Fetcha v0.2.0 — private access (single unlimited plan, invitation-only signup, /admin/access), managed browser (Patchright, auto-escalation), hardened HTTP fingerprints (ordered headers, HTTP/2, TLS profiles, cookie jar, backoff), expanded block detection, markdown/page/links, crawl & map API, dashboard Crawls, SDK 0.2.0 · 7a8260e · fetcha

Fetcha v0.2.0 — private access (single unlimited plan, invitation-only signup, /admin/access), managed browser (Patchright, auto-escalation), hardened HTTP fingerprints (ordered headers, HTTP/2, TLS profiles, cookie jar, backoff), expanded block detection, markdown/page/links, crawl & map API, dashboard Crawls, SDK 0.2.0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 16 days ago (Sep 8, 2026) parent 67d1ab1

117 changed files +11,929 −1,177

modified .env.example +11 −0

		`@@ -32,3 +32,14 @@ SOAX_PASSWORD=`
32	32
33	33	# ---- Admin ----
34	34	ADMIN_EMAILS=you@example.com
	35	+
	36	+# --- v0.2: managed browser, HTTP engine, crawler ---------------------------
	37	+# Managed Chromium (Playwright). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
	38	+FETCHA_BROWSER_ENABLED=1
	39	+FETCHA_BROWSER_CONCURRENCY=6
	40	+# "chromium" = full Chromium build in new headless mode (best anti-bot realism); "chrome" uses an installed Google Chrome.
	41	+FETCHA_BROWSER_CHANNEL=chromium
	42	+# Prefer HTTP/2 towards origins (set 0 to force HTTP/1.1).
	43	+FETCHA_HTTP2=1
	44	+# Crawl jobs executed in parallel by this API process.
	45	+FETCHA_CRAWL_PARALLEL_JOBS=4

modified CLAUDE.md +15 −5

		`@@ -2,15 +2,20 @@`
2	2
3	3	Fetcha (www.fetcha.co) is "Intelligent Web Access Infrastructure": one API (`POST /v1/fetch`) that routes web requests
4	4	across upstream proxy networks (Oxylabs, Decodo, SOAX — never exposed to customers) with smart routing, retries,
5		−sticky sessions, usage metering and a full dashboard.
	5	+sticky sessions, a managed headless browser (`browser: true` + automatic escalation), crawl/map jobs, usage metering
	6	+and a full dashboard. Private platform (v0.2, 2026-09-08): a single `unlimited` plan, invitation-only signup
	7	+(`signup_allowlist`, managed from `/admin/access`), `ADMIN_EMAILS` become admins. No billing.
6	8
7	9	## Layout
8	10	- `apps/web` — Next.js 16 (marketing site, docs, auth, customer dashboard, admin). Port 8220. Proxies `/v1/*` to the API.
9	11	- `apps/api` — Fastify API service (public `/v1/`, `/health`, `/ready`, internal `/internal/` for the dashboard). Port 8221.
10	12	- `packages/core` — zod schemas, error codes, ids/api-key hashing, SSRF policy, geo model, plan limits.
11	13	- `packages/db` — Drizzle schema + migrations (`drizzle/`), `pnpm db:generate\|migrate\|seed`.
12		−- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct). Provider code lives ONLY here.
13		−- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, block detection, escalation).
	14	+- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct), `http.ts` (undici, HTTP/2, browser-ordered headers, Chrome/Firefox/Safari TLS cipher lists, h2→h1 fallback), `fingerprint.ts` (header profiles), `cookies.ts` (jar across hops / sessions). Provider code lives ONLY here.
	15	+- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, fingerprint rotation, jittered backoff, block detection, HTTP → browser escalation).
	16	+- `packages/browser` — `BrowserPool` (Patchright = patched Playwright, full Chromium new-headless, per-context upstream proxy, stealth init script, challenge wait + Turnstile click). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
	17	+- `packages/core/src/markdown.ts` — HTML → Markdown / main text, page metadata + links, URL normalisation, glob/regex matchers; `robots.ts` — robots.txt + sitemap parsing.
	18	+- `apps/api/src/services/crawl.ts` — durable crawl jobs (`crawl_jobs`/`crawl_pages`, in-process worker, resume after restart) and the sync `/v1/map`.
14	19	- `packages/email` — Resend abstraction + React Email templates. `packages/sdk` (JS), `sdk-python/` (Python).
15	20	- `docs/AGENT-BRIEF.md` — product/engineering rules used when generating UI.
16	21
		`@@ -19,7 +24,9 @@ sticky sessions, usage metering and a full dashboard.`
19	24	- Every external URL goes through `assertUrlAllowed()` (SSRF) — including redirects.
20	25	- Business logic in server actions / `apps/api` / packages, not in React components.
21	26	- API keys: shown once, SHA-256 stored. Request IDs `req_…` on every response (`X-Fetcha-Request-ID`).
22		−- Not implemented yet (say so, don't fake): browser mode, `/v1/extract`, Stripe checkout, teams, webhook delivery, OAuth, 2FA.
	27	+- Not implemented yet (say so, don't fake): scripted browser actions (`POST /v1/browser`), `/v1/extract`, teams, webhook delivery (except the crawl `webhook_url` callback), OAuth, 2FA. Billing/Stripe is intentionally absent (private platform).
	28	+- Plans: always `normalizePlan(org.plan)`; never reintroduce tiers. Signup must stay allowlist-gated (Better Auth `user.create.before` hook in `apps/web/src/lib/auth.ts`).
	29	+- Anti-bot honesty: interactive Cloudflare Turnstile challenges are NOT reliably solved (tested 2026-09-08 with Patchright + residential + headful); non-interactive JS challenges, DataDome/PX/Akamai soft blocks usually pass via residential + browser escalation.
23	30
24	31	## Dev
25	32	`cp .env.example .env` (fill provider creds + RESEND_API_KEY), `createdb fetcha`, `pnpm db:migrate && pnpm db:seed`,
		`@@ -27,4 +34,7 @@ sticky sessions, usage metering and a full dashboard.`
27	34
28	35	## Deploy (MacLustr)
29	36	Deployed by `mld` (gateway M1M32) on M3U96a: web 8220 + api 8221, Postgres 17 `fetcha` + Redis local, ngrok `www.fetcha.co`.
30		−Manifest: `deploy/fetcha.mld.json` → `M1M32:~/dispatch/apps/fetcha.json`. See `deploy/README.md`.
	37	+Manifest: `deploy/fetcha.mld.json` → `M1M32:~/dispatch/apps/fetcha.json`. See `deploy/README.md`. Post-sync hooks: `pnpm install`,
	38	+`db:migrate` + `db:seed` (with `ADMIN_EMAILS`), `patchright install chromium`, `next build`. Env knobs: `FETCHA_BROWSER_ENABLED`,
	39	+`FETCHA_BROWSER_CONCURRENCY`, `FETCHA_BROWSER_CHANNEL`, `FETCHA_BROWSER_HEADLESS`, `FETCHA_HTTP2`, `FETCHA_CRAWL_PARALLEL_JOBS`.
	40	+The `browser_enabled` feature flag (admin → Flags) is the global kill switch for browser mode.

modified README.md +4 −2

		`@@ -35,7 +35,9 @@ pnpm dev:web # http://localhost:8220`
35	35	```
36	36	Tests: `pnpm test` · Typecheck: `pnpm typecheck` · Build: `pnpm build`.
37	37
38		−## Status (v0.1.0 — public preview)
	38	+## Status (v0.2.0 — private platform)
39	39	Live: Fetch API, auto routing (residential network class), sticky sessions, Playground, request logs, usage metering,
40		−API keys, projects, admin console, docs, SDK source. Coming: managed browser, structured extraction, Stripe checkout,
	40	+API keys, projects, admin console, docs, SDK source, managed browser rendering (`browser: true`, automatic escalation),
	41	+`format: "markdown"` + page metadata/links, crawl & map jobs (`/v1/crawl`, `/v1/map`). Private platform: invitation-only
	42	+signup, one unlimited plan, no billing. Coming: scripted browser actions, structured extraction,
41	43	teams, webhooks. See `/changelog`.

modified apps/api/package.json +1 −0

		`@@ -11,6 +11,7 @@`
11	11	},
12	12	"dependencies": {
13	13	"@fastify/cors": "^11.0.0",
	14	+ "@fetcha/browser": "workspace:*",
14	15	"@fetcha/core": "workspace:*",
15	16	"@fetcha/db": "workspace:*",
16	17	"@fetcha/email": "workspace:*",

modified apps/api/src/auth.ts +63 −3

		`@@ -1,4 +1,4 @@`
1		−import { FetchaError, hashApiKey, parseApiKeyMode, safeEqual, type ApiKeyScope, type Plan } from "@fetcha/core";
	1	+import { FetchaError, hashApiKey, normalizePlan, parseApiKeyMode, safeEqual, type ApiKeyScope, type Plan } from "@fetcha/core";
2	2	import { apiKeys, db, eq, organizations, projects, sql, users } from "@fetcha/db";
3	3	import { config } from "./config";
4	4	import { getKV } from "./redis";
		`@@ -74,7 +74,7 @@ async function loadPrincipalByKeyHash(hash: string): Promise<ApiPrincipal \| null`
74	74	scopes: (row.scopes ?? []) as ApiKeyScope[],
75	75	organizationId: row.organizationId,
76	76	organizationName: row.organizationName,
77		− plan: row.plan as Plan,
	77	+ plan: normalizePlan(row.plan),
78	78	providerVisibility: row.providerVisibility,
79	79	suspended: row.suspended,
80	80	orgSoftLimitUsd: row.orgSoft,
		`@@ -172,7 +172,67 @@ export async function principalForProject(projectId: string, userId: string): Pr`
172	172	scopes: ["fetch:execute", "sessions:write", "usage:read"],
173	173	organizationId: row.organizationId,
174	174	organizationName: row.organizationName,
175		− plan: row.plan as Plan,
	175	+ plan: normalizePlan(row.plan),
	176	+ providerVisibility: row.providerVisibility,
	177	+ suspended: row.suspended,
	178	+ orgSoftLimitUsd: row.orgSoft,
	179	+ orgHardLimitUsd: row.orgHard,
	180	+ projectId: row.projectId,
	181	+ projectName: row.projectName,
	182	+ projectLogLevel: row.logLevel as ApiPrincipal["projectLogLevel"],
	183	+ projectSoftLimitUsd: row.projSoft,
	184	+ projectHardLimitUsd: row.projHard,
	185	+ projectMonthlyRequestLimit: row.projMonthly,
	186	+ ownerEmail: row.ownerEmail,
	187	+ ownerVerified: row.ownerVerified,
	188	+ internal: true,
	189	+ };
	190	+}
	191	+
	192	+/** Principal for a background crawl job: the project's owner context, with the originating key when still valid. */
	193	+export async function principalForCrawl(projectId: string, apiKeyId: string \| null): Promise<ApiPrincipal> {
	194	+ const [row] = await db
	195	+ .select({
	196	+ organizationId: organizations.id,
	197	+ organizationName: organizations.name,
	198	+ plan: organizations.plan,
	199	+ providerVisibility: organizations.providerVisibility,
	200	+ suspended: organizations.suspended,
	201	+ orgSoft: organizations.softLimitUsd,
	202	+ orgHard: organizations.hardLimitUsd,
	203	+ projectId: projects.id,
	204	+ projectName: projects.name,
	205	+ logLevel: projects.logLevel,
	206	+ projSoft: projects.softLimitUsd,
	207	+ projHard: projects.hardLimitUsd,
	208	+ projMonthly: projects.monthlyRequestLimit,
	209	+ archivedAt: projects.archivedAt,
	210	+ ownerEmail: users.email,
	211	+ ownerVerified: users.emailVerified,
	212	+ ownerBanned: users.banned,
	213	+ })
	214	+ .from(projects)
	215	+ .innerJoin(organizations, eq(projects.organizationId, organizations.id))
	216	+ .innerJoin(users, eq(organizations.ownerUserId, users.id))
	217	+ .where(eq(projects.id, projectId))
	218	+ .limit(1);
	219	+ if (!row) throw new FetchaError("NOT_FOUND", "Project not found.");
	220	+ if (row.archivedAt) throw new FetchaError("FORBIDDEN", "This project is archived.");
	221	+ if (row.suspended \|\| row.ownerBanned) throw new FetchaError("FORBIDDEN", "This organization is suspended.");
	222	+ let keyName: string \| null = null;
	223	+ if (apiKeyId) {
	224	+ const [k] = await db.select({ name: apiKeys.name, revokedAt: apiKeys.revokedAt }).from(apiKeys).where(eq(apiKeys.id, apiKeyId)).limit(1);
	225	+ if (k?.revokedAt) throw new FetchaError("INVALID_API_KEY", "The API key that started this crawl was revoked.");
	226	+ keyName = k?.name ?? null;
	227	+ }
	228	+ return {
	229	+ keyId: apiKeyId,
	230	+ keyName,
	231	+ mode: "live",
	232	+ scopes: ["fetch:execute", "crawl:execute", "sessions:write", "usage:read"],
	233	+ organizationId: row.organizationId,
	234	+ organizationName: row.organizationName,
	235	+ plan: normalizePlan(row.plan),
176	236	providerVisibility: row.providerVisibility,
177	237	suspended: row.suspended,
178	238	orgSoftLimitUsd: row.orgSoft,

modified apps/api/src/limits.ts +18 −0

		`@@ -23,6 +23,24 @@ export async function acquireConcurrency(p: ApiPrincipal): Promise<() => Promise`
23	23	};
24	24	}
25	25
	26	+/** Reserve a managed-browser slot for the organization (separate, smaller pool than plain fetches). */
	27	+export async function acquireBrowserSlot(p: ApiPrincipal): Promise<() => Promise<void>> {
	28	+ const kv = getKV();
	29	+ const limit = PLAN_LIMITS[p.plan].browser_concurrency;
	30	+ const key = `fch:bconc:${p.organizationId}`;
	31	+ const n = await kv.incrWithTtl(key, CONCURRENCY_TTL_SEC);
	32	+ if (n > limit) {
	33	+ await kv.decr(key);
	34	+ throw new FetchaError("CONCURRENCY_LIMIT", `At most ${limit} browser renders can run at once for your organization.`, { details: { limit, scope: "browser" } });
	35	+ }
	36	+ let released = false;
	37	+ return async () => {
	38	+ if (released) return;
	39	+ released = true;
	40	+ await kv.decr(key).catch(() => {});
	41	+ };
	42	+}
	43	+
26	44	/** Sliding-window rate limits: per key (burst), per organization (sustained), per IP (anonymous abuse). */
27	45	export async function checkRateLimits(p: ApiPrincipal, ip: string \| null): Promise<void> {
28	46	const kv = getKV();

added apps/api/src/routes/crawl.ts +96 −0

		`@@ -0,0 +1,96 @@`
	1	+import type { FastifyInstance } from "fastify";
	2	+import { z } from "zod";
	3	+import { FetchaError } from "@fetcha/core";
	4	+import { authenticateApiKey, principalForProject, requireScope } from "../auth";
	5	+import { cancelCrawl, createCrawl, getCrawl, listCrawlPages, listCrawls, mapSite } from "../services/crawl";
	6	+import { getEngine } from "../services/engine";
	7	+
	8	+const pagesQuery = z.object({
	9	+ cursor: z.string().optional(),
	10	+ limit: z.coerce.number().int().min(1).max(500).optional(),
	11	+ status: z.enum(["pending", "success", "blocked", "failed", "skipped"]).optional(),
	12	+ include_content: z
	13	+ .union([z.literal("true"), z.literal("false"), z.literal("1"), z.literal("0")])
	14	+ .optional()
	15	+ .transform((v) => (v === undefined ? undefined : v === "true" \|\| v === "1")),
	16	+});
	17	+
	18	+const clientIp = (req: { headers: Record<string, unknown>; ip: string }) => ((req.headers["x-forwarded-for"] as string \| undefined)?.split(",")[0]?.trim() ?? req.ip) \|\| null;
	19	+const sourceOf = (req: { headers: Record<string, unknown> }) => (/fetcha-sdk/i.test(String(req.headers["user-agent"] ?? "")) ? "sdk" : "api") as "sdk" \| "api";
	20	+
	21	+/** Public crawl & map endpoints. */
	22	+export async function registerCrawlRoutes(app: FastifyInstance) {
	23	+ app.post("/v1/crawl", async (req, reply) => {
	24	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	25	+ requireScope(principal, "fetch:execute");
	26	+ const job = await createCrawl(principal, req.body, sourceOf(req));
	27	+ return reply.status(202).send(job);
	28	+ });
	29	+ app.get<{ Querystring: { limit?: string } }>("/v1/crawl", async (req) => {
	30	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	31	+ return listCrawls(principal, Number(req.query.limit ?? 50) \|\| 50);
	32	+ });
	33	+ app.get<{ Params: { id: string } }>("/v1/crawl/:id", async (req) => {
	34	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	35	+ return getCrawl(principal, req.params.id);
	36	+ });
	37	+ app.get<{ Params: { id: string }; Querystring: Record<string, string \| undefined> }>("/v1/crawl/:id/pages", async (req) => {
	38	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	39	+ const q = pagesQuery.safeParse(req.query);
	40	+ if (!q.success) throw new FetchaError("INVALID_REQUEST", "Invalid query parameters.", { details: { issues: q.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
	41	+ return listCrawlPages(principal, req.params.id, q.data);
	42	+ });
	43	+ app.delete<{ Params: { id: string } }>("/v1/crawl/:id", async (req) => {
	44	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	45	+ requireScope(principal, "fetch:execute");
	46	+ return cancelCrawl(principal, req.params.id);
	47	+ });
	48	+ app.post("/v1/map", async (req) => {
	49	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	50	+ requireScope(principal, "fetch:execute");
	51	+ return mapSite(principal, req.body, sourceOf(req), clientIp(req));
	52	+ });
	53	+}
	54	+
	55	+/** Internal (dashboard) crawl routes; the service-token check is applied by the internal onRequest hook. */
	56	+export async function registerInternalCrawlRoutes(app: FastifyInstance) {
	57	+ const bodySchema = z.object({ project_id: z.string(), user_id: z.string(), options: z.unknown().optional() });
	58	+ type Q = { project_id: string; user_id: string; limit?: string; cursor?: string; status?: string; include_content?: string };
	59	+
	60	+ app.post("/internal/crawls", async (req, reply) => {
	61	+ const parsed = bodySchema.safeParse(req.body);
	62	+ if (!parsed.success) throw new FetchaError("INVALID_REQUEST", "project_id, user_id and options are required.");
	63	+ const principal = await principalForProject(parsed.data.project_id, parsed.data.user_id);
	64	+ const job = await createCrawl(principal, parsed.data.options ?? {}, "playground");
	65	+ return reply.status(202).send(job);
	66	+ });
	67	+ app.get<{ Querystring: Q }>("/internal/crawls", async (req) => {
	68	+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
	69	+ return listCrawls(principal, Number(req.query.limit ?? 50) \|\| 50);
	70	+ });
	71	+ app.get<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id", async (req) => {
	72	+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
	73	+ return getCrawl(principal, req.params.id);
	74	+ });
	75	+ app.get<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id/pages", async (req) => {
	76	+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
	77	+ const q = pagesQuery.safeParse(req.query);
	78	+ if (!q.success) throw new FetchaError("INVALID_REQUEST", "Invalid query parameters.");
	79	+ return listCrawlPages(principal, req.params.id, q.data);
	80	+ });
	81	+ app.delete<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id", async (req) => {
	82	+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
	83	+ return cancelCrawl(principal, req.params.id);
	84	+ });
	85	+ app.post("/internal/map", async (req) => {
	86	+ const parsed = bodySchema.safeParse(req.body);
	87	+ if (!parsed.success) throw new FetchaError("INVALID_REQUEST", "project_id, user_id and options are required.");
	88	+ const principal = await principalForProject(parsed.data.project_id, parsed.data.user_id);
	89	+ return mapSite(principal, parsed.data.options ?? {}, "playground", (req.headers["x-forwarded-for"] as string \| undefined)?.split(",")[0]?.trim() ?? req.ip);
	90	+ });
	91	+ app.get("/internal/browser", async () => {
	92	+ const engine = await getEngine();
	93	+ const s = engine.browser.status();
	94	+ return { ...s, enabled: engine.browserEnabled, flag_enabled: engine.browserEnabled \|\| !engine.browser.enabled ? engine.browserEnabled : false };
	95	+ });
	96	+}

modified apps/api/src/routes/fetch.ts +34 −11

		`@@ -1,11 +1,11 @@`
1	1	import { FetchaError, PLAN_LIMITS, extractDomain, fetchRequestSchema, newId, type FetchRequest } from "@fetcha/core";
2	2	import { db, eq, proxySessions, sql } from "@fetcha/db";
3	3	import type { ProviderId } from "@fetcha/providers";
4		−import type { AttemptRecord } from "@fetcha/routing";
	4	+import type { AttemptRecord, SerializedCookie } from "@fetcha/routing";
5	5	import type { ConcreteNetwork } from "@fetcha/core";
6	6	import type { ApiPrincipal } from "../auth";
7	7	import { config } from "../config";
8		−import { acquireConcurrency, checkMonthlyLimits, checkRateLimits, invalidateUsageCache } from "../limits";
	8	+import { acquireBrowserSlot, acquireConcurrency, checkMonthlyLimits, checkRateLimits, invalidateUsageCache } from "../limits";
9	9	import { getEngine } from "../services/engine";
10	10	import { completeRequest, createRequestRow, failRequest, recordAttempt, type RequestContext } from "../services/persist";
11	11	import { checkSpendAlerts, notifyHardLimit, recordAbuse } from "../services/alerts";
		`@@ -13,7 +13,7 @@ import { checkSpendAlerts, notifyHardLimit, recordAbuse } from "../services/aler`
13	13	export interface FetchHandlerInput {
14	14	principal: ApiPrincipal;
15	15	body: unknown;
16		− source: "api" \| "playground" \| "sdk";
	16	+ source: "api" \| "playground" \| "sdk" \| "crawl";
17	17	clientIp: string \| null;
18	18	userAgent: string \| null;
19	19	}
		`@@ -22,10 +22,12 @@ export interface FetchHandlerOutput {`
22	22	requestId: string;
23	23	status: number;
24	24	body: unknown;
	25	+ /** Internal: execution summary for callers that need more than the public document (crawler). */
	26	+ summary?: { mode: "http" \| "browser"; bytes: number; costUsd: number; attempts: number; domain: string };
25	27	}
26	28
27	29	/**
28		− * Shared /v1/fetch pipeline used by the public API and the dashboard playground.
	30	+ * Shared /v1/fetch pipeline used by the public API, the dashboard playground and the crawler.
29	31	* Order matters: validate → limits → SSRF (inside executor) → route → persist.
30	32	*/
31	33	export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandlerOutput> {
		`@@ -46,12 +48,12 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
46	48	if (req.network !== "auto" && !limits.networks.includes(req.network)) {
47	49	throw new FetchaError("NETWORK_UNAVAILABLE", `The "${req.network}" network is not included in the ${limits.label} plan.`, { requestId });
48	50	}
49		− if (req.browser) throw new FetchaError("BROWSER_UNAVAILABLE", undefined, { requestId });
	51	+ const engine = await getEngine();
	52	+ if (req.browser && (!limits.browser \|\| !engine.browserEnabled)) throw new FetchaError("BROWSER_UNAVAILABLE", undefined, { requestId });
50	53
51	54	await checkRateLimits(principal, input.clientIp);
52		− let monthly: Awaited<ReturnType<typeof checkMonthlyLimits>>;
53	55	try {
54		− monthly = await checkMonthlyLimits(principal);
	56	+ await checkMonthlyLimits(principal);
55	57	} catch (e) {
56	58	if (e instanceof FetchaError && e.code === "USAGE_LIMIT_REACHED" && (principal.projectHardLimitUsd !== null \|\| principal.orgHardLimitUsd !== null)) {
57	59	const { monthlyUsage } = await import("../limits");
		`@@ -60,6 +62,15 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
60	62	throw e;
61	63	}
62	64	const release = await acquireConcurrency(principal);
	65	+ let releaseBrowser: (() => Promise<void>) \| null = null;
	66	+ if (req.browser) {
	67	+ try {
	68	+ releaseBrowser = await acquireBrowserSlot(principal);
	69	+ } catch (e) {
	70	+ await release();
	71	+ throw e;
	72	+ }
	73	+ }
63	74
64	75	const domain = extractDomain(req.url);
65	76	const ctx: RequestContext = {
		`@@ -77,19 +88,22 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
77	88	const attempts: AttemptRecord[] = [];
78	89	try {
79	90	await createRequestRow(ctx, req, domain);
80		− const engine = await getEngine();
81	91
82		− // Resolve a Fetcha session (sess_…) into a provider-pinned sticky key.
	92	+ // Resolve a Fetcha session (sess_…) into a provider-pinned sticky key (+ its cookie jar).
83	93	let sessionKey: string \| null = null;
84	94	let sessionProvider: ProviderId \| null = null;
85	95	let sessionNetwork: ConcreteNetwork \| null = null;
	96	+ let sessionCookies: SerializedCookie[] \| null = null;
	97	+ let sessionId: string \| null = null;
86	98	if (req.session) {
87	99	const [s] = await db.select().from(proxySessions).where(eq(proxySessions.id, req.session)).limit(1);
88	100	if (!s \|\| s.projectId !== principal.projectId) throw new FetchaError("SESSION_NOT_FOUND", undefined, { requestId });
89	101	if (s.status !== "active" \|\| s.expiresAt.getTime() < Date.now()) throw new FetchaError("SESSION_EXPIRED", undefined, { requestId });
90	102	sessionKey = s.stickyKey;
	103	+ sessionId = s.id;
91	104	sessionProvider = s.provider as ProviderId;
92	105	sessionNetwork = s.network as ConcreteNetwork;
	106	+ sessionCookies = (s.cookies ?? []) as unknown as SerializedCookie[];
93	107	if (!req.country && s.country) req.country = s.country;
94	108	if (req.network === "auto") req.network = s.network as FetchRequest["network"];
95	109	db.update(proxySessions)
		`@@ -106,9 +120,12 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
106	120	sessionKey,
107	121	sessionProvider,
108	122	sessionNetwork,
	123	+ sessionCookies,
109	124	knowledge,
110	125	providerVisibility: principal.providerVisibility,
111	126	maxResponseBytes: Math.min(req.max_response_bytes ?? config.maxResponseBytesDefault, config.maxResponseBytesDefault),
	127	+ browserPool: engine.browserEnabled && limits.browser ? engine.browser : null,
	128	+ browserAllowed: limits.browser,
112	129	onAttempt: async (a) => {
113	130	attempts.push(a);
114	131	await recordAttempt(requestId, a).catch(() => {});
		`@@ -116,13 +133,18 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
116	133	});
117	134	const latencyMs = Math.round(performance.now() - started);
118	135	await completeRequest(ctx, req, result, latencyMs);
	136	+ if (sessionId && result.cookies.length) {
	137	+ db.update(proxySessions)
	138	+ .set({ cookies: result.cookies.slice(-200) as unknown as Array<{ name: string; value: string; domain?: string; path?: string }> })
	139	+ .where(eq(proxySessions.id, sessionId))
	140	+ .catch(() => {});
	141	+ }
119	142	await invalidateUsageCache(principal.organizationId, principal.projectId);
120	143	if (principal.projectSoftLimitUsd !== null \|\| principal.orgSoftLimitUsd !== null) {
121	144	const { monthlyUsage } = await import("../limits");
122	145	monthlyUsage(principal).then((u) => checkSpendAlerts(principal, u)).catch(() => {});
123	146	}
124		− void monthly;
125		− return { requestId, status: 200, body: result.body };
	147	+ return { requestId, status: 200, body: result.body, summary: { mode: result.mode, bytes: result.bytesIn + result.bytesOut, costUsd: result.costUsd, attempts: result.attempts.length, domain } };
126	148	} catch (e) {
127	149	const latencyMs = Math.round(performance.now() - started);
128	150	const err = e instanceof FetchaError ? e : new FetchaError("INTERNAL_ERROR", undefined, { requestId, cause: e });
		`@@ -131,6 +153,7 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
131	153	if (!(e instanceof FetchaError)) console.error(`[fetch] ${requestId} internal error`, e);
132	154	throw Object.assign(err, { requestId });
133	155	} finally {
	156	+ await releaseBrowser?.();
134	157	await release();
135	158	}
136	159	}

modified apps/api/src/routes/misc.ts +3 −1

		`@@ -40,7 +40,9 @@ export async function usageResponse(p: ApiPrincipal) {`
40	40	latency_p50_ms: Math.round(agg?.p50 ?? 0),
41	41	latency_p95_ms: Math.round(agg?.p95 ?? 0),
42	42	},
43		− remaining_requests: Math.max(0, limits.monthly_requests - usage.requests),
	43	+ remaining_requests: limits.monthly_requests >= Number.MAX_SAFE_INTEGER ? null : Math.max(0, limits.monthly_requests - usage.requests),
	44	+ browser: { enabled: true, concurrency: limits.browser_concurrency },
	45	+ crawl: { max_pages: limits.crawl_max_pages, concurrent_jobs: limits.crawl_concurrent_jobs },
44	46	};
45	47	}
46	48

modified apps/api/src/server.ts +13 −4

		`@@ -3,11 +3,14 @@ import cors from "@fastify/cors";`
3	3	import { FetchaError, isFetchaError, newId } from "@fetcha/core";
4	4	import { closeDb, db, sql } from "@fetcha/db";
5	5	import { closeAllDispatchers } from "@fetcha/providers";
	6	+import { closeBrowserPool } from "@fetcha/browser";
6	7	import { authenticateApiKey, requireScope } from "./auth";
7	8	import { assertProductionConfig, config } from "./config";
8	9	import { closeKV, getKV } from "./redis";
9	10	import { handleFetch } from "./routes/fetch";
10	11	import { registerInternalRoutes } from "./routes/internal";
	12	+import { registerCrawlRoutes, registerInternalCrawlRoutes } from "./routes/crawl";
	13	+import { startCrawlWorker } from "./services/crawl";
11	14	import { meResponse, usageResponse } from "./routes/misc";
12	15	import { closeSession, createSession, getSession, listSessions } from "./routes/sessions";
13	16	import { getEngine, startHealthLoop } from "./services/engine";
		`@@ -74,7 +77,7 @@ export async function buildServer() {`
74	77	const engine = await getEngine().catch(() => null);
75	78	checks.providers = Boolean(engine && engine.registry.available().length > 0);
76	79	const ok = checks.database && checks.cache && checks.providers;
77		− return reply.status(ok ? 200 : 503).send({ status: ok ? "ready" : "degraded", checks, available_networks: engine?.registry.availableNetworks() ?? [] });
	80	+ return reply.status(ok ? 200 : 503).send({ status: ok ? "ready" : "degraded", checks, available_networks: engine?.registry.availableNetworks() ?? [], browser: engine ? { enabled: engine.browserEnabled, launched: engine.browser.status().launched } : null });
78	81	});
79	82
80	83	// ---- Public API v1 ----------------------------------------------------------
		`@@ -118,15 +121,18 @@ export async function buildServer() {`
118	121	return usageResponse(principal);
119	122	});
120	123
	124	+ await registerCrawlRoutes(app);
	125	+
121	126	// Explicitly unavailable surfaces (never fake functionality).
122	127	app.post("/v1/browser", async () => {
123		− throw new FetchaError("BROWSER_UNAVAILABLE", "Browser actions are not yet available. Follow the changelog at https://www.fetcha.co/changelog.");
	128	+ throw new FetchaError("BROWSER_UNAVAILABLE", "Scripted browser actions are not available yet. Rendered fetches are: send `browser: true` to POST /v1/fetch.");
124	129	});
125	130	app.post("/v1/extract", async () => {
126		− throw new FetchaError("INVALID_REQUEST", "Structured extraction is not yet available on this plan. Follow the changelog at https://www.fetcha.co/changelog.");
	131	+ throw new FetchaError("INVALID_REQUEST", "Structured extraction is not yet available. Use `format: \"markdown\"` on POST /v1/fetch in the meantime.");
127	132	});
128	133
129	134	await registerInternalRoutes(app);
	135	+ await registerInternalCrawlRoutes(app);
130	136	return app;
131	137	}
132	138
		`@@ -136,12 +142,15 @@ async function main() {`
136	142	const engine = await getEngine();
137	143	app.log.info({ providers: engine.registry.available().map((p) => p.id), networks: engine.registry.availableNetworks() }, "providers loaded");
138	144	const stopHealth = startHealthLoop(engine, app.log);
	145	+ const stopCrawl = startCrawlWorker(app.log);
	146	+ app.log.info({ browser: engine.browser.status() }, "managed browser pool ready (lazy launch)");
139	147
140	148	const shutdown = async (signal: string) => {
141	149	app.log.info(`${signal} received, shutting down`);
142	150	stopHealth();
	151	+ await Promise.race([stopCrawl(), new Promise((r) => setTimeout(r, 8000))]);
143	152	await app.close();
144		− await Promise.all([closeAllDispatchers(), closeKV(), closeDb()]);
	153	+ await Promise.all([closeAllDispatchers(), closeBrowserPool(), closeKV(), closeDb()]);
145	154	process.exit(0);
146	155	};
147	156	process.on("SIGINT", () => void shutdown("SIGINT"));

added apps/api/src/services/crawl.ts +592 −0

		`@@ -0,0 +1,592 @@`
	1	+import {
	2	+ FetchaError,
	3	+ PLAN_LIMITS,
	4	+ assertUrlAllowed,
	5	+ crawlCreateSchema,
	6	+ extractDomain,
	7	+ extractPageMetadata,
	8	+ looksLikePage,
	9	+ mapCreateSchema,
	10	+ newId,
	11	+ normalizeCrawlUrl,
	12	+ parseRobots,
	13	+ parseSitemap,
	14	+ registrableHost,
	15	+ robotsAllows,
	16	+ urlPatternMatcher,
	17	+ type CrawlCreateInput,
	18	+ type FetchResponseBody,
	19	+ type MapCreateInput,
	20	+ type RobotsRules,
	21	+} from "@fetcha/core";
	22	+import { and, crawlJobs, crawlPages, db, desc, eq, inArray, lt, sql, type CrawlJob, type CrawlPage } from "@fetcha/db";
	23	+import { hostname } from "node:os";
	24	+import type { ApiPrincipal } from "../auth";
	25	+import { handleFetch, type FetchHandlerOutput } from "../routes/fetch";
	26	+
	27	+/**
	28	+ * Crawl service: durable jobs (crawl_jobs / crawl_pages) executed in-process by a small worker
	29	+ * loop. Every page goes through `handleFetch`, so quotas, routing intelligence, block detection and
	30	+ * browser escalation apply exactly as for a single fetch.
	31	+ */
	32	+
	33	+const WORKER_ID = `${hostname()}:${process.pid}`;
	34	+const PAGE_CONTENT_CAP = 600_000; // chars stored per page
	35	+const HEARTBEAT_MS = 10_000;
	36	+const STALE_MS = 3 * 60_000;
	37	+
	38	+export function serializeJob(j: CrawlJob) {
	39	+ return {
	40	+ id: j.id,
	41	+ status: j.status,
	42	+ label: j.label,
	43	+ seed_url: j.seedUrl,
	44	+ domain: j.domain,
	45	+ source: j.source,
	46	+ options: j.options,
	47	+ stats: { discovered: j.pagesDiscovered, fetched: j.pagesFetched, ok: j.pagesOk, blocked: j.pagesBlocked, failed: j.pagesFailed, bytes: j.bytes },
	48	+ error: j.errorCode ? { code: j.errorCode, message: j.errorMessage } : null,
	49	+ webhook_status: j.webhookStatus,
	50	+ created_at: j.createdAt.toISOString(),
	51	+ started_at: j.startedAt?.toISOString() ?? null,
	52	+ completed_at: j.completedAt?.toISOString() ?? null,
	53	+ };
	54	+}
	55	+
	56	+export function serializePage(p: CrawlPage, includeContent = true) {
	57	+ return {
	58	+ id: p.id,
	59	+ url: p.url,
	60	+ final_url: p.finalUrl,
	61	+ depth: p.depth,
	62	+ parent_url: p.parentUrl,
	63	+ status: p.status,
	64	+ http_status: p.httpStatus,
	65	+ error_code: p.errorCode,
	66	+ request_id: p.requestId,
	67	+ title: p.title,
	68	+ description: p.description,
	69	+ content_type: p.contentType,
	70	+ content: includeContent ? p.content : undefined,
	71	+ links_count: p.linksCount,
	72	+ bytes: p.bytes,
	73	+ duration_ms: p.durationMs,
	74	+ mode: p.mode,
	75	+ fetched_at: p.fetchedAt?.toISOString() ?? null,
	76	+ };
	77	+}
	78	+
	79	+// ---------------------------------------------------------------------------
	80	+// Public operations
	81	+// ---------------------------------------------------------------------------
	82	+export async function createCrawl(principal: ApiPrincipal, body: unknown, source: "api" \| "playground" \| "sdk") {
	83	+ const parsed = crawlCreateSchema.safeParse(body ?? {});
	84	+ if (!parsed.success) {
	85	+ throw new FetchaError("INVALID_REQUEST", "Invalid crawl options.", { details: { issues: parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
	86	+ }
	87	+ const opts = parsed.data;
	88	+ const limits = PLAN_LIMITS[principal.plan];
	89	+ if (opts.max_pages > limits.crawl_max_pages) opts.max_pages = limits.crawl_max_pages;
	90	+ if (opts.timeout > limits.max_timeout_ms) opts.timeout = limits.max_timeout_ms;
	91	+ if (opts.network !== "auto" && !limits.networks.includes(opts.network)) throw new FetchaError("NETWORK_UNAVAILABLE");
	92	+ const allowed = await assertUrlAllowed(opts.url);
	93	+ const seed = normalizeCrawlUrl(allowed.url.toString());
	94	+ if (!seed) throw new FetchaError("INVALID_REQUEST", "The seed URL is invalid.");
	95	+ if (opts.webhook_url) await assertUrlAllowed(opts.webhook_url);
	96	+
	97	+ const [active] = await db
	98	+ .select({ n: sql<number>`count(*)::int` })
	99	+ .from(crawlJobs)
	100	+ .where(and(eq(crawlJobs.organizationId, principal.organizationId), inArray(crawlJobs.status, ["queued", "running"])));
	101	+ if ((active?.n ?? 0) >= limits.crawl_concurrent_jobs) {
	102	+ throw new FetchaError("CRAWL_LIMIT_REACHED", `At most ${limits.crawl_concurrent_jobs} crawl jobs can be queued or running at once.`, { details: { limit: limits.crawl_concurrent_jobs } });
	103	+ }
	104	+
	105	+ const id = newId("crawl");
	106	+ await db.insert(crawlJobs).values({
	107	+ id,
	108	+ organizationId: principal.organizationId,
	109	+ projectId: principal.projectId,
	110	+ apiKeyId: principal.keyId,
	111	+ source,
	112	+ label: opts.label ?? null,
	113	+ seedUrl: seed,
	114	+ domain: extractDomain(seed),
	115	+ options: { ...opts, url: seed } as Record<string, unknown>,
	116	+ status: "queued",
	117	+ pagesDiscovered: 1,
	118	+ });
	119	+ await db.insert(crawlPages).values({ id: newId("cpg"), jobId: id, url: seed, depth: 0, status: "pending" }).onConflictDoNothing();
	120	+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, id)).limit(1);
	121	+ kick();
	122	+ return serializeJob(job!);
	123	+}
	124	+
	125	+export async function getCrawl(principal: ApiPrincipal, id: string) {
	126	+ const [job] = await db.select().from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
	127	+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
	128	+ return serializeJob(job);
	129	+}
	130	+
	131	+export async function listCrawls(principal: ApiPrincipal, limit = 50) {
	132	+ const rows = await db.select().from(crawlJobs).where(eq(crawlJobs.projectId, principal.projectId)).orderBy(desc(crawlJobs.createdAt)).limit(Math.min(Math.max(1, limit), 200));
	133	+ return { data: rows.map(serializeJob) };
	134	+}
	135	+
	136	+export async function listCrawlPages(principal: ApiPrincipal, id: string, q: { cursor?: string; limit?: number; status?: string; include_content?: boolean }) {
	137	+ const [job] = await db.select({ id: crawlJobs.id }).from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
	138	+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
	139	+ const limit = Math.min(Math.max(1, q.limit ?? 100), 500);
	140	+ const conds = [eq(crawlPages.jobId, id)];
	141	+ if (q.status) conds.push(eq(crawlPages.status, q.status));
	142	+ else conds.push(sql`${crawlPages.status} <> 'pending'`);
	143	+ if (q.cursor) {
	144	+ const [ts, cid] = decodeCursor(q.cursor);
	145	+ conds.push(sql`(${crawlPages.createdAt}, ${crawlPages.id}) > (${new Date(ts)}, ${cid})`);
	146	+ }
	147	+ const rows = await db
	148	+ .select()
	149	+ .from(crawlPages)
	150	+ .where(and(...conds))
	151	+ .orderBy(crawlPages.createdAt, crawlPages.id)
	152	+ .limit(limit + 1);
	153	+ const page = rows.slice(0, limit);
	154	+ const last = page.at(-1);
	155	+ return { data: page.map((p) => serializePage(p, q.include_content !== false)), next_cursor: rows.length > limit && last ? encodeCursor(last.createdAt, last.id) : null };
	156	+}
	157	+
	158	+export async function cancelCrawl(principal: ApiPrincipal, id: string) {
	159	+ const res = await db
	160	+ .update(crawlJobs)
	161	+ .set({ status: "cancelled", completedAt: new Date() })
	162	+ .where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId), inArray(crawlJobs.status, ["queued", "running"])))
	163	+ .returning({ id: crawlJobs.id });
	164	+ if (!res.length) {
	165	+ const [job] = await db.select({ id: crawlJobs.id, status: crawlJobs.status }).from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
	166	+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
	167	+ return { id, status: job.status };
	168	+ }
	169	+ cancelled.add(id);
	170	+ return { id, status: "cancelled" };
	171	+}
	172	+
	173	+function encodeCursor(d: Date, id: string): string {
	174	+ return Buffer.from(`${d.toISOString()}\|${id}`).toString("base64url");
	175	+}
	176	+function decodeCursor(c: string): [string, string] {
	177	+ try {
	178	+ const [ts, id] = Buffer.from(c, "base64url").toString("utf8").split("\|");
	179	+ if (!ts \|\| !id \|\| Number.isNaN(Date.parse(ts))) throw new Error("bad");
	180	+ return [ts, id];
	181	+ } catch {
	182	+ throw new FetchaError("INVALID_REQUEST", "Invalid cursor.");
	183	+ }
	184	+}
	185	+
	186	+// ---------------------------------------------------------------------------
	187	+// Map (sync)
	188	+// ---------------------------------------------------------------------------
	189	+export async function mapSite(principal: ApiPrincipal, body: unknown, source: "api" \| "playground" \| "sdk", clientIp: string \| null) {
	190	+ const parsed = mapCreateSchema.safeParse(body ?? {});
	191	+ if (!parsed.success) {
	192	+ throw new FetchaError("INVALID_REQUEST", "Invalid map options.", { details: { issues: parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
	193	+ }
	194	+ const opts = parsed.data;
	195	+ const allowed = await assertUrlAllowed(opts.url);
	196	+ const seed = normalizeCrawlUrl(allowed.url.toString())!;
	197	+ const seedUrl = new URL(seed);
	198	+ const host = registrableHost(seedUrl.hostname);
	199	+ const inScope = scopeFilter(seedUrl, opts.same_domain, opts.allow_subdomains);
	200	+ const search = opts.search ? urlPatternMatcher([opts.search.includes("") \|\| /^\/.+\/[a-z]$/.test(opts.search) ? opts.search : `${opts.search}`]) : null;
	201	+ const deadline = Date.now() + Math.min(55_000, opts.timeout * 2);
	202	+ const found = new Map<string, "sitemap" \| "links">();
	203	+ const add = (u: string, src: "sitemap" \| "links") => {
	204	+ const n = normalizeCrawlUrl(u);
	205	+ if (!n \|\| found.has(n) \|\| !inScope(n) \|\| !looksLikePage(n)) return;
	206	+ if (search && !search(n)) return;
	207	+ found.set(n, src);
	208	+ };
	209	+ const fetchText = async (url: string, format: "raw" \| "html" = "raw", extra: Record<string, unknown> = {}): Promise<FetchResponseBody \| null> => {
	210	+ if (Date.now() > deadline - 2000) return null;
	211	+ try {
	212	+ const out = await handleFetch({ principal, body: { url, format, timeout: Math.min(opts.timeout, Math.max(3000, deadline - Date.now() - 1000)), country: opts.country, network: opts.network, browser_fallback: false, retries: 1, ...extra }, source, clientIp, userAgent: `fetcha-map/${source}` });
	213	+ return out.body as FetchResponseBody;
	214	+ } catch {
	215	+ return null;
	216	+ }
	217	+ };
	218	+ let sitemapCount = 0;
	219	+ let linksCount = 0;
	220	+ if (opts.use_sitemap) {
	221	+ const sitemapUrls = new Set<string>([`${seedUrl.origin}/sitemap.xml`, `${seedUrl.origin}/sitemap_index.xml`]);
	222	+ const robots = await fetchText(`${seedUrl.origin}/robots.txt`);
	223	+ if (robots?.success && robots.content) for (const s of parseRobots(robots.content).sitemaps) sitemapUrls.add(s);
	224	+ const queue = [...sitemapUrls];
	225	+ const seen = new Set<string>();
	226	+ while (queue.length && seen.size < 25 && found.size < opts.limit && Date.now() < deadline - 3000) {
	227	+ const u = queue.shift()!;
	228	+ if (seen.has(u)) continue;
	229	+ seen.add(u);
	230	+ const r = await fetchText(u);
	231	+ if (!r?.success \|\| !r.content \|\| !/<(urlset\|sitemapindex)\|<loc>/i.test(r.content.slice(0, 5000)) && !/^https?:\/\//m.test(r.content.slice(0, 500))) continue;
	232	+ const { urls, sitemaps } = parseSitemap(r.content);
	233	+ for (const s of sitemaps) if (host === safeHost(s) \|\| opts.allow_subdomains) queue.push(s);
	234	+ for (const x of urls) {
	235	+ if (found.size >= opts.limit) break;
	236	+ const before = found.size;
	237	+ add(x, "sitemap");
	238	+ if (found.size > before) sitemapCount++;
	239	+ }
	240	+ }
	241	+ }
	242	+ if (opts.use_links && found.size < opts.limit && Date.now() < deadline - 3000) {
	243	+ const r = await fetchText(seed, "html", { links: true });
	244	+ if (r?.success && r.links) {
	245	+ for (const l of r.links) {
	246	+ if (found.size >= opts.limit) break;
	247	+ const before = found.size;
	248	+ add(l.url, "links");
	249	+ if (found.size > before) linksCount++;
	250	+ }
	251	+ } else if (r?.content) {
	252	+ const meta = extractPageMetadata(r.content, r.final_url);
	253	+ for (const l of meta.links) {
	254	+ if (found.size >= opts.limit) break;
	255	+ const before = found.size;
	256	+ add(l.url, "links");
	257	+ if (found.size > before) linksCount++;
	258	+ }
	259	+ }
	260	+ }
	261	+ const urls = [...found.keys()];
	262	+ return { url: seed, count: urls.length, urls, sources: { sitemap: sitemapCount, links: linksCount }, truncated: urls.length >= opts.limit };
	263	+}
	264	+
	265	+function safeHost(u: string): string {
	266	+ try {
	267	+ return registrableHost(new URL(u).hostname);
	268	+ } catch {
	269	+ return "";
	270	+ }
	271	+}
	272	+
	273	+function scopeFilter(seed: URL, sameDomain: boolean, allowSubdomains: boolean): (url: string) => boolean {
	274	+ const seedHost = seed.hostname.toLowerCase().replace(/^www\./, "");
	275	+ const reg = registrableHost(seed.hostname);
	276	+ return (url: string) => {
	277	+ if (!sameDomain) return true;
	278	+ let h: string;
	279	+ try {
	280	+ h = new URL(url).hostname.toLowerCase().replace(/^www\./, "");
	281	+ } catch {
	282	+ return false;
	283	+ }
	284	+ if (h === seedHost) return true;
	285	+ if (allowSubdomains) return registrableHost(h) === reg;
	286	+ return false;
	287	+ };
	288	+}
	289	+
	290	+// ---------------------------------------------------------------------------
	291	+// Worker
	292	+// ---------------------------------------------------------------------------
	293	+const cancelled = new Set<string>();
	294	+let running = false;
	295	+let stopped = false;
	296	+let wake: (() => void) \| null = null;
	297	+const activeJobs = new Map<string, Promise<void>>();
	298	+const MAX_PARALLEL_JOBS = Number(process.env.FETCHA_CRAWL_PARALLEL_JOBS ?? 4);
	299	+
	300	+function kick() {
	301	+ wake?.();
	302	+}
	303	+
	304	+export function startCrawlWorker(log: { info: (m: string) => void; warn: (m: string) => void }): () => Promise<void> {
	305	+ if (running) return async () => {};
	306	+ running = true;
	307	+ stopped = false;
	308	+ const loop = async () => {
	309	+ // Re-queue jobs left "running" by a previous process (crash / redeploy).
	310	+ await db
	311	+ .update(crawlJobs)
	312	+ .set({ status: "queued", workerId: null })
	313	+ .where(and(eq(crawlJobs.status, "running"), lt(crawlJobs.heartbeatAt, new Date(Date.now() - STALE_MS))))
	314	+ .catch(() => {});
	315	+ while (!stopped) {
	316	+ try {
	317	+ if (activeJobs.size < MAX_PARALLEL_JOBS) {
	318	+ const claimed = await claimJob();
	319	+ if (claimed) {
	320	+ const p = runJob(claimed, log)
	321	+ .catch((e) => log.warn(`crawl ${claimed.id} crashed: ${(e as Error).message}`))
	322	+ .finally(() => activeJobs.delete(claimed.id));
	323	+ activeJobs.set(claimed.id, p);
	324	+ continue;
	325	+ }
	326	+ }
	327	+ } catch (e) {
	328	+ log.warn(`crawl worker loop error: ${(e as Error).message}`);
	329	+ }
	330	+ await new Promise<void>((r) => {
	331	+ wake = r;
	332	+ setTimeout(r, 4000);
	333	+ });
	334	+ wake = null;
	335	+ }
	336	+ };
	337	+ void loop();
	338	+ log.info(`crawl worker started (${WORKER_ID}, parallel jobs=${MAX_PARALLEL_JOBS})`);
	339	+ return async () => {
	340	+ stopped = true;
	341	+ kick();
	342	+ await Promise.allSettled([...activeJobs.values()]);
	343	+ running = false;
	344	+ };
	345	+}
	346	+
	347	+async function claimJob(): Promise<CrawlJob \| null> {
	348	+ const rows = await db.execute(sql`
	349	+ update crawl_jobs set status = 'running', worker_id = ${WORKER_ID}, started_at = coalesce(started_at, now()), heartbeat_at = now()
	350	+ where id = (select id from crawl_jobs where status = 'queued' order by created_at asc limit 1 for update skip locked)
	351	+ returning *
	352	+ `);
	353	+ const r = (rows as unknown as { rows: Record<string, unknown>[] }).rows?.[0];
	354	+ if (!r) return null;
	355	+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, r.id as string)).limit(1);
	356	+ return job ?? null;
	357	+}
	358	+
	359	+interface Frontier {
	360	+ url: string;
	361	+ depth: number;
	362	+ parent: string \| null;
	363	+}
	364	+
	365	+async function runJob(job: CrawlJob, log: { info: (m: string) => void; warn: (m: string) => void }): Promise<void> {
	366	+ const opts = crawlCreateSchema.parse(job.options) as CrawlCreateInput;
	367	+ const { principalForCrawl } = await import("../auth");
	368	+ let principal: ApiPrincipal;
	369	+ try {
	370	+ principal = await principalForCrawl(job.projectId, job.apiKeyId);
	371	+ } catch (e) {
	372	+ await finishJob(job.id, "failed", { code: (e as FetchaError).code ?? "INTERNAL_ERROR", message: (e as Error).message });
	373	+ return;
	374	+ }
	375	+ const seedUrl = new URL(job.seedUrl);
	376	+ const inScope = scopeFilter(seedUrl, opts.same_domain, opts.allow_subdomains);
	377	+ const include = urlPatternMatcher(opts.include_patterns);
	378	+ const exclude = urlPatternMatcher(opts.exclude_patterns);
	379	+ const seen = new Set<string>();
	380	+ const frontier: Frontier[] = [];
	381	+ let discovered = 0;
	382	+ let fetched = 0;
	383	+ let ok = 0;
	384	+ let blocked = 0;
	385	+ let failed = 0;
	386	+ let bytes = 0;
	387	+ let costUsd = 0;
	388	+ let robots: RobotsRules \| null = null;
	389	+ let delayMs = opts.delay_ms;
	390	+
	391	+ // Resume support: already-known pages of this job.
	392	+ const existing = await db.select({ url: crawlPages.url, status: crawlPages.status, depth: crawlPages.depth, parentUrl: crawlPages.parentUrl }).from(crawlPages).where(eq(crawlPages.jobId, job.id));
	393	+ for (const p of existing) {
	394	+ seen.add(p.url);
	395	+ discovered++;
	396	+ if (p.status === "pending") frontier.push({ url: p.url, depth: p.depth, parent: p.parentUrl });
	397	+ else {
	398	+ fetched++;
	399	+ if (p.status === "success") ok++;
	400	+ else if (p.status === "blocked") blocked++;
	401	+ else if (p.status === "failed") failed++;
	402	+ }
	403	+ }
	404	+ bytes = job.bytes;
	405	+ costUsd = job.costUsd;
	406	+
	407	+ const fetchPage = async (url: string, extra: Record<string, unknown> = {}): Promise<FetchHandlerOutput \| FetchaError> => {
	408	+ try {
	409	+ return await handleFetch({
	410	+ principal,
	411	+ body: {
	412	+ url,
	413	+ format: opts.format === "html" ? "html" : opts.format,
	414	+ timeout: opts.timeout,
	415	+ country: opts.country,
	416	+ network: opts.network,
	417	+ browser: opts.browser,
	418	+ browser_fallback: opts.browser_fallback,
	419	+ headers: opts.headers,
	420	+ links: true,
	421	+ ...extra,
	422	+ },
	423	+ source: "crawl",
	424	+ clientIp: null,
	425	+ userAgent: `fetcha-crawl/${job.id}`,
	426	+ });
	427	+ } catch (e) {
	428	+ return e instanceof FetchaError ? e : new FetchaError("INTERNAL_ERROR", (e as Error).message);
	429	+ }
	430	+ };
	431	+
	432	+ // robots.txt + optional sitemap seeding
	433	+ if (opts.respect_robots) {
	434	+ const r = await fetchPage(`${seedUrl.origin}/robots.txt`, { format: "raw", links: false, browser: false, browser_fallback: false, retries: 1, timeout: Math.min(opts.timeout, 15_000) });
	435	+ if (!(r instanceof FetchaError)) {
	436	+ const b = r.body as FetchResponseBody;
	437	+ if (b.success && b.content && (b.content_type ?? "").includes("text/plain")) {
	438	+ robots = parseRobots(b.content, "fetchabot");
	439	+ if (robots.crawlDelayMs && robots.crawlDelayMs > delayMs) delayMs = Math.min(robots.crawlDelayMs, 10_000);
	440	+ }
	441	+ }
	442	+ }
	443	+ const enqueue = async (candidates: Array<{ url: string; depth: number; parent: string \| null }>) => {
	444	+ const rows: Array<{ id: string; jobId: string; url: string; depth: number; parentUrl: string \| null; status: string }> = [];
	445	+ for (const c of candidates) {
	446	+ if (discovered >= opts.max_pages * 3 \|\| frontier.length + fetched >= opts.max_pages) break;
	447	+ const n = normalizeCrawlUrl(c.url);
	448	+ if (!n \|\| seen.has(n)) continue;
	449	+ if (!inScope(n) \|\| !looksLikePage(n)) continue;
	450	+ if (exclude && exclude(n)) continue;
	451	+ if (include && !include(n) && c.depth > 0) continue;
	452	+ if (robots && !robotsAllows(robots, n)) continue;
	453	+ seen.add(n);
	454	+ discovered++;
	455	+ frontier.push({ url: n, depth: c.depth, parent: c.parent });
	456	+ rows.push({ id: newId("cpg"), jobId: job.id, url: n, depth: c.depth, parentUrl: c.parent, status: "pending" });
	457	+ }
	458	+ if (rows.length) await db.insert(crawlPages).values(rows).onConflictDoNothing().catch(() => {});
	459	+ };
	460	+ if (opts.use_sitemap && fetched === 0) {
	461	+ const sitemapUrls = new Set<string>([`${seedUrl.origin}/sitemap.xml`]);
	462	+ for (const s of robots?.sitemaps ?? []) sitemapUrls.add(s);
	463	+ let count = 0;
	464	+ for (const su of sitemapUrls) {
	465	+ if (count++ >= 5 \|\| frontier.length >= opts.max_pages) break;
	466	+ const r = await fetchPage(su, { format: "raw", links: false, browser: false, browser_fallback: false, retries: 1 });
	467	+ if (r instanceof FetchaError) continue;
	468	+ const b = r.body as FetchResponseBody;
	469	+ if (!b.success \|\| !b.content) continue;
	470	+ const { urls, sitemaps } = parseSitemap(b.content);
	471	+ for (const s of sitemaps.slice(0, 5)) sitemapUrls.add(s);
	472	+ await enqueue(urls.slice(0, opts.max_pages).map((u) => ({ url: u, depth: 1, parent: su })));
	473	+ }
	474	+ }
	475	+
	476	+ let lastHeartbeat = 0;
	477	+ let lastPersist = 0;
	478	+ const persistStats = async (force = false) => {
	479	+ const now = Date.now();
	480	+ if (!force && now - lastPersist < 2000) return;
	481	+ lastPersist = now;
	482	+ await db
	483	+ .update(crawlJobs)
	484	+ .set({ pagesDiscovered: discovered, pagesFetched: fetched, pagesOk: ok, pagesBlocked: blocked, pagesFailed: failed, bytes, costUsd, heartbeatAt: new Date() })
	485	+ .where(eq(crawlJobs.id, job.id))
	486	+ .catch(() => {});
	487	+ lastHeartbeat = now;
	488	+ };
	489	+ const isCancelled = async (): Promise<boolean> => {
	490	+ if (cancelled.has(job.id)) return true;
	491	+ if (Date.now() - lastHeartbeat > HEARTBEAT_MS) {
	492	+ const [row] = await db.select({ status: crawlJobs.status }).from(crawlJobs).where(eq(crawlJobs.id, job.id)).limit(1);
	493	+ lastHeartbeat = Date.now();
	494	+ if (row?.status === "cancelled") {
	495	+ cancelled.add(job.id);
	496	+ return true;
	497	+ }
	498	+ }
	499	+ return false;
	500	+ };
	501	+
	502	+ const worker = async () => {
	503	+ while (!stopped) {
	504	+ if (fetched >= opts.max_pages) return;
	505	+ if (await isCancelled()) return;
	506	+ const item = frontier.shift();
	507	+ if (!item) return;
	508	+ fetched++;
	509	+ const t0 = Date.now();
	510	+ const r = await fetchPage(item.url);
	511	+ const durationMs = Date.now() - t0;
	512	+ const update: Partial<typeof crawlPages.$inferInsert> = { fetchedAt: new Date(), durationMs };
	513	+ if (r instanceof FetchaError) {
	514	+ failed++;
	515	+ Object.assign(update, { status: "failed", errorCode: r.code, requestId: r.requestId ?? null });
	516	+ } else {
	517	+ const b = r.body as FetchResponseBody;
	518	+ const content = opts.format === "markdown" ? b.markdown : opts.format === "text" ? b.text : b.content;
	519	+ const isBlocked = !b.success && b.status >= 400 && r.summary && b.metadata.attempts > 0 && !(b.status === 404 \|\| b.status === 410);
	520	+ bytes += r.summary?.bytes ?? b.metadata.bytes;
	521	+ costUsd += r.summary?.costUsd ?? 0;
	522	+ if (b.success) ok++;
	523	+ else if (isBlocked && b.status !== 500) blocked++;
	524	+ else failed++;
	525	+ Object.assign(update, {
	526	+ status: b.success ? "success" : isBlocked && b.status !== 500 ? "blocked" : "failed",
	527	+ httpStatus: b.status,
	528	+ errorCode: b.success ? null : b.status === 403 \|\| b.status === 429 \|\| b.status === 503 ? "TARGET_BLOCKED" : `HTTP_${b.status}`,
	529	+ requestId: b.request_id,
	530	+ finalUrl: b.final_url,
	531	+ title: b.page?.title ?? null,
	532	+ description: b.page?.description ?? null,
	533	+ contentType: b.content_type,
	534	+ content: b.success && typeof content === "string" ? content.slice(0, PAGE_CONTENT_CAP) : null,
	535	+ linksCount: b.page?.links_count ?? 0,
	536	+ bytes: r.summary?.bytes ?? b.metadata.bytes,
	537	+ mode: b.metadata.mode,
	538	+ });
	539	+ if (b.success && item.depth < opts.max_depth && b.links?.length) {
	540	+ await enqueue(b.links.filter((l) => l.internal \|\| !opts.same_domain).map((l) => ({ url: l.url, depth: item.depth + 1, parent: item.url })));
	541	+ }
	542	+ }
	543	+ await db.update(crawlPages).set(update).where(and(eq(crawlPages.jobId, job.id), eq(crawlPages.url, item.url))).catch(() => {});
	544	+ await persistStats();
	545	+ if (delayMs > 0) await new Promise((res) => setTimeout(res, delayMs));
	546	+ }
	547	+ };
	548	+
	549	+ try {
	550	+ await Promise.all(Array.from({ length: Math.max(1, Math.min(opts.concurrency, 10)) }, () => worker()));
	551	+ await persistStats(true);
	552	+ if (cancelled.has(job.id)) {
	553	+ await finishJob(job.id, "cancelled");
	554	+ cancelled.delete(job.id);
	555	+ } else if (stopped && frontier.length) {
	556	+ await db.update(crawlJobs).set({ status: "queued", workerId: null }).where(eq(crawlJobs.id, job.id)).catch(() => {});
	557	+ } else {
	558	+ await finishJob(job.id, "completed");
	559	+ }
	560	+ log.info(`crawl ${job.id} ${cancelled.has(job.id) ? "cancelled" : "done"}: ${fetched} fetched, ${ok} ok, ${blocked} blocked, ${failed} failed`);
	561	+ if (opts.webhook_url && !stopped) await deliverWebhook(job.id, opts.webhook_url).catch(() => {});
	562	+ } catch (e) {
	563	+ await persistStats(true);
	564	+ await finishJob(job.id, "failed", { code: "INTERNAL_ERROR", message: (e as Error).message?.slice(0, 500) });
	565	+ throw e;
	566	+ }
	567	+}
	568	+
	569	+async function finishJob(id: string, status: "completed" \| "failed" \| "cancelled", error?: { code: string; message?: string }) {
	570	+ await db
	571	+ .update(crawlJobs)
	572	+ .set({ status, completedAt: new Date(), errorCode: error?.code ?? null, errorMessage: error?.message ?? null })
	573	+ .where(and(eq(crawlJobs.id, id), inArray(crawlJobs.status, ["running", "queued", "cancelled"])))
	574	+ .catch(() => {});
	575	+}
	576	+
	577	+async function deliverWebhook(jobId: string, url: string) {
	578	+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, jobId)).limit(1);
	579	+ if (!job) return;
	580	+ let status = "failed";
	581	+ try {
	582	+ await assertUrlAllowed(url);
	583	+ const ac = new AbortController();
	584	+ const t = setTimeout(() => ac.abort(), 10_000);
	585	+ const res = await fetch(url, { method: "POST", headers: { "content-type": "application/json", "user-agent": "fetcha-webhook/0.2" }, body: JSON.stringify({ type: "crawl.completed", job: serializeJob(job) }), signal: ac.signal });
	586	+ clearTimeout(t);
	587	+ status = res.ok ? `delivered:${res.status}` : `failed:${res.status}`;
	588	+ } catch (e) {
	589	+ status = `failed:${(e as Error).message?.slice(0, 80)}`;
	590	+ }
	591	+ await db.update(crawlJobs).set({ webhookStatus: status }).where(eq(crawlJobs.id, jobId)).catch(() => {});
	592	+}

modified apps/api/src/services/engine.ts +14 −2

		`@@ -1,6 +1,7 @@`
1		−import { db, providerConfigs, providerHealth, domainProfiles, eq } from "@fetcha/db";
	1	+import { db, providerConfigs, providerHealth, domainProfiles, featureFlags, eq } from "@fetcha/db";
2	2	import { ProviderRegistry, type ProviderId } from "@fetcha/providers";
3	3	import { CircuitBreaker, FetchExecutor, RoutingEngine, type DomainKnowledge } from "@fetcha/routing";
	4	+import { getBrowserPool, type BrowserPool } from "@fetcha/browser";
4	5	import { newId } from "@fetcha/core";
5	6	import { config } from "../config";
6	7
		`@@ -9,6 +10,9 @@ export interface Engine {`
9	10	circuit: CircuitBreaker;
10	11	routing: RoutingEngine;
11	12	executor: FetchExecutor;
	13	+ browser: BrowserPool;
	14	+ /** Global kill-switch from the `browser_enabled` feature flag (default on). */
	15	+ browserEnabled: boolean;
12	16	reload(): Promise<void>;
13	17	probeAll(): Promise<void>;
14	18	knowledge(domain: string): Promise<DomainKnowledge \| null>;
		`@@ -22,12 +26,18 @@ export async function getEngine(): Promise<Engine> {`
22	26	let registry = new ProviderRegistry();
23	27	let routing = new RoutingEngine(registry, circuit);
24	28	let executor = new FetchExecutor(routing, circuit);
	29	+ const browser = getBrowserPool({ log: { info: (m) => console.log("[browser]", m), warn: (m) => console.warn("[browser]", m) } });
	30	+ let browserEnabled = true;
25	31
26	32	const engine: Engine = {
27	33	get registry() {
28	34	return registry;
29	35	},
30	36	circuit,
	37	+ browser,
	38	+ get browserEnabled() {
	39	+ return browserEnabled && browser.enabled;
	40	+ },
31	41	get routing() {
32	42	return routing;
33	43	},
		`@@ -46,6 +56,8 @@ export async function getEngine(): Promise<Engine> {`
46	56	registry = new ProviderRegistry({ prices, disabled });
47	57	routing = new RoutingEngine(registry, circuit);
48	58	executor = new FetchExecutor(routing, circuit);
	59	+ const [flag] = await db.select().from(featureFlags).where(eq(featureFlags.key, "browser_enabled")).limit(1).catch(() => []);
	60	+ browserEnabled = flag ? flag.enabled : true;
49	61	},
50	62	async probeAll() {
51	63	const results = await Promise.all(
		`@@ -62,7 +74,7 @@ export async function getEngine(): Promise<Engine> {`
62	74	if (!domain) return null;
63	75	const [row] = await db.select().from(domainProfiles).where(eq(domainProfiles.domain, domain)).limit(1).catch(() => []);
64	76	if (!row) return null;
65		− return { domain, routeStats: row.routeStats ?? {}, policy: row.policy ?? null, browserRequiredRate: row.requests ? row.browserRequired / row.requests : 0 };
	77	+ return { domain, routeStats: row.routeStats ?? {}, policy: row.policy ?? null, browserRequiredRate: row.requests ? row.browserRequired / row.requests : 0, browserSamples: row.requests };
66	78	},
67	79	};
68	80	await engine.reload();

modified apps/api/src/services/persist.ts +11 −5

		`@@ -8,7 +8,7 @@ export interface RequestContext {`
8	8	organizationId: string;
9	9	projectId: string;
10	10	apiKeyId: string \| null;
11		− source: "api" \| "playground" \| "sdk";
	11	+ source: "api" \| "playground" \| "sdk" \| "crawl";
12	12	plan: Plan;
13	13	clientIp: string \| null;
14	14	userAgent: string \| null;
		`@@ -46,12 +46,13 @@ export async function recordAttempt(requestId: string, a: AttemptRecord): Promis`
46	46	attemptNo: a.attemptNo,
47	47	provider: a.provider,
48	48	network: a.network,
	49	+ mode: a.mode,
49	50	country: a.country,
50	51	sessionKey: a.sessionKey,
51	52	outcome: a.outcome,
52	53	httpStatus: a.httpStatus,
53	54	errorCode: a.errorCode,
54		− errorDetail: a.errorDetail,
	55	+ errorDetail: a.errorDetail ?? (a.blockVendor ? `vendor=${a.blockVendor} profile=${a.profileId ?? "-"}` : a.profileId ? `profile=${a.profileId}` : null),
55	56	blockReason: a.blockReason,
56	57	durationMs: a.durationMs,
57	58	bytesIn: a.bytesIn,
		`@@ -83,6 +84,8 @@ export async function completeRequest(ctx: RequestContext, req: FetchRequest, re`
83	84	errorMessage: result.body.success ? null : "The target blocked every route we tried.",
84	85	finalUrl: result.finalUrl,
85	86	network,
	87	+ mode: result.mode,
	88	+ browser: result.mode === "browser",
86	89	attempts: result.attempts.length,
87	90	latencyMs,
88	91	bytesIn: result.bytesIn,
		`@@ -96,7 +99,7 @@ export async function completeRequest(ctx: RequestContext, req: FetchRequest, re`
96	99	.where(sql`${fetchRequests.id} = ${ctx.requestId}`);
97	100
98	101	await writeUsage(ctx, result, price.totalUsd, network);
99		− await Promise.all([updateDomainProfile(result.domain, result.attempts, result.body.success), updateRoutingMetrics(result.attempts)]);
	102	+ await Promise.all([updateDomainProfile(result.domain, result.attempts, result.body.success, result.browserRequired), updateRoutingMetrics(result.attempts)]);
100	103	return { priceUsd: price.totalUsd, costUsd: result.costUsd };
101	104	}
102	105
		`@@ -118,7 +121,7 @@ export async function failRequest(ctx: RequestContext, code: string, message: st`
118	121	costUsd: 0,
119	122	upstreamCostUsd: costUsd,
120	123	});
121		− if (attempts.length) await Promise.all([updateDomainProfile(domain, attempts, false), updateRoutingMetrics(attempts)]);
	124	+ if (attempts.length) await Promise.all([updateDomainProfile(domain, attempts, false, false), updateRoutingMetrics(attempts)]);
122	125	}
123	126
124	127	async function writeUsage(ctx: RequestContext, result: ExecutionResult, priceUsd: number, network: ConcreteNetwork \| null) {
		`@@ -134,7 +137,7 @@ async function writeUsage(ctx: RequestContext, result: ExecutionResult, priceUsd`
134	137	);
135	138	}
136	139
137		−async function updateDomainProfile(domain: string, attempts: AttemptRecord[], success: boolean) {
	140	+async function updateDomainProfile(domain: string, attempts: AttemptRecord[], success: boolean, browserRequired: boolean) {
138	141	if (!domain) return;
139	142	const [existing] = await db.select().from(domainProfiles).where(sql`${domainProfiles.domain} = ${domain}`).limit(1);
140	143	const stats = { ...(existing?.routeStats ?? {}) };
		`@@ -145,6 +148,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su`
145	148	if (a.outcome === "blocked") blocks++;
146	149	if (a.blockReason === "captcha" \|\| a.blockReason === "cloudflare_challenge") captchas++;
147	150	}
	151	+ const browserInc = browserRequired ? 1 : 0;
148	152	const pref = preferredRoute(stats);
149	153	const totalLatency = attempts.reduce((s, a) => s + a.durationMs, 0);
150	154	const n = (existing?.requests ?? 0) + 1;
		`@@ -159,6 +163,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su`
159	163	successes: success ? 1 : 0,
160	164	blocks,
161	165	captchas,
	166	+ browserRequired: browserInc,
162	167	avgLatencyMs: totalLatency,
163	168	routeStats: stats,
164	169	lastSeenAt: new Date(),
		`@@ -172,6 +177,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su`
172	177	successes: sql`${domainProfiles.successes} + ${success ? 1 : 0}`,
173	178	blocks: sql`${domainProfiles.blocks} + ${blocks}`,
174	179	captchas: sql`${domainProfiles.captchas} + ${captchas}`,
	180	+ browserRequired: sql`${domainProfiles.browserRequired} + ${browserInc}`,
175	181	avgLatencyMs: avgLatency,
176	182	routeStats: stats,
177	183	lastSeenAt: new Date(),

modified apps/api/src/services/pricing.ts +5 −0

		`@@ -27,6 +27,11 @@ export interface PriceBreakdown {`
27	27
28	28	export function priceRequest(input: PriceInput): PriceBreakdown {
29	29	const limits = PLAN_LIMITS[input.plan];
	30	+ // Private platform: the single plan has no unit prices → nothing is billed. Upstream cost is
	31	+ // still tracked for the admin's unit economics.
	32	+ if (limits.price_usd_month === 0 && limits.overage_per_1k_requests_usd === 0 && limits.residential_per_gb_usd === 0) {
	33	+ return { requestUsd: 0, bandwidthUsd: 0, totalUsd: 0, marginUsd: round6(-input.upstreamCostUsd) };
	34	+ }
30	35	// Free/enterprise have no per-request rate; failed requests are never charged a request fee.
31	36	const requestUsd = input.success ? limits.overage_per_1k_requests_usd / 1000 : 0;
32	37	let bandwidthUsd = 0;

modified apps/api/src/services/retention.ts +10 −0

		`@@ -19,6 +19,16 @@ export async function pruneRetention(log: { info: (m: string) => void } = consol`
19	19	`);
20	20	total += Number((res as { rowCount?: number }).rowCount ?? 0);
21	21	}
	22	+ for (const plan of PLANS) {
	23	+ const days = PLAN_LIMITS[plan].retention_days;
	24	+ await db.execute(sql`
	25	+ delete from crawl_jobs j
	26	+ using organizations o
	27	+ where j.organization_id = o.id
	28	+ and o.plan = ${plan}
	29	+ and j.created_at < now() - make_interval(days => ${days})
	30	+ `);
	31	+ }
22	32	await db.execute(sql`update proxy_sessions set status = 'expired' where status = 'active' and expires_at < now()`);
23	33	await db.execute(sql`delete from provider_health where checked_at < now() - interval '30 days'`);
24	34	await db.execute(sql`delete from verifications where expires_at < now() - interval '1 day'`);

added apps/web/src/actions/access.ts +148 −0

		`@@ -0,0 +1,148 @@`
	1	+"use server";
	2	+
	3	+import { revalidatePath } from "next/cache";
	4	+import { z } from "zod";
	5	+import { getDb, signupAllowlist, eq, inArray } from "@fetcha/db";
	6	+import { getEmailService } from "@fetcha/email";
	7	+import { requireAdmin, requestMeta } from "@/lib/session";
	8	+import { writeAudit } from "@/lib/audit";
	9	+import { SITE_URL } from "@/lib/utils";
	10	+import type { AuthUser } from "@/lib/auth";
	11	+import type { AdminActionResult } from "@/actions/admin";
	12	+
	13	+/**
	14	+ * Signup allowlist management (invitation-only platform). Every mutation is audited as
	15	+ * `admin.action` with metadata.type in { access.allow, access.invite, access.revoke }.
	16	+ */
	17	+
	18	+async function audit(admin: AuthUser, type: "access.allow" \| "access.invite" \| "access.revoke", target: string, metadata: Record<string, unknown> = {}) {
	19	+ const meta = await requestMeta();
	20	+ await writeAudit({ userId: admin.id, action: "admin.action", target, metadata: { type, ...metadata }, ipAddress: meta.ip, userAgent: meta.userAgent });
	21	+}
	22	+
	23	+function fail(e: unknown): { ok: false; error: string } {
	24	+ if (e instanceof z.ZodError) return { ok: false, error: e.issues[0]?.message ?? "Invalid input" };
	25	+ return { ok: false, error: (e as Error)?.message ?? "Unexpected error" };
	26	+}
	27	+
	28	+function revalidate() {
	29	+ revalidatePath("/admin/access");
	30	+}
	31	+
	32	+const emailSchema = z.string().trim().toLowerCase().email("Invalid email address").max(254);
	33	+
	34	+/** Split a comma / newline / whitespace separated list into unique, lower-cased, valid emails. */
	35	+function parseEmailList(raw: string): { emails: string[]; invalid: string[] } {
	36	+ const seen = new Set<string>();
	37	+ const emails: string[] = [];
	38	+ const invalid: string[] = [];
	39	+ for (const token of raw.split(/[\s,;]+/)) {
	40	+ const t = token.trim();
	41	+ if (!t) continue;
	42	+ const r = emailSchema.safeParse(t);
	43	+ if (!r.success) {
	44	+ invalid.push(t);
	45	+ continue;
	46	+ }
	47	+ if (seen.has(r.data)) continue;
	48	+ seen.add(r.data);
	49	+ emails.push(r.data);
	50	+ }
	51	+ return { emails, invalid };
	52	+}
	53	+
	54	+function signupUrlFor(email: string): string {
	55	+ return `${SITE_URL}/signup?email=${encodeURIComponent(email)}`;
	56	+}
	57	+
	58	+const allowSchema = z.object({
	59	+ emails: z.string().trim().min(3, "Enter at least one email address").max(20_000),
	60	+ note: z.string().trim().max(280, "Note is too long (280 characters max)").optional().or(z.literal("")),
	61	+ sendInvite: z.boolean().default(false),
	62	+});
	63	+
	64	+export type AllowEmailsInput = z.input<typeof allowSchema>;
	65	+export type AllowEmailsData = { added: number; existing: number; invited: number; invalid: string[]; failed: string[] };
	66	+
	67	+export async function allowEmails(input: AllowEmailsInput): Promise<AdminActionResult<AllowEmailsData>> {
	68	+ const admin = await requireAdmin();
	69	+ try {
	70	+ const d = allowSchema.parse(input);
	71	+ const { emails, invalid } = parseEmailList(d.emails);
	72	+ if (emails.length === 0) return { ok: false, error: invalid.length ? `No valid email address found (rejected: ${invalid.slice(0, 3).join(", ")}${invalid.length > 3 ? "…" : ""}).` : "Enter at least one email address." };
	73	+ if (emails.length > 200) return { ok: false, error: "Add at most 200 addresses at a time." };
	74	+
	75	+ const db = getDb();
	76	+ const note = d.note?.trim() \|\| null;
	77	+ let invited = 0;
	78	+ const failed: string[] = [];
	79	+
	80	+ const already = new Set((await db.select({ email: signupAllowlist.email }).from(signupAllowlist).where(inArray(signupAllowlist.email, emails))).map((r) => r.email));
	81	+ const fresh = emails.filter((e) => !already.has(e));
	82	+ if (fresh.length) await db.insert(signupAllowlist).values(fresh.map((email) => ({ email, note, invitedByUserId: admin.id }))).onConflictDoNothing();
	83	+ // Existing entries keep their inviter; only refresh the note when one was provided.
	84	+ if (note && already.size) await db.update(signupAllowlist).set({ note }).where(inArray(signupAllowlist.email, Array.from(already)));
	85	+ const added = fresh.length;
	86	+ const existing = already.size;
	87	+ await audit(admin, "access.allow", emails.length === 1 ? emails[0]! : `${emails.length} emails`, { emails, note, added, existing });
	88	+
	89	+ if (d.sendInvite) {
	90	+ const inviterName = admin.name?.trim() \|\| admin.email;
	91	+ for (const email of emails) {
	92	+ try {
	93	+ await getEmailService().sendInvite(email, { inviterName, signupUrl: signupUrlFor(email) });
	94	+ await db.update(signupAllowlist).set({ invitedAt: new Date() }).where(eq(signupAllowlist.email, email));
	95	+ invited++;
	96	+ } catch (e) {
	97	+ failed.push(email);
	98	+ console.error("[access] invite failed", email, (e as Error).message);
	99	+ }
	100	+ }
	101	+ await audit(admin, "access.invite", emails.length === 1 ? emails[0]! : `${emails.length} emails`, { emails, invited, failed });
	102	+ }
	103	+
	104	+ revalidate();
	105	+ const warnings: string[] = [];
	106	+ if (invalid.length) warnings.push(`${invalid.length} entr${invalid.length === 1 ? "y was" : "ies were"} not a valid email and ${invalid.length === 1 ? "was" : "were"} skipped: ${invalid.slice(0, 5).join(", ")}${invalid.length > 5 ? "…" : ""}`);
	107	+ if (failed.length) warnings.push(`Invitation email could not be sent to ${failed.join(", ")}. Use “Resend invite” later.`);
	108	+ return { ok: true, data: { added, existing, invited, invalid, failed }, warning: warnings.length ? warnings.join(" ") : undefined };
	109	+ } catch (e) {
	110	+ return fail(e);
	111	+ }
	112	+}
	113	+
	114	+export async function resendInvite(email: string): Promise<AdminActionResult> {
	115	+ const admin = await requireAdmin();
	116	+ try {
	117	+ const target = emailSchema.parse(email);
	118	+ const db = getDb();
	119	+ const [row] = await db.select().from(signupAllowlist).where(eq(signupAllowlist.email, target)).limit(1);
	120	+ if (!row) return { ok: false, error: "This address is not on the access list." };
	121	+ if (row.userId) return { ok: false, error: "This address already has an account; there is nothing to invite." };
	122	+ const inviterName = admin.name?.trim() \|\| admin.email;
	123	+ await getEmailService().sendInvite(target, { inviterName, signupUrl: signupUrlFor(target) });
	124	+ await db.update(signupAllowlist).set({ invitedAt: new Date() }).where(eq(signupAllowlist.email, target));
	125	+ await audit(admin, "access.invite", target, { emails: [target], invited: 1, resend: true });
	126	+ revalidate();
	127	+ return { ok: true };
	128	+ } catch (e) {
	129	+ return fail(e);
	130	+ }
	131	+}
	132	+
	133	+export async function revokeAllow(email: string): Promise<AdminActionResult> {
	134	+ const admin = await requireAdmin();
	135	+ try {
	136	+ const target = emailSchema.parse(email);
	137	+ const db = getDb();
	138	+ const [row] = await db.select({ email: signupAllowlist.email, userId: signupAllowlist.userId, note: signupAllowlist.note }).from(signupAllowlist).where(eq(signupAllowlist.email, target)).limit(1);
	139	+ if (!row) return { ok: false, error: "This address is not on the access list." };
	140	+ if (row.userId) return { ok: false, error: "This address already has an account; ban the user instead." };
	141	+ await db.delete(signupAllowlist).where(eq(signupAllowlist.email, target));
	142	+ await audit(admin, "access.revoke", target, { note: row.note });
	143	+ revalidate();
	144	+ return { ok: true };
	145	+ } catch (e) {
	146	+ return fail(e);
	147	+ }
	148	+}

modified apps/web/src/actions/admin.ts +8 −13

		`@@ -118,20 +118,15 @@ export async function forceVerifyEmail(userId: string): Promise<AdminActionResul`
118	118	// ---------------------------------------------------------------------------
119	119	// Organizations
120	120	// ---------------------------------------------------------------------------
	121	+/**
	122	+ * Single-plan platform: kept for API compatibility with older UI code, but plans can no longer be
	123	+ * changed. Every organization is `unlimited` (see `normalizePlan` in @fetcha/core).
	124	+ */
121	125	export async function updateOrganizationPlan(orgId: string, plan: string): Promise<AdminActionResult> {
122		− const admin = await requireAdmin();
123		− try {
124		− const p = z.enum(PLANS).parse(plan);
125		− const db = getDb();
126		− const [o] = await db.select({ id: organizations.id, plan: organizations.plan }).from(organizations).where(eq(organizations.id, orgId)).limit(1);
127		− if (!o) return { ok: false, error: "Organization not found" };
128		− await db.update(organizations).set({ plan: p, updatedAt: new Date() }).where(eq(organizations.id, orgId));
129		− await audit(admin, "org.plan", orgId, { from: o.plan, to: p }, orgId);
130		− revalidateAdmin();
131		− return { ok: true };
132		− } catch (e) {
133		− return fail(e);
134		− }
	126	+ await requireAdmin();
	127	+ void orgId;
	128	+ void plan;
	129	+ return { ok: false, error: "Plans are not configurable on this platform" };
135	130	}
136	131
137	132	export async function setOrganizationProviderVisibility(orgId: string, enabled: boolean): Promise<AdminActionResult> {

added apps/web/src/actions/crawls.ts +87 −0

		`@@ -0,0 +1,87 @@`
	1	+"use server";
	2	+
	3	+import { revalidatePath } from "next/cache";
	4	+import { crawlCreateSchema, mapCreateSchema } from "@fetcha/core";
	5	+import { getWorkspace } from "@/lib/session";
	6	+import { internalApi, InternalApiError, type CrawlJob, type MapResult } from "@/lib/api";
	7	+
	8	+export interface CrawlActionError {
	9	+ code: string;
	10	+ message: string;
	11	+ requestId: string \| null;
	12	+ details?: Record<string, unknown>;
	13	+}
	14	+
	15	+export type CrawlActionResult<T> = { ok: true; data: T } \| { ok: false; error: CrawlActionError };
	16	+
	17	+function invalid(issues: Array<{ path: string; message: string }>): CrawlActionError {
	18	+ const first = issues[0];
	19	+ return {
	20	+ code: "INVALID_REQUEST",
	21	+ message: first ? `${first.path ? `${first.path}: ` : ""}${first.message}` : "The request is invalid.",
	22	+ requestId: null,
	23	+ details: { issues },
	24	+ };
	25	+}
	26	+
	27	+function fromError(e: unknown, fallback: string): CrawlActionError {
	28	+ if (e instanceof InternalApiError) return { code: e.code, message: e.message, requestId: e.requestId, details: e.details };
	29	+ console.error("[crawls] unexpected error", e);
	30	+ return { code: "INTERNAL_ERROR", message: fallback, requestId: null };
	31	+}
	32	+
	33	+/** Strip empty strings / undefined so optional fields do not trip the strict schema. */
	34	+function clean(input: unknown): Record<string, unknown> {
	35	+ if (!input \|\| typeof input !== "object") return {};
	36	+ return Object.fromEntries(Object.entries(input as Record<string, unknown>).filter(([, v]) => v !== undefined && v !== "" && v !== null));
	37	+}
	38	+
	39	+/**
	40	+ * Start a crawl job for the current project. Validated with the shared `crawlCreateSchema`; the API
	41	+ * enforces plan limits (pages per job, concurrent jobs). Never throws to the client.
	42	+ */
	43	+export async function createCrawl(input: unknown): Promise<CrawlActionResult<CrawlJob>> {
	44	+ const ws = await getWorkspace();
	45	+ try {
	46	+ const parsed = crawlCreateSchema.safeParse(clean(input));
	47	+ if (!parsed.success) return { ok: false, error: invalid(parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message }))) };
	48	+ const job = await internalApi.createCrawl(ws.project.id, ws.user.id, parsed.data);
	49	+ if (!job \|\| typeof job !== "object" \|\| typeof job.id !== "string") {
	50	+ return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };
	51	+ }
	52	+ revalidatePath("/dashboard/crawls");
	53	+ return { ok: true, data: job };
	54	+ } catch (e) {
	55	+ return { ok: false, error: fromError(e, "Could not start the crawl. Please try again.") };
	56	+ }
	57	+}
	58	+
	59	+/** Cancel a queued or running crawl job. */
	60	+export async function cancelCrawl(id: string): Promise<CrawlActionResult<{ id: string; status: "cancelled" }>> {
	61	+ const ws = await getWorkspace();
	62	+ if (!/^crawl_[A-Za-z0-9]{4,64}$/.test(id)) return { ok: false, error: { code: "INVALID_REQUEST", message: "Invalid crawl id.", requestId: null } };
	63	+ try {
	64	+ const res = await internalApi.cancelCrawl(ws.project.id, ws.user.id, id);
	65	+ revalidatePath("/dashboard/crawls");
	66	+ revalidatePath(`/dashboard/crawls/${id}`);
	67	+ return { ok: true, data: res };
	68	+ } catch (e) {
	69	+ return { ok: false, error: fromError(e, "Could not cancel the crawl. Please try again.") };
	70	+ }
	71	+}
	72	+
	73	+/** Discover the URLs of a site (sitemap + links), synchronously. */
	74	+export async function runMap(input: unknown): Promise<CrawlActionResult<MapResult>> {
	75	+ const ws = await getWorkspace();
	76	+ try {
	77	+ const parsed = mapCreateSchema.safeParse(clean(input));
	78	+ if (!parsed.success) return { ok: false, error: invalid(parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message }))) };
	79	+ const result = await internalApi.mapSite(ws.project.id, ws.user.id, parsed.data);
	80	+ if (!result \|\| typeof result !== "object" \|\| !Array.isArray(result.urls)) {
	81	+ return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };
	82	+ }
	83	+ return { ok: true, data: result };
	84	+ } catch (e) {
	85	+ return { ok: false, error: fromError(e, "Could not map the site. Please try again.") };
	86	+ }
	87	+}

modified apps/web/src/actions/playground.ts +2 −7

		`@@ -1,6 +1,6 @@`
1	1	"use server";
2	2
3		−import { PLAN_LIMITS, fetchRequestSchema, type FetchResponseBody, type Plan } from "@fetcha/core";
	3	+import { PLAN_LIMITS, fetchRequestSchema, normalizePlan, type FetchResponseBody } from "@fetcha/core";
4	4	import { getWorkspace } from "@/lib/session";
5	5	import { internalApi, InternalApiError } from "@/lib/api";
6	6
		`@@ -39,8 +39,7 @@ export async function runPlayground(input: unknown): Promise<PlaygroundResult> {`
39	39	}
40	40
41	41	const request = parsed.data;
42		− const plan = (ws.organization.plan as Plan) in PLAN_LIMITS ? (ws.organization.plan as Plan) : "free";
43		− const limits = PLAN_LIMITS[plan];
	42	+ const limits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
44	43	if (request.timeout > limits.max_timeout_ms) request.timeout = limits.max_timeout_ms;
45	44	if (request.retries !== undefined && request.retries > limits.max_retries) request.retries = limits.max_retries;
46	45	if (request.network !== "auto" && !limits.networks.includes(request.network)) {
		`@@ -49,10 +48,6 @@ export async function runPlayground(input: unknown): Promise<PlaygroundResult> {`
49	48	error: { code: "NETWORK_UNAVAILABLE", message: `The "${request.network}" network is not included in the ${limits.label} plan.`, requestId: null },
50	49	};
51	50	}
52		− if (request.browser) {
53		− return { ok: false, error: { code: "BROWSER_UNAVAILABLE", message: "Managed browser execution is not yet available.", requestId: null } };
54		− }
55		−
56	51	const data = (await internalApi.playgroundFetch(ws.project.id, ws.user.id, request)) as FetchResponseBody;
57	52	if (!data \|\| typeof data !== "object" \|\| typeof data.request_id !== "string") {
58	53	return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };

modified apps/web/src/app/(auth)/login/login-form.tsx +3 −3

		`@@ -35,12 +35,12 @@ export function LoginForm({ next, verified, reset }: { next?: string; verified?:`
35	35	return (
36	36	<AuthCard
37	37	title="Welcome back"
38		− description="Log in to your Fetcha account."
	38	+ description="Log in to your Fetcha account. Fetcha is a private platform: accounts are created by invitation."
39	39	footer={
40	40	<>
41		− New to Fetcha?{" "}
	41	+ Invitation only — ask your administrator.{" "}
42	42	<Link href={`/signup${next ? `?next=${encodeURIComponent(next)}` : ""}`} className="font-medium text-fg underline-offset-4 hover:underline">
43		− Create an account
	43	+ Invited? Create your account
44	44	</Link>
45	45	</>
46	46	}

modified apps/web/src/app/(auth)/signup/page.tsx +4 −2

		`@@ -5,9 +5,11 @@ import { SignupForm } from "./signup-form";`
5	5
6	6	export const metadata: Metadata = { title: "Create account" };
7	7
8		−export default async function SignupPage({ searchParams }: { searchParams: Promise<{ next?: string }> }) {
	8	+export default async function SignupPage({ searchParams }: { searchParams: Promise<{ next?: string; email?: string \| string[] }> }) {
9	9	const sp = await searchParams;
10	10	const user = await getUser();
11	11	if (user) redirect("/dashboard");
12		− return <SignupForm next={sp.next} />;
	12	+ const raw = Array.isArray(sp.email) ? sp.email[0] : sp.email;
	13	+ const email = raw ? raw.trim().toLowerCase().slice(0, 254) : undefined;
	14	+ return <SignupForm next={sp.next} initialEmail={email} />;
13	15	}

modified apps/web/src/app/(auth)/signup/signup-form.tsx +33 −8

		`@@ -12,10 +12,12 @@ import { Input } from "@/components/ui/input";`
12	12	import { Field, FieldError, Label } from "@/components/ui/label";
13	13	import { Alert } from "@/components/ui/alert";
14	14
15		−export function SignupForm({ next }: { next?: string }) {
	15	+const ACCESS_DENIED = "This email is not on the access list. Ask your Fetcha administrator to invite you.";
	16	+
	17	+export function SignupForm({ next, initialEmail }: { next?: string; initialEmail?: string }) {
16	18	const router = useRouter();
17	19	const [name, setName] = React.useState("");
18		− const [email, setEmail] = React.useState("");
	20	+ const [email, setEmail] = React.useState(initialEmail ?? "");
19	21	const [password, setPassword] = React.useState("");
20	22	const [confirm, setConfirm] = React.useState("");
21	23	const [agree, setAgree] = React.useState(false);
		`@@ -32,10 +34,13 @@ export function SignupForm({ next }: { next?: string }) {`
32	34	if (password !== confirm) return setError("Passwords do not match.");
33	35	if (!agree) return setError("You must accept the Terms of Service and Privacy Policy.");
34	36	setLoading(true);
35		− const { error } = await authClient.signUp.email({ name: name.trim() \|\| email.split("@")[0]!, email: email.trim(), password, callbackURL: "/dashboard?verified=1" });
	37	+ const { error } = await authClient.signUp.email({ name: name.trim() \|\| email.split("@")[0]!, email: email.trim().toLowerCase(), password, callbackURL: "/dashboard?verified=1" });
36	38	if (error) {
37	39	setLoading(false);
38		− setError(error.status === 422 \|\| /exist/i.test(error.message ?? "") ? "An account with this email already exists. Try logging in." : error.message ?? "Could not create the account.");
	40	+ const msg = error.message ?? "";
	41	+ if (error.status === 403 \|\| /access list/i.test(msg)) setError(ACCESS_DENIED);
	42	+ else if (error.status === 422 \|\| /exist/i.test(msg)) setError("An account with this email already exists. Try logging in.");
	43	+ else setError(msg \|\| "Could not create the account.");
39	44	return;
40	45	}
41	46	await recordLegalAcceptance().catch(() => {});
		`@@ -49,7 +54,7 @@ export function SignupForm({ next }: { next?: string }) {`
49	54
50	55	if (done) {
51	56	return (
52		− <AuthCard title="Check your inbox" description={<>We sent a verification link to <strong className="text-fg">{email}</strong>. Verify it to unlock production API access — the Playground is available right away.</>}>
	57	+ <AuthCard title="Check your inbox" description={<>We sent a verification link to <strong className="text-fg">{email}</strong>. Verify it to unlock API access — the Playground is available right away.</>}>
53	58	<div className="flex items-center gap-3 rounded-lg border border-border bg-bg-subtle p-4 text-[13.5px]">
54	59	<MailCheck className="size-5 text-success" />
55	60	<span>Redirecting you to the dashboard…</span>
		`@@ -64,7 +69,15 @@ export function SignupForm({ next }: { next?: string }) {`
64	69	return (
65	70	<AuthCard
66	71	title="Create your account"
67		− description="Free plan included: 1,000 requests a month, no credit card."
	72	+ description={
	73	+ <>
	74	+ Fetcha is invitation-only. Use the address your administrator approved.{" "}
	75	+ <a href="mailto:hello@fetcha.co?subject=Fetcha%20access" className="text-fg underline-offset-4 hover:underline">
	76	+ Request access
	77	+ </a>{" "}
	78	+ if you have not been invited yet.
	79	+ </>
	80	+ }
68	81	footer={
69	82	<>
70	83	Already have an account?{" "}
		`@@ -81,8 +94,20 @@ export function SignupForm({ next }: { next?: string }) {`
81	94	<Input id="name" name="name" autoComplete="name" value={name} onChange={(e) => setName(e.target.value)} placeholder="Ada Lovelace" />
82	95	</Field>
83	96	<Field>
84		− <Label htmlFor="email">Work email</Label>
85		− <Input id="email" name="email" type="email" autoComplete="email" required value={email} onChange={(e) => setEmail(e.target.value)} placeholder="you@company.com" />
	97	+ <Label htmlFor="email">Approved email</Label>
	98	+ <Input id="email" name="email" type="email" autoComplete="email" required value={email} onChange={(e) => setEmail(e.target.value)} placeholder="you@company.com" readOnly={Boolean(initialEmail) && email === initialEmail} aria-describedby="email-hint" />
	99	+ <p id="email-hint" className="text-xs text-fg-subtle">
	100	+ {initialEmail && email === initialEmail ? (
	101	+ <>
	102	+ Pre-filled from your invitation.{" "}
	103	+ <button type="button" className="underline-offset-4 hover:underline" onClick={() => setEmail("")}>
	104	+ Use another address
	105	+ </button>
	106	+ </>
	107	+ ) : (
	108	+ "Must match the address on the access list exactly."
	109	+ )}
	110	+ </p>
86	111	</Field>
87	112	<Field>
88	113	<Label htmlFor="password">Password</Label>

modified apps/web/src/app/(marketing)/changelog/page.tsx +23 −2

		`@@ -23,12 +23,33 @@ type Entry = {`
23	23	};
24	24
25	25	const ENTRIES: Entry[] = [
	26	+ {
	27	+ date: "2026-09-08",
	28	+ version: "v0.2.0",
	29	+ title: "Private access, managed browser, crawl & map",
	30	+ tag: "stable",
	31	+ summary: "Fetcha becomes a private, invitation-only platform with a single unlimited plan, and the browser layer goes live: rendered fetches with automatic escalation from plain HTTP, hardened HTTP fingerprints, much broader block detection, Markdown output with page metadata and links, and a new Crawl & Map API with its dashboard page.",
	32	+ changes: [
	33	+ { kind: "changed", text: "Private access: signup is invitation-only (administrators manage the allowlist and send invitations). One plan, unlimited: no request quota, 200 concurrent requests, 120 s max timeout, 5 retries, all network classes, 90-day retention, browser rendering and crawl jobs. No billing, no checkout." },
	34	+ { kind: "added", text: "Managed browser rendering: browser: true renders the page in a headless Chromium on the same network, country and session; wait_for, wait_ms, wait_until, javascript, block_resources and screenshot control the render. 8 concurrent renders per organization; BROWSER_TIMEOUT when a page does not settle." },
	35	+ { kind: "added", text: "Automatic escalation: with browser_fallback (default on), an HTTP attempt blocked by a JavaScript challenge is retried in the browser. metadata.mode reports http or browser; debug attempts carry mode and block_reason." },
	36	+ { kind: "changed", text: "Hardened HTTP fingerprints: ordered Chrome, Firefox and Safari header profiles rotated per attempt, HTTP/2, Chrome-like TLS cipher ordering, a cookie jar carried across redirects, jittered backoff between retries and Retry-After honoured within the timeout. referer: auto \| none \| url controls the Referer strategy." },
	37	+ { kind: "changed", text: "Expanded block detection: Cloudflare challenge and Turnstile, DataDome, PerimeterX / HUMAN, Akamai, Kasada, Imperva, AWS WAF, Vercel attack mode and soft 200 blocks (challenge, interstitial or empty-shell pages returned with a 200)." },
	38	+ { kind: "added", text: "format: \"markdown\" returns the page as Markdown (main content first, boilerplate removed) in markdown. Every HTML response now includes page { title, description, canonical, lang, og, links_count }; links: true adds links[] with absolute URLs, anchor text, internal and nofollow flags." },
	39	+ { kind: "added", text: "Crawl API: POST /v1/crawl starts an asynchronous job (max_pages, max_depth, include/exclude patterns, robots, sitemap, concurrency, delay, format, browser); GET /v1/crawl, GET /v1/crawl/:id, GET /v1/crawl/:id/pages (cursor pagination, content per page) and DELETE /v1/crawl/:id. 2,000 pages per job, 5 concurrent jobs. Every page is a normal request in the log with source crawl." },
	40	+ { kind: "added", text: "Map API: POST /v1/map lists a site's URLs from its sitemap and links, synchronously, with search filtering." },
	41	+ { kind: "added", text: "Dashboard: Crawls page (jobs, status, stats, create dialog, job detail with pages and content preview, cancel, Map tool). Playground: markdown format, live Browser rendering section, Include links, Referer, mode badge, page metadata, Markdown / Links / Screenshot tabs." },
	42	+ { kind: "added", text: "SDKs 0.2.0 (JavaScript and Python): markdown format, browser options, links and referer on fetch; crawl.create / get / pages / cancel / wait and map(). User-Agent fetcha-sdk-js/0.2.0 and fetcha-sdk-python/0.2.0." },
	43	+ { kind: "changed", text: "BROWSER_UNAVAILABLE is now only returned when the browser pool is disabled or unavailable. New codes CRAWL_NOT_FOUND (404) and CRAWL_LIMIT_REACHED (429)." },
	44	+ { kind: "not-yet", text: "Browser actions (POST /v1/browser), structured extraction (/v1/extract), webhook delivery (crawl webhook_url is stored, not called), datacenter, ISP and mobile network classes, teams, OAuth, 2FA, CLI." },
	45	+ ],
	46	+ },
26	47	{
27	48	date: "2026-09-07",
28	49	version: "v0.1.0",
29	50	title: "Public preview",
30	51	tag: "preview",
31		− summary: "First public release of Fetcha. One endpoint, automatic routing over the residential network, sticky sessions, and a dashboard to try requests and read logs. Billing checkout is intentionally not part of this release; every account runs on the Free plan.",
	52	+ summary: "First public release of Fetcha. One endpoint, automatic routing over the residential network, sticky sessions, and a dashboard to try requests and read logs. Billing checkout was intentionally not part of this release; every account ran on the Free plan (superseded by the single unlimited plan in 0.2.0).",
32	53	changes: [
33	54	{ kind: "added", text: "Fetch API: POST /v1/fetch with country, region and city targeting, custom headers, cookies, body, timeout, format (html, text, json, raw), redirect controls and per-request retries." },
34	55	{ kind: "added", text: "Automatic routing (network: \"auto\"): weighted per-domain scoring, circuit breakers and cheap-to-premium escalation with a fresh IP on every retry." },
		`@@ -60,7 +81,7 @@ export default function ChangelogPage() {`
60	81	<p className="text-[11.5px] font-semibold uppercase tracking-[0.12em] text-accent">Changelog</p>
61	82	<h1 className="mt-2 text-[34px] font-semibold leading-[1.1] tracking-tight sm:text-[44px]">What shipped, and what has not.</h1>
62	83	<p className="mt-4 text-[15.5px] leading-relaxed text-fg-muted">
63		− Release notes for the API, dashboard and SDKs. We list unavailable features explicitly so you never build against something that does not exist yet. Follow the <Link href="/status" className="text-accent underline-offset-4 hover:underline">status page</Link> for incidents.
	84	+ Release notes for the API, dashboard and SDKs. We list unavailable features explicitly so you never build against something that does not exist yet. Follow the <Link href="/status" className="text-accent underline-offset-4 hover:underline">status page</Link> for incidents. Fetcha is invitation-only: <a href="mailto:hello@fetcha.co?subject=Fetcha%20access" className="text-accent underline-offset-4 hover:underline">request access</a> or <Link href="/login" className="text-accent underline-offset-4 hover:underline">log in</Link>.
64	85	</p>
65	86	</header>
66	87

modified apps/web/src/app/(marketing)/docs/authentication/page.tsx +11 −11

		`@@ -18,8 +18,8 @@ export default function AuthenticationPage() {`
18	18	<DocPage path="/docs/authentication" title="Authentication" description="Every request to the Fetcha API is authenticated with a project-scoped API key sent in a header. Keys carry scopes, a mode and an optional expiry." status="Stable">
19	19	<H2>API keys</H2>
20	20	<P>
21		− Keys are created in the dashboard under a project. A key belongs to exactly one project and one organization; the requests it makes are logged, counted and billed against them. The
22		− plaintext has a recognisable prefix so you can tell keys apart in configuration:
	21	+ Keys are created in the dashboard under a project. A key belongs to exactly one project and one organization; the requests it makes are logged and counted against them (Fetcha is a private
	22	+ platform with a single unlimited plan, so nothing is billed). The plaintext has a recognisable prefix so you can tell keys apart in configuration:
23	23	</P>
24	24	<Table>
25	25	<THead>
		`@@ -39,7 +39,7 @@ export default function AuthenticationPage() {`
39	39	<Td mono>fch_test_…</Td>
40	40	<Td>test</Td>
41	41	<Td>
42		− Marked as a test key in logs and in <Code>GET /v1/me</Code>. Behaves like a live key today: requests are real and count toward your plan.
	42	+ Marked as a test key in logs and in <Code>GET /v1/me</Code>. Behaves like a live key today: requests are real and appear in your usage.
43	43	</Td>
44	44	</Tr>
45	45	</TBody>
		`@@ -62,7 +62,7 @@ export default function AuthenticationPage() {`
62	62	</P>
63	63	<CodeBlock lang="text" code={`X-API-Key: fch_live_…`} />
64	64	<Callout variant="warning" title="Server-side only">
65		− Never ship a key in a browser bundle or mobile app: anyone who can read it can spend your quota. Call Fetcha from your backend. The API allows cross-origin requests so that the dashboard
	65	+ Never ship a key in a browser bundle or mobile app: anyone who can read it can fetch on your behalf, fill your request log and trip your spending limits. Call Fetcha from your backend. The API allows cross-origin requests so that the dashboard
66	66	can talk to it, not so that keys can live in the browser.
67	67	</Callout>
68	68
		`@@ -103,9 +103,9 @@ export default function AuthenticationPage() {`
103	103	<Tr>
104	104	<Td mono>browser:use</Td>
105	105	<Td>
106		− Reserved for the managed browser (<A href="/docs/browser">coming soon</A>). Can be granted today but unlocks nothing yet.
	106	+ Kept for compatibility. The <A href="/docs/browser">managed browser</A> is part of <Code>POST /v1/fetch</Code> and needs only <Code>fetch:execute</Code>.
107	107	</Td>
108		− <Td>Reserved</Td>
	108	+ <Td>Compatibility</Td>
109	109	</Tr>
110	110	</TBody>
111	111	</Table>
		`@@ -113,11 +113,11 @@ export default function AuthenticationPage() {`
113	113	<Code>GET /v1/me</Code> and <Code>GET /v1/sessions</Code> only require a valid key.
114	114	</P>
115	115
116		− <H2>Email verification</H2>
	116	+ <H2>Accounts and email verification</H2>
117	117	<P>
118		− API keys work only once the organization owner's email address is verified. Until then every authenticated call, including <Code>/v1/me</Code>, fails with{" "}
119		− <Code>403 EMAIL_NOT_VERIFIED</Code>. Resend the verification email from the dashboard if you did not receive it. The dashboard Playground is exempt so you can evaluate the product before
120		− verifying.
	118	+ Fetcha is invitation-only: an account can only be created with an email address that a Fetcha administrator placed on the access list. Once the account exists, API keys work only after
	119	+ that email address is verified. Until then every authenticated call, including <Code>/v1/me</Code>, fails with <Code>403 EMAIL_NOT_VERIFIED</Code>. Resend the verification email from the
	120	+ dashboard if you did not receive it. The dashboard Playground is exempt so you can try the product before verifying.
121	121	</P>
122	122	<ResponseExample
123	123	status={403}
		`@@ -157,7 +157,7 @@ export default function AuthenticationPage() {`
157	157	status={200}
158	158	body={{
159	159	project: { id: "proj_2k8d1m3p9q4r7s6t", name: "Default" },
160		− organization: { id: "org_9a1b2c3d4e5f6g7h", name: "Acme Data", plan: "developer" },
	160	+ organization: { id: "org_9a1b2c3d4e5f6g7h", name: "Acme Data", plan: "unlimited" },
161	161	key: { id: "key_5t6y7u8i9o0p1a2s", name: "backend", mode: "live", scopes: ["fetch:execute", "sessions:write", "usage:read"] },
162	162	}}
163	163	/>

modified apps/web/src/app/(marketing)/docs/browser/page.tsx +189 −45

		`@@ -1,14 +1,46 @@`
1	1	import type { Metadata } from "next";
2	2	import { CodeBlock } from "@/components/ui/code-block";
3	3	import { DocPage } from "@/components/docs/doc-page";
4		−import { A, Code, H2, Li, P, Strong, Table, TBody, Td, Th, THead, Tr, Ul } from "@/components/docs/prose";
	4	+import { A, Code, H2, H3, Li, P, Strong, Table, TBody, Td, Th, THead, Tr, Ul } from "@/components/docs/prose";
5	5	import { Callout, ComingSoon } from "@/components/docs/callout";
6	6	import { Endpoint } from "@/components/docs/endpoint";
	7	+import { ParamTable } from "@/components/docs/param-table";
	8	+import { CodeTabs } from "@/components/docs/code-tabs";
7	9	import { ResponseExample } from "@/components/docs/response-example";
	10	+import { fetchTabs } from "@/components/docs/snippets";
8	11
9	12	export const metadata: Metadata = {
10	13	title: "Browser",
11		− description: "Managed browser rendering is coming soon. This page describes the planned browser: true option and POST /v1/browser actions, and what the API returns today.",
	14	+ description: "Managed browser rendering on POST /v1/fetch: browser, browser_fallback, wait_for, wait_ms, wait_until, javascript, block_resources and screenshot; how automatic escalation works, what is captured, limits and errors.",
	15	+};
	16	+
	17	+const RENDERED = {
	18	+ request_id: "req_5d6e7f8g9h0i1j2k",
	19	+ success: true,
	20	+ status: 200,
	21	+ url: "https://app.example.com/dashboard",
	22	+ final_url: "https://app.example.com/dashboard?tab=results",
	23	+ content: null,
	24	+ content_type: "text/html; charset=utf-8",
	25	+ headers: { "content-type": "text/html; charset=utf-8", "cache-control": "no-store" },
	26	+ cookies: [
	27	+ { name: "session", value: "9f3a…", domain: "app.example.com", path: "/" },
	28	+ { name: "cf_clearance", value: "…", domain: ".example.com", path: "/" },
	29	+ ],
	30	+ text: "Results\nRef\tPrice\nA-1041\tCA$ 1,250\n…",
	31	+ page: { title: "Dashboard — Results", description: null, canonical: "https://app.example.com/dashboard", lang: "en", og: {}, links_count: 64 },
	32	+ screenshot: "iVBORw0KGgoAAAANSUhEUgAA…",
	33	+ metadata: {
	34	+ network: "residential",
	35	+ country: "CA",
	36	+ mode: "browser",
	37	+ attempts: 1,
	38	+ duration_ms: 4930,
	39	+ bytes: 612340,
	40	+ session: "sess_8f2k1m9d3p7q4r6s",
	41	+ cached: false,
	42	+ timing: { dns_ms: 17, proxy_connect_ms: 0, tls_ms: 0, origin_ms: 4480, processing_ms: 61, total_ms: 4930 },
	43	+ },
12	44	};
13	45
14	46	const ACTIONS: Array<[string, string, string]> = [
		`@@ -24,63 +56,183 @@ const ACTIONS: Array<[string, string, string]> = [`
24	56
25	57	export default function BrowserPage() {
26	58	return (
27		− <DocPage path="/docs/browser" eyebrow="Core API" title="Browser" description="Render JavaScript-heavy pages and drive multi-step interactions in a managed browser that inherits Fetcha's routing, geography and sessions." status="Coming soon">
28		− <ComingSoon title="Managed browser is not yet available">
29		− Nothing on this page is callable today. <Code>browser: true</Code> on <Code>/v1/fetch</Code> returns <Code>400 BROWSER_UNAVAILABLE</Code> and <Code>POST /v1/browser</Code> returns the same
30		− code. The design below is what we are building; field names may change before launch. Follow the <A href="/changelog">changelog</A> for the release.
31		− </ComingSoon>
32		−
33		− <H2>Current behaviour</H2>
	59	+ <DocPage path="/docs/browser" eyebrow="Core API" title="Browser" description="Render JavaScript-heavy pages in a managed headless Chromium that inherits Fetcha's routing, geography and sessions. Available on every fetch with browser: true, and used automatically when a plain request is blocked by a JavaScript challenge." status="Stable">
	60	+ <Endpoint method="POST" path="/v1/fetch { browser: true }" scope="fetch:execute" status="Live" />
34	61	<P>
35		− The API already reserves the surface so that clients can be written against it. Today both entry points fail fast, before any network activity, and the request is not counted as an
36		− attempt:
	62	+ Browser rendering is part of the fetch request, not a separate endpoint. The browser runs on Fetcha's infrastructure, connects through the <Strong>same proxy network, country and
	63	+ session</Strong> as a plain fetch, loads the page, waits for it to settle and returns the rendered DOM in the usual response document. No <Code>browser:use</Code> scope is needed (the scope
	64	+ name still exists for compatibility).
37	65	</P>
38		− <Endpoint method="POST" path="/v1/fetch { browser: true }" status="Coming soon" />
39		− <ResponseExample status={400} statusText="Bad Request" body={{ error: { code: "BROWSER_UNAVAILABLE", message: "Managed browser execution is not yet available.", request_id: "req_5d6e7f8g9h0i1j2k" } }} />
40		− <Endpoint method="POST" path="/v1/browser" status="Coming soon" />
41		− <ResponseExample
42		− status={400}
43		− statusText="Bad Request"
44		− body={{ error: { code: "BROWSER_UNAVAILABLE", message: "Browser actions are not yet available. Follow the changelog at https://www.fetcha.co/changelog.", request_id: "req_5d6e7f8g9h0i1j2k" } }}
	66	+
	67	+ <H2>Rendered fetch</H2>
	68	+ <ParamTable
	69	+ rows={[
	70	+ { name: "browser", type: "boolean", default: "false", description: <>Render the page in the managed browser instead of fetching it over plain HTTP.</> },
	71	+ { name: "browser_fallback", type: "boolean", default: "true", description: <>Escalate to the browser automatically when an HTTP attempt is blocked by a JavaScript challenge or anti-bot page. Set <code>false</code> to never render.</> },
	72	+ { name: "wait_until", type: '"load" \| "domcontentloaded" \| "networkidle"', default: '"domcontentloaded"', description: <>Navigation event to wait for. <code>networkidle</code> waits until no network request has been made for 500 ms; use it for pages that fetch their data after load.</> },
	73	+ { name: "wait_for", type: "string (CSS selector)", constraints: "≤ 512 chars", description: <>Selector that must be present before capture, after <code>wait_until</code>. Fails with <code>BROWSER_TIMEOUT</code> if it never appears within the timeout.</> },
	74	+ { name: "wait_ms", type: "integer (ms)", constraints: "0–30,000", description: <>Extra settle time after the wait condition and selector. Useful for pages that animate content in.</> },
	75	+ { name: "javascript", type: "boolean", default: "true", description: <>Set <code>false</code> to render with scripting disabled (useful to bypass client-side redirects or paywalls that rely on JS).</> },
	76	+ { name: "block_resources", type: "boolean", default: "true", description: <>Skip images, fonts and media. Saves bandwidth and time; scripts, stylesheets and XHR still load.</> },
	77	+ { name: "screenshot", type: "boolean", default: "false", description: <>Return a PNG of the viewport in <code>screenshot</code> (base64).</> },
	78	+ { name: "device", type: '"desktop" \| "mobile" \| "tablet"', description: <><code>mobile</code> renders with an iPhone viewport and User-Agent; <code>desktop</code> (default) uses 1366 × 768.</> },
	79	+ { name: "locale", type: "string", description: <>Sets the browser language and <code>Accept-Language</code>, e.g. <code>{`"fr-CA"`}</code>.</> },
	80	+ ]}
45	81	/>
46	82	<P>
47		− The fetch schema already accepts the companion fields <Code>javascript</Code>, <Code>wait_for</Code> and <Code>wait_ms</Code>; they are validated and ignored. The <Code>browser:use</Code> API
48		− key scope exists and can be granted, but unlocks nothing yet.
	83	+ All other fetch fields apply unchanged: <Code>country</Code>, <Code>region</Code>, <Code>city</Code>, <Code>network</Code>, <Code>session</Code>, <Code>headers</Code>, <Code>cookies</Code>,{" "}
	84	+ <Code>format</Code>, <Code>links</Code>, <Code>timeout</Code>, <Code>retries</Code> and <Code>debug</Code>. <Code>method</Code> and <Code>body</Code> are ignored in the browser; navigation is always a{" "}
	85	+ <Code>GET</Code>.
49	86	</P>
50	87
51		− <H2>Planned: rendered fetch</H2>
52		− <P>
53		− The simplest form will be a flag on the existing fetch request. Fetcha will open the URL in a headless browser routed through the same network class, geography and session as a normal fetch,
54		− wait for the page to settle, and return the rendered HTML in the usual response document.
55		− </P>
56		− <CodeBlock
57		− lang="json"
58		− title="Planned request"
59		− code={JSON.stringify({ url: "https://app.example.com/dashboard", browser: true, wait_for: "table.results", wait_ms: 500, country: "CA", session: "sess_…", format: "text" }, null, 2)}
	88	+ <H3>Example</H3>
	89	+ <P>Render a dashboard behind a login, reusing the sticky session that holds the login cookies, wait for the results table, and return readable text plus a screenshot.</P>
	90	+ <CodeTabs
	91	+ tabs={fetchTabs(
	92	+ { url: "https://app.example.com/dashboard", browser: true, wait_for: "table.results", wait_ms: 500, country: "CA", session: "sess_8f2k1m9d3p7q4r6s", format: "text", screenshot: true },
	93	+ {
	94	+ javascript: `console.log(data.metadata.mode, data.page.title); // "browser" "Dashboard — Results"\nawait fs.promises.writeFile("dashboard.png", Buffer.from(data.screenshot, "base64"));`,
	95	+ python: `print(data["metadata"]["mode"], data["page"]["title"])\nopen("dashboard.png", "wb").write(base64.b64decode(data["screenshot"]))`,
	96	+ },
	97	+ )}
60	98	/>
	99	+ <ResponseExample status={200} title="200 OK · metadata.mode: browser" body={RENDERED} />
	100	+
	101	+ <H2>What is captured</H2>
61	102	<Ul>
62	103	<Li>
63		− <Code>wait_for</Code>: CSS selector that must be present before the DOM is captured.
	104	+ <Strong>DOM after settle.</Strong> <Code>content</Code> is the serialised document (<Code>document.documentElement.outerHTML</Code>) once <Code>wait_until</Code>, <Code>wait_for</Code> and{" "}
	105	+ <Code>wait_ms</Code> are satisfied. <Code>format: text</Code> and <Code>markdown</Code> convert this rendered DOM, so client-side content is included.
64	106	</Li>
65	107	<Li>
66		− <Code>wait_ms</Code>: extra settle time after load or after the selector appears (0–30,000 ms).
	108	+ <Strong>Final URL.</Strong> <Code>final_url</Code> reflects server redirects and client-side navigations (<Code>location.replace</Code>, meta refresh, framework routers) that happened before capture.
67	109	</Li>
68	110	<Li>
69		− <Code>javascript</Code>: set to <Code>false</Code> to render with scripting disabled.
	111	+ <Strong>Status and headers of the main document.</Strong> <Code>status</Code> and <Code>headers</Code> come from the main navigation response, not from sub-resources. A page that loads but shows an
	112	+ error inside the app still reports the document's status.
70	113	</Li>
71	114	<Li>
72		− Response: same shape as today, with <Code>content</Code> holding the post-render DOM and <Code>metadata.timing</Code> gaining a browser phase.
	115	+ <Strong>Cookies.</Strong> <Code>cookies</Code> contains the browser's cookie jar for the site after rendering, including cookies set by JavaScript and challenge clearances. Replay them via the{" "}
	116	+ <Code>cookies</Code> field or keep the <Code>session</Code>.
73	117	</Li>
74	118	<Li>
75		− Errors: <Code>504 BROWSER_TIMEOUT</Code> when the page does not settle within <Code>timeout</Code>. The code is already defined.
	119	+ <Strong>Page metadata and links.</Strong> <Code>page</Code> (title, description, canonical, lang, Open Graph, link count) and, with <Code>links: true</Code>, <Code>links[]</Code> are extracted from the
	120	+ rendered DOM.
	121	+ </Li>
	122	+ <Li>
	123	+ <Strong>Optional screenshot.</Strong> A PNG of the viewport (1366 × 768 desktop, 390 × 844 mobile), base64-encoded in <Code>screenshot</Code>.
76	124	</Li>
77	125	</Ul>
78	126
79		− <H2>Planned: browser actions</H2>
	127	+ <H2>Automatic escalation</H2>
	128	+ <P>
	129	+ You rarely need to set <Code>browser: true</Code> yourself. With the default <Code>browser_fallback: true</Code>, Fetcha starts every request over plain HTTP because it is faster and cheaper.
	130	+ When an attempt is classified as a JavaScript challenge or an anti-bot interstitial (Cloudflare challenge or Turnstile, DataDome, PerimeterX, Akamai, Kasada, Imperva, AWS WAF, Vercel attack
	131	+ mode, or a soft 200 challenge page), the router escalates: it re-plays the request in the browser through the same route, letting the challenge script run and the clearance cookie be set.
	132	+ </P>
	133	+ <Ul>
	134	+ <Li>
	135	+ Escalation counts as one attempt and shares the request's single <Code>timeout</Code> and <Code>retries</Code> budget.
	136	+ </Li>
	137	+ <Li>
	138	+ <Code>metadata.mode</Code> is <Code>browser</Code> when the final attempt was rendered. With <Code>debug: true</Code>, each attempt lists its <Code>mode</Code> and, for blocked ones, the{" "}
	139	+ <Code>block_reason</Code>.
	140	+ </Li>
	141	+ <Li>Plain 403/429 blocks without a JavaScript challenge are retried on a new IP or a premium route first; the browser is used when the block needs a script to be executed.</Li>
	142	+ <Li>
	143	+ Set <Code>browser_fallback: false</Code> for latency-sensitive calls where a blocked answer is acceptable, or when you handle challenges yourself.
	144	+ </Li>
	145	+ </Ul>
	146	+ <Callout variant="info" title="Learning">
	147	+ Escalations feed the per-domain profile like any other attempt. A site that consistently needs the browser will be rendered directly on later requests, saving the wasted HTTP attempt.
	148	+ </Callout>
	149	+
	150	+ <H2>Limits and errors</H2>
	151	+ <Table dense>
	152	+ <THead>
	153	+ <Tr>
	154	+ <Th>Limit</Th>
	155	+ <Th>Value</Th>
	156	+ </Tr>
	157	+ </THead>
	158	+ <TBody>
	159	+ <Tr>
	160	+ <Td>Concurrent renders per organization</Td>
	161	+ <Td mono>8</Td>
	162	+ </Tr>
	163	+ <Tr>
	164	+ <Td>Render time</Td>
	165	+ <Td>
	166	+ Bounded by the request <Code>timeout</Code> (max 120 s), shared with any HTTP attempts made before escalation
	167	+ </Td>
	168	+ </Tr>
	169	+ <Tr>
	170	+ <Td>Viewport</Td>
	171	+ <Td mono>1366 × 768 desktop · 390 × 844 mobile</Td>
	172	+ </Tr>
	173	+ <Tr>
	174	+ <Td>Response size</Td>
	175	+ <Td>Same 20 MB cap as plain fetches, applied to the serialised DOM. Screenshots are not counted.</Td>
	176	+ </Tr>
	177	+ </TBody>
	178	+ </Table>
	179	+ <Table dense>
	180	+ <THead>
	181	+ <Tr>
	182	+ <Th>Situation</Th>
	183	+ <Th>Result</Th>
	184	+ </Tr>
	185	+ </THead>
	186	+ <TBody>
	187	+ <Tr>
	188	+ <Td>
	189	+ Page did not reach <Code>wait_until</Code>, <Code>wait_for</Code> never appeared, or the render exceeded the remaining <Code>timeout</Code>
	190	+ </Td>
	191	+ <Td mono>504 BROWSER_TIMEOUT</Td>
	192	+ </Tr>
	193	+ <Tr>
	194	+ <Td>Browser pool disabled or unavailable</Td>
	195	+ <Td mono>400 BROWSER_UNAVAILABLE</Td>
	196	+ </Tr>
	197	+ <Tr>
	198	+ <Td>More than 8 renders in flight for the organization</Td>
	199	+ <Td mono>429 CONCURRENCY_LIMIT</Td>
	200	+ </Tr>
	201	+ <Tr>
	202	+ <Td>Target blocked even in the browser, on every attempt</Td>
	203	+ <Td>
	204	+ <Code>200</Code> with <Code>success: false</Code> and the last page, like any block
	205	+ </Td>
	206	+ </Tr>
	207	+ </TBody>
	208	+ </Table>
80	209	<P>
81		− For interactions that need several steps, <Code>POST /v1/browser</Code> will accept an ordered list of actions executed in one browser context. Each action returns its own result and the
82		− final response includes the DOM, cookies and any screenshots.
	210	+ Rendered requests are logged like any other request: the request log shows <Strong>mode</Strong> per attempt, and bandwidth is metered on the bytes the browser actually transferred (which is
	211	+ why <Code>block_resources</Code> is on by default).
83	212	</P>
	213	+
	214	+ <H2>Tips</H2>
	215	+ <Ul>
	216	+ <Li>
	217	+ Prefer <Code>wait_for</Code> over a large <Code>wait_ms</Code>: it returns as soon as the content exists and fails clearly when it never does.
	218	+ </Li>
	219	+ <Li>
	220	+ Use <Code>{`format: "markdown"`}</Code> or <Code>{`"text"`}</Code> with the browser to get the rendered content without shipping the framework's HTML.
	221	+ </Li>
	222	+ <Li>
	223	+ Combine with a <A href="/docs/sessions">session</A> for logged-in areas: the browser reuses the session's IP and the cookies you pass, and returns the updated jar.
	224	+ </Li>
	225	+ <Li>
	226	+ If the site exposes the JSON endpoint the page calls, fetching it directly with <Code>{`format: "json"`}</Code> is still faster than rendering. See <A href="/docs/examples">Examples</A>.
	227	+ </Li>
	228	+ </Ul>
	229	+
	230	+ <H2>Coming soon: browser actions</H2>
	231	+ <ComingSoon title="POST /v1/browser is not yet available">
	232	+ Multi-step interactions (click, type, scroll, evaluate) in one browser context are planned. <Code>POST /v1/browser</Code> currently returns <Code>400 BROWSER_UNAVAILABLE</Code>. The design below
	233	+ is indicative; field names may change before launch. Follow the <A href="/changelog">changelog</A>.
	234	+ </ComingSoon>
	235	+ <Endpoint method="POST" path="/v1/browser" status="Coming soon" />
84	236	<Table>
85	237	<THead>
86	238	<Tr>
		`@@ -120,14 +272,6 @@ export default function BrowserPage() {`
120	272	2,
121	273	)}
122	274	/>
123		− <Callout variant="info" title="Pricing and limits">
124		− Browser requests will be priced separately from plain fetches and subject to their own concurrency limit; they require the <Code>browser:use</Code> scope. Details will be published with
125		− the launch. Until then, sites that need JavaScript are best handled by calling their underlying JSON endpoints with <Code>{`format: "json"`}</Code>; see{" "}
126		− <Strong>
127		− <A href="/docs/examples">Examples</A>
128		− </Strong>
129		− .
130		− </Callout>
131	275	</DocPage>
132	276	);
133	277	}

added apps/web/src/app/(marketing)/docs/crawl/page.tsx +379 −0

		`@@ -0,0 +1,379 @@`
	1	+import type { Metadata } from "next";
	2	+import { CodeBlock } from "@/components/ui/code-block";
	3	+import { DocPage } from "@/components/docs/doc-page";
	4	+import { A, Code, H2, H3, Li, P, Strong, Table, TBody, Td, Th, THead, Tr, Ul } from "@/components/docs/prose";
	5	+import { Callout } from "@/components/docs/callout";
	6	+import { Endpoint } from "@/components/docs/endpoint";
	7	+import { ParamTable } from "@/components/docs/param-table";
	8	+import { CodeTabs } from "@/components/docs/code-tabs";
	9	+import { ResponseExample } from "@/components/docs/response-example";
	10	+import { apiTabs } from "@/components/docs/snippets";
	11	+
	12	+export const metadata: Metadata = {
	13	+ title: "Crawl & Map",
	14	+ description: "Crawl a site into Markdown, text or HTML with POST /v1/crawl (asynchronous jobs, pages, cancel) and discover its URLs with POST /v1/map. Options, statuses, pagination, limits and polling examples.",
	15	+};
	16	+
	17	+const CREATE_BODY = {
	18	+ url: "https://docs.example.com/",
	19	+ max_pages: 200,
	20	+ max_depth: 3,
	21	+ include_patterns: ["/docs/*"],
	22	+ exclude_patterns: ["/changelog", "/\\.(pdf\|zip)$/"],
	23	+ format: "markdown",
	24	+ country: "CA",
	25	+ label: "docs-site",
	26	+};
	27	+
	28	+const CREATED = {
	29	+ id: "crawl_3k9d0f2a8b1c7e4m",
	30	+ status: "queued",
	31	+ seed_url: "https://docs.example.com/",
	32	+ created_at: "2026-09-08T14:02:11.000Z",
	33	+ options: {
	34	+ url: "https://docs.example.com/",
	35	+ max_pages: 200,
	36	+ max_depth: 3,
	37	+ same_domain: true,
	38	+ allow_subdomains: false,
	39	+ include_patterns: ["/docs/*"],
	40	+ exclude_patterns: ["/changelog", "/\\.(pdf\|zip)$/"],
	41	+ respect_robots: true,
	42	+ use_sitemap: false,
	43	+ concurrency: 3,
	44	+ delay_ms: 0,
	45	+ timeout: 30000,
	46	+ format: "markdown",
	47	+ main_content: true,
	48	+ country: "CA",
	49	+ network: "auto",
	50	+ browser: false,
	51	+ browser_fallback: true,
	52	+ label: "docs-site",
	53	+ },
	54	+};
	55	+
	56	+const JOB = {
	57	+ id: "crawl_3k9d0f2a8b1c7e4m",
	58	+ status: "completed",
	59	+ label: "docs-site",
	60	+ seed_url: "https://docs.example.com/",
	61	+ domain: "docs.example.com",
	62	+ options: { "…": "as submitted, with defaults filled in" },
	63	+ stats: { discovered: 312, fetched: 200, ok: 196, blocked: 1, failed: 3, bytes: 18422911 },
	64	+ error: null,
	65	+ created_at: "2026-09-08T14:02:11.000Z",
	66	+ started_at: "2026-09-08T14:02:12.000Z",
	67	+ completed_at: "2026-09-08T14:06:48.000Z",
	68	+};
	69	+
	70	+const PAGES = {
	71	+ data: [
	72	+ {
	73	+ id: "cpg_9a1b2c3d4e5f6g7h",
	74	+ url: "https://docs.example.com/docs/getting-started",
	75	+ final_url: "https://docs.example.com/docs/getting-started",
	76	+ depth: 1,
	77	+ status: "success",
	78	+ http_status: 200,
	79	+ error_code: null,
	80	+ title: "Getting started — Example Docs",
	81	+ description: "Install the CLI and run your first command.",
	82	+ content_type: "text/html; charset=utf-8",
	83	+ content: "# Getting started\n\nInstall the CLI:\n\n```bash\nnpm i -g example\n```\n…",
	84	+ links_count: 41,
	85	+ bytes: 88412,
	86	+ duration_ms: 812,
	87	+ mode: "http",
	88	+ fetched_at: "2026-09-08T14:02:14.000Z",
	89	+ },
	90	+ {
	91	+ id: "cpg_1h2g3f4e5d6c7b8a",
	92	+ url: "https://docs.example.com/docs/pricing",
	93	+ final_url: "https://docs.example.com/docs/pricing",
	94	+ depth: 1,
	95	+ status: "success",
	96	+ http_status: 200,
	97	+ error_code: null,
	98	+ title: "Pricing — Example Docs",
	99	+ description: null,
	100	+ content_type: "text/html; charset=utf-8",
	101	+ content: "# Pricing\n\n\| Plan \| Price \|\n\| --- \| --- \|\n…",
	102	+ links_count: 27,
	103	+ bytes: 210934,
	104	+ duration_ms: 5104,
	105	+ mode: "browser",
	106	+ fetched_at: "2026-09-08T14:02:21.000Z",
	107	+ },
	108	+ ],
	109	+ next_cursor: "eyJmIjoiMjAyNi0wOS0wOFQxNDowMjoyMS4wMDBaIiwiaWQiOiJjcGdfMWgyZzNmNGU1ZDZjN2I4YSJ9",
	110	+};
	111	+
	112	+const MAP_RESULT = {
	113	+ url: "https://docs.example.com/",
	114	+ count: 4,
	115	+ urls: ["https://docs.example.com/", "https://docs.example.com/docs/getting-started", "https://docs.example.com/docs/pricing", "https://docs.example.com/docs/api/fetch"],
	116	+ sources: { sitemap: 312, links: 58 },
	117	+ truncated: false,
	118	+};
	119	+
	120	+const POLL_JS = `import { Fetcha } from "@fetcha/sdk";
	121	+
	122	+const fetcha = new Fetcha({ apiKey: process.env.FETCHA_API_KEY! });
	123	+
	124	+// 1. Start the job (returns immediately with status "queued")
	125	+const job = await fetcha.crawl.create({ url: "https://docs.example.com/", max_pages: 200, max_depth: 3, format: "markdown" });
	126	+
	127	+// 2. Wait for a terminal status (polls GET /v1/crawl/:id every 2 s, up to 10 min)
	128	+const done = await fetcha.crawl.wait(job.id, { pollMs: 2000, timeoutMs: 600_000 });
	129	+console.log(done.status, done.stats); // "completed" { discovered, fetched, ok, blocked, failed, bytes }
	130	+
	131	+// 3. Page through the results with the cursor
	132	+let cursor: string \| null = null;
	133	+do {
	134	+ const page = await fetcha.crawl.pages(job.id, { cursor, limit: 100, status: "success" });
	135	+ for (const p of page.data) console.log(p.url, p.title, p.content?.length);
	136	+ cursor = page.next_cursor;
	137	+} while (cursor);`;
	138	+
	139	+const POLL_PY = `import os
	140	+from fetcha import Fetcha
	141	+
	142	+client = Fetcha(api_key=os.environ["FETCHA_API_KEY"])
	143	+
	144	+# 1. Start the job
	145	+job = client.crawl.create(url="https://docs.example.com/", max_pages=200, max_depth=3, format="markdown")
	146	+
	147	+# 2. Wait for a terminal status (polls GET /v1/crawl/:id)
	148	+job = client.crawl.wait(job.id, poll_s=2.0, timeout_s=600)
	149	+print(job.status, job.stats)
	150	+
	151	+# 3. Page through the results
	152	+cursor = None
	153	+while True:
	154	+ page = client.crawl.pages(job.id, cursor=cursor, limit=100, status="success")
	155	+ for p in page.data:
	156	+ print(p.url, p.title, len(p.content or ""))
	157	+ cursor = page.next_cursor
	158	+ if not cursor:
	159	+ break`;
	160	+
	161	+const POLL_RAW = `# Without the SDK: poll until status is completed \| failed \| cancelled
	162	+JOB=$(curl -s https://www.fetcha.co/v1/crawl -X POST \\
	163	+ -H "Authorization: Bearer $FETCHA_API_KEY" -H "Content-Type: application/json" \\
	164	+ -d '{"url": "https://docs.example.com/", "max_pages": 200}' \| jq -r .id)
	165	+
	166	+until [ "$(curl -s https://www.fetcha.co/v1/crawl/$JOB -H "Authorization: Bearer $FETCHA_API_KEY" \| jq -r .status)" != "running" ]; do sleep 2; done
	167	+
	168	+curl -s "https://www.fetcha.co/v1/crawl/$JOB/pages?limit=100" -H "Authorization: Bearer $FETCHA_API_KEY" \| jq '.data[] \| {url, status, title}'`;
	169	+
	170	+export default function CrawlPage() {
	171	+ return (
	172	+ <DocPage path="/docs/crawl" eyebrow="Core API" title="Crawl & Map" description="Turn a site into Markdown, text or HTML in one call: POST /v1/crawl starts an asynchronous job that follows links from a seed URL and fetches every page through the routing engine. POST /v1/map lists a site's URLs synchronously without fetching them." status="Stable">
	173	+ <H2>How a crawl works</H2>
	174	+ <Ul>
	175	+ <Li>
	176	+ <Strong>Seed and frontier.</Strong> The seed URL is fetched first. Links are extracted, made absolute, normalised (fragments removed, tracking parameters kept) and added to the frontier if they pass the
	177	+ scope rules: <Code>same_domain</Code> / <Code>allow_subdomains</Code>, <Code>include_patterns</Code> / <Code>exclude_patterns</Code>, <Code>max_depth</Code> and, unless disabled,{" "}
	178	+ <Code>robots.txt</Code>.
	179	+ </Li>
	180	+ <Li>
	181	+ <Strong>Fetching.</Strong> Up to <Code>concurrency</Code> workers fetch pages until <Code>max_pages</Code> is reached or the frontier is empty. Each page is a normal fetch: routing intelligence,
	182	+ retries, block detection and automatic browser escalation apply, and the page appears in the request log with <Code>{`source: "crawl"`}</Code>.
	183	+ </Li>
	184	+ <Li>
	185	+ <Strong>Content.</Strong> Each page is stored in the requested <Code>format</Code> with its title, description, HTTP status, mode, bytes and duration. <Code>markdown</Code> (default) keeps the main
	186	+ content and drops navigation, footers and cookie banners when <Code>main_content</Code> is true.
	187	+ </Li>
	188	+ <Li>
	189	+ <Strong>Completion.</Strong> The job ends <Code>completed</Code> when the frontier is exhausted or the page budget is spent, <Code>failed</Code> if the seed could not be fetched, or{" "}
	190	+ <Code>cancelled</Code> if you deleted it. Pages fetched before a cancellation remain available.
	191	+ </Li>
	192	+ </Ul>
	193	+ <Callout variant="info" title="Limits">
	194	+ Up to <Strong>2,000 pages per job</Strong> and <Strong>5 concurrent jobs</Strong> per organization (a sixth returns <Code>429 CRAWL_LIMIT_REACHED</Code>). Jobs and their pages are kept for the
	195	+ log retention period (90 days).
	196	+ </Callout>
	197	+
	198	+ <H2>Start a crawl</H2>
	199	+ <Endpoint method="POST" path="/v1/crawl" scope="fetch:execute" status="Live" />
	200	+ <ParamTable
	201	+ rows={[
	202	+ { name: "url", type: "string", required: true, constraints: "1–8,192 chars, http or https", description: <>Seed URL. Same URL policy as fetch (<code>URL_NOT_ALLOWED</code> for private or internal hosts).</> },
	203	+ { name: "max_pages", type: "integer", default: "25", constraints: "1–5,000, capped at 2,000", description: <>Maximum number of pages to fetch. The seed counts as one.</> },
	204	+ { name: "max_depth", type: "integer", default: "2", constraints: "0–10", description: <>Maximum link depth from the seed. <code>0</code> fetches the seed only.</> },
	205	+ { name: "same_domain", type: "boolean", default: "true", description: <>Only follow links whose registrable domain matches the seed.</> },
	206	+ { name: "allow_subdomains", type: "boolean", default: "false", description: <>With <code>same_domain</code>, also follow links on subdomains of the seed host.</> },
	207	+ { name: "include_patterns", type: "string[]", constraints: "≤ 50 patterns, each ≤ 512 chars", description: <>Only crawl URLs matching at least one pattern. Glob with <code>*</code> (matched against the full URL or the path) or a regular expression written <code>/…/</code>.</> },
	208	+ { name: "exclude_patterns", type: "string[]", constraints: "≤ 50 patterns", description: <>Never crawl URLs matching one of these patterns. Evaluated after <code>include_patterns</code>.</> },
	209	+ { name: "respect_robots", type: "boolean", default: "true", description: <>Honour <code>Disallow</code> rules of the seed host's <code>robots.txt</code>.</> },
	210	+ { name: "use_sitemap", type: "boolean", default: "false", description: <>Also seed the frontier with URLs from <code>sitemap.xml</code> and sitemaps listed in <code>robots.txt</code> (subject to the same scope rules).</> },
	211	+ { name: "concurrency", type: "integer", default: "3", constraints: "1–10", description: <>Parallel page fetches within the job.</> },
	212	+ { name: "delay_ms", type: "integer", default: "0", constraints: "0–30,000", description: <>Pause between fetches per worker (politeness).</> },
	213	+ { name: "timeout", type: "integer (ms)", default: "30000", constraints: "1,000–120,000", description: <>Per-page timeout, including retries and browser escalation.</> },
	214	+ { name: "format", type: '"markdown" \| "text" \| "html"', default: '"markdown"', description: <>Format of <code>content</code> for each page.</> },
	215	+ { name: "main_content", type: "boolean", default: "true", description: <>Keep only the main content (article, main, largest text block) when converting to Markdown or text.</> },
	216	+ { name: "country", type: "string", constraints: "2 chars", description: <>Exit country for every page.</> },
	217	+ { name: "network", type: '"auto" \| "datacenter" \| "residential" \| "isp" \| "mobile"', default: '"auto"', description: <>Network class for every page.</> },
	218	+ { name: "browser", type: "boolean", default: "false", description: <>Render every page in the managed browser. Slower and heavier; prefer the default and let escalation handle JavaScript challenges.</> },
	219	+ { name: "browser_fallback", type: "boolean", default: "true", description: <>Escalate blocked pages to the browser automatically.</> },
	220	+ { name: "headers", type: "object<string, string>", constraints: "≤ 64 entries", description: <>Headers sent with every page fetch.</> },
	221	+ { name: "webhook_url", type: "string (url)", constraints: "≤ 2,048 chars", description: <>Called with a <code>POST</code> containing the job when it reaches a terminal status. Accepted and stored; delivery is part of the upcoming webhooks release.</> },
	222	+ { name: "label", type: "string", constraints: "≤ 128 chars", description: <>Free-form label shown in the dashboard.</> },
	223	+ ]}
	224	+ />
	225	+ <CodeTabs tabs={apiTabs({ method: "POST", path: "/v1/crawl", body: CREATE_BODY, after: { javascript: `console.log(data.id, data.status); // "crawl_…" "queued"`, python: `print(data["id"], data["status"]) # crawl_… queued` } })} />
	226	+ <ResponseExample status={202} statusText="Accepted" body={CREATED} />
	227	+ <P>
	228	+ The response is <Code>202 Accepted</Code>: the job is queued and the call returns immediately. Poll <Code>GET /v1/crawl/:id</Code> (or use the SDK's <Code>crawl.wait</Code>) to follow progress.
	229	+ </P>
	230	+
	231	+ <H2>Get a job</H2>
	232	+ <Endpoint method="GET" path="/v1/crawl/:id" scope={null} status="Live" />
	233	+ <CodeTabs tabs={apiTabs({ method: "GET", path: "/v1/crawl/crawl_3k9d0f2a8b1c7e4m" }, ["curl", "javascript", "python"])} />
	234	+ <ResponseExample status={200} body={JOB} />
	235	+ <ParamTable
	236	+ showDefault={false}
	237	+ rows={[
	238	+ { name: "id", type: "string", description: <>Job identifier (<code>crawl_…</code>).</> },
	239	+ { name: "status", type: '"queued" \| "running" \| "completed" \| "failed" \| "cancelled"', description: <><code>completed</code>, <code>failed</code> and <code>cancelled</code> are terminal.</> },
	240	+ { name: "label", type: "string \| null", description: <>Your label.</> },
	241	+ { name: "seed_url / domain", type: "string", description: <>The seed and its host.</> },
	242	+ { name: "options", type: "object", description: <>The options as submitted, with defaults filled in.</> },
	243	+ { name: "stats", type: "object", description: <><code>{`{ discovered, fetched, ok, blocked, failed, bytes }`}</code>. <code>discovered</code> counts URLs added to the frontier; <code>fetched = ok + blocked + failed</code>.</> },
	244	+ { name: "error", type: "object \| null", description: <><code>{`{ code, message }`}</code> when <code>status</code> is <code>failed</code> (for example the seed returned <code>URL_NOT_ALLOWED</code> or was blocked on every attempt).</> },
	245	+ { name: "created_at / started_at / completed_at", type: "string \| null", description: <>ISO 8601 timestamps. <code>started_at</code> is <code>null</code> while queued; <code>completed_at</code> is set on any terminal status.</> },
	246	+ ]}
	247	+ />
	248	+
	249	+ <H2>List pages</H2>
	250	+ <Endpoint method="GET" path="/v1/crawl/:id/pages" scope={null} status="Live" />
	251	+ <ParamTable
	252	+ caption="Query parameters"
	253	+ rows={[
	254	+ { name: "cursor", type: "string", description: <>Opaque cursor from the previous response's <code>next_cursor</code>. Omit for the first page.</> },
	255	+ { name: "limit", type: "integer", default: "100", description: <>Pages per response.</> },
	256	+ { name: "status", type: '"success" \| "blocked" \| "failed"', description: <>Only return pages with this status.</> },
	257	+ ]}
	258	+ />
	259	+ <CodeTabs tabs={apiTabs({ method: "GET", path: "/v1/crawl/crawl_3k9d0f2a8b1c7e4m/pages?limit=100&status=success" }, ["curl", "javascript", "python"])} />
	260	+ <ResponseExample status={200} body={PAGES} />
	261	+ <ParamTable
	262	+ showDefault={false}
	263	+ rows={[
	264	+ { name: "id", type: "string", description: <>Page identifier.</> },
	265	+ { name: "url / final_url", type: "string", description: <>Requested URL and URL after redirects.</> },
	266	+ { name: "depth", type: "integer", description: <>Link distance from the seed (seed = 0).</> },
	267	+ { name: "status", type: '"success" \| "blocked" \| "failed"', description: <>Outcome of the fetch. <code>blocked</code> means every attempt, including browser escalation, was classified as a block.</> },
	268	+ { name: "http_status", type: "integer \| null", description: <>Status of the final attempt.</> },
	269	+ { name: "error_code", type: "string \| null", description: <>Fetch error code for failed pages (<code>TARGET_TIMEOUT</code>, <code>TARGET_UNAVAILABLE</code>, …).</> },
	270	+ { name: "title / description", type: "string \| null", description: <>From the page's <code><title></code> and meta description.</> },
	271	+ { name: "content_type", type: "string \| null", description: <>The target's Content-Type.</> },
	272	+ { name: "content", type: "string \| null", description: <>Page content in the job's <code>format</code>. <code>null</code> for non-HTML bodies and blocked pages.</> },
	273	+ { name: "links_count", type: "integer \| null", description: <>Hyperlinks found in the page.</> },
	274	+ { name: "bytes / duration_ms", type: "integer \| null", description: <>Bytes transferred and wall-clock time for this page, all attempts included.</> },
	275	+ { name: "mode", type: '"http" \| "browser" \| null', description: <>How the final attempt was made.</> },
	276	+ { name: "fetched_at", type: "string \| null", description: <>ISO 8601 timestamp.</> },
	277	+ ]}
	278	+ />
	279	+ <P>
	280	+ Pages are ordered by <Code>fetched_at</Code>. <Code>next_cursor</Code> is <Code>null</Code> on the last page. While a job is running, new pages appear at the end; keep the last cursor to fetch only
	281	+ what is new.
	282	+ </P>
	283	+
	284	+ <H2>Cancel a job</H2>
	285	+ <Endpoint method="DELETE" path="/v1/crawl/:id" scope="fetch:execute" status="Live" />
	286	+ <CodeTabs tabs={apiTabs({ method: "DELETE", path: "/v1/crawl/crawl_3k9d0f2a8b1c7e4m" }, ["curl", "javascript", "python"])} />
	287	+ <ResponseExample status={200} body={{ id: "crawl_3k9d0f2a8b1c7e4m", status: "cancelled" }} />
	288	+ <P>
	289	+ Cancellation is immediate for queued jobs; running jobs stop after the in-flight pages finish. Pages already fetched stay available. Cancelling a job that is already terminal returns it unchanged.
	290	+ </P>
	291	+
	292	+ <H2>List jobs</H2>
	293	+ <Endpoint method="GET" path="/v1/crawl" scope={null} status="Live" />
	294	+ <P>
	295	+ Returns the most recent jobs of the project, newest first. <Code>?limit=</Code> defaults to 50.
	296	+ </P>
	297	+ <CodeTabs tabs={apiTabs({ method: "GET", path: "/v1/crawl?limit=50" }, ["curl", "javascript", "python"])} />
	298	+ <ResponseExample status={200} body={{ data: [JOB] }} />
	299	+ <Table dense>
	300	+ <THead>
	301	+ <Tr>
	302	+ <Th>Condition</Th>
	303	+ <Th>Result</Th>
	304	+ </Tr>
	305	+ </THead>
	306	+ <TBody>
	307	+ <Tr>
	308	+ <Td>Invalid body (unknown field, pattern too long, out-of-range value)</Td>
	309	+ <Td mono>400 INVALID_REQUEST</Td>
	310	+ </Tr>
	311	+ <Tr>
	312	+ <Td>Seed URL private, internal or non-http</Td>
	313	+ <Td mono>400 URL_NOT_ALLOWED</Td>
	314	+ </Tr>
	315	+ <Tr>
	316	+ <Td>Id does not exist or belongs to another project</Td>
	317	+ <Td mono>404 CRAWL_NOT_FOUND</Td>
	318	+ </Tr>
	319	+ <Tr>
	320	+ <Td>Five jobs already queued or running</Td>
	321	+ <Td mono>429 CRAWL_LIMIT_REACHED</Td>
	322	+ </Tr>
	323	+ </TBody>
	324	+ </Table>
	325	+
	326	+ <H2>Polling example</H2>
	327	+ <P>
	328	+ The SDKs wrap the four calls and add <Code>crawl.wait</Code>, which polls until a terminal status. Both examples start a job, wait for it, then stream the pages with the cursor.
	329	+ </P>
	330	+ <CodeBlock lang="typescript" title="crawl.ts" code={POLL_JS} />
	331	+ <CodeBlock lang="python" title="crawl.py" code={POLL_PY} />
	332	+ <CodeBlock lang="bash" title="Terminal" code={POLL_RAW} />
	333	+
	334	+ <H2>Map a site</H2>
	335	+ <Endpoint method="POST" path="/v1/map" scope="fetch:execute" status="Live" />
	336	+ <P>
	337	+ <Code>map</Code> answers the question “which URLs does this site have?” without fetching every page. It reads the sitemap(s) and the links of the seed page, filters them, and returns a
	338	+ de-duplicated list, <Strong>synchronously</Strong> (the call takes up to 60 s). Use it to pick <Code>include_patterns</Code> before a crawl, or to feed URLs into your own fetch loop.
	339	+ </P>
	340	+ <ParamTable
	341	+ rows={[
	342	+ { name: "url", type: "string", required: true, description: <>Site or page to map.</> },
	343	+ { name: "limit", type: "integer", default: "1000", constraints: "1–10,000", description: <>Maximum number of URLs returned. <code>truncated</code> is <code>true</code> when more were found.</> },
	344	+ { name: "use_sitemap", type: "boolean", default: "true", description: <>Read <code>sitemap.xml</code>, sitemap indexes and sitemaps listed in <code>robots.txt</code>.</> },
	345	+ { name: "use_links", type: "boolean", default: "true", description: <>Include hyperlinks found on the seed page.</> },
	346	+ { name: "same_domain", type: "boolean", default: "true", description: <>Drop URLs outside the seed's registrable domain.</> },
	347	+ { name: "allow_subdomains", type: "boolean", default: "false", description: <>Keep subdomains of the seed host.</> },
	348	+ { name: "search", type: "string", constraints: "≤ 256 chars", description: <>Keep only URLs matching this substring, glob (<code>*</code>) or <code>/regex/</code>.</> },
	349	+ { name: "country / network / timeout", type: "—", description: <>Same meaning as in fetch; apply to the sitemap and seed-page requests.</> },
	350	+ ]}
	351	+ />
	352	+ <CodeTabs tabs={apiTabs({ method: "POST", path: "/v1/map", body: { url: "https://docs.example.com/", search: "/docs/*", limit: 500 }, after: { javascript: `console.log(data.count, data.urls.slice(0, 3));`, python: `print(data["count"], data["urls"][:3])` } })} />
	353	+ <ResponseExample status={200} body={MAP_RESULT} />
	354	+ <ParamTable
	355	+ showDefault={false}
	356	+ rows={[
	357	+ { name: "url", type: "string", description: <>The seed you passed.</> },
	358	+ { name: "count", type: "integer", description: <>Number of URLs in <code>urls</code>.</> },
	359	+ { name: "urls", type: "string[]", description: <>Absolute, de-duplicated URLs; sitemap entries first, then links in document order.</> },
	360	+ { name: "sources", type: "object", description: <><code>{`{ sitemap, links }`}</code>: how many candidates each source contributed before filtering and de-duplication.</> },
	361	+ { name: "truncated", type: "boolean", description: <><code>true</code> when the result was cut at <code>limit</code>.</> },
	362	+ ]}
	363	+ />
	364	+ <H3>Map, then crawl</H3>
	365	+ <CodeBlock
	366	+ lang="typescript"
	367	+ code={`const map = await fetcha.map({ url: "https://docs.example.com/", search: "/docs/api/*" });
	368	+console.log(map.count, "API pages");
	369	+
	370	+// Crawl exactly that section
	371	+const job = await fetcha.crawl.create({ url: "https://docs.example.com/docs/api/", include_patterns: ["/docs/api/*"], max_pages: map.count });
	372	+const done = await fetcha.crawl.wait(job.id);`}
	373	+ />
	374	+ <P>
	375	+ A map costs a handful of fetches (one per sitemap file plus the seed page) and does not create a job. See the <A href="/docs/sdks">SDKs</A> page for the full client reference.
	376	+ </P>
	377	+ </DocPage>
	378	+ );
	379	+}

modified apps/web/src/app/(marketing)/docs/errors/page.tsx +6 −4

		`@@ -26,8 +26,8 @@ const NOTES: Record<ErrorCode, { retry: "no" \| "after-fix" \| "backoff" \| "later"`
26	26	TARGET_UNAVAILABLE: { retry: "later", note: "DNS resolution failed, no address records, or the connection was refused on every attempt." },
27	27	PROVIDER_UNAVAILABLE: { retry: "backoff", note: "No route is available for an auto request (all circuits open or nothing configured for the geography). Transient; retry with backoff." },
28	28	NETWORK_UNAVAILABLE: { retry: "after-fix", note: "Explicit class not in your plan, not live yet, or without a sticky-capable route (sessions). Use auto or residential." },
29		− BROWSER_UNAVAILABLE: { retry: "no", note: "browser: true or POST /v1/browser. Managed browser is not yet available." },
30		− BROWSER_TIMEOUT: { retry: "later", note: "Reserved for the managed browser. Not emitted today." },
	29	+ BROWSER_UNAVAILABLE: { retry: "later", note: "Browser pool disabled or unavailable. Retry later, or send the request without browser: true." },
	30	+ BROWSER_TIMEOUT: { retry: "later", note: "The page did not settle in the managed browser within the request timeout. Raise timeout, relax wait_for / wait_until, or retry." },
31	31	RESPONSE_TOO_LARGE: { retry: "after-fix", note: "Body exceeded max_response_bytes or the 20 MB platform cap. Not retried across routes." },
32	32	TOO_MANY_REDIRECTS: { retry: "after-fix", note: "More than max_redirects hops. Not retried across routes." },
33	33	INSUFFICIENT_CREDITS: { retry: "after-fix", note: "Reserved for prepaid balances. Not emitted today." },
		`@@ -37,6 +37,8 @@ const NOTES: Record<ErrorCode, { retry: "no" \| "after-fix" \| "backoff" \| "later"`
37	37	NOT_FOUND: { retry: "no", note: "Unknown route or resource. The message names the route." },
38	38	FORBIDDEN: { retry: "after-fix", note: "Key lacks the required scope, project archived, or organization/account suspended." },
39	39	INTERNAL_ERROR: { retry: "backoff", note: "Unexpected failure inside Fetcha. Retry with backoff; if it persists, send us the request_id." },
	40	+ CRAWL_NOT_FOUND: { retry: "after-fix", note: "GET/DELETE /v1/crawl/:id with an id that does not exist or belongs to another project." },
	41	+ CRAWL_LIMIT_REACHED: { retry: "later", note: "Too many crawl jobs running for the organization (5 concurrent). Wait for one to finish or cancel it." },
40	42	};
41	43
42	44	const RETRY_LABEL: Record<(typeof NOTES)[ErrorCode]["retry"], string> = {
		`@@ -49,9 +51,9 @@ const RETRY_LABEL: Record<(typeof NOTES)[ErrorCode]["retry"], string> = {`
49	51	const GROUPS: Array<{ title: string; codes: ErrorCode[] }> = [
50	52	{ title: "Authentication and authorization", codes: ["INVALID_API_KEY", "EMAIL_NOT_VERIFIED", "FORBIDDEN"] },
51	53	{ title: "Request validation", codes: ["INVALID_REQUEST", "URL_NOT_ALLOWED", "NETWORK_UNAVAILABLE", "BROWSER_UNAVAILABLE", "NOT_FOUND"] },
52		− { title: "Limits and quotas", codes: ["RATE_LIMITED", "CONCURRENCY_LIMIT", "USAGE_LIMIT_REACHED", "INSUFFICIENT_CREDITS"] },
	54	+ { title: "Limits and quotas", codes: ["RATE_LIMITED", "CONCURRENCY_LIMIT", "USAGE_LIMIT_REACHED", "INSUFFICIENT_CREDITS", "CRAWL_LIMIT_REACHED"] },
53	55	{ title: "Target and network", codes: ["TARGET_TIMEOUT", "TARGET_BLOCKED", "TARGET_UNAVAILABLE", "PROVIDER_UNAVAILABLE", "RESPONSE_TOO_LARGE", "TOO_MANY_REDIRECTS", "BROWSER_TIMEOUT"] },
54		− { title: "Sessions", codes: ["SESSION_NOT_FOUND", "SESSION_EXPIRED"] },
	56	+ { title: "Sessions and crawls", codes: ["SESSION_NOT_FOUND", "SESSION_EXPIRED", "CRAWL_NOT_FOUND"] },
55	57	{ title: "Server", codes: ["INTERNAL_ERROR"] },
56	58	];
57	59

modified apps/web/src/app/(marketing)/docs/fetch/page.tsx +85 −21

		`@@ -14,7 +14,7 @@ import { fetchTabs } from "@/components/docs/snippets";`
14	14
15	15	export const metadata: Metadata = {
16	16	title: "Fetch API",
17		− description: "Complete reference for POST /v1/fetch: every request field, the response shape, formats, redirects, size limits and error behaviour.",
	17	+ description: "Complete reference for POST /v1/fetch: every request field including browser rendering, the response shape, formats (html, text, markdown, json, raw), page metadata, links, redirects, size limits and error behaviour.",
18	18	};
19	19
20	20	const EXAMPLE_BODY = {
		`@@ -43,16 +43,49 @@ const SUCCESS = {`
43	43	},
44	44	cookies: [{ name: "sid", value: "3f9a…", domain: "example.com", path: "/" }],
45	45	text: "Example Product 42\nCA$ 129.00\nIn stock — ships from Montréal…",
	46	+ page: { title: "Example Product 42 — Example Shop", description: "Buy Example Product 42 for CA$ 129.00.", canonical: "https://example.com/products/42", lang: "en", og: { "og:title": "Example Product 42", "og:type": "product" }, links_count: 38 },
46	47	metadata: {
47	48	network: "residential",
48	49	country: "CA",
	50	+ mode: "http",
49	51	attempts: 1,
50	52	duration_ms: 1184,
51	53	bytes: 48211,
52	54	session: null,
53	55	cached: false,
54	56	timing: { dns_ms: 21, proxy_connect_ms: 0, tls_ms: 0, origin_ms: 934, processing_ms: 12, total_ms: 1184 },
55		− debug: { attempts: [{ provider: "network-a", network: "residential", country: "CA", outcome: "success", status: 200, duration_ms: 1102 }] },
	57	+ debug: { attempts: [{ provider: "network-a", network: "residential", mode: "http", country: "CA", outcome: "success", block_reason: null, status: 200, duration_ms: 1102 }] },
	58	+ },
	59	+};
	60	+
	61	+const ESCALATED = {
	62	+ request_id: "req_7h2k9m4p1q8r5s3t",
	63	+ success: true,
	64	+ status: 200,
	65	+ url: "https://app.example.io/listings",
	66	+ final_url: "https://app.example.io/listings",
	67	+ content: null,
	68	+ content_type: "text/html; charset=utf-8",
	69	+ headers: { "content-type": "text/html; charset=utf-8" },
	70	+ cookies: [{ name: "cf_clearance", value: "…", domain: ".example.io", path: "/" }],
	71	+ markdown: "# Listings\n\n\| Ref \| Price \|\n\| --- \| --- \|\n\| A-1041 \| CA$ 1,250 \|\n…",
	72	+ page: { title: "Listings", description: null, canonical: null, lang: "en", og: {}, links_count: 112 },
	73	+ metadata: {
	74	+ network: "residential",
	75	+ country: "CA",
	76	+ mode: "browser",
	77	+ attempts: 2,
	78	+ duration_ms: 6820,
	79	+ bytes: 391204,
	80	+ session: null,
	81	+ cached: false,
	82	+ timing: { dns_ms: 19, proxy_connect_ms: 0, tls_ms: 0, origin_ms: 5210, processing_ms: 88, total_ms: 6820 },
	83	+ debug: {
	84	+ attempts: [
	85	+ { provider: "network-a", network: "residential", mode: "http", country: "CA", outcome: "blocked", block_reason: "cloudflare_challenge", status: 403, duration_ms: 1204 },
	86	+ { provider: "network-a", network: "residential", mode: "browser", country: "CA", outcome: "success", block_reason: null, status: 200, duration_ms: 5480 },
	87	+ ],
	88	+ },
56	89	},
57	90	};
58	91
		`@@ -69,12 +102,13 @@ const BLOCKED = {`
69	102	metadata: {
70	103	network: "residential",
71	104	country: "US",
72		− attempts: 3,
73		− duration_ms: 6410,
	105	+ mode: "browser",
	106	+ attempts: 4,
	107	+ duration_ms: 11410,
74	108	bytes: 9120,
75	109	session: null,
76	110	cached: false,
77		− timing: { dns_ms: 18, proxy_connect_ms: 0, tls_ms: 0, origin_ms: 1710, processing_ms: 4, total_ms: 6410 },
	111	+ timing: { dns_ms: 18, proxy_connect_ms: 0, tls_ms: 0, origin_ms: 1710, processing_ms: 4, total_ms: 11410 },
78	112	},
79	113	};
80	114
		`@@ -86,7 +120,7 @@ const INVALID = {`
86	120	details: {
87	121	issues: [
88	122	{ path: "timeout", message: "Number must be greater than or equal to 1000" },
89		− { path: "format", message: "Invalid enum value. Expected 'html' \| 'text' \| 'json' \| 'raw', received 'markdown'" },
	123	+ { path: "format", message: "Invalid enum value. Expected 'html' \| 'text' \| 'markdown' \| 'json' \| 'raw', received 'pdf'" },
90	124	],
91	125	},
92	126	},
		`@@ -103,7 +137,7 @@ export default function FetchApiPage() {`
103	137	Download OpenAPI 3.1
104	138	</Link>
105	139	</Button>
106		− <span>Machine-readable description of the public endpoints (fetch, sessions, me, usage).</span>
	140	+ <span>Machine-readable description of the public endpoints (fetch, crawl, map, sessions, me, usage).</span>
107	141	</div>
108	142
109	143	<H2>Request</H2>
		`@@ -115,30 +149,39 @@ export default function FetchApiPage() {`
115	149	rows={[
116	150	{ name: "url", type: "string", required: true, constraints: "1–8,192 chars, http or https", description: <>Absolute URL to fetch. Credentials in the URL, private/internal hosts and non-http schemes are refused with <code>URL_NOT_ALLOWED</code>.</> },
117	151	{ name: "method", type: '"GET" \| "POST" \| "PUT" \| "PATCH" \| "DELETE" \| "HEAD" \| "OPTIONS"', default: '"GET"', description: <>HTTP method sent to the target.</> },
118		− { name: "headers", type: "object<string, string>", constraints: "≤ 64 entries; name ≤ 256, value ≤ 8,192 chars", description: <>Headers forwarded to the target. They override Fetcha's defaults (a desktop Chrome <code>User-Agent</code>, <code>Accept</code>, <code>Accept-Language: en-US,en;q=0.9</code>, <code>Accept-Encoding</code>). Hop-by-hop headers and <code>Host</code> are dropped.</> },
	152	+ { name: "headers", type: "object<string, string>", constraints: "≤ 64 entries; name ≤ 256, value ≤ 8,192 chars", description: <>Headers forwarded to the target. They override Fetcha's defaults: a complete, ordered browser header profile (Chrome, Firefox or Safari, rotated per attempt) with matching <code>User-Agent</code>, <code>Accept</code>, <code>Accept-Language</code>, <code>Accept-Encoding</code> and client hints. Hop-by-hop headers and <code>Host</code> are dropped.</> },
119	153	{ name: "cookies", type: "object<string, string>", constraints: "name ≤ 256, value ≤ 4,096 chars", description: <>Cookies sent with the request. They are serialised into a <code>Cookie</code> header and appended to any <code>Cookie</code> header you also set.</> },
120	154	{ name: "body", type: "string \| object", constraints: "string ≤ 2,000,000 chars", description: <>Request body. Strings are sent verbatim; objects are JSON-serialised. If you do not set <code>Content-Type</code>, it defaults to <code>application/json</code>. Ignored for <code>GET</code> and <code>HEAD</code>.</> },
121		− { name: "timeout", type: "integer (ms)", default: "30000", constraints: "1,000–120,000; silently capped to your plan maximum", description: <>Overall deadline for the whole request, including every retry. Free 30 s, Developer 60 s, Growth 90 s, Business and Enterprise 120 s. When fewer than 500 ms remain, Fetcha stops retrying and returns <code>TARGET_TIMEOUT</code>.</> },
	155	+ { name: "timeout", type: "integer (ms)", default: "30000", constraints: "1,000–120,000", description: <>Overall deadline for the whole request, including every retry and any browser render (maximum 120 s). When fewer than 500 ms remain, Fetcha stops retrying and returns <code>TARGET_TIMEOUT</code>.</> },
122	156	{ name: "country", type: "string", constraints: "exactly 2 chars, upper-cased", description: <>ISO 3166-1 alpha-2 country of the exit IP, e.g. <code>{`"CA"`}</code>. See <Link href="/docs/geolocation" className="text-accent underline underline-offset-2">Geolocation</Link>.</> },
123	157	{ name: "region", type: "string", constraints: "≤ 64 chars", description: <>State or province. US states and Canadian provinces accept two-letter codes (<code>{`"QC"`}</code>, <code>{`"NY"`}</code>) or full names; other values are slugified.</> },
124	158	{ name: "city", type: "string", constraints: "≤ 128 chars", description: <>City name, slugified (<code>{`"Quebec"`}</code> → <code>quebec</code>, <code>{`"New York"`}</code> → <code>new_york</code>).</> },
125	159	{ name: "network", type: '"auto" \| "datacenter" \| "residential" \| "isp" \| "mobile"', default: '"auto"', description: <>Network class. <code>auto</code> lets the engine choose and escalate. Only <code>residential</code> is live today; explicitly requesting another class returns <code>NETWORK_UNAVAILABLE</code>. See <Link href="/docs/networks" className="text-accent underline underline-offset-2">Network selection</Link>.</> },
126	160	{ name: "session", type: "string", constraints: "≤ 64 chars", description: <>Id of a session created with <code>POST /v1/sessions</code> (<code>sess_…</code>). Pins the exit identity, network and default country for this request. Unknown ids return <code>SESSION_NOT_FOUND</code>; expired or closed ones return <code>SESSION_EXPIRED</code>.</> },
127		− { name: "format", type: '"html" \| "text" \| "json" \| "raw"', default: '"html"', description: <>Controls the response. <code>html</code> and <code>raw</code> return the body in <code>content</code>. <code>text</code> returns readable text in <code>text</code> (scripts, styles and tags removed) and sets <code>content</code> to <code>null</code>. <code>json</code> returns the body in <code>content</code> and, when it parses, the parsed value in <code>json</code>.</> },
	161	+ { name: "format", type: '"html" \| "text" \| "markdown" \| "json" \| "raw"', default: '"html"', description: <>Controls the response. <code>html</code> and <code>raw</code> return the body in <code>content</code>. <code>text</code> returns readable text in <code>text</code> (scripts, styles and tags removed) and sets <code>content</code> to <code>null</code>. <code>markdown</code> converts the page to Markdown in <code>markdown</code> (main content first, navigation and boilerplate removed; headings, lists, links, tables and code preserved) and sets <code>content</code> to <code>null</code>. <code>json</code> returns the body in <code>content</code> and, when it parses, the parsed value in <code>json</code>.</> },
128	162	{ name: "follow_redirects", type: "boolean", default: "true", description: <>Follow 301/302/303/307/308 responses. When <code>false</code>, the redirect response is returned as-is (<code>success</code> is <code>true</code> for 3xx).</> },
129	163	{ name: "max_redirects", type: "integer", default: "10", constraints: "0–20", description: <>Maximum hops before <code>TOO_MANY_REDIRECTS</code>.</> },
130	164	{ name: "max_response_bytes", type: "integer", constraints: "1,024–50,000,000", description: <>Lower the response size cap for this request. The platform limit is 20 MB; values above it are reduced to 20 MB. Exceeding the cap returns <code>RESPONSE_TOO_LARGE</code>.</> },
131		− { name: "retries", type: "integer", default: "plan maximum", constraints: "0–5, capped by plan", description: <>Number of additional attempts after the first. Free 2, Developer 3, Growth 4, Business and Enterprise 5. See <Link href="/docs/retries" className="text-accent underline underline-offset-2">Retries</Link>.</> },
	165	+ { name: "retries", type: "integer", default: "5", constraints: "0–5", description: <>Number of additional attempts after the first. Retries use a fresh exit IP and a different header profile, with jittered backoff; a <code>Retry-After</code> header from the target is honoured within the timeout. See <Link href="/docs/retries" className="text-accent underline underline-offset-2">Retries</Link>.</> },
132	166	{ name: "device", type: '"desktop" \| "mobile" \| "tablet"', description: <><code>mobile</code> sets an iPhone Safari <code>User-Agent</code> unless you provide your own. <code>desktop</code> is the default behaviour. <code>tablet</code> is accepted but currently changes nothing.</> },
133	167	{ name: "locale", type: "string", constraints: "≤ 16 chars", description: <>Sets the <code>Accept-Language</code> header, e.g. <code>{`"fr-CA"`}</code>.</> },
134		− { name: "debug", type: "boolean", default: "false", description: <>Adds <code>metadata.debug.attempts</code> with one entry per attempt (route alias, network, country, outcome, status, duration).</> },
135		− { name: "browser", type: "boolean", default: "false", description: <>Managed browser rendering. Not available yet: <code>true</code> returns <code>400 BROWSER_UNAVAILABLE</code> before any network activity.</> },
136		− { name: "javascript", type: "boolean", reserved: true, description: <>Reserved for the managed browser. Accepted, ignored.</> },
137		− { name: "wait_for", type: "string", reserved: true, constraints: "≤ 512 chars", description: <>Reserved for the managed browser (CSS selector to wait for). Accepted, ignored.</> },
138		− { name: "wait_ms", type: "integer", reserved: true, constraints: "0–30,000", description: <>Reserved for the managed browser. Accepted, ignored.</> },
	168	+ { name: "debug", type: "boolean", default: "false", description: <>Adds <code>metadata.debug.attempts</code> with one entry per attempt (route alias, network, mode, country, outcome, block reason, status, duration).</> },
	169	+ { name: "links", type: "boolean", default: "false", description: <>Return every hyperlink of the page in <code>links[]</code> as absolute URLs with anchor text, <code>internal</code> (same registrable domain) and <code>nofollow</code> flags. HTML responses only. <code>page.links_count</code> is always present.</> },
	170	+ { name: "referer", type: '"auto" \| "none" \| url', default: '"auto"', constraints: "url ≤ 2,048 chars", description: <><code>Referer</code> strategy. <code>auto</code> sends none on the first attempt and a search-engine referer on retries; <code>none</code> never sends one; a literal URL is sent as-is.</> },
	171	+ { name: "browser", type: "boolean", default: "false", description: <>Render the page in the managed headless Chromium, routed through the same network class, geography and session as a plain fetch. Returns the DOM after the page settles. See <Link href="/docs/browser" className="text-accent underline underline-offset-2">Browser</Link>.</> },
	172	+ { name: "browser_fallback", type: "boolean", default: "true", description: <>When an HTTP attempt is blocked by a JavaScript challenge or anti-bot page, automatically retry in the browser. Set to <code>false</code> to stay on plain HTTP.</> },
	173	+ { name: "wait_for", type: "string (CSS selector)", constraints: "≤ 512 chars", description: <>Browser: selector that must be present before the DOM is captured.</> },
	174	+ { name: "wait_ms", type: "integer (ms)", constraints: "0–30,000", description: <>Browser: extra settle time after the wait condition (and after <code>wait_for</code> when set).</> },
	175	+ { name: "wait_until", type: '"load" \| "domcontentloaded" \| "networkidle"', default: '"domcontentloaded"', description: <>Browser: navigation event to wait for before applying <code>wait_for</code> / <code>wait_ms</code>.</> },
	176	+ { name: "javascript", type: "boolean", default: "true", description: <>Browser: set to <code>false</code> to render with scripting disabled.</> },
	177	+ { name: "block_resources", type: "boolean", default: "true", description: <>Browser: skip images, fonts and media to save bandwidth and time. Page scripts and XHR still run.</> },
	178	+ { name: "screenshot", type: "boolean", default: "false", description: <>Browser: return a PNG of the viewport, base64-encoded, in <code>screenshot</code>.</> },
139	179	{ name: "cache", type: "{ enabled?: boolean, ttl?: integer }", reserved: true, constraints: "ttl 1–86,400 s", description: <>Reserved for response caching. Accepted, ignored; <code>metadata.cached</code> is always <code>false</code> today.</> },
140	180	]}
141	181	/>
	182	+ <Callout variant="info" title="Browser fields outside browser mode">
	183	+ <Code>wait_for</Code>, <Code>wait_ms</Code>, <Code>wait_until</Code>, <Code>javascript</Code>, <Code>block_resources</Code> and <Code>screenshot</Code> only take effect when a render happens: either because <Code>browser: true</Code>, or because an HTTP attempt was blocked and <Code>browser_fallback</Code> escalated to the browser. On a plain HTTP response they are accepted and have no effect.
	184	+ </Callout>
142	185
143	186	<H3>Example</H3>
144	187	<P>Fetch a product page as readable text through a Canadian residential exit in Québec, with a 20-second budget and debug metadata.</P>
		`@@ -172,10 +215,20 @@ export default function FetchApiPage() {`
172	215	{ name: "headers", type: "object<string, string>", description: <>Response headers from the target with lower-cased names. Multiple values are joined with <code>{`", "`}</code>. <code>content-encoding</code> is removed because Fetcha decompresses gzip, deflate, brotli and zstd bodies for you.</> },
173	216	{ name: "cookies", type: "array", description: <>Cookies parsed from <code>Set-Cookie</code>: <code>{`{ name, value, domain?, path? }`}</code>. Replay them via the <code>cookies</code> request field.</> },
174	217	{ name: "text", type: "string \| null", description: <>Present only for <code>{`format: "text"`}</code>. Readable text extracted from HTML; <code>null</code> when the body was binary.</> },
	218	+ { name: "markdown", type: "string \| null", description: <>Present only for <code>{`format: "markdown"`}</code>. The page converted to Markdown, main content first with navigation, footers, cookie banners and scripts removed; <code>null</code> when the body was not HTML.</> },
175	219	{ name: "json", type: "any", description: <>Present only for <code>{`format: "json"`}</code> and only when the body parsed as JSON. Absent otherwise; check <code>content</code> in that case.</> },
	220	+ { name: "page", type: "object \| null", description: <>Parsed page metadata for HTML responses: <code>{`{ title, description, canonical, lang, og, links_count }`}</code>. <code>og</code> maps Open Graph property names to values. <code>null</code> for non-HTML bodies.</> },
	221	+ { name: "links", type: "array", description: <>Present only with <code>links: true</code> on an HTML response. Items are <code>{`{ url, text, internal, nofollow }`}</code> with absolute URLs, de-duplicated, in document order.</> },
	222	+ { name: "screenshot", type: "string", description: <>Present only when the page was rendered in the browser with <code>screenshot: true</code>. PNG, base64-encoded.</> },
176	223	{ name: "metadata", type: "object", description: <>Routing information, see below.</> },
177	224	]}
178	225	/>
	226	+ <H3>Escalated to the browser</H3>
	227	+ <P>
	228	+ The same document is returned when Fetcha had to render the page. <Code>metadata.mode</Code> tells you how the final attempt was made, and with <Code>debug: true</Code> each attempt shows
	229	+ its <Code>mode</Code> and, when blocked, the <Code>block_reason</Code> that triggered the escalation.
	230	+ </P>
	231	+ <ResponseExample status={200} title="200 OK · format: markdown · escalated to the browser" body={ESCALATED} />
179	232
180	233	<H3>metadata</H3>
181	234	<ParamTable
		`@@ -183,13 +236,14 @@ export default function FetchApiPage() {`
183	236	rows={[
184	237	{ name: "network", type: '"datacenter" \| "residential" \| "isp" \| "mobile"', description: <>Concrete network class that served the final attempt. Never <code>auto</code>.</> },
185	238	{ name: "country", type: "string \| null", description: <>Country targeted by the final attempt (the request's <code>country</code>, or the session's). <code>null</code> when no geography was requested.</> },
	239	+ { name: "mode", type: '"http" \| "browser"', description: <><code>http</code> for a plain fetch, <code>browser</code> when the final attempt was rendered in the managed browser (requested or escalated).</> },
186	240	{ name: "attempts", type: "integer", description: <>Number of attempts made, including the successful one. Greater than 1 means Fetcha retried or escalated.</> },
187	241	{ name: "duration_ms", type: "integer", description: <>Wall-clock time of the whole request inside Fetcha, all attempts included.</> },
188	242	{ name: "bytes", type: "integer", description: <>Bytes transferred across <em>all</em> attempts (request and response). This is the quantity used for bandwidth pricing on premium networks.</> },
189	243	{ name: "session", type: "string \| null", description: <>Session id used, if any.</> },
190	244	{ name: "cached", type: "boolean", description: <>Always <code>false</code> today (caching is reserved).</> },
191	245	{ name: "timing", type: "object", description: <>Breakdown of the final attempt, see below.</> },
192		− { name: "debug", type: "object", description: <>Only with <code>debug: true</code>. <code>attempts[]</code> of <code>{`{ provider, network, country, outcome, status, duration_ms }`}</code>. <code>provider</code> is a neutral route alias (<code>network-a</code>, <code>network-b</code>, …); <code>outcome</code> is one of <code>success</code>, <code>blocked</code>, <code>timeout</code>, <code>error</code>, <code>provider_error</code>, <code>too_large</code>.</> },
	246	+ { name: "debug", type: "object", description: <>Only with <code>debug: true</code>. <code>attempts[]</code> of <code>{`{ provider, network, mode, country, outcome, block_reason, status, duration_ms }`}</code>. <code>provider</code> is a neutral route alias (<code>network-a</code>, <code>network-b</code>, …); <code>mode</code> is <code>http</code> or <code>browser</code>; <code>outcome</code> is one of <code>success</code>, <code>blocked</code>, <code>timeout</code>, <code>error</code>, <code>provider_error</code>, <code>too_large</code>; <code>block_reason</code> names the detector that classified a blocked attempt (for example <code>cloudflare_challenge</code>, <code>datadome</code>, <code>captcha</code>, <code>soft_block</code>) and is <code>null</code> otherwise.</> },
193	247	]}
194	248	/>
195	249
		`@@ -238,9 +292,15 @@ export default function FetchApiPage() {`
238	292
239	293	<H2>Blocked targets</H2>
240	294	<P>
241		− A block is any response Fetcha classifies as anti-bot interference: HTTP 403, 407, 429 or 999, a 503 challenge page, captcha markup, known anti-bot markers in the first 20 KB, or a WAF
242		− server signature on a 4xx/5xx. Blocks trigger escalation. If <Strong>every</Strong> attempt is blocked, Fetcha still returns <Code>200</Code> with the last blocked page so you can inspect
243		− it, but with <Code>success: false</Code>. The request log records it as <Code>TARGET_BLOCKED</Code>.
	295	+ A block is any response Fetcha classifies as anti-bot interference. Detection covers HTTP 403, 407, 429 and 999, 503 challenge pages, captcha markup and WAF signatures on 4xx/5xx, plus
	296	+ vendor-specific fingerprints in headers, cookies and the first part of the body: Cloudflare challenges and Turnstile, DataDome, PerimeterX / HUMAN, Akamai Bot Manager, Kasada, Imperva /
	297	+ Incapsula, AWS WAF challenges and Vercel attack-mode pages. <Strong>Soft blocks</Strong> are detected too: a 200 whose body is a challenge, an interstitial, an empty shell or a
	298	+ “verify you are human” page rather than the real content.
	299	+ </P>
	300	+ <P>
	301	+ Blocks trigger escalation: a new exit IP and header profile, then a premium route, and, when the block is a JavaScript challenge and <Code>browser_fallback</Code> is on (the default), a
	302	+ render in the managed browser, which solves most challenge pages. If <Strong>every</Strong> attempt is blocked, Fetcha still returns <Code>200</Code> with the last blocked page so you can
	303	+ inspect it, but with <Code>success: false</Code>. The request log records it as <Code>TARGET_BLOCKED</Code>, and <Code>debug.attempts[].block_reason</Code> tells you what was detected.
244	304	</P>
245	305	<ResponseExample status={200} title="200 OK · success: false" body={BLOCKED} />
246	306	<Callout variant="info" title="Non-blocked errors are returned immediately">
		`@@ -318,10 +378,14 @@ if (r.content_type?.startsWith("application/pdf")) {`
318	378	</Tr>
319	379	<Tr>
320	380	<Td>
321		− <Code>browser: true</Code>
	381	+ <Code>browser: true</Code> while the browser pool is disabled or down
322	382	</Td>
323	383	<Td mono>400 BROWSER_UNAVAILABLE</Td>
324	384	</Tr>
	385	+ <Tr>
	386	+ <Td>Page did not settle in the browser before the timeout</Td>
	387	+ <Td mono>504 BROWSER_TIMEOUT</Td>
	388	+ </Tr>
325	389	<Tr>
326	390	<Td>Timeout exhausted across all attempts</Td>
327	391	<Td mono>504 TARGET_TIMEOUT</Td>

modified apps/web/src/app/(marketing)/docs/networks/page.tsx +30 −33

		`@@ -1,5 +1,5 @@`
1	1	import type { Metadata } from "next";
2		−import { PLAN_LIMITS, PLANS } from "@fetcha/core";
	2	+import { PLAN_LIMITS } from "@fetcha/core";
3	3	import { CodeBlock } from "@/components/ui/code-block";
4	4	import { Badge } from "@/components/ui/badge";
5	5	import { DocPage } from "@/components/docs/doc-page";
		`@@ -11,18 +11,19 @@ import { fetchTabs } from "@/components/docs/snippets";`
11	11
12	12	export const metadata: Metadata = {
13	13	title: "Network Selection",
14		− description: "Network classes, what auto does, how routes are scored, per-domain intelligence, escalation, circuit breakers and plan gating.",
	14	+ description: "Network classes, what auto does, how routes are scored, per-domain intelligence, escalation, circuit breakers and which classes are available (all of them).",
15	15	};
16	16
17	17	const CLASSES: Array<{ name: string; live: boolean; desc: string }> = [
18	18	{ name: "auto", live: true, desc: "Let the engine choose. Cheapest class first, escalating to more reliable classes on blocks. Resolves to residential today." },
19		− { name: "residential", live: true, desc: "Exit IPs assigned by consumer ISPs to households. Highest acceptance on sites that filter datacenter traffic. Geo-targetable. Billed per GB on paid plans." },
20		− { name: "datacenter", live: false, desc: "Exit IPs from hosting providers. Fastest and cheapest, most often blocked. Not live yet." },
21		− { name: "isp", live: false, desc: "Static IPs registered to consumer ISPs but hosted in datacenters. Residential reputation with datacenter speed. Not live yet." },
22		− { name: "mobile", live: false, desc: "Exit IPs from cellular carriers. Highest acceptance, highest cost. Not live yet." },
	19	+ { name: "residential", live: true, desc: "Exit IPs assigned by consumer ISPs to households. Highest acceptance on sites that filter datacenter traffic. Geo-targetable." },
	20	+ { name: "datacenter", live: false, desc: "Exit IPs from hosting providers. Fastest and cheapest, most often blocked. Included, no live route yet." },
	21	+ { name: "isp", live: false, desc: "Static IPs registered to consumer ISPs but hosted in datacenters. Residential reputation with datacenter speed. Included, no live route yet." },
	22	+ { name: "mobile", live: false, desc: "Exit IPs from cellular carriers. Highest acceptance, highest cost. Included, no live route yet." },
23	23	];
24	24
25	25	const ALL_NETWORKS = ["datacenter", "residential", "isp", "mobile"] as const;
	26	+const L = PLAN_LIMITS.unlimited;
26	27
27	28	export default function NetworksPage() {
28	29	return (
		`@@ -56,8 +57,8 @@ export default function NetworksPage() {`
56	57	</Table>
57	58	<Callout variant="warning" title="What happens today">
58	59	Only the <Code>residential</Code> class has live capacity. <Code>auto</Code> therefore resolves to residential and <Code>metadata.network</Code> reports <Code>{`"residential"`}</Code>. Requesting{" "}
59		− <Code>datacenter</Code>, <Code>isp</Code> or <Code>mobile</Code> explicitly returns <Code>400 NETWORK_UNAVAILABLE</Code>: either because the class is not part of your plan, or because it has no
60		− live route yet. When those classes launch, <Code>auto</Code> will start using them without any change on your side.
	60	+ <Code>datacenter</Code>, <Code>isp</Code> or <Code>mobile</Code> explicitly returns <Code>400 NETWORK_UNAVAILABLE</Code> because the class has no live route yet — never because of your plan: all
	61	+ classes are included for every organization. When those classes launch, <Code>auto</Code> will start using them without any change on your side.
61	62	</Callout>
62	63
63	64	<H2>What auto does</H2>
		`@@ -67,8 +68,8 @@ export default function NetworksPage() {`
67	68	</P>
68	69	<H3>1. Eligible classes</H3>
69	70	<P>
70		− The engine starts from the escalation order <Code>datacenter → isp → residential → mobile</Code>, keeps only the classes included in your plan, and drops classes that have no configured,
71		− healthy route right now. If the domain has an admin-pinned <Code>force_network</Code> policy, only that class is considered.
	71	+ The engine starts from the escalation order <Code>datacenter → isp → residential → mobile</Code> (all four classes are available to every organization) and drops classes that have no
	72	+ configured, healthy route right now. If the domain has an admin-pinned <Code>force_network</Code> policy, only that class is considered.
72	73	</P>
73	74	<H3>2. Scoring</H3>
74	75	<P>Every eligible route receives a score between 0 and 1, computed as a weighted sum of six signals:</P>
		`@@ -127,7 +128,7 @@ export default function NetworksPage() {`
127	128	the domain. Residential is never skipped by this rule.
128	129	</Li>
129	130	<Li>
130		− The list is cut to your attempt budget: <Code>retries + 1</Code>, bounded by the plan maximum. See <A href="/docs/retries">Retries</A>.
	131	+ The list is cut to your attempt budget: <Code>retries + 1</Code>, at most {L.max_retries + 1}. See <A href="/docs/retries">Retries</A>.
131	132	</Li>
132	133	<Li>
133	134	With an explicit class (for example <Code>{`"network": "residential"`}</Code>), candidates are simply sorted by score; there is no cross-class escalation.
		`@@ -158,40 +159,36 @@ export default function NetworksPage() {`
158	159	<Code>400 NETWORK_UNAVAILABLE</Code>. Neither counts as a target failure and both are safe to retry after a short pause.
159	160	</P>
160	161
161		− <H2>Plan gating</H2>
	162	+ <H2>Availability</H2>
162	163	<P>
163		− Each plan includes a set of network classes. Requesting a class outside your plan is refused before any attempt. The table reflects the plan definitions; remember that only residential is
164		− live regardless of plan.
	164	+ Fetcha has a single plan and no network class is gated: every organization may request any class. The only reason an explicit class is refused is the absence of a live route for it. The table
	165	+ reflects the platform configuration; remember that only residential is live today.
165	166	</P>
166	167	<Table>
167	168	<THead>
168	169	<Tr>
169		− <Th>Plan</Th>
170		− {ALL_NETWORKS.map((n) => (
171		− <Th key={n} className="text-center">
172		− {n}
173		− </Th>
174		− ))}
	170	+ <Th>Class</Th>
	171	+ <Th className="text-center">Included</Th>
	172	+ <Th className="text-center">Live route</Th>
175	173	</Tr>
176	174	</THead>
177	175	<TBody>
178		− {PLANS.map((p) => (
179		− <Tr key={p}>
180		− <Td className="font-medium text-fg">{PLAN_LIMITS[p].label}</Td>
181		− {ALL_NETWORKS.map((n) => (
182		− <Td key={n} className="text-center">
183		− {PLAN_LIMITS[p].networks.includes(n) ? <span className="text-success">Included</span> : <span className="text-fg-subtle">—</span>}
184		− </Td>
185		− ))}
186		− </Tr>
187		− ))}
	176	+ {ALL_NETWORKS.map((n) => {
	177	+ const live = CLASSES.find((c) => c.name === n)?.live ?? false;
	178	+ return (
	179	+ <Tr key={n}>
	180	+ <Td mono>{n}</Td>
	181	+ <Td className="text-center">{L.networks.includes(n) ? <span className="text-success">Yes</span> : <span className="text-fg-subtle">—</span>}</Td>
	182	+ <Td className="text-center">{live ? <span className="text-success">Yes</span> : <span className="text-fg-subtle">Not yet</span>}</Td>
	183	+ </Tr>
	184	+ );
	185	+ })}
188	186	</TBody>
189	187	</Table>
190	188
191	189	<H2>Choosing a class</H2>
192	190	<P>
193		− Prefer <Code>auto</Code>. Pin <Code>residential</Code> when you need to guarantee that no cheaper class is ever tried on a sensitive target, or when you want bandwidth cost to be fully
194		− predictable. Read the served class from <Code>metadata.network</Code> and, with <Code>debug: true</Code>, the per-attempt route aliases from <Code>metadata.debug.attempts</Code>.
	191	+ Prefer <Code>auto</Code>. Pin <Code>residential</Code> when you need to guarantee that no cheaper class is ever tried on a sensitive target. Read the served class from <Code>metadata.network</Code> and, with <Code>debug: true</Code>, the per-attempt route aliases from <Code>metadata.debug.attempts</Code>.
195	192	</P>
196	193	<CodeTabs
197	194	tabs={fetchTabs(
		`@@ -207,7 +204,7 @@ export default function NetworksPage() {`
207	204	status={400}
208	205	statusText="Bad Request"
209	206	title="400 Bad Request · explicit class not available"
210		− body={{ error: { code: "NETWORK_UNAVAILABLE", message: 'The "mobile" network is not included in the Developer plan.', request_id: "req_0a1b2c3d4e5f6g7h" } }}
	207	+ body={{ error: { code: "NETWORK_UNAVAILABLE", message: 'The "mobile" network has no live route right now.', request_id: "req_0a1b2c3d4e5f6g7h" } }}
211	208	/>
212	209	<CodeBlock
213	210	lang="json"

modified apps/web/src/app/(marketing)/docs/openapi.json/route.ts +277 −20

		`@@ -1,8 +1,9 @@`
1		−import { ERROR_CODES, ERROR_HTTP_STATUS, ERROR_MESSAGES, HTTP_METHODS, NETWORK_CLASSES, OUTPUT_FORMATS, DEVICES, PLAN_LIMITS } from "@fetcha/core";
	1	+import { CRAWL_FORMATS, CRAWL_STATUSES, ERROR_CODES, ERROR_HTTP_STATUS, ERROR_MESSAGES, HTTP_METHODS, NETWORK_CLASSES, OUTPUT_FORMATS, DEVICES, PLAN_LIMITS } from "@fetcha/core";
2	2
3	3	/**
4	4	* OpenAPI 3.1 description of the public Fetcha API, derived by hand from `fetchRequestSchema`,
5		− * `sessionCreateSchema`, the error catalogue and the route handlers. Served at /docs/openapi.json.
	5	+ * `sessionCreateSchema`, `crawlCreateSchema`, `mapCreateSchema`, the error catalogue and the route
	6	+ * handlers. Served at /docs/openapi.json.
6	7	*/
7	8
8	9	const BASE_URL = "https://www.fetcha.co";
		`@@ -65,17 +66,23 @@ const fetchRequest = {`
65	66	headers: { type: "object", additionalProperties: { type: "string", maxLength: 8192 }, maxProperties: 64, description: "Headers forwarded to the target; override Fetcha defaults." },
66	67	cookies: { type: "object", additionalProperties: { type: "string", maxLength: 4096 }, description: "Cookies serialised into the Cookie header." },
67	68	body: { oneOf: [{ type: "string", maxLength: 2_000_000 }, { type: "object", additionalProperties: true }], description: "Request body. Objects are JSON-serialised. Content-Type defaults to application/json. Ignored for GET/HEAD." },
68		− timeout: { type: "integer", minimum: 1000, maximum: 120_000, default: 30_000, description: "Overall deadline in ms for all attempts; capped to the plan maximum (Free 30 s, Developer 60 s, Growth 90 s, Business/Enterprise 120 s)." },
	69	+ timeout: { type: "integer", minimum: 1000, maximum: 120_000, default: 30_000, description: "Overall deadline in ms for all attempts, including any browser render (max 120 s)." },
69	70	...geoProps,
70	71	network: { type: "string", enum: [...NETWORK_CLASSES], default: "auto", description: "Network class. Only `residential` is live; `auto` resolves to it. Other explicit classes return NETWORK_UNAVAILABLE." },
71	72	session: { type: "string", maxLength: 64, description: "Session id (sess_…) from POST /v1/sessions." },
72		− browser: { type: "boolean", default: false, description: "Managed browser. Not available yet: true returns BROWSER_UNAVAILABLE." },
73		− javascript: { type: "boolean", description: "Reserved for the managed browser. Accepted, ignored." },
74		− wait_for: { type: "string", maxLength: 512, description: "Reserved for the managed browser. Accepted, ignored." },
75		− wait_ms: { type: "integer", minimum: 0, maximum: 30_000, description: "Reserved for the managed browser. Accepted, ignored." },
76		− device: { type: "string", enum: [...DEVICES], description: "`mobile` sets an iPhone User-Agent unless one is provided. `tablet` currently has no effect." },
	73	+ browser: { type: "boolean", default: false, description: "Render the page in the managed headless browser routed through the same network, country and session." },
	74	+ browser_fallback: { type: "boolean", default: true, description: "When an HTTP attempt is blocked by a JavaScript challenge / anti-bot page, automatically retry in the browser." },
	75	+ javascript: { type: "boolean", default: true, description: "Browser: set false to render with scripting disabled." },
	76	+ wait_for: { type: "string", maxLength: 512, description: "Browser: CSS selector that must be present before capture." },
	77	+ wait_ms: { type: "integer", minimum: 0, maximum: 30_000, description: "Browser: extra settle time in ms after the wait condition." },
	78	+ wait_until: { type: "string", enum: ["load", "domcontentloaded", "networkidle"], default: "domcontentloaded", description: "Browser: navigation event to wait for." },
	79	+ block_resources: { type: "boolean", default: true, description: "Browser: skip images, fonts and media." },
	80	+ screenshot: { type: "boolean", default: false, description: "Browser: return a PNG screenshot (base64) in `screenshot`." },
	81	+ links: { type: "boolean", default: false, description: "Return every hyperlink of the page in `links[]` (absolute URLs)." },
	82	+ referer: { oneOf: [{ type: "string", enum: ["auto", "none"] }, { type: "string", format: "uri", maxLength: 2048 }], default: "auto", description: "Referer strategy: auto (none first, search-engine referer on retries), none, or a literal URL." },
	83	+ device: { type: "string", enum: [...DEVICES], description: "`mobile` sets an iPhone User-Agent (and viewport in the browser) unless one is provided. `tablet` currently has no effect." },
77	84	locale: { type: "string", maxLength: 16, description: "Sets the Accept-Language header.", example: "fr-CA" },
78		− format: { type: "string", enum: [...OUTPUT_FORMATS], default: "html", description: "html/raw: body in `content`; text: readable text in `text` (content null); json: body in `content` and parsed value in `json`." },
	85	+ format: { type: "string", enum: [...OUTPUT_FORMATS], default: "html", description: "html/raw: body in `content`; text: readable text in `text` (content null); markdown: Markdown in `markdown` (content null); json: body in `content` and parsed value in `json`." },
79	86	follow_redirects: { type: "boolean", default: true },
80	87	max_redirects: { type: "integer", minimum: 0, maximum: 20, default: 10 },
81	88	max_response_bytes: { type: "integer", minimum: 1024, maximum: 50_000_000, description: "Lower the response size cap for this request. Effective cap is min(value, 20 MB)." },
		`@@ -85,11 +92,36 @@ const fetchRequest = {`
85	92	properties: { enabled: { type: "boolean", default: false }, ttl: { type: "integer", minimum: 1, maximum: 86_400, default: 300 } },
86	93	description: "Reserved. Accepted, ignored; metadata.cached is always false.",
87	94	},
88		− retries: { type: "integer", minimum: 0, maximum: 5, description: "Additional attempts after the first. Defaults to and is capped by the plan maximum (Free 2, Developer 3, Growth 4, Business/Enterprise 5)." },
	95	+ retries: { type: "integer", minimum: 0, maximum: 5, default: 5, description: "Additional attempts after the first (max 5)." },
89	96	debug: { type: "boolean", default: false, description: "Include metadata.debug.attempts." },
90	97	},
91	98	} as const;
92	99
	100	+const pageMetadata = {
	101	+ type: ["object", "null"],
	102	+ required: ["title", "description", "canonical", "lang", "og", "links_count"],
	103	+ properties: {
	104	+ title: { type: ["string", "null"] },
	105	+ description: { type: ["string", "null"] },
	106	+ canonical: { type: ["string", "null"] },
	107	+ lang: { type: ["string", "null"] },
	108	+ og: { type: "object", additionalProperties: { type: "string" }, description: "Open Graph properties." },
	109	+ links_count: { type: "integer" },
	110	+ },
	111	+ description: "Parsed page metadata (HTML responses only; null otherwise).",
	112	+} as const;
	113	+
	114	+const pageLink = {
	115	+ type: "object",
	116	+ required: ["url", "text", "internal", "nofollow"],
	117	+ properties: {
	118	+ url: { type: "string", format: "uri" },
	119	+ text: { type: "string" },
	120	+ internal: { type: "boolean", description: "Same registrable domain as the page." },
	121	+ nofollow: { type: "boolean" },
	122	+ },
	123	+} as const;
	124	+
93	125	const timing = {
94	126	type: "object",
95	127	required: ["dns_ms", "proxy_connect_ms", "tls_ms", "origin_ms", "processing_ms", "total_ms"],
		`@@ -120,13 +152,18 @@ const fetchResponse = {`
120	152	items: { type: "object", required: ["name", "value"], properties: { name: { type: "string" }, value: { type: "string" }, domain: { type: "string" }, path: { type: "string" } } },
121	153	},
122	154	text: { type: ["string", "null"], description: "Only for format=text." },
	155	+ markdown: { type: ["string", "null"], description: "Only for format=markdown. Main content first, boilerplate removed." },
123	156	json: { description: "Only for format=json when the body parsed as JSON." },
	157	+ page: { $ref: "#/components/schemas/PageMetadata" },
	158	+ links: { type: "array", items: { $ref: "#/components/schemas/PageLink" }, description: "Only with links=true on an HTML response." },
	159	+ screenshot: { type: "string", description: "Only with browser rendering and screenshot=true. PNG, base64." },
124	160	metadata: {
125	161	type: "object",
126		− required: ["network", "country", "attempts", "duration_ms", "bytes", "session", "cached"],
	162	+ required: ["network", "country", "mode", "attempts", "duration_ms", "bytes", "session", "cached"],
127	163	properties: {
128	164	network: { type: "string", enum: ["datacenter", "residential", "isp", "mobile"] },
129	165	country: { type: ["string", "null"] },
	166	+ mode: { type: "string", enum: ["http", "browser"], description: "How the final attempt was made." },
130	167	attempts: { type: "integer", minimum: 1 },
131	168	duration_ms: { type: "integer" },
132	169	bytes: { type: "integer", description: "Bytes transferred across all attempts." },
		`@@ -143,8 +180,10 @@ const fetchResponse = {`
143	180	properties: {
144	181	provider: { type: "string", description: "Neutral route alias (network-a, network-b, …)." },
145	182	network: { type: "string" },
	183	+ mode: { type: "string", enum: ["http", "browser"] },
146	184	country: { type: ["string", "null"] },
147	185	outcome: { type: "string", enum: ["success", "blocked", "timeout", "error", "provider_error", "too_large"] },
	186	+ block_reason: { type: ["string", "null"], description: "Detector that classified a blocked attempt (cloudflare_challenge, datadome, captcha, soft_block, …)." },
148	187	status: { type: ["integer", "null"] },
149	188	duration_ms: { type: "integer" },
150	189	},
		`@@ -186,22 +225,149 @@ const session = {`
186	225	},
187	226	} as const;
188	227
189		−const plans = Object.values(PLAN_LIMITS).map((p) => `${p.label}: ${p.concurrency} concurrent, ${p.monthly_requests >= Number.MAX_SAFE_INTEGER ? "unlimited" : p.monthly_requests.toLocaleString("en-US")} requests/month, ${p.max_timeout_ms / 1000} s max timeout, ${p.max_retries} max retries`);
	228	+const crawlCreate = {
	229	+ type: "object",
	230	+ additionalProperties: false,
	231	+ required: ["url"],
	232	+ properties: {
	233	+ url: { type: "string", minLength: 1, maxLength: 8192, format: "uri", description: "Seed URL (same URL policy as fetch)." },
	234	+ max_pages: { type: "integer", minimum: 1, maximum: 5000, default: 25, description: "Maximum pages to fetch (capped at 2,000)." },
	235	+ max_depth: { type: "integer", minimum: 0, maximum: 10, default: 2 },
	236	+ same_domain: { type: "boolean", default: true },
	237	+ allow_subdomains: { type: "boolean", default: false },
	238	+ include_patterns: { type: "array", maxItems: 50, items: { type: "string", maxLength: 512 }, description: "Glob with * or /regex/." },
	239	+ exclude_patterns: { type: "array", maxItems: 50, items: { type: "string", maxLength: 512 } },
	240	+ respect_robots: { type: "boolean", default: true },
	241	+ use_sitemap: { type: "boolean", default: false },
	242	+ concurrency: { type: "integer", minimum: 1, maximum: 10, default: 3 },
	243	+ delay_ms: { type: "integer", minimum: 0, maximum: 30_000, default: 0 },
	244	+ timeout: { type: "integer", minimum: 1000, maximum: 120_000, default: 30_000, description: "Per-page timeout." },
	245	+ format: { type: "string", enum: [...CRAWL_FORMATS], default: "markdown" },
	246	+ main_content: { type: "boolean", default: true },
	247	+ country: geoProps.country,
	248	+ network: { type: "string", enum: [...NETWORK_CLASSES], default: "auto" },
	249	+ browser: { type: "boolean", default: false },
	250	+ browser_fallback: { type: "boolean", default: true },
	251	+ headers: { type: "object", additionalProperties: { type: "string", maxLength: 8192 }, maxProperties: 64 },
	252	+ webhook_url: { type: "string", format: "uri", maxLength: 2048, description: "Called once when the job finishes (delivery not yet active)." },
	253	+ label: { type: "string", maxLength: 128 },
	254	+ },
	255	+} as const;
	256	+
	257	+const crawlStats = {
	258	+ type: "object",
	259	+ required: ["discovered", "fetched", "ok", "blocked", "failed", "bytes"],
	260	+ properties: {
	261	+ discovered: { type: "integer" },
	262	+ fetched: { type: "integer" },
	263	+ ok: { type: "integer" },
	264	+ blocked: { type: "integer" },
	265	+ failed: { type: "integer" },
	266	+ bytes: { type: "integer" },
	267	+ },
	268	+} as const;
	269	+
	270	+const crawlJob = {
	271	+ type: "object",
	272	+ required: ["id", "status", "label", "seed_url", "domain", "options", "stats", "error", "created_at", "started_at", "completed_at"],
	273	+ properties: {
	274	+ id: { type: "string", example: "crawl_3k9d0f2a8b1c7e4m" },
	275	+ status: { type: "string", enum: [...CRAWL_STATUSES] },
	276	+ label: { type: ["string", "null"] },
	277	+ seed_url: { type: "string" },
	278	+ domain: { type: "string" },
	279	+ options: { $ref: "#/components/schemas/CrawlCreate" },
	280	+ stats: { $ref: "#/components/schemas/CrawlStats" },
	281	+ error: { type: ["object", "null"], properties: { code: { type: "string" }, message: { type: "string" } } },
	282	+ created_at: { type: "string", format: "date-time" },
	283	+ started_at: { type: ["string", "null"], format: "date-time" },
	284	+ completed_at: { type: ["string", "null"], format: "date-time" },
	285	+ },
	286	+} as const;
	287	+
	288	+const crawlCreated = {
	289	+ type: "object",
	290	+ required: ["id", "status", "seed_url", "created_at", "options"],
	291	+ properties: {
	292	+ id: { type: "string", example: "crawl_3k9d0f2a8b1c7e4m" },
	293	+ status: { type: "string", const: "queued" },
	294	+ seed_url: { type: "string" },
	295	+ created_at: { type: "string", format: "date-time" },
	296	+ options: { $ref: "#/components/schemas/CrawlCreate" },
	297	+ },
	298	+} as const;
	299	+
	300	+const crawlPage = {
	301	+ type: "object",
	302	+ required: ["id", "url", "final_url", "depth", "status", "http_status", "error_code", "title", "description", "content_type", "content", "links_count", "bytes", "duration_ms", "mode", "fetched_at"],
	303	+ properties: {
	304	+ id: { type: "string" },
	305	+ url: { type: "string" },
	306	+ final_url: { type: ["string", "null"] },
	307	+ depth: { type: "integer" },
	308	+ status: { type: "string", enum: ["success", "blocked", "failed"] },
	309	+ http_status: { type: ["integer", "null"] },
	310	+ error_code: { type: ["string", "null"] },
	311	+ title: { type: ["string", "null"] },
	312	+ description: { type: ["string", "null"] },
	313	+ content_type: { type: ["string", "null"] },
	314	+ content: { type: ["string", "null"], description: "Page content in the job's format." },
	315	+ links_count: { type: ["integer", "null"] },
	316	+ bytes: { type: ["integer", "null"] },
	317	+ duration_ms: { type: ["integer", "null"] },
	318	+ mode: { type: ["string", "null"], enum: ["http", "browser", null] },
	319	+ fetched_at: { type: ["string", "null"], format: "date-time" },
	320	+ },
	321	+} as const;
	322	+
	323	+const mapCreate = {
	324	+ type: "object",
	325	+ additionalProperties: false,
	326	+ required: ["url"],
	327	+ properties: {
	328	+ url: { type: "string", minLength: 1, maxLength: 8192, format: "uri" },
	329	+ limit: { type: "integer", minimum: 1, maximum: 10_000, default: 1000 },
	330	+ use_sitemap: { type: "boolean", default: true },
	331	+ use_links: { type: "boolean", default: true },
	332	+ same_domain: { type: "boolean", default: true },
	333	+ allow_subdomains: { type: "boolean", default: false },
	334	+ search: { type: "string", maxLength: 256, description: "Substring, glob (*) or /regex/ filter." },
	335	+ country: geoProps.country,
	336	+ network: { type: "string", enum: [...NETWORK_CLASSES], default: "auto" },
	337	+ timeout: { type: "integer", minimum: 1000, maximum: 120_000, default: 30_000 },
	338	+ },
	339	+} as const;
	340	+
	341	+const mapResult = {
	342	+ type: "object",
	343	+ required: ["url", "count", "urls", "sources", "truncated"],
	344	+ properties: {
	345	+ url: { type: "string" },
	346	+ count: { type: "integer" },
	347	+ urls: { type: "array", items: { type: "string", format: "uri" } },
	348	+ sources: { type: "object", required: ["sitemap", "links"], properties: { sitemap: { type: "integer" }, links: { type: "integer" } } },
	349	+ truncated: { type: "boolean" },
	350	+ },
	351	+} as const;
	352	+
	353	+const CRAWL_ERRORS = [...AUTH_ERRORS, "INVALID_REQUEST", "URL_NOT_ALLOWED", "CRAWL_LIMIT_REACHED", "RATE_LIMITED", "INTERNAL_ERROR"];
	354	+
	355	+const plans = Object.values(PLAN_LIMITS).map((p) => `${p.label}: ${p.concurrency} concurrent, ${p.monthly_requests >= Number.MAX_SAFE_INTEGER ? "unlimited" : p.monthly_requests.toLocaleString("en-US")} requests/month, ${p.max_timeout_ms / 1000} s max timeout, ${p.max_retries} max retries, ${p.browser_concurrency} concurrent browser renders, ${p.crawl_max_pages.toLocaleString("en-US")} pages per crawl job, ${p.crawl_concurrent_jobs} concurrent crawl jobs`);
190	356
191	357	const document = {
192	358	openapi: "3.1.0",
193	359	info: {
194	360	title: "Fetcha API",
195		− version: "0.1.0",
	361	+ version: "0.2.0",
196	362	summary: "Intelligent Web Access Infrastructure",
197	363	description: [
198		− "One API to fetch web pages through Fetcha's routing engine. Authenticate with `Authorization: Bearer fch_live_…` (or `X-API-Key`).",
	364	+ "One API to fetch, render and crawl web pages through Fetcha's routing engine. Authenticate with `Authorization: Bearer fch_live_…` (or `X-API-Key`).",
199	365	"",
200	366	"Every response carries `X-Fetcha-Request-ID`. Errors use a single envelope `{ error: { code, message, request_id, details? } }`.",
201	367	"",
202		− "Plans: " + plans.join("; ") + ".",
	368	+ "Private platform (invitation-only). Plan: " + plans.join("; ") + ".",
203	369	"",
204		− "Not yet available: managed browser (`browser: true`, `POST /v1/browser`), structured extraction (`POST /v1/extract`), webhooks, datacenter/isp/mobile network classes.",
	370	+ "Not yet available: browser actions (`POST /v1/browser`), structured extraction (`POST /v1/extract`), webhook delivery, datacenter/isp/mobile network classes.",
205	371	].join("\n"),
206	372	contact: { name: "Fetcha support", email: "support@fetcha.co", url: `${BASE_URL}/docs` },
207	373	termsOfService: `${BASE_URL}/legal/terms`,
		`@@ -210,6 +376,7 @@ const document = {`
210	376	externalDocs: { url: `${BASE_URL}/docs`, description: "Fetcha documentation" },
211	377	tags: [
212	378	{ name: "Fetch", description: "Retrieve URLs through the routing engine." },
	379	+ { name: "Crawl", description: "Asynchronous site crawls and synchronous URL discovery." },
213	380	{ name: "Sessions", description: "Sticky exit identities." },
214	381	{ name: "Account", description: "Key introspection and usage." },
215	382	{ name: "Health", description: "Public health probes (no authentication)." },
		`@@ -222,11 +389,11 @@ const document = {`
222	389	operationId: "fetch",
223	390	summary: "Fetch a URL",
224	391	description:
225		− "Fetches the URL through the best available route, retrying and escalating on blocks within the request's single `timeout`. Returns 200 whenever the target answered, including 4xx/5xx and fully blocked targets (`success: false`). Requires scope `fetch:execute`.",
226		− requestBody: { required: true, content: { "application/json": { schema: { $ref: "#/components/schemas/FetchRequest" }, example: { url: "https://example.com", country: "CA", format: "text" } } } },
	392	+ "Fetches the URL through the best available route, retrying and escalating on blocks (up to a managed browser render for JavaScript challenges) within the request's single `timeout`. Returns 200 whenever the target answered, including 4xx/5xx and fully blocked targets (`success: false`). Requires scope `fetch:execute`.",
	393	+ requestBody: { required: true, content: { "application/json": { schema: { $ref: "#/components/schemas/FetchRequest" }, example: { url: "https://example.com", country: "CA", format: "markdown" } } } },
227	394	responses: {
228	395	"200": {
229		− description: "Fetcha obtained a response from the target. Inspect `success` and `status`.",
	396	+ description: "Fetcha obtained a response from the target. Inspect `success`, `status` and `metadata.mode`.",
230	397	headers: { "X-Fetcha-Request-ID": { $ref: "#/components/headers/X-Fetcha-Request-ID" } },
231	398	content: { "application/json": { schema: { $ref: "#/components/schemas/FetchResponse" } } },
232	399	},
		`@@ -247,10 +414,91 @@ const document = {`
247	414	"TOO_MANY_REDIRECTS",
248	415	"PROVIDER_UNAVAILABLE",
249	416	"TARGET_TIMEOUT",
	417	+ "BROWSER_TIMEOUT",
250	418	]),
251	419	},
252	420	},
253	421	},
	422	+ "/v1/crawl": {
	423	+ post: {
	424	+ tags: ["Crawl"],
	425	+ operationId: "createCrawl",
	426	+ summary: "Start a crawl job",
	427	+ description: "Queues an asynchronous crawl from a seed URL. Each crawled page is a normal fetch request (source `crawl`). Requires scope `fetch:execute`. Limits: 2,000 pages per job, 5 concurrent jobs per organization.",
	428	+ requestBody: { required: true, content: { "application/json": { schema: { $ref: "#/components/schemas/CrawlCreate" }, example: { url: "https://docs.example.com/", max_pages: 200, max_depth: 3, include_patterns: ["/docs/*"], format: "markdown" } } } },
	429	+ responses: {
	430	+ "202": { description: "Job queued.", headers: { "X-Fetcha-Request-ID": { $ref: "#/components/headers/X-Fetcha-Request-ID" } }, content: { "application/json": { schema: { $ref: "#/components/schemas/CrawlCreated" } } } },
	431	+ ...byStatus(CRAWL_ERRORS),
	432	+ },
	433	+ },
	434	+ get: {
	435	+ tags: ["Crawl"],
	436	+ operationId: "listCrawls",
	437	+ summary: "List crawl jobs",
	438	+ description: "Most recent jobs of the project, newest first.",
	439	+ parameters: [{ name: "limit", in: "query", required: false, schema: { type: "integer", default: 50 } }],
	440	+ responses: {
	441	+ "200": { description: "Jobs.", content: { "application/json": { schema: { type: "object", required: ["data"], properties: { data: { type: "array", items: { $ref: "#/components/schemas/CrawlJob" } } } } } } },
	442	+ ...byStatus([...AUTH_ERRORS, "INTERNAL_ERROR"]),
	443	+ },
	444	+ },
	445	+ },
	446	+ "/v1/crawl/{id}": {
	447	+ parameters: [{ name: "id", in: "path", required: true, schema: { type: "string" }, example: "crawl_3k9d0f2a8b1c7e4m" }],
	448	+ get: {
	449	+ tags: ["Crawl"],
	450	+ operationId: "getCrawl",
	451	+ summary: "Get a crawl job",
	452	+ responses: {
	453	+ "200": { description: "Job with live status and stats.", content: { "application/json": { schema: { $ref: "#/components/schemas/CrawlJob" } } } },
	454	+ ...byStatus([...AUTH_ERRORS, "CRAWL_NOT_FOUND", "INTERNAL_ERROR"]),
	455	+ },
	456	+ },
	457	+ delete: {
	458	+ tags: ["Crawl"],
	459	+ operationId: "cancelCrawl",
	460	+ summary: "Cancel a crawl job",
	461	+ description: "Stops a queued or running job. Pages already fetched remain available. Requires scope `fetch:execute`.",
	462	+ responses: {
	463	+ "200": { description: "Cancelled.", content: { "application/json": { schema: { type: "object", required: ["id", "status"], properties: { id: { type: "string" }, status: { type: "string", const: "cancelled" } } } } } },
	464	+ ...byStatus([...AUTH_ERRORS, "CRAWL_NOT_FOUND", "INTERNAL_ERROR"]),
	465	+ },
	466	+ },
	467	+ },
	468	+ "/v1/crawl/{id}/pages": {
	469	+ get: {
	470	+ tags: ["Crawl"],
	471	+ operationId: "listCrawlPages",
	472	+ summary: "List crawled pages",
	473	+ description: "Pages of a job, ordered by fetch time, with cursor pagination.",
	474	+ parameters: [
	475	+ { name: "id", in: "path", required: true, schema: { type: "string" } },
	476	+ { name: "cursor", in: "query", required: false, schema: { type: "string" }, description: "Opaque cursor from `next_cursor`." },
	477	+ { name: "limit", in: "query", required: false, schema: { type: "integer", default: 100 } },
	478	+ { name: "status", in: "query", required: false, schema: { type: "string", enum: ["success", "blocked", "failed"] } },
	479	+ ],
	480	+ responses: {
	481	+ "200": {
	482	+ description: "A page of results.",
	483	+ content: { "application/json": { schema: { type: "object", required: ["data", "next_cursor"], properties: { data: { type: "array", items: { $ref: "#/components/schemas/CrawlPage" } }, next_cursor: { type: ["string", "null"] } } } } },
	484	+ },
	485	+ ...byStatus([...AUTH_ERRORS, "CRAWL_NOT_FOUND", "INTERNAL_ERROR"]),
	486	+ },
	487	+ },
	488	+ },
	489	+ "/v1/map": {
	490	+ post: {
	491	+ tags: ["Crawl"],
	492	+ operationId: "mapSite",
	493	+ summary: "Map a site's URLs",
	494	+ description: "Synchronously lists the URLs of a site from its sitemap(s) and the links of the seed page (up to 60 s). Requires scope `fetch:execute`.",
	495	+ requestBody: { required: true, content: { "application/json": { schema: { $ref: "#/components/schemas/MapCreate" }, example: { url: "https://docs.example.com/", search: "/docs/*", limit: 500 } } } },
	496	+ responses: {
	497	+ "200": { description: "URL list.", headers: { "X-Fetcha-Request-ID": { $ref: "#/components/headers/X-Fetcha-Request-ID" } }, content: { "application/json": { schema: { $ref: "#/components/schemas/MapResult" } } } },
	498	+ ...byStatus([...AUTH_ERRORS, "INVALID_REQUEST", "URL_NOT_ALLOWED", "RATE_LIMITED", "TARGET_TIMEOUT", "TARGET_UNAVAILABLE", "INTERNAL_ERROR"]),
	499	+ },
	500	+ },
	501	+ },
254	502	"/v1/sessions": {
255	503	post: {
256	504	tags: ["Sessions"],
		`@@ -320,7 +568,7 @@ const document = {`
320	568	id: { type: "string" },
321	569	name: { type: "string" },
322	570	mode: { type: "string", enum: ["live", "test"] },
323		− scopes: { type: "array", items: { type: "string", enum: ["fetch:execute", "browser:use", "sessions:write", "usage:read"] } },
	571	+ scopes: { type: "array", items: { type: "string", enum: ["fetch:execute", "browser:use", "crawl:execute", "sessions:write", "usage:read"] } },
324	572	},
325	573	},
326	574	},
		`@@ -421,8 +669,17 @@ const document = {`
421	669	FetchRequest: fetchRequest,
422	670	FetchResponse: fetchResponse,
423	671	Timing: timing,
	672	+ PageMetadata: pageMetadata,
	673	+ PageLink: pageLink,
424	674	SessionCreate: sessionCreate,
425	675	Session: session,
	676	+ CrawlCreate: crawlCreate,
	677	+ CrawlCreated: crawlCreated,
	678	+ CrawlJob: crawlJob,
	679	+ CrawlStats: crawlStats,
	680	+ CrawlPage: crawlPage,
	681	+ MapCreate: mapCreate,
	682	+ MapResult: mapResult,
426	683	},
427	684	},
428	685	"x-error-catalogue": Object.fromEntries(ERROR_CODES.map((c) => [c, { status: ERROR_HTTP_STATUS[c], message: ERROR_MESSAGES[c] }])),

modified apps/web/src/app/(marketing)/docs/page.tsx +15 −11

		`@@ -11,21 +11,22 @@ export const metadata: Metadata = {`
11	11	};
12	12
13	13	const LIVE: Array<[string, string]> = [
14		− ["POST /v1/fetch", "Fetch any public http(s) URL through the routing engine. HTML, text, JSON or raw output."],
15		− ["Residential network + auto", "The residential class is live in every plan. auto currently resolves to residential."],
	14	+ ["POST /v1/fetch", "Fetch any public http(s) URL through the routing engine. HTML, text, Markdown, JSON or raw output, page metadata and links."],
	15	+ ["Managed browser", "browser: true renders the page in a headless Chromium on the same route; blocked HTTP attempts escalate to it automatically (browser_fallback)."],
	16	+ ["Crawl & map", "POST /v1/crawl turns a site into Markdown/text/HTML as an asynchronous job; POST /v1/map lists a site's URLs."],
	17	+ ["Residential network + auto", "The residential class is live. auto currently resolves to residential."],
16	18	["Geolocation", "Country targeting (ISO 3166-1 alpha-2), with region and city hints."],
17	19	["Sessions", "Sticky exit identity for 60 to 1,800 seconds via /v1/sessions."],
18		− ["Retries & escalation", "Automatic retry with a new exit IP, block-page detection, per-domain intelligence, circuit breakers."],
	20	+ ["Retries & escalation", "Automatic retry with a new exit IP and header profile, expanded block detection (Cloudflare, DataDome, PerimeterX, Akamai, Kasada, Imperva, AWS WAF, soft blocks), per-domain intelligence, circuit breakers."],
19	21	["Account endpoints", "GET /v1/me and GET /v1/usage for key introspection and monthly usage."],
20		− ["Dashboard", "Projects, API keys, request logs with request IDs, usage and a Playground."],
	22	+ ["Dashboard", "Projects, API keys, request logs with request IDs, crawls, usage and a Playground."],
21	23	];
22	24
23	25	const SOON: Array<[string, string]> = [
24		− ["Managed browser", "browser: true and POST /v1/browser. Today browser: true returns BROWSER_UNAVAILABLE."],
	26	+ ["Browser actions", "POST /v1/browser (click, type, scroll, evaluate). Today it returns BROWSER_UNAVAILABLE. Rendered fetches are live."],
25	27	["Structured extraction", "POST /v1/extract. Today it returns INVALID_REQUEST with an explanatory message."],
26	28	["Datacenter, ISP and mobile classes", "Requesting them explicitly returns NETWORK_UNAVAILABLE until they launch."],
27		− ["Webhooks", "Event delivery signed with HMAC-SHA256. Not configurable yet."],
28		− ["Self-serve billing", "Plan upgrades and checkout. Contact sales@fetcha.co in the meantime."],
	29	+ ["Webhooks", "Event delivery signed with HMAC-SHA256 (including crawl webhook_url). Not delivered yet."],
29	30	["CLI", "A command-line client. Not available."],
30	31	["Teams & invitations", "Multi-member organizations."],
31	32	["Published SDK packages", "@fetcha/sdk on npm and fetcha on PyPI. Both exist as source in the repository today."],
		`@@ -61,8 +62,9 @@ export default function IntroductionPage() {`
61	62	orders them from cheapest to most reliable.
62	63	</Li>
63	64	<Li>
64		− <Strong>Execute and escalate.</Strong> The request is sent through the best route. If the response looks like a block page (403, 429, captcha, anti-bot challenge, WAF page), Fetcha retries with
65		− a fresh exit identity or an alternate route within your plan's retry budget and the request's single overall timeout.
	65	+ <Strong>Execute and escalate.</Strong> The request is sent through the best route with a complete browser header profile. If the response looks like a block page (403, 429, captcha, anti-bot
	66	+ challenge, WAF page, soft 200 block), Fetcha retries with a fresh exit identity or an alternate route, and renders the page in the managed browser when the block is a JavaScript
	67	+ challenge, all within the request's retry budget and single overall timeout.
66	68	</Li>
67	69	<Li>
68	70	<Strong>Learn.</Strong> Each attempt updates a per-domain profile so future requests to the same site start from the route most likely to succeed.
		@@ -108,8 +110,8 @@ Cache-Control: no-store`}
108	110
109	111	<H2>What is live today</H2>
110	112	<P>
111		− Fetcha is in its first public version. The table below is the authoritative list of what you can call right now; the section after it lists what is planned. The docs never describe a
112		− capability as working unless it is.
	113	+ Fetcha is a private platform: access is granted by invitation and every account runs on the single unlimited plan. The table below is the authoritative list of what you can call right
	114	+ now; the section after it lists what is planned. The docs never describe a capability as working unless it is.
113	115	</P>
114	116	<Table>
115	117	<THead>
		@@ -166,6 +168,8 @@ Cache-Control: no-store`}
166	168	<CardGrid>
167	169	<LinkCard href="/docs/quickstart" title="Quickstart" description="Create a key and make your first request in under five minutes." />
168	170	<LinkCard href="/docs/fetch" title="Fetch API reference" description="Every request field, the response shape, size limits and redirect rules." />
	171	+ <LinkCard href="/docs/browser" title="Browser" description="Rendered fetches, automatic escalation on JavaScript challenges, screenshots." />
	172	+ <LinkCard href="/docs/crawl" title="Crawl & Map" description="Turn a whole site into Markdown with an asynchronous job, or list its URLs." />
169	173	<LinkCard href="/docs/networks" title="Network selection" description="What auto does, how routes are scored and what each class means." />
170	174	<LinkCard href="/docs/errors" title="Errors" description="All error codes with HTTP status, meaning and how to recover." />
171	175	<LinkCard href="/docs/sdks" title="SDKs" description="JavaScript/TypeScript and Python clients, installed from source." />

modified apps/web/src/app/(marketing)/docs/rate-limits/page.tsx +83 −74

		`@@ -1,5 +1,5 @@`
1	1	import type { Metadata } from "next";
2		−import { PLAN_LIMITS, PLANS } from "@fetcha/core";
	2	+import { PLAN_LIMITS } from "@fetcha/core";
3	3	import { CodeBlock } from "@/components/ui/code-block";
4	4	import { DocPage } from "@/components/docs/doc-page";
5	5	import { A, Code, H2, H3, Li, P, Strong, Table, TBody, Td, Th, THead, Tr, Ul } from "@/components/docs/prose";
		`@@ -11,14 +11,27 @@ import { apiTabs } from "@/components/docs/snippets";`
11	11
12	12	export const metadata: Metadata = {
13	13	title: "Rate Limits",
14		− description: "Concurrency per plan, sliding-window rate limits, RATE_LIMITED vs CONCURRENCY_LIMIT, monthly quotas, spending limits, USAGE_LIMIT_REACHED and response headers.",
	14	+ description: "Concurrency, sliding-window rate limits, RATE_LIMITED vs CONCURRENCY_LIMIT, optional spending limits, USAGE_LIMIT_REACHED and response headers on the single unlimited plan.",
15	15	};
16	16
17		−const fmt = (n: number) => (n >= Number.MAX_SAFE_INTEGER ? "Unlimited" : n.toLocaleString("en-US"));
	17	+const L = PLAN_LIMITS.unlimited;
	18	+const fmt = (n: number) => n.toLocaleString("en-US");
18	19
19	20	export default function RateLimitsPage() {
	21	+ const perOrgMinute = Math.max(60, L.concurrency * 60);
	22	+ const perKeySecond = Math.max(10, L.concurrency * 2);
20	23	return (
21		− <DocPage path="/docs/rate-limits" eyebrow="Reliability" title="Rate limits" description="Fetcha enforces three kinds of limits: how many requests run at once, how fast you can submit them, and how many you can make per month. All are derived from your plan." status="Stable">
	24	+ <DocPage
	25	+ path="/docs/rate-limits"
	26	+ eyebrow="Reliability"
	27	+ title="Rate limits"
	28	+ description="Fetcha is a private platform with a single plan: there is no monthly request quota. Two operational limits remain — how many requests run at once and how fast you can submit them — plus spending limits you can set yourself."
	29	+ status="Stable"
	30	+ >
	31	+ <Callout variant="info" title="One plan, no meters">
	32	+ Every organization has the same limits, listed below and returned by <Code>GET /v1/usage</Code>. There is nothing to upgrade: hitting a limit means smoothing your traffic, not paying more.
	33	+ </Callout>
	34	+
22	35	<H2>Concurrency</H2>
23	36	<P>
24	37	Concurrency is the number of fetch requests in flight at the same time, counted per <Strong>organization</Strong> (all projects and keys together). It is the limit that matters most in
		`@@ -27,31 +40,44 @@ export default function RateLimitsPage() {`
27	40	<Table>
28	41	<THead>
29	42	<Tr>
30		− <Th>Plan</Th>
31		− <Th>Concurrent requests</Th>
32		− <Th>Monthly requests</Th>
33		− <Th>Log retention</Th>
	43	+ <Th>Limit</Th>
	44	+ <Th>Value</Th>
	45	+ <Th>Scope</Th>
34	46	</Tr>
35	47	</THead>
36	48	<TBody>
37		− {PLANS.map((p) => (
38		− <Tr key={p}>
39		− <Td className="font-medium text-fg">{PLAN_LIMITS[p].label}</Td>
40		− <Td mono>
41		− {PLAN_LIMITS[p].concurrency.toLocaleString("en-US")}
42		− {p === "enterprise" ? <span className="ml-1 font-sans text-fg-subtle">(custom on request)</span> : null}
43		− </Td>
44		− <Td mono>{fmt(PLAN_LIMITS[p].monthly_requests)}</Td>
45		− <Td mono>{PLAN_LIMITS[p].retention_days} days</Td>
46		− </Tr>
47		− ))}
	49	+ <Tr>
	50	+ <Td className="font-medium text-fg">Concurrent requests</Td>
	51	+ <Td mono>{fmt(L.concurrency)}</Td>
	52	+ <Td>Organization</Td>
	53	+ </Tr>
	54	+ <Tr>
	55	+ <Td className="font-medium text-fg">Concurrent browser renders</Td>
	56	+ <Td mono>{fmt(L.browser_concurrency)}</Td>
	57	+ <Td>Organization (subset of the above)</Td>
	58	+ </Tr>
	59	+ <Tr>
	60	+ <Td className="font-medium text-fg">Concurrent crawl jobs</Td>
	61	+ <Td mono>{fmt(L.crawl_concurrent_jobs)}</Td>
	62	+ <Td>Organization</Td>
	63	+ </Tr>
	64	+ <Tr>
	65	+ <Td className="font-medium text-fg">Monthly requests</Td>
	66	+ <Td mono>Unlimited</Td>
	67	+ <Td>—</Td>
	68	+ </Tr>
	69	+ <Tr>
	70	+ <Td className="font-medium text-fg">Log retention</Td>
	71	+ <Td mono>{L.retention_days} days</Td>
	72	+ <Td>Organization</Td>
	73	+ </Tr>
48	74	</TBody>
49	75	</Table>
50	76	<P>
51	77	When a request would exceed the limit it is refused immediately with <Code>429 CONCURRENCY_LIMIT</Code>; nothing is queued on Fetcha's side. A slot is released as soon as a request
52	78	completes (successfully or not). As a safety net, a slot that is somehow never released expires after 150 seconds.
53	79	</P>
54		− <ResponseExample status={429} statusText="Too Many Requests" body={{ error: { code: "CONCURRENCY_LIMIT", message: "Your plan allows 25 concurrent requests.", request_id: "req_1l2m3n4o5p6q7r8s", details: { limit: 25 } } }} />
	80	+ <ResponseExample status={429} statusText="Too Many Requests" body={{ error: { code: "CONCURRENCY_LIMIT", message: `Your organization allows ${L.concurrency} concurrent requests.`, request_id: "req_1l2m3n4o5p6q7r8s", details: { limit: L.concurrency } } }} />
55	81
56	82	<H2>Request rate</H2>
57	83	<P>
		`@@ -63,7 +89,8 @@ export default function RateLimitsPage() {`
63	89	<Tr>
64	90	<Th>Window</Th>
65	91	<Th>Scope</Th>
66		− <Th>Limit</Th>
	92	+ <Th>Rule</Th>
	93	+ <Th>Value</Th>
67	94	</Tr>
68	95	</THead>
69	96	<TBody>
		`@@ -71,49 +98,30 @@ export default function RateLimitsPage() {`
71	98	<Td>60 seconds</Td>
72	99	<Td>Organization (sustained)</Td>
73	100	<Td>
74		− <Code>max(60, concurrency × 60)</Code> requests per minute
	101	+ <Code>concurrency × 60</Code> per minute
75	102	</Td>
	103	+ <Td mono>{fmt(perOrgMinute)} / min</Td>
76	104	</Tr>
77	105	<Tr>
78	106	<Td>1 second</Td>
79	107	<Td>API key (burst)</Td>
80	108	<Td>
81		− <Code>max(10, concurrency × 2)</Code> requests per second
	109	+ <Code>concurrency × 2</Code> per second
82	110	</Td>
	111	+ <Td mono>{fmt(perKeySecond)} / s</Td>
83	112	</Tr>
84	113	<Tr>
85	114	<Td>1 second</Td>
86	115	<Td>Client IP (abuse)</Td>
87		− <Td>200 requests per second</Td>
	116	+ <Td>Fixed</Td>
	117	+ <Td mono>200 / s</Td>
88	118	</Tr>
89	119	</TBody>
90	120	</Table>
91		− <H3>Resulting values per plan</H3>
92		− <Table dense>
93		− <THead>
94		− <Tr>
95		− <Th>Plan</Th>
96		− <Th>Per organization / minute</Th>
97		− <Th>Per key / second</Th>
98		− </Tr>
99		− </THead>
100		− <TBody>
101		− {PLANS.map((p) => {
102		− const c = PLAN_LIMITS[p].concurrency;
103		− return (
104		− <Tr key={p}>
105		− <Td className="font-medium text-fg">{PLAN_LIMITS[p].label}</Td>
106		− <Td mono>{Math.max(60, c * 60).toLocaleString("en-US")}</Td>
107		− <Td mono>{Math.max(10, c * 2).toLocaleString("en-US")}</Td>
108		− </Tr>
109		− );
110		− })}
111		− </TBody>
112		− </Table>
113	121	<P>
114	122	Exceeding any window returns <Code>429 RATE_LIMITED</Code> with a <Code>Retry-After</Code> header (seconds, rounded up) and <Code>details.retry_after_ms</Code>.
115	123	</P>
116		− <ResponseExample status={429} statusText="Too Many Requests" body={{ error: { code: "RATE_LIMITED", message: "Too many requests. Slow down or upgrade your plan.", request_id: "req_4e5f6g7h8i9j0k1l", details: { retry_after_ms: 640 } } }} />
	124	+ <ResponseExample status={429} statusText="Too Many Requests" body={{ error: { code: "RATE_LIMITED", message: "Too many requests. Slow down and retry after the indicated delay.", request_id: "req_4e5f6g7h8i9j0k1l", details: { retry_after_ms: 640 } } }} />
117	125
118	126	<H3>RATE_LIMITED vs CONCURRENCY_LIMIT</H3>
119	127	<Ul>
		`@@ -122,28 +130,22 @@ export default function RateLimitsPage() {`
122	130	</Li>
123	131	<Li>
124	132	<Code>CONCURRENCY_LIMIT</Code>: too many requests are running right now. Do not sleep a fixed time; retry when one of your in-flight requests finishes. Use a semaphore or worker pool sized
125		− to your plan (see the <A href="/docs/examples#concurrency-within-plan-limits">concurrency example</A>).
	133	+ to {fmt(L.concurrency)} or less (see the <A href="/docs/examples#concurrency-within-plan-limits">concurrency example</A>).
126	134	</Li>
127	135	</Ul>
128	136
129		− <H2>Monthly quota</H2>
	137	+ <H2>No monthly quota</H2>
130	138	<P>
131		− Each plan includes a number of requests per calendar month (UTC). Usage is counted per organization across all projects and keys. <Strong>Every request that reaches the execution
132		− pipeline counts</Strong>, including those that end blocked or failed; requests refused earlier (authentication, validation, rate limits, session errors) do not. Free and Enterprise have no
133		− overage; paid plans continue past the included volume only once self-serve billing is available.
	139	+ There is no cap on the number of requests per month and no overage. Usage is still counted per organization and per project (<Strong>every request that reaches the execution pipeline
	140	+ counts</Strong>, including those that end blocked or failed) so that the dashboard, <Code>GET /v1/usage</Code> and your optional spending limits have accurate figures. Requests refused
	141	+ earlier (authentication, validation, rate limits, session errors) are not counted.
134	142	</P>
135		− <P>
136		− When the quota is exhausted, fetches return <Code>402 USAGE_LIMIT_REACHED</Code> until the first day of the next month. Usage counters are cached for up to 20 seconds, so enforcement can
137		− lag a few requests behind the exact limit.
138		− </P>
139		− <ResponseExample
140		− status={402}
141		− statusText="Payment Required"
142		− body={{ error: { code: "USAGE_LIMIT_REACHED", message: "Monthly request quota of 1,000 reached for the Free plan.", request_id: "req_8s9t0u1v2w3x4y5z", details: { limit: 1000, used: 1000 } } }}
143		− />
144	143
145	144	<H2>Spending and project limits</H2>
146		− <P>Beyond the plan quota, you can cap usage yourself in the dashboard. All of these produce the same <Code>USAGE_LIMIT_REACHED</Code> code with a specific message and details:</P>
	145	+ <P>
	146	+ You can cap usage yourself in the dashboard. These are guard-rails against runaway jobs, not billing: spend is an internal cost estimate and nothing is invoiced. All of them produce the same{" "}
	147	+ <Code>USAGE_LIMIT_REACHED</Code> code with a specific message and details:
	148	+ </P>
147	149	<Table dense>
148	150	<THead>
149	151	<Tr>
		`@@ -161,28 +163,34 @@ export default function RateLimitsPage() {`
161	163	<Td mono>{`{ limit, used }`}</Td>
162	164	</Tr>
163	165	<Tr>
164		− <Td>Hard spending limit (USD)</Td>
	166	+ <Td>Hard spending limit (USD estimate)</Td>
165	167	<Td>Organization</Td>
166	168	<Td>All fetches are refused</Td>
167	169	<Td mono>{`{ limit_usd, spent_usd }`}</Td>
168	170	</Tr>
169	171	<Tr>
170		− <Td>Hard spending limit (USD)</Td>
	172	+ <Td>Hard spending limit (USD estimate)</Td>
171	173	<Td>Project</Td>
172	174	<Td>Fetches in that project are refused</Td>
173	175	<Td mono>{`{ limit_usd, spent_usd }`}</Td>
174	176	</Tr>
175	177	<Tr>
176		− <Td>Soft spending limit (USD)</Td>
	178	+ <Td>Soft spending limit (USD estimate)</Td>
177	179	<Td>Organization or project</Td>
178	180	<Td>Alert only; traffic continues</Td>
179	181	<Td>—</Td>
180	182	</Tr>
181	183	</TBody>
182	184	</Table>
183		− <Callout variant="info" title="Spend is computed from your plan's prices">
184		− Spend is the customer price of your usage (per-request overage plus residential bandwidth at your plan rate), accumulated over the calendar month. It is what <Code>GET /v1/usage</Code>{" "}
185		− reports as <Code>spend_usd</Code>.
	185	+ <ResponseExample
	186	+ status={402}
	187	+ statusText="Payment Required"
	188	+ title="402 · project request limit reached (set by you)"
	189	+ body={{ error: { code: "USAGE_LIMIT_REACHED", message: "Monthly request limit of 50,000 reached for this project.", request_id: "req_8s9t0u1v2w3x4y5z", details: { limit: 50000, used: 50000 } } }}
	190	+ />
	191	+ <Callout variant="info" title="Spend is an estimate">
	192	+ Spend is Fetcha's internal cost estimate of your usage, accumulated over the calendar month (UTC). It is what <Code>GET /v1/usage</Code> reports as <Code>spend_usd</Code>. Usage counters
	193	+ are cached for up to 20 seconds, so enforcement of a limit you set can lag a few requests behind.
186	194	</Callout>
187	195
188	196	<H2>Checking your usage</H2>
		`@@ -192,18 +200,19 @@ export default function RateLimitsPage() {`
192	200	status={200}
193	201	body={{
194	202	period_start: "2026-09-01T00:00:00.000Z",
195		− plan: { id: "developer", label: "Developer", monthly_requests: 50000, concurrency: 25 },
	203	+ plan: { id: "unlimited", label: "Unlimited", monthly_requests: null, concurrency: L.concurrency },
196	204	organization: { requests: 12840, spend_usd: 3.2115 },
197	205	project: { requests: 9911, spend_usd: 2.4402, successful_requests: 9640, success_rate: 97.3, bandwidth_bytes: 3892214411, latency_p50_ms: 1180, latency_p95_ms: 4210 },
198		− remaining_requests: 37160,
	206	+ remaining_requests: null,
199	207	}}
200	208	/>
201	209	<Ul>
202	210	<Li>
203		− <Code>organization</Code> covers every project; <Code>project</Code> is the key's project only. Quotas apply to the organization figures.
	211	+ <Code>organization</Code> covers every project; <Code>project</Code> is the key's project only.
204	212	</Li>
205	213	<Li>
206		− <Code>remaining_requests</Code> is the plan quota minus organization requests this month, never negative.
	214	+ <Code>plan.monthly_requests</Code> and <Code>remaining_requests</Code> are <Code>null</Code>: there is no quota. A project-level request limit you configured is reported in the
	215	+ dashboard, not here.
207	216	</Li>
208	217	<Li>
209	218	<Code>success_rate</Code> is a percentage with one decimal, or <Code>null</Code> when the project made no requests this month.
		`@@ -211,11 +220,11 @@ export default function RateLimitsPage() {`
211	220	</Ul>
212	221
213	222	<H2>Response headers</H2>
214		− <P>Fetcha currently sets these headers on every API response. There are no <Code>X-RateLimit-*</Code> counters yet; use <Code>GET /v1/usage</Code> for remaining quota.</P>
	223	+ <P>Fetcha currently sets these headers on every API response. There are no <Code>X-RateLimit-*</Code> counters; use <Code>GET /v1/usage</Code> for month-to-date figures.</P>
215	224	<CodeBlock
216	225	lang="text"
217	226	code={`X-Fetcha-Request-ID: req_… # always; quote it to support
218		−X-Fetcha-Version: 0.1.0 # API version
	227	+X-Fetcha-Version: 0.2.0 # API version
219	228	Cache-Control: no-store # responses are never cacheable
220	229	Retry-After: <seconds> # only on 429 RATE_LIMITED`}
221	230	/>

modified apps/web/src/app/(marketing)/docs/retries/page.tsx +33 −26

		`@@ -1,5 +1,5 @@`
1	1	import type { Metadata } from "next";
2		−import { PLAN_LIMITS, PLANS } from "@fetcha/core";
	2	+import { PLAN_LIMITS } from "@fetcha/core";
3	3	import { CodeBlock } from "@/components/ui/code-block";
4	4	import { DocPage } from "@/components/docs/doc-page";
5	5	import { A, Code, H2, H3, Li, P, Strong, Table, TBody, Td, Th, THead, Tr, Ul } from "@/components/docs/prose";
		`@@ -10,9 +10,11 @@ import { fetchTabs } from "@/components/docs/snippets";`
10	10
11	11	export const metadata: Metadata = {
12	12	title: "Retries",
13		− description: "How Fetcha retries and escalates: what triggers a retry, the retries option and plan maximums, the shared timeout, attempt metadata, cost implications and idempotency.",
	13	+ description: "How Fetcha retries and escalates: what triggers a retry, the retries option and its maximum, the shared timeout, attempt metadata, bandwidth implications and idempotency.",
14	14	};
15	15
	16	+const L = PLAN_LIMITS.unlimited;
	17	+
16	18	export default function RetriesPage() {
17	19	return (
18	20	<DocPage path="/docs/retries" eyebrow="Reliability" title="Retries" description="Retries are built into every fetch. When a target blocks or a route fails, Fetcha tries again with a new exit identity and, when it helps, a different route, all within a single request and a single timeout." status="Stable">
		`@@ -101,27 +103,35 @@ export default function RetriesPage() {`
101	103
102	104	<H2>The retries option</H2>
103	105	<P>
104		− <Code>retries</Code> is the number of <Strong>additional</Strong> attempts after the first; total attempts are <Code>retries + 1</Code>. Omit it to use your plan's maximum. Values above
105		− the plan maximum are reduced silently; the schema accepts 0 to 5.
	106	+ <Code>retries</Code> is the number of <Strong>additional</Strong> attempts after the first; total attempts are <Code>retries + 1</Code>. Omit it to use the maximum. The schema accepts 0 to{" "}
	107	+ {L.max_retries}; the same maximum applies to every organization (single plan).
106	108	</P>
107	109	<Table>
108	110	<THead>
109	111	<Tr>
110		− <Th>Plan</Th>
111		− <Th>Max retries</Th>
112		− <Th>Max attempts</Th>
113		− <Th>Max timeout</Th>
	112	+ <Th>Limit</Th>
	113	+ <Th>Value</Th>
114	114	</Tr>
115	115	</THead>
116	116	<TBody>
117		− {PLANS.map((p) => (
118		− <Tr key={p}>
119		− <Td className="font-medium text-fg">{PLAN_LIMITS[p].label}</Td>
120		− <Td mono>{PLAN_LIMITS[p].max_retries}</Td>
121		− <Td mono>{PLAN_LIMITS[p].max_retries + 1}</Td>
122		− <Td mono>{PLAN_LIMITS[p].max_timeout_ms / 1000} s</Td>
123		− </Tr>
124		− ))}
	117	+ <Tr>
	118	+ <Td className="font-medium text-fg">Max retries</Td>
	119	+ <Td mono>{L.max_retries}</Td>
	120	+ </Tr>
	121	+ <Tr>
	122	+ <Td className="font-medium text-fg">Max attempts</Td>
	123	+ <Td mono>{L.max_retries + 1}</Td>
	124	+ </Tr>
	125	+ <Tr>
	126	+ <Td className="font-medium text-fg">Max timeout</Td>
	127	+ <Td mono>{L.max_timeout_ms / 1000} s</Td>
	128	+ </Tr>
	129	+ <Tr>
	130	+ <Td className="font-medium text-fg">Browser fallback</Td>
	131	+ <Td>
	132	+ On by default (<Code>browser_fallback: true</Code>): a JavaScript challenge on an HTTP attempt escalates to the managed browser within the same budget.
	133	+ </Td>
	134	+ </Tr>
125	135	</TBody>
126	136	</Table>
127	137	<P>
		`@@ -133,7 +143,7 @@ export default function RetriesPage() {`
133	143	<H2>One timeout for everything</H2>
134	144	<P>
135	145	<Code>timeout</Code> is the budget for the <Strong>whole</Strong> request, not per attempt. Each attempt receives what is left; when fewer than 500 ms remain, Fetcha stops and returns{" "}
136		− <Code>TARGET_TIMEOUT</Code>. If you rely on several attempts against slow targets, size <Code>timeout</Code> accordingly (up to your plan cap) rather than raising <Code>retries</Code>.
	146	+ <Code>TARGET_TIMEOUT</Code>. If you rely on several attempts against slow targets, size <Code>timeout</Code> accordingly (up to {L.max_timeout_ms / 1000} s) rather than raising <Code>retries</Code>.
137	147	</P>
138	148
139	149	<H2>Attempts in metadata</H2>
		`@@ -164,21 +174,18 @@ export default function RetriesPage() {`
164	174	}}
165	175	/>
166	176
167		− <H2>Cost implications</H2>
	177	+ <H2>Usage implications</H2>
	178	+ <P>Fetcha has no billing, but retries still show up in your usage figures and in the internal cost estimate behind optional spending limits:</P>
168	179	<Ul>
169	180	<Li>
170		− <Strong>Bandwidth is counted for every attempt.</Strong> <Code>metadata.bytes</Code> sums request and response bytes across attempts, and on paid plans residential bandwidth is billed
171		− per GB on that total. A block page is usually small, but a large page that gets blocked after being fully downloaded costs its size.
172		− </Li>
173		− <Li>
174		− <Strong>The per-request fee applies only to successful requests.</Strong> On plans with an overage rate, a request that ends with <Code>success: false</Code> or an error carries no
175		− request fee.
	181	+ <Strong>Bandwidth is counted for every attempt.</Strong> <Code>metadata.bytes</Code> sums request and response bytes across attempts. A block page is usually small, but a large page that
	182	+ gets blocked after being fully downloaded counts its full size.
176	183	</Li>
177	184	<Li>
178		− <Strong>Every request counts toward the monthly quota</Strong>, including failed and blocked ones. This is an abuse control; see <A href="/docs/rate-limits">Rate limits</A>.
	185	+ <Strong>Every request counts in your usage</Strong>, including failed and blocked ones, but there is no monthly quota; see <A href="/docs/rate-limits">Rate limits</A>.
179	186	</Li>
180	187	<Li>
181		− Requests refused before any attempt (validation, limits, <Code>URL_NOT_ALLOWED</Code>, session errors) transfer no bytes and cost nothing beyond the quota count.
	188	+ Requests refused before any attempt (validation, limits, <Code>URL_NOT_ALLOWED</Code>, session errors) transfer no bytes and are not counted.
182	189	</Li>
183	190	</Ul>
184	191

modified apps/web/src/app/(marketing)/page.tsx +30 −25

		`@@ -13,6 +13,7 @@ import {`
13	13	Layers,
14	14	MapPin,
15	15	MonitorSmartphone,
	16	+ Network,
16	17	RefreshCw,
17	18	ShieldCheck,
18	19	Terminal,
		`@@ -29,7 +30,7 @@ import { cn } from "@/lib/utils";`
29	30	export const metadata: Metadata = {
30	31	title: { absolute: "Fetcha — The web, accessible through one API." },
31	32	description:
32		− "Fetcha intelligently routes every request across premium proxy networks, sessions and browser infrastructure to deliver reliable web access at scale. 1,000 free requests a month, no credit card.",
	33	+ "Fetcha intelligently routes every request across premium proxy networks, sessions and a managed browser to deliver reliable web access at scale. Private, invitation-only platform.",
33	34	alternates: { canonical: "/" },
34	35	};
35	36
		`@@ -46,28 +47,29 @@ const SCORE_WEIGHTS = [`
46	47	{ label: "Session stability", weight: 5, tone: "bg-accent/20" },
47	48	] as const;
48	49
49		−const ESCALATION = [
	50	+const ESCALATION: ReadonlyArray<{ step: string; title: string; body: string; cost: string; state: "live" \| "soon" }> = [
50	51	{ step: "Attempt 1", title: "Best-scored route", body: "The cheapest route with a strong success history for this domain, from the requested geography.", cost: "~$0.0005", state: "live" },
51	52	{ step: "Attempt 2", title: "Alternate route, fresh IP", body: "Blocked or timed out? Fetcha rotates the exit IP and switches to an alternate route in the same class.", cost: "~$0.0009", state: "live" },
52	53	{ step: "Attempt 3", title: "Premium network", body: "Escalates to a premium residential route pinned to the target country, with tuned headers and pacing.", cost: "~$0.002", state: "live" },
53		− { step: "Attempt 4", title: "Browser execution", body: "For JavaScript-heavy targets, a managed headless browser renders the page. Not available yet.", cost: "coming soon", state: "soon" },
54		−] as const;
	54	+ { step: "Attempt 4", title: "Managed browser rendering", body: "When the block is a JavaScript challenge, a managed headless Chromium renders the page on the same route, solves the challenge and returns the rendered DOM.", cost: "~$0.004", state: "live" },
	55	+];
55	56
56	57	const FEATURES: Array<{ icon: React.ComponentType<{ className?: string }>; title: string; body: string; soon?: boolean; detail?: string }> = [
57	58	{ icon: Globe, title: "Global proxy network", body: "Residential exits in 45 target countries, aggregated behind one endpoint. Fetcha manages upstream capacity, so you never juggle vendors or IP pools.", detail: "residential · live" },
58	59	{ icon: Layers, title: "Session management", body: "Sticky sessions keep the same exit IP for up to 30 minutes. Create one with POST /v1/sessions and pass its id on every follow-up request.", detail: "ttl 60–1800 s" },
59	60	{ icon: MapPin, title: "Geo targeting", body: "Pin requests to a country, region or city with ISO codes. Prices, availability and search results as your users actually see them.", detail: "country · region · city" },
60		− { icon: RefreshCw, title: "Automatic retries", body: "Blocks, CAPTCHAs and timeouts trigger retries on a new IP or alternate network, within your plan's retry budget and timeout.", detail: "up to 5 retries" },
61		− { icon: Terminal, title: "Developer playground", body: "Run real requests from the dashboard, inspect attempts and timings, then copy production-ready code for your language.", detail: "dashboard" },
62		− { icon: Activity, title: "Observability", body: "Every call carries a request id. Logs show status, network class, attempts, latency and bytes, with sensitive headers redacted.", detail: "X-Fetcha-Request-ID" },
63		− { icon: MonitorSmartphone, title: "Browser execution", body: "Render JavaScript-heavy pages in a managed headless browser with the same routing and geo controls. Requests with browser: true currently return BROWSER_UNAVAILABLE.", soon: true },
	61	+ { icon: RefreshCw, title: "Automatic retries", body: "Blocks, CAPTCHAs, challenges and timeouts trigger retries on a new IP, a new header profile or an alternate network, with jittered backoff, within the request timeout.", detail: "up to 5 retries" },
	62	+ { icon: MonitorSmartphone, title: "Managed browser rendering", body: "Render JavaScript-heavy pages in a managed headless Chromium with the same routing, geo and session controls. Blocked HTTP attempts escalate to it automatically; screenshots on request.", detail: "browser: true · auto-escalation" },
	63	+ { icon: Network, title: "Crawl & map", body: "Turn a whole site into Markdown, text or HTML with one asynchronous job, or list its URLs from the sitemap and links. Every page is a normal, logged request.", detail: "/v1/crawl · /v1/map" },
	64	+ { icon: Terminal, title: "Developer playground", body: "Run real requests from the dashboard, inspect attempts, modes and timings, then copy production-ready code for your language.", detail: "dashboard" },
	65	+ { icon: Activity, title: "Observability", body: "Every call carries a request id. Logs show status, network class, mode, attempts, latency and bytes, with sensitive headers redacted.", detail: "X-Fetcha-Request-ID" },
64	66	{ icon: Braces, title: "Structured extraction", body: "Turn fetched pages into typed JSON with a schema you define. The /v1/extract endpoint is on the roadmap and not yet callable.", soon: true },
65		− { icon: Building2, title: "Enterprise scale", body: "Dedicated concurrency, 90-day and longer retention, custom limits, invoicing and a DPA. Talk to us for volume above 5M requests a month.", detail: "sales@fetcha.co" },
	67	+ { icon: Building2, title: "Private platform", body: "Access is granted by invitation. One unlimited plan: no request quota, 200 concurrent requests, all network classes, 90-day retention, browser and crawl included. No billing.", detail: "hello@fetcha.co" },
66	68	];
67	69
68	70	const STEPS = [
69		− { title: "Sign up", body: "Email and password. No card." },
70		− { title: "Verify", body: "Confirm your email address." },
	71	+ { title: "Get invited", body: "An administrator adds your email." },
	72	+ { title: "Log in", body: "Set your password, confirm your email." },
71	73	{ title: "Create a key", body: "Scoped fch_live_ or fch_test_ keys." },
72	74	{ title: "Open the Playground", body: "Try a URL and a country." },
73	75	{ title: "Run", body: "Inspect attempts, timing, bytes." },
		`@@ -78,7 +80,7 @@ const SECURITY = [`
78	80	{ icon: ShieldCheck, title: "SSRF protection", body: "Only http and https. Localhost, private ranges, link-local, metadata and internal hosts are refused, and every redirect is re-validated." },
79	81	{ icon: EyeOff, title: "Redacted logs", body: "Authorization, Cookie and other sensitive headers are stripped before anything is written. Response bodies are not stored by default." },
80	82	{ icon: Fingerprint, title: "Request IDs", body: "A unique req_ id on every response header and every log line, so support and audits point at exactly one call." },
81		− { icon: Clock, title: "Retention by plan", body: "Request metadata is kept 3, 7, 30 or 90 days depending on your plan, then deleted. Enterprise retention is configurable." },
	83	+ { icon: Clock, title: "90-day retention", body: "Request metadata and crawl results are kept 90 days, then deleted. Response bodies are not stored unless you enable it per project." },
82	84	];
83	85
84	86	const SESSION_SNIPPET = `# 1. Open a sticky session pinned to Canada
		`@@ -129,21 +131,24 @@ export default function HomePage() {`
129	131	The web, accessible through one API.
130	132	</h1>
131	133	<p className="mt-5 max-w-xl text-[16.5px] leading-relaxed text-fg-muted sm:text-[17.5px]">
132		− Fetcha intelligently routes every request across premium proxy networks, sessions and browser infrastructure to deliver reliable web access at scale.
	134	+ Fetcha intelligently routes every request across premium proxy networks, sticky sessions and a managed browser to deliver reliable web access at scale. Fetch, render, crawl.
133	135	</p>
134	136	<div className="mt-8 flex flex-wrap items-center gap-3">
135	137	<Button asChild size="lg">
136		− <Link href="/signup">
137		− Start building <ArrowRight aria-hidden />
	138	+ <Link href="/login">
	139	+ Log in <ArrowRight aria-hidden />
138	140	</Link>
139	141	</Button>
140	142	<Button asChild size="lg" variant="outline">
141		− <Link href="/docs">View documentation</Link>
	143	+ <a href="mailto:hello@fetcha.co?subject=Fetcha%20access">Request access</a>
142	144	</Button>
	145	+ <Link href="/docs" className="text-[14px] font-medium text-fg-muted underline-offset-4 hover:text-fg hover:underline">
	146	+ View documentation
	147	+ </Link>
143	148	</div>
144	149	<dl className="mt-10 grid max-w-md grid-cols-3 gap-4 border-t border-border pt-6">
145	150	{[
146		− ["1,000", "free requests / mo"],
	151	+ ["Unlimited", "requests, one plan"],
147	152	["45", "target countries"],
148	153	["1 call", "to get a page"],
149	154	].map(([v, l]) => (
		`@@ -229,9 +234,9 @@ export default function HomePage() {`
229	234	</ol>
230	235	</Reveal>
231	236	<div className="lg:sticky lg:top-24">
232		− <SectionHeading eyebrow="Escalation" id="escalation-title" title="Cheap first. Premium only when it earns its keep." body="A block is not a failure, it is a signal. Fetcha starts on the most economical route and escalates step by step, so you pay premium bandwidth only for the requests that need it. A blocked target still returns 200 with success: false, the upstream status and the attempt count, never a silent retry loop." />
	237	+ <SectionHeading eyebrow="Escalation" id="escalation-title" title="Cheap first. Premium and browser only when they earn their keep." body="A block is not a failure, it is a signal. Fetcha starts on the most economical route and escalates step by step, up to a full browser render for JavaScript challenges, so heavy infrastructure is used only for the requests that need it. A blocked target still returns 200 with success: false, the upstream status, the attempt count and the block reason, never a silent retry loop." />
233	238	<Reveal delay={0.1} className="mt-6">
234		− <p className="text-[12.5px] leading-relaxed text-fg-subtle">Costs shown are illustrative averages per request for a typical 200 KB HTML page. Actual billing is per request plus residential bandwidth, per your plan.</p>
	239	+ <p className="text-[12.5px] leading-relaxed text-fg-subtle">Costs shown are illustrative upstream averages per request for a typical 200 KB HTML page, for transparency. Fetcha is a private platform with a single unlimited plan; there is no per-request billing.</p>
235	240	</Reveal>
236	241	</div>
237	242	</div>
		`@@ -283,7 +288,7 @@ export default function HomePage() {`
283	288	{/* -------------------------------------------------- Developer experience */}
284	289	<section className="border-b border-border bg-bg-subtle/50 py-20 lg:py-28" aria-labelledby="dx-title">
285	290	<div className="container-page">
286		− <SectionHeading eyebrow="Developer experience" id="dx-title" title="From sign-up to a working request in a few minutes." align="center" />
	291	+ <SectionHeading eyebrow="Developer experience" id="dx-title" title="From invitation to a working request in a few minutes." align="center" />
287	292	<Reveal className="mt-12" delay={0.05}>
288	293	<ol className="grid gap-px overflow-hidden rounded-lg border border-border bg-border sm:grid-cols-3 lg:grid-cols-6">
289	294	{STEPS.map((s, i) => (
		`@@ -340,19 +345,19 @@ export default function HomePage() {`
340	345	Stop building proxy plumbing. Start fetching.
341	346	</h2>
342	347	<p className="mx-auto mt-4 max-w-xl text-[15.5px] leading-relaxed text-fg-muted">
343		− Fetcha is built and operated from Québec, Canada, by engineers who spent too many years maintaining scrapers. One endpoint, honest limits, and a public preview you can try today.
	348	+ Fetcha is built and operated from Québec, Canada, by engineers who spent too many years maintaining scrapers. One endpoint, honest limits, and a private platform for invited teams.
344	349	</p>
345	350	<div className="mt-8 flex flex-wrap items-center justify-center gap-3">
346	351	<Button asChild size="lg">
347		− <Link href="/signup">
348		− Start building <ArrowRight aria-hidden />
	352	+ <Link href="/login">
	353	+ Log in <ArrowRight aria-hidden />
349	354	</Link>
350	355	</Button>
351	356	<Button asChild size="lg" variant="outline">
352		− <Link href="/pricing">See pricing</Link>
	357	+ <a href="mailto:hello@fetcha.co?subject=Fetcha%20access">Request access</a>
353	358	</Button>
354	359	</div>
355		− <p className="mt-5 text-[13px] text-fg-subtle">No credit card. 1,000 free requests a month.</p>
	360	+ <p className="mt-5 text-[13px] text-fg-subtle">Invitation-only. One unlimited plan, no billing.</p>
356	361	</Reveal>
357	362	</div>
358	363	</section>

modified apps/web/src/app/(marketing)/pricing/page.tsx +127 −249

		`@@ -1,280 +1,158 @@`
1	1	import type { Metadata } from "next";
2	2	import Link from "next/link";
3		−import { Check, Minus } from "lucide-react";
4		−import { PLAN_LIMITS, PLANS, type Plan, type PlanLimits } from "@fetcha/core";
	3	+import { Check, KeyRound, LogIn, Mail } from "lucide-react";
	4	+import { PLAN_LIMITS } from "@fetcha/core";
5	5	import { Button } from "@/components/ui/button";
6	6	import { Badge } from "@/components/ui/badge";
7		−import { Alert } from "@/components/ui/alert";
8		−import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table";
9		−import { formatNumber, formatUsd } from "@/lib/format";
10		−import { cn } from "@/lib/utils";
	7	+import { formatNumber } from "@/lib/format";
11	8
12	9	export const metadata: Metadata = {
13		− title: "Pricing",
14		− description: "Simple, usage-based pricing for the Fetcha web access API. Start free with 1,000 requests a month. Developer $29, Growth $149, Business $599, Enterprise custom.",
	10	+ title: "Access",
	11	+ description: "Fetcha is a private, invitation-only web access platform. One plan, no meters: unlimited requests, 200 concurrent, all network classes, managed browser and crawl jobs.",
15	12	alternates: { canonical: "/pricing" },
16	13	};
17	14
18		−/* ------------------------------------------------------------------ */
19		−/* Helpers (pure formatting, no business logic) */
20		−/* ------------------------------------------------------------------ */
	15	+const REQUEST_ACCESS = "mailto:hello@fetcha.co?subject=Fetcha%20access";
21	16
22		−const isEnterprise = (p: PlanLimits) => p.plan === "enterprise";
23		−const isFree = (p: PlanLimits) => p.plan === "free";
24		−const RECOMMENDED: Plan = "developer";
25		−
26		−function requests(p: PlanLimits) {
27		− return isEnterprise(p) ? "Custom" : formatNumber(p.monthly_requests);
28		−}
29		−function price(p: PlanLimits) {
30		− return isEnterprise(p) ? "Custom" : p.price_usd_month === 0 ? "$0" : `$${p.price_usd_month}`;
31		−}
32	17	function seconds(ms: number) {
33	18	return `${ms / 1000} s`;
34	19	}
35		−function gb(p: PlanLimits) {
36		− return isEnterprise(p) ? "Custom" : `${p.included_gb} GB`;
37		−}
38		−function overage(p: PlanLimits) {
39		− if (isEnterprise(p)) return "Custom";
40		− if (isFree(p)) return "Hard stop";
41		− return `${formatUsd(p.overage_per_1k_requests_usd)} / 1k`;
42		−}
43		−function residentialGb(p: PlanLimits) {
44		− if (isEnterprise(p)) return "Custom";
45		− if (isFree(p)) return "Included only";
46		− return `${formatUsd(p.residential_per_gb_usd)} / GB`;
47		−}
48		−function retention(p: PlanLimits) {
49		− return isEnterprise(p) ? "Custom (365 d default)" : `${p.retention_days} days`;
50		−}
51		−function networksLabel(p: PlanLimits) {
52		− return p.networks.map((n) => n[0].toUpperCase() + n.slice(1)).join(", ");
53		−}
54	20
55		−const ROWS: Array<{ label: string; hint?: string; value: (p: PlanLimits) => string }> = [
56		− { label: "Monthly price", value: (p) => (isEnterprise(p) ? "Custom" : `${price(p)} / mo`) },
57		− { label: "Included requests", hint: "Per calendar month", value: requests },
58		− { label: "Concurrency", hint: "Simultaneous in-flight requests", value: (p) => formatNumber(p.concurrency) },
59		− { label: "Max timeout", hint: "Per request", value: (p) => seconds(p.max_timeout_ms) },
60		− { label: "Retries", hint: "Maximum automatic retries per request", value: (p) => String(p.max_retries) },
61		− { label: "Log retention", hint: "Request metadata, then deleted", value: retention },
62		− { label: "Included residential bandwidth", hint: "Per month", value: gb },
63		− { label: "Overage per 1,000 requests", value: overage },
64		− { label: "Residential bandwidth overage", value: residentialGb },
65		− { label: "Network classes on plan", hint: "Residential is live; other classes are coming soon", value: networksLabel },
66		−];
	21	+export default function AccessPage() {
	22	+ const p = PLAN_LIMITS.unlimited;
	23	+ const networks = p.networks.map((n) => n[0]!.toUpperCase() + n.slice(1)).join(", ");
67	24
68		−const FAQ: Array<{ q: string; a: React.ReactNode }> = [
69		− {
70		− q: "How does billing work?",
71		− a: (
72		− <>
73		− Plans are monthly and usage-based: a flat fee includes a request quota and a residential bandwidth allowance; anything above is metered as overage. <strong>Self-serve checkout is not live yet.</strong> Every account today runs on the Free plan. To move to a paid plan, email <a href="mailto:sales@fetcha.co">sales@fetcha.co</a> and we will provision it manually and invoice you. Card checkout will open in the dashboard shortly.
74		− </>
75		− ),
76		− },
77		− {
78		− q: "What counts as a request?",
79		− a: (
80		− <>
81		− One call to <code>POST /v1/fetch</code> is one request, whether it succeeds or the target blocks it. Requests rejected before any upstream work (invalid body, disallowed URL, quota exceeded) are not counted. Session management calls (<code>/v1/sessions</code>), <code>/v1/me</code> and <code>/v1/usage</code> are free.
82		− </>
83		− ),
84		− },
85		− {
86		− q: "Do retries cost extra?",
87		− a: <>No. Automatic retries and route escalation happen inside a single request and are covered by that request. You are billed once per call regardless of how many attempts Fetcha made behind the scenes. Bandwidth consumed by attempts on residential routes does count toward your residential allowance.</>,
88		− },
89		− {
90		− q: "How is bandwidth measured?",
91		− a: <>Residential bandwidth is the sum of bytes transferred through residential exits (request plus response) for the month, rounded to the byte. Each plan includes an allowance; above it, the per-GB rate of your plan applies. Only residential traffic is metered for bandwidth today.</>,
92		− },
93		− {
94		− q: "Is geo targeting included?",
95		− a: <>Yes, on every plan. Country, region and city targeting are part of the request body and do not change the price. Availability of a specific city depends on upstream capacity at the time of the request; when a location is unavailable you receive a clear error rather than a silently different geography.</>,
96		− },
97		− {
98		− q: "How do sessions work and are they billed?",
99		− a: <>A sticky session pins one exit IP for 60 to 1,800 seconds. Creating, listing and closing sessions is free; only the fetches you make through the session count as requests. Session limits scale with your plan’s concurrency.</>,
100		− },
101		− {
102		− q: "Is browser rendering available?",
103		− a: <>Not yet. Sending <code>browser: true</code> returns <code>BROWSER_UNAVAILABLE</code> and is not billed. Browser execution and structured extraction are on the roadmap; they will be priced separately and announced in the <Link href="/changelog">changelog</Link> before launch.</>,
104		− },
105		− {
106		− q: "Can I cancel or downgrade?",
107		− a: <>Yes, at any time, with no minimum term on self-serve plans. Downgrades take effect at the end of the current billing period. Until checkout is live, email <a href="mailto:sales@fetcha.co">sales@fetcha.co</a> and we will process the change within one business day. Your request logs remain accessible for the retention window of your plan.</>,
108		− },
109		−];
	25	+ const included: Array<{ label: string; value: string; hint: string }> = [
	26	+ { label: "Requests", value: "Unlimited", hint: "No monthly quota, no overage, no request fee." },
	27	+ { label: "Concurrency", value: formatNumber(p.concurrency), hint: "Requests in flight at once, per organization." },
	28	+ { label: "Timeout", value: `up to ${seconds(p.max_timeout_ms)}`, hint: "One budget for the whole request, retries included." },
	29	+ { label: "Retries", value: `up to ${p.max_retries}`, hint: "New exit IP and alternate route on every attempt." },
	30	+ { label: "Network classes", value: "All", hint: networks + ", chosen automatically or pinned per request." },
	31	+ { label: "Managed browser", value: `${p.browser_concurrency} concurrent`, hint: "Real Chromium, same networks, automatic fallback on JS challenges." },
	32	+ { label: "Crawl jobs", value: `${formatNumber(p.crawl_max_pages)} pages / job`, hint: `${p.crawl_concurrent_jobs} jobs in parallel, sitemap discovery, Markdown output.` },
	33	+ { label: "Sessions and geo", value: "Included", hint: "Sticky sessions up to 30 minutes; country, region and city targeting." },
	34	+ { label: "Request logs", value: `${p.retention_days} days`, hint: "Every attempt, route alias, timing and cost, in the dashboard and the API." },
	35	+ ];
110	36
111		−/* ------------------------------------------------------------------ */
112		−/* Page */
113		−/* ------------------------------------------------------------------ */
	37	+ const features = ["POST /v1/fetch with HTML, text, Markdown, JSON or raw output", "Managed headless browser with screenshots and wait conditions", "Crawl and map endpoints for whole sites", "Sticky sessions and geo targeting", "Playground, request explorer and usage analytics", "JavaScript and Python SDKs", "Optional spending limits as guard-rails"];
114	38
115		−export default function PricingPage() {
116		− const plans = PLANS.map((k) => PLAN_LIMITS[k]);
117	39	return (
118	40	<div className="container-page py-14 sm:py-20">
119	41	<header className="mx-auto max-w-2xl text-center">
120		− <p className="text-[11.5px] font-semibold uppercase tracking-[0.12em] text-accent">Pricing</p>
121		− <h1 className="mt-2 text-[34px] font-semibold leading-[1.1] tracking-tight sm:text-[44px]">Start free. Pay for what you fetch.</h1>
122		− <p className="mt-4 text-[15.5px] leading-relaxed text-fg-muted">A flat monthly fee with included requests and residential bandwidth, metered overage above. Retries are never billed twice. Prices in USD.</p>
	42	+ <p className="text-[11.5px] font-semibold uppercase tracking-[0.12em] text-accent">Access</p>
	43	+ <h1 className="mt-2 text-[34px] font-semibold leading-[1.1] tracking-tight sm:text-[44px]">Private access. No plans, no meters.</h1>
	44	+ <p className="mt-4 text-[15.5px] leading-relaxed text-fg-muted">
	45	+ Fetcha is an invitation-only platform. There is a single, unlimited plan for everyone who is let in, and there is nothing to buy: no tiers, no quotas, no invoices. Accounts are created
	46	+ from an access list managed by a Fetcha administrator.
	47	+ </p>
	48	+ <div className="mt-8 flex flex-wrap items-center justify-center gap-3">
	49	+ <Button asChild size="lg">
	50	+ <Link href="/login">
	51	+ <LogIn /> Log in
	52	+ </Link>
	53	+ </Button>
	54	+ <Button asChild size="lg" variant="outline">
	55	+ <a href={REQUEST_ACCESS}>
	56	+ <Mail /> Request access
	57	+ </a>
	58	+ </Button>
	59	+ </div>
	60	+ <p className="mt-4 text-[12.5px] text-fg-subtle">Already invited? Sign up with the email address your administrator approved.</p>
123	61	</header>
124	62
125		− <Alert variant="info" title="Public preview" className="mx-auto mt-8 max-w-3xl">
126		− Self-serve upgrades and card checkout are not live yet. Every account runs on the Free plan today. To start on a paid plan, email <a className="underline underline-offset-4" href="mailto:sales@fetcha.co">sales@fetcha.co</a>.
127		− </Alert>
128		−
129		− {/* Plan columns */}
130		− <section aria-labelledby="plans-title" className="mt-12">
131		− <h2 id="plans-title" className="sr-only">
132		− Plans
133		− </h2>
134		− <ul className="grid gap-4 md:grid-cols-2 lg:grid-cols-5 lg:gap-0 lg:divide-x lg:divide-border lg:overflow-hidden lg:rounded-lg lg:border lg:border-border" role="list">
135		− {plans.map((p) => {
136		− const recommended = p.plan === RECOMMENDED;
137		− return (
138		− <li key={p.plan} className={cn("relative flex flex-col rounded-lg border border-border bg-bg p-5 lg:rounded-none lg:border-0", recommended && "bg-bg-subtle/70")}>
139		− <div className="flex items-center justify-between gap-2">
140		− <h3 className="text-[15px] font-semibold tracking-tight">{p.label}</h3>
141		− {recommended ? <Badge variant="accent">Most popular</Badge> : null}
142		− </div>
143		− <div className="mt-4 flex items-baseline gap-1">
144		− <span className="font-mono text-[30px] font-semibold tabular tracking-tight">{price(p)}</span>
145		− {!isEnterprise(p) ? <span className="text-[13px] text-fg-subtle">/ month</span> : null}
146		− </div>
147		− <p className="mt-1 text-[13px] text-fg-muted">
148		− {isEnterprise(p) ? "Volume above 5M requests, custom terms." : <><span className="font-mono tabular text-fg">{requests(p)}</span> requests / mo</>}
149		− </p>
150		−
151		− <dl className="mt-5 space-y-2 border-t border-border pt-4 text-[13px]">
152		− {[
153		− ["Concurrency", formatNumber(p.concurrency)],
154		− ["Timeout", seconds(p.max_timeout_ms)],
155		− ["Retries", String(p.max_retries)],
156		− ["Logs", retention(p)],
157		− ["Residential GB", gb(p)],
158		− ["Overage", overage(p)],
159		− ["Residential $/GB", residentialGb(p)],
160		− ].map(([k, v]) => (
161		− <div key={k} className="flex items-center justify-between gap-3">
162		− <dt className="text-fg-muted">{k}</dt>
163		− <dd className="font-mono tabular text-fg">{v}</dd>
164		− </div>
165		− ))}
166		− </dl>
167		− <p className="mt-3 text-[12px] leading-snug text-fg-subtle">Networks: {networksLabel(p)}</p>
168		−
169		− <div className="mt-auto pt-5">
170		− {isEnterprise(p) ? (
171		− <Button asChild variant="outline" className="w-full">
172		− <a href="mailto:sales@fetcha.co?subject=Fetcha%20Enterprise">Contact sales</a>
173		− </Button>
174		− ) : (
175		− <Button asChild variant={recommended ? "primary" : isFree(p) ? "default" : "outline"} className="w-full">
176		− <Link href="/signup">Start free</Link>
177		− </Button>
178		− )}
179		− {!isEnterprise(p) && !isFree(p) ? (
180		− <p className="mt-2 text-center text-[11.5px] leading-snug text-fg-subtle">
181		− Self-serve upgrades open soon — email <a className="underline underline-offset-2 hover:text-fg" href="mailto:sales@fetcha.co">sales@fetcha.co</a>
182		− </p>
183		− ) : null}
184		− {isFree(p) ? <p className="mt-2 text-center text-[11.5px] text-fg-subtle">No credit card required.</p> : null}
185		− </div>
186		− </li>
187		− );
188		− })}
189		− </ul>
	63	+ {/* Single plan */}
	64	+ <section aria-labelledby="plan-title" className="mt-16">
	65	+ <div className="overflow-hidden rounded-xl border border-border bg-bg">
	66	+ <div className="flex flex-col gap-4 border-b border-border bg-bg-subtle/60 px-6 py-6 sm:flex-row sm:items-center sm:justify-between sm:px-8">
	67	+ <div>
	68	+ <div className="flex items-center gap-2">
	69	+ <h2 id="plan-title" className="text-[22px] font-semibold tracking-tight">
	70	+ {p.label}
	71	+ </h2>
	72	+ <Badge variant="solid">private platform</Badge>
	73	+ </div>
	74	+ <p className="mt-1 text-[14px] text-fg-muted">The only plan. Same limits for every organization, enforced by the API from the configuration below.</p>
	75	+ </div>
	76	+ <div className="text-left sm:text-right">
	77	+ <div className="font-mono text-[30px] font-semibold tabular tracking-tight">$0</div>
	78	+ <div className="text-[12.5px] text-fg-subtle">no billing, by invitation</div>
	79	+ </div>
	80	+ </div>
	81	+ <dl className="grid divide-y divide-border sm:grid-cols-2 sm:divide-y-0 lg:grid-cols-3 [&>div]:border-border sm:[&>div:nth-child(n+3)]:border-t lg:[&>div:nth-child(n+3)]:border-t-0 lg:[&>div:nth-child(n+4)]:border-t sm:[&>div:nth-child(odd)]:border-r lg:[&>div:nth-child(odd)]:border-r-0 lg:[&>div:not(:nth-child(3n))]:border-r">
	82	+ {included.map((row) => (
	83	+ <div key={row.label} className="px-6 py-5 sm:px-8">
	84	+ <dt className="text-[11.5px] font-medium uppercase tracking-wide text-fg-subtle">{row.label}</dt>
	85	+ <dd className="mt-1 font-mono text-[18px] font-semibold tabular tracking-tight">{row.value}</dd>
	86	+ <dd className="mt-1 text-[12.5px] leading-snug text-fg-muted">{row.hint}</dd>
	87	+ </div>
	88	+ ))}
	89	+ </dl>
	90	+ </div>
190	91	</section>
191	92
192		− {/* Comparison table */}
193		− <section aria-labelledby="compare-title" className="mt-20">
194		− <div className="mb-5 max-w-2xl">
195		− <h2 id="compare-title" className="text-[24px] font-semibold tracking-tight">
196		− Compare plans
197		− </h2>
198		− <p className="mt-2 text-[14px] text-fg-muted">All limits are enforced by the API and visible in your dashboard. Values come from the same configuration the API uses.</p>
	93	+ {/* What is included + how to get access */}
	94	+ <section className="mt-16 grid gap-8 lg:grid-cols-2 lg:gap-12">
	95	+ <div>
	96	+ <h2 className="text-[24px] font-semibold tracking-tight">Everything is included</h2>
	97	+ <p className="mt-2 text-[14px] text-fg-muted">No feature is gated behind a tier. When a capability ships, every account gets it the same day.</p>
	98	+ <ul className="mt-5 grid gap-2.5 text-[14px]">
	99	+ {features.map((f) => (
	100	+ <li key={f} className="flex items-start gap-2.5">
	101	+ <Check className="mt-0.5 size-4 shrink-0 text-success" aria-hidden />
	102	+ <span>{f}</span>
	103	+ </li>
	104	+ ))}
	105	+ </ul>
199	106	</div>
200		− <div className="rounded-lg border border-border">
201		− <Table>
202		− <TableHeader>
203		− <TableRow className="hover:bg-transparent">
204		− <TableHead className="w-[240px]">Limit</TableHead>
205		− {plans.map((p) => (
206		− <TableHead key={p.plan} className={cn("text-right", p.plan === RECOMMENDED && "text-fg")}>
207		− {p.label}
208		− </TableHead>
209		− ))}
210		− </TableRow>
211		− </TableHeader>
212		− <TableBody>
213		− {ROWS.map((r) => (
214		− <TableRow key={r.label}>
215		− <TableCell>
216		− <div className="font-medium text-fg">{r.label}</div>
217		− {r.hint ? <div className="text-[12px] text-fg-subtle">{r.hint}</div> : null}
218		− </TableCell>
219		− {plans.map((p) => (
220		− <TableCell key={p.plan} className={cn("text-right font-mono tabular text-[13px]", p.plan === RECOMMENDED && "bg-bg-subtle/60")}>
221		− {r.value(p)}
222		− </TableCell>
223		− ))}
224		− </TableRow>
225		− ))}
226		− {[
227		− { label: "Sticky sessions", live: true },
228		− { label: "Geo targeting (country, region, city)", live: true },
229		− { label: "Playground and request logs", live: true },
230		− { label: "SDKs (JavaScript, Python — from source)", live: true },
231		− { label: "Browser execution", live: false },
232		− { label: "Structured extraction", live: false },
233		− { label: "Webhooks", live: false },
234		− { label: "Teams and invitations", live: false },
235		− ].map((f) => (
236		− <TableRow key={f.label}>
237		− <TableCell>
238		− <div className="flex items-center gap-2 font-medium text-fg">
239		− {f.label}
240		− {!f.live ? <Badge variant="outline">Coming soon</Badge> : null}
241		− </div>
242		− </TableCell>
243		− {plans.map((p) => (
244		− <TableCell key={p.plan} className={cn("text-right", p.plan === RECOMMENDED && "bg-bg-subtle/60")}>
245		− {f.live ? (
246		− <>
247		− <Check className="ml-auto size-4 text-success" aria-hidden />
248		− <span className="sr-only">Included</span>
249		− </>
250		− ) : (
251		− <>
252		− <Minus className="ml-auto size-4 text-fg-subtle" aria-hidden />
253		− <span className="sr-only">Not yet available</span>
254		− </>
255		− )}
256		− </TableCell>
257		− ))}
258		− </TableRow>
259		− ))}
260		− </TableBody>
261		− </Table>
	107	+ <div className="rounded-xl border border-border bg-bg-subtle/60 p-6 sm:p-8">
	108	+ <div className="flex items-center gap-2">
	109	+ <KeyRound className="size-4 text-accent" aria-hidden />
	110	+ <h2 className="text-[20px] font-semibold tracking-tight">How to get access</h2>
	111	+ </div>
	112	+ <ol className="mt-4 grid gap-4 text-[14px]">
	113	+ {[
	114	+ { title: "Ask for an invitation", body: <>Write to <a className="text-accent underline-offset-4 hover:underline" href={REQUEST_ACCESS}>hello@fetcha.co</a> with the email address you want to use and a sentence about what you plan to fetch.</> },
	115	+ { title: "Your email is added to the access list", body: <>A Fetcha administrator adds it and sends you an invitation. Only listed addresses can create an account; anyone else is refused at signup.</> },
	116	+ { title: "Create your account", body: <>Follow the link in the invitation (or open <Link className="text-accent underline-offset-4 hover:underline" href="/signup">/signup</Link>) using the same address. Your workspace, a default project and the Playground are ready immediately.</> },
	117	+ ].map((s, i) => (
	118	+ <li key={s.title} className="flex gap-3">
	119	+ <span className="mt-0.5 flex size-6 shrink-0 items-center justify-center rounded-full bg-fg font-mono text-[12px] font-semibold text-bg">{i + 1}</span>
	120	+ <div>
	121	+ <div className="font-medium">{s.title}</div>
	122	+ <p className="mt-0.5 text-[13.5px] leading-relaxed text-fg-muted">{s.body}</p>
	123	+ </div>
	124	+ </li>
	125	+ ))}
	126	+ </ol>
	127	+ <div className="mt-6 flex flex-wrap gap-2">
	128	+ <Button asChild>
	129	+ <Link href="/login">Log in</Link>
	130	+ </Button>
	131	+ <Button asChild variant="outline">
	132	+ <a href={REQUEST_ACCESS}>Request access</a>
	133	+ </Button>
	134	+ </div>
262	135	</div>
263		− <p className="mt-3 text-[12.5px] text-fg-subtle">Datacenter, ISP and mobile network classes are listed per plan but not yet routable; <code className="font-mono">auto</code> resolves to residential during the public preview.</p>
264	136	</section>
265	137
266		− {/* FAQ */}
267		− <section aria-labelledby="faq-title" className="mt-20 grid gap-8 lg:grid-cols-[280px_minmax(0,1fr)] lg:gap-16">
	138	+ {/* Notes */}
	139	+ <section aria-labelledby="notes-title" className="mt-16 grid gap-8 lg:grid-cols-[280px_minmax(0,1fr)] lg:gap-16">
268	140	<div>
269		− <h2 id="faq-title" className="text-[24px] font-semibold tracking-tight">
270		− Questions
	141	+ <h2 id="notes-title" className="text-[24px] font-semibold tracking-tight">
	142	+ Good to know
271	143	</h2>
272	144	<p className="mt-2 text-[14px] text-fg-muted">
273		− Anything else? Write to <a className="text-accent underline-offset-4 hover:underline" href="mailto:support@fetcha.co">support@fetcha.co</a>.
	145	+ Questions? Write to <a className="text-accent underline-offset-4 hover:underline" href="mailto:support@fetcha.co">support@fetcha.co</a>.
274	146	</p>
275	147	</div>
276	148	<div className="divide-y divide-border rounded-lg border border-border">
277		− {FAQ.map((f) => (
	149	+ {[
	150	+ { q: "Why is there no pricing?", a: "Fetcha runs as a private platform for invited teams. Access is granted by an administrator rather than sold, so there is no checkout, no card on file and no invoice. The dashboard's “Plan & access” page shows your usage for information only." },
	151	+ { q: "Are the limits really unlimited?", a: `There is no monthly request quota and no overage. Operational limits still apply to keep the platform healthy: ${formatNumber(p.concurrency)} concurrent requests per organization, a ${seconds(p.max_timeout_ms)} maximum timeout, up to ${p.max_retries} retries, ${p.browser_concurrency} concurrent browser renders and ${formatNumber(p.crawl_max_pages)} pages per crawl job. Burst rate limits protect the API from runaway loops.` },
	152	+ { q: "What are spending limits, then?", a: "Optional guard-rails you set yourself, per organization or per project. They are based on an internal cost estimate and stop requests with USAGE_LIMIT_REACHED when a hard limit is reached. Nothing is charged." },
	153	+ { q: "Can I invite my teammates?", a: "Ask your Fetcha administrator to add their email addresses to the access list; each of them receives an invitation and creates their own account. Shared organizations and roles are not available yet." },
	154	+ { q: "What happens if I sign up with an email that was not invited?", a: "Signup is refused with the message “This email is not on the access list. Ask your Fetcha administrator to invite you.” No account is created." },
	155	+ ].map((f) => (
278	156	<details key={f.q} className="group px-5 py-4 open:bg-bg-subtle/40">
279	157	<summary className="flex cursor-pointer list-none items-center justify-between gap-4 text-[15px] font-medium tracking-tight marker:content-none [&::-webkit-details-marker]:hidden">
280	158	{f.q}
		`@@ -282,24 +160,24 @@ export default function PricingPage() {`
282	160	+
283	161	</span>
284	162	</summary>
285		− <div className="mt-3 max-w-2xl text-[14px] leading-relaxed text-fg-muted [&_a]:text-accent [&_a]:underline-offset-4 hover:[&_a]:underline [&_code]:rounded-[4px] [&_code]:border [&_code]:border-border [&_code]:bg-bg-muted [&_code]:px-1.5 [&_code]:py-0.5 [&_code]:font-mono [&_code]:text-[0.85em] [&_code]:text-fg [&_strong]:font-semibold [&_strong]:text-fg">{f.a}</div>
	163	+ <div className="mt-3 max-w-2xl text-[14px] leading-relaxed text-fg-muted">{f.a}</div>
286	164	</details>
287	165	))}
288	166	</div>
289	167	</section>
290	168
291	169	<section className="mt-20 rounded-xl border border-border bg-bg-subtle/60 px-6 py-12 text-center">
292		− <h2 className="text-[26px] font-semibold tracking-tight">Try it on the Free plan today.</h2>
293		− <p className="mx-auto mt-3 max-w-lg text-[14.5px] text-fg-muted">1,000 requests a month, sticky sessions, geo targeting and full request logs. Upgrade when you outgrow it.</p>
	170	+ <h2 className="text-[26px] font-semibold tracking-tight">Invited? Your workspace is waiting.</h2>
	171	+ <p className="mx-auto mt-3 max-w-lg text-[14.5px] text-fg-muted">Log in with the address your administrator approved, or ask us for access.</p>
294	172	<div className="mt-6 flex flex-wrap items-center justify-center gap-3">
295	173	<Button asChild size="lg">
296		− <Link href="/signup">Start free</Link>
	174	+ <Link href="/login">Log in</Link>
297	175	</Button>
298	176	<Button asChild size="lg" variant="outline">
299		− <a href="mailto:sales@fetcha.co">Talk to sales</a>
	177	+ <a href={REQUEST_ACCESS}>Request access</a>
300	178	</Button>
301	179	</div>
302		− <p className="mt-4 text-[12.5px] text-fg-subtle">No credit card. 1,000 free requests a month.</p>
	180	+ <p className="mt-4 text-[12.5px] text-fg-subtle">Private platform. No plans, no meters, no billing.</p>
303	181	</section>
304	182	</div>
305	183	);

added apps/web/src/app/admin/access/loading.tsx +5 −0

		`@@ -0,0 +1,5 @@`
	1	+import { AdminLoading } from "@/components/admin/primitives";
	2	+
	3	+export default function Loading() {
	4	+ return <AdminLoading stats={4} rows={6} />;
	5	+}

added apps/web/src/app/admin/access/page.tsx +159 −0

		`@@ -0,0 +1,159 @@`
	1	+import Link from "next/link";
	2	+import { KeyRound } from "lucide-react";
	3	+import { requireAdmin } from "@/lib/session";
	4	+import { getAllowlistCounts, listAllowlist, str } from "@/lib/queries/admin";
	5	+import { adminEmails } from "@/lib/access";
	6	+import { formatNumber } from "@/lib/format";
	7	+import { PageHeader } from "@/components/ui/page-header";
	8	+import { Badge } from "@/components/ui/badge";
	9	+import { Alert } from "@/components/ui/alert";
	10	+import { EmptyState } from "@/components/ui/empty-state";
	11	+import { Stat, StatGrid } from "@/components/ui/stat";
	12	+import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table";
	13	+import { DateCell, Mono, Panel, SearchForm } from "@/components/admin/primitives";
	14	+import { AddEmailsDialog, RemoveAllowButton, ResendInviteButton } from "@/components/admin/access-controls";
	15	+
	16	+export const dynamic = "force-dynamic";
	17	+
	18	+function StatusBadgeFor({ status, banned }: { status: "account" \| "invited" \| "pending"; banned: boolean \| null }) {
	19	+ if (status === "account") {
	20	+ return banned ? (
	21	+ <Badge variant="danger" dot>
	22	+ account banned
	23	+ </Badge>
	24	+ ) : (
	25	+ <Badge variant="success" dot>
	26	+ account created
	27	+ </Badge>
	28	+ );
	29	+ }
	30	+ if (status === "invited")
	31	+ return (
	32	+ <Badge variant="info" dot>
	33	+ invited
	34	+ </Badge>
	35	+ );
	36	+ return (
	37	+ <Badge variant="outline" dot>
	38	+ pending
	39	+ </Badge>
	40	+ );
	41	+}
	42	+
	43	+export default async function AdminAccessPage({ searchParams }: { searchParams: Promise<Record<string, string \| string[] \| undefined>> }) {
	44	+ await requireAdmin();
	45	+ const sp = await searchParams;
	46	+ const q = str(sp.q);
	47	+ const [rows, counts] = await Promise.all([listAllowlist({ q: q \|\| undefined }), getAllowlistCounts()]);
	48	+
	49	+ return (
	50	+ <>
	51	+ <PageHeader
	52	+ eyebrow="Accounts"
	53	+ title="Access"
	54	+ description="Fetcha is invitation-only. Only emails on this list (plus the ADMIN_EMAILS environment variable) can create an account. Removing an entry does not affect an existing account — ban the user instead."
	55	+ actions={
	56	+ <>
	57	+ <SearchForm action="/admin/access" q={q} placeholder="email or note" />
	58	+ <AddEmailsDialog />
	59	+ </>
	60	+ }
	61	+ />
	62	+
	63	+ <StatGrid cols={4} className="mb-6">
	64	+ <Stat label="On the list" value={formatNumber(counts.total)} hint="allowlisted emails" />
	65	+ <Stat label="Accounts created" value={formatNumber(counts.accounts)} hint="invitation used" />
	66	+ <Stat label="Invited" value={formatNumber(counts.invited)} hint="email sent, no account yet" />
	67	+ <Stat label="Pending" value={formatNumber(counts.pending)} hint="listed, no email sent" />
	68	+ </StatGrid>
	69	+
	70	+ {adminEmails.length ? (
	71	+ <Alert variant="info" title="Administrators from the environment" className="mb-4">
	72	+ {adminEmails.length === 1 ? "This address" : "These addresses"} can always sign up and {adminEmails.length === 1 ? "is" : "are"} created with the admin role, whether or not {adminEmails.length === 1 ? "it is" : "they are"} listed
	73	+ below: {adminEmails.map((e, i) => (
	74	+ <span key={e}>
	75	+ {i > 0 ? ", " : null}
	76	+ <Mono>{e}</Mono>
	77	+ </span>
	78	+ ))}
	79	+ . Run <Mono>pnpm db:seed</Mono> to add them to the list and promote existing accounts.
	80	+ </Alert>
	81	+ ) : (
	82	+ <Alert variant="warning" title="ADMIN_EMAILS is not set" className="mb-4">
	83	+ Without it, only addresses on this list can sign up and nobody is promoted to admin automatically. Set <Mono>ADMIN_EMAILS</Mono> in the web environment and run <Mono>pnpm db:seed</Mono>.
	84	+ </Alert>
	85	+ )}
	86	+
	87	+ <Panel flush>
	88	+ {rows.length === 0 ? (
	89	+ <div className="p-4">
	90	+ <EmptyState
	91	+ icon={KeyRound}
	92	+ title={q ? "No entries match" : "The access list is empty"}
	93	+ description={q ? "Try a partial email or a different spelling." : "Add the emails of the people you want to let in. Optionally send them an invitation email with a link to the signup page."}
	94	+ action={q ? undefined : <AddEmailsDialog />}
	95	+ compact
	96	+ />
	97	+ </div>
	98	+ ) : (
	99	+ <Table>
	100	+ <TableHeader>
	101	+ <TableRow>
	102	+ <TableHead>Email</TableHead>
	103	+ <TableHead>Note</TableHead>
	104	+ <TableHead>Status</TableHead>
	105	+ <TableHead>Invited by</TableHead>
	106	+ <TableHead className="text-right">Invited</TableHead>
	107	+ <TableHead className="text-right">Added</TableHead>
	108	+ <TableHead className="text-right">Actions</TableHead>
	109	+ </TableRow>
	110	+ </TableHeader>
	111	+ <TableBody>
	112	+ {rows.map((r) => (
	113	+ <TableRow key={r.email}>
	114	+ <TableCell>
	115	+ {r.userId ? (
	116	+ <Link href={`/admin/users/${r.userId}`} className="font-medium hover:underline">
	117	+ {r.email}
	118	+ </Link>
	119	+ ) : (
	120	+ <span className="font-medium">{r.email}</span>
	121	+ )}
	122	+ {r.accountEmail && r.accountEmail.toLowerCase() !== r.email ? <div className="text-[11.5px] text-fg-subtle">account now {r.accountEmail}</div> : null}
	123	+ </TableCell>
	124	+ <TableCell className="max-w-[280px] truncate text-fg-muted" title={r.note ?? undefined}>
	125	+ {r.note ?? <span className="text-fg-subtle">—</span>}
	126	+ </TableCell>
	127	+ <TableCell>
	128	+ <StatusBadgeFor status={r.status} banned={r.accountBanned} />
	129	+ </TableCell>
	130	+ <TableCell className="max-w-[220px] truncate text-fg-muted">
	131	+ {r.invitedByUserId ? (
	132	+ <Link href={`/admin/users/${r.invitedByUserId}`} className="hover:underline">
	133	+ {r.invitedByEmail ?? r.invitedByUserId}
	134	+ </Link>
	135	+ ) : (
	136	+ <span className="text-fg-subtle">seed</span>
	137	+ )}
	138	+ </TableCell>
	139	+ <TableCell className="text-right">
	140	+ <DateCell value={r.invitedAt} />
	141	+ </TableCell>
	142	+ <TableCell className="text-right">
	143	+ <DateCell value={r.createdAt} />
	144	+ </TableCell>
	145	+ <TableCell className="text-right">
	146	+ <div className="inline-flex items-center justify-end gap-1">
	147	+ {r.status !== "account" ? <ResendInviteButton email={r.email} invitedBefore={Boolean(r.invitedAt)} /> : null}
	148	+ <RemoveAllowButton email={r.email} hasAccount={r.status === "account"} />
	149	+ </div>
	150	+ </TableCell>
	151	+ </TableRow>
	152	+ ))}
	153	+ </TableBody>
	154	+ </Table>
	155	+ )}
	156	+ </Panel>
	157	+ </>
	158	+ );
	159	+}

modified apps/web/src/app/admin/billing/page.tsx +4 −4

		`@@ -20,11 +20,11 @@ export default async function AdminBillingPage() {`
20	20
21	21	return (
22	22	<>
23		− <PageHeader eyebrow="Revenue" title="Billing" description="Plan distribution and MRR estimate from plan list prices. Stripe checkout and invoicing are not connected yet." />
	23	+ <PageHeader eyebrow="Revenue" title="Billing" description="Private platform: a single Unlimited plan with no list price, so MRR is always zero. Usage revenue reflects metered price_usd only. Stripe checkout and invoicing are not connected." />
24	24
25	25	{!b.stripeConfigured ? (
26	26	<Alert variant="info" title="Stripe is not connected" className="mb-4">
27		− `STRIPE_SECRET_KEY` is not set. Plans are changed manually from the organization page; no charges are made. Subscriptions and billing events below will populate once the Stripe integration ships.
	27	+ `STRIPE_SECRET_KEY` is not set and there is nothing to bill: every organization is on the Unlimited plan (private platform, no billing). Subscriptions and billing events below stay empty unless a Stripe integration is added later.
28	28	</Alert>
29	29	) : null}
30	30
		`@@ -53,7 +53,7 @@ export default async function AdminBillingPage() {`
53	53	<PlanBadge plan={d.plan} />
54	54	</TableCell>
55	55	<TableCell className={numCell}>{formatNumber(d.orgs)}</TableCell>
56		− <TableCell className={numCell}>{d.plan === "enterprise" ? "custom" : formatUsd(d.price)}</TableCell>
	56	+ <TableCell className={numCell}>{formatUsd(d.price)}</TableCell>
57	57	<TableCell className={numCell}>{formatUsd(d.mrr)}</TableCell>
58	58	<TableCell className="text-right">
59	59	<Link href={`/admin/organizations?plan=${d.plan}`} className="text-[12.5px] text-accent hover:underline">
		`@@ -65,7 +65,7 @@ export default async function AdminBillingPage() {`
65	65	{b.unknown.map((u) => (
66	66	<TableRow key={u.plan}>
67	67	<TableCell>
68		− <Badge variant="danger">{u.plan}</Badge> <span className="text-[12px] text-fg-muted">unknown plan value</span>
	68	+ <Badge variant="danger">{u.plan}</Badge> <span className="text-[12px] text-fg-muted">legacy plan value — treated as Unlimited; run `pnpm db:seed` to normalize</span>
69	69	</TableCell>
70	70	<TableCell className={numCell}>{u.n}</TableCell>
71	71	<TableCell className={numCell}>—</TableCell>

modified apps/web/src/app/admin/organizations/[id]/page.tsx +6 −3

		`@@ -2,7 +2,7 @@ import Link from "next/link";`
2	2	import { notFound } from "next/navigation";
3	3	import { ArrowLeft } from "lucide-react";
4	4	import { requireAdmin } from "@/lib/session";
5		−import { getOrganizationDetail, PLAN_OPTIONS } from "@/lib/queries/admin";
	5	+import { getOrganizationDetail } from "@/lib/queries/admin";
6	6	import { formatBytes, formatDate, formatMs, formatNumber, formatPercent, formatUsd } from "@/lib/format";
7	7	import { PageHeader } from "@/components/ui/page-header";
8	8	import { Badge, StatusBadge } from "@/components/ui/badge";
		`@@ -12,7 +12,7 @@ import { Table, TableBody, TableCell, TableEmpty, TableHead, TableHeader, TableR`
12	12	import { CopyButton } from "@/components/ui/copy-button";
13	13	import { Alert } from "@/components/ui/alert";
14	14	import { DateCell, KV, MarginText, Mono, NetworkBadge, Panel, PlanBadge, numCell } from "@/components/admin/primitives";
15		−import { OrgLimitsForm, OrgPlanSelect, OrgProviderVisibilitySwitch, OrgSuspendSwitch } from "@/components/admin/org-actions";
	15	+import { OrgLimitsForm, OrgProviderVisibilitySwitch, OrgSuspendSwitch } from "@/components/admin/org-actions";
16	16
17	17	export const dynamic = "force-dynamic";
18	18
		`@@ -73,7 +73,10 @@ export default async function AdminOrganizationDetail({ params }: { params: Prom`
73	73	<div className="grid gap-5">
74	74	<div className="grid gap-1.5">
75	75	<div className="text-[13px] font-medium">Plan</div>
76		− <OrgPlanSelect orgId={org.id} plan={org.plan} options={PLAN_OPTIONS} />
	76	+ <div className="flex items-center gap-2">
	77	+ <Badge variant="solid">Unlimited</Badge>
	78	+ <span className="text-[12.5px] text-fg-muted">private platform — plans are not configurable</span>
	79	+ </div>
77	80	</div>
78	81	<OrgProviderVisibilitySwitch orgId={org.id} enabled={org.providerVisibility} />
79	82	<OrgSuspendSwitch orgId={org.id} suspended={org.suspended} />

modified apps/web/src/app/dashboard/billing/page.tsx +86 −143

		`@@ -1,83 +1,70 @@`
1	1	import Link from "next/link";
2		−import { ArrowRight, Check, CreditCard, Receipt } from "lucide-react";
3		−import { PLAN_LIMITS, PLANS, type Plan } from "@fetcha/core";
	2	+import { ArrowRight, Check, KeyRound, ShieldCheck } from "lucide-react";
	3	+import { PLAN_LIMITS, isUnlimited, normalizePlan } from "@fetcha/core";
4	4	import { getWorkspace } from "@/lib/session";
5	5	import { monthlyUsage } from "@/lib/queries/account";
6		−import { formatNumber, formatPercent, formatUsd, formatDateOnly } from "@/lib/format";
7		−import { cn } from "@/lib/utils";
8		−import { PageHeader, SectionTitle } from "@/components/ui/page-header";
	6	+import { formatBytes, formatNumber, formatUsd, formatDateOnly } from "@/lib/format";
	7	+import { PageHeader } from "@/components/ui/page-header";
9	8	import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card";
10	9	import { Badge } from "@/components/ui/badge";
11	10	import { Button } from "@/components/ui/button";
12		−import { EmptyState } from "@/components/ui/empty-state";
13	11	import { Alert } from "@/components/ui/alert";
14		−import { UpgradeDialog } from "@/components/dashboard/billing/upgrade-dialog";
15	12
16	13	export const dynamic = "force-dynamic";
17	14
18		−export const metadata = { title: "Billing · Fetcha" };
	15	+export const metadata = { title: "Plan & access · Fetcha" };
19	16
20		−function planOf(plan: string): Plan {
21		− return (PLANS as readonly string[]).includes(plan) ? (plan as Plan) : "free";
22		−}
23		−
24		−function priceLabel(p: Plan): string {
25		− const l = PLAN_LIMITS[p];
26		− if (p === "enterprise") return "Custom pricing";
27		− if (l.price_usd_month === 0) return "$0 / month";
28		− return `$${l.price_usd_month} / month`;
29		−}
30		−
31		−function limitLines(p: Plan): string[] {
32		− const l = PLAN_LIMITS[p];
33		− const unlimited = l.monthly_requests >= Number.MAX_SAFE_INTEGER;
34		− const lines = [
35		− unlimited ? "Unlimited requests" : `${formatNumber(l.monthly_requests)} requests / month`,
36		− `${formatNumber(l.concurrency)} concurrent requests`,
37		− `${l.max_timeout_ms / 1000} s max timeout · ${l.max_retries} retries`,
38		− `${l.retention_days}-day request logs`,
39		− ];
40		− if (l.included_gb > 0) lines.push(`${l.included_gb} GB residential bandwidth included`);
41		− if (l.overage_per_1k_requests_usd > 0) lines.push(`$${l.overage_per_1k_requests_usd.toFixed(2)} per extra 1k requests`);
42		− if (l.residential_per_gb_usd > 0) lines.push(`$${l.residential_per_gb_usd.toFixed(2)} per extra residential GB`);
43		− if (p === "enterprise") lines.push("Custom bandwidth and overage", "SLA, invoicing, dedicated support");
44		− return lines;
	17	+function seconds(ms: number) {
	18	+ return `${ms / 1000} s`;
45	19	}
46	20
47	21	export default async function BillingPage() {
48	22	const ws = await getWorkspace();
49		− const plan = planOf(ws.organization.plan);
50		− const limits = PLAN_LIMITS[plan];
51		− const usage = await monthlyUsage(ws.organization.id);
52		− const unlimited = limits.monthly_requests >= Number.MAX_SAFE_INTEGER;
53		− const pct = unlimited ? null : Math.min(100, (usage.requests / limits.monthly_requests) * 100);
	23	+ const limits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
	24	+ const unlimited = isUnlimited(limits);
54	25	const now = new Date();
55		− const periodEnd = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() + 1, 1));
56		− const planIndex = PLANS.indexOf(plan);
	26	+ const monthStart = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 1));
	27	+ const nextMonth = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() + 1, 1));
	28	+ const usage = await monthlyUsage(ws.organization.id);
	29	+
	30	+ const included: Array<{ label: string; detail?: string }> = [
	31	+ { label: unlimited ? "Unlimited requests" : `${formatNumber(limits.monthly_requests)} requests / month`, detail: "no monthly quota, no overage" },
	32	+ { label: `${formatNumber(limits.concurrency)} concurrent requests`, detail: "per organization, all projects together" },
	33	+ { label: `${seconds(limits.max_timeout_ms)} maximum timeout`, detail: `up to ${limits.max_retries} automatic retries per request` },
	34	+ { label: "All network classes", detail: limits.networks.map((n) => n[0]!.toUpperCase() + n.slice(1)).join(", ") },
	35	+ { label: "Managed browser rendering", detail: `${limits.browser_concurrency} concurrent renders, routed through the same networks` },
	36	+ { label: `Crawl jobs up to ${formatNumber(limits.crawl_max_pages)} pages`, detail: `${limits.crawl_concurrent_jobs} jobs in parallel, sitemap discovery and Markdown output` },
	37	+ { label: "Sticky sessions and geo targeting", detail: "country, region and city on every request" },
	38	+ { label: `${limits.retention_days}-day request logs`, detail: "full metadata, attempts and timings" },
	39	+ ];
57	40
58	41	return (
59	42	<div className="grid gap-8">
60		− <PageHeader title="Billing" description="Your plan, usage against its limits, and invoices. Checkout and card payments open at the end of the public preview." />
	43	+ <PageHeader
	44	+ title="Plan & access"
	45	+ description="Fetcha is a private platform. There is a single plan, nothing to upgrade and nothing to pay: no invoices, no payment method, no meters. Usage below is informational."
	46	+ />
61	47
62	48	<div className="grid gap-6 lg:grid-cols-[1fr_360px]">
63	49	<Card>
64	50	<CardHeader>
65	51	<div className="flex items-center justify-between gap-3">
66		− <CardTitle>Current plan</CardTitle>
67		− <Badge variant="accent" dot>
	52	+ <CardTitle>Unlimited — private platform</CardTitle>
	53	+ <Badge variant="solid" dot>
68	54	{limits.label}
69	55	</Badge>
70	56	</div>
71		− <CardDescription>
72		− {priceLabel(plan)} · usage period resets {formatDateOnly(periodEnd)}. Included with every plan: auto routing, retries with new IPs, sticky sessions and the Playground.
73		− </CardDescription>
	57	+ <CardDescription>Every organization on this platform gets the same limits. They are enforced by the API and shown here from the same configuration the API uses.</CardDescription>
74	58	</CardHeader>
75	59	<CardContent>
76		− <ul className="grid gap-2 text-[13.5px] sm:grid-cols-2">
77		− {limitLines(plan).map((line) => (
78		− <li key={line} className="flex items-start gap-2">
	60	+ <ul className="grid gap-3 text-[13.5px] sm:grid-cols-2">
	61	+ {included.map((line) => (
	62	+ <li key={line.label} className="flex items-start gap-2">
79	63	<Check className="mt-0.5 size-4 shrink-0 text-success" aria-hidden />
80		− <span className="tabular">{line}</span>
	64	+ <span>
	65	+ <span className="font-medium tabular">{line.label}</span>
	66	+ {line.detail ? <span className="block text-[12.5px] text-fg-muted">{line.detail}</span> : null}
	67	+ </span>
81	68	</li>
82	69	))}
83	70	</ul>
		`@@ -87,29 +74,30 @@ export default async function BillingPage() {`
87	74	<Card>
88	75	<CardHeader>
89	76	<CardTitle>Usage this month</CardTitle>
90		− <CardDescription>Across all projects of {ws.organization.name}.</CardDescription>
	77	+ <CardDescription>
	78	+ Across all projects of {ws.organization.name}, since {formatDateOnly(monthStart)}. Resets {formatDateOnly(nextMonth)}.
	79	+ </CardDescription>
91	80	</CardHeader>
92		− <CardContent className="grid gap-4">
93		− <div className="grid gap-1.5">
94		− <div className="flex items-baseline justify-between text-[13px]">
95		− <span className="text-fg-muted">Requests</span>
96		− <span className="font-mono tabular">
97		− {formatNumber(usage.requests)} <span className="text-fg-subtle">/ {unlimited ? "∞" : formatNumber(limits.monthly_requests)}</span>
98		− </span>
99		− </div>
100		− <div className="h-1.5 overflow-hidden rounded-full bg-bg-muted" role="progressbar" aria-valuenow={pct ?? undefined} aria-valuemin={0} aria-valuemax={100} aria-label="Requests used">
101		− {pct !== null ? <div className={cn("h-full", pct >= 90 ? "bg-danger" : pct >= 70 ? "bg-warning" : "bg-accent")} style={{ width: `${pct}%` }} /> : null}
102		− </div>
103		− <div className="text-[11.5px] text-fg-subtle">{pct !== null ? `${formatPercent(pct, 0)} of plan allowance` : "no cap on this plan"}</div>
	81	+ <CardContent className="grid gap-3 text-[13px]">
	82	+ <div className="flex items-baseline justify-between">
	83	+ <span className="text-fg-muted">Requests</span>
	84	+ <span className="font-mono tabular">
	85	+ {formatNumber(usage.requests)} <span className="text-fg-subtle">/ ∞</span>
	86	+ </span>
	87	+ </div>
	88	+ <div className="flex items-baseline justify-between">
	89	+ <span className="text-fg-muted">Successful</span>
	90	+ <span className="font-mono tabular">{formatNumber(usage.successful)}</span>
	91	+ </div>
	92	+ <div className="flex items-baseline justify-between">
	93	+ <span className="text-fg-muted">Bandwidth</span>
	94	+ <span className="font-mono tabular">{formatBytes(usage.bytes)}</span>
104	95	</div>
105		− <div className="flex items-baseline justify-between border-t border-border pt-3 text-[13px]">
	96	+ <div className="flex items-baseline justify-between border-t border-border pt-3">
106	97	<span className="text-fg-muted">Metered spend</span>
107	98	<span className="font-mono tabular">{formatUsd(usage.spendUsd)}</span>
108	99	</div>
109		− <div className="flex items-baseline justify-between text-[13px]">
110		− <span className="text-fg-muted">Plan fee</span>
111		− <span className="font-mono tabular">{plan === "enterprise" ? "custom" : formatUsd(limits.price_usd_month)}</span>
112		− </div>
	100	+ <p className="text-[11.5px] leading-snug text-fg-subtle">Spend is an internal cost estimate used only for your optional spending limits. Nothing is invoiced.</p>
113	101	<Link href="/dashboard/usage" className="inline-flex items-center gap-1 text-[12.5px] text-fg-muted underline-offset-4 hover:text-fg hover:underline">
114	102	Detailed usage <ArrowRight className="size-3.5" />
115	103	</Link>
		`@@ -117,84 +105,39 @@ export default async function BillingPage() {`
117	105	</Card>
118	106	</div>
119	107
120		− <section>
121		− <SectionTitle right={<span className="text-[12px] font-normal normal-case tracking-normal text-fg-subtle">Prices in USD, billed monthly</span>}>Plans</SectionTitle>
122		− <div className="grid gap-3 md:grid-cols-2 xl:grid-cols-5">
123		− {PLANS.map((p, i) => {
124		− const l = PLAN_LIMITS[p];
125		− const current = p === plan;
126		− const downgrade = i < planIndex;
127		− return (
128		− <div key={p} className={cn("flex flex-col rounded-lg border bg-bg-elevated p-4 shadow-xs", current ? "border-accent ring-1 ring-accent/30" : "border-border")}>
129		− <div className="flex items-center justify-between gap-2">
130		− <h3 className="text-[14px] font-semibold tracking-tight">{l.label}</h3>
131		− {current ? <Badge variant="accent">Current</Badge> : null}
132		− </div>
133		− <div className="mt-2 font-mono text-[20px] font-semibold tabular leading-none">{p === "enterprise" ? "Custom" : `$${l.price_usd_month}`}</div>
134		− <div className="mt-1 text-[11.5px] text-fg-subtle">{p === "enterprise" ? "annual contract" : "per month"}</div>
135		− <ul className="mt-4 grid flex-1 gap-1.5 text-[12.5px] text-fg-muted">
136		− {limitLines(p)
137		− .slice(0, 5)
138		− .map((line) => (
139		− <li key={line} className="tabular">
140		− {line}
141		− </li>
142		− ))}
143		− </ul>
144		− <div className="mt-4">
145		− {current ? (
146		− <Button variant="secondary" size="sm" className="w-full" disabled>
147		− Your plan
148		− </Button>
149		− ) : downgrade ? (
150		− <Button asChild variant="ghost" size="sm" className="w-full">
151		− <a href={`mailto:sales@fetcha.co?subject=${encodeURIComponent(`Downgrade to ${l.label}`)}`}>Request downgrade</a>
152		− </Button>
153		− ) : (
154		− <UpgradeDialog planLabel={l.label} priceLabel={priceLabel(p)} currentPlanLabel={limits.label} variant={i === planIndex + 1 ? "primary" : "outline"} />
155		− )}
156		− </div>
157		− </div>
158		− );
159		− })}
160		− </div>
161		− </section>
162		−
163	108	<div className="grid gap-6 lg:grid-cols-2">
164		− <section>
165		− <SectionTitle>Invoices</SectionTitle>
166		− <EmptyState compact icon={Receipt} title="No invoices yet" description="Billing is not enabled during the public preview. Invoices and receipts will appear here once checkout opens." />
167		− </section>
168		− <section className="grid gap-4">
169		− <div>
170		− <SectionTitle>Payment method</SectionTitle>
171		− <Card>
172		− <CardContent className="flex items-center gap-3 pt-5">
173		− <div className="flex size-9 items-center justify-center rounded-md border border-border bg-bg">
174		− <CreditCard className="size-4 text-fg-muted" aria-hidden />
175		− </div>
176		− <div className="min-w-0 flex-1">
177		− <div className="text-[13.5px] font-medium">Not configured</div>
178		− <div className="text-[12.5px] text-fg-muted">Card checkout opens soon. You will be asked for a payment method before any paid plan is charged.</div>
179		− </div>
180		− <Button variant="outline" size="sm" disabled>
181		− Add card
182		− </Button>
183		− </CardContent>
184		− </Card>
185		− </div>
186		− <Alert
187		− variant="info"
188		− title="Spending limits"
189		− action={
	109	+ <Alert
	110	+ variant="info"
	111	+ title="Spending limits"
	112	+ action={
	113	+ <Button asChild variant="outline" size="xs">
	114	+ <Link href="/dashboard/settings">Settings</Link>
	115	+ </Button>
	116	+ }
	117	+ >
	118	+ Optional guard-rails, not billing: cap the estimated monthly spend for the whole organization{ws.organization.hardLimitUsd ? ` (currently ${formatUsd(ws.organization.hardLimitUsd)} hard limit)` : ""} or per project. Soft limits email you; hard limits stop requests with USAGE_LIMIT_REACHED.
	119	+ </Alert>
	120	+ <Alert
	121	+ variant="success"
	122	+ title="Invitation-only access"
	123	+ action={
	124	+ ws.isAdmin ? (
	125	+ <Button asChild variant="outline" size="xs">
	126	+ <Link href="/admin/access">
	127	+ <KeyRound className="size-3.5" /> Manage access
	128	+ </Link>
	129	+ </Button>
	130	+ ) : (
190	131	<Button asChild variant="outline" size="xs">
191		− <Link href="/dashboard/settings">Settings</Link>
	132	+ <a href="mailto:hello@fetcha.co?subject=Fetcha%20access">
	133	+ <ShieldCheck className="size-3.5" /> Contact
	134	+ </a>
192	135	</Button>
193		− }
194		− >
195		− Cap monthly spend for the whole organization{ws.organization.hardLimitUsd ? ` (currently ${formatUsd(ws.organization.hardLimitUsd)} hard limit)` : ""} or per project. Hard limits stop requests with USAGE_LIMIT_REACHED.
196		− </Alert>
197		− </section>
	136	+ )
	137	+ }
	138	+ >
	139	+ Accounts are created by invitation. To bring a teammate in, ask your Fetcha administrator to add their email to the access list; they will receive an invitation to sign up.
	140	+ </Alert>
198	141	</div>
199	142	</div>
200	143	);

added apps/web/src/app/dashboard/crawls/[id]/loading.tsx +28 −0

		`@@ -0,0 +1,28 @@`
	1	+import { Skeleton, TableSkeleton } from "@/components/ui/skeleton";
	2	+
	3	+export default function CrawlDetailLoading() {
	4	+ return (
	5	+ <div className="flex flex-col gap-5" aria-busy="true" aria-label="Loading crawl">
	6	+ <Skeleton className="h-3.5 w-20" />
	7	+ <div className="flex items-start justify-between gap-3">
	8	+ <div className="flex flex-col gap-2">
	9	+ <Skeleton className="h-3 w-16" />
	10	+ <Skeleton className="h-7 w-72 max-w-full" />
	11	+ <Skeleton className="h-3.5 w-[380px] max-w-full" />
	12	+ </div>
	13	+ <Skeleton className="h-8 w-24" />
	14	+ </div>
	15	+ <div className="grid gap-px overflow-hidden rounded-lg border border-border bg-border sm:grid-cols-3 lg:grid-cols-6">
	16	+ {Array.from({ length: 6 }).map((_, i) => (
	17	+ <div key={i} className="flex flex-col gap-2 bg-bg-elevated px-5 py-4">
	18	+ <Skeleton className="h-3 w-20" />
	19	+ <Skeleton className="h-6 w-14" />
	20	+ </div>
	21	+ ))}
	22	+ </div>
	23	+ <div className="rounded-lg border border-border bg-bg-elevated">
	24	+ <TableSkeleton rows={10} cols={8} />
	25	+ </div>
	26	+ </div>
	27	+ );
	28	+}

added apps/web/src/app/dashboard/crawls/[id]/page.tsx +344 −0

		`@@ -0,0 +1,344 @@`
	1	+import type { Metadata } from "next";
	2	+import Link from "next/link";
	3	+import { notFound } from "next/navigation";
	4	+import { ArrowLeft, ChevronDown, RefreshCw } from "lucide-react";
	5	+import { getWorkspace } from "@/lib/session";
	6	+import { internalApi, InternalApiError, type CrawlJob, type CrawlPage, type CrawlPagesPage } from "@/lib/api";
	7	+import { formatBytes, formatDate, formatMs, formatNumber, timeAgo } from "@/lib/format";
	8	+import { PageHeader } from "@/components/ui/page-header";
	9	+import { Alert } from "@/components/ui/alert";
	10	+import { Badge } from "@/components/ui/badge";
	11	+import { Button } from "@/components/ui/button";
	12	+import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
	13	+import { CopyButton } from "@/components/ui/copy-button";
	14	+import { EmptyState } from "@/components/ui/empty-state";
	15	+import { Stat, StatGrid } from "@/components/ui/stat";
	16	+import { Table, TableBody, TableCell, TableEmpty, TableHead, TableHeader, TableRow } from "@/components/ui/table";
	17	+import { CrawlPageStatusBadge, CrawlStatusBadge } from "@/components/dashboard/crawls/crawl-status-badge";
	18	+import { CancelCrawlButton } from "@/components/dashboard/crawls/cancel-crawl-button";
	19	+
	20	+export const dynamic = "force-dynamic";
	21	+export const metadata: Metadata = { title: "Crawl" };
	22	+
	23	+const PAGE_SIZE = 100;
	24	+const PREVIEW_CHARS = 2000;
	25	+const PAGE_STATUSES = ["success", "blocked", "failed"] as const;
	26	+
	27	+type SearchParams = Promise<{ cursor?: string \| string[]; status?: string \| string[] }>;
	28	+
	29	+function first(v: string \| string[] \| undefined): string \| undefined {
	30	+ return Array.isArray(v) ? v[0] : v;
	31	+}
	32	+
	33	+function Row({ label, children, mono }: { label: string; children: React.ReactNode; mono?: boolean }) {
	34	+ return (
	35	+ <div className="grid grid-cols-[140px_1fr] gap-3 px-4 py-2.5 text-[13px] sm:grid-cols-[160px_1fr]">
	36	+ <dt className="text-fg-subtle">{label}</dt>
	37	+ <dd className={`min-w-0 break-all ${mono ? "font-mono tabular" : ""}`}>{children}</dd>
	38	+ </div>
	39	+ );
	40	+}
	41	+
	42	+function optionValue(v: unknown): string {
	43	+ if (v === undefined \|\| v === null) return "—";
	44	+ if (Array.isArray(v)) return v.length ? v.join(", ") : "—";
	45	+ if (typeof v === "object") return JSON.stringify(v);
	46	+ return String(v);
	47	+}
	48	+
	49	+const OPTION_KEYS = ["max_pages", "max_depth", "format", "same_domain", "allow_subdomains", "respect_robots", "use_sitemap", "concurrency", "delay_ms", "timeout", "main_content", "country", "network", "browser", "browser_fallback", "include_patterns", "exclude_patterns", "webhook_url"] as const;
	50	+
	51	+export default async function CrawlDetailPage({ params, searchParams }: { params: Promise<{ id: string }>; searchParams: SearchParams }) {
	52	+ const [ws, { id }, sp] = await Promise.all([getWorkspace(), params, searchParams]);
	53	+ if (!/^crawl_[A-Za-z0-9]{4,64}$/.test(id)) notFound();
	54	+ const cursor = first(sp.cursor)?.trim() \|\| null;
	55	+ const statusFilter = first(sp.status)?.trim() \|\| null;
	56	+ const status = statusFilter && (PAGE_STATUSES as readonly string[]).includes(statusFilter) ? statusFilter : null;
	57	+
	58	+ let job: CrawlJob \| null = null;
	59	+ let pages: CrawlPagesPage = { data: [], next_cursor: null };
	60	+ let loadError: string \| null = null;
	61	+ try {
	62	+ job = await internalApi.getCrawl(ws.project.id, ws.user.id, id);
	63	+ } catch (e) {
	64	+ if (e instanceof InternalApiError && (e.status === 404 \|\| e.code === "NOT_FOUND")) notFound();
	65	+ loadError = e instanceof InternalApiError ? e.message : "The Fetcha API service is unreachable.";
	66	+ }
	67	+ if (!job) {
	68	+ return (
	69	+ <div className="flex flex-col gap-5">
	70	+ <Link href="/dashboard/crawls" className="inline-flex items-center gap-1 text-[12.5px] text-fg-muted underline-offset-4 hover:text-fg hover:underline">
	71	+ <ArrowLeft className="size-3.5" /> Crawls
	72	+ </Link>
	73	+ <Alert variant="danger" title="Could not load this crawl">
	74	+ {loadError ?? "Unknown error."}
	75	+ </Alert>
	76	+ </div>
	77	+ );
	78	+ }
	79	+ try {
	80	+ pages = await internalApi.crawlPages(ws.project.id, ws.user.id, id, { cursor, limit: PAGE_SIZE, status });
	81	+ } catch (e) {
	82	+ loadError = e instanceof InternalApiError ? e.message : "Could not load the crawled pages.";
	83	+ }
	84	+
	85	+ const active = job.status === "queued" \|\| job.status === "running";
	86	+ const stats = job.stats ?? { discovered: 0, fetched: 0, ok: 0, blocked: 0, failed: 0, bytes: 0 };
	87	+ const hrefFor = (next: { cursor?: string \| null; status?: string \| null }) => {
	88	+ const q = new URLSearchParams();
	89	+ const st = next.status === undefined ? status : next.status;
	90	+ if (st) q.set("status", st);
	91	+ if (next.cursor) q.set("cursor", next.cursor);
	92	+ const s = q.toString();
	93	+ return `/dashboard/crawls/${job!.id}${s ? `?${s}` : ""}`;
	94	+ };
	95	+ // Only shown for finished jobs (a live elapsed counter would need a client component).
	96	+ const durationMs = job.started_at && job.completed_at ? new Date(job.completed_at).getTime() - new Date(job.started_at).getTime() : null;
	97	+
	98	+ return (
	99	+ <div className="flex flex-col gap-5">
	100	+ <Link href="/dashboard/crawls" className="inline-flex items-center gap-1 text-[12.5px] text-fg-muted underline-offset-4 hover:text-fg hover:underline">
	101	+ <ArrowLeft className="size-3.5" /> Crawls
	102	+ </Link>
	103	+ <PageHeader
	104	+ eyebrow="Crawl"
	105	+ title={
	106	+ <span className="inline-flex flex-wrap items-center gap-2">
	107	+ <span className="min-w-0 break-all">{job.label ?? job.domain ?? job.seed_url}</span>
	108	+ <CrawlStatusBadge status={job.status} />
	109	+ </span>
	110	+ }
	111	+ description={
	112	+ <span className="flex flex-wrap items-center gap-x-2 gap-y-1">
	113	+ <span className="inline-flex items-center gap-1 font-mono text-[12.5px] text-fg">
	114	+ {job.id} <CopyButton value={job.id} className="size-6" />
	115	+ </span>
	116	+ <span className="text-fg-subtle">·</span>
	117	+ <a href={job.seed_url} target="_blank" rel="noreferrer noopener" className="min-w-0 break-all font-mono text-[12.5px] text-accent underline-offset-4 hover:underline">
	118	+ {job.seed_url}
	119	+ </a>
	120	+ <span className="text-fg-subtle">·</span>
	121	+ <span title={formatDate(job.created_at, { timeStyle: "medium" })}>created {timeAgo(job.created_at)}</span>
	122	+ </span>
	123	+ }
	124	+ actions={
	125	+ <>
	126	+ {active ? (
	127	+ <Button variant="outline" size="sm" asChild>
	128	+ <Link href={hrefFor({ cursor })} prefetch={false}>
	129	+ <RefreshCw className="size-3.5" /> Refresh
	130	+ </Link>
	131	+ </Button>
	132	+ ) : null}
	133	+ {active ? <CancelCrawlButton id={job.id} /> : null}
	134	+ </>
	135	+ }
	136	+ />
	137	+
	138	+ {job.error ? (
	139	+ <Alert variant="danger" title={job.error.code}>
	140	+ {job.error.message}
	141	+ </Alert>
	142	+ ) : null}
	143	+ {active ? (
	144	+ <Alert variant="info" title={job.status === "queued" ? "Queued" : "Running"}>
	145	+ {job.status === "queued" ? "The job is waiting for a worker slot. " : "Pages are being fetched. "}
	146	+ This page does not update on its own; use Refresh to see progress.
	147	+ </Alert>
	148	+ ) : null}
	149	+
	150	+ <StatGrid cols={6}>
	151	+ <Stat label="Discovered" value={formatNumber(stats.discovered)} />
	152	+ <Stat label="Fetched" value={formatNumber(stats.fetched)} hint={`of ${formatNumber(Number(job.options?.max_pages ?? 0)) \|\| "—"} max`} />
	153	+ <Stat label="OK" value={formatNumber(stats.ok)} />
	154	+ <Stat label="Blocked" value={formatNumber(stats.blocked)} />
	155	+ <Stat label="Failed" value={formatNumber(stats.failed)} />
	156	+ <Stat label="Bytes" value={formatBytes(stats.bytes)} hint={durationMs !== null ? `in ${formatMs(durationMs)}` : undefined} />
	157	+ </StatGrid>
	158	+
	159	+ <div className="grid gap-5 lg:grid-cols-[minmax(0,1fr)_320px]">
	160	+ <div className="flex min-w-0 flex-col gap-3">
	161	+ <div className="flex flex-wrap items-center justify-between gap-2">
	162	+ <h2 className="text-[13px] font-semibold uppercase tracking-wide text-fg-subtle">Pages</h2>
	163	+ <nav className="flex items-center gap-1 text-[12.5px]" aria-label="Filter pages by status">
	164	+ <FilterLink href={hrefFor({ cursor: null, status: null })} active={!status}>
	165	+ All
	166	+ </FilterLink>
	167	+ {PAGE_STATUSES.map((s) => (
	168	+ <FilterLink key={s} href={hrefFor({ cursor: null, status: s })} active={status === s}>
	169	+ {s}
	170	+ </FilterLink>
	171	+ ))}
	172	+ </nav>
	173	+ </div>
	174	+ {loadError ? (
	175	+ <Alert variant="danger" title="Could not load pages">
	176	+ {loadError}
	177	+ </Alert>
	178	+ ) : null}
	179	+ <Card className="overflow-hidden">
	180	+ <Table>
	181	+ <TableHeader>
	182	+ <TableRow className="hover:bg-transparent">
	183	+ <TableHead>URL</TableHead>
	184	+ <TableHead>Status</TableHead>
	185	+ <TableHead className="text-right">HTTP</TableHead>
	186	+ <TableHead className="text-right">Depth</TableHead>
	187	+ <TableHead className="text-right">Bytes</TableHead>
	188	+ <TableHead className="text-right">Duration</TableHead>
	189	+ <TableHead>Mode</TableHead>
	190	+ </TableRow>
	191	+ </TableHeader>
	192	+ <TableBody>
	193	+ {pages.data.length === 0 ? (
	194	+ <TableEmpty colSpan={7}>{active ? "No pages fetched yet." : status ? `No ${status} pages.` : "No pages were fetched."}</TableEmpty>
	195	+ ) : (
	196	+ pages.data.map((p) => <PageRow key={p.id} page={p} />)
	197	+ )}
	198	+ </TableBody>
	199	+ </Table>
	200	+ </Card>
	201	+ <div className="flex flex-wrap items-center justify-between gap-3 text-[12.5px] text-fg-muted">
	202	+ <span className="font-mono tabular">
	203	+ {formatNumber(pages.data.length)} page{pages.data.length === 1 ? "" : "s"} shown{cursor ? " (continued)" : ""}
	204	+ </span>
	205	+ <div className="flex items-center gap-2">
	206	+ {cursor ? (
	207	+ <Button variant="ghost" size="sm" asChild>
	208	+ <Link href={hrefFor({ cursor: null })}>First page</Link>
	209	+ </Button>
	210	+ ) : null}
	211	+ {pages.next_cursor ? (
	212	+ <Button variant="outline" size="sm" asChild>
	213	+ <Link href={hrefFor({ cursor: pages.next_cursor })} rel="next" prefetch={false}>
	214	+ Load more <ChevronDown className="size-3.5" />
	215	+ </Link>
	216	+ </Button>
	217	+ ) : null}
	218	+ </div>
	219	+ </div>
	220	+ </div>
	221	+
	222	+ <aside className="flex flex-col gap-4">
	223	+ <Card>
	224	+ <CardHeader className="pb-1">
	225	+ <CardTitle className="text-[14px]">Job</CardTitle>
	226	+ </CardHeader>
	227	+ <dl className="divide-y divide-border">
	228	+ <Row label="Domain" mono>
	229	+ {job.domain}
	230	+ </Row>
	231	+ <Row label="Created">{formatDate(job.created_at, { timeStyle: "medium" })}</Row>
	232	+ <Row label="Started">{job.started_at ? formatDate(job.started_at, { timeStyle: "medium" }) : <span className="text-fg-subtle">not yet</span>}</Row>
	233	+ <Row label="Completed">{job.completed_at ? formatDate(job.completed_at, { timeStyle: "medium" }) : <span className="text-fg-subtle">{active ? "in progress" : "—"}</span>}</Row>
	234	+ </dl>
	235	+ </Card>
	236	+ <Card>
	237	+ <CardHeader className="pb-1">
	238	+ <CardTitle className="text-[14px]">Options</CardTitle>
	239	+ </CardHeader>
	240	+ <dl className="divide-y divide-border">
	241	+ {OPTION_KEYS.filter((k) => job!.options && job!.options[k] !== undefined && job!.options[k] !== null && !(Array.isArray(job!.options[k]) && (job!.options[k] as unknown[]).length === 0)).map((k) => (
	242	+ <Row key={k} label={k} mono>
	243	+ {optionValue(job!.options[k])}
	244	+ </Row>
	245	+ ))}
	246	+ </dl>
	247	+ </Card>
	248	+ <Card>
	249	+ <CardHeader className="pb-2">
	250	+ <CardTitle className="text-[14px]">Reading this page</CardTitle>
	251	+ </CardHeader>
	252	+ <CardContent className="space-y-2 text-[12.5px] text-fg-muted">
	253	+ <p>
	254	+ <strong className="text-fg">Mode</strong> shows whether a page was fetched over plain HTTP or rendered in the managed browser after a JavaScript challenge.
	255	+ </p>
	256	+ <p>
	257	+ Click a URL row to expand the first {formatNumber(PREVIEW_CHARS)} characters of the stored content. Retrieve the full content with <code className="font-mono">GET /v1/crawl/:id/pages</code>.
	258	+ </p>
	259	+ <p>
	260	+ Each page is also a request in <Link href="/dashboard/requests" className="underline-offset-4 hover:underline">Requests</Link> with source <code className="font-mono">crawl</code>.
	261	+ </p>
	262	+ </CardContent>
	263	+ </Card>
	264	+ </aside>
	265	+ </div>
	266	+ {!pages.data.length && !active && !loadError && !status ? <EmptyState compact title="Nothing to show" description="The crawl finished without fetching a page. Check the seed URL, robots.txt and the include/exclude patterns." /> : null}
	267	+ </div>
	268	+ );
	269	+}
	270	+
	271	+function FilterLink({ href, active, children }: { href: string; active: boolean; children: React.ReactNode }) {
	272	+ return (
	273	+ <Link href={href} prefetch={false} className={`rounded-md px-2 py-1 capitalize transition-colors ${active ? "bg-bg-muted font-medium text-fg" : "text-fg-muted hover:bg-bg-subtle hover:text-fg"}`} aria-current={active ? "page" : undefined}>
	274	+ {children}
	275	+ </Link>
	276	+ );
	277	+}
	278	+
	279	+function PageRow({ page: p }: { page: CrawlPage }) {
	280	+ const preview = typeof p.content === "string" && p.content.length ? p.content.slice(0, PREVIEW_CHARS) : null;
	281	+ const truncated = typeof p.content === "string" && p.content.length > PREVIEW_CHARS;
	282	+ return (
	283	+ <TableRow className="group">
	284	+ <TableCell colSpan={7} className="p-0">
	285	+ <details className="[&_summary::-webkit-details-marker]:hidden">
	286	+ <summary className="grid cursor-pointer list-none grid-cols-[minmax(0,1fr)_7rem_4rem_4rem_5.5rem_5.5rem_5rem] items-center gap-3 px-4 py-2.5 text-[13px] hover:bg-bg-subtle/60">
	287	+ <span className="min-w-0">
	288	+ <span className="flex items-center gap-1.5">
	289	+ <ChevronDown className="size-3.5 shrink-0 text-fg-subtle transition-transform group-has-[details[open]]:rotate-180" aria-hidden />
	290	+ <span className="truncate font-mono text-[12.5px]" title={p.url}>
	291	+ {p.url}
	292	+ </span>
	293	+ </span>
	294	+ {p.title ? (
	295	+ <span className="block truncate pl-5 text-[12px] text-fg-muted" title={p.title}>
	296	+ {p.title}
	297	+ </span>
	298	+ ) : null}
	299	+ {p.error_code ? <span className="block pl-5 font-mono text-[11.5px] text-danger">{p.error_code}</span> : null}
	300	+ </span>
	301	+ <span>
	302	+ <CrawlPageStatusBadge status={p.status} />
	303	+ </span>
	304	+ <span className="text-right font-mono tabular">{p.http_status ?? <span className="text-fg-subtle">—</span>}</span>
	305	+ <span className="text-right font-mono tabular text-fg-muted">{p.depth}</span>
	306	+ <span className="text-right font-mono tabular text-fg-muted">{formatBytes(p.bytes)}</span>
	307	+ <span className="text-right font-mono tabular text-fg-muted">{formatMs(p.duration_ms)}</span>
	308	+ <span>{p.mode ? <Badge variant={p.mode === "browser" ? "accent" : "outline"}>{p.mode === "browser" ? "Browser" : "HTTP"}</Badge> : <span className="text-fg-subtle">—</span>}</span>
	309	+ </summary>
	310	+ <div className="border-t border-border bg-bg-subtle/40 px-4 py-3">
	311	+ <dl className="mb-3 grid gap-x-6 gap-y-1 text-[12px] text-fg-muted sm:grid-cols-[auto_minmax(0,1fr)]">
	312	+ {p.final_url && p.final_url !== p.url ? (
	313	+ <>
	314	+ <dt className="text-fg-subtle">Final URL</dt>
	315	+ <dd className="min-w-0 break-all font-mono">{p.final_url}</dd>
	316	+ </>
	317	+ ) : null}
	318	+ {p.description ? (
	319	+ <>
	320	+ <dt className="text-fg-subtle">Description</dt>
	321	+ <dd className="min-w-0 break-words">{p.description}</dd>
	322	+ </>
	323	+ ) : null}
	324	+ <dt className="text-fg-subtle">Content type</dt>
	325	+ <dd className="font-mono">{p.content_type ?? "—"}</dd>
	326	+ <dt className="text-fg-subtle">Links</dt>
	327	+ <dd className="font-mono tabular">{p.links_count ?? "—"}</dd>
	328	+ <dt className="text-fg-subtle">Fetched</dt>
	329	+ <dd>{p.fetched_at ? formatDate(p.fetched_at, { timeStyle: "medium" }) : "—"}</dd>
	330	+ </dl>
	331	+ {preview ? (
	332	+ <>
	333	+ <pre className="max-h-[360px] overflow-auto whitespace-pre-wrap break-words rounded-md border border-border bg-bg p-3 font-mono text-[12px] leading-relaxed scrollbar-thin">{preview}</pre>
	334	+ {truncated ? <p className="mt-1.5 text-[11.5px] text-fg-subtle">Showing the first {formatNumber(PREVIEW_CHARS)} of {formatNumber(p.content!.length)} characters.</p> : null}
	335	+ </>
	336	+ ) : (
	337	+ <p className="text-[12px] text-fg-subtle">No stored content for this page.</p>
	338	+ )}
	339	+ </div>
	340	+ </details>
	341	+ </TableCell>
	342	+ </TableRow>
	343	+ );
	344	+}

added apps/web/src/app/dashboard/crawls/loading.tsx +31 −0

		`@@ -0,0 +1,31 @@`
	1	+import { Skeleton, TableSkeleton } from "@/components/ui/skeleton";
	2	+
	3	+export default function CrawlsLoading() {
	4	+ return (
	5	+ <div className="flex flex-col gap-5" aria-busy="true" aria-label="Loading crawls">
	6	+ <div className="flex items-start justify-between gap-3">
	7	+ <div className="flex flex-col gap-2">
	8	+ <Skeleton className="h-7 w-32" />
	9	+ <Skeleton className="h-3.5 w-[420px] max-w-full" />
	10	+ </div>
	11	+ <Skeleton className="h-8 w-28" />
	12	+ </div>
	13	+ <div className="grid gap-px overflow-hidden rounded-lg border border-border bg-border sm:grid-cols-2 lg:grid-cols-4">
	14	+ {Array.from({ length: 4 }).map((_, i) => (
	15	+ <div key={i} className="flex flex-col gap-2 bg-bg-elevated px-5 py-4">
	16	+ <Skeleton className="h-3 w-24" />
	17	+ <Skeleton className="h-6 w-16" />
	18	+ </div>
	19	+ ))}
	20	+ </div>
	21	+ <div className="rounded-lg border border-border bg-bg-elevated">
	22	+ <TableSkeleton rows={6} cols={8} />
	23	+ </div>
	24	+ <div className="grid gap-4 lg:grid-cols-2">
	25	+ <Skeleton className="h-64 w-full" />
	26	+ <Skeleton className="h-64 w-full" />
	27	+ </div>
	28	+ <Skeleton className="h-48 w-full" />
	29	+ </div>
	30	+ );
	31	+}

added apps/web/src/app/dashboard/crawls/page.tsx +179 −0

		`@@ -0,0 +1,179 @@`
	1	+import type { Metadata } from "next";
	2	+import Link from "next/link";
	3	+import { BookOpen, Network } from "lucide-react";
	4	+import { PLAN_LIMITS, normalizePlan } from "@fetcha/core";
	5	+import { getWorkspace } from "@/lib/session";
	6	+import { internalApi, InternalApiError, type CrawlJob } from "@/lib/api";
	7	+import { formatBytes, formatDate, formatNumber, timeAgo } from "@/lib/format";
	8	+import { API_PUBLIC_URL } from "@/lib/utils";
	9	+import { PageHeader } from "@/components/ui/page-header";
	10	+import { Alert } from "@/components/ui/alert";
	11	+import { Button } from "@/components/ui/button";
	12	+import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card";
	13	+import { CodeBlock } from "@/components/ui/code-block";
	14	+import { EmptyState } from "@/components/ui/empty-state";
	15	+import { Stat, StatGrid } from "@/components/ui/stat";
	16	+import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table";
	17	+import { CrawlStatusBadge } from "@/components/dashboard/crawls/crawl-status-badge";
	18	+import { CreateCrawlDialog } from "@/components/dashboard/crawls/create-crawl-dialog";
	19	+import { CancelCrawlButton } from "@/components/dashboard/crawls/cancel-crawl-button";
	20	+import { MapTool } from "@/components/dashboard/crawls/map-tool";
	21	+
	22	+export const dynamic = "force-dynamic";
	23	+export const metadata: Metadata = { title: "Crawls" };
	24	+
	25	+const API_SNIPPET = `# 1. Start a crawl (returns 202 with the job)
	26	+curl -X POST ${API_PUBLIC_URL}/v1/crawl \\
	27	+ -H "Authorization: Bearer fch_live_YOUR_KEY" \\
	28	+ -H "Content-Type: application/json" \\
	29	+ -d '{ "url": "https://docs.example.com/", "max_pages": 100, "max_depth": 3, "format": "markdown" }'
	30	+# → { "id": "crawl_…", "status": "queued", … }
	31	+
	32	+# 2. Poll the job, then page through the results
	33	+curl ${API_PUBLIC_URL}/v1/crawl/crawl_… -H "Authorization: Bearer fch_live_YOUR_KEY"
	34	+curl "${API_PUBLIC_URL}/v1/crawl/crawl_…/pages?limit=100" -H "Authorization: Bearer fch_live_YOUR_KEY"`;
	35	+
	36	+function seedLabel(job: CrawlJob): string {
	37	+ try {
	38	+ const u = new URL(job.seed_url);
	39	+ return u.host + (u.pathname !== "/" ? u.pathname : "");
	40	+ } catch {
	41	+ return job.seed_url;
	42	+ }
	43	+}
	44	+
	45	+export default async function CrawlsPage() {
	46	+ const ws = await getWorkspace();
	47	+ const limits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
	48	+
	49	+ let jobs: CrawlJob[] = [];
	50	+ let loadError: string \| null = null;
	51	+ try {
	52	+ const res = await internalApi.listCrawls(ws.project.id, ws.user.id, 50);
	53	+ jobs = Array.isArray(res?.data) ? res.data : [];
	54	+ } catch (e) {
	55	+ loadError = e instanceof InternalApiError ? e.message : "The Fetcha API service is unreachable.";
	56	+ }
	57	+
	58	+ const active = jobs.filter((j) => j.status === "queued" \|\| j.status === "running");
	59	+ const pagesFetched = jobs.reduce((a, j) => a + (j.stats?.fetched ?? 0), 0);
	60	+ const bytes = jobs.reduce((a, j) => a + (j.stats?.bytes ?? 0), 0);
	61	+ const createButton = <CreateCrawlDialog defaultCountry={ws.project.defaultCountry} maxPages={limits.crawl_max_pages} />;
	62	+
	63	+ return (
	64	+ <div className="flex flex-col gap-5">
	65	+ <PageHeader
	66	+ eyebrow={ws.project.name}
	67	+ title="Crawls"
	68	+ description={`Crawl jobs for ${ws.project.name}. A crawl follows links from a seed URL, fetches each page through the routing engine and stores the content as Markdown, text or HTML.`}
	69	+ actions={
	70	+ <>
	71	+ <Button variant="outline" size="sm" asChild>
	72	+ <Link href="/docs/crawl">
	73	+ <BookOpen className="size-3.5" /> Crawl API
	74	+ </Link>
	75	+ </Button>
	76	+ {createButton}
	77	+ </>
	78	+ }
	79	+ />
	80	+
	81	+ <StatGrid cols={4}>
	82	+ <Stat label="Active jobs" value={formatNumber(active.length)} hint={`up to ${limits.crawl_concurrent_jobs} concurrent`} />
	83	+ <Stat label="Jobs" value={formatNumber(jobs.length)} hint="most recent 50" />
	84	+ <Stat label="Pages fetched" value={formatNumber(pagesFetched)} hint={`max ${formatNumber(limits.crawl_max_pages)} per job`} />
	85	+ <Stat label="Bytes transferred" value={formatBytes(bytes)} hint="all listed jobs" />
	86	+ </StatGrid>
	87	+
	88	+ {loadError ? (
	89	+ <Alert variant="danger" title="Could not load crawl jobs">
	90	+ {loadError}
	91	+ </Alert>
	92	+ ) : null}
	93	+
	94	+ {jobs.length ? (
	95	+ <Card className="overflow-hidden">
	96	+ <Table>
	97	+ <TableHeader>
	98	+ <TableRow className="hover:bg-transparent">
	99	+ <TableHead>Crawl</TableHead>
	100	+ <TableHead>Status</TableHead>
	101	+ <TableHead className="text-right">Fetched</TableHead>
	102	+ <TableHead className="text-right">OK</TableHead>
	103	+ <TableHead className="text-right">Blocked</TableHead>
	104	+ <TableHead className="text-right">Failed</TableHead>
	105	+ <TableHead className="text-right">Bytes</TableHead>
	106	+ <TableHead>Created</TableHead>
	107	+ <TableHead className="text-right"> </TableHead>
	108	+ </TableRow>
	109	+ </TableHeader>
	110	+ <TableBody>
	111	+ {jobs.map((j) => (
	112	+ <TableRow key={j.id}>
	113	+ <TableCell className="max-w-[320px]">
	114	+ <Link href={`/dashboard/crawls/${j.id}`} className="block min-w-0">
	115	+ <span className="block truncate text-[13.5px] font-medium text-fg underline-offset-4 hover:underline" title={j.label ?? j.seed_url}>
	116	+ {j.label ?? seedLabel(j)}
	117	+ </span>
	118	+ <span className="block truncate font-mono text-[11.5px] text-fg-subtle" title={j.seed_url}>
	119	+ {j.label ? j.seed_url : j.id}
	120	+ </span>
	121	+ </Link>
	122	+ </TableCell>
	123	+ <TableCell>
	124	+ <CrawlStatusBadge status={j.status} />
	125	+ </TableCell>
	126	+ <TableCell className="text-right font-mono tabular">
	127	+ {formatNumber(j.stats?.fetched ?? 0)}
	128	+ <span className="text-fg-subtle"> / {formatNumber(j.stats?.discovered ?? 0)}</span>
	129	+ </TableCell>
	130	+ <TableCell className="text-right font-mono tabular text-success">{formatNumber(j.stats?.ok ?? 0)}</TableCell>
	131	+ <TableCell className="text-right font-mono tabular">{j.stats?.blocked ? <span className="text-warning">{formatNumber(j.stats.blocked)}</span> : <span className="text-fg-subtle">0</span>}</TableCell>
	132	+ <TableCell className="text-right font-mono tabular">{j.stats?.failed ? <span className="text-danger">{formatNumber(j.stats.failed)}</span> : <span className="text-fg-subtle">0</span>}</TableCell>
	133	+ <TableCell className="text-right font-mono tabular text-fg-muted">{formatBytes(j.stats?.bytes ?? 0)}</TableCell>
	134	+ <TableCell className="whitespace-nowrap text-fg-muted" title={formatDate(j.created_at, { timeStyle: "medium" })}>
	135	+ {timeAgo(j.created_at)}
	136	+ </TableCell>
	137	+ <TableCell className="text-right">{j.status === "queued" \|\| j.status === "running" ? <CancelCrawlButton id={j.id} size="xs" /> : null}</TableCell>
	138	+ </TableRow>
	139	+ ))}
	140	+ </TableBody>
	141	+ </Table>
	142	+ </Card>
	143	+ ) : !loadError ? (
	144	+ <EmptyState
	145	+ icon={Network}
	146	+ title="No crawls yet"
	147	+ description="Start one here to try it, or from your code with POST /v1/crawl. Each crawled page is a normal fetch request: quotas, retries, escalation and routing intelligence apply."
	148	+ action={createButton}
	149	+ />
	150	+ ) : null}
	151	+
	152	+ <div className="grid gap-4 lg:grid-cols-2">
	153	+ <Card>
	154	+ <CardHeader>
	155	+ <CardTitle>How crawls work</CardTitle>
	156	+ <CardDescription>Scope, politeness and what you get back.</CardDescription>
	157	+ </CardHeader>
	158	+ <CardContent className="space-y-3 text-[13px] leading-relaxed text-fg-muted">
	159	+ <p>
	160	+ <strong className="text-fg">Frontier.</strong> The seed is fetched first; links are extracted, normalised and filtered by <code className="font-mono">same_domain</code>, <code className="font-mono">include_patterns</code> / <code className="font-mono">exclude_patterns</code> and <code className="font-mono">max_depth</code>, then fetched with up to <code className="font-mono">concurrency</code> workers until <code className="font-mono">max_pages</code> is reached.
	161	+ </p>
	162	+ <p>
	163	+ <strong className="text-fg">Politeness.</strong> <code className="font-mono">robots.txt</code> is honoured by default and <code className="font-mono">delay_ms</code> adds a pause between fetches per worker. Sitemaps can seed the frontier with <code className="font-mono">use_sitemap</code>.
	164	+ </p>
	165	+ <p>
	166	+ <strong className="text-fg">Content.</strong> Each page is stored as Markdown (main content, boilerplate removed), text or HTML, with title, description, status, mode (HTTP or browser), bytes and duration. Blocked pages escalate to the managed browser automatically.
	167	+ </p>
	168	+ <p className="text-fg-subtle">
	169	+ Every page also appears in <Link href="/dashboard/requests" className="underline-offset-4 hover:underline">Requests</Link> with source <code className="font-mono">crawl</code>. Jobs are limited to {formatNumber(limits.crawl_max_pages)} pages and {limits.crawl_concurrent_jobs} concurrent jobs per organization.
	170	+ </p>
	171	+ </CardContent>
	172	+ </Card>
	173	+ <CodeBlock code={API_SNIPPET} lang="bash" title="Crawls from the API" className="self-start" />
	174	+ </div>
	175	+
	176	+ <MapTool />
	177	+ </div>
	178	+ );
	179	+}

modified apps/web/src/app/dashboard/page.tsx +5 −6

		`@@ -1,6 +1,6 @@`
1	1	import Link from "next/link";
2	2	import { ArrowRight, BookOpen, CheckCircle2, Circle, KeyRound, Play } from "lucide-react";
3		−import { PLAN_LIMITS, type Plan } from "@fetcha/core";
	3	+import { PLAN_LIMITS, isUnlimited, normalizePlan } from "@fetcha/core";
4	4	import { getWorkspace } from "@/lib/session";
5	5	import { formatBytes, formatCompact, formatMs, formatNumber, formatPercent, formatUsd, formatDateOnly } from "@/lib/format";
6	6	import { getActiveSessionCount, getMonthStats, getNetworkDistribution, getOnboardingState, getRecentRequests, getRequestSeries, type Scope } from "@/lib/queries/dashboard";
		`@@ -35,9 +35,8 @@ export default async function DashboardOverviewPage({ searchParams }: { searchPa`
35	35	getActiveSessionCount(scope),
36	36	]);
37	37
38		− const plan = (ws.organization.plan in PLAN_LIMITS ? ws.organization.plan : "free") as Plan;
39		− const limits = PLAN_LIMITS[plan];
40		− const unlimited = limits.monthly_requests >= Number.MAX_SAFE_INTEGER;
	38	+ const limits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
	39	+ const unlimited = isUnlimited(limits);
41	40	const projectLimit = ws.project.monthlyRequestLimit;
42	41	const effectiveLimit = projectLimit && (unlimited \|\| projectLimit < limits.monthly_requests) ? projectLimit : unlimited ? null : limits.monthly_requests;
43	42	const firstName = ws.user.name?.trim().split(/\s+/)[0] \|\| ws.user.email.split("@")[0];
		`@@ -138,12 +137,12 @@ export default async function DashboardOverviewPage({ searchParams }: { searchPa`
138	137	<CardHeader className="pb-3">
139	138	<CardTitle>Quota and limits</CardTitle>
140	139	<CardDescription>
141		− Monthly request allowance on the {limits.label} plan
	140	+ {unlimited ? "Requests are unlimited on this private platform" : `Monthly request allowance on the ${limits.label} plan`}
142	141	{projectLimit ? ` (project cap: ${formatNumber(projectLimit)})` : ""}. Spending limits stop new requests with <code className="font-mono text-[12px]">USAGE_LIMIT_REACHED</code>.
143	142	</CardDescription>
144	143	</CardHeader>
145	144	<CardContent className="grid gap-6 lg:grid-cols-2">
146		− <QuotaBar label="Requests this month" used={month.total} limit={effectiveLimit} unlimited={effectiveLimit === null} usedLabel={formatNumber(month.total)} limitLabel={effectiveLimit ? formatCompact(effectiveLimit) : undefined} hint={effectiveLimit ? `${formatNumber(Math.max(0, effectiveLimit - month.total))} remaining` : "Enterprise plan"} />
	145	+ <QuotaBar label="Requests this month" used={month.total} limit={effectiveLimit} unlimited={effectiveLimit === null} usedLabel={formatNumber(month.total)} limitLabel={effectiveLimit ? formatCompact(effectiveLimit) : undefined} hint={effectiveLimit ? `${formatNumber(Math.max(0, effectiveLimit - month.total))} remaining` : "no monthly cap"} />
147	146	<div className="grid grid-cols-2 gap-4 text-[13px]">
148	147	<div>
149	148	<div className="text-[12px] font-medium text-fg-subtle">Soft limit</div>

modified apps/web/src/app/dashboard/playground/page.tsx +4 −5

		`@@ -1,7 +1,7 @@`
1	1	import type { Metadata } from "next";
2	2	import Link from "next/link";
3	3	import { BookOpen } from "lucide-react";
4		−import { COUNTRIES, PLAN_LIMITS, type Plan } from "@fetcha/core";
	4	+import { COUNTRIES, PLAN_LIMITS, normalizePlan } from "@fetcha/core";
5	5	import { Button } from "@/components/ui/button";
6	6	import { PageHeader } from "@/components/ui/page-header";
7	7	import { Playground } from "@/components/playground/playground";
		`@@ -22,8 +22,7 @@ function first(v: string \| string[] \| undefined): string \| undefined {`
22	22
23	23	export default async function PlaygroundPage({ searchParams }: { searchParams: SearchParams }) {
24	24	const [sp, ws] = await Promise.all([searchParams, getWorkspace()]);
25		− const planKey = (ws.organization.plan in PLAN_LIMITS ? ws.organization.plan : "free") as Plan;
26		− const limits = PLAN_LIMITS[planKey];
	25	+ const limits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
27	26
28	27	const replayId = first(sp.request)?.trim();
29	28	const prefillUrl = first(sp.url)?.trim();
		`@@ -63,7 +62,7 @@ export default async function PlaygroundPage({ searchParams }: { searchParams: S`
63	62	<PageHeader
64	63	eyebrow={ws.project.name}
65	64	title="Playground"
66		− description="Send a request through Fetcha and inspect the response, the network route it took and where the time went. Playground requests count toward your plan usage."
	65	+ description="Send a request through Fetcha and inspect the response, the network route it took and where the time went. Turn on browser rendering for JavaScript-heavy pages. Playground requests appear in the request log."
67	66	actions={
68	67	<Button variant="outline" size="sm" asChild>
69	68	<Link href="/docs">
		`@@ -74,7 +73,7 @@ export default async function PlaygroundPage({ searchParams }: { searchParams: S`
74	73	/>
75	74	<Playground
76	75	keys={keys}
77		− plan={{ label: limits.label, concurrency: limits.concurrency, max_timeout_ms: limits.max_timeout_ms, max_retries: limits.max_retries, networks: limits.networks }}
	76	+ plan={{ label: limits.label, concurrency: limits.concurrency, max_timeout_ms: limits.max_timeout_ms, max_retries: limits.max_retries, networks: limits.networks, browser_concurrency: limits.browser_concurrency }}
78	77	availableNetworks={ready.available_networks ?? []}
79	78	countries={countries}
80	79	baseUrl={API_PUBLIC_URL}

modified apps/web/src/app/dashboard/projects/[id]/page.tsx +3 −3

		`@@ -1,7 +1,7 @@`
1	1	import Link from "next/link";
2	2	import { notFound } from "next/navigation";
3	3	import { ArrowLeft, ArrowRight, KeyRound } from "lucide-react";
4		−import { PLAN_LIMITS, type Plan } from "@fetcha/core";
	4	+import { PLAN_LIMITS, normalizePlan } from "@fetcha/core";
5	5	import { getWorkspace } from "@/lib/session";
6	6	import { getOrgProject, listApiKeys, monthlyUsage } from "@/lib/queries/account";
7	7	import { formatDate, formatDateOnly, formatNumber, formatPercent, formatUsd, timeAgo } from "@/lib/format";
		`@@ -30,7 +30,7 @@ export default async function ProjectDetailPage({ params }: { params: Promise<{`
30	30	const activeKeys = keys.filter((k) => k.status === "active");
31	31	const current = ws.project.id === project.id;
32	32	const archived = Boolean(project.archivedAt);
33		− const planLimits = PLAN_LIMITS[(ws.organization.plan as Plan) in PLAN_LIMITS ? (ws.organization.plan as Plan) : "free"];
	33	+ const planLimits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
34	34	const requestCap = project.monthlyRequestLimit ?? planLimits.monthly_requests;
35	35	const requestPct = requestCap > 0 && requestCap < Number.MAX_SAFE_INTEGER ? Math.min(100, (usage.requests / requestCap) * 100) : null;
36	36	const spendPct = project.hardLimitUsd ? Math.min(100, (usage.spendUsd / project.hardLimitUsd) * 100) : null;
		`@@ -175,7 +175,7 @@ export default async function ProjectDetailPage({ params }: { params: Promise<{`
175	175	<CardDescription>Month-to-date against this project's limits.</CardDescription>
176	176	</CardHeader>
177	177	<CardContent className="grid gap-4 text-[13px]">
178		− <LimitRow label="Requests" value={`${formatNumber(usage.requests)} / ${requestCap < Number.MAX_SAFE_INTEGER ? formatNumber(requestCap) : "∞"}`} pct={requestPct} note={project.monthlyRequestLimit ? "project cap" : `${planLimits.label} plan`} />
	178	+ <LimitRow label="Requests" value={`${formatNumber(usage.requests)} / ${requestCap < Number.MAX_SAFE_INTEGER ? formatNumber(requestCap) : "∞"}`} pct={requestPct} note={project.monthlyRequestLimit ? "project cap" : `${planLimits.label} — no monthly cap`} />
179	179	<LimitRow label="Spend" value={`${formatUsd(usage.spendUsd)}${project.hardLimitUsd ? ` / ${formatUsd(project.hardLimitUsd)}` : ""}`} pct={spendPct} note={project.hardLimitUsd ? "hard limit" : "no hard limit"} />
180	180	<div className="flex items-center justify-between border-t border-border pt-3 text-fg-muted">
181	181	<span>Soft limit (alert)</span>

modified apps/web/src/app/dashboard/usage/page.tsx +18 −17

		`@@ -1,6 +1,6 @@`
1	1	import Link from "next/link";
2	2	import { Activity, ArrowRight } from "lucide-react";
3		−import { PLAN_LIMITS, type Plan } from "@fetcha/core";
	3	+import { PLAN_LIMITS, isUnlimited, normalizePlan } from "@fetcha/core";
4	4	import { getWorkspace } from "@/lib/session";
5	5	import { formatBytes, formatCompact, formatDate, formatDateOnly, formatNumber, formatUsd } from "@/lib/format";
6	6	import { getUsageMonth, lastMonths, type Scope } from "@/lib/queries/dashboard";
		`@@ -50,9 +50,8 @@ export default async function UsagePage({ searchParams }: { searchParams: Promis`
50	50	const months = lastMonths(6);
51	51	const isCurrent = usage.month === months[0]!.key;
52	52
53		− const plan = (ws.organization.plan in PLAN_LIMITS ? ws.organization.plan : "free") as Plan;
54		− const limits = PLAN_LIMITS[plan];
55		− const unlimited = limits.monthly_requests >= Number.MAX_SAFE_INTEGER;
	53	+ const limits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
	54	+ const unlimited = isUnlimited(limits);
56	55	const projectCap = ws.project.monthlyRequestLimit;
57	56	const requestLimit = projectCap && (unlimited \|\| projectCap < limits.monthly_requests) ? projectCap : unlimited ? null : limits.monthly_requests;
58	57	const includedBytes = limits.included_gb * GB;
		`@@ -65,7 +64,7 @@ export default async function UsagePage({ searchParams }: { searchParams: Promis`
65	64	<div className="flex flex-col gap-6">
66	65	<PageHeader
67	66	title="Usage"
68		− description={`Metered usage for ${ws.project.name}. Amounts are estimates until the monthly statement is issued; the ledger below is the source of truth.`}
	67	+ description={`Usage for ${ws.project.name}. Fetcha is a private platform with no billing: amounts are internal cost estimates used only for optional spending limits. The ledger below is the source of truth.`}
69	68	actions={
70	69	<nav className="inline-flex h-9 items-center gap-0.5 rounded-md bg-bg-muted p-1" aria-label="Month">
71	70	{months.map((m) => (
		`@@ -88,12 +87,12 @@ export default async function UsagePage({ searchParams }: { searchParams: Promis`
88	87	<Stat label="Requests" value={formatNumber(usage.requests)} hint={requestLimit ? `of ${formatCompact(requestLimit)} included` : "unlimited"} />
89	88	<Stat label="Successful" value={formatNumber(usage.successful)} hint={usage.requests ? `${((usage.successful / usage.requests) * 100).toFixed(1)}% of requests` : "—"} />
90	89	<Stat label="Bandwidth" value={formatBytes(usage.bytes)} hint="transferred in + out" />
91		− <Stat label="Estimated spend" value={formatUsd(usage.spendUsd)} hint={limits.price_usd_month ? `plus ${formatUsd(limits.price_usd_month)} plan fee` : "no plan fee"} />
	90	+ <Stat label="Estimated spend" value={formatUsd(usage.spendUsd)} hint="internal estimate, not billed" />
92	91	</StatGrid>
93	92	<StatGrid cols={3} className="mt-3">
94		− <Stat label="Residential bandwidth" value={formatBytes(residentialUsed)} hint={limits.included_gb ? `${limits.included_gb} GB included` : "metered"} />
95		− <Stat label="Mobile bandwidth" value={formatBytes(usage.mobileBytes)} hint={<Badge variant="outline">coming soon</Badge>} />
96		− <Stat label="Browser seconds" value={formatNumber(usage.browserSeconds)} hint={<Badge variant="outline">not launched</Badge>} />
	93	+ <Stat label="Residential bandwidth" value={formatBytes(residentialUsed)} hint="no allowance cap" />
	94	+ <Stat label="Mobile bandwidth" value={formatBytes(usage.mobileBytes)} hint="no allowance cap" />
	95	+ <Stat label="Browser seconds" value={formatNumber(usage.browserSeconds)} hint={<Badge variant="success">live</Badge>} />
97	96	</StatGrid>
98	97	</section>
99	98
		`@@ -101,20 +100,20 @@ export default async function UsagePage({ searchParams }: { searchParams: Promis`
101	100	<Card className="lg:col-span-2">
102	101	<CardHeader>
103	102	<CardTitle>Quotas</CardTitle>
104		− <CardDescription>Included allowances on the {limits.label} plan for {monthLabel}.</CardDescription>
	103	+ <CardDescription>{unlimited ? `No monthly quota on this private platform for ${monthLabel}; only a project cap you set yourself can limit requests.` : `Included allowances on the ${limits.label} plan for ${monthLabel}.`}</CardDescription>
105	104	</CardHeader>
106	105	<CardContent className="grid gap-6 sm:grid-cols-2">
107		− <QuotaBar label="Requests" used={usage.requests} limit={requestLimit} unlimited={requestLimit === null} usedLabel={formatNumber(usage.requests)} limitLabel={requestLimit ? formatNumber(requestLimit) : undefined} hint={projectCap && requestLimit === projectCap ? "project cap" : limits.overage_per_1k_requests_usd ? `then ${formatUsd(limits.overage_per_1k_requests_usd)} per 1k` : "hard stop at the limit"} />
108		− <QuotaBar label="Included residential bandwidth" used={residentialUsed} limit={includedBytes > 0 ? includedBytes : null} unlimited={includedBytes <= 0} usedLabel={formatBytes(residentialUsed)} limitLabel={includedBytes > 0 ? `${limits.included_gb} GB` : undefined} hint={limits.residential_per_gb_usd ? `then ${formatUsd(limits.residential_per_gb_usd)} per GB` : "metered per plan"} />
	106	+ <QuotaBar label="Requests" used={usage.requests} limit={requestLimit} unlimited={requestLimit === null} usedLabel={formatNumber(usage.requests)} limitLabel={requestLimit ? formatNumber(requestLimit) : undefined} hint={projectCap && requestLimit === projectCap ? "project cap" : requestLimit === null ? "unlimited" : "hard stop at the limit"} />
	107	+ <QuotaBar label="Residential bandwidth" used={residentialUsed} limit={includedBytes > 0 ? includedBytes : null} unlimited={includedBytes <= 0} usedLabel={formatBytes(residentialUsed)} limitLabel={includedBytes > 0 ? `${limits.included_gb} GB` : undefined} hint={includedBytes > 0 ? "included allowance" : "unmetered"} />
109	108	</CardContent>
110	109	</Card>
111	110	<Card>
112	111	<CardHeader>
113	112	<CardTitle className="flex items-center justify-between">
114	113	{limits.label} plan
115		− <Badge variant="accent">{limits.price_usd_month ? `${formatUsd(limits.price_usd_month)}/mo` : plan === "enterprise" ? "custom" : "free"}</Badge>
	114	+ <Badge variant="solid">private platform</Badge>
116	115	</CardTitle>
117		− <CardDescription>Limits that apply to every project in {ws.organization.name}.</CardDescription>
	116	+ <CardDescription>Limits that apply to every project in {ws.organization.name}. No billing.</CardDescription>
118	117	</CardHeader>
119	118	<CardContent>
120	119	<dl className="divide-y divide-border text-[13px]">
		`@@ -123,8 +122,10 @@ export default async function UsagePage({ searchParams }: { searchParams: Promis`
123	122	["Concurrency", formatNumber(limits.concurrency)],
124	123	["Max timeout", `${limits.max_timeout_ms / 1000} s`],
125	124	["Max retries", String(limits.max_retries)],
	125	+ ["Network classes", limits.networks.length === 4 ? "All" : limits.networks.join(", ")],
	126	+ ["Browser renders", `${limits.browser_concurrency} concurrent`],
	127	+ ["Crawl jobs", `${formatNumber(limits.crawl_max_pages)} pages · ${limits.crawl_concurrent_jobs} parallel`],
126	128	["Log retention", `${limits.retention_days} days`],
127		− ["Included residential", limits.included_gb ? `${limits.included_gb} GB` : "—"],
128	129	].map(([k, v]) => (
129	130	<div key={k} className="flex items-center justify-between py-1.5">
130	131	<dt className="text-fg-muted">{k}</dt>
		`@@ -134,7 +135,7 @@ export default async function UsagePage({ searchParams }: { searchParams: Promis`
134	135	</dl>
135	136	<Button asChild variant="outline" size="sm" className="mt-4 w-full">
136	137	<Link href="/dashboard/billing">
137		− Plans and billing <ArrowRight />
	138	+ Plan & access <ArrowRight />
138	139	</Link>
139	140	</Button>
140	141	</CardContent>
		`@@ -268,7 +269,7 @@ export default async function UsagePage({ searchParams }: { searchParams: Promis`
268	269	</TableBody>
269	270	</Table>
270	271	</Card>
271		− <p className="mt-2 text-[12px] text-fg-subtle">The ledger is immutable: corrections are posted as new entries, never by editing past ones. Prices are what you are billed; upstream costs are not part of your account.</p>
	272	+ <p className="mt-2 text-[12px] text-fg-subtle">The ledger is immutable: corrections are posted as new entries, never by editing past ones. Prices are internal estimates used for spending limits only; nothing is invoiced on this platform.</p>
272	273	</section>
273	274	</>
274	275	)}

added apps/web/src/components/admin/access-controls.tsx +149 −0

		`@@ -0,0 +1,149 @@`
	1	+"use client";
	2	+import * as React from "react";
	3	+import { useRouter } from "next/navigation";
	4	+import { MailPlus, Send, Trash2, UserPlus } from "lucide-react";
	5	+import { allowEmails, resendInvite, revokeAllow } from "@/actions/access";
	6	+import type { AdminActionResult } from "@/actions/admin";
	7	+import { Button } from "@/components/ui/button";
	8	+import { Input, Textarea } from "@/components/ui/input";
	9	+import { Field, Hint, Label } from "@/components/ui/label";
	10	+import { Switch } from "@/components/ui/switch";
	11	+import { Dialog, DialogContent, DialogDescription, DialogFooter, DialogHeader, DialogTitle } from "@/components/ui/dialog";
	12	+import { ActionButton, ResultMessage } from "./action-button";
	13	+
	14	+export function AddEmailsDialog({ variant = "primary" }: { variant?: "primary" \| "outline" }) {
	15	+ const router = useRouter();
	16	+ const [open, setOpen] = React.useState(false);
	17	+ const [pending, start] = React.useTransition();
	18	+ const [emails, setEmails] = React.useState("");
	19	+ const [note, setNote] = React.useState("");
	20	+ const [sendInvite, setSendInvite] = React.useState(true);
	21	+ const [result, setResult] = React.useState<AdminActionResult<{ added: number; existing: number; invited: number }> \| null>(null);
	22	+
	23	+ const count = emails.split(/[\s,;]+/).filter((s) => s.includes("@")).length;
	24	+
	25	+ const reset = () => {
	26	+ setEmails("");
	27	+ setNote("");
	28	+ setSendInvite(true);
	29	+ setResult(null);
	30	+ };
	31	+
	32	+ return (
	33	+ <>
	34	+ <Button variant={variant} size="sm" onClick={() => setOpen(true)}>
	35	+ <UserPlus className="size-3.5" /> Add emails
	36	+ </Button>
	37	+ <Dialog
	38	+ open={open}
	39	+ onOpenChange={(v) => {
	40	+ setOpen(v);
	41	+ if (!v) reset();
	42	+ }}
	43	+ >
	44	+ <DialogContent size="md">
	45	+ <DialogHeader>
	46	+ <DialogTitle>Add emails to the access list</DialogTitle>
	47	+ <DialogDescription>Only listed addresses can create an account. Paste one or many emails separated by commas, spaces or new lines.</DialogDescription>
	48	+ </DialogHeader>
	49	+ <form
	50	+ className="grid gap-3"
	51	+ onSubmit={(e) => {
	52	+ e.preventDefault();
	53	+ start(async () => {
	54	+ const r = await allowEmails({ emails, note, sendInvite });
	55	+ setResult(r);
	56	+ if (r.ok) {
	57	+ router.refresh();
	58	+ if (!r.warning) {
	59	+ setOpen(false);
	60	+ reset();
	61	+ } else {
	62	+ setEmails("");
	63	+ }
	64	+ }
	65	+ });
	66	+ }}
	67	+ >
	68	+ <Field>
	69	+ <Label htmlFor="access-emails">Emails</Label>
	70	+ <Textarea id="access-emails" value={emails} onChange={(e) => setEmails(e.target.value)} placeholder={"ada@example.com\ngrace@example.com, linus@example.com"} className="min-h-[120px] font-mono text-[12.5px]" required autoFocus />
	71	+ <Hint>{count === 0 ? "Addresses are lower-cased and de-duplicated." : `${count} address${count === 1 ? "" : "es"} detected. Existing entries are kept (the note is updated when provided).`}</Hint>
	72	+ </Field>
	73	+ <Field>
	74	+ <Label htmlFor="access-note">
	75	+ Note <span className="font-normal text-fg-subtle">(optional)</span>
	76	+ </Label>
	77	+ <Input id="access-note" value={note} onChange={(e) => setNote(e.target.value)} placeholder="e.g. Acme data team, pilot Q4" maxLength={280} />
	78	+ <Hint>Internal only — never shown to the invitee.</Hint>
	79	+ </Field>
	80	+ <div className="flex items-start gap-3 rounded-md border border-border bg-bg-subtle/60 px-3 py-2.5">
	81	+ <Switch id="access-send" checked={sendInvite} onCheckedChange={setSendInvite} aria-label="Send invitation email" />
	82	+ <div className="grid gap-0.5">
	83	+ <Label htmlFor="access-send">Send invitation email</Label>
	84	+ <Hint>Each address receives “You're invited to Fetcha” with a link to the signup page, pre-filled with their email. You can resend it later.</Hint>
	85	+ </div>
	86	+ </div>
	87	+ {result && result.ok ? (
	88	+ <p role="status" className="text-[12.5px] text-success">
	89	+ {result.data ? `${result.data.added} added, ${result.data.existing} already listed${sendInvite ? `, ${result.data.invited} invitation${result.data.invited === 1 ? "" : "s"} sent` : ""}.` : "Saved."}
	90	+ </p>
	91	+ ) : null}
	92	+ <ResultMessage result={result && (!result.ok \|\| result.warning) ? result : null} />
	93	+ <DialogFooter>
	94	+ <Button type="button" variant="outline" onClick={() => setOpen(false)}>
	95	+ {result?.ok ? "Close" : "Cancel"}
	96	+ </Button>
	97	+ <Button type="submit" variant="primary" loading={pending} disabled={count === 0}>
	98	+ {sendInvite ? (
	99	+ <>
	100	+ <Send className="size-3.5" /> Add and invite
	101	+ </>
	102	+ ) : (
	103	+ <>
	104	+ <UserPlus className="size-3.5" /> Add to list
	105	+ </>
	106	+ )}
	107	+ </Button>
	108	+ </DialogFooter>
	109	+ </form>
	110	+ </DialogContent>
	111	+ </Dialog>
	112	+ </>
	113	+ );
	114	+}
	115	+
	116	+export function ResendInviteButton({ email, invitedBefore }: { email: string; invitedBefore: boolean }) {
	117	+ return (
	118	+ <ActionButton
	119	+ variant="outline"
	120	+ size="xs"
	121	+ aria-label={`${invitedBefore ? "Resend" : "Send"} invitation to ${email}`}
	122	+ action={() => resendInvite(email)}
	123	+ confirm={{
	124	+ title: `${invitedBefore ? "Resend" : "Send"} the invitation to ${email}?`,
	125	+ description: "An email with a link to the signup page (pre-filled with this address) is sent from hello@fetcha.co.",
	126	+ confirmLabel: invitedBefore ? "Resend invite" : "Send invite",
	127	+ variant: "primary",
	128	+ }}
	129	+ >
	130	+ <MailPlus className="size-3.5" /> {invitedBefore ? "Resend invite" : "Send invite"}
	131	+ </ActionButton>
	132	+ );
	133	+}
	134	+
	135	+export function RemoveAllowButton({ email, hasAccount }: { email: string; hasAccount: boolean }) {
	136	+ return (
	137	+ <ActionButton
	138	+ variant="ghost"
	139	+ size="icon-sm"
	140	+ aria-label={`Remove ${email} from the access list`}
	141	+ disabled={hasAccount}
	142	+ title={hasAccount ? "This address already has an account; ban the user instead." : undefined}
	143	+ action={() => revokeAllow(email)}
	144	+ confirm={{ title: `Remove ${email}?`, description: "The address will no longer be able to create an account. Existing accounts are not affected.", confirmLabel: "Remove" }}
	145	+ >
	146	+ <Trash2 className="size-3.5 text-fg-subtle" />
	147	+ </ActionButton>
	148	+ );
	149	+}

modified apps/web/src/components/admin/admin-shell.tsx +2 −0

		`@@ -11,6 +11,7 @@ import {`
11	11	Flag,
12	12	FolderKanban,
13	13	Globe,
	14	+ KeyRound,
14	15	LayoutDashboard,
15	16	ListTree,
16	17	Menu,
		`@@ -30,6 +31,7 @@ import { Button } from "@/components/ui/button";`
30	31	export const ADMIN_NAV = [
31	32	{ href: "/admin", label: "Overview", icon: LayoutDashboard, exact: true },
32	33	{ href: "/admin/users", label: "Users", icon: Users },
	34	+ { href: "/admin/access", label: "Access", icon: KeyRound },
33	35	{ href: "/admin/organizations", label: "Organizations", icon: Building2 },
34	36	{ href: "/admin/projects", label: "Projects", icon: FolderKanban },
35	37	{ href: "/admin/requests", label: "Requests", icon: ListTree },

modified apps/web/src/components/admin/org-actions.tsx +1 −36

		`@@ -1,49 +1,14 @@`
1	1	"use client";
2	2	import * as React from "react";
3	3	import { useRouter } from "next/navigation";
4		−import { setOrganizationProviderVisibility, setOrganizationSuspended, updateOrganizationLimits, updateOrganizationPlan, type AdminActionResult } from "@/actions/admin";
	4	+import { setOrganizationProviderVisibility, setOrganizationSuspended, updateOrganizationLimits, type AdminActionResult } from "@/actions/admin";
5	5	import { Button } from "@/components/ui/button";
6	6	import { Input, Textarea } from "@/components/ui/input";
7	7	import { Field, Hint, Label } from "@/components/ui/label";
8	8	import { Switch } from "@/components/ui/switch";
9		−import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
10	9	import { Dialog, DialogContent, DialogDescription, DialogFooter, DialogHeader, DialogTitle } from "@/components/ui/dialog";
11	10	import { ResultMessage } from "./action-button";
12	11
13		−export function OrgPlanSelect({ orgId, plan, options }: { orgId: string; plan: string; options: Array<{ value: string; label: string }> }) {
14		− const router = useRouter();
15		− const [pending, start] = React.useTransition();
16		− const [result, setResult] = React.useState<AdminActionResult \| null>(null);
17		− return (
18		− <div className="grid gap-1.5">
19		− <Select
20		− value={plan}
21		− disabled={pending}
22		− onValueChange={(v) =>
23		− start(async () => {
24		− const r = await updateOrganizationPlan(orgId, v);
25		− setResult(r);
26		− if (r.ok) router.refresh();
27		− })
28		− }
29		− >
30		− <SelectTrigger aria-label="Plan" className="w-[200px]">
31		− <SelectValue />
32		− </SelectTrigger>
33		− <SelectContent>
34		− {options.map((o) => (
35		− <SelectItem key={o.value} value={o.value}>
36		− {o.label}
37		− </SelectItem>
38		− ))}
39		− </SelectContent>
40		− </Select>
41		− <Hint>Changes limits immediately. Billing is not connected yet, so no invoice is generated.</Hint>
42		− <ResultMessage result={result} />
43		− </div>
44		− );
45		−}
46		−
47	12	export function OrgProviderVisibilitySwitch({ orgId, enabled }: { orgId: string; enabled: boolean }) {
48	13	const router = useRouter();
49	14	const [pending, start] = React.useTransition();

modified apps/web/src/components/admin/primitives.tsx +3 −2

		`@@ -39,10 +39,11 @@ export function NetworkBadge({ network }: { network: string \| null \| undefined }`
39	39	return <Badge variant={network === "residential" ? "accent" : network === "mobile" ? "info" : "default"}>{network}</Badge>;
40	40	}
41	41
	42	+/** Single-plan platform: `unlimited` is the only valid value; anything else is a legacy row not yet migrated. */
42	43	export function PlanBadge({ plan }: { plan: string }) {
43		− const v = plan === "free" ? "outline" : plan === "developer" ? "default" : plan === "growth" ? "info" : plan === "business" ? "accent" : "solid";
	44	+ const v = plan === "unlimited" ? "solid" : "outline";
44	45	return (
45		− <Badge variant={v} className="capitalize">
	46	+ <Badge variant={v} className="capitalize" title={plan === "unlimited" ? undefined : "Legacy plan value (treated as unlimited)"}>
46	47	{plan}
47	48	</Badge>
48	49	);

deleted apps/web/src/components/dashboard/billing/upgrade-dialog.tsx +0 −39

		`@@ -1,39 +0,0 @@`
1		−"use client";
2		−import * as React from "react";
3		−import { Mail } from "lucide-react";
4		−import { Button, type ButtonProps } from "@/components/ui/button";
5		−import { Dialog, DialogContent, DialogDescription, DialogFooter, DialogHeader, DialogTitle, DialogTrigger } from "@/components/ui/dialog";
6		−import { Alert } from "@/components/ui/alert";
7		−
8		−export function UpgradeDialog({ planLabel, priceLabel, currentPlanLabel, variant = "outline" }: { planLabel: string; priceLabel: string; currentPlanLabel: string; variant?: ButtonProps["variant"] }) {
9		− const isEnterprise = planLabel.toLowerCase() === "enterprise";
10		− const subject = encodeURIComponent(`Upgrade to ${planLabel}`);
11		− const body = encodeURIComponent(`Hi Fetcha team,\n\nI'd like to move from ${currentPlanLabel} to ${planLabel}.\n\nExpected monthly volume:\nMain use case:\n\nThanks`);
12		− return (
13		− <Dialog>
14		− <DialogTrigger asChild>
15		− <Button variant={variant} size="sm" className="w-full">
16		− {isEnterprise ? "Contact sales" : "Request upgrade"}
17		− </Button>
18		− </DialogTrigger>
19		− <DialogContent size="sm">
20		− <DialogHeader>
21		− <DialogTitle>{isEnterprise ? "Talk to us about Enterprise" : `Upgrade to ${planLabel}`}</DialogTitle>
22		− <DialogDescription>
23		− {isEnterprise ? "Custom volumes, SLAs, invoicing and dedicated support." : `${priceLabel}. Billed monthly once checkout is live.`}
24		− </DialogDescription>
25		− </DialogHeader>
26		− <Alert variant="info" title="Self-serve checkout is coming soon">
27		− During the public preview, plan changes are handled by our team. Email us and we will switch your organization to {planLabel} within one business day — no card is charged until billing opens.
28		− </Alert>
29		− <DialogFooter>
30		− <Button asChild variant="primary">
31		− <a href={`mailto:sales@fetcha.co?subject=${subject}&body=${body}`}>
32		− <Mail /> Email sales@fetcha.co
33		− </a>
34		− </Button>
35		− </DialogFooter>
36		− </DialogContent>
37		− </Dialog>
38		− );
39		−}

added apps/web/src/components/dashboard/crawls/cancel-crawl-button.tsx +33 −0

		`@@ -0,0 +1,33 @@`
	1	+"use client";
	2	+import * as React from "react";
	3	+import { useRouter } from "next/navigation";
	4	+import { Ban } from "lucide-react";
	5	+import { cancelCrawl } from "@/actions/crawls";
	6	+import { Button } from "@/components/ui/button";
	7	+
	8	+export function CancelCrawlButton({ id, size = "sm" }: { id: string; size?: "xs" \| "sm" }) {
	9	+ const router = useRouter();
	10	+ const [pending, startTransition] = React.useTransition();
	11	+ const [error, setError] = React.useState<string \| null>(null);
	12	+ return (
	13	+ <span className="inline-flex items-center gap-2">
	14	+ {error ? <span className="text-[11.5px] text-danger">{error}</span> : null}
	15	+ <Button
	16	+ variant="outline"
	17	+ size={size}
	18	+ loading={pending}
	19	+ onClick={() =>
	20	+ startTransition(async () => {
	21	+ setError(null);
	22	+ const res = await cancelCrawl(id);
	23	+ if (!res.ok) setError(res.error.message);
	24	+ else router.refresh();
	25	+ })
	26	+ }
	27	+ aria-label={`Cancel crawl ${id}`}
	28	+ >
	29	+ {!pending ? <Ban className="size-3.5" /> : null} Cancel
	30	+ </Button>
	31	+ </span>
	32	+ );
	33	+}

added apps/web/src/components/dashboard/crawls/crawl-status-badge.tsx +39 −0

		`@@ -0,0 +1,39 @@`
	1	+import { Badge, type BadgeProps } from "@/components/ui/badge";
	2	+
	3	+const VARIANT: Record<string, BadgeProps["variant"]> = {
	4	+ queued: "default",
	5	+ running: "info",
	6	+ completed: "success",
	7	+ failed: "danger",
	8	+ cancelled: "warning",
	9	+};
	10	+
	11	+/** Status badge for crawl jobs (queued / running / completed / failed / cancelled). */
	12	+export function CrawlStatusBadge({ status, className }: { status: string; className?: string }) {
	13	+ return (
	14	+ <Badge variant={VARIANT[status] ?? "default"} dot className={className}>
	15	+ {status}
	16	+ </Badge>
	17	+ );
	18	+}
	19	+
	20	+const PAGE_VARIANT: Record<string, BadgeProps["variant"]> = {
	21	+ success: "success",
	22	+ ok: "success",
	23	+ blocked: "warning",
	24	+ failed: "danger",
	25	+ error: "danger",
	26	+ timeout: "danger",
	27	+ skipped: "outline",
	28	+ pending: "default",
	29	+ queued: "default",
	30	+};
	31	+
	32	+/** Status badge for individual crawled pages. */
	33	+export function CrawlPageStatusBadge({ status }: { status: string }) {
	34	+ return (
	35	+ <Badge variant={PAGE_VARIANT[status] ?? "default"} dot>
	36	+ {status}
	37	+ </Badge>
	38	+ );
	39	+}

added apps/web/src/components/dashboard/crawls/create-crawl-dialog.tsx +201 −0

		`@@ -0,0 +1,201 @@`
	1	+"use client";
	2	+import * as React from "react";
	3	+import { useRouter } from "next/navigation";
	4	+import { Plus } from "lucide-react";
	5	+import { COUNTRIES } from "@fetcha/core/client";
	6	+import { createCrawl } from "@/actions/crawls";
	7	+import { Alert } from "@/components/ui/alert";
	8	+import { Button } from "@/components/ui/button";
	9	+import { Dialog, DialogClose, DialogContent, DialogDescription, DialogFooter, DialogHeader, DialogTitle, DialogTrigger } from "@/components/ui/dialog";
	10	+import { Input, Textarea } from "@/components/ui/input";
	11	+import { Field, Hint, Label } from "@/components/ui/label";
	12	+import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
	13	+import { Switch } from "@/components/ui/switch";
	14	+
	15	+const ANY = "__any__";
	16	+const FORMATS = ["markdown", "text", "html"] as const;
	17	+
	18	+function splitPatterns(s: string): string[] \| undefined {
	19	+ const list = s
	20	+ .split(/[\n,]+/)
	21	+ .map((x) => x.trim())
	22	+ .filter(Boolean)
	23	+ .slice(0, 50);
	24	+ return list.length ? list : undefined;
	25	+}
	26	+
	27	+export function CreateCrawlDialog({ defaultCountry, maxPages, variant = "primary" }: { defaultCountry?: string \| null; maxPages: number; variant?: "primary" \| "outline" }) {
	28	+ const router = useRouter();
	29	+ const [open, setOpen] = React.useState(false);
	30	+ const [pending, startTransition] = React.useTransition();
	31	+ const [error, setError] = React.useState<string \| null>(null);
	32	+
	33	+ const [url, setUrl] = React.useState("");
	34	+ const [label, setLabel] = React.useState("");
	35	+ const [maxPagesValue, setMaxPagesValue] = React.useState(25);
	36	+ const [maxDepth, setMaxDepth] = React.useState(2);
	37	+ const [format, setFormat] = React.useState<(typeof FORMATS)[number]>("markdown");
	38	+ const [sameDomain, setSameDomain] = React.useState(true);
	39	+ const [respectRobots, setRespectRobots] = React.useState(true);
	40	+ const [browser, setBrowser] = React.useState(false);
	41	+ const [country, setCountry] = React.useState<string>(defaultCountry && defaultCountry in COUNTRIES ? defaultCountry : ANY);
	42	+ const [include, setInclude] = React.useState("");
	43	+ const [exclude, setExclude] = React.useState("");
	44	+
	45	+ function reset() {
	46	+ setError(null);
	47	+ setUrl("");
	48	+ setLabel("");
	49	+ setMaxPagesValue(25);
	50	+ setMaxDepth(2);
	51	+ setFormat("markdown");
	52	+ setSameDomain(true);
	53	+ setRespectRobots(true);
	54	+ setBrowser(false);
	55	+ setInclude("");
	56	+ setExclude("");
	57	+ }
	58	+
	59	+ function submit(e: React.FormEvent) {
	60	+ e.preventDefault();
	61	+ setError(null);
	62	+ startTransition(async () => {
	63	+ const res = await createCrawl({
	64	+ url: url.trim(),
	65	+ label: label.trim() \|\| undefined,
	66	+ max_pages: Math.min(maxPages, Math.max(1, Math.round(maxPagesValue \|\| 1))),
	67	+ max_depth: Math.min(10, Math.max(0, Math.round(maxDepth \|\| 0))),
	68	+ format,
	69	+ same_domain: sameDomain,
	70	+ respect_robots: respectRobots,
	71	+ browser,
	72	+ country: country === ANY ? undefined : country,
	73	+ include_patterns: splitPatterns(include),
	74	+ exclude_patterns: splitPatterns(exclude),
	75	+ });
	76	+ if (!res.ok) {
	77	+ setError(res.error.message);
	78	+ return;
	79	+ }
	80	+ setOpen(false);
	81	+ reset();
	82	+ router.push(`/dashboard/crawls/${res.data.id}`);
	83	+ router.refresh();
	84	+ });
	85	+ }
	86	+
	87	+ return (
	88	+ <Dialog
	89	+ open={open}
	90	+ onOpenChange={(o) => {
	91	+ setOpen(o);
	92	+ if (!o) setError(null);
	93	+ }}
	94	+ >
	95	+ <DialogTrigger asChild>
	96	+ <Button variant={variant} size="sm">
	97	+ <Plus /> New crawl
	98	+ </Button>
	99	+ </DialogTrigger>
	100	+ <DialogContent size="lg">
	101	+ <form onSubmit={submit} className="contents">
	102	+ <DialogHeader>
	103	+ <DialogTitle>Start a crawl</DialogTitle>
	104	+ <DialogDescription>
	105	+ Fetcha follows links from the seed URL, fetches each page through the routing engine and stores the content. Every page is a normal request in the log with source <code className="font-mono">crawl</code>.
	106	+ </DialogDescription>
	107	+ </DialogHeader>
	108	+ {error ? <Alert variant="danger">{error}</Alert> : null}
	109	+ <div className="grid gap-4 sm:grid-cols-2">
	110	+ <Field className="sm:col-span-2">
	111	+ <Label htmlFor="c-url">Seed URL</Label>
	112	+ <Input id="c-url" type="url" inputMode="url" required value={url} onChange={(e) => setUrl(e.target.value)} placeholder="https://docs.example.com/" autoComplete="off" autoCapitalize="off" spellCheck={false} className="font-mono text-[13px]" />
	113	+ </Field>
	114	+ <Field>
	115	+ <Label htmlFor="c-max-pages">Max pages</Label>
	116	+ <Input id="c-max-pages" type="number" inputMode="numeric" min={1} max={maxPages} value={maxPagesValue} onChange={(e) => setMaxPagesValue(Number(e.target.value))} className="font-mono tabular" />
	117	+ <Hint>1–{maxPages.toLocaleString("en-US")}. The seed counts as one.</Hint>
	118	+ </Field>
	119	+ <Field>
	120	+ <Label htmlFor="c-max-depth">Max depth</Label>
	121	+ <Input id="c-max-depth" type="number" inputMode="numeric" min={0} max={10} value={maxDepth} onChange={(e) => setMaxDepth(Number(e.target.value))} className="font-mono tabular" />
	122	+ <Hint>0 = seed only, up to 10 link hops.</Hint>
	123	+ </Field>
	124	+ <Field>
	125	+ <Label htmlFor="c-format">Content format</Label>
	126	+ <Select value={format} onValueChange={(v) => setFormat(v as (typeof FORMATS)[number])}>
	127	+ <SelectTrigger id="c-format">
	128	+ <SelectValue />
	129	+ </SelectTrigger>
	130	+ <SelectContent>
	131	+ {FORMATS.map((f) => (
	132	+ <SelectItem key={f} value={f} className="font-mono text-[13px]">
	133	+ {f}
	134	+ </SelectItem>
	135	+ ))}
	136	+ </SelectContent>
	137	+ </Select>
	138	+ </Field>
	139	+ <Field>
	140	+ <Label htmlFor="c-country">Country</Label>
	141	+ <Select value={country} onValueChange={setCountry}>
	142	+ <SelectTrigger id="c-country">
	143	+ <SelectValue />
	144	+ </SelectTrigger>
	145	+ <SelectContent>
	146	+ <SelectItem value={ANY}>Any country</SelectItem>
	147	+ {Object.entries(COUNTRIES).map(([code, name]) => (
	148	+ <SelectItem key={code} value={code}>
	149	+ <span className="font-mono text-[12px] text-fg-subtle">{code}</span> {name}
	150	+ </SelectItem>
	151	+ ))}
	152	+ </SelectContent>
	153	+ </Select>
	154	+ </Field>
	155	+ <Field>
	156	+ <Label htmlFor="c-include">Include patterns</Label>
	157	+ <Textarea id="c-include" value={include} onChange={(e) => setInclude(e.target.value)} placeholder={"/docs/*\n/regex/"} rows={3} spellCheck={false} className="font-mono text-[12.5px]" />
	158	+ <Hint>One per line. Glob with <span className="font-mono">*</span> or <span className="font-mono">/regex/</span>. Empty = everything.</Hint>
	159	+ </Field>
	160	+ <Field>
	161	+ <Label htmlFor="c-exclude">Exclude patterns</Label>
	162	+ <Textarea id="c-exclude" value={exclude} onChange={(e) => setExclude(e.target.value)} placeholder={"/login\n*.pdf"} rows={3} spellCheck={false} className="font-mono text-[12.5px]" />
	163	+ <Hint>URLs matching any pattern are never crawled.</Hint>
	164	+ </Field>
	165	+ <div className="grid gap-3 rounded-md border border-border bg-bg-subtle/50 p-3 sm:col-span-2">
	166	+ <ToggleRow id="c-same-domain" label="Stay on the seed domain" hint="Only follow links on the same registrable host." checked={sameDomain} onChange={setSameDomain} />
	167	+ <ToggleRow id="c-robots" label="Respect robots.txt" hint="Skip paths disallowed for the seed host." checked={respectRobots} onChange={setRespectRobots} />
	168	+ <ToggleRow id="c-browser" label="Browser rendering" hint="Render every page in the managed browser. Slower; use for JavaScript-only sites. Blocked pages escalate automatically either way." checked={browser} onChange={setBrowser} />
	169	+ </div>
	170	+ <Field className="sm:col-span-2">
	171	+ <Label htmlFor="c-label">Label (optional)</Label>
	172	+ <Input id="c-label" value={label} onChange={(e) => setLabel(e.target.value)} placeholder="docs-site-weekly" maxLength={128} autoComplete="off" />
	173	+ </Field>
	174	+ </div>
	175	+ <DialogFooter>
	176	+ <DialogClose asChild>
	177	+ <Button type="button" variant="ghost">
	178	+ Cancel
	179	+ </Button>
	180	+ </DialogClose>
	181	+ <Button type="submit" variant="primary" loading={pending}>
	182	+ Start crawl
	183	+ </Button>
	184	+ </DialogFooter>
	185	+ </form>
	186	+ </DialogContent>
	187	+ </Dialog>
	188	+ );
	189	+}
	190	+
	191	+function ToggleRow({ id, label, hint, checked, onChange }: { id: string; label: string; hint?: string; checked: boolean; onChange: (v: boolean) => void }) {
	192	+ return (
	193	+ <div className="flex items-center justify-between gap-3">
	194	+ <div className="min-w-0">
	195	+ <Label htmlFor={id}>{label}</Label>
	196	+ {hint ? <Hint className="mt-1">{hint}</Hint> : null}
	197	+ </div>
	198	+ <Switch id={id} checked={checked} onCheckedChange={onChange} aria-label={label} />
	199	+ </div>
	200	+ );
	201	+}

added apps/web/src/components/dashboard/crawls/map-tool.tsx +99 −0

		`@@ -0,0 +1,99 @@`
	1	+"use client";
	2	+import * as React from "react";
	3	+import { Map as MapIcon, Search } from "lucide-react";
	4	+import type { MapResult } from "@/lib/api";
	5	+import { runMap } from "@/actions/crawls";
	6	+import { Alert } from "@/components/ui/alert";
	7	+import { Badge } from "@/components/ui/badge";
	8	+import { Button } from "@/components/ui/button";
	9	+import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card";
	10	+import { CopyButton } from "@/components/ui/copy-button";
	11	+import { Input } from "@/components/ui/input";
	12	+import { Field, Hint, Label } from "@/components/ui/label";
	13	+import { formatNumber } from "@/lib/format";
	14	+
	15	+/** Mini tool: discover the URLs of a site (sitemap + links) with POST /v1/map, synchronously. */
	16	+export function MapTool() {
	17	+ const [url, setUrl] = React.useState("");
	18	+ const [search, setSearch] = React.useState("");
	19	+ const [limit, setLimit] = React.useState(200);
	20	+ const [pending, startTransition] = React.useTransition();
	21	+ const [error, setError] = React.useState<string \| null>(null);
	22	+ const [result, setResult] = React.useState<MapResult \| null>(null);
	23	+
	24	+ function submit(e: React.FormEvent) {
	25	+ e.preventDefault();
	26	+ setError(null);
	27	+ startTransition(async () => {
	28	+ const res = await runMap({ url: url.trim(), search: search.trim() \|\| undefined, limit: Math.min(10_000, Math.max(1, Math.round(limit \|\| 1))) });
	29	+ if (!res.ok) {
	30	+ setResult(null);
	31	+ setError(res.error.message);
	32	+ return;
	33	+ }
	34	+ setResult(res.data);
	35	+ });
	36	+ }
	37	+
	38	+ return (
	39	+ <Card>
	40	+ <CardHeader>
	41	+ <CardTitle className="flex items-center gap-2">
	42	+ <MapIcon className="size-4 text-fg-subtle" aria-hidden /> Map a site
	43	+ </CardTitle>
	44	+ <CardDescription>
	45	+ List the URLs of a site from its sitemap and the links on the seed page, without fetching every page. Same as <code className="font-mono">POST /v1/map</code>; runs synchronously (up to 60 s).
	46	+ </CardDescription>
	47	+ </CardHeader>
	48	+ <CardContent className="grid gap-4">
	49	+ <form onSubmit={submit} className="grid gap-3 sm:grid-cols-[minmax(0,1fr)_minmax(0,14rem)_6rem_auto] sm:items-end">
	50	+ <Field>
	51	+ <Label htmlFor="map-url">URL</Label>
	52	+ <Input id="map-url" type="url" inputMode="url" required value={url} onChange={(e) => setUrl(e.target.value)} placeholder="https://example.com" autoComplete="off" autoCapitalize="off" spellCheck={false} className="font-mono text-[13px]" />
	53	+ </Field>
	54	+ <Field>
	55	+ <Label htmlFor="map-search">Filter (optional)</Label>
	56	+ <Input id="map-search" value={search} onChange={(e) => setSearch(e.target.value)} placeholder="blog, /docs/*, /regex/" maxLength={256} autoComplete="off" spellCheck={false} className="font-mono text-[12.5px]" />
	57	+ </Field>
	58	+ <Field>
	59	+ <Label htmlFor="map-limit">Limit</Label>
	60	+ <Input id="map-limit" type="number" inputMode="numeric" min={1} max={10_000} value={limit} onChange={(e) => setLimit(Number(e.target.value))} className="font-mono tabular" />
	61	+ </Field>
	62	+ <Button type="submit" variant="primary" loading={pending} className="sm:mb-px">
	63	+ {!pending ? <Search className="size-3.5" /> : null} Map
	64	+ </Button>
	65	+ </form>
	66	+ {error ? <Alert variant="danger">{error}</Alert> : null}
	67	+ {result ? (
	68	+ <div className="grid gap-2">
	69	+ <div className="flex flex-wrap items-center gap-2 text-[12.5px] text-fg-muted">
	70	+ <span className="font-mono tabular text-fg">{formatNumber(result.count)}</span> URL{result.count === 1 ? "" : "s"}
	71	+ <Badge variant="outline">sitemap {formatNumber(result.sources?.sitemap ?? 0)}</Badge>
	72	+ <Badge variant="outline">links {formatNumber(result.sources?.links ?? 0)}</Badge>
	73	+ {result.truncated ? <Badge variant="warning">truncated at {formatNumber(limit)}</Badge> : null}
	74	+ <span className="ml-auto">
	75	+ <CopyButton value={result.urls.join("\n")} label="Copy list" />
	76	+ </span>
	77	+ </div>
	78	+ {result.urls.length ? (
	79	+ <ol className="max-h-[360px] overflow-auto rounded-lg border border-border bg-bg-subtle p-3 font-mono text-[12px] leading-relaxed scrollbar-thin">
	80	+ {result.urls.map((u, i) => (
	81	+ <li key={`${u}-${i}`} className="flex gap-3">
	82	+ <span className="w-8 shrink-0 select-none text-right text-fg-subtle/70">{i + 1}</span>
	83	+ <a href={u} target="_blank" rel="noreferrer noopener nofollow" className="min-w-0 truncate text-fg hover:text-accent hover:underline" title={u}>
	84	+ {u}
	85	+ </a>
	86	+ </li>
	87	+ ))}
	88	+ </ol>
	89	+ ) : (
	90	+ <Hint>No URLs found. The site may have no sitemap and the seed page no links, or the filter matched nothing.</Hint>
	91	+ )}
	92	+ </div>
	93	+ ) : (
	94	+ <Hint>Use the result to pick include/exclude patterns before starting a crawl.</Hint>
	95	+ )}
	96	+ </CardContent>
	97	+ </Card>
	98	+ );
	99	+}

modified apps/web/src/components/dashboard/sidebar.tsx +2 −1

		`@@ -1,7 +1,7 @@`
1	1	"use client";
2	2	import Link from "next/link";
3	3	import { usePathname } from "next/navigation";
4		−import { Activity, BarChart3, BookOpen, CreditCard, FolderKanban, KeyRound, LayoutDashboard, ListTree, Play, Settings, Shield, Waypoints } from "lucide-react";
	4	+import { Activity, BarChart3, BookOpen, CreditCard, FolderKanban, KeyRound, LayoutDashboard, ListTree, Network, Play, Settings, Shield, Waypoints } from "lucide-react";
5	5	import { cn } from "@/lib/utils";
6	6	import { Logo } from "@/components/ui/logo";
7	7
		`@@ -11,6 +11,7 @@ export const DASHBOARD_NAV = [`
11	11	{ href: "/dashboard/api-keys", label: "API Keys", icon: KeyRound },
12	12	{ href: "/dashboard/projects", label: "Projects", icon: FolderKanban },
13	13	{ href: "/dashboard/requests", label: "Requests", icon: ListTree },
	14	+ { href: "/dashboard/crawls", label: "Crawls", icon: Network },
14	15	{ href: "/dashboard/sessions", label: "Sessions", icon: Waypoints },
15	16	{ href: "/dashboard/usage", label: "Usage", icon: Activity },
16	17	{ href: "/dashboard/analytics", label: "Analytics", icon: BarChart3 },

modified apps/web/src/components/docs/nav-config.ts +2 −1

		`@@ -27,7 +27,8 @@ export const DOCS_NAV: DocsNavSection[] = [`
27	27	{ title: "Network Selection", href: "/docs/networks" },
28	28	{ title: "Geolocation", href: "/docs/geolocation" },
29	29	{ title: "Sessions", href: "/docs/sessions" },
30		− { title: "Browser", href: "/docs/browser", badge: "Coming soon" },
	30	+ { title: "Crawl & Map", href: "/docs/crawl" },
	31	+ { title: "Browser", href: "/docs/browser" },
31	32	{ title: "Extraction", href: "/docs/extraction", badge: "Coming soon" },
32	33	],
33	34	},

modified apps/web/src/components/marketing/architecture-diagram.tsx +2 −2

		`@@ -38,7 +38,7 @@ const NETWORKS: Array<{ label: string; live: boolean }> = [`
38	38	{ label: "Residential", live: true },
39	39	{ label: "ISP", live: false },
40	40	{ label: "Mobile", live: false },
41		− { label: "Browser", live: false },
	41	+ { label: "Browser", live: true },
42	42	];
43	43
44	44	export function ArchitectureDiagram() {
		`@@ -72,7 +72,7 @@ export function ArchitectureDiagram() {`
72	72	</li>
73	73	))}
74	74	</ul>
75		− <p className="mt-2 text-[11px] text-fg-subtle">Residential is live today. Other classes are coming soon; <span className="font-mono">auto</span> resolves to the best available.</p>
	75	+ <p className="mt-2 text-[11px] text-fg-subtle">Residential and the managed browser are live today. Other classes are coming soon; <span className="font-mono">auto</span> resolves to the best available and escalates to the browser on JavaScript challenges.</p>
76	76	</Node>
77	77	<Flow />
78	78	<Node step="04 · Target" title="Global web" subtitle="The page you asked for, from the geography you asked for.">

modified apps/web/src/components/marketing/footer.tsx +1 −1

		`@@ -2,7 +2,7 @@ import Link from "next/link";`
2	2	import { Logo } from "@/components/ui/logo";
3	3
4	4	const columns = [
5		− { title: "Product", links: [["Platform", "/#platform"], ["Playground", "/dashboard/playground"], ["Pricing", "/pricing"], ["Changelog", "/changelog"], ["Status", "/status"]] },
	5	+ { title: "Product", links: [["Platform", "/#platform"], ["Playground", "/dashboard/playground"], ["Access", "/pricing"], ["Changelog", "/changelog"], ["Status", "/status"]] },
6	6	{ title: "Developers", links: [["Documentation", "/docs"], ["Quickstart", "/docs/quickstart"], ["Fetch API", "/docs/fetch"], ["SDKs", "/docs/sdks"], ["Errors", "/docs/errors"]] },
7	7	{ title: "Company", links: [["About", "/#about"], ["Support", "mailto:support@fetcha.co"], ["Contact", "mailto:hello@fetcha.co"]] },
8	8	{ title: "Legal", links: [["Terms of Service", "/legal/terms"], ["Privacy Policy", "/legal/privacy"], ["Acceptable Use", "/legal/acceptable-use"], ["DPA", "/legal/dpa"], ["Cookies", "/legal/cookies"]] },

modified apps/web/src/components/marketing/nav.tsx +5 −5

		`@@ -10,7 +10,7 @@ import { cn } from "@/lib/utils";`
10	10
11	11	const links = [
12	12	{ href: "/#platform", label: "Platform" },
13		− { href: "/pricing", label: "Pricing" },
	13	+ { href: "/pricing", label: "Access" },
14	14	{ href: "/docs", label: "Docs" },
15	15	{ href: "/changelog", label: "Changelog" },
16	16	{ href: "/status", label: "Status" },
		`@@ -42,8 +42,8 @@ export function MarketingNav({ signedIn }: { signedIn: boolean }) {`
42	42	<Button asChild size="sm"><Link href="/dashboard">Dashboard</Link></Button>
43	43	) : (
44	44	<>
45		− <Button asChild variant="ghost" size="sm"><Link href="/login">Log in</Link></Button>
46		− <Button asChild size="sm"><Link href="/signup">Start building</Link></Button>
	45	+ <Button asChild variant="ghost" size="sm"><a href="mailto:hello@fetcha.co?subject=Fetcha%20access">Request access</a></Button>
	46	+ <Button asChild size="sm"><Link href="/login">Log in</Link></Button>
47	47	</>
48	48	)}
49	49	</div>
		`@@ -65,8 +65,8 @@ export function MarketingNav({ signedIn }: { signedIn: boolean }) {`
65	65	<Button asChild className="flex-1"><Link href="/dashboard">Dashboard</Link></Button>
66	66	) : (
67	67	<>
68		− <Button asChild variant="outline" className="flex-1"><Link href="/login">Log in</Link></Button>
69		− <Button asChild className="flex-1"><Link href="/signup">Start building</Link></Button>
	68	+ <Button asChild variant="outline" className="flex-1"><a href="mailto:hello@fetcha.co?subject=Fetcha%20access">Request access</a></Button>
	69	+ <Button asChild className="flex-1"><Link href="/login">Log in</Link></Button>
70	70	</>
71	71	)}
72	72	</div>

modified apps/web/src/components/playground/request-builder.tsx +102 −11

		`@@ -1,6 +1,6 @@`
1	1	"use client";
2	2	import * as React from "react";
3		−import { ChevronRight, History, Play, RotateCcw, Trash2 } from "lucide-react";
	3	+import { ChevronRight, History, MonitorSmartphone, Play, RotateCcw, Trash2 } from "lucide-react";
4	4	import { Badge } from "@/components/ui/badge";
5	5	import { Button } from "@/components/ui/button";
6	6	import { DropdownMenu, DropdownMenuContent, DropdownMenuItem, DropdownMenuLabel, DropdownMenuSeparator, DropdownMenuTrigger } from "@/components/ui/dropdown-menu";
		`@@ -8,12 +8,21 @@ import { Input, Textarea } from "@/components/ui/input";`
8	8	import { Field, Hint, Label } from "@/components/ui/label";
9	9	import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
10	10	import { Switch } from "@/components/ui/switch";
11		−import { SimpleTooltip } from "@/components/ui/tooltip";
12	11	import type { UrlCheck } from "@/lib/playground-url";
13	12	import { cn } from "@/lib/utils";
14	13	import { CountrySelect } from "./country-select";
15	14	import { KvRows } from "./kv-rows";
16		−import { DEVICES, FORMATS, HTTP_METHODS, NETWORKS, NETWORK_LABELS, inspectBody, methodHasBody, type BuilderState, type CountryOption, type HttpMethod, type NetworkClass, type OutputFormat, type PlanSummary } from "./types";
	15	+import { DEVICES, FORMATS, HTTP_METHODS, NETWORKS, NETWORK_LABELS, WAIT_UNTIL, inspectBody, methodHasBody, type BuilderState, type CountryOption, type HttpMethod, type NetworkClass, type OutputFormat, type PlanSummary, type RefererMode, type WaitUntil } from "./types";
	16	+
	17	+const FORMAT_HINTS: Record<OutputFormat, string> = {
	18	+ html: "Body as returned by the target",
	19	+ text: "Readable text, tags removed",
	20	+ markdown: "Main content converted to Markdown",
	21	+ json: "Body plus parsed JSON",
	22	+ raw: "Untouched body (base64 for binary)",
	23	+};
	24	+
	25	+const WAIT_UNTIL_LABELS: Record<WaitUntil, string> = { domcontentloaded: "DOM content loaded (default)", load: "Load event", networkidle: "Network idle" };
17	26
18	27	export interface RequestBuilderProps {
19	28	state: BuilderState;
		`@@ -58,7 +67,7 @@ function Section({ title, count, open, onToggle, children, hint }: { title: stri`
58	67
59	68	export function RequestBuilder(p: RequestBuilderProps) {
60	69	const { state: s, onChange, plan, availableNetworks, countries, urlCheck, urlTouched, onUrlBlur, onRun, onReset, pending, recent, onPickRecent, onClearRecent, isMac } = p;
61		− const [open, setOpen] = React.useState({ headers: s.headers.length > 0, query: s.query.length > 0, body: s.body.length > 0, cookies: s.cookies.length > 0, options: true });
	70	+ const [open, setOpen] = React.useState({ headers: s.headers.length > 0, query: s.query.length > 0, body: s.body.length > 0, cookies: s.cookies.length > 0, options: true, browser: s.browser });
62	71	const toggle = (k: keyof typeof open) => setOpen((o) => ({ ...o, [k]: !o[k] }));
63	72	const bodyState = inspectBody(s.body);
64	73	const showUrlMessage = urlTouched && urlCheck.level !== "ok";
		`@@ -251,6 +260,7 @@ export function RequestBuilder(p: RequestBuilderProps) {`
251	260	{FORMATS.map((f) => (
252	261	<SelectItem key={f} value={f} className="font-mono text-[13px]">
253	262	{f}
	263	+ <span className="ml-1.5 font-sans text-xs text-fg-subtle">— {FORMAT_HINTS[f]}</span>
254	264	</SelectItem>
255	265	))}
256	266	</SelectContent>
		`@@ -277,17 +287,98 @@ export function RequestBuilder(p: RequestBuilderProps) {`
277	287	<Input id="pg-locale" value={s.locale} onChange={(e) => onChange({ locale: e.target.value })} placeholder="en-CA" maxLength={16} autoCapitalize="off" spellCheck={false} className="font-mono text-[12.5px]" />
278	288	</Field>
279	289	</div>
	290	+ <div className="grid gap-4 sm:grid-cols-2">
	291	+ <Field>
	292	+ <Label htmlFor="pg-referer">Referer</Label>
	293	+ <Select value={s.refererMode} onValueChange={(v) => onChange({ refererMode: v as RefererMode })}>
	294	+ <SelectTrigger id="pg-referer" aria-label="Referer strategy">
	295	+ <SelectValue />
	296	+ </SelectTrigger>
	297	+ <SelectContent>
	298	+ <SelectItem value="auto">Auto (search-engine referer on retries)</SelectItem>
	299	+ <SelectItem value="none">None</SelectItem>
	300	+ <SelectItem value="custom">Custom URL</SelectItem>
	301	+ </SelectContent>
	302	+ </Select>
	303	+ </Field>
	304	+ {s.refererMode === "custom" ? (
	305	+ <Field>
	306	+ <Label htmlFor="pg-referer-url">Referer URL</Label>
	307	+ <Input id="pg-referer-url" type="url" inputMode="url" value={s.refererUrl} onChange={(e) => onChange({ refererUrl: e.target.value })} placeholder="https://www.google.com/" maxLength={2048} autoCapitalize="off" spellCheck={false} className="font-mono text-[12.5px]" />
	308	+ </Field>
	309	+ ) : null}
	310	+ </div>
280	311	<div className="grid gap-3 rounded-md border border-border bg-bg-subtle/50 p-3">
281	312	<ToggleRow id="pg-follow" label="Follow redirects" hint="Each hop is validated against the URL policy." checked={s.followRedirects} onChange={(v) => onChange({ followRedirects: v })} />
	313	+ <ToggleRow id="pg-links" label="Include links" hint="Return every hyperlink of the page as absolute URLs in links[]." checked={s.links} onChange={(v) => onChange({ links: v })} />
282	314	<ToggleRow id="pg-debug" label="Debug" hint="Include per-attempt routing details in the response." checked={s.debug} onChange={(v) => onChange({ debug: v })} />
283		− <SimpleTooltip content="Managed browser rendering is not available yet.">
284		− <div>
285		− <ToggleRow id="pg-browser" label="Browser rendering" hint="Execute JavaScript in a managed browser." checked={false} onChange={() => {}} disabled badge="Coming soon" />
286		− </div>
287		− </SimpleTooltip>
288	315	</div>
289	316	</div>
290	317	</Section>
	318	+
	319	+ <Section title="Browser rendering" open={open.browser} onToggle={() => toggle("browser")} hint={s.browser ? "on" : s.browserFallback ? "fallback" : "off"}>
	320	+ <div className="grid gap-4">
	321	+ <div className="grid gap-3 rounded-md border border-border bg-bg-subtle/50 p-3">
	322	+ <ToggleRow
	323	+ id="pg-browser"
	324	+ label={
	325	+ <span className="inline-flex items-center gap-1.5">
	326	+ <MonitorSmartphone className="size-3.5 text-fg-subtle" aria-hidden /> Render in a managed browser
	327	+ </span>
	328	+ }
	329	+ hint="Real Chromium routed through the same network, geography and session. Returns the DOM after the page settles."
	330	+ checked={s.browser}
	331	+ onChange={(v) => onChange({ browser: v })}
	332	+ />
	333	+ <ToggleRow id="pg-browser-fallback" label="Escalate on JavaScript challenges" hint="When an HTTP attempt is blocked by a JS challenge or anti-bot page, retry automatically in the browser." checked={s.browserFallback} onChange={(v) => onChange({ browserFallback: v })} />
	334	+ </div>
	335	+ <div className="grid gap-4 sm:grid-cols-2">
	336	+ <Field>
	337	+ <Label htmlFor="pg-wait-for">Wait for selector</Label>
	338	+ <Input id="pg-wait-for" value={s.waitFor} onChange={(e) => onChange({ waitFor: e.target.value })} placeholder="e.g. table.results" maxLength={512} autoCapitalize="off" spellCheck={false} className="font-mono text-[12.5px]" />
	339	+ <Hint>CSS selector that must be present before capture.</Hint>
	340	+ </Field>
	341	+ <Field>
	342	+ <Label htmlFor="pg-wait-ms">Extra settle time (ms)</Label>
	343	+ <Input
	344	+ id="pg-wait-ms"
	345	+ type="number"
	346	+ inputMode="numeric"
	347	+ min={0}
	348	+ max={30_000}
	349	+ step={100}
	350	+ value={s.waitMs}
	351	+ onChange={(e) => onChange({ waitMs: Number(e.target.value) })}
	352	+ onBlur={() => onChange({ waitMs: Math.min(30_000, Math.max(0, Math.round(s.waitMs \|\| 0))) })}
	353	+ className="font-mono tabular text-[12.5px]"
	354	+ />
	355	+ <Hint>0–30,000 ms after load or after the selector appears.</Hint>
	356	+ </Field>
	357	+ </div>
	358	+ <Field>
	359	+ <Label htmlFor="pg-wait-until">Navigation wait condition</Label>
	360	+ <Select value={s.waitUntil} onValueChange={(v) => onChange({ waitUntil: v as WaitUntil })}>
	361	+ <SelectTrigger id="pg-wait-until" aria-label="Navigation wait condition">
	362	+ <SelectValue />
	363	+ </SelectTrigger>
	364	+ <SelectContent>
	365	+ {WAIT_UNTIL.map((w) => (
	366	+ <SelectItem key={w} value={w}>
	367	+ {WAIT_UNTIL_LABELS[w]}
	368	+ </SelectItem>
	369	+ ))}
	370	+ </SelectContent>
	371	+ </Select>
	372	+ </Field>
	373	+ <div className="grid gap-3 rounded-md border border-border bg-bg-subtle/50 p-3">
	374	+ <ToggleRow id="pg-block-resources" label="Block images, fonts and media" hint="Saves bandwidth and time; page scripts still run." checked={s.blockResources} onChange={(v) => onChange({ blockResources: v })} />
	375	+ <ToggleRow id="pg-screenshot" label="Screenshot" hint="Return a PNG of the viewport (base64) in screenshot." checked={s.screenshot} onChange={(v) => onChange({ screenshot: v })} />
	376	+ </div>
	377	+ <Hint>
	378	+ Up to {plan.browser_concurrency} concurrent renders. A page that does not settle within the timeout returns <span className="font-mono">BROWSER_TIMEOUT</span>.
	379	+ </Hint>
	380	+ </div>
	381	+ </Section>
291	382	</div>
292	383
293	384	{/* Actions (desktop) */}
		`@@ -308,7 +399,7 @@ export function RequestBuilder(p: RequestBuilderProps) {`
308	399	);
309	400	}
310	401
311		−function ToggleRow({ id, label, hint, checked, onChange, disabled, badge }: { id: string; label: string; hint?: string; checked: boolean; onChange: (v: boolean) => void; disabled?: boolean; badge?: string }) {
	402	+function ToggleRow({ id, label, hint, checked, onChange, disabled, badge }: { id: string; label: React.ReactNode; hint?: string; checked: boolean; onChange: (v: boolean) => void; disabled?: boolean; badge?: string }) {
312	403	return (
313	404	<div className="flex items-center justify-between gap-3">
314	405	<div className="min-w-0">
		`@@ -318,7 +409,7 @@ function ToggleRow({ id, label, hint, checked, onChange, disabled, badge }: { id`
318	409	</Label>
319	410	{hint ? <Hint className="mt-1">{hint}</Hint> : null}
320	411	</div>
321		− <Switch id={id} checked={checked} onCheckedChange={onChange} disabled={disabled} aria-label={label} />
	412	+ <Switch id={id} checked={checked} onCheckedChange={onChange} disabled={disabled} aria-label={typeof label === "string" ? label : undefined} />
322	413	</div>
323	414	);
324	415	}

modified apps/web/src/components/playground/results-panel.tsx +124 −3

		`@@ -2,7 +2,7 @@`
2	2	import * as React from "react";
3	3	import Link from "next/link";
4	4	import type { FetchRequestInput, FetchResponseBody } from "@fetcha/core/client";
5		−import { ArrowUpRight, Download, Loader2, PlayCircle, ShieldAlert } from "lucide-react";
	5	+import { ArrowUpRight, Download, Loader2, MonitorSmartphone, PlayCircle, ShieldAlert } from "lucide-react";
6	6	import type { PlaygroundError } from "@/actions/playground";
7	7	import { Alert } from "@/components/ui/alert";
8	8	import { Badge } from "@/components/ui/badge";
		`@@ -172,11 +172,18 @@ function SuccessView({ result: r, request, elapsedMs, onSaveCurl, code }: Result`
172	172	const clone: Record<string, unknown> = { ...r };
173	173	if (typeof r.content === "string") clone.content = `[omitted: ${r.content.length.toLocaleString("en-US")} chars — see the ${html ? "HTML" : "Body"} tab]`;
174	174	if (typeof r.text === "string") clone.text = `[omitted: ${r.text.length.toLocaleString("en-US")} chars — see the Text tab]`;
	175	+ if (typeof r.markdown === "string") clone.markdown = `[omitted: ${r.markdown.length.toLocaleString("en-US")} chars — see the Markdown tab]`;
	176	+ if (typeof r.screenshot === "string") clone.screenshot = `[omitted: ${r.screenshot.length.toLocaleString("en-US")} base64 chars — see the Screenshot tab]`;
	177	+ if (Array.isArray(r.links)) clone.links = `[${r.links.length.toLocaleString("en-US")} links — see the Links tab]`;
175	178	if (r.json !== undefined) clone.json = "[see the JSON tab]";
176	179	return JSON.stringify(clone, null, 2);
177	180	}, [r, html]);
178	181
179	182	const bodyLabel = html ? "HTML" : "Body";
	183	+ const hasMarkdown = typeof r.markdown === "string";
	184	+ const links = Array.isArray(r.links) ? r.links : null;
	185	+ const hasScreenshot = typeof r.screenshot === "string" && r.screenshot.length > 0;
	186	+ const mode = meta.mode ?? "http";
180	187
181	188	return (
182	189	<div className="grid gap-4">
		`@@ -195,6 +202,10 @@ function SuccessView({ result: r, request, elapsedMs, onSaveCurl, code }: Result`
195	202	blocked
196	203	</Badge>
197	204	)}
	205	+ <Badge variant={mode === "browser" ? "accent" : "outline"} title={mode === "browser" ? "Rendered in the managed browser" : "Plain HTTP fetch"}>
	206	+ {mode === "browser" ? <MonitorSmartphone className="size-3" aria-hidden /> : null}
	207	+ {mode === "browser" ? "Browser" : "HTTP"}
	208	+ </Badge>
198	209	{meta.cached ? <Badge variant="outline">cached</Badge> : null}
199	210	<span className="hidden text-fg-subtle sm:inline">·</span>
200	211	<Metric label="Duration" value={formatMs(meta.duration_ms ?? elapsedMs)} />
		`@@ -230,18 +241,27 @@ function SuccessView({ result: r, request, elapsedMs, onSaveCurl, code }: Result`
230	241	{!r.success ? (
231	242	<p className="flex items-start gap-2 text-xs text-fg-muted">
232	243	<ShieldAlert className="mt-0.5 size-3.5 shrink-0 text-danger" aria-hidden />
233		− The target answered with HTTP {r.status} on every route Fetcha tried ({meta.attempts} attempt{meta.attempts === 1 ? "" : "s"}). Try a residential network, a different country or a session.
	244	+ The target answered with HTTP {r.status} on every route Fetcha tried ({meta.attempts} attempt{meta.attempts === 1 ? "" : "s"}). Try browser rendering, a residential network, a different country or a session.
234	245	</p>
235	246	) : null}
236	247	</div>
237	248
	249	+ {r.page ? <PageCard page={r.page} /> : null}
	250	+
238	251	{/* Tabs */}
239	252	<Tabs value={tab} onValueChange={setTab}>
240	253	<TabsList variant="underline" className="-mx-1 px-1">
241	254	<TabsTrigger value="preview">Preview</TabsTrigger>
242	255	<TabsTrigger value="html">{bodyLabel}</TabsTrigger>
243	256	<TabsTrigger value="text">Text</TabsTrigger>
	257	+ {hasMarkdown ? <TabsTrigger value="markdown">Markdown</TabsTrigger> : null}
244	258	<TabsTrigger value="json">JSON</TabsTrigger>
	259	+ {links ? (
	260	+ <TabsTrigger value="links">
	261	+ Links <Count n={links.length} />
	262	+ </TabsTrigger>
	263	+ ) : null}
	264	+ {hasScreenshot ? <TabsTrigger value="screenshot">Screenshot</TabsTrigger> : null}
245	265	<TabsTrigger value="headers">
246	266	Headers <Count n={headerEntries.length} />
247	267	</TabsTrigger>
		`@@ -273,6 +293,60 @@ function SuccessView({ result: r, request, elapsedMs, onSaveCurl, code }: Result`
273	293
274	294	<TabsContent value="text">{!textValue ? <NoBody isHead={isHead} label="No text content." /> : <TextBlock text={textValue} />}</TabsContent>
275	295
	296	+ {hasMarkdown ? (
	297	+ <TabsContent value="markdown">
	298	+ {!r.markdown ? <NoBody isHead={isHead} label="No Markdown content." /> : <TextBlock text={r.markdown} />}
	299	+ <p className="mt-2 text-xs text-fg-subtle">Main content first, navigation and boilerplate removed. Shown as plain text.</p>
	300	+ </TabsContent>
	301	+ ) : null}
	302	+
	303	+ {links ? (
	304	+ <TabsContent value="links">
	305	+ <div className="rounded-lg border border-border">
	306	+ <Table>
	307	+ <TableHeader>
	308	+ <TableRow>
	309	+ <TableHead>URL</TableHead>
	310	+ <TableHead>Text</TableHead>
	311	+ <TableHead>Scope</TableHead>
	312	+ <TableHead>Rel</TableHead>
	313	+ </TableRow>
	314	+ </TableHeader>
	315	+ <TableBody>
	316	+ {links.length === 0 ? <TableEmpty colSpan={4}>No hyperlinks found in the page.</TableEmpty> : null}
	317	+ {links.slice(0, 500).map((l, i) => (
	318	+ <TableRow key={`${l.url}-${i}`}>
	319	+ <TableCell className="max-w-[22rem] truncate font-mono text-[12px]" title={l.url}>
	320	+ <a href={l.url} target="_blank" rel="noreferrer noopener nofollow" className="text-accent underline-offset-4 hover:underline">
	321	+ {l.url}
	322	+ </a>
	323	+ </TableCell>
	324	+ <TableCell className="max-w-[14rem] truncate text-[12.5px] text-fg-muted" title={l.text}>
	325	+ {l.text \|\| <span className="text-fg-subtle">—</span>}
	326	+ </TableCell>
	327	+ <TableCell>
	328	+ <Badge variant={l.internal ? "default" : "outline"}>{l.internal ? "internal" : "external"}</Badge>
	329	+ </TableCell>
	330	+ <TableCell className="text-[12px] text-fg-muted">{l.nofollow ? "nofollow" : "—"}</TableCell>
	331	+ </TableRow>
	332	+ ))}
	333	+ </TableBody>
	334	+ </Table>
	335	+ </div>
	336	+ {links.length > 500 ? <p className="mt-2 text-xs text-fg-muted">Showing the first 500 of {links.length.toLocaleString("en-US")} links.</p> : null}
	337	+ </TabsContent>
	338	+ ) : null}
	339	+
	340	+ {hasScreenshot ? (
	341	+ <TabsContent value="screenshot">
	342	+ <div className="grid gap-2">
	343	+ {/* eslint-disable-next-line @next/next/no-img-element -- data URL from the API, not an optimisable asset */}
	344	+ <img src={`data:image/png;base64,${r.screenshot}`} alt="Screenshot of the rendered page" className="w-full rounded-lg border border-border bg-white" />
	345	+ <p className="text-xs text-fg-subtle">PNG captured after the page settled in the managed browser.</p>
	346	+ </div>
	347	+ </TabsContent>
	348	+ ) : null}
	349	+
276	350	<TabsContent value="json">
277	351	{jsonValue.ok ? (
278	352	<JsonBlock value={jsonValue.value} />
		`@@ -349,6 +423,45 @@ function SuccessView({ result: r, request, elapsedMs, onSaveCurl, code }: Result`
349	423	);
350	424	}
351	425
	426	+function PageCard({ page }: { page: NonNullable<FetchResponseBody["page"]> }) {
	427	+ const rows: Array<{ label: string; value: string \| null; mono?: boolean }> = [
	428	+ { label: "Title", value: page.title },
	429	+ { label: "Description", value: page.description },
	430	+ { label: "Canonical", value: page.canonical, mono: true },
	431	+ { label: "Language", value: page.lang, mono: true },
	432	+ { label: "Links", value: Number.isFinite(page.links_count) ? page.links_count.toLocaleString("en-US") : null, mono: true },
	433	+ ];
	434	+ const og = Object.entries(page.og ?? {});
	435	+ return (
	436	+ <section className="rounded-lg border border-border px-4 py-3" aria-label="Page metadata">
	437	+ <h2 className="text-[11.5px] font-semibold uppercase tracking-wide text-fg-subtle">Page</h2>
	438	+ <dl className="mt-2 grid gap-x-6 gap-y-1.5 text-[12.5px] sm:grid-cols-[auto_minmax(0,1fr)]">
	439	+ {rows.map((row) => (
	440	+ <React.Fragment key={row.label}>
	441	+ <dt className="text-fg-subtle">{row.label}</dt>
	442	+ <dd className={cn("min-w-0 break-words", row.mono && "font-mono", !row.value && "text-fg-subtle")} title={row.value ?? undefined}>
	443	+ {row.value \|\| "—"}
	444	+ </dd>
	445	+ </React.Fragment>
	446	+ ))}
	447	+ </dl>
	448	+ {og.length ? (
	449	+ <details className="mt-2 text-xs text-fg-muted">
	450	+ <summary className="cursor-pointer select-none text-fg-subtle hover:text-fg">Open Graph ({og.length})</summary>
	451	+ <dl className="mt-1.5 grid gap-x-4 gap-y-1 sm:grid-cols-[auto_minmax(0,1fr)]">
	452	+ {og.map(([k, v]) => (
	453	+ <React.Fragment key={k}>
	454	+ <dt className="font-mono text-fg-subtle">{k}</dt>
	455	+ <dd className="min-w-0 break-words">{v}</dd>
	456	+ </React.Fragment>
	457	+ ))}
	458	+ </dl>
	459	+ </details>
	460	+ ) : null}
	461	+ </section>
	462	+ );
	463	+}
	464	+
352	465	function Metric({ label, value }: { label: string; value: string }) {
353	466	return (
354	467	<span className="inline-flex items-baseline gap-1 text-[12.5px]">
		`@@ -414,6 +527,7 @@ function NetworkTab({ result: r, request }: { result: FetchResponseBody; request`
414	527	const rows: Array<{ label: string; value: string; muted?: boolean }> = [
415	528	{ label: "Requested network", value: request.network ?? "auto" },
416	529	{ label: "Network used", value: m.network },
	530	+ { label: "Mode", value: m.mode === "browser" ? "browser (rendered)" : "http" },
417	531	{ label: "Country", value: m.country ?? "any" },
418	532	{ label: "Region / city", value: geo \|\| "—", muted: !geo },
419	533	{ label: "Attempts", value: String(m.attempts) },
		`@@ -441,6 +555,7 @@ function NetworkTab({ result: r, request }: { result: FetchResponseBody; request`
441	555	<TableHead>#</TableHead>
442	556	<TableHead>Route</TableHead>
443	557	<TableHead>Network</TableHead>
	558	+ <TableHead>Mode</TableHead>
444	559	<TableHead>Country</TableHead>
445	560	<TableHead>Outcome</TableHead>
446	561	<TableHead>Status</TableHead>
		`@@ -453,11 +568,17 @@ function NetworkTab({ result: r, request }: { result: FetchResponseBody; request`
453	568	<TableCell className="font-mono tabular text-fg-muted">{i + 1}</TableCell>
454	569	<TableCell className="font-mono text-[12.5px]">{a.provider}</TableCell>
455	570	<TableCell className="font-mono text-[12.5px]">{a.network}</TableCell>
	571	+ <TableCell className="font-mono text-[12.5px]">{a.mode ?? "http"}</TableCell>
456	572	<TableCell className="font-mono text-[12.5px]">{a.country ?? "any"}</TableCell>
457	573	<TableCell>
458	574	<Badge variant={a.outcome === "success" ? "success" : a.outcome === "blocked" ? "warning" : "danger"} dot>
459	575	{a.outcome}
460	576	</Badge>
	577	+ {a.block_reason ? (
	578	+ <span className="ml-2 font-mono text-[11.5px] text-fg-muted" title="Block reason">
	579	+ {a.block_reason}
	580	+ </span>
	581	+ ) : null}
461	582	{a.error ? <span className="ml-2 text-xs text-fg-muted">{a.error}</span> : null}
462	583	</TableCell>
463	584	<TableCell className="font-mono tabular">{a.status ?? "—"}</TableCell>
		`@@ -475,7 +596,7 @@ function NetworkTab({ result: r, request }: { result: FetchResponseBody; request`
475	596	<div className={cn("rounded-lg border border-border bg-bg-subtle/50 p-4 text-[13px] leading-relaxed text-fg-muted")}>
476	597	<p className="font-medium text-fg">How routing works</p>
477	598	<p className="mt-1">
478		− In <span className="font-mono">auto</span> mode Fetcha scores every available route for this domain — historical success (35%), cost (20%), latency (15%), network health (15%), geography (10%) and session stability (5%) — starts with the cheapest network likely to succeed and escalates to premium networks when the target blocks or times out. Each retry uses a new IP. Requesting a specific network pins the first attempt to that class.
	599	+ In <span className="font-mono">auto</span> mode Fetcha scores every available route for this domain — historical success (35%), cost (20%), latency (15%), network health (15%), geography (10%) and session stability (5%) — starts with the cheapest network likely to succeed and escalates to premium networks when the target blocks or times out. Each retry uses a new IP. When an attempt is blocked by a JavaScript challenge and browser fallback is on, the next attempt is rendered in the managed browser. Requesting a specific network pins the first attempt to that class.
479	600	</p>
480	601	</div>
481	602	</div>

modified apps/web/src/components/playground/types.ts +54 −1

		`@@ -13,12 +13,18 @@ export const NETWORKS = ["auto", "datacenter", "residential", "isp", "mobile"] a`
13	13	export type NetworkClass = (typeof NETWORKS)[number];
14	14	export const NETWORK_LABELS: Record<NetworkClass, string> = { auto: "Auto (recommended)", datacenter: "Datacenter", residential: "Residential", isp: "ISP", mobile: "Mobile" };
15	15
16		−export const FORMATS = ["html", "text", "json", "raw"] as const;
	16	+export const FORMATS = ["html", "text", "markdown", "json", "raw"] as const;
17	17	export type OutputFormat = (typeof FORMATS)[number];
18	18
19	19	export const DEVICES = ["desktop", "mobile", "tablet"] as const;
20	20	export type Device = (typeof DEVICES)[number];
21	21
	22	+export const WAIT_UNTIL = ["domcontentloaded", "load", "networkidle"] as const;
	23	+export type WaitUntil = (typeof WAIT_UNTIL)[number];
	24	+
	25	+export const REFERER_MODES = ["auto", "none", "custom"] as const;
	26	+export type RefererMode = (typeof REFERER_MODES)[number];
	27	+
22	28	export interface KV {
23	29	id: string;
24	30	key: string;
		`@@ -43,6 +49,19 @@ export interface BuilderState {`
43	49	debug: boolean;
44	50	device: Device \| "";
45	51	locale: string;
	52	+ /** Managed browser rendering. */
	53	+ browser: boolean;
	54	+ /** Escalate to the browser automatically when an HTTP attempt is blocked by a JS challenge. */
	55	+ browserFallback: boolean;
	56	+ waitFor: string;
	57	+ waitMs: number; // 0 = not set
	58	+ waitUntil: WaitUntil;
	59	+ blockResources: boolean;
	60	+ screenshot: boolean;
	61	+ /** Return `links[]` with the response. */
	62	+ links: boolean;
	63	+ refererMode: RefererMode;
	64	+ refererUrl: string;
46	65	}
47	66
48	67	export interface PlaygroundKey {
		`@@ -59,6 +78,8 @@ export interface PlanSummary {`
59	78	max_timeout_ms: number;
60	79	max_retries: number;
61	80	networks: string[];
	81	+ /** Max concurrent browser renders for the organization. */
	82	+ browser_concurrency: number;
62	83	}
63	84
64	85	export interface CountryOption {
		`@@ -104,6 +125,16 @@ export const DEFAULT_STATE: BuilderState = {`
104	125	debug: false,
105	126	device: "",
106	127	locale: "",
	128	+ browser: false,
	129	+ browserFallback: true,
	130	+ waitFor: "",
	131	+ waitMs: 0,
	132	+ waitUntil: "domcontentloaded",
	133	+ blockResources: true,
	134	+ screenshot: false,
	135	+ links: false,
	136	+ refererMode: "auto",
	137	+ refererUrl: "",
107	138	};
108	139
109	140	export function methodHasBody(m: HttpMethod): boolean {
		`@@ -140,6 +171,18 @@ export function toFetchRequest(s: BuilderState, maxTimeoutMs?: number): FetchReq`
140	171	if (s.debug) req.debug = true;
141	172	if (s.device) req.device = s.device;
142	173	if (s.locale.trim()) req.locale = s.locale.trim();
	174	+ // Browser rendering. The wait/resource/screenshot options also apply to renders reached through
	175	+ // automatic escalation, so they are emitted whenever they differ from the API defaults.
	176	+ if (s.browser) req.browser = true;
	177	+ if (!s.browserFallback) req.browser_fallback = false;
	178	+ if (s.waitFor.trim()) req.wait_for = s.waitFor.trim().slice(0, 512);
	179	+ if (s.waitMs > 0) req.wait_ms = Math.min(30_000, Math.round(s.waitMs));
	180	+ if (s.waitUntil !== "domcontentloaded") req.wait_until = s.waitUntil;
	181	+ if (!s.blockResources) req.block_resources = false;
	182	+ if (s.screenshot) req.screenshot = true;
	183	+ if (s.links) req.links = true;
	184	+ if (s.refererMode === "none") req.referer = "none";
	185	+ else if (s.refererMode === "custom" && s.refererUrl.trim()) req.referer = s.refererUrl.trim();
143	186	return req;
144	187	}
145	188
		`@@ -188,6 +231,16 @@ export function sanitizeState(input: unknown): Partial<BuilderState> {`
188	231	if (typeof s.debug === "boolean") out.debug = s.debug;
189	232	if (s.device === "" \|\| (typeof s.device === "string" && (DEVICES as readonly string[]).includes(s.device))) out.device = s.device as Device \| "";
190	233	if (typeof s.locale === "string") out.locale = s.locale.slice(0, 16);
	234	+ if (typeof s.browser === "boolean") out.browser = s.browser;
	235	+ if (typeof s.browserFallback === "boolean") out.browserFallback = s.browserFallback;
	236	+ if (typeof s.waitFor === "string") out.waitFor = s.waitFor.slice(0, 512);
	237	+ if (typeof s.waitMs === "number" && Number.isFinite(s.waitMs)) out.waitMs = Math.min(30_000, Math.max(0, Math.round(s.waitMs)));
	238	+ if (typeof s.waitUntil === "string" && (WAIT_UNTIL as readonly string[]).includes(s.waitUntil)) out.waitUntil = s.waitUntil as WaitUntil;
	239	+ if (typeof s.blockResources === "boolean") out.blockResources = s.blockResources;
	240	+ if (typeof s.screenshot === "boolean") out.screenshot = s.screenshot;
	241	+ if (typeof s.links === "boolean") out.links = s.links;
	242	+ if (typeof s.refererMode === "string" && (REFERER_MODES as readonly string[]).includes(s.refererMode)) out.refererMode = s.refererMode as RefererMode;
	243	+ if (typeof s.refererUrl === "string") out.refererUrl = s.refererUrl.slice(0, 2048);
191	244	return out;
192	245	}
193	246

added apps/web/src/lib/access.ts +20 −0

		`@@ -0,0 +1,20 @@`
	1	+/**
	2	+ * Access-model helpers shared by Better Auth hooks, the session layer and the admin UI.
	3	+ * Pure functions only (no "server-only", no database) so they can be imported anywhere on the server.
	4	+ */
	5	+
	6	+/** Emails listed in `ADMIN_EMAILS` (comma-separated, case-insensitive). */
	7	+export const adminEmails: readonly string[] = (process.env.ADMIN_EMAILS ?? "")
	8	+ .split(",")
	9	+ .map((s) => s.trim().toLowerCase())
	10	+ .filter(Boolean);
	11	+
	12	+export function normalizeEmail(email: string): string {
	13	+ return email.trim().toLowerCase();
	14	+}
	15	+
	16	+export function isAdminEmail(email: string): boolean {
	17	+ return adminEmails.includes(normalizeEmail(email));
	18	+}
	19	+
	20	+export const ACCESS_DENIED_MESSAGE = "This email is not on the access list. Ask your Fetcha administrator to invite you.";

modified apps/web/src/lib/api.ts +88 −0

		`@@ -55,6 +55,81 @@ async function call<T>(path: string, init: RequestInit & { timeoutMs?: number }`
55	55	}
56	56	}
57	57
	58	+// ---------------------------------------------------------------------------
	59	+// Crawl & map (section 3 of docs/API-v0.2.md). Provider details are never included.
	60	+// ---------------------------------------------------------------------------
	61	+
	62	+export type CrawlJobStatus = "queued" \| "running" \| "completed" \| "failed" \| "cancelled";
	63	+
	64	+export interface CrawlJobStats {
	65	+ discovered: number;
	66	+ fetched: number;
	67	+ ok: number;
	68	+ blocked: number;
	69	+ failed: number;
	70	+ bytes: number;
	71	+}
	72	+
	73	+export interface CrawlJob {
	74	+ id: string;
	75	+ status: CrawlJobStatus;
	76	+ label: string \| null;
	77	+ seed_url: string;
	78	+ domain: string;
	79	+ options: Record<string, unknown>;
	80	+ stats: CrawlJobStats;
	81	+ error: { code: string; message: string } \| null;
	82	+ created_at: string;
	83	+ started_at: string \| null;
	84	+ completed_at: string \| null;
	85	+}
	86	+
	87	+export interface CrawlPage {
	88	+ id: string;
	89	+ url: string;
	90	+ final_url: string \| null;
	91	+ depth: number;
	92	+ status: string;
	93	+ http_status: number \| null;
	94	+ error_code: string \| null;
	95	+ title: string \| null;
	96	+ description: string \| null;
	97	+ content_type: string \| null;
	98	+ content: string \| null;
	99	+ links_count: number \| null;
	100	+ bytes: number \| null;
	101	+ duration_ms: number \| null;
	102	+ mode: "http" \| "browser" \| null;
	103	+ fetched_at: string \| null;
	104	+}
	105	+
	106	+export interface CrawlPagesPage {
	107	+ data: CrawlPage[];
	108	+ next_cursor: string \| null;
	109	+}
	110	+
	111	+export interface MapResult {
	112	+ url: string;
	113	+ count: number;
	114	+ urls: string[];
	115	+ sources: { sitemap: number; links: number };
	116	+ truncated: boolean;
	117	+}
	118	+
	119	+export interface BrowserStatus {
	120	+ enabled: boolean;
	121	+ running: number;
	122	+ capacity: number;
	123	+ queue: number;
	124	+}
	125	+
	126	+function qs(params: Record<string, string \| number \| undefined \| null>): string {
	127	+ const sp = new URLSearchParams();
	128	+ for (const [k, v] of Object.entries(params)) if (v !== undefined && v !== null && v !== "") sp.set(k, String(v));
	129	+ const s = sp.toString();
	130	+ return s ? `?${s}` : "";
	131	+}
	132	+
58	133	export const internalApi = {
59	134	health: () => call<{ status: string; version: string }>("/health", { timeoutMs: 5000 }),
60	135	ready: () => call<{ status: string; checks: Record<string, boolean>; available_networks: string[] }>("/ready", { timeoutMs: 8000 }).catch((e) => (e instanceof InternalApiError && e.status === 503 ? { status: "degraded", checks: {}, available_networks: [] } : Promise.reject(e))),
		`@@ -82,4 +157,17 @@ export const internalApi = {`
82	157	reloadProviders: () => call<{ ok: boolean }>("/internal/providers/reload", { method: "POST" }),
83	158	resetCircuit: (key?: string) => call<{ ok: boolean }>("/internal/circuits/reset", { method: "POST", body: JSON.stringify({ key }) }),
84	159	invalidateKey: (keyHash: string) => call<{ ok: boolean }>("/internal/keys/invalidate", { method: "POST", body: JSON.stringify({ key_hash: keyHash }) }).catch(() => ({ ok: false })),
	160	+
	161	+ // Crawl & map
	162	+ createCrawl: (projectId: string, userId: string, options: unknown) =>
	163	+ call<CrawlJob>("/internal/crawls", { method: "POST", body: JSON.stringify({ project_id: projectId, user_id: userId, options }), timeoutMs: 30_000 }),
	164	+ listCrawls: (projectId: string, userId: string, limit = 50) => call<{ data: CrawlJob[] }>(`/internal/crawls${qs({ project_id: projectId, user_id: userId, limit })}`, { timeoutMs: 15_000 }),
	165	+ getCrawl: (projectId: string, userId: string, id: string) => call<CrawlJob>(`/internal/crawls/${encodeURIComponent(id)}${qs({ project_id: projectId, user_id: userId })}`, { timeoutMs: 15_000 }),
	166	+ crawlPages: (projectId: string, userId: string, id: string, opts: { cursor?: string \| null; limit?: number; status?: string \| null } = {}) =>
	167	+ call<CrawlPagesPage>(`/internal/crawls/${encodeURIComponent(id)}/pages${qs({ project_id: projectId, user_id: userId, cursor: opts.cursor, limit: opts.limit, status: opts.status })}`, { timeoutMs: 20_000 }),
	168	+ cancelCrawl: (projectId: string, userId: string, id: string) =>
	169	+ call<{ id: string; status: "cancelled" }>(`/internal/crawls/${encodeURIComponent(id)}${qs({ project_id: projectId, user_id: userId })}`, { method: "DELETE", timeoutMs: 15_000 }),
	170	+ mapSite: (projectId: string, userId: string, options: unknown) =>
	171	+ call<MapResult>("/internal/map", { method: "POST", body: JSON.stringify({ project_id: projectId, user_id: userId, options }), timeoutMs: 90_000 }),
	172	+ browserStatus: () => call<BrowserStatus>("/internal/browser", { timeoutMs: 5000 }).catch(() => ({ enabled: false, running: 0, capacity: 0, queue: 0 }) as BrowserStatus),
85	173	};

modified apps/web/src/lib/auth.ts +22 −1

		`@@ -1,12 +1,14 @@`
1	1	import "server-only";
2	2	import { betterAuth } from "better-auth";
	3	+import { APIError } from "better-auth/api";
3	4	import { drizzleAdapter } from "better-auth/adapters/drizzle";
4	5	import { nextCookies } from "better-auth/next-js";
5		−import { getDb, users, sessions, accounts, verifications } from "@fetcha/db";
	6	+import { getDb, users, sessions, accounts, verifications, signupAllowlist, eq } from "@fetcha/db";
6	7	import { getEmailService } from "@fetcha/email";
7	8	import { ensureWorkspace } from "./workspace";
8	9	import { writeAudit } from "./audit";
9	10	import { SITE_URL } from "./utils";
	11	+import { ACCESS_DENIED_MESSAGE, isAdminEmail, normalizeEmail } from "./access";
10	12
11	13	const secret = process.env.AUTH_SECRET;
12	14	if (!secret \|\| secret.length < 32) {
		`@@ -93,9 +95,28 @@ export const auth = betterAuth({`
93	95	databaseHooks: {
94	96	user: {
95	97	create: {
	98	+ /**
	99	+ * Invitation-only platform: refuse any email that is neither on `signup_allowlist` nor in
	100	+ * `ADMIN_EMAILS`. Admin emails are created with `role = "admin"`; everyone else is a plain user.
	101	+ */
	102	+ before: async (user) => {
	103	+ const email = normalizeEmail(user.email);
	104	+ const admin = isAdminEmail(email);
	105	+ if (!admin) {
	106	+ const [row] = await getDb().select({ email: signupAllowlist.email }).from(signupAllowlist).where(eq(signupAllowlist.email, email)).limit(1);
	107	+ if (!row) throw new APIError("FORBIDDEN", { message: ACCESS_DENIED_MESSAGE });
	108	+ }
	109	+ return { data: { ...user, email, role: admin ? "admin" : "user" } };
	110	+ },
96	111	after: async (user) => {
97	112	await ensureWorkspace({ id: user.id, name: user.name, email: user.email });
98	113	await writeAudit({ userId: user.id, action: "account.created" });
	114	+ // Mark the invitation as consumed (no-op for admin emails that were never on the list).
	115	+ await getDb()
	116	+ .update(signupAllowlist)
	117	+ .set({ usedAt: new Date(), userId: user.id })
	118	+ .where(eq(signupAllowlist.email, normalizeEmail(user.email)))
	119	+ .catch((e: unknown) => console.error("[auth] allowlist update failed", (e as Error).message));
99	120	},
100	121	},
101	122	},

modified apps/web/src/lib/codegen.test.ts +56 −0

		`@@ -31,6 +31,40 @@ describe("buildRequestBody", () => {`
31	31	expect(Object.keys(body)).toEqual(["url", "method", "country", "timeout", "debug"]);
32	32	expect(body.country).toBe("ca");
33	33	});
	34	+
	35	+ it("drops browser/links/referer fields equal to the API defaults", () => {
	36	+ const body = buildRequestBody({
	37	+ url: "https://example.com",
	38	+ browser: false,
	39	+ browser_fallback: true,
	40	+ javascript: true,
	41	+ wait_until: "domcontentloaded",
	42	+ block_resources: true,
	43	+ screenshot: false,
	44	+ links: false,
	45	+ referer: "auto",
	46	+ });
	47	+ expect(body).toEqual({ url: "https://example.com" });
	48	+ });
	49	+
	50	+ it("serialises browser rendering, links and referer options", () => {
	51	+ const body = buildRequestBody({
	52	+ url: "https://app.example.com",
	53	+ browser: true,
	54	+ browser_fallback: false,
	55	+ wait_for: "table.results",
	56	+ wait_ms: 500,
	57	+ wait_until: "networkidle",
	58	+ javascript: false,
	59	+ block_resources: false,
	60	+ screenshot: true,
	61	+ links: true,
	62	+ referer: "https://www.google.com/",
	63	+ format: "markdown",
	64	+ });
	65	+ expect(Object.keys(body)).toEqual(["url", "browser", "browser_fallback", "wait_for", "wait_ms", "wait_until", "javascript", "block_resources", "screenshot", "links", "referer", "format"]);
	66	+ expect(body).toMatchObject({ browser: true, browser_fallback: false, wait_for: "table.results", wait_ms: 500, wait_until: "networkidle", javascript: false, block_resources: false, screenshot: true, links: true, referer: "https://www.google.com/", format: "markdown" });
	67	+ });
34	68	});
35	69
36	70	describe("generateCode", () => {
		`@@ -57,6 +91,28 @@ describe("generateCode", () => {`
57	91	expect(code).toContain("Authorization: \"Bearer fch_live_••••1234\"");
58	92	});
59	93
	94	+ it("browser and markdown options reach the cURL, JavaScript and Python snippets", () => {
	95	+ const req = { url: "https://app.example.com", browser: true, wait_for: "#app", wait_ms: 500, screenshot: true, links: true, referer: "none" as const, format: "markdown" as const };
	96	+ const curl = generateCode("curl", req, opts);
	97	+ expect(curl).toContain('"browser": true');
	98	+ expect(curl).toContain('"wait_for": "#app"');
	99	+ expect(curl).toContain('"wait_ms": 500');
	100	+ expect(curl).toContain('"screenshot": true');
	101	+ expect(curl).toContain('"links": true');
	102	+ expect(curl).toContain('"referer": "none"');
	103	+ expect(curl).toContain('"format": "markdown"');
	104	+ const js = generateCode("javascript", req, opts);
	105	+ expect(js).toContain('"browser": true');
	106	+ expect(js).toContain('"format": "markdown"');
	107	+ const py = generateCode("python", req, opts);
	108	+ expect(py).toContain('"browser": True');
	109	+ expect(py).toContain('"screenshot": True');
	110	+ expect(py).toContain('"format": "markdown"');
	111	+ const sdk = generateCode("python-sdk", req, opts);
	112	+ expect(sdk).toContain("browser=True");
	113	+ expect(sdk).toContain('format="markdown"');
	114	+ });
	115	+
60	116	it("Python with country uses Python literals", () => {
61	117	const code = generateCode("python", { url: "https://example.com", country: "CA", debug: true, follow_redirects: false }, opts);
62	118	expect(code).toContain("import requests");

modified apps/web/src/lib/codegen.ts +22 −0

		`@@ -50,6 +50,15 @@ const FIELD_ORDER = [`
50	50	"network",
51	51	"session",
52	52	"browser",
	53	+ "browser_fallback",
	54	+ "wait_for",
	55	+ "wait_ms",
	56	+ "wait_until",
	57	+ "javascript",
	58	+ "block_resources",
	59	+ "screenshot",
	60	+ "links",
	61	+ "referer",
53	62	"device",
54	63	"locale",
55	64	"timeout",
		`@@ -95,9 +104,22 @@ export function buildRequestBody(request: FetchRequestInput): RequestBody {`
95	104	if (v !== "auto") out.network = v as string;
96	105	break;
97	106	case "browser":
	107	+ case "screenshot":
	108	+ case "links":
98	109	case "debug":
99	110	if (v === true) out[key] = true;
100	111	break;
	112	+ case "browser_fallback":
	113	+ case "javascript":
	114	+ case "block_resources":
	115	+ if (v === false) out[key] = false;
	116	+ break;
	117	+ case "wait_until":
	118	+ if (v !== "domcontentloaded") out.wait_until = v as string;
	119	+ break;
	120	+ case "referer":
	121	+ if (v !== "auto") out.referer = v as string;
	122	+ break;
101	123	case "timeout":
102	124	if (v !== 30_000) out.timeout = v as number;
103	125	break;

modified apps/web/src/lib/queries/account.ts +13 −3

		`@@ -59,10 +59,13 @@ export async function listApiKeys(organizationId: string, projectId?: string \| n`
59	59
60	60	export interface MonthlyUsage {
61	61	requests: number;
	62	+ successful: number;
	63	+ /** Bytes in + out across all attempts. */
	64	+ bytes: number;
62	65	spendUsd: number;
63	66	}
64	67
65		−/** Requests (fetch_requests) and billed spend (usage_events.cost_usd) for the current month. */
	68	+/** Requests (fetch_requests), bandwidth and estimated spend (usage_events.cost_usd) for the current month. */
66	69	export async function monthlyUsage(organizationId: string, projectId?: string \| null): Promise<MonthlyUsage> {
67	70	const db = getDb();
68	71	const since = monthStart();
		`@@ -73,10 +76,17 @@ export async function monthlyUsage(organizationId: string, projectId?: string \|`
73	76	? and(eq(usageEvents.organizationId, organizationId), eq(usageEvents.projectId, projectId), gte(usageEvents.createdAt, since))
74	77	: and(eq(usageEvents.organizationId, organizationId), gte(usageEvents.createdAt, since));
75	78	const [[req], [usage]] = await Promise.all([
76		− db.select({ n: sql<number>`count(*)::int` }).from(fetchRequests).where(reqWhere),
	79	+ db
	80	+ .select({
	81	+ n: sql<number>`count(*)::int`,
	82	+ ok: sql<number>`count(*) filter (where ${fetchRequests.status} = 'success')::int`,
	83	+ bytes: sql<number>`coalesce(sum(${fetchRequests.bytesIn} + ${fetchRequests.bytesOut}), 0)::float8`,
	84	+ })
	85	+ .from(fetchRequests)
	86	+ .where(reqWhere),
77	87	db.select({ spend: sql<number>`coalesce(sum(${usageEvents.costUsd}), 0)::float8` }).from(usageEvents).where(usageWhere),
78	88	]);
79		− return { requests: Number(req?.n ?? 0), spendUsd: Number(usage?.spend ?? 0) };
	89	+ return { requests: Number(req?.n ?? 0), successful: Number(req?.ok ?? 0), bytes: Number(req?.bytes ?? 0), spendUsd: Number(usage?.spend ?? 0) };
80	90	}
81	91
82	92	export interface ProjectStats {

modified apps/web/src/lib/queries/admin.ts +73 −4

		`@@ -1,5 +1,6 @@`
1	1	import "server-only";
2	2	import type { SQL } from "drizzle-orm";
	3	+import { alias } from "drizzle-orm/pg-core";
3	4	import {
4	5	getDb,
5	6	users,
		`@@ -23,6 +24,7 @@ import {`
23	24	statusIncidents,
24	25	proxySessions,
25	26	webhooks,
	27	+ signupAllowlist,
26	28	eq,
27	29	and,
28	30	or,
		`@@ -35,7 +37,7 @@ import {`
35	37	inArray,
36	38	count,
37	39	} from "@fetcha/db";
38		−import { PLAN_LIMITS, PLANS, type Plan } from "@fetcha/core";
	40	+import { PLAN_LIMITS, PLANS, normalizePlan } from "@fetcha/core";
39	41
40	42	/**
41	43	* Admin-only data access. Everything here may expose upstream provider names,
		`@@ -630,7 +632,7 @@ export async function getEconomics(range: Range) {`
630	632	return {
631	633	unit,
632	634	byBucket: byBucketRaw.map(econ),
633		− byPlan: byPlanRaw.map(econ).sort((a, b) => PLANS.indexOf(a.key as Plan) - PLANS.indexOf(b.key as Plan)),
	635	+ byPlan: byPlanRaw.map(econ).sort((a, b) => PLANS.indexOf(normalizePlan(a.key)) - PLANS.indexOf(normalizePlan(b.key)) \|\| a.key.localeCompare(b.key)),
634	636	byNetwork: byNetworkRaw.map(econ),
635	637	byProvider: byProviderRaw.map((r) => ({
636	638	provider: r.key,
		`@@ -836,12 +838,14 @@ export async function getBillingOverview() {`
836	838	.orderBy(desc(billingEvents.createdAt))
837	839	.limit(100);
838	840	const dist = await db.select({ plan: organizations.plan, n: count() }).from(organizations).groupBy(organizations.plan);
	841	+ // Single plan: every stored value (including legacy ones) normalizes to it, so nothing is "unknown".
839	842	const distribution = PLANS.map((p) => {
840		− const n = dist.find((d) => d.plan === p)?.n ?? 0;
	843	+ const n = dist.filter((d) => normalizePlan(d.plan) === p).reduce((s, d) => s + Number(d.n ?? 0), 0);
841	844	const price = PLAN_LIMITS[p].price_usd_month;
842	845	return { plan: p, label: PLAN_LIMITS[p].label, orgs: n, price, mrr: n * price };
843	846	});
844		− const unknown = dist.filter((d) => !(PLANS as readonly string[]).includes(d.plan));
	847	+ /** Legacy plan values still present in `organizations.plan` (should be empty after migration 0001). */
	848	+ const unknown = dist.filter((d) => !(PLANS as readonly string[]).includes(d.plan)).map((d) => ({ plan: d.plan, n: Number(d.n ?? 0) }));
845	849	const mrr = distribution.reduce((s, d) => s + d.mrr, 0);
846	850	const [stripe] = await db.select({ n: count() }).from(organizations).where(sql`${organizations.stripeCustomerId} is not null`);
847	851	const since = rangeStart("30d");
		`@@ -852,6 +856,71 @@ export async function getBillingOverview() {`
852	856	return { subs, events, distribution, unknown, mrr, stripeCustomers: stripe?.n ?? 0, usageRevenue30d: usage30?.revenue ?? 0, usageCost30d: usage30?.cost ?? 0, stripeConfigured: Boolean(process.env.STRIPE_SECRET_KEY) };
853	857	}
854	858
	859	+// ---------------------------------------------------------------------------
	860	+// Access (signup allowlist)
	861	+// ---------------------------------------------------------------------------
	862	+export type AllowlistStatus = "account" \| "invited" \| "pending";
	863	+
	864	+export interface AllowlistRow {
	865	+ email: string;
	866	+ note: string \| null;
	867	+ invitedByUserId: string \| null;
	868	+ invitedByEmail: string \| null;
	869	+ invitedAt: Date \| null;
	870	+ usedAt: Date \| null;
	871	+ userId: string \| null;
	872	+ /** Email of the account that consumed the invitation (join on users), when it exists. */
	873	+ accountEmail: string \| null;
	874	+ accountBanned: boolean \| null;
	875	+ createdAt: Date;
	876	+ status: AllowlistStatus;
	877	+}
	878	+
	879	+/** Whole allowlist (it is small by construction), newest first, with inviter and account information. */
	880	+export async function listAllowlist(opts: { q?: string } = {}): Promise<AllowlistRow[]> {
	881	+ const db = getDb();
	882	+ const inviter = alias(users, "inviter");
	883	+ const account = alias(users, "account");
	884	+ const where = opts.q ? or(ilike(signupAllowlist.email, `%${opts.q}%`), ilike(signupAllowlist.note, `%${opts.q}%`)) : undefined;
	885	+ const rows = await db
	886	+ .select({
	887	+ email: signupAllowlist.email,
	888	+ note: signupAllowlist.note,
	889	+ invitedByUserId: signupAllowlist.invitedByUserId,
	890	+ invitedByEmail: inviter.email,
	891	+ invitedAt: signupAllowlist.invitedAt,
	892	+ usedAt: signupAllowlist.usedAt,
	893	+ userId: signupAllowlist.userId,
	894	+ accountEmail: account.email,
	895	+ accountBanned: account.banned,
	896	+ createdAt: signupAllowlist.createdAt,
	897	+ })
	898	+ .from(signupAllowlist)
	899	+ .leftJoin(inviter, eq(inviter.id, signupAllowlist.invitedByUserId))
	900	+ .leftJoin(account, eq(account.id, signupAllowlist.userId))
	901	+ .where(where)
	902	+ .orderBy(desc(signupAllowlist.createdAt))
	903	+ .limit(2000);
	904	+ return rows.map((r) => ({
	905	+ ...r,
	906	+ status: r.userId \|\| r.usedAt ? "account" : r.invitedAt ? "invited" : "pending",
	907	+ }));
	908	+}
	909	+
	910	+export async function getAllowlistCounts(): Promise<{ total: number; accounts: number; invited: number; pending: number }> {
	911	+ const [r] = await getDb()
	912	+ .select({
	913	+ total: count(),
	914	+ accounts: sumCase(sql`${signupAllowlist.userId} is not null or ${signupAllowlist.usedAt} is not null`),
	915	+ invited: sumCase(sql`${signupAllowlist.userId} is null and ${signupAllowlist.usedAt} is null and ${signupAllowlist.invitedAt} is not null`),
	916	+ })
	917	+ .from(signupAllowlist);
	918	+ const total = r?.total ?? 0;
	919	+ const accounts = r?.accounts ?? 0;
	920	+ const invited = r?.invited ?? 0;
	921	+ return { total, accounts, invited, pending: Math.max(0, total - accounts - invited) };
	922	+}
	923	+
855	924	// ---------------------------------------------------------------------------
856	925	// Abuse
857	926	// ---------------------------------------------------------------------------

modified apps/web/src/lib/session.ts +2 −6

		`@@ -6,6 +6,7 @@ import { getDb, organizations, organizationMembers, projects, eq, and, isNull, a`
6	6	import type { Organization, Project } from "@fetcha/db";
7	7	import { auth, type AuthUser } from "./auth";
8	8	import { ensureWorkspace } from "./workspace";
	9	+import { isAdminEmail } from "./access";
9	10
10	11	export const PROJECT_COOKIE = "fetcha_project";
11	12
		`@@ -26,13 +27,8 @@ export async function requireUser(next?: string): Promise<AuthUser> {`
26	27	return user;
27	28	}
28	29
29		−const adminEmails = (process.env.ADMIN_EMAILS ?? "")
30		− .split(",")
31		− .map((s) => s.trim().toLowerCase())
32		− .filter(Boolean);
33		−
34	30	export function isAdmin(user: { email: string; role?: string \| null }): boolean {
35		− return user.role === "admin" \|\| adminEmails.includes(user.email.toLowerCase());
	31	+ return user.role === "admin" \|\| isAdminEmail(user.email);
36	32	}
37	33
38	34	export async function requireAdmin(): Promise<AuthUser> {

modified deploy/README.md +2 −1

		`@@ -11,5 +11,6 @@ scp deploy/fetcha.mld.json M1M32:~/dispatch/apps/fetcha.json`
11	11	# 3. verify
12	12	curl -s https://www.fetcha.co/api/ready
13	13	```
14		−Post-sync hooks run `pnpm install`, `db:migrate`, `db:seed` and `next build`. PM2 processes: `fetcha-api`, `fetcha-web`.
	14	+Post-sync hooks run `pnpm install`, `db:migrate`, `db:seed` (with `ADMIN_EMAILS` → admins + allowlist), `patchright install chromium`
	15	+(managed browser binary, ~170 MB once) and `next build`. PM2 processes: `fetcha-api` (also runs the crawl worker and the browser pool), `fetcha-web`.
15	16	Remote source of truth after first deploy: `M3U96a:~/apps/fetcha` (git remote `gitsrv:fetcha.git`).

added docs/API-v0.2.md +124 −0

		`@@ -0,0 +1,124 @@`
	1	+# Fetcha v0.2 — contract for the access model, browser mode and crawling
	2	+
	3	+This document is the single source of truth for the v0.2 changes while the backend (packages/core,
	4	+packages/providers, packages/browser, packages/routing, apps/api) and the web app are built in
	5	+parallel. Field names below are final.
	6	+
	7	+## 1. Access model (private platform)
	8	+
	9	+- There is one plan: `unlimited` (`PLANS = ["unlimited"]`, `PLAN_LIMITS.unlimited`). Legacy values in
	10	+ `organizations.plan` are migrated to `unlimited`; always go through `normalizePlan(org.plan)` from
	11	+ `@fetcha/core` instead of casting.
	12	+- Limits: unlimited monthly requests, 200 concurrent requests, 120 s max timeout, 5 retries, all
	13	+ network classes, 90-day log retention, browser rendering (8 concurrent renders), crawl jobs
	14	+ (2,000 pages/job, 5 concurrent jobs). No prices, no invoices, no checkout: billing UI must say
	15	+ "private platform, no billing" (do not show plan grids, upgrade dialogs or "Start free").
	16	+- Signup is invitation-only: table `signup_allowlist(email pk lower-cased, note, invited_by_user_id,
	17	+ invited_at, used_at, user_id, created_at)`. Better Auth `databaseHooks.user.create.before` must
	18	+ refuse (`APIError("FORBIDDEN", { message })`) any email that is neither in the allowlist nor in
	19	+ `ADMIN_EMAILS`. The `after` hook sets `used_at`/`user_id`.
	20	+- Emails listed in `ADMIN_EMAILS` (env, comma-separated; prod = `spbou4@icloud.com`) get
	21	+ `users.role = 'admin'` at creation, and `pnpm db:seed` (packages/db/src/seed.ts) promotes existing
	22	+ users with those emails + inserts them in the allowlist (idempotent).
	23	+- Admin UI `/admin/access`: list allowlist (email, note, invited by, invited at, account created?),
	24	+ add one or many emails (textarea, comma/newline separated) with optional note and "send
	25	+ invitation email" checkbox, resend invitation, remove entry (only if no account yet). Server
	26	+ actions in `apps/web/src/actions/access.ts`, audited via `admin.action` metadata types
	27	+ `access.allow`, `access.invite`, `access.revoke`.
	28	+- Invitation email: `EmailService.sendInvite(to, { inviterName, signupUrl })` in packages/email
	29	+ (template `InviteEmail`): "You have been invited to Fetcha", button → `${siteUrl}/signup?email=…`.
	30	+- Signup form: pre-fill email from `?email=`, copy "Fetcha is invitation-only. Use the address your
	31	+ administrator approved." On a 403 from sign-up show: "This email is not on the access list. Ask
	32	+ your Fetcha administrator to invite you." Marketing CTAs "Start free" → "Log in" (primary) and
	33	+ "Request access" → `mailto:hello@fetcha.co?subject=Fetcha%20access`.
	34	+- `/pricing` becomes an "Access" page (same route, nav label "Access"): explains private access,
	35	+ what is included (unlimited, browser, crawl, sessions, geo), how to get invited.
	36	+
	37	+## 2. `POST /v1/fetch` additions (all optional, strict schema)
	38	+
	39	+\| Field \| Type \| Default \| Meaning \|
	40	+\|---\|---\|---\|---\|
	41	+\| `format` \| `"html" \\| "text" \\| "markdown" \\| "json" \\| "raw"` \| `"html"` \| `markdown` returns the page converted to Markdown in `markdown` (main content first, boilerplate removed). \|
	42	+\| `browser` \| boolean \| false \| Render in the managed headless Chromium routed through the same proxy network/geo/session. Live. \|
	43	+\| `browser_fallback` \| boolean \| true \| If an HTTP attempt is blocked by a JS challenge/anti-bot, automatically retry in the browser. \|
	44	+\| `wait_for` \| string (CSS) \| — \| Browser: selector to wait for before capturing. \|
	45	+\| `wait_ms` \| 0–30000 \| — \| Browser: extra settle time. \|
	46	+\| `wait_until` \| `"load" \\| "domcontentloaded" \\| "networkidle"` \| `"domcontentloaded"` \| Browser navigation wait condition. \|
	47	+\| `javascript` \| boolean \| true \| Browser: disable scripting when false. \|
	48	+\| `block_resources` \| boolean \| true \| Browser: skip images/fonts/media. \|
	49	+\| `screenshot` \| boolean \| false \| Browser: PNG base64 in `screenshot`. \|
	50	+\| `links` \| boolean \| false \| Return `links[]` (all hyperlinks, absolute). \|
	51	+\| `referer` \| `"auto" \\| "none" \\| url` \| `"auto"` \| Referer strategy (auto = none first, search-engine referer on retries). \|
	52	+
	53	+Response additions:
	54	+
	55	+```jsonc
	56	+{
	57	+ "markdown": "# Title…", // only for format=markdown
	58	+ "page": { "title": "…", "description": "…", "canonical": "…", "lang": "en", "og": { "og:title": "…" }, "links_count": 42 },
	59	+ "links": [{ "url": "https://…", "text": "About", "internal": true, "nofollow": false }], // only with links:true
	60	+ "screenshot": "iVBORw0…", // only with browser + screenshot
	61	+ "metadata": { "mode": "http" \| "browser", …, "debug": { "attempts": [{ "provider", "network", "mode", "country", "outcome", "block_reason", "status", "duration_ms" }] } }
	62	+}
	63	+```
	64	+
	65	+New error semantics: `BROWSER_UNAVAILABLE` is now only returned when the browser pool is disabled or
	66	+down (503-ish situations), `BROWSER_TIMEOUT` when the page does not settle in time. Scope
	67	+`browser:use` is not required (browser is part of fetch); keep the scope name for compatibility.
	68	+
	69	+## 3. Crawl API
	70	+
	71	+`POST /v1/crawl` (scope `fetch:execute`) — body = `crawlCreateSchema` (`@fetcha/core`):
	72	+`url`, `max_pages` (1–5000, default 25), `max_depth` (0–10, default 2), `same_domain` (true),
	73	+`allow_subdomains` (false), `include_patterns[]`, `exclude_patterns[]` (glob with `*` or `/regex/`),
	74	+`respect_robots` (true), `use_sitemap` (false), `concurrency` (1–10, default 3), `delay_ms`, `timeout`,
	75	+`format` (`markdown\|text\|html`, default markdown), `main_content` (true), `country`, `network`,
	76	+`browser`, `browser_fallback`, `headers`, `webhook_url`, `label`.
	77	+
	78	+Returns `202`:
	79	+
	80	+```json
	81	+{ "id": "crawl_…", "status": "queued", "seed_url": "…", "created_at": "…", "options": { … } }
	82	+```
	83	+
	84	+`GET /v1/crawl/:id` → job:
	85	+
	86	+```json
	87	+{ "id", "status": "queued\|running\|completed\|failed\|cancelled", "label", "seed_url", "domain", "options",
	88	+ "stats": { "discovered", "fetched", "ok", "blocked", "failed", "bytes" },
	89	+ "error": { "code", "message" } \| null, "created_at", "started_at", "completed_at" }
	90	+```
	91	+
	92	+`GET /v1/crawl/:id/pages?cursor=&limit=100&status=success` → `{ "data": [page…], "next_cursor": "…" \| null }` where page =
	93	+`{ "id", "url", "final_url", "depth", "status", "http_status", "error_code", "title", "description", "content_type", "content", "links_count", "bytes", "duration_ms", "mode", "fetched_at" }`.
	94	+
	95	+`DELETE /v1/crawl/:id` → `{ "id", "status": "cancelled" }`. `GET /v1/crawl?limit=50` → `{ "data": [job…] }`.
	96	+
	97	+Every crawled page is also a normal fetch request (visible in the Requests log with `source:
	98	+"crawl"`), so quotas, retries, escalation and routing intelligence apply.
	99	+
	100	+`POST /v1/map` (sync, ≤ 60 s) — body = `mapCreateSchema`: `url`, `limit` (default 1000), `use_sitemap`
	101	+(true), `use_links` (true), `same_domain`, `allow_subdomains`, `search`, `country`, `network`, `timeout`.
	102	+Returns `{ "url", "count", "urls": ["…"], "sources": { "sitemap": n, "links": n }, "truncated": bool }`.
	103	+
	104	+## 4. Internal routes (dashboard → API, service token)
	105	+
	106	+- `POST /internal/crawls` `{ project_id, user_id, options }` → job (202 body as above)
	107	+- `GET /internal/crawls?project_id&user_id&limit=50` → `{ data: [job…] }`
	108	+- `GET /internal/crawls/:id?project_id&user_id` → job
	109	+- `GET /internal/crawls/:id/pages?project_id&user_id&cursor&limit&status` → pages page
	110	+- `DELETE /internal/crawls/:id?project_id&user_id` → cancel
	111	+- `POST /internal/map` `{ project_id, user_id, options }` → map result
	112	+- `GET /internal/browser` → `{ enabled, running, capacity, queue }` (admin System page)
	113	+
	114	+Dashboard: new page `/dashboard/crawls` (list jobs of the current project, status badges, stats,
	115	+create dialog) and `/dashboard/crawls/[id]` (job detail, pages table with content preview, cancel).
	116	+Sidebar item "Crawls" after "Requests". Add crawl/map client methods to `apps/web/src/lib/api.ts`.
	117	+
	118	+## 5. SDKs
	119	+
	120	+JS (`packages/sdk`) and Python (`sdk-python`): add `format: "markdown"`, `browser*`, `links`,
	121	+`referer` to fetch options and `markdown`/`page`/`links`/`screenshot` to results; add
	122	+`crawl.create(options)`, `crawl.get(id)`, `crawl.pages(id, { cursor, limit })`, `crawl.cancel(id)`,
	123	+`crawl.wait(id, { pollMs, timeoutMs })` (polls until terminal) and `map(options)`. Bump SDK user-agent
	124	+to `fetcha-sdk-js/0.2.0` / `fetcha-sdk-python/0.2.0`.

added packages/browser/package.json +26 −0

		`@@ -0,0 +1,26 @@`
	1	+{
	2	+ "name": "@fetcha/browser",
	3	+ "version": "0.2.0",
	4	+ "private": true,
	5	+ "type": "module",
	6	+ "main": "./src/index.ts",
	7	+ "types": "./src/index.ts",
	8	+ "exports": {
	9	+ ".": "./src/index.ts"
	10	+ },
	11	+ "scripts": {
	12	+ "typecheck": "tsc -p tsconfig.json --noEmit",
	13	+ "test": "vitest run --passWithNoTests",
	14	+ "install-browser": "patchright install chromium"
	15	+ },
	16	+ "dependencies": {
	17	+ "@fetcha/core": "workspace:*",
	18	+ "@fetcha/providers": "workspace:*",
	19	+ "patchright": "1.62.3"
	20	+ },
	21	+ "devDependencies": {
	22	+ "@types/node": "^24.0.0",
	23	+ "typescript": "^5.9.3",
	24	+ "vitest": "^3.2.0"
	25	+ }
	26	+}

added packages/browser/src/index.ts +550 −0

		`@@ -0,0 +1,550 @@`
	1	+/**
	2	+ * @fetcha/browser — managed headless Chromium for rendered fetches.
	3	+ *
	4	+ * One Chromium process (full Chromium in "new headless" mode, not the headless shell, driven by
	5	+ * Patchright — Playwright patched against the classic CDP leaks) hosts many isolated contexts; every
	6	+ * render gets its own context with the upstream proxy credentials of the chosen route, a consistent
	7	+ * fingerprint (UA, viewport, locale, timezone, platform) and stealth patches. Cloudflare/DataDome
	8	+ * style JavaScript challenges are given time to complete (and a best-effort Turnstile click) before
	9	+ * the DOM is captured. Set FETCHA_BROWSER_HEADLESS=0 on a node with a graphical session to run a
	10	+ * real (off-screen) window, which defeats more headless-detection heuristics.
	11	+ */
	12	+import { FetchaError, isBlockedHostname, isIP, isBlockedIP, looksBlocked, type HttpMethod } from "@fetcha/core";
	13	+import type { FingerprintProfile, ProxyEndpoint } from "@fetcha/providers";
	14	+import { acceptLanguage } from "@fetcha/providers";
	15	+import type { Browser, BrowserContext, Page, Response as PwResponse } from "patchright";
	16	+import { stealthScript, timezoneFor, webglFor } from "./stealth";
	17	+
	18	+export interface RenderRequest {
	19	+ url: string;
	20	+ method: HttpMethod;
	21	+ body?: string \| Buffer;
	22	+ proxy: ProxyEndpoint \| null;
	23	+ profile: FingerprintProfile;
	24	+ locale?: string \| null;
	25	+ country?: string \| null;
	26	+ headers?: Record<string, string>;
	27	+ cookies?: Array<{ name: string; value: string; domain: string; path: string; secure?: boolean; expires?: number \| null }>;
	28	+ referer?: string \| null;
	29	+ timeoutMs: number;
	30	+ waitUntil: "load" \| "domcontentloaded" \| "networkidle";
	31	+ waitFor?: string \| null;
	32	+ waitMs?: number \| null;
	33	+ javascript: boolean;
	34	+ blockResources: boolean;
	35	+ screenshot: boolean;
	36	+ maxResponseBytes: number;
	37	+ /** Called for every main-frame navigation to a new URL (redirects). Throw to abort. */
	38	+ onRedirect?: (nextUrl: string) => Promise<void>;
	39	+}
	40	+
	41	+export interface RenderResult {
	42	+ status: number;
	43	+ headers: Record<string, string>;
	44	+ /** Serialised DOM (UTF-8 HTML). */
	45	+ body: Buffer;
	46	+ finalUrl: string;
	47	+ redirects: number;
	48	+ bytesIn: number;
	49	+ bytesOut: number;
	50	+ timing: { navigation_ms: number; challenge_ms: number; settle_ms: number; capture_ms: number; total_ms: number };
	51	+ cookies: Array<{ name: string; value: string; domain: string; path: string; secure: boolean; expires: number \| null }>;
	52	+ screenshot?: Buffer;
	53	+ /** True when a challenge page was detected and later replaced by real content. */
	54	+ challengeSolved: boolean;
	55	+ /** Block verdict on the captured DOM (after the challenge wait). */
	56	+ block: ReturnType<typeof looksBlocked>;
	57	+}
	58	+
	59	+export interface BrowserPoolOptions {
	60	+ /** Max concurrent renders across the process. */
	61	+ maxConcurrency?: number;
	62	+ /** How long a render may wait for a slot before failing with BROWSER_UNAVAILABLE. */
	63	+ queueTimeoutMs?: number;
	64	+ /** Close the browser after this idle time. */
	65	+ idleCloseMs?: number;
	66	+ /** Playwright channel: "chromium" (full build, new headless) or "chrome". */
	67	+ channel?: string;
	68	+ enabled?: boolean;
	69	+ /** Maximum time to give a JS challenge to resolve. */
	70	+ challengeWaitMs?: number;
	71	+ /** Run a real (off-screen) window instead of headless mode. Requires a graphical session. */
	72	+ headless?: boolean;
	73	+ log?: { info: (m: string) => void; warn: (m: string) => void };
	74	+}
	75	+
	76	+export interface BrowserStatus {
	77	+ enabled: boolean;
	78	+ launched: boolean;
	79	+ running: number;
	80	+ capacity: number;
	81	+ queue: number;
	82	+ renders: number;
	83	+ failures: number;
	84	+ challengesSolved: number;
	85	+ lastError: string \| null;
	86	+ version: string \| null;
	87	+}
	88	+
	89	+const CHALLENGE_SELECTORS = ["#challenge-running", "#challenge-form", "#challenge-stage", ".cf-turnstile", "iframe[src='challenges.cloudflare.com']", "#px-captcha", "#datadome", "iframe[src='captcha-delivery.com']", "#sec-cpt-if", "form#challenge", "#cmsg", "[data-testid='challenge']"];
	90	+
	91	+export class BrowserPool {
	92	+ private browser: Browser \| null = null;
	93	+ private launching: Promise<Browser> \| null = null;
	94	+ /** User agent reported by the real Chromium build (with "HeadlessChrome" normalised to "Chrome"). */
	95	+ private nativeUa: string \| null = null;
	96	+ private running = 0;
	97	+ private queue: Array<() => void> = [];
	98	+ private idleTimer: NodeJS.Timeout \| null = null;
	99	+ private stats = { renders: 0, failures: 0, challengesSolved: 0, lastError: null as string \| null };
	100	+ private readonly opts: Required<Omit<BrowserPoolOptions, "log">> & { log: NonNullable<BrowserPoolOptions["log"]> };
	101	+
	102	+ constructor(opts: BrowserPoolOptions = {}) {
	103	+ this.opts = {
	104	+ maxConcurrency: opts.maxConcurrency ?? Number(process.env.FETCHA_BROWSER_CONCURRENCY ?? 6),
	105	+ queueTimeoutMs: opts.queueTimeoutMs ?? 25_000,
	106	+ idleCloseMs: opts.idleCloseMs ?? 10 * 60_000,
	107	+ channel: opts.channel ?? process.env.FETCHA_BROWSER_CHANNEL ?? "chromium",
	108	+ enabled: opts.enabled ?? process.env.FETCHA_BROWSER_ENABLED !== "0",
	109	+ challengeWaitMs: opts.challengeWaitMs ?? 18_000,
	110	+ headless: opts.headless ?? process.env.FETCHA_BROWSER_HEADLESS !== "0",
	111	+ log: opts.log ?? { info: (m) => console.log("[browser]", m), warn: (m) => console.warn("[browser]", m) },
	112	+ };
	113	+ }
	114	+
	115	+ get enabled(): boolean {
	116	+ return this.opts.enabled;
	117	+ }
	118	+
	119	+ status(): BrowserStatus {
	120	+ return {
	121	+ enabled: this.opts.enabled,
	122	+ launched: Boolean(this.browser?.isConnected()),
	123	+ running: this.running,
	124	+ capacity: this.opts.maxConcurrency,
	125	+ queue: this.queue.length,
	126	+ renders: this.stats.renders,
	127	+ failures: this.stats.failures,
	128	+ challengesSolved: this.stats.challengesSolved,
	129	+ lastError: this.stats.lastError,
	130	+ version: this.browser?.version() ?? null,
	131	+ };
	132	+ }
	133	+
	134	+ private async getBrowser(): Promise<Browser> {
	135	+ if (this.browser?.isConnected()) return this.browser;
	136	+ if (this.launching) return this.launching;
	137	+ this.launching = (async () => {
	138	+ // Patchright = Playwright patched against CDP leaks (Runtime.enable, command-line flags, console).
	139	+ const { chromium } = await import("patchright");
	140	+ const args = [
	141	+ "--no-first-run",
	142	+ "--no-default-browser-check",
	143	+ "--disable-dev-shm-usage",
	144	+ "--disable-background-timer-throttling",
	145	+ "--disable-renderer-backgrounding",
	146	+ "--disable-backgrounding-occluded-windows",
	147	+ "--disable-features=IsolateOrigins,site-per-process,Translate,MediaRouter,OptimizationHints",
	148	+ "--disable-infobars",
	149	+ "--no-service-autorun",
	150	+ "--password-store=basic",
	151	+ "--use-mock-keychain",
	152	+ "--export-tagged-pdf",
	153	+ "--force-color-profile=srgb",
	154	+ "--window-size=1920,1080",
	155	+ ...(this.opts.headless ? [] : ["--window-position=-32000,-32000"]),
	156	+ ];
	157	+ const launch = async (channel: string \| undefined) =>
	158	+ chromium.launch({
	159	+ headless: this.opts.headless,
	160	+ channel,
	161	+ args,
	162	+ proxy: { server: "per-context" },
	163	+ timeout: 60_000,
	164	+ });
	165	+ let b: Browser;
	166	+ try {
	167	+ b = await launch(this.opts.channel);
	168	+ } catch (e) {
	169	+ this.opts.log.warn(`launch with channel=${this.opts.channel} failed (${(e as Error).message.split("\n")[0]}), retrying default build`);
	170	+ b = await launch(undefined);
	171	+ }
	172	+ b.on("disconnected", () => {
	173	+ this.opts.log.warn("browser disconnected");
	174	+ if (this.browser === b) this.browser = null;
	175	+ });
	176	+ this.browser = b;
	177	+ // Client hints (Sec-CH-UA*) come from the real build and cannot be spoofed coherently, so the UA
	178	+ // we present is the native one minus the headless marker — never a made-up version.
	179	+ try {
	180	+ const probe = await b.newContext({ proxy: { server: "http://127.0.0.1:1", bypass: "*" } });
	181	+ const pg = await probe.newPage();
	182	+ this.nativeUa = (await pg.evaluate(() => navigator.userAgent)).replace(/HeadlessChrome/g, "Chrome");
	183	+ await probe.close();
	184	+ } catch {
	185	+ this.nativeUa = null;
	186	+ }
	187	+ this.opts.log.info(`chromium ${b.version()} launched (channel=${this.opts.channel}, headless=${this.opts.headless}, concurrency=${this.opts.maxConcurrency})`);
	188	+ return b;
	189	+ })();
	190	+ try {
	191	+ return await this.launching;
	192	+ } finally {
	193	+ this.launching = null;
	194	+ }
	195	+ }
	196	+
	197	+ private touchIdle() {
	198	+ if (this.idleTimer) clearTimeout(this.idleTimer);
	199	+ this.idleTimer = setTimeout(() => {
	200	+ if (this.running === 0 && this.browser) {
	201	+ this.opts.log.info("closing idle browser");
	202	+ this.browser.close().catch(() => {});
	203	+ this.browser = null;
	204	+ }
	205	+ }, this.opts.idleCloseMs);
	206	+ this.idleTimer.unref?.();
	207	+ }
	208	+
	209	+ private async acquire(): Promise<() => void> {
	210	+ if (this.running < this.opts.maxConcurrency) {
	211	+ this.running++;
	212	+ return () => this.release();
	213	+ }
	214	+ return new Promise<() => void>((resolve, reject) => {
	215	+ const timer = setTimeout(() => {
	216	+ this.queue = this.queue.filter((f) => f !== wake);
	217	+ reject(new FetchaError("BROWSER_UNAVAILABLE", "The managed browser is busy. Retry in a few seconds."));
	218	+ }, this.opts.queueTimeoutMs);
	219	+ const wake = () => {
	220	+ clearTimeout(timer);
	221	+ this.running++;
	222	+ resolve(() => this.release());
	223	+ };
	224	+ this.queue.push(wake);
	225	+ });
	226	+ }
	227	+
	228	+ private release() {
	229	+ this.running = Math.max(0, this.running - 1);
	230	+ const next = this.queue.shift();
	231	+ if (next) next();
	232	+ else this.touchIdle();
	233	+ }
	234	+
	235	+ async close(): Promise<void> {
	236	+ if (this.idleTimer) clearTimeout(this.idleTimer);
	237	+ await this.browser?.close().catch(() => {});
	238	+ this.browser = null;
	239	+ }
	240	+
	241	+ async render(req: RenderRequest): Promise<RenderResult> {
	242	+ if (!this.opts.enabled) throw new FetchaError("BROWSER_UNAVAILABLE");
	243	+ const release = await this.acquire();
	244	+ const started = performance.now();
	245	+ const deadline = started + req.timeoutMs;
	246	+ let context: BrowserContext \| null = null;
	247	+ try {
	248	+ const browser = await this.getBrowser();
	249	+ const profile = req.profile;
	250	+ const languages = acceptLanguage(req.locale, req.country)
	251	+ .split(",")
	252	+ .map((s) => s.split(";")[0]!.trim())
	253	+ .filter(Boolean);
	254	+ const isMobile = profile.device === "mobile";
	255	+ const extraHeaders: Record<string, string> = {};
	256	+ let customUa: string \| null = null;
	257	+ for (const [k, v] of Object.entries(req.headers ?? {})) {
	258	+ const lk = k.toLowerCase();
	259	+ if (lk === "user-agent") {
	260	+ customUa = v;
	261	+ continue;
	262	+ }
	263	+ if (["cookie", "host", "content-length", "accept-encoding", "connection"].includes(lk)) continue;
	264	+ extraHeaders[lk] = v;
	265	+ }
	266	+ const userAgent = customUa ?? (isMobile ? profile.userAgent : this.nativeUa ?? profile.userAgent);
	267	+ if (!extraHeaders["accept-language"]) extraHeaders["accept-language"] = acceptLanguage(req.locale, req.country);
	268	+
	269	+ context = await browser.newContext({
	270	+ // The browser is launched with per-context proxying; a context without an upstream proxy
	271	+ // must still provide one, so we point at an unreachable local proxy and bypass every host (= direct).
	272	+ proxy: req.proxy ? { server: `http://${req.proxy.host}:${req.proxy.port}`, username: req.proxy.username, password: req.proxy.password } : { server: "http://127.0.0.1:1", bypass: "*" },
	273	+ userAgent,
	274	+ viewport: profile.viewport,
	275	+ deviceScaleFactor: isMobile ? 3 : profile.tls === "safari" ? 2 : 1,
	276	+ isMobile,
	277	+ hasTouch: isMobile \|\| profile.device === "tablet",
	278	+ locale: languages[0] ?? "en-US",
	279	+ timezoneId: timezoneFor(req.country),
	280	+ javaScriptEnabled: req.javascript,
	281	+ extraHTTPHeaders: extraHeaders,
	282	+ ignoreHTTPSErrors: false,
	283	+ colorScheme: "light",
	284	+ serviceWorkers: "block",
	285	+ acceptDownloads: false,
	286	+ });
	287	+ const webgl = webglFor(customUa \|\| isMobile ? profile.platform : /Windows/.test(userAgent) ? "Win32" : /Linux/.test(userAgent) ? "Linux x86_64" : "MacIntel", profile.tls);
	288	+ const platform = customUa \|\| isMobile ? profile.platform : /Windows/.test(userAgent) ? "Win32" : /Linux/.test(userAgent) ? "Linux x86_64" : "MacIntel";
	289	+ await context.addInitScript(stealthScript({ platform, languages, hardwareConcurrency: isMobile ? 8 : 12, deviceMemory: 8, vendor: webgl.vendor, renderer: webgl.renderer, mobile: isMobile }));
	290	+ if (req.cookies?.length) {
	291	+ await context
	292	+ .addCookies(
	293	+ req.cookies.map((c) => ({
	294	+ name: c.name,
	295	+ value: c.value,
	296	+ domain: c.domain.startsWith(".") ? c.domain : c.domain,
	297	+ path: c.path \|\| "/",
	298	+ secure: Boolean(c.secure),
	299	+ httpOnly: false,
	300	+ expires: c.expires ? Math.floor(c.expires / 1000) : -1,
	301	+ sameSite: "Lax" as const,
	302	+ })),
	303	+ )
	304	+ .catch(() => {});
	305	+ }
	306	+
	307	+ let bytesIn = 0;
	308	+ let bytesOut = 0;
	309	+ const pendingSizes: Promise<void>[] = [];
	310	+ const initialUrl = new URL(req.url);
	311	+ let redirects = 0;
	312	+ let policyError: FetchaError \| null = null;
	313	+ let firstNav = true;
	314	+
	315	+ const page = await context.newPage();
	316	+ page.setDefaultTimeout(Math.min(req.timeoutMs, 60_000));
	317	+
	318	+ await page.route("*/", async (route) => {
	319	+ const r = route.request();
	320	+ const isMain = r.isNavigationRequest() && r.frame() === page.mainFrame();
	321	+ let target: URL;
	322	+ try {
	323	+ target = new URL(r.url());
	324	+ } catch {
	325	+ return route.abort("blockedbyclient");
	326	+ }
	327	+ // SSRF: never let the browser touch private hosts (even via subresources / redirects).
	328	+ const host = target.hostname.replace(/^\[\|\]$/g, "");
	329	+ if (isBlockedHostname(host) \|\| (isIP(host) && isBlockedIP(host)) \|\| !/^https?:$/.test(target.protocol)) {
	330	+ if (isMain) policyError = new FetchaError("URL_NOT_ALLOWED", "The page redirected to a local, private or internal host.");
	331	+ return route.abort("blockedbyclient");
	332	+ }
	333	+ if (isMain) {
	334	+ if (firstNav) {
	335	+ firstNav = false;
	336	+ if (req.method !== "GET") {
	337	+ const headers = { ...r.headers() };
	338	+ if (req.body !== undefined && !headers["content-type"]) headers["content-type"] = typeof req.body === "string" ? "application/json" : "application/octet-stream";
	339	+ return route.continue({ method: req.method, postData: req.body, headers });
	340	+ }
	341	+ } else if (r.url() !== initialUrl.toString()) {
	342	+ redirects++;
	343	+ if (req.onRedirect) {
	344	+ try {
	345	+ await req.onRedirect(r.url());
	346	+ } catch (e) {
	347	+ policyError = e instanceof FetchaError ? e : new FetchaError("URL_NOT_ALLOWED");
	348	+ return route.abort("blockedbyclient");
	349	+ }
	350	+ }
	351	+ }
	352	+ return route.continue();
	353	+ }
	354	+ if (req.blockResources) {
	355	+ const type = r.resourceType();
	356	+ if (type === "image" \|\| type === "media" \|\| type === "font" \|\| type === "manifest" \|\| type === "texttrack") return route.abort("blockedbyclient");
	357	+ }
	358	+ return route.continue();
	359	+ });
	360	+ page.on("requestfinished", (r) => {
	361	+ pendingSizes.push(
	362	+ r
	363	+ .sizes()
	364	+ .then((s) => {
	365	+ bytesOut += s.requestBodySize + s.requestHeadersSize;
	366	+ bytesIn += s.responseBodySize + s.responseHeadersSize;
	367	+ })
	368	+ .catch(() => {}),
	369	+ );
	370	+ });
	371	+ page.on("dialog", (d) => d.dismiss().catch(() => {}));
	372	+
	373	+ // Navigate
	374	+ const tNav0 = performance.now();
	375	+ let response: PwResponse \| null = null;
	376	+ try {
	377	+ response = await page.goto(req.url, { waitUntil: req.waitUntil, timeout: Math.max(1000, Math.min(req.timeoutMs - 1500, deadline - performance.now())), referer: req.referer ?? undefined });
	378	+ } catch (e) {
	379	+ if (policyError) throw policyError;
	380	+ const msg = (e as Error).message ?? String(e);
	381	+ if (/Timeout/i.test(msg)) {
	382	+ // Partial content may still be useful: if the document exists, capture what we have.
	383	+ if (!(await page.content().catch(() => "")).includes("<body")) throw new FetchaError("BROWSER_TIMEOUT", "The page did not finish loading in time.");
	384	+ } else if (/ERR_TUNNEL_CONNECTION_FAILED\|ERR_PROXY\|ERR_NO_SUPPORTED_PROXIES\|407/i.test(msg)) {
	385	+ throw new FetchaError("PROVIDER_UNAVAILABLE", "The upstream network rejected the browser connection.");
	386	+ } else if (/ERR_NAME_NOT_RESOLVED\|ERR_CONNECTION_REFUSED\|ERR_CONNECTION_RESET\|ERR_CONNECTION_CLOSED\|ERR_ADDRESS_UNREACHABLE\|ERR_CONNECTION_TIMED_OUT\|ERR_EMPTY_RESPONSE/i.test(msg)) {
	387	+ throw new FetchaError("TARGET_UNAVAILABLE", "The browser could not reach the target.");
	388	+ } else if (/ERR_BLOCKED_BY_CLIENT/i.test(msg)) {
	389	+ throw policyError ?? new FetchaError("URL_NOT_ALLOWED");
	390	+ } else if (/ERR_ABORTED/i.test(msg) && response) {
	391	+ /* download / non-HTML navigation: continue with what we have */
	392	+ } else if (/ERR_TOO_MANY_REDIRECTS/i.test(msg)) {
	393	+ throw new FetchaError("TOO_MANY_REDIRECTS");
	394	+ } else if (/ERR_CERT\|ERR_SSL/i.test(msg)) {
	395	+ throw new FetchaError("TARGET_UNAVAILABLE", "TLS error while connecting to the target.");
	396	+ } else throw new FetchaError("INTERNAL_ERROR", "Browser navigation failed.", { cause: e });
	397	+ }
	398	+ if (policyError) throw policyError;
	399	+ const navigationMs = Math.round(performance.now() - tNav0);
	400	+
	401	+ // Challenge handling: give the page time to solve JS challenges (Cloudflare, DataDome, PX…).
	402	+ const tCh0 = performance.now();
	403	+ let challengeSolved = false;
	404	+ let status = response?.status() ?? 200;
	405	+ let headers = lower(response?.headers() ?? {});
	406	+ let html = await page.content().catch(() => "");
	407	+ let verdict = looksBlocked(status, html, headers);
	408	+ if (verdict.blocked && verdict.challenge !== false && req.javascript) {
	409	+ const budget = Math.min(this.opts.challengeWaitMs, deadline - performance.now() - 1500);
	410	+ const challengeStatus = status;
	411	+ const until = performance.now() + Math.max(0, budget);
	412	+ let clickedTurnstile = false;
	413	+ while (performance.now() < until) {
	414	+ await page.waitForTimeout(600).catch(() => {});
	415	+ // Best-effort click on a Turnstile / hCaptcha checkbox when one is visible.
	416	+ if (!clickedTurnstile) {
	417	+ clickedTurnstile = await this.tryClickChallenge(page);
	418	+ }
	419	+ const navResp = await page.waitForNavigation({ timeout: 1200, waitUntil: "domcontentloaded" }).catch(() => null);
	420	+ if (navResp) {
	421	+ response = navResp;
	422	+ status = navResp.status();
	423	+ headers = lower(navResp.headers());
	424	+ }
	425	+ html = await page.content().catch(() => "");
	426	+ const stillChallenge = CHALLENGE_SELECTORS.some((sel) => html.includes(sel.replace(/^[#.]/, "")) && /challenge\|turnstile\|captcha\|datadome\|px-/i.test(sel)) && looksBlocked(status, html, headers).blocked;
	427	+ const v = looksBlocked(navResp ? status : 200, html, headers);
	428	+ if (!v.blocked && !stillChallenge && html.length > 1500) {
	429	+ challengeSolved = true;
	430	+ if (!navResp) status = 200; // the document was replaced in place
	431	+ break;
	432	+ }
	433	+ }
	434	+ verdict = looksBlocked(status, html, headers);
	435	+ if (challengeSolved) {
	436	+ this.stats.challengesSolved++;
	437	+ headers = { ...headers, "x-fetcha-challenge": `solved:${challengeStatus}` };
	438	+ verdict = { blocked: false };
	439	+ }
	440	+ }
	441	+ const challengeMs = Math.round(performance.now() - tCh0);
	442	+
	443	+ // Settle: wait for selector / extra time; light human-like interaction.
	444	+ const tSettle0 = performance.now();
	445	+ if (req.waitFor && !verdict.blocked) {
	446	+ const remaining = deadline - performance.now() - 800;
	447	+ if (remaining > 200) {
	448	+ try {
	449	+ await page.waitForSelector(req.waitFor, { timeout: Math.min(remaining, 45_000), state: "attached" });
	450	+ } catch {
	451	+ throw new FetchaError("BROWSER_TIMEOUT", `The selector "${req.waitFor}" did not appear in time.`, { details: { wait_for: req.waitFor } });
	452	+ }
	453	+ }
	454	+ }
	455	+ if (req.javascript && !verdict.blocked) {
	456	+ try {
	457	+ const vp = profile.viewport;
	458	+ await page.mouse.move(vp.width * (0.3 + Math.random() * 0.4), vp.height * (0.3 + Math.random() * 0.3), { steps: 5 });
	459	+ await page.mouse.wheel(0, 200 + Math.round(Math.random() * 400));
	460	+ } catch {
	461	+ /* ignore */
	462	+ }
	463	+ }
	464	+ if (req.waitMs) await page.waitForTimeout(Math.min(req.waitMs, Math.max(0, deadline - performance.now() - 500)));
	465	+ else if (req.javascript && req.waitUntil !== "networkidle") await page.waitForLoadState("networkidle", { timeout: Math.min(2500, Math.max(0, deadline - performance.now() - 500)) }).catch(() => {});
	466	+ const settleMs = Math.round(performance.now() - tSettle0);
	467	+
	468	+ // Capture
	469	+ const tCap0 = performance.now();
	470	+ html = await page.content().catch(() => html);
	471	+ const finalUrl = page.url();
	472	+ const body = Buffer.from(html, "utf8");
	473	+ if (body.length > req.maxResponseBytes) throw new FetchaError("RESPONSE_TOO_LARGE");
	474	+ let screenshot: Buffer \| undefined;
	475	+ if (req.screenshot) screenshot = await page.screenshot({ type: "png", fullPage: false, timeout: 15_000 }).catch(() => undefined);
	476	+ const cookies = (await context.cookies().catch(() => [])).map((c) => ({ name: c.name, value: c.value, domain: c.domain, path: c.path, secure: c.secure, expires: c.expires && c.expires > 0 ? Math.round(c.expires * 1000) : null }));
	477	+ await Promise.race([Promise.allSettled(pendingSizes), new Promise((r) => setTimeout(r, 500))]);
	478	+ const captureMs = Math.round(performance.now() - tCap0);
	479	+ const totalMs = Math.round(performance.now() - started);
	480	+ const finalVerdict = challengeSolved ? { blocked: false } : looksBlocked(status, html, headers);
	481	+ this.stats.renders++;
	482	+ const outHeaders: Record<string, string> = { ...headers };
	483	+ delete outHeaders["content-encoding"];
	484	+ delete outHeaders["content-length"];
	485	+ if (!outHeaders["content-type"]) outHeaders["content-type"] = "text/html; charset=utf-8";
	486	+ return {
	487	+ status,
	488	+ headers: outHeaders,
	489	+ body,
	490	+ finalUrl,
	491	+ redirects,
	492	+ bytesIn: bytesIn \|\| body.length,
	493	+ bytesOut,
	494	+ timing: { navigation_ms: navigationMs, challenge_ms: challengeMs, settle_ms: settleMs, capture_ms: captureMs, total_ms: totalMs },
	495	+ cookies,
	496	+ screenshot,
	497	+ challengeSolved,
	498	+ block: finalVerdict,
	499	+ };
	500	+ } catch (e) {
	501	+ this.stats.failures++;
	502	+ this.stats.lastError = (e as Error).message?.slice(0, 300) ?? String(e);
	503	+ throw e;
	504	+ } finally {
	505	+ await context?.close().catch(() => {});
	506	+ release();
	507	+ }
	508	+ }
	509	+
	510	+ private async tryClickChallenge(page: Page): Promise<boolean> {
	511	+ try {
	512	+ for (const frame of page.frames()) {
	513	+ const u = frame.url();
	514	+ if (!/challenges\.cloudflare\.com\|hcaptcha\.com\|captcha-delivery\.com/.test(u)) continue;
	515	+ const el = await frame.frameElement().catch(() => null);
	516	+ const box = el ? await el.boundingBox().catch(() => null) : null;
	517	+ if (box && box.width > 20 && box.height > 20) {
	518	+ await page.mouse.move(box.x + 28 + Math.random() * 6, box.y + box.height / 2 + (Math.random() * 4 - 2), { steps: 8 });
	519	+ await page.waitForTimeout(150 + Math.random() * 200);
	520	+ await page.mouse.click(box.x + 30, box.y + box.height / 2, { delay: 40 + Math.random() * 60 });
	521	+ return true;
	522	+ }
	523	+ }
	524	+ const cb = page.locator("#challenge-stage input[type=checkbox], .ctp-checkbox-label, label.ctp-checkbox-label").first();
	525	+ if (await cb.isVisible({ timeout: 200 }).catch(() => false)) {
	526	+ await cb.click({ timeout: 1500 }).catch(() => {});
	527	+ return true;
	528	+ }
	529	+ } catch {
	530	+ /* ignore */
	531	+ }
	532	+ return false;
	533	+ }
	534	+}
	535	+
	536	+function lower(h: Record<string, string>): Record<string, string> {
	537	+ const out: Record<string, string> = {};
	538	+ for (const [k, v] of Object.entries(h)) out[k.toLowerCase()] = v;
	539	+ return out;
	540	+}
	541	+
	542	+let _pool: BrowserPool \| null = null;
	543	+export function getBrowserPool(opts?: BrowserPoolOptions): BrowserPool {
	544	+ if (!_pool) _pool = new BrowserPool(opts);
	545	+ return _pool;
	546	+}
	547	+export async function closeBrowserPool(): Promise<void> {
	548	+ await _pool?.close();
	549	+ _pool = null;
	550	+}

added packages/browser/src/stealth.ts +126 −0

		`@@ -0,0 +1,126 @@`
	1	+/**
	2	+ * Init script injected in every page before any site script runs. Removes the usual headless /
	3	+ * automation tells that anti-bot vendors probe (webdriver flag, missing `chrome` object, empty
	4	+ * plugin list, SwiftShader WebGL renderer, permission API quirks, notification permission
	5	+ * mismatch, `outerWidth === 0`, headless UA hints).
	6	+ */
	7	+export function stealthScript(opts: { platform: string; languages: string[]; hardwareConcurrency: number; deviceMemory: number; vendor: string; renderer: string; mobile: boolean }): string {
	8	+ return `(() => {
	9	+ const opts = ${JSON.stringify(opts)};
	10	+ const define = (obj, key, value) => { try { Object.defineProperty(obj, key, { get: () => value, configurable: true }); } catch {} };
	11	+
	12	+ // navigator.webdriver → false (and not enumerable like real Chrome)
	13	+ try { Object.defineProperty(Navigator.prototype, 'webdriver', { get: () => false, configurable: true }); } catch {}
	14	+
	15	+ // window.chrome (present in every Chrome, absent in headless-shell)
	16	+ if (!window.chrome) {
	17	+ const chrome = {
	18	+ app: { isInstalled: false, InstallState: { DISABLED: 'disabled', INSTALLED: 'installed', NOT_INSTALLED: 'not_installed' }, RunningState: { CANNOT_RUN: 'cannot_run', READY_TO_RUN: 'ready_to_run', RUNNING: 'running' } },
	19	+ runtime: { OnInstalledReason: { CHROME_UPDATE: 'chrome_update', INSTALL: 'install', SHARED_MODULE_UPDATE: 'shared_module_update', UPDATE: 'update' }, PlatformArch: { ARM: 'arm', ARM64: 'arm64', MIPS: 'mips', MIPS64: 'mips64', X86_32: 'x86-32', X86_64: 'x86-64' }, PlatformOs: { ANDROID: 'android', CROS: 'cros', LINUX: 'linux', MAC: 'mac', OPENBSD: 'openbsd', WIN: 'win' }, RequestUpdateCheckStatus: { NO_UPDATE: 'no_update', THROTTLED: 'throttled', UPDATE_AVAILABLE: 'update_available' }, connect: function() {}, sendMessage: function() {}, id: undefined },
	20	+ csi: function() { return { onloadT: Date.now(), pageT: Math.random() * 1000, startE: Date.now(), tran: 15 }; },
	21	+ loadTimes: function() { const t = Date.now() / 1000; return { commitLoadTime: t, connectionInfo: 'h2', finishDocumentLoadTime: t, finishLoadTime: t, firstPaintAfterLoadTime: 0, firstPaintTime: t, navigationType: 'Other', npnNegotiatedProtocol: 'h2', requestTime: t - 0.5, startLoadTime: t - 0.4, wasAlternateProtocolAvailable: false, wasFetchedViaSpdy: true, wasNpnNegotiated: true }; },
	22	+ };
	23	+ define(window, 'chrome', chrome);
	24	+ }
	25	+
	26	+ // Plugins & mimeTypes (Chrome ships 5 PDF-related plugins)
	27	+ try {
	28	+ const mk = (name, filename, description) => ({ name, filename, description, length: 1, item: () => null, namedItem: () => null, 0: { type: 'application/pdf', suffixes: 'pdf', description } });
	29	+ const list = [mk('PDF Viewer', 'internal-pdf-viewer', 'Portable Document Format'), mk('Chrome PDF Viewer', 'internal-pdf-viewer', 'Portable Document Format'), mk('Chromium PDF Viewer', 'internal-pdf-viewer', 'Portable Document Format'), mk('Microsoft Edge PDF Viewer', 'internal-pdf-viewer', 'Portable Document Format'), mk('WebKit built-in PDF', 'internal-pdf-viewer', 'Portable Document Format')];
	30	+ const plugins = Object.create(PluginArray.prototype);
	31	+ list.forEach((p, i) => { plugins[i] = p; });
	32	+ define(plugins, 'length', opts.mobile ? 0 : list.length);
	33	+ plugins.item = (i) => plugins[i] ?? null; plugins.namedItem = (n) => list.find((p) => p.name === n) ?? null; plugins.refresh = () => {};
	34	+ if (navigator.plugins.length === 0) define(Navigator.prototype, 'plugins', plugins);
	35	+ } catch {}
	36	+
	37	+ define(Navigator.prototype, 'languages', opts.languages);
	38	+ define(Navigator.prototype, 'platform', opts.platform);
	39	+ define(Navigator.prototype, 'hardwareConcurrency', opts.hardwareConcurrency);
	40	+ define(Navigator.prototype, 'deviceMemory', opts.deviceMemory);
	41	+ if (opts.mobile) define(Navigator.prototype, 'maxTouchPoints', 5);
	42	+
	43	+ // Permissions: headless reports 'denied' for notifications while Notification.permission is 'default'
	44	+ try {
	45	+ const q = Permissions.prototype.query;
	46	+ Permissions.prototype.query = function(p) {
	47	+ if (p && p.name === 'notifications') return Promise.resolve({ state: Notification.permission === 'default' ? 'prompt' : Notification.permission, onchange: null });
	48	+ return q.call(this, p);
	49	+ };
	50	+ } catch {}
	51	+
	52	+ // WebGL vendor/renderer (SwiftShader / "Google Inc. (Google)" is a headless tell)
	53	+ try {
	54	+ const patch = (proto) => {
	55	+ const gp = proto.getParameter;
	56	+ proto.getParameter = function(param) {
	57	+ if (param === 37445) return opts.vendor;
	58	+ if (param === 37446) return opts.renderer;
	59	+ return gp.call(this, param);
	60	+ };
	61	+ };
	62	+ if (window.WebGLRenderingContext) patch(WebGLRenderingContext.prototype);
	63	+ if (window.WebGL2RenderingContext) patch(WebGL2RenderingContext.prototype);
	64	+ } catch {}
	65	+
	66	+ // outerWidth/outerHeight are 0 in headless
	67	+ if (!window.outerWidth) define(window, 'outerWidth', window.innerWidth);
	68	+ if (!window.outerHeight) define(window, 'outerHeight', window.innerHeight + 85);
	69	+
	70	+ // Hide automation-only properties leaking through Function.prototype.toString
	71	+ try {
	72	+ const nativeToString = Function.prototype.toString;
	73	+ const patched = new WeakSet([Permissions.prototype.query]);
	74	+ Function.prototype.toString = function() {
	75	+ if (patched.has(this)) return 'function query() { [native code] }';
	76	+ return nativeToString.call(this);
	77	+ };
	78	+ } catch {}
	79	+
	80	+ // Connection / battery plausibility
	81	+ try { if (navigator.connection) define(navigator.connection, 'rtt', 50); } catch {}
	82	+})();`;
	83	+}
	84	+
	85	+const TIMEZONES: Record<string, string> = {
	86	+ CA: "America/Toronto",
	87	+ US: "America/New_York",
	88	+ GB: "Europe/London",
	89	+ FR: "Europe/Paris",
	90	+ DE: "Europe/Berlin",
	91	+ ES: "Europe/Madrid",
	92	+ IT: "Europe/Rome",
	93	+ NL: "Europe/Amsterdam",
	94	+ BE: "Europe/Brussels",
	95	+ CH: "Europe/Zurich",
	96	+ SE: "Europe/Stockholm",
	97	+ PL: "Europe/Warsaw",
	98	+ PT: "Europe/Lisbon",
	99	+ IE: "Europe/Dublin",
	100	+ AU: "Australia/Sydney",
	101	+ NZ: "Pacific/Auckland",
	102	+ JP: "Asia/Tokyo",
	103	+ KR: "Asia/Seoul",
	104	+ IN: "Asia/Kolkata",
	105	+ SG: "Asia/Singapore",
	106	+ HK: "Asia/Hong_Kong",
	107	+ BR: "America/Sao_Paulo",
	108	+ MX: "America/Mexico_City",
	109	+ AR: "America/Argentina/Buenos_Aires",
	110	+ ZA: "Africa/Johannesburg",
	111	+ AE: "Asia/Dubai",
	112	+ TR: "Europe/Istanbul",
	113	+ RU: "Europe/Moscow",
	114	+ IL: "Asia/Jerusalem",
	115	+};
	116	+
	117	+export function timezoneFor(country: string \| null \| undefined): string {
	118	+ return (country && TIMEZONES[country.toUpperCase()]) \|\| "America/New_York";
	119	+}
	120	+
	121	+export function webglFor(platform: string, tls: string): { vendor: string; renderer: string } {
	122	+ if (tls === "safari" \|\| /mac/i.test(platform)) return { vendor: "Google Inc. (Apple)", renderer: "ANGLE (Apple, ANGLE Metal Renderer: Apple M2, Unspecified Version)" };
	123	+ if (/linux/i.test(platform) && !/arm/i.test(platform)) return { vendor: "Google Inc. (Intel)", renderer: "ANGLE (Intel, Mesa Intel(R) UHD Graphics 630 (CFL GT2), OpenGL 4.6)" };
	124	+ if (/arm/i.test(platform)) return { vendor: "Google Inc. (Qualcomm)", renderer: "ANGLE (Qualcomm, Adreno (TM) 750, OpenGL ES 3.2 V@0744.0)" };
	125	+ return { vendor: "Google Inc. (NVIDIA)", renderer: "ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 (0x00002504) Direct3D11 vs_5_0 ps_5_0, D3D11)" };
	126	+}

added packages/browser/tsconfig.json +8 −0

		`@@ -0,0 +1,8 @@`
	1	+{
	2	+ "extends": "../../tsconfig.base.json",
	3	+ "compilerOptions": {
	4	+ "types": ["node"],
	5	+ "outDir": "dist"
	6	+ },
	7	+ "include": ["src/*/.ts", "test/*/.ts"]
	8	+}

modified packages/core/src/client.ts +1 −0

		`@@ -4,3 +4,4 @@ export * from "./schema";`
4	4	export * from "./redact";
5	5	export * from "./text";
6	6	export * from "./geo";
	7	+export * from "./markdown";

modified packages/core/src/errors.ts +10 −4

		`@@ -18,6 +18,8 @@ export const ERROR_CODES = [`
18	18	"USAGE_LIMIT_REACHED",
19	19	"SESSION_NOT_FOUND",
20	20	"SESSION_EXPIRED",
	21	+ "CRAWL_NOT_FOUND",
	22	+ "CRAWL_LIMIT_REACHED",
21	23	"NOT_FOUND",
22	24	"FORBIDDEN",
23	25	"INTERNAL_ERROR",
		`@@ -45,6 +47,8 @@ export const ERROR_HTTP_STATUS: Record<ErrorCode, number> = {`
45	47	USAGE_LIMIT_REACHED: 402,
46	48	SESSION_NOT_FOUND: 404,
47	49	SESSION_EXPIRED: 410,
	50	+ CRAWL_NOT_FOUND: 404,
	51	+ CRAWL_LIMIT_REACHED: 429,
48	52	NOT_FOUND: 404,
49	53	FORBIDDEN: 403,
50	54	INTERNAL_ERROR: 500,
		`@@ -53,16 +57,16 @@ export const ERROR_HTTP_STATUS: Record<ErrorCode, number> = {`
53	57	export const ERROR_MESSAGES: Record<ErrorCode, string> = {
54	58	INVALID_API_KEY: "The API key is missing, malformed, revoked or expired.",
55	59	EMAIL_NOT_VERIFIED: "Verify your email address before using the production API.",
56		− RATE_LIMITED: "Too many requests. Slow down or upgrade your plan.",
57		− CONCURRENCY_LIMIT: "Concurrent request limit reached for your plan.",
	60	+ RATE_LIMITED: "Too many requests. Slow down and retry after the indicated delay.",
	61	+ CONCURRENCY_LIMIT: "Concurrent request limit reached for your organization.",
58	62	INVALID_REQUEST: "The request body is invalid.",
59	63	URL_NOT_ALLOWED: "The target URL is not allowed.",
60	64	TARGET_TIMEOUT: "The target did not respond in time.",
61	65	TARGET_BLOCKED: "The target blocked every route we tried.",
62	66	TARGET_UNAVAILABLE: "The target could not be reached.",
63	67	PROVIDER_UNAVAILABLE: "No network route is currently available for this request.",
64		− NETWORK_UNAVAILABLE: "The requested network class is not available on your plan or region.",
65		− BROWSER_UNAVAILABLE: "Managed browser execution is not yet available.",
	68	+ NETWORK_UNAVAILABLE: "The requested network class is not available right now.",
	69	+ BROWSER_UNAVAILABLE: "The managed browser is disabled or temporarily unavailable.",
66	70	BROWSER_TIMEOUT: "The browser did not finish in time.",
67	71	RESPONSE_TOO_LARGE: "The response exceeded the maximum allowed size.",
68	72	TOO_MANY_REDIRECTS: "The target redirected too many times.",
		`@@ -70,6 +74,8 @@ export const ERROR_MESSAGES: Record<ErrorCode, string> = {`
70	74	USAGE_LIMIT_REACHED: "A spending or usage limit configured on this project was reached.",
71	75	SESSION_NOT_FOUND: "The session does not exist.",
72	76	SESSION_EXPIRED: "The session has expired.",
	77	+ CRAWL_NOT_FOUND: "The crawl job does not exist.",
	78	+ CRAWL_LIMIT_REACHED: "Too many crawl jobs are running for your organization.",
73	79	NOT_FOUND: "Resource not found.",
74	80	FORBIDDEN: "You do not have access to this resource.",
75	81	INTERNAL_ERROR: "An internal error occurred.",

modified packages/core/src/ids.ts +3 −1

		`@@ -19,7 +19,9 @@ export type IdPrefix =`
19	19	\| "evt"
20	20	\| "wh"
21	21	\| "aud"
22		− \| "abuse";
	22	+ \| "abuse"
	23	+ \| "crawl"
	24	+ \| "cpg";
23	25
24	26	export function newId(prefix: IdPrefix): string {
25	27	return `${prefix}_${nano()}`;

modified packages/core/src/index.ts +2 −0

		`@@ -5,3 +5,5 @@ export * from "./ssrf";`
5	5	export * from "./redact";
6	6	export * from "./text";
7	7	export * from "./geo";
	8	+export * from "./markdown";
	9	+export * from "./robots";

added packages/core/src/markdown.ts +687 −0

		`@@ -0,0 +1,687 @@`
	1	+/**
	2	+ * Dependency-free HTML → Markdown conversion, page metadata and link extraction.
	3	+ *
	4	+ * The converter builds a lightweight DOM from a forgiving tokenizer, optionally isolates the main
	5	+ * content (readability-style density scoring), then serialises to Markdown. It is intended for
	6	+ * LLM/RAG pipelines and crawling: stable, compact output rather than perfect fidelity.
	7	+ */
	8	+import { decodeEntities } from "./text";
	9	+import type { PageLink, PageMetadata } from "./schema";
	10	+
	11	+// ---------------------------------------------------------------------------
	12	+// Tokenizer / DOM-lite
	13	+// ---------------------------------------------------------------------------
	14	+export interface HNode {
	15	+ type: "element" \| "text";
	16	+ tag: string;
	17	+ attrs: Record<string, string>;
	18	+ children: HNode[];
	19	+ text: string;
	20	+ parent: HNode \| null;
	21	+}
	22	+
	23	+const VOID = new Set(["area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr"]);
	24	+const RAW_TEXT = new Set(["script", "style", "noscript", "template", "svg", "math", "iframe", "canvas", "object", "textarea"]);
	25	+const BLOCK = new Set([
	26	+ "address", "article", "aside", "blockquote", "body", "center", "dd", "details", "dialog", "div", "dl", "dt", "fieldset", "figcaption", "figure", "footer", "form", "h1", "h2", "h3", "h4", "h5", "h6",
	27	+ "header", "hr", "html", "li", "main", "nav", "ol", "p", "pre", "section", "summary", "table", "tbody", "td", "tfoot", "th", "thead", "tr", "ul",
	28	+]);
	29	+/** Elements whose content is closed implicitly by a new block start. */
	30	+const AUTO_CLOSE: Record<string, Set<string>> = {
	31	+ p: BLOCK,
	32	+ li: new Set(["li"]),
	33	+ dt: new Set(["dt", "dd"]),
	34	+ dd: new Set(["dt", "dd"]),
	35	+ tr: new Set(["tr"]),
	36	+ td: new Set(["td", "th", "tr"]),
	37	+ th: new Set(["td", "th", "tr"]),
	38	+ option: new Set(["option"]),
	39	+};
	40	+
	41	+function makeNode(type: HNode["type"], tag: string, parent: HNode \| null): HNode {
	42	+ return { type, tag, attrs: {}, children: [], text: "", parent };
	43	+}
	44	+
	45	+const ATTR_RE = /([^\s"'<>\/=]+)(?:\s=\s(?:"([^"])"\|'([^'])'\|([^\s"'=<>`]+)))?/g;
	46	+
	47	+export function parseHtml(html: string): HNode {
	48	+ const root = makeNode("element", "#root", null);
	49	+ let cur = root;
	50	+ let i = 0;
	51	+ const n = html.length;
	52	+ const pushText = (t: string) => {
	53	+ if (!t) return;
	54	+ const node = makeNode("text", "#text", cur);
	55	+ node.text = t;
	56	+ cur.children.push(node);
	57	+ };
	58	+ while (i < n) {
	59	+ const lt = html.indexOf("<", i);
	60	+ if (lt === -1) {
	61	+ pushText(html.slice(i));
	62	+ break;
	63	+ }
	64	+ if (lt > i) pushText(html.slice(i, lt));
	65	+ if (html.startsWith("<!--", lt)) {
	66	+ const end = html.indexOf("-->", lt + 4);
	67	+ i = end === -1 ? n : end + 3;
	68	+ continue;
	69	+ }
	70	+ if (html.startsWith("<![CDATA[", lt)) {
	71	+ const end = html.indexOf("]]>", lt);
	72	+ i = end === -1 ? n : end + 3;
	73	+ continue;
	74	+ }
	75	+ if (html[lt + 1] === "!" \|\| html[lt + 1] === "?") {
	76	+ const end = html.indexOf(">", lt);
	77	+ i = end === -1 ? n : end + 1;
	78	+ continue;
	79	+ }
	80	+ const gt = findTagEnd(html, lt);
	81	+ if (gt === -1) {
	82	+ pushText(html.slice(lt));
	83	+ break;
	84	+ }
	85	+ const raw = html.slice(lt + 1, gt);
	86	+ i = gt + 1;
	87	+ if (raw.startsWith("/")) {
	88	+ const tag = raw.slice(1).trim().toLowerCase().split(/\s/)[0]!;
	89	+ // close up to matching open element
	90	+ let p: HNode \| null = cur;
	91	+ while (p && p !== root && p.tag !== tag) p = p.parent;
	92	+ if (p && p !== root) cur = p.parent ?? root;
	93	+ continue;
	94	+ }
	95	+ const m = raw.match(/^([a-zA-Z][a-zA-Z0-9:-]*)/);
	96	+ if (!m) {
	97	+ pushText("<" + raw + ">");
	98	+ continue;
	99	+ }
	100	+ const tag = m[1]!.toLowerCase();
	101	+ const selfClosing = raw.endsWith("/");
	102	+ const attrs: Record<string, string> = {};
	103	+ const attrStr = raw.slice(m[0].length, selfClosing ? -1 : undefined);
	104	+ for (const am of attrStr.matchAll(ATTR_RE)) {
	105	+ const k = am[1]!.toLowerCase();
	106	+ attrs[k] = decodeEntities(am[2] ?? am[3] ?? am[4] ?? "");
	107	+ }
	108	+ // implicit closes
	109	+ let p: HNode \| null = cur;
	110	+ while (p && p !== root) {
	111	+ const ac = AUTO_CLOSE[p.tag];
	112	+ if (ac && ac.has(tag)) {
	113	+ cur = p.parent ?? root;
	114	+ break;
	115	+ }
	116	+ if (p.tag === "p" && BLOCK.has(tag)) {
	117	+ cur = p.parent ?? root;
	118	+ break;
	119	+ }
	120	+ p = p.parent;
	121	+ }
	122	+ const node = makeNode("element", tag, cur);
	123	+ node.attrs = attrs;
	124	+ cur.children.push(node);
	125	+ if (RAW_TEXT.has(tag) && !selfClosing) {
	126	+ const close = html.toLowerCase().indexOf(`</${tag}`, i);
	127	+ const end = close === -1 ? n : close;
	128	+ const t = makeNode("text", "#text", node);
	129	+ t.text = html.slice(i, end);
	130	+ node.children.push(t);
	131	+ i = close === -1 ? n : html.indexOf(">", close) + 1 \|\| n;
	132	+ continue;
	133	+ }
	134	+ if (!VOID.has(tag) && !selfClosing) cur = node;
	135	+ }
	136	+ return root;
	137	+}
	138	+
	139	+function findTagEnd(html: string, from: number): number {
	140	+ let q: string \| null = null;
	141	+ for (let j = from + 1; j < html.length; j++) {
	142	+ const c = html[j]!;
	143	+ if (q) {
	144	+ if (c === q) q = null;
	145	+ } else if (c === '"' \|\| c === "'") {
	146	+ // only treat as quote when inside attribute area (after a space or =)
	147	+ const prev = html[j - 1];
	148	+ if (prev === "=" \|\| prev === " " \|\| prev === "\t" \|\| prev === "\n") q = c;
	149	+ } else if (c === ">") return j;
	150	+ else if (c === "<" && j > from + 1) return -1 + 0 * j; // malformed: give up on this tag
	151	+ }
	152	+ return -1;
	153	+}
	154	+
	155	+// ---------------------------------------------------------------------------
	156	+// Queries
	157	+// ---------------------------------------------------------------------------
	158	+export function walk(node: HNode, fn: (n: HNode) => boolean \| void): void {
	159	+ if (fn(node) === false) return;
	160	+ for (const c of node.children) walk(c, fn);
	161	+}
	162	+
	163	+export function findAll(root: HNode, pred: (n: HNode) => boolean): HNode[] {
	164	+ const out: HNode[] = [];
	165	+ walk(root, (n) => {
	166	+ if (n.type === "element" && pred(n)) out.push(n);
	167	+ });
	168	+ return out;
	169	+}
	170	+
	171	+export function textOf(node: HNode): string {
	172	+ let s = "";
	173	+ walk(node, (n) => {
	174	+ if (n.type === "element" && RAW_TEXT.has(n.tag)) return false;
	175	+ if (n.type === "text") s += n.text;
	176	+ });
	177	+ return decodeEntities(s).replace(/\s+/g, " ").trim();
	178	+}
	179	+
	180	+// ---------------------------------------------------------------------------
	181	+// Main content extraction (readability-lite)
	182	+// ---------------------------------------------------------------------------
	183	+const NOISE_TAGS = new Set(["nav", "header", "footer", "aside", "form", "button", "select", "input", "label", "dialog", "menu"]);
	184	+const NOISE_RE = /(^\|[\s_-])(nav\|menu\|sidebar\|side-bar\|footer\|header\|banner\|cookie\|consent\|gdpr\|modal\|popup\|newsletter\|subscribe\|social\|share\|sharing\|related\|recommend\|promo\|advert\|ad-\|ads\|sponsor\|breadcrumb\|comment\|widget\|toolbar\|skip\|masthead\|sitemap\|legal\|copyright\|login\|signup\|search)([\s_-]\|$)/i;
	185	+const CONTENT_RE = /(^\|[\s_-])(article\|content\|main\|post\|entry\|body\|story\|text\|blog\|product\|description\|prose\|markdown\|documentation\|docs)([\s_-]\|$)/i;
	186	+
	187	+function classId(n: HNode): string {
	188	+ return `${n.attrs["class"] ?? ""} ${n.attrs["id"] ?? ""}`;
	189	+}
	190	+
	191	+export function isNoise(n: HNode): boolean {
	192	+ if (n.type !== "element") return false;
	193	+ if (RAW_TEXT.has(n.tag)) return true;
	194	+ if (NOISE_TAGS.has(n.tag)) return true;
	195	+ if (n.attrs["hidden"] !== undefined \|\| /display\s:\snone/i.test(n.attrs["style"] ?? "")) return true;
	196	+ if (n.attrs["aria-hidden"] === "true") return true;
	197	+ const role = n.attrs["role"];
	198	+ if (role && /navigation\|banner\|contentinfo\|complementary\|dialog\|search\|menu/i.test(role)) return true;
	199	+ if (n.tag === "div" \|\| n.tag === "section" \|\| n.tag === "ul" \|\| n.tag === "span") {
	200	+ const ci = classId(n);
	201	+ if (NOISE_RE.test(ci) && !CONTENT_RE.test(ci)) return true;
	202	+ }
	203	+ return false;
	204	+}
	205	+
	206	+/** Pick the element that most likely holds the primary content. Falls back to <body>/root. */
	207	+export function mainContent(root: HNode): HNode {
	208	+ const explicit = findAll(root, (n) => n.tag === "article" \|\| n.tag === "main" \|\| n.attrs["role"] === "main" \|\| n.attrs["itemprop"] === "articleBody");
	209	+ const body = findAll(root, (n) => n.tag === "body")[0] ?? root;
	210	+ const scored: Array<{ node: HNode; score: number }> = [];
	211	+ const candidates = explicit.length ? explicit : findAll(body, (n) => ["div", "section", "td", "article", "main"].includes(n.tag));
	212	+ const bodyLen = Math.max(1, textOf(body).length);
	213	+ for (const c of candidates) {
	214	+ if (isNoise(c)) continue;
	215	+ const text = textOf(c);
	216	+ if (text.length < 200) continue;
	217	+ const paragraphs = findAll(c, (n) => n.tag === "p" \|\| n.tag === "h1" \|\| n.tag === "h2" \|\| n.tag === "h3" \|\| n.tag === "li" \|\| n.tag === "pre").length;
	218	+ const links = findAll(c, (n) => n.tag === "a").reduce((s, a) => s + textOf(a).length, 0);
	219	+ const linkDensity = links / Math.max(1, text.length);
	220	+ let score = text.length / bodyLen + Math.min(paragraphs, 40) / 40;
	221	+ score *= 1 - Math.min(0.9, linkDensity);
	222	+ if (c.tag === "article" \|\| c.tag === "main" \|\| c.attrs["role"] === "main") score *= 1.6;
	223	+ if (CONTENT_RE.test(classId(c))) score *= 1.2;
	224	+ scored.push({ node: c, score });
	225	+ }
	226	+ if (!scored.length) return body;
	227	+ scored.sort((a, b) => b.score - a.score);
	228	+ const best = scored[0]!;
	229	+ // Guard against choosing a tiny fragment: require the pick to hold ≥ 25% of the body text.
	230	+ if (textOf(best.node).length < bodyLen * 0.25 && !explicit.length) return body;
	231	+ return best.node;
	232	+}
	233	+
	234	+// ---------------------------------------------------------------------------
	235	+// Markdown serialisation
	236	+// ---------------------------------------------------------------------------
	237	+export interface MarkdownOptions {
	238	+ baseUrl?: string;
	239	+ /** Isolate the main content (default true). */
	240	+ mainContent?: boolean;
	241	+ /** Keep images as ![alt](src) (default true). */
	242	+ images?: boolean;
	243	+ /** Keep hyperlinks as [text](href) (default true); false renders plain text. */
	244	+ links?: boolean;
	245	+ /** Drop navigation/footers/asides etc. even inside the main content (default true). */
	246	+ stripNoise?: boolean;
	247	+ /** Maximum output length in characters (default 2,000,000). */
	248	+ maxLength?: number;
	249	+}
	250	+
	251	+function absolutize(href: string \| undefined, base?: string): string \| null {
	252	+ if (!href) return null;
	253	+ const h = href.trim();
	254	+ if (!h \|\| h.startsWith("javascript:") \|\| h.startsWith("data:") \|\| h.startsWith("mailto:") \|\| h.startsWith("tel:") \|\| h === "#") return h.startsWith("mailto:") \|\| h.startsWith("tel:") ? h : null;
	255	+ try {
	256	+ return base ? new URL(h, base).toString() : new URL(h).toString();
	257	+ } catch {
	258	+ return null;
	259	+ }
	260	+}
	261	+
	262	+function esc(s: string): string {
	263	+ return s.replace(/([\\`*_{}[\]<>])/g, "\\$1");
	264	+}
	265	+
	266	+class MdWriter {
	267	+ out: string[] = [];
	268	+ private listStack: Array<{ ordered: boolean; index: number }> = [];
	269	+ constructor(private readonly opts: Required<Pick<MarkdownOptions, "images" \| "links" \| "stripNoise">> & { baseUrl?: string }) {}
	270	+
	271	+ block(s: string) {
	272	+ const t = s.replace(/\n{3,}/g, "\n\n").trim();
	273	+ if (!t) return;
	274	+ this.out.push(t);
	275	+ }
	276	+
	277	+ render(node: HNode): string {
	278	+ this.out = [];
	279	+ this.renderChildren(node, "");
	280	+ return this.out.join("\n\n").replace(/\n{3,}/g, "\n\n").trim();
	281	+ }
	282	+
	283	+ private inline(node: HNode): string {
	284	+ let s = "";
	285	+ for (const c of node.children) {
	286	+ if (c.type === "text") {
	287	+ s += decodeEntities(c.text).replace(/\s+/g, " ");
	288	+ continue;
	289	+ }
	290	+ if (this.opts.stripNoise && isNoise(c)) continue;
	291	+ switch (c.tag) {
	292	+ case "br":
	293	+ s += " \n";
	294	+ break;
	295	+ case "strong":
	296	+ case "b": {
	297	+ const t = this.inline(c).trim();
	298	+ s += t ? `${t}` : "";
	299	+ break;
	300	+ }
	301	+ case "em":
	302	+ case "i": {
	303	+ const t = this.inline(c).trim();
	304	+ s += t ? `${t}` : "";
	305	+ break;
	306	+ }
	307	+ case "del":
	308	+ case "s":
	309	+ case "strike": {
	310	+ const t = this.inline(c).trim();
	311	+ s += t ? `~~${t}~~` : "";
	312	+ break;
	313	+ }
	314	+ case "code":
	315	+ case "kbd":
	316	+ case "samp":
	317	+ case "var": {
	318	+ const t = textOf(c);
	319	+ s += t ? `\`${t.replace(/`/g, "\\`")}\`` : "";
	320	+ break;
	321	+ }
	322	+ case "a": {
	323	+ const t = this.inline(c).trim();
	324	+ const href = absolutize(c.attrs["href"], this.opts.baseUrl);
	325	+ if (!t && !href) break;
	326	+ if (!this.opts.links \|\| !href) s += t;
	327	+ else s += `[${t \|\| href}](${href})`;
	328	+ break;
	329	+ }
	330	+ case "img": {
	331	+ if (!this.opts.images) break;
	332	+ const src = absolutize(c.attrs["src"] ?? c.attrs["data-src"] ?? (c.attrs["srcset"] ?? "").split(/[\s,]/)[0], this.opts.baseUrl);
	333	+ if (!src) break;
	334	+ const alt = (c.attrs["alt"] ?? "").replace(/\s+/g, " ").trim();
	335	+ s += `![${alt}](${src})`;
	336	+ break;
	337	+ }
	338	+ case "sup":
	339	+ s += `^${this.inline(c).trim()}`;
	340	+ break;
	341	+ case "sub":
	342	+ s += `_${this.inline(c).trim()}`;
	343	+ break;
	344	+ case "q":
	345	+ s += `"${this.inline(c).trim()}"`;
	346	+ break;
	347	+ case "input": {
	348	+ if (c.attrs["type"] === "checkbox") s += c.attrs["checked"] !== undefined ? "[x] " : "[ ] ";
	349	+ break;
	350	+ }
	351	+ default:
	352	+ if (BLOCK.has(c.tag)) {
	353	+ // block inside inline context: flush as text with breaks
	354	+ s += "\n" + this.inlineBlock(c) + "\n";
	355	+ } else s += this.inline(c);
	356	+ }
	357	+ }
	358	+ return s;
	359	+ }
	360	+
	361	+ private inlineBlock(node: HNode): string {
	362	+ const w = new MdWriter(this.opts);
	363	+ return w.render(node);
	364	+ }
	365	+
	366	+ private renderChildren(node: HNode, prefix: string) {
	367	+ let inlineBuf = "";
	368	+ const flush = () => {
	369	+ const t = inlineBuf.replace(/[ \t]+/g, " ").replace(/ \n /g, "\n").trim();
	370	+ if (t) this.block(prefix + t.split("\n").join("\n" + prefix));
	371	+ inlineBuf = "";
	372	+ };
	373	+ for (const c of node.children) {
	374	+ if (c.type === "text") {
	375	+ inlineBuf += decodeEntities(c.text).replace(/\s+/g, " ");
	376	+ continue;
	377	+ }
	378	+ if (this.opts.stripNoise && isNoise(c)) continue;
	379	+ if (!BLOCK.has(c.tag) && !["table", "ul", "ol", "dl"].includes(c.tag)) {
	380	+ inlineBuf += this.inline({ ...c, children: [c] } as HNode);
	381	+ continue;
	382	+ }
	383	+ flush();
	384	+ this.renderBlock(c, prefix);
	385	+ }
	386	+ flush();
	387	+ }
	388	+
	389	+ private renderBlock(c: HNode, prefix: string) {
	390	+ switch (c.tag) {
	391	+ case "h1":
	392	+ case "h2":
	393	+ case "h3":
	394	+ case "h4":
	395	+ case "h5":
	396	+ case "h6": {
	397	+ const level = Number(c.tag[1]);
	398	+ const t = this.inline(c).replace(/\s+/g, " ").trim();
	399	+ if (t) this.block(`${prefix}${"#".repeat(level)} ${t}`);
	400	+ return;
	401	+ }
	402	+ case "p":
	403	+ case "div":
	404	+ case "section":
	405	+ case "article":
	406	+ case "main":
	407	+ case "header":
	408	+ case "footer":
	409	+ case "aside":
	410	+ case "nav":
	411	+ case "address":
	412	+ case "center":
	413	+ case "details":
	414	+ case "summary":
	415	+ case "figure":
	416	+ case "dialog":
	417	+ case "fieldset":
	418	+ case "form":
	419	+ case "body":
	420	+ case "html": {
	421	+ const hasBlocks = c.children.some((k) => k.type === "element" && (BLOCK.has(k.tag) \|\| ["table", "ul", "ol", "dl"].includes(k.tag)));
	422	+ if (hasBlocks) this.renderChildren(c, prefix);
	423	+ else {
	424	+ const t = this.inline(c).replace(/[ \t]+/g, " ").trim();
	425	+ if (t) this.block(prefix + t.split("\n").map((l) => l.trim()).join("\n" + prefix));
	426	+ }
	427	+ return;
	428	+ }
	429	+ case "figcaption": {
	430	+ const t = this.inline(c).trim();
	431	+ if (t) this.block(`${prefix}${t}`);
	432	+ return;
	433	+ }
	434	+ case "blockquote": {
	435	+ const inner = new MdWriter(this.opts).render(c);
	436	+ if (inner) this.block(inner.split("\n").map((l) => `${prefix}> ${l}`).join("\n"));
	437	+ return;
	438	+ }
	439	+ case "pre": {
	440	+ const codeEl = c.children.find((k) => k.type === "element" && k.tag === "code");
	441	+ const lang = ((codeEl?.attrs["class"] ?? c.attrs["class"] ?? "").match(/(?:language\|lang)-([a-z0-9+#-]+)/i) ?? [])[1] ?? "";
	442	+ let code = "";
	443	+ walk(c, (n) => {
	444	+ if (n.type === "text") code += decodeEntities(n.text);
	445	+ });
	446	+ code = code.replace(/^\n+\|\n+$/g, "");
	447	+ const fence = code.includes("```") ? "````" : "```";
	448	+ this.block(`${prefix}${fence}${lang}\n${code.split("\n").map((l) => prefix + l).join("\n")}\n${prefix}${fence}`);
	449	+ return;
	450	+ }
	451	+ case "hr":
	452	+ this.block(`${prefix}---`);
	453	+ return;
	454	+ case "ul":
	455	+ case "ol": {
	456	+ const ordered = c.tag === "ol";
	457	+ let idx = Number(c.attrs["start"] ?? 1) \|\| 1;
	458	+ const items: string[] = [];
	459	+ for (const li of c.children) {
	460	+ if (li.type !== "element" \|\| li.tag !== "li") continue;
	461	+ const marker = ordered ? `${idx++}. ` : "- ";
	462	+ const w = new MdWriter(this.opts);
	463	+ const inner = w.render(li);
	464	+ if (!inner) continue;
	465	+ const lines = inner.split("\n");
	466	+ const pad = " ".repeat(marker.length);
	467	+ items.push(prefix + marker + lines[0] + (lines.length > 1 ? "\n" + lines.slice(1).map((l) => (l ? prefix + pad + l : "")).join("\n") : ""));
	468	+ }
	469	+ if (items.length) this.block(items.join("\n"));
	470	+ return;
	471	+ }
	472	+ case "li": {
	473	+ const w = new MdWriter(this.opts);
	474	+ const inner = w.render(c);
	475	+ if (inner) this.block(prefix + "- " + inner.split("\n").join("\n" + prefix + " "));
	476	+ return;
	477	+ }
	478	+ case "dl": {
	479	+ const lines: string[] = [];
	480	+ for (const k of c.children) {
	481	+ if (k.type !== "element") continue;
	482	+ if (k.tag === "dt") lines.push(`${prefix}${this.inline(k).trim()}`);
	483	+ if (k.tag === "dd") lines.push(`${prefix}: ${this.inline(k).trim()}`);
	484	+ }
	485	+ if (lines.length) this.block(lines.join("\n"));
	486	+ return;
	487	+ }
	488	+ case "table": {
	489	+ const rows: string[][] = [];
	490	+ let headerRow: string[] \| null = null;
	491	+ const trs = findAll(c, (n) => n.tag === "tr").filter((tr) => {
	492	+ // exclude nested tables' rows
	493	+ let p = tr.parent;
	494	+ while (p && p !== c) {
	495	+ if (p.tag === "table") return false;
	496	+ p = p.parent;
	497	+ }
	498	+ return true;
	499	+ });
	500	+ for (const tr of trs) {
	501	+ const cells = tr.children.filter((k) => k.type === "element" && (k.tag === "td" \|\| k.tag === "th"));
	502	+ if (!cells.length) continue;
	503	+ const vals = cells.map((cell) => this.inline(cell).replace(/\s\n\s/g, " ").replace(/\\|/g, "\\\|").trim());
	504	+ const isHeader = !headerRow && cells.every((cell) => cell.tag === "th");
	505	+ if (isHeader) headerRow = vals;
	506	+ else rows.push(vals);
	507	+ }
	508	+ if (!headerRow && !rows.length) return;
	509	+ const width = Math.max(headerRow?.length ?? 0, ...rows.map((r) => r.length));
	510	+ const pad = (r: string[]) => [...r, ...Array(Math.max(0, width - r.length)).fill("")];
	511	+ const head = headerRow ? pad(headerRow) : Array(width).fill(" ");
	512	+ const lines = [`${prefix}\| ${head.join(" \| ")} \|`, `${prefix}\| ${Array(width).fill("---").join(" \| ")} \|`, ...rows.map((r) => `${prefix}\| ${pad(r).join(" \| ")} \|`)];
	513	+ this.block(lines.join("\n"));
	514	+ return;
	515	+ }
	516	+ case "tr":
	517	+ case "td":
	518	+ case "th":
	519	+ case "thead":
	520	+ case "tbody":
	521	+ case "tfoot":
	522	+ case "dd":
	523	+ case "dt": {
	524	+ const t = this.inline(c).trim();
	525	+ if (t) this.block(prefix + t);
	526	+ return;
	527	+ }
	528	+ default: {
	529	+ this.renderChildren(c, prefix);
	530	+ }
	531	+ }
	532	+ }
	533	+}
	534	+
	535	+export function htmlToMarkdown(html: string, opts: MarkdownOptions = {}): string {
	536	+ const root = parseHtml(html);
	537	+ const scope = opts.mainContent === false ? (findAll(root, (n) => n.tag === "body")[0] ?? root) : mainContent(root);
	538	+ const writer = new MdWriter({ images: opts.images ?? true, links: opts.links ?? true, stripNoise: opts.stripNoise ?? true, baseUrl: opts.baseUrl });
	539	+ let md = writer.render(scope);
	540	+ // Prepend the document title when the content does not already start with a heading.
	541	+ const title = findAll(root, (n) => n.tag === "title")[0];
	542	+ const t = title ? textOf(title) : "";
	543	+ if (t && !/^#\s/.test(md) && !md.startsWith(`# ${t}`)) md = `# ${esc(t)}\n\n${md}`;
	544	+ const max = opts.maxLength ?? 2_000_000;
	545	+ return md.length > max ? md.slice(0, max) : md;
	546	+}
	547	+
	548	+/** Readable text (main content, boilerplate removed). Falls back to whole document. */
	549	+export function htmlToMainText(html: string): string {
	550	+ const root = parseHtml(html);
	551	+ const scope = mainContent(root);
	552	+ const w = new MdWriter({ images: false, links: false, stripNoise: true });
	553	+ return w
	554	+ .render(scope)
	555	+ .replace(/^#+\s*/gm, "")
	556	+ .replace(/\\\|~~\|(?<!\\)\*/g, "")
	557	+ .replace(/\\([\\`*_{}[\]<>])/g, "$1")
	558	+ .trim();
	559	+}
	560	+
	561	+// ---------------------------------------------------------------------------
	562	+// Metadata & links
	563	+// ---------------------------------------------------------------------------
	564	+export function registrableHost(hostname: string): string {
	565	+ const parts = hostname.toLowerCase().replace(/^www\./, "").split(".");
	566	+ if (parts.length <= 2) return parts.join(".");
	567	+ const sld = new Set(["co", "com", "org", "net", "gov", "edu", "ac", "gc", "qc", "on", "bc"]);
	568	+ if (parts.length >= 3 && sld.has(parts[parts.length - 2]!) && parts[parts.length - 1]!.length === 2) return parts.slice(-3).join(".");
	569	+ return parts.slice(-2).join(".");
	570	+}
	571	+
	572	+export function extractPageMetadata(html: string, baseUrl: string): { page: PageMetadata; links: PageLink[] } {
	573	+ const root = parseHtml(html.length > 3_000_000 ? html.slice(0, 3_000_000) : html);
	574	+ const head = findAll(root, (n) => n.tag === "head")[0] ?? root;
	575	+ const titleEl = findAll(head, (n) => n.tag === "title")[0] ?? findAll(root, (n) => n.tag === "title")[0];
	576	+ const og: Record<string, string> = {};
	577	+ let description: string \| null = null;
	578	+ let canonical: string \| null = null;
	579	+ for (const m of findAll(root, (n) => n.tag === "meta")) {
	580	+ const name = (m.attrs["name"] ?? m.attrs["property"] ?? "").toLowerCase();
	581	+ const content = (m.attrs["content"] ?? "").trim();
	582	+ if (!name \|\| !content) continue;
	583	+ if (name === "description" && !description) description = content.slice(0, 1000);
	584	+ if (name.startsWith("og:") \|\| name.startsWith("twitter:") \|\| name.startsWith("article:")) og[name] = content.slice(0, 1000);
	585	+ }
	586	+ for (const l of findAll(root, (n) => n.tag === "link")) {
	587	+ if ((l.attrs["rel"] ?? "").toLowerCase().split(/\s+/).includes("canonical")) {
	588	+ canonical = absolutize(l.attrs["href"], baseUrl);
	589	+ break;
	590	+ }
	591	+ }
	592	+ const htmlEl = findAll(root, (n) => n.tag === "html")[0];
	593	+ const lang = htmlEl?.attrs["lang"]?.trim().slice(0, 16) \|\| og["og:locale"]?.slice(0, 16) \|\| null;
	594	+
	595	+ let baseHref = baseUrl;
	596	+ const baseEl = findAll(head, (n) => n.tag === "base")[0];
	597	+ if (baseEl?.attrs["href"]) baseHref = absolutize(baseEl.attrs["href"], baseUrl) ?? baseUrl;
	598	+
	599	+ const seen = new Set<string>();
	600	+ const links: PageLink[] = [];
	601	+ let host = "";
	602	+ try {
	603	+ host = registrableHost(new URL(baseUrl).hostname);
	604	+ } catch {
	605	+ /* ignore */
	606	+ }
	607	+ for (const a of findAll(root, (n) => n.tag === "a" \|\| n.tag === "area")) {
	608	+ const url = absolutize(a.attrs["href"], baseHref);
	609	+ if (!url \|\| !/^https?:/i.test(url)) continue;
	610	+ const clean = url.replace(/#.*$/, "");
	611	+ if (!clean \|\| seen.has(clean)) continue;
	612	+ seen.add(clean);
	613	+ let internal = false;
	614	+ try {
	615	+ internal = registrableHost(new URL(clean).hostname) === host;
	616	+ } catch {
	617	+ /* ignore */
	618	+ }
	619	+ const rel = (a.attrs["rel"] ?? "").toLowerCase();
	620	+ links.push({ url: clean, text: textOf(a).slice(0, 200) \|\| (a.attrs["title"] ?? a.attrs["aria-label"] ?? "").slice(0, 200), internal, nofollow: /\bnofollow\b/.test(rel) });
	621	+ if (links.length >= 5000) break;
	622	+ }
	623	+ return {
	624	+ page: {
	625	+ title: titleEl ? textOf(titleEl).slice(0, 300) \|\| null : og["og:title"]?.slice(0, 300) ?? null,
	626	+ description: description ?? og["og:description"] ?? null,
	627	+ canonical,
	628	+ lang,
	629	+ og,
	630	+ links_count: links.length,
	631	+ },
	632	+ links,
	633	+ };
	634	+}
	635	+
	636	+/** Glob (``, ``) or `/regex/flags` pattern matcher for URL filtering. /
	637	+export function urlPatternMatcher(patterns: string[] \| undefined): ((url: string) => boolean) \| null {
	638	+ if (!patterns?.length) return null;
	639	+ const res: RegExp[] = [];
	640	+ for (const raw of patterns) {
	641	+ const p = raw.trim();
	642	+ if (!p) continue;
	643	+ const rx = p.match(/^\/(.+)\/([a-z]*)$/);
	644	+ if (rx) {
	645	+ try {
	646	+ res.push(new RegExp(rx[1]!, rx[2]!.replace(/[^gimsuy]/g, "")));
	647	+ continue;
	648	+ } catch {
	649	+ /* fall through to glob */
	650	+ }
	651	+ }
	652	+ const escaped = p.replace(/[.+^${}()\|[\]\\?]/g, "\\$&").replace(/\\/g, "\u0000").replace(/\/g, "[^/]").replace(/\u0000/g, ".*");
	653	+ res.push(new RegExp(p.includes("://") \|\| p.startsWith("/") ? `^${escaped}$` : escaped, "i"));
	654	+ }
	655	+ if (!res.length) return null;
	656	+ return (url: string) => res.some((r) => r.test(url));
	657	+}
	658	+
	659	+/** Normalise a URL for crawl de-duplication: strip fragment, tracking params, trailing slash, sort query. */
	660	+export function normalizeCrawlUrl(url: string): string \| null {
	661	+ try {
	662	+ const u = new URL(url);
	663	+ if (!/^https?:$/.test(u.protocol)) return null;
	664	+ u.hash = "";
	665	+ u.hostname = u.hostname.toLowerCase();
	666	+ const drop = ["utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content", "utm_id", "gclid", "fbclid", "mc_cid", "mc_eid", "ref", "ref_", "_ga", "yclid", "msclkid", "igshid", "spm"];
	667	+ for (const k of drop) u.searchParams.delete(k);
	668	+ u.searchParams.sort();
	669	+ if (u.pathname.length > 1 && u.pathname.endsWith("/")) u.pathname = u.pathname.slice(0, -1);
	670	+ if ((u.protocol === "http:" && u.port === "80") \|\| (u.protocol === "https:" && u.port === "443")) u.port = "";
	671	+ return u.toString();
	672	+ } catch {
	673	+ return null;
	674	+ }
	675	+}
	676	+
	677	+const NON_HTML_EXT = /\.(jpe?g\|png\|gif\|webp\|avif\|svg\|ico\|bmp\|tiff?\|mp4\|mp3\|wav\|ogg\|webm\|mov\|avi\|zip\|gz\|tgz\|rar\|7z\|tar\|pdf\|docx?\|xlsx?\|pptx?\|exe\|dmg\|apk\|css\|js\|mjs\|json\|xml\|rss\|atom\|woff2?\|ttf\|eot\|otf)(\?.*)?$/i;
	678	+
	679	+/** Whether a URL is likely to be an HTML page worth crawling. */
	680	+export function looksLikePage(url: string): boolean {
	681	+ try {
	682	+ const u = new URL(url);
	683	+ return !NON_HTML_EXT.test(u.pathname);
	684	+ } catch {
	685	+ return false;
	686	+ }
	687	+}

added packages/core/src/robots.ts +101 −0

		`@@ -0,0 +1,101 @@`
	1	+/** Minimal robots.txt parser (RFC 9309 semantics: longest-match, allow wins on tie). */
	2	+export interface RobotsRules {
	3	+ allow: string[];
	4	+ disallow: string[];
	5	+ crawlDelayMs: number \| null;
	6	+ sitemaps: string[];
	7	+}
	8	+
	9	+export function parseRobots(txt: string, userAgent = "fetchabot"): RobotsRules {
	10	+ const ua = userAgent.toLowerCase();
	11	+ const groups: Array<{ agents: string[]; allow: string[]; disallow: string[]; crawlDelay: number \| null }> = [];
	12	+ const sitemaps: string[] = [];
	13	+ let cur: (typeof groups)[number] \| null = null;
	14	+ let lastWasAgent = false;
	15	+ for (const rawLine of txt.split(/\r?\n/)) {
	16	+ const line = rawLine.replace(/#.*$/, "").trim();
	17	+ if (!line) continue;
	18	+ const idx = line.indexOf(":");
	19	+ if (idx === -1) continue;
	20	+ const key = line.slice(0, idx).trim().toLowerCase();
	21	+ const value = line.slice(idx + 1).trim();
	22	+ if (key === "sitemap") {
	23	+ if (value) sitemaps.push(value);
	24	+ continue;
	25	+ }
	26	+ if (key === "user-agent") {
	27	+ if (!cur \|\| !lastWasAgent) {
	28	+ cur = { agents: [], allow: [], disallow: [], crawlDelay: null };
	29	+ groups.push(cur);
	30	+ }
	31	+ cur.agents.push(value.toLowerCase());
	32	+ lastWasAgent = true;
	33	+ continue;
	34	+ }
	35	+ lastWasAgent = false;
	36	+ if (!cur) continue;
	37	+ if (key === "allow") cur.allow.push(value);
	38	+ else if (key === "disallow") cur.disallow.push(value);
	39	+ else if (key === "crawl-delay") {
	40	+ const n = Number(value);
	41	+ if (Number.isFinite(n)) cur.crawlDelay = n;
	42	+ }
	43	+ }
	44	+ // Pick the most specific group: exact UA token match, else "*".
	45	+ let chosen = groups.find((g) => g.agents.some((a) => a !== "*" && ua.includes(a)));
	46	+ if (!chosen) chosen = groups.find((g) => g.agents.includes("*"));
	47	+ return {
	48	+ allow: chosen?.allow ?? [],
	49	+ disallow: chosen?.disallow.filter(Boolean) ?? [],
	50	+ crawlDelayMs: chosen?.crawlDelay !== null && chosen?.crawlDelay !== undefined ? Math.min(30_000, Math.round(chosen.crawlDelay * 1000)) : null,
	51	+ sitemaps,
	52	+ };
	53	+}
	54	+
	55	+function patternToRegex(p: string): RegExp {
	56	+ let anchored = false;
	57	+ let pat = p;
	58	+ if (pat.endsWith("$")) {
	59	+ anchored = true;
	60	+ pat = pat.slice(0, -1);
	61	+ }
	62	+ const esc = pat.replace(/[.+^${}()\|[\]\\?]/g, "\\$&").replace(/\/g, ".");
	63	+ return new RegExp(`^${esc}${anchored ? "$" : ""}`);
	64	+}
	65	+
	66	+export function robotsAllows(rules: RobotsRules, url: string): boolean {
	67	+ let path: string;
	68	+ try {
	69	+ const u = new URL(url);
	70	+ path = u.pathname + u.search;
	71	+ } catch {
	72	+ return false;
	73	+ }
	74	+ let best: { allow: boolean; len: number } \| null = null;
	75	+ for (const p of rules.allow) {
	76	+ if (p && patternToRegex(p).test(path) && (!best \|\| p.length > best.len \|\| (p.length === best.len && !best.allow))) best = { allow: true, len: p.length };
	77	+ }
	78	+ for (const p of rules.disallow) {
	79	+ if (p && patternToRegex(p).test(path) && (!best \|\| p.length > best.len)) best = { allow: false, len: p.length };
	80	+ }
	81	+ return best ? best.allow : true;
	82	+}
	83	+
	84	+/** Extract <loc> URLs from a sitemap or sitemap index. Returns { urls, sitemaps }. */
	85	+export function parseSitemap(xml: string): { urls: string[]; sitemaps: string[] } {
	86	+ const urls: string[] = [];
	87	+ const sitemaps: string[] = [];
	88	+ const isIndex = /<sitemapindex/i.test(xml);
	89	+ for (const m of xml.matchAll(/<loc>\s([^<\s]+)\s<\/loc>/gi)) {
	90	+ const loc = m[1]!.replace(/&/g, "&").trim();
	91	+ (isIndex ? sitemaps : urls).push(loc);
	92	+ }
	93	+ if (!urls.length && !sitemaps.length) {
	94	+ // plain-text sitemap
	95	+ for (const line of xml.split(/\r?\n/)) {
	96	+ const t = line.trim();
	97	+ if (/^https?:\/\//i.test(t)) urls.push(t);
	98	+ }
	99	+ }
	100	+ return { urls, sitemaps };
	101	+}

modified packages/core/src/schema.ts +150 −64

		`@@ -9,7 +9,7 @@ export type ConcreteNetwork = Exclude<NetworkClass, "auto">;`
9	9	export const HTTP_METHODS = ["GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"] as const;
10	10	export type HttpMethod = (typeof HTTP_METHODS)[number];
11	11
12		−export const OUTPUT_FORMATS = ["html", "text", "json", "raw"] as const;
	12	+export const OUTPUT_FORMATS = ["html", "text", "markdown", "json", "raw"] as const;
13	13	export type OutputFormat = (typeof OUTPUT_FORMATS)[number];
14	14
15	15	export const DEVICES = ["desktop", "mobile", "tablet"] as const;
		`@@ -36,10 +36,22 @@ export const fetchRequestSchema = z`
36	36	city: z.string().max(128).optional(),
37	37	network: z.enum(NETWORK_CLASSES).default("auto"),
38	38	session: z.string().max(64).optional(),
	39	+ /** Render in a managed headless browser (real Chromium) routed through the same network. */
39	40	browser: z.boolean().default(false),
	41	+ /** When an HTTP attempt is blocked by a JavaScript challenge / anti-bot, automatically escalate to the browser. */
	42	+ browser_fallback: z.boolean().default(true),
40	43	javascript: z.boolean().optional(),
41	44	wait_for: z.string().max(512).optional(),
42	45	wait_ms: z.number().int().min(0).max(30_000).optional(),
	46	+ wait_until: z.enum(["load", "domcontentloaded", "networkidle"]).default("domcontentloaded"),
	47	+ /** Browser: skip images, fonts and media to save bandwidth (default true). */
	48	+ block_resources: z.boolean().default(true),
	49	+ /** Browser: return a PNG screenshot (base64) in `screenshot`. */
	50	+ screenshot: z.boolean().default(false),
	51	+ /** Include the list of hyperlinks found in the page (`links`). */
	52	+ links: z.boolean().default(false),
	53	+ /** Referer strategy: "auto" (none on first try, search-engine referer on retries), "none", or a literal URL. */
	54	+ referer: z.union([z.enum(["auto", "none"]), z.string().url().max(2048)]).default("auto"),
43	55	device: z.enum(DEVICES).optional(),
44	56	locale: z.string().max(16).optional(),
45	57	format: z.enum(OUTPUT_FORMATS).default("html"),
		`@@ -85,9 +97,28 @@ export interface FetchTiming {`
85	97	total_ms: number;
86	98	}
87	99
	100	+export interface PageMetadata {
	101	+ title: string \| null;
	102	+ description: string \| null;
	103	+ canonical: string \| null;
	104	+ lang: string \| null;
	105	+ og: Record<string, string>;
	106	+ /** Number of hyperlinks found (the list itself is only returned with `links: true`). */
	107	+ links_count: number;
	108	+}
	109	+
	110	+export interface PageLink {
	111	+ url: string;
	112	+ text: string;
	113	+ internal: boolean;
	114	+ nofollow: boolean;
	115	+}
	116	+
88	117	export interface FetchMetadata {
89	118	network: ConcreteNetwork \| "direct";
90	119	country: string \| null;
	120	+ /** "http" for a plain fetch, "browser" when the final attempt was rendered in the managed browser. */
	121	+ mode: "http" \| "browser";
91	122	attempts: number;
92	123	duration_ms: number;
93	124	bytes: number;
		`@@ -99,8 +130,10 @@ export interface FetchMetadata {`
99	130	attempts: Array<{
100	131	provider: string;
101	132	network: string;
	133	+ mode: "http" \| "browser";
102	134	country: string \| null;
103	135	outcome: string;
	136	+ block_reason?: string \| null;
104	137	status: number \| null;
105	138	duration_ms: number;
106	139	error?: string;
		`@@ -123,9 +156,21 @@ export interface FetchResponseBody {`
123	156	text?: string \| null;
124	157	/** Present when `format` = json and the body parsed. */
125	158	json?: unknown;
	159	+ /** Present when `format` = markdown. */
	160	+ markdown?: string \| null;
	161	+ /** Parsed page metadata (HTML responses only). */
	162	+ page?: PageMetadata \| null;
	163	+ /** Present when `links: true` (HTML responses only). */
	164	+ links?: PageLink[];
	165	+ /** Present when `screenshot: true` in browser mode: PNG, base64. */
	166	+ screenshot?: string \| null;
126	167	}
127	168
128		−export const PLANS = ["free", "developer", "growth", "business", "enterprise"] as const;
	169	+/**
	170	+ * Fetcha is a private platform: there is a single plan and it is unlimited. Access is granted by
	171	+ * an administrator (signup allowlist). Legacy plan names from the public preview map to it.
	172	+ */
	173	+export const PLANS = ["unlimited"] as const;
129	174	export type Plan = (typeof PLANS)[number];
130	175
131	176	export interface PlanLimits {
		`@@ -141,80 +186,121 @@ export interface PlanLimits {`
141	186	included_gb: number;
142	187	overage_per_1k_requests_usd: number;
143	188	residential_per_gb_usd: number;
	189	+ /** Managed browser rendering available. */
	190	+ browser: boolean;
	191	+ /** Max concurrent browser renders per organization. */
	192	+ browser_concurrency: number;
	193	+ /** Crawl jobs: max pages per job and concurrent jobs per organization. */
	194	+ crawl_max_pages: number;
	195	+ crawl_concurrent_jobs: number;
144	196	}
145	197
146	198	export const PLAN_LIMITS: Record<Plan, PlanLimits> = {
147		− free: {
148		− plan: "free",
149		− label: "Free",
150		− monthly_requests: 1000,
151		− concurrency: 5,
152		− max_timeout_ms: 30_000,
153		− max_retries: 2,
154		− networks: ["datacenter", "residential"],
155		− retention_days: 3,
156		− price_usd_month: 0,
157		− included_gb: 0.5,
158		− overage_per_1k_requests_usd: 0,
159		− residential_per_gb_usd: 0,
160		− },
161		− developer: {
162		− plan: "developer",
163		− label: "Developer",
164		− monthly_requests: 50_000,
165		− concurrency: 25,
166		− max_timeout_ms: 60_000,
167		− max_retries: 3,
168		− networks: ["datacenter", "residential", "isp"],
169		− retention_days: 7,
170		− price_usd_month: 29,
171		− included_gb: 5,
172		− overage_per_1k_requests_usd: 0.6,
173		− residential_per_gb_usd: 9,
174		− },
175		− growth: {
176		− plan: "growth",
177		− label: "Growth",
178		− monthly_requests: 500_000,
179		− concurrency: 100,
180		− max_timeout_ms: 90_000,
181		− max_retries: 4,
182		− networks: ["datacenter", "residential", "isp", "mobile"],
183		− retention_days: 30,
184		− price_usd_month: 149,
185		− included_gb: 30,
186		− overage_per_1k_requests_usd: 0.4,
187		− residential_per_gb_usd: 7.5,
188		− },
189		− business: {
190		− plan: "business",
191		− label: "Business",
192		− monthly_requests: 5_000_000,
193		− concurrency: 500,
194		− max_timeout_ms: 120_000,
195		− max_retries: 5,
196		− networks: ["datacenter", "residential", "isp", "mobile"],
197		− retention_days: 90,
198		− price_usd_month: 599,
199		− included_gb: 150,
200		− overage_per_1k_requests_usd: 0.25,
201		− residential_per_gb_usd: 6,
202		− },
203		− enterprise: {
204		− plan: "enterprise",
205		− label: "Enterprise",
	199	+ unlimited: {
	200	+ plan: "unlimited",
	201	+ label: "Unlimited",
206	202	monthly_requests: Number.MAX_SAFE_INTEGER,
207		− concurrency: 2000,
	203	+ concurrency: 200,
208	204	max_timeout_ms: 120_000,
209	205	max_retries: 5,
210	206	networks: ["datacenter", "residential", "isp", "mobile"],
211		− retention_days: 365,
	207	+ retention_days: 90,
212	208	price_usd_month: 0,
213	209	included_gb: 0,
214	210	overage_per_1k_requests_usd: 0,
215	211	residential_per_gb_usd: 0,
	212	+ browser: true,
	213	+ browser_concurrency: 8,
	214	+ crawl_max_pages: 2000,
	215	+ crawl_concurrent_jobs: 5,
216	216	},
217	217	};
218	218
219		−export const API_KEY_SCOPES = ["fetch:execute", "browser:use", "sessions:write", "usage:read"] as const;
	219	+/** Map any stored plan value (including legacy free/developer/growth/business/enterprise) to the single plan. */
	220	+export function normalizePlan(_plan: string \| null \| undefined): Plan {
	221	+ return "unlimited";
	222	+}
	223	+
	224	+export function isUnlimited(limits: PlanLimits): boolean {
	225	+ return limits.monthly_requests >= Number.MAX_SAFE_INTEGER;
	226	+}
	227	+
	228	+// ---------------------------------------------------------------------------
	229	+// Crawl & map
	230	+// ---------------------------------------------------------------------------
	231	+export const CRAWL_FORMATS = ["markdown", "text", "html"] as const;
	232	+export type CrawlFormat = (typeof CRAWL_FORMATS)[number];
	233	+
	234	+export const crawlCreateSchema = z
	235	+ .object({
	236	+ url: z.string().min(1).max(8192),
	237	+ /** Maximum number of pages to fetch (the seed counts as one). */
	238	+ max_pages: z.number().int().min(1).max(5000).default(25),
	239	+ /** Maximum link depth from the seed (0 = seed only). */
	240	+ max_depth: z.number().int().min(0).max(10).default(2),
	241	+ /** Only follow links on the seed's registrable host (default true). */
	242	+ same_domain: z.boolean().default(true),
	243	+ /** Also follow links on subdomains of the seed host. */
	244	+ allow_subdomains: z.boolean().default(false),
	245	+ /** Only crawl URLs matching at least one of these patterns (glob with ``, or /regex/). /
	246	+ include_patterns: z.array(z.string().max(512)).max(50).optional(),
	247	+ /** Never crawl URLs matching one of these patterns. */
	248	+ exclude_patterns: z.array(z.string().max(512)).max(50).optional(),
	249	+ /** Honour robots.txt disallow rules for the seed host (default true). */
	250	+ respect_robots: z.boolean().default(true),
	251	+ /** Also seed the frontier with URLs from the site's sitemap(s). */
	252	+ use_sitemap: z.boolean().default(false),
	253	+ /** Parallel page fetches within this job. */
	254	+ concurrency: z.number().int().min(1).max(10).default(3),
	255	+ /** Fixed pause between page fetches per worker (politeness). */
	256	+ delay_ms: z.number().int().min(0).max(30_000).default(0),
	257	+ /** Per-page timeout. */
	258	+ timeout: z.number().int().min(1000).max(120_000).default(30_000),
	259	+ format: z.enum(CRAWL_FORMATS).default("markdown"),
	260	+ /** Keep only the main content (article/main) when converting to markdown/text. */
	261	+ main_content: z.boolean().default(true),
	262	+ country: z
	263	+ .string()
	264	+ .length(2)
	265	+ .transform((s) => s.toUpperCase())
	266	+ .optional(),
	267	+ network: z.enum(NETWORK_CLASSES).default("auto"),
	268	+ browser: z.boolean().default(false),
	269	+ browser_fallback: z.boolean().default(true),
	270	+ headers: headerRecord.optional(),
	271	+ /** Optional POST-back URL called once when the job finishes. */
	272	+ webhook_url: z.string().url().max(2048).optional(),
	273	+ label: z.string().max(128).optional(),
	274	+ })
	275	+ .strict();
	276	+export type CrawlCreateInput = z.output<typeof crawlCreateSchema>;
	277	+
	278	+export const CRAWL_STATUSES = ["queued", "running", "completed", "failed", "cancelled"] as const;
	279	+export type CrawlStatus = (typeof CRAWL_STATUSES)[number];
	280	+
	281	+export const mapCreateSchema = z
	282	+ .object({
	283	+ url: z.string().min(1).max(8192),
	284	+ /** Maximum number of URLs to return. */
	285	+ limit: z.number().int().min(1).max(10_000).default(1000),
	286	+ /** Include sitemap.xml (and sitemaps listed in robots.txt). */
	287	+ use_sitemap: z.boolean().default(true),
	288	+ /** Include hyperlinks from the seed page. */
	289	+ use_links: z.boolean().default(true),
	290	+ same_domain: z.boolean().default(true),
	291	+ allow_subdomains: z.boolean().default(false),
	292	+ /** Filter results by substring / glob / regex. */
	293	+ search: z.string().max(256).optional(),
	294	+ country: z
	295	+ .string()
	296	+ .length(2)
	297	+ .transform((s) => s.toUpperCase())
	298	+ .optional(),
	299	+ network: z.enum(NETWORK_CLASSES).default("auto"),
	300	+ timeout: z.number().int().min(1000).max(120_000).default(30_000),
	301	+ })
	302	+ .strict();
	303	+export type MapCreateInput = z.output<typeof mapCreateSchema>;
	304	+
	305	+export const API_KEY_SCOPES = ["fetch:execute", "browser:use", "crawl:execute", "sessions:write", "usage:read"] as const;
220	306	export type ApiKeyScope = (typeof API_KEY_SCOPES)[number];

modified packages/core/src/text.ts +180 −13

		`@@ -4,8 +4,11 @@ export function htmlToText(html: string): string {`
4	4	s = s.replace(/<script[\s\S]*?<\/script>/gi, " ");
5	5	s = s.replace(/<style[\s\S]*?<\/style>/gi, " ");
6	6	s = s.replace(/<noscript[\s\S]*?<\/noscript>/gi, " ");
	7	+ s = s.replace(/<template[\s\S]*?<\/template>/gi, " ");
	8	+ s = s.replace(/<svg[\s\S]*?<\/svg>/gi, " ");
7	9	s = s.replace(/<!--[\s\S]*?-->/g, " ");
8		− s = s.replace(/<(br\|\/p\|\/div\|\/li\|\/h[1-6]\|\/tr\|\/section\|\/article\|\/header\|\/footer)[^>]*>/gi, "\n");
	10	+ s = s.replace(/<(br\|\/p\|\/div\|\/li\|\/h[1-6]\|\/tr\|\/section\|\/article\|\/header\|\/footer\|\/blockquote\|\/pre\|\/table\|\/ul\|\/ol\|\/dd\|\/dt)[^>]*>/gi, "\n");
	11	+ s = s.replace(/<\/t[dh]>/gi, "\t");
9	12	s = s.replace(/<[^>]+>/g, " ");
10	13	s = decodeEntities(s);
11	14	s = s.replace(/[ \t\f\v]+/g, " ");
		`@@ -23,22 +26,59 @@ const ENTITIES: Record<string, string> = {`
23	26	nbsp: " ",
24	27	copy: "©",
25	28	reg: "®",
	29	+ trade: "™",
26	30	hellip: "…",
27	31	mdash: "—",
28	32	ndash: "–",
29	33	laquo: "«",
30	34	raquo: "»",
	35	+ lsquo: "‘",
	36	+ rsquo: "’",
	37	+ ldquo: "“",
	38	+ rdquo: "”",
	39	+ bull: "•",
	40	+ middot: "·",
	41	+ euro: "€",
	42	+ pound: "£",
	43	+ yen: "¥",
	44	+ cent: "¢",
	45	+ deg: "°",
	46	+ times: "×",
	47	+ divide: "÷",
	48	+ plusmn: "±",
	49	+ frac12: "½",
	50	+ frac14: "¼",
	51	+ frac34: "¾",
31	52	eacute: "é",
32	53	egrave: "è",
	54	+ ecirc: "ê",
	55	+ euml: "ë",
33	56	agrave: "à",
	57	+ aacute: "á",
	58	+ acirc: "â",
	59	+ auml: "ä",
34	60	ccedil: "ç",
	61	+ iuml: "ï",
	62	+ icirc: "î",
	63	+ ocirc: "ô",
	64	+ ouml: "ö",
	65	+ ugrave: "ù",
	66	+ ucirc: "û",
	67	+ uuml: "ü",
	68	+ ntilde: "ñ",
	69	+ szlig: "ß",
	70	+ oelig: "œ",
	71	+ aelig: "æ",
	72	+ Eacute: "É",
	73	+ Agrave: "À",
	74	+ Ccedil: "Ç",
35	75	};
36	76
37	77	export function decodeEntities(s: string): string {
38	78	return s
39	79	.replace(/&#x([0-9a-f]+);/gi, (_, h) => safeChar(parseInt(h, 16)))
40	80	.replace(/&#(\d+);/g, (_, d) => safeChar(parseInt(d, 10)))
41		− .replace(/&([a-z]+);/gi, (m, name) => ENTITIES[name.toLowerCase()] ?? m);
	81	+ .replace(/&([a-z0-9]+);/gi, (m, name) => ENTITIES[name] ?? ENTITIES[name.toLowerCase()] ?? m);
42	82	}
43	83
44	84	function safeChar(code: number): string {
		`@@ -51,18 +91,145 @@ function safeChar(code: number): string {`
51	91
52	92	export function extractTitle(html: string): string \| null {
53	93	const m = html.match(/<title[^>]>([\s\S]?)<\/title>/i);
54		− return m ? decodeEntities(m[1]!).trim().slice(0, 300) : null;
	94	+ return m ? decodeEntities(m[1]!).replace(/\s+/g, " ").trim().slice(0, 300) \|\| null : null;
55	95	}
56	96
57		−/** Heuristic block-page detection used by the retry engine. */
58		−export function looksBlocked(status: number, body: string, headers: Record<string, string>): { blocked: boolean; reason?: string } {
59		− if (status === 403 \|\| status === 429 \|\| status === 407 \|\| status === 999) return { blocked: true, reason: `http_${status}` };
60		− if (status === 503 && /cloudflare\|just a moment\|attention required/i.test(body)) return { blocked: true, reason: "challenge" };
61		− const head = body.slice(0, 20_000);
62		− if (/cf-chl\|challenge-platform\|__cf_chl\|Just a moment\.\.\./i.test(head)) return { blocked: true, reason: "cloudflare_challenge" };
63		− if (/captcha\|recaptcha\|hcaptcha\|g-recaptcha\|Are you a robot\|verify you are human\|unusual traffic/i.test(head)) return { blocked: true, reason: "captcha" };
64		− if (/access denied\|request blocked\|has been blocked\|perimeterx\|_px[a-z0-9]\|datadome\|incapsula\|distil_r/i.test(head)) return { blocked: true, reason: "anti_bot" };
65		− const server = (headers["server"] ?? "").toLowerCase();
66		− if (status >= 400 && /akamai\|imperva\|sucuri/.test(server)) return { blocked: true, reason: "waf" };
	97	+// ---------------------------------------------------------------------------
	98	+// Block / anti-bot detection
	99	+// ---------------------------------------------------------------------------
	100	+
	101	+export type BlockVendor =
	102	+ \| "cloudflare"
	103	+ \| "datadome"
	104	+ \| "perimeterx"
	105	+ \| "akamai"
	106	+ \| "kasada"
	107	+ \| "imperva"
	108	+ \| "aws_waf"
	109	+ \| "vercel"
	110	+ \| "shape"
	111	+ \| "distil"
	112	+ \| "fastly"
	113	+ \| "google"
	114	+ \| "sucuri"
	115	+ \| "generic";
	116	+
	117	+export interface BlockVerdict {
	118	+ blocked: boolean;
	119	+ /** Stable reason code: http_403, http_429, cloudflare_challenge, captcha, anti_bot, waf, soft_block, empty_html, rate_limited… */
	120	+ reason?: string;
	121	+ vendor?: BlockVendor;
	122	+ /** True when the block is a JavaScript challenge that a real browser can typically pass. */
	123	+ challenge?: boolean;
	124	+ /** Retry-After hint in milliseconds, when the origin sent one. */
	125	+ retryAfterMs?: number;
	126	+}
	127	+
	128	+const HTML_CT = /text\/html\|application\/xhtml/i;
	129	+
	130	+function parseRetryAfter(v: string \| undefined): number \| undefined {
	131	+ if (!v) return undefined;
	132	+ const n = Number(v);
	133	+ if (Number.isFinite(n)) return Math.max(0, Math.round(n * 1000));
	134	+ const d = Date.parse(v);
	135	+ if (!Number.isNaN(d)) return Math.max(0, d - Date.now());
	136	+ return undefined;
	137	+}
	138	+
	139	+/**
	140	+ * Heuristic block-page detection used by the retry engine. Looks at the status, the response
	141	+ * headers (WAF signatures) and the first 40 KB of the body (challenge markup, captcha vendors,
	142	+ * "access denied" pages served with a 200).
	143	+ */
	144	+export function looksBlocked(status: number, body: string, headers: Record<string, string>): BlockVerdict {
	145	+ const h = lowerKeys(headers);
	146	+ const server = (h["server"] ?? "").toLowerCase();
	147	+ const ct = h["content-type"] ?? "";
	148	+ const isHtml = !ct \|\| HTML_CT.test(ct);
	149	+ const head = body.slice(0, 40_000);
	150	+ const retryAfterMs = parseRetryAfter(h["retry-after"]);
	151	+
	152	+ // --- header-level signals (strongest) -------------------------------------
	153	+ if (h["cf-mitigated"] === "challenge") return { blocked: true, reason: "cloudflare_challenge", vendor: "cloudflare", challenge: true, retryAfterMs };
	154	+ if (h["x-datadome"] \|\| h["x-dd-b"] \|\| /datadome/i.test(h["set-cookie"] ?? "")) {
	155	+ if (status === 403 \|\| status === 401 \|\| status === 429 \|\| /captcha-delivery\|geo\.captcha\|dd\.js\|DataDome/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true, retryAfterMs };
	156	+ }
	157	+ if (h["x-kpsdk-ct"] \|\| h["x-kpsdk-c"] \|\| /kpsdk\|ips\.js/i.test(head) && status >= 400) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true, retryAfterMs };
	158	+ if (h["x-amzn-waf-action"] === "challenge" \|\| h["x-amzn-waf-action"] === "captcha") return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true, retryAfterMs };
	159	+ if (h["x-vercel-mitigated"] === "challenge" \|\| h["x-vercel-protection-bypass"] !== undefined && status === 403) return { blocked: true, reason: "anti_bot", vendor: "vercel", challenge: true, retryAfterMs };
	160	+ if (status === 403 && (h["x-iinfo"] \|\| /incap_ses\|visid_incap/i.test(h["set-cookie"] ?? ""))) return { blocked: true, reason: "waf", vendor: "imperva", challenge: /_Incapsula_Resource/i.test(head), retryAfterMs };
	161	+ if (status === 403 && /_abck\|ak_bmsc\|bm_sz/i.test(h["set-cookie"] ?? "")) return { blocked: true, reason: "anti_bot", vendor: "akamai", challenge: false, retryAfterMs };
	162	+
	163	+ // --- status-level signals --------------------------------------------------
	164	+ if (status === 407) return { blocked: true, reason: "http_407", vendor: "generic" };
	165	+ if (status === 429) return { blocked: true, reason: "rate_limited", vendor: vendorFrom(server, head), retryAfterMs };
	166	+ if (status === 999) return { blocked: true, reason: "http_999", vendor: "generic" };
	167	+ if (status === 403) return { blocked: true, reason: "http_403", vendor: vendorFrom(server, head), challenge: /cf-chl\|challenge-platform\|__cf_chl\|Just a moment/i.test(head), retryAfterMs };
	168	+ if (status === 401 && /captcha\|challenge\|bot\|automated/i.test(head)) return { blocked: true, reason: "http_401", vendor: vendorFrom(server, head) };
	169	+ if (status === 503 && /cloudflare\|just a moment\|attention required\|checking your browser\|ddos-guard\|checking if the site connection is secure/i.test(head)) {
	170	+ return { blocked: true, reason: "cloudflare_challenge", vendor: "cloudflare", challenge: true, retryAfterMs };
	171	+ }
	172	+ if ((status === 503 \|\| status === 520 \|\| status === 521 \|\| status === 522 \|\| status === 523 \|\| status === 524 \|\| status === 525 \|\| status === 526 \|\| status === 530) && /cloudflare/i.test(server + " " + head)) {
	173	+ return { blocked: true, reason: `origin_${status}`, vendor: "cloudflare", challenge: false, retryAfterMs };
	174	+ }
	175	+ if (status === 405 && /akamai/i.test(server)) return { blocked: true, reason: "waf", vendor: "akamai" };
	176	+ if (status === 202 && /datadome\|captcha-delivery/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };
	177	+
	178	+ // --- body-level signals (only meaningful for HTML) -------------------------
	179	+ if (!isHtml) return { blocked: false };
	180	+ if (/cf-chl\|challenge-platform\|__cf_chl\|cf_chl_opt\|<title>\s*Just a moment\|cf-turnstile\|turnstile\.js\|challenges\.cloudflare\.com/i.test(head)) {
	181	+ return { blocked: true, reason: "cloudflare_challenge", vendor: "cloudflare", challenge: true, retryAfterMs };
	182	+ }
	183	+ if (/geo\.captcha-delivery\.com\|dd\.js\|datadome\|<title>\s*DataDome/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };
	184	+ if (/px-captcha\|_pxhd\|_pxvid\|perimeterx\|human-challenge\|<title>\s*Access to this page has been denied/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "perimeterx", challenge: true };
	185	+ if (/Reference #\d\|Reference #\d+\.[0-9a-f]+\.\d+\.[0-9a-f]+\|errors\.edgesuite\.net\|akamai\.com\/us\/en\/policies/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "akamai", challenge: false };
	186	+ if (/_Incapsula_Resource\|Incapsula incident\|Request unsuccessful\. Incapsula/i.test(head)) return { blocked: true, reason: "waf", vendor: "imperva", challenge: true };
	187	+ if (/awswaf\|aws-waf-token\|challenge\.js\?\|<title>\s*Human Verification/i.test(head)) return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true };
	188	+ if (/kpsdk\|ips\.js\|<title>\s*Kasada/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true };
	189	+ if (/Pardon Our Interruption\|distil_r_captcha\|distil_referrer/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "distil", challenge: true };
	190	+ if (/Vercel Security Checkpoint\|_vercel_challenge/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "vercel", challenge: true };
	191	+ if (/shape-security\|_imp_apg_r_\|<title>\s*Blocked by Shape/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "shape" };
	192	+ if (/sucuri\.net\|Sucuri WebSite Firewall/i.test(head)) return { blocked: true, reason: "waf", vendor: "sucuri" };
	193	+ if (/www\.google\.com\/sorry\/\|Our systems have detected unusual traffic/i.test(head)) return { blocked: true, reason: "captcha", vendor: "google", challenge: false };
	194	+ if (/recaptcha\/api\.js\|g-recaptcha\|hcaptcha\.com\|h-captcha\|arkoselabs\|funcaptcha\|<title>\s[^<]captcha/i.test(head) && /verify\|robot\|human\|unusual traffic\|security check\|confirm you are/i.test(head)) {
	195	+ return { blocked: true, reason: "captcha", vendor: vendorFrom(server, head), challenge: false };
	196	+ }
	197	+ if (/<title>\s*(Access Denied\|Request Blocked\|Forbidden\|Blocked\|Bot Detected\|Security Check\|Attention Required\|Verification Required\|Are you a robot)/i.test(head)) {
	198	+ return { blocked: true, reason: "soft_block", vendor: vendorFrom(server, head) };
	199	+ }
	200	+ if (/enable javascript and cookies to continue\|please enable javascript\|checking your browser before accessing\|verify you are human\|verifying you are human\|are you a robot\|unusual traffic from your (computer\|network)\|automated access to this (site\|page)\|suspected bot\|request could not be satisfied.*(bot\|blocked)/i.test(head) && body.length < 60_000) {
	201	+ return { blocked: true, reason: "soft_block", vendor: vendorFrom(server, head), challenge: /enable javascript\|checking your browser\|verifying/i.test(head) };
	202	+ }
	203	+ if (status >= 400 && /akamai\|imperva\|sucuri\|cloudflare\|awselb\|varnish.*block/i.test(server)) return { blocked: true, reason: "waf", vendor: vendorFrom(server, head) };
	204	+ // A 200 whose document is a script-only shell with no visible text is typically a JS gate.
	205	+ if (status === 200 && /<html/i.test(head) && /<script/i.test(head) && body.length < 20_000) {
	206	+ const visible = body.replace(/<script[\s\S]?<\/script>/gi, "").replace(/<style[\s\S]?<\/style>/gi, "").replace(/<[^>]+>/g, "").replace(/\s+/g, "");
	207	+ if (visible.length < 20 && !/<(img\|a\|form\|input\|p\|h1\|h2\|main\|article)\b/i.test(head)) return { blocked: true, reason: "empty_html", vendor: "generic", challenge: true };
	208	+ }
67	209	return { blocked: false };
68	210	}
	211	+
	212	+function vendorFrom(server: string, head: string): BlockVendor {
	213	+ if (/cloudflare/i.test(server) \|\| /cloudflare/i.test(head)) return "cloudflare";
	214	+ if (/akamai/i.test(server) \|\| /akamai/i.test(head)) return "akamai";
	215	+ if (/imperva\|incapsula/i.test(server + head)) return "imperva";
	216	+ if (/datadome/i.test(head)) return "datadome";
	217	+ if (/perimeterx\|_px/i.test(head)) return "perimeterx";
	218	+ if (/kasada\|kpsdk/i.test(head)) return "kasada";
	219	+ if (/awselb\|aws/i.test(server) && /waf/i.test(head)) return "aws_waf";
	220	+ if (/vercel/i.test(server)) return "vercel";
	221	+ if (/sucuri/i.test(server + head)) return "sucuri";
	222	+ if (/fastly\|varnish/i.test(server)) return "fastly";
	223	+ return "generic";
	224	+}
	225	+
	226	+function lowerKeys(h: Record<string, string>): Record<string, string> {
	227	+ const out: Record<string, string> = {};
	228	+ for (const [k, v] of Object.entries(h)) out[k.toLowerCase()] = v;
	229	+ return out;
	230	+}
	231	+
	232	+/** Whether an origin status is worth retrying on another route (transient upstream/edge errors). */
	233	+export function isTransientStatus(status: number): boolean {
	234	+ return status === 502 \|\| status === 503 \|\| status === 504 \|\| status === 408 \|\| (status >= 520 && status <= 530);
	235	+}

added packages/core/test/markdown.test.ts +80 −0

		`@@ -0,0 +1,80 @@`
	1	+import { describe, expect, it } from "vitest";
	2	+import { extractPageMetadata, htmlToMarkdown, htmlToMainText, normalizeCrawlUrl, parseRobots, parseSitemap, robotsAllows, urlPatternMatcher, looksLikePage } from "../src/index";
	3	+
	4	+const PAGE = `<!doctype html><html lang="fr"><head><title>Titre & test</title>
	5	+<meta name="description" content="Une description."><meta property="og:title" content="OG Titre"><link rel="canonical" href="/canon">
	6	+</head><body>
	7	+<nav><a href="/a">Nav A</a><a href="/b">Nav B</a></nav>
	8	+<main><article>
	9	+<h1>Bonjour</h1>
	10	+<p>Premier <strong>paragraphe</strong> avec un <a href="/lien?utm_source=x#frag">lien</a> et une <img src="/img.png" alt="image">.</p>
	11	+<h2>Liste</h2>
	12	+<ul><li>Un</li><li>Deux <em>deux</em></li></ul>
	13	+<ol start="3"><li>Trois</li></ol>
	14	+<pre><code class="language-js">const x = 1;\nconsole.log(x);</code></pre>
	15	+<table><thead><tr><th>A</th><th>B</th></tr></thead><tbody><tr><td>1</td><td>2 \| 3</td></tr></tbody></table>
	16	+<blockquote><p>Citation</p></blockquote>
	17	+</article></main>
	18	+<footer><a href="https://other.example/x" rel="nofollow">Externe</a> © 2026</footer>
	19	+<script>var a = 1;</script>
	20	+</body></html>`;
	21	+
	22	+describe("markdown", () => {
	23	+ it("converts the main content to markdown", () => {
	24	+ const md = htmlToMarkdown(PAGE, { baseUrl: "https://site.example/dir/page" });
	25	+ expect(md).toContain("# Bonjour");
	26	+ expect(md).toContain("Premier paragraphe avec un [lien](https://site.example/lien?utm_source=x#frag) et une ![image](https://site.example/img.png).");
	27	+ expect(md).toContain("- Un\n- Deux deux");
	28	+ expect(md).toContain("3. Trois");
	29	+ expect(md).toContain("```js\nconst x = 1;\nconsole.log(x);\n```");
	30	+ expect(md).toContain("\| A \| B \|\n\| --- \| --- \|\n\| 1 \| 2 \\\| 3 \|");
	31	+ expect(md).toContain("> Citation");
	32	+ expect(md).not.toContain("Nav A");
	33	+ expect(md).not.toContain("var a = 1");
	34	+ });
	35	+ it("extracts readable main text", () => {
	36	+ const t = htmlToMainText(PAGE);
	37	+ expect(t).toContain("Bonjour");
	38	+ expect(t).toContain("Premier paragraphe");
	39	+ expect(t).not.toContain("Nav A");
	40	+ });
	41	+ it("extracts metadata and links", () => {
	42	+ const { page, links } = extractPageMetadata(PAGE, "https://site.example/dir/page");
	43	+ expect(page.title).toBe("Titre & test");
	44	+ expect(page.description).toBe("Une description.");
	45	+ expect(page.canonical).toBe("https://site.example/canon");
	46	+ expect(page.lang).toBe("fr");
	47	+ expect(page.og["og:title"]).toBe("OG Titre");
	48	+ expect(page.links_count).toBe(links.length);
	49	+ expect(links.find((l) => l.url === "https://site.example/a")?.internal).toBe(true);
	50	+ const ext = links.find((l) => l.url.startsWith("https://other.example"));
	51	+ expect(ext).toMatchObject({ internal: false, nofollow: true, text: "Externe" });
	52	+ expect(links.find((l) => l.url.includes("#"))).toBeUndefined();
	53	+ });
	54	+ it("normalises crawl URLs", () => {
	55	+ expect(normalizeCrawlUrl("HTTPS://Site.Example/path/?b=2&a=1&utm_source=x#frag")).toBe("https://site.example/path?a=1&b=2");
	56	+ expect(normalizeCrawlUrl("mailto:x@y.z")).toBeNull();
	57	+ expect(looksLikePage("https://a.b/c.pdf")).toBe(false);
	58	+ expect(looksLikePage("https://a.b/c")).toBe(true);
	59	+ });
	60	+ it("matches glob and regex patterns", () => {
	61	+ const m = urlPatternMatcher(["https://a.b/blog/*", "/\\/docs\\//"])!;
	62	+ expect(m("https://a.b/blog/post-1")).toBe(true);
	63	+ expect(m("https://a.b/blog/2024/x")).toBe(false);
	64	+ expect(m("https://a.b/docs/x")).toBe(true);
	65	+ expect(m("https://a.b/other")).toBe(false);
	66	+ });
	67	+ it("parses robots.txt and sitemaps", () => {
	68	+ const r = parseRobots("User-agent: *\nDisallow: /private/\nAllow: /private/ok\nCrawl-delay: 2\nSitemap: https://a.b/sitemap.xml\n\nUser-agent: fetchabot\nDisallow: /secret");
	69	+ expect(r.disallow).toEqual(["/secret"]);
	70	+ expect(robotsAllows(r, "https://a.b/secret/x")).toBe(false);
	71	+ expect(robotsAllows(r, "https://a.b/public")).toBe(true);
	72	+ const star = parseRobots("User-agent: *\nDisallow: /private/\nAllow: /private/ok");
	73	+ expect(robotsAllows(star, "https://a.b/private/x")).toBe(false);
	74	+ expect(robotsAllows(star, "https://a.b/private/ok")).toBe(true);
	75	+ expect(star.sitemaps).toEqual([]);
	76	+ expect(r.sitemaps).toEqual(["https://a.b/sitemap.xml"]);
	77	+ expect(parseSitemap("<urlset><url><loc>https://a.b/1</loc></url><url><loc>https://a.b/2</loc></url></urlset>").urls).toEqual(["https://a.b/1", "https://a.b/2"]);
	78	+ expect(parseSitemap("<sitemapindex><sitemap><loc>https://a.b/s1.xml</loc></sitemap></sitemapindex>").sitemaps).toEqual(["https://a.b/s1.xml"]);
	79	+ });
	80	+});

modified packages/core/test/ssrf.test.ts +7 −1

		`@@ -76,8 +76,14 @@ describe("block detection & text", () => {`
76	76	it("detects challenge pages and captchas", () => {
77	77	expect(looksBlocked(403, "", {}).blocked).toBe(true);
78	78	expect(looksBlocked(503, "<title>Just a moment...</title>", {}).blocked).toBe(true);
79		− expect(looksBlocked(200, "<div class='g-recaptcha'></div>", {}).blocked).toBe(true);
	79	+ expect(looksBlocked(200, "<html><head><title>Security check</title></head><body><div class='g-recaptcha'></div>Please verify you are human</body></html>", {}).blocked).toBe(true);
	80	+ expect(looksBlocked(200, "<html><body><h1>Contact</h1><form><div class='g-recaptcha'></div><button>Send</button></form></body></html>", {}).blocked).toBe(false);
80	81	expect(looksBlocked(200, "<html><body>Hello</body></html>", {}).blocked).toBe(false);
	82	+ expect(looksBlocked(200, "<html><body></body></html>", { "cf-mitigated": "challenge" })).toMatchObject({ blocked: true, reason: "cloudflare_challenge", challenge: true });
	83	+ expect(looksBlocked(403, "<html><body>Access Denied</body></html>", { server: "AkamaiGHost", "set-cookie": "_abck=1; Path=/" })).toMatchObject({ blocked: true, vendor: "akamai" });
	84	+ expect(looksBlocked(429, "", { "retry-after": "3" })).toMatchObject({ blocked: true, reason: "rate_limited", retryAfterMs: 3000 });
	85	+ expect(looksBlocked(200, "<html><head><script src='/x.js'></script></head><body></body></html>", {})).toMatchObject({ blocked: true, reason: "empty_html" });
	86	+ expect(looksBlocked(200, "<html><head><title>DataDome</title></head><body><script src='https://geo.captcha-delivery.com/captcha/'></script></body></html>", {})).toMatchObject({ blocked: true, vendor: "datadome" });
81	87	});
82	88	it("extracts readable text", () => {
83	89	expect(htmlToText("<h1>Hi</h1><script>x()</script><p>There & back</p>")).toBe("Hi\nThere & back");

added packages/db/drizzle/0001_furry_red_skull.sql +76 −0

		`@@ -0,0 +1,76 @@`
	1	+CREATE TABLE "crawl_jobs" (
	2	+ "id" text PRIMARY KEY NOT NULL,
	3	+ "organization_id" text NOT NULL,
	4	+ "project_id" text NOT NULL,
	5	+ "api_key_id" text,
	6	+ "source" text DEFAULT 'api' NOT NULL,
	7	+ "label" text,
	8	+ "seed_url" text NOT NULL,
	9	+ "domain" text NOT NULL,
	10	+ "options" jsonb NOT NULL,
	11	+ "status" text DEFAULT 'queued' NOT NULL,
	12	+ "worker_id" text,
	13	+ "pages_discovered" integer DEFAULT 0 NOT NULL,
	14	+ "pages_fetched" integer DEFAULT 0 NOT NULL,
	15	+ "pages_ok" integer DEFAULT 0 NOT NULL,
	16	+ "pages_blocked" integer DEFAULT 0 NOT NULL,
	17	+ "pages_failed" integer DEFAULT 0 NOT NULL,
	18	+ "bytes" bigint DEFAULT 0 NOT NULL,
	19	+ "cost_usd" double precision DEFAULT 0 NOT NULL,
	20	+ "error_code" text,
	21	+ "error_message" text,
	22	+ "webhook_status" text,
	23	+ "created_at" timestamp with time zone DEFAULT now() NOT NULL,
	24	+ "started_at" timestamp with time zone,
	25	+ "heartbeat_at" timestamp with time zone,
	26	+ "completed_at" timestamp with time zone
	27	+);
	28	+--> statement-breakpoint
	29	+CREATE TABLE "crawl_pages" (
	30	+ "id" text PRIMARY KEY NOT NULL,
	31	+ "job_id" text NOT NULL,
	32	+ "url" text NOT NULL,
	33	+ "final_url" text,
	34	+ "depth" integer DEFAULT 0 NOT NULL,
	35	+ "parent_url" text,
	36	+ "status" text DEFAULT 'pending' NOT NULL,
	37	+ "http_status" integer,
	38	+ "error_code" text,
	39	+ "request_id" text,
	40	+ "title" text,
	41	+ "description" text,
	42	+ "content_type" text,
	43	+ "content" text,
	44	+ "links_count" integer DEFAULT 0 NOT NULL,
	45	+ "bytes" bigint DEFAULT 0 NOT NULL,
	46	+ "duration_ms" integer,
	47	+ "mode" text,
	48	+ "created_at" timestamp with time zone DEFAULT now() NOT NULL,
	49	+ "fetched_at" timestamp with time zone
	50	+);
	51	+--> statement-breakpoint
	52	+CREATE TABLE "signup_allowlist" (
	53	+ "email" text PRIMARY KEY NOT NULL,
	54	+ "note" text,
	55	+ "invited_by_user_id" text,
	56	+ "invited_at" timestamp with time zone,
	57	+ "used_at" timestamp with time zone,
	58	+ "user_id" text,
	59	+ "created_at" timestamp with time zone DEFAULT now() NOT NULL
	60	+);
	61	+--> statement-breakpoint
	62	+ALTER TABLE "organizations" ALTER COLUMN "plan" SET DEFAULT 'unlimited';--> statement-breakpoint
	63	+ALTER TABLE "fetch_requests" ADD COLUMN "mode" text;--> statement-breakpoint
	64	+ALTER TABLE "request_attempts" ADD COLUMN "mode" text DEFAULT 'http' NOT NULL;--> statement-breakpoint
	65	+ALTER TABLE "crawl_jobs" ADD CONSTRAINT "crawl_jobs_organization_id_organizations_id_fk" FOREIGN KEY ("organization_id") REFERENCES "public"."organizations"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
	66	+ALTER TABLE "crawl_jobs" ADD CONSTRAINT "crawl_jobs_project_id_projects_id_fk" FOREIGN KEY ("project_id") REFERENCES "public"."projects"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
	67	+ALTER TABLE "crawl_jobs" ADD CONSTRAINT "crawl_jobs_api_key_id_api_keys_id_fk" FOREIGN KEY ("api_key_id") REFERENCES "public"."api_keys"("id") ON DELETE set null ON UPDATE no action;--> statement-breakpoint
	68	+ALTER TABLE "crawl_pages" ADD CONSTRAINT "crawl_pages_job_id_crawl_jobs_id_fk" FOREIGN KEY ("job_id") REFERENCES "public"."crawl_jobs"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
	69	+ALTER TABLE "signup_allowlist" ADD CONSTRAINT "signup_allowlist_invited_by_user_id_users_id_fk" FOREIGN KEY ("invited_by_user_id") REFERENCES "public"."users"("id") ON DELETE set null ON UPDATE no action;--> statement-breakpoint
	70	+ALTER TABLE "signup_allowlist" ADD CONSTRAINT "signup_allowlist_user_id_users_id_fk" FOREIGN KEY ("user_id") REFERENCES "public"."users"("id") ON DELETE set null ON UPDATE no action;--> statement-breakpoint
	71	+CREATE INDEX "crawl_jobs_project_created_idx" ON "crawl_jobs" USING btree ("project_id","created_at");--> statement-breakpoint
	72	+CREATE INDEX "crawl_jobs_status_idx" ON "crawl_jobs" USING btree ("status");--> statement-breakpoint
	73	+CREATE INDEX "crawl_pages_job_idx" ON "crawl_pages" USING btree ("job_id","created_at");--> statement-breakpoint
	74	+CREATE UNIQUE INDEX "crawl_pages_job_url_uq" ON "crawl_pages" USING btree ("job_id","url");--> statement-breakpoint
	75	+CREATE INDEX "signup_allowlist_user_idx" ON "signup_allowlist" USING btree ("user_id");--> statement-breakpoint
	76	+UPDATE "organizations" SET "plan" = 'unlimited' WHERE "plan" <> 'unlimited';

added packages/db/drizzle/meta/0001_snapshot.json +3572 −0

		`@@ -0,0 +1,3572 @@`
	1	+{
	2	+ "id": "d9d93da3-7999-4c5e-b296-16790579a722",
	3	+ "prevId": "faa23eec-a36e-40bd-b9e5-570b18c400ec",
	4	+ "version": "7",
	5	+ "dialect": "postgresql",
	6	+ "tables": {
	7	+ "public.abuse_events": {
	8	+ "name": "abuse_events",
	9	+ "schema": "",
	10	+ "columns": {
	11	+ "id": {
	12	+ "name": "id",
	13	+ "type": "text",
	14	+ "primaryKey": true,
	15	+ "notNull": true
	16	+ },
	17	+ "organization_id": {
	18	+ "name": "organization_id",
	19	+ "type": "text",
	20	+ "primaryKey": false,
	21	+ "notNull": false
	22	+ },
	23	+ "project_id": {
	24	+ "name": "project_id",
	25	+ "type": "text",
	26	+ "primaryKey": false,
	27	+ "notNull": false
	28	+ },
	29	+ "request_id": {
	30	+ "name": "request_id",
	31	+ "type": "text",
	32	+ "primaryKey": false,
	33	+ "notNull": false
	34	+ },
	35	+ "kind": {
	36	+ "name": "kind",
	37	+ "type": "text",
	38	+ "primaryKey": false,
	39	+ "notNull": true
	40	+ },
	41	+ "severity": {
	42	+ "name": "severity",
	43	+ "type": "text",
	44	+ "primaryKey": false,
	45	+ "notNull": true,
	46	+ "default": "'low'"
	47	+ },
	48	+ "detail": {
	49	+ "name": "detail",
	50	+ "type": "text",
	51	+ "primaryKey": false,
	52	+ "notNull": false
	53	+ },
	54	+ "resolved": {
	55	+ "name": "resolved",
	56	+ "type": "boolean",
	57	+ "primaryKey": false,
	58	+ "notNull": true,
	59	+ "default": false
	60	+ },
	61	+ "created_at": {
	62	+ "name": "created_at",
	63	+ "type": "timestamp with time zone",
	64	+ "primaryKey": false,
	65	+ "notNull": true,
	66	+ "default": "now()"
	67	+ }
	68	+ },
	69	+ "indexes": {
	70	+ "abuse_events_org_idx": {
	71	+ "name": "abuse_events_org_idx",
	72	+ "columns": [
	73	+ {
	74	+ "expression": "organization_id",
	75	+ "isExpression": false,
	76	+ "asc": true,
	77	+ "nulls": "last"
	78	+ }
	79	+ ],
	80	+ "isUnique": false,
	81	+ "concurrently": false,
	82	+ "method": "btree",
	83	+ "with": {}
	84	+ }
	85	+ },
	86	+ "foreignKeys": {
	87	+ "abuse_events_organization_id_organizations_id_fk": {
	88	+ "name": "abuse_events_organization_id_organizations_id_fk",
	89	+ "tableFrom": "abuse_events",
	90	+ "tableTo": "organizations",
	91	+ "columnsFrom": [
	92	+ "organization_id"
	93	+ ],
	94	+ "columnsTo": [
	95	+ "id"
	96	+ ],
	97	+ "onDelete": "cascade",
	98	+ "onUpdate": "no action"
	99	+ }
	100	+ },
	101	+ "compositePrimaryKeys": {},
	102	+ "uniqueConstraints": {},
	103	+ "policies": {},
	104	+ "checkConstraints": {},
	105	+ "isRLSEnabled": false
	106	+ },
	107	+ "public.accounts": {
	108	+ "name": "accounts",
	109	+ "schema": "",
	110	+ "columns": {
	111	+ "id": {
	112	+ "name": "id",
	113	+ "type": "text",
	114	+ "primaryKey": true,
	115	+ "notNull": true
	116	+ },
	117	+ "account_id": {
	118	+ "name": "account_id",
	119	+ "type": "text",
	120	+ "primaryKey": false,
	121	+ "notNull": true
	122	+ },
	123	+ "provider_id": {
	124	+ "name": "provider_id",
	125	+ "type": "text",
	126	+ "primaryKey": false,
	127	+ "notNull": true
	128	+ },
	129	+ "user_id": {
	130	+ "name": "user_id",
	131	+ "type": "text",
	132	+ "primaryKey": false,
	133	+ "notNull": true
	134	+ },
	135	+ "access_token": {
	136	+ "name": "access_token",
	137	+ "type": "text",
	138	+ "primaryKey": false,
	139	+ "notNull": false
	140	+ },
	141	+ "refresh_token": {
	142	+ "name": "refresh_token",
	143	+ "type": "text",
	144	+ "primaryKey": false,
	145	+ "notNull": false
	146	+ },
	147	+ "id_token": {
	148	+ "name": "id_token",
	149	+ "type": "text",
	150	+ "primaryKey": false,
	151	+ "notNull": false
	152	+ },
	153	+ "access_token_expires_at": {
	154	+ "name": "access_token_expires_at",
	155	+ "type": "timestamp with time zone",
	156	+ "primaryKey": false,
	157	+ "notNull": false
	158	+ },
	159	+ "refresh_token_expires_at": {
	160	+ "name": "refresh_token_expires_at",
	161	+ "type": "timestamp with time zone",
	162	+ "primaryKey": false,
	163	+ "notNull": false
	164	+ },
	165	+ "scope": {
	166	+ "name": "scope",
	167	+ "type": "text",
	168	+ "primaryKey": false,
	169	+ "notNull": false
	170	+ },
	171	+ "password": {
	172	+ "name": "password",
	173	+ "type": "text",
	174	+ "primaryKey": false,
	175	+ "notNull": false
	176	+ },
	177	+ "created_at": {
	178	+ "name": "created_at",
	179	+ "type": "timestamp with time zone",
	180	+ "primaryKey": false,
	181	+ "notNull": true,
	182	+ "default": "now()"
	183	+ },
	184	+ "updated_at": {
	185	+ "name": "updated_at",
	186	+ "type": "timestamp with time zone",
	187	+ "primaryKey": false,
	188	+ "notNull": true,
	189	+ "default": "now()"
	190	+ }
	191	+ },
	192	+ "indexes": {
	193	+ "accounts_user_idx": {
	194	+ "name": "accounts_user_idx",
	195	+ "columns": [
	196	+ {
	197	+ "expression": "user_id",
	198	+ "isExpression": false,
	199	+ "asc": true,
	200	+ "nulls": "last"
	201	+ }
	202	+ ],
	203	+ "isUnique": false,
	204	+ "concurrently": false,
	205	+ "method": "btree",
	206	+ "with": {}
	207	+ }
	208	+ },
	209	+ "foreignKeys": {
	210	+ "accounts_user_id_users_id_fk": {
	211	+ "name": "accounts_user_id_users_id_fk",
	212	+ "tableFrom": "accounts",
	213	+ "tableTo": "users",
	214	+ "columnsFrom": [
	215	+ "user_id"
	216	+ ],
	217	+ "columnsTo": [
	218	+ "id"
	219	+ ],
	220	+ "onDelete": "cascade",
	221	+ "onUpdate": "no action"
	222	+ }
	223	+ },
	224	+ "compositePrimaryKeys": {},
	225	+ "uniqueConstraints": {},
	226	+ "policies": {},
	227	+ "checkConstraints": {},
	228	+ "isRLSEnabled": false
	229	+ },
	230	+ "public.api_keys": {
	231	+ "name": "api_keys",
	232	+ "schema": "",
	233	+ "columns": {
	234	+ "id": {
	235	+ "name": "id",
	236	+ "type": "text",
	237	+ "primaryKey": true,
	238	+ "notNull": true
	239	+ },
	240	+ "organization_id": {
	241	+ "name": "organization_id",
	242	+ "type": "text",
	243	+ "primaryKey": false,
	244	+ "notNull": true
	245	+ },
	246	+ "project_id": {
	247	+ "name": "project_id",
	248	+ "type": "text",
	249	+ "primaryKey": false,
	250	+ "notNull": true
	251	+ },
	252	+ "created_by_user_id": {
	253	+ "name": "created_by_user_id",
	254	+ "type": "text",
	255	+ "primaryKey": false,
	256	+ "notNull": false
	257	+ },
	258	+ "name": {
	259	+ "name": "name",
	260	+ "type": "text",
	261	+ "primaryKey": false,
	262	+ "notNull": true
	263	+ },
	264	+ "key_hash": {
	265	+ "name": "key_hash",
	266	+ "type": "text",
	267	+ "primaryKey": false,
	268	+ "notNull": true
	269	+ },
	270	+ "key_prefix": {
	271	+ "name": "key_prefix",
	272	+ "type": "text",
	273	+ "primaryKey": false,
	274	+ "notNull": true
	275	+ },
	276	+ "last4": {
	277	+ "name": "last4",
	278	+ "type": "text",
	279	+ "primaryKey": false,
	280	+ "notNull": true
	281	+ },
	282	+ "mode": {
	283	+ "name": "mode",
	284	+ "type": "text",
	285	+ "primaryKey": false,
	286	+ "notNull": true,
	287	+ "default": "'live'"
	288	+ },
	289	+ "scopes": {
	290	+ "name": "scopes",
	291	+ "type": "jsonb",
	292	+ "primaryKey": false,
	293	+ "notNull": true,
	294	+ "default": "'[\"fetch:execute\",\"sessions:write\",\"usage:read\"]'::jsonb"
	295	+ },
	296	+ "expires_at": {
	297	+ "name": "expires_at",
	298	+ "type": "timestamp with time zone",
	299	+ "primaryKey": false,
	300	+ "notNull": false
	301	+ },
	302	+ "last_used_at": {
	303	+ "name": "last_used_at",
	304	+ "type": "timestamp with time zone",
	305	+ "primaryKey": false,
	306	+ "notNull": false
	307	+ },
	308	+ "revoked_at": {
	309	+ "name": "revoked_at",
	310	+ "type": "timestamp with time zone",
	311	+ "primaryKey": false,
	312	+ "notNull": false
	313	+ },
	314	+ "rotated_from_id": {
	315	+ "name": "rotated_from_id",
	316	+ "type": "text",
	317	+ "primaryKey": false,
	318	+ "notNull": false
	319	+ },
	320	+ "created_at": {
	321	+ "name": "created_at",
	322	+ "type": "timestamp with time zone",
	323	+ "primaryKey": false,
	324	+ "notNull": true,
	325	+ "default": "now()"
	326	+ }
	327	+ },
	328	+ "indexes": {
	329	+ "api_keys_hash_uq": {
	330	+ "name": "api_keys_hash_uq",
	331	+ "columns": [
	332	+ {
	333	+ "expression": "key_hash",
	334	+ "isExpression": false,
	335	+ "asc": true,
	336	+ "nulls": "last"
	337	+ }
	338	+ ],
	339	+ "isUnique": true,
	340	+ "concurrently": false,
	341	+ "method": "btree",
	342	+ "with": {}
	343	+ },
	344	+ "api_keys_project_idx": {
	345	+ "name": "api_keys_project_idx",
	346	+ "columns": [
	347	+ {
	348	+ "expression": "project_id",
	349	+ "isExpression": false,
	350	+ "asc": true,
	351	+ "nulls": "last"
	352	+ }
	353	+ ],
	354	+ "isUnique": false,
	355	+ "concurrently": false,
	356	+ "method": "btree",
	357	+ "with": {}
	358	+ },
	359	+ "api_keys_org_idx": {
	360	+ "name": "api_keys_org_idx",
	361	+ "columns": [
	362	+ {
	363	+ "expression": "organization_id",
	364	+ "isExpression": false,
	365	+ "asc": true,
	366	+ "nulls": "last"
	367	+ }
	368	+ ],
	369	+ "isUnique": false,
	370	+ "concurrently": false,
	371	+ "method": "btree",
	372	+ "with": {}
	373	+ }
	374	+ },
	375	+ "foreignKeys": {
	376	+ "api_keys_organization_id_organizations_id_fk": {
	377	+ "name": "api_keys_organization_id_organizations_id_fk",
	378	+ "tableFrom": "api_keys",
	379	+ "tableTo": "organizations",
	380	+ "columnsFrom": [
	381	+ "organization_id"
	382	+ ],
	383	+ "columnsTo": [
	384	+ "id"
	385	+ ],
	386	+ "onDelete": "cascade",
	387	+ "onUpdate": "no action"
	388	+ },
	389	+ "api_keys_project_id_projects_id_fk": {
	390	+ "name": "api_keys_project_id_projects_id_fk",
	391	+ "tableFrom": "api_keys",
	392	+ "tableTo": "projects",
	393	+ "columnsFrom": [
	394	+ "project_id"
	395	+ ],
	396	+ "columnsTo": [
	397	+ "id"
	398	+ ],
	399	+ "onDelete": "cascade",
	400	+ "onUpdate": "no action"
	401	+ },
	402	+ "api_keys_created_by_user_id_users_id_fk": {
	403	+ "name": "api_keys_created_by_user_id_users_id_fk",
	404	+ "tableFrom": "api_keys",
	405	+ "tableTo": "users",
	406	+ "columnsFrom": [
	407	+ "created_by_user_id"
	408	+ ],
	409	+ "columnsTo": [
	410	+ "id"
	411	+ ],
	412	+ "onDelete": "set null",
	413	+ "onUpdate": "no action"
	414	+ }
	415	+ },
	416	+ "compositePrimaryKeys": {},
	417	+ "uniqueConstraints": {},
	418	+ "policies": {},
	419	+ "checkConstraints": {},
	420	+ "isRLSEnabled": false
	421	+ },
	422	+ "public.audit_logs": {
	423	+ "name": "audit_logs",
	424	+ "schema": "",
	425	+ "columns": {
	426	+ "id": {
	427	+ "name": "id",
	428	+ "type": "text",
	429	+ "primaryKey": true,
	430	+ "notNull": true
	431	+ },
	432	+ "organization_id": {
	433	+ "name": "organization_id",
	434	+ "type": "text",
	435	+ "primaryKey": false,
	436	+ "notNull": false
	437	+ },
	438	+ "user_id": {
	439	+ "name": "user_id",
	440	+ "type": "text",
	441	+ "primaryKey": false,
	442	+ "notNull": false
	443	+ },
	444	+ "action": {
	445	+ "name": "action",
	446	+ "type": "text",
	447	+ "primaryKey": false,
	448	+ "notNull": true
	449	+ },
	450	+ "target": {
	451	+ "name": "target",
	452	+ "type": "text",
	453	+ "primaryKey": false,
	454	+ "notNull": false
	455	+ },
	456	+ "metadata": {
	457	+ "name": "metadata",
	458	+ "type": "jsonb",
	459	+ "primaryKey": false,
	460	+ "notNull": false
	461	+ },
	462	+ "ip_address": {
	463	+ "name": "ip_address",
	464	+ "type": "text",
	465	+ "primaryKey": false,
	466	+ "notNull": false
	467	+ },
	468	+ "user_agent": {
	469	+ "name": "user_agent",
	470	+ "type": "text",
	471	+ "primaryKey": false,
	472	+ "notNull": false
	473	+ },
	474	+ "created_at": {
	475	+ "name": "created_at",
	476	+ "type": "timestamp with time zone",
	477	+ "primaryKey": false,
	478	+ "notNull": true,
	479	+ "default": "now()"
	480	+ }
	481	+ },
	482	+ "indexes": {
	483	+ "audit_logs_org_created_idx": {
	484	+ "name": "audit_logs_org_created_idx",
	485	+ "columns": [
	486	+ {
	487	+ "expression": "organization_id",
	488	+ "isExpression": false,
	489	+ "asc": true,
	490	+ "nulls": "last"
	491	+ },
	492	+ {
	493	+ "expression": "created_at",
	494	+ "isExpression": false,
	495	+ "asc": true,
	496	+ "nulls": "last"
	497	+ }
	498	+ ],
	499	+ "isUnique": false,
	500	+ "concurrently": false,
	501	+ "method": "btree",
	502	+ "with": {}
	503	+ },
	504	+ "audit_logs_user_idx": {
	505	+ "name": "audit_logs_user_idx",
	506	+ "columns": [
	507	+ {
	508	+ "expression": "user_id",
	509	+ "isExpression": false,
	510	+ "asc": true,
	511	+ "nulls": "last"
	512	+ }
	513	+ ],
	514	+ "isUnique": false,
	515	+ "concurrently": false,
	516	+ "method": "btree",
	517	+ "with": {}
	518	+ }
	519	+ },
	520	+ "foreignKeys": {
	521	+ "audit_logs_organization_id_organizations_id_fk": {
	522	+ "name": "audit_logs_organization_id_organizations_id_fk",
	523	+ "tableFrom": "audit_logs",
	524	+ "tableTo": "organizations",
	525	+ "columnsFrom": [
	526	+ "organization_id"
	527	+ ],
	528	+ "columnsTo": [
	529	+ "id"
	530	+ ],
	531	+ "onDelete": "cascade",
	532	+ "onUpdate": "no action"
	533	+ },
	534	+ "audit_logs_user_id_users_id_fk": {
	535	+ "name": "audit_logs_user_id_users_id_fk",
	536	+ "tableFrom": "audit_logs",
	537	+ "tableTo": "users",
	538	+ "columnsFrom": [
	539	+ "user_id"
	540	+ ],
	541	+ "columnsTo": [
	542	+ "id"
	543	+ ],
	544	+ "onDelete": "set null",
	545	+ "onUpdate": "no action"
	546	+ }
	547	+ },
	548	+ "compositePrimaryKeys": {},
	549	+ "uniqueConstraints": {},
	550	+ "policies": {},
	551	+ "checkConstraints": {},
	552	+ "isRLSEnabled": false
	553	+ },
	554	+ "public.billing_events": {
	555	+ "name": "billing_events",
	556	+ "schema": "",
	557	+ "columns": {
	558	+ "id": {
	559	+ "name": "id",
	560	+ "type": "text",
	561	+ "primaryKey": true,
	562	+ "notNull": true
	563	+ },
	564	+ "organization_id": {
	565	+ "name": "organization_id",
	566	+ "type": "text",
	567	+ "primaryKey": false,
	568	+ "notNull": false
	569	+ },
	570	+ "type": {
	571	+ "name": "type",
	572	+ "type": "text",
	573	+ "primaryKey": false,
	574	+ "notNull": true
	575	+ },
	576	+ "stripe_event_id": {
	577	+ "name": "stripe_event_id",
	578	+ "type": "text",
	579	+ "primaryKey": false,
	580	+ "notNull": false
	581	+ },
	582	+ "payload": {
	583	+ "name": "payload",
	584	+ "type": "jsonb",
	585	+ "primaryKey": false,
	586	+ "notNull": false
	587	+ },
	588	+ "created_at": {
	589	+ "name": "created_at",
	590	+ "type": "timestamp with time zone",
	591	+ "primaryKey": false,
	592	+ "notNull": true,
	593	+ "default": "now()"
	594	+ }
	595	+ },
	596	+ "indexes": {
	597	+ "billing_events_org_idx": {
	598	+ "name": "billing_events_org_idx",
	599	+ "columns": [
	600	+ {
	601	+ "expression": "organization_id",
	602	+ "isExpression": false,
	603	+ "asc": true,
	604	+ "nulls": "last"
	605	+ }
	606	+ ],
	607	+ "isUnique": false,
	608	+ "concurrently": false,
	609	+ "method": "btree",
	610	+ "with": {}
	611	+ }
	612	+ },
	613	+ "foreignKeys": {
	614	+ "billing_events_organization_id_organizations_id_fk": {
	615	+ "name": "billing_events_organization_id_organizations_id_fk",
	616	+ "tableFrom": "billing_events",
	617	+ "tableTo": "organizations",
	618	+ "columnsFrom": [
	619	+ "organization_id"
	620	+ ],
	621	+ "columnsTo": [
	622	+ "id"
	623	+ ],
	624	+ "onDelete": "set null",
	625	+ "onUpdate": "no action"
	626	+ }
	627	+ },
	628	+ "compositePrimaryKeys": {},
	629	+ "uniqueConstraints": {},
	630	+ "policies": {},
	631	+ "checkConstraints": {},
	632	+ "isRLSEnabled": false
	633	+ },
	634	+ "public.crawl_jobs": {
	635	+ "name": "crawl_jobs",
	636	+ "schema": "",
	637	+ "columns": {
	638	+ "id": {
	639	+ "name": "id",
	640	+ "type": "text",
	641	+ "primaryKey": true,
	642	+ "notNull": true
	643	+ },
	644	+ "organization_id": {
	645	+ "name": "organization_id",
	646	+ "type": "text",
	647	+ "primaryKey": false,
	648	+ "notNull": true
	649	+ },
	650	+ "project_id": {
	651	+ "name": "project_id",
	652	+ "type": "text",
	653	+ "primaryKey": false,
	654	+ "notNull": true
	655	+ },
	656	+ "api_key_id": {
	657	+ "name": "api_key_id",
	658	+ "type": "text",
	659	+ "primaryKey": false,
	660	+ "notNull": false
	661	+ },
	662	+ "source": {
	663	+ "name": "source",
	664	+ "type": "text",
	665	+ "primaryKey": false,
	666	+ "notNull": true,
	667	+ "default": "'api'"
	668	+ },
	669	+ "label": {
	670	+ "name": "label",
	671	+ "type": "text",
	672	+ "primaryKey": false,
	673	+ "notNull": false
	674	+ },
	675	+ "seed_url": {
	676	+ "name": "seed_url",
	677	+ "type": "text",
	678	+ "primaryKey": false,
	679	+ "notNull": true
	680	+ },
	681	+ "domain": {
	682	+ "name": "domain",
	683	+ "type": "text",
	684	+ "primaryKey": false,
	685	+ "notNull": true
	686	+ },
	687	+ "options": {
	688	+ "name": "options",
	689	+ "type": "jsonb",
	690	+ "primaryKey": false,
	691	+ "notNull": true
	692	+ },
	693	+ "status": {
	694	+ "name": "status",
	695	+ "type": "text",
	696	+ "primaryKey": false,
	697	+ "notNull": true,
	698	+ "default": "'queued'"
	699	+ },
	700	+ "worker_id": {
	701	+ "name": "worker_id",
	702	+ "type": "text",
	703	+ "primaryKey": false,
	704	+ "notNull": false
	705	+ },
	706	+ "pages_discovered": {
	707	+ "name": "pages_discovered",
	708	+ "type": "integer",
	709	+ "primaryKey": false,
	710	+ "notNull": true,
	711	+ "default": 0
	712	+ },
	713	+ "pages_fetched": {
	714	+ "name": "pages_fetched",
	715	+ "type": "integer",
	716	+ "primaryKey": false,
	717	+ "notNull": true,
	718	+ "default": 0
	719	+ },
	720	+ "pages_ok": {
	721	+ "name": "pages_ok",
	722	+ "type": "integer",
	723	+ "primaryKey": false,
	724	+ "notNull": true,
	725	+ "default": 0
	726	+ },
	727	+ "pages_blocked": {
	728	+ "name": "pages_blocked",
	729	+ "type": "integer",
	730	+ "primaryKey": false,
	731	+ "notNull": true,
	732	+ "default": 0
	733	+ },
	734	+ "pages_failed": {
	735	+ "name": "pages_failed",
	736	+ "type": "integer",
	737	+ "primaryKey": false,
	738	+ "notNull": true,
	739	+ "default": 0
	740	+ },
	741	+ "bytes": {
	742	+ "name": "bytes",
	743	+ "type": "bigint",
	744	+ "primaryKey": false,
	745	+ "notNull": true,
	746	+ "default": 0
	747	+ },
	748	+ "cost_usd": {
	749	+ "name": "cost_usd",
	750	+ "type": "double precision",
	751	+ "primaryKey": false,
	752	+ "notNull": true,
	753	+ "default": 0
	754	+ },
	755	+ "error_code": {
	756	+ "name": "error_code",
	757	+ "type": "text",
	758	+ "primaryKey": false,
	759	+ "notNull": false
	760	+ },
	761	+ "error_message": {
	762	+ "name": "error_message",
	763	+ "type": "text",
	764	+ "primaryKey": false,
	765	+ "notNull": false
	766	+ },
	767	+ "webhook_status": {
	768	+ "name": "webhook_status",
	769	+ "type": "text",
	770	+ "primaryKey": false,
	771	+ "notNull": false
	772	+ },
	773	+ "created_at": {
	774	+ "name": "created_at",
	775	+ "type": "timestamp with time zone",
	776	+ "primaryKey": false,
	777	+ "notNull": true,
	778	+ "default": "now()"
	779	+ },
	780	+ "started_at": {
	781	+ "name": "started_at",
	782	+ "type": "timestamp with time zone",
	783	+ "primaryKey": false,
	784	+ "notNull": false
	785	+ },
	786	+ "heartbeat_at": {
	787	+ "name": "heartbeat_at",
	788	+ "type": "timestamp with time zone",
	789	+ "primaryKey": false,
	790	+ "notNull": false
	791	+ },
	792	+ "completed_at": {
	793	+ "name": "completed_at",
	794	+ "type": "timestamp with time zone",
	795	+ "primaryKey": false,
	796	+ "notNull": false
	797	+ }
	798	+ },
	799	+ "indexes": {
	800	+ "crawl_jobs_project_created_idx": {
	801	+ "name": "crawl_jobs_project_created_idx",
	802	+ "columns": [
	803	+ {
	804	+ "expression": "project_id",
	805	+ "isExpression": false,
	806	+ "asc": true,
	807	+ "nulls": "last"
	808	+ },
	809	+ {
	810	+ "expression": "created_at",
	811	+ "isExpression": false,
	812	+ "asc": true,
	813	+ "nulls": "last"
	814	+ }
	815	+ ],
	816	+ "isUnique": false,
	817	+ "concurrently": false,
	818	+ "method": "btree",
	819	+ "with": {}
	820	+ },
	821	+ "crawl_jobs_status_idx": {
	822	+ "name": "crawl_jobs_status_idx",
	823	+ "columns": [
	824	+ {
	825	+ "expression": "status",
	826	+ "isExpression": false,
	827	+ "asc": true,
	828	+ "nulls": "last"
	829	+ }
	830	+ ],
	831	+ "isUnique": false,
	832	+ "concurrently": false,
	833	+ "method": "btree",
	834	+ "with": {}
	835	+ }
	836	+ },
	837	+ "foreignKeys": {
	838	+ "crawl_jobs_organization_id_organizations_id_fk": {
	839	+ "name": "crawl_jobs_organization_id_organizations_id_fk",
	840	+ "tableFrom": "crawl_jobs",
	841	+ "tableTo": "organizations",
	842	+ "columnsFrom": [
	843	+ "organization_id"
	844	+ ],
	845	+ "columnsTo": [
	846	+ "id"
	847	+ ],
	848	+ "onDelete": "cascade",
	849	+ "onUpdate": "no action"
	850	+ },
	851	+ "crawl_jobs_project_id_projects_id_fk": {
	852	+ "name": "crawl_jobs_project_id_projects_id_fk",
	853	+ "tableFrom": "crawl_jobs",
	854	+ "tableTo": "projects",
	855	+ "columnsFrom": [
	856	+ "project_id"
	857	+ ],
	858	+ "columnsTo": [
	859	+ "id"
	860	+ ],
	861	+ "onDelete": "cascade",
	862	+ "onUpdate": "no action"
	863	+ },
	864	+ "crawl_jobs_api_key_id_api_keys_id_fk": {
	865	+ "name": "crawl_jobs_api_key_id_api_keys_id_fk",
	866	+ "tableFrom": "crawl_jobs",
	867	+ "tableTo": "api_keys",
	868	+ "columnsFrom": [
	869	+ "api_key_id"
	870	+ ],
	871	+ "columnsTo": [
	872	+ "id"
	873	+ ],
	874	+ "onDelete": "set null",
	875	+ "onUpdate": "no action"
	876	+ }
	877	+ },
	878	+ "compositePrimaryKeys": {},
	879	+ "uniqueConstraints": {},
	880	+ "policies": {},
	881	+ "checkConstraints": {},
	882	+ "isRLSEnabled": false
	883	+ },
	884	+ "public.crawl_pages": {
	885	+ "name": "crawl_pages",
	886	+ "schema": "",
	887	+ "columns": {
	888	+ "id": {
	889	+ "name": "id",
	890	+ "type": "text",
	891	+ "primaryKey": true,
	892	+ "notNull": true
	893	+ },
	894	+ "job_id": {
	895	+ "name": "job_id",
	896	+ "type": "text",
	897	+ "primaryKey": false,
	898	+ "notNull": true
	899	+ },
	900	+ "url": {
	901	+ "name": "url",
	902	+ "type": "text",
	903	+ "primaryKey": false,
	904	+ "notNull": true
	905	+ },
	906	+ "final_url": {
	907	+ "name": "final_url",
	908	+ "type": "text",
	909	+ "primaryKey": false,
	910	+ "notNull": false
	911	+ },
	912	+ "depth": {
	913	+ "name": "depth",
	914	+ "type": "integer",
	915	+ "primaryKey": false,
	916	+ "notNull": true,
	917	+ "default": 0
	918	+ },
	919	+ "parent_url": {
	920	+ "name": "parent_url",
	921	+ "type": "text",
	922	+ "primaryKey": false,
	923	+ "notNull": false
	924	+ },
	925	+ "status": {
	926	+ "name": "status",
	927	+ "type": "text",
	928	+ "primaryKey": false,
	929	+ "notNull": true,
	930	+ "default": "'pending'"
	931	+ },
	932	+ "http_status": {
	933	+ "name": "http_status",
	934	+ "type": "integer",
	935	+ "primaryKey": false,
	936	+ "notNull": false
	937	+ },
	938	+ "error_code": {
	939	+ "name": "error_code",
	940	+ "type": "text",
	941	+ "primaryKey": false,
	942	+ "notNull": false
	943	+ },
	944	+ "request_id": {
	945	+ "name": "request_id",
	946	+ "type": "text",
	947	+ "primaryKey": false,
	948	+ "notNull": false
	949	+ },
	950	+ "title": {
	951	+ "name": "title",
	952	+ "type": "text",
	953	+ "primaryKey": false,
	954	+ "notNull": false
	955	+ },
	956	+ "description": {
	957	+ "name": "description",
	958	+ "type": "text",
	959	+ "primaryKey": false,
	960	+ "notNull": false
	961	+ },
	962	+ "content_type": {
	963	+ "name": "content_type",
	964	+ "type": "text",
	965	+ "primaryKey": false,
	966	+ "notNull": false
	967	+ },
	968	+ "content": {
	969	+ "name": "content",
	970	+ "type": "text",
	971	+ "primaryKey": false,
	972	+ "notNull": false
	973	+ },
	974	+ "links_count": {
	975	+ "name": "links_count",
	976	+ "type": "integer",
	977	+ "primaryKey": false,
	978	+ "notNull": true,
	979	+ "default": 0
	980	+ },
	981	+ "bytes": {
	982	+ "name": "bytes",
	983	+ "type": "bigint",
	984	+ "primaryKey": false,
	985	+ "notNull": true,
	986	+ "default": 0
	987	+ },
	988	+ "duration_ms": {
	989	+ "name": "duration_ms",
	990	+ "type": "integer",
	991	+ "primaryKey": false,
	992	+ "notNull": false
	993	+ },
	994	+ "mode": {
	995	+ "name": "mode",
	996	+ "type": "text",
	997	+ "primaryKey": false,
	998	+ "notNull": false
	999	+ },
	1000	+ "created_at": {
	1001	+ "name": "created_at",
	1002	+ "type": "timestamp with time zone",
	1003	+ "primaryKey": false,
	1004	+ "notNull": true,
	1005	+ "default": "now()"
	1006	+ },
	1007	+ "fetched_at": {
	1008	+ "name": "fetched_at",
	1009	+ "type": "timestamp with time zone",
	1010	+ "primaryKey": false,
	1011	+ "notNull": false
	1012	+ }
	1013	+ },
	1014	+ "indexes": {
	1015	+ "crawl_pages_job_idx": {
	1016	+ "name": "crawl_pages_job_idx",
	1017	+ "columns": [
	1018	+ {
	1019	+ "expression": "job_id",
	1020	+ "isExpression": false,
	1021	+ "asc": true,
	1022	+ "nulls": "last"
	1023	+ },
	1024	+ {
	1025	+ "expression": "created_at",
	1026	+ "isExpression": false,
	1027	+ "asc": true,
	1028	+ "nulls": "last"
	1029	+ }
	1030	+ ],
	1031	+ "isUnique": false,
	1032	+ "concurrently": false,
	1033	+ "method": "btree",
	1034	+ "with": {}
	1035	+ },
	1036	+ "crawl_pages_job_url_uq": {
	1037	+ "name": "crawl_pages_job_url_uq",
	1038	+ "columns": [
	1039	+ {
	1040	+ "expression": "job_id",
	1041	+ "isExpression": false,
	1042	+ "asc": true,
	1043	+ "nulls": "last"
	1044	+ },
	1045	+ {
	1046	+ "expression": "url",
	1047	+ "isExpression": false,
	1048	+ "asc": true,
	1049	+ "nulls": "last"
	1050	+ }
	1051	+ ],
	1052	+ "isUnique": true,
	1053	+ "concurrently": false,
	1054	+ "method": "btree",
	1055	+ "with": {}
	1056	+ }
	1057	+ },
	1058	+ "foreignKeys": {
	1059	+ "crawl_pages_job_id_crawl_jobs_id_fk": {
	1060	+ "name": "crawl_pages_job_id_crawl_jobs_id_fk",
	1061	+ "tableFrom": "crawl_pages",
	1062	+ "tableTo": "crawl_jobs",
	1063	+ "columnsFrom": [
	1064	+ "job_id"
	1065	+ ],
	1066	+ "columnsTo": [
	1067	+ "id"
	1068	+ ],
	1069	+ "onDelete": "cascade",
	1070	+ "onUpdate": "no action"
	1071	+ }
	1072	+ },
	1073	+ "compositePrimaryKeys": {},
	1074	+ "uniqueConstraints": {},
	1075	+ "policies": {},
	1076	+ "checkConstraints": {},
	1077	+ "isRLSEnabled": false
	1078	+ },
	1079	+ "public.domain_profiles": {
	1080	+ "name": "domain_profiles",
	1081	+ "schema": "",
	1082	+ "columns": {
	1083	+ "domain": {
	1084	+ "name": "domain",
	1085	+ "type": "text",
	1086	+ "primaryKey": true,
	1087	+ "notNull": true
	1088	+ },
	1089	+ "preferred_network": {
	1090	+ "name": "preferred_network",
	1091	+ "type": "text",
	1092	+ "primaryKey": false,
	1093	+ "notNull": false
	1094	+ },
	1095	+ "preferred_provider": {
	1096	+ "name": "preferred_provider",
	1097	+ "type": "text",
	1098	+ "primaryKey": false,
	1099	+ "notNull": false
	1100	+ },
	1101	+ "requests": {
	1102	+ "name": "requests",
	1103	+ "type": "integer",
	1104	+ "primaryKey": false,
	1105	+ "notNull": true,
	1106	+ "default": 0
	1107	+ },
	1108	+ "successes": {
	1109	+ "name": "successes",
	1110	+ "type": "integer",
	1111	+ "primaryKey": false,
	1112	+ "notNull": true,
	1113	+ "default": 0
	1114	+ },
	1115	+ "blocks": {
	1116	+ "name": "blocks",
	1117	+ "type": "integer",
	1118	+ "primaryKey": false,
	1119	+ "notNull": true,
	1120	+ "default": 0
	1121	+ },
	1122	+ "captchas": {
	1123	+ "name": "captchas",
	1124	+ "type": "integer",
	1125	+ "primaryKey": false,
	1126	+ "notNull": true,
	1127	+ "default": 0
	1128	+ },
	1129	+ "browser_required": {
	1130	+ "name": "browser_required",
	1131	+ "type": "integer",
	1132	+ "primaryKey": false,
	1133	+ "notNull": true,
	1134	+ "default": 0
	1135	+ },
	1136	+ "avg_latency_ms": {
	1137	+ "name": "avg_latency_ms",
	1138	+ "type": "double precision",
	1139	+ "primaryKey": false,
	1140	+ "notNull": true,
	1141	+ "default": 0
	1142	+ },
	1143	+ "route_stats": {
	1144	+ "name": "route_stats",
	1145	+ "type": "jsonb",
	1146	+ "primaryKey": false,
	1147	+ "notNull": true,
	1148	+ "default": "'{}'::jsonb"
	1149	+ },
	1150	+ "policy": {
	1151	+ "name": "policy",
	1152	+ "type": "jsonb",
	1153	+ "primaryKey": false,
	1154	+ "notNull": false
	1155	+ },
	1156	+ "last_seen_at": {
	1157	+ "name": "last_seen_at",
	1158	+ "type": "timestamp with time zone",
	1159	+ "primaryKey": false,
	1160	+ "notNull": true,
	1161	+ "default": "now()"
	1162	+ },
	1163	+ "updated_at": {
	1164	+ "name": "updated_at",
	1165	+ "type": "timestamp with time zone",
	1166	+ "primaryKey": false,
	1167	+ "notNull": true,
	1168	+ "default": "now()"
	1169	+ }
	1170	+ },
	1171	+ "indexes": {},
	1172	+ "foreignKeys": {},
	1173	+ "compositePrimaryKeys": {},
	1174	+ "uniqueConstraints": {},
	1175	+ "policies": {},
	1176	+ "checkConstraints": {},
	1177	+ "isRLSEnabled": false
	1178	+ },
	1179	+ "public.feature_flags": {
	1180	+ "name": "feature_flags",
	1181	+ "schema": "",
	1182	+ "columns": {
	1183	+ "key": {
	1184	+ "name": "key",
	1185	+ "type": "text",
	1186	+ "primaryKey": true,
	1187	+ "notNull": true
	1188	+ },
	1189	+ "enabled": {
	1190	+ "name": "enabled",
	1191	+ "type": "boolean",
	1192	+ "primaryKey": false,
	1193	+ "notNull": true,
	1194	+ "default": false
	1195	+ },
	1196	+ "description": {
	1197	+ "name": "description",
	1198	+ "type": "text",
	1199	+ "primaryKey": false,
	1200	+ "notNull": false
	1201	+ },
	1202	+ "plans": {
	1203	+ "name": "plans",
	1204	+ "type": "jsonb",
	1205	+ "primaryKey": false,
	1206	+ "notNull": false
	1207	+ },
	1208	+ "organization_ids": {
	1209	+ "name": "organization_ids",
	1210	+ "type": "jsonb",
	1211	+ "primaryKey": false,
	1212	+ "notNull": false
	1213	+ },
	1214	+ "updated_at": {
	1215	+ "name": "updated_at",
	1216	+ "type": "timestamp with time zone",
	1217	+ "primaryKey": false,
	1218	+ "notNull": true,
	1219	+ "default": "now()"
	1220	+ }
	1221	+ },
	1222	+ "indexes": {},
	1223	+ "foreignKeys": {},
	1224	+ "compositePrimaryKeys": {},
	1225	+ "uniqueConstraints": {},
	1226	+ "policies": {},
	1227	+ "checkConstraints": {},
	1228	+ "isRLSEnabled": false
	1229	+ },
	1230	+ "public.fetch_requests": {
	1231	+ "name": "fetch_requests",
	1232	+ "schema": "",
	1233	+ "columns": {
	1234	+ "id": {
	1235	+ "name": "id",
	1236	+ "type": "text",
	1237	+ "primaryKey": true,
	1238	+ "notNull": true
	1239	+ },
	1240	+ "organization_id": {
	1241	+ "name": "organization_id",
	1242	+ "type": "text",
	1243	+ "primaryKey": false,
	1244	+ "notNull": true
	1245	+ },
	1246	+ "project_id": {
	1247	+ "name": "project_id",
	1248	+ "type": "text",
	1249	+ "primaryKey": false,
	1250	+ "notNull": true
	1251	+ },
	1252	+ "api_key_id": {
	1253	+ "name": "api_key_id",
	1254	+ "type": "text",
	1255	+ "primaryKey": false,
	1256	+ "notNull": false
	1257	+ },
	1258	+ "source": {
	1259	+ "name": "source",
	1260	+ "type": "text",
	1261	+ "primaryKey": false,
	1262	+ "notNull": true,
	1263	+ "default": "'api'"
	1264	+ },
	1265	+ "url": {
	1266	+ "name": "url",
	1267	+ "type": "text",
	1268	+ "primaryKey": false,
	1269	+ "notNull": true
	1270	+ },
	1271	+ "final_url": {
	1272	+ "name": "final_url",
	1273	+ "type": "text",
	1274	+ "primaryKey": false,
	1275	+ "notNull": false
	1276	+ },
	1277	+ "domain": {
	1278	+ "name": "domain",
	1279	+ "type": "text",
	1280	+ "primaryKey": false,
	1281	+ "notNull": true
	1282	+ },
	1283	+ "method": {
	1284	+ "name": "method",
	1285	+ "type": "text",
	1286	+ "primaryKey": false,
	1287	+ "notNull": true,
	1288	+ "default": "'GET'"
	1289	+ },
	1290	+ "requested_network": {
	1291	+ "name": "requested_network",
	1292	+ "type": "text",
	1293	+ "primaryKey": false,
	1294	+ "notNull": true,
	1295	+ "default": "'auto'"
	1296	+ },
	1297	+ "network": {
	1298	+ "name": "network",
	1299	+ "type": "text",
	1300	+ "primaryKey": false,
	1301	+ "notNull": false
	1302	+ },
	1303	+ "country": {
	1304	+ "name": "country",
	1305	+ "type": "text",
	1306	+ "primaryKey": false,
	1307	+ "notNull": false
	1308	+ },
	1309	+ "region": {
	1310	+ "name": "region",
	1311	+ "type": "text",
	1312	+ "primaryKey": false,
	1313	+ "notNull": false
	1314	+ },
	1315	+ "city": {
	1316	+ "name": "city",
	1317	+ "type": "text",
	1318	+ "primaryKey": false,
	1319	+ "notNull": false
	1320	+ },
	1321	+ "session_id": {
	1322	+ "name": "session_id",
	1323	+ "type": "text",
	1324	+ "primaryKey": false,
	1325	+ "notNull": false
	1326	+ },
	1327	+ "browser": {
	1328	+ "name": "browser",
	1329	+ "type": "boolean",
	1330	+ "primaryKey": false,
	1331	+ "notNull": true,
	1332	+ "default": false
	1333	+ },
	1334	+ "mode": {
	1335	+ "name": "mode",
	1336	+ "type": "text",
	1337	+ "primaryKey": false,
	1338	+ "notNull": false
	1339	+ },
	1340	+ "format": {
	1341	+ "name": "format",
	1342	+ "type": "text",
	1343	+ "primaryKey": false,
	1344	+ "notNull": true,
	1345	+ "default": "'html'"
	1346	+ },
	1347	+ "status": {
	1348	+ "name": "status",
	1349	+ "type": "text",
	1350	+ "primaryKey": false,
	1351	+ "notNull": true,
	1352	+ "default": "'pending'"
	1353	+ },
	1354	+ "http_status": {
	1355	+ "name": "http_status",
	1356	+ "type": "integer",
	1357	+ "primaryKey": false,
	1358	+ "notNull": false
	1359	+ },
	1360	+ "error_code": {
	1361	+ "name": "error_code",
	1362	+ "type": "text",
	1363	+ "primaryKey": false,
	1364	+ "notNull": false
	1365	+ },
	1366	+ "error_message": {
	1367	+ "name": "error_message",
	1368	+ "type": "text",
	1369	+ "primaryKey": false,
	1370	+ "notNull": false
	1371	+ },
	1372	+ "attempts": {
	1373	+ "name": "attempts",
	1374	+ "type": "integer",
	1375	+ "primaryKey": false,
	1376	+ "notNull": true,
	1377	+ "default": 0
	1378	+ },
	1379	+ "latency_ms": {
	1380	+ "name": "latency_ms",
	1381	+ "type": "integer",
	1382	+ "primaryKey": false,
	1383	+ "notNull": false
	1384	+ },
	1385	+ "bytes_in": {
	1386	+ "name": "bytes_in",
	1387	+ "type": "bigint",
	1388	+ "primaryKey": false,
	1389	+ "notNull": true,
	1390	+ "default": 0
	1391	+ },
	1392	+ "bytes_out": {
	1393	+ "name": "bytes_out",
	1394	+ "type": "bigint",
	1395	+ "primaryKey": false,
	1396	+ "notNull": true,
	1397	+ "default": 0
	1398	+ },
	1399	+ "cost_usd": {
	1400	+ "name": "cost_usd",
	1401	+ "type": "double precision",
	1402	+ "primaryKey": false,
	1403	+ "notNull": true,
	1404	+ "default": 0
	1405	+ },
	1406	+ "price_usd": {
	1407	+ "name": "price_usd",
	1408	+ "type": "double precision",
	1409	+ "primaryKey": false,
	1410	+ "notNull": true,
	1411	+ "default": 0
	1412	+ },
	1413	+ "cached": {
	1414	+ "name": "cached",
	1415	+ "type": "boolean",
	1416	+ "primaryKey": false,
	1417	+ "notNull": true,
	1418	+ "default": false
	1419	+ },
	1420	+ "request_headers": {
	1421	+ "name": "request_headers",
	1422	+ "type": "jsonb",
	1423	+ "primaryKey": false,
	1424	+ "notNull": false
	1425	+ },
	1426	+ "response_headers": {
	1427	+ "name": "response_headers",
	1428	+ "type": "jsonb",
	1429	+ "primaryKey": false,
	1430	+ "notNull": false
	1431	+ },
	1432	+ "timing": {
	1433	+ "name": "timing",
	1434	+ "type": "jsonb",
	1435	+ "primaryKey": false,
	1436	+ "notNull": false
	1437	+ },
	1438	+ "client_ip": {
	1439	+ "name": "client_ip",
	1440	+ "type": "text",
	1441	+ "primaryKey": false,
	1442	+ "notNull": false
	1443	+ },
	1444	+ "user_agent": {
	1445	+ "name": "user_agent",
	1446	+ "type": "text",
	1447	+ "primaryKey": false,
	1448	+ "notNull": false
	1449	+ },
	1450	+ "created_at": {
	1451	+ "name": "created_at",
	1452	+ "type": "timestamp with time zone",
	1453	+ "primaryKey": false,
	1454	+ "notNull": true,
	1455	+ "default": "now()"
	1456	+ },
	1457	+ "completed_at": {
	1458	+ "name": "completed_at",
	1459	+ "type": "timestamp with time zone",
	1460	+ "primaryKey": false,
	1461	+ "notNull": false
	1462	+ }
	1463	+ },
	1464	+ "indexes": {
	1465	+ "fetch_requests_project_created_idx": {
	1466	+ "name": "fetch_requests_project_created_idx",
	1467	+ "columns": [
	1468	+ {
	1469	+ "expression": "project_id",
	1470	+ "isExpression": false,
	1471	+ "asc": true,
	1472	+ "nulls": "last"
	1473	+ },
	1474	+ {
	1475	+ "expression": "created_at",
	1476	+ "isExpression": false,
	1477	+ "asc": true,
	1478	+ "nulls": "last"
	1479	+ }
	1480	+ ],
	1481	+ "isUnique": false,
	1482	+ "concurrently": false,
	1483	+ "method": "btree",
	1484	+ "with": {}
	1485	+ },
	1486	+ "fetch_requests_org_created_idx": {
	1487	+ "name": "fetch_requests_org_created_idx",
	1488	+ "columns": [
	1489	+ {
	1490	+ "expression": "organization_id",
	1491	+ "isExpression": false,
	1492	+ "asc": true,
	1493	+ "nulls": "last"
	1494	+ },
	1495	+ {
	1496	+ "expression": "created_at",
	1497	+ "isExpression": false,
	1498	+ "asc": true,
	1499	+ "nulls": "last"
	1500	+ }
	1501	+ ],
	1502	+ "isUnique": false,
	1503	+ "concurrently": false,
	1504	+ "method": "btree",
	1505	+ "with": {}
	1506	+ },
	1507	+ "fetch_requests_domain_idx": {
	1508	+ "name": "fetch_requests_domain_idx",
	1509	+ "columns": [
	1510	+ {
	1511	+ "expression": "domain",
	1512	+ "isExpression": false,
	1513	+ "asc": true,
	1514	+ "nulls": "last"
	1515	+ }
	1516	+ ],
	1517	+ "isUnique": false,
	1518	+ "concurrently": false,
	1519	+ "method": "btree",
	1520	+ "with": {}
	1521	+ },
	1522	+ "fetch_requests_created_idx": {
	1523	+ "name": "fetch_requests_created_idx",
	1524	+ "columns": [
	1525	+ {
	1526	+ "expression": "created_at",
	1527	+ "isExpression": false,
	1528	+ "asc": true,
	1529	+ "nulls": "last"
	1530	+ }
	1531	+ ],
	1532	+ "isUnique": false,
	1533	+ "concurrently": false,
	1534	+ "method": "btree",
	1535	+ "with": {}
	1536	+ }
	1537	+ },
	1538	+ "foreignKeys": {
	1539	+ "fetch_requests_organization_id_organizations_id_fk": {
	1540	+ "name": "fetch_requests_organization_id_organizations_id_fk",
	1541	+ "tableFrom": "fetch_requests",
	1542	+ "tableTo": "organizations",
	1543	+ "columnsFrom": [
	1544	+ "organization_id"
	1545	+ ],
	1546	+ "columnsTo": [
	1547	+ "id"
	1548	+ ],
	1549	+ "onDelete": "cascade",
	1550	+ "onUpdate": "no action"
	1551	+ },
	1552	+ "fetch_requests_project_id_projects_id_fk": {
	1553	+ "name": "fetch_requests_project_id_projects_id_fk",
	1554	+ "tableFrom": "fetch_requests",
	1555	+ "tableTo": "projects",
	1556	+ "columnsFrom": [
	1557	+ "project_id"
	1558	+ ],
	1559	+ "columnsTo": [
	1560	+ "id"
	1561	+ ],
	1562	+ "onDelete": "cascade",
	1563	+ "onUpdate": "no action"
	1564	+ },
	1565	+ "fetch_requests_api_key_id_api_keys_id_fk": {
	1566	+ "name": "fetch_requests_api_key_id_api_keys_id_fk",
	1567	+ "tableFrom": "fetch_requests",
	1568	+ "tableTo": "api_keys",
	1569	+ "columnsFrom": [
	1570	+ "api_key_id"
	1571	+ ],
	1572	+ "columnsTo": [
	1573	+ "id"
	1574	+ ],
	1575	+ "onDelete": "set null",
	1576	+ "onUpdate": "no action"
	1577	+ }
	1578	+ },
	1579	+ "compositePrimaryKeys": {},
	1580	+ "uniqueConstraints": {},
	1581	+ "policies": {},
	1582	+ "checkConstraints": {},
	1583	+ "isRLSEnabled": false
	1584	+ },
	1585	+ "public.legal_acceptances": {
	1586	+ "name": "legal_acceptances",
	1587	+ "schema": "",
	1588	+ "columns": {
	1589	+ "id": {
	1590	+ "name": "id",
	1591	+ "type": "text",
	1592	+ "primaryKey": true,
	1593	+ "notNull": true
	1594	+ },
	1595	+ "user_id": {
	1596	+ "name": "user_id",
	1597	+ "type": "text",
	1598	+ "primaryKey": false,
	1599	+ "notNull": true
	1600	+ },
	1601	+ "document": {
	1602	+ "name": "document",
	1603	+ "type": "text",
	1604	+ "primaryKey": false,
	1605	+ "notNull": true
	1606	+ },
	1607	+ "version": {
	1608	+ "name": "version",
	1609	+ "type": "text",
	1610	+ "primaryKey": false,
	1611	+ "notNull": true
	1612	+ },
	1613	+ "ip_address": {
	1614	+ "name": "ip_address",
	1615	+ "type": "text",
	1616	+ "primaryKey": false,
	1617	+ "notNull": false
	1618	+ },
	1619	+ "accepted_at": {
	1620	+ "name": "accepted_at",
	1621	+ "type": "timestamp with time zone",
	1622	+ "primaryKey": false,
	1623	+ "notNull": true,
	1624	+ "default": "now()"
	1625	+ }
	1626	+ },
	1627	+ "indexes": {
	1628	+ "legal_acceptances_user_idx": {
	1629	+ "name": "legal_acceptances_user_idx",
	1630	+ "columns": [
	1631	+ {
	1632	+ "expression": "user_id",
	1633	+ "isExpression": false,
	1634	+ "asc": true,
	1635	+ "nulls": "last"
	1636	+ }
	1637	+ ],
	1638	+ "isUnique": false,
	1639	+ "concurrently": false,
	1640	+ "method": "btree",
	1641	+ "with": {}
	1642	+ }
	1643	+ },
	1644	+ "foreignKeys": {
	1645	+ "legal_acceptances_user_id_users_id_fk": {
	1646	+ "name": "legal_acceptances_user_id_users_id_fk",
	1647	+ "tableFrom": "legal_acceptances",
	1648	+ "tableTo": "users",
	1649	+ "columnsFrom": [
	1650	+ "user_id"
	1651	+ ],
	1652	+ "columnsTo": [
	1653	+ "id"
	1654	+ ],
	1655	+ "onDelete": "cascade",
	1656	+ "onUpdate": "no action"
	1657	+ }
	1658	+ },
	1659	+ "compositePrimaryKeys": {},
	1660	+ "uniqueConstraints": {},
	1661	+ "policies": {},
	1662	+ "checkConstraints": {},
	1663	+ "isRLSEnabled": false
	1664	+ },
	1665	+ "public.organization_members": {
	1666	+ "name": "organization_members",
	1667	+ "schema": "",
	1668	+ "columns": {
	1669	+ "organization_id": {
	1670	+ "name": "organization_id",
	1671	+ "type": "text",
	1672	+ "primaryKey": false,
	1673	+ "notNull": true
	1674	+ },
	1675	+ "user_id": {
	1676	+ "name": "user_id",
	1677	+ "type": "text",
	1678	+ "primaryKey": false,
	1679	+ "notNull": true
	1680	+ },
	1681	+ "role": {
	1682	+ "name": "role",
	1683	+ "type": "text",
	1684	+ "primaryKey": false,
	1685	+ "notNull": true,
	1686	+ "default": "'owner'"
	1687	+ },
	1688	+ "created_at": {
	1689	+ "name": "created_at",
	1690	+ "type": "timestamp with time zone",
	1691	+ "primaryKey": false,
	1692	+ "notNull": true,
	1693	+ "default": "now()"
	1694	+ }
	1695	+ },
	1696	+ "indexes": {
	1697	+ "org_members_user_idx": {
	1698	+ "name": "org_members_user_idx",
	1699	+ "columns": [
	1700	+ {
	1701	+ "expression": "user_id",
	1702	+ "isExpression": false,
	1703	+ "asc": true,
	1704	+ "nulls": "last"
	1705	+ }
	1706	+ ],
	1707	+ "isUnique": false,
	1708	+ "concurrently": false,
	1709	+ "method": "btree",
	1710	+ "with": {}
	1711	+ }
	1712	+ },
	1713	+ "foreignKeys": {
	1714	+ "organization_members_organization_id_organizations_id_fk": {
	1715	+ "name": "organization_members_organization_id_organizations_id_fk",
	1716	+ "tableFrom": "organization_members",
	1717	+ "tableTo": "organizations",
	1718	+ "columnsFrom": [
	1719	+ "organization_id"
	1720	+ ],
	1721	+ "columnsTo": [
	1722	+ "id"
	1723	+ ],
	1724	+ "onDelete": "cascade",
	1725	+ "onUpdate": "no action"
	1726	+ },
	1727	+ "organization_members_user_id_users_id_fk": {
	1728	+ "name": "organization_members_user_id_users_id_fk",
	1729	+ "tableFrom": "organization_members",
	1730	+ "tableTo": "users",
	1731	+ "columnsFrom": [
	1732	+ "user_id"
	1733	+ ],
	1734	+ "columnsTo": [
	1735	+ "id"
	1736	+ ],
	1737	+ "onDelete": "cascade",
	1738	+ "onUpdate": "no action"
	1739	+ }
	1740	+ },
	1741	+ "compositePrimaryKeys": {
	1742	+ "organization_members_organization_id_user_id_pk": {
	1743	+ "name": "organization_members_organization_id_user_id_pk",
	1744	+ "columns": [
	1745	+ "organization_id",
	1746	+ "user_id"
	1747	+ ]
	1748	+ }
	1749	+ },
	1750	+ "uniqueConstraints": {},
	1751	+ "policies": {},
	1752	+ "checkConstraints": {},
	1753	+ "isRLSEnabled": false
	1754	+ },
	1755	+ "public.organizations": {
	1756	+ "name": "organizations",
	1757	+ "schema": "",
	1758	+ "columns": {
	1759	+ "id": {
	1760	+ "name": "id",
	1761	+ "type": "text",
	1762	+ "primaryKey": true,
	1763	+ "notNull": true
	1764	+ },
	1765	+ "name": {
	1766	+ "name": "name",
	1767	+ "type": "text",
	1768	+ "primaryKey": false,
	1769	+ "notNull": true
	1770	+ },
	1771	+ "slug": {
	1772	+ "name": "slug",
	1773	+ "type": "text",
	1774	+ "primaryKey": false,
	1775	+ "notNull": true
	1776	+ },
	1777	+ "plan": {
	1778	+ "name": "plan",
	1779	+ "type": "text",
	1780	+ "primaryKey": false,
	1781	+ "notNull": true,
	1782	+ "default": "'unlimited'"
	1783	+ },
	1784	+ "owner_user_id": {
	1785	+ "name": "owner_user_id",
	1786	+ "type": "text",
	1787	+ "primaryKey": false,
	1788	+ "notNull": true
	1789	+ },
	1790	+ "provider_visibility": {
	1791	+ "name": "provider_visibility",
	1792	+ "type": "boolean",
	1793	+ "primaryKey": false,
	1794	+ "notNull": true,
	1795	+ "default": false
	1796	+ },
	1797	+ "soft_limit_usd": {
	1798	+ "name": "soft_limit_usd",
	1799	+ "type": "double precision",
	1800	+ "primaryKey": false,
	1801	+ "notNull": false
	1802	+ },
	1803	+ "hard_limit_usd": {
	1804	+ "name": "hard_limit_usd",
	1805	+ "type": "double precision",
	1806	+ "primaryKey": false,
	1807	+ "notNull": false
	1808	+ },
	1809	+ "stripe_customer_id": {
	1810	+ "name": "stripe_customer_id",
	1811	+ "type": "text",
	1812	+ "primaryKey": false,
	1813	+ "notNull": false
	1814	+ },
	1815	+ "suspended": {
	1816	+ "name": "suspended",
	1817	+ "type": "boolean",
	1818	+ "primaryKey": false,
	1819	+ "notNull": true,
	1820	+ "default": false
	1821	+ },
	1822	+ "created_at": {
	1823	+ "name": "created_at",
	1824	+ "type": "timestamp with time zone",
	1825	+ "primaryKey": false,
	1826	+ "notNull": true,
	1827	+ "default": "now()"
	1828	+ },
	1829	+ "updated_at": {
	1830	+ "name": "updated_at",
	1831	+ "type": "timestamp with time zone",
	1832	+ "primaryKey": false,
	1833	+ "notNull": true,
	1834	+ "default": "now()"
	1835	+ }
	1836	+ },
	1837	+ "indexes": {
	1838	+ "organizations_slug_uq": {
	1839	+ "name": "organizations_slug_uq",
	1840	+ "columns": [
	1841	+ {
	1842	+ "expression": "slug",
	1843	+ "isExpression": false,
	1844	+ "asc": true,
	1845	+ "nulls": "last"
	1846	+ }
	1847	+ ],
	1848	+ "isUnique": true,
	1849	+ "concurrently": false,
	1850	+ "method": "btree",
	1851	+ "with": {}
	1852	+ },
	1853	+ "organizations_owner_idx": {
	1854	+ "name": "organizations_owner_idx",
	1855	+ "columns": [
	1856	+ {
	1857	+ "expression": "owner_user_id",
	1858	+ "isExpression": false,
	1859	+ "asc": true,
	1860	+ "nulls": "last"
	1861	+ }
	1862	+ ],
	1863	+ "isUnique": false,
	1864	+ "concurrently": false,
	1865	+ "method": "btree",
	1866	+ "with": {}
	1867	+ }
	1868	+ },
	1869	+ "foreignKeys": {
	1870	+ "organizations_owner_user_id_users_id_fk": {
	1871	+ "name": "organizations_owner_user_id_users_id_fk",
	1872	+ "tableFrom": "organizations",
	1873	+ "tableTo": "users",
	1874	+ "columnsFrom": [
	1875	+ "owner_user_id"
	1876	+ ],
	1877	+ "columnsTo": [
	1878	+ "id"
	1879	+ ],
	1880	+ "onDelete": "restrict",
	1881	+ "onUpdate": "no action"
	1882	+ }
	1883	+ },
	1884	+ "compositePrimaryKeys": {},
	1885	+ "uniqueConstraints": {},
	1886	+ "policies": {},
	1887	+ "checkConstraints": {},
	1888	+ "isRLSEnabled": false
	1889	+ },
	1890	+ "public.projects": {
	1891	+ "name": "projects",
	1892	+ "schema": "",
	1893	+ "columns": {
	1894	+ "id": {
	1895	+ "name": "id",
	1896	+ "type": "text",
	1897	+ "primaryKey": true,
	1898	+ "notNull": true
	1899	+ },
	1900	+ "organization_id": {
	1901	+ "name": "organization_id",
	1902	+ "type": "text",
	1903	+ "primaryKey": false,
	1904	+ "notNull": true
	1905	+ },
	1906	+ "name": {
	1907	+ "name": "name",
	1908	+ "type": "text",
	1909	+ "primaryKey": false,
	1910	+ "notNull": true
	1911	+ },
	1912	+ "slug": {
	1913	+ "name": "slug",
	1914	+ "type": "text",
	1915	+ "primaryKey": false,
	1916	+ "notNull": true
	1917	+ },
	1918	+ "description": {
	1919	+ "name": "description",
	1920	+ "type": "text",
	1921	+ "primaryKey": false,
	1922	+ "notNull": false
	1923	+ },
	1924	+ "environment": {
	1925	+ "name": "environment",
	1926	+ "type": "text",
	1927	+ "primaryKey": false,
	1928	+ "notNull": true,
	1929	+ "default": "'production'"
	1930	+ },
	1931	+ "default_country": {
	1932	+ "name": "default_country",
	1933	+ "type": "text",
	1934	+ "primaryKey": false,
	1935	+ "notNull": false
	1936	+ },
	1937	+ "default_network": {
	1938	+ "name": "default_network",
	1939	+ "type": "text",
	1940	+ "primaryKey": false,
	1941	+ "notNull": true,
	1942	+ "default": "'auto'"
	1943	+ },
	1944	+ "log_level": {
	1945	+ "name": "log_level",
	1946	+ "type": "text",
	1947	+ "primaryKey": false,
	1948	+ "notNull": true,
	1949	+ "default": "'metadata'"
	1950	+ },
	1951	+ "soft_limit_usd": {
	1952	+ "name": "soft_limit_usd",
	1953	+ "type": "double precision",
	1954	+ "primaryKey": false,
	1955	+ "notNull": false
	1956	+ },
	1957	+ "hard_limit_usd": {
	1958	+ "name": "hard_limit_usd",
	1959	+ "type": "double precision",
	1960	+ "primaryKey": false,
	1961	+ "notNull": false
	1962	+ },
	1963	+ "monthly_request_limit": {
	1964	+ "name": "monthly_request_limit",
	1965	+ "type": "integer",
	1966	+ "primaryKey": false,
	1967	+ "notNull": false
	1968	+ },
	1969	+ "archived_at": {
	1970	+ "name": "archived_at",
	1971	+ "type": "timestamp with time zone",
	1972	+ "primaryKey": false,
	1973	+ "notNull": false
	1974	+ },
	1975	+ "created_at": {
	1976	+ "name": "created_at",
	1977	+ "type": "timestamp with time zone",
	1978	+ "primaryKey": false,
	1979	+ "notNull": true,
	1980	+ "default": "now()"
	1981	+ },
	1982	+ "updated_at": {
	1983	+ "name": "updated_at",
	1984	+ "type": "timestamp with time zone",
	1985	+ "primaryKey": false,
	1986	+ "notNull": true,
	1987	+ "default": "now()"
	1988	+ }
	1989	+ },
	1990	+ "indexes": {
	1991	+ "projects_org_slug_uq": {
	1992	+ "name": "projects_org_slug_uq",
	1993	+ "columns": [
	1994	+ {
	1995	+ "expression": "organization_id",
	1996	+ "isExpression": false,
	1997	+ "asc": true,
	1998	+ "nulls": "last"
	1999	+ },
	2000	+ {
	2001	+ "expression": "slug",
	2002	+ "isExpression": false,
	2003	+ "asc": true,
	2004	+ "nulls": "last"
	2005	+ }
	2006	+ ],
	2007	+ "isUnique": true,
	2008	+ "concurrently": false,
	2009	+ "method": "btree",
	2010	+ "with": {}
	2011	+ },
	2012	+ "projects_org_idx": {
	2013	+ "name": "projects_org_idx",
	2014	+ "columns": [
	2015	+ {
	2016	+ "expression": "organization_id",
	2017	+ "isExpression": false,
	2018	+ "asc": true,
	2019	+ "nulls": "last"
	2020	+ }
	2021	+ ],
	2022	+ "isUnique": false,
	2023	+ "concurrently": false,
	2024	+ "method": "btree",
	2025	+ "with": {}
	2026	+ }
	2027	+ },
	2028	+ "foreignKeys": {
	2029	+ "projects_organization_id_organizations_id_fk": {
	2030	+ "name": "projects_organization_id_organizations_id_fk",
	2031	+ "tableFrom": "projects",
	2032	+ "tableTo": "organizations",
	2033	+ "columnsFrom": [
	2034	+ "organization_id"
	2035	+ ],
	2036	+ "columnsTo": [
	2037	+ "id"
	2038	+ ],
	2039	+ "onDelete": "cascade",
	2040	+ "onUpdate": "no action"
	2041	+ }
	2042	+ },
	2043	+ "compositePrimaryKeys": {},
	2044	+ "uniqueConstraints": {},
	2045	+ "policies": {},
	2046	+ "checkConstraints": {},
	2047	+ "isRLSEnabled": false
	2048	+ },
	2049	+ "public.provider_configs": {
	2050	+ "name": "provider_configs",
	2051	+ "schema": "",
	2052	+ "columns": {
	2053	+ "id": {
	2054	+ "name": "id",
	2055	+ "type": "text",
	2056	+ "primaryKey": true,
	2057	+ "notNull": true
	2058	+ },
	2059	+ "label": {
	2060	+ "name": "label",
	2061	+ "type": "text",
	2062	+ "primaryKey": false,
	2063	+ "notNull": true
	2064	+ },
	2065	+ "enabled": {
	2066	+ "name": "enabled",
	2067	+ "type": "boolean",
	2068	+ "primaryKey": false,
	2069	+ "notNull": true,
	2070	+ "default": true
	2071	+ },
	2072	+ "networks": {
	2073	+ "name": "networks",
	2074	+ "type": "jsonb",
	2075	+ "primaryKey": false,
	2076	+ "notNull": true,
	2077	+ "default": "'[\"residential\"]'::jsonb"
	2078	+ },
	2079	+ "price_per_gb_usd": {
	2080	+ "name": "price_per_gb_usd",
	2081	+ "type": "jsonb",
	2082	+ "primaryKey": false,
	2083	+ "notNull": true,
	2084	+ "default": "'{}'::jsonb"
	2085	+ },
	2086	+ "weight": {
	2087	+ "name": "weight",
	2088	+ "type": "double precision",
	2089	+ "primaryKey": false,
	2090	+ "notNull": true,
	2091	+ "default": 1
	2092	+ },
	2093	+ "max_concurrency": {
	2094	+ "name": "max_concurrency",
	2095	+ "type": "integer",
	2096	+ "primaryKey": false,
	2097	+ "notNull": true,
	2098	+ "default": 200
	2099	+ },
	2100	+ "notes": {
	2101	+ "name": "notes",
	2102	+ "type": "text",
	2103	+ "primaryKey": false,
	2104	+ "notNull": false
	2105	+ },
	2106	+ "updated_at": {
	2107	+ "name": "updated_at",
	2108	+ "type": "timestamp with time zone",
	2109	+ "primaryKey": false,
	2110	+ "notNull": true,
	2111	+ "default": "now()"
	2112	+ }
	2113	+ },
	2114	+ "indexes": {},
	2115	+ "foreignKeys": {},
	2116	+ "compositePrimaryKeys": {},
	2117	+ "uniqueConstraints": {},
	2118	+ "policies": {},
	2119	+ "checkConstraints": {},
	2120	+ "isRLSEnabled": false
	2121	+ },
	2122	+ "public.provider_health": {
	2123	+ "name": "provider_health",
	2124	+ "schema": "",
	2125	+ "columns": {
	2126	+ "id": {
	2127	+ "name": "id",
	2128	+ "type": "text",
	2129	+ "primaryKey": true,
	2130	+ "notNull": true
	2131	+ },
	2132	+ "provider": {
	2133	+ "name": "provider",
	2134	+ "type": "text",
	2135	+ "primaryKey": false,
	2136	+ "notNull": true
	2137	+ },
	2138	+ "network": {
	2139	+ "name": "network",
	2140	+ "type": "text",
	2141	+ "primaryKey": false,
	2142	+ "notNull": true
	2143	+ },
	2144	+ "status": {
	2145	+ "name": "status",
	2146	+ "type": "text",
	2147	+ "primaryKey": false,
	2148	+ "notNull": true
	2149	+ },
	2150	+ "latency_ms": {
	2151	+ "name": "latency_ms",
	2152	+ "type": "integer",
	2153	+ "primaryKey": false,
	2154	+ "notNull": false
	2155	+ },
	2156	+ "success_rate": {
	2157	+ "name": "success_rate",
	2158	+ "type": "double precision",
	2159	+ "primaryKey": false,
	2160	+ "notNull": false
	2161	+ },
	2162	+ "detail": {
	2163	+ "name": "detail",
	2164	+ "type": "text",
	2165	+ "primaryKey": false,
	2166	+ "notNull": false
	2167	+ },
	2168	+ "checked_at": {
	2169	+ "name": "checked_at",
	2170	+ "type": "timestamp with time zone",
	2171	+ "primaryKey": false,
	2172	+ "notNull": true,
	2173	+ "default": "now()"
	2174	+ }
	2175	+ },
	2176	+ "indexes": {
	2177	+ "provider_health_provider_checked_idx": {
	2178	+ "name": "provider_health_provider_checked_idx",
	2179	+ "columns": [
	2180	+ {
	2181	+ "expression": "provider",
	2182	+ "isExpression": false,
	2183	+ "asc": true,
	2184	+ "nulls": "last"
	2185	+ },
	2186	+ {
	2187	+ "expression": "checked_at",
	2188	+ "isExpression": false,
	2189	+ "asc": true,
	2190	+ "nulls": "last"
	2191	+ }
	2192	+ ],
	2193	+ "isUnique": false,
	2194	+ "concurrently": false,
	2195	+ "method": "btree",
	2196	+ "with": {}
	2197	+ }
	2198	+ },
	2199	+ "foreignKeys": {},
	2200	+ "compositePrimaryKeys": {},
	2201	+ "uniqueConstraints": {},
	2202	+ "policies": {},
	2203	+ "checkConstraints": {},
	2204	+ "isRLSEnabled": false
	2205	+ },
	2206	+ "public.proxy_sessions": {
	2207	+ "name": "proxy_sessions",
	2208	+ "schema": "",
	2209	+ "columns": {
	2210	+ "id": {
	2211	+ "name": "id",
	2212	+ "type": "text",
	2213	+ "primaryKey": true,
	2214	+ "notNull": true
	2215	+ },
	2216	+ "organization_id": {
	2217	+ "name": "organization_id",
	2218	+ "type": "text",
	2219	+ "primaryKey": false,
	2220	+ "notNull": true
	2221	+ },
	2222	+ "project_id": {
	2223	+ "name": "project_id",
	2224	+ "type": "text",
	2225	+ "primaryKey": false,
	2226	+ "notNull": true
	2227	+ },
	2228	+ "label": {
	2229	+ "name": "label",
	2230	+ "type": "text",
	2231	+ "primaryKey": false,
	2232	+ "notNull": false
	2233	+ },
	2234	+ "provider": {
	2235	+ "name": "provider",
	2236	+ "type": "text",
	2237	+ "primaryKey": false,
	2238	+ "notNull": true
	2239	+ },
	2240	+ "network": {
	2241	+ "name": "network",
	2242	+ "type": "text",
	2243	+ "primaryKey": false,
	2244	+ "notNull": true
	2245	+ },
	2246	+ "country": {
	2247	+ "name": "country",
	2248	+ "type": "text",
	2249	+ "primaryKey": false,
	2250	+ "notNull": false
	2251	+ },
	2252	+ "region": {
	2253	+ "name": "region",
	2254	+ "type": "text",
	2255	+ "primaryKey": false,
	2256	+ "notNull": false
	2257	+ },
	2258	+ "city": {
	2259	+ "name": "city",
	2260	+ "type": "text",
	2261	+ "primaryKey": false,
	2262	+ "notNull": false
	2263	+ },
	2264	+ "sticky_key": {
	2265	+ "name": "sticky_key",
	2266	+ "type": "text",
	2267	+ "primaryKey": false,
	2268	+ "notNull": true
	2269	+ },
	2270	+ "cookies": {
	2271	+ "name": "cookies",
	2272	+ "type": "jsonb",
	2273	+ "primaryKey": false,
	2274	+ "notNull": true,
	2275	+ "default": "'[]'::jsonb"
	2276	+ },
	2277	+ "status": {
	2278	+ "name": "status",
	2279	+ "type": "text",
	2280	+ "primaryKey": false,
	2281	+ "notNull": true,
	2282	+ "default": "'active'"
	2283	+ },
	2284	+ "request_count": {
	2285	+ "name": "request_count",
	2286	+ "type": "integer",
	2287	+ "primaryKey": false,
	2288	+ "notNull": true,
	2289	+ "default": 0
	2290	+ },
	2291	+ "last_used_at": {
	2292	+ "name": "last_used_at",
	2293	+ "type": "timestamp with time zone",
	2294	+ "primaryKey": false,
	2295	+ "notNull": false
	2296	+ },
	2297	+ "expires_at": {
	2298	+ "name": "expires_at",
	2299	+ "type": "timestamp with time zone",
	2300	+ "primaryKey": false,
	2301	+ "notNull": true
	2302	+ },
	2303	+ "created_at": {
	2304	+ "name": "created_at",
	2305	+ "type": "timestamp with time zone",
	2306	+ "primaryKey": false,
	2307	+ "notNull": true,
	2308	+ "default": "now()"
	2309	+ }
	2310	+ },
	2311	+ "indexes": {
	2312	+ "proxy_sessions_project_idx": {
	2313	+ "name": "proxy_sessions_project_idx",
	2314	+ "columns": [
	2315	+ {
	2316	+ "expression": "project_id",
	2317	+ "isExpression": false,
	2318	+ "asc": true,
	2319	+ "nulls": "last"
	2320	+ }
	2321	+ ],
	2322	+ "isUnique": false,
	2323	+ "concurrently": false,
	2324	+ "method": "btree",
	2325	+ "with": {}
	2326	+ },
	2327	+ "proxy_sessions_expires_idx": {
	2328	+ "name": "proxy_sessions_expires_idx",
	2329	+ "columns": [
	2330	+ {
	2331	+ "expression": "expires_at",
	2332	+ "isExpression": false,
	2333	+ "asc": true,
	2334	+ "nulls": "last"
	2335	+ }
	2336	+ ],
	2337	+ "isUnique": false,
	2338	+ "concurrently": false,
	2339	+ "method": "btree",
	2340	+ "with": {}
	2341	+ }
	2342	+ },
	2343	+ "foreignKeys": {
	2344	+ "proxy_sessions_organization_id_organizations_id_fk": {
	2345	+ "name": "proxy_sessions_organization_id_organizations_id_fk",
	2346	+ "tableFrom": "proxy_sessions",
	2347	+ "tableTo": "organizations",
	2348	+ "columnsFrom": [
	2349	+ "organization_id"
	2350	+ ],
	2351	+ "columnsTo": [
	2352	+ "id"
	2353	+ ],
	2354	+ "onDelete": "cascade",
	2355	+ "onUpdate": "no action"
	2356	+ },
	2357	+ "proxy_sessions_project_id_projects_id_fk": {
	2358	+ "name": "proxy_sessions_project_id_projects_id_fk",
	2359	+ "tableFrom": "proxy_sessions",
	2360	+ "tableTo": "projects",
	2361	+ "columnsFrom": [
	2362	+ "project_id"
	2363	+ ],
	2364	+ "columnsTo": [
	2365	+ "id"
	2366	+ ],
	2367	+ "onDelete": "cascade",
	2368	+ "onUpdate": "no action"
	2369	+ }
	2370	+ },
	2371	+ "compositePrimaryKeys": {},
	2372	+ "uniqueConstraints": {},
	2373	+ "policies": {},
	2374	+ "checkConstraints": {},
	2375	+ "isRLSEnabled": false
	2376	+ },
	2377	+ "public.request_attempts": {
	2378	+ "name": "request_attempts",
	2379	+ "schema": "",
	2380	+ "columns": {
	2381	+ "id": {
	2382	+ "name": "id",
	2383	+ "type": "text",
	2384	+ "primaryKey": true,
	2385	+ "notNull": true
	2386	+ },
	2387	+ "request_id": {
	2388	+ "name": "request_id",
	2389	+ "type": "text",
	2390	+ "primaryKey": false,
	2391	+ "notNull": true
	2392	+ },
	2393	+ "attempt_no": {
	2394	+ "name": "attempt_no",
	2395	+ "type": "integer",
	2396	+ "primaryKey": false,
	2397	+ "notNull": true
	2398	+ },
	2399	+ "provider": {
	2400	+ "name": "provider",
	2401	+ "type": "text",
	2402	+ "primaryKey": false,
	2403	+ "notNull": true
	2404	+ },
	2405	+ "network": {
	2406	+ "name": "network",
	2407	+ "type": "text",
	2408	+ "primaryKey": false,
	2409	+ "notNull": true
	2410	+ },
	2411	+ "mode": {
	2412	+ "name": "mode",
	2413	+ "type": "text",
	2414	+ "primaryKey": false,
	2415	+ "notNull": true,
	2416	+ "default": "'http'"
	2417	+ },
	2418	+ "country": {
	2419	+ "name": "country",
	2420	+ "type": "text",
	2421	+ "primaryKey": false,
	2422	+ "notNull": false
	2423	+ },
	2424	+ "session_key": {
	2425	+ "name": "session_key",
	2426	+ "type": "text",
	2427	+ "primaryKey": false,
	2428	+ "notNull": false
	2429	+ },
	2430	+ "outcome": {
	2431	+ "name": "outcome",
	2432	+ "type": "text",
	2433	+ "primaryKey": false,
	2434	+ "notNull": true
	2435	+ },
	2436	+ "http_status": {
	2437	+ "name": "http_status",
	2438	+ "type": "integer",
	2439	+ "primaryKey": false,
	2440	+ "notNull": false
	2441	+ },
	2442	+ "error_code": {
	2443	+ "name": "error_code",
	2444	+ "type": "text",
	2445	+ "primaryKey": false,
	2446	+ "notNull": false
	2447	+ },
	2448	+ "error_detail": {
	2449	+ "name": "error_detail",
	2450	+ "type": "text",
	2451	+ "primaryKey": false,
	2452	+ "notNull": false
	2453	+ },
	2454	+ "block_reason": {
	2455	+ "name": "block_reason",
	2456	+ "type": "text",
	2457	+ "primaryKey": false,
	2458	+ "notNull": false
	2459	+ },
	2460	+ "duration_ms": {
	2461	+ "name": "duration_ms",
	2462	+ "type": "integer",
	2463	+ "primaryKey": false,
	2464	+ "notNull": true,
	2465	+ "default": 0
	2466	+ },
	2467	+ "bytes_in": {
	2468	+ "name": "bytes_in",
	2469	+ "type": "bigint",
	2470	+ "primaryKey": false,
	2471	+ "notNull": true,
	2472	+ "default": 0
	2473	+ },
	2474	+ "bytes_out": {
	2475	+ "name": "bytes_out",
	2476	+ "type": "bigint",
	2477	+ "primaryKey": false,
	2478	+ "notNull": true,
	2479	+ "default": 0
	2480	+ },
	2481	+ "unit_price_per_gb": {
	2482	+ "name": "unit_price_per_gb",
	2483	+ "type": "double precision",
	2484	+ "primaryKey": false,
	2485	+ "notNull": true,
	2486	+ "default": 0
	2487	+ },
	2488	+ "cost_usd": {
	2489	+ "name": "cost_usd",
	2490	+ "type": "double precision",
	2491	+ "primaryKey": false,
	2492	+ "notNull": true,
	2493	+ "default": 0
	2494	+ },
	2495	+ "routing_score": {
	2496	+ "name": "routing_score",
	2497	+ "type": "double precision",
	2498	+ "primaryKey": false,
	2499	+ "notNull": false
	2500	+ },
	2501	+ "timing": {
	2502	+ "name": "timing",
	2503	+ "type": "jsonb",
	2504	+ "primaryKey": false,
	2505	+ "notNull": false
	2506	+ },
	2507	+ "created_at": {
	2508	+ "name": "created_at",
	2509	+ "type": "timestamp with time zone",
	2510	+ "primaryKey": false,
	2511	+ "notNull": true,
	2512	+ "default": "now()"
	2513	+ }
	2514	+ },
	2515	+ "indexes": {
	2516	+ "request_attempts_request_idx": {
	2517	+ "name": "request_attempts_request_idx",
	2518	+ "columns": [
	2519	+ {
	2520	+ "expression": "request_id",
	2521	+ "isExpression": false,
	2522	+ "asc": true,
	2523	+ "nulls": "last"
	2524	+ }
	2525	+ ],
	2526	+ "isUnique": false,
	2527	+ "concurrently": false,
	2528	+ "method": "btree",
	2529	+ "with": {}
	2530	+ },
	2531	+ "request_attempts_provider_created_idx": {
	2532	+ "name": "request_attempts_provider_created_idx",
	2533	+ "columns": [
	2534	+ {
	2535	+ "expression": "provider",
	2536	+ "isExpression": false,
	2537	+ "asc": true,
	2538	+ "nulls": "last"
	2539	+ },
	2540	+ {
	2541	+ "expression": "created_at",
	2542	+ "isExpression": false,
	2543	+ "asc": true,
	2544	+ "nulls": "last"
	2545	+ }
	2546	+ ],
	2547	+ "isUnique": false,
	2548	+ "concurrently": false,
	2549	+ "method": "btree",
	2550	+ "with": {}
	2551	+ }
	2552	+ },
	2553	+ "foreignKeys": {
	2554	+ "request_attempts_request_id_fetch_requests_id_fk": {
	2555	+ "name": "request_attempts_request_id_fetch_requests_id_fk",
	2556	+ "tableFrom": "request_attempts",
	2557	+ "tableTo": "fetch_requests",
	2558	+ "columnsFrom": [
	2559	+ "request_id"
	2560	+ ],
	2561	+ "columnsTo": [
	2562	+ "id"
	2563	+ ],
	2564	+ "onDelete": "cascade",
	2565	+ "onUpdate": "no action"
	2566	+ }
	2567	+ },
	2568	+ "compositePrimaryKeys": {},
	2569	+ "uniqueConstraints": {},
	2570	+ "policies": {},
	2571	+ "checkConstraints": {},
	2572	+ "isRLSEnabled": false
	2573	+ },
	2574	+ "public.routing_metrics": {
	2575	+ "name": "routing_metrics",
	2576	+ "schema": "",
	2577	+ "columns": {
	2578	+ "id": {
	2579	+ "name": "id",
	2580	+ "type": "text",
	2581	+ "primaryKey": true,
	2582	+ "notNull": true
	2583	+ },
	2584	+ "bucket": {
	2585	+ "name": "bucket",
	2586	+ "type": "timestamp with time zone",
	2587	+ "primaryKey": false,
	2588	+ "notNull": true
	2589	+ },
	2590	+ "provider": {
	2591	+ "name": "provider",
	2592	+ "type": "text",
	2593	+ "primaryKey": false,
	2594	+ "notNull": true
	2595	+ },
	2596	+ "network": {
	2597	+ "name": "network",
	2598	+ "type": "text",
	2599	+ "primaryKey": false,
	2600	+ "notNull": true
	2601	+ },
	2602	+ "country": {
	2603	+ "name": "country",
	2604	+ "type": "text",
	2605	+ "primaryKey": false,
	2606	+ "notNull": false
	2607	+ },
	2608	+ "requests": {
	2609	+ "name": "requests",
	2610	+ "type": "integer",
	2611	+ "primaryKey": false,
	2612	+ "notNull": true,
	2613	+ "default": 0
	2614	+ },
	2615	+ "successes": {
	2616	+ "name": "successes",
	2617	+ "type": "integer",
	2618	+ "primaryKey": false,
	2619	+ "notNull": true,
	2620	+ "default": 0
	2621	+ },
	2622	+ "blocked": {
	2623	+ "name": "blocked",
	2624	+ "type": "integer",
	2625	+ "primaryKey": false,
	2626	+ "notNull": true,
	2627	+ "default": 0
	2628	+ },
	2629	+ "errors": {
	2630	+ "name": "errors",
	2631	+ "type": "integer",
	2632	+ "primaryKey": false,
	2633	+ "notNull": true,
	2634	+ "default": 0
	2635	+ },
	2636	+ "latency_sum_ms": {
	2637	+ "name": "latency_sum_ms",
	2638	+ "type": "bigint",
	2639	+ "primaryKey": false,
	2640	+ "notNull": true,
	2641	+ "default": 0
	2642	+ },
	2643	+ "bytes": {
	2644	+ "name": "bytes",
	2645	+ "type": "bigint",
	2646	+ "primaryKey": false,
	2647	+ "notNull": true,
	2648	+ "default": 0
	2649	+ },
	2650	+ "cost_usd": {
	2651	+ "name": "cost_usd",
	2652	+ "type": "double precision",
	2653	+ "primaryKey": false,
	2654	+ "notNull": true,
	2655	+ "default": 0
	2656	+ }
	2657	+ },
	2658	+ "indexes": {
	2659	+ "routing_metrics_bucket_uq": {
	2660	+ "name": "routing_metrics_bucket_uq",
	2661	+ "columns": [
	2662	+ {
	2663	+ "expression": "bucket",
	2664	+ "isExpression": false,
	2665	+ "asc": true,
	2666	+ "nulls": "last"
	2667	+ },
	2668	+ {
	2669	+ "expression": "provider",
	2670	+ "isExpression": false,
	2671	+ "asc": true,
	2672	+ "nulls": "last"
	2673	+ },
	2674	+ {
	2675	+ "expression": "network",
	2676	+ "isExpression": false,
	2677	+ "asc": true,
	2678	+ "nulls": "last"
	2679	+ },
	2680	+ {
	2681	+ "expression": "country",
	2682	+ "isExpression": false,
	2683	+ "asc": true,
	2684	+ "nulls": "last"
	2685	+ }
	2686	+ ],
	2687	+ "isUnique": true,
	2688	+ "concurrently": false,
	2689	+ "method": "btree",
	2690	+ "with": {}
	2691	+ }
	2692	+ },
	2693	+ "foreignKeys": {},
	2694	+ "compositePrimaryKeys": {},
	2695	+ "uniqueConstraints": {},
	2696	+ "policies": {},
	2697	+ "checkConstraints": {},
	2698	+ "isRLSEnabled": false
	2699	+ },
	2700	+ "public.sessions": {
	2701	+ "name": "sessions",
	2702	+ "schema": "",
	2703	+ "columns": {
	2704	+ "id": {
	2705	+ "name": "id",
	2706	+ "type": "text",
	2707	+ "primaryKey": true,
	2708	+ "notNull": true
	2709	+ },
	2710	+ "expires_at": {
	2711	+ "name": "expires_at",
	2712	+ "type": "timestamp with time zone",
	2713	+ "primaryKey": false,
	2714	+ "notNull": true
	2715	+ },
	2716	+ "token": {
	2717	+ "name": "token",
	2718	+ "type": "text",
	2719	+ "primaryKey": false,
	2720	+ "notNull": true
	2721	+ },
	2722	+ "created_at": {
	2723	+ "name": "created_at",
	2724	+ "type": "timestamp with time zone",
	2725	+ "primaryKey": false,
	2726	+ "notNull": true,
	2727	+ "default": "now()"
	2728	+ },
	2729	+ "updated_at": {
	2730	+ "name": "updated_at",
	2731	+ "type": "timestamp with time zone",
	2732	+ "primaryKey": false,
	2733	+ "notNull": true,
	2734	+ "default": "now()"
	2735	+ },
	2736	+ "ip_address": {
	2737	+ "name": "ip_address",
	2738	+ "type": "text",
	2739	+ "primaryKey": false,
	2740	+ "notNull": false
	2741	+ },
	2742	+ "user_agent": {
	2743	+ "name": "user_agent",
	2744	+ "type": "text",
	2745	+ "primaryKey": false,
	2746	+ "notNull": false
	2747	+ },
	2748	+ "user_id": {
	2749	+ "name": "user_id",
	2750	+ "type": "text",
	2751	+ "primaryKey": false,
	2752	+ "notNull": true
	2753	+ }
	2754	+ },
	2755	+ "indexes": {
	2756	+ "sessions_token_uq": {
	2757	+ "name": "sessions_token_uq",
	2758	+ "columns": [
	2759	+ {
	2760	+ "expression": "token",
	2761	+ "isExpression": false,
	2762	+ "asc": true,
	2763	+ "nulls": "last"
	2764	+ }
	2765	+ ],
	2766	+ "isUnique": true,
	2767	+ "concurrently": false,
	2768	+ "method": "btree",
	2769	+ "with": {}
	2770	+ },
	2771	+ "sessions_user_idx": {
	2772	+ "name": "sessions_user_idx",
	2773	+ "columns": [
	2774	+ {
	2775	+ "expression": "user_id",
	2776	+ "isExpression": false,
	2777	+ "asc": true,
	2778	+ "nulls": "last"
	2779	+ }
	2780	+ ],
	2781	+ "isUnique": false,
	2782	+ "concurrently": false,
	2783	+ "method": "btree",
	2784	+ "with": {}
	2785	+ }
	2786	+ },
	2787	+ "foreignKeys": {
	2788	+ "sessions_user_id_users_id_fk": {
	2789	+ "name": "sessions_user_id_users_id_fk",
	2790	+ "tableFrom": "sessions",
	2791	+ "tableTo": "users",
	2792	+ "columnsFrom": [
	2793	+ "user_id"
	2794	+ ],
	2795	+ "columnsTo": [
	2796	+ "id"
	2797	+ ],
	2798	+ "onDelete": "cascade",
	2799	+ "onUpdate": "no action"
	2800	+ }
	2801	+ },
	2802	+ "compositePrimaryKeys": {},
	2803	+ "uniqueConstraints": {},
	2804	+ "policies": {},
	2805	+ "checkConstraints": {},
	2806	+ "isRLSEnabled": false
	2807	+ },
	2808	+ "public.signup_allowlist": {
	2809	+ "name": "signup_allowlist",
	2810	+ "schema": "",
	2811	+ "columns": {
	2812	+ "email": {
	2813	+ "name": "email",
	2814	+ "type": "text",
	2815	+ "primaryKey": true,
	2816	+ "notNull": true
	2817	+ },
	2818	+ "note": {
	2819	+ "name": "note",
	2820	+ "type": "text",
	2821	+ "primaryKey": false,
	2822	+ "notNull": false
	2823	+ },
	2824	+ "invited_by_user_id": {
	2825	+ "name": "invited_by_user_id",
	2826	+ "type": "text",
	2827	+ "primaryKey": false,
	2828	+ "notNull": false
	2829	+ },
	2830	+ "invited_at": {
	2831	+ "name": "invited_at",
	2832	+ "type": "timestamp with time zone",
	2833	+ "primaryKey": false,
	2834	+ "notNull": false
	2835	+ },
	2836	+ "used_at": {
	2837	+ "name": "used_at",
	2838	+ "type": "timestamp with time zone",
	2839	+ "primaryKey": false,
	2840	+ "notNull": false
	2841	+ },
	2842	+ "user_id": {
	2843	+ "name": "user_id",
	2844	+ "type": "text",
	2845	+ "primaryKey": false,
	2846	+ "notNull": false
	2847	+ },
	2848	+ "created_at": {
	2849	+ "name": "created_at",
	2850	+ "type": "timestamp with time zone",
	2851	+ "primaryKey": false,
	2852	+ "notNull": true,
	2853	+ "default": "now()"
	2854	+ }
	2855	+ },
	2856	+ "indexes": {
	2857	+ "signup_allowlist_user_idx": {
	2858	+ "name": "signup_allowlist_user_idx",
	2859	+ "columns": [
	2860	+ {
	2861	+ "expression": "user_id",
	2862	+ "isExpression": false,
	2863	+ "asc": true,
	2864	+ "nulls": "last"
	2865	+ }
	2866	+ ],
	2867	+ "isUnique": false,
	2868	+ "concurrently": false,
	2869	+ "method": "btree",
	2870	+ "with": {}
	2871	+ }
	2872	+ },
	2873	+ "foreignKeys": {
	2874	+ "signup_allowlist_invited_by_user_id_users_id_fk": {
	2875	+ "name": "signup_allowlist_invited_by_user_id_users_id_fk",
	2876	+ "tableFrom": "signup_allowlist",
	2877	+ "tableTo": "users",
	2878	+ "columnsFrom": [
	2879	+ "invited_by_user_id"
	2880	+ ],
	2881	+ "columnsTo": [
	2882	+ "id"
	2883	+ ],
	2884	+ "onDelete": "set null",
	2885	+ "onUpdate": "no action"
	2886	+ },
	2887	+ "signup_allowlist_user_id_users_id_fk": {
	2888	+ "name": "signup_allowlist_user_id_users_id_fk",
	2889	+ "tableFrom": "signup_allowlist",
	2890	+ "tableTo": "users",
	2891	+ "columnsFrom": [
	2892	+ "user_id"
	2893	+ ],
	2894	+ "columnsTo": [
	2895	+ "id"
	2896	+ ],
	2897	+ "onDelete": "set null",
	2898	+ "onUpdate": "no action"
	2899	+ }
	2900	+ },
	2901	+ "compositePrimaryKeys": {},
	2902	+ "uniqueConstraints": {},
	2903	+ "policies": {},
	2904	+ "checkConstraints": {},
	2905	+ "isRLSEnabled": false
	2906	+ },
	2907	+ "public.status_incidents": {
	2908	+ "name": "status_incidents",
	2909	+ "schema": "",
	2910	+ "columns": {
	2911	+ "id": {
	2912	+ "name": "id",
	2913	+ "type": "text",
	2914	+ "primaryKey": true,
	2915	+ "notNull": true
	2916	+ },
	2917	+ "component": {
	2918	+ "name": "component",
	2919	+ "type": "text",
	2920	+ "primaryKey": false,
	2921	+ "notNull": true
	2922	+ },
	2923	+ "title": {
	2924	+ "name": "title",
	2925	+ "type": "text",
	2926	+ "primaryKey": false,
	2927	+ "notNull": true
	2928	+ },
	2929	+ "body": {
	2930	+ "name": "body",
	2931	+ "type": "text",
	2932	+ "primaryKey": false,
	2933	+ "notNull": false
	2934	+ },
	2935	+ "severity": {
	2936	+ "name": "severity",
	2937	+ "type": "text",
	2938	+ "primaryKey": false,
	2939	+ "notNull": true,
	2940	+ "default": "'minor'"
	2941	+ },
	2942	+ "started_at": {
	2943	+ "name": "started_at",
	2944	+ "type": "timestamp with time zone",
	2945	+ "primaryKey": false,
	2946	+ "notNull": true,
	2947	+ "default": "now()"
	2948	+ },
	2949	+ "resolved_at": {
	2950	+ "name": "resolved_at",
	2951	+ "type": "timestamp with time zone",
	2952	+ "primaryKey": false,
	2953	+ "notNull": false
	2954	+ }
	2955	+ },
	2956	+ "indexes": {},
	2957	+ "foreignKeys": {},
	2958	+ "compositePrimaryKeys": {},
	2959	+ "uniqueConstraints": {},
	2960	+ "policies": {},
	2961	+ "checkConstraints": {},
	2962	+ "isRLSEnabled": false
	2963	+ },
	2964	+ "public.subscriptions": {
	2965	+ "name": "subscriptions",
	2966	+ "schema": "",
	2967	+ "columns": {
	2968	+ "id": {
	2969	+ "name": "id",
	2970	+ "type": "text",
	2971	+ "primaryKey": true,
	2972	+ "notNull": true
	2973	+ },
	2974	+ "organization_id": {
	2975	+ "name": "organization_id",
	2976	+ "type": "text",
	2977	+ "primaryKey": false,
	2978	+ "notNull": true
	2979	+ },
	2980	+ "plan": {
	2981	+ "name": "plan",
	2982	+ "type": "text",
	2983	+ "primaryKey": false,
	2984	+ "notNull": true
	2985	+ },
	2986	+ "status": {
	2987	+ "name": "status",
	2988	+ "type": "text",
	2989	+ "primaryKey": false,
	2990	+ "notNull": true,
	2991	+ "default": "'active'"
	2992	+ },
	2993	+ "stripe_subscription_id": {
	2994	+ "name": "stripe_subscription_id",
	2995	+ "type": "text",
	2996	+ "primaryKey": false,
	2997	+ "notNull": false
	2998	+ },
	2999	+ "current_period_start": {
	3000	+ "name": "current_period_start",
	3001	+ "type": "timestamp with time zone",
	3002	+ "primaryKey": false,
	3003	+ "notNull": false
	3004	+ },
	3005	+ "current_period_end": {
	3006	+ "name": "current_period_end",
	3007	+ "type": "timestamp with time zone",
	3008	+ "primaryKey": false,
	3009	+ "notNull": false
	3010	+ },
	3011	+ "cancel_at_period_end": {
	3012	+ "name": "cancel_at_period_end",
	3013	+ "type": "boolean",
	3014	+ "primaryKey": false,
	3015	+ "notNull": true,
	3016	+ "default": false
	3017	+ },
	3018	+ "created_at": {
	3019	+ "name": "created_at",
	3020	+ "type": "timestamp with time zone",
	3021	+ "primaryKey": false,
	3022	+ "notNull": true,
	3023	+ "default": "now()"
	3024	+ },
	3025	+ "updated_at": {
	3026	+ "name": "updated_at",
	3027	+ "type": "timestamp with time zone",
	3028	+ "primaryKey": false,
	3029	+ "notNull": true,
	3030	+ "default": "now()"
	3031	+ }
	3032	+ },
	3033	+ "indexes": {},
	3034	+ "foreignKeys": {
	3035	+ "subscriptions_organization_id_organizations_id_fk": {
	3036	+ "name": "subscriptions_organization_id_organizations_id_fk",
	3037	+ "tableFrom": "subscriptions",
	3038	+ "tableTo": "organizations",
	3039	+ "columnsFrom": [
	3040	+ "organization_id"
	3041	+ ],
	3042	+ "columnsTo": [
	3043	+ "id"
	3044	+ ],
	3045	+ "onDelete": "cascade",
	3046	+ "onUpdate": "no action"
	3047	+ }
	3048	+ },
	3049	+ "compositePrimaryKeys": {},
	3050	+ "uniqueConstraints": {},
	3051	+ "policies": {},
	3052	+ "checkConstraints": {},
	3053	+ "isRLSEnabled": false
	3054	+ },
	3055	+ "public.usage_events": {
	3056	+ "name": "usage_events",
	3057	+ "schema": "",
	3058	+ "columns": {
	3059	+ "id": {
	3060	+ "name": "id",
	3061	+ "type": "text",
	3062	+ "primaryKey": true,
	3063	+ "notNull": true
	3064	+ },
	3065	+ "organization_id": {
	3066	+ "name": "organization_id",
	3067	+ "type": "text",
	3068	+ "primaryKey": false,
	3069	+ "notNull": true
	3070	+ },
	3071	+ "project_id": {
	3072	+ "name": "project_id",
	3073	+ "type": "text",
	3074	+ "primaryKey": false,
	3075	+ "notNull": false
	3076	+ },
	3077	+ "request_id": {
	3078	+ "name": "request_id",
	3079	+ "type": "text",
	3080	+ "primaryKey": false,
	3081	+ "notNull": false
	3082	+ },
	3083	+ "metric": {
	3084	+ "name": "metric",
	3085	+ "type": "text",
	3086	+ "primaryKey": false,
	3087	+ "notNull": true
	3088	+ },
	3089	+ "quantity": {
	3090	+ "name": "quantity",
	3091	+ "type": "double precision",
	3092	+ "primaryKey": false,
	3093	+ "notNull": true
	3094	+ },
	3095	+ "unit": {
	3096	+ "name": "unit",
	3097	+ "type": "text",
	3098	+ "primaryKey": false,
	3099	+ "notNull": true
	3100	+ },
	3101	+ "cost_usd": {
	3102	+ "name": "cost_usd",
	3103	+ "type": "double precision",
	3104	+ "primaryKey": false,
	3105	+ "notNull": true,
	3106	+ "default": 0
	3107	+ },
	3108	+ "upstream_cost_usd": {
	3109	+ "name": "upstream_cost_usd",
	3110	+ "type": "double precision",
	3111	+ "primaryKey": false,
	3112	+ "notNull": true,
	3113	+ "default": 0
	3114	+ },
	3115	+ "created_at": {
	3116	+ "name": "created_at",
	3117	+ "type": "timestamp with time zone",
	3118	+ "primaryKey": false,
	3119	+ "notNull": true,
	3120	+ "default": "now()"
	3121	+ }
	3122	+ },
	3123	+ "indexes": {
	3124	+ "usage_events_org_created_idx": {
	3125	+ "name": "usage_events_org_created_idx",
	3126	+ "columns": [
	3127	+ {
	3128	+ "expression": "organization_id",
	3129	+ "isExpression": false,
	3130	+ "asc": true,
	3131	+ "nulls": "last"
	3132	+ },
	3133	+ {
	3134	+ "expression": "created_at",
	3135	+ "isExpression": false,
	3136	+ "asc": true,
	3137	+ "nulls": "last"
	3138	+ }
	3139	+ ],
	3140	+ "isUnique": false,
	3141	+ "concurrently": false,
	3142	+ "method": "btree",
	3143	+ "with": {}
	3144	+ },
	3145	+ "usage_events_project_created_idx": {
	3146	+ "name": "usage_events_project_created_idx",
	3147	+ "columns": [
	3148	+ {
	3149	+ "expression": "project_id",
	3150	+ "isExpression": false,
	3151	+ "asc": true,
	3152	+ "nulls": "last"
	3153	+ },
	3154	+ {
	3155	+ "expression": "created_at",
	3156	+ "isExpression": false,
	3157	+ "asc": true,
	3158	+ "nulls": "last"
	3159	+ }
	3160	+ ],
	3161	+ "isUnique": false,
	3162	+ "concurrently": false,
	3163	+ "method": "btree",
	3164	+ "with": {}
	3165	+ }
	3166	+ },
	3167	+ "foreignKeys": {
	3168	+ "usage_events_organization_id_organizations_id_fk": {
	3169	+ "name": "usage_events_organization_id_organizations_id_fk",
	3170	+ "tableFrom": "usage_events",
	3171	+ "tableTo": "organizations",
	3172	+ "columnsFrom": [
	3173	+ "organization_id"
	3174	+ ],
	3175	+ "columnsTo": [
	3176	+ "id"
	3177	+ ],
	3178	+ "onDelete": "cascade",
	3179	+ "onUpdate": "no action"
	3180	+ },
	3181	+ "usage_events_project_id_projects_id_fk": {
	3182	+ "name": "usage_events_project_id_projects_id_fk",
	3183	+ "tableFrom": "usage_events",
	3184	+ "tableTo": "projects",
	3185	+ "columnsFrom": [
	3186	+ "project_id"
	3187	+ ],
	3188	+ "columnsTo": [
	3189	+ "id"
	3190	+ ],
	3191	+ "onDelete": "set null",
	3192	+ "onUpdate": "no action"
	3193	+ }
	3194	+ },
	3195	+ "compositePrimaryKeys": {},
	3196	+ "uniqueConstraints": {},
	3197	+ "policies": {},
	3198	+ "checkConstraints": {},
	3199	+ "isRLSEnabled": false
	3200	+ },
	3201	+ "public.users": {
	3202	+ "name": "users",
	3203	+ "schema": "",
	3204	+ "columns": {
	3205	+ "id": {
	3206	+ "name": "id",
	3207	+ "type": "text",
	3208	+ "primaryKey": true,
	3209	+ "notNull": true
	3210	+ },
	3211	+ "name": {
	3212	+ "name": "name",
	3213	+ "type": "text",
	3214	+ "primaryKey": false,
	3215	+ "notNull": true,
	3216	+ "default": "''"
	3217	+ },
	3218	+ "email": {
	3219	+ "name": "email",
	3220	+ "type": "text",
	3221	+ "primaryKey": false,
	3222	+ "notNull": true
	3223	+ },
	3224	+ "email_verified": {
	3225	+ "name": "email_verified",
	3226	+ "type": "boolean",
	3227	+ "primaryKey": false,
	3228	+ "notNull": true,
	3229	+ "default": false
	3230	+ },
	3231	+ "image": {
	3232	+ "name": "image",
	3233	+ "type": "text",
	3234	+ "primaryKey": false,
	3235	+ "notNull": false
	3236	+ },
	3237	+ "role": {
	3238	+ "name": "role",
	3239	+ "type": "text",
	3240	+ "primaryKey": false,
	3241	+ "notNull": true,
	3242	+ "default": "'user'"
	3243	+ },
	3244	+ "banned": {
	3245	+ "name": "banned",
	3246	+ "type": "boolean",
	3247	+ "primaryKey": false,
	3248	+ "notNull": true,
	3249	+ "default": false
	3250	+ },
	3251	+ "ban_reason": {
	3252	+ "name": "ban_reason",
	3253	+ "type": "text",
	3254	+ "primaryKey": false,
	3255	+ "notNull": false
	3256	+ },
	3257	+ "onboarding_completed_at": {
	3258	+ "name": "onboarding_completed_at",
	3259	+ "type": "timestamp with time zone",
	3260	+ "primaryKey": false,
	3261	+ "notNull": false
	3262	+ },
	3263	+ "created_at": {
	3264	+ "name": "created_at",
	3265	+ "type": "timestamp with time zone",
	3266	+ "primaryKey": false,
	3267	+ "notNull": true,
	3268	+ "default": "now()"
	3269	+ },
	3270	+ "updated_at": {
	3271	+ "name": "updated_at",
	3272	+ "type": "timestamp with time zone",
	3273	+ "primaryKey": false,
	3274	+ "notNull": true,
	3275	+ "default": "now()"
	3276	+ }
	3277	+ },
	3278	+ "indexes": {
	3279	+ "users_email_uq": {
	3280	+ "name": "users_email_uq",
	3281	+ "columns": [
	3282	+ {
	3283	+ "expression": "email",
	3284	+ "isExpression": false,
	3285	+ "asc": true,
	3286	+ "nulls": "last"
	3287	+ }
	3288	+ ],
	3289	+ "isUnique": true,
	3290	+ "concurrently": false,
	3291	+ "method": "btree",
	3292	+ "with": {}
	3293	+ }
	3294	+ },
	3295	+ "foreignKeys": {},
	3296	+ "compositePrimaryKeys": {},
	3297	+ "uniqueConstraints": {},
	3298	+ "policies": {},
	3299	+ "checkConstraints": {},
	3300	+ "isRLSEnabled": false
	3301	+ },
	3302	+ "public.verifications": {
	3303	+ "name": "verifications",
	3304	+ "schema": "",
	3305	+ "columns": {
	3306	+ "id": {
	3307	+ "name": "id",
	3308	+ "type": "text",
	3309	+ "primaryKey": true,
	3310	+ "notNull": true
	3311	+ },
	3312	+ "identifier": {
	3313	+ "name": "identifier",
	3314	+ "type": "text",
	3315	+ "primaryKey": false,
	3316	+ "notNull": true
	3317	+ },
	3318	+ "value": {
	3319	+ "name": "value",
	3320	+ "type": "text",
	3321	+ "primaryKey": false,
	3322	+ "notNull": true
	3323	+ },
	3324	+ "expires_at": {
	3325	+ "name": "expires_at",
	3326	+ "type": "timestamp with time zone",
	3327	+ "primaryKey": false,
	3328	+ "notNull": true
	3329	+ },
	3330	+ "created_at": {
	3331	+ "name": "created_at",
	3332	+ "type": "timestamp with time zone",
	3333	+ "primaryKey": false,
	3334	+ "notNull": true,
	3335	+ "default": "now()"
	3336	+ },
	3337	+ "updated_at": {
	3338	+ "name": "updated_at",
	3339	+ "type": "timestamp with time zone",
	3340	+ "primaryKey": false,
	3341	+ "notNull": true,
	3342	+ "default": "now()"
	3343	+ }
	3344	+ },
	3345	+ "indexes": {
	3346	+ "verifications_identifier_idx": {
	3347	+ "name": "verifications_identifier_idx",
	3348	+ "columns": [
	3349	+ {
	3350	+ "expression": "identifier",
	3351	+ "isExpression": false,
	3352	+ "asc": true,
	3353	+ "nulls": "last"
	3354	+ }
	3355	+ ],
	3356	+ "isUnique": false,
	3357	+ "concurrently": false,
	3358	+ "method": "btree",
	3359	+ "with": {}
	3360	+ }
	3361	+ },
	3362	+ "foreignKeys": {},
	3363	+ "compositePrimaryKeys": {},
	3364	+ "uniqueConstraints": {},
	3365	+ "policies": {},
	3366	+ "checkConstraints": {},
	3367	+ "isRLSEnabled": false
	3368	+ },
	3369	+ "public.webhook_deliveries": {
	3370	+ "name": "webhook_deliveries",
	3371	+ "schema": "",
	3372	+ "columns": {
	3373	+ "id": {
	3374	+ "name": "id",
	3375	+ "type": "text",
	3376	+ "primaryKey": true,
	3377	+ "notNull": true
	3378	+ },
	3379	+ "webhook_id": {
	3380	+ "name": "webhook_id",
	3381	+ "type": "text",
	3382	+ "primaryKey": false,
	3383	+ "notNull": true
	3384	+ },
	3385	+ "event": {
	3386	+ "name": "event",
	3387	+ "type": "text",
	3388	+ "primaryKey": false,
	3389	+ "notNull": true
	3390	+ },
	3391	+ "payload": {
	3392	+ "name": "payload",
	3393	+ "type": "jsonb",
	3394	+ "primaryKey": false,
	3395	+ "notNull": true
	3396	+ },
	3397	+ "status": {
	3398	+ "name": "status",
	3399	+ "type": "text",
	3400	+ "primaryKey": false,
	3401	+ "notNull": true,
	3402	+ "default": "'pending'"
	3403	+ },
	3404	+ "response_status": {
	3405	+ "name": "response_status",
	3406	+ "type": "integer",
	3407	+ "primaryKey": false,
	3408	+ "notNull": false
	3409	+ },
	3410	+ "attempts": {
	3411	+ "name": "attempts",
	3412	+ "type": "integer",
	3413	+ "primaryKey": false,
	3414	+ "notNull": true,
	3415	+ "default": 0
	3416	+ },
	3417	+ "next_attempt_at": {
	3418	+ "name": "next_attempt_at",
	3419	+ "type": "timestamp with time zone",
	3420	+ "primaryKey": false,
	3421	+ "notNull": false
	3422	+ },
	3423	+ "created_at": {
	3424	+ "name": "created_at",
	3425	+ "type": "timestamp with time zone",
	3426	+ "primaryKey": false,
	3427	+ "notNull": true,
	3428	+ "default": "now()"
	3429	+ }
	3430	+ },
	3431	+ "indexes": {
	3432	+ "webhook_deliveries_webhook_idx": {
	3433	+ "name": "webhook_deliveries_webhook_idx",
	3434	+ "columns": [
	3435	+ {
	3436	+ "expression": "webhook_id",
	3437	+ "isExpression": false,
	3438	+ "asc": true,
	3439	+ "nulls": "last"
	3440	+ }
	3441	+ ],
	3442	+ "isUnique": false,
	3443	+ "concurrently": false,
	3444	+ "method": "btree",
	3445	+ "with": {}
	3446	+ }
	3447	+ },
	3448	+ "foreignKeys": {
	3449	+ "webhook_deliveries_webhook_id_webhooks_id_fk": {
	3450	+ "name": "webhook_deliveries_webhook_id_webhooks_id_fk",
	3451	+ "tableFrom": "webhook_deliveries",
	3452	+ "tableTo": "webhooks",
	3453	+ "columnsFrom": [
	3454	+ "webhook_id"
	3455	+ ],
	3456	+ "columnsTo": [
	3457	+ "id"
	3458	+ ],
	3459	+ "onDelete": "cascade",
	3460	+ "onUpdate": "no action"
	3461	+ }
	3462	+ },
	3463	+ "compositePrimaryKeys": {},
	3464	+ "uniqueConstraints": {},
	3465	+ "policies": {},
	3466	+ "checkConstraints": {},
	3467	+ "isRLSEnabled": false
	3468	+ },
	3469	+ "public.webhooks": {
	3470	+ "name": "webhooks",
	3471	+ "schema": "",
	3472	+ "columns": {
	3473	+ "id": {
	3474	+ "name": "id",
	3475	+ "type": "text",
	3476	+ "primaryKey": true,
	3477	+ "notNull": true
	3478	+ },
	3479	+ "organization_id": {
	3480	+ "name": "organization_id",
	3481	+ "type": "text",
	3482	+ "primaryKey": false,
	3483	+ "notNull": true
	3484	+ },
	3485	+ "project_id": {
	3486	+ "name": "project_id",
	3487	+ "type": "text",
	3488	+ "primaryKey": false,
	3489	+ "notNull": false
	3490	+ },
	3491	+ "url": {
	3492	+ "name": "url",
	3493	+ "type": "text",
	3494	+ "primaryKey": false,
	3495	+ "notNull": true
	3496	+ },
	3497	+ "secret": {
	3498	+ "name": "secret",
	3499	+ "type": "text",
	3500	+ "primaryKey": false,
	3501	+ "notNull": true
	3502	+ },
	3503	+ "events": {
	3504	+ "name": "events",
	3505	+ "type": "jsonb",
	3506	+ "primaryKey": false,
	3507	+ "notNull": true,
	3508	+ "default": "'[]'::jsonb"
	3509	+ },
	3510	+ "enabled": {
	3511	+ "name": "enabled",
	3512	+ "type": "boolean",
	3513	+ "primaryKey": false,
	3514	+ "notNull": true,
	3515	+ "default": true
	3516	+ },
	3517	+ "created_at": {
	3518	+ "name": "created_at",
	3519	+ "type": "timestamp with time zone",
	3520	+ "primaryKey": false,
	3521	+ "notNull": true,
	3522	+ "default": "now()"
	3523	+ }
	3524	+ },
	3525	+ "indexes": {},
	3526	+ "foreignKeys": {
	3527	+ "webhooks_organization_id_organizations_id_fk": {
	3528	+ "name": "webhooks_organization_id_organizations_id_fk",
	3529	+ "tableFrom": "webhooks",
	3530	+ "tableTo": "organizations",
	3531	+ "columnsFrom": [
	3532	+ "organization_id"
	3533	+ ],
	3534	+ "columnsTo": [
	3535	+ "id"
	3536	+ ],
	3537	+ "onDelete": "cascade",
	3538	+ "onUpdate": "no action"
	3539	+ },
	3540	+ "webhooks_project_id_projects_id_fk": {
	3541	+ "name": "webhooks_project_id_projects_id_fk",
	3542	+ "tableFrom": "webhooks",
	3543	+ "tableTo": "projects",
	3544	+ "columnsFrom": [
	3545	+ "project_id"
	3546	+ ],
	3547	+ "columnsTo": [
	3548	+ "id"
	3549	+ ],
	3550	+ "onDelete": "cascade",
	3551	+ "onUpdate": "no action"
	3552	+ }
	3553	+ },
	3554	+ "compositePrimaryKeys": {},
	3555	+ "uniqueConstraints": {},
	3556	+ "policies": {},
	3557	+ "checkConstraints": {},
	3558	+ "isRLSEnabled": false
	3559	+ }
	3560	+ },
	3561	+ "enums": {},
	3562	+ "schemas": {},
	3563	+ "sequences": {},
	3564	+ "roles": {},
	3565	+ "policies": {},
	3566	+ "views": {},
	3567	+ "_meta": {
	3568	+ "columns": {},
	3569	+ "schemas": {},
	3570	+ "tables": {}
	3571	+ }
	3572	+}
		\ No newline at end of file

modified packages/db/drizzle/meta/_journal.json +7 −0

		`@@ -8,6 +8,13 @@`
8	8	"when": 1788811423052,
9	9	"tag": "0000_closed_golden_guardian",
10	10	"breakpoints": true
	11	+ },
	12	+ {
	13	+ "idx": 1,
	14	+ "version": "7",
	15	+ "when": 1788845554374,
	16	+ "tag": "0001_furry_red_skull",
	17	+ "breakpoints": true
11	18	}
12	19	]
13	20	}
		\ No newline at end of file

modified packages/db/src/schema.ts +96 −1

		`@@ -89,6 +89,26 @@ export const verifications = pgTable(`
89	89	(t) => [index("verifications_identifier_idx").on(t.identifier)],
90	90	);
91	91
	92	+/**
	93	+ * Signup allowlist: Fetcha is invitation-only. An administrator adds an email here (optionally
	94	+ * sending an invitation), and only then can that address create an account.
	95	+ */
	96	+export const signupAllowlist = pgTable(
	97	+ "signup_allowlist",
	98	+ {
	99	+ email: text("email").primaryKey(), // lower-cased
	100	+ note: text("note"),
	101	+ invitedByUserId: text("invited_by_user_id").references(() => users.id, { onDelete: "set null" }),
	102	+ /** Last time an invitation email was sent. */
	103	+ invitedAt: ts("invited_at"),
	104	+ /** Set when the account was created. */
	105	+ usedAt: ts("used_at"),
	106	+ userId: text("user_id").references(() => users.id, { onDelete: "set null" }),
	107	+ createdAt: ts("created_at").notNull().default(now()),
	108	+ },
	109	+ (t) => [index("signup_allowlist_user_idx").on(t.userId)],
	110	+);
	111	+
92	112	// ---------------------------------------------------------------------------
93	113	// Organizations & projects
94	114	// ---------------------------------------------------------------------------
		`@@ -98,7 +118,7 @@ export const organizations = pgTable(`
98	118	id: text("id").primaryKey(),
99	119	name: text("name").notNull(),
100	120	slug: text("slug").notNull(),
101		− plan: text("plan").notNull().default("free"),
	121	+ plan: text("plan").notNull().default("unlimited"),
102	122	ownerUserId: text("owner_user_id")
103	123	.notNull()
104	124	.references(() => users.id, { onDelete: "restrict" }),
		`@@ -205,6 +225,8 @@ export const fetchRequests = pgTable(`
205	225	city: text("city"),
206	226	sessionId: text("session_id"),
207	227	browser: boolean("browser").notNull().default(false),
	228	+ /** Execution mode of the final attempt: http \| browser. */
	229	+ mode: text("mode"),
208	230	format: text("format").notNull().default("html"),
209	231	status: text("status").notNull().default("pending"), // pending \| success \| failed
210	232	httpStatus: integer("http_status"),
		`@@ -243,6 +265,7 @@ export const requestAttempts = pgTable(`
243	265	attemptNo: integer("attempt_no").notNull(),
244	266	provider: text("provider").notNull(), // oxylabs \| decodo \| soax \| direct
245	267	network: text("network").notNull(),
	268	+ mode: text("mode").notNull().default("http"), // http \| browser
246	269	country: text("country"),
247	270	sessionKey: text("session_key"),
248	271	outcome: text("outcome").notNull(), // success \| blocked \| timeout \| error \| provider_error
		`@@ -340,6 +363,75 @@ export const billingEvents = pgTable(`
340	363	(t) => [index("billing_events_org_idx").on(t.organizationId)],
341	364	);
342	365
	366	+// ---------------------------------------------------------------------------
	367	+// Crawl jobs
	368	+// ---------------------------------------------------------------------------
	369	+export const crawlJobs = pgTable(
	370	+ "crawl_jobs",
	371	+ {
	372	+ id: text("id").primaryKey(), // crawl_xxx
	373	+ organizationId: text("organization_id")
	374	+ .notNull()
	375	+ .references(() => organizations.id, { onDelete: "cascade" }),
	376	+ projectId: text("project_id")
	377	+ .notNull()
	378	+ .references(() => projects.id, { onDelete: "cascade" }),
	379	+ apiKeyId: text("api_key_id").references(() => apiKeys.id, { onDelete: "set null" }),
	380	+ source: text("source").notNull().default("api"), // api \| playground \| sdk
	381	+ label: text("label"),
	382	+ seedUrl: text("seed_url").notNull(),
	383	+ domain: text("domain").notNull(),
	384	+ options: jsonb("options").$type<Record<string, unknown>>().notNull(),
	385	+ status: text("status").notNull().default("queued"), // queued \| running \| completed \| failed \| cancelled
	386	+ workerId: text("worker_id"),
	387	+ pagesDiscovered: integer("pages_discovered").notNull().default(0),
	388	+ pagesFetched: integer("pages_fetched").notNull().default(0),
	389	+ pagesOk: integer("pages_ok").notNull().default(0),
	390	+ pagesBlocked: integer("pages_blocked").notNull().default(0),
	391	+ pagesFailed: integer("pages_failed").notNull().default(0),
	392	+ bytes: bigint("bytes", { mode: "number" }).notNull().default(0),
	393	+ costUsd: doublePrecision("cost_usd").notNull().default(0),
	394	+ errorCode: text("error_code"),
	395	+ errorMessage: text("error_message"),
	396	+ webhookStatus: text("webhook_status"),
	397	+ createdAt: ts("created_at").notNull().default(now()),
	398	+ startedAt: ts("started_at"),
	399	+ heartbeatAt: ts("heartbeat_at"),
	400	+ completedAt: ts("completed_at"),
	401	+ },
	402	+ (t) => [index("crawl_jobs_project_created_idx").on(t.projectId, t.createdAt), index("crawl_jobs_status_idx").on(t.status)],
	403	+);
	404	+
	405	+export const crawlPages = pgTable(
	406	+ "crawl_pages",
	407	+ {
	408	+ id: text("id").primaryKey(), // cpg_xxx
	409	+ jobId: text("job_id")
	410	+ .notNull()
	411	+ .references(() => crawlJobs.id, { onDelete: "cascade" }),
	412	+ url: text("url").notNull(),
	413	+ finalUrl: text("final_url"),
	414	+ depth: integer("depth").notNull().default(0),
	415	+ parentUrl: text("parent_url"),
	416	+ status: text("status").notNull().default("pending"), // pending \| success \| blocked \| failed \| skipped
	417	+ httpStatus: integer("http_status"),
	418	+ errorCode: text("error_code"),
	419	+ requestId: text("request_id"),
	420	+ title: text("title"),
	421	+ description: text("description"),
	422	+ contentType: text("content_type"),
	423	+ /** Converted content (markdown / text / html) — bounded by the job's per-page cap. */
	424	+ content: text("content"),
	425	+ linksCount: integer("links_count").notNull().default(0),
	426	+ bytes: bigint("bytes", { mode: "number" }).notNull().default(0),
	427	+ durationMs: integer("duration_ms"),
	428	+ mode: text("mode"), // http \| browser
	429	+ createdAt: ts("created_at").notNull().default(now()),
	430	+ fetchedAt: ts("fetched_at"),
	431	+ },
	432	+ (t) => [index("crawl_pages_job_idx").on(t.jobId, t.createdAt), uniqueIndex("crawl_pages_job_url_uq").on(t.jobId, t.url)],
	433	+);
	434	+
343	435	// ---------------------------------------------------------------------------
344	436	// Providers, routing intelligence
345	437	// ---------------------------------------------------------------------------
		`@@ -560,3 +652,6 @@ export type UsageEvent = typeof usageEvents.$inferSelect;`
560	652	export type DomainProfile = typeof domainProfiles.$inferSelect;
561	653	export type ProviderHealthRow = typeof providerHealth.$inferSelect;
562	654	export type AuditLog = typeof auditLogs.$inferSelect;
	655	+export type SignupAllowlistRow = typeof signupAllowlist.$inferSelect;
	656	+export type CrawlJob = typeof crawlJobs.$inferSelect;
	657	+export type CrawlPage = typeof crawlPages.$inferSelect;

modified packages/db/src/seed.ts +47 −3

		`@@ -1,6 +1,18 @@`
1	1	import { getDb, closeDb } from "./index";
2		−import { providerConfigs, featureFlags } from "./schema";
3		−import { sql } from "drizzle-orm";
	2	+import { providerConfigs, featureFlags, organizations, signupAllowlist, users } from "./schema";
	3	+import { and, eq, ne, sql } from "drizzle-orm";
	4	+
	5	+/** Emails from `ADMIN_EMAILS` (comma-separated, case-insensitive). */
	6	+function adminEmails(): string[] {
	7	+ return Array.from(
	8	+ new Set(
	9	+ (process.env.ADMIN_EMAILS ?? "")
	10	+ .split(",")
	11	+ .map((s) => s.trim().toLowerCase())
	12	+ .filter(Boolean),
	13	+ ),
	14	+ );
	15	+}
4	16
5	17	async function main() {
6	18	const db = getDb();
		`@@ -15,8 +27,9 @@ async function main() {`
15	27	.values(p)
16	28	.onConflictDoUpdate({ target: providerConfigs.id, set: { label: p.label, networks: p.networks, pricePerGbUsd: p.pricePerGbUsd, updatedAt: sql`now()` } });
17	29	}
	30	+
18	31	const flags = [
19		− { key: "browser_enabled", enabled: false, description: "Managed browser execution (/v1/fetch browser=true)" },
	32	+ { key: "browser_enabled", enabled: true, description: "Managed browser rendering (live)" },
20	33	{ key: "extract_enabled", enabled: false, description: "Structured extraction endpoint /v1/extract" },
21	34	{ key: "mobile_proxy_enabled", enabled: false, description: "Mobile network class" },
22	35	{ key: "organizations_enabled", enabled: false, description: "Team organizations UI" },
		`@@ -25,6 +38,37 @@ async function main() {`
25	38	for (const f of flags) {
26	39	await db.insert(featureFlags).values(f).onConflictDoNothing();
27	40	}
	41	+
	42	+ // Single-plan platform: defensively normalize any legacy plan value (migration 0001 already does this).
	43	+ const normalized = await db.update(organizations).set({ plan: "unlimited", updatedAt: sql`now()` }).where(ne(organizations.plan, "unlimited")).returning({ id: organizations.id });
	44	+ if (normalized.length) console.log(`[db] ${normalized.length} organization(s) moved to plan=unlimited`);
	45	+
	46	+ // Administrators: allowlisted + promoted (idempotent).
	47	+ const admins = adminEmails();
	48	+ if (!admins.length) {
	49	+ console.warn("[db] ADMIN_EMAILS is empty — no administrator allowlisted or promoted");
	50	+ }
	51	+ for (const email of admins) {
	52	+ await db.insert(signupAllowlist).values({ email, note: "administrator" }).onConflictDoNothing();
	53	+ const promoted = await db
	54	+ .update(users)
	55	+ .set({ role: "admin", updatedAt: sql`now()` })
	56	+ .where(and(sql`lower(${users.email}) = ${email}`, ne(users.role, "admin")))
	57	+ .returning({ id: users.id });
	58	+ if (promoted.length) console.log(`[db] promoted ${email} to admin`);
	59	+ const [existing] = await db.select({ id: users.id, createdAt: users.createdAt }).from(users).where(sql`lower(${users.email}) = ${email}`).limit(1);
	60	+ if (existing) {
	61	+ await db
	62	+ .update(signupAllowlist)
	63	+ .set({ userId: existing.id, usedAt: sql`coalesce(${signupAllowlist.usedAt}, ${existing.createdAt})` })
	64	+ .where(eq(signupAllowlist.email, email));
	65	+ }
	66	+ console.log(`[db] admin ${email}: allowlisted${existing ? ", account linked" : ", no account yet"}`);
	67	+ }
	68	+
	69	+ // v0.2: the managed browser is live. Rows seeded by v0.1 (still carrying the old description) are
	70	+ // switched on once; an admin toggling the flag off afterwards is respected on later seeds.
	71	+ await db.execute(sql`update feature_flags set enabled = true, description = 'Managed browser rendering (live)', updated_at = now() where key = 'browser_enabled' and description = 'Managed browser execution (/v1/fetch browser=true)'`);
28	72	console.log("[db] seed ok");
29	73	await closeDb();
30	74	}

modified packages/email/src/index.ts +6 −0

		`@@ -8,6 +8,7 @@ import {`
8	8	BillingEmail,
9	9	EmailChangeApprovalEmail,
10	10	EmailChangedEmail,
	11	+ InviteEmail,
11	12	NewLoginEmail,
12	13	PasswordResetEmail,
13	14	UsageAlertEmail,
		`@@ -81,6 +82,11 @@ export class EmailService {`
81	82	);
82	83	}
83	84
	85	+ /** Invitation to join the private platform (marketing sender). `signupUrl` should pre-fill the email. */
	86	+ sendInvite(to: string, p: { inviterName: string; signupUrl: string }) {
	87	+ return this.send(to, "You're invited to Fetcha", React.createElement(InviteEmail, { ...p, siteUrl: this.siteUrl }), { replyTo: "hello@fetcha.co" });
	88	+ }
	89	+
84	90	sendPasswordReset(to: string, url: string) {
85	91	return this.send(to, "Reset your Fetcha password", React.createElement(PasswordResetEmail, { url, siteUrl: this.siteUrl }), { transactional: true });
86	92	}

modified packages/email/src/templates.tsx +26 −1

		`@@ -86,12 +86,37 @@ export function WelcomeEmail({ name, dashboardUrl, docsUrl, siteUrl }: { name: s`
86	86	<Link href={docsUrl} style={{ color: colors.accent }}>
87	87	{docsUrl}
88	88	</Link>
89		− . Your free plan includes 1,000 requests per month.
	89	+ . Fetcha is a private platform: unlimited requests, no plans, no meters.
90	90	</Text>
91	91	</Layout>
92	92	);
93	93	}
94	94
	95	+export function InviteEmail({ inviterName, signupUrl, siteUrl }: { inviterName: string; signupUrl: string; siteUrl: string }) {
	96	+ const by = inviterName.trim();
	97	+ return (
	98	+ <Layout preview="You have been invited to Fetcha" siteUrl={siteUrl}>
	99	+ <Heading style={h1}>You have been invited to Fetcha</Heading>
	100	+ <Text style={p}>
	101	+ {by ? <><strong>{by}</strong> has added your email address to the Fetcha access list.</> : <>Your email address was added to the Fetcha access list.</>} Fetcha is a private,
	102	+ invitation-only platform: one API to reach any web page through managed proxy networks, a headless browser and crawl jobs, with no plans and no meters.
	103	+ </Text>
	104	+ <Text style={p}>Create your account with this same email address to get started. Your workspace, a default project and the Playground are ready as soon as you sign up.</Text>
	105	+ <Button href={signupUrl} style={btn}>
	106	+ Create my account
	107	+ </Button>
	108	+ <Text style={small}>
	109	+ If the button does not work, paste this URL in your browser:
	110	+ <br />
	111	+ <Link href={signupUrl} style={{ color: colors.accent, wordBreak: "break-all" }}>
	112	+ {signupUrl}
	113	+ </Link>
	114	+ </Text>
	115	+ <Text style={small}>If you were not expecting this invitation, you can safely ignore this email — no account is created until you sign up.</Text>
	116	+ </Layout>
	117	+ );
	118	+}
	119	+
95	120	export function PasswordResetEmail({ url, siteUrl }: { url: string; siteUrl: string }) {
96	121	return (
97	122	<Layout preview="Reset your Fetcha password" siteUrl={siteUrl}>

added packages/providers/src/cookies.ts +129 −0

		`@@ -0,0 +1,129 @@`
	1	+/**
	2	+ * Small RFC 6265-ish cookie jar used within a single fetch (across redirect hops) and, for sticky
	3	+ * sessions, persisted between requests. No dependency; ignores SameSite (irrelevant server-side).
	4	+ */
	5	+export interface JarCookie {
	6	+ name: string;
	7	+ value: string;
	8	+ domain: string; // without leading dot, lower-case
	9	+ hostOnly: boolean;
	10	+ path: string;
	11	+ secure: boolean;
	12	+ expires: number \| null; // epoch ms, null = session
	13	+}
	14	+
	15	+export class CookieJar {
	16	+ private cookies: JarCookie[] = [];
	17	+
	18	+ static fromSerialized(list: Array<{ name: string; value: string; domain?: string; path?: string; secure?: boolean; expires?: number \| null }> \| null \| undefined, fallbackHost?: string): CookieJar {
	19	+ const jar = new CookieJar();
	20	+ for (const c of list ?? []) {
	21	+ const domain = (c.domain ?? fallbackHost ?? "").toLowerCase().replace(/^\./, "");
	22	+ if (!domain) continue;
	23	+ jar.cookies.push({ name: c.name, value: c.value, domain, hostOnly: !c.domain, path: c.path ?? "/", secure: Boolean(c.secure), expires: c.expires ?? null });
	24	+ }
	25	+ return jar;
	26	+ }
	27	+
	28	+ size(): number {
	29	+ return this.cookies.length;
	30	+ }
	31	+
	32	+ serialize(): Array<{ name: string; value: string; domain: string; path: string; secure: boolean; expires: number \| null }> {
	33	+ const now = Date.now();
	34	+ return this.cookies.filter((c) => c.expires === null \|\| c.expires > now).map((c) => ({ name: c.name, value: c.value, domain: c.hostOnly ? c.domain : `.${c.domain}`, path: c.path, secure: c.secure, expires: c.expires }));
	35	+ }
	36	+
	37	+ /** Cookies exposed to the customer: name/value/domain/path only. */
	38	+ publicList(): Array<{ name: string; value: string; domain?: string; path?: string }> {
	39	+ return this.serialize().map((c) => ({ name: c.name, value: c.value, domain: c.domain.replace(/^\./, ""), path: c.path }));
	40	+ }
	41	+
	42	+ /** Store every Set-Cookie header value received from `url`. Accepts the flattened ", " joined form too. */
	43	+ storeFromHeaders(setCookie: string \| string[] \| undefined, url: URL): void {
	44	+ if (!setCookie) return;
	45	+ const values = Array.isArray(setCookie) ? setCookie : splitSetCookie(setCookie);
	46	+ for (const v of values) this.store(v, url);
	47	+ }
	48	+
	49	+ store(setCookieValue: string, url: URL): void {
	50	+ const segs = setCookieValue.split(";").map((s) => s.trim());
	51	+ const nv = segs.shift();
	52	+ if (!nv) return;
	53	+ const eq = nv.indexOf("=");
	54	+ if (eq <= 0) return;
	55	+ const name = nv.slice(0, eq).trim();
	56	+ const value = nv.slice(eq + 1).trim();
	57	+ const host = url.hostname.toLowerCase();
	58	+ let domain = host;
	59	+ let hostOnly = true;
	60	+ let path = defaultPath(url.pathname);
	61	+ let secure = false;
	62	+ let expires: number \| null = null;
	63	+ for (const a of segs) {
	64	+ const [k0, ...rest] = a.split("=");
	65	+ const k = (k0 ?? "").trim().toLowerCase();
	66	+ const val = rest.join("=").trim();
	67	+ if (k === "domain" && val) {
	68	+ const d = val.toLowerCase().replace(/^\./, "");
	69	+ // reject cookies for unrelated domains (and public suffix-ish single labels)
	70	+ if (d && (host === d \|\| host.endsWith(`.${d}`)) && d.includes(".")) {
	71	+ domain = d;
	72	+ hostOnly = false;
	73	+ }
	74	+ } else if (k === "path" && val.startsWith("/")) path = val;
	75	+ else if (k === "secure") secure = true;
	76	+ else if (k === "max-age") {
	77	+ const n = Number(val);
	78	+ if (Number.isFinite(n)) expires = n <= 0 ? 0 : Date.now() + n * 1000;
	79	+ } else if (k === "expires" && expires === null) {
	80	+ const t = Date.parse(val);
	81	+ if (!Number.isNaN(t)) expires = t;
	82	+ }
	83	+ }
	84	+ // remove existing
	85	+ this.cookies = this.cookies.filter((c) => !(c.name === name && c.domain === domain && c.path === path));
	86	+ if (expires !== null && expires <= Date.now()) return; // deletion
	87	+ this.cookies.push({ name, value, domain, hostOnly, path, secure, expires });
	88	+ if (this.cookies.length > 300) this.cookies.splice(0, this.cookies.length - 300);
	89	+ }
	90	+
	91	+ /** Cookie header value for `url` (or null). */
	92	+ headerFor(url: URL): string \| null {
	93	+ const host = url.hostname.toLowerCase();
	94	+ const path = url.pathname \|\| "/";
	95	+ const secure = url.protocol === "https:";
	96	+ const now = Date.now();
	97	+ const matches = this.cookies.filter((c) => {
	98	+ if (c.expires !== null && c.expires <= now) return false;
	99	+ if (c.secure && !secure) return false;
	100	+ if (c.hostOnly ? host !== c.domain : !(host === c.domain \|\| host.endsWith(`.${c.domain}`))) return false;
	101	+ return path === c.path \|\| (path.startsWith(c.path) && (c.path.endsWith("/") \|\| path[c.path.length] === "/"));
	102	+ });
	103	+ if (!matches.length) return null;
	104	+ matches.sort((a, b) => b.path.length - a.path.length);
	105	+ return matches.map((c) => `${c.name}=${c.value}`).join("; ");
	106	+ }
	107	+
	108	+ /** Merge a raw caller-provided Cookie header (name=value; …) as host-only cookies for `url`. */
	109	+ addRawCookieHeader(header: string \| undefined, url: URL): void {
	110	+ if (!header) return;
	111	+ for (const part of header.split(";")) {
	112	+ const t = part.trim();
	113	+ const eq = t.indexOf("=");
	114	+ if (eq <= 0) continue;
	115	+ this.store(`${t.slice(0, eq)}=${t.slice(eq + 1)}; Path=/`, url);
	116	+ }
	117	+ }
	118	+}
	119	+
	120	+function defaultPath(p: string): string {
	121	+ if (!p \|\| !p.startsWith("/")) return "/";
	122	+ const i = p.lastIndexOf("/");
	123	+ return i <= 0 ? "/" : p.slice(0, i);
	124	+}
	125	+
	126	+/** Split a flattened Set-Cookie header (joined with ", ") on cookie boundaries (not on Expires dates). */
	127	+export function splitSetCookie(raw: string): string[] {
	128	+ return raw.split(/,(?=\s[A-Za-z0-9_\-!#$%&'+.^`\|~]+=)/).map((s) => s.trim()).filter(Boolean);
	129	+}

modified packages/providers/src/decodo.ts +6 −2

		`@@ -1,6 +1,6 @@`
1	1	import type { ConcreteNetwork, GeoTarget } from "@fetcha/core";
2		−import { executeHttp, type ProxyEndpoint } from "./http";
3		−import type { ProviderHealth, ProviderRequest, ProviderResponse, ProxyProvider } from "./types";
	2	+import { executeHttp } from "./http";
	3	+import type { ProviderHealth, ProviderRequest, ProviderResponse, ProxyEndpoint, ProxyProvider } from "./types";
4	4	import { probeHealth } from "./probe";
5	5
6	6	/**
		`@@ -54,6 +54,10 @@ export class DecodoProvider implements ProxyProvider {`
54	54	return { host: "gate.decodo.com", port: 7000, username: user, password: this.password };
55	55	}
56	56
	57	+ proxyEndpoint(req: Pick<ProviderRequest, "geo" \| "sessionKey" \| "sessionMinutes">): ProxyEndpoint \| null {
	58	+ return this.endpoint(req);
	59	+ }
	60	+
57	61	async fetch(request: ProviderRequest): Promise<ProviderResponse> {
58	62	return executeHttp(this.id, this.endpoint(request), request);
59	63	}

modified packages/providers/src/direct.ts +4 −0

		`@@ -34,6 +34,10 @@ export class DirectProvider implements ProxyProvider {`
34	34	return (bytes / 1_073_741_824) * this.pricePerGb();
35	35	}
36	36
	37	+ proxyEndpoint(): null {
	38	+ return null;
	39	+ }
	40	+
37	41	async fetch(request: ProviderRequest): Promise<ProviderResponse> {
38	42	return executeHttp(this.id, null, request);
39	43	}

added packages/providers/src/fingerprint.ts +369 −0

		`@@ -0,0 +1,369 @@`
	1	+/**
	2	+ * Browser fingerprint profiles used for plain HTTP fetches.
	3	+ *
	4	+ * A profile is a coherent set of request headers (in the order the real browser sends them), the
	5	+ * matching client-hints, and a TLS cipher/curve preference list approximating the browser's
	6	+ * ClientHello. Sticky sessions pick a profile deterministically from their key so the identity stays
	7	+ * stable across requests; otherwise a weighted-random profile is used and rotated on every retry.
	8	+ */
	9	+import type { ConcreteNetwork } from "@fetcha/core";
	10	+
	11	+export type ProfileDevice = "desktop" \| "mobile" \| "tablet";
	12	+export type TlsFamily = "chrome" \| "firefox" \| "safari";
	13	+
	14	+export interface FingerprintProfile {
	15	+ id: string;
	16	+ device: ProfileDevice;
	17	+ tls: TlsFamily;
	18	+ /** Weight for random selection (desktop Chrome dominates real traffic). */
	19	+ weight: number;
	20	+ /** Ordered header list; `accept-language` is filled from the request locale. */
	21	+ headers: Array<[string, string]>;
	22	+ /** Ordered navigational Sec-Fetch headers (Chromium & Firefox). */
	23	+ secFetch: boolean;
	24	+ userAgent: string;
	25	+ /** Playwright-facing attributes for the managed browser. */
	26	+ viewport: { width: number; height: number };
	27	+ platform: string;
	28	+}
	29	+
	30	+const CHROME_VER = "140";
	31	+const CHROME_FULL = "140.0.7339.128";
	32	+const FIREFOX_VER = "143";
	33	+const SAFARI_VER = "18.6";
	34	+
	35	+const chromeAccept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7";
	36	+const firefoxAccept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,/;q=0.8";
	37	+const safariAccept = "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8";
	38	+
	39	+function chromeDesktop(id: string, platformToken: string, uaPlatform: string, uaPlatformHint: string, weight: number, brandExtra = ""): FingerprintProfile {
	40	+ const ua = `Mozilla/5.0 (${uaPlatform}) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/${CHROME_FULL} Safari/537.36${brandExtra}`;
	41	+ const brands = brandExtra.includes("Edg")
	42	+ ? `"Chromium";v="${CHROME_VER}", "Microsoft Edge";v="${CHROME_VER}", "Not_A Brand";v="24"`
	43	+ : `"Chromium";v="${CHROME_VER}", "Google Chrome";v="${CHROME_VER}", "Not_A Brand";v="24"`;
	44	+ return {
	45	+ id,
	46	+ device: "desktop",
	47	+ tls: "chrome",
	48	+ weight,
	49	+ userAgent: ua,
	50	+ platform: platformToken,
	51	+ viewport: { width: 1920, height: 1080 },
	52	+ secFetch: true,
	53	+ headers: [
	54	+ ["sec-ch-ua", brands],
	55	+ ["sec-ch-ua-mobile", "?0"],
	56	+ ["sec-ch-ua-platform", `"${uaPlatformHint}"`],
	57	+ ["upgrade-insecure-requests", "1"],
	58	+ ["user-agent", ua],
	59	+ ["accept", chromeAccept],
	60	+ ["sec-fetch-site", "none"],
	61	+ ["sec-fetch-mode", "navigate"],
	62	+ ["sec-fetch-user", "?1"],
	63	+ ["sec-fetch-dest", "document"],
	64	+ ["accept-encoding", "gzip, deflate, br, zstd"],
	65	+ ["accept-language", "{lang}"],
	66	+ ["priority", "u=0, i"],
	67	+ ],
	68	+ };
	69	+}
	70	+
	71	+export const PROFILES: FingerprintProfile[] = [
	72	+ chromeDesktop("chrome-win", "Win32", "Windows NT 10.0; Win64; x64", "Windows", 40),
	73	+ chromeDesktop("chrome-mac", "MacIntel", "Macintosh; Intel Mac OS X 10_15_7", "macOS", 20),
	74	+ chromeDesktop("edge-win", "Win32", "Windows NT 10.0; Win64; x64", "Windows", 8, ` Edg/${CHROME_FULL}`),
	75	+ chromeDesktop("chrome-linux", "Linux x86_64", "X11; Linux x86_64", "Linux", 4),
	76	+ {
	77	+ id: "firefox-win",
	78	+ device: "desktop",
	79	+ tls: "firefox",
	80	+ weight: 8,
	81	+ platform: "Win32",
	82	+ viewport: { width: 1920, height: 1080 },
	83	+ secFetch: true,
	84	+ userAgent: `Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:${FIREFOX_VER}.0) Gecko/20100101 Firefox/${FIREFOX_VER}.0`,
	85	+ headers: [
	86	+ ["user-agent", `Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:${FIREFOX_VER}.0) Gecko/20100101 Firefox/${FIREFOX_VER}.0`],
	87	+ ["accept", firefoxAccept],
	88	+ ["accept-language", "{lang}"],
	89	+ ["accept-encoding", "gzip, deflate, br, zstd"],
	90	+ ["upgrade-insecure-requests", "1"],
	91	+ ["sec-fetch-dest", "document"],
	92	+ ["sec-fetch-mode", "navigate"],
	93	+ ["sec-fetch-site", "none"],
	94	+ ["sec-fetch-user", "?1"],
	95	+ ["priority", "u=0, i"],
	96	+ ["te", "trailers"],
	97	+ ],
	98	+ },
	99	+ {
	100	+ id: "safari-mac",
	101	+ device: "desktop",
	102	+ tls: "safari",
	103	+ weight: 8,
	104	+ platform: "MacIntel",
	105	+ viewport: { width: 1728, height: 1117 },
	106	+ secFetch: true,
	107	+ userAgent: `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/${SAFARI_VER} Safari/605.1.15`,
	108	+ headers: [
	109	+ ["accept", safariAccept],
	110	+ ["sec-fetch-site", "none"],
	111	+ ["accept-encoding", "gzip, deflate, br"],
	112	+ ["sec-fetch-mode", "navigate"],
	113	+ ["user-agent", `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/${SAFARI_VER} Safari/605.1.15`],
	114	+ ["accept-language", "{lang}"],
	115	+ ["sec-fetch-dest", "document"],
	116	+ ["priority", "u=0, i"],
	117	+ ],
	118	+ },
	119	+ {
	120	+ id: "safari-iphone",
	121	+ device: "mobile",
	122	+ tls: "safari",
	123	+ weight: 7,
	124	+ platform: "iPhone",
	125	+ viewport: { width: 393, height: 852 },
	126	+ secFetch: true,
	127	+ userAgent: `Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/${SAFARI_VER} Mobile/15E148 Safari/604.1`,
	128	+ headers: [
	129	+ ["accept", safariAccept],
	130	+ ["sec-fetch-site", "none"],
	131	+ ["accept-encoding", "gzip, deflate, br"],
	132	+ ["sec-fetch-mode", "navigate"],
	133	+ ["user-agent", `Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/${SAFARI_VER} Mobile/15E148 Safari/604.1`],
	134	+ ["accept-language", "{lang}"],
	135	+ ["sec-fetch-dest", "document"],
	136	+ ["priority", "u=0, i"],
	137	+ ],
	138	+ },
	139	+ {
	140	+ id: "chrome-android",
	141	+ device: "mobile",
	142	+ tls: "chrome",
	143	+ weight: 5,
	144	+ platform: "Linux armv81",
	145	+ viewport: { width: 412, height: 915 },
	146	+ secFetch: true,
	147	+ userAgent: `Mozilla/5.0 (Linux; Android 15; Pixel 9) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/${CHROME_FULL} Mobile Safari/537.36`,
	148	+ headers: [
	149	+ ["sec-ch-ua", `"Chromium";v="${CHROME_VER}", "Google Chrome";v="${CHROME_VER}", "Not_A Brand";v="24"`],
	150	+ ["sec-ch-ua-mobile", "?1"],
	151	+ ["sec-ch-ua-platform", '"Android"'],
	152	+ ["upgrade-insecure-requests", "1"],
	153	+ ["user-agent", `Mozilla/5.0 (Linux; Android 15; Pixel 9) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/${CHROME_FULL} Mobile Safari/537.36`],
	154	+ ["accept", chromeAccept],
	155	+ ["sec-fetch-site", "none"],
	156	+ ["sec-fetch-mode", "navigate"],
	157	+ ["sec-fetch-user", "?1"],
	158	+ ["sec-fetch-dest", "document"],
	159	+ ["accept-encoding", "gzip, deflate, br, zstd"],
	160	+ ["accept-language", "{lang}"],
	161	+ ["priority", "u=0, i"],
	162	+ ],
	163	+ },
	164	+ {
	165	+ id: "safari-ipad",
	166	+ device: "tablet",
	167	+ tls: "safari",
	168	+ weight: 2,
	169	+ platform: "MacIntel",
	170	+ viewport: { width: 1024, height: 1366 },
	171	+ secFetch: true,
	172	+ userAgent: `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/${SAFARI_VER} Safari/605.1.15`,
	173	+ headers: [
	174	+ ["accept", safariAccept],
	175	+ ["sec-fetch-site", "none"],
	176	+ ["accept-encoding", "gzip, deflate, br"],
	177	+ ["sec-fetch-mode", "navigate"],
	178	+ ["user-agent", `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/${SAFARI_VER} Safari/605.1.15`],
	179	+ ["accept-language", "{lang}"],
	180	+ ["sec-fetch-dest", "document"],
	181	+ ["priority", "u=0, i"],
	182	+ ],
	183	+ },
	184	+];
	185	+
	186	+/** Cipher suites in browser preference order (OpenSSL names). Node cannot reorder TLS extensions,
	187	+ * but matching the suite list and curves removes the most obvious "Node.js" JA3 tells. */
	188	+export const TLS_CIPHERS: Record<TlsFamily, string> = {
	189	+ chrome: [
	190	+ "TLS_AES_128_GCM_SHA256",
	191	+ "TLS_AES_256_GCM_SHA384",
	192	+ "TLS_CHACHA20_POLY1305_SHA256",
	193	+ "ECDHE-ECDSA-AES128-GCM-SHA256",
	194	+ "ECDHE-RSA-AES128-GCM-SHA256",
	195	+ "ECDHE-ECDSA-AES256-GCM-SHA384",
	196	+ "ECDHE-RSA-AES256-GCM-SHA384",
	197	+ "ECDHE-ECDSA-CHACHA20-POLY1305",
	198	+ "ECDHE-RSA-CHACHA20-POLY1305",
	199	+ "ECDHE-RSA-AES128-SHA",
	200	+ "ECDHE-RSA-AES256-SHA",
	201	+ "AES128-GCM-SHA256",
	202	+ "AES256-GCM-SHA384",
	203	+ "AES128-SHA",
	204	+ "AES256-SHA",
	205	+ ].join(":"),
	206	+ firefox: [
	207	+ "TLS_AES_128_GCM_SHA256",
	208	+ "TLS_CHACHA20_POLY1305_SHA256",
	209	+ "TLS_AES_256_GCM_SHA384",
	210	+ "ECDHE-ECDSA-AES128-GCM-SHA256",
	211	+ "ECDHE-RSA-AES128-GCM-SHA256",
	212	+ "ECDHE-ECDSA-CHACHA20-POLY1305",
	213	+ "ECDHE-RSA-CHACHA20-POLY1305",
	214	+ "ECDHE-ECDSA-AES256-GCM-SHA384",
	215	+ "ECDHE-RSA-AES256-GCM-SHA384",
	216	+ "ECDHE-ECDSA-AES256-SHA",
	217	+ "ECDHE-ECDSA-AES128-SHA",
	218	+ "ECDHE-RSA-AES128-SHA",
	219	+ "ECDHE-RSA-AES256-SHA",
	220	+ "AES128-GCM-SHA256",
	221	+ "AES256-GCM-SHA384",
	222	+ "AES128-SHA",
	223	+ "AES256-SHA",
	224	+ ].join(":"),
	225	+ safari: [
	226	+ "TLS_AES_128_GCM_SHA256",
	227	+ "TLS_AES_256_GCM_SHA384",
	228	+ "TLS_CHACHA20_POLY1305_SHA256",
	229	+ "ECDHE-ECDSA-AES256-GCM-SHA384",
	230	+ "ECDHE-ECDSA-AES128-GCM-SHA256",
	231	+ "ECDHE-ECDSA-CHACHA20-POLY1305",
	232	+ "ECDHE-RSA-AES256-GCM-SHA384",
	233	+ "ECDHE-RSA-AES128-GCM-SHA256",
	234	+ "ECDHE-RSA-CHACHA20-POLY1305",
	235	+ "ECDHE-ECDSA-AES256-SHA384",
	236	+ "ECDHE-ECDSA-AES128-SHA256",
	237	+ "ECDHE-RSA-AES256-SHA384",
	238	+ "ECDHE-RSA-AES128-SHA256",
	239	+ "ECDHE-ECDSA-AES256-SHA",
	240	+ "ECDHE-ECDSA-AES128-SHA",
	241	+ "ECDHE-RSA-AES256-SHA",
	242	+ "ECDHE-RSA-AES128-SHA",
	243	+ "AES256-GCM-SHA384",
	244	+ "AES128-GCM-SHA256",
	245	+ "AES256-SHA256",
	246	+ "AES128-SHA256",
	247	+ "AES256-SHA",
	248	+ "AES128-SHA",
	249	+ ].join(":"),
	250	+};
	251	+
	252	+export const TLS_CURVES: Record<TlsFamily, string> = {
	253	+ chrome: "X25519:P-256:P-384",
	254	+ firefox: "X25519:P-256:P-384:P-521",
	255	+ safari: "X25519:P-256:P-384:P-521",
	256	+};
	257	+
	258	+export const TLS_SIGALGS: Record<TlsFamily, string> = {
	259	+ chrome: "ecdsa_secp256r1_sha256:rsa_pss_rsae_sha256:rsa_pkcs1_sha256:ecdsa_secp384r1_sha384:rsa_pss_rsae_sha384:rsa_pkcs1_sha384:rsa_pss_rsae_sha512:rsa_pkcs1_sha512",
	260	+ firefox: "ecdsa_secp256r1_sha256:ecdsa_secp384r1_sha384:ecdsa_secp521r1_sha512:rsa_pss_rsae_sha256:rsa_pss_rsae_sha384:rsa_pss_rsae_sha512:rsa_pkcs1_sha256:rsa_pkcs1_sha384:rsa_pkcs1_sha512:ecdsa_sha1:rsa_pkcs1_sha1",
	261	+ safari: "ecdsa_secp256r1_sha256:rsa_pss_rsae_sha256:rsa_pkcs1_sha256:ecdsa_secp384r1_sha384:ecdsa_sha1:rsa_pss_rsae_sha384:rsa_pss_rsae_sha384:rsa_pkcs1_sha384:rsa_pss_rsae_sha512:rsa_pkcs1_sha512:rsa_pkcs1_sha1",
	262	+};
	263	+
	264	+function hash32(s: string): number {
	265	+ let h = 2166136261;
	266	+ for (let i = 0; i < s.length; i++) {
	267	+ h ^= s.charCodeAt(i);
	268	+ h = Math.imul(h, 16777619);
	269	+ }
	270	+ return h >>> 0;
	271	+}
	272	+
	273	+export interface ProfileSelection {
	274	+ device?: ProfileDevice \| null;
	275	+ /** Deterministic seed (sticky session key). */
	276	+ seed?: string \| null;
	277	+ /** Attempt number: rotates the profile on retries when there is no seed. */
	278	+ attempt?: number;
	279	+ /** Exclude a profile id (e.g. the one that just got blocked). */
	280	+ exclude?: string[];
	281	+}
	282	+
	283	+export function pickProfile(sel: ProfileSelection = {}): FingerprintProfile {
	284	+ let pool = PROFILES.filter((p) => (sel.device ? p.device === sel.device : p.device === "desktop"));
	285	+ if (sel.exclude?.length) {
	286	+ const filtered = pool.filter((p) => !sel.exclude!.includes(p.id));
	287	+ if (filtered.length) pool = filtered;
	288	+ }
	289	+ const total = pool.reduce((s, p) => s + p.weight, 0);
	290	+ let r: number;
	291	+ if (sel.seed) r = (hash32(`${sel.seed}:${sel.device ?? "desktop"}`) % 10_000) / 10_000;
	292	+ else r = Math.random();
	293	+ let acc = 0;
	294	+ for (const p of pool) {
	295	+ acc += p.weight / total;
	296	+ if (r <= acc) return p;
	297	+ }
	298	+ return pool[pool.length - 1]!;
	299	+}
	300	+
	301	+export function profileById(id: string): FingerprintProfile \| undefined {
	302	+ return PROFILES.find((p) => p.id === id);
	303	+}
	304	+
	305	+/** Accept-Language for a locale, with a realistic q-cascade. */
	306	+export function acceptLanguage(locale: string \| null \| undefined, country: string \| null \| undefined): string {
	307	+ if (locale) {
	308	+ const l = locale.trim();
	309	+ if (l.includes(",")) return l; // caller provided a full header value
	310	+ const base = l.split("-")[0]!;
	311	+ return base === l ? `${l},en;q=0.9` : `${l},${base};q=0.9,en;q=0.8`;
	312	+ }
	313	+ const byCountry: Record<string, string> = {
	314	+ CA: "en-CA,en;q=0.9,fr-CA;q=0.8,fr;q=0.7",
	315	+ US: "en-US,en;q=0.9",
	316	+ GB: "en-GB,en;q=0.9",
	317	+ AU: "en-AU,en;q=0.9",
	318	+ FR: "fr-FR,fr;q=0.9,en;q=0.8",
	319	+ DE: "de-DE,de;q=0.9,en;q=0.8",
	320	+ ES: "es-ES,es;q=0.9,en;q=0.8",
	321	+ IT: "it-IT,it;q=0.9,en;q=0.8",
	322	+ BR: "pt-BR,pt;q=0.9,en;q=0.8",
	323	+ PT: "pt-PT,pt;q=0.9,en;q=0.8",
	324	+ NL: "nl-NL,nl;q=0.9,en;q=0.8",
	325	+ JP: "ja-JP,ja;q=0.9,en;q=0.8",
	326	+ KR: "ko-KR,ko;q=0.9,en;q=0.8",
	327	+ MX: "es-MX,es;q=0.9,en;q=0.8",
	328	+ IN: "en-IN,en;q=0.9,hi;q=0.8",
	329	+ SE: "sv-SE,sv;q=0.9,en;q=0.8",
	330	+ PL: "pl-PL,pl;q=0.9,en;q=0.8",
	331	+ CH: "de-CH,de;q=0.9,fr;q=0.8,en;q=0.7",
	332	+ BE: "fr-BE,fr;q=0.9,nl;q=0.8,en;q=0.7",
	333	+ };
	334	+ return (country && byCountry[country.toUpperCase()]) \|\| "en-US,en;q=0.9";
	335	+}
	336	+
	337	+const SEARCH_REFERERS = ["https://www.google.com/", "https://www.bing.com/", "https://duckduckgo.com/", "https://www.google.ca/", "https://www.google.co.uk/"];
	338	+
	339	+/** Referer for a retry: a search engine landing referer looks like organic traffic. */
	340	+export function retryReferer(attempt: number, country?: string \| null): string {
	341	+ if (country === "CA") return attempt % 2 ? "https://www.google.ca/" : "https://www.google.com/";
	342	+ if (country === "GB") return "https://www.google.co.uk/";
	343	+ return SEARCH_REFERERS[attempt % SEARCH_REFERERS.length]!;
	344	+}
	345	+
	346	+/** Build the ordered header list for a request. Caller overrides win and keep the profile position when the name exists. */
	347	+export function buildHeaders(profile: FingerprintProfile, opts: { locale?: string \| null; country?: string \| null; referer?: string \| null; overrides?: Record<string, string>; hasBody?: boolean; contentType?: string \| null; network?: ConcreteNetwork }): Array<[string, string]> {
	348	+ const lang = acceptLanguage(opts.locale, opts.country);
	349	+ const overrides = new Map<string, string>();
	350	+ for (const [k, v] of Object.entries(opts.overrides ?? {})) overrides.set(k.toLowerCase(), v);
	351	+ const out: Array<[string, string]> = [];
	352	+ const placed = new Set<string>();
	353	+ for (const [k, v0] of profile.headers) {
	354	+ let v = v0 === "{lang}" ? lang : v0;
	355	+ if (overrides.has(k)) v = overrides.get(k)!;
	356	+ if (k === "sec-fetch-site" && opts.referer && !overrides.has(k)) v = "cross-site";
	357	+ out.push([k, v]);
	358	+ placed.add(k);
	359	+ }
	360	+ if (opts.referer && !placed.has("referer")) {
	361	+ // Chrome places referer right after the sec-fetch-* block; Safari/Firefox after accept-language.
	362	+ const idx = out.findIndex(([k]) => k === "accept-encoding");
	363	+ out.splice(idx === -1 ? out.length : idx, 0, ["referer", opts.referer]);
	364	+ placed.add("referer");
	365	+ }
	366	+ if (opts.hasBody && !placed.has("content-type") && !overrides.has("content-type")) out.push(["content-type", opts.contentType ?? "application/json"]);
	367	+ for (const [k, v] of overrides) if (!placed.has(k)) out.push([k, v]);
	368	+ return out;
	369	+}

modified packages/providers/src/http.ts +171 −82

		`@@ -1,90 +1,116 @@`
1	1	import { Agent, ProxyAgent, request, type Dispatcher } from "undici";
2		−import { ProviderError, type ProviderId, type ProviderRequest, type ProviderResponse, type ProviderTiming } from "./types";
	2	+import { CookieJar } from "./cookies";
	3	+import { TLS_CIPHERS, TLS_CURVES, TLS_SIGALGS, buildHeaders, pickProfile, type FingerprintProfile } from "./fingerprint";
	4	+import { ProviderError, type ProviderId, type ProviderRequest, type ProviderResponse, type ProviderTiming, type ProxyEndpoint } from "./types";
3	5
4		−export interface ProxyEndpoint {
5		− host: string;
6		− port: number;
7		− username: string;
8		− password: string;
	6	+export type { ProxyEndpoint } from "./types";
	7	+
	8	+const HTTP2_ENABLED = process.env.FETCHA_HTTP2 !== "0";
	9	+
	10	+const agentCache = new Map<string, { d: Dispatcher; lastUsed: number }>();
	11	+
	12	+interface DispatcherOpts {
	13	+ endpoint: ProxyEndpoint \| null;
	14	+ tls: FingerprintProfile["tls"];
	15	+ h2: boolean;
	16	+ timeoutMs: number;
9	17	}
10	18
11		−const agentCache = new Map<string, Dispatcher>();
	19	+function tlsOptions(tls: FingerprintProfile["tls"]) {
	20	+ return {
	21	+ ciphers: TLS_CIPHERS[tls],
	22	+ ecdhCurve: TLS_CURVES[tls],
	23	+ sigalgs: TLS_SIGALGS[tls],
	24	+ minVersion: "TLSv1.2" as const,
	25	+ maxVersion: "TLSv1.3" as const,
	26	+ honorCipherOrder: false,
	27	+ // Browsers never send the legacy renegotiation info in a way Node does by default; keep session reuse on.
	28	+ sessionTimeout: 300,
	29	+ };
	30	+}
12	31
13		−function dispatcherFor(endpoint: ProxyEndpoint \| null, timeoutMs: number): Dispatcher {
14		− const key = endpoint ? `${endpoint.host}:${endpoint.port}:${endpoint.username}` : "direct";
15		− let d = agentCache.get(key);
16		− if (d) return d;
	32	+function dispatcherFor(o: DispatcherOpts): Dispatcher {
	33	+ const key = `${o.endpoint ? `${o.endpoint.host}:${o.endpoint.port}:${o.endpoint.username}` : "direct"}\|${o.tls}\|${o.h2 ? "h2" : "h1"}`;
	34	+ const hit = agentCache.get(key);
	35	+ if (hit) {
	36	+ hit.lastUsed = Date.now();
	37	+ return hit.d;
	38	+ }
17	39	const common = {
18		− connect: { timeout: Math.min(timeoutMs, 20_000) },
19	40	headersTimeout: 120_000,
20	41	bodyTimeout: 120_000,
21	42	keepAliveTimeout: 15_000,
22		− connections: 256,
	43	+ connections: 128,
	44	+ allowH2: o.h2,
	45	+ // Browser-like ALPN and TLS parameters for the origin connection.
	46	+ connect: { timeout: Math.min(o.timeoutMs, 20_000), ...tlsOptions(o.tls), ALPNProtocols: o.h2 ? ["h2", "http/1.1"] : ["http/1.1"] },
23	47	};
24		− if (!endpoint) {
	48	+ let d: Dispatcher;
	49	+ if (!o.endpoint) {
25	50	d = new Agent(common);
26	51	} else {
27		− const token = "Basic " + Buffer.from(`${endpoint.username}:${endpoint.password}`).toString("base64");
	52	+ const token = "Basic " + Buffer.from(`${o.endpoint.username}:${o.endpoint.password}`).toString("base64");
28	53	d = new ProxyAgent({
29		− uri: `http://${endpoint.host}:${endpoint.port}`,
	54	+ uri: `http://${o.endpoint.host}:${o.endpoint.port}`,
30	55	token,
31	56	...common,
	57	+ // TLS parameters used for the tunnelled origin connection (through CONNECT).
	58	+ requestTls: { ...tlsOptions(o.tls), ALPNProtocols: o.h2 ? ["h2", "http/1.1"] : ["http/1.1"] },
32	59	});
33	60	}
34		− // Bound the cache: sticky sessions create many usernames.
35		− if (agentCache.size > 500) {
36		− const first = agentCache.keys().next().value;
37		− if (first) {
38		− agentCache.get(first)?.close().catch(() => {});
39		− agentCache.delete(first);
	61	+ // Bound the cache: sticky sessions create many usernames. Evict least recently used.
	62	+ if (agentCache.size >= 400) {
	63	+ let oldest: string \| null = null;
	64	+ let t = Infinity;
	65	+ for (const [k, v] of agentCache) if (v.lastUsed < t) (t = v.lastUsed), (oldest = k);
	66	+ if (oldest) {
	67	+ agentCache.get(oldest)?.d.close().catch(() => {});
	68	+ agentCache.delete(oldest);
40	69	}
41	70	}
42		− agentCache.set(key, d);
	71	+ agentCache.set(key, { d, lastUsed: Date.now() });
43	72	return d;
44	73	}
45	74
46		−const HOP_BY_HOP = new Set(["connection", "keep-alive", "proxy-authenticate", "proxy-authorization", "te", "trailer", "transfer-encoding", "upgrade", "host", "content-length"]);
47		−
48		−function defaultHeaders(url: URL, overrides: Record<string, string>): Record<string, string> {
49		− const base: Record<string, string> = {
50		− "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
51		− accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,/;q=0.8",
52		− "accept-language": "en-US,en;q=0.9",
53		− "accept-encoding": "gzip, deflate, br",
54		− "upgrade-insecure-requests": "1",
55		− "sec-fetch-dest": "document",
56		− "sec-fetch-mode": "navigate",
57		− "sec-fetch-site": "none",
58		− "sec-fetch-user": "?1",
59		− };
60		− const out: Record<string, string> = {};
61		− for (const [k, v] of Object.entries(base)) out[k] = v;
62		− for (const [k, v] of Object.entries(overrides)) {
63		− const lk = k.toLowerCase();
64		− if (HOP_BY_HOP.has(lk)) continue;
65		− out[lk] = v;
66		− }
67		− void url;
68		− return out;
	75	+const HOP_BY_HOP = new Set(["connection", "keep-alive", "proxy-authenticate", "proxy-authorization", "te", "trailer", "transfer-encoding", "upgrade", "host", "content-length", "cookie"]);
	76	+
	77	+/** Header names as browsers spell them in HTTP/1.1 (HTTP/2 lower-cases everything anyway). */
	78	+const SPECIAL_CASE: Record<string, string> = { "sec-ch-ua": "Sec-CH-UA", "sec-ch-ua-mobile": "Sec-CH-UA-Mobile", "sec-ch-ua-platform": "Sec-CH-UA-Platform", te: "TE", dnt: "DNT", "x-requested-with": "X-Requested-With" };
	79	+function titleCase(name: string): string {
	80	+ if (SPECIAL_CASE[name]) return SPECIAL_CASE[name]!;
	81	+ return name
	82	+ .split("-")
	83	+ .map((p) => (p ? p[0]!.toUpperCase() + p.slice(1) : p))
	84	+ .join("-");
69	85	}
70	86
71		−function flattenHeaders(h: Record<string, string \| string[] \| undefined>): Record<string, string> {
72		− const out: Record<string, string> = {};
	87	+function flattenHeaders(h: Record<string, string \| string[] \| undefined>): { flat: Record<string, string>; setCookie: string[] } {
	88	+ const flat: Record<string, string> = {};
	89	+ let setCookie: string[] = [];
73	90	for (const [k, v] of Object.entries(h)) {
74	91	if (v === undefined) continue;
75		− out[k.toLowerCase()] = Array.isArray(v) ? v.join(", ") : v;
	92	+ const lk = k.toLowerCase();
	93	+ if (lk === "set-cookie") {
	94	+ setCookie = Array.isArray(v) ? v : [v];
	95	+ flat[lk] = setCookie.join(", ");
	96	+ continue;
	97	+ }
	98	+ flat[lk] = Array.isArray(v) ? v.join(", ") : v;
76	99	}
77		− return out;
	100	+ return { flat, setCookie };
78	101	}
79	102
80	103	/**
81		− * Execute an HTTP request, optionally through an upstream proxy tunnel.
82		− * Handles redirects manually so every hop can be validated by the caller (SSRF).
	104	+ * Execute an HTTP request, optionally through an upstream proxy tunnel, emulating a real browser's
	105	+ * header order, client hints, TLS preferences and HTTP/2 usage. Redirects are followed manually so
	106	+ * every hop can be validated by the caller (SSRF) and cookies set along the way are replayed.
83	107	*/
84	108	export async function executeHttp(provider: ProviderId, endpoint: ProxyEndpoint \| null, req: ProviderRequest): Promise<ProviderResponse> {
85	109	const started = performance.now();
86	110	const deadline = started + req.timeoutMs;
87		− const dispatcher = dispatcherFor(endpoint, req.timeoutMs);
	111	+ const profile = req.profile ?? pickProfile({ device: null, seed: req.sessionKey ?? null });
	112	+ const jar = req.jar ?? new CookieJar();
	113	+ let h2 = HTTP2_ENABLED && req.http2 !== false;
88	114
89	115	let url = new URL(req.url);
90	116	let method = req.method;
		`@@ -92,14 +118,44 @@ export async function executeHttp(provider: ProviderId, endpoint: ProxyEndpoint`
92	118	let redirects = 0;
93	119	let bytesOut = 0;
94	120	let bytesIn = 0;
95		− let connectMs = 0;
96	121	let originMs = 0;
	122	+ let referer: string \| null = req.referer ?? null;
	123	+ let retriedH1 = false;
	124	+
	125	+ // Caller-supplied cookies (explicit header) become host-only jar cookies for the first URL.
	126	+ const overrideHeaders: Record<string, string> = {};
	127	+ for (const [k, v] of Object.entries(req.headers)) {
	128	+ const lk = k.toLowerCase();
	129	+ if (lk === "cookie") {
	130	+ jar.addRawCookieHeader(v, url);
	131	+ continue;
	132	+ }
	133	+ if (HOP_BY_HOP.has(lk)) continue;
	134	+ overrideHeaders[lk] = v;
	135	+ }
97	136
98	137	for (;;) {
99	138	const remaining = deadline - performance.now();
100	139	if (remaining <= 0) throw new ProviderError(provider, "timeout", "Timed out before the target responded.");
101		− const headers = defaultHeaders(url, req.headers);
102		− if (body !== undefined && !headers["content-type"]) headers["content-type"] = typeof body === "string" ? "application/json" : "application/octet-stream";
	140	+ const hasBody = body !== undefined && method !== "GET" && method !== "HEAD";
	141	+ const ordered = buildHeaders(profile, {
	142	+ locale: req.locale,
	143	+ country: req.geo.country,
	144	+ referer,
	145	+ overrides: overrideHeaders,
	146	+ hasBody,
	147	+ contentType: hasBody ? (typeof body === "string" ? "application/json" : "application/octet-stream") : null,
	148	+ });
	149	+ const cookie = jar.headerFor(url);
	150	+ if (cookie) {
	151	+ // Browsers send Cookie after Accept-Language / before Priority.
	152	+ const idx = ordered.findIndex(([k]) => k === "priority");
	153	+ ordered.splice(idx === -1 ? ordered.length : idx, 0, ["cookie", cookie]);
	154	+ }
	155	+ const headers: Record<string, string> = {};
	156	+ for (const [k, v] of ordered) headers[titleCase(k)] = v;
	157	+
	158	+ const dispatcher = dispatcherFor({ endpoint, tls: profile.tls, h2, timeoutMs: req.timeoutMs });
103	159	const ac = new AbortController();
104	160	const timer = setTimeout(() => ac.abort(), remaining);
105	161	const t0 = performance.now();
		`@@ -108,17 +164,25 @@ export async function executeHttp(provider: ProviderId, endpoint: ProxyEndpoint`
108	164	res = await request(url, {
109	165	method,
110	166	headers,
111		− body: method === "GET" \|\| method === "HEAD" ? undefined : body,
	167	+ body: hasBody ? body : undefined,
112	168	dispatcher,
113	169	signal: ac.signal,
114	170	});
115	171	} catch (e) {
116	172	clearTimeout(timer);
117		− throw classifyError(provider, e);
	173	+ const err = classifyError(provider, e);
	174	+ // Some origins/proxies mis-handle h2 through CONNECT tunnels: fall back to HTTP/1.1 once.
	175	+ if (h2 && !retriedH1 && isH2Failure(e) && deadline - performance.now() > 1000) {
	176	+ retriedH1 = true;
	177	+ h2 = false;
	178	+ continue;
	179	+ }
	180	+ throw err;
118	181	}
119	182	const tFirstByte = performance.now();
120		− bytesOut += approxRequestBytes(method, url, headers, body);
121		− const resHeaders = flattenHeaders(res.headers as Record<string, string \| string[] \| undefined>);
	183	+ bytesOut += approxRequestBytes(method, url, headers, hasBody ? body : undefined);
	184	+ const { flat: resHeaders, setCookie } = flattenHeaders(res.headers as Record<string, string \| string[] \| undefined>);
	185	+ jar.storeFromHeaders(setCookie.length ? setCookie : resHeaders["set-cookie"], url);
122	186
123	187	// Proxy-level auth/quota errors surface as 407 from the gateway before reaching the target.
124	188	if (endpoint && res.statusCode === 407) {
		`@@ -132,9 +196,12 @@ export async function executeHttp(provider: ProviderId, endpoint: ProxyEndpoint`
132	196	await res.body.dump().catch(() => {});
133	197	clearTimeout(timer);
134	198	redirects += 1;
	199	+ originMs += tFirstByte - t0;
135	200	if (redirects > req.maxRedirects) throw new ProviderError(provider, "redirect", `Exceeded ${req.maxRedirects} redirects.`);
136	201	const next = new URL(resHeaders["location"]!, url);
137	202	if (req.onRedirect) await req.onRedirect(next.toString());
	203	+ // Browsers send the previous URL as referer on redirects (same-origin: full URL; cross-origin: origin only).
	204	+ referer = next.origin === url.origin ? url.toString() : url.protocol === "https:" && next.protocol !== "https:" ? null : url.origin + "/";
138	205	url = next;
139	206	if (res.statusCode === 303 \|\| ((res.statusCode === 301 \|\| res.statusCode === 302) && method === "POST")) {
140	207	method = "GET";
		`@@ -165,7 +232,6 @@ export async function executeHttp(provider: ProviderId, endpoint: ProxyEndpoint`
165	232	const tEnd = performance.now();
166	233	const raw = Buffer.concat(chunks);
167	234	bytesIn += total + approxHeaderBytes(resHeaders);
168		− connectMs += 0;
169	235	originMs += tFirstByte - t0;
170	236
171	237	const decoded = await decodeBody(raw, resHeaders["content-encoding"]);
		`@@ -173,7 +239,7 @@ export async function executeHttp(provider: ProviderId, endpoint: ProxyEndpoint`
173	239	const totalMs = Math.round(tDecoded - started);
174	240	const timing: ProviderTiming = {
175	241	dns_ms: 0,
176		− proxy_connect_ms: Math.round(connectMs),
	242	+ proxy_connect_ms: 0,
177	243	tls_ms: 0,
178	244	origin_ms: Math.round(originMs),
179	245	processing_ms: Math.round(tDecoded - tEnd),
		`@@ -189,10 +255,19 @@ export async function executeHttp(provider: ProviderId, endpoint: ProxyEndpoint`
189	255	bytesIn,
190	256	bytesOut,
191	257	timing,
	258	+ profileId: profile.id,
	259	+ protocol: h2 ? "h2?" : "http/1.1",
192	260	};
193	261	}
194	262	}
195	263
	264	+function isH2Failure(e: unknown): boolean {
	265	+ const err = e as { code?: string; message?: string; cause?: { code?: string; message?: string } };
	266	+ const code = err.code ?? err.cause?.code ?? "";
	267	+ const msg = `${err.message ?? ""} ${err.cause?.message ?? ""}`;
	268	+ return /HTTP\/2\|h2\|ERR_HTTP2\|NGHTTP2\|GOAWAY\|RST_STREAM\|PROTOCOL_ERROR\|UND_ERR_INVALID_ARG/i.test(code + " " + msg) \|\| code === "ERR_HTTP2_ERROR" \|\| code === "ECONNRESET" && /h2\|http2/i.test(msg);
	269	+}
	270	+
196	271	function approxRequestBytes(method: string, url: URL, headers: Record<string, string>, body?: string \| Buffer): number {
197	272	let n = method.length + url.pathname.length + url.search.length + 12;
198	273	for (const [k, v] of Object.entries(headers)) n += k.length + v.length + 4;
		`@@ -210,36 +285,50 @@ async function decodeBody(raw: Buffer, encoding?: string): Promise<Buffer> {`
210	285	if (!encoding \|\| raw.length === 0) return raw;
211	286	const zlib = await import("node:zlib");
212	287	const { promisify } = await import("node:util");
213		− try {
214		− switch (encoding.toLowerCase().trim()) {
215		− case "gzip":
216		− case "x-gzip":
217		− return await promisify(zlib.gunzip)(raw);
218		− case "deflate":
219		− return await promisify(zlib.inflate)(raw).catch(() => promisify(zlib.inflateRaw)(raw));
220		− case "br":
221		− return await promisify(zlib.brotliDecompress)(raw);
222		− case "zstd":
223		− if (typeof (zlib as unknown as { zstdDecompress?: unknown }).zstdDecompress === "function") {
224		− return await promisify((zlib as unknown as { zstdDecompress: (b: Buffer, cb: (e: Error \| null, r: Buffer) => void) => void }).zstdDecompress)(raw);
	288	+ // Handle stacked encodings ("gzip, br") right-to-left.
	289	+ const encodings = encoding.split(",").map((s) => s.trim().toLowerCase()).filter(Boolean).reverse();
	290	+ let buf = raw;
	291	+ for (const enc of encodings) {
	292	+ try {
	293	+ switch (enc) {
	294	+ case "gzip":
	295	+ case "x-gzip":
	296	+ buf = await promisify(zlib.gunzip)(buf);
	297	+ break;
	298	+ case "deflate":
	299	+ buf = await promisify(zlib.inflate)(buf).catch(() => promisify(zlib.inflateRaw)(buf));
	300	+ break;
	301	+ case "br":
	302	+ buf = await promisify(zlib.brotliDecompress)(buf);
	303	+ break;
	304	+ case "zstd": {
	305	+ const z = zlib as unknown as { zstdDecompress?: (b: Buffer, cb: (e: Error \| null, r: Buffer) => void) => void };
	306	+ if (typeof z.zstdDecompress === "function") buf = await promisify(z.zstdDecompress)(buf);
	307	+ break;
225	308	}
226		− return raw;
227		− default:
228		− return raw;
	309	+ case "identity":
	310	+ break;
	311	+ default:
	312	+ return buf;
	313	+ }
	314	+ } catch {
	315	+ return buf;
229	316	}
230		− } catch {
231		− return raw;
232	317	}
	318	+ return buf;
233	319	}
234	320
235	321	function classifyError(provider: ProviderId, e: unknown): ProviderError {
236	322	const err = e as { name?: string; code?: string; message?: string; cause?: { code?: string; message?: string } };
237	323	const code = err.code ?? err.cause?.code ?? "";
238	324	const msg = err.message ?? err.cause?.message ?? String(e);
239		− if (err.name === "AbortError" \|\| code === "UND_ERR_ABORTED" \|\| code === "UND_ERR_HEADERS_TIMEOUT" \|\| code === "UND_ERR_BODY_TIMEOUT" \|\| code === "UND_ERR_CONNECT_TIMEOUT") {
	325	+ if (err.name === "AbortError" \|\| code === "UND_ERR_ABORTED" \|\| code === "UND_ERR_HEADERS_TIMEOUT" \|\| code === "UND_ERR_BODY_TIMEOUT" \|\| code === "UND_ERR_CONNECT_TIMEOUT" \|\| code === "ETIMEDOUT") {
240	326	return new ProviderError(provider, "timeout", msg, { cause: e });
241	327	}
242		− if (code === "ECONNREFUSED" \|\| code === "ENOTFOUND" \|\| code === "EAI_AGAIN" \|\| code === "ECONNRESET" \|\| code === "EPIPE" \|\| code === "UND_ERR_SOCKET") {
	328	+ if (/CERT_\|SSL\|TLS\|ERR_TLS\|EPROTO\|handshake/i.test(code + " " + msg)) {
	329	+ return new ProviderError(provider, "tls", msg, { cause: e });
	330	+ }
	331	+ if (code === "ECONNREFUSED" \|\| code === "ENOTFOUND" \|\| code === "EAI_AGAIN" \|\| code === "ECONNRESET" \|\| code === "EPIPE" \|\| code === "UND_ERR_SOCKET" \|\| code === "EHOSTUNREACH" \|\| code === "ENETUNREACH") {
243	332	return new ProviderError(provider, "connect", msg, { cause: e });
244	333	}
245	334	if (code === "UND_ERR_PRX_TLS" \|\| /proxy/i.test(msg)) {
		`@@ -249,6 +338,6 @@ function classifyError(provider: ProviderId, e: unknown): ProviderError {`
249	338	}
250	339
251	340	export async function closeAllDispatchers(): Promise<void> {
252		− await Promise.all([...agentCache.values()].map((d) => d.close().catch(() => {})));
	341	+ await Promise.all([...agentCache.values()].map((v) => v.d.close().catch(() => {})));
253	342	agentCache.clear();
254	343	}

modified packages/providers/src/index.ts +2 −0

		`@@ -11,6 +11,8 @@ export { DecodoProvider } from "./decodo";`
11	11	export { SoaxProvider } from "./soax";
12	12	export { DirectProvider } from "./direct";
13	13	export { closeAllDispatchers } from "./http";
	14	+export * from "./fingerprint";
	15	+export * from "./cookies";
14	16
15	17	export interface ProviderRegistryOptions {
16	18	env?: NodeJS.ProcessEnv;

modified packages/providers/src/oxylabs.ts +6 −2

		`@@ -1,6 +1,6 @@`
1	1	import type { ConcreteNetwork, GeoTarget } from "@fetcha/core";
2		−import { executeHttp, type ProxyEndpoint } from "./http";
3		−import type { ProviderHealth, ProviderRequest, ProviderResponse, ProxyProvider } from "./types";
	2	+import { executeHttp } from "./http";
	3	+import type { ProviderHealth, ProviderRequest, ProviderResponse, ProxyEndpoint, ProxyProvider } from "./types";
4	4	import { probeHealth } from "./probe";
5	5
6	6	/**
		`@@ -54,6 +54,10 @@ export class OxylabsProvider implements ProxyProvider {`
54	54	return { host: "pr.oxylabs.io", port: 7777, username: user, password: this.password };
55	55	}
56	56
	57	+ proxyEndpoint(req: Pick<ProviderRequest, "geo" \| "sessionKey" \| "sessionMinutes">): ProxyEndpoint \| null {
	58	+ return this.endpoint(req);
	59	+ }
	60	+
57	61	async fetch(request: ProviderRequest): Promise<ProviderResponse> {
58	62	return executeHttp(this.id, this.endpoint(request), request);
59	63	}

modified packages/providers/src/soax.ts +6 −2

		`@@ -1,6 +1,6 @@`
1	1	import type { ConcreteNetwork, GeoTarget } from "@fetcha/core";
2		−import { executeHttp, type ProxyEndpoint } from "./http";
3		−import type { ProviderHealth, ProviderRequest, ProviderResponse, ProxyProvider } from "./types";
	2	+import { executeHttp } from "./http";
	3	+import type { ProviderHealth, ProviderRequest, ProviderResponse, ProxyEndpoint, ProxyProvider } from "./types";
4	4	import { probeHealth } from "./probe";
5	5
6	6	/**
		`@@ -56,6 +56,10 @@ export class SoaxProvider implements ProxyProvider {`
56	56	return { host: "proxy.soax.com", port: 5000, username: user, password: this.password };
57	57	}
58	58
	59	+ proxyEndpoint(req: Pick<ProviderRequest, "geo" \| "sessionKey" \| "sessionMinutes">): ProxyEndpoint \| null {
	60	+ return this.endpoint(req);
	61	+ }
	62	+
59	63	async fetch(request: ProviderRequest): Promise<ProviderResponse> {
60	64	return executeHttp(this.id, this.endpoint(request), request);
61	65	}

modified packages/providers/src/types.ts +26 −1

		`@@ -1,7 +1,16 @@`
1	1	import type { ConcreteNetwork, GeoTarget, HttpMethod } from "@fetcha/core";
	2	+import type { CookieJar } from "./cookies";
	3	+import type { FingerprintProfile } from "./fingerprint";
2	4
3	5	export type ProviderId = "oxylabs" \| "decodo" \| "soax" \| "direct";
4	6
	7	+export interface ProxyEndpoint {
	8	+ host: string;
	9	+ port: number;
	10	+ username: string;
	11	+ password: string;
	12	+}
	13	+
5	14	export interface ProviderRequest {
6	15	requestId: string;
7	16	attemptId: string;
		`@@ -21,6 +30,16 @@ export interface ProviderRequest {`
21	30	maxResponseBytes: number;
22	31	/** Called for every redirect hop; must throw to abort. */
23	32	onRedirect?: (nextUrl: string) => Promise<void>;
	33	+ /** Browser fingerprint profile to emulate (headers, TLS). Picked by the executor. */
	34	+ profile?: FingerprintProfile;
	35	+ /** Accept-Language source. */
	36	+ locale?: string \| null;
	37	+ /** Referer to send on the first hop (null/undefined = none). */
	38	+ referer?: string \| null;
	39	+ /** Cookie jar shared across hops (and across requests for sticky sessions). */
	40	+ jar?: CookieJar;
	41	+ /** Prefer HTTP/2 when the origin supports it (default true). */
	42	+ http2?: boolean;
24	43	}
25	44
26	45	export interface ProviderTiming {
		`@@ -43,6 +62,10 @@ export interface ProviderResponse {`
43	62	timing: ProviderTiming;
44	63	/** Provider exit info when known (never exposed to customers). */
45	64	exit?: { ip?: string; country?: string };
	65	+ /** Fingerprint profile that was used. */
	66	+ profileId?: string;
	67	+ /** Negotiated protocol of the final hop when known ("h2" \| "http/1.1"). */
	68	+ protocol?: string;
46	69	}
47	70
48	71	export type ProviderHealthStatus = "healthy" \| "degraded" \| "down" \| "unconfigured";
		`@@ -58,7 +81,7 @@ export interface ProviderHealth {`
58	81
59	82	export class ProviderError extends Error {
60	83	readonly provider: ProviderId;
61		− readonly kind: "timeout" \| "connect" \| "auth" \| "proxy" \| "too_large" \| "redirect" \| "unknown";
	84	+ readonly kind: "timeout" \| "connect" \| "auth" \| "proxy" \| "too_large" \| "redirect" \| "tls" \| "unknown";
62	85	readonly status?: number;
63	86	constructor(provider: ProviderId, kind: ProviderError["kind"], message: string, opts: { status?: number; cause?: unknown } = {}) {
64	87	super(message, { cause: opts.cause });
		`@@ -85,4 +108,6 @@ export interface ProxyProvider {`
85	108	estimateCost(network: ConcreteNetwork, bytes: number): number;
86	109	/** Unit price in USD per GB for a network class. */
87	110	pricePerGb(network: ConcreteNetwork): number;
	111	+ /** Upstream proxy endpoint for a request (null = direct egress). Used by the managed browser. */
	112	+ proxyEndpoint(req: Pick<ProviderRequest, "geo" \| "sessionKey" \| "sessionMinutes">): ProxyEndpoint \| null;
88	113	}

modified packages/routing/package.json +1 −0

		`@@ -13,6 +13,7 @@`
13	13	"test": "vitest run --passWithNoTests"
14	14	},
15	15	"dependencies": {
	16	+ "@fetcha/browser": "workspace:*",
16	17	"@fetcha/core": "workspace:*",
17	18	"@fetcha/providers": "workspace:*"
18	19	},

modified packages/routing/src/engine.ts +3 −0

		`@@ -16,7 +16,10 @@ export interface DomainKnowledge {`
16	16	domain: string;
17	17	routeStats: RouteStats;
18	18	policy?: { order?: string[]; force_network?: string; browser?: boolean } \| null;
	19	+ /** Share of requests where the browser was needed after HTTP attempts were blocked. */
19	20	browserRequiredRate?: number;
	21	+ /** Number of requests behind `browserRequiredRate`. */
	22	+ browserSamples?: number;
20	23	}
21	24
22	25	export interface RoutingInput {

modified packages/routing/src/executor.ts +230 −61

		`@@ -2,22 +2,31 @@ import {`
2	2	FetchaError,
3	3	assertUrlAllowed,
4	4	extractDomain,
	5	+ extractPageMetadata,
	6	+ htmlToMainText,
	7	+ htmlToMarkdown,
5	8	htmlToText,
	9	+ isTransientStatus,
6	10	looksBlocked,
7	11	newId,
8	12	normalizeGeo,
9	13	scrubProviderText,
	14	+ type BlockVerdict,
10	15	type ConcreteNetwork,
11	16	type FetchRequest,
12	17	type FetchResponseBody,
13	18	type FetchTiming,
	19	+ type PageLink,
	20	+ type PageMetadata,
14	21	type Plan,
15	22	} from "@fetcha/core";
16		−import { ProviderError, type ProviderId, type ProviderResponse } from "@fetcha/providers";
	23	+import { CookieJar, ProviderError, pickProfile, retryReferer, type FingerprintProfile, type ProviderId, type ProviderResponse } from "@fetcha/providers";
	24	+import type { BrowserPool, RenderResult } from "@fetcha/browser";
17	25	import type { CircuitBreaker } from "./circuit";
18	26	import { routeKey, type DomainKnowledge, type RouteCandidate, type RoutingEngine } from "./engine";
19	27
20	28	export type AttemptOutcome = "success" \| "blocked" \| "timeout" \| "error" \| "provider_error" \| "too_large";
	29	+export type AttemptMode = "http" \| "browser";
21	30
22	31	/** Customer-facing aliases: upstream names are never revealed without provider visibility. */
23	32	export const PUBLIC_PROVIDER_ALIAS: Record<ProviderId, string> = {
		`@@ -32,6 +41,7 @@ export interface AttemptRecord {`
32	41	attemptNo: number;
33	42	provider: ProviderId;
34	43	network: ConcreteNetwork;
	44	+ mode: AttemptMode;
35	45	country: string \| null;
36	46	sessionKey: string \| null;
37	47	outcome: AttemptOutcome;
		`@@ -39,6 +49,8 @@ export interface AttemptRecord {`
39	49	errorCode: string \| null;
40	50	errorDetail: string \| null;
41	51	blockReason: string \| null;
	52	+ blockVendor: string \| null;
	53	+ profileId: string \| null;
42	54	durationMs: number;
43	55	bytesIn: number;
44	56	bytesOut: number;
		`@@ -48,6 +60,8 @@ export interface AttemptRecord {`
48	60	timing: FetchTiming \| null;
49	61	}
50	62
	63	+export type SerializedCookie = { name: string; value: string; domain: string; path: string; secure: boolean; expires: number \| null };
	64	+
51	65	export interface ExecutionContext {
52	66	requestId: string;
53	67	plan: Plan;
		`@@ -56,12 +70,18 @@ export interface ExecutionContext {`
56	70	sessionKey?: string \| null;
57	71	sessionProvider?: ProviderId \| null;
58	72	sessionNetwork?: ConcreteNetwork \| null;
	73	+ /** Cookies persisted on the session from earlier requests. */
	74	+ sessionCookies?: SerializedCookie[] \| null;
59	75	knowledge?: DomainKnowledge \| null;
60	76	/** Called after every attempt (success or failure) for persistence & metrics. */
61	77	onAttempt?: (a: AttemptRecord) => Promise<void> \| void;
62	78	/** Whether to include provider names in the debug payload. */
63	79	providerVisibility?: boolean;
64	80	maxResponseBytes?: number;
	81	+ /** Managed browser pool (null/undefined = browser mode unavailable). */
	82	+ browserPool?: BrowserPool \| null;
	83	+ /** Whether the plan allows browser rendering. */
	84	+ browserAllowed?: boolean;
65	85	}
66	86
67	87	export interface ExecutionResult {
		`@@ -69,15 +89,31 @@ export interface ExecutionResult {`
69	89	attempts: AttemptRecord[];
70	90	network: ConcreteNetwork \| null;
71	91	provider: ProviderId \| null;
	92	+ mode: AttemptMode;
72	93	costUsd: number;
73	94	bytesIn: number;
74	95	bytesOut: number;
75	96	finalUrl: string;
76	97	domain: string;
	98	+ /** Cookie jar state at the end of the request (for sticky-session persistence). */
	99	+ cookies: SerializedCookie[];
	100	+ /** True when the browser was needed after HTTP attempts were blocked (feeds domain intelligence). */
	101	+ browserRequired: boolean;
77	102	/** Raw response for optional storage. */
78	103	raw?: ProviderResponse;
79	104	}
80	105
	106	+interface FinalResponse {
	107	+ status: number;
	108	+ headers: Record<string, string>;
	109	+ body: Buffer;
	110	+ finalUrl: string;
	111	+ timing: FetchTiming;
	112	+ screenshot?: Buffer;
	113	+}
	114	+
	115	+const HTML_CT = /text\/html\|application\/xhtml/i;
	116	+
81	117	export class FetchExecutor {
82	118	constructor(
83	119	private readonly engine: RoutingEngine,
		`@@ -88,8 +124,9 @@ export class FetchExecutor {`
88	124	const { request, requestId } = ctx;
89	125	const started = performance.now();
90	126	const domain = extractDomain(request.url);
	127	+ const browserAvailable = Boolean(ctx.browserPool?.enabled) && ctx.browserAllowed !== false;
91	128
92		− if (request.browser) {
	129	+ if (request.browser && !browserAvailable) {
93	130	throw new FetchaError("BROWSER_UNAVAILABLE", undefined, { requestId });
94	131	}
95	132
		`@@ -121,97 +158,184 @@ export class FetchExecutor {`
121	158	throw new FetchaError("PROVIDER_UNAVAILABLE", undefined, { requestId });
122	159	}
123	160
	161	+ // Domain intelligence: sites that consistently need the browser go straight there.
	162	+ const knowledgeSaysBrowser = Boolean(ctx.knowledge && (ctx.knowledge.browserRequiredRate ?? 0) >= 0.5 && (ctx.knowledge.browserSamples ?? 0) >= 3);
	163	+ let browserMode = request.browser \|\| (browserAvailable && request.browser_fallback && knowledgeSaysBrowser);
	164	+ let browserEscalated = false;
	165	+
124	166	const attempts: AttemptRecord[] = [];
125	167	let lastError: FetchaError \| null = null;
126		− let lastBlocked: { res: ProviderResponse; cand: RouteCandidate } \| null = null;
	168	+ let lastBlocked: { res: FinalResponse; cand: RouteCandidate; mode: AttemptMode; verdict: BlockVerdict } \| null = null;
	169	+ let lastTransient: { res: FinalResponse; cand: RouteCandidate; mode: AttemptMode } \| null = null;
127	170	const maxBytes = ctx.maxResponseBytes ?? request.max_response_bytes ?? 20_000_000;
128	171	const deadline = started + request.timeout;
	172	+ const usedProfiles: string[] = [];
	173	+ const jar = CookieJar.fromSerialized(ctx.sessionCookies ?? null, allowed.hostname);
	174	+ if (request.cookies) for (const [k, v] of Object.entries(request.cookies)) jar.store(`${k}=${v}; Path=/`, allowed.url);
	175	+ let browserRequired = false;
	176	+ // The browser fallback is one extra attempt on top of the plan's HTTP budget.
	177	+ const maxAttempts = plan.maxAttempts + (browserAvailable && request.browser_fallback && !request.browser ? 1 : 0);
129	178
130		− for (let i = 0; i < candidates.length && i < plan.maxAttempts; i++) {
131		− const cand = candidates[i]!;
	179	+ for (let i = 0; i < maxAttempts; i++) {
	180	+ const cand = candidates[Math.min(i, candidates.length - 1)]!;
132	181	const remaining = deadline - performance.now();
133		− if (remaining < 500) {
	182	+ if (remaining < 800) {
134	183	lastError = new FetchaError("TARGET_TIMEOUT", undefined, { requestId });
135	184	break;
136	185	}
	186	+ if (i > 0) await this.backoff(i, lastBlocked?.verdict.retryAfterMs, deadline);
	187	+
137	188	const attemptId = newId("att");
138	189	const key = routeKey(cand.provider.id, cand.network);
139	190	const t0 = performance.now();
140		− // Retry with a fresh identity unless the caller pinned a session.
141	191	const sessionKey = ctx.sessionKey ?? null;
	192	+ const profile: FingerprintProfile = pickProfile({ device: request.device ?? null, seed: sessionKey, attempt: i, exclude: sessionKey ? [] : usedProfiles });
	193	+ usedProfiles.push(profile.id);
142	194	const headers = { ...(request.headers ?? {}) };
143		− if (request.cookies && Object.keys(request.cookies).length) {
144		− headers["cookie"] = [headers["cookie"], Object.entries(request.cookies).map(([k, v]) => `${k}=${v}`).join("; ")].filter(Boolean).join("; ");
145		− }
146		− if (request.locale) headers["accept-language"] = request.locale;
147		− if (request.device === "mobile" && !headers["user-agent"]) {
148		− headers["user-agent"] = "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1";
149		− }
	195	+ if (request.device === "mobile" && headers["user-agent"]) delete headers["user-agent"]; // profile already mobile
	196	+ const referer = request.referer === "none" ? null : request.referer === "auto" ? (i === 0 && !browserEscalated ? null : retryReferer(i, geo.country)) : request.referer;
150	197	const body = request.body === undefined ? undefined : typeof request.body === "string" ? request.body : JSON.stringify(request.body);
	198	+ const mode: AttemptMode = browserMode ? "browser" : "http";
	199	+ const timeoutMs = Math.max(1000, Math.min(deadline - performance.now(), request.timeout));
151	200
152		− let res: ProviderResponse \| null = null;
	201	+ let res: FinalResponse \| null = null;
	202	+ let verdict: BlockVerdict = { blocked: false };
153	203	let record: AttemptRecord;
154	204	try {
155		− res = await cand.provider.fetch({
156		− requestId,
157		− attemptId,
158		− url: allowed.url.toString(),
159		− method: request.method,
160		− headers,
161		− body,
162		− timeoutMs: Math.max(1000, Math.min(remaining, request.timeout)),
163		− network: cand.network,
164		− geo,
165		− sessionKey,
166		− sessionMinutes: 10,
167		− followRedirects: request.follow_redirects,
168		− maxRedirects: request.max_redirects,
169		− maxResponseBytes: maxBytes,
170		− onRedirect: async (next) => {
171		− await assertUrlAllowed(next);
172		− },
173		− });
	205	+ let bytesIn = 0;
	206	+ let bytesOut = 0;
	207	+ if (mode === "browser") {
	208	+ const render: RenderResult = await ctx.browserPool!.render({
	209	+ url: allowed.url.toString(),
	210	+ method: request.method,
	211	+ body,
	212	+ proxy: cand.provider.proxyEndpoint({ geo, sessionKey, sessionMinutes: 10 }),
	213	+ profile,
	214	+ locale: request.locale ?? null,
	215	+ country: geo.country,
	216	+ headers,
	217	+ cookies: jar.serialize(),
	218	+ referer,
	219	+ timeoutMs,
	220	+ waitUntil: request.wait_until,
	221	+ waitFor: request.wait_for ?? null,
	222	+ waitMs: request.wait_ms ?? null,
	223	+ javascript: request.javascript !== false,
	224	+ blockResources: request.block_resources,
	225	+ screenshot: request.screenshot,
	226	+ maxResponseBytes: maxBytes,
	227	+ onRedirect: async (next) => {
	228	+ await assertUrlAllowed(next);
	229	+ },
	230	+ });
	231	+ for (const c of render.cookies) jar.store(`${c.name}=${c.value}; Domain=${c.domain}; Path=${c.path}${c.secure ? "; Secure" : ""}`, new URL(render.finalUrl));
	232	+ bytesIn = render.bytesIn;
	233	+ bytesOut = render.bytesOut;
	234	+ res = {
	235	+ status: render.status,
	236	+ headers: render.headers,
	237	+ body: render.body,
	238	+ finalUrl: render.finalUrl,
	239	+ screenshot: render.screenshot,
	240	+ timing: { dns_ms: allowed.dns_ms, proxy_connect_ms: 0, tls_ms: 0, origin_ms: render.timing.navigation_ms, processing_ms: render.timing.challenge_ms + render.timing.settle_ms + render.timing.capture_ms, total_ms: render.timing.total_ms },
	241	+ };
	242	+ verdict = render.block;
	243	+ if (render.challengeSolved) browserRequired = true;
	244	+ } else {
	245	+ const pr = await cand.provider.fetch({
	246	+ requestId,
	247	+ attemptId,
	248	+ url: allowed.url.toString(),
	249	+ method: request.method,
	250	+ headers,
	251	+ body,
	252	+ timeoutMs,
	253	+ network: cand.network,
	254	+ geo,
	255	+ sessionKey,
	256	+ sessionMinutes: 10,
	257	+ followRedirects: request.follow_redirects,
	258	+ maxRedirects: request.max_redirects,
	259	+ maxResponseBytes: maxBytes,
	260	+ profile,
	261	+ locale: request.locale ?? null,
	262	+ referer,
	263	+ jar,
	264	+ onRedirect: async (next) => {
	265	+ await assertUrlAllowed(next);
	266	+ },
	267	+ });
	268	+ bytesIn = pr.bytesIn;
	269	+ bytesOut = pr.bytesOut;
	270	+ res = { status: pr.status, headers: pr.headers, body: pr.body, finalUrl: pr.finalUrl, timing: { ...pr.timing, dns_ms: allowed.dns_ms } };
	271	+ const ct = pr.headers["content-type"] ?? "";
	272	+ verdict = looksBlocked(pr.status, !ct \|\| HTML_CT.test(ct) \|\| /json\|xml\|text/i.test(ct) ? pr.body.subarray(0, 40_000).toString("utf8") : "", pr.headers);
	273	+ }
174	274	const durationMs = Math.round(performance.now() - t0);
175		− const text = res.body.toString("utf8");
176		− const block = looksBlocked(res.status, text, res.headers);
177		− const cost = cand.provider.estimateCost(cand.network, res.bytesIn + res.bytesOut);
	275	+ const transient = !verdict.blocked && isTransientStatus(res.status);
	276	+ const cost = cand.provider.estimateCost(cand.network, bytesIn + bytesOut);
178	277	record = {
179	278	attemptId,
180	279	attemptNo: i + 1,
181	280	provider: cand.provider.id,
182	281	network: cand.network,
	282	+ mode,
183	283	country: geo.country,
184	284	sessionKey,
185		− outcome: block.blocked ? "blocked" : "success",
	285	+ outcome: verdict.blocked ? "blocked" : transient ? "error" : "success",
186	286	httpStatus: res.status,
187		− errorCode: block.blocked ? "TARGET_BLOCKED" : null,
	287	+ errorCode: verdict.blocked ? "TARGET_BLOCKED" : transient ? "TARGET_UNAVAILABLE" : null,
188	288	errorDetail: null,
189		− blockReason: block.reason ?? null,
	289	+ blockReason: verdict.reason ?? null,
	290	+ blockVendor: verdict.vendor ?? null,
	291	+ profileId: profile.id,
190	292	durationMs,
191		− bytesIn: res.bytesIn,
192		− bytesOut: res.bytesOut,
	293	+ bytesIn,
	294	+ bytesOut,
193	295	unitPricePerGb: cand.provider.pricePerGb(cand.network),
194	296	costUsd: cost,
195	297	routingScore: cand.score,
196		− timing: { ...res.timing, dns_ms: allowed.dns_ms },
	298	+ timing: res.timing,
197	299	};
198	300	attempts.push(record);
199	301	await ctx.onAttempt?.(record);
200		− this.circuit.record(key, res.status < 500 && !(block.blocked && block.reason?.startsWith("http_407")));
	302	+ // Provider health: only proxy-side failures count against the route.
	303	+ this.circuit.record(key, !(verdict.blocked && verdict.reason === "http_407"));
201	304
202		− if (!block.blocked) {
203		− return this.finish(ctx, res, cand, attempts, started, domain, allowed.dns_ms);
	305	+ if (!verdict.blocked && !transient) {
	306	+ if (mode === "browser" && browserEscalated) browserRequired = true;
	307	+ return this.finish(ctx, res, cand, mode, attempts, started, domain, jar, browserRequired);
204	308	}
205		− lastBlocked = { res, cand };
206		− continue; // escalate
	309	+ if (transient) {
	310	+ lastTransient = { res, cand, mode };
	311	+ continue;
	312	+ }
	313	+ lastBlocked = { res, cand, mode, verdict };
	314	+ // Escalate to the browser after an HTTP block that a real browser can typically pass.
	315	+ if (mode === "http" && browserAvailable && request.browser_fallback && (verdict.challenge \|\| i >= plan.maxAttempts - 1 \|\| knowledgeSaysBrowser)) {
	316	+ browserMode = true;
	317	+ browserEscalated = true;
	318	+ }
	319	+ continue;
207	320	} catch (e) {
208	321	const durationMs = Math.round(performance.now() - t0);
209	322	if (e instanceof FetchaError) {
210		− // SSRF on redirect or similar policy violation: do not retry.
211		− record = this.errorRecord(attemptId, i, cand, geo.country, sessionKey, "error", e.code, e.message, durationMs);
	323	+ if (e.code === "URL_NOT_ALLOWED" \|\| e.code === "RESPONSE_TOO_LARGE" \|\| e.code === "TOO_MANY_REDIRECTS" \|\| e.code === "INVALID_REQUEST") {
	324	+ // Policy violation or non-retryable: do not retry.
	325	+ record = this.errorRecord(attemptId, i, cand, mode, profile.id, geo.country, sessionKey, "error", e.code, e.message, durationMs);
	326	+ attempts.push(record);
	327	+ await ctx.onAttempt?.(record);
	328	+ throw e;
	329	+ }
	330	+ // Browser-side failure (timeout / unavailable): record and let the loop continue (HTTP or another route).
	331	+ const outcome: AttemptOutcome = e.code === "BROWSER_TIMEOUT" \|\| e.code === "TARGET_TIMEOUT" ? "timeout" : e.code === "BROWSER_UNAVAILABLE" ? "provider_error" : "error";
	332	+ record = this.errorRecord(attemptId, i, cand, mode, profile.id, geo.country, sessionKey, outcome, e.code, e.message, durationMs);
212	333	attempts.push(record);
213	334	await ctx.onAttempt?.(record);
214		− throw e;
	335	+ lastError = new FetchaError(e.code, e.message, { requestId });
	336	+ if (e.code === "BROWSER_UNAVAILABLE" && !request.browser) browserMode = false; // fall back to HTTP for the remaining budget
	337	+ if (request.browser && e.code === "BROWSER_UNAVAILABLE") throw lastError;
	338	+ continue;
215	339	}
216	340	const pe = e instanceof ProviderError ? e : new ProviderError(cand.provider.id, "unknown", (e as Error).message ?? String(e), { cause: e });
217	341	let outcome: AttemptOutcome = "provider_error";
		`@@ -225,17 +349,22 @@ export class FetchExecutor {`
225	349	} else if (pe.kind === "redirect") {
226	350	outcome = "error";
227	351	code = "TOO_MANY_REDIRECTS";
228		− } else if (pe.kind === "connect") {
	352	+ } else if (pe.kind === "connect" \|\| pe.kind === "tls") {
229	353	outcome = "error";
230	354	code = "TARGET_UNAVAILABLE";
231	355	}
232		− record = this.errorRecord(attemptId, i, cand, geo.country, sessionKey, outcome, code, pe.message, durationMs);
	356	+ record = this.errorRecord(attemptId, i, cand, mode, profile.id, geo.country, sessionKey, outcome, code, pe.message, durationMs);
233	357	attempts.push(record);
234	358	await ctx.onAttempt?.(record);
235	359	// Provider-side failures count against the circuit; target-side ones don't.
236		− this.circuit.record(key, !(pe.kind === "auth" \|\| pe.kind === "proxy" \|\| pe.kind === "connect"));
	360	+ this.circuit.record(key, !(pe.kind === "auth" \|\| pe.kind === "proxy"));
237	361	lastError = new FetchaError(code as FetchaError["code"], undefined, { requestId });
238	362	if (pe.kind === "too_large" \|\| pe.kind === "redirect") throw lastError;
	363	+ // A TLS failure through a proxy is often the route, not the site: escalate to the browser when we can.
	364	+ if (pe.kind === "tls" && browserAvailable && request.browser_fallback && !browserMode) {
	365	+ browserMode = true;
	366	+ browserEscalated = true;
	367	+ }
239	368	continue;
240	369	}
241	370	}
		`@@ -243,17 +372,32 @@ export class FetchExecutor {`
243	372	// Exhausted: if the last route returned a blocked page, surface that page (customer still
244	373	// gets the status) but mark success=false with TARGET_BLOCKED.
245	374	if (lastBlocked) {
246		− const result = this.finish(ctx, lastBlocked.res, lastBlocked.cand, attempts, started, domain, allowed.dns_ms);
	375	+ const result = this.finish(ctx, lastBlocked.res, lastBlocked.cand, lastBlocked.mode, attempts, started, domain, jar, browserRequired);
	376	+ result.body.success = false;
	377	+ return result;
	378	+ }
	379	+ if (lastTransient) {
	380	+ const result = this.finish(ctx, lastTransient.res, lastTransient.cand, lastTransient.mode, attempts, started, domain, jar, browserRequired);
247	381	result.body.success = false;
248	382	return result;
249	383	}
250	384	throw lastError ?? new FetchaError("TARGET_UNAVAILABLE", undefined, { requestId });
251	385	}
252	386
	387	+ /** Jittered backoff between attempts; honours short Retry-After hints. */
	388	+ private async backoff(attempt: number, retryAfterMs: number \| undefined, deadline: number): Promise<void> {
	389	+ let wait = 250 + Math.random() * 650 + Math.min(attempt, 3) * 200;
	390	+ if (retryAfterMs && retryAfterMs > 0) wait = Math.max(wait, Math.min(retryAfterMs, 6000));
	391	+ wait = Math.min(wait, Math.max(0, deadline - performance.now() - 1500));
	392	+ if (wait > 0) await new Promise((r) => setTimeout(r, wait));
	393	+ }
	394	+
253	395	private errorRecord(
254	396	attemptId: string,
255	397	i: number,
256	398	cand: RouteCandidate,
	399	+ mode: AttemptMode,
	400	+ profileId: string,
257	401	country: string \| null,
258	402	sessionKey: string \| null,
259	403	outcome: AttemptOutcome,
		`@@ -266,6 +410,7 @@ export class FetchExecutor {`
266	410	attemptNo: i + 1,
267	411	provider: cand.provider.id,
268	412	network: cand.network,
	413	+ mode,
269	414	country,
270	415	sessionKey,
271	416	outcome,
		`@@ -273,6 +418,8 @@ export class FetchExecutor {`
273	418	errorCode: code,
274	419	errorDetail: detail.slice(0, 500),
275	420	blockReason: null,
	421	+ blockVendor: null,
	422	+ profileId,
276	423	durationMs,
277	424	bytesIn: 0,
278	425	bytesOut: 0,
		`@@ -283,16 +430,30 @@ export class FetchExecutor {`
283	430	};
284	431	}
285	432
286		− private finish(ctx: ExecutionContext, res: ProviderResponse, cand: RouteCandidate, attempts: AttemptRecord[], started: number, domain: string, dnsMs: number): ExecutionResult {
	433	+ private finish(ctx: ExecutionContext, res: FinalResponse, cand: RouteCandidate, mode: AttemptMode, attempts: AttemptRecord[], started: number, domain: string, jar: CookieJar, browserRequired: boolean): ExecutionResult {
287	434	const { request, requestId } = ctx;
288	435	const contentType = res.headers["content-type"] ?? null;
289	436	const isText = !contentType \|\| /text\/\|json\|xml\|javascript\|x-www-form-urlencoded/i.test(contentType);
	437	+ const isHtml = isText && (!contentType \|\| HTML_CT.test(contentType) \|\| (/^\s*<(!doctype\|html)/i.test(res.body.subarray(0, 200).toString("utf8")) && !/json\|xml\|javascript/i.test(contentType ?? "")));
290	438	const content = isText ? res.body.toString("utf8") : res.body.toString("base64");
291	439	const tProc = performance.now();
292	440
293	441	let text: string \| null \| undefined;
	442	+ let markdown: string \| null \| undefined;
294	443	let json: unknown;
295		− if (request.format === "text") text = isText ? htmlToText(content) : null;
	444	+ let page: PageMetadata \| null = null;
	445	+ let links: PageLink[] \| undefined;
	446	+ if (isHtml) {
	447	+ try {
	448	+ const meta = extractPageMetadata(content, res.finalUrl);
	449	+ page = meta.page;
	450	+ if (request.links) links = meta.links;
	451	+ } catch {
	452	+ page = null;
	453	+ }
	454	+ }
	455	+ if (request.format === "text") text = isText ? (isHtml ? htmlToMainText(content) \|\| htmlToText(content) : content) : null;
	456	+ if (request.format === "markdown") markdown = isText ? (isHtml ? htmlToMarkdown(content, { baseUrl: res.finalUrl }) : content) : null;
296	457	if (request.format === "json") {
297	458	try {
298	459	json = JSON.parse(content);
		`@@ -300,14 +461,12 @@ export class FetchExecutor {`
300	461	json = undefined;
301	462	}
302	463	}
303		− const cookies = parseSetCookies(res.headers["set-cookie"]);
304	464	const durationMs = Math.round(performance.now() - started);
305	465	const bytesIn = attempts.reduce((s, a) => s + a.bytesIn, 0);
306	466	const bytesOut = attempts.reduce((s, a) => s + a.bytesOut, 0);
307	467	const costUsd = attempts.reduce((s, a) => s + a.costUsd, 0);
308	468	const timing: FetchTiming = {
309	469	...res.timing,
310		− dns_ms: dnsMs,
311	470	processing_ms: res.timing.processing_ms + Math.round(performance.now() - tProc),
312	471	total_ms: durationMs,
313	472	};
		`@@ -320,10 +479,12 @@ export class FetchExecutor {`
320	479	content: request.format === "raw" \|\| request.format === "html" \|\| request.format === "json" ? content : null,
321	480	content_type: contentType,
322	481	headers: res.headers,
323		− cookies,
	482	+ cookies: jar.publicList(),
	483	+ page,
324	484	metadata: {
325	485	network: cand.network,
326	486	country: attempts.at(-1)?.country ?? null,
	487	+ mode,
327	488	attempts: attempts.length,
328	489	duration_ms: durationMs,
329	490	bytes: bytesIn + bytesOut,
		`@@ -333,14 +494,19 @@ export class FetchExecutor {`
333	494	},
334	495	};
335	496	if (request.format === "text") body.text = text ?? null;
	497	+ if (request.format === "markdown") body.markdown = markdown ?? null;
336	498	if (request.format === "json") body.json = json;
	499	+ if (links) body.links = links;
	500	+ if (request.screenshot && mode === "browser") body.screenshot = res.screenshot ? res.screenshot.toString("base64") : null;
337	501	if (request.debug) {
338	502	body.metadata.debug = {
339	503	attempts: attempts.map((a) => ({
340	504	provider: ctx.providerVisibility ? a.provider : PUBLIC_PROVIDER_ALIAS[a.provider],
341	505	network: a.network,
	506	+ mode: a.mode,
342	507	country: a.country,
343	508	outcome: a.outcome,
	509	+ block_reason: a.blockReason,
344	510	status: a.httpStatus,
345	511	duration_ms: a.durationMs,
346	512	...(a.errorDetail && ctx.providerVisibility ? { error: scrubProviderText(a.errorDetail) } : {}),
		`@@ -352,12 +518,15 @@ export class FetchExecutor {`
352	518	attempts,
353	519	network: cand.network,
354	520	provider: cand.provider.id,
	521	+ mode,
355	522	costUsd,
356	523	bytesIn,
357	524	bytesOut,
358	525	finalUrl: res.finalUrl,
359	526	domain,
360		− raw: res,
	527	+ cookies: jar.serialize(),
	528	+ browserRequired,
	529	+ raw: { status: res.status, headers: res.headers, body: res.body, finalUrl: res.finalUrl, redirects: 0, bytesIn, bytesOut, timing: res.timing },
361	530	};
362	531	}
363	532	}

modified packages/routing/test/engine.test.ts +7 −7

		`@@ -17,16 +17,16 @@ describe("RoutingEngine", () => {`
17	17	expect(registry.available().map((p) => p.id).sort()).toEqual(["decodo", "oxylabs"]);
18	18	expect(registry.availableNetworks()).toEqual(["residential"]);
19	19	const engine = new RoutingEngine(registry, new CircuitBreaker());
20		− const plan = engine.plan({ domain: "example.com", network: "auto", geo, plan: "free", sessionRequired: false, browser: false });
	20	+ const plan = engine.plan({ domain: "example.com", network: "auto", geo, plan: "unlimited", sessionRequired: false, browser: false });
21	21	expect(plan.candidates.length).toBeGreaterThan(0);
22	22	expect(plan.candidates.every((c) => c.network === "residential")).toBe(true);
23		− expect(plan.maxAttempts).toBe(3); // free plan: 2 retries + 1
	23	+ expect(plan.maxAttempts).toBe(6); // unlimited plan: 5 retries + 1
24	24	});
25	25
26	26	it("prefers the cheaper provider when nothing else differs", () => {
27	27	const registry = new ProviderRegistry({ env, prices: { oxylabs: { residential: 8 }, decodo: { residential: 7 } } });
28	28	const engine = new RoutingEngine(registry, new CircuitBreaker());
29		− const plan = engine.plan({ domain: "example.com", network: "residential", geo, plan: "developer", sessionRequired: false, browser: false });
	29	+ const plan = engine.plan({ domain: "example.com", network: "residential", geo, plan: "unlimited", sessionRequired: false, browser: false });
30	30	expect(plan.candidates[0]!.provider.id).toBe("decodo");
31	31	});
32	32
		`@@ -34,7 +34,7 @@ describe("RoutingEngine", () => {`
34	34	const registry = new ProviderRegistry({ env, prices: { oxylabs: { residential: 7 }, decodo: { residential: 7 } } });
35	35	const engine = new RoutingEngine(registry, new CircuitBreaker());
36	36	const stats = { [routeKey("oxylabs", "residential")]: { n: 50, ok: 50, blocked: 0, lat: 400, cost: 0.01 }, [routeKey("decodo", "residential")]: { n: 50, ok: 10, blocked: 40, lat: 1500, cost: 0.01 } };
37		− const plan = engine.plan({ domain: "hard.example", network: "residential", geo, plan: "growth", sessionRequired: false, browser: false, knowledge: { domain: "hard.example", routeStats: stats } });
	37	+ const plan = engine.plan({ domain: "hard.example", network: "residential", geo, plan: "unlimited", sessionRequired: false, browser: false, knowledge: { domain: "hard.example", routeStats: stats } });
38	38	expect(plan.candidates[0]!.provider.id).toBe("oxylabs");
39	39	});
40	40
		`@@ -44,7 +44,7 @@ describe("RoutingEngine", () => {`
44	44	for (let i = 0; i < 4; i++) circuit.record(routeKey("decodo", "residential"), false);
45	45	expect(circuit.state(routeKey("decodo", "residential"))).toBe("open");
46	46	const engine = new RoutingEngine(registry, circuit);
47		− const plan = engine.plan({ domain: "x.com", network: "residential", geo, plan: "free", sessionRequired: false, browser: false });
	47	+ const plan = engine.plan({ domain: "x.com", network: "residential", geo, plan: "unlimited", sessionRequired: false, browser: false });
48	48	expect(plan.candidates.every((c) => c.provider.id !== "decodo")).toBe(true);
49	49	});
50	50
		`@@ -55,7 +55,7 @@ describe("RoutingEngine", () => {`
55	55	domain: "pinned.example",
56	56	network: "residential",
57	57	geo,
58		− plan: "business",
	58	+ plan: "unlimited",
59	59	sessionRequired: false,
60	60	browser: false,
61	61	knowledge: { domain: "pinned.example", routeStats: {}, policy: { order: [routeKey("oxylabs", "residential")] } },
		`@@ -66,7 +66,7 @@ describe("RoutingEngine", () => {`
66	66	it("returns no candidates for unavailable network classes", () => {
67	67	const registry = new ProviderRegistry({ env });
68	68	const engine = new RoutingEngine(registry, new CircuitBreaker());
69		− const plan = engine.plan({ domain: "x.com", network: "mobile", geo, plan: "business", sessionRequired: false, browser: false });
	69	+ const plan = engine.plan({ domain: "x.com", network: "mobile", geo, plan: "unlimited", sessionRequired: false, browser: false });
70	70	expect(plan.candidates).toHaveLength(0);
71	71	});
72	72	});

modified packages/sdk/package.json +1 −1

		`@@ -1,6 +1,6 @@`
1	1	{
2	2	"name": "@fetcha/sdk",
3		− "version": "0.1.0",
	3	+ "version": "0.2.0",
4	4	"description": "Official JavaScript/TypeScript SDK for Fetcha — Intelligent Web Access Infrastructure",
5	5	"license": "MIT",
6	6	"type": "module",

modified packages/sdk/src/index.ts +274 −9

		`@@ -4,13 +4,23 @@`
4	4	* ```ts
5	5	* import { Fetcha } from "@fetcha/sdk";
6	6	* const fetcha = new Fetcha({ apiKey: process.env.FETCHA_API_KEY! });
7		− * const result = await fetcha.fetch({ url: "https://example.com", country: "CA" });
8		− * console.log(result.status, result.content?.slice(0, 200));
	7	+ * const result = await fetcha.fetch({ url: "https://example.com", country: "CA", format: "markdown" });
	8	+ * console.log(result.status, result.metadata.mode, result.markdown?.slice(0, 200));
	9	+ *
	10	+ * const job = await fetcha.crawl.create({ url: "https://docs.example.com/", max_pages: 100 });
	11	+ * const done = await fetcha.crawl.wait(job.id);
	12	+ * const pages = await fetcha.crawl.pages(job.id, { limit: 100 });
9	13	* ```
	14	+ *
	15	+ * Zero dependencies; works on Node 18+, Deno, Bun and modern browsers (global `fetch`).
10	16	*/
11	17
	18	+export const SDK_VERSION = "0.2.0";
	19	+
12	20	export type FetchaNetwork = "auto" \| "datacenter" \| "residential" \| "isp" \| "mobile";
13		−export type FetchaFormat = "html" \| "text" \| "json" \| "raw";
	21	+export type FetchaFormat = "html" \| "text" \| "markdown" \| "json" \| "raw";
	22	+export type FetchaMode = "http" \| "browser";
	23	+export type FetchaWaitUntil = "load" \| "domcontentloaded" \| "networkidle";
14	24
15	25	export interface FetchOptions {
16	26	url: string;
		`@@ -18,13 +28,33 @@ export interface FetchOptions {`
18	28	headers?: Record<string, string>;
19	29	cookies?: Record<string, string>;
20	30	body?: string \| Record<string, unknown>;
	31	+ /** Overall deadline in ms for all attempts, including browser renders (1,000–120,000). */
21	32	timeout?: number;
22	33	country?: string;
23	34	region?: string;
24	35	city?: string;
25	36	network?: FetchaNetwork;
26	37	session?: string;
	38	+ /** Render in the managed headless browser routed through the same network / geo / session. */
27	39	browser?: boolean;
	40	+ /** Escalate to the browser automatically when an HTTP attempt is blocked by a JS challenge (default true). */
	41	+ browser_fallback?: boolean;
	42	+ /** Browser: CSS selector to wait for before capturing. */
	43	+ wait_for?: string;
	44	+ /** Browser: extra settle time in ms (0–30,000). */
	45	+ wait_ms?: number;
	46	+ /** Browser: navigation wait condition (default "domcontentloaded"). */
	47	+ wait_until?: FetchaWaitUntil;
	48	+ /** Browser: disable scripting when false. */
	49	+ javascript?: boolean;
	50	+ /** Browser: skip images, fonts and media (default true). */
	51	+ block_resources?: boolean;
	52	+ /** Browser: return a PNG screenshot (base64) in `screenshot`. */
	53	+ screenshot?: boolean;
	54	+ /** Return `links[]` (all hyperlinks, absolute). */
	55	+ links?: boolean;
	56	+ /** Referer strategy: "auto" (none first, search-engine referer on retries), "none", or a literal URL. */
	57	+ referer?: "auto" \| "none" \| string;
28	58	device?: "desktop" \| "mobile" \| "tablet";
29	59	locale?: string;
30	60	format?: FetchaFormat;
		`@@ -35,6 +65,34 @@ export interface FetchOptions {`
35	65	debug?: boolean;
36	66	}
37	67
	68	+export interface PageMetadata {
	69	+ title: string \| null;
	70	+ description: string \| null;
	71	+ canonical: string \| null;
	72	+ lang: string \| null;
	73	+ og: Record<string, string>;
	74	+ links_count: number;
	75	+}
	76	+
	77	+export interface PageLink {
	78	+ url: string;
	79	+ text: string;
	80	+ internal: boolean;
	81	+ nofollow: boolean;
	82	+}
	83	+
	84	+export interface FetchAttempt {
	85	+ provider: string;
	86	+ network: string;
	87	+ mode: FetchaMode;
	88	+ country: string \| null;
	89	+ outcome: string;
	90	+ block_reason?: string \| null;
	91	+ status: number \| null;
	92	+ duration_ms: number;
	93	+ error?: string;
	94	+}
	95	+
38	96	export interface FetchResult {
39	97	request_id: string;
40	98	success: boolean;
		`@@ -45,18 +103,30 @@ export interface FetchResult {`
45	103	content_type: string \| null;
46	104	headers: Record<string, string>;
47	105	cookies: Array<{ name: string; value: string; domain?: string; path?: string }>;
	106	+ /** Present for format "text". */
48	107	text?: string \| null;
	108	+ /** Present for format "json" when the body parsed. */
49	109	json?: unknown;
	110	+ /** Present for format "markdown". */
	111	+ markdown?: string \| null;
	112	+ /** Parsed page metadata (HTML responses). */
	113	+ page?: PageMetadata \| null;
	114	+ /** Present with `links: true`. */
	115	+ links?: PageLink[];
	116	+ /** Present with browser rendering and `screenshot: true` (PNG, base64). */
	117	+ screenshot?: string \| null;
50	118	metadata: {
51	119	network: string;
52	120	country: string \| null;
	121	+ /** "http" for a plain fetch, "browser" when the final attempt was rendered. */
	122	+ mode: FetchaMode;
53	123	attempts: number;
54	124	duration_ms: number;
55	125	bytes: number;
56	126	session: string \| null;
57	127	cached: boolean;
58	128	timing?: Record<string, number>;
59		− debug?: unknown;
	129	+ debug?: { attempts: FetchAttempt[] };
60	130	};
61	131	}
62	132
		`@@ -78,6 +148,136 @@ export interface Session {`
78	148	created_at: string;
79	149	}
80	150
	151	+// ---------------------------------------------------------------------------
	152	+// Crawl & map
	153	+// ---------------------------------------------------------------------------
	154	+
	155	+export type CrawlFormat = "markdown" \| "text" \| "html";
	156	+export type CrawlStatus = "queued" \| "running" \| "completed" \| "failed" \| "cancelled";
	157	+
	158	+export interface CrawlOptions {
	159	+ url: string;
	160	+ /** Maximum pages to fetch (1–5,000; the seed counts as one). Default 25. */
	161	+ max_pages?: number;
	162	+ /** Maximum link depth from the seed (0–10). Default 2. */
	163	+ max_depth?: number;
	164	+ same_domain?: boolean;
	165	+ allow_subdomains?: boolean;
	166	+ /** Glob with `` or `/regex/`. /
	167	+ include_patterns?: string[];
	168	+ exclude_patterns?: string[];
	169	+ respect_robots?: boolean;
	170	+ use_sitemap?: boolean;
	171	+ /** Parallel page fetches (1–10). Default 3. */
	172	+ concurrency?: number;
	173	+ delay_ms?: number;
	174	+ /** Per-page timeout in ms. */
	175	+ timeout?: number;
	176	+ format?: CrawlFormat;
	177	+ main_content?: boolean;
	178	+ country?: string;
	179	+ network?: FetchaNetwork;
	180	+ browser?: boolean;
	181	+ browser_fallback?: boolean;
	182	+ headers?: Record<string, string>;
	183	+ webhook_url?: string;
	184	+ label?: string;
	185	+}
	186	+
	187	+export interface CrawlStats {
	188	+ discovered: number;
	189	+ fetched: number;
	190	+ ok: number;
	191	+ blocked: number;
	192	+ failed: number;
	193	+ bytes: number;
	194	+}
	195	+
	196	+export interface CrawlJob {
	197	+ id: string;
	198	+ status: CrawlStatus;
	199	+ label: string \| null;
	200	+ seed_url: string;
	201	+ domain: string;
	202	+ options: Record<string, unknown>;
	203	+ stats: CrawlStats;
	204	+ error: { code: string; message: string } \| null;
	205	+ created_at: string;
	206	+ started_at: string \| null;
	207	+ completed_at: string \| null;
	208	+}
	209	+
	210	+/** Body of the 202 returned by `crawl.create`. */
	211	+export interface CrawlCreated {
	212	+ id: string;
	213	+ status: "queued";
	214	+ seed_url: string;
	215	+ created_at: string;
	216	+ options: Record<string, unknown>;
	217	+}
	218	+
	219	+export interface CrawlPage {
	220	+ id: string;
	221	+ url: string;
	222	+ final_url: string \| null;
	223	+ depth: number;
	224	+ status: string;
	225	+ http_status: number \| null;
	226	+ error_code: string \| null;
	227	+ title: string \| null;
	228	+ description: string \| null;
	229	+ content_type: string \| null;
	230	+ content: string \| null;
	231	+ links_count: number \| null;
	232	+ bytes: number \| null;
	233	+ duration_ms: number \| null;
	234	+ mode: FetchaMode \| null;
	235	+ fetched_at: string \| null;
	236	+}
	237	+
	238	+export interface CrawlPagesResult {
	239	+ data: CrawlPage[];
	240	+ next_cursor: string \| null;
	241	+}
	242	+
	243	+export interface CrawlPagesOptions {
	244	+ cursor?: string \| null;
	245	+ limit?: number;
	246	+ status?: "success" \| "blocked" \| "failed";
	247	+}
	248	+
	249	+export interface CrawlWaitOptions {
	250	+ /** Interval between polls in ms. Default 2,000. */
	251	+ pollMs?: number;
	252	+ /** Give up after this many ms. Default 600,000 (10 min). */
	253	+ timeoutMs?: number;
	254	+ /** Called after each poll with the current job. */
	255	+ onPoll?: (job: CrawlJob) => void;
	256	+}
	257	+
	258	+export interface MapOptions {
	259	+ url: string;
	260	+ /** Maximum URLs returned (1–10,000). Default 1,000. */
	261	+ limit?: number;
	262	+ use_sitemap?: boolean;
	263	+ use_links?: boolean;
	264	+ same_domain?: boolean;
	265	+ allow_subdomains?: boolean;
	266	+ /** Substring, glob (``) or `/regex/` filter. /
	267	+ search?: string;
	268	+ country?: string;
	269	+ network?: FetchaNetwork;
	270	+ timeout?: number;
	271	+}
	272	+
	273	+export interface MapResult {
	274	+ url: string;
	275	+ count: number;
	276	+ urls: string[];
	277	+ sources: { sitemap: number; links: number };
	278	+ truncated: boolean;
	279	+}
	280	+
81	281	export interface FetchaClientOptions {
82	282	apiKey: string;
83	283	baseUrl?: string;
		`@@ -90,15 +290,26 @@ export class FetchaError extends Error {`
90	290	readonly code: string;
91	291	readonly status: number;
92	292	readonly requestId: string \| null;
93		− constructor(code: string, message: string, status: number, requestId: string \| null) {
	293	+ readonly details?: Record<string, unknown>;
	294	+ constructor(code: string, message: string, status: number, requestId: string \| null, details?: Record<string, unknown>) {
94	295	super(message);
95	296	this.name = "FetchaError";
96	297	this.code = code;
97	298	this.status = status;
98	299	this.requestId = requestId;
	300	+ this.details = details;
99	301	}
100	302	}
101	303
	304	+const TERMINAL: ReadonlySet<CrawlStatus> = new Set(["completed", "failed", "cancelled"]);
	305	+
	306	+function query(params: Record<string, string \| number \| null \| undefined>): string {
	307	+ const sp = new URLSearchParams();
	308	+ for (const [k, v] of Object.entries(params)) if (v !== undefined && v !== null && v !== "") sp.set(k, String(v));
	309	+ const s = sp.toString();
	310	+ return s ? `?${s}` : "";
	311	+}
	312	+
102	313	export class Fetcha {
103	314	private readonly apiKey: string;
104	315	private readonly baseUrl: string;
		`@@ -122,7 +333,7 @@ export class Fetcha {`
122	333	headers: {
123	334	authorization: `Bearer ${this.apiKey}`,
124	335	"content-type": "application/json",
125		− "user-agent": "fetcha-sdk-js/0.1.0",
	336	+ "user-agent": `fetcha-sdk-js/${SDK_VERSION}`,
126	337	...(init.idempotencyKey ? { "idempotency-key": init.idempotencyKey } : {}),
127	338	},
128	339	body: init.body === undefined ? undefined : JSON.stringify(init.body),
		`@@ -137,8 +348,8 @@ export class Fetcha {`
137	348	data = null;
138	349	}
139	350	if (!res.ok) {
140		− const err = (data as { error?: { code?: string; message?: string; request_id?: string } } \| null)?.error;
141		− throw new FetchaError(err?.code ?? "HTTP_ERROR", err?.message ?? `HTTP ${res.status}`, res.status, err?.request_id ?? requestId);
	351	+ const err = (data as { error?: { code?: string; message?: string; request_id?: string; details?: Record<string, unknown> } } \| null)?.error;
	352	+ throw new FetchaError(err?.code ?? "HTTP_ERROR", err?.message ?? `HTTP ${res.status}`, res.status, err?.request_id ?? requestId, err?.details);
142	353	}
143	354	return data as T;
144	355	} finally {
		`@@ -146,7 +357,7 @@ export class Fetcha {`
146	357	}
147	358	}
148	359
149		− /** Fetch a URL through Fetcha's routing engine. */
	360	+ /** Fetch a URL through Fetcha's routing engine (optionally rendered in the managed browser). */
150	361	fetch(options: FetchOptions): Promise<FetchResult> {
151	362	return this.call<FetchResult>("/v1/fetch", { method: "POST", body: options });
152	363	}
		`@@ -157,12 +368,66 @@ export class Fetcha {`
157	368	return r.text ?? "";
158	369	}
159	370
	371	+ /** Convenience: GET a page and return it as Markdown. */
	372	+ async markdown(url: string, options: Omit<FetchOptions, "url" \| "format"> = {}): Promise<string> {
	373	+ const r = await this.fetch({ ...options, url, format: "markdown" });
	374	+ return r.markdown ?? "";
	375	+ }
	376	+
160	377	/** Convenience: GET a JSON endpoint. */
161	378	async json<T = unknown>(url: string, options: Omit<FetchOptions, "url" \| "format"> = {}): Promise<T> {
162	379	const r = await this.fetch({ ...options, url, format: "json" });
163	380	return r.json as T;
164	381	}
165	382
	383	+ /** Convenience: render a page in the managed browser. */
	384	+ render(url: string, options: Omit<FetchOptions, "url" \| "browser"> = {}): Promise<FetchResult> {
	385	+ return this.fetch({ ...options, url, browser: true });
	386	+ }
	387	+
	388	+ /** Discover the URLs of a site (sitemap + links), synchronously. */
	389	+ map(options: MapOptions): Promise<MapResult> {
	390	+ return this.call<MapResult>("/v1/map", { method: "POST", body: options });
	391	+ }
	392	+
	393	+ readonly crawl = {
	394	+ /** Start a crawl job. Returns immediately with status "queued". */
	395	+ create: (options: CrawlOptions) => this.call<CrawlCreated>("/v1/crawl", { method: "POST", body: options }),
	396	+ /** Get a job with its live status and stats. */
	397	+ get: (id: string) => this.call<CrawlJob>(`/v1/crawl/${encodeURIComponent(id)}`, { method: "GET" }),
	398	+ /** List the most recent jobs of the project. */
	399	+ list: (limit?: number) => this.call<{ data: CrawlJob[] }>(`/v1/crawl${query({ limit })}`, { method: "GET" }),
	400	+ /** Page through the crawled pages with `cursor` / `next_cursor`. */
	401	+ pages: (id: string, options: CrawlPagesOptions = {}) => this.call<CrawlPagesResult>(`/v1/crawl/${encodeURIComponent(id)}/pages${query({ cursor: options.cursor, limit: options.limit, status: options.status })}`, { method: "GET" }),
	402	+ /** Cancel a queued or running job. */
	403	+ cancel: (id: string) => this.call<{ id: string; status: "cancelled" }>(`/v1/crawl/${encodeURIComponent(id)}`, { method: "DELETE" }),
	404	+ /** Poll `get` until the job reaches a terminal status (completed, failed or cancelled). */
	405	+ wait: async (id: string, options: CrawlWaitOptions = {}): Promise<CrawlJob> => {
	406	+ const pollMs = Math.max(250, options.pollMs ?? 2000);
	407	+ const timeoutMs = options.timeoutMs ?? 600_000;
	408	+ const deadline = Date.now() + timeoutMs;
	409	+ for (;;) {
	410	+ const job = await this.crawl.get(id);
	411	+ options.onPoll?.(job);
	412	+ if (TERMINAL.has(job.status)) return job;
	413	+ if (Date.now() + pollMs > deadline) throw new FetchaError("CRAWL_WAIT_TIMEOUT", `Crawl ${id} did not finish within ${timeoutMs} ms (status: ${job.status}).`, 0, null);
	414	+ await new Promise((r) => setTimeout(r, pollMs));
	415	+ }
	416	+ },
	417	+ /** Iterate over every page of a job, following cursors. */
	418	+ iteratePages: (id: string, options: Omit<CrawlPagesOptions, "cursor"> = {}): AsyncGenerator<CrawlPage, void, undefined> => {
	419	+ const pages = this.crawl.pages;
	420	+ return (async function* () {
	421	+ let cursor: string \| null = null;
	422	+ do {
	423	+ const page: CrawlPagesResult = await pages(id, { ...options, cursor });
	424	+ for (const p of page.data) yield p;
	425	+ cursor = page.next_cursor;
	426	+ } while (cursor);
	427	+ })();
	428	+ },
	429	+ };
	430	+
166	431	readonly sessions = {
167	432	create: (options: SessionOptions = {}, idempotencyKey?: string) => this.call<Session>("/v1/sessions", { method: "POST", body: options, idempotencyKey }),
168	433	get: (id: string) => this.call<Session>(`/v1/sessions/${encodeURIComponent(id)}`, { method: "GET" }),

modified pnpm-lock.yaml +54 −0

		`@@ -29,6 +29,9 @@ importers:`
29	29	'@fastify/cors':
30	30	specifier: ^11.0.0
31	31	version: 11.3.0
	32	+ '@fetcha/browser':
	33	+ specifier: workspace:*
	34	+ version: link:../../packages/browser
32	35	'@fetcha/core':
33	36	specifier: workspace:*
34	37	version: link:../../packages/core
		`@@ -182,6 +185,28 @@ importers:`
182	185	specifier: ^3.2.0
183	186	version: 3.2.7(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13)
184	187
	188	+ packages/browser:
	189	+ dependencies:
	190	+ '@fetcha/core':
	191	+ specifier: workspace:*
	192	+ version: link:../core
	193	+ '@fetcha/providers':
	194	+ specifier: workspace:*
	195	+ version: link:../providers
	196	+ patchright:
	197	+ specifier: 1.62.3
	198	+ version: 1.62.3
	199	+ devDependencies:
	200	+ '@types/node':
	201	+ specifier: ^24.0.0
	202	+ version: 24.13.3
	203	+ typescript:
	204	+ specifier: ^5.9.3
	205	+ version: 5.9.3
	206	+ vitest:
	207	+ specifier: ^3.2.0
	208	+ version: 3.2.7(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.32.0)(tsx@4.23.13)
	209	+
185	210	packages/core:
186	211	dependencies:
187	212	nanoid:
		`@@ -284,6 +309,9 @@ importers:`
284	309
285	310	packages/routing:
286	311	dependencies:
	312	+ '@fetcha/browser':
	313	+ specifier: workspace:*
	314	+ version: link:../browser
287	315	'@fetcha/core':
288	316	specifier: workspace:*
289	317	version: link:../core
		`@@ -3275,6 +3303,11 @@ packages:`
3275	3303	react-dom:
3276	3304	optional: true
3277	3305
	3306	+ fsevents@2.3.2:
	3307	+ resolution: {integrity: sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==}
	3308	+ engines: {node: ^8.16.0 \|\| ^10.6.0 \|\| >=11.0.0}
	3309	+ os: [darwin]
	3310	+
3278	3311	fsevents@2.3.3:
3279	3312	resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
3280	3313	engines: {node: ^8.16.0 \|\| ^10.6.0 \|\| >=11.0.0}
		`@@ -3874,6 +3907,16 @@ packages:`
3874	3907	parseley@0.12.1:
3875	3908	resolution: {integrity: sha512-e6qHKe3a9HWr0oMRVDTRhKce+bRO8VGQR3NyVwcjwrbhMmFCX9KszEV35+rn4AdilFAq9VPxP/Fe1wC9Qjd2lw==}
3876	3909
	3910	+ patchright-core@1.62.3:
	3911	+ resolution: {integrity: sha512-RQf0M2THMf4TL9HNNUxdYbd4Oe3DOVPni6G/bJYsEpD9F1cgqEpWiTJIh+pfuK9JwGRo4Du6lzagJosMUXbh/Q==}
	3912	+ engines: {node: '>=20'}
	3913	+ hasBin: true
	3914	+
	3915	+ patchright@1.62.3:
	3916	+ resolution: {integrity: sha512-TMpWzcZVWUmOe1251PHWpk0gTP2d7+mvS00h1CLL2IQAUKX7UexKfPqEjdKkH8F2TT5AlyrSbwZKr5lu1dThAQ==}
	3917	+ engines: {node: '>=20'}
	3918	+ hasBin: true
	3919	+
3877	3920	path-exists@4.0.0:
3878	3921	resolution: {integrity: sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w==}
3879	3922	engines: {node: '>=8'}
		`@@ -7265,6 +7308,9 @@ snapshots:`
7265	7308	react: 19.2.8
7266	7309	react-dom: 19.2.8(react@19.2.8)
7267	7310
	7311	+ fsevents@2.3.2:
	7312	+ optional: true
	7313	+
7268	7314	fsevents@2.3.3:
7269	7315	optional: true
7270	7316
		`@@ -7848,6 +7894,14 @@ snapshots:`
7848	7894	leac: 0.6.0
7849	7895	peberminta: 0.9.0
7850	7896
	7897	+ patchright-core@1.62.3: {}
	7898	+
	7899	+ patchright@1.62.3:
	7900	+ dependencies:
	7901	+ patchright-core: 1.62.3
	7902	+ optionalDependencies:
	7903	+ fsevents: 2.3.2
	7904	+
7851	7905	path-exists@4.0.0: {}
7852	7906
7853	7907	path-key@3.1.1: {}

modified scripts/dev-seed.ts +7 −4

		`@@ -1,15 +1,18 @@`
1		−/** Creates a verified dev user + org + project + API key. Prints the key once. */
2		−import { getDb, closeDb, users, organizations, organizationMembers, projects, apiKeys } from "@fetcha/db";
	1	+/** Creates a verified dev admin user + org + project + API key and allowlists the email. Prints the key once. */
	2	+import { getDb, closeDb, users, organizations, organizationMembers, projects, apiKeys, signupAllowlist, eq } from "@fetcha/db";
3	3	import { generateApiKey, newId } from "@fetcha/core";
4	4	import { randomUUID } from "node:crypto";
5	5
6		−const email = process.argv[2] ?? "dev@fetcha.local";
	6	+const email = (process.argv[2] ?? "dev@fetcha.local").trim().toLowerCase();
7	7	async function main() {
8	8	const db = getDb();
9	9	const userId = randomUUID();
10	10	await db.insert(users).values({ id: userId, name: "Dev User", email, emailVerified: true, role: "admin" }).onConflictDoNothing();
11		− const [u] = await db.select().from(users).where((await import("drizzle-orm")).eq(users.email, email));
	11	+ const [u] = await db.select().from(users).where(eq(users.email, email));
	12	+ // Invitation-only platform: keep the allowlist consistent with the account we just created.
	13	+ await db.insert(signupAllowlist).values({ email, note: "dev-seed", userId: u!.id, usedAt: new Date() }).onConflictDoNothing();
12	14	const orgId = newId("org");
	15	+ // `plan` defaults to "unlimited" (the only plan).
13	16	await db.insert(organizations).values({ id: orgId, name: "Dev Org", slug: `dev-${orgId.slice(4, 10)}`, ownerUserId: u!.id, providerVisibility: true });
14	17	await db.insert(organizationMembers).values({ organizationId: orgId, userId: u!.id, role: "owner" });
15	18	const projectId = newId("proj");

modified sdk-python/README.md +28 −4

		`@@ -1,6 +1,6 @@`
1	1	# fetcha — Python SDK
2	2
3		−Official Python client for [Fetcha](https://www.fetcha.co), the intelligent web-access API.
	3	+Official Python client for [Fetcha](https://www.fetcha.co), the intelligent web-access API. Standard library only, Python 3.9+.
4	4
5	5	```bash
6	6	pip install fetcha
		`@@ -13,17 +13,41 @@ client = Fetcha(api_key="fch_live_...")`
13	13
14	14	# Fetch a page — Fetcha picks the network, geography and retries for you
15	15	result = client.fetch("https://example.com", country="CA")
16		−print(result.status, result.metadata["network"], result.metadata["attempts"])
	16	+print(result.status, result.metadata["network"], result.metadata["mode"], result.metadata["attempts"])
17	17	print(result.content[:500])
18	18
19		−# Readable text or JSON helpers
	19	+# Readable text, Markdown or JSON helpers
20	20	text = client.text("https://example.com")
	21	+md = client.markdown("https://example.com/article") # main content as Markdown
21	22	data = client.json("https://api.example.com/items")
22	23
	24	+# Page metadata and links come with every HTML response
	25	+r = client.fetch("https://example.com", links=True)
	26	+print(r.page["title"], r.page["links_count"], r.links[:3])
	27	+
	28	+# Managed browser rendering (same network / country / session as a plain fetch)
	29	+r = client.render("https://app.example.com/dashboard", wait_for="table.results", screenshot=True)
	30	+open("dashboard.png", "wb").write(__import__("base64").b64decode(r.screenshot))
	31	+# Blocked HTTP attempts escalate to the browser automatically (browser_fallback=True by default);
	32	+# r.mode tells you whether the final attempt was "http" or "browser".
	33	+
23	34	# Sticky sessions keep the same exit identity across requests
24	35	session = client.sessions.create(country="CA", ttl=600)
25	36	page1 = client.fetch("https://example.com/login", session=session.id)
26	37	page2 = client.fetch("https://example.com/account", session=session.id)
	38	+
	39	+# Crawl a site into Markdown (asynchronous job)
	40	+job = client.crawl.create(url="https://docs.example.com/", max_pages=200, max_depth=3, include_patterns=["/docs/*"])
	41	+job = client.crawl.wait(job.id, poll_s=2.0, timeout_s=600) # polls until completed \| failed \| cancelled
	42	+print(job.status, job.stats) # {'discovered': …, 'fetched': …, 'ok': …, 'blocked': …, 'failed': …, 'bytes': …}
	43	+for page in client.crawl.iter_pages(job.id, status="success"):
	44	+ print(page.url, page.title, len(page.content or ""))
	45	+# or page manually: client.crawl.pages(job.id, cursor=None, limit=100).next_cursor
	46	+client.crawl.cancel(job.id)
	47	+
	48	+# Map a site's URLs (sitemap + links) without fetching every page
	49	+m = client.map("https://docs.example.com/", search="/docs/*", limit=500)
	50	+print(m.count, m.urls[:5], m.sources, m.truncated)
27	51	```
28	52
29		−Errors raise `fetcha.FetchaError` with `.code`, `.status` and `.request_id`.
	53	+Errors raise `fetcha.FetchaError` with `.code`, `.status`, `.request_id` and `.details`.

modified sdk-python/fetcha/__init__.py +266 −27

		`@@ -3,30 +3,51 @@`
3	3	from fetcha import Fetcha
4	4
5	5	client = Fetcha(api_key="fch_live_...")
6		− response = client.fetch(url="https://example.com", country="CA")
7		− print(response.status, response.content[:200])
	6	+ result = client.fetch("https://example.com", country="CA", format="markdown")
	7	+ print(result.status, result.metadata["mode"], (result.markdown or "")[:200])
	8	+
	9	+ job = client.crawl.create(url="https://docs.example.com/", max_pages=100)
	10	+ job = client.crawl.wait(job.id)
	11	+ page = client.crawl.pages(job.id, limit=100)
	12	+
	13	+Zero third-party dependencies: uses the standard library (urllib) only.
8	14	"""
9	15
10	16	from __future__ import annotations
11	17
	18	+import json as _json
	19	+import time
	20	+import urllib.error
	21	+import urllib.parse
	22	+import urllib.request
12	23	from dataclasses import dataclass, field
13		−from typing import Any, Dict, List, Optional
	24	+from typing import Any, Dict, Iterator, List, Optional
14	25
15		−import httpx
	26	+__all__ = [
	27	+ "Fetcha",
	28	+ "FetchaError",
	29	+ "FetchResult",
	30	+ "Session",
	31	+ "CrawlJob",
	32	+ "CrawlPage",
	33	+ "CrawlPages",
	34	+ "MapResult",
	35	+]
	36	+__version__ = "0.2.0"
16	37
17		−__all__ = ["Fetcha", "FetchaError", "FetchResult", "Session"]
18		−__version__ = "0.1.0"
	38	+_TERMINAL = frozenset({"completed", "failed", "cancelled"})
19	39
20	40
21	41	class FetchaError(Exception):
22	42	"""Raised when the Fetcha API returns an error envelope."""
23	43
24		− def __init__(self, code: str, message: str, status: int, request_id: Optional[str] = None):
	44	+ def __init__(self, code: str, message: str, status: int, request_id: Optional[str] = None, details: Optional[Dict[str, Any]] = None):
25	45	super().__init__(f"{code}: {message}")
26	46	self.code = code
27	47	self.message = message
28	48	self.status = status
29	49	self.request_id = request_id
	50	+ self.details = details or {}
30	51
31	52
32	53	@dataclass
		`@@ -43,8 +64,17 @@ class FetchResult:`
43	64	metadata: Dict[str, Any]
44	65	text: Optional[str] = None
45	66	json: Any = None
	67	+ markdown: Optional[str] = None
	68	+ page: Optional[Dict[str, Any]] = None
	69	+ links: Optional[List[Dict[str, Any]]] = None
	70	+ screenshot: Optional[str] = None
46	71	raw: Dict[str, Any] = field(default_factory=dict, repr=False)
47	72
	73	+ @property
	74	+ def mode(self) -> str:
	75	+ """"http" for a plain fetch, "browser" when the final attempt was rendered."""
	76	+ return str(self.metadata.get("mode") or "http")
	77	+
48	78	@classmethod
49	79	def from_dict(cls, d: Dict[str, Any]) -> "FetchResult":
50	80	return cls(
		`@@ -60,6 +90,10 @@ class FetchResult:`
60	90	metadata=d.get("metadata") or {},
61	91	text=d.get("text"),
62	92	json=d.get("json"),
	93	+ markdown=d.get("markdown"),
	94	+ page=d.get("page"),
	95	+ links=d.get("links"),
	96	+ screenshot=d.get("screenshot"),
63	97	raw=d,
64	98	)
65	99
		`@@ -87,6 +121,118 @@ class Session:`
87	121	)
88	122
89	123
	124	+@dataclass
	125	+class CrawlJob:
	126	+ id: str
	127	+ status: str
	128	+ seed_url: str
	129	+ label: Optional[str] = None
	130	+ domain: Optional[str] = None
	131	+ options: Dict[str, Any] = field(default_factory=dict)
	132	+ stats: Dict[str, int] = field(default_factory=dict)
	133	+ error: Optional[Dict[str, Any]] = None
	134	+ created_at: str = ""
	135	+ started_at: Optional[str] = None
	136	+ completed_at: Optional[str] = None
	137	+ raw: Dict[str, Any] = field(default_factory=dict, repr=False)
	138	+
	139	+ @property
	140	+ def done(self) -> bool:
	141	+ return self.status in _TERMINAL
	142	+
	143	+ @classmethod
	144	+ def from_dict(cls, d: Dict[str, Any]) -> "CrawlJob":
	145	+ return cls(
	146	+ id=d["id"],
	147	+ status=d.get("status", ""),
	148	+ seed_url=d.get("seed_url", ""),
	149	+ label=d.get("label"),
	150	+ domain=d.get("domain"),
	151	+ options=d.get("options") or {},
	152	+ stats=d.get("stats") or {},
	153	+ error=d.get("error"),
	154	+ created_at=d.get("created_at", ""),
	155	+ started_at=d.get("started_at"),
	156	+ completed_at=d.get("completed_at"),
	157	+ raw=d,
	158	+ )
	159	+
	160	+
	161	+@dataclass
	162	+class CrawlPage:
	163	+ id: str
	164	+ url: str
	165	+ status: str
	166	+ depth: int = 0
	167	+ final_url: Optional[str] = None
	168	+ http_status: Optional[int] = None
	169	+ error_code: Optional[str] = None
	170	+ title: Optional[str] = None
	171	+ description: Optional[str] = None
	172	+ content_type: Optional[str] = None
	173	+ content: Optional[str] = None
	174	+ links_count: Optional[int] = None
	175	+ bytes: Optional[int] = None
	176	+ duration_ms: Optional[int] = None
	177	+ mode: Optional[str] = None
	178	+ fetched_at: Optional[str] = None
	179	+ raw: Dict[str, Any] = field(default_factory=dict, repr=False)
	180	+
	181	+ @classmethod
	182	+ def from_dict(cls, d: Dict[str, Any]) -> "CrawlPage":
	183	+ return cls(
	184	+ id=d.get("id", ""),
	185	+ url=d.get("url", ""),
	186	+ status=d.get("status", ""),
	187	+ depth=int(d.get("depth") or 0),
	188	+ final_url=d.get("final_url"),
	189	+ http_status=d.get("http_status"),
	190	+ error_code=d.get("error_code"),
	191	+ title=d.get("title"),
	192	+ description=d.get("description"),
	193	+ content_type=d.get("content_type"),
	194	+ content=d.get("content"),
	195	+ links_count=d.get("links_count"),
	196	+ bytes=d.get("bytes"),
	197	+ duration_ms=d.get("duration_ms"),
	198	+ mode=d.get("mode"),
	199	+ fetched_at=d.get("fetched_at"),
	200	+ raw=d,
	201	+ )
	202	+
	203	+
	204	+@dataclass
	205	+class CrawlPages:
	206	+ data: List[CrawlPage]
	207	+ next_cursor: Optional[str]
	208	+ raw: Dict[str, Any] = field(default_factory=dict, repr=False)
	209	+
	210	+ @classmethod
	211	+ def from_dict(cls, d: Dict[str, Any]) -> "CrawlPages":
	212	+ return cls(data=[CrawlPage.from_dict(p) for p in d.get("data", [])], next_cursor=d.get("next_cursor"), raw=d)
	213	+
	214	+
	215	+@dataclass
	216	+class MapResult:
	217	+ url: str
	218	+ count: int
	219	+ urls: List[str]
	220	+ sources: Dict[str, int]
	221	+ truncated: bool
	222	+ raw: Dict[str, Any] = field(default_factory=dict, repr=False)
	223	+
	224	+ @classmethod
	225	+ def from_dict(cls, d: Dict[str, Any]) -> "MapResult":
	226	+ return cls(
	227	+ url=d.get("url", ""),
	228	+ count=int(d.get("count") or 0),
	229	+ urls=list(d.get("urls") or []),
	230	+ sources=d.get("sources") or {},
	231	+ truncated=bool(d.get("truncated")),
	232	+ raw=d,
	233	+ )
	234	+
	235	+
90	236	class _Sessions:
91	237	def __init__(self, client: "Fetcha"):
92	238	self._c = client
		`@@ -95,67 +241,160 @@ class _Sessions:`
95	241	return Session.from_dict(self._c._request("POST", "/v1/sessions", json=options))
96	242
97	243	def get(self, session_id: str) -> Session:
98		− return Session.from_dict(self._c._request("GET", f"/v1/sessions/{session_id}"))
	244	+ return Session.from_dict(self._c._request("GET", f"/v1/sessions/{urllib.parse.quote(session_id, safe='')}"))
99	245
100	246	def close(self, session_id: str) -> Dict[str, Any]:
101		− return self._c._request("DELETE", f"/v1/sessions/{session_id}")
	247	+ return self._c._request("DELETE", f"/v1/sessions/{urllib.parse.quote(session_id, safe='')}")
102	248
103	249	def list(self) -> List[Session]:
104	250	data = self._c._request("GET", "/v1/sessions")
105	251	return [Session.from_dict(s) for s in data.get("data", [])]
106	252
107	253
	254	+class _Crawl:
	255	+ """`client.crawl`: asynchronous site crawls (POST /v1/crawl)."""
	256	+
	257	+ def __init__(self, client: "Fetcha"):
	258	+ self._c = client
	259	+
	260	+ def create(self, url: str, **options: Any) -> CrawlJob:
	261	+ """Start a crawl job. Options: max_pages, max_depth, same_domain, allow_subdomains,
	262	+ include_patterns, exclude_patterns, respect_robots, use_sitemap, concurrency, delay_ms,
	263	+ timeout, format (markdown\|text\|html), main_content, country, network, browser,
	264	+ browser_fallback, headers, webhook_url, label. Returns immediately with status "queued"."""
	265	+ payload = {"url": url, **_clean(options)}
	266	+ return CrawlJob.from_dict(self._c._request("POST", "/v1/crawl", json=payload))
	267	+
	268	+ def get(self, job_id: str) -> CrawlJob:
	269	+ return CrawlJob.from_dict(self._c._request("GET", f"/v1/crawl/{urllib.parse.quote(job_id, safe='')}"))
	270	+
	271	+ def list(self, limit: Optional[int] = None) -> List[CrawlJob]:
	272	+ data = self._c._request("GET", "/v1/crawl" + _query({"limit": limit}))
	273	+ return [CrawlJob.from_dict(j) for j in data.get("data", [])]
	274	+
	275	+ def pages(self, job_id: str, cursor: Optional[str] = None, limit: Optional[int] = None, status: Optional[str] = None) -> CrawlPages:
	276	+ """One page of results. Pass `cursor=result.next_cursor` to continue; it is None on the last page."""
	277	+ path = f"/v1/crawl/{urllib.parse.quote(job_id, safe='')}/pages" + _query({"cursor": cursor, "limit": limit, "status": status})
	278	+ return CrawlPages.from_dict(self._c._request("GET", path))
	279	+
	280	+ def iter_pages(self, job_id: str, limit: Optional[int] = None, status: Optional[str] = None) -> Iterator[CrawlPage]:
	281	+ """Iterate over every page of a job, following cursors."""
	282	+ cursor: Optional[str] = None
	283	+ while True:
	284	+ page = self.pages(job_id, cursor=cursor, limit=limit, status=status)
	285	+ for p in page.data:
	286	+ yield p
	287	+ cursor = page.next_cursor
	288	+ if not cursor:
	289	+ return
	290	+
	291	+ def cancel(self, job_id: str) -> Dict[str, Any]:
	292	+ return self._c._request("DELETE", f"/v1/crawl/{urllib.parse.quote(job_id, safe='')}")
	293	+
	294	+ def wait(self, job_id: str, poll_s: float = 2.0, timeout_s: float = 600.0, on_poll: Any = None) -> CrawlJob:
	295	+ """Poll `get` until the job is completed, failed or cancelled."""
	296	+ poll_s = max(0.25, float(poll_s))
	297	+ deadline = time.monotonic() + float(timeout_s)
	298	+ while True:
	299	+ job = self.get(job_id)
	300	+ if on_poll is not None:
	301	+ on_poll(job)
	302	+ if job.done:
	303	+ return job
	304	+ if time.monotonic() + poll_s > deadline:
	305	+ raise FetchaError("CRAWL_WAIT_TIMEOUT", f"Crawl {job_id} did not finish within {timeout_s:g} s (status: {job.status}).", 0)
	306	+ time.sleep(poll_s)
	307	+
	308	+
	309	+def _clean(options: Dict[str, Any]) -> Dict[str, Any]:
	310	+ return {k: v for k, v in options.items() if v is not None}
	311	+
	312	+
	313	+def _query(params: Dict[str, Any]) -> str:
	314	+ items = {k: str(v) for k, v in params.items() if v is not None and v != ""}
	315	+ return f"?{urllib.parse.urlencode(items)}" if items else ""
	316	+
	317	+
108	318	class Fetcha:
109		− """Synchronous Fetcha client."""
	319	+ """Synchronous Fetcha client (standard library only)."""
110	320
111	321	def __init__(self, api_key: str, base_url: str = "https://www.fetcha.co", timeout: float = 150.0):
112	322	if not api_key:
113	323	raise ValueError("api_key is required")
114	324	self._api_key = api_key
115	325	self._base_url = base_url.rstrip("/")
116		− self._http = httpx.Client(
117		− timeout=timeout,
118		− headers={
119		− "Authorization": f"Bearer {api_key}",
120		− "Content-Type": "application/json",
121		− "User-Agent": f"fetcha-sdk-python/{__version__}",
122		− },
123		− )
	326	+ self._timeout = timeout
	327	+ self._headers = {
	328	+ "Authorization": f"Bearer {api_key}",
	329	+ "Content-Type": "application/json",
	330	+ "Accept": "application/json",
	331	+ "User-Agent": f"fetcha-sdk-python/{__version__}",
	332	+ }
124	333	self.sessions = _Sessions(self)
	334	+ self.crawl = _Crawl(self)
125	335
126	336	# -- low level -----------------------------------------------------------------
127	337	def _request(self, method: str, path: str, json: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
128		− res = self._http.request(method, f"{self._base_url}{path}", json=json)
129		− request_id = res.headers.get("x-fetcha-request-id")
	338	+ body = _json.dumps(json).encode("utf-8") if json is not None else None
	339	+ req = urllib.request.Request(f"{self._base_url}{path}", data=body, method=method, headers=self._headers)
	340	+ try:
	341	+ with urllib.request.urlopen(req, timeout=self._timeout) as res:
	342	+ status = res.status
	343	+ request_id = res.headers.get("x-fetcha-request-id")
	344	+ raw = res.read()
	345	+ except urllib.error.HTTPError as e:
	346	+ status = e.code
	347	+ request_id = e.headers.get("x-fetcha-request-id") if e.headers else None
	348	+ raw = e.read()
	349	+ except urllib.error.URLError as e:
	350	+ raise FetchaError("NETWORK_ERROR", f"Could not reach the Fetcha API: {e.reason}", 0) from e
130	351	try:
131		− data = res.json() if res.content else {}
	352	+ data = _json.loads(raw.decode("utf-8")) if raw else {}
132	353	except ValueError:
133	354	data = {}
134		− if res.status_code >= 400:
	355	+ if status >= 400:
135	356	err = (data or {}).get("error") or {}
136	357	raise FetchaError(
137	358	err.get("code", "HTTP_ERROR"),
138		− err.get("message", f"HTTP {res.status_code}"),
139		− res.status_code,
	359	+ err.get("message", f"HTTP {status}"),
	360	+ status,
140	361	err.get("request_id") or request_id,
	362	+ err.get("details"),
141	363	)
142	364	return data
143	365
144	366	# -- public --------------------------------------------------------------------
145	367	def fetch(self, url: str, **options: Any) -> FetchResult:
146	368	"""POST /v1/fetch. Options: method, headers, cookies, body, timeout, country, region, city,
147		− network, session, device, locale, format, follow_redirects, max_redirects, retries, debug."""
148		− payload = {"url": url, **options}
	369	+ network, session, browser, browser_fallback, wait_for, wait_ms, wait_until, javascript,
	370	+ block_resources, screenshot, links, referer, device, locale, format (html\|text\|markdown\|json\|raw),
	371	+ follow_redirects, max_redirects, max_response_bytes, retries, debug."""
	372	+ payload = {"url": url, **_clean(options)}
149	373	return FetchResult.from_dict(self._request("POST", "/v1/fetch", json=payload))
150	374
151	375	def text(self, url: str, **options: Any) -> str:
152	376	options["format"] = "text"
153	377	return self.fetch(url, **options).text or ""
154	378
	379	+ def markdown(self, url: str, **options: Any) -> str:
	380	+ options["format"] = "markdown"
	381	+ return self.fetch(url, **options).markdown or ""
	382	+
155	383	def json(self, url: str, **options: Any) -> Any:
156	384	options["format"] = "json"
157	385	return self.fetch(url, **options).json
158	386
	387	+ def render(self, url: str, **options: Any) -> FetchResult:
	388	+ """Render a page in the managed browser (browser=True)."""
	389	+ options["browser"] = True
	390	+ return self.fetch(url, **options)
	391	+
	392	+ def map(self, url: str, **options: Any) -> MapResult:
	393	+ """POST /v1/map: list a site's URLs (sitemap + links), synchronously. Options: limit,
	394	+ use_sitemap, use_links, same_domain, allow_subdomains, search, country, network, timeout."""
	395	+ payload = {"url": url, **_clean(options)}
	396	+ return MapResult.from_dict(self._request("POST", "/v1/map", json=payload))
	397	+
159	398	def me(self) -> Dict[str, Any]:
160	399	return self._request("GET", "/v1/me")
161	400
		`@@ -163,7 +402,7 @@ class Fetcha:`
163	402	return self._request("GET", "/v1/usage")
164	403
165	404	def close(self) -> None:
166		− self._http.close()
	405	+ """Kept for API compatibility; the client holds no persistent connection."""
167	406
168	407	def __enter__(self) -> "Fetcha":
169	408	return self

added sdk-python/fetcha/__pycache__/__init__.cpython-314.pyc +0 −0

Binary file not shown.

modified sdk-python/pyproject.toml +2 −2

		`@@ -4,14 +4,14 @@ build-backend = "hatchling.build"`
4	4
5	5	[project]
6	6	name = "fetcha"
7		−version = "0.1.0"
	7	+version = "0.2.0"
8	8	description = "Official Python SDK for Fetcha — Intelligent Web Access Infrastructure"
9	9	readme = "README.md"
10	10	license = { text = "MIT" }
11	11	requires-python = ">=3.9"
12	12	authors = [{ name = "Fetcha", email = "hello@fetcha.co" }]
13	13	keywords = ["fetcha", "proxy", "scraping", "web access", "residential proxy"]
14		−dependencies = ["httpx>=0.27"]
	14	+dependencies = []
15	15
16	16	[project.urls]
17	17	Homepage = "https://www.fetcha.co"
18	18