Fetcha v0.2.0 — private access (single unlimited plan, invitation-only signup, /admin/access), managed browser (Patchright, auto-escalation), hardened HTTP fingerprints (ordered headers, HTTP/2, TLS profiles, cookie jar, backoff), expanded block detection, markdown/page/links, crawl & map API, dashboard Crawls, SDK 0.2.0 · 7a8260e · fetcha

modified .env.example +11 −0

		`@@ -32,3 +32,14 @@ SOAX_PASSWORD=`
32	32
33	33	# ---- Admin ----
34	34	ADMIN_EMAILS=you@example.com
	35	+
	36	+# --- v0.2: managed browser, HTTP engine, crawler ---------------------------
	37	+# Managed Chromium (Playwright). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
	38	+FETCHA_BROWSER_ENABLED=1
	39	+FETCHA_BROWSER_CONCURRENCY=6
	40	+# "chromium" = full Chromium build in new headless mode (best anti-bot realism); "chrome" uses an installed Google Chrome.
	41	+FETCHA_BROWSER_CHANNEL=chromium
	42	+# Prefer HTTP/2 towards origins (set 0 to force HTTP/1.1).
	43	+FETCHA_HTTP2=1
	44	+# Crawl jobs executed in parallel by this API process.
	45	+FETCHA_CRAWL_PARALLEL_JOBS=4

modified CLAUDE.md +15 −5

		`@@ -2,15 +2,20 @@`
2	2
3	3	Fetcha (www.fetcha.co) is "Intelligent Web Access Infrastructure": one API (`POST /v1/fetch`) that routes web requests
4	4	across upstream proxy networks (Oxylabs, Decodo, SOAX — never exposed to customers) with smart routing, retries,
5		−sticky sessions, usage metering and a full dashboard.
	5	+sticky sessions, a managed headless browser (`browser: true` + automatic escalation), crawl/map jobs, usage metering
	6	+and a full dashboard. Private platform (v0.2, 2026-09-08): a single `unlimited` plan, invitation-only signup
	7	+(`signup_allowlist`, managed from `/admin/access`), `ADMIN_EMAILS` become admins. No billing.
6	8
7	9	## Layout
8	10	- `apps/web` — Next.js 16 (marketing site, docs, auth, customer dashboard, admin). Port 8220. Proxies `/v1/*` to the API.
9	11	- `apps/api` — Fastify API service (public `/v1/`, `/health`, `/ready`, internal `/internal/` for the dashboard). Port 8221.
10	12	- `packages/core` — zod schemas, error codes, ids/api-key hashing, SSRF policy, geo model, plan limits.
11	13	- `packages/db` — Drizzle schema + migrations (`drizzle/`), `pnpm db:generate\|migrate\|seed`.
12		−- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct). Provider code lives ONLY here.
13		−- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, block detection, escalation).
	14	+- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct), `http.ts` (undici, HTTP/2, browser-ordered headers, Chrome/Firefox/Safari TLS cipher lists, h2→h1 fallback), `fingerprint.ts` (header profiles), `cookies.ts` (jar across hops / sessions). Provider code lives ONLY here.
	15	+- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, fingerprint rotation, jittered backoff, block detection, HTTP → browser escalation).
	16	+- `packages/browser` — `BrowserPool` (Patchright = patched Playwright, full Chromium new-headless, per-context upstream proxy, stealth init script, challenge wait + Turnstile click). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
	17	+- `packages/core/src/markdown.ts` — HTML → Markdown / main text, page metadata + links, URL normalisation, glob/regex matchers; `robots.ts` — robots.txt + sitemap parsing.
	18	+- `apps/api/src/services/crawl.ts` — durable crawl jobs (`crawl_jobs`/`crawl_pages`, in-process worker, resume after restart) and the sync `/v1/map`.
14	19	- `packages/email` — Resend abstraction + React Email templates. `packages/sdk` (JS), `sdk-python/` (Python).
15	20	- `docs/AGENT-BRIEF.md` — product/engineering rules used when generating UI.
16	21
		`@@ -19,7 +24,9 @@ sticky sessions, usage metering and a full dashboard.`
19	24	- Every external URL goes through `assertUrlAllowed()` (SSRF) — including redirects.
20	25	- Business logic in server actions / `apps/api` / packages, not in React components.
21	26	- API keys: shown once, SHA-256 stored. Request IDs `req_…` on every response (`X-Fetcha-Request-ID`).
22		−- Not implemented yet (say so, don't fake): browser mode, `/v1/extract`, Stripe checkout, teams, webhook delivery, OAuth, 2FA.
	27	+- Not implemented yet (say so, don't fake): scripted browser actions (`POST /v1/browser`), `/v1/extract`, teams, webhook delivery (except the crawl `webhook_url` callback), OAuth, 2FA. Billing/Stripe is intentionally absent (private platform).
	28	+- Plans: always `normalizePlan(org.plan)`; never reintroduce tiers. Signup must stay allowlist-gated (Better Auth `user.create.before` hook in `apps/web/src/lib/auth.ts`).
	29	+- Anti-bot honesty: interactive Cloudflare Turnstile challenges are NOT reliably solved (tested 2026-09-08 with Patchright + residential + headful); non-interactive JS challenges, DataDome/PX/Akamai soft blocks usually pass via residential + browser escalation.
23	30
24	31	## Dev
25	32	`cp .env.example .env` (fill provider creds + RESEND_API_KEY), `createdb fetcha`, `pnpm db:migrate && pnpm db:seed`,
		`@@ -27,4 +34,7 @@ sticky sessions, usage metering and a full dashboard.`
27	34
28	35	## Deploy (MacLustr)
29	36	Deployed by `mld` (gateway M1M32) on M3U96a: web 8220 + api 8221, Postgres 17 `fetcha` + Redis local, ngrok `www.fetcha.co`.
30		−Manifest: `deploy/fetcha.mld.json` → `M1M32:~/dispatch/apps/fetcha.json`. See `deploy/README.md`.
	37	+Manifest: `deploy/fetcha.mld.json` → `M1M32:~/dispatch/apps/fetcha.json`. See `deploy/README.md`. Post-sync hooks: `pnpm install`,
	38	+`db:migrate` + `db:seed` (with `ADMIN_EMAILS`), `patchright install chromium`, `next build`. Env knobs: `FETCHA_BROWSER_ENABLED`,
	39	+`FETCHA_BROWSER_CONCURRENCY`, `FETCHA_BROWSER_CHANNEL`, `FETCHA_BROWSER_HEADLESS`, `FETCHA_HTTP2`, `FETCHA_CRAWL_PARALLEL_JOBS`.
	40	+The `browser_enabled` feature flag (admin → Flags) is the global kill switch for browser mode.

modified README.md +4 −2

		`@@ -35,7 +35,9 @@ pnpm dev:web # http://localhost:8220`
35	35	```
36	36	Tests: `pnpm test` · Typecheck: `pnpm typecheck` · Build: `pnpm build`.
37	37
38		−## Status (v0.1.0 — public preview)
	38	+## Status (v0.2.0 — private platform)
39	39	Live: Fetch API, auto routing (residential network class), sticky sessions, Playground, request logs, usage metering,
40		−API keys, projects, admin console, docs, SDK source. Coming: managed browser, structured extraction, Stripe checkout,
	40	+API keys, projects, admin console, docs, SDK source, managed browser rendering (`browser: true`, automatic escalation),
	41	+`format: "markdown"` + page metadata/links, crawl & map jobs (`/v1/crawl`, `/v1/map`). Private platform: invitation-only
	42	+signup, one unlimited plan, no billing. Coming: scripted browser actions, structured extraction,
41	43	teams, webhooks. See `/changelog`.

modified apps/api/package.json +1 −0

		`@@ -11,6 +11,7 @@`
11	11	},
12	12	"dependencies": {
13	13	"@fastify/cors": "^11.0.0",
	14	+ "@fetcha/browser": "workspace:*",
14	15	"@fetcha/core": "workspace:*",
15	16	"@fetcha/db": "workspace:*",
16	17	"@fetcha/email": "workspace:*",

modified apps/api/src/auth.ts +63 −3

		`@@ -1,4 +1,4 @@`
1		−import { FetchaError, hashApiKey, parseApiKeyMode, safeEqual, type ApiKeyScope, type Plan } from "@fetcha/core";
	1	+import { FetchaError, hashApiKey, normalizePlan, parseApiKeyMode, safeEqual, type ApiKeyScope, type Plan } from "@fetcha/core";
2	2	import { apiKeys, db, eq, organizations, projects, sql, users } from "@fetcha/db";
3	3	import { config } from "./config";
4	4	import { getKV } from "./redis";
		`@@ -74,7 +74,7 @@ async function loadPrincipalByKeyHash(hash: string): Promise<ApiPrincipal \| null`
74	74	scopes: (row.scopes ?? []) as ApiKeyScope[],
75	75	organizationId: row.organizationId,
76	76	organizationName: row.organizationName,
77		− plan: row.plan as Plan,
	77	+ plan: normalizePlan(row.plan),
78	78	providerVisibility: row.providerVisibility,
79	79	suspended: row.suspended,
80	80	orgSoftLimitUsd: row.orgSoft,
		`@@ -172,7 +172,67 @@ export async function principalForProject(projectId: string, userId: string): Pr`
172	172	scopes: ["fetch:execute", "sessions:write", "usage:read"],
173	173	organizationId: row.organizationId,
174	174	organizationName: row.organizationName,
175		− plan: row.plan as Plan,
	175	+ plan: normalizePlan(row.plan),
	176	+ providerVisibility: row.providerVisibility,
	177	+ suspended: row.suspended,
	178	+ orgSoftLimitUsd: row.orgSoft,
	179	+ orgHardLimitUsd: row.orgHard,
	180	+ projectId: row.projectId,
	181	+ projectName: row.projectName,
	182	+ projectLogLevel: row.logLevel as ApiPrincipal["projectLogLevel"],
	183	+ projectSoftLimitUsd: row.projSoft,
	184	+ projectHardLimitUsd: row.projHard,
	185	+ projectMonthlyRequestLimit: row.projMonthly,
	186	+ ownerEmail: row.ownerEmail,
	187	+ ownerVerified: row.ownerVerified,
	188	+ internal: true,
	189	+ };
	190	+}
	191	+
	192	+/** Principal for a background crawl job: the project's owner context, with the originating key when still valid. */
	193	+export async function principalForCrawl(projectId: string, apiKeyId: string \| null): Promise<ApiPrincipal> {
	194	+ const [row] = await db
	195	+ .select({
	196	+ organizationId: organizations.id,
	197	+ organizationName: organizations.name,
	198	+ plan: organizations.plan,
	199	+ providerVisibility: organizations.providerVisibility,
	200	+ suspended: organizations.suspended,
	201	+ orgSoft: organizations.softLimitUsd,
	202	+ orgHard: organizations.hardLimitUsd,
	203	+ projectId: projects.id,
	204	+ projectName: projects.name,
	205	+ logLevel: projects.logLevel,
	206	+ projSoft: projects.softLimitUsd,
	207	+ projHard: projects.hardLimitUsd,
	208	+ projMonthly: projects.monthlyRequestLimit,
	209	+ archivedAt: projects.archivedAt,
	210	+ ownerEmail: users.email,
	211	+ ownerVerified: users.emailVerified,
	212	+ ownerBanned: users.banned,
	213	+ })
	214	+ .from(projects)
	215	+ .innerJoin(organizations, eq(projects.organizationId, organizations.id))
	216	+ .innerJoin(users, eq(organizations.ownerUserId, users.id))
	217	+ .where(eq(projects.id, projectId))
	218	+ .limit(1);
	219	+ if (!row) throw new FetchaError("NOT_FOUND", "Project not found.");
	220	+ if (row.archivedAt) throw new FetchaError("FORBIDDEN", "This project is archived.");
	221	+ if (row.suspended \|\| row.ownerBanned) throw new FetchaError("FORBIDDEN", "This organization is suspended.");
	222	+ let keyName: string \| null = null;
	223	+ if (apiKeyId) {
	224	+ const [k] = await db.select({ name: apiKeys.name, revokedAt: apiKeys.revokedAt }).from(apiKeys).where(eq(apiKeys.id, apiKeyId)).limit(1);
	225	+ if (k?.revokedAt) throw new FetchaError("INVALID_API_KEY", "The API key that started this crawl was revoked.");
	226	+ keyName = k?.name ?? null;
	227	+ }
	228	+ return {
	229	+ keyId: apiKeyId,
	230	+ keyName,
	231	+ mode: "live",
	232	+ scopes: ["fetch:execute", "crawl:execute", "sessions:write", "usage:read"],
	233	+ organizationId: row.organizationId,
	234	+ organizationName: row.organizationName,
	235	+ plan: normalizePlan(row.plan),
176	236	providerVisibility: row.providerVisibility,
177	237	suspended: row.suspended,
178	238	orgSoftLimitUsd: row.orgSoft,

modified apps/api/src/limits.ts +18 −0

		`@@ -23,6 +23,24 @@ export async function acquireConcurrency(p: ApiPrincipal): Promise<() => Promise`
23	23	};
24	24	}
25	25
	26	+/** Reserve a managed-browser slot for the organization (separate, smaller pool than plain fetches). */
	27	+export async function acquireBrowserSlot(p: ApiPrincipal): Promise<() => Promise<void>> {
	28	+ const kv = getKV();
	29	+ const limit = PLAN_LIMITS[p.plan].browser_concurrency;
	30	+ const key = `fch:bconc:${p.organizationId}`;
	31	+ const n = await kv.incrWithTtl(key, CONCURRENCY_TTL_SEC);
	32	+ if (n > limit) {
	33	+ await kv.decr(key);
	34	+ throw new FetchaError("CONCURRENCY_LIMIT", `At most ${limit} browser renders can run at once for your organization.`, { details: { limit, scope: "browser" } });
	35	+ }
	36	+ let released = false;
	37	+ return async () => {
	38	+ if (released) return;
	39	+ released = true;
	40	+ await kv.decr(key).catch(() => {});
	41	+ };
	42	+}
	43	+
26	44	/** Sliding-window rate limits: per key (burst), per organization (sustained), per IP (anonymous abuse). */
27	45	export async function checkRateLimits(p: ApiPrincipal, ip: string \| null): Promise<void> {
28	46	const kv = getKV();

added apps/api/src/routes/crawl.ts +96 −0

		`@@ -0,0 +1,96 @@`
	1	+import type { FastifyInstance } from "fastify";
	2	+import { z } from "zod";
	3	+import { FetchaError } from "@fetcha/core";
	4	+import { authenticateApiKey, principalForProject, requireScope } from "../auth";
	5	+import { cancelCrawl, createCrawl, getCrawl, listCrawlPages, listCrawls, mapSite } from "../services/crawl";
	6	+import { getEngine } from "../services/engine";
	7	+
	8	+const pagesQuery = z.object({
	9	+ cursor: z.string().optional(),
	10	+ limit: z.coerce.number().int().min(1).max(500).optional(),
	11	+ status: z.enum(["pending", "success", "blocked", "failed", "skipped"]).optional(),
	12	+ include_content: z
	13	+ .union([z.literal("true"), z.literal("false"), z.literal("1"), z.literal("0")])
	14	+ .optional()
	15	+ .transform((v) => (v === undefined ? undefined : v === "true" \|\| v === "1")),
	16	+});
	17	+
	18	+const clientIp = (req: { headers: Record<string, unknown>; ip: string }) => ((req.headers["x-forwarded-for"] as string \| undefined)?.split(",")[0]?.trim() ?? req.ip) \|\| null;
	19	+const sourceOf = (req: { headers: Record<string, unknown> }) => (/fetcha-sdk/i.test(String(req.headers["user-agent"] ?? "")) ? "sdk" : "api") as "sdk" \| "api";
	20	+
	21	+/** Public crawl & map endpoints. */
	22	+export async function registerCrawlRoutes(app: FastifyInstance) {
	23	+ app.post("/v1/crawl", async (req, reply) => {
	24	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	25	+ requireScope(principal, "fetch:execute");
	26	+ const job = await createCrawl(principal, req.body, sourceOf(req));
	27	+ return reply.status(202).send(job);
	28	+ });
	29	+ app.get<{ Querystring: { limit?: string } }>("/v1/crawl", async (req) => {
	30	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	31	+ return listCrawls(principal, Number(req.query.limit ?? 50) \|\| 50);
	32	+ });
	33	+ app.get<{ Params: { id: string } }>("/v1/crawl/:id", async (req) => {
	34	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	35	+ return getCrawl(principal, req.params.id);
	36	+ });
	37	+ app.get<{ Params: { id: string }; Querystring: Record<string, string \| undefined> }>("/v1/crawl/:id/pages", async (req) => {
	38	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	39	+ const q = pagesQuery.safeParse(req.query);
	40	+ if (!q.success) throw new FetchaError("INVALID_REQUEST", "Invalid query parameters.", { details: { issues: q.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
	41	+ return listCrawlPages(principal, req.params.id, q.data);
	42	+ });
	43	+ app.delete<{ Params: { id: string } }>("/v1/crawl/:id", async (req) => {
	44	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	45	+ requireScope(principal, "fetch:execute");
	46	+ return cancelCrawl(principal, req.params.id);
	47	+ });
	48	+ app.post("/v1/map", async (req) => {
	49	+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string \| undefined);
	50	+ requireScope(principal, "fetch:execute");
	51	+ return mapSite(principal, req.body, sourceOf(req), clientIp(req));
	52	+ });
	53	+}
	54	+
	55	+/** Internal (dashboard) crawl routes; the service-token check is applied by the internal onRequest hook. */
	56	+export async function registerInternalCrawlRoutes(app: FastifyInstance) {
	57	+ const bodySchema = z.object({ project_id: z.string(), user_id: z.string(), options: z.unknown().optional() });
	58	+ type Q = { project_id: string; user_id: string; limit?: string; cursor?: string; status?: string; include_content?: string };
	59	+
	60	+ app.post("/internal/crawls", async (req, reply) => {
	61	+ const parsed = bodySchema.safeParse(req.body);
	62	+ if (!parsed.success) throw new FetchaError("INVALID_REQUEST", "project_id, user_id and options are required.");
	63	+ const principal = await principalForProject(parsed.data.project_id, parsed.data.user_id);
	64	+ const job = await createCrawl(principal, parsed.data.options ?? {}, "playground");
	65	+ return reply.status(202).send(job);
	66	+ });
	67	+ app.get<{ Querystring: Q }>("/internal/crawls", async (req) => {
	68	+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
	69	+ return listCrawls(principal, Number(req.query.limit ?? 50) \|\| 50);
	70	+ });
	71	+ app.get<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id", async (req) => {
	72	+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
	73	+ return getCrawl(principal, req.params.id);
	74	+ });
	75	+ app.get<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id/pages", async (req) => {
	76	+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
	77	+ const q = pagesQuery.safeParse(req.query);
	78	+ if (!q.success) throw new FetchaError("INVALID_REQUEST", "Invalid query parameters.");
	79	+ return listCrawlPages(principal, req.params.id, q.data);
	80	+ });
	81	+ app.delete<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id", async (req) => {
	82	+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
	83	+ return cancelCrawl(principal, req.params.id);
	84	+ });
	85	+ app.post("/internal/map", async (req) => {
	86	+ const parsed = bodySchema.safeParse(req.body);
	87	+ if (!parsed.success) throw new FetchaError("INVALID_REQUEST", "project_id, user_id and options are required.");
	88	+ const principal = await principalForProject(parsed.data.project_id, parsed.data.user_id);
	89	+ return mapSite(principal, parsed.data.options ?? {}, "playground", (req.headers["x-forwarded-for"] as string \| undefined)?.split(",")[0]?.trim() ?? req.ip);
	90	+ });
	91	+ app.get("/internal/browser", async () => {
	92	+ const engine = await getEngine();
	93	+ const s = engine.browser.status();
	94	+ return { ...s, enabled: engine.browserEnabled, flag_enabled: engine.browserEnabled \|\| !engine.browser.enabled ? engine.browserEnabled : false };
	95	+ });
	96	+}

modified apps/api/src/routes/fetch.ts +34 −11

		`@@ -1,11 +1,11 @@`
1	1	import { FetchaError, PLAN_LIMITS, extractDomain, fetchRequestSchema, newId, type FetchRequest } from "@fetcha/core";
2	2	import { db, eq, proxySessions, sql } from "@fetcha/db";
3	3	import type { ProviderId } from "@fetcha/providers";
4		−import type { AttemptRecord } from "@fetcha/routing";
	4	+import type { AttemptRecord, SerializedCookie } from "@fetcha/routing";
5	5	import type { ConcreteNetwork } from "@fetcha/core";
6	6	import type { ApiPrincipal } from "../auth";
7	7	import { config } from "../config";
8		−import { acquireConcurrency, checkMonthlyLimits, checkRateLimits, invalidateUsageCache } from "../limits";
	8	+import { acquireBrowserSlot, acquireConcurrency, checkMonthlyLimits, checkRateLimits, invalidateUsageCache } from "../limits";
9	9	import { getEngine } from "../services/engine";
10	10	import { completeRequest, createRequestRow, failRequest, recordAttempt, type RequestContext } from "../services/persist";
11	11	import { checkSpendAlerts, notifyHardLimit, recordAbuse } from "../services/alerts";
		`@@ -13,7 +13,7 @@ import { checkSpendAlerts, notifyHardLimit, recordAbuse } from "../services/aler`
13	13	export interface FetchHandlerInput {
14	14	principal: ApiPrincipal;
15	15	body: unknown;
16		− source: "api" \| "playground" \| "sdk";
	16	+ source: "api" \| "playground" \| "sdk" \| "crawl";
17	17	clientIp: string \| null;
18	18	userAgent: string \| null;
19	19	}
		`@@ -22,10 +22,12 @@ export interface FetchHandlerOutput {`
22	22	requestId: string;
23	23	status: number;
24	24	body: unknown;
	25	+ /** Internal: execution summary for callers that need more than the public document (crawler). */
	26	+ summary?: { mode: "http" \| "browser"; bytes: number; costUsd: number; attempts: number; domain: string };
25	27	}
26	28
27	29	/**
28		− * Shared /v1/fetch pipeline used by the public API and the dashboard playground.
	30	+ * Shared /v1/fetch pipeline used by the public API, the dashboard playground and the crawler.
29	31	* Order matters: validate → limits → SSRF (inside executor) → route → persist.
30	32	*/
31	33	export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandlerOutput> {
		`@@ -46,12 +48,12 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
46	48	if (req.network !== "auto" && !limits.networks.includes(req.network)) {
47	49	throw new FetchaError("NETWORK_UNAVAILABLE", `The "${req.network}" network is not included in the ${limits.label} plan.`, { requestId });
48	50	}
49		− if (req.browser) throw new FetchaError("BROWSER_UNAVAILABLE", undefined, { requestId });
	51	+ const engine = await getEngine();
	52	+ if (req.browser && (!limits.browser \|\| !engine.browserEnabled)) throw new FetchaError("BROWSER_UNAVAILABLE", undefined, { requestId });
50	53
51	54	await checkRateLimits(principal, input.clientIp);
52		− let monthly: Awaited<ReturnType<typeof checkMonthlyLimits>>;
53	55	try {
54		− monthly = await checkMonthlyLimits(principal);
	56	+ await checkMonthlyLimits(principal);
55	57	} catch (e) {
56	58	if (e instanceof FetchaError && e.code === "USAGE_LIMIT_REACHED" && (principal.projectHardLimitUsd !== null \|\| principal.orgHardLimitUsd !== null)) {
57	59	const { monthlyUsage } = await import("../limits");
		`@@ -60,6 +62,15 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
60	62	throw e;
61	63	}
62	64	const release = await acquireConcurrency(principal);
	65	+ let releaseBrowser: (() => Promise<void>) \| null = null;
	66	+ if (req.browser) {
	67	+ try {
	68	+ releaseBrowser = await acquireBrowserSlot(principal);
	69	+ } catch (e) {
	70	+ await release();
	71	+ throw e;
	72	+ }
	73	+ }
63	74
64	75	const domain = extractDomain(req.url);
65	76	const ctx: RequestContext = {
		`@@ -77,19 +88,22 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
77	88	const attempts: AttemptRecord[] = [];
78	89	try {
79	90	await createRequestRow(ctx, req, domain);
80		− const engine = await getEngine();
81	91
82		− // Resolve a Fetcha session (sess_…) into a provider-pinned sticky key.
	92	+ // Resolve a Fetcha session (sess_…) into a provider-pinned sticky key (+ its cookie jar).
83	93	let sessionKey: string \| null = null;
84	94	let sessionProvider: ProviderId \| null = null;
85	95	let sessionNetwork: ConcreteNetwork \| null = null;
	96	+ let sessionCookies: SerializedCookie[] \| null = null;
	97	+ let sessionId: string \| null = null;
86	98	if (req.session) {
87	99	const [s] = await db.select().from(proxySessions).where(eq(proxySessions.id, req.session)).limit(1);
88	100	if (!s \|\| s.projectId !== principal.projectId) throw new FetchaError("SESSION_NOT_FOUND", undefined, { requestId });
89	101	if (s.status !== "active" \|\| s.expiresAt.getTime() < Date.now()) throw new FetchaError("SESSION_EXPIRED", undefined, { requestId });
90	102	sessionKey = s.stickyKey;
	103	+ sessionId = s.id;
91	104	sessionProvider = s.provider as ProviderId;
92	105	sessionNetwork = s.network as ConcreteNetwork;
	106	+ sessionCookies = (s.cookies ?? []) as unknown as SerializedCookie[];
93	107	if (!req.country && s.country) req.country = s.country;
94	108	if (req.network === "auto") req.network = s.network as FetchRequest["network"];
95	109	db.update(proxySessions)
		`@@ -106,9 +120,12 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
106	120	sessionKey,
107	121	sessionProvider,
108	122	sessionNetwork,
	123	+ sessionCookies,
109	124	knowledge,
110	125	providerVisibility: principal.providerVisibility,
111	126	maxResponseBytes: Math.min(req.max_response_bytes ?? config.maxResponseBytesDefault, config.maxResponseBytesDefault),
	127	+ browserPool: engine.browserEnabled && limits.browser ? engine.browser : null,
	128	+ browserAllowed: limits.browser,
112	129	onAttempt: async (a) => {
113	130	attempts.push(a);
114	131	await recordAttempt(requestId, a).catch(() => {});
		`@@ -116,13 +133,18 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
116	133	});
117	134	const latencyMs = Math.round(performance.now() - started);
118	135	await completeRequest(ctx, req, result, latencyMs);
	136	+ if (sessionId && result.cookies.length) {
	137	+ db.update(proxySessions)
	138	+ .set({ cookies: result.cookies.slice(-200) as unknown as Array<{ name: string; value: string; domain?: string; path?: string }> })
	139	+ .where(eq(proxySessions.id, sessionId))
	140	+ .catch(() => {});
	141	+ }
119	142	await invalidateUsageCache(principal.organizationId, principal.projectId);
120	143	if (principal.projectSoftLimitUsd !== null \|\| principal.orgSoftLimitUsd !== null) {
121	144	const { monthlyUsage } = await import("../limits");
122	145	monthlyUsage(principal).then((u) => checkSpendAlerts(principal, u)).catch(() => {});
123	146	}
124		− void monthly;
125		− return { requestId, status: 200, body: result.body };
	147	+ return { requestId, status: 200, body: result.body, summary: { mode: result.mode, bytes: result.bytesIn + result.bytesOut, costUsd: result.costUsd, attempts: result.attempts.length, domain } };
126	148	} catch (e) {
127	149	const latencyMs = Math.round(performance.now() - started);
128	150	const err = e instanceof FetchaError ? e : new FetchaError("INTERNAL_ERROR", undefined, { requestId, cause: e });
		`@@ -131,6 +153,7 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle`
131	153	if (!(e instanceof FetchaError)) console.error(`[fetch] ${requestId} internal error`, e);
132	154	throw Object.assign(err, { requestId });
133	155	} finally {
	156	+ await releaseBrowser?.();
134	157	await release();
135	158	}
136	159	}

modified apps/api/src/routes/misc.ts +3 −1

		`@@ -40,7 +40,9 @@ export async function usageResponse(p: ApiPrincipal) {`
40	40	latency_p50_ms: Math.round(agg?.p50 ?? 0),
41	41	latency_p95_ms: Math.round(agg?.p95 ?? 0),
42	42	},
43		− remaining_requests: Math.max(0, limits.monthly_requests - usage.requests),
	43	+ remaining_requests: limits.monthly_requests >= Number.MAX_SAFE_INTEGER ? null : Math.max(0, limits.monthly_requests - usage.requests),
	44	+ browser: { enabled: true, concurrency: limits.browser_concurrency },
	45	+ crawl: { max_pages: limits.crawl_max_pages, concurrent_jobs: limits.crawl_concurrent_jobs },
44	46	};
45	47	}
46	48

modified apps/api/src/server.ts +13 −4

		`@@ -3,11 +3,14 @@ import cors from "@fastify/cors";`
3	3	import { FetchaError, isFetchaError, newId } from "@fetcha/core";
4	4	import { closeDb, db, sql } from "@fetcha/db";
5	5	import { closeAllDispatchers } from "@fetcha/providers";
	6	+import { closeBrowserPool } from "@fetcha/browser";
6	7	import { authenticateApiKey, requireScope } from "./auth";
7	8	import { assertProductionConfig, config } from "./config";
8	9	import { closeKV, getKV } from "./redis";
9	10	import { handleFetch } from "./routes/fetch";
10	11	import { registerInternalRoutes } from "./routes/internal";
	12	+import { registerCrawlRoutes, registerInternalCrawlRoutes } from "./routes/crawl";
	13	+import { startCrawlWorker } from "./services/crawl";
11	14	import { meResponse, usageResponse } from "./routes/misc";
12	15	import { closeSession, createSession, getSession, listSessions } from "./routes/sessions";
13	16	import { getEngine, startHealthLoop } from "./services/engine";
		`@@ -74,7 +77,7 @@ export async function buildServer() {`
74	77	const engine = await getEngine().catch(() => null);
75	78	checks.providers = Boolean(engine && engine.registry.available().length > 0);
76	79	const ok = checks.database && checks.cache && checks.providers;
77		− return reply.status(ok ? 200 : 503).send({ status: ok ? "ready" : "degraded", checks, available_networks: engine?.registry.availableNetworks() ?? [] });
	80	+ return reply.status(ok ? 200 : 503).send({ status: ok ? "ready" : "degraded", checks, available_networks: engine?.registry.availableNetworks() ?? [], browser: engine ? { enabled: engine.browserEnabled, launched: engine.browser.status().launched } : null });
78	81	});
79	82
80	83	// ---- Public API v1 ----------------------------------------------------------
		`@@ -118,15 +121,18 @@ export async function buildServer() {`
118	121	return usageResponse(principal);
119	122	});
120	123
	124	+ await registerCrawlRoutes(app);
	125	+
121	126	// Explicitly unavailable surfaces (never fake functionality).
122	127	app.post("/v1/browser", async () => {
123		− throw new FetchaError("BROWSER_UNAVAILABLE", "Browser actions are not yet available. Follow the changelog at https://www.fetcha.co/changelog.");
	128	+ throw new FetchaError("BROWSER_UNAVAILABLE", "Scripted browser actions are not available yet. Rendered fetches are: send `browser: true` to POST /v1/fetch.");
124	129	});
125	130	app.post("/v1/extract", async () => {
126		− throw new FetchaError("INVALID_REQUEST", "Structured extraction is not yet available on this plan. Follow the changelog at https://www.fetcha.co/changelog.");
	131	+ throw new FetchaError("INVALID_REQUEST", "Structured extraction is not yet available. Use `format: \"markdown\"` on POST /v1/fetch in the meantime.");
127	132	});
128	133
129	134	await registerInternalRoutes(app);
	135	+ await registerInternalCrawlRoutes(app);
130	136	return app;
131	137	}
132	138
		`@@ -136,12 +142,15 @@ async function main() {`
136	142	const engine = await getEngine();
137	143	app.log.info({ providers: engine.registry.available().map((p) => p.id), networks: engine.registry.availableNetworks() }, "providers loaded");
138	144	const stopHealth = startHealthLoop(engine, app.log);
	145	+ const stopCrawl = startCrawlWorker(app.log);
	146	+ app.log.info({ browser: engine.browser.status() }, "managed browser pool ready (lazy launch)");
139	147
140	148	const shutdown = async (signal: string) => {
141	149	app.log.info(`${signal} received, shutting down`);
142	150	stopHealth();
	151	+ await Promise.race([stopCrawl(), new Promise((r) => setTimeout(r, 8000))]);
143	152	await app.close();
144		− await Promise.all([closeAllDispatchers(), closeKV(), closeDb()]);
	153	+ await Promise.all([closeAllDispatchers(), closeBrowserPool(), closeKV(), closeDb()]);
145	154	process.exit(0);
146	155	};
147	156	process.on("SIGINT", () => void shutdown("SIGINT"));

added apps/api/src/services/crawl.ts +592 −0

		`@@ -0,0 +1,592 @@`
	1	+import {
	2	+ FetchaError,
	3	+ PLAN_LIMITS,
	4	+ assertUrlAllowed,
	5	+ crawlCreateSchema,
	6	+ extractDomain,
	7	+ extractPageMetadata,
	8	+ looksLikePage,
	9	+ mapCreateSchema,
	10	+ newId,
	11	+ normalizeCrawlUrl,
	12	+ parseRobots,
	13	+ parseSitemap,
	14	+ registrableHost,
	15	+ robotsAllows,
	16	+ urlPatternMatcher,
	17	+ type CrawlCreateInput,
	18	+ type FetchResponseBody,
	19	+ type MapCreateInput,
	20	+ type RobotsRules,
	21	+} from "@fetcha/core";
	22	+import { and, crawlJobs, crawlPages, db, desc, eq, inArray, lt, sql, type CrawlJob, type CrawlPage } from "@fetcha/db";
	23	+import { hostname } from "node:os";
	24	+import type { ApiPrincipal } from "../auth";
	25	+import { handleFetch, type FetchHandlerOutput } from "../routes/fetch";
	26	+
	27	+/**
	28	+ * Crawl service: durable jobs (crawl_jobs / crawl_pages) executed in-process by a small worker
	29	+ * loop. Every page goes through `handleFetch`, so quotas, routing intelligence, block detection and
	30	+ * browser escalation apply exactly as for a single fetch.
	31	+ */
	32	+
	33	+const WORKER_ID = `${hostname()}:${process.pid}`;
	34	+const PAGE_CONTENT_CAP = 600_000; // chars stored per page
	35	+const HEARTBEAT_MS = 10_000;
	36	+const STALE_MS = 3 * 60_000;
	37	+
	38	+export function serializeJob(j: CrawlJob) {
	39	+ return {
	40	+ id: j.id,
	41	+ status: j.status,
	42	+ label: j.label,
	43	+ seed_url: j.seedUrl,
	44	+ domain: j.domain,
	45	+ source: j.source,
	46	+ options: j.options,
	47	+ stats: { discovered: j.pagesDiscovered, fetched: j.pagesFetched, ok: j.pagesOk, blocked: j.pagesBlocked, failed: j.pagesFailed, bytes: j.bytes },
	48	+ error: j.errorCode ? { code: j.errorCode, message: j.errorMessage } : null,
	49	+ webhook_status: j.webhookStatus,
	50	+ created_at: j.createdAt.toISOString(),
	51	+ started_at: j.startedAt?.toISOString() ?? null,
	52	+ completed_at: j.completedAt?.toISOString() ?? null,
	53	+ };
	54	+}
	55	+
	56	+export function serializePage(p: CrawlPage, includeContent = true) {
	57	+ return {
	58	+ id: p.id,
	59	+ url: p.url,
	60	+ final_url: p.finalUrl,
	61	+ depth: p.depth,
	62	+ parent_url: p.parentUrl,
	63	+ status: p.status,
	64	+ http_status: p.httpStatus,
	65	+ error_code: p.errorCode,
	66	+ request_id: p.requestId,
	67	+ title: p.title,
	68	+ description: p.description,
	69	+ content_type: p.contentType,
	70	+ content: includeContent ? p.content : undefined,
	71	+ links_count: p.linksCount,
	72	+ bytes: p.bytes,
	73	+ duration_ms: p.durationMs,
	74	+ mode: p.mode,
	75	+ fetched_at: p.fetchedAt?.toISOString() ?? null,
	76	+ };
	77	+}
	78	+
	79	+// ---------------------------------------------------------------------------
	80	+// Public operations
	81	+// ---------------------------------------------------------------------------
	82	+export async function createCrawl(principal: ApiPrincipal, body: unknown, source: "api" \| "playground" \| "sdk") {
	83	+ const parsed = crawlCreateSchema.safeParse(body ?? {});
	84	+ if (!parsed.success) {
	85	+ throw new FetchaError("INVALID_REQUEST", "Invalid crawl options.", { details: { issues: parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
	86	+ }
	87	+ const opts = parsed.data;
	88	+ const limits = PLAN_LIMITS[principal.plan];
	89	+ if (opts.max_pages > limits.crawl_max_pages) opts.max_pages = limits.crawl_max_pages;
	90	+ if (opts.timeout > limits.max_timeout_ms) opts.timeout = limits.max_timeout_ms;
	91	+ if (opts.network !== "auto" && !limits.networks.includes(opts.network)) throw new FetchaError("NETWORK_UNAVAILABLE");
	92	+ const allowed = await assertUrlAllowed(opts.url);
	93	+ const seed = normalizeCrawlUrl(allowed.url.toString());
	94	+ if (!seed) throw new FetchaError("INVALID_REQUEST", "The seed URL is invalid.");
	95	+ if (opts.webhook_url) await assertUrlAllowed(opts.webhook_url);
	96	+
	97	+ const [active] = await db
	98	+ .select({ n: sql<number>`count(*)::int` })
	99	+ .from(crawlJobs)
	100	+ .where(and(eq(crawlJobs.organizationId, principal.organizationId), inArray(crawlJobs.status, ["queued", "running"])));
	101	+ if ((active?.n ?? 0) >= limits.crawl_concurrent_jobs) {
	102	+ throw new FetchaError("CRAWL_LIMIT_REACHED", `At most ${limits.crawl_concurrent_jobs} crawl jobs can be queued or running at once.`, { details: { limit: limits.crawl_concurrent_jobs } });
	103	+ }
	104	+
	105	+ const id = newId("crawl");
	106	+ await db.insert(crawlJobs).values({
	107	+ id,
	108	+ organizationId: principal.organizationId,
	109	+ projectId: principal.projectId,
	110	+ apiKeyId: principal.keyId,
	111	+ source,
	112	+ label: opts.label ?? null,
	113	+ seedUrl: seed,
	114	+ domain: extractDomain(seed),
	115	+ options: { ...opts, url: seed } as Record<string, unknown>,
	116	+ status: "queued",
	117	+ pagesDiscovered: 1,
	118	+ });
	119	+ await db.insert(crawlPages).values({ id: newId("cpg"), jobId: id, url: seed, depth: 0, status: "pending" }).onConflictDoNothing();
	120	+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, id)).limit(1);
	121	+ kick();
	122	+ return serializeJob(job!);
	123	+}
	124	+
	125	+export async function getCrawl(principal: ApiPrincipal, id: string) {
	126	+ const [job] = await db.select().from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
	127	+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
	128	+ return serializeJob(job);
	129	+}
	130	+
	131	+export async function listCrawls(principal: ApiPrincipal, limit = 50) {
	132	+ const rows = await db.select().from(crawlJobs).where(eq(crawlJobs.projectId, principal.projectId)).orderBy(desc(crawlJobs.createdAt)).limit(Math.min(Math.max(1, limit), 200));
	133	+ return { data: rows.map(serializeJob) };
	134	+}
	135	+
	136	+export async function listCrawlPages(principal: ApiPrincipal, id: string, q: { cursor?: string; limit?: number; status?: string; include_content?: boolean }) {
	137	+ const [job] = await db.select({ id: crawlJobs.id }).from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
	138	+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
	139	+ const limit = Math.min(Math.max(1, q.limit ?? 100), 500);
	140	+ const conds = [eq(crawlPages.jobId, id)];
	141	+ if (q.status) conds.push(eq(crawlPages.status, q.status));
	142	+ else conds.push(sql`${crawlPages.status} <> 'pending'`);
	143	+ if (q.cursor) {
	144	+ const [ts, cid] = decodeCursor(q.cursor);
	145	+ conds.push(sql`(${crawlPages.createdAt}, ${crawlPages.id}) > (${new Date(ts)}, ${cid})`);
	146	+ }
	147	+ const rows = await db
	148	+ .select()
	149	+ .from(crawlPages)
	150	+ .where(and(...conds))
	151	+ .orderBy(crawlPages.createdAt, crawlPages.id)
	152	+ .limit(limit + 1);
	153	+ const page = rows.slice(0, limit);
	154	+ const last = page.at(-1);
	155	+ return { data: page.map((p) => serializePage(p, q.include_content !== false)), next_cursor: rows.length > limit && last ? encodeCursor(last.createdAt, last.id) : null };
	156	+}
	157	+
	158	+export async function cancelCrawl(principal: ApiPrincipal, id: string) {
	159	+ const res = await db
	160	+ .update(crawlJobs)
	161	+ .set({ status: "cancelled", completedAt: new Date() })
	162	+ .where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId), inArray(crawlJobs.status, ["queued", "running"])))
	163	+ .returning({ id: crawlJobs.id });
	164	+ if (!res.length) {
	165	+ const [job] = await db.select({ id: crawlJobs.id, status: crawlJobs.status }).from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
	166	+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
	167	+ return { id, status: job.status };
	168	+ }
	169	+ cancelled.add(id);
	170	+ return { id, status: "cancelled" };
	171	+}
	172	+
	173	+function encodeCursor(d: Date, id: string): string {
	174	+ return Buffer.from(`${d.toISOString()}\|${id}`).toString("base64url");
	175	+}
	176	+function decodeCursor(c: string): [string, string] {
	177	+ try {
	178	+ const [ts, id] = Buffer.from(c, "base64url").toString("utf8").split("\|");
	179	+ if (!ts \|\| !id \|\| Number.isNaN(Date.parse(ts))) throw new Error("bad");
	180	+ return [ts, id];
	181	+ } catch {
	182	+ throw new FetchaError("INVALID_REQUEST", "Invalid cursor.");
	183	+ }
	184	+}
	185	+
	186	+// ---------------------------------------------------------------------------
	187	+// Map (sync)
	188	+// ---------------------------------------------------------------------------
	189	+export async function mapSite(principal: ApiPrincipal, body: unknown, source: "api" \| "playground" \| "sdk", clientIp: string \| null) {
	190	+ const parsed = mapCreateSchema.safeParse(body ?? {});
	191	+ if (!parsed.success) {
	192	+ throw new FetchaError("INVALID_REQUEST", "Invalid map options.", { details: { issues: parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
	193	+ }
	194	+ const opts = parsed.data;
	195	+ const allowed = await assertUrlAllowed(opts.url);
	196	+ const seed = normalizeCrawlUrl(allowed.url.toString())!;
	197	+ const seedUrl = new URL(seed);
	198	+ const host = registrableHost(seedUrl.hostname);
	199	+ const inScope = scopeFilter(seedUrl, opts.same_domain, opts.allow_subdomains);
	200	+ const search = opts.search ? urlPatternMatcher([opts.search.includes("") \|\| /^\/.+\/[a-z]$/.test(opts.search) ? opts.search : `${opts.search}`]) : null;
	201	+ const deadline = Date.now() + Math.min(55_000, opts.timeout * 2);
	202	+ const found = new Map<string, "sitemap" \| "links">();
	203	+ const add = (u: string, src: "sitemap" \| "links") => {
	204	+ const n = normalizeCrawlUrl(u);
	205	+ if (!n \|\| found.has(n) \|\| !inScope(n) \|\| !looksLikePage(n)) return;
	206	+ if (search && !search(n)) return;
	207	+ found.set(n, src);
	208	+ };
	209	+ const fetchText = async (url: string, format: "raw" \| "html" = "raw", extra: Record<string, unknown> = {}): Promise<FetchResponseBody \| null> => {
	210	+ if (Date.now() > deadline - 2000) return null;
	211	+ try {
	212	+ const out = await handleFetch({ principal, body: { url, format, timeout: Math.min(opts.timeout, Math.max(3000, deadline - Date.now() - 1000)), country: opts.country, network: opts.network, browser_fallback: false, retries: 1, ...extra }, source, clientIp, userAgent: `fetcha-map/${source}` });
	213	+ return out.body as FetchResponseBody;
	214	+ } catch {
	215	+ return null;
	216	+ }
	217	+ };
	218	+ let sitemapCount = 0;
	219	+ let linksCount = 0;
	220	+ if (opts.use_sitemap) {
	221	+ const sitemapUrls = new Set<string>([`${seedUrl.origin}/sitemap.xml`, `${seedUrl.origin}/sitemap_index.xml`]);
	222	+ const robots = await fetchText(`${seedUrl.origin}/robots.txt`);
	223	+ if (robots?.success && robots.content) for (const s of parseRobots(robots.content).sitemaps) sitemapUrls.add(s);
	224	+ const queue = [...sitemapUrls];
	225	+ const seen = new Set<string>();
	226	+ while (queue.length && seen.size < 25 && found.size < opts.limit && Date.now() < deadline - 3000) {
	227	+ const u = queue.shift()!;
	228	+ if (seen.has(u)) continue;
	229	+ seen.add(u);
	230	+ const r = await fetchText(u);
	231	+ if (!r?.success \|\| !r.content \|\| !/<(urlset\|sitemapindex)\|<loc>/i.test(r.content.slice(0, 5000)) && !/^https?:\/\//m.test(r.content.slice(0, 500))) continue;
	232	+ const { urls, sitemaps } = parseSitemap(r.content);
	233	+ for (const s of sitemaps) if (host === safeHost(s) \|\| opts.allow_subdomains) queue.push(s);
	234	+ for (const x of urls) {
	235	+ if (found.size >= opts.limit) break;
	236	+ const before = found.size;
	237	+ add(x, "sitemap");
	238	+ if (found.size > before) sitemapCount++;
	239	+ }
	240	+ }
	241	+ }
	242	+ if (opts.use_links && found.size < opts.limit && Date.now() < deadline - 3000) {
	243	+ const r = await fetchText(seed, "html", { links: true });
	244	+ if (r?.success && r.links) {
	245	+ for (const l of r.links) {
	246	+ if (found.size >= opts.limit) break;
	247	+ const before = found.size;
	248	+ add(l.url, "links");
	249	+ if (found.size > before) linksCount++;
	250	+ }
	251	+ } else if (r?.content) {
	252	+ const meta = extractPageMetadata(r.content, r.final_url);
	253	+ for (const l of meta.links) {
	254	+ if (found.size >= opts.limit) break;
	255	+ const before = found.size;
	256	+ add(l.url, "links");
	257	+ if (found.size > before) linksCount++;
	258	+ }
	259	+ }
	260	+ }
	261	+ const urls = [...found.keys()];
	262	+ return { url: seed, count: urls.length, urls, sources: { sitemap: sitemapCount, links: linksCount }, truncated: urls.length >= opts.limit };
	263	+}
	264	+
	265	+function safeHost(u: string): string {
	266	+ try {
	267	+ return registrableHost(new URL(u).hostname);
	268	+ } catch {
	269	+ return "";
	270	+ }
	271	+}
	272	+
	273	+function scopeFilter(seed: URL, sameDomain: boolean, allowSubdomains: boolean): (url: string) => boolean {
	274	+ const seedHost = seed.hostname.toLowerCase().replace(/^www\./, "");
	275	+ const reg = registrableHost(seed.hostname);
	276	+ return (url: string) => {
	277	+ if (!sameDomain) return true;
	278	+ let h: string;
	279	+ try {
	280	+ h = new URL(url).hostname.toLowerCase().replace(/^www\./, "");
	281	+ } catch {
	282	+ return false;
	283	+ }
	284	+ if (h === seedHost) return true;
	285	+ if (allowSubdomains) return registrableHost(h) === reg;
	286	+ return false;
	287	+ };
	288	+}
	289	+
	290	+// ---------------------------------------------------------------------------
	291	+// Worker
	292	+// ---------------------------------------------------------------------------
	293	+const cancelled = new Set<string>();
	294	+let running = false;
	295	+let stopped = false;
	296	+let wake: (() => void) \| null = null;
	297	+const activeJobs = new Map<string, Promise<void>>();
	298	+const MAX_PARALLEL_JOBS = Number(process.env.FETCHA_CRAWL_PARALLEL_JOBS ?? 4);
	299	+
	300	+function kick() {
	301	+ wake?.();
	302	+}
	303	+
	304	+export function startCrawlWorker(log: { info: (m: string) => void; warn: (m: string) => void }): () => Promise<void> {
	305	+ if (running) return async () => {};
	306	+ running = true;
	307	+ stopped = false;
	308	+ const loop = async () => {
	309	+ // Re-queue jobs left "running" by a previous process (crash / redeploy).
	310	+ await db
	311	+ .update(crawlJobs)
	312	+ .set({ status: "queued", workerId: null })
	313	+ .where(and(eq(crawlJobs.status, "running"), lt(crawlJobs.heartbeatAt, new Date(Date.now() - STALE_MS))))
	314	+ .catch(() => {});
	315	+ while (!stopped) {
	316	+ try {
	317	+ if (activeJobs.size < MAX_PARALLEL_JOBS) {
	318	+ const claimed = await claimJob();
	319	+ if (claimed) {
	320	+ const p = runJob(claimed, log)
	321	+ .catch((e) => log.warn(`crawl ${claimed.id} crashed: ${(e as Error).message}`))
	322	+ .finally(() => activeJobs.delete(claimed.id));
	323	+ activeJobs.set(claimed.id, p);
	324	+ continue;
	325	+ }
	326	+ }
	327	+ } catch (e) {
	328	+ log.warn(`crawl worker loop error: ${(e as Error).message}`);
	329	+ }
	330	+ await new Promise<void>((r) => {
	331	+ wake = r;
	332	+ setTimeout(r, 4000);
	333	+ });
	334	+ wake = null;
	335	+ }
	336	+ };
	337	+ void loop();
	338	+ log.info(`crawl worker started (${WORKER_ID}, parallel jobs=${MAX_PARALLEL_JOBS})`);
	339	+ return async () => {
	340	+ stopped = true;
	341	+ kick();
	342	+ await Promise.allSettled([...activeJobs.values()]);
	343	+ running = false;
	344	+ };
	345	+}
	346	+
	347	+async function claimJob(): Promise<CrawlJob \| null> {
	348	+ const rows = await db.execute(sql`
	349	+ update crawl_jobs set status = 'running', worker_id = ${WORKER_ID}, started_at = coalesce(started_at, now()), heartbeat_at = now()
	350	+ where id = (select id from crawl_jobs where status = 'queued' order by created_at asc limit 1 for update skip locked)
	351	+ returning *
	352	+ `);
	353	+ const r = (rows as unknown as { rows: Record<string, unknown>[] }).rows?.[0];
	354	+ if (!r) return null;
	355	+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, r.id as string)).limit(1);
	356	+ return job ?? null;
	357	+}
	358	+
	359	+interface Frontier {
	360	+ url: string;
	361	+ depth: number;
	362	+ parent: string \| null;
	363	+}
	364	+
	365	+async function runJob(job: CrawlJob, log: { info: (m: string) => void; warn: (m: string) => void }): Promise<void> {
	366	+ const opts = crawlCreateSchema.parse(job.options) as CrawlCreateInput;
	367	+ const { principalForCrawl } = await import("../auth");
	368	+ let principal: ApiPrincipal;
	369	+ try {
	370	+ principal = await principalForCrawl(job.projectId, job.apiKeyId);
	371	+ } catch (e) {
	372	+ await finishJob(job.id, "failed", { code: (e as FetchaError).code ?? "INTERNAL_ERROR", message: (e as Error).message });
	373	+ return;
	374	+ }
	375	+ const seedUrl = new URL(job.seedUrl);
	376	+ const inScope = scopeFilter(seedUrl, opts.same_domain, opts.allow_subdomains);
	377	+ const include = urlPatternMatcher(opts.include_patterns);
	378	+ const exclude = urlPatternMatcher(opts.exclude_patterns);
	379	+ const seen = new Set<string>();
	380	+ const frontier: Frontier[] = [];
	381	+ let discovered = 0;
	382	+ let fetched = 0;
	383	+ let ok = 0;
	384	+ let blocked = 0;
	385	+ let failed = 0;
	386	+ let bytes = 0;
	387	+ let costUsd = 0;
	388	+ let robots: RobotsRules \| null = null;
	389	+ let delayMs = opts.delay_ms;
	390	+
	391	+ // Resume support: already-known pages of this job.
	392	+ const existing = await db.select({ url: crawlPages.url, status: crawlPages.status, depth: crawlPages.depth, parentUrl: crawlPages.parentUrl }).from(crawlPages).where(eq(crawlPages.jobId, job.id));
	393	+ for (const p of existing) {
	394	+ seen.add(p.url);
	395	+ discovered++;
	396	+ if (p.status === "pending") frontier.push({ url: p.url, depth: p.depth, parent: p.parentUrl });
	397	+ else {
	398	+ fetched++;
	399	+ if (p.status === "success") ok++;
	400	+ else if (p.status === "blocked") blocked++;
	401	+ else if (p.status === "failed") failed++;
	402	+ }
	403	+ }
	404	+ bytes = job.bytes;
	405	+ costUsd = job.costUsd;
	406	+
	407	+ const fetchPage = async (url: string, extra: Record<string, unknown> = {}): Promise<FetchHandlerOutput \| FetchaError> => {
	408	+ try {
	409	+ return await handleFetch({
	410	+ principal,
	411	+ body: {
	412	+ url,
	413	+ format: opts.format === "html" ? "html" : opts.format,
	414	+ timeout: opts.timeout,
	415	+ country: opts.country,
	416	+ network: opts.network,
	417	+ browser: opts.browser,
	418	+ browser_fallback: opts.browser_fallback,
	419	+ headers: opts.headers,
	420	+ links: true,
	421	+ ...extra,
	422	+ },
	423	+ source: "crawl",
	424	+ clientIp: null,
	425	+ userAgent: `fetcha-crawl/${job.id}`,
	426	+ });
	427	+ } catch (e) {
	428	+ return e instanceof FetchaError ? e : new FetchaError("INTERNAL_ERROR", (e as Error).message);
	429	+ }
	430	+ };
	431	+
	432	+ // robots.txt + optional sitemap seeding
	433	+ if (opts.respect_robots) {
	434	+ const r = await fetchPage(`${seedUrl.origin}/robots.txt`, { format: "raw", links: false, browser: false, browser_fallback: false, retries: 1, timeout: Math.min(opts.timeout, 15_000) });
	435	+ if (!(r instanceof FetchaError)) {
	436	+ const b = r.body as FetchResponseBody;
	437	+ if (b.success && b.content && (b.content_type ?? "").includes("text/plain")) {
	438	+ robots = parseRobots(b.content, "fetchabot");
	439	+ if (robots.crawlDelayMs && robots.crawlDelayMs > delayMs) delayMs = Math.min(robots.crawlDelayMs, 10_000);
	440	+ }
	441	+ }
	442	+ }
	443	+ const enqueue = async (candidates: Array<{ url: string; depth: number; parent: string \| null }>) => {
	444	+ const rows: Array<{ id: string; jobId: string; url: string; depth: number; parentUrl: string \| null; status: string }> = [];
	445	+ for (const c of candidates) {
	446	+ if (discovered >= opts.max_pages * 3 \|\| frontier.length + fetched >= opts.max_pages) break;
	447	+ const n = normalizeCrawlUrl(c.url);
	448	+ if (!n \|\| seen.has(n)) continue;
	449	+ if (!inScope(n) \|\| !looksLikePage(n)) continue;
	450	+ if (exclude && exclude(n)) continue;
	451	+ if (include && !include(n) && c.depth > 0) continue;
	452	+ if (robots && !robotsAllows(robots, n)) continue;
	453	+ seen.add(n);
	454	+ discovered++;
	455	+ frontier.push({ url: n, depth: c.depth, parent: c.parent });
	456	+ rows.push({ id: newId("cpg"), jobId: job.id, url: n, depth: c.depth, parentUrl: c.parent, status: "pending" });
	457	+ }
	458	+ if (rows.length) await db.insert(crawlPages).values(rows).onConflictDoNothing().catch(() => {});
	459	+ };
	460	+ if (opts.use_sitemap && fetched === 0) {
	461	+ const sitemapUrls = new Set<string>([`${seedUrl.origin}/sitemap.xml`]);
	462	+ for (const s of robots?.sitemaps ?? []) sitemapUrls.add(s);
	463	+ let count = 0;
	464	+ for (const su of sitemapUrls) {
	465	+ if (count++ >= 5 \|\| frontier.length >= opts.max_pages) break;
	466	+ const r = await fetchPage(su, { format: "raw", links: false, browser: false, browser_fallback: false, retries: 1 });
	467	+ if (r instanceof FetchaError) continue;
	468	+ const b = r.body as FetchResponseBody;
	469	+ if (!b.success \|\| !b.content) continue;
	470	+ const { urls, sitemaps } = parseSitemap(b.content);
	471	+ for (const s of sitemaps.slice(0, 5)) sitemapUrls.add(s);
	472	+ await enqueue(urls.slice(0, opts.max_pages).map((u) => ({ url: u, depth: 1, parent: su })));
	473	+ }
	474	+ }
	475	+
	476	+ let lastHeartbeat = 0;
	477	+ let lastPersist = 0;
	478	+ const persistStats = async (force = false) => {
	479	+ const now = Date.now();
	480	+ if (!force && now - lastPersist < 2000) return;
	481	+ lastPersist = now;
	482	+ await db
	483	+ .update(crawlJobs)
	484	+ .set({ pagesDiscovered: discovered, pagesFetched: fetched, pagesOk: ok, pagesBlocked: blocked, pagesFailed: failed, bytes, costUsd, heartbeatAt: new Date() })
	485	+ .where(eq(crawlJobs.id, job.id))
	486	+ .catch(() => {});
	487	+ lastHeartbeat = now;
	488	+ };
	489	+ const isCancelled = async (): Promise<boolean> => {
	490	+ if (cancelled.has(job.id)) return true;
	491	+ if (Date.now() - lastHeartbeat > HEARTBEAT_MS) {
	492	+ const [row] = await db.select({ status: crawlJobs.status }).from(crawlJobs).where(eq(crawlJobs.id, job.id)).limit(1);
	493	+ lastHeartbeat = Date.now();
	494	+ if (row?.status === "cancelled") {
	495	+ cancelled.add(job.id);
	496	+ return true;
	497	+ }
	498	+ }
	499	+ return false;
	500	+ };
	501	+
	502	+ const worker = async () => {
	503	+ while (!stopped) {
	504	+ if (fetched >= opts.max_pages) return;
	505	+ if (await isCancelled()) return;
	506	+ const item = frontier.shift();
	507	+ if (!item) return;
	508	+ fetched++;
	509	+ const t0 = Date.now();
	510	+ const r = await fetchPage(item.url);
	511	+ const durationMs = Date.now() - t0;
	512	+ const update: Partial<typeof crawlPages.$inferInsert> = { fetchedAt: new Date(), durationMs };
	513	+ if (r instanceof FetchaError) {
	514	+ failed++;
	515	+ Object.assign(update, { status: "failed", errorCode: r.code, requestId: r.requestId ?? null });
	516	+ } else {
	517	+ const b = r.body as FetchResponseBody;
	518	+ const content = opts.format === "markdown" ? b.markdown : opts.format === "text" ? b.text : b.content;
	519	+ const isBlocked = !b.success && b.status >= 400 && r.summary && b.metadata.attempts > 0 && !(b.status === 404 \|\| b.status === 410);
	520	+ bytes += r.summary?.bytes ?? b.metadata.bytes;
	521	+ costUsd += r.summary?.costUsd ?? 0;
	522	+ if (b.success) ok++;
	523	+ else if (isBlocked && b.status !== 500) blocked++;
	524	+ else failed++;
	525	+ Object.assign(update, {
	526	+ status: b.success ? "success" : isBlocked && b.status !== 500 ? "blocked" : "failed",
	527	+ httpStatus: b.status,
	528	+ errorCode: b.success ? null : b.status === 403 \|\| b.status === 429 \|\| b.status === 503 ? "TARGET_BLOCKED" : `HTTP_${b.status}`,
	529	+ requestId: b.request_id,
	530	+ finalUrl: b.final_url,
	531	+ title: b.page?.title ?? null,
	532	+ description: b.page?.description ?? null,
	533	+ contentType: b.content_type,
	534	+ content: b.success && typeof content === "string" ? content.slice(0, PAGE_CONTENT_CAP) : null,
	535	+ linksCount: b.page?.links_count ?? 0,
	536	+ bytes: r.summary?.bytes ?? b.metadata.bytes,
	537	+ mode: b.metadata.mode,
	538	+ });
	539	+ if (b.success && item.depth < opts.max_depth && b.links?.length) {
	540	+ await enqueue(b.links.filter((l) => l.internal \|\| !opts.same_domain).map((l) => ({ url: l.url, depth: item.depth + 1, parent: item.url })));
	541	+ }
	542	+ }
	543	+ await db.update(crawlPages).set(update).where(and(eq(crawlPages.jobId, job.id), eq(crawlPages.url, item.url))).catch(() => {});
	544	+ await persistStats();
	545	+ if (delayMs > 0) await new Promise((res) => setTimeout(res, delayMs));
	546	+ }
	547	+ };
	548	+
	549	+ try {
	550	+ await Promise.all(Array.from({ length: Math.max(1, Math.min(opts.concurrency, 10)) }, () => worker()));
	551	+ await persistStats(true);
	552	+ if (cancelled.has(job.id)) {
	553	+ await finishJob(job.id, "cancelled");
	554	+ cancelled.delete(job.id);
	555	+ } else if (stopped && frontier.length) {
	556	+ await db.update(crawlJobs).set({ status: "queued", workerId: null }).where(eq(crawlJobs.id, job.id)).catch(() => {});
	557	+ } else {
	558	+ await finishJob(job.id, "completed");
	559	+ }
	560	+ log.info(`crawl ${job.id} ${cancelled.has(job.id) ? "cancelled" : "done"}: ${fetched} fetched, ${ok} ok, ${blocked} blocked, ${failed} failed`);
	561	+ if (opts.webhook_url && !stopped) await deliverWebhook(job.id, opts.webhook_url).catch(() => {});
	562	+ } catch (e) {
	563	+ await persistStats(true);
	564	+ await finishJob(job.id, "failed", { code: "INTERNAL_ERROR", message: (e as Error).message?.slice(0, 500) });
	565	+ throw e;
	566	+ }
	567	+}
	568	+
	569	+async function finishJob(id: string, status: "completed" \| "failed" \| "cancelled", error?: { code: string; message?: string }) {
	570	+ await db
	571	+ .update(crawlJobs)
	572	+ .set({ status, completedAt: new Date(), errorCode: error?.code ?? null, errorMessage: error?.message ?? null })
	573	+ .where(and(eq(crawlJobs.id, id), inArray(crawlJobs.status, ["running", "queued", "cancelled"])))
	574	+ .catch(() => {});
	575	+}
	576	+
	577	+async function deliverWebhook(jobId: string, url: string) {
	578	+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, jobId)).limit(1);
	579	+ if (!job) return;
	580	+ let status = "failed";
	581	+ try {
	582	+ await assertUrlAllowed(url);
	583	+ const ac = new AbortController();
	584	+ const t = setTimeout(() => ac.abort(), 10_000);
	585	+ const res = await fetch(url, { method: "POST", headers: { "content-type": "application/json", "user-agent": "fetcha-webhook/0.2" }, body: JSON.stringify({ type: "crawl.completed", job: serializeJob(job) }), signal: ac.signal });
	586	+ clearTimeout(t);
	587	+ status = res.ok ? `delivered:${res.status}` : `failed:${res.status}`;
	588	+ } catch (e) {
	589	+ status = `failed:${(e as Error).message?.slice(0, 80)}`;
	590	+ }
	591	+ await db.update(crawlJobs).set({ webhookStatus: status }).where(eq(crawlJobs.id, jobId)).catch(() => {});
	592	+}

modified apps/api/src/services/engine.ts +14 −2

		`@@ -1,6 +1,7 @@`
1		−import { db, providerConfigs, providerHealth, domainProfiles, eq } from "@fetcha/db";
	1	+import { db, providerConfigs, providerHealth, domainProfiles, featureFlags, eq } from "@fetcha/db";
2	2	import { ProviderRegistry, type ProviderId } from "@fetcha/providers";
3	3	import { CircuitBreaker, FetchExecutor, RoutingEngine, type DomainKnowledge } from "@fetcha/routing";
	4	+import { getBrowserPool, type BrowserPool } from "@fetcha/browser";
4	5	import { newId } from "@fetcha/core";
5	6	import { config } from "../config";
6	7
		`@@ -9,6 +10,9 @@ export interface Engine {`
9	10	circuit: CircuitBreaker;
10	11	routing: RoutingEngine;
11	12	executor: FetchExecutor;
	13	+ browser: BrowserPool;
	14	+ /** Global kill-switch from the `browser_enabled` feature flag (default on). */
	15	+ browserEnabled: boolean;
12	16	reload(): Promise<void>;
13	17	probeAll(): Promise<void>;
14	18	knowledge(domain: string): Promise<DomainKnowledge \| null>;
		`@@ -22,12 +26,18 @@ export async function getEngine(): Promise<Engine> {`
22	26	let registry = new ProviderRegistry();
23	27	let routing = new RoutingEngine(registry, circuit);
24	28	let executor = new FetchExecutor(routing, circuit);
	29	+ const browser = getBrowserPool({ log: { info: (m) => console.log("[browser]", m), warn: (m) => console.warn("[browser]", m) } });
	30	+ let browserEnabled = true;
25	31
26	32	const engine: Engine = {
27	33	get registry() {
28	34	return registry;
29	35	},
30	36	circuit,
	37	+ browser,
	38	+ get browserEnabled() {
	39	+ return browserEnabled && browser.enabled;
	40	+ },
31	41	get routing() {
32	42	return routing;
33	43	},
		`@@ -46,6 +56,8 @@ export async function getEngine(): Promise<Engine> {`
46	56	registry = new ProviderRegistry({ prices, disabled });
47	57	routing = new RoutingEngine(registry, circuit);
48	58	executor = new FetchExecutor(routing, circuit);
	59	+ const [flag] = await db.select().from(featureFlags).where(eq(featureFlags.key, "browser_enabled")).limit(1).catch(() => []);
	60	+ browserEnabled = flag ? flag.enabled : true;
49	61	},
50	62	async probeAll() {
51	63	const results = await Promise.all(
		`@@ -62,7 +74,7 @@ export async function getEngine(): Promise<Engine> {`
62	74	if (!domain) return null;
63	75	const [row] = await db.select().from(domainProfiles).where(eq(domainProfiles.domain, domain)).limit(1).catch(() => []);
64	76	if (!row) return null;
65		− return { domain, routeStats: row.routeStats ?? {}, policy: row.policy ?? null, browserRequiredRate: row.requests ? row.browserRequired / row.requests : 0 };
	77	+ return { domain, routeStats: row.routeStats ?? {}, policy: row.policy ?? null, browserRequiredRate: row.requests ? row.browserRequired / row.requests : 0, browserSamples: row.requests };
66	78	},
67	79	};
68	80	await engine.reload();

modified apps/api/src/services/persist.ts +11 −5

		`@@ -8,7 +8,7 @@ export interface RequestContext {`
8	8	organizationId: string;
9	9	projectId: string;
10	10	apiKeyId: string \| null;
11		− source: "api" \| "playground" \| "sdk";
	11	+ source: "api" \| "playground" \| "sdk" \| "crawl";
12	12	plan: Plan;
13	13	clientIp: string \| null;
14	14	userAgent: string \| null;
		`@@ -46,12 +46,13 @@ export async function recordAttempt(requestId: string, a: AttemptRecord): Promis`
46	46	attemptNo: a.attemptNo,
47	47	provider: a.provider,
48	48	network: a.network,
	49	+ mode: a.mode,
49	50	country: a.country,
50	51	sessionKey: a.sessionKey,
51	52	outcome: a.outcome,
52	53	httpStatus: a.httpStatus,
53	54	errorCode: a.errorCode,
54		− errorDetail: a.errorDetail,
	55	+ errorDetail: a.errorDetail ?? (a.blockVendor ? `vendor=${a.blockVendor} profile=${a.profileId ?? "-"}` : a.profileId ? `profile=${a.profileId}` : null),
55	56	blockReason: a.blockReason,
56	57	durationMs: a.durationMs,
57	58	bytesIn: a.bytesIn,
		`@@ -83,6 +84,8 @@ export async function completeRequest(ctx: RequestContext, req: FetchRequest, re`
83	84	errorMessage: result.body.success ? null : "The target blocked every route we tried.",
84	85	finalUrl: result.finalUrl,
85	86	network,
	87	+ mode: result.mode,
	88	+ browser: result.mode === "browser",
86	89	attempts: result.attempts.length,
87	90	latencyMs,
88	91	bytesIn: result.bytesIn,
		`@@ -96,7 +99,7 @@ export async function completeRequest(ctx: RequestContext, req: FetchRequest, re`
96	99	.where(sql`${fetchRequests.id} = ${ctx.requestId}`);
97	100
98	101	await writeUsage(ctx, result, price.totalUsd, network);
99		− await Promise.all([updateDomainProfile(result.domain, result.attempts, result.body.success), updateRoutingMetrics(result.attempts)]);
	102	+ await Promise.all([updateDomainProfile(result.domain, result.attempts, result.body.success, result.browserRequired), updateRoutingMetrics(result.attempts)]);
100	103	return { priceUsd: price.totalUsd, costUsd: result.costUsd };
101	104	}
102	105
		`@@ -118,7 +121,7 @@ export async function failRequest(ctx: RequestContext, code: string, message: st`
118	121	costUsd: 0,
119	122	upstreamCostUsd: costUsd,
120	123	});
121		− if (attempts.length) await Promise.all([updateDomainProfile(domain, attempts, false), updateRoutingMetrics(attempts)]);
	124	+ if (attempts.length) await Promise.all([updateDomainProfile(domain, attempts, false, false), updateRoutingMetrics(attempts)]);
122	125	}
123	126
124	127	async function writeUsage(ctx: RequestContext, result: ExecutionResult, priceUsd: number, network: ConcreteNetwork \| null) {
		`@@ -134,7 +137,7 @@ async function writeUsage(ctx: RequestContext, result: ExecutionResult, priceUsd`
134	137	);
135	138	}
136	139
137		−async function updateDomainProfile(domain: string, attempts: AttemptRecord[], success: boolean) {
	140	+async function updateDomainProfile(domain: string, attempts: AttemptRecord[], success: boolean, browserRequired: boolean) {
138	141	if (!domain) return;
139	142	const [existing] = await db.select().from(domainProfiles).where(sql`${domainProfiles.domain} = ${domain}`).limit(1);
140	143	const stats = { ...(existing?.routeStats ?? {}) };
		`@@ -145,6 +148,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su`
145	148	if (a.outcome === "blocked") blocks++;
146	149	if (a.blockReason === "captcha" \|\| a.blockReason === "cloudflare_challenge") captchas++;
147	150	}
	151	+ const browserInc = browserRequired ? 1 : 0;
148	152	const pref = preferredRoute(stats);
149	153	const totalLatency = attempts.reduce((s, a) => s + a.durationMs, 0);
150	154	const n = (existing?.requests ?? 0) + 1;
		`@@ -159,6 +163,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su`
159	163	successes: success ? 1 : 0,
160	164	blocks,
161	165	captchas,
	166	+ browserRequired: browserInc,
162	167	avgLatencyMs: totalLatency,
163	168	routeStats: stats,
164	169	lastSeenAt: new Date(),
		`@@ -172,6 +177,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su`
172	177	successes: sql`${domainProfiles.successes} + ${success ? 1 : 0}`,
173	178	blocks: sql`${domainProfiles.blocks} + ${blocks}`,
174	179	captchas: sql`${domainProfiles.captchas} + ${captchas}`,
	180	+ browserRequired: sql`${domainProfiles.browserRequired} + ${browserInc}`,
175	181	avgLatencyMs: avgLatency,
176	182	routeStats: stats,
177	183	lastSeenAt: new Date(),

modified apps/api/src/services/pricing.ts +5 −0

		`@@ -27,6 +27,11 @@ export interface PriceBreakdown {`
27	27
28	28	export function priceRequest(input: PriceInput): PriceBreakdown {
29	29	const limits = PLAN_LIMITS[input.plan];
	30	+ // Private platform: the single plan has no unit prices → nothing is billed. Upstream cost is
	31	+ // still tracked for the admin's unit economics.
	32	+ if (limits.price_usd_month === 0 && limits.overage_per_1k_requests_usd === 0 && limits.residential_per_gb_usd === 0) {
	33	+ return { requestUsd: 0, bandwidthUsd: 0, totalUsd: 0, marginUsd: round6(-input.upstreamCostUsd) };
	34	+ }
30	35	// Free/enterprise have no per-request rate; failed requests are never charged a request fee.
31	36	const requestUsd = input.success ? limits.overage_per_1k_requests_usd / 1000 : 0;
32	37	let bandwidthUsd = 0;

modified apps/api/src/services/retention.ts +10 −0

		`@@ -19,6 +19,16 @@ export async function pruneRetention(log: { info: (m: string) => void } = consol`
19	19	`);
20	20	total += Number((res as { rowCount?: number }).rowCount ?? 0);
21	21	}
	22	+ for (const plan of PLANS) {
	23	+ const days = PLAN_LIMITS[plan].retention_days;
	24	+ await db.execute(sql`
	25	+ delete from crawl_jobs j
	26	+ using organizations o
	27	+ where j.organization_id = o.id
	28	+ and o.plan = ${plan}
	29	+ and j.created_at < now() - make_interval(days => ${days})
	30	+ `);
	31	+ }
22	32	await db.execute(sql`update proxy_sessions set status = 'expired' where status = 'active' and expires_at < now()`);
23	33	await db.execute(sql`delete from provider_health where checked_at < now() - interval '30 days'`);
24	34	await db.execute(sql`delete from verifications where expires_at < now() - interval '1 day'`);

added apps/web/src/actions/access.ts +148 −0

		`@@ -0,0 +1,148 @@`
	1	+"use server";
	2	+
	3	+import { revalidatePath } from "next/cache";
	4	+import { z } from "zod";
	5	+import { getDb, signupAllowlist, eq, inArray } from "@fetcha/db";
	6	+import { getEmailService } from "@fetcha/email";
	7	+import { requireAdmin, requestMeta } from "@/lib/session";
	8	+import { writeAudit } from "@/lib/audit";
	9	+import { SITE_URL } from "@/lib/utils";
	10	+import type { AuthUser } from "@/lib/auth";
	11	+import type { AdminActionResult } from "@/actions/admin";
	12	+
	13	+/**
	14	+ * Signup allowlist management (invitation-only platform). Every mutation is audited as
	15	+ * `admin.action` with metadata.type in { access.allow, access.invite, access.revoke }.
	16	+ */
	17	+
	18	+async function audit(admin: AuthUser, type: "access.allow" \| "access.invite" \| "access.revoke", target: string, metadata: Record<string, unknown> = {}) {
	19	+ const meta = await requestMeta();
	20	+ await writeAudit({ userId: admin.id, action: "admin.action", target, metadata: { type, ...metadata }, ipAddress: meta.ip, userAgent: meta.userAgent });
	21	+}
	22	+
	23	+function fail(e: unknown): { ok: false; error: string } {
	24	+ if (e instanceof z.ZodError) return { ok: false, error: e.issues[0]?.message ?? "Invalid input" };
	25	+ return { ok: false, error: (e as Error)?.message ?? "Unexpected error" };
	26	+}
	27	+
	28	+function revalidate() {
	29	+ revalidatePath("/admin/access");
	30	+}
	31	+
	32	+const emailSchema = z.string().trim().toLowerCase().email("Invalid email address").max(254);
	33	+
	34	+/** Split a comma / newline / whitespace separated list into unique, lower-cased, valid emails. */
	35	+function parseEmailList(raw: string): { emails: string[]; invalid: string[] } {
	36	+ const seen = new Set<string>();
	37	+ const emails: string[] = [];
	38	+ const invalid: string[] = [];
	39	+ for (const token of raw.split(/[\s,;]+/)) {
	40	+ const t = token.trim();
	41	+ if (!t) continue;
	42	+ const r = emailSchema.safeParse(t);
	43	+ if (!r.success) {
	44	+ invalid.push(t);
	45	+ continue;
	46	+ }
	47	+ if (seen.has(r.data)) continue;
	48	+ seen.add(r.data);
	49	+ emails.push(r.data);
	50	+ }
	51	+ return { emails, invalid };
	52	+}
	53	+
	54	+function signupUrlFor(email: string): string {
	55	+ return `${SITE_URL}/signup?email=${encodeURIComponent(email)}`;
	56	+}
	57	+
	58	+const allowSchema = z.object({
	59	+ emails: z.string().trim().min(3, "Enter at least one email address").max(20_000),
	60	+ note: z.string().trim().max(280, "Note is too long (280 characters max)").optional().or(z.literal("")),
	61	+ sendInvite: z.boolean().default(false),
	62	+});
	63	+
	64	+export type AllowEmailsInput = z.input<typeof allowSchema>;
	65	+export type AllowEmailsData = { added: number; existing: number; invited: number; invalid: string[]; failed: string[] };
	66	+
	67	+export async function allowEmails(input: AllowEmailsInput): Promise<AdminActionResult<AllowEmailsData>> {
	68	+ const admin = await requireAdmin();
	69	+ try {
	70	+ const d = allowSchema.parse(input);
	71	+ const { emails, invalid } = parseEmailList(d.emails);
	72	+ if (emails.length === 0) return { ok: false, error: invalid.length ? `No valid email address found (rejected: ${invalid.slice(0, 3).join(", ")}${invalid.length > 3 ? "…" : ""}).` : "Enter at least one email address." };
	73	+ if (emails.length > 200) return { ok: false, error: "Add at most 200 addresses at a time." };
	74	+
	75	+ const db = getDb();
	76	+ const note = d.note?.trim() \|\| null;
	77	+ let invited = 0;
	78	+ const failed: string[] = [];
	79	+
	80	+ const already = new Set((await db.select({ email: signupAllowlist.email }).from(signupAllowlist).where(inArray(signupAllowlist.email, emails))).map((r) => r.email));
	81	+ const fresh = emails.filter((e) => !already.has(e));
	82	+ if (fresh.length) await db.insert(signupAllowlist).values(fresh.map((email) => ({ email, note, invitedByUserId: admin.id }))).onConflictDoNothing();
	83	+ // Existing entries keep their inviter; only refresh the note when one was provided.
	84	+ if (note && already.size) await db.update(signupAllowlist).set({ note }).where(inArray(signupAllowlist.email, Array.from(already)));
	85	+ const added = fresh.length;
	86	+ const existing = already.size;
	87	+ await audit(admin, "access.allow", emails.length === 1 ? emails[0]! : `${emails.length} emails`, { emails, note, added, existing });
	88	+
	89	+ if (d.sendInvite) {
	90	+ const inviterName = admin.name?.trim() \|\| admin.email;
	91	+ for (const email of emails) {
	92	+ try {
	93	+ await getEmailService().sendInvite(email, { inviterName, signupUrl: signupUrlFor(email) });
	94	+ await db.update(signupAllowlist).set({ invitedAt: new Date() }).where(eq(signupAllowlist.email, email));
	95	+ invited++;
	96	+ } catch (e) {
	97	+ failed.push(email);
	98	+ console.error("[access] invite failed", email, (e as Error).message);
	99	+ }
	100	+ }
	101	+ await audit(admin, "access.invite", emails.length === 1 ? emails[0]! : `${emails.length} emails`, { emails, invited, failed });
	102	+ }
	103	+
	104	+ revalidate();
	105	+ const warnings: string[] = [];
	106	+ if (invalid.length) warnings.push(`${invalid.length} entr${invalid.length === 1 ? "y was" : "ies were"} not a valid email and ${invalid.length === 1 ? "was" : "were"} skipped: ${invalid.slice(0, 5).join(", ")}${invalid.length > 5 ? "…" : ""}`);
	107	+ if (failed.length) warnings.push(`Invitation email could not be sent to ${failed.join(", ")}. Use “Resend invite” later.`);
	108	+ return { ok: true, data: { added, existing, invited, invalid, failed }, warning: warnings.length ? warnings.join(" ") : undefined };
	109	+ } catch (e) {
	110	+ return fail(e);
	111	+ }
	112	+}
	113	+
	114	+export async function resendInvite(email: string): Promise<AdminActionResult> {
	115	+ const admin = await requireAdmin();
	116	+ try {
	117	+ const target = emailSchema.parse(email);
	118	+ const db = getDb();
	119	+ const [row] = await db.select().from(signupAllowlist).where(eq(signupAllowlist.email, target)).limit(1);
	120	+ if (!row) return { ok: false, error: "This address is not on the access list." };
	121	+ if (row.userId) return { ok: false, error: "This address already has an account; there is nothing to invite." };
	122	+ const inviterName = admin.name?.trim() \|\| admin.email;
	123	+ await getEmailService().sendInvite(target, { inviterName, signupUrl: signupUrlFor(target) });
	124	+ await db.update(signupAllowlist).set({ invitedAt: new Date() }).where(eq(signupAllowlist.email, target));
	125	+ await audit(admin, "access.invite", target, { emails: [target], invited: 1, resend: true });
	126	+ revalidate();
	127	+ return { ok: true };
	128	+ } catch (e) {
	129	+ return fail(e);
	130	+ }
	131	+}
	132	+
	133	+export async function revokeAllow(email: string): Promise<AdminActionResult> {
	134	+ const admin = await requireAdmin();
	135	+ try {
	136	+ const target = emailSchema.parse(email);
	137	+ const db = getDb();
	138	+ const [row] = await db.select({ email: signupAllowlist.email, userId: signupAllowlist.userId, note: signupAllowlist.note }).from(signupAllowlist).where(eq(signupAllowlist.email, target)).limit(1);
	139	+ if (!row) return { ok: false, error: "This address is not on the access list." };
	140	+ if (row.userId) return { ok: false, error: "This address already has an account; ban the user instead." };
	141	+ await db.delete(signupAllowlist).where(eq(signupAllowlist.email, target));
	142	+ await audit(admin, "access.revoke", target, { note: row.note });
	143	+ revalidate();
	144	+ return { ok: true };
	145	+ } catch (e) {
	146	+ return fail(e);
	147	+ }
	148	+}

modified apps/web/src/actions/admin.ts +8 −13

		`@@ -118,20 +118,15 @@ export async function forceVerifyEmail(userId: string): Promise<AdminActionResul`
118	118	// ---------------------------------------------------------------------------
119	119	// Organizations
120	120	// ---------------------------------------------------------------------------
	121	+/**
	122	+ * Single-plan platform: kept for API compatibility with older UI code, but plans can no longer be
	123	+ * changed. Every organization is `unlimited` (see `normalizePlan` in @fetcha/core).
	124	+ */
121	125	export async function updateOrganizationPlan(orgId: string, plan: string): Promise<AdminActionResult> {
122		− const admin = await requireAdmin();
123		− try {
124		− const p = z.enum(PLANS).parse(plan);
125		− const db = getDb();
126		− const [o] = await db.select({ id: organizations.id, plan: organizations.plan }).from(organizations).where(eq(organizations.id, orgId)).limit(1);
127		− if (!o) return { ok: false, error: "Organization not found" };
128		− await db.update(organizations).set({ plan: p, updatedAt: new Date() }).where(eq(organizations.id, orgId));
129		− await audit(admin, "org.plan", orgId, { from: o.plan, to: p }, orgId);
130		− revalidateAdmin();
131		− return { ok: true };
132		− } catch (e) {
133		− return fail(e);
134		− }
	126	+ await requireAdmin();
	127	+ void orgId;
	128	+ void plan;
	129	+ return { ok: false, error: "Plans are not configurable on this platform" };
135	130	}
136	131
137	132	export async function setOrganizationProviderVisibility(orgId: string, enabled: boolean): Promise<AdminActionResult> {

added apps/web/src/actions/crawls.ts +87 −0

		`@@ -0,0 +1,87 @@`
	1	+"use server";
	2	+
	3	+import { revalidatePath } from "next/cache";
	4	+import { crawlCreateSchema, mapCreateSchema } from "@fetcha/core";
	5	+import { getWorkspace } from "@/lib/session";
	6	+import { internalApi, InternalApiError, type CrawlJob, type MapResult } from "@/lib/api";
	7	+
	8	+export interface CrawlActionError {
	9	+ code: string;
	10	+ message: string;
	11	+ requestId: string \| null;
	12	+ details?: Record<string, unknown>;
	13	+}
	14	+
	15	+export type CrawlActionResult<T> = { ok: true; data: T } \| { ok: false; error: CrawlActionError };
	16	+
	17	+function invalid(issues: Array<{ path: string; message: string }>): CrawlActionError {
	18	+ const first = issues[0];
	19	+ return {
	20	+ code: "INVALID_REQUEST",
	21	+ message: first ? `${first.path ? `${first.path}: ` : ""}${first.message}` : "The request is invalid.",
	22	+ requestId: null,
	23	+ details: { issues },
	24	+ };
	25	+}
	26	+
	27	+function fromError(e: unknown, fallback: string): CrawlActionError {
	28	+ if (e instanceof InternalApiError) return { code: e.code, message: e.message, requestId: e.requestId, details: e.details };
	29	+ console.error("[crawls] unexpected error", e);
	30	+ return { code: "INTERNAL_ERROR", message: fallback, requestId: null };
	31	+}
	32	+
	33	+/** Strip empty strings / undefined so optional fields do not trip the strict schema. */
	34	+function clean(input: unknown): Record<string, unknown> {
	35	+ if (!input \|\| typeof input !== "object") return {};
	36	+ return Object.fromEntries(Object.entries(input as Record<string, unknown>).filter(([, v]) => v !== undefined && v !== "" && v !== null));
	37	+}
	38	+
	39	+/**
	40	+ * Start a crawl job for the current project. Validated with the shared `crawlCreateSchema`; the API
	41	+ * enforces plan limits (pages per job, concurrent jobs). Never throws to the client.
	42	+ */
	43	+export async function createCrawl(input: unknown): Promise<CrawlActionResult<CrawlJob>> {
	44	+ const ws = await getWorkspace();
	45	+ try {
	46	+ const parsed = crawlCreateSchema.safeParse(clean(input));
	47	+ if (!parsed.success) return { ok: false, error: invalid(parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message }))) };
	48	+ const job = await internalApi.createCrawl(ws.project.id, ws.user.id, parsed.data);
	49	+ if (!job \|\| typeof job !== "object" \|\| typeof job.id !== "string") {
	50	+ return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };
	51	+ }
	52	+ revalidatePath("/dashboard/crawls");
	53	+ return { ok: true, data: job };
	54	+ } catch (e) {
	55	+ return { ok: false, error: fromError(e, "Could not start the crawl. Please try again.") };
	56	+ }
	57	+}
	58	+
	59	+/** Cancel a queued or running crawl job. */
	60	+export async function cancelCrawl(id: string): Promise<CrawlActionResult<{ id: string; status: "cancelled" }>> {
	61	+ const ws = await getWorkspace();
	62	+ if (!/^crawl_[A-Za-z0-9]{4,64}$/.test(id)) return { ok: false, error: { code: "INVALID_REQUEST", message: "Invalid crawl id.", requestId: null } };
	63	+ try {
	64	+ const res = await internalApi.cancelCrawl(ws.project.id, ws.user.id, id);
	65	+ revalidatePath("/dashboard/crawls");
	66	+ revalidatePath(`/dashboard/crawls/${id}`);
	67	+ return { ok: true, data: res };
	68	+ } catch (e) {
	69	+ return { ok: false, error: fromError(e, "Could not cancel the crawl. Please try again.") };
	70	+ }
	71	+}
	72	+
	73	+/** Discover the URLs of a site (sitemap + links), synchronously. */
	74	+export async function runMap(input: unknown): Promise<CrawlActionResult<MapResult>> {
	75	+ const ws = await getWorkspace();
	76	+ try {
	77	+ const parsed = mapCreateSchema.safeParse(clean(input));
	78	+ if (!parsed.success) return { ok: false, error: invalid(parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message }))) };
	79	+ const result = await internalApi.mapSite(ws.project.id, ws.user.id, parsed.data);
	80	+ if (!result \|\| typeof result !== "object" \|\| !Array.isArray(result.urls)) {
	81	+ return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };
	82	+ }
	83	+ return { ok: true, data: result };
	84	+ } catch (e) {
	85	+ return { ok: false, error: fromError(e, "Could not map the site. Please try again.") };
	86	+ }
	87	+}

modified apps/web/src/actions/playground.ts +2 −7

		`@@ -1,6 +1,6 @@`
1	1	"use server";
2	2
3		−import { PLAN_LIMITS, fetchRequestSchema, type FetchResponseBody, type Plan } from "@fetcha/core";
	3	+import { PLAN_LIMITS, fetchRequestSchema, normalizePlan, type FetchResponseBody } from "@fetcha/core";
4	4	import { getWorkspace } from "@/lib/session";
5	5	import { internalApi, InternalApiError } from "@/lib/api";
6	6
		`@@ -39,8 +39,7 @@ export async function runPlayground(input: unknown): Promise<PlaygroundResult> {`
39	39	}
40	40
41	41	const request = parsed.data;
42		− const plan = (ws.organization.plan as Plan) in PLAN_LIMITS ? (ws.organization.plan as Plan) : "free";
43		− const limits = PLAN_LIMITS[plan];
	42	+ const limits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
44	43	if (request.timeout > limits.max_timeout_ms) request.timeout = limits.max_timeout_ms;
45	44	if (request.retries !== undefined && request.retries > limits.max_retries) request.retries = limits.max_retries;
46	45	if (request.network !== "auto" && !limits.networks.includes(request.network)) {
		`@@ -49,10 +48,6 @@ export async function runPlayground(input: unknown): Promise<PlaygroundResult> {`
49	48	error: { code: "NETWORK_UNAVAILABLE", message: `The "${request.network}" network is not included in the ${limits.label} plan.`, requestId: null },
50	49	};
51	50	}
52		− if (request.browser) {
53		− return { ok: false, error: { code: "BROWSER_UNAVAILABLE", message: "Managed browser execution is not yet available.", requestId: null } };
54		− }
55		−
56	51	const data = (await internalApi.playgroundFetch(ws.project.id, ws.user.id, request)) as FetchResponseBody;
57	52	if (!data \|\| typeof data !== "object" \|\| typeof data.request_id !== "string") {
58	53	return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };

modified apps/web/src/app/(auth)/login/login-form.tsx +3 −3

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(auth)/signup/page.tsx +4 −2

Large diff (13 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(auth)/signup/signup-form.tsx +33 −8

Large diff (75 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/changelog/page.tsx +23 −2

Large diff (42 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/authentication/page.tsx +11 −11

Large diff (60 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/browser/page.tsx +189 −45

Large diff (280 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/(marketing)/docs/crawl/page.tsx +379 −0

Large diff (379 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/errors/page.tsx +6 −4

Large diff (29 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/fetch/page.tsx +85 −21

Large diff (206 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/networks/page.tsx +30 −33

Large diff (123 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/openapi.json/route.ts +277 −20

Large diff (398 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/page.tsx +15 −11

Large diff (58 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/rate-limits/page.tsx +83 −74

Large diff (274 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/docs/retries/page.tsx +33 −26

Large diff (102 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/page.tsx +30 −25

Large diff (134 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/(marketing)/pricing/page.tsx +127 −249

Large diff (431 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/admin/access/loading.tsx +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/admin/access/page.tsx +159 −0

Large diff (159 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/admin/billing/page.tsx +4 −4

Large diff (29 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/admin/organizations/[id]/page.tsx +6 −3

Large diff (27 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/dashboard/billing/page.tsx +86 −143

Large diff (282 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/dashboard/crawls/[id]/loading.tsx +28 −0

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/dashboard/crawls/[id]/page.tsx +344 −0

Large diff (344 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/dashboard/crawls/loading.tsx +31 −0

Large diff (31 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/dashboard/crawls/page.tsx +179 −0

Large diff (179 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/dashboard/page.tsx +5 −6

Large diff (32 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/dashboard/playground/page.tsx +4 −5

Large diff (33 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/dashboard/projects/[id]/page.tsx +3 −3

Large diff (24 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/dashboard/usage/page.tsx +18 −17

Large diff (94 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/admin/access-controls.tsx +149 −0

Large diff (149 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/admin/admin-shell.tsx +2 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/admin/org-actions.tsx +1 −36

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/admin/primitives.tsx +3 −2

Large diff (13 lines) is not loaded by default to keep the page fast.Load diff

deleted apps/web/src/components/dashboard/billing/upgrade-dialog.tsx +0 −39

Large diff (39 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/dashboard/crawls/cancel-crawl-button.tsx +33 −0

Large diff (33 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/dashboard/crawls/crawl-status-badge.tsx +39 −0

Large diff (39 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/dashboard/crawls/create-crawl-dialog.tsx +201 −0

Large diff (201 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/dashboard/crawls/map-tool.tsx +99 −0

Large diff (99 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/dashboard/sidebar.tsx +2 −1

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/docs/nav-config.ts +2 −1

Large diff (9 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/marketing/architecture-diagram.tsx +2 −2

Large diff (16 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/marketing/footer.tsx +1 −1

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/marketing/nav.tsx +5 −5

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/playground/request-builder.tsx +102 −11

Large diff (164 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/playground/results-panel.tsx +124 −3

Large diff (208 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/playground/types.ts +54 −1

Large diff (96 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/access.ts +20 −0

Large diff (20 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/api.ts +88 −0

Large diff (98 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/auth.ts +22 −1

Large diff (43 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/codegen.test.ts +56 −0

Large diff (68 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/codegen.ts +22 −0

Large diff (37 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/queries/account.ts +13 −3

Large diff (33 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/queries/admin.ts +73 −4

Large diff (116 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/session.ts +2 −6

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

modified deploy/README.md +2 −1

Large diff (7 lines) is not loaded by default to keep the page fast.Load diff

added docs/API-v0.2.md +124 −0

Large diff (124 lines) is not loaded by default to keep the page fast.Load diff

added packages/browser/package.json +26 −0

Large diff (26 lines) is not loaded by default to keep the page fast.Load diff

added packages/browser/src/index.ts +550 −0

Large diff (550 lines) is not loaded by default to keep the page fast.Load diff

added packages/browser/src/stealth.ts +126 −0

Large diff (126 lines) is not loaded by default to keep the page fast.Load diff

added packages/browser/tsconfig.json +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

modified packages/core/src/client.ts +1 −0

Large diff (4 lines) is not loaded by default to keep the page fast.Load diff

modified packages/core/src/errors.ts +10 −4

Large diff (44 lines) is not loaded by default to keep the page fast.Load diff

modified packages/core/src/ids.ts +3 −1

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

modified packages/core/src/index.ts +2 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/markdown.ts +687 −0

Large diff (687 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/robots.ts +101 −0

Large diff (101 lines) is not loaded by default to keep the page fast.Load diff

modified packages/core/src/schema.ts +150 −64

Large diff (273 lines) is not loaded by default to keep the page fast.Load diff

modified packages/core/src/text.ts +180 −13

Large diff (228 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/test/markdown.test.ts +80 −0

Large diff (80 lines) is not loaded by default to keep the page fast.Load diff

modified packages/core/test/ssrf.test.ts +7 −1

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/drizzle/0001_furry_red_skull.sql +76 −0

Large diff (76 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/drizzle/meta/0001_snapshot.json +3572 −0

Large diff (3,573 lines) is not loaded by default to keep the page fast.Load diff

modified packages/db/drizzle/meta/_journal.json +7 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

modified packages/db/src/schema.ts +96 −1

Large diff (130 lines) is not loaded by default to keep the page fast.Load diff

modified packages/db/src/seed.ts +47 −3

Large diff (67 lines) is not loaded by default to keep the page fast.Load diff

modified packages/email/src/index.ts +6 −0

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff

modified packages/email/src/templates.tsx +26 −1

Large diff (38 lines) is not loaded by default to keep the page fast.Load diff

added packages/providers/src/cookies.ts +129 −0

Large diff (129 lines) is not loaded by default to keep the page fast.Load diff

modified packages/providers/src/decodo.ts +6 −2

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff

modified packages/providers/src/direct.ts +4 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

added packages/providers/src/fingerprint.ts +369 −0

Large diff (369 lines) is not loaded by default to keep the page fast.Load diff

modified packages/providers/src/http.ts +171 −82

Large diff (367 lines) is not loaded by default to keep the page fast.Load diff

modified packages/providers/src/index.ts +2 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

modified packages/providers/src/oxylabs.ts +6 −2

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff

modified packages/providers/src/soax.ts +6 −2

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff

modified packages/providers/src/types.ts +26 −1

Large diff (56 lines) is not loaded by default to keep the page fast.Load diff

modified packages/routing/package.json +1 −0

Large diff (7 lines) is not loaded by default to keep the page fast.Load diff

modified packages/routing/src/engine.ts +3 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

modified packages/routing/src/executor.ts +230 −61

Large diff (514 lines) is not loaded by default to keep the page fast.Load diff

modified packages/routing/test/engine.test.ts +7 −7

Large diff (51 lines) is not loaded by default to keep the page fast.Load diff

modified packages/sdk/package.json +1 −1

Large diff (7 lines) is not loaded by default to keep the page fast.Load diff

modified packages/sdk/src/index.ts +274 −9

Large diff (379 lines) is not loaded by default to keep the page fast.Load diff

modified pnpm-lock.yaml +54 −0

Large diff (96 lines) is not loaded by default to keep the page fast.Load diff

modified scripts/dev-seed.ts +7 −4

Large diff (22 lines) is not loaded by default to keep the page fast.Load diff

modified sdk-python/README.md +28 −4

Large diff (51 lines) is not loaded by default to keep the page fast.Load diff

modified sdk-python/fetcha/__init__.py +266 −27

Large diff (390 lines) is not loaded by default to keep the page fast.Load diff

added sdk-python/fetcha/__pycache__/__init__.cpython-314.pyc +0 −0

Binary file not shown.

modified sdk-python/pyproject.toml +2 −2

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

spb/fetcha

Theme