SPB Git
spboucher.ai

spb/fetcha

Public

Fetcha v0.2.0 — private access (single unlimited plan, invitation-only signup, /admin/access), managed browser (Patchright, auto-escalation), hardened HTTP fingerprints (ordered headers, HTTP/2, TLS profiles, cookie jar, backoff), expanded block detection, markdown/page/links, crawl & map API, dashboard Crawls, SDK 0.2.0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 8, 2026) parent 67d1ab1 Browse files

117 changed files +11,929 −1,177

117 changed files
Load all 117 files Unified Split
modified .env.example +11 −0
@@ -32,3 +32,14 @@ SOAX_PASSWORD=
32 32
33 # ---- Admin ---- 33 # ---- Admin ----
34 ADMIN_EMAILS=you@example.com 34 ADMIN_EMAILS=you@example.com
35+
36+# --- v0.2: managed browser, HTTP engine, crawler ---------------------------
37+# Managed Chromium (Playwright). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
38+FETCHA_BROWSER_ENABLED=1
39+FETCHA_BROWSER_CONCURRENCY=6
40+# "chromium" = full Chromium build in new headless mode (best anti-bot realism); "chrome" uses an installed Google Chrome.
41+FETCHA_BROWSER_CHANNEL=chromium
42+# Prefer HTTP/2 towards origins (set 0 to force HTTP/1.1).
43+FETCHA_HTTP2=1
44+# Crawl jobs executed in parallel by this API process.
45+FETCHA_CRAWL_PARALLEL_JOBS=4
modified CLAUDE.md +15 −5
@@ -2,15 +2,20 @@
2 2
3 Fetcha (www.fetcha.co) is "Intelligent Web Access Infrastructure": one API (`POST /v1/fetch`) that routes web requests 3 Fetcha (www.fetcha.co) is "Intelligent Web Access Infrastructure": one API (`POST /v1/fetch`) that routes web requests
4 across upstream proxy networks (Oxylabs, Decodo, SOAX — never exposed to customers) with smart routing, retries, 4 across upstream proxy networks (Oxylabs, Decodo, SOAX — never exposed to customers) with smart routing, retries,
5−sticky sessions, usage metering and a full dashboard. 5+sticky sessions, a managed headless browser (`browser: true` + automatic escalation), crawl/map jobs, usage metering
6+and a full dashboard. **Private platform (v0.2, 2026-09-08)**: a single `unlimited` plan, invitation-only signup
7+(`signup_allowlist`, managed from `/admin/access`), `ADMIN_EMAILS` become admins. No billing.
6 8
7 ## Layout 9 ## Layout
8 - `apps/web` — Next.js 16 (marketing site, docs, auth, customer dashboard, admin). Port 8220. Proxies `/v1/*` to the API. 10 - `apps/web` — Next.js 16 (marketing site, docs, auth, customer dashboard, admin). Port 8220. Proxies `/v1/*` to the API.
9 - `apps/api` — Fastify API service (public `/v1/*`, `/health`, `/ready`, internal `/internal/*` for the dashboard). Port 8221. 11 - `apps/api` — Fastify API service (public `/v1/*`, `/health`, `/ready`, internal `/internal/*` for the dashboard). Port 8221.
10 - `packages/core` — zod schemas, error codes, ids/api-key hashing, SSRF policy, geo model, plan limits. 12 - `packages/core` — zod schemas, error codes, ids/api-key hashing, SSRF policy, geo model, plan limits.
11 - `packages/db` — Drizzle schema + migrations (`drizzle/`), `pnpm db:generate|migrate|seed`. 13 - `packages/db` — Drizzle schema + migrations (`drizzle/`), `pnpm db:generate|migrate|seed`.
12−- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct). Provider code lives ONLY here. 14+- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct), `http.ts` (undici, HTTP/2, browser-ordered headers, Chrome/Firefox/Safari TLS cipher lists, h2→h1 fallback), `fingerprint.ts` (header profiles), `cookies.ts` (jar across hops / sessions). Provider code lives ONLY here.
13−- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, block detection, escalation). 15+- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, fingerprint rotation, jittered backoff, block detection, HTTP → browser escalation).
16+- `packages/browser` — `BrowserPool` (Patchright = patched Playwright, full Chromium new-headless, per-context upstream proxy, stealth init script, challenge wait + Turnstile click). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
17+- `packages/core/src/markdown.ts` — HTML → Markdown / main text, page metadata + links, URL normalisation, glob/regex matchers; `robots.ts` — robots.txt + sitemap parsing.
18+- `apps/api/src/services/crawl.ts` — durable crawl jobs (`crawl_jobs`/`crawl_pages`, in-process worker, resume after restart) and the sync `/v1/map`.
14 - `packages/email` — Resend abstraction + React Email templates. `packages/sdk` (JS), `sdk-python/` (Python). 19 - `packages/email` — Resend abstraction + React Email templates. `packages/sdk` (JS), `sdk-python/` (Python).
15 - `docs/AGENT-BRIEF.md` — product/engineering rules used when generating UI. 20 - `docs/AGENT-BRIEF.md` — product/engineering rules used when generating UI.
16 21
@@ -19,7 +24,9 @@ sticky sessions, usage metering and a full dashboard.
19 - Every external URL goes through `assertUrlAllowed()` (SSRF) — including redirects. 24 - Every external URL goes through `assertUrlAllowed()` (SSRF) — including redirects.
20 - Business logic in server actions / `apps/api` / packages, not in React components. 25 - Business logic in server actions / `apps/api` / packages, not in React components.
21 - API keys: shown once, SHA-256 stored. Request IDs `req_…` on every response (`X-Fetcha-Request-ID`). 26 - API keys: shown once, SHA-256 stored. Request IDs `req_…` on every response (`X-Fetcha-Request-ID`).
22−- Not implemented yet (say so, don't fake): browser mode, `/v1/extract`, Stripe checkout, teams, webhook delivery, OAuth, 2FA. 27+- Not implemented yet (say so, don't fake): scripted browser actions (`POST /v1/browser`), `/v1/extract`, teams, webhook delivery (except the crawl `webhook_url` callback), OAuth, 2FA. Billing/Stripe is intentionally absent (private platform).
28+- Plans: always `normalizePlan(org.plan)`; never reintroduce tiers. Signup must stay allowlist-gated (Better Auth `user.create.before` hook in `apps/web/src/lib/auth.ts`).
29+- Anti-bot honesty: interactive Cloudflare Turnstile challenges are NOT reliably solved (tested 2026-09-08 with Patchright + residential + headful); non-interactive JS challenges, DataDome/PX/Akamai soft blocks usually pass via residential + browser escalation.
23 30
24 ## Dev 31 ## Dev
25 `cp .env.example .env` (fill provider creds + RESEND_API_KEY), `createdb fetcha`, `pnpm db:migrate && pnpm db:seed`, 32 `cp .env.example .env` (fill provider creds + RESEND_API_KEY), `createdb fetcha`, `pnpm db:migrate && pnpm db:seed`,
@@ -27,4 +34,7 @@ sticky sessions, usage metering and a full dashboard.
27 34
28 ## Deploy (MacLustr) 35 ## Deploy (MacLustr)
29 Deployed by `mld` (gateway M1M32) on M3U96a: web 8220 + api 8221, Postgres 17 `fetcha` + Redis local, ngrok `www.fetcha.co`. 36 Deployed by `mld` (gateway M1M32) on M3U96a: web 8220 + api 8221, Postgres 17 `fetcha` + Redis local, ngrok `www.fetcha.co`.
30−Manifest: `deploy/fetcha.mld.json` → `M1M32:~/dispatch/apps/fetcha.json`. See `deploy/README.md`. 37+Manifest: `deploy/fetcha.mld.json` → `M1M32:~/dispatch/apps/fetcha.json`. See `deploy/README.md`. Post-sync hooks: `pnpm install`,
38+`db:migrate` + `db:seed` (with `ADMIN_EMAILS`), `patchright install chromium`, `next build`. Env knobs: `FETCHA_BROWSER_ENABLED`,
39+`FETCHA_BROWSER_CONCURRENCY`, `FETCHA_BROWSER_CHANNEL`, `FETCHA_BROWSER_HEADLESS`, `FETCHA_HTTP2`, `FETCHA_CRAWL_PARALLEL_JOBS`.
40+The `browser_enabled` feature flag (admin → Flags) is the global kill switch for browser mode.
modified README.md +4 −2
@@ -35,7 +35,9 @@ pnpm dev:web # http://localhost:8220
35 ``` 35 ```
36 Tests: `pnpm test` · Typecheck: `pnpm typecheck` · Build: `pnpm build`. 36 Tests: `pnpm test` · Typecheck: `pnpm typecheck` · Build: `pnpm build`.
37 37
38−## Status (v0.1.0 — public preview) 38+## Status (v0.2.0 — private platform)
39 Live: Fetch API, auto routing (residential network class), sticky sessions, Playground, request logs, usage metering, 39 Live: Fetch API, auto routing (residential network class), sticky sessions, Playground, request logs, usage metering,
40−API keys, projects, admin console, docs, SDK source. Coming: managed browser, structured extraction, Stripe checkout, 40+API keys, projects, admin console, docs, SDK source, managed browser rendering (`browser: true`, automatic escalation),
41+`format: "markdown"` + page metadata/links, crawl & map jobs (`/v1/crawl`, `/v1/map`). Private platform: invitation-only
42+signup, one unlimited plan, no billing. Coming: scripted browser actions, structured extraction,
41 teams, webhooks. See `/changelog`. 43 teams, webhooks. See `/changelog`.
modified apps/api/package.json +1 −0
@@ -11,6 +11,7 @@
11 }, 11 },
12 "dependencies": { 12 "dependencies": {
13 "@fastify/cors": "^11.0.0", 13 "@fastify/cors": "^11.0.0",
14+ "@fetcha/browser": "workspace:*",
14 "@fetcha/core": "workspace:*", 15 "@fetcha/core": "workspace:*",
15 "@fetcha/db": "workspace:*", 16 "@fetcha/db": "workspace:*",
16 "@fetcha/email": "workspace:*", 17 "@fetcha/email": "workspace:*",
modified apps/api/src/auth.ts +63 −3
@@ -1,4 +1,4 @@
1−import { FetchaError, hashApiKey, parseApiKeyMode, safeEqual, type ApiKeyScope, type Plan } from "@fetcha/core"; 1+import { FetchaError, hashApiKey, normalizePlan, parseApiKeyMode, safeEqual, type ApiKeyScope, type Plan } from "@fetcha/core";
2 import { apiKeys, db, eq, organizations, projects, sql, users } from "@fetcha/db"; 2 import { apiKeys, db, eq, organizations, projects, sql, users } from "@fetcha/db";
3 import { config } from "./config"; 3 import { config } from "./config";
4 import { getKV } from "./redis"; 4 import { getKV } from "./redis";
@@ -74,7 +74,7 @@ async function loadPrincipalByKeyHash(hash: string): Promise<ApiPrincipal | null
74 scopes: (row.scopes ?? []) as ApiKeyScope[], 74 scopes: (row.scopes ?? []) as ApiKeyScope[],
75 organizationId: row.organizationId, 75 organizationId: row.organizationId,
76 organizationName: row.organizationName, 76 organizationName: row.organizationName,
77− plan: row.plan as Plan, 77+ plan: normalizePlan(row.plan),
78 providerVisibility: row.providerVisibility, 78 providerVisibility: row.providerVisibility,
79 suspended: row.suspended, 79 suspended: row.suspended,
80 orgSoftLimitUsd: row.orgSoft, 80 orgSoftLimitUsd: row.orgSoft,
@@ -172,7 +172,67 @@ export async function principalForProject(projectId: string, userId: string): Pr
172 scopes: ["fetch:execute", "sessions:write", "usage:read"], 172 scopes: ["fetch:execute", "sessions:write", "usage:read"],
173 organizationId: row.organizationId, 173 organizationId: row.organizationId,
174 organizationName: row.organizationName, 174 organizationName: row.organizationName,
175− plan: row.plan as Plan, 175+ plan: normalizePlan(row.plan),
176+ providerVisibility: row.providerVisibility,
177+ suspended: row.suspended,
178+ orgSoftLimitUsd: row.orgSoft,
179+ orgHardLimitUsd: row.orgHard,
180+ projectId: row.projectId,
181+ projectName: row.projectName,
182+ projectLogLevel: row.logLevel as ApiPrincipal["projectLogLevel"],
183+ projectSoftLimitUsd: row.projSoft,
184+ projectHardLimitUsd: row.projHard,
185+ projectMonthlyRequestLimit: row.projMonthly,
186+ ownerEmail: row.ownerEmail,
187+ ownerVerified: row.ownerVerified,
188+ internal: true,
189+ };
190+}
191+
192+/** Principal for a background crawl job: the project's owner context, with the originating key when still valid. */
193+export async function principalForCrawl(projectId: string, apiKeyId: string | null): Promise<ApiPrincipal> {
194+ const [row] = await db
195+ .select({
196+ organizationId: organizations.id,
197+ organizationName: organizations.name,
198+ plan: organizations.plan,
199+ providerVisibility: organizations.providerVisibility,
200+ suspended: organizations.suspended,
201+ orgSoft: organizations.softLimitUsd,
202+ orgHard: organizations.hardLimitUsd,
203+ projectId: projects.id,
204+ projectName: projects.name,
205+ logLevel: projects.logLevel,
206+ projSoft: projects.softLimitUsd,
207+ projHard: projects.hardLimitUsd,
208+ projMonthly: projects.monthlyRequestLimit,
209+ archivedAt: projects.archivedAt,
210+ ownerEmail: users.email,
211+ ownerVerified: users.emailVerified,
212+ ownerBanned: users.banned,
213+ })
214+ .from(projects)
215+ .innerJoin(organizations, eq(projects.organizationId, organizations.id))
216+ .innerJoin(users, eq(organizations.ownerUserId, users.id))
217+ .where(eq(projects.id, projectId))
218+ .limit(1);
219+ if (!row) throw new FetchaError("NOT_FOUND", "Project not found.");
220+ if (row.archivedAt) throw new FetchaError("FORBIDDEN", "This project is archived.");
221+ if (row.suspended || row.ownerBanned) throw new FetchaError("FORBIDDEN", "This organization is suspended.");
222+ let keyName: string | null = null;
223+ if (apiKeyId) {
224+ const [k] = await db.select({ name: apiKeys.name, revokedAt: apiKeys.revokedAt }).from(apiKeys).where(eq(apiKeys.id, apiKeyId)).limit(1);
225+ if (k?.revokedAt) throw new FetchaError("INVALID_API_KEY", "The API key that started this crawl was revoked.");
226+ keyName = k?.name ?? null;
227+ }
228+ return {
229+ keyId: apiKeyId,
230+ keyName,
231+ mode: "live",
232+ scopes: ["fetch:execute", "crawl:execute", "sessions:write", "usage:read"],
233+ organizationId: row.organizationId,
234+ organizationName: row.organizationName,
235+ plan: normalizePlan(row.plan),
176 providerVisibility: row.providerVisibility, 236 providerVisibility: row.providerVisibility,
177 suspended: row.suspended, 237 suspended: row.suspended,
178 orgSoftLimitUsd: row.orgSoft, 238 orgSoftLimitUsd: row.orgSoft,
modified apps/api/src/limits.ts +18 −0
@@ -23,6 +23,24 @@ export async function acquireConcurrency(p: ApiPrincipal): Promise<() => Promise
23 }; 23 };
24 } 24 }
25 25
26+/** Reserve a managed-browser slot for the organization (separate, smaller pool than plain fetches). */
27+export async function acquireBrowserSlot(p: ApiPrincipal): Promise<() => Promise<void>> {
28+ const kv = getKV();
29+ const limit = PLAN_LIMITS[p.plan].browser_concurrency;
30+ const key = `fch:bconc:${p.organizationId}`;
31+ const n = await kv.incrWithTtl(key, CONCURRENCY_TTL_SEC);
32+ if (n > limit) {
33+ await kv.decr(key);
34+ throw new FetchaError("CONCURRENCY_LIMIT", `At most ${limit} browser renders can run at once for your organization.`, { details: { limit, scope: "browser" } });
35+ }
36+ let released = false;
37+ return async () => {
38+ if (released) return;
39+ released = true;
40+ await kv.decr(key).catch(() => {});
41+ };
42+}
43+
26 /** Sliding-window rate limits: per key (burst), per organization (sustained), per IP (anonymous abuse). */ 44 /** Sliding-window rate limits: per key (burst), per organization (sustained), per IP (anonymous abuse). */
27 export async function checkRateLimits(p: ApiPrincipal, ip: string | null): Promise<void> { 45 export async function checkRateLimits(p: ApiPrincipal, ip: string | null): Promise<void> {
28 const kv = getKV(); 46 const kv = getKV();
added apps/api/src/routes/crawl.ts +96 −0
@@ -0,0 +1,96 @@
1+import type { FastifyInstance } from "fastify";
2+import { z } from "zod";
3+import { FetchaError } from "@fetcha/core";
4+import { authenticateApiKey, principalForProject, requireScope } from "../auth";
5+import { cancelCrawl, createCrawl, getCrawl, listCrawlPages, listCrawls, mapSite } from "../services/crawl";
6+import { getEngine } from "../services/engine";
7+
8+const pagesQuery = z.object({
9+ cursor: z.string().optional(),
10+ limit: z.coerce.number().int().min(1).max(500).optional(),
11+ status: z.enum(["pending", "success", "blocked", "failed", "skipped"]).optional(),
12+ include_content: z
13+ .union([z.literal("true"), z.literal("false"), z.literal("1"), z.literal("0")])
14+ .optional()
15+ .transform((v) => (v === undefined ? undefined : v === "true" || v === "1")),
16+});
17+
18+const clientIp = (req: { headers: Record<string, unknown>; ip: string }) => ((req.headers["x-forwarded-for"] as string | undefined)?.split(",")[0]?.trim() ?? req.ip) || null;
19+const sourceOf = (req: { headers: Record<string, unknown> }) => (/fetcha-sdk/i.test(String(req.headers["user-agent"] ?? "")) ? "sdk" : "api") as "sdk" | "api";
20+
21+/** Public crawl & map endpoints. */
22+export async function registerCrawlRoutes(app: FastifyInstance) {
23+ app.post("/v1/crawl", async (req, reply) => {
24+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string | undefined);
25+ requireScope(principal, "fetch:execute");
26+ const job = await createCrawl(principal, req.body, sourceOf(req));
27+ return reply.status(202).send(job);
28+ });
29+ app.get<{ Querystring: { limit?: string } }>("/v1/crawl", async (req) => {
30+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string | undefined);
31+ return listCrawls(principal, Number(req.query.limit ?? 50) || 50);
32+ });
33+ app.get<{ Params: { id: string } }>("/v1/crawl/:id", async (req) => {
34+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string | undefined);
35+ return getCrawl(principal, req.params.id);
36+ });
37+ app.get<{ Params: { id: string }; Querystring: Record<string, string | undefined> }>("/v1/crawl/:id/pages", async (req) => {
38+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string | undefined);
39+ const q = pagesQuery.safeParse(req.query);
40+ if (!q.success) throw new FetchaError("INVALID_REQUEST", "Invalid query parameters.", { details: { issues: q.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
41+ return listCrawlPages(principal, req.params.id, q.data);
42+ });
43+ app.delete<{ Params: { id: string } }>("/v1/crawl/:id", async (req) => {
44+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string | undefined);
45+ requireScope(principal, "fetch:execute");
46+ return cancelCrawl(principal, req.params.id);
47+ });
48+ app.post("/v1/map", async (req) => {
49+ const principal = await authenticateApiKey(req.headers.authorization, req.headers["x-api-key"] as string | undefined);
50+ requireScope(principal, "fetch:execute");
51+ return mapSite(principal, req.body, sourceOf(req), clientIp(req));
52+ });
53+}
54+
55+/** Internal (dashboard) crawl routes; the service-token check is applied by the internal onRequest hook. */
56+export async function registerInternalCrawlRoutes(app: FastifyInstance) {
57+ const bodySchema = z.object({ project_id: z.string(), user_id: z.string(), options: z.unknown().optional() });
58+ type Q = { project_id: string; user_id: string; limit?: string; cursor?: string; status?: string; include_content?: string };
59+
60+ app.post("/internal/crawls", async (req, reply) => {
61+ const parsed = bodySchema.safeParse(req.body);
62+ if (!parsed.success) throw new FetchaError("INVALID_REQUEST", "project_id, user_id and options are required.");
63+ const principal = await principalForProject(parsed.data.project_id, parsed.data.user_id);
64+ const job = await createCrawl(principal, parsed.data.options ?? {}, "playground");
65+ return reply.status(202).send(job);
66+ });
67+ app.get<{ Querystring: Q }>("/internal/crawls", async (req) => {
68+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
69+ return listCrawls(principal, Number(req.query.limit ?? 50) || 50);
70+ });
71+ app.get<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id", async (req) => {
72+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
73+ return getCrawl(principal, req.params.id);
74+ });
75+ app.get<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id/pages", async (req) => {
76+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
77+ const q = pagesQuery.safeParse(req.query);
78+ if (!q.success) throw new FetchaError("INVALID_REQUEST", "Invalid query parameters.");
79+ return listCrawlPages(principal, req.params.id, q.data);
80+ });
81+ app.delete<{ Params: { id: string }; Querystring: Q }>("/internal/crawls/:id", async (req) => {
82+ const principal = await principalForProject(req.query.project_id, req.query.user_id);
83+ return cancelCrawl(principal, req.params.id);
84+ });
85+ app.post("/internal/map", async (req) => {
86+ const parsed = bodySchema.safeParse(req.body);
87+ if (!parsed.success) throw new FetchaError("INVALID_REQUEST", "project_id, user_id and options are required.");
88+ const principal = await principalForProject(parsed.data.project_id, parsed.data.user_id);
89+ return mapSite(principal, parsed.data.options ?? {}, "playground", (req.headers["x-forwarded-for"] as string | undefined)?.split(",")[0]?.trim() ?? req.ip);
90+ });
91+ app.get("/internal/browser", async () => {
92+ const engine = await getEngine();
93+ const s = engine.browser.status();
94+ return { ...s, enabled: engine.browserEnabled, flag_enabled: engine.browserEnabled || !engine.browser.enabled ? engine.browserEnabled : false };
95+ });
96+}
modified apps/api/src/routes/fetch.ts +34 −11
@@ -1,11 +1,11 @@
1 import { FetchaError, PLAN_LIMITS, extractDomain, fetchRequestSchema, newId, type FetchRequest } from "@fetcha/core"; 1 import { FetchaError, PLAN_LIMITS, extractDomain, fetchRequestSchema, newId, type FetchRequest } from "@fetcha/core";
2 import { db, eq, proxySessions, sql } from "@fetcha/db"; 2 import { db, eq, proxySessions, sql } from "@fetcha/db";
3 import type { ProviderId } from "@fetcha/providers"; 3 import type { ProviderId } from "@fetcha/providers";
4−import type { AttemptRecord } from "@fetcha/routing"; 4+import type { AttemptRecord, SerializedCookie } from "@fetcha/routing";
5 import type { ConcreteNetwork } from "@fetcha/core"; 5 import type { ConcreteNetwork } from "@fetcha/core";
6 import type { ApiPrincipal } from "../auth"; 6 import type { ApiPrincipal } from "../auth";
7 import { config } from "../config"; 7 import { config } from "../config";
8−import { acquireConcurrency, checkMonthlyLimits, checkRateLimits, invalidateUsageCache } from "../limits"; 8+import { acquireBrowserSlot, acquireConcurrency, checkMonthlyLimits, checkRateLimits, invalidateUsageCache } from "../limits";
9 import { getEngine } from "../services/engine"; 9 import { getEngine } from "../services/engine";
10 import { completeRequest, createRequestRow, failRequest, recordAttempt, type RequestContext } from "../services/persist"; 10 import { completeRequest, createRequestRow, failRequest, recordAttempt, type RequestContext } from "../services/persist";
11 import { checkSpendAlerts, notifyHardLimit, recordAbuse } from "../services/alerts"; 11 import { checkSpendAlerts, notifyHardLimit, recordAbuse } from "../services/alerts";
@@ -13,7 +13,7 @@ import { checkSpendAlerts, notifyHardLimit, recordAbuse } from "../services/aler
13 export interface FetchHandlerInput { 13 export interface FetchHandlerInput {
14 principal: ApiPrincipal; 14 principal: ApiPrincipal;
15 body: unknown; 15 body: unknown;
16− source: "api" | "playground" | "sdk"; 16+ source: "api" | "playground" | "sdk" | "crawl";
17 clientIp: string | null; 17 clientIp: string | null;
18 userAgent: string | null; 18 userAgent: string | null;
19 } 19 }
@@ -22,10 +22,12 @@ export interface FetchHandlerOutput {
22 requestId: string; 22 requestId: string;
23 status: number; 23 status: number;
24 body: unknown; 24 body: unknown;
25+ /** Internal: execution summary for callers that need more than the public document (crawler). */
26+ summary?: { mode: "http" | "browser"; bytes: number; costUsd: number; attempts: number; domain: string };
25 } 27 }
26 28
27 /** 29 /**
28− * Shared /v1/fetch pipeline used by the public API and the dashboard playground. 30+ * Shared /v1/fetch pipeline used by the public API, the dashboard playground and the crawler.
29 * Order matters: validate → limits → SSRF (inside executor) → route → persist. 31 * Order matters: validate → limits → SSRF (inside executor) → route → persist.
30 */ 32 */
31 export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandlerOutput> { 33 export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandlerOutput> {
@@ -46,12 +48,12 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle
46 if (req.network !== "auto" && !limits.networks.includes(req.network)) { 48 if (req.network !== "auto" && !limits.networks.includes(req.network)) {
47 throw new FetchaError("NETWORK_UNAVAILABLE", `The "${req.network}" network is not included in the ${limits.label} plan.`, { requestId }); 49 throw new FetchaError("NETWORK_UNAVAILABLE", `The "${req.network}" network is not included in the ${limits.label} plan.`, { requestId });
48 } 50 }
49− if (req.browser) throw new FetchaError("BROWSER_UNAVAILABLE", undefined, { requestId }); 51+ const engine = await getEngine();
52+ if (req.browser && (!limits.browser || !engine.browserEnabled)) throw new FetchaError("BROWSER_UNAVAILABLE", undefined, { requestId });
50 53
51 await checkRateLimits(principal, input.clientIp); 54 await checkRateLimits(principal, input.clientIp);
52− let monthly: Awaited<ReturnType<typeof checkMonthlyLimits>>;
53 try { 55 try {
54− monthly = await checkMonthlyLimits(principal); 56+ await checkMonthlyLimits(principal);
55 } catch (e) { 57 } catch (e) {
56 if (e instanceof FetchaError && e.code === "USAGE_LIMIT_REACHED" && (principal.projectHardLimitUsd !== null || principal.orgHardLimitUsd !== null)) { 58 if (e instanceof FetchaError && e.code === "USAGE_LIMIT_REACHED" && (principal.projectHardLimitUsd !== null || principal.orgHardLimitUsd !== null)) {
57 const { monthlyUsage } = await import("../limits"); 59 const { monthlyUsage } = await import("../limits");
@@ -60,6 +62,15 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle
60 throw e; 62 throw e;
61 } 63 }
62 const release = await acquireConcurrency(principal); 64 const release = await acquireConcurrency(principal);
65+ let releaseBrowser: (() => Promise<void>) | null = null;
66+ if (req.browser) {
67+ try {
68+ releaseBrowser = await acquireBrowserSlot(principal);
69+ } catch (e) {
70+ await release();
71+ throw e;
72+ }
73+ }
63 74
64 const domain = extractDomain(req.url); 75 const domain = extractDomain(req.url);
65 const ctx: RequestContext = { 76 const ctx: RequestContext = {
@@ -77,19 +88,22 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle
77 const attempts: AttemptRecord[] = []; 88 const attempts: AttemptRecord[] = [];
78 try { 89 try {
79 await createRequestRow(ctx, req, domain); 90 await createRequestRow(ctx, req, domain);
80− const engine = await getEngine();
81 91
82− // Resolve a Fetcha session (sess_…) into a provider-pinned sticky key. 92+ // Resolve a Fetcha session (sess_…) into a provider-pinned sticky key (+ its cookie jar).
83 let sessionKey: string | null = null; 93 let sessionKey: string | null = null;
84 let sessionProvider: ProviderId | null = null; 94 let sessionProvider: ProviderId | null = null;
85 let sessionNetwork: ConcreteNetwork | null = null; 95 let sessionNetwork: ConcreteNetwork | null = null;
96+ let sessionCookies: SerializedCookie[] | null = null;
97+ let sessionId: string | null = null;
86 if (req.session) { 98 if (req.session) {
87 const [s] = await db.select().from(proxySessions).where(eq(proxySessions.id, req.session)).limit(1); 99 const [s] = await db.select().from(proxySessions).where(eq(proxySessions.id, req.session)).limit(1);
88 if (!s || s.projectId !== principal.projectId) throw new FetchaError("SESSION_NOT_FOUND", undefined, { requestId }); 100 if (!s || s.projectId !== principal.projectId) throw new FetchaError("SESSION_NOT_FOUND", undefined, { requestId });
89 if (s.status !== "active" || s.expiresAt.getTime() < Date.now()) throw new FetchaError("SESSION_EXPIRED", undefined, { requestId }); 101 if (s.status !== "active" || s.expiresAt.getTime() < Date.now()) throw new FetchaError("SESSION_EXPIRED", undefined, { requestId });
90 sessionKey = s.stickyKey; 102 sessionKey = s.stickyKey;
103+ sessionId = s.id;
91 sessionProvider = s.provider as ProviderId; 104 sessionProvider = s.provider as ProviderId;
92 sessionNetwork = s.network as ConcreteNetwork; 105 sessionNetwork = s.network as ConcreteNetwork;
106+ sessionCookies = (s.cookies ?? []) as unknown as SerializedCookie[];
93 if (!req.country && s.country) req.country = s.country; 107 if (!req.country && s.country) req.country = s.country;
94 if (req.network === "auto") req.network = s.network as FetchRequest["network"]; 108 if (req.network === "auto") req.network = s.network as FetchRequest["network"];
95 db.update(proxySessions) 109 db.update(proxySessions)
@@ -106,9 +120,12 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle
106 sessionKey, 120 sessionKey,
107 sessionProvider, 121 sessionProvider,
108 sessionNetwork, 122 sessionNetwork,
123+ sessionCookies,
109 knowledge, 124 knowledge,
110 providerVisibility: principal.providerVisibility, 125 providerVisibility: principal.providerVisibility,
111 maxResponseBytes: Math.min(req.max_response_bytes ?? config.maxResponseBytesDefault, config.maxResponseBytesDefault), 126 maxResponseBytes: Math.min(req.max_response_bytes ?? config.maxResponseBytesDefault, config.maxResponseBytesDefault),
127+ browserPool: engine.browserEnabled && limits.browser ? engine.browser : null,
128+ browserAllowed: limits.browser,
112 onAttempt: async (a) => { 129 onAttempt: async (a) => {
113 attempts.push(a); 130 attempts.push(a);
114 await recordAttempt(requestId, a).catch(() => {}); 131 await recordAttempt(requestId, a).catch(() => {});
@@ -116,13 +133,18 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle
116 }); 133 });
117 const latencyMs = Math.round(performance.now() - started); 134 const latencyMs = Math.round(performance.now() - started);
118 await completeRequest(ctx, req, result, latencyMs); 135 await completeRequest(ctx, req, result, latencyMs);
136+ if (sessionId && result.cookies.length) {
137+ db.update(proxySessions)
138+ .set({ cookies: result.cookies.slice(-200) as unknown as Array<{ name: string; value: string; domain?: string; path?: string }> })
139+ .where(eq(proxySessions.id, sessionId))
140+ .catch(() => {});
141+ }
119 await invalidateUsageCache(principal.organizationId, principal.projectId); 142 await invalidateUsageCache(principal.organizationId, principal.projectId);
120 if (principal.projectSoftLimitUsd !== null || principal.orgSoftLimitUsd !== null) { 143 if (principal.projectSoftLimitUsd !== null || principal.orgSoftLimitUsd !== null) {
121 const { monthlyUsage } = await import("../limits"); 144 const { monthlyUsage } = await import("../limits");
122 monthlyUsage(principal).then((u) => checkSpendAlerts(principal, u)).catch(() => {}); 145 monthlyUsage(principal).then((u) => checkSpendAlerts(principal, u)).catch(() => {});
123 } 146 }
124− void monthly; 147+ return { requestId, status: 200, body: result.body, summary: { mode: result.mode, bytes: result.bytesIn + result.bytesOut, costUsd: result.costUsd, attempts: result.attempts.length, domain } };
125− return { requestId, status: 200, body: result.body };
126 } catch (e) { 148 } catch (e) {
127 const latencyMs = Math.round(performance.now() - started); 149 const latencyMs = Math.round(performance.now() - started);
128 const err = e instanceof FetchaError ? e : new FetchaError("INTERNAL_ERROR", undefined, { requestId, cause: e }); 150 const err = e instanceof FetchaError ? e : new FetchaError("INTERNAL_ERROR", undefined, { requestId, cause: e });
@@ -131,6 +153,7 @@ export async function handleFetch(input: FetchHandlerInput): Promise<FetchHandle
131 if (!(e instanceof FetchaError)) console.error(`[fetch] ${requestId} internal error`, e); 153 if (!(e instanceof FetchaError)) console.error(`[fetch] ${requestId} internal error`, e);
132 throw Object.assign(err, { requestId }); 154 throw Object.assign(err, { requestId });
133 } finally { 155 } finally {
156+ await releaseBrowser?.();
134 await release(); 157 await release();
135 } 158 }
136 } 159 }
modified apps/api/src/routes/misc.ts +3 −1
@@ -40,7 +40,9 @@ export async function usageResponse(p: ApiPrincipal) {
40 latency_p50_ms: Math.round(agg?.p50 ?? 0), 40 latency_p50_ms: Math.round(agg?.p50 ?? 0),
41 latency_p95_ms: Math.round(agg?.p95 ?? 0), 41 latency_p95_ms: Math.round(agg?.p95 ?? 0),
42 }, 42 },
43− remaining_requests: Math.max(0, limits.monthly_requests - usage.requests), 43+ remaining_requests: limits.monthly_requests >= Number.MAX_SAFE_INTEGER ? null : Math.max(0, limits.monthly_requests - usage.requests),
44+ browser: { enabled: true, concurrency: limits.browser_concurrency },
45+ crawl: { max_pages: limits.crawl_max_pages, concurrent_jobs: limits.crawl_concurrent_jobs },
44 }; 46 };
45 } 47 }
46 48
modified apps/api/src/server.ts +13 −4
@@ -3,11 +3,14 @@ import cors from "@fastify/cors";
3 import { FetchaError, isFetchaError, newId } from "@fetcha/core"; 3 import { FetchaError, isFetchaError, newId } from "@fetcha/core";
4 import { closeDb, db, sql } from "@fetcha/db"; 4 import { closeDb, db, sql } from "@fetcha/db";
5 import { closeAllDispatchers } from "@fetcha/providers"; 5 import { closeAllDispatchers } from "@fetcha/providers";
6+import { closeBrowserPool } from "@fetcha/browser";
6 import { authenticateApiKey, requireScope } from "./auth"; 7 import { authenticateApiKey, requireScope } from "./auth";
7 import { assertProductionConfig, config } from "./config"; 8 import { assertProductionConfig, config } from "./config";
8 import { closeKV, getKV } from "./redis"; 9 import { closeKV, getKV } from "./redis";
9 import { handleFetch } from "./routes/fetch"; 10 import { handleFetch } from "./routes/fetch";
10 import { registerInternalRoutes } from "./routes/internal"; 11 import { registerInternalRoutes } from "./routes/internal";
12+import { registerCrawlRoutes, registerInternalCrawlRoutes } from "./routes/crawl";
13+import { startCrawlWorker } from "./services/crawl";
11 import { meResponse, usageResponse } from "./routes/misc"; 14 import { meResponse, usageResponse } from "./routes/misc";
12 import { closeSession, createSession, getSession, listSessions } from "./routes/sessions"; 15 import { closeSession, createSession, getSession, listSessions } from "./routes/sessions";
13 import { getEngine, startHealthLoop } from "./services/engine"; 16 import { getEngine, startHealthLoop } from "./services/engine";
@@ -74,7 +77,7 @@ export async function buildServer() {
74 const engine = await getEngine().catch(() => null); 77 const engine = await getEngine().catch(() => null);
75 checks.providers = Boolean(engine && engine.registry.available().length > 0); 78 checks.providers = Boolean(engine && engine.registry.available().length > 0);
76 const ok = checks.database && checks.cache && checks.providers; 79 const ok = checks.database && checks.cache && checks.providers;
77− return reply.status(ok ? 200 : 503).send({ status: ok ? "ready" : "degraded", checks, available_networks: engine?.registry.availableNetworks() ?? [] }); 80+ return reply.status(ok ? 200 : 503).send({ status: ok ? "ready" : "degraded", checks, available_networks: engine?.registry.availableNetworks() ?? [], browser: engine ? { enabled: engine.browserEnabled, launched: engine.browser.status().launched } : null });
78 }); 81 });
79 82
80 // ---- Public API v1 ---------------------------------------------------------- 83 // ---- Public API v1 ----------------------------------------------------------
@@ -118,15 +121,18 @@ export async function buildServer() {
118 return usageResponse(principal); 121 return usageResponse(principal);
119 }); 122 });
120 123
124+ await registerCrawlRoutes(app);
125+
121 // Explicitly unavailable surfaces (never fake functionality). 126 // Explicitly unavailable surfaces (never fake functionality).
122 app.post("/v1/browser", async () => { 127 app.post("/v1/browser", async () => {
123− throw new FetchaError("BROWSER_UNAVAILABLE", "Browser actions are not yet available. Follow the changelog at https://www.fetcha.co/changelog."); 128+ throw new FetchaError("BROWSER_UNAVAILABLE", "Scripted browser actions are not available yet. Rendered fetches are: send `browser: true` to POST /v1/fetch.");
124 }); 129 });
125 app.post("/v1/extract", async () => { 130 app.post("/v1/extract", async () => {
126− throw new FetchaError("INVALID_REQUEST", "Structured extraction is not yet available on this plan. Follow the changelog at https://www.fetcha.co/changelog."); 131+ throw new FetchaError("INVALID_REQUEST", "Structured extraction is not yet available. Use `format: \"markdown\"` on POST /v1/fetch in the meantime.");
127 }); 132 });
128 133
129 await registerInternalRoutes(app); 134 await registerInternalRoutes(app);
135+ await registerInternalCrawlRoutes(app);
130 return app; 136 return app;
131 } 137 }
132 138
@@ -136,12 +142,15 @@ async function main() {
136 const engine = await getEngine(); 142 const engine = await getEngine();
137 app.log.info({ providers: engine.registry.available().map((p) => p.id), networks: engine.registry.availableNetworks() }, "providers loaded"); 143 app.log.info({ providers: engine.registry.available().map((p) => p.id), networks: engine.registry.availableNetworks() }, "providers loaded");
138 const stopHealth = startHealthLoop(engine, app.log); 144 const stopHealth = startHealthLoop(engine, app.log);
145+ const stopCrawl = startCrawlWorker(app.log);
146+ app.log.info({ browser: engine.browser.status() }, "managed browser pool ready (lazy launch)");
139 147
140 const shutdown = async (signal: string) => { 148 const shutdown = async (signal: string) => {
141 app.log.info(`${signal} received, shutting down`); 149 app.log.info(`${signal} received, shutting down`);
142 stopHealth(); 150 stopHealth();
151+ await Promise.race([stopCrawl(), new Promise((r) => setTimeout(r, 8000))]);
143 await app.close(); 152 await app.close();
144− await Promise.all([closeAllDispatchers(), closeKV(), closeDb()]); 153+ await Promise.all([closeAllDispatchers(), closeBrowserPool(), closeKV(), closeDb()]);
145 process.exit(0); 154 process.exit(0);
146 }; 155 };
147 process.on("SIGINT", () => void shutdown("SIGINT")); 156 process.on("SIGINT", () => void shutdown("SIGINT"));
added apps/api/src/services/crawl.ts +592 −0
@@ -0,0 +1,592 @@
1+import {
2+ FetchaError,
3+ PLAN_LIMITS,
4+ assertUrlAllowed,
5+ crawlCreateSchema,
6+ extractDomain,
7+ extractPageMetadata,
8+ looksLikePage,
9+ mapCreateSchema,
10+ newId,
11+ normalizeCrawlUrl,
12+ parseRobots,
13+ parseSitemap,
14+ registrableHost,
15+ robotsAllows,
16+ urlPatternMatcher,
17+ type CrawlCreateInput,
18+ type FetchResponseBody,
19+ type MapCreateInput,
20+ type RobotsRules,
21+} from "@fetcha/core";
22+import { and, crawlJobs, crawlPages, db, desc, eq, inArray, lt, sql, type CrawlJob, type CrawlPage } from "@fetcha/db";
23+import { hostname } from "node:os";
24+import type { ApiPrincipal } from "../auth";
25+import { handleFetch, type FetchHandlerOutput } from "../routes/fetch";
26+
27+/**
28+ * Crawl service: durable jobs (crawl_jobs / crawl_pages) executed in-process by a small worker
29+ * loop. Every page goes through `handleFetch`, so quotas, routing intelligence, block detection and
30+ * browser escalation apply exactly as for a single fetch.
31+ */
32+
33+const WORKER_ID = `${hostname()}:${process.pid}`;
34+const PAGE_CONTENT_CAP = 600_000; // chars stored per page
35+const HEARTBEAT_MS = 10_000;
36+const STALE_MS = 3 * 60_000;
37+
38+export function serializeJob(j: CrawlJob) {
39+ return {
40+ id: j.id,
41+ status: j.status,
42+ label: j.label,
43+ seed_url: j.seedUrl,
44+ domain: j.domain,
45+ source: j.source,
46+ options: j.options,
47+ stats: { discovered: j.pagesDiscovered, fetched: j.pagesFetched, ok: j.pagesOk, blocked: j.pagesBlocked, failed: j.pagesFailed, bytes: j.bytes },
48+ error: j.errorCode ? { code: j.errorCode, message: j.errorMessage } : null,
49+ webhook_status: j.webhookStatus,
50+ created_at: j.createdAt.toISOString(),
51+ started_at: j.startedAt?.toISOString() ?? null,
52+ completed_at: j.completedAt?.toISOString() ?? null,
53+ };
54+}
55+
56+export function serializePage(p: CrawlPage, includeContent = true) {
57+ return {
58+ id: p.id,
59+ url: p.url,
60+ final_url: p.finalUrl,
61+ depth: p.depth,
62+ parent_url: p.parentUrl,
63+ status: p.status,
64+ http_status: p.httpStatus,
65+ error_code: p.errorCode,
66+ request_id: p.requestId,
67+ title: p.title,
68+ description: p.description,
69+ content_type: p.contentType,
70+ content: includeContent ? p.content : undefined,
71+ links_count: p.linksCount,
72+ bytes: p.bytes,
73+ duration_ms: p.durationMs,
74+ mode: p.mode,
75+ fetched_at: p.fetchedAt?.toISOString() ?? null,
76+ };
77+}
78+
79+// ---------------------------------------------------------------------------
80+// Public operations
81+// ---------------------------------------------------------------------------
82+export async function createCrawl(principal: ApiPrincipal, body: unknown, source: "api" | "playground" | "sdk") {
83+ const parsed = crawlCreateSchema.safeParse(body ?? {});
84+ if (!parsed.success) {
85+ throw new FetchaError("INVALID_REQUEST", "Invalid crawl options.", { details: { issues: parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
86+ }
87+ const opts = parsed.data;
88+ const limits = PLAN_LIMITS[principal.plan];
89+ if (opts.max_pages > limits.crawl_max_pages) opts.max_pages = limits.crawl_max_pages;
90+ if (opts.timeout > limits.max_timeout_ms) opts.timeout = limits.max_timeout_ms;
91+ if (opts.network !== "auto" && !limits.networks.includes(opts.network)) throw new FetchaError("NETWORK_UNAVAILABLE");
92+ const allowed = await assertUrlAllowed(opts.url);
93+ const seed = normalizeCrawlUrl(allowed.url.toString());
94+ if (!seed) throw new FetchaError("INVALID_REQUEST", "The seed URL is invalid.");
95+ if (opts.webhook_url) await assertUrlAllowed(opts.webhook_url);
96+
97+ const [active] = await db
98+ .select({ n: sql<number>`count(*)::int` })
99+ .from(crawlJobs)
100+ .where(and(eq(crawlJobs.organizationId, principal.organizationId), inArray(crawlJobs.status, ["queued", "running"])));
101+ if ((active?.n ?? 0) >= limits.crawl_concurrent_jobs) {
102+ throw new FetchaError("CRAWL_LIMIT_REACHED", `At most ${limits.crawl_concurrent_jobs} crawl jobs can be queued or running at once.`, { details: { limit: limits.crawl_concurrent_jobs } });
103+ }
104+
105+ const id = newId("crawl");
106+ await db.insert(crawlJobs).values({
107+ id,
108+ organizationId: principal.organizationId,
109+ projectId: principal.projectId,
110+ apiKeyId: principal.keyId,
111+ source,
112+ label: opts.label ?? null,
113+ seedUrl: seed,
114+ domain: extractDomain(seed),
115+ options: { ...opts, url: seed } as Record<string, unknown>,
116+ status: "queued",
117+ pagesDiscovered: 1,
118+ });
119+ await db.insert(crawlPages).values({ id: newId("cpg"), jobId: id, url: seed, depth: 0, status: "pending" }).onConflictDoNothing();
120+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, id)).limit(1);
121+ kick();
122+ return serializeJob(job!);
123+}
124+
125+export async function getCrawl(principal: ApiPrincipal, id: string) {
126+ const [job] = await db.select().from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
127+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
128+ return serializeJob(job);
129+}
130+
131+export async function listCrawls(principal: ApiPrincipal, limit = 50) {
132+ const rows = await db.select().from(crawlJobs).where(eq(crawlJobs.projectId, principal.projectId)).orderBy(desc(crawlJobs.createdAt)).limit(Math.min(Math.max(1, limit), 200));
133+ return { data: rows.map(serializeJob) };
134+}
135+
136+export async function listCrawlPages(principal: ApiPrincipal, id: string, q: { cursor?: string; limit?: number; status?: string; include_content?: boolean }) {
137+ const [job] = await db.select({ id: crawlJobs.id }).from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
138+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
139+ const limit = Math.min(Math.max(1, q.limit ?? 100), 500);
140+ const conds = [eq(crawlPages.jobId, id)];
141+ if (q.status) conds.push(eq(crawlPages.status, q.status));
142+ else conds.push(sql`${crawlPages.status} <> 'pending'`);
143+ if (q.cursor) {
144+ const [ts, cid] = decodeCursor(q.cursor);
145+ conds.push(sql`(${crawlPages.createdAt}, ${crawlPages.id}) > (${new Date(ts)}, ${cid})`);
146+ }
147+ const rows = await db
148+ .select()
149+ .from(crawlPages)
150+ .where(and(...conds))
151+ .orderBy(crawlPages.createdAt, crawlPages.id)
152+ .limit(limit + 1);
153+ const page = rows.slice(0, limit);
154+ const last = page.at(-1);
155+ return { data: page.map((p) => serializePage(p, q.include_content !== false)), next_cursor: rows.length > limit && last ? encodeCursor(last.createdAt, last.id) : null };
156+}
157+
158+export async function cancelCrawl(principal: ApiPrincipal, id: string) {
159+ const res = await db
160+ .update(crawlJobs)
161+ .set({ status: "cancelled", completedAt: new Date() })
162+ .where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId), inArray(crawlJobs.status, ["queued", "running"])))
163+ .returning({ id: crawlJobs.id });
164+ if (!res.length) {
165+ const [job] = await db.select({ id: crawlJobs.id, status: crawlJobs.status }).from(crawlJobs).where(and(eq(crawlJobs.id, id), eq(crawlJobs.projectId, principal.projectId))).limit(1);
166+ if (!job) throw new FetchaError("CRAWL_NOT_FOUND");
167+ return { id, status: job.status };
168+ }
169+ cancelled.add(id);
170+ return { id, status: "cancelled" };
171+}
172+
173+function encodeCursor(d: Date, id: string): string {
174+ return Buffer.from(`${d.toISOString()}|${id}`).toString("base64url");
175+}
176+function decodeCursor(c: string): [string, string] {
177+ try {
178+ const [ts, id] = Buffer.from(c, "base64url").toString("utf8").split("|");
179+ if (!ts || !id || Number.isNaN(Date.parse(ts))) throw new Error("bad");
180+ return [ts, id];
181+ } catch {
182+ throw new FetchaError("INVALID_REQUEST", "Invalid cursor.");
183+ }
184+}
185+
186+// ---------------------------------------------------------------------------
187+// Map (sync)
188+// ---------------------------------------------------------------------------
189+export async function mapSite(principal: ApiPrincipal, body: unknown, source: "api" | "playground" | "sdk", clientIp: string | null) {
190+ const parsed = mapCreateSchema.safeParse(body ?? {});
191+ if (!parsed.success) {
192+ throw new FetchaError("INVALID_REQUEST", "Invalid map options.", { details: { issues: parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message })) } });
193+ }
194+ const opts = parsed.data;
195+ const allowed = await assertUrlAllowed(opts.url);
196+ const seed = normalizeCrawlUrl(allowed.url.toString())!;
197+ const seedUrl = new URL(seed);
198+ const host = registrableHost(seedUrl.hostname);
199+ const inScope = scopeFilter(seedUrl, opts.same_domain, opts.allow_subdomains);
200+ const search = opts.search ? urlPatternMatcher([opts.search.includes("*") || /^\/.+\/[a-z]*$/.test(opts.search) ? opts.search : `*${opts.search}*`]) : null;
201+ const deadline = Date.now() + Math.min(55_000, opts.timeout * 2);
202+ const found = new Map<string, "sitemap" | "links">();
203+ const add = (u: string, src: "sitemap" | "links") => {
204+ const n = normalizeCrawlUrl(u);
205+ if (!n || found.has(n) || !inScope(n) || !looksLikePage(n)) return;
206+ if (search && !search(n)) return;
207+ found.set(n, src);
208+ };
209+ const fetchText = async (url: string, format: "raw" | "html" = "raw", extra: Record<string, unknown> = {}): Promise<FetchResponseBody | null> => {
210+ if (Date.now() > deadline - 2000) return null;
211+ try {
212+ const out = await handleFetch({ principal, body: { url, format, timeout: Math.min(opts.timeout, Math.max(3000, deadline - Date.now() - 1000)), country: opts.country, network: opts.network, browser_fallback: false, retries: 1, ...extra }, source, clientIp, userAgent: `fetcha-map/${source}` });
213+ return out.body as FetchResponseBody;
214+ } catch {
215+ return null;
216+ }
217+ };
218+ let sitemapCount = 0;
219+ let linksCount = 0;
220+ if (opts.use_sitemap) {
221+ const sitemapUrls = new Set<string>([`${seedUrl.origin}/sitemap.xml`, `${seedUrl.origin}/sitemap_index.xml`]);
222+ const robots = await fetchText(`${seedUrl.origin}/robots.txt`);
223+ if (robots?.success && robots.content) for (const s of parseRobots(robots.content).sitemaps) sitemapUrls.add(s);
224+ const queue = [...sitemapUrls];
225+ const seen = new Set<string>();
226+ while (queue.length && seen.size < 25 && found.size < opts.limit && Date.now() < deadline - 3000) {
227+ const u = queue.shift()!;
228+ if (seen.has(u)) continue;
229+ seen.add(u);
230+ const r = await fetchText(u);
231+ if (!r?.success || !r.content || !/<(urlset|sitemapindex)|<loc>/i.test(r.content.slice(0, 5000)) && !/^https?:\/\//m.test(r.content.slice(0, 500))) continue;
232+ const { urls, sitemaps } = parseSitemap(r.content);
233+ for (const s of sitemaps) if (host === safeHost(s) || opts.allow_subdomains) queue.push(s);
234+ for (const x of urls) {
235+ if (found.size >= opts.limit) break;
236+ const before = found.size;
237+ add(x, "sitemap");
238+ if (found.size > before) sitemapCount++;
239+ }
240+ }
241+ }
242+ if (opts.use_links && found.size < opts.limit && Date.now() < deadline - 3000) {
243+ const r = await fetchText(seed, "html", { links: true });
244+ if (r?.success && r.links) {
245+ for (const l of r.links) {
246+ if (found.size >= opts.limit) break;
247+ const before = found.size;
248+ add(l.url, "links");
249+ if (found.size > before) linksCount++;
250+ }
251+ } else if (r?.content) {
252+ const meta = extractPageMetadata(r.content, r.final_url);
253+ for (const l of meta.links) {
254+ if (found.size >= opts.limit) break;
255+ const before = found.size;
256+ add(l.url, "links");
257+ if (found.size > before) linksCount++;
258+ }
259+ }
260+ }
261+ const urls = [...found.keys()];
262+ return { url: seed, count: urls.length, urls, sources: { sitemap: sitemapCount, links: linksCount }, truncated: urls.length >= opts.limit };
263+}
264+
265+function safeHost(u: string): string {
266+ try {
267+ return registrableHost(new URL(u).hostname);
268+ } catch {
269+ return "";
270+ }
271+}
272+
273+function scopeFilter(seed: URL, sameDomain: boolean, allowSubdomains: boolean): (url: string) => boolean {
274+ const seedHost = seed.hostname.toLowerCase().replace(/^www\./, "");
275+ const reg = registrableHost(seed.hostname);
276+ return (url: string) => {
277+ if (!sameDomain) return true;
278+ let h: string;
279+ try {
280+ h = new URL(url).hostname.toLowerCase().replace(/^www\./, "");
281+ } catch {
282+ return false;
283+ }
284+ if (h === seedHost) return true;
285+ if (allowSubdomains) return registrableHost(h) === reg;
286+ return false;
287+ };
288+}
289+
290+// ---------------------------------------------------------------------------
291+// Worker
292+// ---------------------------------------------------------------------------
293+const cancelled = new Set<string>();
294+let running = false;
295+let stopped = false;
296+let wake: (() => void) | null = null;
297+const activeJobs = new Map<string, Promise<void>>();
298+const MAX_PARALLEL_JOBS = Number(process.env.FETCHA_CRAWL_PARALLEL_JOBS ?? 4);
299+
300+function kick() {
301+ wake?.();
302+}
303+
304+export function startCrawlWorker(log: { info: (m: string) => void; warn: (m: string) => void }): () => Promise<void> {
305+ if (running) return async () => {};
306+ running = true;
307+ stopped = false;
308+ const loop = async () => {
309+ // Re-queue jobs left "running" by a previous process (crash / redeploy).
310+ await db
311+ .update(crawlJobs)
312+ .set({ status: "queued", workerId: null })
313+ .where(and(eq(crawlJobs.status, "running"), lt(crawlJobs.heartbeatAt, new Date(Date.now() - STALE_MS))))
314+ .catch(() => {});
315+ while (!stopped) {
316+ try {
317+ if (activeJobs.size < MAX_PARALLEL_JOBS) {
318+ const claimed = await claimJob();
319+ if (claimed) {
320+ const p = runJob(claimed, log)
321+ .catch((e) => log.warn(`crawl ${claimed.id} crashed: ${(e as Error).message}`))
322+ .finally(() => activeJobs.delete(claimed.id));
323+ activeJobs.set(claimed.id, p);
324+ continue;
325+ }
326+ }
327+ } catch (e) {
328+ log.warn(`crawl worker loop error: ${(e as Error).message}`);
329+ }
330+ await new Promise<void>((r) => {
331+ wake = r;
332+ setTimeout(r, 4000);
333+ });
334+ wake = null;
335+ }
336+ };
337+ void loop();
338+ log.info(`crawl worker started (${WORKER_ID}, parallel jobs=${MAX_PARALLEL_JOBS})`);
339+ return async () => {
340+ stopped = true;
341+ kick();
342+ await Promise.allSettled([...activeJobs.values()]);
343+ running = false;
344+ };
345+}
346+
347+async function claimJob(): Promise<CrawlJob | null> {
348+ const rows = await db.execute(sql`
349+ update crawl_jobs set status = 'running', worker_id = ${WORKER_ID}, started_at = coalesce(started_at, now()), heartbeat_at = now()
350+ where id = (select id from crawl_jobs where status = 'queued' order by created_at asc limit 1 for update skip locked)
351+ returning *
352+ `);
353+ const r = (rows as unknown as { rows: Record<string, unknown>[] }).rows?.[0];
354+ if (!r) return null;
355+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, r.id as string)).limit(1);
356+ return job ?? null;
357+}
358+
359+interface Frontier {
360+ url: string;
361+ depth: number;
362+ parent: string | null;
363+}
364+
365+async function runJob(job: CrawlJob, log: { info: (m: string) => void; warn: (m: string) => void }): Promise<void> {
366+ const opts = crawlCreateSchema.parse(job.options) as CrawlCreateInput;
367+ const { principalForCrawl } = await import("../auth");
368+ let principal: ApiPrincipal;
369+ try {
370+ principal = await principalForCrawl(job.projectId, job.apiKeyId);
371+ } catch (e) {
372+ await finishJob(job.id, "failed", { code: (e as FetchaError).code ?? "INTERNAL_ERROR", message: (e as Error).message });
373+ return;
374+ }
375+ const seedUrl = new URL(job.seedUrl);
376+ const inScope = scopeFilter(seedUrl, opts.same_domain, opts.allow_subdomains);
377+ const include = urlPatternMatcher(opts.include_patterns);
378+ const exclude = urlPatternMatcher(opts.exclude_patterns);
379+ const seen = new Set<string>();
380+ const frontier: Frontier[] = [];
381+ let discovered = 0;
382+ let fetched = 0;
383+ let ok = 0;
384+ let blocked = 0;
385+ let failed = 0;
386+ let bytes = 0;
387+ let costUsd = 0;
388+ let robots: RobotsRules | null = null;
389+ let delayMs = opts.delay_ms;
390+
391+ // Resume support: already-known pages of this job.
392+ const existing = await db.select({ url: crawlPages.url, status: crawlPages.status, depth: crawlPages.depth, parentUrl: crawlPages.parentUrl }).from(crawlPages).where(eq(crawlPages.jobId, job.id));
393+ for (const p of existing) {
394+ seen.add(p.url);
395+ discovered++;
396+ if (p.status === "pending") frontier.push({ url: p.url, depth: p.depth, parent: p.parentUrl });
397+ else {
398+ fetched++;
399+ if (p.status === "success") ok++;
400+ else if (p.status === "blocked") blocked++;
401+ else if (p.status === "failed") failed++;
402+ }
403+ }
404+ bytes = job.bytes;
405+ costUsd = job.costUsd;
406+
407+ const fetchPage = async (url: string, extra: Record<string, unknown> = {}): Promise<FetchHandlerOutput | FetchaError> => {
408+ try {
409+ return await handleFetch({
410+ principal,
411+ body: {
412+ url,
413+ format: opts.format === "html" ? "html" : opts.format,
414+ timeout: opts.timeout,
415+ country: opts.country,
416+ network: opts.network,
417+ browser: opts.browser,
418+ browser_fallback: opts.browser_fallback,
419+ headers: opts.headers,
420+ links: true,
421+ ...extra,
422+ },
423+ source: "crawl",
424+ clientIp: null,
425+ userAgent: `fetcha-crawl/${job.id}`,
426+ });
427+ } catch (e) {
428+ return e instanceof FetchaError ? e : new FetchaError("INTERNAL_ERROR", (e as Error).message);
429+ }
430+ };
431+
432+ // robots.txt + optional sitemap seeding
433+ if (opts.respect_robots) {
434+ const r = await fetchPage(`${seedUrl.origin}/robots.txt`, { format: "raw", links: false, browser: false, browser_fallback: false, retries: 1, timeout: Math.min(opts.timeout, 15_000) });
435+ if (!(r instanceof FetchaError)) {
436+ const b = r.body as FetchResponseBody;
437+ if (b.success && b.content && (b.content_type ?? "").includes("text/plain")) {
438+ robots = parseRobots(b.content, "fetchabot");
439+ if (robots.crawlDelayMs && robots.crawlDelayMs > delayMs) delayMs = Math.min(robots.crawlDelayMs, 10_000);
440+ }
441+ }
442+ }
443+ const enqueue = async (candidates: Array<{ url: string; depth: number; parent: string | null }>) => {
444+ const rows: Array<{ id: string; jobId: string; url: string; depth: number; parentUrl: string | null; status: string }> = [];
445+ for (const c of candidates) {
446+ if (discovered >= opts.max_pages * 3 || frontier.length + fetched >= opts.max_pages) break;
447+ const n = normalizeCrawlUrl(c.url);
448+ if (!n || seen.has(n)) continue;
449+ if (!inScope(n) || !looksLikePage(n)) continue;
450+ if (exclude && exclude(n)) continue;
451+ if (include && !include(n) && c.depth > 0) continue;
452+ if (robots && !robotsAllows(robots, n)) continue;
453+ seen.add(n);
454+ discovered++;
455+ frontier.push({ url: n, depth: c.depth, parent: c.parent });
456+ rows.push({ id: newId("cpg"), jobId: job.id, url: n, depth: c.depth, parentUrl: c.parent, status: "pending" });
457+ }
458+ if (rows.length) await db.insert(crawlPages).values(rows).onConflictDoNothing().catch(() => {});
459+ };
460+ if (opts.use_sitemap && fetched === 0) {
461+ const sitemapUrls = new Set<string>([`${seedUrl.origin}/sitemap.xml`]);
462+ for (const s of robots?.sitemaps ?? []) sitemapUrls.add(s);
463+ let count = 0;
464+ for (const su of sitemapUrls) {
465+ if (count++ >= 5 || frontier.length >= opts.max_pages) break;
466+ const r = await fetchPage(su, { format: "raw", links: false, browser: false, browser_fallback: false, retries: 1 });
467+ if (r instanceof FetchaError) continue;
468+ const b = r.body as FetchResponseBody;
469+ if (!b.success || !b.content) continue;
470+ const { urls, sitemaps } = parseSitemap(b.content);
471+ for (const s of sitemaps.slice(0, 5)) sitemapUrls.add(s);
472+ await enqueue(urls.slice(0, opts.max_pages).map((u) => ({ url: u, depth: 1, parent: su })));
473+ }
474+ }
475+
476+ let lastHeartbeat = 0;
477+ let lastPersist = 0;
478+ const persistStats = async (force = false) => {
479+ const now = Date.now();
480+ if (!force && now - lastPersist < 2000) return;
481+ lastPersist = now;
482+ await db
483+ .update(crawlJobs)
484+ .set({ pagesDiscovered: discovered, pagesFetched: fetched, pagesOk: ok, pagesBlocked: blocked, pagesFailed: failed, bytes, costUsd, heartbeatAt: new Date() })
485+ .where(eq(crawlJobs.id, job.id))
486+ .catch(() => {});
487+ lastHeartbeat = now;
488+ };
489+ const isCancelled = async (): Promise<boolean> => {
490+ if (cancelled.has(job.id)) return true;
491+ if (Date.now() - lastHeartbeat > HEARTBEAT_MS) {
492+ const [row] = await db.select({ status: crawlJobs.status }).from(crawlJobs).where(eq(crawlJobs.id, job.id)).limit(1);
493+ lastHeartbeat = Date.now();
494+ if (row?.status === "cancelled") {
495+ cancelled.add(job.id);
496+ return true;
497+ }
498+ }
499+ return false;
500+ };
501+
502+ const worker = async () => {
503+ while (!stopped) {
504+ if (fetched >= opts.max_pages) return;
505+ if (await isCancelled()) return;
506+ const item = frontier.shift();
507+ if (!item) return;
508+ fetched++;
509+ const t0 = Date.now();
510+ const r = await fetchPage(item.url);
511+ const durationMs = Date.now() - t0;
512+ const update: Partial<typeof crawlPages.$inferInsert> = { fetchedAt: new Date(), durationMs };
513+ if (r instanceof FetchaError) {
514+ failed++;
515+ Object.assign(update, { status: "failed", errorCode: r.code, requestId: r.requestId ?? null });
516+ } else {
517+ const b = r.body as FetchResponseBody;
518+ const content = opts.format === "markdown" ? b.markdown : opts.format === "text" ? b.text : b.content;
519+ const isBlocked = !b.success && b.status >= 400 && r.summary && b.metadata.attempts > 0 && !(b.status === 404 || b.status === 410);
520+ bytes += r.summary?.bytes ?? b.metadata.bytes;
521+ costUsd += r.summary?.costUsd ?? 0;
522+ if (b.success) ok++;
523+ else if (isBlocked && b.status !== 500) blocked++;
524+ else failed++;
525+ Object.assign(update, {
526+ status: b.success ? "success" : isBlocked && b.status !== 500 ? "blocked" : "failed",
527+ httpStatus: b.status,
528+ errorCode: b.success ? null : b.status === 403 || b.status === 429 || b.status === 503 ? "TARGET_BLOCKED" : `HTTP_${b.status}`,
529+ requestId: b.request_id,
530+ finalUrl: b.final_url,
531+ title: b.page?.title ?? null,
532+ description: b.page?.description ?? null,
533+ contentType: b.content_type,
534+ content: b.success && typeof content === "string" ? content.slice(0, PAGE_CONTENT_CAP) : null,
535+ linksCount: b.page?.links_count ?? 0,
536+ bytes: r.summary?.bytes ?? b.metadata.bytes,
537+ mode: b.metadata.mode,
538+ });
539+ if (b.success && item.depth < opts.max_depth && b.links?.length) {
540+ await enqueue(b.links.filter((l) => l.internal || !opts.same_domain).map((l) => ({ url: l.url, depth: item.depth + 1, parent: item.url })));
541+ }
542+ }
543+ await db.update(crawlPages).set(update).where(and(eq(crawlPages.jobId, job.id), eq(crawlPages.url, item.url))).catch(() => {});
544+ await persistStats();
545+ if (delayMs > 0) await new Promise((res) => setTimeout(res, delayMs));
546+ }
547+ };
548+
549+ try {
550+ await Promise.all(Array.from({ length: Math.max(1, Math.min(opts.concurrency, 10)) }, () => worker()));
551+ await persistStats(true);
552+ if (cancelled.has(job.id)) {
553+ await finishJob(job.id, "cancelled");
554+ cancelled.delete(job.id);
555+ } else if (stopped && frontier.length) {
556+ await db.update(crawlJobs).set({ status: "queued", workerId: null }).where(eq(crawlJobs.id, job.id)).catch(() => {});
557+ } else {
558+ await finishJob(job.id, "completed");
559+ }
560+ log.info(`crawl ${job.id} ${cancelled.has(job.id) ? "cancelled" : "done"}: ${fetched} fetched, ${ok} ok, ${blocked} blocked, ${failed} failed`);
561+ if (opts.webhook_url && !stopped) await deliverWebhook(job.id, opts.webhook_url).catch(() => {});
562+ } catch (e) {
563+ await persistStats(true);
564+ await finishJob(job.id, "failed", { code: "INTERNAL_ERROR", message: (e as Error).message?.slice(0, 500) });
565+ throw e;
566+ }
567+}
568+
569+async function finishJob(id: string, status: "completed" | "failed" | "cancelled", error?: { code: string; message?: string }) {
570+ await db
571+ .update(crawlJobs)
572+ .set({ status, completedAt: new Date(), errorCode: error?.code ?? null, errorMessage: error?.message ?? null })
573+ .where(and(eq(crawlJobs.id, id), inArray(crawlJobs.status, ["running", "queued", "cancelled"])))
574+ .catch(() => {});
575+}
576+
577+async function deliverWebhook(jobId: string, url: string) {
578+ const [job] = await db.select().from(crawlJobs).where(eq(crawlJobs.id, jobId)).limit(1);
579+ if (!job) return;
580+ let status = "failed";
581+ try {
582+ await assertUrlAllowed(url);
583+ const ac = new AbortController();
584+ const t = setTimeout(() => ac.abort(), 10_000);
585+ const res = await fetch(url, { method: "POST", headers: { "content-type": "application/json", "user-agent": "fetcha-webhook/0.2" }, body: JSON.stringify({ type: "crawl.completed", job: serializeJob(job) }), signal: ac.signal });
586+ clearTimeout(t);
587+ status = res.ok ? `delivered:${res.status}` : `failed:${res.status}`;
588+ } catch (e) {
589+ status = `failed:${(e as Error).message?.slice(0, 80)}`;
590+ }
591+ await db.update(crawlJobs).set({ webhookStatus: status }).where(eq(crawlJobs.id, jobId)).catch(() => {});
592+}
modified apps/api/src/services/engine.ts +14 −2
@@ -1,6 +1,7 @@
1−import { db, providerConfigs, providerHealth, domainProfiles, eq } from "@fetcha/db"; 1+import { db, providerConfigs, providerHealth, domainProfiles, featureFlags, eq } from "@fetcha/db";
2 import { ProviderRegistry, type ProviderId } from "@fetcha/providers"; 2 import { ProviderRegistry, type ProviderId } from "@fetcha/providers";
3 import { CircuitBreaker, FetchExecutor, RoutingEngine, type DomainKnowledge } from "@fetcha/routing"; 3 import { CircuitBreaker, FetchExecutor, RoutingEngine, type DomainKnowledge } from "@fetcha/routing";
4+import { getBrowserPool, type BrowserPool } from "@fetcha/browser";
4 import { newId } from "@fetcha/core"; 5 import { newId } from "@fetcha/core";
5 import { config } from "../config"; 6 import { config } from "../config";
6 7
@@ -9,6 +10,9 @@ export interface Engine {
9 circuit: CircuitBreaker; 10 circuit: CircuitBreaker;
10 routing: RoutingEngine; 11 routing: RoutingEngine;
11 executor: FetchExecutor; 12 executor: FetchExecutor;
13+ browser: BrowserPool;
14+ /** Global kill-switch from the `browser_enabled` feature flag (default on). */
15+ browserEnabled: boolean;
12 reload(): Promise<void>; 16 reload(): Promise<void>;
13 probeAll(): Promise<void>; 17 probeAll(): Promise<void>;
14 knowledge(domain: string): Promise<DomainKnowledge | null>; 18 knowledge(domain: string): Promise<DomainKnowledge | null>;
@@ -22,12 +26,18 @@ export async function getEngine(): Promise<Engine> {
22 let registry = new ProviderRegistry(); 26 let registry = new ProviderRegistry();
23 let routing = new RoutingEngine(registry, circuit); 27 let routing = new RoutingEngine(registry, circuit);
24 let executor = new FetchExecutor(routing, circuit); 28 let executor = new FetchExecutor(routing, circuit);
29+ const browser = getBrowserPool({ log: { info: (m) => console.log("[browser]", m), warn: (m) => console.warn("[browser]", m) } });
30+ let browserEnabled = true;
25 31
26 const engine: Engine = { 32 const engine: Engine = {
27 get registry() { 33 get registry() {
28 return registry; 34 return registry;
29 }, 35 },
30 circuit, 36 circuit,
37+ browser,
38+ get browserEnabled() {
39+ return browserEnabled && browser.enabled;
40+ },
31 get routing() { 41 get routing() {
32 return routing; 42 return routing;
33 }, 43 },
@@ -46,6 +56,8 @@ export async function getEngine(): Promise<Engine> {
46 registry = new ProviderRegistry({ prices, disabled }); 56 registry = new ProviderRegistry({ prices, disabled });
47 routing = new RoutingEngine(registry, circuit); 57 routing = new RoutingEngine(registry, circuit);
48 executor = new FetchExecutor(routing, circuit); 58 executor = new FetchExecutor(routing, circuit);
59+ const [flag] = await db.select().from(featureFlags).where(eq(featureFlags.key, "browser_enabled")).limit(1).catch(() => []);
60+ browserEnabled = flag ? flag.enabled : true;
49 }, 61 },
50 async probeAll() { 62 async probeAll() {
51 const results = await Promise.all( 63 const results = await Promise.all(
@@ -62,7 +74,7 @@ export async function getEngine(): Promise<Engine> {
62 if (!domain) return null; 74 if (!domain) return null;
63 const [row] = await db.select().from(domainProfiles).where(eq(domainProfiles.domain, domain)).limit(1).catch(() => []); 75 const [row] = await db.select().from(domainProfiles).where(eq(domainProfiles.domain, domain)).limit(1).catch(() => []);
64 if (!row) return null; 76 if (!row) return null;
65− return { domain, routeStats: row.routeStats ?? {}, policy: row.policy ?? null, browserRequiredRate: row.requests ? row.browserRequired / row.requests : 0 }; 77+ return { domain, routeStats: row.routeStats ?? {}, policy: row.policy ?? null, browserRequiredRate: row.requests ? row.browserRequired / row.requests : 0, browserSamples: row.requests };
66 }, 78 },
67 }; 79 };
68 await engine.reload(); 80 await engine.reload();
modified apps/api/src/services/persist.ts +11 −5
@@ -8,7 +8,7 @@ export interface RequestContext {
8 organizationId: string; 8 organizationId: string;
9 projectId: string; 9 projectId: string;
10 apiKeyId: string | null; 10 apiKeyId: string | null;
11− source: "api" | "playground" | "sdk"; 11+ source: "api" | "playground" | "sdk" | "crawl";
12 plan: Plan; 12 plan: Plan;
13 clientIp: string | null; 13 clientIp: string | null;
14 userAgent: string | null; 14 userAgent: string | null;
@@ -46,12 +46,13 @@ export async function recordAttempt(requestId: string, a: AttemptRecord): Promis
46 attemptNo: a.attemptNo, 46 attemptNo: a.attemptNo,
47 provider: a.provider, 47 provider: a.provider,
48 network: a.network, 48 network: a.network,
49+ mode: a.mode,
49 country: a.country, 50 country: a.country,
50 sessionKey: a.sessionKey, 51 sessionKey: a.sessionKey,
51 outcome: a.outcome, 52 outcome: a.outcome,
52 httpStatus: a.httpStatus, 53 httpStatus: a.httpStatus,
53 errorCode: a.errorCode, 54 errorCode: a.errorCode,
54− errorDetail: a.errorDetail, 55+ errorDetail: a.errorDetail ?? (a.blockVendor ? `vendor=${a.blockVendor} profile=${a.profileId ?? "-"}` : a.profileId ? `profile=${a.profileId}` : null),
55 blockReason: a.blockReason, 56 blockReason: a.blockReason,
56 durationMs: a.durationMs, 57 durationMs: a.durationMs,
57 bytesIn: a.bytesIn, 58 bytesIn: a.bytesIn,
@@ -83,6 +84,8 @@ export async function completeRequest(ctx: RequestContext, req: FetchRequest, re
83 errorMessage: result.body.success ? null : "The target blocked every route we tried.", 84 errorMessage: result.body.success ? null : "The target blocked every route we tried.",
84 finalUrl: result.finalUrl, 85 finalUrl: result.finalUrl,
85 network, 86 network,
87+ mode: result.mode,
88+ browser: result.mode === "browser",
86 attempts: result.attempts.length, 89 attempts: result.attempts.length,
87 latencyMs, 90 latencyMs,
88 bytesIn: result.bytesIn, 91 bytesIn: result.bytesIn,
@@ -96,7 +99,7 @@ export async function completeRequest(ctx: RequestContext, req: FetchRequest, re
96 .where(sql`${fetchRequests.id} = ${ctx.requestId}`); 99 .where(sql`${fetchRequests.id} = ${ctx.requestId}`);
97 100
98 await writeUsage(ctx, result, price.totalUsd, network); 101 await writeUsage(ctx, result, price.totalUsd, network);
99− await Promise.all([updateDomainProfile(result.domain, result.attempts, result.body.success), updateRoutingMetrics(result.attempts)]); 102+ await Promise.all([updateDomainProfile(result.domain, result.attempts, result.body.success, result.browserRequired), updateRoutingMetrics(result.attempts)]);
100 return { priceUsd: price.totalUsd, costUsd: result.costUsd }; 103 return { priceUsd: price.totalUsd, costUsd: result.costUsd };
101 } 104 }
102 105
@@ -118,7 +121,7 @@ export async function failRequest(ctx: RequestContext, code: string, message: st
118 costUsd: 0, 121 costUsd: 0,
119 upstreamCostUsd: costUsd, 122 upstreamCostUsd: costUsd,
120 }); 123 });
121− if (attempts.length) await Promise.all([updateDomainProfile(domain, attempts, false), updateRoutingMetrics(attempts)]); 124+ if (attempts.length) await Promise.all([updateDomainProfile(domain, attempts, false, false), updateRoutingMetrics(attempts)]);
122 } 125 }
123 126
124 async function writeUsage(ctx: RequestContext, result: ExecutionResult, priceUsd: number, network: ConcreteNetwork | null) { 127 async function writeUsage(ctx: RequestContext, result: ExecutionResult, priceUsd: number, network: ConcreteNetwork | null) {
@@ -134,7 +137,7 @@ async function writeUsage(ctx: RequestContext, result: ExecutionResult, priceUsd
134 ); 137 );
135 } 138 }
136 139
137−async function updateDomainProfile(domain: string, attempts: AttemptRecord[], success: boolean) { 140+async function updateDomainProfile(domain: string, attempts: AttemptRecord[], success: boolean, browserRequired: boolean) {
138 if (!domain) return; 141 if (!domain) return;
139 const [existing] = await db.select().from(domainProfiles).where(sql`${domainProfiles.domain} = ${domain}`).limit(1); 142 const [existing] = await db.select().from(domainProfiles).where(sql`${domainProfiles.domain} = ${domain}`).limit(1);
140 const stats = { ...(existing?.routeStats ?? {}) }; 143 const stats = { ...(existing?.routeStats ?? {}) };
@@ -145,6 +148,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su
145 if (a.outcome === "blocked") blocks++; 148 if (a.outcome === "blocked") blocks++;
146 if (a.blockReason === "captcha" || a.blockReason === "cloudflare_challenge") captchas++; 149 if (a.blockReason === "captcha" || a.blockReason === "cloudflare_challenge") captchas++;
147 } 150 }
151+ const browserInc = browserRequired ? 1 : 0;
148 const pref = preferredRoute(stats); 152 const pref = preferredRoute(stats);
149 const totalLatency = attempts.reduce((s, a) => s + a.durationMs, 0); 153 const totalLatency = attempts.reduce((s, a) => s + a.durationMs, 0);
150 const n = (existing?.requests ?? 0) + 1; 154 const n = (existing?.requests ?? 0) + 1;
@@ -159,6 +163,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su
159 successes: success ? 1 : 0, 163 successes: success ? 1 : 0,
160 blocks, 164 blocks,
161 captchas, 165 captchas,
166+ browserRequired: browserInc,
162 avgLatencyMs: totalLatency, 167 avgLatencyMs: totalLatency,
163 routeStats: stats, 168 routeStats: stats,
164 lastSeenAt: new Date(), 169 lastSeenAt: new Date(),
@@ -172,6 +177,7 @@ async function updateDomainProfile(domain: string, attempts: AttemptRecord[], su
172 successes: sql`${domainProfiles.successes} + ${success ? 1 : 0}`, 177 successes: sql`${domainProfiles.successes} + ${success ? 1 : 0}`,
173 blocks: sql`${domainProfiles.blocks} + ${blocks}`, 178 blocks: sql`${domainProfiles.blocks} + ${blocks}`,
174 captchas: sql`${domainProfiles.captchas} + ${captchas}`, 179 captchas: sql`${domainProfiles.captchas} + ${captchas}`,
180+ browserRequired: sql`${domainProfiles.browserRequired} + ${browserInc}`,
175 avgLatencyMs: avgLatency, 181 avgLatencyMs: avgLatency,
176 routeStats: stats, 182 routeStats: stats,
177 lastSeenAt: new Date(), 183 lastSeenAt: new Date(),
modified apps/api/src/services/pricing.ts +5 −0
@@ -27,6 +27,11 @@ export interface PriceBreakdown {
27 27
28 export function priceRequest(input: PriceInput): PriceBreakdown { 28 export function priceRequest(input: PriceInput): PriceBreakdown {
29 const limits = PLAN_LIMITS[input.plan]; 29 const limits = PLAN_LIMITS[input.plan];
30+ // Private platform: the single plan has no unit prices → nothing is billed. Upstream cost is
31+ // still tracked for the admin's unit economics.
32+ if (limits.price_usd_month === 0 && limits.overage_per_1k_requests_usd === 0 && limits.residential_per_gb_usd === 0) {
33+ return { requestUsd: 0, bandwidthUsd: 0, totalUsd: 0, marginUsd: round6(-input.upstreamCostUsd) };
34+ }
30 // Free/enterprise have no per-request rate; failed requests are never charged a request fee. 35 // Free/enterprise have no per-request rate; failed requests are never charged a request fee.
31 const requestUsd = input.success ? limits.overage_per_1k_requests_usd / 1000 : 0; 36 const requestUsd = input.success ? limits.overage_per_1k_requests_usd / 1000 : 0;
32 let bandwidthUsd = 0; 37 let bandwidthUsd = 0;
modified apps/api/src/services/retention.ts +10 −0
@@ -19,6 +19,16 @@ export async function pruneRetention(log: { info: (m: string) => void } = consol
19 `); 19 `);
20 total += Number((res as { rowCount?: number }).rowCount ?? 0); 20 total += Number((res as { rowCount?: number }).rowCount ?? 0);
21 } 21 }
22+ for (const plan of PLANS) {
23+ const days = PLAN_LIMITS[plan].retention_days;
24+ await db.execute(sql`
25+ delete from crawl_jobs j
26+ using organizations o
27+ where j.organization_id = o.id
28+ and o.plan = ${plan}
29+ and j.created_at < now() - make_interval(days => ${days})
30+ `);
31+ }
22 await db.execute(sql`update proxy_sessions set status = 'expired' where status = 'active' and expires_at < now()`); 32 await db.execute(sql`update proxy_sessions set status = 'expired' where status = 'active' and expires_at < now()`);
23 await db.execute(sql`delete from provider_health where checked_at < now() - interval '30 days'`); 33 await db.execute(sql`delete from provider_health where checked_at < now() - interval '30 days'`);
24 await db.execute(sql`delete from verifications where expires_at < now() - interval '1 day'`); 34 await db.execute(sql`delete from verifications where expires_at < now() - interval '1 day'`);
added apps/web/src/actions/access.ts +148 −0
@@ -0,0 +1,148 @@
1+"use server";
2+
3+import { revalidatePath } from "next/cache";
4+import { z } from "zod";
5+import { getDb, signupAllowlist, eq, inArray } from "@fetcha/db";
6+import { getEmailService } from "@fetcha/email";
7+import { requireAdmin, requestMeta } from "@/lib/session";
8+import { writeAudit } from "@/lib/audit";
9+import { SITE_URL } from "@/lib/utils";
10+import type { AuthUser } from "@/lib/auth";
11+import type { AdminActionResult } from "@/actions/admin";
12+
13+/**
14+ * Signup allowlist management (invitation-only platform). Every mutation is audited as
15+ * `admin.action` with metadata.type in { access.allow, access.invite, access.revoke }.
16+ */
17+
18+async function audit(admin: AuthUser, type: "access.allow" | "access.invite" | "access.revoke", target: string, metadata: Record<string, unknown> = {}) {
19+ const meta = await requestMeta();
20+ await writeAudit({ userId: admin.id, action: "admin.action", target, metadata: { type, ...metadata }, ipAddress: meta.ip, userAgent: meta.userAgent });
21+}
22+
23+function fail(e: unknown): { ok: false; error: string } {
24+ if (e instanceof z.ZodError) return { ok: false, error: e.issues[0]?.message ?? "Invalid input" };
25+ return { ok: false, error: (e as Error)?.message ?? "Unexpected error" };
26+}
27+
28+function revalidate() {
29+ revalidatePath("/admin/access");
30+}
31+
32+const emailSchema = z.string().trim().toLowerCase().email("Invalid email address").max(254);
33+
34+/** Split a comma / newline / whitespace separated list into unique, lower-cased, valid emails. */
35+function parseEmailList(raw: string): { emails: string[]; invalid: string[] } {
36+ const seen = new Set<string>();
37+ const emails: string[] = [];
38+ const invalid: string[] = [];
39+ for (const token of raw.split(/[\s,;]+/)) {
40+ const t = token.trim();
41+ if (!t) continue;
42+ const r = emailSchema.safeParse(t);
43+ if (!r.success) {
44+ invalid.push(t);
45+ continue;
46+ }
47+ if (seen.has(r.data)) continue;
48+ seen.add(r.data);
49+ emails.push(r.data);
50+ }
51+ return { emails, invalid };
52+}
53+
54+function signupUrlFor(email: string): string {
55+ return `${SITE_URL}/signup?email=${encodeURIComponent(email)}`;
56+}
57+
58+const allowSchema = z.object({
59+ emails: z.string().trim().min(3, "Enter at least one email address").max(20_000),
60+ note: z.string().trim().max(280, "Note is too long (280 characters max)").optional().or(z.literal("")),
61+ sendInvite: z.boolean().default(false),
62+});
63+
64+export type AllowEmailsInput = z.input<typeof allowSchema>;
65+export type AllowEmailsData = { added: number; existing: number; invited: number; invalid: string[]; failed: string[] };
66+
67+export async function allowEmails(input: AllowEmailsInput): Promise<AdminActionResult<AllowEmailsData>> {
68+ const admin = await requireAdmin();
69+ try {
70+ const d = allowSchema.parse(input);
71+ const { emails, invalid } = parseEmailList(d.emails);
72+ if (emails.length === 0) return { ok: false, error: invalid.length ? `No valid email address found (rejected: ${invalid.slice(0, 3).join(", ")}${invalid.length > 3 ? "…" : ""}).` : "Enter at least one email address." };
73+ if (emails.length > 200) return { ok: false, error: "Add at most 200 addresses at a time." };
74+
75+ const db = getDb();
76+ const note = d.note?.trim() || null;
77+ let invited = 0;
78+ const failed: string[] = [];
79+
80+ const already = new Set((await db.select({ email: signupAllowlist.email }).from(signupAllowlist).where(inArray(signupAllowlist.email, emails))).map((r) => r.email));
81+ const fresh = emails.filter((e) => !already.has(e));
82+ if (fresh.length) await db.insert(signupAllowlist).values(fresh.map((email) => ({ email, note, invitedByUserId: admin.id }))).onConflictDoNothing();
83+ // Existing entries keep their inviter; only refresh the note when one was provided.
84+ if (note && already.size) await db.update(signupAllowlist).set({ note }).where(inArray(signupAllowlist.email, Array.from(already)));
85+ const added = fresh.length;
86+ const existing = already.size;
87+ await audit(admin, "access.allow", emails.length === 1 ? emails[0]! : `${emails.length} emails`, { emails, note, added, existing });
88+
89+ if (d.sendInvite) {
90+ const inviterName = admin.name?.trim() || admin.email;
91+ for (const email of emails) {
92+ try {
93+ await getEmailService().sendInvite(email, { inviterName, signupUrl: signupUrlFor(email) });
94+ await db.update(signupAllowlist).set({ invitedAt: new Date() }).where(eq(signupAllowlist.email, email));
95+ invited++;
96+ } catch (e) {
97+ failed.push(email);
98+ console.error("[access] invite failed", email, (e as Error).message);
99+ }
100+ }
101+ await audit(admin, "access.invite", emails.length === 1 ? emails[0]! : `${emails.length} emails`, { emails, invited, failed });
102+ }
103+
104+ revalidate();
105+ const warnings: string[] = [];
106+ if (invalid.length) warnings.push(`${invalid.length} entr${invalid.length === 1 ? "y was" : "ies were"} not a valid email and ${invalid.length === 1 ? "was" : "were"} skipped: ${invalid.slice(0, 5).join(", ")}${invalid.length > 5 ? "…" : ""}`);
107+ if (failed.length) warnings.push(`Invitation email could not be sent to ${failed.join(", ")}. Use “Resend invite” later.`);
108+ return { ok: true, data: { added, existing, invited, invalid, failed }, warning: warnings.length ? warnings.join(" ") : undefined };
109+ } catch (e) {
110+ return fail(e);
111+ }
112+}
113+
114+export async function resendInvite(email: string): Promise<AdminActionResult> {
115+ const admin = await requireAdmin();
116+ try {
117+ const target = emailSchema.parse(email);
118+ const db = getDb();
119+ const [row] = await db.select().from(signupAllowlist).where(eq(signupAllowlist.email, target)).limit(1);
120+ if (!row) return { ok: false, error: "This address is not on the access list." };
121+ if (row.userId) return { ok: false, error: "This address already has an account; there is nothing to invite." };
122+ const inviterName = admin.name?.trim() || admin.email;
123+ await getEmailService().sendInvite(target, { inviterName, signupUrl: signupUrlFor(target) });
124+ await db.update(signupAllowlist).set({ invitedAt: new Date() }).where(eq(signupAllowlist.email, target));
125+ await audit(admin, "access.invite", target, { emails: [target], invited: 1, resend: true });
126+ revalidate();
127+ return { ok: true };
128+ } catch (e) {
129+ return fail(e);
130+ }
131+}
132+
133+export async function revokeAllow(email: string): Promise<AdminActionResult> {
134+ const admin = await requireAdmin();
135+ try {
136+ const target = emailSchema.parse(email);
137+ const db = getDb();
138+ const [row] = await db.select({ email: signupAllowlist.email, userId: signupAllowlist.userId, note: signupAllowlist.note }).from(signupAllowlist).where(eq(signupAllowlist.email, target)).limit(1);
139+ if (!row) return { ok: false, error: "This address is not on the access list." };
140+ if (row.userId) return { ok: false, error: "This address already has an account; ban the user instead." };
141+ await db.delete(signupAllowlist).where(eq(signupAllowlist.email, target));
142+ await audit(admin, "access.revoke", target, { note: row.note });
143+ revalidate();
144+ return { ok: true };
145+ } catch (e) {
146+ return fail(e);
147+ }
148+}
modified apps/web/src/actions/admin.ts +8 −13
@@ -118,20 +118,15 @@ export async function forceVerifyEmail(userId: string): Promise<AdminActionResul
118 // --------------------------------------------------------------------------- 118 // ---------------------------------------------------------------------------
119 // Organizations 119 // Organizations
120 // --------------------------------------------------------------------------- 120 // ---------------------------------------------------------------------------
121+/**
122+ * Single-plan platform: kept for API compatibility with older UI code, but plans can no longer be
123+ * changed. Every organization is `unlimited` (see `normalizePlan` in @fetcha/core).
124+ */
121 export async function updateOrganizationPlan(orgId: string, plan: string): Promise<AdminActionResult> { 125 export async function updateOrganizationPlan(orgId: string, plan: string): Promise<AdminActionResult> {
122− const admin = await requireAdmin(); 126+ await requireAdmin();
123− try { 127+ void orgId;
124− const p = z.enum(PLANS).parse(plan); 128+ void plan;
125− const db = getDb(); 129+ return { ok: false, error: "Plans are not configurable on this platform" };
126− const [o] = await db.select({ id: organizations.id, plan: organizations.plan }).from(organizations).where(eq(organizations.id, orgId)).limit(1);
127− if (!o) return { ok: false, error: "Organization not found" };
128− await db.update(organizations).set({ plan: p, updatedAt: new Date() }).where(eq(organizations.id, orgId));
129− await audit(admin, "org.plan", orgId, { from: o.plan, to: p }, orgId);
130− revalidateAdmin();
131− return { ok: true };
132− } catch (e) {
133− return fail(e);
134− }
135 } 130 }
136 131
137 export async function setOrganizationProviderVisibility(orgId: string, enabled: boolean): Promise<AdminActionResult> { 132 export async function setOrganizationProviderVisibility(orgId: string, enabled: boolean): Promise<AdminActionResult> {
added apps/web/src/actions/crawls.ts +87 −0
@@ -0,0 +1,87 @@
1+"use server";
2+
3+import { revalidatePath } from "next/cache";
4+import { crawlCreateSchema, mapCreateSchema } from "@fetcha/core";
5+import { getWorkspace } from "@/lib/session";
6+import { internalApi, InternalApiError, type CrawlJob, type MapResult } from "@/lib/api";
7+
8+export interface CrawlActionError {
9+ code: string;
10+ message: string;
11+ requestId: string | null;
12+ details?: Record<string, unknown>;
13+}
14+
15+export type CrawlActionResult<T> = { ok: true; data: T } | { ok: false; error: CrawlActionError };
16+
17+function invalid(issues: Array<{ path: string; message: string }>): CrawlActionError {
18+ const first = issues[0];
19+ return {
20+ code: "INVALID_REQUEST",
21+ message: first ? `${first.path ? `${first.path}: ` : ""}${first.message}` : "The request is invalid.",
22+ requestId: null,
23+ details: { issues },
24+ };
25+}
26+
27+function fromError(e: unknown, fallback: string): CrawlActionError {
28+ if (e instanceof InternalApiError) return { code: e.code, message: e.message, requestId: e.requestId, details: e.details };
29+ console.error("[crawls] unexpected error", e);
30+ return { code: "INTERNAL_ERROR", message: fallback, requestId: null };
31+}
32+
33+/** Strip empty strings / undefined so optional fields do not trip the strict schema. */
34+function clean(input: unknown): Record<string, unknown> {
35+ if (!input || typeof input !== "object") return {};
36+ return Object.fromEntries(Object.entries(input as Record<string, unknown>).filter(([, v]) => v !== undefined && v !== "" && v !== null));
37+}
38+
39+/**
40+ * Start a crawl job for the current project. Validated with the shared `crawlCreateSchema`; the API
41+ * enforces plan limits (pages per job, concurrent jobs). Never throws to the client.
42+ */
43+export async function createCrawl(input: unknown): Promise<CrawlActionResult<CrawlJob>> {
44+ const ws = await getWorkspace();
45+ try {
46+ const parsed = crawlCreateSchema.safeParse(clean(input));
47+ if (!parsed.success) return { ok: false, error: invalid(parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message }))) };
48+ const job = await internalApi.createCrawl(ws.project.id, ws.user.id, parsed.data);
49+ if (!job || typeof job !== "object" || typeof job.id !== "string") {
50+ return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };
51+ }
52+ revalidatePath("/dashboard/crawls");
53+ return { ok: true, data: job };
54+ } catch (e) {
55+ return { ok: false, error: fromError(e, "Could not start the crawl. Please try again.") };
56+ }
57+}
58+
59+/** Cancel a queued or running crawl job. */
60+export async function cancelCrawl(id: string): Promise<CrawlActionResult<{ id: string; status: "cancelled" }>> {
61+ const ws = await getWorkspace();
62+ if (!/^crawl_[A-Za-z0-9]{4,64}$/.test(id)) return { ok: false, error: { code: "INVALID_REQUEST", message: "Invalid crawl id.", requestId: null } };
63+ try {
64+ const res = await internalApi.cancelCrawl(ws.project.id, ws.user.id, id);
65+ revalidatePath("/dashboard/crawls");
66+ revalidatePath(`/dashboard/crawls/${id}`);
67+ return { ok: true, data: res };
68+ } catch (e) {
69+ return { ok: false, error: fromError(e, "Could not cancel the crawl. Please try again.") };
70+ }
71+}
72+
73+/** Discover the URLs of a site (sitemap + links), synchronously. */
74+export async function runMap(input: unknown): Promise<CrawlActionResult<MapResult>> {
75+ const ws = await getWorkspace();
76+ try {
77+ const parsed = mapCreateSchema.safeParse(clean(input));
78+ if (!parsed.success) return { ok: false, error: invalid(parsed.error.issues.map((i) => ({ path: i.path.join("."), message: i.message }))) };
79+ const result = await internalApi.mapSite(ws.project.id, ws.user.id, parsed.data);
80+ if (!result || typeof result !== "object" || !Array.isArray(result.urls)) {
81+ return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };
82+ }
83+ return { ok: true, data: result };
84+ } catch (e) {
85+ return { ok: false, error: fromError(e, "Could not map the site. Please try again.") };
86+ }
87+}
modified apps/web/src/actions/playground.ts +2 −7
@@ -1,6 +1,6 @@
1 "use server"; 1 "use server";
2 2
3−import { PLAN_LIMITS, fetchRequestSchema, type FetchResponseBody, type Plan } from "@fetcha/core"; 3+import { PLAN_LIMITS, fetchRequestSchema, normalizePlan, type FetchResponseBody } from "@fetcha/core";
4 import { getWorkspace } from "@/lib/session"; 4 import { getWorkspace } from "@/lib/session";
5 import { internalApi, InternalApiError } from "@/lib/api"; 5 import { internalApi, InternalApiError } from "@/lib/api";
6 6
@@ -39,8 +39,7 @@ export async function runPlayground(input: unknown): Promise<PlaygroundResult> {
39 } 39 }
40 40
41 const request = parsed.data; 41 const request = parsed.data;
42− const plan = (ws.organization.plan as Plan) in PLAN_LIMITS ? (ws.organization.plan as Plan) : "free"; 42+ const limits = PLAN_LIMITS[normalizePlan(ws.organization.plan)];
43− const limits = PLAN_LIMITS[plan];
44 if (request.timeout > limits.max_timeout_ms) request.timeout = limits.max_timeout_ms; 43 if (request.timeout > limits.max_timeout_ms) request.timeout = limits.max_timeout_ms;
45 if (request.retries !== undefined && request.retries > limits.max_retries) request.retries = limits.max_retries; 44 if (request.retries !== undefined && request.retries > limits.max_retries) request.retries = limits.max_retries;
46 if (request.network !== "auto" && !limits.networks.includes(request.network)) { 45 if (request.network !== "auto" && !limits.networks.includes(request.network)) {
@@ -49,10 +48,6 @@ export async function runPlayground(input: unknown): Promise<PlaygroundResult> {
49 error: { code: "NETWORK_UNAVAILABLE", message: `The "${request.network}" network is not included in the ${limits.label} plan.`, requestId: null }, 48 error: { code: "NETWORK_UNAVAILABLE", message: `The "${request.network}" network is not included in the ${limits.label} plan.`, requestId: null },
50 }; 49 };
51 } 50 }
52− if (request.browser) {
53− return { ok: false, error: { code: "BROWSER_UNAVAILABLE", message: "Managed browser execution is not yet available.", requestId: null } };
54− }
55−
56 const data = (await internalApi.playgroundFetch(ws.project.id, ws.user.id, request)) as FetchResponseBody; 51 const data = (await internalApi.playgroundFetch(ws.project.id, ws.user.id, request)) as FetchResponseBody;
57 if (!data || typeof data !== "object" || typeof data.request_id !== "string") { 52 if (!data || typeof data !== "object" || typeof data.request_id !== "string") {
58 return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } }; 53 return { ok: false, error: { code: "INTERNAL_ERROR", message: "The API returned an unexpected response.", requestId: null } };
modified apps/web/src/app/(auth)/login/login-form.tsx +3 −3
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(auth)/signup/page.tsx +4 −2
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(auth)/signup/signup-form.tsx +33 −8
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/changelog/page.tsx +23 −2
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/authentication/page.tsx +11 −11
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/browser/page.tsx +189 −45
Large diff (280 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/(marketing)/docs/crawl/page.tsx +379 −0
Large diff (379 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/errors/page.tsx +6 −4
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/fetch/page.tsx +85 −21
Large diff (206 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/networks/page.tsx +30 −33
Large diff (123 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/openapi.json/route.ts +277 −20
Large diff (398 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/page.tsx +15 −11
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/rate-limits/page.tsx +83 −74
Large diff (274 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/docs/retries/page.tsx +33 −26
Large diff (102 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/page.tsx +30 −25
Large diff (134 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/(marketing)/pricing/page.tsx +127 −249
Large diff (431 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/admin/access/loading.tsx +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/admin/access/page.tsx +159 −0
Large diff (159 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/admin/billing/page.tsx +4 −4
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/admin/organizations/[id]/page.tsx +6 −3
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/dashboard/billing/page.tsx +86 −143
Large diff (282 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/dashboard/crawls/[id]/loading.tsx +28 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/dashboard/crawls/[id]/page.tsx +344 −0
Large diff (344 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/dashboard/crawls/loading.tsx +31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/dashboard/crawls/page.tsx +179 −0
Large diff (179 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/dashboard/page.tsx +5 −6
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/dashboard/playground/page.tsx +4 −5
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/dashboard/projects/[id]/page.tsx +3 −3
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/dashboard/usage/page.tsx +18 −17
Large diff (94 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/admin/access-controls.tsx +149 −0
Large diff (149 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/admin/admin-shell.tsx +2 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/admin/org-actions.tsx +1 −36
Large diff (50 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/admin/primitives.tsx +3 −2
Large diff (13 lines) is not loaded by default to keep the page fast.Load diff
deleted apps/web/src/components/dashboard/billing/upgrade-dialog.tsx +0 −39
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/dashboard/crawls/cancel-crawl-button.tsx +33 −0
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/dashboard/crawls/crawl-status-badge.tsx +39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/dashboard/crawls/create-crawl-dialog.tsx +201 −0
Large diff (201 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/dashboard/crawls/map-tool.tsx +99 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/dashboard/sidebar.tsx +2 −1
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/docs/nav-config.ts +2 −1
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/marketing/architecture-diagram.tsx +2 −2
Large diff (16 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/marketing/footer.tsx +1 −1
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/marketing/nav.tsx +5 −5
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/playground/request-builder.tsx +102 −11
Large diff (164 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/playground/results-panel.tsx +124 −3
Large diff (208 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/playground/types.ts +54 −1
Large diff (96 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/lib/access.ts +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/lib/api.ts +88 −0
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/lib/auth.ts +22 −1
Large diff (43 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/lib/codegen.test.ts +56 −0
Large diff (68 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/lib/codegen.ts +22 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/lib/queries/account.ts +13 −3
Large diff (33 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/lib/queries/admin.ts +73 −4
Large diff (116 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/lib/session.ts +2 −6
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
modified deploy/README.md +2 −1
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added docs/API-v0.2.md +124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added packages/browser/package.json +26 −0
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added packages/browser/src/index.ts +550 −0
Large diff (550 lines) is not loaded by default to keep the page fast.Load diff
added packages/browser/src/stealth.ts +126 −0
Large diff (126 lines) is not loaded by default to keep the page fast.Load diff
added packages/browser/tsconfig.json +8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified packages/core/src/client.ts +1 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
modified packages/core/src/errors.ts +10 −4
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
modified packages/core/src/ids.ts +3 −1
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
modified packages/core/src/index.ts +2 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added packages/core/src/markdown.ts +687 −0
Large diff (687 lines) is not loaded by default to keep the page fast.Load diff
added packages/core/src/robots.ts +101 −0
Large diff (101 lines) is not loaded by default to keep the page fast.Load diff
modified packages/core/src/schema.ts +150 −64
Large diff (273 lines) is not loaded by default to keep the page fast.Load diff
modified packages/core/src/text.ts +180 −13
Large diff (228 lines) is not loaded by default to keep the page fast.Load diff
added packages/core/test/markdown.test.ts +80 −0
Large diff (80 lines) is not loaded by default to keep the page fast.Load diff
modified packages/core/test/ssrf.test.ts +7 −1
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
added packages/db/drizzle/0001_furry_red_skull.sql +76 −0
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
added packages/db/drizzle/meta/0001_snapshot.json +3572 −0
Large diff (3,573 lines) is not loaded by default to keep the page fast.Load diff
modified packages/db/drizzle/meta/_journal.json +7 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
modified packages/db/src/schema.ts +96 −1
Large diff (130 lines) is not loaded by default to keep the page fast.Load diff
modified packages/db/src/seed.ts +47 −3
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
modified packages/email/src/index.ts +6 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
modified packages/email/src/templates.tsx +26 −1
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added packages/providers/src/cookies.ts +129 −0
Large diff (129 lines) is not loaded by default to keep the page fast.Load diff
modified packages/providers/src/decodo.ts +6 −2
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
modified packages/providers/src/direct.ts +4 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added packages/providers/src/fingerprint.ts +369 −0
Large diff (369 lines) is not loaded by default to keep the page fast.Load diff
modified packages/providers/src/http.ts +171 −82
Large diff (367 lines) is not loaded by default to keep the page fast.Load diff
modified packages/providers/src/index.ts +2 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified packages/providers/src/oxylabs.ts +6 −2
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
modified packages/providers/src/soax.ts +6 −2
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
modified packages/providers/src/types.ts +26 −1
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
modified packages/routing/package.json +1 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
modified packages/routing/src/engine.ts +3 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
modified packages/routing/src/executor.ts +230 −61
Large diff (514 lines) is not loaded by default to keep the page fast.Load diff
modified packages/routing/test/engine.test.ts +7 −7
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
modified packages/sdk/package.json +1 −1
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
modified packages/sdk/src/index.ts +274 −9
Large diff (379 lines) is not loaded by default to keep the page fast.Load diff
modified pnpm-lock.yaml +54 −0
Large diff (96 lines) is not loaded by default to keep the page fast.Load diff
modified scripts/dev-seed.ts +7 −4
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
modified sdk-python/README.md +28 −4
Large diff (51 lines) is not loaded by default to keep the page fast.Load diff
modified sdk-python/fetcha/__init__.py +266 −27
Large diff (390 lines) is not loaded by default to keep the page fast.Load diff
added sdk-python/fetcha/__pycache__/__init__.cpython-314.pyc +0 −0

Binary file not shown.

modified sdk-python/pyproject.toml +2 −2
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere