v0.2.1 — 2captcha Turnstile solver (onload-hook interception, UA adoption via CDP), solve_captcha option, real Chrome channel + headful mode, tighter 2xx block signatures, Patchright main-world evaluates · ba9ab18 · fetcha

v0.2.1 — 2captcha Turnstile solver (onload-hook interception, UA adoption via CDP), solve_captcha option, real Chrome channel + headful mode, tighter 2xx block signatures, Patchright main-world evaluates

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 16 days ago (Sep 8, 2026) parent 23fd257

14 changed files +457 −27

modified .env.example +4 −1

		`@@ -37,9 +37,12 @@ ADMIN_EMAILS=you@example.com`
37	37	# Managed Chromium (Playwright). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
38	38	FETCHA_BROWSER_ENABLED=1
39	39	FETCHA_BROWSER_CONCURRENCY=6
40		−# "chromium" = full Chromium build in new headless mode (best anti-bot realism); "chrome" uses an installed Google Chrome.
	40	+# "chromium" = full Chromium build (Patchright); "chrome" uses an installed Google Chrome (prod). FETCHA_BROWSER_HEADLESS=0 = real off-screen window (needs a GUI session).
41	41	FETCHA_BROWSER_CHANNEL=chromium
	42	+FETCHA_BROWSER_HEADLESS=1
42	43	# Prefer HTTP/2 towards origins (set 0 to force HTTP/1.1).
43	44	FETCHA_HTTP2=1
44	45	# Crawl jobs executed in parallel by this API process.
45	46	FETCHA_CRAWL_PARALLEL_JOBS=4
	47	+# Captcha solver (2captcha.com) for Cloudflare Turnstile challenges in browser mode. Empty = disabled.
	48	+TWOCAPTCHA_API_KEY=

modified CLAUDE.md +2 −2

		@@ -13,7 +13,7 @@ and a full dashboard. Private platform (v0.2, 2026-09-08): a single `unlimit
13	13	- `packages/db` — Drizzle schema + migrations (`drizzle/`), `pnpm db:generate\|migrate\|seed`.
14	14	- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct), `http.ts` (undici, HTTP/2, browser-ordered headers, Chrome/Firefox/Safari TLS cipher lists, h2→h1 fallback), `fingerprint.ts` (header profiles), `cookies.ts` (jar across hops / sessions). Provider code lives ONLY here.
15	15	- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, fingerprint rotation, jittered backoff, block detection, HTTP → browser escalation).
16		−- `packages/browser` — `BrowserPool` (Patchright = patched Playwright, full Chromium new-headless, per-context upstream proxy, stealth init script, challenge wait + Turnstile click). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
	16	+- `packages/browser` — `BrowserPool` (Patchright = patched Playwright, Chrome/Chromium, per-context upstream proxy, stealth init script, challenge wait, 2captcha Turnstile solver in `captcha.ts`). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
17	17	- `packages/core/src/markdown.ts` — HTML → Markdown / main text, page metadata + links, URL normalisation, glob/regex matchers; `robots.ts` — robots.txt + sitemap parsing.
18	18	- `apps/api/src/services/crawl.ts` — durable crawl jobs (`crawl_jobs`/`crawl_pages`, in-process worker, resume after restart) and the sync `/v1/map`.
19	19	- `packages/email` — Resend abstraction + React Email templates. `packages/sdk` (JS), `sdk-python/` (Python).
		@@ -26,7 +26,7 @@ and a full dashboard. Private platform (v0.2, 2026-09-08): a single `unlimit
26	26	- API keys: shown once, SHA-256 stored. Request IDs `req_…` on every response (`X-Fetcha-Request-ID`).
27	27	- Not implemented yet (say so, don't fake): scripted browser actions (`POST /v1/browser`), `/v1/extract`, teams, webhook delivery (except the crawl `webhook_url` callback), OAuth, 2FA. Billing/Stripe is intentionally absent (private platform).
28	28	- Plans: always `normalizePlan(org.plan)`; never reintroduce tiers. Signup must stay allowlist-gated (Better Auth `user.create.before` hook in `apps/web/src/lib/auth.ts`).
29		−- Anti-bot honesty: interactive Cloudflare Turnstile challenges are NOT reliably solved (tested 2026-09-08 with Patchright + residential + headful); non-interactive JS challenges, DataDome/PX/Akamai soft blocks usually pass via residential + browser escalation.
	29	+- Anti-bot: interactive Cloudflare Turnstile challenges are solved with 2captcha (`TWOCAPTCHA_API_KEY`, `packages/browser/src/captcha.ts`): the init script wraps the api.js `?onload=` callback to capture `turnstile.render` params (sitekey, action, cData, chlPageData, callback), a token is bought (~$0.0015, 5–10 s) and injected through the callback after switching the page UA (CDP `Emulation.setUserAgentOverride`) to the solver's UA. Patchright runs init scripts in the main world but `page.evaluate` in an isolated world — pass `undefined, false` (4th arg) to read main-world state. `looksBlocked()` on 2xx must only match interstitial markers (vendor beacons like `challenge-platform/scripts/jsd`, `tags.js`, `ips.js` are on every protected page). Prod browser = Google Chrome stable (brew cask on M3U96a), `FETCHA_BROWSER_HEADLESS=0` (real off-screen window in the node's GUI session).
30	30
31	31	## Dev
32	32	`cp .env.example .env` (fill provider creds + RESEND_API_KEY), `createdb fetcha`, `pnpm db:migrate && pnpm db:seed`,

modified apps/web/src/app/(marketing)/changelog/page.tsx +12 −0

		`@@ -23,6 +23,18 @@ type Entry = {`
23	23	};
24	24
25	25	const ENTRIES: Entry[] = [
	26	+ {
	27	+ date: "2026-09-08",
	28	+ version: "v0.2.1",
	29	+ title: "Turnstile captcha solving, real Chrome",
	30	+ tag: "stable",
	31	+ summary: "The managed browser now solves interactive Cloudflare Turnstile challenges through a human-verification service and runs on a real Google Chrome build.",
	32	+ changes: [
	33	+ { kind: "added", text: "Captcha solver (2captcha) for Cloudflare Turnstile: widget parameters are intercepted in-page, a token is obtained and injected with the matching user agent. New request field solve_captcha (default true); attempts report captcha_solved in metadata.debug." },
	34	+ { kind: "changed", text: "Managed browser runs Google Chrome (stable channel) as a real window on the node instead of the headless Chromium build, with the native user agent and client hints kept coherent." },
	35	+ { kind: "changed", text: "Admin → System lists the anti-bot configuration (solver key presence, browser channel and mode); /internal/browser exposes solver statistics." },
	36	+ ],
	37	+ },
26	38	{
27	39	date: "2026-09-08",
28	40	version: "v0.2.0",

modified apps/web/src/app/(marketing)/docs/browser/page.tsx +8 −0

		`@@ -124,6 +124,14 @@ export default function BrowserPage() {`
124	124	</Li>
125	125	</Ul>
126	126
	127	+ <H2>Captcha solving</H2>
	128	+ <P>
	129	+ Most JavaScript challenges clear on their own inside the real browser. When Cloudflare asks for an interactive Turnstile verification instead, Fetcha intercepts the widget parameters,
	130	+ obtains a token from a managed human-verification service (2captcha) and hands it to the page, adopting the user agent the token was issued for. This typically adds 10–40 seconds, so keep
	131	+ <Code>timeout</Code> generous (60 s or more) for hard targets. Set <Code>{`solve_captcha: false`}</Code> to opt out on a request. Attempts that needed a token report{" "}
	132	+ <Code>captcha_solved: true</Code> in <Code>metadata.debug.attempts[]</Code>.
	133	+ </P>
	134	+
127	135	<H2>Automatic escalation</H2>
128	136	<P>
129	137	You rarely need to set <Code>browser: true</Code> yourself. With the default <Code>browser_fallback: true</Code>, Fetcha starts every request over plain HTTP because it is faster and cheaper.

modified apps/web/src/app/(marketing)/docs/fetch/page.tsx +1 −0

		`@@ -176,6 +176,7 @@ export default function FetchApiPage() {`
176	176	{ name: "javascript", type: "boolean", default: "true", description: <>Browser: set to <code>false</code> to render with scripting disabled.</> },
177	177	{ name: "block_resources", type: "boolean", default: "true", description: <>Browser: skip images, fonts and media to save bandwidth and time. Page scripts and XHR still run.</> },
178	178	{ name: "screenshot", type: "boolean", default: "false", description: <>Browser: return a PNG of the viewport, base64-encoded, in <code>screenshot</code>.</> },
	179	+ { name: "solve_captcha", type: "boolean", default: "true", description: <>Browser: when a Cloudflare Turnstile challenge blocks the render, obtain a token from the managed captcha solver and pass it to the page. Set <code>false</code> to never spend solver credits on a request. Attempts that needed a token report <code>captcha_solved: true</code> in <code>metadata.debug</code>.</> },
179	180	{ name: "cache", type: "{ enabled?: boolean, ttl?: integer }", reserved: true, constraints: "ttl 1–86,400 s", description: <>Reserved for response caching. Accepted, ignored; <code>metadata.cached</code> is always <code>false</code> today.</> },
180	181	]}
181	182	/>

modified apps/web/src/lib/queries/admin.ts +1 −0

		`@@ -1048,6 +1048,7 @@ export function getEnvPresence(): Array<{ group: string; key: string; present: b`
1048	1048	["Upstream providers", ["OXYLABS_USERNAME", "OXYLABS_PASSWORD", "DECODO_USERNAME", "DECODO_PASSWORD", "SOAX_USERNAME", "SOAX_PASSWORD"]],
1049	1049	["Core", ["DATABASE_URL", "REDIS_URL", "AUTH_SECRET", "INTERNAL_SERVICE_TOKEN", "API_URL", "NEXT_PUBLIC_SITE_URL", "ADMIN_EMAILS"]],
1050	1050	["Email", ["RESEND_API_KEY", "EMAIL_FROM", "EMAIL_FROM_TRANSACTIONAL"]],
	1051	+ ["Anti-bot", ["TWOCAPTCHA_API_KEY", "FETCHA_BROWSER_ENABLED", "FETCHA_BROWSER_CHANNEL", "FETCHA_BROWSER_HEADLESS"]],
1051	1052	["Billing", ["STRIPE_SECRET_KEY", "STRIPE_WEBHOOK_SECRET"]],
1052	1053	];
1053	1054	return spec.flatMap(([group, keys]) => keys.map((key) => ({ group, key, present: Boolean(process.env[key]?.trim()) })));

added packages/browser/src/captcha.ts +233 −0

		`@@ -0,0 +1,233 @@`
	1	+/**
	2	+ * Captcha solving via 2captcha (https://2captcha.com, JSON API v2).
	3	+ *
	4	+ * Used for Cloudflare Turnstile — both the standalone widget (`data-sitekey`) and the managed
	5	+ * challenge page ("Just a moment…"), where the `turnstile.render` call must be intercepted to grab
	6	+ * `sitekey`, `action`, `cData` and `chlPageData`, and the returned token handed to the widget's
	7	+ * callback. The solver also returns the User-Agent the token was produced with; the page must adopt
	8	+ * it before invoking the callback.
	9	+ */
	10	+export interface TurnstileParams {
	11	+ sitekey: string;
	12	+ action?: string \| null;
	13	+ cData?: string \| null;
	14	+ chlPageData?: string \| null;
	15	+ /** Whether a JS callback was captured (challenge page / explicit render). */
	16	+ hasCallback: boolean;
	17	+}
	18	+
	19	+export interface TurnstileSolution {
	20	+ token: string;
	21	+ userAgent: string \| null;
	22	+ taskId: string;
	23	+ ms: number;
	24	+ costUsd: number;
	25	+}
	26	+
	27	+export interface CaptchaSolverOptions {
	28	+ apiKey: string;
	29	+ endpoint?: string;
	30	+ /** Overall budget for a solve (default 110 s). */
	31	+ timeoutMs?: number;
	32	+ /** First poll delay / subsequent poll interval (2captcha recommends ~5 s then 3 s). */
	33	+ firstPollMs?: number;
	34	+ pollMs?: number;
	35	+ /** Price per Turnstile solve, for cost accounting (2captcha lists ~$1.45 / 1000). */
	36	+ pricePerSolveUsd?: number;
	37	+ fetch?: typeof globalThis.fetch;
	38	+ log?: { info: (m: string) => void; warn: (m: string) => void };
	39	+}
	40	+
	41	+export class CaptchaSolverError extends Error {
	42	+ constructor(
	43	+ readonly code: string,
	44	+ message: string,
	45	+ ) {
	46	+ super(message);
	47	+ this.name = "CaptchaSolverError";
	48	+ }
	49	+}
	50	+
	51	+interface TaskResult {
	52	+ errorId: number;
	53	+ errorCode?: string;
	54	+ errorDescription?: string;
	55	+ status?: "processing" \| "ready";
	56	+ solution?: { token?: string; userAgent?: string };
	57	+ taskId?: number \| string;
	58	+ balance?: number;
	59	+}
	60	+
	61	+export class TwoCaptchaSolver {
	62	+ readonly provider = "2captcha" as const;
	63	+ private readonly opts: Required<Omit<CaptchaSolverOptions, "log">> & { log: NonNullable<CaptchaSolverOptions["log"]> };
	64	+ private stats = { requested: 0, solved: 0, failed: 0, spentUsd: 0, lastError: null as string \| null };
	65	+
	66	+ constructor(opts: CaptchaSolverOptions) {
	67	+ if (!opts.apiKey) throw new Error("2captcha: apiKey is required");
	68	+ this.opts = {
	69	+ apiKey: opts.apiKey,
	70	+ endpoint: (opts.endpoint ?? "https://api.2captcha.com").replace(/\/$/, ""),
	71	+ timeoutMs: opts.timeoutMs ?? 110_000,
	72	+ firstPollMs: opts.firstPollMs ?? 5000,
	73	+ pollMs: opts.pollMs ?? 3000,
	74	+ pricePerSolveUsd: opts.pricePerSolveUsd ?? 0.00145,
	75	+ fetch: opts.fetch ?? globalThis.fetch.bind(globalThis),
	76	+ log: opts.log ?? { info: (m) => console.log("[2captcha]", m), warn: (m) => console.warn("[2captcha]", m) },
	77	+ };
	78	+ }
	79	+
	80	+ status() {
	81	+ return { provider: this.provider, ...this.stats };
	82	+ }
	83	+
	84	+ private async call(path: string, body: Record<string, unknown>, timeoutMs = 20_000): Promise<TaskResult> {
	85	+ const ac = new AbortController();
	86	+ const t = setTimeout(() => ac.abort(), timeoutMs);
	87	+ try {
	88	+ const res = await this.opts.fetch(`${this.opts.endpoint}${path}`, {
	89	+ method: "POST",
	90	+ headers: { "content-type": "application/json", "user-agent": "fetcha/0.2 (+https://www.fetcha.co)" },
	91	+ body: JSON.stringify({ clientKey: this.opts.apiKey, ...body }),
	92	+ signal: ac.signal,
	93	+ });
	94	+ const data = (await res.json().catch(() => ({}))) as TaskResult;
	95	+ if (!res.ok) throw new CaptchaSolverError("HTTP_ERROR", `2captcha HTTP ${res.status}`);
	96	+ return data;
	97	+ } finally {
	98	+ clearTimeout(t);
	99	+ }
	100	+ }
	101	+
	102	+ async balance(): Promise<number> {
	103	+ const r = await this.call("/getBalance", {});
	104	+ if (r.errorId) throw new CaptchaSolverError(r.errorCode ?? "ERROR", r.errorDescription ?? "2captcha error");
	105	+ return Number(r.balance ?? 0);
	106	+ }
	107	+
	108	+ /**
	109	+ * Solve a Turnstile challenge. `deadline` (performance.now() based) bounds the wait so that the
	110	+ * caller's request timeout is honoured.
	111	+ */
	112	+ async solveTurnstile(input: { websiteURL: string; params: TurnstileParams; deadline?: number }): Promise<TurnstileSolution> {
	113	+ const t0 = performance.now();
	114	+ const hardDeadline = Math.min(t0 + this.opts.timeoutMs, input.deadline ?? Infinity);
	115	+ this.stats.requested++;
	116	+ const task: Record<string, unknown> = { type: "TurnstileTaskProxyless", websiteURL: input.websiteURL, websiteKey: input.params.sitekey };
	117	+ if (input.params.action) task.action = input.params.action;
	118	+ if (input.params.cData) task.data = input.params.cData;
	119	+ if (input.params.chlPageData) task.pagedata = input.params.chlPageData;
	120	+ let created: TaskResult;
	121	+ try {
	122	+ created = await this.call("/createTask", { task });
	123	+ } catch (e) {
	124	+ this.fail((e as Error).message);
	125	+ throw e instanceof CaptchaSolverError ? e : new CaptchaSolverError("NETWORK", (e as Error).message);
	126	+ }
	127	+ if (created.errorId \|\| !created.taskId) {
	128	+ const code = created.errorCode ?? "CREATE_FAILED";
	129	+ this.fail(`${code}: ${created.errorDescription ?? ""}`);
	130	+ throw new CaptchaSolverError(code, created.errorDescription ?? "2captcha refused the task");
	131	+ }
	132	+ const taskId = String(created.taskId);
	133	+ let wait = this.opts.firstPollMs;
	134	+ for (;;) {
	135	+ const remaining = hardDeadline - performance.now();
	136	+ if (remaining <= 0) {
	137	+ this.fail("timeout");
	138	+ throw new CaptchaSolverError("TIMEOUT", "2captcha did not solve the challenge in time");
	139	+ }
	140	+ await new Promise((r) => setTimeout(r, Math.min(wait, remaining)));
	141	+ wait = this.opts.pollMs;
	142	+ let r: TaskResult;
	143	+ try {
	144	+ r = await this.call("/getTaskResult", { taskId: created.taskId });
	145	+ } catch (e) {
	146	+ // transient network error: keep polling
	147	+ this.opts.log.warn(`poll error: ${(e as Error).message}`);
	148	+ continue;
	149	+ }
	150	+ if (r.errorId) {
	151	+ const code = r.errorCode ?? "SOLVE_FAILED";
	152	+ this.fail(`${code}: ${r.errorDescription ?? ""}`);
	153	+ throw new CaptchaSolverError(code, r.errorDescription ?? "2captcha could not solve the challenge");
	154	+ }
	155	+ if (r.status === "ready" && r.solution?.token) {
	156	+ const ms = Math.round(performance.now() - t0);
	157	+ this.stats.solved++;
	158	+ this.stats.spentUsd += this.opts.pricePerSolveUsd;
	159	+ this.opts.log.info(`turnstile solved in ${ms} ms (task ${taskId})`);
	160	+ return { token: r.solution.token, userAgent: r.solution.userAgent ?? null, taskId, ms, costUsd: this.opts.pricePerSolveUsd };
	161	+ }
	162	+ }
	163	+ }
	164	+
	165	+ private fail(msg: string) {
	166	+ this.stats.failed++;
	167	+ this.stats.lastError = msg.slice(0, 200);
	168	+ this.opts.log.warn(`solve failed: ${msg}`);
	169	+ }
	170	+}
	171	+
	172	+/**
	173	+ * Init script: intercept `turnstile.render` (challenge pages and explicit widgets) to capture the
	174	+ * parameters 2captcha needs, and keep the callback so the token can be injected later. Stored on a
	175	+ * non-enumerable, obscurely named window property.
	176	+ */
	177	+export const TURNSTILE_HOOK_KEY = "__fx_ts_" + "9c1";
	178	+
	179	+export function turnstileHookScript(): string {
	180	+ return `(() => {
	181	+ const KEY = ${JSON.stringify(TURNSTILE_HOOK_KEY)};
	182	+ const store = { params: null, callback: null, rendered: 0, hooked: [] };
	183	+ try { Object.defineProperty(window, KEY, { value: store, enumerable: false, configurable: true, writable: false }); } catch { window[KEY] = store; }
	184	+ const patchTs = (ts) => {
	185	+ if (!ts \|\| typeof ts !== 'object' \|\| ts.__fx) return false;
	186	+ const origRender = ts.render;
	187	+ const patched = function (container, params) {
	188	+ try {
	189	+ const p = params \|\| {};
	190	+ store.rendered++;
	191	+ store.params = { sitekey: p.sitekey \|\| null, action: p.action \|\| null, cData: p.cData \|\| null, chlPageData: p.chlPageData \|\| null, hasCallback: typeof p.callback === 'function' };
	192	+ if (typeof p.callback === 'function') store.callback = p.callback;
	193	+ } catch {}
	194	+ // Challenge pages: do NOT render the real widget (its own attempt would consume the challenge
	195	+ // data we are about to solve out-of-band); return a fake widget id instead (2captcha's guidance).
	196	+ if (store.params && store.params.chlPageData) return 'fx-' + Math.random().toString(36).slice(2, 10);
	197	+ return typeof origRender === 'function' ? origRender.apply(this, arguments) : undefined;
	198	+ };
	199	+ try { Object.defineProperty(patched, 'toString', { value: () => 'function render() { [native code] }' }); } catch {}
	200	+ try { ts.render = patched; Object.defineProperty(ts, '__fx', { value: true, enumerable: false, configurable: true }); } catch { return false; }
	201	+ return true;
	202	+ };
	203	+ // 1) Challenge pages load api.js with ?onload=<name>&render=explicit and call turnstile.render from that
	204	+ // callback. Wrapping the callback catches the turnstile object at the exact moment it exists.
	205	+ const hookOnload = (name) => {
	206	+ if (!name \|\| store.hooked.includes(name)) return;
	207	+ store.hooked.push(name);
	208	+ let cur = window[name];
	209	+ const wrap = (fn) => function () {
	210	+ try { patchTs(arguments[0] && typeof arguments[0].render === 'function' ? arguments[0] : window.turnstile); } catch {}
	211	+ return fn.apply(this, arguments);
	212	+ };
	213	+ try {
	214	+ Object.defineProperty(window, name, { configurable: true, enumerable: true, get() { return cur; }, set(v) { cur = typeof v === 'function' ? wrap(v) : v; } });
	215	+ if (typeof cur === 'function') cur = wrap(cur);
	216	+ } catch {}
	217	+ };
	218	+ const onloadName = (src) => { const m = /[?&]onload=([A-Za-z0-9_$]+)/.exec(src \|\| ''); return m ? m[1] : null; };
	219	+ const scan = (root) => { try { for (const el of root.querySelectorAll('script[src*="turnstile"]')) hookOnload(onloadName(el.src)); } catch {} };
	220	+ try {
	221	+ new MutationObserver((muts) => {
	222	+ for (const m of muts) for (const n of m.addedNodes) {
	223	+ if (n.nodeName === 'SCRIPT' && n.src && /turnstile/.test(n.src)) hookOnload(onloadName(n.src));
	224	+ else if (n.querySelectorAll) scan(n);
	225	+ }
	226	+ }).observe(document, { childList: true, subtree: true });
	227	+ } catch {}
	228	+ document.addEventListener('DOMContentLoaded', () => scan(document));
	229	+ // 2) Standalone widgets / other pages: poll for window.turnstile (api.js sets it on load).
	230	+ let n = 0;
	231	+ const iv = setInterval(() => { if (patchTs(window.turnstile) \|\| ++n > 1200) clearInterval(iv); }, 25);
	232	+})();`;
	233	+}

modified packages/browser/src/index.ts +162 −13

		`@@ -14,6 +14,9 @@ import type { FingerprintProfile, ProxyEndpoint } from "@fetcha/providers";`
14	14	import { acceptLanguage } from "@fetcha/providers";
15	15	import type { Browser, BrowserContext, Page, Response as PwResponse } from "patchright";
16	16	import { stealthScript, timezoneFor, webglFor } from "./stealth";
	17	+import { TURNSTILE_HOOK_KEY, TwoCaptchaSolver, turnstileHookScript, type TurnstileParams, type TurnstileSolution } from "./captcha";
	18	+
	19	+export * from "./captcha";
17	20
18	21	export interface RenderRequest {
19	22	url: string;
		`@@ -34,10 +37,20 @@ export interface RenderRequest {`
34	37	blockResources: boolean;
35	38	screenshot: boolean;
36	39	maxResponseBytes: number;
	40	+ /** Use the configured captcha solver (2captcha) on Turnstile challenges (default true). */
	41	+ solveCaptcha?: boolean;
37	42	/** Called for every main-frame navigation to a new URL (redirects). Throw to abort. */
38	43	onRedirect?: (nextUrl: string) => Promise<void>;
39	44	}
40	45
	46	+export interface CaptchaOutcome {
	47	+ provider: "2captcha";
	48	+ /** Token obtained and the page passed afterwards. */
	49	+ solved: boolean;
	50	+ ms: number;
	51	+ costUsd: number;
	52	+}
	53	+
41	54	export interface RenderResult {
42	55	status: number;
43	56	headers: Record<string, string>;
		`@@ -54,6 +67,8 @@ export interface RenderResult {`
54	67	challengeSolved: boolean;
55	68	/** Block verdict on the captured DOM (after the challenge wait). */
56	69	block: ReturnType<typeof looksBlocked>;
	70	+ /** Captcha solver involvement, when a token was purchased. */
	71	+ captcha: CaptchaOutcome \| null;
57	72	}
58	73
59	74	export interface BrowserPoolOptions {
		`@@ -70,6 +85,8 @@ export interface BrowserPoolOptions {`
70	85	challengeWaitMs?: number;
71	86	/** Run a real (off-screen) window instead of headless mode. Requires a graphical session. */
72	87	headless?: boolean;
	88	+ /** 2captcha API key (default: env TWOCAPTCHA_API_KEY). Empty = no solver. */
	89	+ captchaApiKey?: string \| null;
73	90	log?: { info: (m: string) => void; warn: (m: string) => void };
74	91	}
75	92
		`@@ -84,6 +101,9 @@ export interface BrowserStatus {`
84	101	challengesSolved: number;
85	102	lastError: string \| null;
86	103	version: string \| null;
	104	+ headless: boolean;
	105	+ channel: string;
	106	+ captchaSolver: { provider: string; requested: number; solved: number; failed: number; spentUsd: number; lastError: string \| null } \| null;
87	107	}
88	108
89	109	const CHALLENGE_SELECTORS = ["#challenge-running", "#challenge-form", "#challenge-stage", ".cf-turnstile", "iframe[src='challenges.cloudflare.com']", "#px-captcha", "#datadome", "iframe[src='captcha-delivery.com']", "#sec-cpt-if", "form#challenge", "#cmsg", "[data-testid='challenge']"];
		`@@ -93,6 +113,9 @@ export class BrowserPool {`
93	113	private launching: Promise<Browser> \| null = null;
94	114	/** User agent reported by the real Chromium build (with "HeadlessChrome" normalised to "Chrome"). */
95	115	private nativeUa: string \| null = null;
	116	+ private readonly solver: TwoCaptchaSolver \| null;
	117	+ /** User-Agent of the last solver token: reused up-front so challenge pages see one consistent UA. */
	118	+ private lastSolverUa: string \| null = null;
96	119	private running = 0;
97	120	private queue: Array<() => void> = [];
98	121	private idleTimer: NodeJS.Timeout \| null = null;
		`@@ -108,14 +131,20 @@ export class BrowserPool {`
108	131	enabled: opts.enabled ?? process.env.FETCHA_BROWSER_ENABLED !== "0",
109	132	challengeWaitMs: opts.challengeWaitMs ?? 18_000,
110	133	headless: opts.headless ?? process.env.FETCHA_BROWSER_HEADLESS !== "0",
	134	+ captchaApiKey: opts.captchaApiKey === undefined ? process.env.TWOCAPTCHA_API_KEY?.trim() \|\| null : opts.captchaApiKey,
111	135	log: opts.log ?? { info: (m) => console.log("[browser]", m), warn: (m) => console.warn("[browser]", m) },
112	136	};
	137	+ this.solver = this.opts.captchaApiKey ? new TwoCaptchaSolver({ apiKey: this.opts.captchaApiKey, log: { info: (m) => this.opts.log.info(`2captcha: ${m}`), warn: (m) => this.opts.log.warn(`2captcha: ${m}`) } }) : null;
113	138	}
114	139
115	140	get enabled(): boolean {
116	141	return this.opts.enabled;
117	142	}
118	143
	144	+ get captchaSolver(): TwoCaptchaSolver \| null {
	145	+ return this.solver;
	146	+ }
	147	+
119	148	status(): BrowserStatus {
120	149	return {
121	150	enabled: this.opts.enabled,
		`@@ -128,6 +157,9 @@ export class BrowserPool {`
128	157	challengesSolved: this.stats.challengesSolved,
129	158	lastError: this.stats.lastError,
130	159	version: this.browser?.version() ?? null,
	160	+ headless: this.opts.headless,
	161	+ channel: this.opts.channel,
	162	+ captchaSolver: this.solver ? this.solver.status() : null,
131	163	};
132	164	}
133	165
		`@@ -287,6 +319,7 @@ export class BrowserPool {`
287	319	const webgl = webglFor(customUa \|\| isMobile ? profile.platform : /Windows/.test(userAgent) ? "Win32" : /Linux/.test(userAgent) ? "Linux x86_64" : "MacIntel", profile.tls);
288	320	const platform = customUa \|\| isMobile ? profile.platform : /Windows/.test(userAgent) ? "Win32" : /Linux/.test(userAgent) ? "Linux x86_64" : "MacIntel";
289	321	await context.addInitScript(stealthScript({ platform, languages, hardwareConcurrency: isMobile ? 8 : 12, deviceMemory: 8, vendor: webgl.vendor, renderer: webgl.renderer, mobile: isMobile }));
	322	+ if (this.solver && req.solveCaptcha !== false) await context.addInitScript(turnstileHookScript());
290	323	if (req.cookies?.length) {
291	324	await context
292	325	.addCookies(
		`@@ -336,7 +369,7 @@ export class BrowserPool {`
336	369	if (req.method !== "GET") {
337	370	const headers = { ...r.headers() };
338	371	if (req.body !== undefined && !headers["content-type"]) headers["content-type"] = typeof req.body === "string" ? "application/json" : "application/octet-stream";
339		− return route.continue({ method: req.method, postData: req.body, headers });
	372	+ return route.fallback({ method: req.method, postData: req.body, headers });
340	373	}
341	374	} else if (r.url() !== initialUrl.toString()) {
342	375	redirects++;
		`@@ -349,13 +382,14 @@ export class BrowserPool {`
349	382	}
350	383	}
351	384	}
352		− return route.continue();
	385	+ // fallback() (not continue()) lets Patchright's context-level route inject the init scripts.
	386	+ return route.fallback();
353	387	}
354	388	if (req.blockResources) {
355	389	const type = r.resourceType();
356	390	if (type === "image" \|\| type === "media" \|\| type === "font" \|\| type === "manifest" \|\| type === "texttrack") return route.abort("blockedbyclient");
357	391	}
358		− return route.continue();
	392	+ return route.fallback();
359	393	});
360	394	page.on("requestfinished", (r) => {
361	395	pendingSizes.push(
		`@@ -398,25 +432,27 @@ export class BrowserPool {`
398	432	if (policyError) throw policyError;
399	433	const navigationMs = Math.round(performance.now() - tNav0);
400	434
401		− // Challenge handling: give the page time to solve JS challenges (Cloudflare, DataDome, PX…).
	435	+ // Challenge handling: give the page time to solve JS challenges (Cloudflare, DataDome, PX…),
	436	+ // and buy a Turnstile token from the captcha solver when the challenge is interactive.
402	437	const tCh0 = performance.now();
403	438	let challengeSolved = false;
	439	+ let captcha: CaptchaOutcome \| null = null;
404	440	let status = response?.status() ?? 200;
405	441	let headers = lower(response?.headers() ?? {});
406	442	let html = await page.content().catch(() => "");
407	443	let verdict = looksBlocked(status, html, headers);
408	444	if (verdict.blocked && verdict.challenge !== false && req.javascript) {
409		− const budget = Math.min(this.opts.challengeWaitMs, deadline - performance.now() - 1500);
410	445	const challengeStatus = status;
411		− const until = performance.now() + Math.max(0, budget);
	446	+ const solverAllowed = Boolean(this.solver) && req.solveCaptcha !== false;
	447	+ const loopStart = performance.now();
	448	+ let until = loopStart + Math.max(0, Math.min(this.opts.challengeWaitMs, deadline - performance.now() - 1500));
412	449	let clickedTurnstile = false;
	450	+ let solveState: { started: boolean; done: boolean; sol: TurnstileSolution \| null } = { started: false, done: false, sol: null };
	451	+ let injected = false;
413	452	while (performance.now() < until) {
414		− await page.waitForTimeout(600).catch(() => {});
415		− // Best-effort click on a Turnstile / hCaptcha checkbox when one is visible.
416		− if (!clickedTurnstile) {
417		− clickedTurnstile = await this.tryClickChallenge(page);
418		− }
419		− const navResp = await page.waitForNavigation({ timeout: 1200, waitUntil: "domcontentloaded" }).catch(() => null);
	453	+ await page.waitForTimeout(500).catch(() => {});
	454	+ // Natural resolution: the challenge script redirects / replaces the document.
	455	+ const navResp = await page.waitForNavigation({ timeout: 900, waitUntil: "domcontentloaded" }).catch(() => null);
420	456	if (navResp) {
421	457	response = navResp;
422	458	status = navResp.status();
		`@@ -430,12 +466,43 @@ export class BrowserPool {`
430	466	if (!navResp) status = 200; // the document was replaced in place
431	467	break;
432	468	}
	469	+ // Captcha solver: once the widget parameters are visible, buy a token (in the background) and keep waiting.
	470	+ if (solverAllowed && !solveState.started && performance.now() - loopStart > 1200 && deadline - performance.now() > 15_000) {
	471	+ const params = await this.readTurnstileParams(page);
	472	+ if (params?.sitekey) {
	473	+ solveState.started = true;
	474	+ const websiteURL = page.url();
	475	+ this.solver!.solveTurnstile({ websiteURL, params, deadline: deadline - 4000 })
	476	+ .then((sol) => (solveState = { started: true, done: true, sol }))
	477	+ .catch(() => (solveState = { started: true, done: true, sol: null }));
	478	+ until = deadline - 2500; // wait for the solver instead of the short challenge budget
	479	+ }
	480	+ }
	481	+ if (solveState.done && !injected) {
	482	+ injected = true;
	483	+ const sol = solveState.sol;
	484	+ if (!sol) {
	485	+ until = Math.min(until, performance.now() + 2500);
	486	+ } else {
	487	+ captcha = { provider: "2captcha", solved: false, ms: sol.ms, costUsd: sol.costUsd };
	488	+ if (sol.userAgent && sol.userAgent !== userAgent) {
	489	+ this.lastSolverUa = sol.userAgent;
	490	+ await this.overrideUserAgent(context, page, sol.userAgent, extraHeaders["accept-language"]);
	491	+ }
	492	+ const how = await this.injectTurnstileToken(page, sol.token);
	493	+ this.opts.log.info(`turnstile token injected via ${how} on ${new URL(page.url()).hostname} (solver ua ${sol.userAgent ? (sol.userAgent === userAgent ? "same" : "override") : "none"})`);
	494	+ until = Math.min(deadline - 1500, performance.now() + 15_000);
	495	+ }
	496	+ }
	497	+ // Best-effort click on a Turnstile / hCaptcha checkbox when no solver is available.
	498	+ if (!solverAllowed && !clickedTurnstile) clickedTurnstile = await this.tryClickChallenge(page);
433	499	}
434	500	verdict = looksBlocked(status, html, headers);
435	501	if (challengeSolved) {
436	502	this.stats.challengesSolved++;
437		− headers = { ...headers, "x-fetcha-challenge": `solved:${challengeStatus}` };
	503	+ headers = { ...headers, "x-fetcha-challenge": `solved:${challengeStatus}${captcha ? ":2captcha" : ""}` };
438	504	verdict = { blocked: false };
	505	+ if (captcha) captcha.solved = true;
439	506	}
440	507	}
441	508	const challengeMs = Math.round(performance.now() - tCh0);
		`@@ -496,6 +563,7 @@ export class BrowserPool {`
496	563	screenshot,
497	564	challengeSolved,
498	565	block: finalVerdict,
	566	+ captcha,
499	567	};
500	568	} catch (e) {
501	569	this.stats.failures++;
		`@@ -507,6 +575,87 @@ export class BrowserPool {`
507	575	}
508	576	}
509	577
	578	+ /** Read the intercepted `turnstile.render` parameters, or fall back to a widget's data-* attributes. */
	579	+ private async readTurnstileParams(page: Page): Promise<TurnstileParams \| null> {
	580	+ try {
	581	+ return await page.evaluate((key) => {
	582	+ const w = window as unknown as Record<string, { params?: TurnstileParams \| null; callback?: unknown } \| undefined>;
	583	+ const store = w[key];
	584	+ if (store?.params?.sitekey) return { ...store.params, hasCallback: typeof store.callback === "function" } as TurnstileParams;
	585	+ const el = document.querySelector(".cf-turnstile[data-sitekey], [data-sitekey][data-callback], #cf-chl-widget-container [data-sitekey]") as HTMLElement \| null;
	586	+ if (el?.dataset.sitekey) return { sitekey: el.dataset.sitekey, action: el.dataset.action ?? null, cData: el.dataset.cdata ?? null, chlPageData: null, hasCallback: false } as TurnstileParams;
	587	+ return null;
	588	+ }, TURNSTILE_HOOK_KEY, undefined, false); // Patchright: 4th arg = isolatedContext → false = page's main world
	589	+ } catch {
	590	+ return null;
	591	+ }
	592	+ }
	593	+
	594	+ /** Hand the solved token to the widget's callback (challenge pages) or to the form field (standalone widgets). */
	595	+ private async injectTurnstileToken(page: Page, token: string): Promise<string> {
	596	+ try {
	597	+ return await page.evaluate(
	598	+ ({ key, token }) => {
	599	+ const w = window as unknown as Record<string, { callback?: (t: string) => void } \| undefined>;
	600	+ const store = w[key];
	601	+ if (store && typeof store.callback === "function") {
	602	+ store.callback(token);
	603	+ return "callback";
	604	+ }
	605	+ const input = document.querySelector('input[name="cf-turnstile-response"]') as HTMLInputElement \| null;
	606	+ if (input) {
	607	+ input.value = token;
	608	+ const form = input.closest("form");
	609	+ const cbName = (document.querySelector(".cf-turnstile[data-callback]") as HTMLElement \| null)?.dataset.callback;
	610	+ const cb = cbName ? (window as unknown as Record<string, unknown>)[cbName] : null;
	611	+ if (typeof cb === "function") (cb as (t: string) => void)(token);
	612	+ else if (form) (form as HTMLFormElement & { requestSubmit?: () => void }).requestSubmit ? (form as HTMLFormElement).requestSubmit() : form.submit();
	613	+ return "form";
	614	+ }
	615	+ return "none";
	616	+ },
	617	+ { key: TURNSTILE_HOOK_KEY, token },
	618	+ undefined,
	619	+ false, // main world: the callback captured by the init script lives there
	620	+ );
	621	+ } catch (e) {
	622	+ return `error:${(e as Error).message.split("\n")[0]}`;
	623	+ }
	624	+ }
	625	+
	626	+ /** Adopt the User-Agent the token was produced with (Cloudflare binds tokens to the UA). */
	627	+ private async overrideUserAgent(context: BrowserContext, page: Page, userAgent: string, acceptLanguage?: string): Promise<void> {
	628	+ try {
	629	+ const cdp = await context.newCDPSession(page);
	630	+ const major = /Chrome\/(\d+)/.exec(userAgent)?.[1] ?? "140";
	631	+ const platform = /Windows/.test(userAgent) ? "Windows" : /Macintosh/.test(userAgent) ? "macOS" : /Android/.test(userAgent) ? "Android" : "Linux";
	632	+ const brands = [
	633	+ { brand: "Chromium", version: major },
	634	+ { brand: /Edg\//.test(userAgent) ? "Microsoft Edge" : "Google Chrome", version: major },
	635	+ { brand: "Not_A Brand", version: "24" },
	636	+ ];
	637	+ await cdp.send("Emulation.setUserAgentOverride", {
	638	+ userAgent,
	639	+ acceptLanguage,
	640	+ platform: platform === "Windows" ? "Win32" : platform === "macOS" ? "MacIntel" : platform === "Android" ? "Linux armv81" : "Linux x86_64",
	641	+ userAgentMetadata: {
	642	+ brands,
	643	+ fullVersionList: brands.map((b) => ({ brand: b.brand, version: b.version === "24" ? "24.0.0.0" : `${b.version}.0.0.0` })),
	644	+ platform,
	645	+ platformVersion: platform === "Windows" ? "15.0.0" : platform === "macOS" ? "14.6.1" : platform === "Android" ? "15.0.0" : "6.8.0",
	646	+ architecture: platform === "Android" ? "" : "x86",
	647	+ model: "",
	648	+ mobile: /Mobile/.test(userAgent),
	649	+ bitness: "64",
	650	+ wow64: false,
	651	+ },
	652	+ });
	653	+ await cdp.detach().catch(() => {});
	654	+ } catch (e) {
	655	+ this.opts.log.warn(`user-agent override failed: ${(e as Error).message.split("\n")[0]}`);
	656	+ }
	657	+ }
	658	+
510	659	private async tryClickChallenge(page: Page): Promise<boolean> {
511	660	try {
512	661	for (const frame of page.frames()) {

modified packages/core/src/schema.ts +4 −0

		`@@ -48,6 +48,8 @@ export const fetchRequestSchema = z`
48	48	block_resources: z.boolean().default(true),
49	49	/** Browser: return a PNG screenshot (base64) in `screenshot`. */
50	50	screenshot: z.boolean().default(false),
	51	+ /** Browser: use the captcha solver on Cloudflare Turnstile challenges when the platform has one configured (default true). */
	52	+ solve_captcha: z.boolean().default(true),
51	53	/** Include the list of hyperlinks found in the page (`links`). */
52	54	links: z.boolean().default(false),
53	55	/** Referer strategy: "auto" (none on first try, search-engine referer on retries), "none", or a literal URL. */
		`@@ -134,6 +136,8 @@ export interface FetchMetadata {`
134	136	country: string \| null;
135	137	outcome: string;
136	138	block_reason?: string \| null;
	139	+ /** True when a captcha solver token was needed to pass this attempt. */
	140	+ captcha_solved?: boolean;
137	141	status: number \| null;
138	142	duration_ms: number;
139	143	error?: string;

modified packages/core/src/text.ts +13 −9

		`@@ -154,7 +154,7 @@ export function looksBlocked(status: number, body: string, headers: Record<strin`
154	154	if (h["x-datadome"] \|\| h["x-dd-b"] \|\| /datadome/i.test(h["set-cookie"] ?? "")) {
155	155	if (status === 403 \|\| status === 401 \|\| status === 429 \|\| /captcha-delivery\|geo\.captcha\|dd\.js\|DataDome/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true, retryAfterMs };
156	156	}
157		− if (h["x-kpsdk-ct"] \|\| h["x-kpsdk-c"] \|\| /kpsdk\|ips\.js/i.test(head) && status >= 400) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true, retryAfterMs };
	157	+ if (h["x-kpsdk-ct"] \|\| h["x-kpsdk-c"] \|\| (/x-kpsdk\|kpsdk-/i.test(head) && status >= 400)) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true, retryAfterMs };
158	158	if (h["x-amzn-waf-action"] === "challenge" \|\| h["x-amzn-waf-action"] === "captcha") return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true, retryAfterMs };
159	159	if (h["x-vercel-mitigated"] === "challenge" \|\| h["x-vercel-protection-bypass"] !== undefined && status === 403) return { blocked: true, reason: "anti_bot", vendor: "vercel", challenge: true, retryAfterMs };
160	160	if (status === 403 && (h["x-iinfo"] \|\| /incap_ses\|visid_incap/i.test(h["set-cookie"] ?? ""))) return { blocked: true, reason: "waf", vendor: "imperva", challenge: /_Incapsula_Resource/i.test(head), retryAfterMs };
		`@@ -176,20 +176,24 @@ export function looksBlocked(status: number, body: string, headers: Record<strin`
176	176	if (status === 202 && /datadome\|captcha-delivery/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };
177	177
178	178	// --- body-level signals (only meaningful for HTML) -------------------------
	179	+ // Vendor beacons (Cloudflare JSD, DataDome tags.js, PerimeterX, Imperva, Kasada ips.js, AWS WAF
	180	+ // challenge.js) are present on EVERY page of a protected site, so on a 2xx only interstitial-specific
	181	+ // markers count; on 4xx/5xx the mere presence of a vendor script is enough.
179	182	if (!isHtml) return { blocked: false };
180		− if (/cf-chl\|challenge-platform\|__cf_chl\|cf_chl_opt\|<title>\s*Just a moment\|cf-turnstile\|turnstile\.js\|challenges\.cloudflare\.com/i.test(head)) {
	183	+ const denied = status >= 400;
	184	+ if (/cf-chl-bypass\|__cf_chl_f_tk\|__cf_chl_rt_tk\|__cf_chl_tk\|cf_chl_opt\|<title>\s*Just a moment\|cf-chl-widget\|challenge-error-text\|cf-challenge-running\|id="challenge-(running\|stage\|form)"\|Checking your browser before accessing\|Verify you are human by completing the action/i.test(head) \|\| (denied && /cf-chl\|challenge-platform\|cf-turnstile\|challenges\.cloudflare\.com/i.test(head))) {
181	185	return { blocked: true, reason: "cloudflare_challenge", vendor: "cloudflare", challenge: true, retryAfterMs };
182	186	}
183		− if (/geo\.captcha-delivery\.com\|dd\.js\|datadome\|<title>\s*DataDome/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };
184		− if (/px-captcha\|_pxhd\|_pxvid\|perimeterx\|human-challenge\|<title>\s*Access to this page has been denied/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "perimeterx", challenge: true };
	187	+ if (/geo\.captcha-delivery\.com\|<title>\s*DataDome\|dd\.js\?\|ddCaptcha/i.test(head) \|\| (denied && /datadome/i.test(head))) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };
	188	+ if (/px-captcha\|human-challenge\|<title>\s*Access to this page has been denied/i.test(head) \|\| (denied && /perimeterx\|_pxhd\|_pxvid/i.test(head))) return { blocked: true, reason: "anti_bot", vendor: "perimeterx", challenge: true };
185	189	if (/Reference #\d\|Reference #\d+\.[0-9a-f]+\.\d+\.[0-9a-f]+\|errors\.edgesuite\.net\|akamai\.com\/us\/en\/policies/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "akamai", challenge: false };
186		− if (/_Incapsula_Resource\|Incapsula incident\|Request unsuccessful\. Incapsula/i.test(head)) return { blocked: true, reason: "waf", vendor: "imperva", challenge: true };
187		− if (/awswaf\|aws-waf-token\|challenge\.js\?\|<title>\s*Human Verification/i.test(head)) return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true };
188		− if (/kpsdk\|ips\.js\|<title>\s*Kasada/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true };
	190	+ if (/Incapsula incident\|Request unsuccessful\. Incapsula/i.test(head) \|\| (denied && /_Incapsula_Resource/i.test(head))) return { blocked: true, reason: "waf", vendor: "imperva", challenge: true };
	191	+ if (/<title>\sHuman Verification\|aws-waf-token.captcha\|awswaf.*captcha/i.test(head) \|\| (denied && /awswaf\|challenge\.js\?/i.test(head))) return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true };
	192	+ if (/<title>\s*Kasada/i.test(head) \|\| (denied && /kpsdk\|ips\.js/i.test(head))) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true };
189	193	if (/Pardon Our Interruption\|distil_r_captcha\|distil_referrer/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "distil", challenge: true };
190	194	if (/Vercel Security Checkpoint\|_vercel_challenge/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "vercel", challenge: true };
191		− if (/shape-security\|_imp_apg_r_\|<title>\s*Blocked by Shape/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "shape" };
192		− if (/sucuri\.net\|Sucuri WebSite Firewall/i.test(head)) return { blocked: true, reason: "waf", vendor: "sucuri" };
	195	+ if (/<title>\s*Blocked by Shape\|_imp_apg_r_/i.test(head) \|\| (denied && /shape-security/i.test(head))) return { blocked: true, reason: "anti_bot", vendor: "shape" };
	196	+ if (/Sucuri WebSite Firewall/i.test(head) \|\| (denied && /sucuri\.net/i.test(head))) return { blocked: true, reason: "waf", vendor: "sucuri" };
193	197	if (/www\.google\.com\/sorry\/\|Our systems have detected unusual traffic/i.test(head)) return { blocked: true, reason: "captcha", vendor: "google", challenge: false };
194	198	if (/recaptcha\/api\.js\|g-recaptcha\|hcaptcha\.com\|h-captcha\|arkoselabs\|funcaptcha\|<title>\s[^<]captcha/i.test(head) && /verify\|robot\|human\|unusual traffic\|security check\|confirm you are/i.test(head)) {
195	199	return { blocked: true, reason: "captcha", vendor: vendorFrom(server, head), challenge: false };

modified packages/core/test/ssrf.test.ts +5 −0

		`@@ -84,6 +84,11 @@ describe("block detection & text", () => {`
84	84	expect(looksBlocked(429, "", { "retry-after": "3" })).toMatchObject({ blocked: true, reason: "rate_limited", retryAfterMs: 3000 });
85	85	expect(looksBlocked(200, "<html><head><script src='/x.js'></script></head><body></body></html>", {})).toMatchObject({ blocked: true, reason: "empty_html" });
86	86	expect(looksBlocked(200, "<html><head><title>DataDome</title></head><body><script src='https://geo.captcha-delivery.com/captcha/'></script></body></html>", {})).toMatchObject({ blocked: true, vendor: "datadome" });
	87	+ // Vendor beacons on a normal page must not count as blocks.
	88	+ const normal = `<html><head><title>Cloudflare Challenge - Example</title><script src="https://js.datadome.co/tags.js"></script><script src="/challenge.js?x"></script><script src="/ips.js"></script></head><body><main><h1>You bypassed the Cloudflare challenge!</h1><p>${"content ".repeat(200)}</p></main><script src="/cdn-cgi/challenge-platform/scripts/jsd/main.js"></script><script>window._pxvid='x'; _Incapsula_Resource='y';</script></body></html>`;
	89	+ expect(looksBlocked(200, normal, { server: "cloudflare" }).blocked).toBe(false);
	90	+ expect(looksBlocked(403, normal, { server: "cloudflare" })).toMatchObject({ blocked: true, vendor: "cloudflare" });
	91	+ expect(looksBlocked(200, "<html><head><title>Just a moment...</title><script src='/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1'></script></head><body><div id='challenge-running'></div></body></html>", {})).toMatchObject({ blocked: true, reason: "cloudflare_challenge" });
87	92	});
88	93	it("extracts readable text", () => {
89	94	expect(htmlToText("<h1>Hi</h1><script>x()</script><p>There & back</p>")).toBe("Hi\nThere & back");

modified packages/routing/src/executor.ts +9 −1

		`@@ -51,6 +51,8 @@ export interface AttemptRecord {`
51	51	blockReason: string \| null;
52	52	blockVendor: string \| null;
53	53	profileId: string \| null;
	54	+ /** A captcha solver token was purchased and the attempt passed thanks to it. */
	55	+ captchaSolved: boolean;
54	56	durationMs: number;
55	57	bytesIn: number;
56	58	bytesOut: number;
		`@@ -200,6 +202,7 @@ export class FetchExecutor {`
200	202
201	203	let res: FinalResponse \| null = null;
202	204	let verdict: BlockVerdict = { blocked: false };
	205	+ let captchaOutcome: RenderResult["captcha"] = null;
203	206	let record: AttemptRecord;
204	207	try {
205	208	let bytesIn = 0;
		`@@ -223,6 +226,7 @@ export class FetchExecutor {`
223	226	javascript: request.javascript !== false,
224	227	blockResources: request.block_resources,
225	228	screenshot: request.screenshot,
	229	+ solveCaptcha: request.solve_captcha,
226	230	maxResponseBytes: maxBytes,
227	231	onRedirect: async (next) => {
228	232	await assertUrlAllowed(next);
		`@@ -241,6 +245,7 @@ export class FetchExecutor {`
241	245	};
242	246	verdict = render.block;
243	247	if (render.challengeSolved) browserRequired = true;
	248	+ captchaOutcome = render.captcha;
244	249	} else {
245	250	const pr = await cand.provider.fetch({
246	251	requestId,
		`@@ -273,7 +278,7 @@ export class FetchExecutor {`
273	278	}
274	279	const durationMs = Math.round(performance.now() - t0);
275	280	const transient = !verdict.blocked && isTransientStatus(res.status);
276		− const cost = cand.provider.estimateCost(cand.network, bytesIn + bytesOut);
	281	+ const cost = cand.provider.estimateCost(cand.network, bytesIn + bytesOut) + (captchaOutcome?.costUsd ?? 0);
277	282	record = {
278	283	attemptId,
279	284	attemptNo: i + 1,
		`@@ -289,6 +294,7 @@ export class FetchExecutor {`
289	294	blockReason: verdict.reason ?? null,
290	295	blockVendor: verdict.vendor ?? null,
291	296	profileId: profile.id,
	297	+ captchaSolved: Boolean(captchaOutcome?.solved),
292	298	durationMs,
293	299	bytesIn,
294	300	bytesOut,
		`@@ -420,6 +426,7 @@ export class FetchExecutor {`
420	426	blockReason: null,
421	427	blockVendor: null,
422	428	profileId,
	429	+ captchaSolved: false,
423	430	durationMs,
424	431	bytesIn: 0,
425	432	bytesOut: 0,
		`@@ -507,6 +514,7 @@ export class FetchExecutor {`
507	514	country: a.country,
508	515	outcome: a.outcome,
509	516	block_reason: a.blockReason,
	517	+ ...(a.captchaSolved ? { captcha_solved: true } : {}),
510	518	status: a.httpStatus,
511	519	duration_ms: a.durationMs,
512	520	...(a.errorDetail && ctx.providerVisibility ? { error: scrubProviderText(a.errorDetail) } : {}),

modified packages/sdk/src/index.ts +2 −0

		`@@ -51,6 +51,8 @@ export interface FetchOptions {`
51	51	block_resources?: boolean;
52	52	/** Browser: return a PNG screenshot (base64) in `screenshot`. */
53	53	screenshot?: boolean;
	54	+ /** Browser: solve Cloudflare Turnstile challenges with the platform's captcha solver (default true). */
	55	+ solve_captcha?: boolean;
54	56	/** Return `links[]` (all hyperlinks, absolute). */
55	57	links?: boolean;
56	58	/** Referer strategy: "auto" (none first, search-engine referer on retries), "none", or a literal URL. */

modified sdk-python/fetcha/__init__.py +1 −1

		`@@ -367,7 +367,7 @@ class Fetcha:`
367	367	def fetch(self, url: str, **options: Any) -> FetchResult:
368	368	"""POST /v1/fetch. Options: method, headers, cookies, body, timeout, country, region, city,
369	369	network, session, browser, browser_fallback, wait_for, wait_ms, wait_until, javascript,
370		− block_resources, screenshot, links, referer, device, locale, format (html\|text\|markdown\|json\|raw),
	370	+ block_resources, screenshot, solve_captcha, links, referer, device, locale, format (html\|text\|markdown\|json\|raw),
371	371	follow_redirects, max_redirects, max_response_bytes, retries, debug."""
372	372	payload = {"url": url, **_clean(options)}
373	373	return FetchResult.from_dict(self._request("POST", "/v1/fetch", json=payload))
374	374