v0.2.1 — 2captcha Turnstile solver (onload-hook interception, UA adoption via CDP), solve_captcha option, real Chrome channel + headful mode, tighter 2xx block signatures, Patchright main-world evaluates · ba9ab18 · fetcha

v0.2.1 — 2captcha Turnstile solver (onload-hook interception, UA adoption via CDP), solve_captcha option, real Chrome channel + headful mode, tighter 2xx block signatures, Patchright main-world evaluates

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 2 mo ago (Sep 8, 2026) parent 23fd257 Browse files

14 changed files +457 −27

modified .env.example +4 −1

				`@@ -37,9 +37,12 @@ ADMIN_EMAILS=you@example.com`
37	# Managed Chromium (Playwright). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.	37	# Managed Chromium (Playwright). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
38	FETCHA_BROWSER_ENABLED=1	38	FETCHA_BROWSER_ENABLED=1
39	FETCHA_BROWSER_CONCURRENCY=6	39	FETCHA_BROWSER_CONCURRENCY=6
40	−# "chromium" = full Chromium build in new headless mode (best anti-bot realism); "chrome" uses an installed Google Chrome.	40	+# "chromium" = full Chromium build (Patchright); "chrome" uses an installed Google Chrome (prod). FETCHA_BROWSER_HEADLESS=0 = real off-screen window (needs a GUI session).
41	FETCHA_BROWSER_CHANNEL=chromium	41	FETCHA_BROWSER_CHANNEL=chromium
		42	+FETCHA_BROWSER_HEADLESS=1
42	# Prefer HTTP/2 towards origins (set 0 to force HTTP/1.1).	43	# Prefer HTTP/2 towards origins (set 0 to force HTTP/1.1).
43	FETCHA_HTTP2=1	44	FETCHA_HTTP2=1
44	# Crawl jobs executed in parallel by this API process.	45	# Crawl jobs executed in parallel by this API process.
45	FETCHA_CRAWL_PARALLEL_JOBS=4	46	FETCHA_CRAWL_PARALLEL_JOBS=4
		47	+# Captcha solver (2captcha.com) for Cloudflare Turnstile challenges in browser mode. Empty = disabled.
		48	+TWOCAPTCHA_API_KEY=

modified CLAUDE.md +2 −2

				@@ -13,7 +13,7 @@ and a full dashboard. Private platform (v0.2, 2026-09-08): a single `unlimit
13	- `packages/db` — Drizzle schema + migrations (`drizzle/`), `pnpm db:generate\|migrate\|seed`.	13	- `packages/db` — Drizzle schema + migrations (`drizzle/`), `pnpm db:generate\|migrate\|seed`.
14	- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct), `http.ts` (undici, HTTP/2, browser-ordered headers, Chrome/Firefox/Safari TLS cipher lists, h2→h1 fallback), `fingerprint.ts` (header profiles), `cookies.ts` (jar across hops / sessions). Provider code lives ONLY here.	14	- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct), `http.ts` (undici, HTTP/2, browser-ordered headers, Chrome/Firefox/Safari TLS cipher lists, h2→h1 fallback), `fingerprint.ts` (header profiles), `cookies.ts` (jar across hops / sessions). Provider code lives ONLY here.
15	- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, fingerprint rotation, jittered backoff, block detection, HTTP → browser escalation).	15	- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, fingerprint rotation, jittered backoff, block detection, HTTP → browser escalation).
16	−- `packages/browser` — `BrowserPool` (Patchright = patched Playwright, full Chromium new-headless, per-context upstream proxy, stealth init script, challenge wait + Turnstile click). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.	16	+- `packages/browser` — `BrowserPool` (Patchright = patched Playwright, Chrome/Chromium, per-context upstream proxy, stealth init script, challenge wait, 2captcha Turnstile solver in `captcha.ts`). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
17	- `packages/core/src/markdown.ts` — HTML → Markdown / main text, page metadata + links, URL normalisation, glob/regex matchers; `robots.ts` — robots.txt + sitemap parsing.	17	- `packages/core/src/markdown.ts` — HTML → Markdown / main text, page metadata + links, URL normalisation, glob/regex matchers; `robots.ts` — robots.txt + sitemap parsing.
18	- `apps/api/src/services/crawl.ts` — durable crawl jobs (`crawl_jobs`/`crawl_pages`, in-process worker, resume after restart) and the sync `/v1/map`.	18	- `apps/api/src/services/crawl.ts` — durable crawl jobs (`crawl_jobs`/`crawl_pages`, in-process worker, resume after restart) and the sync `/v1/map`.
19	- `packages/email` — Resend abstraction + React Email templates. `packages/sdk` (JS), `sdk-python/` (Python).	19	- `packages/email` — Resend abstraction + React Email templates. `packages/sdk` (JS), `sdk-python/` (Python).
				@@ -26,7 +26,7 @@ and a full dashboard. Private platform (v0.2, 2026-09-08): a single `unlimit
26	- API keys: shown once, SHA-256 stored. Request IDs `req_…` on every response (`X-Fetcha-Request-ID`).	26	- API keys: shown once, SHA-256 stored. Request IDs `req_…` on every response (`X-Fetcha-Request-ID`).
27	- Not implemented yet (say so, don't fake): scripted browser actions (`POST /v1/browser`), `/v1/extract`, teams, webhook delivery (except the crawl `webhook_url` callback), OAuth, 2FA. Billing/Stripe is intentionally absent (private platform).	27	- Not implemented yet (say so, don't fake): scripted browser actions (`POST /v1/browser`), `/v1/extract`, teams, webhook delivery (except the crawl `webhook_url` callback), OAuth, 2FA. Billing/Stripe is intentionally absent (private platform).
28	- Plans: always `normalizePlan(org.plan)`; never reintroduce tiers. Signup must stay allowlist-gated (Better Auth `user.create.before` hook in `apps/web/src/lib/auth.ts`).	28	- Plans: always `normalizePlan(org.plan)`; never reintroduce tiers. Signup must stay allowlist-gated (Better Auth `user.create.before` hook in `apps/web/src/lib/auth.ts`).
29	−- Anti-bot honesty: interactive Cloudflare Turnstile challenges are NOT reliably solved (tested 2026-09-08 with Patchright + residential + headful); non-interactive JS challenges, DataDome/PX/Akamai soft blocks usually pass via residential + browser escalation.	29	+- Anti-bot: interactive Cloudflare Turnstile challenges are solved with 2captcha (`TWOCAPTCHA_API_KEY`, `packages/browser/src/captcha.ts`): the init script wraps the api.js `?onload=` callback to capture `turnstile.render` params (sitekey, action, cData, chlPageData, callback), a token is bought (~$0.0015, 5–10 s) and injected through the callback after switching the page UA (CDP `Emulation.setUserAgentOverride`) to the solver's UA. Patchright runs init scripts in the main world but `page.evaluate` in an isolated world — pass `undefined, false` (4th arg) to read main-world state. `looksBlocked()` on 2xx must only match interstitial markers (vendor beacons like `challenge-platform/scripts/jsd`, `tags.js`, `ips.js` are on every protected page). Prod browser = Google Chrome stable (brew cask on M3U96a), `FETCHA_BROWSER_HEADLESS=0` (real off-screen window in the node's GUI session).
30		30
31	## Dev	31	## Dev
32	`cp .env.example .env` (fill provider creds + RESEND_API_KEY), `createdb fetcha`, `pnpm db:migrate && pnpm db:seed`,	32	`cp .env.example .env` (fill provider creds + RESEND_API_KEY), `createdb fetcha`, `pnpm db:migrate && pnpm db:seed`,

modified apps/web/src/app/(marketing)/changelog/page.tsx +12 −0

				`@@ -23,6 +23,18 @@ type Entry = {`
23	};	23	};
24		24
25	const ENTRIES: Entry[] = [	25	const ENTRIES: Entry[] = [
		26	+ {
		27	+ date: "2026-09-08",
		28	+ version: "v0.2.1",
		29	+ title: "Turnstile captcha solving, real Chrome",
		30	+ tag: "stable",
		31	+ summary: "The managed browser now solves interactive Cloudflare Turnstile challenges through a human-verification service and runs on a real Google Chrome build.",
		32	+ changes: [
		33	+ { kind: "added", text: "Captcha solver (2captcha) for Cloudflare Turnstile: widget parameters are intercepted in-page, a token is obtained and injected with the matching user agent. New request field solve_captcha (default true); attempts report captcha_solved in metadata.debug." },
		34	+ { kind: "changed", text: "Managed browser runs Google Chrome (stable channel) as a real window on the node instead of the headless Chromium build, with the native user agent and client hints kept coherent." },
		35	+ { kind: "changed", text: "Admin → System lists the anti-bot configuration (solver key presence, browser channel and mode); /internal/browser exposes solver statistics." },
		36	+ ],
		37	+ },
26	{	38	{
27	date: "2026-09-08",	39	date: "2026-09-08",
28	version: "v0.2.0",	40	version: "v0.2.0",

modified apps/web/src/app/(marketing)/docs/browser/page.tsx +8 −0

				`@@ -124,6 +124,14 @@ export default function BrowserPage() {`
124	</Li>	124	</Li>
125	</Ul>	125	</Ul>
126		126
		127	+ <H2>Captcha solving</H2>
		128	+ <P>
		129	+ Most JavaScript challenges clear on their own inside the real browser. When Cloudflare asks for an interactive Turnstile verification instead, Fetcha intercepts the widget parameters,
		130	+ obtains a token from a managed human-verification service (2captcha) and hands it to the page, adopting the user agent the token was issued for. This typically adds 10–40 seconds, so keep
		131	+ <Code>timeout</Code> generous (60 s or more) for hard targets. Set <Code>{`solve_captcha: false`}</Code> to opt out on a request. Attempts that needed a token report{" "}
		132	+ <Code>captcha_solved: true</Code> in <Code>metadata.debug.attempts[]</Code>.
		133	+ </P>
		134	+
127	<H2>Automatic escalation</H2>	135	<H2>Automatic escalation</H2>
128	<P>	136	<P>
129	You rarely need to set <Code>browser: true</Code> yourself. With the default <Code>browser_fallback: true</Code>, Fetcha starts every request over plain HTTP because it is faster and cheaper.	137	You rarely need to set <Code>browser: true</Code> yourself. With the default <Code>browser_fallback: true</Code>, Fetcha starts every request over plain HTTP because it is faster and cheaper.

modified apps/web/src/app/(marketing)/docs/fetch/page.tsx +1 −0

				`@@ -176,6 +176,7 @@ export default function FetchApiPage() {`
176	{ name: "javascript", type: "boolean", default: "true", description: <>Browser: set to <code>false</code> to render with scripting disabled.</> },	176	{ name: "javascript", type: "boolean", default: "true", description: <>Browser: set to <code>false</code> to render with scripting disabled.</> },
177	{ name: "block_resources", type: "boolean", default: "true", description: <>Browser: skip images, fonts and media to save bandwidth and time. Page scripts and XHR still run.</> },	177	{ name: "block_resources", type: "boolean", default: "true", description: <>Browser: skip images, fonts and media to save bandwidth and time. Page scripts and XHR still run.</> },
178	{ name: "screenshot", type: "boolean", default: "false", description: <>Browser: return a PNG of the viewport, base64-encoded, in <code>screenshot</code>.</> },	178	{ name: "screenshot", type: "boolean", default: "false", description: <>Browser: return a PNG of the viewport, base64-encoded, in <code>screenshot</code>.</> },
		179	+ { name: "solve_captcha", type: "boolean", default: "true", description: <>Browser: when a Cloudflare Turnstile challenge blocks the render, obtain a token from the managed captcha solver and pass it to the page. Set <code>false</code> to never spend solver credits on a request. Attempts that needed a token report <code>captcha_solved: true</code> in <code>metadata.debug</code>.</> },
179	{ name: "cache", type: "{ enabled?: boolean, ttl?: integer }", reserved: true, constraints: "ttl 1–86,400 s", description: <>Reserved for response caching. Accepted, ignored; <code>metadata.cached</code> is always <code>false</code> today.</> },	180	{ name: "cache", type: "{ enabled?: boolean, ttl?: integer }", reserved: true, constraints: "ttl 1–86,400 s", description: <>Reserved for response caching. Accepted, ignored; <code>metadata.cached</code> is always <code>false</code> today.</> },
180	]}	181	]}
181	/>	182	/>

modified apps/web/src/lib/queries/admin.ts +1 −0

				`@@ -1048,6 +1048,7 @@ export function getEnvPresence(): Array<{ group: string; key: string; present: b`
1048	["Upstream providers", ["OXYLABS_USERNAME", "OXYLABS_PASSWORD", "DECODO_USERNAME", "DECODO_PASSWORD", "SOAX_USERNAME", "SOAX_PASSWORD"]],	1048	["Upstream providers", ["OXYLABS_USERNAME", "OXYLABS_PASSWORD", "DECODO_USERNAME", "DECODO_PASSWORD", "SOAX_USERNAME", "SOAX_PASSWORD"]],
1049	["Core", ["DATABASE_URL", "REDIS_URL", "AUTH_SECRET", "INTERNAL_SERVICE_TOKEN", "API_URL", "NEXT_PUBLIC_SITE_URL", "ADMIN_EMAILS"]],	1049	["Core", ["DATABASE_URL", "REDIS_URL", "AUTH_SECRET", "INTERNAL_SERVICE_TOKEN", "API_URL", "NEXT_PUBLIC_SITE_URL", "ADMIN_EMAILS"]],
1050	["Email", ["RESEND_API_KEY", "EMAIL_FROM", "EMAIL_FROM_TRANSACTIONAL"]],	1050	["Email", ["RESEND_API_KEY", "EMAIL_FROM", "EMAIL_FROM_TRANSACTIONAL"]],
		1051	+ ["Anti-bot", ["TWOCAPTCHA_API_KEY", "FETCHA_BROWSER_ENABLED", "FETCHA_BROWSER_CHANNEL", "FETCHA_BROWSER_HEADLESS"]],
1051	["Billing", ["STRIPE_SECRET_KEY", "STRIPE_WEBHOOK_SECRET"]],	1052	["Billing", ["STRIPE_SECRET_KEY", "STRIPE_WEBHOOK_SECRET"]],
1052	];	1053	];
1053	return spec.flatMap(([group, keys]) => keys.map((key) => ({ group, key, present: Boolean(process.env[key]?.trim()) })));	1054	return spec.flatMap(([group, keys]) => keys.map((key) => ({ group, key, present: Boolean(process.env[key]?.trim()) })));

added packages/browser/src/captcha.ts +233 −0

++/**
++ * Captcha solving via 2captcha (https://2captcha.com, JSON API v2).
++ *
++ * Used for Cloudflare Turnstile — both the standalone widget (`data-sitekey`) and the managed
++ * challenge page ("Just a moment…"), where the `turnstile.render` call must be intercepted to grab
++ * `sitekey`, `action`, `cData` and `chlPageData`, and the returned token handed to the widget's
++ * callback. The solver also returns the User-Agent the token was produced with; the page must adopt
++ * it before invoking the callback.
++ */
++export interface TurnstileParams {
++  sitekey: string;
++  action?: string | null;
++  cData?: string | null;
++  chlPageData?: string | null;
++  /** Whether a JS callback was captured (challenge page / explicit render). */
++  hasCallback: boolean;
++}
++
++export interface TurnstileSolution {
++  token: string;
++  userAgent: string | null;
++  taskId: string;
++  ms: number;
++  costUsd: number;
++}
++
++export interface CaptchaSolverOptions {
++  apiKey: string;
++  endpoint?: string;
++  /** Overall budget for a solve (default 110 s). */
++  timeoutMs?: number;
++  /** First poll delay / subsequent poll interval (2captcha recommends ~5 s then 3 s). */
++  firstPollMs?: number;
++  pollMs?: number;
++  /** Price per Turnstile solve, for cost accounting (2captcha lists ~$1.45 / 1000). */
++  pricePerSolveUsd?: number;
++  fetch?: typeof globalThis.fetch;
++  log?: { info: (m: string) => void; warn: (m: string) => void };
++}
++
++export class CaptchaSolverError extends Error {
++  constructor(
++    readonly code: string,
++    message: string,
++  ) {
++    super(message);
++    this.name = "CaptchaSolverError";
++  }
++}
++
++interface TaskResult {
++  errorId: number;
++  errorCode?: string;
++  errorDescription?: string;
++  status?: "processing" | "ready";
++  solution?: { token?: string; userAgent?: string };
++  taskId?: number | string;
++  balance?: number;
++}
++
++export class TwoCaptchaSolver {
++  readonly provider = "2captcha" as const;
++  private readonly opts: Required<Omit<CaptchaSolverOptions, "log">> & { log: NonNullable<CaptchaSolverOptions["log"]> };
++  private stats = { requested: 0, solved: 0, failed: 0, spentUsd: 0, lastError: null as string | null };
++
++  constructor(opts: CaptchaSolverOptions) {
++    if (!opts.apiKey) throw new Error("2captcha: apiKey is required");
++    this.opts = {
++      apiKey: opts.apiKey,
++      endpoint: (opts.endpoint ?? "https://api.2captcha.com").replace(/\/$/, ""),
++      timeoutMs: opts.timeoutMs ?? 110_000,
++      firstPollMs: opts.firstPollMs ?? 5000,
++      pollMs: opts.pollMs ?? 3000,
++      pricePerSolveUsd: opts.pricePerSolveUsd ?? 0.00145,
++      fetch: opts.fetch ?? globalThis.fetch.bind(globalThis),
++      log: opts.log ?? { info: (m) => console.log("[2captcha]", m), warn: (m) => console.warn("[2captcha]", m) },
++    };
++  }
++
++  status() {
++    return { provider: this.provider, ...this.stats };
++  }
++
++  private async call(path: string, body: Record<string, unknown>, timeoutMs = 20_000): Promise<TaskResult> {
++    const ac = new AbortController();
++    const t = setTimeout(() => ac.abort(), timeoutMs);
++    try {
++      const res = await this.opts.fetch(`${this.opts.endpoint}${path}`, {
++        method: "POST",
++        headers: { "content-type": "application/json", "user-agent": "fetcha/0.2 (+https://www.fetcha.co)" },
++        body: JSON.stringify({ clientKey: this.opts.apiKey, ...body }),
++        signal: ac.signal,
++      });
++      const data = (await res.json().catch(() => ({}))) as TaskResult;
++      if (!res.ok) throw new CaptchaSolverError("HTTP_ERROR", `2captcha HTTP ${res.status}`);
++      return data;
++    } finally {
++      clearTimeout(t);
++    }
++  }
++
++  async balance(): Promise<number> {
++    const r = await this.call("/getBalance", {});
++    if (r.errorId) throw new CaptchaSolverError(r.errorCode ?? "ERROR", r.errorDescription ?? "2captcha error");
++    return Number(r.balance ?? 0);
++  }
++
++  /**
++   * Solve a Turnstile challenge. `deadline` (performance.now() based) bounds the wait so that the
++   * caller's request timeout is honoured.
++   */
++  async solveTurnstile(input: { websiteURL: string; params: TurnstileParams; deadline?: number }): Promise<TurnstileSolution> {
++    const t0 = performance.now();
++    const hardDeadline = Math.min(t0 + this.opts.timeoutMs, input.deadline ?? Infinity);
++    this.stats.requested++;
++    const task: Record<string, unknown> = { type: "TurnstileTaskProxyless", websiteURL: input.websiteURL, websiteKey: input.params.sitekey };
++    if (input.params.action) task.action = input.params.action;
++    if (input.params.cData) task.data = input.params.cData;
++    if (input.params.chlPageData) task.pagedata = input.params.chlPageData;
++    let created: TaskResult;
++    try {
++      created = await this.call("/createTask", { task });
++    } catch (e) {
++      this.fail((e as Error).message);
++      throw e instanceof CaptchaSolverError ? e : new CaptchaSolverError("NETWORK", (e as Error).message);
++    }
++    if (created.errorId || !created.taskId) {
++      const code = created.errorCode ?? "CREATE_FAILED";
++      this.fail(`${code}: ${created.errorDescription ?? ""}`);
++      throw new CaptchaSolverError(code, created.errorDescription ?? "2captcha refused the task");
++    }
++    const taskId = String(created.taskId);
++    let wait = this.opts.firstPollMs;
++    for (;;) {
++      const remaining = hardDeadline - performance.now();
++      if (remaining <= 0) {
++        this.fail("timeout");
++        throw new CaptchaSolverError("TIMEOUT", "2captcha did not solve the challenge in time");
++      }
++      await new Promise((r) => setTimeout(r, Math.min(wait, remaining)));
++      wait = this.opts.pollMs;
++      let r: TaskResult;
++      try {
++        r = await this.call("/getTaskResult", { taskId: created.taskId });
++      } catch (e) {
++        // transient network error: keep polling
++        this.opts.log.warn(`poll error: ${(e as Error).message}`);
++        continue;
++      }
++      if (r.errorId) {
++        const code = r.errorCode ?? "SOLVE_FAILED";
++        this.fail(`${code}: ${r.errorDescription ?? ""}`);
++        throw new CaptchaSolverError(code, r.errorDescription ?? "2captcha could not solve the challenge");
++      }
++      if (r.status === "ready" && r.solution?.token) {
++        const ms = Math.round(performance.now() - t0);
++        this.stats.solved++;
++        this.stats.spentUsd += this.opts.pricePerSolveUsd;
++        this.opts.log.info(`turnstile solved in ${ms} ms (task ${taskId})`);
++        return { token: r.solution.token, userAgent: r.solution.userAgent ?? null, taskId, ms, costUsd: this.opts.pricePerSolveUsd };
++      }
++    }
++  }
++
++  private fail(msg: string) {
++    this.stats.failed++;
++    this.stats.lastError = msg.slice(0, 200);
++    this.opts.log.warn(`solve failed: ${msg}`);
++  }
++}
++
++/**
++ * Init script: intercept `turnstile.render` (challenge pages and explicit widgets) to capture the
++ * parameters 2captcha needs, and keep the callback so the token can be injected later. Stored on a
++ * non-enumerable, obscurely named window property.
++ */
++export const TURNSTILE_HOOK_KEY = "__fx_ts_" + "9c1";
++
++export function turnstileHookScript(): string {
++  return `(() => {
++  const KEY = ${JSON.stringify(TURNSTILE_HOOK_KEY)};
++  const store = { params: null, callback: null, rendered: 0, hooked: [] };
++  try { Object.defineProperty(window, KEY, { value: store, enumerable: false, configurable: true, writable: false }); } catch { window[KEY] = store; }
++  const patchTs = (ts) => {
++    if (!ts || typeof ts !== 'object' || ts.__fx) return false;
++    const origRender = ts.render;
++    const patched = function (container, params) {
++      try {
++        const p = params || {};
++        store.rendered++;
++        store.params = { sitekey: p.sitekey || null, action: p.action || null, cData: p.cData || null, chlPageData: p.chlPageData || null, hasCallback: typeof p.callback === 'function' };
++        if (typeof p.callback === 'function') store.callback = p.callback;
++      } catch {}
++      // Challenge pages: do NOT render the real widget (its own attempt would consume the challenge
++      // data we are about to solve out-of-band); return a fake widget id instead (2captcha's guidance).
++      if (store.params && store.params.chlPageData) return 'fx-' + Math.random().toString(36).slice(2, 10);
++      return typeof origRender === 'function' ? origRender.apply(this, arguments) : undefined;
++    };
++    try { Object.defineProperty(patched, 'toString', { value: () => 'function render() { [native code] }' }); } catch {}
++    try { ts.render = patched; Object.defineProperty(ts, '__fx', { value: true, enumerable: false, configurable: true }); } catch { return false; }
++    return true;
++  };
++  // 1) Challenge pages load api.js with ?onload=<name>&render=explicit and call turnstile.render from that
++  //    callback. Wrapping the callback catches the turnstile object at the exact moment it exists.
++  const hookOnload = (name) => {
++    if (!name || store.hooked.includes(name)) return;
++    store.hooked.push(name);
++    let cur = window[name];
++    const wrap = (fn) => function () {
++      try { patchTs(arguments[0] && typeof arguments[0].render === 'function' ? arguments[0] : window.turnstile); } catch {}
++      return fn.apply(this, arguments);
++    };
++    try {
++      Object.defineProperty(window, name, { configurable: true, enumerable: true, get() { return cur; }, set(v) { cur = typeof v === 'function' ? wrap(v) : v; } });
++      if (typeof cur === 'function') cur = wrap(cur);
++    } catch {}
++  };
++  const onloadName = (src) => { const m = /[?&]onload=([A-Za-z0-9_$]+)/.exec(src || ''); return m ? m[1] : null; };
++  const scan = (root) => { try { for (const el of root.querySelectorAll('script[src*="turnstile"]')) hookOnload(onloadName(el.src)); } catch {} };
++  try {
++    new MutationObserver((muts) => {
++      for (const m of muts) for (const n of m.addedNodes) {
++        if (n.nodeName === 'SCRIPT' && n.src && /turnstile/.test(n.src)) hookOnload(onloadName(n.src));
++        else if (n.querySelectorAll) scan(n);
++      }
++    }).observe(document, { childList: true, subtree: true });
++  } catch {}
++  document.addEventListener('DOMContentLoaded', () => scan(document));
++  // 2) Standalone widgets / other pages: poll for window.turnstile (api.js sets it on load).
++  let n = 0;
++  const iv = setInterval(() => { if (patchTs(window.turnstile) || ++n > 1200) clearInterval(iv); }, 25);
++})();`;
++}

modified packages/browser/src/index.ts +162 −13

				`@@ -14,6 +14,9 @@ import type { FingerprintProfile, ProxyEndpoint } from "@fetcha/providers";`
14	import { acceptLanguage } from "@fetcha/providers";	14	import { acceptLanguage } from "@fetcha/providers";
15	import type { Browser, BrowserContext, Page, Response as PwResponse } from "patchright";	15	import type { Browser, BrowserContext, Page, Response as PwResponse } from "patchright";
16	import { stealthScript, timezoneFor, webglFor } from "./stealth";	16	import { stealthScript, timezoneFor, webglFor } from "./stealth";
		17	+import { TURNSTILE_HOOK_KEY, TwoCaptchaSolver, turnstileHookScript, type TurnstileParams, type TurnstileSolution } from "./captcha";
		18	+
		19	+export * from "./captcha";
17		20
18	export interface RenderRequest {	21	export interface RenderRequest {
19	url: string;	22	url: string;
				`@@ -34,10 +37,20 @@ export interface RenderRequest {`
34	blockResources: boolean;	37	blockResources: boolean;
35	screenshot: boolean;	38	screenshot: boolean;
36	maxResponseBytes: number;	39	maxResponseBytes: number;
		40	+ /** Use the configured captcha solver (2captcha) on Turnstile challenges (default true). */
		41	+ solveCaptcha?: boolean;
37	/** Called for every main-frame navigation to a new URL (redirects). Throw to abort. */	42	/** Called for every main-frame navigation to a new URL (redirects). Throw to abort. */
38	onRedirect?: (nextUrl: string) => Promise<void>;	43	onRedirect?: (nextUrl: string) => Promise<void>;
39	}	44	}
40		45
		46	+export interface CaptchaOutcome {
		47	+ provider: "2captcha";
		48	+ /** Token obtained and the page passed afterwards. */
		49	+ solved: boolean;
		50	+ ms: number;
		51	+ costUsd: number;
		52	+}
		53	+
41	export interface RenderResult {	54	export interface RenderResult {
42	status: number;	55	status: number;
43	headers: Record<string, string>;	56	headers: Record<string, string>;
				`@@ -54,6 +67,8 @@ export interface RenderResult {`
54	challengeSolved: boolean;	67	challengeSolved: boolean;
55	/** Block verdict on the captured DOM (after the challenge wait). */	68	/** Block verdict on the captured DOM (after the challenge wait). */
56	block: ReturnType<typeof looksBlocked>;	69	block: ReturnType<typeof looksBlocked>;
		70	+ /** Captcha solver involvement, when a token was purchased. */
		71	+ captcha: CaptchaOutcome \| null;
57	}	72	}
58		73
59	export interface BrowserPoolOptions {	74	export interface BrowserPoolOptions {
				`@@ -70,6 +85,8 @@ export interface BrowserPoolOptions {`
70	challengeWaitMs?: number;	85	challengeWaitMs?: number;
71	/** Run a real (off-screen) window instead of headless mode. Requires a graphical session. */	86	/** Run a real (off-screen) window instead of headless mode. Requires a graphical session. */
72	headless?: boolean;	87	headless?: boolean;
		88	+ /** 2captcha API key (default: env TWOCAPTCHA_API_KEY). Empty = no solver. */
		89	+ captchaApiKey?: string \| null;
73	log?: { info: (m: string) => void; warn: (m: string) => void };	90	log?: { info: (m: string) => void; warn: (m: string) => void };
74	}	91	}
75		92
				`@@ -84,6 +101,9 @@ export interface BrowserStatus {`
84	challengesSolved: number;	101	challengesSolved: number;
85	lastError: string \| null;	102	lastError: string \| null;
86	version: string \| null;	103	version: string \| null;
		104	+ headless: boolean;
		105	+ channel: string;
		106	+ captchaSolver: { provider: string; requested: number; solved: number; failed: number; spentUsd: number; lastError: string \| null } \| null;
87	}	107	}
88		108
89	const CHALLENGE_SELECTORS = ["#challenge-running", "#challenge-form", "#challenge-stage", ".cf-turnstile", "iframe[src='challenges.cloudflare.com']", "#px-captcha", "#datadome", "iframe[src='captcha-delivery.com']", "#sec-cpt-if", "form#challenge", "#cmsg", "[data-testid='challenge']"];	109	const CHALLENGE_SELECTORS = ["#challenge-running", "#challenge-form", "#challenge-stage", ".cf-turnstile", "iframe[src='challenges.cloudflare.com']", "#px-captcha", "#datadome", "iframe[src='captcha-delivery.com']", "#sec-cpt-if", "form#challenge", "#cmsg", "[data-testid='challenge']"];
				`@@ -93,6 +113,9 @@ export class BrowserPool {`
93	private launching: Promise<Browser> \| null = null;	113	private launching: Promise<Browser> \| null = null;
94	/** User agent reported by the real Chromium build (with "HeadlessChrome" normalised to "Chrome"). */	114	/** User agent reported by the real Chromium build (with "HeadlessChrome" normalised to "Chrome"). */
95	private nativeUa: string \| null = null;	115	private nativeUa: string \| null = null;
		116	+ private readonly solver: TwoCaptchaSolver \| null;
		117	+ /** User-Agent of the last solver token: reused up-front so challenge pages see one consistent UA. */
		118	+ private lastSolverUa: string \| null = null;
96	private running = 0;	119	private running = 0;
97	private queue: Array<() => void> = [];	120	private queue: Array<() => void> = [];
98	private idleTimer: NodeJS.Timeout \| null = null;	121	private idleTimer: NodeJS.Timeout \| null = null;
				`@@ -108,14 +131,20 @@ export class BrowserPool {`
108	enabled: opts.enabled ?? process.env.FETCHA_BROWSER_ENABLED !== "0",	131	enabled: opts.enabled ?? process.env.FETCHA_BROWSER_ENABLED !== "0",
109	challengeWaitMs: opts.challengeWaitMs ?? 18_000,	132	challengeWaitMs: opts.challengeWaitMs ?? 18_000,
110	headless: opts.headless ?? process.env.FETCHA_BROWSER_HEADLESS !== "0",	133	headless: opts.headless ?? process.env.FETCHA_BROWSER_HEADLESS !== "0",
		134	+ captchaApiKey: opts.captchaApiKey === undefined ? process.env.TWOCAPTCHA_API_KEY?.trim() \|\| null : opts.captchaApiKey,
111	log: opts.log ?? { info: (m) => console.log("[browser]", m), warn: (m) => console.warn("[browser]", m) },	135	log: opts.log ?? { info: (m) => console.log("[browser]", m), warn: (m) => console.warn("[browser]", m) },
112	};	136	};
		137	+ this.solver = this.opts.captchaApiKey ? new TwoCaptchaSolver({ apiKey: this.opts.captchaApiKey, log: { info: (m) => this.opts.log.info(`2captcha: ${m}`), warn: (m) => this.opts.log.warn(`2captcha: ${m}`) } }) : null;
113	}	138	}
114		139
115	get enabled(): boolean {	140	get enabled(): boolean {
116	return this.opts.enabled;	141	return this.opts.enabled;
117	}	142	}
118		143
		144	+ get captchaSolver(): TwoCaptchaSolver \| null {
		145	+ return this.solver;
		146	+ }
		147	+
119	status(): BrowserStatus {	148	status(): BrowserStatus {
120	return {	149	return {
121	enabled: this.opts.enabled,	150	enabled: this.opts.enabled,
				`@@ -128,6 +157,9 @@ export class BrowserPool {`
128	challengesSolved: this.stats.challengesSolved,	157	challengesSolved: this.stats.challengesSolved,
129	lastError: this.stats.lastError,	158	lastError: this.stats.lastError,
130	version: this.browser?.version() ?? null,	159	version: this.browser?.version() ?? null,
		160	+ headless: this.opts.headless,
		161	+ channel: this.opts.channel,
		162	+ captchaSolver: this.solver ? this.solver.status() : null,
131	};	163	};
132	}	164	}
133		165
				`@@ -287,6 +319,7 @@ export class BrowserPool {`
287	const webgl = webglFor(customUa \|\| isMobile ? profile.platform : /Windows/.test(userAgent) ? "Win32" : /Linux/.test(userAgent) ? "Linux x86_64" : "MacIntel", profile.tls);	319	const webgl = webglFor(customUa \|\| isMobile ? profile.platform : /Windows/.test(userAgent) ? "Win32" : /Linux/.test(userAgent) ? "Linux x86_64" : "MacIntel", profile.tls);
288	const platform = customUa \|\| isMobile ? profile.platform : /Windows/.test(userAgent) ? "Win32" : /Linux/.test(userAgent) ? "Linux x86_64" : "MacIntel";	320	const platform = customUa \|\| isMobile ? profile.platform : /Windows/.test(userAgent) ? "Win32" : /Linux/.test(userAgent) ? "Linux x86_64" : "MacIntel";
289	await context.addInitScript(stealthScript({ platform, languages, hardwareConcurrency: isMobile ? 8 : 12, deviceMemory: 8, vendor: webgl.vendor, renderer: webgl.renderer, mobile: isMobile }));	321	await context.addInitScript(stealthScript({ platform, languages, hardwareConcurrency: isMobile ? 8 : 12, deviceMemory: 8, vendor: webgl.vendor, renderer: webgl.renderer, mobile: isMobile }));
		322	+ if (this.solver && req.solveCaptcha !== false) await context.addInitScript(turnstileHookScript());
290	if (req.cookies?.length) {	323	if (req.cookies?.length) {
291	await context	324	await context
292	.addCookies(	325	.addCookies(
				`@@ -336,7 +369,7 @@ export class BrowserPool {`
336	if (req.method !== "GET") {	369	if (req.method !== "GET") {
337	const headers = { ...r.headers() };	370	const headers = { ...r.headers() };
338	if (req.body !== undefined && !headers["content-type"]) headers["content-type"] = typeof req.body === "string" ? "application/json" : "application/octet-stream";	371	if (req.body !== undefined && !headers["content-type"]) headers["content-type"] = typeof req.body === "string" ? "application/json" : "application/octet-stream";
339	− return route.continue({ method: req.method, postData: req.body, headers });	372	+ return route.fallback({ method: req.method, postData: req.body, headers });
340	}	373	}
341	} else if (r.url() !== initialUrl.toString()) {	374	} else if (r.url() !== initialUrl.toString()) {
342	redirects++;	375	redirects++;
				`@@ -349,13 +382,14 @@ export class BrowserPool {`
349	}	382	}
350	}	383	}
351	}	384	}
352	− return route.continue();	385	+ // fallback() (not continue()) lets Patchright's context-level route inject the init scripts.
		386	+ return route.fallback();
353	}	387	}
354	if (req.blockResources) {	388	if (req.blockResources) {
355	const type = r.resourceType();	389	const type = r.resourceType();
356	if (type === "image" \|\| type === "media" \|\| type === "font" \|\| type === "manifest" \|\| type === "texttrack") return route.abort("blockedbyclient");	390	if (type === "image" \|\| type === "media" \|\| type === "font" \|\| type === "manifest" \|\| type === "texttrack") return route.abort("blockedbyclient");
357	}	391	}
358	− return route.continue();	392	+ return route.fallback();
359	});	393	});
360	page.on("requestfinished", (r) => {	394	page.on("requestfinished", (r) => {
361	pendingSizes.push(	395	pendingSizes.push(
				`@@ -398,25 +432,27 @@ export class BrowserPool {`
398	if (policyError) throw policyError;	432	if (policyError) throw policyError;
399	const navigationMs = Math.round(performance.now() - tNav0);	433	const navigationMs = Math.round(performance.now() - tNav0);
400		434
401	− // Challenge handling: give the page time to solve JS challenges (Cloudflare, DataDome, PX…).	435	+ // Challenge handling: give the page time to solve JS challenges (Cloudflare, DataDome, PX…),
		436	+ // and buy a Turnstile token from the captcha solver when the challenge is interactive.
402	const tCh0 = performance.now();	437	const tCh0 = performance.now();
403	let challengeSolved = false;	438	let challengeSolved = false;
		439	+ let captcha: CaptchaOutcome \| null = null;
404	let status = response?.status() ?? 200;	440	let status = response?.status() ?? 200;
405	let headers = lower(response?.headers() ?? {});	441	let headers = lower(response?.headers() ?? {});
406	let html = await page.content().catch(() => "");	442	let html = await page.content().catch(() => "");
407	let verdict = looksBlocked(status, html, headers);	443	let verdict = looksBlocked(status, html, headers);
408	if (verdict.blocked && verdict.challenge !== false && req.javascript) {	444	if (verdict.blocked && verdict.challenge !== false && req.javascript) {
409	− const budget = Math.min(this.opts.challengeWaitMs, deadline - performance.now() - 1500);
410	const challengeStatus = status;	445	const challengeStatus = status;
411	− const until = performance.now() + Math.max(0, budget);	446	+ const solverAllowed = Boolean(this.solver) && req.solveCaptcha !== false;
		447	+ const loopStart = performance.now();
		448	+ let until = loopStart + Math.max(0, Math.min(this.opts.challengeWaitMs, deadline - performance.now() - 1500));
412	let clickedTurnstile = false;	449	let clickedTurnstile = false;
		450	+ let solveState: { started: boolean; done: boolean; sol: TurnstileSolution \| null } = { started: false, done: false, sol: null };
		451	+ let injected = false;
413	while (performance.now() < until) {	452	while (performance.now() < until) {
414	− await page.waitForTimeout(600).catch(() => {});	453	+ await page.waitForTimeout(500).catch(() => {});
415	− // Best-effort click on a Turnstile / hCaptcha checkbox when one is visible.	454	+ // Natural resolution: the challenge script redirects / replaces the document.
416	− if (!clickedTurnstile) {	455	+ const navResp = await page.waitForNavigation({ timeout: 900, waitUntil: "domcontentloaded" }).catch(() => null);
417	− clickedTurnstile = await this.tryClickChallenge(page);
418	− }
419	− const navResp = await page.waitForNavigation({ timeout: 1200, waitUntil: "domcontentloaded" }).catch(() => null);
420	if (navResp) {	456	if (navResp) {
421	response = navResp;	457	response = navResp;
422	status = navResp.status();	458	status = navResp.status();
				`@@ -430,12 +466,43 @@ export class BrowserPool {`
430	if (!navResp) status = 200; // the document was replaced in place	466	if (!navResp) status = 200; // the document was replaced in place
431	break;	467	break;
432	}	468	}
		469	+ // Captcha solver: once the widget parameters are visible, buy a token (in the background) and keep waiting.
		470	+ if (solverAllowed && !solveState.started && performance.now() - loopStart > 1200 && deadline - performance.now() > 15_000) {
		471	+ const params = await this.readTurnstileParams(page);
		472	+ if (params?.sitekey) {
		473	+ solveState.started = true;
		474	+ const websiteURL = page.url();
		475	+ this.solver!.solveTurnstile({ websiteURL, params, deadline: deadline - 4000 })
		476	+ .then((sol) => (solveState = { started: true, done: true, sol }))
		477	+ .catch(() => (solveState = { started: true, done: true, sol: null }));
		478	+ until = deadline - 2500; // wait for the solver instead of the short challenge budget
		479	+ }
		480	+ }
		481	+ if (solveState.done && !injected) {
		482	+ injected = true;
		483	+ const sol = solveState.sol;
		484	+ if (!sol) {
		485	+ until = Math.min(until, performance.now() + 2500);
		486	+ } else {
		487	+ captcha = { provider: "2captcha", solved: false, ms: sol.ms, costUsd: sol.costUsd };
		488	+ if (sol.userAgent && sol.userAgent !== userAgent) {
		489	+ this.lastSolverUa = sol.userAgent;
		490	+ await this.overrideUserAgent(context, page, sol.userAgent, extraHeaders["accept-language"]);
		491	+ }
		492	+ const how = await this.injectTurnstileToken(page, sol.token);
		493	+ this.opts.log.info(`turnstile token injected via ${how} on ${new URL(page.url()).hostname} (solver ua ${sol.userAgent ? (sol.userAgent === userAgent ? "same" : "override") : "none"})`);
		494	+ until = Math.min(deadline - 1500, performance.now() + 15_000);
		495	+ }
		496	+ }
		497	+ // Best-effort click on a Turnstile / hCaptcha checkbox when no solver is available.
		498	+ if (!solverAllowed && !clickedTurnstile) clickedTurnstile = await this.tryClickChallenge(page);
433	}	499	}
434	verdict = looksBlocked(status, html, headers);	500	verdict = looksBlocked(status, html, headers);
435	if (challengeSolved) {	501	if (challengeSolved) {
436	this.stats.challengesSolved++;	502	this.stats.challengesSolved++;
437	− headers = { ...headers, "x-fetcha-challenge": `solved:${challengeStatus}` };	503	+ headers = { ...headers, "x-fetcha-challenge": `solved:${challengeStatus}${captcha ? ":2captcha" : ""}` };
438	verdict = { blocked: false };	504	verdict = { blocked: false };
		505	+ if (captcha) captcha.solved = true;
439	}	506	}
440	}	507	}
441	const challengeMs = Math.round(performance.now() - tCh0);	508	const challengeMs = Math.round(performance.now() - tCh0);
				`@@ -496,6 +563,7 @@ export class BrowserPool {`
496	screenshot,	563	screenshot,
497	challengeSolved,	564	challengeSolved,
498	block: finalVerdict,	565	block: finalVerdict,
		566	+ captcha,
499	};	567	};
500	} catch (e) {	568	} catch (e) {
501	this.stats.failures++;	569	this.stats.failures++;
				`@@ -507,6 +575,87 @@ export class BrowserPool {`
507	}	575	}
508	}	576	}
509		577
		578	+ /** Read the intercepted `turnstile.render` parameters, or fall back to a widget's data-* attributes. */
		579	+ private async readTurnstileParams(page: Page): Promise<TurnstileParams \| null> {
		580	+ try {
		581	+ return await page.evaluate((key) => {
		582	+ const w = window as unknown as Record<string, { params?: TurnstileParams \| null; callback?: unknown } \| undefined>;
		583	+ const store = w[key];
		584	+ if (store?.params?.sitekey) return { ...store.params, hasCallback: typeof store.callback === "function" } as TurnstileParams;
		585	+ const el = document.querySelector(".cf-turnstile[data-sitekey], [data-sitekey][data-callback], #cf-chl-widget-container [data-sitekey]") as HTMLElement \| null;
		586	+ if (el?.dataset.sitekey) return { sitekey: el.dataset.sitekey, action: el.dataset.action ?? null, cData: el.dataset.cdata ?? null, chlPageData: null, hasCallback: false } as TurnstileParams;
		587	+ return null;
		588	+ }, TURNSTILE_HOOK_KEY, undefined, false); // Patchright: 4th arg = isolatedContext → false = page's main world
		589	+ } catch {
		590	+ return null;
		591	+ }
		592	+ }
		593	+
		594	+ /** Hand the solved token to the widget's callback (challenge pages) or to the form field (standalone widgets). */
		595	+ private async injectTurnstileToken(page: Page, token: string): Promise<string> {
		596	+ try {
		597	+ return await page.evaluate(
		598	+ ({ key, token }) => {
		599	+ const w = window as unknown as Record<string, { callback?: (t: string) => void } \| undefined>;
		600	+ const store = w[key];
		601	+ if (store && typeof store.callback === "function") {
		602	+ store.callback(token);
		603	+ return "callback";
		604	+ }
		605	+ const input = document.querySelector('input[name="cf-turnstile-response"]') as HTMLInputElement \| null;
		606	+ if (input) {
		607	+ input.value = token;
		608	+ const form = input.closest("form");
		609	+ const cbName = (document.querySelector(".cf-turnstile[data-callback]") as HTMLElement \| null)?.dataset.callback;
		610	+ const cb = cbName ? (window as unknown as Record<string, unknown>)[cbName] : null;
		611	+ if (typeof cb === "function") (cb as (t: string) => void)(token);
		612	+ else if (form) (form as HTMLFormElement & { requestSubmit?: () => void }).requestSubmit ? (form as HTMLFormElement).requestSubmit() : form.submit();
		613	+ return "form";
		614	+ }
		615	+ return "none";
		616	+ },
		617	+ { key: TURNSTILE_HOOK_KEY, token },
		618	+ undefined,
		619	+ false, // main world: the callback captured by the init script lives there
		620	+ );
		621	+ } catch (e) {
		622	+ return `error:${(e as Error).message.split("\n")[0]}`;
		623	+ }
		624	+ }
		625	+
		626	+ /** Adopt the User-Agent the token was produced with (Cloudflare binds tokens to the UA). */
		627	+ private async overrideUserAgent(context: BrowserContext, page: Page, userAgent: string, acceptLanguage?: string): Promise<void> {
		628	+ try {
		629	+ const cdp = await context.newCDPSession(page);
		630	+ const major = /Chrome\/(\d+)/.exec(userAgent)?.[1] ?? "140";
		631	+ const platform = /Windows/.test(userAgent) ? "Windows" : /Macintosh/.test(userAgent) ? "macOS" : /Android/.test(userAgent) ? "Android" : "Linux";
		632	+ const brands = [
		633	+ { brand: "Chromium", version: major },
		634	+ { brand: /Edg\//.test(userAgent) ? "Microsoft Edge" : "Google Chrome", version: major },
		635	+ { brand: "Not_A Brand", version: "24" },
		636	+ ];
		637	+ await cdp.send("Emulation.setUserAgentOverride", {
		638	+ userAgent,
		639	+ acceptLanguage,
		640	+ platform: platform === "Windows" ? "Win32" : platform === "macOS" ? "MacIntel" : platform === "Android" ? "Linux armv81" : "Linux x86_64",
		641	+ userAgentMetadata: {
		642	+ brands,
		643	+ fullVersionList: brands.map((b) => ({ brand: b.brand, version: b.version === "24" ? "24.0.0.0" : `${b.version}.0.0.0` })),
		644	+ platform,
		645	+ platformVersion: platform === "Windows" ? "15.0.0" : platform === "macOS" ? "14.6.1" : platform === "Android" ? "15.0.0" : "6.8.0",
		646	+ architecture: platform === "Android" ? "" : "x86",
		647	+ model: "",
		648	+ mobile: /Mobile/.test(userAgent),
		649	+ bitness: "64",
		650	+ wow64: false,
		651	+ },
		652	+ });
		653	+ await cdp.detach().catch(() => {});
		654	+ } catch (e) {
		655	+ this.opts.log.warn(`user-agent override failed: ${(e as Error).message.split("\n")[0]}`);
		656	+ }
		657	+ }
		658	+
510	private async tryClickChallenge(page: Page): Promise<boolean> {	659	private async tryClickChallenge(page: Page): Promise<boolean> {
511	try {	660	try {
512	for (const frame of page.frames()) {	661	for (const frame of page.frames()) {

modified packages/core/src/schema.ts +4 −0

				`@@ -48,6 +48,8 @@ export const fetchRequestSchema = z`
48	block_resources: z.boolean().default(true),	48	block_resources: z.boolean().default(true),
49	/** Browser: return a PNG screenshot (base64) in `screenshot`. */	49	/** Browser: return a PNG screenshot (base64) in `screenshot`. */
50	screenshot: z.boolean().default(false),	50	screenshot: z.boolean().default(false),
		51	+ /** Browser: use the captcha solver on Cloudflare Turnstile challenges when the platform has one configured (default true). */
		52	+ solve_captcha: z.boolean().default(true),
51	/** Include the list of hyperlinks found in the page (`links`). */	53	/** Include the list of hyperlinks found in the page (`links`). */
52	links: z.boolean().default(false),	54	links: z.boolean().default(false),
53	/** Referer strategy: "auto" (none on first try, search-engine referer on retries), "none", or a literal URL. */	55	/** Referer strategy: "auto" (none on first try, search-engine referer on retries), "none", or a literal URL. */
				`@@ -134,6 +136,8 @@ export interface FetchMetadata {`
134	country: string \| null;	136	country: string \| null;
135	outcome: string;	137	outcome: string;
136	block_reason?: string \| null;	138	block_reason?: string \| null;
		139	+ /** True when a captcha solver token was needed to pass this attempt. */
		140	+ captcha_solved?: boolean;
137	status: number \| null;	141	status: number \| null;
138	duration_ms: number;	142	duration_ms: number;
139	error?: string;	143	error?: string;

modified packages/core/src/text.ts +13 −9

				`@@ -154,7 +154,7 @@ export function looksBlocked(status: number, body: string, headers: Record<strin`
154	if (h["x-datadome"] \|\| h["x-dd-b"] \|\| /datadome/i.test(h["set-cookie"] ?? "")) {	154	if (h["x-datadome"] \|\| h["x-dd-b"] \|\| /datadome/i.test(h["set-cookie"] ?? "")) {
155	if (status === 403 \|\| status === 401 \|\| status === 429 \|\| /captcha-delivery\|geo\.captcha\|dd\.js\|DataDome/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true, retryAfterMs };	155	if (status === 403 \|\| status === 401 \|\| status === 429 \|\| /captcha-delivery\|geo\.captcha\|dd\.js\|DataDome/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true, retryAfterMs };
156	}	156	}
157	− if (h["x-kpsdk-ct"] \|\| h["x-kpsdk-c"] \|\| /kpsdk\|ips\.js/i.test(head) && status >= 400) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true, retryAfterMs };	157	+ if (h["x-kpsdk-ct"] \|\| h["x-kpsdk-c"] \|\| (/x-kpsdk\|kpsdk-/i.test(head) && status >= 400)) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true, retryAfterMs };
158	if (h["x-amzn-waf-action"] === "challenge" \|\| h["x-amzn-waf-action"] === "captcha") return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true, retryAfterMs };	158	if (h["x-amzn-waf-action"] === "challenge" \|\| h["x-amzn-waf-action"] === "captcha") return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true, retryAfterMs };
159	if (h["x-vercel-mitigated"] === "challenge" \|\| h["x-vercel-protection-bypass"] !== undefined && status === 403) return { blocked: true, reason: "anti_bot", vendor: "vercel", challenge: true, retryAfterMs };	159	if (h["x-vercel-mitigated"] === "challenge" \|\| h["x-vercel-protection-bypass"] !== undefined && status === 403) return { blocked: true, reason: "anti_bot", vendor: "vercel", challenge: true, retryAfterMs };
160	if (status === 403 && (h["x-iinfo"] \|\| /incap_ses\|visid_incap/i.test(h["set-cookie"] ?? ""))) return { blocked: true, reason: "waf", vendor: "imperva", challenge: /_Incapsula_Resource/i.test(head), retryAfterMs };	160	if (status === 403 && (h["x-iinfo"] \|\| /incap_ses\|visid_incap/i.test(h["set-cookie"] ?? ""))) return { blocked: true, reason: "waf", vendor: "imperva", challenge: /_Incapsula_Resource/i.test(head), retryAfterMs };
				`@@ -176,20 +176,24 @@ export function looksBlocked(status: number, body: string, headers: Record<strin`
176	if (status === 202 && /datadome\|captcha-delivery/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };	176	if (status === 202 && /datadome\|captcha-delivery/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };
177		177
178	// --- body-level signals (only meaningful for HTML) -------------------------	178	// --- body-level signals (only meaningful for HTML) -------------------------
		179	+ // Vendor beacons (Cloudflare JSD, DataDome tags.js, PerimeterX, Imperva, Kasada ips.js, AWS WAF
		180	+ // challenge.js) are present on EVERY page of a protected site, so on a 2xx only interstitial-specific
		181	+ // markers count; on 4xx/5xx the mere presence of a vendor script is enough.
179	if (!isHtml) return { blocked: false };	182	if (!isHtml) return { blocked: false };
180	− if (/cf-chl\|challenge-platform\|__cf_chl\|cf_chl_opt\|<title>\s*Just a moment\|cf-turnstile\|turnstile\.js\|challenges\.cloudflare\.com/i.test(head)) {	183	+ const denied = status >= 400;
		184	+ if (/cf-chl-bypass\|__cf_chl_f_tk\|__cf_chl_rt_tk\|__cf_chl_tk\|cf_chl_opt\|<title>\s*Just a moment\|cf-chl-widget\|challenge-error-text\|cf-challenge-running\|id="challenge-(running\|stage\|form)"\|Checking your browser before accessing\|Verify you are human by completing the action/i.test(head) \|\| (denied && /cf-chl\|challenge-platform\|cf-turnstile\|challenges\.cloudflare\.com/i.test(head))) {
181	return { blocked: true, reason: "cloudflare_challenge", vendor: "cloudflare", challenge: true, retryAfterMs };	185	return { blocked: true, reason: "cloudflare_challenge", vendor: "cloudflare", challenge: true, retryAfterMs };
182	}	186	}
183	− if (/geo\.captcha-delivery\.com\|dd\.js\|datadome\|<title>\s*DataDome/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };	187	+ if (/geo\.captcha-delivery\.com\|<title>\s*DataDome\|dd\.js\?\|ddCaptcha/i.test(head) \|\| (denied && /datadome/i.test(head))) return { blocked: true, reason: "anti_bot", vendor: "datadome", challenge: true };
184	− if (/px-captcha\|_pxhd\|_pxvid\|perimeterx\|human-challenge\|<title>\s*Access to this page has been denied/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "perimeterx", challenge: true };	188	+ if (/px-captcha\|human-challenge\|<title>\s*Access to this page has been denied/i.test(head) \|\| (denied && /perimeterx\|_pxhd\|_pxvid/i.test(head))) return { blocked: true, reason: "anti_bot", vendor: "perimeterx", challenge: true };
185	if (/Reference #\d\|Reference #\d+\.[0-9a-f]+\.\d+\.[0-9a-f]+\|errors\.edgesuite\.net\|akamai\.com\/us\/en\/policies/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "akamai", challenge: false };	189	if (/Reference #\d\|Reference #\d+\.[0-9a-f]+\.\d+\.[0-9a-f]+\|errors\.edgesuite\.net\|akamai\.com\/us\/en\/policies/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "akamai", challenge: false };
186	− if (/_Incapsula_Resource\|Incapsula incident\|Request unsuccessful\. Incapsula/i.test(head)) return { blocked: true, reason: "waf", vendor: "imperva", challenge: true };	190	+ if (/Incapsula incident\|Request unsuccessful\. Incapsula/i.test(head) \|\| (denied && /_Incapsula_Resource/i.test(head))) return { blocked: true, reason: "waf", vendor: "imperva", challenge: true };
187	− if (/awswaf\|aws-waf-token\|challenge\.js\?\|<title>\s*Human Verification/i.test(head)) return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true };	191	+ if (/<title>\sHuman Verification\|aws-waf-token.captcha\|awswaf.*captcha/i.test(head) \|\| (denied && /awswaf\|challenge\.js\?/i.test(head))) return { blocked: true, reason: "waf", vendor: "aws_waf", challenge: true };
188	− if (/kpsdk\|ips\.js\|<title>\s*Kasada/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true };	192	+ if (/<title>\s*Kasada/i.test(head) \|\| (denied && /kpsdk\|ips\.js/i.test(head))) return { blocked: true, reason: "anti_bot", vendor: "kasada", challenge: true };
189	if (/Pardon Our Interruption\|distil_r_captcha\|distil_referrer/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "distil", challenge: true };	193	if (/Pardon Our Interruption\|distil_r_captcha\|distil_referrer/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "distil", challenge: true };
190	if (/Vercel Security Checkpoint\|_vercel_challenge/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "vercel", challenge: true };	194	if (/Vercel Security Checkpoint\|_vercel_challenge/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "vercel", challenge: true };
191	− if (/shape-security\|_imp_apg_r_\|<title>\s*Blocked by Shape/i.test(head)) return { blocked: true, reason: "anti_bot", vendor: "shape" };	195	+ if (/<title>\s*Blocked by Shape\|_imp_apg_r_/i.test(head) \|\| (denied && /shape-security/i.test(head))) return { blocked: true, reason: "anti_bot", vendor: "shape" };
192	− if (/sucuri\.net\|Sucuri WebSite Firewall/i.test(head)) return { blocked: true, reason: "waf", vendor: "sucuri" };	196	+ if (/Sucuri WebSite Firewall/i.test(head) \|\| (denied && /sucuri\.net/i.test(head))) return { blocked: true, reason: "waf", vendor: "sucuri" };
193	if (/www\.google\.com\/sorry\/\|Our systems have detected unusual traffic/i.test(head)) return { blocked: true, reason: "captcha", vendor: "google", challenge: false };	197	if (/www\.google\.com\/sorry\/\|Our systems have detected unusual traffic/i.test(head)) return { blocked: true, reason: "captcha", vendor: "google", challenge: false };
194	if (/recaptcha\/api\.js\|g-recaptcha\|hcaptcha\.com\|h-captcha\|arkoselabs\|funcaptcha\|<title>\s[^<]captcha/i.test(head) && /verify\|robot\|human\|unusual traffic\|security check\|confirm you are/i.test(head)) {	198	if (/recaptcha\/api\.js\|g-recaptcha\|hcaptcha\.com\|h-captcha\|arkoselabs\|funcaptcha\|<title>\s[^<]captcha/i.test(head) && /verify\|robot\|human\|unusual traffic\|security check\|confirm you are/i.test(head)) {
195	return { blocked: true, reason: "captcha", vendor: vendorFrom(server, head), challenge: false };	199	return { blocked: true, reason: "captcha", vendor: vendorFrom(server, head), challenge: false };

modified packages/core/test/ssrf.test.ts +5 −0

				`@@ -84,6 +84,11 @@ describe("block detection & text", () => {`
84	expect(looksBlocked(429, "", { "retry-after": "3" })).toMatchObject({ blocked: true, reason: "rate_limited", retryAfterMs: 3000 });	84	expect(looksBlocked(429, "", { "retry-after": "3" })).toMatchObject({ blocked: true, reason: "rate_limited", retryAfterMs: 3000 });
85	expect(looksBlocked(200, "<html><head><script src='/x.js'></script></head><body></body></html>", {})).toMatchObject({ blocked: true, reason: "empty_html" });	85	expect(looksBlocked(200, "<html><head><script src='/x.js'></script></head><body></body></html>", {})).toMatchObject({ blocked: true, reason: "empty_html" });
86	expect(looksBlocked(200, "<html><head><title>DataDome</title></head><body><script src='https://geo.captcha-delivery.com/captcha/'></script></body></html>", {})).toMatchObject({ blocked: true, vendor: "datadome" });	86	expect(looksBlocked(200, "<html><head><title>DataDome</title></head><body><script src='https://geo.captcha-delivery.com/captcha/'></script></body></html>", {})).toMatchObject({ blocked: true, vendor: "datadome" });
		87	+ // Vendor beacons on a normal page must not count as blocks.
		88	+ const normal = `<html><head><title>Cloudflare Challenge - Example</title><script src="https://js.datadome.co/tags.js"></script><script src="/challenge.js?x"></script><script src="/ips.js"></script></head><body><main><h1>You bypassed the Cloudflare challenge!</h1><p>${"content ".repeat(200)}</p></main><script src="/cdn-cgi/challenge-platform/scripts/jsd/main.js"></script><script>window._pxvid='x'; _Incapsula_Resource='y';</script></body></html>`;
		89	+ expect(looksBlocked(200, normal, { server: "cloudflare" }).blocked).toBe(false);
		90	+ expect(looksBlocked(403, normal, { server: "cloudflare" })).toMatchObject({ blocked: true, vendor: "cloudflare" });
		91	+ expect(looksBlocked(200, "<html><head><title>Just a moment...</title><script src='/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1'></script></head><body><div id='challenge-running'></div></body></html>", {})).toMatchObject({ blocked: true, reason: "cloudflare_challenge" });
87	});	92	});
88	it("extracts readable text", () => {	93	it("extracts readable text", () => {
89	expect(htmlToText("<h1>Hi</h1><script>x()</script><p>There & back</p>")).toBe("Hi\nThere & back");	94	expect(htmlToText("<h1>Hi</h1><script>x()</script><p>There & back</p>")).toBe("Hi\nThere & back");

modified packages/routing/src/executor.ts +9 −1

				`@@ -51,6 +51,8 @@ export interface AttemptRecord {`
51	blockReason: string \| null;	51	blockReason: string \| null;
52	blockVendor: string \| null;	52	blockVendor: string \| null;
53	profileId: string \| null;	53	profileId: string \| null;
		54	+ /** A captcha solver token was purchased and the attempt passed thanks to it. */
		55	+ captchaSolved: boolean;
54	durationMs: number;	56	durationMs: number;
55	bytesIn: number;	57	bytesIn: number;
56	bytesOut: number;	58	bytesOut: number;
				`@@ -200,6 +202,7 @@ export class FetchExecutor {`
200		202
201	let res: FinalResponse \| null = null;	203	let res: FinalResponse \| null = null;
202	let verdict: BlockVerdict = { blocked: false };	204	let verdict: BlockVerdict = { blocked: false };
		205	+ let captchaOutcome: RenderResult["captcha"] = null;
203	let record: AttemptRecord;	206	let record: AttemptRecord;
204	try {	207	try {
205	let bytesIn = 0;	208	let bytesIn = 0;
				`@@ -223,6 +226,7 @@ export class FetchExecutor {`
223	javascript: request.javascript !== false,	226	javascript: request.javascript !== false,
224	blockResources: request.block_resources,	227	blockResources: request.block_resources,
225	screenshot: request.screenshot,	228	screenshot: request.screenshot,
		229	+ solveCaptcha: request.solve_captcha,
226	maxResponseBytes: maxBytes,	230	maxResponseBytes: maxBytes,
227	onRedirect: async (next) => {	231	onRedirect: async (next) => {
228	await assertUrlAllowed(next);	232	await assertUrlAllowed(next);
				`@@ -241,6 +245,7 @@ export class FetchExecutor {`
241	};	245	};
242	verdict = render.block;	246	verdict = render.block;
243	if (render.challengeSolved) browserRequired = true;	247	if (render.challengeSolved) browserRequired = true;
		248	+ captchaOutcome = render.captcha;
244	} else {	249	} else {
245	const pr = await cand.provider.fetch({	250	const pr = await cand.provider.fetch({
246	requestId,	251	requestId,
				`@@ -273,7 +278,7 @@ export class FetchExecutor {`
273	}	278	}
274	const durationMs = Math.round(performance.now() - t0);	279	const durationMs = Math.round(performance.now() - t0);
275	const transient = !verdict.blocked && isTransientStatus(res.status);	280	const transient = !verdict.blocked && isTransientStatus(res.status);
276	− const cost = cand.provider.estimateCost(cand.network, bytesIn + bytesOut);	281	+ const cost = cand.provider.estimateCost(cand.network, bytesIn + bytesOut) + (captchaOutcome?.costUsd ?? 0);
277	record = {	282	record = {
278	attemptId,	283	attemptId,
279	attemptNo: i + 1,	284	attemptNo: i + 1,
				`@@ -289,6 +294,7 @@ export class FetchExecutor {`
289	blockReason: verdict.reason ?? null,	294	blockReason: verdict.reason ?? null,
290	blockVendor: verdict.vendor ?? null,	295	blockVendor: verdict.vendor ?? null,
291	profileId: profile.id,	296	profileId: profile.id,
		297	+ captchaSolved: Boolean(captchaOutcome?.solved),
292	durationMs,	298	durationMs,
293	bytesIn,	299	bytesIn,
294	bytesOut,	300	bytesOut,
				`@@ -420,6 +426,7 @@ export class FetchExecutor {`
420	blockReason: null,	426	blockReason: null,
421	blockVendor: null,	427	blockVendor: null,
422	profileId,	428	profileId,
		429	+ captchaSolved: false,
423	durationMs,	430	durationMs,
424	bytesIn: 0,	431	bytesIn: 0,
425	bytesOut: 0,	432	bytesOut: 0,
				`@@ -507,6 +514,7 @@ export class FetchExecutor {`
507	country: a.country,	514	country: a.country,
508	outcome: a.outcome,	515	outcome: a.outcome,
509	block_reason: a.blockReason,	516	block_reason: a.blockReason,
		517	+ ...(a.captchaSolved ? { captcha_solved: true } : {}),
510	status: a.httpStatus,	518	status: a.httpStatus,
511	duration_ms: a.durationMs,	519	duration_ms: a.durationMs,
512	...(a.errorDetail && ctx.providerVisibility ? { error: scrubProviderText(a.errorDetail) } : {}),	520	...(a.errorDetail && ctx.providerVisibility ? { error: scrubProviderText(a.errorDetail) } : {}),

modified packages/sdk/src/index.ts +2 −0

				`@@ -51,6 +51,8 @@ export interface FetchOptions {`
51	block_resources?: boolean;	51	block_resources?: boolean;
52	/** Browser: return a PNG screenshot (base64) in `screenshot`. */	52	/** Browser: return a PNG screenshot (base64) in `screenshot`. */
53	screenshot?: boolean;	53	screenshot?: boolean;
		54	+ /** Browser: solve Cloudflare Turnstile challenges with the platform's captcha solver (default true). */
		55	+ solve_captcha?: boolean;
54	/** Return `links[]` (all hyperlinks, absolute). */	56	/** Return `links[]` (all hyperlinks, absolute). */
55	links?: boolean;	57	links?: boolean;
56	/** Referer strategy: "auto" (none first, search-engine referer on retries), "none", or a literal URL. */	58	/** Referer strategy: "auto" (none first, search-engine referer on retries), "none", or a literal URL. */

modified sdk-python/fetcha/__init__.py +1 −1

				`@@ -367,7 +367,7 @@ class Fetcha:`
367	def fetch(self, url: str, **options: Any) -> FetchResult:	367	def fetch(self, url: str, **options: Any) -> FetchResult:
368	"""POST /v1/fetch. Options: method, headers, cookies, body, timeout, country, region, city,	368	"""POST /v1/fetch. Options: method, headers, cookies, body, timeout, country, region, city,
369	network, session, browser, browser_fallback, wait_for, wait_ms, wait_until, javascript,	369	network, session, browser, browser_fallback, wait_for, wait_ms, wait_until, javascript,
370	− block_resources, screenshot, links, referer, device, locale, format (html\|text\|markdown\|json\|raw),	370	+ block_resources, screenshot, solve_captcha, links, referer, device, locale, format (html\|text\|markdown\|json\|raw),
371	follow_redirects, max_redirects, max_response_bytes, retries, debug."""	371	follow_redirects, max_redirects, max_response_bytes, retries, debug."""
372	payload = {"url": url, **_clean(options)}	372	payload = {"url": url, **_clean(options)}
373	return FetchResult.from_dict(self._request("POST", "/v1/fetch", json=payload))	373	return FetchResult.from_dict(self._request("POST", "/v1/fetch", json=payload))
374		374

spb/fetcha

v0.2.1 — 2captcha Turnstile solver (onload-hook interception, UA adoption via CDP), solve_captcha option, real Chrome channel + headful mode, tighter 2xx block signatures, Patchright main-world evaluates

Theme