SPB Git forge
spboucher.ai ↗

spb/fetcha

Public
11commits 1branches 0releases
1.5 MBsize
maindefault branch
16 days agolast push
TypeScript 97.5% SQL 1.4% Python 0.8%
4.8 KB
View source Raw Blame History

# Fetcha — repository guide

Fetcha (www.fetcha.co) is "Intelligent Web Access Infrastructure": one API (POST /v1/fetch) that routes web requests across upstream proxy networks (Oxylabs, Decodo, SOAX — never exposed to customers) with smart routing, retries, sticky sessions, a managed headless browser (browser: true + automatic escalation), crawl/map jobs, usage metering and a full dashboard. Private platform (v0.2, 2026-09-08): a single unlimited plan, invitation-only signup (signup_allowlist, managed from /admin/access), ADMIN_EMAILS become admins. No billing.

# Layout

  • apps/web — Next.js 16 (marketing site, docs, auth, customer dashboard, admin). Port 8220. Proxies /v1/* to the API.
  • apps/api — Fastify API service (public /v1/*, /health, /ready, internal /internal/* for the dashboard). Port 8221.
  • packages/core — zod schemas, error codes, ids/api-key hashing, SSRF policy, geo model, plan limits.
  • packages/db — Drizzle schema + migrations (drizzle/), pnpm db:generate|migrate|seed.
  • packages/providers — ProxyProvider interface + adapters (oxylabs, decodo, soax, direct), http.ts (undici, HTTP/2, browser-ordered headers, Chrome/Firefox/Safari TLS cipher lists, h2→h1 fallback), fingerprint.ts (header profiles), cookies.ts (jar across hops / sessions). Provider code lives ONLY here.
  • packages/routing — circuit breaker, routing score, FetchExecutor (attempt loop, fingerprint rotation, jittered backoff, block detection, HTTP → browser escalation).
  • packages/browser — BrowserPool (Patchright = patched Playwright, Chrome/Chromium, per-context upstream proxy, stealth init script, challenge wait, 2captcha Turnstile solver in captcha.ts). Install the browser once: pnpm --filter @fetcha/browser install-browser.
  • packages/core/src/markdown.ts — HTML → Markdown / main text, page metadata + links, URL normalisation, glob/regex matchers; robots.ts — robots.txt + sitemap parsing.
  • apps/api/src/services/crawl.ts — durable crawl jobs (crawl_jobs/crawl_pages, in-process worker, resume after restart) and the sync /v1/map.
  • packages/email — Resend abstraction + React Email templates. packages/sdk (JS), sdk-python/ (Python).
  • docs/AGENT-BRIEF.md — product/engineering rules used when generating UI.

# Rules

  • Never expose provider names, credentials or provider error text to customers. Public debug output uses aliases (network-a/b/c).
  • Every external URL goes through assertUrlAllowed() (SSRF) — including redirects.
  • Business logic in server actions / apps/api / packages, not in React components.
  • API keys: shown once, SHA-256 stored. Request IDs req_… on every response (X-Fetcha-Request-ID).
  • Not implemented yet (say so, don't fake): scripted browser actions (POST /v1/browser), /v1/extract, teams, webhook delivery (except the crawl webhook_url callback), OAuth, 2FA. Billing/Stripe is intentionally absent (private platform).
  • Plans: always normalizePlan(org.plan); never reintroduce tiers. Signup must stay allowlist-gated (Better Auth user.create.before hook in apps/web/src/lib/auth.ts).
  • Anti-bot: interactive Cloudflare Turnstile challenges are solved with 2captcha (TWOCAPTCHA_API_KEY, packages/browser/src/captcha.ts): the init script wraps the api.js ?onload= callback to capture turnstile.render params (sitekey, action, cData, chlPageData, callback), a token is bought (~$0.0015, 5–10 s) and injected through the callback — never switch the page UA to the solver's (Cloudflare then rejects the token). Patchright runs init scripts in the main world but page.evaluate in an isolated world — pass undefined, false (4th arg) to read main-world state. looksBlocked() on 2xx must only match interstitial markers (vendor beacons like challenge-platform/scripts/jsd, tags.js, ips.js are on every protected page). Prod browser = Google Chrome stable (brew cask on M3U96a), FETCHA_BROWSER_HEADLESS=0 (real off-screen window in the node's GUI session).

# Dev

cp .env.example .env (fill provider creds + RESEND_API_KEY), createdb fetcha, pnpm db:migrate && pnpm db:seed, pnpm dev:api + pnpm dev:web. Tests: pnpm test. Dev seed: cd scripts && npx tsx dev-seed.ts you@example.com prints an API key.

# Deploy (MacLustr)

Deployed by mld (gateway M1M32) on M3U96a: web 8220 + api 8221, Postgres 17 fetcha + Redis local, ngrok www.fetcha.co. Manifest: deploy/fetcha.mld.json → M1M32:~/dispatch/apps/fetcha.json. See deploy/README.md. Post-sync hooks: pnpm install, db:migrate + db:seed (with ADMIN_EMAILS), patchright install chromium, next build. Env knobs: FETCHA_BROWSER_ENABLED, FETCHA_BROWSER_CONCURRENCY, FETCHA_BROWSER_CHANNEL, FETCHA_BROWSER_HEADLESS, FETCHA_HTTP2, FETCHA_CRAWL_PARALLEL_JOBS. The browser_enabled feature flag (admin → Flags) is the global kill switch for browser mode.