# Fetcha — repository guide

Fetcha (www.fetcha.co) is "Intelligent Web Access Infrastructure": one API (`POST /v1/fetch`) that routes web requests
across upstream proxy networks (Oxylabs, Decodo, SOAX — never exposed to customers) with smart routing, retries,
sticky sessions, a managed headless browser (`browser: true` + automatic escalation), crawl/map jobs, usage metering
and a full dashboard. **Private platform (v0.2, 2026-09-08)**: a single `unlimited` plan, invitation-only signup
(`signup_allowlist`, managed from `/admin/access`), `ADMIN_EMAILS` become admins. No billing.

## Layout
- `apps/web` — Next.js 16 (marketing site, docs, auth, customer dashboard, admin). Port 8220. Proxies `/v1/*` to the API.
- `apps/api` — Fastify API service (public `/v1/*`, `/health`, `/ready`, internal `/internal/*` for the dashboard). Port 8221.
- `packages/core` — zod schemas, error codes, ids/api-key hashing, SSRF policy, geo model, plan limits.
- `packages/db` — Drizzle schema + migrations (`drizzle/`), `pnpm db:generate|migrate|seed`.
- `packages/providers` — `ProxyProvider` interface + adapters (oxylabs, decodo, soax, direct), `http.ts` (undici, HTTP/2, browser-ordered headers, Chrome/Firefox/Safari TLS cipher lists, h2→h1 fallback), `fingerprint.ts` (header profiles), `cookies.ts` (jar across hops / sessions). Provider code lives ONLY here.
- `packages/routing` — circuit breaker, routing score, `FetchExecutor` (attempt loop, fingerprint rotation, jittered backoff, block detection, HTTP → browser escalation).
- `packages/browser` — `BrowserPool` (Patchright = patched Playwright, Chrome/Chromium, per-context upstream proxy, stealth init script, challenge wait, 2captcha Turnstile solver in `captcha.ts`). Install the browser once: `pnpm --filter @fetcha/browser install-browser`.
- `packages/core/src/markdown.ts` — HTML → Markdown / main text, page metadata + links, URL normalisation, glob/regex matchers; `robots.ts` — robots.txt + sitemap parsing.
- `apps/api/src/services/crawl.ts` — durable crawl jobs (`crawl_jobs`/`crawl_pages`, in-process worker, resume after restart) and the sync `/v1/map`.
- `packages/email` — Resend abstraction + React Email templates. `packages/sdk` (JS), `sdk-python/` (Python).
- `docs/AGENT-BRIEF.md` — product/engineering rules used when generating UI.

## Rules
- Never expose provider names, credentials or provider error text to customers. Public debug output uses aliases (network-a/b/c).
- Every external URL goes through `assertUrlAllowed()` (SSRF) — including redirects.
- Business logic in server actions / `apps/api` / packages, not in React components.
- API keys: shown once, SHA-256 stored. Request IDs `req_…` on every response (`X-Fetcha-Request-ID`).
- Not implemented yet (say so, don't fake): scripted browser actions (`POST /v1/browser`), `/v1/extract`, teams, webhook delivery (except the crawl `webhook_url` callback), OAuth, 2FA. Billing/Stripe is intentionally absent (private platform).
- Plans: always `normalizePlan(org.plan)`; never reintroduce tiers. Signup must stay allowlist-gated (Better Auth `user.create.before` hook in `apps/web/src/lib/auth.ts`).
- Anti-bot: interactive Cloudflare Turnstile challenges are solved with 2captcha (`TWOCAPTCHA_API_KEY`, `packages/browser/src/captcha.ts`): the init script wraps the api.js `?onload=` callback to capture `turnstile.render` params (sitekey, action, cData, chlPageData, callback), a token is bought (~$0.0015, 5–10 s) and injected through the callback — never switch the page UA to the solver's (Cloudflare then rejects the token). Patchright runs init scripts in the main world but `page.evaluate` in an isolated world — pass `undefined, false` (4th arg) to read main-world state. `looksBlocked()` on 2xx must only match interstitial markers (vendor beacons like `challenge-platform/scripts/jsd`, `tags.js`, `ips.js` are on every protected page). Prod browser = Google Chrome stable (brew cask on M3U96a), `FETCHA_BROWSER_HEADLESS=0` (real off-screen window in the node's GUI session).

## Dev
`cp .env.example .env` (fill provider creds + RESEND_API_KEY), `createdb fetcha`, `pnpm db:migrate && pnpm db:seed`,
`pnpm dev:api` + `pnpm dev:web`. Tests: `pnpm test`. Dev seed: `cd scripts && npx tsx dev-seed.ts you@example.com` prints an API key.

## Deploy (MacLustr)
Deployed by `mld` (gateway M1M32) on M3U96a: web 8220 + api 8221, Postgres 17 `fetcha` + Redis local, ngrok `www.fetcha.co`.
Manifest: `deploy/fetcha.mld.json` → `M1M32:~/dispatch/apps/fetcha.json`. See `deploy/README.md`. Post-sync hooks: `pnpm install`,
`db:migrate` + `db:seed` (with `ADMIN_EMAILS`), `patchright install chromium`, `next build`. Env knobs: `FETCHA_BROWSER_ENABLED`,
`FETCHA_BROWSER_CONCURRENCY`, `FETCHA_BROWSER_CHANNEL`, `FETCHA_BROWSER_HEADLESS`, `FETCHA_HTTP2`, `FETCHA_CRAWL_PARALLEL_JOBS`.
The `browser_enabled` feature flag (admin → Flags) is the global kill switch for browser mode.