SPB Git forge
spboucher.ai ↗

spb/fetcha

Public
11commits 1branches 0releases
1.5 MBsize
maindefault branch
16 days agolast push
TypeScript 97.5% SQL 1.4% Python 0.8%
14.4 KB
View source Raw Blame History

# Fetcha — shared engineering brief for parallel agents

You are building part of Fetcha (fetcha.co), "Intelligent Web Access Infrastructure": one API that routes web requests across upstream proxy networks. Repo root: ~/Desktop/Projets/apps-web/fetcha (pnpm + Turborepo monorepo). The product spec is long; the essentials you need are here. Read the files you touch or depend on before writing.

# Non-negotiable rules

  1. Never expose upstream provider names (Oxylabs, Decodo, SOAX) anywhere customers can see: marketing, docs, dashboard, API responses, error messages. Customers see network classes: datacenter, residential, isp, mobile. Only the /admin area may show provider names.
  2. No fake functionality. Not available in V1 (say so explicitly, mark "coming soon" / "not yet available"): managed browser (browser: true → BROWSER_UNAVAILABLE), /v1/extract, /v1/browser, Stripe checkout, teams/invitations, webhooks delivery, OAuth login, 2FA/passkeys, datacenter/isp/mobile network classes (currently only residential is live; auto resolves to residential). Say what exists; keep the UI for future things honest ("Coming soon").
  3. Business logic never lives in React components. Use server components + server actions (src/actions/*.ts) or src/lib/queries/*.ts for data access. Client components only for interactivity.
  4. Security: never show plaintext API keys except at creation; redact Authorization/Cookie; never trust user HTML — render fetched HTML previews only inside a sandboxed <iframe sandbox="" srcDoc=…>.
  5. Every screen needs: strong hierarchy, loading state (loading.tsx or Suspense + Skeleton), empty state (EmptyState that teaches the next action), error handling, responsive layout, keyboard accessibility. Dark + light both first-class.
  6. Design direction: Cloudflare / Vercel / Linear / Stripe. Minimal, technical, generous whitespace, precise 13–15px type, tabular numbers for data (className="tabular font-mono"), tables/inline metrics over card grids. No gradients/neon/AI-SaaS clichés. Do not clone Scrapfly. Accent is blue (text-accent, bg-accent). Radii are small (rounded-md/lg).
  7. Do not edit files you don't own (listed per task). Do not modify packages/*, apps/api/*, globals.css, layout.tsx roots, components/ui/*, lib/* (except adding new files under src/lib/queries/ or src/lib/<yourarea>-*.ts). If you need a shared change, write a note at the end of your report instead.
  8. When done, run cd apps/web && npx tsc -p tsconfig.json --noEmit and fix errors in your files. Ignore errors from other agents' files. Also pnpm --filter @fetcha/web lint if quick. Do not run next build (another process will).

# Stack

Next.js 16 (App Router, src/app), React 19, TypeScript strict, Tailwind v4 (@theme tokens in src/app/globals.css), Radix primitives wrapped in src/components/ui/*, lucide-react icons, recharts (client only), framer-motion available, Drizzle ORM + Postgres via @fetcha/db, Better Auth (src/lib/auth.ts, client src/lib/auth-client.ts). Path alias @/ → apps/web/src/. Pages default to server components; add "use client" only when needed. Route segments: (marketing) public pages with nav/footer, (auth) login/signup, dashboard/*, admin/*. Dashboard pages must export const dynamic = "force-dynamic" (they read cookies/session).

# Design tokens (Tailwind classes)

Colors: bg-bg, bg-bg-subtle, bg-bg-muted, bg-bg-elevated, text-fg, text-fg-muted, text-fg-subtle, border-border, border-border-strong, bg-accent/text-accent/bg-accent-soft, text-success/bg-success-soft, warning, danger, info. Utilities: container-page (max-w-6xl centered), tabular, grid-bg, scrollbar-thin, animate-fade-in, animate-fade-up. Fonts: sans default; font-mono for ids, code, numbers.

# Shared components (@/components/ui/*) — use these, don't reinvent

  • button → Button (variant: default|primary|secondary|outline|ghost|danger|link; size: default|sm|xs|lg|icon|icon-sm; loading, asChild)
  • input → Input, Textarea, NativeSelect, inputClass
  • label → Label, Hint, FieldError, Field (grid gap wrapper)
  • card → Card, CardHeader, CardTitle, CardDescription, CardContent, CardFooter
  • badge → Badge (variant default|outline|accent|success|warning|danger|info|solid, dot), StatusBadge status="success|failed|pending|active|…"
  • table → Table, TableHeader, TableBody, TableRow, TableHead, TableCell, TableEmpty colSpan
  • tabs → Tabs, TabsList variant="pill"|"underline", TabsTrigger, TabsContent (client)
  • dialog → Dialog, DialogTrigger, DialogContent size="sm|md|lg|xl", DialogHeader, DialogTitle, DialogDescription, DialogFooter, DialogClose
  • select → Radix Select, SelectTrigger, SelectValue, SelectContent, SelectItem, SelectLabel, SelectGroup
  • switch → Switch; dropdown-menu → DropdownMenu*; tooltip → Tooltip*, SimpleTooltip content=
  • skeleton → Skeleton, TableSkeleton rows cols
  • empty-state → EmptyState icon title description action compact
  • copy-button → CopyButton value label?
  • code-block → CodeBlock code lang title? lineNumbers? maxHeight? copy? (lang: bash|json|javascript|typescript|python|go|php|ruby|java|csharp|html|text), InlineCode, highlight()
  • page-header → PageHeader title description actions eyebrow, SectionTitle right?
  • stat → Stat label value hint delta, StatGrid cols=2|3|4|5|6 (children are Stats; renders divided grid)
  • alert → Alert variant="info|success|warning|danger" title action
  • logo → Logo href size, LogoMark size
  • theme → ThemeToggle

# Server helpers (@/lib/*)

  • session.ts: getSession(), getUser(), requireUser(next?), requireAdmin(), isAdmin(user), getWorkspace() → { user, organization, role, projects, project /*current*/, isAdmin } (cached per request; redirects to /login if anon), requestMeta() → { ip, userAgent }, PROJECT_COOKIE.
  • api.ts (server-only): internalApi.playgroundFetch(projectId, userId, request), createSession, listSessions, closeSession, usage(projectId,userId), providers() (admin: provider list + health + circuits), probeProviders(), reloadProviders(), resetCircuit(key?), health(), ready(). Throws InternalApiError { status, code, message, requestId, details }.
  • format.ts: formatNumber, formatCompact, formatBytes, formatMs, formatUsd(n, precise?), formatPercent, formatDate, formatDateOnly, timeAgo, truncate, titleCase.
  • utils.ts: cn, slugify, SITE_URL, API_PUBLIC_URL.
  • audit.ts: writeAudit({ userId, organizationId, action, target, metadata, ipAddress, userAgent }).
  • auth-client.ts (client): authClient (Better Auth React client: signIn.email, signUp.email, signOut, changePassword, changeEmail, listSessions, revokeSession, revokeOtherSessions, deleteUser, sendVerificationEmail, updateUser).
  • Existing server actions: @/actions/projects (createProject(formData), updateProject(id, formData), archiveProject(id), selectProject(id); ActionResult<T> type), @/actions/api-keys (createApiKey({name, projectId, mode, scopes, expiresInDays}) → { ok, data: { id, plaintext, prefix, name, projectId } }, revokeApiKey(id), rotateApiKey(id)), @/actions/account (completeOnboarding(), updateOrganization(formData), updateProfileName(name), recordAuditFromClient(action, meta)).

# Data model (@fetcha/db, Drizzle; import { getDb, <table>, eq, and, desc, sql, gte, … } from "@fetcha/db")

Tables (TS names): users, sessions, accounts, verifications, organizations (plan, providerVisibility, softLimitUsd, hardLimitUsd, suspended, ownerUserId), organizationMembers, projects (name, slug, environment, defaultCountry, defaultNetwork, logLevel, soft/hardLimitUsd, monthlyRequestLimit, archivedAt), apiKeys (name, keyPrefix, last4, mode, scopes jsonb string[], expiresAt, lastUsedAt, revokedAt, keyHash — never display), fetchRequests (id req_…, organizationId, projectId, apiKeyId, source api|playground|sdk, url, finalUrl, domain, method, requestedNetwork, network, country, region, city, sessionId, browser, format, status pending|success|failed, httpStatus, errorCode, errorMessage, attempts, latencyMs, bytesIn, bytesOut, costUsd (upstream, admin-only), priceUsd (billed), cached, requestHeaders, responseHeaders, timing jsonb, clientIp, userAgent, createdAt, completedAt), requestAttempts (requestId, attemptNo, provider ⚠ admin-only, network, country, outcome success|blocked|timeout|error|provider_error|too_large, httpStatus, errorCode, errorDetail ⚠ admin-only, blockReason, durationMs, bytesIn, bytesOut, costUsd, routingScore, timing), proxySessions (id sess_…, projectId, label, provider ⚠, network, country, status active|expired|closed, requestCount, lastUsedAt, expiresAt), usageEvents (organizationId, projectId, requestId, metric request|bandwidth|residential_bandwidth|mobile_bandwidth, quantity, unit, costUsd = customer price, upstreamCostUsd ⚠, createdAt), subscriptions, billingEvents, providerConfigs (id, label, enabled, networks, pricePerGbUsd jsonb, weight, maxConcurrency), providerHealth (provider, network, status healthy|degraded|down|unconfigured, latencyMs, detail, checkedAt), domainProfiles (domain, preferredNetwork, preferredProvider ⚠, requests, successes, blocks, captchas, browserRequired, avgLatencyMs, routeStats jsonb, policy jsonb {order?, force_network?}, lastSeenAt), routingMetrics (hourly bucket, provider, network, country, requests, successes, blocked, errors, latencySumMs, bytes, costUsd), webhooks, webhookDeliveries, auditLogs (userId, organizationId, action, target, metadata, ipAddress, userAgent, createdAt), abuseEvents, featureFlags (key, enabled, description, plans, organizationIds), legalAcceptances, statusIncidents (component, title, body, severity, startedAt, resolvedAt). Always scope customer queries by organizationId/projectId from getWorkspace().

@fetcha/core exports: PLAN_LIMITS (per plan: label, monthly_requests, concurrency, max_timeout_ms, max_retries, networks, retention_days, price_usd_month, included_gb, overage_per_1k_requests_usd, residential_per_gb_usd), PLANS, NETWORK_CLASSES, API_KEY_SCOPES (fetch:execute, browser:use, sessions:write, usage:read), ERROR_CODES, ERROR_MESSAGES, ERROR_HTTP_STATUS, COUNTRIES (ISO2 → name), fetchRequestSchema (zod), FetchRequest, FetchResponseBody types.

# Public API surface (what docs / codegen must describe)

Base URL: https://www.fetcha.co (API is served under the same domain: POST https://www.fetcha.co/v1/fetch). Auth header: Authorization: Bearer fch_live_… (test keys fch_test_…). Response header X-Fetcha-Request-ID: req_….

  • POST /v1/fetch body: { url (required), method="GET", headers, cookies, body, timeout=30000 (ms, ≤ plan max), country (ISO2), region, city, network="auto"|datacenter|residential|isp|mobile, session ("sess_…"), browser=false, device, locale, format="html"|text|json|raw, follow_redirects=true, max_redirects=10, max_response_bytes, retries (≤ plan), debug=false }. Response 200: { request_id, success, status, url, final_url, content, content_type, headers, cookies:[{name,value,domain,path}], metadata: { network, country, attempts, duration_ms, bytes, session, cached, timing:{dns_ms,proxy_connect_ms,tls_ms,origin_ms,processing_ms,total_ms}, debug?: { attempts:[{provider: "network-a"…, network, country, outcome, status, duration_ms}] } }, text? (format=text), json? (format=json) }. A blocked target still returns 200 with success:false, status (e.g. 403) and metadata.attempts > 1.
  • POST /v1/sessions { country?, region?, city?, network?="auto", ttl?=600 (60–1800 s), label? } → { id:"sess_…", status, network, country, request_count, last_used_at, expires_at, created_at }; GET /v1/sessions, GET /v1/sessions/:id, DELETE /v1/sessions/:id. Idempotency-Key header supported on create.
  • GET /v1/me → { project, organization:{plan}, key:{scopes} }; GET /v1/usage → monthly usage summary.
  • GET /health, GET /ready (public, at https://www.fetcha.co/api/health and /api/ready).
  • Errors: { "error": { "code", "message", "request_id", "details"? } }. Codes: INVALID_API_KEY 401, EMAIL_NOT_VERIFIED 403, RATE_LIMITED 429 (+ Retry-After), CONCURRENCY_LIMIT 429, INVALID_REQUEST 400, URL_NOT_ALLOWED 400, TARGET_TIMEOUT 504, TARGET_BLOCKED 502, TARGET_UNAVAILABLE 502, PROVIDER_UNAVAILABLE 503, NETWORK_UNAVAILABLE 400, BROWSER_UNAVAILABLE 400, BROWSER_TIMEOUT 504, RESPONSE_TOO_LARGE 502, TOO_MANY_REDIRECTS 502, INSUFFICIENT_CREDITS 402, USAGE_LIMIT_REACHED 402, SESSION_NOT_FOUND 404, SESSION_EXPIRED 410, NOT_FOUND 404, FORBIDDEN 403, INTERNAL_ERROR 500.
  • Plans (from PLAN_LIMITS): Free $0 — 1,000 req/mo, 5 concurrent, 30 s timeout, 2 retries, 3-day logs, 0.5 GB residential; Developer $29 — 50k req, 25 concurrent, 60 s, 7 days, 5 GB incl., $0.60/1k overage, $9/GB residential; Growth $149 — 500k req, 100 concurrent, 90 s, 30 days, 30 GB, $0.40/1k, $7.50/GB; Business $599 — 5M req, 500 concurrent, 120 s, 90 days, 150 GB, $0.25/1k, $6/GB; Enterprise custom. Billing/checkout is NOT live yet: pricing page shows plans with "Contact sales" / "Start free"; dashboard Billing says upgrades open soon (email sales@fetcha.co).
  • SDKs: @fetcha/sdk (JS/TS: new Fetcha({ apiKey }).fetch({ url, country }), .text(url), .json(url), .sessions.create()) and fetcha (Python: Fetcha(api_key).fetch(url, country="CA"), .text, .json, .sessions.create(...)). Not yet published to npm/PyPI — docs must say "install from source (repo packages/sdk, sdk-python) until the packages are published". CLI: not available yet.
  • Routing intelligence ("auto" mode): weighted score (35 % historical success, 20 % cost, 15 % latency, 15 % network health, 10 % geography, 5 % session stability), per-domain profiles, circuit breakers, escalation cheap→premium, retries with new IP / alternate network. Explain this in marketing/docs without naming providers.
  • SSRF policy: only http/https; localhost, private, link-local, metadata and internal hosts blocked; every redirect validated.
  • Retention: request metadata kept per plan; response bodies not stored by default; sensitive headers redacted in logs.