# CLAUDE.md — Tendril

Web ingestion platform (search / scrape / crawl / map) running on macOS Apple Silicon. Built to beat Firecrawl on three axes: stealth, access to authenticated pages, and deterministic extraction quality.

This file is the project's persistent context. Read it fully before making any change.

Production host: m3u96a (Mac, M3, 96 GB unified memory) — single node, self-hosted. Public endpoint: https://www.ten-dril.com via ngrok reserved domain.

# Table of contents

# 0. Name & identity

Codename: Tendril — the climbing shoot that latches on and grows. Domain: ten-dril.com.

If the name changes, update: package.json names, Swift bundle IDs (com.tendril.worker), Redis key prefix (tdr:*), Postgres schema name, ngrok config, API key prefix (tdr_live_ / tdr_test_), LaunchAgent labels, and the User-Agent string.

# 1. Product thesis

Firecrawl runs in the cloud, on datacenter IPs, in headless Chromium. Tendril runs on a residential machine, in WebKit (Safari's actual engine).

There is no local LLM in this system. Extraction is deterministic: parsers, selectors, and structural inference. This is a deliberate choice with real consequences, so be honest about them:

• ✅ Predictable cost, predictable latency, reproducible output. The same page always yields the same result — you can write golden-file tests against extraction, which is impossible with a sampling model.
• ✅ No 20 GB resident model, no 40 s cold start, no GPU contention with the WebView pool.
• ❌ No zero-shot extraction from arbitrary prose. If a user wants "the CEO's name" from an unstructured about-page, deterministic parsing will not find it reliably.

The gap in ❌ is covered by BYOK (§8.7): the user supplies their own LLM API key, Tendril passes the cleaned markdown through and returns the result. Tendril never pays for inference and never stores the key beyond the request.

So the three real differentiators are:

1. Browser fidelity. WebKit on macOS ARM behind a residential IP is not a headless Chromium signature. Sites that block Firecrawl outright often do not block Tendril at all.
2. Authenticated pages. Tier 2 drives a real Safari profile with real logged-in sessions. No cloud service can offer this.
3. Extraction you can test. Deterministic output, versioned rules, golden files. Enterprise buyers care about reproducibility more than about magic.

Accepted constraint: one machine, one IP, no native horizontal scaling. Tendril is not built to crawl 10M pages/day. It is built to crawl 100k pages/day that nobody else can reach, with output you can regression-test.

# 2. Architecture — three-tier router

The core of the system is an escalation router. Always start at the cheapest tier; escalate only on evidence of failure.

                       ┌──────────────┐
   request ──────────► │   Router     │
                       └──────┬───────┘
                              │  shouldEscalate()
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
   ┌─────────┐          ┌───────────┐        ┌──────────────┐
   │ TIER 0  │          │  TIER 1   │        │   TIER 2     │
   │ undici  │─fail────►│ WKWebView │─fail──►│   Safari     │
   │  HTTP   │          │   pool    │        │ safaridriver │
   └────┬────┘          └─────┬─────┘        └──────┬───────┘
        │                     │                     │
        └─────────────────────┴─────────────────────┘
                              │  raw HTML + metadata
                              ▼
                     ┌──────────────────┐
                     │ Extraction engine │
                     └──────────────────┘

Escalation is one-way and capped. A request escalates at most twice. Every escalation is recorded with its reason so you can see, per domain, which tier actually works — and then pin that domain in domain_hints so future requests skip straight to it. The pinning table is what makes the system fast over time; without it you pay the escalation cost forever.

# 3. Tier 0 spec — HTTP (packages/fetcher-http)

# 3.1 Client configuration

undici.Agent with:

{
  connections: 64,
  pipelining: 1,              // pipelining breaks on many CDNs, leave at 1
  keepAliveTimeout: 30_000,
  keepAliveMaxTimeout: 120_000,
  connect: { timeout: 8_000, rejectUnauthorized: true },
  maxRedirections: 5,         // handled manually, see 3.3
  bodyTimeout: 20_000,
  headersTimeout: 10_000,
}

# 3.2 Header fidelity

Header order is a fingerprint. Send them in Safari's order, not alphabetically, and not in the order a JS object happens to iterate:

Host
Accept                    text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language           en-US,en;q=0.9      (or match the target's likely locale)
Accept-Encoding           gzip, deflate, br
User-Agent                Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 ...
                          + " Tendril/1.0 (+https://www.ten-dril.com/bot)"
Connection                keep-alive
Sec-Fetch-Dest            document
Sec-Fetch-Mode            navigate
Sec-Fetch-Site            none
Sec-Fetch-User            ?1
Upgrade-Insecure-Requests 1

The bot suffix in the UA is non-negotiable (§26). Stealth here means not looking broken, not lying about who you are.

TLS: Node's default cipher order differs from Safari's, which is a JA3-visible signal. If a target fingerprints TLS, that target belongs on Tier 1 anyway — do not try to fix it at Tier 0.

# 3.3 Redirect handling

Handle redirects manually rather than letting undici follow them, because you need to:

• record the full chain in metadata.redirects[] (users need it for canonicalization);
• re-run SSRF validation on every hop, not just the first (§16.5);
• detect meta-refresh and JS redirects in the body, which undici cannot see;
• stop on a redirect loop (same normalized URL twice) with ERR_REDIRECT_LOOP.

# 3.4 Escalation decision

shouldEscalate(response): TierDecision — a pure function, unit tested, no I/O.

Decisive → escalate. Two or more strong/moderate → escalate. Return the reason string; it goes in timings.escalationReason and into the metrics label.

# 4. Tier 1 spec — WKWebView pool (native/tendril-worker/)

This is the heart of the system. A Swift daemon exposing HTTP on 127.0.0.1:8787.

# 4.1 Why native Swift, not Playwright-WebKit

Playwright ships a patched WebKit build with detectable artifacts: modified navigator.webdriver handling, injected bindings, altered timing. A native WKWebView is the same binary as Safari — same CFNetwork stack, same TLS, same font cache, same media codecs. That fidelity is the product.

# 4.2 Process model

• One process, N WebViews. The daemon must run as a real AppKit app (NSApplication with .accessory activation policy), not a plain CLI, or WebKit refuses to lay out and render.
• Pool of 24 on m3u96a. Each WebView is 150-400 MB resident. Measure with footprint before raising it; the number is bounded by memory, not cores.
• The pool is a Swift actor. Acquire/release with an async semaphore, FIFO waiters, 30 s acquisition timeout → ERR_POOL_EXHAUSTED.
• Recycle aggressively. Destroy and recreate a WebView after 50 renders or 10 minutes of life, whichever comes first. WebKit leaks slowly under adversarial pages, and a recycled view has a clean JS heap.
• Hard per-render watchdog: if a render exceeds timeout + 5 s, tear the WebView down rather than waiting. A hung page must never hold a slot.

# 4.3 Isolation

Each WebView gets its own WKWebsiteDataStore:

• .nonPersistent() by default → disposable profile, no cookie bleed between customers. This is a security boundary, not an optimization.
• Named persistent stores for profileId (§7).
• Between renders on a non-persistent store, call removeData(ofTypes:modifiedSince:) with the full type set anyway. Belt and braces.

# 4.4 Render-complete detection

Never rely on didFinish alone — it fires at DOM ready, long before an SPA has content. Combine:

1. webView(_:didFinish:) fires, and
2. network idle: zero in-flight requests for 500 ms (track via a WKURLSchemeHandler shim or a PerformanceObserver injected at document start), and
3. DOM quiet: a MutationObserver reports no mutations for 300 ms, and
4. optional waitFor: a CSS selector present, or a fixed delay, or a JS predicate returning true.

Cap the whole thing at timeout. On cap, return what you have with metadata.renderTimedOut: true rather than failing — a partially rendered page is usually still useful, and the caller can decide.

Many sites poll forever (chat widgets, analytics beacons, websockets). Condition 2 must ignore requests to known-noisy hosts, or it never settles. Maintain a blocklist and apply it as a WKContentRuleList — which also speeds up every render substantially.

# 4.5 Injected user script (.atDocumentStart, main frame + subframes)

• Stub Notification.requestPermission, navigator.geolocation, navigator.mediaDevices so permission prompts never appear (a prompt blocks the run loop).
• Neutralize window.alert/confirm/prompt — otherwise a modal deadlocks the WebView.
• Install the MutationObserver and PerformanceObserver used by §4.4.
• Capture window.onerror and console.* into an array the daemon reads back. Return it in console[]; it is the single most useful debugging field for "why is this page empty".
• Do not try to spoof fingerprinting surfaces (canvas, WebGL, audio). WebKit's real values are the asset here. Spoofing creates inconsistencies that are more detectable than the truth.

# 4.6 Daemon API

POST /render
  { url, waitFor?, timeout?, profileId?, actions?, blockResources?, viewport?, userAgent? }
  → { html, status, finalURL, redirects[], console[], timings, renderTimedOut }

POST /screenshot
  { url, fullPage?, width?, height?, format?, quality? }  → image/png | image/jpeg

POST /pdf
  { url, paperSize? }                                      → application/pdf

GET  /health
  → { pool: { total, busy, idle, recycled }, rss, uptime, rendersTotal }

POST /profile   { id, cookies[], userAgent? }   → creates/updates a persistent data store
DELETE /profile/:id                             → wipes it

actions mini-DSL, executed in order after initial load:

[
  { "click": "#accept-cookies" },
  { "wait": 1500 },
  { "type": { "selector": "#search", "text": "query" } },
  { "press": "Enter" },
  { "scroll": 2000 },
  { "scrollToBottom": { "maxScrolls": 10, "delay": 500 } },
  { "waitForSelector": ".results-loaded" },
  { "evaluate": "document.querySelector('.more').click()" }
]

scrollToBottom is the workhorse for infinite-scroll pages; implement it as scroll → wait → compare scrollHeight → repeat until stable or maxScrolls.

# 4.7 Transport between Node and the daemon

HTTP over loopback with keep-alive. Not Unix sockets (harder to debug), not stdio (framing pain). Node client (packages/fetcher-webkit) must implement: 2 s connect timeout, single retry on ECONNREFUSED after a 1 s delay, circuit breaker that opens after 5 consecutive failures and routes everything to Tier 0 with a loud alert. If the daemon is dead, the API must degrade rather than 500 across the board.

# 5. Tier 2 spec — real Safari (packages/fetcher-safari)

Last resort, for targets that defeat Tier 1 or require the real user's logged-in state.

# 5.1 Two modes

• safaridriver (W3C WebDriver) — preferred. Stable, scriptable, proper timeouts. Hard limit: one session at a time on the machine. Safari Technology Preview provides a second, independent session; that is the ceiling — two, total.
• AppleScript via osascript — fallback. Serial, fragile, breaks on focus changes, but drives the real Safari with the real user profile and all its logged-in sessions.

# 5.2 Operating rules

• Its own BullMQ queue, concurrency: 1, hard 60 s timeout, max 200 jobs/day by default.
• Never reachable via tier: "auto" for anonymous requests. Tier 2 requires either an explicit tier: "safari" or a profileId bound to the real profile.
• Between jobs: close the tab, do not clear cookies (that is the point), but do clear the URL bar and history entry if the customer is not the profile owner.
• Fails closed. If safaridriver reports a session conflict, queue the job rather than falling back to AppleScript mid-flight — mixing the two corrupts state.
• Log every Tier 2 job with the requesting API key. This is the audit trail you will need if a target complains.

# 6. Proxy & identity layer (packages/egress)

One machine means one IP means one ban ends everything. Design for this from day one.

• Tier 0 and Tier 1 egress through rotating residential proxies. Tier 2 uses the host IP directly — that is precisely its value.
• Proxy selection is sticky per host for the duration of a crawl job: switching IPs mid-crawl of one site looks far more suspicious than staying put.
• Health-check the proxy pool every 60 s against a known-good endpoint. Eject on 3 consecutive failures, re-test after 10 min.
• Track per-(proxy, host) success rate in Redis. Stop routing a proxy to a host once its rate drops below 50% over the last 20 attempts — it is burned there.
• Geo-pin when the target is geo-sensitive: egressRegion option on the API, defaulting to the account's region.
• WKWebView proxy configuration is set per-WKWebsiteDataStore via proxyConfigurations (macOS 14+). It cannot be changed on a live store, so the pool must hold buckets of WebViews per proxy and acquire from the matching bucket. Plan the pool shape accordingly: e.g. 4 proxies × 6 WebViews rather than 24 undifferentiated ones.

# 7. Session profiles (packages/profiles)

A profile is a named, persistent browser identity: cookies, localStorage, and a pinned UA.

• Created by POST /v1/profiles with either an explicit cookie array, or an interactive login flow (§7.2).
• Stored as a WKWebsiteDataStore keyed by UUID on disk, plus a Postgres row holding metadata (owner, target domains, created/last-used, expiry).
• Cookies are secrets. Encrypt the Postgres-side copy with a key from the macOS Keychain, never log them, redact them from error payloads, and exclude the profile directory from any backup that leaves the machine unencrypted.
• Profiles expire: default 30 days idle, then wiped. Re-auth is the user's job.
• A profile is bound to one account. Cross-account use is a hard error, not a warning.

# 7.2 Interactive login flow

For targets where cookie import is impractical: POST /v1/profiles/:id/login opens a visible WKWebView on m3u96a, the operator logs in by hand, the daemon detects navigation to a success URL and seals the profile. This is a manual, low-volume operation and that is fine — it is what makes Tier 2 valuable.

# 8. Extraction engine (packages/extract)

Deterministic, versioned, testable. No model inference anywhere in this package.

# 8.1 Pipeline

raw HTML
  → encoding detection (charset header → <meta> → BOM → heuristic)
  → parse with linkedom
  → sanitize: strip <script> <style> <svg> <noscript>, ad iframes, tracking pixels
  → structured-data harvest (§8.3)  ─────────┐
  → boilerplate removal (§8.4)               │
  → relative → absolute URL rewriting        ├─► metadata
  → Turndown with custom rules (§8.5)        │
  → post-process                             │
  → markdown ────────────────────────────────┘

Every stage is a pure function (input, options) => output. The whole pipeline is testable offline against saved HTML fixtures, which is the entire reason for choosing determinism.

# 8.2 Boilerplate removal

Two modes, selected by onlyMainContent:

• Readability (@mozilla/readability) for article-shaped pages. Reliable on news, blogs, docs.
• Density-based fallback when Readability returns under 200 characters — which it does on listing pages, product pages, and dashboards. Implement a simple text-density scorer: for each block element, textLength / (1 + linkTextLength), pick the subtree maximizing the sum. Do not skip this fallback; Readability silently failing to a near-empty result is the single most common extraction bug.

Always strip, in both modes: <header>, <footer>, <nav>, [role=navigation], [aria-hidden=true], cookie banners (match on common class patterns), newsletter modals, and elements whose class matches /(^|[-_])(ad|ads|advert|sponsor|promo|share|social|related|comment)([-_]|$)/i.

# 8.3 Structured data harvest

This is where deterministic extraction earns its keep. Harvest, in priority order, and merge:

1. JSON-LD (<script type="application/ld+json">) — parse all blocks, handle @graph, resolve @type to a normalized shape. Covers Article, Product, Recipe, Event, JobPosting, BreadcrumbList, Organization, FAQPage.
2. Microdata (itemscope/itemprop) and RDFa — older but still common on e-commerce.
3. OpenGraph and Twitter cards — title, description, image, type, publish time.
4. Standard meta — description, author, keywords, canonical, hreflang alternates.
5. Heuristics — <time datetime> for dates, <h1> for title, first <p> over 100 chars for description.

Output as metadata plus a normalized structured object. A Product page should yield price, currency, availability, SKU, and images without any model involved. Roughly 60% of commercially interesting pages carry usable structured data; this is the highest-leverage code in the repo.

# 8.4 Selector-based extraction (/v1/extract core)

Users supply a schema where each field maps to an extractor:

{
  "schema": {
    "title":  { "selector": "h1", "type": "text" },
    "price":  { "selector": ".price", "type": "number", "clean": "currency" },
    "images": { "selector": "img.product", "type": "attr", "attr": "src", "multiple": true },
    "specs":  { "selector": "table.specs", "type": "table" },
    "sku":    { "jsonld": "$.sku" },
    "body":   { "selector": "#desc", "type": "markdown" }
  }
}

Types: text, html, markdown, number, date, url, attr, table, list, boolean (presence). Cleaners: currency, whitespace, trim, stripTags, parseDate (with locale hint). Sources: selector (CSS), xpath, jsonld (JSONPath into harvested JSON-LD), regex, meta.

Field resolution order: jsonld → meta → selector → xpath → regex. First non-empty wins. This lets one schema work across sites with different markup.

# 8.5 Repeated-structure inference (/v1/extract on list pages)

For listing pages, users should not have to write selectors. Implement auto-detection:

1. Compute a structural signature for every element: tag path + class set, normalized.
2. Find the deepest parent whose children contain ≥ 5 siblings sharing a signature.
3. That parent is the list container; the siblings are the records.
4. Within one record, each leaf position becomes a candidate field; label it from itemprop, class, or a nearby <dt>/<th>.

This handles search results, product grids, and job boards without configuration. Return the inferred selectors alongside the data so users can pin and refine them — inference that cannot be inspected and overridden is worse than useless.

# 8.6 Markdown conversion rules

Turndown with hand-written rules. Write these; the defaults are not good enough:

Post-process: collapse 3+ blank lines to 2, trim trailing whitespace, normalize list markers to -, deduplicate consecutive identical links.

# 8.7 BYOK LLM pass-through (optional, packages/byok)

Thin and stateless. /v1/extract accepts:

{ "llm": { "provider": "anthropic|openai|...", "apiKey": "sk-...", "model": "...", "prompt": "..." } }

Tendril runs its normal deterministic pipeline, then forwards the resulting markdown plus the user's JSON schema to their provider with their key, and returns the response.

Rules: the key is held in memory for the request only, never written to disk, never logged, never included in error payloads; chunk markdown over ~50k characters by heading and merge results; on provider error, return the deterministic result with llm: { error } rather than failing the whole request. Tendril is a pass-through, not an LLM vendor — do not build a model router, do not cache completions, do not offer a Tendril-supplied key.

# 8.8 Versioning

The extraction pipeline has a pipelineVersion (semver) stored with every cached result. Bumping it invalidates derived markdown but not the raw HTML cache, so you can re-extract the entire corpus after a rules improvement. This is the single most valuable property of keeping raw HTML forever.

# 9. Non-HTML content (packages/extract/formats)

Content-Type routing, before any HTML logic:

Never let a Content-Type mismatch reach the HTML parser. A 200 MB video streamed into linkedom will take the process down.

# 10. Crawl frontier (packages/frontier)

Non-negotiable rules:

1. Normalize before anything. Lowercase host, strip default port, sort query params, remove utm_*/fbclid/gclid/ref/mc_cid/_ga, drop fragment (unless #! hashbang), resolve .., consistent trailing slash, punycode hosts. Dedup runs on the normalized form. Keep the original for output.
2. Two-stage dedup. In-memory Bloom filter (1% FP, sized limit × 3) → Postgres confirmation on hit. Avoids a DB round-trip per discovered link, which at 8 concurrent renders is thousands per minute.
3. Per-host rate limiting, never global. Token bucket in Redis, configurable delayMs, honor robots.txt Crawl-delay when higher. A crawl of one site must not slow another customer's job.
4. Priority. Strict BFS by depth, then heuristics: /blog/, /docs/, /article/, /product/ ahead of /tag/, /author/, /page/47, ?sort=, ?filter=.
5. Trap detection. If more than 200 URLs on one host share a path pattern with only a varying numeric or date segment, sample instead of exhausting. Infinite calendars, paginated tag archives, and faceted-search URL explosions are the three classics; each can generate unbounded URLs.
6. Backoff. 429 or 503 → pause the entire host, exponential 2^n s, capped at 5 min, 3 attempts, then mark the host degraded for the job.
7. Budgets. Page count and wall-clock and total bytes. First limit reached ends the job cleanly with stopReason.
8. Canonical awareness. If a page declares <link rel=canonical> pointing elsewhere on the same host, record the mapping and do not crawl both. Saves 20-40% on most e-commerce sites.
9. Sitemap seeding. Always attempt sitemap discovery first, even for a crawl — starting from a complete URL list beats link-following, and it lets you report accurate progress instead of an unknown denominator.
10. Resumability. Frontier state lives in Postgres, not memory. A restart mid-crawl resumes; it does not start over.

# 10.1 robots.txt

Full RFC 9309 parsing: User-agent groups with correct specificity matching, Allow/Disallow with longest-match-wins, wildcards * and $, Sitemap directives, Crawl-delay (non-standard but honored). Cache per host for 24 h. On fetch failure treat as allow-all, but log it; on 5xx treat as disallow-all for 1 h, per the spec's intent.

# 11. Cache & storage (packages/cache)

• Content-addressed blob store. blobs/<sha256[0:2]>/<sha256[2:4]>/<sha256>. Raw HTML, screenshots, PDFs. Deduplicates automatically across customers — the same page fetched by two accounts is stored once.
• Metadata in Postgres, pointing at blob hashes. Never store blobs in Postgres.
• Cache key = sha256(normalizedURL + tier + profileId + relevantOptions). Options that do not change the fetch (output format, extraction schema) must not be in the key, or the hit rate collapses.
• maxAge on the request decides whether a hit is served. Default 0 (always fetch) for /scrape, 3600 for /crawl sub-pages.
• Respect ETag and Last-Modified: on a stale hit, revalidate with a conditional request. A 304 costs almost nothing and refreshes the entry.
• Eviction: LRU by last access, triggered when the blob store exceeds a configured ceiling (default 200 GB). Run it as a scheduled job, not inline.
• GDPR: cache entries carry a retention date; a purge job runs nightly (§26).

# 12. Data model

Postgres 16, schema tendril, migrations via drizzle. Core tables:

accounts        id, name, plan, created_at, egress_region
api_keys        id, account_id, hash (argon2id), prefix, scopes[], last_used_at, revoked_at
jobs            id, account_id, type(scrape|crawl|map|search|extract), status, params jsonb,
                created_at, started_at, finished_at, stop_reason, error_code
pages           id, job_id, url, normalized_url, canonical_url, depth, status_code,
                tier_used, escalation_reason, html_hash, markdown_hash, screenshot_hash,
                metadata jsonb, structured jsonb, pipeline_version, fetched_at, duration_ms
frontier        job_id, normalized_url, depth, priority, state(pending|active|done|failed),
                attempts, next_attempt_at            -- PK (job_id, normalized_url)
domain_hints    host, preferred_tier, requires_profile, crawl_delay_ms, success_rate,
                last_updated                          -- the learned routing table
profiles        id, account_id, name, domains[], data_store_path, cookies_enc,
                created_at, last_used_at, expires_at
robots_cache    host, body, fetched_at, expires_at
usage           account_id, day, pages_by_tier jsonb, bytes_out, llm_passthrough_calls
webhooks        id, job_id, url, events[], secret, deliveries jsonb

Indexes that matter: frontier(job_id, state, priority) for the dequeue path, pages(job_id, fetched_at) for pagination, pages(html_hash) for dedup lookups, partial index on frontier(next_attempt_at) WHERE state='failed'.

# 13. Queue & job lifecycle

BullMQ on Redis. Queues:

Job lifecycle: queued → running → (succeeded | failed | cancelled). Retries: 3 attempts, exponential backoff with jitter, only for transient error codes (§15). Never retry ERR_ROBOTS_DENIED, ERR_SSRF_BLOCKED, ERR_UNSUPPORTED_TYPE, or any 4xx other than 408/429.

Idempotency: /scrape, /crawl, /extract accept an Idempotency-Key header. Store key → jobId for 24 h; a repeat returns the original job rather than starting a new one. Customers with retry logic will otherwise double-bill themselves and blame you.

Cancellation is cooperative: set a flag in Redis, workers check it between pages, in-flight renders complete. Never kill a WebView to cancel — it corrupts the pool.

# 14. API contracts

Base: https://www.ten-dril.com/v1. Auth: Authorization: Bearer tdr_live_.... All responses: { success: boolean, data?, error?: { code, message, details? } }. All list responses are cursor-paginated: { data: [...], next?: "cursor" }. No offset pagination.

# 14.1 POST /v1/scrape

{
  "url": "https://example.com/article",
  "formats": ["markdown", "html", "links", "screenshot", "structured", "extract"],
  "extractSchema": { /* §8.4 */ },
  "llm": { /* §8.7, optional */ },
  "tier": "auto",                 // auto | http | webkit | safari
  "profileId": "uuid",            // forces tier >= 1
  "actions": [ /* §4.6 */ ],
  "onlyMainContent": true,
  "includeTags": ["article", "main"],
  "excludeTags": [".sidebar", "#comments"],
  "waitFor": 0,
  "timeout": 30000,
  "maxAge": 0,
  "headers": { /* custom, merged over defaults */ },
  "egressRegion": "us",
  "blockResources": ["image", "media", "font"]
}

Response data: { markdown, html, rawHtml, links[], structured, extract, screenshot, metadata, tierUsed, cached, timings }.

timings: { total, dns, connect, ttfb, download, render, extract, escalations[] }. Expose this — customers debugging slow crawls will otherwise blame you for their target's latency.

links[] is { url, text, rel, isInternal }, deduplicated, absolute.

# 14.2 POST /v1/crawl

{
  "url": "https://example.com",
  "limit": 1000,
  "maxDepth": 4,
  "maxDurationSeconds": 3600,
  "maxBytes": 5368709120,
  "includePaths": ["^/blog/.*"],
  "excludePaths": ["\\.pdf$", "^/tag/"],
  "allowSubdomains": false,
  "allowBackwardLinks": false,
  "allowExternalLinks": false,
  "concurrency": 8,
  "delayMs": 250,
  "respectRobots": true,
  "deduplicateSimilar": true,
  "scrapeOptions": { /* §14.1 minus url */ },
  "webhook": { "url": "...", "events": ["page","completed","failed"], "secret": "..." }
}

→ 202 { jobId, statusUrl, streamUrl }.

• GET /v1/crawl/:id → { status, total, completed, failed, creditsUsed, data[], next }
• GET /v1/crawl/:id/stream → SSE, one event per page plus a terminal event. Heartbeat every 15 s or ngrok will drop the connection.
• DELETE /v1/crawl/:id → cooperative cancel, returns partial results.
• GET /v1/crawl/:id/errors → per-URL failures with codes. Customers need this and Firecrawl handles it poorly.

Webhook payloads are signed: X-Tendril-Signature: sha256=<hmac(secret, body)>, plus X-Tendril-Timestamp for replay protection. Retry 5 times with exponential backoff, then mark the webhook degraded.

# 14.3 POST /v1/map

URL discovery without rendering. Target: under 3 s for a normal site. Sources merged and deduplicated: sitemap.xml (recursive through sitemap indexes, .gz support, cap at 50k URLs), robots.txt Sitemap directives, homepage links, /llms.txt, RSS/Atom feeds. Options: search (substring filter, ranked by match position), limit, includeSubdomains, sitemapOnly, ignoreSitemap. Returns { links: [{ url, title?, lastModified?, source }] } — source tells the user where each URL came from, which builds trust in the result.

# 14.4 POST /v1/search

{ "query": "...", "limit": 10, "lang": "en", "country": "us", "timeRange": "month",
  "scrapeResults": false, "scrapeOptions": {} }

Pipeline: SearXNG (self-hosted, §16.1) → dedup by domain+normalized title → rank → optionally scrape the top N in parallel (Tier 0 forced, 8 s timeout, partial failures tolerated and reported per-result).

Be explicit in the docs: Tendril has no index of its own. Search quality is SearXNG's quality. Under-promise here.

# 14.5 POST /v1/extract

{
  "urls": ["https://example.com/products/*"],   // wildcards expand via /map
  "schema": { /* §8.4 */ },
  "inferList": true,                            // §8.5 auto-detection
  "llm": { /* §8.7, optional */ },
  "scrapeOptions": {}
}

Returns one record per matched URL, or one aggregated array when inferList finds a list page. Include the resolved selectors in the response so users can pin them.

# 14.6 Ancillary

• POST /v1/profiles, GET /v1/profiles, DELETE /v1/profiles/:id, POST /v1/profiles/:id/login
• GET /v1/usage?from=&to= — pages by tier, bytes, cost
• GET /healthz — unauthenticated, shallow
• GET /v1/status — authenticated, per-tier health (§18.3)

Rate limits returned on every response: X-RateLimit-Limit, -Remaining, -Reset.

# 15. Error taxonomy

One file maps codes to HTTP statuses and retry-ability. Never invent an ad-hoc string.

Every error response carries a requestId that appears in the logs. Support requests without one are unanswerable.

# 16. Deployment — node m3u96a + ngrok

Single host. No Kubernetes, no cloud VM. Ingress is an ngrok tunnel on a reserved domain.

# 16.1 Topology

   Internet
      │
      ▼
 ngrok edge  ──── TLS terminated here, certificate managed by ngrok
 www.ten-dril.com
      │  (encrypted tunnel, outbound connection initiated by the Mac)
      ▼
 ┌──────────────────────── m3u96a ────────────────────────┐
 │  ngrok agent (LaunchAgent)                             │
 │        │                                               │
 │        ▼                                               │
 │  Caddy :8080 ── reverse proxy, routing, access logs    │
 │        ├─► Fastify API            :3000                │
 │        └─► /healthz                                    │
 │                                                        │
 │  Internal only, bound to 127.0.0.1, never tunneled:    │
 │    tendril-worker (Swift)         :8787                │
 │    Redis                          :6379                │
 │    PostgreSQL                     :5432                │
 │    SearXNG                        :8888                │
 │    Prometheus / Grafana           :9090 / :3001        │
 └────────────────────────────────────────────────────────┘

Caddy sits between ngrok and the API deliberately: one place for routing, access logs, and local rate limiting, and only one port is ever tunneled.

# 16.2 ngrok configuration

deploy/ngrok.yml:

version: "3"
agent:
  authtoken: ${NGROK_AUTHTOKEN}
  log: /usr/local/var/log/ngrok.log
  log_level: info
  connect_timeout: 10s
endpoints:
  - name: tendril-api
    url: https://www.ten-dril.com
    upstream:
      url: 8080
    traffic_policy:
      inbound:
        - actions:
            - type: rate-limit
              config:
                name: global
                algorithm: sliding_window
                capacity: 600
                rate: 60s
        - expressions:
            - "req.url.path.startsWith('/internal')"
          actions:
            - type: deny
              config:
                status_code: 404

DNS: CNAME for www.ten-dril.com → the target ngrok returns when you reserve the domain. Reserve the apex ten-dril.com too and 301 it to www at the ngrok edge, not in the application.

A reserved domain requires a paid ngrok plan; free-tier URLs rotate on restart and are unusable for a customer-facing API. Confirm current plan tiers and Traffic Policy feature availability on ngrok's site before building against a specific capability — this changes.

# 16.3 Process supervision

LaunchAgents (not LaunchDaemons — WebKit requires a user GUI context), in ~/Library/LaunchAgents/:

Set SoftResourceLimits: { NumberOfFiles: 65536 } in every plist, or the pool saturates around 40 connections.

Start ordering: com.tendril.ngrok runs a wrapper that polls http://127.0.0.1:8080/healthz until healthy (max 120 s) before starting the tunnel. Exposing an unhealthy API is worse than a short outage.

# 16.4 Deploy procedure (scripts/deploy.sh)

1. git pull on a deploy branch, never on a dirty tree
2. pnpm install --frozen-lockfile && pnpm build && swift build -c release
3. pnpm db:migrate — migrations must be backward-compatible for one version, so a rollback does not need a down-migration
4. Drain: stop accepting new jobs, wait up to 60 s for in-flight ones
5. launchctl kickstart -k each agent in dependency order
6. ./scripts/health.sh — all tiers, all services
7. Smoke test through the public URL, not localhost: scrape a known page via https://www.ten-dril.com/v1/scrape and assert the markdown contains an expected string
8. On failure: git checkout previous tag, repeat from 2. Keep the last 3 builds on disk.

# 16.5 Security posture

Exposing a home machine to the internet deserves more care than a cloud VM.

• Only port 8080 is tunneled. 8787, 6379, 5432, 8888, 9090 bind to 127.0.0.1 explicitly. Verify with lsof -iTCP -sTCP:LISTEN after every deploy.
• API keys hashed with argon2id, prefix stored in clear for identification, full key shown once at creation.
• Quotas enforced in-app and a global rate limit at the ngrok edge, so a burst never reaches the Mac.
• Grafana and any admin surface: IP allowlist in Traffic Policy, or a separate non-public tunnel.
• SSRF is the top risk of any scraping API. Users control the target URL. Enforce: block RFC1918, loopback, link-local, 169.254.169.254, IPv6 ULA and mapped-IPv4; allow only http/https; allow only ports 80/443/8080/8443. Resolve DNS first and validate the resolved IP, then connect to that IP with the Host header set — otherwise DNS rebinding walks straight into Redis and Postgres. Re-validate on every redirect hop.
• Redact from logs: Authorization, Cookie, Set-Cookie, BYOK keys, webhook secrets, profile cookie blobs.
• macOS firewall on, Remote Login restricted to key auth on a non-default port, FileVault enabled.

# 16.6 Failure modes specific to this setup

These are what will actually take the service down. Treat them as design constraints.

• The tunnel is a single point of failure. Sleep, Wi-Fi loss, or reboot means total outage. External uptime monitor on https://www.ten-dril.com/healthz every 60 s, alerting to your phone. Nothing else in this document matters if you find out from a customer.
• Residential upload bandwidth is the real ceiling. Screenshots and large HTML saturate a typical uplink long before CPU. Gzip everything (Caddy), cap screenshot dimensions, and serve large blobs via short-lived signed links rather than inline base64.
• Reboots break the GUI session. Without auto-login, Tiers 1 and 2 die while Tier 0 keeps returning bot-blocked pages. That looks like a quality regression, not an outage — which is why health checks must probe each tier independently (§18.3).
• ngrok reconnects silently but not instantly. Expect 5-30 s gaps on network flaps. SDKs must retry with backoff; document it.
• One IP, one ban surface. See §6.
• Thermal throttling. 24 WebViews rendering continuously will heat an M3 and reduce clocks. Monitor powermetrics and cap the pool lower if sustained throughput drops.
• Backups. Postgres and the blob CAS live on one SSD. Nightly pg_dump plus a CAS rsync to an external encrypted volume, offsite weekly, and a tested restore. An untested backup is not a backup.

# 17. macOS prerequisites

Scripted in scripts/setup-mac.sh, idempotent, but several steps require a human.

• A GUI session is mandatory. WKWebView and Safari will not render without a window server session. SSH alone is insufficient. Enable auto-login; connect via Screen Sharing after the graphical session exists.
• caffeinate -dimsu as a LaunchAgent. Sleep kills the pool.
• Screen lock and screen saver disabled. A locked screen suspends rendering in some WebViews.
• Safari → Develop → "Allow JavaScript from Apple Events" must be ticked by hand. Not scriptable.
• safaridriver --enable once, with an admin password.
• Automation and Accessibility permissions: the first osascript triggers a TCC dialog requiring a click. No clean bypass. Document it in the runbook.
• Full Disk Access for the terminal and the daemon if touching the Safari profile directory.
• Disable Spotlight indexing on the blob store (mdutil -i off) — it will otherwise index millions of files.
• Increase kern.maxfiles and kern.maxfilesperproc via a sysctl plist.

# 18. Observability

# 18.1 Logs

Pino, JSON, to stdout and a rotated file. Every line carries { requestId, jobId, accountId, url, tier } where applicable. No console.log anywhere, enforced by lint rule.

# 18.2 Metrics (Prometheus, scraped locally)

• tendril_requests_total{endpoint,status}
• tendril_fetch_duration_seconds{tier} — histogram
• tendril_escalations_total{from,to,reason} — the most important metric in the system
• tendril_pool_slots{state} — gauge: busy/idle/recycling
• tendril_extraction_empty_total{reason} — pages yielding under 200 chars; a rising line here means a silent quality regression
• tendril_proxy_success_rate{proxy,host}
• tendril_tunnel_up — 0/1 from the agent's local API
• tendril_queue_depth{queue}, tendril_job_duration_seconds{type}

# 18.3 Health checks

GET /v1/status probes each tier independently against a known-stable control URL and returns per-tier latency and last success. /healthz stays shallow and fast for the uptime monitor.

Alert on: tunnel down, daemon circuit open, escalation rate above 40% (a target changed its defenses), extraction-empty rate above 10%, queue depth growing for 10 minutes, disk above 85%.

# 19. Performance targets

Measured on m3u96a, pnpm bench against 100 reference URLs. Regressions block a release.

# 20. Testing

• Unit: URL normalization, shouldEscalate, Turndown rules, robots parsing, SSRF validation, structural inference. Fast, hermetic, no network. These are the majority of the suite.
• Fixtures: test/fixtures/html/ with ~60 saved real pages — React SPA, Next.js, WordPress, Shopify, static docs, a Cloudflare interstitial, a paginated listing, a page with nested tables, a page with JSON-LD, one with broken encoding. Extraction is tested only here.
• Golden files for markdown output. Any rule change must show its diff and be reviewed.
• Contract tests on the API schemas — the OpenAPI document is generated from Zod, so schema drift breaks the build.
• E2E against 20 stable public sites, tagged @slow, excluded from CI. Asserts escalation rates and non-empty extraction, never exact content.
• Deploy smoke test through the public URL (§16.4). Half of all outages are ingress, not application.
• Chaos: kill the Swift daemon mid-crawl and assert the circuit breaker degrades to Tier 0 instead of failing the job. Do this in CI with a fake daemon.

Coverage target: 85% on packages/, no target on apps/.

# 21. Code conventions

• TypeScript strict, noUncheckedIndexedAccess, exactOptionalPropertyTypes. No any; unknown plus narrowing.
• No exceptions for normal control flow. Fetchers and extractors return Result<T, TendrilError>. Exceptions signal bugs and are caught only at the top-level handler.
• Every error carries a code from §15. The code → HTTP mapping lives in exactly one file.
• Pure functions wherever possible; I/O confined to fetcher-* and cache. This is what makes the fixture-based tests viable.
• No comments restating code. Comment the why, especially for site-specific workarounds — and date them, they rot within months.
• Swift: async/await only, no completion handlers. The pool is an actor. No force-unwraps outside tests.
• Commits: conventional commits. A commit touching extraction rules must include the golden-file diff.

# 22. SDKs & docs

• TypeScript SDK — generated types from the OpenAPI schema, hand-written ergonomics on top. Must handle: automatic retry with backoff (for tunnel flaps, §16.6), SSE streaming for crawls with reconnect, idempotency keys generated by default.
• Python SDK — same surface, sync and async clients.
• Docs site with runnable examples per endpoint, plus a migration guide from Firecrawl mapping their parameter names to yours. Compatibility of naming where it costs nothing is worth more than elegance.
• Publish the OpenAPI document at https://www.ten-dril.com/openapi.json.

# 23. Repository layout

tendril/
├── CLAUDE.md
├── package.json                 # pnpm workspaces
├── turbo.json
├── apps/
│   ├── api/                     # Fastify — public surface
│   │   ├── src/routes/{scrape,crawl,map,search,extract,profiles,usage}.ts
│   │   ├── src/schemas/         # Zod → generated OpenAPI
│   │   ├── src/auth/            # keys, quotas, rate limits
│   │   └── src/errors.ts        # §15 mapping, single source of truth
│   └── worker/                  # BullMQ consumers
├── packages/
│   ├── router/                  # tier decision, escalation, domain_hints
│   ├── fetcher-http/            # Tier 0
│   ├── fetcher-webkit/          # Swift daemon client + circuit breaker
│   ├── fetcher-safari/          # Tier 2
│   ├── egress/                  # proxy pool, SSRF validation
│   ├── profiles/                # session management
│   ├── extract/                 # deterministic pipeline
│   │   ├── src/structured/      # JSON-LD, microdata, OG
│   │   ├── src/boilerplate/     # Readability + density fallback
│   │   ├── src/markdown/        # Turndown rules
│   │   ├── src/selectors/       # schema-driven extraction
│   │   ├── src/infer/           # repeated-structure detection
│   │   └── src/formats/         # PDF, CSV, XML, feeds
│   ├── byok/                    # optional LLM pass-through
│   ├── frontier/                # crawl logic, robots, sitemaps
│   ├── cache/                   # CAS + metadata
│   └── shared/                  # types, errors, logger, Result
├── native/
│   └── tendril-worker/
│       ├── Package.swift
│       └── Sources/TendrilWorker/{main,Pool,Renderer,Server,Actions,Profiles}.swift
├── deploy/
│   ├── ngrok.yml
│   ├── Caddyfile
│   ├── launchagents/*.plist
│   └── runbook.md
├── sdks/{typescript,python}/
├── test/fixtures/
└── scripts/{setup-mac,deploy,health,backup}.sh

# 24. Roadmap

Phase 1 — foundations (2 wks). Tier 0, extraction pipeline (§8.1-8.3, 8.6), /scrape, CAS cache, error taxonomy. Must handle 80% of the fixture set. Ship nothing publicly yet.

Phase 2 — rendering (3 wks). Swift daemon, pool, escalation router, domain_hints, /screenshot. Targets: escalation under 25%, Tier 1 p95 under 2.5 s.

Phase 3 — crawl (2 wks). Frontier, BullMQ, /crawl with SSE and signed webhooks, /map, resumability.

Phase 4 — extraction depth (3 wks). Selector schemas (§8.4), structure inference (§8.5), non-HTML formats (§9), /extract. This is the phase that differentiates the product — do not cut it. Optional BYOK at the end.

Phase 5 — stealth & identity (2 wks). Proxy layer, Tier 2 Safari, session profiles, /search via SearXNG.

Phase 6 — production (2 wks). Deploy on m3u96a, ngrok reserved domain, quotas, billing, observability, SDKs, docs, migration guide.

Do not start Phase 4 before Phase 2's escalation rate target is met. Extraction quality on pages you cannot fetch is worth nothing.

# 25. Anti-patterns — do not do this

• ❌ Driving Safari via AppleScript for volume. Serial, breaks on focus change. A last resort, not a foundation.
• ❌ Spawning a WKWebView per request. Init costs ~800 ms. Pool or nothing.
• ❌ Using jsdom. linkedom is 5-10× faster on this workload and the difference is the whole extraction budget.
• ❌ Storing markdown without the source HTML. You will never re-extract after improving rules (§8.8).
• ❌ Putting output-format options in the cache key. Hit rate collapses.
• ❌ A global rate limit instead of per-host. You will get banned from one site while crawling another fast.
• ❌ Trusting networkidle alone as a render signal. Many pages poll forever.
• ❌ Letting Readability's empty result pass through silently. Always run the density fallback.
• ❌ Ignoring robots.txt by default. Explicit opt-out, logged per client, never implicit.
• ❌ Putting Tier 2 in the general queue. One stuck session freezes everything.
• ❌ Validating the hostname instead of the resolved IP for SSRF. That is a rebinding hole straight into your database.
• ❌ Tunneling any port other than 8080.
• ❌ Assuming the tunnel is up. Verify after every deploy, monitor continuously.
• ❌ Adding a local model "just for the hard cases". That decision reverses §1 and turns a testable system into a probabilistic one. If it becomes necessary, do it as an explicit, documented rewrite — not as a quiet addition.

# 26. Legal & ethical

Not decorative. This is real product risk.

• respectRobots: true is the default. Disabling requires an explicit flag, is logged per client, and is refused entirely for domains on an internal blocklist.
• Honest rate limiting: minimum 100 ms delayMs per host, not overridable.
• Identifiable User-Agent with a contact URL (+https://www.ten-dril.com/bot) and a published page explaining who you are and how to block you. Stealth means matching a real browser's fingerprint, not hiding your identity.
• Scraping authenticated pages (Tier 2) puts the user on the hook for the target's ToS. State it plainly and require acceptance at signup.
• No CAPTCHA solving, no paywall bypass, no login-credential stuffing. Commercial red lines as much as ethical ones.
• Personal data collected incidentally has a retention period; the nightly purge job enforces it. Support deletion requests by URL and by domain.
• Honor noindex/nofollow as a signal for crawl scope even though they are indexing directives — it costs little and is defensible.
• Running a public service from a residential connection may conflict with your ISP's terms on operating servers. Check the contract before going live on www.ten-dril.com.

I am not a lawyer. Have this framework reviewed by one before commercializing, particularly regarding the CFAA and state computer-crime statutes (US), the Database Directive and GDPR (EU), and the terms of the specific high-value targets you intend to support.