spb/tendril Public

1# CLAUDE.md — Tendril23> Web ingestion platform (search / scrape / crawl / map) running on macOS Apple Silicon.4> Built to beat Firecrawl on three axes: **stealth**, **access to authenticated pages**, and5> **deterministic extraction quality**.67This file is the project's persistent context. Read it fully before making any change.89**Production host:** `m3u96a` (Mac, M3, 96 GB unified memory) — single node, self-hosted.10**Public endpoint:** `https://www.ten-dril.com` via ngrok reserved domain.1112---1314## Table of contents1516| § | Section |17|---|---|18| 0 | Name & identity |19| 1 | Product thesis |20| 2 | Architecture — three-tier router |21| 3 | Tier 0 spec — HTTP |22| 4 | Tier 1 spec — WKWebView pool |23| 5 | Tier 2 spec — real Safari |24| 6 | Proxy & identity layer |25| 7 | Session profiles |26| 8 | Extraction engine (deterministic) |27| 9 | Non-HTML content |28| 10 | Crawl frontier |29| 11 | Cache & storage |30| 12 | Data model |31| 13 | Queue & job lifecycle |32| 14 | API contracts |33| 15 | Error taxonomy |34| 16 | Deployment — m3u96a + ngrok |35| 17 | macOS prerequisites |36| 18 | Observability |37| 19 | Performance targets |38| 20 | Testing |39| 21 | Code conventions |40| 22 | SDKs & docs |41| 23 | Repository layout |42| 24 | Roadmap |43| 25 | Anti-patterns |44| 26 | Legal & ethical |4546---4748## 0. Name & identity4950Codename: **Tendril** — the climbing shoot that latches on and grows.51Domain: `ten-dril.com`.5253If the name changes, update: `package.json` names, Swift bundle IDs (`com.tendril.worker`), Redis key prefix (`tdr:*`), Postgres schema name, ngrok config, API key prefix (`tdr_live_` / `tdr_test_`), LaunchAgent labels, and the User-Agent string.5455---5657## 1. Product thesis5859Firecrawl runs in the cloud, on datacenter IPs, in headless Chromium. Tendril runs on a **residential machine**, in **WebKit** (Safari's actual engine).6061**There is no local LLM in this system.** Extraction is deterministic: parsers, selectors, and structural inference. This is a deliberate choice with real consequences, so be honest about them:6263- ✅ Predictable cost, predictable latency, reproducible output. The same page always yields the same result — you can write golden-file tests against extraction, which is impossible with a sampling model.64- ✅ No 20 GB resident model, no 40 s cold start, no GPU contention with the WebView pool.65- ❌ No zero-shot extraction from arbitrary prose. If a user wants "the CEO's name" from an unstructured about-page, deterministic parsing will not find it reliably.6667The gap in ❌ is covered by **BYOK** (§8.7): the user supplies their own LLM API key, Tendril passes the cleaned markdown through and returns the result. Tendril never pays for inference and never stores the key beyond the request.6869So the three real differentiators are:70711. **Browser fidelity.** WebKit on macOS ARM behind a residential IP is not a headless Chromium signature. Sites that block Firecrawl outright often do not block Tendril at all.722. **Authenticated pages.** Tier 2 drives a real Safari profile with real logged-in sessions. No cloud service can offer this.733. **Extraction you can test.** Deterministic output, versioned rules, golden files. Enterprise buyers care about reproducibility more than about magic.7475**Accepted constraint:** one machine, one IP, no native horizontal scaling.76Tendril is not built to crawl 10M pages/day. It is built to crawl 100k pages/day *that nobody else can reach*, with output you can regression-test.7778---7980## 2. Architecture — three-tier router8182The core of the system is an **escalation router**. Always start at the cheapest tier; escalate only on evidence of failure.8384```85                       ┌──────────────┐86   request ──────────► │   Router     │87                       └──────┬───────┘88                              │  shouldEscalate()89        ┌─────────────────────┼─────────────────────┐90        ▼                     ▼                     ▼91   ┌─────────┐          ┌───────────┐        ┌──────────────┐92   │ TIER 0  │          │  TIER 1   │        │   TIER 2     │93   │ undici  │─fail────►│ WKWebView │─fail──►│   Safari     │94   │  HTTP   │          │   pool    │        │ safaridriver │95   └────┬────┘          └─────┬─────┘        └──────┬───────┘96        │                     │                     │97        └─────────────────────┴─────────────────────┘98                              │  raw HTML + metadata99                              ▼100                     ┌──────────────────┐101                     │ Extraction engine │102                     └──────────────────┘103```104105| | Tier 0 | Tier 1 | Tier 2 |106|---|---|---|---|107| Engine | undici | WKWebView | Safari + safaridriver |108| Latency p50 | 50 ms | 500 ms | 3-8 s |109| Concurrency | 200 | 24 | **1** |110| JS execution | no | yes | yes |111| Cookies/session | per-request | per-profile | real user profile |112| Share of traffic | 80% | 18% | 2% |113| Cost per page | ~0 | ~15 MB·s RAM | a human-scale amount of time |114115**Escalation is one-way and capped.** A request escalates at most twice. Every escalation is recorded with its reason so you can see, per domain, which tier actually works — and then pin that domain in `domain_hints` so future requests skip straight to it. The pinning table is what makes the system fast over time; without it you pay the escalation cost forever.116117---118119## 3. Tier 0 spec — HTTP (`packages/fetcher-http`)120121### 3.1 Client configuration122123`undici.Agent` with:124125```ts126{127  connections: 64,128  pipelining: 1,              // pipelining breaks on many CDNs, leave at 1129  keepAliveTimeout: 30_000,130  keepAliveMaxTimeout: 120_000,131  connect: { timeout: 8_000, rejectUnauthorized: true },132  maxRedirections: 5,         // handled manually, see 3.3133  bodyTimeout: 20_000,134  headersTimeout: 10_000,135}136```137138### 3.2 Header fidelity139140Header **order** is a fingerprint. Send them in Safari's order, not alphabetically, and not in the order a JS object happens to iterate:141142```143Host144Accept                    text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8145Accept-Language           en-US,en;q=0.9      (or match the target's likely locale)146Accept-Encoding           gzip, deflate, br147User-Agent                Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 ...148                          + " Tendril/1.0 (+https://www.ten-dril.com/bot)"149Connection                keep-alive150Sec-Fetch-Dest            document151Sec-Fetch-Mode            navigate152Sec-Fetch-Site            none153Sec-Fetch-User            ?1154Upgrade-Insecure-Requests 1155```156157The bot suffix in the UA is non-negotiable (§26). Stealth here means *not looking broken*, not lying about who you are.158159TLS: Node's default cipher order differs from Safari's, which is a JA3-visible signal. If a target fingerprints TLS, that target belongs on Tier 1 anyway — do not try to fix it at Tier 0.160161### 3.3 Redirect handling162163Handle redirects manually rather than letting undici follow them, because you need to:164165- record the full chain in `metadata.redirects[]` (users need it for canonicalization);166- re-run SSRF validation on **every** hop, not just the first (§16.5);167- detect meta-refresh and JS redirects in the body, which undici cannot see;168- stop on a redirect loop (same normalized URL twice) with `ERR_REDIRECT_LOOP`.169170### 3.4 Escalation decision171172`shouldEscalate(response): TierDecision` — a pure function, unit tested, no I/O.173174| Signal | Threshold | Weight |175|---|---|---|176| Status 403 / 429 / 503 | — | decisive |177| Body < 2 KB with empty `#root`, `#__next`, `#app`, `[ng-version]` | — | decisive |178| Body contains `challenge-platform`, `cf-browser-verification`, `_Incapsula_`, `px-captcha`, `datadome` | — | decisive |179| Text/HTML ratio | < 0.05 | strong |180| `<noscript>` containing "enable JavaScript" / "JavaScript is required" | — | strong |181| Zero `<a href>` on a page over 10 KB | — | moderate |182| `<title>` matching `/^(just a moment|attention required|access denied)/i` | — | decisive |183| Content-Type is not HTML | — | route to §9, do not escalate |184185Decisive → escalate. Two or more strong/moderate → escalate. Return the reason string; it goes in `timings.escalationReason` and into the metrics label.186187---188189## 4. Tier 1 spec — WKWebView pool (`native/tendril-worker/`)190191**This is the heart of the system.** A Swift daemon exposing HTTP on `127.0.0.1:8787`.192193### 4.1 Why native Swift, not Playwright-WebKit194195Playwright ships a patched WebKit build with detectable artifacts: modified `navigator.webdriver` handling, injected bindings, altered timing. A native `WKWebView` is the *same binary* as Safari — same CFNetwork stack, same TLS, same font cache, same media codecs. That fidelity is the product.196197### 4.2 Process model198199- **One process, N WebViews.** The daemon **must** run as a real AppKit app (`NSApplication` with `.accessory` activation policy), not a plain CLI, or WebKit refuses to lay out and render.200- Pool of **24** on m3u96a. Each WebView is 150-400 MB resident. Measure with `footprint` before raising it; the number is bounded by memory, not cores.201- The pool is a Swift `actor`. Acquire/release with an async semaphore, FIFO waiters, 30 s acquisition timeout → `ERR_POOL_EXHAUSTED`.202- **Recycle aggressively.** Destroy and recreate a WebView after 50 renders or 10 minutes of life, whichever comes first. WebKit leaks slowly under adversarial pages, and a recycled view has a clean JS heap.203- Hard per-render watchdog: if a render exceeds `timeout + 5 s`, tear the WebView down rather than waiting. A hung page must never hold a slot.204205### 4.3 Isolation206207Each WebView gets its own `WKWebsiteDataStore`:208209- `.nonPersistent()` by default → disposable profile, no cookie bleed between customers. This is a **security boundary**, not an optimization.210- Named persistent stores for `profileId` (§7).211- Between renders on a non-persistent store, call `removeData(ofTypes:modifiedSince:)` with the full type set anyway. Belt and braces.212213### 4.4 Render-complete detection214215Never rely on `didFinish` alone — it fires at DOM ready, long before an SPA has content. Combine:2162171. `webView(_:didFinish:)` fires, **and**2182. network idle: zero in-flight requests for 500 ms (track via a `WKURLSchemeHandler` shim or a `PerformanceObserver` injected at document start), **and**2193. DOM quiet: a `MutationObserver` reports no mutations for 300 ms, **and**2204. optional `waitFor`: a CSS selector present, or a fixed delay, or a JS predicate returning true.221222Cap the whole thing at `timeout`. On cap, **return what you have** with `metadata.renderTimedOut: true` rather than failing — a partially rendered page is usually still useful, and the caller can decide.223224Many sites poll forever (chat widgets, analytics beacons, websockets). Condition 2 must ignore requests to known-noisy hosts, or it never settles. Maintain a blocklist and apply it as a `WKContentRuleList` — which also speeds up every render substantially.225226### 4.5 Injected user script (`.atDocumentStart`, main frame + subframes)227228- Stub `Notification.requestPermission`, `navigator.geolocation`, `navigator.mediaDevices` so permission prompts never appear (a prompt blocks the run loop).229- Neutralize `window.alert/confirm/prompt` — otherwise a modal deadlocks the WebView.230- Install the `MutationObserver` and `PerformanceObserver` used by §4.4.231- Capture `window.onerror` and `console.*` into an array the daemon reads back. Return it in `console[]`; it is the single most useful debugging field for "why is this page empty".232- Do **not** try to spoof fingerprinting surfaces (canvas, WebGL, audio). WebKit's real values are the asset here. Spoofing creates inconsistencies that are *more* detectable than the truth.233234### 4.6 Daemon API235236```237POST /render238  { url, waitFor?, timeout?, profileId?, actions?, blockResources?, viewport?, userAgent? }239  → { html, status, finalURL, redirects[], console[], timings, renderTimedOut }240241POST /screenshot242  { url, fullPage?, width?, height?, format?, quality? }  → image/png | image/jpeg243244POST /pdf245  { url, paperSize? }                                      → application/pdf246247GET  /health248  → { pool: { total, busy, idle, recycled }, rss, uptime, rendersTotal }249250POST /profile   { id, cookies[], userAgent? }   → creates/updates a persistent data store251DELETE /profile/:id                             → wipes it252```253254`actions` mini-DSL, executed in order after initial load:255256```jsonc257[258  { "click": "#accept-cookies" },259  { "wait": 1500 },260  { "type": { "selector": "#search", "text": "query" } },261  { "press": "Enter" },262  { "scroll": 2000 },263  { "scrollToBottom": { "maxScrolls": 10, "delay": 500 } },264  { "waitForSelector": ".results-loaded" },265  { "evaluate": "document.querySelector('.more').click()" }266]267```268269`scrollToBottom` is the workhorse for infinite-scroll pages; implement it as scroll → wait → compare `scrollHeight` → repeat until stable or `maxScrolls`.270271### 4.7 Transport between Node and the daemon272273HTTP over loopback with keep-alive. Not Unix sockets (harder to debug), not stdio (framing pain).274Node client (`packages/fetcher-webkit`) must implement: 2 s connect timeout, single retry on `ECONNREFUSED` after a 1 s delay, circuit breaker that opens after 5 consecutive failures and routes everything to Tier 0 with a loud alert. **If the daemon is dead, the API must degrade rather than 500 across the board.**275276---277278## 5. Tier 2 spec — real Safari (`packages/fetcher-safari`)279280Last resort, for targets that defeat Tier 1 or require the real user's logged-in state.281282### 5.1 Two modes283284- **`safaridriver`** (W3C WebDriver) — preferred. Stable, scriptable, proper timeouts. **Hard limit: one session at a time on the machine.** Safari Technology Preview provides a second, independent session; that is the ceiling — two, total.285- **AppleScript** via `osascript` — fallback. Serial, fragile, breaks on focus changes, but drives the *real* Safari with the *real* user profile and all its logged-in sessions.286287### 5.2 Operating rules288289- Its own BullMQ queue, `concurrency: 1`, hard 60 s timeout, max 200 jobs/day by default.290- Never reachable via `tier: "auto"` for anonymous requests. Tier 2 requires either an explicit `tier: "safari"` or a `profileId` bound to the real profile.291- Between jobs: close the tab, do **not** clear cookies (that is the point), but do clear the URL bar and history entry if the customer is not the profile owner.292- Fails closed. If safaridriver reports a session conflict, queue the job rather than falling back to AppleScript mid-flight — mixing the two corrupts state.293- Log every Tier 2 job with the requesting API key. This is the audit trail you will need if a target complains.294295---296297## 6. Proxy & identity layer (`packages/egress`)298299One machine means one IP means one ban ends everything. Design for this from day one.300301- **Tier 0 and Tier 1 egress through rotating residential proxies.** Tier 2 uses the host IP directly — that is precisely its value.302- Proxy selection is **sticky per host** for the duration of a crawl job: switching IPs mid-crawl of one site looks far more suspicious than staying put.303- Health-check the proxy pool every 60 s against a known-good endpoint. Eject on 3 consecutive failures, re-test after 10 min.304- Track per-(proxy, host) success rate in Redis. Stop routing a proxy to a host once its rate drops below 50% over the last 20 attempts — it is burned there.305- Geo-pin when the target is geo-sensitive: `egressRegion` option on the API, defaulting to the account's region.306- WKWebView proxy configuration is set per-`WKWebsiteDataStore` via `proxyConfigurations` (macOS 14+). It cannot be changed on a live store, so the pool must hold **buckets of WebViews per proxy** and acquire from the matching bucket. Plan the pool shape accordingly: e.g. 4 proxies × 6 WebViews rather than 24 undifferentiated ones.307308---309310## 7. Session profiles (`packages/profiles`)311312A profile is a named, persistent browser identity: cookies, localStorage, and a pinned UA.313314- Created by `POST /v1/profiles` with either an explicit cookie array, or an interactive login flow (§7.2).315- Stored as a `WKWebsiteDataStore` keyed by UUID on disk, plus a Postgres row holding metadata (owner, target domains, created/last-used, expiry).316- **Cookies are secrets.** Encrypt the Postgres-side copy with a key from the macOS Keychain, never log them, redact them from error payloads, and exclude the profile directory from any backup that leaves the machine unencrypted.317- Profiles expire: default 30 days idle, then wiped. Re-auth is the user's job.318- A profile is bound to one account. Cross-account use is a hard error, not a warning.319320### 7.2 Interactive login flow321322For targets where cookie import is impractical: `POST /v1/profiles/:id/login` opens a visible WKWebView on `m3u96a`, the operator logs in by hand, the daemon detects navigation to a success URL and seals the profile. This is a manual, low-volume operation and that is fine — it is what makes Tier 2 valuable.323324---325326## 8. Extraction engine (`packages/extract`)327328Deterministic, versioned, testable. **No model inference anywhere in this package.**329330### 8.1 Pipeline331332```333raw HTML334  → encoding detection (charset header → <meta> → BOM → heuristic)335  → parse with linkedom336  → sanitize: strip <script> <style> <svg> <noscript>, ad iframes, tracking pixels337  → structured-data harvest (§8.3)  ─────────┐338  → boilerplate removal (§8.4)               │339  → relative → absolute URL rewriting        ├─► metadata340  → Turndown with custom rules (§8.5)        │341  → post-process                             │342  → markdown ────────────────────────────────┘343```344345Every stage is a pure function `(input, options) => output`. The whole pipeline is testable offline against saved HTML fixtures, which is the entire reason for choosing determinism.346347### 8.2 Boilerplate removal348349Two modes, selected by `onlyMainContent`:350351- **Readability** (`@mozilla/readability`) for article-shaped pages. Reliable on news, blogs, docs.352- **Density-based fallback** when Readability returns under 200 characters — which it does on listing pages, product pages, and dashboards. Implement a simple text-density scorer: for each block element, `textLength / (1 + linkTextLength)`, pick the subtree maximizing the sum. Do not skip this fallback; Readability silently failing to a near-empty result is the single most common extraction bug.353354Always strip, in both modes: `<header>`, `<footer>`, `<nav>`, `[role=navigation]`, `[aria-hidden=true]`, cookie banners (match on common class patterns), newsletter modals, and elements whose class matches `/(^|[-_])(ad|ads|advert|sponsor|promo|share|social|related|comment)([-_]|$)/i`.355356### 8.3 Structured data harvest357358This is where deterministic extraction earns its keep. Harvest, in priority order, and merge:3593601. **JSON-LD** (`<script type="application/ld+json">`) — parse all blocks, handle `@graph`, resolve `@type` to a normalized shape. Covers Article, Product, Recipe, Event, JobPosting, BreadcrumbList, Organization, FAQPage.3612. **Microdata** (`itemscope`/`itemprop`) and **RDFa** — older but still common on e-commerce.3623. **OpenGraph** and **Twitter cards** — title, description, image, type, publish time.3634. **Standard meta** — `description`, `author`, `keywords`, `canonical`, `hreflang` alternates.3645. **Heuristics** — `<time datetime>` for dates, `<h1>` for title, first `<p>` over 100 chars for description.365366Output as `metadata` plus a normalized `structured` object. A Product page should yield price, currency, availability, SKU, and images **without any model involved**. Roughly 60% of commercially interesting pages carry usable structured data; this is the highest-leverage code in the repo.367368### 8.4 Selector-based extraction (`/v1/extract` core)369370Users supply a schema where each field maps to an extractor:371372```jsonc373{374  "schema": {375    "title":  { "selector": "h1", "type": "text" },376    "price":  { "selector": ".price", "type": "number", "clean": "currency" },377    "images": { "selector": "img.product", "type": "attr", "attr": "src", "multiple": true },378    "specs":  { "selector": "table.specs", "type": "table" },379    "sku":    { "jsonld": "$.sku" },380    "body":   { "selector": "#desc", "type": "markdown" }381  }382}383```384385Types: `text`, `html`, `markdown`, `number`, `date`, `url`, `attr`, `table`, `list`, `boolean` (presence).386Cleaners: `currency`, `whitespace`, `trim`, `stripTags`, `parseDate` (with locale hint).387Sources: `selector` (CSS), `xpath`, `jsonld` (JSONPath into harvested JSON-LD), `regex`, `meta`.388389Field resolution order: `jsonld` → `meta` → `selector` → `xpath` → `regex`. First non-empty wins. This lets one schema work across sites with different markup.390391### 8.5 Repeated-structure inference (`/v1/extract` on list pages)392393For listing pages, users should not have to write selectors. Implement auto-detection:3943951. Compute a structural signature for every element: tag path + class set, normalized.3962. Find the deepest parent whose children contain **≥ 5 siblings sharing a signature**.3973. That parent is the list container; the siblings are the records.3984. Within one record, each leaf position becomes a candidate field; label it from `itemprop`, `class`, or a nearby `<dt>`/`<th>`.399400This handles search results, product grids, and job boards without configuration. Return the inferred selectors alongside the data so users can pin and refine them — inference that cannot be inspected and overridden is worse than useless.401402### 8.6 Markdown conversion rules403404Turndown with hand-written rules. Write these; the defaults are not good enough:405406| Input | Output |407|---|---|408| `<pre><code class="language-x">` | fenced block with language tag |409| `<pre>` without language | fenced block, attempt language detection from content |410| `<table>` | GFM table, pipes escaped, empty cells preserved, `<th>` → header row |411| nested/spanning tables | HTML passthrough (GFM cannot express them) |412| `<figure>` + `<figcaption>` | image followed by italic caption |413| `<dl>/<dt>/<dd>` | bold term + indented definition |414| heading anchors (`#`, `¶`, `.anchor`) | removed |415| `<img>` with `data-src`/`srcset` only | resolve to the highest-resolution real URL |416| `<br>` inside a table cell | `<br>` kept (newline breaks the table) |417| inline `<svg>` | removed, unless it has `<title>` → alt-text image reference |418| `<abbr title>` | text + parenthetical |419| MathML / `.katex` | `$...$` / `$$...$$` |420421Post-process: collapse 3+ blank lines to 2, trim trailing whitespace, normalize list markers to `-`, deduplicate consecutive identical links.422423### 8.7 BYOK LLM pass-through (optional, `packages/byok`)424425Thin and stateless. `/v1/extract` accepts:426427```jsonc428{ "llm": { "provider": "anthropic|openai|...", "apiKey": "sk-...", "model": "...", "prompt": "..." } }429```430431Tendril runs its normal deterministic pipeline, then forwards the resulting markdown plus the user's JSON schema to their provider with their key, and returns the response.432433Rules: the key is held in memory for the request only, never written to disk, never logged, never included in error payloads; chunk markdown over ~50k characters by heading and merge results; on provider error, return the deterministic result with `llm: { error }` rather than failing the whole request. Tendril is a pass-through, not an LLM vendor — do not build a model router, do not cache completions, do not offer a Tendril-supplied key.434435### 8.8 Versioning436437The extraction pipeline has a `pipelineVersion` (semver) stored with every cached result. Bumping it invalidates derived markdown but **not** the raw HTML cache, so you can re-extract the entire corpus after a rules improvement. This is the single most valuable property of keeping raw HTML forever.438439---440441## 9. Non-HTML content (`packages/extract/formats`)442443Content-Type routing, before any HTML logic:444445| Type | Handling |446|---|---|447| `application/pdf` | text layer extraction; if under 100 chars, flag `needsOCR` and skip (do not silently return empty) |448| `application/json` | pretty-print; if it looks like an API response, return as `json` directly |449| `text/plain`, `text/markdown` | pass through |450| `text/csv` | parse to a GFM table, cap at 500 rows |451| `application/xml`, RSS, Atom | parse feeds into items; this makes `/map` much stronger on blogs |452| `image/*` | metadata only, plus the blob; no OCR |453| `application/zip`, archives | reject with `ERR_UNSUPPORTED_TYPE` |454| anything > `maxSizeBytes` (default 20 MB) | reject before download completes, via streaming size check |455456Never let a Content-Type mismatch reach the HTML parser. A 200 MB video streamed into linkedom will take the process down.457458---459460## 10. Crawl frontier (`packages/frontier`)461462Non-negotiable rules:4634641. **Normalize before anything.** Lowercase host, strip default port, sort query params, remove `utm_*`/`fbclid`/`gclid`/`ref`/`mc_cid`/`_ga`, drop fragment (unless `#!` hashbang), resolve `..`, consistent trailing slash, punycode hosts. Dedup runs on the normalized form. Keep the original for output.4652. **Two-stage dedup.** In-memory Bloom filter (1% FP, sized `limit × 3`) → Postgres confirmation on hit. Avoids a DB round-trip per discovered link, which at 8 concurrent renders is thousands per minute.4663. **Per-host rate limiting**, never global. Token bucket in Redis, configurable `delayMs`, honor robots.txt `Crawl-delay` when higher. A crawl of one site must not slow another customer's job.4674. **Priority.** Strict BFS by depth, then heuristics: `/blog/`, `/docs/`, `/article/`, `/product/` ahead of `/tag/`, `/author/`, `/page/47`, `?sort=`, `?filter=`.4685. **Trap detection.** If more than 200 URLs on one host share a path pattern with only a varying numeric or date segment, sample instead of exhausting. Infinite calendars, paginated tag archives, and faceted-search URL explosions are the three classics; each can generate unbounded URLs.4696. **Backoff.** 429 or 503 → pause the entire host, exponential 2^n s, capped at 5 min, 3 attempts, then mark the host degraded for the job.4707. **Budgets.** Page count *and* wall-clock *and* total bytes. First limit reached ends the job cleanly with `stopReason`.4718. **Canonical awareness.** If a page declares `<link rel=canonical>` pointing elsewhere on the same host, record the mapping and do not crawl both. Saves 20-40% on most e-commerce sites.4729. **Sitemap seeding.** Always attempt sitemap discovery first, even for a crawl — starting from a complete URL list beats link-following, and it lets you report accurate progress instead of an unknown denominator.47310. **Resumability.** Frontier state lives in Postgres, not memory. A restart mid-crawl resumes; it does not start over.474475### 10.1 robots.txt476477Full RFC 9309 parsing: `User-agent` groups with correct specificity matching, `Allow`/`Disallow` with longest-match-wins, wildcards `*` and `$`, `Sitemap` directives, `Crawl-delay` (non-standard but honored). Cache per host for 24 h. On fetch failure treat as allow-all, but log it; on 5xx treat as disallow-all for 1 h, per the spec's intent.478479---480481## 11. Cache & storage (`packages/cache`)482483- **Content-addressed blob store.** `blobs/<sha256[0:2]>/<sha256[2:4]>/<sha256>`. Raw HTML, screenshots, PDFs. Deduplicates automatically across customers — the same page fetched by two accounts is stored once.484- **Metadata in Postgres**, pointing at blob hashes. Never store blobs in Postgres.485- **Cache key** = `sha256(normalizedURL + tier + profileId + relevantOptions)`. Options that do not change the fetch (output format, extraction schema) must **not** be in the key, or the hit rate collapses.486- `maxAge` on the request decides whether a hit is served. Default 0 (always fetch) for `/scrape`, 3600 for `/crawl` sub-pages.487- Respect `ETag` and `Last-Modified`: on a stale hit, revalidate with a conditional request. A 304 costs almost nothing and refreshes the entry.488- **Eviction:** LRU by last access, triggered when the blob store exceeds a configured ceiling (default 200 GB). Run it as a scheduled job, not inline.489- GDPR: cache entries carry a retention date; a purge job runs nightly (§26).490491---492493## 12. Data model494495Postgres 16, schema `tendril`, migrations via drizzle. Core tables:496497```498accounts        id, name, plan, created_at, egress_region499api_keys        id, account_id, hash (argon2id), prefix, scopes[], last_used_at, revoked_at500jobs            id, account_id, type(scrape|crawl|map|search|extract), status, params jsonb,501                created_at, started_at, finished_at, stop_reason, error_code502pages           id, job_id, url, normalized_url, canonical_url, depth, status_code,503                tier_used, escalation_reason, html_hash, markdown_hash, screenshot_hash,504                metadata jsonb, structured jsonb, pipeline_version, fetched_at, duration_ms505frontier        job_id, normalized_url, depth, priority, state(pending|active|done|failed),506                attempts, next_attempt_at            -- PK (job_id, normalized_url)507domain_hints    host, preferred_tier, requires_profile, crawl_delay_ms, success_rate,508                last_updated                          -- the learned routing table509profiles        id, account_id, name, domains[], data_store_path, cookies_enc,510                created_at, last_used_at, expires_at511robots_cache    host, body, fetched_at, expires_at512usage           account_id, day, pages_by_tier jsonb, bytes_out, llm_passthrough_calls513webhooks        id, job_id, url, events[], secret, deliveries jsonb514```515516Indexes that matter: `frontier(job_id, state, priority)` for the dequeue path, `pages(job_id, fetched_at)` for pagination, `pages(html_hash)` for dedup lookups, partial index on `frontier(next_attempt_at) WHERE state='failed'`.517518---519520## 13. Queue & job lifecycle521522BullMQ on Redis. Queues:523524| Queue | Concurrency | Notes |525|---|---|---|526| `fetch:http` | 32 | Tier 0 |527| `fetch:webkit` | 24 | matches the pool size exactly |528| `fetch:safari` | **1** | never raise this |529| `crawl:control` | 8 | one job per active crawl, manages its frontier |530| `extract` | 16 | CPU-bound, pure |531| `webhook` | 8 | with retries |532| `maintenance` | 1 | cache eviction, purges, robots refresh |533534Job lifecycle: `queued → running → (succeeded | failed | cancelled)`.535Retries: 3 attempts, exponential backoff with jitter, only for transient error codes (§15). Never retry `ERR_ROBOTS_DENIED`, `ERR_SSRF_BLOCKED`, `ERR_UNSUPPORTED_TYPE`, or any 4xx other than 408/429.536537**Idempotency:** `/scrape`, `/crawl`, `/extract` accept an `Idempotency-Key` header. Store key → jobId for 24 h; a repeat returns the original job rather than starting a new one. Customers with retry logic will otherwise double-bill themselves and blame you.538539Cancellation is cooperative: set a flag in Redis, workers check it between pages, in-flight renders complete. Never kill a WebView to cancel — it corrupts the pool.540541---542543## 14. API contracts544545Base: `https://www.ten-dril.com/v1`. Auth: `Authorization: Bearer tdr_live_...`.546All responses: `{ success: boolean, data?, error?: { code, message, details? } }`.547All list responses are cursor-paginated: `{ data: [...], next?: "cursor" }`. No offset pagination.548549### 14.1 `POST /v1/scrape`550551```jsonc552{553  "url": "https://example.com/article",554  "formats": ["markdown", "html", "links", "screenshot", "structured", "extract"],555  "extractSchema": { /* §8.4 */ },556  "llm": { /* §8.7, optional */ },557  "tier": "auto",                 // auto | http | webkit | safari558  "profileId": "uuid",            // forces tier >= 1559  "actions": [ /* §4.6 */ ],560  "onlyMainContent": true,561  "includeTags": ["article", "main"],562  "excludeTags": [".sidebar", "#comments"],563  "waitFor": 0,564  "timeout": 30000,565  "maxAge": 0,566  "headers": { /* custom, merged over defaults */ },567  "egressRegion": "us",568  "blockResources": ["image", "media", "font"]569}570```571572Response `data`: `{ markdown, html, rawHtml, links[], structured, extract, screenshot, metadata, tierUsed, cached, timings }`.573574`timings`: `{ total, dns, connect, ttfb, download, render, extract, escalations[] }`. Expose this — customers debugging slow crawls will otherwise blame you for their target's latency.575576`links[]` is `{ url, text, rel, isInternal }`, deduplicated, absolute.577578### 14.2 `POST /v1/crawl`579580```jsonc581{582  "url": "https://example.com",583  "limit": 1000,584  "maxDepth": 4,585  "maxDurationSeconds": 3600,586  "maxBytes": 5368709120,587  "includePaths": ["^/blog/.*"],588  "excludePaths": ["\\.pdf$", "^/tag/"],589  "allowSubdomains": false,590  "allowBackwardLinks": false,591  "allowExternalLinks": false,592  "concurrency": 8,593  "delayMs": 250,594  "respectRobots": true,595  "deduplicateSimilar": true,596  "scrapeOptions": { /* §14.1 minus url */ },597  "webhook": { "url": "...", "events": ["page","completed","failed"], "secret": "..." }598}599```600601→ `202 { jobId, statusUrl, streamUrl }`.602603- `GET /v1/crawl/:id` → `{ status, total, completed, failed, creditsUsed, data[], next }`604- `GET /v1/crawl/:id/stream` → SSE, one event per page plus a terminal event. Heartbeat every 15 s or ngrok will drop the connection.605- `DELETE /v1/crawl/:id` → cooperative cancel, returns partial results.606- `GET /v1/crawl/:id/errors` → per-URL failures with codes. Customers need this and Firecrawl handles it poorly.607608Webhook payloads are signed: `X-Tendril-Signature: sha256=<hmac(secret, body)>`, plus `X-Tendril-Timestamp` for replay protection. Retry 5 times with exponential backoff, then mark the webhook degraded.609610### 14.3 `POST /v1/map`611612URL discovery **without rendering**. Target: under 3 s for a normal site.613Sources merged and deduplicated: `sitemap.xml` (recursive through sitemap indexes, `.gz` support, cap at 50k URLs), `robots.txt` Sitemap directives, homepage links, `/llms.txt`, RSS/Atom feeds.614Options: `search` (substring filter, ranked by match position), `limit`, `includeSubdomains`, `sitemapOnly`, `ignoreSitemap`.615Returns `{ links: [{ url, title?, lastModified?, source }] }` — `source` tells the user where each URL came from, which builds trust in the result.616617### 14.4 `POST /v1/search`618619```jsonc620{ "query": "...", "limit": 10, "lang": "en", "country": "us", "timeRange": "month",621  "scrapeResults": false, "scrapeOptions": {} }622```623624Pipeline: SearXNG (self-hosted, §16.1) → dedup by domain+normalized title → rank → optionally scrape the top N in parallel (Tier 0 forced, 8 s timeout, partial failures tolerated and reported per-result).625626Be explicit in the docs: Tendril has no index of its own. Search quality is SearXNG's quality. Under-promise here.627628### 14.5 `POST /v1/extract`629630```jsonc631{632  "urls": ["https://example.com/products/*"],   // wildcards expand via /map633  "schema": { /* §8.4 */ },634  "inferList": true,                            // §8.5 auto-detection635  "llm": { /* §8.7, optional */ },636  "scrapeOptions": {}637}638```639640Returns one record per matched URL, or one aggregated array when `inferList` finds a list page. Include the resolved selectors in the response so users can pin them.641642### 14.6 Ancillary643644- `POST /v1/profiles`, `GET /v1/profiles`, `DELETE /v1/profiles/:id`, `POST /v1/profiles/:id/login`645- `GET /v1/usage?from=&to=` — pages by tier, bytes, cost646- `GET /healthz` — unauthenticated, shallow647- `GET /v1/status` — authenticated, per-tier health (§18.3)648649Rate limits returned on every response: `X-RateLimit-Limit`, `-Remaining`, `-Reset`.650651---652653## 15. Error taxonomy654655One file maps codes to HTTP statuses and retry-ability. Never invent an ad-hoc string.656657| Code | HTTP | Retryable | Meaning |658|---|---|---|---|659| `ERR_INVALID_URL` | 400 | no | malformed or unsupported scheme |660| `ERR_SSRF_BLOCKED` | 400 | no | resolved to a private or forbidden address |661| `ERR_UNSUPPORTED_TYPE` | 415 | no | content type not handled |662| `ERR_TOO_LARGE` | 413 | no | exceeded `maxSizeBytes` |663| `ERR_UNAUTHORIZED` | 401 | no | bad or revoked key |664| `ERR_QUOTA_EXCEEDED` | 402 | no | plan limit hit |665| `ERR_RATE_LIMITED` | 429 | yes | our limit, not the target's |666| `ERR_ROBOTS_DENIED` | 403 | no | robots.txt disallows |667| `ERR_TARGET_BLOCKED` | 502 | maybe | target returned a challenge at every tier |668| `ERR_TARGET_4XX` / `_5XX` | 502 | 5xx only | upstream status passed through |669| `ERR_TIER_TIMEOUT` | 504 | yes | render exceeded timeout |670| `ERR_POOL_EXHAUSTED` | 503 | yes | no WebView available in 30 s |671| `ERR_DAEMON_DOWN` | 503 | yes | Swift daemon unreachable, circuit open |672| `ERR_SAFARI_BUSY` | 503 | yes | Tier 2 session conflict |673| `ERR_PROFILE_EXPIRED` | 409 | no | session profile needs re-auth |674| `ERR_EXTRACT_FAILED` | 422 | no | schema produced no fields |675| `ERR_REDIRECT_LOOP` | 502 | no | same URL twice in a chain |676| `ERR_INTERNAL` | 500 | yes | a bug; page it |677678Every error response carries a `requestId` that appears in the logs. Support requests without one are unanswerable.679680---681682## 16. Deployment — node `m3u96a` + ngrok683684Single host. No Kubernetes, no cloud VM. Ingress is an ngrok tunnel on a reserved domain.685686### 16.1 Topology687688```689   Internet690      │691      ▼692 ngrok edge  ──── TLS terminated here, certificate managed by ngrok693 www.ten-dril.com694      │  (encrypted tunnel, outbound connection initiated by the Mac)695      ▼696 ┌──────────────────────── m3u96a ────────────────────────┐697 │  ngrok agent (LaunchAgent)                             │698 │        │                                               │699 │        ▼                                               │700 │  Caddy :8080 ── reverse proxy, routing, access logs    │701 │        ├─► Fastify API            :3000                │702 │        └─► /healthz                                    │703 │                                                        │704 │  Internal only, bound to 127.0.0.1, never tunneled:    │705 │    tendril-worker (Swift)         :8787                │706 │    Redis                          :6379                │707 │    PostgreSQL                     :5432                │708 │    SearXNG                        :8888                │709 │    Prometheus / Grafana           :9090 / :3001        │710 └────────────────────────────────────────────────────────┘711```712713Caddy sits between ngrok and the API deliberately: one place for routing, access logs, and local rate limiting, and only one port is ever tunneled.714715### 16.2 ngrok configuration716717`deploy/ngrok.yml`:718719```yaml720version: "3"721agent:722  authtoken: ${NGROK_AUTHTOKEN}723  log: /usr/local/var/log/ngrok.log724  log_level: info725  connect_timeout: 10s726endpoints:727  - name: tendril-api728    url: https://www.ten-dril.com729    upstream:730      url: 8080731    traffic_policy:732      inbound:733        - actions:734            - type: rate-limit735              config:736                name: global737                algorithm: sliding_window738                capacity: 600739                rate: 60s740        - expressions:741            - "req.url.path.startsWith('/internal')"742          actions:743            - type: deny744              config:745                status_code: 404746```747748DNS: `CNAME` for `www.ten-dril.com` → the target ngrok returns when you reserve the domain. Reserve the apex `ten-dril.com` too and 301 it to `www` **at the ngrok edge**, not in the application.749750A reserved domain requires a paid ngrok plan; free-tier URLs rotate on restart and are unusable for a customer-facing API. Confirm current plan tiers and Traffic Policy feature availability on ngrok's site before building against a specific capability — this changes.751752### 16.3 Process supervision753754LaunchAgents (not LaunchDaemons — WebKit requires a user GUI context), in `~/Library/LaunchAgents/`:755756| Label | Process | Notes |757|---|---|---|758| `com.tendril.caffeinate` | `caffeinate -dimsu` | first to start, mandatory |759| `com.tendril.postgres` | postgres | |760| `com.tendril.redis` | redis-server | |761| `com.tendril.worker` | Swift daemon | needs GUI session, `KeepAlive: true` |762| `com.tendril.searxng` | uvicorn/docker | |763| `com.tendril.api` | `node apps/api/dist/main.js` | |764| `com.tendril.caddy` | `caddy run` | |765| `com.tendril.ngrok` | `ngrok start --all` | **last**, gated on health |766767Set `SoftResourceLimits: { NumberOfFiles: 65536 }` in every plist, or the pool saturates around 40 connections.768769Start ordering: `com.tendril.ngrok` runs a wrapper that polls `http://127.0.0.1:8080/healthz` until healthy (max 120 s) before starting the tunnel. Exposing an unhealthy API is worse than a short outage.770771### 16.4 Deploy procedure (`scripts/deploy.sh`)7727731. `git pull` on a deploy branch, never on a dirty tree7742. `pnpm install --frozen-lockfile && pnpm build && swift build -c release`7753. `pnpm db:migrate` — migrations must be backward-compatible for one version, so a rollback does not need a down-migration7764. Drain: stop accepting new jobs, wait up to 60 s for in-flight ones7775. `launchctl kickstart -k` each agent in dependency order7786. `./scripts/health.sh` — all tiers, all services7797. **Smoke test through the public URL**, not localhost: scrape a known page via `https://www.ten-dril.com/v1/scrape` and assert the markdown contains an expected string7808. On failure: `git checkout` previous tag, repeat from 2. Keep the last 3 builds on disk.781782### 16.5 Security posture783784Exposing a home machine to the internet deserves more care than a cloud VM.785786- **Only port 8080 is tunneled.** 8787, 6379, 5432, 8888, 9090 bind to `127.0.0.1` explicitly. Verify with `lsof -iTCP -sTCP:LISTEN` after every deploy.787- API keys hashed with argon2id, prefix stored in clear for identification, full key shown once at creation.788- Quotas enforced in-app **and** a global rate limit at the ngrok edge, so a burst never reaches the Mac.789- Grafana and any admin surface: IP allowlist in Traffic Policy, or a separate non-public tunnel.790- **SSRF is the top risk of any scraping API.** Users control the target URL. Enforce: block RFC1918, loopback, link-local, `169.254.169.254`, IPv6 ULA and mapped-IPv4; allow only `http`/`https`; allow only ports 80/443/8080/8443. **Resolve DNS first and validate the resolved IP**, then connect to that IP with the Host header set — otherwise DNS rebinding walks straight into Redis and Postgres. Re-validate on every redirect hop.791- Redact from logs: `Authorization`, `Cookie`, `Set-Cookie`, BYOK keys, webhook secrets, profile cookie blobs.792- macOS firewall on, `Remote Login` restricted to key auth on a non-default port, FileVault enabled.793794### 16.6 Failure modes specific to this setup795796These are what will actually take the service down. Treat them as design constraints.797798- **The tunnel is a single point of failure.** Sleep, Wi-Fi loss, or reboot means total outage. External uptime monitor on `https://www.ten-dril.com/healthz` every 60 s, alerting to your phone. Nothing else in this document matters if you find out from a customer.799- **Residential upload bandwidth is the real ceiling.** Screenshots and large HTML saturate a typical uplink long before CPU. Gzip everything (Caddy), cap screenshot dimensions, and serve large blobs via short-lived signed links rather than inline base64.800- **Reboots break the GUI session.** Without auto-login, Tiers 1 and 2 die while Tier 0 keeps returning bot-blocked pages. That looks like a quality regression, not an outage — which is why health checks must probe each tier independently (§18.3).801- **ngrok reconnects silently but not instantly.** Expect 5-30 s gaps on network flaps. SDKs must retry with backoff; document it.802- **One IP, one ban surface.** See §6.803- **Thermal throttling.** 24 WebViews rendering continuously will heat an M3 and reduce clocks. Monitor `powermetrics` and cap the pool lower if sustained throughput drops.804- **Backups.** Postgres and the blob CAS live on one SSD. Nightly `pg_dump` plus a CAS rsync to an external encrypted volume, offsite weekly, **and a tested restore**. An untested backup is not a backup.805806---807808## 17. macOS prerequisites809810Scripted in `scripts/setup-mac.sh`, idempotent, but several steps require a human.811812- **A GUI session is mandatory.** WKWebView and Safari will not render without a window server session. SSH alone is insufficient. Enable auto-login; connect via Screen Sharing *after* the graphical session exists.813- **`caffeinate -dimsu`** as a LaunchAgent. Sleep kills the pool.814- **Screen lock and screen saver disabled.** A locked screen suspends rendering in some WebViews.815- **Safari → Develop → "Allow JavaScript from Apple Events"** must be ticked by hand. Not scriptable.816- **`safaridriver --enable`** once, with an admin password.817- **Automation and Accessibility permissions:** the first `osascript` triggers a TCC dialog requiring a click. No clean bypass. Document it in the runbook.818- **Full Disk Access** for the terminal and the daemon if touching the Safari profile directory.819- Disable Spotlight indexing on the blob store (`mdutil -i off`) — it will otherwise index millions of files.820- Increase `kern.maxfiles` and `kern.maxfilesperproc` via a `sysctl` plist.821822---823824## 18. Observability825826### 18.1 Logs827828Pino, JSON, to stdout and a rotated file. Every line carries `{ requestId, jobId, accountId, url, tier }` where applicable. No `console.log` anywhere, enforced by lint rule.829830### 18.2 Metrics (Prometheus, scraped locally)831832- `tendril_requests_total{endpoint,status}`833- `tendril_fetch_duration_seconds{tier}` — histogram834- `tendril_escalations_total{from,to,reason}` — **the most important metric in the system**835- `tendril_pool_slots{state}` — gauge: busy/idle/recycling836- `tendril_extraction_empty_total{reason}` — pages yielding under 200 chars; a rising line here means a silent quality regression837- `tendril_proxy_success_rate{proxy,host}`838- `tendril_tunnel_up` — 0/1 from the agent's local API839- `tendril_queue_depth{queue}`, `tendril_job_duration_seconds{type}`840841### 18.3 Health checks842843`GET /v1/status` probes each tier independently against a known-stable control URL and returns per-tier latency and last success. `/healthz` stays shallow and fast for the uptime monitor.844845Alert on: tunnel down, daemon circuit open, escalation rate above 40% (a target changed its defenses), extraction-empty rate above 10%, queue depth growing for 10 minutes, disk above 85%.846847---848849## 19. Performance targets850851Measured on m3u96a, `pnpm bench` against 100 reference URLs. Regressions block a release.852853| Metric | Target |854|---|---|855| Tier 0 p50 / p95 | 60 ms / 400 ms |856| Tier 1 p50 / p95 | 550 ms / 2.5 s |857| Escalation rate | < 25% |858| Extraction (HTML → markdown), 500 KB page | < 120 ms |859| `/map` on a 5k-URL sitemap | < 3 s |860| Sustained crawl throughput | 40 pages/s Tier 0, 12 pages/s Tier 1 |861| Memory, full pool, steady state | < 14 GB |862| Cold start to first successful render | < 45 s |863864---865866## 20. Testing867868- **Unit:** URL normalization, `shouldEscalate`, Turndown rules, robots parsing, SSRF validation, structural inference. Fast, hermetic, no network. These are the majority of the suite.869- **Fixtures:** `test/fixtures/html/` with ~60 saved real pages — React SPA, Next.js, WordPress, Shopify, static docs, a Cloudflare interstitial, a paginated listing, a page with nested tables, a page with JSON-LD, one with broken encoding. Extraction is tested only here.870- **Golden files** for markdown output. Any rule change must show its diff and be reviewed.871- **Contract tests** on the API schemas — the OpenAPI document is generated from Zod, so schema drift breaks the build.872- **E2E** against 20 stable public sites, tagged `@slow`, excluded from CI. Asserts escalation rates and non-empty extraction, never exact content.873- **Deploy smoke test** through the public URL (§16.4). Half of all outages are ingress, not application.874- **Chaos:** kill the Swift daemon mid-crawl and assert the circuit breaker degrades to Tier 0 instead of failing the job. Do this in CI with a fake daemon.875876Coverage target: 85% on `packages/`, no target on `apps/`.877878---879880## 21. Code conventions881882- TypeScript strict, `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`. No `any`; `unknown` plus narrowing.883- **No exceptions for normal control flow.** Fetchers and extractors return `Result<T, TendrilError>`. Exceptions signal bugs and are caught only at the top-level handler.884- Every error carries a code from §15. The code → HTTP mapping lives in exactly one file.885- Pure functions wherever possible; I/O confined to `fetcher-*` and `cache`. This is what makes the fixture-based tests viable.886- No comments restating code. Comment the *why*, especially for site-specific workarounds — and date them, they rot within months.887- Swift: `async/await` only, no completion handlers. The pool is an `actor`. No force-unwraps outside tests.888- Commits: conventional commits. A commit touching extraction rules must include the golden-file diff.889890---891892## 22. SDKs & docs893894- **TypeScript SDK** — generated types from the OpenAPI schema, hand-written ergonomics on top. Must handle: automatic retry with backoff (for tunnel flaps, §16.6), SSE streaming for crawls with reconnect, idempotency keys generated by default.895- **Python SDK** — same surface, sync and async clients.896- Docs site with runnable examples per endpoint, plus a **migration guide from Firecrawl** mapping their parameter names to yours. Compatibility of naming where it costs nothing is worth more than elegance.897- Publish the OpenAPI document at `https://www.ten-dril.com/openapi.json`.898899---900901## 23. Repository layout902903```904tendril/905├── CLAUDE.md906├── package.json                 # pnpm workspaces907├── turbo.json908├── apps/909│   ├── api/                     # Fastify — public surface910│   │   ├── src/routes/{scrape,crawl,map,search,extract,profiles,usage}.ts911│   │   ├── src/schemas/         # Zod → generated OpenAPI912│   │   ├── src/auth/            # keys, quotas, rate limits913│   │   └── src/errors.ts        # §15 mapping, single source of truth914│   └── worker/                  # BullMQ consumers915├── packages/916│   ├── router/                  # tier decision, escalation, domain_hints917│   ├── fetcher-http/            # Tier 0918│   ├── fetcher-webkit/          # Swift daemon client + circuit breaker919│   ├── fetcher-safari/          # Tier 2920│   ├── egress/                  # proxy pool, SSRF validation921│   ├── profiles/                # session management922│   ├── extract/                 # deterministic pipeline923│   │   ├── src/structured/      # JSON-LD, microdata, OG924│   │   ├── src/boilerplate/     # Readability + density fallback925│   │   ├── src/markdown/        # Turndown rules926│   │   ├── src/selectors/       # schema-driven extraction927│   │   ├── src/infer/           # repeated-structure detection928│   │   └── src/formats/         # PDF, CSV, XML, feeds929│   ├── byok/                    # optional LLM pass-through930│   ├── frontier/                # crawl logic, robots, sitemaps931│   ├── cache/                   # CAS + metadata932│   └── shared/                  # types, errors, logger, Result933├── native/934│   └── tendril-worker/935│       ├── Package.swift936│       └── Sources/TendrilWorker/{main,Pool,Renderer,Server,Actions,Profiles}.swift937├── deploy/938│   ├── ngrok.yml939│   ├── Caddyfile940│   ├── launchagents/*.plist941│   └── runbook.md942├── sdks/{typescript,python}/943├── test/fixtures/944└── scripts/{setup-mac,deploy,health,backup}.sh945```946947---948949## 24. Roadmap950951**Phase 1 — foundations (2 wks).** Tier 0, extraction pipeline (§8.1-8.3, 8.6), `/scrape`, CAS cache, error taxonomy. Must handle 80% of the fixture set. Ship nothing publicly yet.952953**Phase 2 — rendering (3 wks).** Swift daemon, pool, escalation router, `domain_hints`, `/screenshot`. Targets: escalation under 25%, Tier 1 p95 under 2.5 s.954955**Phase 3 — crawl (2 wks).** Frontier, BullMQ, `/crawl` with SSE and signed webhooks, `/map`, resumability.956957**Phase 4 — extraction depth (3 wks).** Selector schemas (§8.4), structure inference (§8.5), non-HTML formats (§9), `/extract`. **This is the phase that differentiates the product — do not cut it.** Optional BYOK at the end.958959**Phase 5 — stealth & identity (2 wks).** Proxy layer, Tier 2 Safari, session profiles, `/search` via SearXNG.960961**Phase 6 — production (2 wks).** Deploy on `m3u96a`, ngrok reserved domain, quotas, billing, observability, SDKs, docs, migration guide.962963Do not start Phase 4 before Phase 2's escalation rate target is met. Extraction quality on pages you cannot fetch is worth nothing.964965---966967## 25. Anti-patterns — do not do this968969- ❌ Driving Safari via AppleScript for volume. Serial, breaks on focus change. A last resort, not a foundation.970- ❌ Spawning a `WKWebView` per request. Init costs ~800 ms. Pool or nothing.971- ❌ Using jsdom. linkedom is 5-10× faster on this workload and the difference is the whole extraction budget.972- ❌ Storing markdown without the source HTML. You will never re-extract after improving rules (§8.8).973- ❌ Putting output-format options in the cache key. Hit rate collapses.974- ❌ A global rate limit instead of per-host. You will get banned from one site while crawling another fast.975- ❌ Trusting `networkidle` alone as a render signal. Many pages poll forever.976- ❌ Letting Readability's empty result pass through silently. Always run the density fallback.977- ❌ Ignoring robots.txt by default. Explicit opt-out, logged per client, never implicit.978- ❌ Putting Tier 2 in the general queue. One stuck session freezes everything.979- ❌ Validating the hostname instead of the resolved IP for SSRF. That is a rebinding hole straight into your database.980- ❌ Tunneling any port other than 8080.981- ❌ Assuming the tunnel is up. Verify after every deploy, monitor continuously.982- ❌ Adding a local model "just for the hard cases". That decision reverses §1 and turns a testable system into a probabilistic one. If it becomes necessary, do it as an explicit, documented rewrite — not as a quiet addition.983984---985986## 26. Legal & ethical987988Not decorative. This is real product risk.989990- `respectRobots: true` is the default. Disabling requires an explicit flag, is logged per client, and is refused entirely for domains on an internal blocklist.991- Honest rate limiting: minimum 100 ms `delayMs` per host, not overridable.992- Identifiable User-Agent with a contact URL (`+https://www.ten-dril.com/bot`) and a published page explaining who you are and how to block you. Stealth means matching a real browser's fingerprint, not hiding your identity.993- Scraping authenticated pages (Tier 2) puts the user on the hook for the target's ToS. State it plainly and require acceptance at signup.994- **No CAPTCHA solving, no paywall bypass, no login-credential stuffing.** Commercial red lines as much as ethical ones.995- Personal data collected incidentally has a retention period; the nightly purge job enforces it. Support deletion requests by URL and by domain.996- Honor `noindex`/`nofollow` as a signal for crawl scope even though they are indexing directives — it costs little and is defensible.997- Running a public service from a residential connection may conflict with your ISP's terms on operating servers. Check the contract before going live on `www.ten-dril.com`.998999I am not a lawyer. Have this framework reviewed by one before commercializing, particularly regarding the CFAA and state computer-crime statutes (US), the Database Directive and GDPR (EU), and the terms of the specific high-value targets you intend to support.1000