# CLAUDE.md — Tendril

> Web ingestion platform (search / scrape / crawl / map) running on macOS Apple Silicon.
> Built to beat Firecrawl on three axes: **stealth**, **access to authenticated pages**, and
> **deterministic extraction quality**.

This file is the project's persistent context. Read it fully before making any change.

**Production host:** `m3u96a` (Mac, M3, 96 GB unified memory) — single node, self-hosted.
**Public endpoint:** `https://www.ten-dril.com` via ngrok reserved domain.

---

## Table of contents

| § | Section |
|---|---|
| 0 | Name & identity |
| 1 | Product thesis |
| 2 | Architecture — three-tier router |
| 3 | Tier 0 spec — HTTP |
| 4 | Tier 1 spec — WKWebView pool |
| 5 | Tier 2 spec — real Safari |
| 6 | Proxy & identity layer |
| 7 | Session profiles |
| 8 | Extraction engine (deterministic) |
| 9 | Non-HTML content |
| 10 | Crawl frontier |
| 11 | Cache & storage |
| 12 | Data model |
| 13 | Queue & job lifecycle |
| 14 | API contracts |
| 15 | Error taxonomy |
| 16 | Deployment — m3u96a + ngrok |
| 17 | macOS prerequisites |
| 18 | Observability |
| 19 | Performance targets |
| 20 | Testing |
| 21 | Code conventions |
| 22 | SDKs & docs |
| 23 | Repository layout |
| 24 | Roadmap |
| 25 | Anti-patterns |
| 26 | Legal & ethical |

---

## 0. Name & identity

Codename: **Tendril** — the climbing shoot that latches on and grows.
Domain: `ten-dril.com`.

If the name changes, update: `package.json` names, Swift bundle IDs (`com.tendril.worker`), Redis key prefix (`tdr:*`), Postgres schema name, ngrok config, API key prefix (`tdr_live_` / `tdr_test_`), LaunchAgent labels, and the User-Agent string.

---

## 1. Product thesis

Firecrawl runs in the cloud, on datacenter IPs, in headless Chromium. Tendril runs on a **residential machine**, in **WebKit** (Safari's actual engine).

**There is no local LLM in this system.** Extraction is deterministic: parsers, selectors, and structural inference. This is a deliberate choice with real consequences, so be honest about them:

- ✅ Predictable cost, predictable latency, reproducible output. The same page always yields the same result — you can write golden-file tests against extraction, which is impossible with a sampling model.
- ✅ No 20 GB resident model, no 40 s cold start, no GPU contention with the WebView pool.
- ❌ No zero-shot extraction from arbitrary prose. If a user wants "the CEO's name" from an unstructured about-page, deterministic parsing will not find it reliably.

The gap in ❌ is covered by **BYOK** (§8.7): the user supplies their own LLM API key, Tendril passes the cleaned markdown through and returns the result. Tendril never pays for inference and never stores the key beyond the request.

So the three real differentiators are:

1. **Browser fidelity.** WebKit on macOS ARM behind a residential IP is not a headless Chromium signature. Sites that block Firecrawl outright often do not block Tendril at all.
2. **Authenticated pages.** Tier 2 drives a real Safari profile with real logged-in sessions. No cloud service can offer this.
3. **Extraction you can test.** Deterministic output, versioned rules, golden files. Enterprise buyers care about reproducibility more than about magic.

**Accepted constraint:** one machine, one IP, no native horizontal scaling.
Tendril is not built to crawl 10M pages/day. It is built to crawl 100k pages/day *that nobody else can reach*, with output you can regression-test.

---

## 2. Architecture — three-tier router

The core of the system is an **escalation router**. Always start at the cheapest tier; escalate only on evidence of failure.

```
                       ┌──────────────┐
   request ──────────► │   Router     │
                       └──────┬───────┘
                              │  shouldEscalate()
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
   ┌─────────┐          ┌───────────┐        ┌──────────────┐
   │ TIER 0  │          │  TIER 1   │        │   TIER 2     │
   │ undici  │─fail────►│ WKWebView │─fail──►│   Safari     │
   │  HTTP   │          │   pool    │        │ safaridriver │
   └────┬────┘          └─────┬─────┘        └──────┬───────┘
        │                     │                     │
        └─────────────────────┴─────────────────────┘
                              │  raw HTML + metadata
                              ▼
                     ┌──────────────────┐
                     │ Extraction engine │
                     └──────────────────┘
```

| | Tier 0 | Tier 1 | Tier 2 |
|---|---|---|---|
| Engine | undici | WKWebView | Safari + safaridriver |
| Latency p50 | 50 ms | 500 ms | 3-8 s |
| Concurrency | 200 | 24 | **1** |
| JS execution | no | yes | yes |
| Cookies/session | per-request | per-profile | real user profile |
| Share of traffic | 80% | 18% | 2% |
| Cost per page | ~0 | ~15 MB·s RAM | a human-scale amount of time |

**Escalation is one-way and capped.** A request escalates at most twice. Every escalation is recorded with its reason so you can see, per domain, which tier actually works — and then pin that domain in `domain_hints` so future requests skip straight to it. The pinning table is what makes the system fast over time; without it you pay the escalation cost forever.

---

## 3. Tier 0 spec — HTTP (`packages/fetcher-http`)

### 3.1 Client configuration

`undici.Agent` with:

```ts
{
  connections: 64,
  pipelining: 1,              // pipelining breaks on many CDNs, leave at 1
  keepAliveTimeout: 30_000,
  keepAliveMaxTimeout: 120_000,
  connect: { timeout: 8_000, rejectUnauthorized: true },
  maxRedirections: 5,         // handled manually, see 3.3
  bodyTimeout: 20_000,
  headersTimeout: 10_000,
}
```

### 3.2 Header fidelity

Header **order** is a fingerprint. Send them in Safari's order, not alphabetically, and not in the order a JS object happens to iterate:

```
Host
Accept                    text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language           en-US,en;q=0.9      (or match the target's likely locale)
Accept-Encoding           gzip, deflate, br
User-Agent                Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 ...
                          + " Tendril/1.0 (+https://www.ten-dril.com/bot)"
Connection                keep-alive
Sec-Fetch-Dest            document
Sec-Fetch-Mode            navigate
Sec-Fetch-Site            none
Sec-Fetch-User            ?1
Upgrade-Insecure-Requests 1
```

The bot suffix in the UA is non-negotiable (§26). Stealth here means *not looking broken*, not lying about who you are.

TLS: Node's default cipher order differs from Safari's, which is a JA3-visible signal. If a target fingerprints TLS, that target belongs on Tier 1 anyway — do not try to fix it at Tier 0.

### 3.3 Redirect handling

Handle redirects manually rather than letting undici follow them, because you need to:

- record the full chain in `metadata.redirects[]` (users need it for canonicalization);
- re-run SSRF validation on **every** hop, not just the first (§16.5);
- detect meta-refresh and JS redirects in the body, which undici cannot see;
- stop on a redirect loop (same normalized URL twice) with `ERR_REDIRECT_LOOP`.

### 3.4 Escalation decision

`shouldEscalate(response): TierDecision` — a pure function, unit tested, no I/O.

| Signal | Threshold | Weight |
|---|---|---|
| Status 403 / 429 / 503 | — | decisive |
| Body < 2 KB with empty `#root`, `#__next`, `#app`, `[ng-version]` | — | decisive |
| Body contains `challenge-platform`, `cf-browser-verification`, `_Incapsula_`, `px-captcha`, `datadome` | — | decisive |
| Text/HTML ratio | < 0.05 | strong |
| `<noscript>` containing "enable JavaScript" / "JavaScript is required" | — | strong |
| Zero `<a href>` on a page over 10 KB | — | moderate |
| `<title>` matching `/^(just a moment|attention required|access denied)/i` | — | decisive |
| Content-Type is not HTML | — | route to §9, do not escalate |

Decisive → escalate. Two or more strong/moderate → escalate. Return the reason string; it goes in `timings.escalationReason` and into the metrics label.

---

## 4. Tier 1 spec — WKWebView pool (`native/tendril-worker/`)

**This is the heart of the system.** A Swift daemon exposing HTTP on `127.0.0.1:8787`.

### 4.1 Why native Swift, not Playwright-WebKit

Playwright ships a patched WebKit build with detectable artifacts: modified `navigator.webdriver` handling, injected bindings, altered timing. A native `WKWebView` is the *same binary* as Safari — same CFNetwork stack, same TLS, same font cache, same media codecs. That fidelity is the product.

### 4.2 Process model

- **One process, N WebViews.** The daemon **must** run as a real AppKit app (`NSApplication` with `.accessory` activation policy), not a plain CLI, or WebKit refuses to lay out and render.
- Pool of **24** on m3u96a. Each WebView is 150-400 MB resident. Measure with `footprint` before raising it; the number is bounded by memory, not cores.
- The pool is a Swift `actor`. Acquire/release with an async semaphore, FIFO waiters, 30 s acquisition timeout → `ERR_POOL_EXHAUSTED`.
- **Recycle aggressively.** Destroy and recreate a WebView after 50 renders or 10 minutes of life, whichever comes first. WebKit leaks slowly under adversarial pages, and a recycled view has a clean JS heap.
- Hard per-render watchdog: if a render exceeds `timeout + 5 s`, tear the WebView down rather than waiting. A hung page must never hold a slot.

### 4.3 Isolation

Each WebView gets its own `WKWebsiteDataStore`:

- `.nonPersistent()` by default → disposable profile, no cookie bleed between customers. This is a **security boundary**, not an optimization.
- Named persistent stores for `profileId` (§7).
- Between renders on a non-persistent store, call `removeData(ofTypes:modifiedSince:)` with the full type set anyway. Belt and braces.

### 4.4 Render-complete detection

Never rely on `didFinish` alone — it fires at DOM ready, long before an SPA has content. Combine:

1. `webView(_:didFinish:)` fires, **and**
2. network idle: zero in-flight requests for 500 ms (track via a `WKURLSchemeHandler` shim or a `PerformanceObserver` injected at document start), **and**
3. DOM quiet: a `MutationObserver` reports no mutations for 300 ms, **and**
4. optional `waitFor`: a CSS selector present, or a fixed delay, or a JS predicate returning true.

Cap the whole thing at `timeout`. On cap, **return what you have** with `metadata.renderTimedOut: true` rather than failing — a partially rendered page is usually still useful, and the caller can decide.

Many sites poll forever (chat widgets, analytics beacons, websockets). Condition 2 must ignore requests to known-noisy hosts, or it never settles. Maintain a blocklist and apply it as a `WKContentRuleList` — which also speeds up every render substantially.

### 4.5 Injected user script (`.atDocumentStart`, main frame + subframes)

- Stub `Notification.requestPermission`, `navigator.geolocation`, `navigator.mediaDevices` so permission prompts never appear (a prompt blocks the run loop).
- Neutralize `window.alert/confirm/prompt` — otherwise a modal deadlocks the WebView.
- Install the `MutationObserver` and `PerformanceObserver` used by §4.4.
- Capture `window.onerror` and `console.*` into an array the daemon reads back. Return it in `console[]`; it is the single most useful debugging field for "why is this page empty".
- Do **not** try to spoof fingerprinting surfaces (canvas, WebGL, audio). WebKit's real values are the asset here. Spoofing creates inconsistencies that are *more* detectable than the truth.

### 4.6 Daemon API

```
POST /render
  { url, waitFor?, timeout?, profileId?, actions?, blockResources?, viewport?, userAgent? }
  → { html, status, finalURL, redirects[], console[], timings, renderTimedOut }

POST /screenshot
  { url, fullPage?, width?, height?, format?, quality? }  → image/png | image/jpeg

POST /pdf
  { url, paperSize? }                                      → application/pdf

GET  /health
  → { pool: { total, busy, idle, recycled }, rss, uptime, rendersTotal }

POST /profile   { id, cookies[], userAgent? }   → creates/updates a persistent data store
DELETE /profile/:id                             → wipes it
```

`actions` mini-DSL, executed in order after initial load:

```jsonc
[
  { "click": "#accept-cookies" },
  { "wait": 1500 },
  { "type": { "selector": "#search", "text": "query" } },
  { "press": "Enter" },
  { "scroll": 2000 },
  { "scrollToBottom": { "maxScrolls": 10, "delay": 500 } },
  { "waitForSelector": ".results-loaded" },
  { "evaluate": "document.querySelector('.more').click()" }
]
```

`scrollToBottom` is the workhorse for infinite-scroll pages; implement it as scroll → wait → compare `scrollHeight` → repeat until stable or `maxScrolls`.

### 4.7 Transport between Node and the daemon

HTTP over loopback with keep-alive. Not Unix sockets (harder to debug), not stdio (framing pain).
Node client (`packages/fetcher-webkit`) must implement: 2 s connect timeout, single retry on `ECONNREFUSED` after a 1 s delay, circuit breaker that opens after 5 consecutive failures and routes everything to Tier 0 with a loud alert. **If the daemon is dead, the API must degrade rather than 500 across the board.**

---

## 5. Tier 2 spec — real Safari (`packages/fetcher-safari`)

Last resort, for targets that defeat Tier 1 or require the real user's logged-in state.

### 5.1 Two modes

- **`safaridriver`** (W3C WebDriver) — preferred. Stable, scriptable, proper timeouts. **Hard limit: one session at a time on the machine.** Safari Technology Preview provides a second, independent session; that is the ceiling — two, total.
- **AppleScript** via `osascript` — fallback. Serial, fragile, breaks on focus changes, but drives the *real* Safari with the *real* user profile and all its logged-in sessions.

### 5.2 Operating rules

- Its own BullMQ queue, `concurrency: 1`, hard 60 s timeout, max 200 jobs/day by default.
- Never reachable via `tier: "auto"` for anonymous requests. Tier 2 requires either an explicit `tier: "safari"` or a `profileId` bound to the real profile.
- Between jobs: close the tab, do **not** clear cookies (that is the point), but do clear the URL bar and history entry if the customer is not the profile owner.
- Fails closed. If safaridriver reports a session conflict, queue the job rather than falling back to AppleScript mid-flight — mixing the two corrupts state.
- Log every Tier 2 job with the requesting API key. This is the audit trail you will need if a target complains.

---

## 6. Proxy & identity layer (`packages/egress`)

One machine means one IP means one ban ends everything. Design for this from day one.

- **Tier 0 and Tier 1 egress through rotating residential proxies.** Tier 2 uses the host IP directly — that is precisely its value.
- Proxy selection is **sticky per host** for the duration of a crawl job: switching IPs mid-crawl of one site looks far more suspicious than staying put.
- Health-check the proxy pool every 60 s against a known-good endpoint. Eject on 3 consecutive failures, re-test after 10 min.
- Track per-(proxy, host) success rate in Redis. Stop routing a proxy to a host once its rate drops below 50% over the last 20 attempts — it is burned there.
- Geo-pin when the target is geo-sensitive: `egressRegion` option on the API, defaulting to the account's region.
- WKWebView proxy configuration is set per-`WKWebsiteDataStore` via `proxyConfigurations` (macOS 14+). It cannot be changed on a live store, so the pool must hold **buckets of WebViews per proxy** and acquire from the matching bucket. Plan the pool shape accordingly: e.g. 4 proxies × 6 WebViews rather than 24 undifferentiated ones.

---

## 7. Session profiles (`packages/profiles`)

A profile is a named, persistent browser identity: cookies, localStorage, and a pinned UA.

- Created by `POST /v1/profiles` with either an explicit cookie array, or an interactive login flow (§7.2).
- Stored as a `WKWebsiteDataStore` keyed by UUID on disk, plus a Postgres row holding metadata (owner, target domains, created/last-used, expiry).
- **Cookies are secrets.** Encrypt the Postgres-side copy with a key from the macOS Keychain, never log them, redact them from error payloads, and exclude the profile directory from any backup that leaves the machine unencrypted.
- Profiles expire: default 30 days idle, then wiped. Re-auth is the user's job.
- A profile is bound to one account. Cross-account use is a hard error, not a warning.

### 7.2 Interactive login flow

For targets where cookie import is impractical: `POST /v1/profiles/:id/login` opens a visible WKWebView on `m3u96a`, the operator logs in by hand, the daemon detects navigation to a success URL and seals the profile. This is a manual, low-volume operation and that is fine — it is what makes Tier 2 valuable.

---

## 8. Extraction engine (`packages/extract`)

Deterministic, versioned, testable. **No model inference anywhere in this package.**

### 8.1 Pipeline

```
raw HTML
  → encoding detection (charset header → <meta> → BOM → heuristic)
  → parse with linkedom
  → sanitize: strip <script> <style> <svg> <noscript>, ad iframes, tracking pixels
  → structured-data harvest (§8.3)  ─────────┐
  → boilerplate removal (§8.4)               │
  → relative → absolute URL rewriting        ├─► metadata
  → Turndown with custom rules (§8.5)        │
  → post-process                             │
  → markdown ────────────────────────────────┘
```

Every stage is a pure function `(input, options) => output`. The whole pipeline is testable offline against saved HTML fixtures, which is the entire reason for choosing determinism.

### 8.2 Boilerplate removal

Two modes, selected by `onlyMainContent`:

- **Readability** (`@mozilla/readability`) for article-shaped pages. Reliable on news, blogs, docs.
- **Density-based fallback** when Readability returns under 200 characters — which it does on listing pages, product pages, and dashboards. Implement a simple text-density scorer: for each block element, `textLength / (1 + linkTextLength)`, pick the subtree maximizing the sum. Do not skip this fallback; Readability silently failing to a near-empty result is the single most common extraction bug.

Always strip, in both modes: `<header>`, `<footer>`, `<nav>`, `[role=navigation]`, `[aria-hidden=true]`, cookie banners (match on common class patterns), newsletter modals, and elements whose class matches `/(^|[-_])(ad|ads|advert|sponsor|promo|share|social|related|comment)([-_]|$)/i`.

### 8.3 Structured data harvest

This is where deterministic extraction earns its keep. Harvest, in priority order, and merge:

1. **JSON-LD** (`<script type="application/ld+json">`) — parse all blocks, handle `@graph`, resolve `@type` to a normalized shape. Covers Article, Product, Recipe, Event, JobPosting, BreadcrumbList, Organization, FAQPage.
2. **Microdata** (`itemscope`/`itemprop`) and **RDFa** — older but still common on e-commerce.
3. **OpenGraph** and **Twitter cards** — title, description, image, type, publish time.
4. **Standard meta** — `description`, `author`, `keywords`, `canonical`, `hreflang` alternates.
5. **Heuristics** — `<time datetime>` for dates, `<h1>` for title, first `<p>` over 100 chars for description.

Output as `metadata` plus a normalized `structured` object. A Product page should yield price, currency, availability, SKU, and images **without any model involved**. Roughly 60% of commercially interesting pages carry usable structured data; this is the highest-leverage code in the repo.

### 8.4 Selector-based extraction (`/v1/extract` core)

Users supply a schema where each field maps to an extractor:

```jsonc
{
  "schema": {
    "title":  { "selector": "h1", "type": "text" },
    "price":  { "selector": ".price", "type": "number", "clean": "currency" },
    "images": { "selector": "img.product", "type": "attr", "attr": "src", "multiple": true },
    "specs":  { "selector": "table.specs", "type": "table" },
    "sku":    { "jsonld": "$.sku" },
    "body":   { "selector": "#desc", "type": "markdown" }
  }
}
```

Types: `text`, `html`, `markdown`, `number`, `date`, `url`, `attr`, `table`, `list`, `boolean` (presence).
Cleaners: `currency`, `whitespace`, `trim`, `stripTags`, `parseDate` (with locale hint).
Sources: `selector` (CSS), `xpath`, `jsonld` (JSONPath into harvested JSON-LD), `regex`, `meta`.

Field resolution order: `jsonld` → `meta` → `selector` → `xpath` → `regex`. First non-empty wins. This lets one schema work across sites with different markup.

### 8.5 Repeated-structure inference (`/v1/extract` on list pages)

For listing pages, users should not have to write selectors. Implement auto-detection:

1. Compute a structural signature for every element: tag path + class set, normalized.
2. Find the deepest parent whose children contain **≥ 5 siblings sharing a signature**.
3. That parent is the list container; the siblings are the records.
4. Within one record, each leaf position becomes a candidate field; label it from `itemprop`, `class`, or a nearby `<dt>`/`<th>`.

This handles search results, product grids, and job boards without configuration. Return the inferred selectors alongside the data so users can pin and refine them — inference that cannot be inspected and overridden is worse than useless.

### 8.6 Markdown conversion rules

Turndown with hand-written rules. Write these; the defaults are not good enough:

| Input | Output |
|---|---|
| `<pre><code class="language-x">` | fenced block with language tag |
| `<pre>` without language | fenced block, attempt language detection from content |
| `<table>` | GFM table, pipes escaped, empty cells preserved, `<th>` → header row |
| nested/spanning tables | HTML passthrough (GFM cannot express them) |
| `<figure>` + `<figcaption>` | image followed by italic caption |
| `<dl>/<dt>/<dd>` | bold term + indented definition |
| heading anchors (`#`, `¶`, `.anchor`) | removed |
| `<img>` with `data-src`/`srcset` only | resolve to the highest-resolution real URL |
| `<br>` inside a table cell | `<br>` kept (newline breaks the table) |
| inline `<svg>` | removed, unless it has `<title>` → alt-text image reference |
| `<abbr title>` | text + parenthetical |
| MathML / `.katex` | `$...$` / `$$...$$` |

Post-process: collapse 3+ blank lines to 2, trim trailing whitespace, normalize list markers to `-`, deduplicate consecutive identical links.

### 8.7 BYOK LLM pass-through (optional, `packages/byok`)

Thin and stateless. `/v1/extract` accepts:

```jsonc
{ "llm": { "provider": "anthropic|openai|...", "apiKey": "sk-...", "model": "...", "prompt": "..." } }
```

Tendril runs its normal deterministic pipeline, then forwards the resulting markdown plus the user's JSON schema to their provider with their key, and returns the response.

Rules: the key is held in memory for the request only, never written to disk, never logged, never included in error payloads; chunk markdown over ~50k characters by heading and merge results; on provider error, return the deterministic result with `llm: { error }` rather than failing the whole request. Tendril is a pass-through, not an LLM vendor — do not build a model router, do not cache completions, do not offer a Tendril-supplied key.

### 8.8 Versioning

The extraction pipeline has a `pipelineVersion` (semver) stored with every cached result. Bumping it invalidates derived markdown but **not** the raw HTML cache, so you can re-extract the entire corpus after a rules improvement. This is the single most valuable property of keeping raw HTML forever.

---

## 9. Non-HTML content (`packages/extract/formats`)

Content-Type routing, before any HTML logic:

| Type | Handling |
|---|---|
| `application/pdf` | text layer extraction; if under 100 chars, flag `needsOCR` and skip (do not silently return empty) |
| `application/json` | pretty-print; if it looks like an API response, return as `json` directly |
| `text/plain`, `text/markdown` | pass through |
| `text/csv` | parse to a GFM table, cap at 500 rows |
| `application/xml`, RSS, Atom | parse feeds into items; this makes `/map` much stronger on blogs |
| `image/*` | metadata only, plus the blob; no OCR |
| `application/zip`, archives | reject with `ERR_UNSUPPORTED_TYPE` |
| anything > `maxSizeBytes` (default 20 MB) | reject before download completes, via streaming size check |

Never let a Content-Type mismatch reach the HTML parser. A 200 MB video streamed into linkedom will take the process down.

---

## 10. Crawl frontier (`packages/frontier`)

Non-negotiable rules:

1. **Normalize before anything.** Lowercase host, strip default port, sort query params, remove `utm_*`/`fbclid`/`gclid`/`ref`/`mc_cid`/`_ga`, drop fragment (unless `#!` hashbang), resolve `..`, consistent trailing slash, punycode hosts. Dedup runs on the normalized form. Keep the original for output.
2. **Two-stage dedup.** In-memory Bloom filter (1% FP, sized `limit × 3`) → Postgres confirmation on hit. Avoids a DB round-trip per discovered link, which at 8 concurrent renders is thousands per minute.
3. **Per-host rate limiting**, never global. Token bucket in Redis, configurable `delayMs`, honor robots.txt `Crawl-delay` when higher. A crawl of one site must not slow another customer's job.
4. **Priority.** Strict BFS by depth, then heuristics: `/blog/`, `/docs/`, `/article/`, `/product/` ahead of `/tag/`, `/author/`, `/page/47`, `?sort=`, `?filter=`.
5. **Trap detection.** If more than 200 URLs on one host share a path pattern with only a varying numeric or date segment, sample instead of exhausting. Infinite calendars, paginated tag archives, and faceted-search URL explosions are the three classics; each can generate unbounded URLs.
6. **Backoff.** 429 or 503 → pause the entire host, exponential 2^n s, capped at 5 min, 3 attempts, then mark the host degraded for the job.
7. **Budgets.** Page count *and* wall-clock *and* total bytes. First limit reached ends the job cleanly with `stopReason`.
8. **Canonical awareness.** If a page declares `<link rel=canonical>` pointing elsewhere on the same host, record the mapping and do not crawl both. Saves 20-40% on most e-commerce sites.
9. **Sitemap seeding.** Always attempt sitemap discovery first, even for a crawl — starting from a complete URL list beats link-following, and it lets you report accurate progress instead of an unknown denominator.
10. **Resumability.** Frontier state lives in Postgres, not memory. A restart mid-crawl resumes; it does not start over.

### 10.1 robots.txt

Full RFC 9309 parsing: `User-agent` groups with correct specificity matching, `Allow`/`Disallow` with longest-match-wins, wildcards `*` and `$`, `Sitemap` directives, `Crawl-delay` (non-standard but honored). Cache per host for 24 h. On fetch failure treat as allow-all, but log it; on 5xx treat as disallow-all for 1 h, per the spec's intent.

---

## 11. Cache & storage (`packages/cache`)

- **Content-addressed blob store.** `blobs/<sha256[0:2]>/<sha256[2:4]>/<sha256>`. Raw HTML, screenshots, PDFs. Deduplicates automatically across customers — the same page fetched by two accounts is stored once.
- **Metadata in Postgres**, pointing at blob hashes. Never store blobs in Postgres.
- **Cache key** = `sha256(normalizedURL + tier + profileId + relevantOptions)`. Options that do not change the fetch (output format, extraction schema) must **not** be in the key, or the hit rate collapses.
- `maxAge` on the request decides whether a hit is served. Default 0 (always fetch) for `/scrape`, 3600 for `/crawl` sub-pages.
- Respect `ETag` and `Last-Modified`: on a stale hit, revalidate with a conditional request. A 304 costs almost nothing and refreshes the entry.
- **Eviction:** LRU by last access, triggered when the blob store exceeds a configured ceiling (default 200 GB). Run it as a scheduled job, not inline.
- GDPR: cache entries carry a retention date; a purge job runs nightly (§26).

---

## 12. Data model

Postgres 16, schema `tendril`, migrations via drizzle. Core tables:

```
accounts        id, name, plan, created_at, egress_region
api_keys        id, account_id, hash (argon2id), prefix, scopes[], last_used_at, revoked_at
jobs            id, account_id, type(scrape|crawl|map|search|extract), status, params jsonb,
                created_at, started_at, finished_at, stop_reason, error_code
pages           id, job_id, url, normalized_url, canonical_url, depth, status_code,
                tier_used, escalation_reason, html_hash, markdown_hash, screenshot_hash,
                metadata jsonb, structured jsonb, pipeline_version, fetched_at, duration_ms
frontier        job_id, normalized_url, depth, priority, state(pending|active|done|failed),
                attempts, next_attempt_at            -- PK (job_id, normalized_url)
domain_hints    host, preferred_tier, requires_profile, crawl_delay_ms, success_rate,
                last_updated                          -- the learned routing table
profiles        id, account_id, name, domains[], data_store_path, cookies_enc,
                created_at, last_used_at, expires_at
robots_cache    host, body, fetched_at, expires_at
usage           account_id, day, pages_by_tier jsonb, bytes_out, llm_passthrough_calls
webhooks        id, job_id, url, events[], secret, deliveries jsonb
```

Indexes that matter: `frontier(job_id, state, priority)` for the dequeue path, `pages(job_id, fetched_at)` for pagination, `pages(html_hash)` for dedup lookups, partial index on `frontier(next_attempt_at) WHERE state='failed'`.

---

## 13. Queue & job lifecycle

BullMQ on Redis. Queues:

| Queue | Concurrency | Notes |
|---|---|---|
| `fetch:http` | 32 | Tier 0 |
| `fetch:webkit` | 24 | matches the pool size exactly |
| `fetch:safari` | **1** | never raise this |
| `crawl:control` | 8 | one job per active crawl, manages its frontier |
| `extract` | 16 | CPU-bound, pure |
| `webhook` | 8 | with retries |
| `maintenance` | 1 | cache eviction, purges, robots refresh |

Job lifecycle: `queued → running → (succeeded | failed | cancelled)`.
Retries: 3 attempts, exponential backoff with jitter, only for transient error codes (§15). Never retry `ERR_ROBOTS_DENIED`, `ERR_SSRF_BLOCKED`, `ERR_UNSUPPORTED_TYPE`, or any 4xx other than 408/429.

**Idempotency:** `/scrape`, `/crawl`, `/extract` accept an `Idempotency-Key` header. Store key → jobId for 24 h; a repeat returns the original job rather than starting a new one. Customers with retry logic will otherwise double-bill themselves and blame you.

Cancellation is cooperative: set a flag in Redis, workers check it between pages, in-flight renders complete. Never kill a WebView to cancel — it corrupts the pool.

---

## 14. API contracts

Base: `https://www.ten-dril.com/v1`. Auth: `Authorization: Bearer tdr_live_...`.
All responses: `{ success: boolean, data?, error?: { code, message, details? } }`.
All list responses are cursor-paginated: `{ data: [...], next?: "cursor" }`. No offset pagination.

### 14.1 `POST /v1/scrape`

```jsonc
{
  "url": "https://example.com/article",
  "formats": ["markdown", "html", "links", "screenshot", "structured", "extract"],
  "extractSchema": { /* §8.4 */ },
  "llm": { /* §8.7, optional */ },
  "tier": "auto",                 // auto | http | webkit | safari
  "profileId": "uuid",            // forces tier >= 1
  "actions": [ /* §4.6 */ ],
  "onlyMainContent": true,
  "includeTags": ["article", "main"],
  "excludeTags": [".sidebar", "#comments"],
  "waitFor": 0,
  "timeout": 30000,
  "maxAge": 0,
  "headers": { /* custom, merged over defaults */ },
  "egressRegion": "us",
  "blockResources": ["image", "media", "font"]
}
```

Response `data`: `{ markdown, html, rawHtml, links[], structured, extract, screenshot, metadata, tierUsed, cached, timings }`.

`timings`: `{ total, dns, connect, ttfb, download, render, extract, escalations[] }`. Expose this — customers debugging slow crawls will otherwise blame you for their target's latency.

`links[]` is `{ url, text, rel, isInternal }`, deduplicated, absolute.

### 14.2 `POST /v1/crawl`

```jsonc
{
  "url": "https://example.com",
  "limit": 1000,
  "maxDepth": 4,
  "maxDurationSeconds": 3600,
  "maxBytes": 5368709120,
  "includePaths": ["^/blog/.*"],
  "excludePaths": ["\\.pdf$", "^/tag/"],
  "allowSubdomains": false,
  "allowBackwardLinks": false,
  "allowExternalLinks": false,
  "concurrency": 8,
  "delayMs": 250,
  "respectRobots": true,
  "deduplicateSimilar": true,
  "scrapeOptions": { /* §14.1 minus url */ },
  "webhook": { "url": "...", "events": ["page","completed","failed"], "secret": "..." }
}
```

→ `202 { jobId, statusUrl, streamUrl }`.

- `GET /v1/crawl/:id` → `{ status, total, completed, failed, creditsUsed, data[], next }`
- `GET /v1/crawl/:id/stream` → SSE, one event per page plus a terminal event. Heartbeat every 15 s or ngrok will drop the connection.
- `DELETE /v1/crawl/:id` → cooperative cancel, returns partial results.
- `GET /v1/crawl/:id/errors` → per-URL failures with codes. Customers need this and Firecrawl handles it poorly.

Webhook payloads are signed: `X-Tendril-Signature: sha256=<hmac(secret, body)>`, plus `X-Tendril-Timestamp` for replay protection. Retry 5 times with exponential backoff, then mark the webhook degraded.

### 14.3 `POST /v1/map`

URL discovery **without rendering**. Target: under 3 s for a normal site.
Sources merged and deduplicated: `sitemap.xml` (recursive through sitemap indexes, `.gz` support, cap at 50k URLs), `robots.txt` Sitemap directives, homepage links, `/llms.txt`, RSS/Atom feeds.
Options: `search` (substring filter, ranked by match position), `limit`, `includeSubdomains`, `sitemapOnly`, `ignoreSitemap`.
Returns `{ links: [{ url, title?, lastModified?, source }] }` — `source` tells the user where each URL came from, which builds trust in the result.

### 14.4 `POST /v1/search`

```jsonc
{ "query": "...", "limit": 10, "lang": "en", "country": "us", "timeRange": "month",
  "scrapeResults": false, "scrapeOptions": {} }
```

Pipeline: SearXNG (self-hosted, §16.1) → dedup by domain+normalized title → rank → optionally scrape the top N in parallel (Tier 0 forced, 8 s timeout, partial failures tolerated and reported per-result).

Be explicit in the docs: Tendril has no index of its own. Search quality is SearXNG's quality. Under-promise here.

### 14.5 `POST /v1/extract`

```jsonc
{
  "urls": ["https://example.com/products/*"],   // wildcards expand via /map
  "schema": { /* §8.4 */ },
  "inferList": true,                            // §8.5 auto-detection
  "llm": { /* §8.7, optional */ },
  "scrapeOptions": {}
}
```

Returns one record per matched URL, or one aggregated array when `inferList` finds a list page. Include the resolved selectors in the response so users can pin them.

### 14.6 Ancillary

- `POST /v1/profiles`, `GET /v1/profiles`, `DELETE /v1/profiles/:id`, `POST /v1/profiles/:id/login`
- `GET /v1/usage?from=&to=` — pages by tier, bytes, cost
- `GET /healthz` — unauthenticated, shallow
- `GET /v1/status` — authenticated, per-tier health (§18.3)

Rate limits returned on every response: `X-RateLimit-Limit`, `-Remaining`, `-Reset`.

---

## 15. Error taxonomy

One file maps codes to HTTP statuses and retry-ability. Never invent an ad-hoc string.

| Code | HTTP | Retryable | Meaning |
|---|---|---|---|
| `ERR_INVALID_URL` | 400 | no | malformed or unsupported scheme |
| `ERR_SSRF_BLOCKED` | 400 | no | resolved to a private or forbidden address |
| `ERR_UNSUPPORTED_TYPE` | 415 | no | content type not handled |
| `ERR_TOO_LARGE` | 413 | no | exceeded `maxSizeBytes` |
| `ERR_UNAUTHORIZED` | 401 | no | bad or revoked key |
| `ERR_QUOTA_EXCEEDED` | 402 | no | plan limit hit |
| `ERR_RATE_LIMITED` | 429 | yes | our limit, not the target's |
| `ERR_ROBOTS_DENIED` | 403 | no | robots.txt disallows |
| `ERR_TARGET_BLOCKED` | 502 | maybe | target returned a challenge at every tier |
| `ERR_TARGET_4XX` / `_5XX` | 502 | 5xx only | upstream status passed through |
| `ERR_TIER_TIMEOUT` | 504 | yes | render exceeded timeout |
| `ERR_POOL_EXHAUSTED` | 503 | yes | no WebView available in 30 s |
| `ERR_DAEMON_DOWN` | 503 | yes | Swift daemon unreachable, circuit open |
| `ERR_SAFARI_BUSY` | 503 | yes | Tier 2 session conflict |
| `ERR_PROFILE_EXPIRED` | 409 | no | session profile needs re-auth |
| `ERR_EXTRACT_FAILED` | 422 | no | schema produced no fields |
| `ERR_REDIRECT_LOOP` | 502 | no | same URL twice in a chain |
| `ERR_INTERNAL` | 500 | yes | a bug; page it |

Every error response carries a `requestId` that appears in the logs. Support requests without one are unanswerable.

---

## 16. Deployment — node `m3u96a` + ngrok

Single host. No Kubernetes, no cloud VM. Ingress is an ngrok tunnel on a reserved domain.

### 16.1 Topology

```
   Internet
      │
      ▼
 ngrok edge  ──── TLS terminated here, certificate managed by ngrok
 www.ten-dril.com
      │  (encrypted tunnel, outbound connection initiated by the Mac)
      ▼
 ┌──────────────────────── m3u96a ────────────────────────┐
 │  ngrok agent (LaunchAgent)                             │
 │        │                                               │
 │        ▼                                               │
 │  Caddy :8080 ── reverse proxy, routing, access logs    │
 │        ├─► Fastify API            :3000                │
 │        └─► /healthz                                    │
 │                                                        │
 │  Internal only, bound to 127.0.0.1, never tunneled:    │
 │    tendril-worker (Swift)         :8787                │
 │    Redis                          :6379                │
 │    PostgreSQL                     :5432                │
 │    SearXNG                        :8888                │
 │    Prometheus / Grafana           :9090 / :3001        │
 └────────────────────────────────────────────────────────┘
```

Caddy sits between ngrok and the API deliberately: one place for routing, access logs, and local rate limiting, and only one port is ever tunneled.

### 16.2 ngrok configuration

`deploy/ngrok.yml`:

```yaml
version: "3"
agent:
  authtoken: ${NGROK_AUTHTOKEN}
  log: /usr/local/var/log/ngrok.log
  log_level: info
  connect_timeout: 10s
endpoints:
  - name: tendril-api
    url: https://www.ten-dril.com
    upstream:
      url: 8080
    traffic_policy:
      inbound:
        - actions:
            - type: rate-limit
              config:
                name: global
                algorithm: sliding_window
                capacity: 600
                rate: 60s
        - expressions:
            - "req.url.path.startsWith('/internal')"
          actions:
            - type: deny
              config:
                status_code: 404
```

DNS: `CNAME` for `www.ten-dril.com` → the target ngrok returns when you reserve the domain. Reserve the apex `ten-dril.com` too and 301 it to `www` **at the ngrok edge**, not in the application.

A reserved domain requires a paid ngrok plan; free-tier URLs rotate on restart and are unusable for a customer-facing API. Confirm current plan tiers and Traffic Policy feature availability on ngrok's site before building against a specific capability — this changes.

### 16.3 Process supervision

LaunchAgents (not LaunchDaemons — WebKit requires a user GUI context), in `~/Library/LaunchAgents/`:

| Label | Process | Notes |
|---|---|---|
| `com.tendril.caffeinate` | `caffeinate -dimsu` | first to start, mandatory |
| `com.tendril.postgres` | postgres | |
| `com.tendril.redis` | redis-server | |
| `com.tendril.worker` | Swift daemon | needs GUI session, `KeepAlive: true` |
| `com.tendril.searxng` | uvicorn/docker | |
| `com.tendril.api` | `node apps/api/dist/main.js` | |
| `com.tendril.caddy` | `caddy run` | |
| `com.tendril.ngrok` | `ngrok start --all` | **last**, gated on health |

Set `SoftResourceLimits: { NumberOfFiles: 65536 }` in every plist, or the pool saturates around 40 connections.

Start ordering: `com.tendril.ngrok` runs a wrapper that polls `http://127.0.0.1:8080/healthz` until healthy (max 120 s) before starting the tunnel. Exposing an unhealthy API is worse than a short outage.

### 16.4 Deploy procedure (`scripts/deploy.sh`)

1. `git pull` on a deploy branch, never on a dirty tree
2. `pnpm install --frozen-lockfile && pnpm build && swift build -c release`
3. `pnpm db:migrate` — migrations must be backward-compatible for one version, so a rollback does not need a down-migration
4. Drain: stop accepting new jobs, wait up to 60 s for in-flight ones
5. `launchctl kickstart -k` each agent in dependency order
6. `./scripts/health.sh` — all tiers, all services
7. **Smoke test through the public URL**, not localhost: scrape a known page via `https://www.ten-dril.com/v1/scrape` and assert the markdown contains an expected string
8. On failure: `git checkout` previous tag, repeat from 2. Keep the last 3 builds on disk.

### 16.5 Security posture

Exposing a home machine to the internet deserves more care than a cloud VM.

- **Only port 8080 is tunneled.** 8787, 6379, 5432, 8888, 9090 bind to `127.0.0.1` explicitly. Verify with `lsof -iTCP -sTCP:LISTEN` after every deploy.
- API keys hashed with argon2id, prefix stored in clear for identification, full key shown once at creation.
- Quotas enforced in-app **and** a global rate limit at the ngrok edge, so a burst never reaches the Mac.
- Grafana and any admin surface: IP allowlist in Traffic Policy, or a separate non-public tunnel.
- **SSRF is the top risk of any scraping API.** Users control the target URL. Enforce: block RFC1918, loopback, link-local, `169.254.169.254`, IPv6 ULA and mapped-IPv4; allow only `http`/`https`; allow only ports 80/443/8080/8443. **Resolve DNS first and validate the resolved IP**, then connect to that IP with the Host header set — otherwise DNS rebinding walks straight into Redis and Postgres. Re-validate on every redirect hop.
- Redact from logs: `Authorization`, `Cookie`, `Set-Cookie`, BYOK keys, webhook secrets, profile cookie blobs.
- macOS firewall on, `Remote Login` restricted to key auth on a non-default port, FileVault enabled.

### 16.6 Failure modes specific to this setup

These are what will actually take the service down. Treat them as design constraints.

- **The tunnel is a single point of failure.** Sleep, Wi-Fi loss, or reboot means total outage. External uptime monitor on `https://www.ten-dril.com/healthz` every 60 s, alerting to your phone. Nothing else in this document matters if you find out from a customer.
- **Residential upload bandwidth is the real ceiling.** Screenshots and large HTML saturate a typical uplink long before CPU. Gzip everything (Caddy), cap screenshot dimensions, and serve large blobs via short-lived signed links rather than inline base64.
- **Reboots break the GUI session.** Without auto-login, Tiers 1 and 2 die while Tier 0 keeps returning bot-blocked pages. That looks like a quality regression, not an outage — which is why health checks must probe each tier independently (§18.3).
- **ngrok reconnects silently but not instantly.** Expect 5-30 s gaps on network flaps. SDKs must retry with backoff; document it.
- **One IP, one ban surface.** See §6.
- **Thermal throttling.** 24 WebViews rendering continuously will heat an M3 and reduce clocks. Monitor `powermetrics` and cap the pool lower if sustained throughput drops.
- **Backups.** Postgres and the blob CAS live on one SSD. Nightly `pg_dump` plus a CAS rsync to an external encrypted volume, offsite weekly, **and a tested restore**. An untested backup is not a backup.

---

## 17. macOS prerequisites

Scripted in `scripts/setup-mac.sh`, idempotent, but several steps require a human.

- **A GUI session is mandatory.** WKWebView and Safari will not render without a window server session. SSH alone is insufficient. Enable auto-login; connect via Screen Sharing *after* the graphical session exists.
- **`caffeinate -dimsu`** as a LaunchAgent. Sleep kills the pool.
- **Screen lock and screen saver disabled.** A locked screen suspends rendering in some WebViews.
- **Safari → Develop → "Allow JavaScript from Apple Events"** must be ticked by hand. Not scriptable.
- **`safaridriver --enable`** once, with an admin password.
- **Automation and Accessibility permissions:** the first `osascript` triggers a TCC dialog requiring a click. No clean bypass. Document it in the runbook.
- **Full Disk Access** for the terminal and the daemon if touching the Safari profile directory.
- Disable Spotlight indexing on the blob store (`mdutil -i off`) — it will otherwise index millions of files.
- Increase `kern.maxfiles` and `kern.maxfilesperproc` via a `sysctl` plist.

---

## 18. Observability

### 18.1 Logs

Pino, JSON, to stdout and a rotated file. Every line carries `{ requestId, jobId, accountId, url, tier }` where applicable. No `console.log` anywhere, enforced by lint rule.

### 18.2 Metrics (Prometheus, scraped locally)

- `tendril_requests_total{endpoint,status}`
- `tendril_fetch_duration_seconds{tier}` — histogram
- `tendril_escalations_total{from,to,reason}` — **the most important metric in the system**
- `tendril_pool_slots{state}` — gauge: busy/idle/recycling
- `tendril_extraction_empty_total{reason}` — pages yielding under 200 chars; a rising line here means a silent quality regression
- `tendril_proxy_success_rate{proxy,host}`
- `tendril_tunnel_up` — 0/1 from the agent's local API
- `tendril_queue_depth{queue}`, `tendril_job_duration_seconds{type}`

### 18.3 Health checks

`GET /v1/status` probes each tier independently against a known-stable control URL and returns per-tier latency and last success. `/healthz` stays shallow and fast for the uptime monitor.

Alert on: tunnel down, daemon circuit open, escalation rate above 40% (a target changed its defenses), extraction-empty rate above 10%, queue depth growing for 10 minutes, disk above 85%.

---

## 19. Performance targets

Measured on m3u96a, `pnpm bench` against 100 reference URLs. Regressions block a release.

| Metric | Target |
|---|---|
| Tier 0 p50 / p95 | 60 ms / 400 ms |
| Tier 1 p50 / p95 | 550 ms / 2.5 s |
| Escalation rate | < 25% |
| Extraction (HTML → markdown), 500 KB page | < 120 ms |
| `/map` on a 5k-URL sitemap | < 3 s |
| Sustained crawl throughput | 40 pages/s Tier 0, 12 pages/s Tier 1 |
| Memory, full pool, steady state | < 14 GB |
| Cold start to first successful render | < 45 s |

---

## 20. Testing

- **Unit:** URL normalization, `shouldEscalate`, Turndown rules, robots parsing, SSRF validation, structural inference. Fast, hermetic, no network. These are the majority of the suite.
- **Fixtures:** `test/fixtures/html/` with ~60 saved real pages — React SPA, Next.js, WordPress, Shopify, static docs, a Cloudflare interstitial, a paginated listing, a page with nested tables, a page with JSON-LD, one with broken encoding. Extraction is tested only here.
- **Golden files** for markdown output. Any rule change must show its diff and be reviewed.
- **Contract tests** on the API schemas — the OpenAPI document is generated from Zod, so schema drift breaks the build.
- **E2E** against 20 stable public sites, tagged `@slow`, excluded from CI. Asserts escalation rates and non-empty extraction, never exact content.
- **Deploy smoke test** through the public URL (§16.4). Half of all outages are ingress, not application.
- **Chaos:** kill the Swift daemon mid-crawl and assert the circuit breaker degrades to Tier 0 instead of failing the job. Do this in CI with a fake daemon.

Coverage target: 85% on `packages/`, no target on `apps/`.

---

## 21. Code conventions

- TypeScript strict, `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`. No `any`; `unknown` plus narrowing.
- **No exceptions for normal control flow.** Fetchers and extractors return `Result<T, TendrilError>`. Exceptions signal bugs and are caught only at the top-level handler.
- Every error carries a code from §15. The code → HTTP mapping lives in exactly one file.
- Pure functions wherever possible; I/O confined to `fetcher-*` and `cache`. This is what makes the fixture-based tests viable.
- No comments restating code. Comment the *why*, especially for site-specific workarounds — and date them, they rot within months.
- Swift: `async/await` only, no completion handlers. The pool is an `actor`. No force-unwraps outside tests.
- Commits: conventional commits. A commit touching extraction rules must include the golden-file diff.

---

## 22. SDKs & docs

- **TypeScript SDK** — generated types from the OpenAPI schema, hand-written ergonomics on top. Must handle: automatic retry with backoff (for tunnel flaps, §16.6), SSE streaming for crawls with reconnect, idempotency keys generated by default.
- **Python SDK** — same surface, sync and async clients.
- Docs site with runnable examples per endpoint, plus a **migration guide from Firecrawl** mapping their parameter names to yours. Compatibility of naming where it costs nothing is worth more than elegance.
- Publish the OpenAPI document at `https://www.ten-dril.com/openapi.json`.

---

## 23. Repository layout

```
tendril/
├── CLAUDE.md
├── package.json                 # pnpm workspaces
├── turbo.json
├── apps/
│   ├── api/                     # Fastify — public surface
│   │   ├── src/routes/{scrape,crawl,map,search,extract,profiles,usage}.ts
│   │   ├── src/schemas/         # Zod → generated OpenAPI
│   │   ├── src/auth/            # keys, quotas, rate limits
│   │   └── src/errors.ts        # §15 mapping, single source of truth
│   └── worker/                  # BullMQ consumers
├── packages/
│   ├── router/                  # tier decision, escalation, domain_hints
│   ├── fetcher-http/            # Tier 0
│   ├── fetcher-webkit/          # Swift daemon client + circuit breaker
│   ├── fetcher-safari/          # Tier 2
│   ├── egress/                  # proxy pool, SSRF validation
│   ├── profiles/                # session management
│   ├── extract/                 # deterministic pipeline
│   │   ├── src/structured/      # JSON-LD, microdata, OG
│   │   ├── src/boilerplate/     # Readability + density fallback
│   │   ├── src/markdown/        # Turndown rules
│   │   ├── src/selectors/       # schema-driven extraction
│   │   ├── src/infer/           # repeated-structure detection
│   │   └── src/formats/         # PDF, CSV, XML, feeds
│   ├── byok/                    # optional LLM pass-through
│   ├── frontier/                # crawl logic, robots, sitemaps
│   ├── cache/                   # CAS + metadata
│   └── shared/                  # types, errors, logger, Result
├── native/
│   └── tendril-worker/
│       ├── Package.swift
│       └── Sources/TendrilWorker/{main,Pool,Renderer,Server,Actions,Profiles}.swift
├── deploy/
│   ├── ngrok.yml
│   ├── Caddyfile
│   ├── launchagents/*.plist
│   └── runbook.md
├── sdks/{typescript,python}/
├── test/fixtures/
└── scripts/{setup-mac,deploy,health,backup}.sh
```

---

## 24. Roadmap

**Phase 1 — foundations (2 wks).** Tier 0, extraction pipeline (§8.1-8.3, 8.6), `/scrape`, CAS cache, error taxonomy. Must handle 80% of the fixture set. Ship nothing publicly yet.

**Phase 2 — rendering (3 wks).** Swift daemon, pool, escalation router, `domain_hints`, `/screenshot`. Targets: escalation under 25%, Tier 1 p95 under 2.5 s.

**Phase 3 — crawl (2 wks).** Frontier, BullMQ, `/crawl` with SSE and signed webhooks, `/map`, resumability.

**Phase 4 — extraction depth (3 wks).** Selector schemas (§8.4), structure inference (§8.5), non-HTML formats (§9), `/extract`. **This is the phase that differentiates the product — do not cut it.** Optional BYOK at the end.

**Phase 5 — stealth & identity (2 wks).** Proxy layer, Tier 2 Safari, session profiles, `/search` via SearXNG.

**Phase 6 — production (2 wks).** Deploy on `m3u96a`, ngrok reserved domain, quotas, billing, observability, SDKs, docs, migration guide.

Do not start Phase 4 before Phase 2's escalation rate target is met. Extraction quality on pages you cannot fetch is worth nothing.

---

## 25. Anti-patterns — do not do this

- ❌ Driving Safari via AppleScript for volume. Serial, breaks on focus change. A last resort, not a foundation.
- ❌ Spawning a `WKWebView` per request. Init costs ~800 ms. Pool or nothing.
- ❌ Using jsdom. linkedom is 5-10× faster on this workload and the difference is the whole extraction budget.
- ❌ Storing markdown without the source HTML. You will never re-extract after improving rules (§8.8).
- ❌ Putting output-format options in the cache key. Hit rate collapses.
- ❌ A global rate limit instead of per-host. You will get banned from one site while crawling another fast.
- ❌ Trusting `networkidle` alone as a render signal. Many pages poll forever.
- ❌ Letting Readability's empty result pass through silently. Always run the density fallback.
- ❌ Ignoring robots.txt by default. Explicit opt-out, logged per client, never implicit.
- ❌ Putting Tier 2 in the general queue. One stuck session freezes everything.
- ❌ Validating the hostname instead of the resolved IP for SSRF. That is a rebinding hole straight into your database.
- ❌ Tunneling any port other than 8080.
- ❌ Assuming the tunnel is up. Verify after every deploy, monitor continuously.
- ❌ Adding a local model "just for the hard cases". That decision reverses §1 and turns a testable system into a probabilistic one. If it becomes necessary, do it as an explicit, documented rewrite — not as a quiet addition.

---

## 26. Legal & ethical

Not decorative. This is real product risk.

- `respectRobots: true` is the default. Disabling requires an explicit flag, is logged per client, and is refused entirely for domains on an internal blocklist.
- Honest rate limiting: minimum 100 ms `delayMs` per host, not overridable.
- Identifiable User-Agent with a contact URL (`+https://www.ten-dril.com/bot`) and a published page explaining who you are and how to block you. Stealth means matching a real browser's fingerprint, not hiding your identity.
- Scraping authenticated pages (Tier 2) puts the user on the hook for the target's ToS. State it plainly and require acceptance at signup.
- **No CAPTCHA solving, no paywall bypass, no login-credential stuffing.** Commercial red lines as much as ethical ones.
- Personal data collected incidentally has a retention period; the nightly purge job enforces it. Support deletion requests by URL and by domain.
- Honor `noindex`/`nofollow` as a signal for crawl scope even though they are indexing directives — it costs little and is defensible.
- Running a public service from a residential connection may conflict with your ISP's terms on operating servers. Check the contract before going live on `www.ten-dril.com`.

I am not a lawyer. Have this framework reviewed by one before commercializing, particularly regarding the CFAA and state computer-crime statutes (US), the Database Directive and GDPR (EU), and the terms of the specific high-value targets you intend to support.