# Tendril

Web ingestion that reaches what other crawlers can't.

Scrape · Map — running on a residential Mac, in real WebKit, with deterministic, testable extraction.

🌐 Live · ▶ Playground · 🤖 For agents · 📖 Docs · 📦 Source

# ✦ Why Tendril

Firecrawl runs in the cloud, on datacenter IPs, in headless Chromium. Tendril runs on a residential machine, in WebKit — Safari's actual engine — behind a real IP.


🕸️ Browser fidelity	WebKit on macOS ARM behind a residential IP is not a headless-Chromium signature. Sites that block others often don't block Tendril.
📐 Deterministic extraction	No LLM in the extraction path. The same page always yields the same Markdown — so you can write golden-file tests against it.
🧭 Map without rendering	Merge sitemaps, robots directives, homepage links and `/llms.txt` into one deduplicated URL list in seconds.
🔒 Fair by default	Identifiable bot UA + contact URL, `robots.txt` honoured by default, per-host rate limiting, SSRF-hardened.

# ⚡ Quickstart

bash

# Scrape a page → clean Markdown + structured data + links
curl -s -X POST https://www.ten-dril.com/v1/scrape \
  -H 'content-type: application/json' \
  -d '{"url":"https://www.iana.org","formats":["markdown","structured","links"]}'

# Map a site → every URL, no rendering
curl -s -X POST https://www.ten-dril.com/v1/map \
  -H 'content-type: application/json' \
  -d '{"url":"https://www.iana.org","limit":50}'

Every response is { "success": boolean, "data"?: …, "error"?: { "code", "message" } } with an X-Request-Id header.

# 🤖 Give it to your AI agent

Hand Tendril to any agent — two drop-in integrations, downloadable from the site:

	Download	Install
MCP server — single file, no `npm install`	`tendril-mcp.mjs` · `.zip`	`claude mcp add tendril -- node /path/to/tendril-mcp.mjs`
Agent Skill — `SKILL.md`	`tendril-skill.zip`	`unzip tendril-skill.zip -d ~/.claude/skills/`

The MCP server exposes tendril_scrape, tendril_map, tendril_status. Both point at https://www.ten-dril.com and need no API key today.

# 🏛️ Architecture — three-tier escalation router

Always start at the cheapest tier; escalate only on evidence of failure.

text

   request ──▶ Router ──shouldEscalate()──┐
                 │                         │
        ┌────────┼─────────────┐          │
        ▼        ▼             ▼          │
   ┌─────────┐ ┌───────────┐ ┌──────────┐ │
   │ TIER 0  │ │  TIER 1   │ │  TIER 2  │ │
   │ undici  │▶│ WKWebView │▶│  Safari  │ │  ← Tiers 1/2 = next phase
   │  HTTP   │ │   pool    │ │driver    │ │
   └────┬────┘ └─────┬─────┘ └────┬─────┘ │
        └────────────┴────────────┴───────┘
                      ▼
             Extraction engine (deterministic)
                      ▼
              markdown · structured · links

# 📦 Endpoints

Method	Path	Purpose
`POST`	`/v1/scrape`	Fetch one page → Markdown, structured data, links, metadata
`POST`	`/v1/map`	Discover a site's URLs (sitemaps + robots + homepage + `/llms.txt`)
`GET`	`/v1/status`	Per-tier availability + pipeline version
`GET`	`/healthz`	Shallow health for uptime monitors

# 🧩 What's built

Area	Spec	Status
Error taxonomy (single source)	§15	✅
`Result<T,E>` + typed errors, strict TS	§21	✅
URL normalization	§10.1	✅
Escalation decision `shouldEscalate` (pure)	§3.4	✅
SSRF validation (resolve-then-validate + IP pinning)	§16.5	✅
Tier 0 HTTP fetch (Safari header order, manual redirects, gzip/br)	§3	✅
Deterministic extraction (JSON-LD/OG, Readability + density, Turndown)	§8	✅
robots.txt (RFC 9309) + `/v1/map`	§10.1, §14.3	✅
`/v1/scrape`, `/v1/status`, playground + docs UI	§14	✅
MCP server + agent Skill	—	✅
Deployment: m3u96a + PM2 + ngrok	§16	✅
Tier 1 WKWebView (Swift), Tier 2 Safari, proxies, crawl, cache	§4–13	🔜

Under tier: "auto", a page that needs escalation returns ERR_TARGET_BLOCKED until Tiers 1/2 ship — force tier: "http" for raw Tier 0 output.

# 🛠️ Tech stack

# 💻 Develop

bash

pnpm install
pnpm test          # 86 hermetic unit/contract tests, no network
pnpm build         # tsc -b, strict, emits dist/
pnpm dev:api       # Fastify on 127.0.0.1:3000

# 🚀 Deploy

Production host m3u96a; ingress via ngrok reserved domain www.ten-dril.com. API binds 127.0.0.1 only — ngrok is the sole ingress.

bash

TENDRIL_NODE=M3U96a TENDRIL_PORT=8092 ./scripts/deploy.sh   # rsync + install + build + PM2 + smoke
./scripts/health.sh https://www.ten-dril.com                # smoke an existing deployment

# 🗂️ Layout

text

packages/shared        Result, error taxonomy, types, logger, URL normalization
packages/router        shouldEscalate (§3.4)
packages/egress        SSRF validation + IP classification (§16.5)
packages/fetcher-http  Tier 0 undici client (§3)
packages/extract       deterministic pipeline (§8): structured · boilerplate · markdown
packages/frontier      robots.txt (§10.1), sitemaps, /map discovery (§14.3)
apps/api               Fastify public surface + playground UI (§14)
integrations/mcp       Model Context Protocol server (single-file bundle)
integrations/skill     Agent Skill (SKILL.md)
deploy/ · scripts/     PM2 ecosystem, ngrok config, deploy.sh, health.sh (§16)
test/fixtures/html     saved pages for offline extraction tests (§20)

Built by Simon-Pierre Boucher · contact@spboucher.ai

_{Stealth means matching a real browser's fingerprint — not hiding who we are. Tendril honours robots.txt and identifies itself.}