spb/tendril Public
Tendril — web ingestion platform (scrape/crawl/map/search) on macOS Apple Silicon: WebKit fidelity, authenticated pages, deterministic testable extraction. A self-hosted Firecrawl alternative.
JavaScript 82.6%
TypeScript 11.8%
HTML 5.3%
1<div align="center">23<img src="https://www.ten-dril.com/logo.svg" width="96" height="96" alt="Tendril" />45# Tendril67**Web ingestion that reaches what other crawlers can't.**89Scrape · Map — running on a residential Mac, in **real WebKit**, with **deterministic, testable** extraction.1011<br />1213[](https://www.ten-dril.com)1415161718192021222324252627**[🌐 Live](https://www.ten-dril.com) · [▶ Playground](https://www.ten-dril.com/#playground) · [🤖 For agents](https://www.ten-dril.com/#agents) · [📖 Docs](https://www.ten-dril.com/#docs) · [📦 Source](https://git.spboucher.ai/tendril)**2829</div>3031---3233## ✦ Why Tendril3435Firecrawl runs in the cloud, on datacenter IPs, in headless Chromium. Tendril runs on a **residential machine**, in **WebKit** — Safari's actual engine — behind a real IP.3637| | |38|---|---|39| 🕸️ **Browser fidelity** | WebKit on macOS ARM behind a residential IP is *not* a headless-Chromium signature. Sites that block others often don't block Tendril. |40| 📐 **Deterministic extraction** | No LLM in the extraction path. The same page always yields the same Markdown — so you can write **golden-file tests** against it. |41| 🧭 **Map without rendering** | Merge sitemaps, robots directives, homepage links and `/llms.txt` into one deduplicated URL list in seconds. |42| 🔒 **Fair by default** | Identifiable bot UA + contact URL, `robots.txt` honoured by default, per-host rate limiting, SSRF-hardened. |4344---4546## ⚡ Quickstart4748```bash49# Scrape a page → clean Markdown + structured data + links50curl -s -X POST https://www.ten-dril.com/v1/scrape \51 -H 'content-type: application/json' \52 -d '{"url":"https://www.iana.org","formats":["markdown","structured","links"]}'5354# Map a site → every URL, no rendering55curl -s -X POST https://www.ten-dril.com/v1/map \56 -H 'content-type: application/json' \57 -d '{"url":"https://www.iana.org","limit":50}'58```5960Every response is `{ "success": boolean, "data"?: …, "error"?: { "code", "message" } }` with an `X-Request-Id` header.6162---6364## 🤖 Give it to your AI agent6566Hand Tendril to any agent — two drop-in integrations, downloadable from the site:6768| | Download | Install |69|---|---|---|70| **MCP server** — single file, no `npm install` | [`tendril-mcp.mjs`](https://www.ten-dril.com/downloads/tendril-mcp.mjs) · [`.zip`](https://www.ten-dril.com/downloads/tendril-mcp.zip) | `claude mcp add tendril -- node /path/to/tendril-mcp.mjs` |71| **Agent Skill** — `SKILL.md` | [`tendril-skill.zip`](https://www.ten-dril.com/downloads/tendril-skill.zip) | `unzip tendril-skill.zip -d ~/.claude/skills/` |7273The MCP server exposes `tendril_scrape`, `tendril_map`, `tendril_status`. Both point at `https://www.ten-dril.com` and need no API key today.7475---7677## 🏛️ Architecture — three-tier escalation router7879Always start at the cheapest tier; escalate only on evidence of failure.8081```82 request ──▶ Router ──shouldEscalate()──┐83 │ │84 ┌────────┼─────────────┐ │85 ▼ ▼ ▼ │86 ┌─────────┐ ┌───────────┐ ┌──────────┐ │87 │ TIER 0 │ │ TIER 1 │ │ TIER 2 │ │88 │ undici │▶│ WKWebView │▶│ Safari │ │ ← Tiers 1/2 = next phase89 │ HTTP │ │ pool │ │driver │ │90 └────┬────┘ └─────┬─────┘ └────┬─────┘ │91 └────────────┴────────────┴───────┘92 ▼93 Extraction engine (deterministic)94 ▼95 markdown · structured · links96```9798---99100## 📦 Endpoints101102| Method | Path | Purpose |103|---|---|---|104| `POST` | `/v1/scrape` | Fetch one page → Markdown, structured data, links, metadata |105| `POST` | `/v1/map` | Discover a site's URLs (sitemaps + robots + homepage + `/llms.txt`) |106| `GET` | `/v1/status` | Per-tier availability + pipeline version |107| `GET` | `/healthz` | Shallow health for uptime monitors |108109---110111## 🧩 What's built112113| Area | Spec | Status |114|---|---|---|115| Error taxonomy (single source) | §15 | ✅ |116| `Result<T,E>` + typed errors, strict TS | §21 | ✅ |117| URL normalization | §10.1 | ✅ |118| Escalation decision `shouldEscalate` (pure) | §3.4 | ✅ |119| SSRF validation (resolve-then-validate + IP pinning) | §16.5 | ✅ |120| Tier 0 HTTP fetch (Safari header order, manual redirects, gzip/br) | §3 | ✅ |121| Deterministic extraction (JSON-LD/OG, Readability + density, Turndown) | §8 | ✅ |122| robots.txt (RFC 9309) + `/v1/map` | §10.1, §14.3 | ✅ |123| `/v1/scrape`, `/v1/status`, playground + docs UI | §14 | ✅ |124| MCP server + agent Skill | — | ✅ |125| Deployment: m3u96a + PM2 + ngrok | §16 | ✅ |126| **Tier 1 WKWebView (Swift)**, Tier 2 Safari, proxies, crawl, cache | §4–13 | 🔜 |127128> Under `tier: "auto"`, a page that needs escalation returns `ERR_TARGET_BLOCKED` until Tiers 1/2 ship — force `tier: "http"` for raw Tier 0 output.129130---131132## 🛠️ Tech stack133134135136137138139140141142143---144145## 💻 Develop146147```bash148pnpm install149pnpm test # 86 hermetic unit/contract tests, no network150pnpm build # tsc -b, strict, emits dist/151pnpm dev:api # Fastify on 127.0.0.1:3000152```153154## 🚀 Deploy155156Production host `m3u96a`; ingress via ngrok reserved domain `www.ten-dril.com`. API binds `127.0.0.1` only — ngrok is the sole ingress.157158```bash159TENDRIL_NODE=M3U96a TENDRIL_PORT=8092 ./scripts/deploy.sh # rsync + install + build + PM2 + smoke160./scripts/health.sh https://www.ten-dril.com # smoke an existing deployment161```162163---164165## 🗂️ Layout166167```168packages/shared Result, error taxonomy, types, logger, URL normalization169packages/router shouldEscalate (§3.4)170packages/egress SSRF validation + IP classification (§16.5)171packages/fetcher-http Tier 0 undici client (§3)172packages/extract deterministic pipeline (§8): structured · boilerplate · markdown173packages/frontier robots.txt (§10.1), sitemaps, /map discovery (§14.3)174apps/api Fastify public surface + playground UI (§14)175integrations/mcp Model Context Protocol server (single-file bundle)176integrations/skill Agent Skill (SKILL.md)177deploy/ · scripts/ PM2 ecosystem, ngrok config, deploy.sh, health.sh (§16)178test/fixtures/html saved pages for offline extraction tests (§20)179```180181---182183<div align="center">184185Built by **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai)186187<sub>Stealth means matching a real browser's fingerprint — not hiding who we are. Tendril honours robots.txt and identifies itself.</sub>188189</div>190