SPB Git

spb/tendril Public

Tendril — web ingestion platform (scrape/crawl/map/search) on macOS Apple Silicon: WebKit fidelity, authenticated pages, deterministic testable extraction. A self-hosted Firecrawl alternative.

web-scrapingcrawlerwebkitmacostypescriptfirecrawlswiftfastify
JavaScript 82.6% TypeScript 11.8% HTML 5.3%

docs: legendary README — badges, live status, agent integrations, architecture

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 9 h ago (Aug 10, 2026) parent ad2e90d

Showing 1 changed file with +162 and −68

modified README.md +162 −68
@@ -1,95 +1,189 @@
1 +<div align="center">
2 +
3 +<img src="https://www.ten-dril.com/logo.svg" width="96" height="96" alt="Tendril" />
4 +
1 5 # Tendril
2 6
3 −Web ingestion platform (search / scrape / crawl / map) for macOS Apple Silicon.
4 −See [`CLAUDE.md`](./CLAUDE.md) for the full product and architecture spec.
7 +**Web ingestion that reaches what other crawlers can't.**
8 +
9 +Scrape · Map — running on a residential Mac, in **real WebKit**, with **deterministic, testable** extraction.
10 +
11 +<br />
12 +
13 +[![live](https://img.shields.io/website?url=https%3A%2F%2Fwww.ten-dril.com%2Fhealthz&style=flat-square&label=www.ten-dril.com&up_message=live&up_color=4ade80&down_message=down&down_color=ff6b6b)](https://www.ten-dril.com)
14 +![tests](https://img.shields.io/badge/tests-86_passing-4ade80?style=flat-square&logo=vitest&logoColor=white)
15 +![TypeScript](https://img.shields.io/badge/TypeScript-strict-3178C6?style=flat-square&logo=typescript&logoColor=white)
16 +![Node](https://img.shields.io/badge/Node-%E2%89%A522-5FA04E?style=flat-square&logo=nodedotjs&logoColor=white)
17 +![Fastify](https://img.shields.io/badge/Fastify-5-000000?style=flat-square&logo=fastify&logoColor=white)
18 +![pnpm](https://img.shields.io/badge/pnpm-workspaces-F69220?style=flat-square&logo=pnpm&logoColor=white)
19 +
20 +![engine](https://img.shields.io/badge/engine-WebKit-2dd4bf?style=flat-square&logo=safari&logoColor=white)
21 +![extraction](https://img.shields.io/badge/extraction-deterministic-f5c451?style=flat-square)
22 +![LLM in pipeline](https://img.shields.io/badge/LLMs_in_extraction-0-07100c?style=flat-square&labelColor=4ade80)
23 +![host](https://img.shields.io/badge/host-m3u96a_·_macOS_ARM-111?style=flat-square&logo=apple&logoColor=white)
24 +![ingress](https://img.shields.io/badge/ingress-ngrok-1F1E37?style=flat-square&logo=ngrok&logoColor=white)
25 +![agent ready](https://img.shields.io/badge/agent-MCP_%2B_Skill-4ade80?style=flat-square)
26 +
27 +**[🌐 Live](https://www.ten-dril.com) · [▶ Playground](https://www.ten-dril.com/#playground) · [🤖 For agents](https://www.ten-dril.com/#agents) · [📖 Docs](https://www.ten-dril.com/#docs) · [📦 Source](https://git.spboucher.ai/tendril)**
28 +
29 +</div>
30 +
31 +---
32 +
33 +## ✦ Why Tendril
34 +
35 +Firecrawl runs in the cloud, on datacenter IPs, in headless Chromium. Tendril runs on a **residential machine**, in **WebKit** — Safari's actual engine — behind a real IP.
36 +
37 +| | |
38 +|---|---|
39 +| 🕸️ **Browser fidelity** | WebKit on macOS ARM behind a residential IP is *not* a headless-Chromium signature. Sites that block others often don't block Tendril. |
40 +| 📐 **Deterministic extraction** | No LLM in the extraction path. The same page always yields the same Markdown — so you can write **golden-file tests** against it. |
41 +| 🧭 **Map without rendering** | Merge sitemaps, robots directives, homepage links and `/llms.txt` into one deduplicated URL list in seconds. |
42 +| 🔒 **Fair by default** | Identifiable bot UA + contact URL, `robots.txt` honoured by default, per-host rate limiting, SSRF-hardened. |
43 +
44 +---
45 +
46 +## ⚡ Quickstart
5 47
6 −> **Author:** Simon-Pierre Boucher — <contact@spboucher.ai>
48 +```bash
49 +# Scrape a page → clean Markdown + structured data + links
50 +curl -s -X POST https://www.ten-dril.com/v1/scrape \
51 + -H 'content-type: application/json' \
52 + -d '{"url":"https://www.iana.org","formats":["markdown","structured","links"]}'
53 +
54 +# Map a site → every URL, no rendering
55 +curl -s -X POST https://www.ten-dril.com/v1/map \
56 + -H 'content-type: application/json' \
57 + -d '{"url":"https://www.iana.org","limit":50}'
58 +```
59 +
60 +Every response is `{ "success": boolean, "data"?: …, "error"?: { "code", "message" } }` with an `X-Request-Id` header.
61 +
62 +---
7 63
8 −## Status — Phase 1 (foundations)
64 +## 🤖 Give it to your AI agent
9 65
10 −Per the roadmap (`CLAUDE.md` §24), Phase 1 delivers Tier 0 fetching, the
11 −deterministic extraction pipeline, `/scrape`, and the error taxonomy. This is
12 −what currently exists and is tested:
66 +Hand Tendril to any agent — two drop-in integrations, downloadable from the site:
13 67
14 −| Area | Spec | State |
68 +| | Download | Install |
15 69 |---|---|---|
16 −| Error taxonomy (single source) | §15 | ✅ `packages/shared/src/errors.ts` |
17 −| `Result<T,E>` + typed errors | §21 | ✅ `packages/shared` |
18 −| URL normalization | §10 rule 1, §10.1 | ✅ `packages/shared/src/url.ts` |
19 −| Escalation decision `shouldEscalate` (pure) | §3.4 | ✅ `packages/router` |
20 −| SSRF validation (resolve-then-validate) | §16.5 | ✅ `packages/egress` |
21 −| Tier 0 HTTP fetch (header order, manual redirects, IP pinning) | §3 | ✅ `packages/fetcher-http` |
22 −| Extraction: structured-data harvest | §8.3 | ✅ `packages/extract` |
23 −| Extraction: boilerplate (Readability + density fallback) | §8.2 | ✅ |
24 −| Extraction: Turndown rules (tables, code, figures, dl…) | §8.6 | ✅ |
25 −| robots.txt parsing + enforcement (RFC 9309) | §10.1, §26 | ✅ `packages/frontier` |
26 −| Sitemap discovery / `POST /v1/map` | §14.3 | ✅ `packages/frontier` |
27 −| `POST /v1/scrape`, `/v1/status`, `/healthz` | §14.1 | ✅ `apps/api` |
28 −| Deployment: m3u96a + PM2 + ngrok | §16 | ✅ `scripts/deploy.sh`, `deploy/` |
29 −
30 −86 unit/contract tests, all hermetic (no network).
31 −
32 −**Not yet built** (later phases): Tier 1 WKWebView daemon (§4), Tier 2 Safari
33 −(§5), proxy layer (§6), profiles (§7), selector/inference extraction (§8.4–8.5),
34 −non-HTML formats (§9), full crawl frontier (§10), cache (§11), Postgres/Redis/BullMQ
35 −(§12–13), `/crawl` `/search` `/extract`, SDKs (§22).
36 −
37 −Because Tiers 1–2 do not exist yet, a page that `shouldEscalate` flags under
38 −`tier: "auto"` **fails closed** with `ERR_TARGET_BLOCKED` and the decisive
39 −reason, rather than returning a challenge/interstitial page as if it were
40 −content. Force `tier: "http"` to extract Tier 0 output regardless.
41 −
42 −## Layout
43 −
44 −Monorepo (pnpm workspaces) following `CLAUDE.md` §23:
70 +| **MCP server** — single file, no `npm install` | [`tendril-mcp.mjs`](https://www.ten-dril.com/downloads/tendril-mcp.mjs) · [`.zip`](https://www.ten-dril.com/downloads/tendril-mcp.zip) | `claude mcp add tendril -- node /path/to/tendril-mcp.mjs` |
71 +| **Agent Skill** — `SKILL.md` | [`tendril-skill.zip`](https://www.ten-dril.com/downloads/tendril-skill.zip) | `unzip tendril-skill.zip -d ~/.claude/skills/` |
72 +
73 +The MCP server exposes `tendril_scrape`, `tendril_map`, `tendril_status`. Both point at `https://www.ten-dril.com` and need no API key today.
74 +
75 +---
76 +
77 +## 🏛️ Architecture — three-tier escalation router
78 +
79 +Always start at the cheapest tier; escalate only on evidence of failure.
45 80
46 81 ```
47 −packages/shared Result, error taxonomy, types, logger, URL normalization
48 −packages/router shouldEscalate (§3.4)
49 −packages/egress SSRF validation + IP classification (§16.5)
50 −packages/fetcher-http Tier 0 undici client (§3)
51 −packages/extract deterministic pipeline (§8): structured, boilerplate, markdown
52 −packages/frontier robots.txt (§10.1), sitemaps, /map discovery (§14.3)
53 −apps/api Fastify public surface (§14)
54 −deploy/ PM2 ecosystem, ngrok config (§16)
55 −scripts/ deploy.sh, health.sh (§16.4)
56 −test/fixtures/html saved pages for offline extraction tests (§20)
82 + request ──▶ Router ──shouldEscalate()──┐
83 + │ │
84 + ┌────────┼─────────────┐ │
85 + ▼ ▼ ▼ │
86 + ┌─────────┐ ┌───────────┐ ┌──────────┐ │
87 + │ TIER 0 │ │ TIER 1 │ │ TIER 2 │ │
88 + │ undici │▶│ WKWebView │▶│ Safari │ │ ← Tiers 1/2 = next phase
89 + │ HTTP │ │ pool │ │driver │ │
90 + └────┬────┘ └─────┬─────┘ └────┬─────┘ │
91 + └────────────┴────────────┴───────┘
92 + ▼
93 + Extraction engine (deterministic)
94 + ▼
95 + markdown · structured · links
57 96 ```
58 97
59 −## Deploy
98 +---
60 99
61 −Production host is `m3u96a` (§16), ingress via ngrok reserved domain
62 −`www.ten-dril.com`. The API binds to `127.0.0.1` only; ngrok is the sole ingress.
100 +## 📦 Endpoints
63 101
64 −```bash
65 −TENDRIL_NODE=M3U96a TENDRIL_PORT=8092 ./scripts/deploy.sh # rsync + install + build + PM2 + smoke
66 −./scripts/health.sh https://www.ten-dril.com # smoke an existing deployment
67 −```
102 +| Method | Path | Purpose |
103 +|---|---|---|
104 +| `POST` | `/v1/scrape` | Fetch one page → Markdown, structured data, links, metadata |
105 +| `POST` | `/v1/map` | Discover a site's URLs (sitemaps + robots + homepage + `/llms.txt`) |
106 +| `GET` | `/v1/status` | Per-tier availability + pipeline version |
107 +| `GET` | `/healthz` | Shallow health for uptime monitors |
108 +
109 +---
68 110
69 −The ngrok domain must be reserved once at
70 −`https://dashboard.ngrok.com/domains/new` (the DNS CNAME is already in place);
71 −`tendril-ngrok` connects automatically once it is.
111 +## 🧩 What's built
72 112
73 −## Develop
113 +| Area | Spec | Status |
114 +|---|---|---|
115 +| Error taxonomy (single source) | §15 | ✅ |
116 +| `Result<T,E>` + typed errors, strict TS | §21 | ✅ |
117 +| URL normalization | §10.1 | ✅ |
118 +| Escalation decision `shouldEscalate` (pure) | §3.4 | ✅ |
119 +| SSRF validation (resolve-then-validate + IP pinning) | §16.5 | ✅ |
120 +| Tier 0 HTTP fetch (Safari header order, manual redirects, gzip/br) | §3 | ✅ |
121 +| Deterministic extraction (JSON-LD/OG, Readability + density, Turndown) | §8 | ✅ |
122 +| robots.txt (RFC 9309) + `/v1/map` | §10.1, §14.3 | ✅ |
123 +| `/v1/scrape`, `/v1/status`, playground + docs UI | §14 | ✅ |
124 +| MCP server + agent Skill | — | ✅ |
125 +| Deployment: m3u96a + PM2 + ngrok | §16 | ✅ |
126 +| **Tier 1 WKWebView (Swift)**, Tier 2 Safari, proxies, crawl, cache | §4–13 | 🔜 |
127 +
128 +> Under `tier: "auto"`, a page that needs escalation returns `ERR_TARGET_BLOCKED` until Tiers 1/2 ship — force `tier: "http"` for raw Tier 0 output.
129 +
130 +---
131 +
132 +## 🛠️ Tech stack
133 +
134 +![TypeScript](https://img.shields.io/badge/TypeScript-3178C6?style=flat-square&logo=typescript&logoColor=white)
135 +![undici](https://img.shields.io/badge/undici-HTTP-000?style=flat-square)
136 +![linkedom](https://img.shields.io/badge/linkedom-DOM-2dd4bf?style=flat-square)
137 +![Turndown](https://img.shields.io/badge/Turndown-Markdown-f5c451?style=flat-square)
138 +![Readability](https://img.shields.io/badge/Readability-boilerplate-4ade80?style=flat-square)
139 +![Zod](https://img.shields.io/badge/Zod-schemas-3E67B1?style=flat-square&logo=zod&logoColor=white)
140 +![Vitest](https://img.shields.io/badge/Vitest-86_tests-6E9F18?style=flat-square&logo=vitest&logoColor=white)
141 +![MCP](https://img.shields.io/badge/MCP-server-4ade80?style=flat-square)
142 +
143 +---
144 +
145 +## 💻 Develop
74 146
75 147 ```bash
76 148 pnpm install
77 −pnpm test # 74 unit/contract tests, hermetic, no network
149 +pnpm test # 86 hermetic unit/contract tests, no network
78 150 pnpm build # tsc -b, strict, emits dist/
79 151 pnpm dev:api # Fastify on 127.0.0.1:3000
80 152 ```
81 153
82 −Try it:
154 +## 🚀 Deploy
155 +
156 +Production host `m3u96a`; ingress via ngrok reserved domain `www.ten-dril.com`. API binds `127.0.0.1` only — ngrok is the sole ingress.
83 157
84 158 ```bash
85 −curl -s -X POST http://127.0.0.1:3000/v1/scrape \
86 − -H 'content-type: application/json' \
87 − -d '{"url":"https://example.com","formats":["markdown","links","structured"]}'
159 +TENDRIL_NODE=M3U96a TENDRIL_PORT=8092 ./scripts/deploy.sh # rsync + install + build + PM2 + smoke
160 +./scripts/health.sh https://www.ten-dril.com # smoke an existing deployment
88 161 ```
89 162
90 −## Conventions
163 +---
164 +
165 +## 🗂️ Layout
166 +
167 +```
168 +packages/shared Result, error taxonomy, types, logger, URL normalization
169 +packages/router shouldEscalate (§3.4)
170 +packages/egress SSRF validation + IP classification (§16.5)
171 +packages/fetcher-http Tier 0 undici client (§3)
172 +packages/extract deterministic pipeline (§8): structured · boilerplate · markdown
173 +packages/frontier robots.txt (§10.1), sitemaps, /map discovery (§14.3)
174 +apps/api Fastify public surface + playground UI (§14)
175 +integrations/mcp Model Context Protocol server (single-file bundle)
176 +integrations/skill Agent Skill (SKILL.md)
177 +deploy/ · scripts/ PM2 ecosystem, ngrok config, deploy.sh, health.sh (§16)
178 +test/fixtures/html saved pages for offline extraction tests (§20)
179 +```
180 +
181 +---
182 +
183 +<div align="center">
184 +
185 +Built by **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai)
186 +
187 +<sub>Stealth means matching a real browser's fingerprint — not hiding who we are. Tendril honours robots.txt and identifies itself.</sub>
91 188
92 −- TypeScript strict, `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`, no `any`.
93 −- Fetchers/extractors return `Result<T, TendrilError>`; exceptions only signal bugs (§21).
94 −- Every code file starts with an author header comment.
95 −- Extraction stages are pure functions tested offline against fixtures (§8, §20).
189 +</div>
96 190