spb/tendril Public
Tendril — web ingestion platform (scrape/crawl/map/search) on macOS Apple Silicon: WebKit fidelity, authenticated pages, deterministic testable extraction. A self-hosted Firecrawl alternative.
JavaScript 82.6%
TypeScript 11.8%
HTML 5.3%
-
feat: add /map, robots.txt enforcement, and m3u96a deployment
…
- frontier: RFC 9309 robots.txt parser + matcher (longest-match, *,$), 24h cache with 5xx=disallow-1h, recursive sitemap/index discovery, mapSite() merging sitemaps + homepage links + /llms.txt (§10.1, §14.3) - scrape: enforce robots.txt by default (respectRobots), ERR_ROBOTS_DENIED (§26) - api: POST /v1/map, GET /v1/status, @fastify/compress (br/gzip), graceful shutdown; workspace packages now expose dist/ for production node execution - deploy: PM2 ecosystem (loopback API + ngrok www.ten-dril.com), ngrok.yml, scripts/deploy.sh + health.sh, deployed to m3u96a:8092 (§16) 96 tests green. Ingress blocked only on reserving www.ten-dril.com in the ngrok dashboard (DNS CNAME already live). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>