spb/tendril

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Integrations (downloadable from the site so a user can hand Tendril to an agent):
- integrations/mcp: single-file MCP server (tendril_scrape/map/status), bundled
  to one .mjs (no install), tested end-to-end via the MCP SDK client
- integrations/skill: SKILL.md teaching an agent when/how to use Tendril over HTTP
- served at /downloads/{tendril-mcp.mjs,tendril-mcp.zip,tendril-skill.zip}

Site:
- real Tendril logo (spiral-vine SVG) used as favicon + header/footer brand
- "Give it to your agent" section with download buttons + copy-paste MCP/skill config
- playground result upgrade: metrics row (status/latency/tier/cached/pipeline),
  metadata card with thumbnail, reading view with word count + copy/download .md,
  link explorer with internal/external filter + search + export, structured/metadata
  tabs, loading skeleton, richer error view

Deployed to m3u96a; live at https://www.ten-dril.com.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Self-contained landing page served via @fastify/static (apps/api/public):
- hero with Tendril identity (botanical/terminal theme, animated vine SVG)
- live playground: scrape/map/status, formats & options, latency, rendered
  markdown preview, links/metadata/structured tabs, curl/python/js snippets
  with copy, live /v1/status pill
- collapsible API reference with param tables + examples, §15 error table
Explicit API routes keep priority over the static wildcard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- frontier: RFC 9309 robots.txt parser + matcher (longest-match, *,$),
  24h cache with 5xx=disallow-1h, recursive sitemap/index discovery,
  mapSite() merging sitemaps + homepage links + /llms.txt (§10.1, §14.3)
- scrape: enforce robots.txt by default (respectRobots), ERR_ROBOTS_DENIED (§26)
- api: POST /v1/map, GET /v1/status, @fastify/compress (br/gzip), graceful
  shutdown; workspace packages now expose dist/ for production node execution
- deploy: PM2 ecosystem (loopback API + ngrok www.ten-dril.com), ngrok.yml,
  scripts/deploy.sh + health.sh, deployed to m3u96a:8092 (§16)

96 tests green. Ingress blocked only on reserving www.ten-dril.com in the
ngrok dashboard (DNS CNAME already live).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Web ingestion platform (scrape/crawl/map/search) for macOS Apple Silicon,
following CLAUDE.md. Phase 1 (§24) delivered and fully tested (74 tests):

- shared: Result<T,E>, §15 error taxonomy (single source), types, Pino
  logger with redaction, URL normalization (§10.1)
- router: pure shouldEscalate escalation decision (§3.4)
- egress: SSRF validation with resolve-then-validate + IP pinning (§16.5)
- fetcher-http: Tier 0 undici client, Safari header order, manual redirects
  with per-hop SSRF re-validation, meta-refresh, gzip/br (§3)
- extract: deterministic pipeline (§8) — JSON-LD/OG/Twitter harvest,
  Readability + density fallback, Turndown rules (GFM tables, fenced code,
  figures, dl), link extraction
- api: Fastify POST /v1/scrape + /healthz, errors.ts HTTP mapping (§15)

TypeScript strict (noUncheckedIndexedAccess, exactOptionalPropertyTypes),
fetchers/extractors return Result, author header on every code file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>