SPB Git forge

spb/websensor

Public
33commits 1branches 0releases
3.4 MBsize
maindefault branch
10 days agolast push
TypeScript 55.4% Python 43.2% SQL 1.2%
774 B · 9 lines markdown
Rendered Raw Blame History
1# Discovery engine2For a domain: `robots.txt` (Sitemap: lines, image/video sitemaps skipped, news first, ≤ 8), homepage3`<link rel="alternate">` feeds and links to `status.*`/`*.statuspage.io`, well-known feed paths (`/feed`,4`/rss.xml`, `/atom.xml`, `/changelog/feed`, `/releases.atom`…), well-known sitemap paths, and (if `pages`)5HEAD probes of `/changelog`, `/news`, `/pricing`, `/security`… Every candidate is **fetched and parsed**; only6parseable feeds/sitemaps/statuspages are stored in `discovery_candidates` and promoted (≤ 3 feeds, 1 sitemap,71 statuspage, pricing/changelog/security pages) according to the source's `discover` flags. Re-run every8`WS_DISCOVERY_INTERVAL_DAYS` (7). CLI: `cli.ts probe <domain>` (dry run), `cli.ts discover [sourceId…]`.9