SPB Git forge

spb/websensor

Public
33commits 1branches 0releases
3.4 MBsize
maindefault branch
10 days agolast push
TypeScript 55.4% Python 43.2% SQL 1.2%
1.4 KB

# Connector: RSS / Atom / JSON Feed (rss)

Purpose: newsrooms, blogs, changelogs, advisories, government news APIs (Canada.ca Atom), releases. Sensor types: RSS, ATOM. Retrieval: conditional GET; Accept prefers feed types. Parsing: RSS 2.0, Atom, RSS 1.0/RDF, JSON Feed 1.x; tolerant to BOM/leading junk, CDATA, HTML in descriptions (stripped), missing GUIDs (fallback link → hash of title+summary), feedburner:origLink, odd RFC-822 zones. Items deduplicated by key. Normalization: keyed list (key = GUID/id/link), compareFields: title, summary (an edited item is a "modified" change). Sensor state.seenKeys (last 2000) makes items "new" only when never seen; items leaving the window are never "removed"; items older than 14 days are ignored as backfills. publishedAt = newest item. Config: maxItems (default 100). Failure handling: HTML instead of a feed → parse_error (html_not_feed), health DEGRADED after repeated failures, ERROR after 5. Known quirks: Blogger feeds embed full posts (use feeds/posts/summary?max-results=N); the SEC EDGAR current filings Atom is extremely high volume (not monitored by default); many press feeds are behind bot management. Tests: packages/connectors/src/connectors.test.ts with tests/fixtures/rss-before.xml, rss-after.xml, atom.xml. Verified: 2026-09-08 on ~140 live feeds (AWS What's New, Fed press, CISA advisories, arXiv…).