# Connector: Sitemap (sitemap)

Purpose: page-level discovery — new URLs, removed URLs, lastmod bumps — including news sitemaps. Sensor type: SITEMAP. Retrieval: conditional GET, 30 MB limit, gzip transparently decompressed (magic-byte check), sitemap index followed (maxChildren, default 6; newest first when lastmod is present). Normalization: keyed list by URL, compareFields: lastmod; sorted newest first; include/exclude regexes; maxUrls (default 5000). A > 50 % shrink versus the previous snapshot is treated as a partial response (no removals reported). Events: page_created (new URL), page_removed (URL gone, only after confirmations), documentation_change etc. via heuristics on titles/URLs. Known quirks: huge multilingual indexes (BMW: 100+ children) — choose a specific child sitemap; image/video sitemaps are skipped by discovery. Tests: fixtures tests/fixtures/sitemap.xml. Verified: 2026-09-08 (anthropic.com sitemap 524 URLs, BMW press sitemap).