# WebSensor

Detect What Changed. Know Why It Matters. — https://www.websensor.io

WebSensor is a global sensor network for the changing Web. It continuously monitors official public sources (newsrooms, status pages, changelogs, regulators, registries, APIs), preserves every observation, detects raw changes, filters noise, interprets meaningful changes into events, links them to entities, scores their importance and confidence, clusters related events and publishes them live over WebSocket.

Sources → Sensors/Connectors → Fetch (conditional) → Snapshot (immutable) → Diff → Heuristics
       → Interpretation (Claude, budgeted) → Entities → Novelty/Clustering → Importance → Event
       → Redis stream → WebSocket gateway → www.websensor.io

• 2 681 organizations, 4 052 curated sensors (2026-09-08), every one fetched and parsed by the registry validator before being committed: the founding 271 (config/sources.yaml) plus 22 site-class fragments in config/sources.d/ — open source, central banks & finance, energy/climate/weather, telecom & internet infrastructure, retail/consumer/travel, gaming & entertainment, universities & research, international organizations/NGOs/standards, sports, crypto, transport/aviation/space, consumer safety/food/agriculture, housing/labour/open data, enterprise SaaS, health systems & medtech, politics/elections/courts, SEC EDGAR filings (254 issuers), package registries, web posture (robots.txt, TLS, DNS, headers, RDAP), OpenAPI contracts, status JSON pages, documents & data files — plus validated discovery.
• 16 connector families: HTTP (HTML/JSON/HEAD), RSS/Atom/JSON Feed, sitemaps, Statuspage, Instatus/incident.io/ Status.io (statusjson), GitHub, JSON APIs (jsonlist), package registries (npm, PyPI, crates.io, RubyGems, NuGet, Packagist, Hex, Go proxy, Homebrew, Docker Hub), SEC EDGAR submissions (8-K items decoded), OpenAPI/Swagger contract fingerprints, CSV/TSV time series (FRED, ECB, NOAA…), PDF text (pdf.js), DNS records, TLS certificates, HTTP security headers, RDAP registration — and a budgeted Scrapfly fallback.
• Authoring guide and connector cheat-sheet: docs/registry/AUTHORING.md; validator: node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment.yaml> (no database needed).
• Every event is auditable: event → change → snapshot A/B → fetch run → sensor → source, with raw evidence.
• Observed vs inferred is always labelled; silent (unannounced) changes are flagged.

# Quick start

createdb websensor && cp .env.example .env && pnpm install
npx tsx apps/engine/src/cli.ts sync        # load the registry
pnpm dev:api & pnpm dev:web & pnpm dev:engine
open http://localhost:8260

Docs: CLAUDE.md (repo guide), docs/ARCHITECTURE.md, docs/connectors/*.md, deploy/README.md, API docs at /api on the site. Public API: GET /api/v1/events, /api/v1/entities/{id}, /api/v1/sources/{id}, /api/v1/domains/{domain}/timeline, /api/v1/changes/{id}, wss://…/api/v1/live, /api/v1/feed.rss.