# WebSensor

**Detect What Changed. Know Why It Matters.** — https://www.websensor.io

WebSensor is a global sensor network for the changing Web. It continuously monitors official public sources
(newsrooms, status pages, changelogs, regulators, registries, APIs), preserves every observation, detects raw
changes, filters noise, interprets meaningful changes into events, links them to entities, scores their
importance and confidence, clusters related events and publishes them live over WebSocket.

```
Sources → Sensors/Connectors → Fetch (conditional) → Snapshot (immutable) → Diff → Heuristics
       → Interpretation (Claude, budgeted) → Entities → Novelty/Clustering → Importance → Event
       → Redis stream → WebSocket gateway → www.websensor.io
```

- **2 681 organizations, 4 052 curated sensors** (2026-09-08), every one fetched and parsed by the registry validator
  before being committed: the founding 271 (`config/sources.yaml`) plus 22 site-class fragments in
  `config/sources.d/` — open source, central banks & finance, energy/climate/weather, telecom & internet
  infrastructure, retail/consumer/travel, gaming & entertainment, universities & research, international
  organizations/NGOs/standards, sports, crypto, transport/aviation/space, consumer safety/food/agriculture,
  housing/labour/open data, enterprise SaaS, health systems & medtech, politics/elections/courts, SEC EDGAR filings
  (254 issuers), package registries, web posture (robots.txt, TLS, DNS, headers, RDAP), OpenAPI contracts, status
  JSON pages, documents & data files — plus validated discovery.
- **16 connector families**: HTTP (HTML/JSON/HEAD), RSS/Atom/JSON Feed, sitemaps, Statuspage, Instatus/incident.io/
  Status.io (`statusjson`), GitHub, JSON APIs (`jsonlist`), package registries (npm, PyPI, crates.io, RubyGems, NuGet,
  Packagist, Hex, Go proxy, Homebrew, Docker Hub), SEC EDGAR submissions (8-K items decoded), OpenAPI/Swagger
  contract fingerprints, CSV/TSV time series (FRED, ECB, NOAA…), PDF text (pdf.js), DNS records, TLS certificates,
  HTTP security headers, RDAP registration — and a budgeted Scrapfly fallback.
- Authoring guide and connector cheat-sheet: `docs/registry/AUTHORING.md`; validator:
  `node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment.yaml>` (no database needed).
- Every event is auditable: event → change → snapshot A/B → fetch run → sensor → source, with raw evidence.
- Observed vs inferred is always labelled; silent (unannounced) changes are flagged.

## Quick start
```
createdb websensor && cp .env.example .env && pnpm install
npx tsx apps/engine/src/cli.ts sync        # load the registry
pnpm dev:api & pnpm dev:web & pnpm dev:engine
open http://localhost:8260
```

Docs: `CLAUDE.md` (repo guide), `docs/ARCHITECTURE.md`, `docs/connectors/*.md`, `deploy/README.md`,
API docs at `/api` on the site. Public API: `GET /api/v1/events`, `/api/v1/entities/{id}`,
`/api/v1/sources/{id}`, `/api/v1/domains/{domain}/timeline`, `/api/v1/changes/{id}`, `wss://…/api/v1/live`,
`/api/v1/feed.rss`.