# WebSensor.io — product brief (condensed from the founding CLAUDE.md, 2026-09-08)

Mission. A production-grade real-time web intelligence platform: a global sensor network for the changing Web. Not a generic crawler. Sources → sensors/connectors → fetch → change detection → AI interpretation → entity resolution → event clustering → importance scoring → historical event store → real-time WebSocket feed. Feel: Bloomberg Terminal × GitHub activity × Cloudflare Radar × security monitoring × an Internet-wide changelog.

Philosophy. Never just "HTML changed". Answer: what / when / where changed, is it meaningful, which entity, how important, part of a larger event, publicly announced or silent, has it happened before. Preserve both the RAW OBSERVATION and the INTERPRETED EVENT.

Engineering principles. Read docs and test integrations; prefer webhooks → WebSocket → SSE → official API → RSS/Atom → structured feeds → JSON/XML → sitemaps → conditional GET → static HTML → rendered HTML → Firecrawl → Scrapfly (fallback only, where permitted). Never defeat CAPTCHAs or explicit access controls; honour robots, terms, rate limits. Cache and deduplicate aggressively, retain history, keep connectors independently testable, never discard raw evidence, idempotent processing, UTC everywhere. Build a connector framework, not one crawler.

Sensors and tiers. Every endpoint is a sensor (type, tier, interval, importance weight). Tiers S (15–60 s), A (1–5 min), B (5–30 min), C (30 min–6 h), D (6–24 h). Adaptive polling from change frequency/recency, patterns, importance history, error rate and cache behaviour. Conditional requests (ETag / Last-Modified) preferred.

Snapshots, canonicalization, diffs. Immutable snapshots with raw/canonical/semantic hashes, blobs in object storage (content-addressed, compressed). Strip render noise before diffing. Diff engines: text, DOM, JSON, table, sitemap, PDF. Staged semantic detection: raw diff → heuristics → structured extraction → semantic classification → LLM interpretation → event candidate.

Events. Canonical schema (ids, sensor/entities, detected/observed timestamps, type, title, summary, importance/confidence/novelty, snapshots, diff, categories, silent_change, processing version). Open taxonomy (announcement, product_launch, model_release, pricing_change, policy/terms/API changes, security_advisory, vulnerability, breach, incident, outage, recall, drug_approval, filings, earnings, leadership, acquisition, funding, layoffs, new_region, DNS/certificate changes, releases, standards, government/economic releases, page_created/removed, unknown…). Importance 0–100 = 25 % severity + 20 % source + 15 % entity + 15 % novelty + 10 % magnitude + 5 % confirmation + 5 % user impact + 5 % unusualness (components stored). Confidence is separate. Novelty against recent events (no 50 events for one syndicated press release). Clustering by entity overlap, similarity, time proximity. Silent change = important change without announcement (flagship).

Entities & graph. Everything resolves to entities (organization, product, AI_model, API, drug, person, standard, vulnerability…), aliases, relations (owns, released, approved, acquired…), PostgreSQL first. Timelines per entity and per URL ("git history for the public Web"), compare any two versions. Latency fields (published / first seen / detected / processed / published to feed) — compete on detecting first.

Real-time & product. Redis Streams → WebSocket gateway with channels (global, breaking, categories, entity, source, watchlist). Frontend: the live feed is the hero; nav Live · Breaking · Explore · Sources · Entities · Timelines · Silent Changes · Watchlists · Alerts · API; event detail with importance/confidence, evidence, semantic + raw diffs, related events, cluster, timeline, reliability; Explore, Trending, activity anomaly per source, early signals (never speculation as fact), watchlists, alerts, search, dark + light, performance targets, stable canonical URLs, machine-readable (API, RSS, JSON-LD, sitemaps).

Platform. TypeScript/Node/Fastify, PostgreSQL, Redis, object store, Playwright later; Python micro-services where useful; Next.js frontend. Deployed on the private Apple Silicon cluster as separable processes, exposed through ngrok at https://www.websensor.io (apex redirects to www). Never expose datastores publicly. Strict security: rate limiting, validation, SSRF protection (private ranges, metadata endpoints, cluster hosts blocked), redirect validation, size/time limits. Observability: Prometheus metrics, structured logs, request ids.

Registry. 200 launch organizations across AI, cloud, developer platforms, cybersecurity, consumer tech, finance/regulators, government/statistics, healthcare/pharma, science/space, automotive, commerce/payments, internet platforms/standards. Seed format in config/sources.yaml; discovery validates every candidate. Expected 1 000–4 000 sensors at phase 1, then 10 k, 100 k, millions of monitored endpoints.

MVP definition. Register sources, discover endpoints, schedule, fetch efficiently, preserve snapshots, detect raw changes, eliminate noise, classify, create events, resolve entities, compute importance, cluster, publish over WebSocket, live feed, event detail, raw + semantic diffs, source history, watchlists, connector health, continuous operation on the cluster. No fake connectors, no placeholder production data.

Build order. Foundation → structured sources → intelligence → product → real-time → advanced acquisition. Tagline. WebSensor — Detect What Changed. Know Why It Matters.