# WebSensor.io — product brief (condensed from the founding CLAUDE.md, 2026-09-08)

**Mission.** A production-grade real-time web intelligence platform: *a global sensor network for the changing
Web*. Not a generic crawler. Sources → sensors/connectors → fetch → change detection → AI interpretation →
entity resolution → event clustering → importance scoring → historical event store → real-time WebSocket feed.
Feel: Bloomberg Terminal × GitHub activity × Cloudflare Radar × security monitoring × an Internet-wide changelog.

**Philosophy.** Never just "HTML changed". Answer: what / when / where changed, is it meaningful, which entity,
how important, part of a larger event, publicly announced or silent, has it happened before. Preserve both the
RAW OBSERVATION and the INTERPRETED EVENT.

**Engineering principles.** Read docs and test integrations; prefer webhooks → WebSocket → SSE → official API →
RSS/Atom → structured feeds → JSON/XML → sitemaps → conditional GET → static HTML → rendered HTML → Firecrawl →
Scrapfly (fallback only, where permitted). Never defeat CAPTCHAs or explicit access controls; honour robots,
terms, rate limits. Cache and deduplicate aggressively, retain history, keep connectors independently testable,
never discard raw evidence, idempotent processing, UTC everywhere. Build a connector framework, not one crawler.

**Sensors and tiers.** Every endpoint is a sensor (type, tier, interval, importance weight). Tiers S (15–60 s),
A (1–5 min), B (5–30 min), C (30 min–6 h), D (6–24 h). Adaptive polling from change frequency/recency, patterns,
importance history, error rate and cache behaviour. Conditional requests (ETag / Last-Modified) preferred.

**Snapshots, canonicalization, diffs.** Immutable snapshots with raw/canonical/semantic hashes, blobs in object
storage (content-addressed, compressed). Strip render noise before diffing. Diff engines: text, DOM, JSON,
table, sitemap, PDF. Staged semantic detection: raw diff → heuristics → structured extraction → semantic
classification → LLM interpretation → event candidate.

**Events.** Canonical schema (ids, sensor/entities, detected/observed timestamps, type, title, summary,
importance/confidence/novelty, snapshots, diff, categories, silent_change, processing version). Open taxonomy
(announcement, product_launch, model_release, pricing_change, policy/terms/API changes, security_advisory,
vulnerability, breach, incident, outage, recall, drug_approval, filings, earnings, leadership, acquisition,
funding, layoffs, new_region, DNS/certificate changes, releases, standards, government/economic releases,
page_created/removed, unknown…). Importance 0–100 = 25 % severity + 20 % source + 15 % entity + 15 % novelty +
10 % magnitude + 5 % confirmation + 5 % user impact + 5 % unusualness (components stored). Confidence is
separate. Novelty against recent events (no 50 events for one syndicated press release). Clustering by entity
overlap, similarity, time proximity. **Silent change** = important change without announcement (flagship).

**Entities & graph.** Everything resolves to entities (organization, product, AI_model, API, drug, person,
standard, vulnerability…), aliases, relations (owns, released, approved, acquired…), PostgreSQL first.
Timelines per entity and per URL ("git history for the public Web"), compare any two versions. Latency fields
(published / first seen / detected / processed / published to feed) — compete on detecting first.

**Real-time & product.** Redis Streams → WebSocket gateway with channels (global, breaking, categories,
entity, source, watchlist). Frontend: the live feed is the hero; nav Live · Breaking · Explore · Sources ·
Entities · Timelines · Silent Changes · Watchlists · Alerts · API; event detail with importance/confidence,
evidence, semantic + raw diffs, related events, cluster, timeline, reliability; Explore, Trending, activity
anomaly per source, early signals (never speculation as fact), watchlists, alerts, search, dark + light,
performance targets, stable canonical URLs, machine-readable (API, RSS, JSON-LD, sitemaps).

**Platform.** TypeScript/Node/Fastify, PostgreSQL, Redis, object store, Playwright later; Python micro-services
where useful; Next.js frontend. Deployed on the private Apple Silicon cluster as separable processes, exposed
through ngrok at https://www.websensor.io (apex redirects to www). Never expose datastores publicly. Strict
security: rate limiting, validation, SSRF protection (private ranges, metadata endpoints, cluster hosts blocked),
redirect validation, size/time limits. Observability: Prometheus metrics, structured logs, request ids.

**Registry.** 200 launch organizations across AI, cloud, developer platforms, cybersecurity, consumer tech,
finance/regulators, government/statistics, healthcare/pharma, science/space, automotive, commerce/payments,
internet platforms/standards. Seed format in `config/sources.yaml`; discovery validates every candidate.
Expected 1 000–4 000 sensors at phase 1, then 10 k, 100 k, millions of monitored endpoints.

**MVP definition.** Register sources, discover endpoints, schedule, fetch efficiently, preserve snapshots,
detect raw changes, eliminate noise, classify, create events, resolve entities, compute importance, cluster,
publish over WebSocket, live feed, event detail, raw + semantic diffs, source history, watchlists, connector
health, continuous operation on the cluster. No fake connectors, no placeholder production data.

**Build order.** Foundation → structured sources → intelligence → product → real-time → advanced acquisition.
**Tagline.** *WebSensor — Detect What Changed. Know Why It Matters.*