spb/websensor

1# WebSensor.io — product brief (condensed from the founding CLAUDE.md, 2026-09-08)23**Mission.** A production-grade real-time web intelligence platform: *a global sensor network for the changing4Web*. Not a generic crawler. Sources → sensors/connectors → fetch → change detection → AI interpretation →5entity resolution → event clustering → importance scoring → historical event store → real-time WebSocket feed.6Feel: Bloomberg Terminal × GitHub activity × Cloudflare Radar × security monitoring × an Internet-wide changelog.78**Philosophy.** Never just "HTML changed". Answer: what / when / where changed, is it meaningful, which entity,9how important, part of a larger event, publicly announced or silent, has it happened before. Preserve both the10RAW OBSERVATION and the INTERPRETED EVENT.1112**Engineering principles.** Read docs and test integrations; prefer webhooks → WebSocket → SSE → official API →13RSS/Atom → structured feeds → JSON/XML → sitemaps → conditional GET → static HTML → rendered HTML → Firecrawl →14Scrapfly (fallback only, where permitted). Never defeat CAPTCHAs or explicit access controls; honour robots,15terms, rate limits. Cache and deduplicate aggressively, retain history, keep connectors independently testable,16never discard raw evidence, idempotent processing, UTC everywhere. Build a connector framework, not one crawler.1718**Sensors and tiers.** Every endpoint is a sensor (type, tier, interval, importance weight). Tiers S (15–60 s),19A (1–5 min), B (5–30 min), C (30 min–6 h), D (6–24 h). Adaptive polling from change frequency/recency, patterns,20importance history, error rate and cache behaviour. Conditional requests (ETag / Last-Modified) preferred.2122**Snapshots, canonicalization, diffs.** Immutable snapshots with raw/canonical/semantic hashes, blobs in object23storage (content-addressed, compressed). Strip render noise before diffing. Diff engines: text, DOM, JSON,24table, sitemap, PDF. Staged semantic detection: raw diff → heuristics → structured extraction → semantic25classification → LLM interpretation → event candidate.2627**Events.** Canonical schema (ids, sensor/entities, detected/observed timestamps, type, title, summary,28importance/confidence/novelty, snapshots, diff, categories, silent_change, processing version). Open taxonomy29(announcement, product_launch, model_release, pricing_change, policy/terms/API changes, security_advisory,30vulnerability, breach, incident, outage, recall, drug_approval, filings, earnings, leadership, acquisition,31funding, layoffs, new_region, DNS/certificate changes, releases, standards, government/economic releases,32page_created/removed, unknown…). Importance 0–100 = 25 % severity + 20 % source + 15 % entity + 15 % novelty +3310 % magnitude + 5 % confirmation + 5 % user impact + 5 % unusualness (components stored). Confidence is34separate. Novelty against recent events (no 50 events for one syndicated press release). Clustering by entity35overlap, similarity, time proximity. **Silent change** = important change without announcement (flagship).3637**Entities & graph.** Everything resolves to entities (organization, product, AI_model, API, drug, person,38standard, vulnerability…), aliases, relations (owns, released, approved, acquired…), PostgreSQL first.39Timelines per entity and per URL ("git history for the public Web"), compare any two versions. Latency fields40(published / first seen / detected / processed / published to feed) — compete on detecting first.4142**Real-time & product.** Redis Streams → WebSocket gateway with channels (global, breaking, categories,43entity, source, watchlist). Frontend: the live feed is the hero; nav Live · Breaking · Explore · Sources ·44Entities · Timelines · Silent Changes · Watchlists · Alerts · API; event detail with importance/confidence,45evidence, semantic + raw diffs, related events, cluster, timeline, reliability; Explore, Trending, activity46anomaly per source, early signals (never speculation as fact), watchlists, alerts, search, dark + light,47performance targets, stable canonical URLs, machine-readable (API, RSS, JSON-LD, sitemaps).4849**Platform.** TypeScript/Node/Fastify, PostgreSQL, Redis, object store, Playwright later; Python micro-services50where useful; Next.js frontend. Deployed on the private Apple Silicon cluster as separable processes, exposed51through ngrok at https://www.websensor.io (apex redirects to www). Never expose datastores publicly. Strict52security: rate limiting, validation, SSRF protection (private ranges, metadata endpoints, cluster hosts blocked),53redirect validation, size/time limits. Observability: Prometheus metrics, structured logs, request ids.5455**Registry.** 200 launch organizations across AI, cloud, developer platforms, cybersecurity, consumer tech,56finance/regulators, government/statistics, healthcare/pharma, science/space, automotive, commerce/payments,57internet platforms/standards. Seed format in `config/sources.yaml`; discovery validates every candidate.58Expected 1 000–4 000 sensors at phase 1, then 10 k, 100 k, millions of monitored endpoints.5960**MVP definition.** Register sources, discover endpoints, schedule, fetch efficiently, preserve snapshots,61detect raw changes, eliminate noise, classify, create events, resolve entities, compute importance, cluster,62publish over WebSocket, live feed, event detail, raw + semantic diffs, source history, watchlists, connector63health, continuous operation on the cluster. No fake connectors, no placeholder production data.6465**Build order.** Foundation → structured sources → intelligence → product → real-time → advanced acquisition.66**Tagline.** *WebSensor — Detect What Changed. Know Why It Matters.*67