WebSensor

The Web is changing. We are watching.
Real-time Internet change intelligence — www.websensor.io

 5 669 organizations · 10 969 validated sensors · 16 connector families · 94 event classes · immutable evidence · live WebSocket feed

WebSensor continuously watches the public web for meaningful changes, preserves the evidence, connects related signals into real-world events and surfaces important developments before they become obvious. It is not a website monitor: it answers what changed, where, when, exactly what, whether it was public or silent, how important and unusual it is, who is responsible, which other sources confirm it, whether it is part of a larger event, how fast the story is spreading and whether WebSensor was early.

Built by Simon-Pierre Boucher · contact@spboucher.ai · hosted on MacLustr, an Apple-silicon compute cluster in Québec.

# Table of contents

1. Brand
2. What WebSensor does
3. Architecture
4. The pipeline, step by step
5. Scoring model
6. Event taxonomy
7. Source registry
8. Connectors
9. Data model
10. Real-time protocol
11. REST API
12. Frontend
13. Repository layout
14. Development
15. Testing
16. Deployment and operations
17. Security and crawling etiquette
18. Roadmap
19. Credits

# Brand

The mark is a pulse in the shape of a W: the waveform is grey until the moment a change is detected, then turns signal green and ends in the sensor reading. It is drawn once as SVG and reused everywhere:

Colours: background #0a0e15, ink #c9d1e0, signal #22d3a5, line #2d394e. Type: Inter (UI), JetBrains Mono (numbers, identifiers, timestamps). The product aesthetic is Bloomberg Terminal × Linear × Vercel × Datadog × threat-intelligence desk: dense information, one-pixel borders, small badges, precise spacing, dark first.

# What WebSensor does

# Architecture

                         ┌──────────────────────── MacLustr node M4M64b ────────────────────────┐
 Internet ──TLS──▶ BHS64 │ websensor-api  :8260  Fastify gateway                                │
 (OVH, Caddy,     (wg1)  │   /api/v1/*   REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │
  MacLustr Tunnel)       │   /*          reverse proxy → websensor-web :8261 (Next.js 16)       │
                         │ websensor-engine        scheduler + pipeline, Prometheus :8262       │
                         │ websensor-factory       Source Factory: seeds → discovery → shadow    │
                         │ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │
                         └───────────────────────────────────────────────────────────────────────┘

Everything is TypeScript ESM in a pnpm workspace (Node ≥ 22.15). One node runs everything today; every role is separable (several engine processes can run concurrently thanks to FOR UPDATE SKIP LOCKED claims).

# The pipeline, step by step

Every sensor run goes through apps/engine/src/pipeline.ts:

1. Schedule — the scheduler claims due sensors by priority (0 critical … 3 low) with FOR UPDATE SKIP LOCKED, enforces global concurrency and per-host limits (higher for large public APIs such as github.com, data.sec.gov), applies a per-domain circuit breaker (5 consecutive failures → 10 min pause) and publishes a heartbeat to Redis (ws:engine:status).
2. Fetch — conditional GET (ETag / If-None-Match, Last-Modified / If-Modified-Since), manual redirects validated per hop by the SSRF policy, HTTP/2 → HTTP/1.1 pinning on NGHTTP2 errors, a second attempt with a browser identity on 403/resets, size and time limits, gzip and UTF-16 BOM handling. 304 = cheap check.
3. Normalize — the connector turns the observation into a comparable representation: text (canonical HTML/plain text with navigation, footers, cookie banners, timestamps, tokens and counters stripped), json (sorted keys, volatile paths ignored) or list (keyed items: feed entries, sitemap URLs, incidents, releases, filings, records).
4. Snapshot — if the canonical hash changed, raw body and canonical form are stored immutably and a snapshots row records hashes, HTTP metadata, extraction confidence and the source's own timestamp.
5. Diff — text (paired modifications), json (path operations) or list (added / removed / modified) with connector guards: feed items are "new" only if never seen and < 14 days old, status pages losing an item is a resolution, a > 50 % sitemap shrink is a partial response, a 404 becomes page_removed only after separated confirmations.
6. Heuristics + semantic class — evaluateChange proposes an event type, magnitude, facts (prices, percents, versions, dates) and a signal 0–1 from ~90 typed rules and record-shape priors (KEV records, CVE ids, 8-K items, ClinicalTrials statuses…). classifyChange labels the change cosmetic / navigation / timestamp / advertisement / boilerplate (noise) or meaningful / pricing / policy / product / personnel (signal) and extracts field-level changes. A changes row is always written; noise classes never become events.
7. Event candidate — fingerprint idempotency (sensor + before/after canonical hashes), entity resolution (aliases, longest first; ordinary-word aliases such as first or has only match as exact upper-case acronyms), novelty vs the 72 h window (near-duplicates suppressed, redundant third-party reports suppressed), source anomaly, preliminary importance (routine firehose batches damped), optional Claude interpretation above WS_LLM_MIN_IMPORTANCE (strict JSON: type, title, summary, why it matters, observed vs inferred, severity, confidence, announced?), final importance + confidence + impact, silent-change bar.
8. Cluster — attach to an open cluster (shared entity or source, textual similarity within 6 h; same-entity cross-source stories within 2 h) or open one; update propagation timeline, first-party/external counts, velocity, lead time and breaking state.
9. Signal score & publish — computeSignalScore with stored reasons; events row; entity and source daily counters; alert evaluation (web notifications, HMAC-signed webhooks); Redis stream ws:events (durable, with stream ids) + pub/sub ws:live.
10. Re-schedule — adaptive interval from tier bounds, recency of change, change frequency, 304 ratio and errors (tier S 15 s–5 min · A 1–30 min · B 5 min–2 h · C 30 min–6 h · D 6–24 h).

Background jobs: connector health rollups (1 min), cluster state ageing (5 min), retention (30 min: raw bodies of snapshots without an event are dropped after 21 days, canonical forms and hashes kept, event snapshots kept forever, blobs reference-counted), run log pruning, weekly discovery.

# Scoring model

Importance (0–100) = 25 % intrinsic severity of the event type · 20 % source authority (tier × weight) · 15 % entity importance · 15 % novelty · 10 % magnitude · 5 % cross-source confirmation · 5 % user impact · 5 % unusualness. All eight components are stored on the event.

Confidence (0–100) blends source authenticity, extraction confidence, diff clarity, structured vs free text, confirmations and LLM/heuristic agreement.

Impact (0–100) weights the event type's user impact, entity importance, magnitude and the largest numeric delta among extracted fields ($100 → $500 scores higher than $100 → $105).

Velocity (0–100) grows with signals per hour, unique sources and first-party signals inside a cluster.

WebSensor Signal Score (0–100) = 42 % importance · 14 % confidence · 14 % novelty · 12 % velocity · 12 % impact · 6 % anomaly, then adjusted with explainable reasons: + first-party evidence, + N independent confirmations, + silent change, + tier S source, - third-party report, - low confidence, - unconfirmed, - routine batch from a high-volume feed, and a hard cap of 20 for noise classes. The reason list is shown on every event ("Why this score").

Breaking state is not recency: breaking needs a strong signal, freshness (< 6 h) and either an independent confirmation, a tier-S first-party source or signal ≥ 90; developing = signals accumulating; confirmed = ≥ 3 sources within 24 h; watching otherwise; closed after 72 h.

Entity rank = importance × activity (24 h, 7 d) × average signal × confirmation ratio × unique sources × acceleration vs the 30-day baseline — never raw event counts.

# Event taxonomy

94 event classes grouped for filters and desks:

Each class carries an intrinsic severity and a usuallyAnnounced flag used by silent-change detection. The full list lives in packages/core/src/taxonomy.ts.

# Source registry

The registry is declarative YAML, merged in file-name order by apps/engine/src/seeds.ts:

• config/sources.yaml — the founding 271 organizations;
• config/sources.d/10–35 — one fragment per site class (open source, central banks & finance, energy / climate / weather, telecom & internet infrastructure, retail / consumer / travel, gaming & entertainment, universities & research, international organizations / NGOs / standards, sports, crypto, transport / aviation / space, consumer safety / food / agriculture, housing / labour / open data, enterprise SaaS, health systems & medtech, politics / elections / courts, EDGAR filings, package registries, web posture, OpenAPI, status JSON, documents & data);
• config/sources.d/40–46 — depth fragments (AI frontier, cloud infrastructure, cybersecurity, finance & markets, governments incl. Canadian provinces, science & health, transport / telecom / sports / world news);
• config/sources.d/47–55 — breadth fragments, wave 2 (2026-09-11): the open-source long tail (482 sensors), SaaS status pages and changelogs (923, incl. 208 new status sensors), 424 more EDGAR issuers, 184 US federal agencies via the Federal Register API, cities / regions / public bodies (356), world governments and regulators in 55 countries (535), the corporate pricing / legal / careers "silent change" surface of ~250 companies (702), sports clubs / entertainment / education (396), regional media / trade press / think tanks in 68 countries (638), and a small cross-fragment extend file. Every sensor was validated live before entering the registry; the fragment headers record what was probed and blocked (WAF, JS shells, broken TLS chains).

sources:
  - id: bank-of-canada                 # kebab-case, unique across all files
    name: Bank of Canada
    domain: bankofcanada.ca
    categories: [finance, central-bank]
    tier: S                            # S 15–60 s · A 1–5 min · B 5–30 min · C 30 min–6 h · D 6–24 h
    weight: 1.4
    country: CA                        # ISO 3166-1 alpha-2, EU, INT
    language: en
    products:
      - { name: Valet API, type: API }
    sensors:
      - { name: press releases, url: "https://www.bankofcanada.ca/…/feed/", type: RSS, connector: rss, tier: A }
      - { name: policy rate, url: "https://www.bankofcanada.ca/valet/observations/V39079/json?recent=3", type: REST_API, connector: jsonlist, tier: A, config: { … } }
  - id: cbc                            # media report ABOUT others → third-party evidence
    extend: true
    first_party: false

Rules that never bend:

• Nothing enters the registry without the validator's OK. apps/engine/src/validate.ts runs every curated sensor through its connector (fetch + normalize) and prints OK / WARN / FAIL; WARN (empty list, thin client-rendered page) and FAIL (403, 404, parse error) sensors are pruned. Sites blocked by bot management are documented in notes:, never faked.
• One source = one organization (the entity people search for). extend: true adds sensors, products, aliases, categories, country, language or first-party flag to a source declared in an earlier file.
• Media and aggregators carry first_party: false; the organization's own channels are first-party and weigh more. High-volume feeds (news wires, arXiv, NVD, package streams, sports) run with llm: false.
• Country and language are inferred from unambiguous TLDs when not declared; scheduling priority is derived from tier and category.

Registry sync (cli.ts sync) upserts sources, sensors, organization and product entities, aliases and owns relations; seed sensors removed from YAML are disabled, discovery-created ones are kept. The discovery engine probes robots.txt sitemaps, <link rel="alternate"> feeds, well-known feed paths and linked status pages, and promotes only candidates that fetch and parse. Sensors carry a lifecycle status (PENDING → VALIDATED → ACTIVE / DEGRADED / DISABLED). Bulk import (JSON or YAML) is available through the admin API with a dry-run mode. Authoring guide: docs/registry/AUTHORING.md.

# Connectors

All connectors implement fetch(endpoint) and normalize(endpoint, observation); the shared pipeline does compare, classify, score and persist (packages/connectors/src/types.ts).

Per-connector documentation: docs/connectors/*.md.

# Data model

PostgreSQL 17; migrations in packages/db/migrations (0001_init … 0006_backfill_states), all additive.

sources ─┬─ sensors ─┬─ sensor_runs
         │           ├─ snapshots ─┐
         │           └─ changes ───┼─ events ─┬─ event_entities ─ entities ─┬─ entity_aliases
         │                         │          ├─ interpretations (versioned) ├─ entity_relations
         │                         │          └─ event_clusters (timeline, states)
         ├─ source_entities         └─ urls / url_history
         ├─ source_daily            entity_daily · metrics_daily · llm_usage · connector_health · discovery_candidates
         └─ (kind = custom, owner_token)      watchlists · watchlist_items · alerts · notifications · bookmarks · saved_views

Key columns on events: event_type, title, summary, why_it_matters, importance + importance_components, confidence, novelty, signal_score, velocity_score, impact_score, anomaly_score, score_reasons, change_class, field_changes, silent_change, evidence_label, first_party, country, language, fingerprint (unique), cluster_id, published_at / observed_from / detected_at / processed_at / published_to_feed_at, detection_latency_ms, processing_latency_ms, a generated tsvector for full-text search. Raw evidence is immutable: snapshots and diffs are never rewritten; re-interpretation adds a row to interpretations.

# Real-time protocol

wss://www.websensor.io/api/v1/live (protocol 2). One Redis subscriber fans out to every client; the engine publishes each event once with its stream id.

client → {"subscribe":["events:breaking","entity:org_openai","country:CA"]}
client → {"since":"1789105849242-0"}          # after a reconnection: replay up to 500 missed events
client → {"ping":1}
server → {"type":"hello","protocol":2,"channels":[…]}
server → {"type":"event","sid":"1789105849242-0","channels":["events:global","events:ai"],"event":{…}}
server → {"type":"replay_done","since":"…","count":17}
server → {"type":"heartbeat","t":1789105849242}       # every 25 s

Channels: events:global, events:breaking, events:silent, events:first-party, events:<ai|cyber|finance|health|government|science|products|infrastructure|news>, group:<security|reliability|product|commercial|corporate|government|science|transport|sports|web>, country:<CC>, state:<breaking|developing>, type:<event_type>, entity:<id>, source:<id>, watchlist:<id> (server-side matching of entities, sources, keywords, categories, URLs, event types, countries). The browser client (apps/web/src/lib/use-live.ts) reconnects with jittered backoff, detects half-dead sockets with a 70 s watchdog and replays on reconnection; the feed de-duplicates by event id.

# REST API

Public, unauthenticated, 600 requests / minute / IP (x-ratelimit-* headers), consistent errors { "error": "…", "detail"?: "…" }, cursor pagination (nextCursor → cursor), all timestamps UTC. Documentation with examples: https://www.websensor.io/api.

# Frontend

Next.js 16 (App Router, React 19, Tailwind v4). Server components by default; client components only where interaction is needed.

Interaction model: click any event → intelligence drawer (right on desktop, full-screen sheet on mobile); ⌘K / Ctrl+K command palette (entities, sources, events, clusters, URLs, navigation, commands); three density modes (compact / normal / comfortable) persisted; pause live; ↑ N new events without scroll jumps; badges BREAKING · DEVELOPING · SILENT · FIRST PARTY · EXTERNAL · CONFIRMED · ANOMALOUS; skeleton loaders; real 404s; dark and light themes; audited at 375, 390, 430, 768, 1024, 1440 and 1920 px.

# Repository layout

apps/
  api/        Fastify gateway: routes.ts (public), routes-user.ts (owner), routes-admin.ts, intel.ts, live.ts, cache.ts
  engine/     scheduler.ts, pipeline.ts, cluster.ts, entities.ts, interpret.ts (Claude), alerts.ts, retention.ts,
              registry.ts, seeds.ts, discovery, metrics.ts, cli.ts, validate.ts
  web/        Next.js app: src/app (routes), src/components (ui.tsx design system, live-feed, event-drawer,
              command-palette, live-strip, brand.tsx…), src/lib (api client, format, feed-filters, use-live, owner)
packages/
  core/       taxonomy · scoring · semantic · heuristics · diff · canonical · search · ssrf · schedule · registry-schema
  db/         schema.ts + migrations/*.sql
  store/      content-addressed blob store
  connectors/ fetcher + 16 connector families + discovery + scrapfly
config/
  sources.yaml, sources.d/*.yaml     the registry
docs/
  ARCHITECTURE.md, PRODUCT-BRIEF.md, registry/AUTHORING.md, connectors/*.md, brand/
scripts/      registry generators (EDGAR, Federal Register, web posture, status probe) and pruning helpers
deploy/       mld manifest example and README
tests/        fixtures (RSS before/after, Atlassian summary, KEV, sitemap…)

# Development

createdb websensor && cp .env.example .env
pnpm install
pnpm db:migrate                                   # or let the engine migrate on start
npx tsx apps/engine/src/cli.ts sync               # registry → database (~6 s for 3 000 sources)
npx tsx apps/engine/src/cli.ts run-due 50         # run 50 due sensors through the pipeline
pnpm dev:api      # gateway :8260
pnpm dev:engine   # scheduler + pipeline
NEXT_PUBLIC_API_URL=http://localhost:8260 pnpm dev:web   # :8261 — open the site here in dev
pnpm test · pnpm typecheck · pnpm --filter @websensor/web build

Useful CLI commands (apps/engine/src/cli.ts): sync, discover [sourceId…], probe <domain>, run-once <sensorId>, run-due [n], relink-entities [days] (re-resolve mentioned entities after alias-rule changes), refresh-clusters, prune-blobs, llm-test. Registry validator: node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment.yaml> [--all] [--json report.json] [--concurrency 6] [--probe <domain>]; scripts/prune-fragment.py <fragment> <report.json> [--keep-empty] removes failed sensors.

Environment (see .env.example): DATABASE_URL, REDIS_URL, BLOB_STORE_DIR, WS_FETCH_CONCURRENCY, WS_PER_HOST_CONCURRENCY (+ WS_PER_HOST_OVERRIDES), WS_MEANINGFUL_SIGNAL, WS_SILENT_MIN_IMPORTANCE, WS_RETENTION_*, ANTHROPIC_API_KEY + WS_LLM_* (models, daily call budget, thresholds), SCRAPFLY_API_KEY, WS_ADMIN_TOKEN, WS_MONITORS_PER_OWNER, WS_USER_AGENT.

Dev tip: a copy of production data for UI work is ssh <node> 'pg_dump websensor --data-only -t events …' | psql websensor, then cli.ts sync.

# Testing

pnpm test runs Vitest across the workspace:

• packages/core — diff engines, canonicalization, heuristics, semantic classifier and field extraction (source-simulator fixtures: footer-year bump, relative-time bump, pricing change with deltas, leadership change, site redesign burst, JSON volatile paths, new feed items), new event classes (KEV → active exploitation, zero-day, supply chain, guidance, buyback, emergency notice, outbreak, sports), search syntax round-trip, impact / velocity / signal / breaking / anomaly scoring, title generation for URL-only and batch items;
• packages/connectors — RSS / Atom / JSON Feed, sitemap, Statuspage, KEV, package registries, EDGAR, OpenAPI, CSV, DNS/TLS/RDAP normalizers on fixtures;
• apps/engine — alert rule matching, country / language inference, priority tiers.

Type safety is enforced with pnpm typecheck (strict TypeScript across all packages); the web app is also linted with the React Compiler rules.

# Deployment and operations

Production runs on the MacLustr cluster, orchestrated by mld from the M1M32 gateway:

~/Desktop/cluster-skill/mld stage . websensor
~/Desktop/cluster-skill/mld deploy websensor --node M4M64b

Post-sync hooks (fail-fast): pnpm install --frozen-lockfile → createdb + migrations → registry sync → next build. Processes (PM2): websensor-engine, websensor-api (:8260, published as https://www.websensor.io through the MacLustr Tunnel — WireGuard to the OVH gateway BHS64 with Caddy TLS), websensor-web (:8261, loopback). Secrets live only in the manifest on the gateway. After a deploy that changes alias or scoring rules: cli.ts relink-entities 7 and cli.ts refresh-clusters on the node.

Observability: Prometheus metrics (/api/metrics, engine :8262/metrics: checks by outcome, bytes, changes by class, events by type, LLM calls and tokens, stage latencies, suppressed candidates, queue depth, inflight), /health (connectors, throughput, 304 ratio, failing domains, slowest sensors), /ops (engine heartbeat, queue, storage, LLM usage, recent errors, connector dry-run, import). Retention keeps the database and blob store bounded without ever deleting event evidence. Details: deploy/README.md, docs/ARCHITECTURE.md.

# Security and crawling etiquette

• Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom monitors — passes assertUrlAllowed(): private ranges, loopback, link-local, cloud metadata endpoints, .maclustr.io / .ts.net, single-label hosts and NAT64-embedded private IPv4 are blocked, and the HTTP dispatcher only connects to validated addresses (DNS-rebinding safe).
• Clear User-Agent (WebSensorBot/0.2 (+https://www.websensor.io/bot; contact@spboucher.ai)), conditional requests, per-host concurrency, adaptive intervals, backoff, timeouts and a per-domain circuit breaker. Official APIs, feeds and structured endpoints are preferred; WebSensor does not bypass authentication or anti-bot protections (Scrapfly is a budgeted fallback only where a source explicitly allows it).
• Webhooks: HTTPS only, public hosts only, X-WebSensor-Signature: sha256=<HMAC>; custom monitors: public URLs only, tested before activation, hard per-owner limit, never shown in public feeds.
• Admin API disabled unless WS_ADMIN_TOKEN is set; owner tools use an anonymous browser token; rate limiting on every /api/* route; strict input validation (zod); snapshots served raw get nosniff and a restrictive CSP.

# Source Factory and coverage (0.3, 2026-09-13)

WebSensor no longer grows by hand-written lists alone. config/coverage/*.yaml declares, sector by sector, the organizations that should be observed (listed companies by index, governments of every country by institutional role, AI, cloud, cybersecurity, markets, science, healthcare, energy, transport, telecom, logistics, open source, commerce), each with a verified domain, a country, an importance and optional hints (CIK, GitHub organization, Hugging Face author, status page, explicit URLs). Those universes are the denominator of the Global Observation Coverage Score (breadth = importance-weighted share of members with an active sensor, depth = sensors per organization, score = 100·(0.7·breadth + 0.3·depth)) and the input of the Source Factory:

seed (organization + hints)
  → deep discovery   robots · sitemaps · feeds (+ official sub-domains) · navigation-classified pages
                     (news · press · IR · changelog · security · pricing · legal · careers · leadership · docs)
                     · status-page providers · GitHub org repositories · EDGAR · Hugging Face · OpenAPI · posture
  → validation       every candidate fetched and parsed by its connector; wildcard DNS and anti-bot detected;
                     duplicates removed by content (shared feed items, identical page text)
  → scoring          evidence × page class × organization importance × first-party confidence × change
                     frequency − fetch cost, with explainable reasons; caps per organization and per class
  → shadow sensors   polled like any sensor, evidence stored, nothing published
  → evaluation       reject (errors, pure noise, duplicate of an active sensor) · accept (≥ 5 checks, 24 h) · defer
  → production       accepted sensors publish; `cli.ts factory export` graduates them into a registry fragment

cli.ts expand <domain> shows what the Factory would find for one organization; /ops has the funnel, per-sector progress and a candidate review table; /coverage and /coverage/<sector> show the score and every member's state (observed · shadow · queued · not seeded).

# Roadmap

World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel abstraction is in place); accounts and API keys; embeddings-based similarity; MinIO/S3 blob driver; browser rendering for the few high-value client-rendered pages; MCP server; more depth fragments.

# Credits

WebSensor is designed and built by Simon-Pierre Boucher — contact@spboucher.ai. It runs on MacLustr, an Apple-silicon compute cluster in Québec. Source of truth: websensor.git on spbgit (git.spboucher.ai). AI-generated summaries are labelled as analysis and never replace the original evidence.