# WebSensor architecture (v0.1, 2026-09-08)

# Roles (one node today, separable by design)

Role	Process	Notes
gateway + API + WebSocket	`apps/api` (`websensor-api`, :8260)	Fastify; reverse-proxies the site; Redis pub/sub fan-out
frontend	`apps/web` (`websensor-web`, :8261 loopback)	Next.js 16, server components call the API over loopback
scheduler + workers	`apps/engine` (`websensor-engine`, metrics :8262)	several engine processes can run concurrently (`FOR UPDATE SKIP LOCKED`)
storage	PostgreSQL 17, Redis, content-addressed blob store	Postgres = metadata/events; blobs = raw bodies, canonical text, diffs

# Data model (PostgreSQL)

sources → sensors → sensor_runs, snapshots → changes → events (+ interpretations versions, event_entities, event_clusters), entities (+ entity_aliases, entity_relations, source_entities), urls + url_history, watchlists/watchlist_items, alerts/notifications, connector_health, discovery_candidates, metrics_daily, llm_usage. Full-text search: generated tsvector columns on events and entities (GIN). Migrations: packages/db/migrations/*.sql.

# Pipeline (per sensor run)

Fetch through the connector (conditional GET with ETag/Last-Modified; 304 = cheap check). SSRF policy on every hop. Anti-bot 403/429/503 → Scrapfly fallback only if the source allows it and budget remains.
Normalize to a comparable representation: text (canonical HTML/plain text), json (sorted keys, ignored volatile paths) or list (keyed items: feed entries, sitemap URLs, incidents, releases, records).
Compare canonical hashes; unchanged → done. Else store an immutable snapshot (raw + canonical blob).
Diff against the previous canonical blob: text (paired modifications), json (path ops), list (added/removed/modified with connector-specific guards: feed window, partial sitemaps).
Heuristics (evaluateChange): noise ratio, extracted facts (prices, versions, percents, dates), type rules + sensor/URL priors → signal, eventType, magnitude. A changes row is always written.
If signal ≥ WS_MEANINGFUL_SIGNAL and extraction is trustworthy: event candidate — entity resolution (aliases), novelty vs the 72 h window (shingle Jaccard; near-duplicates suppressed), preliminary importance, optional Claude interpretation (strict JSON: type, title, summary, why it matters, observed/inferred, severity, confidence, announced?), final importance (8 stored components) + confidence, silent-change flag, clustering (shared entity/source + textual similarity within 6 h), events row, Redis stream + pub/sub.
Schedule the next check adaptively (tier bounds, recency of change, change frequency, 304s, errors).

# Latency fields

published_at (claimed by the source) · observed_from (previous snapshot) · detected_at (fetch) · processed_at · published_to_feed_at; detection_latency_ms (only when published < 7 days before detection) and processing_latency_ms.

# Real-time

Engine → Redis stream ws:events (durable, MAXLEN ~20k) + channel ws:live. Gateway subscribes once and routes to clients by channel: events:global, events:breaking (≥ 80), events:silent, events:<ai|cyber|finance| health|government|science|products|infrastructure>, type:<event_type>, entity:<id>, source:<id>, watchlist:<id> (server-side matching of the watchlist items).

# Not in v0.1 (explicitly)

Browser rendering (Playwright), Firecrawl, PDF diffing, OpenSearch (Postgres FTS is used), MinIO (fs blob store with the same interface), accounts/email delivery (watchlists and alerts are keyed by an anonymous owner token), embeddings (shingle similarity is used for novelty/clustering), MCP server, customer webhooks.

spb/websensor