# WebSensor architecture (v0.2, 2026-09-11)

# Roles (one node today, separable by design)

# Data model (PostgreSQL)

sources → sensors → sensor_runs, snapshots → changes → events (+ interpretations versions, event_entities, event_clusters), entities (+ entity_aliases, entity_relations, source_entities), urls + url_history, watchlists/watchlist_items, alerts/notifications, connector_health, discovery_candidates, metrics_daily, llm_usage. Full-text search: generated tsvector columns on events and entities (GIN). Migrations: packages/db/migrations/*.sql.

# Pipeline (per sensor run)

1. Fetch through the connector (conditional GET with ETag/Last-Modified; 304 = cheap check). SSRF policy on every hop. Anti-bot 403/429/503 → Scrapfly fallback only if the source allows it and budget remains.
2. Normalize to a comparable representation: text (canonical HTML/plain text), json (sorted keys, ignored volatile paths) or list (keyed items: feed entries, sitemap URLs, incidents, releases, records).
3. Compare canonical hashes; unchanged → done. Else store an immutable snapshot (raw + canonical blob).
4. Diff against the previous canonical blob: text (paired modifications), json (path ops), list (added/removed/modified with connector-specific guards: feed window, partial sitemaps).
5. Heuristics + semantic class (evaluateChange, classifyChange): noise ratio, extracted facts, type rules
   • sensor/URL/record-shape priors → signal, eventType, magnitude; the semantic stage classifies the change (cosmetic / navigation / timestamp / advertisement / boilerplate = noise; meaningful / pricing / policy / product / personnel = signal) and extracts field-level changes (label, before → after, % delta). A changes row is always written (change_class, field_changes); noise classes never become events.
6. If signal ≥ WS_MEANINGFUL_SIGNAL, extraction is trustworthy and the class is not noise: event candidate — fingerprint idempotency (sensor + before/after canonical hashes), entity resolution (aliases; ordinary-word aliases only as exact acronyms), novelty vs the 72 h window (near-duplicates suppressed; redundant third-party reports suppressed), source activity anomaly, preliminary importance (routine firehose batches damped), optional Claude interpretation (strict JSON), final importance (8 components) + confidence + impact, silent-change bar (first-party · silent-eligible type · no announcement · importance ≥ 45), clustering (shared entity/source + similarity within 6 h; cross-source same-entity stories within 2 h) with propagation timeline, first-party/external counts, velocity, lead time and breaking state, WebSensor Signal Score with score_reasons, entity_daily/source_daily counters, alert evaluation (web + signed webhooks), Redis stream (sid) + pub/sub.
7. Schedule the next check adaptively (tier bounds, recency, change frequency, 304s, errors); the scheduler claims by priority (0 critical … 3 low), applies a per-domain circuit breaker and publishes a heartbeat.
8. Retention (every 30 min): raw bodies of snapshots without an event are dropped after 21 days (canonical + hashes stay, blobs are reference-counted); event snapshots are permanent.

# Latency fields

published_at (claimed by the source) · observed_from (previous snapshot) · detected_at (fetch) · processed_at · published_to_feed_at; detection_latency_ms (only when published < 7 days before detection) and processing_latency_ms.

# Real-time

Engine → Redis stream ws:events (durable, MAXLEN ~20k) + channel ws:live (payload carries the stream id sid). Gateway subscribes once and routes to clients by channel: events:global, events:breaking (signal ≥ 80), events:silent, events:first-party, events:<ai|cyber|finance|health|government|science|products|infrastructure| news>, group:<security|reliability|product|commercial|corporate|government|science|transport|sports|web>, country:<CC>, state:<breaking|developing>, type:<event_type>, entity:<id>, source:<id>, watchlist:<id> (server-side matching incl. url / event_type / country kinds). Clients send {"since": sid} after a reconnection and the gateway replays up to 500 missed events from the stream (replay_done frame).

# Read models (apps/api/src/intel.ts)

Breaking desk (clusters by state), Pulse (15-minute activity series, per-desk top signals, rising entities, anomalies, by-group counts), Radar (weak signals: quiet bursts, silent clusters, documentation/repository bursts, developing clusters, new coverage), entity insights (35-day heatmap, baseline, daily anomaly, velocity ratio, rank), entity rankings (importance × activity × velocity × quality × confirmation), cluster propagation, country and category desks. Aggregates are cached 5–120 s in process (cache.ts); the live feed is never cached.

# Source Factory (v0.3)

Separate process (factory-main.ts). Tables: factory_seeds (organization + hints + funnel counters), discovery_candidates (extended: connector/type/name/kind_class/config/tier/score_value/reason/shadow_sensor_id), factory_daily. Sensors gain status = 'SHADOW' (pipeline stores snapshots and changes, never events; priority 3); sources gain origin (seed | import | factory) and sector. Discovery (packages/connectors/src/discovery-deep.ts) is budgeted per organization (90 requests) and polite (4 in flight per organization, 6 organizations in parallel). Acceptance is evidence-based (evaluateShadows). Coverage (apps/api/src/coverage.ts) matches universe members to sensor hosts by registrable domain (coverageKey) and is cached 5 min.

# Not in v0.3 (explicitly)

Browser rendering (Playwright), Firecrawl, OpenSearch (Postgres FTS + search syntax is used), MinIO (fs blob store with the same interface), accounts and email/Slack/Discord/Telegram/push delivery (watchlists, alerts, bookmarks, saved views and custom monitors are keyed by an anonymous owner token; alert channels are web + signed webhooks), embeddings (shingle similarity is used for novelty/clustering), MCP server, world map.