spb/websensor

1# WebSensor architecture (v0.2, 2026-09-11)23## Roles (one node today, separable by design)4| Role | Process | Notes |5|---|---|---|6| gateway + API + WebSocket | `apps/api` (`websensor-api`, :8260) | Fastify; reverse-proxies the site; Redis pub/sub fan-out |7| frontend | `apps/web` (`websensor-web`, :8261 loopback) | Next.js 16, server components call the API over loopback |8| scheduler + workers | `apps/engine` (`websensor-engine`, metrics :8262) | several engine processes can run concurrently (`FOR UPDATE SKIP LOCKED`) |9| storage | PostgreSQL 17, Redis, content-addressed blob store | Postgres = metadata/events; blobs = raw bodies, canonical text, diffs |1011## Data model (PostgreSQL)12`sources` → `sensors` → `sensor_runs`, `snapshots` → `changes` → `events` (+ `interpretations` versions,13`event_entities`, `event_clusters`), `entities` (+ `entity_aliases`, `entity_relations`, `source_entities`),14`urls` + `url_history`, `watchlists`/`watchlist_items`, `alerts`/`notifications`, `connector_health`,15`discovery_candidates`, `metrics_daily`, `llm_usage`. Full-text search: generated `tsvector` columns on16events and entities (GIN). Migrations: `packages/db/migrations/*.sql`.1718## Pipeline (per sensor run)191. **Fetch** through the connector (conditional GET with ETag/Last-Modified; 304 = cheap check). SSRF policy on20   every hop. Anti-bot 403/429/503 → Scrapfly fallback only if the source allows it and budget remains.212. **Normalize** to a comparable representation: `text` (canonical HTML/plain text), `json` (sorted keys,22   ignored volatile paths) or `list` (keyed items: feed entries, sitemap URLs, incidents, releases, records).233. **Compare** canonical hashes; unchanged → done. Else store an immutable **snapshot** (raw + canonical blob).244. **Diff** against the previous canonical blob: text (paired modifications), json (path ops), list25   (added/removed/modified with connector-specific guards: feed window, partial sitemaps).265. **Heuristics + semantic class** (`evaluateChange`, `classifyChange`): noise ratio, extracted facts, type rules27   + sensor/URL/record-shape priors → `signal`, `eventType`, `magnitude`; the semantic stage classifies the change28   (cosmetic / navigation / timestamp / advertisement / boilerplate = noise; meaningful / pricing / policy /29   product / personnel = signal) and extracts **field-level changes** (label, before → after, % delta). A `changes`30   row is always written (`change_class`, `field_changes`); noise classes never become events.316. If `signal ≥ WS_MEANINGFUL_SIGNAL`, extraction is trustworthy and the class is not noise: **event candidate** —32   fingerprint idempotency (sensor + before/after canonical hashes), entity resolution (aliases; ordinary-word33   aliases only as exact acronyms), novelty vs the 72 h window (near-duplicates suppressed; redundant third-party34   reports suppressed), source activity anomaly, preliminary importance (routine firehose batches damped),35   optional **Claude interpretation** (strict JSON), final importance (8 components) + confidence + **impact**,36   silent-change bar (first-party · silent-eligible type · no announcement · importance ≥ 45), clustering (shared37   entity/source + similarity within 6 h; cross-source same-entity stories within 2 h) with **propagation timeline**,38   first-party/external counts, velocity, **lead time** and **breaking state**, **WebSensor Signal Score** with39   `score_reasons`, `entity_daily`/`source_daily` counters, alert evaluation (web + signed webhooks), Redis stream40   (`sid`) + pub/sub.417. **Schedule** the next check adaptively (tier bounds, recency, change frequency, 304s, errors); the scheduler42   claims by priority (0 critical … 3 low), applies a per-domain circuit breaker and publishes a heartbeat.438. **Retention** (every 30 min): raw bodies of snapshots without an event are dropped after 21 days (canonical +44   hashes stay, blobs are reference-counted); event snapshots are permanent.4546## Latency fields47`published_at` (claimed by the source) · `observed_from` (previous snapshot) · `detected_at` (fetch) ·48`processed_at` · `published_to_feed_at`; `detection_latency_ms` (only when published < 7 days before detection)49and `processing_latency_ms`.5051## Real-time52Engine → Redis stream `ws:events` (durable, MAXLEN ~20k) + channel `ws:live` (payload carries the stream id53`sid`). Gateway subscribes once and routes to clients by channel: `events:global`, `events:breaking` (signal ≥ 80),54`events:silent`, `events:first-party`, `events:<ai|cyber|finance|health|government|science|products|infrastructure|55news>`, `group:<security|reliability|product|commercial|corporate|government|science|transport|sports|web>`,56`country:<CC>`, `state:<breaking|developing>`, `type:<event_type>`, `entity:<id>`, `source:<id>`, `watchlist:<id>`57(server-side matching incl. url / event_type / country kinds). Clients send `{"since": sid}` after a reconnection58and the gateway replays up to 500 missed events from the stream (`replay_done` frame).5960## Read models (apps/api/src/intel.ts)61Breaking desk (clusters by state), Pulse (15-minute activity series, per-desk top signals, rising entities,62anomalies, by-group counts), Radar (weak signals: quiet bursts, silent clusters, documentation/repository bursts,63developing clusters, new coverage), entity insights (35-day heatmap, baseline, daily anomaly, velocity ratio,64rank), entity rankings (importance × activity × velocity × quality × confirmation), cluster propagation, country and65category desks. Aggregates are cached 5–120 s in process (`cache.ts`); the live feed is never cached.6667## Source Factory (v0.3)68Separate process (`factory-main.ts`). Tables: `factory_seeds` (organization + hints + funnel counters),69`discovery_candidates` (extended: connector/type/name/kind_class/config/tier/score_value/reason/shadow_sensor_id),70`factory_daily`. Sensors gain `status = 'SHADOW'` (pipeline stores snapshots and changes, never events; priority 3);71sources gain `origin` (seed | import | factory) and `sector`. Discovery (`packages/connectors/src/discovery-deep.ts`)72is budgeted per organization (90 requests) and polite (4 in flight per organization, 6 organizations in parallel).73Acceptance is evidence-based (`evaluateShadows`). Coverage (`apps/api/src/coverage.ts`) matches universe members to74sensor hosts by registrable domain (`coverageKey`) and is cached 5 min.7576## Not in v0.3 (explicitly)77Browser rendering (Playwright), Firecrawl, OpenSearch (Postgres FTS + search syntax is used), MinIO (fs blob store78with the same interface), accounts and email/Slack/Discord/Telegram/push delivery (watchlists, alerts, bookmarks,79saved views and custom monitors are keyed by an anonymous owner token; alert channels are web + signed webhooks),80embeddings (shingle similarity is used for novelty/clustering), MCP server, world map.81