# WebSensor architecture (v0.1, 2026-09-08)

## Roles (one node today, separable by design)
| Role | Process | Notes |
|---|---|---|
| gateway + API + WebSocket | `apps/api` (`websensor-api`, :8260) | Fastify; reverse-proxies the site; Redis pub/sub fan-out |
| frontend | `apps/web` (`websensor-web`, :8261 loopback) | Next.js 16, server components call the API over loopback |
| scheduler + workers | `apps/engine` (`websensor-engine`, metrics :8262) | several engine processes can run concurrently (`FOR UPDATE SKIP LOCKED`) |
| storage | PostgreSQL 17, Redis, content-addressed blob store | Postgres = metadata/events; blobs = raw bodies, canonical text, diffs |

## Data model (PostgreSQL)
`sources` → `sensors` → `sensor_runs`, `snapshots` → `changes` → `events` (+ `interpretations` versions,
`event_entities`, `event_clusters`), `entities` (+ `entity_aliases`, `entity_relations`, `source_entities`),
`urls` + `url_history`, `watchlists`/`watchlist_items`, `alerts`/`notifications`, `connector_health`,
`discovery_candidates`, `metrics_daily`, `llm_usage`. Full-text search: generated `tsvector` columns on
events and entities (GIN). Migrations: `packages/db/migrations/*.sql`.

## Pipeline (per sensor run)
1. **Fetch** through the connector (conditional GET with ETag/Last-Modified; 304 = cheap check). SSRF policy on
   every hop. Anti-bot 403/429/503 → Scrapfly fallback only if the source allows it and budget remains.
2. **Normalize** to a comparable representation: `text` (canonical HTML/plain text), `json` (sorted keys,
   ignored volatile paths) or `list` (keyed items: feed entries, sitemap URLs, incidents, releases, records).
3. **Compare** canonical hashes; unchanged → done. Else store an immutable **snapshot** (raw + canonical blob).
4. **Diff** against the previous canonical blob: text (paired modifications), json (path ops), list
   (added/removed/modified with connector-specific guards: feed window, partial sitemaps).
5. **Heuristics** (`evaluateChange`): noise ratio, extracted facts (prices, versions, percents, dates), type
   rules + sensor/URL priors → `signal`, `eventType`, `magnitude`. A `changes` row is always written.
6. If `signal ≥ WS_MEANINGFUL_SIGNAL` and extraction is trustworthy: **event candidate** — entity resolution
   (aliases), novelty vs the 72 h window (shingle Jaccard; near-duplicates suppressed), preliminary importance,
   optional **Claude interpretation** (strict JSON: type, title, summary, why it matters, observed/inferred,
   severity, confidence, announced?), final importance (8 stored components) + confidence, silent-change flag,
   clustering (shared entity/source + textual similarity within 6 h), `events` row, Redis stream + pub/sub.
7. **Schedule** the next check adaptively (tier bounds, recency of change, change frequency, 304s, errors).

## Latency fields
`published_at` (claimed by the source) · `observed_from` (previous snapshot) · `detected_at` (fetch) ·
`processed_at` · `published_to_feed_at`; `detection_latency_ms` (only when published < 7 days before detection)
and `processing_latency_ms`.

## Real-time
Engine → Redis stream `ws:events` (durable, MAXLEN ~20k) + channel `ws:live`. Gateway subscribes once and routes
to clients by channel: `events:global`, `events:breaking` (≥ 80), `events:silent`, `events:<ai|cyber|finance|
health|government|science|products|infrastructure>`, `type:<event_type>`, `entity:<id>`, `source:<id>`,
`watchlist:<id>` (server-side matching of the watchlist items).

## Not in v0.1 (explicitly)
Browser rendering (Playwright), Firecrawl, PDF diffing, OpenSearch (Postgres FTS is used), MinIO (fs blob store
with the same interface), accounts/email delivery (watchlists and alerts are keyed by an anonymous owner token),
embeddings (shingle similarity is used for novelty/clustering), MCP server, customer webhooks.