# WebSensor — repository guide

WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing Web. It monitors official public sources, detects meaningful changes, classifies them semantically, links entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score, preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through a replayable WebSocket feed. Product brief: docs/PRODUCT-BRIEF.md; architecture: docs/ARCHITECTURE.md. This file is the working guide for the code (v0.2, 2026-09-11).

# Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)

• packages/core — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups, silent-eligible types, change classes, cluster states, countries), ids, SSRF policy (assertUrlAllowed, safeLookup), hashing, canonical extraction, diff engines (text / json / keyed list), stage-1 heuristics (evaluateChange, describeChange, humanizeUrl), semantic diff (semantic.ts: classifyChange → cosmetic / navigation / timestamp / advertisement / boilerplate vs meaningful / pricing / policy / product / personnel; extractFieldChanges → field-level before → after pairs with % deltas), scoring (scoring.ts: importance components, confidence, impact, velocity, WebSensor Signal Score with explainable reasons, breakingState, entity rank, daily anomaly), adaptive schedule, search syntax (search.ts: entity: type: after:7d silent:true importance:>70 country:CA …), registry seed schema (registry-schema.ts, shared by engine, validator and the admin import endpoint). @websensor/core/client is the browser-safe subset.
• packages/db — plain SQL migrations (migrations/*.sql, applied by migrate() with an advisory lock) + Drizzle schema. 0005_intelligence.sql added provenance (first_party / country / language / kind / owner_token on sources), sensor lifecycle (status, priority, validated_at), semantic class + field changes on changes, signal/velocity/impact/anomaly scores, fingerprints, score reasons on events, cluster propagation columns, entity_daily, source_daily, bookmarks, saved_views, alert channel config, notification delivery. 0006_backfill_states.sql backfills them. textArray() must be used for = any(...) / && with JS arrays.
• packages/store — content-addressed blob store (sha256/ab/cd/<hash>.zst). Raw bodies, canonical forms and full diffs live there, never in Postgres.
• packages/connectors — connector SDK (fetch/normalize), safe fetcher (conditional GET, per-hop SSRF checks, HTTP/2→1.1 pin, browser-UA retry, UTF-16 BOM transcoding, postJson for signed webhooks), 16 connector families (http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap) + discovery + scrapfly fallback. jsonlist.urlTemplate supports {key} (path separators kept) and {field.path}.
• apps/engine — scheduler (priority-aware FOR UPDATE SKIP LOCKED claims, global + per-host concurrency, domain circuit breaker, heartbeat to Redis ws:engine:status), pipeline (fetch → normalize → snapshot → diff → heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias rules, novelty, optional Claude interpretation, impact / anomaly / signal scores, silent-change bar, clustering with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook delivery, Redis stream + pub/sub), registry sync (config/sources.yaml + config/sources.d/), discovery, connector health rollups, retention (retention.ts), Prometheus metrics :8262. CLI: src/cli.ts (sync, discover, probe, run-once, run-due, relink-entities [days], prune-blobs, refresh-clusters, llm-test); src/validate.ts (registry validator).
• apps/api — Fastify gateway (:8260): REST /api/v1/* (routes.ts public, routes-user.ts owner-scoped: watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, routes-admin.ts behind WS_ADMIN_TOKEN: ops, sensor & source actions, connector test, bulk import), intelligence read-models (intel.ts: breaking desk, pulse, radar, entity insights, rankings, cluster detail, country & category desks), TTL cache (cache.ts), WebSocket /api/v1/live protocol 2 (live.ts: sid on every frame, {"since"} replay from the durable stream, channels events:* group:* country:* state:* type:* entity:* source:* watchlist:*), /api/v1/feed.rss, /api/health|ready|metrics, apex→www redirect, reverse proxy to Next.
• apps/web — Next.js 16 (:8261 loopback). Design system in components/ui.tsx (Panel, Badge, Score, Tabs, Sparkline, Heatmap, Skeleton…), prefs.tsx (density compact/normal/comfortable, pause), event-drawer.tsx (intelligence panel), command-palette.tsx (⌘K), live-strip.tsx, live-feed.tsx v2 (URL-synced filters, pause, "↑ N new events", replay), field-changes.tsx. Pages: / /live /breaking /silent /pulse /radar /explore /entity/[id] (/company redirects) /cluster/[slug] /source/[id] /sensor/[id] /category/[c] /country/[slug] /event/[slug] (+ OG image) /bookmarks /watchlists /alerts /monitors /ops /health /api.
• config/sources.yaml (founding registry) + config/sources.d/*.yaml fragments merged in file-name order (extend: true adds to an earlier source). Fragments 10–35 = site classes; 40–46 (2026-09-11) = depth: AI frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport / telecom / sports / news. Seeds may carry country: (ISO-2, EU, INT), language: and first_party: false (media). Guide: docs/registry/AUTHORING.md. Validator: node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment> [--all] [--json report.json] — nothing enters the registry without OK.

# Rules

• Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom monitors — goes through assertUrlAllowed(). Private ranges, metadata endpoints, .maclustr.io/.ts.net, single-label hosts and NAT64-embedded private IPv4 are blocked.
• Raw evidence is immutable: snapshots and diffs are never rewritten. Retention may drop the RAW body of snapshots that are not referenced by any event (canonical form and hashes stay); event snapshots are kept forever. Re-interpretation creates a new interpretations row.
• Noise never becomes an event: changes classified cosmetic / navigation / timestamp / advertisement / boilerplate are stored as changes only. Routine batches from firehose feeds are damped. Ingestion is idempotent (events.fingerprint = sensor + before/after canonical hashes).
• Silent change = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs, shutdown, feature removed, leadership, page removed, crawler policy…) · no matching announcement within 12 h · importance ≥ WS_SILENT_MIN_IMPORTANCE · not a noise class. Never asserted as "unannounced" otherwise.
• Never label inference as fact: evidence_label OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED; AI text is labelled analysis; score_reasons explain every signal score. First-party evidence outweighs media reports (sources.first_party, media seeds carry first_party: false).
• Entity aliases that are ordinary words (first, has, who, make…) only match as exact upper-case acronyms in text (AMBIGUOUS_ALIASES in apps/engine/src/entities.ts); run cli.ts relink-entities 7 after changing alias rules.
• LLM cost control: heuristics first; Claude only above WS_LLM_MIN_IMPORTANCE, daily budget, strict JSON output; llm: false on firehoses (news, arXiv, NVD, package streams). No key → the system still works.
• Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes page_removed after WS_DELETE_CONFIRMATIONS separated checks.
• Admin API is off unless WS_ADMIN_TOKEN is set; custom monitors are limited per owner and never appear in public feeds (sources.kind = 'custom').
• All timestamps UTC. Ids are prefixed (src_, sen_, snap_, chg_, evt_, clu_, ent_, wl_, alr_…).

# Dev

createdb websensor && cp .env.example .env
pnpm install
pnpm db:migrate                        # or let the engine migrate on start
npx tsx apps/engine/src/cli.ts sync    # registry → DB
npx tsx apps/engine/src/cli.ts run-due 50
pnpm dev:api · pnpm dev:engine · pnpm dev:web   # open the site through the gateway (:8260) so /api is same-origin
pnpm test · pnpm typecheck

Run CLI/engine from the repo root. A copy of production data for UI work: ssh M4M64b 'pg_dump websensor --data-only -t events -t …' | psql websensor then cli.ts sync.

# Deploy (MacLustr)

mld stage . websensor && mld deploy websensor --node M4M64b. Manifest M1M32:~/dispatch/apps/websensor.json (secrets incl. WS_ADMIN_TOKEN; gitignored copy deploy/websensor.mld.json). Processes: websensor-api (:8260, public via MacLustr Tunnel www.websensor.io on BHS64), websensor-web (:8261 loopback), websensor-engine (metrics :8262). Postgres 17 websensor + Redis local. Blob store ~/websensor-data/blobs. After a deploy that changes alias or scoring rules: cli.ts relink-entities 7 and cli.ts refresh-clusters on the node. See deploy/README.md.