# WebSensor — repository guide

WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing Web. It monitors official public sources, detects meaningful changes, classifies them, links entities, scores importance, preserves evidence and publishes events through a WebSocket feed. The product brief that drove the design is docs/PRODUCT-BRIEF.md; this file is the working guide for the code.

# Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)

• packages/core — taxonomy (event types + intrinsic severity, tiers, categories), ids, SSRF policy (assertUrlAllowed, safeLookup), hashing (sha256, simhash, shingles/Jaccard), canonical extraction (canonicalizeHtml: strips scripts/nav/footer/cookie chrome, timestamps, tokens, counters), diff engines (text / json / keyed list), stage-1 heuristics (evaluateChange, describeChange), scoring (importance components, confidence, novelty helpers, trending, activity anomaly), adaptive schedule. @websensor/core/client is the browser-safe subset (no node: imports) — client components must import it.
• packages/db — plain SQL migrations (migrations/*.sql, applied by migrate() with an advisory lock) + Drizzle schema for typed access. textArray() must be used for = any(...) / && with JS arrays (Drizzle spreads arrays into parameter lists).
• packages/store — content-addressed blob store (sha256/ab/cd/<hash>.zst, zstd via node:zlib). Raw bodies, canonical representations and full diffs live there, never in Postgres. Interface ready for S3/MinIO.
• packages/connectors — connector SDK (WebSensorConnector.fetch/normalize), safe fetcher (conditional GET, manual redirects validated per hop, HTTP/2→1.1 pin on NGHTTP2 errors, browser-UA second attempt on 403/resets, size + time limits), connectors: http (HTML/JSON/text/HEAD), rss (RSS/Atom/RDF/JSON Feed), sitemap (index + news), statuspage (Atlassian v2 summary), github (releases/tags/commits Atom, advisories REST), jsonlist (keyed records from any JSON API; {now-2h} placeholders), discovery (robots sitemaps, <link rel=alternate>, well-known feed paths, linked status pages — every candidate is fetched and parsed), scrapfly (fallback, budgeted, only when fallback.scrapfly is set on the source).
• apps/engine — scheduler (FOR UPDATE SKIP LOCKED claims, global + per-host concurrency), pipeline (fetch → normalize → snapshot → diff → heuristics → change → event: entities, novelty, LLM interpretation, clustering, importance/confidence, silent-change detection, publish to Redis), registry sync from config/sources.yaml, discovery, connector health rollups, Prometheus metrics on :8262. CLI: src/cli.ts (sync, discover, probe <domain>, run-once <sensor>, run-due [n]).
• apps/api — Fastify gateway (:8260): REST /api/v1/*, WebSocket /api/v1/live (Redis pub/sub fan-out, channels events:*, entity:*, source:*, watchlist:*), /api/v1/feed.rss, /api/health /api/ready /api/metrics, apex→www redirect, and a reverse proxy to the Next.js app for everything else.
• apps/web — Next.js 16 frontend (:8261, loopback). Pages: live, breaking, explore, sources, entities, timelines, silent changes, watchlists, alerts, event detail with diff viewer, health, API docs.
• config/sources.yaml — the 200-organization registry (sources, curated sensors, product entities, discovery flags, fallbacks). docs/connectors/*.md document each connector family.

# Rules

• Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets — goes through assertUrlAllowed(). Private ranges, metadata endpoints, .maclustr.io/.ts.net and single-label hosts are blocked; the dispatcher's DNS lookup only returns approved addresses.
• Raw evidence is immutable: snapshots and diffs are never rewritten. Re-interpretation creates a new row in interpretations (versioned); events.interpretation holds the latest.
• LLM cost control: heuristics first; Claude is called only for candidates above WS_LLM_MIN_IMPORTANCE, with a daily call budget; the deep model only above WS_LLM_DEEP_MIN_IMPORTANCE. Output is strict JSON (output_config.format), never free text. No key → heuristics only, the system still works.
• Feeds: items are "new" only if their key was never seen (sensor state.seenKeys) and they are not older than 14 days; items scrolling out of the window are never "removed". Sitemaps/statuspages: a >50 % shrink is treated as a partial response, not mass deletion. A 404 becomes page_removed only after WS_DELETE_CONFIRMATIONS checks separated by WS_DELETE_SEPARATION_MIN.
• Never label inference as fact: events carry evidence_label (OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED) and the interpretation keeps observed and inferred apart. Silent changes are flagged, never asserted as "unannounced" without checking recent announcement-type events of the same source.
• Do not add sensors that are blocked with 403 by design (Akamai/Cloudflare bot management) unless the source has fallback.scrapfly: true and a tier ≥ C; Scrapfly is a budgeted fallback, not the foundation.
• All timestamps UTC. Ids are prefixed (src_, sen_, snap_, chg_, evt_, clu_, ent_…).

# Dev

createdb websensor && cp .env.example .env
pnpm install
pnpm db:migrate                      # or let the engine migrate on start
npx tsx apps/engine/src/cli.ts sync  # registry → DB
npx tsx apps/engine/src/cli.ts run-due 50
pnpm dev:api · pnpm dev:engine · pnpm dev:web
pnpm test · pnpm typecheck

Run CLI/engine from the repo root (config paths are relative to cwd).

# Deploy (MacLustr)

mld stage . websensor && mld deploy websensor --node M4M64b. Manifest deploy/websensor.mld.json (secrets only on M1M32). Processes: websensor-api (:8260, ngrok www.websensor.io), websensor-web (:8261 loopback), websensor-engine (metrics :8262). Postgres 17 websensor + Redis local. Blob store ~/websensor-data/blobs. See deploy/README.md.