# WebSensor — repository guide

WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing
Web. It monitors official public sources, detects meaningful changes, classifies them, links entities, scores
importance, preserves evidence and publishes events through a WebSocket feed. The product brief that drove the
design is `docs/PRODUCT-BRIEF.md`; this file is the working guide for the code.

## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)
- `packages/core` — taxonomy (event types + intrinsic severity, tiers, categories), ids, SSRF policy
  (`assertUrlAllowed`, `safeLookup`), hashing (sha256, simhash, shingles/Jaccard), canonical extraction
  (`canonicalizeHtml`: strips scripts/nav/footer/cookie chrome, timestamps, tokens, counters), diff engines
  (text / json / keyed list), stage-1 heuristics (`evaluateChange`, `describeChange`), scoring
  (importance components, confidence, novelty helpers, trending, activity anomaly), adaptive schedule.
  `@websensor/core/client` is the browser-safe subset (no `node:` imports) — client components must import it.
- `packages/db` — plain SQL migrations (`migrations/*.sql`, applied by `migrate()` with an advisory lock) +
  Drizzle schema for typed access. **`textArray()`** must be used for `= any(...)` / `&&` with JS arrays
  (Drizzle spreads arrays into parameter lists).
- `packages/store` — content-addressed blob store (`sha256/ab/cd/<hash>.zst`, zstd via `node:zlib`). Raw
  bodies, canonical representations and full diffs live there, never in Postgres. Interface ready for S3/MinIO.
- `packages/connectors` — connector SDK (`WebSensorConnector.fetch/normalize`), safe fetcher (conditional GET,
  manual redirects validated per hop, HTTP/2→1.1 pin on NGHTTP2 errors, browser-UA second attempt on
  403/resets, size + time limits), connectors: `http` (HTML/JSON/text/HEAD), `rss` (RSS/Atom/RDF/JSON Feed),
  `sitemap` (index + news), `statuspage` (Atlassian v2 summary), `github` (releases/tags/commits Atom,
  advisories REST), `jsonlist` (keyed records from any JSON API; `{now-2h}` placeholders), `discovery`
  (robots sitemaps, `<link rel=alternate>`, well-known feed paths, linked status pages — every candidate is
  fetched and parsed), `scrapfly` (fallback, budgeted, only when `fallback.scrapfly` is set on the source).
- `apps/engine` — scheduler (`FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency), pipeline
  (fetch → normalize → snapshot → diff → heuristics → change → event: entities, novelty, LLM interpretation,
  clustering, importance/confidence, silent-change detection, publish to Redis), registry sync from
  `config/sources.yaml`, discovery, connector health rollups, Prometheus metrics on :8262. CLI: `src/cli.ts`
  (`sync`, `discover`, `probe <domain>`, `run-once <sensor>`, `run-due [n]`).
- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*`, WebSocket `/api/v1/live` (Redis pub/sub fan-out,
  channels `events:*`, `entity:*`, `source:*`, `watchlist:*`), `/api/v1/feed.rss`, `/api/health` `/api/ready` `/api/metrics`, apex→www
  redirect, and a reverse proxy to the Next.js app for everything else.
- `apps/web` — Next.js 16 frontend (:8261, loopback). Pages: live, breaking, explore, sources, entities,
  timelines, silent changes, watchlists, alerts, event detail with diff viewer, health, API docs.
- `config/sources.yaml` — the 200-organization registry (sources, curated sensors, product entities, discovery
  flags, fallbacks). `docs/connectors/*.md` document each connector family.

## Rules
- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets — goes through
  `assertUrlAllowed()`. Private ranges, metadata endpoints, `.maclustr.io`/`.ts.net` and single-label hosts are
  blocked; the dispatcher's DNS lookup only returns approved addresses.
- Raw evidence is immutable: snapshots and diffs are never rewritten. Re-interpretation creates a new row in
  `interpretations` (versioned); `events.interpretation` holds the latest.
- LLM cost control: heuristics first; Claude is called only for candidates above `WS_LLM_MIN_IMPORTANCE`, with a
  daily call budget; the deep model only above `WS_LLM_DEEP_MIN_IMPORTANCE`. Output is strict JSON
  (`output_config.format`), never free text. No key → heuristics only, the system still works.
- Feeds: items are "new" only if their key was never seen (sensor `state.seenKeys`) and they are not older than
  14 days; items scrolling out of the window are never "removed". Sitemaps/statuspages: a >50 % shrink is
  treated as a partial response, not mass deletion. A 404 becomes `page_removed` only after
  `WS_DELETE_CONFIRMATIONS` checks separated by `WS_DELETE_SEPARATION_MIN`.
- Never label inference as fact: events carry `evidence_label` (OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED)
  and the interpretation keeps `observed` and `inferred` apart. Silent changes are flagged, never asserted as
  "unannounced" without checking recent announcement-type events of the same source.
- Do not add sensors that are blocked with 403 by design (Akamai/Cloudflare bot management) unless the source
  has `fallback.scrapfly: true` and a tier ≥ C; Scrapfly is a budgeted fallback, not the foundation.
- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`…).

## Dev
```
createdb websensor && cp .env.example .env
pnpm install
pnpm db:migrate                      # or let the engine migrate on start
npx tsx apps/engine/src/cli.ts sync  # registry → DB
npx tsx apps/engine/src/cli.ts run-due 50
pnpm dev:api · pnpm dev:engine · pnpm dev:web
pnpm test · pnpm typecheck
```
Run CLI/engine from the repo root (config paths are relative to cwd).

## Deploy (MacLustr)
`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `deploy/websensor.mld.json` (secrets only
on M1M32). Processes: `websensor-api` (:8260, ngrok www.websensor.io), `websensor-web` (:8261 loopback),
`websensor-engine` (metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`.
See `deploy/README.md`.