WebSensor v0.1.0 — real-time web intelligence platform · aafdcf9 · websensor

added .env.example +30 −0

++# WebSensor — environment (never commit real values)
++NODE_ENV=development
++DATABASE_URL=postgres://localhost:5432/websensor
++REDIS_URL=redis://127.0.0.1:6379
++
++# Content-addressed blob store (fs driver). S3/MinIO driver can be plugged in later.
++BLOB_STORE_DIR=./data/blobs
++
++PUBLIC_BASE_URL=http://localhost:8260
++API_PORT=8260
++API_HOST=127.0.0.1
++WEB_PORT=8261
++WEB_URL=http://127.0.0.1:8261
++ENGINE_METRICS_PORT=8262
++
++# Optional acquisition fallbacks
++FIRECRAWL_API_KEY=
++SCRAPFLY_API_KEY=
++
++# AI interpretation (heuristics run without any key)
++ANTHROPIC_API_KEY=
++WS_LLM_MODEL_FAST=claude-haiku-4-5
++WS_LLM_MODEL_DEEP=claude-opus-5
++WS_LLM_DAILY_CALL_BUDGET=600
++WS_LLM_MIN_IMPORTANCE=35
++
++# Engine tuning
++WS_FETCH_CONCURRENCY=16
++WS_USER_AGENT=WebSensorBot/0.1 (+https://www.websensor.io/bot)
++WS_SOURCES_FILE=./config/sources.yaml

added .gitignore +17 −0

++node_modules/
++.next/
++dist/
++*.tsbuildinfo
++.env
++.env.*
++!.env.example
++logs/
++tmp/
++coverage/
++.DS_Store
++.claude/
++deploy/*.mld.json
++data/
++qa/test-results/
++qa/.e2e-*
++apps/web/next-env.d.ts

added CLAUDE.md +76 −0

++# WebSensor — repository guide
++
++WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing
++Web. It monitors official public sources, detects meaningful changes, classifies them, links entities, scores
++importance, preserves evidence and publishes events through a WebSocket feed. The product brief that drove the
++design is `docs/PRODUCT-BRIEF.md`; this file is the working guide for the code.
++
++## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)
++- `packages/core` — taxonomy (event types + intrinsic severity, tiers, categories), ids, SSRF policy
++  (`assertUrlAllowed`, `safeLookup`), hashing (sha256, simhash, shingles/Jaccard), canonical extraction
++  (`canonicalizeHtml`: strips scripts/nav/footer/cookie chrome, timestamps, tokens, counters), diff engines
++  (text / json / keyed list), stage-1 heuristics (`evaluateChange`, `describeChange`), scoring
++  (importance components, confidence, novelty helpers, trending, activity anomaly), adaptive schedule.
++  `@websensor/core/client` is the browser-safe subset (no `node:` imports) — client components must import it.
++- `packages/db` — plain SQL migrations (`migrations/*.sql`, applied by `migrate()` with an advisory lock) +
++  Drizzle schema for typed access. **`textArray()`** must be used for `= any(...)` / `&&` with JS arrays
++  (Drizzle spreads arrays into parameter lists).
++- `packages/store` — content-addressed blob store (`sha256/ab/cd/<hash>.zst`, zstd via `node:zlib`). Raw
++  bodies, canonical representations and full diffs live there, never in Postgres. Interface ready for S3/MinIO.
++- `packages/connectors` — connector SDK (`WebSensorConnector.fetch/normalize`), safe fetcher (conditional GET,
++  manual redirects validated per hop, HTTP/2→1.1 pin on NGHTTP2 errors, browser-UA second attempt on
++  403/resets, size + time limits), connectors: `http` (HTML/JSON/text/HEAD), `rss` (RSS/Atom/RDF/JSON Feed),
++  `sitemap` (index + news), `statuspage` (Atlassian v2 summary), `github` (releases/tags/commits Atom,
++  advisories REST), `jsonlist` (keyed records from any JSON API; `{now-2h}` placeholders), `discovery`
++  (robots sitemaps, `<link rel=alternate>`, well-known feed paths, linked status pages — every candidate is
++  fetched and parsed), `scrapfly` (fallback, budgeted, only when `fallback.scrapfly` is set on the source).
++- `apps/engine` — scheduler (`FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency), pipeline
++  (fetch → normalize → snapshot → diff → heuristics → change → event: entities, novelty, LLM interpretation,
++  clustering, importance/confidence, silent-change detection, publish to Redis), registry sync from
++  `config/sources.yaml`, discovery, connector health rollups, Prometheus metrics on :8262. CLI: `src/cli.ts`
++  (`sync`, `discover`, `probe <domain>`, `run-once <sensor>`, `run-due [n]`).
++- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*`, WebSocket `/api/v1/live` (Redis pub/sub fan-out,
++  channels `events:*`, `entity:*`, `source:*`, `watchlist:*`), `/api/v1/feed.rss`, `/api/health` `/api/ready` `/api/metrics`, apex→www
++  redirect, and a reverse proxy to the Next.js app for everything else.
++- `apps/web` — Next.js 16 frontend (:8261, loopback). Pages: live, breaking, explore, sources, entities,
++  timelines, silent changes, watchlists, alerts, event detail with diff viewer, health, API docs.
++- `config/sources.yaml` — the 200-organization registry (sources, curated sensors, product entities, discovery
++  flags, fallbacks). `docs/connectors/*.md` document each connector family.
++
++## Rules
++- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets — goes through
++  `assertUrlAllowed()`. Private ranges, metadata endpoints, `.maclustr.io`/`.ts.net` and single-label hosts are
++  blocked; the dispatcher's DNS lookup only returns approved addresses.
++- Raw evidence is immutable: snapshots and diffs are never rewritten. Re-interpretation creates a new row in
++  `interpretations` (versioned); `events.interpretation` holds the latest.
++- LLM cost control: heuristics first; Claude is called only for candidates above `WS_LLM_MIN_IMPORTANCE`, with a
++  daily call budget; the deep model only above `WS_LLM_DEEP_MIN_IMPORTANCE`. Output is strict JSON
++  (`output_config.format`), never free text. No key → heuristics only, the system still works.
++- Feeds: items are "new" only if their key was never seen (sensor `state.seenKeys`) and they are not older than
++  14 days; items scrolling out of the window are never "removed". Sitemaps/statuspages: a >50 % shrink is
++  treated as a partial response, not mass deletion. A 404 becomes `page_removed` only after
++  `WS_DELETE_CONFIRMATIONS` checks separated by `WS_DELETE_SEPARATION_MIN`.
++- Never label inference as fact: events carry `evidence_label` (OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED)
++  and the interpretation keeps `observed` and `inferred` apart. Silent changes are flagged, never asserted as
++  "unannounced" without checking recent announcement-type events of the same source.
++- Do not add sensors that are blocked with 403 by design (Akamai/Cloudflare bot management) unless the source
++  has `fallback.scrapfly: true` and a tier ≥ C; Scrapfly is a budgeted fallback, not the foundation.
++- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`…).
++
++## Dev
++```
++createdb websensor && cp .env.example .env
++pnpm install
++pnpm db:migrate                      # or let the engine migrate on start
++npx tsx apps/engine/src/cli.ts sync  # registry → DB
++npx tsx apps/engine/src/cli.ts run-due 50
++pnpm dev:api · pnpm dev:engine · pnpm dev:web
++pnpm test · pnpm typecheck
++```
++Run CLI/engine from the repo root (config paths are relative to cwd).
++
++## Deploy (MacLustr)
++`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `deploy/websensor.mld.json` (secrets only
++on M1M32). Processes: `websensor-api` (:8260, ngrok www.websensor.io), `websensor-web` (:8261 loopback),
++`websensor-engine` (metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`.
++See `deploy/README.md`.

added README.md +33 −0

++# WebSensor
++
++**Detect What Changed. Know Why It Matters.** — https://www.websensor.io
++
++WebSensor is a global sensor network for the changing Web. It continuously monitors official public sources
++(newsrooms, status pages, changelogs, regulators, registries, APIs), preserves every observation, detects raw
++changes, filters noise, interprets meaningful changes into events, links them to entities, scores their
++importance and confidence, clusters related events and publishes them live over WebSocket.
++
++```
++Sources → Sensors/Connectors → Fetch (conditional) → Snapshot (immutable) → Diff → Heuristics
++       → Interpretation (Claude, budgeted) → Entities → Novelty/Clustering → Importance → Event
++       → Redis stream → WebSocket gateway → www.websensor.io
++```
++
++- 200 organizations, ~290 curated sensors at launch (`config/sources.yaml`), plus validated discovery.
++- Connector families: HTTP (HTML/JSON/HEAD), RSS/Atom/JSON Feed, sitemaps, Statuspage, GitHub, JSON APIs
++  (CISA KEV, NVD, Federal Register, USGS, HIBP, Google Cloud incidents…), Scrapfly fallback.
++- Every event is auditable: event → change → snapshot A/B → fetch run → sensor → source, with raw evidence.
++- Observed vs inferred is always labelled; silent (unannounced) changes are flagged.
++
++## Quick start
++```
++createdb websensor && cp .env.example .env && pnpm install
++npx tsx apps/engine/src/cli.ts sync        # load the registry
++pnpm dev:api & pnpm dev:web & pnpm dev:engine
++open http://localhost:8260
++```
++
++Docs: `CLAUDE.md` (repo guide), `docs/ARCHITECTURE.md`, `docs/connectors/*.md`, `deploy/README.md`,
++API docs at `/api` on the site. Public API: `GET /api/v1/events`, `/api/v1/entities/{id}`,
++`/api/v1/sources/{id}`, `/api/v1/domains/{domain}/timeline`, `/api/v1/changes/{id}`, `wss://…/api/v1/live`,
++`/api/v1/feed.rss`.

added apps/api/package.json +35 −0

++{
++  "name": "@websensor/api",
++  "version": "0.1.0",
++  "private": true,
++  "type": "module",
++  "scripts": {
++    "dev": "tsx watch src/server.ts",
++    "start": "tsx src/server.ts",
++    "typecheck": "tsc -p tsconfig.json --noEmit",
++    "test": "vitest run --passWithNoTests"
++  },
++  "dependencies": {
++    "@fastify/cors": "^11.0.0",
++    "@fastify/rate-limit": "^10.3.0",
++    "@fastify/reply-from": "^12.6.5",
++    "@fastify/websocket": "^11.2.0",
++    "@websensor/core": "workspace:*",
++    "@websensor/db": "workspace:*",
++    "@websensor/store": "workspace:*",
++    "fastify": "^5.4.0",
++    "ioredis": "^5.6.0",
++    "pino": "^9.7.0",
++    "pino-pretty": "^13.0.0",
++    "prom-client": "^15.1.0",
++    "tsx": "^4.20.0",
++    "ws": "^8.21.3",
++    "zod": "^4.0.0"
++  },
++  "devDependencies": {
++    "@types/node": "^24.0.0",
++    "@types/ws": "^8.18.1",
++    "typescript": "^5.9.3",
++    "vitest": "^3.2.0"
++  }
++}

added apps/api/src/config.ts +14 −0

++const env = process.env;
++
++export const config = {
++  env: env.NODE_ENV ?? "development",
++  logLevel: env.LOG_LEVEL ?? "info",
++  port: Number(env.API_PORT ?? 8260),
++  host: env.API_HOST ?? "127.0.0.1",
++  redisUrl: env.REDIS_URL ?? "redis://127.0.0.1:6379",
++  webUrl: env.WEB_URL ?? "http://127.0.0.1:8261",
++  publicBaseUrl: env.PUBLIC_BASE_URL ?? "http://localhost:8260",
++  canonicalHost: env.CANONICAL_HOST ?? "www.websensor.io",
++  redirectApexToWww: (env.WS_REDIRECT_APEX ?? "1") !== "0",
++  version: "0.1.0",
++};

added apps/api/src/live.ts +129 −0

++import type { FastifyInstance } from "fastify";
++import type { WebSocket } from "ws";
++import Redis from "ioredis";
++import { FEED_CHANNELS } from "@websensor/core";
++import { db, sql } from "@websensor/db";
++import { config } from "./config";
++
++/**
++ * WebSocket gateway `/api/v1/live`. One Redis subscriber fans out to every client; clients
++ * pick channels: events:global · events:breaking · events:<ai|cyber|finance|health|government|
++ * science|products|infrastructure> · entity:<id> · source:<id> · watchlist:<id>.
++ * Protocol (JSON): client → {"subscribe":[…]} | {"unsubscribe":[…]} | {"ping":1}
++ *                  server → {"type":"hello"} | {"type":"event", "channels":[…], "event":{…}} | {"type":"pong"}
++ */
++interface Client {
++  ws: WebSocket;
++  channels: Set<string>;
++  watchlists: Map<string, { entities: Set<string>; sources: Set<string>; keywords: string[]; categories: Set<string> }>;
++}
++
++const clients = new Set<Client>();
++let sub: Redis | null = null;
++let published = 0;
++
++export function liveStats(): { clients: number; published: number } {
++  return { clients: clients.size, published };
++}
++
++export async function registerLive(app: FastifyInstance): Promise<void> {
++  sub = new Redis(config.redisUrl, { maxRetriesPerRequest: 3 });
++  sub.on("error", (e) => app.log.warn({ err: e.message }, "redis sub error"));
++  await sub.subscribe("ws:live");
++  sub.on("message", (_ch, msg) => {
++    let ev: Record<string, unknown>;
++    try {
++      ev = JSON.parse(msg) as Record<string, unknown>;
++    } catch {
++      return;
++    }
++    published++;
++    const chans = channelsFor(ev);
++    for (const c of clients) {
++      const hit = [...chans].filter((ch) => c.channels.has(ch));
++      for (const [wid, w] of c.watchlists) if (matchesWatchlist(ev, w)) hit.push(`watchlist:${wid}`);
++      if (!hit.length) continue;
++      if (c.ws.readyState === c.ws.OPEN) c.ws.send(JSON.stringify({ type: "event", channels: hit, event: ev }));
++    }
++  });
++
++  app.get("/api/v1/live", { websocket: true }, (socket) => {
++    const client: Client = { ws: socket, channels: new Set(["events:global"]), watchlists: new Map() };
++    clients.add(client);
++    socket.send(JSON.stringify({ type: "hello", channels: [...client.channels], serverTime: new Date().toISOString() }));
++    socket.on("message", async (raw: Buffer | string) => {
++      let msg: { subscribe?: string[]; unsubscribe?: string[]; ping?: number };
++      try {
++        msg = JSON.parse(raw.toString()) as typeof msg;
++      } catch {
++        return;
++      }
++      if (msg.ping) socket.send(JSON.stringify({ type: "pong", t: Date.now() }));
++      for (const ch of msg.subscribe ?? []) {
++        if (typeof ch !== "string" || ch.length > 120 || client.channels.size > 64) continue;
++        if (ch.startsWith("watchlist:")) await loadWatchlist(client, ch.slice(10));
++        else client.channels.add(ch);
++      }
++      for (const ch of msg.unsubscribe ?? []) {
++        client.channels.delete(ch);
++        if (ch.startsWith("watchlist:")) client.watchlists.delete(ch.slice(10));
++      }
++      socket.send(JSON.stringify({ type: "subscribed", channels: [...client.channels, ...[...client.watchlists.keys()].map((w) => `watchlist:${w}`)] }));
++    });
++    const hb = setInterval(() => {
++      if (socket.readyState === socket.OPEN) socket.send(JSON.stringify({ type: "heartbeat", t: Date.now() }));
++    }, 25_000);
++    socket.on("close", () => {
++      clearInterval(hb);
++      clients.delete(client);
++    });
++    socket.on("error", () => {
++      clearInterval(hb);
++      clients.delete(client);
++    });
++  });
++}
++
++export function channelsFor(ev: Record<string, unknown>): Set<string> {
++  const out = new Set<string>(["events:global"]);
++  const importance = Number(ev.importance ?? 0);
++  if (importance >= 80) out.add("events:breaking");
++  if (ev.silent) out.add("events:silent");
++  const cats = (ev.categories as string[] | undefined) ?? [];
++  for (const [ch, list] of Object.entries(FEED_CHANNELS)) if (list.some((c) => cats.includes(c)) || cats.includes(ch)) out.add(`events:${ch}`);
++  const src = ev.source as { id?: string } | undefined;
++  if (src?.id) out.add(`source:${src.id}`);
++  for (const e of (ev.entities as { id: string }[] | undefined) ?? []) out.add(`entity:${e.id}`);
++  out.add(`type:${String(ev.type)}`);
++  return out;
++}
++
++async function loadWatchlist(client: Client, id: string): Promise<void> {
++  const rows = await db.execute<{ kind: string; value: string }>(sql`select kind, value from watchlist_items where watchlist_id = ${id}`);
++  const w = { entities: new Set<string>(), sources: new Set<string>(), keywords: [] as string[], categories: new Set<string>() };
++  for (const r of rows.rows) {
++    if (r.kind === "entity") w.entities.add(r.value);
++    else if (r.kind === "source") w.sources.add(r.value);
++    else if (r.kind === "keyword") w.keywords.push(r.value.toLowerCase());
++    else if (r.kind === "category") w.categories.add(r.value);
++  }
++  client.watchlists.set(id, w);
++}
++
++function matchesWatchlist(ev: Record<string, unknown>, w: { entities: Set<string>; sources: Set<string>; keywords: string[]; categories: Set<string> }): boolean {
++  const src = ev.source as { id?: string } | undefined;
++  if (src?.id && w.sources.has(src.id)) return true;
++  for (const e of (ev.entities as { id: string }[] | undefined) ?? []) if (w.entities.has(e.id)) return true;
++  for (const c of (ev.categories as string[] | undefined) ?? []) if (w.categories.has(c)) return true;
++  if (w.keywords.length) {
++    const hay = `${String(ev.title)} ${String(ev.summary)}`.toLowerCase();
++    if (w.keywords.some((k) => hay.includes(k))) return true;
++  }
++  return false;
++}
++
++export async function closeLive(): Promise<void> {
++  for (const c of clients) c.ws.close(1001, "server shutdown");
++  clients.clear();
++  if (sub) await sub.quit().catch(() => undefined);
++}

added apps/api/src/queries.ts +146 −0

++import { db, sql, textArray } from "@websensor/db";
++
++/** Read-model helpers shared by REST routes. All return plain JSON-ready objects. */
++
++export interface EventFilters {
++  after?: string;
++  before?: string;
++  category?: string;
++  entity?: string;
++  source?: string;
++  domain?: string;
++  sensor?: string;
++  cluster?: string;
++  importance_min?: number;
++  confidence_min?: number;
++  event_type?: string;
++  silent_change?: boolean;
++  q?: string;
++  limit: number;
++  cursor?: string;
++  order?: "recent" | "importance";
++}
++
++export const EVENT_SELECT = sql`
++  e.id, e.slug, e.event_type, e.title, e.summary, e.why_it_matters, e.importance, e.confidence, e.novelty, e.categories, e.keywords,
++  e.silent_change, e.evidence_label, e.url, e.detected_at, e.published_at, e.observed_from, e.processed_at, e.published_to_feed_at,
++  e.detection_latency_ms, e.processing_latency_ms, e.cluster_id, e.sensor_id, e.source_id, e.change_id, e.old_snapshot_id, e.new_snapshot_id,
++  e.importance_components, e.processing_version,
++  json_build_object('id', s.id, 'name', s.name, 'domain', s.domain, 'tier', s.tier, 'categories', s.categories) as source,
++  json_build_object('id', sen.id, 'name', sen.name, 'type', sen.type, 'connector', sen.connector, 'tier', sen.tier) as sensor,
++  coalesce((select json_agg(json_build_object('id', en.id, 'name', en.name, 'type', en.type, 'role', ee.role) order by ee.role, en.name)
++            from event_entities ee join entities en on en.id = ee.entity_id where ee.event_id = e.id), '[]'::json) as entities,
++  (select event_count from event_clusters c where c.id = e.cluster_id) as cluster_size`;
++
++export async function listEvents(f: EventFilters): Promise<{ items: Record<string, unknown>[]; nextCursor: string | null }> {
++  const conds = [sql`true`];
++  if (f.after) conds.push(sql`e.detected_at > ${new Date(f.after)}`);
++  if (f.before) conds.push(sql`e.detected_at < ${new Date(f.before)}`);
++  if (f.category) conds.push(sql`${f.category} = any(e.categories)`);
++  if (f.source) conds.push(sql`e.source_id = ${f.source}`);
++  if (f.sensor) conds.push(sql`e.sensor_id = ${f.sensor}`);
++  if (f.cluster) conds.push(sql`e.cluster_id = ${f.cluster}`);
++  if (f.domain) conds.push(sql`s.domain = ${f.domain}`);
++  if (f.entity) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = ${f.entity})`);
++  if (f.importance_min !== undefined) conds.push(sql`e.importance >= ${f.importance_min}`);
++  if (f.confidence_min !== undefined) conds.push(sql`e.confidence >= ${f.confidence_min}`);
++  if (f.event_type) conds.push(sql`e.event_type = any(${textArray(f.event_type.split(","))})`);
++  if (f.silent_change !== undefined) conds.push(sql`e.silent_change = ${f.silent_change}`);
++  if (f.q) conds.push(sql`e.search @@ websearch_to_tsquery('english', ${f.q})`);
++  if (f.cursor) {
++    const [ts, id] = decodeCursor(f.cursor);
++    if (f.order === "importance") conds.push(sql`(e.importance, e.id) < (${Number(ts)}, ${id})`);
++    else conds.push(sql`(e.detected_at, e.id) < (${new Date(Number(ts))}, ${id})`);
++  }
++  const order = f.order === "importance" ? sql`e.importance desc, e.id desc` : sql`e.detected_at desc, e.id desc`;
++  const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where ${sql.join(conds, sql` and `)} order by ${order} limit ${f.limit + 1}`);
++  const items = rows.rows.slice(0, f.limit);
++  const last = items[items.length - 1];
++  const nextCursor = rows.rows.length > f.limit && last ? encodeCursor(f.order === "importance" ? String(last.importance) : String(new Date(last.detected_at as string).getTime()), String(last.id)) : null;
++  return { items, nextCursor };
++}
++
++export async function getEvent(idOrSlug: string): Promise<Record<string, unknown> | null> {
++  const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT}, e.interpretation from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.id = ${idOrSlug} or e.slug = ${idOrSlug} limit 1`);
++  return rows.rows[0] ?? null;
++}
++
++export async function relatedEvents(ev: Record<string, unknown>, limit = 8): Promise<Record<string, unknown>[]> {
++  const rows = await db.execute<Record<string, unknown>>(sql`
++    select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id
++    where e.id <> ${String(ev.id)} and (e.cluster_id = ${String(ev.cluster_id ?? "")} or e.source_id = ${String(ev.source_id)} or exists (select 1 from event_entities a join event_entities b on a.entity_id = b.entity_id where a.event_id = e.id and b.event_id = ${String(ev.id)}))
++    order by (e.cluster_id = ${String(ev.cluster_id ?? "")}) desc, e.detected_at desc limit ${limit}`);
++  return rows.rows;
++}
++
++export async function stats(): Promise<Record<string, unknown>> {
++  const [r] = (
++    await db.execute<Record<string, unknown>>(sql`
++    select
++      (select count(*) from sources where enabled) as sources,
++      (select count(*) from sensors where enabled) as sensors,
++      (select count(*) from entities) as entities,
++      (select count(*) from snapshots) as snapshots,
++      (select count(*) from events) as events_total,
++      (select count(*) from events where detected_at >= now() - interval '24 hours') as events_24h,
++      (select count(*) from events where detected_at >= now() - interval '24 hours' and silent_change) as silent_24h,
++      (select count(*) from events where detected_at >= now() - interval '24 hours' and importance >= 80) as breaking_24h,
++      (select count(*) from changes where detected_at >= now() - interval '24 hours') as changes_24h,
++      (select coalesce(sum(checks),0) from metrics_daily where day = current_date) as checks_today,
++      (select coalesce(sum(not_modified),0) from metrics_daily where day = current_date) as not_modified_today,
++      (select coalesce(sum(bytes),0) from metrics_daily where day = current_date) as bytes_today,
++      (select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour,
++      (select count(*) from sensors where health = 'UP' and enabled) as sensors_up,
++      (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled) as sensors_degraded,
++      (select max(started_at) from sensor_runs) as last_check_at,
++      (select max(detected_at) from events) as last_event_at,
++      (select percentile_cont(0.5) within group (order by processing_latency_ms) from events where detected_at >= now() - interval '24 hours') as p50_processing_ms,
++      (select percentile_cont(0.5) within group (order by detection_latency_ms) from events where detected_at >= now() - interval '24 hours' and detection_latency_ms is not null and detection_latency_ms < 86400000) as p50_detection_ms`)
++  ).rows;
++  return Object.fromEntries(Object.entries(r ?? {}).map(([k, v]) => [k, typeof v === "string" && /^\d+$/.test(v) ? Number(v) : v]));
++}
++
++export async function trending(hours = 24, limit = 12): Promise<Record<string, unknown>[]> {
++  const rows = await db.execute<Record<string, unknown>>(sql`
++    with cur as (
++      select ee.entity_id, count(*) as n, sum(e.importance) as imp, count(distinct e.source_id) as sources, sum(case when e.silent_change then 1 else 0 end) as silent, max(e.importance) as max_imp
++      from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - make_interval(hours => ${hours}) group by ee.entity_id),
++    prev as (
++      select ee.entity_id, count(*) as n from events e join event_entities ee on ee.event_id = e.id
++      where e.detected_at >= now() - make_interval(hours => ${hours * 2}) and e.detected_at < now() - make_interval(hours => ${hours}) group by ee.entity_id)
++    select en.id, en.name, en.type, en.domain, cur.n::int as events, cur.imp::float as importance_sum, cur.sources::int as sources, cur.silent::int as silent, cur.max_imp::float as max_importance, coalesce(prev.n,0)::int as prev_events,
++      round((18*(ln(1+cur.n)/ln(2)) + 0.35*(cur.imp/greatest(1,cur.n)) + 10*(ln(1+cur.sources)/ln(2)) + 12*least(2, case when coalesce(prev.n,0)=0 then 2 else cur.n::float/prev.n end) + 5*least(3,cur.silent))::numeric, 1)::float as score
++    from cur join entities en on en.id = cur.entity_id left join prev on prev.entity_id = cur.entity_id
++    order by score desc limit ${limit}`);
++  return rows.rows;
++}
++
++export async function sourceActivity(sourceId: string): Promise<Record<string, unknown>> {
++  const [r] = (
++    await db.execute<Record<string, unknown>>(sql`
++    select
++      (select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '2 hours')::int as changes_2h,
++      (select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '14 days')::int as changes_14d,
++      (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '24 hours')::int as events_24h,
++      (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '14 days')::int as events_14d`)
++  ).rows;
++  const c2h = Number(r?.changes_2h ?? 0);
++  const baselinePerHour = Number(r?.changes_14d ?? 0) / (14 * 24);
++  const currentPerHour = c2h / 2;
++  let anomaly = 0;
++  if (baselinePerHour <= 0) anomaly = currentPerHour > 2 ? 70 : currentPerHour > 0 ? 40 : 0;
++  else {
++    const ratio = currentPerHour / baselinePerHour;
++    anomaly = ratio <= 1 ? ratio * 30 : Math.min(100, 30 + 25 * Math.log2(ratio));
++  }
++  return { ...r, baseline_changes_per_day: Math.round(baselinePerHour * 24 * 10) / 10, activity_score: Math.round(anomaly * 10) / 10 };
++}
++
++export function encodeCursor(a: string, b: string): string {
++  return Buffer.from(`${a}|${b}`).toString("base64url");
++}
++export function decodeCursor(c: string): [string, string] {
++  const s = Buffer.from(c, "base64url").toString("utf8");
++  const i = s.indexOf("|");
++  return [s.slice(0, i), s.slice(i + 1)];
++}

added apps/api/src/routes.ts +349 −0

++import type { FastifyInstance } from "fastify";
++import { z } from "zod";
++import { diffText, EVENT_TYPES, FEED_CHANNELS, newId } from "@websensor/core";
++import { db, sql, textArray } from "@websensor/db";
++import { getBlobStore } from "@websensor/store";
++import { config } from "./config";
++import { liveStats } from "./live";
++import { getEvent, listEvents, relatedEvents, sourceActivity, stats, trending, EVENT_SELECT } from "./queries";
++
++const eventsQuery = z.object({
++  after: z.string().optional(),
++  before: z.string().optional(),
++  category: z.string().optional(),
++  entity: z.string().optional(),
++  source: z.string().optional(),
++  domain: z.string().optional(),
++  sensor: z.string().optional(),
++  cluster: z.string().optional(),
++  importance_min: z.coerce.number().min(0).max(100).optional(),
++  confidence_min: z.coerce.number().min(0).max(100).optional(),
++  event_type: z.string().optional(),
++  silent_change: z
++    .enum(["true", "false"])
++    .transform((v) => v === "true")
++    .optional(),
++  q: z.string().max(200).optional(),
++  limit: z.coerce.number().int().min(1).max(200).default(50),
++  cursor: z.string().optional(),
++  order: z.enum(["recent", "importance"]).default("recent"),
++});
++
++function ownerToken(headers: Record<string, unknown>): string | null {
++  const t = headers["x-websensor-owner"];
++  return typeof t === "string" && /^[A-Za-z0-9_-]{16,80}$/.test(t) ? t : null;
++}
++
++export async function registerRoutes(app: FastifyInstance): Promise<void> {
++  // ---- Health ---------------------------------------------------------------------------
++  app.get("/api/health", async () => ({ status: "ok", service: "api", version: config.version, time: new Date().toISOString() }));
++  app.get("/api/ready", async (_req, reply) => {
++    const checks: Record<string, boolean | number> = {};
++    try {
++      await db.execute(sql`select 1`);
++      checks.database = true;
++    } catch {
++      checks.database = false;
++    }
++    const engine = await db.execute<{ last: Date | null }>(sql`select max(started_at) as last from sensor_runs where started_at >= now() - interval '15 minutes'`);
++    checks.engine_recent = Boolean(engine.rows[0]?.last);
++    checks.ws_clients = liveStats().clients;
++    return reply.status(checks.database ? 200 : 503).send({ status: checks.database ? "ready" : "degraded", checks });
++  });
++
++  // ---- Events ---------------------------------------------------------------------------
++  app.get("/api/v1/events", async (req) => {
++    const q = eventsQuery.parse(req.query);
++    return listEvents(q);
++  });
++  app.get<{ Params: { id: string } }>("/api/v1/events/:id", async (req, reply) => {
++    const ev = await getEvent(req.params.id);
++    if (!ev) return reply.status(404).send({ error: "not_found" });
++    const [related, cluster, change, interp] = await Promise.all([
++      relatedEvents(ev),
++      ev.cluster_id ? db.execute<Record<string, unknown>>(sql`select * from event_clusters where id = ${String(ev.cluster_id)}`).then((r) => r.rows[0] ?? null) : null,
++      ev.change_id ? db.execute<Record<string, unknown>>(sql`select id, kind, diff, signal, noise_ratio, magnitude, heuristic, detected_at, old_snapshot_id, new_snapshot_id from changes where id = ${String(ev.change_id)}`).then((r) => r.rows[0] ?? null) : null,
++      db.execute<Record<string, unknown>>(sql`select version, model, created_at from interpretations where event_id = ${String(ev.id)} order by version`).then((r) => r.rows),
++    ]);
++    const snaps = await db.execute<Record<string, unknown>>(sql`select id, url, captured_at, http_status, content_type, content_length, content_hash, canonical_hash, etag, last_modified, title, mode, fetch_duration_ms, extraction_confidence from snapshots where id in (${String(ev.old_snapshot_id ?? "")}, ${String(ev.new_snapshot_id ?? "")})`);
++    const sourceRel = await db.execute<Record<string, unknown>>(sql`select health, success_rate, avg_latency_ms, total_runs, raw_changes, meaningful_changes, last_check_at from (select s.health, ch.success_rate, s.avg_latency_ms, s.total_runs, s.raw_changes, s.meaningful_changes, s.last_check_at from sensors s left join connector_health ch on ch.connector = s.connector where s.id = ${String(ev.sensor_id)}) x`);
++    return { event: ev, related, cluster, change, interpretations: interp, snapshots: snaps.rows, sensor_reliability: sourceRel.rows[0] ?? null };
++  });
++
++  // ---- Changes / snapshots / diffs -------------------------------------------------------
++  app.get<{ Params: { id: string } }>("/api/v1/changes/:id", async (req, reply) => {
++    const r = await db.execute<Record<string, unknown>>(sql`select c.*, s.url as sensor_url, s.name as sensor_name, s.source_id from changes c join sensors s on s.id = c.sensor_id where c.id = ${req.params.id}`);
++    const c = r.rows[0];
++    if (!c) return reply.status(404).send({ error: "not_found" });
++    let unified: string | null = null;
++    if (c.diff_storage_key) unified = await getBlobStore().getText(String(c.diff_storage_key)).catch(() => null);
++    return { change: c, unified };
++  });
++  app.get<{ Params: { id: string }; Querystring: { raw?: string } }>("/api/v1/snapshots/:id", async (req, reply) => {
++    const r = await db.execute<Record<string, unknown>>(sql`select * from snapshots where id = ${req.params.id}`);
++    const s = r.rows[0];
++    if (!s) return reply.status(404).send({ error: "not_found" });
++    const store = getBlobStore();
++    if (req.query.raw === "1" && s.storage_key) {
++      const buf = await store.get(String(s.storage_key));
++      reply.header("content-type", String(s.content_type ?? "text/plain") + (String(s.content_type ?? "").includes("charset") ? "" : "; charset=utf-8"));
++      reply.header("x-content-type-options", "nosniff");
++      reply.header("content-security-policy", "default-src 'none'; style-src 'unsafe-inline'; img-src data:");
++      return reply.send(buf);
++    }
++    const canonical = s.canonical_storage_key ? await store.getText(String(s.canonical_storage_key)).catch(() => null) : null;
++    return { snapshot: s, canonical: canonical ? JSON.parse(canonical) : null };
++  });
++  app.get<{ Querystring: { a: string; b: string } }>("/api/v1/snapshots/compare", async (req, reply) => {
++    const { a, b } = req.query;
++    if (!a || !b) return reply.status(400).send({ error: "a and b required" });
++    const rows = await db.execute<Record<string, unknown>>(sql`select id, captured_at, canonical_storage_key, mode, url, sensor_id from snapshots where id in (${a}, ${b})`);
++    const sa = rows.rows.find((r) => r.id === a);
++    const sb = rows.rows.find((r) => r.id === b);
++    if (!sa || !sb) return reply.status(404).send({ error: "not_found" });
++    const store = getBlobStore();
++    const [ca, cb] = await Promise.all([store.getText(String(sa.canonical_storage_key)), store.getText(String(sb.canonical_storage_key))]);
++    const ta = renderCanonical(JSON.parse(ca));
++    const tb = renderCanonical(JSON.parse(cb));
++    const d = diffText(ta, tb, `${a}@${String(sa.captured_at)}`, `${b}@${String(sb.captured_at)}`);
++    return { a: sa, b: sb, before: ta, after: tb, diff: { unified: d.unified, stats: d.stats, added: d.added, removed: d.removed, modified: d.modified } };
++  });
++
++  // ---- Sources & sensors ------------------------------------------------------------------
++  app.get<{ Querystring: { category?: string; q?: string; limit?: string } }>("/api/v1/sources", async (req) => {
++    const cat = req.query.category;
++    const q = req.query.q;
++    const rows = await db.execute<Record<string, unknown>>(sql`
++      select s.*, (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count,
++        (select count(*) from events e where e.source_id = s.id)::int as event_count,
++        (select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h,
++        (select max(detected_at) from events e where e.source_id = s.id) as last_event_at,
++        (select max(last_check_at) from sensors x where x.source_id = s.id) as last_check_at,
++        (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.health <> 'UP')::int as sensors_degraded
++      from sources s where s.enabled ${cat ? sql`and ${cat} = any(s.categories)` : sql``} ${q ? sql`and (s.name ilike ${"%" + q + "%"} or s.domain ilike ${"%" + q + "%"})` : sql``}
++      order by s.tier asc, events_24h desc, s.name asc limit ${Math.min(500, Number(req.query.limit ?? 300))}`);
++    return { items: rows.rows };
++  });
++  app.get<{ Params: { id: string } }>("/api/v1/sources/:id", async (req, reply) => {
++    const s = (await db.execute<Record<string, unknown>>(sql`select * from sources where id = ${req.params.id} or domain = ${req.params.id} limit 1`)).rows[0];
++    if (!s) return reply.status(404).send({ error: "not_found" });
++    const [sensors, ents, activity, candidates] = await Promise.all([
++      db.execute<Record<string, unknown>>(sql`select id, name, url, type, connector, tier, health, enabled, next_check_at, last_check_at, last_change_at, last_event_at, last_status, last_error, consecutive_errors, total_runs, total_not_modified, raw_changes, meaningful_changes, avg_latency_ms, base_interval_seconds, etag is not null as has_etag, last_modified is not null as has_last_modified from sensors where source_id = ${String(s.id)} order by tier, name`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select en.id, en.name, en.type, en.importance, en.event_count from source_entities se join entities en on en.id = se.entity_id where se.source_id = ${String(s.id)} order by en.type, en.name`).then((r) => r.rows),
++      sourceActivity(String(s.id)),
++      db.execute<Record<string, unknown>>(sql`select url, kind, evidence, score, status, found_at from discovery_candidates where source_id = ${String(s.id)} order by (score->>'value')::float desc nulls last limit 50`).then((r) => r.rows),
++    ]);
++    return { source: s, sensors, entities: ents, activity, discovery: candidates };
++  });
++  app.get<{ Params: { id: string } }>("/api/v1/sensors/:id", async (req, reply) => {
++    const s = (await db.execute<Record<string, unknown>>(sql`select s.*, so.name as source_name, so.domain from sensors s join sources so on so.id = s.source_id where s.id = ${req.params.id}`)).rows[0];
++    if (!s) return reply.status(404).send({ error: "not_found" });
++    const [runs, snaps, changes] = await Promise.all([
++      db.execute<Record<string, unknown>>(sql`select id, started_at, finished_at, http_status, outcome, error, duration_ms, bytes, fetch_method, snapshot_id from sensor_runs where sensor_id = ${String(s.id)} order by started_at desc limit 50`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_type, content_length, canonical_hash, title, mode, extraction_confidence from snapshots where sensor_id = ${String(s.id)} order by captured_at desc limit 50`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select id, detected_at, kind, signal, noise_ratio, magnitude, meaningful, event_id, old_snapshot_id, new_snapshot_id, heuristic->>'eventType' as heuristic_type from changes where sensor_id = ${String(s.id)} order by detected_at desc limit 50`).then((r) => r.rows),
++    ]);
++    const { etag: _e, last_modified: _lm, state: _st, ...pub } = s as Record<string, unknown>;
++    return { sensor: pub, runs, snapshots: snaps, changes };
++  });
++
++  // ---- Entities ----------------------------------------------------------------------------
++  app.get<{ Querystring: { type?: string; q?: string; limit?: string } }>("/api/v1/entities", async (req) => {
++    const rows = await db.execute<Record<string, unknown>>(sql`
++      select en.*, (select count(*) from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = en.id and e.detected_at >= now() - interval '24 hours')::int as events_24h
++      from entities en where true ${req.query.type ? sql`and en.type = ${req.query.type}` : sql``} ${req.query.q ? sql`and (en.search @@ plainto_tsquery('simple', ${req.query.q}) or en.name ilike ${"%" + req.query.q + "%"})` : sql``}
++      order by en.event_count desc, en.importance desc, en.name limit ${Math.min(500, Number(req.query.limit ?? 200))}`);
++    return { items: rows.rows };
++  });
++  app.get<{ Params: { id: string } }>("/api/v1/entities/:id", async (req, reply) => {
++    const e = (await db.execute<Record<string, unknown>>(sql`select * from entities where id = ${req.params.id} or id = ${"org_" + req.params.id} limit 1`)).rows[0];
++    if (!e) return reply.status(404).send({ error: "not_found" });
++    const id = String(e.id);
++    const [children, relations, sources, aliases, recent, byType] = await Promise.all([
++      db.execute<Record<string, unknown>>(sql`select id, name, type, importance, event_count, last_event_at from entities where parent_id = ${id} order by event_count desc, name`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select r.relation, r.from_id, r.to_id, f.name as from_name, t.name as to_name, t.type as to_type from entity_relations r join entities f on f.id = r.from_id join entities t on t.id = r.to_id where r.from_id = ${id} or r.to_id = ${id} limit 100`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier from source_entities se join sources s on s.id = se.source_id where se.entity_id = ${id}`).then((r) => r.rows),
++      db.execute<{ alias: string }>(sql`select alias from entity_aliases where entity_id = ${id} order by alias`).then((r) => r.rows.map((x) => x.alias)),
++      listEvents({ entity: id, limit: 30 }),
++      db.execute<Record<string, unknown>>(sql`select e.event_type, count(*)::int as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${id} group by 1 order by 2 desc`).then((r) => r.rows),
++    ]);
++    return { entity: e, children, relations, sources, aliases, recent: recent.items, by_type: byType };
++  });
++  app.get<{ Params: { id: string }; Querystring: { limit?: string; cursor?: string } }>("/api/v1/entities/:id/timeline", async (req) => {
++    const id = req.params.id.startsWith("org_") || req.params.id.startsWith("prd_") ? req.params.id : `org_${req.params.id}`;
++    return listEvents({ entity: id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor });
++  });
++
++  // ---- Domains / URLs ----------------------------------------------------------------------
++  app.get<{ Params: { domain: string }; Querystring: { limit?: string; cursor?: string } }>("/api/v1/domains/:domain/timeline", async (req) => {
++    const d = req.params.domain.toLowerCase();
++    const src = (await db.execute<{ id: string }>(sql`select id from sources where domain = ${d} or domain = ${"www." + d} or ${d} = 'www.' || domain limit 1`)).rows[0];
++    const urls = await db.execute<Record<string, unknown>>(sql`select url, status, first_seen_at, last_seen_at, snapshot_count, change_count, sensor_id from urls where domain = ${d} or domain like ${"%." + d} order by change_count desc, last_seen_at desc limit 200`);
++    const ev = src ? await listEvents({ source: src.id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor }) : { items: [], nextCursor: null };
++    return { domain: d, source_id: src?.id ?? null, urls: urls.rows, events: ev.items, nextCursor: ev.nextCursor };
++  });
++  app.get<{ Querystring: { url: string } }>("/api/v1/urls/history", async (req, reply) => {
++    if (!req.query.url) return reply.status(400).send({ error: "url required" });
++    const u = (await db.execute<Record<string, unknown>>(sql`select * from urls where url = ${req.query.url}`)).rows[0];
++    const history = await db.execute<Record<string, unknown>>(sql`
++      select h.id, h.at, h.kind, h.snapshot_id, h.change_id, h.event_id, h.note, e.title as event_title, e.importance, e.event_type, e.slug as event_slug, c.kind as change_kind, c.signal
++      from url_history h left join events e on e.id = h.event_id left join changes c on c.id = h.change_id where h.url = ${req.query.url} order by h.at desc limit 300`);
++    const snaps = await db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_length, canonical_hash, title from snapshots where url = ${req.query.url} or sensor_id = (select sensor_id from urls where url = ${req.query.url}) order by captured_at desc limit 200`);
++    return { url: u ?? { url: req.query.url }, history: history.rows, snapshots: snaps.rows };
++  });
++
++  // ---- Explore / trending / stats ------------------------------------------------------------
++  app.get("/api/v1/stats", async () => stats());
++  app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => ({ items: await trending(Math.min(168, Number(req.query.hours ?? 24)), Math.min(50, Number(req.query.limit ?? 12))) }));
++  app.get("/api/v1/explore", async () => {
++    const [mostActive, biggest, silent, clusters, unusual, byType, byCategory] = await Promise.all([
++      db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, count(*)::int as events_24h, max(e.importance)::float as max_importance from events e join sources s on s.id = e.source_id where e.detected_at >= now() - interval '24 hours' group by s.id, s.name, s.domain order by events_24h desc limit 10`).then((r) => r.rows),
++      listEvents({ limit: 10, order: "importance", after: new Date(Date.now() - 48 * 3600e3).toISOString() }).then((r) => r.items),
++      listEvents({ limit: 10, silent_change: true }).then((r) => r.items),
++      db.execute<Record<string, unknown>>(sql`select c.*, (select json_build_object('id', s.id, 'name', s.name, 'domain', s.domain) from events e join sources s on s.id = e.source_id where e.id = c.primary_event_id) as source from event_clusters c where c.event_count >= 2 and c.last_at >= now() - interval '48 hours' order by c.max_importance desc, c.event_count desc limit 10`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`
++        with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '2 hours' group by s.source_id),
++             base as (select s.source_id, count(*)::float / (14*24) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id)
++        select so.id, so.name, so.domain, cur.n::int as changes_2h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day,
++          round((case when coalesce(base.per_hour,0) = 0 then (case when cur.n/2.0 > 2 then 70 else 40 end) else least(100, case when cur.n/2.0/base.per_hour <= 1 then cur.n/2.0/base.per_hour*30 else 30 + 25*(ln(cur.n/2.0/base.per_hour)/ln(2)) end) end)::numeric, 1)::float as activity_score
++        from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id order by activity_score desc limit 10`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select c as category, count(*)::int as n from events e, unnest(e.categories) c where e.detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows),
++    ]);
++    return { most_active_sources: mostActive, biggest_changes: biggest, silent_changes: silent, clusters, unusual_activity: unusual, by_type: byType, by_category: byCategory, channels: FEED_CHANNELS, event_types: Object.fromEntries(Object.entries(EVENT_TYPES).map(([k, v]) => [k, v.label])) };
++  });
++  app.get<{ Querystring: { limit?: string; since?: string } }>("/api/v1/clusters", async (req) => {
++    const rows = await db.execute<Record<string, unknown>>(sql`
++      select c.*, (select json_agg(json_build_object('id', e.id, 'slug', e.slug, 'title', e.title, 'importance', e.importance, 'event_type', e.event_type, 'detected_at', e.detected_at, 'source_id', e.source_id, 'url', e.url) order by e.importance desc) from events e where e.cluster_id = c.id) as events
++      from event_clusters c where c.last_at >= now() - make_interval(hours => ${Math.min(720, Number(req.query.since ?? 72))}) order by c.last_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
++    return { items: rows.rows };
++  });
++
++  // ---- Search ----------------------------------------------------------------------------
++  app.get<{ Querystring: { q?: string; limit?: string } }>("/api/v1/search", async (req) => {
++    const q = (req.query.q ?? "").trim();
++    if (q.length < 2) return { query: q, events: [], entities: [], sources: [], urls: [] };
++    const lim = Math.min(50, Number(req.query.limit ?? 20));
++    const [ev, ents, srcs, urls] = await Promise.all([
++      listEvents({ q, limit: lim }).then((r) => r.items),
++      db.execute<Record<string, unknown>>(sql`select id, name, type, domain, importance, event_count from entities where search @@ plainto_tsquery('simple', ${q}) or name ilike ${"%" + q + "%"} order by event_count desc, importance desc limit ${lim}`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select id, name, domain, tier, categories from sources where name ilike ${"%" + q + "%"} or domain ilike ${"%" + q + "%"} limit ${lim}`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select url, domain, status, change_count, last_seen_at from urls where url ilike ${"%" + q + "%"} order by change_count desc limit ${lim}`).then((r) => r.rows),
++    ]);
++    return { query: q, events: ev, entities: ents, sources: srcs, urls };
++  });
++
++  // ---- Connector health ----------------------------------------------------------------------
++  app.get("/api/v1/health/connectors", async () => {
++    const [connectors, sensorsByHealth, worst, noisy, daily] = await Promise.all([
++      db.execute<Record<string, unknown>>(sql`select * from connector_health order by connector`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select health, count(*)::int as n from sensors where enabled group by health`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.url, s.source_id, s.connector, s.health, s.consecutive_errors, s.last_error, s.last_status, s.last_check_at from sensors s where s.enabled and s.health <> 'UP' order by s.consecutive_errors desc, s.last_check_at desc limit 50`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.url, s.raw_changes, s.meaningful_changes, case when s.raw_changes > 0 then round(1 - s.meaningful_changes::numeric / s.raw_changes, 3) else null end as noise_ratio from sensors s where s.raw_changes >= 5 order by noise_ratio desc nulls last, raw_changes desc limit 25`).then((r) => r.rows),
++      db.execute<Record<string, unknown>>(sql`select * from metrics_daily order by day desc limit 30`).then((r) => r.rows),
++    ]);
++    return { connectors, sensors_by_health: sensorsByHealth, degraded_sensors: worst, noisy_sensors: noisy, daily, live: liveStats() };
++  });
++
++  // ---- Watchlists & alerts (anonymous owner token, phase 1) ----------------------------------
++  app.get("/api/v1/watchlists", async (req, reply) => {
++    const owner = ownerToken(req.headers as Record<string, unknown>);
++    if (!owner) return reply.status(401).send({ error: "owner token required (X-WebSensor-Owner)" });
++    const rows = await db.execute<Record<string, unknown>>(sql`select w.*, coalesce((select json_agg(json_build_object('kind', i.kind, 'value', i.value, 'added_at', i.added_at)) from watchlist_items i where i.watchlist_id = w.id), '[]'::json) as items from watchlists w where owner_token = ${owner} order by created_at`);
++    return { items: rows.rows };
++  });
++  app.post<{ Body: { name?: string; items?: { kind: string; value: string }[] } }>("/api/v1/watchlists", async (req, reply) => {
++    const owner = ownerToken(req.headers as Record<string, unknown>);
++    if (!owner) return reply.status(401).send({ error: "owner token required" });
++    const body = z.object({ name: z.string().min(1).max(80).default("My watchlist"), items: z.array(z.object({ kind: z.enum(["entity", "source", "keyword", "category", "url"]), value: z.string().min(1).max(200) })).max(200).default([]) }).parse(req.body ?? {});
++    const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from watchlists where owner_token = ${owner}`)).rows[0]?.n;
++    if (Number(count) >= 20) return reply.status(429).send({ error: "too many watchlists" });
++    const id = newId("wl");
++    await db.execute(sql`insert into watchlists (id, owner_token, name) values (${id}, ${owner}, ${body.name})`);
++    for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${id}, ${it.kind}, ${it.value}) on conflict do nothing`);
++    return { id, name: body.name, items: body.items };
++  });
++  app.put<{ Params: { id: string }; Body: { name?: string; items?: { kind: string; value: string }[] } }>("/api/v1/watchlists/:id", async (req, reply) => {
++    const owner = ownerToken(req.headers as Record<string, unknown>);
++    if (!owner) return reply.status(401).send({ error: "owner token required" });
++    const w = (await db.execute<{ id: string }>(sql`select id from watchlists where id = ${req.params.id} and owner_token = ${owner}`)).rows[0];
++    if (!w) return reply.status(404).send({ error: "not_found" });
++    const body = z.object({ name: z.string().min(1).max(80).optional(), items: z.array(z.object({ kind: z.enum(["entity", "source", "keyword", "category", "url"]), value: z.string().min(1).max(200) })).max(200).optional() }).parse(req.body ?? {});
++    if (body.name) await db.execute(sql`update watchlists set name = ${body.name} where id = ${w.id}`);
++    if (body.items) {
++      await db.execute(sql`delete from watchlist_items where watchlist_id = ${w.id}`);
++      for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${w.id}, ${it.kind}, ${it.value}) on conflict do nothing`);
++    }
++    return { ok: true };
++  });
++  app.delete<{ Params: { id: string } }>("/api/v1/watchlists/:id", async (req, reply) => {
++    const owner = ownerToken(req.headers as Record<string, unknown>);
++    if (!owner) return reply.status(401).send({ error: "owner token required" });
++    await db.execute(sql`delete from watchlists where id = ${req.params.id} and owner_token = ${owner}`);
++    return { ok: true };
++  });
++  app.get<{ Params: { id: string }; Querystring: { limit?: string } }>("/api/v1/watchlists/:id/events", async (req, reply) => {
++    const owner = ownerToken(req.headers as Record<string, unknown>);
++    if (!owner) return reply.status(401).send({ error: "owner token required" });
++    const items = (await db.execute<{ kind: string; value: string }>(sql`select i.kind, i.value from watchlist_items i join watchlists w on w.id = i.watchlist_id where w.id = ${req.params.id} and w.owner_token = ${owner}`)).rows;
++    if (!items.length) return reply.send({ items: [] });
++    const ents = items.filter((i) => i.kind === "entity").map((i) => i.value);
++    const srcs = items.filter((i) => i.kind === "source").map((i) => i.value);
++    const cats = items.filter((i) => i.kind === "category").map((i) => i.value);
++    const kws = items.filter((i) => i.kind === "keyword").map((i) => i.value);
++    const conds = [] as ReturnType<typeof sql>[];
++    if (ents.length) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = any(${textArray(ents)}))`);
++    if (srcs.length) conds.push(sql`e.source_id = any(${textArray(srcs)})`);
++    if (cats.length) conds.push(sql`e.categories && ${textArray(cats)}`);
++    for (const k of kws) conds.push(sql`(e.title ilike ${"%" + k + "%"} or e.summary ilike ${"%" + k + "%"})`);
++    if (!conds.length) return { items: [] };
++    const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where ${sql.join(conds, sql` or `)} order by e.detected_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
++    return { items: rows.rows };
++  });
++
++  app.get("/api/v1/alerts", async (req, reply) => {
++    const owner = ownerToken(req.headers as Record<string, unknown>);
++    if (!owner) return reply.status(401).send({ error: "owner token required" });
++    return { items: (await db.execute<Record<string, unknown>>(sql`select * from alerts where owner_token = ${owner} order by created_at`)).rows };
++  });
++  app.post<{ Body: Record<string, unknown> }>("/api/v1/alerts", async (req, reply) => {
++    const owner = ownerToken(req.headers as Record<string, unknown>);
++    if (!owner) return reply.status(401).send({ error: "owner token required" });
++    const body = z.object({ name: z.string().min(1).max(80), rule: z.object({ importance_min: z.number().min(0).max(100).optional(), event_types: z.array(z.string()).optional(), entities: z.array(z.string()).optional(), sources: z.array(z.string()).optional(), keywords: z.array(z.string()).optional(), silent_only: z.boolean().optional(), categories: z.array(z.string()).optional() }), channel: z.enum(["web"]).default("web") }).parse(req.body ?? {});
++    const id = newId("alr");
++    await db.execute(sql`insert into alerts (id, owner_token, name, rule, channel) values (${id}, ${owner}, ${body.name}, ${JSON.stringify(body.rule)}::jsonb, ${body.channel})`);
++    return { id, ...body };
++  });
++  app.delete<{ Params: { id: string } }>("/api/v1/alerts/:id", async (req, reply) => {
++    const owner = ownerToken(req.headers as Record<string, unknown>);
++    if (!owner) return reply.status(401).send({ error: "owner token required" });
++    await db.execute(sql`delete from alerts where id = ${req.params.id} and owner_token = ${owner}`);
++    return { ok: true };
++  });
++
++  // ---- Machine-readable feeds ------------------------------------------------------------------
++  app.get<{ Querystring: { category?: string; importance_min?: string; silent_change?: string } }>("/api/v1/feed.rss", async (req, reply) => {
++    const q = eventsQuery.parse({ ...req.query, limit: 50 });
++    const { items } = await listEvents(q);
++    const esc = (s: unknown): string => String(s ?? "").replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;").replace(/"/g, "&quot;");
++    const base = config.publicBaseUrl;
++    const xml = `<?xml version="1.0" encoding="UTF-8"?>\n<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>WebSensor — ${esc(q.category ? q.category + " events" : "live events")}</title><link>${base}</link><description>Meaningful changes detected on the public Web by WebSensor.</description><atom:link href="${base}/api/v1/feed.rss" rel="self" type="application/rss+xml"/>${items
++      .map((e) => `<item><title>${esc(e.title)}</title><link>${base}/event/${esc(e.slug)}</link><guid isPermaLink="false">${esc(e.id)}</guid><pubDate>${new Date(String(e.detected_at)).toUTCString()}</pubDate><category>${esc(e.event_type)}</category><description>${esc(e.summary)} (importance ${esc(e.importance)}, confidence ${esc(e.confidence)}${e.silent_change ? ", silent change" : ""}) — source: ${esc(e.url)}</description></item>`)
++      .join("")}</channel></rss>`;
++    reply.header("content-type", "application/rss+xml; charset=utf-8");
++    return reply.send(xml);
++  });
++
++  app.get("/api/v1", async () => ({
++    name: "WebSensor API",
++    version: "v1",
++    docs: `${config.publicBaseUrl}/api`,
++    endpoints: ["/api/v1/events", "/api/v1/events/{id|slug}", "/api/v1/changes/{id}", "/api/v1/snapshots/{id}", "/api/v1/snapshots/compare?a=&b=", "/api/v1/sources", "/api/v1/sources/{id}", "/api/v1/sensors/{id}", "/api/v1/entities", "/api/v1/entities/{id}", "/api/v1/entities/{id}/timeline", "/api/v1/domains/{domain}/timeline", "/api/v1/urls/history?url=", "/api/v1/search?q=", "/api/v1/stats", "/api/v1/trending", "/api/v1/explore", "/api/v1/clusters", "/api/v1/health/connectors", "/api/v1/watchlists", "/api/v1/alerts", "/api/v1/feed.rss", "wss://…/api/v1/live"],
++  }));
++}
++
++function renderCanonical(c: { mode: string; text?: string; items?: Record<string, unknown>[]; json?: unknown }): string {
++  if (c.mode === "text") return c.text ?? "";
++  if (c.mode === "list") return (c.items ?? []).map((i) => [i.title ?? i.url ?? i.key, i.url && i.title ? i.url : null, i.summary ? String(i.summary).slice(0, 300) : null].filter(Boolean).join(" — ")).join("\n");
++  return JSON.stringify(c.json ?? null, null, 2);
++}

added apps/api/src/server.ts +107 −0

++import Fastify, { type FastifyReply, type FastifyRequest } from "fastify";
++import cors from "@fastify/cors";
++import rateLimit from "@fastify/rate-limit";
++import websocket from "@fastify/websocket";
++import replyFrom from "@fastify/reply-from";
++import client from "prom-client";
++import { newId } from "@websensor/core";
++import { closeDb } from "@websensor/db";
++import { config } from "./config";
++import { closeLive, registerLive } from "./live";
++import { registerRoutes } from "./routes";
++
++/**
++ * WebSensor gateway: public entry point behind ngrok.
++ *   /api/v1/*        REST + WebSocket (/api/v1/live)
++ *   /api/health /api/ready /api/metrics
++ *   everything else  → Next.js frontend (loopback), with forwarded headers.
++ * Also enforces the canonical host (websensor.io → www.websensor.io).
++ */
++const registry = new client.Registry();
++client.collectDefaultMetrics({ register: registry, prefix: "websensor_api_" });
++const httpRequests = new client.Counter({ name: "websensor_api_requests_total", help: "API requests", labelNames: ["route", "status"], registers: [registry] });
++
++export async function buildServer() {
++  const app = Fastify({
++    logger: {
++      level: config.logLevel,
++      ...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}),
++      redact: ["req.headers.authorization", "req.headers.cookie", "req.headers['x-websensor-owner']"],
++    },
++    trustProxy: true,
++    bodyLimit: 512 * 1024,
++    genReqId: () => newId("req"),
++    disableRequestLogging: config.env === "production",
++  });
++
++  // Canonical host redirect (apex → www) for every path.
++  app.addHook("onRequest", async (req, reply) => {
++    const host = (req.headers.host ?? "").split(":")[0];
++    if (config.redirectApexToWww && host && host !== config.canonicalHost && host === config.canonicalHost.replace(/^www\./, "")) {
++      return reply.redirect(`https://${config.canonicalHost}${req.url}`, 301);
++    }
++  });
++
++  await app.register(cors, { origin: true, methods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"], allowedHeaders: ["content-type", "x-websensor-owner", "authorization"], exposedHeaders: ["x-request-id"] });
++  await app.register(rateLimit, { max: 600, timeWindow: "1 minute", allowList: (req) => !req.url.startsWith("/api/") });
++  await app.register(websocket, { options: { maxPayload: 64 * 1024 } });
++
++  app.addHook("onSend", async (req, reply) => {
++    reply.header("x-request-id", req.id);
++    if (req.url.startsWith("/api/")) {
++      reply.header("cache-control", req.method === "GET" ? "public, max-age=5, stale-while-revalidate=30" : "no-store");
++      httpRequests.inc({ route: req.routeOptions?.url ?? "unknown", status: String(reply.statusCode) });
++    }
++  });
++
++  app.setErrorHandler((err: Error & { statusCode?: number; validation?: unknown; issues?: unknown }, req, reply) => {
++    if (err.name === "ZodError" || err.issues) return reply.status(400).send({ error: "invalid_request", details: err.issues });
++    if (err.statusCode && err.statusCode < 500) return reply.status(err.statusCode).send({ error: err.message });
++    req.log.error({ err }, "unhandled error");
++    return reply.status(500).send({ error: "internal_error", request_id: req.id });
++  });
++
++  app.get("/api/metrics", async (_req, reply) => {
++    reply.header("content-type", registry.contentType);
++    return registry.metrics();
++  });
++
++  await registerRoutes(app);
++  await registerLive(app);
++
++  // Frontend proxy — everything that is not /api/* goes to Next.js (loopback)
++  // with forwarded headers so the app knows the public host/proto and the real client IP.
++  await app.register(replyFrom, { base: config.webUrl, http: { requestOptions: { timeout: 60_000 } }, undici: { connections: 64, pipelining: 1 } });
++  const forward = (req: FastifyRequest, reply: FastifyReply): FastifyReply => {
++    if (req.url.startsWith("/api/")) return reply.status(404).send({ error: "not_found" });
++    return reply.from(req.raw.url ?? "/", {
++      rewriteRequestHeaders: (r, headers) => ({ ...headers, "x-forwarded-host": String(r.headers["x-forwarded-host"] ?? r.headers.host ?? ""), "x-forwarded-proto": String(r.headers["x-forwarded-proto"] ?? "https"), "x-real-ip": r.ip, "x-forwarded-for": r.ip }),
++    });
++  };
++  const methods = ["GET", "HEAD", "POST", "PUT", "DELETE", "PATCH"] as const;
++  app.route({ method: [...methods], url: "/", handler: forward });
++  app.route({ method: [...methods], url: "/*", handler: forward });
++
++  return app;
++}
++
++async function main(): Promise<void> {
++  const app = await buildServer();
++  await app.listen({ port: config.port, host: config.host });
++  app.log.info({ port: config.port, web: config.webUrl, canonical: config.canonicalHost }, "websensor gateway listening");
++  const shutdown = async (): Promise<void> => {
++    await closeLive();
++    await app.close();
++    await closeDb();
++    process.exit(0);
++  };
++  process.on("SIGINT", () => void shutdown());
++  process.on("SIGTERM", () => void shutdown());
++}
++
++if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split("/").pop() ?? "")) {
++  main().catch((e) => {
++    console.error(e);
++    process.exit(1);
++  });
++}

added apps/api/tsconfig.json +5 −0

++{
++  "extends": "../../tsconfig.base.json",
++  "compilerOptions": { "types": ["node"] },
++  "include": ["src/**/*.ts"]
++}

added apps/engine/package.json +35 −0

++{
++  "name": "@websensor/engine",
++  "version": "0.1.0",
++  "private": true,
++  "type": "module",
++  "scripts": {
++    "dev": "tsx watch src/index.ts",
++    "start": "tsx src/index.ts",
++    "registry:sync": "tsx src/cli.ts sync",
++    "discover": "tsx src/cli.ts discover",
++    "run-once": "tsx src/cli.ts run-once",
++    "typecheck": "tsc -p tsconfig.json --noEmit",
++    "test": "vitest run --passWithNoTests"
++  },
++  "dependencies": {
++    "@anthropic-ai/sdk": "^0.124.0",
++    "@websensor/connectors": "workspace:*",
++    "@websensor/core": "workspace:*",
++    "@websensor/db": "workspace:*",
++    "@websensor/store": "workspace:*",
++    "fastify": "^5.4.0",
++    "ioredis": "^5.6.0",
++    "pino": "^9.7.0",
++    "pino-pretty": "^13.0.0",
++    "prom-client": "^15.1.0",
++    "tsx": "^4.20.0",
++    "yaml": "^2.8.0",
++    "zod": "^4.0.0"
++  },
++  "devDependencies": {
++    "@types/node": "^24.0.0",
++    "typescript": "^5.9.3",
++    "vitest": "^3.2.0"
++  }
++}

added apps/engine/src/cli.ts +81 −0

++import { closeDb, db, migrate, sensors, sources, sql } from "@websensor/db";
++import { closeDispatcher, discoverDomain } from "@websensor/connectors";
++import { loadRecent } from "./cluster";
++import { config, log } from "./config";
++import { closeRedis } from "./redis";
++import { runDiscovery, syncRegistry } from "./registry";
++import { runSensor } from "./pipeline";
++import { diffText, evaluateChange } from "@websensor/core";
++import { interpretChange, llmAvailable } from "./interpret";
++
++/**
++ * Operator CLI:
++ *   tsx src/cli.ts sync                      — upsert sources/sensors/entities from config/sources.yaml
++ *   tsx src/cli.ts discover [sourceId…]      — run discovery (+ promote validated endpoints)
++ *   tsx src/cli.ts probe <domain>            — discovery dry-run for a domain (no DB writes)
++ *   tsx src/cli.ts run-once <sensorId>       — run the full pipeline for one sensor now
++ *   tsx src/cli.ts run-due [n]               — run up to n due sensors sequentially
++ */
++async function main(): Promise<void> {
++  const [cmd, ...args] = process.argv.slice(2);
++  await migrate(config.databaseUrl);
++  switch (cmd) {
++    case "sync":
++      await syncRegistry();
++      break;
++    case "discover":
++      await syncRegistry();
++      await runDiscovery({ sourceIds: args.length ? args : undefined });
++      break;
++    case "probe": {
++      const r = await discoverDomain(args[0]!, { probePages: true });
++      for (const e of r) console.log(`${e.type.padEnd(10)} ${e.value.toFixed(2)} ${String(e.itemCount ?? "").padStart(5)} ${e.url}  (${e.evidence})`);
++      break;
++    }
++    case "run-once": {
++      await loadRecent();
++      const s = (await db.select().from(sensors).where(sql`id = ${args[0]}`))[0];
++      if (!s) throw new Error(`sensor ${args[0]} not found`);
++      const src = (await db.select().from(sources).where(sql`id = ${s.sourceId}`))[0]!;
++      const out = await runSensor(s, src);
++      console.log(`${s.id}: ${out}`);
++      break;
++    }
++    case "run-due": {
++      await loadRecent();
++      const n = Number(args[0] ?? 20);
++      const rows = await db.select().from(sensors).where(sql`enabled and next_check_at <= now()`).orderBy(sql`next_check_at asc`).limit(n);
++      const srcs = new Map((await db.select().from(sources)).map((s) => [s.id, s]));
++      for (const s of rows) {
++        const out = await runSensor(s, srcs.get(s.sourceId)!);
++        console.log(`${s.id.padEnd(50)} ${out}`);
++      }
++      break;
++    }
++    case "llm-test": {
++      if (!llmAvailable()) throw new Error("ANTHROPIC_API_KEY not set");
++      const before = "API Pricing\nInput: $10 / million tokens\nOutput: $30 / million tokens\nBatch API: 50% discount";
++      const after = "API Pricing\nInput: $8 / million tokens\nOutput: $24 / million tokens\nBatch API: 50% discount\nPrompt caching: 90% discount on cached input";
++      const diff = diffText(before, after);
++      const heuristic = evaluateChange(diff, { sensorType: "HTML", url: "https://example-ai.com/pricing", sourceCategories: ["ai"], title: "API Pricing" });
++      const t0 = Date.now();
++      const out = await interpretChange({ sourceName: "Example AI", sourceCategories: ["ai"], url: "https://example-ai.com/pricing", sensorName: "pricing", sensorType: "HTML", heuristic, diff, prelimImportance: Number(args[0] ?? 60), title: "API Pricing" });
++      console.log(JSON.stringify(out, null, 2), `\n${Date.now() - t0} ms`);
++      break;
++    }
++    default:
++      console.error("usage: cli.ts sync | discover [sourceId…] | probe <domain> | run-once <sensorId> | run-due [n]");
++      process.exitCode = 1;
++  }
++}
++
++main()
++  .catch((e) => {
++    log.error({ err: (e as Error).stack ?? String(e) }, "cli failed");
++    process.exitCode = 1;
++  })
++  .finally(async () => {
++    await closeDispatcher();
++    await closeRedis();
++    await closeDb();
++  });

added apps/engine/src/cluster.ts +135 −0

++import { jaccard, newId, shingles } from "@websensor/core";
++import { db, eventClusters, events, gte, sql, textArray } from "@websensor/db";
++
++/**
++ * Novelty + clustering over a rolling in-memory window of recent events (loaded from
++ * Postgres at startup). Similarity = Jaccard over word 3-shingles of title+summary.
++ */
++interface RecentEvent {
++  id: string;
++  clusterId: string | null;
++  sourceId: string;
++  sensorId: string;
++  eventType: string;
++  entityIds: string[];
++  detectedAt: number;
++  sh: Set<string>;
++  importance: number;
++}
++
++const WINDOW_MS = 72 * 3600e3;
++const CLUSTER_WINDOW_MS = 6 * 3600e3;
++let recent: RecentEvent[] = [];
++let loaded = false;
++
++export async function loadRecent(): Promise<void> {
++  const since = new Date(Date.now() - WINDOW_MS);
++  const rows = await db.execute<{ id: string; cluster_id: string | null; source_id: string; sensor_id: string; event_type: string; title: string; summary: string; detected_at: Date; importance: number; entity_ids: string[] | null }>(sql`
++    select e.id, e.cluster_id, e.source_id, e.sensor_id, e.event_type, e.title, e.summary, e.detected_at, e.importance,
++           (select array_agg(entity_id) from event_entities ee where ee.event_id = e.id) as entity_ids
++    from events e where e.detected_at >= ${since} order by e.detected_at desc limit 3000`);
++  recent = rows.rows.map((r) => ({ id: r.id, clusterId: r.cluster_id, sourceId: r.source_id, sensorId: r.sensor_id, eventType: r.event_type, entityIds: r.entity_ids ?? [], detectedAt: new Date(r.detected_at).getTime(), sh: shingles(`${r.title}\n${r.summary}`), importance: r.importance }));
++  loaded = true;
++}
++
++function prune(): void {
++  const cutoff = Date.now() - WINDOW_MS;
++  recent = recent.filter((r) => r.detectedAt >= cutoff);
++}
++
++export interface NoveltyResult {
++  novelty: number;
++  nearest: { id: string; similarity: number } | null;
++  /** number of distinct sources reporting near-identical content */
++  confirmations: number;
++}
++
++export async function assessNovelty(text: string, sourceId: string): Promise<NoveltyResult> {
++  if (!loaded) await loadRecent();
++  prune();
++  const sh = shingles(text);
++  let best = 0;
++  let nearest: RecentEvent | null = null;
++  const confirmingSources = new Set<string>();
++  for (const r of recent) {
++    const s = jaccard(sh, r.sh);
++    if (s > best) {
++      best = s;
++      nearest = r;
++    }
++    if (s >= 0.45 && r.sourceId !== sourceId) confirmingSources.add(r.sourceId);
++  }
++  return { novelty: Math.round((1 - best) * 100), nearest: nearest ? { id: nearest.id, similarity: Math.round(best * 100) / 100 } : null, confirmations: confirmingSources.size };
++}
++
++export interface ClusterDecision {
++  clusterId: string;
++  created: boolean;
++}
++
++/**
++ * Attach to an existing open cluster when the event shares an entity (or the same source)
++ * with a recent event and is textually related, or when it is the same event type on the
++ * same source within 30 minutes (e.g. one launch touching six pages). Otherwise open one.
++ */
++export async function clusterEvent(ev: { id: string; sourceId: string; sensorId: string; eventType: string; entityIds: string[]; detectedAt: Date; title: string; summary: string; importance: number; categories: string[] }): Promise<ClusterDecision> {
++  if (!loaded) await loadRecent();
++  const sh = shingles(`${ev.title}\n${ev.summary}`);
++  const now = ev.detectedAt.getTime();
++  let bestCluster: string | null = null;
++  let bestScore = 0;
++  for (const r of recent) {
++    if (!r.clusterId || now - r.detectedAt > CLUSTER_WINDOW_MS) continue;
++    const sharedEntity = r.entityIds.some((e) => ev.entityIds.includes(e));
++    const sameSource = r.sourceId === ev.sourceId;
++    if (!sharedEntity && !sameSource) continue;
++    const sim = jaccard(sh, r.sh);
++    const closeInTime = now - r.detectedAt < 30 * 60e3;
++    let score = 0;
++    if (sim >= 0.22) score = sim + (sharedEntity ? 0.2 : 0);
++    else if (sameSource && r.eventType === ev.eventType && closeInTime && r.sensorId !== ev.sensorId) score = 0.3;
++    else if (sameSource && closeInTime && sim >= 0.12) score = 0.25;
++    if (score > bestScore) {
++      bestScore = score;
++      bestCluster = r.clusterId;
++    }
++  }
++  let clusterId: string;
++  let created = false;
++  if (bestCluster) {
++    clusterId = bestCluster;
++    await db.execute(sql`update event_clusters set event_count = event_count + 1, last_at = greatest(last_at, ${ev.detectedAt}), max_importance = greatest(max_importance, ${ev.importance}),
++      entity_ids = (select array(select distinct unnest(entity_ids || ${textArray(ev.entityIds)}))),
++      categories = (select array(select distinct unnest(categories || ${textArray(ev.categories)}))),
++      title = case when ${ev.importance} > max_importance then ${ev.title} else title end,
++      primary_event_id = case when ${ev.importance} > max_importance then ${ev.id} else primary_event_id end
++      where id = ${clusterId}`);
++  } else {
++    clusterId = newId("clu");
++    created = true;
++    await db.insert(eventClusters).values({ id: clusterId, title: ev.title, summary: ev.summary, primaryEventId: ev.id, entityIds: ev.entityIds, categories: ev.categories, eventCount: 1, maxImportance: ev.importance, firstAt: ev.detectedAt, lastAt: ev.detectedAt });
++  }
++  recent.unshift({ id: ev.id, clusterId, sourceId: ev.sourceId, sensorId: ev.sensorId, eventType: ev.eventType, entityIds: ev.entityIds, detectedAt: now, sh, importance: ev.importance });
++  if (recent.length > 5000) recent.length = 5000;
++  return { clusterId, created };
++}
++
++/** Recent events of the same source published through announcement-type sensors (for silent-change detection). */
++export function recentAnnouncementSimilarity(sourceId: string, text: string, sinceMs: number): number {
++  const sh = shingles(text);
++  let best = 0;
++  const cutoff = Date.now() - sinceMs;
++  for (const r of recent) {
++    if (r.sourceId !== sourceId || r.detectedAt < cutoff) continue;
++    if (!/announcement|product_launch|model_release|software_release|repository_release|incident|outage|maintenance|security_advisory/.test(r.eventType)) continue;
++    best = Math.max(best, jaccard(sh, r.sh));
++  }
++  return best;
++}
++
++export async function recentClusterCount(sinceMs: number): Promise<number> {
++  const rows = await db.select({ n: sql<number>`count(*)` }).from(eventClusters).where(gte(eventClusters.lastAt, new Date(Date.now() - sinceMs)));
++  return Number(rows[0]?.n ?? 0);
++}
++
++export { events };

added apps/engine/src/config.ts +40 −0

++import pino from "pino";
++
++const env = process.env;
++
++export const config = {
++  env: env.NODE_ENV ?? "development",
++  logLevel: env.LOG_LEVEL ?? "info",
++  databaseUrl: env.DATABASE_URL ?? "postgres://localhost:5432/websensor",
++  redisUrl: env.REDIS_URL ?? "redis://127.0.0.1:6379",
++  sourcesFile: env.WS_SOURCES_FILE ?? "./config/sources.yaml",
++  entitiesFile: env.WS_ENTITIES_FILE ?? "./config/entities.yaml",
++  fetchConcurrency: Number(env.WS_FETCH_CONCURRENCY ?? 16),
++  perHostConcurrency: Number(env.WS_PER_HOST_CONCURRENCY ?? 2),
++  metricsPort: Number(env.ENGINE_METRICS_PORT ?? 8262),
++  metricsHost: env.ENGINE_METRICS_HOST ?? "127.0.0.1",
++  processingVersion: "event-pipeline-v1",
++  /** heuristic signal threshold to promote a raw change to an event */
++  meaningfulSignal: Number(env.WS_MEANINGFUL_SIGNAL ?? 0.32),
++  llm: {
++    apiKey: env.ANTHROPIC_API_KEY ?? "",
++    modelFast: env.WS_LLM_MODEL_FAST ?? "claude-haiku-4-5",
++    modelDeep: env.WS_LLM_MODEL_DEEP ?? "claude-opus-5",
++    dailyCallBudget: Number(env.WS_LLM_DAILY_CALL_BUDGET ?? 600),
++    minImportance: Number(env.WS_LLM_MIN_IMPORTANCE ?? 35),
++    deepMinImportance: Number(env.WS_LLM_DEEP_MIN_IMPORTANCE ?? 78),
++  },
++  discovery: {
++    enabled: (env.WS_DISCOVERY ?? "1") !== "0",
++    /** re-run discovery for a source after this many days */
++    intervalDays: Number(env.WS_DISCOVERY_INTERVAL_DAYS ?? 7),
++  },
++  deletion: { confirmations: Number(env.WS_DELETE_CONFIRMATIONS ?? 3), minSeparationMin: Number(env.WS_DELETE_SEPARATION_MIN ?? 60) },
++  version: "0.1.0",
++};
++
++export const log = pino({
++  level: config.logLevel,
++  ...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}),
++  base: { service: "websensor-engine" },
++});

added apps/engine/src/entities.ts +62 −0

++import { db, entityAliases, entities, eq, sourceEntities, sql, textArray } from "@websensor/db";
++import { log } from "./config";
++
++/**
++ * Entity resolution. The source's own entities are always attached (subject). Then aliases
++ * of all known entities are matched against the event text (word-boundary, case-insensitive,
++ * longest alias first) so products/models/regulators named in the change are linked too.
++ */
++interface AliasIndex {
++  loadedAt: number;
++  byLen: { alias: string; re: RegExp; entityId: string }[];
++  importance: Map<string, number>;
++}
++
++let index: AliasIndex | null = null;
++
++async function loadIndex(): Promise<AliasIndex> {
++  if (index && Date.now() - index.loadedAt < 5 * 60_000) return index;
++  const rows = await db.select({ alias: entityAliases.alias, entityId: entityAliases.entityId }).from(entityAliases);
++  const ents = await db.select({ id: entities.id, importance: entities.importance }).from(entities);
++  const byLen = rows
++    .filter((r) => r.alias.length >= 3 && !/^\d+$/.test(r.alias))
++    .map((r) => ({ alias: r.alias, entityId: r.entityId, re: new RegExp(`(^|[^\\p{L}\\p{N}])${escapeRe(r.alias)}(?=$|[^\\p{L}\\p{N}])`, "iu") }))
++    .sort((a, b) => b.alias.length - a.alias.length);
++  index = { loadedAt: Date.now(), byLen, importance: new Map(ents.map((e) => [e.id, e.importance])) };
++  log.debug({ aliases: byLen.length }, "alias index loaded");
++  return index;
++}
++
++function escapeRe(s: string): string {
++  return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
++}
++
++export function invalidateEntityIndex(): void {
++  index = null;
++}
++
++export async function resolveEntities(input: { sourceId: string; text: string; hints: string[] }): Promise<{ subject: string[]; mentioned: string[]; importance: number }> {
++  const idx = await loadIndex();
++  const subject = (await db.select({ entityId: sourceEntities.entityId }).from(sourceEntities).where(eq(sourceEntities.sourceId, input.sourceId))).map((r) => r.entityId);
++  // Only the organization (not every product) is a default subject; products are attached when named.
++  const orgSubjects = subject.filter((s) => s.startsWith("org_"));
++  const mentioned = new Set<string>();
++  const hay = `${input.text}\n${input.hints.join("\n")}`;
++  const hayLower = hay.toLowerCase();
++  for (const a of idx.byLen) {
++    if (mentioned.size >= 12) break;
++    if (orgSubjects.includes(a.entityId)) continue;
++    if (!hayLower.includes(a.alias)) continue;
++    if (a.re.test(hay)) mentioned.add(a.entityId);
++  }
++  // Products of the source named in the text are subjects too.
++  for (const s of subject) if (!s.startsWith("org_") && mentioned.has(s)) mentioned.delete(s), orgSubjects.push(s);
++  const all = [...orgSubjects, ...mentioned];
++  const importance = all.length ? Math.max(...all.map((e) => idx.importance.get(e) ?? 40)) : 40;
++  return { subject: orgSubjects, mentioned: [...mentioned], importance };
++}
++
++export async function bumpEntityCounters(entityIds: string[], at: Date): Promise<void> {
++  if (!entityIds.length) return;
++  await db.execute(sql`update entities set event_count = event_count + 1, last_event_at = ${at} where id = any(${textArray(entityIds)})`);
++}

added apps/engine/src/index.ts +52 −0

++import { closeDb, migrate } from "@websensor/db";
++import { closeDispatcher } from "@websensor/connectors";
++import { loadRecent } from "./cluster";
++import { config, log } from "./config";
++import { startMetricsServer } from "./metrics";
++import { closeRedis } from "./redis";
++import { runDiscovery, syncRegistry } from "./registry";
++import { pruneOldRuns, rollupConnectorHealth, Scheduler } from "./scheduler";
++
++async function main(): Promise<void> {
++  log.info({ env: config.env, version: config.version, llm: Boolean(config.llm.apiKey) }, "websensor engine starting");
++  const applied = await migrate(config.databaseUrl);
++  if (applied.length) log.info({ applied }, "migrations applied");
++  await syncRegistry();
++  await loadRecent();
++  await startMetricsServer();
++
++  const scheduler = new Scheduler();
++  await scheduler.start();
++
++  const timers: NodeJS.Timeout[] = [];
++  timers.push(setInterval(() => rollupConnectorHealth().catch((e) => log.warn({ err: (e as Error).message }, "health rollup failed")), 60_000));
++  timers.push(setInterval(() => pruneOldRuns().catch(() => undefined), 6 * 3600e3));
++  await rollupConnectorHealth().catch(() => undefined);
++
++  if (config.discovery.enabled) {
++    // Background: validate feeds/sitemaps/status pages for sources not discovered recently, then weekly.
++    setTimeout(() => runDiscovery({ onlyMissing: true }).catch((e) => log.warn({ err: (e as Error).message }, "discovery failed")), 5_000);
++    timers.push(setInterval(() => runDiscovery({ onlyMissing: true }).catch(() => undefined), 24 * 3600e3));
++  }
++
++  let shuttingDown = false;
++  const shutdown = async (signal: string): Promise<void> => {
++    if (shuttingDown) return;
++    shuttingDown = true;
++    log.info({ signal }, "shutting down");
++    for (const t of timers) clearInterval(t);
++    await scheduler.stop();
++    await closeDispatcher();
++    await closeRedis();
++    await closeDb();
++    process.exit(0);
++  };
++  process.on("SIGINT", () => void shutdown("SIGINT"));
++  process.on("SIGTERM", () => void shutdown("SIGTERM"));
++  process.on("unhandledRejection", (e) => log.error({ err: e instanceof Error ? e.stack : String(e) }, "unhandled rejection"));
++}
++
++main().catch((e) => {
++  log.fatal({ err: (e as Error).stack ?? String(e) }, "engine failed to start");
++  process.exit(1);
++});

added apps/engine/src/interpret.ts +153 −0

++import Anthropic from "@anthropic-ai/sdk";
++import { EVENT_TYPES, type DiffResult, type HeuristicResult } from "@websensor/core";
++import { db, llmUsage, sql } from "@websensor/db";
++import { config, log } from "./config";
++import { bumpDaily, m } from "./metrics";
++
++/**
++ * Stage-2 interpretation with Claude. Only called for candidates that already cleared the
++ * heuristic bar and a preliminary importance threshold; a daily call budget caps spend.
++ * Routine changes go to the fast model; high-importance candidates get the deep model.
++ * Output is strictly structured (JSON schema) — never free text.
++ */
++export interface Interpretation {
++  event_type: string;
++  title: string;
++  summary: string;
++  why_it_matters: string;
++  who_it_affects: string;
++  observed: string;
++  inferred: string;
++  meaningful: boolean;
++  severity: number;
++  confidence: number;
++  entities: string[];
++  keywords: string[];
++  announced: boolean | null;
++  model: string;
++  input_tokens: number;
++  output_tokens: number;
++}
++
++const client = config.llm.apiKey ? new Anthropic({ apiKey: config.llm.apiKey, timeout: 60_000, maxRetries: 1 }) : null;
++
++const SYSTEM = `You are the interpretation stage of WebSensor, a platform that monitors official public web sources and turns detected changes into precise, sober intelligence events.
++You receive a detected change on one monitored endpoint: the source, the URL, the sensor kind, the heuristic pre-classification, extracted facts and the diff (before → after, or new/removed items).
++Your job: decide whether the change is meaningful to a professional audience, classify it, and write a factual title and summary.
++Rules:
++- Never sensationalize. No marketing tone. State what is observed; keep inference clearly separate.
++- "observed" must only contain facts visible in the diff. "inferred" may contain a cautious interpretation or be empty.
++- Titles: ≤ 110 characters, start with the organization name, no trailing period, no emoji.
++- Summary: 1–3 sentences, concrete (numbers, names, versions when present).
++- meaningful=false for cosmetic edits, typos, navigation/footer churn, copyright years, tracking or timestamp noise, and generic marketing rewording with no new information.
++- severity: 0–100 intrinsic importance of this kind of change for people who follow this organization (pricing/security/outage/model launch high; doc typo low).
++- confidence: 0–100 how sure you are of the classification given the evidence.
++- entities: organizations, products, models, APIs, drugs, standards explicitly named in the change (canonical names).
++- announced: true if the diff itself is an announcement (news/blog/release item), false if it is a silent modification of an existing page, null if unclear.
++Allowed event_type values: ${Object.keys(EVENT_TYPES).join(", ")}.`;
++
++const schema = {
++  type: "object",
++  additionalProperties: false,
++  properties: {
++    event_type: { type: "string", enum: Object.keys(EVENT_TYPES) },
++    title: { type: "string" },
++    summary: { type: "string" },
++    why_it_matters: { type: "string" },
++    who_it_affects: { type: "string" },
++    observed: { type: "string" },
++    inferred: { type: "string" },
++    meaningful: { type: "boolean" },
++    severity: { type: "integer", description: "0–100" },
++    confidence: { type: "integer", description: "0–100" },
++    entities: { type: "array", items: { type: "string" } },
++    keywords: { type: "array", items: { type: "string" } },
++    announced: { type: ["boolean", "null"] },
++  },
++  required: ["event_type", "title", "summary", "why_it_matters", "who_it_affects", "observed", "inferred", "meaningful", "severity", "confidence", "entities", "keywords", "announced"],
++} as const;
++
++let budgetDay = "";
++let budgetUsed = 0;
++
++async function loadBudget(): Promise<void> {
++  const today = new Date().toISOString().slice(0, 10);
++  if (budgetDay === today) return;
++  budgetDay = today;
++  const r = await db.execute<{ n: string }>(sql`select count(*)::text as n from llm_usage where at >= current_date`);
++  budgetUsed = Number(r.rows[0]?.n ?? 0);
++}
++
++export function llmAvailable(): boolean {
++  return client !== null;
++}
++
++export async function interpretChange(input: { sourceName: string; sourceCategories: string[]; url: string; sensorName: string; sensorType: string; heuristic: HeuristicResult; diff: DiffResult; prelimImportance: number; title?: string | null }): Promise<Interpretation | null> {
++  if (!client) return null;
++  await loadBudget();
++  if (budgetUsed >= config.llm.dailyCallBudget) {
++    log.warn({ used: budgetUsed }, "LLM daily budget exhausted — heuristics only");
++    return null;
++  }
++  const deep = input.prelimImportance >= config.llm.deepMinImportance;
++  const model = deep ? config.llm.modelDeep : config.llm.modelFast;
++  const diffText = renderDiff(input.diff).slice(0, 14_000);
++  const user = `SOURCE: ${input.sourceName} (${input.sourceCategories.join(", ") || "n/a"})
++URL: ${input.url}
++SENSOR: ${input.sensorName} [${input.sensorType}]${input.title ? `\nPAGE TITLE: ${input.title}` : ""}
++HEURISTIC: type=${input.heuristic.eventType} signal=${input.heuristic.signal.toFixed(2)} magnitude=${input.heuristic.magnitude} noise=${input.heuristic.noiseRatio.toFixed(2)} reasons=${input.heuristic.reasons.join("; ") || "none"}
++FACTS: ${input.heuristic.facts.map((f) => `${f.kind}: ${f.before ?? "∅"} → ${f.after ?? "∅"}`).join(" | ") || "none"}
++
++DIFF:
++${diffText}`;
++
++  budgetUsed++;
++  const started = Date.now();
++  try {
++    // Haiku 4.5 does not accept `effort`; the deep model (Opus 5) runs adaptive thinking at medium effort.
++    const params: Anthropic.MessageCreateParamsNonStreaming = {
++      model,
++      max_tokens: 1500,
++      system: [{ type: "text", text: SYSTEM, cache_control: { type: "ephemeral" } }],
++      messages: [{ role: "user", content: user }],
++      output_config: { format: { type: "json_schema", schema: schema as unknown as Record<string, unknown> }, ...(deep ? { effort: "medium" as const } : {}) },
++    };
++    const res = await client.messages.create(params);
++    const text = res.content.find((b) => b.type === "text")?.text ?? "";
++    if (res.stop_reason === "refusal" || !text) throw new Error(`no text (stop_reason=${res.stop_reason})`);
++    const parsed = JSON.parse(text) as Omit<Interpretation, "model" | "input_tokens" | "output_tokens">;
++    const usage = { input: res.usage.input_tokens + (res.usage.cache_read_input_tokens ?? 0) + (res.usage.cache_creation_input_tokens ?? 0), output: res.usage.output_tokens };
++    m.llmCalls.inc({ model, ok: "true" });
++    m.llmTokens.inc({ model, direction: "input" }, usage.input);
++    m.llmTokens.inc({ model, direction: "output" }, usage.output);
++    await db.insert(llmUsage).values({ model, purpose: "interpret", inputTokens: usage.input, outputTokens: usage.output, ok: true });
++    await bumpDaily({ llm_calls: 1, llm_input_tokens: usage.input, llm_output_tokens: usage.output });
++    log.debug({ model, ms: Date.now() - started, type: parsed.event_type, meaningful: parsed.meaningful }, "llm interpretation");
++    if (!(parsed.event_type in EVENT_TYPES)) parsed.event_type = input.heuristic.eventType;
++    parsed.severity = Math.max(0, Math.min(100, Math.round(Number(parsed.severity) || 0)));
++    parsed.confidence = Math.max(0, Math.min(100, Math.round(Number(parsed.confidence) || 0)));
++    return { ...parsed, model, input_tokens: usage.input, output_tokens: usage.output };
++  } catch (e) {
++    m.llmCalls.inc({ model, ok: "false" });
++    await db.insert(llmUsage).values({ model, purpose: "interpret", inputTokens: 0, outputTokens: 0, ok: false }).catch(() => undefined);
++    const err = e as Error & { status?: number };
++    log.warn({ model, status: err.status, err: err.message }, "llm interpretation failed — falling back to heuristics");
++    return null;
++  }
++}
++
++export function renderDiff(d: DiffResult): string {
++  if (d.kind === "text") {
++    const lines: string[] = [];
++    for (const mo of d.modified.slice(0, 60)) lines.push(`- ${mo.before}\n+ ${mo.after}`);
++    for (const r of d.removed.slice(0, 60)) lines.push(`- ${r}`);
++    for (const a of d.added.slice(0, 80)) lines.push(`+ ${a}`);
++    return lines.join("\n");
++  }
++  if (d.kind === "json") return d.unified;
++  const lines: string[] = [];
++  for (const a of d.added.slice(0, 30)) lines.push(`+ NEW: ${String(a.title ?? a.url ?? a.key)}${a.summary ? `\n    ${String(a.summary).slice(0, 500)}` : ""}${a.url ? `\n    ${String(a.url)}` : ""}${a.publishedAt ? `\n    published ${String(a.publishedAt)}` : ""}`);
++  for (const r of d.removed.slice(0, 30)) lines.push(`- REMOVED: ${String(r.title ?? r.url ?? r.key)}`);
++  for (const mo of d.modified.slice(0, 30)) lines.push(`~ UPDATED: ${String(mo.after.title ?? mo.key)} (${mo.fields.join(", ")})\n    before: ${mo.fields.map((f) => `${f}=${JSON.stringify(mo.before[f])}`).join(" ")}\n    after:  ${mo.fields.map((f) => `${f}=${JSON.stringify(mo.after[f])}`).join(" ")}`);
++  return lines.join("\n");
++}

added apps/engine/src/metrics.ts +46 −0

++import Fastify from "fastify";
++import client from "prom-client";
++import { db, sql } from "@websensor/db";
++import { config, log } from "./config";
++
++export const registry = new client.Registry();
++client.collectDefaultMetrics({ register: registry, prefix: "websensor_engine_" });
++
++export const m = {
++  checks: new client.Counter({ name: "websensor_checks_total", help: "Sensor checks", labelNames: ["connector", "outcome"], registers: [registry] }),
++  bytes: new client.Counter({ name: "websensor_fetched_bytes_total", help: "Bytes fetched", labelNames: ["connector"], registers: [registry] }),
++  changes: new client.Counter({ name: "websensor_changes_total", help: "Raw changes", labelNames: ["connector", "meaningful"], registers: [registry] }),
++  events: new client.Counter({ name: "websensor_events_total", help: "Events published", labelNames: ["event_type", "silent"], registers: [registry] }),
++  llmCalls: new client.Counter({ name: "websensor_llm_calls_total", help: "LLM calls", labelNames: ["model", "ok"], registers: [registry] }),
++  llmTokens: new client.Counter({ name: "websensor_llm_tokens_total", help: "LLM tokens", labelNames: ["model", "direction"], registers: [registry] }),
++  fetchDuration: new client.Histogram({ name: "websensor_fetch_duration_seconds", help: "Fetch latency", labelNames: ["connector"], buckets: [0.1, 0.25, 0.5, 1, 2, 5, 10, 25], registers: [registry] }),
++  processingLatency: new client.Histogram({ name: "websensor_processing_latency_seconds", help: "Detected → published", buckets: [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 15], registers: [registry] }),
++  queueDue: new client.Gauge({ name: "websensor_sensors_due", help: "Sensors due for a check", registers: [registry] }),
++  inflight: new client.Gauge({ name: "websensor_inflight_checks", help: "Checks in flight", registers: [registry] }),
++  httpStatus: new client.Counter({ name: "websensor_http_status_total", help: "HTTP status codes", labelNames: ["status"], registers: [registry] }),
++};
++
++/** Daily counters in Postgres (feed the public homepage stats). */
++export async function bumpDaily(fields: Partial<Record<"checks" | "not_modified" | "bytes" | "raw_changes" | "events" | "silent_events" | "errors" | "llm_calls" | "llm_input_tokens" | "llm_output_tokens" | "scrapfly_calls", number>>): Promise<void> {
++  const keys = Object.keys(fields) as (keyof typeof fields)[];
++  if (!keys.length) return;
++  const sets = keys.map((k) => sql.raw(`${k} = metrics_daily.${k} + ${Number(fields[k] ?? 0)}`));
++  const cols = keys.map((k) => sql.raw(k));
++  const vals = keys.map((k) => sql`${Number(fields[k] ?? 0)}`);
++  try {
++    await db.execute(sql`insert into metrics_daily (day, ${sql.join(cols, sql`, `)}) values (current_date, ${sql.join(vals, sql`, `)}) on conflict (day) do update set ${sql.join(sets, sql`, `)}`);
++  } catch (e) {
++    log.warn({ err: (e as Error).message }, "metrics_daily update failed");
++  }
++}
++
++export async function startMetricsServer(): Promise<void> {
++  const app = Fastify({ logger: false });
++  app.get("/metrics", async (_req, reply) => {
++    reply.header("content-type", registry.contentType);
++    return registry.metrics();
++  });
++  app.get("/health", async () => ({ status: "ok", service: "engine", version: config.version, time: new Date().toISOString() }));
++  await app.listen({ port: config.metricsPort, host: config.metricsHost });
++  log.info({ port: config.metricsPort }, "engine metrics listening");
++}

added apps/engine/src/pipeline.ts +404 −0

Large diff (404 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/redis.ts +41 −0

Large diff (41 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/registry.ts +179 −0

Large diff (179 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/scheduler.ts +162 −0

Large diff (162 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/README.md +18 −0

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/eslint.config.mjs +6 −0

Large diff (6 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/next.config.ts +23 −0

Large diff (23 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/package.json +32 −0

Large diff (32 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/postcss.config.mjs +2 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/alerts/alerts.tsx +186 −0

Large diff (186 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/alerts/page.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/api/page.tsx +131 −0

Large diff (131 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/bot/page.tsx +36 −0

Large diff (36 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/breaking/page.tsx +17 −0

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/category/[channel]/page.tsx +29 −0

Large diff (29 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/company/[id]/page.tsx +111 −0

Large diff (111 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/compare/page.tsx +24 −0

Large diff (24 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/domain/[domain]/page.tsx +51 −0

Large diff (51 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/entities/page.tsx +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/error.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/event/[slug]/page.tsx +254 −0

Large diff (254 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/explore/page.tsx +112 −0

Large diff (112 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/globals.css +161 −0

Large diff (161 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/health/page.tsx +107 −0

Large diff (107 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/icon.svg +1 −0

Large diff (1 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/layout.tsx +39 −0

Large diff (39 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/manifest.ts +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/not-found.tsx +15 −0

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/opengraph-image.tsx +33 −0

Large diff (33 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/page.tsx +28 −0

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/robots.ts +10 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/search/page.tsx +74 −0

Large diff (74 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/sensor/[id]/page.tsx +105 −0

Large diff (105 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/silent/page.tsx +17 −0

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/sitemap.ts +19 −0

Large diff (19 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/source/[id]/page.tsx +134 −0

Large diff (134 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/sources/page.tsx +53 −0

Large diff (53 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/timeline/[id]/page.tsx +33 −0

Large diff (33 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/url/compare-form.tsx +35 −0

Large diff (35 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/url/page.tsx +72 −0

Large diff (72 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/watchlists/page.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/watchlists/watchlists.tsx +154 −0

Large diff (154 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/code.tsx +27 −0

Large diff (27 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/diff-viewer.tsx +165 −0

Large diff (165 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/event-row.tsx +48 −0

Large diff (48 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/footer.tsx +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/live-feed.tsx +135 −0

Large diff (135 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/nav.tsx +105 −0

Large diff (105 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/rail.tsx +54 −0

Large diff (54 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/stats-strip.tsx +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/theme.tsx +30 −0

Large diff (30 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/timeline.tsx +38 −0

Large diff (38 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/ui.tsx +167 −0

Large diff (167 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/watch-button.tsx +54 −0

Large diff (54 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/api.ts +491 −0

Large diff (491 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/format.ts +143 −0

Large diff (143 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/owner.ts +37 −0

Large diff (37 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/use-live.ts +80 −0

Large diff (80 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/tsconfig.json +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added config/sources.yaml +2088 −0

Large diff (2,088 lines) is not loaded by default to keep the page fast.Load diff

added deploy/README.md +44 −0

Large diff (44 lines) is not loaded by default to keep the page fast.Load diff

added deploy/websensor.mld.json.example +128 −0

Large diff (128 lines) is not loaded by default to keep the page fast.Load diff

added docs/ARCHITECTURE.md +49 −0

Large diff (49 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/discovery.md +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/github.md +10 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/http.md +16 −0

Large diff (16 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/jsonlist.md +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/rss.md +15 −0

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/scrapfly.md +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/sitemap.md +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/statuspage.md +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added package.json +31 −0

Large diff (31 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/package.json +23 −0

Large diff (23 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/connectors.test.ts +89 −0

Large diff (89 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/discovery.ts +150 −0

Large diff (150 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/fetcher.ts +238 −0

Large diff (238 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/github.ts +64 −0

Large diff (64 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/http.ts +100 −0

Large diff (100 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/index.ts +38 −0

Large diff (38 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/jsonlist.ts +89 −0

Large diff (89 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/rss.ts +161 −0

Large diff (161 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/scrapfly.ts +68 −0

Large diff (68 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/sitemap.ts +106 −0

Large diff (106 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/statuspage.ts +69 −0

Large diff (69 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/types.ts +28 −0

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/xml.ts +65 −0

Large diff (65 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/package.json +26 −0

Large diff (26 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/canonical.ts +196 −0

Large diff (196 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/client.ts +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/core.test.ts +131 −0

Large diff (131 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/diff.ts +176 −0

Large diff (176 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/hash.ts +65 −0

Large diff (65 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/heuristics.ts +242 −0

Large diff (242 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/ids.ts +36 −0

Large diff (36 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/index.ts +10 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/schedule.ts +53 −0

Large diff (53 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/scoring.ts +116 −0

Large diff (116 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/ssrf.ts +189 −0

Large diff (189 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/taxonomy.ts +185 −0

Large diff (185 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/types.ts +102 −0

Large diff (102 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/migrations/0001_init.sql +361 −0

Large diff (361 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/migrations/0002_scrapfly.sql +1 −0

Large diff (1 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/migrations/0003_latency_bigint.sql +2 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/package.json +31 −0

Large diff (31 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/src/index.ts +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/src/migrate.ts +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/src/schema.ts +342 −0

Large diff (342 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/store/package.json +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added packages/store/src/index.ts +69 −0

Large diff (69 lines) is not loaded by default to keep the page fast.Load diff

added packages/store/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added pnpm-lock.yaml +6482 −0

Large diff (6,482 lines) is not loaded by default to keep the page fast.Load diff

added pnpm-workspace.yaml +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/atom.xml +4 −0

Large diff (4 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/kev.json +3 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/rss-after.xml +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/rss-before.xml +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/sitemap.xml +6 −0

Large diff (6 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/statuspage.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added tsconfig.base.json +17 −0

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff

spb/websensor

WebSensor v0.1.0 — real-time web intelligence platform

Theme