spb/websensor

Core (taxonomy, SSRF policy, canonical extraction, diff engines, heuristics, scoring, adaptive schedule),
db (SQL migrations + Drizzle), content-addressed zstd blob store, connectors (http, rss/atom, sitemap,
statuspage, github, jsonlist, discovery, scrapfly fallback), engine (scheduler, pipeline, Claude
interpretation, entities, novelty/clustering, silent changes, metrics), Fastify gateway (REST + WebSocket +
proxy), Next.js 16 frontend, 200-source registry, tests, docs, mld manifest.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

added .env.example +30 −0

		@@ -0,0 +1,30 @@
	1	+# WebSensor — environment (never commit real values)
	2	+NODE_ENV=development
	3	+DATABASE_URL=postgres://localhost:5432/websensor
	4	+REDIS_URL=redis://127.0.0.1:6379
	5	+
	6	+# Content-addressed blob store (fs driver). S3/MinIO driver can be plugged in later.
	7	+BLOB_STORE_DIR=./data/blobs
	8	+
	9	+PUBLIC_BASE_URL=http://localhost:8260
	10	+API_PORT=8260
	11	+API_HOST=127.0.0.1
	12	+WEB_PORT=8261
	13	+WEB_URL=http://127.0.0.1:8261
	14	+ENGINE_METRICS_PORT=8262
	15	+
	16	+# Optional acquisition fallbacks
	17	+FIRECRAWL_API_KEY=
	18	+SCRAPFLY_API_KEY=
	19	+
	20	+# AI interpretation (heuristics run without any key)
	21	+ANTHROPIC_API_KEY=
	22	+WS_LLM_MODEL_FAST=claude-haiku-4-5
	23	+WS_LLM_MODEL_DEEP=claude-opus-5
	24	+WS_LLM_DAILY_CALL_BUDGET=600
	25	+WS_LLM_MIN_IMPORTANCE=35
	26	+
	27	+# Engine tuning
	28	+WS_FETCH_CONCURRENCY=16
	29	+WS_USER_AGENT=WebSensorBot/0.1 (+https://www.websensor.io/bot)
	30	+WS_SOURCES_FILE=./config/sources.yaml

added .gitignore +17 −0

		@@ -0,0 +1,17 @@
	1	+node_modules/
	2	+.next/
	3	+dist/
	4	+*.tsbuildinfo
	5	+.env
	6	+.env.*
	7	+!.env.example
	8	+logs/
	9	+tmp/
	10	+coverage/
	11	+.DS_Store
	12	+.claude/
	13	+deploy/*.mld.json
	14	+data/
	15	+qa/test-results/
	16	+qa/.e2e-*
	17	+apps/web/next-env.d.ts

added CLAUDE.md +76 −0

		@@ -0,0 +1,76 @@
	1	+# WebSensor — repository guide
	2	+
	3	+WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing
	4	+Web. It monitors official public sources, detects meaningful changes, classifies them, links entities, scores
	5	+importance, preserves evidence and publishes events through a WebSocket feed. The product brief that drove the
	6	+design is `docs/PRODUCT-BRIEF.md`; this file is the working guide for the code.
	7	+
	8	+## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)
	9	+- `packages/core` — taxonomy (event types + intrinsic severity, tiers, categories), ids, SSRF policy
	10	+ (`assertUrlAllowed`, `safeLookup`), hashing (sha256, simhash, shingles/Jaccard), canonical extraction
	11	+ (`canonicalizeHtml`: strips scripts/nav/footer/cookie chrome, timestamps, tokens, counters), diff engines
	12	+ (text / json / keyed list), stage-1 heuristics (`evaluateChange`, `describeChange`), scoring
	13	+ (importance components, confidence, novelty helpers, trending, activity anomaly), adaptive schedule.
	14	+ `@websensor/core/client` is the browser-safe subset (no `node:` imports) — client components must import it.
	15	+- `packages/db` — plain SQL migrations (`migrations/*.sql`, applied by `migrate()` with an advisory lock) +
	16	+ Drizzle schema for typed access. **`textArray()`** must be used for `= any(...)` / `&&` with JS arrays
	17	+ (Drizzle spreads arrays into parameter lists).
	18	+- `packages/store` — content-addressed blob store (`sha256/ab/cd/<hash>.zst`, zstd via `node:zlib`). Raw
	19	+ bodies, canonical representations and full diffs live there, never in Postgres. Interface ready for S3/MinIO.
	20	+- `packages/connectors` — connector SDK (`WebSensorConnector.fetch/normalize`), safe fetcher (conditional GET,
	21	+ manual redirects validated per hop, HTTP/2→1.1 pin on NGHTTP2 errors, browser-UA second attempt on
	22	+ 403/resets, size + time limits), connectors: `http` (HTML/JSON/text/HEAD), `rss` (RSS/Atom/RDF/JSON Feed),
	23	+ `sitemap` (index + news), `statuspage` (Atlassian v2 summary), `github` (releases/tags/commits Atom,
	24	+ advisories REST), `jsonlist` (keyed records from any JSON API; `{now-2h}` placeholders), `discovery`
	25	+ (robots sitemaps, `<link rel=alternate>`, well-known feed paths, linked status pages — every candidate is
	26	+ fetched and parsed), `scrapfly` (fallback, budgeted, only when `fallback.scrapfly` is set on the source).
	27	+- `apps/engine` — scheduler (`FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency), pipeline
	28	+ (fetch → normalize → snapshot → diff → heuristics → change → event: entities, novelty, LLM interpretation,
	29	+ clustering, importance/confidence, silent-change detection, publish to Redis), registry sync from
	30	+ `config/sources.yaml`, discovery, connector health rollups, Prometheus metrics on :8262. CLI: `src/cli.ts`
	31	+ (`sync`, `discover`, `probe <domain>`, `run-once <sensor>`, `run-due [n]`).
	32	+- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*`, WebSocket `/api/v1/live` (Redis pub/sub fan-out,
	33	+ channels `events:*`, `entity:*`, `source:*`, `watchlist:*`), `/api/v1/feed.rss`, `/api/health` `/api/ready` `/api/metrics`, apex→www
	34	+ redirect, and a reverse proxy to the Next.js app for everything else.
	35	+- `apps/web` — Next.js 16 frontend (:8261, loopback). Pages: live, breaking, explore, sources, entities,
	36	+ timelines, silent changes, watchlists, alerts, event detail with diff viewer, health, API docs.
	37	+- `config/sources.yaml` — the 200-organization registry (sources, curated sensors, product entities, discovery
	38	+ flags, fallbacks). `docs/connectors/*.md` document each connector family.
	39	+
	40	+## Rules
	41	+- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets — goes through
	42	+ `assertUrlAllowed()`. Private ranges, metadata endpoints, `.maclustr.io`/`.ts.net` and single-label hosts are
	43	+ blocked; the dispatcher's DNS lookup only returns approved addresses.
	44	+- Raw evidence is immutable: snapshots and diffs are never rewritten. Re-interpretation creates a new row in
	45	+ `interpretations` (versioned); `events.interpretation` holds the latest.
	46	+- LLM cost control: heuristics first; Claude is called only for candidates above `WS_LLM_MIN_IMPORTANCE`, with a
	47	+ daily call budget; the deep model only above `WS_LLM_DEEP_MIN_IMPORTANCE`. Output is strict JSON
	48	+ (`output_config.format`), never free text. No key → heuristics only, the system still works.
	49	+- Feeds: items are "new" only if their key was never seen (sensor `state.seenKeys`) and they are not older than
	50	+ 14 days; items scrolling out of the window are never "removed". Sitemaps/statuspages: a >50 % shrink is
	51	+ treated as a partial response, not mass deletion. A 404 becomes `page_removed` only after
	52	+ `WS_DELETE_CONFIRMATIONS` checks separated by `WS_DELETE_SEPARATION_MIN`.
	53	+- Never label inference as fact: events carry `evidence_label` (OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED)
	54	+ and the interpretation keeps `observed` and `inferred` apart. Silent changes are flagged, never asserted as
	55	+ "unannounced" without checking recent announcement-type events of the same source.
	56	+- Do not add sensors that are blocked with 403 by design (Akamai/Cloudflare bot management) unless the source
	57	+ has `fallback.scrapfly: true` and a tier ≥ C; Scrapfly is a budgeted fallback, not the foundation.
	58	+- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`…).
	59	+
	60	+## Dev
	61	+```
	62	+createdb websensor && cp .env.example .env
	63	+pnpm install
	64	+pnpm db:migrate # or let the engine migrate on start
	65	+npx tsx apps/engine/src/cli.ts sync # registry → DB
	66	+npx tsx apps/engine/src/cli.ts run-due 50
	67	+pnpm dev:api · pnpm dev:engine · pnpm dev:web
	68	+pnpm test · pnpm typecheck
	69	+```
	70	+Run CLI/engine from the repo root (config paths are relative to cwd).
	71	+
	72	+## Deploy (MacLustr)
	73	+`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `deploy/websensor.mld.json` (secrets only
	74	+on M1M32). Processes: `websensor-api` (:8260, ngrok www.websensor.io), `websensor-web` (:8261 loopback),
	75	+`websensor-engine` (metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`.
	76	+See `deploy/README.md`.

added README.md +33 −0

		@@ -0,0 +1,33 @@
	1	+# WebSensor
	2	+
	3	+**Detect What Changed. Know Why It Matters.** — https://www.websensor.io
	4	+
	5	+WebSensor is a global sensor network for the changing Web. It continuously monitors official public sources
	6	+(newsrooms, status pages, changelogs, regulators, registries, APIs), preserves every observation, detects raw
	7	+changes, filters noise, interprets meaningful changes into events, links them to entities, scores their
	8	+importance and confidence, clusters related events and publishes them live over WebSocket.
	9	+
	10	+```
	11	+Sources → Sensors/Connectors → Fetch (conditional) → Snapshot (immutable) → Diff → Heuristics
	12	+ → Interpretation (Claude, budgeted) → Entities → Novelty/Clustering → Importance → Event
	13	+ → Redis stream → WebSocket gateway → www.websensor.io
	14	+```
	15	+
	16	+- 200 organizations, ~290 curated sensors at launch (`config/sources.yaml`), plus validated discovery.
	17	+- Connector families: HTTP (HTML/JSON/HEAD), RSS/Atom/JSON Feed, sitemaps, Statuspage, GitHub, JSON APIs
	18	+ (CISA KEV, NVD, Federal Register, USGS, HIBP, Google Cloud incidents…), Scrapfly fallback.
	19	+- Every event is auditable: event → change → snapshot A/B → fetch run → sensor → source, with raw evidence.
	20	+- Observed vs inferred is always labelled; silent (unannounced) changes are flagged.
	21	+
	22	+## Quick start
	23	+```
	24	+createdb websensor && cp .env.example .env && pnpm install
	25	+npx tsx apps/engine/src/cli.ts sync # load the registry
	26	+pnpm dev:api & pnpm dev:web & pnpm dev:engine
	27	+open http://localhost:8260
	28	+```
	29	+
	30	+Docs: `CLAUDE.md` (repo guide), `docs/ARCHITECTURE.md`, `docs/connectors/*.md`, `deploy/README.md`,
	31	+API docs at `/api` on the site. Public API: `GET /api/v1/events`, `/api/v1/entities/{id}`,
	32	+`/api/v1/sources/{id}`, `/api/v1/domains/{domain}/timeline`, `/api/v1/changes/{id}`, `wss://…/api/v1/live`,
	33	+`/api/v1/feed.rss`.

added apps/api/package.json +35 −0

		@@ -0,0 +1,35 @@
	1	+{
	2	+ "name": "@websensor/api",
	3	+ "version": "0.1.0",
	4	+ "private": true,
	5	+ "type": "module",
	6	+ "scripts": {
	7	+ "dev": "tsx watch src/server.ts",
	8	+ "start": "tsx src/server.ts",
	9	+ "typecheck": "tsc -p tsconfig.json --noEmit",
	10	+ "test": "vitest run --passWithNoTests"
	11	+ },
	12	+ "dependencies": {
	13	+ "@fastify/cors": "^11.0.0",
	14	+ "@fastify/rate-limit": "^10.3.0",
	15	+ "@fastify/reply-from": "^12.6.5",
	16	+ "@fastify/websocket": "^11.2.0",
	17	+ "@websensor/core": "workspace:*",
	18	+ "@websensor/db": "workspace:*",
	19	+ "@websensor/store": "workspace:*",
	20	+ "fastify": "^5.4.0",
	21	+ "ioredis": "^5.6.0",
	22	+ "pino": "^9.7.0",
	23	+ "pino-pretty": "^13.0.0",
	24	+ "prom-client": "^15.1.0",
	25	+ "tsx": "^4.20.0",
	26	+ "ws": "^8.21.3",
	27	+ "zod": "^4.0.0"
	28	+ },
	29	+ "devDependencies": {
	30	+ "@types/node": "^24.0.0",
	31	+ "@types/ws": "^8.18.1",
	32	+ "typescript": "^5.9.3",
	33	+ "vitest": "^3.2.0"
	34	+ }
	35	+}

added apps/api/src/config.ts +14 −0

		@@ -0,0 +1,14 @@
	1	+const env = process.env;
	2	+
	3	+export const config = {
	4	+ env: env.NODE_ENV ?? "development",
	5	+ logLevel: env.LOG_LEVEL ?? "info",
	6	+ port: Number(env.API_PORT ?? 8260),
	7	+ host: env.API_HOST ?? "127.0.0.1",
	8	+ redisUrl: env.REDIS_URL ?? "redis://127.0.0.1:6379",
	9	+ webUrl: env.WEB_URL ?? "http://127.0.0.1:8261",
	10	+ publicBaseUrl: env.PUBLIC_BASE_URL ?? "http://localhost:8260",
	11	+ canonicalHost: env.CANONICAL_HOST ?? "www.websensor.io",
	12	+ redirectApexToWww: (env.WS_REDIRECT_APEX ?? "1") !== "0",
	13	+ version: "0.1.0",
	14	+};

added apps/api/src/live.ts +129 −0

		@@ -0,0 +1,129 @@
	1	+import type { FastifyInstance } from "fastify";
	2	+import type { WebSocket } from "ws";
	3	+import Redis from "ioredis";
	4	+import { FEED_CHANNELS } from "@websensor/core";
	5	+import { db, sql } from "@websensor/db";
	6	+import { config } from "./config";
	7	+
	8	+/**
	9	+ * WebSocket gateway `/api/v1/live`. One Redis subscriber fans out to every client; clients
	10	+ * pick channels: events:global · events:breaking · events:<ai|cyber|finance|health|government|
	11	+ * science|products|infrastructure> · entity:<id> · source:<id> · watchlist:<id>.
	12	+ * Protocol (JSON): client → {"subscribe":[…]} | {"unsubscribe":[…]} | {"ping":1}
	13	+ * server → {"type":"hello"} | {"type":"event", "channels":[…], "event":{…}} | {"type":"pong"}
	14	+ */
	15	+interface Client {
	16	+ ws: WebSocket;
	17	+ channels: Set<string>;
	18	+ watchlists: Map<string, { entities: Set<string>; sources: Set<string>; keywords: string[]; categories: Set<string> }>;
	19	+}
	20	+
	21	+const clients = new Set<Client>();
	22	+let sub: Redis | null = null;
	23	+let published = 0;
	24	+
	25	+export function liveStats(): { clients: number; published: number } {
	26	+ return { clients: clients.size, published };
	27	+}
	28	+
	29	+export async function registerLive(app: FastifyInstance): Promise<void> {
	30	+ sub = new Redis(config.redisUrl, { maxRetriesPerRequest: 3 });
	31	+ sub.on("error", (e) => app.log.warn({ err: e.message }, "redis sub error"));
	32	+ await sub.subscribe("ws:live");
	33	+ sub.on("message", (_ch, msg) => {
	34	+ let ev: Record<string, unknown>;
	35	+ try {
	36	+ ev = JSON.parse(msg) as Record<string, unknown>;
	37	+ } catch {
	38	+ return;
	39	+ }
	40	+ published++;
	41	+ const chans = channelsFor(ev);
	42	+ for (const c of clients) {
	43	+ const hit = [...chans].filter((ch) => c.channels.has(ch));
	44	+ for (const [wid, w] of c.watchlists) if (matchesWatchlist(ev, w)) hit.push(`watchlist:${wid}`);
	45	+ if (!hit.length) continue;
	46	+ if (c.ws.readyState === c.ws.OPEN) c.ws.send(JSON.stringify({ type: "event", channels: hit, event: ev }));
	47	+ }
	48	+ });
	49	+
	50	+ app.get("/api/v1/live", { websocket: true }, (socket) => {
	51	+ const client: Client = { ws: socket, channels: new Set(["events:global"]), watchlists: new Map() };
	52	+ clients.add(client);
	53	+ socket.send(JSON.stringify({ type: "hello", channels: [...client.channels], serverTime: new Date().toISOString() }));
	54	+ socket.on("message", async (raw: Buffer | string) => {
	55	+ let msg: { subscribe?: string[]; unsubscribe?: string[]; ping?: number };
	56	+ try {
	57	+ msg = JSON.parse(raw.toString()) as typeof msg;
	58	+ } catch {
	59	+ return;
	60	+ }
	61	+ if (msg.ping) socket.send(JSON.stringify({ type: "pong", t: Date.now() }));
	62	+ for (const ch of msg.subscribe ?? []) {
	63	+ if (typeof ch !== "string" || ch.length > 120 || client.channels.size > 64) continue;
	64	+ if (ch.startsWith("watchlist:")) await loadWatchlist(client, ch.slice(10));
	65	+ else client.channels.add(ch);
	66	+ }
	67	+ for (const ch of msg.unsubscribe ?? []) {
	68	+ client.channels.delete(ch);
	69	+ if (ch.startsWith("watchlist:")) client.watchlists.delete(ch.slice(10));
	70	+ }
	71	+ socket.send(JSON.stringify({ type: "subscribed", channels: [...client.channels, ...[...client.watchlists.keys()].map((w) => `watchlist:${w}`)] }));
	72	+ });
	73	+ const hb = setInterval(() => {
	74	+ if (socket.readyState === socket.OPEN) socket.send(JSON.stringify({ type: "heartbeat", t: Date.now() }));
	75	+ }, 25_000);
	76	+ socket.on("close", () => {
	77	+ clearInterval(hb);
	78	+ clients.delete(client);
	79	+ });
	80	+ socket.on("error", () => {
	81	+ clearInterval(hb);
	82	+ clients.delete(client);
	83	+ });
	84	+ });
	85	+}
	86	+
	87	+export function channelsFor(ev: Record<string, unknown>): Set<string> {
	88	+ const out = new Set<string>(["events:global"]);
	89	+ const importance = Number(ev.importance ?? 0);
	90	+ if (importance >= 80) out.add("events:breaking");
	91	+ if (ev.silent) out.add("events:silent");
	92	+ const cats = (ev.categories as string[] | undefined) ?? [];
	93	+ for (const [ch, list] of Object.entries(FEED_CHANNELS)) if (list.some((c) => cats.includes(c)) || cats.includes(ch)) out.add(`events:${ch}`);
	94	+ const src = ev.source as { id?: string } | undefined;
	95	+ if (src?.id) out.add(`source:${src.id}`);
	96	+ for (const e of (ev.entities as { id: string }[] | undefined) ?? []) out.add(`entity:${e.id}`);
	97	+ out.add(`type:${String(ev.type)}`);
	98	+ return out;
	99	+}
	100	+
	101	+async function loadWatchlist(client: Client, id: string): Promise<void> {
	102	+ const rows = await db.execute<{ kind: string; value: string }>(sql`select kind, value from watchlist_items where watchlist_id = ${id}`);
	103	+ const w = { entities: new Set<string>(), sources: new Set<string>(), keywords: [] as string[], categories: new Set<string>() };
	104	+ for (const r of rows.rows) {
	105	+ if (r.kind === "entity") w.entities.add(r.value);
	106	+ else if (r.kind === "source") w.sources.add(r.value);
	107	+ else if (r.kind === "keyword") w.keywords.push(r.value.toLowerCase());
	108	+ else if (r.kind === "category") w.categories.add(r.value);
	109	+ }
	110	+ client.watchlists.set(id, w);
	111	+}
	112	+
	113	+function matchesWatchlist(ev: Record<string, unknown>, w: { entities: Set<string>; sources: Set<string>; keywords: string[]; categories: Set<string> }): boolean {
	114	+ const src = ev.source as { id?: string } | undefined;
	115	+ if (src?.id && w.sources.has(src.id)) return true;
	116	+ for (const e of (ev.entities as { id: string }[] | undefined) ?? []) if (w.entities.has(e.id)) return true;
	117	+ for (const c of (ev.categories as string[] | undefined) ?? []) if (w.categories.has(c)) return true;
	118	+ if (w.keywords.length) {
	119	+ const hay = `${String(ev.title)} ${String(ev.summary)}`.toLowerCase();
	120	+ if (w.keywords.some((k) => hay.includes(k))) return true;
	121	+ }
	122	+ return false;
	123	+}
	124	+
	125	+export async function closeLive(): Promise<void> {
	126	+ for (const c of clients) c.ws.close(1001, "server shutdown");
	127	+ clients.clear();
	128	+ if (sub) await sub.quit().catch(() => undefined);
	129	+}

added apps/api/src/queries.ts +146 −0

		@@ -0,0 +1,146 @@
	1	+import { db, sql, textArray } from "@websensor/db";
	2	+
	3	+/** Read-model helpers shared by REST routes. All return plain JSON-ready objects. */
	4	+
	5	+export interface EventFilters {
	6	+ after?: string;
	7	+ before?: string;
	8	+ category?: string;
	9	+ entity?: string;
	10	+ source?: string;
	11	+ domain?: string;
	12	+ sensor?: string;
	13	+ cluster?: string;
	14	+ importance_min?: number;
	15	+ confidence_min?: number;
	16	+ event_type?: string;
	17	+ silent_change?: boolean;
	18	+ q?: string;
	19	+ limit: number;
	20	+ cursor?: string;
	21	+ order?: "recent" | "importance";
	22	+}
	23	+
	24	+export const EVENT_SELECT = sql`
	25	+ e.id, e.slug, e.event_type, e.title, e.summary, e.why_it_matters, e.importance, e.confidence, e.novelty, e.categories, e.keywords,
	26	+ e.silent_change, e.evidence_label, e.url, e.detected_at, e.published_at, e.observed_from, e.processed_at, e.published_to_feed_at,
	27	+ e.detection_latency_ms, e.processing_latency_ms, e.cluster_id, e.sensor_id, e.source_id, e.change_id, e.old_snapshot_id, e.new_snapshot_id,
	28	+ e.importance_components, e.processing_version,
	29	+ json_build_object('id', s.id, 'name', s.name, 'domain', s.domain, 'tier', s.tier, 'categories', s.categories) as source,
	30	+ json_build_object('id', sen.id, 'name', sen.name, 'type', sen.type, 'connector', sen.connector, 'tier', sen.tier) as sensor,
	31	+ coalesce((select json_agg(json_build_object('id', en.id, 'name', en.name, 'type', en.type, 'role', ee.role) order by ee.role, en.name)
	32	+ from event_entities ee join entities en on en.id = ee.entity_id where ee.event_id = e.id), '[]'::json) as entities,
	33	+ (select event_count from event_clusters c where c.id = e.cluster_id) as cluster_size`;
	34	+
	35	+export async function listEvents(f: EventFilters): Promise<{ items: Record<string, unknown>[]; nextCursor: string | null }> {
	36	+ const conds = [sql`true`];
	37	+ if (f.after) conds.push(sql`e.detected_at > ${new Date(f.after)}`);
	38	+ if (f.before) conds.push(sql`e.detected_at < ${new Date(f.before)}`);
	39	+ if (f.category) conds.push(sql`${f.category} = any(e.categories)`);
	40	+ if (f.source) conds.push(sql`e.source_id = ${f.source}`);
	41	+ if (f.sensor) conds.push(sql`e.sensor_id = ${f.sensor}`);
	42	+ if (f.cluster) conds.push(sql`e.cluster_id = ${f.cluster}`);
	43	+ if (f.domain) conds.push(sql`s.domain = ${f.domain}`);
	44	+ if (f.entity) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = ${f.entity})`);
	45	+ if (f.importance_min !== undefined) conds.push(sql`e.importance >= ${f.importance_min}`);
	46	+ if (f.confidence_min !== undefined) conds.push(sql`e.confidence >= ${f.confidence_min}`);
	47	+ if (f.event_type) conds.push(sql`e.event_type = any(${textArray(f.event_type.split(","))})`);
	48	+ if (f.silent_change !== undefined) conds.push(sql`e.silent_change = ${f.silent_change}`);
	49	+ if (f.q) conds.push(sql`e.search @@ websearch_to_tsquery('english', ${f.q})`);
	50	+ if (f.cursor) {
	51	+ const [ts, id] = decodeCursor(f.cursor);
	52	+ if (f.order === "importance") conds.push(sql`(e.importance, e.id) < (${Number(ts)}, ${id})`);
	53	+ else conds.push(sql`(e.detected_at, e.id) < (${new Date(Number(ts))}, ${id})`);
	54	+ }
	55	+ const order = f.order === "importance" ? sql`e.importance desc, e.id desc` : sql`e.detected_at desc, e.id desc`;
	56	+ const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where ${sql.join(conds, sql` and `)} order by ${order} limit ${f.limit + 1}`);
	57	+ const items = rows.rows.slice(0, f.limit);
	58	+ const last = items[items.length - 1];
	59	+ const nextCursor = rows.rows.length > f.limit && last ? encodeCursor(f.order === "importance" ? String(last.importance) : String(new Date(last.detected_at as string).getTime()), String(last.id)) : null;
	60	+ return { items, nextCursor };
	61	+}
	62	+
	63	+export async function getEvent(idOrSlug: string): Promise<Record<string, unknown> | null> {
	64	+ const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT}, e.interpretation from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.id = ${idOrSlug} or e.slug = ${idOrSlug} limit 1`);
	65	+ return rows.rows[0] ?? null;
	66	+}
	67	+
	68	+export async function relatedEvents(ev: Record<string, unknown>, limit = 8): Promise<Record<string, unknown>[]> {
	69	+ const rows = await db.execute<Record<string, unknown>>(sql`
	70	+ select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id
	71	+ where e.id <> ${String(ev.id)} and (e.cluster_id = ${String(ev.cluster_id ?? "")} or e.source_id = ${String(ev.source_id)} or exists (select 1 from event_entities a join event_entities b on a.entity_id = b.entity_id where a.event_id = e.id and b.event_id = ${String(ev.id)}))
	72	+ order by (e.cluster_id = ${String(ev.cluster_id ?? "")}) desc, e.detected_at desc limit ${limit}`);
	73	+ return rows.rows;
	74	+}
	75	+
	76	+export async function stats(): Promise<Record<string, unknown>> {
	77	+ const [r] = (
	78	+ await db.execute<Record<string, unknown>>(sql`
	79	+ select
	80	+ (select count(*) from sources where enabled) as sources,
	81	+ (select count(*) from sensors where enabled) as sensors,
	82	+ (select count(*) from entities) as entities,
	83	+ (select count(*) from snapshots) as snapshots,
	84	+ (select count(*) from events) as events_total,
	85	+ (select count(*) from events where detected_at >= now() - interval '24 hours') as events_24h,
	86	+ (select count(*) from events where detected_at >= now() - interval '24 hours' and silent_change) as silent_24h,
	87	+ (select count(*) from events where detected_at >= now() - interval '24 hours' and importance >= 80) as breaking_24h,
	88	+ (select count(*) from changes where detected_at >= now() - interval '24 hours') as changes_24h,
	89	+ (select coalesce(sum(checks),0) from metrics_daily where day = current_date) as checks_today,
	90	+ (select coalesce(sum(not_modified),0) from metrics_daily where day = current_date) as not_modified_today,
	91	+ (select coalesce(sum(bytes),0) from metrics_daily where day = current_date) as bytes_today,
	92	+ (select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour,
	93	+ (select count(*) from sensors where health = 'UP' and enabled) as sensors_up,
	94	+ (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled) as sensors_degraded,
	95	+ (select max(started_at) from sensor_runs) as last_check_at,
	96	+ (select max(detected_at) from events) as last_event_at,
	97	+ (select percentile_cont(0.5) within group (order by processing_latency_ms) from events where detected_at >= now() - interval '24 hours') as p50_processing_ms,
	98	+ (select percentile_cont(0.5) within group (order by detection_latency_ms) from events where detected_at >= now() - interval '24 hours' and detection_latency_ms is not null and detection_latency_ms < 86400000) as p50_detection_ms`)
	99	+ ).rows;
	100	+ return Object.fromEntries(Object.entries(r ?? {}).map(([k, v]) => [k, typeof v === "string" && /^\d+$/.test(v) ? Number(v) : v]));
	101	+}
	102	+
	103	+export async function trending(hours = 24, limit = 12): Promise<Record<string, unknown>[]> {
	104	+ const rows = await db.execute<Record<string, unknown>>(sql`
	105	+ with cur as (
	106	+ select ee.entity_id, count(*) as n, sum(e.importance) as imp, count(distinct e.source_id) as sources, sum(case when e.silent_change then 1 else 0 end) as silent, max(e.importance) as max_imp
	107	+ from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - make_interval(hours => ${hours}) group by ee.entity_id),
	108	+ prev as (
	109	+ select ee.entity_id, count(*) as n from events e join event_entities ee on ee.event_id = e.id
	110	+ where e.detected_at >= now() - make_interval(hours => ${hours * 2}) and e.detected_at < now() - make_interval(hours => ${hours}) group by ee.entity_id)
	111	+ select en.id, en.name, en.type, en.domain, cur.n::int as events, cur.imp::float as importance_sum, cur.sources::int as sources, cur.silent::int as silent, cur.max_imp::float as max_importance, coalesce(prev.n,0)::int as prev_events,
	112	+ round((18*(ln(1+cur.n)/ln(2)) + 0.35*(cur.imp/greatest(1,cur.n)) + 10*(ln(1+cur.sources)/ln(2)) + 12*least(2, case when coalesce(prev.n,0)=0 then 2 else cur.n::float/prev.n end) + 5*least(3,cur.silent))::numeric, 1)::float as score
	113	+ from cur join entities en on en.id = cur.entity_id left join prev on prev.entity_id = cur.entity_id
	114	+ order by score desc limit ${limit}`);
	115	+ return rows.rows;
	116	+}
	117	+
	118	+export async function sourceActivity(sourceId: string): Promise<Record<string, unknown>> {
	119	+ const [r] = (
	120	+ await db.execute<Record<string, unknown>>(sql`
	121	+ select
	122	+ (select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '2 hours')::int as changes_2h,
	123	+ (select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '14 days')::int as changes_14d,
	124	+ (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '24 hours')::int as events_24h,
	125	+ (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '14 days')::int as events_14d`)
	126	+ ).rows;
	127	+ const c2h = Number(r?.changes_2h ?? 0);
	128	+ const baselinePerHour = Number(r?.changes_14d ?? 0) / (14 * 24);
	129	+ const currentPerHour = c2h / 2;
	130	+ let anomaly = 0;
	131	+ if (baselinePerHour <= 0) anomaly = currentPerHour > 2 ? 70 : currentPerHour > 0 ? 40 : 0;
	132	+ else {
	133	+ const ratio = currentPerHour / baselinePerHour;
	134	+ anomaly = ratio <= 1 ? ratio * 30 : Math.min(100, 30 + 25 * Math.log2(ratio));
	135	+ }
	136	+ return { ...r, baseline_changes_per_day: Math.round(baselinePerHour * 24 * 10) / 10, activity_score: Math.round(anomaly * 10) / 10 };
	137	+}
	138	+
	139	+export function encodeCursor(a: string, b: string): string {
	140	+ return Buffer.from(`${a}|${b}`).toString("base64url");
	141	+}
	142	+export function decodeCursor(c: string): [string, string] {
	143	+ const s = Buffer.from(c, "base64url").toString("utf8");
	144	+ const i = s.indexOf("|");
	145	+ return [s.slice(0, i), s.slice(i + 1)];
	146	+}

added apps/api/src/routes.ts +349 −0

		@@ -0,0 +1,349 @@
	1	+import type { FastifyInstance } from "fastify";
	2	+import { z } from "zod";
	3	+import { diffText, EVENT_TYPES, FEED_CHANNELS, newId } from "@websensor/core";
	4	+import { db, sql, textArray } from "@websensor/db";
	5	+import { getBlobStore } from "@websensor/store";
	6	+import { config } from "./config";
	7	+import { liveStats } from "./live";
	8	+import { getEvent, listEvents, relatedEvents, sourceActivity, stats, trending, EVENT_SELECT } from "./queries";
	9	+
	10	+const eventsQuery = z.object({
	11	+ after: z.string().optional(),
	12	+ before: z.string().optional(),
	13	+ category: z.string().optional(),
	14	+ entity: z.string().optional(),
	15	+ source: z.string().optional(),
	16	+ domain: z.string().optional(),
	17	+ sensor: z.string().optional(),
	18	+ cluster: z.string().optional(),
	19	+ importance_min: z.coerce.number().min(0).max(100).optional(),
	20	+ confidence_min: z.coerce.number().min(0).max(100).optional(),
	21	+ event_type: z.string().optional(),
	22	+ silent_change: z
	23	+ .enum(["true", "false"])
	24	+ .transform((v) => v === "true")
	25	+ .optional(),
	26	+ q: z.string().max(200).optional(),
	27	+ limit: z.coerce.number().int().min(1).max(200).default(50),
	28	+ cursor: z.string().optional(),
	29	+ order: z.enum(["recent", "importance"]).default("recent"),
	30	+});
	31	+
	32	+function ownerToken(headers: Record<string, unknown>): string | null {
	33	+ const t = headers["x-websensor-owner"];
	34	+ return typeof t === "string" && /^[A-Za-z0-9_-]{16,80}$/.test(t) ? t : null;
	35	+}
	36	+
	37	+export async function registerRoutes(app: FastifyInstance): Promise<void> {
	38	+ // ---- Health ---------------------------------------------------------------------------
	39	+ app.get("/api/health", async () => ({ status: "ok", service: "api", version: config.version, time: new Date().toISOString() }));
	40	+ app.get("/api/ready", async (_req, reply) => {
	41	+ const checks: Record<string, boolean | number> = {};
	42	+ try {
	43	+ await db.execute(sql`select 1`);
	44	+ checks.database = true;
	45	+ } catch {
	46	+ checks.database = false;
	47	+ }
	48	+ const engine = await db.execute<{ last: Date | null }>(sql`select max(started_at) as last from sensor_runs where started_at >= now() - interval '15 minutes'`);
	49	+ checks.engine_recent = Boolean(engine.rows[0]?.last);
	50	+ checks.ws_clients = liveStats().clients;
	51	+ return reply.status(checks.database ? 200 : 503).send({ status: checks.database ? "ready" : "degraded", checks });
	52	+ });
	53	+
	54	+ // ---- Events ---------------------------------------------------------------------------
	55	+ app.get("/api/v1/events", async (req) => {
	56	+ const q = eventsQuery.parse(req.query);
	57	+ return listEvents(q);
	58	+ });
	59	+ app.get<{ Params: { id: string } }>("/api/v1/events/:id", async (req, reply) => {
	60	+ const ev = await getEvent(req.params.id);
	61	+ if (!ev) return reply.status(404).send({ error: "not_found" });
	62	+ const [related, cluster, change, interp] = await Promise.all([
	63	+ relatedEvents(ev),
	64	+ ev.cluster_id ? db.execute<Record<string, unknown>>(sql`select * from event_clusters where id = ${String(ev.cluster_id)}`).then((r) => r.rows[0] ?? null) : null,
	65	+ ev.change_id ? db.execute<Record<string, unknown>>(sql`select id, kind, diff, signal, noise_ratio, magnitude, heuristic, detected_at, old_snapshot_id, new_snapshot_id from changes where id = ${String(ev.change_id)}`).then((r) => r.rows[0] ?? null) : null,
	66	+ db.execute<Record<string, unknown>>(sql`select version, model, created_at from interpretations where event_id = ${String(ev.id)} order by version`).then((r) => r.rows),
	67	+ ]);
	68	+ const snaps = await db.execute<Record<string, unknown>>(sql`select id, url, captured_at, http_status, content_type, content_length, content_hash, canonical_hash, etag, last_modified, title, mode, fetch_duration_ms, extraction_confidence from snapshots where id in (${String(ev.old_snapshot_id ?? "")}, ${String(ev.new_snapshot_id ?? "")})`);
	69	+ const sourceRel = await db.execute<Record<string, unknown>>(sql`select health, success_rate, avg_latency_ms, total_runs, raw_changes, meaningful_changes, last_check_at from (select s.health, ch.success_rate, s.avg_latency_ms, s.total_runs, s.raw_changes, s.meaningful_changes, s.last_check_at from sensors s left join connector_health ch on ch.connector = s.connector where s.id = ${String(ev.sensor_id)}) x`);
	70	+ return { event: ev, related, cluster, change, interpretations: interp, snapshots: snaps.rows, sensor_reliability: sourceRel.rows[0] ?? null };
	71	+ });
	72	+
	73	+ // ---- Changes / snapshots / diffs -------------------------------------------------------
	74	+ app.get<{ Params: { id: string } }>("/api/v1/changes/:id", async (req, reply) => {
	75	+ const r = await db.execute<Record<string, unknown>>(sql`select c.*, s.url as sensor_url, s.name as sensor_name, s.source_id from changes c join sensors s on s.id = c.sensor_id where c.id = ${req.params.id}`);
	76	+ const c = r.rows[0];
	77	+ if (!c) return reply.status(404).send({ error: "not_found" });
	78	+ let unified: string | null = null;
	79	+ if (c.diff_storage_key) unified = await getBlobStore().getText(String(c.diff_storage_key)).catch(() => null);
	80	+ return { change: c, unified };
	81	+ });
	82	+ app.get<{ Params: { id: string }; Querystring: { raw?: string } }>("/api/v1/snapshots/:id", async (req, reply) => {
	83	+ const r = await db.execute<Record<string, unknown>>(sql`select * from snapshots where id = ${req.params.id}`);
	84	+ const s = r.rows[0];
	85	+ if (!s) return reply.status(404).send({ error: "not_found" });
	86	+ const store = getBlobStore();
	87	+ if (req.query.raw === "1" && s.storage_key) {
	88	+ const buf = await store.get(String(s.storage_key));
	89	+ reply.header("content-type", String(s.content_type ?? "text/plain") + (String(s.content_type ?? "").includes("charset") ? "" : "; charset=utf-8"));
	90	+ reply.header("x-content-type-options", "nosniff");
	91	+ reply.header("content-security-policy", "default-src 'none'; style-src 'unsafe-inline'; img-src data:");
	92	+ return reply.send(buf);
	93	+ }
	94	+ const canonical = s.canonical_storage_key ? await store.getText(String(s.canonical_storage_key)).catch(() => null) : null;
	95	+ return { snapshot: s, canonical: canonical ? JSON.parse(canonical) : null };
	96	+ });
	97	+ app.get<{ Querystring: { a: string; b: string } }>("/api/v1/snapshots/compare", async (req, reply) => {
	98	+ const { a, b } = req.query;
	99	+ if (!a || !b) return reply.status(400).send({ error: "a and b required" });
	100	+ const rows = await db.execute<Record<string, unknown>>(sql`select id, captured_at, canonical_storage_key, mode, url, sensor_id from snapshots where id in (${a}, ${b})`);
	101	+ const sa = rows.rows.find((r) => r.id === a);
	102	+ const sb = rows.rows.find((r) => r.id === b);
	103	+ if (!sa || !sb) return reply.status(404).send({ error: "not_found" });
	104	+ const store = getBlobStore();
	105	+ const [ca, cb] = await Promise.all([store.getText(String(sa.canonical_storage_key)), store.getText(String(sb.canonical_storage_key))]);
	106	+ const ta = renderCanonical(JSON.parse(ca));
	107	+ const tb = renderCanonical(JSON.parse(cb));
	108	+ const d = diffText(ta, tb, `${a}@${String(sa.captured_at)}`, `${b}@${String(sb.captured_at)}`);
	109	+ return { a: sa, b: sb, before: ta, after: tb, diff: { unified: d.unified, stats: d.stats, added: d.added, removed: d.removed, modified: d.modified } };
	110	+ });
	111	+
	112	+ // ---- Sources & sensors ------------------------------------------------------------------
	113	+ app.get<{ Querystring: { category?: string; q?: string; limit?: string } }>("/api/v1/sources", async (req) => {
	114	+ const cat = req.query.category;
	115	+ const q = req.query.q;
	116	+ const rows = await db.execute<Record<string, unknown>>(sql`
	117	+ select s.*, (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count,
	118	+ (select count(*) from events e where e.source_id = s.id)::int as event_count,
	119	+ (select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h,
	120	+ (select max(detected_at) from events e where e.source_id = s.id) as last_event_at,
	121	+ (select max(last_check_at) from sensors x where x.source_id = s.id) as last_check_at,
	122	+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.health <> 'UP')::int as sensors_degraded
	123	+ from sources s where s.enabled ${cat ? sql`and ${cat} = any(s.categories)` : sql``} ${q ? sql`and (s.name ilike ${"%" + q + "%"} or s.domain ilike ${"%" + q + "%"})` : sql``}
	124	+ order by s.tier asc, events_24h desc, s.name asc limit ${Math.min(500, Number(req.query.limit ?? 300))}`);
	125	+ return { items: rows.rows };
	126	+ });
	127	+ app.get<{ Params: { id: string } }>("/api/v1/sources/:id", async (req, reply) => {
	128	+ const s = (await db.execute<Record<string, unknown>>(sql`select * from sources where id = ${req.params.id} or domain = ${req.params.id} limit 1`)).rows[0];
	129	+ if (!s) return reply.status(404).send({ error: "not_found" });
	130	+ const [sensors, ents, activity, candidates] = await Promise.all([
	131	+ db.execute<Record<string, unknown>>(sql`select id, name, url, type, connector, tier, health, enabled, next_check_at, last_check_at, last_change_at, last_event_at, last_status, last_error, consecutive_errors, total_runs, total_not_modified, raw_changes, meaningful_changes, avg_latency_ms, base_interval_seconds, etag is not null as has_etag, last_modified is not null as has_last_modified from sensors where source_id = ${String(s.id)} order by tier, name`).then((r) => r.rows),
	132	+ db.execute<Record<string, unknown>>(sql`select en.id, en.name, en.type, en.importance, en.event_count from source_entities se join entities en on en.id = se.entity_id where se.source_id = ${String(s.id)} order by en.type, en.name`).then((r) => r.rows),
	133	+ sourceActivity(String(s.id)),
	134	+ db.execute<Record<string, unknown>>(sql`select url, kind, evidence, score, status, found_at from discovery_candidates where source_id = ${String(s.id)} order by (score->>'value')::float desc nulls last limit 50`).then((r) => r.rows),
	135	+ ]);
	136	+ return { source: s, sensors, entities: ents, activity, discovery: candidates };
	137	+ });
	138	+ app.get<{ Params: { id: string } }>("/api/v1/sensors/:id", async (req, reply) => {
	139	+ const s = (await db.execute<Record<string, unknown>>(sql`select s.*, so.name as source_name, so.domain from sensors s join sources so on so.id = s.source_id where s.id = ${req.params.id}`)).rows[0];
	140	+ if (!s) return reply.status(404).send({ error: "not_found" });
	141	+ const [runs, snaps, changes] = await Promise.all([
	142	+ db.execute<Record<string, unknown>>(sql`select id, started_at, finished_at, http_status, outcome, error, duration_ms, bytes, fetch_method, snapshot_id from sensor_runs where sensor_id = ${String(s.id)} order by started_at desc limit 50`).then((r) => r.rows),
	143	+ db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_type, content_length, canonical_hash, title, mode, extraction_confidence from snapshots where sensor_id = ${String(s.id)} order by captured_at desc limit 50`).then((r) => r.rows),
	144	+ db.execute<Record<string, unknown>>(sql`select id, detected_at, kind, signal, noise_ratio, magnitude, meaningful, event_id, old_snapshot_id, new_snapshot_id, heuristic->>'eventType' as heuristic_type from changes where sensor_id = ${String(s.id)} order by detected_at desc limit 50`).then((r) => r.rows),
	145	+ ]);
	146	+ const { etag: _e, last_modified: _lm, state: _st, ...pub } = s as Record<string, unknown>;
	147	+ return { sensor: pub, runs, snapshots: snaps, changes };
	148	+ });
	149	+
	150	+ // ---- Entities ----------------------------------------------------------------------------
	151	+ app.get<{ Querystring: { type?: string; q?: string; limit?: string } }>("/api/v1/entities", async (req) => {
	152	+ const rows = await db.execute<Record<string, unknown>>(sql`
	153	+ select en.*, (select count(*) from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = en.id and e.detected_at >= now() - interval '24 hours')::int as events_24h
	154	+ from entities en where true ${req.query.type ? sql`and en.type = ${req.query.type}` : sql``} ${req.query.q ? sql`and (en.search @@ plainto_tsquery('simple', ${req.query.q}) or en.name ilike ${"%" + req.query.q + "%"})` : sql``}
	155	+ order by en.event_count desc, en.importance desc, en.name limit ${Math.min(500, Number(req.query.limit ?? 200))}`);
	156	+ return { items: rows.rows };
	157	+ });
	158	+ app.get<{ Params: { id: string } }>("/api/v1/entities/:id", async (req, reply) => {
	159	+ const e = (await db.execute<Record<string, unknown>>(sql`select * from entities where id = ${req.params.id} or id = ${"org_" + req.params.id} limit 1`)).rows[0];
	160	+ if (!e) return reply.status(404).send({ error: "not_found" });
	161	+ const id = String(e.id);
	162	+ const [children, relations, sources, aliases, recent, byType] = await Promise.all([
	163	+ db.execute<Record<string, unknown>>(sql`select id, name, type, importance, event_count, last_event_at from entities where parent_id = ${id} order by event_count desc, name`).then((r) => r.rows),
	164	+ db.execute<Record<string, unknown>>(sql`select r.relation, r.from_id, r.to_id, f.name as from_name, t.name as to_name, t.type as to_type from entity_relations r join entities f on f.id = r.from_id join entities t on t.id = r.to_id where r.from_id = ${id} or r.to_id = ${id} limit 100`).then((r) => r.rows),
	165	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier from source_entities se join sources s on s.id = se.source_id where se.entity_id = ${id}`).then((r) => r.rows),
	166	+ db.execute<{ alias: string }>(sql`select alias from entity_aliases where entity_id = ${id} order by alias`).then((r) => r.rows.map((x) => x.alias)),
	167	+ listEvents({ entity: id, limit: 30 }),
	168	+ db.execute<Record<string, unknown>>(sql`select e.event_type, count(*)::int as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${id} group by 1 order by 2 desc`).then((r) => r.rows),
	169	+ ]);
	170	+ return { entity: e, children, relations, sources, aliases, recent: recent.items, by_type: byType };
	171	+ });
	172	+ app.get<{ Params: { id: string }; Querystring: { limit?: string; cursor?: string } }>("/api/v1/entities/:id/timeline", async (req) => {
	173	+ const id = req.params.id.startsWith("org_") || req.params.id.startsWith("prd_") ? req.params.id : `org_${req.params.id}`;
	174	+ return listEvents({ entity: id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor });
	175	+ });
	176	+
	177	+ // ---- Domains / URLs ----------------------------------------------------------------------
	178	+ app.get<{ Params: { domain: string }; Querystring: { limit?: string; cursor?: string } }>("/api/v1/domains/:domain/timeline", async (req) => {
	179	+ const d = req.params.domain.toLowerCase();
	180	+ const src = (await db.execute<{ id: string }>(sql`select id from sources where domain = ${d} or domain = ${"www." + d} or ${d} = 'www.' || domain limit 1`)).rows[0];
	181	+ const urls = await db.execute<Record<string, unknown>>(sql`select url, status, first_seen_at, last_seen_at, snapshot_count, change_count, sensor_id from urls where domain = ${d} or domain like ${"%." + d} order by change_count desc, last_seen_at desc limit 200`);
	182	+ const ev = src ? await listEvents({ source: src.id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor }) : { items: [], nextCursor: null };
	183	+ return { domain: d, source_id: src?.id ?? null, urls: urls.rows, events: ev.items, nextCursor: ev.nextCursor };
	184	+ });
	185	+ app.get<{ Querystring: { url: string } }>("/api/v1/urls/history", async (req, reply) => {
	186	+ if (!req.query.url) return reply.status(400).send({ error: "url required" });
	187	+ const u = (await db.execute<Record<string, unknown>>(sql`select * from urls where url = ${req.query.url}`)).rows[0];
	188	+ const history = await db.execute<Record<string, unknown>>(sql`
	189	+ select h.id, h.at, h.kind, h.snapshot_id, h.change_id, h.event_id, h.note, e.title as event_title, e.importance, e.event_type, e.slug as event_slug, c.kind as change_kind, c.signal
	190	+ from url_history h left join events e on e.id = h.event_id left join changes c on c.id = h.change_id where h.url = ${req.query.url} order by h.at desc limit 300`);
	191	+ const snaps = await db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_length, canonical_hash, title from snapshots where url = ${req.query.url} or sensor_id = (select sensor_id from urls where url = ${req.query.url}) order by captured_at desc limit 200`);
	192	+ return { url: u ?? { url: req.query.url }, history: history.rows, snapshots: snaps.rows };
	193	+ });
	194	+
	195	+ // ---- Explore / trending / stats ------------------------------------------------------------
	196	+ app.get("/api/v1/stats", async () => stats());
	197	+ app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => ({ items: await trending(Math.min(168, Number(req.query.hours ?? 24)), Math.min(50, Number(req.query.limit ?? 12))) }));
	198	+ app.get("/api/v1/explore", async () => {
	199	+ const [mostActive, biggest, silent, clusters, unusual, byType, byCategory] = await Promise.all([
	200	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, count(*)::int as events_24h, max(e.importance)::float as max_importance from events e join sources s on s.id = e.source_id where e.detected_at >= now() - interval '24 hours' group by s.id, s.name, s.domain order by events_24h desc limit 10`).then((r) => r.rows),
	201	+ listEvents({ limit: 10, order: "importance", after: new Date(Date.now() - 48 * 3600e3).toISOString() }).then((r) => r.items),
	202	+ listEvents({ limit: 10, silent_change: true }).then((r) => r.items),
	203	+ db.execute<Record<string, unknown>>(sql`select c.*, (select json_build_object('id', s.id, 'name', s.name, 'domain', s.domain) from events e join sources s on s.id = e.source_id where e.id = c.primary_event_id) as source from event_clusters c where c.event_count >= 2 and c.last_at >= now() - interval '48 hours' order by c.max_importance desc, c.event_count desc limit 10`).then((r) => r.rows),
	204	+ db.execute<Record<string, unknown>>(sql`
	205	+ with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '2 hours' group by s.source_id),
	206	+ base as (select s.source_id, count(*)::float / (14*24) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id)
	207	+ select so.id, so.name, so.domain, cur.n::int as changes_2h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day,
	208	+ round((case when coalesce(base.per_hour,0) = 0 then (case when cur.n/2.0 > 2 then 70 else 40 end) else least(100, case when cur.n/2.0/base.per_hour <= 1 then cur.n/2.0/base.per_hour*30 else 30 + 25*(ln(cur.n/2.0/base.per_hour)/ln(2)) end) end)::numeric, 1)::float as activity_score
	209	+ from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id order by activity_score desc limit 10`).then((r) => r.rows),
	210	+ db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows),
	211	+ db.execute<Record<string, unknown>>(sql`select c as category, count(*)::int as n from events e, unnest(e.categories) c where e.detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows),
	212	+ ]);
	213	+ return { most_active_sources: mostActive, biggest_changes: biggest, silent_changes: silent, clusters, unusual_activity: unusual, by_type: byType, by_category: byCategory, channels: FEED_CHANNELS, event_types: Object.fromEntries(Object.entries(EVENT_TYPES).map(([k, v]) => [k, v.label])) };
	214	+ });
	215	+ app.get<{ Querystring: { limit?: string; since?: string } }>("/api/v1/clusters", async (req) => {
	216	+ const rows = await db.execute<Record<string, unknown>>(sql`
	217	+ select c.*, (select json_agg(json_build_object('id', e.id, 'slug', e.slug, 'title', e.title, 'importance', e.importance, 'event_type', e.event_type, 'detected_at', e.detected_at, 'source_id', e.source_id, 'url', e.url) order by e.importance desc) from events e where e.cluster_id = c.id) as events
	218	+ from event_clusters c where c.last_at >= now() - make_interval(hours => ${Math.min(720, Number(req.query.since ?? 72))}) order by c.last_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
	219	+ return { items: rows.rows };
	220	+ });
	221	+
	222	+ // ---- Search ----------------------------------------------------------------------------
	223	+ app.get<{ Querystring: { q?: string; limit?: string } }>("/api/v1/search", async (req) => {
	224	+ const q = (req.query.q ?? "").trim();
	225	+ if (q.length < 2) return { query: q, events: [], entities: [], sources: [], urls: [] };
	226	+ const lim = Math.min(50, Number(req.query.limit ?? 20));
	227	+ const [ev, ents, srcs, urls] = await Promise.all([
	228	+ listEvents({ q, limit: lim }).then((r) => r.items),
	229	+ db.execute<Record<string, unknown>>(sql`select id, name, type, domain, importance, event_count from entities where search @@ plainto_tsquery('simple', ${q}) or name ilike ${"%" + q + "%"} order by event_count desc, importance desc limit ${lim}`).then((r) => r.rows),
	230	+ db.execute<Record<string, unknown>>(sql`select id, name, domain, tier, categories from sources where name ilike ${"%" + q + "%"} or domain ilike ${"%" + q + "%"} limit ${lim}`).then((r) => r.rows),
	231	+ db.execute<Record<string, unknown>>(sql`select url, domain, status, change_count, last_seen_at from urls where url ilike ${"%" + q + "%"} order by change_count desc limit ${lim}`).then((r) => r.rows),
	232	+ ]);
	233	+ return { query: q, events: ev, entities: ents, sources: srcs, urls };
	234	+ });
	235	+
	236	+ // ---- Connector health ----------------------------------------------------------------------
	237	+ app.get("/api/v1/health/connectors", async () => {
	238	+ const [connectors, sensorsByHealth, worst, noisy, daily] = await Promise.all([
	239	+ db.execute<Record<string, unknown>>(sql`select * from connector_health order by connector`).then((r) => r.rows),
	240	+ db.execute<Record<string, unknown>>(sql`select health, count(*)::int as n from sensors where enabled group by health`).then((r) => r.rows),
	241	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.url, s.source_id, s.connector, s.health, s.consecutive_errors, s.last_error, s.last_status, s.last_check_at from sensors s where s.enabled and s.health <> 'UP' order by s.consecutive_errors desc, s.last_check_at desc limit 50`).then((r) => r.rows),
	242	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.url, s.raw_changes, s.meaningful_changes, case when s.raw_changes > 0 then round(1 - s.meaningful_changes::numeric / s.raw_changes, 3) else null end as noise_ratio from sensors s where s.raw_changes >= 5 order by noise_ratio desc nulls last, raw_changes desc limit 25`).then((r) => r.rows),
	243	+ db.execute<Record<string, unknown>>(sql`select * from metrics_daily order by day desc limit 30`).then((r) => r.rows),
	244	+ ]);
	245	+ return { connectors, sensors_by_health: sensorsByHealth, degraded_sensors: worst, noisy_sensors: noisy, daily, live: liveStats() };
	246	+ });
	247	+
	248	+ // ---- Watchlists & alerts (anonymous owner token, phase 1) ----------------------------------
	249	+ app.get("/api/v1/watchlists", async (req, reply) => {
	250	+ const owner = ownerToken(req.headers as Record<string, unknown>);
	251	+ if (!owner) return reply.status(401).send({ error: "owner token required (X-WebSensor-Owner)" });
	252	+ const rows = await db.execute<Record<string, unknown>>(sql`select w.*, coalesce((select json_agg(json_build_object('kind', i.kind, 'value', i.value, 'added_at', i.added_at)) from watchlist_items i where i.watchlist_id = w.id), '[]'::json) as items from watchlists w where owner_token = ${owner} order by created_at`);
	253	+ return { items: rows.rows };
	254	+ });
	255	+ app.post<{ Body: { name?: string; items?: { kind: string; value: string }[] } }>("/api/v1/watchlists", async (req, reply) => {
	256	+ const owner = ownerToken(req.headers as Record<string, unknown>);
	257	+ if (!owner) return reply.status(401).send({ error: "owner token required" });
	258	+ const body = z.object({ name: z.string().min(1).max(80).default("My watchlist"), items: z.array(z.object({ kind: z.enum(["entity", "source", "keyword", "category", "url"]), value: z.string().min(1).max(200) })).max(200).default([]) }).parse(req.body ?? {});
	259	+ const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from watchlists where owner_token = ${owner}`)).rows[0]?.n;
	260	+ if (Number(count) >= 20) return reply.status(429).send({ error: "too many watchlists" });
	261	+ const id = newId("wl");
	262	+ await db.execute(sql`insert into watchlists (id, owner_token, name) values (${id}, ${owner}, ${body.name})`);
	263	+ for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${id}, ${it.kind}, ${it.value}) on conflict do nothing`);
	264	+ return { id, name: body.name, items: body.items };
	265	+ });
	266	+ app.put<{ Params: { id: string }; Body: { name?: string; items?: { kind: string; value: string }[] } }>("/api/v1/watchlists/:id", async (req, reply) => {
	267	+ const owner = ownerToken(req.headers as Record<string, unknown>);
	268	+ if (!owner) return reply.status(401).send({ error: "owner token required" });
	269	+ const w = (await db.execute<{ id: string }>(sql`select id from watchlists where id = ${req.params.id} and owner_token = ${owner}`)).rows[0];
	270	+ if (!w) return reply.status(404).send({ error: "not_found" });
	271	+ const body = z.object({ name: z.string().min(1).max(80).optional(), items: z.array(z.object({ kind: z.enum(["entity", "source", "keyword", "category", "url"]), value: z.string().min(1).max(200) })).max(200).optional() }).parse(req.body ?? {});
	272	+ if (body.name) await db.execute(sql`update watchlists set name = ${body.name} where id = ${w.id}`);
	273	+ if (body.items) {
	274	+ await db.execute(sql`delete from watchlist_items where watchlist_id = ${w.id}`);
	275	+ for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${w.id}, ${it.kind}, ${it.value}) on conflict do nothing`);
	276	+ }
	277	+ return { ok: true };
	278	+ });
	279	+ app.delete<{ Params: { id: string } }>("/api/v1/watchlists/:id", async (req, reply) => {
	280	+ const owner = ownerToken(req.headers as Record<string, unknown>);
	281	+ if (!owner) return reply.status(401).send({ error: "owner token required" });
	282	+ await db.execute(sql`delete from watchlists where id = ${req.params.id} and owner_token = ${owner}`);
	283	+ return { ok: true };
	284	+ });
	285	+ app.get<{ Params: { id: string }; Querystring: { limit?: string } }>("/api/v1/watchlists/:id/events", async (req, reply) => {
	286	+ const owner = ownerToken(req.headers as Record<string, unknown>);
	287	+ if (!owner) return reply.status(401).send({ error: "owner token required" });
	288	+ const items = (await db.execute<{ kind: string; value: string }>(sql`select i.kind, i.value from watchlist_items i join watchlists w on w.id = i.watchlist_id where w.id = ${req.params.id} and w.owner_token = ${owner}`)).rows;
	289	+ if (!items.length) return reply.send({ items: [] });
	290	+ const ents = items.filter((i) => i.kind === "entity").map((i) => i.value);
	291	+ const srcs = items.filter((i) => i.kind === "source").map((i) => i.value);
	292	+ const cats = items.filter((i) => i.kind === "category").map((i) => i.value);
	293	+ const kws = items.filter((i) => i.kind === "keyword").map((i) => i.value);
	294	+ const conds = [] as ReturnType<typeof sql>[];
	295	+ if (ents.length) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = any(${textArray(ents)}))`);
	296	+ if (srcs.length) conds.push(sql`e.source_id = any(${textArray(srcs)})`);
	297	+ if (cats.length) conds.push(sql`e.categories && ${textArray(cats)}`);
	298	+ for (const k of kws) conds.push(sql`(e.title ilike ${"%" + k + "%"} or e.summary ilike ${"%" + k + "%"})`);
	299	+ if (!conds.length) return { items: [] };
	300	+ const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where ${sql.join(conds, sql` or `)} order by e.detected_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
	301	+ return { items: rows.rows };
	302	+ });
	303	+
	304	+ app.get("/api/v1/alerts", async (req, reply) => {
	305	+ const owner = ownerToken(req.headers as Record<string, unknown>);
	306	+ if (!owner) return reply.status(401).send({ error: "owner token required" });
	307	+ return { items: (await db.execute<Record<string, unknown>>(sql`select * from alerts where owner_token = ${owner} order by created_at`)).rows };
	308	+ });
	309	+ app.post<{ Body: Record<string, unknown> }>("/api/v1/alerts", async (req, reply) => {
	310	+ const owner = ownerToken(req.headers as Record<string, unknown>);
	311	+ if (!owner) return reply.status(401).send({ error: "owner token required" });
	312	+ const body = z.object({ name: z.string().min(1).max(80), rule: z.object({ importance_min: z.number().min(0).max(100).optional(), event_types: z.array(z.string()).optional(), entities: z.array(z.string()).optional(), sources: z.array(z.string()).optional(), keywords: z.array(z.string()).optional(), silent_only: z.boolean().optional(), categories: z.array(z.string()).optional() }), channel: z.enum(["web"]).default("web") }).parse(req.body ?? {});
	313	+ const id = newId("alr");
	314	+ await db.execute(sql`insert into alerts (id, owner_token, name, rule, channel) values (${id}, ${owner}, ${body.name}, ${JSON.stringify(body.rule)}::jsonb, ${body.channel})`);
	315	+ return { id, ...body };
	316	+ });
	317	+ app.delete<{ Params: { id: string } }>("/api/v1/alerts/:id", async (req, reply) => {
	318	+ const owner = ownerToken(req.headers as Record<string, unknown>);
	319	+ if (!owner) return reply.status(401).send({ error: "owner token required" });
	320	+ await db.execute(sql`delete from alerts where id = ${req.params.id} and owner_token = ${owner}`);
	321	+ return { ok: true };
	322	+ });
	323	+
	324	+ // ---- Machine-readable feeds ------------------------------------------------------------------
	325	+ app.get<{ Querystring: { category?: string; importance_min?: string; silent_change?: string } }>("/api/v1/feed.rss", async (req, reply) => {
	326	+ const q = eventsQuery.parse({ ...req.query, limit: 50 });
	327	+ const { items } = await listEvents(q);
	328	+ const esc = (s: unknown): string => String(s ?? "").replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;").replace(/"/g, "&quot;");
	329	+ const base = config.publicBaseUrl;
	330	+ const xml = `<?xml version="1.0" encoding="UTF-8"?>\n<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>WebSensor — ${esc(q.category ? q.category + " events" : "live events")}</title><link>${base}</link><description>Meaningful changes detected on the public Web by WebSensor.</description><atom:link href="${base}/api/v1/feed.rss" rel="self" type="application/rss+xml"/>${items
	331	+ .map((e) => `<item><title>${esc(e.title)}</title><link>${base}/event/${esc(e.slug)}</link><guid isPermaLink="false">${esc(e.id)}</guid><pubDate>${new Date(String(e.detected_at)).toUTCString()}</pubDate><category>${esc(e.event_type)}</category><description>${esc(e.summary)} (importance ${esc(e.importance)}, confidence ${esc(e.confidence)}${e.silent_change ? ", silent change" : ""}) — source: ${esc(e.url)}</description></item>`)
	332	+ .join("")}</channel></rss>`;
	333	+ reply.header("content-type", "application/rss+xml; charset=utf-8");
	334	+ return reply.send(xml);
	335	+ });
	336	+
	337	+ app.get("/api/v1", async () => ({
	338	+ name: "WebSensor API",
	339	+ version: "v1",
	340	+ docs: `${config.publicBaseUrl}/api`,
	341	+ endpoints: ["/api/v1/events", "/api/v1/events/{id|slug}", "/api/v1/changes/{id}", "/api/v1/snapshots/{id}", "/api/v1/snapshots/compare?a=&b=", "/api/v1/sources", "/api/v1/sources/{id}", "/api/v1/sensors/{id}", "/api/v1/entities", "/api/v1/entities/{id}", "/api/v1/entities/{id}/timeline", "/api/v1/domains/{domain}/timeline", "/api/v1/urls/history?url=", "/api/v1/search?q=", "/api/v1/stats", "/api/v1/trending", "/api/v1/explore", "/api/v1/clusters", "/api/v1/health/connectors", "/api/v1/watchlists", "/api/v1/alerts", "/api/v1/feed.rss", "wss://…/api/v1/live"],
	342	+ }));
	343	+}
	344	+
	345	+function renderCanonical(c: { mode: string; text?: string; items?: Record<string, unknown>[]; json?: unknown }): string {
	346	+ if (c.mode === "text") return c.text ?? "";
	347	+ if (c.mode === "list") return (c.items ?? []).map((i) => [i.title ?? i.url ?? i.key, i.url && i.title ? i.url : null, i.summary ? String(i.summary).slice(0, 300) : null].filter(Boolean).join(" — ")).join("\n");
	348	+ return JSON.stringify(c.json ?? null, null, 2);
	349	+}

added apps/api/src/server.ts +107 −0

		@@ -0,0 +1,107 @@
	1	+import Fastify, { type FastifyReply, type FastifyRequest } from "fastify";
	2	+import cors from "@fastify/cors";
	3	+import rateLimit from "@fastify/rate-limit";
	4	+import websocket from "@fastify/websocket";
	5	+import replyFrom from "@fastify/reply-from";
	6	+import client from "prom-client";
	7	+import { newId } from "@websensor/core";
	8	+import { closeDb } from "@websensor/db";
	9	+import { config } from "./config";
	10	+import { closeLive, registerLive } from "./live";
	11	+import { registerRoutes } from "./routes";
	12	+
	13	+/**
	14	+ * WebSensor gateway: public entry point behind ngrok.
	15	+ * /api/v1/* REST + WebSocket (/api/v1/live)
	16	+ * /api/health /api/ready /api/metrics
	17	+ * everything else → Next.js frontend (loopback), with forwarded headers.
	18	+ * Also enforces the canonical host (websensor.io → www.websensor.io).
	19	+ */
	20	+const registry = new client.Registry();
	21	+client.collectDefaultMetrics({ register: registry, prefix: "websensor_api_" });
	22	+const httpRequests = new client.Counter({ name: "websensor_api_requests_total", help: "API requests", labelNames: ["route", "status"], registers: [registry] });
	23	+
	24	+export async function buildServer() {
	25	+ const app = Fastify({
	26	+ logger: {
	27	+ level: config.logLevel,
	28	+ ...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}),
	29	+ redact: ["req.headers.authorization", "req.headers.cookie", "req.headers['x-websensor-owner']"],
	30	+ },
	31	+ trustProxy: true,
	32	+ bodyLimit: 512 * 1024,
	33	+ genReqId: () => newId("req"),
	34	+ disableRequestLogging: config.env === "production",
	35	+ });
	36	+
	37	+ // Canonical host redirect (apex → www) for every path.
	38	+ app.addHook("onRequest", async (req, reply) => {
	39	+ const host = (req.headers.host ?? "").split(":")[0];
	40	+ if (config.redirectApexToWww && host && host !== config.canonicalHost && host === config.canonicalHost.replace(/^www\./, "")) {
	41	+ return reply.redirect(`https://${config.canonicalHost}${req.url}`, 301);
	42	+ }
	43	+ });
	44	+
	45	+ await app.register(cors, { origin: true, methods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"], allowedHeaders: ["content-type", "x-websensor-owner", "authorization"], exposedHeaders: ["x-request-id"] });
	46	+ await app.register(rateLimit, { max: 600, timeWindow: "1 minute", allowList: (req) => !req.url.startsWith("/api/") });
	47	+ await app.register(websocket, { options: { maxPayload: 64 * 1024 } });
	48	+
	49	+ app.addHook("onSend", async (req, reply) => {
	50	+ reply.header("x-request-id", req.id);
	51	+ if (req.url.startsWith("/api/")) {
	52	+ reply.header("cache-control", req.method === "GET" ? "public, max-age=5, stale-while-revalidate=30" : "no-store");
	53	+ httpRequests.inc({ route: req.routeOptions?.url ?? "unknown", status: String(reply.statusCode) });
	54	+ }
	55	+ });
	56	+
	57	+ app.setErrorHandler((err: Error & { statusCode?: number; validation?: unknown; issues?: unknown }, req, reply) => {
	58	+ if (err.name === "ZodError" || err.issues) return reply.status(400).send({ error: "invalid_request", details: err.issues });
	59	+ if (err.statusCode && err.statusCode < 500) return reply.status(err.statusCode).send({ error: err.message });
	60	+ req.log.error({ err }, "unhandled error");
	61	+ return reply.status(500).send({ error: "internal_error", request_id: req.id });
	62	+ });
	63	+
	64	+ app.get("/api/metrics", async (_req, reply) => {
	65	+ reply.header("content-type", registry.contentType);
	66	+ return registry.metrics();
	67	+ });
	68	+
	69	+ await registerRoutes(app);
	70	+ await registerLive(app);
	71	+
	72	+ // Frontend proxy — everything that is not /api/* goes to Next.js (loopback)
	73	+ // with forwarded headers so the app knows the public host/proto and the real client IP.
	74	+ await app.register(replyFrom, { base: config.webUrl, http: { requestOptions: { timeout: 60_000 } }, undici: { connections: 64, pipelining: 1 } });
	75	+ const forward = (req: FastifyRequest, reply: FastifyReply): FastifyReply => {
	76	+ if (req.url.startsWith("/api/")) return reply.status(404).send({ error: "not_found" });
	77	+ return reply.from(req.raw.url ?? "/", {
	78	+ rewriteRequestHeaders: (r, headers) => ({ ...headers, "x-forwarded-host": String(r.headers["x-forwarded-host"] ?? r.headers.host ?? ""), "x-forwarded-proto": String(r.headers["x-forwarded-proto"] ?? "https"), "x-real-ip": r.ip, "x-forwarded-for": r.ip }),
	79	+ });
	80	+ };
	81	+ const methods = ["GET", "HEAD", "POST", "PUT", "DELETE", "PATCH"] as const;
	82	+ app.route({ method: [...methods], url: "/", handler: forward });
	83	+ app.route({ method: [...methods], url: "/*", handler: forward });
	84	+
	85	+ return app;
	86	+}
	87	+
	88	+async function main(): Promise<void> {
	89	+ const app = await buildServer();
	90	+ await app.listen({ port: config.port, host: config.host });
	91	+ app.log.info({ port: config.port, web: config.webUrl, canonical: config.canonicalHost }, "websensor gateway listening");
	92	+ const shutdown = async (): Promise<void> => {
	93	+ await closeLive();
	94	+ await app.close();
	95	+ await closeDb();
	96	+ process.exit(0);
	97	+ };
	98	+ process.on("SIGINT", () => void shutdown());
	99	+ process.on("SIGTERM", () => void shutdown());
	100	+}
	101	+
	102	+if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split("/").pop() ?? "")) {
	103	+ main().catch((e) => {
	104	+ console.error(e);
	105	+ process.exit(1);
	106	+ });
	107	+}

added apps/api/tsconfig.json +5 −0

		@@ -0,0 +1,5 @@
	1	+{
	2	+ "extends": "../../tsconfig.base.json",
	3	+ "compilerOptions": { "types": ["node"] },
	4	+ "include": ["src/**/*.ts"]
	5	+}

added apps/engine/package.json +35 −0

		@@ -0,0 +1,35 @@
	1	+{
	2	+ "name": "@websensor/engine",
	3	+ "version": "0.1.0",
	4	+ "private": true,
	5	+ "type": "module",
	6	+ "scripts": {
	7	+ "dev": "tsx watch src/index.ts",
	8	+ "start": "tsx src/index.ts",
	9	+ "registry:sync": "tsx src/cli.ts sync",
	10	+ "discover": "tsx src/cli.ts discover",
	11	+ "run-once": "tsx src/cli.ts run-once",
	12	+ "typecheck": "tsc -p tsconfig.json --noEmit",
	13	+ "test": "vitest run --passWithNoTests"
	14	+ },
	15	+ "dependencies": {
	16	+ "@anthropic-ai/sdk": "^0.124.0",
	17	+ "@websensor/connectors": "workspace:*",
	18	+ "@websensor/core": "workspace:*",
	19	+ "@websensor/db": "workspace:*",
	20	+ "@websensor/store": "workspace:*",
	21	+ "fastify": "^5.4.0",
	22	+ "ioredis": "^5.6.0",
	23	+ "pino": "^9.7.0",
	24	+ "pino-pretty": "^13.0.0",
	25	+ "prom-client": "^15.1.0",
	26	+ "tsx": "^4.20.0",
	27	+ "yaml": "^2.8.0",
	28	+ "zod": "^4.0.0"
	29	+ },
	30	+ "devDependencies": {
	31	+ "@types/node": "^24.0.0",
	32	+ "typescript": "^5.9.3",
	33	+ "vitest": "^3.2.0"
	34	+ }
	35	+}

added apps/engine/src/cli.ts +81 −0

		@@ -0,0 +1,81 @@
	1	+import { closeDb, db, migrate, sensors, sources, sql } from "@websensor/db";
	2	+import { closeDispatcher, discoverDomain } from "@websensor/connectors";
	3	+import { loadRecent } from "./cluster";
	4	+import { config, log } from "./config";
	5	+import { closeRedis } from "./redis";
	6	+import { runDiscovery, syncRegistry } from "./registry";
	7	+import { runSensor } from "./pipeline";
	8	+import { diffText, evaluateChange } from "@websensor/core";
	9	+import { interpretChange, llmAvailable } from "./interpret";
	10	+
	11	+/**
	12	+ * Operator CLI:
	13	+ * tsx src/cli.ts sync — upsert sources/sensors/entities from config/sources.yaml
	14	+ * tsx src/cli.ts discover [sourceId…] — run discovery (+ promote validated endpoints)
	15	+ * tsx src/cli.ts probe <domain> — discovery dry-run for a domain (no DB writes)
	16	+ * tsx src/cli.ts run-once <sensorId> — run the full pipeline for one sensor now
	17	+ * tsx src/cli.ts run-due [n] — run up to n due sensors sequentially
	18	+ */
	19	+async function main(): Promise<void> {
	20	+ const [cmd, ...args] = process.argv.slice(2);
	21	+ await migrate(config.databaseUrl);
	22	+ switch (cmd) {
	23	+ case "sync":
	24	+ await syncRegistry();
	25	+ break;
	26	+ case "discover":
	27	+ await syncRegistry();
	28	+ await runDiscovery({ sourceIds: args.length ? args : undefined });
	29	+ break;
	30	+ case "probe": {
	31	+ const r = await discoverDomain(args[0]!, { probePages: true });
	32	+ for (const e of r) console.log(`${e.type.padEnd(10)} ${e.value.toFixed(2)} ${String(e.itemCount ?? "").padStart(5)} ${e.url} (${e.evidence})`);
	33	+ break;
	34	+ }
	35	+ case "run-once": {
	36	+ await loadRecent();
	37	+ const s = (await db.select().from(sensors).where(sql`id = ${args[0]}`))[0];
	38	+ if (!s) throw new Error(`sensor ${args[0]} not found`);
	39	+ const src = (await db.select().from(sources).where(sql`id = ${s.sourceId}`))[0]!;
	40	+ const out = await runSensor(s, src);
	41	+ console.log(`${s.id}: ${out}`);
	42	+ break;
	43	+ }
	44	+ case "run-due": {
	45	+ await loadRecent();
	46	+ const n = Number(args[0] ?? 20);
	47	+ const rows = await db.select().from(sensors).where(sql`enabled and next_check_at <= now()`).orderBy(sql`next_check_at asc`).limit(n);
	48	+ const srcs = new Map((await db.select().from(sources)).map((s) => [s.id, s]));
	49	+ for (const s of rows) {
	50	+ const out = await runSensor(s, srcs.get(s.sourceId)!);
	51	+ console.log(`${s.id.padEnd(50)} ${out}`);
	52	+ }
	53	+ break;
	54	+ }
	55	+ case "llm-test": {
	56	+ if (!llmAvailable()) throw new Error("ANTHROPIC_API_KEY not set");
	57	+ const before = "API Pricing\nInput: $10 / million tokens\nOutput: $30 / million tokens\nBatch API: 50% discount";
	58	+ const after = "API Pricing\nInput: $8 / million tokens\nOutput: $24 / million tokens\nBatch API: 50% discount\nPrompt caching: 90% discount on cached input";
	59	+ const diff = diffText(before, after);
	60	+ const heuristic = evaluateChange(diff, { sensorType: "HTML", url: "https://example-ai.com/pricing", sourceCategories: ["ai"], title: "API Pricing" });
	61	+ const t0 = Date.now();
	62	+ const out = await interpretChange({ sourceName: "Example AI", sourceCategories: ["ai"], url: "https://example-ai.com/pricing", sensorName: "pricing", sensorType: "HTML", heuristic, diff, prelimImportance: Number(args[0] ?? 60), title: "API Pricing" });
	63	+ console.log(JSON.stringify(out, null, 2), `\n${Date.now() - t0} ms`);
	64	+ break;
	65	+ }
	66	+ default:
	67	+ console.error("usage: cli.ts sync | discover [sourceId…] | probe <domain> | run-once <sensorId> | run-due [n]");
	68	+ process.exitCode = 1;
	69	+ }
	70	+}
	71	+
	72	+main()
	73	+ .catch((e) => {
	74	+ log.error({ err: (e as Error).stack ?? String(e) }, "cli failed");
	75	+ process.exitCode = 1;
	76	+ })
	77	+ .finally(async () => {
	78	+ await closeDispatcher();
	79	+ await closeRedis();
	80	+ await closeDb();
	81	+ });

added apps/engine/src/cluster.ts +135 −0

		@@ -0,0 +1,135 @@
	1	+import { jaccard, newId, shingles } from "@websensor/core";
	2	+import { db, eventClusters, events, gte, sql, textArray } from "@websensor/db";
	3	+
	4	+/**
	5	+ * Novelty + clustering over a rolling in-memory window of recent events (loaded from
	6	+ * Postgres at startup). Similarity = Jaccard over word 3-shingles of title+summary.
	7	+ */
	8	+interface RecentEvent {
	9	+ id: string;
	10	+ clusterId: string | null;
	11	+ sourceId: string;
	12	+ sensorId: string;
	13	+ eventType: string;
	14	+ entityIds: string[];
	15	+ detectedAt: number;
	16	+ sh: Set<string>;
	17	+ importance: number;
	18	+}
	19	+
	20	+const WINDOW_MS = 72 * 3600e3;
	21	+const CLUSTER_WINDOW_MS = 6 * 3600e3;
	22	+let recent: RecentEvent[] = [];
	23	+let loaded = false;
	24	+
	25	+export async function loadRecent(): Promise<void> {
	26	+ const since = new Date(Date.now() - WINDOW_MS);
	27	+ const rows = await db.execute<{ id: string; cluster_id: string | null; source_id: string; sensor_id: string; event_type: string; title: string; summary: string; detected_at: Date; importance: number; entity_ids: string[] | null }>(sql`
	28	+ select e.id, e.cluster_id, e.source_id, e.sensor_id, e.event_type, e.title, e.summary, e.detected_at, e.importance,
	29	+ (select array_agg(entity_id) from event_entities ee where ee.event_id = e.id) as entity_ids
	30	+ from events e where e.detected_at >= ${since} order by e.detected_at desc limit 3000`);
	31	+ recent = rows.rows.map((r) => ({ id: r.id, clusterId: r.cluster_id, sourceId: r.source_id, sensorId: r.sensor_id, eventType: r.event_type, entityIds: r.entity_ids ?? [], detectedAt: new Date(r.detected_at).getTime(), sh: shingles(`${r.title}\n${r.summary}`), importance: r.importance }));
	32	+ loaded = true;
	33	+}
	34	+
	35	+function prune(): void {
	36	+ const cutoff = Date.now() - WINDOW_MS;
	37	+ recent = recent.filter((r) => r.detectedAt >= cutoff);
	38	+}
	39	+
	40	+export interface NoveltyResult {
	41	+ novelty: number;
	42	+ nearest: { id: string; similarity: number } | null;
	43	+ /** number of distinct sources reporting near-identical content */
	44	+ confirmations: number;
	45	+}
	46	+
	47	+export async function assessNovelty(text: string, sourceId: string): Promise<NoveltyResult> {
	48	+ if (!loaded) await loadRecent();
	49	+ prune();
	50	+ const sh = shingles(text);
	51	+ let best = 0;
	52	+ let nearest: RecentEvent | null = null;
	53	+ const confirmingSources = new Set<string>();
	54	+ for (const r of recent) {
	55	+ const s = jaccard(sh, r.sh);
	56	+ if (s > best) {
	57	+ best = s;
	58	+ nearest = r;
	59	+ }
	60	+ if (s >= 0.45 && r.sourceId !== sourceId) confirmingSources.add(r.sourceId);
	61	+ }
	62	+ return { novelty: Math.round((1 - best) * 100), nearest: nearest ? { id: nearest.id, similarity: Math.round(best * 100) / 100 } : null, confirmations: confirmingSources.size };
	63	+}
	64	+
	65	+export interface ClusterDecision {
	66	+ clusterId: string;
	67	+ created: boolean;
	68	+}
	69	+
	70	+/**
	71	+ * Attach to an existing open cluster when the event shares an entity (or the same source)
	72	+ * with a recent event and is textually related, or when it is the same event type on the
	73	+ * same source within 30 minutes (e.g. one launch touching six pages). Otherwise open one.
	74	+ */
	75	+export async function clusterEvent(ev: { id: string; sourceId: string; sensorId: string; eventType: string; entityIds: string[]; detectedAt: Date; title: string; summary: string; importance: number; categories: string[] }): Promise<ClusterDecision> {
	76	+ if (!loaded) await loadRecent();
	77	+ const sh = shingles(`${ev.title}\n${ev.summary}`);
	78	+ const now = ev.detectedAt.getTime();
	79	+ let bestCluster: string | null = null;
	80	+ let bestScore = 0;
	81	+ for (const r of recent) {
	82	+ if (!r.clusterId || now - r.detectedAt > CLUSTER_WINDOW_MS) continue;
	83	+ const sharedEntity = r.entityIds.some((e) => ev.entityIds.includes(e));
	84	+ const sameSource = r.sourceId === ev.sourceId;
	85	+ if (!sharedEntity && !sameSource) continue;
	86	+ const sim = jaccard(sh, r.sh);
	87	+ const closeInTime = now - r.detectedAt < 30 * 60e3;
	88	+ let score = 0;
	89	+ if (sim >= 0.22) score = sim + (sharedEntity ? 0.2 : 0);
	90	+ else if (sameSource && r.eventType === ev.eventType && closeInTime && r.sensorId !== ev.sensorId) score = 0.3;
	91	+ else if (sameSource && closeInTime && sim >= 0.12) score = 0.25;
	92	+ if (score > bestScore) {
	93	+ bestScore = score;
	94	+ bestCluster = r.clusterId;
	95	+ }
	96	+ }
	97	+ let clusterId: string;
	98	+ let created = false;
	99	+ if (bestCluster) {
	100	+ clusterId = bestCluster;
	101	+ await db.execute(sql`update event_clusters set event_count = event_count + 1, last_at = greatest(last_at, ${ev.detectedAt}), max_importance = greatest(max_importance, ${ev.importance}),
	102	+ entity_ids = (select array(select distinct unnest(entity_ids || ${textArray(ev.entityIds)}))),
	103	+ categories = (select array(select distinct unnest(categories || ${textArray(ev.categories)}))),
	104	+ title = case when ${ev.importance} > max_importance then ${ev.title} else title end,
	105	+ primary_event_id = case when ${ev.importance} > max_importance then ${ev.id} else primary_event_id end
	106	+ where id = ${clusterId}`);
	107	+ } else {
	108	+ clusterId = newId("clu");
	109	+ created = true;
	110	+ await db.insert(eventClusters).values({ id: clusterId, title: ev.title, summary: ev.summary, primaryEventId: ev.id, entityIds: ev.entityIds, categories: ev.categories, eventCount: 1, maxImportance: ev.importance, firstAt: ev.detectedAt, lastAt: ev.detectedAt });
	111	+ }
	112	+ recent.unshift({ id: ev.id, clusterId, sourceId: ev.sourceId, sensorId: ev.sensorId, eventType: ev.eventType, entityIds: ev.entityIds, detectedAt: now, sh, importance: ev.importance });
	113	+ if (recent.length > 5000) recent.length = 5000;
	114	+ return { clusterId, created };
	115	+}
	116	+
	117	+/** Recent events of the same source published through announcement-type sensors (for silent-change detection). */
	118	+export function recentAnnouncementSimilarity(sourceId: string, text: string, sinceMs: number): number {
	119	+ const sh = shingles(text);
	120	+ let best = 0;
	121	+ const cutoff = Date.now() - sinceMs;
	122	+ for (const r of recent) {
	123	+ if (r.sourceId !== sourceId || r.detectedAt < cutoff) continue;
	124	+ if (!/announcement|product_launch|model_release|software_release|repository_release|incident|outage|maintenance|security_advisory/.test(r.eventType)) continue;
	125	+ best = Math.max(best, jaccard(sh, r.sh));
	126	+ }
	127	+ return best;
	128	+}
	129	+
	130	+export async function recentClusterCount(sinceMs: number): Promise<number> {
	131	+ const rows = await db.select({ n: sql<number>`count(*)` }).from(eventClusters).where(gte(eventClusters.lastAt, new Date(Date.now() - sinceMs)));
	132	+ return Number(rows[0]?.n ?? 0);
	133	+}
	134	+
	135	+export { events };

added apps/engine/src/config.ts +40 −0

		@@ -0,0 +1,40 @@
	1	+import pino from "pino";
	2	+
	3	+const env = process.env;
	4	+
	5	+export const config = {
	6	+ env: env.NODE_ENV ?? "development",
	7	+ logLevel: env.LOG_LEVEL ?? "info",
	8	+ databaseUrl: env.DATABASE_URL ?? "postgres://localhost:5432/websensor",
	9	+ redisUrl: env.REDIS_URL ?? "redis://127.0.0.1:6379",
	10	+ sourcesFile: env.WS_SOURCES_FILE ?? "./config/sources.yaml",
	11	+ entitiesFile: env.WS_ENTITIES_FILE ?? "./config/entities.yaml",
	12	+ fetchConcurrency: Number(env.WS_FETCH_CONCURRENCY ?? 16),
	13	+ perHostConcurrency: Number(env.WS_PER_HOST_CONCURRENCY ?? 2),
	14	+ metricsPort: Number(env.ENGINE_METRICS_PORT ?? 8262),
	15	+ metricsHost: env.ENGINE_METRICS_HOST ?? "127.0.0.1",
	16	+ processingVersion: "event-pipeline-v1",
	17	+ /** heuristic signal threshold to promote a raw change to an event */
	18	+ meaningfulSignal: Number(env.WS_MEANINGFUL_SIGNAL ?? 0.32),
	19	+ llm: {
	20	+ apiKey: env.ANTHROPIC_API_KEY ?? "",
	21	+ modelFast: env.WS_LLM_MODEL_FAST ?? "claude-haiku-4-5",
	22	+ modelDeep: env.WS_LLM_MODEL_DEEP ?? "claude-opus-5",
	23	+ dailyCallBudget: Number(env.WS_LLM_DAILY_CALL_BUDGET ?? 600),
	24	+ minImportance: Number(env.WS_LLM_MIN_IMPORTANCE ?? 35),
	25	+ deepMinImportance: Number(env.WS_LLM_DEEP_MIN_IMPORTANCE ?? 78),
	26	+ },
	27	+ discovery: {
	28	+ enabled: (env.WS_DISCOVERY ?? "1") !== "0",
	29	+ /** re-run discovery for a source after this many days */
	30	+ intervalDays: Number(env.WS_DISCOVERY_INTERVAL_DAYS ?? 7),
	31	+ },
	32	+ deletion: { confirmations: Number(env.WS_DELETE_CONFIRMATIONS ?? 3), minSeparationMin: Number(env.WS_DELETE_SEPARATION_MIN ?? 60) },
	33	+ version: "0.1.0",
	34	+};
	35	+
	36	+export const log = pino({
	37	+ level: config.logLevel,
	38	+ ...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}),
	39	+ base: { service: "websensor-engine" },
	40	+});

added apps/engine/src/entities.ts +62 −0

		@@ -0,0 +1,62 @@
	1	+import { db, entityAliases, entities, eq, sourceEntities, sql, textArray } from "@websensor/db";
	2	+import { log } from "./config";
	3	+
	4	+/**
	5	+ * Entity resolution. The source's own entities are always attached (subject). Then aliases
	6	+ * of all known entities are matched against the event text (word-boundary, case-insensitive,
	7	+ * longest alias first) so products/models/regulators named in the change are linked too.
	8	+ */
	9	+interface AliasIndex {
	10	+ loadedAt: number;
	11	+ byLen: { alias: string; re: RegExp; entityId: string }[];
	12	+ importance: Map<string, number>;
	13	+}
	14	+
	15	+let index: AliasIndex | null = null;
	16	+
	17	+async function loadIndex(): Promise<AliasIndex> {
	18	+ if (index && Date.now() - index.loadedAt < 5 * 60_000) return index;
	19	+ const rows = await db.select({ alias: entityAliases.alias, entityId: entityAliases.entityId }).from(entityAliases);
	20	+ const ents = await db.select({ id: entities.id, importance: entities.importance }).from(entities);
	21	+ const byLen = rows
	22	+ .filter((r) => r.alias.length >= 3 && !/^\d+$/.test(r.alias))
	23	+ .map((r) => ({ alias: r.alias, entityId: r.entityId, re: new RegExp(`(^|[^\\p{L}\\p{N}])${escapeRe(r.alias)}(?=$|[^\\p{L}\\p{N}])`, "iu") }))
	24	+ .sort((a, b) => b.alias.length - a.alias.length);
	25	+ index = { loadedAt: Date.now(), byLen, importance: new Map(ents.map((e) => [e.id, e.importance])) };
	26	+ log.debug({ aliases: byLen.length }, "alias index loaded");
	27	+ return index;
	28	+}
	29	+
	30	+function escapeRe(s: string): string {
	31	+ return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
	32	+}
	33	+
	34	+export function invalidateEntityIndex(): void {
	35	+ index = null;
	36	+}
	37	+
	38	+export async function resolveEntities(input: { sourceId: string; text: string; hints: string[] }): Promise<{ subject: string[]; mentioned: string[]; importance: number }> {
	39	+ const idx = await loadIndex();
	40	+ const subject = (await db.select({ entityId: sourceEntities.entityId }).from(sourceEntities).where(eq(sourceEntities.sourceId, input.sourceId))).map((r) => r.entityId);
	41	+ // Only the organization (not every product) is a default subject; products are attached when named.
	42	+ const orgSubjects = subject.filter((s) => s.startsWith("org_"));
	43	+ const mentioned = new Set<string>();
	44	+ const hay = `${input.text}\n${input.hints.join("\n")}`;
	45	+ const hayLower = hay.toLowerCase();
	46	+ for (const a of idx.byLen) {
	47	+ if (mentioned.size >= 12) break;
	48	+ if (orgSubjects.includes(a.entityId)) continue;
	49	+ if (!hayLower.includes(a.alias)) continue;
	50	+ if (a.re.test(hay)) mentioned.add(a.entityId);
	51	+ }
	52	+ // Products of the source named in the text are subjects too.
	53	+ for (const s of subject) if (!s.startsWith("org_") && mentioned.has(s)) mentioned.delete(s), orgSubjects.push(s);
	54	+ const all = [...orgSubjects, ...mentioned];
	55	+ const importance = all.length ? Math.max(...all.map((e) => idx.importance.get(e) ?? 40)) : 40;
	56	+ return { subject: orgSubjects, mentioned: [...mentioned], importance };
	57	+}
	58	+
	59	+export async function bumpEntityCounters(entityIds: string[], at: Date): Promise<void> {
	60	+ if (!entityIds.length) return;
	61	+ await db.execute(sql`update entities set event_count = event_count + 1, last_event_at = ${at} where id = any(${textArray(entityIds)})`);
	62	+}

added apps/engine/src/index.ts +52 −0

		@@ -0,0 +1,52 @@
	1	+import { closeDb, migrate } from "@websensor/db";
	2	+import { closeDispatcher } from "@websensor/connectors";
	3	+import { loadRecent } from "./cluster";
	4	+import { config, log } from "./config";
	5	+import { startMetricsServer } from "./metrics";
	6	+import { closeRedis } from "./redis";
	7	+import { runDiscovery, syncRegistry } from "./registry";
	8	+import { pruneOldRuns, rollupConnectorHealth, Scheduler } from "./scheduler";
	9	+
	10	+async function main(): Promise<void> {
	11	+ log.info({ env: config.env, version: config.version, llm: Boolean(config.llm.apiKey) }, "websensor engine starting");
	12	+ const applied = await migrate(config.databaseUrl);
	13	+ if (applied.length) log.info({ applied }, "migrations applied");
	14	+ await syncRegistry();
	15	+ await loadRecent();
	16	+ await startMetricsServer();
	17	+
	18	+ const scheduler = new Scheduler();
	19	+ await scheduler.start();
	20	+
	21	+ const timers: NodeJS.Timeout[] = [];
	22	+ timers.push(setInterval(() => rollupConnectorHealth().catch((e) => log.warn({ err: (e as Error).message }, "health rollup failed")), 60_000));
	23	+ timers.push(setInterval(() => pruneOldRuns().catch(() => undefined), 6 * 3600e3));
	24	+ await rollupConnectorHealth().catch(() => undefined);
	25	+
	26	+ if (config.discovery.enabled) {
	27	+ // Background: validate feeds/sitemaps/status pages for sources not discovered recently, then weekly.
	28	+ setTimeout(() => runDiscovery({ onlyMissing: true }).catch((e) => log.warn({ err: (e as Error).message }, "discovery failed")), 5_000);
	29	+ timers.push(setInterval(() => runDiscovery({ onlyMissing: true }).catch(() => undefined), 24 * 3600e3));
	30	+ }
	31	+
	32	+ let shuttingDown = false;
	33	+ const shutdown = async (signal: string): Promise<void> => {
	34	+ if (shuttingDown) return;
	35	+ shuttingDown = true;
	36	+ log.info({ signal }, "shutting down");
	37	+ for (const t of timers) clearInterval(t);
	38	+ await scheduler.stop();
	39	+ await closeDispatcher();
	40	+ await closeRedis();
	41	+ await closeDb();
	42	+ process.exit(0);
	43	+ };
	44	+ process.on("SIGINT", () => void shutdown("SIGINT"));
	45	+ process.on("SIGTERM", () => void shutdown("SIGTERM"));
	46	+ process.on("unhandledRejection", (e) => log.error({ err: e instanceof Error ? e.stack : String(e) }, "unhandled rejection"));
	47	+}
	48	+
	49	+main().catch((e) => {
	50	+ log.fatal({ err: (e as Error).stack ?? String(e) }, "engine failed to start");
	51	+ process.exit(1);
	52	+});

added apps/engine/src/interpret.ts +153 −0

		@@ -0,0 +1,153 @@
	1	+import Anthropic from "@anthropic-ai/sdk";
	2	+import { EVENT_TYPES, type DiffResult, type HeuristicResult } from "@websensor/core";
	3	+import { db, llmUsage, sql } from "@websensor/db";
	4	+import { config, log } from "./config";
	5	+import { bumpDaily, m } from "./metrics";
	6	+
	7	+/**
	8	+ * Stage-2 interpretation with Claude. Only called for candidates that already cleared the
	9	+ * heuristic bar and a preliminary importance threshold; a daily call budget caps spend.
	10	+ * Routine changes go to the fast model; high-importance candidates get the deep model.
	11	+ * Output is strictly structured (JSON schema) — never free text.
	12	+ */
	13	+export interface Interpretation {
	14	+ event_type: string;
	15	+ title: string;
	16	+ summary: string;
	17	+ why_it_matters: string;
	18	+ who_it_affects: string;
	19	+ observed: string;
	20	+ inferred: string;
	21	+ meaningful: boolean;
	22	+ severity: number;
	23	+ confidence: number;
	24	+ entities: string[];
	25	+ keywords: string[];
	26	+ announced: boolean | null;
	27	+ model: string;
	28	+ input_tokens: number;
	29	+ output_tokens: number;
	30	+}
	31	+
	32	+const client = config.llm.apiKey ? new Anthropic({ apiKey: config.llm.apiKey, timeout: 60_000, maxRetries: 1 }) : null;
	33	+
	34	+const SYSTEM = `You are the interpretation stage of WebSensor, a platform that monitors official public web sources and turns detected changes into precise, sober intelligence events.
	35	+You receive a detected change on one monitored endpoint: the source, the URL, the sensor kind, the heuristic pre-classification, extracted facts and the diff (before → after, or new/removed items).
	36	+Your job: decide whether the change is meaningful to a professional audience, classify it, and write a factual title and summary.
	37	+Rules:
	38	+- Never sensationalize. No marketing tone. State what is observed; keep inference clearly separate.
	39	+- "observed" must only contain facts visible in the diff. "inferred" may contain a cautious interpretation or be empty.
	40	+- Titles: ≤ 110 characters, start with the organization name, no trailing period, no emoji.
	41	+- Summary: 1–3 sentences, concrete (numbers, names, versions when present).
	42	+- meaningful=false for cosmetic edits, typos, navigation/footer churn, copyright years, tracking or timestamp noise, and generic marketing rewording with no new information.
	43	+- severity: 0–100 intrinsic importance of this kind of change for people who follow this organization (pricing/security/outage/model launch high; doc typo low).
	44	+- confidence: 0–100 how sure you are of the classification given the evidence.
	45	+- entities: organizations, products, models, APIs, drugs, standards explicitly named in the change (canonical names).
	46	+- announced: true if the diff itself is an announcement (news/blog/release item), false if it is a silent modification of an existing page, null if unclear.
	47	+Allowed event_type values: ${Object.keys(EVENT_TYPES).join(", ")}.`;
	48	+
	49	+const schema = {
	50	+ type: "object",
	51	+ additionalProperties: false,
	52	+ properties: {
	53	+ event_type: { type: "string", enum: Object.keys(EVENT_TYPES) },
	54	+ title: { type: "string" },
	55	+ summary: { type: "string" },
	56	+ why_it_matters: { type: "string" },
	57	+ who_it_affects: { type: "string" },
	58	+ observed: { type: "string" },
	59	+ inferred: { type: "string" },
	60	+ meaningful: { type: "boolean" },
	61	+ severity: { type: "integer", description: "0–100" },
	62	+ confidence: { type: "integer", description: "0–100" },
	63	+ entities: { type: "array", items: { type: "string" } },
	64	+ keywords: { type: "array", items: { type: "string" } },
	65	+ announced: { type: ["boolean", "null"] },
	66	+ },
	67	+ required: ["event_type", "title", "summary", "why_it_matters", "who_it_affects", "observed", "inferred", "meaningful", "severity", "confidence", "entities", "keywords", "announced"],
	68	+} as const;
	69	+
	70	+let budgetDay = "";
	71	+let budgetUsed = 0;
	72	+
	73	+async function loadBudget(): Promise<void> {
	74	+ const today = new Date().toISOString().slice(0, 10);
	75	+ if (budgetDay === today) return;
	76	+ budgetDay = today;
	77	+ const r = await db.execute<{ n: string }>(sql`select count(*)::text as n from llm_usage where at >= current_date`);
	78	+ budgetUsed = Number(r.rows[0]?.n ?? 0);
	79	+}
	80	+
	81	+export function llmAvailable(): boolean {
	82	+ return client !== null;
	83	+}
	84	+
	85	+export async function interpretChange(input: { sourceName: string; sourceCategories: string[]; url: string; sensorName: string; sensorType: string; heuristic: HeuristicResult; diff: DiffResult; prelimImportance: number; title?: string | null }): Promise<Interpretation | null> {
	86	+ if (!client) return null;
	87	+ await loadBudget();
	88	+ if (budgetUsed >= config.llm.dailyCallBudget) {
	89	+ log.warn({ used: budgetUsed }, "LLM daily budget exhausted — heuristics only");
	90	+ return null;
	91	+ }
	92	+ const deep = input.prelimImportance >= config.llm.deepMinImportance;
	93	+ const model = deep ? config.llm.modelDeep : config.llm.modelFast;
	94	+ const diffText = renderDiff(input.diff).slice(0, 14_000);
	95	+ const user = `SOURCE: ${input.sourceName} (${input.sourceCategories.join(", ") || "n/a"})
	96	+URL: ${input.url}
	97	+SENSOR: ${input.sensorName} [${input.sensorType}]${input.title ? `\nPAGE TITLE: ${input.title}` : ""}
	98	+HEURISTIC: type=${input.heuristic.eventType} signal=${input.heuristic.signal.toFixed(2)} magnitude=${input.heuristic.magnitude} noise=${input.heuristic.noiseRatio.toFixed(2)} reasons=${input.heuristic.reasons.join("; ") || "none"}
	99	+FACTS: ${input.heuristic.facts.map((f) => `${f.kind}: ${f.before ?? "∅"} → ${f.after ?? "∅"}`).join(" | ") || "none"}
	100	+
	101	+DIFF:
	102	+${diffText}`;
	103	+
	104	+ budgetUsed++;
	105	+ const started = Date.now();
	106	+ try {
	107	+ // Haiku 4.5 does not accept `effort`; the deep model (Opus 5) runs adaptive thinking at medium effort.
	108	+ const params: Anthropic.MessageCreateParamsNonStreaming = {
	109	+ model,
	110	+ max_tokens: 1500,
	111	+ system: [{ type: "text", text: SYSTEM, cache_control: { type: "ephemeral" } }],
	112	+ messages: [{ role: "user", content: user }],
	113	+ output_config: { format: { type: "json_schema", schema: schema as unknown as Record<string, unknown> }, ...(deep ? { effort: "medium" as const } : {}) },
	114	+ };
	115	+ const res = await client.messages.create(params);
	116	+ const text = res.content.find((b) => b.type === "text")?.text ?? "";
	117	+ if (res.stop_reason === "refusal" || !text) throw new Error(`no text (stop_reason=${res.stop_reason})`);
	118	+ const parsed = JSON.parse(text) as Omit<Interpretation, "model" | "input_tokens" | "output_tokens">;
	119	+ const usage = { input: res.usage.input_tokens + (res.usage.cache_read_input_tokens ?? 0) + (res.usage.cache_creation_input_tokens ?? 0), output: res.usage.output_tokens };
	120	+ m.llmCalls.inc({ model, ok: "true" });
	121	+ m.llmTokens.inc({ model, direction: "input" }, usage.input);
	122	+ m.llmTokens.inc({ model, direction: "output" }, usage.output);
	123	+ await db.insert(llmUsage).values({ model, purpose: "interpret", inputTokens: usage.input, outputTokens: usage.output, ok: true });
	124	+ await bumpDaily({ llm_calls: 1, llm_input_tokens: usage.input, llm_output_tokens: usage.output });
	125	+ log.debug({ model, ms: Date.now() - started, type: parsed.event_type, meaningful: parsed.meaningful }, "llm interpretation");
	126	+ if (!(parsed.event_type in EVENT_TYPES)) parsed.event_type = input.heuristic.eventType;
	127	+ parsed.severity = Math.max(0, Math.min(100, Math.round(Number(parsed.severity) || 0)));
	128	+ parsed.confidence = Math.max(0, Math.min(100, Math.round(Number(parsed.confidence) || 0)));
	129	+ return { ...parsed, model, input_tokens: usage.input, output_tokens: usage.output };
	130	+ } catch (e) {
	131	+ m.llmCalls.inc({ model, ok: "false" });
	132	+ await db.insert(llmUsage).values({ model, purpose: "interpret", inputTokens: 0, outputTokens: 0, ok: false }).catch(() => undefined);
	133	+ const err = e as Error & { status?: number };
	134	+ log.warn({ model, status: err.status, err: err.message }, "llm interpretation failed — falling back to heuristics");
	135	+ return null;
	136	+ }
	137	+}
	138	+
	139	+export function renderDiff(d: DiffResult): string {
	140	+ if (d.kind === "text") {
	141	+ const lines: string[] = [];
	142	+ for (const mo of d.modified.slice(0, 60)) lines.push(`- ${mo.before}\n+ ${mo.after}`);
	143	+ for (const r of d.removed.slice(0, 60)) lines.push(`- ${r}`);
	144	+ for (const a of d.added.slice(0, 80)) lines.push(`+ ${a}`);
	145	+ return lines.join("\n");
	146	+ }
	147	+ if (d.kind === "json") return d.unified;
	148	+ const lines: string[] = [];
	149	+ for (const a of d.added.slice(0, 30)) lines.push(`+ NEW: ${String(a.title ?? a.url ?? a.key)}${a.summary ? `\n ${String(a.summary).slice(0, 500)}` : ""}${a.url ? `\n ${String(a.url)}` : ""}${a.publishedAt ? `\n published ${String(a.publishedAt)}` : ""}`);
	150	+ for (const r of d.removed.slice(0, 30)) lines.push(`- REMOVED: ${String(r.title ?? r.url ?? r.key)}`);
	151	+ for (const mo of d.modified.slice(0, 30)) lines.push(`~ UPDATED: ${String(mo.after.title ?? mo.key)} (${mo.fields.join(", ")})\n before: ${mo.fields.map((f) => `${f}=${JSON.stringify(mo.before[f])}`).join(" ")}\n after: ${mo.fields.map((f) => `${f}=${JSON.stringify(mo.after[f])}`).join(" ")}`);
	152	+ return lines.join("\n");
	153	+}

added apps/engine/src/metrics.ts +46 −0

		@@ -0,0 +1,46 @@
	1	+import Fastify from "fastify";
	2	+import client from "prom-client";
	3	+import { db, sql } from "@websensor/db";
	4	+import { config, log } from "./config";
	5	+
	6	+export const registry = new client.Registry();
	7	+client.collectDefaultMetrics({ register: registry, prefix: "websensor_engine_" });
	8	+
	9	+export const m = {
	10	+ checks: new client.Counter({ name: "websensor_checks_total", help: "Sensor checks", labelNames: ["connector", "outcome"], registers: [registry] }),
	11	+ bytes: new client.Counter({ name: "websensor_fetched_bytes_total", help: "Bytes fetched", labelNames: ["connector"], registers: [registry] }),
	12	+ changes: new client.Counter({ name: "websensor_changes_total", help: "Raw changes", labelNames: ["connector", "meaningful"], registers: [registry] }),
	13	+ events: new client.Counter({ name: "websensor_events_total", help: "Events published", labelNames: ["event_type", "silent"], registers: [registry] }),
	14	+ llmCalls: new client.Counter({ name: "websensor_llm_calls_total", help: "LLM calls", labelNames: ["model", "ok"], registers: [registry] }),
	15	+ llmTokens: new client.Counter({ name: "websensor_llm_tokens_total", help: "LLM tokens", labelNames: ["model", "direction"], registers: [registry] }),
	16	+ fetchDuration: new client.Histogram({ name: "websensor_fetch_duration_seconds", help: "Fetch latency", labelNames: ["connector"], buckets: [0.1, 0.25, 0.5, 1, 2, 5, 10, 25], registers: [registry] }),
	17	+ processingLatency: new client.Histogram({ name: "websensor_processing_latency_seconds", help: "Detected → published", buckets: [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 15], registers: [registry] }),
	18	+ queueDue: new client.Gauge({ name: "websensor_sensors_due", help: "Sensors due for a check", registers: [registry] }),
	19	+ inflight: new client.Gauge({ name: "websensor_inflight_checks", help: "Checks in flight", registers: [registry] }),
	20	+ httpStatus: new client.Counter({ name: "websensor_http_status_total", help: "HTTP status codes", labelNames: ["status"], registers: [registry] }),
	21	+};
	22	+
	23	+/** Daily counters in Postgres (feed the public homepage stats). */
	24	+export async function bumpDaily(fields: Partial<Record<"checks" | "not_modified" | "bytes" | "raw_changes" | "events" | "silent_events" | "errors" | "llm_calls" | "llm_input_tokens" | "llm_output_tokens" | "scrapfly_calls", number>>): Promise<void> {
	25	+ const keys = Object.keys(fields) as (keyof typeof fields)[];
	26	+ if (!keys.length) return;
	27	+ const sets = keys.map((k) => sql.raw(`${k} = metrics_daily.${k} + ${Number(fields[k] ?? 0)}`));
	28	+ const cols = keys.map((k) => sql.raw(k));
	29	+ const vals = keys.map((k) => sql`${Number(fields[k] ?? 0)}`);
	30	+ try {
	31	+ await db.execute(sql`insert into metrics_daily (day, ${sql.join(cols, sql`, `)}) values (current_date, ${sql.join(vals, sql`, `)}) on conflict (day) do update set ${sql.join(sets, sql`, `)}`);
	32	+ } catch (e) {
	33	+ log.warn({ err: (e as Error).message }, "metrics_daily update failed");
	34	+ }
	35	+}
	36	+
	37	+export async function startMetricsServer(): Promise<void> {
	38	+ const app = Fastify({ logger: false });
	39	+ app.get("/metrics", async (_req, reply) => {
	40	+ reply.header("content-type", registry.contentType);
	41	+ return registry.metrics();
	42	+ });
	43	+ app.get("/health", async () => ({ status: "ok", service: "engine", version: config.version, time: new Date().toISOString() }));
	44	+ await app.listen({ port: config.metricsPort, host: config.metricsHost });
	45	+ log.info({ port: config.metricsPort }, "engine metrics listening");
	46	+}

added apps/engine/src/pipeline.ts +404 −0

Large diff (404 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/redis.ts +41 −0

Large diff (41 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/registry.ts +179 −0

Large diff (179 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/scheduler.ts +162 −0

Large diff (162 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/README.md +18 −0

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/eslint.config.mjs +6 −0

Large diff (6 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/next.config.ts +23 −0

Large diff (23 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/package.json +32 −0

Large diff (32 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/postcss.config.mjs +2 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/alerts/alerts.tsx +186 −0

Large diff (186 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/alerts/page.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/api/page.tsx +131 −0

Large diff (131 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/bot/page.tsx +36 −0

Large diff (36 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/breaking/page.tsx +17 −0

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/category/[channel]/page.tsx +29 −0

Large diff (29 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/company/[id]/page.tsx +111 −0

Large diff (111 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/compare/page.tsx +24 −0

Large diff (24 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/domain/[domain]/page.tsx +51 −0

Large diff (51 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/entities/page.tsx +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/error.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/event/[slug]/page.tsx +254 −0

Large diff (254 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/explore/page.tsx +112 −0

Large diff (112 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/globals.css +161 −0

Large diff (161 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/health/page.tsx +107 −0

Large diff (107 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/icon.svg +1 −0

Large diff (1 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/layout.tsx +39 −0

Large diff (39 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/manifest.ts +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/not-found.tsx +15 −0

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/opengraph-image.tsx +33 −0

Large diff (33 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/page.tsx +28 −0

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/robots.ts +10 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/search/page.tsx +74 −0

Large diff (74 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/sensor/[id]/page.tsx +105 −0

Large diff (105 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/silent/page.tsx +17 −0

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/sitemap.ts +19 −0

Large diff (19 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/source/[id]/page.tsx +134 −0

Large diff (134 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/sources/page.tsx +53 −0

Large diff (53 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/timeline/[id]/page.tsx +33 −0

Large diff (33 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/url/compare-form.tsx +35 −0

Large diff (35 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/url/page.tsx +72 −0

Large diff (72 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/watchlists/page.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/watchlists/watchlists.tsx +154 −0

Large diff (154 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/code.tsx +27 −0

Large diff (27 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/diff-viewer.tsx +165 −0

Large diff (165 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/event-row.tsx +48 −0

Large diff (48 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/footer.tsx +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/live-feed.tsx +135 −0

Large diff (135 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/nav.tsx +105 −0

Large diff (105 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/rail.tsx +54 −0

Large diff (54 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/stats-strip.tsx +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/theme.tsx +30 −0

Large diff (30 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/timeline.tsx +38 −0

Large diff (38 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/ui.tsx +167 −0

Large diff (167 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/watch-button.tsx +54 −0

Large diff (54 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/api.ts +491 −0

Large diff (491 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/format.ts +143 −0

Large diff (143 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/owner.ts +37 −0

Large diff (37 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/use-live.ts +80 −0

Large diff (80 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/tsconfig.json +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added config/sources.yaml +2088 −0

Large diff (2,088 lines) is not loaded by default to keep the page fast.Load diff

added deploy/README.md +44 −0

Large diff (44 lines) is not loaded by default to keep the page fast.Load diff

added deploy/websensor.mld.json.example +128 −0

Large diff (128 lines) is not loaded by default to keep the page fast.Load diff

added docs/ARCHITECTURE.md +49 −0

Large diff (49 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/discovery.md +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/github.md +10 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/http.md +16 −0

Large diff (16 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/jsonlist.md +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/rss.md +15 −0

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/scrapfly.md +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/sitemap.md +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added docs/connectors/statuspage.md +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added package.json +31 −0

Large diff (31 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/package.json +23 −0

Large diff (23 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/connectors.test.ts +89 −0

Large diff (89 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/discovery.ts +150 −0

Large diff (150 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/fetcher.ts +238 −0

Large diff (238 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/github.ts +64 −0

Large diff (64 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/http.ts +100 −0

Large diff (100 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/index.ts +38 −0

Large diff (38 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/jsonlist.ts +89 −0

Large diff (89 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/rss.ts +161 −0

Large diff (161 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/scrapfly.ts +68 −0

Large diff (68 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/sitemap.ts +106 −0

Large diff (106 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/statuspage.ts +69 −0

Large diff (69 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/types.ts +28 −0

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/xml.ts +65 −0

Large diff (65 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/package.json +26 −0

Large diff (26 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/canonical.ts +196 −0

Large diff (196 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/client.ts +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/core.test.ts +131 −0

Large diff (131 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/diff.ts +176 −0

Large diff (176 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/hash.ts +65 −0

Large diff (65 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/heuristics.ts +242 −0

Large diff (242 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/ids.ts +36 −0

Large diff (36 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/index.ts +10 −0

Large diff (10 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/schedule.ts +53 −0

Large diff (53 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/scoring.ts +116 −0

Large diff (116 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/ssrf.ts +189 −0

Large diff (189 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/taxonomy.ts +185 −0

Large diff (185 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/types.ts +102 −0

Large diff (102 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/migrations/0001_init.sql +361 −0

Large diff (361 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/migrations/0002_scrapfly.sql +1 −0

Large diff (1 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/migrations/0003_latency_bigint.sql +2 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/package.json +31 −0

Large diff (31 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/src/index.ts +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/src/migrate.ts +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/src/schema.ts +342 −0

Large diff (342 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added packages/store/package.json +21 −0

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

added packages/store/src/index.ts +69 −0

Large diff (69 lines) is not loaded by default to keep the page fast.Load diff

added packages/store/tsconfig.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added pnpm-lock.yaml +6482 −0

Large diff (6,482 lines) is not loaded by default to keep the page fast.Load diff

added pnpm-workspace.yaml +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/atom.xml +4 −0

Large diff (4 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/kev.json +3 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/rss-after.xml +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/rss-before.xml +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/sitemap.xml +6 −0

Large diff (6 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/statuspage.json +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added tsconfig.base.json +17 −0

Large diff (18 lines) is not loaded by default to keep the page fast.Load diff