WebSensor 0.2 — signal intelligence upgrade · af43f2c · websensor

WebSensor 0.2 — signal intelligence upgrade

Registry: 7 depth fragments (40–46: AI frontier, cloud infrastructure, cybersecurity,
finance & markets, governments, science & health, transport/telecom/sports/news) →
3 167 organizations / 6 320 validated sensors (was 2 681 / 4 052); seeds carry
country, language and first_party; sync infers country/language by TLD and a
0–3 scheduling priority.

Core: semantic diff (noise classes vs pricing/policy/product/personnel, field-level
before → after with % deltas), WebSensor Signal Score with explainable reasons,
impact / velocity / anomaly scores, breaking state, entity rank, ~40 new event
classes (zero_day, active_exploitation, supply_chain_attack, guidance, buyback,
sanction, legislation, outbreak, grounding, service_shutdown…), search syntax
(entity: type: after:7d silent: importance:>70 country:), shared registry schema.

Engine: noise never becomes an event, routine firehose batches damped, fingerprint
idempotency, ambiguous-alias rule for entity resolution (+ relink-entities CLI),
silent-change bar (first-party · eligible type · importance ≥ 45), clusters with
propagation timeline, first-party/external counts, velocity, lead time and
breaking/developing/confirmed states, server-side alerts with HMAC-signed
webhooks, raw-body retention with reference counting, priority-aware scheduler
with per-domain circuit breaker and Redis heartbeat, UTF-16 BOM transcoding.

API: new filters (signal_min, group, first_party, confirmed, country, language,
change_class, q syntax, order=signal), /breaking /pulse /radar /clusters/:slug
/entities/rank /countries /categories/:channel /sensors/:id/snapshots, owner
routes (bookmarks, saved views, notifications, custom URL monitors with SSRF
checks), admin routes behind WS_ADMIN_TOKEN (ops, sensor/source actions,
connector test, YAML/JSON bulk import), TTL cache, WebSocket protocol 2 with
stream ids and replay.

Web: terminal-style design system (badges, density modes, skeletons, sparklines,
heatmaps), live system strip, ⌘K command palette, intelligence drawer, live feed
v2 (URL-synced filters, pause, "↑ N new events", replay), pages /live /pulse
/radar /entity/[id] /cluster/[slug] /country /bookmarks /monitors /ops, redesigned
breaking / silent / explore / event / source / sensor / search / alerts /
watchlists / health / api docs, OG image per event, sitemap, loading skeletons.

Migrations 0005 (additive schema) and 0006 (backfills). Docs updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 1 mo ago (Sep 11, 2026) parent 3c59155 Browse files

102 changed files +16,500 −1,428

modified .env.example +15 −1

		`@@ -26,5 +26,19 @@ WS_LLM_MIN_IMPORTANCE=35`
26	26
27	27	# Engine tuning
28	28	WS_FETCH_CONCURRENCY=16
29		−WS_USER_AGENT=WebSensorBot/0.1 (+https://www.websensor.io/bot)
	29	+WS_USER_AGENT="WebSensorBot/0.2 (+https://www.websensor.io/bot; contact@websensor.io)"
30	30	WS_SOURCES_FILE=./config/sources.yaml
	31	+WS_PER_HOST_CONCURRENCY=2
	32	+WS_MEANINGFUL_SIGNAL=0.32
	33	+# Silent-change bar: only silent-eligible types with importance >= this are flagged (spec §23)
	34	+WS_SILENT_MIN_IMPORTANCE=45
	35	+# Storage lifecycle (spec §58): raw bodies of unchanged snapshots are dropped after N days (canonical + hashes kept);
	36	+# snapshots referenced by an event are never pruned. WS_RETENTION=0 disables the job.
	37	+WS_RETENTION=1
	38	+WS_RETENTION_RAW_DAYS=21
	39	+WS_RETENTION_CHANGED_RAW_DAYS=60
	40	+
	41	+# API — internal operations (/api/v1/admin/*, /ops). Unset = admin API disabled.
	42	+WS_ADMIN_TOKEN=
	43	+# Custom URL monitors per anonymous owner (spec §45)
	44	+WS_MONITORS_PER_OWNER=5

modified CLAUDE.md +93 −80

		`@@ -1,97 +1,110 @@`
1	1	# WebSensor — repository guide
2	2
3	3	WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing
4		−Web. It monitors official public sources, detects meaningful changes, classifies them, links entities, scores
5		−importance, preserves evidence and publishes events through a WebSocket feed. The product brief that drove the
6		−design is `docs/PRODUCT-BRIEF.md`; this file is the working guide for the code.
	4	+Web. It monitors official public sources, detects meaningful changes, classifies them semantically, links
	5	+entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score,
	6	+preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through
	7	+a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file
	8	+is the working guide for the code (v0.2, 2026-09-11).
7	9
8	10	## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)
9		−- `packages/core` — taxonomy (event types + intrinsic severity, tiers, categories), ids, SSRF policy
10		− (`assertUrlAllowed`, `safeLookup`), hashing (sha256, simhash, shingles/Jaccard), canonical extraction
11		− (`canonicalizeHtml`: strips scripts/nav/footer/cookie chrome, timestamps, tokens, counters), diff engines
12		− (text / json / keyed list), stage-1 heuristics (`evaluateChange`, `describeChange`), scoring
13		− (importance components, confidence, novelty helpers, trending, activity anomaly), adaptive schedule.
14		− `@websensor/core/client` is the browser-safe subset (no `node:` imports) — client components must import it.
15		−- `packages/db` — plain SQL migrations (`migrations/*.sql`, applied by `migrate()` with an advisory lock) +
16		− Drizzle schema for typed access. `textArray()` must be used for `= any(...)` / `&&` with JS arrays
17		− (Drizzle spreads arrays into parameter lists).
18		−- `packages/store` — content-addressed blob store (`sha256/ab/cd/<hash>.zst`, zstd via `node:zlib`). Raw
19		− bodies, canonical representations and full diffs live there, never in Postgres. Interface ready for S3/MinIO.
20		−- `packages/connectors` — connector SDK (`WebSensorConnector.fetch/normalize`), safe fetcher (conditional GET,
21		− manual redirects validated per hop, HTTP/2→1.1 pin on NGHTTP2 errors, browser-UA second attempt on
22		− 403/resets, size + time limits, `keepAllHeaders`), 16 connector families: `http` (HTML/JSON/text/HEAD), `rss`
23		− (RSS/Atom/RDF/JSON Feed), `sitemap` (index + news), `statuspage` (Atlassian v2 summary), `statusjson` (Instatus /
24		− incident.io / Status.io), `github` (releases/tags/commits Atom, advisories REST), `jsonlist` (keyed records from
25		− any JSON API; `{now-2h}` placeholders), `package` (npm/PyPI/crates/RubyGems/NuGet/Packagist/Hex/Go proxy/Homebrew/
26		− Docker Hub version streams), `edgar` (SEC submissions → filings, 8-K items decoded), `openapi` (API contract
27		− fingerprints), `csv` (open-data rows, `tail`), `pdf` (pdf.js text via unpdf), `dns` (A/AAAA/NS/MX/TXT/CAA/SOA/DMARC),
28		− `tls` (certificate identity, chain, ALPN, expiry), `headers` (security/infra response headers), `rdap` (registrar,
29		− EPP status, expiry via IANA bootstrap), `discovery` (robots sitemaps, `<link rel=alternate>`, well-known feed
30		− paths, linked status pages — every candidate is fetched and parsed), `scrapfly` (fallback, budgeted, only when
31		− `fallback.scrapfly` is set on the source). `dns://` and `tls://` sensor URLs are accepted (host only).
32		−- `apps/engine` — scheduler (`FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency), pipeline
33		− (fetch → normalize → snapshot → diff → heuristics → change → event: entities, novelty, LLM interpretation,
34		− clustering, importance/confidence, silent-change detection, publish to Redis), registry sync from
35		− `config/sources.yaml` + `config/sources.d/`, discovery, connector health rollups, Prometheus metrics on :8262.
36		− CLI: `src/cli.ts` (`sync`, `discover`, `probe <domain>`, `run-once <sensor>`, `run-due [n]`); `src/validate.ts`
37		− (registry validator, see below).
38		−- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*`, WebSocket `/api/v1/live` (Redis pub/sub fan-out,
39		− channels `events:`, `entity:`, `source:`, `watchlist:`), `/api/v1/feed.rss`, `/api/health` `/api/ready` `/api/metrics`, apex→www
40		− redirect, and a reverse proxy to the Next.js app for everything else.
41		−- `apps/web` — Next.js 16 frontend (:8261, loopback). Pages: live, breaking, explore, sources, entities,
42		− timelines, silent changes, watchlists, alerts, event detail with diff viewer, health, API docs.
43		−- `config/sources.yaml` — the founding 271-organization registry (incl. governments and news media; news sources
44		− run with `llm: false`, heuristics only) + `config/sources.d/.yaml` fragments, one per site class* (open source,
45		− central banks & finance, energy/climate/weather, telecom & internet infrastructure, retail/consumer/travel,
46		− gaming & entertainment, universities & research, international orgs/NGOs/standards, sports, crypto,
47		− transport/aviation/space, consumer safety/food/agri, housing/labour/open data, enterprise SaaS, health systems &
48		− medtech, politics/elections/courts, EDGAR filings, package registries, web posture, OpenAPI, status JSON,
49		− documents & data). Fragments are merged by `apps/engine/src/seeds.ts` in file-name order; an entry with
50		− `extend: true` adds sensors/products/aliases to a source declared earlier. Authoring guide + connector
51		− cheat-sheet: `docs/registry/AUTHORING.md`; `docs/connectors/*.md` document each connector family.
52		−- Registry validator (no DB): `node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts [fragment.yaml] [--all]
53		− [--source id] [--connector key] [--json report.json]` runs every curated sensor through its connector and prints
54		− OK/WARN/FAIL; `--probe <domain>` is a discovery dry-run. `scripts/prune-fragment.py` removes FAIL/WARN sensors from
55		− a fragment using the JSON report; `scripts/gen-web-posture.py` regenerates the web-posture fragment.
	11	+- `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups,
	12	+ silent-eligible types, change classes, cluster states, countries), ids, SSRF policy (`assertUrlAllowed`,
	13	+ `safeLookup`), hashing, canonical extraction, diff engines (text / json / keyed list), stage-1 heuristics
	14	+ (`evaluateChange`, `describeChange`, `humanizeUrl`), semantic diff (`semantic.ts`: `classifyChange` →
	15	+ cosmetic / navigation / timestamp / advertisement / boilerplate vs meaningful / pricing / policy / product /
	16	+ personnel; `extractFieldChanges` → field-level before → after pairs with % deltas), scoring (`scoring.ts`:
	17	+ importance components, confidence, impact, velocity, WebSensor Signal Score with explainable reasons,
	18	+ `breakingState`, entity rank, daily anomaly), adaptive schedule, search syntax (`search.ts`: `entity:` `type:`
	19	+ `after:7d` `silent:true` `importance:>70` `country:CA` …), registry seed schema (`registry-schema.ts`, shared by
	20	+ engine, validator and the admin import endpoint). `@websensor/core/client` is the browser-safe subset.
	21	+- `packages/db` — plain SQL migrations (`migrations/*.sql`, applied by `migrate()` with an advisory lock) + Drizzle
	22	+ schema. `0005_intelligence.sql` added provenance (first_party / country / language / kind / owner_token on
	23	+ sources), sensor lifecycle (`status`, `priority`, `validated_at`), semantic class + field changes on changes,
	24	+ signal/velocity/impact/anomaly scores, fingerprints, score reasons on events, cluster propagation columns,
	25	+ `entity_daily`, `source_daily`, `bookmarks`, `saved_views`, alert channel config, notification delivery.
	26	+ `0006_backfill_states.sql` backfills them. `textArray()` must be used for `= any(...)` / `&&` with JS arrays.
	27	+- `packages/store` — content-addressed blob store (`sha256/ab/cd/<hash>.zst`). Raw bodies, canonical forms and full
	28	+ diffs live there, never in Postgres.
	29	+- `packages/connectors` — connector SDK (`fetch/normalize`), safe fetcher (conditional GET, per-hop SSRF checks,
	30	+ HTTP/2→1.1 pin, browser-UA retry, UTF-16 BOM transcoding, `postJson` for signed webhooks), 16 connector families
	31	+ (`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) +
	32	+ `discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and
	33	+ `{field.path}`.
	34	+- `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency,
	35	+ domain circuit breaker, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff →
	36	+ heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias
	37	+ rules, novelty, optional Claude interpretation, impact / anomaly / signal scores, silent-change bar, clustering
	38	+ with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook
	39	+ delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery,
	40	+ connector health rollups, retention (`retention.ts`), Prometheus metrics :8262.
	41	+ CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`,
	42	+ `refresh-clusters`, `llm-test`); `src/validate.ts` (registry validator).
	43	+- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped:
	44	+ watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts`
	45	+ behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models
	46	+ (`intel.ts`: breaking desk, pulse, radar, entity insights, rankings, cluster detail, country & category desks),
	47	+ TTL cache (`cache.ts`), WebSocket `/api/v1/live` protocol 2 (`live.ts`: `sid` on every frame, `{"since"}` replay
	48	+ from the durable stream, channels `events:* group:* country:* state:* type:* entity:* source:* watchlist:*`),
	49	+ `/api/v1/feed.rss`, `/api/health\|ready\|metrics`, apex→www redirect, reverse proxy to Next.
	50	+- `apps/web` — Next.js 16 (:8261 loopback). Design system in `components/ui.tsx` (Panel, Badge, Score, Tabs,
	51	+ Sparkline, Heatmap, Skeleton…), `prefs.tsx` (density compact/normal/comfortable, pause), `event-drawer.tsx`
	52	+ (intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters,
	53	+ pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar`
	54	+ `/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]`
	55	+ `/country/[slug]` `/event/[slug]` (+ OG image) `/bookmarks` `/watchlists` `/alerts` `/monitors` `/ops` `/health`
	56	+ `/api`.
	57	+- `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order
	58	+ (`extend: true` adds to an earlier source). Fragments 10–35 = site classes; 40–46 (2026-09-11) = depth: AI
	59	+ frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport /
	60	+ telecom / sports / news. Seeds may carry `country:` (ISO-2, `EU`, `INT`), `language:` and `first_party: false`
	61	+ (media). Guide: `docs/registry/AUTHORING.md`. Validator: `node node_modules/tsx/dist/cli.mjs
	62	+ apps/engine/src/validate.ts <fragment> [--all] [--json report.json]` — nothing enters the registry without OK.
56	63
57	64	## Rules
58		−- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets — goes through
59		− `assertUrlAllowed()`. Private ranges, metadata endpoints, `.maclustr.io`/`.ts.net` and single-label hosts are
60		− blocked; the dispatcher's DNS lookup only returns approved addresses.
61		−- Raw evidence is immutable: snapshots and diffs are never rewritten. Re-interpretation creates a new row in
62		− `interpretations` (versioned); `events.interpretation` holds the latest.
63		−- LLM cost control: heuristics first; Claude is called only for candidates above `WS_LLM_MIN_IMPORTANCE`, with a
64		− daily call budget; the deep model only above `WS_LLM_DEEP_MIN_IMPORTANCE`. Output is strict JSON
65		− (`output_config.format`), never free text. No key → heuristics only, the system still works.
66		−- Feeds: items are "new" only if their key was never seen (sensor `state.seenKeys`) and they are not older than
67		− 14 days; items scrolling out of the window are never "removed". Sitemaps/statuspages: a >50 % shrink is
68		− treated as a partial response, not mass deletion. A 404 becomes `page_removed` only after
69		− `WS_DELETE_CONFIRMATIONS` checks separated by `WS_DELETE_SEPARATION_MIN`.
70		−- Never label inference as fact: events carry `evidence_label` (OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED)
71		− and the interpretation keeps `observed` and `inferred` apart. Silent changes are flagged, never asserted as
72		− "unannounced" without checking recent announcement-type events of the same source.
73		−- Every sensor in the registry has been fetched and parsed successfully by the validator before being committed;
74		− a fragment that fails validation is not merged. Never add sensors to the registry without running it.
75		−- SSRF policy: NAT64 addresses (`64:ff9b::/96`) are judged by the embedded IPv4 address (IPv6-only networks with
76		− 464XLAT resolve every IPv4-only host that way), not blocked wholesale.
77		−- Do not add sensors that are blocked with 403 by design (Akamai/Cloudflare bot management) unless the source
78		− has `fallback.scrapfly: true` and a tier ≥ C; Scrapfly is a budgeted fallback, not the foundation.
79		−- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`…).
	65	+- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom
	66	+ monitors — goes through `assertUrlAllowed()`. Private ranges, metadata endpoints, `.maclustr.io`/`.ts.net`,
	67	+ single-label hosts and NAT64-embedded private IPv4 are blocked.
	68	+- Raw evidence is immutable: snapshots and diffs are never rewritten. Retention may drop the RAW body of
	69	+ snapshots that are not referenced by any event (canonical form and hashes stay); event snapshots are kept forever.
	70	+ Re-interpretation creates a new `interpretations` row.
	71	+- Noise never becomes an event: changes classified cosmetic / navigation / timestamp / advertisement /
	72	+ boilerplate are stored as changes only. Routine batches from firehose feeds are damped. Ingestion is idempotent
	73	+ (`events.fingerprint` = sensor + before/after canonical hashes).
	74	+- Silent change = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs,
	75	+ shutdown, feature removed, leadership, page removed, crawler policy…) · no matching announcement within 12 h ·
	76	+ importance ≥ `WS_SILENT_MIN_IMPORTANCE` · not a noise class. Never asserted as "unannounced" otherwise.
	77	+- Never label inference as fact: `evidence_label` OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED; AI text is
	78	+ labelled analysis; `score_reasons` explain every signal score. First-party evidence outweighs media reports
	79	+ (`sources.first_party`, media seeds carry `first_party: false`).
	80	+- Entity aliases that are ordinary words (`first`, `has`, `who`, `make`…) only match as exact upper-case acronyms
	81	+ in text (`AMBIGUOUS_ALIASES` in `apps/engine/src/entities.ts`); run `cli.ts relink-entities 7` after changing
	82	+ alias rules.
	83	+- LLM cost control: heuristics first; Claude only above `WS_LLM_MIN_IMPORTANCE`, daily budget, strict JSON output;
	84	+ `llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works.
	85	+- Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes
	86	+ `page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks.
	87	+- Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public
	88	+ feeds (`sources.kind = 'custom'`).
	89	+- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…).
80	90
81	91	## Dev
82	92	```
83	93	createdb websensor && cp .env.example .env
84	94	pnpm install
85		−pnpm db:migrate # or let the engine migrate on start
86		−npx tsx apps/engine/src/cli.ts sync # registry → DB
	95	+pnpm db:migrate # or let the engine migrate on start
	96	+npx tsx apps/engine/src/cli.ts sync # registry → DB
87	97	npx tsx apps/engine/src/cli.ts run-due 50
88		−pnpm dev:api · pnpm dev:engine · pnpm dev:web
	98	+pnpm dev:api · pnpm dev:engine · pnpm dev:web # open the site through the gateway (:8260) so /api is same-origin
89	99	pnpm test · pnpm typecheck
90	100	```
91		−Run CLI/engine from the repo root (config paths are relative to cwd).
	101	+Run CLI/engine from the repo root. A copy of production data for UI work:
	102	+`ssh M4M64b 'pg_dump websensor --data-only -t events -t …' \| psql websensor` then `cli.ts sync`.
92	103
93	104	## Deploy (MacLustr)
94		−`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `deploy/websensor.mld.json` (secrets only
95		−on M1M32). Processes: `websensor-api` (:8260, ngrok www.websensor.io), `websensor-web` (:8261 loopback),
96		−`websensor-engine` (metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`.
97		−See `deploy/README.md`.
	105	+`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json`
	106	+(secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260,
	107	+public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine`
	108	+(metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that
	109	+changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See
	110	+`deploy/README.md`.

modified apps/api/package.json +2 −0

		`@@ -14,6 +14,7 @@`
14	14	"@fastify/rate-limit": "^10.3.0",
15	15	"@fastify/reply-from": "^12.6.5",
16	16	"@fastify/websocket": "^11.2.0",
	17	+ "@websensor/connectors": "workspace:*",
17	18	"@websensor/core": "workspace:*",
18	19	"@websensor/db": "workspace:*",
19	20	"@websensor/store": "workspace:*",
		`@@ -24,6 +25,7 @@`
24	25	"prom-client": "^15.1.0",
25	26	"tsx": "^4.20.0",
26	27	"ws": "^8.21.3",
	28	+ "yaml": "^2.8.0",
27	29	"zod": "^4.0.0"
28	30	},
29	31	"devDependencies": {

added apps/api/src/cache.ts +36 −0

		`@@ -0,0 +1,36 @@`
	1	+/**
	2	+ * Tiny in-process TTL cache with request coalescing (spec §59). Used for aggregates only
	3	+ * (stats, trending, pulse, radar, rankings) — never for the live feed itself.
	4	+ */
	5	+const store = new Map<string, { at: number; ttl: number; value: unknown; pending?: Promise<unknown> }>();
	6	+
	7	+export async function cached<T>(key: string, ttlMs: number, fn: () => Promise<T>): Promise<T> {
	8	+ const now = Date.now();
	9	+ const hit = store.get(key);
	10	+ if (hit && now - hit.at < hit.ttl) return hit.value as T;
	11	+ if (hit?.pending) return hit.pending as Promise<T>;
	12	+ const pending = fn()
	13	+ .then((value) => {
	14	+ store.set(key, { at: Date.now(), ttl: ttlMs, value });
	15	+ return value;
	16	+ })
	17	+ .catch((e) => {
	18	+ // serve stale on failure when we have anything at all
	19	+ if (hit) {
	20	+ store.set(key, { ...hit, pending: undefined });
	21	+ return hit.value as T;
	22	+ }
	23	+ store.delete(key);
	24	+ throw e;
	25	+ });
	26	+ store.set(key, { at: hit?.at ?? 0, ttl: hit?.ttl ?? ttlMs, value: hit?.value, pending });
	27	+ return pending;
	28	+}
	29	+
	30	+export function cacheStats(): { entries: number; keys: string[] } {
	31	+ return { entries: store.size, keys: [...store.keys()].slice(0, 50) };
	32	+}
	33	+
	34	+export function invalidate(prefix: string): void {
	35	+ for (const k of store.keys()) if (k.startsWith(prefix)) store.delete(k);
	36	+}

modified apps/api/src/config.ts +5 −1

		`@@ -10,5 +10,9 @@ export const config = {`
10	10	publicBaseUrl: env.PUBLIC_BASE_URL ?? "http://localhost:8260",
11	11	canonicalHost: env.CANONICAL_HOST ?? "www.websensor.io",
12	12	redirectApexToWww: (env.WS_REDIRECT_APEX ?? "1") !== "0",
13		− version: "0.1.0",
	13	+ /** Admin endpoints (`/api/v1/admin/`) are enabled only when this token is set; sent as `X-WebSensor-Admin`. /
	14	+ adminToken: env.WS_ADMIN_TOKEN ?? "",
	15	+ /** Custom URL monitors per anonymous owner (spec §45). */
	16	+ monitorsPerOwner: Number(env.WS_MONITORS_PER_OWNER ?? 5),
	17	+ version: "0.2.0",
14	18	};

added apps/api/src/intel.ts +238 −0

		`@@ -0,0 +1,238 @@`
	1	+import { COUNTRIES, dailyAnomaly, entityRankScore, EVENT_GROUPS, FEED_CHANNELS } from "@websensor/core";
	2	+import { db, sql, textArray } from "@websensor/db";
	3	+import { cached } from "./cache";
	4	+import { EVENT_SELECT, listEvents } from "./queries";
	5	+
	6	+/**
	7	+ * Intelligence read-models (spec §34–39, §101–105): breaking desk, pulse, radar, entity insights,
	8	+ * rankings, cluster propagation, country and category desks. Aggregates are cached briefly.
	9	+ */
	10	+
	11	+const CLUSTER_SELECT = sql`c.id, c.slug, c.title, c.summary, c.primary_event_id, c.entity_ids, c.categories, c.event_count, c.max_importance, c.first_at, c.last_at, c.source_count, c.first_party_count, c.external_count, c.velocity, c.state, c.lead_time_ms, c.first_party_at, c.first_external_at`;
	12	+
	13	+async function clusterPrimaryEvents(where: ReturnType<typeof sql>, order: ReturnType<typeof sql>, limit: number): Promise<Record<string, unknown>[]> {
	14	+ const rows = await db.execute<Record<string, unknown>>(sql`
	15	+ select ${CLUSTER_SELECT},
	16	+ (select row_to_json(x) from (select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.id = c.primary_event_id) x) as event,
	17	+ (select json_agg(json_build_object('id', s2.id, 'name', s2.name, 'domain', s2.domain, 'first_party', s2.first_party) order by s2.name) from (select distinct s3.id, s3.name, s3.domain, s3.first_party from events e3 join sources s3 on s3.id = e3.source_id where e3.cluster_id = c.id limit 12) s2) as sources
	18	+ from event_clusters c where ${where} order by ${order} limit ${limit}`);
	19	+ return rows.rows;
	20	+}
	21	+
	22	+export async function breakingDesk(): Promise<Record<string, unknown>> {
	23	+ return cached("breaking", 8_000, async () => {
	24	+ const [breaking, developing, confirmed, watching] = await Promise.all([
	25	+ clusterPrimaryEvents(sql`c.state = 'breaking' and c.last_at >= now() - interval '24 hours'`, sql`c.max_importance desc, c.velocity desc, c.last_at desc`, 20),
	26	+ clusterPrimaryEvents(sql`c.state = 'developing' and c.last_at >= now() - interval '24 hours'`, sql`c.velocity desc, c.max_importance desc, c.last_at desc`, 20),
	27	+ clusterPrimaryEvents(sql`c.state = 'confirmed' and c.last_at >= now() - interval '48 hours'`, sql`c.last_at desc`, 20),
	28	+ listEvents({ limit: 25, order: "signal", signal_min: 60, after: new Date(Date.now() - 12 * 3600e3).toISOString() }).then((r) => r.items.filter((e) => !["breaking", "developing", "confirmed"].includes(String((e.cluster as { state?: string } \| null)?.state ?? "")))),
	29	+ ]);
	30	+ return { breaking_now: breaking, developing, recently_confirmed: confirmed, watching, generated_at: new Date().toISOString() };
	31	+ });
	32	+}
	33	+
	34	+export async function pulse(): Promise<Record<string, unknown>> {
	35	+ return cached("pulse", 10_000, async () => {
	36	+ const desks = ["ai", "cyber", "finance", "government", "infrastructure", "health", "science", "products"];
	37	+ const [activity, byDesk, rising, anomalies, silent, incidents, breaking, groups, totals] = await Promise.all([
	38	+ db.execute<Record<string, unknown>>(sql`
	39	+ with b as (select generate_series(date_trunc('hour', now() at time zone 'UTC') - interval '6 hours', date_trunc('minute', now() at time zone 'UTC'), interval '15 minutes') as bucket),
	40	+ ev as (select to_timestamp(floor(extract(epoch from detected_at) / 900) * 900) at time zone 'UTC' as bk, count(*) as n from events where detected_at >= now() - interval '7 hours' group by 1),
	41	+ ch as (select to_timestamp(floor(extract(epoch from detected_at) / 900) * 900) at time zone 'UTC' as bk, count(*) as n from changes where detected_at >= now() - interval '7 hours' group by 1)
	42	+ select to_char(b.bucket, 'YYYY-MM-DD"T"HH24:MI:00"Z"') as t, coalesce(ev.n, 0)::int as events, coalesce(ch.n, 0)::int as changes
	43	+ from b left join ev on ev.bk = b.bucket left join ch on ch.bk = b.bucket order by b.bucket`).then((r) => r.rows).catch(() => []),
	44	+ Promise.all(desks.map(async (d) => ({ desk: d, items: (await listEvents({ limit: 5, order: "signal", category: d, after: new Date(Date.now() - 12 * 3600e3).toISOString() })).items }))),
	45	+ db.execute<Record<string, unknown>>(sql`
	46	+ with cur as (select ee.entity_id, count(*) n from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - interval '3 hours' group by 1),
	47	+ prev as (select ee.entity_id, count(*) n from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - interval '27 hours' and e.detected_at < now() - interval '3 hours' group by 1)
	48	+ select en.id, en.name, en.type, cur.n::int as events_3h, coalesce(prev.n,0)::int as events_prev_24h, round((cur.n::float / greatest(0.125, coalesce(prev.n,0)/8.0))::numeric, 1)::float as acceleration
	49	+ from cur join entities en on en.id = cur.entity_id left join prev on prev.entity_id = cur.entity_id where cur.n >= 2 order by acceleration desc, cur.n desc limit 10`).then((r) => r.rows),
	50	+ db.execute<Record<string, unknown>>(sql`
	51	+ with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '2 hours' group by s.source_id),
	52	+ base as (select s.source_id, count()::float / (1424) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id)
	53	+ select so.id, so.name, so.domain, cur.n::int as changes_2h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day,
	54	+ round((case when coalesce(base.per_hour,0) = 0 then (case when cur.n/2.0 > 2 then 70 else 40 end) else least(100, case when cur.n/2.0/base.per_hour <= 1 then cur.n/2.0/base.per_hour30 else 30 + 25(ln(cur.n/2.0/base.per_hour)/ln(2)) end) end)::numeric, 1)::float as activity_score,
	55	+ (case when coalesce(base.per_hour,0) > 0 then round(((cur.n/2.0/base.per_hour - 1) * 100)::numeric) else null end)::int as pct_vs_baseline
	56	+ from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id where so.kind = 'registry' order by activity_score desc limit 8`).then((r) => r.rows),
	57	+ listEvents({ limit: 8, silent_change: true, order: "signal", after: new Date(Date.now() - 24 * 3600e3).toISOString() }).then((r) => r.items),
	58	+ listEvents({ limit: 8, group: "reliability", order: "recent", after: new Date(Date.now() - 6 * 3600e3).toISOString() }).then((r) => r.items),
	59	+ clusterPrimaryEvents(sql`c.state in ('breaking','developing') and c.last_at >= now() - interval '24 hours'`, sql`(c.state = 'breaking') desc, c.max_importance desc, c.velocity desc`, 8),
	60	+ db.execute<Record<string, unknown>>(sql`select e.event_type, count(*)::int as n from events e where e.detected_at >= now() - interval '24 hours' group by 1`).then((r) => {
	61	+ const byGroup: Record<string, number> = {};
	62	+ for (const row of r.rows) {
	63	+ const g = Object.entries(EVENT_GROUPS).find(([, spec]) => spec.types.includes(String(row.event_type)))?.[0] ?? "web";
	64	+ byGroup[g] = (byGroup[g] ?? 0) + Number(row.n);
	65	+ }
	66	+ return byGroup;
	67	+ }),
	68	+ db.execute<Record<string, unknown>>(sql`select (select count() from events where detected_at >= now() - interval '1 hour')::int as events_1h, (select count() from changes where detected_at >= now() - interval '1 hour')::int as changes_1h, (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes')::int as checks_5m, (select count(distinct source_id) from events where detected_at >= now() - interval '24 hours')::int as active_sources_24h, (select count(distinct country) from events where detected_at >= now() - interval '24 hours' and country is not null)::int as active_countries_24h`).then((r) => r.rows[0]),
	69	+ ]);
	70	+ return { activity, desks: byDesk, rising_entities: rising, anomalies, silent_changes: silent, infrastructure: incidents, breaking, by_group_24h: groups, totals, generated_at: new Date().toISOString() };
	71	+ });
	72	+}
	73	+
	74	+/** Weak signals (spec §101): things that are NOT breaking yet but could become important. Clearly labelled as indicators. */
	75	+export async function radar(): Promise<Record<string, unknown>> {
	76	+ return cached("radar", 30_000, async () => {
	77	+ const [quietBursts, silentClusters, docBursts, repoBursts, statusChanges, developing, newSensorsFiring] = await Promise.all([
	78	+ // Sources far above their baseline but without any high-importance event
	79	+ db.execute<Record<string, unknown>>(sql`
	80	+ with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '3 hours' group by s.source_id),
	81	+ base as (select s.source_id, count()::float / (1424) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id),
	82	+ hot as (select distinct source_id from events where detected_at >= now() - interval '6 hours' and coalesce(signal_score, importance) >= 75)
	83	+ select so.id, so.name, so.domain, so.categories, cur.n::int as changes_3h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day,
	84	+ round((cur.n/3.0 / greatest(0.02, coalesce(base.per_hour, 0)))::numeric, 1)::float as ratio
	85	+ from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id
	86	+ where so.kind = 'registry' and cur.n >= 4 and cur.source_id not in (select source_id from hot) and (coalesce(base.per_hour,0) = 0 or cur.n/3.0 / base.per_hour >= 4)
	87	+ order by ratio desc limit 12`).then((r) => r.rows),
	88	+ // Entities with several silent changes in 24 h
	89	+ db.execute<Record<string, unknown>>(sql`
	90	+ select en.id, en.name, en.type, count(*)::int as silent_24h, array_agg(distinct e.event_type) as types, max(e.detected_at) as last_at
	91	+ from events e join event_entities ee on ee.event_id = e.id join entities en on en.id = ee.entity_id
	92	+ where e.silent_change and e.detected_at >= now() - interval '24 hours' group by en.id, en.name, en.type having count(*) >= 2 order by silent_24h desc, last_at desc limit 12`).then((r) => r.rows),
	93	+ // Documentation / API modification bursts
	94	+ db.execute<Record<string, unknown>>(sql`
	95	+ select s.id, s.name, s.domain, count(*)::int as doc_changes_6h, array_agg(distinct e.event_type) as types, max(e.detected_at) as last_at
	96	+ from events e join sources s on s.id = e.source_id where e.detected_at >= now() - interval '6 hours' and e.event_type in ('documentation_change','API_change','api_change','availability_change')
	97	+ group by s.id, s.name, s.domain having count(*) >= 3 order by doc_changes_6h desc limit 10`).then((r) => r.rows),
	98	+ // Repository activity bursts (commits/tags/releases)
	99	+ db.execute<Record<string, unknown>>(sql`
	100	+ select s.id, s.name, s.domain, count(*)::int as repo_events_6h, max(e.detected_at) as last_at
	101	+ from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.detected_at >= now() - interval '6 hours' and sen.connector = 'github'
	102	+ group by s.id, s.name, s.domain having count(*) >= 3 order by repo_events_6h desc limit 10`).then((r) => r.rows),
	103	+ // Fresh status-page changes below the breaking bar
	104	+ listEvents({ limit: 10, group: "reliability", after: new Date(Date.now() - 3 * 3600e3).toISOString() }).then((r) => r.items.filter((e) => Number(e.importance) < 80)),
	105	+ clusterPrimaryEvents(sql`c.state = 'developing' and c.last_at >= now() - interval '12 hours'`, sql`c.velocity desc, c.last_at desc`, 8),
	106	+ // Sensors that produced their first-ever event in the last 24 h (new coverage lighting up)
	107	+ db.execute<Record<string, unknown>>(sql`
	108	+ select sen.id, sen.name, sen.source_id, s.name as source_name, min(e.detected_at) as first_event_at, count(*)::int as events
	109	+ from events e join sensors sen on sen.id = e.sensor_id join sources s on s.id = sen.source_id
	110	+ where e.detected_at >= now() - interval '24 hours' and not exists (select 1 from events e2 where e2.sensor_id = sen.id and e2.detected_at < now() - interval '24 hours')
	111	+ group by sen.id, sen.name, sen.source_id, s.name order by events desc limit 10`).then((r) => r.rows),
	112	+ ]);
	113	+ return { unusual_source_activity: quietBursts, silent_clusters: silentClusters, documentation_bursts: docBursts, repository_bursts: repoBursts, status_changes: statusChanges, developing, new_coverage: newSensorsFiring, generated_at: new Date().toISOString(), disclaimer: "Indicators, not facts: each item is a pattern in raw observations that has not (yet) produced a high-importance event." };
	114	+ });
	115	+}
	116	+
	117	+/** Entity insights (spec §26, §38, §102): heatmap, baseline, anomaly, velocity, rank, most active sensors. */
	118	+export async function entityInsights(entityId: string): Promise<Record<string, unknown>> {
	119	+ const [heat, today, last24, prev24, sensorsTop, silent24, breaking24, sources24, rank] = await Promise.all([
	120	+ db.execute<Record<string, unknown>>(sql`select day::text as day, events, silent, breaking, max_importance from entity_daily where entity_id = ${entityId} and day >= (now() at time zone 'UTC')::date - 34 order by day`).then((r) => r.rows),
	121	+ db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.detected_at >= date_trunc('day', now() at time zone 'UTC')`).then((r) => Number(r.rows[0]?.n ?? 0)),
	122	+ db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.detected_at >= now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)),
	123	+ db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.detected_at >= now() - interval '48 hours' and e.detected_at < now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)),
	124	+ db.execute<Record<string, unknown>>(sql`select sen.id, sen.name, sen.type, sen.connector, sen.source_id, count(*)::int as events_7d, max(e.detected_at) as last_event_at from events e join event_entities ee on ee.event_id = e.id join sensors sen on sen.id = e.sensor_id where ee.entity_id = ${entityId} and e.detected_at >= now() - interval '7 days' group by sen.id, sen.name, sen.type, sen.connector, sen.source_id order by events_7d desc limit 8`).then((r) => r.rows),
	125	+ db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.silent_change and e.detected_at >= now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)),
	126	+ db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and coalesce(e.signal_score, e.importance) >= 80 and e.detected_at >= now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)),
	127	+ db.execute<{ n: string }>(sql`select count(distinct e.source_id)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.detected_at >= now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)),
	128	+ entityRank(entityId),
	129	+ ]);
	130	+ const baselineDays = heat.filter((d) => String(d.day) < new Date().toISOString().slice(0, 10));
	131	+ const baselinePerDay = baselineDays.length ? baselineDays.reduce((n, d) => n + Number(d.events), 0) / Math.max(baselineDays.length, 7) : 0;
	132	+ // Rolling 24 h vs the 30-day daily baseline (a UTC "today" window is misleading in the first hours of the day).
	133	+ const anomaly = dailyAnomaly(last24, baselinePerDay, 24);
	134	+ const velocity = last24 && prev24 ? Math.round((last24 / prev24) * 100) / 100 : last24 ? 2 : 0;
	135	+ // Fill the 35-day heatmap with zero days
	136	+ const map = new Map(heat.map((d) => [String(d.day), d]));
	137	+ const days: Record<string, unknown>[] = [];
	138	+ for (let i = 34; i >= 0; i--) {
	139	+ const d = new Date(Date.now() - i * 86400e3).toISOString().slice(0, 10);
	140	+ days.push(map.get(d) ?? { day: d, events: 0, silent: 0, breaking: 0, max_importance: 0 });
	141	+ }
	142	+ return { heatmap: days, baseline_per_day: Math.round(baselinePerDay * 10) / 10, today, events_24h: last24, events_prev_24h: prev24, velocity_ratio: velocity, anomaly, silent_24h: silent24, breaking_24h: breaking24, sources_24h: sources24, most_active_sensors: sensorsTop, rank };
	143	+}
	144	+
	145	+/** WebSensor entity ranking (spec §105): computed over entities active in the last 7 days, cached 2 minutes. */
	146	+export async function entityRankings(limit = 100): Promise<Record<string, unknown>[]> {
	147	+ return cached(`rank:${limit}`, 120_000, async () => {
	148	+ const rows = await db.execute<Record<string, unknown>>(sql`
	149	+ with agg as (
	150	+ select ee.entity_id,
	151	+ count(*) filter (where e.detected_at >= now() - interval '24 hours') as e24,
	152	+ count(*) as e7,
	153	+ avg(coalesce(e.signal_score, e.importance)) as avg_signal,
	154	+ avg(case when e.evidence_label = 'CONFIRMED' then 1 else 0 end) as confirmed_ratio,
	155	+ count(distinct e.source_id) as sources,
	156	+ count(*) filter (where e.silent_change and e.detected_at >= now() - interval '24 hours') as silent24,
	157	+ count(*) filter (where coalesce(e.signal_score, e.importance) >= 80 and e.detected_at >= now() - interval '24 hours') as breaking24,
	158	+ max(e.detected_at) as last_at
	159	+ from events e join event_entities ee on ee.event_id = e.id join sources s on s.id = e.source_id where s.kind = 'registry' and e.detected_at >= now() - interval '7 days' group by ee.entity_id),
	160	+ base as (select entity_id, avg(events)::float as per_day from entity_daily where day >= (now() at time zone 'UTC')::date - 30 and day < (now() at time zone 'UTC')::date group by entity_id)
	161	+ select en.id, en.name, en.type, en.domain, en.importance, agg.e24::int as events_24h, agg.e7::int as events_7d, round(agg.avg_signal::numeric,1)::float as avg_signal, round(agg.confirmed_ratio::numeric,2)::float as confirmed_ratio, agg.sources::int as sources, agg.silent24::int as silent_24h, agg.breaking24::int as breaking_24h, agg.last_at, coalesce(base.per_day,0)::float as baseline_per_day
	162	+ from agg join entities en on en.id = agg.entity_id left join base on base.entity_id = agg.entity_id where en.type <> 'person'`);
	163	+ const scored = rows.rows.map((r) => ({ ...r, rank_score: entityRankScore({ importance: Number(r.importance), events24h: Number(r.events_24h), events7d: Number(r.events_7d), avgSignal: Number(r.avg_signal), confirmedRatio: Number(r.confirmed_ratio), uniqueSources: Number(r.sources), baselinePerDay: Number(r.baseline_per_day) }) }));
	164	+ scored.sort((a, b) => b.rank_score - a.rank_score);
	165	+ return scored.slice(0, limit).map((r, i) => ({ ...r, rank: i + 1 }));
	166	+ });
	167	+}
	168	+
	169	+export async function entityRank(entityId: string): Promise<{ rank: number \| null; total: number; score: number \| null }> {
	170	+ const all = await entityRankings(2000);
	171	+ const i = all.findIndex((r) => r.id === entityId);
	172	+ return { rank: i >= 0 ? i + 1 : null, total: all.length, score: i >= 0 ? Number(all[i]!.rank_score) : null };
	173	+}
	174	+
	175	+export async function clusterDetail(idOrSlug: string): Promise<Record<string, unknown> \| null> {
	176	+ const c = (await db.execute<Record<string, unknown>>(sql`select ${CLUSTER_SELECT}, c.timeline from event_clusters c where c.id = ${idOrSlug} or c.slug = ${idOrSlug} limit 1`)).rows[0];
	177	+ if (!c) return null;
	178	+ const events = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.cluster_id = ${String(c.id)} order by e.detected_at asc limit 200`);
	179	+ const ents = (c.entity_ids as string[]).length ? await db.execute<Record<string, unknown>>(sql`select id, name, type, importance from entities where id = any(${textArray(c.entity_ids as string[])}) order by importance desc`) : { rows: [] };
	180	+ // Propagation timeline (spec §35): offsets from the first signal.
	181	+ const first = new Date(String(c.first_at)).getTime();
	182	+ const propagation = events.rows.map((e) => ({ id: e.id, slug: e.slug, at: e.detected_at, offset_ms: new Date(String(e.detected_at)).getTime() - first, source: e.source, sensor: e.sensor, first_party: e.first_party, event_type: e.event_type, importance: e.importance, title: e.title }));
	183	+ const firstExternal = events.rows.find((e) => e.first_party === false);
	184	+ const firstParty = events.rows.find((e) => e.first_party !== false);
	185	+ const leadTime = firstParty && firstExternal ? new Date(String(firstExternal.detected_at)).getTime() - new Date(String(firstParty.detected_at)).getTime() : null;
	186	+ return { cluster: c, events: events.rows, entities: ents.rows, propagation, lead_time_ms: leadTime ?? c.lead_time_ms ?? null, first_party_signals: events.rows.filter((e) => e.first_party !== false).length, external_signals: events.rows.filter((e) => e.first_party === false).length };
	187	+}
	188	+
	189	+export async function countryList(): Promise<Record<string, unknown>[]> {
	190	+ return cached("countries", 60_000, async () => {
	191	+ const rows = await db.execute<Record<string, unknown>>(sql`
	192	+ select s.country, count(distinct s.id)::int as sources, (select count(*) from events e where e.country = s.country and e.detected_at >= now() - interval '24 hours')::int as events_24h,
	193	+ (select count(*) from events e where e.country = s.country and e.detected_at >= now() - interval '24 hours' and coalesce(e.signal_score, e.importance) >= 80)::int as breaking_24h
	194	+ from sources s where s.country is not null and s.enabled and s.kind = 'registry' group by s.country order by events_24h desc, sources desc`);
	195	+ return rows.rows.map((r) => ({ ...r, name: COUNTRIES[String(r.country)]?.name ?? String(r.country), slug: COUNTRIES[String(r.country)]?.slug ?? String(r.country).toLowerCase(), flag: COUNTRIES[String(r.country)]?.flag ?? "" }));
	196	+ });
	197	+}
	198	+
	199	+export async function countryDesk(code: string): Promise<Record<string, unknown>> {
	200	+ const c = code.toUpperCase();
	201	+ const cats = ["government", "finance", "infrastructure", "health", "news", "cyber", "ai", "science"];
	202	+ const [breaking, byCategory, sources, byType, silent, recent] = await Promise.all([
	203	+ listEvents({ limit: 10, country: c, order: "signal", after: new Date(Date.now() - 48 * 3600e3).toISOString() }).then((r) => r.items),
	204	+ Promise.all(cats.map(async (cat) => ({ category: cat, items: (await listEvents({ limit: 6, country: c, category: cat })).items }))).then((x) => x.filter((d) => d.items.length)),
	205	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier, s.categories, s.first_party, (select count() from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count, (select count() from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h from sources s where s.country = ${c} and s.enabled and s.kind = 'registry' order by events_24h desc, s.tier, s.name limit 200`).then((r) => r.rows),
	206	+ db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where country = ${c} and detected_at >= now() - interval '7 days' group by 1 order by 2 desc limit 20`).then((r) => r.rows),
	207	+ listEvents({ limit: 8, country: c, silent_change: true }).then((r) => r.items),
	208	+ listEvents({ limit: 40, country: c }),
	209	+ ]);
	210	+ return { country: { code: c, name: COUNTRIES[c]?.name ?? c, flag: COUNTRIES[c]?.flag ?? "" }, breaking, by_category: byCategory, sources, by_type: byType, silent, recent: recent.items, nextCursor: recent.nextCursor };
	211	+}
	212	+
	213	+export async function categoryDesk(channel: string): Promise<Record<string, unknown>> {
	214	+ const cats = FEED_CHANNELS[channel] ?? [channel];
	215	+ const [breaking, silent, activeSources, byType, trendingEntities, recent, series] = await Promise.all([
	216	+ listEvents({ limit: 8, category: channel, order: "signal", after: new Date(Date.now() - 24 * 3600e3).toISOString() }).then((r) => r.items),
	217	+ listEvents({ limit: 8, category: channel, silent_change: true }).then((r) => r.items),
	218	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier, s.first_party, count(*)::int as events_24h, max(e.importance)::float as max_importance from events e join sources s on s.id = e.source_id where ${channel} = any(e.categories) and e.detected_at >= now() - interval '24 hours' and s.kind = 'registry' group by s.id, s.name, s.domain, s.tier, s.first_party order by events_24h desc limit 12`).then((r) => r.rows),
	219	+ db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where ${channel} = any(categories) and detected_at >= now() - interval '7 days' group by 1 order by 2 desc limit 16`).then((r) => r.rows),
	220	+ db.execute<Record<string, unknown>>(sql`select en.id, en.name, en.type, count(*)::int as events_24h, count(distinct e.source_id)::int as sources from events e join event_entities ee on ee.event_id = e.id join entities en on en.id = ee.entity_id where ${channel} = any(e.categories) and e.detected_at >= now() - interval '24 hours' group by en.id, en.name, en.type order by events_24h desc limit 10`).then((r) => r.rows),
	221	+ listEvents({ limit: 60, category: channel }),
	222	+ db.execute<Record<string, unknown>>(sql`select to_char(date_trunc('hour', detected_at at time zone 'UTC'), 'YYYY-MM-DD"T"HH24:00:00"Z"') as t, count(*)::int as n from events where ${channel} = any(categories) and detected_at >= now() - interval '48 hours' group by 1 order by 1`).then((r) => r.rows),
	223	+ ]);
	224	+ return { channel, categories: cats, breaking, silent, active_sources: activeSources, by_type: byType, trending_entities: trendingEntities, recent: recent.items, nextCursor: recent.nextCursor, series };
	225	+}
	226	+
	227	+/** Sensors table for a source, with polling metadata the source page shows (spec §27). */
	228	+export async function sourceSensors(sourceId: string): Promise<Record<string, unknown>[]> {
	229	+ const r = await db.execute<Record<string, unknown>>(sql`
	230	+ select id, name, url, type, connector, tier, health, status, priority, enabled, next_check_at, last_check_at, last_change_at, last_event_at, last_status, last_error, consecutive_errors, total_runs, total_not_modified, raw_changes, meaningful_changes, avg_latency_ms, base_interval_seconds, validated_at,
	231	+ etag is not null as has_etag, last_modified is not null as has_last_modified, etag, last_modified,
	232	+ (select count(*) from sensor_runs r where r.sensor_id = sensors.id and r.started_at >= now() - interval '24 hours')::int as checks_24h,
	233	+ (select count(*) from changes c where c.sensor_id = sensors.id and c.detected_at >= now() - interval '24 hours')::int as changes_24h,
	234	+ (select count(*) from snapshots sn where sn.sensor_id = sensors.id)::int as snapshot_count,
	235	+ case when last_check_at is not null and next_check_at is not null then extract(epoch from (next_check_at - last_check_at))::int else null end as current_interval_seconds
	236	+ from sensors where source_id = ${sourceId} order by priority, tier, name`);
	237	+ return r.rows;
	238	+}

modified apps/api/src/live.ts +90 −19

		`@@ -1,34 +1,63 @@`
1	1	import type { FastifyInstance } from "fastify";
2	2	import type { WebSocket } from "ws";
3	3	import Redis from "ioredis";
4		−import { FEED_CHANNELS } from "@websensor/core";
	4	+import { eventGroupOf, FEED_CHANNELS } from "@websensor/core";
5	5	import { db, sql } from "@websensor/db";
6	6	import { config } from "./config";
7	7
8	8	/**
9	9	* WebSocket gateway `/api/v1/live`. One Redis subscriber fans out to every client; clients
10		− * pick channels: events:global · events:breaking · events:<ai\|cyber\|finance\|health\|government\|
11		− * science\|products\|infrastructure> · entity:<id> · source:<id> · watchlist:<id>.
12		− * Protocol (JSON): client → {"subscribe":[…]} \| {"unsubscribe":[…]} \| {"ping":1}
13		− * server → {"type":"hello"} \| {"type":"event", "channels":[…], "event":{…}} \| {"type":"pong"}
	10	+ * pick channels: events:global · events:breaking · events:silent · events:first-party ·
	11	+ * events:<ai\|cyber\|finance\|health\|government\|science\|products\|infrastructure\|news> · group:<security\|…> ·
	12	+ * country:<CA> · type:<event_type> · entity:<id> · source:<id> · watchlist:<id>.
	13	+ * Protocol (JSON): client → {"subscribe":[…]} \| {"unsubscribe":[…]} \| {"ping":1} \| {"since":"<sid>"}
	14	+ * server → {"type":"hello"} \| {"type":"event", "sid":"…", "channels":[…], "event":{…}} \| {"type":"replay_done"} \| {"type":"pong"} \| {"type":"heartbeat"}
	15	+ * Every event frame carries the Redis stream id `sid`; after a reconnection the client sends
	16	+ * {"since": lastSid} and missed events (up to 500) are replayed from the durable stream (spec §60).
14	17	*/
15	18	interface Client {
16	19	ws: WebSocket;
17	20	channels: Set<string>;
18		− watchlists: Map<string, { entities: Set<string>; sources: Set<string>; keywords: string[]; categories: Set<string> }>;
	21	+ watchlists: Map<string, WatchRules>;
	22	+}
	23	+interface WatchRules {
	24	+ entities: Set<string>;
	25	+ sources: Set<string>;
	26	+ keywords: string[];
	27	+ categories: Set<string>;
	28	+ types: Set<string>;
	29	+ countries: Set<string>;
	30	+ urls: string[];
19	31	}
20	32
21	33	const clients = new Set<Client>();
22	34	let sub: Redis \| null = null;
	35	+let cmd: Redis \| null = null;
23	36	let published = 0;
24	37
25	38	export function liveStats(): { clients: number; published: number } {
26	39	return { clients: clients.size, published };
27	40	}
28	41
	42	+/** Engine heartbeat written by the scheduler (`ws:engine:status`). */
	43	+export async function engineStatus(): Promise<Record<string, unknown> \| null> {
	44	+ try {
	45	+ const raw = await getCmd().get("ws:engine:status");
	46	+ return raw ? (JSON.parse(raw) as Record<string, unknown>) : null;
	47	+ } catch {
	48	+ return null;
	49	+ }
	50	+}
	51	+
	52	+function getCmd(): Redis {
	53	+ if (!cmd) cmd = new Redis(config.redisUrl, { maxRetriesPerRequest: 2, lazyConnect: false });
	54	+ return cmd;
	55	+}
	56	+
29	57	export async function registerLive(app: FastifyInstance): Promise<void> {
30	58	sub = new Redis(config.redisUrl, { maxRetriesPerRequest: 3 });
31	59	sub.on("error", (e) => app.log.warn({ err: e.message }, "redis sub error"));
	60	+ getCmd().on("error", (e) => app.log.warn({ err: e.message }, "redis cmd error"));
32	61	await sub.subscribe("ws:live");
33	62	sub.on("message", (_ch, msg) => {
34	63	let ev: Record<string, unknown>;
		`@@ -38,21 +67,15 @@ export async function registerLive(app: FastifyInstance): Promise<void> {`
38	67	return;
39	68	}
40	69	published++;
41		− const chans = channelsFor(ev);
42		− for (const c of clients) {
43		− const hit = [...chans].filter((ch) => c.channels.has(ch));
44		− for (const [wid, w] of c.watchlists) if (matchesWatchlist(ev, w)) hit.push(`watchlist:${wid}`);
45		− if (!hit.length) continue;
46		− if (c.ws.readyState === c.ws.OPEN) c.ws.send(JSON.stringify({ type: "event", channels: hit, event: ev }));
47		− }
	70	+ deliver(ev, String(ev.sid ?? ""));
48	71	});
49	72
50	73	app.get("/api/v1/live", { websocket: true }, (socket) => {
51	74	const client: Client = { ws: socket, channels: new Set(["events:global"]), watchlists: new Map() };
52	75	clients.add(client);
53		− socket.send(JSON.stringify({ type: "hello", channels: [...client.channels], serverTime: new Date().toISOString() }));
	76	+ socket.send(JSON.stringify({ type: "hello", channels: [...client.channels], serverTime: new Date().toISOString(), protocol: 2 }));
54	77	socket.on("message", async (raw: Buffer \| string) => {
55		− let msg: { subscribe?: string[]; unsubscribe?: string[]; ping?: number };
	78	+ let msg: { subscribe?: string[]; unsubscribe?: string[]; ping?: number; since?: string };
56	79	try {
57	80	msg = JSON.parse(raw.toString()) as typeof msg;
58	81	} catch {
		`@@ -68,7 +91,8 @@ export async function registerLive(app: FastifyInstance): Promise<void> {`
68	91	client.channels.delete(ch);
69	92	if (ch.startsWith("watchlist:")) client.watchlists.delete(ch.slice(10));
70	93	}
71		− socket.send(JSON.stringify({ type: "subscribed", channels: [...client.channels, ...[...client.watchlists.keys()].map((w) => `watchlist:${w}`)] }));
	94	+ if (msg.subscribe \|\| msg.unsubscribe) socket.send(JSON.stringify({ type: "subscribed", channels: [...client.channels, ...[...client.watchlists.keys()].map((w) => `watchlist:${w}`)] }));
	95	+ if (typeof msg.since === "string" && /^\d{10,16}-\d{1,6}$/.test(msg.since)) await replay(client, msg.since);
72	96	});
73	97	const hb = setInterval(() => {
74	98	if (socket.readyState === socket.OPEN) socket.send(JSON.stringify({ type: "heartbeat", t: Date.now() }));
		`@@ -84,37 +108,83 @@ export async function registerLive(app: FastifyInstance): Promise<void> {`
84	108	});
85	109	}
86	110
	111	+function deliver(ev: Record<string, unknown>, sid: string, only?: Client): void {
	112	+ const chans = channelsFor(ev);
	113	+ for (const c of only ? [only] : clients) {
	114	+ const hit = [...chans].filter((ch) => c.channels.has(ch));
	115	+ for (const [wid, w] of c.watchlists) if (matchesWatchlist(ev, w)) hit.push(`watchlist:${wid}`);
	116	+ if (!hit.length) continue;
	117	+ if (c.ws.readyState === c.ws.OPEN) c.ws.send(JSON.stringify({ type: "event", sid, channels: hit, event: ev }));
	118	+ }
	119	+}
	120	+
	121	+/** Replay missed events from the durable stream (exclusive of `since`). */
	122	+async function replay(client: Client, since: string): Promise<void> {
	123	+ try {
	124	+ const [ms, seq] = since.split("-");
	125	+ const start = `${ms}-${Number(seq) + 1}`;
	126	+ const rows = (await getCmd().xrange("ws:events", start, "+", "COUNT", 500)) as [string, string[]][];
	127	+ let n = 0;
	128	+ for (const [sid, fields] of rows) {
	129	+ const i = fields.indexOf("event");
	130	+ if (i < 0) continue;
	131	+ try {
	132	+ const ev = JSON.parse(fields[i + 1]!) as Record<string, unknown>;
	133	+ deliver({ ...ev, sid, replayed: true }, sid, client);
	134	+ n++;
	135	+ } catch {
	136	+ // skip malformed
	137	+ }
	138	+ }
	139	+ if (client.ws.readyState === client.ws.OPEN) client.ws.send(JSON.stringify({ type: "replay_done", since, count: n, truncated: rows.length >= 500 }));
	140	+ } catch {
	141	+ if (client.ws.readyState === client.ws.OPEN) client.ws.send(JSON.stringify({ type: "replay_done", since, count: 0, error: "replay_unavailable" }));
	142	+ }
	143	+}
	144	+
87	145	export function channelsFor(ev: Record<string, unknown>): Set<string> {
88	146	const out = new Set<string>(["events:global"]);
89	147	const importance = Number(ev.importance ?? 0);
90		− if (importance >= 80) out.add("events:breaking");
	148	+ const signal = Number(ev.signal ?? importance);
	149	+ if (signal >= 80 \|\| importance >= 80) out.add("events:breaking");
91	150	if (ev.silent) out.add("events:silent");
	151	+ if (ev.firstParty !== false) out.add("events:first-party");
92	152	const cats = (ev.categories as string[] \| undefined) ?? [];
93	153	for (const [ch, list] of Object.entries(FEED_CHANNELS)) if (list.some((c) => cats.includes(c)) \|\| cats.includes(ch)) out.add(`events:${ch}`);
94	154	const src = ev.source as { id?: string } \| undefined;
95	155	if (src?.id) out.add(`source:${src.id}`);
96	156	for (const e of (ev.entities as { id: string }[] \| undefined) ?? []) out.add(`entity:${e.id}`);
97	157	out.add(`type:${String(ev.type)}`);
	158	+ out.add(`group:${String(ev.group ?? eventGroupOf(String(ev.type)))}`);
	159	+ if (ev.country) out.add(`country:${String(ev.country).toUpperCase()}`);
	160	+ if (ev.clusterState === "breaking" \|\| ev.clusterState === "developing") out.add(`state:${String(ev.clusterState)}`);
98	161	return out;
99	162	}
100	163
101	164	async function loadWatchlist(client: Client, id: string): Promise<void> {
102	165	const rows = await db.execute<{ kind: string; value: string }>(sql`select kind, value from watchlist_items where watchlist_id = ${id}`);
103		− const w = { entities: new Set<string>(), sources: new Set<string>(), keywords: [] as string[], categories: new Set<string>() };
	166	+ const w: WatchRules = { entities: new Set(), sources: new Set(), keywords: [], categories: new Set(), types: new Set(), countries: new Set(), urls: [] };
104	167	for (const r of rows.rows) {
105	168	if (r.kind === "entity") w.entities.add(r.value);
106	169	else if (r.kind === "source") w.sources.add(r.value);
107	170	else if (r.kind === "keyword") w.keywords.push(r.value.toLowerCase());
108	171	else if (r.kind === "category") w.categories.add(r.value);
	172	+ else if (r.kind === "event_type") w.types.add(r.value);
	173	+ else if (r.kind === "country") w.countries.add(r.value.toUpperCase());
	174	+ else if (r.kind === "url") w.urls.push(r.value.replace(/\/$/, ""));
109	175	}
110	176	client.watchlists.set(id, w);
111	177	}
112	178
113		−function matchesWatchlist(ev: Record<string, unknown>, w: { entities: Set<string>; sources: Set<string>; keywords: string[]; categories: Set<string> }): boolean {
	179	+function matchesWatchlist(ev: Record<string, unknown>, w: WatchRules): boolean {
114	180	const src = ev.source as { id?: string } \| undefined;
115	181	if (src?.id && w.sources.has(src.id)) return true;
116	182	for (const e of (ev.entities as { id: string }[] \| undefined) ?? []) if (w.entities.has(e.id)) return true;
117	183	for (const c of (ev.categories as string[] \| undefined) ?? []) if (w.categories.has(c)) return true;
	184	+ if (w.types.has(String(ev.type))) return true;
	185	+ if (ev.country && w.countries.has(String(ev.country).toUpperCase())) return true;
	186	+ const url = String(ev.url ?? "");
	187	+ if (w.urls.some((u) => url === u \|\| url.startsWith(u + "/"))) return true;
118	188	if (w.keywords.length) {
119	189	const hay = `${String(ev.title)} ${String(ev.summary)}`.toLowerCase();
120	190	if (w.keywords.some((k) => hay.includes(k))) return true;
		`@@ -126,4 +196,5 @@ export async function closeLive(): Promise<void> {`
126	196	for (const c of clients) c.ws.close(1001, "server shutdown");
127	197	clients.clear();
128	198	if (sub) await sub.quit().catch(() => undefined);
	199	+ if (cmd) await cmd.quit().catch(() => undefined);
129	200	}

modified apps/api/src/queries.ts +124 −23

		`@@ -1,3 +1,4 @@`
	1	+import { EVENT_GROUPS, parseSearch } from "@websensor/core";
1	2	import { db, sql, textArray } from "@websensor/db";
2	3
3	4	/** Read-model helpers shared by REST routes. All return plain JSON-ready objects. */
		`@@ -13,27 +14,48 @@ export interface EventFilters {`
13	14	cluster?: string;
14	15	importance_min?: number;
15	16	confidence_min?: number;
	17	+ signal_min?: number;
16	18	event_type?: string;
	19	+ group?: string;
17	20	silent_change?: boolean;
	21	+ first_party?: boolean;
	22	+ confirmed?: boolean;
	23	+ country?: string;
	24	+ language?: string;
	25	+ change_class?: string;
18	26	q?: string;
19	27	limit: number;
20	28	cursor?: string;
21		− order?: "recent" \| "importance";
	29	+ order?: "recent" \| "importance" \| "signal";
	30	+ /** include events from custom (owner) sources — only set by owner-scoped routes */
	31	+ includeCustom?: boolean;
22	32	}
23	33
24	34	export const EVENT_SELECT = sql`
25	35	e.id, e.slug, e.event_type, e.title, e.summary, e.why_it_matters, e.importance, e.confidence, e.novelty, e.categories, e.keywords,
26		− e.silent_change, e.evidence_label, e.url, e.detected_at, e.published_at, e.observed_from, e.processed_at, e.published_to_feed_at,
	36	+ e.silent_change, e.evidence_label, e.url, e.canonical_url, e.detected_at, e.published_at, e.observed_from, e.processed_at, e.published_to_feed_at,
27	37	e.detection_latency_ms, e.processing_latency_ms, e.cluster_id, e.sensor_id, e.source_id, e.change_id, e.old_snapshot_id, e.new_snapshot_id,
28	38	e.importance_components, e.processing_version,
29		− json_build_object('id', s.id, 'name', s.name, 'domain', s.domain, 'tier', s.tier, 'categories', s.categories) as source,
	39	+ e.signal_score, e.velocity_score, e.impact_score, e.anomaly_score, e.change_class, e.first_party, e.country, e.language, e.field_changes, e.score_reasons,
	40	+ json_build_object('id', s.id, 'name', s.name, 'domain', s.domain, 'tier', s.tier, 'categories', s.categories, 'first_party', s.first_party, 'country', s.country) as source,
30	41	json_build_object('id', sen.id, 'name', sen.name, 'type', sen.type, 'connector', sen.connector, 'tier', sen.tier) as sensor,
31	42	coalesce((select json_agg(json_build_object('id', en.id, 'name', en.name, 'type', en.type, 'role', ee.role) order by ee.role, en.name)
32	43	from event_entities ee join entities en on en.id = ee.entity_id where ee.event_id = e.id), '[]'::json) as entities,
33		− (select event_count from event_clusters c where c.id = e.cluster_id) as cluster_size`;
	44	+ (select event_count from event_clusters c where c.id = e.cluster_id) as cluster_size,
	45	+ (select json_build_object('id', c.id, 'slug', c.slug, 'state', c.state, 'event_count', c.event_count, 'source_count', c.source_count, 'first_party_count', c.first_party_count, 'external_count', c.external_count, 'velocity', c.velocity, 'lead_time_ms', c.lead_time_ms) from event_clusters c where c.id = e.cluster_id) as cluster`;
34	46
35		−export async function listEvents(f: EventFilters): Promise<{ items: Record<string, unknown>[]; nextCursor: string \| null }> {
	47	+/** Build WHERE conditions from filters (shared by list, RSS, watchlist and desk queries). */
	48	+export function eventConditions(f: EventFilters): ReturnType<typeof sql>[] {
36	49	const conds = [sql`true`];
	50	+ let free = f.q?.trim() ?? "";
	51	+ // Advanced syntax inside q (entity:… type:… after:…) is merged into the filters.
	52	+ if (free) {
	53	+ const p = parseSearch(free);
	54	+ free = p.text;
	55	+ const pf = p.filters;
	56	+ f = { ...f, entity: f.entity ?? pf.entity, source: f.source ?? pf.source, domain: f.domain ?? pf.domain, event_type: f.event_type ?? pf.event_type, group: f.group ?? pf.group, category: f.category ?? pf.category, country: f.country ?? pf.country, language: f.language ?? pf.language, after: f.after ?? pf.after, before: f.before ?? pf.before, silent_change: f.silent_change ?? pf.silent_change, first_party: f.first_party ?? pf.first_party, confirmed: f.confirmed ?? pf.confirmed, importance_min: f.importance_min ?? pf.importance_min, confidence_min: f.confidence_min ?? pf.confidence_min, signal_min: f.signal_min ?? pf.signal_min, change_class: f.change_class ?? pf.change_class, cluster: f.cluster ?? pf.cluster, sensor: f.sensor ?? pf.sensor };
	57	+ }
	58	+ if (!f.includeCustom) conds.push(sql`s.kind = 'registry'`);
37	59	if (f.after) conds.push(sql`e.detected_at > ${new Date(f.after)}`);
38	60	if (f.before) conds.push(sql`e.detected_at < ${new Date(f.before)}`);
39	61	if (f.category) conds.push(sql`${f.category} = any(e.categories)`);
		`@@ -44,22 +66,41 @@ export async function listEvents(f: EventFilters): Promise<{ items: Record<strin`
44	66	if (f.entity) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = ${f.entity})`);
45	67	if (f.importance_min !== undefined) conds.push(sql`e.importance >= ${f.importance_min}`);
46	68	if (f.confidence_min !== undefined) conds.push(sql`e.confidence >= ${f.confidence_min}`);
47		− if (f.event_type) conds.push(sql`e.event_type = any(${textArray(f.event_type.split(","))})`);
	69	+ if (f.signal_min !== undefined) conds.push(sql`coalesce(e.signal_score, e.importance) >= ${f.signal_min}`);
	70	+ if (f.event_type) conds.push(sql`e.event_type = any(${textArray(f.event_type.split(",").map((t) => t.trim()).filter(Boolean))})`);
	71	+ if (f.group && EVENT_GROUPS[f.group]) conds.push(sql`e.event_type = any(${textArray(EVENT_GROUPS[f.group]!.types)})`);
48	72	if (f.silent_change !== undefined) conds.push(sql`e.silent_change = ${f.silent_change}`);
49		− if (f.q) conds.push(sql`e.search @@ websearch_to_tsquery('english', ${f.q})`);
	73	+ if (f.first_party !== undefined) conds.push(sql`e.first_party = ${f.first_party}`);
	74	+ if (f.confirmed) conds.push(sql`e.evidence_label = 'CONFIRMED'`);
	75	+ if (f.country) conds.push(sql`e.country = ${f.country.toUpperCase()}`);
	76	+ if (f.language) conds.push(sql`e.language = ${f.language.toLowerCase()}`);
	77	+ if (f.change_class) conds.push(sql`e.change_class = ${f.change_class}`);
	78	+ if (free) conds.push(sql`(e.search @@ websearch_to_tsquery('english', ${free}) or e.title ilike ${"%" + free + "%"})`);
	79	+ return conds;
	80	+}
	81	+
	82	+export async function listEvents(f: EventFilters): Promise<{ items: Record<string, unknown>[]; nextCursor: string \| null }> {
	83	+ const conds = eventConditions(f);
50	84	if (f.cursor) {
51	85	const [ts, id] = decodeCursor(f.cursor);
52	86	if (f.order === "importance") conds.push(sql`(e.importance, e.id) < (${Number(ts)}, ${id})`);
	87	+ else if (f.order === "signal") conds.push(sql`(coalesce(e.signal_score, e.importance), e.id) < (${Number(ts)}, ${id})`);
53	88	else conds.push(sql`(e.detected_at, e.id) < (${new Date(Number(ts))}, ${id})`);
54	89	}
55		− const order = f.order === "importance" ? sql`e.importance desc, e.id desc` : sql`e.detected_at desc, e.id desc`;
	90	+ const order = f.order === "importance" ? sql`e.importance desc, e.id desc` : f.order === "signal" ? sql`coalesce(e.signal_score, e.importance) desc, e.id desc` : sql`e.detected_at desc, e.id desc`;
56	91	const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where ${sql.join(conds, sql` and `)} order by ${order} limit ${f.limit + 1}`);
57	92	const items = rows.rows.slice(0, f.limit);
58	93	const last = items[items.length - 1];
59		− const nextCursor = rows.rows.length > f.limit && last ? encodeCursor(f.order === "importance" ? String(last.importance) : String(new Date(last.detected_at as string).getTime()), String(last.id)) : null;
	94	+ const nextCursor = rows.rows.length > f.limit && last ? encodeCursor(f.order === "importance" ? String(last.importance) : f.order === "signal" ? String(last.signal_score ?? last.importance) : String(new Date(last.detected_at as string).getTime()), String(last.id)) : null;
60	95	return { items, nextCursor };
61	96	}
62	97
	98	+export async function countEvents(f: EventFilters): Promise<number> {
	99	+ const conds = eventConditions(f);
	100	+ const r = await db.execute<{ n: string }>(sql`select count(*)::text as n from events e join sources s on s.id = e.source_id where ${sql.join(conds, sql` and `)}`);
	101	+ return Number(r.rows[0]?.n ?? 0);
	102	+}
	103	+
63	104	export async function getEvent(idOrSlug: string): Promise<Record<string, unknown> \| null> {
64	105	const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT}, e.interpretation from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.id = ${idOrSlug} or e.slug = ${idOrSlug} limit 1`);
65	106	return rows.rows[0] ?? null;
		`@@ -68,49 +109,78 @@ export async function getEvent(idOrSlug: string): Promise<Record<string, unknown`
68	109	export async function relatedEvents(ev: Record<string, unknown>, limit = 8): Promise<Record<string, unknown>[]> {
69	110	const rows = await db.execute<Record<string, unknown>>(sql`
70	111	select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id
71		− where e.id <> ${String(ev.id)} and (e.cluster_id = ${String(ev.cluster_id ?? "")} or e.source_id = ${String(ev.source_id)} or exists (select 1 from event_entities a join event_entities b on a.entity_id = b.entity_id where a.event_id = e.id and b.event_id = ${String(ev.id)}))
	112	+ where e.id <> ${String(ev.id)} and s.kind = 'registry' and (e.cluster_id = ${String(ev.cluster_id ?? "")} or e.source_id = ${String(ev.source_id)} or exists (select 1 from event_entities a join event_entities b on a.entity_id = b.entity_id where a.event_id = e.id and b.event_id = ${String(ev.id)}))
72	113	order by (e.cluster_id = ${String(ev.cluster_id ?? "")}) desc, e.detected_at desc limit ${limit}`);
73	114	return rows.rows;
74	115	}
75	116
	117	+/** Previous meaningful events on the same sensor (historical context for "what changed" — spec §79–80). */
	118	+export async function sensorHistory(sensorId: string, beforeEventId: string, limit = 6): Promise<Record<string, unknown>[]> {
	119	+ const rows = await db.execute<Record<string, unknown>>(sql`
	120	+ select e.id, e.slug, e.title, e.event_type, e.importance, e.silent_change, e.detected_at, e.field_changes from events e
	121	+ where e.sensor_id = ${sensorId} and e.id <> ${beforeEventId} order by e.detected_at desc limit ${limit}`);
	122	+ return rows.rows;
	123	+}
	124	+
76	125	export async function stats(): Promise<Record<string, unknown>> {
77	126	const [r] = (
78	127	await db.execute<Record<string, unknown>>(sql`
79	128	select
80		− (select count(*) from sources where enabled) as sources,
	129	+ (select count(*) from sources where enabled and kind = 'registry') as sources,
81	130	(select count(*) from sensors where enabled) as sensors,
82	131	(select count(*) from entities) as entities,
83	132	(select count(*) from snapshots) as snapshots,
84	133	(select count(*) from events) as events_total,
85	134	(select count(*) from events where detected_at >= now() - interval '24 hours') as events_24h,
	135	+ (select count(*) from events where detected_at >= now() - interval '1 hour') as events_1h,
86	136	(select count(*) from events where detected_at >= now() - interval '24 hours' and silent_change) as silent_24h,
87		− (select count(*) from events where detected_at >= now() - interval '24 hours' and importance >= 80) as breaking_24h,
	137	+ (select count(*) from events where detected_at >= now() - interval '24 hours' and coalesce(signal_score, importance) >= 80) as breaking_24h,
	138	+ (select count(*) from event_clusters where state = 'breaking') as breaking_now,
	139	+ (select count(*) from event_clusters where state = 'developing') as developing_now,
88	140	(select count(*) from changes where detected_at >= now() - interval '24 hours') as changes_24h,
89		− (select coalesce(sum(checks),0) from metrics_daily where day = current_date) as checks_today,
90		− (select coalesce(sum(not_modified),0) from metrics_daily where day = current_date) as not_modified_today,
91		− (select coalesce(sum(bytes),0) from metrics_daily where day = current_date) as bytes_today,
	141	+ (select coalesce(sum(checks),0) from metrics_daily where day = (now() at time zone 'UTC')::date) as checks_today,
	142	+ (select coalesce(sum(not_modified),0) from metrics_daily where day = (now() at time zone 'UTC')::date) as not_modified_today,
	143	+ (select coalesce(sum(bytes),0) from metrics_daily where day = (now() at time zone 'UTC')::date) as bytes_today,
92	144	(select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour,
	145	+ (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes') as checks_last_5m,
	146	+ (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified') as not_modified_last_5m,
93	147	(select count(*) from sensors where health = 'UP' and enabled) as sensors_up,
94	148	(select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled) as sensors_degraded,
	149	+ (select count(*) from sources where enabled and first_party and kind = 'registry') as sources_first_party,
	150	+ (select count(distinct country) from sources where country is not null) as countries,
95	151	(select max(started_at) from sensor_runs) as last_check_at,
96	152	(select max(detected_at) from events) as last_event_at,
97	153	(select percentile_cont(0.5) within group (order by processing_latency_ms) from events where detected_at >= now() - interval '24 hours') as p50_processing_ms,
98	154	(select percentile_cont(0.5) within group (order by detection_latency_ms) from events where detected_at >= now() - interval '24 hours' and detection_latency_ms is not null and detection_latency_ms < 86400000) as p50_detection_ms`)
99	155	).rows;
100		− return Object.fromEntries(Object.entries(r ?? {}).map(([k, v]) => [k, typeof v === "string" && /^\d+$/.test(v) ? Number(v) : v]));
	156	+ const out = Object.fromEntries(Object.entries(r ?? {}).map(([k, v]) => [k, typeof v === "string" && /^\d+(\.\d+)?$/.test(v) ? Number(v) : v]));
	157	+ out.checks_per_min = Math.round(Number(out.checks_last_5m ?? 0) / 5);
	158	+ out.events_per_min = Math.round((Number(out.events_1h ?? 0) / 60) * 10) / 10;
	159	+ out.not_modified_ratio_5m = Number(out.checks_last_5m) ? Math.round((Number(out.not_modified_last_5m) / Number(out.checks_last_5m)) * 100) / 100 : null;
	160	+ return out;
101	161	}
102	162
103	163	export async function trending(hours = 24, limit = 12): Promise<Record<string, unknown>[]> {
104	164	const rows = await db.execute<Record<string, unknown>>(sql`
105	165	with cur as (
106		− select ee.entity_id, count(*) as n, sum(e.importance) as imp, count(distinct e.source_id) as sources, sum(case when e.silent_change then 1 else 0 end) as silent, max(e.importance) as max_imp
107		− from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - make_interval(hours => ${hours}) group by ee.entity_id),
	166	+ select ee.entity_id, count(*) as n, sum(e.importance) as imp, count(distinct e.source_id) as sources, sum(case when e.silent_change then 1 else 0 end) as silent, max(e.importance) as max_imp,
	167	+ sum(case when e.first_party then 1 else 0 end) as first_party, sum(case when e.evidence_label = 'CONFIRMED' then 1 else 0 end) as confirmed, avg(coalesce(e.signal_score, e.importance)) as avg_signal,
	168	+ count(*) filter (where e.detected_at >= now() - make_interval(hours => ${hours}) / 4) as recent_quarter
	169	+ from events e join event_entities ee on ee.event_id = e.id join sources s on s.id = e.source_id where s.kind = 'registry' and e.detected_at >= now() - make_interval(hours => ${hours}) group by ee.entity_id),
108	170	prev as (
109	171	select ee.entity_id, count(*) as n from events e join event_entities ee on ee.event_id = e.id
110		− where e.detected_at >= now() - make_interval(hours => ${hours * 2}) and e.detected_at < now() - make_interval(hours => ${hours}) group by ee.entity_id)
111		− select en.id, en.name, en.type, en.domain, cur.n::int as events, cur.imp::float as importance_sum, cur.sources::int as sources, cur.silent::int as silent, cur.max_imp::float as max_importance, coalesce(prev.n,0)::int as prev_events,
112		− round((18(ln(1+cur.n)/ln(2)) + 0.35(cur.imp/greatest(1,cur.n)) + 10(ln(1+cur.sources)/ln(2)) + 12least(2, case when coalesce(prev.n,0)=0 then 2 else cur.n::float/prev.n end) + 5*least(3,cur.silent))::numeric, 1)::float as score
113		− from cur join entities en on en.id = cur.entity_id left join prev on prev.entity_id = cur.entity_id
	172	+ where e.detected_at >= now() - make_interval(hours => ${hours * 2}) and e.detected_at < now() - make_interval(hours => ${hours}) group by ee.entity_id),
	173	+ base as (
	174	+ select entity_id, avg(events)::float as per_day from entity_daily where day >= (now() at time zone 'UTC')::date - 30 and day < (now() at time zone 'UTC')::date group by entity_id)
	175	+ select en.id, en.name, en.type, en.domain, en.importance as entity_importance, cur.n::int as events, cur.imp::float as importance_sum, cur.sources::int as sources, cur.silent::int as silent, cur.max_imp::float as max_importance,
	176	+ cur.first_party::int as first_party, cur.confirmed::int as confirmed, round(cur.avg_signal::numeric, 1)::float as avg_signal, coalesce(prev.n,0)::int as prev_events, coalesce(base.per_day, 0)::float as baseline_per_day,
	177	+ cur.recent_quarter::int as recent_quarter,
	178	+ case when cur.recent_quarter::float / greatest(1, cur.n) > 0.5 then 'up' when cur.recent_quarter = 0 and cur.n >= 3 then 'down' else 'flat' end as direction,
	179	+ round((100 * (1 - exp(-(
	180	+ 18(ln(1+cur.n)/ln(2)) + 0.35(cur.imp/greatest(1,cur.n)) + 10(ln(1+cur.sources)/ln(2)) + 12least(2, case when coalesce(prev.n,0)=0 then 2 else cur.n::float/prev.n end) + 5*least(3,cur.silent)
	181	+ + 6least(3, cur.first_party) + 4least(3, cur.confirmed) + (case when coalesce(base.per_day,0) > 0 then 8least(3, cur.n / (base.per_day ${hours} / 24.0)) else 0 end)
	182	+ ) / 110.0)))::numeric, 1)::float as score
	183	+ from cur join entities en on en.id = cur.entity_id left join prev on prev.entity_id = cur.entity_id left join base on base.entity_id = cur.entity_id
114	184	order by score desc limit ${limit}`);
115	185	return rows.rows;
116	186	}
		`@@ -122,7 +192,9 @@ export async function sourceActivity(sourceId: string): Promise<Record<string, u`
122	192	(select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '2 hours')::int as changes_2h,
123	193	(select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '14 days')::int as changes_14d,
124	194	(select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '24 hours')::int as events_24h,
125		− (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '14 days')::int as events_14d`)
	195	+ (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '14 days')::int as events_14d,
	196	+ (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '24 hours' and silent_change)::int as silent_24h,
	197	+ (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '24 hours' and coalesce(signal_score, importance) >= 80)::int as breaking_24h`)
126	198	).rows;
127	199	const c2h = Number(r?.changes_2h ?? 0);
128	200	const baselinePerHour = Number(r?.changes_14d ?? 0) / (14 * 24);
		`@@ -136,6 +208,35 @@ export async function sourceActivity(sourceId: string): Promise<Record<string, u`
136	208	return { ...r, baseline_changes_per_day: Math.round(baselinePerHour * 24 * 10) / 10, activity_score: Math.round(anomaly * 10) / 10 };
137	209	}
138	210
	211	+/** Source quality score (spec §48): success rate, latency, structured share, usefulness (events/raw), consistency. Distinct from importance. */
	212	+export async function sourceQuality(sourceId: string): Promise<Record<string, unknown>> {
	213	+ const [r] = (
	214	+ await db.execute<Record<string, unknown>>(sql`
	215	+ select
	216	+ (select count(*) from sensors where source_id = ${sourceId} and enabled)::int as sensors,
	217	+ (select count(*) from sensors where source_id = ${sourceId} and enabled and health = 'UP')::int as sensors_up,
	218	+ (select avg(avg_latency_ms) from sensors where source_id = ${sourceId} and enabled)::int as avg_latency_ms,
	219	+ (select coalesce(sum(raw_changes),0) from sensors where source_id = ${sourceId})::int as raw_changes,
	220	+ (select coalesce(sum(meaningful_changes),0) from sensors where source_id = ${sourceId})::int as meaningful_changes,
	221	+ (select coalesce(sum(total_runs),0) from sensors where source_id = ${sourceId})::int as total_runs,
	222	+ (select coalesce(sum(total_not_modified),0) from sensors where source_id = ${sourceId})::int as total_not_modified,
	223	+ (select count(*) from sensors where source_id = ${sourceId} and enabled and connector <> 'http')::int as structured_sensors,
	224	+ (select coalesce(sum(checks),0) from source_daily where source_id = ${sourceId} and day >= (now() at time zone 'UTC')::date - 7)::int as checks_7d,
	225	+ (select coalesce(sum(errors),0) from source_daily where source_id = ${sourceId} and day >= (now() at time zone 'UTC')::date - 7)::int as errors_7d,
	226	+ (select avg(confidence) from events where source_id = ${sourceId} and detected_at >= now() - interval '30 days')::float as avg_confidence`)
	227	+ ).rows;
	228	+ const sensors = Number(r?.sensors ?? 0);
	229	+ const success = Number(r?.checks_7d) ? 1 - Number(r?.errors_7d) / Number(r?.checks_7d) : sensors ? Number(r?.sensors_up) / sensors : 1;
	230	+ const latency = Number(r?.avg_latency_ms ?? 800);
	231	+ const latencyScore = latency <= 400 ? 1 : latency <= 1500 ? 0.8 : latency <= 4000 ? 0.6 : 0.4;
	232	+ const structured = sensors ? Number(r?.structured_sensors) / sensors : 0;
	233	+ const raw = Number(r?.raw_changes ?? 0);
	234	+ const usefulness = raw >= 5 ? Math.min(1, (Number(r?.meaningful_changes) / raw) * 2) : 0.6;
	235	+ const conf = Number(r?.avg_confidence ?? 70) / 100;
	236	+ const score = Math.round(100 * (0.35 * success + 0.15 * latencyScore + 0.15 * structured + 0.2 * usefulness + 0.15 * conf));
	237	+ return { ...r, success_rate: Math.round(success * 1000) / 1000, structured_share: Math.round(structured * 100) / 100, usefulness: Math.round(usefulness * 100) / 100, quality_score: score };
	238	+}
	239	+
139	240	export function encodeCursor(a: string, b: string): string {
140	241	return Buffer.from(`${a}\|${b}`).toString("base64url");
141	242	}

added apps/api/src/routes-admin.ts +228 −0

		`@@ -0,0 +1,228 @@`
	1	+import { readdir, stat } from "node:fs/promises";
	2	+import { join, resolve } from "node:path";
	3	+import type { FastifyInstance } from "fastify";
	4	+import YAML from "yaml";
	5	+import { z } from "zod";
	6	+import { extendSchema, importDocumentSchema, sourceSchema, type SensorEndpoint, type Tier } from "@websensor/core";
	7	+import { getConnector, listConnectors, NormalizeError } from "@websensor/connectors";
	8	+import { db, sql } from "@websensor/db";
	9	+import { cacheStats, invalidate } from "./cache";
	10	+import { config } from "./config";
	11	+import { engineStatus, liveStats } from "./live";
	12	+
	13	+/**
	14	+ * Internal operations (spec §70–72, §85). Enabled only when WS_ADMIN_TOKEN is set; every request
	15	+ * must carry `X-WebSensor-Admin: <token>`. Never linked from public pages; `/ops` in the web app
	16	+ * asks for the token and stores it in sessionStorage.
	17	+ */
	18	+export async function registerAdminRoutes(app: FastifyInstance): Promise<void> {
	19	+ app.addHook("onRequest", async (req, reply) => {
	20	+ if (!req.url.startsWith("/api/v1/admin")) return;
	21	+ if (!config.adminToken) return reply.status(404).send({ error: "not_found" });
	22	+ const t = req.headers["x-websensor-admin"];
	23	+ if (typeof t !== "string" \|\| t !== config.adminToken) return reply.status(401).send({ error: "admin_token_required" });
	24	+ });
	25	+
	26	+ app.get("/api/v1/admin/ops", async () => {
	27	+ const t0 = Date.now();
	28	+ await db.execute(sql`select 1`);
	29	+ const dbLatency = Date.now() - t0;
	30	+ const [queue, failing, slow, throughput, storage, llm, jobs, es, byStatus, recentErrors] = await Promise.all([
	31	+ db.execute<Record<string, unknown>>(sql`select count() filter (where next_check_at <= now())::int as due, count() filter (where next_check_at <= now() - interval '10 minutes')::int as overdue_10m, count() filter (where priority = 0)::int as p0, count() filter (where priority = 0 and next_check_at <= now())::int as p0_due, min(next_check_at) as oldest_due from sensors where enabled`).then((r) => r.rows[0]),
	32	+ db.execute<Record<string, unknown>>(sql`select split_part(split_part(s.url, '/', 3), ':', 1) as host, count(*)::int as failures, count(distinct s.id)::int as sensors, max(r.error) as last_error, max(r.started_at) as last_at from sensor_runs r join sensors s on s.id = r.sensor_id where r.started_at >= now() - interval '6 hours' and r.outcome in ('error','parse_error','rate_limited') group by 1 order by failures desc limit 20`).then((r) => r.rows),
	33	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.url, s.connector, s.avg_latency_ms, s.total_runs from sensors s where s.enabled and s.avg_latency_ms is not null order by s.avg_latency_ms desc limit 20`).then((r) => r.rows),
	34	+ db.execute<Record<string, unknown>>(sql`
	35	+ select to_char(date_trunc('hour', started_at at time zone 'UTC'), 'YYYY-MM-DD"T"HH24:00:00"Z"') as t, count()::int as checks, count() filter (where outcome = 'not_modified')::int as not_modified, count() filter (where outcome in ('error','parse_error','rate_limited'))::int as errors, count() filter (where outcome = 'event')::int as events, avg(duration_ms)::int as avg_ms
	36	+ from sensor_runs where started_at >= now() - interval '24 hours' group by 1 order by 1`).then((r) => r.rows),
	37	+ Promise.all([
	38	+ db.execute<Record<string, unknown>>(sql`select pg_size_pretty(pg_database_size(current_database())) as db_size, pg_database_size(current_database())::bigint as db_bytes, (select count() from snapshots)::int as snapshots, (select count() from snapshots where storage_key is not null)::int as snapshots_with_raw, (select coalesce(sum(content_length),0) from snapshots where storage_key is not null)::bigint as raw_bytes_uncompressed, (select count() from changes)::int as changes, (select count() from events)::int as events, (select count(*) from sensor_runs)::int as runs`).then((r) => r.rows[0]),
	39	+ blobDirSize(),
	40	+ ]).then(([d, blobs]) => ({ ...d, blobs })),
	41	+ db.execute<Record<string, unknown>>(sql`select model, count()::int as calls, sum(input_tokens)::bigint as input_tokens, sum(output_tokens)::bigint as output_tokens, count() filter (where not ok)::int as failures from llm_usage where at >= now() - interval '24 hours' group by model`).then((r) => r.rows),
	42	+ db.execute<Record<string, unknown>>(sql`select outcome, count(*)::int as n from sensor_runs where started_at >= now() - interval '1 hour' group by outcome order by n desc`).then((r) => r.rows),
	43	+ engineStatus(),
	44	+ db.execute<Record<string, unknown>>(sql`select status, health, count(*)::int as n from sensors group by status, health order by n desc`).then((r) => r.rows),
	45	+ db.execute<Record<string, unknown>>(sql`select r.started_at, s.id as sensor_id, s.source_id, s.connector, r.http_status, r.outcome, r.error from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error') and r.started_at >= now() - interval '1 hour' order by r.started_at desc limit 30`).then((r) => r.rows),
	46	+ ]);
	47	+ return { db: { latency_ms: dbLatency }, queue, engine: es, workers: es ? { inflight: es.inflight, concurrency: es.concurrency, busy_hosts: es.busyHosts, circuit_open: es.circuitOpen } : null, outcomes_1h: jobs, top_failing_domains: failing, slowest_sensors: slow, throughput_24h: throughput, storage, llm_24h: llm, sensors_by_status: byStatus, recent_errors: recentErrors, cache: cacheStats(), live: liveStats(), connectors: listConnectors().map((c) => c.metadata()), generated_at: new Date().toISOString() };
	48	+ });
	49	+
	50	+ // ---- Sensor actions ------------------------------------------------------------------------------
	51	+ app.post<{ Params: { id: string } }>("/api/v1/admin/sensors/:id/run-now", async (req, reply) => {
	52	+ const r = await db.execute(sql`update sensors set next_check_at = now() - interval '1 second', enabled = true where id = ${req.params.id}`);
	53	+ if (!r.rowCount) return reply.status(404).send({ error: "not_found" });
	54	+ return { ok: true, scheduled: "now" };
	55	+ });
	56	+ app.post<{ Params: { id: string } }>("/api/v1/admin/sensors/:id/enable", async (req, reply) => {
	57	+ const r = await db.execute(sql`update sensors set enabled = true, status = 'ACTIVE', consecutive_errors = 0, next_check_at = now(), updated_at = now() where id = ${req.params.id}`);
	58	+ if (!r.rowCount) return reply.status(404).send({ error: "not_found" });
	59	+ return { ok: true };
	60	+ });
	61	+ app.post<{ Params: { id: string } }>("/api/v1/admin/sensors/:id/disable", async (req, reply) => {
	62	+ const r = await db.execute(sql`update sensors set enabled = false, status = 'DISABLED', updated_at = now() where id = ${req.params.id}`);
	63	+ if (!r.rowCount) return reply.status(404).send({ error: "not_found" });
	64	+ return { ok: true };
	65	+ });
	66	+ app.patch<{ Params: { id: string } }>("/api/v1/admin/sensors/:id", async (req, reply) => {
	67	+ const body = z.object({ tier: z.enum(["S", "A", "B", "C", "D"]).optional(), priority: z.number().int().min(0).max(3).optional(), base_interval_seconds: z.number().int().min(15).max(7 * 86400).nullable().optional(), config: z.record(z.string(), z.unknown()).optional(), name: z.string().min(1).max(120).optional() }).parse(req.body ?? {});
	68	+ const sets: ReturnType<typeof sql>[] = [sql`updated_at = now()`];
	69	+ if (body.tier) sets.push(sql`tier = ${body.tier}`);
	70	+ if (body.priority !== undefined) sets.push(sql`priority = ${body.priority}`);
	71	+ if (body.base_interval_seconds !== undefined) sets.push(sql`base_interval_seconds = ${body.base_interval_seconds}`);
	72	+ if (body.config) sets.push(sql`config = config \|\| ${JSON.stringify(body.config)}::jsonb`);
	73	+ if (body.name) sets.push(sql`name = ${body.name}`);
	74	+ const r = await db.execute(sql`update sensors set ${sql.join(sets, sql`, `)} where id = ${req.params.id}`);
	75	+ if (!r.rowCount) return reply.status(404).send({ error: "not_found" });
	76	+ return { ok: true };
	77	+ });
	78	+ /** Dry-run a connector against a URL (or an existing sensor) without persisting anything: inspect response + normalized output. */
	79	+ app.post("/api/v1/admin/sensors/test", async (req, reply) => {
	80	+ const body = z.object({ sensor_id: z.string().optional(), url: z.string().url().optional(), connector: z.string().default("http"), type: z.string().default("HTML"), config: z.record(z.string(), z.unknown()).default({}) }).parse(req.body ?? {});
	81	+ let endpoint: SensorEndpoint;
	82	+ if (body.sensor_id) {
	83	+ const s = (await db.execute<Record<string, unknown>>(sql`select * from sensors where id = ${body.sensor_id}`)).rows[0];
	84	+ if (!s) return reply.status(404).send({ error: "not_found" });
	85	+ endpoint = { id: String(s.id), sourceId: String(s.source_id), name: String(s.name), url: String(s.url), type: String(s.type) as SensorEndpoint["type"], tier: String(s.tier) as Tier, connector: String(s.connector), config: (s.config as Record<string, unknown>) ?? {}, etag: null, lastModified: null, state: null };
	86	+ } else if (body.url) {
	87	+ endpoint = { id: "admin_test", sourceId: "admin", name: "test", url: body.url, type: body.type as SensorEndpoint["type"], tier: "C", connector: body.connector, config: body.config, etag: null, lastModified: null, state: null };
	88	+ } else return reply.status(400).send({ error: "sensor_id_or_url_required" });
	89	+ let connector;
	90	+ try {
	91	+ connector = getConnector(endpoint.connector);
	92	+ } catch (e) {
	93	+ return reply.status(400).send({ error: "unknown_connector", detail: (e as Error).message });
	94	+ }
	95	+ const t0 = Date.now();
	96	+ const obs = await connector.fetch(endpoint);
	97	+ const fetchMs = Date.now() - t0;
	98	+ if (obs.error) return { ok: false, stage: "fetch", error: obs.error, meta: obs.meta, fetch_ms: fetchMs };
	99	+ if (obs.meta.status >= 400) return { ok: false, stage: "fetch", http_status: obs.meta.status, meta: obs.meta, fetch_ms: fetchMs };
	100	+ try {
	101	+ const norm = await connector.normalize(endpoint, obs);
	102	+ return { ok: true, fetch_ms: fetchMs, meta: obs.meta, normalized: { mode: norm.mode, title: norm.title ?? null, extractionConfidence: norm.extractionConfidence, items: norm.items?.length ?? null, sample_items: norm.items?.slice(0, 5) ?? null, text_preview: norm.text?.slice(0, 1500) ?? null, json_preview: norm.json !== undefined ? JSON.stringify(norm.json).slice(0, 1500) : null, canonicalHash: norm.canonicalHash, publishedAt: norm.publishedAt ?? null } };
	103	+ } catch (e) {
	104	+ return { ok: false, stage: "normalize", error: e instanceof NormalizeError ? { code: e.code, message: e.message } : { code: "error", message: (e as Error).message }, meta: obs.meta, fetch_ms: fetchMs, body_preview: obs.body?.toString("utf8").slice(0, 800) ?? null };
	105	+ }
	106	+ });
	107	+
	108	+ // ---- Source actions ------------------------------------------------------------------------------
	109	+ app.post<{ Params: { id: string } }>("/api/v1/admin/sources/:id/enable", async (req) => {
	110	+ await db.execute(sql`update sources set enabled = true, updated_at = now() where id = ${req.params.id}`);
	111	+ invalidate("");
	112	+ return { ok: true };
	113	+ });
	114	+ app.post<{ Params: { id: string } }>("/api/v1/admin/sources/:id/disable", async (req) => {
	115	+ await db.execute(sql`update sources set enabled = false, updated_at = now() where id = ${req.params.id}`);
	116	+ invalidate("");
	117	+ return { ok: true };
	118	+ });
	119	+ app.post<{ Params: { id: string } }>("/api/v1/admin/sources/:id/run-now", async (req) => {
	120	+ const r = await db.execute(sql`update sensors set next_check_at = now() - interval '1 second' where source_id = ${req.params.id} and enabled`);
	121	+ return { ok: true, sensors: r.rowCount ?? 0 };
	122	+ });
	123	+ app.post("/api/v1/admin/sensors/bulk", async (req) => {
	124	+ const body = z.object({ ids: z.array(z.string()).min(1).max(2000), action: z.enum(["enable", "disable", "run-now"]) }).parse(req.body ?? {});
	125	+ const arr = sql.raw("array[" + body.ids.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]");
	126	+ let r;
	127	+ if (body.action === "enable") r = await db.execute(sql`update sensors set enabled = true, status = 'ACTIVE', consecutive_errors = 0, next_check_at = now() where id = any(${arr})`);
	128	+ else if (body.action === "disable") r = await db.execute(sql`update sensors set enabled = false, status = 'DISABLED' where id = any(${arr})`);
	129	+ else r = await db.execute(sql`update sensors set next_check_at = now() - interval '1 second' where id = any(${arr}) and enabled`);
	130	+ return { ok: true, affected: r.rowCount ?? 0 };
	131	+ });
	132	+
	133	+ /**
	134	+ * Bulk import (spec §71): JSON body `{ sources: [...] }` or raw YAML (content-type text/yaml). Validates with the
	135	+ * registry schema first; `dry_run=1` returns the validation report without writing. Imported sources are stored
	136	+ * with `kind = 'registry'` and `config.seed = false` so the file-based sync never disables them.
	137	+ */
	138	+ app.post<{ Querystring: { dry_run?: string } }>("/api/v1/admin/sources/import", async (req, reply) => {
	139	+ let doc: unknown = req.body;
	140	+ if (typeof req.body === "string") {
	141	+ try {
	142	+ doc = YAML.parse(req.body);
	143	+ } catch (e) {
	144	+ return reply.status(400).send({ error: "yaml_parse_error", detail: (e as Error).message });
	145	+ }
	146	+ }
	147	+ const parsed = importDocumentSchema.safeParse(doc);
	148	+ if (!parsed.success) return reply.status(400).send({ error: "invalid_document", issues: parsed.error.issues });
	149	+ const report: { id: string \| undefined; ok: boolean; extend: boolean; issues?: string[]; sensors?: number }[] = [];
	150	+ const existing = new Set((await db.execute<{ id: string }>(sql`select id from sources`)).rows.map((r) => r.id));
	151	+ const valid: { extend: boolean; data: z.infer<typeof sourceSchema> \| z.infer<typeof extendSchema> }[] = [];
	152	+ for (const s of parsed.data.sources) {
	153	+ const isExtend = (s as { extend?: boolean })?.extend === true;
	154	+ const res = isExtend ? extendSchema.safeParse(s) : sourceSchema.safeParse(s);
	155	+ const id = (s as { id?: string })?.id;
	156	+ if (!res.success) {
	157	+ report.push({ id, ok: false, extend: isExtend, issues: res.error.issues.map((i) => `${i.path.join(".")}: ${i.message}`) });
	158	+ continue;
	159	+ }
	160	+ if (isExtend && !existing.has(res.data.id)) {
	161	+ report.push({ id, ok: false, extend: true, issues: ["extend: true but source does not exist"] });
	162	+ continue;
	163	+ }
	164	+ if (!isExtend && existing.has(res.data.id)) {
	165	+ report.push({ id, ok: false, extend: false, issues: ["source id already exists (use extend: true)"] });
	166	+ continue;
	167	+ }
	168	+ valid.push({ extend: isExtend, data: res.data });
	169	+ report.push({ id, ok: true, extend: isExtend, sensors: res.data.sensors.length });
	170	+ }
	171	+ if (req.query.dry_run === "1") return { dry_run: true, valid: valid.length, invalid: report.filter((r) => !r.ok).length, report };
	172	+ let sensors = 0;
	173	+ for (const v of valid) {
	174	+ if (!v.extend) {
	175	+ const s = v.data as z.infer<typeof sourceSchema>;
	176	+ await db.execute(sql`insert into sources (id, name, domain, homepage, description, categories, tier, importance_weight, discover, fallback, notes, enabled, llm_enabled, first_party, country, language, kind)
	177	+ values (${s.id}, ${s.name}, ${s.domain}, ${s.homepage ?? `https://${s.domain}`}, ${s.description ?? null}, ${sql.raw("'{" + s.categories.map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.tier}, ${s.weight}, ${JSON.stringify(s.discover)}::jsonb, ${JSON.stringify(s.fallback)}::jsonb, ${s.notes ?? null}, ${s.enabled}, ${s.llm}, ${s.first_party ?? true}, ${s.country ?? null}, ${s.language ?? null}, 'registry')`);
	178	+ const entId = `org_${s.id}`;
	179	+ await db.execute(sql`insert into entities (id, name, type, domain, homepage, description, importance, categories) values (${entId}, ${s.name}, ${s.entity_type}, ${s.domain}, ${s.homepage ?? `https://${s.domain}`}, ${s.description ?? null}, 60, ${sql.raw("'{" + s.categories.map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}) on conflict (id) do nothing`);
	180	+ await db.execute(sql`insert into source_entities (source_id, entity_id) values (${s.id}, ${entId}) on conflict do nothing`);
	181	+ for (const a of new Set([s.name, s.domain, ...s.aliases])) await db.execute(sql`insert into entity_aliases (alias, entity_id) values (${a.toLowerCase()}, ${entId}) on conflict do nothing`);
	182	+ }
	183	+ const base = v.data;
	184	+ for (const sen of base.sensors) {
	185	+ const id = sen.id ?? `${base.id}_${sen.name.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-\|-$/g, "")}`;
	186	+ await db.execute(sql`insert into sensors (id, source_id, name, url, type, connector, tier, importance_weight, config, base_interval_seconds, status, priority)
	187	+ values (${id}, ${base.id}, ${sen.name}, ${sen.url}, ${sen.type}, ${sen.connector}, ${sen.tier ?? (v.extend ? "B" : (v.data as z.infer<typeof sourceSchema>).tier)}, ${sen.weight ?? 1}, ${JSON.stringify({ ...sen.config, seed: false, imported_at: new Date().toISOString() })}::jsonb, ${sen.interval ?? null}, 'PENDING', 2)
	188	+ on conflict (id) do update set name = excluded.name, url = excluded.url, type = excluded.type, connector = excluded.connector, config = excluded.config, enabled = true, updated_at = now()`);
	189	+ sensors++;
	190	+ }
	191	+ }
	192	+ invalidate("");
	193	+ return { dry_run: false, imported_sources: valid.filter((v) => !v.extend).length, extended_sources: valid.filter((v) => v.extend).length, sensors, invalid: report.filter((r) => !r.ok).length, report };
	194	+ });
	195	+
	196	+ app.get("/api/v1/admin/failures", async () => {
	197	+ const rows = await db.execute<Record<string, unknown>>(sql`select r.started_at, r.outcome, r.http_status, r.error, r.duration_ms, s.id as sensor_id, s.name, s.source_id, s.url, s.connector from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error','rate_limited') and r.started_at >= now() - interval '24 hours' order by r.started_at desc limit 200`);
	198	+ return { items: rows.rows };
	199	+ });
	200	+}
	201	+
	202	+async function blobDirSize(): Promise<{ files: number; bytes: number } \| null> {
	203	+ const root = resolve(process.env.BLOB_STORE_DIR ?? "./data/blobs");
	204	+ let files = 0;
	205	+ let bytes = 0;
	206	+ const started = Date.now();
	207	+ const walk = async (dir: string): Promise<void> => {
	208	+ if (Date.now() - started > 4000) return; // bounded
	209	+ let entries: string[];
	210	+ try {
	211	+ entries = await readdir(dir);
	212	+ } catch {
	213	+ return;
	214	+ }
	215	+ for (const e of entries) {
	216	+ const p = join(dir, e);
	217	+ const st = await stat(p).catch(() => null);
	218	+ if (!st) continue;
	219	+ if (st.isDirectory()) await walk(p);
	220	+ else {
	221	+ files++;
	222	+ bytes += st.size;
	223	+ }
	224	+ }
	225	+ };
	226	+ await walk(root);
	227	+ return { files, bytes };
	228	+}

added apps/api/src/routes-user.ts +299 −0

		`@@ -0,0 +1,299 @@`
	1	+import { createHash } from "node:crypto";
	2	+import type { FastifyInstance } from "fastify";
	3	+import { z } from "zod";
	4	+import { assertUrlAllowed, newId, UrlPolicyError } from "@websensor/core";
	5	+import { getConnector } from "@websensor/connectors";
	6	+import { db, sql, textArray } from "@websensor/db";
	7	+import { config } from "./config";
	8	+import { EVENT_SELECT, eventConditions, listEvents } from "./queries";
	9	+
	10	+/**
	11	+ * Owner-scoped routes (anonymous owner token, phase 1): watchlists, alert rules (+ webhook
	12	+ * channel), notifications, bookmarks, saved views and custom URL monitors (spec §42–45, §81, §100).
	13	+ */
	14	+export function ownerToken(headers: Record<string, unknown>): string \| null {
	15	+ const t = headers["x-websensor-owner"];
	16	+ return typeof t === "string" && /^[A-Za-z0-9_-]{16,80}$/.test(t) ? t : null;
	17	+}
	18	+
	19	+const WATCH_KINDS = ["entity", "source", "keyword", "category", "url", "event_type", "country", "group"] as const;
	20	+
	21	+export async function registerUserRoutes(app: FastifyInstance): Promise<void> {
	22	+ const requireOwner = (req: { headers: Record<string, unknown> }, reply: { status: (n: number) => { send: (b: unknown) => unknown } }): string \| null => {
	23	+ const owner = ownerToken(req.headers);
	24	+ if (!owner) reply.status(401).send({ error: "owner_token_required", detail: "Send an X-WebSensor-Owner header (16–80 URL-safe characters)." });
	25	+ return owner;
	26	+ };
	27	+
	28	+ // ---- Watchlists -----------------------------------------------------------------------------
	29	+ app.get("/api/v1/watchlists", async (req, reply) => {
	30	+ const owner = requireOwner(req, reply);
	31	+ if (!owner) return;
	32	+ const rows = await db.execute<Record<string, unknown>>(sql`select w.id, w.name, w.created_at, coalesce((select json_agg(json_build_object('kind', i.kind, 'value', i.value, 'added_at', i.added_at) order by i.added_at) from watchlist_items i where i.watchlist_id = w.id), '[]'::json) as items from watchlists w where owner_token = ${owner} order by created_at`);
	33	+ return { items: rows.rows };
	34	+ });
	35	+ const wlBody = z.object({ name: z.string().min(1).max(80).default("My watchlist"), items: z.array(z.object({ kind: z.enum(WATCH_KINDS), value: z.string().min(1).max(300) })).max(300).default([]) });
	36	+ app.post("/api/v1/watchlists", async (req, reply) => {
	37	+ const owner = requireOwner(req, reply);
	38	+ if (!owner) return;
	39	+ const body = wlBody.parse(req.body ?? {});
	40	+ const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from watchlists where owner_token = ${owner}`)).rows[0]?.n;
	41	+ if (Number(count) >= 20) return reply.status(429).send({ error: "too_many_watchlists" });
	42	+ const id = newId("wl");
	43	+ await db.execute(sql`insert into watchlists (id, owner_token, name) values (${id}, ${owner}, ${body.name})`);
	44	+ for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${id}, ${it.kind}, ${it.value}) on conflict do nothing`);
	45	+ return { id, name: body.name, items: body.items };
	46	+ });
	47	+ app.put<{ Params: { id: string } }>("/api/v1/watchlists/:id", async (req, reply) => {
	48	+ const owner = requireOwner(req, reply);
	49	+ if (!owner) return;
	50	+ const w = (await db.execute<{ id: string }>(sql`select id from watchlists where id = ${req.params.id} and owner_token = ${owner}`)).rows[0];
	51	+ if (!w) return reply.status(404).send({ error: "not_found" });
	52	+ const body = wlBody.partial().parse(req.body ?? {});
	53	+ if (body.name) await db.execute(sql`update watchlists set name = ${body.name} where id = ${w.id}`);
	54	+ if (body.items) {
	55	+ await db.execute(sql`delete from watchlist_items where watchlist_id = ${w.id}`);
	56	+ for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${w.id}, ${it.kind}, ${it.value}) on conflict do nothing`);
	57	+ }
	58	+ return { ok: true };
	59	+ });
	60	+ app.delete<{ Params: { id: string } }>("/api/v1/watchlists/:id", async (req, reply) => {
	61	+ const owner = requireOwner(req, reply);
	62	+ if (!owner) return;
	63	+ await db.execute(sql`delete from watchlists where id = ${req.params.id} and owner_token = ${owner}`);
	64	+ return { ok: true };
	65	+ });
	66	+ app.get<{ Params: { id: string }; Querystring: { limit?: string } }>("/api/v1/watchlists/:id/events", async (req, reply) => {
	67	+ const owner = requireOwner(req, reply);
	68	+ if (!owner) return;
	69	+ const items = (await db.execute<{ kind: string; value: string }>(sql`select i.kind, i.value from watchlist_items i join watchlists w on w.id = i.watchlist_id where w.id = ${req.params.id} and w.owner_token = ${owner}`)).rows;
	70	+ if (!items.length) return reply.send({ items: [] });
	71	+ const by = (k: string): string[] => items.filter((i) => i.kind === k).map((i) => i.value);
	72	+ const conds = [] as ReturnType<typeof sql>[];
	73	+ const ents = by("entity");
	74	+ const srcs = by("source");
	75	+ const cats = by("category");
	76	+ const types = by("event_type");
	77	+ const countries = by("country");
	78	+ const urls = by("url");
	79	+ if (ents.length) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = any(${textArray(ents)}))`);
	80	+ if (srcs.length) conds.push(sql`e.source_id = any(${textArray(srcs)})`);
	81	+ if (cats.length) conds.push(sql`e.categories && ${textArray(cats)}`);
	82	+ if (types.length) conds.push(sql`e.event_type = any(${textArray(types)})`);
	83	+ if (countries.length) conds.push(sql`e.country = any(${textArray(countries.map((c) => c.toUpperCase()))})`);
	84	+ for (const u of urls) conds.push(sql`(e.url = ${u} or e.url like ${u.replace(/\/$/, "") + "/%"})`);
	85	+ for (const k of by("keyword")) conds.push(sql`(e.title ilike ${"%" + k + "%"} or e.summary ilike ${"%" + k + "%"})`);
	86	+ if (!conds.length) return { items: [] };
	87	+ const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where (s.kind = 'registry' or s.owner_token = ${owner}) and (${sql.join(conds, sql` or `)}) order by e.detected_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
	88	+ return { items: rows.rows };
	89	+ });
	90	+
	91	+ // ---- Alerts -----------------------------------------------------------------------------------
	92	+ const ruleSchema = z.object({
	93	+ importance_min: z.number().min(0).max(100).optional(),
	94	+ signal_min: z.number().min(0).max(100).optional(),
	95	+ event_types: z.array(z.string().max(60)).max(60).optional(),
	96	+ groups: z.array(z.string().max(30)).max(12).optional(),
	97	+ entities: z.array(z.string().max(120)).max(100).optional(),
	98	+ sources: z.array(z.string().max(120)).max(100).optional(),
	99	+ keywords: z.array(z.string().max(80)).max(50).optional(),
	100	+ categories: z.array(z.string().max(40)).max(40).optional(),
	101	+ countries: z.array(z.string().max(3)).max(60).optional(),
	102	+ silent_only: z.boolean().optional(),
	103	+ first_party_only: z.boolean().optional(),
	104	+ confirmed_only: z.boolean().optional(),
	105	+ });
	106	+ const alertBody = z.object({ name: z.string().min(1).max(80), rule: ruleSchema, channel: z.enum(["web", "webhook"]).default("web"), channel_config: z.object({ url: z.string().url().max(500).optional(), secret: z.string().max(200).optional() }).default({}) });
	107	+ app.get("/api/v1/alerts", async (req, reply) => {
	108	+ const owner = requireOwner(req, reply);
	109	+ if (!owner) return;
	110	+ const rows = await db.execute<Record<string, unknown>>(sql`select a.id, a.name, a.rule, a.channel, a.channel_config - 'secret' as channel_config, a.enabled, a.created_at, a.last_fired_at, a.fired_count, (select count(*) from notifications n where n.alert_id = a.id and n.created_at >= now() - interval '24 hours')::int as fired_24h from alerts a where owner_token = ${owner} order by created_at`);
	111	+ return { items: rows.rows, channels: { web: "active", webhook: "active", email: "planned", slack: "planned", discord: "planned", telegram: "planned", push: "planned" } };
	112	+ });
	113	+ app.post("/api/v1/alerts", async (req, reply) => {
	114	+ const owner = requireOwner(req, reply);
	115	+ if (!owner) return;
	116	+ const body = alertBody.parse(req.body ?? {});
	117	+ if (body.channel === "webhook") {
	118	+ if (!body.channel_config.url) return reply.status(400).send({ error: "webhook_url_required" });
	119	+ try {
	120	+ const u = await assertUrlAllowed(body.channel_config.url);
	121	+ if (u.url.protocol !== "https:") return reply.status(400).send({ error: "webhook_https_required" });
	122	+ } catch (e) {
	123	+ return reply.status(400).send({ error: "webhook_url_rejected", detail: e instanceof UrlPolicyError ? e.message : String(e) });
	124	+ }
	125	+ }
	126	+ const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from alerts where owner_token = ${owner}`)).rows[0]?.n;
	127	+ if (Number(count) >= 50) return reply.status(429).send({ error: "too_many_alerts" });
	128	+ const id = newId("alr");
	129	+ await db.execute(sql`insert into alerts (id, owner_token, name, rule, channel, channel_config) values (${id}, ${owner}, ${body.name}, ${JSON.stringify(body.rule)}::jsonb, ${body.channel}, ${JSON.stringify(body.channel_config)}::jsonb)`);
	130	+ return { id, name: body.name, rule: body.rule, channel: body.channel };
	131	+ });
	132	+ app.patch<{ Params: { id: string } }>("/api/v1/alerts/:id", async (req, reply) => {
	133	+ const owner = requireOwner(req, reply);
	134	+ if (!owner) return;
	135	+ const body = z.object({ enabled: z.boolean().optional(), name: z.string().min(1).max(80).optional() }).parse(req.body ?? {});
	136	+ if (body.enabled !== undefined) await db.execute(sql`update alerts set enabled = ${body.enabled} where id = ${req.params.id} and owner_token = ${owner}`);
	137	+ if (body.name) await db.execute(sql`update alerts set name = ${body.name} where id = ${req.params.id} and owner_token = ${owner}`);
	138	+ return { ok: true };
	139	+ });
	140	+ app.delete<{ Params: { id: string } }>("/api/v1/alerts/:id", async (req, reply) => {
	141	+ const owner = requireOwner(req, reply);
	142	+ if (!owner) return;
	143	+ await db.execute(sql`delete from alerts where id = ${req.params.id} and owner_token = ${owner}`);
	144	+ return { ok: true };
	145	+ });
	146	+ app.get<{ Querystring: { limit?: string; unread?: string } }>("/api/v1/notifications", async (req, reply) => {
	147	+ const owner = requireOwner(req, reply);
	148	+ if (!owner) return;
	149	+ const rows = await db.execute<Record<string, unknown>>(sql`
	150	+ select n.id, n.alert_id, a.name as alert_name, n.event_id, n.channel, n.status, n.created_at, n.read_at, n.delivered_at, n.error,
	151	+ json_build_object('id', e.id, 'slug', e.slug, 'title', e.title, 'importance', e.importance, 'signal_score', e.signal_score, 'event_type', e.event_type, 'silent_change', e.silent_change, 'detected_at', e.detected_at, 'source', json_build_object('id', s.id, 'name', s.name)) as event
	152	+ from notifications n join alerts a on a.id = n.alert_id join events e on e.id = n.event_id join sources s on s.id = e.source_id
	153	+ where a.owner_token = ${owner} ${req.query.unread === "1" ? sql`and n.read_at is null` : sql``} order by n.created_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
	154	+ const unread = (await db.execute<{ n: string }>(sql`select count(*)::text as n from notifications n join alerts a on a.id = n.alert_id where a.owner_token = ${owner} and n.read_at is null`)).rows[0]?.n;
	155	+ return { items: rows.rows, unread: Number(unread ?? 0) };
	156	+ });
	157	+ app.post("/api/v1/notifications/read", async (req, reply) => {
	158	+ const owner = requireOwner(req, reply);
	159	+ if (!owner) return;
	160	+ const body = z.object({ ids: z.array(z.number().int()).max(500).optional() }).parse(req.body ?? {});
	161	+ if (body.ids?.length) await db.execute(sql`update notifications n set read_at = now() from alerts a where a.id = n.alert_id and a.owner_token = ${owner} and n.id = any(${sql.raw("array[" + body.ids.map((i) => Number(i)).join(",") + "]::bigint[]")})`);
	162	+ else await db.execute(sql`update notifications n set read_at = now() from alerts a where a.id = n.alert_id and a.owner_token = ${owner} and n.read_at is null`);
	163	+ return { ok: true };
	164	+ });
	165	+
	166	+ // ---- Bookmarks ---------------------------------------------------------------------------------
	167	+ app.get<{ Querystring: { limit?: string } }>("/api/v1/bookmarks", async (req, reply) => {
	168	+ const owner = requireOwner(req, reply);
	169	+ if (!owner) return;
	170	+ const rows = await db.execute<Record<string, unknown>>(sql`select b.created_at as bookmarked_at, b.note, x.* from bookmarks b join lateral (select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.id = b.event_id) x on true where b.owner_token = ${owner} order by b.created_at desc limit ${Math.min(500, Number(req.query.limit ?? 100))}`);
	171	+ return { items: rows.rows };
	172	+ });
	173	+ app.post("/api/v1/bookmarks", async (req, reply) => {
	174	+ const owner = requireOwner(req, reply);
	175	+ if (!owner) return;
	176	+ const body = z.object({ event_id: z.string().min(4).max(80), note: z.string().max(500).optional() }).parse(req.body ?? {});
	177	+ const ev = (await db.execute<{ id: string }>(sql`select id from events where id = ${body.event_id} or slug = ${body.event_id}`)).rows[0];
	178	+ if (!ev) return reply.status(404).send({ error: "event_not_found" });
	179	+ const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from bookmarks where owner_token = ${owner}`)).rows[0]?.n;
	180	+ if (Number(count) >= 2000) return reply.status(429).send({ error: "too_many_bookmarks" });
	181	+ await db.execute(sql`insert into bookmarks (owner_token, event_id, note) values (${owner}, ${ev.id}, ${body.note ?? null}) on conflict (owner_token, event_id) do update set note = coalesce(excluded.note, bookmarks.note)`);
	182	+ return { ok: true, event_id: ev.id };
	183	+ });
	184	+ app.delete<{ Params: { id: string } }>("/api/v1/bookmarks/:id", async (req, reply) => {
	185	+ const owner = requireOwner(req, reply);
	186	+ if (!owner) return;
	187	+ await db.execute(sql`delete from bookmarks where owner_token = ${owner} and (event_id = ${req.params.id} or event_id = (select id from events where slug = ${req.params.id}))`);
	188	+ return { ok: true };
	189	+ });
	190	+ app.get("/api/v1/bookmarks/ids", async (req, reply) => {
	191	+ const owner = requireOwner(req, reply);
	192	+ if (!owner) return;
	193	+ const rows = await db.execute<{ event_id: string }>(sql`select event_id from bookmarks where owner_token = ${owner}`);
	194	+ return { ids: rows.rows.map((r) => r.event_id) };
	195	+ });
	196	+
	197	+ // ---- Saved views -------------------------------------------------------------------------------
	198	+ app.get("/api/v1/views", async (req, reply) => {
	199	+ const owner = requireOwner(req, reply);
	200	+ if (!owner) return;
	201	+ return { items: (await db.execute<Record<string, unknown>>(sql`select id, name, query, created_at from saved_views where owner_token = ${owner} order by created_at`)).rows };
	202	+ });
	203	+ app.post("/api/v1/views", async (req, reply) => {
	204	+ const owner = requireOwner(req, reply);
	205	+ if (!owner) return;
	206	+ const body = z.object({ name: z.string().min(1).max(60), query: z.string().min(1).max(600) }).parse(req.body ?? {});
	207	+ const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from saved_views where owner_token = ${owner}`)).rows[0]?.n;
	208	+ if (Number(count) >= 50) return reply.status(429).send({ error: "too_many_views" });
	209	+ const id = newId("view");
	210	+ await db.execute(sql`insert into saved_views (id, owner_token, name, query) values (${id}, ${owner}, ${body.name}, ${body.query})`);
	211	+ return { id, ...body };
	212	+ });
	213	+ app.delete<{ Params: { id: string } }>("/api/v1/views/:id", async (req, reply) => {
	214	+ const owner = requireOwner(req, reply);
	215	+ if (!owner) return;
	216	+ await db.execute(sql`delete from saved_views where id = ${req.params.id} and owner_token = ${owner}`);
	217	+ return { ok: true };
	218	+ });
	219	+
	220	+ // ---- Custom URL monitors (spec §45, §87) -----------------------------------------------------------
	221	+ const monitorBody = z.object({
	222	+ url: z.string().url().max(2000),
	223	+ name: z.string().min(1).max(80).optional(),
	224	+ /** hourly \| daily (maps to tier C / D) */
	225	+ frequency: z.enum(["hourly", "daily"]).default("hourly"),
	226	+ /** low = only big changes, normal, high = every meaningful change */
	227	+ sensitivity: z.enum(["low", "normal", "high"]).default("normal"),
	228	+ selector: z.string().max(200).optional(),
	229	+ keywords: z.array(z.string().min(1).max(60)).max(20).optional(),
	230	+ });
	231	+ const ownerSourceId = (owner: string): string => `custom-${createHash("sha256").update(owner).digest("hex").slice(0, 12)}`;
	232	+
	233	+ app.get("/api/v1/monitors", async (req, reply) => {
	234	+ const owner = requireOwner(req, reply);
	235	+ if (!owner) return;
	236	+ const sid = ownerSourceId(owner);
	237	+ const rows = await db.execute<Record<string, unknown>>(sql`select id, name, url, tier, health, status, enabled, config, next_check_at, last_check_at, last_change_at, last_event_at, last_status, last_error, total_runs, raw_changes, meaningful_changes, created_at from sensors where source_id = ${sid} order by created_at`);
	238	+ return { items: rows.rows, limit: config.monitorsPerOwner, source_id: sid };
	239	+ });
	240	+ app.post("/api/v1/monitors", async (req, reply) => {
	241	+ const owner = requireOwner(req, reply);
	242	+ if (!owner) return;
	243	+ const body = monitorBody.parse(req.body ?? {});
	244	+ let host: string;
	245	+ try {
	246	+ const u = await assertUrlAllowed(body.url);
	247	+ if (!["http:", "https:"].includes(u.url.protocol)) return reply.status(400).send({ error: "scheme_not_allowed" });
	248	+ host = u.url.hostname;
	249	+ } catch (e) {
	250	+ return reply.status(400).send({ error: "url_rejected", detail: e instanceof UrlPolicyError ? e.message : "URL not allowed" });
	251	+ }
	252	+ const sid = ownerSourceId(owner);
	253	+ const n = Number((await db.execute<{ n: string }>(sql`select count(*)::text as n from sensors where source_id = ${sid} and enabled`)).rows[0]?.n ?? 0);
	254	+ if (n >= config.monitorsPerOwner) return reply.status(429).send({ error: "monitor_limit_reached", limit: config.monitorsPerOwner });
	255	+ // Sensor auto-test (spec §72): fetch + normalize before activation; baseline is taken by the engine.
	256	+ const sensorId = `${sid}_${newId("m").slice(2)}`;
	257	+ const cfg: Record<string, unknown> = { custom: true, sensitivity: body.sensitivity, ...(body.selector ? { selector: body.selector } : {}), ...(body.keywords?.length ? { keywords: body.keywords } : {}) };
	258	+ const tier = body.frequency === "hourly" ? "C" : "D";
	259	+ const connector = getConnector("http");
	260	+ const endpoint = { id: sensorId, sourceId: sid, name: body.name ?? host, url: body.url, type: "HTML" as const, tier: tier as "C" \| "D", connector: "http", config: cfg, etag: null, lastModified: null, state: null };
	261	+ const obs = await connector.fetch(endpoint);
	262	+ if (obs.error) return reply.status(422).send({ error: "fetch_failed", detail: `${obs.error.code}: ${obs.error.message}` });
	263	+ if (obs.meta.status >= 400) return reply.status(422).send({ error: "http_error", status: obs.meta.status });
	264	+ let norm;
	265	+ try {
	266	+ norm = await connector.normalize(endpoint, obs);
	267	+ } catch (e) {
	268	+ return reply.status(422).send({ error: "unparseable", detail: (e as Error).message });
	269	+ }
	270	+ if ((norm.text ?? "").length < 40 && norm.mode === "text") return reply.status(422).send({ error: "thin_content", detail: "The page has almost no server-rendered text (client-side app?)." });
	271	+ await db.execute(sql`insert into sources (id, name, domain, homepage, description, categories, tier, importance_weight, enabled, llm_enabled, kind, owner_token, first_party, notes)
	272	+ values (${sid}, ${"Custom monitors"}, ${"custom.websensor.io"}, ${null}, ${"Private URL monitors"}, ${"{custom}"}::text[], 'C', 0.5, true, false, 'custom', ${owner}, true, 'owner-scoped custom monitors') on conflict (id) do nothing`);
	273	+ await db.execute(sql`insert into sensors (id, source_id, name, url, type, connector, tier, importance_weight, config, priority, status, validated_at, enabled, next_check_at)
	274	+ values (${sensorId}, ${sid}, ${body.name ?? host}, ${body.url}, 'HTML', 'http', ${tier}, ${body.sensitivity === "high" ? 1.3 : body.sensitivity === "low" ? 0.7 : 1}, ${JSON.stringify(cfg)}::jsonb, 3, 'VALIDATED', now(), true, now())`);
	275	+ return { id: sensorId, source_id: sid, url: body.url, tier, status: "VALIDATED", test: { http_status: obs.meta.status, content_type: obs.meta.contentType, bytes: obs.meta.contentLength, title: norm.title ?? null, mode: norm.mode, extraction_confidence: norm.extractionConfidence } };
	276	+ });
	277	+ app.delete<{ Params: { id: string } }>("/api/v1/monitors/:id", async (req, reply) => {
	278	+ const owner = requireOwner(req, reply);
	279	+ if (!owner) return;
	280	+ const sid = ownerSourceId(owner);
	281	+ await db.execute(sql`delete from sensors where id = ${req.params.id} and source_id = ${sid}`);
	282	+ return { ok: true };
	283	+ });
	284	+ app.get<{ Params: { id: string }; Querystring: { limit?: string } }>("/api/v1/monitors/:id/events", async (req, reply) => {
	285	+ const owner = requireOwner(req, reply);
	286	+ if (!owner) return;
	287	+ const sid = ownerSourceId(owner);
	288	+ const sensor = (await db.execute<{ id: string }>(sql`select id from sensors where id = ${req.params.id} and source_id = ${sid}`)).rows[0];
	289	+ if (!sensor) return reply.status(404).send({ error: "not_found" });
	290	+ const [events, changes] = await Promise.all([
	291	+ listEvents({ sensor: sensor.id, limit: Math.min(100, Number(req.query.limit ?? 30)), includeCustom: true }),
	292	+ db.execute<Record<string, unknown>>(sql`select id, detected_at, kind, signal, change_class, field_changes, meaningful, event_id, old_snapshot_id, new_snapshot_id from changes where sensor_id = ${sensor.id} order by detected_at desc limit 30`).then((r) => r.rows),
	293	+ ]);
	294	+ return { events: events.items, changes };
	295	+ });
	296	+
	297	+ // keep eventConditions referenced for owner-scoped searches later
	298	+ void eventConditions;
	299	+}

modified apps/api/src/routes.ts +226 −173

		`@@ -1,13 +1,20 @@`
1	1	import type { FastifyInstance } from "fastify";
2	2	import { z } from "zod";
3		−import { diffText, EVENT_TYPES, FEED_CHANNELS, newId } from "@websensor/core";
	3	+import { COUNTRIES, countryBySlug, diffText, EVENT_GROUPS, EVENT_TYPES, FEED_CHANNELS, parseSearch } from "@websensor/core";
4	4	import { db, sql, textArray } from "@websensor/db";
5	5	import { getBlobStore } from "@websensor/store";
	6	+import { cached } from "./cache";
6	7	import { config } from "./config";
7		−import { liveStats } from "./live";
8		−import { getEvent, listEvents, relatedEvents, sourceActivity, stats, trending, EVENT_SELECT } from "./queries";
	8	+import { breakingDesk, categoryDesk, clusterDetail, countryDesk, countryList, entityInsights, entityRankings, pulse, radar, sourceSensors } from "./intel";
	9	+import { engineStatus, liveStats } from "./live";
	10	+import { countEvents, getEvent, listEvents, relatedEvents, sensorHistory, sourceActivity, sourceQuality, stats, trending } from "./queries";
9	11
10		−const eventsQuery = z.object({
	12	+const bool = z
	13	+ .enum(["true", "false", "1", "0"])
	14	+ .transform((v) => v === "true" \|\| v === "1")
	15	+ .optional();
	16	+
	17	+export const eventsQuery = z.object({
11	18	after: z.string().optional(),
12	19	before: z.string().optional(),
13	20	category: z.string().optional(),
		`@@ -18,56 +25,66 @@ const eventsQuery = z.object({`
18	25	cluster: z.string().optional(),
19	26	importance_min: z.coerce.number().min(0).max(100).optional(),
20	27	confidence_min: z.coerce.number().min(0).max(100).optional(),
	28	+ signal_min: z.coerce.number().min(0).max(100).optional(),
21	29	event_type: z.string().optional(),
22		− silent_change: z
23		− .enum(["true", "false"])
24		− .transform((v) => v === "true")
25		− .optional(),
26		− q: z.string().max(200).optional(),
	30	+ group: z.string().optional(),
	31	+ silent_change: bool,
	32	+ first_party: bool,
	33	+ confirmed: bool,
	34	+ country: z.string().max(3).optional(),
	35	+ language: z.string().max(2).optional(),
	36	+ change_class: z.string().optional(),
	37	+ q: z.string().max(300).optional(),
27	38	limit: z.coerce.number().int().min(1).max(200).default(50),
28	39	cursor: z.string().optional(),
29		− order: z.enum(["recent", "importance"]).default("recent"),
	40	+ order: z.enum(["recent", "importance", "signal"]).default("recent"),
30	41	});
31	42
32		−function ownerToken(headers: Record<string, unknown>): string \| null {
33		− const t = headers["x-websensor-owner"];
34		− return typeof t === "string" && /^[A-Za-z0-9_-]{16,80}$/.test(t) ? t : null;
35		−}
36		−
37	43	export async function registerRoutes(app: FastifyInstance): Promise<void> {
38	44	// ---- Health ---------------------------------------------------------------------------
39	45	app.get("/api/health", async () => ({ status: "ok", service: "api", version: config.version, time: new Date().toISOString() }));
40	46	app.get("/api/ready", async (_req, reply) => {
41		− const checks: Record<string, boolean \| number> = {};
	47	+ const checks: Record<string, boolean \| number \| string \| null> = {};
	48	+ const t0 = Date.now();
42	49	try {
43	50	await db.execute(sql`select 1`);
44	51	checks.database = true;
	52	+ checks.database_latency_ms = Date.now() - t0;
45	53	} catch {
46	54	checks.database = false;
47	55	}
48		− const engine = await db.execute<{ last: Date \| null }>(sql`select max(started_at) as last from sensor_runs where started_at >= now() - interval '15 minutes'`);
	56	+ const engine = await db.execute<{ last: Date \| null }>(sql`select max(started_at) as last from sensor_runs where started_at >= now() - interval '15 minutes'`).catch(() => ({ rows: [{ last: null }] }));
49	57	checks.engine_recent = Boolean(engine.rows[0]?.last);
	58	+ const es = await engineStatus();
	59	+ checks.engine_heartbeat = es ? String(es.at) : null;
50	60	checks.ws_clients = liveStats().clients;
51	61	return reply.status(checks.database ? 200 : 503).send({ status: checks.database ? "ready" : "degraded", checks });
52	62	});
53	63
54	64	// ---- Events ---------------------------------------------------------------------------
55		− app.get("/api/v1/events", async (req) => {
	65	+ app.get("/api/v1/events", async (req, reply) => {
	66	+ const q = eventsQuery.parse(req.query);
	67	+ const page = await listEvents(q);
	68	+ reply.header("x-websensor-order", q.order);
	69	+ return { ...page, meta: { limit: q.limit, order: q.order, filters: stripEmpty({ ...q, limit: undefined, cursor: undefined, order: undefined }) } };
	70	+ });
	71	+ app.get("/api/v1/events/count", async (req) => {
56	72	const q = eventsQuery.parse(req.query);
57		− return listEvents(q);
	73	+ return { count: await countEvents(q) };
58	74	});
59	75	app.get<{ Params: { id: string } }>("/api/v1/events/:id", async (req, reply) => {
60	76	const ev = await getEvent(req.params.id);
61	77	if (!ev) return reply.status(404).send({ error: "not_found" });
62		− const [related, cluster, change, interp] = await Promise.all([
	78	+ const [related, cluster, change, interp, history] = await Promise.all([
63	79	relatedEvents(ev),
64	80	ev.cluster_id ? db.execute<Record<string, unknown>>(sql`select * from event_clusters where id = ${String(ev.cluster_id)}`).then((r) => r.rows[0] ?? null) : null,
65		− ev.change_id ? db.execute<Record<string, unknown>>(sql`select id, kind, diff, signal, noise_ratio, magnitude, heuristic, detected_at, old_snapshot_id, new_snapshot_id from changes where id = ${String(ev.change_id)}`).then((r) => r.rows[0] ?? null) : null,
	81	+ ev.change_id ? db.execute<Record<string, unknown>>(sql`select id, kind, diff, signal, noise_ratio, magnitude, heuristic, detected_at, old_snapshot_id, new_snapshot_id, change_class, field_changes from changes where id = ${String(ev.change_id)}`).then((r) => r.rows[0] ?? null) : null,
66	82	db.execute<Record<string, unknown>>(sql`select version, model, created_at from interpretations where event_id = ${String(ev.id)} order by version`).then((r) => r.rows),
	83	+ sensorHistory(String(ev.sensor_id), String(ev.id)),
67	84	]);
68		− const snaps = await db.execute<Record<string, unknown>>(sql`select id, url, captured_at, http_status, content_type, content_length, content_hash, canonical_hash, etag, last_modified, title, mode, fetch_duration_ms, extraction_confidence from snapshots where id in (${String(ev.old_snapshot_id ?? "")}, ${String(ev.new_snapshot_id ?? "")})`);
	85	+ const snaps = await db.execute<Record<string, unknown>>(sql`select id, url, captured_at, http_status, content_type, content_length, content_hash, canonical_hash, etag, last_modified, title, mode, fetch_duration_ms, extraction_confidence, storage_key is not null as has_raw from snapshots where id in (${String(ev.old_snapshot_id ?? "")}, ${String(ev.new_snapshot_id ?? "")})`);
69	86	const sourceRel = await db.execute<Record<string, unknown>>(sql`select health, success_rate, avg_latency_ms, total_runs, raw_changes, meaningful_changes, last_check_at from (select s.health, ch.success_rate, s.avg_latency_ms, s.total_runs, s.raw_changes, s.meaningful_changes, s.last_check_at from sensors s left join connector_health ch on ch.connector = s.connector where s.id = ${String(ev.sensor_id)}) x`);
70		− return { event: ev, related, cluster, change, interpretations: interp, snapshots: snaps.rows, sensor_reliability: sourceRel.rows[0] ?? null };
	87	+ return { event: ev, related, cluster, change, interpretations: interp, snapshots: snaps.rows, sensor_reliability: sourceRel.rows[0] ?? null, history };
71	88	});
72	89
73	90	// ---- Changes / snapshots / diffs -------------------------------------------------------
		`@@ -84,7 +101,8 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {`
84	101	const s = r.rows[0];
85	102	if (!s) return reply.status(404).send({ error: "not_found" });
86	103	const store = getBlobStore();
87		− if (req.query.raw === "1" && s.storage_key) {
	104	+ if (req.query.raw === "1") {
	105	+ if (!s.storage_key) return reply.status(410).send({ error: "raw_body_pruned", detail: "The raw body of this snapshot was pruned by the retention policy; the canonical representation and hashes are preserved." });
88	106	const buf = await store.get(String(s.storage_key));
89	107	reply.header("content-type", String(s.content_type ?? "text/plain") + (String(s.content_type ?? "").includes("charset") ? "" : "; charset=utf-8"));
90	108	reply.header("x-content-type-options", "nosniff");
		`@@ -108,76 +126,111 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {`
108	126	const d = diffText(ta, tb, `${a}@${String(sa.captured_at)}`, `${b}@${String(sb.captured_at)}`);
109	127	return { a: sa, b: sb, before: ta, after: tb, diff: { unified: d.unified, stats: d.stats, added: d.added, removed: d.removed, modified: d.modified } };
110	128	});
	129	+ /** Historical memory (spec §79): how a sensor's page looked at any point — list of snapshots with day grouping. */
	130	+ app.get<{ Params: { id: string }; Querystring: { limit?: string; before?: string } }>("/api/v1/sensors/:id/snapshots", async (req) => {
	131	+ const lim = Math.min(500, Number(req.query.limit ?? 200));
	132	+ const rows = await db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_length, canonical_hash, title, mode, storage_key is not null as has_raw, (select count(*) from changes c where c.new_snapshot_id = snapshots.id) > 0 as has_change, (select e.slug from events e where e.new_snapshot_id = snapshots.id limit 1) as event_slug from snapshots where sensor_id = ${req.params.id} ${req.query.before ? sql`and captured_at < ${new Date(req.query.before)}` : sql``} order by captured_at desc limit ${lim}`);
	133	+ return { items: rows.rows };
	134	+ });
111	135
112	136	// ---- Sources & sensors ------------------------------------------------------------------
113		− app.get<{ Querystring: { category?: string; q?: string; limit?: string } }>("/api/v1/sources", async (req) => {
	137	+ app.get<{ Querystring: { category?: string; q?: string; limit?: string; country?: string; tier?: string; first_party?: string } }>("/api/v1/sources", async (req) => {
114	138	const cat = req.query.category;
115	139	const q = req.query.q;
116	140	const rows = await db.execute<Record<string, unknown>>(sql`
117		− select s., (select count() from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count,
	141	+ select s.id, s.name, s.domain, s.homepage, s.description, s.categories, s.tier, s.importance_weight, s.enabled, s.notes, s.first_party, s.country, s.language, s.robots_checked_at,
	142	+ (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count,
118	143	(select count(*) from events e where e.source_id = s.id)::int as event_count,
119	144	(select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h,
120	145	(select max(detected_at) from events e where e.source_id = s.id) as last_event_at,
121	146	(select max(last_check_at) from sensors x where x.source_id = s.id) as last_check_at,
122	147	(select count(*) from sensors x where x.source_id = s.id and x.enabled and x.health <> 'UP')::int as sensors_degraded
123		− from sources s where s.enabled ${cat ? sql`and ${cat} = any(s.categories)` : sql``} ${q ? sql`and (s.name ilike ${"%" + q + "%"} or s.domain ilike ${"%" + q + "%"})` : sql``}
124		− order by s.tier asc, events_24h desc, s.name asc limit ${Math.min(500, Number(req.query.limit ?? 300))}`);
	148	+ from sources s where s.enabled and s.kind = 'registry' ${cat ? sql`and ${cat} = any(s.categories)` : sql``} ${q ? sql`and (s.name ilike ${"%" + q + "%"} or s.domain ilike ${"%" + q + "%"})` : sql``} ${req.query.country ? sql`and s.country = ${req.query.country.toUpperCase()}` : sql``} ${req.query.tier ? sql`and s.tier = ${req.query.tier.toUpperCase()}` : sql``} ${req.query.first_party ? sql`and s.first_party = ${req.query.first_party === "true"}` : sql``}
	149	+ order by s.tier asc, events_24h desc, s.name asc limit ${Math.min(5000, Number(req.query.limit ?? 300))}`);
125	150	return { items: rows.rows };
126	151	});
127	152	app.get<{ Params: { id: string } }>("/api/v1/sources/:id", async (req, reply) => {
128		− const s = (await db.execute<Record<string, unknown>>(sql`select * from sources where id = ${req.params.id} or domain = ${req.params.id} limit 1`)).rows[0];
	153	+ const s = (await db.execute<Record<string, unknown>>(sql`select * from sources where (id = ${req.params.id} or domain = ${req.params.id}) and kind = 'registry' limit 1`)).rows[0];
129	154	if (!s) return reply.status(404).send({ error: "not_found" });
130		− const [sensors, ents, activity, candidates] = await Promise.all([
131		− db.execute<Record<string, unknown>>(sql`select id, name, url, type, connector, tier, health, enabled, next_check_at, last_check_at, last_change_at, last_event_at, last_status, last_error, consecutive_errors, total_runs, total_not_modified, raw_changes, meaningful_changes, avg_latency_ms, base_interval_seconds, etag is not null as has_etag, last_modified is not null as has_last_modified from sensors where source_id = ${String(s.id)} order by tier, name`).then((r) => r.rows),
	155	+ const { owner_token: _o, ...pub } = s;
	156	+ const [sensors, ents, activity, quality, candidates, byType, series] = await Promise.all([
	157	+ sourceSensors(String(s.id)),
132	158	db.execute<Record<string, unknown>>(sql`select en.id, en.name, en.type, en.importance, en.event_count from source_entities se join entities en on en.id = se.entity_id where se.source_id = ${String(s.id)} order by en.type, en.name`).then((r) => r.rows),
133	159	sourceActivity(String(s.id)),
	160	+ sourceQuality(String(s.id)),
134	161	db.execute<Record<string, unknown>>(sql`select url, kind, evidence, score, status, found_at from discovery_candidates where source_id = ${String(s.id)} order by (score->>'value')::float desc nulls last limit 50`).then((r) => r.rows),
	162	+ db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where source_id = ${String(s.id)} group by 1 order by 2 desc limit 16`).then((r) => r.rows),
	163	+ db.execute<Record<string, unknown>>(sql`select day::text as day, checks, not_modified, errors, raw_changes, events from source_daily where source_id = ${String(s.id)} and day >= (now() at time zone 'UTC')::date - 30 order by day`).then((r) => r.rows),
135	164	]);
136		− return { source: s, sensors, entities: ents, activity, discovery: candidates };
	165	+ return { source: pub, sensors, entities: ents, activity, quality, discovery: candidates, by_type: byType, daily: series };
137	166	});
138	167	app.get<{ Params: { id: string } }>("/api/v1/sensors/:id", async (req, reply) => {
139		− const s = (await db.execute<Record<string, unknown>>(sql`select s.*, so.name as source_name, so.domain from sensors s join sources so on so.id = s.source_id where s.id = ${req.params.id}`)).rows[0];
140		− if (!s) return reply.status(404).send({ error: "not_found" });
141		− const [runs, snaps, changes] = await Promise.all([
	168	+ const s = (await db.execute<Record<string, unknown>>(sql`select s.*, so.name as source_name, so.domain, so.kind as source_kind from sensors s join sources so on so.id = s.source_id where s.id = ${req.params.id}`)).rows[0];
	169	+ if (!s \|\| s.source_kind === "custom") return reply.status(404).send({ error: "not_found" });
	170	+ const [runs, snaps, changes, events] = await Promise.all([
142	171	db.execute<Record<string, unknown>>(sql`select id, started_at, finished_at, http_status, outcome, error, duration_ms, bytes, fetch_method, snapshot_id from sensor_runs where sensor_id = ${String(s.id)} order by started_at desc limit 50`).then((r) => r.rows),
143		− db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_type, content_length, canonical_hash, title, mode, extraction_confidence from snapshots where sensor_id = ${String(s.id)} order by captured_at desc limit 50`).then((r) => r.rows),
144		− db.execute<Record<string, unknown>>(sql`select id, detected_at, kind, signal, noise_ratio, magnitude, meaningful, event_id, old_snapshot_id, new_snapshot_id, heuristic->>'eventType' as heuristic_type from changes where sensor_id = ${String(s.id)} order by detected_at desc limit 50`).then((r) => r.rows),
	172	+ db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_type, content_length, canonical_hash, title, mode, extraction_confidence, storage_key is not null as has_raw from snapshots where sensor_id = ${String(s.id)} order by captured_at desc limit 50`).then((r) => r.rows),
	173	+ db.execute<Record<string, unknown>>(sql`select id, detected_at, kind, signal, noise_ratio, magnitude, meaningful, event_id, old_snapshot_id, new_snapshot_id, change_class, field_changes, heuristic->>'eventType' as heuristic_type from changes where sensor_id = ${String(s.id)} order by detected_at desc limit 50`).then((r) => r.rows),
	174	+ listEvents({ sensor: String(s.id), limit: 30 }).then((r) => r.items),
145	175	]);
146		− const { etag: _e, last_modified: _lm, state: _st, ...pub } = s as Record<string, unknown>;
147		− return { sensor: pub, runs, snapshots: snaps, changes };
	176	+ const { state: _st, source_kind: _k, ...pub } = s as Record<string, unknown>;
	177	+ const runs24 = await db.execute<Record<string, unknown>>(sql`select count()::int as checks_24h, count() filter (where outcome = 'not_modified')::int as not_modified_24h, count(*) filter (where outcome in ('error','parse_error','rate_limited'))::int as errors_24h, avg(duration_ms)::int as avg_ms_24h from sensor_runs where sensor_id = ${String(s.id)} and started_at >= now() - interval '24 hours'`);
	178	+ return { sensor: { ...pub, ...runs24.rows[0] }, runs, snapshots: snaps, changes, events };
148	179	});
149	180
150	181	// ---- Entities ----------------------------------------------------------------------------
151		− app.get<{ Querystring: { type?: string; q?: string; limit?: string } }>("/api/v1/entities", async (req) => {
	182	+ app.get<{ Querystring: { type?: string; q?: string; limit?: string; category?: string } }>("/api/v1/entities", async (req) => {
152	183	const rows = await db.execute<Record<string, unknown>>(sql`
153	184	select en., (select count() from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = en.id and e.detected_at >= now() - interval '24 hours')::int as events_24h
154		− from entities en where true ${req.query.type ? sql`and en.type = ${req.query.type}` : sql``} ${req.query.q ? sql`and (en.search @@ plainto_tsquery('simple', ${req.query.q}) or en.name ilike ${"%" + req.query.q + "%"})` : sql``}
155		− order by en.event_count desc, en.importance desc, en.name limit ${Math.min(500, Number(req.query.limit ?? 200))}`);
	185	+ from entities en where true ${req.query.type ? sql`and en.type = ${req.query.type}` : sql``} ${req.query.category ? sql`and ${req.query.category} = any(en.categories)` : sql``} ${req.query.q ? sql`and (en.search @@ plainto_tsquery('simple', ${req.query.q}) or en.name ilike ${"%" + req.query.q + "%"})` : sql``}
	186	+ order by en.event_count desc, en.importance desc, en.name limit ${Math.min(1000, Number(req.query.limit ?? 200))}`);
156	187	return { items: rows.rows };
157	188	});
	189	+ app.get<{ Querystring: { limit?: string } }>("/api/v1/entities/rank", async (req) => ({ items: await entityRankings(Math.min(500, Number(req.query.limit ?? 100))) }));
158	190	app.get<{ Params: { id: string } }>("/api/v1/entities/:id", async (req, reply) => {
159	191	const e = (await db.execute<Record<string, unknown>>(sql`select * from entities where id = ${req.params.id} or id = ${"org_" + req.params.id} limit 1`)).rows[0];
160	192	if (!e) return reply.status(404).send({ error: "not_found" });
161	193	const id = String(e.id);
162		− const [children, relations, sources, aliases, recent, byType] = await Promise.all([
	194	+ const [children, relations, sources, aliases, recent, byType, insights, silent, parent] = await Promise.all([
163	195	db.execute<Record<string, unknown>>(sql`select id, name, type, importance, event_count, last_event_at from entities where parent_id = ${id} order by event_count desc, name`).then((r) => r.rows),
164	196	db.execute<Record<string, unknown>>(sql`select r.relation, r.from_id, r.to_id, f.name as from_name, t.name as to_name, t.type as to_type from entity_relations r join entities f on f.id = r.from_id join entities t on t.id = r.to_id where r.from_id = ${id} or r.to_id = ${id} limit 100`).then((r) => r.rows),
165		− db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier from source_entities se join sources s on s.id = se.source_id where se.entity_id = ${id}`).then((r) => r.rows),
	197	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier, s.first_party, s.country, (select count() from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count, (select count() from events ev where ev.source_id = s.id and ev.detected_at >= now() - interval '24 hours')::int as events_24h from source_entities se join sources s on s.id = se.source_id where se.entity_id = ${id} and s.kind = 'registry'`).then((r) => r.rows),
166	198	db.execute<{ alias: string }>(sql`select alias from entity_aliases where entity_id = ${id} order by alias`).then((r) => r.rows.map((x) => x.alias)),
167	199	listEvents({ entity: id, limit: 30 }),
168	200	db.execute<Record<string, unknown>>(sql`select e.event_type, count(*)::int as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${id} group by 1 order by 2 desc`).then((r) => r.rows),
	201	+ entityInsights(id),
	202	+ listEvents({ entity: id, silent_change: true, limit: 10 }).then((r) => r.items),
	203	+ e.parent_id ? db.execute<Record<string, unknown>>(sql`select id, name, type from entities where id = ${String(e.parent_id)}`).then((r) => r.rows[0] ?? null) : null,
169	204	]);
170		− return { entity: e, children, relations, sources, aliases, recent: recent.items, by_type: byType };
	205	+ // Related entities: co-occurring in events over 30 days
	206	+ const related = await db.execute<Record<string, unknown>>(sql`
	207	+ select en.id, en.name, en.type, count(*)::int as shared_events from event_entities a join event_entities b on a.event_id = b.event_id and b.entity_id <> a.entity_id join events e on e.id = a.event_id join entities en on en.id = b.entity_id
	208	+ where a.entity_id = ${id} and e.detected_at >= now() - interval '30 days' group by en.id, en.name, en.type order by shared_events desc limit 12`).then((r) => r.rows);
	209	+ return { entity: e, parent, children, relations, sources, aliases, recent: recent.items, nextCursor: recent.nextCursor, by_type: byType, insights, silent, related };
171	210	});
172		− app.get<{ Params: { id: string }; Querystring: { limit?: string; cursor?: string } }>("/api/v1/entities/:id/timeline", async (req) => {
	211	+ app.get<{ Params: { id: string }; Querystring: { limit?: string; cursor?: string; after?: string; before?: string; event_type?: string; silent_change?: string } }>("/api/v1/entities/:id/timeline", async (req) => {
173	212	const id = req.params.id.startsWith("org_") \|\| req.params.id.startsWith("prd_") ? req.params.id : `org_${req.params.id}`;
174		− return listEvents({ entity: id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor });
	213	+ return listEvents({ entity: id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor, after: req.query.after, before: req.query.before, event_type: req.query.event_type, silent_change: req.query.silent_change === "true" ? true : undefined });
	214	+ });
	215	+
	216	+ // ---- Clusters ------------------------------------------------------------------------------
	217	+ app.get<{ Querystring: { limit?: string; since?: string; state?: string; min_events?: string } }>("/api/v1/clusters", async (req) => {
	218	+ const rows = await db.execute<Record<string, unknown>>(sql`
	219	+ select c.*, (select json_agg(json_build_object('id', e.id, 'slug', e.slug, 'title', e.title, 'importance', e.importance, 'event_type', e.event_type, 'detected_at', e.detected_at, 'source_id', e.source_id, 'url', e.url, 'first_party', e.first_party) order by e.importance desc) from events e where e.cluster_id = c.id) as events,
	220	+ (select json_build_object('id', s.id, 'name', s.name, 'domain', s.domain) from events e join sources s on s.id = e.source_id where e.id = c.primary_event_id) as source
	221	+ from event_clusters c where c.last_at >= now() - make_interval(hours => ${Math.min(720, Number(req.query.since ?? 72))}) ${req.query.state ? sql`and c.state = ${req.query.state}` : sql``} ${req.query.min_events ? sql`and c.event_count >= ${Number(req.query.min_events)}` : sql``} order by c.last_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
	222	+ return { items: rows.rows };
	223	+ });
	224	+ app.get<{ Params: { id: string } }>("/api/v1/clusters/:id", async (req, reply) => {
	225	+ const d = await clusterDetail(req.params.id);
	226	+ if (!d) return reply.status(404).send({ error: "not_found" });
	227	+ return d;
175	228	});
176	229
177	230	// ---- Domains / URLs ----------------------------------------------------------------------
178	231	app.get<{ Params: { domain: string }; Querystring: { limit?: string; cursor?: string } }>("/api/v1/domains/:domain/timeline", async (req) => {
179	232	const d = req.params.domain.toLowerCase();
180		− const src = (await db.execute<{ id: string }>(sql`select id from sources where domain = ${d} or domain = ${"www." + d} or ${d} = 'www.' \|\| domain limit 1`)).rows[0];
	233	+ const src = (await db.execute<{ id: string }>(sql`select id from sources where (domain = ${d} or domain = ${"www." + d} or ${d} = 'www.' \|\| domain) and kind = 'registry' limit 1`)).rows[0];
181	234	const urls = await db.execute<Record<string, unknown>>(sql`select url, status, first_seen_at, last_seen_at, snapshot_count, change_count, sensor_id from urls where domain = ${d} or domain like ${"%." + d} order by change_count desc, last_seen_at desc limit 200`);
182	235	const ev = src ? await listEvents({ source: src.id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor }) : { items: [], nextCursor: null };
183	236	return { domain: d, source_id: src?.id ?? null, urls: urls.rows, events: ev.items, nextCursor: ev.nextCursor };
		`@@ -186,149 +239,103 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {`
186	239	if (!req.query.url) return reply.status(400).send({ error: "url required" });
187	240	const u = (await db.execute<Record<string, unknown>>(sql`select * from urls where url = ${req.query.url}`)).rows[0];
188	241	const history = await db.execute<Record<string, unknown>>(sql`
189		− select h.id, h.at, h.kind, h.snapshot_id, h.change_id, h.event_id, h.note, e.title as event_title, e.importance, e.event_type, e.slug as event_slug, c.kind as change_kind, c.signal
	242	+ select h.id, h.at, h.kind, h.snapshot_id, h.change_id, h.event_id, h.note, e.title as event_title, e.importance, e.event_type, e.slug as event_slug, c.kind as change_kind, c.signal, c.change_class
190	243	from url_history h left join events e on e.id = h.event_id left join changes c on c.id = h.change_id where h.url = ${req.query.url} order by h.at desc limit 300`);
191		− const snaps = await db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_length, canonical_hash, title from snapshots where url = ${req.query.url} or sensor_id = (select sensor_id from urls where url = ${req.query.url}) order by captured_at desc limit 200`);
	244	+ const snaps = await db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_length, canonical_hash, title, storage_key is not null as has_raw from snapshots where url = ${req.query.url} or sensor_id = (select sensor_id from urls where url = ${req.query.url}) order by captured_at desc limit 200`);
192	245	return { url: u ?? { url: req.query.url }, history: history.rows, snapshots: snaps.rows };
193	246	});
194	247
195		− // ---- Explore / trending / stats ------------------------------------------------------------
196		− app.get("/api/v1/stats", async () => stats());
197		− app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => ({ items: await trending(Math.min(168, Number(req.query.hours ?? 24)), Math.min(50, Number(req.query.limit ?? 12))) }));
198		− app.get("/api/v1/explore", async () => {
199		− const [mostActive, biggest, silent, clusters, unusual, byType, byCategory] = await Promise.all([
200		− db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, count(*)::int as events_24h, max(e.importance)::float as max_importance from events e join sources s on s.id = e.source_id where e.detected_at >= now() - interval '24 hours' group by s.id, s.name, s.domain order by events_24h desc limit 10`).then((r) => r.rows),
201		− listEvents({ limit: 10, order: "importance", after: new Date(Date.now() - 48 * 3600e3).toISOString() }).then((r) => r.items),
202		− listEvents({ limit: 10, silent_change: true }).then((r) => r.items),
203		− db.execute<Record<string, unknown>>(sql`select c.*, (select json_build_object('id', s.id, 'name', s.name, 'domain', s.domain) from events e join sources s on s.id = e.source_id where e.id = c.primary_event_id) as source from event_clusters c where c.event_count >= 2 and c.last_at >= now() - interval '48 hours' order by c.max_importance desc, c.event_count desc limit 10`).then((r) => r.rows),
204		− db.execute<Record<string, unknown>>(sql`
205		− with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '2 hours' group by s.source_id),
206		− base as (select s.source_id, count()::float / (1424) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id)
207		− select so.id, so.name, so.domain, cur.n::int as changes_2h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day,
208		− round((case when coalesce(base.per_hour,0) = 0 then (case when cur.n/2.0 > 2 then 70 else 40 end) else least(100, case when cur.n/2.0/base.per_hour <= 1 then cur.n/2.0/base.per_hour30 else 30 + 25(ln(cur.n/2.0/base.per_hour)/ln(2)) end) end)::numeric, 1)::float as activity_score
209		− from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id order by activity_score desc limit 10`).then((r) => r.rows),
210		− db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows),
211		− db.execute<Record<string, unknown>>(sql`select c as category, count(*)::int as n from events e, unnest(e.categories) c where e.detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows),
212		− ]);
213		− return { most_active_sources: mostActive, biggest_changes: biggest, silent_changes: silent, clusters, unusual_activity: unusual, by_type: byType, by_category: byCategory, channels: FEED_CHANNELS, event_types: Object.fromEntries(Object.entries(EVENT_TYPES).map(([k, v]) => [k, v.label])) };
	248	+ // ---- Intelligence desks ----------------------------------------------------------------------
	249	+ app.get("/api/v1/stats", async () => cached("stats", 5_000, stats));
	250	+ app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => {
	251	+ const hours = Math.min(168, Number(req.query.hours ?? 24));
	252	+ const limit = Math.min(50, Number(req.query.limit ?? 12));
	253	+ return { items: await cached(`trending:${hours}:${limit}`, 15_000, () => trending(hours, limit)) };
214	254	});
215		− app.get<{ Querystring: { limit?: string; since?: string } }>("/api/v1/clusters", async (req) => {
216		− const rows = await db.execute<Record<string, unknown>>(sql`
217		− select c.*, (select json_agg(json_build_object('id', e.id, 'slug', e.slug, 'title', e.title, 'importance', e.importance, 'event_type', e.event_type, 'detected_at', e.detected_at, 'source_id', e.source_id, 'url', e.url) order by e.importance desc) from events e where e.cluster_id = c.id) as events
218		− from event_clusters c where c.last_at >= now() - make_interval(hours => ${Math.min(720, Number(req.query.since ?? 72))}) order by c.last_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
219		− return { items: rows.rows };
	255	+ app.get("/api/v1/breaking", async () => breakingDesk());
	256	+ app.get("/api/v1/pulse", async () => pulse());
	257	+ app.get("/api/v1/radar", async () => radar());
	258	+ app.get("/api/v1/countries", async () => ({ items: await countryList(), known: COUNTRIES }));
	259	+ app.get<{ Params: { code: string } }>("/api/v1/countries/:code", async (req, reply) => {
	260	+ const c = countryBySlug(req.params.code) ?? (COUNTRIES[req.params.code.toUpperCase()] ? { code: req.params.code.toUpperCase() } : null);
	261	+ if (!c) return reply.status(404).send({ error: "not_found" });
	262	+ return cached(`country:${c.code}`, 20_000, () => countryDesk(c.code));
	263	+ });
	264	+ app.get<{ Params: { channel: string } }>("/api/v1/categories/:channel", async (req, reply) => {
	265	+ const ch = req.params.channel.toLowerCase();
	266	+ if (!FEED_CHANNELS[ch] && !/^[a-z-]{2,30}$/.test(ch)) return reply.status(404).send({ error: "not_found" });
	267	+ return cached(`category:${ch}`, 15_000, () => categoryDesk(ch));
220	268	});
	269	+ app.get("/api/v1/explore", async () =>
	270	+ cached("explore", 20_000, async () => {
	271	+ const since48 = new Date(Date.now() - 48 * 3600e3).toISOString();
	272	+ const [mostActive, biggest, silent, clusters, unusual, byType, byCategory, newly, firstParty] = await Promise.all([
	273	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.first_party, count(*)::int as events_24h, max(e.importance)::float as max_importance from events e join sources s on s.id = e.source_id where e.detected_at >= now() - interval '24 hours' and s.kind = 'registry' group by s.id, s.name, s.domain, s.first_party order by events_24h desc limit 12`).then((r) => r.rows),
	274	+ listEvents({ limit: 12, order: "signal", after: since48 }).then((r) => r.items),
	275	+ listEvents({ limit: 12, silent_change: true, order: "signal", after: since48 }).then((r) => r.items),
	276	+ db.execute<Record<string, unknown>>(sql`select c.*, (select json_build_object('id', s.id, 'name', s.name, 'domain', s.domain) from events e join sources s on s.id = e.source_id where e.id = c.primary_event_id) as source, (select e.slug from events e where e.id = c.primary_event_id) as primary_slug from event_clusters c where c.event_count >= 2 and c.last_at >= now() - interval '48 hours' order by c.max_importance desc, c.event_count desc limit 12`).then((r) => r.rows),
	277	+ db.execute<Record<string, unknown>>(sql`
	278	+ with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '2 hours' group by s.source_id),
	279	+ base as (select s.source_id, count()::float / (1424) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id)
	280	+ select so.id, so.name, so.domain, cur.n::int as changes_2h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day,
	281	+ round((case when coalesce(base.per_hour,0) = 0 then (case when cur.n/2.0 > 2 then 70 else 40 end) else least(100, case when cur.n/2.0/base.per_hour <= 1 then cur.n/2.0/base.per_hour30 else 30 + 25(ln(cur.n/2.0/base.per_hour)/ln(2)) end) end)::numeric, 1)::float as activity_score,
	282	+ (case when coalesce(base.per_hour,0) > 0 then round(((cur.n/2.0/base.per_hour - 1) * 100)::numeric) else null end)::int as pct_vs_baseline
	283	+ from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id where so.kind = 'registry' order by activity_score desc limit 12`).then((r) => r.rows),
	284	+ db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows),
	285	+ db.execute<Record<string, unknown>>(sql`select c as category, count(*)::int as n from events e, unnest(e.categories) c where e.detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows),
	286	+ listEvents({ limit: 12, order: "recent", confidence_min: 60 }).then((r) => r.items),
	287	+ listEvents({ limit: 12, order: "signal", first_party: true, confirmed: true, after: since48 }).then((r) => r.items),
	288	+ ]);
	289	+ return { most_active_sources: mostActive, biggest_changes: biggest, silent_changes: silent, clusters, unusual_activity: unusual, newly_detected: newly, confirmed_first_party: firstParty, by_type: byType, by_category: byCategory, channels: FEED_CHANNELS, groups: Object.fromEntries(Object.entries(EVENT_GROUPS).map(([k, v]) => [k, v.label])), event_types: Object.fromEntries(Object.entries(EVENT_TYPES).map(([k, v]) => [k, v.label])) };
	290	+ }),
	291	+ );
221	292
222	293	// ---- Search ----------------------------------------------------------------------------
223	294	app.get<{ Querystring: { q?: string; limit?: string } }>("/api/v1/search", async (req) => {
224	295	const q = (req.query.q ?? "").trim();
225		− if (q.length < 2) return { query: q, events: [], entities: [], sources: [], urls: [] };
	296	+ const parsed = parseSearch(q);
	297	+ if (q.length < 2) return { query: q, parsed, events: [], entities: [], sources: [], urls: [], clusters: [] };
226	298	const lim = Math.min(50, Number(req.query.limit ?? 20));
227		− const [ev, ents, srcs, urls] = await Promise.all([
228		− listEvents({ q, limit: lim }).then((r) => r.items),
229		− db.execute<Record<string, unknown>>(sql`select id, name, type, domain, importance, event_count from entities where search @@ plainto_tsquery('simple', ${q}) or name ilike ${"%" + q + "%"} order by event_count desc, importance desc limit ${lim}`).then((r) => r.rows),
230		− db.execute<Record<string, unknown>>(sql`select id, name, domain, tier, categories from sources where name ilike ${"%" + q + "%"} or domain ilike ${"%" + q + "%"} limit ${lim}`).then((r) => r.rows),
231		− db.execute<Record<string, unknown>>(sql`select url, domain, status, change_count, last_seen_at from urls where url ilike ${"%" + q + "%"} order by change_count desc limit ${lim}`).then((r) => r.rows),
	299	+ const free = parsed.text;
	300	+ const [ev, ents, srcs, urls, clusters] = await Promise.all([
	301	+ listEvents({ q, limit: lim, order: Object.keys(parsed.filters).length && !free ? "recent" : "recent" }).then((r) => r.items),
	302	+ free.length >= 2 ? db.execute<Record<string, unknown>>(sql`select id, name, type, domain, importance, event_count from entities where search @@ plainto_tsquery('simple', ${free}) or name ilike ${"%" + free + "%"} order by event_count desc, importance desc limit ${lim}`).then((r) => r.rows) : [],
	303	+ free.length >= 2 ? db.execute<Record<string, unknown>>(sql`select id, name, domain, tier, categories, first_party, country from sources where kind = 'registry' and (name ilike ${"%" + free + "%"} or domain ilike ${"%" + free + "%"}) limit ${lim}`).then((r) => r.rows) : [],
	304	+ free.length >= 4 ? db.execute<Record<string, unknown>>(sql`select url, domain, status, change_count, last_seen_at from urls where url ilike ${"%" + free + "%"} order by change_count desc limit ${lim}`).then((r) => r.rows) : [],
	305	+ free.length >= 2 ? db.execute<Record<string, unknown>>(sql`select id, slug, title, state, event_count, source_count, max_importance, last_at from event_clusters where title ilike ${"%" + free + "%"} and last_at >= now() - interval '30 days' order by max_importance desc limit 10`).then((r) => r.rows) : [],
232	306	]);
233		− return { query: q, events: ev, entities: ents, sources: srcs, urls };
	307	+ return { query: q, parsed, events: ev, entities: ents, sources: srcs, urls, clusters };
234	308	});
235	309
236		− // ---- Connector health ----------------------------------------------------------------------
237		− app.get("/api/v1/health/connectors", async () => {
238		− const [connectors, sensorsByHealth, worst, noisy, daily] = await Promise.all([
239		− db.execute<Record<string, unknown>>(sql`select * from connector_health order by connector`).then((r) => r.rows),
240		− db.execute<Record<string, unknown>>(sql`select health, count(*)::int as n from sensors where enabled group by health`).then((r) => r.rows),
241		− db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.url, s.source_id, s.connector, s.health, s.consecutive_errors, s.last_error, s.last_status, s.last_check_at from sensors s where s.enabled and s.health <> 'UP' order by s.consecutive_errors desc, s.last_check_at desc limit 50`).then((r) => r.rows),
242		− db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.url, s.raw_changes, s.meaningful_changes, case when s.raw_changes > 0 then round(1 - s.meaningful_changes::numeric / s.raw_changes, 3) else null end as noise_ratio from sensors s where s.raw_changes >= 5 order by noise_ratio desc nulls last, raw_changes desc limit 25`).then((r) => r.rows),
243		− db.execute<Record<string, unknown>>(sql`select * from metrics_daily order by day desc limit 30`).then((r) => r.rows),
244		− ]);
245		− return { connectors, sensors_by_health: sensorsByHealth, degraded_sensors: worst, noisy_sensors: noisy, daily, live: liveStats() };
246		− });
247		−
248		− // ---- Watchlists & alerts (anonymous owner token, phase 1) ----------------------------------
249		− app.get("/api/v1/watchlists", async (req, reply) => {
250		− const owner = ownerToken(req.headers as Record<string, unknown>);
251		− if (!owner) return reply.status(401).send({ error: "owner token required (X-WebSensor-Owner)" });
252		− const rows = await db.execute<Record<string, unknown>>(sql`select w.*, coalesce((select json_agg(json_build_object('kind', i.kind, 'value', i.value, 'added_at', i.added_at)) from watchlist_items i where i.watchlist_id = w.id), '[]'::json) as items from watchlists w where owner_token = ${owner} order by created_at`);
253		− return { items: rows.rows };
254		− });
255		− app.post<{ Body: { name?: string; items?: { kind: string; value: string }[] } }>("/api/v1/watchlists", async (req, reply) => {
256		− const owner = ownerToken(req.headers as Record<string, unknown>);
257		− if (!owner) return reply.status(401).send({ error: "owner token required" });
258		− const body = z.object({ name: z.string().min(1).max(80).default("My watchlist"), items: z.array(z.object({ kind: z.enum(["entity", "source", "keyword", "category", "url"]), value: z.string().min(1).max(200) })).max(200).default([]) }).parse(req.body ?? {});
259		− const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from watchlists where owner_token = ${owner}`)).rows[0]?.n;
260		− if (Number(count) >= 20) return reply.status(429).send({ error: "too many watchlists" });
261		− const id = newId("wl");
262		− await db.execute(sql`insert into watchlists (id, owner_token, name) values (${id}, ${owner}, ${body.name})`);
263		− for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${id}, ${it.kind}, ${it.value}) on conflict do nothing`);
264		− return { id, name: body.name, items: body.items };
265		− });
266		− app.put<{ Params: { id: string }; Body: { name?: string; items?: { kind: string; value: string }[] } }>("/api/v1/watchlists/:id", async (req, reply) => {
267		− const owner = ownerToken(req.headers as Record<string, unknown>);
268		− if (!owner) return reply.status(401).send({ error: "owner token required" });
269		− const w = (await db.execute<{ id: string }>(sql`select id from watchlists where id = ${req.params.id} and owner_token = ${owner}`)).rows[0];
270		− if (!w) return reply.status(404).send({ error: "not_found" });
271		− const body = z.object({ name: z.string().min(1).max(80).optional(), items: z.array(z.object({ kind: z.enum(["entity", "source", "keyword", "category", "url"]), value: z.string().min(1).max(200) })).max(200).optional() }).parse(req.body ?? {});
272		− if (body.name) await db.execute(sql`update watchlists set name = ${body.name} where id = ${w.id}`);
273		− if (body.items) {
274		− await db.execute(sql`delete from watchlist_items where watchlist_id = ${w.id}`);
275		− for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${w.id}, ${it.kind}, ${it.value}) on conflict do nothing`);
276		− }
277		− return { ok: true };
278		− });
279		− app.delete<{ Params: { id: string } }>("/api/v1/watchlists/:id", async (req, reply) => {
280		− const owner = ownerToken(req.headers as Record<string, unknown>);
281		− if (!owner) return reply.status(401).send({ error: "owner token required" });
282		− await db.execute(sql`delete from watchlists where id = ${req.params.id} and owner_token = ${owner}`);
283		− return { ok: true };
284		− });
285		− app.get<{ Params: { id: string }; Querystring: { limit?: string } }>("/api/v1/watchlists/:id/events", async (req, reply) => {
286		− const owner = ownerToken(req.headers as Record<string, unknown>);
287		− if (!owner) return reply.status(401).send({ error: "owner token required" });
288		− const items = (await db.execute<{ kind: string; value: string }>(sql`select i.kind, i.value from watchlist_items i join watchlists w on w.id = i.watchlist_id where w.id = ${req.params.id} and w.owner_token = ${owner}`)).rows;
289		− if (!items.length) return reply.send({ items: [] });
290		− const ents = items.filter((i) => i.kind === "entity").map((i) => i.value);
291		− const srcs = items.filter((i) => i.kind === "source").map((i) => i.value);
292		− const cats = items.filter((i) => i.kind === "category").map((i) => i.value);
293		− const kws = items.filter((i) => i.kind === "keyword").map((i) => i.value);
294		− const conds = [] as ReturnType<typeof sql>[];
295		− if (ents.length) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = any(${textArray(ents)}))`);
296		− if (srcs.length) conds.push(sql`e.source_id = any(${textArray(srcs)})`);
297		− if (cats.length) conds.push(sql`e.categories && ${textArray(cats)}`);
298		− for (const k of kws) conds.push(sql`(e.title ilike ${"%" + k + "%"} or e.summary ilike ${"%" + k + "%"})`);
299		− if (!conds.length) return { items: [] };
300		− const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where ${sql.join(conds, sql` or `)} order by e.detected_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`);
301		− return { items: rows.rows };
302		− });
303		−
304		− app.get("/api/v1/alerts", async (req, reply) => {
305		− const owner = ownerToken(req.headers as Record<string, unknown>);
306		− if (!owner) return reply.status(401).send({ error: "owner token required" });
307		− return { items: (await db.execute<Record<string, unknown>>(sql`select * from alerts where owner_token = ${owner} order by created_at`)).rows };
308		− });
309		− app.post<{ Body: Record<string, unknown> }>("/api/v1/alerts", async (req, reply) => {
310		− const owner = ownerToken(req.headers as Record<string, unknown>);
311		− if (!owner) return reply.status(401).send({ error: "owner token required" });
312		− const body = z.object({ name: z.string().min(1).max(80), rule: z.object({ importance_min: z.number().min(0).max(100).optional(), event_types: z.array(z.string()).optional(), entities: z.array(z.string()).optional(), sources: z.array(z.string()).optional(), keywords: z.array(z.string()).optional(), silent_only: z.boolean().optional(), categories: z.array(z.string()).optional() }), channel: z.enum(["web"]).default("web") }).parse(req.body ?? {});
313		− const id = newId("alr");
314		− await db.execute(sql`insert into alerts (id, owner_token, name, rule, channel) values (${id}, ${owner}, ${body.name}, ${JSON.stringify(body.rule)}::jsonb, ${body.channel})`);
315		− return { id, ...body };
316		− });
317		− app.delete<{ Params: { id: string } }>("/api/v1/alerts/:id", async (req, reply) => {
318		− const owner = ownerToken(req.headers as Record<string, unknown>);
319		− if (!owner) return reply.status(401).send({ error: "owner token required" });
320		− await db.execute(sql`delete from alerts where id = ${req.params.id} and owner_token = ${owner}`);
321		− return { ok: true };
322		− });
	310	+ // ---- Connector health / system --------------------------------------------------------------
	311	+ app.get("/api/v1/health/connectors", async () =>
	312	+ cached("health", 10_000, async () => {
	313	+ const [connectors, sensorsByHealth, sensorsByStatus, worst, noisy, daily, byConnectorSensors, throughput, es, topFailingDomains, slowest] = await Promise.all([
	314	+ db.execute<Record<string, unknown>>(sql`select * from connector_health order by connector`).then((r) => r.rows),
	315	+ db.execute<Record<string, unknown>>(sql`select health, count(*)::int as n from sensors where enabled group by health`).then((r) => r.rows),
	316	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from sensors group by status`).then((r) => r.rows),
	317	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.url, s.source_id, s.connector, s.health, s.consecutive_errors, s.last_error, s.last_status, s.last_check_at from sensors s where s.enabled and s.health <> 'UP' order by s.consecutive_errors desc, s.last_check_at desc limit 50`).then((r) => r.rows),
	318	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.url, s.raw_changes, s.meaningful_changes, case when s.raw_changes > 0 then round(1 - s.meaningful_changes::numeric / s.raw_changes, 3) else null end as noise_ratio from sensors s where s.raw_changes >= 5 order by noise_ratio desc nulls last, raw_changes desc limit 25`).then((r) => r.rows),
	319	+ db.execute<Record<string, unknown>>(sql`select * from metrics_daily order by day desc limit 30`).then((r) => r.rows),
	320	+ db.execute<Record<string, unknown>>(sql`select connector, count()::int as sensors, count() filter (where health = 'UP')::int as up, avg(avg_latency_ms)::int as avg_latency_ms from sensors where enabled group by connector order by sensors desc`).then((r) => r.rows),
	321	+ db.execute<Record<string, unknown>>(sql`select (select count() from sensor_runs where started_at >= now() - interval '5 minutes')::int as checks_5m, (select count() from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified')::int as not_modified_5m, (select count() from events where detected_at >= now() - interval '5 minutes')::int as events_5m, (select count() from changes where detected_at >= now() - interval '5 minutes')::int as changes_5m, (select count() from sensors where enabled and next_check_at <= now())::int as queue_due, (select avg(duration_ms)::int from sensor_runs where started_at >= now() - interval '5 minutes')::int as avg_latency_5m_ms, (select count() from changes where detected_at >= now() - interval '24 hours' and change_class in ('cosmetic','navigation','timestamp','advertisement','boilerplate'))::int as noise_filtered_24h`).then((r) => r.rows[0]),
	322	+ engineStatus(),
	323	+ db.execute<Record<string, unknown>>(sql`select split_part(split_part(s.url, '/', 3), ':', 1) as host, count(*)::int as failures, max(r.error) as last_error from sensor_runs r join sensors s on s.id = r.sensor_id where r.started_at >= now() - interval '24 hours' and r.outcome in ('error','parse_error','rate_limited') group by 1 order by failures desc limit 15`).then((r) => r.rows),
	324	+ db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.avg_latency_ms, s.connector from sensors s where s.enabled and s.avg_latency_ms is not null order by s.avg_latency_ms desc limit 15`).then((r) => r.rows),
	325	+ ]);
	326	+ const t = throughput ?? {};
	327	+ return { connectors, sensors_by_health: sensorsByHealth, sensors_by_status: sensorsByStatus, degraded_sensors: worst, noisy_sensors: noisy, daily, sensors_by_connector: byConnectorSensors, throughput: { ...t, checks_per_min: Math.round(Number(t.checks_5m ?? 0) / 5), events_per_min: Math.round((Number(t.events_5m ?? 0) / 5) * 10) / 10, not_modified_ratio: Number(t.checks_5m) ? Math.round((Number(t.not_modified_5m) / Number(t.checks_5m)) * 100) / 100 : null }, engine: es, top_failing_domains: topFailingDomains, slowest_sensors: slowest, live: liveStats() };
	328	+ }),
	329	+ );
323	330
324	331	// ---- Machine-readable feeds ------------------------------------------------------------------
325		− app.get<{ Querystring: { category?: string; importance_min?: string; silent_change?: string } }>("/api/v1/feed.rss", async (req, reply) => {
326		− const q = eventsQuery.parse({ ...req.query, limit: 50 });
	332	+ app.get("/api/v1/feed.rss", async (req, reply) => {
	333	+ const q = eventsQuery.parse({ ...(req.query as Record<string, unknown>), limit: 50 });
327	334	const { items } = await listEvents(q);
328	335	const esc = (s: unknown): string => String(s ?? "").replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">").replace(/"/g, """);
329	336	const base = config.publicBaseUrl;
330		− const xml = `<?xml version="1.0" encoding="UTF-8"?>\n<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>WebSensor — ${esc(q.category ? q.category + " events" : "live events")}</title><link>${base}</link><description>Meaningful changes detected on the public Web by WebSensor.</description><atom:link href="${base}/api/v1/feed.rss" rel="self" type="application/rss+xml"/>${items
331		− .map((e) => `<item><title>${esc(e.title)}</title><link>${base}/event/${esc(e.slug)}</link><guid isPermaLink="false">${esc(e.id)}</guid><pubDate>${new Date(String(e.detected_at)).toUTCString()}</pubDate><category>${esc(e.event_type)}</category><description>${esc(e.summary)} (importance ${esc(e.importance)}, confidence ${esc(e.confidence)}${e.silent_change ? ", silent change" : ""}) — source: ${esc(e.url)}</description></item>`)
	337	+ const xml = `<?xml version="1.0" encoding="UTF-8"?>\n<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>WebSensor — ${esc(q.category ? q.category + " events" : q.silent_change ? "silent changes" : "live events")}</title><link>${base}</link><description>Meaningful changes detected on the public Web by WebSensor.</description><atom:link href="${base}/api/v1/feed.rss" rel="self" type="application/rss+xml"/>${items
	338	+ .map((e) => `<item><title>${esc(e.title)}</title><link>${base}/event/${esc(e.slug)}</link><guid isPermaLink="false">${esc(e.id)}</guid><pubDate>${new Date(String(e.detected_at)).toUTCString()}</pubDate><category>${esc(e.event_type)}</category><description>${esc(e.summary)} (signal ${esc(e.signal_score ?? e.importance)}, importance ${esc(e.importance)}, confidence ${esc(e.confidence)}${e.silent_change ? ", silent change" : ""}${e.first_party ? ", first-party" : ""}) — source: ${esc(e.url)}</description></item>`)
332	339	.join("")}</channel></rss>`;
333	340	reply.header("content-type", "application/rss+xml; charset=utf-8");
334	341	return reply.send(xml);
		`@@ -337,13 +344,59 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {`
337	344	app.get("/api/v1", async () => ({
338	345	name: "WebSensor API",
339	346	version: "v1",
	347	+ release: config.version,
340	348	docs: `${config.publicBaseUrl}/api`,
341		− endpoints: ["/api/v1/events", "/api/v1/events/{id\|slug}", "/api/v1/changes/{id}", "/api/v1/snapshots/{id}", "/api/v1/snapshots/compare?a=&b=", "/api/v1/sources", "/api/v1/sources/{id}", "/api/v1/sensors/{id}", "/api/v1/entities", "/api/v1/entities/{id}", "/api/v1/entities/{id}/timeline", "/api/v1/domains/{domain}/timeline", "/api/v1/urls/history?url=", "/api/v1/search?q=", "/api/v1/stats", "/api/v1/trending", "/api/v1/explore", "/api/v1/clusters", "/api/v1/health/connectors", "/api/v1/watchlists", "/api/v1/alerts", "/api/v1/feed.rss", "wss://…/api/v1/live"],
	349	+ rate_limit: "600 requests / minute / IP (headers x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-reset)",
	350	+ endpoints: [
	351	+ "/api/v1/events",
	352	+ "/api/v1/events/count",
	353	+ "/api/v1/events/{id\|slug}",
	354	+ "/api/v1/changes/{id}",
	355	+ "/api/v1/snapshots/{id}",
	356	+ "/api/v1/snapshots/compare?a=&b=",
	357	+ "/api/v1/sensors/{id}",
	358	+ "/api/v1/sensors/{id}/snapshots",
	359	+ "/api/v1/sources",
	360	+ "/api/v1/sources/{id}",
	361	+ "/api/v1/entities",
	362	+ "/api/v1/entities/rank",
	363	+ "/api/v1/entities/{id}",
	364	+ "/api/v1/entities/{id}/timeline",
	365	+ "/api/v1/clusters",
	366	+ "/api/v1/clusters/{id\|slug}",
	367	+ "/api/v1/domains/{domain}/timeline",
	368	+ "/api/v1/urls/history?url=",
	369	+ "/api/v1/search?q=",
	370	+ "/api/v1/stats",
	371	+ "/api/v1/trending",
	372	+ "/api/v1/breaking",
	373	+ "/api/v1/pulse",
	374	+ "/api/v1/radar",
	375	+ "/api/v1/explore",
	376	+ "/api/v1/countries",
	377	+ "/api/v1/countries/{code\|slug}",
	378	+ "/api/v1/categories/{channel}",
	379	+ "/api/v1/health/connectors",
	380	+ "/api/v1/watchlists",
	381	+ "/api/v1/alerts",
	382	+ "/api/v1/notifications",
	383	+ "/api/v1/bookmarks",
	384	+ "/api/v1/views",
	385	+ "/api/v1/monitors",
	386	+ "/api/v1/feed.rss",
	387	+ "wss://…/api/v1/live",
	388	+ ],
342	389	}));
343	390	}
344	391
	392	+function stripEmpty(o: Record<string, unknown>): Record<string, unknown> {
	393	+ return Object.fromEntries(Object.entries(o).filter(([, v]) => v !== undefined && v !== null && v !== ""));
	394	+}
	395	+
345	396	function renderCanonical(c: { mode: string; text?: string; items?: Record<string, unknown>[]; json?: unknown }): string {
346	397	if (c.mode === "text") return c.text ?? "";
347	398	if (c.mode === "list") return (c.items ?? []).map((i) => [i.title ?? i.url ?? i.key, i.url && i.title ? i.url : null, i.summary ? String(i.summary).slice(0, 300) : null].filter(Boolean).join(" — ")).join("\n");
348	399	return JSON.stringify(c.json ?? null, null, 2);
349	400	}
	401	+
	402	+export { textArray };

modified apps/api/src/server.ts +9 −2

Large diff (35 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/alerts.ts +132 −0

Large diff (132 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/cli.ts +18 −1

Large diff (31 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/cluster.ts +78 −22

Large diff (180 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/config.ts +6 −2

Large diff (20 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/engine.test.ts +73 −0

Large diff (73 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/entities.ts +79 −3

Large diff (111 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/index.ts +8 −1

Large diff (24 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/metrics.ts +20 −0

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/pipeline.ts +142 −30

Large diff (368 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/redis.ts +19 −4

Large diff (34 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/registry.ts +46 −6

Large diff (83 lines) is not loaded by default to keep the page fast.Load diff

added apps/engine/src/retention.ts +49 −0

Large diff (49 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/scheduler.ts +41 −2

Large diff (82 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/seeds.ts +6 −52

Large diff (76 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/alerts/alerts.tsx +362 −71

Large diff (528 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/alerts/page.tsx +7 −4

Large diff (21 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/api/page.tsx +144 −62

Large diff (270 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/bookmarks/bookmarks.tsx +184 −0

Large diff (184 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/bookmarks/page.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/breaking/loading.tsx +41 −0

Large diff (41 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/breaking/page.tsx +67 −6

Large diff (84 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/category/[channel]/loading.tsx +28 −0

Large diff (28 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/category/[channel]/page.tsx +111 −10

Large diff (140 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/cluster/[slug]/layout.tsx +11 −0

Large diff (11 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/cluster/[slug]/loading.tsx +50 −0

Large diff (50 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/cluster/[slug]/page.tsx +240 −0

Large diff (240 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/company/[id]/page.tsx +10 −106

Large diff (121 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/country/[slug]/loading.tsx +42 −0

Large diff (42 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/country/[slug]/page.tsx +115 −0

Large diff (115 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/country/loading.tsx +17 −0

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/country/page.tsx +39 −0

Large diff (39 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/entities/loading.tsx +35 −0

Large diff (35 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/entities/page.tsx +70 −11

Large diff (112 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/entity/[id]/layout.tsx +15 −0

Large diff (15 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/entity/[id]/loading.tsx +56 −0

Large diff (56 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/entity/[id]/page.tsx +480 −0

Large diff (480 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/error.tsx +26 −7

Large diff (40 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/event/[slug]/loading.tsx +57 −0

Large diff (57 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/event/[slug]/opengraph-image.tsx +89 −0

Large diff (89 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/event/[slug]/page.tsx +248 −113

Large diff (475 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/explore/loading.tsx +27 −0

Large diff (27 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/explore/page.tsx +373 −98

Large diff (485 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/globals.css +103 −10

Large diff (177 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/health/loading.tsx +38 −0

Large diff (38 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/health/page.tsx +122 −14

Large diff (175 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/layout.tsx +18 −9

Large diff (52 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/live/loading.tsx +30 −0

Large diff (30 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/live/page.tsx +39 −0

Large diff (39 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/monitors/monitors.tsx +310 −0

Large diff (310 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/monitors/page.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/not-found.tsx +19 −8

Large diff (34 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/ops/ops.tsx +706 −0

Large diff (706 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/ops/page.tsx +14 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/page.tsx +35 −10

Large diff (63 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/pulse/loading.tsx +48 −0

Large diff (48 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/pulse/page.tsx +115 −0

Large diff (115 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/radar/loading.tsx +30 −0

Large diff (30 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/radar/page.tsx +127 −0

Large diff (127 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/search/page.tsx +134 −56

Large diff (208 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/sensor/[id]/loading.tsx +41 −0

Large diff (41 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/sensor/[id]/page.tsx +165 −73

Large diff (267 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/silent/loading.tsx +38 −0

Large diff (38 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/silent/page.tsx +44 −5

Large diff (61 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/sitemap.ts +27 −6

Large diff (46 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/source/[id]/loading.tsx +41 −0

Large diff (41 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/source/[id]/page.tsx +190 −51

Large diff (317 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/sources/loading.tsx +32 −0

Large diff (32 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/sources/page.tsx +58 −21

Large diff (106 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/timeline/[id]/page.tsx +7 −28

Large diff (40 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/watchlists/page.tsx +2 −2

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/watchlists/watchlists.tsx +210 −45

Large diff (337 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/command-palette.tsx +205 −0

Large diff (205 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/event-drawer.tsx +330 −0

Large diff (330 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/event-row.tsx +68 −22

Large diff (116 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/field-changes.tsx +52 −0

Large diff (52 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/live-feed.tsx +196 −38

Large diff (318 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/live-strip.tsx +76 −0

Large diff (76 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/nav.tsx +78 −52

Large diff (177 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/prefs.tsx +87 −0

Large diff (87 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/rail.tsx +108 −17

Large diff (156 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/ui.tsx +194 −35

Large diff (290 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/api.ts +331 −10

Large diff (499 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/lib/feed-filters.ts +34 −0

Large diff (34 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/format.ts +83 −0

Large diff (86 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/use-live.ts +30 −6

Large diff (76 lines) is not loaded by default to keep the page fast.Load diff

added config/sources.d/40-ai-frontier.yaml +1129 −0

Large diff (1,129 lines) is not loaded by default to keep the page fast.Load diff

added config/sources.d/41-cloud-infrastructure.yaml +994 −0

Large diff (994 lines) is not loaded by default to keep the page fast.Load diff

added config/sources.d/42-cybersecurity.yaml +1770 −0

Large diff (1,770 lines) is not loaded by default to keep the page fast.Load diff

added config/sources.d/43-finance-markets.yaml +1278 −0

Large diff (1,278 lines) is not loaded by default to keep the page fast.Load diff

added config/sources.d/44-governments.yaml +1660 −0

Large diff (1,660 lines) is not loaded by default to keep the page fast.Load diff

Diff truncated — file too large.

spb/websensor

WebSensor 0.2 — signal intelligence upgrade

Theme