WebSensor 0.2 — signal intelligence upgrade
Registry: 7 depth fragments (40–46: AI frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport/telecom/sports/news) → 3 167 organizations / 6 320 validated sensors (was 2 681 / 4 052); seeds carry country, language and first_party; sync infers country/language by TLD and a 0–3 scheduling priority. Core: semantic diff (noise classes vs pricing/policy/product/personnel, field-level before → after with % deltas), WebSensor Signal Score with explainable reasons, impact / velocity / anomaly scores, breaking state, entity rank, ~40 new event classes (zero_day, active_exploitation, supply_chain_attack, guidance, buyback, sanction, legislation, outbreak, grounding, service_shutdown…), search syntax (entity: type: after:7d silent: importance:>70 country:), shared registry schema. Engine: noise never becomes an event, routine firehose batches damped, fingerprint idempotency, ambiguous-alias rule for entity resolution (+ relink-entities CLI), silent-change bar (first-party · eligible type · importance ≥ 45), clusters with propagation timeline, first-party/external counts, velocity, lead time and breaking/developing/confirmed states, server-side alerts with HMAC-signed webhooks, raw-body retention with reference counting, priority-aware scheduler with per-domain circuit breaker and Redis heartbeat, UTF-16 BOM transcoding. API: new filters (signal_min, group, first_party, confirmed, country, language, change_class, q syntax, order=signal), /breaking /pulse /radar /clusters/:slug /entities/rank /countries /categories/:channel /sensors/:id/snapshots, owner routes (bookmarks, saved views, notifications, custom URL monitors with SSRF checks), admin routes behind WS_ADMIN_TOKEN (ops, sensor/source actions, connector test, YAML/JSON bulk import), TTL cache, WebSocket protocol 2 with stream ids and replay. Web: terminal-style design system (badges, density modes, skeletons, sparklines, heatmaps), live system strip, ⌘K command palette, intelligence drawer, live feed v2 (URL-synced filters, pause, "↑ N new events", replay), pages /live /pulse /radar /entity/[id] /cluster/[slug] /country /bookmarks /monitors /ops, redesigned breaking / silent / explore / event / source / sensor / search / alerts / watchlists / health / api docs, OG image per event, sitemap, loading skeletons. Migrations 0005 (additive schema) and 0006 (backfills). Docs updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
102 changed files +16,500 −1,428
modified
.env.example
+15 −1
@@ -26,5 +26,19 @@ WS_LLM_MIN_IMPORTANCE=35 | ||
| 26 | 26 | |
| 27 | 27 | # Engine tuning |
| 28 | 28 | WS_FETCH_CONCURRENCY=16 |
| 29 | −WS_USER_AGENT=WebSensorBot/0.1 (+https://www.websensor.io/bot) | |
| 29 | +WS_USER_AGENT="WebSensorBot/0.2 (+https://www.websensor.io/bot; contact@websensor.io)" | |
| 30 | 30 | WS_SOURCES_FILE=./config/sources.yaml |
| 31 | +WS_PER_HOST_CONCURRENCY=2 | |
| 32 | +WS_MEANINGFUL_SIGNAL=0.32 | |
| 33 | +# Silent-change bar: only silent-eligible types with importance >= this are flagged (spec §23) | |
| 34 | +WS_SILENT_MIN_IMPORTANCE=45 | |
| 35 | +# Storage lifecycle (spec §58): raw bodies of unchanged snapshots are dropped after N days (canonical + hashes kept); | |
| 36 | +# snapshots referenced by an event are never pruned. WS_RETENTION=0 disables the job. | |
| 37 | +WS_RETENTION=1 | |
| 38 | +WS_RETENTION_RAW_DAYS=21 | |
| 39 | +WS_RETENTION_CHANGED_RAW_DAYS=60 | |
| 40 | + | |
| 41 | +# API — internal operations (/api/v1/admin/*, /ops). Unset = admin API disabled. | |
| 42 | +WS_ADMIN_TOKEN= | |
| 43 | +# Custom URL monitors per anonymous owner (spec §45) | |
| 44 | +WS_MONITORS_PER_OWNER=5 | |
modified
CLAUDE.md
+93 −80
@@ -1,97 +1,110 @@ | ||
| 1 | 1 | # WebSensor — repository guide |
| 2 | 2 | |
| 3 | 3 | WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing |
| 4 | −Web. It monitors official public sources, detects meaningful changes, classifies them, links entities, scores | |
| 5 | −importance, preserves evidence and publishes events through a WebSocket feed. The product brief that drove the | |
| 6 | −design is `docs/PRODUCT-BRIEF.md`; this file is the working guide for the code. | |
| 4 | +Web. It monitors official public sources, detects meaningful changes, classifies them semantically, links | |
| 5 | +entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score, | |
| 6 | +preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through | |
| 7 | +a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file | |
| 8 | +is the working guide for the code (v0.2, 2026-09-11). | |
| 7 | 9 | |
| 8 | 10 | ## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15) |
| 9 | −- `packages/core` — taxonomy (event types + intrinsic severity, tiers, categories), ids, SSRF policy | |
| 10 | − (`assertUrlAllowed`, `safeLookup`), hashing (sha256, simhash, shingles/Jaccard), canonical extraction | |
| 11 | − (`canonicalizeHtml`: strips scripts/nav/footer/cookie chrome, timestamps, tokens, counters), diff engines | |
| 12 | − (text / json / keyed list), stage-1 heuristics (`evaluateChange`, `describeChange`), scoring | |
| 13 | − (importance components, confidence, novelty helpers, trending, activity anomaly), adaptive schedule. | |
| 14 | − `@websensor/core/client` is the browser-safe subset (no `node:` imports) — client components must import it. | |
| 15 | −- `packages/db` — plain SQL migrations (`migrations/*.sql`, applied by `migrate()` with an advisory lock) + | |
| 16 | − Drizzle schema for typed access. **`textArray()`** must be used for `= any(...)` / `&&` with JS arrays | |
| 17 | − (Drizzle spreads arrays into parameter lists). | |
| 18 | −- `packages/store` — content-addressed blob store (`sha256/ab/cd/<hash>.zst`, zstd via `node:zlib`). Raw | |
| 19 | − bodies, canonical representations and full diffs live there, never in Postgres. Interface ready for S3/MinIO. | |
| 20 | −- `packages/connectors` — connector SDK (`WebSensorConnector.fetch/normalize`), safe fetcher (conditional GET, | |
| 21 | − manual redirects validated per hop, HTTP/2→1.1 pin on NGHTTP2 errors, browser-UA second attempt on | |
| 22 | − 403/resets, size + time limits, `keepAllHeaders`), 16 connector families: `http` (HTML/JSON/text/HEAD), `rss` | |
| 23 | − (RSS/Atom/RDF/JSON Feed), `sitemap` (index + news), `statuspage` (Atlassian v2 summary), `statusjson` (Instatus / | |
| 24 | − incident.io / Status.io), `github` (releases/tags/commits Atom, advisories REST), `jsonlist` (keyed records from | |
| 25 | − any JSON API; `{now-2h}` placeholders), `package` (npm/PyPI/crates/RubyGems/NuGet/Packagist/Hex/Go proxy/Homebrew/ | |
| 26 | − Docker Hub version streams), `edgar` (SEC submissions → filings, 8-K items decoded), `openapi` (API contract | |
| 27 | − fingerprints), `csv` (open-data rows, `tail`), `pdf` (pdf.js text via unpdf), `dns` (A/AAAA/NS/MX/TXT/CAA/SOA/DMARC), | |
| 28 | − `tls` (certificate identity, chain, ALPN, expiry), `headers` (security/infra response headers), `rdap` (registrar, | |
| 29 | − EPP status, expiry via IANA bootstrap), `discovery` (robots sitemaps, `<link rel=alternate>`, well-known feed | |
| 30 | − paths, linked status pages — every candidate is fetched and parsed), `scrapfly` (fallback, budgeted, only when | |
| 31 | − `fallback.scrapfly` is set on the source). `dns://` and `tls://` sensor URLs are accepted (host only). | |
| 32 | −- `apps/engine` — scheduler (`FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency), pipeline | |
| 33 | − (fetch → normalize → snapshot → diff → heuristics → change → event: entities, novelty, LLM interpretation, | |
| 34 | − clustering, importance/confidence, silent-change detection, publish to Redis), registry sync from | |
| 35 | − `config/sources.yaml` + `config/sources.d/`, discovery, connector health rollups, Prometheus metrics on :8262. | |
| 36 | − CLI: `src/cli.ts` (`sync`, `discover`, `probe <domain>`, `run-once <sensor>`, `run-due [n]`); `src/validate.ts` | |
| 37 | − (registry validator, see below). | |
| 38 | −- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*`, WebSocket `/api/v1/live` (Redis pub/sub fan-out, | |
| 39 | − channels `events:*`, `entity:*`, `source:*`, `watchlist:*`), `/api/v1/feed.rss`, `/api/health` `/api/ready` `/api/metrics`, apex→www | |
| 40 | − redirect, and a reverse proxy to the Next.js app for everything else. | |
| 41 | −- `apps/web` — Next.js 16 frontend (:8261, loopback). Pages: live, breaking, explore, sources, entities, | |
| 42 | − timelines, silent changes, watchlists, alerts, event detail with diff viewer, health, API docs. | |
| 43 | −- `config/sources.yaml` — the founding 271-organization registry (incl. governments and news media; news sources | |
| 44 | − run with `llm: false`, heuristics only) + `config/sources.d/*.yaml` fragments, one per **site class** (open source, | |
| 45 | − central banks & finance, energy/climate/weather, telecom & internet infrastructure, retail/consumer/travel, | |
| 46 | − gaming & entertainment, universities & research, international orgs/NGOs/standards, sports, crypto, | |
| 47 | − transport/aviation/space, consumer safety/food/agri, housing/labour/open data, enterprise SaaS, health systems & | |
| 48 | − medtech, politics/elections/courts, EDGAR filings, package registries, web posture, OpenAPI, status JSON, | |
| 49 | − documents & data). Fragments are merged by `apps/engine/src/seeds.ts` in file-name order; an entry with | |
| 50 | − `extend: true` adds sensors/products/aliases to a source declared earlier. Authoring guide + connector | |
| 51 | − cheat-sheet: `docs/registry/AUTHORING.md`; `docs/connectors/*.md` document each connector family. | |
| 52 | −- Registry validator (no DB): `node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts [fragment.yaml] [--all] | |
| 53 | − [--source id] [--connector key] [--json report.json]` runs every curated sensor through its connector and prints | |
| 54 | − OK/WARN/FAIL; `--probe <domain>` is a discovery dry-run. `scripts/prune-fragment.py` removes FAIL/WARN sensors from | |
| 55 | − a fragment using the JSON report; `scripts/gen-web-posture.py` regenerates the web-posture fragment. | |
| 11 | +- `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups, | |
| 12 | + silent-eligible types, change classes, cluster states, countries), ids, SSRF policy (`assertUrlAllowed`, | |
| 13 | + `safeLookup`), hashing, canonical extraction, diff engines (text / json / keyed list), stage-1 heuristics | |
| 14 | + (`evaluateChange`, `describeChange`, `humanizeUrl`), **semantic diff** (`semantic.ts`: `classifyChange` → | |
| 15 | + cosmetic / navigation / timestamp / advertisement / boilerplate vs meaningful / pricing / policy / product / | |
| 16 | + personnel; `extractFieldChanges` → field-level before → after pairs with % deltas), scoring (`scoring.ts`: | |
| 17 | + importance components, confidence, impact, velocity, **WebSensor Signal Score** with explainable reasons, | |
| 18 | + `breakingState`, entity rank, daily anomaly), adaptive schedule, **search syntax** (`search.ts`: `entity:` `type:` | |
| 19 | + `after:7d` `silent:true` `importance:>70` `country:CA` …), registry seed schema (`registry-schema.ts`, shared by | |
| 20 | + engine, validator and the admin import endpoint). `@websensor/core/client` is the browser-safe subset. | |
| 21 | +- `packages/db` — plain SQL migrations (`migrations/*.sql`, applied by `migrate()` with an advisory lock) + Drizzle | |
| 22 | + schema. `0005_intelligence.sql` added provenance (first_party / country / language / kind / owner_token on | |
| 23 | + sources), sensor lifecycle (`status`, `priority`, `validated_at`), semantic class + field changes on changes, | |
| 24 | + signal/velocity/impact/anomaly scores, fingerprints, score reasons on events, cluster propagation columns, | |
| 25 | + `entity_daily`, `source_daily`, `bookmarks`, `saved_views`, alert channel config, notification delivery. | |
| 26 | + `0006_backfill_states.sql` backfills them. **`textArray()`** must be used for `= any(...)` / `&&` with JS arrays. | |
| 27 | +- `packages/store` — content-addressed blob store (`sha256/ab/cd/<hash>.zst`). Raw bodies, canonical forms and full | |
| 28 | + diffs live there, never in Postgres. | |
| 29 | +- `packages/connectors` — connector SDK (`fetch/normalize`), safe fetcher (conditional GET, per-hop SSRF checks, | |
| 30 | + HTTP/2→1.1 pin, browser-UA retry, UTF-16 BOM transcoding, `postJson` for signed webhooks), 16 connector families | |
| 31 | + (`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) + | |
| 32 | + `discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and | |
| 33 | + `{field.path}`. | |
| 34 | +- `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency, | |
| 35 | + **domain circuit breaker**, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff → | |
| 36 | + heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias | |
| 37 | + rules, novelty, optional Claude interpretation, impact / anomaly / signal scores, silent-change bar, clustering | |
| 38 | + with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook | |
| 39 | + delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery, | |
| 40 | + connector health rollups, **retention** (`retention.ts`), Prometheus metrics :8262. | |
| 41 | + CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`, | |
| 42 | + `refresh-clusters`, `llm-test`); `src/validate.ts` (registry validator). | |
| 43 | +- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped: | |
| 44 | + watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts` | |
| 45 | + behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models | |
| 46 | + (`intel.ts`: breaking desk, pulse, radar, entity insights, rankings, cluster detail, country & category desks), | |
| 47 | + TTL cache (`cache.ts`), WebSocket `/api/v1/live` protocol 2 (`live.ts`: `sid` on every frame, `{"since"}` replay | |
| 48 | + from the durable stream, channels `events:* group:* country:* state:* type:* entity:* source:* watchlist:*`), | |
| 49 | + `/api/v1/feed.rss`, `/api/health|ready|metrics`, apex→www redirect, reverse proxy to Next. | |
| 50 | +- `apps/web` — Next.js 16 (:8261 loopback). Design system in `components/ui.tsx` (Panel, Badge, Score, Tabs, | |
| 51 | + Sparkline, Heatmap, Skeleton…), `prefs.tsx` (density compact/normal/comfortable, pause), `event-drawer.tsx` | |
| 52 | + (intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters, | |
| 53 | + pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar` | |
| 54 | + `/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]` | |
| 55 | + `/country/[slug]` `/event/[slug]` (+ OG image) `/bookmarks` `/watchlists` `/alerts` `/monitors` `/ops` `/health` | |
| 56 | + `/api`. | |
| 57 | +- `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order | |
| 58 | + (`extend: true` adds to an earlier source). Fragments 10–35 = site classes; **40–46 (2026-09-11)** = depth: AI | |
| 59 | + frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport / | |
| 60 | + telecom / sports / news. Seeds may carry `country:` (ISO-2, `EU`, `INT`), `language:` and `first_party: false` | |
| 61 | + (media). Guide: `docs/registry/AUTHORING.md`. Validator: `node node_modules/tsx/dist/cli.mjs | |
| 62 | + apps/engine/src/validate.ts <fragment> [--all] [--json report.json]` — nothing enters the registry without OK. | |
| 56 | 63 | |
| 57 | 64 | ## Rules |
| 58 | −- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets — goes through | |
| 59 | − `assertUrlAllowed()`. Private ranges, metadata endpoints, `.maclustr.io`/`.ts.net` and single-label hosts are | |
| 60 | − blocked; the dispatcher's DNS lookup only returns approved addresses. | |
| 61 | −- Raw evidence is immutable: snapshots and diffs are never rewritten. Re-interpretation creates a new row in | |
| 62 | − `interpretations` (versioned); `events.interpretation` holds the latest. | |
| 63 | −- LLM cost control: heuristics first; Claude is called only for candidates above `WS_LLM_MIN_IMPORTANCE`, with a | |
| 64 | − daily call budget; the deep model only above `WS_LLM_DEEP_MIN_IMPORTANCE`. Output is strict JSON | |
| 65 | − (`output_config.format`), never free text. No key → heuristics only, the system still works. | |
| 66 | −- Feeds: items are "new" only if their key was never seen (sensor `state.seenKeys`) and they are not older than | |
| 67 | − 14 days; items scrolling out of the window are never "removed". Sitemaps/statuspages: a >50 % shrink is | |
| 68 | − treated as a partial response, not mass deletion. A 404 becomes `page_removed` only after | |
| 69 | − `WS_DELETE_CONFIRMATIONS` checks separated by `WS_DELETE_SEPARATION_MIN`. | |
| 70 | −- Never label inference as fact: events carry `evidence_label` (OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED) | |
| 71 | − and the interpretation keeps `observed` and `inferred` apart. Silent changes are flagged, never asserted as | |
| 72 | − "unannounced" without checking recent announcement-type events of the same source. | |
| 73 | −- Every sensor in the registry has been fetched and parsed successfully by the validator before being committed; | |
| 74 | − a fragment that fails validation is not merged. Never add sensors to the registry without running it. | |
| 75 | −- SSRF policy: NAT64 addresses (`64:ff9b::/96`) are judged by the embedded IPv4 address (IPv6-only networks with | |
| 76 | − 464XLAT resolve every IPv4-only host that way), not blocked wholesale. | |
| 77 | −- Do not add sensors that are blocked with 403 by design (Akamai/Cloudflare bot management) unless the source | |
| 78 | − has `fallback.scrapfly: true` and a tier ≥ C; Scrapfly is a budgeted fallback, not the foundation. | |
| 79 | −- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`…). | |
| 65 | +- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom | |
| 66 | + monitors — goes through `assertUrlAllowed()`. Private ranges, metadata endpoints, `.maclustr.io`/`.ts.net`, | |
| 67 | + single-label hosts and NAT64-embedded private IPv4 are blocked. | |
| 68 | +- Raw evidence is immutable: snapshots and diffs are never rewritten. Retention may drop the RAW body of | |
| 69 | + snapshots that are not referenced by any event (canonical form and hashes stay); event snapshots are kept forever. | |
| 70 | + Re-interpretation creates a new `interpretations` row. | |
| 71 | +- Noise never becomes an event: changes classified cosmetic / navigation / timestamp / advertisement / | |
| 72 | + boilerplate are stored as changes only. Routine batches from firehose feeds are damped. Ingestion is idempotent | |
| 73 | + (`events.fingerprint` = sensor + before/after canonical hashes). | |
| 74 | +- Silent change = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs, | |
| 75 | + shutdown, feature removed, leadership, page removed, crawler policy…) · no matching announcement within 12 h · | |
| 76 | + importance ≥ `WS_SILENT_MIN_IMPORTANCE` · not a noise class. Never asserted as "unannounced" otherwise. | |
| 77 | +- Never label inference as fact: `evidence_label` OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED; AI text is | |
| 78 | + labelled analysis; `score_reasons` explain every signal score. First-party evidence outweighs media reports | |
| 79 | + (`sources.first_party`, media seeds carry `first_party: false`). | |
| 80 | +- Entity aliases that are ordinary words (`first`, `has`, `who`, `make`…) only match as exact upper-case acronyms | |
| 81 | + in text (`AMBIGUOUS_ALIASES` in `apps/engine/src/entities.ts`); run `cli.ts relink-entities 7` after changing | |
| 82 | + alias rules. | |
| 83 | +- LLM cost control: heuristics first; Claude only above `WS_LLM_MIN_IMPORTANCE`, daily budget, strict JSON output; | |
| 84 | + `llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works. | |
| 85 | +- Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes | |
| 86 | + `page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks. | |
| 87 | +- Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public | |
| 88 | + feeds (`sources.kind = 'custom'`). | |
| 89 | +- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…). | |
| 80 | 90 | |
| 81 | 91 | ## Dev |
| 82 | 92 | ``` |
| 83 | 93 | createdb websensor && cp .env.example .env |
| 84 | 94 | pnpm install |
| 85 | −pnpm db:migrate # or let the engine migrate on start | |
| 86 | −npx tsx apps/engine/src/cli.ts sync # registry → DB | |
| 95 | +pnpm db:migrate # or let the engine migrate on start | |
| 96 | +npx tsx apps/engine/src/cli.ts sync # registry → DB | |
| 87 | 97 | npx tsx apps/engine/src/cli.ts run-due 50 |
| 88 | −pnpm dev:api · pnpm dev:engine · pnpm dev:web | |
| 98 | +pnpm dev:api · pnpm dev:engine · pnpm dev:web # open the site through the gateway (:8260) so /api is same-origin | |
| 89 | 99 | pnpm test · pnpm typecheck |
| 90 | 100 | ``` |
| 91 | −Run CLI/engine from the repo root (config paths are relative to cwd). | |
| 101 | +Run CLI/engine from the repo root. A copy of production data for UI work: | |
| 102 | +`ssh M4M64b 'pg_dump websensor --data-only -t events -t …' | psql websensor` then `cli.ts sync`. | |
| 92 | 103 | |
| 93 | 104 | ## Deploy (MacLustr) |
| 94 | −`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `deploy/websensor.mld.json` (secrets only | |
| 95 | −on M1M32). Processes: `websensor-api` (:8260, ngrok www.websensor.io), `websensor-web` (:8261 loopback), | |
| 96 | −`websensor-engine` (metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. | |
| 97 | −See `deploy/README.md`. | |
| 105 | +`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json` | |
| 106 | +(secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260, | |
| 107 | +public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine` | |
| 108 | +(metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that | |
| 109 | +changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See | |
| 110 | +`deploy/README.md`. | |
modified
apps/api/package.json
+2 −0
@@ -14,6 +14,7 @@ | ||
| 14 | 14 | "@fastify/rate-limit": "^10.3.0", |
| 15 | 15 | "@fastify/reply-from": "^12.6.5", |
| 16 | 16 | "@fastify/websocket": "^11.2.0", |
| 17 | + "@websensor/connectors": "workspace:*", | |
| 17 | 18 | "@websensor/core": "workspace:*", |
| 18 | 19 | "@websensor/db": "workspace:*", |
| 19 | 20 | "@websensor/store": "workspace:*", |
@@ -24,6 +25,7 @@ | ||
| 24 | 25 | "prom-client": "^15.1.0", |
| 25 | 26 | "tsx": "^4.20.0", |
| 26 | 27 | "ws": "^8.21.3", |
| 28 | + "yaml": "^2.8.0", | |
| 27 | 29 | "zod": "^4.0.0" |
| 28 | 30 | }, |
| 29 | 31 | "devDependencies": { |
added
apps/api/src/cache.ts
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +/** | |
| 2 | + * Tiny in-process TTL cache with request coalescing (spec §59). Used for aggregates only | |
| 3 | + * (stats, trending, pulse, radar, rankings) — never for the live feed itself. | |
| 4 | + */ | |
| 5 | +const store = new Map<string, { at: number; ttl: number; value: unknown; pending?: Promise<unknown> }>(); | |
| 6 | + | |
| 7 | +export async function cached<T>(key: string, ttlMs: number, fn: () => Promise<T>): Promise<T> { | |
| 8 | + const now = Date.now(); | |
| 9 | + const hit = store.get(key); | |
| 10 | + if (hit && now - hit.at < hit.ttl) return hit.value as T; | |
| 11 | + if (hit?.pending) return hit.pending as Promise<T>; | |
| 12 | + const pending = fn() | |
| 13 | + .then((value) => { | |
| 14 | + store.set(key, { at: Date.now(), ttl: ttlMs, value }); | |
| 15 | + return value; | |
| 16 | + }) | |
| 17 | + .catch((e) => { | |
| 18 | + // serve stale on failure when we have anything at all | |
| 19 | + if (hit) { | |
| 20 | + store.set(key, { ...hit, pending: undefined }); | |
| 21 | + return hit.value as T; | |
| 22 | + } | |
| 23 | + store.delete(key); | |
| 24 | + throw e; | |
| 25 | + }); | |
| 26 | + store.set(key, { at: hit?.at ?? 0, ttl: hit?.ttl ?? ttlMs, value: hit?.value, pending }); | |
| 27 | + return pending; | |
| 28 | +} | |
| 29 | + | |
| 30 | +export function cacheStats(): { entries: number; keys: string[] } { | |
| 31 | + return { entries: store.size, keys: [...store.keys()].slice(0, 50) }; | |
| 32 | +} | |
| 33 | + | |
| 34 | +export function invalidate(prefix: string): void { | |
| 35 | + for (const k of store.keys()) if (k.startsWith(prefix)) store.delete(k); | |
| 36 | +} | |
modified
apps/api/src/config.ts
+5 −1
@@ -10,5 +10,9 @@ export const config = { | ||
| 10 | 10 | publicBaseUrl: env.PUBLIC_BASE_URL ?? "http://localhost:8260", |
| 11 | 11 | canonicalHost: env.CANONICAL_HOST ?? "www.websensor.io", |
| 12 | 12 | redirectApexToWww: (env.WS_REDIRECT_APEX ?? "1") !== "0", |
| 13 | − version: "0.1.0", | |
| 13 | + /** Admin endpoints (`/api/v1/admin/*`) are enabled only when this token is set; sent as `X-WebSensor-Admin`. */ | |
| 14 | + adminToken: env.WS_ADMIN_TOKEN ?? "", | |
| 15 | + /** Custom URL monitors per anonymous owner (spec §45). */ | |
| 16 | + monitorsPerOwner: Number(env.WS_MONITORS_PER_OWNER ?? 5), | |
| 17 | + version: "0.2.0", | |
| 14 | 18 | }; |
added
apps/api/src/intel.ts
+238 −0
@@ -0,0 +1,238 @@ | ||
| 1 | +import { COUNTRIES, dailyAnomaly, entityRankScore, EVENT_GROUPS, FEED_CHANNELS } from "@websensor/core"; | |
| 2 | +import { db, sql, textArray } from "@websensor/db"; | |
| 3 | +import { cached } from "./cache"; | |
| 4 | +import { EVENT_SELECT, listEvents } from "./queries"; | |
| 5 | + | |
| 6 | +/** | |
| 7 | + * Intelligence read-models (spec §34–39, §101–105): breaking desk, pulse, radar, entity insights, | |
| 8 | + * rankings, cluster propagation, country and category desks. Aggregates are cached briefly. | |
| 9 | + */ | |
| 10 | + | |
| 11 | +const CLUSTER_SELECT = sql`c.id, c.slug, c.title, c.summary, c.primary_event_id, c.entity_ids, c.categories, c.event_count, c.max_importance, c.first_at, c.last_at, c.source_count, c.first_party_count, c.external_count, c.velocity, c.state, c.lead_time_ms, c.first_party_at, c.first_external_at`; | |
| 12 | + | |
| 13 | +async function clusterPrimaryEvents(where: ReturnType<typeof sql>, order: ReturnType<typeof sql>, limit: number): Promise<Record<string, unknown>[]> { | |
| 14 | + const rows = await db.execute<Record<string, unknown>>(sql` | |
| 15 | + select ${CLUSTER_SELECT}, | |
| 16 | + (select row_to_json(x) from (select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.id = c.primary_event_id) x) as event, | |
| 17 | + (select json_agg(json_build_object('id', s2.id, 'name', s2.name, 'domain', s2.domain, 'first_party', s2.first_party) order by s2.name) from (select distinct s3.id, s3.name, s3.domain, s3.first_party from events e3 join sources s3 on s3.id = e3.source_id where e3.cluster_id = c.id limit 12) s2) as sources | |
| 18 | + from event_clusters c where ${where} order by ${order} limit ${limit}`); | |
| 19 | + return rows.rows; | |
| 20 | +} | |
| 21 | + | |
| 22 | +export async function breakingDesk(): Promise<Record<string, unknown>> { | |
| 23 | + return cached("breaking", 8_000, async () => { | |
| 24 | + const [breaking, developing, confirmed, watching] = await Promise.all([ | |
| 25 | + clusterPrimaryEvents(sql`c.state = 'breaking' and c.last_at >= now() - interval '24 hours'`, sql`c.max_importance desc, c.velocity desc, c.last_at desc`, 20), | |
| 26 | + clusterPrimaryEvents(sql`c.state = 'developing' and c.last_at >= now() - interval '24 hours'`, sql`c.velocity desc, c.max_importance desc, c.last_at desc`, 20), | |
| 27 | + clusterPrimaryEvents(sql`c.state = 'confirmed' and c.last_at >= now() - interval '48 hours'`, sql`c.last_at desc`, 20), | |
| 28 | + listEvents({ limit: 25, order: "signal", signal_min: 60, after: new Date(Date.now() - 12 * 3600e3).toISOString() }).then((r) => r.items.filter((e) => !["breaking", "developing", "confirmed"].includes(String((e.cluster as { state?: string } | null)?.state ?? "")))), | |
| 29 | + ]); | |
| 30 | + return { breaking_now: breaking, developing, recently_confirmed: confirmed, watching, generated_at: new Date().toISOString() }; | |
| 31 | + }); | |
| 32 | +} | |
| 33 | + | |
| 34 | +export async function pulse(): Promise<Record<string, unknown>> { | |
| 35 | + return cached("pulse", 10_000, async () => { | |
| 36 | + const desks = ["ai", "cyber", "finance", "government", "infrastructure", "health", "science", "products"]; | |
| 37 | + const [activity, byDesk, rising, anomalies, silent, incidents, breaking, groups, totals] = await Promise.all([ | |
| 38 | + db.execute<Record<string, unknown>>(sql` | |
| 39 | + with b as (select generate_series(date_trunc('hour', now() at time zone 'UTC') - interval '6 hours', date_trunc('minute', now() at time zone 'UTC'), interval '15 minutes') as bucket), | |
| 40 | + ev as (select to_timestamp(floor(extract(epoch from detected_at) / 900) * 900) at time zone 'UTC' as bk, count(*) as n from events where detected_at >= now() - interval '7 hours' group by 1), | |
| 41 | + ch as (select to_timestamp(floor(extract(epoch from detected_at) / 900) * 900) at time zone 'UTC' as bk, count(*) as n from changes where detected_at >= now() - interval '7 hours' group by 1) | |
| 42 | + select to_char(b.bucket, 'YYYY-MM-DD"T"HH24:MI:00"Z"') as t, coalesce(ev.n, 0)::int as events, coalesce(ch.n, 0)::int as changes | |
| 43 | + from b left join ev on ev.bk = b.bucket left join ch on ch.bk = b.bucket order by b.bucket`).then((r) => r.rows).catch(() => []), | |
| 44 | + Promise.all(desks.map(async (d) => ({ desk: d, items: (await listEvents({ limit: 5, order: "signal", category: d, after: new Date(Date.now() - 12 * 3600e3).toISOString() })).items }))), | |
| 45 | + db.execute<Record<string, unknown>>(sql` | |
| 46 | + with cur as (select ee.entity_id, count(*) n from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - interval '3 hours' group by 1), | |
| 47 | + prev as (select ee.entity_id, count(*) n from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - interval '27 hours' and e.detected_at < now() - interval '3 hours' group by 1) | |
| 48 | + select en.id, en.name, en.type, cur.n::int as events_3h, coalesce(prev.n,0)::int as events_prev_24h, round((cur.n::float / greatest(0.125, coalesce(prev.n,0)/8.0))::numeric, 1)::float as acceleration | |
| 49 | + from cur join entities en on en.id = cur.entity_id left join prev on prev.entity_id = cur.entity_id where cur.n >= 2 order by acceleration desc, cur.n desc limit 10`).then((r) => r.rows), | |
| 50 | + db.execute<Record<string, unknown>>(sql` | |
| 51 | + with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '2 hours' group by s.source_id), | |
| 52 | + base as (select s.source_id, count(*)::float / (14*24) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id) | |
| 53 | + select so.id, so.name, so.domain, cur.n::int as changes_2h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day, | |
| 54 | + round((case when coalesce(base.per_hour,0) = 0 then (case when cur.n/2.0 > 2 then 70 else 40 end) else least(100, case when cur.n/2.0/base.per_hour <= 1 then cur.n/2.0/base.per_hour*30 else 30 + 25*(ln(cur.n/2.0/base.per_hour)/ln(2)) end) end)::numeric, 1)::float as activity_score, | |
| 55 | + (case when coalesce(base.per_hour,0) > 0 then round(((cur.n/2.0/base.per_hour - 1) * 100)::numeric) else null end)::int as pct_vs_baseline | |
| 56 | + from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id where so.kind = 'registry' order by activity_score desc limit 8`).then((r) => r.rows), | |
| 57 | + listEvents({ limit: 8, silent_change: true, order: "signal", after: new Date(Date.now() - 24 * 3600e3).toISOString() }).then((r) => r.items), | |
| 58 | + listEvents({ limit: 8, group: "reliability", order: "recent", after: new Date(Date.now() - 6 * 3600e3).toISOString() }).then((r) => r.items), | |
| 59 | + clusterPrimaryEvents(sql`c.state in ('breaking','developing') and c.last_at >= now() - interval '24 hours'`, sql`(c.state = 'breaking') desc, c.max_importance desc, c.velocity desc`, 8), | |
| 60 | + db.execute<Record<string, unknown>>(sql`select e.event_type, count(*)::int as n from events e where e.detected_at >= now() - interval '24 hours' group by 1`).then((r) => { | |
| 61 | + const byGroup: Record<string, number> = {}; | |
| 62 | + for (const row of r.rows) { | |
| 63 | + const g = Object.entries(EVENT_GROUPS).find(([, spec]) => spec.types.includes(String(row.event_type)))?.[0] ?? "web"; | |
| 64 | + byGroup[g] = (byGroup[g] ?? 0) + Number(row.n); | |
| 65 | + } | |
| 66 | + return byGroup; | |
| 67 | + }), | |
| 68 | + db.execute<Record<string, unknown>>(sql`select (select count(*) from events where detected_at >= now() - interval '1 hour')::int as events_1h, (select count(*) from changes where detected_at >= now() - interval '1 hour')::int as changes_1h, (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes')::int as checks_5m, (select count(distinct source_id) from events where detected_at >= now() - interval '24 hours')::int as active_sources_24h, (select count(distinct country) from events where detected_at >= now() - interval '24 hours' and country is not null)::int as active_countries_24h`).then((r) => r.rows[0]), | |
| 69 | + ]); | |
| 70 | + return { activity, desks: byDesk, rising_entities: rising, anomalies, silent_changes: silent, infrastructure: incidents, breaking, by_group_24h: groups, totals, generated_at: new Date().toISOString() }; | |
| 71 | + }); | |
| 72 | +} | |
| 73 | + | |
| 74 | +/** Weak signals (spec §101): things that are NOT breaking yet but could become important. Clearly labelled as indicators. */ | |
| 75 | +export async function radar(): Promise<Record<string, unknown>> { | |
| 76 | + return cached("radar", 30_000, async () => { | |
| 77 | + const [quietBursts, silentClusters, docBursts, repoBursts, statusChanges, developing, newSensorsFiring] = await Promise.all([ | |
| 78 | + // Sources far above their baseline but without any high-importance event | |
| 79 | + db.execute<Record<string, unknown>>(sql` | |
| 80 | + with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '3 hours' group by s.source_id), | |
| 81 | + base as (select s.source_id, count(*)::float / (14*24) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id), | |
| 82 | + hot as (select distinct source_id from events where detected_at >= now() - interval '6 hours' and coalesce(signal_score, importance) >= 75) | |
| 83 | + select so.id, so.name, so.domain, so.categories, cur.n::int as changes_3h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day, | |
| 84 | + round((cur.n/3.0 / greatest(0.02, coalesce(base.per_hour, 0)))::numeric, 1)::float as ratio | |
| 85 | + from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id | |
| 86 | + where so.kind = 'registry' and cur.n >= 4 and cur.source_id not in (select source_id from hot) and (coalesce(base.per_hour,0) = 0 or cur.n/3.0 / base.per_hour >= 4) | |
| 87 | + order by ratio desc limit 12`).then((r) => r.rows), | |
| 88 | + // Entities with several silent changes in 24 h | |
| 89 | + db.execute<Record<string, unknown>>(sql` | |
| 90 | + select en.id, en.name, en.type, count(*)::int as silent_24h, array_agg(distinct e.event_type) as types, max(e.detected_at) as last_at | |
| 91 | + from events e join event_entities ee on ee.event_id = e.id join entities en on en.id = ee.entity_id | |
| 92 | + where e.silent_change and e.detected_at >= now() - interval '24 hours' group by en.id, en.name, en.type having count(*) >= 2 order by silent_24h desc, last_at desc limit 12`).then((r) => r.rows), | |
| 93 | + // Documentation / API modification bursts | |
| 94 | + db.execute<Record<string, unknown>>(sql` | |
| 95 | + select s.id, s.name, s.domain, count(*)::int as doc_changes_6h, array_agg(distinct e.event_type) as types, max(e.detected_at) as last_at | |
| 96 | + from events e join sources s on s.id = e.source_id where e.detected_at >= now() - interval '6 hours' and e.event_type in ('documentation_change','API_change','api_change','availability_change') | |
| 97 | + group by s.id, s.name, s.domain having count(*) >= 3 order by doc_changes_6h desc limit 10`).then((r) => r.rows), | |
| 98 | + // Repository activity bursts (commits/tags/releases) | |
| 99 | + db.execute<Record<string, unknown>>(sql` | |
| 100 | + select s.id, s.name, s.domain, count(*)::int as repo_events_6h, max(e.detected_at) as last_at | |
| 101 | + from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.detected_at >= now() - interval '6 hours' and sen.connector = 'github' | |
| 102 | + group by s.id, s.name, s.domain having count(*) >= 3 order by repo_events_6h desc limit 10`).then((r) => r.rows), | |
| 103 | + // Fresh status-page changes below the breaking bar | |
| 104 | + listEvents({ limit: 10, group: "reliability", after: new Date(Date.now() - 3 * 3600e3).toISOString() }).then((r) => r.items.filter((e) => Number(e.importance) < 80)), | |
| 105 | + clusterPrimaryEvents(sql`c.state = 'developing' and c.last_at >= now() - interval '12 hours'`, sql`c.velocity desc, c.last_at desc`, 8), | |
| 106 | + // Sensors that produced their first-ever event in the last 24 h (new coverage lighting up) | |
| 107 | + db.execute<Record<string, unknown>>(sql` | |
| 108 | + select sen.id, sen.name, sen.source_id, s.name as source_name, min(e.detected_at) as first_event_at, count(*)::int as events | |
| 109 | + from events e join sensors sen on sen.id = e.sensor_id join sources s on s.id = sen.source_id | |
| 110 | + where e.detected_at >= now() - interval '24 hours' and not exists (select 1 from events e2 where e2.sensor_id = sen.id and e2.detected_at < now() - interval '24 hours') | |
| 111 | + group by sen.id, sen.name, sen.source_id, s.name order by events desc limit 10`).then((r) => r.rows), | |
| 112 | + ]); | |
| 113 | + return { unusual_source_activity: quietBursts, silent_clusters: silentClusters, documentation_bursts: docBursts, repository_bursts: repoBursts, status_changes: statusChanges, developing, new_coverage: newSensorsFiring, generated_at: new Date().toISOString(), disclaimer: "Indicators, not facts: each item is a pattern in raw observations that has not (yet) produced a high-importance event." }; | |
| 114 | + }); | |
| 115 | +} | |
| 116 | + | |
| 117 | +/** Entity insights (spec §26, §38, §102): heatmap, baseline, anomaly, velocity, rank, most active sensors. */ | |
| 118 | +export async function entityInsights(entityId: string): Promise<Record<string, unknown>> { | |
| 119 | + const [heat, today, last24, prev24, sensorsTop, silent24, breaking24, sources24, rank] = await Promise.all([ | |
| 120 | + db.execute<Record<string, unknown>>(sql`select day::text as day, events, silent, breaking, max_importance from entity_daily where entity_id = ${entityId} and day >= (now() at time zone 'UTC')::date - 34 order by day`).then((r) => r.rows), | |
| 121 | + db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.detected_at >= date_trunc('day', now() at time zone 'UTC')`).then((r) => Number(r.rows[0]?.n ?? 0)), | |
| 122 | + db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.detected_at >= now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)), | |
| 123 | + db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.detected_at >= now() - interval '48 hours' and e.detected_at < now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)), | |
| 124 | + db.execute<Record<string, unknown>>(sql`select sen.id, sen.name, sen.type, sen.connector, sen.source_id, count(*)::int as events_7d, max(e.detected_at) as last_event_at from events e join event_entities ee on ee.event_id = e.id join sensors sen on sen.id = e.sensor_id where ee.entity_id = ${entityId} and e.detected_at >= now() - interval '7 days' group by sen.id, sen.name, sen.type, sen.connector, sen.source_id order by events_7d desc limit 8`).then((r) => r.rows), | |
| 125 | + db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.silent_change and e.detected_at >= now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)), | |
| 126 | + db.execute<{ n: string }>(sql`select count(*)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and coalesce(e.signal_score, e.importance) >= 80 and e.detected_at >= now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)), | |
| 127 | + db.execute<{ n: string }>(sql`select count(distinct e.source_id)::text as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${entityId} and e.detected_at >= now() - interval '24 hours'`).then((r) => Number(r.rows[0]?.n ?? 0)), | |
| 128 | + entityRank(entityId), | |
| 129 | + ]); | |
| 130 | + const baselineDays = heat.filter((d) => String(d.day) < new Date().toISOString().slice(0, 10)); | |
| 131 | + const baselinePerDay = baselineDays.length ? baselineDays.reduce((n, d) => n + Number(d.events), 0) / Math.max(baselineDays.length, 7) : 0; | |
| 132 | + // Rolling 24 h vs the 30-day daily baseline (a UTC "today" window is misleading in the first hours of the day). | |
| 133 | + const anomaly = dailyAnomaly(last24, baselinePerDay, 24); | |
| 134 | + const velocity = last24 && prev24 ? Math.round((last24 / prev24) * 100) / 100 : last24 ? 2 : 0; | |
| 135 | + // Fill the 35-day heatmap with zero days | |
| 136 | + const map = new Map(heat.map((d) => [String(d.day), d])); | |
| 137 | + const days: Record<string, unknown>[] = []; | |
| 138 | + for (let i = 34; i >= 0; i--) { | |
| 139 | + const d = new Date(Date.now() - i * 86400e3).toISOString().slice(0, 10); | |
| 140 | + days.push(map.get(d) ?? { day: d, events: 0, silent: 0, breaking: 0, max_importance: 0 }); | |
| 141 | + } | |
| 142 | + return { heatmap: days, baseline_per_day: Math.round(baselinePerDay * 10) / 10, today, events_24h: last24, events_prev_24h: prev24, velocity_ratio: velocity, anomaly, silent_24h: silent24, breaking_24h: breaking24, sources_24h: sources24, most_active_sensors: sensorsTop, rank }; | |
| 143 | +} | |
| 144 | + | |
| 145 | +/** WebSensor entity ranking (spec §105): computed over entities active in the last 7 days, cached 2 minutes. */ | |
| 146 | +export async function entityRankings(limit = 100): Promise<Record<string, unknown>[]> { | |
| 147 | + return cached(`rank:${limit}`, 120_000, async () => { | |
| 148 | + const rows = await db.execute<Record<string, unknown>>(sql` | |
| 149 | + with agg as ( | |
| 150 | + select ee.entity_id, | |
| 151 | + count(*) filter (where e.detected_at >= now() - interval '24 hours') as e24, | |
| 152 | + count(*) as e7, | |
| 153 | + avg(coalesce(e.signal_score, e.importance)) as avg_signal, | |
| 154 | + avg(case when e.evidence_label = 'CONFIRMED' then 1 else 0 end) as confirmed_ratio, | |
| 155 | + count(distinct e.source_id) as sources, | |
| 156 | + count(*) filter (where e.silent_change and e.detected_at >= now() - interval '24 hours') as silent24, | |
| 157 | + count(*) filter (where coalesce(e.signal_score, e.importance) >= 80 and e.detected_at >= now() - interval '24 hours') as breaking24, | |
| 158 | + max(e.detected_at) as last_at | |
| 159 | + from events e join event_entities ee on ee.event_id = e.id join sources s on s.id = e.source_id where s.kind = 'registry' and e.detected_at >= now() - interval '7 days' group by ee.entity_id), | |
| 160 | + base as (select entity_id, avg(events)::float as per_day from entity_daily where day >= (now() at time zone 'UTC')::date - 30 and day < (now() at time zone 'UTC')::date group by entity_id) | |
| 161 | + select en.id, en.name, en.type, en.domain, en.importance, agg.e24::int as events_24h, agg.e7::int as events_7d, round(agg.avg_signal::numeric,1)::float as avg_signal, round(agg.confirmed_ratio::numeric,2)::float as confirmed_ratio, agg.sources::int as sources, agg.silent24::int as silent_24h, agg.breaking24::int as breaking_24h, agg.last_at, coalesce(base.per_day,0)::float as baseline_per_day | |
| 162 | + from agg join entities en on en.id = agg.entity_id left join base on base.entity_id = agg.entity_id where en.type <> 'person'`); | |
| 163 | + const scored = rows.rows.map((r) => ({ ...r, rank_score: entityRankScore({ importance: Number(r.importance), events24h: Number(r.events_24h), events7d: Number(r.events_7d), avgSignal: Number(r.avg_signal), confirmedRatio: Number(r.confirmed_ratio), uniqueSources: Number(r.sources), baselinePerDay: Number(r.baseline_per_day) }) })); | |
| 164 | + scored.sort((a, b) => b.rank_score - a.rank_score); | |
| 165 | + return scored.slice(0, limit).map((r, i) => ({ ...r, rank: i + 1 })); | |
| 166 | + }); | |
| 167 | +} | |
| 168 | + | |
| 169 | +export async function entityRank(entityId: string): Promise<{ rank: number | null; total: number; score: number | null }> { | |
| 170 | + const all = await entityRankings(2000); | |
| 171 | + const i = all.findIndex((r) => r.id === entityId); | |
| 172 | + return { rank: i >= 0 ? i + 1 : null, total: all.length, score: i >= 0 ? Number(all[i]!.rank_score) : null }; | |
| 173 | +} | |
| 174 | + | |
| 175 | +export async function clusterDetail(idOrSlug: string): Promise<Record<string, unknown> | null> { | |
| 176 | + const c = (await db.execute<Record<string, unknown>>(sql`select ${CLUSTER_SELECT}, c.timeline from event_clusters c where c.id = ${idOrSlug} or c.slug = ${idOrSlug} limit 1`)).rows[0]; | |
| 177 | + if (!c) return null; | |
| 178 | + const events = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.cluster_id = ${String(c.id)} order by e.detected_at asc limit 200`); | |
| 179 | + const ents = (c.entity_ids as string[]).length ? await db.execute<Record<string, unknown>>(sql`select id, name, type, importance from entities where id = any(${textArray(c.entity_ids as string[])}) order by importance desc`) : { rows: [] }; | |
| 180 | + // Propagation timeline (spec §35): offsets from the first signal. | |
| 181 | + const first = new Date(String(c.first_at)).getTime(); | |
| 182 | + const propagation = events.rows.map((e) => ({ id: e.id, slug: e.slug, at: e.detected_at, offset_ms: new Date(String(e.detected_at)).getTime() - first, source: e.source, sensor: e.sensor, first_party: e.first_party, event_type: e.event_type, importance: e.importance, title: e.title })); | |
| 183 | + const firstExternal = events.rows.find((e) => e.first_party === false); | |
| 184 | + const firstParty = events.rows.find((e) => e.first_party !== false); | |
| 185 | + const leadTime = firstParty && firstExternal ? new Date(String(firstExternal.detected_at)).getTime() - new Date(String(firstParty.detected_at)).getTime() : null; | |
| 186 | + return { cluster: c, events: events.rows, entities: ents.rows, propagation, lead_time_ms: leadTime ?? c.lead_time_ms ?? null, first_party_signals: events.rows.filter((e) => e.first_party !== false).length, external_signals: events.rows.filter((e) => e.first_party === false).length }; | |
| 187 | +} | |
| 188 | + | |
| 189 | +export async function countryList(): Promise<Record<string, unknown>[]> { | |
| 190 | + return cached("countries", 60_000, async () => { | |
| 191 | + const rows = await db.execute<Record<string, unknown>>(sql` | |
| 192 | + select s.country, count(distinct s.id)::int as sources, (select count(*) from events e where e.country = s.country and e.detected_at >= now() - interval '24 hours')::int as events_24h, | |
| 193 | + (select count(*) from events e where e.country = s.country and e.detected_at >= now() - interval '24 hours' and coalesce(e.signal_score, e.importance) >= 80)::int as breaking_24h | |
| 194 | + from sources s where s.country is not null and s.enabled and s.kind = 'registry' group by s.country order by events_24h desc, sources desc`); | |
| 195 | + return rows.rows.map((r) => ({ ...r, name: COUNTRIES[String(r.country)]?.name ?? String(r.country), slug: COUNTRIES[String(r.country)]?.slug ?? String(r.country).toLowerCase(), flag: COUNTRIES[String(r.country)]?.flag ?? "" })); | |
| 196 | + }); | |
| 197 | +} | |
| 198 | + | |
| 199 | +export async function countryDesk(code: string): Promise<Record<string, unknown>> { | |
| 200 | + const c = code.toUpperCase(); | |
| 201 | + const cats = ["government", "finance", "infrastructure", "health", "news", "cyber", "ai", "science"]; | |
| 202 | + const [breaking, byCategory, sources, byType, silent, recent] = await Promise.all([ | |
| 203 | + listEvents({ limit: 10, country: c, order: "signal", after: new Date(Date.now() - 48 * 3600e3).toISOString() }).then((r) => r.items), | |
| 204 | + Promise.all(cats.map(async (cat) => ({ category: cat, items: (await listEvents({ limit: 6, country: c, category: cat })).items }))).then((x) => x.filter((d) => d.items.length)), | |
| 205 | + db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier, s.categories, s.first_party, (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count, (select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h from sources s where s.country = ${c} and s.enabled and s.kind = 'registry' order by events_24h desc, s.tier, s.name limit 200`).then((r) => r.rows), | |
| 206 | + db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where country = ${c} and detected_at >= now() - interval '7 days' group by 1 order by 2 desc limit 20`).then((r) => r.rows), | |
| 207 | + listEvents({ limit: 8, country: c, silent_change: true }).then((r) => r.items), | |
| 208 | + listEvents({ limit: 40, country: c }), | |
| 209 | + ]); | |
| 210 | + return { country: { code: c, name: COUNTRIES[c]?.name ?? c, flag: COUNTRIES[c]?.flag ?? "" }, breaking, by_category: byCategory, sources, by_type: byType, silent, recent: recent.items, nextCursor: recent.nextCursor }; | |
| 211 | +} | |
| 212 | + | |
| 213 | +export async function categoryDesk(channel: string): Promise<Record<string, unknown>> { | |
| 214 | + const cats = FEED_CHANNELS[channel] ?? [channel]; | |
| 215 | + const [breaking, silent, activeSources, byType, trendingEntities, recent, series] = await Promise.all([ | |
| 216 | + listEvents({ limit: 8, category: channel, order: "signal", after: new Date(Date.now() - 24 * 3600e3).toISOString() }).then((r) => r.items), | |
| 217 | + listEvents({ limit: 8, category: channel, silent_change: true }).then((r) => r.items), | |
| 218 | + db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier, s.first_party, count(*)::int as events_24h, max(e.importance)::float as max_importance from events e join sources s on s.id = e.source_id where ${channel} = any(e.categories) and e.detected_at >= now() - interval '24 hours' and s.kind = 'registry' group by s.id, s.name, s.domain, s.tier, s.first_party order by events_24h desc limit 12`).then((r) => r.rows), | |
| 219 | + db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where ${channel} = any(categories) and detected_at >= now() - interval '7 days' group by 1 order by 2 desc limit 16`).then((r) => r.rows), | |
| 220 | + db.execute<Record<string, unknown>>(sql`select en.id, en.name, en.type, count(*)::int as events_24h, count(distinct e.source_id)::int as sources from events e join event_entities ee on ee.event_id = e.id join entities en on en.id = ee.entity_id where ${channel} = any(e.categories) and e.detected_at >= now() - interval '24 hours' group by en.id, en.name, en.type order by events_24h desc limit 10`).then((r) => r.rows), | |
| 221 | + listEvents({ limit: 60, category: channel }), | |
| 222 | + db.execute<Record<string, unknown>>(sql`select to_char(date_trunc('hour', detected_at at time zone 'UTC'), 'YYYY-MM-DD"T"HH24:00:00"Z"') as t, count(*)::int as n from events where ${channel} = any(categories) and detected_at >= now() - interval '48 hours' group by 1 order by 1`).then((r) => r.rows), | |
| 223 | + ]); | |
| 224 | + return { channel, categories: cats, breaking, silent, active_sources: activeSources, by_type: byType, trending_entities: trendingEntities, recent: recent.items, nextCursor: recent.nextCursor, series }; | |
| 225 | +} | |
| 226 | + | |
| 227 | +/** Sensors table for a source, with polling metadata the source page shows (spec §27). */ | |
| 228 | +export async function sourceSensors(sourceId: string): Promise<Record<string, unknown>[]> { | |
| 229 | + const r = await db.execute<Record<string, unknown>>(sql` | |
| 230 | + select id, name, url, type, connector, tier, health, status, priority, enabled, next_check_at, last_check_at, last_change_at, last_event_at, last_status, last_error, consecutive_errors, total_runs, total_not_modified, raw_changes, meaningful_changes, avg_latency_ms, base_interval_seconds, validated_at, | |
| 231 | + etag is not null as has_etag, last_modified is not null as has_last_modified, etag, last_modified, | |
| 232 | + (select count(*) from sensor_runs r where r.sensor_id = sensors.id and r.started_at >= now() - interval '24 hours')::int as checks_24h, | |
| 233 | + (select count(*) from changes c where c.sensor_id = sensors.id and c.detected_at >= now() - interval '24 hours')::int as changes_24h, | |
| 234 | + (select count(*) from snapshots sn where sn.sensor_id = sensors.id)::int as snapshot_count, | |
| 235 | + case when last_check_at is not null and next_check_at is not null then extract(epoch from (next_check_at - last_check_at))::int else null end as current_interval_seconds | |
| 236 | + from sensors where source_id = ${sourceId} order by priority, tier, name`); | |
| 237 | + return r.rows; | |
| 238 | +} | |
modified
apps/api/src/live.ts
+90 −19
@@ -1,34 +1,63 @@ | ||
| 1 | 1 | import type { FastifyInstance } from "fastify"; |
| 2 | 2 | import type { WebSocket } from "ws"; |
| 3 | 3 | import Redis from "ioredis"; |
| 4 | −import { FEED_CHANNELS } from "@websensor/core"; | |
| 4 | +import { eventGroupOf, FEED_CHANNELS } from "@websensor/core"; | |
| 5 | 5 | import { db, sql } from "@websensor/db"; |
| 6 | 6 | import { config } from "./config"; |
| 7 | 7 | |
| 8 | 8 | /** |
| 9 | 9 | * WebSocket gateway `/api/v1/live`. One Redis subscriber fans out to every client; clients |
| 10 | − * pick channels: events:global · events:breaking · events:<ai|cyber|finance|health|government| | |
| 11 | − * science|products|infrastructure> · entity:<id> · source:<id> · watchlist:<id>. | |
| 12 | − * Protocol (JSON): client → {"subscribe":[…]} | {"unsubscribe":[…]} | {"ping":1} | |
| 13 | − * server → {"type":"hello"} | {"type":"event", "channels":[…], "event":{…}} | {"type":"pong"} | |
| 10 | + * pick channels: events:global · events:breaking · events:silent · events:first-party · | |
| 11 | + * events:<ai|cyber|finance|health|government|science|products|infrastructure|news> · group:<security|…> · | |
| 12 | + * country:<CA> · type:<event_type> · entity:<id> · source:<id> · watchlist:<id>. | |
| 13 | + * Protocol (JSON): client → {"subscribe":[…]} | {"unsubscribe":[…]} | {"ping":1} | {"since":"<sid>"} | |
| 14 | + * server → {"type":"hello"} | {"type":"event", "sid":"…", "channels":[…], "event":{…}} | {"type":"replay_done"} | {"type":"pong"} | {"type":"heartbeat"} | |
| 15 | + * Every event frame carries the Redis stream id `sid`; after a reconnection the client sends | |
| 16 | + * {"since": lastSid} and missed events (up to 500) are replayed from the durable stream (spec §60). | |
| 14 | 17 | */ |
| 15 | 18 | interface Client { |
| 16 | 19 | ws: WebSocket; |
| 17 | 20 | channels: Set<string>; |
| 18 | − watchlists: Map<string, { entities: Set<string>; sources: Set<string>; keywords: string[]; categories: Set<string> }>; | |
| 21 | + watchlists: Map<string, WatchRules>; | |
| 22 | +} | |
| 23 | +interface WatchRules { | |
| 24 | + entities: Set<string>; | |
| 25 | + sources: Set<string>; | |
| 26 | + keywords: string[]; | |
| 27 | + categories: Set<string>; | |
| 28 | + types: Set<string>; | |
| 29 | + countries: Set<string>; | |
| 30 | + urls: string[]; | |
| 19 | 31 | } |
| 20 | 32 | |
| 21 | 33 | const clients = new Set<Client>(); |
| 22 | 34 | let sub: Redis | null = null; |
| 35 | +let cmd: Redis | null = null; | |
| 23 | 36 | let published = 0; |
| 24 | 37 | |
| 25 | 38 | export function liveStats(): { clients: number; published: number } { |
| 26 | 39 | return { clients: clients.size, published }; |
| 27 | 40 | } |
| 28 | 41 | |
| 42 | +/** Engine heartbeat written by the scheduler (`ws:engine:status`). */ | |
| 43 | +export async function engineStatus(): Promise<Record<string, unknown> | null> { | |
| 44 | + try { | |
| 45 | + const raw = await getCmd().get("ws:engine:status"); | |
| 46 | + return raw ? (JSON.parse(raw) as Record<string, unknown>) : null; | |
| 47 | + } catch { | |
| 48 | + return null; | |
| 49 | + } | |
| 50 | +} | |
| 51 | + | |
| 52 | +function getCmd(): Redis { | |
| 53 | + if (!cmd) cmd = new Redis(config.redisUrl, { maxRetriesPerRequest: 2, lazyConnect: false }); | |
| 54 | + return cmd; | |
| 55 | +} | |
| 56 | + | |
| 29 | 57 | export async function registerLive(app: FastifyInstance): Promise<void> { |
| 30 | 58 | sub = new Redis(config.redisUrl, { maxRetriesPerRequest: 3 }); |
| 31 | 59 | sub.on("error", (e) => app.log.warn({ err: e.message }, "redis sub error")); |
| 60 | + getCmd().on("error", (e) => app.log.warn({ err: e.message }, "redis cmd error")); | |
| 32 | 61 | await sub.subscribe("ws:live"); |
| 33 | 62 | sub.on("message", (_ch, msg) => { |
| 34 | 63 | let ev: Record<string, unknown>; |
@@ -38,21 +67,15 @@ export async function registerLive(app: FastifyInstance): Promise<void> { | ||
| 38 | 67 | return; |
| 39 | 68 | } |
| 40 | 69 | published++; |
| 41 | − const chans = channelsFor(ev); | |
| 42 | − for (const c of clients) { | |
| 43 | − const hit = [...chans].filter((ch) => c.channels.has(ch)); | |
| 44 | − for (const [wid, w] of c.watchlists) if (matchesWatchlist(ev, w)) hit.push(`watchlist:${wid}`); | |
| 45 | − if (!hit.length) continue; | |
| 46 | − if (c.ws.readyState === c.ws.OPEN) c.ws.send(JSON.stringify({ type: "event", channels: hit, event: ev })); | |
| 47 | − } | |
| 70 | + deliver(ev, String(ev.sid ?? "")); | |
| 48 | 71 | }); |
| 49 | 72 | |
| 50 | 73 | app.get("/api/v1/live", { websocket: true }, (socket) => { |
| 51 | 74 | const client: Client = { ws: socket, channels: new Set(["events:global"]), watchlists: new Map() }; |
| 52 | 75 | clients.add(client); |
| 53 | − socket.send(JSON.stringify({ type: "hello", channels: [...client.channels], serverTime: new Date().toISOString() })); | |
| 76 | + socket.send(JSON.stringify({ type: "hello", channels: [...client.channels], serverTime: new Date().toISOString(), protocol: 2 })); | |
| 54 | 77 | socket.on("message", async (raw: Buffer | string) => { |
| 55 | − let msg: { subscribe?: string[]; unsubscribe?: string[]; ping?: number }; | |
| 78 | + let msg: { subscribe?: string[]; unsubscribe?: string[]; ping?: number; since?: string }; | |
| 56 | 79 | try { |
| 57 | 80 | msg = JSON.parse(raw.toString()) as typeof msg; |
| 58 | 81 | } catch { |
@@ -68,7 +91,8 @@ export async function registerLive(app: FastifyInstance): Promise<void> { | ||
| 68 | 91 | client.channels.delete(ch); |
| 69 | 92 | if (ch.startsWith("watchlist:")) client.watchlists.delete(ch.slice(10)); |
| 70 | 93 | } |
| 71 | − socket.send(JSON.stringify({ type: "subscribed", channels: [...client.channels, ...[...client.watchlists.keys()].map((w) => `watchlist:${w}`)] })); | |
| 94 | + if (msg.subscribe || msg.unsubscribe) socket.send(JSON.stringify({ type: "subscribed", channels: [...client.channels, ...[...client.watchlists.keys()].map((w) => `watchlist:${w}`)] })); | |
| 95 | + if (typeof msg.since === "string" && /^\d{10,16}-\d{1,6}$/.test(msg.since)) await replay(client, msg.since); | |
| 72 | 96 | }); |
| 73 | 97 | const hb = setInterval(() => { |
| 74 | 98 | if (socket.readyState === socket.OPEN) socket.send(JSON.stringify({ type: "heartbeat", t: Date.now() })); |
@@ -84,37 +108,83 @@ export async function registerLive(app: FastifyInstance): Promise<void> { | ||
| 84 | 108 | }); |
| 85 | 109 | } |
| 86 | 110 | |
| 111 | +function deliver(ev: Record<string, unknown>, sid: string, only?: Client): void { | |
| 112 | + const chans = channelsFor(ev); | |
| 113 | + for (const c of only ? [only] : clients) { | |
| 114 | + const hit = [...chans].filter((ch) => c.channels.has(ch)); | |
| 115 | + for (const [wid, w] of c.watchlists) if (matchesWatchlist(ev, w)) hit.push(`watchlist:${wid}`); | |
| 116 | + if (!hit.length) continue; | |
| 117 | + if (c.ws.readyState === c.ws.OPEN) c.ws.send(JSON.stringify({ type: "event", sid, channels: hit, event: ev })); | |
| 118 | + } | |
| 119 | +} | |
| 120 | + | |
| 121 | +/** Replay missed events from the durable stream (exclusive of `since`). */ | |
| 122 | +async function replay(client: Client, since: string): Promise<void> { | |
| 123 | + try { | |
| 124 | + const [ms, seq] = since.split("-"); | |
| 125 | + const start = `${ms}-${Number(seq) + 1}`; | |
| 126 | + const rows = (await getCmd().xrange("ws:events", start, "+", "COUNT", 500)) as [string, string[]][]; | |
| 127 | + let n = 0; | |
| 128 | + for (const [sid, fields] of rows) { | |
| 129 | + const i = fields.indexOf("event"); | |
| 130 | + if (i < 0) continue; | |
| 131 | + try { | |
| 132 | + const ev = JSON.parse(fields[i + 1]!) as Record<string, unknown>; | |
| 133 | + deliver({ ...ev, sid, replayed: true }, sid, client); | |
| 134 | + n++; | |
| 135 | + } catch { | |
| 136 | + // skip malformed | |
| 137 | + } | |
| 138 | + } | |
| 139 | + if (client.ws.readyState === client.ws.OPEN) client.ws.send(JSON.stringify({ type: "replay_done", since, count: n, truncated: rows.length >= 500 })); | |
| 140 | + } catch { | |
| 141 | + if (client.ws.readyState === client.ws.OPEN) client.ws.send(JSON.stringify({ type: "replay_done", since, count: 0, error: "replay_unavailable" })); | |
| 142 | + } | |
| 143 | +} | |
| 144 | + | |
| 87 | 145 | export function channelsFor(ev: Record<string, unknown>): Set<string> { |
| 88 | 146 | const out = new Set<string>(["events:global"]); |
| 89 | 147 | const importance = Number(ev.importance ?? 0); |
| 90 | − if (importance >= 80) out.add("events:breaking"); | |
| 148 | + const signal = Number(ev.signal ?? importance); | |
| 149 | + if (signal >= 80 || importance >= 80) out.add("events:breaking"); | |
| 91 | 150 | if (ev.silent) out.add("events:silent"); |
| 151 | + if (ev.firstParty !== false) out.add("events:first-party"); | |
| 92 | 152 | const cats = (ev.categories as string[] | undefined) ?? []; |
| 93 | 153 | for (const [ch, list] of Object.entries(FEED_CHANNELS)) if (list.some((c) => cats.includes(c)) || cats.includes(ch)) out.add(`events:${ch}`); |
| 94 | 154 | const src = ev.source as { id?: string } | undefined; |
| 95 | 155 | if (src?.id) out.add(`source:${src.id}`); |
| 96 | 156 | for (const e of (ev.entities as { id: string }[] | undefined) ?? []) out.add(`entity:${e.id}`); |
| 97 | 157 | out.add(`type:${String(ev.type)}`); |
| 158 | + out.add(`group:${String(ev.group ?? eventGroupOf(String(ev.type)))}`); | |
| 159 | + if (ev.country) out.add(`country:${String(ev.country).toUpperCase()}`); | |
| 160 | + if (ev.clusterState === "breaking" || ev.clusterState === "developing") out.add(`state:${String(ev.clusterState)}`); | |
| 98 | 161 | return out; |
| 99 | 162 | } |
| 100 | 163 | |
| 101 | 164 | async function loadWatchlist(client: Client, id: string): Promise<void> { |
| 102 | 165 | const rows = await db.execute<{ kind: string; value: string }>(sql`select kind, value from watchlist_items where watchlist_id = ${id}`); |
| 103 | − const w = { entities: new Set<string>(), sources: new Set<string>(), keywords: [] as string[], categories: new Set<string>() }; | |
| 166 | + const w: WatchRules = { entities: new Set(), sources: new Set(), keywords: [], categories: new Set(), types: new Set(), countries: new Set(), urls: [] }; | |
| 104 | 167 | for (const r of rows.rows) { |
| 105 | 168 | if (r.kind === "entity") w.entities.add(r.value); |
| 106 | 169 | else if (r.kind === "source") w.sources.add(r.value); |
| 107 | 170 | else if (r.kind === "keyword") w.keywords.push(r.value.toLowerCase()); |
| 108 | 171 | else if (r.kind === "category") w.categories.add(r.value); |
| 172 | + else if (r.kind === "event_type") w.types.add(r.value); | |
| 173 | + else if (r.kind === "country") w.countries.add(r.value.toUpperCase()); | |
| 174 | + else if (r.kind === "url") w.urls.push(r.value.replace(/\/$/, "")); | |
| 109 | 175 | } |
| 110 | 176 | client.watchlists.set(id, w); |
| 111 | 177 | } |
| 112 | 178 | |
| 113 | −function matchesWatchlist(ev: Record<string, unknown>, w: { entities: Set<string>; sources: Set<string>; keywords: string[]; categories: Set<string> }): boolean { | |
| 179 | +function matchesWatchlist(ev: Record<string, unknown>, w: WatchRules): boolean { | |
| 114 | 180 | const src = ev.source as { id?: string } | undefined; |
| 115 | 181 | if (src?.id && w.sources.has(src.id)) return true; |
| 116 | 182 | for (const e of (ev.entities as { id: string }[] | undefined) ?? []) if (w.entities.has(e.id)) return true; |
| 117 | 183 | for (const c of (ev.categories as string[] | undefined) ?? []) if (w.categories.has(c)) return true; |
| 184 | + if (w.types.has(String(ev.type))) return true; | |
| 185 | + if (ev.country && w.countries.has(String(ev.country).toUpperCase())) return true; | |
| 186 | + const url = String(ev.url ?? ""); | |
| 187 | + if (w.urls.some((u) => url === u || url.startsWith(u + "/"))) return true; | |
| 118 | 188 | if (w.keywords.length) { |
| 119 | 189 | const hay = `${String(ev.title)} ${String(ev.summary)}`.toLowerCase(); |
| 120 | 190 | if (w.keywords.some((k) => hay.includes(k))) return true; |
@@ -126,4 +196,5 @@ export async function closeLive(): Promise<void> { | ||
| 126 | 196 | for (const c of clients) c.ws.close(1001, "server shutdown"); |
| 127 | 197 | clients.clear(); |
| 128 | 198 | if (sub) await sub.quit().catch(() => undefined); |
| 199 | + if (cmd) await cmd.quit().catch(() => undefined); | |
| 129 | 200 | } |
modified
apps/api/src/queries.ts
+124 −23
@@ -1,3 +1,4 @@ | ||
| 1 | +import { EVENT_GROUPS, parseSearch } from "@websensor/core"; | |
| 1 | 2 | import { db, sql, textArray } from "@websensor/db"; |
| 2 | 3 | |
| 3 | 4 | /** Read-model helpers shared by REST routes. All return plain JSON-ready objects. */ |
@@ -13,27 +14,48 @@ export interface EventFilters { | ||
| 13 | 14 | cluster?: string; |
| 14 | 15 | importance_min?: number; |
| 15 | 16 | confidence_min?: number; |
| 17 | + signal_min?: number; | |
| 16 | 18 | event_type?: string; |
| 19 | + group?: string; | |
| 17 | 20 | silent_change?: boolean; |
| 21 | + first_party?: boolean; | |
| 22 | + confirmed?: boolean; | |
| 23 | + country?: string; | |
| 24 | + language?: string; | |
| 25 | + change_class?: string; | |
| 18 | 26 | q?: string; |
| 19 | 27 | limit: number; |
| 20 | 28 | cursor?: string; |
| 21 | − order?: "recent" | "importance"; | |
| 29 | + order?: "recent" | "importance" | "signal"; | |
| 30 | + /** include events from custom (owner) sources — only set by owner-scoped routes */ | |
| 31 | + includeCustom?: boolean; | |
| 22 | 32 | } |
| 23 | 33 | |
| 24 | 34 | export const EVENT_SELECT = sql` |
| 25 | 35 | e.id, e.slug, e.event_type, e.title, e.summary, e.why_it_matters, e.importance, e.confidence, e.novelty, e.categories, e.keywords, |
| 26 | − e.silent_change, e.evidence_label, e.url, e.detected_at, e.published_at, e.observed_from, e.processed_at, e.published_to_feed_at, | |
| 36 | + e.silent_change, e.evidence_label, e.url, e.canonical_url, e.detected_at, e.published_at, e.observed_from, e.processed_at, e.published_to_feed_at, | |
| 27 | 37 | e.detection_latency_ms, e.processing_latency_ms, e.cluster_id, e.sensor_id, e.source_id, e.change_id, e.old_snapshot_id, e.new_snapshot_id, |
| 28 | 38 | e.importance_components, e.processing_version, |
| 29 | − json_build_object('id', s.id, 'name', s.name, 'domain', s.domain, 'tier', s.tier, 'categories', s.categories) as source, | |
| 39 | + e.signal_score, e.velocity_score, e.impact_score, e.anomaly_score, e.change_class, e.first_party, e.country, e.language, e.field_changes, e.score_reasons, | |
| 40 | + json_build_object('id', s.id, 'name', s.name, 'domain', s.domain, 'tier', s.tier, 'categories', s.categories, 'first_party', s.first_party, 'country', s.country) as source, | |
| 30 | 41 | json_build_object('id', sen.id, 'name', sen.name, 'type', sen.type, 'connector', sen.connector, 'tier', sen.tier) as sensor, |
| 31 | 42 | coalesce((select json_agg(json_build_object('id', en.id, 'name', en.name, 'type', en.type, 'role', ee.role) order by ee.role, en.name) |
| 32 | 43 | from event_entities ee join entities en on en.id = ee.entity_id where ee.event_id = e.id), '[]'::json) as entities, |
| 33 | − (select event_count from event_clusters c where c.id = e.cluster_id) as cluster_size`; | |
| 44 | + (select event_count from event_clusters c where c.id = e.cluster_id) as cluster_size, | |
| 45 | + (select json_build_object('id', c.id, 'slug', c.slug, 'state', c.state, 'event_count', c.event_count, 'source_count', c.source_count, 'first_party_count', c.first_party_count, 'external_count', c.external_count, 'velocity', c.velocity, 'lead_time_ms', c.lead_time_ms) from event_clusters c where c.id = e.cluster_id) as cluster`; | |
| 34 | 46 | |
| 35 | −export async function listEvents(f: EventFilters): Promise<{ items: Record<string, unknown>[]; nextCursor: string | null }> { | |
| 47 | +/** Build WHERE conditions from filters (shared by list, RSS, watchlist and desk queries). */ | |
| 48 | +export function eventConditions(f: EventFilters): ReturnType<typeof sql>[] { | |
| 36 | 49 | const conds = [sql`true`]; |
| 50 | + let free = f.q?.trim() ?? ""; | |
| 51 | + // Advanced syntax inside q (entity:… type:… after:…) is merged into the filters. | |
| 52 | + if (free) { | |
| 53 | + const p = parseSearch(free); | |
| 54 | + free = p.text; | |
| 55 | + const pf = p.filters; | |
| 56 | + f = { ...f, entity: f.entity ?? pf.entity, source: f.source ?? pf.source, domain: f.domain ?? pf.domain, event_type: f.event_type ?? pf.event_type, group: f.group ?? pf.group, category: f.category ?? pf.category, country: f.country ?? pf.country, language: f.language ?? pf.language, after: f.after ?? pf.after, before: f.before ?? pf.before, silent_change: f.silent_change ?? pf.silent_change, first_party: f.first_party ?? pf.first_party, confirmed: f.confirmed ?? pf.confirmed, importance_min: f.importance_min ?? pf.importance_min, confidence_min: f.confidence_min ?? pf.confidence_min, signal_min: f.signal_min ?? pf.signal_min, change_class: f.change_class ?? pf.change_class, cluster: f.cluster ?? pf.cluster, sensor: f.sensor ?? pf.sensor }; | |
| 57 | + } | |
| 58 | + if (!f.includeCustom) conds.push(sql`s.kind = 'registry'`); | |
| 37 | 59 | if (f.after) conds.push(sql`e.detected_at > ${new Date(f.after)}`); |
| 38 | 60 | if (f.before) conds.push(sql`e.detected_at < ${new Date(f.before)}`); |
| 39 | 61 | if (f.category) conds.push(sql`${f.category} = any(e.categories)`); |
@@ -44,22 +66,41 @@ export async function listEvents(f: EventFilters): Promise<{ items: Record<strin | ||
| 44 | 66 | if (f.entity) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = ${f.entity})`); |
| 45 | 67 | if (f.importance_min !== undefined) conds.push(sql`e.importance >= ${f.importance_min}`); |
| 46 | 68 | if (f.confidence_min !== undefined) conds.push(sql`e.confidence >= ${f.confidence_min}`); |
| 47 | − if (f.event_type) conds.push(sql`e.event_type = any(${textArray(f.event_type.split(","))})`); | |
| 69 | + if (f.signal_min !== undefined) conds.push(sql`coalesce(e.signal_score, e.importance) >= ${f.signal_min}`); | |
| 70 | + if (f.event_type) conds.push(sql`e.event_type = any(${textArray(f.event_type.split(",").map((t) => t.trim()).filter(Boolean))})`); | |
| 71 | + if (f.group && EVENT_GROUPS[f.group]) conds.push(sql`e.event_type = any(${textArray(EVENT_GROUPS[f.group]!.types)})`); | |
| 48 | 72 | if (f.silent_change !== undefined) conds.push(sql`e.silent_change = ${f.silent_change}`); |
| 49 | − if (f.q) conds.push(sql`e.search @@ websearch_to_tsquery('english', ${f.q})`); | |
| 73 | + if (f.first_party !== undefined) conds.push(sql`e.first_party = ${f.first_party}`); | |
| 74 | + if (f.confirmed) conds.push(sql`e.evidence_label = 'CONFIRMED'`); | |
| 75 | + if (f.country) conds.push(sql`e.country = ${f.country.toUpperCase()}`); | |
| 76 | + if (f.language) conds.push(sql`e.language = ${f.language.toLowerCase()}`); | |
| 77 | + if (f.change_class) conds.push(sql`e.change_class = ${f.change_class}`); | |
| 78 | + if (free) conds.push(sql`(e.search @@ websearch_to_tsquery('english', ${free}) or e.title ilike ${"%" + free + "%"})`); | |
| 79 | + return conds; | |
| 80 | +} | |
| 81 | + | |
| 82 | +export async function listEvents(f: EventFilters): Promise<{ items: Record<string, unknown>[]; nextCursor: string | null }> { | |
| 83 | + const conds = eventConditions(f); | |
| 50 | 84 | if (f.cursor) { |
| 51 | 85 | const [ts, id] = decodeCursor(f.cursor); |
| 52 | 86 | if (f.order === "importance") conds.push(sql`(e.importance, e.id) < (${Number(ts)}, ${id})`); |
| 87 | + else if (f.order === "signal") conds.push(sql`(coalesce(e.signal_score, e.importance), e.id) < (${Number(ts)}, ${id})`); | |
| 53 | 88 | else conds.push(sql`(e.detected_at, e.id) < (${new Date(Number(ts))}, ${id})`); |
| 54 | 89 | } |
| 55 | − const order = f.order === "importance" ? sql`e.importance desc, e.id desc` : sql`e.detected_at desc, e.id desc`; | |
| 90 | + const order = f.order === "importance" ? sql`e.importance desc, e.id desc` : f.order === "signal" ? sql`coalesce(e.signal_score, e.importance) desc, e.id desc` : sql`e.detected_at desc, e.id desc`; | |
| 56 | 91 | const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where ${sql.join(conds, sql` and `)} order by ${order} limit ${f.limit + 1}`); |
| 57 | 92 | const items = rows.rows.slice(0, f.limit); |
| 58 | 93 | const last = items[items.length - 1]; |
| 59 | − const nextCursor = rows.rows.length > f.limit && last ? encodeCursor(f.order === "importance" ? String(last.importance) : String(new Date(last.detected_at as string).getTime()), String(last.id)) : null; | |
| 94 | + const nextCursor = rows.rows.length > f.limit && last ? encodeCursor(f.order === "importance" ? String(last.importance) : f.order === "signal" ? String(last.signal_score ?? last.importance) : String(new Date(last.detected_at as string).getTime()), String(last.id)) : null; | |
| 60 | 95 | return { items, nextCursor }; |
| 61 | 96 | } |
| 62 | 97 | |
| 98 | +export async function countEvents(f: EventFilters): Promise<number> { | |
| 99 | + const conds = eventConditions(f); | |
| 100 | + const r = await db.execute<{ n: string }>(sql`select count(*)::text as n from events e join sources s on s.id = e.source_id where ${sql.join(conds, sql` and `)}`); | |
| 101 | + return Number(r.rows[0]?.n ?? 0); | |
| 102 | +} | |
| 103 | + | |
| 63 | 104 | export async function getEvent(idOrSlug: string): Promise<Record<string, unknown> | null> { |
| 64 | 105 | const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT}, e.interpretation from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.id = ${idOrSlug} or e.slug = ${idOrSlug} limit 1`); |
| 65 | 106 | return rows.rows[0] ?? null; |
@@ -68,49 +109,78 @@ export async function getEvent(idOrSlug: string): Promise<Record<string, unknown | ||
| 68 | 109 | export async function relatedEvents(ev: Record<string, unknown>, limit = 8): Promise<Record<string, unknown>[]> { |
| 69 | 110 | const rows = await db.execute<Record<string, unknown>>(sql` |
| 70 | 111 | select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id |
| 71 | − where e.id <> ${String(ev.id)} and (e.cluster_id = ${String(ev.cluster_id ?? "")} or e.source_id = ${String(ev.source_id)} or exists (select 1 from event_entities a join event_entities b on a.entity_id = b.entity_id where a.event_id = e.id and b.event_id = ${String(ev.id)})) | |
| 112 | + where e.id <> ${String(ev.id)} and s.kind = 'registry' and (e.cluster_id = ${String(ev.cluster_id ?? "")} or e.source_id = ${String(ev.source_id)} or exists (select 1 from event_entities a join event_entities b on a.entity_id = b.entity_id where a.event_id = e.id and b.event_id = ${String(ev.id)})) | |
| 72 | 113 | order by (e.cluster_id = ${String(ev.cluster_id ?? "")}) desc, e.detected_at desc limit ${limit}`); |
| 73 | 114 | return rows.rows; |
| 74 | 115 | } |
| 75 | 116 | |
| 117 | +/** Previous meaningful events on the same sensor (historical context for "what changed" — spec §79–80). */ | |
| 118 | +export async function sensorHistory(sensorId: string, beforeEventId: string, limit = 6): Promise<Record<string, unknown>[]> { | |
| 119 | + const rows = await db.execute<Record<string, unknown>>(sql` | |
| 120 | + select e.id, e.slug, e.title, e.event_type, e.importance, e.silent_change, e.detected_at, e.field_changes from events e | |
| 121 | + where e.sensor_id = ${sensorId} and e.id <> ${beforeEventId} order by e.detected_at desc limit ${limit}`); | |
| 122 | + return rows.rows; | |
| 123 | +} | |
| 124 | + | |
| 76 | 125 | export async function stats(): Promise<Record<string, unknown>> { |
| 77 | 126 | const [r] = ( |
| 78 | 127 | await db.execute<Record<string, unknown>>(sql` |
| 79 | 128 | select |
| 80 | − (select count(*) from sources where enabled) as sources, | |
| 129 | + (select count(*) from sources where enabled and kind = 'registry') as sources, | |
| 81 | 130 | (select count(*) from sensors where enabled) as sensors, |
| 82 | 131 | (select count(*) from entities) as entities, |
| 83 | 132 | (select count(*) from snapshots) as snapshots, |
| 84 | 133 | (select count(*) from events) as events_total, |
| 85 | 134 | (select count(*) from events where detected_at >= now() - interval '24 hours') as events_24h, |
| 135 | + (select count(*) from events where detected_at >= now() - interval '1 hour') as events_1h, | |
| 86 | 136 | (select count(*) from events where detected_at >= now() - interval '24 hours' and silent_change) as silent_24h, |
| 87 | − (select count(*) from events where detected_at >= now() - interval '24 hours' and importance >= 80) as breaking_24h, | |
| 137 | + (select count(*) from events where detected_at >= now() - interval '24 hours' and coalesce(signal_score, importance) >= 80) as breaking_24h, | |
| 138 | + (select count(*) from event_clusters where state = 'breaking') as breaking_now, | |
| 139 | + (select count(*) from event_clusters where state = 'developing') as developing_now, | |
| 88 | 140 | (select count(*) from changes where detected_at >= now() - interval '24 hours') as changes_24h, |
| 89 | − (select coalesce(sum(checks),0) from metrics_daily where day = current_date) as checks_today, | |
| 90 | − (select coalesce(sum(not_modified),0) from metrics_daily where day = current_date) as not_modified_today, | |
| 91 | − (select coalesce(sum(bytes),0) from metrics_daily where day = current_date) as bytes_today, | |
| 141 | + (select coalesce(sum(checks),0) from metrics_daily where day = (now() at time zone 'UTC')::date) as checks_today, | |
| 142 | + (select coalesce(sum(not_modified),0) from metrics_daily where day = (now() at time zone 'UTC')::date) as not_modified_today, | |
| 143 | + (select coalesce(sum(bytes),0) from metrics_daily where day = (now() at time zone 'UTC')::date) as bytes_today, | |
| 92 | 144 | (select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour, |
| 145 | + (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes') as checks_last_5m, | |
| 146 | + (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified') as not_modified_last_5m, | |
| 93 | 147 | (select count(*) from sensors where health = 'UP' and enabled) as sensors_up, |
| 94 | 148 | (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled) as sensors_degraded, |
| 149 | + (select count(*) from sources where enabled and first_party and kind = 'registry') as sources_first_party, | |
| 150 | + (select count(distinct country) from sources where country is not null) as countries, | |
| 95 | 151 | (select max(started_at) from sensor_runs) as last_check_at, |
| 96 | 152 | (select max(detected_at) from events) as last_event_at, |
| 97 | 153 | (select percentile_cont(0.5) within group (order by processing_latency_ms) from events where detected_at >= now() - interval '24 hours') as p50_processing_ms, |
| 98 | 154 | (select percentile_cont(0.5) within group (order by detection_latency_ms) from events where detected_at >= now() - interval '24 hours' and detection_latency_ms is not null and detection_latency_ms < 86400000) as p50_detection_ms`) |
| 99 | 155 | ).rows; |
| 100 | − return Object.fromEntries(Object.entries(r ?? {}).map(([k, v]) => [k, typeof v === "string" && /^\d+$/.test(v) ? Number(v) : v])); | |
| 156 | + const out = Object.fromEntries(Object.entries(r ?? {}).map(([k, v]) => [k, typeof v === "string" && /^\d+(\.\d+)?$/.test(v) ? Number(v) : v])); | |
| 157 | + out.checks_per_min = Math.round(Number(out.checks_last_5m ?? 0) / 5); | |
| 158 | + out.events_per_min = Math.round((Number(out.events_1h ?? 0) / 60) * 10) / 10; | |
| 159 | + out.not_modified_ratio_5m = Number(out.checks_last_5m) ? Math.round((Number(out.not_modified_last_5m) / Number(out.checks_last_5m)) * 100) / 100 : null; | |
| 160 | + return out; | |
| 101 | 161 | } |
| 102 | 162 | |
| 103 | 163 | export async function trending(hours = 24, limit = 12): Promise<Record<string, unknown>[]> { |
| 104 | 164 | const rows = await db.execute<Record<string, unknown>>(sql` |
| 105 | 165 | with cur as ( |
| 106 | − select ee.entity_id, count(*) as n, sum(e.importance) as imp, count(distinct e.source_id) as sources, sum(case when e.silent_change then 1 else 0 end) as silent, max(e.importance) as max_imp | |
| 107 | − from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - make_interval(hours => ${hours}) group by ee.entity_id), | |
| 166 | + select ee.entity_id, count(*) as n, sum(e.importance) as imp, count(distinct e.source_id) as sources, sum(case when e.silent_change then 1 else 0 end) as silent, max(e.importance) as max_imp, | |
| 167 | + sum(case when e.first_party then 1 else 0 end) as first_party, sum(case when e.evidence_label = 'CONFIRMED' then 1 else 0 end) as confirmed, avg(coalesce(e.signal_score, e.importance)) as avg_signal, | |
| 168 | + count(*) filter (where e.detected_at >= now() - make_interval(hours => ${hours}) / 4) as recent_quarter | |
| 169 | + from events e join event_entities ee on ee.event_id = e.id join sources s on s.id = e.source_id where s.kind = 'registry' and e.detected_at >= now() - make_interval(hours => ${hours}) group by ee.entity_id), | |
| 108 | 170 | prev as ( |
| 109 | 171 | select ee.entity_id, count(*) as n from events e join event_entities ee on ee.event_id = e.id |
| 110 | − where e.detected_at >= now() - make_interval(hours => ${hours * 2}) and e.detected_at < now() - make_interval(hours => ${hours}) group by ee.entity_id) | |
| 111 | − select en.id, en.name, en.type, en.domain, cur.n::int as events, cur.imp::float as importance_sum, cur.sources::int as sources, cur.silent::int as silent, cur.max_imp::float as max_importance, coalesce(prev.n,0)::int as prev_events, | |
| 112 | − round((18*(ln(1+cur.n)/ln(2)) + 0.35*(cur.imp/greatest(1,cur.n)) + 10*(ln(1+cur.sources)/ln(2)) + 12*least(2, case when coalesce(prev.n,0)=0 then 2 else cur.n::float/prev.n end) + 5*least(3,cur.silent))::numeric, 1)::float as score | |
| 113 | − from cur join entities en on en.id = cur.entity_id left join prev on prev.entity_id = cur.entity_id | |
| 172 | + where e.detected_at >= now() - make_interval(hours => ${hours * 2}) and e.detected_at < now() - make_interval(hours => ${hours}) group by ee.entity_id), | |
| 173 | + base as ( | |
| 174 | + select entity_id, avg(events)::float as per_day from entity_daily where day >= (now() at time zone 'UTC')::date - 30 and day < (now() at time zone 'UTC')::date group by entity_id) | |
| 175 | + select en.id, en.name, en.type, en.domain, en.importance as entity_importance, cur.n::int as events, cur.imp::float as importance_sum, cur.sources::int as sources, cur.silent::int as silent, cur.max_imp::float as max_importance, | |
| 176 | + cur.first_party::int as first_party, cur.confirmed::int as confirmed, round(cur.avg_signal::numeric, 1)::float as avg_signal, coalesce(prev.n,0)::int as prev_events, coalesce(base.per_day, 0)::float as baseline_per_day, | |
| 177 | + cur.recent_quarter::int as recent_quarter, | |
| 178 | + case when cur.recent_quarter::float / greatest(1, cur.n) > 0.5 then 'up' when cur.recent_quarter = 0 and cur.n >= 3 then 'down' else 'flat' end as direction, | |
| 179 | + round((100 * (1 - exp(-( | |
| 180 | + 18*(ln(1+cur.n)/ln(2)) + 0.35*(cur.imp/greatest(1,cur.n)) + 10*(ln(1+cur.sources)/ln(2)) + 12*least(2, case when coalesce(prev.n,0)=0 then 2 else cur.n::float/prev.n end) + 5*least(3,cur.silent) | |
| 181 | + + 6*least(3, cur.first_party) + 4*least(3, cur.confirmed) + (case when coalesce(base.per_day,0) > 0 then 8*least(3, cur.n / (base.per_day * ${hours} / 24.0)) else 0 end) | |
| 182 | + ) / 110.0)))::numeric, 1)::float as score | |
| 183 | + from cur join entities en on en.id = cur.entity_id left join prev on prev.entity_id = cur.entity_id left join base on base.entity_id = cur.entity_id | |
| 114 | 184 | order by score desc limit ${limit}`); |
| 115 | 185 | return rows.rows; |
| 116 | 186 | } |
@@ -122,7 +192,9 @@ export async function sourceActivity(sourceId: string): Promise<Record<string, u | ||
| 122 | 192 | (select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '2 hours')::int as changes_2h, |
| 123 | 193 | (select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '14 days')::int as changes_14d, |
| 124 | 194 | (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '24 hours')::int as events_24h, |
| 125 | − (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '14 days')::int as events_14d`) | |
| 195 | + (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '14 days')::int as events_14d, | |
| 196 | + (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '24 hours' and silent_change)::int as silent_24h, | |
| 197 | + (select count(*) from events where source_id = ${sourceId} and detected_at >= now() - interval '24 hours' and coalesce(signal_score, importance) >= 80)::int as breaking_24h`) | |
| 126 | 198 | ).rows; |
| 127 | 199 | const c2h = Number(r?.changes_2h ?? 0); |
| 128 | 200 | const baselinePerHour = Number(r?.changes_14d ?? 0) / (14 * 24); |
@@ -136,6 +208,35 @@ export async function sourceActivity(sourceId: string): Promise<Record<string, u | ||
| 136 | 208 | return { ...r, baseline_changes_per_day: Math.round(baselinePerHour * 24 * 10) / 10, activity_score: Math.round(anomaly * 10) / 10 }; |
| 137 | 209 | } |
| 138 | 210 | |
| 211 | +/** Source quality score (spec §48): success rate, latency, structured share, usefulness (events/raw), consistency. Distinct from importance. */ | |
| 212 | +export async function sourceQuality(sourceId: string): Promise<Record<string, unknown>> { | |
| 213 | + const [r] = ( | |
| 214 | + await db.execute<Record<string, unknown>>(sql` | |
| 215 | + select | |
| 216 | + (select count(*) from sensors where source_id = ${sourceId} and enabled)::int as sensors, | |
| 217 | + (select count(*) from sensors where source_id = ${sourceId} and enabled and health = 'UP')::int as sensors_up, | |
| 218 | + (select avg(avg_latency_ms) from sensors where source_id = ${sourceId} and enabled)::int as avg_latency_ms, | |
| 219 | + (select coalesce(sum(raw_changes),0) from sensors where source_id = ${sourceId})::int as raw_changes, | |
| 220 | + (select coalesce(sum(meaningful_changes),0) from sensors where source_id = ${sourceId})::int as meaningful_changes, | |
| 221 | + (select coalesce(sum(total_runs),0) from sensors where source_id = ${sourceId})::int as total_runs, | |
| 222 | + (select coalesce(sum(total_not_modified),0) from sensors where source_id = ${sourceId})::int as total_not_modified, | |
| 223 | + (select count(*) from sensors where source_id = ${sourceId} and enabled and connector <> 'http')::int as structured_sensors, | |
| 224 | + (select coalesce(sum(checks),0) from source_daily where source_id = ${sourceId} and day >= (now() at time zone 'UTC')::date - 7)::int as checks_7d, | |
| 225 | + (select coalesce(sum(errors),0) from source_daily where source_id = ${sourceId} and day >= (now() at time zone 'UTC')::date - 7)::int as errors_7d, | |
| 226 | + (select avg(confidence) from events where source_id = ${sourceId} and detected_at >= now() - interval '30 days')::float as avg_confidence`) | |
| 227 | + ).rows; | |
| 228 | + const sensors = Number(r?.sensors ?? 0); | |
| 229 | + const success = Number(r?.checks_7d) ? 1 - Number(r?.errors_7d) / Number(r?.checks_7d) : sensors ? Number(r?.sensors_up) / sensors : 1; | |
| 230 | + const latency = Number(r?.avg_latency_ms ?? 800); | |
| 231 | + const latencyScore = latency <= 400 ? 1 : latency <= 1500 ? 0.8 : latency <= 4000 ? 0.6 : 0.4; | |
| 232 | + const structured = sensors ? Number(r?.structured_sensors) / sensors : 0; | |
| 233 | + const raw = Number(r?.raw_changes ?? 0); | |
| 234 | + const usefulness = raw >= 5 ? Math.min(1, (Number(r?.meaningful_changes) / raw) * 2) : 0.6; | |
| 235 | + const conf = Number(r?.avg_confidence ?? 70) / 100; | |
| 236 | + const score = Math.round(100 * (0.35 * success + 0.15 * latencyScore + 0.15 * structured + 0.2 * usefulness + 0.15 * conf)); | |
| 237 | + return { ...r, success_rate: Math.round(success * 1000) / 1000, structured_share: Math.round(structured * 100) / 100, usefulness: Math.round(usefulness * 100) / 100, quality_score: score }; | |
| 238 | +} | |
| 239 | + | |
| 139 | 240 | export function encodeCursor(a: string, b: string): string { |
| 140 | 241 | return Buffer.from(`${a}|${b}`).toString("base64url"); |
| 141 | 242 | } |
added
apps/api/src/routes-admin.ts
+228 −0
@@ -0,0 +1,228 @@ | ||
| 1 | +import { readdir, stat } from "node:fs/promises"; | |
| 2 | +import { join, resolve } from "node:path"; | |
| 3 | +import type { FastifyInstance } from "fastify"; | |
| 4 | +import YAML from "yaml"; | |
| 5 | +import { z } from "zod"; | |
| 6 | +import { extendSchema, importDocumentSchema, sourceSchema, type SensorEndpoint, type Tier } from "@websensor/core"; | |
| 7 | +import { getConnector, listConnectors, NormalizeError } from "@websensor/connectors"; | |
| 8 | +import { db, sql } from "@websensor/db"; | |
| 9 | +import { cacheStats, invalidate } from "./cache"; | |
| 10 | +import { config } from "./config"; | |
| 11 | +import { engineStatus, liveStats } from "./live"; | |
| 12 | + | |
| 13 | +/** | |
| 14 | + * Internal operations (spec §70–72, §85). Enabled only when WS_ADMIN_TOKEN is set; every request | |
| 15 | + * must carry `X-WebSensor-Admin: <token>`. Never linked from public pages; `/ops` in the web app | |
| 16 | + * asks for the token and stores it in sessionStorage. | |
| 17 | + */ | |
| 18 | +export async function registerAdminRoutes(app: FastifyInstance): Promise<void> { | |
| 19 | + app.addHook("onRequest", async (req, reply) => { | |
| 20 | + if (!req.url.startsWith("/api/v1/admin")) return; | |
| 21 | + if (!config.adminToken) return reply.status(404).send({ error: "not_found" }); | |
| 22 | + const t = req.headers["x-websensor-admin"]; | |
| 23 | + if (typeof t !== "string" || t !== config.adminToken) return reply.status(401).send({ error: "admin_token_required" }); | |
| 24 | + }); | |
| 25 | + | |
| 26 | + app.get("/api/v1/admin/ops", async () => { | |
| 27 | + const t0 = Date.now(); | |
| 28 | + await db.execute(sql`select 1`); | |
| 29 | + const dbLatency = Date.now() - t0; | |
| 30 | + const [queue, failing, slow, throughput, storage, llm, jobs, es, byStatus, recentErrors] = await Promise.all([ | |
| 31 | + db.execute<Record<string, unknown>>(sql`select count(*) filter (where next_check_at <= now())::int as due, count(*) filter (where next_check_at <= now() - interval '10 minutes')::int as overdue_10m, count(*) filter (where priority = 0)::int as p0, count(*) filter (where priority = 0 and next_check_at <= now())::int as p0_due, min(next_check_at) as oldest_due from sensors where enabled`).then((r) => r.rows[0]), | |
| 32 | + db.execute<Record<string, unknown>>(sql`select split_part(split_part(s.url, '/', 3), ':', 1) as host, count(*)::int as failures, count(distinct s.id)::int as sensors, max(r.error) as last_error, max(r.started_at) as last_at from sensor_runs r join sensors s on s.id = r.sensor_id where r.started_at >= now() - interval '6 hours' and r.outcome in ('error','parse_error','rate_limited') group by 1 order by failures desc limit 20`).then((r) => r.rows), | |
| 33 | + db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.url, s.connector, s.avg_latency_ms, s.total_runs from sensors s where s.enabled and s.avg_latency_ms is not null order by s.avg_latency_ms desc limit 20`).then((r) => r.rows), | |
| 34 | + db.execute<Record<string, unknown>>(sql` | |
| 35 | + select to_char(date_trunc('hour', started_at at time zone 'UTC'), 'YYYY-MM-DD"T"HH24:00:00"Z"') as t, count(*)::int as checks, count(*) filter (where outcome = 'not_modified')::int as not_modified, count(*) filter (where outcome in ('error','parse_error','rate_limited'))::int as errors, count(*) filter (where outcome = 'event')::int as events, avg(duration_ms)::int as avg_ms | |
| 36 | + from sensor_runs where started_at >= now() - interval '24 hours' group by 1 order by 1`).then((r) => r.rows), | |
| 37 | + Promise.all([ | |
| 38 | + db.execute<Record<string, unknown>>(sql`select pg_size_pretty(pg_database_size(current_database())) as db_size, pg_database_size(current_database())::bigint as db_bytes, (select count(*) from snapshots)::int as snapshots, (select count(*) from snapshots where storage_key is not null)::int as snapshots_with_raw, (select coalesce(sum(content_length),0) from snapshots where storage_key is not null)::bigint as raw_bytes_uncompressed, (select count(*) from changes)::int as changes, (select count(*) from events)::int as events, (select count(*) from sensor_runs)::int as runs`).then((r) => r.rows[0]), | |
| 39 | + blobDirSize(), | |
| 40 | + ]).then(([d, blobs]) => ({ ...d, blobs })), | |
| 41 | + db.execute<Record<string, unknown>>(sql`select model, count(*)::int as calls, sum(input_tokens)::bigint as input_tokens, sum(output_tokens)::bigint as output_tokens, count(*) filter (where not ok)::int as failures from llm_usage where at >= now() - interval '24 hours' group by model`).then((r) => r.rows), | |
| 42 | + db.execute<Record<string, unknown>>(sql`select outcome, count(*)::int as n from sensor_runs where started_at >= now() - interval '1 hour' group by outcome order by n desc`).then((r) => r.rows), | |
| 43 | + engineStatus(), | |
| 44 | + db.execute<Record<string, unknown>>(sql`select status, health, count(*)::int as n from sensors group by status, health order by n desc`).then((r) => r.rows), | |
| 45 | + db.execute<Record<string, unknown>>(sql`select r.started_at, s.id as sensor_id, s.source_id, s.connector, r.http_status, r.outcome, r.error from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error') and r.started_at >= now() - interval '1 hour' order by r.started_at desc limit 30`).then((r) => r.rows), | |
| 46 | + ]); | |
| 47 | + return { db: { latency_ms: dbLatency }, queue, engine: es, workers: es ? { inflight: es.inflight, concurrency: es.concurrency, busy_hosts: es.busyHosts, circuit_open: es.circuitOpen } : null, outcomes_1h: jobs, top_failing_domains: failing, slowest_sensors: slow, throughput_24h: throughput, storage, llm_24h: llm, sensors_by_status: byStatus, recent_errors: recentErrors, cache: cacheStats(), live: liveStats(), connectors: listConnectors().map((c) => c.metadata()), generated_at: new Date().toISOString() }; | |
| 48 | + }); | |
| 49 | + | |
| 50 | + // ---- Sensor actions ------------------------------------------------------------------------------ | |
| 51 | + app.post<{ Params: { id: string } }>("/api/v1/admin/sensors/:id/run-now", async (req, reply) => { | |
| 52 | + const r = await db.execute(sql`update sensors set next_check_at = now() - interval '1 second', enabled = true where id = ${req.params.id}`); | |
| 53 | + if (!r.rowCount) return reply.status(404).send({ error: "not_found" }); | |
| 54 | + return { ok: true, scheduled: "now" }; | |
| 55 | + }); | |
| 56 | + app.post<{ Params: { id: string } }>("/api/v1/admin/sensors/:id/enable", async (req, reply) => { | |
| 57 | + const r = await db.execute(sql`update sensors set enabled = true, status = 'ACTIVE', consecutive_errors = 0, next_check_at = now(), updated_at = now() where id = ${req.params.id}`); | |
| 58 | + if (!r.rowCount) return reply.status(404).send({ error: "not_found" }); | |
| 59 | + return { ok: true }; | |
| 60 | + }); | |
| 61 | + app.post<{ Params: { id: string } }>("/api/v1/admin/sensors/:id/disable", async (req, reply) => { | |
| 62 | + const r = await db.execute(sql`update sensors set enabled = false, status = 'DISABLED', updated_at = now() where id = ${req.params.id}`); | |
| 63 | + if (!r.rowCount) return reply.status(404).send({ error: "not_found" }); | |
| 64 | + return { ok: true }; | |
| 65 | + }); | |
| 66 | + app.patch<{ Params: { id: string } }>("/api/v1/admin/sensors/:id", async (req, reply) => { | |
| 67 | + const body = z.object({ tier: z.enum(["S", "A", "B", "C", "D"]).optional(), priority: z.number().int().min(0).max(3).optional(), base_interval_seconds: z.number().int().min(15).max(7 * 86400).nullable().optional(), config: z.record(z.string(), z.unknown()).optional(), name: z.string().min(1).max(120).optional() }).parse(req.body ?? {}); | |
| 68 | + const sets: ReturnType<typeof sql>[] = [sql`updated_at = now()`]; | |
| 69 | + if (body.tier) sets.push(sql`tier = ${body.tier}`); | |
| 70 | + if (body.priority !== undefined) sets.push(sql`priority = ${body.priority}`); | |
| 71 | + if (body.base_interval_seconds !== undefined) sets.push(sql`base_interval_seconds = ${body.base_interval_seconds}`); | |
| 72 | + if (body.config) sets.push(sql`config = config || ${JSON.stringify(body.config)}::jsonb`); | |
| 73 | + if (body.name) sets.push(sql`name = ${body.name}`); | |
| 74 | + const r = await db.execute(sql`update sensors set ${sql.join(sets, sql`, `)} where id = ${req.params.id}`); | |
| 75 | + if (!r.rowCount) return reply.status(404).send({ error: "not_found" }); | |
| 76 | + return { ok: true }; | |
| 77 | + }); | |
| 78 | + /** Dry-run a connector against a URL (or an existing sensor) without persisting anything: inspect response + normalized output. */ | |
| 79 | + app.post("/api/v1/admin/sensors/test", async (req, reply) => { | |
| 80 | + const body = z.object({ sensor_id: z.string().optional(), url: z.string().url().optional(), connector: z.string().default("http"), type: z.string().default("HTML"), config: z.record(z.string(), z.unknown()).default({}) }).parse(req.body ?? {}); | |
| 81 | + let endpoint: SensorEndpoint; | |
| 82 | + if (body.sensor_id) { | |
| 83 | + const s = (await db.execute<Record<string, unknown>>(sql`select * from sensors where id = ${body.sensor_id}`)).rows[0]; | |
| 84 | + if (!s) return reply.status(404).send({ error: "not_found" }); | |
| 85 | + endpoint = { id: String(s.id), sourceId: String(s.source_id), name: String(s.name), url: String(s.url), type: String(s.type) as SensorEndpoint["type"], tier: String(s.tier) as Tier, connector: String(s.connector), config: (s.config as Record<string, unknown>) ?? {}, etag: null, lastModified: null, state: null }; | |
| 86 | + } else if (body.url) { | |
| 87 | + endpoint = { id: "admin_test", sourceId: "admin", name: "test", url: body.url, type: body.type as SensorEndpoint["type"], tier: "C", connector: body.connector, config: body.config, etag: null, lastModified: null, state: null }; | |
| 88 | + } else return reply.status(400).send({ error: "sensor_id_or_url_required" }); | |
| 89 | + let connector; | |
| 90 | + try { | |
| 91 | + connector = getConnector(endpoint.connector); | |
| 92 | + } catch (e) { | |
| 93 | + return reply.status(400).send({ error: "unknown_connector", detail: (e as Error).message }); | |
| 94 | + } | |
| 95 | + const t0 = Date.now(); | |
| 96 | + const obs = await connector.fetch(endpoint); | |
| 97 | + const fetchMs = Date.now() - t0; | |
| 98 | + if (obs.error) return { ok: false, stage: "fetch", error: obs.error, meta: obs.meta, fetch_ms: fetchMs }; | |
| 99 | + if (obs.meta.status >= 400) return { ok: false, stage: "fetch", http_status: obs.meta.status, meta: obs.meta, fetch_ms: fetchMs }; | |
| 100 | + try { | |
| 101 | + const norm = await connector.normalize(endpoint, obs); | |
| 102 | + return { ok: true, fetch_ms: fetchMs, meta: obs.meta, normalized: { mode: norm.mode, title: norm.title ?? null, extractionConfidence: norm.extractionConfidence, items: norm.items?.length ?? null, sample_items: norm.items?.slice(0, 5) ?? null, text_preview: norm.text?.slice(0, 1500) ?? null, json_preview: norm.json !== undefined ? JSON.stringify(norm.json).slice(0, 1500) : null, canonicalHash: norm.canonicalHash, publishedAt: norm.publishedAt ?? null } }; | |
| 103 | + } catch (e) { | |
| 104 | + return { ok: false, stage: "normalize", error: e instanceof NormalizeError ? { code: e.code, message: e.message } : { code: "error", message: (e as Error).message }, meta: obs.meta, fetch_ms: fetchMs, body_preview: obs.body?.toString("utf8").slice(0, 800) ?? null }; | |
| 105 | + } | |
| 106 | + }); | |
| 107 | + | |
| 108 | + // ---- Source actions ------------------------------------------------------------------------------ | |
| 109 | + app.post<{ Params: { id: string } }>("/api/v1/admin/sources/:id/enable", async (req) => { | |
| 110 | + await db.execute(sql`update sources set enabled = true, updated_at = now() where id = ${req.params.id}`); | |
| 111 | + invalidate(""); | |
| 112 | + return { ok: true }; | |
| 113 | + }); | |
| 114 | + app.post<{ Params: { id: string } }>("/api/v1/admin/sources/:id/disable", async (req) => { | |
| 115 | + await db.execute(sql`update sources set enabled = false, updated_at = now() where id = ${req.params.id}`); | |
| 116 | + invalidate(""); | |
| 117 | + return { ok: true }; | |
| 118 | + }); | |
| 119 | + app.post<{ Params: { id: string } }>("/api/v1/admin/sources/:id/run-now", async (req) => { | |
| 120 | + const r = await db.execute(sql`update sensors set next_check_at = now() - interval '1 second' where source_id = ${req.params.id} and enabled`); | |
| 121 | + return { ok: true, sensors: r.rowCount ?? 0 }; | |
| 122 | + }); | |
| 123 | + app.post("/api/v1/admin/sensors/bulk", async (req) => { | |
| 124 | + const body = z.object({ ids: z.array(z.string()).min(1).max(2000), action: z.enum(["enable", "disable", "run-now"]) }).parse(req.body ?? {}); | |
| 125 | + const arr = sql.raw("array[" + body.ids.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]"); | |
| 126 | + let r; | |
| 127 | + if (body.action === "enable") r = await db.execute(sql`update sensors set enabled = true, status = 'ACTIVE', consecutive_errors = 0, next_check_at = now() where id = any(${arr})`); | |
| 128 | + else if (body.action === "disable") r = await db.execute(sql`update sensors set enabled = false, status = 'DISABLED' where id = any(${arr})`); | |
| 129 | + else r = await db.execute(sql`update sensors set next_check_at = now() - interval '1 second' where id = any(${arr}) and enabled`); | |
| 130 | + return { ok: true, affected: r.rowCount ?? 0 }; | |
| 131 | + }); | |
| 132 | + | |
| 133 | + /** | |
| 134 | + * Bulk import (spec §71): JSON body `{ sources: [...] }` or raw YAML (content-type text/yaml). Validates with the | |
| 135 | + * registry schema first; `dry_run=1` returns the validation report without writing. Imported sources are stored | |
| 136 | + * with `kind = 'registry'` and `config.seed = false` so the file-based sync never disables them. | |
| 137 | + */ | |
| 138 | + app.post<{ Querystring: { dry_run?: string } }>("/api/v1/admin/sources/import", async (req, reply) => { | |
| 139 | + let doc: unknown = req.body; | |
| 140 | + if (typeof req.body === "string") { | |
| 141 | + try { | |
| 142 | + doc = YAML.parse(req.body); | |
| 143 | + } catch (e) { | |
| 144 | + return reply.status(400).send({ error: "yaml_parse_error", detail: (e as Error).message }); | |
| 145 | + } | |
| 146 | + } | |
| 147 | + const parsed = importDocumentSchema.safeParse(doc); | |
| 148 | + if (!parsed.success) return reply.status(400).send({ error: "invalid_document", issues: parsed.error.issues }); | |
| 149 | + const report: { id: string | undefined; ok: boolean; extend: boolean; issues?: string[]; sensors?: number }[] = []; | |
| 150 | + const existing = new Set((await db.execute<{ id: string }>(sql`select id from sources`)).rows.map((r) => r.id)); | |
| 151 | + const valid: { extend: boolean; data: z.infer<typeof sourceSchema> | z.infer<typeof extendSchema> }[] = []; | |
| 152 | + for (const s of parsed.data.sources) { | |
| 153 | + const isExtend = (s as { extend?: boolean })?.extend === true; | |
| 154 | + const res = isExtend ? extendSchema.safeParse(s) : sourceSchema.safeParse(s); | |
| 155 | + const id = (s as { id?: string })?.id; | |
| 156 | + if (!res.success) { | |
| 157 | + report.push({ id, ok: false, extend: isExtend, issues: res.error.issues.map((i) => `${i.path.join(".")}: ${i.message}`) }); | |
| 158 | + continue; | |
| 159 | + } | |
| 160 | + if (isExtend && !existing.has(res.data.id)) { | |
| 161 | + report.push({ id, ok: false, extend: true, issues: ["extend: true but source does not exist"] }); | |
| 162 | + continue; | |
| 163 | + } | |
| 164 | + if (!isExtend && existing.has(res.data.id)) { | |
| 165 | + report.push({ id, ok: false, extend: false, issues: ["source id already exists (use extend: true)"] }); | |
| 166 | + continue; | |
| 167 | + } | |
| 168 | + valid.push({ extend: isExtend, data: res.data }); | |
| 169 | + report.push({ id, ok: true, extend: isExtend, sensors: res.data.sensors.length }); | |
| 170 | + } | |
| 171 | + if (req.query.dry_run === "1") return { dry_run: true, valid: valid.length, invalid: report.filter((r) => !r.ok).length, report }; | |
| 172 | + let sensors = 0; | |
| 173 | + for (const v of valid) { | |
| 174 | + if (!v.extend) { | |
| 175 | + const s = v.data as z.infer<typeof sourceSchema>; | |
| 176 | + await db.execute(sql`insert into sources (id, name, domain, homepage, description, categories, tier, importance_weight, discover, fallback, notes, enabled, llm_enabled, first_party, country, language, kind) | |
| 177 | + values (${s.id}, ${s.name}, ${s.domain}, ${s.homepage ?? `https://${s.domain}`}, ${s.description ?? null}, ${sql.raw("'{" + s.categories.map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.tier}, ${s.weight}, ${JSON.stringify(s.discover)}::jsonb, ${JSON.stringify(s.fallback)}::jsonb, ${s.notes ?? null}, ${s.enabled}, ${s.llm}, ${s.first_party ?? true}, ${s.country ?? null}, ${s.language ?? null}, 'registry')`); | |
| 178 | + const entId = `org_${s.id}`; | |
| 179 | + await db.execute(sql`insert into entities (id, name, type, domain, homepage, description, importance, categories) values (${entId}, ${s.name}, ${s.entity_type}, ${s.domain}, ${s.homepage ?? `https://${s.domain}`}, ${s.description ?? null}, 60, ${sql.raw("'{" + s.categories.map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}) on conflict (id) do nothing`); | |
| 180 | + await db.execute(sql`insert into source_entities (source_id, entity_id) values (${s.id}, ${entId}) on conflict do nothing`); | |
| 181 | + for (const a of new Set([s.name, s.domain, ...s.aliases])) await db.execute(sql`insert into entity_aliases (alias, entity_id) values (${a.toLowerCase()}, ${entId}) on conflict do nothing`); | |
| 182 | + } | |
| 183 | + const base = v.data; | |
| 184 | + for (const sen of base.sensors) { | |
| 185 | + const id = sen.id ?? `${base.id}_${sen.name.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "")}`; | |
| 186 | + await db.execute(sql`insert into sensors (id, source_id, name, url, type, connector, tier, importance_weight, config, base_interval_seconds, status, priority) | |
| 187 | + values (${id}, ${base.id}, ${sen.name}, ${sen.url}, ${sen.type}, ${sen.connector}, ${sen.tier ?? (v.extend ? "B" : (v.data as z.infer<typeof sourceSchema>).tier)}, ${sen.weight ?? 1}, ${JSON.stringify({ ...sen.config, seed: false, imported_at: new Date().toISOString() })}::jsonb, ${sen.interval ?? null}, 'PENDING', 2) | |
| 188 | + on conflict (id) do update set name = excluded.name, url = excluded.url, type = excluded.type, connector = excluded.connector, config = excluded.config, enabled = true, updated_at = now()`); | |
| 189 | + sensors++; | |
| 190 | + } | |
| 191 | + } | |
| 192 | + invalidate(""); | |
| 193 | + return { dry_run: false, imported_sources: valid.filter((v) => !v.extend).length, extended_sources: valid.filter((v) => v.extend).length, sensors, invalid: report.filter((r) => !r.ok).length, report }; | |
| 194 | + }); | |
| 195 | + | |
| 196 | + app.get("/api/v1/admin/failures", async () => { | |
| 197 | + const rows = await db.execute<Record<string, unknown>>(sql`select r.started_at, r.outcome, r.http_status, r.error, r.duration_ms, s.id as sensor_id, s.name, s.source_id, s.url, s.connector from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error','rate_limited') and r.started_at >= now() - interval '24 hours' order by r.started_at desc limit 200`); | |
| 198 | + return { items: rows.rows }; | |
| 199 | + }); | |
| 200 | +} | |
| 201 | + | |
| 202 | +async function blobDirSize(): Promise<{ files: number; bytes: number } | null> { | |
| 203 | + const root = resolve(process.env.BLOB_STORE_DIR ?? "./data/blobs"); | |
| 204 | + let files = 0; | |
| 205 | + let bytes = 0; | |
| 206 | + const started = Date.now(); | |
| 207 | + const walk = async (dir: string): Promise<void> => { | |
| 208 | + if (Date.now() - started > 4000) return; // bounded | |
| 209 | + let entries: string[]; | |
| 210 | + try { | |
| 211 | + entries = await readdir(dir); | |
| 212 | + } catch { | |
| 213 | + return; | |
| 214 | + } | |
| 215 | + for (const e of entries) { | |
| 216 | + const p = join(dir, e); | |
| 217 | + const st = await stat(p).catch(() => null); | |
| 218 | + if (!st) continue; | |
| 219 | + if (st.isDirectory()) await walk(p); | |
| 220 | + else { | |
| 221 | + files++; | |
| 222 | + bytes += st.size; | |
| 223 | + } | |
| 224 | + } | |
| 225 | + }; | |
| 226 | + await walk(root); | |
| 227 | + return { files, bytes }; | |
| 228 | +} | |
added
apps/api/src/routes-user.ts
+299 −0
@@ -0,0 +1,299 @@ | ||
| 1 | +import { createHash } from "node:crypto"; | |
| 2 | +import type { FastifyInstance } from "fastify"; | |
| 3 | +import { z } from "zod"; | |
| 4 | +import { assertUrlAllowed, newId, UrlPolicyError } from "@websensor/core"; | |
| 5 | +import { getConnector } from "@websensor/connectors"; | |
| 6 | +import { db, sql, textArray } from "@websensor/db"; | |
| 7 | +import { config } from "./config"; | |
| 8 | +import { EVENT_SELECT, eventConditions, listEvents } from "./queries"; | |
| 9 | + | |
| 10 | +/** | |
| 11 | + * Owner-scoped routes (anonymous owner token, phase 1): watchlists, alert rules (+ webhook | |
| 12 | + * channel), notifications, bookmarks, saved views and custom URL monitors (spec §42–45, §81, §100). | |
| 13 | + */ | |
| 14 | +export function ownerToken(headers: Record<string, unknown>): string | null { | |
| 15 | + const t = headers["x-websensor-owner"]; | |
| 16 | + return typeof t === "string" && /^[A-Za-z0-9_-]{16,80}$/.test(t) ? t : null; | |
| 17 | +} | |
| 18 | + | |
| 19 | +const WATCH_KINDS = ["entity", "source", "keyword", "category", "url", "event_type", "country", "group"] as const; | |
| 20 | + | |
| 21 | +export async function registerUserRoutes(app: FastifyInstance): Promise<void> { | |
| 22 | + const requireOwner = (req: { headers: Record<string, unknown> }, reply: { status: (n: number) => { send: (b: unknown) => unknown } }): string | null => { | |
| 23 | + const owner = ownerToken(req.headers); | |
| 24 | + if (!owner) reply.status(401).send({ error: "owner_token_required", detail: "Send an X-WebSensor-Owner header (16–80 URL-safe characters)." }); | |
| 25 | + return owner; | |
| 26 | + }; | |
| 27 | + | |
| 28 | + // ---- Watchlists ----------------------------------------------------------------------------- | |
| 29 | + app.get("/api/v1/watchlists", async (req, reply) => { | |
| 30 | + const owner = requireOwner(req, reply); | |
| 31 | + if (!owner) return; | |
| 32 | + const rows = await db.execute<Record<string, unknown>>(sql`select w.id, w.name, w.created_at, coalesce((select json_agg(json_build_object('kind', i.kind, 'value', i.value, 'added_at', i.added_at) order by i.added_at) from watchlist_items i where i.watchlist_id = w.id), '[]'::json) as items from watchlists w where owner_token = ${owner} order by created_at`); | |
| 33 | + return { items: rows.rows }; | |
| 34 | + }); | |
| 35 | + const wlBody = z.object({ name: z.string().min(1).max(80).default("My watchlist"), items: z.array(z.object({ kind: z.enum(WATCH_KINDS), value: z.string().min(1).max(300) })).max(300).default([]) }); | |
| 36 | + app.post("/api/v1/watchlists", async (req, reply) => { | |
| 37 | + const owner = requireOwner(req, reply); | |
| 38 | + if (!owner) return; | |
| 39 | + const body = wlBody.parse(req.body ?? {}); | |
| 40 | + const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from watchlists where owner_token = ${owner}`)).rows[0]?.n; | |
| 41 | + if (Number(count) >= 20) return reply.status(429).send({ error: "too_many_watchlists" }); | |
| 42 | + const id = newId("wl"); | |
| 43 | + await db.execute(sql`insert into watchlists (id, owner_token, name) values (${id}, ${owner}, ${body.name})`); | |
| 44 | + for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${id}, ${it.kind}, ${it.value}) on conflict do nothing`); | |
| 45 | + return { id, name: body.name, items: body.items }; | |
| 46 | + }); | |
| 47 | + app.put<{ Params: { id: string } }>("/api/v1/watchlists/:id", async (req, reply) => { | |
| 48 | + const owner = requireOwner(req, reply); | |
| 49 | + if (!owner) return; | |
| 50 | + const w = (await db.execute<{ id: string }>(sql`select id from watchlists where id = ${req.params.id} and owner_token = ${owner}`)).rows[0]; | |
| 51 | + if (!w) return reply.status(404).send({ error: "not_found" }); | |
| 52 | + const body = wlBody.partial().parse(req.body ?? {}); | |
| 53 | + if (body.name) await db.execute(sql`update watchlists set name = ${body.name} where id = ${w.id}`); | |
| 54 | + if (body.items) { | |
| 55 | + await db.execute(sql`delete from watchlist_items where watchlist_id = ${w.id}`); | |
| 56 | + for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${w.id}, ${it.kind}, ${it.value}) on conflict do nothing`); | |
| 57 | + } | |
| 58 | + return { ok: true }; | |
| 59 | + }); | |
| 60 | + app.delete<{ Params: { id: string } }>("/api/v1/watchlists/:id", async (req, reply) => { | |
| 61 | + const owner = requireOwner(req, reply); | |
| 62 | + if (!owner) return; | |
| 63 | + await db.execute(sql`delete from watchlists where id = ${req.params.id} and owner_token = ${owner}`); | |
| 64 | + return { ok: true }; | |
| 65 | + }); | |
| 66 | + app.get<{ Params: { id: string }; Querystring: { limit?: string } }>("/api/v1/watchlists/:id/events", async (req, reply) => { | |
| 67 | + const owner = requireOwner(req, reply); | |
| 68 | + if (!owner) return; | |
| 69 | + const items = (await db.execute<{ kind: string; value: string }>(sql`select i.kind, i.value from watchlist_items i join watchlists w on w.id = i.watchlist_id where w.id = ${req.params.id} and w.owner_token = ${owner}`)).rows; | |
| 70 | + if (!items.length) return reply.send({ items: [] }); | |
| 71 | + const by = (k: string): string[] => items.filter((i) => i.kind === k).map((i) => i.value); | |
| 72 | + const conds = [] as ReturnType<typeof sql>[]; | |
| 73 | + const ents = by("entity"); | |
| 74 | + const srcs = by("source"); | |
| 75 | + const cats = by("category"); | |
| 76 | + const types = by("event_type"); | |
| 77 | + const countries = by("country"); | |
| 78 | + const urls = by("url"); | |
| 79 | + if (ents.length) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = any(${textArray(ents)}))`); | |
| 80 | + if (srcs.length) conds.push(sql`e.source_id = any(${textArray(srcs)})`); | |
| 81 | + if (cats.length) conds.push(sql`e.categories && ${textArray(cats)}`); | |
| 82 | + if (types.length) conds.push(sql`e.event_type = any(${textArray(types)})`); | |
| 83 | + if (countries.length) conds.push(sql`e.country = any(${textArray(countries.map((c) => c.toUpperCase()))})`); | |
| 84 | + for (const u of urls) conds.push(sql`(e.url = ${u} or e.url like ${u.replace(/\/$/, "") + "/%"})`); | |
| 85 | + for (const k of by("keyword")) conds.push(sql`(e.title ilike ${"%" + k + "%"} or e.summary ilike ${"%" + k + "%"})`); | |
| 86 | + if (!conds.length) return { items: [] }; | |
| 87 | + const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where (s.kind = 'registry' or s.owner_token = ${owner}) and (${sql.join(conds, sql` or `)}) order by e.detected_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`); | |
| 88 | + return { items: rows.rows }; | |
| 89 | + }); | |
| 90 | + | |
| 91 | + // ---- Alerts ----------------------------------------------------------------------------------- | |
| 92 | + const ruleSchema = z.object({ | |
| 93 | + importance_min: z.number().min(0).max(100).optional(), | |
| 94 | + signal_min: z.number().min(0).max(100).optional(), | |
| 95 | + event_types: z.array(z.string().max(60)).max(60).optional(), | |
| 96 | + groups: z.array(z.string().max(30)).max(12).optional(), | |
| 97 | + entities: z.array(z.string().max(120)).max(100).optional(), | |
| 98 | + sources: z.array(z.string().max(120)).max(100).optional(), | |
| 99 | + keywords: z.array(z.string().max(80)).max(50).optional(), | |
| 100 | + categories: z.array(z.string().max(40)).max(40).optional(), | |
| 101 | + countries: z.array(z.string().max(3)).max(60).optional(), | |
| 102 | + silent_only: z.boolean().optional(), | |
| 103 | + first_party_only: z.boolean().optional(), | |
| 104 | + confirmed_only: z.boolean().optional(), | |
| 105 | + }); | |
| 106 | + const alertBody = z.object({ name: z.string().min(1).max(80), rule: ruleSchema, channel: z.enum(["web", "webhook"]).default("web"), channel_config: z.object({ url: z.string().url().max(500).optional(), secret: z.string().max(200).optional() }).default({}) }); | |
| 107 | + app.get("/api/v1/alerts", async (req, reply) => { | |
| 108 | + const owner = requireOwner(req, reply); | |
| 109 | + if (!owner) return; | |
| 110 | + const rows = await db.execute<Record<string, unknown>>(sql`select a.id, a.name, a.rule, a.channel, a.channel_config - 'secret' as channel_config, a.enabled, a.created_at, a.last_fired_at, a.fired_count, (select count(*) from notifications n where n.alert_id = a.id and n.created_at >= now() - interval '24 hours')::int as fired_24h from alerts a where owner_token = ${owner} order by created_at`); | |
| 111 | + return { items: rows.rows, channels: { web: "active", webhook: "active", email: "planned", slack: "planned", discord: "planned", telegram: "planned", push: "planned" } }; | |
| 112 | + }); | |
| 113 | + app.post("/api/v1/alerts", async (req, reply) => { | |
| 114 | + const owner = requireOwner(req, reply); | |
| 115 | + if (!owner) return; | |
| 116 | + const body = alertBody.parse(req.body ?? {}); | |
| 117 | + if (body.channel === "webhook") { | |
| 118 | + if (!body.channel_config.url) return reply.status(400).send({ error: "webhook_url_required" }); | |
| 119 | + try { | |
| 120 | + const u = await assertUrlAllowed(body.channel_config.url); | |
| 121 | + if (u.url.protocol !== "https:") return reply.status(400).send({ error: "webhook_https_required" }); | |
| 122 | + } catch (e) { | |
| 123 | + return reply.status(400).send({ error: "webhook_url_rejected", detail: e instanceof UrlPolicyError ? e.message : String(e) }); | |
| 124 | + } | |
| 125 | + } | |
| 126 | + const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from alerts where owner_token = ${owner}`)).rows[0]?.n; | |
| 127 | + if (Number(count) >= 50) return reply.status(429).send({ error: "too_many_alerts" }); | |
| 128 | + const id = newId("alr"); | |
| 129 | + await db.execute(sql`insert into alerts (id, owner_token, name, rule, channel, channel_config) values (${id}, ${owner}, ${body.name}, ${JSON.stringify(body.rule)}::jsonb, ${body.channel}, ${JSON.stringify(body.channel_config)}::jsonb)`); | |
| 130 | + return { id, name: body.name, rule: body.rule, channel: body.channel }; | |
| 131 | + }); | |
| 132 | + app.patch<{ Params: { id: string } }>("/api/v1/alerts/:id", async (req, reply) => { | |
| 133 | + const owner = requireOwner(req, reply); | |
| 134 | + if (!owner) return; | |
| 135 | + const body = z.object({ enabled: z.boolean().optional(), name: z.string().min(1).max(80).optional() }).parse(req.body ?? {}); | |
| 136 | + if (body.enabled !== undefined) await db.execute(sql`update alerts set enabled = ${body.enabled} where id = ${req.params.id} and owner_token = ${owner}`); | |
| 137 | + if (body.name) await db.execute(sql`update alerts set name = ${body.name} where id = ${req.params.id} and owner_token = ${owner}`); | |
| 138 | + return { ok: true }; | |
| 139 | + }); | |
| 140 | + app.delete<{ Params: { id: string } }>("/api/v1/alerts/:id", async (req, reply) => { | |
| 141 | + const owner = requireOwner(req, reply); | |
| 142 | + if (!owner) return; | |
| 143 | + await db.execute(sql`delete from alerts where id = ${req.params.id} and owner_token = ${owner}`); | |
| 144 | + return { ok: true }; | |
| 145 | + }); | |
| 146 | + app.get<{ Querystring: { limit?: string; unread?: string } }>("/api/v1/notifications", async (req, reply) => { | |
| 147 | + const owner = requireOwner(req, reply); | |
| 148 | + if (!owner) return; | |
| 149 | + const rows = await db.execute<Record<string, unknown>>(sql` | |
| 150 | + select n.id, n.alert_id, a.name as alert_name, n.event_id, n.channel, n.status, n.created_at, n.read_at, n.delivered_at, n.error, | |
| 151 | + json_build_object('id', e.id, 'slug', e.slug, 'title', e.title, 'importance', e.importance, 'signal_score', e.signal_score, 'event_type', e.event_type, 'silent_change', e.silent_change, 'detected_at', e.detected_at, 'source', json_build_object('id', s.id, 'name', s.name)) as event | |
| 152 | + from notifications n join alerts a on a.id = n.alert_id join events e on e.id = n.event_id join sources s on s.id = e.source_id | |
| 153 | + where a.owner_token = ${owner} ${req.query.unread === "1" ? sql`and n.read_at is null` : sql``} order by n.created_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`); | |
| 154 | + const unread = (await db.execute<{ n: string }>(sql`select count(*)::text as n from notifications n join alerts a on a.id = n.alert_id where a.owner_token = ${owner} and n.read_at is null`)).rows[0]?.n; | |
| 155 | + return { items: rows.rows, unread: Number(unread ?? 0) }; | |
| 156 | + }); | |
| 157 | + app.post("/api/v1/notifications/read", async (req, reply) => { | |
| 158 | + const owner = requireOwner(req, reply); | |
| 159 | + if (!owner) return; | |
| 160 | + const body = z.object({ ids: z.array(z.number().int()).max(500).optional() }).parse(req.body ?? {}); | |
| 161 | + if (body.ids?.length) await db.execute(sql`update notifications n set read_at = now() from alerts a where a.id = n.alert_id and a.owner_token = ${owner} and n.id = any(${sql.raw("array[" + body.ids.map((i) => Number(i)).join(",") + "]::bigint[]")})`); | |
| 162 | + else await db.execute(sql`update notifications n set read_at = now() from alerts a where a.id = n.alert_id and a.owner_token = ${owner} and n.read_at is null`); | |
| 163 | + return { ok: true }; | |
| 164 | + }); | |
| 165 | + | |
| 166 | + // ---- Bookmarks --------------------------------------------------------------------------------- | |
| 167 | + app.get<{ Querystring: { limit?: string } }>("/api/v1/bookmarks", async (req, reply) => { | |
| 168 | + const owner = requireOwner(req, reply); | |
| 169 | + if (!owner) return; | |
| 170 | + const rows = await db.execute<Record<string, unknown>>(sql`select b.created_at as bookmarked_at, b.note, x.* from bookmarks b join lateral (select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where e.id = b.event_id) x on true where b.owner_token = ${owner} order by b.created_at desc limit ${Math.min(500, Number(req.query.limit ?? 100))}`); | |
| 171 | + return { items: rows.rows }; | |
| 172 | + }); | |
| 173 | + app.post("/api/v1/bookmarks", async (req, reply) => { | |
| 174 | + const owner = requireOwner(req, reply); | |
| 175 | + if (!owner) return; | |
| 176 | + const body = z.object({ event_id: z.string().min(4).max(80), note: z.string().max(500).optional() }).parse(req.body ?? {}); | |
| 177 | + const ev = (await db.execute<{ id: string }>(sql`select id from events where id = ${body.event_id} or slug = ${body.event_id}`)).rows[0]; | |
| 178 | + if (!ev) return reply.status(404).send({ error: "event_not_found" }); | |
| 179 | + const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from bookmarks where owner_token = ${owner}`)).rows[0]?.n; | |
| 180 | + if (Number(count) >= 2000) return reply.status(429).send({ error: "too_many_bookmarks" }); | |
| 181 | + await db.execute(sql`insert into bookmarks (owner_token, event_id, note) values (${owner}, ${ev.id}, ${body.note ?? null}) on conflict (owner_token, event_id) do update set note = coalesce(excluded.note, bookmarks.note)`); | |
| 182 | + return { ok: true, event_id: ev.id }; | |
| 183 | + }); | |
| 184 | + app.delete<{ Params: { id: string } }>("/api/v1/bookmarks/:id", async (req, reply) => { | |
| 185 | + const owner = requireOwner(req, reply); | |
| 186 | + if (!owner) return; | |
| 187 | + await db.execute(sql`delete from bookmarks where owner_token = ${owner} and (event_id = ${req.params.id} or event_id = (select id from events where slug = ${req.params.id}))`); | |
| 188 | + return { ok: true }; | |
| 189 | + }); | |
| 190 | + app.get("/api/v1/bookmarks/ids", async (req, reply) => { | |
| 191 | + const owner = requireOwner(req, reply); | |
| 192 | + if (!owner) return; | |
| 193 | + const rows = await db.execute<{ event_id: string }>(sql`select event_id from bookmarks where owner_token = ${owner}`); | |
| 194 | + return { ids: rows.rows.map((r) => r.event_id) }; | |
| 195 | + }); | |
| 196 | + | |
| 197 | + // ---- Saved views ------------------------------------------------------------------------------- | |
| 198 | + app.get("/api/v1/views", async (req, reply) => { | |
| 199 | + const owner = requireOwner(req, reply); | |
| 200 | + if (!owner) return; | |
| 201 | + return { items: (await db.execute<Record<string, unknown>>(sql`select id, name, query, created_at from saved_views where owner_token = ${owner} order by created_at`)).rows }; | |
| 202 | + }); | |
| 203 | + app.post("/api/v1/views", async (req, reply) => { | |
| 204 | + const owner = requireOwner(req, reply); | |
| 205 | + if (!owner) return; | |
| 206 | + const body = z.object({ name: z.string().min(1).max(60), query: z.string().min(1).max(600) }).parse(req.body ?? {}); | |
| 207 | + const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from saved_views where owner_token = ${owner}`)).rows[0]?.n; | |
| 208 | + if (Number(count) >= 50) return reply.status(429).send({ error: "too_many_views" }); | |
| 209 | + const id = newId("view"); | |
| 210 | + await db.execute(sql`insert into saved_views (id, owner_token, name, query) values (${id}, ${owner}, ${body.name}, ${body.query})`); | |
| 211 | + return { id, ...body }; | |
| 212 | + }); | |
| 213 | + app.delete<{ Params: { id: string } }>("/api/v1/views/:id", async (req, reply) => { | |
| 214 | + const owner = requireOwner(req, reply); | |
| 215 | + if (!owner) return; | |
| 216 | + await db.execute(sql`delete from saved_views where id = ${req.params.id} and owner_token = ${owner}`); | |
| 217 | + return { ok: true }; | |
| 218 | + }); | |
| 219 | + | |
| 220 | + // ---- Custom URL monitors (spec §45, §87) ----------------------------------------------------------- | |
| 221 | + const monitorBody = z.object({ | |
| 222 | + url: z.string().url().max(2000), | |
| 223 | + name: z.string().min(1).max(80).optional(), | |
| 224 | + /** hourly | daily (maps to tier C / D) */ | |
| 225 | + frequency: z.enum(["hourly", "daily"]).default("hourly"), | |
| 226 | + /** low = only big changes, normal, high = every meaningful change */ | |
| 227 | + sensitivity: z.enum(["low", "normal", "high"]).default("normal"), | |
| 228 | + selector: z.string().max(200).optional(), | |
| 229 | + keywords: z.array(z.string().min(1).max(60)).max(20).optional(), | |
| 230 | + }); | |
| 231 | + const ownerSourceId = (owner: string): string => `custom-${createHash("sha256").update(owner).digest("hex").slice(0, 12)}`; | |
| 232 | + | |
| 233 | + app.get("/api/v1/monitors", async (req, reply) => { | |
| 234 | + const owner = requireOwner(req, reply); | |
| 235 | + if (!owner) return; | |
| 236 | + const sid = ownerSourceId(owner); | |
| 237 | + const rows = await db.execute<Record<string, unknown>>(sql`select id, name, url, tier, health, status, enabled, config, next_check_at, last_check_at, last_change_at, last_event_at, last_status, last_error, total_runs, raw_changes, meaningful_changes, created_at from sensors where source_id = ${sid} order by created_at`); | |
| 238 | + return { items: rows.rows, limit: config.monitorsPerOwner, source_id: sid }; | |
| 239 | + }); | |
| 240 | + app.post("/api/v1/monitors", async (req, reply) => { | |
| 241 | + const owner = requireOwner(req, reply); | |
| 242 | + if (!owner) return; | |
| 243 | + const body = monitorBody.parse(req.body ?? {}); | |
| 244 | + let host: string; | |
| 245 | + try { | |
| 246 | + const u = await assertUrlAllowed(body.url); | |
| 247 | + if (!["http:", "https:"].includes(u.url.protocol)) return reply.status(400).send({ error: "scheme_not_allowed" }); | |
| 248 | + host = u.url.hostname; | |
| 249 | + } catch (e) { | |
| 250 | + return reply.status(400).send({ error: "url_rejected", detail: e instanceof UrlPolicyError ? e.message : "URL not allowed" }); | |
| 251 | + } | |
| 252 | + const sid = ownerSourceId(owner); | |
| 253 | + const n = Number((await db.execute<{ n: string }>(sql`select count(*)::text as n from sensors where source_id = ${sid} and enabled`)).rows[0]?.n ?? 0); | |
| 254 | + if (n >= config.monitorsPerOwner) return reply.status(429).send({ error: "monitor_limit_reached", limit: config.monitorsPerOwner }); | |
| 255 | + // Sensor auto-test (spec §72): fetch + normalize before activation; baseline is taken by the engine. | |
| 256 | + const sensorId = `${sid}_${newId("m").slice(2)}`; | |
| 257 | + const cfg: Record<string, unknown> = { custom: true, sensitivity: body.sensitivity, ...(body.selector ? { selector: body.selector } : {}), ...(body.keywords?.length ? { keywords: body.keywords } : {}) }; | |
| 258 | + const tier = body.frequency === "hourly" ? "C" : "D"; | |
| 259 | + const connector = getConnector("http"); | |
| 260 | + const endpoint = { id: sensorId, sourceId: sid, name: body.name ?? host, url: body.url, type: "HTML" as const, tier: tier as "C" | "D", connector: "http", config: cfg, etag: null, lastModified: null, state: null }; | |
| 261 | + const obs = await connector.fetch(endpoint); | |
| 262 | + if (obs.error) return reply.status(422).send({ error: "fetch_failed", detail: `${obs.error.code}: ${obs.error.message}` }); | |
| 263 | + if (obs.meta.status >= 400) return reply.status(422).send({ error: "http_error", status: obs.meta.status }); | |
| 264 | + let norm; | |
| 265 | + try { | |
| 266 | + norm = await connector.normalize(endpoint, obs); | |
| 267 | + } catch (e) { | |
| 268 | + return reply.status(422).send({ error: "unparseable", detail: (e as Error).message }); | |
| 269 | + } | |
| 270 | + if ((norm.text ?? "").length < 40 && norm.mode === "text") return reply.status(422).send({ error: "thin_content", detail: "The page has almost no server-rendered text (client-side app?)." }); | |
| 271 | + await db.execute(sql`insert into sources (id, name, domain, homepage, description, categories, tier, importance_weight, enabled, llm_enabled, kind, owner_token, first_party, notes) | |
| 272 | + values (${sid}, ${"Custom monitors"}, ${"custom.websensor.io"}, ${null}, ${"Private URL monitors"}, ${"{custom}"}::text[], 'C', 0.5, true, false, 'custom', ${owner}, true, 'owner-scoped custom monitors') on conflict (id) do nothing`); | |
| 273 | + await db.execute(sql`insert into sensors (id, source_id, name, url, type, connector, tier, importance_weight, config, priority, status, validated_at, enabled, next_check_at) | |
| 274 | + values (${sensorId}, ${sid}, ${body.name ?? host}, ${body.url}, 'HTML', 'http', ${tier}, ${body.sensitivity === "high" ? 1.3 : body.sensitivity === "low" ? 0.7 : 1}, ${JSON.stringify(cfg)}::jsonb, 3, 'VALIDATED', now(), true, now())`); | |
| 275 | + return { id: sensorId, source_id: sid, url: body.url, tier, status: "VALIDATED", test: { http_status: obs.meta.status, content_type: obs.meta.contentType, bytes: obs.meta.contentLength, title: norm.title ?? null, mode: norm.mode, extraction_confidence: norm.extractionConfidence } }; | |
| 276 | + }); | |
| 277 | + app.delete<{ Params: { id: string } }>("/api/v1/monitors/:id", async (req, reply) => { | |
| 278 | + const owner = requireOwner(req, reply); | |
| 279 | + if (!owner) return; | |
| 280 | + const sid = ownerSourceId(owner); | |
| 281 | + await db.execute(sql`delete from sensors where id = ${req.params.id} and source_id = ${sid}`); | |
| 282 | + return { ok: true }; | |
| 283 | + }); | |
| 284 | + app.get<{ Params: { id: string }; Querystring: { limit?: string } }>("/api/v1/monitors/:id/events", async (req, reply) => { | |
| 285 | + const owner = requireOwner(req, reply); | |
| 286 | + if (!owner) return; | |
| 287 | + const sid = ownerSourceId(owner); | |
| 288 | + const sensor = (await db.execute<{ id: string }>(sql`select id from sensors where id = ${req.params.id} and source_id = ${sid}`)).rows[0]; | |
| 289 | + if (!sensor) return reply.status(404).send({ error: "not_found" }); | |
| 290 | + const [events, changes] = await Promise.all([ | |
| 291 | + listEvents({ sensor: sensor.id, limit: Math.min(100, Number(req.query.limit ?? 30)), includeCustom: true }), | |
| 292 | + db.execute<Record<string, unknown>>(sql`select id, detected_at, kind, signal, change_class, field_changes, meaningful, event_id, old_snapshot_id, new_snapshot_id from changes where sensor_id = ${sensor.id} order by detected_at desc limit 30`).then((r) => r.rows), | |
| 293 | + ]); | |
| 294 | + return { events: events.items, changes }; | |
| 295 | + }); | |
| 296 | + | |
| 297 | + // keep eventConditions referenced for owner-scoped searches later | |
| 298 | + void eventConditions; | |
| 299 | +} | |
modified
apps/api/src/routes.ts
+226 −173
@@ -1,13 +1,20 @@ | ||
| 1 | 1 | import type { FastifyInstance } from "fastify"; |
| 2 | 2 | import { z } from "zod"; |
| 3 | −import { diffText, EVENT_TYPES, FEED_CHANNELS, newId } from "@websensor/core"; | |
| 3 | +import { COUNTRIES, countryBySlug, diffText, EVENT_GROUPS, EVENT_TYPES, FEED_CHANNELS, parseSearch } from "@websensor/core"; | |
| 4 | 4 | import { db, sql, textArray } from "@websensor/db"; |
| 5 | 5 | import { getBlobStore } from "@websensor/store"; |
| 6 | +import { cached } from "./cache"; | |
| 6 | 7 | import { config } from "./config"; |
| 7 | −import { liveStats } from "./live"; | |
| 8 | −import { getEvent, listEvents, relatedEvents, sourceActivity, stats, trending, EVENT_SELECT } from "./queries"; | |
| 8 | +import { breakingDesk, categoryDesk, clusterDetail, countryDesk, countryList, entityInsights, entityRankings, pulse, radar, sourceSensors } from "./intel"; | |
| 9 | +import { engineStatus, liveStats } from "./live"; | |
| 10 | +import { countEvents, getEvent, listEvents, relatedEvents, sensorHistory, sourceActivity, sourceQuality, stats, trending } from "./queries"; | |
| 9 | 11 | |
| 10 | −const eventsQuery = z.object({ | |
| 12 | +const bool = z | |
| 13 | + .enum(["true", "false", "1", "0"]) | |
| 14 | + .transform((v) => v === "true" || v === "1") | |
| 15 | + .optional(); | |
| 16 | + | |
| 17 | +export const eventsQuery = z.object({ | |
| 11 | 18 | after: z.string().optional(), |
| 12 | 19 | before: z.string().optional(), |
| 13 | 20 | category: z.string().optional(), |
@@ -18,56 +25,66 @@ const eventsQuery = z.object({ | ||
| 18 | 25 | cluster: z.string().optional(), |
| 19 | 26 | importance_min: z.coerce.number().min(0).max(100).optional(), |
| 20 | 27 | confidence_min: z.coerce.number().min(0).max(100).optional(), |
| 28 | + signal_min: z.coerce.number().min(0).max(100).optional(), | |
| 21 | 29 | event_type: z.string().optional(), |
| 22 | − silent_change: z | |
| 23 | − .enum(["true", "false"]) | |
| 24 | − .transform((v) => v === "true") | |
| 25 | − .optional(), | |
| 26 | − q: z.string().max(200).optional(), | |
| 30 | + group: z.string().optional(), | |
| 31 | + silent_change: bool, | |
| 32 | + first_party: bool, | |
| 33 | + confirmed: bool, | |
| 34 | + country: z.string().max(3).optional(), | |
| 35 | + language: z.string().max(2).optional(), | |
| 36 | + change_class: z.string().optional(), | |
| 37 | + q: z.string().max(300).optional(), | |
| 27 | 38 | limit: z.coerce.number().int().min(1).max(200).default(50), |
| 28 | 39 | cursor: z.string().optional(), |
| 29 | − order: z.enum(["recent", "importance"]).default("recent"), | |
| 40 | + order: z.enum(["recent", "importance", "signal"]).default("recent"), | |
| 30 | 41 | }); |
| 31 | 42 | |
| 32 | −function ownerToken(headers: Record<string, unknown>): string | null { | |
| 33 | − const t = headers["x-websensor-owner"]; | |
| 34 | − return typeof t === "string" && /^[A-Za-z0-9_-]{16,80}$/.test(t) ? t : null; | |
| 35 | −} | |
| 36 | − | |
| 37 | 43 | export async function registerRoutes(app: FastifyInstance): Promise<void> { |
| 38 | 44 | // ---- Health --------------------------------------------------------------------------- |
| 39 | 45 | app.get("/api/health", async () => ({ status: "ok", service: "api", version: config.version, time: new Date().toISOString() })); |
| 40 | 46 | app.get("/api/ready", async (_req, reply) => { |
| 41 | − const checks: Record<string, boolean | number> = {}; | |
| 47 | + const checks: Record<string, boolean | number | string | null> = {}; | |
| 48 | + const t0 = Date.now(); | |
| 42 | 49 | try { |
| 43 | 50 | await db.execute(sql`select 1`); |
| 44 | 51 | checks.database = true; |
| 52 | + checks.database_latency_ms = Date.now() - t0; | |
| 45 | 53 | } catch { |
| 46 | 54 | checks.database = false; |
| 47 | 55 | } |
| 48 | − const engine = await db.execute<{ last: Date | null }>(sql`select max(started_at) as last from sensor_runs where started_at >= now() - interval '15 minutes'`); | |
| 56 | + const engine = await db.execute<{ last: Date | null }>(sql`select max(started_at) as last from sensor_runs where started_at >= now() - interval '15 minutes'`).catch(() => ({ rows: [{ last: null }] })); | |
| 49 | 57 | checks.engine_recent = Boolean(engine.rows[0]?.last); |
| 58 | + const es = await engineStatus(); | |
| 59 | + checks.engine_heartbeat = es ? String(es.at) : null; | |
| 50 | 60 | checks.ws_clients = liveStats().clients; |
| 51 | 61 | return reply.status(checks.database ? 200 : 503).send({ status: checks.database ? "ready" : "degraded", checks }); |
| 52 | 62 | }); |
| 53 | 63 | |
| 54 | 64 | // ---- Events --------------------------------------------------------------------------- |
| 55 | − app.get("/api/v1/events", async (req) => { | |
| 65 | + app.get("/api/v1/events", async (req, reply) => { | |
| 66 | + const q = eventsQuery.parse(req.query); | |
| 67 | + const page = await listEvents(q); | |
| 68 | + reply.header("x-websensor-order", q.order); | |
| 69 | + return { ...page, meta: { limit: q.limit, order: q.order, filters: stripEmpty({ ...q, limit: undefined, cursor: undefined, order: undefined }) } }; | |
| 70 | + }); | |
| 71 | + app.get("/api/v1/events/count", async (req) => { | |
| 56 | 72 | const q = eventsQuery.parse(req.query); |
| 57 | − return listEvents(q); | |
| 73 | + return { count: await countEvents(q) }; | |
| 58 | 74 | }); |
| 59 | 75 | app.get<{ Params: { id: string } }>("/api/v1/events/:id", async (req, reply) => { |
| 60 | 76 | const ev = await getEvent(req.params.id); |
| 61 | 77 | if (!ev) return reply.status(404).send({ error: "not_found" }); |
| 62 | − const [related, cluster, change, interp] = await Promise.all([ | |
| 78 | + const [related, cluster, change, interp, history] = await Promise.all([ | |
| 63 | 79 | relatedEvents(ev), |
| 64 | 80 | ev.cluster_id ? db.execute<Record<string, unknown>>(sql`select * from event_clusters where id = ${String(ev.cluster_id)}`).then((r) => r.rows[0] ?? null) : null, |
| 65 | − ev.change_id ? db.execute<Record<string, unknown>>(sql`select id, kind, diff, signal, noise_ratio, magnitude, heuristic, detected_at, old_snapshot_id, new_snapshot_id from changes where id = ${String(ev.change_id)}`).then((r) => r.rows[0] ?? null) : null, | |
| 81 | + ev.change_id ? db.execute<Record<string, unknown>>(sql`select id, kind, diff, signal, noise_ratio, magnitude, heuristic, detected_at, old_snapshot_id, new_snapshot_id, change_class, field_changes from changes where id = ${String(ev.change_id)}`).then((r) => r.rows[0] ?? null) : null, | |
| 66 | 82 | db.execute<Record<string, unknown>>(sql`select version, model, created_at from interpretations where event_id = ${String(ev.id)} order by version`).then((r) => r.rows), |
| 83 | + sensorHistory(String(ev.sensor_id), String(ev.id)), | |
| 67 | 84 | ]); |
| 68 | − const snaps = await db.execute<Record<string, unknown>>(sql`select id, url, captured_at, http_status, content_type, content_length, content_hash, canonical_hash, etag, last_modified, title, mode, fetch_duration_ms, extraction_confidence from snapshots where id in (${String(ev.old_snapshot_id ?? "")}, ${String(ev.new_snapshot_id ?? "")})`); | |
| 85 | + const snaps = await db.execute<Record<string, unknown>>(sql`select id, url, captured_at, http_status, content_type, content_length, content_hash, canonical_hash, etag, last_modified, title, mode, fetch_duration_ms, extraction_confidence, storage_key is not null as has_raw from snapshots where id in (${String(ev.old_snapshot_id ?? "")}, ${String(ev.new_snapshot_id ?? "")})`); | |
| 69 | 86 | const sourceRel = await db.execute<Record<string, unknown>>(sql`select health, success_rate, avg_latency_ms, total_runs, raw_changes, meaningful_changes, last_check_at from (select s.health, ch.success_rate, s.avg_latency_ms, s.total_runs, s.raw_changes, s.meaningful_changes, s.last_check_at from sensors s left join connector_health ch on ch.connector = s.connector where s.id = ${String(ev.sensor_id)}) x`); |
| 70 | − return { event: ev, related, cluster, change, interpretations: interp, snapshots: snaps.rows, sensor_reliability: sourceRel.rows[0] ?? null }; | |
| 87 | + return { event: ev, related, cluster, change, interpretations: interp, snapshots: snaps.rows, sensor_reliability: sourceRel.rows[0] ?? null, history }; | |
| 71 | 88 | }); |
| 72 | 89 | |
| 73 | 90 | // ---- Changes / snapshots / diffs ------------------------------------------------------- |
@@ -84,7 +101,8 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> { | ||
| 84 | 101 | const s = r.rows[0]; |
| 85 | 102 | if (!s) return reply.status(404).send({ error: "not_found" }); |
| 86 | 103 | const store = getBlobStore(); |
| 87 | − if (req.query.raw === "1" && s.storage_key) { | |
| 104 | + if (req.query.raw === "1") { | |
| 105 | + if (!s.storage_key) return reply.status(410).send({ error: "raw_body_pruned", detail: "The raw body of this snapshot was pruned by the retention policy; the canonical representation and hashes are preserved." }); | |
| 88 | 106 | const buf = await store.get(String(s.storage_key)); |
| 89 | 107 | reply.header("content-type", String(s.content_type ?? "text/plain") + (String(s.content_type ?? "").includes("charset") ? "" : "; charset=utf-8")); |
| 90 | 108 | reply.header("x-content-type-options", "nosniff"); |
@@ -108,76 +126,111 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> { | ||
| 108 | 126 | const d = diffText(ta, tb, `${a}@${String(sa.captured_at)}`, `${b}@${String(sb.captured_at)}`); |
| 109 | 127 | return { a: sa, b: sb, before: ta, after: tb, diff: { unified: d.unified, stats: d.stats, added: d.added, removed: d.removed, modified: d.modified } }; |
| 110 | 128 | }); |
| 129 | + /** Historical memory (spec §79): how a sensor's page looked at any point — list of snapshots with day grouping. */ | |
| 130 | + app.get<{ Params: { id: string }; Querystring: { limit?: string; before?: string } }>("/api/v1/sensors/:id/snapshots", async (req) => { | |
| 131 | + const lim = Math.min(500, Number(req.query.limit ?? 200)); | |
| 132 | + const rows = await db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_length, canonical_hash, title, mode, storage_key is not null as has_raw, (select count(*) from changes c where c.new_snapshot_id = snapshots.id) > 0 as has_change, (select e.slug from events e where e.new_snapshot_id = snapshots.id limit 1) as event_slug from snapshots where sensor_id = ${req.params.id} ${req.query.before ? sql`and captured_at < ${new Date(req.query.before)}` : sql``} order by captured_at desc limit ${lim}`); | |
| 133 | + return { items: rows.rows }; | |
| 134 | + }); | |
| 111 | 135 | |
| 112 | 136 | // ---- Sources & sensors ------------------------------------------------------------------ |
| 113 | − app.get<{ Querystring: { category?: string; q?: string; limit?: string } }>("/api/v1/sources", async (req) => { | |
| 137 | + app.get<{ Querystring: { category?: string; q?: string; limit?: string; country?: string; tier?: string; first_party?: string } }>("/api/v1/sources", async (req) => { | |
| 114 | 138 | const cat = req.query.category; |
| 115 | 139 | const q = req.query.q; |
| 116 | 140 | const rows = await db.execute<Record<string, unknown>>(sql` |
| 117 | − select s.*, (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count, | |
| 141 | + select s.id, s.name, s.domain, s.homepage, s.description, s.categories, s.tier, s.importance_weight, s.enabled, s.notes, s.first_party, s.country, s.language, s.robots_checked_at, | |
| 142 | + (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count, | |
| 118 | 143 | (select count(*) from events e where e.source_id = s.id)::int as event_count, |
| 119 | 144 | (select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h, |
| 120 | 145 | (select max(detected_at) from events e where e.source_id = s.id) as last_event_at, |
| 121 | 146 | (select max(last_check_at) from sensors x where x.source_id = s.id) as last_check_at, |
| 122 | 147 | (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.health <> 'UP')::int as sensors_degraded |
| 123 | − from sources s where s.enabled ${cat ? sql`and ${cat} = any(s.categories)` : sql``} ${q ? sql`and (s.name ilike ${"%" + q + "%"} or s.domain ilike ${"%" + q + "%"})` : sql``} | |
| 124 | − order by s.tier asc, events_24h desc, s.name asc limit ${Math.min(500, Number(req.query.limit ?? 300))}`); | |
| 148 | + from sources s where s.enabled and s.kind = 'registry' ${cat ? sql`and ${cat} = any(s.categories)` : sql``} ${q ? sql`and (s.name ilike ${"%" + q + "%"} or s.domain ilike ${"%" + q + "%"})` : sql``} ${req.query.country ? sql`and s.country = ${req.query.country.toUpperCase()}` : sql``} ${req.query.tier ? sql`and s.tier = ${req.query.tier.toUpperCase()}` : sql``} ${req.query.first_party ? sql`and s.first_party = ${req.query.first_party === "true"}` : sql``} | |
| 149 | + order by s.tier asc, events_24h desc, s.name asc limit ${Math.min(5000, Number(req.query.limit ?? 300))}`); | |
| 125 | 150 | return { items: rows.rows }; |
| 126 | 151 | }); |
| 127 | 152 | app.get<{ Params: { id: string } }>("/api/v1/sources/:id", async (req, reply) => { |
| 128 | − const s = (await db.execute<Record<string, unknown>>(sql`select * from sources where id = ${req.params.id} or domain = ${req.params.id} limit 1`)).rows[0]; | |
| 153 | + const s = (await db.execute<Record<string, unknown>>(sql`select * from sources where (id = ${req.params.id} or domain = ${req.params.id}) and kind = 'registry' limit 1`)).rows[0]; | |
| 129 | 154 | if (!s) return reply.status(404).send({ error: "not_found" }); |
| 130 | − const [sensors, ents, activity, candidates] = await Promise.all([ | |
| 131 | − db.execute<Record<string, unknown>>(sql`select id, name, url, type, connector, tier, health, enabled, next_check_at, last_check_at, last_change_at, last_event_at, last_status, last_error, consecutive_errors, total_runs, total_not_modified, raw_changes, meaningful_changes, avg_latency_ms, base_interval_seconds, etag is not null as has_etag, last_modified is not null as has_last_modified from sensors where source_id = ${String(s.id)} order by tier, name`).then((r) => r.rows), | |
| 155 | + const { owner_token: _o, ...pub } = s; | |
| 156 | + const [sensors, ents, activity, quality, candidates, byType, series] = await Promise.all([ | |
| 157 | + sourceSensors(String(s.id)), | |
| 132 | 158 | db.execute<Record<string, unknown>>(sql`select en.id, en.name, en.type, en.importance, en.event_count from source_entities se join entities en on en.id = se.entity_id where se.source_id = ${String(s.id)} order by en.type, en.name`).then((r) => r.rows), |
| 133 | 159 | sourceActivity(String(s.id)), |
| 160 | + sourceQuality(String(s.id)), | |
| 134 | 161 | db.execute<Record<string, unknown>>(sql`select url, kind, evidence, score, status, found_at from discovery_candidates where source_id = ${String(s.id)} order by (score->>'value')::float desc nulls last limit 50`).then((r) => r.rows), |
| 162 | + db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where source_id = ${String(s.id)} group by 1 order by 2 desc limit 16`).then((r) => r.rows), | |
| 163 | + db.execute<Record<string, unknown>>(sql`select day::text as day, checks, not_modified, errors, raw_changes, events from source_daily where source_id = ${String(s.id)} and day >= (now() at time zone 'UTC')::date - 30 order by day`).then((r) => r.rows), | |
| 135 | 164 | ]); |
| 136 | − return { source: s, sensors, entities: ents, activity, discovery: candidates }; | |
| 165 | + return { source: pub, sensors, entities: ents, activity, quality, discovery: candidates, by_type: byType, daily: series }; | |
| 137 | 166 | }); |
| 138 | 167 | app.get<{ Params: { id: string } }>("/api/v1/sensors/:id", async (req, reply) => { |
| 139 | − const s = (await db.execute<Record<string, unknown>>(sql`select s.*, so.name as source_name, so.domain from sensors s join sources so on so.id = s.source_id where s.id = ${req.params.id}`)).rows[0]; | |
| 140 | − if (!s) return reply.status(404).send({ error: "not_found" }); | |
| 141 | − const [runs, snaps, changes] = await Promise.all([ | |
| 168 | + const s = (await db.execute<Record<string, unknown>>(sql`select s.*, so.name as source_name, so.domain, so.kind as source_kind from sensors s join sources so on so.id = s.source_id where s.id = ${req.params.id}`)).rows[0]; | |
| 169 | + if (!s || s.source_kind === "custom") return reply.status(404).send({ error: "not_found" }); | |
| 170 | + const [runs, snaps, changes, events] = await Promise.all([ | |
| 142 | 171 | db.execute<Record<string, unknown>>(sql`select id, started_at, finished_at, http_status, outcome, error, duration_ms, bytes, fetch_method, snapshot_id from sensor_runs where sensor_id = ${String(s.id)} order by started_at desc limit 50`).then((r) => r.rows), |
| 143 | − db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_type, content_length, canonical_hash, title, mode, extraction_confidence from snapshots where sensor_id = ${String(s.id)} order by captured_at desc limit 50`).then((r) => r.rows), | |
| 144 | − db.execute<Record<string, unknown>>(sql`select id, detected_at, kind, signal, noise_ratio, magnitude, meaningful, event_id, old_snapshot_id, new_snapshot_id, heuristic->>'eventType' as heuristic_type from changes where sensor_id = ${String(s.id)} order by detected_at desc limit 50`).then((r) => r.rows), | |
| 172 | + db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_type, content_length, canonical_hash, title, mode, extraction_confidence, storage_key is not null as has_raw from snapshots where sensor_id = ${String(s.id)} order by captured_at desc limit 50`).then((r) => r.rows), | |
| 173 | + db.execute<Record<string, unknown>>(sql`select id, detected_at, kind, signal, noise_ratio, magnitude, meaningful, event_id, old_snapshot_id, new_snapshot_id, change_class, field_changes, heuristic->>'eventType' as heuristic_type from changes where sensor_id = ${String(s.id)} order by detected_at desc limit 50`).then((r) => r.rows), | |
| 174 | + listEvents({ sensor: String(s.id), limit: 30 }).then((r) => r.items), | |
| 145 | 175 | ]); |
| 146 | − const { etag: _e, last_modified: _lm, state: _st, ...pub } = s as Record<string, unknown>; | |
| 147 | − return { sensor: pub, runs, snapshots: snaps, changes }; | |
| 176 | + const { state: _st, source_kind: _k, ...pub } = s as Record<string, unknown>; | |
| 177 | + const runs24 = await db.execute<Record<string, unknown>>(sql`select count(*)::int as checks_24h, count(*) filter (where outcome = 'not_modified')::int as not_modified_24h, count(*) filter (where outcome in ('error','parse_error','rate_limited'))::int as errors_24h, avg(duration_ms)::int as avg_ms_24h from sensor_runs where sensor_id = ${String(s.id)} and started_at >= now() - interval '24 hours'`); | |
| 178 | + return { sensor: { ...pub, ...runs24.rows[0] }, runs, snapshots: snaps, changes, events }; | |
| 148 | 179 | }); |
| 149 | 180 | |
| 150 | 181 | // ---- Entities ---------------------------------------------------------------------------- |
| 151 | − app.get<{ Querystring: { type?: string; q?: string; limit?: string } }>("/api/v1/entities", async (req) => { | |
| 182 | + app.get<{ Querystring: { type?: string; q?: string; limit?: string; category?: string } }>("/api/v1/entities", async (req) => { | |
| 152 | 183 | const rows = await db.execute<Record<string, unknown>>(sql` |
| 153 | 184 | select en.*, (select count(*) from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = en.id and e.detected_at >= now() - interval '24 hours')::int as events_24h |
| 154 | − from entities en where true ${req.query.type ? sql`and en.type = ${req.query.type}` : sql``} ${req.query.q ? sql`and (en.search @@ plainto_tsquery('simple', ${req.query.q}) or en.name ilike ${"%" + req.query.q + "%"})` : sql``} | |
| 155 | − order by en.event_count desc, en.importance desc, en.name limit ${Math.min(500, Number(req.query.limit ?? 200))}`); | |
| 185 | + from entities en where true ${req.query.type ? sql`and en.type = ${req.query.type}` : sql``} ${req.query.category ? sql`and ${req.query.category} = any(en.categories)` : sql``} ${req.query.q ? sql`and (en.search @@ plainto_tsquery('simple', ${req.query.q}) or en.name ilike ${"%" + req.query.q + "%"})` : sql``} | |
| 186 | + order by en.event_count desc, en.importance desc, en.name limit ${Math.min(1000, Number(req.query.limit ?? 200))}`); | |
| 156 | 187 | return { items: rows.rows }; |
| 157 | 188 | }); |
| 189 | + app.get<{ Querystring: { limit?: string } }>("/api/v1/entities/rank", async (req) => ({ items: await entityRankings(Math.min(500, Number(req.query.limit ?? 100))) })); | |
| 158 | 190 | app.get<{ Params: { id: string } }>("/api/v1/entities/:id", async (req, reply) => { |
| 159 | 191 | const e = (await db.execute<Record<string, unknown>>(sql`select * from entities where id = ${req.params.id} or id = ${"org_" + req.params.id} limit 1`)).rows[0]; |
| 160 | 192 | if (!e) return reply.status(404).send({ error: "not_found" }); |
| 161 | 193 | const id = String(e.id); |
| 162 | − const [children, relations, sources, aliases, recent, byType] = await Promise.all([ | |
| 194 | + const [children, relations, sources, aliases, recent, byType, insights, silent, parent] = await Promise.all([ | |
| 163 | 195 | db.execute<Record<string, unknown>>(sql`select id, name, type, importance, event_count, last_event_at from entities where parent_id = ${id} order by event_count desc, name`).then((r) => r.rows), |
| 164 | 196 | db.execute<Record<string, unknown>>(sql`select r.relation, r.from_id, r.to_id, f.name as from_name, t.name as to_name, t.type as to_type from entity_relations r join entities f on f.id = r.from_id join entities t on t.id = r.to_id where r.from_id = ${id} or r.to_id = ${id} limit 100`).then((r) => r.rows), |
| 165 | − db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier from source_entities se join sources s on s.id = se.source_id where se.entity_id = ${id}`).then((r) => r.rows), | |
| 197 | + db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.tier, s.first_party, s.country, (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count, (select count(*) from events ev where ev.source_id = s.id and ev.detected_at >= now() - interval '24 hours')::int as events_24h from source_entities se join sources s on s.id = se.source_id where se.entity_id = ${id} and s.kind = 'registry'`).then((r) => r.rows), | |
| 166 | 198 | db.execute<{ alias: string }>(sql`select alias from entity_aliases where entity_id = ${id} order by alias`).then((r) => r.rows.map((x) => x.alias)), |
| 167 | 199 | listEvents({ entity: id, limit: 30 }), |
| 168 | 200 | db.execute<Record<string, unknown>>(sql`select e.event_type, count(*)::int as n from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = ${id} group by 1 order by 2 desc`).then((r) => r.rows), |
| 201 | + entityInsights(id), | |
| 202 | + listEvents({ entity: id, silent_change: true, limit: 10 }).then((r) => r.items), | |
| 203 | + e.parent_id ? db.execute<Record<string, unknown>>(sql`select id, name, type from entities where id = ${String(e.parent_id)}`).then((r) => r.rows[0] ?? null) : null, | |
| 169 | 204 | ]); |
| 170 | − return { entity: e, children, relations, sources, aliases, recent: recent.items, by_type: byType }; | |
| 205 | + // Related entities: co-occurring in events over 30 days | |
| 206 | + const related = await db.execute<Record<string, unknown>>(sql` | |
| 207 | + select en.id, en.name, en.type, count(*)::int as shared_events from event_entities a join event_entities b on a.event_id = b.event_id and b.entity_id <> a.entity_id join events e on e.id = a.event_id join entities en on en.id = b.entity_id | |
| 208 | + where a.entity_id = ${id} and e.detected_at >= now() - interval '30 days' group by en.id, en.name, en.type order by shared_events desc limit 12`).then((r) => r.rows); | |
| 209 | + return { entity: e, parent, children, relations, sources, aliases, recent: recent.items, nextCursor: recent.nextCursor, by_type: byType, insights, silent, related }; | |
| 171 | 210 | }); |
| 172 | − app.get<{ Params: { id: string }; Querystring: { limit?: string; cursor?: string } }>("/api/v1/entities/:id/timeline", async (req) => { | |
| 211 | + app.get<{ Params: { id: string }; Querystring: { limit?: string; cursor?: string; after?: string; before?: string; event_type?: string; silent_change?: string } }>("/api/v1/entities/:id/timeline", async (req) => { | |
| 173 | 212 | const id = req.params.id.startsWith("org_") || req.params.id.startsWith("prd_") ? req.params.id : `org_${req.params.id}`; |
| 174 | − return listEvents({ entity: id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor }); | |
| 213 | + return listEvents({ entity: id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor, after: req.query.after, before: req.query.before, event_type: req.query.event_type, silent_change: req.query.silent_change === "true" ? true : undefined }); | |
| 214 | + }); | |
| 215 | + | |
| 216 | + // ---- Clusters ------------------------------------------------------------------------------ | |
| 217 | + app.get<{ Querystring: { limit?: string; since?: string; state?: string; min_events?: string } }>("/api/v1/clusters", async (req) => { | |
| 218 | + const rows = await db.execute<Record<string, unknown>>(sql` | |
| 219 | + select c.*, (select json_agg(json_build_object('id', e.id, 'slug', e.slug, 'title', e.title, 'importance', e.importance, 'event_type', e.event_type, 'detected_at', e.detected_at, 'source_id', e.source_id, 'url', e.url, 'first_party', e.first_party) order by e.importance desc) from events e where e.cluster_id = c.id) as events, | |
| 220 | + (select json_build_object('id', s.id, 'name', s.name, 'domain', s.domain) from events e join sources s on s.id = e.source_id where e.id = c.primary_event_id) as source | |
| 221 | + from event_clusters c where c.last_at >= now() - make_interval(hours => ${Math.min(720, Number(req.query.since ?? 72))}) ${req.query.state ? sql`and c.state = ${req.query.state}` : sql``} ${req.query.min_events ? sql`and c.event_count >= ${Number(req.query.min_events)}` : sql``} order by c.last_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`); | |
| 222 | + return { items: rows.rows }; | |
| 223 | + }); | |
| 224 | + app.get<{ Params: { id: string } }>("/api/v1/clusters/:id", async (req, reply) => { | |
| 225 | + const d = await clusterDetail(req.params.id); | |
| 226 | + if (!d) return reply.status(404).send({ error: "not_found" }); | |
| 227 | + return d; | |
| 175 | 228 | }); |
| 176 | 229 | |
| 177 | 230 | // ---- Domains / URLs ---------------------------------------------------------------------- |
| 178 | 231 | app.get<{ Params: { domain: string }; Querystring: { limit?: string; cursor?: string } }>("/api/v1/domains/:domain/timeline", async (req) => { |
| 179 | 232 | const d = req.params.domain.toLowerCase(); |
| 180 | − const src = (await db.execute<{ id: string }>(sql`select id from sources where domain = ${d} or domain = ${"www." + d} or ${d} = 'www.' || domain limit 1`)).rows[0]; | |
| 233 | + const src = (await db.execute<{ id: string }>(sql`select id from sources where (domain = ${d} or domain = ${"www." + d} or ${d} = 'www.' || domain) and kind = 'registry' limit 1`)).rows[0]; | |
| 181 | 234 | const urls = await db.execute<Record<string, unknown>>(sql`select url, status, first_seen_at, last_seen_at, snapshot_count, change_count, sensor_id from urls where domain = ${d} or domain like ${"%." + d} order by change_count desc, last_seen_at desc limit 200`); |
| 182 | 235 | const ev = src ? await listEvents({ source: src.id, limit: Math.min(200, Number(req.query.limit ?? 100)), cursor: req.query.cursor }) : { items: [], nextCursor: null }; |
| 183 | 236 | return { domain: d, source_id: src?.id ?? null, urls: urls.rows, events: ev.items, nextCursor: ev.nextCursor }; |
@@ -186,149 +239,103 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> { | ||
| 186 | 239 | if (!req.query.url) return reply.status(400).send({ error: "url required" }); |
| 187 | 240 | const u = (await db.execute<Record<string, unknown>>(sql`select * from urls where url = ${req.query.url}`)).rows[0]; |
| 188 | 241 | const history = await db.execute<Record<string, unknown>>(sql` |
| 189 | − select h.id, h.at, h.kind, h.snapshot_id, h.change_id, h.event_id, h.note, e.title as event_title, e.importance, e.event_type, e.slug as event_slug, c.kind as change_kind, c.signal | |
| 242 | + select h.id, h.at, h.kind, h.snapshot_id, h.change_id, h.event_id, h.note, e.title as event_title, e.importance, e.event_type, e.slug as event_slug, c.kind as change_kind, c.signal, c.change_class | |
| 190 | 243 | from url_history h left join events e on e.id = h.event_id left join changes c on c.id = h.change_id where h.url = ${req.query.url} order by h.at desc limit 300`); |
| 191 | − const snaps = await db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_length, canonical_hash, title from snapshots where url = ${req.query.url} or sensor_id = (select sensor_id from urls where url = ${req.query.url}) order by captured_at desc limit 200`); | |
| 244 | + const snaps = await db.execute<Record<string, unknown>>(sql`select id, captured_at, http_status, content_length, canonical_hash, title, storage_key is not null as has_raw from snapshots where url = ${req.query.url} or sensor_id = (select sensor_id from urls where url = ${req.query.url}) order by captured_at desc limit 200`); | |
| 192 | 245 | return { url: u ?? { url: req.query.url }, history: history.rows, snapshots: snaps.rows }; |
| 193 | 246 | }); |
| 194 | 247 | |
| 195 | − // ---- Explore / trending / stats ------------------------------------------------------------ | |
| 196 | − app.get("/api/v1/stats", async () => stats()); | |
| 197 | − app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => ({ items: await trending(Math.min(168, Number(req.query.hours ?? 24)), Math.min(50, Number(req.query.limit ?? 12))) })); | |
| 198 | − app.get("/api/v1/explore", async () => { | |
| 199 | − const [mostActive, biggest, silent, clusters, unusual, byType, byCategory] = await Promise.all([ | |
| 200 | − db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, count(*)::int as events_24h, max(e.importance)::float as max_importance from events e join sources s on s.id = e.source_id where e.detected_at >= now() - interval '24 hours' group by s.id, s.name, s.domain order by events_24h desc limit 10`).then((r) => r.rows), | |
| 201 | − listEvents({ limit: 10, order: "importance", after: new Date(Date.now() - 48 * 3600e3).toISOString() }).then((r) => r.items), | |
| 202 | − listEvents({ limit: 10, silent_change: true }).then((r) => r.items), | |
| 203 | − db.execute<Record<string, unknown>>(sql`select c.*, (select json_build_object('id', s.id, 'name', s.name, 'domain', s.domain) from events e join sources s on s.id = e.source_id where e.id = c.primary_event_id) as source from event_clusters c where c.event_count >= 2 and c.last_at >= now() - interval '48 hours' order by c.max_importance desc, c.event_count desc limit 10`).then((r) => r.rows), | |
| 204 | − db.execute<Record<string, unknown>>(sql` | |
| 205 | − with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '2 hours' group by s.source_id), | |
| 206 | − base as (select s.source_id, count(*)::float / (14*24) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id) | |
| 207 | − select so.id, so.name, so.domain, cur.n::int as changes_2h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day, | |
| 208 | − round((case when coalesce(base.per_hour,0) = 0 then (case when cur.n/2.0 > 2 then 70 else 40 end) else least(100, case when cur.n/2.0/base.per_hour <= 1 then cur.n/2.0/base.per_hour*30 else 30 + 25*(ln(cur.n/2.0/base.per_hour)/ln(2)) end) end)::numeric, 1)::float as activity_score | |
| 209 | − from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id order by activity_score desc limit 10`).then((r) => r.rows), | |
| 210 | − db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows), | |
| 211 | − db.execute<Record<string, unknown>>(sql`select c as category, count(*)::int as n from events e, unnest(e.categories) c where e.detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows), | |
| 212 | − ]); | |
| 213 | − return { most_active_sources: mostActive, biggest_changes: biggest, silent_changes: silent, clusters, unusual_activity: unusual, by_type: byType, by_category: byCategory, channels: FEED_CHANNELS, event_types: Object.fromEntries(Object.entries(EVENT_TYPES).map(([k, v]) => [k, v.label])) }; | |
| 248 | + // ---- Intelligence desks ---------------------------------------------------------------------- | |
| 249 | + app.get("/api/v1/stats", async () => cached("stats", 5_000, stats)); | |
| 250 | + app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => { | |
| 251 | + const hours = Math.min(168, Number(req.query.hours ?? 24)); | |
| 252 | + const limit = Math.min(50, Number(req.query.limit ?? 12)); | |
| 253 | + return { items: await cached(`trending:${hours}:${limit}`, 15_000, () => trending(hours, limit)) }; | |
| 214 | 254 | }); |
| 215 | − app.get<{ Querystring: { limit?: string; since?: string } }>("/api/v1/clusters", async (req) => { | |
| 216 | − const rows = await db.execute<Record<string, unknown>>(sql` | |
| 217 | − select c.*, (select json_agg(json_build_object('id', e.id, 'slug', e.slug, 'title', e.title, 'importance', e.importance, 'event_type', e.event_type, 'detected_at', e.detected_at, 'source_id', e.source_id, 'url', e.url) order by e.importance desc) from events e where e.cluster_id = c.id) as events | |
| 218 | − from event_clusters c where c.last_at >= now() - make_interval(hours => ${Math.min(720, Number(req.query.since ?? 72))}) order by c.last_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`); | |
| 219 | − return { items: rows.rows }; | |
| 255 | + app.get("/api/v1/breaking", async () => breakingDesk()); | |
| 256 | + app.get("/api/v1/pulse", async () => pulse()); | |
| 257 | + app.get("/api/v1/radar", async () => radar()); | |
| 258 | + app.get("/api/v1/countries", async () => ({ items: await countryList(), known: COUNTRIES })); | |
| 259 | + app.get<{ Params: { code: string } }>("/api/v1/countries/:code", async (req, reply) => { | |
| 260 | + const c = countryBySlug(req.params.code) ?? (COUNTRIES[req.params.code.toUpperCase()] ? { code: req.params.code.toUpperCase() } : null); | |
| 261 | + if (!c) return reply.status(404).send({ error: "not_found" }); | |
| 262 | + return cached(`country:${c.code}`, 20_000, () => countryDesk(c.code)); | |
| 263 | + }); | |
| 264 | + app.get<{ Params: { channel: string } }>("/api/v1/categories/:channel", async (req, reply) => { | |
| 265 | + const ch = req.params.channel.toLowerCase(); | |
| 266 | + if (!FEED_CHANNELS[ch] && !/^[a-z-]{2,30}$/.test(ch)) return reply.status(404).send({ error: "not_found" }); | |
| 267 | + return cached(`category:${ch}`, 15_000, () => categoryDesk(ch)); | |
| 220 | 268 | }); |
| 269 | + app.get("/api/v1/explore", async () => | |
| 270 | + cached("explore", 20_000, async () => { | |
| 271 | + const since48 = new Date(Date.now() - 48 * 3600e3).toISOString(); | |
| 272 | + const [mostActive, biggest, silent, clusters, unusual, byType, byCategory, newly, firstParty] = await Promise.all([ | |
| 273 | + db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.domain, s.first_party, count(*)::int as events_24h, max(e.importance)::float as max_importance from events e join sources s on s.id = e.source_id where e.detected_at >= now() - interval '24 hours' and s.kind = 'registry' group by s.id, s.name, s.domain, s.first_party order by events_24h desc limit 12`).then((r) => r.rows), | |
| 274 | + listEvents({ limit: 12, order: "signal", after: since48 }).then((r) => r.items), | |
| 275 | + listEvents({ limit: 12, silent_change: true, order: "signal", after: since48 }).then((r) => r.items), | |
| 276 | + db.execute<Record<string, unknown>>(sql`select c.*, (select json_build_object('id', s.id, 'name', s.name, 'domain', s.domain) from events e join sources s on s.id = e.source_id where e.id = c.primary_event_id) as source, (select e.slug from events e where e.id = c.primary_event_id) as primary_slug from event_clusters c where c.event_count >= 2 and c.last_at >= now() - interval '48 hours' order by c.max_importance desc, c.event_count desc limit 12`).then((r) => r.rows), | |
| 277 | + db.execute<Record<string, unknown>>(sql` | |
| 278 | + with cur as (select s.source_id, count(*) as n from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '2 hours' group by s.source_id), | |
| 279 | + base as (select s.source_id, count(*)::float / (14*24) as per_hour from changes c join sensors s on s.id = c.sensor_id where c.detected_at >= now() - interval '14 days' group by s.source_id) | |
| 280 | + select so.id, so.name, so.domain, cur.n::int as changes_2h, round(coalesce(base.per_hour,0)::numeric*24,1)::float as baseline_per_day, | |
| 281 | + round((case when coalesce(base.per_hour,0) = 0 then (case when cur.n/2.0 > 2 then 70 else 40 end) else least(100, case when cur.n/2.0/base.per_hour <= 1 then cur.n/2.0/base.per_hour*30 else 30 + 25*(ln(cur.n/2.0/base.per_hour)/ln(2)) end) end)::numeric, 1)::float as activity_score, | |
| 282 | + (case when coalesce(base.per_hour,0) > 0 then round(((cur.n/2.0/base.per_hour - 1) * 100)::numeric) else null end)::int as pct_vs_baseline | |
| 283 | + from cur join sources so on so.id = cur.source_id left join base on base.source_id = cur.source_id where so.kind = 'registry' order by activity_score desc limit 12`).then((r) => r.rows), | |
| 284 | + db.execute<Record<string, unknown>>(sql`select event_type, count(*)::int as n from events where detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows), | |
| 285 | + db.execute<Record<string, unknown>>(sql`select c as category, count(*)::int as n from events e, unnest(e.categories) c where e.detected_at >= now() - interval '7 days' group by 1 order by 2 desc`).then((r) => r.rows), | |
| 286 | + listEvents({ limit: 12, order: "recent", confidence_min: 60 }).then((r) => r.items), | |
| 287 | + listEvents({ limit: 12, order: "signal", first_party: true, confirmed: true, after: since48 }).then((r) => r.items), | |
| 288 | + ]); | |
| 289 | + return { most_active_sources: mostActive, biggest_changes: biggest, silent_changes: silent, clusters, unusual_activity: unusual, newly_detected: newly, confirmed_first_party: firstParty, by_type: byType, by_category: byCategory, channels: FEED_CHANNELS, groups: Object.fromEntries(Object.entries(EVENT_GROUPS).map(([k, v]) => [k, v.label])), event_types: Object.fromEntries(Object.entries(EVENT_TYPES).map(([k, v]) => [k, v.label])) }; | |
| 290 | + }), | |
| 291 | + ); | |
| 221 | 292 | |
| 222 | 293 | // ---- Search ---------------------------------------------------------------------------- |
| 223 | 294 | app.get<{ Querystring: { q?: string; limit?: string } }>("/api/v1/search", async (req) => { |
| 224 | 295 | const q = (req.query.q ?? "").trim(); |
| 225 | − if (q.length < 2) return { query: q, events: [], entities: [], sources: [], urls: [] }; | |
| 296 | + const parsed = parseSearch(q); | |
| 297 | + if (q.length < 2) return { query: q, parsed, events: [], entities: [], sources: [], urls: [], clusters: [] }; | |
| 226 | 298 | const lim = Math.min(50, Number(req.query.limit ?? 20)); |
| 227 | − const [ev, ents, srcs, urls] = await Promise.all([ | |
| 228 | − listEvents({ q, limit: lim }).then((r) => r.items), | |
| 229 | − db.execute<Record<string, unknown>>(sql`select id, name, type, domain, importance, event_count from entities where search @@ plainto_tsquery('simple', ${q}) or name ilike ${"%" + q + "%"} order by event_count desc, importance desc limit ${lim}`).then((r) => r.rows), | |
| 230 | − db.execute<Record<string, unknown>>(sql`select id, name, domain, tier, categories from sources where name ilike ${"%" + q + "%"} or domain ilike ${"%" + q + "%"} limit ${lim}`).then((r) => r.rows), | |
| 231 | − db.execute<Record<string, unknown>>(sql`select url, domain, status, change_count, last_seen_at from urls where url ilike ${"%" + q + "%"} order by change_count desc limit ${lim}`).then((r) => r.rows), | |
| 299 | + const free = parsed.text; | |
| 300 | + const [ev, ents, srcs, urls, clusters] = await Promise.all([ | |
| 301 | + listEvents({ q, limit: lim, order: Object.keys(parsed.filters).length && !free ? "recent" : "recent" }).then((r) => r.items), | |
| 302 | + free.length >= 2 ? db.execute<Record<string, unknown>>(sql`select id, name, type, domain, importance, event_count from entities where search @@ plainto_tsquery('simple', ${free}) or name ilike ${"%" + free + "%"} order by event_count desc, importance desc limit ${lim}`).then((r) => r.rows) : [], | |
| 303 | + free.length >= 2 ? db.execute<Record<string, unknown>>(sql`select id, name, domain, tier, categories, first_party, country from sources where kind = 'registry' and (name ilike ${"%" + free + "%"} or domain ilike ${"%" + free + "%"}) limit ${lim}`).then((r) => r.rows) : [], | |
| 304 | + free.length >= 4 ? db.execute<Record<string, unknown>>(sql`select url, domain, status, change_count, last_seen_at from urls where url ilike ${"%" + free + "%"} order by change_count desc limit ${lim}`).then((r) => r.rows) : [], | |
| 305 | + free.length >= 2 ? db.execute<Record<string, unknown>>(sql`select id, slug, title, state, event_count, source_count, max_importance, last_at from event_clusters where title ilike ${"%" + free + "%"} and last_at >= now() - interval '30 days' order by max_importance desc limit 10`).then((r) => r.rows) : [], | |
| 232 | 306 | ]); |
| 233 | − return { query: q, events: ev, entities: ents, sources: srcs, urls }; | |
| 307 | + return { query: q, parsed, events: ev, entities: ents, sources: srcs, urls, clusters }; | |
| 234 | 308 | }); |
| 235 | 309 | |
| 236 | − // ---- Connector health ---------------------------------------------------------------------- | |
| 237 | − app.get("/api/v1/health/connectors", async () => { | |
| 238 | − const [connectors, sensorsByHealth, worst, noisy, daily] = await Promise.all([ | |
| 239 | − db.execute<Record<string, unknown>>(sql`select * from connector_health order by connector`).then((r) => r.rows), | |
| 240 | − db.execute<Record<string, unknown>>(sql`select health, count(*)::int as n from sensors where enabled group by health`).then((r) => r.rows), | |
| 241 | − db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.url, s.source_id, s.connector, s.health, s.consecutive_errors, s.last_error, s.last_status, s.last_check_at from sensors s where s.enabled and s.health <> 'UP' order by s.consecutive_errors desc, s.last_check_at desc limit 50`).then((r) => r.rows), | |
| 242 | − db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.url, s.raw_changes, s.meaningful_changes, case when s.raw_changes > 0 then round(1 - s.meaningful_changes::numeric / s.raw_changes, 3) else null end as noise_ratio from sensors s where s.raw_changes >= 5 order by noise_ratio desc nulls last, raw_changes desc limit 25`).then((r) => r.rows), | |
| 243 | − db.execute<Record<string, unknown>>(sql`select * from metrics_daily order by day desc limit 30`).then((r) => r.rows), | |
| 244 | − ]); | |
| 245 | − return { connectors, sensors_by_health: sensorsByHealth, degraded_sensors: worst, noisy_sensors: noisy, daily, live: liveStats() }; | |
| 246 | − }); | |
| 247 | − | |
| 248 | − // ---- Watchlists & alerts (anonymous owner token, phase 1) ---------------------------------- | |
| 249 | − app.get("/api/v1/watchlists", async (req, reply) => { | |
| 250 | − const owner = ownerToken(req.headers as Record<string, unknown>); | |
| 251 | − if (!owner) return reply.status(401).send({ error: "owner token required (X-WebSensor-Owner)" }); | |
| 252 | − const rows = await db.execute<Record<string, unknown>>(sql`select w.*, coalesce((select json_agg(json_build_object('kind', i.kind, 'value', i.value, 'added_at', i.added_at)) from watchlist_items i where i.watchlist_id = w.id), '[]'::json) as items from watchlists w where owner_token = ${owner} order by created_at`); | |
| 253 | − return { items: rows.rows }; | |
| 254 | − }); | |
| 255 | − app.post<{ Body: { name?: string; items?: { kind: string; value: string }[] } }>("/api/v1/watchlists", async (req, reply) => { | |
| 256 | − const owner = ownerToken(req.headers as Record<string, unknown>); | |
| 257 | − if (!owner) return reply.status(401).send({ error: "owner token required" }); | |
| 258 | − const body = z.object({ name: z.string().min(1).max(80).default("My watchlist"), items: z.array(z.object({ kind: z.enum(["entity", "source", "keyword", "category", "url"]), value: z.string().min(1).max(200) })).max(200).default([]) }).parse(req.body ?? {}); | |
| 259 | − const count = (await db.execute<{ n: string }>(sql`select count(*)::text as n from watchlists where owner_token = ${owner}`)).rows[0]?.n; | |
| 260 | − if (Number(count) >= 20) return reply.status(429).send({ error: "too many watchlists" }); | |
| 261 | − const id = newId("wl"); | |
| 262 | − await db.execute(sql`insert into watchlists (id, owner_token, name) values (${id}, ${owner}, ${body.name})`); | |
| 263 | − for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${id}, ${it.kind}, ${it.value}) on conflict do nothing`); | |
| 264 | − return { id, name: body.name, items: body.items }; | |
| 265 | − }); | |
| 266 | − app.put<{ Params: { id: string }; Body: { name?: string; items?: { kind: string; value: string }[] } }>("/api/v1/watchlists/:id", async (req, reply) => { | |
| 267 | − const owner = ownerToken(req.headers as Record<string, unknown>); | |
| 268 | − if (!owner) return reply.status(401).send({ error: "owner token required" }); | |
| 269 | − const w = (await db.execute<{ id: string }>(sql`select id from watchlists where id = ${req.params.id} and owner_token = ${owner}`)).rows[0]; | |
| 270 | − if (!w) return reply.status(404).send({ error: "not_found" }); | |
| 271 | − const body = z.object({ name: z.string().min(1).max(80).optional(), items: z.array(z.object({ kind: z.enum(["entity", "source", "keyword", "category", "url"]), value: z.string().min(1).max(200) })).max(200).optional() }).parse(req.body ?? {}); | |
| 272 | − if (body.name) await db.execute(sql`update watchlists set name = ${body.name} where id = ${w.id}`); | |
| 273 | − if (body.items) { | |
| 274 | − await db.execute(sql`delete from watchlist_items where watchlist_id = ${w.id}`); | |
| 275 | − for (const it of body.items) await db.execute(sql`insert into watchlist_items (watchlist_id, kind, value) values (${w.id}, ${it.kind}, ${it.value}) on conflict do nothing`); | |
| 276 | − } | |
| 277 | − return { ok: true }; | |
| 278 | − }); | |
| 279 | − app.delete<{ Params: { id: string } }>("/api/v1/watchlists/:id", async (req, reply) => { | |
| 280 | − const owner = ownerToken(req.headers as Record<string, unknown>); | |
| 281 | − if (!owner) return reply.status(401).send({ error: "owner token required" }); | |
| 282 | − await db.execute(sql`delete from watchlists where id = ${req.params.id} and owner_token = ${owner}`); | |
| 283 | − return { ok: true }; | |
| 284 | − }); | |
| 285 | − app.get<{ Params: { id: string }; Querystring: { limit?: string } }>("/api/v1/watchlists/:id/events", async (req, reply) => { | |
| 286 | − const owner = ownerToken(req.headers as Record<string, unknown>); | |
| 287 | − if (!owner) return reply.status(401).send({ error: "owner token required" }); | |
| 288 | − const items = (await db.execute<{ kind: string; value: string }>(sql`select i.kind, i.value from watchlist_items i join watchlists w on w.id = i.watchlist_id where w.id = ${req.params.id} and w.owner_token = ${owner}`)).rows; | |
| 289 | − if (!items.length) return reply.send({ items: [] }); | |
| 290 | − const ents = items.filter((i) => i.kind === "entity").map((i) => i.value); | |
| 291 | − const srcs = items.filter((i) => i.kind === "source").map((i) => i.value); | |
| 292 | − const cats = items.filter((i) => i.kind === "category").map((i) => i.value); | |
| 293 | − const kws = items.filter((i) => i.kind === "keyword").map((i) => i.value); | |
| 294 | − const conds = [] as ReturnType<typeof sql>[]; | |
| 295 | − if (ents.length) conds.push(sql`exists (select 1 from event_entities x where x.event_id = e.id and x.entity_id = any(${textArray(ents)}))`); | |
| 296 | − if (srcs.length) conds.push(sql`e.source_id = any(${textArray(srcs)})`); | |
| 297 | − if (cats.length) conds.push(sql`e.categories && ${textArray(cats)}`); | |
| 298 | − for (const k of kws) conds.push(sql`(e.title ilike ${"%" + k + "%"} or e.summary ilike ${"%" + k + "%"})`); | |
| 299 | − if (!conds.length) return { items: [] }; | |
| 300 | − const rows = await db.execute<Record<string, unknown>>(sql`select ${EVENT_SELECT} from events e join sources s on s.id = e.source_id join sensors sen on sen.id = e.sensor_id where ${sql.join(conds, sql` or `)} order by e.detected_at desc limit ${Math.min(200, Number(req.query.limit ?? 50))}`); | |
| 301 | − return { items: rows.rows }; | |
| 302 | − }); | |
| 303 | − | |
| 304 | − app.get("/api/v1/alerts", async (req, reply) => { | |
| 305 | − const owner = ownerToken(req.headers as Record<string, unknown>); | |
| 306 | − if (!owner) return reply.status(401).send({ error: "owner token required" }); | |
| 307 | − return { items: (await db.execute<Record<string, unknown>>(sql`select * from alerts where owner_token = ${owner} order by created_at`)).rows }; | |
| 308 | − }); | |
| 309 | − app.post<{ Body: Record<string, unknown> }>("/api/v1/alerts", async (req, reply) => { | |
| 310 | − const owner = ownerToken(req.headers as Record<string, unknown>); | |
| 311 | − if (!owner) return reply.status(401).send({ error: "owner token required" }); | |
| 312 | − const body = z.object({ name: z.string().min(1).max(80), rule: z.object({ importance_min: z.number().min(0).max(100).optional(), event_types: z.array(z.string()).optional(), entities: z.array(z.string()).optional(), sources: z.array(z.string()).optional(), keywords: z.array(z.string()).optional(), silent_only: z.boolean().optional(), categories: z.array(z.string()).optional() }), channel: z.enum(["web"]).default("web") }).parse(req.body ?? {}); | |
| 313 | − const id = newId("alr"); | |
| 314 | − await db.execute(sql`insert into alerts (id, owner_token, name, rule, channel) values (${id}, ${owner}, ${body.name}, ${JSON.stringify(body.rule)}::jsonb, ${body.channel})`); | |
| 315 | − return { id, ...body }; | |
| 316 | − }); | |
| 317 | − app.delete<{ Params: { id: string } }>("/api/v1/alerts/:id", async (req, reply) => { | |
| 318 | − const owner = ownerToken(req.headers as Record<string, unknown>); | |
| 319 | − if (!owner) return reply.status(401).send({ error: "owner token required" }); | |
| 320 | − await db.execute(sql`delete from alerts where id = ${req.params.id} and owner_token = ${owner}`); | |
| 321 | − return { ok: true }; | |
| 322 | − }); | |
| 310 | + // ---- Connector health / system -------------------------------------------------------------- | |
| 311 | + app.get("/api/v1/health/connectors", async () => | |
| 312 | + cached("health", 10_000, async () => { | |
| 313 | + const [connectors, sensorsByHealth, sensorsByStatus, worst, noisy, daily, byConnectorSensors, throughput, es, topFailingDomains, slowest] = await Promise.all([ | |
| 314 | + db.execute<Record<string, unknown>>(sql`select * from connector_health order by connector`).then((r) => r.rows), | |
| 315 | + db.execute<Record<string, unknown>>(sql`select health, count(*)::int as n from sensors where enabled group by health`).then((r) => r.rows), | |
| 316 | + db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from sensors group by status`).then((r) => r.rows), | |
| 317 | + db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.url, s.source_id, s.connector, s.health, s.consecutive_errors, s.last_error, s.last_status, s.last_check_at from sensors s where s.enabled and s.health <> 'UP' order by s.consecutive_errors desc, s.last_check_at desc limit 50`).then((r) => r.rows), | |
| 318 | + db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.url, s.raw_changes, s.meaningful_changes, case when s.raw_changes > 0 then round(1 - s.meaningful_changes::numeric / s.raw_changes, 3) else null end as noise_ratio from sensors s where s.raw_changes >= 5 order by noise_ratio desc nulls last, raw_changes desc limit 25`).then((r) => r.rows), | |
| 319 | + db.execute<Record<string, unknown>>(sql`select * from metrics_daily order by day desc limit 30`).then((r) => r.rows), | |
| 320 | + db.execute<Record<string, unknown>>(sql`select connector, count(*)::int as sensors, count(*) filter (where health = 'UP')::int as up, avg(avg_latency_ms)::int as avg_latency_ms from sensors where enabled group by connector order by sensors desc`).then((r) => r.rows), | |
| 321 | + db.execute<Record<string, unknown>>(sql`select (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes')::int as checks_5m, (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified')::int as not_modified_5m, (select count(*) from events where detected_at >= now() - interval '5 minutes')::int as events_5m, (select count(*) from changes where detected_at >= now() - interval '5 minutes')::int as changes_5m, (select count(*) from sensors where enabled and next_check_at <= now())::int as queue_due, (select avg(duration_ms)::int from sensor_runs where started_at >= now() - interval '5 minutes')::int as avg_latency_5m_ms, (select count(*) from changes where detected_at >= now() - interval '24 hours' and change_class in ('cosmetic','navigation','timestamp','advertisement','boilerplate'))::int as noise_filtered_24h`).then((r) => r.rows[0]), | |
| 322 | + engineStatus(), | |
| 323 | + db.execute<Record<string, unknown>>(sql`select split_part(split_part(s.url, '/', 3), ':', 1) as host, count(*)::int as failures, max(r.error) as last_error from sensor_runs r join sensors s on s.id = r.sensor_id where r.started_at >= now() - interval '24 hours' and r.outcome in ('error','parse_error','rate_limited') group by 1 order by failures desc limit 15`).then((r) => r.rows), | |
| 324 | + db.execute<Record<string, unknown>>(sql`select s.id, s.name, s.source_id, s.avg_latency_ms, s.connector from sensors s where s.enabled and s.avg_latency_ms is not null order by s.avg_latency_ms desc limit 15`).then((r) => r.rows), | |
| 325 | + ]); | |
| 326 | + const t = throughput ?? {}; | |
| 327 | + return { connectors, sensors_by_health: sensorsByHealth, sensors_by_status: sensorsByStatus, degraded_sensors: worst, noisy_sensors: noisy, daily, sensors_by_connector: byConnectorSensors, throughput: { ...t, checks_per_min: Math.round(Number(t.checks_5m ?? 0) / 5), events_per_min: Math.round((Number(t.events_5m ?? 0) / 5) * 10) / 10, not_modified_ratio: Number(t.checks_5m) ? Math.round((Number(t.not_modified_5m) / Number(t.checks_5m)) * 100) / 100 : null }, engine: es, top_failing_domains: topFailingDomains, slowest_sensors: slowest, live: liveStats() }; | |
| 328 | + }), | |
| 329 | + ); | |
| 323 | 330 | |
| 324 | 331 | // ---- Machine-readable feeds ------------------------------------------------------------------ |
| 325 | − app.get<{ Querystring: { category?: string; importance_min?: string; silent_change?: string } }>("/api/v1/feed.rss", async (req, reply) => { | |
| 326 | − const q = eventsQuery.parse({ ...req.query, limit: 50 }); | |
| 332 | + app.get("/api/v1/feed.rss", async (req, reply) => { | |
| 333 | + const q = eventsQuery.parse({ ...(req.query as Record<string, unknown>), limit: 50 }); | |
| 327 | 334 | const { items } = await listEvents(q); |
| 328 | 335 | const esc = (s: unknown): string => String(s ?? "").replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">").replace(/"/g, """); |
| 329 | 336 | const base = config.publicBaseUrl; |
| 330 | − const xml = `<?xml version="1.0" encoding="UTF-8"?>\n<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>WebSensor — ${esc(q.category ? q.category + " events" : "live events")}</title><link>${base}</link><description>Meaningful changes detected on the public Web by WebSensor.</description><atom:link href="${base}/api/v1/feed.rss" rel="self" type="application/rss+xml"/>${items | |
| 331 | − .map((e) => `<item><title>${esc(e.title)}</title><link>${base}/event/${esc(e.slug)}</link><guid isPermaLink="false">${esc(e.id)}</guid><pubDate>${new Date(String(e.detected_at)).toUTCString()}</pubDate><category>${esc(e.event_type)}</category><description>${esc(e.summary)} (importance ${esc(e.importance)}, confidence ${esc(e.confidence)}${e.silent_change ? ", silent change" : ""}) — source: ${esc(e.url)}</description></item>`) | |
| 337 | + const xml = `<?xml version="1.0" encoding="UTF-8"?>\n<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>WebSensor — ${esc(q.category ? q.category + " events" : q.silent_change ? "silent changes" : "live events")}</title><link>${base}</link><description>Meaningful changes detected on the public Web by WebSensor.</description><atom:link href="${base}/api/v1/feed.rss" rel="self" type="application/rss+xml"/>${items | |
| 338 | + .map((e) => `<item><title>${esc(e.title)}</title><link>${base}/event/${esc(e.slug)}</link><guid isPermaLink="false">${esc(e.id)}</guid><pubDate>${new Date(String(e.detected_at)).toUTCString()}</pubDate><category>${esc(e.event_type)}</category><description>${esc(e.summary)} (signal ${esc(e.signal_score ?? e.importance)}, importance ${esc(e.importance)}, confidence ${esc(e.confidence)}${e.silent_change ? ", silent change" : ""}${e.first_party ? ", first-party" : ""}) — source: ${esc(e.url)}</description></item>`) | |
| 332 | 339 | .join("")}</channel></rss>`; |
| 333 | 340 | reply.header("content-type", "application/rss+xml; charset=utf-8"); |
| 334 | 341 | return reply.send(xml); |
@@ -337,13 +344,59 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> { | ||
| 337 | 344 | app.get("/api/v1", async () => ({ |
| 338 | 345 | name: "WebSensor API", |
| 339 | 346 | version: "v1", |
| 347 | + release: config.version, | |
| 340 | 348 | docs: `${config.publicBaseUrl}/api`, |
| 341 | − endpoints: ["/api/v1/events", "/api/v1/events/{id|slug}", "/api/v1/changes/{id}", "/api/v1/snapshots/{id}", "/api/v1/snapshots/compare?a=&b=", "/api/v1/sources", "/api/v1/sources/{id}", "/api/v1/sensors/{id}", "/api/v1/entities", "/api/v1/entities/{id}", "/api/v1/entities/{id}/timeline", "/api/v1/domains/{domain}/timeline", "/api/v1/urls/history?url=", "/api/v1/search?q=", "/api/v1/stats", "/api/v1/trending", "/api/v1/explore", "/api/v1/clusters", "/api/v1/health/connectors", "/api/v1/watchlists", "/api/v1/alerts", "/api/v1/feed.rss", "wss://…/api/v1/live"], | |
| 349 | + rate_limit: "600 requests / minute / IP (headers x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-reset)", | |
| 350 | + endpoints: [ | |
| 351 | + "/api/v1/events", | |
| 352 | + "/api/v1/events/count", | |
| 353 | + "/api/v1/events/{id|slug}", | |
| 354 | + "/api/v1/changes/{id}", | |
| 355 | + "/api/v1/snapshots/{id}", | |
| 356 | + "/api/v1/snapshots/compare?a=&b=", | |
| 357 | + "/api/v1/sensors/{id}", | |
| 358 | + "/api/v1/sensors/{id}/snapshots", | |
| 359 | + "/api/v1/sources", | |
| 360 | + "/api/v1/sources/{id}", | |
| 361 | + "/api/v1/entities", | |
| 362 | + "/api/v1/entities/rank", | |
| 363 | + "/api/v1/entities/{id}", | |
| 364 | + "/api/v1/entities/{id}/timeline", | |
| 365 | + "/api/v1/clusters", | |
| 366 | + "/api/v1/clusters/{id|slug}", | |
| 367 | + "/api/v1/domains/{domain}/timeline", | |
| 368 | + "/api/v1/urls/history?url=", | |
| 369 | + "/api/v1/search?q=", | |
| 370 | + "/api/v1/stats", | |
| 371 | + "/api/v1/trending", | |
| 372 | + "/api/v1/breaking", | |
| 373 | + "/api/v1/pulse", | |
| 374 | + "/api/v1/radar", | |
| 375 | + "/api/v1/explore", | |
| 376 | + "/api/v1/countries", | |
| 377 | + "/api/v1/countries/{code|slug}", | |
| 378 | + "/api/v1/categories/{channel}", | |
| 379 | + "/api/v1/health/connectors", | |
| 380 | + "/api/v1/watchlists", | |
| 381 | + "/api/v1/alerts", | |
| 382 | + "/api/v1/notifications", | |
| 383 | + "/api/v1/bookmarks", | |
| 384 | + "/api/v1/views", | |
| 385 | + "/api/v1/monitors", | |
| 386 | + "/api/v1/feed.rss", | |
| 387 | + "wss://…/api/v1/live", | |
| 388 | + ], | |
| 342 | 389 | })); |
| 343 | 390 | } |
| 344 | 391 | |
| 392 | +function stripEmpty(o: Record<string, unknown>): Record<string, unknown> { | |
| 393 | + return Object.fromEntries(Object.entries(o).filter(([, v]) => v !== undefined && v !== null && v !== "")); | |
| 394 | +} | |
| 395 | + | |
| 345 | 396 | function renderCanonical(c: { mode: string; text?: string; items?: Record<string, unknown>[]; json?: unknown }): string { |
| 346 | 397 | if (c.mode === "text") return c.text ?? ""; |
| 347 | 398 | if (c.mode === "list") return (c.items ?? []).map((i) => [i.title ?? i.url ?? i.key, i.url && i.title ? i.url : null, i.summary ? String(i.summary).slice(0, 300) : null].filter(Boolean).join(" — ")).join("\n"); |
| 348 | 399 | return JSON.stringify(c.json ?? null, null, 2); |
| 349 | 400 | } |
| 401 | + | |
| 402 | +export { textArray }; | |
modified
apps/api/src/server.ts
+9 −2
@@ -9,6 +9,8 @@ import { closeDb } from "@websensor/db"; | ||
| 9 | 9 | import { config } from "./config"; |
| 10 | 10 | import { closeLive, registerLive } from "./live"; |
| 11 | 11 | import { registerRoutes } from "./routes"; |
| 12 | +import { registerAdminRoutes } from "./routes-admin"; | |
| 13 | +import { registerUserRoutes } from "./routes-user"; | |
| 12 | 14 | |
| 13 | 15 | /** |
| 14 | 16 | * WebSensor gateway: public entry point behind ngrok. |
@@ -42,14 +44,17 @@ export async function buildServer() { | ||
| 42 | 44 | } |
| 43 | 45 | }); |
| 44 | 46 | |
| 45 | − await app.register(cors, { origin: true, methods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"], allowedHeaders: ["content-type", "x-websensor-owner", "authorization"], exposedHeaders: ["x-request-id"] }); | |
| 47 | + await app.register(cors, { origin: true, methods: ["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"], allowedHeaders: ["content-type", "x-websensor-owner", "x-websensor-admin", "authorization"], exposedHeaders: ["x-request-id", "x-ratelimit-limit", "x-ratelimit-remaining", "x-ratelimit-reset"] }); | |
| 46 | 48 | await app.register(rateLimit, { max: 600, timeWindow: "1 minute", allowList: (req) => !req.url.startsWith("/api/") }); |
| 47 | 49 | await app.register(websocket, { options: { maxPayload: 64 * 1024 } }); |
| 50 | + // Raw YAML bodies for the admin import endpoint. | |
| 51 | + app.addContentTypeParser(["text/yaml", "application/yaml", "application/x-yaml", "text/plain"], { parseAs: "string" }, (_req, body, done) => done(null, body)); | |
| 48 | 52 | |
| 49 | 53 | app.addHook("onSend", async (req, reply) => { |
| 50 | 54 | reply.header("x-request-id", req.id); |
| 51 | 55 | if (req.url.startsWith("/api/")) { |
| 52 | − reply.header("cache-control", req.method === "GET" ? "public, max-age=5, stale-while-revalidate=30" : "no-store"); | |
| 56 | + const isOwner = Boolean(req.headers["x-websensor-owner"]) || req.url.includes("/admin/"); | |
| 57 | + reply.header("cache-control", req.method === "GET" && !isOwner ? "public, max-age=5, stale-while-revalidate=30" : "no-store"); | |
| 53 | 58 | httpRequests.inc({ route: req.routeOptions?.url ?? "unknown", status: String(reply.statusCode) }); |
| 54 | 59 | } |
| 55 | 60 | }); |
@@ -67,6 +72,8 @@ export async function buildServer() { | ||
| 67 | 72 | }); |
| 68 | 73 | |
| 69 | 74 | await registerRoutes(app); |
| 75 | + await registerUserRoutes(app); | |
| 76 | + await registerAdminRoutes(app); | |
| 70 | 77 | await registerLive(app); |
| 71 | 78 | |
| 72 | 79 | // Frontend proxy — everything that is not /api/* goes to Next.js (loopback) |
added
apps/engine/src/alerts.ts
+132 −0
@@ -0,0 +1,132 @@ | ||
| 1 | +import { createHmac } from "node:crypto"; | |
| 2 | +import { db, sql } from "@websensor/db"; | |
| 3 | +import { postJson } from "@websensor/connectors"; | |
| 4 | +import { log } from "./config"; | |
| 5 | +import { m } from "./metrics"; | |
| 6 | + | |
| 7 | +/** | |
| 8 | + * Server-side alert evaluation (spec §43–44). Rules are matched against every published event; | |
| 9 | + * matches become `notifications` rows (channel web = read by the UI) and, for webhook alerts, an | |
| 10 | + * HTTP POST signed with HMAC-SHA256 (`X-WebSensor-Signature: sha256=<hex>`). Delivery goes through | |
| 11 | + * the same SSRF policy as every other URL the engine touches. | |
| 12 | + */ | |
| 13 | +export interface AlertRule { | |
| 14 | + importance_min?: number; | |
| 15 | + signal_min?: number; | |
| 16 | + event_types?: string[]; | |
| 17 | + groups?: string[]; | |
| 18 | + entities?: string[]; | |
| 19 | + sources?: string[]; | |
| 20 | + keywords?: string[]; | |
| 21 | + categories?: string[]; | |
| 22 | + countries?: string[]; | |
| 23 | + silent_only?: boolean; | |
| 24 | + first_party_only?: boolean; | |
| 25 | + confirmed_only?: boolean; | |
| 26 | +} | |
| 27 | + | |
| 28 | +interface AlertRow { | |
| 29 | + id: string; | |
| 30 | + owner_token: string; | |
| 31 | + name: string; | |
| 32 | + rule: AlertRule; | |
| 33 | + channel: string; | |
| 34 | + channel_config: { url?: string; secret?: string } | null; | |
| 35 | + enabled: boolean; | |
| 36 | +} | |
| 37 | + | |
| 38 | +export interface PublishedEvent { | |
| 39 | + id: string; | |
| 40 | + slug: string; | |
| 41 | + type: string; | |
| 42 | + group?: string; | |
| 43 | + title: string; | |
| 44 | + summary: string; | |
| 45 | + importance: number; | |
| 46 | + signal?: number; | |
| 47 | + confidence: number; | |
| 48 | + silent: boolean; | |
| 49 | + evidence: string; | |
| 50 | + firstParty?: boolean; | |
| 51 | + country?: string | null; | |
| 52 | + source: { id: string; name: string; domain: string; tier?: string }; | |
| 53 | + entities: { id: string; name: string; type: string }[]; | |
| 54 | + categories: string[]; | |
| 55 | + url: string; | |
| 56 | + detectedAt: string; | |
| 57 | +} | |
| 58 | + | |
| 59 | +let cache: { at: number; rows: AlertRow[] } | null = null; | |
| 60 | + | |
| 61 | +async function loadAlerts(): Promise<AlertRow[]> { | |
| 62 | + if (cache && Date.now() - cache.at < 30_000) return cache.rows; | |
| 63 | + const r = await db.execute<Record<string, unknown>>(sql`select id, owner_token, name, rule, channel, channel_config, enabled from alerts where enabled`); | |
| 64 | + cache = { at: Date.now(), rows: r.rows as unknown as AlertRow[] }; | |
| 65 | + return cache.rows; | |
| 66 | +} | |
| 67 | + | |
| 68 | +export function invalidateAlerts(): void { | |
| 69 | + cache = null; | |
| 70 | +} | |
| 71 | + | |
| 72 | +export function matchesRule(rule: AlertRule, e: PublishedEvent): boolean { | |
| 73 | + if (rule.importance_min !== undefined && e.importance < rule.importance_min) return false; | |
| 74 | + if (rule.signal_min !== undefined && (e.signal ?? e.importance) < rule.signal_min) return false; | |
| 75 | + if (rule.silent_only && !e.silent) return false; | |
| 76 | + if (rule.first_party_only && e.firstParty === false) return false; | |
| 77 | + if (rule.confirmed_only && e.evidence !== "CONFIRMED") return false; | |
| 78 | + if (rule.event_types?.length && !rule.event_types.includes(e.type)) return false; | |
| 79 | + if (rule.groups?.length && !(e.group && rule.groups.includes(e.group))) return false; | |
| 80 | + if (rule.categories?.length && !rule.categories.some((c) => e.categories.includes(c))) return false; | |
| 81 | + if (rule.countries?.length && !(e.country && rule.countries.includes(e.country))) return false; | |
| 82 | + if (rule.entities?.length && !e.entities.some((x) => rule.entities!.includes(x.id))) return false; | |
| 83 | + if (rule.sources?.length && !rule.sources.includes(e.source.id)) return false; | |
| 84 | + if (rule.keywords?.length) { | |
| 85 | + const hay = `${e.title} ${e.summary}`.toLowerCase(); | |
| 86 | + if (!rule.keywords.some((k) => hay.includes(k.toLowerCase()))) return false; | |
| 87 | + } | |
| 88 | + // A rule with no positive constraint at all would match everything — require at least one. | |
| 89 | + const constrained = rule.importance_min !== undefined || rule.signal_min !== undefined || rule.silent_only || rule.event_types?.length || rule.groups?.length || rule.categories?.length || rule.countries?.length || rule.entities?.length || rule.sources?.length || rule.keywords?.length; | |
| 90 | + return Boolean(constrained); | |
| 91 | +} | |
| 92 | + | |
| 93 | +export async function evaluateAlerts(e: PublishedEvent): Promise<number> { | |
| 94 | + let alerts: AlertRow[]; | |
| 95 | + try { | |
| 96 | + alerts = await loadAlerts(); | |
| 97 | + } catch (err) { | |
| 98 | + log.warn({ err: (err as Error).message }, "alerts load failed"); | |
| 99 | + return 0; | |
| 100 | + } | |
| 101 | + let fired = 0; | |
| 102 | + for (const a of alerts) { | |
| 103 | + if (!matchesRule(a.rule ?? {}, e)) continue; | |
| 104 | + fired++; | |
| 105 | + try { | |
| 106 | + const ins = await db.execute<{ id: number }>(sql`insert into notifications (alert_id, event_id, channel, status) values (${a.id}, ${e.id}, ${a.channel}, ${a.channel === "web" ? "delivered" : "queued"}) returning id`); | |
| 107 | + await db.execute(sql`update alerts set last_fired_at = now(), fired_count = fired_count + 1 where id = ${a.id}`); | |
| 108 | + m.alertsFired.inc({ channel: a.channel }); | |
| 109 | + if (a.channel === "webhook" && a.channel_config?.url) void deliverWebhook(ins.rows[0]!.id, a, e); | |
| 110 | + } catch (err) { | |
| 111 | + log.warn({ alert: a.id, err: (err as Error).message }, "notification insert failed"); | |
| 112 | + } | |
| 113 | + } | |
| 114 | + return fired; | |
| 115 | +} | |
| 116 | + | |
| 117 | +async function deliverWebhook(notificationId: number, a: AlertRow, e: PublishedEvent): Promise<void> { | |
| 118 | + const url = String(a.channel_config?.url ?? ""); | |
| 119 | + const body = JSON.stringify({ alert: { id: a.id, name: a.name }, event: e, delivered_at: new Date().toISOString() }); | |
| 120 | + const headers: Record<string, string> = { "content-type": "application/json", "user-agent": "WebSensor-Alerts/1.0 (+https://www.websensor.io/api)", "x-websensor-alert": a.id, "x-websensor-event": e.id }; | |
| 121 | + if (a.channel_config?.secret) headers["x-websensor-signature"] = "sha256=" + createHmac("sha256", String(a.channel_config.secret)).update(body).digest("hex"); | |
| 122 | + try { | |
| 123 | + const res = await postJson(url, body, headers); | |
| 124 | + const ok = res.status >= 200 && res.status < 300; | |
| 125 | + await db.execute(sql`update notifications set status = ${ok ? "delivered" : "failed"}, delivered_at = ${ok ? new Date() : null}, error = ${ok ? null : (res.error ?? `HTTP ${res.status}`).slice(0, 300)} where id = ${notificationId}`); | |
| 126 | + m.webhookDeliveries.inc({ ok: String(ok) }); | |
| 127 | + } catch (err) { | |
| 128 | + await db.execute(sql`update notifications set status = 'failed', error = ${String((err as Error).message).slice(0, 300)} where id = ${notificationId}`).catch(() => undefined); | |
| 129 | + m.webhookDeliveries.inc({ ok: "false" }); | |
| 130 | + log.warn({ alert: a.id, err: (err as Error).message }, "webhook delivery failed"); | |
| 131 | + } | |
| 132 | +} | |
modified
apps/engine/src/cli.ts
+18 −1
@@ -52,6 +52,23 @@ async function main(): Promise<void> { | ||
| 52 | 52 | } |
| 53 | 53 | break; |
| 54 | 54 | } |
| 55 | + case "relink-entities": { | |
| 56 | + const { relinkMentionedEntities } = await import("./entities"); | |
| 57 | + const r = await relinkMentionedEntities(Number(args[0] ?? 7)); | |
| 58 | + console.log(JSON.stringify(r)); | |
| 59 | + break; | |
| 60 | + } | |
| 61 | + case "prune-blobs": { | |
| 62 | + const { pruneRawSnapshots } = await import("./retention"); | |
| 63 | + console.log(JSON.stringify(await pruneRawSnapshots({ batch: Number(args[0] ?? 5000) }))); | |
| 64 | + break; | |
| 65 | + } | |
| 66 | + case "refresh-clusters": { | |
| 67 | + const { refreshClusterStates } = await import("./cluster"); | |
| 68 | + await refreshClusterStates(); | |
| 69 | + console.log("cluster states refreshed"); | |
| 70 | + break; | |
| 71 | + } | |
| 55 | 72 | case "llm-test": { |
| 56 | 73 | if (!llmAvailable()) throw new Error("ANTHROPIC_API_KEY not set"); |
| 57 | 74 | const before = "API Pricing\nInput: $10 / million tokens\nOutput: $30 / million tokens\nBatch API: 50% discount"; |
@@ -64,7 +81,7 @@ async function main(): Promise<void> { | ||
| 64 | 81 | break; |
| 65 | 82 | } |
| 66 | 83 | default: |
| 67 | − console.error("usage: cli.ts sync | discover [sourceId…] | probe <domain> | run-once <sensorId> | run-due [n]"); | |
| 84 | + console.error("usage: cli.ts sync | discover [sourceId…] | probe <domain> | run-once <sensorId> | run-due [n] | relink-entities [days] | prune-blobs [batch] | refresh-clusters"); | |
| 68 | 85 | process.exitCode = 1; |
| 69 | 86 | } |
| 70 | 87 | } |
modified
apps/engine/src/cluster.ts
+78 −22
@@ -1,9 +1,13 @@ | ||
| 1 | −import { jaccard, newId, shingles } from "@websensor/core"; | |
| 2 | −import { db, eventClusters, events, gte, sql, textArray } from "@websensor/db"; | |
| 1 | +import { breakingState, jaccard, newId, shingles, slugify, velocityScore } from "@websensor/core"; | |
| 2 | +import { db, eventClusters, events, gte, sql, textArray, type ClusterTimelineStep } from "@websensor/db"; | |
| 3 | 3 | |
| 4 | 4 | /** |
| 5 | 5 | * Novelty + clustering over a rolling in-memory window of recent events (loaded from |
| 6 | 6 | * Postgres at startup). Similarity = Jaccard over word 3-shingles of title+summary. |
| 7 | + * | |
| 8 | + * 2026-09-11: clusters now track propagation (spec §24, §35, §36): first-party vs external | |
| 9 | + * signals, a timeline of every signal, velocity, lead time (first-party detection → first | |
| 10 | + * external report) and a breaking state (spec §34). | |
| 7 | 11 | */ |
| 8 | 12 | interface RecentEvent { |
| 9 | 13 | id: string; |
@@ -15,6 +19,7 @@ interface RecentEvent { | ||
| 15 | 19 | detectedAt: number; |
| 16 | 20 | sh: Set<string>; |
| 17 | 21 | importance: number; |
| 22 | + firstParty: boolean; | |
| 18 | 23 | } |
| 19 | 24 | |
| 20 | 25 | const WINDOW_MS = 72 * 3600e3; |
@@ -24,11 +29,11 @@ let loaded = false; | ||
| 24 | 29 | |
| 25 | 30 | export async function loadRecent(): Promise<void> { |
| 26 | 31 | const since = new Date(Date.now() - WINDOW_MS); |
| 27 | − const rows = await db.execute<{ id: string; cluster_id: string | null; source_id: string; sensor_id: string; event_type: string; title: string; summary: string; detected_at: Date; importance: number; entity_ids: string[] | null }>(sql` | |
| 28 | − select e.id, e.cluster_id, e.source_id, e.sensor_id, e.event_type, e.title, e.summary, e.detected_at, e.importance, | |
| 32 | + const rows = await db.execute<{ id: string; cluster_id: string | null; source_id: string; sensor_id: string; event_type: string; title: string; summary: string; detected_at: Date; importance: number; first_party: boolean | null; entity_ids: string[] | null }>(sql` | |
| 33 | + select e.id, e.cluster_id, e.source_id, e.sensor_id, e.event_type, e.title, e.summary, e.detected_at, e.importance, e.first_party, | |
| 29 | 34 | (select array_agg(entity_id) from event_entities ee where ee.event_id = e.id) as entity_ids |
| 30 | − from events e where e.detected_at >= ${since} order by e.detected_at desc limit 3000`); | |
| 31 | − recent = rows.rows.map((r) => ({ id: r.id, clusterId: r.cluster_id, sourceId: r.source_id, sensorId: r.sensor_id, eventType: r.event_type, entityIds: r.entity_ids ?? [], detectedAt: new Date(r.detected_at).getTime(), sh: shingles(`${r.title}\n${r.summary}`), importance: r.importance })); | |
| 35 | + from events e where e.detected_at >= ${since} order by e.detected_at desc limit 4000`); | |
| 36 | + recent = rows.rows.map((r) => ({ id: r.id, clusterId: r.cluster_id, sourceId: r.source_id, sensorId: r.sensor_id, eventType: r.event_type, entityIds: r.entity_ids ?? [], detectedAt: new Date(r.detected_at).getTime(), sh: shingles(`${r.title}\n${r.summary}`), importance: r.importance, firstParty: r.first_party ?? true })); | |
| 32 | 37 | loaded = true; |
| 33 | 38 | } |
| 34 | 39 | |
@@ -40,8 +45,10 @@ function prune(): void { | ||
| 40 | 45 | export interface NoveltyResult { |
| 41 | 46 | novelty: number; |
| 42 | 47 | nearest: { id: string; similarity: number } | null; |
| 43 | − /** number of distinct sources reporting near-identical content */ | |
| 48 | + /** number of distinct OTHER sources reporting near-identical content */ | |
| 44 | 49 | confirmations: number; |
| 50 | + /** among those, how many are first-party channels */ | |
| 51 | + firstPartyConfirmations: number; | |
| 45 | 52 | } |
| 46 | 53 | |
| 47 | 54 | export async function assessNovelty(text: string, sourceId: string): Promise<NoveltyResult> { |
@@ -50,21 +57,30 @@ export async function assessNovelty(text: string, sourceId: string): Promise<Nov | ||
| 50 | 57 | const sh = shingles(text); |
| 51 | 58 | let best = 0; |
| 52 | 59 | let nearest: RecentEvent | null = null; |
| 53 | − const confirmingSources = new Set<string>(); | |
| 60 | + const confirming = new Map<string, boolean>(); | |
| 54 | 61 | for (const r of recent) { |
| 55 | 62 | const s = jaccard(sh, r.sh); |
| 56 | 63 | if (s > best) { |
| 57 | 64 | best = s; |
| 58 | 65 | nearest = r; |
| 59 | 66 | } |
| 60 | − if (s >= 0.45 && r.sourceId !== sourceId) confirmingSources.add(r.sourceId); | |
| 67 | + if (s >= 0.45 && r.sourceId !== sourceId) confirming.set(r.sourceId, r.firstParty); | |
| 61 | 68 | } |
| 62 | − return { novelty: Math.round((1 - best) * 100), nearest: nearest ? { id: nearest.id, similarity: Math.round(best * 100) / 100 } : null, confirmations: confirmingSources.size }; | |
| 69 | + return { novelty: Math.round((1 - best) * 100), nearest: nearest ? { id: nearest.id, similarity: Math.round(best * 100) / 100 } : null, confirmations: confirming.size, firstPartyConfirmations: [...confirming.values()].filter(Boolean).length }; | |
| 63 | 70 | } |
| 64 | 71 | |
| 65 | 72 | export interface ClusterDecision { |
| 66 | 73 | clusterId: string; |
| 74 | + slug: string; | |
| 67 | 75 | created: boolean; |
| 76 | + /** stats after this event was attached */ | |
| 77 | + eventCount: number; | |
| 78 | + sourceCount: number; | |
| 79 | + firstPartyCount: number; | |
| 80 | + externalCount: number; | |
| 81 | + velocity: number; | |
| 82 | + state: string; | |
| 83 | + leadTimeMs: number | null; | |
| 68 | 84 | } |
| 69 | 85 | |
| 70 | 86 | /** |
@@ -72,7 +88,7 @@ export interface ClusterDecision { | ||
| 72 | 88 | * with a recent event and is textually related, or when it is the same event type on the |
| 73 | 89 | * same source within 30 minutes (e.g. one launch touching six pages). Otherwise open one. |
| 74 | 90 | */ |
| 75 | −export async function clusterEvent(ev: { id: string; sourceId: string; sensorId: string; eventType: string; entityIds: string[]; detectedAt: Date; title: string; summary: string; importance: number; categories: string[] }): Promise<ClusterDecision> { | |
| 91 | +export async function clusterEvent(ev: { id: string; sourceId: string; sourceName: string; sensorId: string; sensorType: string; eventType: string; entityIds: string[]; detectedAt: Date; title: string; summary: string; importance: number; signal: number; categories: string[]; firstParty: boolean; sourceTier?: string }): Promise<ClusterDecision> { | |
| 76 | 92 | if (!loaded) await loadRecent(); |
| 77 | 93 | const sh = shingles(`${ev.title}\n${ev.summary}`); |
| 78 | 94 | const now = ev.detectedAt.getTime(); |
@@ -89,29 +105,57 @@ export async function clusterEvent(ev: { id: string; sourceId: string; sensorId: | ||
| 89 | 105 | if (sim >= 0.22) score = sim + (sharedEntity ? 0.2 : 0); |
| 90 | 106 | else if (sameSource && r.eventType === ev.eventType && closeInTime && r.sensorId !== ev.sensorId) score = 0.3; |
| 91 | 107 | else if (sameSource && closeInTime && sim >= 0.12) score = 0.25; |
| 108 | + // cross-source, shared entity, moderately similar and same broad type → the same story reported elsewhere | |
| 109 | + else if (!sameSource && sharedEntity && sim >= 0.15 && now - r.detectedAt < 2 * 3600e3) score = 0.2 + sim; | |
| 92 | 110 | if (score > bestScore) { |
| 93 | 111 | bestScore = score; |
| 94 | 112 | bestCluster = r.clusterId; |
| 95 | 113 | } |
| 96 | 114 | } |
| 115 | + const step: ClusterTimelineStep = { at: ev.detectedAt.toISOString(), eventId: ev.id, sourceId: ev.sourceId, sourceName: ev.sourceName, sensorType: ev.sensorType, firstParty: ev.firstParty, eventType: ev.eventType, importance: ev.importance }; | |
| 97 | 116 | let clusterId: string; |
| 117 | + let slug: string; | |
| 98 | 118 | let created = false; |
| 119 | + let row: { event_count: number; source_count: number; first_party_count: number; external_count: number; first_at: Date; last_at: Date; first_party_at: Date | null; first_external_at: Date | null; lead_time_ms: number | null; slug: string | null; max_importance: number }; | |
| 99 | 120 | if (bestCluster) { |
| 100 | 121 | clusterId = bestCluster; |
| 101 | − await db.execute(sql`update event_clusters set event_count = event_count + 1, last_at = greatest(last_at, ${ev.detectedAt}), max_importance = greatest(max_importance, ${ev.importance}), | |
| 102 | − entity_ids = (select array(select distinct unnest(entity_ids || ${textArray(ev.entityIds)}))), | |
| 103 | − categories = (select array(select distinct unnest(categories || ${textArray(ev.categories)}))), | |
| 104 | − title = case when ${ev.importance} > max_importance then ${ev.title} else title end, | |
| 105 | − primary_event_id = case when ${ev.importance} > max_importance then ${ev.id} else primary_event_id end | |
| 106 | − where id = ${clusterId}`); | |
| 122 | + const r = await db.execute<typeof row>(sql` | |
| 123 | + update event_clusters set | |
| 124 | + event_count = event_count + 1, | |
| 125 | + last_at = greatest(last_at, ${ev.detectedAt}), | |
| 126 | + max_importance = greatest(max_importance, ${ev.importance}), | |
| 127 | + entity_ids = (select array(select distinct unnest(entity_ids || ${textArray(ev.entityIds)}))), | |
| 128 | + categories = (select array(select distinct unnest(categories || ${textArray(ev.categories)}))), | |
| 129 | + title = case when ${ev.importance} > max_importance then ${ev.title} else title end, | |
| 130 | + summary = case when ${ev.importance} > max_importance then ${ev.summary} else summary end, | |
| 131 | + primary_event_id = case when ${ev.importance} > max_importance then ${ev.id} else primary_event_id end, | |
| 132 | + source_count = (select count(distinct source_id) from events where cluster_id = ${clusterId}) + (case when exists (select 1 from events where cluster_id = ${clusterId} and source_id = ${ev.sourceId}) then 0 else 1 end), | |
| 133 | + first_party_count = first_party_count + ${ev.firstParty ? 1 : 0}, | |
| 134 | + external_count = external_count + ${ev.firstParty ? 0 : 1}, | |
| 135 | + first_party_at = case when ${ev.firstParty} then least(coalesce(first_party_at, ${ev.detectedAt}), ${ev.detectedAt}) else first_party_at end, | |
| 136 | + first_external_at = case when ${!ev.firstParty} then least(coalesce(first_external_at, ${ev.detectedAt}), ${ev.detectedAt}) else first_external_at end, | |
| 137 | + timeline = (case when jsonb_array_length(timeline) < 200 then timeline || ${JSON.stringify([step])}::jsonb else timeline end) | |
| 138 | + where id = ${clusterId} | |
| 139 | + returning event_count, source_count, first_party_count, external_count, first_at, last_at, first_party_at, first_external_at, lead_time_ms, slug, max_importance`); | |
| 140 | + row = r.rows[0]!; | |
| 141 | + slug = row.slug ?? clusterId; | |
| 107 | 142 | } else { |
| 108 | 143 | clusterId = newId("clu"); |
| 144 | + slug = `${slugify(ev.title).slice(0, 60)}-${clusterId.slice(-6)}`; | |
| 109 | 145 | created = true; |
| 110 | − await db.insert(eventClusters).values({ id: clusterId, title: ev.title, summary: ev.summary, primaryEventId: ev.id, entityIds: ev.entityIds, categories: ev.categories, eventCount: 1, maxImportance: ev.importance, firstAt: ev.detectedAt, lastAt: ev.detectedAt }); | |
| 146 | + await db.insert(eventClusters).values({ id: clusterId, slug, title: ev.title, summary: ev.summary, primaryEventId: ev.id, entityIds: ev.entityIds, categories: ev.categories, eventCount: 1, maxImportance: ev.importance, firstAt: ev.detectedAt, lastAt: ev.detectedAt, sourceCount: 1, firstPartyCount: ev.firstParty ? 1 : 0, externalCount: ev.firstParty ? 0 : 1, firstPartyAt: ev.firstParty ? ev.detectedAt : null, firstExternalAt: ev.firstParty ? null : ev.detectedAt, timeline: [step], state: "watching" }); | |
| 147 | + row = { event_count: 1, source_count: 1, first_party_count: ev.firstParty ? 1 : 0, external_count: ev.firstParty ? 0 : 1, first_at: ev.detectedAt, last_at: ev.detectedAt, first_party_at: ev.firstParty ? ev.detectedAt : null, first_external_at: ev.firstParty ? null : ev.detectedAt, lead_time_ms: null, slug, max_importance: ev.importance }; | |
| 111 | 148 | } |
| 112 | − recent.unshift({ id: ev.id, clusterId, sourceId: ev.sourceId, sensorId: ev.sensorId, eventType: ev.eventType, entityIds: ev.entityIds, detectedAt: now, sh, importance: ev.importance }); | |
| 113 | − if (recent.length > 5000) recent.length = 5000; | |
| 114 | − return { clusterId, created }; | |
| 149 | + // Derived: velocity, lead time, state. | |
| 150 | + const windowHours = Math.max(0.25, (new Date(row.last_at).getTime() - new Date(row.first_at).getTime()) / 3600e3); | |
| 151 | + const velocity = velocityScore({ signals: row.event_count, windowHours: Math.min(windowHours, 6), uniqueSources: row.source_count, firstPartySignals: row.first_party_count }); | |
| 152 | + const leadTimeMs = row.first_party_at && row.first_external_at ? new Date(row.first_external_at).getTime() - new Date(row.first_party_at).getTime() : null; | |
| 153 | + const confirmations = Math.max(0, row.source_count - 1); | |
| 154 | + const state = breakingState({ signal: ev.signal, importance: Math.max(ev.importance, row.max_importance), velocity, confirmations, firstPartyCount: row.first_party_count, ageMinutes: (Date.now() - new Date(row.first_at).getTime()) / 60e3, sourceTier: ev.sourceTier }); | |
| 155 | + await db.execute(sql`update event_clusters set velocity = ${velocity}, state = ${state}, lead_time_ms = ${leadTimeMs} where id = ${clusterId}`); | |
| 156 | + recent.unshift({ id: ev.id, clusterId, sourceId: ev.sourceId, sensorId: ev.sensorId, eventType: ev.eventType, entityIds: ev.entityIds, detectedAt: now, sh, importance: ev.importance, firstParty: ev.firstParty }); | |
| 157 | + if (recent.length > 6000) recent.length = 6000; | |
| 158 | + return { clusterId, slug, created, eventCount: row.event_count, sourceCount: row.source_count, firstPartyCount: row.first_party_count, externalCount: row.external_count, velocity, state, leadTimeMs }; | |
| 115 | 159 | } |
| 116 | 160 | |
| 117 | 161 | /** Recent events of the same source published through announcement-type sensors (for silent-change detection). */ |
@@ -121,7 +165,7 @@ export function recentAnnouncementSimilarity(sourceId: string, text: string, sin | ||
| 121 | 165 | const cutoff = Date.now() - sinceMs; |
| 122 | 166 | for (const r of recent) { |
| 123 | 167 | if (r.sourceId !== sourceId || r.detectedAt < cutoff) continue; |
| 124 | − if (!/announcement|product_launch|model_release|software_release|repository_release|incident|outage|maintenance|security_advisory/.test(r.eventType)) continue; | |
| 168 | + if (!/announcement|product_launch|model_release|software_release|repository_release|incident|outage|maintenance|security_advisory|service_launch|patch_release/.test(r.eventType)) continue; | |
| 125 | 169 | best = Math.max(best, jaccard(sh, r.sh)); |
| 126 | 170 | } |
| 127 | 171 | return best; |
@@ -132,4 +176,16 @@ export async function recentClusterCount(sinceMs: number): Promise<number> { | ||
| 132 | 176 | return Number(rows[0]?.n ?? 0); |
| 133 | 177 | } |
| 134 | 178 | |
| 179 | +/** Periodic: age out states (breaking → developing/confirmed → closed) for clusters that stopped receiving signals. */ | |
| 180 | +export async function refreshClusterStates(): Promise<void> { | |
| 181 | + await db.execute(sql` | |
| 182 | + update event_clusters set state = case | |
| 183 | + when last_at < now() - interval '72 hours' then 'closed' | |
| 184 | + when state = 'breaking' and first_at < now() - interval '6 hours' then (case when source_count >= 3 then 'confirmed' else 'developing' end) | |
| 185 | + when state = 'developing' and first_at < now() - interval '12 hours' then (case when source_count >= 3 then 'confirmed' else 'watching' end) | |
| 186 | + when state = 'confirmed' and last_at < now() - interval '24 hours' then 'watching' | |
| 187 | + else state end | |
| 188 | + where state <> 'closed' and last_at >= now() - interval '10 days'`); | |
| 189 | +} | |
| 190 | + | |
| 135 | 191 | export { events }; |
modified
apps/engine/src/config.ts
+6 −2
@@ -15,7 +15,7 @@ export const config = { | ||
| 15 | 15 | perHostConcurrency: Number(env.WS_PER_HOST_CONCURRENCY ?? 2), |
| 16 | 16 | metricsPort: Number(env.ENGINE_METRICS_PORT ?? 8262), |
| 17 | 17 | metricsHost: env.ENGINE_METRICS_HOST ?? "127.0.0.1", |
| 18 | − processingVersion: "event-pipeline-v1", | |
| 18 | + processingVersion: "event-pipeline-v2", | |
| 19 | 19 | /** heuristic signal threshold to promote a raw change to an event */ |
| 20 | 20 | meaningfulSignal: Number(env.WS_MEANINGFUL_SIGNAL ?? 0.32), |
| 21 | 21 | llm: { |
@@ -32,7 +32,11 @@ export const config = { | ||
| 32 | 32 | intervalDays: Number(env.WS_DISCOVERY_INTERVAL_DAYS ?? 7), |
| 33 | 33 | }, |
| 34 | 34 | deletion: { confirmations: Number(env.WS_DELETE_CONFIRMATIONS ?? 3), minSeparationMin: Number(env.WS_DELETE_SEPARATION_MIN ?? 60) }, |
| 35 | − version: "0.1.0", | |
| 35 | + /** storage lifecycle (spec §58): raw bodies of unchanged snapshots are dropped after N days; canonical + hashes kept */ | |
| 36 | + retention: { rawDays: Number(env.WS_RETENTION_RAW_DAYS ?? 21), changedRawDays: Number(env.WS_RETENTION_CHANGED_RAW_DAYS ?? 60), enabled: (env.WS_RETENTION ?? "1") !== "0" }, | |
| 37 | + /** silent-change bar: only eligible types with importance ≥ this become "silent" */ | |
| 38 | + silentMinImportance: Number(env.WS_SILENT_MIN_IMPORTANCE ?? 45), | |
| 39 | + version: "0.2.0", | |
| 36 | 40 | }; |
| 37 | 41 | |
| 38 | 42 | export const log = pino({ |
added
apps/engine/src/engine.test.ts
+73 −0
@@ -0,0 +1,73 @@ | ||
| 1 | +import { describe, expect, it } from "vitest"; | |
| 2 | +import { matchesRule, type PublishedEvent } from "./alerts"; | |
| 3 | +import { inferCountry, inferLanguage, priorityFor } from "./registry"; | |
| 4 | + | |
| 5 | +const ev: PublishedEvent = { | |
| 6 | + id: "evt_1", | |
| 7 | + slug: "openai-pricing", | |
| 8 | + type: "pricing_change", | |
| 9 | + group: "commercial", | |
| 10 | + title: "OpenAI: price changed $10 → $8", | |
| 11 | + summary: "Input tokens now $8 per million.", | |
| 12 | + importance: 84, | |
| 13 | + signal: 88, | |
| 14 | + confidence: 90, | |
| 15 | + silent: true, | |
| 16 | + evidence: "OBSERVED", | |
| 17 | + firstParty: true, | |
| 18 | + country: "US", | |
| 19 | + source: { id: "openai", name: "OpenAI", domain: "openai.com", tier: "S" }, | |
| 20 | + entities: [{ id: "org_openai", name: "OpenAI", type: "organization" }], | |
| 21 | + categories: ["ai", "technology"], | |
| 22 | + url: "https://openai.com/api/pricing/", | |
| 23 | + detectedAt: new Date().toISOString(), | |
| 24 | +}; | |
| 25 | + | |
| 26 | +describe("alert rules (spec §43)", () => { | |
| 27 | + it("matches 'OpenAI changes pricing'", () => { | |
| 28 | + expect(matchesRule({ entities: ["org_openai"], event_types: ["pricing_change"] }, ev)).toBe(true); | |
| 29 | + }); | |
| 30 | + it("matches importance thresholds and silent-only", () => { | |
| 31 | + expect(matchesRule({ importance_min: 80 }, ev)).toBe(true); | |
| 32 | + expect(matchesRule({ importance_min: 90 }, ev)).toBe(false); | |
| 33 | + expect(matchesRule({ silent_only: true, categories: ["ai"] }, ev)).toBe(true); | |
| 34 | + }); | |
| 35 | + it("matches groups, countries, keywords and first-party", () => { | |
| 36 | + expect(matchesRule({ groups: ["commercial"] }, ev)).toBe(true); | |
| 37 | + expect(matchesRule({ groups: ["security"] }, ev)).toBe(false); | |
| 38 | + expect(matchesRule({ countries: ["US"] }, ev)).toBe(true); | |
| 39 | + expect(matchesRule({ keywords: ["million"] }, ev)).toBe(true); | |
| 40 | + expect(matchesRule({ first_party_only: true, sources: ["openai"] }, ev)).toBe(true); | |
| 41 | + expect(matchesRule({ confirmed_only: true, sources: ["openai"] }, ev)).toBe(false); | |
| 42 | + }); | |
| 43 | + it("rejects a rule without any positive constraint", () => { | |
| 44 | + expect(matchesRule({}, ev)).toBe(false); | |
| 45 | + expect(matchesRule({ first_party_only: true }, ev)).toBe(false); | |
| 46 | + }); | |
| 47 | +}); | |
| 48 | + | |
| 49 | +describe("registry inference", () => { | |
| 50 | + it("infers country from unambiguous TLDs only", () => { | |
| 51 | + expect(inferCountry("canada.ca", [])).toBe("CA"); | |
| 52 | + expect(inferCountry("quebec.ca", [])).toBe("CA"); | |
| 53 | + expect(inferCountry("gov.uk", [])).toBe("GB"); | |
| 54 | + expect(inferCountry("legifrance.gouv.fr", [])).toBe("FR"); | |
| 55 | + expect(inferCountry("cisa.gov", [])).toBe("US"); | |
| 56 | + expect(inferCountry("europa.eu", [])).toBe("EU"); | |
| 57 | + expect(inferCountry("who.int", [])).toBe("INT"); | |
| 58 | + expect(inferCountry("openai.com", [])).toBeNull(); | |
| 59 | + expect(inferCountry("example.io", [])).toBeNull(); | |
| 60 | + }); | |
| 61 | + it("infers language for francophone and other national domains", () => { | |
| 62 | + expect(inferLanguage("quebec.ca")).toBe("fr"); | |
| 63 | + expect(inferLanguage("spiegel.de")).toBe("de"); | |
| 64 | + expect(inferLanguage("openai.com")).toBeNull(); | |
| 65 | + }); | |
| 66 | + it("assigns priority tiers", () => { | |
| 67 | + expect(priorityFor("S", ["cloud"])).toBe(0); | |
| 68 | + expect(priorityFor("A", ["cyber"])).toBe(0); | |
| 69 | + expect(priorityFor("A", ["retail"])).toBe(1); | |
| 70 | + expect(priorityFor("B", ["retail"])).toBe(2); | |
| 71 | + expect(priorityFor("D", ["web-policy"])).toBe(3); | |
| 72 | + }); | |
| 73 | +}); | |
modified
apps/engine/src/entities.ts
+79 −3
@@ -8,19 +8,34 @@ import { log } from "./config"; | ||
| 8 | 8 | */ |
| 9 | 9 | interface AliasIndex { |
| 10 | 10 | loadedAt: number; |
| 11 | − byLen: { alias: string; re: RegExp; entityId: string }[]; | |
| 11 | + byLen: { alias: string; re: RegExp; entityId: string; ambiguous: boolean }[]; | |
| 12 | 12 | importance: Map<string, number>; |
| 13 | 13 | } |
| 14 | 14 | |
| 15 | 15 | let index: AliasIndex | null = null; |
| 16 | 16 | |
| 17 | +/** | |
| 18 | + * Aliases that are ordinary words. Matching them in free text would attach the entity to almost every | |
| 19 | + * event ("FIRST" → Forum of Incident Response, "HAS" → Haute Autorité de santé, "make", "who", "cell"…). | |
| 20 | + * They only count when written exactly as an upper-case acronym in the original text (WHO, NASA, FIRST), | |
| 21 | + * or when the entity already belongs to the source (subject). | |
| 22 | + */ | |
| 23 | +export const AMBIGUOUS_ALIASES = new Set( | |
| 24 | + "first has who make cell nature science shell orange apple bell block square meta oracle mint target gap next box slack zoom stripe uber lyft ring nest arm hp ge box tesla sky sun star time life people vice wired verge edge chrome safari brave signal telegram discord slack notion linear figma canva medium substack ghost dash zapier make render fly neon turso tigris planet scale cockroach confluent elastic redis mongo mongodb vercel netlify heroku render railway supabase firebase play store market cloud one plus max pro air mini studio watch music tv news post times globe mail star sun herald standard journal press daily weekly review register wire record hill point line frontier alliance alpha beta gamma delta omega origin echo nova atlas titan vector prism pulse radar forge anchor arc arrow beam bolt bond bridge canvas circle core crest crown drift ember flare flow fuse glow grid halo haven horizon iris jet key lift loop lumen mesh nexus node oasis orbit peak pillar pivot quest realm relay ridge rise river rock root sage scope shift spark sphere spire stone stream summit surge swift tide torch trace trail unity vault venture verge vista wave zenith bank trust fund capital global national international federal united american canadian european royal general standard central pacific atlantic western eastern northern southern".split(/\s+/), | |
| 25 | +); | |
| 26 | + | |
| 17 | 27 | async function loadIndex(): Promise<AliasIndex> { |
| 18 | 28 | if (index && Date.now() - index.loadedAt < 5 * 60_000) return index; |
| 19 | 29 | const rows = await db.select({ alias: entityAliases.alias, entityId: entityAliases.entityId }).from(entityAliases); |
| 20 | 30 | const ents = await db.select({ id: entities.id, importance: entities.importance }).from(entities); |
| 21 | 31 | const byLen = rows |
| 22 | 32 | .filter((r) => r.alias.length >= 3 && !/^\d+$/.test(r.alias)) |
| 23 | − .map((r) => ({ alias: r.alias, entityId: r.entityId, re: new RegExp(`(^|[^\\p{L}\\p{N}])${escapeRe(r.alias)}(?=$|[^\\p{L}\\p{N}])`, "iu") })) | |
| 33 | + .map((r) => { | |
| 34 | + const ambiguous = AMBIGUOUS_ALIASES.has(r.alias) || (r.alias.length <= 3 && !/[.-]/.test(r.alias)); | |
| 35 | + // Ambiguous / very short aliases: case-sensitive upper-case acronym match only (WHO, FIRST, HAS as acronym). | |
| 36 | + const re = ambiguous ? new RegExp(`(^|[^\\p{L}\\p{N}])${escapeRe(r.alias.toUpperCase())}(?=$|[^\\p{L}\\p{N}])`, "u") : new RegExp(`(^|[^\\p{L}\\p{N}])${escapeRe(r.alias)}(?=$|[^\\p{L}\\p{N}])`, "iu"); | |
| 37 | + return { alias: r.alias, entityId: r.entityId, re, ambiguous }; | |
| 38 | + }) | |
| 24 | 39 | .sort((a, b) => b.alias.length - a.alias.length); |
| 25 | 40 | index = { loadedAt: Date.now(), byLen, importance: new Map(ents.map((e) => [e.id, e.importance])) }; |
| 26 | 41 | log.debug({ aliases: byLen.length }, "alias index loaded"); |
@@ -47,6 +62,7 @@ export async function resolveEntities(input: { sourceId: string; text: string; h | ||
| 47 | 62 | if (mentioned.size >= 12) break; |
| 48 | 63 | if (orgSubjects.includes(a.entityId)) continue; |
| 49 | 64 | if (!hayLower.includes(a.alias)) continue; |
| 65 | + if (a.ambiguous && !subject.includes(a.entityId) && !a.re.test(hay)) continue; // needs the exact acronym form | |
| 50 | 66 | if (a.re.test(hay)) mentioned.add(a.entityId); |
| 51 | 67 | } |
| 52 | 68 | // Products of the source named in the text are subjects too. |
@@ -56,7 +72,67 @@ export async function resolveEntities(input: { sourceId: string; text: string; h | ||
| 56 | 72 | return { subject: orgSubjects, mentioned: [...mentioned], importance }; |
| 57 | 73 | } |
| 58 | 74 | |
| 59 | −export async function bumpEntityCounters(entityIds: string[], at: Date): Promise<void> { | |
| 75 | +/** | |
| 76 | + * Maintenance: re-resolve "mentioned" entity links for recent events with the current alias rules | |
| 77 | + * (drops links created by ambiguous aliases). Subjects (source entities) are never touched. | |
| 78 | + */ | |
| 79 | +export async function relinkMentionedEntities(days = 7): Promise<{ events: number; removed: number; added: number }> { | |
| 80 | + invalidateEntityIndex(); | |
| 81 | + const idx = await loadIndex(); | |
| 82 | + const rows = await db.execute<{ id: string; source_id: string; title: string; summary: string; keywords: string[] | null }>(sql`select id, source_id, title, summary, keywords from events where detected_at >= now() - make_interval(days => ${days}) order by detected_at desc`); | |
| 83 | + let removed = 0; | |
| 84 | + let added = 0; | |
| 85 | + const subjectsBySource = new Map<string, string[]>(); | |
| 86 | + for (const ev of rows.rows) { | |
| 87 | + let subs = subjectsBySource.get(ev.source_id); | |
| 88 | + if (!subs) { | |
| 89 | + subs = (await db.select({ entityId: sourceEntities.entityId }).from(sourceEntities).where(eq(sourceEntities.sourceId, ev.source_id))).map((r) => r.entityId); | |
| 90 | + subjectsBySource.set(ev.source_id, subs); | |
| 91 | + } | |
| 92 | + const hay = `${ev.title}\n${ev.summary}\n${(ev.keywords ?? []).join("\n")}`; | |
| 93 | + const hayLower = hay.toLowerCase(); | |
| 94 | + const want = new Set<string>(); | |
| 95 | + for (const a of idx.byLen) { | |
| 96 | + if (want.size >= 12) break; | |
| 97 | + if (subs.some((s) => s.startsWith("org_") && s === a.entityId)) continue; | |
| 98 | + if (!hayLower.includes(a.alias)) continue; | |
| 99 | + if (a.ambiguous && !subs.includes(a.entityId) && !a.re.test(hay)) continue; | |
| 100 | + if (a.re.test(hay)) want.add(a.entityId); | |
| 101 | + } | |
| 102 | + const current = (await db.execute<{ entity_id: string; role: string }>(sql`select entity_id, role from event_entities where event_id = ${ev.id}`)).rows; | |
| 103 | + for (const c of current) { | |
| 104 | + if (c.role === "mentioned" && !want.has(c.entity_id) && !subs.includes(c.entity_id)) { | |
| 105 | + await db.execute(sql`delete from event_entities where event_id = ${ev.id} and entity_id = ${c.entity_id}`); | |
| 106 | + removed++; | |
| 107 | + } | |
| 108 | + } | |
| 109 | + for (const w of want) { | |
| 110 | + if (!current.some((c) => c.entity_id === w)) { | |
| 111 | + await db.execute(sql`insert into event_entities (event_id, entity_id, role) values (${ev.id}, ${w}, 'mentioned') on conflict do nothing`); | |
| 112 | + added++; | |
| 113 | + } | |
| 114 | + } | |
| 115 | + } | |
| 116 | + // Recount entity totals + daily table from the corrected links. | |
| 117 | + await db.execute(sql`update entities en set event_count = (select count(*) from event_entities ee where ee.entity_id = en.id), last_event_at = (select max(e.detected_at) from events e join event_entities ee on ee.event_id = e.id where ee.entity_id = en.id)`); | |
| 118 | + await db.execute(sql`delete from entity_daily where day >= (now() at time zone 'UTC')::date - ${sql.raw(String(Math.max(1, Math.floor(days))))}`); | |
| 119 | + await db.execute(sql`insert into entity_daily (entity_id, day, events, silent, breaking, max_importance) | |
| 120 | + select ee.entity_id, (e.detected_at at time zone 'UTC')::date, count(*), sum(case when e.silent_change then 1 else 0 end), sum(case when e.importance >= 80 then 1 else 0 end), max(e.importance) | |
| 121 | + from events e join event_entities ee on ee.event_id = e.id where e.detected_at >= now() - make_interval(days => ${days}) group by 1, 2 | |
| 122 | + on conflict (entity_id, day) do update set events = excluded.events, silent = excluded.silent, breaking = excluded.breaking, max_importance = excluded.max_importance`); | |
| 123 | + log.info({ events: rows.rows.length, removed, added }, "mentioned entity links re-resolved"); | |
| 124 | + return { events: rows.rows.length, removed, added }; | |
| 125 | +} | |
| 126 | + | |
| 127 | +export async function bumpEntityCounters(entityIds: string[], at: Date, opts: { silent?: boolean; importance?: number } = {}): Promise<void> { | |
| 60 | 128 | if (!entityIds.length) return; |
| 61 | 129 | await db.execute(sql`update entities set event_count = event_count + 1, last_event_at = ${at} where id = any(${textArray(entityIds)})`); |
| 130 | + const imp = Number(opts.importance ?? 0); | |
| 131 | + await db | |
| 132 | + .execute( | |
| 133 | + sql`insert into entity_daily (entity_id, day, events, silent, breaking, max_importance) | |
| 134 | + select unnest(${textArray(entityIds)}), (${at} at time zone 'UTC')::date, 1, ${opts.silent ? 1 : 0}, ${imp >= 80 ? 1 : 0}, ${imp} | |
| 135 | + on conflict (entity_id, day) do update set events = entity_daily.events + 1, silent = entity_daily.silent + ${opts.silent ? 1 : 0}, breaking = entity_daily.breaking + ${imp >= 80 ? 1 : 0}, max_importance = greatest(entity_daily.max_importance, ${imp})`, | |
| 136 | + ) | |
| 137 | + .catch((e) => log.warn({ err: (e as Error).message }, "entity_daily update failed")); | |
| 62 | 138 | } |
modified
apps/engine/src/index.ts
+8 −1
@@ -1,10 +1,11 @@ | ||
| 1 | 1 | import { closeDb, migrate } from "@websensor/db"; |
| 2 | 2 | import { closeDispatcher } from "@websensor/connectors"; |
| 3 | −import { loadRecent } from "./cluster"; | |
| 3 | +import { loadRecent, refreshClusterStates } from "./cluster"; | |
| 4 | 4 | import { config, log } from "./config"; |
| 5 | 5 | import { startMetricsServer } from "./metrics"; |
| 6 | 6 | import { closeRedis } from "./redis"; |
| 7 | 7 | import { runDiscovery, syncRegistry } from "./registry"; |
| 8 | +import { pruneNotifications, pruneRawSnapshots } from "./retention"; | |
| 8 | 9 | import { pruneOldRuns, rollupConnectorHealth, Scheduler } from "./scheduler"; |
| 9 | 10 | |
| 10 | 11 | async function main(): Promise<void> { |
@@ -21,6 +22,12 @@ async function main(): Promise<void> { | ||
| 21 | 22 | const timers: NodeJS.Timeout[] = []; |
| 22 | 23 | timers.push(setInterval(() => rollupConnectorHealth().catch((e) => log.warn({ err: (e as Error).message }, "health rollup failed")), 60_000)); |
| 23 | 24 | timers.push(setInterval(() => pruneOldRuns().catch(() => undefined), 6 * 3600e3)); |
| 25 | + timers.push(setInterval(() => refreshClusterStates().catch((e) => log.warn({ err: (e as Error).message }, "cluster state refresh failed")), 5 * 60_000)); | |
| 26 | + if (config.retention.enabled) { | |
| 27 | + timers.push(setInterval(() => pruneRawSnapshots().catch((e) => log.warn({ err: (e as Error).message }, "retention failed")), 30 * 60_000)); | |
| 28 | + timers.push(setInterval(() => pruneNotifications().catch(() => undefined), 24 * 3600e3)); | |
| 29 | + setTimeout(() => pruneRawSnapshots().catch(() => undefined), 60_000); | |
| 30 | + } | |
| 24 | 31 | await rollupConnectorHealth().catch(() => undefined); |
| 25 | 32 | |
| 26 | 33 | if (config.discovery.enabled) { |
modified
apps/engine/src/metrics.ts
+20 −0
@@ -18,8 +18,28 @@ export const m = { | ||
| 18 | 18 | queueDue: new client.Gauge({ name: "websensor_sensors_due", help: "Sensors due for a check", registers: [registry] }), |
| 19 | 19 | inflight: new client.Gauge({ name: "websensor_inflight_checks", help: "Checks in flight", registers: [registry] }), |
| 20 | 20 | httpStatus: new client.Counter({ name: "websensor_http_status_total", help: "HTTP status codes", labelNames: ["status"], registers: [registry] }), |
| 21 | + alertsFired: new client.Counter({ name: "websensor_alerts_fired_total", help: "Alert rules matched", labelNames: ["channel"], registers: [registry] }), | |
| 22 | + webhookDeliveries: new client.Counter({ name: "websensor_webhook_deliveries_total", help: "Webhook deliveries", labelNames: ["ok"], registers: [registry] }), | |
| 23 | + changeClass: new client.Counter({ name: "websensor_change_class_total", help: "Semantic class of raw changes", labelNames: ["class"], registers: [registry] }), | |
| 24 | + suppressed: new client.Counter({ name: "websensor_suppressed_total", help: "Candidates suppressed before becoming events", labelNames: ["reason"], registers: [registry] }), | |
| 25 | + stageLatency: new client.Histogram({ name: "websensor_stage_seconds", help: "Pipeline stage latency", labelNames: ["stage"], buckets: [0.005, 0.02, 0.05, 0.1, 0.25, 0.5, 1, 2, 5], registers: [registry] }), | |
| 26 | + prunedBlobs: new client.Counter({ name: "websensor_pruned_blobs_total", help: "Raw snapshot bodies pruned by retention", registers: [registry] }), | |
| 21 | 27 | }; |
| 22 | 28 | |
| 29 | +/** Per-source daily counters (source quality, heatmaps). */ | |
| 30 | +export async function bumpSourceDaily(sourceId: string, fields: Partial<Record<"checks" | "not_modified" | "errors" | "raw_changes" | "events", number>>): Promise<void> { | |
| 31 | + const keys = Object.keys(fields) as (keyof typeof fields)[]; | |
| 32 | + if (!keys.length) return; | |
| 33 | + const sets = keys.map((k) => sql.raw(`${k} = source_daily.${k} + ${Number(fields[k] ?? 0)}`)); | |
| 34 | + const cols = keys.map((k) => sql.raw(k)); | |
| 35 | + const vals = keys.map((k) => sql`${Number(fields[k] ?? 0)}`); | |
| 36 | + try { | |
| 37 | + await db.execute(sql`insert into source_daily (source_id, day, ${sql.join(cols, sql`, `)}) values (${sourceId}, (now() at time zone 'UTC')::date, ${sql.join(vals, sql`, `)}) on conflict (source_id, day) do update set ${sql.join(sets, sql`, `)}`); | |
| 38 | + } catch (e) { | |
| 39 | + log.warn({ err: (e as Error).message }, "source_daily update failed"); | |
| 40 | + } | |
| 41 | +} | |
| 42 | + | |
| 23 | 43 | /** Daily counters in Postgres (feed the public homepage stats). */ |
| 24 | 44 | export async function bumpDaily(fields: Partial<Record<"checks" | "not_modified" | "bytes" | "raw_changes" | "events" | "silent_events" | "errors" | "llm_calls" | "llm_input_tokens" | "llm_output_tokens" | "scrapfly_calls", number>>): Promise<void> { |
| 25 | 45 | const keys = Object.keys(fields) as (keyof typeof fields)[]; |
modified
apps/engine/src/pipeline.ts
+142 −30
@@ -1,12 +1,13 @@ | ||
| 1 | −import { computeConfidence, computeImportance, describeChange, diffIsEmpty, diffJson, diffList, diffText, evaluateChange, eventTypeSpec, newId, nextIntervalSeconds, slugify, sourceImportanceFromTier, summarizeDiff, type DiffResult, type HeuristicResult, type NormalizedContent, type Observation, type SensorEndpoint, type Tier, FEED_CHANNELS } from "@websensor/core"; | |
| 1 | +import { activityAnomaly, classifyChange, computeConfidence, computeImpact, computeImportance, computeSignalScore, describeChange, describeFieldChanges, diffIsEmpty, diffJson, diffList, diffText, evaluateChange, eventGroupOf, eventTypeSpec, newId, nextIntervalSeconds, NOISE_CLASSES, sha256, SILENT_ELIGIBLE_TYPES, slugify, sourceImportanceFromTier, summarizeDiff, type DiffResult, type HeuristicResult, type NormalizedContent, type Observation, type SemanticResult, type SensorEndpoint, type Tier, FEED_CHANNELS } from "@websensor/core"; | |
| 2 | 2 | import { getConnector, NormalizeError, PARSER_VERSION, scrapflyAvailable, scrapflyFetch } from "@websensor/connectors"; |
| 3 | 3 | import { changes, db, eventEntities, events, interpretations, sensorRuns, sensors, snapshots, sql, textArray, type Sensor, type Source } from "@websensor/db"; |
| 4 | 4 | import { getBlobStore } from "@websensor/store"; |
| 5 | +import { evaluateAlerts, type PublishedEvent } from "./alerts"; | |
| 5 | 6 | import { assessNovelty, clusterEvent, recentAnnouncementSimilarity } from "./cluster"; |
| 6 | 7 | import { config, log } from "./config"; |
| 7 | 8 | import { bumpEntityCounters, resolveEntities } from "./entities"; |
| 8 | 9 | import { interpretChange, llmAvailable, renderDiff, type Interpretation } from "./interpret"; |
| 9 | −import { bumpDaily, m } from "./metrics"; | |
| 10 | +import { bumpDaily, bumpSourceDaily, m } from "./metrics"; | |
| 10 | 11 | import { publishChange, publishEvent } from "./redis"; |
| 11 | 12 | |
| 12 | 13 | export type RunOutcome = "baseline" | "unchanged" | "not_modified" | "changed" | "event" | "error" | "missing" | "rate_limited" | "parse_error"; |
@@ -53,6 +54,7 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc | ||
| 53 | 54 | const isErr = outcome === "error" || outcome === "parse_error" || outcome === "rate_limited"; |
| 54 | 55 | const consecutive = isErr ? sensor.consecutiveErrors + 1 : 0; |
| 55 | 56 | const health = outcome === "rate_limited" ? "RATE_LIMITED" : isErr ? (consecutive >= 5 ? "ERROR" : "DEGRADED") : outcome === "missing" ? "DEGRADED" : "UP"; |
| 57 | + const status = !sensor.enabled ? "DISABLED" : health === "ERROR" || health === "RATE_LIMITED" ? "DEGRADED" : "ACTIVE"; | |
| 56 | 58 | const changes7d = await countChanges7d(sensor.id); |
| 57 | 59 | const events7d = await countEvents7d(sensor.id); |
| 58 | 60 | let interval = nextIntervalSeconds({ tier: sensor.tier as Tier, baseIntervalSeconds: sensor.baseIntervalSeconds, lastChangeAt: extra.changed ? new Date() : sensor.lastChangeAt, changes7d, events7d, consecutiveErrors: consecutive, lastWas304: outcome === "not_modified" }); |
@@ -67,6 +69,7 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc | ||
| 67 | 69 | lastError: isErr || outcome === "missing" ? (extra.error ?? obs.error?.message ?? `HTTP ${obs.meta.status}`).slice(0, 500) : null, |
| 68 | 70 | consecutiveErrors: consecutive, |
| 69 | 71 | health, |
| 72 | + status, | |
| 70 | 73 | totalRuns: sensor.totalRuns + 1, |
| 71 | 74 | totalNotModified: sensor.totalNotModified + (outcome === "not_modified" ? 1 : 0), |
| 72 | 75 | avgLatencyMs: avg, |
@@ -74,11 +77,13 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc | ||
| 74 | 77 | updatedAt: new Date(), |
| 75 | 78 | ...(extra.changed ? { lastChangeAt: new Date() } : {}), |
| 76 | 79 | ...(extra.eventAt ? { lastEventAt: extra.eventAt } : {}), |
| 80 | + ...(!sensor.validatedAt && !isErr && outcome !== "missing" ? { validatedAt: new Date() } : {}), | |
| 77 | 81 | ...(obs.meta.status >= 200 && obs.meta.status < 300 ? { etag: obs.meta.etag ?? sensor.etag, lastModified: obs.meta.lastModified ?? sensor.lastModified } : {}), |
| 78 | 82 | }) |
| 79 | 83 | .where(sql`id = ${sensor.id}`); |
| 80 | 84 | m.checks.inc({ connector: sensor.connector, outcome }); |
| 81 | 85 | await bumpDaily({ checks: 1, bytes: obs.meta.contentLength, not_modified: outcome === "not_modified" ? 1 : 0, errors: isErr ? 1 : 0 }); |
| 86 | + await bumpSourceDaily(source.id, { checks: 1, not_modified: outcome === "not_modified" ? 1 : 0, errors: isErr ? 1 : 0 }); | |
| 82 | 87 | return outcome; |
| 83 | 88 | }; |
| 84 | 89 | |
@@ -96,12 +101,15 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc | ||
| 96 | 101 | |
| 97 | 102 | // ---- Normalize -------------------------------------------------------------------- |
| 98 | 103 | let norm: NormalizedContent; |
| 104 | + const tNorm = m.stageLatency.startTimer({ stage: "normalize" }); | |
| 99 | 105 | try { |
| 100 | 106 | norm = await connector.normalize(endpoint, obs); |
| 101 | 107 | } catch (e) { |
| 108 | + tNorm(); | |
| 102 | 109 | const msg = e instanceof NormalizeError ? `${e.code}: ${e.message}` : (e as Error).message; |
| 103 | 110 | return finish("parse_error", { error: msg }); |
| 104 | 111 | } |
| 112 | + tNorm(); | |
| 105 | 113 | if (norm.state) await db.update(sensors).set({ state: norm.state }).where(sql`id = ${sensor.id}`); |
| 106 | 114 | |
| 107 | 115 | // Restore: page was flagged missing but is back. |
@@ -151,13 +159,17 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc | ||
| 151 | 159 | if (!prev) return finish("baseline", { snapshotId: snapId }); |
| 152 | 160 | |
| 153 | 161 | // ---- Diff ----------------------------------------------------------------------------- |
| 162 | + const tDiff = m.stageLatency.startTimer({ stage: "diff" }); | |
| 154 | 163 | let prevBlob: CanonicalBlob | null = null; |
| 155 | 164 | try { |
| 156 | 165 | prevBlob = prev.canonicalStorageKey ? (JSON.parse(await store.getText(prev.canonicalStorageKey)) as CanonicalBlob) : null; |
| 157 | 166 | } catch (e) { |
| 158 | 167 | log.warn({ sensor: sensor.id, err: (e as Error).message }, "previous canonical blob unreadable"); |
| 159 | 168 | } |
| 160 | − if (!prevBlob || prevBlob.mode !== norm.mode) return finish("changed", { snapshotId: snapId, changed: true }); | |
| 169 | + if (!prevBlob || prevBlob.mode !== norm.mode) { | |
| 170 | + tDiff(); | |
| 171 | + return finish("changed", { snapshotId: snapId, changed: true }); | |
| 172 | + } | |
| 161 | 173 | |
| 162 | 174 | let diff: DiffResult; |
| 163 | 175 | if (norm.mode === "list") { |
@@ -184,45 +196,88 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc | ||
| 184 | 196 | } else { |
| 185 | 197 | diff = diffText(prevBlob.text ?? "", norm.text ?? "", `${sensor.id}@${prev.capturedAt.toISOString()}`, `${sensor.id}@${obs.fetchedAt.toISOString()}`); |
| 186 | 198 | } |
| 199 | + tDiff(); | |
| 187 | 200 | if (diffIsEmpty(diff)) return finish("unchanged", { snapshotId: snapId }); |
| 188 | 201 | |
| 202 | + // ---- Heuristics + semantic classification ------------------------------------------------ | |
| 203 | + const tCls = m.stageLatency.startTimer({ stage: "classify" }); | |
| 189 | 204 | const heuristic = evaluateChange(diff, { sensorType: sensor.type, url: sensor.url, sourceCategories: source.categories, title: norm.title }); |
| 205 | + const semantic = classifyChange(diff, { url: sensor.url, sensorType: sensor.type, title: norm.title }); | |
| 206 | + tCls(); | |
| 207 | + m.changeClass.inc({ class: semantic.class }); | |
| 190 | 208 | const thinFlip = (prevBlob.extractionConfidence ?? 1) < 0.5 || norm.extractionConfidence < 0.5; |
| 191 | − const meaningfulByRules = heuristic.signal >= config.meaningfulSignal && !thinFlip; | |
| 209 | + const isNoise = NOISE_CLASSES.has(semantic.class) && !heuristic.facts.some((f) => f.kind === "price" && f.before && f.after); | |
| 210 | + const meaningfulByRules = heuristic.signal >= config.meaningfulSignal && !thinFlip && !isNoise; | |
| 192 | 211 | const changeId = newId("chg"); |
| 193 | 212 | const diffBlob = await store.put(diff.kind === "text" ? diff.unified : renderDiff(diff)); |
| 194 | − await db.insert(changes).values({ id: changeId, sensorId: sensor.id, oldSnapshotId: prev.id, newSnapshotId: snapId, detectedAt: obs.fetchedAt, kind: diff.kind, diff: summarizeDiff(diff), diffStorageKey: diffBlob.key, signal: heuristic.signal, noiseRatio: heuristic.noiseRatio, magnitude: heuristic.magnitude, heuristic: heuristic as unknown as Record<string, unknown>, meaningful: false }); | |
| 213 | + await db.insert(changes).values({ id: changeId, sensorId: sensor.id, oldSnapshotId: prev.id, newSnapshotId: snapId, detectedAt: obs.fetchedAt, kind: diff.kind, diff: summarizeDiff(diff), diffStorageKey: diffBlob.key, signal: heuristic.signal, noiseRatio: Math.max(heuristic.noiseRatio, semantic.noiseRatio), magnitude: heuristic.magnitude, heuristic: { ...(heuristic as unknown as Record<string, unknown>), semantic: { class: semantic.class, confidence: semantic.confidence, reasons: semantic.reasons } }, meaningful: false, changeClass: semantic.class, fieldChanges: semantic.fieldChanges.length ? semantic.fieldChanges : null }); | |
| 195 | 214 | await db.update(sensors).set({ rawChanges: sensor.rawChanges + 1 }).where(sql`id = ${sensor.id}`); |
| 196 | 215 | await db.execute(sql`insert into url_history (url, at, kind, snapshot_id, change_id) values (${sensor.url}, ${obs.fetchedAt}, 'change', ${snapId}, ${changeId})`); |
| 197 | 216 | await db.execute(sql`update urls set change_count = change_count + 1 where url = ${sensor.url}`); |
| 198 | 217 | await bumpDaily({ raw_changes: 1 }); |
| 218 | + await bumpSourceDaily(source.id, { raw_changes: 1 }); | |
| 199 | 219 | m.changes.inc({ connector: sensor.connector, meaningful: String(meaningfulByRules) }); |
| 200 | − await publishChange({ id: changeId, sensorId: sensor.id, sourceId: source.id, kind: diff.kind, signal: heuristic.signal, at: obs.fetchedAt.toISOString() }); | |
| 201 | − log.info({ sensor: sensor.id, kind: diff.kind, signal: heuristic.signal.toFixed(2), type: heuristic.eventType, thin: thinFlip }, "change detected"); | |
| 220 | + await publishChange({ id: changeId, sensorId: sensor.id, sourceId: source.id, kind: diff.kind, signal: heuristic.signal, class: semantic.class, at: obs.fetchedAt.toISOString() }); | |
| 221 | + log.info({ sensor: sensor.id, kind: diff.kind, signal: heuristic.signal.toFixed(2), type: heuristic.eventType, class: semantic.class, thin: thinFlip }, "change detected"); | |
| 202 | 222 | |
| 203 | − if (!meaningfulByRules) return finish("changed", { snapshotId: snapId, changed: true }); | |
| 223 | + if (!meaningfulByRules) { | |
| 224 | + if (isNoise) m.suppressed.inc({ reason: `noise_${semantic.class}` }); | |
| 225 | + return finish("changed", { snapshotId: snapId, changed: true }); | |
| 226 | + } | |
| 204 | 227 | |
| 205 | 228 | // ---- Event -------------------------------------------------------------------------- |
| 206 | − const ev = await createEvent({ sensor, source, obs, norm, prev, snapId, changeId, diff, heuristic }); | |
| 229 | + const ev = await createEvent({ sensor, source, obs, norm, prev, snapId, changeId, diff, heuristic, semantic }); | |
| 207 | 230 | if (!ev) return finish("changed", { snapshotId: snapId, changed: true }); |
| 208 | 231 | return finish("event", { snapshotId: snapId, changed: true, eventAt: ev.detectedAt }); |
| 209 | 232 | } |
| 210 | 233 | |
| 211 | −async function createEvent(ctx: { sensor: Sensor; source: Source; obs: Observation; norm: NormalizedContent; prev: { id: string; capturedAt: Date }; snapId: string; changeId: string; diff: DiffResult; heuristic: HeuristicResult }): Promise<{ id: string; detectedAt: Date } | null> { | |
| 212 | − const { sensor, source, obs, norm, prev, snapId, changeId, diff, heuristic } = ctx; | |
| 234 | +async function createEvent(ctx: { sensor: Sensor; source: Source; obs: Observation; norm: NormalizedContent; prev: { id: string; capturedAt: Date; canonicalHash: string }; snapId: string; changeId: string; diff: DiffResult; heuristic: HeuristicResult; semantic: SemanticResult }): Promise<{ id: string; detectedAt: Date } | null> { | |
| 235 | + const { sensor, source, obs, norm, prev, snapId, changeId, diff, heuristic, semantic } = ctx; | |
| 213 | 236 | const detectedAt = obs.fetchedAt; |
| 237 | + const tEv = m.stageLatency.startTimer({ stage: "event" }); | |
| 238 | + | |
| 239 | + // Idempotency (spec §90): the same observation (same sensor, same before/after canonical content) is one event. | |
| 240 | + const fingerprint = sha256(`${sensor.id}|${prev.canonicalHash}|${norm.canonicalHash}`); | |
| 241 | + const dup = await db.execute<{ id: string }>(sql`select id from events where fingerprint = ${fingerprint} limit 1`); | |
| 242 | + if (dup.rows[0]) { | |
| 243 | + m.suppressed.inc({ reason: "fingerprint_duplicate" }); | |
| 244 | + log.info({ sensor: sensor.id, existing: dup.rows[0].id }, "duplicate observation — event already exists"); | |
| 245 | + tEv(); | |
| 246 | + return null; | |
| 247 | + } | |
| 248 | + | |
| 214 | 249 | const base = describeChange(heuristic, diff, { sourceName: source.name, url: sensor.url, sensorName: sensor.name }); |
| 250 | + if (semantic.fieldChanges.length && diff.kind !== "list" && !heuristic.facts.some((f) => f.kind === "price" && f.before && f.after)) { | |
| 251 | + base.summary = `${describeFieldChanges(semantic.fieldChanges, 4)}. ${base.summary}`.slice(0, 1500); | |
| 252 | + } | |
| 215 | 253 | const textForMatching = `${base.title}\n${base.summary}\n${renderDiff(diff).slice(0, 4000)}`; |
| 216 | 254 | |
| 217 | 255 | const ent = await resolveEntities({ sourceId: source.id, text: textForMatching, hints: heuristic.keywords }); |
| 218 | 256 | const nov = await assessNovelty(`${base.title}\n${base.summary}`, source.id); |
| 219 | 257 | const sourceImportance = sourceImportanceFromTier(sensor.tier, source.importanceWeight * sensor.importanceWeight); |
| 258 | + const firstParty = source.firstParty !== false; | |
| 259 | + const anomaly = await sourceAnomaly(source.id); | |
| 220 | 260 | let eventType = heuristic.eventType; |
| 221 | − let prelim = computeImportance({ eventType, sourceImportance, entityImportance: ent.importance, novelty: nov.novelty, magnitude: heuristic.magnitude, confirmations: nov.confirmations }); | |
| 261 | + let prelim = computeImportance({ eventType, sourceImportance, entityImportance: ent.importance, novelty: nov.novelty, magnitude: heuristic.magnitude, confirmations: nov.confirmations, unusualness: anomaly }); | |
| 262 | + // Routine batches (spec §13, §55): "21 new CVEs" from a firehose that fires several times a day is normal | |
| 263 | + // operation, not a signal. Damp importance for high-volume list sensors; single items keep full weight. | |
| 264 | + const routineBatch = diff.kind === "list" && diff.added.length >= 8 && (await countEvents24h(sensor.id)) >= 3; | |
| 265 | + if (routineBatch) { | |
| 266 | + prelim.score = Math.round(prelim.score * 0.72 * 10) / 10; | |
| 267 | + prelim.components.novelty = Math.min(prelim.components.novelty, 35); | |
| 268 | + } | |
| 222 | 269 | |
| 223 | 270 | // Duplicate suppression: near-identical to something we already published (syndication / re-fetch). |
| 224 | 271 | if (nov.novelty < 12 && nov.nearest) { |
| 272 | + m.suppressed.inc({ reason: "near_duplicate" }); | |
| 225 | 273 | log.info({ sensor: sensor.id, nearest: nov.nearest.id, sim: nov.nearest.similarity }, "suppressed near-duplicate event"); |
| 274 | + tEv(); | |
| 275 | + return null; | |
| 276 | + } | |
| 277 | + // Third-party reports of a story we already hold from several sources add little: raise the bar. | |
| 278 | + if (!firstParty && nov.novelty < 30 && nov.confirmations >= 3 && prelim.score < 70) { | |
| 279 | + m.suppressed.inc({ reason: "redundant_external" }); | |
| 280 | + tEv(); | |
| 226 | 281 | return null; |
| 227 | 282 | } |
| 228 | 283 | |
@@ -230,14 +285,16 @@ async function createEvent(ctx: { sensor: Sensor; source: Source; obs: Observati | ||
| 230 | 285 | if (llmAvailable() && source.llmEnabled && prelim.score >= config.llm.minImportance) { |
| 231 | 286 | llm = await interpretChange({ sourceName: source.name, sourceCategories: source.categories, url: sensor.url, sensorName: sensor.name, sensorType: sensor.type, heuristic, diff, prelimImportance: prelim.score, title: norm.title }); |
| 232 | 287 | if (llm && !llm.meaningful) { |
| 288 | + m.suppressed.inc({ reason: "llm_not_meaningful" }); | |
| 233 | 289 | log.info({ sensor: sensor.id, type: llm.event_type }, "LLM judged change not meaningful"); |
| 234 | − await db.update(changes).set({ heuristic: { ...(heuristic as unknown as Record<string, unknown>), llm: { meaningful: false, model: llm.model, title: llm.title } } }).where(sql`id = ${changeId}`); | |
| 290 | + await db.update(changes).set({ heuristic: { ...(heuristic as unknown as Record<string, unknown>), semantic: { class: semantic.class, confidence: semantic.confidence }, llm: { meaningful: false, model: llm.model, title: llm.title } } }).where(sql`id = ${changeId}`); | |
| 291 | + tEv(); | |
| 235 | 292 | return null; |
| 236 | 293 | } |
| 237 | 294 | if (llm) { |
| 238 | 295 | eventType = llm.event_type; |
| 239 | 296 | const spec = eventTypeSpec(eventType); |
| 240 | − prelim = computeImportance({ eventType, sourceImportance, entityImportance: ent.importance, novelty: nov.novelty, magnitude: heuristic.magnitude, confirmations: nov.confirmations }); | |
| 297 | + prelim = computeImportance({ eventType, sourceImportance, entityImportance: ent.importance, novelty: nov.novelty, magnitude: heuristic.magnitude, confirmations: nov.confirmations, unusualness: anomaly }); | |
| 241 | 298 | prelim.components.severity = Math.round((spec.severity + llm.severity) / 2); |
| 242 | 299 | prelim.score = Math.round(10 * (0.25 * prelim.components.severity + 0.2 * prelim.components.source + 0.15 * prelim.components.entity + 0.15 * prelim.components.novelty + 0.1 * prelim.components.magnitude + 0.05 * prelim.components.confirmation + 0.05 * prelim.components.userImpact + 0.05 * prelim.components.unusualness)) / 10; |
| 243 | 300 | } |
@@ -248,16 +305,17 @@ async function createEvent(ctx: { sensor: Sensor; source: Source; obs: Observati | ||
| 248 | 305 | const spec = eventTypeSpec(eventType); |
| 249 | 306 | const isAnnouncementSensor = ANNOUNCEMENT_SENSORS.has(sensor.type); |
| 250 | 307 | const announcedSim = isAnnouncementSensor ? 1 : recentAnnouncementSimilarity(source.id, `${title}\n${summary}`, 12 * 3600e3); |
| 251 | − const silentChange = !isAnnouncementSensor && (llm ? llm.announced === false : !spec.usuallyAnnounced) && announcedSim < 0.3 && prelim.score >= 35; | |
| 308 | + // Silent-change bar (spec §23): first-party, silent-eligible type, no matching announcement, real content, importance ≥ bar. | |
| 309 | + const silentChange = firstParty && !isAnnouncementSensor && SILENT_ELIGIBLE_TYPES.has(eventType) && (llm ? llm.announced === false : !spec.usuallyAnnounced) && announcedSim < 0.3 && prelim.score >= config.silentMinImportance && !NOISE_CLASSES.has(semantic.class); | |
| 252 | 310 | |
| 253 | 311 | // Entities named by the LLM that resolve to known aliases |
| 254 | 312 | const extra = llm ? await resolveEntities({ sourceId: source.id, text: llm.entities.join("\n"), hints: [] }) : null; |
| 255 | 313 | const entityIds = [...new Set([...ent.subject, ...ent.mentioned, ...(extra?.mentioned ?? [])])]; |
| 256 | 314 | |
| 257 | 315 | const confidence = computeConfidence({ |
| 258 | − sourceAuthenticity: 1, | |
| 316 | + sourceAuthenticity: firstParty ? 1 : 0.6, | |
| 259 | 317 | extraction: norm.extractionConfidence, |
| 260 | − diffClarity: 1 - heuristic.noiseRatio, | |
| 318 | + diffClarity: 1 - Math.max(heuristic.noiseRatio, semantic.noiseRatio), | |
| 261 | 319 | structured: diff.kind !== "text", |
| 262 | 320 | confirmations: nov.confirmations, |
| 263 | 321 | llmAgreement: llm ? (llm.event_type === heuristic.eventType ? 1 : 0.6) * (llm.confidence / 100) : heuristic.signal, |
@@ -265,11 +323,15 @@ async function createEvent(ctx: { sensor: Sensor; source: Source; obs: Observati | ||
| 265 | 323 | const evidenceLabel = nov.confirmations > 0 ? "CONFIRMED" : llm && llm.inferred && llm.confidence < 60 ? "INFERRED" : !llm && heuristic.signal < 0.5 ? "UNCONFIRMED" : "OBSERVED"; |
| 266 | 324 | const categories = [...new Set([...source.categories, ...Object.entries(FEED_CHANNELS).filter(([, cats]) => cats.some((c) => source.categories.includes(c))).map(([ch]) => ch)])]; |
| 267 | 325 | const keywords = [...new Set([...(llm?.keywords ?? []), ...heuristic.keywords])].slice(0, 20); |
| 326 | + const maxDelta = semantic.fieldChanges.reduce((mx, f) => Math.max(mx, Math.abs(f.deltaPct ?? 0)), 0); | |
| 327 | + const impact = computeImpact({ eventType, magnitude: heuristic.magnitude, entityImportance: ent.importance, maxDeltaPct: maxDelta, fieldChanges: semantic.fieldChanges.length, firstParty }); | |
| 268 | 328 | const id = newId("evt"); |
| 269 | 329 | const slug = `${slugify(title).slice(0, 70)}-${id.slice(-6)}`; |
| 270 | 330 | const processedAt = new Date(); |
| 271 | 331 | const publishedAt = pickPublishedAt(diff, norm); |
| 272 | 332 | const observedFrom = prev.capturedAt; |
| 333 | + const country = source.country ?? null; | |
| 334 | + const language = source.language ?? "en"; | |
| 273 | 335 | |
| 274 | 336 | await db.insert(events).values({ |
| 275 | 337 | id, |
@@ -280,6 +342,7 @@ async function createEvent(ctx: { sensor: Sensor; source: Source; obs: Observati | ||
| 280 | 342 | oldSnapshotId: prev.id, |
| 281 | 343 | newSnapshotId: snapId, |
| 282 | 344 | url: sensor.url, |
| 345 | + canonicalUrl: obs.meta.finalUrl || sensor.url, | |
| 283 | 346 | eventType, |
| 284 | 347 | title, |
| 285 | 348 | summary, |
@@ -288,10 +351,18 @@ async function createEvent(ctx: { sensor: Sensor; source: Source; obs: Observati | ||
| 288 | 351 | importanceComponents: prelim.components as unknown as Record<string, number>, |
| 289 | 352 | confidence, |
| 290 | 353 | novelty: nov.novelty, |
| 354 | + impactScore: impact, | |
| 355 | + anomalyScore: anomaly, | |
| 291 | 356 | categories, |
| 292 | 357 | keywords, |
| 293 | 358 | silentChange, |
| 294 | 359 | evidenceLabel, |
| 360 | + firstParty, | |
| 361 | + country, | |
| 362 | + language, | |
| 363 | + changeClass: semantic.class, | |
| 364 | + fieldChanges: semantic.fieldChanges.length ? semantic.fieldChanges : null, | |
| 365 | + fingerprint, | |
| 295 | 366 | publishedAt, |
| 296 | 367 | observedFrom, |
| 297 | 368 | detectedAt, |
@@ -299,43 +370,63 @@ async function createEvent(ctx: { sensor: Sensor; source: Source; obs: Observati | ||
| 299 | 370 | detectionLatencyMs: publishedAt && detectedAt.getTime() - publishedAt.getTime() < 7 * 86400e3 ? Math.max(0, detectedAt.getTime() - publishedAt.getTime()) : null, |
| 300 | 371 | processingLatencyMs: processedAt.getTime() - detectedAt.getTime(), |
| 301 | 372 | processingVersion: config.processingVersion, |
| 302 | − interpretation: llm ? { model: llm.model, observed: llm.observed, inferred: llm.inferred, who_it_affects: llm.who_it_affects, severity: llm.severity, confidence: llm.confidence, announced: llm.announced, entities: llm.entities } : { model: "heuristics-v1", reasons: heuristic.reasons, facts: heuristic.facts }, | |
| 373 | + interpretation: llm ? { model: llm.model, observed: llm.observed, inferred: llm.inferred, who_it_affects: llm.who_it_affects, severity: llm.severity, confidence: llm.confidence, announced: llm.announced, entities: llm.entities } : { model: "heuristics-v2", reasons: [...heuristic.reasons, ...semantic.reasons], facts: heuristic.facts }, | |
| 303 | 374 | }); |
| 304 | − await db.insert(interpretations).values({ eventId: id, version: 1, model: llm?.model ?? "heuristics-v1", payload: { heuristic: heuristic as unknown as Record<string, unknown>, llm: llm as unknown as Record<string, unknown> | null } }); | |
| 375 | + await db.insert(interpretations).values({ eventId: id, version: 1, model: llm?.model ?? "heuristics-v2", payload: { heuristic: heuristic as unknown as Record<string, unknown>, semantic: { class: semantic.class, confidence: semantic.confidence, reasons: semantic.reasons, fieldChanges: semantic.fieldChanges }, llm: llm as unknown as Record<string, unknown> | null } }); | |
| 305 | 376 | for (const e of entityIds) await db.insert(eventEntities).values({ eventId: id, entityId: e, role: ent.subject.includes(e) ? "subject" : "mentioned" }).onConflictDoNothing(); |
| 306 | − await bumpEntityCounters(entityIds, detectedAt); | |
| 377 | + await bumpEntityCounters(entityIds, detectedAt, { silent: silentChange, importance: prelim.score }); | |
| 307 | 378 | await db.update(changes).set({ meaningful: true, eventId: id }).where(sql`id = ${changeId}`); |
| 308 | 379 | await db.update(sensors).set({ meaningfulChanges: sensor.meaningfulChanges + 1 }).where(sql`id = ${sensor.id}`); |
| 309 | 380 | await db.execute(sql`insert into url_history (url, at, kind, snapshot_id, change_id, event_id) values (${sensor.url}, ${detectedAt}, 'event', ${snapId}, ${changeId}, ${id})`); |
| 310 | 381 | |
| 311 | − const cl = await clusterEvent({ id, sourceId: source.id, sensorId: sensor.id, eventType, entityIds, detectedAt, title, summary, importance: prelim.score, categories }); | |
| 312 | − await db.update(events).set({ clusterId: cl.clusterId, publishedToFeedAt: new Date() }).where(sql`id = ${id}`); | |
| 382 | + // Provisional signal for clustering; recomputed with the cluster's velocity right after. | |
| 383 | + const provisional = computeSignalScore({ importance: prelim.score, confidence, novelty: nov.novelty, velocity: 0, impact, anomaly, confirmations: nov.confirmations, firstParty, silent: silentChange, changeClass: semantic.class, sourceTier: sensor.tier, evidenceLabel }); | |
| 384 | + const cl = await clusterEvent({ id, sourceId: source.id, sourceName: source.name, sensorId: sensor.id, sensorType: sensor.type, eventType, entityIds, detectedAt, title, summary, importance: prelim.score, signal: provisional.score, categories, firstParty, sourceTier: sensor.tier }); | |
| 385 | + const signal = computeSignalScore({ importance: prelim.score, confidence, novelty: nov.novelty, velocity: cl.velocity, impact, anomaly, confirmations: Math.max(nov.confirmations, cl.sourceCount - 1), firstParty, silent: silentChange, changeClass: semantic.class, sourceTier: sensor.tier, evidenceLabel }); | |
| 386 | + if (routineBatch) signal.reasons.push({ sign: "-", text: "routine batch from a high-volume feed", points: -8 }); | |
| 387 | + await db.update(events).set({ clusterId: cl.clusterId, publishedToFeedAt: new Date(), signalScore: signal.score, velocityScore: cl.velocity, scoreReasons: signal.reasons }).where(sql`id = ${id}`); | |
| 388 | + await bumpSourceDaily(source.id, { events: 1 }); | |
| 313 | 389 | |
| 314 | 390 | const entNames = entityIds.length ? await db.execute<{ id: string; name: string; type: string }>(sql`select id, name, type from entities where id = any(${textArray(entityIds)})`) : { rows: [] as { id: string; name: string; type: string }[] }; |
| 315 | − await publishEvent({ | |
| 391 | + const payload: PublishedEvent & Record<string, unknown> = { | |
| 316 | 392 | id, |
| 317 | 393 | slug, |
| 318 | 394 | type: eventType, |
| 395 | + group: eventGroupOf(eventType), | |
| 319 | 396 | title, |
| 320 | 397 | summary: summary.slice(0, 280), |
| 321 | 398 | importance: prelim.score, |
| 399 | + signal: signal.score, | |
| 322 | 400 | confidence, |
| 323 | 401 | novelty: nov.novelty, |
| 402 | + impact, | |
| 403 | + velocity: cl.velocity, | |
| 324 | 404 | silent: silentChange, |
| 325 | 405 | evidence: evidenceLabel, |
| 326 | − source: { id: source.id, name: source.name, domain: source.domain }, | |
| 406 | + firstParty, | |
| 407 | + country, | |
| 408 | + language, | |
| 409 | + changeClass: semantic.class, | |
| 410 | + fieldChanges: semantic.fieldChanges.slice(0, 3), | |
| 411 | + source: { id: source.id, name: source.name, domain: source.domain, tier: source.tier }, | |
| 327 | 412 | sensor: { id: sensor.id, name: sensor.name, type: sensor.type }, |
| 328 | 413 | entities: entNames.rows.map((r) => ({ id: r.id, name: r.name, type: r.type })), |
| 329 | 414 | categories, |
| 330 | 415 | url: sensor.url, |
| 331 | 416 | clusterId: cl.clusterId, |
| 417 | + clusterSlug: cl.slug, | |
| 418 | + clusterSize: cl.eventCount, | |
| 419 | + clusterState: cl.state, | |
| 332 | 420 | detectedAt: detectedAt.toISOString(), |
| 333 | 421 | publishedAt: publishedAt?.toISOString() ?? null, |
| 334 | − }); | |
| 422 | + }; | |
| 423 | + await publishEvent(payload); | |
| 424 | + void evaluateAlerts(payload).catch(() => undefined); | |
| 335 | 425 | m.events.inc({ event_type: eventType, silent: String(silentChange) }); |
| 336 | 426 | m.processingLatency.observe((Date.now() - detectedAt.getTime()) / 1000); |
| 337 | 427 | await bumpDaily({ events: 1, silent_events: silentChange ? 1 : 0 }); |
| 338 | − log.info({ event: id, source: source.id, type: eventType, importance: prelim.score, confidence, silent: silentChange, llm: llm?.model ?? "heuristics" }, title); | |
| 428 | + tEv(); | |
| 429 | + log.info({ event: id, source: source.id, type: eventType, importance: prelim.score, signal: signal.score, confidence, silent: silentChange, class: semantic.class, cluster: cl.state, llm: llm?.model ?? "heuristics" }, title); | |
| 339 | 430 | return { id, detectedAt }; |
| 340 | 431 | } |
| 341 | 432 | |
@@ -361,14 +452,21 @@ async function handleMissing(sensor: Sensor, source: Source, obs: Observation, f | ||
| 361 | 452 | const summary = `${sensor.url} has returned HTTP ${obs.meta.status} on ${missing.count} checks since ${new Date(missing.firstAt).toISOString()}. The last known version is preserved as a snapshot.`; |
| 362 | 453 | const imp = computeImportance({ eventType: "page_removed", sourceImportance: sourceImportanceFromTier(sensor.tier, source.importanceWeight), entityImportance: 50, novelty: 80, magnitude: 60, confirmations: 0 }); |
| 363 | 454 | const slug = `${slugify(title).slice(0, 70)}-${id.slice(-6)}`; |
| 364 | − await db.insert(events).values({ id, slug, sensorId: sensor.id, sourceId: source.id, oldSnapshotId: sensor.lastSnapshotId, url: sensor.url, eventType: "page_removed", title, summary, importance: imp.score, importanceComponents: imp.components as unknown as Record<string, number>, confidence: 85, novelty: 80, categories: source.categories, keywords: ["page removed"], silentChange: true, evidenceLabel: "CONFIRMED", detectedAt: obs.fetchedAt, processedAt: new Date(), processingVersion: config.processingVersion, interpretation: { model: "rules", checks: missing.count } }); | |
| 455 | + const firstParty = source.firstParty !== false; | |
| 456 | + const fingerprint = sha256(`${sensor.id}|removed|${sensor.lastSnapshotId}`); | |
| 457 | + const impact = computeImpact({ eventType: "page_removed", magnitude: 60, entityImportance: 50, firstParty }); | |
| 458 | + const sig = computeSignalScore({ importance: imp.score, confidence: 85, novelty: 80, velocity: 0, impact, anomaly: 0, confirmations: 0, firstParty, silent: true, sourceTier: sensor.tier, evidenceLabel: "CONFIRMED" }); | |
| 459 | + await db.insert(events).values({ id, slug, sensorId: sensor.id, sourceId: source.id, oldSnapshotId: sensor.lastSnapshotId, url: sensor.url, eventType: "page_removed", title, summary, importance: imp.score, importanceComponents: imp.components as unknown as Record<string, number>, confidence: 85, novelty: 80, impactScore: impact, signalScore: sig.score, scoreReasons: sig.reasons, categories: source.categories, keywords: ["page removed"], silentChange: true, evidenceLabel: "CONFIRMED", firstParty, country: source.country ?? null, language: source.language ?? "en", changeClass: "meaningful", fingerprint, detectedAt: obs.fetchedAt, processedAt: new Date(), processingVersion: config.processingVersion, interpretation: { model: "rules", checks: missing.count } }).onConflictDoNothing(); | |
| 365 | 460 | const ent = await resolveEntities({ sourceId: source.id, text: title, hints: [] }); |
| 366 | 461 | for (const e of ent.subject) await db.insert(eventEntities).values({ eventId: id, entityId: e, role: "subject" }).onConflictDoNothing(); |
| 462 | + await bumpEntityCounters(ent.subject, obs.fetchedAt, { silent: true, importance: imp.score }); | |
| 367 | 463 | await db.execute(sql`update urls set status = 'removed' where url = ${sensor.url}`); |
| 368 | 464 | await db.execute(sql`insert into url_history (url, at, kind, event_id, note) values (${sensor.url}, ${obs.fetchedAt}, 'removed', ${id}, ${`HTTP ${obs.meta.status}`})`); |
| 369 | − const cl = await clusterEvent({ id, sourceId: source.id, sensorId: sensor.id, eventType: "page_removed", entityIds: ent.subject, detectedAt: obs.fetchedAt, title, summary, importance: imp.score, categories: source.categories }); | |
| 370 | − await db.update(events).set({ clusterId: cl.clusterId, publishedToFeedAt: new Date() }).where(sql`id = ${id}`); | |
| 371 | − await publishEvent({ id, slug, type: "page_removed", title, summary, importance: imp.score, confidence: 85, novelty: 80, silent: true, evidence: "CONFIRMED", source: { id: source.id, name: source.name, domain: source.domain }, sensor: { id: sensor.id, name: sensor.name, type: sensor.type }, entities: [], categories: source.categories, url: sensor.url, clusterId: cl.clusterId, detectedAt: obs.fetchedAt.toISOString(), publishedAt: null }); | |
| 465 | + const cl = await clusterEvent({ id, sourceId: source.id, sourceName: source.name, sensorId: sensor.id, sensorType: sensor.type, eventType: "page_removed", entityIds: ent.subject, detectedAt: obs.fetchedAt, title, summary, importance: imp.score, signal: sig.score, categories: source.categories, firstParty, sourceTier: sensor.tier }); | |
| 466 | + await db.update(events).set({ clusterId: cl.clusterId, publishedToFeedAt: new Date(), velocityScore: cl.velocity }).where(sql`id = ${id}`); | |
| 467 | + const payload: PublishedEvent & Record<string, unknown> = { id, slug, type: "page_removed", group: "web", title, summary, importance: imp.score, signal: sig.score, confidence: 85, novelty: 80, silent: true, evidence: "CONFIRMED", firstParty, country: source.country ?? null, source: { id: source.id, name: source.name, domain: source.domain, tier: source.tier }, sensor: { id: sensor.id, name: sensor.name, type: sensor.type }, entities: [], categories: source.categories, url: sensor.url, clusterId: cl.clusterId, clusterSlug: cl.slug, clusterState: cl.state, detectedAt: obs.fetchedAt.toISOString(), publishedAt: null }; | |
| 468 | + await publishEvent(payload); | |
| 469 | + void evaluateAlerts(payload).catch(() => undefined); | |
| 372 | 470 | await bumpDaily({ events: 1, silent_events: 1 }); |
| 373 | 471 | log.info({ sensor: sensor.id, checks: missing.count }, "page removal confirmed"); |
| 374 | 472 | } else { |
@@ -398,7 +496,21 @@ async function countChanges7d(sensorId: string): Promise<number> { | ||
| 398 | 496 | const r = await db.execute<{ n: string }>(sql`select count(*)::text as n from changes where sensor_id = ${sensorId} and detected_at >= now() - interval '7 days'`); |
| 399 | 497 | return Number(r.rows[0]?.n ?? 0); |
| 400 | 498 | } |
| 499 | +async function countEvents24h(sensorId: string): Promise<number> { | |
| 500 | + const r = await db.execute<{ n: string }>(sql`select count(*)::text as n from events where sensor_id = ${sensorId} and detected_at >= now() - interval '24 hours'`); | |
| 501 | + return Number(r.rows[0]?.n ?? 0); | |
| 502 | +} | |
| 401 | 503 | async function countEvents7d(sensorId: string): Promise<number> { |
| 402 | 504 | const r = await db.execute<{ n: string }>(sql`select count(*)::text as n from events where sensor_id = ${sensorId} and detected_at >= now() - interval '7 days'`); |
| 403 | 505 | return Number(r.rows[0]?.n ?? 0); |
| 404 | 506 | } |
| 507 | + | |
| 508 | +/** Source activity anomaly at detection time (0–100): last 2 h of raw changes vs the 14-day hourly baseline. */ | |
| 509 | +async function sourceAnomaly(sourceId: string): Promise<number> { | |
| 510 | + const r = await db.execute<{ c2h: string; c14d: string }>(sql` | |
| 511 | + select (select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '2 hours')::text as c2h, | |
| 512 | + (select count(*) from changes c join sensors s on s.id = c.sensor_id where s.source_id = ${sourceId} and c.detected_at >= now() - interval '14 days')::text as c14d`); | |
| 513 | + const cur = Number(r.rows[0]?.c2h ?? 0) / 2; | |
| 514 | + const base = Number(r.rows[0]?.c14d ?? 0) / (14 * 24); | |
| 515 | + return activityAnomaly(cur, base); | |
| 516 | +} | |
modified
apps/engine/src/redis.ts
+19 −4
@@ -15,15 +15,30 @@ export const STREAM_EVENTS = "ws:events"; | ||
| 15 | 15 | export const CHANNEL_LIVE = "ws:live"; |
| 16 | 16 | export const STREAM_CHANGES = "ws:changes"; |
| 17 | 17 | |
| 18 | −/** Publish a compact event payload to the stream (durable) and the pub/sub channel (realtime). */ | |
| 19 | −export async function publishEvent(payload: Record<string, unknown>): Promise<void> { | |
| 18 | +/** | |
| 19 | + * Publish a compact event payload to the stream (durable, replayable by stream id) and the pub/sub | |
| 20 | + * channel (realtime). The stream id is embedded as `sid` so clients can resume with `since`. | |
| 21 | + */ | |
| 22 | +export async function publishEvent(payload: Record<string, unknown>): Promise<string | null> { | |
| 20 | 23 | const r = getRedis(); |
| 21 | − const json = JSON.stringify(payload); | |
| 22 | 24 | try { |
| 23 | − await r.xadd(STREAM_EVENTS, "MAXLEN", "~", "20000", "*", "event", json); | |
| 25 | + const sid = await r.xadd(STREAM_EVENTS, "MAXLEN", "~", "20000", "*", "event", JSON.stringify(payload)); | |
| 26 | + const json = JSON.stringify({ ...payload, sid }); | |
| 27 | + // keep the stream entry consistent with what subscribers saw | |
| 24 | 28 | await r.publish(CHANNEL_LIVE, json); |
| 29 | + return sid ?? null; | |
| 25 | 30 | } catch (e) { |
| 26 | 31 | log.warn({ err: (e as Error).message }, "publish failed"); |
| 32 | + return null; | |
| 33 | + } | |
| 34 | +} | |
| 35 | + | |
| 36 | +/** Engine heartbeat for the ops dashboard (`/api/v1/admin/ops`, `/api/ready`). */ | |
| 37 | +export async function publishEngineStatus(status: Record<string, unknown>): Promise<void> { | |
| 38 | + try { | |
| 39 | + await getRedis().set("ws:engine:status", JSON.stringify({ ...status, at: new Date().toISOString(), pid: process.pid }), "EX", 30); | |
| 40 | + } catch { | |
| 41 | + // best effort | |
| 27 | 42 | } |
| 28 | 43 | } |
| 29 | 44 | |
modified
apps/engine/src/registry.ts
+46 −6
@@ -21,10 +21,14 @@ export function loadSeeds(file = config.sourcesFile, dir = config.sourcesDir): S | ||
| 21 | 21 | export async function syncRegistry(seeds = loadSeeds()): Promise<{ sources: number; sensors: number }> { |
| 22 | 22 | let nSensors = 0; |
| 23 | 23 | for (const s of seeds) { |
| 24 | + // Provenance: media/aggregators are third-party; everything else is the organization's own channel. | |
| 25 | + const firstParty = s.first_party ?? !s.categories.some((c) => c === "news" || c === "media"); | |
| 26 | + const country = s.country ?? inferCountry(s.domain, s.categories); | |
| 27 | + const language = s.language ?? inferLanguage(s.domain); | |
| 24 | 28 | await db |
| 25 | 29 | .insert(sources) |
| 26 | − .values({ id: s.id, name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm }) | |
| 27 | − .onConflictDoUpdate({ target: sources.id, set: { name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, updatedAt: new Date() } }); | |
| 30 | + .values({ id: s.id, name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, firstParty, country, language }) | |
| 31 | + .onConflictDoUpdate({ target: sources.id, set: { name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, firstParty, country, language, updatedAt: new Date() } }); | |
| 28 | 32 | |
| 29 | 33 | // Organization entity + aliases |
| 30 | 34 | const entId = `org_${s.id}`; |
@@ -52,14 +56,16 @@ export async function syncRegistry(seeds = loadSeeds()): Promise<{ sources: numb | ||
| 52 | 56 | const id = sen.id ?? `${s.id}_${slugify(sen.name)}`; |
| 53 | 57 | seedIds.push(id); |
| 54 | 58 | const cfg = { ...sen.config, seed: true }; |
| 59 | + const tier = sen.tier ?? s.tier; | |
| 60 | + const priority = priorityFor(tier, s.categories); | |
| 55 | 61 | await db |
| 56 | 62 | .insert(sensors) |
| 57 | − .values({ id, sourceId: s.id, name: sen.name, url: sen.url, type: sen.type, connector: sen.connector, tier: sen.tier ?? s.tier, importanceWeight: sen.weight ?? 1, config: cfg, baseIntervalSeconds: sen.interval ?? null }) | |
| 58 | − .onConflictDoUpdate({ target: sensors.id, set: { name: sen.name, url: sen.url, type: sen.type, connector: sen.connector, tier: sen.tier ?? s.tier, importanceWeight: sen.weight ?? 1, config: cfg, baseIntervalSeconds: sen.interval ?? null, enabled: true, updatedAt: new Date() } }); | |
| 63 | + .values({ id, sourceId: s.id, name: sen.name, url: sen.url, type: sen.type, connector: sen.connector, tier, importanceWeight: sen.weight ?? 1, config: cfg, baseIntervalSeconds: sen.interval ?? null, priority, status: "VALIDATED", validatedAt: new Date() }) | |
| 64 | + .onConflictDoUpdate({ target: sensors.id, set: { name: sen.name, url: sen.url, type: sen.type, connector: sen.connector, tier, importanceWeight: sen.weight ?? 1, config: cfg, baseIntervalSeconds: sen.interval ?? null, priority, enabled: true, updatedAt: new Date() } }); | |
| 59 | 65 | nSensors++; |
| 60 | 66 | } |
| 61 | 67 | // Seed sensors removed from the YAML are disabled (history kept), discovery-created ones are untouched. |
| 62 | − await db.execute(sql`update sensors set enabled = false, updated_at = now() where source_id = ${s.id} and enabled and (config->>'seed') = 'true' and not (id = any(${textArray(seedIds)}))`); | |
| 68 | + await db.execute(sql`update sensors set enabled = false, status = 'DISABLED', updated_at = now() where source_id = ${s.id} and enabled and (config->>'seed') = 'true' and not (id = any(${textArray(seedIds)}))`); | |
| 63 | 69 | } |
| 64 | 70 | log.info({ sources: seeds.length, sensors: nSensors }, "registry synced"); |
| 65 | 71 | return { sources: seeds.length, sensors: nSensors }; |
@@ -112,7 +118,7 @@ export async function runDiscovery(opts: { onlyMissing?: boolean; sourceIds?: st | ||
| 112 | 118 | const tier = f.connector === "statuspage" ? "S" : f.connector === "sitemap" ? (src.tier === "S" ? "A" : src.tier) : src.tier; |
| 113 | 119 | await db |
| 114 | 120 | .insert(sensors) |
| 115 | − .values({ id, sourceId: src.id, name, url: f.url, type: f.type, connector: f.connector, tier, config: f.connector === "sitemap" ? { maxChildren: 4, maxUrls: 3000 } : {} }) | |
| 121 | + .values({ id, sourceId: src.id, name, url: f.url, type: f.type, connector: f.connector, tier, config: f.connector === "sitemap" ? { maxChildren: 4, maxUrls: 3000 } : {}, status: "VALIDATED", validatedAt: new Date(), priority: priorityFor(tier, src.categories) }) | |
| 116 | 122 | .onConflictDoNothing(); |
| 117 | 123 | await db.update(discoveryCandidates).set({ status: "promoted" }).where(sql`source_id = ${src.id} and url = ${f.url}`); |
| 118 | 124 | existingUrls.add(f.url.replace(/\/$/, "")); |
@@ -129,6 +135,40 @@ export async function runDiscovery(opts: { onlyMissing?: boolean; sourceIds?: st | ||
| 129 | 135 | return promoted; |
| 130 | 136 | } |
| 131 | 137 | |
| 138 | +/** Priority tier (spec §76): 0 critical infrastructure/government/major AI/cyber · 1 major companies/markets/science/health · 2 specialized · 3 low. */ | |
| 139 | +export function priorityFor(tier: string, categories: string[]): number { | |
| 140 | + if (tier === "S") return 0; | |
| 141 | + if (tier === "A" && categories.some((c) => ["cyber", "government", "ai", "cloud", "internet", "infrastructure", "finance", "health"].includes(c))) return 0; | |
| 142 | + if (tier === "A" || (tier === "B" && categories.some((c) => ["finance", "science", "health", "pharma", "ai", "cyber", "government"].includes(c)))) return 1; | |
| 143 | + if (tier === "B" || tier === "C") return 2; | |
| 144 | + return 3; | |
| 145 | +} | |
| 146 | + | |
| 147 | +const TLD_COUNTRY: Record<string, string> = { ca: "CA", "gc.ca": "CA", "gouv.qc.ca": "CA", uk: "GB", "gov.uk": "GB", fr: "FR", "gouv.fr": "FR", de: "DE", it: "IT", es: "ES", jp: "JP", "go.jp": "JP", kr: "KR", "go.kr": "KR", au: "AU", "gov.au": "AU", in: "IN", "gov.in": "IN", br: "BR", "gov.br": "BR", mx: "MX", "gob.mx": "MX", ch: "CH", nl: "NL", se: "SE", no: "NO", fi: "FI", dk: "DK", ie: "IE", be: "BE", at: "AT", pt: "PT", pl: "PL", cn: "CN", hk: "HK", tw: "TW", sg: "SG", il: "IL", ae: "AE", sa: "SA", qa: "QA", za: "ZA", ng: "NG", ke: "KE", ar: "AR", cl: "CL", co: "CO", nz: "NZ", ru: "RU", tr: "TR", id: "ID", th: "TH", vn: "VN", my: "MY", ph: "PH", eg: "EG", ua: "UA", cz: "CZ", gr: "GR", lu: "LU", eu: "EU", "europa.eu": "EU", gov: "US", mil: "US", "us.com": "US" }; | |
| 148 | + | |
| 149 | +/** Country inference for seeds without an explicit `country:` — only from unambiguous TLDs; `.com/.org/.io` stay null. */ | |
| 150 | +export function inferCountry(domain: string, categories: string[]): string | null { | |
| 151 | + const d = domain.toLowerCase(); | |
| 152 | + for (const [suffix, code] of Object.entries(TLD_COUNTRY).sort((a, b) => b[0].length - a[0].length)) if (d === suffix || d.endsWith("." + suffix)) return code; | |
| 153 | + if (/\.int$|^un\.org$|\.who\.int$|\.imf\.org$|\.worldbank\.org$|\.oecd\.org$|\.bis\.org$|\.wto\.org$|\.iso\.org$|\.ietf\.org$|\.w3\.org$|\.icann\.org$/.test(d) || categories.includes("international")) return "INT"; | |
| 154 | + return null; | |
| 155 | +} | |
| 156 | + | |
| 157 | +export function inferLanguage(domain: string): string | null { | |
| 158 | + const d = domain.toLowerCase(); | |
| 159 | + if (/\.qc\.ca$|quebec\.ca$|gouv\.fr$|\.fr$|lapresse\.ca|ledevoir\.com|journaldemontreal\.com|tvanouvelles\.ca|radio-canada\.ca|lemonde\.fr|lefigaro\.fr|24heures\.ca|noovo\.ca|lesoleil\.com|ledroit\.com|lapresse\.ca/.test(d)) return "fr"; | |
| 160 | + if (/\.de$|spiegel\.de|faz\.net|sueddeutsche\.de|handelsblatt\.com|\.at$|swissinfo\.ch/.test(d)) return "de"; | |
| 161 | + if (/\.es$|elpais\.com|elmundo\.es|\.mx$|eluniversal\.com\.mx|\.ar$|\.cl$|\.co$/.test(d)) return "es"; | |
| 162 | + if (/\.it$|repubblica\.it|corriere\.it/.test(d)) return "it"; | |
| 163 | + if (/\.br$|\.pt$|globo\.com|folha\.uol\.com\.br/.test(d)) return "pt"; | |
| 164 | + if (/\.jp$|nhk\.or\.jp|nikkei\.com$/.test(d)) return "ja"; | |
| 165 | + if (/\.kr$/.test(d)) return "ko"; | |
| 166 | + if (/\.cn$|xinhuanet\.com/.test(d)) return "zh"; | |
| 167 | + if (/\.nl$/.test(d)) return "nl"; | |
| 168 | + if (/\.ru$|tass\.(ru|com)/.test(d)) return "ru"; | |
| 169 | + return null; | |
| 170 | +} | |
| 171 | + | |
| 132 | 172 | function feedName(url: string, title?: string): string { |
| 133 | 173 | const p = new URL(url).pathname.toLowerCase(); |
| 134 | 174 | if (/blog/.test(p)) return "blog feed"; |
added
apps/engine/src/retention.ts
+49 −0
@@ -0,0 +1,49 @@ | ||
| 1 | +import { unlink } from "node:fs/promises"; | |
| 2 | +import { join, resolve } from "node:path"; | |
| 3 | +import { db, sql } from "@websensor/db"; | |
| 4 | +import { config, log } from "./config"; | |
| 5 | +import { m } from "./metrics"; | |
| 6 | + | |
| 7 | +/** | |
| 8 | + * Storage lifecycle (spec §58). Evidence that matters is never touched: | |
| 9 | + * - snapshots referenced by an event (old or new side) keep raw body + canonical forever; | |
| 10 | + * - snapshots referenced by a *change* keep raw + canonical for `keepChangedRawDays`; | |
| 11 | + * - "baseline / unchanged" snapshots (no change, no event) lose their RAW body after `keepRawDays` | |
| 12 | + * but keep their canonical representation and every hash, so history/compare still work. | |
| 13 | + * Blobs are content-addressed and shared: a raw blob is deleted only when no other snapshot row | |
| 14 | + * still references its key. Runs incrementally (bounded batches). | |
| 15 | + */ | |
| 16 | +export async function pruneRawSnapshots(opts: { keepRawDays?: number; keepChangedRawDays?: number; batch?: number } = {}): Promise<{ examined: number; pruned: number; freedKeys: number }> { | |
| 17 | + const keepRawDays = opts.keepRawDays ?? config.retention.rawDays; | |
| 18 | + const keepChangedRawDays = opts.keepChangedRawDays ?? config.retention.changedRawDays; | |
| 19 | + const batch = opts.batch ?? 2000; | |
| 20 | + const rows = await db.execute<{ id: string; storage_key: string }>(sql` | |
| 21 | + select s.id, s.storage_key from snapshots s | |
| 22 | + where s.storage_key is not null | |
| 23 | + and s.captured_at < now() - make_interval(days => ${keepRawDays}) | |
| 24 | + and not exists (select 1 from events e where e.new_snapshot_id = s.id or e.old_snapshot_id = s.id) | |
| 25 | + and not exists (select 1 from changes c where (c.new_snapshot_id = s.id or c.old_snapshot_id = s.id) and c.detected_at >= now() - make_interval(days => ${keepChangedRawDays})) | |
| 26 | + order by s.captured_at asc limit ${batch}`); | |
| 27 | + let pruned = 0; | |
| 28 | + let freed = 0; | |
| 29 | + const root = resolve(process.env.BLOB_STORE_DIR ?? "./data/blobs"); | |
| 30 | + for (const r of rows.rows) { | |
| 31 | + // detach the raw body from this snapshot (canonical_storage_key stays) | |
| 32 | + await db.execute(sql`update snapshots set storage_key = null, extra = coalesce(extra, '{}'::jsonb) || jsonb_build_object('raw_pruned_at', now()) where id = ${r.id}`); | |
| 33 | + pruned++; | |
| 34 | + // delete the file only if no other row references the same content-addressed key | |
| 35 | + const still = await db.execute<{ n: string }>(sql`select count(*)::text as n from snapshots where storage_key = ${r.storage_key} or canonical_storage_key = ${r.storage_key} union all select count(*)::text from changes where diff_storage_key = ${r.storage_key}`); | |
| 36 | + if (still.rows.every((x) => Number(x.n) === 0)) { | |
| 37 | + await unlink(join(root, r.storage_key + ".zst")).catch(() => undefined); | |
| 38 | + freed++; | |
| 39 | + m.prunedBlobs.inc(); | |
| 40 | + } | |
| 41 | + } | |
| 42 | + if (pruned) log.info({ examined: rows.rows.length, pruned, freed }, "raw snapshot bodies pruned"); | |
| 43 | + return { examined: rows.rows.length, pruned, freedKeys: freed }; | |
| 44 | +} | |
| 45 | + | |
| 46 | +/** Old raw changes that never became events: keep metadata, drop nothing (they are small). Old runs are pruned in scheduler.ts. */ | |
| 47 | +export async function pruneNotifications(): Promise<void> { | |
| 48 | + await db.execute(sql`delete from notifications where created_at < now() - interval '90 days'`); | |
| 49 | +} | |
modified
apps/engine/src/scheduler.ts
+41 −2
@@ -2,6 +2,7 @@ import { db, sensors, sources, sql, type Sensor, type Source } from "@websensor/ | ||
| 2 | 2 | import { config, log } from "./config"; |
| 3 | 3 | import { m } from "./metrics"; |
| 4 | 4 | import { runSensor } from "./pipeline"; |
| 5 | +import { publishEngineStatus } from "./redis"; | |
| 5 | 6 | |
| 6 | 7 | /** |
| 7 | 8 | * Scheduler: claims due sensors with `FOR UPDATE SKIP LOCKED` (safe with several engine |
@@ -44,15 +45,43 @@ export class Scheduler { | ||
| 44 | 45 | this.sourceCacheAt = Date.now(); |
| 45 | 46 | } |
| 46 | 47 | |
| 48 | + private lastStatusAt = 0; | |
| 49 | + private hostFailures = new Map<string, { count: number; until: number }>(); | |
| 50 | + | |
| 51 | + /** Snapshot for the ops dashboard. */ | |
| 52 | + status(): Record<string, unknown> { | |
| 53 | + const busyHosts = [...this.perHost.entries()].filter(([, n]) => n > 0).map(([h, n]) => ({ host: h, inflight: n })); | |
| 54 | + const tripped = [...this.hostFailures.entries()].filter(([, f]) => f.until > Date.now()).map(([h, f]) => ({ host: h, failures: f.count, until: new Date(f.until).toISOString() })); | |
| 55 | + return { inflight: this.inflight, concurrency: config.fetchConcurrency, perHostConcurrency: config.perHostConcurrency, busyHosts, circuitOpen: tripped, sources: this.sourceCache.size }; | |
| 56 | + } | |
| 57 | + | |
| 58 | + /** Domain circuit breaker (spec §86): after 5 consecutive failures on a host, pause it for 10 minutes. */ | |
| 59 | + noteHostOutcome(host: string, failed: boolean): void { | |
| 60 | + if (!failed) { | |
| 61 | + this.hostFailures.delete(host); | |
| 62 | + return; | |
| 63 | + } | |
| 64 | + const f = this.hostFailures.get(host) ?? { count: 0, until: 0 }; | |
| 65 | + f.count++; | |
| 66 | + if (f.count >= 5) f.until = Date.now() + 10 * 60e3; | |
| 67 | + this.hostFailures.set(host, f); | |
| 68 | + } | |
| 69 | + | |
| 47 | 70 | private async tick(): Promise<void> { |
| 48 | 71 | await this.refreshSources(); |
| 49 | 72 | const due = await db.execute<{ n: string }>(sql`select count(*)::text as n from sensors where enabled and next_check_at <= now()`); |
| 50 | − m.queueDue.set(Number(due.rows[0]?.n ?? 0)); | |
| 73 | + const dueN = Number(due.rows[0]?.n ?? 0); | |
| 74 | + m.queueDue.set(dueN); | |
| 75 | + if (Date.now() - this.lastStatusAt > 10_000) { | |
| 76 | + this.lastStatusAt = Date.now(); | |
| 77 | + void publishEngineStatus({ ...this.status(), due: dueN, version: config.version }); | |
| 78 | + } | |
| 51 | 79 | const slots = config.fetchConcurrency - this.inflight; |
| 52 | 80 | if (slots <= 0) return; |
| 81 | + // Priority-aware claim: critical sensors (priority 0) go first among what is due; within a priority the most overdue first. | |
| 53 | 82 | const claimed = await db.execute<Sensor>(sql` |
| 54 | 83 | update sensors set next_check_at = now() + interval '10 minutes' |
| 55 | − where id in (select id from sensors where enabled and next_check_at <= now() order by next_check_at asc limit ${slots} for update skip locked) | |
| 84 | + where id in (select id from sensors where enabled and next_check_at <= now() order by priority asc, next_check_at asc limit ${slots} for update skip locked) | |
| 56 | 85 | returning *`); |
| 57 | 86 | for (const row of claimed.rows) { |
| 58 | 87 | const sensor = normalizeRow(row as unknown as Record<string, unknown>); |
@@ -62,6 +91,11 @@ export class Scheduler { | ||
| 62 | 91 | await db.update(sensors).set({ nextCheckAt: new Date(Date.now() + 5000) }).where(sql`id = ${sensor.id}`); |
| 63 | 92 | continue; |
| 64 | 93 | } |
| 94 | + const trip = this.hostFailures.get(host); | |
| 95 | + if (trip && trip.until > Date.now()) { | |
| 96 | + await db.update(sensors).set({ nextCheckAt: new Date(trip.until + Math.random() * 30_000) }).where(sql`id = ${sensor.id}`); | |
| 97 | + continue; | |
| 98 | + } | |
| 65 | 99 | const source = this.sourceCache.get(sensor.sourceId); |
| 66 | 100 | if (!source || !source.enabled) { |
| 67 | 101 | await db.update(sensors).set({ nextCheckAt: new Date(Date.now() + 3600e3) }).where(sql`id = ${sensor.id}`); |
@@ -71,8 +105,10 @@ export class Scheduler { | ||
| 71 | 105 | this.perHost.set(host, (this.perHost.get(host) ?? 0) + 1); |
| 72 | 106 | m.inflight.set(this.inflight); |
| 73 | 107 | void runSensor(sensor, source) |
| 108 | + .then((outcome) => this.noteHostOutcome(host, outcome === "error" || outcome === "rate_limited")) | |
| 74 | 109 | .catch(async (e) => { |
| 75 | 110 | log.error({ sensor: sensor.id, err: (e as Error).stack ?? (e as Error).message }, "pipeline crashed"); |
| 111 | + this.noteHostOutcome(host, true); | |
| 76 | 112 | await db.update(sensors).set({ nextCheckAt: new Date(Date.now() + 15 * 60e3), lastError: `pipeline: ${(e as Error).message}`.slice(0, 500), consecutiveErrors: sensor.consecutiveErrors + 1, health: "DEGRADED" }).where(sql`id = ${sensor.id}`).catch(() => undefined); |
| 77 | 113 | }) |
| 78 | 114 | .finally(() => { |
@@ -116,6 +152,9 @@ function normalizeRow(r: Record<string, unknown>): Sensor { | ||
| 116 | 152 | rawChanges: Number(r.raw_changes ?? 0), |
| 117 | 153 | meaningfulChanges: Number(r.meaningful_changes ?? 0), |
| 118 | 154 | avgLatencyMs: (r.avg_latency_ms as number | null) ?? null, |
| 155 | + status: (r.status as string) ?? "ACTIVE", | |
| 156 | + validatedAt: d(r.validated_at), | |
| 157 | + priority: Number(r.priority ?? 2), | |
| 119 | 158 | createdAt: d(r.created_at) ?? new Date(), |
| 120 | 159 | updatedAt: d(r.updated_at) ?? new Date(), |
| 121 | 160 | }; |
modified
apps/engine/src/seeds.ts
+6 −52
@@ -1,8 +1,6 @@ | ||
| 1 | 1 | import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; |
| 2 | 2 | import { join } from "node:path"; |
| 3 | 3 | import YAML from "yaml"; |
| 4 | −import { z } from "zod"; | |
| 5 | −import { SENSOR_TYPES, TIERS } from "@websensor/core"; | |
| 6 | 4 | |
| 7 | 5 | /** |
| 8 | 6 | * Source registry seeds. `config/sources.yaml` is the founding file; every `config/sources.d/*.yaml` |
@@ -13,56 +11,9 @@ import { SENSOR_TYPES, TIERS } from "@websensor/core"; | ||
| 13 | 11 | * |
| 14 | 12 | * This module has no database dependency so that the validator CLI can load seeds anywhere. |
| 15 | 13 | */ |
| 16 | −export const sensorSchema = z.object({ | |
| 17 | − id: z.string().optional(), | |
| 18 | − name: z.string(), | |
| 19 | − url: z.string().url(), | |
| 20 | − type: z.enum(SENSOR_TYPES), | |
| 21 | − connector: z.string().default("http"), | |
| 22 | − tier: z.enum(TIERS).optional(), | |
| 23 | − interval: z.number().int().positive().optional(), | |
| 24 | − weight: z.number().positive().optional(), | |
| 25 | − config: z.record(z.string(), z.unknown()).default({}), | |
| 26 | −}); | |
| 27 | −export type SensorSeed = z.infer<typeof sensorSchema>; | |
| 28 | − | |
| 29 | −export const productSchema = z.object({ id: z.string().optional(), name: z.string(), type: z.string().default("product"), aliases: z.array(z.string()).default([]) }); | |
| 30 | − | |
| 31 | −export const sourceSchema = z.object({ | |
| 32 | − id: z.string().regex(/^[a-z0-9][a-z0-9-]*$/, "id must be kebab-case"), | |
| 33 | − name: z.string(), | |
| 34 | − domain: z.string(), | |
| 35 | − homepage: z.string().url().optional(), | |
| 36 | − description: z.string().optional(), | |
| 37 | − categories: z.array(z.string()).default([]), | |
| 38 | − tier: z.enum(TIERS).default("B"), | |
| 39 | − weight: z.number().positive().default(1), | |
| 40 | − entity_type: z.string().default("organization"), | |
| 41 | − aliases: z.array(z.string()).default([]), | |
| 42 | − products: z.array(productSchema).default([]), | |
| 43 | − discover: z.object({ rss: z.boolean().optional(), sitemap: z.boolean().optional(), status: z.boolean().optional(), pages: z.boolean().optional() }).default({}), | |
| 44 | − fallback: z.object({ firecrawl: z.boolean().optional(), scrapfly: z.boolean().optional() }).default({}), | |
| 45 | − sensors: z.array(sensorSchema).default([]), | |
| 46 | − notes: z.string().optional(), | |
| 47 | − enabled: z.boolean().default(true), | |
| 48 | − /** false = heuristics only (high-volume feeds such as news wires) */ | |
| 49 | − llm: z.boolean().default(true), | |
| 50 | − /** fragment-only: merge into an already declared source instead of redefining it */ | |
| 51 | − extend: z.boolean().default(false), | |
| 52 | −}); | |
| 53 | −export type SourceSeed = z.infer<typeof sourceSchema>; | |
| 54 | − | |
| 55 | −/** Partial schema for `extend: true` entries — only `id` is required. */ | |
| 56 | −const extendSchema = z.object({ | |
| 57 | − id: z.string(), | |
| 58 | − extend: z.literal(true), | |
| 59 | − aliases: z.array(z.string()).default([]), | |
| 60 | − products: z.array(productSchema).default([]), | |
| 61 | − sensors: z.array(sensorSchema).default([]), | |
| 62 | − categories: z.array(z.string()).default([]), | |
| 63 | − fallback: z.object({ firecrawl: z.boolean().optional(), scrapfly: z.boolean().optional() }).optional(), | |
| 64 | − notes: z.string().optional(), | |
| 65 | −}); | |
| 14 | +export { sensorSchema, productSchema, sourceSchema, extendSchema } from "@websensor/core"; | |
| 15 | +export type { SensorSeed, SourceSeed } from "@websensor/core"; | |
| 16 | +import { extendSchema, sourceSchema, type SourceSeed } from "@websensor/core"; | |
| 66 | 17 | |
| 67 | 18 | export interface SeedIssue { |
| 68 | 19 | file: string; |
@@ -124,6 +75,9 @@ export function loadSeedsDetailed(file: string, dir: string): LoadedSeeds { | ||
| 124 | 75 | base.sensors = [...base.sensors, ...parsed.data.sensors.filter((n) => !base.sensors.some((o) => o.url === n.url))]; |
| 125 | 76 | if (parsed.data.fallback) base.fallback = { ...base.fallback, ...parsed.data.fallback }; |
| 126 | 77 | if (parsed.data.notes) base.notes = [base.notes, parsed.data.notes].filter(Boolean).join(" "); |
| 78 | + if (parsed.data.country && !base.country) base.country = parsed.data.country; | |
| 79 | + if (parsed.data.language && !base.language) base.language = parsed.data.language; | |
| 80 | + if (parsed.data.first_party !== undefined && base.first_party === undefined) base.first_party = parsed.data.first_party; | |
| 127 | 81 | continue; |
| 128 | 82 | } |
| 129 | 83 | const parsed = sourceSchema.safeParse(s); |
modified
apps/web/src/app/alerts/alerts.tsx
+362 −71
@@ -1,22 +1,28 @@ | ||
| 1 | 1 | "use client"; |
| 2 | 2 | |
| 3 | −import { Bell, BellRing, Trash2 } from "lucide-react"; | |
| 3 | +import { Bell, BellOff, BellRing, Check, Trash2, Webhook } from "lucide-react"; | |
| 4 | 4 | import Link from "next/link"; |
| 5 | −import { useCallback, useEffect, useState } from "react"; | |
| 6 | −import { EVENT_TYPES } from "@websensor/core/client"; | |
| 5 | +import { useSearchParams } from "next/navigation"; | |
| 6 | +import { useEffect, useMemo, useState } from "react"; | |
| 7 | +import { COUNTRIES, EVENT_TYPES, eventGroupOf } from "@websensor/core/client"; | |
| 7 | 8 | import { LiveDot } from "@/components/live-feed"; |
| 8 | 9 | import { useMounted } from "@/components/theme"; |
| 9 | −import { Chip, Empty, Panel, Score } from "@/components/ui"; | |
| 10 | −import type { Alert, AlertRule, LiveEvent } from "@/lib/api"; | |
| 11 | −import { relTime, typeLabel } from "@/lib/format"; | |
| 10 | +import { Chip, Empty, Flag, Panel, Score, SkeletonRows } from "@/components/ui"; | |
| 11 | +import type { Alert, AlertRule, LiveEvent, Notification } from "@/lib/api"; | |
| 12 | +import { CHANNEL_KEYS, GROUP_LABELS, relTime, typeLabel, utcDateTime } from "@/lib/format"; | |
| 12 | 13 | import { ownerFetch } from "@/lib/owner"; |
| 13 | 14 | import { useLive } from "@/lib/use-live"; |
| 14 | 15 | |
| 15 | 16 | function matches(rule: AlertRule, e: LiveEvent): boolean { |
| 16 | 17 | if (rule.importance_min !== undefined && e.importance < rule.importance_min) return false; |
| 18 | + if (rule.signal_min !== undefined && (e.signal ?? e.importance) < rule.signal_min) return false; | |
| 17 | 19 | if (rule.silent_only && !e.silent) return false; |
| 20 | + if (rule.first_party_only && e.firstParty === false) return false; | |
| 21 | + if (rule.confirmed_only && e.evidence !== "CONFIRMED" && e.clusterState !== "confirmed") return false; | |
| 18 | 22 | if (rule.event_types?.length && !rule.event_types.includes(e.type)) return false; |
| 23 | + if (rule.groups?.length && !rule.groups.includes(e.group ?? eventGroupOf(e.type))) return false; | |
| 19 | 24 | if (rule.categories?.length && !rule.categories.some((c) => e.categories.includes(c))) return false; |
| 25 | + if (rule.countries?.length && !(e.country && rule.countries.includes(e.country.toUpperCase()))) return false; | |
| 20 | 26 | if (rule.entities?.length && !e.entities.some((x) => rule.entities!.includes(x.id))) return false; |
| 21 | 27 | if (rule.sources?.length && !rule.sources.includes(e.source?.id)) return false; |
| 22 | 28 | if (rule.keywords?.length) { |
@@ -26,28 +32,112 @@ function matches(rule: AlertRule, e: LiveEvent): boolean { | ||
| 26 | 32 | return true; |
| 27 | 33 | } |
| 28 | 34 | |
| 35 | +interface Form { | |
| 36 | + name: string; | |
| 37 | + importance_min: number; | |
| 38 | + signal_min: number; | |
| 39 | + silent_only: boolean; | |
| 40 | + first_party_only: boolean; | |
| 41 | + confirmed_only: boolean; | |
| 42 | + event_types: string[]; | |
| 43 | + groups: string[]; | |
| 44 | + categories: string[]; | |
| 45 | + countries: string[]; | |
| 46 | + entities: string; | |
| 47 | + sources: string; | |
| 48 | + keywords: string; | |
| 49 | + channel: "web" | "webhook"; | |
| 50 | + webhook_url: string; | |
| 51 | + webhook_secret: string; | |
| 52 | +} | |
| 53 | + | |
| 54 | +const EMPTY: Form = { name: "", importance_min: 0, signal_min: 60, silent_only: false, first_party_only: false, confirmed_only: false, event_types: [], groups: [], categories: [], countries: [], entities: "", sources: "", keywords: "", channel: "web", webhook_url: "", webhook_secret: "" }; | |
| 55 | + | |
| 56 | +const PRESETS: { label: string; form: Partial<Form> }[] = [ | |
| 57 | + { label: "OpenAI pricing", form: { name: "OpenAI pricing", entities: "org_openai", event_types: ["pricing_change"], signal_min: 0 } }, | |
| 58 | + { label: "Anthropic model release", form: { name: "Anthropic model release", entities: "org_anthropic", event_types: ["model_release"], signal_min: 0 } }, | |
| 59 | + { label: "Cloudflare outage", form: { name: "Cloudflare outage", entities: "org_cloudflare", groups: ["reliability"], signal_min: 0 } }, | |
| 60 | + { label: "Critical CISA/KEV", form: { name: "Critical CISA / KEV", sources: "cisa", groups: ["security"], signal_min: 70, first_party_only: true } }, | |
| 61 | + { label: "Tesla pricing", form: { name: "Tesla pricing", entities: "org_tesla", event_types: ["pricing_change"], signal_min: 0 } }, | |
| 62 | + { label: "NVIDIA ≥ 80", form: { name: "NVIDIA ≥ 80", entities: "org_nvidia", signal_min: 80 } }, | |
| 63 | +]; | |
| 64 | + | |
| 65 | +const COUNTRY_CODES = Object.keys(COUNTRIES); | |
| 66 | + | |
| 67 | +function toggle<T>(list: T[], v: T): T[] { | |
| 68 | + return list.includes(v) ? list.filter((x) => x !== v) : [...list, v]; | |
| 69 | +} | |
| 70 | + | |
| 71 | +function ChipToggle({ on, onClick, children, tone = "signal", title }: { on: boolean; onClick: () => void; children: React.ReactNode; tone?: "signal" | "silent" | "info" | "ok"; title?: string }) { | |
| 72 | + const onCls = tone === "silent" ? "border-silent/50 bg-silent-soft text-silent" : tone === "info" ? "border-info/50 bg-info/10 text-info" : tone === "ok" ? "border-ok/50 bg-ok/10 text-ok" : "border-signal/50 bg-signal-soft text-signal"; | |
| 73 | + return ( | |
| 74 | + <button type="button" aria-pressed={on} title={title} onClick={onClick} className={`inline-flex items-center gap-1 rounded-sm border px-1.5 py-px text-[10.5px] leading-4 ${on ? onCls : "border-line text-fg-muted hover:text-fg"}`}> | |
| 75 | + {children} | |
| 76 | + </button> | |
| 77 | + ); | |
| 78 | +} | |
| 79 | + | |
| 80 | +function ruleChips(rule: AlertRule) { | |
| 81 | + const out: React.ReactNode[] = []; | |
| 82 | + if (rule.signal_min) out.push(<Chip key="sig" tone="signal">signal ≥ {rule.signal_min}</Chip>); | |
| 83 | + if (rule.importance_min) out.push(<Chip key="imp">importance ≥ {rule.importance_min}</Chip>); | |
| 84 | + if (rule.silent_only) out.push(<Chip key="silent" tone="silent">silent only</Chip>); | |
| 85 | + if (rule.first_party_only) out.push(<Chip key="fp" tone="signal">first-party</Chip>); | |
| 86 | + if (rule.confirmed_only) out.push(<Chip key="cf" tone="ok">confirmed</Chip>); | |
| 87 | + rule.groups?.forEach((g) => out.push(<Chip key={`g-${g}`} tone="info">{GROUP_LABELS[g] ?? g}</Chip>)); | |
| 88 | + rule.event_types?.forEach((t) => out.push(<Chip key={`t-${t}`}>{typeLabel(t)}</Chip>)); | |
| 89 | + rule.categories?.forEach((c) => out.push(<Chip key={`c-${c}`}>{c}</Chip>)); | |
| 90 | + rule.countries?.forEach((c) => out.push(<Chip key={`co-${c}`}><Flag code={c} /> {c}</Chip>)); | |
| 91 | + rule.entities?.forEach((t) => out.push(<Chip key={`e-${t}`} tone="signal" href={`/entity/${t}`}>{t.replace(/^(org|prd|ent)_/, "")}</Chip>)); | |
| 92 | + rule.sources?.forEach((t) => out.push(<Chip key={`s-${t}`} tone="info" href={`/source/${t}`}>{t}</Chip>)); | |
| 93 | + rule.keywords?.forEach((t) => out.push(<Chip key={`k-${t}`}>“{t}”</Chip>)); | |
| 94 | + if (!out.length) out.push(<Chip key="all">every event</Chip>); | |
| 95 | + return out; | |
| 96 | +} | |
| 97 | + | |
| 29 | 98 | export function Alerts() { |
| 99 | + const sp = useSearchParams(); | |
| 100 | + const prefillEntity = sp.get("entity") ?? ""; | |
| 30 | 101 | const [alerts, setAlerts] = useState<Alert[] | null>(null); |
| 102 | + const [notifs, setNotifs] = useState<{ items: Notification[]; unread: number } | null>(null); | |
| 31 | 103 | const [fired, setFired] = useState<{ alert: Alert; event: LiveEvent; at: number }[]>([]); |
| 32 | 104 | const mounted = useMounted(); |
| 33 | 105 | const [permOverride, setPerm] = useState<NotificationPermission | null>(null); |
| 34 | 106 | const perm: NotificationPermission | "unsupported" = !mounted ? "default" : permOverride ?? (typeof Notification === "undefined" ? "unsupported" : Notification.permission); |
| 35 | − const [form, setForm] = useState<{ name: string; importance_min: number; event_types: string[]; entities: string; sources: string; keywords: string; silent_only: boolean }>({ name: "", importance_min: 70, event_types: [], entities: "", sources: "", keywords: "", silent_only: false }); | |
| 107 | + const [form, setForm] = useState<Form>({ ...EMPTY, entities: prefillEntity, name: prefillEntity ? `${prefillEntity.replace(/^(org|prd|ent)_/, "")} alerts` : "" }); | |
| 36 | 108 | const [err, setErr] = useState<string | null>(null); |
| 109 | + const [busy, setBusy] = useState(false); | |
| 110 | + const [created, setCreated] = useState<string | null>(null); | |
| 111 | + const [tick, setTick] = useState(0); | |
| 112 | + const reload = (): void => setTick((t) => t + 1); | |
| 37 | 113 | |
| 38 | − const load = useCallback( | |
| 39 | − () => | |
| 40 | − ownerFetch<{ items: Alert[] }>("/api/v1/alerts") | |
| 41 | − .then((r) => setAlerts(r.items)) | |
| 42 | − .catch((e: Error) => { | |
| 43 | − setErr(e.message); | |
| 44 | − setAlerts([]); | |
| 45 | − }), | |
| 46 | − [], | |
| 47 | − ); | |
| 48 | 114 | useEffect(() => { |
| 49 | − void load(); | |
| 50 | − }, [load]); | |
| 115 | + let cancelled = false; | |
| 116 | + ownerFetch<{ items: Alert[] }>("/api/v1/alerts") | |
| 117 | + .then((r) => { | |
| 118 | + if (!cancelled) setAlerts(r.items); | |
| 119 | + }) | |
| 120 | + .catch((e: Error) => { | |
| 121 | + if (cancelled) return; | |
| 122 | + setErr(e.message); | |
| 123 | + setAlerts([]); | |
| 124 | + }); | |
| 125 | + const pull = (): void => { | |
| 126 | + ownerFetch<{ items: Notification[]; unread: number }>("/api/v1/notifications?limit=50") | |
| 127 | + .then((r) => { | |
| 128 | + if (!cancelled) setNotifs(r); | |
| 129 | + }) | |
| 130 | + .catch(() => { | |
| 131 | + if (!cancelled) setNotifs({ items: [], unread: 0 }); | |
| 132 | + }); | |
| 133 | + }; | |
| 134 | + pull(); | |
| 135 | + const t = setInterval(pull, 30_000); | |
| 136 | + return () => { | |
| 137 | + cancelled = true; | |
| 138 | + clearInterval(t); | |
| 139 | + }; | |
| 140 | + }, [tick]); | |
| 51 | 141 | |
| 52 | 142 | const status = useLive(["events:global"], (e) => { |
| 53 | 143 | for (const a of alerts ?? []) { |
@@ -55,7 +145,7 @@ export function Alerts() { | ||
| 55 | 145 | setFired((prev) => [{ alert: a, event: e, at: Date.now() }, ...prev].slice(0, 50)); |
| 56 | 146 | if (typeof Notification !== "undefined" && Notification.permission === "granted") { |
| 57 | 147 | try { |
| 58 | − const n = new Notification(`${e.source?.name ?? "WebSensor"} · ${Math.round(e.importance)}`, { body: e.title, tag: e.id, icon: "/icon.svg" }); | |
| 148 | + const n = new Notification(`${e.source?.name ?? "WebSensor"} · ${Math.round(e.signal ?? e.importance)}`, { body: e.title, tag: e.id, icon: "/icon.svg" }); | |
| 59 | 149 | n.onclick = () => window.open(`/event/${e.slug}`, "_blank"); |
| 60 | 150 | } catch { |
| 61 | 151 | // notifications unavailable |
@@ -64,69 +154,232 @@ export function Alerts() { | ||
| 64 | 154 | } |
| 65 | 155 | }); |
| 66 | 156 | |
| 157 | + const split = (s: string): string[] | undefined => { | |
| 158 | + const arr = s.split(/[,\n]/).map((x) => x.trim()).filter(Boolean); | |
| 159 | + return arr.length ? arr : undefined; | |
| 160 | + }; | |
| 161 | + | |
| 67 | 162 | const create = async (): Promise<void> => { |
| 68 | − const split = (s: string): string[] | undefined => s.split(",").map((x) => x.trim()).filter(Boolean).length ? s.split(",").map((x) => x.trim()).filter(Boolean) : undefined; | |
| 69 | − const rule: AlertRule = { importance_min: form.importance_min, event_types: form.event_types.length ? form.event_types : undefined, entities: split(form.entities), sources: split(form.sources), keywords: split(form.keywords), silent_only: form.silent_only || undefined }; | |
| 163 | + setErr(null); | |
| 164 | + setCreated(null); | |
| 165 | + const rule: AlertRule = { | |
| 166 | + importance_min: form.importance_min > 0 ? form.importance_min : undefined, | |
| 167 | + signal_min: form.signal_min > 0 ? form.signal_min : undefined, | |
| 168 | + event_types: form.event_types.length ? form.event_types : undefined, | |
| 169 | + groups: form.groups.length ? form.groups : undefined, | |
| 170 | + categories: form.categories.length ? form.categories : undefined, | |
| 171 | + countries: form.countries.length ? form.countries : undefined, | |
| 172 | + entities: split(form.entities), | |
| 173 | + sources: split(form.sources), | |
| 174 | + keywords: split(form.keywords), | |
| 175 | + silent_only: form.silent_only || undefined, | |
| 176 | + first_party_only: form.first_party_only || undefined, | |
| 177 | + confirmed_only: form.confirmed_only || undefined, | |
| 178 | + }; | |
| 179 | + if (form.channel === "webhook" && !/^https:\/\//i.test(form.webhook_url.trim())) { | |
| 180 | + setErr("Webhook URL must start with https:// and be publicly reachable."); | |
| 181 | + return; | |
| 182 | + } | |
| 183 | + setBusy(true); | |
| 184 | + try { | |
| 185 | + const body = { name: form.name.trim() || "Alert", rule, channel: form.channel, channel_config: form.channel === "webhook" ? { url: form.webhook_url.trim(), ...(form.webhook_secret.trim() ? { secret: form.webhook_secret.trim() } : {}) } : {} }; | |
| 186 | + const r = await ownerFetch<{ id: string; name: string }>("/api/v1/alerts", { method: "POST", body: JSON.stringify(body) }); | |
| 187 | + setForm(EMPTY); | |
| 188 | + setCreated(r.name); | |
| 189 | + reload(); | |
| 190 | + } catch (e) { | |
| 191 | + setErr((e as Error).message); | |
| 192 | + } finally { | |
| 193 | + setBusy(false); | |
| 194 | + } | |
| 195 | + }; | |
| 196 | + | |
| 197 | + const setEnabled = async (a: Alert, enabled: boolean): Promise<void> => { | |
| 198 | + setAlerts((prev) => prev?.map((x) => (x.id === a.id ? { ...x, enabled } : x)) ?? prev); | |
| 70 | 199 | try { |
| 71 | − await ownerFetch("/api/v1/alerts", { method: "POST", body: JSON.stringify({ name: form.name.trim() || "Alert", rule, channel: "web" }) }); | |
| 72 | − setForm({ name: "", importance_min: 70, event_types: [], entities: "", sources: "", keywords: "", silent_only: false }); | |
| 73 | − await load(); | |
| 200 | + await ownerFetch(`/api/v1/alerts/${a.id}`, { method: "PATCH", body: JSON.stringify({ enabled }) }); | |
| 74 | 201 | } catch (e) { |
| 75 | 202 | setErr((e as Error).message); |
| 203 | + reload(); | |
| 76 | 204 | } |
| 77 | 205 | }; |
| 206 | + const remove = async (a: Alert): Promise<void> => { | |
| 207 | + setAlerts((prev) => prev?.filter((x) => x.id !== a.id) ?? prev); | |
| 208 | + try { | |
| 209 | + await ownerFetch(`/api/v1/alerts/${a.id}`, { method: "DELETE" }); | |
| 210 | + } catch (e) { | |
| 211 | + setErr((e as Error).message); | |
| 212 | + reload(); | |
| 213 | + } | |
| 214 | + }; | |
| 215 | + const markAllRead = async (): Promise<void> => { | |
| 216 | + try { | |
| 217 | + await ownerFetch("/api/v1/notifications/read", { method: "POST", body: JSON.stringify({}) }); | |
| 218 | + reload(); | |
| 219 | + } catch (e) { | |
| 220 | + setErr((e as Error).message); | |
| 221 | + } | |
| 222 | + }; | |
| 223 | + | |
| 224 | + const applyPreset = (p: Partial<Form>): void => setForm({ ...EMPTY, ...p }); | |
| 225 | + const activeFilters = useMemo(() => [form.signal_min > 0, form.importance_min > 0, form.silent_only, form.first_party_only, form.confirmed_only, form.event_types.length > 0, form.groups.length > 0, form.categories.length > 0, form.countries.length > 0, Boolean(form.entities.trim()), Boolean(form.sources.trim()), Boolean(form.keywords.trim())].filter(Boolean).length, [form]); | |
| 78 | 226 | |
| 79 | 227 | return ( |
| 80 | − <div className="grid gap-4 lg:grid-cols-[380px_1fr]"> | |
| 81 | − <aside className="flex flex-col gap-4"> | |
| 82 | − <Panel title="New rule"> | |
| 228 | + <div className="grid gap-4 lg:grid-cols-[400px_minmax(0,1fr)]"> | |
| 229 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 230 | + <Panel title={<span>New rule <span className="normal-case tracking-normal text-fg-subtle">· {activeFilters} condition{activeFilters === 1 ? "" : "s"}</span></span>}> | |
| 83 | 231 | <form |
| 84 | − className="flex flex-col gap-2 text-[12.5px]" | |
| 232 | + className="flex flex-col gap-3 text-[12.5px]" | |
| 85 | 233 | onSubmit={(e) => { |
| 86 | 234 | e.preventDefault(); |
| 87 | 235 | void create(); |
| 88 | 236 | }} |
| 89 | 237 | > |
| 90 | − <input value={form.name} onChange={(e) => setForm({ ...form, name: e.target.value })} placeholder="Rule name" className="h-8 rounded-md border border-line bg-panel px-2" /> | |
| 91 | − <label className="flex items-center justify-between gap-2"> | |
| 92 | − <span className="text-fg-muted">Importance ≥ <span className="font-mono">{form.importance_min}</span></span> | |
| 93 | − <input type="range" min={0} max={100} value={form.importance_min} onChange={(e) => setForm({ ...form, importance_min: Number(e.target.value) })} className="w-40 accent-[var(--signal)]" /> | |
| 94 | − </label> | |
| 95 | − <label className="flex items-center gap-2"> | |
| 96 | − <input type="checkbox" checked={form.silent_only} onChange={(e) => setForm({ ...form, silent_only: e.target.checked })} className="accent-[var(--silent)]" /> | |
| 97 | − <span className="text-silent">Silent changes only</span> | |
| 238 | + <div> | |
| 239 | + <div className="label mb-1">Presets</div> | |
| 240 | + <div className="flex flex-wrap gap-1"> | |
| 241 | + {PRESETS.map((p) => ( | |
| 242 | + <button key={p.label} type="button" onClick={() => applyPreset(p.form)} className="rounded-sm border border-line px-1.5 py-px text-[10.5px] leading-4 text-fg-muted hover:border-line-strong hover:text-fg"> | |
| 243 | + {p.label} | |
| 244 | + </button> | |
| 245 | + ))} | |
| 246 | + </div> | |
| 247 | + </div> | |
| 248 | + | |
| 249 | + <label className="flex flex-col gap-1"> | |
| 250 | + <span className="label">Name</span> | |
| 251 | + <input value={form.name} onChange={(e) => setForm({ ...form, name: e.target.value })} placeholder="Rule name" maxLength={80} className="h-8 rounded-md border border-line bg-panel px-2 text-[12.5px]" /> | |
| 98 | 252 | </label> |
| 253 | + | |
| 254 | + <div className="grid grid-cols-2 gap-3"> | |
| 255 | + <label className="flex flex-col gap-1"> | |
| 256 | + <span className="flex items-center justify-between"><span className="label">Signal ≥</span><span className="font-mono text-[12px] tabular">{form.signal_min || "any"}</span></span> | |
| 257 | + <input type="range" min={0} max={100} step={5} value={form.signal_min} onChange={(e) => setForm({ ...form, signal_min: Number(e.target.value) })} className="w-full accent-[var(--signal)]" aria-label="Minimum signal score" /> | |
| 258 | + </label> | |
| 259 | + <label className="flex flex-col gap-1"> | |
| 260 | + <span className="flex items-center justify-between"><span className="label">Importance ≥</span><span className="font-mono text-[12px] tabular">{form.importance_min || "any"}</span></span> | |
| 261 | + <input type="range" min={0} max={100} step={5} value={form.importance_min} onChange={(e) => setForm({ ...form, importance_min: Number(e.target.value) })} className="w-full accent-[var(--signal)]" aria-label="Minimum importance" /> | |
| 262 | + </label> | |
| 263 | + </div> | |
| 264 | + | |
| 265 | + <div className="flex flex-wrap gap-x-4 gap-y-1"> | |
| 266 | + <label className="flex items-center gap-1.5"> | |
| 267 | + <input type="checkbox" checked={form.silent_only} onChange={(e) => setForm({ ...form, silent_only: e.target.checked })} className="accent-[var(--silent)]" /> | |
| 268 | + <span className="text-silent">Silent only</span> | |
| 269 | + </label> | |
| 270 | + <label className="flex items-center gap-1.5"> | |
| 271 | + <input type="checkbox" checked={form.first_party_only} onChange={(e) => setForm({ ...form, first_party_only: e.target.checked })} className="accent-[var(--signal)]" /> | |
| 272 | + <span>First-party only</span> | |
| 273 | + </label> | |
| 274 | + <label className="flex items-center gap-1.5"> | |
| 275 | + <input type="checkbox" checked={form.confirmed_only} onChange={(e) => setForm({ ...form, confirmed_only: e.target.checked })} className="accent-[var(--ok)]" /> | |
| 276 | + <span>Confirmed only</span> | |
| 277 | + </label> | |
| 278 | + </div> | |
| 279 | + | |
| 280 | + <div> | |
| 281 | + <div className="label mb-1">Groups</div> | |
| 282 | + <div className="flex flex-wrap gap-1"> | |
| 283 | + {Object.entries(GROUP_LABELS).map(([k, v]) => ( | |
| 284 | + <ChipToggle key={k} on={form.groups.includes(k)} tone="info" onClick={() => setForm({ ...form, groups: toggle(form.groups, k) })}>{v}</ChipToggle> | |
| 285 | + ))} | |
| 286 | + </div> | |
| 287 | + </div> | |
| 288 | + | |
| 289 | + <div> | |
| 290 | + <div className="mb-1 flex items-center justify-between"> | |
| 291 | + <span className="label">Event types</span> | |
| 292 | + {form.event_types.length > 0 && <button type="button" onClick={() => setForm({ ...form, event_types: [] })} className="text-[10.5px] text-fg-subtle hover:text-fg">clear {form.event_types.length}</button>} | |
| 293 | + </div> | |
| 294 | + <div className="flex max-h-28 flex-wrap gap-1 overflow-auto rounded-md border border-line p-1.5"> | |
| 295 | + {Object.entries(EVENT_TYPES).map(([k, v]) => ( | |
| 296 | + <ChipToggle key={k} on={form.event_types.includes(k)} onClick={() => setForm({ ...form, event_types: toggle(form.event_types, k) })} title={`${k} · severity ${v.severity}`}>{v.label}</ChipToggle> | |
| 297 | + ))} | |
| 298 | + </div> | |
| 299 | + </div> | |
| 300 | + | |
| 99 | 301 | <div> |
| 100 | − <div className="label mb-1">Event types</div> | |
| 101 | − <div className="flex max-h-32 flex-wrap gap-1 overflow-auto rounded-md border border-line p-1.5"> | |
| 102 | − {Object.entries(EVENT_TYPES).map(([k, v]) => { | |
| 103 | − const on = form.event_types.includes(k); | |
| 104 | − return ( | |
| 105 | − <button key={k} type="button" onClick={() => setForm({ ...form, event_types: on ? form.event_types.filter((x) => x !== k) : [...form.event_types, k] })} className={`rounded-sm border px-1.5 py-px text-[10.5px] ${on ? "border-signal/50 bg-signal-soft text-signal" : "border-line text-fg-muted hover:text-fg"}`}> | |
| 106 | − {v.label} | |
| 107 | − </button> | |
| 108 | − ); | |
| 109 | − })} | |
| 302 | + <div className="label mb-1">Categories</div> | |
| 303 | + <div className="flex flex-wrap gap-1"> | |
| 304 | + {CHANNEL_KEYS.map((c) => ( | |
| 305 | + <ChipToggle key={c} on={form.categories.includes(c)} tone="ok" onClick={() => setForm({ ...form, categories: toggle(form.categories, c) })}>{c}</ChipToggle> | |
| 306 | + ))} | |
| 110 | 307 | </div> |
| 111 | 308 | </div> |
| 112 | − <input value={form.entities} onChange={(e) => setForm({ ...form, entities: e.target.value })} placeholder="Entity ids, comma-separated (org_openai, prd_openai_api)" className="h-8 rounded-md border border-line bg-panel px-2 font-mono text-[12px]" /> | |
| 113 | − <input value={form.sources} onChange={(e) => setForm({ ...form, sources: e.target.value })} placeholder="Source ids (openai, cisa)" className="h-8 rounded-md border border-line bg-panel px-2 font-mono text-[12px]" /> | |
| 114 | − <input value={form.keywords} onChange={(e) => setForm({ ...form, keywords: e.target.value })} placeholder="Keywords (pricing, CVE, outage)" className="h-8 rounded-md border border-line bg-panel px-2" /> | |
| 115 | − <button type="submit" className="inline-flex h-8 items-center justify-center gap-1 rounded-md border border-line bg-panel-2 px-3 hover:border-line-strong"><Bell className="size-3.5" /> Create rule</button> | |
| 309 | + | |
| 310 | + <div> | |
| 311 | + <div className="mb-1 flex items-center justify-between"> | |
| 312 | + <span className="label">Countries</span> | |
| 313 | + {form.countries.length > 0 && <button type="button" onClick={() => setForm({ ...form, countries: [] })} className="text-[10.5px] text-fg-subtle hover:text-fg">clear {form.countries.length}</button>} | |
| 314 | + </div> | |
| 315 | + <div className="flex max-h-20 flex-wrap gap-1 overflow-auto rounded-md border border-line p-1.5"> | |
| 316 | + {COUNTRY_CODES.map((c) => ( | |
| 317 | + <ChipToggle key={c} on={form.countries.includes(c)} onClick={() => setForm({ ...form, countries: toggle(form.countries, c) })} title={COUNTRIES[c]?.name}> | |
| 318 | + <span className="font-mono">{c}</span> | |
| 319 | + </ChipToggle> | |
| 320 | + ))} | |
| 321 | + </div> | |
| 322 | + </div> | |
| 323 | + | |
| 324 | + <label className="flex flex-col gap-1"> | |
| 325 | + <span className="label">Entities <span className="normal-case tracking-normal">· ids, comma-separated</span></span> | |
| 326 | + <input value={form.entities} onChange={(e) => setForm({ ...form, entities: e.target.value })} placeholder="org_openai, prd_openai_openai-api" className="h-8 rounded-md border border-line bg-panel px-2 font-mono text-[12px]" /> | |
| 327 | + </label> | |
| 328 | + <label className="flex flex-col gap-1"> | |
| 329 | + <span className="label">Sources <span className="normal-case tracking-normal">· ids</span></span> | |
| 330 | + <input value={form.sources} onChange={(e) => setForm({ ...form, sources: e.target.value })} placeholder="openai, cisa" className="h-8 rounded-md border border-line bg-panel px-2 font-mono text-[12px]" /> | |
| 331 | + </label> | |
| 332 | + <label className="flex flex-col gap-1"> | |
| 333 | + <span className="label">Keywords <span className="normal-case tracking-normal">· in title or summary</span></span> | |
| 334 | + <input value={form.keywords} onChange={(e) => setForm({ ...form, keywords: e.target.value })} placeholder="pricing, CVE, outage" className="h-8 rounded-md border border-line bg-panel px-2 text-[12.5px]" /> | |
| 335 | + </label> | |
| 336 | + | |
| 337 | + <fieldset className="rounded-md border border-line p-2"> | |
| 338 | + <legend className="label px-1">Delivery channel</legend> | |
| 339 | + <div className="flex flex-wrap gap-x-4 gap-y-1"> | |
| 340 | + <label className="flex items-center gap-1.5"> | |
| 341 | + <input type="radio" name="channel" checked={form.channel === "web"} onChange={() => setForm({ ...form, channel: "web" })} className="accent-[var(--signal)]" /> | |
| 342 | + <Bell className="size-3.5 text-fg-muted" /> Web (this browser) | |
| 343 | + </label> | |
| 344 | + <label className="flex items-center gap-1.5"> | |
| 345 | + <input type="radio" name="channel" checked={form.channel === "webhook"} onChange={() => setForm({ ...form, channel: "webhook" })} className="accent-[var(--signal)]" /> | |
| 346 | + <Webhook className="size-3.5 text-fg-muted" /> Webhook | |
| 347 | + </label> | |
| 348 | + </div> | |
| 349 | + {form.channel === "webhook" && ( | |
| 350 | + <div className="mt-2 flex flex-col gap-1.5"> | |
| 351 | + <input type="url" value={form.webhook_url} onChange={(e) => setForm({ ...form, webhook_url: e.target.value })} placeholder="https://example.com/hooks/websensor" required className="h-8 rounded-md border border-line bg-panel px-2 font-mono text-[12px]" aria-label="Webhook URL" /> | |
| 352 | + <input value={form.webhook_secret} onChange={(e) => setForm({ ...form, webhook_secret: e.target.value })} placeholder="Shared secret (optional)" className="h-8 rounded-md border border-line bg-panel px-2 font-mono text-[12px]" aria-label="Webhook secret" autoComplete="off" /> | |
| 353 | + <p className="text-[11px] leading-relaxed text-fg-subtle"> | |
| 354 | + POST JSON per matching event. With a secret, each request carries <code className="text-fg-muted">X-WebSensor-Signature: sha256=…</code> — an HMAC-SHA256 of the raw body you can verify. HTTPS only; private, loopback and cloud-metadata addresses are rejected. | |
| 355 | + </p> | |
| 356 | + </div> | |
| 357 | + )} | |
| 358 | + </fieldset> | |
| 359 | + | |
| 360 | + <button type="submit" disabled={busy} className="inline-flex h-8 items-center justify-center gap-1 rounded-md border border-line bg-panel-2 px-3 text-[12.5px] hover:border-line-strong disabled:opacity-60"> | |
| 361 | + <Bell className="size-3.5" /> {busy ? "Creating…" : "Create rule"} | |
| 362 | + </button> | |
| 116 | 363 | {err && <p className="text-[11px] text-danger">{err}</p>} |
| 364 | + {created && <p className="flex items-center gap-1 text-[11px] text-signal"><Check className="size-3" /> Rule “{created}” created.</p>} | |
| 117 | 365 | </form> |
| 118 | 366 | </Panel> |
| 119 | − <Panel title="Delivery"> | |
| 367 | + | |
| 368 | + <Panel title="Delivery status"> | |
| 120 | 369 | <div className="flex flex-col gap-2 text-[12.5px]"> |
| 121 | 370 | <div className="flex items-center justify-between"> |
| 122 | 371 | <span>Web (this browser)</span> |
| 123 | 372 | <Chip tone="ok">active</Chip> |
| 124 | 373 | </div> |
| 374 | + <div className="flex items-center justify-between"> | |
| 375 | + <span>Webhook (HTTPS, signed)</span> | |
| 376 | + <Chip tone="ok">active</Chip> | |
| 377 | + </div> | |
| 125 | 378 | <div className="flex items-center justify-between"> |
| 126 | 379 | <span>Browser notifications</span> |
| 127 | − {perm === "granted" ? <Chip tone="ok">granted</Chip> : perm === "unsupported" ? <Chip>unsupported</Chip> : perm === "denied" ? <Chip tone="danger">denied</Chip> : <button type="button" onClick={() => Notification.requestPermission().then(setPerm)} className="rounded-md border border-line bg-panel-2 px-2 py-0.5 text-[12px]">Enable</button>} | |
| 380 | + {perm === "granted" ? <Chip tone="ok">granted</Chip> : perm === "unsupported" ? <Chip>unsupported</Chip> : perm === "denied" ? <Chip tone="danger">denied</Chip> : <button type="button" onClick={() => Notification.requestPermission().then(setPerm)} className="h-7 rounded-md border border-line bg-panel-2 px-2 text-[12px] hover:border-line-strong">Enable</button>} | |
| 128 | 381 | </div> |
| 129 | − {["Email", "Push", "Webhook", "Slack", "Discord"].map((c) => ( | |
| 382 | + {["Email", "Push", "Slack", "Discord", "Telegram"].map((c) => ( | |
| 130 | 383 | <div key={c} className="flex items-center justify-between text-fg-muted"> |
| 131 | 384 | <span>{c}</span> |
| 132 | 385 | <Chip>planned</Chip> |
@@ -135,40 +388,78 @@ export function Alerts() { | ||
| 135 | 388 | </div> |
| 136 | 389 | </Panel> |
| 137 | 390 | </aside> |
| 138 | − <div className="flex flex-col gap-4"> | |
| 139 | − <Panel title={`Rules · ${alerts?.length ?? 0}`} dense> | |
| 391 | + | |
| 392 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 393 | + <Panel title={<span>Rules <span className="font-mono text-fg-subtle">{alerts?.length ?? "…"}</span></span>} dense> | |
| 140 | 394 | {alerts === null ? ( |
| 141 | − <Empty>Loading…</Empty> | |
| 395 | + <SkeletonRows rows={3} /> | |
| 142 | 396 | ) : alerts.length ? ( |
| 143 | 397 | <ul className="divide-y divide-line"> |
| 144 | 398 | {alerts.map((a) => ( |
| 145 | − <li key={a.id} className="flex items-start gap-3 px-3 py-2 text-[13px]"> | |
| 146 | − <BellRing className="mt-0.5 size-4 text-signal" /> | |
| 147 | − <div className="min-w-0 flex-1"> | |
| 148 | − <div className="font-medium">{a.name}</div> | |
| 149 | − <div className="mt-0.5 flex flex-wrap gap-1"> | |
| 150 | − {a.rule.importance_min !== undefined && <Chip>importance ≥ {a.rule.importance_min}</Chip>} | |
| 151 | − {a.rule.silent_only && <Chip tone="silent">silent only</Chip>} | |
| 152 | − {a.rule.event_types?.map((t) => <Chip key={t}>{typeLabel(t)}</Chip>)} | |
| 153 | − {a.rule.entities?.map((t) => <Chip key={t} tone="signal">{t}</Chip>)} | |
| 154 | − {a.rule.sources?.map((t) => <Chip key={t} tone="info">{t}</Chip>)} | |
| 155 | − {a.rule.keywords?.map((t) => <Chip key={t}>“{t}”</Chip>)} | |
| 399 | + <li key={a.id} className={`grid grid-cols-[auto_minmax(0,1fr)_auto] items-start gap-3 px-3 py-2 text-[13px] ${a.enabled ? "" : "opacity-60"}`}> | |
| 400 | + <button type="button" role="switch" aria-checked={a.enabled} aria-label={a.enabled ? "Disable rule" : "Enable rule"} title={a.enabled ? "Disable" : "Enable"} onClick={() => setEnabled(a, !a.enabled)} className={`mt-0.5 inline-flex size-6 items-center justify-center rounded-md border ${a.enabled ? "border-signal/40 bg-signal-soft text-signal" : "border-line text-fg-subtle"}`}> | |
| 401 | + {a.enabled ? <BellRing className="size-3.5" /> : <BellOff className="size-3.5" />} | |
| 402 | + </button> | |
| 403 | + <div className="min-w-0"> | |
| 404 | + <div className="flex flex-wrap items-center gap-2"> | |
| 405 | + <span className="font-medium">{a.name}</span> | |
| 406 | + <Chip tone={a.channel === "webhook" ? "info" : "default"} className="font-mono">{a.channel === "webhook" ? <><Webhook className="size-3" /> webhook</> : "web"}</Chip> | |
| 407 | + {!a.enabled && <Chip>paused</Chip>} | |
| 408 | + </div> | |
| 409 | + <div className="mt-1 flex flex-wrap gap-1">{ruleChips(a.rule)}</div> | |
| 410 | + <div className="mt-1 flex flex-wrap gap-x-3 font-mono text-[11px] text-fg-subtle tabular"> | |
| 411 | + <span>fired 24 h <span className="text-fg-muted">{a.fired_24h ?? 0}</span></span> | |
| 412 | + <span>total <span className="text-fg-muted">{a.fired_count ?? 0}</span></span> | |
| 413 | + <span>last {a.last_fired_at ? relTime(a.last_fired_at) : "never"}</span> | |
| 414 | + {a.channel === "webhook" && a.channel_config?.url && <span className="truncate">{a.channel_config.url}</span>} | |
| 415 | + </div> | |
| 416 | + </div> | |
| 417 | + <button type="button" aria-label={`Delete rule ${a.name}`} onClick={() => remove(a)} className="mt-0.5 text-fg-subtle hover:text-danger"><Trash2 className="size-3.5" /></button> | |
| 418 | + </li> | |
| 419 | + ))} | |
| 420 | + </ul> | |
| 421 | + ) : ( | |
| 422 | + <Empty>No rules yet — pick a preset or define conditions on the left.</Empty> | |
| 423 | + )} | |
| 424 | + </Panel> | |
| 425 | + | |
| 426 | + <Panel | |
| 427 | + title={<span>Notifications {notifs && notifs.unread > 0 && <span className="ml-1 rounded-sm border border-signal/40 bg-signal-soft px-1 font-mono text-[10px] text-signal">{notifs.unread} unread</span>}</span>} | |
| 428 | + action={notifs && notifs.unread > 0 ? <button type="button" onClick={markAllRead} className="text-[11px] text-fg-subtle hover:text-fg">mark all read</button> : undefined} | |
| 429 | + dense | |
| 430 | + > | |
| 431 | + {notifs === null ? ( | |
| 432 | + <SkeletonRows rows={3} /> | |
| 433 | + ) : notifs.items.length ? ( | |
| 434 | + <ul className="divide-y divide-line"> | |
| 435 | + {notifs.items.map((n) => ( | |
| 436 | + <li key={n.id} className={`grid grid-cols-[auto_minmax(0,1fr)_auto] items-center gap-3 px-3 py-1.5 text-[12.5px] ${n.read_at ? "" : "bg-panel-2/60"}`}> | |
| 437 | + <Score value={n.event.signal_score ?? n.event.importance} size="sm" kind="signal" /> | |
| 438 | + <div className="min-w-0"> | |
| 439 | + <Link href={`/event/${n.event.slug}`} className="block truncate font-medium hover:underline">{n.event.title}</Link> | |
| 440 | + <div className="flex flex-wrap gap-x-2 text-[11px] text-fg-subtle"> | |
| 441 | + <span>rule “{n.alert_name}”</span> | |
| 442 | + <span>· {n.event.source.name}</span> | |
| 443 | + <span>· <span className="font-mono">{n.channel}</span></span> | |
| 444 | + <span className={n.status === "failed" || n.status === "error" ? "text-danger" : n.status === "delivered" || n.status === "sent" ? "text-ok" : ""}>· {n.status}</span> | |
| 445 | + {!n.read_at && <span className="text-signal">· new</span>} | |
| 156 | 446 | </div> |
| 157 | 447 | </div> |
| 158 | − <button type="button" aria-label="Delete" onClick={() => ownerFetch(`/api/v1/alerts/${a.id}`, { method: "DELETE" }).then(load)} className="text-fg-subtle hover:text-danger"><Trash2 className="size-3.5" /></button> | |
| 448 | + <time dateTime={n.created_at} title={utcDateTime(n.created_at)} className="whitespace-nowrap font-mono text-[11px] text-fg-subtle tabular">{relTime(n.created_at)}</time> | |
| 159 | 449 | </li> |
| 160 | 450 | ))} |
| 161 | 451 | </ul> |
| 162 | 452 | ) : ( |
| 163 | − <Empty>No rules yet.</Empty> | |
| 453 | + <Empty>No notifications yet. Rules are evaluated by the engine on every new event; deliveries are logged here.</Empty> | |
| 164 | 454 | )} |
| 165 | 455 | </Panel> |
| 456 | + | |
| 166 | 457 | <Panel title={<span className="flex items-center gap-3">Fired in this session <LiveDot status={status} /></span>} dense> |
| 167 | 458 | {fired.length ? ( |
| 168 | 459 | <ul className="divide-y divide-line"> |
| 169 | 460 | {fired.map((f, i) => ( |
| 170 | 461 | <li key={`${f.event.id}-${i}`} className="flex items-center gap-3 px-3 py-2 text-[13px] animate-fade-in"> |
| 171 | − <Score value={f.event.importance} size="sm" /> | |
| 462 | + <Score value={f.event.signal ?? f.event.importance} size="sm" kind="signal" /> | |
| 172 | 463 | <div className="min-w-0 flex-1"> |
| 173 | 464 | <Link href={`/event/${f.event.slug}`} className="block truncate font-medium hover:underline">{f.event.title}</Link> |
| 174 | 465 | <div className="text-[11px] text-fg-subtle">rule “{f.alert.name}” · {f.event.source?.name} · {relTime(new Date(f.at))}</div> |
modified
apps/web/src/app/alerts/page.tsx
+7 −4
@@ -1,14 +1,17 @@ | ||
| 1 | 1 | import type { Metadata } from "next"; |
| 2 | −import { PageHeader } from "@/components/ui"; | |
| 2 | +import { Suspense } from "react"; | |
| 3 | +import { PageHeader, SkeletonPanel } from "@/components/ui"; | |
| 3 | 4 | import { Alerts } from "./alerts"; |
| 4 | 5 | |
| 5 | −export const metadata: Metadata = { title: "Alerts", description: "Alert rules on importance, event type, entity, source, keyword or silent changes. Delivered in this browser today; email, webhooks, Slack and Discord are planned.", robots: { index: false } }; | |
| 6 | +export const metadata: Metadata = { title: "Alerts", description: "Alert rules on signal, importance, event type, group, entity, source, keyword, country, silent or first-party changes. Delivered in this browser or to a signed HTTPS webhook.", robots: { index: false } }; | |
| 6 | 7 | |
| 7 | 8 | export default function AlertsPage() { |
| 8 | 9 | return ( |
| 9 | 10 | <> |
| 10 | − <PageHeader kicker="Rules evaluated on the live stream" title="Alerts" description="Define conditions; while this page (or any WebSensor tab with alerts enabled) is open, matching events trigger a toast and a browser notification. Email, push, webhook, Slack and Discord delivery are planned." /> | |
| 11 | − <Alerts /> | |
| 11 | + <PageHeader compact kicker="Rules evaluated by the engine on every new event" title="Alerts" description="Define conditions once. Web rules notify this browser while any WebSensor tab is open; webhook rules POST signed JSON to your HTTPS endpoint. Every delivery is logged below." /> | |
| 12 | + <Suspense fallback={<div className="grid gap-4 lg:grid-cols-[400px_minmax(0,1fr)]"><SkeletonPanel lines={10} /><SkeletonPanel lines={6} /></div>}> | |
| 13 | + <Alerts /> | |
| 14 | + </Suspense> | |
| 12 | 15 | </> |
| 13 | 16 | ); |
| 14 | 17 | } |
modified
apps/web/src/app/api/page.tsx
+144 −62
@@ -1,34 +1,63 @@ | ||
| 1 | 1 | import type { Metadata } from "next"; |
| 2 | 2 | import { Code } from "@/components/code"; |
| 3 | −import { PageHeader, Panel, Table, Td } from "@/components/ui"; | |
| 3 | +import { Chip, PageHeader, Panel, Table, Td } from "@/components/ui"; | |
| 4 | 4 | import { SITE_URL } from "@/lib/api"; |
| 5 | 5 | |
| 6 | −export const metadata: Metadata = { title: "API", description: "WebSensor REST API, RSS feed and WebSocket live stream. Public, no key required in phase 1." }; | |
| 6 | +export const metadata: Metadata = { title: "API", description: "WebSensor REST API, RSS feed and replayable WebSocket live stream (protocol 2). Public, no key required in phase 1; owner-scoped tools use an anonymous header." }; | |
| 7 | 7 | |
| 8 | −const ENDPOINTS: [string, string][] = [ | |
| 9 | − ["GET /api/v1/events", "List events. Filters: after, before, category, entity, source, domain, sensor, cluster, importance_min, confidence_min, event_type (comma list), silent_change, q, limit (≤200), cursor, order=recent|importance"], | |
| 10 | − ["GET /api/v1/events/{id|slug}", "Event detail: related events, cluster, change summary, interpretation versions, snapshots, sensor reliability"], | |
| 11 | − ["GET /api/v1/changes/{id}", "Raw change with stored heuristic and the unified patch"], | |
| 12 | − ["GET /api/v1/snapshots/{id}", "Snapshot metadata + canonical content; ?raw=1 streams the original body"], | |
| 8 | +const PUBLIC: [string, string][] = [ | |
| 9 | + ["GET /api/v1/events", "List events (cursor pagination). See filters below. order=recent|importance|signal"], | |
| 10 | + ["GET /api/v1/events/count", "Count of events matching the same filters as /events"], | |
| 11 | + ["GET /api/v1/events/{id|slug}", "Event detail: change + field changes + unified diff, score reasons, related, cluster with propagation, interpretations, snapshots (has_raw), sensor reliability, history of the URL"], | |
| 12 | + ["GET /api/v1/changes/{id}", "Raw change with heuristic, semantic class, field changes and the unified patch"], | |
| 13 | + ["GET /api/v1/snapshots/{id}", "Snapshot metadata + canonical content; ?raw=1 streams the original body when retention kept it"], | |
| 13 | 14 | ["GET /api/v1/snapshots/compare?a=&b=", "Diff between any two snapshots of the same URL"], |
| 14 | − ["GET /api/v1/sources", "Monitored organizations (category, q)"], | |
| 15 | − ["GET /api/v1/sources/{id}", "Source detail: sensors, entities, activity anomaly, discovered endpoints"], | |
| 16 | − ["GET /api/v1/sensors/{id}", "Sensor detail: runs, snapshots, changes"], | |
| 15 | + ["GET /api/v1/breaking", "Breaking desk: breaking_now, developing, recently_confirmed clusters + high-signal watching list"], | |
| 16 | + ["GET /api/v1/pulse", "Pulse: 6 h activity series, desks, rising entities, anomalous sources, silent changes, group totals"], | |
| 17 | + ["GET /api/v1/radar", "Radar: indicators (unusual source activity, silent clusters, doc/repo bursts, status changes, new coverage)"], | |
| 18 | + ["GET /api/v1/clusters?limit=&since=", "Event clusters (state, signals, sources, lead time)"], | |
| 19 | + ["GET /api/v1/clusters/{id|slug}", "Cluster detail: events, entities, propagation timeline, first-party vs external signals, lead_time_ms"], | |
| 17 | 20 | ["GET /api/v1/entities", "Entities (type, q)"], |
| 18 | − ["GET /api/v1/entities/{id}", "Entity detail: children, relations, sources, aliases, recent events"], | |
| 21 | + ["GET /api/v1/entities/rank?limit=", "Entity ranking: 24 h / 7 d activity, avg signal, confirmed ratio, baseline, rank score"], | |
| 22 | + ["GET /api/v1/entities/{id}", "Entity detail: children, relations, sources, aliases, recent, insights (heatmap, anomaly, rank)"], | |
| 19 | 23 | ["GET /api/v1/entities/{id}/timeline", "Entity timeline (cursor pagination)"], |
| 24 | + ["GET /api/v1/countries", "Countries with sources, events and breaking counts (24 h)"], | |
| 25 | + ["GET /api/v1/countries/{code}", "Country desk: breaking, by category, sources, by type, silent, recent"], | |
| 26 | + ["GET /api/v1/categories/{channel}", "Category desk (ai · cyber · finance · health · government · science · products · infrastructure · news)"], | |
| 27 | + ["GET /api/v1/sources", "Monitored organizations (category, q, country, tier, first_party)"], | |
| 28 | + ["GET /api/v1/sources/{id}", "Source detail: sensors, entities, activity anomaly, quality, discovery, daily"], | |
| 29 | + ["GET /api/v1/sensors/{id}", "Sensor detail: runs, snapshots, changes, events"], | |
| 30 | + ["GET /api/v1/sensors/{id}/snapshots?limit=", "Snapshot history of one sensor (hashes, HTTP status, has_raw, event slug)"], | |
| 20 | 31 | ["GET /api/v1/domains/{domain}/timeline", "Domain: monitored URLs + events"], |
| 21 | 32 | ["GET /api/v1/urls/history?url=", "URL history: snapshots, changes, events, removals"], |
| 22 | − ["GET /api/v1/search?q=", "Search events, entities, sources, URLs"], | |
| 23 | − ["GET /api/v1/stats", "Platform counters"], | |
| 33 | + ["GET /api/v1/search?q=", "Search events, entities, sources, URLs, clusters — accepts the q syntax below"], | |
| 34 | + ["GET /api/v1/stats", "Platform counters (checks/min, events/min, breaking_now, 304 ratio…)"], | |
| 24 | 35 | ["GET /api/v1/trending?hours=24", "Trending entities"], |
| 25 | 36 | ["GET /api/v1/explore", "Explore aggregates"], |
| 26 | − ["GET /api/v1/clusters", "Event clusters"], | |
| 27 | − ["GET /api/v1/health/connectors", "Connector health"], | |
| 28 | − ["GET/POST/PUT/DELETE /api/v1/watchlists", "Anonymous watchlists (header X-WebSensor-Owner)"], | |
| 29 | − ["GET/POST/DELETE /api/v1/alerts", "Alert rules (header X-WebSensor-Owner)"], | |
| 37 | + ["GET /api/v1/health/connectors", "Connector health, throughput, engine heartbeat, failing domains, slowest sensors"], | |
| 30 | 38 | ["GET /api/v1/feed.rss", "RSS 2.0 of the latest events (same filters as /events)"], |
| 31 | − ["WSS /api/v1/live", "Real-time event stream"], | |
| 39 | + ["WSS /api/v1/live", "Real-time event stream, protocol 2 (replayable)"], | |
| 40 | +]; | |
| 41 | + | |
| 42 | +const OWNER: [string, string][] = [ | |
| 43 | + ["GET/POST/PUT/DELETE /api/v1/watchlists", "Watchlists; item kinds entity · source · keyword · category · url · event_type · country · group. GET /watchlists/{id}/events"], | |
| 44 | + ["GET/POST/PATCH/DELETE /api/v1/alerts", "Alert rules {name, rule, channel: web|webhook, channel_config: {url, secret}}. PATCH {enabled, name}. Webhooks: https only, public hosts only"], | |
| 45 | + ["GET /api/v1/notifications?limit=&unread=1", "Deliveries of your rules with event summary; POST /notifications/read {ids?} marks read (all when omitted)"], | |
| 46 | + ["GET/POST/DELETE /api/v1/bookmarks", "Saved events {event_id, note?}; GET /bookmarks/ids for a quick membership check"], | |
| 47 | + ["GET/POST/DELETE /api/v1/views", "Saved live-feed views {name, query} where query is a /live query string"], | |
| 48 | + ["GET/POST/DELETE /api/v1/monitors", "Custom URL monitors {url, name?, frequency: hourly|daily, sensitivity: low|normal|high, selector?, keywords?}; GET /monitors/{id}/events. 5 per owner"], | |
| 49 | +]; | |
| 50 | + | |
| 51 | +const FILTERS: [string, string][] = [ | |
| 52 | + ["after · before", "ISO-8601 bounds on detected_at"], | |
| 53 | + ["category · group", "Category slug (ai, cyber…) · event group (security, reliability, product, commercial, corporate, government, science, transport, sports, web)"], | |
| 54 | + ["event_type", "Comma list of event types"], | |
| 55 | + ["entity · source · domain · sensor · cluster", "Scope to one id"], | |
| 56 | + ["importance_min · confidence_min · signal_min", "Score floors (0–100)"], | |
| 57 | + ["silent_change · first_party · confirmed", "Booleans (true)"], | |
| 58 | + ["country · language · change_class", "ISO-2 country (CA, US, EU…) · language code · semantic class (pricing, policy, product, personnel, meaningful…)"], | |
| 59 | + ["q", "Free text + search syntax (right)"], | |
| 60 | + ["order · limit · cursor", "recent (default) | importance | signal · ≤ 200 · opaque nextCursor"], | |
| 32 | 61 | ]; |
| 33 | 62 | |
| 34 | 63 | export default function ApiPage() { |
@@ -36,92 +65,145 @@ export default function ApiPage() { | ||
| 36 | 65 | const wss = base.replace(/^http/, "ws"); |
| 37 | 66 | return ( |
| 38 | 67 | <> |
| 39 | − <PageHeader kicker="Machine-readable WebSensor" title="API" description="WebSensor practises what it monitors: stable URLs, JSON, RSS, JSON-LD and a WebSocket feed. Phase 1 is public and unauthenticated; rate limit 600 requests / minute / IP." /> | |
| 40 | − <div className="grid gap-4 lg:grid-cols-[1fr_380px]"> | |
| 41 | − <div className="flex flex-col gap-4"> | |
| 42 | − <Panel title="REST endpoints" dense> | |
| 68 | + <PageHeader compact kicker="Machine-readable WebSensor" title="API" description="WebSensor practises what it monitors: stable URLs, JSON, RSS, JSON-LD and a replayable WebSocket feed. Phase 1 is public and unauthenticated; rate limit 600 requests / minute / IP. Owner-scoped tools (watchlists, alerts, bookmarks, views, monitors) use an anonymous X-WebSensor-Owner token generated by your browser." /> | |
| 69 | + <div className="grid grid-cols-1 gap-4 lg:grid-cols-[minmax(0,1fr)_400px]"> | |
| 70 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 71 | + <Panel title="Public REST endpoints" dense> | |
| 43 | 72 | <Table head={["Endpoint", "Description"]}> |
| 44 | − {ENDPOINTS.map(([ep, desc]) => ( | |
| 73 | + {PUBLIC.map(([ep, desc]) => ( | |
| 45 | 74 | <tr key={ep}> |
| 46 | − <Td mono className="whitespace-nowrap">{ep}</Td> | |
| 75 | + <Td mono className="sm:whitespace-nowrap">{ep}</Td> | |
| 47 | 76 | <Td className="text-fg-muted">{desc}</Td> |
| 48 | 77 | </tr> |
| 49 | 78 | ))} |
| 50 | 79 | </Table> |
| 51 | 80 | </Panel> |
| 81 | + <Panel title={<span>Owner-scoped endpoints <span className="normal-case tracking-normal text-fg-subtle">· header X-WebSensor-Owner: <16–80 chars [A-Za-z0-9_-]></span></span>} dense> | |
| 82 | + <Table head={["Endpoint", "Description"]}> | |
| 83 | + {OWNER.map(([ep, desc]) => ( | |
| 84 | + <tr key={ep}> | |
| 85 | + <Td mono className="sm:whitespace-nowrap">{ep}</Td> | |
| 86 | + <Td className="text-fg-muted">{desc}</Td> | |
| 87 | + </tr> | |
| 88 | + ))} | |
| 89 | + </Table> | |
| 90 | + <p className="border-t border-line px-3 py-2 text-[11.5px] text-fg-subtle">No accounts yet: the token is your identity. Anyone holding it can read and edit these resources. Webhook deliveries are signed when a secret is set: <code>X-WebSensor-Signature: sha256=<HMAC-SHA256(secret, raw body)></code>.</p> | |
| 91 | + </Panel> | |
| 92 | + <Panel title="Event filters · /events, /events/count, /feed.rss" dense> | |
| 93 | + <Table head={["Parameter", "Meaning"]}> | |
| 94 | + {FILTERS.map(([k, v]) => ( | |
| 95 | + <tr key={k}> | |
| 96 | + <Td mono className="sm:whitespace-nowrap">{k}</Td> | |
| 97 | + <Td className="text-fg-muted">{v}</Td> | |
| 98 | + </tr> | |
| 99 | + ))} | |
| 100 | + </Table> | |
| 101 | + </Panel> | |
| 52 | 102 | <Panel title="Examples"> |
| 53 | 103 | <div className="flex flex-col gap-3 text-[13px]"> |
| 54 | − <p>Latest AI events with importance ≥ 70:</p> | |
| 55 | − <Code>{`curl "${base}/api/v1/events?category=ai&importance_min=70&limit=20"`}</Code> | |
| 56 | − <p>Silent pricing / terms changes only:</p> | |
| 57 | − <Code>{`curl "${base}/api/v1/events?silent_change=true&event_type=pricing_change,terms_change"`}</Code> | |
| 58 | − <p>OpenAI timeline as git-log style history:</p> | |
| 59 | − <Code>{`curl "${base}/api/v1/entities/org_openai/timeline?limit=50"`}</Code> | |
| 60 | − <p>Subscribe to an RSS reader:</p> | |
| 61 | − <Code>{`${base}/api/v1/feed.rss?importance_min=60`}</Code> | |
| 62 | − <p>Compare two snapshots of the same URL:</p> | |
| 63 | − <Code>{`curl "${base}/api/v1/snapshots/compare?a=snap_…&b=snap_…"`}</Code> | |
| 104 | + <p>Strongest signals of the last 48 h, first-party only:</p> | |
| 105 | + <Code>{`curl "${base}/api/v1/events?signal_min=80&first_party=true&order=signal&after=$(date -u -v-48H +%FT%TZ)"`}</Code> | |
| 106 | + <p>Silent pricing / terms changes in the commercial group:</p> | |
| 107 | + <Code>{`curl "${base}/api/v1/events?silent_change=true&group=commercial&change_class=pricing"`}</Code> | |
| 108 | + <p>Everything from Canada in the government category, counted:</p> | |
| 109 | + <Code>{`curl "${base}/api/v1/events/count?country=CA&category=government"`}</Code> | |
| 110 | + <p>Search syntax (same as the ⌘K palette):</p> | |
| 111 | + <Code>{`curl "${base}/api/v1/search?q=$(printf %s 'openai pricing type:pricing_change signal:>70 after:2026-09-01' | jq -sRr @uri)"`}</Code> | |
| 112 | + <p>Cluster with propagation timeline and lead time:</p> | |
| 113 | + <Code>{`curl "${base}/api/v1/clusters/<slug>" | jq '{lead_time_ms, first_party_signals, external_signals, propagation: .propagation[:3]}'`}</Code> | |
| 114 | + <p>Create a signed webhook alert (owner token = any 16–80 char string you keep):</p> | |
| 115 | + <Code>{`curl -X POST "${base}/api/v1/alerts" \\ | |
| 116 | + -H "content-type: application/json" -H "X-WebSensor-Owner: $OWNER" \\ | |
| 117 | + -d '{"name":"Critical security","rule":{"groups":["security"],"signal_min":80,"first_party_only":true}, | |
| 118 | + "channel":"webhook","channel_config":{"url":"https://example.com/hooks/ws","secret":"…"}}'`}</Code> | |
| 119 | + <p>Subscribe an RSS reader:</p> | |
| 120 | + <Code>{`${base}/api/v1/feed.rss?signal_min=60&group=reliability`}</Code> | |
| 64 | 121 | </div> |
| 65 | 122 | </Panel> |
| 66 | 123 | <Panel title="Event object"> |
| 67 | 124 | <Code lang="json">{`{ |
| 68 | 125 | "id": "evt_…", "slug": "openai-api-pricing-changed-…", |
| 69 | − "event_type": "pricing_change", | |
| 126 | + "event_type": "pricing_change", "change_class": "pricing", | |
| 70 | 127 | "title": "OpenAI: price changed $10 / million tokens → $8 / million tokens", |
| 71 | 128 | "summary": "…", "why_it_matters": "…", |
| 72 | − "importance": 91.2, "confidence": 98.1, "novelty": 87.4, | |
| 129 | + "signal_score": 91, "importance": 88.2, "confidence": 98.1, "novelty": 87.4, | |
| 130 | + "impact_score": 85, "velocity_score": 40, "anomaly_score": 12, | |
| 131 | + "score_reasons": [{ "sign": "+", "text": "First-party source", "points": 8 }, { "sign": "-", "text": "Single source so far", "points": -5 }], | |
| 73 | 132 | "importance_components": { "severity": 82, "source": 92, "entity": 92, "novelty": 87, "magnitude": 40, "confirmation": 0, "userImpact": 85, "unusualness": 30 }, |
| 133 | + "field_changes": [{ "label": "gpt-4o input", "kind": "price", "before": "$10.00", "after": "$8.00", "deltaPct": -20 }], | |
| 74 | 134 | "categories": ["ai", "technology"], "keywords": ["pricing"], |
| 75 | − "silent_change": true, "evidence_label": "OBSERVED", | |
| 135 | + "silent_change": true, "first_party": true, "country": "US", "language": "en", | |
| 136 | + "evidence_label": "OBSERVED", | |
| 76 | 137 | "url": "https://openai.com/api/pricing/", |
| 77 | 138 | "published_at": null, "observed_from": "…", "detected_at": "…", "processed_at": "…", |
| 78 | 139 | "detection_latency_ms": null, "processing_latency_ms": 412, |
| 79 | − "cluster_id": "clu_…", "change_id": "chg_…", "old_snapshot_id": "snap_…", "new_snapshot_id": "snap_…", | |
| 140 | + "cluster": { "id": "clu_…", "slug": "openai-pricing-…", "state": "developing", "event_count": 3, "source_count": 2, | |
| 141 | + "first_party_count": 1, "external_count": 2, "velocity": 61, "lead_time_ms": 5400000 }, | |
| 142 | + "change_id": "chg_…", "old_snapshot_id": "snap_…", "new_snapshot_id": "snap_…", | |
| 80 | 143 | "source": { "id": "openai", "name": "OpenAI", "domain": "openai.com", "tier": "S" }, |
| 81 | 144 | "sensor": { "id": "openai_pricing", "name": "pricing", "type": "HTML", "connector": "http" }, |
| 82 | − "entities": [{ "id": "org_openai", "name": "OpenAI", "type": "organization", "role": "subject" }], | |
| 83 | − "cluster_size": 3 | |
| 145 | + "entities": [{ "id": "org_openai", "name": "OpenAI", "type": "organization", "role": "subject" }] | |
| 84 | 146 | }`}</Code> |
| 85 | 147 | </Panel> |
| 86 | 148 | </div> |
| 87 | − <aside className="flex flex-col gap-4"> | |
| 88 | − <Panel title="WebSocket live stream"> | |
| 149 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 150 | + <Panel title="WebSocket live stream · protocol 2"> | |
| 89 | 151 | <div className="flex flex-col gap-3 text-[13px]"> |
| 90 | 152 | <Code>{`wscat -c ${wss}/api/v1/live |
| 91 | −> {"subscribe":["events:breaking","entity:org_openai"]}`}</Code> | |
| 153 | +> {"subscribe":["group:security","country:CA","entity:org_openai"]} | |
| 154 | +> {"since":"1757333722000-0"} # after a reconnect`}</Code> | |
| 92 | 155 | <p className="text-fg-muted">Channels:</p> |
| 93 | − <ul className="list-disc space-y-0.5 pl-5 font-mono text-[12px] text-fg-muted"> | |
| 94 | − <li>events:global (default)</li> | |
| 95 | − <li>events:breaking (importance ≥ 80)</li> | |
| 96 | − <li>events:silent</li> | |
| 97 | − <li>events:ai · cyber · finance · health · government · science · products · infrastructure · news</li> | |
| 98 | − <li>entity:{"{entity_id}"}</li> | |
| 99 | − <li>source:{"{source_id}"}</li> | |
| 100 | − <li>type:{"{event_type}"}</li> | |
| 101 | − <li>watchlist:{"{watchlist_id}"}</li> | |
| 156 | + <ul className="list-disc space-y-0.5 pl-5 font-mono text-[12px] text-fg-muted [overflow-wrap:anywhere]"> | |
| 157 | + <li>events:global (default) · events:breaking · events:silent · events:first-party</li> | |
| 158 | + <li>events:{"{ai|cyber|finance|health|government|science|products|infrastructure|news}"}</li> | |
| 159 | + <li>group:{"{security|reliability|product|commercial|corporate|government|science|transport|sports|web}"}</li> | |
| 160 | + <li>country:{"{CA}"} · state:{"{breaking|developing|confirmed}"} · type:{"{event_type}"}</li> | |
| 161 | + <li>entity:{"{entity_id}"} · source:{"{source_id}"} · watchlist:{"{watchlist_id}"}</li> | |
| 102 | 162 | </ul> |
| 103 | 163 | <p className="text-fg-muted">Frames:</p> |
| 104 | 164 | <Code lang="json">{`{"type":"hello","channels":["events:global"]} |
| 105 | −{"type":"event","channels":["events:global","events:ai"], | |
| 165 | +{"type":"event","sid":"1757333722000-0","channels":["events:global","group:product"], | |
| 106 | 166 | "event":{"id":"evt_…","slug":"…","type":"model_release","title":"…", |
| 107 | − "importance":97,"confidence":95,"novelty":90,"silent":false, | |
| 108 | − "evidence":"OBSERVED","source":{"id":"openai","name":"OpenAI","domain":"openai.com"}, | |
| 167 | + "signal":93,"importance":97,"confidence":95,"novelty":90,"impact":80,"velocity":55, | |
| 168 | + "silent":false,"firstParty":true,"country":"US","changeClass":"product", | |
| 169 | + "fieldChanges":[…],"evidence":"OBSERVED","group":"product", | |
| 170 | + "source":{"id":"openai","name":"OpenAI","domain":"openai.com"}, | |
| 109 | 171 | "sensor":{"id":"openai_news","name":"news feed","type":"RSS"}, |
| 110 | 172 | "entities":[{"id":"org_openai","name":"OpenAI","type":"organization"}], |
| 111 | − "categories":["ai"],"url":"https://…","clusterId":"clu_…", | |
| 173 | + "categories":["ai"],"url":"https://…","clusterId":"clu_…","clusterSlug":"…","clusterState":"breaking", | |
| 112 | 174 | "detectedAt":"2026-09-08T12:15:22Z","publishedAt":"2026-09-08T12:14:55Z"}} |
| 175 | +{"type":"replay_done","since":"1757333722000-0","count":12,"truncated":false} | |
| 113 | 176 | {"type":"heartbeat","t":1757333722000}`}</Code> |
| 114 | − <p className="text-fg-muted">Send <code>{`{"ping":1}`}</code> for a <code>pong</code>. Heartbeats every 25 s keep proxies alive.</p> | |
| 177 | + <ul className="list-disc space-y-1 pl-5 text-[12.5px] text-fg-muted"> | |
| 178 | + <li>Every event frame carries a stream id <code>sid</code>. Keep the last one; after reconnecting send <code>{`{"since": sid}`}</code> and up to 500 missed events are replayed in order, then <code>replay_done</code>.</li> | |
| 179 | + <li>Send <code>{`{"ping":1}`}</code> for a <code>pong</code>. Heartbeats every 25 s keep proxies alive.</li> | |
| 180 | + <li>Subscriptions are additive; <code>unsubscribe</code> removes channels (including the default <code>events:global</code>).</li> | |
| 181 | + </ul> | |
| 115 | 182 | </div> |
| 116 | 183 | </Panel> |
| 184 | + <Panel title="Search syntax · q"> | |
| 185 | + <Code>{`openai pricing entity:org_openai type:pricing_change,terms_change | |
| 186 | +after:2026-09-01 before:2026-09-10 after:7d | |
| 187 | +silent:true first_party:true confirmed:true | |
| 188 | +importance:>70 confidence:>=50 signal:>80 | |
| 189 | +source:"bank of canada" category:ai country:CA lang:fr | |
| 190 | +class:pricing group:security cluster:clu_…`}</Code> | |
| 191 | + <p className="mt-2 text-[12px] text-fg-muted">Quoted values allowed; unknown keys stay in the free-text part. Works in <code>/events?q=</code>, <code>/search?q=</code> and the ⌘K palette.</p> | |
| 192 | + </Panel> | |
| 117 | 193 | <Panel title="Conventions"> |
| 118 | 194 | <ul className="list-disc space-y-1 pl-5 text-[12.5px] text-fg-muted"> |
| 119 | − <li>All timestamps are UTC ISO-8601.</li> | |
| 195 | + <li>All timestamps are UTC ISO-8601. Ids are prefixed (<code>evt_ chg_ snap_ clu_ src_ sen_ ent_</code>).</li> | |
| 120 | 196 | <li>Cursor pagination: pass <code>nextCursor</code> back as <code>cursor</code>.</li> |
| 121 | − <li>Scores are 0–100. Importance and confidence are independent.</li> | |
| 122 | − <li>Every event traces to change → snapshots → sensor → source; snapshots are immutable.</li> | |
| 123 | − <li>Labels OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED are always exposed.</li> | |
| 124 | − <li>Planned: API keys, customer webhooks, MCP server.</li> | |
| 197 | + <li>Scores are 0–100. Signal, importance and confidence are independent; <code>score_reasons</code> explain each signal score.</li> | |
| 198 | + <li>Every event traces to change → snapshots → sensor → source; snapshots are immutable. Retention may drop raw bodies (<code>has_raw: false</code>) but never canonical forms or hashes.</li> | |
| 199 | + <li>Labels OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED are always exposed; AI text is labelled analysis.</li> | |
| 200 | + <li> | |
| 201 | + Errors share one shape: <code>{`{"error": "snake_case_code", "detail"?: "…"}`}</code>. Validation failures add <code>details</code> (issues); 404 is <code>not_found</code>, 429 <code>monitor_limit_reached</code> / <code>too_many_*</code>. | |
| 202 | + </li> | |
| 203 | + <li> | |
| 204 | + Rate limit 600 req / min / IP on <code>/api/*</code>, exposed as <Chip className="font-mono">x-ratelimit-limit</Chip> <Chip className="font-mono">x-ratelimit-remaining</Chip> <Chip className="font-mono">x-ratelimit-reset</Chip>; every response has <code>x-request-id</code>. | |
| 205 | + </li> | |
| 206 | + <li>Planned: API keys, MCP server, e-mail / Slack / Discord channels.</li> | |
| 125 | 207 | </ul> |
| 126 | 208 | </Panel> |
| 127 | 209 | </aside> |
added
apps/web/src/app/bookmarks/bookmarks.tsx
+184 −0
@@ -0,0 +1,184 @@ | ||
| 1 | +"use client"; | |
| 2 | + | |
| 3 | +import { Bookmark as BookmarkIcon, Plus, Trash2 } from "lucide-react"; | |
| 4 | +import Link from "next/link"; | |
| 5 | +import { useCallback, useEffect, useState } from "react"; | |
| 6 | +import { EventRow } from "@/components/event-row"; | |
| 7 | +import { Chip, Empty, Panel, SkeletonRows } from "@/components/ui"; | |
| 8 | +import type { Bookmark } from "@/lib/api"; | |
| 9 | +import { relTime, SAVED_VIEWS, utcDateTime } from "@/lib/format"; | |
| 10 | +import { ownerFetch } from "@/lib/owner"; | |
| 11 | + | |
| 12 | +interface SavedView { | |
| 13 | + id: string; | |
| 14 | + name: string; | |
| 15 | + query: string; | |
| 16 | + created_at?: string; | |
| 17 | +} | |
| 18 | + | |
| 19 | +function normalizeQuery(q: string): string { | |
| 20 | + const s = q.trim().replace(/^\/?live/, "").replace(/^\?/, ""); | |
| 21 | + return s; | |
| 22 | +} | |
| 23 | + | |
| 24 | +export function Bookmarks() { | |
| 25 | + const [items, setItems] = useState<Bookmark[] | null>(null); | |
| 26 | + const [views, setViews] = useState<SavedView[] | null>(null); | |
| 27 | + const [err, setErr] = useState<string | null>(null); | |
| 28 | + const [viewErr, setViewErr] = useState<string | null>(null); | |
| 29 | + const [form, setForm] = useState({ name: "", query: "" }); | |
| 30 | + | |
| 31 | + const load = useCallback( | |
| 32 | + () => | |
| 33 | + ownerFetch<{ items: Bookmark[] }>("/api/v1/bookmarks?limit=200") | |
| 34 | + .then((r) => setItems(r.items)) | |
| 35 | + .catch((e: Error) => { | |
| 36 | + setErr(e.message); | |
| 37 | + setItems([]); | |
| 38 | + }), | |
| 39 | + [], | |
| 40 | + ); | |
| 41 | + const loadViews = useCallback( | |
| 42 | + () => | |
| 43 | + ownerFetch<{ items: SavedView[] }>("/api/v1/views") | |
| 44 | + .then((r) => setViews(r.items)) | |
| 45 | + .catch((e: Error) => { | |
| 46 | + setViewErr(e.message); | |
| 47 | + setViews([]); | |
| 48 | + }), | |
| 49 | + [], | |
| 50 | + ); | |
| 51 | + useEffect(() => { | |
| 52 | + void load(); | |
| 53 | + void loadViews(); | |
| 54 | + }, [load, loadViews]); | |
| 55 | + | |
| 56 | + const remove = async (id: string): Promise<void> => { | |
| 57 | + setItems((prev) => prev?.filter((b) => b.id !== id) ?? prev); | |
| 58 | + try { | |
| 59 | + await ownerFetch(`/api/v1/bookmarks/${encodeURIComponent(id)}`, { method: "DELETE" }); | |
| 60 | + } catch (e) { | |
| 61 | + setErr((e as Error).message); | |
| 62 | + await load(); | |
| 63 | + } | |
| 64 | + }; | |
| 65 | + | |
| 66 | + const createView = async (): Promise<void> => { | |
| 67 | + const query = normalizeQuery(form.query); | |
| 68 | + if (!query) { | |
| 69 | + setViewErr("Paste a live-feed query string, e.g. group=security&signal_min=70"); | |
| 70 | + return; | |
| 71 | + } | |
| 72 | + setViewErr(null); | |
| 73 | + try { | |
| 74 | + await ownerFetch("/api/v1/views", { method: "POST", body: JSON.stringify({ name: form.name.trim() || "My view", query }) }); | |
| 75 | + setForm({ name: "", query: "" }); | |
| 76 | + await loadViews(); | |
| 77 | + } catch (e) { | |
| 78 | + setViewErr((e as Error).message); | |
| 79 | + } | |
| 80 | + }; | |
| 81 | + const removeView = async (id: string): Promise<void> => { | |
| 82 | + setViews((prev) => prev?.filter((v) => v.id !== id) ?? prev); | |
| 83 | + try { | |
| 84 | + await ownerFetch(`/api/v1/views/${encodeURIComponent(id)}`, { method: "DELETE" }); | |
| 85 | + } catch (e) { | |
| 86 | + setViewErr((e as Error).message); | |
| 87 | + await loadViews(); | |
| 88 | + } | |
| 89 | + }; | |
| 90 | + | |
| 91 | + return ( | |
| 92 | + <div className="grid gap-4 lg:grid-cols-[minmax(0,1fr)_340px]"> | |
| 93 | + <Panel title={<span>Saved events <span className="font-mono text-fg-subtle">{items?.length ?? "…"}</span></span>} dense> | |
| 94 | + {items === null ? ( | |
| 95 | + <SkeletonRows rows={6} /> | |
| 96 | + ) : items.length ? ( | |
| 97 | + <ul className="divide-y divide-line"> | |
| 98 | + {items.map((b) => ( | |
| 99 | + <li key={b.id} className="grid grid-cols-[minmax(0,1fr)_auto] items-start gap-2 pr-2"> | |
| 100 | + <div className="min-w-0 [&>article]:border-b-0"> | |
| 101 | + <EventRow ev={b} showDate /> | |
| 102 | + </div> | |
| 103 | + <div className="flex flex-col items-end gap-1 pt-2"> | |
| 104 | + <button type="button" aria-label="Remove bookmark" title="Remove bookmark" onClick={() => remove(b.id)} className="inline-flex size-7 items-center justify-center rounded-md border border-line text-fg-subtle hover:border-line-strong hover:text-danger"> | |
| 105 | + <Trash2 className="size-3.5" /> | |
| 106 | + </button> | |
| 107 | + <span className="inline-flex items-center gap-1 whitespace-nowrap font-mono text-[10.5px] text-fg-subtle" title={`Saved ${utcDateTime(b.bookmarked_at)}${b.note ? ` · ${b.note}` : ""}`}> | |
| 108 | + <BookmarkIcon className="size-3 text-signal" /> {relTime(b.bookmarked_at)} | |
| 109 | + </span> | |
| 110 | + </div> | |
| 111 | + </li> | |
| 112 | + ))} | |
| 113 | + </ul> | |
| 114 | + ) : ( | |
| 115 | + <Empty icon={<BookmarkIcon className="size-5 text-fg-subtle" />}> | |
| 116 | + No saved events yet — use the bookmark icon in the intelligence panel. | |
| 117 | + <div className="mt-2"> | |
| 118 | + <Link href="/live" className="text-info hover:underline">Open the live feed →</Link> | |
| 119 | + </div> | |
| 120 | + </Empty> | |
| 121 | + )} | |
| 122 | + {err && <p className="border-t border-line px-3 py-2 text-[11px] text-danger">{err}</p>} | |
| 123 | + </Panel> | |
| 124 | + | |
| 125 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 126 | + <Panel title={<span>Saved views <span className="font-mono text-fg-subtle">{views?.length ?? "…"}</span></span>} dense> | |
| 127 | + {views === null ? ( | |
| 128 | + <SkeletonRows rows={3} /> | |
| 129 | + ) : views.length ? ( | |
| 130 | + <ul className="divide-y divide-line"> | |
| 131 | + {views.map((v) => ( | |
| 132 | + <li key={v.id} className="flex items-start gap-2 px-3 py-2 text-[13px]"> | |
| 133 | + <div className="min-w-0 flex-1"> | |
| 134 | + <Link href={`/live?${v.query}`} className="block truncate font-medium hover:underline">{v.name}</Link> | |
| 135 | + <div className="mt-0.5 truncate font-mono text-[11px] text-fg-subtle" title={v.query}>?{v.query}</div> | |
| 136 | + </div> | |
| 137 | + <button type="button" aria-label={`Delete view ${v.name}`} onClick={() => removeView(v.id)} className="mt-0.5 text-fg-subtle hover:text-danger"> | |
| 138 | + <Trash2 className="size-3.5" /> | |
| 139 | + </button> | |
| 140 | + </li> | |
| 141 | + ))} | |
| 142 | + </ul> | |
| 143 | + ) : ( | |
| 144 | + <Empty>No saved views yet. Filter the live feed, then paste its query string below.</Empty> | |
| 145 | + )} | |
| 146 | + <form | |
| 147 | + className="flex flex-col gap-1.5 border-t border-line p-2 text-[12.5px]" | |
| 148 | + onSubmit={(e) => { | |
| 149 | + e.preventDefault(); | |
| 150 | + void createView(); | |
| 151 | + }} | |
| 152 | + > | |
| 153 | + <label className="sr-only" htmlFor="view-name">View name</label> | |
| 154 | + <input id="view-name" value={form.name} onChange={(e) => setForm({ ...form, name: e.target.value })} placeholder="View name" maxLength={60} className="h-8 rounded-md border border-line bg-panel px-2 text-[12.5px]" /> | |
| 155 | + <label className="sr-only" htmlFor="view-query">Query string</label> | |
| 156 | + <input id="view-query" value={form.query} onChange={(e) => setForm({ ...form, query: e.target.value })} placeholder="group=security&signal_min=70" maxLength={600} className="h-8 rounded-md border border-line bg-panel px-2 font-mono text-[12px]" /> | |
| 157 | + <div className="flex items-center justify-between gap-2"> | |
| 158 | + <button type="button" onClick={() => setForm({ ...form, query: typeof window !== "undefined" && window.location.pathname.startsWith("/live") ? normalizeQuery(window.location.search) : form.query })} className="text-[11px] text-fg-subtle hover:text-fg" title="Only works from a /live page">use current query</button> | |
| 159 | + <button type="submit" className="inline-flex h-8 items-center gap-1 rounded-md border border-line bg-panel-2 px-2 text-[12px] hover:border-line-strong"> | |
| 160 | + <Plus className="size-3.5" /> Save view | |
| 161 | + </button> | |
| 162 | + </div> | |
| 163 | + {viewErr && <p className="text-[11px] text-danger">{viewErr}</p>} | |
| 164 | + </form> | |
| 165 | + </Panel> | |
| 166 | + <Panel title="Built-in views" dense> | |
| 167 | + <ul className="divide-y divide-line"> | |
| 168 | + {SAVED_VIEWS.map((v) => { | |
| 169 | + const q = new URLSearchParams(Object.entries(v.query).map(([k, val]) => [k, String(val)])).toString(); | |
| 170 | + return ( | |
| 171 | + <li key={v.key} className="flex items-center justify-between gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 172 | + <Link href={`/live?${q}`} className="min-w-0 truncate hover:underline">{v.label}</Link> | |
| 173 | + <button type="button" onClick={() => setForm({ name: v.label, query: q })} className="shrink-0" title="Copy into the form"> | |
| 174 | + <Chip className="hover:border-line-strong">copy</Chip> | |
| 175 | + </button> | |
| 176 | + </li> | |
| 177 | + ); | |
| 178 | + })} | |
| 179 | + </ul> | |
| 180 | + </Panel> | |
| 181 | + </aside> | |
| 182 | + </div> | |
| 183 | + ); | |
| 184 | +} | |
added
apps/web/src/app/bookmarks/page.tsx
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +import type { Metadata } from "next"; | |
| 2 | +import { PageHeader } from "@/components/ui"; | |
| 3 | +import { Bookmarks } from "./bookmarks"; | |
| 4 | + | |
| 5 | +export const metadata: Metadata = { title: "Saved events", description: "Events you bookmarked in this browser, plus your saved live-feed views.", robots: { index: false } }; | |
| 6 | + | |
| 7 | +export default function BookmarksPage() { | |
| 8 | + return ( | |
| 9 | + <> | |
| 10 | + <PageHeader compact kicker="Stored in this browser (no account yet)" title="Saved" description="Bookmarked events and saved live-feed views. Everything is scoped to this browser's anonymous owner token." /> | |
| 11 | + <Bookmarks /> | |
| 12 | + </> | |
| 13 | + ); | |
| 14 | +} | |
added
apps/web/src/app/breaking/loading.tsx
+41 −0
@@ -0,0 +1,41 @@ | ||
| 1 | +import { Skeleton, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +function PanelSkeleton({ rows }: { rows: number }) { | |
| 4 | + return ( | |
| 5 | + <div className="panel"> | |
| 6 | + <div className="border-b border-line px-3 py-2"> | |
| 7 | + <Skeleton className="h-2.5 w-28" /> | |
| 8 | + </div> | |
| 9 | + <SkeletonRows rows={rows} /> | |
| 10 | + </div> | |
| 11 | + ); | |
| 12 | +} | |
| 13 | + | |
| 14 | +export default function Loading() { | |
| 15 | + return ( | |
| 16 | + <> | |
| 17 | + <div className="mb-4 flex flex-col gap-2"> | |
| 18 | + <div className="flex items-center gap-2"> | |
| 19 | + <Skeleton className="h-4 w-20" /> | |
| 20 | + <Skeleton className="h-3 w-80 max-w-full" /> | |
| 21 | + </div> | |
| 22 | + <Skeleton className="h-7 w-40" /> | |
| 23 | + </div> | |
| 24 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-2"> | |
| 25 | + <PanelSkeleton rows={4} /> | |
| 26 | + <PanelSkeleton rows={4} /> | |
| 27 | + </div> | |
| 28 | + <div className="mt-4 grid grid-cols-1 gap-4 xl:grid-cols-2"> | |
| 29 | + <PanelSkeleton rows={3} /> | |
| 30 | + <PanelSkeleton rows={3} /> | |
| 31 | + </div> | |
| 32 | + <div className="mt-4 panel"> | |
| 33 | + <div className="flex items-center gap-3 border-b border-line px-3 py-2"> | |
| 34 | + <Skeleton className="h-3 w-48" /> | |
| 35 | + <Skeleton className="h-3 w-12" /> | |
| 36 | + </div> | |
| 37 | + <SkeletonRows rows={8} /> | |
| 38 | + </div> | |
| 39 | + </> | |
| 40 | + ); | |
| 41 | +} | |
modified
apps/web/src/app/breaking/page.tsx
+67 −6
@@ -1,17 +1,78 @@ | ||
| 1 | 1 | import type { Metadata } from "next"; |
| 2 | +import Link from "next/link"; | |
| 3 | +import { EventRow } from "@/components/event-row"; | |
| 2 | 4 | import { LiveFeed } from "@/components/live-feed"; |
| 3 | −import { PageHeader } from "@/components/ui"; | |
| 4 | −import { api } from "@/lib/api"; | |
| 5 | +import { Badge, Chip, Empty, Flag, PageHeader, Panel, Score, StateLabelText } from "@/components/ui"; | |
| 6 | +import { api, type Cluster, type EventItem } from "@/lib/api"; | |
| 7 | +import { fmtOffset, relTime, typeLabel, utcTime } from "@/lib/format"; | |
| 5 | 8 | |
| 6 | 9 | export const dynamic = "force-dynamic"; |
| 7 | −export const metadata: Metadata = { title: "Breaking", description: "Events with importance ≥ 80 detected in the last 48 hours." }; | |
| 10 | +export const metadata: Metadata = { title: "Breaking", description: "Breaking now, developing, recently confirmed and watching — ranked by signal, velocity, novelty and confirmation, not by recency." }; | |
| 8 | 11 | |
| 12 | +/** Breaking desk (spec §34). */ | |
| 9 | 13 | export default async function BreakingPage() { |
| 10 | − const events = await api.breaking(60); | |
| 14 | + const [desk, live] = await Promise.all([api.breakingDesk(), api.breaking(40)]); | |
| 15 | + const d = desk ?? { breaking_now: [], developing: [], recently_confirmed: [], watching: [], generated_at: "" }; | |
| 11 | 16 | return ( |
| 12 | 17 | <> |
| 13 | − <PageHeader kicker="Importance ≥ 80 · last 48 h" title="Breaking" description="Highest-importance events across every category, ranked by score. New breaking events stream in live." /> | |
| 14 | − <LiveFeed initial={events.items} initialCursor={events.nextCursor} fixed="breaking" title="BREAKING" /> | |
| 18 | + <PageHeader kicker={<span className="flex items-center gap-2"><Badge kind="breaking" /> <span className="text-fg-subtle">Breaking is not recent: it needs strong signal, velocity, novelty, source quality, confirmation and cross-source activity.</span></span>} title="Breaking" /> | |
| 19 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-[1fr_1fr] [&>*]:min-w-0"> | |
| 20 | + <ClusterSection title="Breaking now" tone="hot" items={d.breaking_now} empty="Nothing is breaking right now." /> | |
| 21 | + <ClusterSection title="Developing" tone="high" items={d.developing} empty="No developing story: signals are not accumulating fast enough anywhere." /> | |
| 22 | + </div> | |
| 23 | + <div className="mt-4 grid grid-cols-1 gap-4 xl:grid-cols-[1fr_1fr] [&>*]:min-w-0"> | |
| 24 | + <ClusterSection title="Recently confirmed" tone="ok" items={d.recently_confirmed} empty="No cluster reached independent confirmation in the last 48 h." /> | |
| 25 | + <Panel title="Watching · high signal, not yet breaking" dense> | |
| 26 | + {d.watching.length ? d.watching.slice(0, 12).map((e) => <EventRow key={e.id} ev={e} />) : <Empty>Nothing to watch.</Empty>} | |
| 27 | + </Panel> | |
| 28 | + </div> | |
| 29 | + <div className="mt-4"> | |
| 30 | + <LiveFeed initial={live.items} initialCursor={live.nextCursor} fixed="breaking" title="BREAKING · SIGNAL ≥ 80 · LIVE" showFilters={false} /> | |
| 31 | + </div> | |
| 15 | 32 | </> |
| 16 | 33 | ); |
| 17 | 34 | } |
| 35 | + | |
| 36 | +function ClusterSection({ title, tone, items, empty }: { title: string; tone: "hot" | "high" | "ok"; items: Cluster[]; empty: string }) { | |
| 37 | + const color = tone === "hot" ? "text-hot" : tone === "high" ? "text-high" : "text-ok"; | |
| 38 | + return ( | |
| 39 | + <Panel title={<span className={color}>{title} <span className="font-mono text-fg-subtle">{items.length}</span></span>} dense> | |
| 40 | + {items.length === 0 ? ( | |
| 41 | + <Empty>{empty}</Empty> | |
| 42 | + ) : ( | |
| 43 | + <ul className="divide-y divide-line"> | |
| 44 | + {items.map((c) => { | |
| 45 | + const e = c.event as EventItem | null | undefined; | |
| 46 | + return ( | |
| 47 | + <li key={c.id} className={`px-3 py-2.5 ${tone === "hot" ? "border-l-2 border-l-hot/70" : tone === "high" ? "border-l-2 border-l-high/60" : "border-l-2 border-l-ok/50"}`}> | |
| 48 | + <div className="flex flex-wrap items-center gap-x-2 gap-y-1 text-[11px] text-fg-subtle"> | |
| 49 | + <StateLabelText state={c.state} /> | |
| 50 | + {e?.source && <Link href={`/source/${e.source.id}`} className="font-mono uppercase text-fg-muted hover:text-fg">{e.source.name}</Link>} | |
| 51 | + {e?.country && <Flag code={e.country} />} | |
| 52 | + {e?.event_type && <Chip>{typeLabel(e.event_type)}</Chip>} | |
| 53 | + {e?.silent_change && <Badge kind="silent" compact />} | |
| 54 | + <span className="ml-auto font-mono tabular">{utcTime(c.last_at)} · {relTime(c.last_at)}</span> | |
| 55 | + </div> | |
| 56 | + <div className="mt-1 flex items-start gap-3"> | |
| 57 | + <Score value={e?.signal_score ?? c.max_importance} kind="signal" /> | |
| 58 | + <div className="min-w-0 flex-1"> | |
| 59 | + <Link href={e?.slug ? `/event/${e.slug}` : `/cluster/${c.slug ?? c.id}`} className="block font-medium leading-snug hover:underline">{c.title}</Link> | |
| 60 | + {e?.summary && <p className="mt-0.5 line-clamp-2 text-[12.5px] text-fg-muted">{e.summary}</p>} | |
| 61 | + <div className="mt-1 flex flex-wrap items-center gap-1.5 text-[11px] text-fg-subtle"> | |
| 62 | + <Link href={`/cluster/${c.slug ?? c.id}`} className="text-info hover:underline">{c.event_count} signal{c.event_count === 1 ? "" : "s"} · {c.source_count ?? 1} source{(c.source_count ?? 1) === 1 ? "" : "s"}</Link> | |
| 63 | + {(c.first_party_count ?? 0) > 0 && <Chip tone="signal">{c.first_party_count} first-party</Chip>} | |
| 64 | + {(c.external_count ?? 0) > 0 && <Chip>{c.external_count} external</Chip>} | |
| 65 | + {c.velocity !== undefined && c.velocity >= 40 && <Chip tone="high">velocity {Math.round(c.velocity)}</Chip>} | |
| 66 | + {c.lead_time_ms && c.lead_time_ms > 0 ? <Chip tone="ok" title="WebSensor detected the first-party signal this long before the first external report">lead {fmtOffset(c.lead_time_ms).replace("+", "")}</Chip> : null} | |
| 67 | + {c.sources && c.sources.length > 1 && <span className="min-w-0 max-w-full truncate">{c.sources.slice(0, 4).map((s) => s.name).join(" · ")}{c.sources.length > 4 ? ` +${c.sources.length - 4}` : ""}</span>} | |
| 68 | + </div> | |
| 69 | + </div> | |
| 70 | + </div> | |
| 71 | + </li> | |
| 72 | + ); | |
| 73 | + })} | |
| 74 | + </ul> | |
| 75 | + )} | |
| 76 | + </Panel> | |
| 77 | + ); | |
| 78 | +} | |
added
apps/web/src/app/category/[channel]/loading.tsx
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3 flex items-end justify-between gap-3"> | |
| 7 | + <div> | |
| 8 | + <Skeleton className="mb-1 h-2.5 w-24" /> | |
| 9 | + <Skeleton className="h-5 w-40" /> | |
| 10 | + <Skeleton className="mt-2 h-3 w-72" /> | |
| 11 | + </div> | |
| 12 | + <Skeleton className="h-9 w-40" /> | |
| 13 | + </div> | |
| 14 | + <div className="grid gap-4 xl:grid-cols-[1fr_340px]"> | |
| 15 | + <div className="panel"> | |
| 16 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-32" /></div> | |
| 17 | + <SkeletonRows rows={14} /> | |
| 18 | + </div> | |
| 19 | + <div className="flex flex-col gap-4"> | |
| 20 | + <SkeletonPanel lines={6} /> | |
| 21 | + <SkeletonPanel lines={6} /> | |
| 22 | + <SkeletonPanel lines={8} /> | |
| 23 | + <SkeletonPanel lines={6} /> | |
| 24 | + </div> | |
| 25 | + </div> | |
| 26 | + </> | |
| 27 | + ); | |
| 28 | +} | |
modified
apps/web/src/app/category/[channel]/page.tsx
+111 −10
@@ -1,29 +1,130 @@ | ||
| 1 | +import Link from "next/link"; | |
| 1 | 2 | import type { Metadata } from "next"; |
| 2 | 3 | import { notFound } from "next/navigation"; |
| 4 | +import { EventRow } from "@/components/event-row"; | |
| 3 | 5 | import { LiveFeed } from "@/components/live-feed"; |
| 4 | −import { PageHeader } from "@/components/ui"; | |
| 6 | +import { Badge, Bar, Chip, Empty, PageHeader, Panel, Score, Sparkline, Table, Td, TierBadge } from "@/components/ui"; | |
| 5 | 7 | import { api } from "@/lib/api"; |
| 6 | −import { CHANNELS } from "@/lib/format"; | |
| 8 | +import { CHANNELS, CHANNEL_KEYS, feedHref, fmtInt, typeLabel } from "@/lib/format"; | |
| 7 | 9 | |
| 8 | 10 | export const dynamic = "force-dynamic"; |
| 9 | 11 | |
| 10 | −const ALLOWED = new Set(["ai", "cyber", "finance", "health", "government", "science", "products", "infrastructure", "cloud", "developer", "consumer-tech", "semiconductors", "pharma", "statistics", "space", "automotive", "commerce", "payments", "crypto", "enterprise", "internet", "standards", "technology", "news", "media", "politics"]); | |
| 12 | +/** Categories that are valid desks even when they have no recent event. */ | |
| 13 | +const KNOWN = new Set<string>([...CHANNEL_KEYS, "cloud", "developer", "consumer-tech", "semiconductors", "pharma", "statistics", "space", "automotive", "commerce", "payments", "crypto", "enterprise", "internet", "standards", "technology", "media", "politics", "sports", "energy", "transport", "telecom", "education", "retail", "gaming", "web-policy", "packages", "api"]); | |
| 14 | + | |
| 15 | +function channelOf(channel: string) { | |
| 16 | + return CHANNELS.find((c) => c.key === channel && c.key !== "all" && c.key !== "breaking" && c.key !== "silent"); | |
| 17 | +} | |
| 11 | 18 | |
| 12 | 19 | export async function generateMetadata({ params }: { params: Promise<{ channel: string }> }): Promise<Metadata> { |
| 13 | 20 | const { channel } = await params; |
| 14 | − const label = CHANNELS.find((c) => c.key === channel)?.label ?? channel; | |
| 15 | − return { title: `${label} events`, description: `Live ${label} events detected by WebSensor.`, alternates: { canonical: `/category/${channel}` } }; | |
| 21 | + const label = channelOf(channel)?.label ?? channel; | |
| 22 | + return { title: `${label} desk`, description: `Live ${label} events detected by WebSensor: breaking, silent changes, active sources and trending entities.`, alternates: { canonical: `/category/${channel}` } }; | |
| 16 | 23 | } |
| 17 | 24 | |
| 25 | +/** Specialized real-time desk (spec §103). */ | |
| 18 | 26 | export default async function CategoryPage({ params }: { params: Promise<{ channel: string }> }) { |
| 19 | 27 | const { channel } = await params; |
| 20 | − if (!ALLOWED.has(channel)) notFound(); | |
| 21 | − const chan = CHANNELS.find((c) => c.key === channel); | |
| 22 | − const events = await api.events({ category: channel, limit: 60 }); | |
| 28 | + if (!/^[a-z0-9-]{2,32}$/.test(channel)) notFound(); | |
| 29 | + const desk = await api.categoryDesk(channel); | |
| 30 | + if (!desk || (desk.recent.length === 0 && !KNOWN.has(channel))) notFound(); | |
| 31 | + const chan = channelOf(channel); | |
| 32 | + const label = chan?.label ?? channel; | |
| 33 | + const series = desk.series.map((p) => p.n); | |
| 34 | + const total48 = series.reduce((a, b) => a + b, 0); | |
| 35 | + const maxType = Math.max(1, ...desk.by_type.map((t) => t.n)); | |
| 36 | + const maxSrc = Math.max(1, ...desk.active_sources.map((s) => s.events_24h)); | |
| 23 | 37 | return ( |
| 24 | 38 | <> |
| 25 | − <PageHeader kicker="Category" title={chan?.label ?? channel} description={`Meaningful changes from sources categorized as “${channel}”.`} /> | |
| 26 | − <LiveFeed initial={events.items} initialCursor={events.nextCursor} fixed={chan ? chan.key : undefined} title={(chan?.label ?? channel).toUpperCase()} extraQuery={chan ? undefined : { category: channel }} showTabs={false} /> | |
| 39 | + <PageHeader | |
| 40 | + compact | |
| 41 | + kicker={ | |
| 42 | + <span className="flex flex-wrap items-center gap-1.5"> | |
| 43 | + <span>Desk</span> | |
| 44 | + <span className="text-fg-subtle">·</span> | |
| 45 | + {desk.categories.map((c) => ( | |
| 46 | + <Chip key={c} href={`/category/${c}`} tone={c === channel ? "signal" : "default"}>{c}</Chip> | |
| 47 | + ))} | |
| 48 | + </span> | |
| 49 | + } | |
| 50 | + title={label} | |
| 51 | + description={`Meaningful changes from sources categorized as ${desk.categories.map((c) => `“${c}”`).join(", ")} — official pages, feeds, status pages, documentation, filings.`} | |
| 52 | + actions={ | |
| 53 | + <div className="flex items-center gap-3"> | |
| 54 | + <div className="text-right"> | |
| 55 | + <div className="label">48 h</div> | |
| 56 | + <div className="font-mono text-base font-semibold tabular">{fmtInt(total48)}</div> | |
| 57 | + </div> | |
| 58 | + <Sparkline values={series} width={160} height={36} tone="signal" /> | |
| 59 | + </div> | |
| 60 | + } | |
| 61 | + /> | |
| 62 | + <div className="grid gap-4 xl:grid-cols-[1fr_340px]"> | |
| 63 | + <div className="min-w-0"> | |
| 64 | + <LiveFeed initial={desk.recent} initialCursor={desk.nextCursor} fixed={chan ? chan.key : undefined} extraQuery={chan ? undefined : { category: channel }} showTabs={false} title={`${label.toUpperCase()} · LIVE`} /> | |
| 65 | + </div> | |
| 66 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 67 | + <Panel title={<span className="text-hot">Breaking · 24 h</span>} dense action={<Link href={feedHref({ category: channel, signal_min: 80 }, "/live")} className="text-[11px] text-fg-subtle hover:text-fg">live →</Link>}> | |
| 68 | + {desk.breaking.length ? desk.breaking.slice(0, 8).map((e) => <EventRow key={e.id} ev={e} />) : <Empty>Nothing is breaking in this desk right now.</Empty>} | |
| 69 | + </Panel> | |
| 70 | + <Panel title={<span className="text-silent">Silent</span>} dense action={<Link href={feedHref({ category: channel, silent_change: true }, "/live")} className="text-[11px] text-fg-subtle hover:text-fg">all →</Link>}> | |
| 71 | + {desk.silent.length ? desk.silent.slice(0, 8).map((e) => <EventRow key={e.id} ev={e} />) : <Empty>No silent change in this desk.</Empty>} | |
| 72 | + </Panel> | |
| 73 | + <Panel title="Active sources · 24 h" dense action={<Link href={`/sources?category=${channel}`} className="text-[11px] text-fg-subtle hover:text-fg">sources →</Link>}> | |
| 74 | + {desk.active_sources.length ? ( | |
| 75 | + <Table head={["Source", "Kind", "24 h", "", "Max"]}> | |
| 76 | + {desk.active_sources.map((s) => ( | |
| 77 | + <tr key={s.id} className="hover:bg-panel-2/60"> | |
| 78 | + <Td> | |
| 79 | + <span className="flex items-center gap-1.5"><TierBadge tier={s.tier} /><Link href={`/source/${s.id}`} className="truncate font-medium hover:underline">{s.name}</Link></span> | |
| 80 | + <div className="truncate font-mono text-[10.5px] text-fg-subtle">{s.domain}</div> | |
| 81 | + </Td> | |
| 82 | + <Td>{s.first_party === false ? <Badge kind="external" compact /> : <Badge kind="first-party" compact />}</Td> | |
| 83 | + <Td mono>{fmtInt(s.events_24h)}</Td> | |
| 84 | + <Td><div className="w-12"><Bar value={s.events_24h} max={maxSrc} /></div></Td> | |
| 85 | + <Td><Score value={s.max_importance} size="sm" /></Td> | |
| 86 | + </tr> | |
| 87 | + ))} | |
| 88 | + </Table> | |
| 89 | + ) : ( | |
| 90 | + <Empty>No source produced an event in this desk in the last 24 h.</Empty> | |
| 91 | + )} | |
| 92 | + </Panel> | |
| 93 | + <Panel title="Trending entities · 24 h" dense action={<Link href="/explore?tab=trending" className="text-[11px] text-fg-subtle hover:text-fg">all →</Link>}> | |
| 94 | + {desk.trending_entities.length ? ( | |
| 95 | + <ol className="divide-y divide-line"> | |
| 96 | + {desk.trending_entities.map((t, i) => ( | |
| 97 | + <li key={t.id} className="grid grid-cols-[1.25rem_1fr_auto] items-center gap-2 px-3 py-1.5 text-[13px]"> | |
| 98 | + <span className="font-mono text-[11px] text-fg-subtle tabular">{i + 1}</span> | |
| 99 | + <div className="min-w-0"> | |
| 100 | + <Link href={`/entity/${t.id}`} className="block truncate font-medium hover:underline">{t.name}</Link> | |
| 101 | + <div className="truncate text-[11px] text-fg-subtle">{t.type.replace(/_/g, " ")} · {t.sources} source{t.sources === 1 ? "" : "s"}</div> | |
| 102 | + </div> | |
| 103 | + <span className="font-mono text-[12px] font-semibold tabular">{fmtInt(t.events_24h)}</span> | |
| 104 | + </li> | |
| 105 | + ))} | |
| 106 | + </ol> | |
| 107 | + ) : ( | |
| 108 | + <Empty>No entity is trending in this desk.</Empty> | |
| 109 | + )} | |
| 110 | + </Panel> | |
| 111 | + <Panel title="By type · 7 d" dense> | |
| 112 | + {desk.by_type.length ? ( | |
| 113 | + <ul className="divide-y divide-line"> | |
| 114 | + {desk.by_type.map((t) => ( | |
| 115 | + <li key={t.event_type} className="grid grid-cols-[8.5rem_1fr_3rem] items-center gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 116 | + <Link href={feedHref({ category: channel, event_type: t.event_type }, "/live")} className="truncate hover:underline">{typeLabel(t.event_type)}</Link> | |
| 117 | + <Bar value={t.n} max={maxType} tone="info" /> | |
| 118 | + <span className="text-right font-mono text-fg-subtle tabular">{fmtInt(t.n)}</span> | |
| 119 | + </li> | |
| 120 | + ))} | |
| 121 | + </ul> | |
| 122 | + ) : ( | |
| 123 | + <Empty /> | |
| 124 | + )} | |
| 125 | + </Panel> | |
| 126 | + </aside> | |
| 127 | + </div> | |
| 27 | 128 | </> |
| 28 | 129 | ); |
| 29 | 130 | } |
added
apps/web/src/app/cluster/[slug]/layout.tsx
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +import { notFound } from "next/navigation"; | |
| 2 | +import type { ReactNode } from "react"; | |
| 3 | +import { api } from "@/lib/api"; | |
| 4 | + | |
| 5 | +/** Existence check outside the `loading.tsx` boundary so unknown clusters answer with a real 404 (see entity/[id]/layout.tsx). */ | |
| 6 | +export default async function ClusterLayout({ params, children }: { params: Promise<{ slug: string }>; children: ReactNode }) { | |
| 7 | + const { slug } = await params; | |
| 8 | + const d = await api.cluster(slug); | |
| 9 | + if (!d) notFound(); | |
| 10 | + return children; | |
| 11 | +} | |
added
apps/web/src/app/cluster/[slug]/loading.tsx
+50 −0
@@ -0,0 +1,50 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3 flex flex-wrap items-end justify-between gap-3"> | |
| 7 | + <div className="flex min-w-0 flex-col gap-2"> | |
| 8 | + <div className="flex items-center gap-2"> | |
| 9 | + <Skeleton className="h-4 w-20" /> | |
| 10 | + <Skeleton className="h-4 w-24" /> | |
| 11 | + <Skeleton className="h-4 w-16" /> | |
| 12 | + </div> | |
| 13 | + <Skeleton className="h-6 w-[28rem] max-w-full" /> | |
| 14 | + <Skeleton className="h-3 w-80 max-w-full" /> | |
| 15 | + </div> | |
| 16 | + <div className="flex gap-2"> | |
| 17 | + <Skeleton className="h-7 w-20" /> | |
| 18 | + <Skeleton className="h-7 w-28" /> | |
| 19 | + </div> | |
| 20 | + </div> | |
| 21 | + <Skeleton className="mb-3 h-3 w-96 max-w-full" /> | |
| 22 | + <div className="mb-4 grid grid-cols-2 gap-px overflow-hidden rounded-lg border border-line bg-line sm:grid-cols-4"> | |
| 23 | + {Array.from({ length: 4 }, (_, i) => ( | |
| 24 | + <div key={i} className="flex flex-col gap-1.5 bg-panel px-3 py-2"> | |
| 25 | + <Skeleton className="h-2.5 w-16" /> | |
| 26 | + <Skeleton className="h-5 w-12" /> | |
| 27 | + <Skeleton className="h-2.5 w-24" /> | |
| 28 | + </div> | |
| 29 | + ))} | |
| 30 | + </div> | |
| 31 | + <div className="grid gap-4 xl:grid-cols-[1fr_340px]"> | |
| 32 | + <div className="flex flex-col gap-4"> | |
| 33 | + <div className="panel"> | |
| 34 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-40" /></div> | |
| 35 | + <SkeletonRows rows={6} /> | |
| 36 | + </div> | |
| 37 | + <div className="panel"> | |
| 38 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-20" /></div> | |
| 39 | + <SkeletonRows rows={6} /> | |
| 40 | + </div> | |
| 41 | + </div> | |
| 42 | + <aside className="flex flex-col gap-4"> | |
| 43 | + <SkeletonPanel lines={2} /> | |
| 44 | + <SkeletonPanel lines={6} /> | |
| 45 | + <SkeletonPanel lines={4} /> | |
| 46 | + </aside> | |
| 47 | + </div> | |
| 48 | + </> | |
| 49 | + ); | |
| 50 | +} | |
added
apps/web/src/app/cluster/[slug]/page.tsx
+240 −0
@@ -0,0 +1,240 @@ | ||
| 1 | +import Link from "next/link"; | |
| 2 | +import type { Metadata } from "next"; | |
| 3 | +import { notFound } from "next/navigation"; | |
| 4 | +import { EventRow } from "@/components/event-row"; | |
| 5 | +import { Badge, Chip, Empty, Flag, PageHeader, Panel, Score, Sparkline, StateBadge, Stat, TypeChip } from "@/components/ui"; | |
| 6 | +import { ShareButton } from "@/components/watch-button"; | |
| 7 | +import { api, SITE_URL, type ClusterDetail, type EventItem } from "@/lib/api"; | |
| 8 | +import { fmtInt, fmtOffset, fmtScore, relTime, typeLabel, utcDate, utcDateTime, utcTime } from "@/lib/format"; | |
| 9 | + | |
| 10 | +export const dynamic = "force-dynamic"; | |
| 11 | + | |
| 12 | +function num(v: number | string | null | undefined): number { | |
| 13 | + if (v === null || v === undefined) return 0; | |
| 14 | + const n = typeof v === "string" ? Number(v) : v; | |
| 15 | + return Number.isFinite(n) ? n : 0; | |
| 16 | +} | |
| 17 | + | |
| 18 | +export async function generateMetadata({ params }: { params: Promise<{ slug: string }> }): Promise<Metadata> { | |
| 19 | + const { slug } = await params; | |
| 20 | + const d = await api.cluster(slug); | |
| 21 | + if (!d) return { title: "Cluster not found" }; | |
| 22 | + const c = d.cluster; | |
| 23 | + const desc = (c.summary?.trim() || `${c.event_count} signal${c.event_count === 1 ? "" : "s"} from ${c.source_count ?? 1} source${(c.source_count ?? 1) === 1 ? "" : "s"} — ${d.first_party_signals} first-party, ${d.external_signals} external. Propagation timeline with WebSensor lead time.`).slice(0, 200); | |
| 24 | + const path = `/cluster/${c.slug ?? c.id}`; | |
| 25 | + return { | |
| 26 | + title: c.title, | |
| 27 | + description: desc, | |
| 28 | + alternates: { canonical: path }, | |
| 29 | + openGraph: { type: "article", title: c.title, description: desc, url: `${SITE_URL}${path}`, publishedTime: new Date(c.first_at).toISOString(), modifiedTime: new Date(c.last_at).toISOString(), tags: c.categories }, | |
| 30 | + twitter: { card: "summary", title: c.title, description: desc }, | |
| 31 | + }; | |
| 32 | +} | |
| 33 | + | |
| 34 | +/** Signals per hour (or per day when the story spans more than 3 days) — computed from the events. */ | |
| 35 | +function series(events: EventItem[], first: number, last: number): { values: number[]; unit: "hour" | "day" } { | |
| 36 | + const span = Math.max(0, last - first); | |
| 37 | + const unit: "hour" | "day" = span > 72 * 3600e3 ? "day" : "hour"; | |
| 38 | + const step = unit === "hour" ? 3600e3 : 86400e3; | |
| 39 | + const start = Math.floor(first / step) * step; | |
| 40 | + const n = Math.min(120, Math.floor((last - start) / step) + 1); | |
| 41 | + const values = new Array<number>(Math.max(1, n)).fill(0); | |
| 42 | + for (const e of events) { | |
| 43 | + const i = Math.floor((new Date(e.detected_at).getTime() - start) / step); | |
| 44 | + if (i >= 0 && i < values.length) values[i]! += 1; | |
| 45 | + } | |
| 46 | + return { values, unit }; | |
| 47 | +} | |
| 48 | + | |
| 49 | +export default async function ClusterPage({ params }: { params: Promise<{ slug: string }> }) { | |
| 50 | + const { slug } = await params; | |
| 51 | + const d = await api.cluster(slug); | |
| 52 | + if (!d) notFound(); | |
| 53 | + const c = d.cluster; | |
| 54 | + const path = `/cluster/${c.slug ?? c.id}`; | |
| 55 | + const lead = num(d.lead_time_ms ?? c.lead_time_ms); | |
| 56 | + const velocity = num(c.velocity); | |
| 57 | + const firstMs = new Date(c.first_at).getTime(); | |
| 58 | + const lastMs = new Date(c.last_at).getTime(); | |
| 59 | + const spanMs = Math.max(0, lastMs - firstMs); | |
| 60 | + const sources = sourcesOf(d); | |
| 61 | + const countryOf = (p: ClusterDetail["propagation"][number]): string | null => (p.source as { country?: string | null }).country ?? null; | |
| 62 | + const spark = series(d.events, firstMs, lastMs); | |
| 63 | + const primary = d.events.find((e) => e.id === c.primary_event_id) ?? d.events[0]; | |
| 64 | + const jsonLd = { | |
| 65 | + "@context": "https://schema.org", | |
| 66 | + "@type": "NewsArticle", | |
| 67 | + headline: c.title, | |
| 68 | + description: c.summary ?? undefined, | |
| 69 | + datePublished: new Date(c.first_at).toISOString(), | |
| 70 | + dateModified: new Date(c.last_at).toISOString(), | |
| 71 | + url: `${SITE_URL}${path}`, | |
| 72 | + mainEntityOfPage: `${SITE_URL}${path}`, | |
| 73 | + author: { "@type": "Organization", name: "WebSensor", url: SITE_URL }, | |
| 74 | + publisher: { "@type": "Organization", name: "WebSensor", url: SITE_URL }, | |
| 75 | + about: d.entities.slice(0, 12).map((x) => ({ "@type": "Thing", name: x.name, url: `${SITE_URL}/entity/${x.id}` })), | |
| 76 | + keywords: c.categories.join(", "), | |
| 77 | + ...(primary?.url ? { isBasedOn: primary.url } : {}), | |
| 78 | + }; | |
| 79 | + | |
| 80 | + return ( | |
| 81 | + <> | |
| 82 | + <script type="application/ld+json" dangerouslySetInnerHTML={{ __html: JSON.stringify(jsonLd) }} /> | |
| 83 | + <PageHeader | |
| 84 | + compact | |
| 85 | + kicker={ | |
| 86 | + <span className="flex flex-wrap items-center gap-x-2 gap-y-1"> | |
| 87 | + <StateBadge state={c.state} /> | |
| 88 | + <span className="font-mono text-fg-subtle">CLUSTER</span> | |
| 89 | + {d.entities.slice(0, 6).map((x) => ( | |
| 90 | + <Chip key={x.id} href={`/entity/${x.id}`} title={typeLabel(x.type)}>{x.name}</Chip> | |
| 91 | + ))} | |
| 92 | + {d.entities.length > 6 && <span className="font-mono text-fg-subtle tabular">+{d.entities.length - 6}</span>} | |
| 93 | + {c.categories.slice(0, 3).map((k) => <Chip key={k} href={`/category/${k}`} className="!text-fg-subtle">{k}</Chip>)} | |
| 94 | + </span> | |
| 95 | + } | |
| 96 | + title={c.title} | |
| 97 | + description={c.summary?.trim() ? <span className="whitespace-pre-line">{c.summary.trim()}</span> : undefined} | |
| 98 | + actions={ | |
| 99 | + <> | |
| 100 | + <ShareButton path={path} /> | |
| 101 | + {primary && <Link href={`/event/${primary.slug}`} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">Primary event →</Link>} | |
| 102 | + </> | |
| 103 | + } | |
| 104 | + /> | |
| 105 | + | |
| 106 | + <div className="mb-3 flex flex-wrap items-center gap-x-3 gap-y-1 text-[12.5px] text-fg-muted"> | |
| 107 | + <span><span className="font-mono font-semibold text-fg tabular">{fmtInt(c.event_count)}</span> signal{c.event_count === 1 ? "" : "s"}</span> | |
| 108 | + <span className="text-fg-subtle">·</span> | |
| 109 | + <span><span className="font-mono font-semibold text-signal tabular">{fmtInt(d.first_party_signals)}</span> first-party</span> | |
| 110 | + <span className="text-fg-subtle">·</span> | |
| 111 | + <span><span className="font-mono font-semibold text-fg tabular">{fmtInt(d.external_signals)}</span> external</span> | |
| 112 | + <span className="text-fg-subtle">·</span> | |
| 113 | + <span><span className="font-mono font-semibold text-fg tabular">{fmtInt(c.source_count ?? sources.length)}</span> source{(c.source_count ?? sources.length) === 1 ? "" : "s"}</span> | |
| 114 | + <span className="text-fg-subtle">·</span> | |
| 115 | + <span className="font-mono text-[11.5px] text-fg-subtle tabular" title={`${utcDateTime(c.first_at)} → ${utcDateTime(c.last_at)}`}>{utcDateTime(c.first_at).replace(" UTC", "")} → {utcDate(c.last_at) === utcDate(c.first_at) ? utcTime(c.last_at) : utcDateTime(c.last_at).replace(" UTC", "")} UTC</span> | |
| 116 | + <span className="text-fg-subtle">·</span> | |
| 117 | + <span className="text-fg-subtle">updated {relTime(c.last_at)}</span> | |
| 118 | + </div> | |
| 119 | + | |
| 120 | + {lead > 0 && ( | |
| 121 | + <div className="mb-3 flex flex-wrap items-center gap-x-3 gap-y-1 border-l-2 border-l-signal bg-signal-soft/50 px-3 py-2 text-[12.5px]" role="note"> | |
| 122 | + <span className="font-mono text-[10.5px] font-semibold tracking-wider text-signal">WEBSENSOR LEAD TIME</span> | |
| 123 | + <span className="text-fg"> | |
| 124 | + detected <span className="font-mono font-semibold text-signal tabular">{fmtOffset(lead).replace("+", "")}</span> before the first external report | |
| 125 | + </span> | |
| 126 | + {c.first_party_at && c.first_external_at && ( | |
| 127 | + <span className="font-mono text-[11px] text-fg-subtle tabular">first-party {utcTime(c.first_party_at, false)} · external {utcTime(c.first_external_at, false)} UTC</span> | |
| 128 | + )} | |
| 129 | + </div> | |
| 130 | + )} | |
| 131 | + | |
| 132 | + <div className="mb-4 grid grid-cols-2 gap-px overflow-hidden rounded-lg border border-line bg-line sm:grid-cols-4"> | |
| 133 | + <div className="bg-panel"><Stat label="Peak signal" value={fmtScore(Math.max(c.max_importance, ...d.events.map((e) => e.signal_score ?? 0)))} hint={`importance ${fmtScore(c.max_importance)}`} tone={c.max_importance >= 80 ? "hot" : undefined} /></div> | |
| 134 | + <div className="bg-panel"><Stat label="Velocity" value={velocity > 0 ? fmtScore(velocity) : "0"} hint={velocity >= 40 ? "signals accumulating fast" : "signals per unit of time"} tone={velocity >= 40 ? "warn" : undefined} /></div> | |
| 135 | + <div className="bg-panel"><Stat label="Span" value={fmtOffset(spanMs).replace("+", "")} hint={`${utcTime(c.first_at, false)} → ${utcTime(c.last_at, false)} UTC`} /></div> | |
| 136 | + <div className="bg-panel"><Stat label="Lead time" value={lead > 0 ? fmtOffset(lead).replace("+", "") : "—"} hint={lead > 0 ? "first-party before external" : d.external_signals === 0 ? "no external report yet" : lead < 0 ? `external first by ${fmtOffset(-lead).replace("+", "")}` : "n/a"} tone={lead > 0 ? "signal" : undefined} /></div> | |
| 137 | + </div> | |
| 138 | + | |
| 139 | + <div className="grid gap-4 xl:grid-cols-[1fr_340px]"> | |
| 140 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 141 | + <Panel title={<>Web velocity · propagation <span className="font-mono text-fg-subtle">{d.propagation.length}</span></>} dense action={<span className="hidden items-center gap-3 font-mono text-[10.5px] text-fg-subtle sm:inline-flex"><span className="inline-flex items-center gap-1"><span className="inline-block h-3 w-0.5 bg-signal" /> first-party</span><span className="inline-flex items-center gap-1"><span className="inline-block h-3 w-0.5 bg-line-strong" /> external</span></span>}> | |
| 142 | + {d.propagation.length === 0 ? ( | |
| 143 | + <Empty>No propagation recorded yet — the cluster has a single signal.</Empty> | |
| 144 | + ) : ( | |
| 145 | + <ol className="divide-y divide-line"> | |
| 146 | + {d.propagation.map((p, i) => ( | |
| 147 | + <li key={p.id} className={`grid grid-cols-[4.25rem_1fr] gap-x-3 border-l-2 px-3 py-2 sm:grid-cols-[4.25rem_1fr_auto] ${p.first_party ? "border-l-signal" : "border-l-line-strong"}`}> | |
| 148 | + <div className="flex flex-col font-mono text-[12px] leading-4 tabular"> | |
| 149 | + <span className={`font-semibold ${i === 0 ? "text-fg" : p.first_party ? "text-signal" : "text-fg-muted"}`}>{i === 0 || p.offset_ms === 0 ? "00:00" : fmtOffset(p.offset_ms)}</span> | |
| 150 | + <time dateTime={p.at} title={utcDateTime(p.at)} className="text-[10.5px] text-fg-subtle">{utcTime(p.at)}</time> | |
| 151 | + </div> | |
| 152 | + <div className="min-w-0 sm:col-start-2"> | |
| 153 | + <div className="flex flex-wrap items-center gap-x-2 gap-y-0.5 text-[11px]"> | |
| 154 | + <Link href={`/source/${p.source.id}`} className="font-mono font-semibold uppercase tracking-wide text-fg-muted hover:text-fg">{p.source.name}</Link> | |
| 155 | + {countryOf(p) && <Flag code={countryOf(p)} />} | |
| 156 | + <span className="sm:hidden"><Score value={p.importance} size="sm" /></span> | |
| 157 | + <Badge kind={p.first_party ? "first-party" : "external"} compact /> | |
| 158 | + {p.sensor?.type && <Chip className="font-mono" title={p.sensor.name}>{p.sensor.type}</Chip>} | |
| 159 | + <TypeChip type={p.event_type} /> | |
| 160 | + </div> | |
| 161 | + <Link href={`/event/${p.slug}`} className="mt-0.5 block font-medium leading-snug text-fg hover:underline">{p.title}</Link> | |
| 162 | + </div> | |
| 163 | + <div className="hidden items-start pt-0.5 sm:flex"><Score value={p.importance} /></div> | |
| 164 | + </li> | |
| 165 | + ))} | |
| 166 | + </ol> | |
| 167 | + )} | |
| 168 | + </Panel> | |
| 169 | + <Panel title={<>Signals <span className="font-mono text-fg-subtle">{d.events.length}</span></>} dense action={primary && <Link href={`/event/${primary.slug}`} className="text-[11px] text-fg-subtle hover:text-fg">primary event →</Link>}> | |
| 170 | + {d.events.length ? d.events.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>No signal in this cluster.</Empty>} | |
| 171 | + </Panel> | |
| 172 | + </div> | |
| 173 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 174 | + <Panel title={spark.unit === "hour" ? "Signals per hour" : "Signals per day"}> | |
| 175 | + <Sparkline values={spark.values} width={300} height={44} tone={c.state === "breaking" ? "hot" : "signal"} /> | |
| 176 | + <div className="mt-1 flex justify-between font-mono text-[10.5px] text-fg-subtle tabular"> | |
| 177 | + <span>{utcDate(c.first_at)} {utcTime(c.first_at, false)}</span> | |
| 178 | + <span>peak {Math.max(0, ...spark.values)}/{spark.unit === "hour" ? "h" : "d"}</span> | |
| 179 | + <span>{utcDate(c.last_at)} {utcTime(c.last_at, false)}</span> | |
| 180 | + </div> | |
| 181 | + </Panel> | |
| 182 | + <Panel title={<>Entities <span className="font-mono text-fg-subtle">{d.entities.length}</span></>} dense> | |
| 183 | + {d.entities.length === 0 ? ( | |
| 184 | + <Empty>No entity resolved for this cluster.</Empty> | |
| 185 | + ) : ( | |
| 186 | + <ul className="divide-y divide-line"> | |
| 187 | + {d.entities.slice(0, 14).map((x) => ( | |
| 188 | + <li key={x.id} className="flex items-center justify-between gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 189 | + <Link href={`/entity/${x.id}`} className="min-w-0 truncate font-medium hover:underline">{x.name}</Link> | |
| 190 | + <span className="flex shrink-0 items-center gap-1.5"><Chip>{typeLabel(x.type)}</Chip><Score value={x.importance} size="sm" /></span> | |
| 191 | + </li> | |
| 192 | + ))} | |
| 193 | + {d.entities.length > 14 && <li className="px-3 py-1.5 text-[11px] text-fg-subtle">+{d.entities.length - 14} more entities</li>} | |
| 194 | + </ul> | |
| 195 | + )} | |
| 196 | + </Panel> | |
| 197 | + <Panel title={<>Sources <span className="font-mono text-fg-subtle">{sources.length}</span></>} dense> | |
| 198 | + {sources.length === 0 ? ( | |
| 199 | + <Empty>No source.</Empty> | |
| 200 | + ) : ( | |
| 201 | + <ul className="divide-y divide-line"> | |
| 202 | + {sources.map((s) => ( | |
| 203 | + <li key={s.id} className="grid grid-cols-[1fr_auto_2rem] items-center gap-x-2 px-3 py-1.5 text-[12.5px]"> | |
| 204 | + <div className="min-w-0"> | |
| 205 | + <Link href={`/source/${s.id}`} className="block truncate font-medium hover:underline">{s.name}</Link> | |
| 206 | + <div className="flex items-center gap-1.5 font-mono text-[10.5px] text-fg-subtle"><span className="truncate">{s.domain}</span>{s.country && <Flag code={s.country} />}</div> | |
| 207 | + </div> | |
| 208 | + <Badge kind={s.first_party ? "first-party" : "external"} compact /> | |
| 209 | + <span className="text-right font-mono text-[12px] tabular text-fg-muted" title="Signals from this source">{s.n}</span> | |
| 210 | + </li> | |
| 211 | + ))} | |
| 212 | + </ul> | |
| 213 | + )} | |
| 214 | + </Panel> | |
| 215 | + <Panel title="Share"> | |
| 216 | + <div className="flex flex-wrap items-center gap-2"> | |
| 217 | + <ShareButton path={path} /> | |
| 218 | + <Link href={`/api/v1/clusters/${encodeURIComponent(c.slug ?? c.id)}`} className="rounded-md border border-line bg-panel px-2.5 py-1 font-mono text-[11.5px] hover:border-line-strong">JSON</Link> | |
| 219 | + </div> | |
| 220 | + <p className="mt-2 break-all font-mono text-[10.5px] text-fg-subtle">{SITE_URL}{path}</p> | |
| 221 | + </Panel> | |
| 222 | + </aside> | |
| 223 | + </div> | |
| 224 | + </> | |
| 225 | + ); | |
| 226 | +} | |
| 227 | + | |
| 228 | +function sourcesOf(d: ClusterDetail): { id: string; name: string; domain: string; first_party: boolean; country: string | null; n: number }[] { | |
| 229 | + const m = new Map<string, { id: string; name: string; domain: string; first_party: boolean; country: string | null; n: number }>(); | |
| 230 | + for (const e of d.events) { | |
| 231 | + const id = e.source?.id ?? e.source_id; | |
| 232 | + const cur = m.get(id); | |
| 233 | + if (cur) cur.n += 1; | |
| 234 | + else m.set(id, { id, name: e.source?.name ?? id, domain: e.source?.domain ?? "", first_party: e.first_party !== false, country: e.country ?? null, n: 1 }); | |
| 235 | + } | |
| 236 | + for (const p of d.propagation) { | |
| 237 | + if (!m.has(p.source.id)) m.set(p.source.id, { id: p.source.id, name: p.source.name, domain: p.source.domain, first_party: p.first_party, country: (p.source as { country?: string | null }).country ?? null, n: 1 }); | |
| 238 | + } | |
| 239 | + return [...m.values()].sort((a, b) => Number(b.first_party) - Number(a.first_party) || b.n - a.n); | |
| 240 | +} | |
modified
apps/web/src/app/company/[id]/page.tsx
+10 −106
@@ -1,111 +1,15 @@ | ||
| 1 | −import Link from "next/link"; | |
| 2 | −import type { Metadata } from "next"; | |
| 3 | −import { notFound } from "next/navigation"; | |
| 4 | −import { Timeline } from "@/components/timeline"; | |
| 5 | −import { Bar, Chip, Empty, ExtLink, PageHeader, Panel, Stat } from "@/components/ui"; | |
| 6 | −import { WatchButton } from "@/components/watch-button"; | |
| 7 | −import { api } from "@/lib/api"; | |
| 8 | −import { fmtInt, fmtScore, relTime, typeLabel } from "@/lib/format"; | |
| 1 | +import { permanentRedirect } from "next/navigation"; | |
| 9 | 2 | |
| 10 | 3 | export const dynamic = "force-dynamic"; |
| 11 | 4 | |
| 12 | −export async function generateMetadata({ params }: { params: Promise<{ id: string }> }): Promise<Metadata> { | |
| 5 | +/** `/company/:id` is the legacy entity URL — permanently redirected to `/entity/:id` (tab preserved). */ | |
| 6 | +export default async function CompanyPage({ params, searchParams }: { params: Promise<{ id: string }>; searchParams: Promise<{ tab?: string; range?: string; cursor?: string }> }) { | |
| 13 | 7 | const { id } = await params; |
| 14 | − const d = await api.entity(id); | |
| 15 | − if (!d) return { title: "Entity not found" }; | |
| 16 | − return { title: `${d.entity.name} — timeline`, description: d.entity.description ?? `Every meaningful change detected for ${d.entity.name}: pricing, products, documentation, incidents, silent changes.`, alternates: { canonical: `/company/${d.entity.id}` } }; | |
| 17 | −} | |
| 18 | − | |
| 19 | −export default async function CompanyPage({ params }: { params: Promise<{ id: string }> }) { | |
| 20 | − const { id } = await params; | |
| 21 | − const d = await api.entity(id); | |
| 22 | − if (!d) notFound(); | |
| 23 | − const e = d.entity; | |
| 24 | − const maxType = Math.max(1, ...d.by_type.map((t) => t.n)); | |
| 25 | − return ( | |
| 26 | − <> | |
| 27 | − <PageHeader | |
| 28 | − kicker={<span className="flex items-center gap-2"><Chip>{e.type}</Chip>{e.domain && <Link href={`/domain/${e.domain}`} className="font-mono hover:underline">{e.domain}</Link>}</span>} | |
| 29 | − title={e.name} | |
| 30 | − description={e.description} | |
| 31 | − actions={ | |
| 32 | − <> | |
| 33 | − <WatchButton kind="entity" value={e.id} /> | |
| 34 | − <Link href={`/timeline/${e.id}`} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">Full timeline →</Link> | |
| 35 | − {e.homepage && <ExtLink href={e.homepage} className="text-[12px]">Website ↗</ExtLink>} | |
| 36 | − </> | |
| 37 | − } | |
| 38 | − /> | |
| 39 | − <div className="panel mb-4 grid grid-cols-2 divide-x divide-line sm:grid-cols-4"> | |
| 40 | − <Stat label="Events" value={fmtInt(e.event_count)} hint={e.last_event_at ? `last ${relTime(e.last_event_at)}` : undefined} /> | |
| 41 | − <Stat label="Importance" value={fmtScore(e.importance)} hint="entity weight" /> | |
| 42 | − <Stat label="Sources" value={fmtInt(d.sources.length)} /> | |
| 43 | − <Stat label="Silent" value={fmtInt(d.recent.filter((x) => x.silent_change).length)} hint="in recent 30" /> | |
| 44 | − </div> | |
| 45 | − <div className="grid gap-4 lg:grid-cols-[1fr_320px]"> | |
| 46 | − <Panel title="Timeline" dense action={<Link href={`/timeline/${e.id}`} className="text-[11px] text-fg-subtle hover:text-fg">all →</Link>}> | |
| 47 | − <Timeline events={d.recent} /> | |
| 48 | − </Panel> | |
| 49 | − <aside className="flex flex-col gap-4"> | |
| 50 | − {d.children.length > 0 && ( | |
| 51 | − <Panel title="Products & children" dense> | |
| 52 | − <ul className="divide-y divide-line"> | |
| 53 | − {d.children.map((c) => ( | |
| 54 | − <li key={c.id} className="flex items-center justify-between px-3 py-1.5 text-[13px]"> | |
| 55 | − <Link href={`/company/${c.id}`} className="hover:underline">{c.name}</Link> | |
| 56 | − <span className="font-mono text-[11px] text-fg-subtle">{c.type} · {c.event_count}</span> | |
| 57 | − </li> | |
| 58 | − ))} | |
| 59 | − </ul> | |
| 60 | − </Panel> | |
| 61 | − )} | |
| 62 | − <Panel title="Event types"> | |
| 63 | − {d.by_type.length ? ( | |
| 64 | − <ul className="space-y-1.5"> | |
| 65 | − {d.by_type.slice(0, 12).map((t) => ( | |
| 66 | − <li key={t.event_type} className="grid grid-cols-[8rem_1fr_2.5rem] items-center gap-2 text-[12px]"> | |
| 67 | − <span className="truncate">{typeLabel(t.event_type)}</span> | |
| 68 | − <Bar value={t.n} max={maxType} tone="info" /> | |
| 69 | − <span className="text-right font-mono text-fg-subtle tabular">{t.n}</span> | |
| 70 | − </li> | |
| 71 | − ))} | |
| 72 | − </ul> | |
| 73 | − ) : ( | |
| 74 | − <Empty>No events yet.</Empty> | |
| 75 | − )} | |
| 76 | − </Panel> | |
| 77 | − <Panel title="Sources" dense> | |
| 78 | − {d.sources.length ? ( | |
| 79 | − <ul className="divide-y divide-line"> | |
| 80 | − {d.sources.map((s) => ( | |
| 81 | − <li key={s.id} className="flex items-center justify-between px-3 py-1.5 text-[13px]"> | |
| 82 | − <Link href={`/source/${s.id}`} className="hover:underline">{s.name}</Link> | |
| 83 | − <span className="font-mono text-[11px] text-fg-subtle">{s.domain}</span> | |
| 84 | − </li> | |
| 85 | − ))} | |
| 86 | − </ul> | |
| 87 | − ) : ( | |
| 88 | − <Empty>No monitored source linked.</Empty> | |
| 89 | − )} | |
| 90 | − </Panel> | |
| 91 | − {d.relations.length > 0 && ( | |
| 92 | − <Panel title="Knowledge graph" dense> | |
| 93 | − <ul className="divide-y divide-line"> | |
| 94 | − {d.relations.map((r, i) => ( | |
| 95 | − <li key={i} className="px-3 py-1.5 text-[12.5px]"> | |
| 96 | − <Link href={`/company/${r.from_id}`} className="hover:underline">{r.from_name}</Link> <span className="font-mono text-[11px] text-fg-subtle">→ {r.relation} →</span> <Link href={`/company/${r.to_id}`} className="hover:underline">{r.to_name}</Link> | |
| 97 | − </li> | |
| 98 | − ))} | |
| 99 | − </ul> | |
| 100 | − </Panel> | |
| 101 | − )} | |
| 102 | − {d.aliases.length > 0 && ( | |
| 103 | − <Panel title="Aliases"> | |
| 104 | − <div className="flex flex-wrap gap-1">{d.aliases.map((a) => <Chip key={a}>{a}</Chip>)}</div> | |
| 105 | − </Panel> | |
| 106 | − )} | |
| 107 | − </aside> | |
| 108 | − </div> | |
| 109 | − </> | |
| 110 | − ); | |
| 8 | + const sp = await searchParams; | |
| 9 | + const q = new URLSearchParams(); | |
| 10 | + if (sp.tab) q.set("tab", sp.tab); | |
| 11 | + if (sp.range) q.set("range", sp.range); | |
| 12 | + if (sp.cursor) q.set("cursor", sp.cursor); | |
| 13 | + const s = q.toString(); | |
| 14 | + permanentRedirect(`/entity/${encodeURIComponent(id)}${s ? `?${s}` : ""}`); | |
| 111 | 15 | } |
added
apps/web/src/app/country/[slug]/loading.tsx
+42 −0
@@ -0,0 +1,42 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3"> | |
| 7 | + <Skeleton className="mb-1 h-2.5 w-24" /> | |
| 8 | + <Skeleton className="h-5 w-48" /> | |
| 9 | + <Skeleton className="mt-2 h-3 w-full max-w-2xl" /> | |
| 10 | + </div> | |
| 11 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-line sm:grid-cols-4"> | |
| 12 | + {Array.from({ length: 4 }, (_, i) => ( | |
| 13 | + <div key={i} className="flex flex-col gap-1.5 px-3 py-2"> | |
| 14 | + <Skeleton className="h-2.5 w-16" /> | |
| 15 | + <Skeleton className="h-5 w-12" /> | |
| 16 | + <Skeleton className="h-2.5 w-20" /> | |
| 17 | + </div> | |
| 18 | + ))} | |
| 19 | + </div> | |
| 20 | + <div className="grid gap-4 xl:grid-cols-[1fr_360px]"> | |
| 21 | + <div className="flex flex-col gap-4"> | |
| 22 | + <div className="panel"> | |
| 23 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-32" /></div> | |
| 24 | + <SkeletonRows rows={5} /> | |
| 25 | + </div> | |
| 26 | + <div className="grid gap-4 md:grid-cols-2"> | |
| 27 | + {Array.from({ length: 4 }, (_, i) => ( | |
| 28 | + <div key={i} className="panel"> | |
| 29 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-24" /></div> | |
| 30 | + <SkeletonRows rows={4} /> | |
| 31 | + </div> | |
| 32 | + ))} | |
| 33 | + </div> | |
| 34 | + </div> | |
| 35 | + <div className="flex flex-col gap-4"> | |
| 36 | + <SkeletonPanel lines={6} /> | |
| 37 | + <SkeletonPanel lines={10} /> | |
| 38 | + </div> | |
| 39 | + </div> | |
| 40 | + </> | |
| 41 | + ); | |
| 42 | +} | |
added
apps/web/src/app/country/[slug]/page.tsx
+115 −0
@@ -0,0 +1,115 @@ | ||
| 1 | +import Link from "next/link"; | |
| 2 | +import type { Metadata } from "next"; | |
| 3 | +import { notFound } from "next/navigation"; | |
| 4 | +import { EventRow } from "@/components/event-row"; | |
| 5 | +import { LiveFeed } from "@/components/live-feed"; | |
| 6 | +import { Badge, Bar, Chip, Empty, PageHeader, Panel, Stat, Table, Td, TierBadge } from "@/components/ui"; | |
| 7 | +import { api } from "@/lib/api"; | |
| 8 | +import { feedHref, fmtInt, relTime, typeLabel } from "@/lib/format"; | |
| 9 | + | |
| 10 | +export const dynamic = "force-dynamic"; | |
| 11 | + | |
| 12 | +/** Section order and labels for the desk (spec §104). Unknown categories are appended in API order. */ | |
| 13 | +const SECTIONS: { key: string; label: string }[] = [ | |
| 14 | + { key: "government", label: "Government" }, | |
| 15 | + { key: "finance", label: "Business & markets" }, | |
| 16 | + { key: "infrastructure", label: "Infrastructure" }, | |
| 17 | + { key: "health", label: "Health" }, | |
| 18 | + { key: "news", label: "Media" }, | |
| 19 | + { key: "cyber", label: "Cyber" }, | |
| 20 | + { key: "ai", label: "AI" }, | |
| 21 | + { key: "science", label: "Science" }, | |
| 22 | +]; | |
| 23 | + | |
| 24 | +export async function generateMetadata({ params }: { params: Promise<{ slug: string }> }): Promise<Metadata> { | |
| 25 | + const { slug } = await params; | |
| 26 | + const d = await api.country(slug); | |
| 27 | + if (!d) return { title: "Country not found" }; | |
| 28 | + return { title: `${d.country.name} desk`, description: `Government, business, infrastructure, health, media, cyber, AI and science signals from ${d.country.name}, detected live by WebSensor.`, alternates: { canonical: `/country/${slug}` } }; | |
| 29 | +} | |
| 30 | + | |
| 31 | +export default async function CountryPage({ params }: { params: Promise<{ slug: string }> }) { | |
| 32 | + const { slug } = await params; | |
| 33 | + const d = await api.country(slug); | |
| 34 | + if (!d) notFound(); | |
| 35 | + const code = d.country.code; | |
| 36 | + const name = d.country.name; | |
| 37 | + const events24 = d.sources.reduce((n, s) => n + (s.events_24h ?? 0), 0); | |
| 38 | + const ordered = [...SECTIONS.map((s) => ({ ...s, items: d.by_category.find((c) => c.category === s.key)?.items ?? [] })), ...d.by_category.filter((c) => !SECTIONS.some((s) => s.key === c.category)).map((c) => ({ key: c.category, label: c.category, items: c.items }))]; | |
| 39 | + const maxType = Math.max(1, ...d.by_type.map((t) => t.n)); | |
| 40 | + const firstParty = d.sources.filter((s) => s.first_party !== false).length; | |
| 41 | + return ( | |
| 42 | + <> | |
| 43 | + <PageHeader | |
| 44 | + compact | |
| 45 | + kicker={<span className="flex items-center gap-2"><Link href="/country" className="hover:text-fg">Countries</Link> <span className="text-fg-subtle">/</span> <span className="font-mono">{code}</span></span>} | |
| 46 | + title={<span className="flex items-center gap-2"><span aria-hidden>{d.country.flag}</span> {name}</span>} | |
| 47 | + description={`Official government, business, infrastructure, health and media signals from ${name}, grouped by desk. Sources are the organizations' own channels; media reports are marked EXTERNAL.`} | |
| 48 | + actions={<Link href={feedHref({ country: code }, "/live")} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">Live feed →</Link>} | |
| 49 | + /> | |
| 50 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-4 sm:divide-y-0"> | |
| 51 | + <Stat label="Sources" value={fmtInt(d.sources.length)} hint={`${fmtInt(firstParty)} first-party · ${fmtInt(d.sources.length - firstParty)} media`} /> | |
| 52 | + <Stat label="Events · 24 h" value={fmtInt(events24)} tone="signal" hint="meaningful signals" /> | |
| 53 | + <Stat label="Breaking · 24 h" value={fmtInt(d.breaking.length)} tone={d.breaking.length ? "hot" : undefined} hint="signal ≥ 80" /> | |
| 54 | + <Stat label="Silent" value={fmtInt(d.silent.length)} tone={d.silent.length ? "silent" : undefined} hint="no matching announcement" /> | |
| 55 | + </div> | |
| 56 | + | |
| 57 | + <div className="grid gap-4 xl:grid-cols-[1fr_360px]"> | |
| 58 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 59 | + <Panel title={<span className="text-hot">Major events · 24 h</span>} dense action={<Link href={feedHref({ country: code, signal_min: 80 }, "/live")} className="text-[11px] text-fg-subtle hover:text-fg">live →</Link>}> | |
| 60 | + {d.breaking.length ? d.breaking.map((e) => <EventRow key={e.id} ev={e} />) : <Empty>No breaking signal from {name} in the last 24 h.</Empty>} | |
| 61 | + </Panel> | |
| 62 | + <div className="grid gap-4 md:grid-cols-2"> | |
| 63 | + {ordered.map((s) => ( | |
| 64 | + <Panel key={s.key} title={s.label} dense action={<Link href={feedHref({ country: code, category: s.key }, "/live")} className="text-[11px] text-fg-subtle hover:text-fg">live →</Link>}> | |
| 65 | + {s.items.length ? s.items.map((e) => <EventRow key={e.id} ev={e} />) : <Empty>No {s.label.toLowerCase()} signal from {name} recently.</Empty>} | |
| 66 | + </Panel> | |
| 67 | + ))} | |
| 68 | + </div> | |
| 69 | + <LiveFeed initial={d.recent} initialCursor={d.nextCursor} extraQuery={{ country: code }} initialFilters={{ country: code }} showTabs={false} title={`LIVE · ${name.toUpperCase()}`} /> | |
| 70 | + </div> | |
| 71 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 72 | + <Panel title={<span className="text-silent">Silent changes</span>} dense action={<Link href={feedHref({ country: code, silent_change: true }, "/live")} className="text-[11px] text-fg-subtle hover:text-fg">all →</Link>}> | |
| 73 | + {d.silent.length ? d.silent.map((e) => <EventRow key={e.id} ev={e} />) : <Empty>No silent change from {name}.</Empty>} | |
| 74 | + </Panel> | |
| 75 | + <Panel title="By type · 7 d" dense> | |
| 76 | + {d.by_type.length ? ( | |
| 77 | + <ul className="divide-y divide-line"> | |
| 78 | + {d.by_type.map((t) => ( | |
| 79 | + <li key={t.event_type} className="grid grid-cols-[9rem_1fr_3.5rem] items-center gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 80 | + <Link href={feedHref({ country: code, event_type: t.event_type }, "/live")} className="truncate hover:underline">{typeLabel(t.event_type)}</Link> | |
| 81 | + <Bar value={t.n} max={maxType} tone="info" /> | |
| 82 | + <span className="text-right font-mono text-fg-subtle tabular">{fmtInt(t.n)}</span> | |
| 83 | + </li> | |
| 84 | + ))} | |
| 85 | + </ul> | |
| 86 | + ) : ( | |
| 87 | + <Empty /> | |
| 88 | + )} | |
| 89 | + </Panel> | |
| 90 | + </aside> | |
| 91 | + </div> | |
| 92 | + | |
| 93 | + <Panel title={`Sources · ${d.sources.length}`} dense className="mt-4" action={<Link href={`/sources?country=${code}`} className="text-[11px] text-fg-subtle hover:text-fg">filter sources →</Link>}> | |
| 94 | + {d.sources.length === 0 ? ( | |
| 95 | + <Empty>No source is attributed to {name} yet.</Empty> | |
| 96 | + ) : ( | |
| 97 | + <Table head={["Tier", "Source", "Domain", "Kind", "Categories", "Sensors", "Events 24 h", "Last event"]}> | |
| 98 | + {d.sources.map((s) => ( | |
| 99 | + <tr key={s.id} className="hover:bg-panel-2/60"> | |
| 100 | + <Td><TierBadge tier={s.tier} /></Td> | |
| 101 | + <Td><Link href={`/source/${s.id}`} className="font-medium hover:underline">{s.name}</Link></Td> | |
| 102 | + <Td mono><Link href={`/domain/${s.domain}`} className="text-fg-muted hover:underline">{s.domain}</Link></Td> | |
| 103 | + <Td>{s.first_party === false ? <Badge kind="external" compact /> : <Badge kind="first-party" compact />}</Td> | |
| 104 | + <Td><div className="flex flex-wrap gap-1">{s.categories.slice(0, 3).map((c) => <Chip key={c} href={`/category/${c}`}>{c}</Chip>)}</div></Td> | |
| 105 | + <Td mono>{s.sensor_count ?? 0}</Td> | |
| 106 | + <Td mono>{fmtInt(s.events_24h ?? 0)}</Td> | |
| 107 | + <Td mono className="text-fg-subtle">{s.last_event_at ? relTime(s.last_event_at) : "—"}</Td> | |
| 108 | + </tr> | |
| 109 | + ))} | |
| 110 | + </Table> | |
| 111 | + )} | |
| 112 | + </Panel> | |
| 113 | + </> | |
| 114 | + ); | |
| 115 | +} | |
added
apps/web/src/app/country/loading.tsx
+17 −0
@@ -0,0 +1,17 @@ | ||
| 1 | +import { Skeleton, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3"> | |
| 7 | + <Skeleton className="mb-1 h-2.5 w-12" /> | |
| 8 | + <Skeleton className="h-5 w-32" /> | |
| 9 | + <Skeleton className="mt-2 h-3 w-full max-w-2xl" /> | |
| 10 | + </div> | |
| 11 | + <div className="panel"> | |
| 12 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-64" /></div> | |
| 13 | + <SkeletonRows rows={16} /> | |
| 14 | + </div> | |
| 15 | + </> | |
| 16 | + ); | |
| 17 | +} | |
added
apps/web/src/app/country/page.tsx
+39 −0
@@ -0,0 +1,39 @@ | ||
| 1 | +import Link from "next/link"; | |
| 2 | +import type { Metadata } from "next"; | |
| 3 | +import { Bar, Empty, Flag, PageHeader, Panel, Table, Td } from "@/components/ui"; | |
| 4 | +import { api } from "@/lib/api"; | |
| 5 | +import { fmtInt } from "@/lib/format"; | |
| 6 | + | |
| 7 | +export const dynamic = "force-dynamic"; | |
| 8 | +export const metadata: Metadata = { title: "Countries", description: "Country desks: government, business, infrastructure, health, media, cyber, AI and science signals for every country WebSensor covers.", alternates: { canonical: "/country" } }; | |
| 9 | + | |
| 10 | +export default async function CountriesPage() { | |
| 11 | + const { items } = await api.countries(); | |
| 12 | + const totals = items.reduce((a, c) => ({ sources: a.sources + c.sources, events: a.events + c.events_24h, breaking: a.breaking + c.breaking_24h }), { sources: 0, events: 0, breaking: 0 }); | |
| 13 | + const max = Math.max(1, ...items.map((c) => c.events_24h)); | |
| 14 | + return ( | |
| 15 | + <> | |
| 16 | + <PageHeader compact kicker="Desks" title="Countries" description={`${fmtInt(items.length)} countries · ${fmtInt(totals.sources)} sources · ${fmtInt(totals.events)} events and ${fmtInt(totals.breaking)} breaking signals in the last 24 h. Each desk groups official government, business, infrastructure, health and media signals by jurisdiction.`} /> | |
| 17 | + <Panel dense> | |
| 18 | + {items.length === 0 ? ( | |
| 19 | + <Empty>Country desks appear once sources carry a country.</Empty> | |
| 20 | + ) : ( | |
| 21 | + <Table head={["", "Country", "Code", "Sources", "Events 24 h", "", "Breaking 24 h", ""]}> | |
| 22 | + {items.map((c) => ( | |
| 23 | + <tr key={c.country} className="hover:bg-panel-2/60"> | |
| 24 | + <Td className="w-8 text-base leading-none">{c.flag || <Flag code={c.country} />}</Td> | |
| 25 | + <Td><Link href={`/country/${c.slug}`} className="font-medium hover:underline">{c.name}</Link></Td> | |
| 26 | + <Td mono className="text-fg-subtle">{c.country}</Td> | |
| 27 | + <Td mono>{fmtInt(c.sources)}</Td> | |
| 28 | + <Td mono>{fmtInt(c.events_24h)}</Td> | |
| 29 | + <Td><div className="w-20 sm:w-40"><Bar value={c.events_24h} max={max} /></div></Td> | |
| 30 | + <Td mono className={c.breaking_24h ? "text-hot" : "text-fg-subtle"}>{fmtInt(c.breaking_24h)}</Td> | |
| 31 | + <Td><Link href={`/country/${c.slug}`} className="whitespace-nowrap text-[11px] text-fg-subtle hover:text-fg">desk →</Link></Td> | |
| 32 | + </tr> | |
| 33 | + ))} | |
| 34 | + </Table> | |
| 35 | + )} | |
| 36 | + </Panel> | |
| 37 | + </> | |
| 38 | + ); | |
| 39 | +} | |
added
apps/web/src/app/entities/loading.tsx
+35 −0
@@ -0,0 +1,35 @@ | ||
| 1 | +import { Skeleton } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3 flex flex-col gap-2"> | |
| 7 | + <Skeleton className="h-6 w-28" /> | |
| 8 | + <Skeleton className="h-3 w-96 max-w-full" /> | |
| 9 | + </div> | |
| 10 | + <div className="mb-3 flex gap-4 border-b border-line pb-2"> | |
| 11 | + <Skeleton className="h-3 w-14" /> | |
| 12 | + <Skeleton className="h-3 w-10" /> | |
| 13 | + </div> | |
| 14 | + <div className="panel"> | |
| 15 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-32" /></div> | |
| 16 | + <div className="divide-y divide-line"> | |
| 17 | + {Array.from({ length: 14 }, (_, i) => ( | |
| 18 | + <div key={i} className="grid grid-cols-[2rem_1fr_5rem_3rem_3rem_3rem] items-center gap-x-3 px-3 py-2 sm:grid-cols-[2rem_1fr_5rem_3rem_3rem_3rem_4rem_4rem_4rem_4rem]"> | |
| 19 | + <Skeleton className="h-3 w-5" /> | |
| 20 | + <div className="flex flex-col gap-1.5"><Skeleton className={`h-3.5 ${i % 3 === 0 ? "w-40" : i % 3 === 1 ? "w-28" : "w-52"}`} /><Skeleton className="h-2.5 w-24" /></div> | |
| 21 | + <Skeleton className="h-4 w-16" /> | |
| 22 | + <Skeleton className="h-5 w-8" /> | |
| 23 | + <Skeleton className="h-3 w-7" /> | |
| 24 | + <Skeleton className="h-3 w-7" /> | |
| 25 | + <Skeleton className="hidden h-3 w-6 sm:block" /> | |
| 26 | + <Skeleton className="hidden h-3 w-8 sm:block" /> | |
| 27 | + <Skeleton className="hidden h-3 w-8 sm:block" /> | |
| 28 | + <Skeleton className="hidden h-3 w-10 sm:block" /> | |
| 29 | + </div> | |
| 30 | + ))} | |
| 31 | + </div> | |
| 32 | + </div> | |
| 33 | + </> | |
| 34 | + ); | |
| 35 | +} | |
modified
apps/web/src/app/entities/page.tsx
+70 −11
@@ -1,29 +1,88 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | 2 | import type { Metadata } from "next"; |
| 3 | 3 | import { ENTITY_TYPES } from "@websensor/core/client"; |
| 4 | −import { Chip, Empty, PageHeader, Panel, Table, Td } from "@/components/ui"; | |
| 4 | +import { Chip, Empty, PageHeader, Panel, Score, Table, Tabs, Td } from "@/components/ui"; | |
| 5 | 5 | import { api } from "@/lib/api"; |
| 6 | −import { fmtInt, fmtScore, relTime } from "@/lib/format"; | |
| 6 | +import { fmtInt, fmtScore, relTime, typeLabel } from "@/lib/format"; | |
| 7 | 7 | |
| 8 | 8 | export const dynamic = "force-dynamic"; |
| 9 | −export const metadata: Metadata = { title: "Entities", description: "Organizations, products, models, APIs and other entities that events resolve to." }; | |
| 9 | +export const metadata: Metadata = { title: "Entities", description: "Organizations, products, models, APIs and other entities that events resolve to — ranked by importance, activity, velocity, quality and confirmation.", alternates: { canonical: "/entities" } }; | |
| 10 | 10 | |
| 11 | −export default async function EntitiesPage({ searchParams }: { searchParams: Promise<{ type?: string; q?: string }> }) { | |
| 11 | +type Search = { tab?: string; type?: string; q?: string }; | |
| 12 | + | |
| 13 | +export default async function EntitiesPage({ searchParams }: { searchParams: Promise<Search> }) { | |
| 12 | 14 | const sp = await searchParams; |
| 15 | + const tab = sp.tab === "all" || sp.type || sp.q ? "all" : "ranked"; | |
| 16 | + return ( | |
| 17 | + <> | |
| 18 | + <PageHeader compact title="Entities" description="Everything important resolves to an entity. Each entity has a timeline — the equivalent of git history for its public Web presence." /> | |
| 19 | + <Tabs | |
| 20 | + className="mb-3" | |
| 21 | + current={tab} | |
| 22 | + items={[ | |
| 23 | + { key: "ranked", label: "Ranked", href: "/entities" }, | |
| 24 | + { key: "all", label: "All", href: "/entities?tab=all" }, | |
| 25 | + ]} | |
| 26 | + /> | |
| 27 | + {tab === "ranked" ? <Ranked /> : <All sp={sp} />} | |
| 28 | + </> | |
| 29 | + ); | |
| 30 | +} | |
| 31 | + | |
| 32 | +async function Ranked() { | |
| 33 | + const { items } = await api.rank(200); | |
| 34 | + return ( | |
| 35 | + <Panel | |
| 36 | + title={<>WebSensor rank <span className="font-mono text-fg-subtle">{fmtInt(items.length)}</span></>} | |
| 37 | + dense | |
| 38 | + action={<span className="hidden text-[11px] text-fg-subtle md:inline">importance × activity × velocity × quality × confirmation — not raw counts</span>} | |
| 39 | + > | |
| 40 | + <p className="border-b border-line px-3 py-2 text-[12px] text-fg-muted md:hidden">Ranking uses importance × activity × velocity × quality × confirmation, not raw counts.</p> | |
| 41 | + {items.length === 0 ? ( | |
| 42 | + <Empty>The ranking is computed from the last 7 days of activity — nothing to rank yet.</Empty> | |
| 43 | + ) : ( | |
| 44 | + <Table head={["#", "Entity", "Type", "Rank", "24 h", "7 d", "Sources", "Avg signal", "Confirmed", "Silent 24 h", "Breaking 24 h", "Last"]}> | |
| 45 | + {items.map((e) => ( | |
| 46 | + <tr key={e.id} className="hover:bg-panel-2/60"> | |
| 47 | + <Td mono className="text-fg-subtle">{e.rank}</Td> | |
| 48 | + <Td> | |
| 49 | + <Link href={`/entity/${e.id}`} className="font-medium hover:underline">{e.name}</Link> | |
| 50 | + {e.domain && <div className="truncate font-mono text-[10.5px] text-fg-subtle">{e.domain}</div>} | |
| 51 | + </Td> | |
| 52 | + <Td><Chip href={`/entities?tab=all&type=${e.type}`}>{typeLabel(e.type)}</Chip></Td> | |
| 53 | + <Td><Score value={e.rank_score} size="sm" title={`Rank score ${fmtScore(e.rank_score)} · entity importance ${fmtScore(e.importance)}`} /></Td> | |
| 54 | + <Td mono className={e.events_24h ? "" : "text-fg-subtle"}>{fmtInt(e.events_24h)}</Td> | |
| 55 | + <Td mono className="text-fg-muted">{fmtInt(e.events_7d)}</Td> | |
| 56 | + <Td mono>{fmtInt(e.sources)}</Td> | |
| 57 | + <Td mono>{fmtScore(e.avg_signal)}</Td> | |
| 58 | + <Td mono className={e.confirmed_ratio > 0 ? "text-ok" : "text-fg-subtle"}>{Math.round(e.confirmed_ratio * 100)}%</Td> | |
| 59 | + <Td mono className={e.silent_24h ? "text-silent" : "text-fg-subtle"}>{fmtInt(e.silent_24h)}</Td> | |
| 60 | + <Td mono className={e.breaking_24h ? "text-hot" : "text-fg-subtle"}>{fmtInt(e.breaking_24h)}</Td> | |
| 61 | + <Td mono className="whitespace-nowrap text-fg-subtle">{relTime(e.last_at)}</Td> | |
| 62 | + </tr> | |
| 63 | + ))} | |
| 64 | + </Table> | |
| 65 | + )} | |
| 66 | + </Panel> | |
| 67 | + ); | |
| 68 | +} | |
| 69 | + | |
| 70 | +async function All({ sp }: { sp: Search }) { | |
| 13 | 71 | const { items } = await api.entities({ type: sp.type, q: sp.q, limit: 400 }); |
| 14 | 72 | const types = new Set(items.map((e) => e.type)); |
| 15 | 73 | return ( |
| 16 | 74 | <> |
| 17 | − <PageHeader kicker={`${fmtInt(items.length)} entities`} title="Entities" description="Everything important resolves to an entity. Each entity has a timeline — the equivalent of git history for its public Web presence." /> | |
| 18 | 75 | <form className="mb-3 flex flex-wrap items-center gap-2" action="/entities"> |
| 19 | − <input name="q" defaultValue={sp.q ?? ""} placeholder="Search entities…" className="h-8 w-64 rounded-md border border-line bg-panel px-2.5 text-[13px] placeholder:text-fg-subtle" /> | |
| 76 | + <input type="hidden" name="tab" value="all" /> | |
| 77 | + <input name="q" defaultValue={sp.q ?? ""} placeholder="Search entities…" className="h-8 w-64 max-w-full rounded-md border border-line bg-panel px-2.5 text-[13px] placeholder:text-fg-subtle" /> | |
| 20 | 78 | {sp.type && <input type="hidden" name="type" value={sp.type} />} |
| 21 | 79 | <button type="submit" className="h-8 rounded-md border border-line bg-panel-2 px-3 text-[12.5px]">Search</button> |
| 80 | + <span className="font-mono text-[11px] text-fg-subtle tabular">{fmtInt(items.length)} entit{items.length === 1 ? "y" : "ies"}</span> | |
| 22 | 81 | </form> |
| 23 | 82 | <div className="mb-3 flex flex-wrap gap-1"> |
| 24 | − <Chip href="/entities" tone={!sp.type ? "signal" : "default"}>all</Chip> | |
| 83 | + <Chip href={sp.q ? `/entities?tab=all&q=${encodeURIComponent(sp.q)}` : "/entities?tab=all"} tone={!sp.type ? "signal" : "default"}>all</Chip> | |
| 25 | 84 | {ENTITY_TYPES.filter((t) => types.has(t) || t === sp.type).map((t) => ( |
| 26 | − <Chip key={t} href={`/entities?type=${t}`} tone={sp.type === t ? "signal" : "default"}>{t}</Chip> | |
| 85 | + <Chip key={t} href={`/entities?tab=all&type=${t}${sp.q ? `&q=${encodeURIComponent(sp.q)}` : ""}`} tone={sp.type === t ? "signal" : "default"}>{typeLabel(t)}</Chip> | |
| 27 | 86 | ))} |
| 28 | 87 | </div> |
| 29 | 88 | <Panel dense> |
@@ -33,13 +92,13 @@ export default async function EntitiesPage({ searchParams }: { searchParams: Pro | ||
| 33 | 92 | <Table head={["Entity", "Type", "Domain", "Importance", "Events", "24 h", "Last event"]}> |
| 34 | 93 | {items.map((e) => ( |
| 35 | 94 | <tr key={e.id} className="hover:bg-panel-2/60"> |
| 36 | − <Td><Link href={`/company/${e.id}`} className="font-medium hover:underline">{e.name}</Link></Td> | |
| 37 | − <Td><Chip>{e.type}</Chip></Td> | |
| 95 | + <Td><Link href={`/entity/${e.id}`} className="font-medium hover:underline">{e.name}</Link></Td> | |
| 96 | + <Td><Chip>{typeLabel(e.type)}</Chip></Td> | |
| 38 | 97 | <Td mono className="text-fg-subtle">{e.domain ?? "—"}</Td> |
| 39 | 98 | <Td mono>{fmtScore(e.importance)}</Td> |
| 40 | 99 | <Td mono>{fmtInt(e.event_count)}</Td> |
| 41 | 100 | <Td mono className={e.events_24h ? "text-signal" : "text-fg-subtle"}>{e.events_24h ?? 0}</Td> |
| 42 | − <Td mono className="text-fg-subtle">{e.last_event_at ? relTime(e.last_event_at) : "—"}</Td> | |
| 101 | + <Td mono className="whitespace-nowrap text-fg-subtle">{e.last_event_at ? relTime(e.last_event_at) : "—"}</Td> | |
| 43 | 102 | </tr> |
| 44 | 103 | ))} |
| 45 | 104 | </Table> |
added
apps/web/src/app/entity/[id]/layout.tsx
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +import { notFound } from "next/navigation"; | |
| 2 | +import type { ReactNode } from "react"; | |
| 3 | +import { api } from "@/lib/api"; | |
| 4 | + | |
| 5 | +/** | |
| 6 | + * Existence check outside the `loading.tsx` boundary: the page streams behind a skeleton, so a | |
| 7 | + * `notFound()` thrown there would arrive after the 200 shell. Checking here yields a real 404. | |
| 8 | + * (The page's own `api.entity()` call is deduplicated by the fetch memoization of the same render.) | |
| 9 | + */ | |
| 10 | +export default async function EntityLayout({ params, children }: { params: Promise<{ id: string }>; children: ReactNode }) { | |
| 11 | + const { id } = await params; | |
| 12 | + const d = await api.entity(id); | |
| 13 | + if (!d) notFound(); | |
| 14 | + return children; | |
| 15 | +} | |
added
apps/web/src/app/entity/[id]/loading.tsx
+56 −0
@@ -0,0 +1,56 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3 flex flex-wrap items-end justify-between gap-3"> | |
| 7 | + <div className="flex flex-col gap-2"> | |
| 8 | + <div className="flex items-center gap-2"> | |
| 9 | + <Skeleton className="h-4 w-20" /> | |
| 10 | + <Skeleton className="h-3 w-24" /> | |
| 11 | + <Skeleton className="h-4 w-16" /> | |
| 12 | + <Skeleton className="h-3 w-32" /> | |
| 13 | + </div> | |
| 14 | + <Skeleton className="h-6 w-56" /> | |
| 15 | + </div> | |
| 16 | + <div className="flex gap-2"> | |
| 17 | + <Skeleton className="h-7 w-20" /> | |
| 18 | + <Skeleton className="h-7 w-24" /> | |
| 19 | + </div> | |
| 20 | + </div> | |
| 21 | + <div className="mb-3 grid grid-cols-2 gap-px overflow-hidden rounded-lg border border-line bg-line sm:grid-cols-3 xl:grid-cols-6"> | |
| 22 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 23 | + <div key={i} className="flex flex-col gap-1.5 bg-panel px-3 py-2"> | |
| 24 | + <Skeleton className="h-2.5 w-16" /> | |
| 25 | + <Skeleton className="h-5 w-12" /> | |
| 26 | + <Skeleton className="h-2.5 w-24" /> | |
| 27 | + </div> | |
| 28 | + ))} | |
| 29 | + </div> | |
| 30 | + <div className="panel mb-3 flex flex-wrap items-center gap-4 px-3 py-2"> | |
| 31 | + <Skeleton className="h-2.5 w-24" /> | |
| 32 | + <div className="flex flex-wrap gap-[3px]"> | |
| 33 | + {Array.from({ length: 35 }, (_, i) => ( | |
| 34 | + <Skeleton key={i} className="size-3 !rounded-[2px]" /> | |
| 35 | + ))} | |
| 36 | + </div> | |
| 37 | + </div> | |
| 38 | + <div className="mb-3 flex gap-4 border-b border-line pb-2"> | |
| 39 | + {Array.from({ length: 7 }, (_, i) => ( | |
| 40 | + <Skeleton key={i} className="h-3 w-14" /> | |
| 41 | + ))} | |
| 42 | + </div> | |
| 43 | + <div className="grid gap-4 xl:grid-cols-[1fr_340px]"> | |
| 44 | + <div className="panel"> | |
| 45 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-28" /></div> | |
| 46 | + <SkeletonRows rows={10} /> | |
| 47 | + </div> | |
| 48 | + <aside className="flex flex-col gap-4"> | |
| 49 | + <SkeletonPanel lines={6} /> | |
| 50 | + <SkeletonPanel lines={3} /> | |
| 51 | + <SkeletonPanel lines={4} /> | |
| 52 | + </aside> | |
| 53 | + </div> | |
| 54 | + </> | |
| 55 | + ); | |
| 56 | +} | |
added
apps/web/src/app/entity/[id]/page.tsx
+480 −0
@@ -0,0 +1,480 @@ | ||
| 1 | +import Link from "next/link"; | |
| 2 | +import type { Metadata } from "next"; | |
| 3 | +import { notFound } from "next/navigation"; | |
| 4 | +import { EventRow } from "@/components/event-row"; | |
| 5 | +import { FieldChanges } from "@/components/field-changes"; | |
| 6 | +import { LiveFeed } from "@/components/live-feed"; | |
| 7 | +import { Badge, Bar, Chip, Empty, ExtLink, Flag, Heatmap, PageHeader, Panel, Score, Stat, Table, Tabs, Td, TierBadge, TypeChip } from "@/components/ui"; | |
| 8 | +import { WatchButton } from "@/components/watch-button"; | |
| 9 | +import { api, SITE_URL, type EntityDetail, type EntityInsights, type EventItem } from "@/lib/api"; | |
| 10 | +import { agoIso, dayHeader, fmtInt, fmtScore, relTime, typeLabel, utcDate, utcDateTime, utcTime, withinLast } from "@/lib/format"; | |
| 11 | + | |
| 12 | +export const dynamic = "force-dynamic"; | |
| 13 | + | |
| 14 | +type Tab = "overview" | "live" | "sources" | "silent" | "timeline" | "related" | "metrics"; | |
| 15 | +const TABS: Tab[] = ["overview", "live", "sources", "silent", "timeline", "related", "metrics"]; | |
| 16 | +const RANGES: { key: string; label: string; ms: number | null }[] = [ | |
| 17 | + { key: "1h", label: "1 h", ms: 3600e3 }, | |
| 18 | + { key: "6h", label: "6 h", ms: 6 * 3600e3 }, | |
| 19 | + { key: "24h", label: "24 h", ms: 24 * 3600e3 }, | |
| 20 | + { key: "7d", label: "7 d", ms: 7 * 86400e3 }, | |
| 21 | + { key: "30d", label: "30 d", ms: 30 * 86400e3 }, | |
| 22 | + { key: "all", label: "All", ms: null }, | |
| 23 | +]; | |
| 24 | + | |
| 25 | +type Search = { tab?: string; range?: string; cursor?: string }; | |
| 26 | + | |
| 27 | +export async function generateMetadata({ params }: { params: Promise<{ id: string }> }): Promise<Metadata> { | |
| 28 | + const { id } = await params; | |
| 29 | + const d = await api.entity(id); | |
| 30 | + if (!d) return { title: "Entity not found" }; | |
| 31 | + const e = d.entity; | |
| 32 | + const desc = e.description ?? `Every meaningful change WebSensor detected for ${e.name}: announcements, releases, pricing, documentation, incidents and silent changes — with evidence.`; | |
| 33 | + return { title: `${e.name} — ${typeLabel(e.type)}`, description: desc, alternates: { canonical: `/entity/${e.id}` }, openGraph: { title: `${e.name} · WebSensor`, description: desc, url: `${SITE_URL}/entity/${e.id}` } }; | |
| 34 | +} | |
| 35 | + | |
| 36 | +const EMPTY_INSIGHTS: EntityInsights = { heatmap: [], baseline_per_day: 0, today: 0, events_24h: 0, events_prev_24h: 0, velocity_ratio: 0, anomaly: { score: 0, ratio: 0, pct: 0 }, silent_24h: 0, breaking_24h: 0, sources_24h: 0, most_active_sensors: [], rank: { rank: null, total: 0, score: null } }; | |
| 37 | + | |
| 38 | +function mainCountry(d: EntityDetail): string | null { | |
| 39 | + const counts = new Map<string, number>(); | |
| 40 | + for (const s of d.sources) if (s.country) counts.set(s.country, (counts.get(s.country) ?? 0) + (s.first_party ? 2 : 1)); | |
| 41 | + let best: string | null = null; | |
| 42 | + let n = 0; | |
| 43 | + for (const [c, k] of counts) if (k > n) [best, n] = [c, k]; | |
| 44 | + return best; | |
| 45 | +} | |
| 46 | + | |
| 47 | +export default async function EntityPage({ params, searchParams }: { params: Promise<{ id: string }>; searchParams: Promise<Search> }) { | |
| 48 | + const { id } = await params; | |
| 49 | + const sp = await searchParams; | |
| 50 | + const d = await api.entity(id); | |
| 51 | + if (!d) notFound(); | |
| 52 | + const e = d.entity; | |
| 53 | + const tab: Tab = TABS.includes(sp.tab as Tab) ? (sp.tab as Tab) : "overview"; | |
| 54 | + const ins = d.insights ?? EMPTY_INSIGHTS; | |
| 55 | + const silent = d.silent ?? []; | |
| 56 | + const related = d.related ?? []; | |
| 57 | + const country = mainCountry(d); | |
| 58 | + const active = withinLast(e.last_event_at, 7 * 86400e3); | |
| 59 | + const anomalous = ins.anomaly.score >= 60; | |
| 60 | + const delta = ins.events_24h - ins.events_prev_24h; | |
| 61 | + const href = (t: Tab): string => (t === "overview" ? `/entity/${e.id}` : `/entity/${e.id}?tab=${t}`); | |
| 62 | + | |
| 63 | + return ( | |
| 64 | + <> | |
| 65 | + <PageHeader | |
| 66 | + compact | |
| 67 | + kicker={ | |
| 68 | + <span className="flex flex-wrap items-center gap-x-2 gap-y-1"> | |
| 69 | + <TypeChip type={e.type} href={`/entities?tab=all&type=${e.type}`} /> | |
| 70 | + {d.parent && ( | |
| 71 | + <Link href={`/entity/${d.parent.id}`} className="inline-flex items-center gap-1 text-fg-muted hover:text-fg" title={`Parent: ${d.parent.name}`}> | |
| 72 | + <span className="text-fg-subtle">↑</span> {d.parent.name} | |
| 73 | + </Link> | |
| 74 | + )} | |
| 75 | + {e.domain && <Link href={`/domain/${e.domain}`} className="font-mono text-fg-muted hover:text-fg hover:underline">{e.domain}</Link>} | |
| 76 | + {country && <Flag code={country} className="text-[12px]" />} | |
| 77 | + <Chip tone={active ? "signal" : "default"} className="font-mono font-semibold tracking-wider" title={active ? "Events detected in the last 7 days" : "No event in the last 7 days"}> | |
| 78 | + <span className={`inline-block size-1.5 rounded-full ${active ? "bg-signal" : "bg-low"}`} /> {active ? "ACTIVE" : "QUIET"} | |
| 79 | + </Chip> | |
| 80 | + {ins.rank.rank !== null && ( | |
| 81 | + <span className="font-mono text-fg-muted tabular" title={`WebSensor rank score ${fmtScore(ins.rank.score)} — importance × activity × velocity × quality × confirmation`}> | |
| 82 | + <span className="text-fg-subtle">WebSensor rank</span> <Link href="/entities" className="font-semibold text-fg hover:underline">#{fmtInt(ins.rank.rank)}</Link> <span className="text-fg-subtle">of {fmtInt(ins.rank.total)}</span> | |
| 83 | + </span> | |
| 84 | + )} | |
| 85 | + </span> | |
| 86 | + } | |
| 87 | + title={e.name} | |
| 88 | + description={e.description} | |
| 89 | + actions={ | |
| 90 | + <> | |
| 91 | + <WatchButton kind="entity" value={e.id} /> | |
| 92 | + <Link href={`/alerts?entity=${encodeURIComponent(e.id)}`} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">Add to alert</Link> | |
| 93 | + {(e.homepage || e.domain) && <ExtLink href={e.homepage ?? `https://${e.domain}`} className="text-[12px]">Website ↗</ExtLink>} | |
| 94 | + </> | |
| 95 | + } | |
| 96 | + /> | |
| 97 | + | |
| 98 | + <div className="mb-3 grid grid-cols-2 gap-px overflow-hidden rounded-lg border border-line bg-line sm:grid-cols-3 xl:grid-cols-6"> | |
| 99 | + <div className="bg-panel"> | |
| 100 | + <Stat | |
| 101 | + label="Activity 24 h" | |
| 102 | + value={fmtInt(ins.events_24h)} | |
| 103 | + hint={ | |
| 104 | + <span className={delta > 0 ? "text-signal" : delta < 0 ? "text-fg-subtle" : ""}> | |
| 105 | + {delta > 0 ? "↑" : delta < 0 ? "↓" : "→"} {Math.abs(delta)} vs prev 24 h · ×{ins.velocity_ratio.toFixed(2)} | |
| 106 | + </span> | |
| 107 | + } | |
| 108 | + /> | |
| 109 | + </div> | |
| 110 | + <div className="bg-panel"><Stat label="Breaking 24 h" value={fmtInt(ins.breaking_24h)} tone={ins.breaking_24h > 0 ? "hot" : undefined} hint="signal ≥ 80" /></div> | |
| 111 | + <div className="bg-panel"><Stat label="Silent 24 h" value={fmtInt(ins.silent_24h)} tone={ins.silent_24h > 0 ? "silent" : undefined} hint="no matching announcement" /></div> | |
| 112 | + <div className="bg-panel"><Stat label="Sources 24 h" value={fmtInt(ins.sources_24h)} hint={`${fmtInt(d.sources.length)} linked`} /></div> | |
| 113 | + <div className="bg-panel"><Stat label="30-day baseline" value={<>{ins.baseline_per_day.toFixed(1)}<span className="text-[11px] font-normal text-fg-subtle">/day</span></>} hint={`today ${fmtInt(ins.today)}`} /></div> | |
| 114 | + <div className="bg-panel"> | |
| 115 | + <Stat | |
| 116 | + label={anomalous ? "Anomalous activity" : "Anomaly"} | |
| 117 | + value={ins.anomaly.score > 0 ? fmtScore(ins.anomaly.score) : "0"} | |
| 118 | + tone={anomalous ? "hot" : ins.anomaly.score >= 40 ? "warn" : undefined} | |
| 119 | + hint={<span className={anomalous ? "font-semibold text-hot" : ""}>{ins.anomaly.pct > 0 ? "+" : ""}{Math.round(ins.anomaly.pct)}% vs baseline</span>} | |
| 120 | + /> | |
| 121 | + </div> | |
| 122 | + </div> | |
| 123 | + | |
| 124 | + <div className="panel mb-3 flex flex-wrap items-center gap-x-4 gap-y-2 px-3 py-2"> | |
| 125 | + <span className="label">35-day activity</span> | |
| 126 | + <div className="min-w-0"><Heatmap days={ins.heatmap} /></div> | |
| 127 | + <span className="ml-auto flex items-center gap-1 text-[10.5px] text-fg-subtle"> | |
| 128 | + less <span className="size-2.5 rounded-[2px] heat-1" /><span className="size-2.5 rounded-[2px] heat-2" /><span className="size-2.5 rounded-[2px] heat-3" /><span className="size-2.5 rounded-[2px] heat-4" /> more | |
| 129 | + <span className="ml-2 size-2.5 rounded-[2px] heat-hot" /> ≥ 3 breaking | |
| 130 | + </span> | |
| 131 | + </div> | |
| 132 | + | |
| 133 | + <Tabs | |
| 134 | + className="mb-3" | |
| 135 | + current={tab} | |
| 136 | + items={[ | |
| 137 | + { key: "overview", label: "Overview", href: href("overview") }, | |
| 138 | + { key: "live", label: "Live", href: href("live") }, | |
| 139 | + { key: "sources", label: "Sources", href: href("sources"), count: d.sources.length }, | |
| 140 | + { key: "silent", label: <span className={silent.length ? "text-silent" : ""}>Silent</span>, href: href("silent"), count: silent.length }, | |
| 141 | + { key: "timeline", label: "Timeline", href: href("timeline") }, | |
| 142 | + { key: "related", label: "Related", href: href("related"), count: related.length + d.children.length }, | |
| 143 | + { key: "metrics", label: "Metrics", href: href("metrics") }, | |
| 144 | + ]} | |
| 145 | + /> | |
| 146 | + | |
| 147 | + {tab === "overview" && <Overview d={d} ins={ins} />} | |
| 148 | + {tab === "live" && <LiveFeed initial={d.recent} initialCursor={d.nextCursor ?? null} extraQuery={{ entity: e.id }} showTabs={false} title={`LIVE · ${e.name.toUpperCase()}`} />} | |
| 149 | + {tab === "sources" && <SourcesTab d={d} />} | |
| 150 | + {tab === "silent" && <SilentTab items={silent} name={e.name} />} | |
| 151 | + {tab === "timeline" && <TimelineTab id={e.id} range={sp.range} cursor={sp.cursor} />} | |
| 152 | + {tab === "related" && <RelatedTab d={d} />} | |
| 153 | + {tab === "metrics" && <MetricsTab d={d} ins={ins} />} | |
| 154 | + </> | |
| 155 | + ); | |
| 156 | +} | |
| 157 | + | |
| 158 | +// --------------------------------------------------------------------------------------- | |
| 159 | + | |
| 160 | +function Overview({ d, ins }: { d: EntityDetail; ins: EntityInsights }) { | |
| 161 | + const e = d.entity; | |
| 162 | + const related = d.related ?? []; | |
| 163 | + return ( | |
| 164 | + <div className="grid gap-4 xl:grid-cols-[1fr_340px]"> | |
| 165 | + <Panel title={<>Recent signals <span className="font-mono text-fg-subtle">{d.recent.length}</span></>} dense action={<Link href={`/entity/${e.id}?tab=timeline`} className="text-[11px] text-fg-subtle hover:text-fg">full timeline →</Link>}> | |
| 166 | + {d.recent.length ? d.recent.map((ev) => <EventRow key={ev.id} ev={ev} showDate />) : <Empty>No event detected yet for {e.name}. Sensors linked to this entity are checked continuously.</Empty>} | |
| 167 | + </Panel> | |
| 168 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 169 | + <SensorsPanel sensors={ins.most_active_sensors} /> | |
| 170 | + <Panel title="Related entities" dense> | |
| 171 | + {related.length ? ( | |
| 172 | + <div className="flex flex-wrap gap-1 p-2"> | |
| 173 | + {related.slice(0, 16).map((r) => ( | |
| 174 | + <Chip key={r.id} href={`/entity/${r.id}`} title={`${typeLabel(r.type)} · ${r.shared_events} shared event${r.shared_events === 1 ? "" : "s"}`}> | |
| 175 | + {r.name} <span className="font-mono text-fg-subtle tabular">{r.shared_events}</span> | |
| 176 | + </Chip> | |
| 177 | + ))} | |
| 178 | + </div> | |
| 179 | + ) : ( | |
| 180 | + <Empty>No co-occurring entity yet.</Empty> | |
| 181 | + )} | |
| 182 | + </Panel> | |
| 183 | + {d.children.length > 0 && ( | |
| 184 | + <Panel title={<>Products & children <span className="font-mono text-fg-subtle">{d.children.length}</span></>} dense> | |
| 185 | + <ul className="divide-y divide-line"> | |
| 186 | + {d.children.slice(0, 12).map((c) => ( | |
| 187 | + <li key={c.id} className="flex items-center justify-between gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 188 | + <Link href={`/entity/${c.id}`} className="min-w-0 truncate font-medium hover:underline">{c.name}</Link> | |
| 189 | + <span className="shrink-0 font-mono text-[11px] text-fg-subtle tabular">{typeLabel(c.type)} · {fmtInt(c.event_count)}</span> | |
| 190 | + </li> | |
| 191 | + ))} | |
| 192 | + {d.children.length > 12 && <li className="px-3 py-1.5 text-[11px] text-fg-subtle"><Link href={`/entity/${e.id}?tab=related`} className="hover:text-fg">+{d.children.length - 12} more →</Link></li>} | |
| 193 | + </ul> | |
| 194 | + </Panel> | |
| 195 | + )} | |
| 196 | + {d.relations.length > 0 && ( | |
| 197 | + <Panel title="Knowledge graph" dense> | |
| 198 | + <ul className="divide-y divide-line"> | |
| 199 | + {d.relations.slice(0, 10).map((r, i) => ( | |
| 200 | + <li key={`${r.from_id}-${r.relation}-${r.to_id}-${i}`} className="flex flex-wrap items-center gap-x-1.5 px-3 py-1.5 text-[12.5px]"> | |
| 201 | + <Link href={`/entity/${r.from_id}`} className={r.from_id === e.id ? "text-fg-muted" : "font-medium hover:underline"}>{r.from_name}</Link> | |
| 202 | + <span className="font-mono text-[10.5px] uppercase tracking-wider text-fg-subtle">{r.relation.replace(/_/g, " ")}</span> | |
| 203 | + <Link href={`/entity/${r.to_id}`} className={r.to_id === e.id ? "text-fg-muted" : "font-medium hover:underline"}>{r.to_name}</Link> | |
| 204 | + <Chip className="ml-auto">{typeLabel(r.to_type)}</Chip> | |
| 205 | + </li> | |
| 206 | + ))} | |
| 207 | + </ul> | |
| 208 | + </Panel> | |
| 209 | + )} | |
| 210 | + {d.aliases.length > 0 && ( | |
| 211 | + <Panel title="Aliases"> | |
| 212 | + <div className="flex flex-wrap gap-1">{d.aliases.map((a) => <Chip key={a} className="font-mono">{a}</Chip>)}</div> | |
| 213 | + </Panel> | |
| 214 | + )} | |
| 215 | + </aside> | |
| 216 | + </div> | |
| 217 | + ); | |
| 218 | +} | |
| 219 | + | |
| 220 | +function SensorsPanel({ sensors, title = "Most active sensors · 7 d" }: { sensors: EntityInsights["most_active_sensors"]; title?: string }) { | |
| 221 | + return ( | |
| 222 | + <Panel title={title} dense> | |
| 223 | + {sensors.length ? ( | |
| 224 | + <ul className="divide-y divide-line"> | |
| 225 | + {sensors.slice(0, 8).map((s) => ( | |
| 226 | + <li key={s.id} className="grid grid-cols-[1fr_auto_2.5rem] items-center gap-x-2 px-3 py-1.5 text-[12.5px] hover:bg-panel-2/60"> | |
| 227 | + <div className="min-w-0"> | |
| 228 | + <Link href={`/sensor/${s.id}`} className="block truncate font-medium hover:underline">{s.name}</Link> | |
| 229 | + <div className="flex min-w-0 items-center gap-1.5 font-mono text-[10.5px] text-fg-subtle tabular"> | |
| 230 | + <Link href={`/source/${s.source_id}`} className="truncate uppercase tracking-wide hover:text-fg">{s.source_id}</Link> | |
| 231 | + <span className="shrink-0">· {relTime(s.last_event_at)}</span> | |
| 232 | + </div> | |
| 233 | + </div> | |
| 234 | + <Chip className="font-mono">{s.type}</Chip> | |
| 235 | + <span className="text-right font-mono text-[12px] font-semibold text-signal tabular" title="Events in the last 7 days">{s.events_7d}</span> | |
| 236 | + </li> | |
| 237 | + ))} | |
| 238 | + </ul> | |
| 239 | + ) : ( | |
| 240 | + <Empty>No sensor produced an event for this entity in the last 7 days.</Empty> | |
| 241 | + )} | |
| 242 | + </Panel> | |
| 243 | + ); | |
| 244 | +} | |
| 245 | + | |
| 246 | +function SourcesTab({ d }: { d: EntityDetail }) { | |
| 247 | + return ( | |
| 248 | + <Panel title={<>Monitored sources <span className="font-mono text-fg-subtle">{d.sources.length}</span></>} dense> | |
| 249 | + {d.sources.length === 0 ? ( | |
| 250 | + <Empty>No monitored source is linked to {d.entity.name} yet. Events still resolve to it from third-party coverage.</Empty> | |
| 251 | + ) : ( | |
| 252 | + <Table head={["Source", "Tier", "Evidence", "Country", "Sensors", "24 h", ""]}> | |
| 253 | + {[...d.sources].sort((a, b) => (b.events_24h ?? 0) - (a.events_24h ?? 0) || Number(Boolean(b.first_party)) - Number(Boolean(a.first_party))).map((s) => ( | |
| 254 | + <tr key={s.id} className="hover:bg-panel-2/60"> | |
| 255 | + <Td> | |
| 256 | + <Link href={`/source/${s.id}`} className="font-medium hover:underline">{s.name}</Link> | |
| 257 | + <div className="truncate font-mono text-[11px] text-fg-subtle">{s.domain}</div> | |
| 258 | + </Td> | |
| 259 | + <Td><TierBadge tier={s.tier} /></Td> | |
| 260 | + <Td>{s.first_party ? <Badge kind="first-party" /> : <Badge kind="external" />}</Td> | |
| 261 | + <Td>{s.country ? <span className="inline-flex items-center gap-1.5"><Flag code={s.country} /><span className="font-mono text-[11px] text-fg-subtle">{s.country}</span></span> : <span className="text-fg-subtle">—</span>}</Td> | |
| 262 | + <Td mono>{s.sensor_count ?? 0}</Td> | |
| 263 | + <Td mono className={s.events_24h ? "text-signal" : "text-fg-subtle"}>{s.events_24h ?? 0}</Td> | |
| 264 | + <Td className="text-right"><Link href={`/source/${s.id}`} className="whitespace-nowrap text-[11.5px] text-fg-subtle hover:text-fg">sensors →</Link></Td> | |
| 265 | + </tr> | |
| 266 | + ))} | |
| 267 | + </Table> | |
| 268 | + )} | |
| 269 | + </Panel> | |
| 270 | + ); | |
| 271 | +} | |
| 272 | + | |
| 273 | +function SilentTab({ items, name }: { items: EventItem[]; name: string }) { | |
| 274 | + return ( | |
| 275 | + <Panel title={<span className="inline-flex items-center gap-2"><Badge kind="silent" /> Silent changes <span className="font-mono text-fg-subtle">{items.length}</span></span>} dense action={<Link href={`/live?entity=${encodeURIComponent(name)}&silent_change=true`} className="hidden text-[11px] text-fg-subtle hover:text-fg sm:inline">all silent →</Link>}> | |
| 276 | + {items.length === 0 ? ( | |
| 277 | + <Empty>No silent changes detected in this period.</Empty> | |
| 278 | + ) : ( | |
| 279 | + <ul className="divide-y divide-line"> | |
| 280 | + {items.map((ev) => ( | |
| 281 | + <li key={ev.id} className="grid grid-cols-[auto_1fr] gap-x-3 border-l-2 border-l-silent/60 px-3 py-2 sm:grid-cols-[6.25rem_1fr_auto]"> | |
| 282 | + <div className="flex flex-col font-mono text-[11px] leading-4 text-fg-subtle tabular"> | |
| 283 | + <time dateTime={ev.detected_at} title={utcDateTime(ev.detected_at)} className="text-fg-muted">{utcTime(ev.detected_at)}</time> | |
| 284 | + <span>{utcDate(ev.detected_at)}</span> | |
| 285 | + </div> | |
| 286 | + <div className="min-w-0 sm:col-start-2"> | |
| 287 | + <div className="flex flex-wrap items-center gap-x-2 gap-y-0.5 text-[11px]"> | |
| 288 | + <Link href={`/source/${ev.source?.id ?? ev.source_id}`} className="font-mono font-semibold uppercase tracking-wide text-fg-muted hover:text-fg">{ev.source?.name ?? ev.source_id}</Link> | |
| 289 | + {ev.country && <Flag code={ev.country} />} | |
| 290 | + <span className="sm:hidden"><Score value={ev.signal_score ?? ev.importance} size="sm" kind="signal" /></span> | |
| 291 | + <Chip>{typeLabel(ev.event_type)}</Chip> | |
| 292 | + {ev.change_class && <Chip tone="silent">{ev.change_class}</Chip>} | |
| 293 | + </div> | |
| 294 | + <Link href={`/event/${ev.slug}`} className="mt-0.5 block font-medium leading-snug hover:underline">{ev.title}</Link> | |
| 295 | + {ev.field_changes && ev.field_changes.length > 0 ? ( | |
| 296 | + <div className="mt-1.5 max-w-3xl"><FieldChanges items={ev.field_changes} compact max={4} /></div> | |
| 297 | + ) : ( | |
| 298 | + ev.summary && <p className="mt-0.5 line-clamp-2 text-[12.5px] text-fg-muted">{ev.summary}</p> | |
| 299 | + )} | |
| 300 | + </div> | |
| 301 | + <div className="hidden items-start pt-0.5 sm:flex"><Score value={ev.signal_score ?? ev.importance} kind="signal" /></div> | |
| 302 | + </li> | |
| 303 | + ))} | |
| 304 | + </ul> | |
| 305 | + )} | |
| 306 | + </Panel> | |
| 307 | + ); | |
| 308 | +} | |
| 309 | + | |
| 310 | +async function TimelineTab({ id, range, cursor }: { id: string; range?: string; cursor?: string }) { | |
| 311 | + const r = RANGES.find((x) => x.key === range) ?? RANGES[2]!; | |
| 312 | + const after = r.ms ? agoIso(r.ms) : undefined; | |
| 313 | + const page = await api.events({ entity: id, after, limit: 100, cursor }); | |
| 314 | + const groups = new Map<string, EventItem[]>(); | |
| 315 | + for (const ev of page.items) { | |
| 316 | + const day = utcDate(ev.detected_at); | |
| 317 | + if (!groups.has(day)) groups.set(day, []); | |
| 318 | + groups.get(day)!.push(ev); | |
| 319 | + } | |
| 320 | + const rangeHref = (k: string): string => `/entity/${id}?tab=timeline&range=${k}`; | |
| 321 | + return ( | |
| 322 | + <Panel | |
| 323 | + title={<>Timeline <span className="font-mono text-fg-subtle">{page.items.length}{page.nextCursor ? "+" : ""}</span></>} | |
| 324 | + dense | |
| 325 | + action={ | |
| 326 | + <nav className="flex items-center gap-0.5" aria-label="Range"> | |
| 327 | + {RANGES.map((x) => ( | |
| 328 | + <Link key={x.key} href={rangeHref(x.key)} aria-current={x.key === r.key ? "page" : undefined} className={`rounded-sm px-1.5 py-px font-mono text-[11px] tabular ${x.key === r.key ? "bg-panel-2 text-fg" : "text-fg-subtle hover:text-fg"}`}>{x.label}</Link> | |
| 329 | + ))} | |
| 330 | + </nav> | |
| 331 | + } | |
| 332 | + > | |
| 333 | + {page.items.length === 0 ? ( | |
| 334 | + <Empty>No event in the last {r.label.replace(" ", "")}{r.ms ? "" : " — nothing recorded yet"}. Try a wider range.</Empty> | |
| 335 | + ) : ( | |
| 336 | + <div> | |
| 337 | + {[...groups.entries()].map(([day, evs]) => ( | |
| 338 | + <div key={day}> | |
| 339 | + <div className="sticky top-12 z-10 flex items-center justify-between border-y border-line bg-panel-2/95 px-3 py-1 font-mono text-[11px] font-semibold tracking-wider text-fg-muted backdrop-blur"> | |
| 340 | + <span>{dayHeader(evs[0]!.detected_at)}</span> | |
| 341 | + <span className="font-normal text-fg-subtle tabular">{evs.length}</span> | |
| 342 | + </div> | |
| 343 | + {evs.map((ev) => <EventRow key={ev.id} ev={ev} />)} | |
| 344 | + </div> | |
| 345 | + ))} | |
| 346 | + </div> | |
| 347 | + )} | |
| 348 | + <div className="flex items-center justify-between px-3 py-2 text-[12px] text-fg-subtle"> | |
| 349 | + <span>{after ? <>since <span className="font-mono tabular">{utcDateTime(after)}</span></> : "complete history, newest first"}</span> | |
| 350 | + {page.nextCursor && <Link href={`${rangeHref(r.key)}&cursor=${encodeURIComponent(page.nextCursor)}`} className="rounded-md border border-line bg-panel-2 px-2.5 py-1 text-fg hover:border-line-strong">Older →</Link>} | |
| 351 | + </div> | |
| 352 | + </Panel> | |
| 353 | + ); | |
| 354 | +} | |
| 355 | + | |
| 356 | +function RelatedTab({ d }: { d: EntityDetail }) { | |
| 357 | + const related = d.related ?? []; | |
| 358 | + const max = Math.max(1, ...related.map((r) => r.shared_events)); | |
| 359 | + return ( | |
| 360 | + <div className="grid gap-4 xl:grid-cols-[1fr_1fr]"> | |
| 361 | + <Panel title={<>Co-occurring entities <span className="font-mono text-fg-subtle">{related.length}</span></>} dense> | |
| 362 | + {related.length === 0 ? ( | |
| 363 | + <Empty>No entity shares an event with {d.entity.name} yet.</Empty> | |
| 364 | + ) : ( | |
| 365 | + <Table head={["Entity", "Type", "Shared events", ""]}> | |
| 366 | + {related.map((r) => ( | |
| 367 | + <tr key={r.id} className="hover:bg-panel-2/60"> | |
| 368 | + <Td><Link href={`/entity/${r.id}`} className="font-medium hover:underline">{r.name}</Link></Td> | |
| 369 | + <Td><Chip>{typeLabel(r.type)}</Chip></Td> | |
| 370 | + <Td mono>{r.shared_events}</Td> | |
| 371 | + <Td className="w-32 min-w-24 pt-3"><Bar value={r.shared_events} max={max} tone="info" /></Td> | |
| 372 | + </tr> | |
| 373 | + ))} | |
| 374 | + </Table> | |
| 375 | + )} | |
| 376 | + </Panel> | |
| 377 | + <div className="flex flex-col gap-4"> | |
| 378 | + <Panel title={<>Products & children <span className="font-mono text-fg-subtle">{d.children.length}</span></>} dense> | |
| 379 | + {d.children.length === 0 ? ( | |
| 380 | + <Empty>No child entity.</Empty> | |
| 381 | + ) : ( | |
| 382 | + <Table head={["Entity", "Type", "Events", "Last"]}> | |
| 383 | + {d.children.map((c) => ( | |
| 384 | + <tr key={c.id} className="hover:bg-panel-2/60"> | |
| 385 | + <Td><Link href={`/entity/${c.id}`} className="font-medium hover:underline">{c.name}</Link></Td> | |
| 386 | + <Td><Chip>{typeLabel(c.type)}</Chip></Td> | |
| 387 | + <Td mono>{fmtInt(c.event_count)}</Td> | |
| 388 | + <Td mono className="whitespace-nowrap text-fg-subtle">{c.last_event_at ? relTime(c.last_event_at) : "—"}</Td> | |
| 389 | + </tr> | |
| 390 | + ))} | |
| 391 | + </Table> | |
| 392 | + )} | |
| 393 | + </Panel> | |
| 394 | + <Panel title={<>Knowledge graph <span className="font-mono text-fg-subtle">{d.relations.length}</span></>} dense> | |
| 395 | + {d.relations.length === 0 ? ( | |
| 396 | + <Empty>No relation recorded.</Empty> | |
| 397 | + ) : ( | |
| 398 | + <ul className="divide-y divide-line"> | |
| 399 | + {d.relations.map((r, i) => ( | |
| 400 | + <li key={`${r.from_id}-${r.relation}-${r.to_id}-${i}`} className="flex flex-wrap items-center gap-x-1.5 px-3 py-1.5 text-[12.5px]"> | |
| 401 | + <Link href={`/entity/${r.from_id}`} className={r.from_id === d.entity.id ? "text-fg-muted" : "font-medium hover:underline"}>{r.from_name}</Link> | |
| 402 | + <span className="font-mono text-[10.5px] uppercase tracking-wider text-fg-subtle">{r.relation.replace(/_/g, " ")}</span> | |
| 403 | + <Link href={`/entity/${r.to_id}`} className={r.to_id === d.entity.id ? "text-fg-muted" : "font-medium hover:underline"}>{r.to_name}</Link> | |
| 404 | + <Chip className="ml-auto">{typeLabel(r.to_type)}</Chip> | |
| 405 | + </li> | |
| 406 | + ))} | |
| 407 | + </ul> | |
| 408 | + )} | |
| 409 | + </Panel> | |
| 410 | + {d.aliases.length > 0 && ( | |
| 411 | + <Panel title="Aliases"> | |
| 412 | + <div className="flex flex-wrap gap-1">{d.aliases.map((a) => <Chip key={a} className="font-mono">{a}</Chip>)}</div> | |
| 413 | + </Panel> | |
| 414 | + )} | |
| 415 | + </div> | |
| 416 | + </div> | |
| 417 | + ); | |
| 418 | +} | |
| 419 | + | |
| 420 | +function MetricsTab({ d, ins }: { d: EntityDetail; ins: EntityInsights }) { | |
| 421 | + const maxType = Math.max(1, ...d.by_type.map((t) => t.n)); | |
| 422 | + const total = d.by_type.reduce((n, t) => n + t.n, 0); | |
| 423 | + const anomalous = ins.anomaly.score >= 60; | |
| 424 | + const days7 = ins.heatmap.slice(-7).reduce((n, x) => n + x.events, 0); | |
| 425 | + const days35 = ins.heatmap.reduce((n, x) => n + x.events, 0); | |
| 426 | + const peak = ins.heatmap.reduce<{ day: string; events: number } | null>((best, x) => (!best || x.events > best.events ? { day: x.day, events: x.events } : best), null); | |
| 427 | + return ( | |
| 428 | + <div className="grid gap-4 xl:grid-cols-[1fr_1fr]"> | |
| 429 | + <div className="flex flex-col gap-4"> | |
| 430 | + <Panel title={<>Event types <span className="font-mono text-fg-subtle">{fmtInt(total)}</span></>}> | |
| 431 | + {d.by_type.length ? ( | |
| 432 | + <ul className="space-y-1.5"> | |
| 433 | + {d.by_type.map((t) => ( | |
| 434 | + <li key={t.event_type} className="grid grid-cols-[9rem_1fr_3.5rem] items-center gap-2 text-[12px] sm:grid-cols-[11rem_1fr_3.5rem]"> | |
| 435 | + <Link href={`/live?entity=${encodeURIComponent(d.entity.id)}&event_type=${t.event_type}`} className="truncate hover:underline">{typeLabel(t.event_type)}</Link> | |
| 436 | + <Bar value={t.n} max={maxType} tone="info" /> | |
| 437 | + <span className="text-right font-mono text-fg-subtle tabular">{t.n} <span className="text-[10.5px]">{total ? Math.round((t.n / total) * 100) : 0}%</span></span> | |
| 438 | + </li> | |
| 439 | + ))} | |
| 440 | + </ul> | |
| 441 | + ) : ( | |
| 442 | + <Empty>No events yet.</Empty> | |
| 443 | + )} | |
| 444 | + </Panel> | |
| 445 | + <SensorsPanel sensors={ins.most_active_sensors} /> | |
| 446 | + </div> | |
| 447 | + <div className="flex flex-col gap-4"> | |
| 448 | + <Panel title="Anomaly"> | |
| 449 | + <div className="flex items-baseline justify-between"> | |
| 450 | + <span className={`font-mono text-3xl font-semibold tabular ${anomalous ? "text-hot" : ins.anomaly.score >= 40 ? "text-warn" : "text-fg"}`}>{ins.anomaly.score > 0 ? fmtScore(ins.anomaly.score) : "0"}</span> | |
| 451 | + <span className={`label ${anomalous ? "!text-hot" : ""}`}>{anomalous ? "ANOMALOUS ACTIVITY" : ins.anomaly.score >= 40 ? "ELEVATED" : "NORMAL"}</span> | |
| 452 | + </div> | |
| 453 | + <div className="mt-2"><Bar value={ins.anomaly.score} tone={anomalous ? "hot" : ins.anomaly.score >= 40 ? "high" : "signal"} /></div> | |
| 454 | + <dl className="mt-3 grid grid-cols-2 gap-y-1 text-[12.5px]"> | |
| 455 | + <dt className="text-fg-subtle">30-day baseline</dt><dd className="text-right font-mono tabular">{ins.baseline_per_day.toFixed(1)} events/day</dd> | |
| 456 | + <dt className="text-fg-subtle">Last 24 h</dt><dd className="text-right font-mono tabular">{fmtInt(ins.events_24h)} events</dd> | |
| 457 | + <dt className="text-fg-subtle">Previous 24 h</dt><dd className="text-right font-mono tabular">{fmtInt(ins.events_prev_24h)} events</dd> | |
| 458 | + <dt className="text-fg-subtle">Velocity ratio</dt><dd className="text-right font-mono tabular">×{ins.velocity_ratio.toFixed(2)}</dd> | |
| 459 | + <dt className="text-fg-subtle">vs baseline</dt><dd className={`text-right font-mono tabular ${anomalous ? "font-semibold text-hot" : ""}`}>{ins.anomaly.pct > 0 ? "+" : ""}{Math.round(ins.anomaly.pct)}% (×{ins.anomaly.ratio.toFixed(2)})</dd> | |
| 460 | + </dl> | |
| 461 | + <p className="mt-3 text-[12px] leading-relaxed text-fg-muted"> | |
| 462 | + The anomaly score compares the last 24 h with this entity's own 30-day baseline, so a busy organization is not flagged for being busy. Scores ≥ 60 mark anomalous activity; the ratio above 1 means more events than usual. | |
| 463 | + </p> | |
| 464 | + </Panel> | |
| 465 | + <Panel title="35-day activity"> | |
| 466 | + <Heatmap days={ins.heatmap} /> | |
| 467 | + <dl className="mt-3 grid grid-cols-3 gap-2 text-[12.5px]"> | |
| 468 | + <div><dt className="label">7 d</dt><dd className="font-mono text-base font-semibold tabular">{fmtInt(days7)}</dd></div> | |
| 469 | + <div><dt className="label">35 d</dt><dd className="font-mono text-base font-semibold tabular">{fmtInt(days35)}</dd></div> | |
| 470 | + <div className="min-w-0"><dt className="label">Peak day</dt><dd className="truncate font-mono text-base font-semibold tabular">{peak && peak.events ? <>{peak.events} <span className="text-[11px] font-normal text-fg-subtle">{peak.day}</span></> : "—"}</dd></div> | |
| 471 | + </dl> | |
| 472 | + </Panel> | |
| 473 | + <div className="grid grid-cols-2 gap-px overflow-hidden rounded-lg border border-line bg-line"> | |
| 474 | + <div className="bg-panel"><Stat label="Total events" value={fmtInt(d.entity.event_count)} hint={d.entity.last_event_at ? `last ${relTime(d.entity.last_event_at)}` : "none yet"} /></div> | |
| 475 | + <div className="bg-panel"><Stat label="Entity importance" value={fmtScore(d.entity.importance)} hint="weight in scoring" /></div> | |
| 476 | + </div> | |
| 477 | + </div> | |
| 478 | + </div> | |
| 479 | + ); | |
| 480 | +} | |
modified
apps/web/src/app/error.tsx
+26 −7
@@ -1,14 +1,33 @@ | ||
| 1 | 1 | "use client"; |
| 2 | 2 | |
| 3 | +import { AlertTriangle, RotateCcw } from "lucide-react"; | |
| 4 | +import Link from "next/link"; | |
| 5 | +import { useEffect } from "react"; | |
| 6 | +import { Kbd, Panel } from "@/components/ui"; | |
| 7 | + | |
| 3 | 8 | export default function ErrorPage({ error, reset }: { error: Error & { digest?: string }; reset: () => void }) { |
| 9 | + useEffect(() => { | |
| 10 | + console.error(error); | |
| 11 | + }, [error]); | |
| 4 | 12 | return ( |
| 5 | − <div className="mx-auto max-w-lg py-20 text-center"> | |
| 6 | − <div className="label mb-2">Error</div> | |
| 7 | − <h1 className="text-xl font-semibold">Something went wrong while rendering</h1> | |
| 8 | − <p className="mt-2 font-mono text-[12px] text-fg-subtle">{error.digest ?? error.message}</p> | |
| 9 | − <button type="button" onClick={reset} className="mt-5 rounded-md border border-line bg-panel px-3 py-1.5 text-[13px] hover:border-line-strong"> | |
| 10 | − Try again | |
| 11 | − </button> | |
| 13 | + <div className="mx-auto max-w-lg py-12 sm:py-20"> | |
| 14 | + <Panel dense> | |
| 15 | + <div className="flex flex-col items-center gap-2 px-4 py-10 text-center"> | |
| 16 | + <span className="font-mono text-[11px] font-semibold tracking-wider text-danger">RENDER ERROR</span> | |
| 17 | + <AlertTriangle className="mt-1 size-6 text-danger" aria-hidden /> | |
| 18 | + <h1 className="mt-1 text-lg font-semibold leading-tight">Something went wrong while rendering</h1> | |
| 19 | + <p className="max-w-sm text-[13px] text-fg-muted">The sensors and the API are unaffected; only this view failed. Retrying usually works — the gateway may have been warming up.</p> | |
| 20 | + {(error.digest || error.message) && <code className="mt-1 max-w-full truncate rounded-sm border border-line bg-panel-2 px-2 py-0.5 font-mono text-[11px] text-fg-subtle" title={error.message}>{error.digest ?? error.message}</code>} | |
| 21 | + <div className="mt-4 flex flex-wrap justify-center gap-2 text-[12.5px]"> | |
| 22 | + <button type="button" onClick={reset} className="inline-flex h-8 items-center gap-1.5 rounded-md border border-signal/40 bg-signal-soft px-3 text-signal hover:border-signal"><RotateCcw className="size-3.5" /> Try again</button> | |
| 23 | + <Link href="/live" className="inline-flex h-8 items-center rounded-md border border-line bg-panel px-3 hover:border-line-strong">Live feed</Link> | |
| 24 | + <Link href="/explore" className="inline-flex h-8 items-center rounded-md border border-line bg-panel px-3 hover:border-line-strong">Explore</Link> | |
| 25 | + </div> | |
| 26 | + <p className="mt-3 flex items-center gap-1.5 text-[11px] text-fg-subtle"> | |
| 27 | + <Kbd>⌘</Kbd><Kbd>K</Kbd> opens search from any page. | |
| 28 | + </p> | |
| 29 | + </div> | |
| 30 | + </Panel> | |
| 12 | 31 | </div> |
| 13 | 32 | ); |
| 14 | 33 | } |
added
apps/web/src/app/event/[slug]/loading.tsx
+57 −0
@@ -0,0 +1,57 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-4 flex flex-col gap-2"> | |
| 7 | + <div className="flex gap-2"> | |
| 8 | + <Skeleton className="h-4 w-24" /> | |
| 9 | + <Skeleton className="h-4 w-16" /> | |
| 10 | + <Skeleton className="h-4 w-20" /> | |
| 11 | + </div> | |
| 12 | + <Skeleton className="h-7 w-11/12 max-w-3xl" /> | |
| 13 | + <Skeleton className="h-7 w-2/3 max-w-xl" /> | |
| 14 | + </div> | |
| 15 | + <div className="grid gap-4 lg:grid-cols-[minmax(0,1fr)_340px]"> | |
| 16 | + <div className="flex flex-col gap-4"> | |
| 17 | + <SkeletonPanel lines={4} /> | |
| 18 | + <div className="panel overflow-hidden"> | |
| 19 | + <div className="flex gap-1 border-b border-line px-2 py-1.5"> | |
| 20 | + <Skeleton className="h-6 w-16" /> | |
| 21 | + <Skeleton className="h-6 w-24" /> | |
| 22 | + <Skeleton className="h-6 w-20" /> | |
| 23 | + </div> | |
| 24 | + <div className="flex flex-col gap-1.5 p-3"> | |
| 25 | + {Array.from({ length: 8 }, (_, i) => ( | |
| 26 | + <Skeleton key={i} className={`h-3.5 ${i % 3 === 0 ? "w-11/12" : i % 3 === 1 ? "w-3/4" : "w-5/6"}`} /> | |
| 27 | + ))} | |
| 28 | + </div> | |
| 29 | + </div> | |
| 30 | + <SkeletonPanel lines={5} /> | |
| 31 | + <div className="panel"> | |
| 32 | + <Skeleton className="m-3 h-2.5 w-20" /> | |
| 33 | + <SkeletonRows rows={4} /> | |
| 34 | + </div> | |
| 35 | + </div> | |
| 36 | + <aside className="flex flex-col gap-4"> | |
| 37 | + <div className="panel p-3"> | |
| 38 | + <div className="flex items-center gap-3"> | |
| 39 | + <Skeleton className="h-9 w-14" /> | |
| 40 | + <Skeleton className="h-3 w-32" /> | |
| 41 | + </div> | |
| 42 | + <div className="mt-3 grid grid-cols-2 gap-x-4 gap-y-3"> | |
| 43 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 44 | + <div key={i} className="flex flex-col gap-1.5"> | |
| 45 | + <Skeleton className="h-2.5 w-16" /> | |
| 46 | + <Skeleton className="h-1 w-full" /> | |
| 47 | + </div> | |
| 48 | + ))} | |
| 49 | + </div> | |
| 50 | + </div> | |
| 51 | + <SkeletonPanel lines={7} /> | |
| 52 | + <SkeletonPanel lines={3} /> | |
| 53 | + </aside> | |
| 54 | + </div> | |
| 55 | + </> | |
| 56 | + ); | |
| 57 | +} | |
added
apps/web/src/app/event/[slug]/opengraph-image.tsx
+89 −0
@@ -0,0 +1,89 @@ | ||
| 1 | +import { ImageResponse } from "next/og"; | |
| 2 | +import { api } from "@/lib/api"; | |
| 3 | +import { fmtScore, utcDateTime } from "@/lib/format"; | |
| 4 | + | |
| 5 | +export const runtime = "nodejs"; | |
| 6 | +export const alt = "WebSensor event"; | |
| 7 | +export const size = { width: 1200, height: 630 }; | |
| 8 | +export const contentType = "image/png"; | |
| 9 | + | |
| 10 | +const BG = "#0a0e15"; | |
| 11 | +const PANEL = "#10161f"; | |
| 12 | +const LINE = "#2d394e"; | |
| 13 | +const FG = "#e6e8ee"; | |
| 14 | +const MUTED = "#a3adc2"; | |
| 15 | +const SUBTLE = "#6b7591"; | |
| 16 | +const SIGNAL = "#22d3a5"; | |
| 17 | +const HOT = "#ff5c5c"; | |
| 18 | +const HIGH = "#ff9640"; | |
| 19 | +const MID = "#e4b53b"; | |
| 20 | +const SILENT = "#a78bfa"; | |
| 21 | + | |
| 22 | +function scoreColor(v: number): string { | |
| 23 | + return v >= 90 ? HOT : v >= 75 ? HIGH : v >= 50 ? MID : SUBTLE; | |
| 24 | +} | |
| 25 | + | |
| 26 | +function clip(s: string, max: number): string { | |
| 27 | + return s.length > max ? s.slice(0, max - 1).trimEnd() + "…" : s; | |
| 28 | +} | |
| 29 | + | |
| 30 | +export default async function OpenGraphImage({ params }: { params: Promise<{ slug: string }> }) { | |
| 31 | + const { slug } = await params; | |
| 32 | + const d = await api.event(slug); | |
| 33 | + const e = d?.event ?? null; | |
| 34 | + const signal = e ? Math.round(e.signal_score ?? e.importance) : 0; | |
| 35 | + const badges: { label: string; color: string }[] = []; | |
| 36 | + if (e?.cluster?.state === "breaking") badges.push({ label: "BREAKING", color: HOT }); | |
| 37 | + if (e?.cluster?.state === "developing") badges.push({ label: "DEVELOPING", color: HIGH }); | |
| 38 | + if (e?.silent_change) badges.push({ label: "SILENT", color: SILENT }); | |
| 39 | + if (e && e.first_party !== false) badges.push({ label: "FIRST PARTY", color: SIGNAL }); | |
| 40 | + if (e?.first_party === false) badges.push({ label: "EXTERNAL", color: SUBTLE }); | |
| 41 | + if (e?.evidence_label === "CONFIRMED") badges.push({ label: "CONFIRMED", color: SIGNAL }); | |
| 42 | + const title = e ? clip(e.title, 150) : "Event not found"; | |
| 43 | + const fontSize = title.length > 110 ? 40 : title.length > 70 ? 46 : 54; | |
| 44 | + | |
| 45 | + return new ImageResponse( | |
| 46 | + ( | |
| 47 | + <div style={{ width: "100%", height: "100%", display: "flex", flexDirection: "column", justifyContent: "space-between", padding: 56, background: BG, color: FG, fontFamily: "Inter, system-ui, sans-serif" }}> | |
| 48 | + <div style={{ display: "flex", alignItems: "center", justifyContent: "space-between" }}> | |
| 49 | + <div style={{ display: "flex", alignItems: "center", gap: 14, fontSize: 30, fontWeight: 700 }}> | |
| 50 | + <div style={{ width: 16, height: 16, borderRadius: 16, background: SIGNAL, boxShadow: "0 0 0 8px rgba(34,211,165,0.22)" }} /> | |
| 51 | + <span> | |
| 52 | + Web<span style={{ color: SIGNAL }}>Sensor</span> | |
| 53 | + </span> | |
| 54 | + </div> | |
| 55 | + <div style={{ display: "flex", fontSize: 22, color: SUBTLE, fontFamily: "monospace", letterSpacing: 2 }}>{e ? (e.event_type ?? "").replace(/_/g, " ").toUpperCase() : ""}</div> | |
| 56 | + </div> | |
| 57 | + | |
| 58 | + <div style={{ display: "flex", flexDirection: "column", gap: 20, flex: 1, justifyContent: "center", paddingTop: 24, paddingBottom: 24 }}> | |
| 59 | + <div style={{ display: "flex", alignItems: "center", gap: 14, fontFamily: "monospace", fontSize: 26, color: MUTED, letterSpacing: 3, fontWeight: 700 }}> | |
| 60 | + <span>{(e?.source?.name ?? "").toUpperCase()}</span> | |
| 61 | + {e?.country ? <span style={{ color: SUBTLE, fontSize: 22 }}>· {e.country}</span> : null} | |
| 62 | + </div> | |
| 63 | + <div style={{ display: "flex", flexDirection: "row", gap: 32, alignItems: "flex-start" }}> | |
| 64 | + <div style={{ display: "flex", flex: 1, fontSize, fontWeight: 700, letterSpacing: -1.2, lineHeight: 1.12, color: FG, overflow: "hidden", maxHeight: fontSize * 1.12 * 3 + 6 }}>{title}</div> | |
| 65 | + <div style={{ display: "flex", flexDirection: "column", alignItems: "center", justifyContent: "center", width: 200, height: 160, borderRadius: 12, border: `2px solid ${scoreColor(signal)}`, background: PANEL, flexShrink: 0 }}> | |
| 66 | + <div style={{ display: "flex", fontFamily: "monospace", fontSize: 88, fontWeight: 700, color: scoreColor(signal), lineHeight: 1 }}>{e ? fmtScore(signal) : "—"}</div> | |
| 67 | + <div style={{ display: "flex", fontSize: 16, letterSpacing: 3, color: SUBTLE, marginTop: 10, fontWeight: 600 }}>SIGNAL</div> | |
| 68 | + </div> | |
| 69 | + </div> | |
| 70 | + {badges.length > 0 && ( | |
| 71 | + <div style={{ display: "flex", gap: 10 }}> | |
| 72 | + {badges.map((b) => ( | |
| 73 | + <div key={b.label} style={{ display: "flex", padding: "6px 12px", borderRadius: 6, border: `1.5px solid ${b.color}`, color: b.color, fontFamily: "monospace", fontSize: 20, fontWeight: 700, letterSpacing: 2 }}> | |
| 74 | + {b.label} | |
| 75 | + </div> | |
| 76 | + ))} | |
| 77 | + </div> | |
| 78 | + )} | |
| 79 | + </div> | |
| 80 | + | |
| 81 | + <div style={{ display: "flex", justifyContent: "space-between", alignItems: "center", borderTop: `1px solid ${LINE}`, paddingTop: 20, fontSize: 22, color: SUBTLE, fontFamily: "monospace" }}> | |
| 82 | + <span>www.websensor.io</span> | |
| 83 | + <span>{e ? `DETECTED ${utcDateTime(e.detected_at)}` : "LIVE · UTC"}</span> | |
| 84 | + </div> | |
| 85 | + </div> | |
| 86 | + ), | |
| 87 | + { ...size }, | |
| 88 | + ); | |
| 89 | +} | |
modified
apps/web/src/app/event/[slug]/page.tsx
+248 −113
@@ -1,12 +1,16 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | 2 | import type { Metadata } from "next"; |
| 3 | 3 | import { notFound } from "next/navigation"; |
| 4 | +import type { ReactNode } from "react"; | |
| 5 | +import { IMPORTANCE_WEIGHTS } from "@websensor/core/client"; | |
| 4 | 6 | import { DiffViewer } from "@/components/diff-viewer"; |
| 7 | +import { BookmarkButton } from "@/components/event-drawer"; | |
| 5 | 8 | import { EventRow } from "@/components/event-row"; |
| 6 | −import { Chip, Empty, EvidenceTag, ExtLink, Gauge, HealthPill, PageHeader, Panel, Score, SilentBadge, Table, Td, TypeChip } from "@/components/ui"; | |
| 9 | +import { FieldChangeInline, FieldChanges } from "@/components/field-changes"; | |
| 10 | +import { Badge, Chip, Empty, EvidenceTag, ExtLink, Flag, Gauge, HealthPill, PageHeader, Panel, Score, StateBadge, Table, Td, TypeChip } from "@/components/ui"; | |
| 7 | 11 | import { ShareButton, WatchButton } from "@/components/watch-button"; |
| 8 | −import { api, SITE_URL } from "@/lib/api"; | |
| 9 | −import { fmtBytes, fmtMs, fmtPct, fmtScore, relTime, shortHash, typeLabel, utcDateTime } from "@/lib/format"; | |
| 12 | +import { api, SITE_URL, type SnapshotRow } from "@/lib/api"; | |
| 13 | +import { CLASS_LABELS, fmtBytes, fmtMs, fmtOffset, fmtPct, fmtScore, relTime, shortHash, typeLabel, utcDateTime } from "@/lib/format"; | |
| 10 | 14 | |
| 11 | 15 | export const dynamic = "force-dynamic"; |
| 12 | 16 | |
@@ -25,16 +29,23 @@ export async function generateMetadata({ params }: { params: Promise<{ slug: str | ||
| 25 | 29 | }; |
| 26 | 30 | } |
| 27 | 31 | |
| 28 | −const COMPONENT_WEIGHTS: [string, string, number][] = [ | |
| 29 | − ["severity", "Intrinsic event severity", 25], | |
| 30 | − ["source", "Source importance", 20], | |
| 31 | − ["entity", "Entity importance", 15], | |
| 32 | − ["novelty", "Novelty", 15], | |
| 33 | − ["magnitude", "Magnitude of change", 10], | |
| 34 | − ["confirmation", "Cross-source confirmation", 5], | |
| 35 | − ["userImpact", "User impact", 5], | |
| 36 | − ["unusualness", "Unusualness", 5], | |
| 37 | −]; | |
| 32 | +function Row({ label, value, tone = "" }: { label: string; value: ReactNode; tone?: string }) { | |
| 33 | + return ( | |
| 34 | + <> | |
| 35 | + <dt className="text-fg-subtle">{label}</dt> | |
| 36 | + <dd className={`text-right font-mono tabular ${tone}`}>{value}</dd> | |
| 37 | + </> | |
| 38 | + ); | |
| 39 | +} | |
| 40 | + | |
| 41 | +function Cell({ v, l }: { v: number | string; l: string }) { | |
| 42 | + return ( | |
| 43 | + <div className="rounded-md border border-line bg-panel-2 px-2 py-1.5 text-center"> | |
| 44 | + <div className="font-mono text-base font-semibold tabular">{v}</div> | |
| 45 | + <div className="label !text-[9.5px]">{l}</div> | |
| 46 | + </div> | |
| 47 | + ); | |
| 48 | +} | |
| 38 | 49 | |
| 39 | 50 | export default async function EventPage({ params }: { params: Promise<{ slug: string }> }) { |
| 40 | 51 | const { slug } = await params; |
@@ -50,6 +61,16 @@ export default async function EventPage({ params }: { params: Promise<{ slug: st | ||
| 50 | 61 | const oldSnap = d.snapshots.find((s) => s.id === e.old_snapshot_id); |
| 51 | 62 | const newSnap = d.snapshots.find((s) => s.id === e.new_snapshot_id); |
| 52 | 63 | const subject = e.entities.find((x) => x.role === "subject") ?? e.entities[0]; |
| 64 | + const signal = e.signal_score ?? e.importance; | |
| 65 | + const cluster = e.cluster; | |
| 66 | + const clusterHref = cluster ? `/cluster/${cluster.slug ?? cluster.id}` : d.cluster ? `/cluster/${d.cluster.slug ?? d.cluster.id}` : null; | |
| 67 | + const leadTime = cluster?.lead_time_ms ?? d.cluster?.lead_time_ms ?? null; | |
| 68 | + const hasFields = (e.field_changes?.length ?? 0) > 0; | |
| 69 | + const diffKind = change?.change.kind ?? d.change?.kind ?? null; | |
| 70 | + const diffSummary = change?.change.diff ?? d.change?.diff ?? null; | |
| 71 | + const defaultTab = diffKind === "text" ? "split" : "semantic"; | |
| 72 | + const reasons = e.score_reasons ?? []; | |
| 73 | + const history = d.history ?? []; | |
| 53 | 74 | const jsonLd = { |
| 54 | 75 | "@context": "https://schema.org", |
| 55 | 76 | "@type": "NewsArticle", |
@@ -59,136 +80,267 @@ export default async function EventPage({ params }: { params: Promise<{ slug: st | ||
| 59 | 80 | dateModified: e.processed_at ?? e.detected_at, |
| 60 | 81 | url: `${SITE_URL}/event/${e.slug}`, |
| 61 | 82 | mainEntityOfPage: `${SITE_URL}/event/${e.slug}`, |
| 83 | + image: [`${SITE_URL}/event/${e.slug}/opengraph-image`], | |
| 62 | 84 | author: { "@type": "Organization", name: "WebSensor", url: SITE_URL }, |
| 63 | 85 | publisher: { "@type": "Organization", name: "WebSensor", url: SITE_URL }, |
| 64 | − about: e.entities.map((x) => ({ "@type": "Thing", name: x.name, url: `${SITE_URL}/company/${x.id}` })), | |
| 86 | + about: e.entities.map((x) => ({ "@type": "Thing", name: x.name, url: `${SITE_URL}/entity/${x.id}` })), | |
| 65 | 87 | isBasedOn: e.url, |
| 66 | 88 | keywords: [e.event_type, ...e.categories, ...e.keywords].join(", "), |
| 67 | 89 | }; |
| 90 | + const snapRows: [string, SnapshotRow | undefined][] = [ | |
| 91 | + ["Before", oldSnap], | |
| 92 | + ["After", newSnap], | |
| 93 | + ]; | |
| 68 | 94 | return ( |
| 69 | 95 | <> |
| 70 | 96 | <script type="application/ld+json" dangerouslySetInnerHTML={{ __html: JSON.stringify(jsonLd) }} /> |
| 71 | 97 | <PageHeader |
| 72 | 98 | kicker={ |
| 73 | − <span className="flex flex-wrap items-center gap-2"> | |
| 74 | − <Link href={`/source/${e.source?.id ?? e.source_id}`} className="font-mono uppercase tracking-wide text-fg-muted hover:text-fg">{e.source?.name}</Link> | |
| 75 | − <TypeChip type={e.event_type} /> | |
| 76 | − {e.silent_change && <SilentBadge />} | |
| 99 | + <span className="flex flex-wrap items-center gap-1.5"> | |
| 100 | + <Link href={`/source/${e.source?.id ?? e.source_id}`} className="font-mono text-[11px] font-semibold uppercase tracking-wide text-fg-muted hover:text-fg">{e.source?.name ?? e.source_id}</Link> | |
| 101 | + {e.country && <Flag code={e.country} className="text-[12px]" />} | |
| 102 | + <StateBadge state={cluster?.state} /> | |
| 103 | + {e.silent_change && <Badge kind="silent" />} | |
| 104 | + {e.first_party === false ? <Badge kind="external" /> : <Badge kind="first-party" />} | |
| 77 | 105 | <EvidenceTag label={e.evidence_label} /> |
| 106 | + <TypeChip type={e.event_type} href={`/live?event_type=${e.event_type}`} /> | |
| 78 | 107 | {e.categories.map((c) => <Chip key={c} href={`/category/${c}`}>{c}</Chip>)} |
| 79 | 108 | </span> |
| 80 | 109 | } |
| 81 | − title={e.title} | |
| 110 | + title={<span className="[overflow-wrap:anywhere]">{e.title}</span>} | |
| 82 | 111 | actions={ |
| 83 | 112 | <> |
| 84 | 113 | <ShareButton path={`/event/${e.slug}`} /> |
| 114 | + <BookmarkButton eventId={e.id} label /> | |
| 85 | 115 | {subject && <WatchButton kind="entity" value={subject.id} label={`Watch ${subject.name}`} />} |
| 86 | − <Link href="/alerts" className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">Alert</Link> | |
| 116 | + <Link href={subject ? `/alerts?entity=${encodeURIComponent(subject.id)}` : "/alerts"} className="inline-flex h-7 items-center rounded-md border border-line bg-panel px-2.5 text-[12px] hover:border-line-strong">Alert</Link> | |
| 87 | 117 | </> |
| 88 | 118 | } |
| 89 | 119 | /> |
| 90 | − <div className="grid gap-4 lg:grid-cols-[1fr_340px]"> | |
| 91 | − <div className="flex flex-col gap-4"> | |
| 92 | − <Panel> | |
| 93 | − <p className="text-[15px] leading-relaxed">{e.summary}</p> | |
| 94 | − {e.why_it_matters && ( | |
| 95 | − <div className="mt-4"> | |
| 96 | − <div className="label mb-1">Why it matters</div> | |
| 97 | − <p className="text-[13.5px] leading-relaxed text-fg-muted">{e.why_it_matters}</p> | |
| 98 | − </div> | |
| 120 | + <div className="grid gap-4 lg:grid-cols-[minmax(0,1fr)_340px]"> | |
| 121 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 122 | + {/* WHAT CHANGED */} | |
| 123 | + <Panel title={<span>What changed{e.change_class && <span className="ml-2 normal-case tracking-normal text-fg-subtle">· {CLASS_LABELS[e.change_class] ?? e.change_class}</span>}</span>} action={<span className="font-mono text-[11px] text-fg-subtle tabular">{utcDateTime(e.detected_at)}</span>}> | |
| 124 | + {hasFields ? ( | |
| 125 | + <> | |
| 126 | + <FieldChanges items={e.field_changes} /> | |
| 127 | + <p className="mt-3 text-[13px] leading-relaxed text-fg-muted [overflow-wrap:anywhere]">{e.summary}</p> | |
| 128 | + </> | |
| 129 | + ) : ( | |
| 130 | + <p className="text-[15px] leading-relaxed [overflow-wrap:anywhere]">{e.summary}</p> | |
| 99 | 131 | )} |
| 100 | − {who && ( | |
| 101 | − <div className="mt-3"> | |
| 102 | − <div className="label mb-1">Who it affects</div> | |
| 103 | − <p className="text-[13.5px] text-fg-muted">{who}</p> | |
| 104 | − </div> | |
| 132 | + </Panel> | |
| 133 | + | |
| 134 | + {/* BEFORE / AFTER */} | |
| 135 | + <div id="diff" className="scroll-mt-16"> | |
| 136 | + <div className="mb-2 flex flex-wrap items-center justify-between gap-2"> | |
| 137 | + <h2 className="label">Before / after{diffKind ? <span className="ml-2 normal-case tracking-normal text-fg-subtle">· {diffKind} diff</span> : null}</h2> | |
| 138 | + <span className="flex flex-wrap gap-3 text-[11px] text-fg-subtle"> | |
| 139 | + {oldSnap && newSnap && <Link href={`/compare?a=${oldSnap.id}&b=${newSnap.id}`} className="hover:text-fg">open in compare →</Link>} | |
| 140 | + <Link href={`/url?u=${encodeURIComponent(e.url)}`} className="hover:text-fg">URL history →</Link> | |
| 141 | + </span> | |
| 142 | + </div> | |
| 143 | + {change || diffSummary ? ( | |
| 144 | + <DiffViewer unified={change?.unified ?? null} summary={diffSummary} defaultTab={defaultTab} /> | |
| 145 | + ) : ( | |
| 146 | + <Panel dense> | |
| 147 | + <Empty>No diff is attached to this event{e.event_type === "page_removed" ? " — the page disappeared; its last snapshot is preserved below." : "."}</Empty> | |
| 148 | + </Panel> | |
| 105 | 149 | )} |
| 106 | − <div className="mt-4 grid gap-3 sm:grid-cols-2"> | |
| 150 | + </div> | |
| 151 | + | |
| 152 | + {/* WHY IT MATTERS */} | |
| 153 | + <Panel title={<span>Why it matters <span className="normal-case tracking-normal text-fg-subtle">· analysis</span></span>}> | |
| 154 | + {e.why_it_matters ? <p className="text-[13.5px] leading-relaxed">{e.why_it_matters}</p> : <p className="text-[13px] text-fg-subtle">No analysis was generated for this event; the observed change speaks for itself.</p>} | |
| 155 | + <div className="mt-3 grid gap-3 sm:grid-cols-2"> | |
| 107 | 156 | <div className="rounded-md border border-line bg-panel-2 p-3"> |
| 108 | 157 | <div className="label mb-1 !text-ok">Observed</div> |
| 109 | − <p className="text-[13px]">{observed ?? "Facts are limited to the diff below: the monitored endpoint changed between the two preserved snapshots."}</p> | |
| 158 | + <p className="text-[13px]">{observed ?? "Facts are limited to the diff above: the monitored endpoint changed between the two preserved snapshots."}</p> | |
| 110 | 159 | </div> |
| 111 | 160 | <div className="rounded-md border border-line bg-panel-2 p-3"> |
| 112 | 161 | <div className="label mb-1 !text-info">Inferred</div> |
| 113 | 162 | <p className="text-[13px] text-fg-muted">{inferred ?? "No inference beyond the observed change."}</p> |
| 114 | − <p className="mt-1 font-mono text-[11px] text-fg-subtle">confidence {fmtScore(e.confidence)}% · {model}</p> | |
| 163 | + <p className="mt-1 font-mono text-[11px] text-fg-subtle">confidence {fmtScore(e.confidence)} · {model}</p> | |
| 115 | 164 | </div> |
| 116 | 165 | </div> |
| 166 | + {who && ( | |
| 167 | + <div className="mt-3"> | |
| 168 | + <div className="label mb-1">Who it affects</div> | |
| 169 | + <p className="text-[13px] text-fg-muted">{who}</p> | |
| 170 | + </div> | |
| 171 | + )} | |
| 117 | 172 | </Panel> |
| 118 | 173 | |
| 119 | − <div> | |
| 120 | − <div className="mb-2 flex items-center justify-between"> | |
| 121 | − <h2 className="label">Diff · {change?.change.kind ?? d.change?.kind ?? "—"}</h2> | |
| 122 | − {oldSnap && newSnap && <Link href={`/compare?a=${oldSnap.id}&b=${newSnap.id}`} className="text-[11px] text-fg-subtle hover:text-fg">open in compare →</Link>} | |
| 123 | − </div> | |
| 124 | − {change ? <DiffViewer unified={change.unified} summary={change.change.diff} defaultTab={change.change.kind === "text" ? "unified" : "semantic"} /> : <Panel dense><Empty>No diff is attached to this event{e.event_type === "page_removed" ? " — the page disappeared; its last snapshot is preserved below." : "."}</Empty></Panel>} | |
| 125 | − </div> | |
| 126 | − | |
| 174 | + {/* EVIDENCE */} | |
| 127 | 175 | <Panel title="Evidence" dense> |
| 128 | − <div className="px-3 py-2 text-[13px]"> | |
| 129 | − <div className="label mb-1">Monitored URL</div> | |
| 130 | − <ExtLink href={e.url} className="break-all font-mono text-[12.5px]">{e.url}</ExtLink> | |
| 131 | − <span className="ml-2 text-[11px] text-fg-subtle">· <Link href={`/url?u=${encodeURIComponent(e.url)}`} className="hover:underline">URL history</Link> · <Link href={`/sensor/${e.sensor_id}`} className="hover:underline">sensor {e.sensor?.name}</Link></span> | |
| 176 | + <div className="flex flex-wrap items-baseline gap-x-2 gap-y-1 px-3 py-2 text-[13px]"> | |
| 177 | + <span className="label">Monitored URL</span> | |
| 178 | + <ExtLink href={e.url} className="min-w-0 break-all font-mono text-[12.5px]">{e.url}</ExtLink> | |
| 132 | 179 | </div> |
| 133 | 180 | <Table head={["Snapshot", "Captured", "HTTP", "Size", "Canonical hash", "Raw hash", ""]}> |
| 134 | − {[["Before", oldSnap], ["After", newSnap]].map(([label, s]) => { | |
| 135 | − const snap = s as typeof oldSnap; | |
| 136 | − return ( | |
| 137 | − <tr key={String(label)}> | |
| 138 | − <Td className="font-medium">{String(label)}</Td> | |
| 139 | − <Td mono className="text-fg-subtle">{snap ? utcDateTime(snap.captured_at) : "—"}</Td> | |
| 140 | − <Td mono>{snap?.http_status ?? "—"}</Td> | |
| 141 | − <Td mono>{snap ? fmtBytes(snap.content_length) : "—"}</Td> | |
| 142 | − <Td mono className="text-fg-subtle">{shortHash(snap?.canonical_hash)}</Td> | |
| 143 | − <Td mono className="text-fg-subtle">{shortHash(snap?.content_hash)}</Td> | |
| 144 | − <Td>{snap ? <a href={`/api/v1/snapshots/${snap.id}?raw=1`} target="_blank" rel="noopener noreferrer" className="text-info hover:underline">view raw</a> : ""}</Td> | |
| 145 | − </tr> | |
| 146 | − ); | |
| 147 | − })} | |
| 181 | + {snapRows.map(([label, snap]) => ( | |
| 182 | + <tr key={label}> | |
| 183 | + <Td className="font-medium">{label}</Td> | |
| 184 | + <Td mono className="whitespace-nowrap text-fg-subtle">{snap ? utcDateTime(snap.captured_at) : "—"}</Td> | |
| 185 | + <Td mono>{snap?.http_status ?? "—"}</Td> | |
| 186 | + <Td mono className="whitespace-nowrap">{snap ? fmtBytes(snap.content_length) : "—"}</Td> | |
| 187 | + <Td mono className="text-fg-subtle"><span title={snap?.canonical_hash ?? undefined}>{shortHash(snap?.canonical_hash)}</span></Td> | |
| 188 | + <Td mono className="text-fg-subtle"><span title={snap?.content_hash ?? undefined}>{shortHash(snap?.content_hash)}</span></Td> | |
| 189 | + <Td className="whitespace-nowrap"> | |
| 190 | + {snap ? ( | |
| 191 | + snap.has_raw === false ? ( | |
| 192 | + <span className="text-fg-subtle" title="Raw body pruned by retention; canonical preserved">view raw · pruned</span> | |
| 193 | + ) : ( | |
| 194 | + <a href={`/api/v1/snapshots/${snap.id}?raw=1`} target="_blank" rel="noopener noreferrer" className="text-info hover:underline">view raw</a> | |
| 195 | + ) | |
| 196 | + ) : ""} | |
| 197 | + </Td> | |
| 198 | + </tr> | |
| 199 | + ))} | |
| 148 | 200 | </Table> |
| 149 | − <div className="px-3 py-2 font-mono text-[11px] text-fg-subtle"> | |
| 150 | − event {e.id} · change {e.change_id ?? "—"} · {e.processing_version ?? ""} · trace: event → change → snapshots → sensor {e.sensor_id} → source {e.source_id} | |
| 201 | + {snapRows.some(([, s]) => s && s.has_raw === false) && <p className="px-3 pt-2 text-[11px] text-fg-subtle">Raw body pruned by retention; canonical preserved. Hashes and the canonical diff remain verifiable.</p>} | |
| 202 | + <dl className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1 px-3 py-2 text-[12px]"> | |
| 203 | + <dt className="text-fg-subtle">Evidence hash</dt> | |
| 204 | + <dd className="min-w-0 break-all font-mono text-fg-muted tabular">{newSnap?.canonical_hash ?? "—"}</dd> | |
| 205 | + <dt className="text-fg-subtle">Sensor</dt> | |
| 206 | + <dd className="min-w-0 truncate"><Link href={`/sensor/${e.sensor_id}`} className="text-info hover:underline">{e.sensor?.name ?? e.sensor_id}</Link> <span className="font-mono text-[11px] text-fg-subtle">{e.sensor?.type}{e.sensor?.connector ? ` · ${e.sensor.connector}` : ""}</span></dd> | |
| 207 | + <dt className="text-fg-subtle">Trace</dt> | |
| 208 | + <dd className="min-w-0 break-all font-mono text-[11px] text-fg-subtle">event {e.id} → change {e.change_id ?? "—"} → snapshots → sensor {e.sensor_id} → source {e.source_id}{e.processing_version ? ` · ${e.processing_version}` : ""}</dd> | |
| 209 | + </dl> | |
| 210 | + <div className="flex flex-wrap gap-3 border-t border-line px-3 py-2 text-[11.5px]"> | |
| 211 | + {oldSnap && newSnap && <Link href={`/compare?a=${oldSnap.id}&b=${newSnap.id}`} className="text-info hover:underline">Compare snapshots</Link>} | |
| 212 | + <Link href={`/sensor/${e.sensor_id}`} className="text-info hover:underline">Sensor</Link> | |
| 213 | + <Link href={`/url?u=${encodeURIComponent(e.url)}`} className="text-info hover:underline">URL history</Link> | |
| 214 | + <Link href={`/source/${e.source?.id ?? e.source_id}`} className="text-info hover:underline">Source</Link> | |
| 151 | 215 | </div> |
| 152 | 216 | </Panel> |
| 153 | 217 | |
| 154 | − <Panel title="Related events" dense> | |
| 155 | − {d.related.length ? d.related.map((r) => <EventRow key={r.id} ev={r} showDate />) : <Empty>No related events yet.</Empty>} | |
| 218 | + {/* HISTORICAL CONTEXT */} | |
| 219 | + <Panel title={<span>Previous changes on this page <span className="font-mono text-fg-subtle">{history.length}</span></span>} dense> | |
| 220 | + {history.length ? ( | |
| 221 | + <ul className="divide-y divide-line"> | |
| 222 | + {history.map((h) => ( | |
| 223 | + <li key={h.id} className="grid grid-cols-[5.5rem_1fr_auto] items-start gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 224 | + <span className="font-mono text-[11px] text-fg-subtle tabular" title={utcDateTime(h.detected_at)}>{relTime(h.detected_at)}</span> | |
| 225 | + <div className="min-w-0"> | |
| 226 | + <div className="flex flex-wrap items-center gap-1.5"> | |
| 227 | + <Link href={`/event/${h.slug}`} className="min-w-0 truncate hover:underline">{h.title}</Link> | |
| 228 | + {h.silent_change && <Badge kind="silent" compact />} | |
| 229 | + </div> | |
| 230 | + <div className="mt-0.5 flex flex-wrap items-center gap-2 text-[11px] text-fg-subtle"> | |
| 231 | + <span>{typeLabel(h.event_type)}</span> | |
| 232 | + {h.field_changes && h.field_changes.length > 0 && <FieldChangeInline items={h.field_changes} max={2} />} | |
| 233 | + </div> | |
| 234 | + </div> | |
| 235 | + <Score value={h.importance} size="sm" /> | |
| 236 | + </li> | |
| 237 | + ))} | |
| 238 | + </ul> | |
| 239 | + ) : ( | |
| 240 | + <Empty>First change observed on this URL.</Empty> | |
| 241 | + )} | |
| 242 | + </Panel> | |
| 243 | + | |
| 244 | + {/* RELATED SIGNALS */} | |
| 245 | + <Panel title={<span>Related signals <span className="font-mono text-fg-subtle">{d.related.length}</span></span>} dense> | |
| 246 | + {d.related.length ? d.related.map((r) => <EventRow key={r.id} ev={r} showDate />) : <Empty>No related signals yet.</Empty>} | |
| 156 | 247 | </Panel> |
| 248 | + | |
| 249 | + {/* CLUSTER / PROPAGATION */} | |
| 250 | + {(cluster || d.cluster) && ( | |
| 251 | + <Panel title="Event cluster · propagation" action={clusterHref && <Link href={clusterHref} className="text-[11px] text-info hover:underline">propagation timeline →</Link>}> | |
| 252 | + {d.cluster?.title && <div className="mb-2 text-[13px] font-medium [overflow-wrap:anywhere]">{d.cluster.title}</div>} | |
| 253 | + <div className="grid grid-cols-2 gap-2 sm:grid-cols-4"> | |
| 254 | + <Cell v={cluster?.event_count ?? d.cluster?.event_count ?? 1} l="signals" /> | |
| 255 | + <Cell v={cluster?.first_party_count ?? d.cluster?.first_party_count ?? 0} l="first-party" /> | |
| 256 | + <Cell v={cluster?.external_count ?? d.cluster?.external_count ?? 0} l="external" /> | |
| 257 | + <Cell v={cluster?.source_count ?? d.cluster?.source_count ?? 1} l="sources" /> | |
| 258 | + </div> | |
| 259 | + {leadTime !== null && leadTime > 0 && ( | |
| 260 | + <p className="mt-3 rounded-md border border-signal/30 bg-signal-soft px-3 py-2 text-[12.5px]"> | |
| 261 | + <span className="font-semibold text-signal">WebSensor lead time {fmtOffset(leadTime).replace("+", "")}</span> <span className="text-fg-muted">— the first-party change was observed before the first external report.</span> | |
| 262 | + </p> | |
| 263 | + )} | |
| 264 | + <div className="mt-2 flex flex-wrap items-center gap-x-3 gap-y-1 text-[11.5px] text-fg-subtle"> | |
| 265 | + {(cluster?.state ?? d.cluster?.state) && <span>state <span className="font-mono uppercase text-fg-muted">{cluster?.state ?? d.cluster?.state}</span></span>} | |
| 266 | + {(cluster?.velocity ?? d.cluster?.velocity) !== undefined && <span>velocity <span className="font-mono text-fg-muted tabular">{Math.round(cluster?.velocity ?? d.cluster?.velocity ?? 0)}</span></span>} | |
| 267 | + {d.cluster && <span>{relTime(d.cluster.first_at)} → {relTime(d.cluster.last_at)}</span>} | |
| 268 | + </div> | |
| 269 | + {d.cluster && d.cluster.entity_ids.length > 0 && <div className="mt-2 flex flex-wrap gap-1">{d.cluster.entity_ids.slice(0, 8).map((id) => <Chip key={id} href={`/entity/${id}`}>{id.replace(/^(org|prd|ent)_/, "")}</Chip>)}</div>} | |
| 270 | + </Panel> | |
| 271 | + )} | |
| 157 | 272 | </div> |
| 158 | 273 | |
| 159 | − <aside className="flex flex-col gap-4"> | |
| 274 | + {/* RIGHT RAIL */} | |
| 275 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 160 | 276 | <Panel> |
| 161 | − <div className="mb-3 flex items-center gap-3"> | |
| 162 | − <Score value={e.importance} size="lg" /> | |
| 163 | − <div> | |
| 164 | − <div className="label">Importance</div> | |
| 165 | − <div className="text-[12px] text-fg-muted">{e.importance >= 90 ? "critical" : e.importance >= 75 ? "major" : e.importance >= 50 ? "notable" : "minor"}</div> | |
| 277 | + <div className="flex items-center gap-3"> | |
| 278 | + <Score value={signal} size="lg" kind="signal" /> | |
| 279 | + <div className="min-w-0"> | |
| 280 | + <div className="label">WebSensor signal score</div> | |
| 281 | + <div className="text-[12px] text-fg-muted">{signal >= 80 ? "attention now" : signal >= 60 ? "worth a look" : "informational"}</div> | |
| 166 | 282 | </div> |
| 167 | 283 | </div> |
| 168 | − <div className="flex flex-col gap-3"> | |
| 284 | + <div className="mt-3 grid grid-cols-2 gap-x-4 gap-y-3"> | |
| 285 | + <Gauge label="Importance" value={e.importance} /> | |
| 169 | 286 | <Gauge label="Confidence" value={e.confidence} tone="info" /> |
| 170 | 287 | <Gauge label="Novelty" value={e.novelty} tone="signal" /> |
| 288 | + <Gauge label="Impact" value={e.impact_score} tone="high" /> | |
| 289 | + <Gauge label="Velocity" value={e.velocity_score} tone="silent" /> | |
| 290 | + <Gauge label="Anomaly" value={e.anomaly_score} tone="mid" /> | |
| 291 | + </div> | |
| 292 | + <div className="mt-4"> | |
| 293 | + <div className="label mb-1.5">Why this score</div> | |
| 294 | + {reasons.length ? ( | |
| 295 | + <ul className="divide-y divide-line rounded-md border border-line text-[12px]"> | |
| 296 | + {reasons.map((r, i) => ( | |
| 297 | + <li key={i} className="flex items-start gap-2 px-2 py-1"> | |
| 298 | + <span className={`w-2 shrink-0 font-mono font-semibold ${r.sign === "+" ? "text-signal" : "text-danger"}`}>{r.sign}</span> | |
| 299 | + <span className="min-w-0 flex-1 text-fg-muted">{r.text}</span> | |
| 300 | + {r.points !== undefined && <span className={`shrink-0 font-mono tabular ${r.sign === "+" ? "text-signal" : "text-danger"}`}>{r.points > 0 ? "+" : ""}{r.points}</span>} | |
| 301 | + </li> | |
| 302 | + ))} | |
| 303 | + </ul> | |
| 304 | + ) : ( | |
| 305 | + <p className="text-[12px] text-fg-subtle">Scored by heuristics only; no explicit adjustments were recorded for this event.</p> | |
| 306 | + )} | |
| 171 | 307 | </div> |
| 308 | + {e.change_class && <div className="mt-3 text-[11.5px] text-fg-subtle">Semantic class <span className="font-medium text-fg-muted">{CLASS_LABELS[e.change_class] ?? e.change_class}</span></div>} | |
| 309 | + </Panel> | |
| 310 | + | |
| 311 | + <Panel title="Importance components" dense> | |
| 312 | + <Table head={["Component", "Weight", "Score"]}> | |
| 313 | + {IMPORTANCE_WEIGHTS.map((w) => ( | |
| 314 | + <tr key={w.key}> | |
| 315 | + <Td>{w.label}</Td> | |
| 316 | + <Td mono className="text-fg-subtle">{w.weight}%</Td> | |
| 317 | + <Td mono>{e.importance_components?.[w.key] !== undefined ? fmtScore(e.importance_components[w.key]) : "—"}</Td> | |
| 318 | + </tr> | |
| 319 | + ))} | |
| 320 | + </Table> | |
| 172 | 321 | </Panel> |
| 322 | + | |
| 173 | 323 | <Panel title="Timing"> |
| 174 | 324 | <dl className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1 text-[12.5px]"> |
| 175 | − <dt className="text-fg-subtle">Published</dt><dd className="text-right font-mono tabular">{e.published_at ? utcDateTime(e.published_at) : "—"}</dd> | |
| 176 | − <dt className="text-fg-subtle">Observed from</dt><dd className="text-right font-mono tabular">{e.observed_from ? utcDateTime(e.observed_from) : "—"}</dd> | |
| 177 | − <dt className="text-fg-subtle">Detected</dt><dd className="text-right font-mono tabular">{utcDateTime(e.detected_at)}</dd> | |
| 178 | − <dt className="text-fg-subtle">Processed</dt><dd className="text-right font-mono tabular">{e.processed_at ? utcDateTime(e.processed_at) : "—"}</dd> | |
| 179 | − <dt className="text-fg-subtle">To feed</dt><dd className="text-right font-mono tabular">{e.published_to_feed_at ? utcDateTime(e.published_to_feed_at) : "—"}</dd> | |
| 180 | − <dt className="text-fg-subtle">Detection latency</dt><dd className="text-right font-mono tabular">{fmtMs(e.detection_latency_ms)}</dd> | |
| 181 | − <dt className="text-fg-subtle">Processing latency</dt><dd className="text-right font-mono tabular">{fmtMs(e.processing_latency_ms)}</dd> | |
| 325 | + <Row label="Published" value={e.published_at ? utcDateTime(e.published_at) : "—"} /> | |
| 326 | + <Row label="Observed from" value={e.observed_from ? utcDateTime(e.observed_from) : "—"} /> | |
| 327 | + <Row label="Detected" value={utcDateTime(e.detected_at)} /> | |
| 328 | + <Row label="Processed" value={e.processed_at ? utcDateTime(e.processed_at) : "—"} /> | |
| 329 | + <Row label="To feed" value={e.published_to_feed_at ? utcDateTime(e.published_to_feed_at) : "—"} /> | |
| 330 | + <Row label="Detection latency" value={fmtMs(e.detection_latency_ms)} /> | |
| 331 | + <Row label="Processing latency" value={fmtMs(e.processing_latency_ms)} /> | |
| 332 | + {leadTime !== null && leadTime > 0 && <Row label="WebSensor lead time" value={fmtOffset(leadTime).replace("+", "")} tone="text-signal font-semibold" />} | |
| 182 | 333 | </dl> |
| 183 | 334 | <p className="mt-2 text-[11px] text-fg-subtle">{relTime(e.detected_at)} · all times UTC</p> |
| 184 | 335 | </Panel> |
| 336 | + | |
| 185 | 337 | <Panel title="Entities" dense> |
| 186 | 338 | {e.entities.length ? ( |
| 187 | 339 | <ul className="divide-y divide-line"> |
| 188 | 340 | {e.entities.map((x) => ( |
| 189 | − <li key={x.id} className="flex items-center justify-between px-3 py-1.5 text-[13px]"> | |
| 190 | − <Link href={`/company/${x.id}`} className="hover:underline">{x.name}</Link> | |
| 191 | − <span className="font-mono text-[11px] text-fg-subtle">{x.type}{x.role === "subject" ? " · subject" : ""}</span> | |
| 341 | + <li key={x.id} className="flex items-center justify-between gap-2 px-3 py-1.5 text-[13px]"> | |
| 342 | + <Link href={`/entity/${x.id}`} className="min-w-0 truncate hover:underline">{x.name}</Link> | |
| 343 | + <span className="shrink-0 font-mono text-[11px] text-fg-subtle">{x.type}{x.role === "subject" ? " · subject" : ""}</span> | |
| 192 | 344 | </li> |
| 193 | 345 | ))} |
| 194 | 346 | </ul> |
@@ -196,57 +348,40 @@ export default async function EventPage({ params }: { params: Promise<{ slug: st | ||
| 196 | 348 | <Empty>No entity resolved.</Empty> |
| 197 | 349 | )} |
| 198 | 350 | </Panel> |
| 199 | − <Panel title="Importance components" dense> | |
| 200 | − <Table head={["Component", "Weight", "Score"]}> | |
| 201 | − {COMPONENT_WEIGHTS.map(([k, label, w]) => ( | |
| 202 | − <tr key={k}> | |
| 203 | − <Td>{label}</Td> | |
| 204 | − <Td mono className="text-fg-subtle">{w}%</Td> | |
| 205 | − <Td mono>{e.importance_components?.[k] !== undefined ? fmtScore(e.importance_components[k]) : "—"}</Td> | |
| 206 | − </tr> | |
| 207 | − ))} | |
| 208 | − </Table> | |
| 209 | − </Panel> | |
| 210 | − {d.cluster && ( | |
| 211 | − <Panel title="Event cluster"> | |
| 212 | − <div id="cluster" className="text-[13px]"> | |
| 213 | − <div className="font-medium">{d.cluster.title}</div> | |
| 214 | − <div className="mt-1 text-[11.5px] text-fg-subtle">{d.cluster.event_count} related observation{d.cluster.event_count === 1 ? "" : "s"} · max importance {fmtScore(d.cluster.max_importance)} · {relTime(d.cluster.first_at)} → {relTime(d.cluster.last_at)}</div> | |
| 215 | − {d.cluster.entity_ids.length > 0 && <div className="mt-2 flex flex-wrap gap-1">{d.cluster.entity_ids.slice(0, 8).map((id) => <Chip key={id} href={`/company/${id}`}>{id.replace(/^(org|prd)_/, "")}</Chip>)}</div>} | |
| 216 | − </div> | |
| 217 | − </Panel> | |
| 218 | − )} | |
| 351 | + | |
| 219 | 352 | <Panel title="Source reliability"> |
| 220 | 353 | <dl className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1 text-[12.5px]"> |
| 221 | 354 | <dt className="text-fg-subtle">Sensor health</dt><dd className="text-right"><HealthPill health={d.sensor_reliability?.health} /></dd> |
| 222 | − <dt className="text-fg-subtle">Connector success</dt><dd className="text-right font-mono tabular">{fmtPct(d.sensor_reliability?.success_rate ?? null)}</dd> | |
| 223 | − <dt className="text-fg-subtle">Latency</dt><dd className="text-right font-mono tabular">{fmtMs(d.sensor_reliability?.avg_latency_ms)}</dd> | |
| 224 | − <dt className="text-fg-subtle">Runs</dt><dd className="text-right font-mono tabular">{d.sensor_reliability?.total_runs ?? "—"}</dd> | |
| 225 | − <dt className="text-fg-subtle">Raw / meaningful</dt><dd className="text-right font-mono tabular">{d.sensor_reliability?.raw_changes ?? "—"} / {d.sensor_reliability?.meaningful_changes ?? "—"}</dd> | |
| 355 | + <Row label="Connector success" value={fmtPct(d.sensor_reliability?.success_rate ?? null)} /> | |
| 356 | + <Row label="Latency" value={fmtMs(d.sensor_reliability?.avg_latency_ms)} /> | |
| 357 | + <Row label="Runs" value={d.sensor_reliability?.total_runs ?? "—"} /> | |
| 358 | + <Row label="Raw / meaningful" value={`${d.sensor_reliability?.raw_changes ?? "—"} / ${d.sensor_reliability?.meaningful_changes ?? "—"}`} /> | |
| 226 | 359 | </dl> |
| 227 | − <p className="mt-2 text-[11px] text-fg-subtle">Sensor type {e.sensor?.type} via {e.sensor?.connector} connector · tier {e.sensor?.tier}</p> | |
| 360 | + <p className="mt-2 text-[11px] text-fg-subtle">Sensor type {e.sensor?.type} via {e.sensor?.connector ?? "—"} connector · tier {e.sensor?.tier ?? e.source?.tier ?? "—"}{e.first_party === false ? " · third-party report" : " · first-party channel"}</p> | |
| 228 | 361 | </Panel> |
| 362 | + | |
| 229 | 363 | <Panel title="Interpretation versions" dense> |
| 230 | 364 | {d.interpretations.length ? ( |
| 231 | 365 | <ul className="divide-y divide-line"> |
| 232 | 366 | {d.interpretations.map((v) => ( |
| 233 | − <li key={v.version} className="flex items-center justify-between px-3 py-1.5 text-[12.5px]"> | |
| 367 | + <li key={v.version} className="flex items-center justify-between gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 234 | 368 | <span>v{v.version} · <span className="font-mono">{v.model}</span></span> |
| 235 | 369 | <span className="font-mono text-[11px] text-fg-subtle">{utcDateTime(v.created_at)}</span> |
| 236 | 370 | </li> |
| 237 | 371 | ))} |
| 238 | 372 | </ul> |
| 239 | 373 | ) : ( |
| 240 | − <Empty>—</Empty> | |
| 374 | + <p className="px-3 py-2 text-[12px] text-fg-subtle">Heuristic interpretation only ({model}).</p> | |
| 241 | 375 | )} |
| 242 | − <p className="px-3 py-2 text-[11px] text-fg-subtle">Raw evidence is immutable; interpretations may be reprocessed and are versioned.</p> | |
| 376 | + <p className="border-t border-line px-3 py-2 text-[11px] text-fg-subtle">Raw evidence is immutable; interpretations may be reprocessed and are versioned.</p> | |
| 243 | 377 | </Panel> |
| 378 | + | |
| 244 | 379 | {e.keywords.length > 0 && ( |
| 245 | 380 | <Panel title="Keywords"> |
| 246 | 381 | <div className="flex flex-wrap gap-1">{e.keywords.map((k) => <Chip key={k} href={`/search?q=${encodeURIComponent(k)}`}>{k}</Chip>)}</div> |
| 247 | 382 | </Panel> |
| 248 | 383 | )} |
| 249 | − <p className="text-[11px] text-fg-subtle">{typeLabel(e.event_type)} · detected by WebSensor · AI-generated summaries never replace the original evidence above.</p> | |
| 384 | + <p className="text-[11px] text-fg-subtle">{typeLabel(e.event_type)} · detected by WebSensor · AI-generated analysis never replaces the original evidence above.</p> | |
| 250 | 385 | </aside> |
| 251 | 386 | </div> |
| 252 | 387 | </> |
added
apps/web/src/app/explore/loading.tsx
+27 −0
@@ -0,0 +1,27 @@ | ||
| 1 | +import { Skeleton, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3"> | |
| 7 | + <Skeleton className="mb-1 h-2.5 w-16" /> | |
| 8 | + <Skeleton className="h-5 w-32" /> | |
| 9 | + <Skeleton className="mt-2 h-3 w-full max-w-xl" /> | |
| 10 | + </div> | |
| 11 | + <div className="mb-3 flex gap-2 border-b border-line pb-2"> | |
| 12 | + {Array.from({ length: 8 }, (_, i) => ( | |
| 13 | + <Skeleton key={i} className="h-3 w-16" /> | |
| 14 | + ))} | |
| 15 | + </div> | |
| 16 | + <div className="mb-3 flex gap-1"> | |
| 17 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 18 | + <Skeleton key={i} className="h-4 w-20" /> | |
| 19 | + ))} | |
| 20 | + </div> | |
| 21 | + <div className="panel"> | |
| 22 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-40" /></div> | |
| 23 | + <SkeletonRows rows={12} /> | |
| 24 | + </div> | |
| 25 | + </> | |
| 26 | + ); | |
| 27 | +} | |
modified
apps/web/src/app/explore/page.tsx
+373 −98
@@ -1,112 +1,387 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | 2 | import type { Metadata } from "next"; |
| 3 | 3 | import { EventRow } from "@/components/event-row"; |
| 4 | −import { Bar, Empty, PageHeader, Panel, Score } from "@/components/ui"; | |
| 5 | −import { api } from "@/lib/api"; | |
| 6 | −import { fmtScore, relTime, typeLabel } from "@/lib/format"; | |
| 4 | +import { Badge, Bar, Chip, Empty, Flag, PageHeader, Panel, Score, StateBadge, Table, Td, Tabs, TierBadge } from "@/components/ui"; | |
| 5 | +import { api, type Cluster, type EventItem } from "@/lib/api"; | |
| 6 | +import { CHANNELS, SAVED_VIEWS, feedHref, fmtInt, fmtOffset, fmtScore, relTime, typeLabel } from "@/lib/format"; | |
| 7 | 7 | |
| 8 | 8 | export const dynamic = "force-dynamic"; |
| 9 | −export const metadata: Metadata = { title: "Explore", description: "Most active sources, biggest changes, silent changes, unusual activity and event clusters." }; | |
| 10 | 9 | |
| 11 | −export default async function ExplorePage() { | |
| 12 | − const x = await api.explore(); | |
| 13 | − const maxType = Math.max(1, ...(x?.by_type ?? []).map((t) => t.n)); | |
| 14 | − const maxCat = Math.max(1, ...(x?.by_category ?? []).map((t) => t.n)); | |
| 10 | +const TABS = [ | |
| 11 | + { key: "trending", label: "Trending" }, | |
| 12 | + { key: "breaking", label: "Breaking" }, | |
| 13 | + { key: "silent", label: "Silent" }, | |
| 14 | + { key: "newly", label: "Newly detected" }, | |
| 15 | + { key: "active", label: "Active sources" }, | |
| 16 | + { key: "unusual", label: "Unusual activity" }, | |
| 17 | + { key: "clusters", label: "Clusters" }, | |
| 18 | + { key: "entities", label: "Entities" }, | |
| 19 | + { key: "sources", label: "Sources" }, | |
| 20 | + { key: "categories", label: "Categories" }, | |
| 21 | + { key: "countries", label: "Countries" }, | |
| 22 | +] as const; | |
| 23 | +type TabKey = (typeof TABS)[number]["key"]; | |
| 24 | + | |
| 25 | +function tabOf(v: string | undefined): TabKey { | |
| 26 | + return (TABS.find((t) => t.key === v)?.key ?? "trending") as TabKey; | |
| 27 | +} | |
| 28 | + | |
| 29 | +export async function generateMetadata({ searchParams }: { searchParams: Promise<{ tab?: string }> }): Promise<Metadata> { | |
| 30 | + const { tab } = await searchParams; | |
| 31 | + const t = TABS.find((x) => x.key === tabOf(tab))!; | |
| 32 | + return { title: `Discover · ${t.label}`, description: "Trending entities, breaking clusters, silent changes, newly detected signals, active and anomalous sources, clusters, entities, sources, categories and countries.", alternates: { canonical: t.key === "trending" ? "/explore" : `/explore?tab=${t.key}` } }; | |
| 33 | +} | |
| 34 | + | |
| 35 | +/** Discover page (spec §28): URL-driven tabs over the derived views of the event store. */ | |
| 36 | +export default async function ExplorePage({ searchParams }: { searchParams: Promise<{ tab?: string }> }) { | |
| 37 | + const sp = await searchParams; | |
| 38 | + const tab = tabOf(sp.tab); | |
| 39 | + const [x, trending, desk, rank, sources, countries] = await Promise.all([ | |
| 40 | + api.explore(), | |
| 41 | + tab === "trending" ? api.trending(24, 30) : Promise.resolve({ items: [] }), | |
| 42 | + tab === "breaking" ? api.breakingDesk() : Promise.resolve(null), | |
| 43 | + tab === "entities" ? api.rank(50) : Promise.resolve({ items: [] }), | |
| 44 | + tab === "sources" ? api.sources({}) : Promise.resolve({ items: [] }), | |
| 45 | + tab === "countries" ? api.countries() : Promise.resolve({ items: [] }), | |
| 46 | + ]); | |
| 47 | + const counts: Partial<Record<TabKey, number>> = { | |
| 48 | + trending: tab === "trending" ? trending.items.length : undefined, | |
| 49 | + breaking: desk ? desk.breaking_now.length + desk.developing.length + desk.recently_confirmed.length : undefined, | |
| 50 | + silent: x?.silent_changes?.length, | |
| 51 | + newly: x?.newly_detected?.length, | |
| 52 | + active: x?.most_active_sources?.length, | |
| 53 | + unusual: x?.unusual_activity?.length, | |
| 54 | + clusters: x?.clusters?.length, | |
| 55 | + entities: tab === "entities" ? rank.items.length : undefined, | |
| 56 | + sources: tab === "sources" ? sources.items.length : undefined, | |
| 57 | + categories: x?.by_category?.length, | |
| 58 | + countries: tab === "countries" ? countries.items.length : undefined, | |
| 59 | + }; | |
| 15 | 60 | return ( |
| 16 | 61 | <> |
| 17 | − <PageHeader kicker="Discover" title="Explore" description="Derived views over the event store: who is changing the most, what mattered most, what changed silently, and where activity is abnormal." /> | |
| 18 | − <div className="grid gap-4 lg:grid-cols-3"> | |
| 19 | − <Panel title="Most active sources · 24 h" dense> | |
| 20 | − {x?.most_active_sources?.length ? ( | |
| 21 | − <ul className="divide-y divide-line"> | |
| 22 | − {x.most_active_sources.map((s) => ( | |
| 23 | − <li key={s.id} className="flex items-center gap-2 px-3 py-1.5 text-[13px]"> | |
| 24 | − <Link href={`/source/${s.id}`} className="min-w-0 flex-1 truncate font-medium hover:underline">{s.name}</Link> | |
| 25 | − <span className="font-mono text-[12px] text-fg-subtle tabular">{s.events_24h} ev</span> | |
| 26 | − <Score value={s.max_importance} size="sm" /> | |
| 27 | − </li> | |
| 28 | − ))} | |
| 29 | − </ul> | |
| 30 | − ) : ( | |
| 31 | − <Empty /> | |
| 32 | − )} | |
| 33 | − </Panel> | |
| 34 | − <Panel title="Unusual activity" dense> | |
| 35 | − {x?.unusual_activity?.length ? ( | |
| 36 | − <ul className="divide-y divide-line"> | |
| 37 | − {x.unusual_activity.map((s) => ( | |
| 38 | − <li key={s.id} className="px-3 py-1.5 text-[13px]"> | |
| 39 | − <div className="flex items-center gap-2"> | |
| 40 | − <Link href={`/source/${s.id}`} className="min-w-0 flex-1 truncate font-medium hover:underline">{s.name}</Link> | |
| 41 | − <span className={`font-mono text-[12px] font-semibold tabular ${s.activity_score >= 70 ? "text-hot" : s.activity_score >= 45 ? "text-high" : "text-fg-muted"}`}>{fmtScore(s.activity_score)}</span> | |
| 42 | − </div> | |
| 43 | − <div className="text-[11px] text-fg-subtle">{s.changes_2h} changes in 2 h · baseline {s.baseline_per_day}/day</div> | |
| 44 | − <div className="mt-1"><Bar value={s.activity_score} tone={s.activity_score >= 70 ? "hot" : s.activity_score >= 45 ? "high" : "signal"} /></div> | |
| 45 | − </li> | |
| 46 | − ))} | |
| 47 | − </ul> | |
| 48 | − ) : ( | |
| 49 | − <Empty>Activity anomalies compare the last 2 h of raw changes with a 14-day baseline.</Empty> | |
| 50 | − )} | |
| 51 | − </Panel> | |
| 52 | − <Panel title="Event clusters · 48 h" dense> | |
| 53 | − {x?.clusters?.length ? ( | |
| 54 | − <ul className="divide-y divide-line"> | |
| 55 | − {x.clusters.map((c) => ( | |
| 56 | − <li key={c.id} className="flex items-start gap-2 px-3 py-1.5 text-[13px]"> | |
| 57 | − <Score value={c.max_importance} size="sm" /> | |
| 58 | − <div className="min-w-0"> | |
| 59 | − <div className="line-clamp-2 font-medium">{c.title}</div> | |
| 60 | − <div className="text-[11px] text-fg-subtle">{c.event_count} observations · {c.source?.name ?? ""} · {relTime(c.last_at)}</div> | |
| 61 | − </div> | |
| 62 | − </li> | |
| 63 | − ))} | |
| 64 | − </ul> | |
| 65 | − ) : ( | |
| 66 | − <Empty>No multi-observation clusters yet.</Empty> | |
| 67 | − )} | |
| 68 | − </Panel> | |
| 62 | + <PageHeader compact kicker="Discover" title="Explore" description="Derived views over the event store: who is changing the most, what mattered most, what changed silently, and where activity is abnormal." /> | |
| 63 | + <Tabs className="mb-3" current={tab} items={TABS.map((t) => ({ key: t.key, label: t.label, href: t.key === "trending" ? "/explore" : `/explore?tab=${t.key}`, count: counts[t.key] }))} /> | |
| 64 | + <div className="mb-3 flex flex-wrap items-center gap-1"> | |
| 65 | + <span className="mr-1 text-[11px] text-fg-subtle">Saved views</span> | |
| 66 | + {SAVED_VIEWS.map((v) => ( | |
| 67 | + <Chip key={v.key} href={feedHref(v.query, "/live")}>{v.label}</Chip> | |
| 68 | + ))} | |
| 69 | 69 | </div> |
| 70 | − <div className="mt-4 grid gap-4 lg:grid-cols-2"> | |
| 71 | − <Panel title="Biggest changes · 48 h" dense action={<Link href="/breaking" className="text-[11px] text-fg-subtle hover:text-fg">breaking →</Link>}> | |
| 72 | − {x?.biggest_changes?.length ? x.biggest_changes.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty />} | |
| 73 | − </Panel> | |
| 74 | − <Panel title="Silent changes" dense action={<Link href="/silent" className="text-[11px] text-fg-subtle hover:text-fg">all silent →</Link>}> | |
| 75 | − {x?.silent_changes?.length ? x.silent_changes.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>No silent changes yet.</Empty>} | |
| 76 | − </Panel> | |
| 77 | − </div> | |
| 78 | − <div className="mt-4 grid gap-4 lg:grid-cols-2"> | |
| 79 | − <Panel title="Events by type · 7 d"> | |
| 80 | − {x?.by_type?.length ? ( | |
| 81 | − <ul className="space-y-1.5"> | |
| 82 | − {x.by_type.slice(0, 16).map((t) => ( | |
| 83 | − <li key={t.event_type} className="grid grid-cols-[10rem_1fr_3rem] items-center gap-2 text-[12.5px]"> | |
| 84 | − <span className="truncate">{typeLabel(t.event_type)}</span> | |
| 85 | − <Bar value={t.n} max={maxType} tone="info" /> | |
| 86 | − <span className="text-right font-mono text-fg-subtle tabular">{t.n}</span> | |
| 87 | − </li> | |
| 88 | − ))} | |
| 89 | − </ul> | |
| 90 | − ) : ( | |
| 91 | − <Empty /> | |
| 92 | − )} | |
| 70 | + | |
| 71 | + {tab === "trending" && <TrendingTab items={trending.items} />} | |
| 72 | + {tab === "breaking" && <BreakingTab desk={desk} />} | |
| 73 | + {tab === "silent" && ( | |
| 74 | + <Panel title="Silent changes · 48 h" dense action={<Link href="/silent" className="text-[11px] text-fg-subtle hover:text-fg">all silent →</Link>}> | |
| 75 | + {x?.silent_changes?.length ? x.silent_changes.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>No silent change detected in this window.</Empty>} | |
| 93 | 76 | </Panel> |
| 94 | − <Panel title="Events by category · 7 d"> | |
| 95 | − {x?.by_category?.length ? ( | |
| 96 | − <ul className="space-y-1.5"> | |
| 97 | − {x.by_category.slice(0, 16).map((t) => ( | |
| 98 | − <li key={t.category} className="grid grid-cols-[10rem_1fr_3rem] items-center gap-2 text-[12.5px]"> | |
| 99 | − <Link href={`/category/${t.category}`} className="truncate hover:underline">{t.category}</Link> | |
| 100 | − <Bar value={t.n} max={maxCat} tone="signal" /> | |
| 101 | − <span className="text-right font-mono text-fg-subtle tabular">{t.n}</span> | |
| 102 | − </li> | |
| 103 | − ))} | |
| 104 | − </ul> | |
| 105 | − ) : ( | |
| 106 | − <Empty /> | |
| 107 | − )} | |
| 77 | + )} | |
| 78 | + {tab === "newly" && ( | |
| 79 | + <Panel title="Newly detected · first observation of a page, feed or entity" dense action={<Link href="/live" className="text-[11px] text-fg-subtle hover:text-fg">live →</Link>}> | |
| 80 | + {x?.newly_detected?.length ? x.newly_detected.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>Nothing newly detected in this window.</Empty>} | |
| 108 | 81 | </Panel> |
| 109 | − </div> | |
| 82 | + )} | |
| 83 | + {tab === "active" && <ActiveTab items={x?.most_active_sources ?? []} />} | |
| 84 | + {tab === "unusual" && <UnusualTab items={x?.unusual_activity ?? []} />} | |
| 85 | + {tab === "clusters" && <ClustersTab items={x?.clusters ?? []} />} | |
| 86 | + {tab === "entities" && <EntitiesTab items={rank.items} />} | |
| 87 | + {tab === "sources" && <SourcesTab items={sources.items} />} | |
| 88 | + {tab === "categories" && <CategoriesTab byCategory={x?.by_category ?? []} byType={x?.by_type ?? []} labels={x?.event_types ?? {}} />} | |
| 89 | + {tab === "countries" && <CountriesTab items={countries.items} />} | |
| 110 | 90 | </> |
| 111 | 91 | ); |
| 112 | 92 | } |
| 93 | + | |
| 94 | +// --------------------------------------------------------------------------------------- | |
| 95 | + | |
| 96 | +function Dir({ d }: { d?: "up" | "down" | "flat" }) { | |
| 97 | + const c = d === "up" ? "text-signal" : d === "down" ? "text-fg-subtle" : "text-fg-muted"; | |
| 98 | + return <span className={`font-mono ${c}`} title={d === "up" ? "Rising vs. previous window" : d === "down" ? "Falling vs. previous window" : "Flat vs. previous window"}>{d === "up" ? "↑" : d === "down" ? "↓" : "→"}</span>; | |
| 99 | +} | |
| 100 | + | |
| 101 | +function TrendingTab({ items }: { items: Awaited<ReturnType<typeof api.trending>>["items"] }) { | |
| 102 | + return ( | |
| 103 | + <Panel title="Trending entities · 24 h" dense action={<span className="text-[11px] text-fg-subtle">score = volume × acceleration × signal, vs. the previous 24 h</span>}> | |
| 104 | + {items.length === 0 ? ( | |
| 105 | + <Empty>Trending is computed from the last 24 h of events.</Empty> | |
| 106 | + ) : ( | |
| 107 | + <Table head={["#", "Entity", "", "Events", "Sources", "1st-party", "Silent", "Avg signal", "Score"]}> | |
| 108 | + {items.map((t, i) => ( | |
| 109 | + <tr key={t.id} className="hover:bg-panel-2/60"> | |
| 110 | + <Td mono className="text-fg-subtle">{i + 1}</Td> | |
| 111 | + <Td> | |
| 112 | + <Link href={`/entity/${t.id}`} className="font-medium hover:underline">{t.name}</Link> | |
| 113 | + <div className="truncate font-mono text-[10.5px] text-fg-subtle">{t.type}{t.domain ? ` · ${t.domain}` : ""}</div> | |
| 114 | + </Td> | |
| 115 | + <Td mono><Dir d={t.direction} /></Td> | |
| 116 | + <Td mono className="whitespace-nowrap">{fmtInt(t.events)}<span className="text-fg-subtle"> / {fmtInt(t.prev_events)}</span></Td> | |
| 117 | + <Td mono>{t.sources}</Td> | |
| 118 | + <Td mono className={t.first_party ? "text-signal" : "text-fg-subtle"}>{t.first_party ?? 0}</Td> | |
| 119 | + <Td mono className={t.silent ? "text-silent" : "text-fg-subtle"}>{t.silent}</Td> | |
| 120 | + <Td mono>{fmtScore(t.avg_signal)}</Td> | |
| 121 | + <Td> | |
| 122 | + <div className="flex w-28 items-center gap-2"> | |
| 123 | + <Bar value={t.score} tone={t.score >= 80 ? "hot" : t.score >= 60 ? "high" : "signal"} /> | |
| 124 | + <span className="w-8 text-right font-mono text-[12px] font-semibold tabular">{fmtScore(t.score)}</span> | |
| 125 | + </div> | |
| 126 | + </Td> | |
| 127 | + </tr> | |
| 128 | + ))} | |
| 129 | + </Table> | |
| 130 | + )} | |
| 131 | + </Panel> | |
| 132 | + ); | |
| 133 | +} | |
| 134 | + | |
| 135 | +function ClusterRows({ items, empty }: { items: Cluster[]; empty: string }) { | |
| 136 | + if (!items.length) return <Empty>{empty}</Empty>; | |
| 137 | + return ( | |
| 138 | + <ul className="divide-y divide-line"> | |
| 139 | + {items.map((c) => { | |
| 140 | + const e = c.event as EventItem | null | undefined; | |
| 141 | + return ( | |
| 142 | + <li key={c.id} className="grid grid-cols-[auto_1fr] items-start gap-x-3 px-3 py-2 text-[13px]"> | |
| 143 | + <Score value={e?.signal_score ?? c.max_importance} kind="signal" size="sm" /> | |
| 144 | + <div className="min-w-0"> | |
| 145 | + <div className="flex flex-wrap items-center gap-x-2 gap-y-0.5 text-[11px] text-fg-subtle"> | |
| 146 | + <StateBadge state={c.state} /> | |
| 147 | + {(e?.source ?? c.source) && <span className="truncate font-mono uppercase text-fg-muted">{(e?.source ?? c.source)?.name}</span>} | |
| 148 | + {e?.country && <Flag code={e.country} />} | |
| 149 | + <span className="ml-auto whitespace-nowrap font-mono tabular">{relTime(c.last_at)}</span> | |
| 150 | + </div> | |
| 151 | + <Link href={`/cluster/${c.primary_slug ?? c.slug ?? c.id}`} className="mt-0.5 block font-medium leading-snug hover:underline">{c.title}</Link> | |
| 152 | + <div className="mt-0.5 flex flex-wrap items-center gap-x-2 gap-y-0.5 text-[11px] text-fg-subtle"> | |
| 153 | + <span>{c.event_count} signal{c.event_count === 1 ? "" : "s"} · {c.source_count ?? 1} source{(c.source_count ?? 1) === 1 ? "" : "s"}</span> | |
| 154 | + {(c.first_party_count ?? 0) > 0 && <span className="text-signal">{c.first_party_count} first-party</span>} | |
| 155 | + {(c.external_count ?? 0) > 0 && <span>{c.external_count} external</span>} | |
| 156 | + {c.lead_time_ms && c.lead_time_ms > 0 ? <span title="WebSensor lead time before the first external report">lead {fmtOffset(c.lead_time_ms).replace("+", "")}</span> : null} | |
| 157 | + {c.categories?.slice(0, 2).map((k) => <Chip key={k} href={`/category/${k}`}>{k}</Chip>)} | |
| 158 | + </div> | |
| 159 | + </div> | |
| 160 | + </li> | |
| 161 | + ); | |
| 162 | + })} | |
| 163 | + </ul> | |
| 164 | + ); | |
| 165 | +} | |
| 166 | + | |
| 167 | +function BreakingTab({ desk }: { desk: Awaited<ReturnType<typeof api.breakingDesk>> }) { | |
| 168 | + const d = desk ?? { breaking_now: [], developing: [], recently_confirmed: [], watching: [] }; | |
| 169 | + return ( | |
| 170 | + <div className="grid gap-4 xl:grid-cols-2"> | |
| 171 | + <Panel title={<span className="text-hot">Breaking now <span className="font-mono text-fg-subtle">{d.breaking_now.length}</span></span>} dense action={<Link href="/breaking" className="text-[11px] text-fg-subtle hover:text-fg">desk →</Link>}> | |
| 172 | + <ClusterRows items={d.breaking_now} empty="Nothing is breaking right now. Breaking requires strong signal, freshness and confirmation — not just recency." /> | |
| 173 | + </Panel> | |
| 174 | + <Panel title={<span className="text-high">Developing <span className="font-mono text-fg-subtle">{d.developing.length}</span></span>} dense> | |
| 175 | + <ClusterRows items={d.developing} empty="No developing story: signals are not accumulating fast enough anywhere." /> | |
| 176 | + </Panel> | |
| 177 | + <Panel title={<span className="text-ok">Recently confirmed <span className="font-mono text-fg-subtle">{d.recently_confirmed.length}</span></span>} dense> | |
| 178 | + <ClusterRows items={d.recently_confirmed} empty="No cluster reached independent confirmation in the last 48 h." /> | |
| 179 | + </Panel> | |
| 180 | + <Panel title="Watching · high signal, not yet breaking" dense> | |
| 181 | + {d.watching.length ? d.watching.slice(0, 12).map((e) => <EventRow key={e.id} ev={e} />) : <Empty>Nothing to watch.</Empty>} | |
| 182 | + </Panel> | |
| 183 | + </div> | |
| 184 | + ); | |
| 185 | +} | |
| 186 | + | |
| 187 | +function ActiveTab({ items }: { items: NonNullable<Awaited<ReturnType<typeof api.explore>>>["most_active_sources"] }) { | |
| 188 | + const max = Math.max(1, ...items.map((s) => s.events_24h)); | |
| 189 | + return ( | |
| 190 | + <Panel title="Most active sources · 24 h" dense action={<Link href="/sources" className="text-[11px] text-fg-subtle hover:text-fg">all sources →</Link>}> | |
| 191 | + {items.length === 0 ? ( | |
| 192 | + <Empty /> | |
| 193 | + ) : ( | |
| 194 | + <Table head={["#", "Source", "Domain", "Kind", "Events 24 h", "", "Max importance"]}> | |
| 195 | + {items.map((s, i) => ( | |
| 196 | + <tr key={s.id} className="hover:bg-panel-2/60"> | |
| 197 | + <Td mono className="text-fg-subtle">{i + 1}</Td> | |
| 198 | + <Td><Link href={`/source/${s.id}`} className="font-medium hover:underline">{s.name}</Link></Td> | |
| 199 | + <Td mono><Link href={`/domain/${s.domain}`} className="text-fg-muted hover:underline">{s.domain}</Link></Td> | |
| 200 | + <Td>{(s as { first_party?: boolean }).first_party === false ? <Badge kind="external" compact /> : <Badge kind="first-party" compact />}</Td> | |
| 201 | + <Td mono>{fmtInt(s.events_24h)}</Td> | |
| 202 | + <Td><div className="w-24 sm:w-40"><Bar value={s.events_24h} max={max} tone="signal" /></div></Td> | |
| 203 | + <Td><Score value={s.max_importance} size="sm" /></Td> | |
| 204 | + </tr> | |
| 205 | + ))} | |
| 206 | + </Table> | |
| 207 | + )} | |
| 208 | + </Panel> | |
| 209 | + ); | |
| 210 | +} | |
| 211 | + | |
| 212 | +function UnusualTab({ items }: { items: NonNullable<Awaited<ReturnType<typeof api.explore>>>["unusual_activity"] }) { | |
| 213 | + return ( | |
| 214 | + <Panel title="Unusual activity · last 2 h vs. 14-day baseline" dense action={<Link href="/radar" className="text-[11px] text-fg-subtle hover:text-fg">radar →</Link>}> | |
| 215 | + {items.length === 0 ? ( | |
| 216 | + <Empty>Activity anomalies compare the last 2 h of raw changes with a 14-day baseline. Nothing is far above its baseline right now.</Empty> | |
| 217 | + ) : ( | |
| 218 | + <Table head={["Source", "Domain", "Changes 2 h", "Baseline / day", "vs. baseline", "Activity score"]}> | |
| 219 | + {items.map((s) => { | |
| 220 | + const tone = s.activity_score >= 70 ? "hot" : s.activity_score >= 45 ? "high" : "signal"; | |
| 221 | + return ( | |
| 222 | + <tr key={s.id} className="hover:bg-panel-2/60"> | |
| 223 | + <Td><Link href={`/source/${s.id}`} className="font-medium hover:underline">{s.name}</Link></Td> | |
| 224 | + <Td mono><Link href={`/domain/${s.domain}`} className="text-fg-muted hover:underline">{s.domain}</Link></Td> | |
| 225 | + <Td mono>{s.changes_2h}</Td> | |
| 226 | + <Td mono className="text-fg-subtle">{s.baseline_per_day}</Td> | |
| 227 | + <Td mono className={`font-semibold ${tone === "hot" ? "text-hot" : tone === "high" ? "text-high" : "text-fg-muted"}`}>{s.pct_vs_baseline !== null && s.pct_vs_baseline !== undefined ? `+${Math.min(9999, Math.round(s.pct_vs_baseline)).toLocaleString("en-US")}%` : "—"}</Td> | |
| 228 | + <Td> | |
| 229 | + <div className="flex w-32 items-center gap-2"> | |
| 230 | + <Bar value={s.activity_score} tone={tone} /> | |
| 231 | + <span className="w-8 text-right font-mono text-[12px] font-semibold tabular">{fmtScore(s.activity_score)}</span> | |
| 232 | + </div> | |
| 233 | + </Td> | |
| 234 | + </tr> | |
| 235 | + ); | |
| 236 | + })} | |
| 237 | + </Table> | |
| 238 | + )} | |
| 239 | + </Panel> | |
| 240 | + ); | |
| 241 | +} | |
| 242 | + | |
| 243 | +function ClustersTab({ items }: { items: Cluster[] }) { | |
| 244 | + return ( | |
| 245 | + <Panel title="Event clusters · 48 h" dense action={<Link href="/breaking" className="text-[11px] text-fg-subtle hover:text-fg">breaking desk →</Link>}> | |
| 246 | + <ClusterRows items={items} empty="No multi-observation clusters yet. Related observations are grouped into clusters as they arrive." /> | |
| 247 | + </Panel> | |
| 248 | + ); | |
| 249 | +} | |
| 250 | + | |
| 251 | +function EntitiesTab({ items }: { items: Awaited<ReturnType<typeof api.rank>>["items"] }) { | |
| 252 | + return ( | |
| 253 | + <Panel title="Entity rankings · 7 d" dense action={<Link href="/entities" className="text-[11px] text-fg-subtle hover:text-fg">all entities →</Link>}> | |
| 254 | + {items.length === 0 ? ( | |
| 255 | + <Empty>Rankings appear once entities accumulate events.</Empty> | |
| 256 | + ) : ( | |
| 257 | + <Table head={["#", "Entity", "Type", "24 h", "7 d", "Baseline / d", "Sources", "Silent", "Breaking", "Avg signal", "Confirmed", "Last", "Rank score"]}> | |
| 258 | + {items.map((e) => ( | |
| 259 | + <tr key={e.id} className="hover:bg-panel-2/60"> | |
| 260 | + <Td mono className="text-fg-subtle">{e.rank}</Td> | |
| 261 | + <Td> | |
| 262 | + <Link href={`/entity/${e.id}`} className="font-medium hover:underline">{e.name}</Link> | |
| 263 | + {e.domain && <div className="truncate font-mono text-[10.5px] text-fg-subtle">{e.domain}</div>} | |
| 264 | + </Td> | |
| 265 | + <Td><Chip>{e.type.replace(/_/g, " ")}</Chip></Td> | |
| 266 | + <Td mono>{fmtInt(e.events_24h)}</Td> | |
| 267 | + <Td mono>{fmtInt(e.events_7d)}</Td> | |
| 268 | + <Td mono className="text-fg-subtle">{e.baseline_per_day.toFixed(1)}</Td> | |
| 269 | + <Td mono>{e.sources}</Td> | |
| 270 | + <Td mono className={e.silent_24h ? "text-silent" : "text-fg-subtle"}>{e.silent_24h}</Td> | |
| 271 | + <Td mono className={e.breaking_24h ? "text-hot" : "text-fg-subtle"}>{e.breaking_24h}</Td> | |
| 272 | + <Td mono>{fmtScore(e.avg_signal)}</Td> | |
| 273 | + <Td mono className="text-fg-subtle">{Math.round(e.confirmed_ratio * 100)}%</Td> | |
| 274 | + <Td mono className="text-fg-subtle">{relTime(e.last_at)}</Td> | |
| 275 | + <Td> | |
| 276 | + <div className="flex w-24 items-center gap-2"> | |
| 277 | + <Bar value={e.rank_score} tone={e.rank_score >= 80 ? "hot" : e.rank_score >= 60 ? "high" : "signal"} /> | |
| 278 | + <span className="w-7 text-right font-mono text-[12px] font-semibold tabular">{fmtScore(e.rank_score)}</span> | |
| 279 | + </div> | |
| 280 | + </Td> | |
| 281 | + </tr> | |
| 282 | + ))} | |
| 283 | + </Table> | |
| 284 | + )} | |
| 285 | + </Panel> | |
| 286 | + ); | |
| 287 | +} | |
| 288 | + | |
| 289 | +function SourcesTab({ items }: { items: Awaited<ReturnType<typeof api.sources>>["items"] }) { | |
| 290 | + const top = [...items].sort((a, b) => (b.events_24h ?? 0) - (a.events_24h ?? 0)).slice(0, 50); | |
| 291 | + const max = Math.max(1, ...top.map((s) => s.events_24h ?? 0)); | |
| 292 | + return ( | |
| 293 | + <Panel title="Sources · top 50 by events in 24 h" dense action={<Link href="/sources" className="text-[11px] text-fg-subtle hover:text-fg">all sources →</Link>}> | |
| 294 | + {top.length === 0 ? ( | |
| 295 | + <Empty /> | |
| 296 | + ) : ( | |
| 297 | + <Table head={["Tier", "Source", "Country", "Kind", "Categories", "Sensors", "Events 24 h", "", "Total", "Last event"]}> | |
| 298 | + {top.map((s) => ( | |
| 299 | + <tr key={s.id} className="hover:bg-panel-2/60"> | |
| 300 | + <Td><TierBadge tier={s.tier} /></Td> | |
| 301 | + <Td> | |
| 302 | + <Link href={`/source/${s.id}`} className="font-medium hover:underline">{s.name}</Link> | |
| 303 | + <div className="truncate font-mono text-[10.5px] text-fg-subtle">{s.domain}</div> | |
| 304 | + </Td> | |
| 305 | + <Td><Flag code={s.country} /></Td> | |
| 306 | + <Td>{s.first_party === false ? <Badge kind="external" compact /> : <Badge kind="first-party" compact />}</Td> | |
| 307 | + <Td><div className="flex flex-wrap gap-1">{s.categories.slice(0, 3).map((c) => <Chip key={c} href={`/category/${c}`}>{c}</Chip>)}</div></Td> | |
| 308 | + <Td mono>{s.sensor_count ?? 0}</Td> | |
| 309 | + <Td mono>{fmtInt(s.events_24h ?? 0)}</Td> | |
| 310 | + <Td><div className="w-20 sm:w-32"><Bar value={s.events_24h ?? 0} max={max} /></div></Td> | |
| 311 | + <Td mono className="text-fg-subtle">{fmtInt(s.event_count)}</Td> | |
| 312 | + <Td mono className="text-fg-subtle">{s.last_event_at ? relTime(s.last_event_at) : "—"}</Td> | |
| 313 | + </tr> | |
| 314 | + ))} | |
| 315 | + </Table> | |
| 316 | + )} | |
| 317 | + </Panel> | |
| 318 | + ); | |
| 319 | +} | |
| 320 | + | |
| 321 | +function CategoriesTab({ byCategory, byType, labels }: { byCategory: { category: string; n: number }[]; byType: { event_type: string; n: number }[]; labels: Record<string, string> }) { | |
| 322 | + const maxCat = Math.max(1, ...byCategory.map((t) => t.n)); | |
| 323 | + const maxType = Math.max(1, ...byType.map((t) => t.n)); | |
| 324 | + const channelOf = (c: string): string | undefined => CHANNELS.find((ch) => ch.query.category === c)?.label; | |
| 325 | + return ( | |
| 326 | + <div className="grid gap-4 lg:grid-cols-2"> | |
| 327 | + <Panel title="Events by category · 7 d" dense> | |
| 328 | + {byCategory.length ? ( | |
| 329 | + <ul className="divide-y divide-line"> | |
| 330 | + {byCategory.slice(0, 30).map((t) => ( | |
| 331 | + <li key={t.category} className="grid grid-cols-[8rem_1fr_3.5rem] items-center gap-2 px-3 py-1.5 text-[12.5px] sm:grid-cols-[11rem_1fr_3.5rem]"> | |
| 332 | + <Link href={`/category/${t.category}`} className="truncate hover:underline" title={channelOf(t.category) ? `${channelOf(t.category)} desk` : t.category}>{t.category}</Link> | |
| 333 | + <Bar value={t.n} max={maxCat} tone="signal" /> | |
| 334 | + <span className="text-right font-mono text-fg-subtle tabular">{fmtInt(t.n)}</span> | |
| 335 | + </li> | |
| 336 | + ))} | |
| 337 | + </ul> | |
| 338 | + ) : ( | |
| 339 | + <Empty /> | |
| 340 | + )} | |
| 341 | + </Panel> | |
| 342 | + <Panel title="Events by type · 7 d" dense> | |
| 343 | + {byType.length ? ( | |
| 344 | + <ul className="divide-y divide-line"> | |
| 345 | + {byType.slice(0, 30).map((t) => ( | |
| 346 | + <li key={t.event_type} className="grid grid-cols-[8rem_1fr_3.5rem] items-center gap-2 px-3 py-1.5 text-[12.5px] sm:grid-cols-[11rem_1fr_3.5rem]"> | |
| 347 | + <Link href={`/live?event_type=${t.event_type}`} className="truncate hover:underline" title={labels[t.event_type] ?? typeLabel(t.event_type)}>{typeLabel(t.event_type)}</Link> | |
| 348 | + <Bar value={t.n} max={maxType} tone="info" /> | |
| 349 | + <span className="text-right font-mono text-fg-subtle tabular">{fmtInt(t.n)}</span> | |
| 350 | + </li> | |
| 351 | + ))} | |
| 352 | + </ul> | |
| 353 | + ) : ( | |
| 354 | + <Empty /> | |
| 355 | + )} | |
| 356 | + </Panel> | |
| 357 | + </div> | |
| 358 | + ); | |
| 359 | +} | |
| 360 | + | |
| 361 | +function CountriesTab({ items }: { items: Awaited<ReturnType<typeof api.countries>>["items"] }) { | |
| 362 | + return ( | |
| 363 | + <Panel title={`Countries · ${items.length}`} dense action={<Link href="/country" className="text-[11px] text-fg-subtle hover:text-fg">country desks →</Link>}> | |
| 364 | + {items.length === 0 ? ( | |
| 365 | + <Empty>Country desks appear once sources carry a country.</Empty> | |
| 366 | + ) : ( | |
| 367 | + <ul className="grid gap-px bg-line sm:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4"> | |
| 368 | + {items.map((c) => ( | |
| 369 | + <li key={c.country} className="bg-panel"> | |
| 370 | + <Link href={`/country/${c.slug}`} className="flex items-center gap-3 px-3 py-2 hover:bg-panel-2/60"> | |
| 371 | + <span className="w-6 text-center text-lg leading-none" aria-hidden>{c.flag || <Flag code={c.country} />}</span> | |
| 372 | + <span className="min-w-0 flex-1"> | |
| 373 | + <span className="block truncate text-[13px] font-medium">{c.name}</span> | |
| 374 | + <span className="block font-mono text-[10.5px] text-fg-subtle">{c.country} · {c.sources} source{c.sources === 1 ? "" : "s"}</span> | |
| 375 | + </span> | |
| 376 | + <span className="flex flex-col items-end font-mono text-[12px] tabular"> | |
| 377 | + <span>{fmtInt(c.events_24h)} <span className="text-[10px] text-fg-subtle">24 h</span></span> | |
| 378 | + <span className={c.breaking_24h ? "text-hot" : "text-fg-subtle"}>{fmtInt(c.breaking_24h)} <span className="text-[10px] text-fg-subtle">brk</span></span> | |
| 379 | + </span> | |
| 380 | + </Link> | |
| 381 | + </li> | |
| 382 | + ))} | |
| 383 | + </ul> | |
| 384 | + )} | |
| 385 | + </Panel> | |
| 386 | + ); | |
| 387 | +} | |
modified
apps/web/src/app/globals.css
+103 −10
@@ -9,6 +9,7 @@ | ||
| 9 | 9 | --color-bg: var(--bg); |
| 10 | 10 | --color-panel: var(--panel); |
| 11 | 11 | --color-panel-2: var(--panel-2); |
| 12 | + --color-panel-3: var(--panel-3); | |
| 12 | 13 | --color-fg: var(--fg); |
| 13 | 14 | --color-fg-muted: var(--fg-muted); |
| 14 | 15 | --color-fg-subtle: var(--fg-subtle); |
@@ -34,6 +35,10 @@ | ||
| 34 | 35 | --animate-pulse-dot: pulse-dot 1.6s ease-in-out infinite; |
| 35 | 36 | --animate-flash: flash 1.4s ease-out both; |
| 36 | 37 | --animate-fade-in: fade-in 0.2s ease-out both; |
| 38 | + --animate-slide-in: slide-in 0.22s cubic-bezier(0.2, 0.8, 0.2, 1) both; | |
| 39 | + --animate-sheet-up: sheet-up 0.24s cubic-bezier(0.2, 0.8, 0.2, 1) both; | |
| 40 | + --animate-row-in: row-in 0.35s ease-out both; | |
| 41 | + --animate-tick: tick 0.6s ease-out both; | |
| 37 | 42 | |
| 38 | 43 | @keyframes pulse-dot { |
| 39 | 44 | 0%, 100% { box-shadow: 0 0 0 0 color-mix(in oklab, var(--signal) 55%, transparent); } |
@@ -47,18 +52,35 @@ | ||
| 47 | 52 | from { opacity: 0; transform: translateY(-2px); } |
| 48 | 53 | to { opacity: 1; transform: translateY(0); } |
| 49 | 54 | } |
| 55 | + @keyframes slide-in { | |
| 56 | + from { opacity: 0; transform: translateX(16px); } | |
| 57 | + to { opacity: 1; transform: translateX(0); } | |
| 58 | + } | |
| 59 | + @keyframes sheet-up { | |
| 60 | + from { opacity: 0; transform: translateY(24px); } | |
| 61 | + to { opacity: 1; transform: translateY(0); } | |
| 62 | + } | |
| 63 | + @keyframes row-in { | |
| 64 | + from { opacity: 0; transform: translateY(-6px); background-color: color-mix(in oklab, var(--signal) 18%, transparent); } | |
| 65 | + to { opacity: 1; transform: translateY(0); background-color: transparent; } | |
| 66 | + } | |
| 67 | + @keyframes tick { | |
| 68 | + 0% { color: var(--signal); } | |
| 69 | + 100% { color: inherit; } | |
| 70 | + } | |
| 50 | 71 | } |
| 51 | 72 | |
| 52 | 73 | :root { |
| 53 | 74 | color-scheme: light; |
| 54 | − --bg: #f7f8fb; | |
| 75 | + --bg: #f6f7fa; | |
| 55 | 76 | --panel: #ffffff; |
| 56 | − --panel-2: #f0f2f6; | |
| 77 | + --panel-2: #eff1f5; | |
| 78 | + --panel-3: #e6e9ef; | |
| 57 | 79 | --fg: #0d1220; |
| 58 | 80 | --fg-muted: #4a5468; |
| 59 | 81 | --fg-subtle: #7f889b; |
| 60 | − --line: #e2e6ee; | |
| 61 | − --line-strong: #c6cdd9; | |
| 82 | + --line: #e1e5ec; | |
| 83 | + --line-strong: #c3cad6; | |
| 62 | 84 | --signal: #0f9f7d; |
| 63 | 85 | --signal-soft: #dcf5ee; |
| 64 | 86 | --hot: #d12b2b; |
@@ -71,18 +93,25 @@ | ||
| 71 | 93 | --warn: #c57a0a; |
| 72 | 94 | --ok: #0f9f7d; |
| 73 | 95 | --info: #2b63d9; |
| 96 | + | |
| 97 | + /* density (spec §32) — overridden by [data-density] */ | |
| 98 | + --row-py: 7px; | |
| 99 | + --row-gap: 4px; | |
| 100 | + --fs-row: 13.5px; | |
| 101 | + --fs-meta: 11.5px; | |
| 74 | 102 | } |
| 75 | 103 | |
| 76 | 104 | .dark { |
| 77 | 105 | color-scheme: dark; |
| 78 | − --bg: #0b0f17; | |
| 79 | − --panel: #111726; | |
| 80 | − --panel-2: #161d2e; | |
| 81 | − --fg: #e5e7eb; | |
| 106 | + --bg: #0a0e15; | |
| 107 | + --panel: #10161f; | |
| 108 | + --panel-2: #151c28; | |
| 109 | + --panel-3: #1b2331; | |
| 110 | + --fg: #e6e8ee; | |
| 82 | 111 | --fg-muted: #a3adc2; |
| 83 | 112 | --fg-subtle: #6b7591; |
| 84 | − --line: #1f2937; | |
| 85 | − --line-strong: #2f3b52; | |
| 113 | + --line: #1c2533; | |
| 114 | + --line-strong: #2d394e; | |
| 86 | 115 | --signal: #22d3a5; |
| 87 | 116 | --signal-soft: #0f2a24; |
| 88 | 117 | --hot: #ff5c5c; |
@@ -97,6 +126,19 @@ | ||
| 97 | 126 | --info: #6ea0ff; |
| 98 | 127 | } |
| 99 | 128 | |
| 129 | +[data-density="compact"] { | |
| 130 | + --row-py: 3px; | |
| 131 | + --row-gap: 2px; | |
| 132 | + --fs-row: 12.5px; | |
| 133 | + --fs-meta: 10.5px; | |
| 134 | +} | |
| 135 | +[data-density="comfortable"] { | |
| 136 | + --row-py: 11px; | |
| 137 | + --row-gap: 6px; | |
| 138 | + --fs-row: 14.5px; | |
| 139 | + --fs-meta: 12px; | |
| 140 | +} | |
| 141 | + | |
| 100 | 142 | @layer base { |
| 101 | 143 | * { |
| 102 | 144 | border-color: var(--line); |
@@ -104,6 +146,7 @@ | ||
| 104 | 146 | html { |
| 105 | 147 | -webkit-text-size-adjust: 100%; |
| 106 | 148 | text-rendering: optimizeLegibility; |
| 149 | + scrollbar-gutter: stable; | |
| 107 | 150 | } |
| 108 | 151 | body { |
| 109 | 152 | @apply bg-bg text-fg font-sans antialiased; |
@@ -134,6 +177,12 @@ | ||
| 134 | 177 | border-radius: 6px; |
| 135 | 178 | border: 2px solid var(--bg); |
| 136 | 179 | } |
| 180 | + @media (prefers-reduced-motion: reduce) { | |
| 181 | + *, *::before, *::after { | |
| 182 | + animation-duration: 0.01ms !important; | |
| 183 | + transition-duration: 0.01ms !important; | |
| 184 | + } | |
| 185 | + } | |
| 137 | 186 | } |
| 138 | 187 | |
| 139 | 188 | @utility hairline { |
@@ -154,8 +203,52 @@ | ||
| 154 | 203 | color: var(--fg-subtle); |
| 155 | 204 | font-weight: 600; |
| 156 | 205 | } |
| 206 | +@utility row-dense { | |
| 207 | + padding-top: var(--row-py); | |
| 208 | + padding-bottom: var(--row-py); | |
| 209 | + font-size: var(--fs-row); | |
| 210 | +} | |
| 211 | +@utility meta { | |
| 212 | + font-size: var(--fs-meta); | |
| 213 | +} | |
| 214 | +@utility offscreen-ok { | |
| 215 | + content-visibility: auto; | |
| 216 | + contain-intrinsic-size: auto 64px; | |
| 217 | +} | |
| 218 | +@utility no-scrollbar { | |
| 219 | + scrollbar-width: none; | |
| 220 | + &::-webkit-scrollbar { display: none; } | |
| 221 | +} | |
| 157 | 222 | |
| 158 | 223 | .diff-line-add { background: color-mix(in oklab, var(--ok) 14%, transparent); color: var(--fg); } |
| 159 | 224 | .diff-line-del { background: color-mix(in oklab, var(--danger) 14%, transparent); color: var(--fg); } |
| 160 | 225 | .diff-line-meta { color: var(--fg-subtle); } |
| 161 | 226 | .diff-line-hunk { color: var(--info); background: color-mix(in oklab, var(--info) 8%, transparent); } |
| 227 | + | |
| 228 | +/* skeletons (spec §113) */ | |
| 229 | +.skeleton { | |
| 230 | + position: relative; | |
| 231 | + overflow: hidden; | |
| 232 | + background: color-mix(in oklab, var(--panel-2) 80%, var(--panel-3)); | |
| 233 | + border-radius: 4px; | |
| 234 | +} | |
| 235 | +.skeleton::after { | |
| 236 | + content: ""; | |
| 237 | + position: absolute; | |
| 238 | + inset: 0; | |
| 239 | + transform: translateX(-100%); | |
| 240 | + background: linear-gradient(90deg, transparent, color-mix(in oklab, var(--fg) 6%, transparent), transparent); | |
| 241 | + animation: shimmer 1.4s infinite; | |
| 242 | +} | |
| 243 | +@keyframes shimmer { 100% { transform: translateX(100%); } } | |
| 244 | + | |
| 245 | +/* heatmap cells (spec §102) */ | |
| 246 | +.heat-0 { background: var(--panel-2); } | |
| 247 | +.heat-1 { background: color-mix(in oklab, var(--signal) 22%, var(--panel-2)); } | |
| 248 | +.heat-2 { background: color-mix(in oklab, var(--signal) 42%, var(--panel-2)); } | |
| 249 | +.heat-3 { background: color-mix(in oklab, var(--signal) 65%, var(--panel-2)); } | |
| 250 | +.heat-4 { background: var(--signal); } | |
| 251 | +.heat-hot { background: var(--hot); } | |
| 252 | + | |
| 253 | +/* sheet / drawer scrim */ | |
| 254 | +.scrim { background: color-mix(in oklab, #000 45%, transparent); backdrop-filter: blur(2px); } | |
added
apps/web/src/app/health/loading.tsx
+38 −0
@@ -0,0 +1,38 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3 flex flex-col gap-2"> | |
| 7 | + <Skeleton className="h-3 w-28" /> | |
| 8 | + <Skeleton className="h-6 w-48" /> | |
| 9 | + <Skeleton className="h-3 w-full max-w-2xl" /> | |
| 10 | + </div> | |
| 11 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-4 xl:grid-cols-8 xl:divide-y-0"> | |
| 12 | + {Array.from({ length: 8 }, (_, i) => ( | |
| 13 | + <div key={i} className="flex flex-col gap-1.5 px-3 py-2"> | |
| 14 | + <Skeleton className="h-2.5 w-16" /> | |
| 15 | + <Skeleton className="h-5 w-12" /> | |
| 16 | + <Skeleton className="h-2.5 w-20" /> | |
| 17 | + </div> | |
| 18 | + ))} | |
| 19 | + </div> | |
| 20 | + <Skeleton className="mb-4 h-8 w-full" /> | |
| 21 | + <div className="grid grid-cols-1 gap-3 sm:grid-cols-2 lg:grid-cols-3"> | |
| 22 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 23 | + <SkeletonPanel key={i} lines={6} /> | |
| 24 | + ))} | |
| 25 | + </div> | |
| 26 | + <div className="mt-4 grid grid-cols-1 gap-4 lg:grid-cols-2"> | |
| 27 | + <div className="panel"> | |
| 28 | + <Skeleton className="m-3 h-2.5 w-32" /> | |
| 29 | + <SkeletonRows rows={4} /> | |
| 30 | + </div> | |
| 31 | + <div className="panel"> | |
| 32 | + <Skeleton className="m-3 h-2.5 w-32" /> | |
| 33 | + <SkeletonRows rows={4} /> | |
| 34 | + </div> | |
| 35 | + </div> | |
| 36 | + </> | |
| 37 | + ); | |
| 38 | +} | |
modified
apps/web/src/app/health/page.tsx
+122 −14
@@ -2,26 +2,62 @@ import Link from "next/link"; | ||
| 2 | 2 | import type { Metadata } from "next"; |
| 3 | 3 | import { Chip, Empty, HealthPill, PageHeader, Panel, Stat, Table, Td } from "@/components/ui"; |
| 4 | 4 | import { api } from "@/lib/api"; |
| 5 | −import { fmtBytes, fmtInt, fmtMs, fmtPct, relTime } from "@/lib/format"; | |
| 5 | +import { fmtBytes, fmtInt, fmtMs, fmtPct, relTime, utcDateTime, utcTime } from "@/lib/format"; | |
| 6 | 6 | |
| 7 | 7 | export const dynamic = "force-dynamic"; |
| 8 | −export const metadata: Metadata = { title: "Connector health", description: "Live health of every connector family and sensor: success rate, latency, HTTP codes, noise ratios." }; | |
| 8 | +export const metadata: Metadata = { title: "Connector health", description: "Live health of every connector family and sensor: success rate, latency, HTTP codes, noise ratios, queue depth and engine heartbeat." }; | |
| 9 | + | |
| 10 | +/** A heartbeat older than two minutes means the scheduler stopped publishing. */ | |
| 11 | +function heartbeatStale(at: string | undefined): boolean { | |
| 12 | + return Boolean(at) && Date.now() - new Date(at as string).getTime() > 120_000; | |
| 13 | +} | |
| 9 | 14 | |
| 10 | 15 | export default async function HealthPage() { |
| 11 | 16 | const h = await api.health(); |
| 12 | 17 | const byHealth = Object.fromEntries((h?.sensors_by_health ?? []).map((x) => [x.health, x.n])); |
| 18 | + const tp = h?.throughput; | |
| 19 | + const eng = h?.engine ?? null; | |
| 20 | + const engStale = heartbeatStale(eng?.at); | |
| 21 | + const healthy = byHealth.UP ?? 0; | |
| 22 | + const degraded = (byHealth.DEGRADED ?? 0) + (byHealth.RATE_LIMITED ?? 0); | |
| 23 | + const failed = byHealth.ERROR ?? 0; | |
| 13 | 24 | return ( |
| 14 | 25 | <> |
| 15 | − <PageHeader kicker="Internal observability" title="Connector health" description="Every connector exposes UP / DEGRADED / ERROR / RATE_LIMITED with 24 h success rate, latency, changes and events. Noisy sensors point to canonicalizer work, not more LLM calls." /> | |
| 16 | − <div className="panel mb-4 grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-3 lg:grid-cols-6 lg:divide-y-0"> | |
| 17 | − <Stat label="Sensors UP" value={fmtInt(byHealth.UP ?? 0)} /> | |
| 18 | − <Stat label="Degraded" value={fmtInt(byHealth.DEGRADED ?? 0)} /> | |
| 19 | − <Stat label="Error" value={fmtInt(byHealth.ERROR ?? 0)} /> | |
| 20 | − <Stat label="Rate limited" value={fmtInt(byHealth.RATE_LIMITED ?? 0)} /> | |
| 21 | − <Stat label="WS clients" value={fmtInt(h?.live.clients ?? 0)} hint={`${fmtInt(h?.live.published ?? 0)} published since start`} /> | |
| 22 | − <Stat label="Connectors" value={fmtInt(h?.connectors.length ?? 0)} /> | |
| 26 | + <PageHeader compact kicker="Public observability" title="Connector health" description="Every connector exposes UP / DEGRADED / ERROR / RATE_LIMITED with 24 h success rate, latency, changes and events. Noisy sensors point to canonicalizer work, not more LLM calls." /> | |
| 27 | + | |
| 28 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-4 xl:grid-cols-8 xl:divide-y-0"> | |
| 29 | + <Stat label="Healthy" value={fmtInt(healthy)} tone="signal" hint="sensors UP" /> | |
| 30 | + <Stat label="Degraded" value={fmtInt(degraded)} tone={degraded ? "warn" : undefined} hint={`${fmtInt(byHealth.RATE_LIMITED ?? 0)} rate-limited`} /> | |
| 31 | + <Stat label="Failed" value={fmtInt(failed)} tone={failed ? "hot" : undefined} hint="consecutive errors" /> | |
| 32 | + <Stat label="Checks / min" value={tp ? tp.checks_per_min.toFixed(1) : "—"} hint={tp ? `${fmtInt(tp.checks_5m)} in 5 min` : undefined} /> | |
| 33 | + <Stat label="Events / min" value={tp ? tp.events_per_min.toFixed(2) : "—"} tone="signal" hint={tp ? `${fmtInt(tp.events_5m)} in 5 min · ${fmtInt(tp.changes_5m)} raw` : undefined} /> | |
| 34 | + <Stat label="304 ratio" value={fmtPct(tp?.not_modified_ratio ?? null)} hint="conditional GET hits · 5 min" /> | |
| 35 | + <Stat label="Queue depth" value={fmtInt(tp?.queue_due ?? 0)} tone={(tp?.queue_due ?? 0) > 500 ? "warn" : undefined} hint="sensors due now" /> | |
| 36 | + <Stat label="Avg latency" value={fmtMs(tp?.avg_latency_5m_ms)} hint={`${fmtInt(tp?.noise_filtered_24h ?? 0)} noise filtered · 24 h`} /> | |
| 23 | 37 | </div> |
| 24 | − <div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-3"> | |
| 38 | + | |
| 39 | + <div className="mb-4 flex flex-wrap items-center gap-x-3 gap-y-1 rounded-md border border-line bg-panel px-3 py-2 font-mono text-[11.5px] text-fg-muted tabular"> | |
| 40 | + <span className="flex items-center gap-1.5"> | |
| 41 | + <span className={`inline-block size-2 rounded-full ${eng && !engStale ? "bg-signal animate-pulse-dot" : "bg-warn"}`} /> | |
| 42 | + <span className="label !text-fg-subtle">engine</span> | |
| 43 | + <span className={eng && !engStale ? "text-signal" : "text-warn"}>{eng ? (engStale ? "STALE" : "LIVE") : "NO HEARTBEAT"}</span> | |
| 44 | + </span> | |
| 45 | + {eng ? ( | |
| 46 | + <> | |
| 47 | + <span>in flight {eng.inflight ?? 0} / {eng.concurrency ?? "—"}</span> | |
| 48 | + <span>due {fmtInt(eng.due)}</span> | |
| 49 | + <span>busy hosts {eng.busyHosts?.length ?? 0}</span> | |
| 50 | + <span className={eng.circuitOpen?.length ? "text-warn" : ""}>circuit open {eng.circuitOpen?.length ?? 0}</span> | |
| 51 | + {eng.version && <span>v{eng.version}</span>} | |
| 52 | + {eng.at && <span className="text-fg-subtle" title={utcDateTime(eng.at)}>heartbeat {utcTime(eng.at)} UTC · {relTime(eng.at)}</span>} | |
| 53 | + </> | |
| 54 | + ) : ( | |
| 55 | + <span className="text-fg-subtle">the scheduler publishes a heartbeat to Redis every few seconds; none is visible right now.</span> | |
| 56 | + )} | |
| 57 | + <span className="ml-auto text-fg-subtle">WS clients {fmtInt(h?.live.clients ?? 0)} · {fmtInt(h?.live.published ?? 0)} published since start</span> | |
| 58 | + </div> | |
| 59 | + | |
| 60 | + <div className="grid grid-cols-1 gap-3 sm:grid-cols-2 lg:grid-cols-3"> | |
| 25 | 61 | {(h?.connectors ?? []).map((c) => ( |
| 26 | 62 | <Panel key={c.connector} title={<span className="flex items-center gap-2 normal-case tracking-normal"><span className="font-mono text-[13px] text-fg">{c.connector}</span><HealthPill health={c.status} /></span>}> |
| 27 | 63 | <dl className="grid grid-cols-2 gap-y-1 text-[12.5px]"> |
@@ -42,7 +78,43 @@ export default async function HealthPage() { | ||
| 42 | 78 | ))} |
| 43 | 79 | {!h?.connectors.length && <Panel dense><Empty>Health rollups appear after the first minute of engine activity.</Empty></Panel>} |
| 44 | 80 | </div> |
| 45 | − <div className="mt-4 grid gap-4 lg:grid-cols-2"> | |
| 81 | + | |
| 82 | + <div className="mt-4 grid grid-cols-1 gap-4 lg:grid-cols-[minmax(0,2fr)_minmax(0,1fr)]"> | |
| 83 | + <Panel title="Sensors by connector" dense> | |
| 84 | + {h?.sensors_by_connector?.length ? ( | |
| 85 | + <Table head={["Connector", "Sensors", "Up", "Up ratio", "Avg latency"]}> | |
| 86 | + {h.sensors_by_connector.map((c) => ( | |
| 87 | + <tr key={c.connector}> | |
| 88 | + <Td mono>{c.connector}</Td> | |
| 89 | + <Td mono>{fmtInt(c.sensors)}</Td> | |
| 90 | + <Td mono>{fmtInt(c.up)}</Td> | |
| 91 | + <Td mono className={c.sensors && c.up / c.sensors < 0.9 ? "text-warn" : ""}>{fmtPct(c.sensors ? c.up / c.sensors : null)}</Td> | |
| 92 | + <Td mono>{fmtMs(c.avg_latency_ms)}</Td> | |
| 93 | + </tr> | |
| 94 | + ))} | |
| 95 | + </Table> | |
| 96 | + ) : ( | |
| 97 | + <Empty /> | |
| 98 | + )} | |
| 99 | + </Panel> | |
| 100 | + <Panel title="Sensors by status" dense> | |
| 101 | + {h?.sensors_by_status?.length ? ( | |
| 102 | + <Table head={["Status", "Sensors"]}> | |
| 103 | + {h.sensors_by_status.map((s) => ( | |
| 104 | + <tr key={s.status}> | |
| 105 | + <Td><HealthPill health={s.status} /></Td> | |
| 106 | + <Td mono>{fmtInt(s.n)}</Td> | |
| 107 | + </tr> | |
| 108 | + ))} | |
| 109 | + </Table> | |
| 110 | + ) : ( | |
| 111 | + <Empty /> | |
| 112 | + )} | |
| 113 | + <p className="border-t border-line px-3 py-2 text-[11px] text-fg-subtle">PENDING → VALIDATED → ACTIVE is the sensor lifecycle; DISABLED sensors are kept for evidence but never scheduled.</p> | |
| 114 | + </Panel> | |
| 115 | + </div> | |
| 116 | + | |
| 117 | + <div className="mt-4 grid grid-cols-1 gap-4 lg:grid-cols-2"> | |
| 46 | 118 | <Panel title="Degraded sensors" dense> |
| 47 | 119 | {h?.degraded_sensors.length ? ( |
| 48 | 120 | <Table head={["Sensor", "Source", "Health", "Errors", "HTTP", "Last error", "Last check"]}> |
@@ -53,8 +125,8 @@ export default async function HealthPage() { | ||
| 53 | 125 | <Td><HealthPill health={s.health} /></Td> |
| 54 | 126 | <Td mono>{s.consecutive_errors}</Td> |
| 55 | 127 | <Td mono>{s.last_status ?? "—"}</Td> |
| 56 | − <Td className="max-w-[16rem] truncate text-danger" >{s.last_error}</Td> | |
| 57 | − <Td mono className="text-fg-subtle">{s.last_check_at ? relTime(s.last_check_at) : "—"}</Td> | |
| 128 | + <Td className="max-w-[16rem] truncate text-danger"><span title={s.last_error ?? ""}>{s.last_error}</span></Td> | |
| 129 | + <Td mono className="whitespace-nowrap text-fg-subtle">{s.last_check_at ? relTime(s.last_check_at) : "—"}</Td> | |
| 58 | 130 | </tr> |
| 59 | 131 | ))} |
| 60 | 132 | </Table> |
@@ -80,6 +152,41 @@ export default async function HealthPage() { | ||
| 80 | 152 | )} |
| 81 | 153 | </Panel> |
| 82 | 154 | </div> |
| 155 | + | |
| 156 | + <div className="mt-4 grid grid-cols-1 gap-4 lg:grid-cols-2"> | |
| 157 | + <Panel title="Top failing domains" dense> | |
| 158 | + {h?.top_failing_domains?.length ? ( | |
| 159 | + <Table head={["Host", "Failures", "Last error"]}> | |
| 160 | + {h.top_failing_domains.map((f) => ( | |
| 161 | + <tr key={f.host}> | |
| 162 | + <Td mono><Link href={`/domain/${f.host}`} className="hover:underline">{f.host}</Link></Td> | |
| 163 | + <Td mono className="text-danger">{fmtInt(f.failures)}</Td> | |
| 164 | + <Td className="max-w-[20rem] truncate font-mono text-[11px] text-fg-muted"><span title={f.last_error ?? ""}>{f.last_error ?? "—"}</span></Td> | |
| 165 | + </tr> | |
| 166 | + ))} | |
| 167 | + </Table> | |
| 168 | + ) : ( | |
| 169 | + <Empty>No failing domains right now.</Empty> | |
| 170 | + )} | |
| 171 | + </Panel> | |
| 172 | + <Panel title="Slowest sensors" dense> | |
| 173 | + {h?.slowest_sensors?.length ? ( | |
| 174 | + <Table head={["Sensor", "Source", "Connector", "Avg latency"]}> | |
| 175 | + {h.slowest_sensors.map((s) => ( | |
| 176 | + <tr key={s.id}> | |
| 177 | + <Td><Link href={`/sensor/${s.id}`} className="hover:underline">{s.name}</Link></Td> | |
| 178 | + <Td><Link href={`/source/${s.source_id}`} className="hover:underline">{s.source_id}</Link></Td> | |
| 179 | + <Td mono>{s.connector}</Td> | |
| 180 | + <Td mono className={s.avg_latency_ms > 10_000 ? "text-warn" : ""}>{fmtMs(s.avg_latency_ms)}</Td> | |
| 181 | + </tr> | |
| 182 | + ))} | |
| 183 | + </Table> | |
| 184 | + ) : ( | |
| 185 | + <Empty /> | |
| 186 | + )} | |
| 187 | + </Panel> | |
| 188 | + </div> | |
| 189 | + | |
| 83 | 190 | <Panel title="Daily metrics" dense className="mt-4"> |
| 84 | 191 | {h?.daily.length ? ( |
| 85 | 192 | <Table head={["Day", "Checks", "304", "Bytes", "Raw changes", "Events", "Silent", "Errors", "LLM calls", "LLM tokens in/out"]}> |
@@ -102,6 +209,7 @@ export default async function HealthPage() { | ||
| 102 | 209 | <Empty /> |
| 103 | 210 | )} |
| 104 | 211 | </Panel> |
| 212 | + <p className="mt-3 text-[11px] text-fg-subtle">Rollups refresh every minute from sensor runs; raw counters are in <code>/api/v1/health/connectors</code> and <code>/api/metrics</code> (Prometheus). Operators: <Link href="/ops" className="hover:text-fg">/ops</Link>.</p> | |
| 105 | 213 | </> |
| 106 | 214 | ); |
| 107 | 215 | } |
modified
apps/web/src/app/layout.tsx
+18 −9
@@ -1,8 +1,11 @@ | ||
| 1 | 1 | import type { Metadata, Viewport } from "next"; |
| 2 | 2 | import { Inter, JetBrains_Mono } from "next/font/google"; |
| 3 | 3 | import type { ReactNode } from "react"; |
| 4 | +import { CommandPalette } from "@/components/command-palette"; | |
| 5 | +import { EventDrawerProvider } from "@/components/event-drawer"; | |
| 4 | 6 | import { Footer } from "@/components/footer"; |
| 5 | 7 | import { MobileNav, TopNav } from "@/components/nav"; |
| 8 | +import { PrefsProvider } from "@/components/prefs"; | |
| 6 | 9 | import { Providers } from "@/components/theme"; |
| 7 | 10 | import { SITE_URL } from "@/lib/api"; |
| 8 | 11 | import "./globals.css"; |
@@ -12,26 +15,32 @@ const mono = JetBrains_Mono({ subsets: ["latin"], variable: "--font-jetbrains", | ||
| 12 | 15 | |
| 13 | 16 | export const metadata: Metadata = { |
| 14 | 17 | metadataBase: new URL(SITE_URL), |
| 15 | − title: { default: "WebSensor — Detect What Changed. Know Why It Matters.", template: "%s · WebSensor" }, | |
| 16 | − description: "Real-time intelligence from the changing Web. WebSensor monitors official public sources, detects meaningful changes, scores their importance and publishes them live.", | |
| 18 | + title: { default: "WebSensor — The Web is changing. We are watching.", template: "%s · WebSensor" }, | |
| 19 | + description: "WebSensor continuously watches the public web for meaningful changes, preserving evidence and connecting signals into real-world events.", | |
| 17 | 20 | applicationName: "WebSensor", |
| 18 | − openGraph: { type: "website", siteName: "WebSensor", url: SITE_URL, title: "WebSensor — The Web, Live.", description: "A global sensor network for the changing Web." }, | |
| 19 | − twitter: { card: "summary_large_image", title: "WebSensor — The Web, Live.", description: "Detect What Changed. Know Why It Matters." }, | |
| 21 | + openGraph: { type: "website", siteName: "WebSensor", url: SITE_URL, title: "WebSensor — The Web is changing. We are watching.", description: "Real-time Internet change intelligence: what changed, where, when, whether it was silent, how important it is — with immutable evidence." }, | |
| 22 | + twitter: { card: "summary_large_image", title: "WebSensor — The Web is changing. We are watching.", description: "Detect what changed. Know why it matters." }, | |
| 20 | 23 | alternates: { canonical: "/", types: { "application/rss+xml": `${SITE_URL}/api/v1/feed.rss` } }, |
| 21 | 24 | robots: { index: true, follow: true }, |
| 22 | 25 | }; |
| 23 | 26 | |
| 24 | −export const viewport: Viewport = { themeColor: [{ media: "(prefers-color-scheme: dark)", color: "#0b0f17" }, { media: "(prefers-color-scheme: light)", color: "#f7f8fb" }], width: "device-width", initialScale: 1 }; | |
| 27 | +export const viewport: Viewport = { themeColor: [{ media: "(prefers-color-scheme: dark)", color: "#0a0e15" }, { media: "(prefers-color-scheme: light)", color: "#f6f7fa" }], width: "device-width", initialScale: 1, viewportFit: "cover" }; | |
| 25 | 28 | |
| 26 | 29 | export default function RootLayout({ children }: { children: ReactNode }) { |
| 27 | 30 | return ( |
| 28 | 31 | <html lang="en" className={`${inter.variable} ${mono.variable} dark`} suppressHydrationWarning> |
| 29 | 32 | <body className="min-h-dvh"> |
| 30 | 33 | <Providers> |
| 31 | − <TopNav /> | |
| 32 | − <main className="mx-auto w-full max-w-[1500px] px-3 py-4 sm:px-4">{children}</main> | |
| 33 | − <Footer /> | |
| 34 | − <MobileNav /> | |
| 34 | + <PrefsProvider> | |
| 35 | + <EventDrawerProvider> | |
| 36 | + <a href="#main" className="sr-only focus:not-sr-only focus:fixed focus:left-2 focus:top-2 focus:z-[70] focus:rounded-md focus:bg-panel focus:px-3 focus:py-2">Skip to content</a> | |
| 37 | + <TopNav /> | |
| 38 | + <main id="main" className="mx-auto w-full max-w-[1600px] px-3 py-4 pb-24 sm:px-4 lg:pb-6">{children}</main> | |
| 39 | + <Footer /> | |
| 40 | + <MobileNav /> | |
| 41 | + <CommandPalette /> | |
| 42 | + </EventDrawerProvider> | |
| 43 | + </PrefsProvider> | |
| 35 | 44 | </Providers> |
| 36 | 45 | </body> |
| 37 | 46 | </html> |
added
apps/web/src/app/live/loading.tsx
+30 −0
@@ -0,0 +1,30 @@ | ||
| 1 | +import { Skeleton, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-4 flex flex-col gap-2"> | |
| 7 | + <Skeleton className="h-3 w-40" /> | |
| 8 | + <Skeleton className="h-7 w-56" /> | |
| 9 | + <div className="flex flex-wrap gap-1"> | |
| 10 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 11 | + <Skeleton key={i} className="h-5 w-20" /> | |
| 12 | + ))} | |
| 13 | + </div> | |
| 14 | + </div> | |
| 15 | + <div className="panel overflow-hidden"> | |
| 16 | + <div className="flex items-center gap-3 border-b border-line px-3 py-2"> | |
| 17 | + <Skeleton className="h-3 w-24" /> | |
| 18 | + <Skeleton className="h-3 w-12" /> | |
| 19 | + <Skeleton className="ml-auto h-6 w-28" /> | |
| 20 | + </div> | |
| 21 | + <div className="flex gap-1 border-b border-line px-2 py-1.5"> | |
| 22 | + {Array.from({ length: 8 }, (_, i) => ( | |
| 23 | + <Skeleton key={i} className="h-5 w-16" /> | |
| 24 | + ))} | |
| 25 | + </div> | |
| 26 | + <SkeletonRows rows={14} /> | |
| 27 | + </div> | |
| 28 | + </> | |
| 29 | + ); | |
| 30 | +} | |
added
apps/web/src/app/live/page.tsx
+39 −0
@@ -0,0 +1,39 @@ | ||
| 1 | +import type { Metadata } from "next"; | |
| 2 | +import Link from "next/link"; | |
| 3 | +import { LiveFeed } from "@/components/live-feed"; | |
| 4 | +import { filtersFromParams } from "@/lib/feed-filters"; | |
| 5 | +import { PageHeader } from "@/components/ui"; | |
| 6 | +import { api, type EventQuery } from "@/lib/api"; | |
| 7 | +import { SAVED_VIEWS, feedHref } from "@/lib/format"; | |
| 8 | + | |
| 9 | +export const dynamic = "force-dynamic"; | |
| 10 | +export const metadata: Metadata = { title: "Live", description: "Every meaningful change detected on the public web, live. Filters persist in the URL and can be shared." }; | |
| 11 | + | |
| 12 | +/** Full-width live feed with URL-persisted filters (spec §99) and saved views (spec §100). */ | |
| 13 | +export default async function LivePage({ searchParams }: { searchParams: Promise<Record<string, string | undefined>> }) { | |
| 14 | + const sp = await searchParams; | |
| 15 | + const f = filtersFromParams(sp); | |
| 16 | + const q: EventQuery = { ...f, limit: 60 }; | |
| 17 | + const events = await api.events(q); | |
| 18 | + const activeView = SAVED_VIEWS.find((v) => Object.entries(v.query).every(([k, val]) => String(sp[k] ?? "") === String(val)) && Object.keys(sp).length === Object.keys(v.query).length); | |
| 19 | + return ( | |
| 20 | + <> | |
| 21 | + <PageHeader | |
| 22 | + compact | |
| 23 | + kicker="Live feed" | |
| 24 | + title={activeView ? activeView.label : "Live"} | |
| 25 | + description="Everything WebSensor detects, as it happens. Filters live in the URL — share the link to share the view." | |
| 26 | + actions={ | |
| 27 | + <div className="flex flex-wrap gap-1"> | |
| 28 | + {SAVED_VIEWS.map((v) => ( | |
| 29 | + <Link key={v.key} href={feedHref(v.query, "/live")} className={`rounded-md border px-2 py-1 text-[11.5px] ${activeView?.key === v.key ? "border-signal/50 bg-signal-soft text-signal" : "border-line text-fg-muted hover:text-fg"}`}> | |
| 30 | + {v.label} | |
| 31 | + </Link> | |
| 32 | + ))} | |
| 33 | + </div> | |
| 34 | + } | |
| 35 | + /> | |
| 36 | + <LiveFeed key={JSON.stringify(f)} initial={events.items} initialCursor={events.nextCursor} initialFilters={f} syncUrl showTabs={!f.category} title="LIVE WEB" /> | |
| 37 | + </> | |
| 38 | + ); | |
| 39 | +} | |
added
apps/web/src/app/monitors/monitors.tsx
+310 −0
@@ -0,0 +1,310 @@ | ||
| 1 | +"use client"; | |
| 2 | + | |
| 3 | +import { Check, Plus, RefreshCw, Trash2 } from "lucide-react"; | |
| 4 | +import { useCallback, useEffect, useState } from "react"; | |
| 5 | +import { EventRow } from "@/components/event-row"; | |
| 6 | +import { FieldChangeInline } from "@/components/field-changes"; | |
| 7 | +import { Chip, Empty, HealthPill, Panel, SkeletonRows, Table, Td } from "@/components/ui"; | |
| 8 | +import { clientApiBase, type ChangeRow, type EventItem, type Monitor } from "@/lib/api"; | |
| 9 | +import { CLASS_LABELS, fmtBytes, hostOf, relTime, untilTime, utcDateTime } from "@/lib/format"; | |
| 10 | +import { ownerToken } from "@/lib/owner"; | |
| 11 | + | |
| 12 | +interface ApiError { | |
| 13 | + status: number; | |
| 14 | + error: string; | |
| 15 | + detail?: string; | |
| 16 | + limit?: number; | |
| 17 | +} | |
| 18 | + | |
| 19 | +/** Owner-scoped request that keeps the API's error shape (`{error, detail?}`) instead of flattening it. */ | |
| 20 | +async function request<T>(path: string, init: RequestInit = {}): Promise<T> { | |
| 21 | + const res = await fetch(`${clientApiBase()}${path}`, { ...init, headers: { "content-type": "application/json", accept: "application/json", "x-websensor-owner": ownerToken(), ...(init.headers ?? {}) } }); | |
| 22 | + if (!res.ok) { | |
| 23 | + const body = (await res.json().catch(() => ({}))) as Partial<ApiError> & { issues?: { path?: (string | number)[]; message?: string }[] }; | |
| 24 | + const detail = body.detail ?? body.issues?.map((i) => `${(i.path ?? []).join(".")}: ${i.message ?? ""}`).join("; "); | |
| 25 | + throw Object.assign(new Error(body.error ?? `HTTP ${res.status}`), { status: res.status, error: body.error ?? `http_${res.status}`, detail, limit: body.limit } satisfies ApiError); | |
| 26 | + } | |
| 27 | + return (await res.json()) as T; | |
| 28 | +} | |
| 29 | + | |
| 30 | +const ERROR_HELP: Record<string, string> = { | |
| 31 | + url_rejected: "Private, loopback, metadata and internal addresses are refused by the SSRF policy.", | |
| 32 | + scheme_not_allowed: "Only http:// and https:// URLs can be monitored.", | |
| 33 | + fetch_failed: "The page could not be fetched. Check the address, TLS certificate and that the host answers public requests.", | |
| 34 | + http_error: "The page answered with an error status. Monitors need a 2xx page to take a baseline.", | |
| 35 | + unparseable: "The response could not be normalized as HTML/text.", | |
| 36 | + thin_content: "Almost no server-rendered text was found (client-side app?). Try a more specific URL or a CSS selector.", | |
| 37 | + monitor_limit_reached: "Monitor limit reached for this browser. Delete one to add another.", | |
| 38 | + not_found: "Monitor not found — it may have been deleted.", | |
| 39 | +}; | |
| 40 | + | |
| 41 | +interface Form { | |
| 42 | + url: string; | |
| 43 | + name: string; | |
| 44 | + frequency: "hourly" | "daily"; | |
| 45 | + sensitivity: "low" | "normal" | "high"; | |
| 46 | + selector: string; | |
| 47 | + keywords: string; | |
| 48 | +} | |
| 49 | +const EMPTY: Form = { url: "", name: "", frequency: "hourly", sensitivity: "normal", selector: "", keywords: "" }; | |
| 50 | + | |
| 51 | +interface CreateResult { | |
| 52 | + id: string; | |
| 53 | + url: string; | |
| 54 | + tier: string; | |
| 55 | + status: string; | |
| 56 | + test?: { http_status: number; content_type?: string | null; bytes?: number | null; title?: string | null; mode?: string; extraction_confidence?: number | null }; | |
| 57 | +} | |
| 58 | + | |
| 59 | +export function Monitors() { | |
| 60 | + const [items, setItems] = useState<Monitor[] | null>(null); | |
| 61 | + const [limit, setLimit] = useState(5); | |
| 62 | + const [form, setForm] = useState<Form>(EMPTY); | |
| 63 | + const [err, setErr] = useState<ApiError | null>(null); | |
| 64 | + const [busy, setBusy] = useState(false); | |
| 65 | + const [created, setCreated] = useState<CreateResult | null>(null); | |
| 66 | + const [selected, setSelected] = useState<string | null>(null); | |
| 67 | + /** Keyed by monitor id so switching monitors shows a loading state without a synchronous reset. */ | |
| 68 | + const [loaded, setLoaded] = useState<{ id: string; events: EventItem[]; changes: ChangeRow[]; error: string | null } | null>(null); | |
| 69 | + const detail = loaded && loaded.id === selected ? loaded : null; | |
| 70 | + | |
| 71 | + const load = useCallback( | |
| 72 | + () => | |
| 73 | + request<{ items: Monitor[]; limit: number }>("/api/v1/monitors") | |
| 74 | + .then((r) => { | |
| 75 | + setItems(r.items); | |
| 76 | + setLimit(r.limit); | |
| 77 | + setSelected((cur) => cur ?? r.items[0]?.id ?? null); | |
| 78 | + }) | |
| 79 | + .catch((e: ApiError) => { | |
| 80 | + setErr(e); | |
| 81 | + setItems([]); | |
| 82 | + }), | |
| 83 | + [], | |
| 84 | + ); | |
| 85 | + useEffect(() => { | |
| 86 | + void load(); | |
| 87 | + const t = setInterval(() => void load(), 30_000); | |
| 88 | + return () => clearInterval(t); | |
| 89 | + }, [load]); | |
| 90 | + | |
| 91 | + useEffect(() => { | |
| 92 | + if (!selected) return; | |
| 93 | + const id = selected; | |
| 94 | + let cancelled = false; | |
| 95 | + request<{ events: EventItem[]; changes: ChangeRow[] }>(`/api/v1/monitors/${encodeURIComponent(id)}/events?limit=50`) | |
| 96 | + .then((r) => { | |
| 97 | + if (!cancelled) setLoaded({ id, events: r.events, changes: r.changes, error: null }); | |
| 98 | + }) | |
| 99 | + .catch((e: ApiError) => { | |
| 100 | + if (!cancelled) setLoaded({ id, events: [], changes: [], error: `${e.error}${e.detail ? ` — ${e.detail}` : ""}` }); | |
| 101 | + }); | |
| 102 | + return () => { | |
| 103 | + cancelled = true; | |
| 104 | + }; | |
| 105 | + }, [selected, items]); | |
| 106 | + | |
| 107 | + const create = async (): Promise<void> => { | |
| 108 | + setErr(null); | |
| 109 | + setCreated(null); | |
| 110 | + const url = form.url.trim(); | |
| 111 | + if (!/^https?:\/\//i.test(url)) { | |
| 112 | + setErr({ status: 0, error: "invalid_url", detail: "Enter a full address starting with http:// or https://." }); | |
| 113 | + return; | |
| 114 | + } | |
| 115 | + const keywords = form.keywords.split(/[,\n]/).map((k) => k.trim()).filter(Boolean).slice(0, 20); | |
| 116 | + setBusy(true); | |
| 117 | + try { | |
| 118 | + const r = await request<CreateResult>("/api/v1/monitors", { method: "POST", body: JSON.stringify({ url, name: form.name.trim() || undefined, frequency: form.frequency, sensitivity: form.sensitivity, selector: form.selector.trim() || undefined, keywords: keywords.length ? keywords : undefined }) }); | |
| 119 | + setCreated(r); | |
| 120 | + setForm(EMPTY); | |
| 121 | + await load(); | |
| 122 | + setSelected(r.id); | |
| 123 | + } catch (e) { | |
| 124 | + setErr(e as ApiError); | |
| 125 | + } finally { | |
| 126 | + setBusy(false); | |
| 127 | + } | |
| 128 | + }; | |
| 129 | + | |
| 130 | + const remove = async (m: Monitor): Promise<void> => { | |
| 131 | + setItems((prev) => prev?.filter((x) => x.id !== m.id) ?? prev); | |
| 132 | + if (selected === m.id) setSelected(null); | |
| 133 | + try { | |
| 134 | + await request(`/api/v1/monitors/${encodeURIComponent(m.id)}`, { method: "DELETE" }); | |
| 135 | + } catch (e) { | |
| 136 | + setErr(e as ApiError); | |
| 137 | + } | |
| 138 | + await load(); | |
| 139 | + }; | |
| 140 | + | |
| 141 | + const active = items?.filter((m) => m.enabled).length ?? 0; | |
| 142 | + const current = items?.find((m) => m.id === selected) ?? null; | |
| 143 | + const input = "h-8 rounded-md border border-line bg-panel px-2 text-[12.5px]"; | |
| 144 | + | |
| 145 | + return ( | |
| 146 | + <div className="grid gap-4 lg:grid-cols-[380px_minmax(0,1fr)]"> | |
| 147 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 148 | + <Panel title={<span>New monitor <span className="normal-case tracking-normal text-fg-subtle">· {active} / {limit} used</span></span>}> | |
| 149 | + <form | |
| 150 | + className="flex flex-col gap-3 text-[12.5px]" | |
| 151 | + onSubmit={(e) => { | |
| 152 | + e.preventDefault(); | |
| 153 | + void create(); | |
| 154 | + }} | |
| 155 | + > | |
| 156 | + <label className="flex flex-col gap-1"> | |
| 157 | + <span className="label">URL</span> | |
| 158 | + <input type="url" required value={form.url} onChange={(e) => setForm({ ...form, url: e.target.value })} placeholder="https://example.com/pricing" className={`${input} font-mono text-[12px]`} /> | |
| 159 | + </label> | |
| 160 | + <label className="flex flex-col gap-1"> | |
| 161 | + <span className="label">Name <span className="normal-case tracking-normal">· optional</span></span> | |
| 162 | + <input value={form.name} onChange={(e) => setForm({ ...form, name: e.target.value })} placeholder={form.url ? hostOf(form.url) : "Defaults to the host name"} maxLength={80} className={input} /> | |
| 163 | + </label> | |
| 164 | + <div className="grid grid-cols-2 gap-3"> | |
| 165 | + <div> | |
| 166 | + <div className="label mb-1">Frequency</div> | |
| 167 | + <div className="flex gap-1" role="radiogroup" aria-label="Frequency"> | |
| 168 | + {(["hourly", "daily"] as const).map((f) => ( | |
| 169 | + <button key={f} type="button" role="radio" aria-checked={form.frequency === f} onClick={() => setForm({ ...form, frequency: f })} className={`h-8 flex-1 rounded-md border text-[12px] ${form.frequency === f ? "border-signal/50 bg-signal-soft text-signal" : "border-line bg-panel text-fg-muted hover:text-fg"}`}> | |
| 170 | + {f} | |
| 171 | + </button> | |
| 172 | + ))} | |
| 173 | + </div> | |
| 174 | + </div> | |
| 175 | + <div> | |
| 176 | + <div className="label mb-1">Sensitivity</div> | |
| 177 | + <div className="flex gap-1" role="radiogroup" aria-label="Sensitivity"> | |
| 178 | + {(["low", "normal", "high"] as const).map((s) => ( | |
| 179 | + <button key={s} type="button" role="radio" aria-checked={form.sensitivity === s} onClick={() => setForm({ ...form, sensitivity: s })} title={s === "low" ? "Only big changes" : s === "high" ? "Every meaningful change" : "Balanced"} className={`h-8 flex-1 rounded-md border text-[12px] ${form.sensitivity === s ? "border-signal/50 bg-signal-soft text-signal" : "border-line bg-panel text-fg-muted hover:text-fg"}`}> | |
| 180 | + {s} | |
| 181 | + </button> | |
| 182 | + ))} | |
| 183 | + </div> | |
| 184 | + </div> | |
| 185 | + </div> | |
| 186 | + <label className="flex flex-col gap-1"> | |
| 187 | + <span className="label">CSS selector <span className="normal-case tracking-normal">· optional, narrows the watched region</span></span> | |
| 188 | + <input value={form.selector} onChange={(e) => setForm({ ...form, selector: e.target.value })} placeholder="main .pricing-table" maxLength={200} className={`${input} font-mono text-[12px]`} /> | |
| 189 | + </label> | |
| 190 | + <label className="flex flex-col gap-1"> | |
| 191 | + <span className="label">Keywords <span className="normal-case tracking-normal">· optional, comma-separated</span></span> | |
| 192 | + <input value={form.keywords} onChange={(e) => setForm({ ...form, keywords: e.target.value })} placeholder="price, deprecated, discontinued" className={input} /> | |
| 193 | + </label> | |
| 194 | + <button type="submit" disabled={busy || active >= limit} className="inline-flex h-8 items-center justify-center gap-1 rounded-md border border-line bg-panel-2 px-3 text-[12.5px] hover:border-line-strong disabled:opacity-60"> | |
| 195 | + {busy ? <RefreshCw className="size-3.5 animate-spin" /> : <Plus className="size-3.5" />} {busy ? "Fetching & testing…" : active >= limit ? "Limit reached" : "Create monitor"} | |
| 196 | + </button> | |
| 197 | + {err && ( | |
| 198 | + <div className="rounded-md border border-danger/40 bg-danger/10 px-2 py-1.5 text-[12px]"> | |
| 199 | + <div className="font-mono font-semibold text-danger">{err.error}{err.status ? ` · HTTP ${err.status}` : ""}</div> | |
| 200 | + {err.detail && <div className="mt-0.5 break-words text-fg">{err.detail}</div>} | |
| 201 | + {ERROR_HELP[err.error] && <div className="mt-0.5 text-fg-muted">{ERROR_HELP[err.error]}</div>} | |
| 202 | + </div> | |
| 203 | + )} | |
| 204 | + {created && ( | |
| 205 | + <div className="rounded-md border border-ok/40 bg-ok/10 px-2 py-1.5 text-[12px]"> | |
| 206 | + <div className="flex items-center gap-1 font-semibold text-ok"><Check className="size-3.5" /> Monitor created · baseline test passed</div> | |
| 207 | + {created.test && ( | |
| 208 | + <div className="mt-0.5 font-mono text-[11px] text-fg-muted tabular"> | |
| 209 | + HTTP {created.test.http_status} · {created.test.content_type ?? "—"} · {fmtBytes(created.test.bytes)} · {created.test.mode ?? "text"} · confidence {created.test.extraction_confidence !== null && created.test.extraction_confidence !== undefined ? Math.round(created.test.extraction_confidence * 100) : "—"}% | |
| 210 | + {created.test.title && <div className="truncate text-fg">“{created.test.title}”</div>} | |
| 211 | + </div> | |
| 212 | + )} | |
| 213 | + </div> | |
| 214 | + )} | |
| 215 | + </form> | |
| 216 | + </Panel> | |
| 217 | + <Panel title="Limits & rules"> | |
| 218 | + <ul className="list-disc space-y-1 pl-4 text-[12.5px] text-fg-muted"> | |
| 219 | + <li><span className="text-fg">{limit} monitors per browser</span>, scoped to this browser's anonymous owner token.</li> | |
| 220 | + <li><span className="text-fg">Public URLs only.</span> Private networks, loopback and cloud-metadata addresses are rejected; redirects are checked hop by hop.</li> | |
| 221 | + <li><span className="text-fg">Respects robots and rate limits.</span> Conditional GET, one request per interval, per-host concurrency caps, circuit breaker on failing hosts.</li> | |
| 222 | + <li><span className="text-fg">No authentication bypass.</span> Pages behind logins, paywalls or bot walls are not fetched with credentials or headless browsers.</li> | |
| 223 | + <li>Hourly = tier C, daily = tier D. The baseline is taken on the first engine run; the first event needs a second snapshot.</li> | |
| 224 | + <li>Custom monitors never enter the public feed, rankings or clusters.</li> | |
| 225 | + </ul> | |
| 226 | + </Panel> | |
| 227 | + </aside> | |
| 228 | + | |
| 229 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 230 | + <Panel title={<span>Your monitors <span className="font-mono text-fg-subtle">{items?.length ?? "…"}</span></span>} dense> | |
| 231 | + {items === null ? ( | |
| 232 | + <SkeletonRows rows={3} /> | |
| 233 | + ) : items.length ? ( | |
| 234 | + <Table head={["Monitor", "Status", "Schedule", "Last check", "HTTP", "Changes", ""]}> | |
| 235 | + {items.map((m) => { | |
| 236 | + const sens = typeof m.config.sensitivity === "string" ? m.config.sensitivity : "normal"; | |
| 237 | + const sel = m.id === selected; | |
| 238 | + return ( | |
| 239 | + <tr key={m.id} className={sel ? "bg-panel-2/60" : ""}> | |
| 240 | + <Td> | |
| 241 | + <button type="button" aria-pressed={sel} onClick={() => setSelected(m.id)} className="block max-w-[18rem] truncate text-left font-medium hover:underline">{m.name}</button> | |
| 242 | + <a href={m.url} target="_blank" rel="noopener noreferrer nofollow" className="block max-w-[18rem] truncate font-mono text-[10.5px] text-fg-subtle hover:text-info" title={m.url}>{m.url}</a> | |
| 243 | + </Td> | |
| 244 | + <Td> | |
| 245 | + <div className="flex flex-wrap gap-1"> | |
| 246 | + <HealthPill health={m.enabled ? m.health : "DISABLED"} /> | |
| 247 | + <Chip className="font-mono">{m.status}</Chip> | |
| 248 | + </div> | |
| 249 | + </Td> | |
| 250 | + <Td> | |
| 251 | + <div className="flex flex-wrap gap-1"> | |
| 252 | + <Chip>{m.tier === "D" ? "daily" : "hourly"}</Chip> | |
| 253 | + <Chip tone={sens === "high" ? "high" : sens === "low" ? "default" : "info"}>{sens}</Chip> | |
| 254 | + </div> | |
| 255 | + <div className="mt-0.5 font-mono text-[10.5px] text-fg-subtle tabular">next {untilTime(m.next_check_at)}</div> | |
| 256 | + </Td> | |
| 257 | + <Td mono className="whitespace-nowrap text-fg-subtle">{m.last_check_at ? <span title={utcDateTime(m.last_check_at)}>{relTime(m.last_check_at)}</span> : "pending"}</Td> | |
| 258 | + <Td mono className={m.last_status && m.last_status >= 400 ? "text-danger" : ""}>{m.last_status ?? "—"}</Td> | |
| 259 | + <Td mono className="whitespace-nowrap" ><span title="raw / meaningful">{m.raw_changes ?? 0} / {m.meaningful_changes ?? 0}</span></Td> | |
| 260 | + <Td> | |
| 261 | + <button type="button" aria-label={`Delete monitor ${m.name}`} onClick={() => remove(m)} className="text-fg-subtle hover:text-danger"><Trash2 className="size-3.5" /></button> | |
| 262 | + </Td> | |
| 263 | + </tr> | |
| 264 | + ); | |
| 265 | + })} | |
| 266 | + </Table> | |
| 267 | + ) : ( | |
| 268 | + <Empty>No monitors yet — add a public URL on the left. The first check takes a baseline; changes are reported from the second check on.</Empty> | |
| 269 | + )} | |
| 270 | + {items?.some((m) => m.last_error) && ( | |
| 271 | + <ul className="border-t border-line px-3 py-2 text-[11px]"> | |
| 272 | + {items.filter((m) => m.last_error).map((m) => ( | |
| 273 | + <li key={m.id} className="truncate font-mono text-danger" title={m.last_error ?? ""}>{m.name}: {m.last_error}</li> | |
| 274 | + ))} | |
| 275 | + </ul> | |
| 276 | + )} | |
| 277 | + </Panel> | |
| 278 | + | |
| 279 | + <Panel title={<span>Events {current && <span className="normal-case tracking-normal text-fg-subtle">· {current.name}</span>}</span>} dense> | |
| 280 | + {!current ? ( | |
| 281 | + <Empty>Select a monitor to see its events and raw changes.</Empty> | |
| 282 | + ) : detail === null ? ( | |
| 283 | + <SkeletonRows rows={3} /> | |
| 284 | + ) : detail.events.length ? ( | |
| 285 | + detail.events.map((e) => <EventRow key={e.id} ev={e} showDate />) | |
| 286 | + ) : ( | |
| 287 | + <Empty>{detail.error ?? (current.total_runs ? "No meaningful change detected yet." : "Waiting for the first check to take a baseline.")}</Empty> | |
| 288 | + )} | |
| 289 | + </Panel> | |
| 290 | + | |
| 291 | + {current && detail && detail.changes.length > 0 && ( | |
| 292 | + <Panel title={<span>Raw changes <span className="font-mono text-fg-subtle">{detail.changes.length}</span> <span className="normal-case tracking-normal text-fg-subtle">· before noise filtering</span></span>} dense> | |
| 293 | + <Table head={["Detected", "Kind", "Class", "Signal", "Fields", "Event"]}> | |
| 294 | + {detail.changes.map((c) => ( | |
| 295 | + <tr key={c.id}> | |
| 296 | + <Td mono className="whitespace-nowrap text-fg-subtle"><span title={utcDateTime(c.detected_at)}>{relTime(c.detected_at)}</span></Td> | |
| 297 | + <Td mono>{c.kind}</Td> | |
| 298 | + <Td>{c.change_class ? <Chip tone={c.meaningful ? "signal" : "default"}>{CLASS_LABELS[c.change_class] ?? c.change_class}</Chip> : <span className="text-fg-subtle">—</span>}</Td> | |
| 299 | + <Td mono>{Math.round(Number(c.signal))}</Td> | |
| 300 | + <Td>{c.field_changes?.length ? <FieldChangeInline items={c.field_changes} max={2} /> : <span className="text-fg-subtle">—</span>}</Td> | |
| 301 | + <Td>{c.event_id ? <Chip tone="ok">event</Chip> : <span className="text-[11px] text-fg-subtle">filtered</span>}</Td> | |
| 302 | + </tr> | |
| 303 | + ))} | |
| 304 | + </Table> | |
| 305 | + </Panel> | |
| 306 | + )} | |
| 307 | + </div> | |
| 308 | + </div> | |
| 309 | + ); | |
| 310 | +} | |
added
apps/web/src/app/monitors/page.tsx
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +import type { Metadata } from "next"; | |
| 2 | +import { PageHeader } from "@/components/ui"; | |
| 3 | +import { Monitors } from "./monitors"; | |
| 4 | + | |
| 5 | +export const metadata: Metadata = { title: "Custom monitors", description: "Watch any public URL with WebSensor's change engine: canonical extraction, semantic diff, noise filtering and evidence, scoped to your browser.", robots: { index: false } }; | |
| 6 | + | |
| 7 | +export default function MonitorsPage() { | |
| 8 | + return ( | |
| 9 | + <> | |
| 10 | + <PageHeader compact kicker="Stored in this browser (no account yet)" title="Custom monitors" description="Point WebSensor at a public page. It is fetched, canonicalized and diffed like every registry sensor; meaningful changes become events visible only to this browser." /> | |
| 11 | + <Monitors /> | |
| 12 | + </> | |
| 13 | + ); | |
| 14 | +} | |
modified
apps/web/src/app/not-found.tsx
+19 −8
@@ -1,15 +1,26 @@ | ||
| 1 | +import { Radar, Search } from "lucide-react"; | |
| 1 | 2 | import Link from "next/link"; |
| 3 | +import { Kbd, Panel } from "@/components/ui"; | |
| 2 | 4 | |
| 3 | 5 | export default function NotFound() { |
| 4 | 6 | return ( |
| 5 | − <div className="mx-auto max-w-lg py-20 text-center"> | |
| 6 | − <div className="label mb-2">404</div> | |
| 7 | − <h1 className="text-xl font-semibold">Nothing observed at this address</h1> | |
| 8 | − <p className="mt-2 text-[13px] text-fg-muted">The page may have moved, or the event, source or entity does not exist.</p> | |
| 9 | − <div className="mt-5 flex justify-center gap-3 text-[13px]"> | |
| 10 | − <Link href="/" className="rounded-md border border-line bg-panel px-3 py-1.5 hover:border-line-strong">Live feed</Link> | |
| 11 | − <Link href="/search" className="rounded-md border border-line bg-panel px-3 py-1.5 hover:border-line-strong">Search</Link> | |
| 12 | − </div> | |
| 7 | + <div className="mx-auto max-w-lg py-12 sm:py-20"> | |
| 8 | + <Panel dense> | |
| 9 | + <div className="flex flex-col items-center gap-2 px-4 py-10 text-center"> | |
| 10 | + <span className="font-mono text-[11px] font-semibold tracking-wider text-fg-subtle">HTTP 404 · NOT OBSERVED</span> | |
| 11 | + <Radar className="mt-1 size-6 text-fg-subtle" aria-hidden /> | |
| 12 | + <h1 className="mt-1 text-lg font-semibold leading-tight">Nothing observed at this address</h1> | |
| 13 | + <p className="max-w-sm text-[13px] text-fg-muted">The page may have moved, or the event, source, entity or cluster does not exist. Sensors keep watching; this URL is not one of them.</p> | |
| 14 | + <div className="mt-4 flex flex-wrap justify-center gap-2 text-[12.5px]"> | |
| 15 | + <Link href="/live" className="inline-flex h-8 items-center rounded-md border border-line bg-panel px-3 hover:border-line-strong">Live feed</Link> | |
| 16 | + <Link href="/explore" className="inline-flex h-8 items-center rounded-md border border-line bg-panel px-3 hover:border-line-strong">Explore</Link> | |
| 17 | + <Link href="/search" className="inline-flex h-8 items-center gap-1.5 rounded-md border border-line bg-panel px-3 hover:border-line-strong"><Search className="size-3.5" /> Search</Link> | |
| 18 | + </div> | |
| 19 | + <p className="mt-3 flex items-center gap-1.5 text-[11px] text-fg-subtle"> | |
| 20 | + Press <Kbd>⌘</Kbd><Kbd>K</Kbd> anywhere to search entities, events and URLs. | |
| 21 | + </p> | |
| 22 | + </div> | |
| 23 | + </Panel> | |
| 13 | 24 | </div> |
| 14 | 25 | ); |
| 15 | 26 | } |
added
apps/web/src/app/ops/ops.tsx
+706 −0
@@ -0,0 +1,706 @@ | ||
| 1 | +"use client"; | |
| 2 | + | |
| 3 | +import { KeyRound, LogOut, Play, RefreshCw, ShieldAlert } from "lucide-react"; | |
| 4 | +import Link from "next/link"; | |
| 5 | +import { useEffect, useMemo, useState, type ReactNode } from "react"; | |
| 6 | +import { SENSOR_TYPES, TIERS } from "@websensor/core/client"; | |
| 7 | +import { useMounted } from "@/components/theme"; | |
| 8 | +import { Chip, Empty, HealthPill, Panel, Skeleton, Sparkline, Stat, Table, Td } from "@/components/ui"; | |
| 9 | +import { clientApiBase } from "@/lib/api"; | |
| 10 | +import { fmtBytes, fmtInt, fmtMs, fmtPct, relTime, untilTime, utcDateTime, utcTime } from "@/lib/format"; | |
| 11 | + | |
| 12 | +const KEY = "ws_admin"; | |
| 13 | + | |
| 14 | +// --------------------------------------------------------------------------------------- | |
| 15 | +// Types for GET /api/v1/admin/ops (kept local: internal shape, not part of the public client) | |
| 16 | +// --------------------------------------------------------------------------------------- | |
| 17 | + | |
| 18 | +interface Ops { | |
| 19 | + db: { latency_ms: number }; | |
| 20 | + queue: { due: number; overdue_10m: number; p0: number; p0_due: number; oldest_due: string | null }; | |
| 21 | + engine: { inflight?: number; concurrency?: number; due?: number; busyHosts?: { host: string; inflight: number }[]; circuitOpen?: { host: string; failures: number; until: string }[]; at?: string; version?: string; [k: string]: unknown } | null; | |
| 22 | + outcomes_1h: { outcome: string; n: number }[]; | |
| 23 | + top_failing_domains: { host: string; failures: number; sensors: number; last_error: string | null; last_at: string }[]; | |
| 24 | + slowest_sensors: { id: string; name: string; source_id: string; url: string; connector: string; avg_latency_ms: number; total_runs: number }[]; | |
| 25 | + throughput_24h: { t: string; checks: number; not_modified: number; errors: number; events: number; avg_ms: number | null }[]; | |
| 26 | + storage: { db_size: string; db_bytes: string | number; snapshots: number; snapshots_with_raw: number; raw_bytes_uncompressed: string | number; changes: number; events: number; runs: number; blobs: { files: number; bytes: number } | null }; | |
| 27 | + llm_24h: { model: string; calls: number; input_tokens: string | number; output_tokens: string | number; failures: number }[]; | |
| 28 | + sensors_by_status: { status: string; health: string; n: number }[]; | |
| 29 | + recent_errors: { started_at: string; sensor_id: string; source_id: string; connector: string; http_status: number | null; outcome: string; error: string | null }[]; | |
| 30 | + cache: { entries: number; keys: string[] }; | |
| 31 | + live: { clients: number; published: number }; | |
| 32 | + connectors: { key: string; name: string; version?: string; sensorTypes?: string[]; description?: string }[]; | |
| 33 | + generated_at: string; | |
| 34 | +} | |
| 35 | + | |
| 36 | +class AdminError extends Error { | |
| 37 | + status: number; | |
| 38 | + body: Record<string, unknown>; | |
| 39 | + constructor(status: number, body: Record<string, unknown>) { | |
| 40 | + super(typeof body.error === "string" ? body.error : `HTTP ${status}`); | |
| 41 | + this.status = status; | |
| 42 | + this.body = body; | |
| 43 | + } | |
| 44 | +} | |
| 45 | + | |
| 46 | +function readToken(): string { | |
| 47 | + if (typeof window === "undefined") return ""; | |
| 48 | + return window.sessionStorage.getItem(KEY) ?? ""; | |
| 49 | +} | |
| 50 | + | |
| 51 | +async function adminFetch<T>(path: string, init: RequestInit = {}, token = readToken()): Promise<T> { | |
| 52 | + const headers: Record<string, string> = { accept: "application/json", "x-websensor-admin": token, ...((init.headers as Record<string, string>) ?? {}) }; | |
| 53 | + if (init.body !== undefined && !headers["content-type"]) headers["content-type"] = "application/json"; | |
| 54 | + const res = await fetch(`${clientApiBase()}${path}`, { ...init, headers }); | |
| 55 | + const body = (await res.json().catch(() => ({}))) as Record<string, unknown>; | |
| 56 | + if (!res.ok) throw new AdminError(res.status, body); | |
| 57 | + return body as T; | |
| 58 | +} | |
| 59 | + | |
| 60 | +/** A heartbeat older than two minutes means the scheduler stopped publishing. */ | |
| 61 | +function heartbeatStale(at: string | undefined): boolean { | |
| 62 | + return Boolean(at) && Date.now() - new Date(at as string).getTime() > 120_000; | |
| 63 | +} | |
| 64 | + | |
| 65 | +function gateMessage(e: unknown): string { | |
| 66 | + if (e instanceof AdminError) { | |
| 67 | + if (e.status === 401) return "invalid token"; | |
| 68 | + if (e.status === 404) return "admin API disabled (WS_ADMIN_TOKEN not set)"; | |
| 69 | + return `${e.message}${typeof e.body.detail === "string" ? ` — ${e.body.detail}` : ""}`; | |
| 70 | + } | |
| 71 | + return (e as Error).message; | |
| 72 | +} | |
| 73 | + | |
| 74 | +// --------------------------------------------------------------------------------------- | |
| 75 | + | |
| 76 | +export function Ops() { | |
| 77 | + const mounted = useMounted(); | |
| 78 | + /** null = use the token stored in sessionStorage; "" = signed out; otherwise the token entered in this session. */ | |
| 79 | + const [override, setOverride] = useState<string | null>(null); | |
| 80 | + const token: string | null = !mounted ? null : override ?? readToken(); | |
| 81 | + const [gate, setGate] = useState<string | null>(null); | |
| 82 | + const [data, setData] = useState<Ops | null>(null); | |
| 83 | + const [err, setErr] = useState<string | null>(null); | |
| 84 | + const [refreshing, setRefreshing] = useState(false); | |
| 85 | + const [input, setInput] = useState(""); | |
| 86 | + const [tick, setTick] = useState(0); | |
| 87 | + | |
| 88 | + // Poll /admin/ops every 15 s while a token is present (and on manual refresh via `tick`). | |
| 89 | + useEffect(() => { | |
| 90 | + if (!token) return; | |
| 91 | + const t = token; | |
| 92 | + let cancelled = false; | |
| 93 | + const pull = (): void => { | |
| 94 | + adminFetch<Ops>("/api/v1/admin/ops", {}, t) | |
| 95 | + .then((d) => { | |
| 96 | + if (cancelled) return; | |
| 97 | + setData(d); | |
| 98 | + setErr(null); | |
| 99 | + setGate(null); | |
| 100 | + }) | |
| 101 | + .catch((e: unknown) => { | |
| 102 | + if (cancelled) return; | |
| 103 | + const msg = gateMessage(e); | |
| 104 | + if (e instanceof AdminError && (e.status === 401 || e.status === 404)) { | |
| 105 | + setGate(msg); | |
| 106 | + setData(null); | |
| 107 | + } else setErr(msg); | |
| 108 | + }) | |
| 109 | + .finally(() => { | |
| 110 | + if (!cancelled) setRefreshing(false); | |
| 111 | + }); | |
| 112 | + }; | |
| 113 | + pull(); | |
| 114 | + const iv = setInterval(pull, 15_000); | |
| 115 | + return () => { | |
| 116 | + cancelled = true; | |
| 117 | + clearInterval(iv); | |
| 118 | + }; | |
| 119 | + }, [token, tick]); | |
| 120 | + | |
| 121 | + const refresh = (): void => { | |
| 122 | + setRefreshing(true); | |
| 123 | + setTick((x) => x + 1); | |
| 124 | + }; | |
| 125 | + const submitToken = async (): Promise<void> => { | |
| 126 | + const t = input.trim(); | |
| 127 | + if (!t) return; | |
| 128 | + try { | |
| 129 | + const d = await adminFetch<Ops>("/api/v1/admin/ops", {}, t); | |
| 130 | + window.sessionStorage.setItem(KEY, t); | |
| 131 | + setData(d); | |
| 132 | + setGate(null); | |
| 133 | + setErr(null); | |
| 134 | + setOverride(t); | |
| 135 | + setInput(""); | |
| 136 | + } catch (e) { | |
| 137 | + window.sessionStorage.removeItem(KEY); | |
| 138 | + setGate(gateMessage(e)); | |
| 139 | + } | |
| 140 | + }; | |
| 141 | + const signOut = (): void => { | |
| 142 | + window.sessionStorage.removeItem(KEY); | |
| 143 | + setOverride(""); | |
| 144 | + setData(null); | |
| 145 | + setGate(null); | |
| 146 | + }; | |
| 147 | + | |
| 148 | + if (token === null) return <Skeleton className="h-24 w-full" />; | |
| 149 | + if (!token || (gate && !data)) { | |
| 150 | + return ( | |
| 151 | + <Panel> | |
| 152 | + <form | |
| 153 | + className="mx-auto flex max-w-md flex-col gap-2 py-6 text-[13px]" | |
| 154 | + onSubmit={(e) => { | |
| 155 | + e.preventDefault(); | |
| 156 | + void submitToken(); | |
| 157 | + }} | |
| 158 | + > | |
| 159 | + <p className="leading-relaxed text-fg-muted"> | |
| 160 | + <KeyRound className="mr-1.5 inline size-4 align-text-bottom" /> | |
| 161 | + Enter the admin token (<code>WS_ADMIN_TOKEN</code>). It is kept in <code>sessionStorage</code> for this tab only and sent as <code>X-WebSensor-Admin</code>. | |
| 162 | + </p> | |
| 163 | + <div className="flex gap-1"> | |
| 164 | + <input type="password" value={input} onChange={(e) => setInput(e.target.value)} placeholder="admin token" autoComplete="off" aria-label="Admin token" className="h-8 min-w-0 flex-1 rounded-md border border-line bg-panel px-2 font-mono text-[12.5px]" /> | |
| 165 | + <button type="submit" className="inline-flex h-8 items-center gap-1 rounded-md border border-line bg-panel-2 px-3 text-[12.5px] hover:border-line-strong">Unlock</button> | |
| 166 | + </div> | |
| 167 | + {gate && <p className="flex items-center gap-1 text-[12px] text-danger"><ShieldAlert className="size-3.5" /> {gate}</p>} | |
| 168 | + <p className="text-[11px] text-fg-subtle">Public health is at <Link href="/health" className="text-info hover:underline">/health</Link>.</p> | |
| 169 | + </form> | |
| 170 | + </Panel> | |
| 171 | + ); | |
| 172 | + } | |
| 173 | + | |
| 174 | + return ( | |
| 175 | + <div className="flex flex-col gap-4"> | |
| 176 | + <div className="flex flex-wrap items-center justify-between gap-2 text-[11.5px] text-fg-subtle"> | |
| 177 | + <span className="flex items-center gap-2"> | |
| 178 | + <span className={`inline-block size-2 rounded-full ${err ? "bg-danger" : "bg-signal animate-pulse-dot"}`} /> | |
| 179 | + {data ? `generated ${utcTime(data.generated_at)} UTC · db ${data.db.latency_ms} ms · ${data.live.clients} WS clients` : "loading…"} | |
| 180 | + {err && <span className="text-danger">· {err}</span>} | |
| 181 | + </span> | |
| 182 | + <span className="flex items-center gap-2"> | |
| 183 | + <button type="button" onClick={refresh} disabled={refreshing} className="inline-flex h-7 items-center gap-1 rounded-md border border-line bg-panel px-2 hover:border-line-strong disabled:opacity-60"><RefreshCw className={`size-3 ${refreshing ? "animate-spin" : ""}`} /> refresh</button> | |
| 184 | + <button type="button" onClick={signOut} className="inline-flex h-7 items-center gap-1 rounded-md border border-line bg-panel px-2 hover:border-line-strong"><LogOut className="size-3" /> forget token</button> | |
| 185 | + </span> | |
| 186 | + </div> | |
| 187 | + | |
| 188 | + {data ? <Dashboard d={data} /> : <Skeleton className="h-64 w-full" />} | |
| 189 | + | |
| 190 | + <Tools connectors={data?.connectors ?? []} onDone={refresh} /> | |
| 191 | + </div> | |
| 192 | + ); | |
| 193 | +} | |
| 194 | + | |
| 195 | +// --------------------------------------------------------------------------------------- | |
| 196 | +// Dashboard panels | |
| 197 | +// --------------------------------------------------------------------------------------- | |
| 198 | + | |
| 199 | +function Dashboard({ d }: { d: Ops }) { | |
| 200 | + const eng = d.engine; | |
| 201 | + const stale = heartbeatStale(eng?.at); | |
| 202 | + const tp = d.throughput_24h; | |
| 203 | + const sum = (k: "checks" | "events" | "errors" | "not_modified"): number => tp.reduce((a, r) => a + Number(r[k] ?? 0), 0); | |
| 204 | + const outcomes = d.outcomes_1h; | |
| 205 | + const oTotal = outcomes.reduce((a, o) => a + o.n, 0); | |
| 206 | + const byStatus = useMemo(() => { | |
| 207 | + const m = new Map<string, { n: number; health: Record<string, number> }>(); | |
| 208 | + for (const r of d.sensors_by_status) { | |
| 209 | + const cur = m.get(r.status) ?? { n: 0, health: {} }; | |
| 210 | + cur.n += r.n; | |
| 211 | + cur.health[r.health] = (cur.health[r.health] ?? 0) + r.n; | |
| 212 | + m.set(r.status, cur); | |
| 213 | + } | |
| 214 | + return [...m.entries()].sort((a, b) => b[1].n - a[1].n); | |
| 215 | + }, [d.sensors_by_status]); | |
| 216 | + const raw = Number(d.storage.raw_bytes_uncompressed); | |
| 217 | + const blobBytes = d.storage.blobs?.bytes ?? 0; | |
| 218 | + | |
| 219 | + return ( | |
| 220 | + <> | |
| 221 | + <div className="panel grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-4 xl:grid-cols-8 xl:divide-y-0"> | |
| 222 | + <Stat label="Engine" value={eng ? (stale ? "STALE" : "LIVE") : "NO HEARTBEAT"} tone={eng && !stale ? "signal" : "warn"} hint={eng?.at ? `${relTime(eng.at)} · ${eng.version ?? "?"}` : "Redis ws:engine:status empty"} /> | |
| 223 | + <Stat label="In flight" value={eng ? `${eng.inflight ?? 0} / ${eng.concurrency ?? "—"}` : "—"} hint="workers / concurrency" /> | |
| 224 | + <Stat label="Queue due" value={fmtInt(d.queue.due)} tone={d.queue.overdue_10m > 0 ? "warn" : undefined} hint={`${fmtInt(d.queue.overdue_10m)} overdue > 10 min`} /> | |
| 225 | + <Stat label="P0 · due" value={`${fmtInt(d.queue.p0)} · ${fmtInt(d.queue.p0_due)}`} tone={d.queue.p0_due > 0 ? "hot" : undefined} hint={d.queue.oldest_due ? `oldest ${untilTime(d.queue.oldest_due)}` : "—"} /> | |
| 226 | + <Stat label="Checks 24 h" value={fmtInt(sum("checks"))} hint={`${fmtPct(sum("checks") ? sum("not_modified") / sum("checks") : null)} 304`} /> | |
| 227 | + <Stat label="Events 24 h" value={fmtInt(sum("events"))} tone="signal" hint={`${fmtInt(sum("errors"))} errors`} /> | |
| 228 | + <Stat label="Circuit open" value={fmtInt(eng?.circuitOpen?.length ?? 0)} tone={(eng?.circuitOpen?.length ?? 0) > 0 ? "warn" : undefined} hint={`${eng?.busyHosts?.length ?? 0} busy hosts`} /> | |
| 229 | + <Stat label="Cache" value={fmtInt(d.cache.entries)} hint="TTL entries" /> | |
| 230 | + </div> | |
| 231 | + | |
| 232 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-3"> | |
| 233 | + <Panel title="Engine heartbeat"> | |
| 234 | + {eng ? ( | |
| 235 | + <dl className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1 text-[12.5px]"> | |
| 236 | + <Row k="At" v={utcDateTime(eng.at)} /> | |
| 237 | + <Row k="Version" v={eng.version ?? "—"} /> | |
| 238 | + <Row k="In flight" v={`${eng.inflight ?? 0} / ${eng.concurrency ?? "—"}`} /> | |
| 239 | + <Row k="Due (engine view)" v={fmtInt(eng.due)} /> | |
| 240 | + <Row k="Busy hosts" v={eng.busyHosts?.length ? eng.busyHosts.slice(0, 8).map((h) => `${h.host} ×${h.inflight}`).join(", ") : "none"} /> | |
| 241 | + <Row k="Circuit open" v={eng.circuitOpen?.length ? eng.circuitOpen.slice(0, 8).map((h) => `${h.host} (${h.failures}, until ${utcTime(h.until, false)})`).join(", ") : "none"} tone={eng.circuitOpen?.length ? "text-warn" : ""} /> | |
| 242 | + </dl> | |
| 243 | + ) : ( | |
| 244 | + <Empty>No heartbeat in Redis — the engine is not running or cannot reach Redis.</Empty> | |
| 245 | + )} | |
| 246 | + </Panel> | |
| 247 | + <Panel title="Outcomes · 1 h"> | |
| 248 | + {outcomes.length ? ( | |
| 249 | + <ul className="flex flex-col gap-1.5 text-[12.5px]"> | |
| 250 | + {outcomes.map((o) => ( | |
| 251 | + <li key={o.outcome} className="grid grid-cols-[7rem_1fr_3.5rem] items-center gap-2"> | |
| 252 | + <span className="truncate font-mono text-[11.5px]">{o.outcome}</span> | |
| 253 | + <div className="h-1.5 overflow-hidden rounded-full bg-panel-2"><div className={`h-full ${o.outcome === "error" || o.outcome === "parse_error" || o.outcome === "rate_limited" ? "bg-danger" : o.outcome === "event" ? "bg-signal" : o.outcome === "not_modified" ? "bg-info" : "bg-low"}`} style={{ width: `${oTotal ? (o.n / oTotal) * 100 : 0}%` }} /></div> | |
| 254 | + <span className="text-right font-mono tabular">{fmtInt(o.n)}</span> | |
| 255 | + </li> | |
| 256 | + ))} | |
| 257 | + </ul> | |
| 258 | + ) : ( | |
| 259 | + <Empty>No runs in the last hour.</Empty> | |
| 260 | + )} | |
| 261 | + </Panel> | |
| 262 | + <Panel title="Storage"> | |
| 263 | + <dl className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1 text-[12.5px]"> | |
| 264 | + <Row k="Database" v={d.storage.db_size} /> | |
| 265 | + <Row k="Snapshots" v={`${fmtInt(d.storage.snapshots)} · ${fmtInt(d.storage.snapshots_with_raw)} with raw`} /> | |
| 266 | + <Row k="Raw (uncompressed)" v={fmtBytes(raw)} /> | |
| 267 | + <Row k="Blob store" v={d.storage.blobs ? `${fmtInt(d.storage.blobs.files)} files · ${fmtBytes(blobBytes)}` : "—"} /> | |
| 268 | + <Row k="Compression" v={raw && blobBytes ? `${(raw / blobBytes).toFixed(1)}×` : "—"} /> | |
| 269 | + <Row k="Changes / events" v={`${fmtInt(d.storage.changes)} / ${fmtInt(d.storage.events)}`} /> | |
| 270 | + <Row k="Runs" v={fmtInt(d.storage.runs)} /> | |
| 271 | + </dl> | |
| 272 | + </Panel> | |
| 273 | + </div> | |
| 274 | + | |
| 275 | + <Panel title="Throughput · 24 h · hourly" dense> | |
| 276 | + {tp.length ? ( | |
| 277 | + <div className="grid grid-cols-1 gap-3 p-3 sm:grid-cols-3"> | |
| 278 | + <Spark label="Checks" values={tp.map((r) => r.checks)} tone="info" /> | |
| 279 | + <Spark label="Events" values={tp.map((r) => r.events)} tone="signal" /> | |
| 280 | + <Spark label="Errors" values={tp.map((r) => r.errors)} tone="hot" /> | |
| 281 | + </div> | |
| 282 | + ) : ( | |
| 283 | + <Empty>No sensor runs recorded in the last 24 h.</Empty> | |
| 284 | + )} | |
| 285 | + {tp.length > 0 && ( | |
| 286 | + <div className="max-h-56 overflow-auto border-t border-line"> | |
| 287 | + <Table head={["Hour (UTC)", "Checks", "304", "Errors", "Events", "Avg ms"]}> | |
| 288 | + {[...tp].reverse().map((r) => ( | |
| 289 | + <tr key={r.t}> | |
| 290 | + <Td mono>{r.t.slice(5, 16).replace("T", " ")}</Td> | |
| 291 | + <Td mono>{fmtInt(r.checks)}</Td> | |
| 292 | + <Td mono className="text-fg-subtle">{fmtInt(r.not_modified)}</Td> | |
| 293 | + <Td mono className={r.errors > 0 ? "text-danger" : ""}>{fmtInt(r.errors)}</Td> | |
| 294 | + <Td mono className="text-signal">{fmtInt(r.events)}</Td> | |
| 295 | + <Td mono>{fmtMs(r.avg_ms)}</Td> | |
| 296 | + </tr> | |
| 297 | + ))} | |
| 298 | + </Table> | |
| 299 | + </div> | |
| 300 | + )} | |
| 301 | + </Panel> | |
| 302 | + | |
| 303 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-3"> | |
| 304 | + <Panel title="Sensors by status / health" dense> | |
| 305 | + {byStatus.length ? ( | |
| 306 | + <Table head={["Status", "Sensors", "Health"]}> | |
| 307 | + {byStatus.map(([status, v]) => ( | |
| 308 | + <tr key={status}> | |
| 309 | + <Td mono>{status}</Td> | |
| 310 | + <Td mono>{fmtInt(v.n)}</Td> | |
| 311 | + <Td> | |
| 312 | + <div className="flex flex-wrap gap-1"> | |
| 313 | + {Object.entries(v.health).sort((a, b) => b[1] - a[1]).map(([h, n]) => ( | |
| 314 | + <span key={h} className="inline-flex items-center gap-1"><HealthPill health={h} /><span className="font-mono text-[11px] text-fg-subtle tabular">{fmtInt(n)}</span></span> | |
| 315 | + ))} | |
| 316 | + </div> | |
| 317 | + </Td> | |
| 318 | + </tr> | |
| 319 | + ))} | |
| 320 | + </Table> | |
| 321 | + ) : ( | |
| 322 | + <Empty /> | |
| 323 | + )} | |
| 324 | + </Panel> | |
| 325 | + <Panel title="LLM · 24 h" dense> | |
| 326 | + {d.llm_24h.length ? ( | |
| 327 | + <Table head={["Model", "Calls", "In", "Out", "Fail"]}> | |
| 328 | + {d.llm_24h.map((m) => ( | |
| 329 | + <tr key={m.model}> | |
| 330 | + <Td mono className="whitespace-nowrap">{m.model}</Td> | |
| 331 | + <Td mono>{fmtInt(m.calls)}</Td> | |
| 332 | + <Td mono>{fmtInt(Number(m.input_tokens))}</Td> | |
| 333 | + <Td mono>{fmtInt(Number(m.output_tokens))}</Td> | |
| 334 | + <Td mono className={m.failures ? "text-danger" : "text-fg-subtle"}>{fmtInt(m.failures)}</Td> | |
| 335 | + </tr> | |
| 336 | + ))} | |
| 337 | + </Table> | |
| 338 | + ) : ( | |
| 339 | + <Empty>No LLM calls in 24 h (heuristics only).</Empty> | |
| 340 | + )} | |
| 341 | + </Panel> | |
| 342 | + <Panel title="Live & cache" dense> | |
| 343 | + <dl className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1 px-3 py-2 text-[12.5px]"> | |
| 344 | + <Row k="WS clients" v={fmtInt(d.live.clients)} /> | |
| 345 | + <Row k="Published since start" v={fmtInt(d.live.published)} /> | |
| 346 | + <Row k="Cache entries" v={fmtInt(d.cache.entries)} /> | |
| 347 | + </dl> | |
| 348 | + <div className="max-h-32 overflow-auto border-t border-line px-3 py-2"> | |
| 349 | + <div className="flex flex-wrap gap-1">{d.cache.keys.slice(0, 40).map((k) => <Chip key={k} className="font-mono">{k}</Chip>)}</div> | |
| 350 | + </div> | |
| 351 | + </Panel> | |
| 352 | + </div> | |
| 353 | + | |
| 354 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-2"> | |
| 355 | + <Panel title="Top failing domains · 6 h" dense> | |
| 356 | + {d.top_failing_domains.length ? ( | |
| 357 | + <Table head={["Host", "Failures", "Sensors", "Last error", "Last"]}> | |
| 358 | + {d.top_failing_domains.map((f) => ( | |
| 359 | + <tr key={f.host}> | |
| 360 | + <Td mono><Link href={`/domain/${f.host}`} className="hover:underline">{f.host}</Link></Td> | |
| 361 | + <Td mono className="text-danger">{fmtInt(f.failures)}</Td> | |
| 362 | + <Td mono>{fmtInt(f.sensors)}</Td> | |
| 363 | + <Td className="max-w-[18rem] truncate font-mono text-[11px] text-fg-muted" ><span title={f.last_error ?? ""}>{f.last_error ?? "—"}</span></Td> | |
| 364 | + <Td mono className="whitespace-nowrap text-fg-subtle">{relTime(f.last_at)}</Td> | |
| 365 | + </tr> | |
| 366 | + ))} | |
| 367 | + </Table> | |
| 368 | + ) : ( | |
| 369 | + <Empty>No failures in the last 6 h.</Empty> | |
| 370 | + )} | |
| 371 | + </Panel> | |
| 372 | + <Panel title="Slowest sensors" dense> | |
| 373 | + {d.slowest_sensors.length ? ( | |
| 374 | + <Table head={["Sensor", "Source", "Connector", "Avg latency", "Runs"]}> | |
| 375 | + {d.slowest_sensors.map((s) => ( | |
| 376 | + <tr key={s.id}> | |
| 377 | + <Td><Link href={`/sensor/${s.id}`} className="hover:underline">{s.name}</Link><div className="max-w-[14rem] truncate font-mono text-[10.5px] text-fg-subtle">{s.url}</div></Td> | |
| 378 | + <Td><Link href={`/source/${s.source_id}`} className="hover:underline">{s.source_id}</Link></Td> | |
| 379 | + <Td mono>{s.connector}</Td> | |
| 380 | + <Td mono className={s.avg_latency_ms > 10_000 ? "text-warn" : ""}>{fmtMs(s.avg_latency_ms)}</Td> | |
| 381 | + <Td mono>{fmtInt(s.total_runs)}</Td> | |
| 382 | + </tr> | |
| 383 | + ))} | |
| 384 | + </Table> | |
| 385 | + ) : ( | |
| 386 | + <Empty /> | |
| 387 | + )} | |
| 388 | + </Panel> | |
| 389 | + </div> | |
| 390 | + | |
| 391 | + <Panel title="Recent errors · 1 h" dense> | |
| 392 | + {d.recent_errors.length ? ( | |
| 393 | + <Table head={["At", "Sensor", "Source", "Connector", "HTTP", "Outcome", "Error"]}> | |
| 394 | + {d.recent_errors.map((r, i) => ( | |
| 395 | + <tr key={`${r.sensor_id}-${i}`}> | |
| 396 | + <Td mono className="whitespace-nowrap text-fg-subtle">{utcTime(r.started_at)}</Td> | |
| 397 | + <Td><Link href={`/sensor/${r.sensor_id}`} className="font-mono text-[11.5px] hover:underline">{r.sensor_id}</Link></Td> | |
| 398 | + <Td><Link href={`/source/${r.source_id}`} className="hover:underline">{r.source_id}</Link></Td> | |
| 399 | + <Td mono>{r.connector}</Td> | |
| 400 | + <Td mono>{r.http_status ?? "—"}</Td> | |
| 401 | + <Td mono className="text-danger">{r.outcome}</Td> | |
| 402 | + <Td className="max-w-[24rem] truncate font-mono text-[11px] text-fg-muted"><span title={r.error ?? ""}>{r.error ?? "—"}</span></Td> | |
| 403 | + </tr> | |
| 404 | + ))} | |
| 405 | + </Table> | |
| 406 | + ) : ( | |
| 407 | + <Empty>No errors in the last hour.</Empty> | |
| 408 | + )} | |
| 409 | + </Panel> | |
| 410 | + </> | |
| 411 | + ); | |
| 412 | +} | |
| 413 | + | |
| 414 | +function Row({ k, v, tone = "" }: { k: string; v: ReactNode; tone?: string }) { | |
| 415 | + return ( | |
| 416 | + <> | |
| 417 | + <dt className="text-fg-subtle">{k}</dt> | |
| 418 | + <dd className={`min-w-0 break-words text-right font-mono text-[12px] tabular ${tone}`}>{v}</dd> | |
| 419 | + </> | |
| 420 | + ); | |
| 421 | +} | |
| 422 | + | |
| 423 | +function Spark({ label, values, tone }: { label: string; values: number[]; tone: "signal" | "hot" | "info" }) { | |
| 424 | + const last = values[values.length - 1] ?? 0; | |
| 425 | + const max = Math.max(...values, 0); | |
| 426 | + return ( | |
| 427 | + <div className="rounded-md border border-line bg-panel-2 p-2"> | |
| 428 | + <div className="flex items-baseline justify-between"> | |
| 429 | + <span className="label">{label}</span> | |
| 430 | + <span className="font-mono text-[11px] text-fg-subtle tabular">last h {fmtInt(last)} · max {fmtInt(max)}</span> | |
| 431 | + </div> | |
| 432 | + <div className="mt-1 w-full overflow-hidden"> | |
| 433 | + <Sparkline values={values} width={360} height={40} tone={tone} /> | |
| 434 | + </div> | |
| 435 | + </div> | |
| 436 | + ); | |
| 437 | +} | |
| 438 | + | |
| 439 | +// --------------------------------------------------------------------------------------- | |
| 440 | +// Tools | |
| 441 | +// --------------------------------------------------------------------------------------- | |
| 442 | + | |
| 443 | +type Json = Record<string, unknown>; | |
| 444 | + | |
| 445 | +function Result({ r }: { r: { ok: boolean; body: Json | null; error?: string } | null }) { | |
| 446 | + if (!r) return null; | |
| 447 | + return ( | |
| 448 | + <div className={`rounded-md border px-2 py-1.5 text-[12px] ${r.ok ? "border-ok/40 bg-ok/10" : "border-danger/40 bg-danger/10"}`}> | |
| 449 | + {r.error && <div className="font-mono font-semibold text-danger">{r.error}</div>} | |
| 450 | + {r.body && <pre className="max-h-64 overflow-auto whitespace-pre-wrap break-words font-mono text-[11px] leading-4 text-fg-muted">{JSON.stringify(r.body, null, 2)}</pre>} | |
| 451 | + </div> | |
| 452 | + ); | |
| 453 | +} | |
| 454 | + | |
| 455 | +function useAction() { | |
| 456 | + const [busy, setBusy] = useState(false); | |
| 457 | + const [res, setRes] = useState<{ ok: boolean; body: Json | null; error?: string } | null>(null); | |
| 458 | + const run = async (fn: () => Promise<Json>): Promise<Json | null> => { | |
| 459 | + setBusy(true); | |
| 460 | + setRes(null); | |
| 461 | + try { | |
| 462 | + const body = await fn(); | |
| 463 | + setRes({ ok: true, body }); | |
| 464 | + return body; | |
| 465 | + } catch (e) { | |
| 466 | + setRes({ ok: false, body: e instanceof AdminError ? e.body : null, error: gateMessage(e) }); | |
| 467 | + return null; | |
| 468 | + } finally { | |
| 469 | + setBusy(false); | |
| 470 | + } | |
| 471 | + }; | |
| 472 | + return { busy, res, run }; | |
| 473 | +} | |
| 474 | + | |
| 475 | +const input = "h-8 min-w-0 rounded-md border border-line bg-panel px-2 font-mono text-[12px]"; | |
| 476 | +const btn = "inline-flex h-8 items-center gap-1 rounded-md border border-line bg-panel-2 px-2.5 text-[12px] hover:border-line-strong disabled:opacity-50"; | |
| 477 | + | |
| 478 | +function Tools({ connectors, onDone }: { connectors: Ops["connectors"]; onDone: () => void }) { | |
| 479 | + return ( | |
| 480 | + <> | |
| 481 | + <h2 className="label mt-2">Tools</h2> | |
| 482 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-2"> | |
| 483 | + <TestConnector connectors={connectors} /> | |
| 484 | + <SensorActions onDone={onDone} /> | |
| 485 | + <SourceAndBulk onDone={onDone} /> | |
| 486 | + <Import onDone={onDone} /> | |
| 487 | + </div> | |
| 488 | + </> | |
| 489 | + ); | |
| 490 | +} | |
| 491 | + | |
| 492 | +function TestConnector({ connectors }: { connectors: Ops["connectors"] }) { | |
| 493 | + const [f, setF] = useState({ url: "", sensor_id: "", connector: "http", type: "HTML", config: "{}" }); | |
| 494 | + const { busy, res, run } = useAction(); | |
| 495 | + const preview = res?.body?.normalized as Json | undefined; | |
| 496 | + const meta = res?.body?.meta as Json | undefined; | |
| 497 | + const submit = (): void => { | |
| 498 | + let config: Json = {}; | |
| 499 | + try { | |
| 500 | + config = f.config.trim() ? (JSON.parse(f.config) as Json) : {}; | |
| 501 | + } catch { | |
| 502 | + void run(() => Promise.reject(new Error("config is not valid JSON"))); | |
| 503 | + return; | |
| 504 | + } | |
| 505 | + void run(() => adminFetch<Json>("/api/v1/admin/sensors/test", { method: "POST", body: JSON.stringify({ ...(f.sensor_id.trim() ? { sensor_id: f.sensor_id.trim() } : { url: f.url.trim() }), connector: f.connector, type: f.type, config }) })); | |
| 506 | + }; | |
| 507 | + return ( | |
| 508 | + <Panel title="Test connector · dry run, nothing persisted"> | |
| 509 | + <form | |
| 510 | + className="flex flex-col gap-2 text-[12.5px]" | |
| 511 | + onSubmit={(e) => { | |
| 512 | + e.preventDefault(); | |
| 513 | + submit(); | |
| 514 | + }} | |
| 515 | + > | |
| 516 | + <input value={f.url} onChange={(e) => setF({ ...f, url: e.target.value })} placeholder="https://example.com/feed.xml" aria-label="URL" className={input} /> | |
| 517 | + <input value={f.sensor_id} onChange={(e) => setF({ ...f, sensor_id: e.target.value })} placeholder="… or an existing sensor id (overrides URL)" aria-label="Sensor id" className={input} /> | |
| 518 | + <div className="grid grid-cols-2 gap-2"> | |
| 519 | + <select value={f.connector} onChange={(e) => setF({ ...f, connector: e.target.value })} aria-label="Connector" className={input}> | |
| 520 | + {(connectors.length ? connectors.map((c) => c.key) : ["http"]).map((k) => ( | |
| 521 | + <option key={k} value={k}>{k}</option> | |
| 522 | + ))} | |
| 523 | + </select> | |
| 524 | + <select value={f.type} onChange={(e) => setF({ ...f, type: e.target.value })} aria-label="Sensor type" className={input}> | |
| 525 | + {SENSOR_TYPES.map((t) => ( | |
| 526 | + <option key={t} value={t}>{t}</option> | |
| 527 | + ))} | |
| 528 | + </select> | |
| 529 | + </div> | |
| 530 | + <textarea value={f.config} onChange={(e) => setF({ ...f, config: e.target.value })} rows={2} aria-label="Config JSON" placeholder='{"selector": "main"}' className="rounded-md border border-line bg-panel px-2 py-1 font-mono text-[12px]" /> | |
| 531 | + <button type="submit" disabled={busy || (!f.url.trim() && !f.sensor_id.trim())} className={btn}><Play className="size-3.5" /> {busy ? "Fetching…" : "Run test"}</button> | |
| 532 | + </form> | |
| 533 | + {res && ( | |
| 534 | + <div className="mt-2 flex flex-col gap-2 text-[12px]"> | |
| 535 | + <div className="flex flex-wrap items-center gap-2"> | |
| 536 | + <Chip tone={res.ok && res.body?.ok ? "ok" : "danger"} className="font-mono">{res.ok && res.body?.ok ? "OK" : `FAILED${typeof res.body?.stage === "string" ? ` · ${res.body.stage}` : ""}`}</Chip> | |
| 537 | + {meta && <span className="font-mono text-[11px] text-fg-subtle tabular">HTTP {String(meta.status ?? "—")} · {String(meta.contentType ?? "—")} · {fmtBytes(Number(meta.contentLength ?? 0))} · {String(res.body?.fetch_ms ?? "—")} ms · {String(meta.redirects ?? 0)} redirects</span>} | |
| 538 | + </div> | |
| 539 | + {preview && ( | |
| 540 | + <dl className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-0.5 rounded-md border border-line bg-panel-2 p-2 text-[12px]"> | |
| 541 | + <Row k="Mode" v={String(preview.mode ?? "—")} /> | |
| 542 | + <Row k="Title" v={String(preview.title ?? "—")} /> | |
| 543 | + <Row k="Items" v={preview.items === null || preview.items === undefined ? "—" : String(preview.items)} /> | |
| 544 | + <Row k="Extraction confidence" v={typeof preview.extractionConfidence === "number" ? `${Math.round(preview.extractionConfidence * 100)}%` : "—"} /> | |
| 545 | + <Row k="Canonical hash" v={String(preview.canonicalHash ?? "").slice(0, 16)} /> | |
| 546 | + <Row k="Published" v={String(preview.publishedAt ?? "—")} /> | |
| 547 | + </dl> | |
| 548 | + )} | |
| 549 | + {preview && typeof preview.text_preview === "string" && <pre className="max-h-40 overflow-auto whitespace-pre-wrap break-words rounded-md border border-line bg-panel-2 p-2 font-mono text-[11px] leading-4 text-fg-muted">{preview.text_preview}</pre>} | |
| 550 | + {preview && typeof preview.json_preview === "string" && <pre className="max-h-40 overflow-auto whitespace-pre-wrap break-words rounded-md border border-line bg-panel-2 p-2 font-mono text-[11px] leading-4 text-fg-muted">{preview.json_preview}</pre>} | |
| 551 | + {preview && Array.isArray(preview.sample_items) && preview.sample_items.length > 0 && <pre className="max-h-40 overflow-auto whitespace-pre-wrap break-words rounded-md border border-line bg-panel-2 p-2 font-mono text-[11px] leading-4 text-fg-muted">{JSON.stringify(preview.sample_items, null, 1)}</pre>} | |
| 552 | + {(!res.ok || !res.body?.ok) && <Result r={{ ok: false, body: (res.body?.error as Json | undefined) ?? res.body, error: res.error }} />} | |
| 553 | + </div> | |
| 554 | + )} | |
| 555 | + </Panel> | |
| 556 | + ); | |
| 557 | +} | |
| 558 | + | |
| 559 | +function SensorActions({ onDone }: { onDone: () => void }) { | |
| 560 | + const [id, setId] = useState(""); | |
| 561 | + const [tier, setTier] = useState(""); | |
| 562 | + const [priority, setPriority] = useState(""); | |
| 563 | + const [interval, setInterval_] = useState(""); | |
| 564 | + const { busy, res, run } = useAction(); | |
| 565 | + const act = (a: "run-now" | "enable" | "disable"): void => { | |
| 566 | + void run(() => adminFetch<Json>(`/api/v1/admin/sensors/${encodeURIComponent(id.trim())}/${a}`, { method: "POST" })).then(onDone); | |
| 567 | + }; | |
| 568 | + const patch = (): void => { | |
| 569 | + const body: Json = {}; | |
| 570 | + if (tier) body.tier = tier; | |
| 571 | + if (priority !== "") body.priority = Number(priority); | |
| 572 | + if (interval !== "") body.base_interval_seconds = Number(interval); | |
| 573 | + void run(() => adminFetch<Json>(`/api/v1/admin/sensors/${encodeURIComponent(id.trim())}`, { method: "PATCH", body: JSON.stringify(body) })).then(onDone); | |
| 574 | + }; | |
| 575 | + return ( | |
| 576 | + <Panel title="Sensor actions"> | |
| 577 | + <div className="flex flex-col gap-2 text-[12.5px]"> | |
| 578 | + <input value={id} onChange={(e) => setId(e.target.value)} placeholder="sensor id (e.g. openai_pricing)" aria-label="Sensor id" className={input} /> | |
| 579 | + <div className="flex flex-wrap gap-1"> | |
| 580 | + <button type="button" disabled={busy || !id.trim()} onClick={() => act("run-now")} className={btn}><Play className="size-3.5" /> run now</button> | |
| 581 | + <button type="button" disabled={busy || !id.trim()} onClick={() => act("enable")} className={btn}>enable</button> | |
| 582 | + <button type="button" disabled={busy || !id.trim()} onClick={() => act("disable")} className={`${btn} text-danger`}>disable</button> | |
| 583 | + <Link href={id.trim() ? `/sensor/${encodeURIComponent(id.trim())}` : "#"} className={`${btn} ${id.trim() ? "" : "pointer-events-none opacity-50"}`}>open →</Link> | |
| 584 | + </div> | |
| 585 | + <div className="grid grid-cols-3 gap-2"> | |
| 586 | + <select value={tier} onChange={(e) => setTier(e.target.value)} aria-label="Tier" className={input}> | |
| 587 | + <option value="">tier …</option> | |
| 588 | + {TIERS.map((t) => ( | |
| 589 | + <option key={t} value={t}>tier {t}</option> | |
| 590 | + ))} | |
| 591 | + </select> | |
| 592 | + <select value={priority} onChange={(e) => setPriority(e.target.value)} aria-label="Priority" className={input}> | |
| 593 | + <option value="">priority …</option> | |
| 594 | + {[0, 1, 2, 3].map((p) => ( | |
| 595 | + <option key={p} value={p}>P{p}</option> | |
| 596 | + ))} | |
| 597 | + </select> | |
| 598 | + <input value={interval} onChange={(e) => setInterval_(e.target.value)} inputMode="numeric" placeholder="interval s" aria-label="Base interval seconds" className={input} /> | |
| 599 | + </div> | |
| 600 | + <button type="button" disabled={busy || !id.trim() || (!tier && priority === "" && interval === "")} onClick={patch} className={btn}>apply patch</button> | |
| 601 | + <Result r={res} /> | |
| 602 | + </div> | |
| 603 | + </Panel> | |
| 604 | + ); | |
| 605 | +} | |
| 606 | + | |
| 607 | +function SourceAndBulk({ onDone }: { onDone: () => void }) { | |
| 608 | + const [sid, setSid] = useState(""); | |
| 609 | + const [ids, setIds] = useState(""); | |
| 610 | + const [action, setAction] = useState<"enable" | "disable" | "run-now">("run-now"); | |
| 611 | + const src = useAction(); | |
| 612 | + const bulk = useAction(); | |
| 613 | + const list = ids.split(/[\s,]+/).map((x) => x.trim()).filter(Boolean); | |
| 614 | + return ( | |
| 615 | + <Panel title="Source actions & bulk"> | |
| 616 | + <div className="flex flex-col gap-2 text-[12.5px]"> | |
| 617 | + <input value={sid} onChange={(e) => setSid(e.target.value)} placeholder="source id (e.g. openai)" aria-label="Source id" className={input} /> | |
| 618 | + <div className="flex flex-wrap gap-1"> | |
| 619 | + {(["run-now", "enable", "disable"] as const).map((a) => ( | |
| 620 | + <button key={a} type="button" disabled={src.busy || !sid.trim()} onClick={() => void src.run(() => adminFetch<Json>(`/api/v1/admin/sources/${encodeURIComponent(sid.trim())}/${a}`, { method: "POST" })).then(onDone)} className={`${btn} ${a === "disable" ? "text-danger" : ""}`}> | |
| 621 | + {a === "run-now" && <Play className="size-3.5" />} {a} | |
| 622 | + </button> | |
| 623 | + ))} | |
| 624 | + <Link href={sid.trim() ? `/source/${encodeURIComponent(sid.trim())}` : "#"} className={`${btn} ${sid.trim() ? "" : "pointer-events-none opacity-50"}`}>open →</Link> | |
| 625 | + </div> | |
| 626 | + <Result r={src.res} /> | |
| 627 | + <div className="label mt-2">Bulk sensor action</div> | |
| 628 | + <textarea value={ids} onChange={(e) => setIds(e.target.value)} rows={3} placeholder="sensor ids, one per line or comma-separated" aria-label="Sensor ids" className="rounded-md border border-line bg-panel px-2 py-1 font-mono text-[12px]" /> | |
| 629 | + <div className="flex flex-wrap items-center gap-1"> | |
| 630 | + <select value={action} onChange={(e) => setAction(e.target.value as typeof action)} aria-label="Bulk action" className={input}> | |
| 631 | + <option value="run-now">run now</option> | |
| 632 | + <option value="enable">enable</option> | |
| 633 | + <option value="disable">disable</option> | |
| 634 | + </select> | |
| 635 | + <button type="button" disabled={bulk.busy || !list.length} onClick={() => void bulk.run(() => adminFetch<Json>("/api/v1/admin/sensors/bulk", { method: "POST", body: JSON.stringify({ ids: list, action }) })).then(onDone)} className={btn}> | |
| 636 | + apply to <span className="font-mono tabular">{list.length}</span> | |
| 637 | + </button> | |
| 638 | + </div> | |
| 639 | + <Result r={bulk.res} /> | |
| 640 | + </div> | |
| 641 | + </Panel> | |
| 642 | + ); | |
| 643 | +} | |
| 644 | + | |
| 645 | +interface ImportReport { | |
| 646 | + dry_run: boolean; | |
| 647 | + valid?: number; | |
| 648 | + invalid: number; | |
| 649 | + imported_sources?: number; | |
| 650 | + extended_sources?: number; | |
| 651 | + sensors?: number; | |
| 652 | + report: { id?: string; ok: boolean; extend: boolean; issues?: string[]; sensors?: number }[]; | |
| 653 | +} | |
| 654 | + | |
| 655 | +function Import({ onDone }: { onDone: () => void }) { | |
| 656 | + const [doc, setDoc] = useState(""); | |
| 657 | + const [report, setReport] = useState<ImportReport | null>(null); | |
| 658 | + const [checkedDoc, setCheckedDoc] = useState<string | null>(null); | |
| 659 | + const { busy, res, run } = useAction(); | |
| 660 | + const isJson = /^\s*[[{]/.test(doc); | |
| 661 | + const send = async (dry: boolean): Promise<void> => { | |
| 662 | + const body = await run(() => adminFetch<Json>(`/api/v1/admin/sources/import${dry ? "?dry_run=1" : ""}`, { method: "POST", headers: { "content-type": isJson ? "application/json" : "text/yaml" }, body: doc })); | |
| 663 | + if (body) { | |
| 664 | + setReport(body as unknown as ImportReport); | |
| 665 | + setCheckedDoc(dry ? doc : null); | |
| 666 | + if (!dry) onDone(); | |
| 667 | + } else setReport(null); | |
| 668 | + }; | |
| 669 | + const canImport = report?.dry_run === true && checkedDoc === doc && (report.valid ?? 0) > 0; | |
| 670 | + return ( | |
| 671 | + <Panel title="Import sources · YAML or JSON"> | |
| 672 | + <div className="flex flex-col gap-2 text-[12.5px]"> | |
| 673 | + <textarea value={doc} onChange={(e) => setDoc(e.target.value)} rows={7} spellCheck={false} aria-label="Registry document" placeholder={"sources:\n - id: example\n name: Example\n domain: example.com\n tier: C\n categories: [web]\n sensors:\n - name: home\n type: HTML\n url: https://example.com/"} className="rounded-md border border-line bg-panel px-2 py-1 font-mono text-[12px] leading-4" /> | |
| 674 | + <div className="flex flex-wrap items-center gap-1"> | |
| 675 | + <Chip className="font-mono">{isJson ? "application/json" : "text/yaml"}</Chip> | |
| 676 | + <button type="button" disabled={busy || !doc.trim()} onClick={() => void send(true)} className={btn}>dry run</button> | |
| 677 | + <button type="button" disabled={busy || !canImport} onClick={() => void send(false)} className={`${btn} ${canImport ? "border-signal/50 text-signal" : ""}`} title={canImport ? "Write the validated document" : "Run a successful dry run first"}>import</button> | |
| 678 | + <span className="text-[11px] text-fg-subtle">Same schema as <code>config/sources.d/*.yaml</code>; <code>extend: true</code> adds to an existing source.</span> | |
| 679 | + </div> | |
| 680 | + {res && !res.ok && <Result r={res} />} | |
| 681 | + {report && ( | |
| 682 | + <div> | |
| 683 | + <div className="mb-1 flex flex-wrap gap-1 text-[11.5px]"> | |
| 684 | + <Chip tone={report.dry_run ? "info" : "ok"} className="font-mono">{report.dry_run ? "DRY RUN" : "IMPORTED"}</Chip> | |
| 685 | + {report.valid !== undefined && <Chip tone="ok">{report.valid} valid</Chip>} | |
| 686 | + {report.invalid > 0 && <Chip tone="danger">{report.invalid} invalid</Chip>} | |
| 687 | + {report.imported_sources !== undefined && <Chip>{report.imported_sources} sources · {report.extended_sources ?? 0} extended · {report.sensors ?? 0} sensors</Chip>} | |
| 688 | + </div> | |
| 689 | + <div className="overflow-hidden rounded-md border border-line"> | |
| 690 | + <Table head={["Source", "Mode", "Result", "Sensors / issues"]}> | |
| 691 | + {report.report.map((r, i) => ( | |
| 692 | + <tr key={`${r.id ?? "?"}-${i}`}> | |
| 693 | + <Td mono>{r.id ?? <span className="text-fg-subtle">(no id)</span>}</Td> | |
| 694 | + <Td mono className="text-fg-subtle">{r.extend ? "extend" : "new"}</Td> | |
| 695 | + <Td>{r.ok ? <Chip tone="ok">OK</Chip> : <Chip tone="danger">FAIL</Chip>}</Td> | |
| 696 | + <Td className="text-[12px]">{r.ok ? <span className="font-mono">{r.sensors ?? 0} sensors</span> : <ul className="list-disc pl-4 text-danger">{(r.issues ?? []).map((x, j) => <li key={j} className="font-mono text-[11px]">{x}</li>)}</ul>}</Td> | |
| 697 | + </tr> | |
| 698 | + ))} | |
| 699 | + </Table> | |
| 700 | + </div> | |
| 701 | + </div> | |
| 702 | + )} | |
| 703 | + </div> | |
| 704 | + </Panel> | |
| 705 | + ); | |
| 706 | +} | |
added
apps/web/src/app/ops/page.tsx
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +import type { Metadata } from "next"; | |
| 2 | +import { PageHeader } from "@/components/ui"; | |
| 3 | +import { Ops } from "./ops"; | |
| 4 | + | |
| 5 | +export const metadata: Metadata = { title: "Operations", description: "Internal WebSensor operations dashboard.", robots: { index: false, follow: false } }; | |
| 6 | + | |
| 7 | +export default function OpsPage() { | |
| 8 | + return ( | |
| 9 | + <> | |
| 10 | + <PageHeader compact kicker="Internal · requires the admin token" title="Operations" description="Engine heartbeat, queue, throughput, storage, LLM usage, failing domains and connector tools. Refreshes every 15 s. Not linked from public navigation." /> | |
| 11 | + <Ops /> | |
| 12 | + </> | |
| 13 | + ); | |
| 14 | +} | |
modified
apps/web/src/app/page.tsx
+35 −10
@@ -1,28 +1,53 @@ | ||
| 1 | +import Link from "next/link"; | |
| 1 | 2 | import { LiveFeed } from "@/components/live-feed"; |
| 2 | −import { ClustersPanel, TrendingPanel } from "@/components/rail"; | |
| 3 | −import { StatsStrip } from "@/components/stats-strip"; | |
| 3 | +import { LiveStrip } from "@/components/live-strip"; | |
| 4 | +import { AnomalyRail, BreakingRail, ClustersPanel, SilentRail, TrendingPanel } from "@/components/rail"; | |
| 5 | +import { EventRow } from "@/components/event-row"; | |
| 6 | +import { Empty, Panel } from "@/components/ui"; | |
| 4 | 7 | import { api } from "@/lib/api"; |
| 8 | +import { agoIso } from "@/lib/format"; | |
| 5 | 9 | |
| 6 | 10 | export const dynamic = "force-dynamic"; |
| 7 | 11 | |
| 8 | −export default async function LivePage() { | |
| 9 | − const [stats, events, trending, clusters] = await Promise.all([api.stats(), api.events({ limit: 60 }), api.trending(24, 10), api.clusters(20, 48)]); | |
| 12 | +/** | |
| 13 | + * Homepage (spec §64, §94): hero line · live system strip · live feed · right rail (breaking now, | |
| 14 | + * trending, anomalous) · lower area (clusters, silent changes, infrastructure pulse). One coherent | |
| 15 | + * data surface; the product sells the product. | |
| 16 | + */ | |
| 17 | +export default async function HomePage() { | |
| 18 | + const [stats, events, trending, clusters, breaking, explore, infra] = await Promise.all([ | |
| 19 | + api.stats(), | |
| 20 | + api.events({ limit: 60 }), | |
| 21 | + api.trending(24, 10), | |
| 22 | + api.clusters(30, 48), | |
| 23 | + api.breakingDesk(), | |
| 24 | + api.explore(), | |
| 25 | + api.events({ limit: 8, group: "reliability", after: agoIso(12 * 3600e3) }), | |
| 26 | + ]); | |
| 10 | 27 | return ( |
| 11 | 28 | <> |
| 12 | 29 | <div className="mb-3 flex flex-wrap items-baseline justify-between gap-2"> |
| 13 | − <h1 className="text-[15px] font-semibold tracking-tight"> | |
| 30 | + <h1 className="text-[16px] font-semibold tracking-tight"> | |
| 14 | 31 | The Web is changing. <span className="text-fg-muted">We are watching.</span> |
| 15 | 32 | </h1> |
| 16 | − <p className="text-[12px] text-fg-subtle">Official sources · conditional fetches · immutable evidence · importance-scored events</p> | |
| 33 | + <p className="text-[12px] text-fg-subtle">WebSensor continuously watches the public web for meaningful changes, preserving evidence and connecting signals into real-world events.</p> | |
| 17 | 34 | </div> |
| 18 | − <StatsStrip stats={stats} /> | |
| 19 | − <div className="grid gap-4 lg:grid-cols-[1fr_320px]"> | |
| 35 | + <LiveStrip initial={stats} /> | |
| 36 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-[1fr_340px] [&>*]:min-w-0"> | |
| 20 | 37 | <LiveFeed initial={events.items} initialCursor={events.nextCursor} /> |
| 21 | − <aside className="hidden flex-col gap-4 lg:flex"> | |
| 38 | + <aside className="flex flex-col gap-4"> | |
| 39 | + <BreakingRail items={breaking?.breaking_now ?? []} developing={breaking?.developing ?? []} /> | |
| 22 | 40 | <TrendingPanel items={trending.items} /> |
| 23 | − <ClustersPanel items={clusters.items} /> | |
| 41 | + <AnomalyRail items={explore?.unusual_activity ?? []} /> | |
| 24 | 42 | </aside> |
| 25 | 43 | </div> |
| 44 | + <div className="mt-4 grid grid-cols-1 gap-4 lg:grid-cols-3 [&>*]:min-w-0"> | |
| 45 | + <ClustersPanel items={clusters.items} title="Event clusters · 48 h" /> | |
| 46 | + <SilentRail items={explore?.silent_changes ?? []} /> | |
| 47 | + <Panel title="Infrastructure pulse · 12 h" dense action={<Link href="/live?group=reliability" className="text-[11px] text-fg-subtle hover:text-fg">live →</Link>}> | |
| 48 | + {infra.items.length ? infra.items.slice(0, 6).map((e) => <EventRow key={e.id} ev={e} />) : <Empty>No outage, incident or maintenance signal in the last 12 h.</Empty>} | |
| 49 | + </Panel> | |
| 50 | + </div> | |
| 26 | 51 | </> |
| 27 | 52 | ); |
| 28 | 53 | } |
added
apps/web/src/app/pulse/loading.tsx
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3 flex flex-col gap-2"> | |
| 7 | + <Skeleton className="h-3 w-20" /> | |
| 8 | + <Skeleton className="h-6 w-24" /> | |
| 9 | + <Skeleton className="h-3 w-full max-w-2xl" /> | |
| 10 | + </div> | |
| 11 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-3 lg:grid-cols-6 lg:divide-y-0"> | |
| 12 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 13 | + <div key={i} className="flex flex-col gap-1.5 px-3 py-2"> | |
| 14 | + <Skeleton className="h-2.5 w-20" /> | |
| 15 | + <Skeleton className="h-5 w-12" /> | |
| 16 | + <Skeleton className="h-2.5 w-16" /> | |
| 17 | + </div> | |
| 18 | + ))} | |
| 19 | + </div> | |
| 20 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-[minmax(0,1fr)_340px]"> | |
| 21 | + <div className="flex flex-col gap-4"> | |
| 22 | + <div className="panel p-3"> | |
| 23 | + <Skeleton className="mb-3 h-2.5 w-56" /> | |
| 24 | + <div className="grid grid-cols-1 gap-4 sm:grid-cols-2"> | |
| 25 | + <Skeleton className="h-16 w-full" /> | |
| 26 | + <Skeleton className="h-16 w-full" /> | |
| 27 | + </div> | |
| 28 | + </div> | |
| 29 | + <div className="grid grid-cols-1 gap-4 md:grid-cols-2"> | |
| 30 | + {Array.from({ length: 4 }, (_, i) => ( | |
| 31 | + <div key={i} className="panel"> | |
| 32 | + <div className="border-b border-line px-3 py-2"> | |
| 33 | + <Skeleton className="h-2.5 w-32" /> | |
| 34 | + </div> | |
| 35 | + <SkeletonRows rows={4} /> | |
| 36 | + </div> | |
| 37 | + ))} | |
| 38 | + </div> | |
| 39 | + </div> | |
| 40 | + <aside className="flex flex-col gap-4"> | |
| 41 | + <SkeletonPanel lines={8} /> | |
| 42 | + <SkeletonPanel lines={6} /> | |
| 43 | + <SkeletonPanel lines={6} /> | |
| 44 | + </aside> | |
| 45 | + </div> | |
| 46 | + </> | |
| 47 | + ); | |
| 48 | +} | |
added
apps/web/src/app/pulse/page.tsx
+115 −0
@@ -0,0 +1,115 @@ | ||
| 1 | +import type { Metadata } from "next"; | |
| 2 | +import Link from "next/link"; | |
| 3 | +import { EventRow } from "@/components/event-row"; | |
| 4 | +import { BreakingRail } from "@/components/rail"; | |
| 5 | +import { Empty, PageHeader, Panel, Sparkline, Stat } from "@/components/ui"; | |
| 6 | +import { api } from "@/lib/api"; | |
| 7 | +import { fmtInt, GROUP_LABELS, relTime, CHANNELS } from "@/lib/format"; | |
| 8 | + | |
| 9 | +export const dynamic = "force-dynamic"; | |
| 10 | +export const metadata: Metadata = { title: "Pulse", description: "What is changing on the Internet right now: global activity, breaking events, fastest-rising entities, cyber incidents, AI developments, market and government events, infrastructure incidents, silent changes." }; | |
| 11 | + | |
| 12 | +/** Real-time summary of the Internet (spec §39). */ | |
| 13 | +export default async function PulsePage() { | |
| 14 | + const p = await api.pulse(); | |
| 15 | + if (!p) return <Empty>Pulse is warming up.</Empty>; | |
| 16 | + const activity = p.activity ?? []; | |
| 17 | + const evSeries = activity.map((a) => Number(a.events)); | |
| 18 | + const chSeries = activity.map((a) => Number(a.changes)); | |
| 19 | + const groups = Object.entries(p.by_group_24h ?? {}).sort((a, b) => b[1] - a[1]); | |
| 20 | + const maxG = Math.max(1, ...groups.map(([, n]) => n)); | |
| 21 | + const deskLabel = (d: string): string => CHANNELS.find((c) => c.key === d)?.label ?? d; | |
| 22 | + return ( | |
| 23 | + <> | |
| 24 | + <PageHeader compact kicker="Right now" title="Pulse" description="A live summary of what is changing on the public web — generated from real observations, refreshed every ten seconds." /> | |
| 25 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-3 lg:grid-cols-6 lg:divide-y-0"> | |
| 26 | + <Stat label="Events · 1 h" value={fmtInt(p.totals?.events_1h)} hint="meaningful signals" tone="signal" /> | |
| 27 | + <Stat label="Raw changes · 1 h" value={fmtInt(p.totals?.changes_1h)} hint="before noise filtering" /> | |
| 28 | + <Stat label="Checks · 5 min" value={fmtInt(p.totals?.checks_5m)} hint={`${Math.round(Number(p.totals?.checks_5m ?? 0) / 5)}/min`} /> | |
| 29 | + <Stat label="Active sources · 24 h" value={fmtInt(p.totals?.active_sources_24h)} /> | |
| 30 | + <Stat label="Countries · 24 h" value={fmtInt(p.totals?.active_countries_24h)} /> | |
| 31 | + <Stat label="Breaking / developing" value={`${p.breaking.filter((c) => c.state === "breaking").length} / ${p.breaking.filter((c) => c.state === "developing").length}`} tone="hot" /> | |
| 32 | + </div> | |
| 33 | + <div className="grid grid-cols-1 gap-4 xl:grid-cols-[1fr_340px] [&>*]:min-w-0"> | |
| 34 | + <div className="flex flex-col gap-4"> | |
| 35 | + <Panel title="Global activity · last 6 h · 15-minute buckets"> | |
| 36 | + <div className="grid grid-cols-1 gap-4 sm:grid-cols-2 [&>*]:min-w-0"> | |
| 37 | + <div> | |
| 38 | + <div className="label mb-1">Events</div> | |
| 39 | + <Sparkline values={evSeries} width={420} height={56} tone="signal" responsive /> | |
| 40 | + </div> | |
| 41 | + <div> | |
| 42 | + <div className="label mb-1">Raw changes</div> | |
| 43 | + <Sparkline values={chSeries} width={420} height={56} tone="muted" responsive /> | |
| 44 | + </div> | |
| 45 | + </div> | |
| 46 | + <p className="mt-2 text-[11px] text-fg-subtle">The gap between raw changes and events is the noise WebSensor filtered out: timestamps, navigation churn, cookie banners, advertising, cosmetic edits.</p> | |
| 47 | + </Panel> | |
| 48 | + <div className="grid gap-4 md:grid-cols-2"> | |
| 49 | + {(p.desks ?? []).filter((d) => d.items.length).map((d) => ( | |
| 50 | + <Panel key={d.desk} title={`${deskLabel(d.desk)} · 12 h`} dense action={<Link href={`/category/${d.desk}`} className="text-[11px] text-fg-subtle hover:text-fg">desk →</Link>}> | |
| 51 | + {d.items.map((e) => ( | |
| 52 | + <EventRow key={e.id} ev={e} /> | |
| 53 | + ))} | |
| 54 | + </Panel> | |
| 55 | + ))} | |
| 56 | + </div> | |
| 57 | + <Panel title="Infrastructure incidents · 6 h" dense action={<Link href="/live?group=reliability" className="text-[11px] text-fg-subtle hover:text-fg">live →</Link>}> | |
| 58 | + {p.infrastructure.length ? p.infrastructure.map((e) => <EventRow key={e.id} ev={e} />) : <Empty>No outage, incident or maintenance signal in the last 6 h.</Empty>} | |
| 59 | + </Panel> | |
| 60 | + <Panel title="Silent changes · 24 h" dense action={<Link href="/silent" className="text-[11px] text-fg-subtle hover:text-fg">all →</Link>}> | |
| 61 | + {p.silent_changes.length ? p.silent_changes.map((e) => <EventRow key={e.id} ev={e} />) : <Empty>No silent change in the last 24 h.</Empty>} | |
| 62 | + </Panel> | |
| 63 | + </div> | |
| 64 | + <aside className="flex flex-col gap-4"> | |
| 65 | + <BreakingRail items={p.breaking.filter((c) => c.state === "breaking")} developing={p.breaking.filter((c) => c.state === "developing")} /> | |
| 66 | + <Panel title="Fastest-rising entities · 3 h" dense> | |
| 67 | + {p.rising_entities.length ? ( | |
| 68 | + <ol className="divide-y divide-line"> | |
| 69 | + {p.rising_entities.map((r, i) => ( | |
| 70 | + <li key={r.id} className="grid grid-cols-[1.25rem_1fr_auto] items-center gap-2 px-3 py-1.5 text-[13px]"> | |
| 71 | + <span className="font-mono text-[11px] text-fg-subtle tabular">{i + 1}</span> | |
| 72 | + <div className="min-w-0"> | |
| 73 | + <Link href={`/entity/${r.id}`} className="block truncate font-medium hover:underline">{r.name}</Link> | |
| 74 | + <div className="text-[11px] text-fg-subtle">{r.events_3h} signals in 3 h · {r.events_prev_24h} in the prior 24 h</div> | |
| 75 | + </div> | |
| 76 | + <span className="font-mono text-[12px] font-semibold text-signal tabular">×{r.acceleration}</span> | |
| 77 | + </li> | |
| 78 | + ))} | |
| 79 | + </ol> | |
| 80 | + ) : ( | |
| 81 | + <Empty>No entity is accelerating right now.</Empty> | |
| 82 | + )} | |
| 83 | + </Panel> | |
| 84 | + <Panel title="Anomalous sources · 2 h" dense> | |
| 85 | + {p.anomalies.length ? ( | |
| 86 | + <ul className="divide-y divide-line"> | |
| 87 | + {p.anomalies.map((s) => ( | |
| 88 | + <li key={s.id} className="flex items-center gap-2 px-3 py-1.5 text-[13px]"> | |
| 89 | + <Link href={`/source/${s.id}`} className="min-w-0 flex-1 truncate hover:underline">{s.name}</Link> | |
| 90 | + <span className="text-[11px] text-fg-subtle">{s.changes_2h} vs {s.baseline_per_day}/d</span> | |
| 91 | + <span className={`font-mono text-[12px] font-semibold tabular ${s.activity_score >= 70 ? "text-hot" : "text-high"}`}>{s.pct_vs_baseline !== null && s.pct_vs_baseline !== undefined ? `+${Math.min(9999, s.pct_vs_baseline)}%` : Math.round(s.activity_score)}</span> | |
| 92 | + </li> | |
| 93 | + ))} | |
| 94 | + </ul> | |
| 95 | + ) : ( | |
| 96 | + <Empty>All sources within baseline.</Empty> | |
| 97 | + )} | |
| 98 | + </Panel> | |
| 99 | + <Panel title="Events by group · 24 h"> | |
| 100 | + <ul className="space-y-1.5"> | |
| 101 | + {groups.map(([g, n]) => ( | |
| 102 | + <li key={g} className="grid grid-cols-[8rem_1fr_3rem] items-center gap-2 text-[12.5px]"> | |
| 103 | + <Link href={`/live?group=${g}`} className="truncate hover:underline">{GROUP_LABELS[g] ?? g}</Link> | |
| 104 | + <div className="h-1 w-full overflow-hidden rounded-full bg-panel-2"><div className="h-full rounded-full bg-info" style={{ width: `${(n / maxG) * 100}%` }} /></div> | |
| 105 | + <span className="text-right font-mono text-fg-subtle tabular">{fmtInt(n)}</span> | |
| 106 | + </li> | |
| 107 | + ))} | |
| 108 | + </ul> | |
| 109 | + </Panel> | |
| 110 | + <p className="text-[11px] text-fg-subtle">Generated {relTime(p.generated_at)} · all values come from stored observations; nothing is estimated.</p> | |
| 111 | + </aside> | |
| 112 | + </div> | |
| 113 | + </> | |
| 114 | + ); | |
| 115 | +} | |
added
apps/web/src/app/radar/loading.tsx
+30 −0
@@ -0,0 +1,30 @@ | ||
| 1 | +import { Skeleton, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3 flex flex-col gap-2"> | |
| 7 | + <Skeleton className="h-4 w-32" /> | |
| 8 | + <Skeleton className="h-6 w-24" /> | |
| 9 | + <Skeleton className="h-3 w-full max-w-3xl" /> | |
| 10 | + <Skeleton className="h-3 w-2/3 max-w-2xl" /> | |
| 11 | + </div> | |
| 12 | + <div className="grid grid-cols-1 gap-4 lg:grid-cols-2 xl:grid-cols-3"> | |
| 13 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 14 | + <div key={i} className="panel"> | |
| 15 | + <div className="border-b border-line px-3 py-2"> | |
| 16 | + <Skeleton className="h-2.5 w-40" /> | |
| 17 | + </div> | |
| 18 | + <SkeletonRows rows={4} /> | |
| 19 | + </div> | |
| 20 | + ))} | |
| 21 | + </div> | |
| 22 | + <div className="mt-4 panel"> | |
| 23 | + <div className="border-b border-line px-3 py-2"> | |
| 24 | + <Skeleton className="h-2.5 w-64" /> | |
| 25 | + </div> | |
| 26 | + <SkeletonRows rows={5} /> | |
| 27 | + </div> | |
| 28 | + </> | |
| 29 | + ); | |
| 30 | +} | |
added
apps/web/src/app/radar/page.tsx
+127 −0
@@ -0,0 +1,127 @@ | ||
| 1 | +import type { Metadata } from "next"; | |
| 2 | +import Link from "next/link"; | |
| 3 | +import { EventRow } from "@/components/event-row"; | |
| 4 | +import { Chip, Empty, PageHeader, Panel, Score, StateLabelText } from "@/components/ui"; | |
| 5 | +import { api } from "@/lib/api"; | |
| 6 | +import { relTime, typeLabel } from "@/lib/format"; | |
| 7 | + | |
| 8 | +export const dynamic = "force-dynamic"; | |
| 9 | +export const metadata: Metadata = { title: "Radar", description: "Weak signals that are not yet breaking: unusual source activity, clusters of silent changes, documentation and repository bursts, fresh status changes." }; | |
| 10 | + | |
| 11 | +/** Radar (spec §101): indicators, not facts. */ | |
| 12 | +export default async function RadarPage() { | |
| 13 | + const r = await api.radar(); | |
| 14 | + if (!r) return <Empty>Radar is warming up.</Empty>; | |
| 15 | + return ( | |
| 16 | + <> | |
| 17 | + <PageHeader compact kicker={<Chip tone="warn">Indicators, not facts</Chip>} title="Radar" description="What might become important next. Each item is a pattern in raw observations — a source far above its baseline, several silent changes around one entity, a burst of documentation or repository edits, a fresh status change — that has not (yet) produced a high-importance event." /> | |
| 18 | + <div className="grid grid-cols-1 gap-4 lg:grid-cols-2 xl:grid-cols-3 [&>*]:min-w-0"> | |
| 19 | + <Panel title="Unusual source activity · 3 h" dense> | |
| 20 | + {r.unusual_source_activity.length ? ( | |
| 21 | + <ul className="divide-y divide-line"> | |
| 22 | + {r.unusual_source_activity.map((s) => ( | |
| 23 | + <li key={s.id} className="px-3 py-2 text-[13px]"> | |
| 24 | + <div className="flex items-center gap-2"> | |
| 25 | + <Link href={`/source/${s.id}`} className="min-w-0 flex-1 truncate font-medium hover:underline">{s.name}</Link> | |
| 26 | + <span className="font-mono text-[12px] font-semibold text-high tabular">×{s.ratio}</span> | |
| 27 | + </div> | |
| 28 | + <div className="text-[11px] text-fg-subtle">{s.changes_3h} raw changes in 3 h · baseline {s.baseline_per_day}/day · no high-importance event yet</div> | |
| 29 | + <div className="mt-1 flex flex-wrap gap-1">{(s.categories ?? []).slice(0, 3).map((c) => <Chip key={c} href={`/category/${c}`}>{c}</Chip>)}</div> | |
| 30 | + </li> | |
| 31 | + ))} | |
| 32 | + </ul> | |
| 33 | + ) : ( | |
| 34 | + <Empty>No source is quietly bursting above its baseline.</Empty> | |
| 35 | + )} | |
| 36 | + </Panel> | |
| 37 | + <Panel title="Silent-change clusters · 24 h" dense> | |
| 38 | + {r.silent_clusters.length ? ( | |
| 39 | + <ul className="divide-y divide-line"> | |
| 40 | + {r.silent_clusters.map((e) => ( | |
| 41 | + <li key={e.id} className="px-3 py-2 text-[13px]"> | |
| 42 | + <div className="flex items-center gap-2"> | |
| 43 | + <Link href={`/entity/${e.id}?tab=silent`} className="min-w-0 flex-1 truncate font-medium hover:underline">{e.name}</Link> | |
| 44 | + <span className="font-mono text-[12px] font-semibold text-silent tabular">{e.silent_24h} silent</span> | |
| 45 | + </div> | |
| 46 | + <div className="mt-1 flex flex-wrap gap-1">{e.types.slice(0, 4).map((t) => <Chip key={t} tone="silent">{typeLabel(t)}</Chip>)}<span className="text-[11px] text-fg-subtle">{relTime(e.last_at)}</span></div> | |
| 47 | + </li> | |
| 48 | + ))} | |
| 49 | + </ul> | |
| 50 | + ) : ( | |
| 51 | + <Empty>No entity accumulated several silent changes.</Empty> | |
| 52 | + )} | |
| 53 | + </Panel> | |
| 54 | + <Panel title="Developing clusters" dense> | |
| 55 | + {r.developing.length ? ( | |
| 56 | + <ul className="divide-y divide-line"> | |
| 57 | + {r.developing.map((c) => ( | |
| 58 | + <li key={c.id} className="flex items-start gap-2 px-3 py-2 text-[13px]"> | |
| 59 | + <Score value={c.max_importance} size="sm" /> | |
| 60 | + <div className="min-w-0"> | |
| 61 | + <Link href={`/cluster/${c.slug ?? c.id}`} className="line-clamp-2 font-medium hover:underline">{c.title}</Link> | |
| 62 | + <div className="flex flex-wrap gap-x-2 text-[11px] text-fg-subtle"><StateLabelText state={c.state} /><span>{c.event_count} signals · {c.source_count} sources · velocity {Math.round(c.velocity ?? 0)}</span><span>{relTime(c.last_at)}</span></div> | |
| 63 | + </div> | |
| 64 | + </li> | |
| 65 | + ))} | |
| 66 | + </ul> | |
| 67 | + ) : ( | |
| 68 | + <Empty>No developing cluster.</Empty> | |
| 69 | + )} | |
| 70 | + </Panel> | |
| 71 | + <Panel title="Documentation & API bursts · 6 h" dense> | |
| 72 | + {r.documentation_bursts.length ? ( | |
| 73 | + <ul className="divide-y divide-line"> | |
| 74 | + {r.documentation_bursts.map((s) => ( | |
| 75 | + <li key={s.id} className="px-3 py-2 text-[13px]"> | |
| 76 | + <div className="flex items-center gap-2"> | |
| 77 | + <Link href={`/source/${s.id}`} className="min-w-0 flex-1 truncate font-medium hover:underline">{s.name}</Link> | |
| 78 | + <span className="font-mono text-[12px] tabular">{s.doc_changes_6h} edits</span> | |
| 79 | + </div> | |
| 80 | + <div className="mt-1 flex flex-wrap gap-1">{s.types.map((t) => <Chip key={t}>{typeLabel(t)}</Chip>)}<span className="text-[11px] text-fg-subtle">{relTime(s.last_at)}</span></div> | |
| 81 | + </li> | |
| 82 | + ))} | |
| 83 | + </ul> | |
| 84 | + ) : ( | |
| 85 | + <Empty>No documentation burst. A launch often starts with several doc/API edits in a short window.</Empty> | |
| 86 | + )} | |
| 87 | + </Panel> | |
| 88 | + <Panel title="Repository bursts · 6 h" dense> | |
| 89 | + {r.repository_bursts.length ? ( | |
| 90 | + <ul className="divide-y divide-line"> | |
| 91 | + {r.repository_bursts.map((s) => ( | |
| 92 | + <li key={s.id} className="flex items-center gap-2 px-3 py-2 text-[13px]"> | |
| 93 | + <Link href={`/source/${s.id}`} className="min-w-0 flex-1 truncate font-medium hover:underline">{s.name}</Link> | |
| 94 | + <span className="font-mono text-[12px] tabular">{s.repo_events_6h} releases/commits</span> | |
| 95 | + <span className="text-[11px] text-fg-subtle">{relTime(s.last_at)}</span> | |
| 96 | + </li> | |
| 97 | + ))} | |
| 98 | + </ul> | |
| 99 | + ) : ( | |
| 100 | + <Empty>No repository burst.</Empty> | |
| 101 | + )} | |
| 102 | + </Panel> | |
| 103 | + <Panel title="New coverage lighting up · 24 h" dense> | |
| 104 | + {r.new_coverage.length ? ( | |
| 105 | + <ul className="divide-y divide-line"> | |
| 106 | + {r.new_coverage.map((s) => ( | |
| 107 | + <li key={s.id} className="flex items-center gap-2 px-3 py-2 text-[13px]"> | |
| 108 | + <div className="min-w-0 flex-1"> | |
| 109 | + <Link href={`/sensor/${s.id}`} className="block truncate font-medium hover:underline">{s.name}</Link> | |
| 110 | + <div className="truncate text-[11px] text-fg-subtle">{s.source_name} · first event {relTime(s.first_event_at)}</div> | |
| 111 | + </div> | |
| 112 | + <span className="font-mono text-[12px] tabular">{s.events}</span> | |
| 113 | + </li> | |
| 114 | + ))} | |
| 115 | + </ul> | |
| 116 | + ) : ( | |
| 117 | + <Empty>No sensor produced its first event today.</Empty> | |
| 118 | + )} | |
| 119 | + </Panel> | |
| 120 | + </div> | |
| 121 | + <Panel title="Fresh status changes below the breaking bar · 3 h" dense className="mt-4"> | |
| 122 | + {r.status_changes.length ? r.status_changes.map((e) => <EventRow key={e.id} ev={e} />) : <Empty>No fresh status-page change.</Empty>} | |
| 123 | + </Panel> | |
| 124 | + <p className="mt-3 text-[11px] text-fg-subtle">{r.disclaimer} Generated {relTime(r.generated_at)}.</p> | |
| 125 | + </> | |
| 126 | + ); | |
| 127 | +} | |
modified
apps/web/src/app/search/page.tsx
+134 −56
@@ -1,74 +1,152 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | 2 | import type { Metadata } from "next"; |
| 3 | 3 | import { EventRow } from "@/components/event-row"; |
| 4 | −import { Chip, Empty, PageHeader, Panel } from "@/components/ui"; | |
| 4 | +import { Badge, Chip, Empty, Flag, Kbd, PageHeader, Panel, Score, StateBadge, TierBadge } from "@/components/ui"; | |
| 5 | 5 | import { api } from "@/lib/api"; |
| 6 | −import { relTime } from "@/lib/format"; | |
| 6 | +import { relTime, typeLabel, utcDateTime } from "@/lib/format"; | |
| 7 | 7 | |
| 8 | 8 | export const dynamic = "force-dynamic"; |
| 9 | 9 | export const metadata: Metadata = { title: "Search", robots: { index: false } }; |
| 10 | 10 | |
| 11 | +const SYNTAX: { key: string; example: string; help: string }[] = [ | |
| 12 | + { key: "entity", example: "entity:openai", help: "events linked to an entity" }, | |
| 13 | + { key: "type", example: "type:pricing_change", help: "event type" }, | |
| 14 | + { key: "after", example: "after:7d", help: "detected after (7d, 24h, 2026-09-01)" }, | |
| 15 | + { key: "silent", example: "silent:true", help: "silent changes only" }, | |
| 16 | + { key: "importance", example: "importance:>70", help: "minimum importance" }, | |
| 17 | + { key: "country", example: "country:CA", help: "source country" }, | |
| 18 | + { key: "first_party", example: "first_party:true", help: "first-party evidence only" }, | |
| 19 | +]; | |
| 20 | + | |
| 21 | +function fmtFilter(k: string, v: unknown): string { | |
| 22 | + if (v === null || v === undefined) return ""; | |
| 23 | + if (typeof v === "string" && /^\d{4}-\d{2}-\d{2}T/.test(v)) return utcDateTime(v); | |
| 24 | + if (Array.isArray(v)) return v.join(", "); | |
| 25 | + return String(v); | |
| 26 | +} | |
| 27 | + | |
| 28 | +/** Search (spec §30): full-text + structured filters, results across events, clusters, entities, sources and URLs. */ | |
| 11 | 29 | export default async function SearchPage({ searchParams }: { searchParams: Promise<{ q?: string }> }) { |
| 12 | 30 | const { q = "" } = await searchParams; |
| 13 | − const r = q.trim().length >= 2 ? await api.search(q.trim()) : null; | |
| 31 | + const query = q.trim(); | |
| 32 | + const r = query.length >= 2 ? await api.search(query) : null; | |
| 33 | + const filters = Object.entries(r?.parsed?.filters ?? {}).filter(([, v]) => v !== undefined && v !== null && v !== ""); | |
| 34 | + const text = r?.parsed?.text?.trim() ?? ""; | |
| 35 | + const clusters = r?.clusters ?? []; | |
| 36 | + const total = r ? r.events.length + clusters.length + r.entities.length + r.sources.length + r.urls.length : 0; | |
| 14 | 37 | return ( |
| 15 | 38 | <> |
| 16 | − <PageHeader kicker="Search everything" title={q ? <span>Results for “{q}”</span> : "Search"} description="Events, entities, sources, URLs and domains. Full-text search over titles, summaries and keywords." /> | |
| 17 | − <form action="/search" className="mb-4 flex gap-2"> | |
| 18 | − <input name="q" defaultValue={q} autoFocus placeholder="OpenAI, CVE-2026, pricing, status.…" className="h-9 w-full max-w-xl rounded-md border border-line bg-panel px-3 text-[13.5px] placeholder:text-fg-subtle" /> | |
| 19 | − <button type="submit" className="h-9 rounded-md border border-line bg-panel-2 px-3 text-[13px]">Search</button> | |
| 39 | + <PageHeader compact kicker="Search everything" title={query ? <span>Results for “{query}”</span> : "Search"} description="Events, clusters, entities, sources and URLs. Full-text over titles, summaries and keywords; structured filters narrow the event set." /> | |
| 40 | + <form action="/search" className="mb-2 flex gap-2"> | |
| 41 | + <input name="q" defaultValue={q} autoFocus placeholder="OpenAI, CVE-2026, pricing, status.…" aria-label="Search" className="h-9 w-full min-w-0 max-w-2xl rounded-md border border-line bg-panel px-3 font-mono text-[13px] placeholder:text-fg-subtle" /> | |
| 42 | + <button type="submit" className="h-9 rounded-md border border-line bg-panel-2 px-3 text-[13px] hover:border-line-strong">Search</button> | |
| 20 | 43 | </form> |
| 44 | + <div className="mb-4 flex flex-wrap items-center gap-x-2 gap-y-1 text-[11.5px] text-fg-subtle"> | |
| 45 | + <span>Syntax</span> | |
| 46 | + {SYNTAX.map((s) => ( | |
| 47 | + <Link key={s.key} href={`/search?q=${encodeURIComponent(`${query ? query + " " : ""}${s.example}`)}`} title={s.help} className="rounded-sm border border-line bg-panel-2 px-1.5 py-px font-mono text-[10.5px] text-fg-muted hover:text-fg">{s.example}</Link> | |
| 48 | + ))} | |
| 49 | + <span className="hidden items-center gap-1 sm:inline-flex"><Kbd>⌘K</Kbd> anywhere</span> | |
| 50 | + </div> | |
| 51 | + | |
| 21 | 52 | {r && ( |
| 22 | − <div className="grid gap-4 lg:grid-cols-[1fr_320px]"> | |
| 23 | − <Panel title={`Events · ${r.events.length}`} dense> | |
| 24 | − {r.events.length ? r.events.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>No events match.</Empty>} | |
| 25 | − </Panel> | |
| 26 | − <aside className="flex flex-col gap-4"> | |
| 27 | − <Panel title={`Entities · ${r.entities.length}`} dense> | |
| 28 | − {r.entities.length ? ( | |
| 29 | − <ul className="divide-y divide-line"> | |
| 30 | − {r.entities.map((e) => ( | |
| 31 | − <li key={e.id} className="flex items-center justify-between px-3 py-1.5 text-[13px]"> | |
| 32 | − <Link href={`/company/${e.id}`} className="hover:underline">{e.name}</Link> | |
| 33 | − <Chip>{e.type}</Chip> | |
| 34 | − </li> | |
| 35 | − ))} | |
| 36 | − </ul> | |
| 37 | − ) : ( | |
| 38 | − <Empty>—</Empty> | |
| 39 | − )} | |
| 40 | − </Panel> | |
| 41 | − <Panel title={`Sources · ${r.sources.length}`} dense> | |
| 42 | − {r.sources.length ? ( | |
| 43 | − <ul className="divide-y divide-line"> | |
| 44 | − {r.sources.map((s) => ( | |
| 45 | − <li key={s.id} className="flex items-center justify-between px-3 py-1.5 text-[13px]"> | |
| 46 | − <Link href={`/source/${s.id}`} className="hover:underline">{s.name}</Link> | |
| 47 | − <Link href={`/domain/${s.domain}`} className="font-mono text-[11px] text-fg-subtle hover:underline">{s.domain}</Link> | |
| 48 | − </li> | |
| 49 | − ))} | |
| 50 | − </ul> | |
| 51 | − ) : ( | |
| 52 | − <Empty>—</Empty> | |
| 53 | − )} | |
| 54 | − </Panel> | |
| 55 | − <Panel title={`URLs · ${r.urls.length}`} dense> | |
| 56 | − {r.urls.length ? ( | |
| 57 | − <ul className="divide-y divide-line"> | |
| 58 | − {r.urls.map((u) => ( | |
| 59 | − <li key={u.url} className="px-3 py-1.5 text-[12px]"> | |
| 60 | − <Link href={`/url?u=${encodeURIComponent(u.url)}`} className="block truncate font-mono hover:underline">{u.url}</Link> | |
| 61 | − <span className="text-[11px] text-fg-subtle">{u.change_count} changes · {relTime(u.last_seen_at)}</span> | |
| 62 | − </li> | |
| 63 | − ))} | |
| 64 | − </ul> | |
| 65 | − ) : ( | |
| 66 | − <Empty>—</Empty> | |
| 67 | − )} | |
| 68 | − </Panel> | |
| 69 | − </aside> | |
| 70 | − </div> | |
| 53 | + <> | |
| 54 | + <div className="mb-4 flex flex-wrap items-center gap-1.5 text-[12px]"> | |
| 55 | + <span className="text-fg-subtle">{total} result{total === 1 ? "" : "s"}</span> | |
| 56 | + {text && <Chip tone="default" title="Full-text term">“{text}”</Chip>} | |
| 57 | + {filters.map(([k, v]) => ( | |
| 58 | + <Chip key={k} tone="info" className="font-mono" title={`Filter ${k}`}>{k}: {fmtFilter(k, v)}</Chip> | |
| 59 | + ))} | |
| 60 | + {filters.length > 0 && <Link href={`/search?q=${encodeURIComponent(text)}`} className="text-[11px] text-fg-subtle hover:text-fg">clear filters</Link>} | |
| 61 | + </div> | |
| 62 | + <div className="grid gap-4 xl:grid-cols-[1fr_360px]"> | |
| 63 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 64 | + <Panel title={`Events · ${r.events.length}`} dense action={r.events.length ? <Link href={`/live?q=${encodeURIComponent(text || query)}`} className="text-[11px] text-fg-subtle hover:text-fg">live →</Link> : undefined}> | |
| 65 | + {r.events.length ? r.events.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>No event matches{filters.length ? " these filters" : ""}. Try a broader term, remove a filter, or extend the window with <span className="font-mono">after:30d</span>.</Empty>} | |
| 66 | + </Panel> | |
| 67 | + <Panel title={`Clusters · ${clusters.length}`} dense> | |
| 68 | + {clusters.length ? ( | |
| 69 | + <ul className="divide-y divide-line"> | |
| 70 | + {clusters.map((c) => ( | |
| 71 | + <li key={c.id} className="grid grid-cols-[auto_1fr] items-start gap-x-3 px-3 py-2 text-[13px]"> | |
| 72 | + <Score value={c.max_importance} size="sm" /> | |
| 73 | + <div className="min-w-0"> | |
| 74 | + <div className="flex flex-wrap items-center gap-x-2 text-[11px] text-fg-subtle"> | |
| 75 | + <StateBadge state={c.state} /> | |
| 76 | + <span>{c.event_count} signal{c.event_count === 1 ? "" : "s"} · {c.source_count} source{c.source_count === 1 ? "" : "s"}</span> | |
| 77 | + <span className="ml-auto font-mono tabular">{relTime(c.last_at)}</span> | |
| 78 | + </div> | |
| 79 | + <Link href={`/cluster/${c.slug ?? c.id}`} className="mt-0.5 block font-medium leading-snug hover:underline">{c.title}</Link> | |
| 80 | + </div> | |
| 81 | + </li> | |
| 82 | + ))} | |
| 83 | + </ul> | |
| 84 | + ) : ( | |
| 85 | + <Empty>No cluster matches. Clusters group related signals across sources.</Empty> | |
| 86 | + )} | |
| 87 | + </Panel> | |
| 88 | + </div> | |
| 89 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 90 | + <Panel title={`Entities · ${r.entities.length}`} dense> | |
| 91 | + {r.entities.length ? ( | |
| 92 | + <ul className="divide-y divide-line"> | |
| 93 | + {r.entities.map((e) => ( | |
| 94 | + <li key={e.id} className="flex items-center gap-2 px-3 py-1.5 text-[13px]"> | |
| 95 | + <div className="min-w-0 flex-1"> | |
| 96 | + <Link href={`/entity/${e.id}`} className="block truncate font-medium hover:underline">{e.name}</Link> | |
| 97 | + <div className="truncate font-mono text-[10.5px] text-fg-subtle">{e.domain ?? e.id}{e.event_count ? ` · ${e.event_count} ev` : ""}</div> | |
| 98 | + </div> | |
| 99 | + <Chip>{e.type.replace(/_/g, " ")}</Chip> | |
| 100 | + </li> | |
| 101 | + ))} | |
| 102 | + </ul> | |
| 103 | + ) : ( | |
| 104 | + <Empty>No entity matches.</Empty> | |
| 105 | + )} | |
| 106 | + </Panel> | |
| 107 | + <Panel title={`Sources · ${r.sources.length}`} dense> | |
| 108 | + {r.sources.length ? ( | |
| 109 | + <ul className="divide-y divide-line"> | |
| 110 | + {r.sources.map((s) => ( | |
| 111 | + <li key={s.id} className="flex items-center gap-2 px-3 py-1.5 text-[13px]"> | |
| 112 | + <TierBadge tier={s.tier} /> | |
| 113 | + <div className="min-w-0 flex-1"> | |
| 114 | + <Link href={`/source/${s.id}`} className="block truncate font-medium hover:underline">{s.name}</Link> | |
| 115 | + <Link href={`/domain/${s.domain}`} className="block truncate font-mono text-[10.5px] text-fg-subtle hover:underline">{s.domain}</Link> | |
| 116 | + </div> | |
| 117 | + {s.country && <Flag code={s.country} className="text-[12px]" />} | |
| 118 | + {s.first_party === false ? <Badge kind="external" compact /> : <Badge kind="first-party" compact />} | |
| 119 | + </li> | |
| 120 | + ))} | |
| 121 | + </ul> | |
| 122 | + ) : ( | |
| 123 | + <Empty>No source matches.</Empty> | |
| 124 | + )} | |
| 125 | + </Panel> | |
| 126 | + <Panel title={`URLs · ${r.urls.length}`} dense> | |
| 127 | + {r.urls.length ? ( | |
| 128 | + <ul className="divide-y divide-line"> | |
| 129 | + {r.urls.map((u) => ( | |
| 130 | + <li key={u.url} className="px-3 py-1.5 text-[12px]"> | |
| 131 | + <Link href={`/url?u=${encodeURIComponent(u.url)}`} className="block truncate font-mono hover:underline" title={u.url}>{u.url}</Link> | |
| 132 | + <span className="flex items-center gap-2 text-[11px] text-fg-subtle"> | |
| 133 | + <span>{u.change_count} change{u.change_count === 1 ? "" : "s"}</span> | |
| 134 | + <span>·</span> | |
| 135 | + <span>{relTime(u.last_seen_at)}</span> | |
| 136 | + {u.status && u.status !== "active" && <Chip tone="warn" className="font-mono">{typeLabel(u.status)}</Chip>} | |
| 137 | + </span> | |
| 138 | + </li> | |
| 139 | + ))} | |
| 140 | + </ul> | |
| 141 | + ) : ( | |
| 142 | + <Empty>No monitored URL matches.</Empty> | |
| 143 | + )} | |
| 144 | + </Panel> | |
| 145 | + </aside> | |
| 146 | + </div> | |
| 147 | + </> | |
| 71 | 148 | )} |
| 149 | + {!r && query.length > 0 && query.length < 2 && <Empty>Type at least two characters.</Empty>} | |
| 72 | 150 | </> |
| 73 | 151 | ); |
| 74 | 152 | } |
added
apps/web/src/app/sensor/[id]/loading.tsx
+41 −0
@@ -0,0 +1,41 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3"> | |
| 7 | + <Skeleton className="mb-1 h-2.5 w-64" /> | |
| 8 | + <Skeleton className="h-5 w-40" /> | |
| 9 | + <Skeleton className="mt-2 h-3 w-full max-w-lg" /> | |
| 10 | + </div> | |
| 11 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-line sm:grid-cols-4 lg:grid-cols-8"> | |
| 12 | + {Array.from({ length: 8 }, (_, i) => ( | |
| 13 | + <div key={i} className="flex flex-col gap-1.5 px-3 py-2"> | |
| 14 | + <Skeleton className="h-2.5 w-16" /> | |
| 15 | + <Skeleton className="h-5 w-12" /> | |
| 16 | + <Skeleton className="h-2.5 w-20" /> | |
| 17 | + </div> | |
| 18 | + ))} | |
| 19 | + </div> | |
| 20 | + <div className="grid gap-4 xl:grid-cols-[1fr_380px]"> | |
| 21 | + <div className="flex flex-col gap-4"> | |
| 22 | + <div className="panel"> | |
| 23 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-56" /></div> | |
| 24 | + <SkeletonRows rows={10} /> | |
| 25 | + </div> | |
| 26 | + <div className="panel"> | |
| 27 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-24" /></div> | |
| 28 | + <SkeletonRows rows={5} /> | |
| 29 | + </div> | |
| 30 | + </div> | |
| 31 | + <div className="flex flex-col gap-4"> | |
| 32 | + <div className="panel"> | |
| 33 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-20" /></div> | |
| 34 | + <SkeletonRows rows={6} /> | |
| 35 | + </div> | |
| 36 | + <SkeletonPanel lines={5} /> | |
| 37 | + </div> | |
| 38 | + </div> | |
| 39 | + </> | |
| 40 | + ); | |
| 41 | +} | |
modified
apps/web/src/app/sensor/[id]/page.tsx
+165 −73
@@ -1,9 +1,12 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | 2 | import type { Metadata } from "next"; |
| 3 | 3 | import { notFound } from "next/navigation"; |
| 4 | +import { PRIORITY_LABELS } from "@websensor/core/client"; | |
| 5 | +import { EventRow } from "@/components/event-row"; | |
| 6 | +import { FieldChangeInline } from "@/components/field-changes"; | |
| 4 | 7 | import { Chip, Empty, ExtLink, HealthPill, PageHeader, Panel, Stat, Table, Td, TierBadge } from "@/components/ui"; |
| 5 | −import { api } from "@/lib/api"; | |
| 6 | −import { fmtBytes, fmtDuration, fmtInt, fmtMs, relTime, shortHash, untilTime, utcDateTime } from "@/lib/format"; | |
| 8 | +import { api, type SnapshotRow } from "@/lib/api"; | |
| 9 | +import { CLASS_LABELS, dayHeader, fmtBytes, fmtDuration, fmtInt, fmtMs, relTime, shortHash, untilTime, utcDate, utcDateTime, utcTime } from "@/lib/format"; | |
| 7 | 10 | |
| 8 | 11 | export const dynamic = "force-dynamic"; |
| 9 | 12 | |
@@ -13,93 +16,182 @@ export async function generateMetadata({ params }: { params: Promise<{ id: strin | ||
| 13 | 16 | return { title: d ? `${d.sensor.source_name} · ${d.sensor.name} — sensor` : "Sensor not found", robots: { index: false } }; |
| 14 | 17 | } |
| 15 | 18 | |
| 19 | +/** Sensor page with historical memory (spec §79): how this page looked over time. */ | |
| 16 | 20 | export default async function SensorPage({ params }: { params: Promise<{ id: string }> }) { |
| 17 | 21 | const { id } = await params; |
| 18 | 22 | const d = await api.sensor(id); |
| 19 | 23 | if (!d) notFound(); |
| 20 | 24 | const s = d.sensor; |
| 25 | + const snaps = await api.sensorSnapshots(s.id, 200); | |
| 26 | + const history = snaps.items.length ? snaps.items : d.snapshots; | |
| 21 | 27 | const runs = s.total_runs ?? 0; |
| 28 | + const checks24 = s.checks_24h ?? 0; | |
| 29 | + const nm24 = checks24 ? Math.round(((s.not_modified_24h ?? 0) / checks24) * 100) : null; | |
| 30 | + const nmAll = runs ? Math.round(((s.total_not_modified ?? 0) / runs) * 100) : null; | |
| 31 | + const events = d.events ?? []; | |
| 32 | + const prio = s.priority ?? null; | |
| 33 | + // Group snapshots by UTC day (newest first, API order). | |
| 34 | + const days: { day: string; rows: SnapshotRow[] }[] = []; | |
| 35 | + for (const sn of history) { | |
| 36 | + const day = utcDate(sn.captured_at); | |
| 37 | + const last = days[days.length - 1]; | |
| 38 | + if (last && last.day === day) last.rows.push(sn); | |
| 39 | + else days.push({ day, rows: [sn] }); | |
| 40 | + } | |
| 41 | + const changeCount = history.filter((x) => x.has_change).length; | |
| 22 | 42 | return ( |
| 23 | 43 | <> |
| 24 | 44 | <PageHeader |
| 25 | − kicker={<span className="flex items-center gap-2"><Link href={`/source/${s.source_id}`} className="hover:underline">{s.source_name}</Link> · <span className="font-mono">{s.type}</span> · {s.connector} · <TierBadge tier={s.tier} /></span>} | |
| 45 | + compact | |
| 46 | + kicker={ | |
| 47 | + <span className="flex flex-wrap items-center gap-2"> | |
| 48 | + <Link href={`/source/${s.source_id}`} className="hover:underline">{s.source_name}</Link> | |
| 49 | + <span className="text-fg-subtle">/</span> | |
| 50 | + <span className="font-mono">{s.type}</span> | |
| 51 | + <span className="text-fg-subtle">·</span> | |
| 52 | + <span className="font-mono text-fg-muted">{s.connector}</span> | |
| 53 | + <TierBadge tier={s.tier} /> | |
| 54 | + {prio !== null && <Chip tone={prio === 0 ? "hot" : prio === 1 ? "high" : "default"} className="font-mono" title={PRIORITY_LABELS[prio] ?? `Priority ${prio}`}>P{prio}</Chip>} | |
| 55 | + {s.status && <HealthPill health={s.enabled ? s.status : "DISABLED"} />} | |
| 56 | + <HealthPill health={s.enabled ? s.health : "DISABLED"} /> | |
| 57 | + </span> | |
| 58 | + } | |
| 26 | 59 | title={s.name} |
| 27 | 60 | description={<ExtLink href={s.url} className="font-mono text-[12px] break-all">{s.url}</ExtLink>} |
| 28 | − actions={<><HealthPill health={s.enabled ? s.health : "DISABLED"} /><Link href={`/url?u=${encodeURIComponent(s.url)}`} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">URL history →</Link></>} | |
| 61 | + actions={ | |
| 62 | + <> | |
| 63 | + <Link href={`/url?u=${encodeURIComponent(s.url)}`} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">URL history →</Link> | |
| 64 | + {s.domain && <Link href={`/domain/${s.domain}`} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">Domain →</Link>} | |
| 65 | + </> | |
| 66 | + } | |
| 29 | 67 | /> |
| 68 | + | |
| 30 | 69 | <div className="panel mb-4 grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-4 lg:grid-cols-8 lg:divide-y-0"> |
| 31 | − <Stat label="Runs" value={fmtInt(runs)} hint={`${runs ? Math.round(((s.total_not_modified ?? 0) / runs) * 100) : 0}% 304`} /> | |
| 32 | − <Stat label="Raw changes" value={fmtInt(s.raw_changes)} /> | |
| 33 | − <Stat label="Meaningful" value={fmtInt(s.meaningful_changes)} hint={s.raw_changes ? `noise ${Math.round((1 - (s.meaningful_changes ?? 0) / Math.max(1, s.raw_changes)) * 100)}%` : undefined} /> | |
| 34 | − <Stat label="Latency" value={fmtMs(s.avg_latency_ms)} /> | |
| 70 | + <Stat label="Checks · 24 h" value={fmtInt(checks24)} hint={`${fmtInt(runs)} total`} /> | |
| 71 | + <Stat label="304 · 24 h" value={nm24 !== null ? `${nm24}%` : "—"} hint={nmAll !== null ? `${nmAll}% all-time` : s.has_etag || s.has_last_modified ? "validators exposed" : "no validators"} /> | |
| 72 | + <Stat label="Errors · 24 h" value={fmtInt(s.errors_24h ?? 0)} tone={(s.errors_24h ?? 0) > 0 ? "warn" : undefined} hint={s.consecutive_errors ? <span className="text-danger">{s.consecutive_errors} consecutive</span> : "0 consecutive"} /> | |
| 73 | + <Stat label="Avg latency" value={fmtMs(s.avg_ms_24h ?? s.avg_latency_ms)} hint={s.avg_ms_24h !== null && s.avg_ms_24h !== undefined ? "last 24 h" : "all-time"} /> | |
| 74 | + <Stat label="Raw / meaningful" value={<span>{fmtInt(s.raw_changes)}<span className="text-fg-subtle"> / </span><span className="text-signal">{fmtInt(s.meaningful_changes)}</span></span>} hint={s.raw_changes ? `noise ${Math.round((1 - (s.meaningful_changes ?? 0) / Math.max(1, s.raw_changes)) * 100)}%` : undefined} /> | |
| 75 | + <Stat label="Snapshots" value={fmtInt(history.length)} hint={`${changeCount} with a change`} /> | |
| 35 | 76 | <Stat label="Last check" value={s.last_check_at ? relTime(s.last_check_at) : "never"} hint={s.last_status ? `HTTP ${s.last_status}` : undefined} /> |
| 36 | − <Stat label="Next check" value={untilTime(s.next_check_at)} hint={s.base_interval_seconds ? `base ${fmtDuration(s.base_interval_seconds)}` : "adaptive"} /> | |
| 37 | − <Stat label="Last change" value={s.last_change_at ? relTime(s.last_change_at) : "—"} /> | |
| 38 | − <Stat label="Errors" value={fmtInt(s.consecutive_errors)} hint={s.last_error ? <span className="text-danger">{s.last_error}</span> : "consecutive"} /> | |
| 77 | + <Stat label="Next check" value={untilTime(s.next_check_at)} hint={s.current_interval_seconds ? `every ${fmtDuration(s.current_interval_seconds)}` : s.base_interval_seconds ? `base ${fmtDuration(s.base_interval_seconds)}` : "adaptive"} /> | |
| 39 | 78 | </div> |
| 40 | − <div className="grid gap-4 lg:grid-cols-2"> | |
| 41 | − <Panel title={`Changes · ${d.changes.length}`} dense> | |
| 42 | − {d.changes.length ? ( | |
| 43 | − <Table head={["Detected", "Kind", "Signal", "Noise", "Magnitude", "Heuristic", "Event", "Compare"]}> | |
| 44 | − {d.changes.map((c) => ( | |
| 45 | − <tr key={c.id}> | |
| 46 | − <Td mono className="text-fg-subtle">{utcDateTime(c.detected_at)}</Td> | |
| 47 | − <Td mono>{c.kind}</Td> | |
| 48 | − <Td mono className={c.signal >= 0.32 ? "text-signal" : "text-fg-subtle"}>{c.signal.toFixed(2)}</Td> | |
| 49 | − <Td mono>{c.noise_ratio !== undefined ? `${Math.round(c.noise_ratio * 100)}%` : "—"}</Td> | |
| 50 | − <Td mono>{c.magnitude ?? "—"}</Td> | |
| 51 | − <Td>{c.heuristic_type ? <Chip>{c.heuristic_type.replace(/_/g, " ")}</Chip> : "—"}</Td> | |
| 52 | − <Td>{c.event_id ? <Link href={`/event/${c.event_id}`} className="text-info hover:underline">event</Link> : <span className="text-fg-subtle">{c.meaningful ? "—" : "filtered"}</span>}</Td> | |
| 53 | − <Td>{c.old_snapshot_id && c.new_snapshot_id ? <Link href={`/compare?a=${c.old_snapshot_id}&b=${c.new_snapshot_id}`} className="text-info hover:underline">diff</Link> : "—"}</Td> | |
| 54 | − </tr> | |
| 55 | − ))} | |
| 56 | − </Table> | |
| 57 | − ) : ( | |
| 58 | − <Empty>No changes detected yet.</Empty> | |
| 59 | − )} | |
| 60 | − </Panel> | |
| 61 | − <Panel title={`Snapshots · ${d.snapshots.length}`} dense> | |
| 62 | − {d.snapshots.length ? ( | |
| 63 | − <Table head={["Captured", "HTTP", "Type", "Size", "Canonical hash", "Confidence", ""]}> | |
| 64 | − {d.snapshots.map((sn, i) => ( | |
| 65 | − <tr key={sn.id}> | |
| 66 | − <Td mono className="text-fg-subtle">{utcDateTime(sn.captured_at)}</Td> | |
| 67 | − <Td mono>{sn.http_status ?? "—"}</Td> | |
| 68 | − <Td mono className="text-fg-subtle">{(sn.content_type ?? sn.mode ?? "").split(";")[0]}</Td> | |
| 69 | − <Td mono>{fmtBytes(sn.content_length)}</Td> | |
| 70 | − <Td mono className="text-fg-subtle">{shortHash(sn.canonical_hash)}</Td> | |
| 71 | − <Td mono>{sn.extraction_confidence !== null && sn.extraction_confidence !== undefined ? sn.extraction_confidence.toFixed(2) : "—"}</Td> | |
| 72 | − <Td> | |
| 73 | − <a href={`/api/v1/snapshots/${sn.id}?raw=1`} target="_blank" rel="noopener noreferrer" className="text-info hover:underline">raw</a> | |
| 74 | − {d.snapshots[i + 1] && <> · <Link href={`/compare?a=${d.snapshots[i + 1]!.id}&b=${sn.id}`} className="text-info hover:underline">vs previous</Link></>} | |
| 75 | − </Td> | |
| 76 | − </tr> | |
| 77 | − ))} | |
| 78 | − </Table> | |
| 79 | − ) : ( | |
| 80 | − <Empty>No snapshots yet.</Empty> | |
| 81 | − )} | |
| 82 | − </Panel> | |
| 79 | + {s.last_error && ( | |
| 80 | + <div className="mb-4 rounded-md border border-danger/40 bg-danger/10 px-3 py-2 font-mono text-[12px] text-danger break-words"> | |
| 81 | + last error · {s.last_error} | |
| 82 | + </div> | |
| 83 | + )} | |
| 84 | + | |
| 85 | + <div className="grid gap-4 xl:grid-cols-[1fr_380px]"> | |
| 86 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 87 | + <Panel title={`How this page looked over time · ${history.length} snapshot${history.length === 1 ? "" : "s"}`} dense action={<span className="hidden text-[11px] text-fg-subtle sm:inline">raw bodies may be pruned by retention · canonical form and hashes are kept</span>}> | |
| 88 | + {history.length === 0 ? ( | |
| 89 | + <Empty>No snapshot yet — the first successful check stores the initial state of this page.</Empty> | |
| 90 | + ) : ( | |
| 91 | + <div className="overflow-x-auto"> | |
| 92 | + <table className="w-full text-[12.5px]"> | |
| 93 | + <thead> | |
| 94 | + <tr className="border-b border-line text-left"> | |
| 95 | + {["Captured", "HTTP", "Size", "Canonical hash", "Change", "Event", "Raw", "Compare"].map((h) => ( | |
| 96 | + <th key={h} className="label whitespace-nowrap px-3 py-2 font-semibold">{h}</th> | |
| 97 | + ))} | |
| 98 | + </tr> | |
| 99 | + </thead> | |
| 100 | + {days.map((g) => ( | |
| 101 | + <tbody key={g.day} className="divide-y divide-line border-b border-line"> | |
| 102 | + <tr className="bg-panel-2/50"> | |
| 103 | + <td colSpan={8} className="px-3 py-1 font-mono text-[10.5px] font-semibold tracking-wider text-fg-subtle">{dayHeader(g.rows[0]!.captured_at)} <span className="font-normal">· {g.rows.length} capture{g.rows.length === 1 ? "" : "s"}</span></td> | |
| 104 | + </tr> | |
| 105 | + {g.rows.map((sn) => { | |
| 106 | + const idx = history.indexOf(sn); | |
| 107 | + const prev = history[idx + 1]; | |
| 108 | + return ( | |
| 109 | + <tr key={sn.id} className={`hover:bg-panel-2/60 ${sn.has_change ? "border-l-2 border-l-signal/60" : "border-l-2 border-l-transparent"}`}> | |
| 110 | + <Td mono className="whitespace-nowrap"><span title={utcDateTime(sn.captured_at)}>{utcTime(sn.captured_at)}</span> <span className="text-fg-subtle">{relTime(sn.captured_at)}</span></Td> | |
| 111 | + <Td mono className={sn.http_status && sn.http_status >= 400 ? "text-warn" : "text-fg-muted"}>{sn.http_status ?? "—"}</Td> | |
| 112 | + <Td mono className="whitespace-nowrap">{fmtBytes(sn.content_length)}</Td> | |
| 113 | + <Td mono className="text-fg-subtle"><span title={sn.canonical_hash ?? undefined}>{shortHash(sn.canonical_hash)}</span>{sn.mode ? <span className="ml-1.5 text-[10.5px]">{sn.mode}</span> : null}</Td> | |
| 114 | + <Td>{sn.has_change ? <Chip tone="signal" className="font-mono">CHANGED</Chip> : <span className="text-fg-subtle">—</span>}</Td> | |
| 115 | + <Td>{sn.event_slug ? <Link href={`/event/${sn.event_slug}`} className="text-info hover:underline">event →</Link> : <span className="text-fg-subtle">—</span>}</Td> | |
| 116 | + <Td> | |
| 117 | + {sn.has_raw === false ? ( | |
| 118 | + <span className="cursor-not-allowed text-fg-subtle line-through" title="raw body pruned by retention">view raw</span> | |
| 119 | + ) : ( | |
| 120 | + <a href={`/api/v1/snapshots/${sn.id}?raw=1`} target="_blank" rel="noopener noreferrer" className="text-info hover:underline">view raw</a> | |
| 121 | + )} | |
| 122 | + </Td> | |
| 123 | + <Td>{prev ? <Link href={`/compare?a=${prev.id}&b=${sn.id}`} className="whitespace-nowrap text-info hover:underline">compare with previous</Link> : <span className="text-fg-subtle">first capture</span>}</Td> | |
| 124 | + </tr> | |
| 125 | + ); | |
| 126 | + })} | |
| 127 | + </tbody> | |
| 128 | + ))} | |
| 129 | + </table> | |
| 130 | + </div> | |
| 131 | + )} | |
| 132 | + </Panel> | |
| 133 | + | |
| 134 | + <Panel title={`Changes · ${d.changes.length}`} dense> | |
| 135 | + {d.changes.length ? ( | |
| 136 | + <Table head={["Detected", "Kind", "Class", "Signal", "Meaningful", "What changed", "Links"]}> | |
| 137 | + {d.changes.map((c) => ( | |
| 138 | + <tr key={c.id} className="hover:bg-panel-2/60"> | |
| 139 | + <Td mono className="whitespace-nowrap text-fg-subtle"><span title={utcDateTime(c.detected_at)}>{utcDate(c.detected_at)} {utcTime(c.detected_at, false)}</span></Td> | |
| 140 | + <Td mono>{c.kind}</Td> | |
| 141 | + <Td>{c.change_class ? <Chip tone={c.change_class === "pricing" || c.change_class === "policy" ? "high" : c.change_class === "meaningful" || c.change_class === "product" || c.change_class === "personnel" ? "signal" : "default"}>{CLASS_LABELS[c.change_class] ?? c.change_class}</Chip> : c.heuristic_type ? <Chip>{c.heuristic_type.replace(/_/g, " ")}</Chip> : <span className="text-fg-subtle">—</span>}</Td> | |
| 142 | + <Td mono className={c.signal >= 0.32 ? "text-signal" : "text-fg-subtle"}>{c.signal.toFixed(2)}{c.noise_ratio !== undefined && c.noise_ratio > 0 ? <span className="ml-1 text-[10.5px] text-fg-subtle">noise {Math.round(c.noise_ratio * 100)}%</span> : null}</Td> | |
| 143 | + <Td>{c.meaningful ? <Chip tone="ok" className="font-mono">YES</Chip> : <span className="font-mono text-[11px] text-fg-subtle">filtered</span>}</Td> | |
| 144 | + <Td className="min-w-[14rem]">{c.field_changes?.length ? <FieldChangeInline items={c.field_changes} max={2} /> : c.magnitude !== undefined && c.magnitude !== null ? <span className="font-mono text-[11px] text-fg-subtle">magnitude {c.magnitude}</span> : <span className="text-fg-subtle">—</span>}</Td> | |
| 145 | + <Td className="whitespace-nowrap"> | |
| 146 | + {c.event_id && <Link href={`/event/${c.event_id}`} className="text-info hover:underline">event</Link>} | |
| 147 | + {c.event_id && c.old_snapshot_id && c.new_snapshot_id && <span className="text-fg-subtle"> · </span>} | |
| 148 | + {c.old_snapshot_id && c.new_snapshot_id && <Link href={`/compare?a=${c.old_snapshot_id}&b=${c.new_snapshot_id}`} className="text-info hover:underline">diff</Link>} | |
| 149 | + {!c.event_id && !(c.old_snapshot_id && c.new_snapshot_id) && <span className="text-fg-subtle">—</span>} | |
| 150 | + </Td> | |
| 151 | + </tr> | |
| 152 | + ))} | |
| 153 | + </Table> | |
| 154 | + ) : ( | |
| 155 | + <Empty>No change detected yet.</Empty> | |
| 156 | + )} | |
| 157 | + </Panel> | |
| 158 | + | |
| 159 | + <Panel title={`Runs · ${d.runs.length}`} dense> | |
| 160 | + {d.runs.length ? ( | |
| 161 | + <Table head={["Started", "Outcome", "HTTP", "Duration", "Bytes", "Method", "Error"]}> | |
| 162 | + {d.runs.map((r) => ( | |
| 163 | + <tr key={r.id} className="hover:bg-panel-2/60"> | |
| 164 | + <Td mono className="whitespace-nowrap text-fg-subtle">{utcDateTime(r.started_at)}</Td> | |
| 165 | + <Td><Chip tone={r.outcome === "event" ? "signal" : r.outcome === "changed" ? "info" : r.outcome === "error" || r.outcome === "parse_error" ? "danger" : r.outcome === "rate_limited" || r.outcome === "missing" ? "warn" : "default"}>{r.outcome}</Chip></Td> | |
| 166 | + <Td mono>{r.http_status ?? "—"}</Td> | |
| 167 | + <Td mono>{fmtMs(r.duration_ms)}</Td> | |
| 168 | + <Td mono>{fmtBytes(r.bytes)}</Td> | |
| 169 | + <Td mono className="text-fg-subtle">{r.fetch_method ?? "—"}</Td> | |
| 170 | + <Td className="max-w-[24rem] truncate text-danger" >{r.error ?? ""}</Td> | |
| 171 | + </tr> | |
| 172 | + ))} | |
| 173 | + </Table> | |
| 174 | + ) : ( | |
| 175 | + <Empty>No run recorded in the retained window.</Empty> | |
| 176 | + )} | |
| 177 | + </Panel> | |
| 178 | + </div> | |
| 179 | + | |
| 180 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 181 | + <Panel title={`Events · ${events.length}`} dense action={<Link href={`/source/${s.source_id}`} className="text-[11px] text-fg-subtle hover:text-fg">source →</Link>}> | |
| 182 | + {events.length ? events.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>No meaningful event from this sensor yet — raw changes that are noise never become events.</Empty>} | |
| 183 | + </Panel> | |
| 184 | + <Panel title="Conditional requests"> | |
| 185 | + <dl className="grid grid-cols-[auto_1fr] gap-x-3 gap-y-1 text-[12.5px]"> | |
| 186 | + <dt className="text-fg-subtle">ETag</dt><dd className="min-w-0 truncate font-mono text-[11.5px]" title={s.etag ?? undefined}>{s.etag ?? <span className="text-fg-subtle">not exposed</span>}</dd> | |
| 187 | + <dt className="text-fg-subtle">Last-Modified</dt><dd className="min-w-0 truncate font-mono text-[11.5px]" title={s.last_modified ?? undefined}>{s.last_modified ?? <span className="text-fg-subtle">not exposed</span>}</dd> | |
| 188 | + <dt className="text-fg-subtle">Last change</dt><dd className="font-mono text-[11.5px]">{s.last_change_at ? relTime(s.last_change_at) : "—"}</dd> | |
| 189 | + <dt className="text-fg-subtle">Last event</dt><dd className="font-mono text-[11.5px]">{s.last_event_at ? relTime(s.last_event_at) : "—"}</dd> | |
| 190 | + <dt className="text-fg-subtle">Validated</dt><dd className="font-mono text-[11.5px]">{s.validated_at ? relTime(s.validated_at) : "—"}</dd> | |
| 191 | + </dl> | |
| 192 | + </Panel> | |
| 193 | + </aside> | |
| 83 | 194 | </div> |
| 84 | − <Panel title={`Runs · ${d.runs.length}`} dense className="mt-4"> | |
| 85 | − {d.runs.length ? ( | |
| 86 | − <Table head={["Started", "Outcome", "HTTP", "Duration", "Bytes", "Method", "Error"]}> | |
| 87 | − {d.runs.map((r) => ( | |
| 88 | − <tr key={r.id}> | |
| 89 | − <Td mono className="text-fg-subtle">{utcDateTime(r.started_at)}</Td> | |
| 90 | − <Td><Chip tone={r.outcome === "event" ? "signal" : r.outcome === "changed" ? "info" : r.outcome === "error" || r.outcome === "parse_error" ? "danger" : r.outcome === "rate_limited" || r.outcome === "missing" ? "warn" : "default"}>{r.outcome}</Chip></Td> | |
| 91 | − <Td mono>{r.http_status ?? "—"}</Td> | |
| 92 | − <Td mono>{fmtMs(r.duration_ms)}</Td> | |
| 93 | − <Td mono>{fmtBytes(r.bytes)}</Td> | |
| 94 | − <Td mono className="text-fg-subtle">{r.fetch_method ?? "—"}</Td> | |
| 95 | − <Td className="max-w-[24rem] truncate text-danger" >{r.error ?? ""}</Td> | |
| 96 | − </tr> | |
| 97 | − ))} | |
| 98 | − </Table> | |
| 99 | − ) : ( | |
| 100 | − <Empty>No runs recorded yet.</Empty> | |
| 101 | − )} | |
| 102 | − </Panel> | |
| 103 | 195 | </> |
| 104 | 196 | ); |
| 105 | 197 | } |
added
apps/web/src/app/silent/loading.tsx
+38 −0
@@ -0,0 +1,38 @@ | ||
| 1 | +import { Skeleton, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-4 flex flex-col gap-2"> | |
| 7 | + <Skeleton className="h-3 w-48" /> | |
| 8 | + <Skeleton className="h-7 w-52" /> | |
| 9 | + <Skeleton className="h-3 w-full max-w-2xl" /> | |
| 10 | + </div> | |
| 11 | + <div className="panel mb-4"> | |
| 12 | + <div className="border-b border-line px-3 py-2"> | |
| 13 | + <Skeleton className="h-2.5 w-36" /> | |
| 14 | + </div> | |
| 15 | + <ul className="grid grid-cols-1 gap-px bg-line md:grid-cols-2 xl:grid-cols-3"> | |
| 16 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 17 | + <li key={i} className="flex flex-col gap-1.5 bg-panel px-3 py-2.5"> | |
| 18 | + <div className="flex items-center gap-2"> | |
| 19 | + <Skeleton className="h-3 w-20" /> | |
| 20 | + <Skeleton className="h-4 w-12" /> | |
| 21 | + <Skeleton className="ml-auto h-3 w-10" /> | |
| 22 | + </div> | |
| 23 | + <Skeleton className={`h-3.5 ${i % 2 ? "w-4/5" : "w-11/12"}`} /> | |
| 24 | + <Skeleton className="h-2.5 w-2/3" /> | |
| 25 | + </li> | |
| 26 | + ))} | |
| 27 | + </ul> | |
| 28 | + </div> | |
| 29 | + <div className="panel overflow-hidden"> | |
| 30 | + <div className="flex items-center gap-3 border-b border-line px-3 py-2"> | |
| 31 | + <Skeleton className="h-3 w-44" /> | |
| 32 | + <Skeleton className="h-3 w-12" /> | |
| 33 | + </div> | |
| 34 | + <SkeletonRows rows={10} /> | |
| 35 | + </div> | |
| 36 | + </> | |
| 37 | + ); | |
| 38 | +} | |
modified
apps/web/src/app/silent/page.tsx
+44 −5
@@ -1,17 +1,56 @@ | ||
| 1 | 1 | import type { Metadata } from "next"; |
| 2 | +import Link from "next/link"; | |
| 3 | +import { FieldChanges } from "@/components/field-changes"; | |
| 2 | 4 | import { LiveFeed } from "@/components/live-feed"; |
| 3 | −import { PageHeader } from "@/components/ui"; | |
| 5 | +import { Badge, Chip, Empty, Flag, PageHeader, Panel, Score } from "@/components/ui"; | |
| 4 | 6 | import { api } from "@/lib/api"; |
| 7 | +import { agoIso, relTime, typeLabel, utcDateTime } from "@/lib/format"; | |
| 5 | 8 | |
| 6 | 9 | export const dynamic = "force-dynamic"; |
| 7 | −export const metadata: Metadata = { title: "Silent changes", description: "Important changes detected without a corresponding public announcement." }; | |
| 10 | +export const metadata: Metadata = { title: "Silent changes", description: "Pricing, terms, API limits, documentation and product pages that changed quietly — no announcement matched. Previous state, current state, exact diff." }; | |
| 8 | 11 | |
| 12 | +/** Dedicated silent-change experience (spec §23). */ | |
| 9 | 13 | export default async function SilentPage() { |
| 10 | − const events = await api.events({ silent_change: true, limit: 60 }); | |
| 14 | + const [top, recent, count24] = await Promise.all([api.events({ silent_change: true, limit: 12, order: "signal", after: agoIso(48 * 3600e3) }), api.events({ silent_change: true, limit: 60 }), api.eventCount({ silent_change: true, after: agoIso(24 * 3600e3) })]); | |
| 15 | + const byType = new Map<string, number>(); | |
| 16 | + for (const e of recent.items) byType.set(e.event_type, (byType.get(e.event_type) ?? 0) + 1); | |
| 11 | 17 | return ( |
| 12 | 18 | <> |
| 13 | − <PageHeader kicker="Flagship signal" title={<span>⚠ Silent changes</span>} description="Pricing, terms, API limits, documentation or product pages that changed quietly — no announcement matched within the observation window. Each item links to the exact diff and both preserved snapshots." /> | |
| 14 | − <LiveFeed initial={events.items} initialCursor={events.nextCursor} fixed="silent" title="SILENT CHANGES" /> | |
| 19 | + <PageHeader | |
| 20 | + kicker={<span className="flex items-center gap-2"><Badge kind="silent" /> <span className="text-fg-subtle">{count24.count} in the last 24 h · first-party pages only · importance ≥ 45 · no matching announcement within 12 h</span></span>} | |
| 21 | + title="Silent changes" | |
| 22 | + description="Something was modified without an obvious public announcement: a price quietly changed, documentation rewritten, a feature removed, terms edited, an API limit adjusted, a product page gone. Each item shows the previous state, the current state and the exact diff." | |
| 23 | + actions={<div className="flex flex-wrap gap-1">{[...byType.entries()].sort((a, b) => b[1] - a[1]).slice(0, 8).map(([t, n]) => <Chip key={t} href={`/live?silent_change=true&event_type=${t}`}>{typeLabel(t)} <span className="font-mono text-fg-subtle">{n}</span></Chip>)}</div>} | |
| 24 | + /> | |
| 25 | + <Panel title="Most significant · 48 h" dense className="mb-4"> | |
| 26 | + {top.items.length === 0 ? ( | |
| 27 | + <Empty>No silent change detected in this period.</Empty> | |
| 28 | + ) : ( | |
| 29 | + <ul className="grid gap-px bg-line md:grid-cols-2 xl:grid-cols-3"> | |
| 30 | + {top.items.map((e) => ( | |
| 31 | + <li key={e.id} className="flex flex-col gap-2 bg-panel p-3"> | |
| 32 | + <div className="flex items-center gap-2 text-[11px] text-fg-subtle"> | |
| 33 | + <Link href={`/source/${e.source?.id ?? e.source_id}`} className="truncate font-mono uppercase text-fg-muted hover:text-fg">{e.source?.name}</Link> | |
| 34 | + {e.country && <Flag code={e.country} />} | |
| 35 | + <Chip>{typeLabel(e.event_type)}</Chip> | |
| 36 | + <span className="ml-auto whitespace-nowrap font-mono tabular" title={utcDateTime(e.detected_at)}>{relTime(e.detected_at)}</span> | |
| 37 | + </div> | |
| 38 | + <div className="flex items-start gap-2"> | |
| 39 | + <Score value={e.signal_score ?? e.importance} kind="signal" size="sm" /> | |
| 40 | + <Link href={`/event/${e.slug}`} className="font-medium leading-snug hover:underline">{e.title}</Link> | |
| 41 | + </div> | |
| 42 | + {e.field_changes?.length ? <FieldChanges items={e.field_changes} compact max={3} /> : <p className="line-clamp-3 text-[12px] text-fg-muted">{e.summary}</p>} | |
| 43 | + <div className="mt-auto flex items-center gap-2 text-[11px]"> | |
| 44 | + <Link href={`/event/${e.slug}#diff`} className="text-info hover:underline">exact diff</Link> | |
| 45 | + {e.old_snapshot_id && e.new_snapshot_id && <Link href={`/compare?a=${e.old_snapshot_id}&b=${e.new_snapshot_id}`} className="text-info hover:underline">before / after</Link>} | |
| 46 | + <span className="ml-auto text-fg-subtle">{e.evidence_label}</span> | |
| 47 | + </div> | |
| 48 | + </li> | |
| 49 | + ))} | |
| 50 | + </ul> | |
| 51 | + )} | |
| 52 | + </Panel> | |
| 53 | + <LiveFeed initial={recent.items} initialCursor={recent.nextCursor} fixed="silent" title="SILENT CHANGES · LIVE" showFilters={false} /> | |
| 15 | 54 | </> |
| 16 | 55 | ); |
| 17 | 56 | } |
modified
apps/web/src/app/sitemap.ts
+27 −6
@@ -1,19 +1,40 @@ | ||
| 1 | 1 | import type { MetadataRoute } from "next"; |
| 2 | 2 | import { api, SITE_URL } from "@/lib/api"; |
| 3 | +import { CHANNEL_KEYS } from "@/lib/format"; | |
| 3 | 4 | |
| 4 | 5 | export const dynamic = "force-dynamic"; |
| 5 | 6 | |
| 7 | +/** Public, crawlable surface only — user tools (/watchlists, /alerts, /bookmarks, /monitors, /ops) are noindex. */ | |
| 6 | 8 | export default async function sitemap(): Promise<MetadataRoute.Sitemap> { |
| 7 | 9 | const now = new Date(); |
| 8 | − const statics: MetadataRoute.Sitemap = ["", "/breaking", "/explore", "/sources", "/entities", "/silent", "/api", "/health", "/bot", "/watchlists", "/alerts", ...["ai", "cyber", "finance", "health", "government", "science", "products", "infrastructure", "news"].map((c) => `/category/${c}`)].map((p) => ({ url: `${SITE_URL}${p}`, lastModified: now, changeFrequency: p === "" ? "always" : "hourly", priority: p === "" ? 1 : 0.7 })); | |
| 9 | − const [events, sources, entities] = await Promise.all([api.events({ limit: 200 }), api.sources(), api.entities({ limit: 500 })]); | |
| 10 | + const statics: MetadataRoute.Sitemap = [ | |
| 11 | + { url: `${SITE_URL}/`, lastModified: now, changeFrequency: "always", priority: 1 }, | |
| 12 | + { url: `${SITE_URL}/live`, lastModified: now, changeFrequency: "always", priority: 0.9 }, | |
| 13 | + { url: `${SITE_URL}/breaking`, lastModified: now, changeFrequency: "always", priority: 0.9 }, | |
| 14 | + { url: `${SITE_URL}/pulse`, lastModified: now, changeFrequency: "always", priority: 0.8 }, | |
| 15 | + { url: `${SITE_URL}/radar`, lastModified: now, changeFrequency: "hourly", priority: 0.7 }, | |
| 16 | + { url: `${SITE_URL}/silent`, lastModified: now, changeFrequency: "hourly", priority: 0.8 }, | |
| 17 | + { url: `${SITE_URL}/explore`, lastModified: now, changeFrequency: "hourly", priority: 0.7 }, | |
| 18 | + { url: `${SITE_URL}/sources`, lastModified: now, changeFrequency: "daily", priority: 0.6 }, | |
| 19 | + { url: `${SITE_URL}/entities`, lastModified: now, changeFrequency: "hourly", priority: 0.6 }, | |
| 20 | + { url: `${SITE_URL}/country`, lastModified: now, changeFrequency: "hourly", priority: 0.6 }, | |
| 21 | + { url: `${SITE_URL}/api`, lastModified: now, changeFrequency: "weekly", priority: 0.5 }, | |
| 22 | + { url: `${SITE_URL}/health`, lastModified: now, changeFrequency: "hourly", priority: 0.3 }, | |
| 23 | + { url: `${SITE_URL}/bot`, lastModified: now, changeFrequency: "monthly", priority: 0.3 }, | |
| 24 | + ...CHANNEL_KEYS.map((c) => ({ url: `${SITE_URL}/category/${c}`, lastModified: now, changeFrequency: "hourly" as const, priority: 0.7 })), | |
| 25 | + ]; | |
| 26 | + const [events, sources, entities, countries, clusters] = await Promise.all([api.events({ limit: 200 }), api.sources(), api.entities({ limit: 500 }), api.countries(), api.clusters(200, 168)]); | |
| 10 | 27 | const more = await (events.nextCursor ? api.events({ limit: 200, cursor: events.nextCursor }) : Promise.resolve({ items: [], nextCursor: null })); |
| 11 | 28 | const third = await (more.nextCursor ? api.events({ limit: 100, cursor: more.nextCursor }) : Promise.resolve({ items: [], nextCursor: null })); |
| 12 | − return [ | |
| 29 | + const seen = new Set<string>(); | |
| 30 | + const unique = <T extends { url: string }>(rows: T[]): T[] => rows.filter((r) => (seen.has(r.url) ? false : (seen.add(r.url), true))); | |
| 31 | + return unique([ | |
| 13 | 32 | ...statics, |
| 14 | − ...[...events.items, ...more.items, ...third.items].map((e) => ({ url: `${SITE_URL}/event/${e.slug}`, lastModified: new Date(e.detected_at), changeFrequency: "daily" as const, priority: Math.min(0.9, 0.4 + e.importance / 200) })), | |
| 33 | + ...[...events.items, ...more.items, ...third.items].map((e) => ({ url: `${SITE_URL}/event/${e.slug}`, lastModified: new Date(e.detected_at), changeFrequency: "daily" as const, priority: Math.min(0.9, 0.4 + (e.signal_score ?? e.importance) / 200) })), | |
| 34 | + ...clusters.items.filter((c) => c.slug).map((c) => ({ url: `${SITE_URL}/cluster/${c.slug}`, lastModified: new Date(c.last_at), changeFrequency: "hourly" as const, priority: c.state === "breaking" ? 0.9 : 0.6 })), | |
| 35 | + ...countries.items.map((c) => ({ url: `${SITE_URL}/country/${c.slug}`, lastModified: now, changeFrequency: "hourly" as const, priority: 0.6 })), | |
| 15 | 36 | ...sources.items.map((s) => ({ url: `${SITE_URL}/source/${s.id}`, lastModified: s.last_event_at ? new Date(s.last_event_at) : now, changeFrequency: "hourly" as const, priority: 0.6 })), |
| 16 | 37 | ...sources.items.map((s) => ({ url: `${SITE_URL}/domain/${s.domain}`, lastModified: now, changeFrequency: "daily" as const, priority: 0.4 })), |
| 17 | − ...entities.items.map((e) => ({ url: `${SITE_URL}/company/${e.id}`, lastModified: e.last_event_at ? new Date(e.last_event_at) : now, changeFrequency: "hourly" as const, priority: 0.6 })), | |
| 18 | − ]; | |
| 38 | + ...entities.items.map((e) => ({ url: `${SITE_URL}/entity/${e.id}`, lastModified: e.last_event_at ? new Date(e.last_event_at) : now, changeFrequency: "hourly" as const, priority: 0.6 })), | |
| 39 | + ]); | |
| 19 | 40 | } |
added
apps/web/src/app/source/[id]/loading.tsx
+41 −0
@@ -0,0 +1,41 @@ | ||
| 1 | +import { Skeleton, SkeletonPanel, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3"> | |
| 7 | + <Skeleton className="mb-1 h-2.5 w-48" /> | |
| 8 | + <Skeleton className="h-5 w-40" /> | |
| 9 | + <Skeleton className="mt-2 h-3 w-full max-w-2xl" /> | |
| 10 | + </div> | |
| 11 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-line sm:grid-cols-3 lg:grid-cols-6"> | |
| 12 | + {Array.from({ length: 6 }, (_, i) => ( | |
| 13 | + <div key={i} className="flex flex-col gap-1.5 px-3 py-2"> | |
| 14 | + <Skeleton className="h-2.5 w-16" /> | |
| 15 | + <Skeleton className="h-5 w-12" /> | |
| 16 | + <Skeleton className="h-2.5 w-20" /> | |
| 17 | + </div> | |
| 18 | + ))} | |
| 19 | + </div> | |
| 20 | + <div className="grid gap-4 xl:grid-cols-[1fr_320px]"> | |
| 21 | + <div className="flex flex-col gap-4"> | |
| 22 | + <SkeletonPanel lines={3} /> | |
| 23 | + <div className="panel"> | |
| 24 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-24" /></div> | |
| 25 | + <SkeletonRows rows={8} /> | |
| 26 | + </div> | |
| 27 | + <div className="panel"> | |
| 28 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-28" /></div> | |
| 29 | + <SkeletonRows rows={8} /> | |
| 30 | + </div> | |
| 31 | + </div> | |
| 32 | + <div className="flex flex-col gap-4"> | |
| 33 | + <SkeletonPanel lines={6} /> | |
| 34 | + <SkeletonPanel lines={6} /> | |
| 35 | + <SkeletonPanel lines={4} /> | |
| 36 | + <SkeletonPanel lines={5} /> | |
| 37 | + </div> | |
| 38 | + </div> | |
| 39 | + </> | |
| 40 | + ); | |
| 41 | +} | |
modified
apps/web/src/app/source/[id]/page.tsx
+190 −51
@@ -1,10 +1,11 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | 2 | import type { Metadata } from "next"; |
| 3 | 3 | import { notFound } from "next/navigation"; |
| 4 | +import { PRIORITY_LABELS } from "@websensor/core/client"; | |
| 4 | 5 | import { EventRow } from "@/components/event-row"; |
| 5 | −import { Bar, Chip, Empty, ExtLink, HealthPill, PageHeader, Panel, Stat, Table, Td, TierBadge } from "@/components/ui"; | |
| 6 | −import { api } from "@/lib/api"; | |
| 7 | −import { fmtDuration, fmtInt, fmtMs, fmtScore, relTime, untilTime } from "@/lib/format"; | |
| 6 | +import { Badge, Bar, Chip, Empty, ExtLink, Flag, HealthPill, PageHeader, Panel, Sparkline, Stat, Table, Td, TierBadge } from "@/components/ui"; | |
| 7 | +import { api, type SensorRow } from "@/lib/api"; | |
| 8 | +import { fmtDuration, fmtInt, fmtMs, fmtPct, fmtScore, relTime, typeLabel, untilTime, utcDateTime } from "@/lib/format"; | |
| 8 | 9 | |
| 9 | 10 | export const dynamic = "force-dynamic"; |
| 10 | 11 | |
@@ -12,83 +13,137 @@ export async function generateMetadata({ params }: { params: Promise<{ id: strin | ||
| 12 | 13 | const { id } = await params; |
| 13 | 14 | const d = await api.source(id); |
| 14 | 15 | if (!d) return { title: "Source not found" }; |
| 15 | − return { title: `${d.source.name} — source`, description: d.source.description ?? `Sensors, activity and events for ${d.source.name} (${d.source.domain}).`, alternates: { canonical: `/source/${d.source.id}` } }; | |
| 16 | + return { title: `${d.source.name} — source intelligence`, description: d.source.description ?? `Sensors, quality, activity and events for ${d.source.name} (${d.source.domain}).`, alternates: { canonical: `/source/${d.source.id}` } }; | |
| 16 | 17 | } |
| 17 | 18 | |
| 19 | +function qualityTone(v: number): "signal" | "warn" | "hot" | undefined { | |
| 20 | + return v >= 80 ? "signal" : v >= 60 ? undefined : v >= 40 ? "warn" : "hot"; | |
| 21 | +} | |
| 22 | + | |
| 23 | +/** Source intelligence page (spec §27, §48). */ | |
| 18 | 24 | export default async function SourcePage({ params }: { params: Promise<{ id: string }> }) { |
| 19 | 25 | const { id } = await params; |
| 20 | 26 | const d = await api.source(id); |
| 21 | 27 | if (!d) notFound(); |
| 22 | 28 | const events = await api.events({ source: d.source.id, limit: 40 }); |
| 29 | + const src = d.source; | |
| 23 | 30 | const a = d.activity ?? {}; |
| 31 | + const q = d.quality; | |
| 24 | 32 | const score = a.activity_score ?? 0; |
| 25 | 33 | const entityId = d.entities.find((e) => e.type === "organization" || e.id.startsWith("org_"))?.id; |
| 34 | + const daily = [...(d.daily ?? [])].sort((x, y) => x.day.localeCompare(y.day)).slice(-30); | |
| 35 | + const sensorsUp = d.sensors.filter((s) => s.enabled && s.health === "UP").length; | |
| 36 | + const rawTotal = q?.raw_changes ?? d.sensors.reduce((n, s) => n + (s.raw_changes ?? 0), 0); | |
| 37 | + const meaningfulTotal = q?.meaningful_changes ?? d.sensors.reduce((n, s) => n + (s.meaningful_changes ?? 0), 0); | |
| 38 | + const maxType = Math.max(1, ...(d.by_type ?? []).map((t) => t.n)); | |
| 39 | + const extra = src as typeof src & { llm_enabled?: boolean; discover?: Record<string, boolean>; fallback?: Record<string, boolean>; rate_limit_per_min?: number | null; terms_reviewed_at?: string | null }; | |
| 26 | 40 | return ( |
| 27 | 41 | <> |
| 28 | 42 | <PageHeader |
| 29 | − kicker={<span className="flex items-center gap-2"><TierBadge tier={d.source.tier} /> Tier {d.source.tier} · <Link href={`/domain/${d.source.domain}`} className="font-mono hover:underline">{d.source.domain}</Link></span>} | |
| 30 | − title={d.source.name} | |
| 31 | − description={d.source.description} | |
| 43 | + compact | |
| 44 | + kicker={ | |
| 45 | + <span className="flex flex-wrap items-center gap-2"> | |
| 46 | + <TierBadge tier={src.tier} /> | |
| 47 | + <span>Tier {src.tier}</span> | |
| 48 | + <span className="text-fg-subtle">·</span> | |
| 49 | + <Link href={`/domain/${src.domain}`} className="font-mono hover:underline">{src.domain}</Link> | |
| 50 | + {src.country && ( | |
| 51 | + <> | |
| 52 | + <span className="text-fg-subtle">·</span> | |
| 53 | + <Link href={`/country/${src.country.toLowerCase()}`} className="inline-flex items-center gap-1 font-mono hover:text-fg"><Flag code={src.country} /> {src.country}</Link> | |
| 54 | + </> | |
| 55 | + )} | |
| 56 | + {src.first_party === false ? <Badge kind="external" /> : <Badge kind="first-party" />} | |
| 57 | + {src.enabled === false && <Chip tone="warn" className="font-mono">DISABLED</Chip>} | |
| 58 | + </span> | |
| 59 | + } | |
| 60 | + title={src.name} | |
| 61 | + description={src.description ?? (src.first_party === false ? `Media / aggregator source: its reports are third-party evidence and never count as first-party confirmation.` : `Official channels of ${src.name}: every sensor below is an endpoint the organization itself publishes.`)} | |
| 32 | 62 | actions={ |
| 33 | 63 | <> |
| 34 | − {d.source.categories.map((c) => <Chip key={c} href={`/category/${c}`}>{c}</Chip>)} | |
| 35 | − {entityId && <Link href={`/company/${entityId}`} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">Entity & timeline →</Link>} | |
| 36 | − <ExtLink href={d.source.homepage ?? `https://${d.source.domain}`} className="text-[12px]">Website ↗</ExtLink> | |
| 64 | + {src.categories.map((c) => <Chip key={c} href={`/category/${c}`}>{c}</Chip>)} | |
| 65 | + {entityId && <Link href={`/entity/${entityId}`} className="rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong">Entity & timeline →</Link>} | |
| 66 | + <Link href={`/live?q=${encodeURIComponent(src.name)}`} className="hidden rounded-md border border-line bg-panel px-2.5 py-1 text-[12px] hover:border-line-strong sm:inline-block">Live →</Link> | |
| 67 | + <ExtLink href={src.homepage ?? `https://${src.domain}`} className="text-[12px]">Website ↗</ExtLink> | |
| 37 | 68 | </> |
| 38 | 69 | } |
| 39 | 70 | /> |
| 40 | − <div className="grid gap-4 lg:grid-cols-[1fr_320px]"> | |
| 41 | − <div className="flex flex-col gap-4"> | |
| 42 | − <Panel title={`Sensors · ${d.sensors.length}`} dense> | |
| 43 | − {d.sensors.length === 0 ? ( | |
| 44 | − <Empty>No sensors yet — discovery validates feeds, sitemaps and status pages before creating sensors.</Empty> | |
| 45 | − ) : ( | |
| 46 | − <Table head={["Sensor", "Type", "Tier", "Health", "Last check", "Next", "Latency", "Raw / meaningful", "304 %", "Status"]}> | |
| 47 | − {d.sensors.map((s) => { | |
| 48 | − const runs = s.total_runs ?? 0; | |
| 49 | − const nm = runs ? Math.round(((s.total_not_modified ?? 0) / runs) * 100) : 0; | |
| 71 | + | |
| 72 | + <div className="panel mb-4 grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-3 lg:grid-cols-6 lg:divide-y-0"> | |
| 73 | + <Stat label="Quality score" value={q ? fmtScore(q.quality_score) : "—"} tone={q ? qualityTone(q.quality_score) : undefined} hint={q ? `${q.sensors_up ?? sensorsUp} / ${q.sensors ?? d.sensors.length} sensors up` : "not computed yet"} /> | |
| 74 | + <Stat label="Success rate" value={q ? fmtPct(q.success_rate) : "—"} tone={q && q.success_rate < 0.9 ? "warn" : undefined} hint={q?.checks_7d ? `${fmtInt(q.checks_7d)} checks · ${fmtInt(q.errors_7d)} errors · 7 d` : `${fmtInt(q?.total_runs)} checks total`} /> | |
| 75 | + <Stat label="Avg latency" value={fmtMs(q?.avg_latency_ms)} hint="across sensors" /> | |
| 76 | + <Stat label="Structured" value={q ? fmtPct(q.structured_share) : "—"} hint={q ? `${q.structured_sensors ?? 0} feeds / APIs / status` : undefined} /> | |
| 77 | + <Stat label="Usefulness" value={q ? fmtPct(q.usefulness) : "—"} hint={`${fmtInt(meaningfulTotal)} meaningful / ${fmtInt(rawTotal)} raw`} /> | |
| 78 | + <Stat label="Sensors" value={<span>{sensorsUp}<span className="text-fg-subtle"> / {d.sensors.length}</span></span>} hint={`${d.sensors.filter((s) => s.health !== "UP").length} not UP`} tone={sensorsUp < d.sensors.length ? "warn" : undefined} /> | |
| 79 | + </div> | |
| 80 | + | |
| 81 | + <div className="grid gap-4 xl:grid-cols-[1fr_320px]"> | |
| 82 | + <div className="flex min-w-0 flex-col gap-4"> | |
| 83 | + <Panel title="30 days · daily activity" action={daily.length ? <span className="text-[11px] text-fg-subtle">{daily[0]!.day} → {daily[daily.length - 1]!.day}</span> : undefined}> | |
| 84 | + {daily.length ? ( | |
| 85 | + <div className="grid gap-4 sm:grid-cols-3"> | |
| 86 | + {( | |
| 87 | + [ | |
| 88 | + { label: "Checks", key: "checks", tone: "muted" }, | |
| 89 | + { label: "Raw changes", key: "raw_changes", tone: "info" }, | |
| 90 | + { label: "Events", key: "events", tone: "signal" }, | |
| 91 | + ] as const | |
| 92 | + ).map((s) => { | |
| 93 | + const values = daily.map((r) => Number(r[s.key] ?? 0)); | |
| 94 | + const sum = values.reduce((x, y) => x + y, 0); | |
| 50 | 95 | return ( |
| 51 | − <tr key={s.id} className="hover:bg-panel-2/60"> | |
| 52 | − <Td> | |
| 53 | − <Link href={`/sensor/${s.id}`} className="font-medium hover:underline">{s.name}</Link> | |
| 54 | − <div className="max-w-[28rem] truncate font-mono text-[11px] text-fg-subtle">{s.url}</div> | |
| 55 | − </Td> | |
| 56 | − <Td mono><span className="text-fg-muted">{s.type}</span><div className="text-[10.5px] text-fg-subtle">{s.connector}</div></Td> | |
| 57 | − <Td><TierBadge tier={s.tier} /></Td> | |
| 58 | − <Td><HealthPill health={s.enabled ? s.health : "DISABLED"} /></Td> | |
| 59 | − <Td mono className="text-fg-subtle">{s.last_check_at ? relTime(s.last_check_at) : "never"}</Td> | |
| 60 | − <Td mono className="text-fg-subtle">{untilTime(s.next_check_at)}{s.base_interval_seconds ? <div className="text-[10.5px]">base {fmtDuration(s.base_interval_seconds)}</div> : null}</Td> | |
| 61 | − <Td mono>{fmtMs(s.avg_latency_ms)}</Td> | |
| 62 | − <Td mono>{s.raw_changes ?? 0} / <span className="text-signal">{s.meaningful_changes ?? 0}</span></Td> | |
| 63 | − <Td mono>{runs ? `${nm}%` : "—"}<div className="text-[10.5px] text-fg-subtle">{s.has_etag ? "etag" : ""}{s.has_etag && s.has_last_modified ? "+" : ""}{s.has_last_modified ? "lm" : ""}</div></Td> | |
| 64 | − <Td mono>{s.last_status ?? "—"}{s.last_error ? <div className="max-w-[14rem] truncate text-[10.5px] text-danger" title={s.last_error}>{s.last_error}</div> : null}</Td> | |
| 65 | − </tr> | |
| 96 | + <div key={s.key} className="min-w-0"> | |
| 97 | + <div className="flex items-baseline justify-between"> | |
| 98 | + <span className="label">{s.label}</span> | |
| 99 | + <span className="font-mono text-[12px] tabular">{fmtInt(sum)}</span> | |
| 100 | + </div> | |
| 101 | + <div className="mt-1 w-full overflow-hidden"> | |
| 102 | + <Sparkline values={values} width={300} height={40} tone={s.tone} /> | |
| 103 | + </div> | |
| 104 | + </div> | |
| 66 | 105 | ); |
| 67 | 106 | })} |
| 107 | + </div> | |
| 108 | + ) : ( | |
| 109 | + <Empty>Daily counters accumulate from the first check onward — nothing recorded for the last 30 days yet.</Empty> | |
| 110 | + )} | |
| 111 | + </Panel> | |
| 112 | + | |
| 113 | + <Panel title={`Sensors · ${d.sensors.length}`} dense action={<span className="hidden text-[11px] text-fg-subtle sm:inline">P0 critical · P1 major · P2 specialized · P3 low urgency</span>}> | |
| 114 | + {d.sensors.length === 0 ? ( | |
| 115 | + <Empty>No sensors yet — discovery validates feeds, sitemaps and status pages before creating sensors.</Empty> | |
| 116 | + ) : ( | |
| 117 | + <Table head={["Sensor", "Type", "Tier", "Prio", "Status", "Health", "Polling", "Last / next", "HTTP", "Cond.", "24 h", "Snap.", "304 %", "Raw / mean.", ""]}> | |
| 118 | + {d.sensors.map((s) => <SensorTr key={s.id} s={s} />)} | |
| 68 | 119 | </Table> |
| 69 | 120 | )} |
| 70 | 121 | </Panel> |
| 71 | − <Panel title="Recent events" dense> | |
| 122 | + | |
| 123 | + <Panel title="Recent events" dense action={<Link href={`/live?q=${encodeURIComponent(src.name)}`} className="text-[11px] text-fg-subtle hover:text-fg">live →</Link>}> | |
| 72 | 124 | {events.items.length ? events.items.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>No meaningful events yet for this source.</Empty>} |
| 73 | 125 | </Panel> |
| 126 | + | |
| 74 | 127 | {d.discovery.length > 0 && ( |
| 75 | − <Panel title="Discovered endpoints" dense> | |
| 76 | − <Table head={["Kind", "URL", "Evidence", "Items", "Value", "Status"]}> | |
| 128 | + <Panel title={`Discovered endpoints · ${d.discovery.length}`} dense> | |
| 129 | + <Table head={["Kind", "URL", "Evidence", "Items", "Value", "Status", "Found"]}> | |
| 77 | 130 | {d.discovery.map((c) => ( |
| 78 | − <tr key={c.url}> | |
| 131 | + <tr key={c.url} className="hover:bg-panel-2/60"> | |
| 79 | 132 | <Td mono>{c.kind}</Td> |
| 80 | − <Td><span className="block max-w-[32rem] truncate font-mono text-[11.5px]">{c.url}</span></Td> | |
| 133 | + <Td><span className="block max-w-[28rem] truncate font-mono text-[11.5px]" title={c.url}>{c.url}</span></Td> | |
| 81 | 134 | <Td className="text-fg-subtle">{c.evidence}</Td> |
| 82 | 135 | <Td mono>{c.score?.itemCount ?? "—"}</Td> |
| 83 | 136 | <Td mono>{c.score?.value !== undefined ? fmtScore((c.score.value ?? 0) * 100) : "—"}</Td> |
| 84 | 137 | <Td><Chip tone={c.status === "promoted" ? "ok" : c.status === "rejected" ? "danger" : "default"}>{c.status}</Chip></Td> |
| 138 | + <Td mono className="whitespace-nowrap text-fg-subtle">{relTime(c.found_at)}</Td> | |
| 85 | 139 | </tr> |
| 86 | 140 | ))} |
| 87 | 141 | </Table> |
| 88 | 142 | </Panel> |
| 89 | 143 | )} |
| 90 | 144 | </div> |
| 91 | − <aside className="flex flex-col gap-4"> | |
| 145 | + | |
| 146 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 92 | 147 | <Panel title="Activity anomaly"> |
| 93 | 148 | <div className="flex items-baseline justify-between"> |
| 94 | 149 | <span className={`font-mono text-3xl font-semibold tabular ${score >= 70 ? "text-hot" : score >= 45 ? "text-high" : "text-fg"}`}>{fmtScore(score)}</span> |
@@ -100,19 +155,34 @@ export default async function SourcePage({ params }: { params: Promise<{ id: str | ||
| 100 | 155 | <dt className="text-fg-subtle">Current</dt><dd className="text-right font-mono tabular">{a.changes_2h ?? 0} in 2 h</dd> |
| 101 | 156 | <dt className="text-fg-subtle">Events 24 h</dt><dd className="text-right font-mono tabular">{a.events_24h ?? 0}</dd> |
| 102 | 157 | <dt className="text-fg-subtle">Events 14 d</dt><dd className="text-right font-mono tabular">{a.events_14d ?? 0}</dd> |
| 158 | + <dt className="text-fg-subtle">Silent 24 h</dt><dd className={`text-right font-mono tabular ${a.silent_24h ? "text-silent" : ""}`}>{a.silent_24h ?? 0}</dd> | |
| 159 | + <dt className="text-fg-subtle">Breaking 24 h</dt><dd className={`text-right font-mono tabular ${a.breaking_24h ? "text-hot" : ""}`}>{a.breaking_24h ?? 0}</dd> | |
| 103 | 160 | </dl> |
| 104 | 161 | </Panel> |
| 105 | − <div className="panel grid grid-cols-2 divide-x divide-line"> | |
| 106 | − <Stat label="Sensors" value={fmtInt(d.sensors.length)} hint={`${d.sensors.filter((s) => s.health === "UP").length} UP`} /> | |
| 107 | − <Stat label="Raw changes" value={fmtInt(d.sensors.reduce((n, s) => n + (s.raw_changes ?? 0), 0))} hint={`${fmtInt(d.sensors.reduce((n, s) => n + (s.meaningful_changes ?? 0), 0))} meaningful`} /> | |
| 108 | − </div> | |
| 109 | − <Panel title="Entities" dense> | |
| 162 | + | |
| 163 | + <Panel title="Events by type" dense> | |
| 164 | + {d.by_type?.length ? ( | |
| 165 | + <ul className="divide-y divide-line"> | |
| 166 | + {d.by_type.map((t) => ( | |
| 167 | + <li key={t.event_type} className="grid grid-cols-[8.5rem_1fr_3rem] items-center gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 168 | + <Link href={`/live?event_type=${t.event_type}&q=${encodeURIComponent(src.name)}`} className="truncate hover:underline">{typeLabel(t.event_type)}</Link> | |
| 169 | + <Bar value={t.n} max={maxType} tone="info" /> | |
| 170 | + <span className="text-right font-mono text-fg-subtle tabular">{fmtInt(t.n)}</span> | |
| 171 | + </li> | |
| 172 | + ))} | |
| 173 | + </ul> | |
| 174 | + ) : ( | |
| 175 | + <Empty>No events yet.</Empty> | |
| 176 | + )} | |
| 177 | + </Panel> | |
| 178 | + | |
| 179 | + <Panel title={`Entities · ${d.entities.length}`} dense> | |
| 110 | 180 | {d.entities.length ? ( |
| 111 | 181 | <ul className="divide-y divide-line"> |
| 112 | 182 | {d.entities.map((e) => ( |
| 113 | − <li key={e.id} className="flex items-center justify-between px-3 py-1.5 text-[13px]"> | |
| 114 | − <Link href={`/company/${e.id}`} className="hover:underline">{e.name}</Link> | |
| 115 | − <span className="font-mono text-[11px] text-fg-subtle">{e.type} · {e.event_count} ev</span> | |
| 183 | + <li key={e.id} className="flex items-center justify-between gap-2 px-3 py-1.5 text-[13px]"> | |
| 184 | + <Link href={`/entity/${e.id}`} className="min-w-0 truncate hover:underline">{e.name}</Link> | |
| 185 | + <span className="whitespace-nowrap font-mono text-[11px] text-fg-subtle">{e.type.replace(/_/g, " ")} · {e.event_count} ev</span> | |
| 116 | 186 | </li> |
| 117 | 187 | ))} |
| 118 | 188 | </ul> |
@@ -120,15 +190,84 @@ export default async function SourcePage({ params }: { params: Promise<{ id: str | ||
| 120 | 190 | <Empty>No entities linked.</Empty> |
| 121 | 191 | )} |
| 122 | 192 | </Panel> |
| 193 | + | |
| 123 | 194 | <Panel title="Source policy"> |
| 124 | 195 | <dl className="grid grid-cols-2 gap-y-1 text-[12.5px]"> |
| 125 | − <dt className="text-fg-subtle">robots.txt checked</dt><dd className="text-right font-mono tabular">{d.source.robots_checked_at ? relTime(d.source.robots_checked_at) : "—"}</dd> | |
| 126 | − <dt className="text-fg-subtle">Importance weight</dt><dd className="text-right font-mono tabular">{d.source.importance_weight ?? 1}</dd> | |
| 196 | + <dt className="text-fg-subtle">Provenance</dt><dd className="text-right">{src.first_party === false ? "third-party media" : "first-party channels"}</dd> | |
| 197 | + <dt className="text-fg-subtle">robots.txt checked</dt><dd className="text-right font-mono tabular" title={src.robots_checked_at ? utcDateTime(src.robots_checked_at) : undefined}>{src.robots_checked_at ? relTime(src.robots_checked_at) : "—"}</dd> | |
| 198 | + <dt className="text-fg-subtle">Importance weight</dt><dd className="text-right font-mono tabular">{src.importance_weight ?? 1}</dd> | |
| 199 | + {extra.llm_enabled !== undefined && ( | |
| 200 | + <> | |
| 201 | + <dt className="text-fg-subtle">LLM interpretation</dt><dd className="text-right font-mono">{extra.llm_enabled ? "on" : "off · heuristics only"}</dd> | |
| 202 | + </> | |
| 203 | + )} | |
| 204 | + {extra.rate_limit_per_min ? ( | |
| 205 | + <> | |
| 206 | + <dt className="text-fg-subtle">Rate limit</dt><dd className="text-right font-mono tabular">{extra.rate_limit_per_min}/min</dd> | |
| 207 | + </> | |
| 208 | + ) : null} | |
| 209 | + {extra.discover && Object.values(extra.discover).some(Boolean) && ( | |
| 210 | + <> | |
| 211 | + <dt className="text-fg-subtle">Discovery</dt><dd className="text-right font-mono">{Object.entries(extra.discover).filter(([, v]) => v).map(([k]) => k).join(" · ")}</dd> | |
| 212 | + </> | |
| 213 | + )} | |
| 214 | + {extra.fallback && Object.values(extra.fallback).some(Boolean) && ( | |
| 215 | + <> | |
| 216 | + <dt className="text-fg-subtle">Fallback</dt><dd className="text-right font-mono">{Object.entries(extra.fallback).filter(([, v]) => v).map(([k]) => k).join(" · ")}</dd> | |
| 217 | + </> | |
| 218 | + )} | |
| 127 | 219 | <dt className="text-fg-subtle">Acquisition</dt><dd className="text-right">official feeds first</dd> |
| 128 | 220 | </dl> |
| 221 | + {src.notes && <p className="mt-3 border-t border-line pt-2 text-[11.5px] leading-snug text-fg-muted break-words">{src.notes}</p>} | |
| 129 | 222 | </Panel> |
| 130 | 223 | </aside> |
| 131 | 224 | </div> |
| 132 | 225 | </> |
| 133 | 226 | ); |
| 134 | 227 | } |
| 228 | + | |
| 229 | +function SensorTr({ s }: { s: SensorRow }) { | |
| 230 | + const runs = s.total_runs ?? 0; | |
| 231 | + const nm = runs ? Math.round(((s.total_not_modified ?? 0) / runs) * 100) : null; | |
| 232 | + const prio = s.priority ?? null; | |
| 233 | + const cond = [s.has_etag || s.etag ? "etag" : null, s.has_last_modified || s.last_modified ? "lm" : null].filter(Boolean); | |
| 234 | + const condTitle = [s.etag ? `ETag: ${s.etag}` : null, s.last_modified ? `Last-Modified: ${s.last_modified}` : null].filter(Boolean).join("\n") || "No conditional-request validators exposed"; | |
| 235 | + return ( | |
| 236 | + <tr className="hover:bg-panel-2/60"> | |
| 237 | + <Td> | |
| 238 | + <Link href={`/sensor/${s.id}`} className="font-medium hover:underline">{s.name}</Link> | |
| 239 | + <div className="max-w-[16rem] truncate font-mono text-[11px] text-fg-subtle" title={s.url}>{s.url}</div> | |
| 240 | + </Td> | |
| 241 | + <Td mono><span className="text-fg-muted">{s.type}</span><div className="text-[10.5px] text-fg-subtle">{s.connector}</div></Td> | |
| 242 | + <Td><TierBadge tier={s.tier} /></Td> | |
| 243 | + <Td>{prio !== null ? <Chip tone={prio === 0 ? "hot" : prio === 1 ? "high" : "default"} className="font-mono" title={PRIORITY_LABELS[prio] ?? `Priority ${prio}`}>P{prio}</Chip> : <span className="text-fg-subtle">—</span>}</Td> | |
| 244 | + <Td>{s.status ? <HealthPill health={s.enabled ? s.status : "DISABLED"} /> : <span className="text-fg-subtle">—</span>}</Td> | |
| 245 | + <Td><HealthPill health={s.enabled ? s.health : "DISABLED"} /></Td> | |
| 246 | + <Td mono className="whitespace-nowrap"> | |
| 247 | + {s.current_interval_seconds ? fmtDuration(s.current_interval_seconds) : <span className="text-fg-subtle">adaptive</span>} | |
| 248 | + <div className="text-[10.5px] text-fg-subtle">{s.base_interval_seconds ? `base ${fmtDuration(s.base_interval_seconds)}` : "no base"}</div> | |
| 249 | + </Td> | |
| 250 | + <Td mono className="whitespace-nowrap text-fg-subtle"> | |
| 251 | + <div title={s.last_check_at ? utcDateTime(s.last_check_at) : undefined}>{s.last_check_at ? relTime(s.last_check_at) : "never"}</div> | |
| 252 | + <div className="text-[10.5px]" title={s.next_check_at ? utcDateTime(s.next_check_at) : undefined}>{untilTime(s.next_check_at)}</div> | |
| 253 | + </Td> | |
| 254 | + <Td mono> | |
| 255 | + {s.last_status ? <span className={s.last_status >= 500 ? "text-danger" : s.last_status >= 400 ? "text-warn" : s.last_status === 304 ? "text-fg-subtle" : ""}>{s.last_status}</span> : <span className="text-fg-subtle">—</span>} | |
| 256 | + {s.last_error ? <div className="max-w-[10rem] truncate text-[10.5px] text-danger" title={s.last_error}>{s.last_error}</div> : null} | |
| 257 | + </Td> | |
| 258 | + <Td> | |
| 259 | + <span className="flex gap-1" title={condTitle}> | |
| 260 | + {cond.length ? cond.map((c) => <Chip key={c} tone="ok" className="font-mono">{c}</Chip>) : <span className="text-fg-subtle">—</span>} | |
| 261 | + </span> | |
| 262 | + </Td> | |
| 263 | + <Td mono className="whitespace-nowrap"> | |
| 264 | + {fmtInt(s.checks_24h ?? 0)} <span className="text-fg-subtle">chk</span> | |
| 265 | + <div className="text-[10.5px]">{s.changes_24h ?? 0} <span className="text-fg-subtle">chg</span></div> | |
| 266 | + </Td> | |
| 267 | + <Td mono>{fmtInt(s.snapshot_count ?? 0)}</Td> | |
| 268 | + <Td mono>{nm !== null ? `${nm}%` : <span className="text-fg-subtle">—</span>}</Td> | |
| 269 | + <Td mono className="whitespace-nowrap">{s.raw_changes ?? 0} / <span className="text-signal">{s.meaningful_changes ?? 0}</span></Td> | |
| 270 | + <Td><Link href={`/sensor/${s.id}`} className="whitespace-nowrap text-[11px] text-info hover:underline">history →</Link></Td> | |
| 271 | + </tr> | |
| 272 | + ); | |
| 273 | +} | |
added
apps/web/src/app/sources/loading.tsx
+32 −0
@@ -0,0 +1,32 @@ | ||
| 1 | +import { Skeleton, SkeletonRows } from "@/components/ui"; | |
| 2 | + | |
| 3 | +export default function Loading() { | |
| 4 | + return ( | |
| 5 | + <> | |
| 6 | + <div className="mb-3"> | |
| 7 | + <Skeleton className="mb-1 h-2.5 w-48" /> | |
| 8 | + <Skeleton className="h-5 w-28" /> | |
| 9 | + <Skeleton className="mt-2 h-3 w-full max-w-2xl" /> | |
| 10 | + </div> | |
| 11 | + <div className="mb-2 flex gap-2"> | |
| 12 | + <Skeleton className="h-8 w-64" /> | |
| 13 | + <Skeleton className="h-8 w-40" /> | |
| 14 | + <Skeleton className="h-8 w-16" /> | |
| 15 | + </div> | |
| 16 | + <div className="mb-2 flex gap-1"> | |
| 17 | + {Array.from({ length: 9 }, (_, i) => ( | |
| 18 | + <Skeleton key={i} className="h-4 w-10" /> | |
| 19 | + ))} | |
| 20 | + </div> | |
| 21 | + <div className="mb-3 flex flex-wrap gap-1"> | |
| 22 | + {Array.from({ length: 16 }, (_, i) => ( | |
| 23 | + <Skeleton key={i} className="h-4 w-16" /> | |
| 24 | + ))} | |
| 25 | + </div> | |
| 26 | + <div className="panel"> | |
| 27 | + <div className="border-b border-line px-3 py-2"><Skeleton className="h-2.5 w-80" /></div> | |
| 28 | + <SkeletonRows rows={16} /> | |
| 29 | + </div> | |
| 30 | + </> | |
| 31 | + ); | |
| 32 | +} | |
modified
apps/web/src/app/sources/page.tsx
+58 −21
@@ -1,48 +1,85 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | 2 | import type { Metadata } from "next"; |
| 3 | −import { Chip, Empty, PageHeader, Panel, Table, Td, TierBadge } from "@/components/ui"; | |
| 3 | +import { Badge, Chip, Empty, Flag, PageHeader, Panel, Table, Td, TierBadge } from "@/components/ui"; | |
| 4 | 4 | import { api } from "@/lib/api"; |
| 5 | 5 | import { fmtInt, relTime } from "@/lib/format"; |
| 6 | 6 | |
| 7 | 7 | export const dynamic = "force-dynamic"; |
| 8 | −export const metadata: Metadata = { title: "Sources", description: "Every organization monitored by WebSensor, with its sensors and recent activity." }; | |
| 8 | +export const metadata: Metadata = { title: "Sources", description: "Every organization monitored by WebSensor, with its sensors and recent activity. Filter by country, tier, category and provenance." }; | |
| 9 | 9 | |
| 10 | −const CATS = ["ai", "cloud", "developer", "cyber", "consumer-tech", "semiconductors", "finance", "government", "statistics", "health", "pharma", "science", "space", "automotive", "commerce", "payments", "crypto", "enterprise", "internet", "standards"]; | |
| 10 | +const CATS = ["ai", "cloud", "developer", "cyber", "consumer-tech", "semiconductors", "finance", "government", "statistics", "health", "pharma", "science", "space", "automotive", "commerce", "payments", "crypto", "enterprise", "internet", "standards", "news", "media", "sports", "transport", "energy"]; | |
| 11 | +const TIERS = ["S", "A", "B", "C", "D"]; | |
| 11 | 12 | |
| 12 | −export default async function SourcesPage({ searchParams }: { searchParams: Promise<{ q?: string; category?: string }> }) { | |
| 13 | − const sp = await searchParams; | |
| 14 | − const { items } = await api.sources({ q: sp.q, category: sp.category }); | |
| 13 | +type Params = { q?: string; category?: string; country?: string; tier?: string; first_party?: string }; | |
| 14 | + | |
| 15 | +function href(p: Params, patch: Partial<Params>): string { | |
| 16 | + const n: Record<string, string | undefined> = { ...p, ...patch }; | |
| 17 | + const sp = new URLSearchParams(); | |
| 18 | + for (const k of ["q", "category", "country", "tier", "first_party"]) if (n[k]) sp.set(k, n[k]!); | |
| 19 | + const s = sp.toString(); | |
| 20 | + return s ? `/sources?${s}` : "/sources"; | |
| 21 | +} | |
| 22 | + | |
| 23 | +export default async function SourcesPage({ searchParams }: { searchParams: Promise<Params> }) { | |
| 24 | + const raw = await searchParams; | |
| 25 | + const sp: Params = { q: raw.q?.trim() || undefined, category: raw.category || undefined, country: raw.country?.toUpperCase() || undefined, tier: TIERS.includes((raw.tier ?? "").toUpperCase()) ? raw.tier!.toUpperCase() : undefined, first_party: raw.first_party === "true" || raw.first_party === "false" ? raw.first_party : undefined }; | |
| 26 | + const [{ items }, countries] = await Promise.all([api.sources({ q: sp.q, category: sp.category, country: sp.country, tier: sp.tier, first_party: sp.first_party }), api.countries()]); | |
| 27 | + const active = [sp.q, sp.category, sp.country, sp.tier, sp.first_party].filter(Boolean).length; | |
| 28 | + const firstParty = items.filter((s) => s.first_party !== false).length; | |
| 15 | 29 | return ( |
| 16 | 30 | <> |
| 17 | − <PageHeader kicker={`${fmtInt(items.length)} organizations`} title="Sources" description="WebSensor monitors sensors, not merely domains: each organization exposes several official endpoints (feeds, status pages, sitemaps, pricing and documentation pages)." /> | |
| 18 | − <form className="mb-3 flex flex-wrap items-center gap-2" action="/sources"> | |
| 19 | − <input name="q" defaultValue={sp.q ?? ""} placeholder="Filter by name or domain…" className="h-8 w-64 rounded-md border border-line bg-panel px-2.5 text-[13px] placeholder:text-fg-subtle" /> | |
| 31 | + <PageHeader compact kicker={`${fmtInt(items.length)} organizations · ${fmtInt(firstParty)} first-party · ${fmtInt(items.length - firstParty)} media`} title="Sources" description="WebSensor monitors sensors, not merely domains: each organization exposes several official endpoints (feeds, status pages, sitemaps, pricing and documentation pages). Media sources are marked EXTERNAL and never count as first-party evidence." /> | |
| 32 | + <form className="mb-2 flex flex-wrap items-center gap-2" action="/sources"> | |
| 33 | + <input name="q" defaultValue={sp.q ?? ""} placeholder="Filter by name or domain…" aria-label="Filter by name or domain" className="h-8 w-full max-w-64 min-w-0 rounded-md border border-line bg-panel px-2.5 text-[13px] placeholder:text-fg-subtle sm:w-64" /> | |
| 34 | + <select name="country" defaultValue={sp.country ?? ""} aria-label="Country" className="h-8 max-w-[14rem] rounded-md border border-line bg-panel px-2 text-[12.5px]"> | |
| 35 | + <option value="">all countries</option> | |
| 36 | + {countries.items.map((c) => ( | |
| 37 | + <option key={c.country} value={c.country}>{c.name} · {c.sources}</option> | |
| 38 | + ))} | |
| 39 | + </select> | |
| 20 | 40 | {sp.category && <input type="hidden" name="category" value={sp.category} />} |
| 21 | − <button type="submit" className="h-8 rounded-md border border-line bg-panel-2 px-3 text-[12.5px]">Filter</button> | |
| 41 | + {sp.tier && <input type="hidden" name="tier" value={sp.tier} />} | |
| 42 | + {sp.first_party && <input type="hidden" name="first_party" value={sp.first_party} />} | |
| 43 | + <button type="submit" className="h-8 rounded-md border border-line bg-panel-2 px-3 text-[12.5px] hover:border-line-strong">Filter</button> | |
| 44 | + {active > 0 && <Link href="/sources" className="text-[12px] text-fg-subtle hover:text-fg">clear {active} filter{active === 1 ? "" : "s"}</Link>} | |
| 22 | 45 | </form> |
| 23 | − <div className="mb-3 flex flex-wrap gap-1"> | |
| 24 | − <Chip href="/sources" tone={!sp.category ? "signal" : "default"}>all</Chip> | |
| 46 | + <div className="mb-2 flex flex-wrap items-center gap-1"> | |
| 47 | + <span className="label mr-1">tier</span> | |
| 48 | + <Chip href={href(sp, { tier: undefined })} tone={!sp.tier ? "signal" : "default"} className="font-mono">all</Chip> | |
| 49 | + {TIERS.map((t) => ( | |
| 50 | + <Chip key={t} href={href(sp, { tier: t })} tone={sp.tier === t ? "signal" : "default"} className="font-mono">{t}</Chip> | |
| 51 | + ))} | |
| 52 | + <span className="label mx-1">·</span> | |
| 53 | + <Chip href={href(sp, { first_party: undefined })} tone={!sp.first_party ? "signal" : "default"}>all kinds</Chip> | |
| 54 | + <Chip href={href(sp, { first_party: "true" })} tone={sp.first_party === "true" ? "signal" : "default"}>first-party</Chip> | |
| 55 | + <Chip href={href(sp, { first_party: "false" })} tone={sp.first_party === "false" ? "signal" : "default"}>media / external</Chip> | |
| 56 | + </div> | |
| 57 | + <div className="mb-3 flex flex-wrap items-center gap-1"> | |
| 58 | + <span className="label mr-1">category</span> | |
| 59 | + <Chip href={href(sp, { category: undefined })} tone={!sp.category ? "signal" : "default"}>all</Chip> | |
| 25 | 60 | {CATS.map((c) => ( |
| 26 | − <Chip key={c} href={`/sources?category=${c}${sp.q ? `&q=${encodeURIComponent(sp.q)}` : ""}`} tone={sp.category === c ? "signal" : "default"}>{c}</Chip> | |
| 61 | + <Chip key={c} href={href(sp, { category: c })} tone={sp.category === c ? "signal" : "default"}>{c}</Chip> | |
| 27 | 62 | ))} |
| 63 | + {sp.category && !CATS.includes(sp.category) && <Chip tone="signal">{sp.category}</Chip>} | |
| 28 | 64 | </div> |
| 29 | 65 | <Panel dense> |
| 30 | 66 | {items.length === 0 ? ( |
| 31 | − <Empty>No sources match.</Empty> | |
| 67 | + <Empty>No source matches these filters.</Empty> | |
| 32 | 68 | ) : ( |
| 33 | − <Table head={["Tier", "Source", "Domain", "Categories", "Sensors", "Events 24h", "Total", "Last event", "Last check", "Health"]}> | |
| 69 | + <Table head={["Tier", "Source", "Domain", "Country", "Kind", "Categories", "Sensors", "Events 24 h", "Total", "Last event", "Health"]}> | |
| 34 | 70 | {items.map((s) => ( |
| 35 | 71 | <tr key={s.id} className="hover:bg-panel-2/60"> |
| 36 | 72 | <Td><TierBadge tier={s.tier} /></Td> |
| 37 | 73 | <Td><Link href={`/source/${s.id}`} className="font-medium hover:underline">{s.name}</Link></Td> |
| 38 | 74 | <Td mono><Link href={`/domain/${s.domain}`} className="text-fg-muted hover:underline">{s.domain}</Link></Td> |
| 39 | − <Td><div className="flex flex-wrap gap-1">{s.categories.slice(0, 3).map((c) => <Chip key={c} href={`/sources?category=${c}`}>{c}</Chip>)}</div></Td> | |
| 75 | + <Td>{s.country ? <Link href={href(sp, { country: s.country })} className="inline-flex items-center gap-1 font-mono text-[11px] text-fg-muted hover:text-fg"><Flag code={s.country} /> {s.country}</Link> : <span className="text-fg-subtle">—</span>}</Td> | |
| 76 | + <Td>{s.first_party === false ? <Badge kind="external" compact /> : <Badge kind="first-party" compact />}</Td> | |
| 77 | + <Td><div className="flex flex-wrap gap-1">{s.categories.slice(0, 3).map((c) => <Chip key={c} href={href(sp, { category: c })}>{c}</Chip>)}{s.categories.length > 3 && <span className="font-mono text-[10.5px] text-fg-subtle">+{s.categories.length - 3}</span>}</div></Td> | |
| 40 | 78 | <Td mono>{s.sensor_count ?? 0}</Td> |
| 41 | − <Td mono>{s.events_24h ?? 0}</Td> | |
| 42 | − <Td mono>{fmtInt(s.event_count)}</Td> | |
| 43 | − <Td mono className="text-fg-subtle">{s.last_event_at ? relTime(s.last_event_at) : "—"}</Td> | |
| 44 | − <Td mono className="text-fg-subtle">{s.last_check_at ? relTime(s.last_check_at) : "—"}</Td> | |
| 45 | − <Td>{(s.sensors_degraded ?? 0) > 0 ? <Chip tone="warn">{s.sensors_degraded} degraded</Chip> : <Chip tone="ok">UP</Chip>}</Td> | |
| 79 | + <Td mono>{fmtInt(s.events_24h ?? 0)}</Td> | |
| 80 | + <Td mono className="text-fg-subtle">{fmtInt(s.event_count)}</Td> | |
| 81 | + <Td mono className="whitespace-nowrap text-fg-subtle">{s.last_event_at ? relTime(s.last_event_at) : "—"}</Td> | |
| 82 | + <Td>{(s.sensors_degraded ?? 0) > 0 ? <Chip tone="warn" className="font-mono">{s.sensors_degraded} DEGRADED</Chip> : <Chip tone="ok" className="font-mono">UP</Chip>}</Td> | |
| 46 | 83 | </tr> |
| 47 | 84 | ))} |
| 48 | 85 | </Table> |
modified
apps/web/src/app/timeline/[id]/page.tsx
+7 −28
@@ -1,33 +1,12 @@ | ||
| 1 | −import Link from "next/link"; | |
| 2 | −import type { Metadata } from "next"; | |
| 3 | −import { notFound } from "next/navigation"; | |
| 4 | −import { Timeline } from "@/components/timeline"; | |
| 5 | −import { PageHeader, Panel } from "@/components/ui"; | |
| 6 | −import { api } from "@/lib/api"; | |
| 1 | +import { permanentRedirect } from "next/navigation"; | |
| 7 | 2 | |
| 8 | 3 | export const dynamic = "force-dynamic"; |
| 9 | 4 | |
| 10 | −export async function generateMetadata({ params }: { params: Promise<{ id: string }> }): Promise<Metadata> { | |
| 5 | +/** `/timeline/:id` lives on the entity page now — permanently redirected to `/entity/:id?tab=timeline`. */ | |
| 6 | +export default async function TimelinePage({ params, searchParams }: { params: Promise<{ id: string }>; searchParams: Promise<{ cursor?: string; range?: string }> }) { | |
| 11 | 7 | const { id } = await params; |
| 12 | − const d = await api.entity(id); | |
| 13 | − return { title: d ? `${d.entity.name} — full timeline` : "Timeline", alternates: { canonical: `/timeline/${id}` } }; | |
| 14 | −} | |
| 15 | − | |
| 16 | −export default async function TimelinePage({ params, searchParams }: { params: Promise<{ id: string }>; searchParams: Promise<{ cursor?: string }> }) { | |
| 17 | − const { id } = await params; | |
| 18 | − const { cursor } = await searchParams; | |
| 19 | − const d = await api.entity(id); | |
| 20 | − if (!d) notFound(); | |
| 21 | − const page = await api.entityTimeline(d.entity.id, cursor, 100); | |
| 22 | − return ( | |
| 23 | − <> | |
| 24 | − <PageHeader kicker={<Link href={`/company/${d.entity.id}`} className="hover:underline">← {d.entity.name}</Link>} title={`${d.entity.name} · timeline`} description="Every meaningful event, newest first. Git history for this organization's public Web." /> | |
| 25 | − <Panel dense> | |
| 26 | − <Timeline events={page.items} showSource /> | |
| 27 | − <div className="flex justify-end px-3 py-2 text-[12px]"> | |
| 28 | − {page.nextCursor && <Link href={`/timeline/${d.entity.id}?cursor=${encodeURIComponent(page.nextCursor)}`} className="rounded-md border border-line bg-panel-2 px-2.5 py-1 hover:border-line-strong">Older →</Link>} | |
| 29 | − </div> | |
| 30 | − </Panel> | |
| 31 | − </> | |
| 32 | − ); | |
| 8 | + const sp = await searchParams; | |
| 9 | + const q = new URLSearchParams({ tab: "timeline", range: sp.range ?? "all" }); | |
| 10 | + if (sp.cursor) q.set("cursor", sp.cursor); | |
| 11 | + permanentRedirect(`/entity/${encodeURIComponent(id)}?${q.toString()}`); | |
| 33 | 12 | } |
modified
apps/web/src/app/watchlists/page.tsx
+2 −2
@@ -2,12 +2,12 @@ import type { Metadata } from "next"; | ||
| 2 | 2 | import { PageHeader } from "@/components/ui"; |
| 3 | 3 | import { Watchlists } from "./watchlists"; |
| 4 | 4 | |
| 5 | −export const metadata: Metadata = { title: "Watchlists", description: "Follow organizations, sources, keywords and categories. Live updates via WebSocket.", robots: { index: false } }; | |
| 5 | +export const metadata: Metadata = { title: "Watchlists", description: "Follow organizations, sources, keywords, categories, URLs, event types, countries and groups. Live updates via WebSocket.", robots: { index: false } }; | |
| 6 | 6 | |
| 7 | 7 | export default function WatchlistsPage() { |
| 8 | 8 | return ( |
| 9 | 9 | <> |
| 10 | − <PageHeader kicker="Stored in this browser (no account yet)" title="Watchlists" description="Follow entities, sources, keywords or categories. Matching events stream in live. Accounts, email and webhook delivery are planned." /> | |
| 10 | + <PageHeader compact kicker="Stored in this browser (no account yet)" title="Watchlists" description="Follow entities, sources, keywords, categories, exact URLs, event types, countries or groups. Matching events stream in live over the watchlist channel; turn a watchlist into deliveries with an alert rule." /> | |
| 11 | 11 | <Watchlists /> |
| 12 | 12 | </> |
| 13 | 13 | ); |
modified
apps/web/src/app/watchlists/watchlists.tsx
+210 −45
@@ -2,24 +2,51 @@ | ||
| 2 | 2 | |
| 3 | 3 | import { Plus, Trash2, X } from "lucide-react"; |
| 4 | 4 | import { useCallback, useEffect, useMemo, useState } from "react"; |
| 5 | +import { COUNTRIES, EVENT_TYPES } from "@websensor/core/client"; | |
| 5 | 6 | import { EventRow } from "@/components/event-row"; |
| 6 | 7 | import { LiveDot } from "@/components/live-feed"; |
| 7 | −import { Chip, Empty, Panel } from "@/components/ui"; | |
| 8 | +import { Chip, Empty, Flag, Panel, SkeletonRows, type Tone } from "@/components/ui"; | |
| 8 | 9 | import { liveToEvent, type EventItem, type LiveEvent, type SearchResult, type Watchlist } from "@/lib/api"; |
| 9 | −import { CHANNEL_KEYS } from "@/lib/format"; | |
| 10 | +import { CHANNEL_KEYS, GROUP_LABELS, hostOf, typeLabel } from "@/lib/format"; | |
| 10 | 11 | import { ownerFetch, publicFetch } from "@/lib/owner"; |
| 11 | 12 | import { useLive } from "@/lib/use-live"; |
| 12 | 13 | |
| 13 | −type Item = { kind: "entity" | "source" | "keyword" | "category"; value: string; label?: string }; | |
| 14 | +type Kind = "entity" | "source" | "keyword" | "category" | "url" | "event_type" | "country" | "group"; | |
| 15 | +type Item = { kind: Kind; value: string }; | |
| 16 | + | |
| 17 | +const KIND_TONE: Record<Kind, Tone> = { entity: "signal", source: "info", category: "ok", keyword: "default", url: "warn", event_type: "high", country: "default", group: "silent" }; | |
| 18 | +const KIND_LABEL: Record<Kind, string> = { entity: "entity", source: "source", keyword: "keyword", category: "category", url: "url", event_type: "type", country: "country", group: "group" }; | |
| 19 | + | |
| 20 | +function itemLabel(i: { kind: string; value: string }): string { | |
| 21 | + switch (i.kind) { | |
| 22 | + case "entity": | |
| 23 | + return i.value.replace(/^(org|prd|ent)_/, ""); | |
| 24 | + case "url": | |
| 25 | + return i.value.length > 48 ? `${hostOf(i.value)}…${i.value.slice(-18)}` : i.value; | |
| 26 | + case "event_type": | |
| 27 | + return EVENT_TYPES[i.value]?.label ?? typeLabel(i.value); | |
| 28 | + case "group": | |
| 29 | + return GROUP_LABELS[i.value] ?? i.value; | |
| 30 | + case "country": | |
| 31 | + return COUNTRIES[i.value]?.name ?? i.value; | |
| 32 | + default: | |
| 33 | + return i.value; | |
| 34 | + } | |
| 35 | +} | |
| 14 | 36 | |
| 15 | 37 | export function Watchlists() { |
| 16 | 38 | const [lists, setLists] = useState<Watchlist[] | null>(null); |
| 17 | 39 | const [active, setActive] = useState<string | null>(null); |
| 18 | − const [events, setEvents] = useState<EventItem[]>([]); | |
| 40 | + /** Keyed by watchlist id: switching lists shows a loading state without a synchronous reset. */ | |
| 41 | + const [feed, setFeed] = useState<{ id: string; items: EventItem[] } | null>(null); | |
| 42 | + const events = feed && feed.id === active ? feed.items : null; | |
| 19 | 43 | const [error, setError] = useState<string | null>(null); |
| 20 | 44 | const [name, setName] = useState(""); |
| 21 | 45 | const [q, setQ] = useState(""); |
| 46 | + const [url, setUrl] = useState(""); | |
| 22 | 47 | const [results, setResults] = useState<SearchResult | null>(null); |
| 48 | + const [typeSel, setTypeSel] = useState(""); | |
| 49 | + const [countrySel, setCountrySel] = useState(""); | |
| 23 | 50 | |
| 24 | 51 | const load = useCallback( |
| 25 | 52 | () => |
@@ -40,7 +67,18 @@ export function Watchlists() { | ||
| 40 | 67 | |
| 41 | 68 | useEffect(() => { |
| 42 | 69 | if (!active) return; |
| 43 | − ownerFetch<{ items: EventItem[] }>(`/api/v1/watchlists/${active}/events?limit=60`).then((r) => setEvents(r.items)).catch(() => setEvents([])); | |
| 70 | + const id = active; | |
| 71 | + let cancelled = false; | |
| 72 | + ownerFetch<{ items: EventItem[] }>(`/api/v1/watchlists/${id}/events?limit=60`) | |
| 73 | + .then((r) => { | |
| 74 | + if (!cancelled) setFeed({ id, items: r.items }); | |
| 75 | + }) | |
| 76 | + .catch(() => { | |
| 77 | + if (!cancelled) setFeed({ id, items: [] }); | |
| 78 | + }); | |
| 79 | + return () => { | |
| 80 | + cancelled = true; | |
| 81 | + }; | |
| 44 | 82 | }, [active, lists]); |
| 45 | 83 | |
| 46 | 84 | useEffect(() => { |
@@ -55,52 +93,76 @@ export function Watchlists() { | ||
| 55 | 93 | }, [q]); |
| 56 | 94 | |
| 57 | 95 | const channels = useMemo(() => (active ? [`watchlist:${active}`] : []), [active]); |
| 58 | − const status = useLive(channels, (e: LiveEvent) => setEvents((prev) => (prev.some((x) => x.id === e.id) ? prev : [liveToEvent(e), ...prev].slice(0, 200)))); | |
| 96 | + const status = useLive(channels, (e: LiveEvent) => setFeed((prev) => (!prev || prev.id !== active || prev.items.some((x) => x.id === e.id) ? prev : { id: prev.id, items: [liveToEvent(e), ...prev.items].slice(0, 200) }))); | |
| 59 | 97 | |
| 60 | 98 | const current = lists?.find((l) => l.id === active) ?? null; |
| 61 | 99 | |
| 62 | 100 | const create = async (): Promise<void> => { |
| 63 | − const wl = await ownerFetch<Watchlist>("/api/v1/watchlists", { method: "POST", body: JSON.stringify({ name: name.trim() || "My watchlist", items: [] }) }); | |
| 64 | − setName(""); | |
| 65 | − await load(); | |
| 66 | − setActive(wl.id); | |
| 101 | + try { | |
| 102 | + const wl = await ownerFetch<Watchlist>("/api/v1/watchlists", { method: "POST", body: JSON.stringify({ name: name.trim() || "My watchlist", items: [] }) }); | |
| 103 | + setName(""); | |
| 104 | + await load(); | |
| 105 | + setActive(wl.id); | |
| 106 | + } catch (e) { | |
| 107 | + setError((e as Error).message); | |
| 108 | + } | |
| 67 | 109 | }; |
| 68 | 110 | const remove = async (id: string): Promise<void> => { |
| 69 | − await ownerFetch(`/api/v1/watchlists/${id}`, { method: "DELETE" }); | |
| 111 | + await ownerFetch(`/api/v1/watchlists/${id}`, { method: "DELETE" }).catch((e: Error) => setError(e.message)); | |
| 70 | 112 | setActive(null); |
| 71 | 113 | await load(); |
| 72 | 114 | }; |
| 73 | 115 | const setItems = async (items: Item[]): Promise<void> => { |
| 74 | 116 | if (!current) return; |
| 75 | − await ownerFetch(`/api/v1/watchlists/${current.id}`, { method: "PUT", body: JSON.stringify({ items: items.map((i) => ({ kind: i.kind, value: i.value })) }) }); | |
| 117 | + try { | |
| 118 | + await ownerFetch(`/api/v1/watchlists/${current.id}`, { method: "PUT", body: JSON.stringify({ items: items.map((i) => ({ kind: i.kind, value: i.value })) }) }); | |
| 119 | + setError(null); | |
| 120 | + } catch (e) { | |
| 121 | + setError((e as Error).message); | |
| 122 | + } | |
| 76 | 123 | await load(); |
| 77 | 124 | }; |
| 125 | + const currentItems = (): Item[] => (current?.items ?? []).map((i) => ({ kind: i.kind as Kind, value: i.value })); | |
| 78 | 126 | const add = (it: Item): void => { |
| 79 | 127 | if (!current) return; |
| 80 | − const items = current.items.map((i) => ({ kind: i.kind as Item["kind"], value: i.value })); | |
| 128 | + const items = currentItems(); | |
| 81 | 129 | if (!items.some((i) => i.kind === it.kind && i.value === it.value)) void setItems([...items, it]); |
| 82 | 130 | setQ(""); |
| 83 | 131 | }; |
| 84 | 132 | const del = (it: { kind: string; value: string }): void => { |
| 85 | 133 | if (!current) return; |
| 86 | − void setItems(current.items.filter((i) => !(i.kind === it.kind && i.value === it.value)).map((i) => ({ kind: i.kind as Item["kind"], value: i.value }))); | |
| 134 | + void setItems(currentItems().filter((i) => !(i.kind === it.kind && i.value === it.value))); | |
| 135 | + }; | |
| 136 | + const addUrl = (): void => { | |
| 137 | + const v = url.trim(); | |
| 138 | + if (!/^https?:\/\//i.test(v)) { | |
| 139 | + setError("URL must start with http:// or https://"); | |
| 140 | + return; | |
| 141 | + } | |
| 142 | + add({ kind: "url", value: v }); | |
| 143 | + setUrl(""); | |
| 87 | 144 | }; |
| 88 | 145 | |
| 146 | + const selectCls = "h-8 min-w-0 flex-1 rounded-md border border-line bg-panel px-2 text-[12.5px]"; | |
| 147 | + const addBtn = "inline-flex h-8 shrink-0 items-center gap-1 rounded-md border border-line bg-panel-2 px-2 text-[12px] hover:border-line-strong disabled:opacity-50"; | |
| 148 | + | |
| 89 | 149 | return ( |
| 90 | − <div className="grid gap-4 lg:grid-cols-[300px_1fr]"> | |
| 91 | − <aside className="flex flex-col gap-4"> | |
| 150 | + <div className="grid gap-4 lg:grid-cols-[320px_minmax(0,1fr)]"> | |
| 151 | + <aside className="flex min-w-0 flex-col gap-4"> | |
| 92 | 152 | <Panel title="Your watchlists" dense> |
| 93 | 153 | {lists === null ? ( |
| 94 | − <Empty>Loading…</Empty> | |
| 154 | + <SkeletonRows rows={2} /> | |
| 95 | 155 | ) : ( |
| 96 | 156 | <ul className="divide-y divide-line"> |
| 97 | 157 | {lists.map((l) => ( |
| 98 | − <li key={l.id} className={`flex items-center justify-between px-3 py-1.5 text-[13px] ${l.id === active ? "bg-panel-2" : ""}`}> | |
| 99 | − <button type="button" onClick={() => setActive(l.id)} className="min-w-0 flex-1 truncate text-left hover:underline">{l.name} <span className="text-fg-subtle">· {l.items.length}</span></button> | |
| 100 | − <button type="button" aria-label="Delete" onClick={() => remove(l.id)} className="text-fg-subtle hover:text-danger"><Trash2 className="size-3.5" /></button> | |
| 158 | + <li key={l.id} className={`flex items-center justify-between gap-2 px-3 py-1.5 text-[13px] ${l.id === active ? "bg-panel-2" : ""}`}> | |
| 159 | + <button type="button" aria-pressed={l.id === active} onClick={() => setActive(l.id)} className="min-w-0 flex-1 truncate text-left hover:underline"> | |
| 160 | + {l.name} <span className="font-mono text-[11px] text-fg-subtle">· {l.items.length}</span> | |
| 161 | + </button> | |
| 162 | + <button type="button" aria-label={`Delete watchlist ${l.name}`} onClick={() => remove(l.id)} className="text-fg-subtle hover:text-danger"><Trash2 className="size-3.5" /></button> | |
| 101 | 163 | </li> |
| 102 | 164 | ))} |
| 103 | − {lists.length === 0 && <Empty>No watchlist yet.</Empty>} | |
| 165 | + {lists.length === 0 && <Empty>No watchlist yet — create one below.</Empty>} | |
| 104 | 166 | </ul> |
| 105 | 167 | )} |
| 106 | 168 | <form |
@@ -110,44 +172,147 @@ export function Watchlists() { | ||
| 110 | 172 | void create(); |
| 111 | 173 | }} |
| 112 | 174 | > |
| 113 | − <input value={name} onChange={(e) => setName(e.target.value)} placeholder="New watchlist name" className="h-8 min-w-0 flex-1 rounded-md border border-line bg-panel px-2 text-[12.5px]" /> | |
| 114 | − <button type="submit" className="inline-flex h-8 items-center gap-1 rounded-md border border-line bg-panel-2 px-2 text-[12px]"><Plus className="size-3.5" /> Add</button> | |
| 175 | + <input value={name} onChange={(e) => setName(e.target.value)} placeholder="New watchlist name" aria-label="New watchlist name" className={selectCls} /> | |
| 176 | + <button type="submit" className={addBtn}><Plus className="size-3.5" /> Add</button> | |
| 115 | 177 | </form> |
| 116 | 178 | {error && <p className="px-3 pb-2 text-[11px] text-danger">{error}</p>} |
| 117 | 179 | </Panel> |
| 180 | + | |
| 118 | 181 | {current && ( |
| 119 | − <Panel title={`Items · ${current.items.length}`}> | |
| 120 | − <div className="mb-2 flex flex-wrap gap-1"> | |
| 182 | + <Panel title={<span>Items <span className="font-mono text-fg-subtle">{current.items.length}</span></span>}> | |
| 183 | + <div className="mb-3 flex flex-wrap gap-1"> | |
| 121 | 184 | {current.items.map((i) => ( |
| 122 | − <Chip key={`${i.kind}:${i.value}`} tone={i.kind === "entity" ? "signal" : i.kind === "source" ? "info" : i.kind === "category" ? "ok" : "default"}> | |
| 123 | − <span className="text-fg-subtle">{i.kind}</span> {i.value.replace(/^(org|prd)_/, "")} | |
| 124 | − <button type="button" aria-label="Remove" onClick={() => del(i)} className="ml-0.5 hover:text-danger"><X className="size-3" /></button> | |
| 185 | + <Chip key={`${i.kind}:${i.value}`} tone={KIND_TONE[i.kind as Kind] ?? "default"} title={`${i.kind}: ${i.value}`}> | |
| 186 | + <span className="font-mono text-[9.5px] uppercase tracking-wider opacity-70">{KIND_LABEL[i.kind as Kind] ?? i.kind}</span> | |
| 187 | + {i.kind === "country" && <Flag code={i.value} />} | |
| 188 | + <span className="max-w-[14rem] truncate">{itemLabel(i)}</span> | |
| 189 | + <button type="button" aria-label={`Remove ${i.kind} ${i.value}`} onClick={() => del(i)} className="ml-0.5 hover:text-danger"><X className="size-3" /></button> | |
| 125 | 190 | </Chip> |
| 126 | 191 | ))} |
| 127 | − {current.items.length === 0 && <span className="text-[12px] text-fg-subtle">Empty — add entities, sources, keywords or categories.</span>} | |
| 192 | + {current.items.length === 0 && <span className="text-[12px] text-fg-subtle">Empty — add entities, sources, keywords, categories, URLs, event types, countries or groups.</span>} | |
| 128 | 193 | </div> |
| 129 | − <input value={q} onChange={(e) => setQ(e.target.value)} placeholder="Search entities / sources, or type a keyword…" className="h-8 w-full rounded-md border border-line bg-panel px-2 text-[12.5px]" /> | |
| 130 | − {q.trim().length >= 2 && ( | |
| 131 | − <div className="mt-1 max-h-64 overflow-auto rounded-md border border-line bg-panel text-[12.5px]"> | |
| 132 | − <button type="button" onClick={() => add({ kind: "keyword", value: q.trim().toLowerCase() })} className="block w-full px-2 py-1.5 text-left hover:bg-panel-2">keyword “{q.trim()}”</button> | |
| 133 | − {results?.entities.map((e) => ( | |
| 134 | − <button key={e.id} type="button" onClick={() => add({ kind: "entity", value: e.id })} className="block w-full px-2 py-1.5 text-left hover:bg-panel-2">entity · {e.name} <span className="text-fg-subtle">{e.type}</span></button> | |
| 135 | − ))} | |
| 136 | − {results?.sources.map((s) => ( | |
| 137 | − <button key={s.id} type="button" onClick={() => add({ kind: "source", value: s.id })} className="block w-full px-2 py-1.5 text-left hover:bg-panel-2">source · {s.name} <span className="text-fg-subtle">{s.domain}</span></button> | |
| 138 | − ))} | |
| 194 | + | |
| 195 | + <div className="flex flex-col gap-3 text-[12.5px]"> | |
| 196 | + <div> | |
| 197 | + <div className="label mb-1">Entity · source · keyword</div> | |
| 198 | + <input value={q} onChange={(e) => setQ(e.target.value)} placeholder="Search entities / sources, or type a keyword…" aria-label="Search entities, sources or keyword" className="h-8 w-full rounded-md border border-line bg-panel px-2 text-[12.5px]" /> | |
| 199 | + {q.trim().length >= 2 && ( | |
| 200 | + <div className="mt-1 max-h-64 overflow-auto rounded-md border border-line bg-panel text-[12.5px]"> | |
| 201 | + <button type="button" onClick={() => add({ kind: "keyword", value: q.trim().toLowerCase() })} className="block w-full px-2 py-1.5 text-left hover:bg-panel-2"><span className="label mr-1 !text-[9.5px]">keyword</span> “{q.trim()}”</button> | |
| 202 | + {results?.entities.map((e) => ( | |
| 203 | + <button key={e.id} type="button" onClick={() => add({ kind: "entity", value: e.id })} className="block w-full truncate px-2 py-1.5 text-left hover:bg-panel-2"><span className="label mr-1 !text-[9.5px] !text-signal">entity</span> {e.name} <span className="text-fg-subtle">{e.type}</span></button> | |
| 204 | + ))} | |
| 205 | + {results?.sources.map((s) => ( | |
| 206 | + <button key={s.id} type="button" onClick={() => add({ kind: "source", value: s.id })} className="block w-full truncate px-2 py-1.5 text-left hover:bg-panel-2"><span className="label mr-1 !text-[9.5px] !text-info">source</span> {s.name} <span className="text-fg-subtle">{s.domain}</span></button> | |
| 207 | + ))} | |
| 208 | + {results && !results.entities.length && !results.sources.length && <div className="px-2 py-1.5 text-[11.5px] text-fg-subtle">No entity or source matches — add as a keyword.</div>} | |
| 209 | + </div> | |
| 210 | + )} | |
| 211 | + </div> | |
| 212 | + | |
| 213 | + <div> | |
| 214 | + <div className="label mb-1">URL</div> | |
| 215 | + <form | |
| 216 | + className="flex gap-1" | |
| 217 | + onSubmit={(e) => { | |
| 218 | + e.preventDefault(); | |
| 219 | + addUrl(); | |
| 220 | + }} | |
| 221 | + > | |
| 222 | + <input type="url" value={url} onChange={(e) => setUrl(e.target.value)} placeholder="https://example.com/pricing" aria-label="URL to watch" className={`${selectCls} font-mono text-[12px]`} /> | |
| 223 | + <button type="submit" disabled={!url.trim()} className={addBtn}><Plus className="size-3.5" /></button> | |
| 224 | + </form> | |
| 225 | + <p className="mt-1 text-[11px] text-fg-subtle">Matches events whose monitored URL equals this address. To watch a page WebSensor does not cover yet, create a <a href="/monitors" className="text-info hover:underline">custom monitor</a>.</p> | |
| 226 | + </div> | |
| 227 | + | |
| 228 | + <div> | |
| 229 | + <div className="label mb-1">Event type</div> | |
| 230 | + <div className="flex gap-1"> | |
| 231 | + <select value={typeSel} onChange={(e) => setTypeSel(e.target.value)} aria-label="Event type" className={selectCls}> | |
| 232 | + <option value="">Choose an event type…</option> | |
| 233 | + {Object.entries(EVENT_TYPES).map(([k, v]) => ( | |
| 234 | + <option key={k} value={k}>{v.label}</option> | |
| 235 | + ))} | |
| 236 | + </select> | |
| 237 | + <button | |
| 238 | + type="button" | |
| 239 | + disabled={!typeSel} | |
| 240 | + onClick={() => { | |
| 241 | + if (typeSel) add({ kind: "event_type", value: typeSel }); | |
| 242 | + setTypeSel(""); | |
| 243 | + }} | |
| 244 | + className={addBtn} | |
| 245 | + > | |
| 246 | + <Plus className="size-3.5" /> | |
| 247 | + </button> | |
| 248 | + </div> | |
| 249 | + </div> | |
| 250 | + | |
| 251 | + <div> | |
| 252 | + <div className="label mb-1">Country</div> | |
| 253 | + <div className="flex gap-1"> | |
| 254 | + <select value={countrySel} onChange={(e) => setCountrySel(e.target.value)} aria-label="Country" className={selectCls}> | |
| 255 | + <option value="">Choose a country…</option> | |
| 256 | + {Object.entries(COUNTRIES).map(([code, c]) => ( | |
| 257 | + <option key={code} value={code}>{c.flag} {c.name} ({code})</option> | |
| 258 | + ))} | |
| 259 | + </select> | |
| 260 | + <button | |
| 261 | + type="button" | |
| 262 | + disabled={!countrySel} | |
| 263 | + onClick={() => { | |
| 264 | + if (countrySel) add({ kind: "country", value: countrySel }); | |
| 265 | + setCountrySel(""); | |
| 266 | + }} | |
| 267 | + className={addBtn} | |
| 268 | + > | |
| 269 | + <Plus className="size-3.5" /> | |
| 270 | + </button> | |
| 271 | + </div> | |
| 272 | + </div> | |
| 273 | + | |
| 274 | + <div> | |
| 275 | + <div className="label mb-1">Group</div> | |
| 276 | + <div className="flex flex-wrap gap-1"> | |
| 277 | + {Object.entries(GROUP_LABELS).map(([k, v]) => { | |
| 278 | + const on = current.items.some((i) => i.kind === "group" && i.value === k); | |
| 279 | + return ( | |
| 280 | + <button key={k} type="button" aria-pressed={on} disabled={on} onClick={() => add({ kind: "group", value: k })} className={`rounded-sm border px-1.5 py-px text-[10.5px] leading-4 ${on ? "border-silent/40 bg-silent-soft text-silent" : "border-line text-fg-muted hover:text-fg"}`}> | |
| 281 | + {on ? "" : "+ "}{v} | |
| 282 | + </button> | |
| 283 | + ); | |
| 284 | + })} | |
| 285 | + </div> | |
| 286 | + </div> | |
| 287 | + | |
| 288 | + <div> | |
| 289 | + <div className="label mb-1">Category</div> | |
| 290 | + <div className="flex flex-wrap gap-1"> | |
| 291 | + {CHANNEL_KEYS.map((c) => { | |
| 292 | + const on = current.items.some((i) => i.kind === "category" && i.value === c); | |
| 293 | + return ( | |
| 294 | + <button key={c} type="button" aria-pressed={on} disabled={on} onClick={() => add({ kind: "category", value: c })} className={`rounded-sm border px-1.5 py-px text-[10.5px] leading-4 ${on ? "border-ok/40 bg-ok/10 text-ok" : "border-line text-fg-muted hover:text-fg"}`}> | |
| 295 | + {on ? "" : "+ "}{c} | |
| 296 | + </button> | |
| 297 | + ); | |
| 298 | + })} | |
| 299 | + </div> | |
| 139 | 300 | </div> |
| 140 | − )} | |
| 141 | − <div className="mt-2 flex flex-wrap gap-1"> | |
| 142 | − {CHANNEL_KEYS.map((c) => ( | |
| 143 | − <button key={c} type="button" onClick={() => add({ kind: "category", value: c })} className="rounded-sm border border-line px-1.5 py-px text-[10.5px] text-fg-muted hover:text-fg">+ {c}</button> | |
| 144 | − ))} | |
| 145 | 301 | </div> |
| 146 | 302 | </Panel> |
| 147 | 303 | )} |
| 148 | 304 | </aside> |
| 305 | + | |
| 149 | 306 | <Panel title={<span className="flex items-center gap-3">{current ? current.name : "Events"} {active && <LiveDot status={status} />}</span>} dense> |
| 150 | − {!current ? <Empty>Select or create a watchlist.</Empty> : events.length ? events.map((e) => <EventRow key={e.id} ev={e} showDate />) : <Empty>No events match this watchlist yet — new ones will appear live.</Empty>} | |
| 307 | + {!current ? ( | |
| 308 | + <Empty>{lists && lists.length === 0 ? "Create a watchlist to start following entities, sources, URLs, types, countries or groups." : "Select a watchlist."}</Empty> | |
| 309 | + ) : events === null ? ( | |
| 310 | + <SkeletonRows rows={6} /> | |
| 311 | + ) : events.length ? ( | |
| 312 | + events.map((e) => <EventRow key={e.id} ev={e} showDate />) | |
| 313 | + ) : ( | |
| 314 | + <Empty>{current.items.length === 0 ? "This watchlist is empty — add items on the left." : "No events match this watchlist yet — new ones will appear live."}</Empty> | |
| 315 | + )} | |
| 151 | 316 | </Panel> |
| 152 | 317 | </div> |
| 153 | 318 | ); |
added
apps/web/src/components/command-palette.tsx
+205 −0
@@ -0,0 +1,205 @@ | ||
| 1 | +"use client"; | |
| 2 | + | |
| 3 | +import { useRouter } from "next/navigation"; | |
| 4 | +import { Activity, Building2, Compass, Globe2, Layers, Radar, Search, Siren, Zap } from "lucide-react"; | |
| 5 | +import { useCallback, useEffect, useMemo, useRef, useState, type ReactNode } from "react"; | |
| 6 | +import type { SearchResult } from "@/lib/api"; | |
| 7 | +import { publicFetch } from "@/lib/owner"; | |
| 8 | +import { useEventDrawer } from "./event-drawer"; | |
| 9 | +import { usePrefs, type Density } from "./prefs"; | |
| 10 | +import { Chip, Kbd, Score } from "./ui"; | |
| 11 | + | |
| 12 | +/** | |
| 13 | + * Global command palette (spec §29): ⌘K / Ctrl+K. Searches entities, sources, events, clusters, | |
| 14 | + * URLs and offers navigation + preference commands. Fully keyboard-driven (↑ ↓ ↵ Esc), ARIA listbox. | |
| 15 | + */ | |
| 16 | +interface Item { | |
| 17 | + id: string; | |
| 18 | + group: string; | |
| 19 | + label: string; | |
| 20 | + hint?: string; | |
| 21 | + icon?: ReactNode; | |
| 22 | + score?: number | null; | |
| 23 | + run: () => void; | |
| 24 | +} | |
| 25 | + | |
| 26 | +const NAV: { label: string; href: string; hint: string; icon: ReactNode }[] = [ | |
| 27 | + { label: "Live feed", href: "/live", hint: "everything, live", icon: <Activity className="size-3.5" /> }, | |
| 28 | + { label: "Breaking", href: "/breaking", hint: "breaking · developing · confirmed", icon: <Siren className="size-3.5" /> }, | |
| 29 | + { label: "Pulse", href: "/pulse", hint: "what is changing on the Internet right now", icon: <Zap className="size-3.5" /> }, | |
| 30 | + { label: "Radar", href: "/radar", hint: "weak signals, not yet breaking", icon: <Radar className="size-3.5" /> }, | |
| 31 | + { label: "Silent changes", href: "/silent", hint: "modified without announcement", icon: <Layers className="size-3.5" /> }, | |
| 32 | + { label: "Explore", href: "/explore", hint: "trending · unusual · entities · sources", icon: <Compass className="size-3.5" /> }, | |
| 33 | + { label: "Entities", href: "/entities", hint: "organizations, products, models", icon: <Building2 className="size-3.5" /> }, | |
| 34 | + { label: "Countries", href: "/country", hint: "national desks", icon: <Globe2 className="size-3.5" /> }, | |
| 35 | + { label: "Sources", href: "/sources", hint: "monitored organizations", icon: <Building2 className="size-3.5" /> }, | |
| 36 | + { label: "Watchlists", href: "/watchlists", hint: "your entities, sources, keywords", icon: <Layers className="size-3.5" /> }, | |
| 37 | + { label: "Alerts", href: "/alerts", hint: "rules · webhooks", icon: <Siren className="size-3.5" /> }, | |
| 38 | + { label: "Bookmarks", href: "/bookmarks", hint: "saved events", icon: <Layers className="size-3.5" /> }, | |
| 39 | + { label: "Health", href: "/health", hint: "sensors, connectors, throughput", icon: <Activity className="size-3.5" /> }, | |
| 40 | + { label: "API", href: "/api", hint: "REST · RSS · WebSocket", icon: <Zap className="size-3.5" /> }, | |
| 41 | +]; | |
| 42 | + | |
| 43 | +export function CommandPalette() { | |
| 44 | + const [open, setOpen] = useState(false); | |
| 45 | + const [q, setQ] = useState(""); | |
| 46 | + const [res, setRes] = useState<SearchResult | null>(null); | |
| 47 | + const [busy, setBusy] = useState(false); | |
| 48 | + const [idx, setIdx] = useState(0); | |
| 49 | + const input = useRef<HTMLInputElement>(null); | |
| 50 | + const router = useRouter(); | |
| 51 | + const drawer = useEventDrawer(); | |
| 52 | + const { setDensity } = usePrefs(); | |
| 53 | + | |
| 54 | + const openPalette = (): void => { | |
| 55 | + setQ(""); | |
| 56 | + setRes(null); | |
| 57 | + setIdx(0); | |
| 58 | + setOpen(true); | |
| 59 | + setTimeout(() => input.current?.focus(), 10); | |
| 60 | + }; | |
| 61 | + useEffect(() => { | |
| 62 | + const onKey = (e: KeyboardEvent): void => { | |
| 63 | + if ((e.metaKey || e.ctrlKey) && e.key.toLowerCase() === "k") { | |
| 64 | + e.preventDefault(); | |
| 65 | + if (open) setOpen(false); | |
| 66 | + else openPalette(); | |
| 67 | + } else if (e.key === "/" && !open && !(e.target instanceof HTMLInputElement || e.target instanceof HTMLTextAreaElement)) { | |
| 68 | + e.preventDefault(); | |
| 69 | + openPalette(); | |
| 70 | + } | |
| 71 | + }; | |
| 72 | + window.addEventListener("keydown", onKey); | |
| 73 | + const onOpen = (): void => openPalette(); | |
| 74 | + window.addEventListener("ws:palette", onOpen); | |
| 75 | + return () => { | |
| 76 | + window.removeEventListener("keydown", onKey); | |
| 77 | + window.removeEventListener("ws:palette", onOpen); | |
| 78 | + }; | |
| 79 | + }, [open]); | |
| 80 | + useEffect(() => { | |
| 81 | + if (!open) return; | |
| 82 | + const t = q.trim(); | |
| 83 | + const h = setTimeout(() => { | |
| 84 | + if (t.length < 2) { | |
| 85 | + setRes(null); | |
| 86 | + return; | |
| 87 | + } | |
| 88 | + setBusy(true); | |
| 89 | + publicFetch<SearchResult>(`/api/v1/search?q=${encodeURIComponent(t)}&limit=6`) | |
| 90 | + .then((r) => setRes(r)) | |
| 91 | + .catch(() => setRes(null)) | |
| 92 | + .finally(() => setBusy(false)); | |
| 93 | + }, 160); | |
| 94 | + return () => clearTimeout(h); | |
| 95 | + }, [q, open]); | |
| 96 | + | |
| 97 | + const go = useCallback( | |
| 98 | + (href: string) => { | |
| 99 | + setOpen(false); | |
| 100 | + router.push(href); | |
| 101 | + }, | |
| 102 | + [router], | |
| 103 | + ); | |
| 104 | + | |
| 105 | + const items = useMemo<Item[]>(() => { | |
| 106 | + const t = q.trim().toLowerCase(); | |
| 107 | + const out: Item[] = []; | |
| 108 | + if (res) { | |
| 109 | + for (const e of res.entities.slice(0, 5)) out.push({ id: `ent:${e.id}`, group: "Entities", label: e.name, hint: `${e.type}${e.domain ? " · " + e.domain : ""}`, icon: <Building2 className="size-3.5" />, run: () => go(`/entity/${e.id}`) }); | |
| 110 | + for (const s of res.sources.slice(0, 5)) out.push({ id: `src:${s.id}`, group: "Sources", label: s.name, hint: `${s.domain} · tier ${s.tier}${s.first_party === false ? " · media" : ""}`, icon: <Globe2 className="size-3.5" />, run: () => go(`/source/${s.id}`) }); | |
| 111 | + for (const c of res.clusters?.slice(0, 3) ?? []) out.push({ id: `clu:${c.id}`, group: "Clusters", label: c.title, hint: `${c.event_count} signals · ${c.source_count} sources · ${c.state}`, icon: <Layers className="size-3.5" />, score: c.max_importance, run: () => go(`/cluster/${c.slug ?? c.id}`) }); | |
| 112 | + for (const e of res.events.slice(0, 6)) out.push({ id: `evt:${e.id}`, group: "Events", label: e.title, hint: `${e.source?.name} · ${e.event_type.replace(/_/g, " ")}`, icon: <Activity className="size-3.5" />, score: e.signal_score ?? e.importance, run: () => { setOpen(false); drawer.open(e.slug, e); } }); | |
| 113 | + for (const u of res.urls.slice(0, 3)) out.push({ id: `url:${u.url}`, group: "URLs", label: u.url, hint: `${u.change_count} changes`, icon: <Globe2 className="size-3.5" />, run: () => go(`/url?u=${encodeURIComponent(u.url)}`) }); | |
| 114 | + } | |
| 115 | + if (t.length >= 2) out.push({ id: "search", group: "Search", label: `Search events for “${q.trim()}”`, hint: "supports entity: type: after: silent: importance:>", icon: <Search className="size-3.5" />, run: () => go(`/search?q=${encodeURIComponent(q.trim())}`) }); | |
| 116 | + const nav = NAV.filter((n) => !t || n.label.toLowerCase().includes(t) || n.hint.includes(t) || n.href.includes(t)).slice(0, t ? 5 : 14); | |
| 117 | + for (const n of nav) out.push({ id: `nav:${n.href}`, group: "Go to", label: n.label, hint: n.hint, icon: n.icon, run: () => go(n.href) }); | |
| 118 | + const cmds: { label: string; hint: string; run: () => void; match: string }[] = [ | |
| 119 | + { label: "Density: compact", hint: "denser rows", match: "density compact", run: () => setDensity("compact" as Density) }, | |
| 120 | + { label: "Density: normal", hint: "default rows", match: "density normal", run: () => setDensity("normal" as Density) }, | |
| 121 | + { label: "Density: comfortable", hint: "roomier rows with summaries", match: "density comfortable", run: () => setDensity("comfortable" as Density) }, | |
| 122 | + { label: "Toggle theme", hint: "dark / light", match: "theme dark light", run: () => document.documentElement.classList.toggle("dark") }, | |
| 123 | + ]; | |
| 124 | + for (const c of cmds.filter((c) => !t || c.match.includes(t) || c.label.toLowerCase().includes(t)).slice(0, t ? 3 : 4)) out.push({ id: `cmd:${c.label}`, group: "Commands", label: c.label, hint: c.hint, icon: <Zap className="size-3.5" />, run: () => { c.run(); setOpen(false); } }); | |
| 125 | + return out; | |
| 126 | + }, [q, res, go, drawer, setDensity]); | |
| 127 | + | |
| 128 | + // keep the highlighted row valid when the result list shrinks | |
| 129 | + const safeIdx = Math.min(idx, Math.max(0, items.length - 1)); | |
| 130 | + | |
| 131 | + if (!open) return null; | |
| 132 | + const groups = [...new Set(items.map((i) => i.group))]; | |
| 133 | + let flat = -1; | |
| 134 | + return ( | |
| 135 | + <div className="fixed inset-0 z-[60] flex items-start justify-center px-3 pt-[10vh]" role="dialog" aria-modal="true" aria-label="Command palette"> | |
| 136 | + <button type="button" aria-label="Close" onClick={() => setOpen(false)} className="scrim absolute inset-0 cursor-default" /> | |
| 137 | + <div className="relative w-full max-w-xl overflow-hidden rounded-lg border border-line bg-panel shadow-2xl animate-fade-in"> | |
| 138 | + <div className="flex items-center gap-2 border-b border-line px-3"> | |
| 139 | + <Search className="size-4 text-fg-subtle" /> | |
| 140 | + <input | |
| 141 | + ref={input} | |
| 142 | + value={q} | |
| 143 | + onChange={(e) => { | |
| 144 | + setQ(e.target.value); | |
| 145 | + setIdx(0); | |
| 146 | + }} | |
| 147 | + onKeyDown={(e) => { | |
| 148 | + if (e.key === "ArrowDown") { | |
| 149 | + e.preventDefault(); | |
| 150 | + setIdx((i) => Math.min(items.length - 1, i + 1)); | |
| 151 | + } else if (e.key === "ArrowUp") { | |
| 152 | + e.preventDefault(); | |
| 153 | + setIdx((i) => Math.max(0, i - 1)); | |
| 154 | + } else if (e.key === "Enter") { | |
| 155 | + e.preventDefault(); | |
| 156 | + items[safeIdx]?.run(); | |
| 157 | + } else if (e.key === "Escape") setOpen(false); | |
| 158 | + }} | |
| 159 | + placeholder="Search entities, sources, events, URLs — or jump to a page…" | |
| 160 | + className="h-11 w-full bg-transparent text-[14px] outline-none placeholder:text-fg-subtle" | |
| 161 | + role="combobox" | |
| 162 | + aria-expanded | |
| 163 | + aria-controls="palette-list" | |
| 164 | + aria-activedescendant={items[safeIdx] ? `pal-${items[safeIdx].id}` : undefined} | |
| 165 | + autoComplete="off" | |
| 166 | + /> | |
| 167 | + {busy ? <span className="size-3 animate-pulse rounded-full bg-signal" /> : <Kbd>esc</Kbd>} | |
| 168 | + </div> | |
| 169 | + <ul id="palette-list" role="listbox" className="max-h-[60vh] overflow-y-auto py-1"> | |
| 170 | + {groups.map((g) => ( | |
| 171 | + <li key={g} role="presentation"> | |
| 172 | + <div className="label px-3 pb-0.5 pt-2">{g}</div> | |
| 173 | + <ul role="group"> | |
| 174 | + {items | |
| 175 | + .filter((i) => i.group === g) | |
| 176 | + .map((it) => { | |
| 177 | + flat++; | |
| 178 | + const active = flat === safeIdx; | |
| 179 | + const my = flat; | |
| 180 | + return ( | |
| 181 | + <li key={it.id} id={`pal-${it.id}`} role="option" aria-selected={active} onMouseEnter={() => setIdx(my)} onClick={it.run} className={`flex cursor-pointer items-center gap-2.5 px-3 py-1.5 text-[13px] ${active ? "bg-panel-2" : ""}`}> | |
| 182 | + <span className="text-fg-subtle">{it.icon}</span> | |
| 183 | + <span className="min-w-0 flex-1"> | |
| 184 | + <span className="block truncate">{it.label}</span> | |
| 185 | + {it.hint && <span className="block truncate text-[11px] text-fg-subtle">{it.hint}</span>} | |
| 186 | + </span> | |
| 187 | + {it.score !== undefined && it.score !== null && <Score value={it.score} size="sm" kind="signal" />} | |
| 188 | + {active && <Kbd>↵</Kbd>} | |
| 189 | + </li> | |
| 190 | + ); | |
| 191 | + })} | |
| 192 | + </ul> | |
| 193 | + </li> | |
| 194 | + ))} | |
| 195 | + {!items.length && <li className="px-3 py-6 text-center text-[13px] text-fg-subtle">Nothing matches.</li>} | |
| 196 | + </ul> | |
| 197 | + <div className="flex items-center gap-3 border-t border-line px-3 py-1.5 text-[11px] text-fg-subtle"> | |
| 198 | + <span className="inline-flex items-center gap-1"><Kbd>↑</Kbd><Kbd>↓</Kbd> navigate</span> | |
| 199 | + <span className="inline-flex items-center gap-1"><Kbd>↵</Kbd> open</span> | |
| 200 | + <span className="ml-auto inline-flex items-center gap-1"><Chip>entity:openai</Chip><Chip>type:pricing_change</Chip><Chip>silent:true</Chip></span> | |
| 201 | + </div> | |
| 202 | + </div> | |
| 203 | + </div> | |
| 204 | + ); | |
| 205 | +} | |
added
apps/web/src/components/event-drawer.tsx
+330 −0
@@ -0,0 +1,330 @@ | ||
| 1 | +"use client"; | |
| 2 | + | |
| 3 | +import Link from "next/link"; | |
| 4 | +import { usePathname } from "next/navigation"; | |
| 5 | +import { Bookmark, BookmarkCheck, ExternalLink, Maximize2, X } from "lucide-react"; | |
| 6 | +import { createContext, useCallback, useContext, useEffect, useMemo, useRef, useState, type ReactNode } from "react"; | |
| 7 | +import type { EventDetail, EventItem } from "@/lib/api"; | |
| 8 | +import { fmtMs, fmtOffset, fmtScore, relTime, typeLabel, utcDateTime, CLASS_LABELS } from "@/lib/format"; | |
| 9 | +import { ownerFetch, publicFetch } from "@/lib/owner"; | |
| 10 | +import { FieldChanges } from "./field-changes"; | |
| 11 | +import { Badge, Bar, Chip, EvidenceTag, Flag, Score, Skeleton, StateBadge, TypeChip } from "./ui"; | |
| 12 | + | |
| 13 | +/** | |
| 14 | + * Intelligence panel (spec §33, §78): a right drawer on desktop, a full-screen sheet on mobile. | |
| 15 | + * Opens from any feed row without navigation; the permanent page stays one click away. | |
| 16 | + */ | |
| 17 | +interface DrawerCtx { | |
| 18 | + open: (slug: string, seed?: EventItem) => void; | |
| 19 | + close: () => void; | |
| 20 | +} | |
| 21 | +const Ctx = createContext<DrawerCtx>({ open: () => {}, close: () => {} }); | |
| 22 | +export const useEventDrawer = (): DrawerCtx => useContext(Ctx); | |
| 23 | + | |
| 24 | +export function EventDrawerProvider({ children }: { children: ReactNode }) { | |
| 25 | + // The drawer state is keyed by pathname: a route change opens a fresh (closed) drawer. | |
| 26 | + const pathname = usePathname(); | |
| 27 | + const [state, setState] = useState<{ path: string; slug: string | null; seed: EventItem | null; detail: EventDetail | null; loading: boolean }>({ path: pathname, slug: null, seed: null, detail: null, loading: false }); | |
| 28 | + const current = state.path === pathname ? state : { path: pathname, slug: null, seed: null, detail: null, loading: false }; | |
| 29 | + const slug = current.slug; | |
| 30 | + const seed = current.seed; | |
| 31 | + const detail = current.detail; | |
| 32 | + const loading = current.loading; | |
| 33 | + const open = useCallback((s: string, e?: EventItem) => setState({ path: pathname, slug: s, seed: e ?? null, detail: null, loading: true }), [pathname]); | |
| 34 | + const close = useCallback(() => setState({ path: pathname, slug: null, seed: null, detail: null, loading: false }), [pathname]); | |
| 35 | + useEffect(() => { | |
| 36 | + if (!slug) return; | |
| 37 | + let cancelled = false; | |
| 38 | + publicFetch<EventDetail>(`/api/v1/events/${encodeURIComponent(slug)}`) | |
| 39 | + .then((d) => { | |
| 40 | + if (!cancelled) setState((st) => (st.slug === slug ? { ...st, detail: d, loading: false } : st)); | |
| 41 | + }) | |
| 42 | + .catch(() => { | |
| 43 | + if (!cancelled) setState((st) => (st.slug === slug ? { ...st, loading: false } : st)); | |
| 44 | + }); | |
| 45 | + return () => { | |
| 46 | + cancelled = true; | |
| 47 | + }; | |
| 48 | + }, [slug]); | |
| 49 | + useEffect(() => { | |
| 50 | + if (!slug) return; | |
| 51 | + const onKey = (e: KeyboardEvent): void => { | |
| 52 | + if (e.key === "Escape") close(); | |
| 53 | + }; | |
| 54 | + document.addEventListener("keydown", onKey); | |
| 55 | + const prev = document.body.style.overflow; | |
| 56 | + if (window.innerWidth < 1024) document.body.style.overflow = "hidden"; | |
| 57 | + return () => { | |
| 58 | + document.removeEventListener("keydown", onKey); | |
| 59 | + document.body.style.overflow = prev; | |
| 60 | + }; | |
| 61 | + }, [slug, close]); | |
| 62 | + const value = useMemo(() => ({ open, close }), [open, close]); | |
| 63 | + return ( | |
| 64 | + <Ctx.Provider value={value}> | |
| 65 | + {children} | |
| 66 | + {slug && <Drawer slug={slug} seed={seed} detail={detail} loading={loading} onClose={close} />} | |
| 67 | + </Ctx.Provider> | |
| 68 | + ); | |
| 69 | +} | |
| 70 | + | |
| 71 | +function Drawer({ slug, seed, detail, loading, onClose }: { slug: string; seed: EventItem | null; detail: EventDetail | null; loading: boolean; onClose: () => void }) { | |
| 72 | + const ev = detail?.event ?? seed; | |
| 73 | + const ref = useRef<HTMLDivElement>(null); | |
| 74 | + useEffect(() => { | |
| 75 | + ref.current?.focus(); | |
| 76 | + }, [slug]); | |
| 77 | + return ( | |
| 78 | + <div className="fixed inset-0 z-50 flex justify-end" role="dialog" aria-modal="true" aria-label={ev?.title ?? "Event"}> | |
| 79 | + <button type="button" aria-label="Close" onClick={onClose} className="scrim absolute inset-0 hidden cursor-default lg:block" /> | |
| 80 | + <div ref={ref} tabIndex={-1} className="relative flex h-full w-full flex-col bg-bg shadow-2xl outline-none animate-sheet-up lg:w-[520px] lg:animate-slide-in lg:border-l lg:border-line xl:w-[600px]"> | |
| 81 | + <header className="flex items-center gap-2 border-b border-line px-3 py-2"> | |
| 82 | + <span className="label">Intelligence panel</span> | |
| 83 | + <span className="ml-auto flex items-center gap-1"> | |
| 84 | + {ev && <BookmarkButton eventId={ev.id} />} | |
| 85 | + {ev && ( | |
| 86 | + <Link href={`/event/${ev.slug}`} className="inline-flex size-7 items-center justify-center rounded-md border border-line text-fg-muted hover:text-fg" title="Open full page" aria-label="Open full page"> | |
| 87 | + <Maximize2 className="size-3.5" /> | |
| 88 | + </Link> | |
| 89 | + )} | |
| 90 | + <button type="button" onClick={onClose} className="inline-flex size-7 items-center justify-center rounded-md border border-line text-fg-muted hover:text-fg" aria-label="Close"> | |
| 91 | + <X className="size-4" /> | |
| 92 | + </button> | |
| 93 | + </span> | |
| 94 | + </header> | |
| 95 | + <div className="min-h-0 flex-1 overflow-y-auto px-3 py-3 pb-24 lg:pb-6"> | |
| 96 | + {!ev ? ( | |
| 97 | + <DrawerSkeleton /> | |
| 98 | + ) : ( | |
| 99 | + <> | |
| 100 | + <div className="flex flex-wrap items-center gap-1.5"> | |
| 101 | + <Link href={`/source/${ev.source?.id ?? ev.source_id}`} className="font-mono text-[11px] font-semibold uppercase tracking-wide text-fg-muted hover:text-fg">{ev.source?.name}</Link> | |
| 102 | + {ev.country && <Flag code={ev.country} className="text-[12px]" />} | |
| 103 | + <StateBadge state={ev.cluster?.state} /> | |
| 104 | + {ev.silent_change && <Badge kind="silent" />} | |
| 105 | + {ev.first_party === false ? <Badge kind="external" /> : <Badge kind="first-party" />} | |
| 106 | + <EvidenceTag label={ev.evidence_label} /> | |
| 107 | + </div> | |
| 108 | + <h2 className="mt-1.5 text-[17px] font-semibold leading-snug">{ev.title}</h2> | |
| 109 | + <div className="mt-1 flex flex-wrap items-center gap-1.5 text-[11.5px] text-fg-subtle"> | |
| 110 | + <TypeChip type={ev.event_type} /> | |
| 111 | + <span>detected {relTime(ev.detected_at)}</span> | |
| 112 | + <span>·</span> | |
| 113 | + <span className="font-mono">{utcDateTime(ev.detected_at)}</span> | |
| 114 | + </div> | |
| 115 | + <p className="mt-3 text-[13.5px] leading-relaxed">{ev.summary}</p> | |
| 116 | + | |
| 117 | + {/* WHAT CHANGED */} | |
| 118 | + {(ev.field_changes?.length ?? 0) > 0 && ( | |
| 119 | + <section className="mt-4"> | |
| 120 | + <div className="label mb-1.5">What changed</div> | |
| 121 | + <FieldChanges items={ev.field_changes} max={8} /> | |
| 122 | + </section> | |
| 123 | + )} | |
| 124 | + {ev.why_it_matters && ( | |
| 125 | + <section className="mt-4"> | |
| 126 | + <div className="label mb-1">Why it matters <span className="normal-case tracking-normal text-fg-subtle">· analysis</span></div> | |
| 127 | + <p className="text-[13px] leading-relaxed text-fg-muted">{ev.why_it_matters}</p> | |
| 128 | + </section> | |
| 129 | + )} | |
| 130 | + | |
| 131 | + {/* SCORES */} | |
| 132 | + <section className="mt-4 rounded-md border border-line bg-panel p-3"> | |
| 133 | + <div className="flex items-center gap-3"> | |
| 134 | + <Score value={ev.signal_score ?? ev.importance} size="lg" kind="signal" /> | |
| 135 | + <div className="min-w-0"> | |
| 136 | + <div className="label">WebSensor signal score</div> | |
| 137 | + <div className="text-[12px] text-fg-muted">{(ev.signal_score ?? ev.importance) >= 80 ? "attention now" : (ev.signal_score ?? ev.importance) >= 60 ? "worth a look" : "informational"}</div> | |
| 138 | + </div> | |
| 139 | + </div> | |
| 140 | + <div className="mt-3 grid grid-cols-2 gap-x-4 gap-y-2 text-[12px]"> | |
| 141 | + <Mini label="Importance" v={ev.importance} /> | |
| 142 | + <Mini label="Confidence" v={ev.confidence} tone="info" /> | |
| 143 | + <Mini label="Novelty" v={ev.novelty} /> | |
| 144 | + <Mini label="Impact" v={ev.impact_score ?? 0} tone="high" /> | |
| 145 | + <Mini label="Velocity" v={ev.velocity_score ?? 0} tone="silent" /> | |
| 146 | + <Mini label="Anomaly" v={ev.anomaly_score ?? 0} tone="mid" /> | |
| 147 | + </div> | |
| 148 | + {(ev.score_reasons?.length ?? 0) > 0 && ( | |
| 149 | + <ul className="mt-3 space-y-0.5 text-[12px]"> | |
| 150 | + {ev.score_reasons!.slice(0, 7).map((r, i) => ( | |
| 151 | + <li key={i} className="flex gap-2"> | |
| 152 | + <span className={`font-mono ${r.sign === "+" ? "text-signal" : "text-danger"}`}>{r.sign}</span> | |
| 153 | + <span className="text-fg-muted">{r.text}</span> | |
| 154 | + {r.points !== undefined && <span className="ml-auto font-mono text-fg-subtle tabular">{r.points > 0 ? "+" : ""}{r.points}</span>} | |
| 155 | + </li> | |
| 156 | + ))} | |
| 157 | + </ul> | |
| 158 | + )} | |
| 159 | + {ev.change_class && <div className="mt-2 text-[11px] text-fg-subtle">Semantic class: <span className="text-fg-muted">{CLASS_LABELS[ev.change_class] ?? ev.change_class}</span></div>} | |
| 160 | + </section> | |
| 161 | + | |
| 162 | + {/* CLUSTER / PROPAGATION */} | |
| 163 | + {ev.cluster && ev.cluster.event_count > 1 && ( | |
| 164 | + <section className="mt-4 rounded-md border border-line bg-panel p-3"> | |
| 165 | + <div className="flex items-center justify-between"> | |
| 166 | + <div className="label">Event cluster</div> | |
| 167 | + <Link href={`/cluster/${ev.cluster.slug ?? ev.cluster.id}`} className="text-[11.5px] text-info hover:underline">propagation timeline →</Link> | |
| 168 | + </div> | |
| 169 | + <div className="mt-2 grid grid-cols-3 gap-2 text-center"> | |
| 170 | + <Cell v={ev.cluster.event_count} l="signals" /> | |
| 171 | + <Cell v={ev.cluster.first_party_count ?? 0} l="first-party" /> | |
| 172 | + <Cell v={ev.cluster.external_count ?? 0} l="external" /> | |
| 173 | + </div> | |
| 174 | + {ev.cluster.lead_time_ms !== null && ev.cluster.lead_time_ms !== undefined && ev.cluster.lead_time_ms > 0 && ( | |
| 175 | + <p className="mt-2 text-[12px] text-fg-muted"> | |
| 176 | + <span className="font-semibold text-signal">WebSensor lead time</span> {fmtOffset(ev.cluster.lead_time_ms).replace("+", "")} before the first external report. | |
| 177 | + </p> | |
| 178 | + )} | |
| 179 | + </section> | |
| 180 | + )} | |
| 181 | + | |
| 182 | + {/* EVIDENCE */} | |
| 183 | + <section className="mt-4"> | |
| 184 | + <div className="label mb-1.5">Evidence</div> | |
| 185 | + <div className="rounded-md border border-line bg-panel p-3 text-[12.5px]"> | |
| 186 | + <a href={ev.url} target="_blank" rel="noopener noreferrer nofollow" className="inline-flex max-w-full items-center gap-1 break-all font-mono text-[12px] text-info hover:underline"> | |
| 187 | + {ev.url} <ExternalLink className="size-3 shrink-0" /> | |
| 188 | + </a> | |
| 189 | + {detail ? ( | |
| 190 | + <dl className="mt-2 grid grid-cols-[auto_1fr] gap-x-3 gap-y-0.5 text-[12px]"> | |
| 191 | + {detail.snapshots.map((s) => ( | |
| 192 | + <FragmentRow key={s.id} label={s.id === ev.old_snapshot_id ? "Before" : "After"} value={`${utcDateTime(s.captured_at)} · HTTP ${s.http_status ?? "—"} · ${s.canonical_hash?.slice(0, 12) ?? ""}`} /> | |
| 193 | + ))} | |
| 194 | + <FragmentRow label="Observed from" value={ev.observed_from ? utcDateTime(ev.observed_from) : "—"} /> | |
| 195 | + <FragmentRow label="Published" value={ev.published_at ? utcDateTime(ev.published_at) : "—"} /> | |
| 196 | + <FragmentRow label="Processing" value={fmtMs(ev.processing_latency_ms)} /> | |
| 197 | + </dl> | |
| 198 | + ) : ( | |
| 199 | + <Skeleton className="mt-2 h-10 w-full" /> | |
| 200 | + )} | |
| 201 | + <div className="mt-2 flex flex-wrap gap-2 text-[11.5px]"> | |
| 202 | + <Link href={`/event/${ev.slug}#diff`} className="text-info hover:underline">Open diff</Link> | |
| 203 | + {ev.old_snapshot_id && ev.new_snapshot_id && <Link href={`/compare?a=${ev.old_snapshot_id}&b=${ev.new_snapshot_id}`} className="text-info hover:underline">Compare snapshots</Link>} | |
| 204 | + <Link href={`/sensor/${ev.sensor_id}`} className="text-info hover:underline">Sensor</Link> | |
| 205 | + <Link href={`/url?u=${encodeURIComponent(ev.url)}`} className="text-info hover:underline">URL history</Link> | |
| 206 | + </div> | |
| 207 | + </div> | |
| 208 | + </section> | |
| 209 | + | |
| 210 | + {/* HISTORICAL CONTEXT */} | |
| 211 | + {detail?.history && detail.history.length > 0 && ( | |
| 212 | + <section className="mt-4"> | |
| 213 | + <div className="label mb-1.5">Previous changes on this page</div> | |
| 214 | + <ul className="divide-y divide-line rounded-md border border-line bg-panel"> | |
| 215 | + {detail.history.map((h) => ( | |
| 216 | + <li key={h.id} className="flex items-center gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 217 | + <span className="w-[5.5rem] shrink-0 font-mono text-[11px] text-fg-subtle tabular">{relTime(h.detected_at)}</span> | |
| 218 | + <Link href={`/event/${h.slug}`} className="min-w-0 flex-1 truncate hover:underline">{h.title}</Link> | |
| 219 | + <Score value={h.importance} size="sm" /> | |
| 220 | + </li> | |
| 221 | + ))} | |
| 222 | + </ul> | |
| 223 | + </section> | |
| 224 | + )} | |
| 225 | + | |
| 226 | + {/* RELATED */} | |
| 227 | + {detail?.related && detail.related.length > 0 && ( | |
| 228 | + <section className="mt-4"> | |
| 229 | + <div className="label mb-1.5">Related signals</div> | |
| 230 | + <ul className="divide-y divide-line rounded-md border border-line bg-panel"> | |
| 231 | + {detail.related.slice(0, 6).map((r) => ( | |
| 232 | + <li key={r.id} className="flex items-center gap-2 px-3 py-1.5 text-[12.5px]"> | |
| 233 | + <span className="w-[5.5rem] shrink-0 truncate font-mono text-[10.5px] uppercase text-fg-subtle">{r.source?.name}</span> | |
| 234 | + <Link href={`/event/${r.slug}`} className="min-w-0 flex-1 truncate hover:underline">{r.title}</Link> | |
| 235 | + <Score value={r.signal_score ?? r.importance} size="sm" kind="signal" /> | |
| 236 | + </li> | |
| 237 | + ))} | |
| 238 | + </ul> | |
| 239 | + </section> | |
| 240 | + )} | |
| 241 | + {ev.entities?.length > 0 && ( | |
| 242 | + <div className="mt-4 flex flex-wrap gap-1"> | |
| 243 | + {ev.entities.map((x) => ( | |
| 244 | + <Chip key={x.id} href={`/entity/${x.id}`} tone={x.role === "subject" ? "signal" : "default"}>{x.name}</Chip> | |
| 245 | + ))} | |
| 246 | + </div> | |
| 247 | + )} | |
| 248 | + <p className="mt-4 text-[11px] text-fg-subtle">{typeLabel(ev.event_type)} · {loading ? "loading details…" : "raw evidence is immutable; analysis is labelled and versioned."}</p> | |
| 249 | + </> | |
| 250 | + )} | |
| 251 | + </div> | |
| 252 | + </div> | |
| 253 | + </div> | |
| 254 | + ); | |
| 255 | +} | |
| 256 | + | |
| 257 | +function Mini({ label, v, tone }: { label: string; v: number; tone?: "signal" | "hot" | "high" | "mid" | "silent" | "info" }) { | |
| 258 | + return ( | |
| 259 | + <div> | |
| 260 | + <div className="flex items-baseline justify-between"> | |
| 261 | + <span className="text-fg-subtle">{label}</span> | |
| 262 | + <span className="font-mono tabular">{fmtScore(v)}</span> | |
| 263 | + </div> | |
| 264 | + <Bar value={v} tone={tone ?? (v >= 80 ? "hot" : v >= 60 ? "high" : "signal")} /> | |
| 265 | + </div> | |
| 266 | + ); | |
| 267 | +} | |
| 268 | +function Cell({ v, l }: { v: number; l: string }) { | |
| 269 | + return ( | |
| 270 | + <div className="rounded border border-line bg-panel-2 py-1.5"> | |
| 271 | + <div className="font-mono text-base font-semibold tabular">{v}</div> | |
| 272 | + <div className="label !text-[9.5px]">{l}</div> | |
| 273 | + </div> | |
| 274 | + ); | |
| 275 | +} | |
| 276 | +function FragmentRow({ label, value }: { label: string; value: string }) { | |
| 277 | + return ( | |
| 278 | + <> | |
| 279 | + <dt className="text-fg-subtle">{label}</dt> | |
| 280 | + <dd className="truncate font-mono text-fg-muted tabular">{value}</dd> | |
| 281 | + </> | |
| 282 | + ); | |
| 283 | +} | |
| 284 | +function DrawerSkeleton() { | |
| 285 | + return ( | |
| 286 | + <div className="flex flex-col gap-3"> | |
| 287 | + <Skeleton className="h-3 w-32" /> | |
| 288 | + <Skeleton className="h-5 w-11/12" /> | |
| 289 | + <Skeleton className="h-5 w-3/4" /> | |
| 290 | + <Skeleton className="h-16 w-full" /> | |
| 291 | + <Skeleton className="h-28 w-full" /> | |
| 292 | + </div> | |
| 293 | + ); | |
| 294 | +} | |
| 295 | + | |
| 296 | +export function BookmarkButton({ eventId, label = false }: { eventId: string; label?: boolean }) { | |
| 297 | + const [state, setState] = useState<"idle" | "on" | "busy">("idle"); | |
| 298 | + useEffect(() => { | |
| 299 | + let c = false; | |
| 300 | + ownerFetch<{ ids: string[] }>("/api/v1/bookmarks/ids") | |
| 301 | + .then((r) => { | |
| 302 | + if (!c && r.ids.includes(eventId)) setState("on"); | |
| 303 | + }) | |
| 304 | + .catch(() => undefined); | |
| 305 | + return () => { | |
| 306 | + c = true; | |
| 307 | + }; | |
| 308 | + }, [eventId]); | |
| 309 | + const toggle = async (): Promise<void> => { | |
| 310 | + const was = state; | |
| 311 | + setState("busy"); | |
| 312 | + try { | |
| 313 | + if (was === "on") { | |
| 314 | + await ownerFetch(`/api/v1/bookmarks/${encodeURIComponent(eventId)}`, { method: "DELETE" }); | |
| 315 | + setState("idle"); | |
| 316 | + } else { | |
| 317 | + await ownerFetch("/api/v1/bookmarks", { method: "POST", body: JSON.stringify({ event_id: eventId }) }); | |
| 318 | + setState("on"); | |
| 319 | + } | |
| 320 | + } catch { | |
| 321 | + setState(was); | |
| 322 | + } | |
| 323 | + }; | |
| 324 | + return ( | |
| 325 | + <button type="button" onClick={toggle} disabled={state === "busy"} aria-pressed={state === "on"} title={state === "on" ? "Remove bookmark" : "Bookmark"} className={`inline-flex h-7 items-center gap-1 rounded-md border border-line px-1.5 text-[12px] ${state === "on" ? "text-signal" : "text-fg-muted hover:text-fg"}`}> | |
| 326 | + {state === "on" ? <BookmarkCheck className="size-3.5" /> : <Bookmark className="size-3.5" />} | |
| 327 | + {label && (state === "on" ? "Saved" : "Save")} | |
| 328 | + </button> | |
| 329 | + ); | |
| 330 | +} | |
modified
apps/web/src/components/event-row.tsx
+68 −22
@@ -1,48 +1,94 @@ | ||
| 1 | 1 | "use client"; |
| 2 | 2 | |
| 3 | 3 | import Link from "next/link"; |
| 4 | +import type { MouseEvent } from "react"; | |
| 4 | 5 | import type { EventItem } from "@/lib/api"; |
| 5 | −import { relTime, utcTime, utcDate } from "@/lib/format"; | |
| 6 | −import { Chip, EvidenceTag, Score, SilentBadge, TypeChip } from "./ui"; | |
| 6 | +import { relTime, utcTime, utcDate, utcDateTime, localDateTime } from "@/lib/format"; | |
| 7 | +import { useEventDrawer } from "./event-drawer"; | |
| 8 | +import { FieldChangeInline } from "./field-changes"; | |
| 9 | +import { usePrefs } from "./prefs"; | |
| 10 | +import { Badge, Chip, Flag, Score, TypeChip } from "./ui"; | |
| 7 | 11 | |
| 8 | −export function EventRow({ ev, flash = false, showDate = false, now }: { ev: EventItem; flash?: boolean; showDate?: boolean; now?: number }) { | |
| 9 | − const cats = (ev.categories ?? []).filter((c) => !["ai", "cyber", "finance", "health", "government", "science", "products", "infrastructure"].includes(c) || !ev.categories.some((x) => x !== c && sameChannel(c, x))).slice(0, 3); | |
| 12 | +const CHANNEL_GROUPS: Record<string, string[]> = { finance: ["finance", "payments", "crypto", "commerce"], health: ["health", "pharma"], government: ["government", "statistics"], science: ["science", "space"], products: ["consumer-tech", "automotive", "semiconductors", "enterprise"], infrastructure: ["cloud", "developer", "internet", "standards"], news: ["news", "media"] }; | |
| 13 | +function sameChannel(a: string, b: string): boolean { | |
| 14 | + return CHANNEL_GROUPS[a]?.includes(b) ?? false; | |
| 15 | +} | |
| 16 | + | |
| 17 | +/** | |
| 18 | + * Feed row (spec §31): TIME · ENTITY/SOURCE · SIGNAL · TYPE · TITLE · STATUS · TAGS, with badges | |
| 19 | + * BREAKING / DEVELOPING / SILENT / FIRST PARTY / CONFIRMED / ANOMALOUS. Density comes from CSS | |
| 20 | + * variables (spec §32). Clicking the title opens the intelligence drawer (spec §33); the link keeps | |
| 21 | + * a real href so middle-click / crawlers / no-JS still reach the permanent page. | |
| 22 | + */ | |
| 23 | +export function EventRow({ ev, flash = false, showDate = false, now, fresh = false, replayed = false }: { ev: EventItem; flash?: boolean; showDate?: boolean; now?: number; fresh?: boolean; replayed?: boolean }) { | |
| 24 | + const { open } = useEventDrawer(); | |
| 25 | + const { exactTime, density, mounted } = usePrefs(); | |
| 26 | + const iso = (() => { | |
| 27 | + const d = new Date(ev.detected_at); | |
| 28 | + return Number.isNaN(d.getTime()) ? ev.detected_at : d.toISOString(); | |
| 29 | + })(); | |
| 30 | + const cats = (ev.categories ?? []).filter((c) => !Object.keys(CHANNEL_GROUPS).includes(c) || !ev.categories.some((x) => x !== c && sameChannel(c, x))).slice(0, density === "compact" ? 1 : 2); | |
| 31 | + const signal = ev.signal_score ?? ev.importance; | |
| 32 | + const state = ev.cluster?.state; | |
| 33 | + const isBreaking = state === "breaking" || (signal >= 80 && !state); | |
| 34 | + const isDeveloping = state === "developing"; | |
| 35 | + const anomalous = (ev.anomaly_score ?? 0) >= 60; | |
| 36 | + const clusterHref = ev.cluster?.slug ? `/cluster/${ev.cluster.slug}` : ev.cluster_id ? `/cluster/${ev.cluster_id}` : null; | |
| 37 | + const onClick = (e: MouseEvent<HTMLAnchorElement>): void => { | |
| 38 | + if (e.metaKey || e.ctrlKey || e.shiftKey || e.button !== 0) return; | |
| 39 | + e.preventDefault(); | |
| 40 | + open(ev.slug, ev); | |
| 41 | + }; | |
| 10 | 42 | return ( |
| 11 | − <article className={`grid grid-cols-[auto_1fr] gap-x-3 px-3 py-2 hairline sm:grid-cols-[6.5rem_1fr_auto] ${flash ? "animate-flash" : ""}`}> | |
| 12 | − <div className="flex flex-col font-mono text-[11.5px] leading-4 text-fg-subtle tabular"> | |
| 13 | − <span className="text-fg-muted">{utcTime(ev.detected_at)}</span> | |
| 14 | − <span>{showDate ? utcDate(ev.detected_at) : relTime(ev.detected_at, now)}</span> | |
| 43 | + <article className={`row-dense offscreen-ok grid grid-cols-[auto_1fr] gap-x-3 px-3 hairline sm:grid-cols-[6.25rem_1fr_auto] ${flash || fresh ? "animate-row-in" : ""} ${isBreaking ? "border-l-2 border-l-hot/70" : isDeveloping ? "border-l-2 border-l-high/60" : ev.silent_change ? "border-l-2 border-l-silent/60" : "border-l-2 border-l-transparent"}`}> | |
| 44 | + <div className="flex flex-col font-mono meta leading-4 text-fg-subtle tabular"> | |
| 45 | + <time dateTime={iso} title={mounted ? localDateTime(ev.detected_at) : utcDateTime(ev.detected_at)} suppressHydrationWarning className="text-fg-muted">{utcTime(ev.detected_at)}</time> | |
| 46 | + <span suppressHydrationWarning>{showDate ? utcDate(ev.detected_at) : exactTime ? "UTC" : relTime(ev.detected_at, now)}</span> | |
| 15 | 47 | </div> |
| 16 | 48 | <div className="min-w-0 sm:col-start-2"> |
| 17 | − <div className="flex flex-wrap items-center gap-x-2 gap-y-1"> | |
| 18 | − <Link href={`/source/${ev.source?.id ?? ev.source_id}`} className="font-mono text-[11px] font-semibold uppercase tracking-wide text-fg-muted hover:text-fg"> | |
| 49 | + <div className="flex flex-wrap items-center gap-x-2 gap-y-0.5"> | |
| 50 | + <Link href={`/source/${ev.source?.id ?? ev.source_id}`} className="font-mono meta font-semibold uppercase tracking-wide text-fg-muted hover:text-fg"> | |
| 19 | 51 | {ev.source?.name ?? ev.source_id} |
| 20 | 52 | </Link> |
| 53 | + {ev.country && <Flag code={ev.country} className="text-[11px]" />} | |
| 21 | 54 | <span className="sm:hidden"> |
| 22 | − <Score value={ev.importance} size="sm" /> | |
| 55 | + <Score value={signal} size="sm" kind="signal" /> | |
| 23 | 56 | </span> |
| 24 | − {ev.silent_change && <SilentBadge compact />} | |
| 57 | + {isBreaking && <Badge kind="breaking" compact={density === "compact"} />} | |
| 58 | + {isDeveloping && <Badge kind="developing" compact={density === "compact"} />} | |
| 59 | + {ev.silent_change && <Badge kind="silent" compact={density === "compact"} />} | |
| 60 | + {ev.first_party === false && <Badge kind="external" compact />} | |
| 61 | + {ev.evidence_label === "CONFIRMED" && <Badge kind="confirmed" compact={density === "compact"} />} | |
| 62 | + {anomalous && <Badge kind="anomalous" compact />} | |
| 63 | + {replayed && <Badge kind="replayed" compact />} | |
| 25 | 64 | </div> |
| 26 | − <Link href={`/event/${ev.slug}`} className="mt-0.5 block text-[13.5px] font-medium leading-snug text-fg hover:underline"> | |
| 65 | + <Link href={`/event/${ev.slug}`} onClick={onClick} className={`mt-0.5 block font-medium leading-snug text-fg hover:underline [overflow-wrap:anywhere] ${density === "compact" ? "truncate" : ""}`}> | |
| 27 | 66 | {ev.title} |
| 28 | 67 | </Link> |
| 68 | + {density !== "compact" && ev.field_changes && ev.field_changes.length > 0 && ( | |
| 69 | + <div className="mt-0.5"> | |
| 70 | + <FieldChangeInline items={ev.field_changes} max={density === "comfortable" ? 3 : 2} /> | |
| 71 | + </div> | |
| 72 | + )} | |
| 73 | + {density === "comfortable" && ev.summary && <p className="mt-0.5 line-clamp-2 text-[12.5px] text-fg-muted">{ev.summary}</p>} | |
| 29 | 74 | <div className="mt-1 flex flex-wrap items-center gap-1"> |
| 75 | + <TypeChip type={ev.event_type} href={`/live?event_type=${ev.event_type}`} /> | |
| 30 | 76 | {cats.map((c) => ( |
| 31 | 77 | <Chip key={c} href={`/category/${c}`}>{c}</Chip> |
| 32 | 78 | ))} |
| 33 | − <TypeChip type={ev.event_type} /> | |
| 34 | − <EvidenceTag label={ev.evidence_label} /> | |
| 35 | − {ev.cluster_size && ev.cluster_size > 1 && <Chip tone="info" href={`/event/${ev.slug}#cluster`}>+{ev.cluster_size - 1} related</Chip>} | |
| 79 | + {ev.entities?.filter((x) => x.role === "subject").slice(0, 1).map((x) => ( | |
| 80 | + <Chip key={x.id} href={`/entity/${x.id}`} tone="default" className="!text-fg-muted">{x.name}</Chip> | |
| 81 | + ))} | |
| 82 | + {ev.cluster_size && ev.cluster_size > 1 && clusterHref && ( | |
| 83 | + <Chip tone="info" href={clusterHref} title="Signals in this event cluster"> | |
| 84 | + {ev.cluster_size} signals{ev.cluster?.source_count && ev.cluster.source_count > 1 ? ` · ${ev.cluster.source_count} sources` : ""} | |
| 85 | + </Chip> | |
| 86 | + )} | |
| 36 | 87 | </div> |
| 37 | 88 | </div> |
| 38 | 89 | <div className="hidden items-start pt-0.5 sm:flex"> |
| 39 | − <Score value={ev.importance} /> | |
| 90 | + <Score value={signal} kind="signal" title={`Signal ${Math.round(signal)} · importance ${Math.round(ev.importance)} · confidence ${Math.round(ev.confidence)}`} /> | |
| 40 | 91 | </div> |
| 41 | 92 | </article> |
| 42 | 93 | ); |
| 43 | 94 | } |
| 44 | − | |
| 45 | −function sameChannel(a: string, b: string): boolean { | |
| 46 | − const groups: Record<string, string[]> = { finance: ["finance", "payments", "crypto", "commerce"], health: ["health", "pharma"], government: ["government", "statistics"], science: ["science", "space"], products: ["consumer-tech", "automotive", "semiconductors", "enterprise"], infrastructure: ["cloud", "developer", "internet", "standards"] }; | |
| 47 | − return groups[a]?.includes(b) ?? false; | |
| 48 | −} | |
added
apps/web/src/components/field-changes.tsx
+52 −0
@@ -0,0 +1,52 @@ | ||
| 1 | +import type { FieldChange } from "@/lib/api"; | |
| 2 | +import { fmtPctDelta } from "@/lib/format"; | |
| 3 | + | |
| 4 | +/** | |
| 5 | + * WHAT CHANGED — field-level before → after table (spec §21, §78). Pure server-renderable component. | |
| 6 | + */ | |
| 7 | +export function FieldChanges({ items, compact = false, max }: { items: FieldChange[] | null | undefined; compact?: boolean; max?: number }) { | |
| 8 | + if (!items?.length) return null; | |
| 9 | + const rows = max ? items.slice(0, max) : items; | |
| 10 | + return ( | |
| 11 | + <div className={`overflow-hidden rounded-md border border-line ${compact ? "text-[12px]" : "text-[12.5px]"}`}> | |
| 12 | + <table className="w-full"> | |
| 13 | + <tbody className="divide-y divide-line"> | |
| 14 | + {rows.map((f, i) => ( | |
| 15 | + <tr key={i} className="align-top"> | |
| 16 | + <td className={`${compact ? "px-2 py-1" : "px-3 py-1.5"} w-[34%] text-fg-muted`}> | |
| 17 | + <span className="block truncate" title={f.label}>{f.label}</span> | |
| 18 | + <span className="label !text-[9.5px] !tracking-wider">{f.kind}</span> | |
| 19 | + </td> | |
| 20 | + <td className={`${compact ? "px-2 py-1" : "px-3 py-1.5"} font-mono tabular`}> | |
| 21 | + <span className="diff-line-del rounded px-1 break-words">{f.before ?? "∅"}</span> | |
| 22 | + <span className="mx-1.5 text-fg-subtle">→</span> | |
| 23 | + <span className="diff-line-add rounded px-1 break-words">{f.after ?? "∅"}</span> | |
| 24 | + {f.deltaPct !== null && f.deltaPct !== undefined && <span className={`ml-2 font-semibold ${f.deltaPct > 0 ? "text-high" : "text-signal"}`}>{fmtPctDelta(f.deltaPct)}</span>} | |
| 25 | + </td> | |
| 26 | + </tr> | |
| 27 | + ))} | |
| 28 | + </tbody> | |
| 29 | + </table> | |
| 30 | + {max && items.length > max && <div className="border-t border-line px-3 py-1 text-[11px] text-fg-subtle">+{items.length - max} more field change{items.length - max === 1 ? "" : "s"}</div>} | |
| 31 | + </div> | |
| 32 | + ); | |
| 33 | +} | |
| 34 | + | |
| 35 | +/** One-line inline version for feed rows / drawer headers. */ | |
| 36 | +export function FieldChangeInline({ items, max = 2 }: { items: FieldChange[] | null | undefined; max?: number }) { | |
| 37 | + if (!items?.length) return null; | |
| 38 | + return ( | |
| 39 | + <span className="inline-flex flex-wrap items-center gap-x-2 font-mono text-[11px] text-fg-muted tabular"> | |
| 40 | + {items.slice(0, max).map((f, i) => ( | |
| 41 | + <span key={i} className="inline-flex items-center gap-1 whitespace-nowrap"> | |
| 42 | + <span className="text-fg-subtle">{f.label.length > 22 ? f.label.slice(0, 21) + "…" : f.label}</span> | |
| 43 | + <span className="diff-line-del rounded px-0.5">{f.before ?? "∅"}</span> | |
| 44 | + <span className="text-fg-subtle">→</span> | |
| 45 | + <span className="diff-line-add rounded px-0.5">{f.after ?? "∅"}</span> | |
| 46 | + {f.deltaPct !== null && f.deltaPct !== undefined && <span className={f.deltaPct > 0 ? "text-high" : "text-signal"}>{fmtPctDelta(f.deltaPct)}</span>} | |
| 47 | + </span> | |
| 48 | + ))} | |
| 49 | + {items.length > max && <span className="text-fg-subtle">+{items.length - max}</span>} | |
| 50 | + </span> | |
| 51 | + ); | |
| 52 | +} | |
modified
apps/web/src/components/live-feed.tsx
+196 −38
@@ -1,56 +1,113 @@ | ||
| 1 | 1 | "use client"; |
| 2 | 2 | |
| 3 | 3 | import Link from "next/link"; |
| 4 | +import { usePathname, useRouter, useSearchParams } from "next/navigation"; | |
| 5 | +import { ArrowUp, Pause, Play, SlidersHorizontal, X } from "lucide-react"; | |
| 4 | 6 | import { useCallback, useEffect, useMemo, useRef, useState } from "react"; |
| 5 | 7 | import { liveToEvent, type EventItem, type EventQuery, type LiveEvent, eventQueryString } from "@/lib/api"; |
| 6 | −import { CHANNELS } from "@/lib/format"; | |
| 8 | +import { CHANNELS, GROUP_LABELS, SAVED_VIEWS, agoIso, feedHref } from "@/lib/format"; | |
| 9 | +import type { FeedFilters } from "@/lib/feed-filters"; | |
| 7 | 10 | import { publicFetch } from "@/lib/owner"; |
| 8 | 11 | import { useLive, type LiveStatus } from "@/lib/use-live"; |
| 9 | 12 | import { EventRow } from "./event-row"; |
| 10 | −import { Empty } from "./ui"; | |
| 13 | +import { DensityToggle, usePrefs } from "./prefs"; | |
| 14 | +import { Chip, Empty, Kbd } from "./ui"; | |
| 11 | 15 | |
| 12 | −const MAX_ROWS = 300; | |
| 16 | +const MAX_ROWS = 400; | |
| 13 | 17 | |
| 14 | −export function LiveDot({ status }: { status: LiveStatus }) { | |
| 15 | − const label = status === "live" ? "LIVE" : status === "connecting" ? "CONNECTING" : status === "reconnecting" ? "RECONNECTING" : "OFFLINE"; | |
| 16 | − const color = status === "live" ? "bg-signal animate-pulse-dot" : status === "offline" ? "bg-danger" : "bg-warn"; | |
| 18 | +export function LiveDot({ status, paused = false }: { status: LiveStatus; paused?: boolean }) { | |
| 19 | + const label = paused ? "PAUSED" : status === "live" ? "LIVE" : status === "connecting" ? "CONNECTING" : status === "reconnecting" ? "RECONNECTING" : "OFFLINE"; | |
| 20 | + const color = paused ? "bg-warn" : status === "live" ? "bg-signal animate-pulse-dot" : status === "offline" ? "bg-danger" : "bg-warn"; | |
| 17 | 21 | return ( |
| 18 | − <span className="inline-flex items-center gap-1.5 font-mono text-[11px] font-semibold tracking-wider text-fg-muted"> | |
| 22 | + <span className="inline-flex items-center gap-1.5 font-mono text-[11px] font-semibold tracking-wider text-fg-muted" aria-live="polite"> | |
| 19 | 23 | <span className={`inline-block size-2 rounded-full ${color}`} /> |
| 20 | 24 | {label} |
| 21 | 25 | </span> |
| 22 | 26 | ); |
| 23 | 27 | } |
| 24 | 28 | |
| 29 | +export type { FeedFilters } from "@/lib/feed-filters"; | |
| 30 | + | |
| 31 | +function matchesFilters(ev: EventItem, f: FeedFilters, fixedQuery?: EventQuery): boolean { | |
| 32 | + const sig = ev.signal_score ?? ev.importance; | |
| 33 | + if (f.signal_min !== undefined && sig < f.signal_min) return false; | |
| 34 | + if (f.importance_min !== undefined && ev.importance < f.importance_min) return false; | |
| 35 | + if (f.silent_change && !ev.silent_change) return false; | |
| 36 | + if (f.first_party && ev.first_party === false) return false; | |
| 37 | + if (f.confirmed && ev.evidence_label !== "CONFIRMED") return false; | |
| 38 | + if (f.country && ev.country !== f.country) return false; | |
| 39 | + if (f.language && ev.language !== f.language) return false; | |
| 40 | + if (f.category && !ev.categories.includes(f.category)) return false; | |
| 41 | + if (f.event_type && !f.event_type.split(",").includes(ev.event_type)) return false; | |
| 42 | + if (f.q) return false; // text search results are not extended live | |
| 43 | + if (fixedQuery?.source && ev.source_id !== fixedQuery.source) return false; | |
| 44 | + if (fixedQuery?.entity && !ev.entities.some((x) => x.id === fixedQuery.entity)) return false; | |
| 45 | + if (fixedQuery?.category && !ev.categories.includes(fixedQuery.category)) return false; | |
| 46 | + if (fixedQuery?.silent_change && !ev.silent_change) return false; | |
| 47 | + return true; | |
| 48 | +} | |
| 49 | + | |
| 25 | 50 | /** |
| 26 | − * The live feed: initial REST page, WebSocket prepends, channel tabs, cursor pagination. | |
| 27 | − * `fixed` restricts the feed to one filter (used by /breaking, /silent, /category/*). | |
| 51 | + * The live feed v2 (spec §31, §97–99): initial REST page, WebSocket prepends, URL-synced filters, | |
| 52 | + * pause, "↑ N new events" without scroll jumps, density modes, cursor pagination, drawer on click. | |
| 53 | + * `fixed` restricts the feed to one channel (used by /breaking, /silent, /category/*). | |
| 28 | 54 | */ |
| 29 | −export function LiveFeed({ initial, initialCursor, fixed, showTabs = true, title = "LIVE WEB", extraQuery }: { initial: EventItem[]; initialCursor: string | null; fixed?: string; showTabs?: boolean; title?: string; extraQuery?: EventQuery }) { | |
| 30 | − const [tab, setTab] = useState(fixed ?? "all"); | |
| 55 | +export function LiveFeed({ initial, initialCursor, fixed, showTabs = true, title = "LIVE WEB", extraQuery, syncUrl = false, initialFilters, showFilters = true, compactHeader = false }: { initial: EventItem[]; initialCursor: string | null; fixed?: string; showTabs?: boolean; title?: string; extraQuery?: EventQuery; syncUrl?: boolean; initialFilters?: FeedFilters; showFilters?: boolean; compactHeader?: boolean }) { | |
| 56 | + const router = useRouter(); | |
| 57 | + const pathname = usePathname(); | |
| 58 | + const sp = useSearchParams(); | |
| 59 | + const { paused, setPaused } = usePrefs(); | |
| 60 | + const [tab, setTab] = useState(fixed ?? initialFilters?.category ?? "all"); | |
| 61 | + const [filters, setFilters] = useState<FeedFilters>(initialFilters ?? {}); | |
| 31 | 62 | const [items, setItems] = useState<EventItem[]>(initial); |
| 32 | 63 | const [cursor, setCursor] = useState<string | null>(initialCursor); |
| 33 | 64 | const [loading, setLoading] = useState(false); |
| 34 | 65 | const [fresh, setFresh] = useState<Set<string>>(new Set()); |
| 66 | + const [pending, setPending] = useState<EventItem[]>([]); | |
| 67 | + const [atTop, setAtTop] = useState(true); | |
| 35 | 68 | const [now, setNow] = useState(() => Date.now()); |
| 69 | + const [showFilterBar, setShowFilterBar] = useState(Boolean(initialFilters && Object.keys(initialFilters).length)); | |
| 36 | 70 | const seen = useRef(new Set(initial.map((i) => i.id))); |
| 71 | + const topRef = useRef<HTMLDivElement>(null); | |
| 37 | 72 | const chan = useMemo(() => CHANNELS.find((c) => c.key === tab) ?? CHANNELS[0]!, [tab]); |
| 38 | 73 | |
| 74 | + // sync URL ← filters (spec §99) | |
| 75 | + useEffect(() => { | |
| 76 | + if (!syncUrl) return; | |
| 77 | + const href = feedHref({ ...filters, category: tab !== "all" && tab !== "breaking" && tab !== "silent" ? tab : filters.category, silent_change: tab === "silent" ? true : filters.silent_change, signal_min: tab === "breaking" ? Math.max(80, filters.signal_min ?? 0) : filters.signal_min }, pathname); | |
| 78 | + const current = `${pathname}${sp.toString() ? `?${sp.toString()}` : ""}`; | |
| 79 | + if (href !== current) router.replace(href, { scroll: false }); | |
| 80 | + // eslint-disable-next-line react-hooks/exhaustive-deps | |
| 81 | + }, [filters, tab, syncUrl, pathname]); | |
| 82 | + | |
| 39 | 83 | useEffect(() => { |
| 40 | 84 | const t = setInterval(() => setNow(Date.now()), 10_000); |
| 41 | 85 | return () => clearInterval(t); |
| 42 | 86 | }, []); |
| 87 | + // Is the top of the list visible? If not, new events are buffered (spec §97). | |
| 88 | + useEffect(() => { | |
| 89 | + const el = topRef.current; | |
| 90 | + if (!el) return; | |
| 91 | + const io = new IntersectionObserver(([e]) => setAtTop(Boolean(e?.isIntersecting)), { rootMargin: "80px 0px 0px 0px" }); | |
| 92 | + io.observe(el); | |
| 93 | + return () => io.disconnect(); | |
| 94 | + }, []); | |
| 95 | + | |
| 96 | + const query = useMemo<EventQuery>(() => { | |
| 97 | + const q: EventQuery = { ...(extraQuery ?? {}), ...(chan.query as EventQuery), ...filters, limit: 60 }; | |
| 98 | + if (chan.key === "breaking") { | |
| 99 | + q.order = q.order ?? "signal"; | |
| 100 | + q.signal_min = Math.max(80, q.signal_min ?? 0); | |
| 101 | + q.after = agoIso(48 * 3600e3); | |
| 102 | + } | |
| 103 | + return q; | |
| 104 | + }, [chan, extraQuery, filters]); | |
| 43 | 105 | |
| 44 | 106 | const load = useCallback( |
| 45 | 107 | async (reset: boolean, c: string | null) => { |
| 46 | 108 | setLoading(true); |
| 47 | 109 | try { |
| 48 | − const q: EventQuery = { ...(extraQuery ?? {}), ...(chan.query as EventQuery), limit: 60, cursor: reset ? undefined : (c ?? undefined) }; | |
| 49 | − if (chan.key === "breaking") { | |
| 50 | − q.order = "importance"; | |
| 51 | − q.after = new Date(Date.now() - 48 * 3600e3).toISOString(); | |
| 52 | − } | |
| 53 | − const page = await publicFetch<{ items: EventItem[]; nextCursor: string | null }>(`/api/v1/events${eventQueryString(q)}`); | |
| 110 | + const page = await publicFetch<{ items: EventItem[]; nextCursor: string | null }>(`/api/v1/events${eventQueryString({ ...query, cursor: reset ? undefined : (c ?? undefined) })}`); | |
| 54 | 111 | setItems((prev) => { |
| 55 | 112 | const base = reset ? [] : prev; |
| 56 | 113 | const ids = new Set(base.map((i) => i.id)); |
@@ -59,13 +116,14 @@ export function LiveFeed({ initial, initialCursor, fixed, showTabs = true, title | ||
| 59 | 116 | return merged; |
| 60 | 117 | }); |
| 61 | 118 | setCursor(page.nextCursor); |
| 119 | + if (reset) setPending([]); | |
| 62 | 120 | } catch { |
| 63 | 121 | // keep current list |
| 64 | 122 | } finally { |
| 65 | 123 | setLoading(false); |
| 66 | 124 | } |
| 67 | 125 | }, |
| 68 | − [chan, extraQuery], | |
| 126 | + [query], | |
| 69 | 127 | ); |
| 70 | 128 | |
| 71 | 129 | const first = useRef(true); |
@@ -77,50 +135,141 @@ export function LiveFeed({ initial, initialCursor, fixed, showTabs = true, title | ||
| 77 | 135 | void load(true, null); |
| 78 | 136 | }, [load]); |
| 79 | 137 | |
| 138 | + const flushPending = useCallback(() => { | |
| 139 | + setPending((p) => { | |
| 140 | + if (!p.length) return p; | |
| 141 | + setItems((prev) => [...p, ...prev].slice(0, MAX_ROWS)); | |
| 142 | + setFresh((prevF) => new Set([...prevF, ...p.map((x) => x.id)])); | |
| 143 | + setTimeout(() => setFresh(new Set()), 1800); | |
| 144 | + return []; | |
| 145 | + }); | |
| 146 | + topRef.current?.scrollIntoView({ block: "start", behavior: "smooth" }); | |
| 147 | + }, []); | |
| 148 | + | |
| 80 | 149 | const onEvent = useCallback( |
| 81 | 150 | (e: LiveEvent) => { |
| 82 | 151 | if (seen.current.has(e.id)) return; |
| 83 | 152 | const ev = liveToEvent(e); |
| 84 | − if (extraQuery?.source && ev.source_id !== extraQuery.source) return; | |
| 85 | − if (extraQuery?.entity && !ev.entities.some((x) => x.id === extraQuery.entity)) return; | |
| 153 | + if (!matchesFilters(ev, filters, extraQuery)) return; | |
| 154 | + if (chan.key === "breaking" && (ev.signal_score ?? ev.importance) < 80) return; | |
| 155 | + if (chan.key === "silent" && !ev.silent_change) return; | |
| 86 | 156 | seen.current.add(e.id); |
| 157 | + if (paused || !atTop || query.order === "signal" || query.order === "importance") { | |
| 158 | + setPending((p) => [ev, ...p].slice(0, 200)); | |
| 159 | + return; | |
| 160 | + } | |
| 87 | 161 | setItems((prev) => [ev, ...prev].slice(0, MAX_ROWS)); |
| 88 | 162 | setFresh((prev) => new Set([...prev, e.id])); |
| 89 | − setTimeout(() => setFresh((prev) => { | |
| 90 | − const n = new Set(prev); | |
| 91 | − n.delete(e.id); | |
| 92 | − return n; | |
| 93 | − }), 1500); | |
| 163 | + setTimeout( | |
| 164 | + () => | |
| 165 | + setFresh((prev) => { | |
| 166 | + const n = new Set(prev); | |
| 167 | + n.delete(e.id); | |
| 168 | + return n; | |
| 169 | + }), | |
| 170 | + 1800, | |
| 171 | + ); | |
| 94 | 172 | }, |
| 95 | − [extraQuery], | |
| 173 | + [filters, extraQuery, chan.key, paused, atTop, query.order], | |
| 96 | 174 | ); |
| 97 | − const status = useLive([chan.ws], onEvent); | |
| 175 | + const wsChannel = extraQuery?.source ? `source:${extraQuery.source}` : extraQuery?.entity ? `entity:${extraQuery.entity}` : filters.country && chan.key === "all" ? `country:${filters.country}` : filters.group && chan.key === "all" ? `group:${filters.group}` : chan.ws; | |
| 176 | + const wsChannels = useMemo(() => [wsChannel], [wsChannel]); | |
| 177 | + const status = useLive(wsChannels, onEvent); | |
| 178 | + | |
| 179 | + const setF = (patch: Partial<FeedFilters>): void => setFilters((f) => { | |
| 180 | + const n = { ...f, ...patch }; | |
| 181 | + for (const k of Object.keys(n) as (keyof FeedFilters)[]) if (n[k] === undefined || n[k] === false || n[k] === "") delete n[k]; | |
| 182 | + return n; | |
| 183 | + }); | |
| 184 | + const activeFilterCount = Object.keys(filters).length; | |
| 98 | 185 | |
| 99 | 186 | return ( |
| 100 | − <section className="panel overflow-hidden"> | |
| 101 | − <header className="flex flex-wrap items-center gap-2 border-b border-line px-3 py-2"> | |
| 102 | − <h2 className="label mr-2 !text-fg">{title}</h2> | |
| 103 | − <LiveDot status={status} /> | |
| 104 | − <span className="ml-auto font-mono text-[11px] text-fg-subtle tabular">{items.length} events</span> | |
| 187 | + <section className="panel overflow-hidden" aria-label={title}> | |
| 188 | + <header className={`flex flex-wrap items-center gap-2 border-b border-line px-3 ${compactHeader ? "py-1.5" : "py-2"}`}> | |
| 189 | + <h2 className="label mr-1 !text-fg">{title}</h2> | |
| 190 | + <LiveDot status={status} paused={paused} /> | |
| 191 | + <span className="hidden font-mono text-[11px] text-fg-subtle tabular sm:inline">{items.length} events</span> | |
| 192 | + <span className="ml-auto flex items-center gap-1"> | |
| 193 | + {showFilters && ( | |
| 194 | + <button type="button" onClick={() => setShowFilterBar((v) => !v)} aria-expanded={showFilterBar} className={`inline-flex h-7 items-center gap-1 rounded-md border border-line px-2 text-[11.5px] ${activeFilterCount ? "border-signal/50 text-signal" : "text-fg-muted hover:text-fg"}`} title="Filters"> | |
| 195 | + <SlidersHorizontal className="size-3.5" /> <span className="hidden sm:inline">Filters</span> | |
| 196 | + {activeFilterCount > 0 && <span className="font-mono">{activeFilterCount}</span>} | |
| 197 | + </button> | |
| 198 | + )} | |
| 199 | + <button type="button" onClick={() => setPaused(!paused)} aria-pressed={paused} className={`inline-flex h-7 items-center gap-1 rounded-md border border-line px-2 text-[11.5px] ${paused ? "border-warn/50 text-warn" : "text-fg-muted hover:text-fg"}`} title={paused ? "Resume live" : "Pause live (backend keeps ingesting)"}> | |
| 200 | + {paused ? <Play className="size-3.5" /> : <Pause className="size-3.5" />} <span className="hidden sm:inline">{paused ? "Resume" : "Pause"}</span> | |
| 201 | + </button> | |
| 202 | + <DensityToggle compact /> | |
| 203 | + </span> | |
| 105 | 204 | </header> |
| 106 | 205 | {showTabs && !fixed && ( |
| 107 | − <div className="flex gap-1 overflow-x-auto border-b border-line px-2 py-1.5 [scrollbar-width:none]"> | |
| 206 | + <div className="flex gap-1 overflow-x-auto border-b border-line px-2 py-1.5 no-scrollbar" role="tablist" aria-label="Channels"> | |
| 108 | 207 | {CHANNELS.map((c) => ( |
| 109 | − <button key={c.key} type="button" onClick={() => setTab(c.key)} className={`whitespace-nowrap rounded-md px-2 py-1 text-[12px] ${tab === c.key ? "bg-panel-2 text-fg" : "text-fg-muted hover:text-fg"} ${c.key === "silent" ? "text-silent" : ""}`}> | |
| 208 | + <button key={c.key} type="button" role="tab" aria-selected={tab === c.key} onClick={() => setTab(c.key)} className={`whitespace-nowrap rounded-md px-2 py-1 text-[12px] ${tab === c.key ? "bg-panel-2 text-fg" : "text-fg-muted hover:text-fg"} ${c.key === "silent" ? "text-silent" : c.key === "breaking" ? "text-hot" : ""}`}> | |
| 110 | 209 | {c.label} |
| 111 | 210 | </button> |
| 112 | 211 | ))} |
| 113 | 212 | </div> |
| 114 | 213 | )} |
| 115 | − <div> | |
| 116 | − {items.length === 0 && !loading && <Empty>No events in this channel yet. Sensors are being checked continuously — meaningful changes will appear here the moment they are detected.</Empty>} | |
| 214 | + {showFilters && showFilterBar && ( | |
| 215 | + <div className="flex flex-wrap items-center gap-1.5 border-b border-line bg-panel-2/40 px-3 py-2 text-[12px]"> | |
| 216 | + <Toggle on={Boolean(filters.silent_change)} onClick={() => setF({ silent_change: !filters.silent_change || undefined })} tone="silent">Silent</Toggle> | |
| 217 | + <Toggle on={Boolean(filters.first_party)} onClick={() => setF({ first_party: !filters.first_party || undefined })} tone="signal">First-party</Toggle> | |
| 218 | + <Toggle on={Boolean(filters.confirmed)} onClick={() => setF({ confirmed: !filters.confirmed || undefined })} tone="ok">Confirmed</Toggle> | |
| 219 | + <select aria-label="Minimum signal" value={filters.signal_min ?? ""} onChange={(e) => setF({ signal_min: e.target.value ? Number(e.target.value) : undefined })} className="h-6 rounded-md border border-line bg-panel px-1.5 text-[11.5px]"> | |
| 220 | + <option value="">signal ≥ any</option> | |
| 221 | + <option value="50">signal ≥ 50</option> | |
| 222 | + <option value="65">signal ≥ 65</option> | |
| 223 | + <option value="80">signal ≥ 80</option> | |
| 224 | + <option value="90">signal ≥ 90</option> | |
| 225 | + </select> | |
| 226 | + <select aria-label="Event group" value={filters.group ?? ""} onChange={(e) => setF({ group: e.target.value || undefined })} className="h-6 max-w-[11rem] rounded-md border border-line bg-panel px-1.5 text-[11.5px]"> | |
| 227 | + <option value="">all groups</option> | |
| 228 | + {Object.entries(GROUP_LABELS).map(([k, v]) => ( | |
| 229 | + <option key={k} value={k}>{v}</option> | |
| 230 | + ))} | |
| 231 | + </select> | |
| 232 | + <select aria-label="Order" value={filters.order ?? "recent"} onChange={(e) => setF({ order: e.target.value === "recent" ? undefined : (e.target.value as FeedFilters["order"]) })} className="h-6 rounded-md border border-line bg-panel px-1.5 text-[11.5px]"> | |
| 233 | + <option value="recent">newest first</option> | |
| 234 | + <option value="signal">highest signal</option> | |
| 235 | + <option value="importance">highest importance</option> | |
| 236 | + </select> | |
| 237 | + <input aria-label="Country" value={filters.country ?? ""} onChange={(e) => setF({ country: e.target.value.toUpperCase().slice(0, 3) || undefined })} placeholder="CC" maxLength={3} className="h-6 w-12 rounded-md border border-line bg-panel px-1.5 font-mono text-[11.5px] uppercase placeholder:text-fg-subtle" title="Country (ISO code)" /> | |
| 238 | + {filters.event_type && <Chip tone="info">type: {filters.event_type.replace(/_/g, " ")} <button type="button" aria-label="Clear type" onClick={() => setF({ event_type: undefined })}><X className="size-3" /></button></Chip>} | |
| 239 | + {filters.q && <Chip tone="info">“{filters.q}” <button type="button" aria-label="Clear search" onClick={() => setF({ q: undefined })}><X className="size-3" /></button></Chip>} | |
| 240 | + <span className="ml-auto flex items-center gap-1"> | |
| 241 | + <span className="text-[11px] text-fg-subtle">views:</span> | |
| 242 | + {SAVED_VIEWS.slice(0, 4).map((v) => ( | |
| 243 | + <Link key={v.key} href={feedHref(v.query, "/live")} className="rounded-sm border border-line px-1.5 py-px text-[10.5px] text-fg-muted hover:text-fg">{v.label}</Link> | |
| 244 | + ))} | |
| 245 | + {activeFilterCount > 0 && ( | |
| 246 | + <button type="button" onClick={() => setFilters({})} className="rounded-sm px-1.5 py-px text-[10.5px] text-fg-subtle hover:text-fg">clear</button> | |
| 247 | + )} | |
| 248 | + </span> | |
| 249 | + </div> | |
| 250 | + )} | |
| 251 | + <div ref={topRef} aria-hidden className="h-px" /> | |
| 252 | + {pending.length > 0 && ( | |
| 253 | + <div className="sticky top-12 z-20 flex justify-center py-1"> | |
| 254 | + <button type="button" onClick={flushPending} className="inline-flex items-center gap-1.5 rounded-full border border-signal/50 bg-panel px-3 py-1 text-[12px] font-medium text-signal shadow-lg hover:bg-signal-soft"> | |
| 255 | + <ArrowUp className="size-3.5" /> {pending.length} new event{pending.length === 1 ? "" : "s"}{paused ? " · paused" : ""} | |
| 256 | + </button> | |
| 257 | + </div> | |
| 258 | + )} | |
| 259 | + <div role="feed" aria-busy={loading}> | |
| 260 | + {items.length === 0 && !loading && ( | |
| 261 | + <Empty> | |
| 262 | + {activeFilterCount ? "No events match these filters. Live events that match will appear here." : "No events in this channel yet. Sensors are being checked continuously — meaningful changes will appear here the moment they are detected."} | |
| 263 | + </Empty> | |
| 264 | + )} | |
| 117 | 265 | {items.map((ev) => ( |
| 118 | − <EventRow key={ev.id} ev={ev} flash={fresh.has(ev.id)} now={now} /> | |
| 266 | + <EventRow key={ev.id} ev={ev} fresh={fresh.has(ev.id)} now={now} replayed={Boolean((ev as EventItem & { replayed?: boolean }).replayed)} /> | |
| 119 | 267 | ))} |
| 120 | 268 | </div> |
| 121 | 269 | <footer className="flex items-center justify-between px-3 py-2 text-[12px] text-fg-subtle"> |
| 122 | − <span> | |
| 270 | + <span className="inline-flex items-center gap-2"> | |
| 123 | 271 | Showing {items.length}{items.length >= MAX_ROWS ? ` (capped at ${MAX_ROWS})` : ""} |
| 272 | + <span className="hidden items-center gap-1 sm:inline-flex"><Kbd>⌘K</Kbd> search</span> | |
| 124 | 273 | </span> |
| 125 | 274 | {cursor ? ( |
| 126 | 275 | <button type="button" disabled={loading} onClick={() => load(false, cursor)} className="rounded-md border border-line bg-panel-2 px-2.5 py-1 text-[12px] text-fg hover:border-line-strong disabled:opacity-50"> |
@@ -133,3 +282,12 @@ export function LiveFeed({ initial, initialCursor, fixed, showTabs = true, title | ||
| 133 | 282 | </section> |
| 134 | 283 | ); |
| 135 | 284 | } |
| 285 | + | |
| 286 | +function Toggle({ on, onClick, children, tone }: { on: boolean; onClick: () => void; children: React.ReactNode; tone: "silent" | "signal" | "ok" }) { | |
| 287 | + const onCls = tone === "silent" ? "border-silent/50 bg-silent-soft text-silent" : tone === "ok" ? "border-ok/50 bg-ok/10 text-ok" : "border-signal/50 bg-signal-soft text-signal"; | |
| 288 | + return ( | |
| 289 | + <button type="button" aria-pressed={on} onClick={onClick} className={`h-6 rounded-md border px-2 text-[11.5px] ${on ? onCls : "border-line text-fg-muted hover:text-fg"}`}> | |
| 290 | + {children} | |
| 291 | + </button> | |
| 292 | + ); | |
| 293 | +} | |
added
apps/web/src/components/live-strip.tsx
+76 −0
@@ -0,0 +1,76 @@ | ||
| 1 | +"use client"; | |
| 2 | + | |
| 3 | +import Link from "next/link"; | |
| 4 | +import { useEffect, useRef, useState } from "react"; | |
| 5 | +import type { Stats } from "@/lib/api"; | |
| 6 | +import { fmtInt, relTime } from "@/lib/format"; | |
| 7 | +import { publicFetch } from "@/lib/owner"; | |
| 8 | + | |
| 9 | +/** | |
| 10 | + * Live system strip (spec §65): SOURCES · SENSORS · CHECKS/MIN · EVENTS 24H · BREAKING · SILENT · STATUS. | |
| 11 | + * Server-rendered with real values, then refreshed every 10 s. Numbers tick when they change. | |
| 12 | + */ | |
| 13 | +export function LiveStrip({ initial }: { initial: Stats }) { | |
| 14 | + const [s, setS] = useState<Stats>(initial); | |
| 15 | + const [now, setNow] = useState(() => Date.now()); | |
| 16 | + useEffect(() => { | |
| 17 | + const tick = (): void => { | |
| 18 | + publicFetch<Stats>("/api/v1/stats").then(setS).catch(() => undefined); | |
| 19 | + setNow(Date.now()); | |
| 20 | + }; | |
| 21 | + const t = setInterval(tick, 10_000); | |
| 22 | + const n = setInterval(() => setNow(Date.now()), 1000); | |
| 23 | + return () => { | |
| 24 | + clearInterval(t); | |
| 25 | + clearInterval(n); | |
| 26 | + }; | |
| 27 | + }, []); | |
| 28 | + const lastCheckAge = s.last_check_at ? now - new Date(s.last_check_at).getTime() : Infinity; | |
| 29 | + const live = lastCheckAge < 120_000; | |
| 30 | + // `now` differs between server and client renders; the status word is re-rendered right after mount. | |
| 31 | + const items: { label: string; value: number | undefined; href?: string; tone?: string; hint?: string }[] = [ | |
| 32 | + { label: "Sources", value: s.sources, href: "/sources", hint: `${fmtInt(s.sources_first_party)} first-party` }, | |
| 33 | + { label: "Sensors", value: s.sensors, href: "/health", hint: s.sensors_degraded ? `${fmtInt(s.sensors_degraded)} degraded` : "all healthy" }, | |
| 34 | + { label: "Checks / min", value: s.checks_per_min, href: "/health", hint: s.not_modified_ratio_5m !== null && s.not_modified_ratio_5m !== undefined ? `${Math.round(s.not_modified_ratio_5m * 100)}% 304` : undefined }, | |
| 35 | + { label: "Events 24h", value: s.events_24h, href: "/live", hint: s.events_per_min !== undefined ? `${s.events_per_min}/min` : undefined }, | |
| 36 | + { label: "Breaking", value: s.breaking_now ?? s.breaking_24h, href: "/breaking", tone: "text-hot", hint: s.developing_now ? `${s.developing_now} developing` : undefined }, | |
| 37 | + { label: "Silent 24h", value: s.silent_24h, href: "/silent", tone: "text-silent" }, | |
| 38 | + ]; | |
| 39 | + return ( | |
| 40 | + <div className="panel mb-4 flex items-stretch overflow-x-auto no-scrollbar" role="region" aria-label="Live system status"> | |
| 41 | + {items.map((it) => ( | |
| 42 | + <Cell key={it.label} {...it} /> | |
| 43 | + ))} | |
| 44 | + <div className="ml-auto flex min-w-[7.5rem] flex-col justify-center gap-0.5 border-l border-line px-3 py-2"> | |
| 45 | + <span className="label">Status</span> | |
| 46 | + <span className="inline-flex items-center gap-1.5 font-mono text-[12px] font-semibold tracking-wider" suppressHydrationWarning> | |
| 47 | + <span className={`inline-block size-2 rounded-full ${live ? "bg-signal animate-pulse-dot" : "bg-warn"}`} /> | |
| 48 | + {live ? "LIVE" : "STALE"} | |
| 49 | + </span> | |
| 50 | + <span className="truncate text-[10.5px] text-fg-subtle" suppressHydrationWarning>{s.last_check_at ? `last check ${relTime(s.last_check_at, now)}` : "—"}</span> | |
| 51 | + </div> | |
| 52 | + </div> | |
| 53 | + ); | |
| 54 | +} | |
| 55 | + | |
| 56 | +function Cell({ label, value, href, tone, hint }: { label: string; value: number | undefined; href?: string; tone?: string; hint?: string }) { | |
| 57 | + const prev = useRef(value); | |
| 58 | + const [tick, setTick] = useState(false); | |
| 59 | + useEffect(() => { | |
| 60 | + if (prev.current !== value) { | |
| 61 | + prev.current = value; | |
| 62 | + setTick(true); | |
| 63 | + const t = setTimeout(() => setTick(false), 600); | |
| 64 | + return () => clearTimeout(t); | |
| 65 | + } | |
| 66 | + }, [value]); | |
| 67 | + const body = ( | |
| 68 | + <> | |
| 69 | + <span className="label">{label}</span> | |
| 70 | + <span className={`font-mono text-lg font-semibold leading-tight tabular ${tone ?? ""} ${tick ? "animate-tick" : ""}`}>{fmtInt(value)}</span> | |
| 71 | + {hint && <span className="truncate text-[10.5px] text-fg-subtle">{hint}</span>} | |
| 72 | + </> | |
| 73 | + ); | |
| 74 | + const cls = "flex min-w-[7.25rem] flex-col gap-0.5 border-r border-line px-3 py-2 last:border-r-0"; | |
| 75 | + return href ? <Link href={href} className={`${cls} hover:bg-panel-2/60`}>{body}</Link> : <div className={cls}>{body}</div>; | |
| 76 | +} | |
modified
apps/web/src/components/nav.tsx
+78 −52
@@ -1,27 +1,29 @@ | ||
| 1 | 1 | "use client"; |
| 2 | 2 | |
| 3 | 3 | import Link from "next/link"; |
| 4 | −import { usePathname, useRouter } from "next/navigation"; | |
| 5 | −import { Activity, Bell, Compass, Eye, Search, User } from "lucide-react"; | |
| 6 | −import { useEffect, useRef, useState } from "react"; | |
| 4 | +import { usePathname } from "next/navigation"; | |
| 5 | +import { Activity, Bell, Bookmark, Compass, Eye, Globe2, LayoutGrid, Radar, Search, Siren, X, Zap } from "lucide-react"; | |
| 6 | +import { useState } from "react"; | |
| 7 | +import { DensityToggle } from "./prefs"; | |
| 7 | 8 | import { ThemeToggle } from "./theme"; |
| 9 | +import { Kbd } from "./ui"; | |
| 8 | 10 | |
| 9 | 11 | const NAV = [ |
| 10 | − ["/", "Live"], | |
| 12 | + ["/live", "Live"], | |
| 11 | 13 | ["/breaking", "Breaking"], |
| 14 | + ["/pulse", "Pulse"], | |
| 15 | + ["/radar", "Radar"], | |
| 16 | + ["/silent", "Silent"], | |
| 12 | 17 | ["/explore", "Explore"], |
| 13 | − ["/sources", "Sources"], | |
| 14 | 18 | ["/entities", "Entities"], |
| 15 | − ["/silent", "Silent"], | |
| 19 | + ["/sources", "Sources"], | |
| 16 | 20 | ["/watchlists", "Watchlists"], |
| 17 | 21 | ["/alerts", "Alerts"], |
| 18 | − ["/api", "API"], | |
| 19 | − ["/health", "Health"], | |
| 20 | 22 | ] as const; |
| 21 | 23 | |
| 22 | 24 | export function Wordmark() { |
| 23 | 25 | return ( |
| 24 | − <Link href="/" className="flex items-center gap-2 font-semibold tracking-tight"> | |
| 26 | + <Link href="/" className="flex items-center gap-2 font-semibold tracking-tight" aria-label="WebSensor home"> | |
| 25 | 27 | <span className="relative inline-flex size-2.5 items-center justify-center"> |
| 26 | 28 | <span className="absolute inset-0 rounded-full bg-signal animate-pulse-dot" /> |
| 27 | 29 | </span> |
@@ -34,72 +36,96 @@ export function Wordmark() { | ||
| 34 | 36 | |
| 35 | 37 | export function TopNav() { |
| 36 | 38 | const path = usePathname(); |
| 37 | − const router = useRouter(); | |
| 38 | − const [q, setQ] = useState(""); | |
| 39 | − const input = useRef<HTMLInputElement>(null); | |
| 40 | − useEffect(() => { | |
| 41 | − const onKey = (e: KeyboardEvent): void => { | |
| 42 | − if ((e.metaKey || e.ctrlKey) && e.key.toLowerCase() === "k") { | |
| 43 | − e.preventDefault(); | |
| 44 | − input.current?.focus(); | |
| 45 | − } | |
| 46 | − }; | |
| 47 | − window.addEventListener("keydown", onKey); | |
| 48 | − return () => window.removeEventListener("keydown", onKey); | |
| 49 | − }, []); | |
| 39 | + const openPalette = (): void => { | |
| 40 | + window.dispatchEvent(new Event("ws:palette")); | |
| 41 | + }; | |
| 50 | 42 | return ( |
| 51 | 43 | <header className="sticky top-0 z-40 border-b border-line bg-bg/90 backdrop-blur"> |
| 52 | − <div className="mx-auto flex h-12 max-w-[1500px] items-center gap-4 px-3 sm:px-4"> | |
| 44 | + <div className="mx-auto flex h-12 max-w-[1600px] items-center gap-3 px-3 sm:px-4"> | |
| 53 | 45 | <Wordmark /> |
| 54 | − <nav className="hidden items-center gap-0.5 lg:flex"> | |
| 46 | + <nav className="hidden items-center gap-0.5 lg:flex" aria-label="Primary"> | |
| 55 | 47 | {NAV.map(([href, label]) => { |
| 56 | − const active = href === "/" ? path === "/" : path.startsWith(href); | |
| 48 | + const active = href === "/live" ? path === "/" || path.startsWith("/live") : path.startsWith(href); | |
| 57 | 49 | return ( |
| 58 | − <Link key={href} href={href} className={`rounded-md px-2 py-1 text-[12.5px] ${active ? "bg-panel-2 text-fg" : "text-fg-muted hover:text-fg"}`}> | |
| 50 | + <Link key={href} href={href} aria-current={active ? "page" : undefined} className={`rounded-md px-1.5 py-1 text-[12px] xl:px-2 xl:text-[12.5px] ${active ? "bg-panel-2 text-fg" : "text-fg-muted hover:text-fg"} ${href === "/breaking" && active ? "!text-hot" : href === "/silent" && active ? "!text-silent" : ""}`}> | |
| 59 | 51 | {label} |
| 60 | 52 | </Link> |
| 61 | 53 | ); |
| 62 | 54 | })} |
| 63 | 55 | </nav> |
| 64 | − <form | |
| 65 | − className="ml-auto flex items-center" | |
| 66 | − onSubmit={(e) => { | |
| 67 | − e.preventDefault(); | |
| 68 | − if (q.trim()) router.push(`/search?q=${encodeURIComponent(q.trim())}`); | |
| 69 | − }} | |
| 70 | − > | |
| 71 | − <label className="relative block"> | |
| 72 | − <Search className="pointer-events-none absolute left-2 top-1/2 size-3.5 -translate-y-1/2 text-fg-subtle" /> | |
| 73 | − <input ref={input} value={q} onChange={(e) => setQ(e.target.value)} placeholder="Search events, entities, URLs…" className="h-8 w-40 rounded-md border border-line bg-panel pl-7 pr-9 text-[12.5px] placeholder:text-fg-subtle focus:w-64 sm:w-56 sm:focus:w-80 transition-[width]" /> | |
| 74 | − <kbd className="pointer-events-none absolute right-1.5 top-1/2 hidden -translate-y-1/2 rounded border border-line bg-panel-2 px-1 font-mono text-[10px] text-fg-subtle sm:block">⌘K</kbd> | |
| 75 | − </label> | |
| 76 | − </form> | |
| 56 | + <button type="button" onClick={openPalette} className="ml-auto inline-flex h-8 w-40 min-w-0 items-center gap-2 rounded-md border border-line bg-panel px-2 text-[12.5px] text-fg-subtle hover:border-line-strong sm:w-56 lg:w-48 xl:w-72" aria-label="Open search and commands (⌘K)"> | |
| 57 | + <Search className="size-3.5 shrink-0" /> | |
| 58 | + <span className="flex-1 truncate text-left">Search entities, events, URLs…</span> | |
| 59 | + <span className="hidden sm:inline-flex"><Kbd>⌘K</Kbd></span> | |
| 60 | + </button> | |
| 61 | + <div className="hidden xl:block"><DensityToggle compact /></div> | |
| 77 | 62 | <ThemeToggle /> |
| 78 | 63 | </div> |
| 79 | 64 | </header> |
| 80 | 65 | ); |
| 81 | 66 | } |
| 82 | 67 | |
| 68 | +/** Mobile bottom navigation (spec §95): Live · Breaking · Explore · Watchlists · More (bottom sheet). */ | |
| 83 | 69 | export function MobileNav() { |
| 84 | 70 | const path = usePathname(); |
| 71 | + const [moreFor, setMoreFor] = useState<string | null>(null); | |
| 72 | + const more = moreFor === path; | |
| 73 | + const setMore = (v: boolean): void => setMoreFor(v ? path : null); | |
| 85 | 74 | const items = [ |
| 86 | − ["/", "Live", Activity], | |
| 87 | − ["/explore", "Explore", Compass], | |
| 88 | − ["/alerts", "Alerts", Bell], | |
| 89 | − ["/watchlists", "Watchlist", Eye], | |
| 90 | − ["/watchlists", "Profile", User], | |
| 75 | + ["/live", "Live", Activity, path === "/" || path.startsWith("/live")], | |
| 76 | + ["/breaking", "Breaking", Siren, path.startsWith("/breaking")], | |
| 77 | + ["/explore", "Explore", Compass, path.startsWith("/explore")], | |
| 78 | + ["/watchlists", "Watch", Eye, path.startsWith("/watchlists")], | |
| 91 | 79 | ] as const; |
| 80 | + const moreItems: [string, string, typeof Activity][] = [ | |
| 81 | + ["/pulse", "Pulse", Zap], | |
| 82 | + ["/radar", "Radar", Radar], | |
| 83 | + ["/silent", "Silent changes", LayoutGrid], | |
| 84 | + ["/entities", "Entities", Globe2], | |
| 85 | + ["/sources", "Sources", Globe2], | |
| 86 | + ["/country", "Countries", Globe2], | |
| 87 | + ["/alerts", "Alerts", Bell], | |
| 88 | + ["/bookmarks", "Bookmarks", Bookmark], | |
| 89 | + ["/health", "Health", Activity], | |
| 90 | + ["/api", "API", Zap], | |
| 91 | + ]; | |
| 92 | 92 | return ( |
| 93 | − <nav className="fixed inset-x-0 bottom-0 z-40 grid grid-cols-5 border-t border-line bg-bg/95 backdrop-blur lg:hidden" style={{ paddingBottom: "env(safe-area-inset-bottom)" }}> | |
| 94 | − {items.map(([href, label, Icon], i) => { | |
| 95 | − const active = href === "/" ? path === "/" : path.startsWith(href) && !(i === 4 && label === "Profile" && path.startsWith("/watchlists") && false); | |
| 96 | − return ( | |
| 97 | − <Link key={label} href={href} className={`flex flex-col items-center gap-0.5 py-2 text-[10.5px] ${active && !(i === 4) ? "text-signal" : "text-fg-muted"}`}> | |
| 93 | + <> | |
| 94 | + <nav className="fixed inset-x-0 bottom-0 z-40 grid grid-cols-5 border-t border-line bg-bg/95 backdrop-blur lg:hidden" style={{ paddingBottom: "env(safe-area-inset-bottom)" }} aria-label="Mobile"> | |
| 95 | + {items.map(([href, label, Icon, active]) => ( | |
| 96 | + <Link key={href} href={href} aria-current={active ? "page" : undefined} className={`flex flex-col items-center gap-0.5 py-2 text-[10.5px] ${active ? (href === "/breaking" ? "text-hot" : "text-signal") : "text-fg-muted"}`}> | |
| 98 | 97 | <Icon className="size-4" /> |
| 99 | 98 | {label} |
| 100 | 99 | </Link> |
| 101 | − ); | |
| 102 | − })} | |
| 103 | − </nav> | |
| 100 | + ))} | |
| 101 | + <button type="button" onClick={() => setMore(true)} aria-expanded={more} className={`flex flex-col items-center gap-0.5 py-2 text-[10.5px] ${more ? "text-signal" : "text-fg-muted"}`}> | |
| 102 | + <LayoutGrid className="size-4" /> | |
| 103 | + More | |
| 104 | + </button> | |
| 105 | + </nav> | |
| 106 | + {more && ( | |
| 107 | + <div className="fixed inset-0 z-50 flex items-end lg:hidden" role="dialog" aria-modal="true" aria-label="More"> | |
| 108 | + <button type="button" aria-label="Close" onClick={() => setMore(false)} className="scrim absolute inset-0 cursor-default" /> | |
| 109 | + <div className="relative w-full rounded-t-xl border-t border-line bg-panel pb-6 animate-sheet-up" style={{ paddingBottom: "calc(1.5rem + env(safe-area-inset-bottom))" }}> | |
| 110 | + <div className="flex items-center justify-between px-4 py-3"> | |
| 111 | + <span className="label">More</span> | |
| 112 | + <span className="flex items-center gap-2"> | |
| 113 | + <DensityToggle /> | |
| 114 | + <button type="button" onClick={() => setMore(false)} aria-label="Close" className="inline-flex size-7 items-center justify-center rounded-md border border-line"><X className="size-4" /></button> | |
| 115 | + </span> | |
| 116 | + </div> | |
| 117 | + <ul className="grid grid-cols-2 gap-1 px-3"> | |
| 118 | + {moreItems.map(([href, label, Icon]) => ( | |
| 119 | + <li key={href}> | |
| 120 | + <Link href={href} className="flex items-center gap-2 rounded-md border border-line px-3 py-2.5 text-[13px] hover:bg-panel-2"> | |
| 121 | + <Icon className="size-4 text-fg-muted" /> {label} | |
| 122 | + </Link> | |
| 123 | + </li> | |
| 124 | + ))} | |
| 125 | + </ul> | |
| 126 | + </div> | |
| 127 | + </div> | |
| 128 | + )} | |
| 129 | + </> | |
| 104 | 130 | ); |
| 105 | 131 | } |
added
apps/web/src/components/prefs.tsx
+87 −0
@@ -0,0 +1,87 @@ | ||
| 1 | +"use client"; | |
| 2 | + | |
| 3 | +import { createContext, useCallback, useContext, useEffect, useMemo, useState, useSyncExternalStore, type ReactNode } from "react"; | |
| 4 | + | |
| 5 | +/** | |
| 6 | + * User preferences (spec §32, §98, §110): density (compact / normal / comfortable), live paused, | |
| 7 | + * exact timestamps. Persisted in localStorage; density is mirrored on <html data-density> so CSS | |
| 8 | + * variables apply everywhere without re-rendering the tree. | |
| 9 | + */ | |
| 10 | +export type Density = "compact" | "normal" | "comfortable"; | |
| 11 | + | |
| 12 | +interface Prefs { | |
| 13 | + density: Density; | |
| 14 | + setDensity: (d: Density) => void; | |
| 15 | + paused: boolean; | |
| 16 | + setPaused: (p: boolean) => void; | |
| 17 | + exactTime: boolean; | |
| 18 | + setExactTime: (v: boolean) => void; | |
| 19 | + mounted: boolean; | |
| 20 | +} | |
| 21 | + | |
| 22 | +const Ctx = createContext<Prefs | null>(null); | |
| 23 | +const KEY = "ws_prefs"; | |
| 24 | + | |
| 25 | +export function PrefsProvider({ children }: { children: ReactNode }) { | |
| 26 | + // Stored preferences are applied once, right after hydration (reading storage during SSR would mismatch). | |
| 27 | + const mounted = useSyncExternalStore(subscribeNoop, () => true, () => false); | |
| 28 | + const stored = useSyncExternalStore(subscribeNoop, readStored, () => null); | |
| 29 | + const [override, setOverride] = useState<{ density?: Density; exactTime?: boolean }>({}); | |
| 30 | + const density = override.density ?? stored?.density ?? "normal"; | |
| 31 | + const exactTime = override.exactTime ?? stored?.exactTime ?? false; | |
| 32 | + const [paused, setPaused] = useState(false); | |
| 33 | + const setDensityState = (d: Density): void => setOverride((o) => ({ ...o, density: d })); | |
| 34 | + const setExactTimeState = (v: boolean): void => setOverride((o) => ({ ...o, exactTime: v })); | |
| 35 | + useEffect(() => { | |
| 36 | + if (!mounted) return; | |
| 37 | + document.documentElement.dataset.density = density; | |
| 38 | + try { | |
| 39 | + window.localStorage.setItem(KEY, JSON.stringify({ density, exactTime })); | |
| 40 | + } catch { | |
| 41 | + // ignore | |
| 42 | + } | |
| 43 | + }, [density, exactTime, mounted]); | |
| 44 | + | |
| 45 | + const setDensity = useCallback((d: Density) => setDensityState(d), []); | |
| 46 | + const setExactTime = useCallback((v: boolean) => setExactTimeState(v), []); | |
| 47 | + const value = useMemo(() => ({ density, setDensity, paused, setPaused, exactTime, setExactTime, mounted }), [density, setDensity, paused, exactTime, setExactTime, mounted]); | |
| 48 | + return <Ctx.Provider value={value}>{children}</Ctx.Provider>; | |
| 49 | +} | |
| 50 | + | |
| 51 | +const subscribeNoop = (): (() => void) => () => {}; | |
| 52 | +let storedCache: { raw: string | null; value: { density?: Density; exactTime?: boolean } | null } = { raw: undefined as unknown as string | null, value: null }; | |
| 53 | +function readStored(): { density?: Density; exactTime?: boolean } | null { | |
| 54 | + try { | |
| 55 | + const raw = window.localStorage.getItem(KEY); | |
| 56 | + if (raw === storedCache.raw) return storedCache.value; | |
| 57 | + const value = raw ? (JSON.parse(raw) as { density?: Density; exactTime?: boolean }) : null; | |
| 58 | + storedCache = { raw, value }; | |
| 59 | + return value; | |
| 60 | + } catch { | |
| 61 | + return null; | |
| 62 | + } | |
| 63 | +} | |
| 64 | + | |
| 65 | +export function usePrefs(): Prefs { | |
| 66 | + const v = useContext(Ctx); | |
| 67 | + if (!v) return { density: "normal", setDensity: () => {}, paused: false, setPaused: () => {}, exactTime: false, setExactTime: () => {}, mounted: false }; | |
| 68 | + return v; | |
| 69 | +} | |
| 70 | + | |
| 71 | +export function DensityToggle({ compact = false }: { compact?: boolean }) { | |
| 72 | + const { density, setDensity } = usePrefs(); | |
| 73 | + const opts: [Density, string][] = [ | |
| 74 | + ["compact", "Compact"], | |
| 75 | + ["normal", "Normal"], | |
| 76 | + ["comfortable", "Comfortable"], | |
| 77 | + ]; | |
| 78 | + return ( | |
| 79 | + <div role="radiogroup" aria-label="Density" className="inline-flex items-center rounded-md border border-line bg-panel p-0.5"> | |
| 80 | + {opts.map(([d, label]) => ( | |
| 81 | + <button key={d} type="button" role="radio" aria-checked={density === d} onClick={() => setDensity(d)} title={label} className={`rounded-sm px-1.5 py-0.5 text-[11px] ${density === d ? "bg-panel-3 text-fg" : "text-fg-subtle hover:text-fg"}`}> | |
| 82 | + {compact ? label[0] : label} | |
| 83 | + </button> | |
| 84 | + ))} | |
| 85 | + </div> | |
| 86 | + ); | |
| 87 | +} | |
modified
apps/web/src/components/rail.tsx
+108 −17
@@ -1,26 +1,27 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | −import type { Cluster, TrendingItem } from "@/lib/api"; | |
| 3 | −import { fmtScore, relTime } from "@/lib/format"; | |
| 4 | −import { Empty, Panel, Score } from "./ui"; | |
| 2 | +import type { Cluster, EventItem, TrendingItem } from "@/lib/api"; | |
| 3 | +import { fmtOffset, fmtScore, relTime } from "@/lib/format"; | |
| 4 | +import { Badge, Chip, Empty, Mark, Panel, Score, StateLabelText } from "./ui"; | |
| 5 | 5 | |
| 6 | −export function TrendingPanel({ items }: { items: TrendingItem[] }) { | |
| 6 | +export function TrendingPanel({ items, title = "Trending entities", hours = 24 }: { items: TrendingItem[]; title?: string; hours?: number }) { | |
| 7 | 7 | return ( |
| 8 | − <Panel title="Trending now" dense action={<Link href="/entities" className="text-[11px] text-fg-subtle hover:text-fg">all entities →</Link>}> | |
| 8 | + <Panel title={`${title} · ${hours} h`} dense action={<Link href="/explore?tab=trending" className="text-[11px] text-fg-subtle hover:text-fg">all →</Link>}> | |
| 9 | 9 | {items.length === 0 ? ( |
| 10 | − <Empty>Trending is computed from the last 24 h of events.</Empty> | |
| 10 | + <Empty>Trending is computed from the last {hours} h of events.</Empty> | |
| 11 | 11 | ) : ( |
| 12 | 12 | <ol className="divide-y divide-line"> |
| 13 | 13 | {items.map((t, i) => ( |
| 14 | − <li key={t.id} className="grid grid-cols-[1.5rem_1fr_auto] items-center gap-2 px-3 py-1.5 text-[13px]"> | |
| 15 | − <span className="font-mono text-[12px] text-fg-subtle tabular">{i + 1}</span> | |
| 14 | + <li key={t.id} className="grid grid-cols-[1.25rem_1.25rem_1fr_auto] items-center gap-2 px-3 py-1.5 text-[13px]"> | |
| 15 | + <span className="font-mono text-[11px] text-fg-subtle tabular">{i + 1}</span> | |
| 16 | + <Mark name={t.name} /> | |
| 16 | 17 | <div className="min-w-0"> |
| 17 | − <Link href={`/company/${t.id}`} className="block truncate font-medium hover:underline">{t.name}</Link> | |
| 18 | + <Link href={`/entity/${t.id}`} className="block truncate font-medium hover:underline">{t.name}</Link> | |
| 18 | 19 | <div className="truncate text-[11px] text-fg-subtle"> |
| 19 | − {t.events} events · {t.sources} source{t.sources === 1 ? "" : "s"}{t.silent ? ` · ${t.silent} silent` : ""} | |
| 20 | + {t.events} signals · {t.sources} source{t.sources === 1 ? "" : "s"}{t.first_party ? ` · ${t.first_party} first-party` : ""}{t.silent ? ` · ${t.silent} silent` : ""} | |
| 20 | 21 | </div> |
| 21 | 22 | </div> |
| 22 | − <span className={`font-mono text-[12px] font-semibold tabular ${t.events > t.prev_events ? "text-signal" : "text-fg-muted"}`}> | |
| 23 | − {t.events > t.prev_events ? "↑" : t.events < t.prev_events ? "↓" : "→"} {fmtScore(t.score)} | |
| 23 | + <span className={`font-mono text-[12px] font-semibold tabular ${t.direction === "up" ? "text-signal" : t.direction === "down" ? "text-fg-subtle" : "text-fg-muted"}`} title={`trend score ${fmtScore(t.score)} · ${t.events} vs ${t.prev_events} in the previous window`}> | |
| 24 | + {t.direction === "up" ? "↑" : t.direction === "down" ? "↓" : "→"} {fmtScore(t.score)} | |
| 24 | 25 | </span> |
| 25 | 26 | </li> |
| 26 | 27 | ))} |
@@ -30,10 +31,10 @@ export function TrendingPanel({ items }: { items: TrendingItem[] }) { | ||
| 30 | 31 | ); |
| 31 | 32 | } |
| 32 | 33 | |
| 33 | −export function ClustersPanel({ items }: { items: Cluster[] }) { | |
| 34 | +export function ClustersPanel({ items, title = "Event clusters" }: { items: Cluster[]; title?: string }) { | |
| 34 | 35 | const multi = items.filter((c) => c.event_count > 1).slice(0, 8); |
| 35 | 36 | return ( |
| 36 | − <Panel title="Event clusters" dense> | |
| 37 | + <Panel title={title} dense action={<Link href="/explore?tab=clusters" className="text-[11px] text-fg-subtle hover:text-fg">all →</Link>}> | |
| 37 | 38 | {multi.length === 0 ? ( |
| 38 | 39 | <Empty>Related observations are grouped into clusters as they arrive.</Empty> |
| 39 | 40 | ) : ( |
@@ -41,9 +42,16 @@ export function ClustersPanel({ items }: { items: Cluster[] }) { | ||
| 41 | 42 | {multi.map((c) => ( |
| 42 | 43 | <li key={c.id} className="flex items-start gap-2 px-3 py-1.5 text-[13px]"> |
| 43 | 44 | <Score value={c.max_importance} size="sm" /> |
| 44 | − <div className="min-w-0"> | |
| 45 | − <Link href={c.events?.[0]?.slug ? `/event/${c.events[0].slug}` : "/explore"} className="line-clamp-2 font-medium hover:underline">{c.title}</Link> | |
| 46 | − <div className="text-[11px] text-fg-subtle">{c.event_count} observations · {relTime(c.last_at)}</div> | |
| 45 | + <div className="min-w-0 flex-1"> | |
| 46 | + <Link href={`/cluster/${c.slug ?? c.id}`} className="line-clamp-2 font-medium hover:underline">{c.title}</Link> | |
| 47 | + <div className="flex flex-wrap items-center gap-x-2 text-[11px] text-fg-subtle"> | |
| 48 | + <StateLabelText state={c.state} /> | |
| 49 | + <span>{c.event_count} signals</span> | |
| 50 | + {c.source_count && c.source_count > 1 && <span>{c.source_count} sources</span>} | |
| 51 | + {(c.first_party_count ?? 0) > 0 && <span className="text-signal">{c.first_party_count} 1st-party</span>} | |
| 52 | + {c.lead_time_ms && c.lead_time_ms > 0 ? <span title="WebSensor lead time before the first external report">lead {fmtOffset(c.lead_time_ms).replace("+", "")}</span> : null} | |
| 53 | + <span>{relTime(c.last_at)}</span> | |
| 54 | + </div> | |
| 47 | 55 | </div> |
| 48 | 56 | </li> |
| 49 | 57 | ))} |
@@ -52,3 +60,86 @@ export function ClustersPanel({ items }: { items: Cluster[] }) { | ||
| 52 | 60 | </Panel> |
| 53 | 61 | ); |
| 54 | 62 | } |
| 63 | + | |
| 64 | +/** Breaking now rail (spec §94): primary event of each breaking / developing cluster. */ | |
| 65 | +export function BreakingRail({ items, developing }: { items: Cluster[]; developing?: Cluster[] }) { | |
| 66 | + const all = [...items.map((c) => ({ c, state: "breaking" })), ...(developing ?? []).map((c) => ({ c, state: "developing" }))].slice(0, 8); | |
| 67 | + return ( | |
| 68 | + <Panel title={<span className="text-hot">Breaking now</span>} dense action={<Link href="/breaking" className="text-[11px] text-fg-subtle hover:text-fg">desk →</Link>}> | |
| 69 | + {all.length === 0 ? ( | |
| 70 | + <Empty>Nothing is breaking right now. Breaking requires strong signal, freshness and confirmation — not just recency.</Empty> | |
| 71 | + ) : ( | |
| 72 | + <ul className="divide-y divide-line"> | |
| 73 | + {all.map(({ c, state }) => { | |
| 74 | + const e = c.event as EventItem | null | undefined; | |
| 75 | + return ( | |
| 76 | + <li key={c.id} className={`px-3 py-2 text-[13px] ${state === "breaking" ? "border-l-2 border-l-hot/70" : "border-l-2 border-l-high/60"}`}> | |
| 77 | + <div className="flex items-center gap-1.5"> | |
| 78 | + <Badge kind={state as "breaking" | "developing"} compact /> | |
| 79 | + <span className="truncate font-mono text-[10.5px] uppercase text-fg-subtle">{e?.source?.name ?? c.source?.name ?? ""}</span> | |
| 80 | + <span className="ml-auto font-mono text-[10.5px] text-fg-subtle">{relTime(c.last_at)}</span> | |
| 81 | + </div> | |
| 82 | + <Link href={e?.slug ? `/event/${e.slug}` : `/cluster/${c.slug ?? c.id}`} className="mt-0.5 line-clamp-2 font-medium hover:underline">{c.title}</Link> | |
| 83 | + <div className="mt-0.5 flex flex-wrap items-center gap-1 text-[11px] text-fg-subtle"> | |
| 84 | + <Score value={e?.signal_score ?? c.max_importance} size="sm" kind="signal" /> | |
| 85 | + <span>{c.event_count} signal{c.event_count === 1 ? "" : "s"} · {c.source_count ?? 1} source{(c.source_count ?? 1) === 1 ? "" : "s"}</span> | |
| 86 | + {(c.first_party_count ?? 0) > 0 && <Chip tone="signal">{c.first_party_count} first-party</Chip>} | |
| 87 | + </div> | |
| 88 | + </li> | |
| 89 | + ); | |
| 90 | + })} | |
| 91 | + </ul> | |
| 92 | + )} | |
| 93 | + </Panel> | |
| 94 | + ); | |
| 95 | +} | |
| 96 | + | |
| 97 | +export function AnomalyRail({ items }: { items: { id: string; name: string; domain: string; changes_2h: number; baseline_per_day: number; activity_score: number; pct_vs_baseline?: number | null }[] }) { | |
| 98 | + const top = items.filter((s) => s.activity_score >= 45).slice(0, 6); | |
| 99 | + return ( | |
| 100 | + <Panel title="Anomalous activity" dense action={<Link href="/radar" className="text-[11px] text-fg-subtle hover:text-fg">radar →</Link>}> | |
| 101 | + {top.length === 0 ? ( | |
| 102 | + <Empty>No source is far above its baseline right now.</Empty> | |
| 103 | + ) : ( | |
| 104 | + <ul className="divide-y divide-line"> | |
| 105 | + {top.map((s) => ( | |
| 106 | + <li key={s.id} className="px-3 py-1.5 text-[13px]"> | |
| 107 | + <div className="flex items-center gap-2"> | |
| 108 | + <Link href={`/source/${s.id}`} className="min-w-0 flex-1 truncate font-medium hover:underline">{s.name}</Link> | |
| 109 | + <span className={`font-mono text-[12px] font-semibold tabular ${s.activity_score >= 70 ? "text-hot" : "text-high"}`}>{s.pct_vs_baseline !== null && s.pct_vs_baseline !== undefined ? `+${Math.min(9999, Math.round(s.pct_vs_baseline)).toLocaleString()}%` : fmtScore(s.activity_score)}</span> | |
| 110 | + </div> | |
| 111 | + <div className="text-[11px] text-fg-subtle">{s.changes_2h} changes in 2 h · baseline {s.baseline_per_day}/day</div> | |
| 112 | + </li> | |
| 113 | + ))} | |
| 114 | + </ul> | |
| 115 | + )} | |
| 116 | + </Panel> | |
| 117 | + ); | |
| 118 | +} | |
| 119 | + | |
| 120 | +export function SilentRail({ items }: { items: EventItem[] }) { | |
| 121 | + return ( | |
| 122 | + <Panel title={<span className="text-silent">Silent changes</span>} dense action={<Link href="/silent" className="text-[11px] text-fg-subtle hover:text-fg">all →</Link>}> | |
| 123 | + {items.length === 0 ? ( | |
| 124 | + <Empty>No silent change in this window.</Empty> | |
| 125 | + ) : ( | |
| 126 | + <ul className="divide-y divide-line"> | |
| 127 | + {items.slice(0, 6).map((e) => ( | |
| 128 | + <li key={e.id} className="px-3 py-1.5 text-[13px]"> | |
| 129 | + <div className="flex items-center gap-2 text-[11px] text-fg-subtle"> | |
| 130 | + <span className="truncate font-mono uppercase">{e.source?.name}</span> | |
| 131 | + <span className="ml-auto">{relTime(e.detected_at)}</span> | |
| 132 | + </div> | |
| 133 | + <Link href={`/event/${e.slug}`} className="line-clamp-2 hover:underline">{e.title}</Link> | |
| 134 | + {e.field_changes?.[0] && ( | |
| 135 | + <div className="mt-0.5 truncate font-mono text-[11px] text-fg-muted"> | |
| 136 | + {e.field_changes[0].label}: <span className="diff-line-del rounded px-0.5">{e.field_changes[0].before ?? "∅"}</span> → <span className="diff-line-add rounded px-0.5">{e.field_changes[0].after ?? "∅"}</span> | |
| 137 | + </div> | |
| 138 | + )} | |
| 139 | + </li> | |
| 140 | + ))} | |
| 141 | + </ul> | |
| 142 | + )} | |
| 143 | + </Panel> | |
| 144 | + ); | |
| 145 | +} | |
modified
apps/web/src/components/ui.tsx
+194 −35
@@ -1,32 +1,65 @@ | ||
| 1 | 1 | import Link from "next/link"; |
| 2 | 2 | import type { ReactNode } from "react"; |
| 3 | −import { fmtScore, importanceBand, typeLabel } from "@/lib/format"; | |
| 3 | +import { fmtScore, importanceBand, stateLabel, typeLabel } from "@/lib/format"; | |
| 4 | 4 | |
| 5 | −export function Score({ value, size = "md", title }: { value: number | null | undefined; size?: "sm" | "md" | "lg"; title?: string }) { | |
| 5 | +export function Score({ value, size = "md", title, kind = "importance" }: { value: number | null | undefined; size?: "sm" | "md" | "lg"; title?: string; kind?: "importance" | "signal" }) { | |
| 6 | 6 | const v = value ?? 0; |
| 7 | 7 | const band = importanceBand(v); |
| 8 | 8 | const color = band === "hot" ? "bg-hot/15 text-hot border-hot/40" : band === "high" ? "bg-high/15 text-high border-high/40" : band === "mid" ? "bg-mid/15 text-mid border-mid/40" : "bg-panel-2 text-fg-muted border-line"; |
| 9 | 9 | const sz = size === "sm" ? "min-w-7 px-1 text-[11px] h-5" : size === "lg" ? "min-w-14 px-2 text-xl h-9" : "min-w-9 px-1.5 text-xs h-6"; |
| 10 | 10 | return ( |
| 11 | − <span title={title ?? `Importance ${fmtScore(v)}`} className={`inline-flex items-center justify-center rounded-sm border font-mono font-semibold tabular ${color} ${sz}`}> | |
| 11 | + <span title={title ?? `${kind === "signal" ? "Signal" : "Importance"} ${fmtScore(v)}`} className={`inline-flex items-center justify-center rounded-sm border font-mono font-semibold tabular ${color} ${sz}`}> | |
| 12 | 12 | {fmtScore(v)} |
| 13 | 13 | </span> |
| 14 | 14 | ); |
| 15 | 15 | } |
| 16 | 16 | |
| 17 | −export function Chip({ children, tone = "default", href, className = "" }: { children: ReactNode; tone?: "default" | "silent" | "signal" | "danger" | "warn" | "info" | "ok"; href?: string; className?: string }) { | |
| 18 | − const tones: Record<string, string> = { | |
| 19 | − default: "border-line bg-panel-2 text-fg-muted", | |
| 20 | − silent: "border-silent/40 bg-silent-soft text-silent", | |
| 21 | − signal: "border-signal/40 bg-signal-soft text-signal", | |
| 22 | − danger: "border-danger/40 bg-danger/10 text-danger", | |
| 23 | − warn: "border-warn/40 bg-warn/10 text-warn", | |
| 24 | − info: "border-info/40 bg-info/10 text-info", | |
| 25 | − ok: "border-ok/40 bg-ok/10 text-ok", | |
| 17 | +export type Tone = "default" | "silent" | "signal" | "danger" | "warn" | "info" | "ok" | "hot" | "high"; | |
| 18 | + | |
| 19 | +const TONES: Record<Tone, string> = { | |
| 20 | + default: "border-line bg-panel-2 text-fg-muted", | |
| 21 | + silent: "border-silent/40 bg-silent-soft text-silent", | |
| 22 | + signal: "border-signal/40 bg-signal-soft text-signal", | |
| 23 | + danger: "border-danger/40 bg-danger/10 text-danger", | |
| 24 | + warn: "border-warn/40 bg-warn/10 text-warn", | |
| 25 | + info: "border-info/40 bg-info/10 text-info", | |
| 26 | + ok: "border-ok/40 bg-ok/10 text-ok", | |
| 27 | + hot: "border-hot/50 bg-hot/12 text-hot", | |
| 28 | + high: "border-high/50 bg-high/12 text-high", | |
| 29 | +}; | |
| 30 | + | |
| 31 | +export function Chip({ children, tone = "default", href, className = "", title }: { children: ReactNode; tone?: Tone; href?: string; className?: string; title?: string }) { | |
| 32 | + const cls = `inline-flex items-center gap-1 rounded-sm border px-1.5 py-px text-[10.5px] font-medium leading-4 whitespace-nowrap ${TONES[tone]} ${className}`; | |
| 33 | + if (href) return <Link href={href} title={title} className={`${cls} hover:border-line-strong`}>{children}</Link>; | |
| 34 | + return <span title={title} className={cls}>{children}</span>; | |
| 35 | +} | |
| 36 | + | |
| 37 | +/** Signal badges (spec §31): BREAKING · SILENT · FIRST PARTY · CONFIRMED · DEVELOPING · ANOMALOUS · EXTERNAL */ | |
| 38 | +export type BadgeKind = "breaking" | "silent" | "first-party" | "confirmed" | "developing" | "anomalous" | "external" | "inferred" | "unconfirmed" | "replayed"; | |
| 39 | +export function Badge({ kind, compact = false }: { kind: BadgeKind; compact?: boolean }) { | |
| 40 | + const map: Record<BadgeKind, { tone: Tone; label: string; short: string; title: string }> = { | |
| 41 | + breaking: { tone: "hot", label: "BREAKING", short: "BRK", title: "Breaking: strong signal, fresh, confirmed or first-party" }, | |
| 42 | + developing: { tone: "high", label: "DEVELOPING", short: "DEV", title: "Developing: signals accumulating across sources" }, | |
| 43 | + silent: { tone: "silent", label: "SILENT", short: "SIL", title: "Silent change: modified without a matching announcement" }, | |
| 44 | + "first-party": { tone: "signal", label: "FIRST PARTY", short: "1ST", title: "First-party evidence: the organization's own channel" }, | |
| 45 | + confirmed: { tone: "ok", label: "CONFIRMED", short: "CFM", title: "Confirmed by independent sources" }, | |
| 46 | + anomalous: { tone: "warn", label: "ANOMALOUS", short: "ANM", title: "Activity far above this source's baseline" }, | |
| 47 | + external: { tone: "default", label: "EXTERNAL", short: "EXT", title: "Third-party report (media / aggregator)" }, | |
| 48 | + inferred: { tone: "info", label: "INFERRED", short: "INF", title: "Classification inferred, limited evidence" }, | |
| 49 | + unconfirmed: { tone: "warn", label: "UNCONFIRMED", short: "UNC", title: "Single-source, low heuristic confidence" }, | |
| 50 | + replayed: { tone: "default", label: "REPLAYED", short: "RPL", title: "Delivered from the durable stream after a reconnection" }, | |
| 26 | 51 | }; |
| 27 | − const cls = `inline-flex items-center gap-1 rounded-sm border px-1.5 py-px text-[10.5px] font-medium leading-4 whitespace-nowrap ${tones[tone]} ${className}`; | |
| 28 | − if (href) return <Link href={href} className={`${cls} hover:border-line-strong`}>{children}</Link>; | |
| 29 | − return <span className={cls}>{children}</span>; | |
| 52 | + const b = map[kind]; | |
| 53 | + return ( | |
| 54 | + <Chip tone={b.tone} title={b.title} className="font-mono font-semibold tracking-wider"> | |
| 55 | + {compact ? b.short : b.label} | |
| 56 | + </Chip> | |
| 57 | + ); | |
| 58 | +} | |
| 59 | + | |
| 60 | +export function StateBadge({ state }: { state: string | null | undefined }) { | |
| 61 | + if (!state || state === "watching" || state === "closed") return null; | |
| 62 | + return <Badge kind={state as BadgeKind} />; | |
| 30 | 63 | } |
| 31 | 64 | |
| 32 | 65 | export function TypeChip({ type, href }: { type: string; href?: string }) { |
@@ -34,25 +67,23 @@ export function TypeChip({ type, href }: { type: string; href?: string }) { | ||
| 34 | 67 | } |
| 35 | 68 | |
| 36 | 69 | export function SilentBadge({ compact = false }: { compact?: boolean }) { |
| 37 | − return ( | |
| 38 | − <Chip tone="silent" className="font-semibold tracking-wide"> | |
| 39 | − <span aria-hidden>⚠</span> {compact ? "SILENT" : "SILENT CHANGE"} | |
| 40 | − </Chip> | |
| 41 | − ); | |
| 70 | + return <Badge kind="silent" compact={compact} />; | |
| 42 | 71 | } |
| 43 | 72 | |
| 44 | 73 | export function EvidenceTag({ label }: { label: string | null | undefined }) { |
| 45 | 74 | const l = (label ?? "OBSERVED").toUpperCase(); |
| 46 | − const tone = l === "CONFIRMED" ? "ok" : l === "INFERRED" ? "info" : l === "UNCONFIRMED" ? "warn" : "default"; | |
| 47 | − return <Chip tone={tone as "ok" | "info" | "warn" | "default"} className="font-mono tracking-wider">{l}</Chip>; | |
| 75 | + if (l === "CONFIRMED") return <Badge kind="confirmed" />; | |
| 76 | + if (l === "INFERRED") return <Badge kind="inferred" />; | |
| 77 | + if (l === "UNCONFIRMED") return <Badge kind="unconfirmed" />; | |
| 78 | + return <Chip className="font-mono tracking-wider">{l}</Chip>; | |
| 48 | 79 | } |
| 49 | 80 | |
| 50 | 81 | export function HealthPill({ health }: { health: string | null | undefined }) { |
| 51 | 82 | const h = (health ?? "UP").toUpperCase(); |
| 52 | − const tone = h === "UP" ? "ok" : h === "DEGRADED" || h === "RATE_LIMITED" ? "warn" : h === "ERROR" ? "danger" : "default"; | |
| 83 | + const tone: Tone = h === "UP" || h === "ACTIVE" || h === "VALIDATED" ? "ok" : h === "DEGRADED" || h === "RATE_LIMITED" || h === "PENDING" ? "warn" : h === "ERROR" ? "danger" : "default"; | |
| 53 | 84 | return ( |
| 54 | − <Chip tone={tone as "ok" | "warn" | "danger" | "default"} className="font-mono"> | |
| 55 | − <span className={`inline-block size-1.5 rounded-full ${h === "UP" ? "bg-ok" : h === "ERROR" ? "bg-danger" : h === "DISABLED" ? "bg-low" : "bg-warn"}`} /> {h} | |
| 85 | + <Chip tone={tone} className="font-mono"> | |
| 86 | + <span className={`inline-block size-1.5 rounded-full ${tone === "ok" ? "bg-ok" : tone === "danger" ? "bg-danger" : tone === "default" ? "bg-low" : "bg-warn"}`} /> {h} | |
| 56 | 87 | </Chip> |
| 57 | 88 | ); |
| 58 | 89 | } |
@@ -61,9 +92,9 @@ export function TierBadge({ tier }: { tier: string | null | undefined }) { | ||
| 61 | 92 | return <span className="inline-flex size-5 items-center justify-center rounded-sm border border-line bg-panel-2 font-mono text-[11px] font-semibold text-fg-muted" title={`Tier ${tier}`}>{tier ?? "?"}</span>; |
| 62 | 93 | } |
| 63 | 94 | |
| 64 | −export function Panel({ title, action, children, className = "", dense = false }: { title?: ReactNode; action?: ReactNode; children: ReactNode; className?: string; dense?: boolean }) { | |
| 95 | +export function Panel({ title, action, children, className = "", dense = false, id }: { title?: ReactNode; action?: ReactNode; children: ReactNode; className?: string; dense?: boolean; id?: string }) { | |
| 65 | 96 | return ( |
| 66 | − <section className={`panel ${className}`}> | |
| 97 | + <section id={id} className={`panel ${className}`}> | |
| 67 | 98 | {title !== undefined && ( |
| 68 | 99 | <header className="flex items-center justify-between gap-3 border-b border-line px-3 py-2"> |
| 69 | 100 | <h2 className="label">{title}</h2> |
@@ -75,15 +106,21 @@ export function Panel({ title, action, children, className = "", dense = false } | ||
| 75 | 106 | ); |
| 76 | 107 | } |
| 77 | 108 | |
| 78 | −export function Empty({ children = "No data yet — the engine is warming up." }: { children?: ReactNode }) { | |
| 79 | − return <div className="px-3 py-8 text-center text-[13px] text-fg-subtle">{children}</div>; | |
| 109 | +export function Empty({ children = "No data yet — the engine is warming up.", icon }: { children?: ReactNode; icon?: ReactNode }) { | |
| 110 | + return ( | |
| 111 | + <div className="flex flex-col items-center gap-1 px-3 py-8 text-center text-[13px] text-fg-subtle"> | |
| 112 | + {icon} | |
| 113 | + <div>{children}</div> | |
| 114 | + </div> | |
| 115 | + ); | |
| 80 | 116 | } |
| 81 | 117 | |
| 82 | −export function Stat({ label, value, hint }: { label: string; value: ReactNode; hint?: ReactNode }) { | |
| 118 | +export function Stat({ label, value, hint, tone }: { label: string; value: ReactNode; hint?: ReactNode; tone?: Tone }) { | |
| 119 | + const color = tone === "hot" ? "text-hot" : tone === "silent" ? "text-silent" : tone === "signal" ? "text-signal" : tone === "warn" ? "text-warn" : ""; | |
| 83 | 120 | return ( |
| 84 | 121 | <div className="flex min-w-0 flex-col gap-0.5 px-3 py-2"> |
| 85 | 122 | <span className="label">{label}</span> |
| 86 | − <span className="font-mono text-lg font-semibold leading-tight tabular">{value}</span> | |
| 123 | + <span className={`font-mono text-lg font-semibold leading-tight tabular ${color}`}>{value}</span> | |
| 87 | 124 | {hint && <span className="truncate text-[11px] text-fg-subtle">{hint}</span>} |
| 88 | 125 | </div> |
| 89 | 126 | ); |
@@ -114,12 +151,12 @@ export function Gauge({ label, value, tone }: { label: string; value: number | n | ||
| 114 | 151 | ); |
| 115 | 152 | } |
| 116 | 153 | |
| 117 | −export function PageHeader({ title, kicker, description, actions }: { title: ReactNode; kicker?: ReactNode; description?: ReactNode; actions?: ReactNode }) { | |
| 154 | +export function PageHeader({ title, kicker, description, actions, compact = false }: { title: ReactNode; kicker?: ReactNode; description?: ReactNode; actions?: ReactNode; compact?: boolean }) { | |
| 118 | 155 | return ( |
| 119 | − <div className="mb-4 flex flex-wrap items-end justify-between gap-3"> | |
| 156 | + <div className={`${compact ? "mb-3" : "mb-4"} flex flex-wrap items-end justify-between gap-3`}> | |
| 120 | 157 | <div className="min-w-0"> |
| 121 | − {kicker && <div className="label mb-1">{kicker}</div>} | |
| 122 | − <h1 className="text-xl font-semibold leading-tight sm:text-2xl">{title}</h1> | |
| 158 | + {kicker && <div className="label mb-1 flex flex-wrap items-center gap-2 normal-case tracking-normal">{kicker}</div>} | |
| 159 | + <h1 className={`${compact ? "text-lg" : "text-xl sm:text-2xl"} font-semibold leading-tight`}>{title}</h1> | |
| 123 | 160 | {description && <p className="mt-1 max-w-3xl text-[13px] text-fg-muted">{description}</p>} |
| 124 | 161 | </div> |
| 125 | 162 | {actions && <div className="flex flex-wrap items-center gap-2">{actions}</div>} |
@@ -165,3 +202,125 @@ export function ExtLink({ href, children, className = "" }: { href: string; chil | ||
| 165 | 202 | </a> |
| 166 | 203 | ); |
| 167 | 204 | } |
| 205 | + | |
| 206 | +export function Kbd({ children }: { children: ReactNode }) { | |
| 207 | + return <kbd className="inline-flex h-[18px] min-w-[18px] items-center justify-center rounded border border-line bg-panel-2 px-1 font-mono text-[10px] text-fg-subtle">{children}</kbd>; | |
| 208 | +} | |
| 209 | + | |
| 210 | +/** Skeleton primitives (spec §113). */ | |
| 211 | +export function Skeleton({ className = "" }: { className?: string }) { | |
| 212 | + return <div aria-hidden className={`skeleton ${className}`} />; | |
| 213 | +} | |
| 214 | +export function SkeletonRows({ rows = 8 }: { rows?: number }) { | |
| 215 | + return ( | |
| 216 | + <div className="divide-y divide-line"> | |
| 217 | + {Array.from({ length: rows }, (_, i) => ( | |
| 218 | + <div key={i} className="grid grid-cols-[6.5rem_1fr_auto] gap-x-3 px-3 py-2.5"> | |
| 219 | + <div className="flex flex-col gap-1.5"> | |
| 220 | + <Skeleton className="h-3 w-14" /> | |
| 221 | + <Skeleton className="h-2.5 w-10" /> | |
| 222 | + </div> | |
| 223 | + <div className="flex flex-col gap-1.5"> | |
| 224 | + <Skeleton className="h-2.5 w-24" /> | |
| 225 | + <Skeleton className={`h-3.5 ${i % 3 === 0 ? "w-3/4" : i % 3 === 1 ? "w-11/12" : "w-2/3"}`} /> | |
| 226 | + <Skeleton className="h-2.5 w-40" /> | |
| 227 | + </div> | |
| 228 | + <Skeleton className="h-6 w-9" /> | |
| 229 | + </div> | |
| 230 | + ))} | |
| 231 | + </div> | |
| 232 | + ); | |
| 233 | +} | |
| 234 | +export function SkeletonPanel({ lines = 5, title = true }: { lines?: number; title?: boolean }) { | |
| 235 | + return ( | |
| 236 | + <div className="panel p-3"> | |
| 237 | + {title && <Skeleton className="mb-3 h-2.5 w-28" />} | |
| 238 | + <div className="flex flex-col gap-2"> | |
| 239 | + {Array.from({ length: lines }, (_, i) => ( | |
| 240 | + <Skeleton key={i} className={`h-3 ${i % 2 ? "w-5/6" : "w-full"}`} /> | |
| 241 | + ))} | |
| 242 | + </div> | |
| 243 | + </div> | |
| 244 | + ); | |
| 245 | +} | |
| 246 | + | |
| 247 | +/** Inline SVG sparkline — no chart library, no client JS. */ | |
| 248 | +export function Sparkline({ values, width = 120, height = 28, tone = "signal", fill = true, responsive = false }: { values: number[]; width?: number; height?: number; tone?: "signal" | "hot" | "info" | "silent" | "muted"; fill?: boolean; /** stretch to the container width (viewBox scaling) */ responsive?: boolean }) { | |
| 249 | + if (!values.length) return <svg width={responsive ? "100%" : width} height={height} aria-hidden />; | |
| 250 | + const max = Math.max(1, ...values); | |
| 251 | + const step = values.length > 1 ? width / (values.length - 1) : width; | |
| 252 | + const pts = values.map((v, i) => [i * step, height - 2 - (v / max) * (height - 4)] as const); | |
| 253 | + const d = pts.map(([x, y], i) => `${i ? "L" : "M"}${x.toFixed(1)},${y.toFixed(1)}`).join(" "); | |
| 254 | + const color = tone === "hot" ? "var(--hot)" : tone === "info" ? "var(--info)" : tone === "silent" ? "var(--silent)" : tone === "muted" ? "var(--fg-subtle)" : "var(--signal)"; | |
| 255 | + return ( | |
| 256 | + <svg width={responsive ? "100%" : width} height={height} viewBox={`0 0 ${width} ${height}`} preserveAspectRatio={responsive ? "none" : undefined} aria-hidden className="block max-w-full overflow-visible"> | |
| 257 | + {fill && <path d={`${d} L${width},${height} L0,${height} Z`} fill={color} opacity={0.12} />} | |
| 258 | + <path d={d} fill="none" stroke={color} strokeWidth={1.5} strokeLinejoin="round" strokeLinecap="round" /> | |
| 259 | + </svg> | |
| 260 | + ); | |
| 261 | +} | |
| 262 | + | |
| 263 | +/** 35-day activity heatmap (spec §102). */ | |
| 264 | +export function Heatmap({ days, max }: { days: { day: string; events: number; silent?: number; breaking?: number }[]; max?: number }) { | |
| 265 | + const m = max ?? Math.max(1, ...days.map((d) => d.events)); | |
| 266 | + const cls = (n: number, breaking?: number): string => { | |
| 267 | + if (!n) return "heat-0"; | |
| 268 | + if (breaking && breaking >= 3) return "heat-hot"; | |
| 269 | + const r = n / m; | |
| 270 | + return r > 0.75 ? "heat-4" : r > 0.5 ? "heat-3" : r > 0.25 ? "heat-2" : "heat-1"; | |
| 271 | + }; | |
| 272 | + return ( | |
| 273 | + <div className="flex flex-wrap gap-[3px]" role="img" aria-label="Daily activity over the last 35 days"> | |
| 274 | + {days.map((d) => ( | |
| 275 | + <span key={d.day} title={`${d.day}: ${d.events} event${d.events === 1 ? "" : "s"}${d.silent ? ` · ${d.silent} silent` : ""}${d.breaking ? ` · ${d.breaking} breaking` : ""}`} className={`size-3 rounded-[2px] ${cls(d.events, d.breaking)}`} /> | |
| 276 | + ))} | |
| 277 | + </div> | |
| 278 | + ); | |
| 279 | +} | |
| 280 | + | |
| 281 | +/** Link-based tabs (URL-driven, server-renderable). */ | |
| 282 | +export function Tabs({ items, current, className = "" }: { items: { key: string; label: ReactNode; href: string; count?: number | null }[]; current: string; className?: string }) { | |
| 283 | + return ( | |
| 284 | + <nav className={`flex gap-0.5 overflow-x-auto border-b border-line no-scrollbar ${className}`} aria-label="Sections"> | |
| 285 | + {items.map((t) => { | |
| 286 | + const active = t.key === current; | |
| 287 | + return ( | |
| 288 | + <Link key={t.key} href={t.href} aria-current={active ? "page" : undefined} className={`-mb-px inline-flex items-center gap-1.5 whitespace-nowrap border-b-2 px-2.5 py-1.5 text-[12.5px] ${active ? "border-signal text-fg" : "border-transparent text-fg-muted hover:text-fg"}`}> | |
| 289 | + {t.label} | |
| 290 | + {t.count !== undefined && t.count !== null && <span className="font-mono text-[10.5px] text-fg-subtle tabular">{t.count}</span>} | |
| 291 | + </Link> | |
| 292 | + ); | |
| 293 | + })} | |
| 294 | + </nav> | |
| 295 | + ); | |
| 296 | +} | |
| 297 | + | |
| 298 | +export function StateLabelText({ state }: { state?: string | null }) { | |
| 299 | + const l = stateLabel(state); | |
| 300 | + if (!l) return null; | |
| 301 | + const c = state === "breaking" ? "text-hot" : state === "developing" ? "text-high" : state === "confirmed" ? "text-ok" : "text-fg-subtle"; | |
| 302 | + return <span className={`font-mono text-[10.5px] font-semibold tracking-wider ${c}`}>{l}</span>; | |
| 303 | +} | |
| 304 | + | |
| 305 | +export function Flag({ code, className = "" }: { code?: string | null; className?: string }) { | |
| 306 | + if (!code) return null; | |
| 307 | + const c = code.toUpperCase(); | |
| 308 | + if (c.length !== 2) return <span className={`font-mono text-[10px] ${className}`}>{c}</span>; | |
| 309 | + const flag = String.fromCodePoint(...[...c].map((ch) => 0x1f1e6 + ch.charCodeAt(0) - 65)); | |
| 310 | + return <span className={className} title={c} aria-label={c}>{flag}</span>; | |
| 311 | +} | |
| 312 | + | |
| 313 | +/** Entity mark (spec §111): deterministic initials fallback — no third-party favicon requests. */ | |
| 314 | +export function Mark({ name, size = 5, className = "" }: { name: string; size?: 4 | 5 | 6 | 8; className?: string }) { | |
| 315 | + const words = name.replace(/[^\p{L}\p{N} ]/gu, " ").trim().split(/\s+/).filter(Boolean); | |
| 316 | + const initials = (words.length >= 2 ? words[0]![0]! + words[1]![0]! : name.slice(0, 2)).toUpperCase(); | |
| 317 | + let h = 0; | |
| 318 | + for (const ch of name) h = (h * 31 + ch.charCodeAt(0)) >>> 0; | |
| 319 | + const hue = h % 360; | |
| 320 | + const sz = size === 4 ? "size-4 text-[8px]" : size === 6 ? "size-6 text-[10px]" : size === 8 ? "size-8 text-[12px]" : "size-5 text-[9px]"; | |
| 321 | + return ( | |
| 322 | + <span aria-hidden className={`inline-flex shrink-0 items-center justify-center rounded-[3px] border font-mono font-semibold tracking-tight ${sz} ${className}`} style={{ background: `color-mix(in oklab, oklch(0.62 0.12 ${hue}) 22%, var(--panel-2))`, borderColor: `color-mix(in oklab, oklch(0.62 0.12 ${hue}) 40%, var(--line))`, color: `oklch(0.78 0.11 ${hue})` }}> | |
| 323 | + {initials} | |
| 324 | + </span> | |
| 325 | + ); | |
| 326 | +} | |
modified
apps/web/src/lib/api.ts
+331 −10
@@ -78,6 +78,45 @@ export interface EventItem { | ||
| 78 | 78 | sensor: SensorRef; |
| 79 | 79 | entities: EntityRef[]; |
| 80 | 80 | cluster_size?: number | null; |
| 81 | + // 0.2 intelligence fields | |
| 82 | + signal_score?: number | null; | |
| 83 | + velocity_score?: number | null; | |
| 84 | + impact_score?: number | null; | |
| 85 | + anomaly_score?: number | null; | |
| 86 | + change_class?: string | null; | |
| 87 | + first_party?: boolean; | |
| 88 | + country?: string | null; | |
| 89 | + language?: string | null; | |
| 90 | + canonical_url?: string | null; | |
| 91 | + field_changes?: FieldChange[] | null; | |
| 92 | + score_reasons?: ScoreReason[] | null; | |
| 93 | + cluster?: ClusterRef | null; | |
| 94 | +} | |
| 95 | + | |
| 96 | +export interface FieldChange { | |
| 97 | + label: string; | |
| 98 | + kind: "price" | "percent" | "number" | "date" | "version" | "status" | "text"; | |
| 99 | + before: string | null; | |
| 100 | + after: string | null; | |
| 101 | + deltaPct?: number | null; | |
| 102 | +} | |
| 103 | + | |
| 104 | +export interface ScoreReason { | |
| 105 | + sign: "+" | "-"; | |
| 106 | + text: string; | |
| 107 | + points?: number; | |
| 108 | +} | |
| 109 | + | |
| 110 | +export interface ClusterRef { | |
| 111 | + id: string; | |
| 112 | + slug?: string | null; | |
| 113 | + state?: string; | |
| 114 | + event_count: number; | |
| 115 | + source_count?: number; | |
| 116 | + first_party_count?: number; | |
| 117 | + external_count?: number; | |
| 118 | + velocity?: number; | |
| 119 | + lead_time_ms?: number | null; | |
| 81 | 120 | } |
| 82 | 121 | |
| 83 | 122 | export interface LiveEvent { |
@@ -97,8 +136,22 @@ export interface LiveEvent { | ||
| 97 | 136 | categories: string[]; |
| 98 | 137 | url: string; |
| 99 | 138 | clusterId?: string | null; |
| 139 | + clusterSlug?: string | null; | |
| 140 | + clusterSize?: number; | |
| 141 | + clusterState?: string; | |
| 100 | 142 | detectedAt: string; |
| 101 | 143 | publishedAt?: string | null; |
| 144 | + signal?: number; | |
| 145 | + impact?: number; | |
| 146 | + velocity?: number; | |
| 147 | + firstParty?: boolean; | |
| 148 | + country?: string | null; | |
| 149 | + language?: string | null; | |
| 150 | + changeClass?: string | null; | |
| 151 | + fieldChanges?: FieldChange[]; | |
| 152 | + group?: string; | |
| 153 | + sid?: string; | |
| 154 | + replayed?: boolean; | |
| 102 | 155 | } |
| 103 | 156 | |
| 104 | 157 | /** Normalize a WebSocket payload to the REST row shape used by all list components. */ |
@@ -125,7 +178,16 @@ export function liveToEvent(e: LiveEvent): EventItem { | ||
| 125 | 178 | source: e.source, |
| 126 | 179 | sensor: e.sensor, |
| 127 | 180 | entities: e.entities ?? [], |
| 128 | − cluster_size: 1, | |
| 181 | + cluster_size: e.clusterSize ?? 1, | |
| 182 | + cluster: e.clusterId ? { id: e.clusterId, slug: e.clusterSlug ?? null, state: e.clusterState, event_count: e.clusterSize ?? 1, velocity: e.velocity } : null, | |
| 183 | + signal_score: e.signal ?? null, | |
| 184 | + velocity_score: e.velocity ?? 0, | |
| 185 | + impact_score: e.impact ?? 0, | |
| 186 | + change_class: e.changeClass ?? null, | |
| 187 | + first_party: e.firstParty !== false, | |
| 188 | + country: e.country ?? null, | |
| 189 | + language: e.language ?? null, | |
| 190 | + field_changes: e.fieldChanges ?? null, | |
| 129 | 191 | }; |
| 130 | 192 | } |
| 131 | 193 | |
@@ -135,6 +197,14 @@ export interface EventsPage { | ||
| 135 | 197 | } |
| 136 | 198 | |
| 137 | 199 | export interface Stats { |
| 200 | + checks_per_min?: number; | |
| 201 | + events_per_min?: number; | |
| 202 | + events_1h?: number; | |
| 203 | + breaking_now?: number; | |
| 204 | + developing_now?: number; | |
| 205 | + sources_first_party?: number; | |
| 206 | + countries?: number; | |
| 207 | + not_modified_ratio_5m?: number | null; | |
| 138 | 208 | sources?: number; |
| 139 | 209 | sensors?: number; |
| 140 | 210 | entities?: number; |
@@ -168,6 +238,12 @@ export interface TrendingItem { | ||
| 168 | 238 | max_importance: number; |
| 169 | 239 | prev_events: number; |
| 170 | 240 | score: number; |
| 241 | + first_party?: number; | |
| 242 | + confirmed?: number; | |
| 243 | + avg_signal?: number; | |
| 244 | + baseline_per_day?: number; | |
| 245 | + direction?: "up" | "down" | "flat"; | |
| 246 | + entity_importance?: number; | |
| 171 | 247 | } |
| 172 | 248 | |
| 173 | 249 | export interface Cluster { |
@@ -182,7 +258,168 @@ export interface Cluster { | ||
| 182 | 258 | first_at: string; |
| 183 | 259 | last_at: string; |
| 184 | 260 | source?: SourceRef | null; |
| 185 | − events?: { id: string; slug: string; title: string; importance: number; event_type: string; detected_at: string; source_id: string; url: string }[] | null; | |
| 261 | + events?: { id: string; slug: string; title: string; importance: number; event_type: string; detected_at: string; source_id: string; url: string; first_party?: boolean }[] | null; | |
| 262 | + slug?: string | null; | |
| 263 | + state?: string; | |
| 264 | + source_count?: number; | |
| 265 | + first_party_count?: number; | |
| 266 | + external_count?: number; | |
| 267 | + velocity?: number; | |
| 268 | + lead_time_ms?: number | null; | |
| 269 | + first_party_at?: string | null; | |
| 270 | + first_external_at?: string | null; | |
| 271 | + primary_slug?: string | null; | |
| 272 | + event?: EventItem | null; | |
| 273 | + sources?: { id: string; name: string; domain: string; first_party?: boolean }[] | null; | |
| 274 | +} | |
| 275 | + | |
| 276 | +export interface ClusterDetail { | |
| 277 | + cluster: Cluster & { timeline?: { at: string; eventId: string; sourceId: string; sourceName?: string; sensorType?: string; firstParty: boolean; eventType: string; importance: number }[] }; | |
| 278 | + events: EventItem[]; | |
| 279 | + entities: { id: string; name: string; type: string; importance: number }[]; | |
| 280 | + propagation: { id: string; slug: string; at: string; offset_ms: number; source: SourceRef; sensor: SensorRef; first_party: boolean; event_type: string; importance: number; title: string }[]; | |
| 281 | + lead_time_ms: number | null; | |
| 282 | + first_party_signals: number; | |
| 283 | + external_signals: number; | |
| 284 | +} | |
| 285 | + | |
| 286 | +export interface BreakingDesk { | |
| 287 | + breaking_now: Cluster[]; | |
| 288 | + developing: Cluster[]; | |
| 289 | + recently_confirmed: Cluster[]; | |
| 290 | + watching: EventItem[]; | |
| 291 | + generated_at: string; | |
| 292 | +} | |
| 293 | + | |
| 294 | +export interface Pulse { | |
| 295 | + activity: { t: string; events: number; changes: number }[]; | |
| 296 | + desks: { desk: string; items: EventItem[] }[]; | |
| 297 | + rising_entities: { id: string; name: string; type: string; events_3h: number; events_prev_24h: number; acceleration: number }[]; | |
| 298 | + anomalies: { id: string; name: string; domain: string; changes_2h: number; baseline_per_day: number; activity_score: number; pct_vs_baseline: number | null }[]; | |
| 299 | + silent_changes: EventItem[]; | |
| 300 | + infrastructure: EventItem[]; | |
| 301 | + breaking: Cluster[]; | |
| 302 | + by_group_24h: Record<string, number>; | |
| 303 | + totals: { events_1h: number; changes_1h: number; checks_5m: number; active_sources_24h: number; active_countries_24h: number }; | |
| 304 | + generated_at: string; | |
| 305 | +} | |
| 306 | + | |
| 307 | +export interface Radar { | |
| 308 | + unusual_source_activity: { id: string; name: string; domain: string; categories: string[]; changes_3h: number; baseline_per_day: number; ratio: number }[]; | |
| 309 | + silent_clusters: { id: string; name: string; type: string; silent_24h: number; types: string[]; last_at: string }[]; | |
| 310 | + documentation_bursts: { id: string; name: string; domain: string; doc_changes_6h: number; types: string[]; last_at: string }[]; | |
| 311 | + repository_bursts: { id: string; name: string; domain: string; repo_events_6h: number; last_at: string }[]; | |
| 312 | + status_changes: EventItem[]; | |
| 313 | + developing: Cluster[]; | |
| 314 | + new_coverage: { id: string; name: string; source_id: string; source_name: string; first_event_at: string; events: number }[]; | |
| 315 | + generated_at: string; | |
| 316 | + disclaimer: string; | |
| 317 | +} | |
| 318 | + | |
| 319 | +export interface EntityInsights { | |
| 320 | + heatmap: { day: string; events: number; silent: number; breaking: number; max_importance: number }[]; | |
| 321 | + baseline_per_day: number; | |
| 322 | + today: number; | |
| 323 | + events_24h: number; | |
| 324 | + events_prev_24h: number; | |
| 325 | + velocity_ratio: number; | |
| 326 | + anomaly: { score: number; ratio: number; pct: number }; | |
| 327 | + silent_24h: number; | |
| 328 | + breaking_24h: number; | |
| 329 | + sources_24h: number; | |
| 330 | + most_active_sensors: { id: string; name: string; type: string; connector: string; source_id: string; events_7d: number; last_event_at: string }[]; | |
| 331 | + rank: { rank: number | null; total: number; score: number | null }; | |
| 332 | +} | |
| 333 | + | |
| 334 | +export interface RankedEntity { | |
| 335 | + id: string; | |
| 336 | + name: string; | |
| 337 | + type: string; | |
| 338 | + domain?: string | null; | |
| 339 | + importance: number; | |
| 340 | + events_24h: number; | |
| 341 | + events_7d: number; | |
| 342 | + avg_signal: number; | |
| 343 | + confirmed_ratio: number; | |
| 344 | + sources: number; | |
| 345 | + silent_24h: number; | |
| 346 | + breaking_24h: number; | |
| 347 | + last_at: string; | |
| 348 | + baseline_per_day: number; | |
| 349 | + rank_score: number; | |
| 350 | + rank: number; | |
| 351 | +} | |
| 352 | + | |
| 353 | +export interface CountryRow { | |
| 354 | + country: string; | |
| 355 | + name: string; | |
| 356 | + slug: string; | |
| 357 | + flag: string; | |
| 358 | + sources: number; | |
| 359 | + events_24h: number; | |
| 360 | + breaking_24h: number; | |
| 361 | +} | |
| 362 | + | |
| 363 | +export interface CountryDesk { | |
| 364 | + country: { code: string; name: string; flag: string }; | |
| 365 | + breaking: EventItem[]; | |
| 366 | + by_category: { category: string; items: EventItem[] }[]; | |
| 367 | + sources: SourceRow[]; | |
| 368 | + by_type: { event_type: string; n: number }[]; | |
| 369 | + silent: EventItem[]; | |
| 370 | + recent: EventItem[]; | |
| 371 | + nextCursor: string | null; | |
| 372 | +} | |
| 373 | + | |
| 374 | +export interface CategoryDesk { | |
| 375 | + channel: string; | |
| 376 | + categories: string[]; | |
| 377 | + breaking: EventItem[]; | |
| 378 | + silent: EventItem[]; | |
| 379 | + active_sources: { id: string; name: string; domain: string; tier: string; first_party?: boolean; events_24h: number; max_importance: number }[]; | |
| 380 | + by_type: { event_type: string; n: number }[]; | |
| 381 | + trending_entities: { id: string; name: string; type: string; events_24h: number; sources: number }[]; | |
| 382 | + recent: EventItem[]; | |
| 383 | + nextCursor: string | null; | |
| 384 | + series: { t: string; n: number }[]; | |
| 385 | +} | |
| 386 | + | |
| 387 | +export interface Bookmark extends EventItem { | |
| 388 | + bookmarked_at: string; | |
| 389 | + note?: string | null; | |
| 390 | +} | |
| 391 | + | |
| 392 | +export interface Notification { | |
| 393 | + id: number; | |
| 394 | + alert_id: string; | |
| 395 | + alert_name: string; | |
| 396 | + event_id: string; | |
| 397 | + channel: string; | |
| 398 | + status: string; | |
| 399 | + created_at: string; | |
| 400 | + read_at?: string | null; | |
| 401 | + event: { id: string; slug: string; title: string; importance: number; signal_score?: number | null; event_type: string; silent_change: boolean; detected_at: string; source: { id: string; name: string } }; | |
| 402 | +} | |
| 403 | + | |
| 404 | +export interface Monitor { | |
| 405 | + id: string; | |
| 406 | + name: string; | |
| 407 | + url: string; | |
| 408 | + tier: string; | |
| 409 | + health: string; | |
| 410 | + status: string; | |
| 411 | + enabled: boolean; | |
| 412 | + config: Record<string, unknown>; | |
| 413 | + next_check_at?: string | null; | |
| 414 | + last_check_at?: string | null; | |
| 415 | + last_change_at?: string | null; | |
| 416 | + last_event_at?: string | null; | |
| 417 | + last_status?: number | null; | |
| 418 | + last_error?: string | null; | |
| 419 | + total_runs?: number; | |
| 420 | + raw_changes?: number; | |
| 421 | + meaningful_changes?: number; | |
| 422 | + created_at: string; | |
| 186 | 423 | } |
| 187 | 424 | |
| 188 | 425 | export interface SourceRow { |
@@ -203,6 +440,9 @@ export interface SourceRow { | ||
| 203 | 440 | last_check_at?: string | null; |
| 204 | 441 | sensors_degraded?: number; |
| 205 | 442 | robots_checked_at?: string | null; |
| 443 | + first_party?: boolean; | |
| 444 | + country?: string | null; | |
| 445 | + language?: string | null; | |
| 206 | 446 | } |
| 207 | 447 | |
| 208 | 448 | export interface SensorRow { |
@@ -233,14 +473,47 @@ export interface SensorRow { | ||
| 233 | 473 | source_name?: string; |
| 234 | 474 | domain?: string; |
| 235 | 475 | config?: Record<string, unknown>; |
| 476 | + status?: string; | |
| 477 | + priority?: number; | |
| 478 | + validated_at?: string | null; | |
| 479 | + etag?: string | null; | |
| 480 | + last_modified?: string | null; | |
| 481 | + checks_24h?: number; | |
| 482 | + changes_24h?: number; | |
| 483 | + snapshot_count?: number; | |
| 484 | + current_interval_seconds?: number | null; | |
| 485 | + not_modified_24h?: number; | |
| 486 | + errors_24h?: number; | |
| 487 | + avg_ms_24h?: number | null; | |
| 488 | +} | |
| 489 | + | |
| 490 | +export interface SourceQuality { | |
| 491 | + sensors?: number; | |
| 492 | + sensors_up?: number; | |
| 493 | + avg_latency_ms?: number | null; | |
| 494 | + raw_changes?: number; | |
| 495 | + meaningful_changes?: number; | |
| 496 | + total_runs?: number; | |
| 497 | + total_not_modified?: number; | |
| 498 | + structured_sensors?: number; | |
| 499 | + checks_7d?: number; | |
| 500 | + errors_7d?: number; | |
| 501 | + avg_confidence?: number | null; | |
| 502 | + success_rate: number; | |
| 503 | + structured_share: number; | |
| 504 | + usefulness: number; | |
| 505 | + quality_score: number; | |
| 236 | 506 | } |
| 237 | 507 | |
| 238 | 508 | export interface SourceDetail { |
| 239 | 509 | source: SourceRow; |
| 240 | 510 | sensors: SensorRow[]; |
| 241 | 511 | entities: { id: string; name: string; type: string; importance: number; event_count: number }[]; |
| 242 | − activity: { changes_2h?: number; changes_14d?: number; events_24h?: number; events_14d?: number; baseline_changes_per_day?: number; activity_score?: number }; | |
| 512 | + activity: { changes_2h?: number; changes_14d?: number; events_24h?: number; events_14d?: number; baseline_changes_per_day?: number; activity_score?: number; silent_24h?: number; breaking_24h?: number }; | |
| 513 | + quality?: SourceQuality; | |
| 243 | 514 | discovery: { url: string; kind: string; evidence?: string | null; score?: { value?: number; itemCount?: number | null; title?: string | null } | null; status: string; found_at: string }[]; |
| 515 | + by_type?: { event_type: string; n: number }[]; | |
| 516 | + daily?: { day: string; checks: number; not_modified: number; errors: number; raw_changes: number; events: number }[]; | |
| 244 | 517 | } |
| 245 | 518 | |
| 246 | 519 | export interface SensorDetail { |
@@ -248,6 +521,7 @@ export interface SensorDetail { | ||
| 248 | 521 | runs: { id: string; started_at: string; finished_at?: string | null; http_status?: number | null; outcome: string; error?: string | null; duration_ms?: number | null; bytes?: number | null; fetch_method?: string | null; snapshot_id?: string | null }[]; |
| 249 | 522 | snapshots: SnapshotRow[]; |
| 250 | 523 | changes: ChangeRow[]; |
| 524 | + events?: EventItem[]; | |
| 251 | 525 | } |
| 252 | 526 | |
| 253 | 527 | export interface SnapshotRow { |
@@ -266,6 +540,9 @@ export interface SnapshotRow { | ||
| 266 | 540 | fetch_duration_ms?: number | null; |
| 267 | 541 | extraction_confidence?: number | null; |
| 268 | 542 | storage_key?: string | null; |
| 543 | + has_raw?: boolean; | |
| 544 | + has_change?: boolean; | |
| 545 | + event_slug?: string | null; | |
| 269 | 546 | } |
| 270 | 547 | |
| 271 | 548 | export interface ChangeRow { |
@@ -282,6 +559,8 @@ export interface ChangeRow { | ||
| 282 | 559 | heuristic_type?: string | null; |
| 283 | 560 | diff?: DiffSummary; |
| 284 | 561 | heuristic?: Record<string, unknown>; |
| 562 | + change_class?: string | null; | |
| 563 | + field_changes?: FieldChange[] | null; | |
| 285 | 564 | } |
| 286 | 565 | |
| 287 | 566 | export interface DiffSummary { |
@@ -303,6 +582,7 @@ export interface EventDetail { | ||
| 303 | 582 | interpretations: { version: number; model: string; created_at: string }[]; |
| 304 | 583 | snapshots: SnapshotRow[]; |
| 305 | 584 | sensor_reliability: { health?: string; success_rate?: number | null; avg_latency_ms?: number | null; total_runs?: number; raw_changes?: number; meaningful_changes?: number; last_check_at?: string | null } | null; |
| 585 | + history?: { id: string; slug: string; title: string; event_type: string; importance: number; silent_change: boolean; detected_at: string; field_changes?: FieldChange[] | null }[]; | |
| 306 | 586 | } |
| 307 | 587 | |
| 308 | 588 | export interface EntityRow { |
@@ -322,23 +602,31 @@ export interface EntityRow { | ||
| 322 | 602 | |
| 323 | 603 | export interface EntityDetail { |
| 324 | 604 | entity: EntityRow; |
| 605 | + parent?: { id: string; name: string; type: string } | null; | |
| 325 | 606 | children: EntityRow[]; |
| 326 | 607 | relations: { relation: string; from_id: string; to_id: string; from_name: string; to_name: string; to_type: string }[]; |
| 327 | − sources: SourceRef[]; | |
| 608 | + sources: (SourceRef & { first_party?: boolean; country?: string | null; sensor_count?: number; events_24h?: number })[]; | |
| 328 | 609 | aliases: string[]; |
| 329 | 610 | recent: EventItem[]; |
| 611 | + nextCursor?: string | null; | |
| 330 | 612 | by_type: { event_type: string; n: number }[]; |
| 613 | + insights?: EntityInsights; | |
| 614 | + silent?: EventItem[]; | |
| 615 | + related?: { id: string; name: string; type: string; shared_events: number }[]; | |
| 331 | 616 | } |
| 332 | 617 | |
| 333 | 618 | export interface Explore { |
| 334 | − most_active_sources: { id: string; name: string; domain: string; events_24h: number; max_importance: number }[]; | |
| 619 | + most_active_sources: { id: string; name: string; domain: string; first_party?: boolean; events_24h: number; max_importance: number }[]; | |
| 335 | 620 | biggest_changes: EventItem[]; |
| 336 | 621 | silent_changes: EventItem[]; |
| 337 | 622 | clusters: Cluster[]; |
| 338 | − unusual_activity: { id: string; name: string; domain: string; changes_2h: number; baseline_per_day: number; activity_score: number }[]; | |
| 623 | + unusual_activity: { id: string; name: string; domain: string; changes_2h: number; baseline_per_day: number; activity_score: number; pct_vs_baseline?: number | null }[]; | |
| 624 | + newly_detected?: EventItem[]; | |
| 625 | + confirmed_first_party?: EventItem[]; | |
| 339 | 626 | by_type: { event_type: string; n: number }[]; |
| 340 | 627 | by_category: { category: string; n: number }[]; |
| 341 | 628 | channels: Record<string, string[]>; |
| 629 | + groups?: Record<string, string>; | |
| 342 | 630 | event_types: Record<string, string>; |
| 343 | 631 | } |
| 344 | 632 | |
@@ -366,14 +654,22 @@ export interface HealthReport { | ||
| 366 | 654 | noisy_sensors: { id: string; name: string; source_id: string; url: string; raw_changes: number; meaningful_changes: number; noise_ratio: number | null }[]; |
| 367 | 655 | daily: Record<string, number | string>[]; |
| 368 | 656 | live: { clients: number; published: number }; |
| 657 | + sensors_by_status?: { status: string; n: number }[]; | |
| 658 | + sensors_by_connector?: { connector: string; sensors: number; up: number; avg_latency_ms: number | null }[]; | |
| 659 | + throughput?: { checks_5m: number; not_modified_5m: number; events_5m: number; changes_5m: number; queue_due: number; avg_latency_5m_ms: number | null; noise_filtered_24h: number; checks_per_min: number; events_per_min: number; not_modified_ratio: number | null }; | |
| 660 | + engine?: { inflight?: number; concurrency?: number; due?: number; busyHosts?: { host: string; inflight: number }[]; circuitOpen?: { host: string; failures: number; until: string }[]; at?: string; version?: string } | null; | |
| 661 | + top_failing_domains?: { host: string; failures: number; last_error: string | null }[]; | |
| 662 | + slowest_sensors?: { id: string; name: string; source_id: string; avg_latency_ms: number; connector: string }[]; | |
| 369 | 663 | } |
| 370 | 664 | |
| 371 | 665 | export interface SearchResult { |
| 372 | 666 | query: string; |
| 667 | + parsed?: { text: string; filters: Record<string, unknown> }; | |
| 373 | 668 | events: EventItem[]; |
| 374 | 669 | entities: EntityRow[]; |
| 375 | 670 | sources: SourceRow[]; |
| 376 | 671 | urls: { url: string; domain: string; status: string; change_count: number; last_seen_at: string }[]; |
| 672 | + clusters?: { id: string; slug: string; title: string; state: string; event_count: number; source_count: number; max_importance: number; last_at: string }[]; | |
| 377 | 673 | } |
| 378 | 674 | |
| 379 | 675 | export interface DomainTimeline { |
@@ -408,12 +704,17 @@ export interface Watchlist { | ||
| 408 | 704 | |
| 409 | 705 | export interface AlertRule { |
| 410 | 706 | importance_min?: number; |
| 707 | + signal_min?: number; | |
| 411 | 708 | event_types?: string[]; |
| 709 | + groups?: string[]; | |
| 412 | 710 | entities?: string[]; |
| 413 | 711 | sources?: string[]; |
| 414 | 712 | keywords?: string[]; |
| 415 | 713 | silent_only?: boolean; |
| 714 | + first_party_only?: boolean; | |
| 715 | + confirmed_only?: boolean; | |
| 416 | 716 | categories?: string[]; |
| 717 | + countries?: string[]; | |
| 417 | 718 | } |
| 418 | 719 | |
| 419 | 720 | export interface Alert { |
@@ -421,9 +722,12 @@ export interface Alert { | ||
| 421 | 722 | name: string; |
| 422 | 723 | rule: AlertRule; |
| 423 | 724 | channel: string; |
| 725 | + channel_config?: { url?: string }; | |
| 424 | 726 | enabled: boolean; |
| 425 | 727 | created_at: string; |
| 426 | 728 | last_fired_at?: string | null; |
| 729 | + fired_count?: number; | |
| 730 | + fired_24h?: number; | |
| 427 | 731 | } |
| 428 | 732 | |
| 429 | 733 | // --------------------------------------------------------------------------------------- |
@@ -452,12 +756,19 @@ export type EventQuery = Partial<{ | ||
| 452 | 756 | cluster: string; |
| 453 | 757 | importance_min: number; |
| 454 | 758 | confidence_min: number; |
| 759 | + signal_min: number; | |
| 455 | 760 | event_type: string; |
| 761 | + group: string; | |
| 456 | 762 | silent_change: boolean; |
| 763 | + first_party: boolean; | |
| 764 | + confirmed: boolean; | |
| 765 | + country: string; | |
| 766 | + language: string; | |
| 767 | + change_class: string; | |
| 457 | 768 | q: string; |
| 458 | 769 | limit: number; |
| 459 | 770 | cursor: string; |
| 460 | − order: "recent" | "importance"; | |
| 771 | + order: "recent" | "importance" | "signal"; | |
| 461 | 772 | }>; |
| 462 | 773 | |
| 463 | 774 | export function eventQueryString(q: EventQuery): string { |
@@ -469,14 +780,24 @@ export function eventQueryString(q: EventQuery): string { | ||
| 469 | 780 | |
| 470 | 781 | export const api = { |
| 471 | 782 | events: (q: EventQuery = {}) => getJson<EventsPage>(`/api/v1/events${eventQueryString(q)}`, { items: [], nextCursor: null }), |
| 472 | − /** Importance ≥ 80 over the last 48 h, ranked. */ | |
| 473 | − breaking: (limit = 60) => getJson<EventsPage>(`/api/v1/events${eventQueryString({ importance_min: 80, order: "importance", after: new Date(Date.now() - 48 * 3600e3).toISOString(), limit })}`, { items: [], nextCursor: null }), | |
| 783 | + /** Signal ≥ 80 over the last 48 h, ranked (legacy list; the /breaking desk uses `breakingDesk`). */ | |
| 784 | + breaking: (limit = 60) => getJson<EventsPage>(`/api/v1/events${eventQueryString({ signal_min: 80, order: "signal", after: new Date(Date.now() - 48 * 3600e3).toISOString(), limit })}`, { items: [], nextCursor: null }), | |
| 785 | + breakingDesk: () => getJson<BreakingDesk | null>("/api/v1/breaking", null), | |
| 786 | + pulse: () => getJson<Pulse | null>("/api/v1/pulse", null), | |
| 787 | + radar: () => getJson<Radar | null>("/api/v1/radar", null), | |
| 788 | + cluster: (idOrSlug: string) => getJson<ClusterDetail | null>(`/api/v1/clusters/${encodeURIComponent(idOrSlug)}`, null), | |
| 789 | + rank: (limit = 100) => getJson<{ items: RankedEntity[] }>(`/api/v1/entities/rank?limit=${limit}`, { items: [] }), | |
| 790 | + countries: () => getJson<{ items: CountryRow[] }>("/api/v1/countries", { items: [] }), | |
| 791 | + country: (code: string) => getJson<CountryDesk | null>(`/api/v1/countries/${encodeURIComponent(code)}`, null), | |
| 792 | + categoryDesk: (channel: string) => getJson<CategoryDesk | null>(`/api/v1/categories/${encodeURIComponent(channel)}`, null), | |
| 793 | + sensorSnapshots: (id: string, limit = 200) => getJson<{ items: SnapshotRow[] }>(`/api/v1/sensors/${encodeURIComponent(id)}/snapshots?limit=${limit}`, { items: [] }), | |
| 794 | + eventCount: (q: EventQuery = {}) => getJson<{ count: number }>(`/api/v1/events/count${eventQueryString(q)}`, { count: 0 }), | |
| 474 | 795 | event: (idOrSlug: string) => getJson<EventDetail | null>(`/api/v1/events/${encodeURIComponent(idOrSlug)}`, null), |
| 475 | 796 | stats: () => getJson<Stats>("/api/v1/stats", {}, { revalidate: 5 }), |
| 476 | 797 | trending: (hours = 24, limit = 10) => getJson<{ items: TrendingItem[] }>(`/api/v1/trending?hours=${hours}&limit=${limit}`, { items: [] }), |
| 477 | 798 | explore: () => getJson<Explore | null>("/api/v1/explore", null), |
| 478 | 799 | clusters: (limit = 20, since = 72) => getJson<{ items: Cluster[] }>(`/api/v1/clusters?limit=${limit}&since=${since}`, { items: [] }), |
| 479 | − sources: (q: { category?: string; q?: string } = {}) => getJson<{ items: SourceRow[] }>(`/api/v1/sources${eventQueryString(q)}`, { items: [] }), | |
| 800 | + sources: (q: { category?: string; q?: string; country?: string; tier?: string; first_party?: string } = {}) => getJson<{ items: SourceRow[] }>(`/api/v1/sources${eventQueryString(q as EventQuery)}`, { items: [] }), | |
| 480 | 801 | source: (id: string) => getJson<SourceDetail | null>(`/api/v1/sources/${encodeURIComponent(id)}`, null), |
| 481 | 802 | sensor: (id: string) => getJson<SensorDetail | null>(`/api/v1/sensors/${encodeURIComponent(id)}`, null), |
| 482 | 803 | entities: (q: { type?: string; q?: string; limit?: number } = {}) => getJson<{ items: EntityRow[] }>(`/api/v1/entities${eventQueryString(q)}`, { items: [] }), |
added
apps/web/src/lib/feed-filters.ts
+34 −0
@@ -0,0 +1,34 @@ | ||
| 1 | +/** Filters that live in the URL (spec §99). Shared by server pages and the client feed. */ | |
| 2 | +export interface FeedFilters { | |
| 3 | + category?: string; | |
| 4 | + group?: string; | |
| 5 | + event_type?: string; | |
| 6 | + signal_min?: number; | |
| 7 | + importance_min?: number; | |
| 8 | + silent_change?: boolean; | |
| 9 | + first_party?: boolean; | |
| 10 | + confirmed?: boolean; | |
| 11 | + country?: string; | |
| 12 | + language?: string; | |
| 13 | + q?: string; | |
| 14 | + order?: "recent" | "signal" | "importance"; | |
| 15 | +} | |
| 16 | + | |
| 17 | +export function filtersFromParams(sp: URLSearchParams | Record<string, string | undefined>): FeedFilters { | |
| 18 | + const get = (k: string): string | undefined => (sp instanceof URLSearchParams ? sp.get(k) ?? undefined : sp[k]); | |
| 19 | + const f: FeedFilters = {}; | |
| 20 | + if (get("category")) f.category = get("category"); | |
| 21 | + if (get("group")) f.group = get("group"); | |
| 22 | + if (get("event_type")) f.event_type = get("event_type"); | |
| 23 | + if (get("signal_min")) f.signal_min = Number(get("signal_min")); | |
| 24 | + if (get("importance_min")) f.importance_min = Number(get("importance_min")); | |
| 25 | + if (get("silent_change") === "true") f.silent_change = true; | |
| 26 | + if (get("first_party") === "true") f.first_party = true; | |
| 27 | + if (get("confirmed") === "true") f.confirmed = true; | |
| 28 | + if (get("country")) f.country = get("country")!.toUpperCase(); | |
| 29 | + if (get("language")) f.language = get("language"); | |
| 30 | + if (get("q")) f.q = get("q"); | |
| 31 | + const o = get("order"); | |
| 32 | + if (o === "signal" || o === "importance" || o === "recent") f.order = o; | |
| 33 | + return f; | |
| 34 | +} | |
modified
apps/web/src/lib/format.ts
+83 −0
@@ -142,3 +142,86 @@ export const CHANNELS: { key: string; label: string; query: Record<string, strin | ||
| 142 | 142 | ]; |
| 143 | 143 | |
| 144 | 144 | export const CHANNEL_KEYS = ["ai", "cyber", "finance", "health", "government", "science", "products", "infrastructure", "news"] as const; |
| 145 | + | |
| 146 | +// --------------------------------------------------------------------------------------- | |
| 147 | +// 0.2 additions | |
| 148 | +// --------------------------------------------------------------------------------------- | |
| 149 | + | |
| 150 | +/** Lead time / offsets: "+14 s", "+2 min", "+3.2 h" */ | |
| 151 | +export function fmtOffset(ms: number | null | undefined): string { | |
| 152 | + if (ms === null || ms === undefined || Number.isNaN(ms)) return "—"; | |
| 153 | + const sign = ms < 0 ? "−" : "+"; | |
| 154 | + const a = Math.abs(ms); | |
| 155 | + if (a < 1000) return `${sign}${Math.round(a)} ms`; | |
| 156 | + if (a < 60_000) return `${sign}${Math.round(a / 1000)} s`; | |
| 157 | + if (a < 3_600_000) return `${sign}${Math.round(a / 60_000)} min`; | |
| 158 | + if (a < 86_400_000) return `${sign}${(a / 3_600_000).toFixed(1)} h`; | |
| 159 | + return `${sign}${(a / 86_400_000).toFixed(1)} d`; | |
| 160 | +} | |
| 161 | + | |
| 162 | +export function fmtPctDelta(pct: number | null | undefined): string { | |
| 163 | + if (pct === null || pct === undefined || Number.isNaN(pct)) return ""; | |
| 164 | + return `${pct > 0 ? "+" : ""}${Math.abs(pct) >= 100 ? Math.round(pct) : pct}%`; | |
| 165 | +} | |
| 166 | + | |
| 167 | +export function signalBand(score: number | null | undefined): "hot" | "high" | "mid" | "low" { | |
| 168 | + return importanceBand(score ?? 0); | |
| 169 | +} | |
| 170 | + | |
| 171 | +export function localDateTime(iso: string | Date | null | undefined): string { | |
| 172 | + if (!iso) return "—"; | |
| 173 | + const d = typeof iso === "string" ? new Date(iso) : iso; | |
| 174 | + if (Number.isNaN(d.getTime())) return "—"; | |
| 175 | + return new Intl.DateTimeFormat(undefined, { year: "numeric", month: "short", day: "2-digit", hour: "2-digit", minute: "2-digit", second: "2-digit", timeZoneName: "short" }).format(d); | |
| 176 | +} | |
| 177 | + | |
| 178 | +export const GROUP_LABELS: Record<string, string> = { security: "Security", reliability: "Reliability", product: "Product & API", commercial: "Pricing & terms", corporate: "Corporate", government: "Government & legal", science: "Science & health", transport: "Transport", sports: "Sports", web: "Web" }; | |
| 179 | + | |
| 180 | +export const CLASS_LABELS: Record<string, string> = { meaningful: "Content", pricing: "Pricing", policy: "Policy", product: "Product", personnel: "Personnel", cosmetic: "Cosmetic", navigation: "Navigation", timestamp: "Timestamp", advertisement: "Advertising", boilerplate: "Boilerplate" }; | |
| 181 | + | |
| 182 | +export function stateLabel(s: string | null | undefined): string { | |
| 183 | + switch (s) { | |
| 184 | + case "breaking": | |
| 185 | + return "BREAKING"; | |
| 186 | + case "developing": | |
| 187 | + return "DEVELOPING"; | |
| 188 | + case "confirmed": | |
| 189 | + return "CONFIRMED"; | |
| 190 | + case "watching": | |
| 191 | + return "WATCHING"; | |
| 192 | + case "closed": | |
| 193 | + return "CLOSED"; | |
| 194 | + default: | |
| 195 | + return ""; | |
| 196 | + } | |
| 197 | +} | |
| 198 | + | |
| 199 | +/** Build the live-feed URL for a filter set (spec §99). */ | |
| 200 | +export function feedHref(q: Record<string, string | number | boolean | undefined | null>, base = "/live"): string { | |
| 201 | + const p = new URLSearchParams(); | |
| 202 | + for (const [k, v] of Object.entries(q)) if (v !== undefined && v !== null && v !== "" && v !== false) p.set(k, String(v)); | |
| 203 | + const s = p.toString(); | |
| 204 | + return s ? `${base}?${s}` : base; | |
| 205 | +} | |
| 206 | + | |
| 207 | +export const SAVED_VIEWS: { key: string; label: string; query: Record<string, string | number | boolean> }[] = [ | |
| 208 | + { key: "ai-releases", label: "AI releases", query: { category: "ai", event_type: "model_release,software_release,product_launch,API_change,api_change" } }, | |
| 209 | + { key: "cyber-critical", label: "Cyber critical", query: { group: "security", signal_min: 70 } }, | |
| 210 | + { key: "canada-gov", label: "Canadian government", query: { country: "CA", category: "government" } }, | |
| 211 | + { key: "market-breaking", label: "Market breaking", query: { category: "finance", signal_min: 75 } }, | |
| 212 | + { key: "cloud-outages", label: "Cloud outages", query: { category: "infrastructure", group: "reliability" } }, | |
| 213 | + { key: "silent-pricing", label: "Silent pricing & terms", query: { silent_change: true, group: "commercial" } }, | |
| 214 | + { key: "first-party-confirmed", label: "First-party & confirmed", query: { first_party: true, confirmed: true } }, | |
| 215 | +]; | |
| 216 | + | |
| 217 | +/** ISO timestamp `ms` milliseconds ago — keeps `Date.now()` out of component bodies (react-hooks/purity). */ | |
| 218 | +export function agoIso(ms: number): string { | |
| 219 | + return new Date(Date.now() - ms).toISOString(); | |
| 220 | +} | |
| 221 | + | |
| 222 | +/** True when `iso` falls within the last `windowMs` milliseconds. */ | |
| 223 | +export function withinLast(iso: string | null | undefined, windowMs: number): boolean { | |
| 224 | + if (!iso) return false; | |
| 225 | + const t = new Date(iso).getTime(); | |
| 226 | + return Number.isFinite(t) && Date.now() - t < windowMs; | |
| 227 | +} | |
modified
apps/web/src/lib/use-live.ts
+30 −6
@@ -14,12 +14,15 @@ export function wsUrl(): string { | ||
| 14 | 14 | } |
| 15 | 15 | |
| 16 | 16 | /** |
| 17 | − * Shared WebSocket to /api/v1/live. Subscribes to the given channels and calls `onEvent` | |
| 18 | − * for every matching event. Reconnects with exponential backoff (1 s → 30 s). | |
| 17 | + * Shared WebSocket to /api/v1/live (protocol 2). Subscribes to the given channels and calls | |
| 18 | + * `onEvent` for every matching event. Reconnects with exponential backoff (1 s → 30 s) and, on | |
| 19 | + * reconnection, asks the gateway to replay everything after the last stream id seen (spec §60), | |
| 20 | + * so a laptop waking from sleep catches up without duplicates (ids are de-duplicated by callers). | |
| 19 | 21 | */ |
| 20 | 22 | export function useLive(channels: string[], onEvent: (ev: LiveEvent, channels: string[]) => void): LiveStatus { |
| 21 | 23 | const [status, setStatus] = useState<LiveStatus>("connecting"); |
| 22 | 24 | const handler = useRef(onEvent); |
| 25 | + const lastSid = useRef<string | null>(null); | |
| 23 | 26 | useEffect(() => { |
| 24 | 27 | handler.current = onEvent; |
| 25 | 28 | }); |
@@ -30,6 +33,8 @@ export function useLive(channels: string[], onEvent: (ev: LiveEvent, channels: s | ||
| 30 | 33 | let closed = false; |
| 31 | 34 | let attempt = 0; |
| 32 | 35 | let timer: ReturnType<typeof setTimeout> | null = null; |
| 36 | + let watchdog: ReturnType<typeof setInterval> | null = null; | |
| 37 | + let lastFrame = Date.now(); | |
| 33 | 38 | const subs = key.split("|").filter(Boolean); |
| 34 | 39 | |
| 35 | 40 | const connect = (): void => { |
@@ -41,14 +46,20 @@ export function useLive(channels: string[], onEvent: (ev: LiveEvent, channels: s | ||
| 41 | 46 | return; |
| 42 | 47 | } |
| 43 | 48 | ws.onopen = () => { |
| 49 | + const wasReconnect = attempt > 0; | |
| 44 | 50 | attempt = 0; |
| 51 | + lastFrame = Date.now(); | |
| 45 | 52 | setStatus("live"); |
| 46 | − ws?.send(JSON.stringify({ subscribe: subs, unsubscribe: subs.includes("events:global") ? [] : ["events:global"] })); | |
| 53 | + ws?.send(JSON.stringify({ subscribe: subs, unsubscribe: subs.includes("events:global") ? [] : ["events:global"], ...(wasReconnect && lastSid.current ? { since: lastSid.current } : {}) })); | |
| 47 | 54 | }; |
| 48 | 55 | ws.onmessage = (m) => { |
| 56 | + lastFrame = Date.now(); | |
| 49 | 57 | try { |
| 50 | − const msg = JSON.parse(String(m.data)) as { type: string; event?: LiveEvent; channels?: string[] }; | |
| 51 | − if (msg.type === "event" && msg.event) handler.current(msg.event, msg.channels ?? []); | |
| 58 | + const msg = JSON.parse(String(m.data)) as { type: string; sid?: string; event?: LiveEvent; channels?: string[] }; | |
| 59 | + if (msg.type === "event" && msg.event) { | |
| 60 | + if (msg.sid) lastSid.current = msg.sid; | |
| 61 | + handler.current({ ...msg.event, sid: msg.sid }, msg.channels ?? []); | |
| 62 | + } | |
| 52 | 63 | } catch { |
| 53 | 64 | // ignore malformed frames |
| 54 | 65 | } |
@@ -64,14 +75,27 @@ export function useLive(channels: string[], onEvent: (ev: LiveEvent, channels: s | ||
| 64 | 75 | }; |
| 65 | 76 | const schedule = (): void => { |
| 66 | 77 | if (closed) return; |
| 67 | − const delay = Math.min(30_000, 1000 * 2 ** Math.min(attempt, 5)); | |
| 78 | + const delay = Math.min(30_000, 1000 * 2 ** Math.min(attempt, 5)) * (0.8 + Math.random() * 0.4); | |
| 68 | 79 | attempt++; |
| 69 | 80 | timer = setTimeout(connect, delay); |
| 70 | 81 | }; |
| 82 | + // Heartbeats arrive every 25 s; if nothing for 70 s the socket is half-dead → reconnect. | |
| 83 | + watchdog = setInterval(() => { | |
| 84 | + if (ws && ws.readyState === WebSocket.OPEN && Date.now() - lastFrame > 70_000) ws.close(); | |
| 85 | + }, 10_000); | |
| 86 | + const onVisible = (): void => { | |
| 87 | + if (document.visibilityState === "visible" && ws && ws.readyState !== WebSocket.OPEN && ws.readyState !== WebSocket.CONNECTING) { | |
| 88 | + if (timer) clearTimeout(timer); | |
| 89 | + connect(); | |
| 90 | + } | |
| 91 | + }; | |
| 92 | + document.addEventListener("visibilitychange", onVisible); | |
| 71 | 93 | connect(); |
| 72 | 94 | return () => { |
| 73 | 95 | closed = true; |
| 74 | 96 | if (timer) clearTimeout(timer); |
| 97 | + if (watchdog) clearInterval(watchdog); | |
| 98 | + document.removeEventListener("visibilitychange", onVisible); | |
| 75 | 99 | ws?.close(); |
| 76 | 100 | }; |
| 77 | 101 | }, [key]); |
added
config/sources.d/40-ai-frontier.yaml
+1129 −0
@@ -0,0 +1,1129 @@ | ||
| 1 | +# WebSensor — AI frontier depth coverage (added 2026-09-11). Many validated sensors per AI organization: model | |
| 2 | +# releases (Hugging Face public API `api/models?author=…`, sorted by lastModified — items carry no URL because the | |
| 3 | +# jsonlist `urlTemplate` URL-encodes the `owner/model` key), API documentation (model lists, pricing, deprecations, | |
| 4 | +# release notes), SDK / runtime GitHub releases, research & blog feeds, system-card / transparency pages, status pages | |
| 5 | +# (Atlassian summary.json only — Betterstack pages such as status.huggingface.co, status.together.ai, status.modal.com, | |
| 6 | +# uptime.runpod.io expose no JSON), llms.txt indexes and Epoch AI open CSV datasets. Existing organizations are | |
| 7 | +# extended (`extend: true`); Chinese labs, inference clouds, coding assistants, tooling and evaluators are declared here. | |
| 8 | +# Blocked / unusable (do not re-probe blindly): x.ai + status.x.ai (403), status.mistral.ai + status.replit.com + | |
| 9 | +# docs.midjourney.com + help.runwayml.com + perplexity.ai/hub (Cloudflare challenge), ai.meta.com + llama.com (400 to | |
| 10 | +# non-browser clients), help.openai.com + openai.com/research (403), ir.baidu.com (403), status.openrouter.ai (403), | |
| 11 | +# api.ngc.nvidia.com (401), status.deepseek.com (custom Flashcat page), status.langchain.com / status.moonshot.ai / | |
| 12 | +# status.z.ai / status.runpod.io (no DNS), platform.stability.ai + platform.openai.com + bigmodel.cn (client-rendered). | |
| 13 | +sources: | |
| 14 | + # ───────────────────────── Frontier labs (extend founding sources) ───────────────────────── | |
| 15 | + - id: openai | |
| 16 | + extend: true | |
| 17 | + country: US | |
| 18 | + products: | |
| 19 | + - { name: gpt-oss, type: AI_model, aliases: [gpt-oss-120b, gpt-oss-20b] } | |
| 20 | + - { name: Whisper, type: AI_model } | |
| 21 | + - { name: Agents SDK, type: software, aliases: [openai agents sdk] } | |
| 22 | + sensors: | |
| 23 | + - { name: api changelog, url: "https://developers.openai.com/api/docs/changelog", type: HTML, connector: http, tier: A } | |
| 24 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=openai&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 25 | + - { name: sdk node releases, url: "https://github.com/openai/openai-node/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: openai/openai-node, kind: releases } } | |
| 26 | + - { name: codex releases, url: "https://github.com/openai/codex/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: openai/codex, kind: releases } } | |
| 27 | + - { name: agents sdk python releases, url: "https://github.com/openai/openai-agents-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: openai/openai-agents-python, kind: releases } } | |
| 28 | + - { name: agents sdk js releases, url: "https://github.com/openai/openai-agents-js/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: openai/openai-agents-js, kind: releases } } | |
| 29 | + - { name: gpt-oss tags, url: "https://github.com/openai/gpt-oss/tags.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: openai/gpt-oss, kind: tags } } | |
| 30 | + - { name: tiktoken releases, url: "https://github.com/openai/tiktoken/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: openai/tiktoken, kind: releases } } | |
| 31 | + - { name: whisper tags, url: "https://github.com/openai/whisper/tags.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: openai/whisper, kind: tags } } | |
| 32 | + - { name: openapi spec commits, url: "https://github.com/openai/openai-openapi/commits/manual_spec.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: openai/openai-openapi, kind: commits, branch: manual_spec } } | |
| 33 | + - { name: cookbook commits, url: "https://github.com/openai/openai-cookbook/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: openai/openai-cookbook, kind: commits, branch: main } } | |
| 34 | + - id: anthropic | |
| 35 | + extend: true | |
| 36 | + country: US | |
| 37 | + aliases: [claude ai] | |
| 38 | + products: | |
| 39 | + - { name: Claude Developer Platform, type: API, aliases: [platform.claude.com, claude platform] } | |
| 40 | + - { name: Claude Agent SDK, type: software, aliases: [anthropic sdk] } | |
| 41 | + sensors: | |
| 42 | + - { name: models overview, url: "https://platform.claude.com/docs/en/models/overview", type: HTML, connector: http, tier: A } | |
| 43 | + - { name: model deprecations, url: "https://platform.claude.com/docs/en/about-claude/model-deprecations", type: HTML, connector: http, tier: B } | |
| 44 | + - { name: platform release notes, url: "https://platform.claude.com/docs/en/release-notes/overview", type: HTML, connector: http, tier: A } | |
| 45 | + - { name: docs llms.txt, url: "https://platform.claude.com/llms.txt", type: FILE, connector: http, tier: D } | |
| 46 | + - { name: claude code changelog, url: "https://raw.githubusercontent.com/anthropics/claude-code/main/CHANGELOG.md", type: FILE, connector: http, tier: A } | |
| 47 | + - { name: research, url: "https://www.anthropic.com/research", type: HTML, connector: http, tier: A } | |
| 48 | + - { name: engineering blog, url: "https://www.anthropic.com/engineering", type: HTML, connector: http, tier: B } | |
| 49 | + - { name: transparency hub, url: "https://www.anthropic.com/transparency", type: HTML, connector: http, tier: C } | |
| 50 | + - { name: alignment science blog, url: "https://alignment.anthropic.com/", type: HTML, connector: http, tier: B } | |
| 51 | + - { name: red team blog, url: "https://red.anthropic.com/", type: HTML, connector: http, tier: B } | |
| 52 | + - { name: transformer circuits, url: "https://transformer-circuits.pub/", type: HTML, connector: http, tier: B } | |
| 53 | + - { name: claude.com pricing, url: "https://claude.com/pricing", type: HTML, connector: http, tier: C } | |
| 54 | + - { name: claude.com blog, url: "https://claude.com/blog", type: HTML, connector: http, tier: B } | |
| 55 | + - { name: sdk python releases, url: "https://github.com/anthropics/anthropic-sdk-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: anthropics/anthropic-sdk-python, kind: releases } } | |
| 56 | + - { name: sdk go releases, url: "https://github.com/anthropics/anthropic-sdk-go/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: anthropics/anthropic-sdk-go, kind: releases } } | |
| 57 | + - { name: sdk java releases, url: "https://github.com/anthropics/anthropic-sdk-java/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: anthropics/anthropic-sdk-java, kind: releases } } | |
| 58 | + - { name: cookbooks commits, url: "https://github.com/anthropics/claude-cookbooks/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: anthropics/claude-cookbooks, kind: commits, branch: main } } | |
| 59 | + - { name: skills commits, url: "https://github.com/anthropics/skills/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: anthropics/skills, kind: commits, branch: main } } | |
| 60 | + - id: deepmind | |
| 61 | + extend: true | |
| 62 | + country: US | |
| 63 | + products: | |
| 64 | + - { name: Gemma, type: AI_model, aliases: [gemma 3] } | |
| 65 | + - { name: Gemini app, type: product, aliases: [gemini.google] } | |
| 66 | + sensors: | |
| 67 | + - { name: google blog deepmind feed, url: "https://blog.google/innovation-and-ai/models-and-research/google-deepmind/rss/", type: RSS, connector: rss, tier: A } | |
| 68 | + - { name: models page, url: "https://deepmind.google/models/", type: HTML, connector: http, tier: A } | |
| 69 | + - { name: publications, url: "https://deepmind.google/research/publications/", type: HTML, connector: http, tier: B } | |
| 70 | + - { name: gemini app release notes, url: "https://gemini.google/release-notes/", type: HTML, connector: http, tier: A } | |
| 71 | + - { name: gemma releases, url: "https://github.com/google-deepmind/gemma/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: google-deepmind/gemma, kind: releases } } | |
| 72 | + - { name: alphafold3 tags, url: "https://github.com/google-deepmind/alphafold3/tags.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: google-deepmind/alphafold3, kind: tags } } | |
| 73 | + - id: google-ai | |
| 74 | + extend: true | |
| 75 | + country: US | |
| 76 | + sensors: | |
| 77 | + - { name: google blog ai feed, url: "https://blog.google/innovation-and-ai/technology/ai/rss/", type: RSS, connector: rss, tier: A } | |
| 78 | + - { name: google research blog feed, url: "https://research.google/blog/rss/", type: RSS, connector: rss, tier: A } | |
| 79 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=google&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 80 | + - id: google-developers-ai | |
| 81 | + extend: true | |
| 82 | + country: US | |
| 83 | + products: | |
| 84 | + - { name: Gemini CLI, type: software } | |
| 85 | + - { name: Agent Development Kit, type: software, aliases: [adk] } | |
| 86 | + sensors: | |
| 87 | + - { name: gemini api pricing, url: "https://ai.google.dev/gemini-api/docs/pricing", type: HTML, connector: http, tier: B } | |
| 88 | + - { name: gemini api models, url: "https://ai.google.dev/gemini-api/docs/models", type: HTML, connector: http, tier: A } | |
| 89 | + - { name: gemini api deprecations, url: "https://ai.google.dev/gemini-api/docs/deprecations", type: HTML, connector: http, tier: B } | |
| 90 | + - { name: gemma releases page, url: "https://ai.google.dev/gemma/docs/releases", type: HTML, connector: http, tier: B } | |
| 91 | + - { name: developers blog gemini feed, url: "https://developers.googleblog.com/feeds/posts/default/?category=Gemini", type: ATOM, connector: rss, tier: B } | |
| 92 | + - { name: python genai sdk releases, url: "https://github.com/googleapis/python-genai/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: googleapis/python-genai, kind: releases } } | |
| 93 | + - { name: js genai sdk releases, url: "https://github.com/googleapis/js-genai/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: googleapis/js-genai, kind: releases } } | |
| 94 | + - { name: gemini cli releases, url: "https://github.com/google-gemini/gemini-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: google-gemini/gemini-cli, kind: releases } } | |
| 95 | + - { name: adk python releases, url: "https://github.com/google/adk-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: google/adk-python, kind: releases } } | |
| 96 | + - { name: gemini cookbook commits, url: "https://github.com/google-gemini/cookbook/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: google-gemini/cookbook, kind: commits, branch: main } } | |
| 97 | + - id: google-cloud | |
| 98 | + extend: true | |
| 99 | + products: | |
| 100 | + - { name: Vertex AI, type: product, aliases: [vertex] } | |
| 101 | + sensors: | |
| 102 | + - { name: vertex ai release notes feed, url: "https://docs.cloud.google.com/feeds/vertex-ai-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 103 | + - { name: gemini for google cloud release notes feed, url: "https://docs.cloud.google.com/feeds/gemini-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 104 | + - id: meta-ai | |
| 105 | + extend: true | |
| 106 | + country: US | |
| 107 | + products: | |
| 108 | + - { name: Llama Stack, type: software } | |
| 109 | + notes: "ai.meta.com and llama.com answer 400 to non-browser clients; coverage comes from Hugging Face and GitHub." | |
| 110 | + sensors: | |
| 111 | + - { name: hugging face meta-llama models, url: "https://huggingface.co/api/models?author=meta-llama&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 112 | + - { name: hugging face facebook models, url: "https://huggingface.co/api/models?author=facebook&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 113 | + - { name: engineering blog feed, url: "https://engineering.fb.com/feed/", type: RSS, connector: rss, tier: B } | |
| 114 | + - { name: llama-models commits, url: "https://github.com/meta-llama/llama-models/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: meta-llama/llama-models, kind: commits, branch: main } } | |
| 115 | + - { name: llama-models tags, url: "https://github.com/meta-llama/llama-models/tags.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: meta-llama/llama-models, kind: tags } } | |
| 116 | + - { name: faiss releases, url: "https://github.com/facebookresearch/faiss/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: facebookresearch/faiss, kind: releases } } | |
| 117 | + - id: microsoft-ai | |
| 118 | + extend: true | |
| 119 | + country: US | |
| 120 | + products: | |
| 121 | + - { name: Phi, type: AI_model, aliases: [phi-4, phi-3] } | |
| 122 | + - { name: Semantic Kernel, type: software } | |
| 123 | + - { name: AutoGen, type: software } | |
| 124 | + - { name: Agent Framework, type: software, aliases: [microsoft agent framework] } | |
| 125 | + sensors: | |
| 126 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=microsoft&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 127 | + - { name: agent framework blog feed, url: "https://devblogs.microsoft.com/agent-framework/feed/", type: RSS, connector: rss, tier: B } | |
| 128 | + - { name: semantic kernel releases, url: "https://github.com/microsoft/semantic-kernel/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: microsoft/semantic-kernel, kind: releases } } | |
| 129 | + - { name: autogen releases, url: "https://github.com/microsoft/autogen/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: microsoft/autogen, kind: releases } } | |
| 130 | + - { name: agent framework releases, url: "https://github.com/microsoft/agent-framework/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: microsoft/agent-framework, kind: releases } } | |
| 131 | + - { name: onnx runtime releases, url: "https://github.com/microsoft/onnxruntime/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: microsoft/onnxruntime, kind: releases } } | |
| 132 | + - { name: deepspeed releases, url: "https://github.com/deepspeedai/DeepSpeed/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: deepspeedai/DeepSpeed, kind: releases } } | |
| 133 | + - { name: markitdown releases, url: "https://github.com/microsoft/markitdown/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: microsoft/markitdown, kind: releases } } | |
| 134 | + - { name: vscode copilot chat releases, url: "https://github.com/microsoft/vscode-copilot-chat/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: microsoft/vscode-copilot-chat, kind: releases } } | |
| 135 | + - id: azure-ai | |
| 136 | + extend: true | |
| 137 | + country: US | |
| 138 | + sensors: | |
| 139 | + - { name: foundry blog feed, url: "https://devblogs.microsoft.com/foundry/feed/", type: RSS, connector: rss, tier: B } | |
| 140 | + - { name: azure blog feed, url: "https://azure.microsoft.com/en-us/blog/feed/", type: RSS, connector: rss, tier: B } | |
| 141 | + - { name: foundry whats new, url: "https://learn.microsoft.com/en-us/azure/foundry/whats-new-foundry", type: HTML, connector: http, tier: B } | |
| 142 | + - { name: azure openai whats new, url: "https://learn.microsoft.com/en-us/azure/foundry-classic/openai/whats-new", type: HTML, connector: http, tier: A } | |
| 143 | + - { name: foundry models sold directly by azure, url: "https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/concepts/models-sold-directly-by-azure", type: HTML, connector: http, tier: B } | |
| 144 | + - id: microsoft-research | |
| 145 | + name: Microsoft Research | |
| 146 | + domain: microsoft.com | |
| 147 | + homepage: https://www.microsoft.com/en-us/research/ | |
| 148 | + categories: [ai, research, technology] | |
| 149 | + tier: B | |
| 150 | + weight: 1.1 | |
| 151 | + country: US | |
| 152 | + aliases: [msr, microsoft research] | |
| 153 | + discover: { rss: true } | |
| 154 | + sensors: | |
| 155 | + - { name: research blog feed, url: "https://www.microsoft.com/en-us/research/blog/feed/", type: RSS, connector: rss, tier: B } | |
| 156 | + - id: xai | |
| 157 | + extend: true | |
| 158 | + country: US | |
| 159 | + notes: "x.ai and status.x.ai return 403 to non-browser clients (Scrapfly fallback on the news page); docs.x.ai is open." | |
| 160 | + sensors: | |
| 161 | + - { name: api release notes, url: "https://docs.x.ai/developers/release-notes", type: HTML, connector: http, tier: A } | |
| 162 | + - { name: docs llms.txt, url: "https://docs.x.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 163 | + - { name: xai sdk python releases, url: "https://github.com/xai-org/xai-sdk-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: xai-org/xai-sdk-python, kind: releases } } | |
| 164 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=xai-org&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 165 | + - id: mistral | |
| 166 | + extend: true | |
| 167 | + country: FR | |
| 168 | + products: | |
| 169 | + - { name: Mistral Vibe, type: software, aliases: [vibe cli] } | |
| 170 | + - { name: Magistral, type: AI_model } | |
| 171 | + - { name: Devstral, type: AI_model } | |
| 172 | + notes: "status.mistral.ai sits behind a Cloudflare challenge." | |
| 173 | + sensors: | |
| 174 | + - { name: news feed, url: "https://mistral.ai/news/rss", type: RSS, connector: rss, tier: A } | |
| 175 | + - { name: pricing, url: "https://mistral.ai/pricing/", type: HTML, connector: http, tier: C } | |
| 176 | + - { name: models docs, url: "https://docs.mistral.ai/models", type: HTML, connector: http, tier: A } | |
| 177 | + - { name: docs llms.txt, url: "https://docs.mistral.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 178 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=mistralai&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 179 | + - { name: client python releases, url: "https://github.com/mistralai/client-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: mistralai/client-python, kind: releases } } | |
| 180 | + - { name: client ts releases, url: "https://github.com/mistralai/client-ts/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: mistralai/client-ts, kind: releases } } | |
| 181 | + - { name: mistral-common releases, url: "https://github.com/mistralai/mistral-common/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: mistralai/mistral-common, kind: releases } } | |
| 182 | + - { name: mistral vibe releases, url: "https://github.com/mistralai/mistral-vibe/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: mistralai/mistral-vibe, kind: releases } } | |
| 183 | + - { name: mistral-inference tags, url: "https://github.com/mistralai/mistral-inference/tags.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: mistralai/mistral-inference, kind: tags } } | |
| 184 | + - id: cohere | |
| 185 | + extend: true | |
| 186 | + country: CA | |
| 187 | + products: | |
| 188 | + - { name: Aya, type: AI_model } | |
| 189 | + - { name: Embed, type: AI_model, aliases: [cohere embed] } | |
| 190 | + - { name: Rerank, type: AI_model, aliases: [cohere rerank] } | |
| 191 | + notes: "cohere.com/pricing and /research are client-hydrated (validator: thin, even with selector main)." | |
| 192 | + sensors: | |
| 193 | + - { name: status, url: "https://status.cohere.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 194 | + - { name: models docs, url: "https://docs.cohere.com/docs/models", type: HTML, connector: http, tier: A } | |
| 195 | + - { name: blog index, url: "https://cohere.com/blog", type: HTML, connector: http, tier: B } | |
| 196 | + - { name: docs llms.txt, url: "https://docs.cohere.com/llms.txt", type: FILE, connector: http, tier: D } | |
| 197 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=CohereLabs&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 198 | + - { name: sdk python releases, url: "https://github.com/cohere-ai/cohere-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cohere-ai/cohere-python, kind: releases } } | |
| 199 | + - { name: sdk typescript releases, url: "https://github.com/cohere-ai/cohere-typescript/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: cohere-ai/cohere-typescript, kind: releases } } | |
| 200 | + - id: huggingface | |
| 201 | + extend: true | |
| 202 | + country: US | |
| 203 | + products: | |
| 204 | + - { name: Diffusers, type: software } | |
| 205 | + - { name: Hugging Face Hub, type: service, aliases: [hf hub] } | |
| 206 | + - { name: SmolLM, type: AI_model, aliases: [smollm3] } | |
| 207 | + - { name: Text Generation Inference, type: software, aliases: [tgi] } | |
| 208 | + notes: "status.huggingface.co is a Betterstack page (no summary JSON)." | |
| 209 | + sensors: | |
| 210 | + - { name: pricing, url: "https://huggingface.co/pricing", type: HTML, connector: http, tier: C } | |
| 211 | + - { name: hub changelog, url: "https://huggingface.co/changelog", type: HTML, connector: http, tier: B } | |
| 212 | + - { name: daily papers api, url: "https://huggingface.co/api/daily_papers", type: REST_API, connector: jsonlist, tier: B, config: { keyField: paper.id, titleField: title, dateField: publishedAt, urlTemplate: "https://huggingface.co/papers/{key}", compareFields: [title], maxItems: 50 } } | |
| 213 | + - { name: hugging face smollm models, url: "https://huggingface.co/api/models?author=HuggingFaceTB&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 214 | + - { name: diffusers releases, url: "https://github.com/huggingface/diffusers/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: huggingface/diffusers, kind: releases } } | |
| 215 | + - { name: huggingface_hub releases, url: "https://github.com/huggingface/huggingface_hub/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: huggingface/huggingface_hub, kind: releases } } | |
| 216 | + - { name: datasets releases, url: "https://github.com/huggingface/datasets/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/datasets, kind: releases } } | |
| 217 | + - { name: tokenizers releases, url: "https://github.com/huggingface/tokenizers/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/tokenizers, kind: releases } } | |
| 218 | + - { name: peft releases, url: "https://github.com/huggingface/peft/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/peft, kind: releases } } | |
| 219 | + - { name: trl releases, url: "https://github.com/huggingface/trl/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/trl, kind: releases } } | |
| 220 | + - { name: accelerate releases, url: "https://github.com/huggingface/accelerate/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/accelerate, kind: releases } } | |
| 221 | + - { name: text generation inference releases, url: "https://github.com/huggingface/text-generation-inference/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: huggingface/text-generation-inference, kind: releases } } | |
| 222 | + - { name: smolagents releases, url: "https://github.com/huggingface/smolagents/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/smolagents, kind: releases } } | |
| 223 | + - { name: lerobot releases, url: "https://github.com/huggingface/lerobot/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/lerobot, kind: releases } } | |
| 224 | + - { name: transformers.js releases, url: "https://github.com/huggingface/transformers.js/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/transformers.js, kind: releases } } | |
| 225 | + - { name: candle tags, url: "https://github.com/huggingface/candle/tags.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: huggingface/candle, kind: tags } } | |
| 226 | + # ───────────────────────── Silicon & platforms ───────────────────────── | |
| 227 | + - id: nvidia | |
| 228 | + extend: true | |
| 229 | + country: US | |
| 230 | + products: | |
| 231 | + - { name: TensorRT-LLM, type: software, aliases: [tensorrt llm] } | |
| 232 | + - { name: NeMo, type: software, aliases: [nvidia nemo] } | |
| 233 | + - { name: Nemotron, type: AI_model } | |
| 234 | + notes: "NGC catalog API (api.ngc.nvidia.com) requires authentication; NIM release notes are client-rendered." | |
| 235 | + sensors: | |
| 236 | + - { name: corporate blog feed, url: "https://blogs.nvidia.com/feed/", type: RSS, connector: rss, tier: A } | |
| 237 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=nvidia&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 238 | + - { name: cuda toolkit release notes, url: "https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html", type: HTML, connector: http, tier: B } | |
| 239 | + - { name: tensorrt release notes, url: "https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes.html", type: HTML, connector: http, tier: C } | |
| 240 | + - { name: research publications, url: "https://research.nvidia.com/publications", type: HTML, connector: http, tier: B } | |
| 241 | + - { name: tensorrt-llm releases, url: "https://github.com/NVIDIA/TensorRT-LLM/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: NVIDIA/TensorRT-LLM, kind: releases } } | |
| 242 | + - { name: tensorrt releases, url: "https://github.com/NVIDIA/TensorRT/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/TensorRT, kind: releases } } | |
| 243 | + - { name: triton inference server releases, url: "https://github.com/triton-inference-server/server/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: triton-inference-server/server, kind: releases } } | |
| 244 | + - { name: megatron-lm releases, url: "https://github.com/NVIDIA/Megatron-LM/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/Megatron-LM, kind: releases } } | |
| 245 | + - { name: cutlass releases, url: "https://github.com/NVIDIA/cutlass/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/cutlass, kind: releases } } | |
| 246 | + - { name: cuda python releases, url: "https://github.com/NVIDIA/cuda-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/cuda-python, kind: releases } } | |
| 247 | + - { name: container toolkit releases, url: "https://github.com/NVIDIA/nvidia-container-toolkit/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/nvidia-container-toolkit, kind: releases } } | |
| 248 | + - { name: transformer engine releases, url: "https://github.com/NVIDIA/TransformerEngine/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/TransformerEngine, kind: releases } } | |
| 249 | + - { name: nemo agent toolkit releases, url: "https://github.com/NVIDIA/NeMo-Agent-Toolkit/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/NeMo-Agent-Toolkit, kind: releases } } | |
| 250 | + - { name: nemo guardrails releases, url: "https://github.com/NVIDIA-NeMo/Guardrails/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA-NeMo/Guardrails, kind: releases } } | |
| 251 | + - { name: open gpu kernel modules releases, url: "https://github.com/NVIDIA/open-gpu-kernel-modules/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/open-gpu-kernel-modules, kind: releases } } | |
| 252 | + - { name: nim deploy commits, url: "https://github.com/NVIDIA/nim-deploy/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: NVIDIA/nim-deploy, kind: commits, branch: main } } | |
| 253 | + - id: amd | |
| 254 | + extend: true | |
| 255 | + country: US | |
| 256 | + products: | |
| 257 | + - { name: ROCm, type: software, aliases: [rocm] } | |
| 258 | + - { name: Instinct, type: product, aliases: [mi300, mi350, mi400] } | |
| 259 | + sensors: | |
| 260 | + - { name: rocm blogs feed, url: "https://rocm.blogs.amd.com/blog/atom.xml", type: ATOM, connector: rss, tier: B } | |
| 261 | + - { name: rocm release notes, url: "https://rocm.docs.amd.com/en/latest/about/release-notes.html", type: HTML, connector: http, tier: B } | |
| 262 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=amd&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 263 | + - { name: rocm systems releases, url: "https://github.com/ROCm/rocm-systems/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: ROCm/rocm-systems, kind: releases } } | |
| 264 | + - { name: rocm libraries releases, url: "https://github.com/ROCm/rocm-libraries/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: ROCm/rocm-libraries, kind: releases } } | |
| 265 | + - { name: rocm vllm releases, url: "https://github.com/ROCm/vllm/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ROCm/vllm, kind: releases } } | |
| 266 | + - { name: composable kernel releases, url: "https://github.com/ROCm/composable_kernel/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ROCm/composable_kernel, kind: releases } } | |
| 267 | + - { name: rocblas releases, url: "https://github.com/ROCm/rocBLAS/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ROCm/rocBLAS, kind: releases } } | |
| 268 | + - id: apple-ml-research | |
| 269 | + name: Apple Machine Learning Research | |
| 270 | + domain: machinelearning.apple.com | |
| 271 | + categories: [ai, research, technology] | |
| 272 | + tier: B | |
| 273 | + weight: 1.1 | |
| 274 | + country: US | |
| 275 | + aliases: [apple ml research, apple machine learning] | |
| 276 | + products: | |
| 277 | + - { name: Core ML, type: software, aliases: [coreml, coremltools] } | |
| 278 | + - { name: Apple Foundation Models, type: AI_model, aliases: [apple intelligence models] } | |
| 279 | + discover: { rss: true } | |
| 280 | + sensors: | |
| 281 | + - { name: research feed, url: "https://machinelearning.apple.com/rss.xml", type: RSS, connector: rss, tier: B } | |
| 282 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=apple&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 283 | + - { name: coremltools releases, url: "https://github.com/apple/coremltools/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: apple/coremltools, kind: releases } } | |
| 284 | + - id: mlx | |
| 285 | + name: MLX | |
| 286 | + domain: ml-explore.github.io | |
| 287 | + homepage: https://ml-explore.github.io/mlx/ | |
| 288 | + categories: [ai, open-source, developer] | |
| 289 | + tier: B | |
| 290 | + country: US | |
| 291 | + aliases: [mlx framework, apple mlx, mlx-lm] | |
| 292 | + products: | |
| 293 | + - { name: mlx-lm, type: software } | |
| 294 | + - { name: MLX Swift, type: software, aliases: [mlx-swift] } | |
| 295 | + discover: { rss: false } | |
| 296 | + sensors: | |
| 297 | + - { name: mlx releases, url: "https://github.com/ml-explore/mlx/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: ml-explore/mlx, kind: releases } } | |
| 298 | + - { name: mlx-lm releases, url: "https://github.com/ml-explore/mlx-lm/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: ml-explore/mlx-lm, kind: releases } } | |
| 299 | + - { name: mlx-swift releases, url: "https://github.com/ml-explore/mlx-swift/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ml-explore/mlx-swift, kind: releases } } | |
| 300 | + - { name: mlx-swift-lm releases, url: "https://github.com/ml-explore/mlx-swift-lm/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ml-explore/mlx-swift-lm, kind: releases } } | |
| 301 | + - { name: mlx-examples commits, url: "https://github.com/ml-explore/mlx-examples/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ml-explore/mlx-examples, kind: commits, branch: main } } | |
| 302 | + - id: aws | |
| 303 | + extend: true | |
| 304 | + country: US | |
| 305 | + products: | |
| 306 | + - { name: Amazon Nova, type: AI_model, aliases: [nova] } | |
| 307 | + - { name: Strands Agents, type: software } | |
| 308 | + - { name: Bedrock AgentCore, type: product, aliases: [agentcore] } | |
| 309 | + sensors: | |
| 310 | + - { name: machine learning blog feed, url: "https://aws.amazon.com/blogs/machine-learning/feed/", type: RSS, connector: rss, tier: B } | |
| 311 | + - { name: news blog ai category feed, url: "https://aws.amazon.com/blogs/aws/category/artificial-intelligence/feed/", type: RSS, connector: rss, tier: B } | |
| 312 | + - { name: bedrock pricing, url: "https://aws.amazon.com/bedrock/pricing/", type: HTML, connector: http, tier: C } | |
| 313 | + - { name: bedrock user guide history, url: "https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-ug-doc-history.html", type: HTML, connector: http, tier: B } | |
| 314 | + - { name: bedrock model lifecycle, url: "https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html", type: HTML, connector: http, tier: B } | |
| 315 | + - { name: strands agents sdk releases, url: "https://github.com/strands-agents/harness-sdk/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: strands-agents/harness-sdk, kind: releases } } | |
| 316 | + - { name: bedrock agentcore sdk releases, url: "https://github.com/aws/bedrock-agentcore-sdk-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: aws/bedrock-agentcore-sdk-python, kind: releases } } | |
| 317 | + # ───────────────────────── Chinese labs ───────────────────────── | |
| 318 | + - id: qwen | |
| 319 | + name: Alibaba Qwen | |
| 320 | + domain: qwen.ai | |
| 321 | + homepage: https://qwenlm.github.io | |
| 322 | + categories: [ai, technology] | |
| 323 | + tier: A | |
| 324 | + weight: 1.3 | |
| 325 | + country: CN | |
| 326 | + aliases: [qwen, qwenlm, alibaba qwen, tongyi qianwen, alibaba cloud model studio] | |
| 327 | + products: | |
| 328 | + - { name: Qwen3, type: AI_model, aliases: [qwen 3, qwen3-max, qwen3-coder, qwen3-vl] } | |
| 329 | + - { name: Qwen-Image, type: AI_model } | |
| 330 | + - { name: Qwen Code, type: software, aliases: [qwen-code] } | |
| 331 | + - { name: Model Studio, type: API, aliases: [dashscope, bailian] } | |
| 332 | + discover: { rss: true } | |
| 333 | + notes: "qwen.ai is client-rendered; the blog lives on qwenlm.github.io. Model Studio pricing page is Alibaba Cloud help." | |
| 334 | + sensors: | |
| 335 | + - { name: blog feed, url: "https://qwenlm.github.io/blog/index.xml", type: RSS, connector: rss, tier: A } | |
| 336 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=Qwen&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 337 | + - { name: model studio pricing, url: "https://www.alibabacloud.com/help/en/model-studio/model-pricing", type: HTML, connector: http, tier: C } | |
| 338 | + - { name: qwen code releases, url: "https://github.com/QwenLM/qwen-code/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: QwenLM/qwen-code, kind: releases } } | |
| 339 | + - { name: qwen-agent releases, url: "https://github.com/QwenLM/Qwen-Agent/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: QwenLM/Qwen-Agent, kind: releases } } | |
| 340 | + - { name: qwen-image commits, url: "https://github.com/QwenLM/Qwen-Image/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: QwenLM/Qwen-Image, kind: commits, branch: main } } | |
| 341 | + - { name: qwen3-omni commits, url: "https://github.com/QwenLM/Qwen3-Omni/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: QwenLM/Qwen3-Omni, kind: commits, branch: main } } | |
| 342 | + - id: deepseek | |
| 343 | + name: DeepSeek | |
| 344 | + domain: deepseek.com | |
| 345 | + homepage: https://www.deepseek.com | |
| 346 | + categories: [ai, technology] | |
| 347 | + tier: A | |
| 348 | + weight: 1.3 | |
| 349 | + country: CN | |
| 350 | + aliases: [deepseek ai, deep seek] | |
| 351 | + products: | |
| 352 | + - { name: DeepSeek-V3, type: AI_model, aliases: [deepseek v3, deepseek-v3.1, deepseek-v3.2] } | |
| 353 | + - { name: DeepSeek-R1, type: AI_model, aliases: [deepseek r1] } | |
| 354 | + - { name: DeepSeek API, type: API } | |
| 355 | + discover: { rss: true, sitemap: true } | |
| 356 | + notes: "platform.deepseek.com and status.deepseek.com (Flashcat) are client-rendered; api-docs.deepseek.com is Docusaurus (server-rendered)." | |
| 357 | + sensors: | |
| 358 | + - { name: api updates, url: "https://api-docs.deepseek.com/updates/", type: HTML, connector: http, tier: A } | |
| 359 | + - { name: api pricing, url: "https://api-docs.deepseek.com/quick_start/pricing/", type: HTML, connector: http, tier: B } | |
| 360 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=deepseek-ai&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 361 | + - { name: deepseek-v3 commits, url: "https://github.com/deepseek-ai/DeepSeek-V3/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: deepseek-ai/DeepSeek-V3, kind: commits, branch: main } } | |
| 362 | + - { name: deepseek-v3 tags, url: "https://github.com/deepseek-ai/DeepSeek-V3/tags.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: deepseek-ai/DeepSeek-V3, kind: tags } } | |
| 363 | + - { name: deepseek-r1 commits, url: "https://github.com/deepseek-ai/DeepSeek-R1/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: deepseek-ai/DeepSeek-R1, kind: commits, branch: main } } | |
| 364 | + - { name: deepseek-v3.2-exp commits, url: "https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: deepseek-ai/DeepSeek-V3.2-Exp, kind: commits, branch: main } } | |
| 365 | + - id: moonshot-ai | |
| 366 | + name: Moonshot AI | |
| 367 | + domain: moonshot.ai | |
| 368 | + homepage: https://platform.kimi.ai | |
| 369 | + categories: [ai, technology] | |
| 370 | + tier: A | |
| 371 | + weight: 1.1 | |
| 372 | + country: CN | |
| 373 | + aliases: [moonshot, kimi, kimi ai, moonshot ai] | |
| 374 | + products: | |
| 375 | + - { name: Kimi K2, type: AI_model, aliases: [kimi-k2, kimi k2 thinking] } | |
| 376 | + - { name: Kimi CLI, type: software } | |
| 377 | + - { name: Kimi API, type: API, aliases: [moonshot api] } | |
| 378 | + discover: { rss: true } | |
| 379 | + notes: "platform.moonshot.ai redirects to platform.kimi.ai; status.moonshot.ai does not resolve." | |
| 380 | + sensors: | |
| 381 | + - { name: api pricing (chat), url: "https://platform.kimi.ai/docs/pricing/chat", type: HTML, connector: http, tier: B } | |
| 382 | + - { name: api rate limits, url: "https://platform.kimi.ai/docs/pricing/limits", type: HTML, connector: http, tier: C } | |
| 383 | + - { name: api overview, url: "https://platform.kimi.ai/docs/overview", type: HTML, connector: http, tier: B } | |
| 384 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=moonshotai&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 385 | + - { name: kimi-k2 commits, url: "https://github.com/MoonshotAI/Kimi-K2/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: MoonshotAI/Kimi-K2, kind: commits, branch: main } } | |
| 386 | + - { name: kimi cli releases, url: "https://github.com/MoonshotAI/kimi-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: MoonshotAI/kimi-cli, kind: releases } } | |
| 387 | + - id: zhipu-ai | |
| 388 | + name: Zhipu AI (Z.ai) | |
| 389 | + domain: z.ai | |
| 390 | + homepage: https://z.ai | |
| 391 | + categories: [ai, technology] | |
| 392 | + tier: A | |
| 393 | + weight: 1.1 | |
| 394 | + country: CN | |
| 395 | + aliases: [zhipu, zhipu ai, z.ai, bigmodel, chatglm, thudm] | |
| 396 | + products: | |
| 397 | + - { name: GLM, type: AI_model, aliases: [glm-4.5, glm-4.6, glm-5, chatglm] } | |
| 398 | + - { name: Z.ai API, type: API, aliases: [bigmodel api] } | |
| 399 | + discover: { rss: true } | |
| 400 | + notes: "open.bigmodel.cn / bigmodel.cn are client-rendered; docs.z.ai (Mintlify) is server-rendered." | |
| 401 | + sensors: | |
| 402 | + - { name: api pricing, url: "https://docs.z.ai/guides/overview/pricing", type: HTML, connector: http, tier: B } | |
| 403 | + - { name: release notes, url: "https://docs.z.ai/release-notes/new-released", type: HTML, connector: http, tier: A } | |
| 404 | + - { name: docs llms.txt, url: "https://docs.z.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 405 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=zai-org&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 406 | + - { name: glm-4.5 commits, url: "https://github.com/zai-org/GLM-4.5/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: zai-org/GLM-4.5, kind: commits, branch: main } } | |
| 407 | + - id: minimax | |
| 408 | + name: MiniMax | |
| 409 | + domain: minimax.io | |
| 410 | + homepage: https://www.minimax.io | |
| 411 | + categories: [ai, technology] | |
| 412 | + tier: B | |
| 413 | + weight: 1.1 | |
| 414 | + country: CN | |
| 415 | + aliases: [minimax ai, hailuo] | |
| 416 | + products: | |
| 417 | + - { name: MiniMax-M1, type: AI_model, aliases: [minimax m1] } | |
| 418 | + - { name: MiniMax-M2, type: AI_model, aliases: [minimax m2] } | |
| 419 | + - { name: Hailuo, type: product, aliases: [hailuo video] } | |
| 420 | + discover: { rss: true } | |
| 421 | + notes: "minimax.io/news is client-rendered." | |
| 422 | + sensors: | |
| 423 | + - { name: status, url: "https://status.minimax.io/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 424 | + - { name: models intro, url: "https://platform.minimax.io/docs/guides/models-intro", type: HTML, connector: http, tier: B } | |
| 425 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=MiniMaxAI&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 426 | + - { name: minimax-m2 commits, url: "https://github.com/MiniMax-AI/MiniMax-M2/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: MiniMax-AI/MiniMax-M2, kind: commits, branch: main } } | |
| 427 | + - { name: minimax-m1 commits, url: "https://github.com/MiniMax-AI/MiniMax-M1/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: MiniMax-AI/MiniMax-M1, kind: commits, branch: main } } | |
| 428 | + - id: baidu | |
| 429 | + name: Baidu | |
| 430 | + domain: baidu.com | |
| 431 | + homepage: https://www.baidu.com | |
| 432 | + categories: [ai, technology, internet] | |
| 433 | + tier: B | |
| 434 | + weight: 1.2 | |
| 435 | + country: CN | |
| 436 | + aliases: [baidu ai, ernie, wenxin, paddlepaddle] | |
| 437 | + products: | |
| 438 | + - { name: ERNIE, type: AI_model, aliases: [ernie 4.5, ernie bot, wenxin yiyan] } | |
| 439 | + - { name: PaddlePaddle, type: software, aliases: [paddle] } | |
| 440 | + - { name: Qianfan, type: API, aliases: [wenxin workshop] } | |
| 441 | + discover: { rss: true } | |
| 442 | + notes: "ir.baidu.com returns 403; Qianfan docs (cloud.baidu.com) are Chinese and mostly client-rendered." | |
| 443 | + sensors: | |
| 444 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=baidu&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 445 | + - { name: ernie blog, url: "https://ernie.baidu.com/blog", type: HTML, connector: http, tier: B } | |
| 446 | + - { name: paddle releases, url: "https://github.com/PaddlePaddle/Paddle/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: PaddlePaddle/Paddle, kind: releases } } | |
| 447 | + - { name: ernie commits, url: "https://github.com/PaddlePaddle/ERNIE/commits/develop.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: PaddlePaddle/ERNIE, kind: commits, branch: develop } } | |
| 448 | + - { name: paddleocr releases, url: "https://github.com/PaddlePaddle/PaddleOCR/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: PaddlePaddle/PaddleOCR, kind: releases } } | |
| 449 | + - id: tencent | |
| 450 | + extend: true | |
| 451 | + country: CN | |
| 452 | + categories: [ai] | |
| 453 | + aliases: [hunyuan, tencent hunyuan] | |
| 454 | + products: | |
| 455 | + - { name: Hunyuan, type: AI_model, aliases: [hunyuan-a13b, hunyuanvideo, hunyuan3d] } | |
| 456 | + sensors: | |
| 457 | + - { name: hugging face tencent models, url: "https://huggingface.co/api/models?author=tencent&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 458 | + - { name: hugging face hunyuan models, url: "https://huggingface.co/api/models?author=Tencent-Hunyuan&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 459 | + - { name: hunyuanvideo tags, url: "https://github.com/Tencent-Hunyuan/HunyuanVideo/tags.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: Tencent-Hunyuan/HunyuanVideo, kind: tags } } | |
| 460 | + - { name: hunyuan3d-2 commits, url: "https://github.com/Tencent-Hunyuan/Hunyuan3D-2/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: Tencent-Hunyuan/Hunyuan3D-2, kind: commits, branch: main } } | |
| 461 | + - { name: hunyuan-a13b commits, url: "https://github.com/Tencent-Hunyuan/Hunyuan-A13B/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: Tencent-Hunyuan/Hunyuan-A13B, kind: commits, branch: main } } | |
| 462 | + - id: bytedance-seed | |
| 463 | + name: ByteDance Seed | |
| 464 | + domain: seed.bytedance.com | |
| 465 | + homepage: https://seed.bytedance.com/en/ | |
| 466 | + categories: [ai, technology] | |
| 467 | + tier: B | |
| 468 | + weight: 1.1 | |
| 469 | + country: CN | |
| 470 | + aliases: [bytedance, bytedance seed, seed team, doubao, volcano engine] | |
| 471 | + products: | |
| 472 | + - { name: Seed, type: AI_model, aliases: [seed-oss, seed-coder, seedance, seedream] } | |
| 473 | + - { name: Doubao, type: AI_model } | |
| 474 | + - { name: verl, type: software } | |
| 475 | + discover: { rss: false } | |
| 476 | + notes: "seed.bytedance.com/en/blog is client-rendered; the research index is server-rendered." | |
| 477 | + sensors: | |
| 478 | + - { name: research index, url: "https://seed.bytedance.com/en/research", type: HTML, connector: http, tier: B } | |
| 479 | + - { name: hugging face seed models, url: "https://huggingface.co/api/models?author=ByteDance-Seed&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 480 | + - { name: hugging face bytedance models, url: "https://huggingface.co/api/models?author=ByteDance&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 481 | + - { name: deer-flow releases, url: "https://github.com/bytedance/deer-flow/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: bytedance/deer-flow, kind: releases } } | |
| 482 | + - { name: verl releases, url: "https://github.com/verl-project/verl/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: verl-project/verl, kind: releases } } | |
| 483 | + - { name: seed-coder commits, url: "https://github.com/ByteDance-Seed/Seed-Coder/commits/master.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ByteDance-Seed/Seed-Coder, kind: commits, branch: master } } | |
| 484 | + - id: shanghai-ai-lab | |
| 485 | + name: Shanghai AI Laboratory | |
| 486 | + domain: shlab.org.cn | |
| 487 | + homepage: https://www.shlab.org.cn | |
| 488 | + categories: [ai, research] | |
| 489 | + tier: B | |
| 490 | + country: CN | |
| 491 | + aliases: [shanghai ai lab, internlm, opengvlab, intern-s1] | |
| 492 | + products: | |
| 493 | + - { name: InternLM, type: AI_model, aliases: [intern-s1, internlm3] } | |
| 494 | + - { name: InternVL, type: AI_model, aliases: [internvl3] } | |
| 495 | + discover: { rss: false } | |
| 496 | + sensors: | |
| 497 | + - { name: hugging face internlm models, url: "https://huggingface.co/api/models?author=internlm&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 498 | + - { name: hugging face opengvlab models, url: "https://huggingface.co/api/models?author=OpenGVLab&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 499 | + - id: stepfun | |
| 500 | + name: StepFun | |
| 501 | + domain: stepfun.com | |
| 502 | + homepage: https://www.stepfun.com | |
| 503 | + categories: [ai, technology] | |
| 504 | + tier: C | |
| 505 | + country: CN | |
| 506 | + aliases: [stepfun ai, step-3] | |
| 507 | + products: | |
| 508 | + - { name: Step, type: AI_model, aliases: [step-3, step3-vl] } | |
| 509 | + discover: { rss: false } | |
| 510 | + sensors: | |
| 511 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=stepfun-ai&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 512 | + # ───────────────────────── Inference clouds & AI infrastructure ───────────────────────── | |
| 513 | + - id: together-ai | |
| 514 | + extend: true | |
| 515 | + country: US | |
| 516 | + notes: "status.together.ai is a Betterstack page (no summary JSON)." | |
| 517 | + sensors: | |
| 518 | + - { name: blog feed, url: "https://www.together.ai/blog/rss.xml", type: RSS, connector: rss, tier: B } | |
| 519 | + - { name: serverless models docs, url: "https://docs.together.ai/docs/serverless/models", type: HTML, connector: http, tier: B } | |
| 520 | + - { name: dedicated endpoint models docs, url: "https://docs.together.ai/docs/dedicated-endpoints/models", type: HTML, connector: http, tier: C } | |
| 521 | + - { name: docs llms.txt, url: "https://docs.together.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 522 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=togethercomputer&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 523 | + - { name: sdk python releases, url: "https://github.com/togethercomputer/together-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: togethercomputer/together-python, kind: releases } } | |
| 524 | + - { name: sdk typescript releases, url: "https://github.com/togethercomputer/together-typescript/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: togethercomputer/together-typescript, kind: releases } } | |
| 525 | + - id: groq | |
| 526 | + extend: true | |
| 527 | + country: US | |
| 528 | + products: | |
| 529 | + - { name: GroqCloud, type: service, aliases: [groq cloud, groq api] } | |
| 530 | + - { name: LPU, type: technology } | |
| 531 | + notes: "groq.com/pricing redirects to the home page." | |
| 532 | + sensors: | |
| 533 | + - { name: models docs, url: "https://console.groq.com/docs/models", type: HTML, connector: http, tier: A } | |
| 534 | + - { name: docs changelog, url: "https://console.groq.com/docs/changelog", type: HTML, connector: http, tier: A } | |
| 535 | + - { name: deprecations, url: "https://console.groq.com/docs/deprecations", type: HTML, connector: http, tier: B } | |
| 536 | + - { name: docs llms.txt, url: "https://console.groq.com/llms.txt", type: FILE, connector: http, tier: D } | |
| 537 | + - { name: blog index, url: "https://groq.com/blog", type: HTML, connector: http, tier: B } | |
| 538 | + - { name: sdk python releases, url: "https://github.com/groq/groq-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: groq/groq-python, kind: releases } } | |
| 539 | + - { name: sdk typescript releases, url: "https://github.com/groq/groq-typescript/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: groq/groq-typescript, kind: releases } } | |
| 540 | + - id: cerebras | |
| 541 | + extend: true | |
| 542 | + country: US | |
| 543 | + products: | |
| 544 | + - { name: Cerebras Inference, type: service, aliases: [cerebras cloud] } | |
| 545 | + - { name: WSE-3, type: product, aliases: [wafer scale engine] } | |
| 546 | + sensors: | |
| 547 | + - { name: status, url: "https://status.cerebras.ai/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 548 | + - { name: inference models overview, url: "https://inference-docs.cerebras.ai/models/overview", type: HTML, connector: http, tier: A } | |
| 549 | + - { name: inference changelog, url: "https://inference-docs.cerebras.ai/support/change-log", type: HTML, connector: http, tier: B } | |
| 550 | + - { name: pricing, url: "https://www.cerebras.ai/pricing", type: HTML, connector: http, tier: C } | |
| 551 | + - { name: docs llms.txt, url: "https://inference-docs.cerebras.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 552 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=cerebras&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 553 | + - id: sambanova | |
| 554 | + name: SambaNova Systems | |
| 555 | + domain: sambanova.ai | |
| 556 | + categories: [ai, semiconductors] | |
| 557 | + tier: B | |
| 558 | + country: US | |
| 559 | + aliases: [sambanova, samba nova, sambacloud] | |
| 560 | + products: | |
| 561 | + - { name: SambaNova Cloud, type: service, aliases: [sambacloud] } | |
| 562 | + - { name: SN40L, type: product } | |
| 563 | + discover: { rss: true } | |
| 564 | + sensors: | |
| 565 | + - { name: status, url: "https://status.sambanova.ai/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 566 | + - { name: blog feed, url: "https://sambanova.ai/blog/rss.xml", type: RSS, connector: rss, tier: B } | |
| 567 | + - { name: cloud models docs, url: "https://docs.sambanova.ai/docs/en/models/sambacloud-models", type: HTML, connector: http, tier: B } | |
| 568 | + - { name: release notes, url: "https://docs.sambanova.ai/docs/en/release-notes/overview", type: HTML, connector: http, tier: B } | |
| 569 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=sambanovasystems&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 570 | + - { name: ai starter kit releases, url: "https://github.com/sambanova/ai-starter-kit/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: sambanova/ai-starter-kit, kind: releases } } | |
| 571 | + - id: fireworks-ai | |
| 572 | + extend: true | |
| 573 | + country: US | |
| 574 | + sensors: | |
| 575 | + - { name: status, url: "https://status.fireworks.ai/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 576 | + - { name: pricing, url: "https://fireworks.ai/pricing", type: HTML, connector: http, tier: C } | |
| 577 | + - { name: models library, url: "https://fireworks.ai/models", type: HTML, connector: http, tier: B } | |
| 578 | + - { name: docs llms.txt, url: "https://docs.fireworks.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 579 | + - { name: blog index, url: "https://fireworks.ai/blog", type: HTML, connector: http, tier: B } | |
| 580 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=fireworks-ai&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 581 | + - id: perplexity | |
| 582 | + extend: true | |
| 583 | + country: US | |
| 584 | + products: | |
| 585 | + - { name: Sonar, type: AI_model, aliases: [sonar pro, sonar api] } | |
| 586 | + - { name: Comet, type: product, aliases: [comet browser] } | |
| 587 | + notes: "perplexity.ai/hub is behind a Cloudflare challenge." | |
| 588 | + sensors: | |
| 589 | + - { name: status, url: "https://status.perplexity.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 590 | + - { name: api pricing, url: "https://docs.perplexity.ai/docs/getting-started/pricing", type: HTML, connector: http, tier: B } | |
| 591 | + - { name: docs llms.txt, url: "https://docs.perplexity.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 592 | + - { name: sdk python releases, url: "https://github.com/perplexityai/perplexity-py/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: perplexityai/perplexity-py, kind: releases } } | |
| 593 | + - id: replicate | |
| 594 | + extend: true | |
| 595 | + country: US | |
| 596 | + notes: "status.replicate.com redirects to cloudflarestatus.com." | |
| 597 | + sensors: | |
| 598 | + - { name: blog feed, url: "https://replicate.com/blog/rss", type: RSS, connector: rss, tier: B } | |
| 599 | + - { name: pricing, url: "https://replicate.com/pricing", type: HTML, connector: http, tier: C } | |
| 600 | + - { name: explore models, url: "https://replicate.com/explore", type: HTML, connector: http, tier: B } | |
| 601 | + - { name: sdk python releases, url: "https://github.com/replicate/replicate-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: replicate/replicate-python, kind: releases } } | |
| 602 | + - { name: sdk javascript releases, url: "https://github.com/replicate/replicate-javascript/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: replicate/replicate-javascript, kind: releases } } | |
| 603 | + - { name: cog releases, url: "https://github.com/replicate/cog/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: replicate/cog, kind: releases } } | |
| 604 | + - id: modal | |
| 605 | + name: Modal | |
| 606 | + domain: modal.com | |
| 607 | + categories: [ai, cloud, developer] | |
| 608 | + tier: B | |
| 609 | + country: US | |
| 610 | + aliases: [modal labs] | |
| 611 | + products: | |
| 612 | + - { name: Modal client, type: software, aliases: [modal python] } | |
| 613 | + discover: { rss: true } | |
| 614 | + notes: "modal.com/blog and the docs changelog are client-rendered; status.modal.com is Betterstack." | |
| 615 | + sensors: | |
| 616 | + - { name: pricing, url: "https://modal.com/pricing", type: HTML, connector: http, tier: C } | |
| 617 | + - { name: llms.txt, url: "https://modal.com/llms.txt", type: FILE, connector: http, tier: D } | |
| 618 | + - { name: client releases, url: "https://github.com/modal-labs/modal-client/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: modal-labs/modal-client, kind: releases } } | |
| 619 | + - id: baseten | |
| 620 | + name: Baseten | |
| 621 | + domain: baseten.co | |
| 622 | + homepage: https://www.baseten.co | |
| 623 | + categories: [ai, cloud, developer] | |
| 624 | + tier: B | |
| 625 | + country: US | |
| 626 | + products: | |
| 627 | + - { name: Truss, type: software } | |
| 628 | + discover: { rss: true } | |
| 629 | + sensors: | |
| 630 | + - { name: status, url: "https://status.baseten.co/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 631 | + - { name: pricing, url: "https://www.baseten.co/pricing/", type: HTML, connector: http, tier: C } | |
| 632 | + - { name: model library, url: "https://www.baseten.co/library/", type: HTML, connector: http, tier: B } | |
| 633 | + - { name: docs llms.txt, url: "https://docs.baseten.co/llms.txt", type: FILE, connector: http, tier: D } | |
| 634 | + - { name: truss releases, url: "https://github.com/basetenlabs/truss/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: basetenlabs/truss, kind: releases } } | |
| 635 | + - id: runpod | |
| 636 | + name: RunPod | |
| 637 | + domain: runpod.io | |
| 638 | + homepage: https://www.runpod.io | |
| 639 | + categories: [ai, cloud] | |
| 640 | + tier: B | |
| 641 | + country: US | |
| 642 | + aliases: [run pod] | |
| 643 | + discover: { rss: true } | |
| 644 | + notes: "uptime.runpod.io is a Betterstack page; status.runpod.io does not resolve." | |
| 645 | + sensors: | |
| 646 | + - { name: blog feed, url: "https://www.runpod.io/blog/rss.xml", type: RSS, connector: rss, tier: B } | |
| 647 | + - { name: release notes, url: "https://docs.runpod.io/release-notes", type: HTML, connector: http, tier: B } | |
| 648 | + - { name: pricing, url: "https://www.runpod.io/pricing", type: HTML, connector: http, tier: C } | |
| 649 | + - { name: docs llms.txt, url: "https://docs.runpod.io/llms.txt", type: FILE, connector: http, tier: D } | |
| 650 | + - { name: sdk python releases, url: "https://github.com/runpod/runpod-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: runpod/runpod-python, kind: releases } } | |
| 651 | + - { name: runpodctl releases, url: "https://github.com/runpod/runpodctl/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: runpod/runpodctl, kind: releases } } | |
| 652 | + - id: lambda | |
| 653 | + name: Lambda | |
| 654 | + domain: lambda.ai | |
| 655 | + categories: [ai, cloud] | |
| 656 | + tier: B | |
| 657 | + country: US | |
| 658 | + aliases: [lambda labs, lambdalabs, lambda cloud] | |
| 659 | + products: | |
| 660 | + - { name: Lambda Cloud, type: service, aliases: [gpu cloud] } | |
| 661 | + discover: { rss: true } | |
| 662 | + sensors: | |
| 663 | + - { name: status, url: "https://status.lambda.ai/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 664 | + - { name: blog feed, url: "https://lambda.ai/blog/rss.xml", type: RSS, connector: rss, tier: B } | |
| 665 | + - { name: pricing, url: "https://lambda.ai/pricing", type: HTML, connector: http, tier: C } | |
| 666 | + - { name: instances, url: "https://lambda.ai/instances", type: HTML, connector: http, tier: C } | |
| 667 | + - { name: on-demand docs, url: "https://docs.lambda.ai/public-cloud/on-demand/", type: HTML, connector: http, tier: C } | |
| 668 | + - id: coreweave | |
| 669 | + name: CoreWeave | |
| 670 | + domain: coreweave.com | |
| 671 | + homepage: https://www.coreweave.com | |
| 672 | + categories: [ai, cloud, filings] | |
| 673 | + tier: B | |
| 674 | + weight: 1.1 | |
| 675 | + country: US | |
| 676 | + aliases: [crwv, core weave] | |
| 677 | + discover: { rss: true } | |
| 678 | + notes: "SIC: Services-Computer Programming, Data Processing · NASDAQ: CRWV · CIK 1769628" | |
| 679 | + sensors: | |
| 680 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001769628.json", type: REST_API, connector: edgar, tier: B } | |
| 681 | + - { name: blog feed, url: "https://www.coreweave.com/blog/rss.xml", type: RSS, connector: rss, tier: B } | |
| 682 | + - { name: pricing, url: "https://www.coreweave.com/pricing", type: HTML, connector: http, tier: C } | |
| 683 | + - { name: docs changelog, url: "https://docs.coreweave.com/changelog", type: HTML, connector: http, tier: B } | |
| 684 | + - { name: docs llms.txt, url: "https://docs.coreweave.com/llms.txt", type: FILE, connector: http, tier: D } | |
| 685 | + - id: openrouter | |
| 686 | + name: OpenRouter | |
| 687 | + domain: openrouter.ai | |
| 688 | + categories: [ai, developer] | |
| 689 | + tier: B | |
| 690 | + weight: 1.1 | |
| 691 | + country: US | |
| 692 | + aliases: [open router] | |
| 693 | + products: | |
| 694 | + - { name: OpenRouter API, type: API } | |
| 695 | + discover: { rss: false } | |
| 696 | + notes: "The public /api/v1/models catalogue (≈440 models with prices) is a cross-provider model-release and price-change signal. status.openrouter.ai returns 403; the blog is client-rendered." | |
| 697 | + sensors: | |
| 698 | + - { name: models api, url: "https://openrouter.ai/api/v1/models", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: data, keyField: id, titleField: name, compareFields: [pricing.prompt, pricing.completion, context_length], maxItems: 1000 } } | |
| 699 | + - { name: docs llms.txt, url: "https://openrouter.ai/docs/llms.txt", type: FILE, connector: http, tier: D } | |
| 700 | + - id: litellm | |
| 701 | + name: LiteLLM | |
| 702 | + domain: litellm.ai | |
| 703 | + homepage: https://docs.litellm.ai | |
| 704 | + categories: [ai, open-source, developer] | |
| 705 | + tier: C | |
| 706 | + country: US | |
| 707 | + aliases: [berriai, lite llm] | |
| 708 | + llm: false | |
| 709 | + discover: { rss: false } | |
| 710 | + notes: "Releases feed publishes several builds a day (heuristics only). model_prices_and_context_window.json is the de-facto open price/context registry for ~2 000 models." | |
| 711 | + sensors: | |
| 712 | + - { name: model prices and context window, url: "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json", type: JSON, connector: http, tier: C } | |
| 713 | + - { name: release notes feed, url: "https://docs.litellm.ai/blog/rss.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 30 } } | |
| 714 | + - { name: releases, url: "https://github.com/BerriAI/litellm/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: BerriAI/litellm, kind: releases } } | |
| 715 | + # ───────────────────────── Coding assistants ───────────────────────── | |
| 716 | + - id: cursor | |
| 717 | + name: Cursor | |
| 718 | + domain: cursor.com | |
| 719 | + categories: [ai, developer] | |
| 720 | + tier: A | |
| 721 | + weight: 1.1 | |
| 722 | + country: US | |
| 723 | + aliases: [anysphere, cursor ai, cursor editor] | |
| 724 | + products: | |
| 725 | + - { name: Cursor, type: software, aliases: [cursor ide] } | |
| 726 | + discover: { rss: true } | |
| 727 | + sensors: | |
| 728 | + - { name: status, url: "https://status.cursor.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 729 | + - { name: changelog feed, url: "https://cursor.com/changelog/rss.xml", type: RSS, connector: rss, tier: A } | |
| 730 | + - { name: changelog, url: "https://cursor.com/changelog", type: HTML, connector: http, tier: B } | |
| 731 | + - { name: pricing, url: "https://cursor.com/pricing", type: HTML, connector: http, tier: C } | |
| 732 | + - { name: blog index, url: "https://cursor.com/blog", type: HTML, connector: http, tier: B } | |
| 733 | + - { name: docs llms.txt, url: "https://cursor.com/docs/llms.txt", type: FILE, connector: http, tier: D } | |
| 734 | + - { name: forum announcements feed, url: "https://forum.cursor.com/c/announcements/11.rss", type: RSS, connector: rss, tier: B } | |
| 735 | + - id: replit | |
| 736 | + name: Replit | |
| 737 | + domain: replit.com | |
| 738 | + categories: [ai, developer] | |
| 739 | + tier: B | |
| 740 | + country: US | |
| 741 | + aliases: [repl.it, replit agent] | |
| 742 | + products: | |
| 743 | + - { name: Replit Agent, type: product } | |
| 744 | + discover: { rss: true } | |
| 745 | + notes: "status.replit.com is behind a Cloudflare challenge." | |
| 746 | + sensors: | |
| 747 | + - { name: blog feed, url: "https://replit.com/blog/feed.xml", type: RSS, connector: rss, tier: B } | |
| 748 | + - { name: pricing, url: "https://replit.com/pricing", type: HTML, connector: http, tier: C } | |
| 749 | + - { name: docs updates, url: "https://docs.replit.com/updates", type: HTML, connector: http, tier: B } | |
| 750 | + - { name: docs llms.txt, url: "https://docs.replit.com/llms.txt", type: FILE, connector: http, tier: D } | |
| 751 | + - id: windsurf | |
| 752 | + name: Windsurf | |
| 753 | + domain: windsurf.com | |
| 754 | + categories: [ai, developer] | |
| 755 | + tier: B | |
| 756 | + country: US | |
| 757 | + aliases: [codeium, cognition, windsurf editor, devin] | |
| 758 | + products: | |
| 759 | + - { name: Windsurf Editor, type: software } | |
| 760 | + - { name: Devin, type: product, aliases: [devin ai] } | |
| 761 | + discover: { rss: true } | |
| 762 | + notes: "Windsurf now belongs to Cognition: the changelog redirects to docs.devin.ai; windsurf.com/pricing redirects to devin.ai and rate-limits (429)." | |
| 763 | + sensors: | |
| 764 | + - { name: status, url: "https://status.windsurf.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 765 | + - { name: changelog, url: "https://windsurf.com/changelog", type: HTML, connector: http, tier: B } | |
| 766 | + - { name: docs llms.txt, url: "https://docs.windsurf.com/llms.txt", type: FILE, connector: http, tier: D } | |
| 767 | + - id: github | |
| 768 | + extend: true | |
| 769 | + aliases: [github copilot, copilot coding agent] | |
| 770 | + sensors: | |
| 771 | + - { name: copilot blog feed, url: "https://github.blog/ai-and-ml/github-copilot/feed/", type: RSS, connector: rss, tier: B } | |
| 772 | + - { name: copilot supported models, url: "https://docs.github.com/en/copilot/reference/ai-models/supported-models", type: HTML, connector: http, tier: B } | |
| 773 | + - { name: copilot plans, url: "https://docs.github.com/en/copilot/get-started/plans", type: HTML, connector: http, tier: C } | |
| 774 | + - { name: copilot sdk releases, url: "https://github.com/github/copilot-sdk/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: github/copilot-sdk, kind: releases } } | |
| 775 | + - { name: copilot cli releases, url: "https://github.com/github/copilot-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: github/copilot-cli, kind: releases } } | |
| 776 | + - { name: github mcp server releases, url: "https://github.com/github/github-mcp-server/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: github/github-mcp-server, kind: releases } } | |
| 777 | + # ───────────────────────── Data & ML platforms ───────────────────────── | |
| 778 | + - id: scale-ai | |
| 779 | + name: Scale AI | |
| 780 | + domain: scale.com | |
| 781 | + categories: [ai, technology] | |
| 782 | + tier: B | |
| 783 | + weight: 1.1 | |
| 784 | + country: US | |
| 785 | + aliases: [scale, scale ai, seal leaderboards] | |
| 786 | + products: | |
| 787 | + - { name: SEAL Leaderboards, type: index, aliases: [seal] } | |
| 788 | + discover: { rss: false } | |
| 789 | + notes: "scale.com/blog is client-rendered; research moved to labs.scale.com." | |
| 790 | + sensors: | |
| 791 | + - { name: status, url: "https://status.scale.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 792 | + - { name: research papers, url: "https://labs.scale.com/papers", type: HTML, connector: http, tier: B } | |
| 793 | + - { name: leaderboard, url: "https://labs.scale.com/leaderboard", type: HTML, connector: http, tier: B } | |
| 794 | + - id: databricks | |
| 795 | + extend: true | |
| 796 | + country: US | |
| 797 | + products: | |
| 798 | + - { name: MLflow, type: software } | |
| 799 | + sensors: | |
| 800 | + - { name: platform release notes, url: "https://docs.databricks.com/aws/en/release-notes/product/", type: HTML, connector: http, tier: B } | |
| 801 | + - { name: foundation model overview, url: "https://docs.databricks.com/aws/en/machine-learning/model-serving/foundation-model-overview", type: HTML, connector: http, tier: C } | |
| 802 | + - { name: mlflow releases, url: "https://github.com/mlflow/mlflow/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: mlflow/mlflow, kind: releases } } | |
| 803 | + - id: snowflake | |
| 804 | + extend: true | |
| 805 | + country: US | |
| 806 | + products: | |
| 807 | + - { name: Arctic, type: AI_model, aliases: [snowflake arctic] } | |
| 808 | + sensors: | |
| 809 | + - { name: cortex ai sql docs, url: "https://docs.snowflake.com/en/user-guide/snowflake-cortex/aisql", type: HTML, connector: http, tier: C } | |
| 810 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=Snowflake&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 811 | + - { name: medium engineering feed, url: "https://medium.com/feed/snowflake", type: RSS, connector: rss, tier: C } | |
| 812 | + - id: wandb | |
| 813 | + name: Weights & Biases | |
| 814 | + domain: wandb.ai | |
| 815 | + categories: [ai, developer] | |
| 816 | + tier: B | |
| 817 | + country: US | |
| 818 | + aliases: [weights and biases, weights & biases, w&b, wandb] | |
| 819 | + products: | |
| 820 | + - { name: Weave, type: software, aliases: [wandb weave] } | |
| 821 | + discover: { rss: true } | |
| 822 | + sensors: | |
| 823 | + - { name: status, url: "https://status.wandb.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 824 | + - { name: sdk releases, url: "https://github.com/wandb/wandb/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: wandb/wandb, kind: releases } } | |
| 825 | + - { name: weave releases, url: "https://github.com/wandb/weave/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: wandb/weave, kind: releases } } | |
| 826 | + - { name: pricing, url: "https://wandb.ai/site/pricing/", type: HTML, connector: http, tier: C } | |
| 827 | + - { name: docs release notes, url: "https://docs.wandb.ai/release-notes", type: HTML, connector: http, tier: B } | |
| 828 | + - { name: docs llms.txt, url: "https://docs.wandb.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 829 | + # ───────────────────────── Generative media ───────────────────────── | |
| 830 | + - id: stability-ai | |
| 831 | + extend: true | |
| 832 | + country: GB | |
| 833 | + notes: "platform.stability.ai is client-rendered; stability.ai/news redirects to /news-updates (Squarespace, no RSS)." | |
| 834 | + sensors: | |
| 835 | + - { name: news updates, url: "https://stability.ai/news-updates", type: HTML, connector: http, tier: B } | |
| 836 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=stabilityai&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 837 | + - { name: generative-models tags, url: "https://github.com/Stability-AI/generative-models/tags.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: Stability-AI/generative-models, kind: tags } } | |
| 838 | + - id: black-forest-labs | |
| 839 | + name: Black Forest Labs | |
| 840 | + domain: bfl.ai | |
| 841 | + categories: [ai, technology] | |
| 842 | + tier: B | |
| 843 | + weight: 1.1 | |
| 844 | + country: DE | |
| 845 | + aliases: [bfl, black forest labs, flux] | |
| 846 | + products: | |
| 847 | + - { name: FLUX, type: AI_model, aliases: [flux.1, flux.2, flux kontext] } | |
| 848 | + - { name: BFL API, type: API } | |
| 849 | + discover: { rss: true } | |
| 850 | + sensors: | |
| 851 | + - { name: status, url: "https://status.bfl.ml/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 852 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=black-forest-labs&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 853 | + - { name: pricing, url: "https://bfl.ai/pricing", type: HTML, connector: http, tier: C } | |
| 854 | + - { name: blog index, url: "https://bfl.ai/blog", type: HTML, connector: http, tier: B } | |
| 855 | + - { name: docs llms.txt, url: "https://docs.bfl.ai/llms.txt", type: FILE, connector: http, tier: D } | |
| 856 | + - { name: flux commits, url: "https://github.com/black-forest-labs/flux/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: black-forest-labs/flux, kind: commits, branch: main } } | |
| 857 | + - id: elevenlabs | |
| 858 | + extend: true | |
| 859 | + country: US | |
| 860 | + sensors: | |
| 861 | + - { name: docs changelog, url: "https://elevenlabs.io/docs/changelog", type: HTML, connector: http, tier: A } | |
| 862 | + - { name: pricing, url: "https://elevenlabs.io/pricing", type: HTML, connector: http, tier: C } | |
| 863 | + - { name: api pricing, url: "https://elevenlabs.io/pricing/api", type: HTML, connector: http, tier: C } | |
| 864 | + - { name: docs llms.txt, url: "https://elevenlabs.io/docs/llms.txt", type: FILE, connector: http, tier: D } | |
| 865 | + - { name: sdk python releases, url: "https://github.com/elevenlabs/elevenlabs-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: elevenlabs/elevenlabs-python, kind: releases } } | |
| 866 | + - { name: sdk js releases, url: "https://github.com/elevenlabs/elevenlabs-js/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: elevenlabs/elevenlabs-js, kind: releases } } | |
| 867 | + - id: midjourney | |
| 868 | + name: Midjourney | |
| 869 | + domain: midjourney.com | |
| 870 | + homepage: https://www.midjourney.com | |
| 871 | + categories: [ai, technology] | |
| 872 | + tier: B | |
| 873 | + weight: 1.1 | |
| 874 | + country: US | |
| 875 | + aliases: [mid journey, mj] | |
| 876 | + discover: { rss: false } | |
| 877 | + notes: "docs.midjourney.com (Zendesk) is behind a Cloudflare challenge; midjourney.com/updates is client-rendered; the Ghost updates site has RSS." | |
| 878 | + sensors: | |
| 879 | + - { name: updates feed, url: "https://updates.midjourney.com/rss/", type: RSS, connector: rss, tier: A } | |
| 880 | + - id: runway | |
| 881 | + name: Runway | |
| 882 | + domain: runway.com | |
| 883 | + categories: [ai, technology, entertainment] | |
| 884 | + tier: B | |
| 885 | + weight: 1.1 | |
| 886 | + country: US | |
| 887 | + aliases: [runwayml, runway ml, runway ai] | |
| 888 | + products: | |
| 889 | + - { name: Gen-4, type: AI_model, aliases: [gen-4 turbo, gen-3 alpha] } | |
| 890 | + - { name: Runway API, type: API } | |
| 891 | + discover: { rss: true } | |
| 892 | + notes: "runwayml.com redirects to runway.com; help.runwayml.com (Zendesk) is behind a Cloudflare challenge; /news is client-rendered." | |
| 893 | + sensors: | |
| 894 | + - { name: status, url: "https://status.runwayml.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 895 | + - { name: research, url: "https://runway.com/research", type: HTML, connector: http, tier: B } | |
| 896 | + - { name: pricing, url: "https://runway.com/pricing", type: HTML, connector: http, tier: C } | |
| 897 | + - { name: api docs llms.txt, url: "https://docs.dev.runwayml.com/llms.txt", type: FILE, connector: http, tier: D } | |
| 898 | + - { name: sdk python releases, url: "https://github.com/runwayml/sdk-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: runwayml/sdk-python, kind: releases } } | |
| 899 | + - { name: sdk node releases, url: "https://github.com/runwayml/sdk-node/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: runwayml/sdk-node, kind: releases } } | |
| 900 | + # ───────────────────────── Frameworks, runtimes, protocols ───────────────────────── | |
| 901 | + - id: langchain | |
| 902 | + extend: true | |
| 903 | + country: US | |
| 904 | + sensors: | |
| 905 | + - { name: langsmith changelog, url: "https://docs.langchain.com/langsmith/changelog", type: HTML, connector: http, tier: B } | |
| 906 | + - { name: langgraph releases, url: "https://github.com/langchain-ai/langgraph/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: langchain-ai/langgraph, kind: releases } } | |
| 907 | + - { name: langchainjs releases, url: "https://github.com/langchain-ai/langchainjs/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: langchain-ai/langchainjs, kind: releases } } | |
| 908 | + - { name: deepagents releases, url: "https://github.com/langchain-ai/deepagents/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: langchain-ai/deepagents, kind: releases } } | |
| 909 | + - id: llamaindex | |
| 910 | + name: LlamaIndex | |
| 911 | + domain: llamaindex.ai | |
| 912 | + homepage: https://www.llamaindex.ai | |
| 913 | + categories: [ai, open-source, developer] | |
| 914 | + tier: B | |
| 915 | + country: US | |
| 916 | + aliases: [llama index, llamacloud, llamaparse, run-llama] | |
| 917 | + products: | |
| 918 | + - { name: LlamaCloud, type: service, aliases: [llamaparse] } | |
| 919 | + discover: { rss: true, status: true } | |
| 920 | + sensors: | |
| 921 | + - { name: status, url: "https://llamaindex.statuspage.io/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 922 | + - { name: releases, url: "https://github.com/run-llama/llama_index/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: run-llama/llama_index, kind: releases } } | |
| 923 | + - { name: llama cloud services releases, url: "https://github.com/run-llama/llama_cloud_services/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: run-llama/llama_cloud_services, kind: releases } } | |
| 924 | + - { name: blog index, url: "https://www.llamaindex.ai/blog", type: HTML, connector: http, tier: B } | |
| 925 | + - { name: pricing, url: "https://www.llamaindex.ai/pricing", type: HTML, connector: http, tier: C } | |
| 926 | + - id: ollama | |
| 927 | + extend: true | |
| 928 | + country: US | |
| 929 | + sensors: | |
| 930 | + - { name: model library, url: "https://ollama.com/library", type: HTML, connector: http, tier: A } | |
| 931 | + - { name: sdk python releases, url: "https://github.com/ollama/ollama-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ollama/ollama-python, kind: releases } } | |
| 932 | + - { name: sdk js releases, url: "https://github.com/ollama/ollama-js/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ollama/ollama-js, kind: releases } } | |
| 933 | + - id: vllm | |
| 934 | + extend: true | |
| 935 | + country: US | |
| 936 | + sensors: | |
| 937 | + - { name: supported models docs, url: "https://docs.vllm.ai/en/latest/models/supported_models/", type: HTML, connector: http, tier: B } | |
| 938 | + - { name: llm-compressor releases, url: "https://github.com/vllm-project/llm-compressor/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: vllm-project/llm-compressor, kind: releases } } | |
| 939 | + - id: llama-cpp | |
| 940 | + name: llama.cpp (ggml) | |
| 941 | + domain: ggml.ai | |
| 942 | + homepage: https://github.com/ggml-org/llama.cpp | |
| 943 | + categories: [ai, open-source, developer] | |
| 944 | + tier: B | |
| 945 | + weight: 1.1 | |
| 946 | + country: BG | |
| 947 | + aliases: [llama.cpp, llamacpp, ggml, gguf, whisper.cpp, ggerganov] | |
| 948 | + products: | |
| 949 | + - { name: GGUF, type: technology } | |
| 950 | + - { name: whisper.cpp, type: software } | |
| 951 | + llm: false | |
| 952 | + discover: { rss: false } | |
| 953 | + notes: "llama.cpp tags a build (b####) several times a day — heuristics only." | |
| 954 | + sensors: | |
| 955 | + - { name: llama.cpp releases, url: "https://github.com/ggml-org/llama.cpp/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: ggml-org/llama.cpp, kind: releases } } | |
| 956 | + - { name: whisper.cpp releases, url: "https://github.com/ggml-org/whisper.cpp/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ggml-org/whisper.cpp, kind: releases } } | |
| 957 | + - { name: ggml tags, url: "https://github.com/ggml-org/ggml/tags.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ggml-org/ggml, kind: tags } } | |
| 958 | + - id: pytorch | |
| 959 | + extend: true | |
| 960 | + country: US | |
| 961 | + sensors: | |
| 962 | + - { name: executorch releases, url: "https://github.com/pytorch/executorch/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: pytorch/executorch, kind: releases } } | |
| 963 | + - { name: torchao releases, url: "https://github.com/pytorch/ao/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: pytorch/ao, kind: releases } } | |
| 964 | + - { name: torchvision releases, url: "https://github.com/pytorch/vision/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: pytorch/vision, kind: releases } } | |
| 965 | + - { name: torchtune releases, url: "https://github.com/meta-pytorch/torchtune/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: meta-pytorch/torchtune, kind: releases } } | |
| 966 | + - id: tensorflow | |
| 967 | + extend: true | |
| 968 | + country: US | |
| 969 | + sensors: | |
| 970 | + - { name: keras releases, url: "https://github.com/keras-team/keras/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: keras-team/keras, kind: releases } } | |
| 971 | + - id: jax | |
| 972 | + extend: true | |
| 973 | + country: US | |
| 974 | + sensors: | |
| 975 | + - { name: flax releases, url: "https://github.com/google/flax/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: google/flax, kind: releases } } | |
| 976 | + - id: model-context-protocol | |
| 977 | + name: Model Context Protocol | |
| 978 | + domain: modelcontextprotocol.io | |
| 979 | + categories: [ai, standards, open-source, developer] | |
| 980 | + tier: B | |
| 981 | + weight: 1.1 | |
| 982 | + country: US | |
| 983 | + aliases: [mcp, model context protocol, mcp specification] | |
| 984 | + products: | |
| 985 | + - { name: MCP specification, type: standard } | |
| 986 | + - { name: MCP TypeScript SDK, type: software } | |
| 987 | + - { name: MCP Python SDK, type: software } | |
| 988 | + discover: { rss: true } | |
| 989 | + sensors: | |
| 990 | + - { name: blog feed, url: "https://blog.modelcontextprotocol.io/index.xml", type: RSS, connector: rss, tier: B } | |
| 991 | + - { name: specification changelog, url: "https://modelcontextprotocol.io/specification/latest/changelog", type: HTML, connector: http, tier: B } | |
| 992 | + - { name: llms.txt, url: "https://modelcontextprotocol.io/llms.txt", type: FILE, connector: http, tier: D } | |
| 993 | + - { name: specification releases, url: "https://github.com/modelcontextprotocol/modelcontextprotocol/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: modelcontextprotocol/modelcontextprotocol, kind: releases } } | |
| 994 | + - { name: typescript sdk releases, url: "https://github.com/modelcontextprotocol/typescript-sdk/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: modelcontextprotocol/typescript-sdk, kind: releases } } | |
| 995 | + - { name: python sdk releases, url: "https://github.com/modelcontextprotocol/python-sdk/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: modelcontextprotocol/python-sdk, kind: releases } } | |
| 996 | + - { name: reference servers releases, url: "https://github.com/modelcontextprotocol/servers/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: modelcontextprotocol/servers, kind: releases } } | |
| 997 | + # ───────────────────────── Evaluators, trackers, other labs ───────────────────────── | |
| 998 | + - id: lmarena | |
| 999 | + name: LMArena | |
| 1000 | + domain: arena.ai | |
| 1001 | + categories: [ai, research, statistics] | |
| 1002 | + tier: B | |
| 1003 | + weight: 1.1 | |
| 1004 | + country: US | |
| 1005 | + first_party: false | |
| 1006 | + aliases: [lmarena, lm arena, chatbot arena, lmsys, arena.ai] | |
| 1007 | + products: | |
| 1008 | + - { name: Arena Leaderboard, type: index, aliases: [chatbot arena leaderboard, text arena] } | |
| 1009 | + discover: { rss: true } | |
| 1010 | + notes: "lmarena.ai redirects to arena.ai; the leaderboard and blog are client-hydrated (validator: thin, even with selector main) and there is no RSS — discovery only until arena.ai exposes a feed or JSON." | |
| 1011 | + - id: artificial-analysis | |
| 1012 | + name: Artificial Analysis | |
| 1013 | + domain: artificialanalysis.ai | |
| 1014 | + categories: [ai, research, statistics] | |
| 1015 | + tier: B | |
| 1016 | + weight: 1.1 | |
| 1017 | + country: US | |
| 1018 | + first_party: false | |
| 1019 | + aliases: [artificial analysis, aa intelligence index] | |
| 1020 | + products: | |
| 1021 | + - { name: Intelligence Index, type: index, aliases: [artificial analysis intelligence index] } | |
| 1022 | + discover: { rss: false } | |
| 1023 | + notes: "The data API (/api/v2) requires a key; the leaderboard page is server-rendered." | |
| 1024 | + sensors: | |
| 1025 | + - { name: models leaderboard, url: "https://artificialanalysis.ai/leaderboards/models", type: HTML, connector: http, tier: B } | |
| 1026 | + - { name: models index, url: "https://artificialanalysis.ai/models", type: HTML, connector: http, tier: B } | |
| 1027 | + - id: epoch-ai | |
| 1028 | + extend: true | |
| 1029 | + country: US | |
| 1030 | + first_party: false | |
| 1031 | + products: | |
| 1032 | + - { name: Notable AI Models, type: dataset, aliases: [epoch ai models database] } | |
| 1033 | + - { name: AI Supercomputers, type: dataset, aliases: [gpu clusters dataset] } | |
| 1034 | + notes: "Open CSV datasets (CC-BY): notable_ai_models (newest first), large_scale_ai_models, ml_hardware, gpu_clusters (ai_supercomputers.csv redirects there). No site-wide RSS (feed.xml 404)." | |
| 1035 | + sensors: | |
| 1036 | + - { name: notable ai models dataset, url: "https://epoch.ai/data/notable_ai_models.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: Model, titleColumn: Model, dateColumn: "Publication date", compareColumns: [Organization, "Publication date", Parameters, "Training compute (FLOP)", Link], maxRows: 2000 } } | |
| 1037 | + - { name: large-scale ai models dataset, url: "https://epoch.ai/data/large_scale_ai_models.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: Model, titleColumn: Model, dateColumn: "Publication date", compareColumns: [Organization, "Publication date", Parameters, "Training compute (FLOP)", "Model accessibility"], maxRows: 2000 } } | |
| 1038 | + - { name: ml hardware dataset, url: "https://epoch.ai/data/ml_hardware.csv", type: FILE, connector: csv, tier: D, config: { keyColumn: "Hardware name", titleColumn: "Hardware name", dateColumn: "Release date", compareColumns: [Manufacturer, "Release date", "Tensor-FP16/BF16 performance (FLOP/s)", "Memory (bytes)", "TDP (W)"], maxRows: 2000 } } | |
| 1039 | + - { name: ai supercomputers dataset, url: "https://epoch.ai/data/gpu_clusters.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: Name, titleColumn: Name, dateColumn: "First Operational Date", compareColumns: [Status, "Chip type (primary)", "Chip quantity (primary)", "Power Capacity (MW)", Owner], maxRows: 2000 } } | |
| 1040 | + - id: ibm-research | |
| 1041 | + name: IBM Research | |
| 1042 | + domain: research.ibm.com | |
| 1043 | + categories: [ai, research, technology] | |
| 1044 | + tier: B | |
| 1045 | + country: US | |
| 1046 | + aliases: [ibm research, ibm granite, granite] | |
| 1047 | + products: | |
| 1048 | + - { name: Granite, type: AI_model, aliases: [granite 4, ibm granite] } | |
| 1049 | + discover: { rss: true } | |
| 1050 | + sensors: | |
| 1051 | + - { name: news feed, url: "https://research.ibm.com/rss", type: RSS, connector: rss, tier: B } | |
| 1052 | + - { name: granite page, url: "https://www.ibm.com/granite", type: HTML, connector: http, tier: C } | |
| 1053 | + - { name: hugging face granite models, url: "https://huggingface.co/api/models?author=ibm-granite&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 1054 | + - { name: granite 4.0 commits, url: "https://github.com/ibm-granite/granite-4.0-language-models/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ibm-granite/granite-4.0-language-models, kind: commits, branch: main } } | |
| 1055 | + - id: tii | |
| 1056 | + name: Technology Innovation Institute | |
| 1057 | + domain: tii.ae | |
| 1058 | + homepage: https://falconllm.tii.ae | |
| 1059 | + categories: [ai, research] | |
| 1060 | + tier: C | |
| 1061 | + country: AE | |
| 1062 | + aliases: [tii, falcon llm, falcon] | |
| 1063 | + products: | |
| 1064 | + - { name: Falcon, type: AI_model, aliases: [falcon-h1, falcon 3, falcon mamba] } | |
| 1065 | + discover: { rss: false } | |
| 1066 | + sensors: | |
| 1067 | + - { name: falcon llm site, url: "https://falconllm.tii.ae/", type: HTML, connector: http, tier: C } | |
| 1068 | + - { name: falcon models, url: "https://falconllm.tii.ae/falcon-models.html", type: HTML, connector: http, tier: C } | |
| 1069 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=tiiuae&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 1070 | + - { name: news, url: "https://www.tii.ae/news", type: HTML, connector: http, tier: C } | |
| 1071 | + - id: ai2 | |
| 1072 | + extend: true | |
| 1073 | + country: US | |
| 1074 | + sensors: | |
| 1075 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=allenai&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 1076 | + - id: nous-research | |
| 1077 | + name: Nous Research | |
| 1078 | + domain: nousresearch.com | |
| 1079 | + categories: [ai, research] | |
| 1080 | + tier: C | |
| 1081 | + country: US | |
| 1082 | + aliases: [nous, hermes] | |
| 1083 | + products: | |
| 1084 | + - { name: Hermes, type: AI_model, aliases: [hermes 4] } | |
| 1085 | + discover: { rss: false } | |
| 1086 | + notes: "No RSS (feed paths 404); the blog index is server-rendered." | |
| 1087 | + sensors: | |
| 1088 | + - { name: blog index, url: "https://nousresearch.com/blog", type: HTML, connector: http, tier: C } | |
| 1089 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=NousResearch&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 1090 | + - id: ai21 | |
| 1091 | + name: AI21 Labs | |
| 1092 | + domain: ai21.com | |
| 1093 | + homepage: https://www.ai21.com | |
| 1094 | + categories: [ai, technology] | |
| 1095 | + tier: C | |
| 1096 | + country: IL | |
| 1097 | + aliases: [ai21 labs, jamba, maestro] | |
| 1098 | + products: | |
| 1099 | + - { name: Jamba, type: AI_model } | |
| 1100 | + discover: { rss: true } | |
| 1101 | + sensors: | |
| 1102 | + - { name: status, url: "https://status.ai21.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 1103 | + - { name: docs changelog, url: "https://docs.ai21.com/changelog", type: HTML, connector: http, tier: B } | |
| 1104 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=ai21labs&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 1105 | + - { name: blog index, url: "https://www.ai21.com/blog/", type: HTML, connector: http, tier: C } | |
| 1106 | + - id: liquid-ai | |
| 1107 | + name: Liquid AI | |
| 1108 | + domain: liquid.ai | |
| 1109 | + homepage: https://www.liquid.ai | |
| 1110 | + categories: [ai, technology] | |
| 1111 | + tier: C | |
| 1112 | + country: US | |
| 1113 | + aliases: [liquid, lfm, leap] | |
| 1114 | + products: | |
| 1115 | + - { name: LFM, type: AI_model, aliases: [liquid foundation models, lfm2] } | |
| 1116 | + discover: { rss: false } | |
| 1117 | + sensors: | |
| 1118 | + - { name: model news, url: "https://www.liquid.ai/news/models", type: HTML, connector: http, tier: C } | |
| 1119 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=LiquidAI&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 1120 | + - id: salesforce | |
| 1121 | + extend: true | |
| 1122 | + categories: [ai] | |
| 1123 | + sensors: | |
| 1124 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=Salesforce&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
| 1125 | + - id: servicenow | |
| 1126 | + extend: true | |
| 1127 | + categories: [ai] | |
| 1128 | + sensors: | |
| 1129 | + - { name: hugging face models, url: "https://huggingface.co/api/models?author=ServiceNow-AI&sort=lastModified&direction=-1&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { keyField: id, titleField: id, dateField: lastModified, compareFields: [pipeline_tag, library_name], maxItems: 50 } } | |
added
config/sources.d/41-cloud-infrastructure.yaml
+994 −0
@@ -0,0 +1,994 @@ | ||
| 1 | +# config/sources.d/41-cloud-infrastructure.yaml — 2026-09-11 — depth coverage of cloud / Internet infrastructure | |
| 2 | +# providers: outages and status changes, regions, new services, pricing, deprecations, incidents, postmortems and | |
| 3 | +# API modifications. Hyperscalers, European/Asian clouds, PaaS, managed databases, object storage/CDN, network | |
| 4 | +# operators, IXPs, RIRs, DNS root, CAs, identity, communications, observability, developer platforms and the | |
| 5 | +# infrastructure software they run on. Every sensor below was fetched and parsed by the validator on 2026-09-11. | |
| 6 | +# Blocked / not machine-readable (documented, not added): Vultr web (Cloudflare 403 — the public v2 API is used | |
| 7 | +# instead), linode.com docs/pricing (Akamai 403), Alibaba/Tencent/Huawei status pages (JS shells), Redis and | |
| 8 | +# Backblaze status (FireHydrant, no API), Railway status (custom), Neon/Databricks/PagerDuty/Okta status (custom), | |
| 9 | +# Equinix corporate status (Akamai 403), DE-CIX/LINX/Zayo/Lumen status hosts (unreachable), PCH (no feed). | |
| 10 | +sources: | |
| 11 | + # ───────────────────────── hyperscalers ───────────────────────── | |
| 12 | + - id: aws | |
| 13 | + extend: true | |
| 14 | + country: US | |
| 15 | + language: en | |
| 16 | + products: | |
| 17 | + - { name: Amazon RDS, type: product, aliases: [rds] } | |
| 18 | + - { name: Amazon EKS, type: product, aliases: [eks] } | |
| 19 | + - { name: Amazon Route 53, type: product, aliases: [route 53, route53] } | |
| 20 | + - { name: Amazon CloudFront, type: product, aliases: [cloudfront] } | |
| 21 | + - { name: AWS Health Dashboard, type: service, aliases: [aws status] } | |
| 22 | + notes: "health.aws.amazon.com/public/currentevents is JSON but served as UTF-16 (BOM FF FE) regardless of Accept headers — the fetcher decodes UTF-8, so it cannot be a jsonlist sensor until the connector handles UTF-16. Per-service status RSS feeds (status.aws.amazon.com/rss/<service>-<region>.rss) are empty until an incident occurs; only route53 and cloudfront had items on 2026-09-11 and were kept." | |
| 23 | + sensors: | |
| 24 | + - { name: route 53 status feed, url: "https://status.aws.amazon.com/rss/route53.rss", type: RSS, connector: rss, tier: S } | |
| 25 | + - { name: cloudfront status feed, url: "https://status.aws.amazon.com/rss/cloudfront.rss", type: RSS, connector: rss, tier: S } | |
| 26 | + - { name: compute blog feed, url: "https://aws.amazon.com/blogs/compute/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 27 | + - { name: architecture blog feed (architecture), url: "https://aws.amazon.com/blogs/architecture/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 28 | + - { name: networking blog feed, url: "https://aws.amazon.com/blogs/networking-and-content-delivery/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 29 | + - { name: database blog feed, url: "https://aws.amazon.com/blogs/database/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 30 | + - { name: containers blog feed, url: "https://aws.amazon.com/blogs/containers/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 31 | + - { name: storage blog feed, url: "https://aws.amazon.com/blogs/storage/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 32 | + - { name: devops blog feed, url: "https://aws.amazon.com/blogs/devops/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 33 | + - { name: open source blog feed, url: "https://aws.amazon.com/blogs/opensource/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 34 | + - { name: regions and availability zones, url: "https://aws.amazon.com/about-aws/global-infrastructure/regions_az/", type: HTML, connector: http, tier: C } | |
| 35 | + - { name: service endpoints and quotas reference, url: "https://docs.aws.amazon.com/general/latest/gr/rande.html", type: HTML, connector: http, tier: C } | |
| 36 | + - { name: ec2 on-demand pricing, url: "https://aws.amazon.com/ec2/pricing/on-demand/", type: HTML, connector: http, tier: C } | |
| 37 | + - { name: ip ranges, url: "https://ip-ranges.amazonaws.com/ip-ranges.json", type: JSON, connector: http, tier: C, config: { ignoreKeys: [syncToken, createDate] } } | |
| 38 | + - { name: aws cli releases, url: "https://github.com/aws/aws-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: aws/aws-cli, kind: releases } } | |
| 39 | + - { name: aws cdk releases, url: "https://github.com/aws/aws-cdk/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: aws/aws-cdk, kind: releases } } | |
| 40 | + - { name: aws sdk js v3 releases, url: "https://github.com/aws/aws-sdk-js-v3/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: aws/aws-sdk-js-v3, kind: releases } } | |
| 41 | + - { name: botocore releases, url: "https://github.com/boto/botocore/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: boto/botocore, kind: releases } } | |
| 42 | + - { name: karpenter releases, url: "https://github.com/aws/karpenter-provider-aws/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: aws/karpenter-provider-aws, kind: releases } } | |
| 43 | + - { name: cloudformation resource schema commits, url: "https://github.com/aws-cloudformation/cloudformation-resource-schema/commits/master.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: aws-cloudformation/cloudformation-resource-schema, kind: commits, branch: master } } | |
| 44 | + - id: azure | |
| 45 | + extend: true | |
| 46 | + country: US | |
| 47 | + language: en | |
| 48 | + products: | |
| 49 | + - { name: Azure Kubernetes Service, type: product, aliases: [aks] } | |
| 50 | + - { name: Azure DevOps, type: product, aliases: [azure devops] } | |
| 51 | + - { name: Bicep, type: software } | |
| 52 | + sensors: | |
| 53 | + - { name: azure sdk blog feed, url: "https://devblogs.microsoft.com/azure-sdk/feed/", type: RSS, connector: rss, tier: B } | |
| 54 | + - { name: azure devops blog feed, url: "https://devblogs.microsoft.com/devops/feed/", type: RSS, connector: rss, tier: B } | |
| 55 | + - { name: learn azure docs feed, url: "https://learn.microsoft.com/api/search/rss?search=azure&locale=en-us&$filter=scopes%2Fany(t%3A%20t%20eq%20%27Azure%27)", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 56 | + - { name: regions list, url: "https://learn.microsoft.com/en-us/azure/reliability/regions-list", type: HTML, connector: http, tier: C } | |
| 57 | + - { name: global infrastructure geographies, url: "https://azure.microsoft.com/en-us/explore/global-infrastructure/geographies/", type: HTML, connector: http, tier: C } | |
| 58 | + - { name: linux vm pricing, url: "https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/", type: HTML, connector: http, tier: C } | |
| 59 | + - { name: status history, url: "https://azure.status.microsoft/en-us/status/history/", type: HTML, connector: http, tier: A } | |
| 60 | + - { name: azure cli releases, url: "https://github.com/Azure/azure-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: Azure/azure-cli, kind: releases } } | |
| 61 | + - { name: aks releases, url: "https://github.com/Azure/AKS/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: Azure/AKS, kind: releases } } | |
| 62 | + - { name: bicep releases, url: "https://github.com/Azure/bicep/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: Azure/bicep, kind: releases } } | |
| 63 | + - { name: azure sdk for js releases, url: "https://github.com/Azure/azure-sdk-for-js/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: Azure/azure-sdk-for-js, kind: releases } } | |
| 64 | + - { name: azure rest api specs commits, url: "https://github.com/Azure/azure-rest-api-specs/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: Azure/azure-rest-api-specs, kind: commits, branch: main } } | |
| 65 | + - id: google-cloud | |
| 66 | + extend: true | |
| 67 | + country: US | |
| 68 | + language: en | |
| 69 | + products: | |
| 70 | + - { name: Compute Engine, type: product, aliases: [gce] } | |
| 71 | + - { name: Google Kubernetes Engine, type: product, aliases: [gke] } | |
| 72 | + - { name: Cloud Run, type: product } | |
| 73 | + - { name: Cloud SQL, type: product } | |
| 74 | + - { name: Spanner, type: product, aliases: [cloud spanner] } | |
| 75 | + - { name: AlloyDB, type: product } | |
| 76 | + sensors: | |
| 77 | + - { name: cloud blog feed, url: "https://cloudblog.withgoogle.com/rss/", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 78 | + - { name: compute engine release notes, url: "https://cloud.google.com/feeds/compute-engine-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 79 | + - { name: gke release notes, url: "https://cloud.google.com/feeds/kubernetes-engine-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 80 | + - { name: cloud run release notes, url: "https://cloud.google.com/feeds/cloud-run-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 81 | + - { name: bigquery release notes, url: "https://cloud.google.com/feeds/bigquery-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 82 | + - { name: vertex ai release notes, url: "https://cloud.google.com/feeds/vertex-ai-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 83 | + - { name: cloud sql release notes, url: "https://cloud.google.com/feeds/cloud-sql-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 84 | + - { name: spanner release notes, url: "https://cloud.google.com/feeds/spanner-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 85 | + - { name: alloydb release notes, url: "https://cloud.google.com/feeds/alloydb-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 86 | + - { name: cloud storage release notes, url: "https://cloud.google.com/feeds/cloud-storage-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 87 | + - { name: iam release notes, url: "https://cloud.google.com/feeds/iam-release-notes.xml", type: ATOM, connector: rss, tier: A } | |
| 88 | + - { name: cloud load balancing release notes, url: "https://cloud.google.com/feeds/cloud-load-balancing-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 89 | + - { name: cloud dns release notes, url: "https://cloud.google.com/feeds/cloud-dns-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 90 | + - { name: vpc release notes, url: "https://cloud.google.com/feeds/vpc-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 91 | + - { name: cloud cdn release notes, url: "https://cloud.google.com/feeds/cloud-cdn-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 92 | + - { name: cloud armor release notes, url: "https://cloud.google.com/feeds/cloud-armor-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 93 | + - { name: pub/sub release notes, url: "https://cloud.google.com/feeds/pubsub-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 94 | + - { name: cloud functions release notes, url: "https://cloud.google.com/feeds/cloud-functions-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 95 | + - { name: bigtable release notes, url: "https://cloud.google.com/feeds/bigtable-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 96 | + - { name: memorystore release notes, url: "https://cloud.google.com/feeds/memorystore-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 97 | + - { name: dataflow release notes, url: "https://cloud.google.com/feeds/dataflow-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 98 | + - { name: dataproc release notes, url: "https://cloud.google.com/feeds/dataproc-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 99 | + - { name: cloud build release notes, url: "https://cloud.google.com/feeds/cloud-build-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 100 | + - { name: cloud logging release notes, url: "https://cloud.google.com/feeds/cloud-logging-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 101 | + - { name: cloud monitoring release notes, url: "https://cloud.google.com/feeds/cloud-monitoring-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 102 | + - { name: secret manager release notes, url: "https://cloud.google.com/feeds/secret-manager-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 103 | + - { name: apigee release notes, url: "https://cloud.google.com/feeds/apigee-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 104 | + - { name: gemini for google cloud release notes, url: "https://cloud.google.com/feeds/gemini-release-notes.xml", type: ATOM, connector: rss, tier: B } | |
| 105 | + - { name: anthos / gdc release notes, url: "https://cloud.google.com/feeds/anthos-release-notes.xml", type: ATOM, connector: rss, tier: C } | |
| 106 | + - { name: locations, url: "https://cloud.google.com/about/locations", type: HTML, connector: http, tier: C } | |
| 107 | + - { name: compute engine deprecations, url: "https://cloud.google.com/compute/docs/deprecations", type: HTML, connector: http, tier: C } | |
| 108 | + - { name: compute engine pricing, url: "https://cloud.google.com/compute/all-pricing", type: HTML, connector: http, tier: C } | |
| 109 | + - { name: gcloud cli release notes, url: "https://cloud.google.com/sdk/docs/release-notes", type: HTML, connector: http, tier: B } | |
| 110 | + - { name: ip ranges, url: "https://www.gstatic.com/ipranges/cloud.json", type: JSON, connector: http, tier: C, config: { ignoreKeys: [syncToken, creationTime] } } | |
| 111 | + - { name: google-cloud-go releases, url: "https://github.com/googleapis/google-cloud-go/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: googleapis/google-cloud-go, kind: releases } } | |
| 112 | + - { name: google-cloud-python releases, url: "https://github.com/googleapis/google-cloud-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: googleapis/google-cloud-python, kind: releases } } | |
| 113 | + - { name: googleapis proto commits, url: "https://github.com/googleapis/googleapis/commits/master.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: googleapis/googleapis, kind: commits, branch: master } } | |
| 114 | + - id: cloudflare | |
| 115 | + extend: true | |
| 116 | + country: US | |
| 117 | + language: en | |
| 118 | + products: | |
| 119 | + - { name: Cloudflare R2, type: product, aliases: [r2] } | |
| 120 | + - { name: Cloudflare D1, type: product, aliases: [d1] } | |
| 121 | + - { name: Cloudflare Pages, type: product } | |
| 122 | + - { name: Workers AI, type: product } | |
| 123 | + - { name: Cloudflare One, type: product, aliases: [zero trust] } | |
| 124 | + - { name: cloudflared, type: software } | |
| 125 | + - { name: workerd, type: software } | |
| 126 | + sensors: | |
| 127 | + - { name: workers changelog feed, url: "https://developers.cloudflare.com/changelog/rss/workers.xml", type: RSS, connector: rss, tier: A } | |
| 128 | + - { name: workers ai changelog feed, url: "https://developers.cloudflare.com/changelog/rss/workers-ai.xml", type: RSS, connector: rss, tier: B } | |
| 129 | + - { name: r2 changelog feed, url: "https://developers.cloudflare.com/changelog/rss/r2.xml", type: RSS, connector: rss, tier: B } | |
| 130 | + - { name: d1 changelog feed, url: "https://developers.cloudflare.com/changelog/rss/d1.xml", type: RSS, connector: rss, tier: B } | |
| 131 | + - { name: pages changelog feed, url: "https://developers.cloudflare.com/changelog/rss/pages.xml", type: RSS, connector: rss, tier: B } | |
| 132 | + - { name: waf changelog feed, url: "https://developers.cloudflare.com/changelog/rss/waf.xml", type: RSS, connector: rss, tier: A } | |
| 133 | + - { name: dns changelog feed, url: "https://developers.cloudflare.com/changelog/rss/dns.xml", type: RSS, connector: rss, tier: B } | |
| 134 | + - { name: cloudflare one changelog feed, url: "https://developers.cloudflare.com/changelog/rss/cloudflare-one.xml", type: RSS, connector: rss, tier: B } | |
| 135 | + - { name: agents changelog feed, url: "https://developers.cloudflare.com/changelog/rss/agents.xml", type: RSS, connector: rss, tier: B } | |
| 136 | + - { name: workers platform changelog feed, url: "https://developers.cloudflare.com/workers/platform/changelog/index.xml", type: RSS, connector: rss, tier: A } | |
| 137 | + - { name: api deprecations, url: "https://developers.cloudflare.com/fundamentals/api/reference/deprecations/", type: HTML, connector: http, tier: B } | |
| 138 | + - { name: workers pricing, url: "https://developers.cloudflare.com/workers/platform/pricing/", type: HTML, connector: http, tier: C } | |
| 139 | + - { name: workers limits, url: "https://developers.cloudflare.com/workers/platform/limits/", type: HTML, connector: http, tier: C } | |
| 140 | + - { name: plans, url: "https://www.cloudflare.com/plans/", type: HTML, connector: http, tier: C } | |
| 141 | + - { name: network map, url: "https://www.cloudflare.com/network/", type: HTML, connector: http, tier: C } | |
| 142 | + - { name: ip ranges api, url: "https://api.cloudflare.com/client/v4/ips", type: JSON, connector: http, tier: D } | |
| 143 | + - { name: terraform provider registry, url: "https://registry.terraform.io/v1/providers/cloudflare/cloudflare", type: JSON, connector: http, tier: C, config: { ignoreKeys: [downloads] } } | |
| 144 | + - { name: workerd releases, url: "https://github.com/cloudflare/workerd/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cloudflare/workerd, kind: releases } } | |
| 145 | + - { name: cloudflared releases, url: "https://github.com/cloudflare/cloudflared/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cloudflare/cloudflared, kind: releases } } | |
| 146 | + - { name: workers-sdk releases, url: "https://github.com/cloudflare/workers-sdk/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cloudflare/workers-sdk, kind: releases } } | |
| 147 | + - { name: terraform provider releases, url: "https://github.com/cloudflare/terraform-provider-cloudflare/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cloudflare/terraform-provider-cloudflare, kind: releases } } | |
| 148 | + - { name: pingora releases, url: "https://github.com/cloudflare/pingora/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cloudflare/pingora, kind: releases } } | |
| 149 | + - id: oracle-cloud | |
| 150 | + extend: true | |
| 151 | + country: US | |
| 152 | + language: en | |
| 153 | + sensors: | |
| 154 | + - { name: oci status indicator, url: "https://ocistatus.oraclecloud.com/api/v2/status.json", type: JSON, connector: http, tier: S } | |
| 155 | + - { name: oci status history feed, url: "https://ocistatus.oraclecloud.com/history.rss", type: RSS, connector: rss, tier: S } | |
| 156 | + - { name: regions and availability domains, url: "https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm", type: HTML, connector: http, tier: C } | |
| 157 | + - { name: oci cli releases, url: "https://github.com/oracle/oci-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: oracle/oci-cli, kind: releases } } | |
| 158 | + - { name: terraform provider releases, url: "https://github.com/oracle/terraform-provider-oci/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: oracle/terraform-provider-oci, kind: releases } } | |
| 159 | + notes: "ocistatus.oraclecloud.com is a custom page: /api/v2/status.json (indicator) and history.rss work, summary/incidents do not; blogs.oracle.com RSS returns 403." | |
| 160 | + - id: ibm-cloud | |
| 161 | + extend: true | |
| 162 | + country: US | |
| 163 | + language: en | |
| 164 | + sensors: | |
| 165 | + - { name: status notifications feed, url: "https://cloud.ibm.com/status/api/notifications/feed.rss", type: RSS, connector: rss, tier: S, config: { maxItems: 80 } } | |
| 166 | + - { name: terraform provider releases, url: "https://github.com/IBM-Cloud/terraform-provider-ibm/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: IBM-Cloud/terraform-provider-ibm, kind: releases } } | |
| 167 | + - { name: cli releases, url: "https://github.com/IBM-Cloud/ibm-cloud-cli-release/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: IBM-Cloud/ibm-cloud-cli-release, kind: releases } } | |
| 168 | + notes: "cloud.ibm.com/status is a JS console; the notifications RSS is the machine-readable side. Release-notes feed redirects to login." | |
| 169 | + # ───────────────────────── european / regional clouds ───────────────────────── | |
| 170 | + - id: ovhcloud | |
| 171 | + extend: true | |
| 172 | + country: FR | |
| 173 | + language: en | |
| 174 | + products: | |
| 175 | + - { name: OVHcloud Public Cloud, type: product } | |
| 176 | + - { name: OVHcloud Bare Metal, type: product } | |
| 177 | + sensors: | |
| 178 | + - { name: public cloud status, url: "https://public-cloud.status-ovhcloud.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 179 | + - { name: network status, url: "https://network.status-ovhcloud.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 180 | + - { name: bare metal status, url: "https://bare-metal-servers.status-ovhcloud.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 181 | + - { name: web cloud status, url: "https://web-cloud.status-ovhcloud.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 182 | + - { name: hosted private cloud status, url: "https://hosted-private-cloud.status-ovhcloud.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 183 | + - { name: customer service status, url: "https://customer-service.status-ovhcloud.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: A } | |
| 184 | + - { name: blog feed, url: "https://blog.ovhcloud.com/feed/", type: RSS, connector: rss, tier: B } | |
| 185 | + - { name: public cloud prices, url: "https://www.ovhcloud.com/en-ca/public-cloud/prices/", type: HTML, connector: http, tier: C } | |
| 186 | + - { name: public cloud compute changelog, url: "https://help.ovhcloud.com/csm/en-ca-public-cloud-compute-changelog", type: HTML, connector: http, tier: B } | |
| 187 | + notes: "status.ovhcloud.com / travaux.ovh.net redirect to a Vue SPA (www.status-ovhcloud.com) that aggregates six Atlassian Statuspage instances — those are polled directly." | |
| 188 | + - id: hetzner | |
| 189 | + extend: true | |
| 190 | + country: DE | |
| 191 | + language: en | |
| 192 | + products: | |
| 193 | + - { name: Hetzner Cloud, type: product, aliases: [hcloud] } | |
| 194 | + sensors: | |
| 195 | + - { name: status feed, url: "https://status.hetzner.com/en.atom", type: ATOM, connector: rss, tier: S } | |
| 196 | + - { name: cloud api openapi, url: "https://docs.hetzner.cloud/cloud.spec.json", type: JSON, connector: openapi, tier: B, config: { docsUrl: "https://docs.hetzner.cloud/reference/cloud" } } | |
| 197 | + - { name: cloud api changelog, url: "https://docs.hetzner.cloud/changelog", type: HTML, connector: http, tier: B } | |
| 198 | + - { name: cloud locations, url: "https://docs.hetzner.com/cloud/general/locations/", type: HTML, connector: http, tier: C } | |
| 199 | + - { name: hcloud cli releases, url: "https://github.com/hetznercloud/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: hetznercloud/cli, kind: releases } } | |
| 200 | + - { name: hcloud-go releases, url: "https://github.com/hetznercloud/hcloud-go/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: hetznercloud/hcloud-go, kind: releases } } | |
| 201 | + - { name: terraform provider releases, url: "https://github.com/hetznercloud/terraform-provider-hcloud/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: hetznercloud/terraform-provider-hcloud, kind: releases } } | |
| 202 | + notes: "status.hetzner.com is a custom page (no /api/v2) but publishes en.atom; www.hetzner.com/news rate-limits (429)." | |
| 203 | + - id: scaleway | |
| 204 | + name: Scaleway | |
| 205 | + domain: scaleway.com | |
| 206 | + homepage: https://www.scaleway.com | |
| 207 | + categories: [cloud, infrastructure] | |
| 208 | + tier: B | |
| 209 | + weight: 1.1 | |
| 210 | + country: FR | |
| 211 | + language: en | |
| 212 | + aliases: [scaleway, online.net, iliad cloud] | |
| 213 | + products: | |
| 214 | + - { name: Scaleway Instances, type: product } | |
| 215 | + - { name: Scaleway Kubernetes Kapsule, type: product, aliases: [kapsule] } | |
| 216 | + discover: { rss: true, status: true } | |
| 217 | + sensors: | |
| 218 | + - { name: status, url: "https://status.scaleway.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 219 | + - { name: pricing, url: "https://www.scaleway.com/en/pricing/", type: HTML, connector: http, tier: C } | |
| 220 | + - { name: instance server catalogue fr-par-1, url: "https://api.scaleway.com/instance/v1/zones/fr-par-1/products/servers", type: JSON, connector: http, tier: C } | |
| 221 | + - { name: cli releases, url: "https://github.com/scaleway/scaleway-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: scaleway/scaleway-cli, kind: releases } } | |
| 222 | + - { name: terraform provider releases, url: "https://github.com/scaleway/terraform-provider-scaleway/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: scaleway/terraform-provider-scaleway, kind: releases } } | |
| 223 | + notes: "Blog has no feed; the public instance products API exposes the catalogue (prices, availability) without auth." | |
| 224 | + - id: upcloud | |
| 225 | + name: UpCloud | |
| 226 | + domain: upcloud.com | |
| 227 | + categories: [cloud, infrastructure] | |
| 228 | + tier: B | |
| 229 | + country: FI | |
| 230 | + language: en | |
| 231 | + aliases: [upcloud] | |
| 232 | + discover: { status: true } | |
| 233 | + sensors: | |
| 234 | + - { name: status, url: "https://status.upcloud.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 235 | + notes: "upcloud.com blog feed is Cloudflare-blocked (403); the API requires auth." | |
| 236 | + - id: aiven | |
| 237 | + name: Aiven | |
| 238 | + domain: aiven.io | |
| 239 | + categories: [cloud, enterprise] | |
| 240 | + tier: B | |
| 241 | + country: FI | |
| 242 | + language: en | |
| 243 | + aliases: [aiven] | |
| 244 | + discover: { rss: true, status: true } | |
| 245 | + sensors: | |
| 246 | + - { name: status, url: "https://status.aiven.io/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 247 | + - { name: changelog, url: "https://aiven.io/changelog", type: HTML, connector: http, tier: B } | |
| 248 | + - id: alibaba-cloud | |
| 249 | + name: Alibaba Cloud | |
| 250 | + domain: alibabacloud.com | |
| 251 | + homepage: https://www.alibabacloud.com | |
| 252 | + categories: [cloud, infrastructure] | |
| 253 | + tier: B | |
| 254 | + weight: 1.2 | |
| 255 | + country: CN | |
| 256 | + language: en | |
| 257 | + aliases: [alibaba cloud, aliyun, alicloud] | |
| 258 | + products: | |
| 259 | + - { name: Elastic Compute Service, type: product, aliases: [ecs] } | |
| 260 | + discover: { rss: false, status: false, pages: true } | |
| 261 | + sensors: | |
| 262 | + - { name: ecs pricing, url: "https://www.alibabacloud.com/en/product/ecs/pricing", type: HTML, connector: http, tier: C } | |
| 263 | + - { name: aliyun cli releases, url: "https://github.com/aliyun/aliyun-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: aliyun/aliyun-cli, kind: releases } } | |
| 264 | + - { name: terraform provider releases, url: "https://github.com/aliyun/terraform-provider-alicloud/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: aliyun/terraform-provider-alicloud, kind: releases } } | |
| 265 | + notes: "status.alibabacloud.com redirects to a Taobao error page for non-browser clients; blog/notice RSS endpoints return HTML." | |
| 266 | + - id: tencent-cloud | |
| 267 | + name: Tencent Cloud | |
| 268 | + domain: tencentcloud.com | |
| 269 | + homepage: https://www.tencentcloud.com | |
| 270 | + categories: [cloud, infrastructure] | |
| 271 | + tier: B | |
| 272 | + country: CN | |
| 273 | + language: en | |
| 274 | + aliases: [tencent cloud, qcloud] | |
| 275 | + discover: { rss: false, status: false } | |
| 276 | + sensors: | |
| 277 | + - { name: cli releases, url: "https://github.com/TencentCloud/tencentcloud-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: TencentCloud/tencentcloud-cli, kind: releases } } | |
| 278 | + - { name: terraform provider releases, url: "https://github.com/tencentcloudstack/terraform-provider-tencentcloud/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: tencentcloudstack/terraform-provider-tencentcloud, kind: releases } } | |
| 279 | + notes: "status.cloud.tencent.com and tencentcloud.com announcements are JS shells (same 29 KB body on every path)." | |
| 280 | + - id: huawei-cloud | |
| 281 | + name: Huawei Cloud | |
| 282 | + domain: huaweicloud.com | |
| 283 | + homepage: https://www.huaweicloud.com/intl/en-us/ | |
| 284 | + categories: [cloud, infrastructure] | |
| 285 | + tier: B | |
| 286 | + country: CN | |
| 287 | + language: en | |
| 288 | + aliases: [huawei cloud] | |
| 289 | + discover: { rss: false, status: false } | |
| 290 | + sensors: | |
| 291 | + - { name: sdk go v3 releases, url: "https://github.com/huaweicloud/huaweicloud-sdk-go-v3/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: huaweicloud/huaweicloud-sdk-go-v3, kind: releases } } | |
| 292 | + - { name: terraform provider releases, url: "https://github.com/huaweicloud/terraform-provider-huaweicloud/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: huaweicloud/terraform-provider-huaweicloud, kind: releases } } | |
| 293 | + notes: "status.huaweicloud.com does not answer from outside China; intl news pages return 404." | |
| 294 | + # ───────────────────────── developer clouds / VPS ───────────────────────── | |
| 295 | + - id: digitalocean | |
| 296 | + extend: true | |
| 297 | + country: US | |
| 298 | + language: en | |
| 299 | + sensors: | |
| 300 | + - { name: blog feed, url: "https://www.digitalocean.com/blog/rss", type: ATOM, connector: rss, tier: B } | |
| 301 | + - { name: droplet pricing, url: "https://www.digitalocean.com/pricing/droplets", type: HTML, connector: http, tier: C } | |
| 302 | + - { name: regional availability matrix, url: "https://docs.digitalocean.com/products/platform/availability-matrix/", type: HTML, connector: http, tier: C } | |
| 303 | + - { name: doctl releases, url: "https://github.com/digitalocean/doctl/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: digitalocean/doctl, kind: releases } } | |
| 304 | + - { name: terraform provider releases, url: "https://github.com/digitalocean/terraform-provider-digitalocean/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: digitalocean/terraform-provider-digitalocean, kind: releases } } | |
| 305 | + - id: vultr | |
| 306 | + extend: true | |
| 307 | + country: US | |
| 308 | + language: en | |
| 309 | + sensors: | |
| 310 | + - { name: regions api, url: "https://api.vultr.com/v2/regions", type: JSON, connector: http, tier: C } | |
| 311 | + - { name: plans and pricing api, url: "https://api.vultr.com/v2/plans", type: JSON, connector: http, tier: C } | |
| 312 | + - { name: vultr cli releases, url: "https://github.com/vultr/vultr-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: vultr/vultr-cli, kind: releases } } | |
| 313 | + - { name: terraform provider releases, url: "https://github.com/vultr/terraform-provider-vultr/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: vultr/terraform-provider-vultr, kind: releases } } | |
| 314 | + notes: "www.vultr.com and status.vultr.com are Cloudflare-challenged (403) for bots; the public v2 API (regions, plans) and docs.vultr.com are open." | |
| 315 | + - id: linode | |
| 316 | + extend: true | |
| 317 | + country: US | |
| 318 | + language: en | |
| 319 | + sensors: | |
| 320 | + - { name: blog feed, url: "https://www.linode.com/blog/feed/", type: RSS, connector: rss, tier: B } | |
| 321 | + - { name: cloud computing changelog feed, url: "https://techdocs.akamai.com/cloud-computing/changelog.rss", type: RSS, connector: rss, tier: A } | |
| 322 | + - { name: api v4 openapi, url: "https://raw.githubusercontent.com/linode/linode-api-docs/development/openapi.json", type: JSON, connector: openapi, tier: B, config: { docsUrl: "https://techdocs.akamai.com/linode-api/reference/api" } } | |
| 323 | + - { name: regions api, url: "https://api.linode.com/v4/regions", type: JSON, connector: http, tier: C } | |
| 324 | + - { name: instance types and pricing api, url: "https://api.linode.com/v4/linode/types", type: JSON, connector: http, tier: C } | |
| 325 | + - { name: linode cli releases, url: "https://github.com/linode/linode-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: linode/linode-cli, kind: releases } } | |
| 326 | + - { name: terraform provider releases, url: "https://github.com/linode/terraform-provider-linode/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: linode/terraform-provider-linode, kind: releases } } | |
| 327 | + notes: "linode.com/docs and /pricing are Akamai-blocked (403); the unauthenticated v4 API exposes regions and types with prices." | |
| 328 | + - id: akamai | |
| 329 | + extend: true | |
| 330 | + country: US | |
| 331 | + language: en | |
| 332 | + sensors: | |
| 333 | + - { name: status, url: "https://www.akamaistatus.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 334 | + # ───────────────────────── CDN / edge ───────────────────────── | |
| 335 | + - id: fastly | |
| 336 | + extend: true | |
| 337 | + country: US | |
| 338 | + language: en | |
| 339 | + sensors: | |
| 340 | + - { name: status history feed, url: "https://www.fastlystatus.com/rss", type: RSS, connector: rss, tier: S } | |
| 341 | + - { name: api and product changes, url: "https://www.fastly.com/documentation/reference/changes/", type: HTML, connector: http, tier: B } | |
| 342 | + - { name: public ip list, url: "https://api.fastly.com/public-ip-list", type: JSON, connector: http, tier: D } | |
| 343 | + - { name: cli releases, url: "https://github.com/fastly/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: fastly/cli, kind: releases } } | |
| 344 | + - { name: terraform provider releases, url: "https://github.com/fastly/terraform-provider-fastly/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: fastly/terraform-provider-fastly, kind: releases } } | |
| 345 | + notes: "status.fastly.com is a custom page (no /api/v2); www.fastlystatus.com/rss is the machine-readable history. Blog has no feed." | |
| 346 | + - id: bunny-net | |
| 347 | + extend: true | |
| 348 | + country: SI | |
| 349 | + language: en | |
| 350 | + sensors: | |
| 351 | + - { name: blog feed, url: "https://bunny.net/blog/rss/", type: RSS, connector: rss, tier: B } | |
| 352 | + - { name: docs changelog, url: "https://docs.bunny.net/changelog", type: HTML, connector: http, tier: B } | |
| 353 | + - { name: network pops, url: "https://bunny.net/network/", type: HTML, connector: http, tier: C } | |
| 354 | + # ───────────────────────── PaaS / frontend clouds ───────────────────────── | |
| 355 | + - id: vercel | |
| 356 | + extend: true | |
| 357 | + country: US | |
| 358 | + language: en | |
| 359 | + sensors: | |
| 360 | + - { name: pricing, url: "https://vercel.com/pricing", type: HTML, connector: http, tier: C } | |
| 361 | + - { name: edge network regions, url: "https://vercel.com/docs/edge-network/regions", type: HTML, connector: http, tier: C } | |
| 362 | + - { name: vercel cli releases, url: "https://github.com/vercel/vercel/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: vercel/vercel, kind: releases } } | |
| 363 | + - { name: turborepo releases, url: "https://github.com/vercel/turborepo/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: vercel/turborepo, kind: releases } } | |
| 364 | + - id: netlify | |
| 365 | + extend: true | |
| 366 | + country: US | |
| 367 | + language: en | |
| 368 | + sensors: | |
| 369 | + - { name: changelog feed, url: "https://www.netlify.com/changelog/feed.xml", type: RSS, connector: rss, tier: A } | |
| 370 | + - { name: blog feed, url: "https://www.netlify.com/feed.xml", type: RSS, connector: rss, tier: B } | |
| 371 | + - { name: pricing, url: "https://www.netlify.com/pricing/", type: HTML, connector: http, tier: C } | |
| 372 | + - { name: api openapi, url: "https://open-api.netlify.com/swagger.json", type: JSON, connector: openapi, tier: B, config: { docsUrl: "https://open-api.netlify.com/" } } | |
| 373 | + - { name: cli releases, url: "https://github.com/netlify/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: netlify/cli, kind: releases } } | |
| 374 | + - { name: build releases, url: "https://github.com/netlify/build/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: netlify/build, kind: releases } } | |
| 375 | + - id: fly-io | |
| 376 | + extend: true | |
| 377 | + country: US | |
| 378 | + language: en | |
| 379 | + sensors: | |
| 380 | + - { name: infra log feed, url: "https://community.fly.io/c/infra-log/34.rss", type: RSS, connector: rss, tier: A } | |
| 381 | + - { name: machines api openapi, url: "https://docs.machines.dev/spec/openapi3.json", type: JSON, connector: openapi, tier: B, config: { docsUrl: "https://docs.machines.dev/" } } | |
| 382 | + - { name: pricing, url: "https://fly.io/docs/about/pricing/", type: HTML, connector: http, tier: C } | |
| 383 | + - { name: regions, url: "https://fly.io/docs/reference/regions/", type: HTML, connector: http, tier: C } | |
| 384 | + - { name: flyctl releases, url: "https://github.com/superfly/flyctl/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: superfly/flyctl, kind: releases } } | |
| 385 | + - id: railway | |
| 386 | + extend: true | |
| 387 | + country: US | |
| 388 | + language: en | |
| 389 | + sensors: | |
| 390 | + - { name: changelog feed, url: "https://railway.com/changelog/rss.xml", type: RSS, connector: rss, tier: A } | |
| 391 | + - { name: blog feed, url: "https://blog.railway.com/rss.xml", type: RSS, connector: rss, tier: B } | |
| 392 | + - { name: pricing, url: "https://railway.com/pricing", type: HTML, connector: http, tier: C } | |
| 393 | + - { name: cli releases, url: "https://github.com/railwayapp/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: railwayapp/cli, kind: releases } } | |
| 394 | + notes: "status.railway.com is a custom Next.js page (every path returns the same HTML); public API is GraphQL only." | |
| 395 | + # ───────────────────────── managed databases / data platforms ───────────────────────── | |
| 396 | + - id: supabase | |
| 397 | + extend: true | |
| 398 | + country: US | |
| 399 | + language: en | |
| 400 | + sensors: | |
| 401 | + - { name: pricing, url: "https://supabase.com/pricing", type: HTML, connector: http, tier: C } | |
| 402 | + - { name: supabase releases, url: "https://github.com/supabase/supabase/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: supabase/supabase, kind: releases } } | |
| 403 | + - { name: cli releases, url: "https://github.com/supabase/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: supabase/cli, kind: releases } } | |
| 404 | + - { name: postgres image releases, url: "https://github.com/supabase/postgres/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: supabase/postgres, kind: releases } } | |
| 405 | + - id: neon | |
| 406 | + extend: true | |
| 407 | + country: US | |
| 408 | + language: en | |
| 409 | + sensors: | |
| 410 | + - { name: changelog feed, url: "https://neon.com/docs/changelog/rss.xml", type: RSS, connector: rss, tier: A } | |
| 411 | + - { name: pricing, url: "https://neon.com/pricing", type: HTML, connector: http, tier: C } | |
| 412 | + - { name: api v2 openapi, url: "https://neon.com/api_spec/release/v2.json", type: JSON, connector: openapi, tier: B, config: { docsUrl: "https://api-docs.neon.tech/reference/getting-started-with-neon-api" } } | |
| 413 | + - id: planetscale | |
| 414 | + extend: true | |
| 415 | + country: US | |
| 416 | + language: en | |
| 417 | + sensors: | |
| 418 | + - { name: changelog feed, url: "https://planetscale.com/changelog/feed.atom", type: ATOM, connector: rss, tier: A } | |
| 419 | + - { name: pricing, url: "https://planetscale.com/pricing", type: HTML, connector: http, tier: C } | |
| 420 | + - { name: api openapi, url: "https://api.planetscale.com/v1/openapi-spec", type: JSON, connector: openapi, tier: B, config: { docsUrl: "https://api-docs.planetscale.com/" } } | |
| 421 | + - id: mongodb | |
| 422 | + extend: true | |
| 423 | + country: US | |
| 424 | + language: en | |
| 425 | + products: | |
| 426 | + - { name: MongoDB Atlas, type: product, aliases: [atlas] } | |
| 427 | + sensors: | |
| 428 | + - { name: status history feed, url: "https://status.mongodb.com/history.rss", type: RSS, connector: rss, tier: A } | |
| 429 | + - { name: blog feed, url: "https://www.mongodb.com/blog/rss", type: RSS, connector: rss, tier: B } | |
| 430 | + - { name: atlas release notes, url: "https://www.mongodb.com/docs/atlas/release-notes/", type: HTML, connector: http, tier: B } | |
| 431 | + - { name: server release notes, url: "https://www.mongodb.com/docs/manual/release-notes/", type: HTML, connector: http, tier: B } | |
| 432 | + - { name: pricing, url: "https://www.mongodb.com/pricing", type: HTML, connector: http, tier: C } | |
| 433 | + - { name: atlas cli releases, url: "https://github.com/mongodb/mongodb-atlas-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: mongodb/mongodb-atlas-cli, kind: releases } } | |
| 434 | + - id: redis | |
| 435 | + extend: true | |
| 436 | + country: US | |
| 437 | + language: en | |
| 438 | + sensors: | |
| 439 | + - { name: blog feed, url: "https://redis.io/blog/feed/", type: RSS, connector: rss, tier: B } | |
| 440 | + - { name: redis software release notes, url: "https://redis.io/docs/latest/operate/rs/release-notes/", type: HTML, connector: http, tier: B } | |
| 441 | + - { name: redis-py releases, url: "https://github.com/redis/redis-py/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: redis/redis-py, kind: releases } } | |
| 442 | + - { name: node-redis releases, url: "https://github.com/redis/node-redis/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: redis/node-redis, kind: releases } } | |
| 443 | + notes: "status.redis.io is a FireHydrant page (HTML only, no JSON/RSS)." | |
| 444 | + - id: elastic | |
| 445 | + extend: true | |
| 446 | + country: NL | |
| 447 | + language: en | |
| 448 | + sensors: | |
| 449 | + - { name: blog feed, url: "https://www.elastic.co/blog/feed", type: RSS, connector: rss, tier: B } | |
| 450 | + - { name: elasticsearch release notes, url: "https://www.elastic.co/docs/release-notes/elasticsearch", type: HTML, connector: http, tier: B } | |
| 451 | + - { name: pricing, url: "https://www.elastic.co/pricing", type: HTML, connector: http, tier: C } | |
| 452 | + - { name: kibana releases, url: "https://github.com/elastic/kibana/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: elastic/kibana, kind: releases } } | |
| 453 | + - { name: eck operator releases, url: "https://github.com/elastic/cloud-on-k8s/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: elastic/cloud-on-k8s, kind: releases } } | |
| 454 | + - id: cockroachdb | |
| 455 | + extend: true | |
| 456 | + country: US | |
| 457 | + language: en | |
| 458 | + sensors: | |
| 459 | + - { name: releases page, url: "https://www.cockroachlabs.com/docs/releases/", type: HTML, connector: http, tier: B } | |
| 460 | + - { name: cloud release notes, url: "https://www.cockroachlabs.com/docs/releases/cloud", type: HTML, connector: http, tier: B } | |
| 461 | + - { name: pricing, url: "https://www.cockroachlabs.com/pricing/", type: HTML, connector: http, tier: C } | |
| 462 | + - id: confluent | |
| 463 | + extend: true | |
| 464 | + country: US | |
| 465 | + language: en | |
| 466 | + sensors: | |
| 467 | + - { name: cloud release notes, url: "https://docs.confluent.io/cloud/current/release-notes/index.html", type: HTML, connector: http, tier: B } | |
| 468 | + - { name: cloud pricing, url: "https://www.confluent.io/confluent-cloud/pricing/", type: HTML, connector: http, tier: C } | |
| 469 | + - { name: cli releases, url: "https://github.com/confluentinc/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: confluentinc/cli, kind: releases } } | |
| 470 | + - { name: confluent-kafka-go releases, url: "https://github.com/confluentinc/confluent-kafka-go/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: confluentinc/confluent-kafka-go, kind: releases } } | |
| 471 | + - { name: terraform provider releases, url: "https://github.com/confluentinc/terraform-provider-confluent/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: confluentinc/terraform-provider-confluent, kind: releases } } | |
| 472 | + - id: snowflake | |
| 473 | + extend: true | |
| 474 | + country: US | |
| 475 | + language: en | |
| 476 | + sensors: | |
| 477 | + - { name: behavior change releases, url: "https://docs.snowflake.com/en/release-notes/behavior-changes", type: HTML, connector: http, tier: B } | |
| 478 | + - { name: pricing, url: "https://www.snowflake.com/pricing/", type: HTML, connector: http, tier: C } | |
| 479 | + - { name: snowflake cli releases, url: "https://github.com/snowflakedb/snowflake-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: snowflakedb/snowflake-cli, kind: releases } } | |
| 480 | + - { name: snowpark python releases, url: "https://github.com/snowflakedb/snowpark-python/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: snowflakedb/snowpark-python, kind: releases } } | |
| 481 | + - id: databricks | |
| 482 | + extend: true | |
| 483 | + country: US | |
| 484 | + language: en | |
| 485 | + sensors: | |
| 486 | + - { name: product release notes, url: "https://docs.databricks.com/aws/en/release-notes/product/index.html", type: HTML, connector: http, tier: B } | |
| 487 | + - { name: pricing, url: "https://www.databricks.com/product/pricing", type: HTML, connector: http, tier: C } | |
| 488 | + - { name: cli releases, url: "https://github.com/databricks/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: databricks/cli, kind: releases } } | |
| 489 | + - { name: terraform provider releases, url: "https://github.com/databricks/terraform-provider-databricks/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: databricks/terraform-provider-databricks, kind: releases } } | |
| 490 | + notes: "status.databricks.com is a custom page (no Statuspage/Instatus API)." | |
| 491 | + - id: upstash | |
| 492 | + extend: true | |
| 493 | + country: US | |
| 494 | + language: en | |
| 495 | + sensors: | |
| 496 | + - { name: pricing, url: "https://upstash.com/pricing", type: HTML, connector: http, tier: C } | |
| 497 | + - { name: redis changelog, url: "https://upstash.com/docs/redis/overall/changelog", type: HTML, connector: http, tier: B } | |
| 498 | + - id: turso | |
| 499 | + name: Turso | |
| 500 | + domain: turso.tech | |
| 501 | + categories: [cloud, developer] | |
| 502 | + tier: B | |
| 503 | + country: US | |
| 504 | + language: en | |
| 505 | + aliases: [turso, libsql] | |
| 506 | + products: | |
| 507 | + - { name: libSQL, type: software, aliases: [libsql] } | |
| 508 | + discover: { rss: true } | |
| 509 | + sensors: | |
| 510 | + - { name: blog feed, url: "https://turso.tech/blog/feed.xml", type: ATOM, connector: rss, tier: B } | |
| 511 | + - { name: pricing, url: "https://turso.tech/pricing", type: HTML, connector: http, tier: C } | |
| 512 | + - { name: turso database releases, url: "https://github.com/tursodatabase/turso/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: tursodatabase/turso, kind: releases } } | |
| 513 | + - { name: libsql releases, url: "https://github.com/tursodatabase/libsql/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: tursodatabase/libsql, kind: releases } } | |
| 514 | + - { name: cli releases, url: "https://github.com/tursodatabase/turso-cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: tursodatabase/turso-cli, kind: releases } } | |
| 515 | + notes: "status.turso.tech is a Better Stack page (HTML only)." | |
| 516 | + # ───────────────────────── object storage ───────────────────────── | |
| 517 | + - id: backblaze | |
| 518 | + name: Backblaze | |
| 519 | + domain: backblaze.com | |
| 520 | + homepage: https://www.backblaze.com | |
| 521 | + categories: [cloud, infrastructure] | |
| 522 | + tier: B | |
| 523 | + country: US | |
| 524 | + language: en | |
| 525 | + aliases: [backblaze, b2] | |
| 526 | + products: | |
| 527 | + - { name: Backblaze B2, type: product, aliases: [b2 cloud storage] } | |
| 528 | + discover: { rss: true } | |
| 529 | + sensors: | |
| 530 | + - { name: blog feed, url: "https://www.backblaze.com/blog/feed/", type: RSS, connector: rss, tier: B } | |
| 531 | + - { name: b2 pricing, url: "https://www.backblaze.com/cloud-storage/pricing", type: HTML, connector: http, tier: C } | |
| 532 | + - { name: b2 cli releases, url: "https://github.com/Backblaze/B2_Command_Line_Tool/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: Backblaze/B2_Command_Line_Tool, kind: releases } } | |
| 533 | + notes: "status.backblaze.com is a FireHydrant page (HTML only)." | |
| 534 | + - id: wasabi | |
| 535 | + name: Wasabi Technologies | |
| 536 | + domain: wasabi.com | |
| 537 | + categories: [cloud, infrastructure] | |
| 538 | + tier: B | |
| 539 | + country: US | |
| 540 | + language: en | |
| 541 | + aliases: [wasabi, wasabi hot cloud storage] | |
| 542 | + discover: { status: true } | |
| 543 | + sensors: | |
| 544 | + - { name: status, url: "https://status.wasabi.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 545 | + - { name: pricing, url: "https://wasabi.com/pricing/", type: HTML, connector: http, tier: C } | |
| 546 | + notes: "wasabi.com has no blog feed; docs release notes 404." | |
| 547 | + - id: tigris | |
| 548 | + name: Tigris Data | |
| 549 | + domain: tigrisdata.com | |
| 550 | + homepage: https://www.tigrisdata.com | |
| 551 | + categories: [cloud, developer] | |
| 552 | + tier: C | |
| 553 | + country: US | |
| 554 | + language: en | |
| 555 | + aliases: [tigris, tigris data] | |
| 556 | + discover: { rss: true, status: true } | |
| 557 | + sensors: | |
| 558 | + - { name: status, url: "https://status.tigrisdata.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 559 | + - { name: blog feed, url: "https://www.tigrisdata.com/blog/rss.xml", type: RSS, connector: rss, tier: B } | |
| 560 | + - { name: pricing, url: "https://www.tigrisdata.com/pricing/", type: HTML, connector: http, tier: C } | |
| 561 | + # ───────────────────────── network operators / IXPs / data centres ───────────────────────── | |
| 562 | + - id: equinix | |
| 563 | + extend: true | |
| 564 | + country: US | |
| 565 | + language: en | |
| 566 | + products: | |
| 567 | + - { name: Equinix Metal, type: product, aliases: [packet] } | |
| 568 | + sensors: | |
| 569 | + - { name: equinix metal status, url: "https://status.equinixmetal.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 570 | + - id: zayo | |
| 571 | + name: Zayo | |
| 572 | + domain: zayo.com | |
| 573 | + homepage: https://www.zayo.com | |
| 574 | + categories: [telecom, infrastructure] | |
| 575 | + tier: C | |
| 576 | + country: US | |
| 577 | + language: en | |
| 578 | + aliases: [zayo, zayo group] | |
| 579 | + discover: { rss: true } | |
| 580 | + sensors: | |
| 581 | + - { name: blog feed, url: "https://www.zayo.com/feed/", type: RSS, connector: rss, tier: C } | |
| 582 | + notes: "newsroom feed is empty; status.zayo.com does not resolve." | |
| 583 | + - id: arelion | |
| 584 | + name: Arelion | |
| 585 | + domain: arelion.com | |
| 586 | + homepage: https://www.arelion.com | |
| 587 | + categories: [telecom, infrastructure] | |
| 588 | + tier: C | |
| 589 | + country: SE | |
| 590 | + language: en | |
| 591 | + aliases: [arelion, telia carrier, as1299] | |
| 592 | + discover: { rss: true } | |
| 593 | + sensors: | |
| 594 | + - { name: blog feed, url: "https://blog.arelion.com/feed", type: RSS, connector: rss, tier: C } | |
| 595 | + notes: "arelion.com press pages return 404 to non-browser clients; the blog is open." | |
| 596 | + - id: hurricane-electric | |
| 597 | + name: Hurricane Electric | |
| 598 | + domain: he.net | |
| 599 | + categories: [telecom, infrastructure, internet] | |
| 600 | + tier: C | |
| 601 | + country: US | |
| 602 | + language: en | |
| 603 | + aliases: [hurricane electric, he.net, as6939] | |
| 604 | + discover: { rss: false } | |
| 605 | + sensors: | |
| 606 | + - { name: news, url: "https://he.net/news.html", type: HTML, connector: http, tier: C } | |
| 607 | + - id: arin | |
| 608 | + extend: true | |
| 609 | + country: US | |
| 610 | + language: en | |
| 611 | + sensors: | |
| 612 | + - { name: announcements feed, url: "https://www.arin.net/announcements/rss.xml", type: RSS, connector: rss, tier: B } | |
| 613 | + # ───────────────────────── DNS root / registries ───────────────────────── | |
| 614 | + - id: root-servers | |
| 615 | + name: Root Server Operators | |
| 616 | + domain: root-servers.org | |
| 617 | + categories: [internet, infrastructure, standards] | |
| 618 | + tier: B | |
| 619 | + weight: 1.3 | |
| 620 | + country: INT | |
| 621 | + language: en | |
| 622 | + aliases: [root servers, root-servers.org, rssac, dns root] | |
| 623 | + discover: { rss: false } | |
| 624 | + sensors: | |
| 625 | + - { name: root servers overview, url: "https://root-servers.org/", type: HTML, connector: http, tier: C } | |
| 626 | + - id: iana | |
| 627 | + extend: true | |
| 628 | + country: INT | |
| 629 | + language: en | |
| 630 | + sensors: | |
| 631 | + - { name: root hints file, url: "https://www.internic.net/domain/named.root", type: FILE, connector: http, tier: C } | |
| 632 | + - { name: dnssec root trust anchors, url: "https://data.iana.org/root-anchors/root-anchors.xml", type: XML, connector: http, tier: C } | |
| 633 | + - { name: root servers list, url: "https://www.iana.org/domains/root/servers", type: HTML, connector: http, tier: C } | |
| 634 | + - id: nlnet-labs | |
| 635 | + name: NLnet Labs | |
| 636 | + domain: nlnetlabs.nl | |
| 637 | + categories: [open-source, internet, infrastructure] | |
| 638 | + tier: B | |
| 639 | + country: NL | |
| 640 | + language: en | |
| 641 | + aliases: [nlnet labs, nlnetlabs] | |
| 642 | + products: | |
| 643 | + - { name: Unbound, type: software } | |
| 644 | + - { name: NSD, type: software } | |
| 645 | + - { name: Routinator, type: software } | |
| 646 | + - { name: Krill, type: software } | |
| 647 | + discover: { rss: true } | |
| 648 | + sensors: | |
| 649 | + - { name: blog feed, url: "https://blog.nlnetlabs.nl/rss/", type: RSS, connector: rss, tier: B } | |
| 650 | + - { name: unbound releases, url: "https://github.com/NLnetLabs/unbound/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: NLnetLabs/unbound, kind: releases } } | |
| 651 | + - { name: nsd releases, url: "https://github.com/NLnetLabs/nsd/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: NLnetLabs/nsd, kind: releases } } | |
| 652 | + - { name: routinator releases, url: "https://github.com/NLnetLabs/routinator/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: NLnetLabs/routinator, kind: releases } } | |
| 653 | + - { name: krill releases, url: "https://github.com/NLnetLabs/krill/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: NLnetLabs/krill, kind: releases } } | |
| 654 | + - id: isc | |
| 655 | + name: Internet Systems Consortium | |
| 656 | + domain: isc.org | |
| 657 | + homepage: https://www.isc.org | |
| 658 | + categories: [open-source, internet, infrastructure] | |
| 659 | + tier: B | |
| 660 | + weight: 1.1 | |
| 661 | + country: US | |
| 662 | + language: en | |
| 663 | + aliases: [isc, internet systems consortium, bind] | |
| 664 | + products: | |
| 665 | + - { name: BIND 9, type: software, aliases: [bind, bind9, named] } | |
| 666 | + - { name: Kea DHCP, type: software, aliases: [kea] } | |
| 667 | + discover: { rss: true } | |
| 668 | + sensors: | |
| 669 | + - { name: bind 9 tags (gitlab), url: "https://gitlab.isc.org/isc-projects/bind9/-/tags?format=atom", type: ATOM, connector: rss, tier: A } | |
| 670 | + - { name: bind 9 releases (github mirror), url: "https://github.com/isc-projects/bind9/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: isc-projects/bind9, kind: releases } } | |
| 671 | + - { name: kea tags, url: "https://github.com/isc-projects/kea/tags.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: isc-projects/kea, kind: tags } } | |
| 672 | + notes: "isc.org and kb.isc.org publish no RSS (404)." | |
| 673 | + # ───────────────────────── certificate authorities ───────────────────────── | |
| 674 | + - id: letsencrypt | |
| 675 | + extend: true | |
| 676 | + country: US | |
| 677 | + language: en | |
| 678 | + sensors: | |
| 679 | + - { name: status (status.io), url: "https://letsencrypt.status.io/1.0/status/55957a99e800baa4470002da", type: STATUSPAGE, connector: statusjson, tier: S, config: { flavor: statusio } } | |
| 680 | + - { name: api announcements feed, url: "https://community.letsencrypt.org/c/api-announcements.rss", type: RSS, connector: rss, tier: A } | |
| 681 | + - { name: rate limits, url: "https://letsencrypt.org/docs/rate-limits/", type: HTML, connector: http, tier: C } | |
| 682 | + - { name: boulder releases, url: "https://github.com/letsencrypt/boulder/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: letsencrypt/boulder, kind: releases } } | |
| 683 | + - id: tailscale | |
| 684 | + extend: true | |
| 685 | + country: CA | |
| 686 | + language: en | |
| 687 | + sensors: | |
| 688 | + - { name: changelog feed, url: "https://tailscale.com/changelog/index.xml", type: RSS, connector: rss, tier: A } | |
| 689 | + - { name: security bulletins feed, url: "https://tailscale.com/security-bulletins/index.xml", type: RSS, connector: rss, tier: S } | |
| 690 | + - { name: pricing, url: "https://tailscale.com/pricing", type: HTML, connector: http, tier: C } | |
| 691 | + - { name: tailscale releases, url: "https://github.com/tailscale/tailscale/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: tailscale/tailscale, kind: releases } } | |
| 692 | + - id: 1password | |
| 693 | + extend: true | |
| 694 | + country: CA | |
| 695 | + language: en | |
| 696 | + sensors: | |
| 697 | + - { name: release notes feed, url: "https://releases.1password.com/index.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 698 | + - { name: pricing, url: "https://1password.com/pricing", type: HTML, connector: http, tier: C } | |
| 699 | + - id: okta | |
| 700 | + extend: true | |
| 701 | + country: US | |
| 702 | + language: en | |
| 703 | + sensors: | |
| 704 | + - { name: developer feed, url: "https://developer.okta.com/feed.xml", type: RSS, connector: rss, tier: B } | |
| 705 | + - { name: developer release notes 2026, url: "https://developer.okta.com/docs/release-notes/2026/", type: HTML, connector: http, tier: B } | |
| 706 | + notes: "status.okta.com/api/v2 requires auth (401) — custom page." | |
| 707 | + - id: auth0 | |
| 708 | + extend: true | |
| 709 | + country: US | |
| 710 | + language: en | |
| 711 | + sensors: | |
| 712 | + - { name: pricing, url: "https://auth0.com/pricing", type: HTML, connector: http, tier: C } | |
| 713 | + # ───────────────────────── communications APIs ───────────────────────── | |
| 714 | + - id: sendgrid | |
| 715 | + name: Twilio SendGrid | |
| 716 | + domain: sendgrid.com | |
| 717 | + categories: [enterprise, developer, cloud] | |
| 718 | + tier: B | |
| 719 | + country: US | |
| 720 | + language: en | |
| 721 | + aliases: [sendgrid, twilio sendgrid] | |
| 722 | + discover: { status: true } | |
| 723 | + sensors: | |
| 724 | + - { name: status, url: "https://status.sendgrid.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 725 | + notes: "sendgrid.com blog feed URLs return HTML; docs changelog 404 (merged into twilio.com changelog)." | |
| 726 | + - id: resend | |
| 727 | + extend: true | |
| 728 | + country: US | |
| 729 | + language: en | |
| 730 | + sensors: | |
| 731 | + - { name: status (incident.io), url: "https://resend-status.com/api/v1/summary", type: STATUSPAGE, connector: statusjson, tier: S, config: { flavor: incidentio } } | |
| 732 | + - id: datadog | |
| 733 | + extend: true | |
| 734 | + country: US | |
| 735 | + language: en | |
| 736 | + notes: "github.com/DataDog/datadog-agent releases.atom and tags.atom return 502/504 (feed too large for GitHub to render) — not a sensor." | |
| 737 | + sensors: | |
| 738 | + - { name: pricing, url: "https://www.datadoghq.com/pricing/", type: HTML, connector: http, tier: C } | |
| 739 | + - { name: api v2 openapi, url: "https://raw.githubusercontent.com/DataDog/datadog-api-client-python/master/.generator/schemas/v2/openapi.yaml", type: JSON, connector: openapi, tier: B, config: { docsUrl: "https://docs.datadoghq.com/api/latest/" } } | |
| 740 | + - id: new-relic | |
| 741 | + extend: true | |
| 742 | + country: US | |
| 743 | + language: en | |
| 744 | + sensors: | |
| 745 | + - { name: whats new feed, url: "https://docs.newrelic.com/whats-new/feed.xml", type: RSS, connector: rss, tier: B } | |
| 746 | + - { name: pricing, url: "https://newrelic.com/pricing", type: HTML, connector: http, tier: C } | |
| 747 | + - { name: infrastructure agent releases, url: "https://github.com/newrelic/infrastructure-agent/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: newrelic/infrastructure-agent, kind: releases } } | |
| 748 | + - id: grafana | |
| 749 | + extend: true | |
| 750 | + country: US | |
| 751 | + language: en | |
| 752 | + products: | |
| 753 | + - { name: Grafana Loki, type: software, aliases: [loki] } | |
| 754 | + - { name: Grafana Tempo, type: software, aliases: [tempo] } | |
| 755 | + - { name: Grafana Mimir, type: software, aliases: [mimir] } | |
| 756 | + - { name: Grafana Alloy, type: software, aliases: [alloy] } | |
| 757 | + sensors: | |
| 758 | + - { name: cloud status, url: "https://status.grafana.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 759 | + - { name: blog feed, url: "https://grafana.com/blog/index.xml", type: RSS, connector: rss, tier: B } | |
| 760 | + - { name: pricing, url: "https://grafana.com/pricing/", type: HTML, connector: http, tier: C } | |
| 761 | + - { name: loki releases, url: "https://github.com/grafana/loki/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: grafana/loki, kind: releases } } | |
| 762 | + - { name: tempo releases, url: "https://github.com/grafana/tempo/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: grafana/tempo, kind: releases } } | |
| 763 | + - { name: mimir releases, url: "https://github.com/grafana/mimir/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: grafana/mimir, kind: releases } } | |
| 764 | + - { name: alloy releases, url: "https://github.com/grafana/alloy/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: grafana/alloy, kind: releases } } | |
| 765 | + - id: sentry | |
| 766 | + extend: true | |
| 767 | + country: US | |
| 768 | + language: en | |
| 769 | + sensors: | |
| 770 | + - { name: blog feed, url: "https://blog.sentry.io/feed.xml", type: RSS, connector: rss, tier: B } | |
| 771 | + - { name: pricing, url: "https://sentry.io/pricing/", type: HTML, connector: http, tier: C } | |
| 772 | + - { name: sentry releases, url: "https://github.com/getsentry/sentry/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: getsentry/sentry, kind: releases } } | |
| 773 | + - { name: self-hosted releases, url: "https://github.com/getsentry/self-hosted/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: getsentry/self-hosted, kind: releases } } | |
| 774 | + - id: honeycomb | |
| 775 | + name: Honeycomb | |
| 776 | + domain: honeycomb.io | |
| 777 | + homepage: https://www.honeycomb.io | |
| 778 | + categories: [enterprise, developer, cloud] | |
| 779 | + tier: B | |
| 780 | + country: US | |
| 781 | + language: en | |
| 782 | + aliases: [honeycomb, honeycomb.io] | |
| 783 | + products: | |
| 784 | + - { name: Refinery, type: software } | |
| 785 | + discover: { rss: true, status: true } | |
| 786 | + sensors: | |
| 787 | + - { name: status, url: "https://status.honeycomb.io/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 788 | + - { name: pricing, url: "https://www.honeycomb.io/pricing", type: HTML, connector: http, tier: C } | |
| 789 | + - { name: refinery releases, url: "https://github.com/honeycombio/refinery/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: honeycombio/refinery, kind: releases } } | |
| 790 | + notes: "honeycomb.io blog has no feed; docs changelog 404." | |
| 791 | + # ───────────────────────── developer platforms / registries ───────────────────────── | |
| 792 | + - id: atlassian | |
| 793 | + extend: true | |
| 794 | + country: AU | |
| 795 | + language: en | |
| 796 | + products: | |
| 797 | + - { name: Bitbucket, type: product, aliases: [bitbucket cloud] } | |
| 798 | + sensors: | |
| 799 | + - { name: bitbucket status, url: "https://bitbucket.status.atlassian.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 800 | + - { name: bitbucket api openapi, url: "https://api.bitbucket.org/swagger.json", type: JSON, connector: openapi, tier: B, config: { docsUrl: "https://developer.atlassian.com/cloud/bitbucket/rest/" } } | |
| 801 | + - { name: bitbucket cloud changelog, url: "https://developer.atlassian.com/cloud/bitbucket/changelog/", type: HTML, connector: http, tier: B } | |
| 802 | + - { name: jira platform changelog, url: "https://developer.atlassian.com/cloud/jira/platform/changelog/", type: HTML, connector: http, tier: B } | |
| 803 | + - id: github | |
| 804 | + extend: true | |
| 805 | + country: US | |
| 806 | + language: en | |
| 807 | + sensors: | |
| 808 | + - { name: engineering blog feed, url: "https://github.blog/engineering/feed/", type: RSS, connector: rss, tier: B } | |
| 809 | + - { name: actions changelog feed, url: "https://github.blog/changelog/label/actions/feed/", type: RSS, connector: rss, tier: A } | |
| 810 | + - { name: rest api breaking changes, url: "https://docs.github.com/en/rest/overview/breaking-changes", type: HTML, connector: http, tier: B } | |
| 811 | + - { name: meta ip ranges api, url: "https://api.github.com/meta", type: JSON, connector: http, tier: C } | |
| 812 | + - { name: gh cli releases, url: "https://github.com/cli/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cli/cli, kind: releases } } | |
| 813 | + - id: gitlab | |
| 814 | + extend: true | |
| 815 | + country: US | |
| 816 | + language: en | |
| 817 | + sensors: | |
| 818 | + - { name: status (status.io), url: "https://api.status.io/1.0/status/5b36dc6502d06804c08349f7", type: STATUSPAGE, connector: statusjson, tier: S, config: { flavor: statusio } } | |
| 819 | + - { name: status history feed, url: "https://status.gitlab.com/pages/5b36dc6502d06804c08349f7/rss", type: RSS, connector: rss, tier: A } | |
| 820 | + - id: npm | |
| 821 | + name: npm Registry | |
| 822 | + domain: npmjs.com | |
| 823 | + homepage: https://www.npmjs.com | |
| 824 | + categories: [packages, developer, infrastructure] | |
| 825 | + tier: A | |
| 826 | + weight: 1.3 | |
| 827 | + country: US | |
| 828 | + language: en | |
| 829 | + aliases: [npm, npmjs, npm registry] | |
| 830 | + products: | |
| 831 | + - { name: npm CLI, type: software } | |
| 832 | + discover: { status: true } | |
| 833 | + sensors: | |
| 834 | + - { name: status, url: "https://status.npmjs.org/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 835 | + - { name: npm changelog feed (github), url: "https://github.blog/changelog/label/npm/feed/", type: RSS, connector: rss, tier: A } | |
| 836 | + - { name: npm cli releases, url: "https://github.com/npm/cli/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: npm/cli, kind: releases } } | |
| 837 | + notes: "blog.npmjs.org is gone; www.npmjs.com/blog is Cloudflare-blocked (403)." | |
| 838 | + - id: pypi | |
| 839 | + name: PyPI | |
| 840 | + domain: pypi.org | |
| 841 | + categories: [packages, developer, infrastructure] | |
| 842 | + tier: A | |
| 843 | + weight: 1.3 | |
| 844 | + country: US | |
| 845 | + language: en | |
| 846 | + aliases: [pypi, python package index, warehouse] | |
| 847 | + products: | |
| 848 | + - { name: Warehouse, type: software } | |
| 849 | + discover: { status: true } | |
| 850 | + sensors: | |
| 851 | + - { name: status (python infrastructure), url: "https://status.python.org/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S } | |
| 852 | + - { name: blog feed, url: "https://blog.pypi.org/feed_rss_created.xml", type: RSS, connector: rss, tier: B } | |
| 853 | + - { name: warehouse commits, url: "https://github.com/pypi/warehouse/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: pypi/warehouse, kind: commits, branch: main } } | |
| 854 | + notes: "pypi.org/rss/updates.xml and packages.xml are firehoses of every release — deliberately not added." | |
| 855 | + # ───────────────────────── infrastructure software / foundations ───────────────────────── | |
| 856 | + - id: kubernetes | |
| 857 | + extend: true | |
| 858 | + country: US | |
| 859 | + language: en | |
| 860 | + sensors: | |
| 861 | + - { name: release schedule, url: "https://kubernetes.io/releases/", type: HTML, connector: http, tier: B } | |
| 862 | + - { name: api deprecation guide, url: "https://kubernetes.io/docs/reference/using-api/deprecation-guide/", type: HTML, connector: http, tier: C } | |
| 863 | + - { name: ingress-nginx releases, url: "https://github.com/kubernetes/ingress-nginx/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: kubernetes/ingress-nginx, kind: releases } } | |
| 864 | + - { name: cluster autoscaler releases, url: "https://github.com/kubernetes/autoscaler/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: kubernetes/autoscaler, kind: releases } } | |
| 865 | + - { name: kubespray releases, url: "https://github.com/kubernetes-sigs/kubespray/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: kubernetes-sigs/kubespray, kind: releases } } | |
| 866 | + - id: cncf | |
| 867 | + extend: true | |
| 868 | + country: US | |
| 869 | + language: en | |
| 870 | + products: | |
| 871 | + - { name: containerd, type: software } | |
| 872 | + - { name: etcd, type: software } | |
| 873 | + - { name: CoreDNS, type: software, aliases: [coredns] } | |
| 874 | + - { name: Cilium, type: software } | |
| 875 | + - { name: Linkerd, type: software } | |
| 876 | + - { name: Flux, type: software, aliases: [fluxcd] } | |
| 877 | + - { name: Crossplane, type: software } | |
| 878 | + - { name: cert-manager, type: software } | |
| 879 | + - { name: OpenTelemetry Collector, type: software, aliases: [otel collector] } | |
| 880 | + sensors: | |
| 881 | + - { name: announcements feed, url: "https://www.cncf.io/announcements/feed/", type: RSS, connector: rss, tier: B } | |
| 882 | + - { name: containerd releases, url: "https://github.com/containerd/containerd/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: containerd/containerd, kind: releases } } | |
| 883 | + - { name: etcd releases, url: "https://github.com/etcd-io/etcd/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: etcd-io/etcd, kind: releases } } | |
| 884 | + - { name: coredns releases, url: "https://github.com/coredns/coredns/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: coredns/coredns, kind: releases } } | |
| 885 | + - { name: cilium releases, url: "https://github.com/cilium/cilium/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cilium/cilium, kind: releases } } | |
| 886 | + - { name: linkerd releases, url: "https://github.com/linkerd/linkerd2/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: linkerd/linkerd2, kind: releases } } | |
| 887 | + - { name: flux releases, url: "https://github.com/fluxcd/flux2/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: fluxcd/flux2, kind: releases } } | |
| 888 | + - { name: crossplane releases, url: "https://github.com/crossplane/crossplane/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: crossplane/crossplane, kind: releases } } | |
| 889 | + - { name: cert-manager releases, url: "https://github.com/cert-manager/cert-manager/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cert-manager/cert-manager, kind: releases } } | |
| 890 | + - { name: external-secrets releases, url: "https://github.com/external-secrets/external-secrets/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: external-secrets/external-secrets, kind: releases } } | |
| 891 | + - { name: opentelemetry collector releases, url: "https://github.com/open-telemetry/opentelemetry-collector/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: open-telemetry/opentelemetry-collector, kind: releases } } | |
| 892 | + - { name: calico releases, url: "https://github.com/projectcalico/calico/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: projectcalico/calico, kind: releases } } | |
| 893 | + - id: hashicorp | |
| 894 | + extend: true | |
| 895 | + country: US | |
| 896 | + language: en | |
| 897 | + products: | |
| 898 | + - { name: Consul, type: software } | |
| 899 | + - { name: Nomad, type: software } | |
| 900 | + - { name: Packer, type: software } | |
| 901 | + - { name: Terraform Registry, type: service } | |
| 902 | + sensors: | |
| 903 | + - { name: blog feed, url: "https://www.hashicorp.com/blog/feed.xml", type: RSS, connector: rss, tier: B } | |
| 904 | + - { name: consul releases, url: "https://github.com/hashicorp/consul/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: hashicorp/consul, kind: releases } } | |
| 905 | + - { name: nomad releases, url: "https://github.com/hashicorp/nomad/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: hashicorp/nomad, kind: releases } } | |
| 906 | + - { name: packer releases, url: "https://github.com/hashicorp/packer/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: hashicorp/packer, kind: releases } } | |
| 907 | + - { name: terraform provider aws releases, url: "https://github.com/hashicorp/terraform-provider-aws/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: hashicorp/terraform-provider-aws, kind: releases } } | |
| 908 | + - { name: terraform provider azurerm releases, url: "https://github.com/hashicorp/terraform-provider-azurerm/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: hashicorp/terraform-provider-azurerm, kind: releases } } | |
| 909 | + - { name: terraform provider google releases, url: "https://github.com/hashicorp/terraform-provider-google/releases.atom", type: GITHUB_RELEASE, connector: github, tier: A, config: { repo: hashicorp/terraform-provider-google, kind: releases } } | |
| 910 | + - { name: registry provider aws, url: "https://registry.terraform.io/v1/providers/hashicorp/aws", type: JSON, connector: http, tier: C, config: { ignoreKeys: [downloads] } } | |
| 911 | + - { name: registry provider azurerm, url: "https://registry.terraform.io/v1/providers/hashicorp/azurerm", type: JSON, connector: http, tier: C, config: { ignoreKeys: [downloads] } } | |
| 912 | + - { name: registry provider google, url: "https://registry.terraform.io/v1/providers/hashicorp/google", type: JSON, connector: http, tier: C, config: { ignoreKeys: [downloads] } } | |
| 913 | + - id: pulumi | |
| 914 | + extend: true | |
| 915 | + country: US | |
| 916 | + language: en | |
| 917 | + sensors: | |
| 918 | + - { name: pulumi-aws releases, url: "https://github.com/pulumi/pulumi-aws/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: pulumi/pulumi-aws, kind: releases } } | |
| 919 | + - id: traefik | |
| 920 | + name: Traefik Labs | |
| 921 | + domain: traefik.io | |
| 922 | + categories: [open-source, infrastructure] | |
| 923 | + tier: B | |
| 924 | + country: FR | |
| 925 | + language: en | |
| 926 | + aliases: [traefik, traefik labs, containous] | |
| 927 | + products: | |
| 928 | + - { name: Traefik Proxy, type: software, aliases: [traefik] } | |
| 929 | + discover: { rss: true } | |
| 930 | + sensors: | |
| 931 | + - { name: traefik releases, url: "https://github.com/traefik/traefik/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: traefik/traefik, kind: releases } } | |
| 932 | + - id: minio | |
| 933 | + name: MinIO | |
| 934 | + domain: min.io | |
| 935 | + categories: [open-source, infrastructure, cloud] | |
| 936 | + tier: B | |
| 937 | + country: US | |
| 938 | + language: en | |
| 939 | + aliases: [minio] | |
| 940 | + discover: { rss: true } | |
| 941 | + sensors: | |
| 942 | + - { name: minio releases, url: "https://github.com/minio/minio/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: minio/minio, kind: releases } } | |
| 943 | + - id: rancher | |
| 944 | + name: Rancher (SUSE) | |
| 945 | + domain: rancher.com | |
| 946 | + categories: [open-source, infrastructure] | |
| 947 | + tier: B | |
| 948 | + country: US | |
| 949 | + language: en | |
| 950 | + aliases: [rancher, k3s, suse rancher] | |
| 951 | + products: | |
| 952 | + - { name: K3s, type: software, aliases: [k3s] } | |
| 953 | + discover: { rss: true } | |
| 954 | + sensors: | |
| 955 | + - { name: rancher releases, url: "https://github.com/rancher/rancher/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: rancher/rancher, kind: releases } } | |
| 956 | + - { name: k3s releases, url: "https://github.com/k3s-io/k3s/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: k3s-io/k3s, kind: releases } } | |
| 957 | + - id: frrouting | |
| 958 | + name: FRRouting | |
| 959 | + domain: frrouting.org | |
| 960 | + categories: [open-source, infrastructure, internet] | |
| 961 | + tier: C | |
| 962 | + country: US | |
| 963 | + language: en | |
| 964 | + aliases: [frr, frrouting] | |
| 965 | + discover: { rss: false } | |
| 966 | + sensors: | |
| 967 | + - { name: frr releases, url: "https://github.com/FRRouting/frr/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: FRRouting/frr, kind: releases } } | |
| 968 | + - id: powerdns | |
| 969 | + name: PowerDNS | |
| 970 | + domain: powerdns.com | |
| 971 | + homepage: https://www.powerdns.com | |
| 972 | + categories: [open-source, infrastructure, internet] | |
| 973 | + tier: C | |
| 974 | + country: NL | |
| 975 | + language: en | |
| 976 | + aliases: [powerdns, pdns] | |
| 977 | + discover: { rss: true } | |
| 978 | + sensors: | |
| 979 | + - { name: pdns releases, url: "https://github.com/PowerDNS/pdns/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: PowerDNS/pdns, kind: releases } } | |
| 980 | + - id: victoriametrics | |
| 981 | + name: VictoriaMetrics | |
| 982 | + domain: victoriametrics.com | |
| 983 | + categories: [open-source, infrastructure] | |
| 984 | + tier: C | |
| 985 | + country: US | |
| 986 | + language: en | |
| 987 | + aliases: [victoriametrics, victoria metrics] | |
| 988 | + discover: { rss: true } | |
| 989 | + sensors: | |
| 990 | + - { name: releases, url: "https://github.com/VictoriaMetrics/VictoriaMetrics/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: VictoriaMetrics/VictoriaMetrics, kind: releases } } | |
| 991 | + - id: prometheus | |
| 992 | + extend: true | |
| 993 | + sensors: | |
| 994 | + - { name: alertmanager releases, url: "https://github.com/prometheus/alertmanager/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: prometheus/alertmanager, kind: releases } } | |
added
config/sources.d/42-cybersecurity.yaml
+1770 −0
@@ -0,0 +1,1770 @@ | ||
| 1 | +# config/sources.d/42-cybersecurity.yaml — cybersecurity coverage (2026-09-11): CVE/KEV/EPSS feeds, vendor PSIRT | |
| 2 | +# advisories, national CERTs, exploit & ransomware trackers, threat-intel research blogs, supply-chain / dependency | |
| 3 | +# vulnerability databases, security tooling releases and security media. Extends the founding cyber block (cisa, nvd, | |
| 4 | +# nist, cert-cc, mitre, microsoft-security, …) and earlier fragments; every URL below was fetched and parsed on | |
| 5 | +# 2026-09-11. Blocked / JS-only pages (Oracle CPU, Adobe PSIRT, Broadcom/VMware advisories, Citrix bulletins, SonicWall | |
| 6 | +# PSIRT, Juniper, F5 MyF5, Akamai blog RSS, Trellix, BSI RSS, ENISA news RSS, ACSC, Shadowserver, Packet Storm TOS wall, | |
| 7 | +# CVE.org list API, OSV POST API, VulnCheck auth) are recorded in `notes:` instead of pretending. | |
| 8 | +sources: | |
| 9 | + # ───────────────────────── A · US government, CVE ecosystem, scoring ───────────────────────── | |
| 10 | + - id: cisa | |
| 11 | + extend: true | |
| 12 | + country: US | |
| 13 | + aliases: [us-cert, ics-cert, cisa kev] | |
| 14 | + products: | |
| 15 | + - { name: ICS Advisories, type: service, aliases: [ics-cert advisories, icsa] } | |
| 16 | + - { name: Vulnrichment, type: dataset, aliases: [cisa vulnrichment, ssvc] } | |
| 17 | + sensors: | |
| 18 | + - { name: ics advisories feed, url: "https://www.cisa.gov/cybersecurity-advisories/ics-advisories.xml", type: RSS, connector: rss, tier: A, config: { maxItems: 60 } } | |
| 19 | + - { name: ics medical advisories feed, url: "https://www.cisa.gov/cybersecurity-advisories/ics-medical-advisories.xml", type: RSS, connector: rss, tier: B } | |
| 20 | + - { name: alerts feed, url: "https://www.cisa.gov/cybersecurity-advisories/alerts.xml", type: RSS, connector: rss, tier: S } | |
| 21 | + - { name: cybersecurity advisories feed, url: "https://www.cisa.gov/cybersecurity-advisories/cybersecurity-advisories.xml", type: RSS, connector: rss, tier: A } | |
| 22 | + - { name: analysis reports feed, url: "https://www.cisa.gov/cybersecurity-advisories/analysis-reports.xml", type: RSS, connector: rss, tier: B } | |
| 23 | + - { name: blog feed, url: "https://www.cisa.gov/blog.xml", type: RSS, connector: rss, tier: C } | |
| 24 | + - { name: vulnrichment commits, url: "https://github.com/cisagov/vulnrichment/commits/develop.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: cisagov/vulnrichment, kind: commits, branch: develop } } | |
| 25 | + - { name: scubagear releases, url: "https://github.com/cisagov/ScubaGear/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: cisagov/ScubaGear, kind: releases } } | |
| 26 | + - id: nvd | |
| 27 | + extend: true | |
| 28 | + country: US | |
| 29 | + llm: false | |
| 30 | + sensors: | |
| 31 | + - name: newly published cves | |
| 32 | + url: "https://services.nvd.nist.gov/rest/json/cves/2.0?pubStartDate={now-2h}&pubEndDate={now}&resultsPerPage=200" | |
| 33 | + type: REST_API | |
| 34 | + connector: jsonlist | |
| 35 | + tier: A | |
| 36 | + interval: 900 | |
| 37 | + config: { itemsPath: vulnerabilities, keyField: cve.id, titleTemplate: "{cve.id} — {cve.vulnStatus} ({cve.sourceIdentifier})", summaryField: cve.descriptions, dateField: cve.published, urlTemplate: "https://nvd.nist.gov/vuln/detail/{key}", compareFields: [cve.vulnStatus, cve.metrics], noConditional: true, maxItems: 200 } | |
| 38 | + - id: nist | |
| 39 | + extend: true | |
| 40 | + country: US | |
| 41 | + sensors: | |
| 42 | + - { name: cybersecurity insights blog, url: "https://www.nist.gov/blogs/cybersecurity-insights/rss.xml", type: RSS, connector: rss, tier: C } | |
| 43 | + - id: cert-cc | |
| 44 | + extend: true | |
| 45 | + country: US | |
| 46 | + sensors: | |
| 47 | + - { name: vulnerability notes atom, url: "https://www.kb.cert.org/vuls/atomfeed/", type: ATOM, connector: rss, tier: A } | |
| 48 | + - id: mitre | |
| 49 | + extend: true | |
| 50 | + country: US | |
| 51 | + products: | |
| 52 | + - { name: CVE Program, type: service, aliases: [cve.org, cve list, cna] } | |
| 53 | + - { name: ATT&CK, type: dataset, aliases: [mitre att&ck, attack framework] } | |
| 54 | + - { name: CWE, type: dataset, aliases: [common weakness enumeration] } | |
| 55 | + notes: "CVE.org list API (cveawg.mitre.org/api/cve-id) requires CNA credentials → not a sensor; mitre.org RSS returns 403 (Akamai)." | |
| 56 | + sensors: | |
| 57 | + - { name: cve program blog, url: "https://medium.com/feed/@cve_program", type: RSS, connector: rss, tier: C } | |
| 58 | + - { name: attack blog, url: "https://medium.com/feed/mitre-attack", type: RSS, connector: rss, tier: C } | |
| 59 | + - { name: cti stix releases, url: "https://github.com/mitre/cti/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: mitre/cti, kind: releases } } | |
| 60 | + - { name: cve schema releases, url: "https://github.com/CVEProject/cve-schema/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: CVEProject/cve-schema, kind: releases } } | |
| 61 | + - { name: attack navigator releases, url: "https://github.com/mitre-attack/attack-navigator/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: mitre-attack/attack-navigator, kind: releases } } | |
| 62 | + - { name: attack flow releases, url: "https://github.com/center-for-threat-informed-defense/attack-flow/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: center-for-threat-informed-defense/attack-flow, kind: releases } } | |
| 63 | + - id: first-org | |
| 64 | + extend: true | |
| 65 | + country: US | |
| 66 | + products: | |
| 67 | + - { name: EPSS, type: dataset, aliases: [exploit prediction scoring system, epss score] } | |
| 68 | + sensors: | |
| 69 | + - { name: epss scores updated today, url: "https://api.first.org/data/v1/epss?days=1&limit=100", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: data, keyField: cve, titleTemplate: "{cve} — EPSS {epss} (percentile {percentile})", dateField: date, urlTemplate: "https://nvd.nist.gov/vuln/detail/{key}", compareFields: [epss, percentile], maxItems: 100 } } | |
| 70 | + - { name: epss top scores, url: "https://api.first.org/data/v1/epss?epss-gt=0.9&order=!date&limit=100", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: data, keyField: cve, titleTemplate: "{cve} — EPSS {epss}", dateField: date, urlTemplate: "https://nvd.nist.gov/vuln/detail/{key}", compareFields: [epss], maxItems: 100 } } | |
| 71 | + - { name: press releases, url: "https://www.first.org/newsroom/releases/rss.xml", type: RSS, connector: rss, tier: C } | |
| 72 | + - id: ic3 | |
| 73 | + name: FBI Internet Crime Complaint Center | |
| 74 | + domain: ic3.gov | |
| 75 | + homepage: https://www.ic3.gov | |
| 76 | + categories: [cyber, government] | |
| 77 | + tier: A | |
| 78 | + country: US | |
| 79 | + aliases: [ic3, fbi ic3, internet crime complaint center] | |
| 80 | + discover: { rss: false } | |
| 81 | + sensors: | |
| 82 | + - { name: public service announcements, url: "https://www.ic3.gov/PSA/RSS", type: RSS, connector: rss, tier: A } | |
| 83 | + - { name: cybersecurity advisories, url: "https://www.ic3.gov/CSA/RSS", type: RSS, connector: rss, tier: A } | |
| 84 | + - id: circl | |
| 85 | + name: CIRCL (Computer Incident Response Center Luxembourg) | |
| 86 | + domain: circl.lu | |
| 87 | + homepage: https://www.circl.lu | |
| 88 | + categories: [cyber, government] | |
| 89 | + tier: B | |
| 90 | + country: LU | |
| 91 | + aliases: [circl, vulnerability-lookup, cve-search] | |
| 92 | + products: | |
| 93 | + - { name: Vulnerability-Lookup, type: service, aliases: [vulnerability.circl.lu] } | |
| 94 | + - { name: MISP, type: software, aliases: [misp project, malware information sharing platform] } | |
| 95 | + discover: { rss: false } | |
| 96 | + sensors: | |
| 97 | + - { name: news feed, url: "https://www.circl.lu/rss.xml", type: RSS, connector: rss, tier: C } | |
| 98 | + - { name: latest vulnerabilities (vulnerability-lookup), url: "https://cve.circl.lu/api/last/30", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "", keyField: id, titleTemplate: "{id}", summaryField: details, dateField: published, urlTemplate: "https://vulnerability.circl.lu/vuln/{key}", compareFields: [modified], maxItems: 30 } } | |
| 99 | + - { name: misp releases, url: "https://github.com/MISP/MISP/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: MISP/MISP, kind: releases } } | |
| 100 | + | |
| 101 | + # ───────────────────────── B · Platform vendors: advisories & security blogs ───────────────────────── | |
| 102 | + - id: microsoft-security | |
| 103 | + extend: true | |
| 104 | + country: US | |
| 105 | + products: | |
| 106 | + - { name: Security Update Guide, type: service, aliases: [msrc update guide, patch tuesday, msrc cvrf] } | |
| 107 | + - { name: Microsoft Threat Intelligence, type: service, aliases: [mstic, microsoft threat intelligence center] } | |
| 108 | + sensors: | |
| 109 | + - { name: security update guide feed, url: "https://api.msrc.microsoft.com/update-guide/rss", type: RSS, connector: rss, tier: S, config: { maxItems: 200 } } | |
| 110 | + - { name: security blog feed, url: "https://www.microsoft.com/en-us/security/blog/feed/", type: RSS, connector: rss, tier: B } | |
| 111 | + - { name: threat intelligence blog feed, url: "https://www.microsoft.com/en-us/security/blog/topic/threat-intelligence/feed/", type: RSS, connector: rss, tier: A } | |
| 112 | + - id: apple-security | |
| 113 | + extend: true | |
| 114 | + country: US | |
| 115 | + sensors: | |
| 116 | + - { name: security research blog, url: "https://security.apple.com/blog/", type: HTML, connector: http, tier: C } | |
| 117 | + - id: android | |
| 118 | + name: Android Security | |
| 119 | + domain: source.android.com | |
| 120 | + homepage: https://source.android.com/docs/security | |
| 121 | + categories: [cyber, consumer-tech] | |
| 122 | + tier: A | |
| 123 | + country: US | |
| 124 | + aliases: [android security bulletin, android open source project security, aosp security] | |
| 125 | + discover: { rss: false } | |
| 126 | + sensors: | |
| 127 | + - { name: security bulletins index, url: "https://source.android.com/docs/security/bulletin", type: HTML, connector: http, tier: A } | |
| 128 | + - id: chromium | |
| 129 | + extend: true | |
| 130 | + country: US | |
| 131 | + sensors: | |
| 132 | + - { name: chrome stable channel updates, url: "https://chromereleases.googleblog.com/feeds/posts/default/-/Stable%20updates", type: ATOM, connector: rss, tier: A } | |
| 133 | + - id: google | |
| 134 | + extend: true | |
| 135 | + sensors: | |
| 136 | + - { name: safety & security blog, url: "https://blog.google/technology/safety-security/rss/", type: RSS, connector: rss, tier: B } | |
| 137 | + - id: google-cloud | |
| 138 | + extend: true | |
| 139 | + products: | |
| 140 | + - { name: Google Cloud Security Bulletins, type: service, aliases: [gcp security bulletins, gke security bulletins] } | |
| 141 | + sensors: | |
| 142 | + - { name: security bulletins feed, url: "https://cloud.google.com/feeds/google-cloud-security-bulletins.xml", type: ATOM, connector: rss, tier: A } | |
| 143 | + - { name: gke security bulletins feed, url: "https://cloud.google.com/feeds/gke-security-bulletins.xml", type: ATOM, connector: rss, tier: A } | |
| 144 | + - { name: identity & security blog, url: "https://cloudblog.withgoogle.com/products/identity-security/rss/", type: RSS, connector: rss, tier: C } | |
| 145 | + - id: aws | |
| 146 | + extend: true | |
| 147 | + sensors: | |
| 148 | + - { name: security blog feed, url: "https://aws.amazon.com/blogs/security/feed/", type: RSS, connector: rss, tier: B } | |
| 149 | + - id: cisco-security | |
| 150 | + extend: true | |
| 151 | + country: US | |
| 152 | + products: | |
| 153 | + - { name: Cisco Talos, type: service, aliases: [talos, talos intelligence] } | |
| 154 | + sensors: | |
| 155 | + - { name: event responses feed, url: "https://sec.cloudapps.cisco.com/security/center/eventResponses_20.xml", type: RSS, connector: rss, tier: A } | |
| 156 | + - { name: talos blog feed, url: "https://blog.talosintelligence.com/rss/", type: RSS, connector: rss, tier: B } | |
| 157 | + - { name: cisco security blog feed, url: "https://blogs.cisco.com/security/feed", type: RSS, connector: rss, tier: C } | |
| 158 | + - id: palo-alto-networks | |
| 159 | + extend: true | |
| 160 | + country: US | |
| 161 | + sensors: | |
| 162 | + - { name: corporate blog feed, url: "https://www.paloaltonetworks.com/blog/feed/", type: RSS, connector: rss, tier: C } | |
| 163 | + - id: fortinet | |
| 164 | + extend: true | |
| 165 | + country: US | |
| 166 | + products: | |
| 167 | + - { name: FortiGuard Outbreak Alerts, type: service, aliases: [outbreak alerts] } | |
| 168 | + notes: "fortinet.com blog RSS paths return 404; feeds.fortinet.com does not resolve." | |
| 169 | + sensors: | |
| 170 | + - { name: outbreak alerts feed, url: "https://filestore.fortinet.com/fortiguard/rss/outbreakalert.xml", type: RSS, connector: rss, tier: A } | |
| 171 | + - id: sentinelone | |
| 172 | + extend: true | |
| 173 | + country: US | |
| 174 | + products: | |
| 175 | + - { name: SentinelLabs, type: service, aliases: [sentinel labs] } | |
| 176 | + sensors: | |
| 177 | + - { name: sentinellabs feed, url: "https://www.sentinelone.com/labs/feed/", type: RSS, connector: rss, tier: B } | |
| 178 | + - id: sophos | |
| 179 | + extend: true | |
| 180 | + country: GB | |
| 181 | + products: | |
| 182 | + - { name: Sophos X-Ops, type: service, aliases: [x-ops, sophoslabs] } | |
| 183 | + sensors: | |
| 184 | + - { name: threat research feed, url: "https://www.sophos.com/en-us/category/threat-research/feed", type: RSS, connector: rss, tier: B } | |
| 185 | + - id: trend-micro | |
| 186 | + extend: true | |
| 187 | + country: JP | |
| 188 | + notes: "trendmicro.com research RSS paths return 404; FeedBurner security-news feed still served." | |
| 189 | + sensors: | |
| 190 | + - { name: security news feed, url: "https://feeds.feedburner.com/TrendMicroSecurityNews", type: RSS, connector: rss, tier: B } | |
| 191 | + - id: zero-day-initiative | |
| 192 | + name: Zero Day Initiative | |
| 193 | + domain: zerodayinitiative.com | |
| 194 | + homepage: https://www.zerodayinitiative.com | |
| 195 | + categories: [cyber] | |
| 196 | + tier: A | |
| 197 | + country: US | |
| 198 | + aliases: [zdi, trend micro zdi, pwn2own] | |
| 199 | + llm: false | |
| 200 | + discover: { rss: false } | |
| 201 | + sensors: | |
| 202 | + - { name: published advisories feed, url: "https://www.zerodayinitiative.com/rss/published/", type: RSS, connector: rss, tier: A, config: { maxItems: 100 } } | |
| 203 | + - { name: upcoming advisories feed, url: "https://www.zerodayinitiative.com/rss/upcoming/", type: RSS, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 204 | + - { name: blog feed, url: "https://www.zerodayinitiative.com/blog?format=rss", type: RSS, connector: rss, tier: C } | |
| 205 | + - id: tenable | |
| 206 | + extend: true | |
| 207 | + country: US | |
| 208 | + sensors: | |
| 209 | + - { name: cyber exposure alerts feed, url: "https://www.tenable.com/blog/cyber-exposure-alerts/feed", type: RSS, connector: rss, tier: A } | |
| 210 | + - { name: product security advisories feed, url: "https://www.tenable.com/security/feed", type: RSS, connector: rss, tier: B } | |
| 211 | + - id: rapid7 | |
| 212 | + extend: true | |
| 213 | + country: US | |
| 214 | + sensors: | |
| 215 | + - { name: emergent threat response feed, url: "https://www.rapid7.com/blog/tag/emergent-threat-response/rss/", type: RSS, connector: rss, tier: A } | |
| 216 | + - id: crowdstrike | |
| 217 | + extend: true | |
| 218 | + country: US | |
| 219 | + - id: mandiant | |
| 220 | + extend: true | |
| 221 | + country: US | |
| 222 | + - id: malwarebytes | |
| 223 | + extend: true | |
| 224 | + country: US | |
| 225 | + - id: ivanti | |
| 226 | + extend: true | |
| 227 | + country: US | |
| 228 | + sensors: | |
| 229 | + - { name: security advisory blog feed, url: "https://www.ivanti.com/blog/topics/security-advisory/rss", type: RSS, connector: rss, tier: A } | |
| 230 | + - id: broadcom | |
| 231 | + extend: true | |
| 232 | + categories: [cyber] | |
| 233 | + notes: "VMware security advisories moved to support.broadcom.com (JS-rendered) → no advisories sensor; VMware security blog RSS kept." | |
| 234 | + sensors: | |
| 235 | + - { name: vmware security blog feed, url: "https://blogs.vmware.com/security/feed", type: RSS, connector: rss, tier: B } | |
| 236 | + - id: atlassian | |
| 237 | + extend: true | |
| 238 | + categories: [cyber] | |
| 239 | + sensors: | |
| 240 | + - { name: security advisories space feed, url: "https://confluence.atlassian.com/createrssfeed.action?types=page&spaces=SECURITY&title=Security+Advisories&labelString=&excludedSpaceKeys=&sort=modified&maxResults=25&timeSpan=600&showContent=true&confirm=Create+RSS+Feed", type: ATOM, connector: rss, tier: A } | |
| 241 | + - id: yubico | |
| 242 | + name: Yubico | |
| 243 | + domain: yubico.com | |
| 244 | + homepage: https://www.yubico.com | |
| 245 | + categories: [cyber, consumer-tech] | |
| 246 | + tier: B | |
| 247 | + country: SE | |
| 248 | + aliases: [yubikey] | |
| 249 | + discover: { rss: true } | |
| 250 | + sensors: | |
| 251 | + - { name: blog feed, url: "https://www.yubico.com/feed/", type: RSS, connector: rss, tier: C } | |
| 252 | + - id: jamf | |
| 253 | + name: Jamf | |
| 254 | + domain: jamf.com | |
| 255 | + homepage: https://www.jamf.com | |
| 256 | + categories: [cyber, enterprise] | |
| 257 | + tier: C | |
| 258 | + country: US | |
| 259 | + aliases: [jamf threat labs] | |
| 260 | + discover: { rss: false } | |
| 261 | + sensors: | |
| 262 | + - { name: blog feed, url: "https://www.jamf.com/blog/rss/", type: RSS, connector: rss, tier: C } | |
| 263 | + - id: hashicorp | |
| 264 | + extend: true | |
| 265 | + sensors: | |
| 266 | + - { name: security announcements forum, url: "https://discuss.hashicorp.com/c/security/52.rss", type: RSS, connector: rss, tier: A } | |
| 267 | + - id: mongodb | |
| 268 | + extend: true | |
| 269 | + categories: [cyber] | |
| 270 | + sensors: | |
| 271 | + - { name: server security issues (jira), url: "https://jira.mongodb.org/sr/jira.issueviews:searchrequest-rss/temp/SearchRequest.xml?jqlQuery=project+%3D+SERVER+AND+labels+%3D+security&tempMax=50", type: RSS, connector: rss, tier: B } | |
| 272 | + - id: postgresql | |
| 273 | + extend: true | |
| 274 | + categories: [cyber] | |
| 275 | + sensors: | |
| 276 | + - { name: security information page, url: "https://www.postgresql.org/support/security/", type: HTML, connector: http, tier: B } | |
| 277 | + | |
| 278 | + # ───────────────────────── C · Source code platforms, Linux, language ecosystems (dependency vulnerabilities) ───────────────────────── | |
| 279 | + - id: github | |
| 280 | + extend: true | |
| 281 | + categories: [cyber] | |
| 282 | + products: | |
| 283 | + - { name: GitHub Advisory Database, type: dataset, aliases: [ghsa, github advisories, advisory database] } | |
| 284 | + - { name: GitHub Security Lab, type: service, aliases: [security lab] } | |
| 285 | + sensors: | |
| 286 | + - { name: advisory database (all ecosystems), url: "https://api.github.com/advisories?per_page=100&sort=published", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, summaryField: description, dateField: published_at, compareFields: [severity, updated_at, withdrawn_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 287 | + - { name: advisories npm, url: "https://api.github.com/advisories?per_page=100&sort=published&ecosystem=npm", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, dateField: published_at, compareFields: [severity, updated_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 288 | + - { name: advisories pypi, url: "https://api.github.com/advisories?per_page=100&sort=published&ecosystem=pip", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, dateField: published_at, compareFields: [severity, updated_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 289 | + - { name: advisories maven, url: "https://api.github.com/advisories?per_page=100&sort=published&ecosystem=maven", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, dateField: published_at, compareFields: [severity, updated_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 290 | + - { name: advisories go, url: "https://api.github.com/advisories?per_page=100&sort=published&ecosystem=go", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, dateField: published_at, compareFields: [severity, updated_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 291 | + - { name: advisories rubygems, url: "https://api.github.com/advisories?per_page=100&sort=published&ecosystem=rubygems", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, dateField: published_at, compareFields: [severity, updated_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 292 | + - { name: advisories rust crates, url: "https://api.github.com/advisories?per_page=100&sort=published&ecosystem=rust", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, dateField: published_at, compareFields: [severity, updated_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 293 | + - { name: advisories composer, url: "https://api.github.com/advisories?per_page=100&sort=published&ecosystem=composer", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, dateField: published_at, compareFields: [severity, updated_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 294 | + - { name: advisories nuget, url: "https://api.github.com/advisories?per_page=100&sort=published&ecosystem=nuget", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — {severity}: {summary}", urlField: html_url, dateField: published_at, compareFields: [severity, updated_at], maxItems: 100, headers: { accept: application/vnd.github+json } } } | |
| 295 | + - { name: critical advisories, url: "https://api.github.com/advisories?per_page=50&sort=published&severity=critical", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "", keyField: ghsa_id, titleTemplate: "{ghsa_id} {cve_id} — CRITICAL: {summary}", urlField: html_url, summaryField: description, dateField: published_at, compareFields: [updated_at, withdrawn_at], maxItems: 50, headers: { accept: application/vnd.github+json } } } | |
| 296 | + - { name: security lab advisories feed, url: "https://securitylab.github.com/advisories/feed.xml", type: RSS, connector: rss, tier: B } | |
| 297 | + - { name: vulnerability research blog feed, url: "https://github.blog/security/vulnerability-research/feed/", type: RSS, connector: rss, tier: C } | |
| 298 | + - { name: advisory database commits, url: "https://github.com/github/advisory-database/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: github/advisory-database, kind: commits, branch: main } } | |
| 299 | + - id: gitlab | |
| 300 | + extend: true | |
| 301 | + categories: [cyber] | |
| 302 | + sensors: | |
| 303 | + - { name: patch & security releases feed, url: "https://docs.gitlab.com/releases/patch-releases.xml", type: ATOM, connector: rss, tier: A, config: { maxItems: 50 } } | |
| 304 | + - id: kubernetes | |
| 305 | + extend: true | |
| 306 | + categories: [cyber] | |
| 307 | + products: | |
| 308 | + - { name: Kubernetes Official CVE Feed, type: dataset, aliases: [k8s cve feed] } | |
| 309 | + sensors: | |
| 310 | + - { name: official cve feed, url: "https://kubernetes.io/docs/reference/issues-security/official-cve-feed/index.json", type: JSON, connector: jsonlist, tier: A, config: { itemsPath: items, keyField: id, titleField: summary, urlField: url, summaryField: content_text, dateField: date_published, compareFields: [status, summary], maxItems: 200 } } | |
| 311 | + - id: docker | |
| 312 | + extend: true | |
| 313 | + categories: [cyber] | |
| 314 | + sensors: | |
| 315 | + - { name: security blog category feed, url: "https://www.docker.com/blog/category/security/feed/", type: RSS, connector: rss, tier: C } | |
| 316 | + - id: red-hat | |
| 317 | + extend: true | |
| 318 | + categories: [cyber] | |
| 319 | + products: | |
| 320 | + - { name: Red Hat Product Security, type: service, aliases: [rhsa, red hat security advisories, red hat security data api] } | |
| 321 | + notes: "securitydata/cve.json returns a JSON-encoded string (double-encoded) → not parseable by jsonlist; csaf.json is a proper array." | |
| 322 | + sensors: | |
| 323 | + - { name: security advisories (csaf api), url: "https://access.redhat.com/hydra/rest/securitydata/csaf.json?per_page=100", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: RHSA, titleTemplate: "{RHSA} — {severity}: {CVEs}", urlField: resource_url, dateField: released_on, compareFields: [severity, released_on], maxItems: 100 } } | |
| 324 | + - { name: security blog channel feed, url: "https://www.redhat.com/en/rss/blog/channel/security", type: RSS, connector: rss, tier: C } | |
| 325 | + - id: suse | |
| 326 | + extend: true | |
| 327 | + categories: [cyber] | |
| 328 | + sensors: | |
| 329 | + - { name: security blog tag feed, url: "https://www.suse.com/c/tag/security/feed/", type: RSS, connector: rss, tier: C } | |
| 330 | + - id: gentoo | |
| 331 | + name: Gentoo Linux | |
| 332 | + domain: gentoo.org | |
| 333 | + homepage: https://www.gentoo.org | |
| 334 | + categories: [open-source, cyber, developer] | |
| 335 | + tier: B | |
| 336 | + country: US | |
| 337 | + aliases: [gentoo, glsa, gentoo linux security advisories] | |
| 338 | + discover: { rss: false } | |
| 339 | + sensors: | |
| 340 | + - { name: glsa feed, url: "https://security.gentoo.org/glsa/feed.rss", type: RSS, connector: rss, tier: A } | |
| 341 | + - id: linux-kernel | |
| 342 | + extend: true | |
| 343 | + categories: [cyber] | |
| 344 | + products: | |
| 345 | + - { name: Linux kernel CVE team, type: service, aliases: [linux-cve-announce, kernel cve] } | |
| 346 | + sensors: | |
| 347 | + - { name: linux-cve-announce feed, url: "https://lore.kernel.org/linux-cve-announce/new.atom", type: ATOM, connector: rss, tier: A, config: { maxItems: 100 } } | |
| 348 | + - id: openssl | |
| 349 | + extend: true | |
| 350 | + sensors: | |
| 351 | + - { name: vulnerabilities page, url: "https://openssl-library.org/news/vulnerabilities/", type: HTML, connector: http, tier: A } | |
| 352 | + - id: python | |
| 353 | + extend: true | |
| 354 | + categories: [cyber] | |
| 355 | + sensors: | |
| 356 | + - { name: psf advisory database commits, url: "https://github.com/psf/advisory-database/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: psf/advisory-database, kind: commits, branch: main } } | |
| 357 | + - { name: pypa advisory database commits, url: "https://github.com/pypa/advisory-database/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: pypa/advisory-database, kind: commits, branch: main } } | |
| 358 | + - id: go | |
| 359 | + extend: true | |
| 360 | + categories: [cyber] | |
| 361 | + products: | |
| 362 | + - { name: Go Vulnerability Database, type: dataset, aliases: [govulndb, vuln.go.dev, govulncheck] } | |
| 363 | + sensors: | |
| 364 | + - { name: vulnerability database index, url: "https://vuln.go.dev/index/vulns.json", type: JSON, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: id, titleTemplate: "{id} — {aliases}", dateField: modified, urlTemplate: "https://pkg.go.dev/vuln/{key}", compareFields: [modified], maxItems: 10000 } } | |
| 365 | + - id: rust | |
| 366 | + extend: true | |
| 367 | + categories: [cyber] | |
| 368 | + products: | |
| 369 | + - { name: RustSec Advisory Database, type: dataset, aliases: [rustsec, cargo-audit] } | |
| 370 | + sensors: | |
| 371 | + - { name: rustsec advisories feed, url: "https://rustsec.org/feed.xml", type: ATOM, connector: rss, tier: A } | |
| 372 | + - { name: rustsec advisory-db commits, url: "https://github.com/rustsec/advisory-db/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: rustsec/advisory-db, kind: commits, branch: main } } | |
| 373 | + - id: ruby | |
| 374 | + extend: true | |
| 375 | + categories: [cyber] | |
| 376 | + sensors: | |
| 377 | + - { name: ruby advisory database commits, url: "https://github.com/rubysec/ruby-advisory-db/commits/master.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: rubysec/ruby-advisory-db, kind: commits, branch: master } } | |
| 378 | + - id: php | |
| 379 | + extend: true | |
| 380 | + categories: [cyber] | |
| 381 | + sensors: | |
| 382 | + - { name: friendsofphp security advisories commits, url: "https://github.com/FriendsOfPHP/security-advisories/commits/master.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: FriendsOfPHP/security-advisories, kind: commits, branch: master } } | |
| 383 | + - id: nodejs | |
| 384 | + extend: true | |
| 385 | + categories: [cyber] | |
| 386 | + sensors: | |
| 387 | + - { name: security working group commits, url: "https://github.com/nodejs/security-wg/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: nodejs/security-wg, kind: commits, branch: main } } | |
| 388 | + - id: wordpress | |
| 389 | + extend: true | |
| 390 | + categories: [cyber] | |
| 391 | + sensors: | |
| 392 | + - { name: security news feed, url: "https://wordpress.org/news/category/security/feed/", type: RSS, connector: rss, tier: A } | |
| 393 | + - id: wordfence | |
| 394 | + name: Wordfence | |
| 395 | + domain: wordfence.com | |
| 396 | + homepage: https://www.wordfence.com | |
| 397 | + categories: [cyber, internet] | |
| 398 | + tier: B | |
| 399 | + country: US | |
| 400 | + aliases: [defiant, wordfence threat intelligence] | |
| 401 | + discover: { rss: false } | |
| 402 | + sensors: | |
| 403 | + - { name: blog feed, url: "https://www.wordfence.com/blog/feed/", type: RSS, connector: rss, tier: B } | |
| 404 | + - id: patchstack | |
| 405 | + name: Patchstack | |
| 406 | + domain: patchstack.com | |
| 407 | + homepage: https://patchstack.com | |
| 408 | + categories: [cyber, internet] | |
| 409 | + tier: B | |
| 410 | + country: EE | |
| 411 | + discover: { rss: false } | |
| 412 | + sensors: | |
| 413 | + - { name: articles feed, url: "https://patchstack.com/feed/", type: RSS, connector: rss, tier: B } | |
| 414 | + - id: wpscan | |
| 415 | + name: WPScan | |
| 416 | + domain: wpscan.com | |
| 417 | + homepage: https://wpscan.com | |
| 418 | + categories: [cyber, internet] | |
| 419 | + tier: C | |
| 420 | + country: GB | |
| 421 | + aliases: [wpscan vulnerability database] | |
| 422 | + discover: { rss: false } | |
| 423 | + sensors: | |
| 424 | + - { name: blog feed, url: "https://wpscan.com/blog/feed/", type: RSS, connector: rss, tier: C } | |
| 425 | + - id: sucuri | |
| 426 | + name: Sucuri | |
| 427 | + domain: sucuri.net | |
| 428 | + homepage: https://sucuri.net | |
| 429 | + categories: [cyber, internet] | |
| 430 | + tier: C | |
| 431 | + country: US | |
| 432 | + aliases: [sucuri labs] | |
| 433 | + discover: { rss: false } | |
| 434 | + sensors: | |
| 435 | + - { name: blog feed, url: "https://blog.sucuri.net/feed", type: RSS, connector: rss, tier: C } | |
| 436 | + | |
| 437 | + # ───────────────────────── D · Supply-chain security, SBOM/VEX standards, security tooling ───────────────────────── | |
| 438 | + - id: openssf | |
| 439 | + extend: true | |
| 440 | + products: | |
| 441 | + - { name: OpenSSF Scorecard, type: software, aliases: [scorecard] } | |
| 442 | + - { name: OSV Schema, type: standard, aliases: [osv schema, open source vulnerability schema] } | |
| 443 | + sensors: | |
| 444 | + - { name: scorecard releases, url: "https://github.com/ossf/scorecard/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: ossf/scorecard, kind: releases } } | |
| 445 | + - { name: osv schema releases, url: "https://github.com/ossf/osv-schema/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: ossf/osv-schema, kind: releases } } | |
| 446 | + - id: osv | |
| 447 | + name: OSV.dev | |
| 448 | + domain: osv.dev | |
| 449 | + homepage: https://osv.dev | |
| 450 | + categories: [cyber, open-source, developer] | |
| 451 | + tier: B | |
| 452 | + country: US | |
| 453 | + aliases: [open source vulnerabilities, osv scanner, google osv] | |
| 454 | + notes: "The OSV query API is POST-only → no jsonlist sensor; release feed of the scanner used instead." | |
| 455 | + discover: { rss: false } | |
| 456 | + sensors: | |
| 457 | + - { name: osv-scanner releases, url: "https://github.com/google/osv-scanner/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: google/osv-scanner, kind: releases } } | |
| 458 | + - id: sigstore | |
| 459 | + name: Sigstore | |
| 460 | + domain: sigstore.dev | |
| 461 | + homepage: https://www.sigstore.dev | |
| 462 | + categories: [cyber, open-source, developer] | |
| 463 | + tier: C | |
| 464 | + country: US | |
| 465 | + aliases: [cosign, fulcio, rekor] | |
| 466 | + discover: { rss: false } | |
| 467 | + sensors: | |
| 468 | + - { name: cosign releases, url: "https://github.com/sigstore/cosign/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: sigstore/cosign, kind: releases } } | |
| 469 | + - id: chainguard | |
| 470 | + name: Chainguard | |
| 471 | + domain: chainguard.dev | |
| 472 | + homepage: https://www.chainguard.dev | |
| 473 | + categories: [cyber, developer, cloud] | |
| 474 | + tier: C | |
| 475 | + country: US | |
| 476 | + aliases: [wolfi, chainguard images] | |
| 477 | + discover: { rss: false } | |
| 478 | + sensors: | |
| 479 | + - { name: unchained blog feed, url: "https://www.chainguard.dev/unchained/rss.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 480 | + - { name: wolfi os commits, url: "https://github.com/wolfi-dev/os/commits/main.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: wolfi-dev/os, kind: commits, branch: main } } | |
| 481 | + - { name: melange releases, url: "https://github.com/chainguard-dev/melange/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: chainguard-dev/melange, kind: releases } } | |
| 482 | + - id: aqua-security | |
| 483 | + name: Aqua Security | |
| 484 | + domain: aquasec.com | |
| 485 | + homepage: https://www.aquasec.com | |
| 486 | + categories: [cyber, cloud, developer] | |
| 487 | + tier: C | |
| 488 | + country: IL | |
| 489 | + aliases: [aqua nautilus, trivy] | |
| 490 | + products: | |
| 491 | + - { name: Trivy, type: software, aliases: [trivy scanner] } | |
| 492 | + discover: { rss: false } | |
| 493 | + sensors: | |
| 494 | + - { name: blog feed, url: "https://blog.aquasec.com/rss.xml", type: RSS, connector: rss, tier: C } | |
| 495 | + - { name: trivy releases, url: "https://github.com/aquasecurity/trivy/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: aquasecurity/trivy, kind: releases } } | |
| 496 | + - id: anchore | |
| 497 | + name: Anchore | |
| 498 | + domain: anchore.com | |
| 499 | + homepage: https://anchore.com | |
| 500 | + categories: [cyber, developer] | |
| 501 | + tier: D | |
| 502 | + country: US | |
| 503 | + aliases: [grype, syft] | |
| 504 | + discover: { rss: true } | |
| 505 | + sensors: | |
| 506 | + - { name: grype releases, url: "https://github.com/anchore/grype/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: anchore/grype, kind: releases } } | |
| 507 | + - { name: syft releases, url: "https://github.com/anchore/syft/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: anchore/syft, kind: releases } } | |
| 508 | + - id: cyclonedx | |
| 509 | + name: OWASP CycloneDX | |
| 510 | + domain: cyclonedx.org | |
| 511 | + homepage: https://cyclonedx.org | |
| 512 | + categories: [cyber, standards, open-source] | |
| 513 | + tier: D | |
| 514 | + country: US | |
| 515 | + aliases: [cyclonedx, sbom standard] | |
| 516 | + discover: { rss: false } | |
| 517 | + sensors: | |
| 518 | + - { name: specification releases, url: "https://github.com/CycloneDX/specification/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: CycloneDX/specification, kind: releases } } | |
| 519 | + - id: spdx | |
| 520 | + name: SPDX | |
| 521 | + domain: spdx.dev | |
| 522 | + homepage: https://spdx.dev | |
| 523 | + categories: [cyber, standards, open-source] | |
| 524 | + tier: D | |
| 525 | + country: US | |
| 526 | + aliases: [software package data exchange] | |
| 527 | + discover: { rss: false } | |
| 528 | + sensors: | |
| 529 | + - { name: specification releases, url: "https://github.com/spdx/spdx-spec/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: spdx/spdx-spec, kind: releases } } | |
| 530 | + - id: oasis-csaf | |
| 531 | + name: OASIS CSAF | |
| 532 | + domain: oasis-open.org | |
| 533 | + homepage: https://oasis-open.github.io/csaf-documentation/ | |
| 534 | + categories: [cyber, standards] | |
| 535 | + tier: D | |
| 536 | + country: US | |
| 537 | + aliases: [csaf, common security advisory framework] | |
| 538 | + discover: { rss: false } | |
| 539 | + sensors: | |
| 540 | + - { name: csaf releases, url: "https://github.com/oasis-tcs/csaf/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: oasis-tcs/csaf, kind: releases } } | |
| 541 | + - id: openvex | |
| 542 | + name: OpenVEX | |
| 543 | + domain: openvex.dev | |
| 544 | + homepage: https://openvex.dev | |
| 545 | + categories: [cyber, standards, open-source] | |
| 546 | + tier: D | |
| 547 | + country: US | |
| 548 | + aliases: [vex, vulnerability exploitability exchange] | |
| 549 | + discover: { rss: false } | |
| 550 | + sensors: | |
| 551 | + - { name: vexctl releases, url: "https://github.com/openvex/vexctl/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: openvex/vexctl, kind: releases } } | |
| 552 | + - id: projectdiscovery | |
| 553 | + name: ProjectDiscovery | |
| 554 | + domain: projectdiscovery.io | |
| 555 | + homepage: https://projectdiscovery.io | |
| 556 | + categories: [cyber, open-source, developer] | |
| 557 | + tier: B | |
| 558 | + country: US | |
| 559 | + aliases: [nuclei, nuclei templates] | |
| 560 | + discover: { rss: false } | |
| 561 | + sensors: | |
| 562 | + - { name: nuclei releases, url: "https://github.com/projectdiscovery/nuclei/releases.atom", type: GITHUB_RELEASE, connector: github, tier: C, config: { repo: projectdiscovery/nuclei, kind: releases } } | |
| 563 | + - { name: nuclei templates releases, url: "https://github.com/projectdiscovery/nuclei-templates/releases.atom", type: GITHUB_RELEASE, connector: github, tier: B, config: { repo: projectdiscovery/nuclei-templates, kind: releases } } | |
| 564 | + - id: owasp | |
| 565 | + extend: true | |
| 566 | + products: | |
| 567 | + - { name: OWASP ZAP, type: software, aliases: [zap, zed attack proxy] } | |
| 568 | + sensors: | |
| 569 | + - { name: zap releases, url: "https://github.com/zaproxy/zaproxy/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: zaproxy/zaproxy, kind: releases } } | |
| 570 | + - id: semgrep | |
| 571 | + name: Semgrep | |
| 572 | + domain: semgrep.dev | |
| 573 | + homepage: https://semgrep.dev | |
| 574 | + categories: [cyber, developer] | |
| 575 | + tier: D | |
| 576 | + country: US | |
| 577 | + discover: { rss: true } | |
| 578 | + sensors: | |
| 579 | + - { name: releases, url: "https://github.com/semgrep/semgrep/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: semgrep/semgrep, kind: releases } } | |
| 580 | + - id: truffle-security | |
| 581 | + name: Truffle Security | |
| 582 | + domain: trufflesecurity.com | |
| 583 | + homepage: https://trufflesecurity.com | |
| 584 | + categories: [cyber, developer] | |
| 585 | + tier: D | |
| 586 | + country: US | |
| 587 | + aliases: [trufflehog] | |
| 588 | + discover: { rss: true } | |
| 589 | + sensors: | |
| 590 | + - { name: trufflehog releases, url: "https://github.com/trufflesecurity/trufflehog/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: trufflesecurity/trufflehog, kind: releases } } | |
| 591 | + - id: gitleaks | |
| 592 | + name: Gitleaks | |
| 593 | + domain: gitleaks.io | |
| 594 | + homepage: https://gitleaks.io | |
| 595 | + categories: [cyber, developer, open-source] | |
| 596 | + tier: D | |
| 597 | + country: US | |
| 598 | + discover: { rss: false } | |
| 599 | + sensors: | |
| 600 | + - { name: releases, url: "https://github.com/gitleaks/gitleaks/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: gitleaks/gitleaks, kind: releases } } | |
| 601 | + - id: wazuh | |
| 602 | + name: Wazuh | |
| 603 | + domain: wazuh.com | |
| 604 | + homepage: https://wazuh.com | |
| 605 | + categories: [cyber, open-source] | |
| 606 | + tier: D | |
| 607 | + country: US | |
| 608 | + discover: { rss: true } | |
| 609 | + sensors: | |
| 610 | + - { name: releases, url: "https://github.com/wazuh/wazuh/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: wazuh/wazuh, kind: releases } } | |
| 611 | + - id: osquery | |
| 612 | + name: osquery | |
| 613 | + domain: osquery.io | |
| 614 | + homepage: https://osquery.io | |
| 615 | + categories: [cyber, open-source] | |
| 616 | + tier: D | |
| 617 | + country: US | |
| 618 | + discover: { rss: false } | |
| 619 | + sensors: | |
| 620 | + - { name: releases, url: "https://github.com/osquery/osquery/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: osquery/osquery, kind: releases } } | |
| 621 | + - id: falco | |
| 622 | + name: Falco | |
| 623 | + domain: falco.org | |
| 624 | + homepage: https://falco.org | |
| 625 | + categories: [cyber, open-source, cloud] | |
| 626 | + tier: D | |
| 627 | + country: US | |
| 628 | + aliases: [falcosecurity] | |
| 629 | + discover: { rss: true } | |
| 630 | + sensors: | |
| 631 | + - { name: releases, url: "https://github.com/falcosecurity/falco/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: falcosecurity/falco, kind: releases } } | |
| 632 | + - id: filigran | |
| 633 | + name: Filigran (OpenCTI) | |
| 634 | + domain: filigran.io | |
| 635 | + homepage: https://filigran.io | |
| 636 | + categories: [cyber, open-source] | |
| 637 | + tier: D | |
| 638 | + country: FR | |
| 639 | + aliases: [opencti, openbas] | |
| 640 | + discover: { rss: true } | |
| 641 | + sensors: | |
| 642 | + - { name: opencti releases, url: "https://github.com/OpenCTI-Platform/opencti/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: OpenCTI-Platform/opencti, kind: releases } } | |
| 643 | + - id: sigmahq | |
| 644 | + name: SigmaHQ | |
| 645 | + domain: sigmahq.io | |
| 646 | + homepage: https://sigmahq.io | |
| 647 | + categories: [cyber, open-source] | |
| 648 | + tier: D | |
| 649 | + country: DE | |
| 650 | + aliases: [sigma rules] | |
| 651 | + discover: { rss: false } | |
| 652 | + sensors: | |
| 653 | + - { name: sigma releases, url: "https://github.com/SigmaHQ/sigma/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: SigmaHQ/sigma, kind: releases } } | |
| 654 | + - id: virustotal | |
| 655 | + name: VirusTotal | |
| 656 | + domain: virustotal.com | |
| 657 | + homepage: https://www.virustotal.com | |
| 658 | + categories: [cyber] | |
| 659 | + tier: C | |
| 660 | + country: US | |
| 661 | + aliases: [yara] | |
| 662 | + products: | |
| 663 | + - { name: YARA, type: software, aliases: [yara rules engine] } | |
| 664 | + discover: { rss: false } | |
| 665 | + sensors: | |
| 666 | + - { name: blog feed, url: "https://blog.virustotal.com/feeds/posts/default", type: ATOM, connector: rss, tier: C } | |
| 667 | + - { name: yara releases, url: "https://github.com/VirusTotal/yara/releases.atom", type: GITHUB_RELEASE, connector: github, tier: D, config: { repo: VirusTotal/yara, kind: releases } } | |
| 668 | + - id: hibp | |
| 669 | + extend: true | |
| 670 | + country: AU | |
| 671 | + aliases: [pwned, haveibeenpwned] | |
| 672 | + - id: exploit-db | |
| 673 | + name: Exploit Database | |
| 674 | + domain: exploit-db.com | |
| 675 | + homepage: https://www.exploit-db.com | |
| 676 | + categories: [cyber] | |
| 677 | + tier: A | |
| 678 | + country: US | |
| 679 | + aliases: [exploit-db, exploitdb, offsec exploit database] | |
| 680 | + llm: false | |
| 681 | + discover: { rss: false } | |
| 682 | + sensors: | |
| 683 | + - { name: latest exploits feed, url: "https://www.exploit-db.com/rss.xml", type: RSS, connector: rss, tier: A, config: { maxItems: 50 } } | |
| 684 | + - { name: exploitdb repository commits, url: "https://gitlab.com/exploit-database/exploitdb/-/commits/main?format=atom", type: ATOM, connector: rss, tier: B, config: { maxItems: 40 } } | |
| 685 | + - id: ransomware-live | |
| 686 | + name: ransomware.live | |
| 687 | + domain: ransomware.live | |
| 688 | + homepage: https://www.ransomware.live | |
| 689 | + categories: [cyber] | |
| 690 | + tier: A | |
| 691 | + country: FR | |
| 692 | + aliases: [ransomware live, ransomware victims tracker] | |
| 693 | + llm: false | |
| 694 | + discover: { rss: false } | |
| 695 | + sensors: | |
| 696 | + - { name: recent victims api, url: "https://api.ransomware.live/v2/recentvictims", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: victim, titleTemplate: "{group} — {victim} ({country})", urlField: url, summaryField: description, dateField: discovered, compareFields: [group, attackdate, ransom], maxItems: 100 } } | |
| 697 | + - { name: victims feed, url: "https://www.ransomware.live/rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 698 | + - id: abuse-ch | |
| 699 | + name: abuse.ch | |
| 700 | + domain: abuse.ch | |
| 701 | + homepage: https://abuse.ch | |
| 702 | + categories: [cyber] | |
| 703 | + tier: A | |
| 704 | + country: CH | |
| 705 | + aliases: [urlhaus, threatfox, malwarebazaar, feodo tracker, sslbl] | |
| 706 | + llm: false | |
| 707 | + notes: "URLhaus/ThreatFox/MalwareBazaar APIs require an Auth-Key since 2025; only the Feodo Tracker JSON blocklist is anonymous." | |
| 708 | + discover: { rss: false } | |
| 709 | + sensors: | |
| 710 | + - { name: feodo tracker c2 blocklist, url: "https://feodotracker.abuse.ch/downloads/ipblocklist.json", type: JSON, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: ip_address, titleTemplate: "{malware} C2 {ip_address}:{port} ({country}, {as_name})", dateField: first_seen, urlTemplate: "https://feodotracker.abuse.ch/browse/host/{key}/", compareFields: [status, last_online], maxItems: 500 } } | |
| 711 | + | |
| 712 | + # ───────────────────────── E · National CERTs, CSIRTs & cyber agencies ───────────────────────── | |
| 713 | + - id: ncsc-uk | |
| 714 | + name: NCSC (UK National Cyber Security Centre) | |
| 715 | + domain: ncsc.gov.uk | |
| 716 | + homepage: https://www.ncsc.gov.uk | |
| 717 | + categories: [cyber, government] | |
| 718 | + tier: A | |
| 719 | + weight: 1.3 | |
| 720 | + country: GB | |
| 721 | + aliases: [ncsc, uk ncsc, national cyber security centre] | |
| 722 | + discover: { rss: false } | |
| 723 | + sensors: | |
| 724 | + - { name: all content feed, url: "https://www.ncsc.gov.uk/api/1/services/v1/all-rss-feed.xml", type: RSS, connector: rss, tier: A } | |
| 725 | + - { name: news feed, url: "https://www.ncsc.gov.uk/api/1/services/v1/news-rss-feed.xml", type: RSS, connector: rss, tier: B } | |
| 726 | + - { name: reports & advisories feed, url: "https://www.ncsc.gov.uk/api/1/services/v1/report-rss-feed.xml", type: RSS, connector: rss, tier: A } | |
| 727 | + - { name: gov.uk publications atom, url: "https://www.gov.uk/government/organisations/national-cyber-security-centre.atom", type: ATOM, connector: rss, tier: C } | |
| 728 | + - id: cccs | |
| 729 | + name: Canadian Centre for Cyber Security | |
| 730 | + domain: cyber.gc.ca | |
| 731 | + homepage: https://www.cyber.gc.ca | |
| 732 | + categories: [cyber, government] | |
| 733 | + tier: A | |
| 734 | + weight: 1.2 | |
| 735 | + country: CA | |
| 736 | + aliases: [cccs, cyber centre, centre canadien pour la cybersécurité, cse cyber centre] | |
| 737 | + discover: { rss: false } | |
| 738 | + sensors: | |
| 739 | + - { name: alerts & advisories atom, url: "https://www.cyber.gc.ca/api/cccs/atom/v1/get?feed=alerts_advisories&lang=en", type: ATOM, connector: rss, tier: A, config: { maxItems: 50 } } | |
| 740 | + - { name: alertes et avis (fr), url: "https://www.cyber.gc.ca/api/cccs/atom/v1/get?feed=alerts_advisories&lang=fr", type: ATOM, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 741 | + - id: cert-fr | |
| 742 | + name: CERT-FR (ANSSI) | |
| 743 | + domain: cert.ssi.gouv.fr | |
| 744 | + homepage: https://www.cert.ssi.gouv.fr | |
| 745 | + categories: [cyber, government] | |
| 746 | + tier: A | |
| 747 | + weight: 1.2 | |
| 748 | + country: FR | |
| 749 | + language: fr | |
| 750 | + aliases: [cert-fr, anssi, agence nationale de la sécurité des systèmes d'information] | |
| 751 | + discover: { rss: false } | |
| 752 | + sensors: | |
| 753 | + - { name: avis de sécurité, url: "https://www.cert.ssi.gouv.fr/avis/feed/", type: RSS, connector: rss, tier: A } | |
| 754 | + - { name: alertes, url: "https://www.cert.ssi.gouv.fr/alerte/feed/", type: RSS, connector: rss, tier: S } | |
| 755 | + - { name: menaces et incidents (cti), url: "https://www.cert.ssi.gouv.fr/cti/feed/", type: RSS, connector: rss, tier: B } | |
| 756 | + - { name: indicateurs de compromission, url: "https://www.cert.ssi.gouv.fr/ioc/feed/", type: RSS, connector: rss, tier: B } | |
| 757 | + - { name: actualités, url: "https://www.cert.ssi.gouv.fr/actualite/feed/", type: RSS, connector: rss, tier: C } | |
| 758 | + - id: cert-bund | |
| 759 | + name: CERT-Bund (BSI) | |
| 760 | + domain: cert-bund.de | |
| 761 | + homepage: https://wid.cert-bund.de/portal/wid/kurzinformationen | |
| 762 | + categories: [cyber, government] | |
| 763 | + tier: A | |
| 764 | + weight: 1.2 | |
| 765 | + country: DE | |
| 766 | + language: de | |
| 767 | + aliases: [cert-bund, bsi, bundesamt für sicherheit in der informationstechnik, warn- und informationsdienst] | |
| 768 | + notes: "bsi.bund.de RSS endpoints return 404; the WID portal's 'rss' endpoint returns a JSON envelope (items[])." | |
| 769 | + discover: { rss: false } | |
| 770 | + sensors: | |
| 771 | + - { name: wid security advisories, url: "https://wid.cert-bund.de/content/public/securityAdvisory/rss", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: items, keyField: link, titleField: title, urlField: link, summaryField: description, dateField: pubDate, compareFields: [title], maxItems: 100 } } | |
| 772 | + - id: cert-eu | |
| 773 | + name: CERT-EU | |
| 774 | + domain: cert.europa.eu | |
| 775 | + homepage: https://cert.europa.eu | |
| 776 | + categories: [cyber, government, international] | |
| 777 | + tier: A | |
| 778 | + country: EU | |
| 779 | + aliases: [cert-eu, cybersecurity service for the union institutions] | |
| 780 | + discover: { rss: false } | |
| 781 | + sensors: | |
| 782 | + - { name: security advisories feed, url: "https://cert.europa.eu/publications/security-advisories-rss", type: RSS, connector: rss, tier: A } | |
| 783 | + - { name: threat intelligence feed, url: "https://cert.europa.eu/publications/threat-intelligence-rss", type: RSS, connector: rss, tier: B } | |
| 784 | + - { name: security guidance feed, url: "https://cert.europa.eu/publications/security-guidance-rss", type: RSS, connector: rss, tier: C } | |
| 785 | + - id: enisa | |
| 786 | + extend: true | |
| 787 | + country: EU | |
| 788 | + products: | |
| 789 | + - { name: EUVD, type: dataset, aliases: [european union vulnerability database, eu vulnerability database] } | |
| 790 | + notes: "enisa.europa.eu RSS endpoints return 404; EUVD API (euvdservices) is public JSON." | |
| 791 | + sensors: | |
| 792 | + - { name: euvd latest vulnerabilities, url: "https://euvdservices.enisa.europa.eu/api/lastvulnerabilities", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: id, titleTemplate: "{id} — CVSS {baseScore} ({assigner})", summaryField: description, dateField: datePublished, urlTemplate: "https://euvd.enisa.europa.eu/vulnerability/{key}", compareFields: [baseScore, dateUpdated, exploitedSince], maxItems: 50 } } | |
| 793 | + - { name: euvd exploited vulnerabilities, url: "https://euvdservices.enisa.europa.eu/api/exploitedvulnerabilities", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: id, titleTemplate: "{id} — exploited since {exploitedSince} (CVSS {baseScore})", summaryField: description, dateField: dateUpdated, urlTemplate: "https://euvd.enisa.europa.eu/vulnerability/{key}", compareFields: [baseScore, exploitedSince], maxItems: 50 } } | |
| 794 | + - { name: euvd critical vulnerabilities, url: "https://euvdservices.enisa.europa.eu/api/criticalvulnerabilities", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: id, titleTemplate: "{id} — critical CVSS {baseScore}", summaryField: description, dateField: datePublished, urlTemplate: "https://euvd.enisa.europa.eu/vulnerability/{key}", compareFields: [baseScore, dateUpdated], maxItems: 50 } } | |
| 795 | + - id: europol | |
| 796 | + extend: true | |
| 797 | + country: EU | |
| 798 | + - id: jpcert | |
| 799 | + name: JPCERT/CC | |
| 800 | + domain: jpcert.or.jp | |
| 801 | + homepage: https://www.jpcert.or.jp/english/ | |
| 802 | + categories: [cyber] | |
| 803 | + tier: A | |
| 804 | + country: JP | |
| 805 | + aliases: [jpcert, japan computer emergency response team coordination center] | |
| 806 | + discover: { rss: false } | |
| 807 | + sensors: | |
| 808 | + - { name: english feed, url: "https://www.jpcert.or.jp/english/rss/jpcert-en.rdf", type: RSS, connector: rss, tier: A } | |
| 809 | + - id: jvn | |
| 810 | + name: JVN (Japan Vulnerability Notes) | |
| 811 | + domain: jvn.jp | |
| 812 | + homepage: https://jvn.jp/en/ | |
| 813 | + categories: [cyber, government] | |
| 814 | + tier: A | |
| 815 | + country: JP | |
| 816 | + aliases: [jvn, jvndb, japan vulnerability notes] | |
| 817 | + llm: false | |
| 818 | + discover: { rss: false } | |
| 819 | + sensors: | |
| 820 | + - { name: jvn notes feed, url: "https://jvn.jp/en/rss/jvn.rdf", type: RSS, connector: rss, tier: A } | |
| 821 | + - { name: jvndb new entries feed, url: "https://jvndb.jvn.jp/en/rss/jvndb_new.rdf", type: RSS, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 822 | + - id: ipa-japan | |
| 823 | + name: IPA (Information-technology Promotion Agency, Japan) | |
| 824 | + domain: ipa.go.jp | |
| 825 | + homepage: https://www.ipa.go.jp/security/ | |
| 826 | + categories: [cyber, government] | |
| 827 | + tier: B | |
| 828 | + country: JP | |
| 829 | + language: ja | |
| 830 | + aliases: [ipa, 情報処理推進機構] | |
| 831 | + discover: { rss: false } | |
| 832 | + sensors: | |
| 833 | + - { name: security alerts feed, url: "https://www.ipa.go.jp/security/rss/alert.rdf", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 834 | + - id: ncsc-nl | |
| 835 | + name: NCSC-NL | |
| 836 | + domain: ncsc.nl | |
| 837 | + homepage: https://www.ncsc.nl | |
| 838 | + categories: [cyber, government] | |
| 839 | + tier: A | |
| 840 | + country: NL | |
| 841 | + aliases: [ncsc-nl, nationaal cyber security centrum] | |
| 842 | + discover: { rss: false } | |
| 843 | + sensors: | |
| 844 | + - { name: security advisories feed, url: "https://advisories.ncsc.nl/rss/advisories", type: RSS, connector: rss, tier: A } | |
| 845 | + - id: incibe | |
| 846 | + name: INCIBE-CERT | |
| 847 | + domain: incibe.es | |
| 848 | + homepage: https://www.incibe.es/en/incibe-cert | |
| 849 | + categories: [cyber, government] | |
| 850 | + tier: A | |
| 851 | + country: ES | |
| 852 | + aliases: [incibe, incibe-cert, instituto nacional de ciberseguridad] | |
| 853 | + discover: { rss: false } | |
| 854 | + sensors: | |
| 855 | + - { name: security advisories feed (en), url: "https://www.incibe.es/en/incibe-cert/early-warning/security-advisories/feed", type: RSS, connector: rss, tier: A } | |
| 856 | + - { name: avisos sci (ics), url: "https://www.incibe.es/incibe-cert/alerta-temprana/avisos-sci/feed", type: RSS, connector: rss, tier: B } | |
| 857 | + - { name: blog feed, url: "https://www.incibe.es/incibe-cert/blog/feed", type: RSS, connector: rss, tier: C } | |
| 858 | + - id: cert-pl | |
| 859 | + name: CERT Polska | |
| 860 | + domain: cert.pl | |
| 861 | + homepage: https://cert.pl/en/ | |
| 862 | + categories: [cyber, government] | |
| 863 | + tier: B | |
| 864 | + country: PL | |
| 865 | + aliases: [cert polska, cert.pl, nask] | |
| 866 | + discover: { rss: false } | |
| 867 | + sensors: | |
| 868 | + - { name: english feed, url: "https://cert.pl/en/rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 869 | + - id: cert-at | |
| 870 | + name: CERT.at | |
| 871 | + domain: cert.at | |
| 872 | + homepage: https://www.cert.at | |
| 873 | + categories: [cyber, government] | |
| 874 | + tier: A | |
| 875 | + country: AT | |
| 876 | + language: de | |
| 877 | + aliases: [cert.at, govcert austria] | |
| 878 | + discover: { rss: false } | |
| 879 | + sensors: | |
| 880 | + - { name: warnungen, url: "https://www.cert.at/cert-at.de.warnings.rss_2.0.xml", type: RSS, connector: rss, tier: A, config: { maxItems: 50 } } | |
| 881 | + - { name: alle meldungen, url: "https://www.cert.at/cert-at.de.all.rss_2.0.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 882 | + - id: ccb-belgium | |
| 883 | + name: Centre for Cybersecurity Belgium | |
| 884 | + domain: ccb.belgium.be | |
| 885 | + homepage: https://ccb.belgium.be | |
| 886 | + categories: [cyber, government] | |
| 887 | + tier: A | |
| 888 | + country: BE | |
| 889 | + aliases: [ccb, cert.be, centre for cyber security belgium] | |
| 890 | + discover: { rss: false } | |
| 891 | + sensors: | |
| 892 | + - { name: advisories feed, url: "https://ccb.belgium.be/advisories.xml", type: RSS, connector: rss, tier: A } | |
| 893 | + - { name: news feed, url: "https://ccb.belgium.be/news.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 894 | + - id: cert-se | |
| 895 | + name: CERT-SE | |
| 896 | + domain: cert.se | |
| 897 | + homepage: https://www.cert.se | |
| 898 | + categories: [cyber, government] | |
| 899 | + tier: B | |
| 900 | + country: SE | |
| 901 | + language: sv | |
| 902 | + aliases: [cert-se, msb cert] | |
| 903 | + discover: { rss: false } | |
| 904 | + sensors: | |
| 905 | + - { name: feed, url: "https://www.cert.se/feed/rss.xml", type: RSS, connector: rss, tier: B } | |
| 906 | + - id: si-cert | |
| 907 | + name: SI-CERT | |
| 908 | + domain: cert.si | |
| 909 | + homepage: https://www.cert.si/en/ | |
| 910 | + categories: [cyber, government] | |
| 911 | + tier: C | |
| 912 | + country: SI | |
| 913 | + aliases: [si-cert, slovenian computer emergency response team] | |
| 914 | + discover: { rss: false } | |
| 915 | + sensors: | |
| 916 | + - { name: english feed, url: "https://www.cert.si/en/feed/", type: RSS, connector: rss, tier: C } | |
| 917 | + - id: cert-hr | |
| 918 | + name: CERT.hr (CARNET) | |
| 919 | + domain: cert.hr | |
| 920 | + homepage: https://www.cert.hr | |
| 921 | + categories: [cyber, government] | |
| 922 | + tier: C | |
| 923 | + country: HR | |
| 924 | + language: hr | |
| 925 | + aliases: [cert.hr, nacionalni cert] | |
| 926 | + discover: { rss: false } | |
| 927 | + sensors: | |
| 928 | + - { name: feed, url: "https://www.cert.hr/feed/", type: RSS, connector: rss, tier: C } | |
| 929 | + - id: csirt-sk | |
| 930 | + name: CSIRT.SK | |
| 931 | + domain: csirt.sk | |
| 932 | + homepage: https://www.csirt.sk | |
| 933 | + categories: [cyber, government] | |
| 934 | + tier: C | |
| 935 | + country: SK | |
| 936 | + language: sk | |
| 937 | + aliases: [csirt.sk, sk-cert] | |
| 938 | + llm: false | |
| 939 | + discover: { rss: false } | |
| 940 | + sensors: | |
| 941 | + - { name: feed, url: "https://www.csirt.sk/feed", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 942 | + - id: dnsc-romania | |
| 943 | + name: DNSC (Romanian National Cyber Security Directorate) | |
| 944 | + domain: dnsc.ro | |
| 945 | + homepage: https://www.dnsc.ro | |
| 946 | + categories: [cyber, government] | |
| 947 | + tier: C | |
| 948 | + country: RO | |
| 949 | + language: ro | |
| 950 | + aliases: [dnsc, cert-ro, directoratul național de securitate cibernetică] | |
| 951 | + discover: { rss: false } | |
| 952 | + sensors: | |
| 953 | + - { name: feed, url: "https://www.dnsc.ro/feed", type: RSS, connector: rss, tier: C } | |
| 954 | + - id: cert-ua | |
| 955 | + name: CERT-UA | |
| 956 | + domain: cert.gov.ua | |
| 957 | + homepage: https://cert.gov.ua | |
| 958 | + categories: [cyber, government] | |
| 959 | + tier: A | |
| 960 | + country: UA | |
| 961 | + language: uk | |
| 962 | + aliases: [cert-ua, урядова команда реагування на комп'ютерні надзвичайні події україни] | |
| 963 | + discover: { rss: false } | |
| 964 | + sensors: | |
| 965 | + - { name: articles feed, url: "https://cert.gov.ua/api/articles/rss", type: RSS, connector: rss, tier: A } | |
| 966 | + - id: kisa-krcert | |
| 967 | + name: KISA KrCERT/CC | |
| 968 | + domain: boho.or.kr | |
| 969 | + homepage: https://www.boho.or.kr | |
| 970 | + categories: [cyber, government] | |
| 971 | + tier: B | |
| 972 | + country: KR | |
| 973 | + language: ko | |
| 974 | + aliases: [krcert, kisa, korea internet & security agency] | |
| 975 | + discover: { rss: false } | |
| 976 | + sensors: | |
| 977 | + - { name: security notices feed, url: "https://www.boho.or.kr/kr/rss.do?bbsId=B0000133", type: RSS, connector: rss, tier: B } | |
| 978 | + - id: cert-br | |
| 979 | + name: CERT.br | |
| 980 | + domain: cert.br | |
| 981 | + homepage: https://www.cert.br | |
| 982 | + categories: [cyber] | |
| 983 | + tier: C | |
| 984 | + country: BR | |
| 985 | + language: pt | |
| 986 | + aliases: [cert.br, nic.br cert] | |
| 987 | + discover: { rss: false } | |
| 988 | + sensors: | |
| 989 | + - { name: feed, url: "https://cert.br/rss/certbr-rss.xml", type: RSS, connector: rss, tier: C } | |
| 990 | + - id: cert-ae | |
| 991 | + name: UAE Cyber Security Council (aeCERT) | |
| 992 | + domain: cert.ae | |
| 993 | + homepage: https://www.cert.ae | |
| 994 | + categories: [cyber, government] | |
| 995 | + tier: C | |
| 996 | + country: AE | |
| 997 | + aliases: [aecert, uae cert] | |
| 998 | + discover: { rss: false } | |
| 999 | + sensors: | |
| 1000 | + - { name: feed, url: "https://www.cert.ae/feed/", type: RSS, connector: rss, tier: C } | |
| 1001 | + - id: traficom-ncsc-fi | |
| 1002 | + name: NCSC-FI (Traficom) | |
| 1003 | + domain: kyberturvallisuuskeskus.fi | |
| 1004 | + homepage: https://www.kyberturvallisuuskeskus.fi/en/ | |
| 1005 | + categories: [cyber, government] | |
| 1006 | + tier: B | |
| 1007 | + country: FI | |
| 1008 | + aliases: [ncsc-fi, kyberturvallisuuskeskus, finnish transport and communications agency cyber security centre] | |
| 1009 | + discover: { rss: false } | |
| 1010 | + sensors: | |
| 1011 | + - { name: english feed, url: "https://www.kyberturvallisuuskeskus.fi/feed/rss/en", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1012 | + - id: cnil | |
| 1013 | + extend: true | |
| 1014 | + sensors: | |
| 1015 | + - { name: english feed, url: "https://www.cnil.fr/en/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1016 | + | |
| 1017 | + # ───────────────────────── F · Threat-intelligence & security research vendors ───────────────────────── | |
| 1018 | + - id: kaspersky | |
| 1019 | + name: Kaspersky | |
| 1020 | + domain: kaspersky.com | |
| 1021 | + homepage: https://www.kaspersky.com | |
| 1022 | + categories: [cyber] | |
| 1023 | + tier: B | |
| 1024 | + country: RU | |
| 1025 | + aliases: [securelist, kaspersky lab, great] | |
| 1026 | + products: | |
| 1027 | + - { name: Securelist, type: service, aliases: [securelist blog] } | |
| 1028 | + - { name: Kaspersky ICS CERT, type: service, aliases: [ics cert kaspersky] } | |
| 1029 | + discover: { rss: false } | |
| 1030 | + sensors: | |
| 1031 | + - { name: securelist feed, url: "https://securelist.com/feed/", type: RSS, connector: rss, tier: B } | |
| 1032 | + - { name: securelist apt reports, url: "https://securelist.com/category/apt-reports/feed/", type: RSS, connector: rss, tier: B } | |
| 1033 | + - { name: ics cert feed, url: "https://ics-cert.kaspersky.com/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1034 | + - { name: daily blog feed, url: "https://www.kaspersky.com/blog/feed/", type: RSS, connector: rss, tier: C } | |
| 1035 | + - id: eset | |
| 1036 | + name: ESET | |
| 1037 | + domain: eset.com | |
| 1038 | + homepage: https://www.eset.com | |
| 1039 | + categories: [cyber] | |
| 1040 | + tier: B | |
| 1041 | + country: SK | |
| 1042 | + aliases: [welivesecurity, eset research] | |
| 1043 | + products: | |
| 1044 | + - { name: WeLiveSecurity, type: service, aliases: [we live security] } | |
| 1045 | + discover: { rss: false } | |
| 1046 | + sensors: | |
| 1047 | + - { name: welivesecurity feed, url: "https://www.welivesecurity.com/en/rss/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1048 | + - id: check-point | |
| 1049 | + name: Check Point Software | |
| 1050 | + domain: checkpoint.com | |
| 1051 | + homepage: https://www.checkpoint.com | |
| 1052 | + categories: [cyber] | |
| 1053 | + tier: B | |
| 1054 | + country: IL | |
| 1055 | + aliases: [check point research, cpr, checkpoint] | |
| 1056 | + products: | |
| 1057 | + - { name: Check Point Research, type: service, aliases: [cpr] } | |
| 1058 | + discover: { rss: false } | |
| 1059 | + sensors: | |
| 1060 | + - { name: research feed, url: "https://research.checkpoint.com/feed/", type: RSS, connector: rss, tier: B } | |
| 1061 | + - { name: blog feed, url: "https://blog.checkpoint.com/feed/", type: RSS, connector: rss, tier: C } | |
| 1062 | + - id: recorded-future | |
| 1063 | + name: Recorded Future | |
| 1064 | + domain: recordedfuture.com | |
| 1065 | + homepage: https://www.recordedfuture.com | |
| 1066 | + categories: [cyber] | |
| 1067 | + tier: B | |
| 1068 | + country: US | |
| 1069 | + aliases: [insikt group] | |
| 1070 | + discover: { rss: false } | |
| 1071 | + sensors: | |
| 1072 | + - { name: blog & research feed, url: "https://www.recordedfuture.com/feed", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1073 | + - id: greynoise | |
| 1074 | + name: GreyNoise Intelligence | |
| 1075 | + domain: greynoise.io | |
| 1076 | + homepage: https://www.greynoise.io | |
| 1077 | + categories: [cyber] | |
| 1078 | + tier: B | |
| 1079 | + country: US | |
| 1080 | + aliases: [greynoise] | |
| 1081 | + discover: { rss: false } | |
| 1082 | + sensors: | |
| 1083 | + - { name: blog feed, url: "https://www.greynoise.io/blog/rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1084 | + - id: huntress | |
| 1085 | + name: Huntress | |
| 1086 | + domain: huntress.com | |
| 1087 | + homepage: https://www.huntress.com | |
| 1088 | + categories: [cyber] | |
| 1089 | + tier: B | |
| 1090 | + country: US | |
| 1091 | + discover: { rss: false } | |
| 1092 | + sensors: | |
| 1093 | + - { name: blog feed, url: "https://www.huntress.com/blog/rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1094 | + - id: datadog | |
| 1095 | + extend: true | |
| 1096 | + categories: [cyber] | |
| 1097 | + products: | |
| 1098 | + - { name: Datadog Security Labs, type: service, aliases: [security labs] } | |
| 1099 | + sensors: | |
| 1100 | + - { name: security labs feed, url: "https://securitylabs.datadoghq.com/rss/feed.xml", type: RSS, connector: rss, tier: B } | |
| 1101 | + - id: orca-security | |
| 1102 | + name: Orca Security | |
| 1103 | + domain: orca.security | |
| 1104 | + homepage: https://orca.security | |
| 1105 | + categories: [cyber, cloud] | |
| 1106 | + tier: C | |
| 1107 | + country: US | |
| 1108 | + aliases: [orca] | |
| 1109 | + discover: { rss: false } | |
| 1110 | + sensors: | |
| 1111 | + - { name: blog feed, url: "https://orca.security/resources/blog/feed/", type: RSS, connector: rss, tier: C } | |
| 1112 | + - id: qualys | |
| 1113 | + name: Qualys | |
| 1114 | + domain: qualys.com | |
| 1115 | + homepage: https://www.qualys.com | |
| 1116 | + categories: [cyber, enterprise] | |
| 1117 | + tier: B | |
| 1118 | + country: US | |
| 1119 | + aliases: [qualys tru, threat research unit] | |
| 1120 | + discover: { rss: false } | |
| 1121 | + sensors: | |
| 1122 | + - { name: blog feed, url: "https://blog.qualys.com/feed", type: RSS, connector: rss, tier: B } | |
| 1123 | + - { name: threatprotect feed, url: "https://threatprotect.qualys.com/feed/", type: RSS, connector: rss, tier: A } | |
| 1124 | + - id: bitdefender | |
| 1125 | + name: Bitdefender | |
| 1126 | + domain: bitdefender.com | |
| 1127 | + homepage: https://www.bitdefender.com | |
| 1128 | + categories: [cyber, consumer-tech] | |
| 1129 | + tier: C | |
| 1130 | + country: RO | |
| 1131 | + aliases: [bitdefender labs, hot for security] | |
| 1132 | + discover: { rss: false } | |
| 1133 | + sensors: | |
| 1134 | + - { name: labs feed, url: "https://www.bitdefender.com/nuxt/api/en-us/rss/labs/", type: RSS, connector: rss, tier: C } | |
| 1135 | + - { name: hot for security feed, url: "https://www.bitdefender.com/nuxt/api/en-us/rss/hotforsecurity/", type: RSS, connector: rss, tier: C } | |
| 1136 | + - id: avast | |
| 1137 | + name: Avast (Gen Digital) | |
| 1138 | + domain: avast.com | |
| 1139 | + homepage: https://www.avast.com | |
| 1140 | + categories: [cyber, consumer-tech] | |
| 1141 | + tier: C | |
| 1142 | + country: CZ | |
| 1143 | + aliases: [gen digital, avast threat labs, norton lifelock] | |
| 1144 | + discover: { rss: false } | |
| 1145 | + sensors: | |
| 1146 | + - { name: blog feed, url: "https://blog.avast.com/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1147 | + - id: cybereason | |
| 1148 | + name: Cybereason | |
| 1149 | + domain: cybereason.com | |
| 1150 | + homepage: https://www.cybereason.com | |
| 1151 | + categories: [cyber] | |
| 1152 | + tier: C | |
| 1153 | + country: US | |
| 1154 | + discover: { rss: false } | |
| 1155 | + sensors: | |
| 1156 | + - { name: blog feed, url: "https://www.cybereason.com/blog/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1157 | + - id: arctic-wolf | |
| 1158 | + name: Arctic Wolf | |
| 1159 | + domain: arcticwolf.com | |
| 1160 | + homepage: https://arcticwolf.com | |
| 1161 | + categories: [cyber] | |
| 1162 | + tier: C | |
| 1163 | + country: US | |
| 1164 | + aliases: [arctic wolf labs] | |
| 1165 | + discover: { rss: false } | |
| 1166 | + sensors: | |
| 1167 | + - { name: feed, url: "https://arcticwolf.com/feed/", type: RSS, connector: rss, tier: C } | |
| 1168 | + - id: forescout | |
| 1169 | + name: Forescout | |
| 1170 | + domain: forescout.com | |
| 1171 | + homepage: https://www.forescout.com | |
| 1172 | + categories: [cyber] | |
| 1173 | + tier: C | |
| 1174 | + country: US | |
| 1175 | + aliases: [vedere labs] | |
| 1176 | + discover: { rss: false } | |
| 1177 | + sensors: | |
| 1178 | + - { name: blog feed, url: "https://www.forescout.com/feed/", type: RSS, connector: rss, tier: C } | |
| 1179 | + - id: censys | |
| 1180 | + name: Censys | |
| 1181 | + domain: censys.com | |
| 1182 | + homepage: https://censys.com | |
| 1183 | + categories: [cyber, internet] | |
| 1184 | + tier: C | |
| 1185 | + country: US | |
| 1186 | + discover: { rss: false } | |
| 1187 | + sensors: | |
| 1188 | + - { name: blog feed, url: "https://censys.com/feed/", type: RSS, connector: rss, tier: C } | |
| 1189 | + - id: shodan | |
| 1190 | + name: Shodan | |
| 1191 | + domain: shodan.io | |
| 1192 | + homepage: https://www.shodan.io | |
| 1193 | + categories: [cyber, internet] | |
| 1194 | + tier: C | |
| 1195 | + country: US | |
| 1196 | + discover: { rss: false } | |
| 1197 | + sensors: | |
| 1198 | + - { name: blog feed, url: "https://blog.shodan.io/rss/", type: RSS, connector: rss, tier: C } | |
| 1199 | + - id: cloudsek | |
| 1200 | + name: CloudSEK | |
| 1201 | + domain: cloudsek.com | |
| 1202 | + homepage: https://www.cloudsek.com | |
| 1203 | + categories: [cyber] | |
| 1204 | + tier: C | |
| 1205 | + country: IN | |
| 1206 | + discover: { rss: false } | |
| 1207 | + sensors: | |
| 1208 | + - { name: blog feed, url: "https://www.cloudsek.com/blog/rss.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 1209 | + - id: intel471 | |
| 1210 | + name: Intel 471 | |
| 1211 | + domain: intel471.com | |
| 1212 | + homepage: https://www.intel471.com | |
| 1213 | + categories: [cyber] | |
| 1214 | + tier: C | |
| 1215 | + country: US | |
| 1216 | + discover: { rss: false } | |
| 1217 | + sensors: | |
| 1218 | + - { name: blog feed, url: "https://www.intel471.com/blog/feed", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 1219 | + - id: flashpoint | |
| 1220 | + name: Flashpoint | |
| 1221 | + domain: flashpoint.io | |
| 1222 | + homepage: https://flashpoint.io | |
| 1223 | + categories: [cyber] | |
| 1224 | + tier: C | |
| 1225 | + country: US | |
| 1226 | + discover: { rss: false } | |
| 1227 | + sensors: | |
| 1228 | + - { name: blog feed, url: "https://flashpoint.io/blog/feed/", type: RSS, connector: rss, tier: C } | |
| 1229 | + - id: socradar | |
| 1230 | + name: SOCRadar | |
| 1231 | + domain: socradar.io | |
| 1232 | + homepage: https://socradar.io | |
| 1233 | + categories: [cyber] | |
| 1234 | + tier: C | |
| 1235 | + country: US | |
| 1236 | + discover: { rss: false } | |
| 1237 | + sensors: | |
| 1238 | + - { name: blog feed, url: "https://socradar.io/feed/", type: RSS, connector: rss, tier: C } | |
| 1239 | + - id: eclecticiq | |
| 1240 | + name: EclecticIQ | |
| 1241 | + domain: eclecticiq.com | |
| 1242 | + homepage: https://www.eclecticiq.com | |
| 1243 | + categories: [cyber] | |
| 1244 | + tier: C | |
| 1245 | + country: NL | |
| 1246 | + discover: { rss: false } | |
| 1247 | + sensors: | |
| 1248 | + - { name: blog feed, url: "https://blog.eclecticiq.com/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1249 | + - id: sygnia | |
| 1250 | + name: Sygnia | |
| 1251 | + domain: sygnia.co | |
| 1252 | + homepage: https://www.sygnia.co | |
| 1253 | + categories: [cyber] | |
| 1254 | + tier: C | |
| 1255 | + country: IL | |
| 1256 | + discover: { rss: false } | |
| 1257 | + sensors: | |
| 1258 | + - { name: feed, url: "https://www.sygnia.co/feed/", type: RSS, connector: rss, tier: C } | |
| 1259 | + - id: resecurity | |
| 1260 | + name: Resecurity | |
| 1261 | + domain: resecurity.com | |
| 1262 | + homepage: https://www.resecurity.com | |
| 1263 | + categories: [cyber] | |
| 1264 | + tier: C | |
| 1265 | + country: US | |
| 1266 | + discover: { rss: false } | |
| 1267 | + sensors: | |
| 1268 | + - { name: feed, url: "https://www.resecurity.com/feed", type: RSS, connector: rss, tier: C } | |
| 1269 | + - id: ahnlab | |
| 1270 | + name: AhnLab | |
| 1271 | + domain: ahnlab.com | |
| 1272 | + homepage: https://asec.ahnlab.com/en/ | |
| 1273 | + categories: [cyber] | |
| 1274 | + tier: C | |
| 1275 | + country: KR | |
| 1276 | + aliases: [asec, ahnlab security intelligence center] | |
| 1277 | + discover: { rss: false } | |
| 1278 | + sensors: | |
| 1279 | + - { name: asec english feed, url: "https://asec.ahnlab.com/en/feed/", type: RSS, connector: rss, tier: C } | |
| 1280 | + - id: any-run | |
| 1281 | + name: ANY.RUN | |
| 1282 | + domain: any.run | |
| 1283 | + homepage: https://any.run | |
| 1284 | + categories: [cyber] | |
| 1285 | + tier: C | |
| 1286 | + country: AE | |
| 1287 | + aliases: [anyrun sandbox] | |
| 1288 | + discover: { rss: false } | |
| 1289 | + sensors: | |
| 1290 | + - { name: blog feed, url: "https://any.run/cybersecurity-blog/feed/", type: RSS, connector: rss, tier: C } | |
| 1291 | + - id: infoblox | |
| 1292 | + name: Infoblox | |
| 1293 | + domain: infoblox.com | |
| 1294 | + homepage: https://www.infoblox.com | |
| 1295 | + categories: [cyber, internet] | |
| 1296 | + tier: C | |
| 1297 | + country: US | |
| 1298 | + aliases: [infoblox threat intel] | |
| 1299 | + discover: { rss: false } | |
| 1300 | + sensors: | |
| 1301 | + - { name: blog feed, url: "https://www.infoblox.com/blog/feed/", type: RSS, connector: rss, tier: C } | |
| 1302 | + - id: exodus-intelligence | |
| 1303 | + name: Exodus Intelligence | |
| 1304 | + domain: exodusintel.com | |
| 1305 | + homepage: https://www.exodusintel.com | |
| 1306 | + categories: [cyber] | |
| 1307 | + tier: C | |
| 1308 | + country: US | |
| 1309 | + discover: { rss: false } | |
| 1310 | + sensors: | |
| 1311 | + - { name: blog feed, url: "https://blog.exodusintel.com/feed/", type: RSS, connector: rss, tier: C } | |
| 1312 | + - id: reversinglabs | |
| 1313 | + name: ReversingLabs | |
| 1314 | + domain: reversinglabs.com | |
| 1315 | + homepage: https://www.reversinglabs.com | |
| 1316 | + categories: [cyber, developer] | |
| 1317 | + tier: C | |
| 1318 | + country: US | |
| 1319 | + discover: { rss: false } | |
| 1320 | + sensors: | |
| 1321 | + - { name: blog feed, url: "https://www.reversinglabs.com/blog/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1322 | + - id: endor-labs | |
| 1323 | + name: Endor Labs | |
| 1324 | + domain: endorlabs.com | |
| 1325 | + homepage: https://www.endorlabs.com | |
| 1326 | + categories: [cyber, developer] | |
| 1327 | + tier: C | |
| 1328 | + country: US | |
| 1329 | + discover: { rss: false } | |
| 1330 | + sensors: | |
| 1331 | + - { name: learn feed, url: "https://www.endorlabs.com/learn/rss.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 1332 | + - id: legit-security | |
| 1333 | + name: Legit Security | |
| 1334 | + domain: legitsecurity.com | |
| 1335 | + homepage: https://www.legitsecurity.com | |
| 1336 | + categories: [cyber, developer] | |
| 1337 | + tier: D | |
| 1338 | + country: US | |
| 1339 | + discover: { rss: false } | |
| 1340 | + sensors: | |
| 1341 | + - { name: blog feed, url: "https://www.legitsecurity.com/blog/rss.xml", type: RSS, connector: rss, tier: D } | |
| 1342 | + - id: cycode | |
| 1343 | + name: Cycode | |
| 1344 | + domain: cycode.com | |
| 1345 | + homepage: https://cycode.com | |
| 1346 | + categories: [cyber, developer] | |
| 1347 | + tier: D | |
| 1348 | + country: US | |
| 1349 | + discover: { rss: false } | |
| 1350 | + sensors: | |
| 1351 | + - { name: feed, url: "https://cycode.com/feed/", type: RSS, connector: rss, tier: D } | |
| 1352 | + - id: mend | |
| 1353 | + name: Mend.io | |
| 1354 | + domain: mend.io | |
| 1355 | + homepage: https://www.mend.io | |
| 1356 | + categories: [cyber, developer] | |
| 1357 | + tier: D | |
| 1358 | + country: IL | |
| 1359 | + aliases: [whitesource, renovate] | |
| 1360 | + discover: { rss: false } | |
| 1361 | + sensors: | |
| 1362 | + - { name: blog feed, url: "https://www.mend.io/blog/feed/", type: RSS, connector: rss, tier: D } | |
| 1363 | + - id: checkmarx | |
| 1364 | + name: Checkmarx | |
| 1365 | + domain: checkmarx.com | |
| 1366 | + homepage: https://checkmarx.com | |
| 1367 | + categories: [cyber, developer] | |
| 1368 | + tier: D | |
| 1369 | + country: US | |
| 1370 | + discover: { rss: false } | |
| 1371 | + sensors: | |
| 1372 | + - { name: feed, url: "https://checkmarx.com/feed/", type: RSS, connector: rss, tier: D } | |
| 1373 | + - id: trail-of-bits | |
| 1374 | + name: Trail of Bits | |
| 1375 | + domain: trailofbits.com | |
| 1376 | + homepage: https://www.trailofbits.com | |
| 1377 | + categories: [cyber, developer] | |
| 1378 | + tier: C | |
| 1379 | + country: US | |
| 1380 | + discover: { rss: false } | |
| 1381 | + sensors: | |
| 1382 | + - { name: blog feed, url: "https://blog.trailofbits.com/index.xml", type: RSS, connector: rss, tier: C } | |
| 1383 | + - id: bishop-fox | |
| 1384 | + name: Bishop Fox | |
| 1385 | + domain: bishopfox.com | |
| 1386 | + homepage: https://bishopfox.com | |
| 1387 | + categories: [cyber] | |
| 1388 | + tier: C | |
| 1389 | + country: US | |
| 1390 | + discover: { rss: false } | |
| 1391 | + sensors: | |
| 1392 | + - { name: blog feed, url: "https://bishopfox.com/feeds/blog.rss", type: RSS, connector: rss, tier: C } | |
| 1393 | + - id: praetorian | |
| 1394 | + name: Praetorian | |
| 1395 | + domain: praetorian.com | |
| 1396 | + homepage: https://www.praetorian.com | |
| 1397 | + categories: [cyber] | |
| 1398 | + tier: D | |
| 1399 | + country: US | |
| 1400 | + discover: { rss: false } | |
| 1401 | + sensors: | |
| 1402 | + - { name: blog feed, url: "https://www.praetorian.com/blog/feed/", type: RSS, connector: rss, tier: D } | |
| 1403 | + - id: synacktiv | |
| 1404 | + name: Synacktiv | |
| 1405 | + domain: synacktiv.com | |
| 1406 | + homepage: https://www.synacktiv.com | |
| 1407 | + categories: [cyber] | |
| 1408 | + tier: C | |
| 1409 | + country: FR | |
| 1410 | + discover: { rss: false } | |
| 1411 | + sensors: | |
| 1412 | + - { name: english blog feed, url: "https://www.synacktiv.com/en/feed/lastblog.xml", type: RSS, connector: rss, tier: C } | |
| 1413 | + - id: assetnote | |
| 1414 | + name: Assetnote (Searchlight Cyber) | |
| 1415 | + domain: assetnote.io | |
| 1416 | + homepage: https://www.assetnote.io | |
| 1417 | + categories: [cyber] | |
| 1418 | + tier: C | |
| 1419 | + country: AU | |
| 1420 | + discover: { rss: false } | |
| 1421 | + sensors: | |
| 1422 | + - { name: research blog feed, url: "https://blog.assetnote.io/feed.xml", type: RSS, connector: rss, tier: C } | |
| 1423 | + - id: horizon3 | |
| 1424 | + name: Horizon3.ai | |
| 1425 | + domain: horizon3.ai | |
| 1426 | + homepage: https://horizon3.ai | |
| 1427 | + categories: [cyber] | |
| 1428 | + tier: C | |
| 1429 | + country: US | |
| 1430 | + aliases: [horizon3 attack team, nodezero] | |
| 1431 | + discover: { rss: false } | |
| 1432 | + sensors: | |
| 1433 | + - { name: feed, url: "https://horizon3.ai/feed/", type: RSS, connector: rss, tier: C } | |
| 1434 | + - id: watchtowr | |
| 1435 | + name: watchTowr | |
| 1436 | + domain: watchtowr.com | |
| 1437 | + homepage: https://watchtowr.com | |
| 1438 | + categories: [cyber] | |
| 1439 | + tier: B | |
| 1440 | + country: SG | |
| 1441 | + aliases: [watchtowr labs] | |
| 1442 | + discover: { rss: false } | |
| 1443 | + sensors: | |
| 1444 | + - { name: labs feed, url: "https://labs.watchtowr.com/rss/", type: RSS, connector: rss, tier: B } | |
| 1445 | + - id: intigriti | |
| 1446 | + name: Intigriti | |
| 1447 | + domain: intigriti.com | |
| 1448 | + homepage: https://www.intigriti.com | |
| 1449 | + categories: [cyber] | |
| 1450 | + tier: D | |
| 1451 | + country: BE | |
| 1452 | + discover: { rss: false } | |
| 1453 | + sensors: | |
| 1454 | + - { name: blog feed, url: "https://www.intigriti.com/blog/feed", type: ATOM, connector: rss, tier: D } | |
| 1455 | + - id: mdsec | |
| 1456 | + name: MDSec | |
| 1457 | + domain: mdsec.co.uk | |
| 1458 | + homepage: https://www.mdsec.co.uk | |
| 1459 | + categories: [cyber] | |
| 1460 | + tier: C | |
| 1461 | + country: GB | |
| 1462 | + discover: { rss: false } | |
| 1463 | + sensors: | |
| 1464 | + - { name: feed, url: "https://www.mdsec.co.uk/feed/", type: RSS, connector: rss, tier: C } | |
| 1465 | + - id: pentest-partners | |
| 1466 | + name: Pen Test Partners | |
| 1467 | + domain: pentestpartners.com | |
| 1468 | + homepage: https://www.pentestpartners.com | |
| 1469 | + categories: [cyber] | |
| 1470 | + tier: D | |
| 1471 | + country: GB | |
| 1472 | + discover: { rss: false } | |
| 1473 | + sensors: | |
| 1474 | + - { name: feed, url: "https://www.pentestpartners.com/feed/", type: RSS, connector: rss, tier: D } | |
| 1475 | + - id: portswigger | |
| 1476 | + name: PortSwigger | |
| 1477 | + domain: portswigger.net | |
| 1478 | + homepage: https://portswigger.net | |
| 1479 | + categories: [cyber, developer] | |
| 1480 | + tier: C | |
| 1481 | + country: GB | |
| 1482 | + aliases: [burp suite, portswigger research] | |
| 1483 | + discover: { rss: false } | |
| 1484 | + sensors: | |
| 1485 | + - { name: research feed, url: "https://portswigger.net/research/rss", type: RSS, connector: rss, tier: C } | |
| 1486 | + - id: cloudflare | |
| 1487 | + extend: true | |
| 1488 | + sensors: | |
| 1489 | + - { name: security tag feed, url: "https://blog.cloudflare.com/tag/security/rss/", type: RSS, connector: rss, tier: B } | |
| 1490 | + - { name: vulnerabilities tag feed, url: "https://blog.cloudflare.com/tag/vulnerabilities/rss/", type: RSS, connector: rss, tier: A } | |
| 1491 | + - { name: post-mortem tag feed, url: "https://blog.cloudflare.com/tag/post-mortem/rss/", type: RSS, connector: rss, tier: A } | |
| 1492 | + - id: proofpoint | |
| 1493 | + extend: true | |
| 1494 | + country: US | |
| 1495 | + sensors: | |
| 1496 | + - { name: threat insight blog feed, url: "https://www.proofpoint.com/us/threat-insight-blog.xml", type: RSS, connector: rss, tier: B } | |
| 1497 | + - id: zscaler | |
| 1498 | + extend: true | |
| 1499 | + products: | |
| 1500 | + - { name: ThreatLabz, type: service, aliases: [zscaler threatlabz] } | |
| 1501 | + sensors: | |
| 1502 | + - { name: security research blog feed, url: "https://www.zscaler.com/blogs/feeds/security-research", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1503 | + - id: akamai | |
| 1504 | + extend: true | |
| 1505 | + notes: "akamai.com blog RSS paths return 403 (Akamai bot manager); the FeedBurner mirror is the only working feed." | |
| 1506 | + - id: sans-isc | |
| 1507 | + extend: true | |
| 1508 | + country: US | |
| 1509 | + sensors: | |
| 1510 | + - { name: diaries feed (short), url: "https://isc.sans.edu/rssfeed.xml", type: RSS, connector: rss, tier: A } | |
| 1511 | + - id: seclists | |
| 1512 | + name: SecLists.org (Full Disclosure / oss-security) | |
| 1513 | + domain: seclists.org | |
| 1514 | + homepage: https://seclists.org | |
| 1515 | + categories: [cyber, open-source] | |
| 1516 | + tier: B | |
| 1517 | + country: US | |
| 1518 | + aliases: [full disclosure, oss-security, seclists] | |
| 1519 | + first_party: false | |
| 1520 | + llm: false | |
| 1521 | + discover: { rss: false } | |
| 1522 | + sensors: | |
| 1523 | + - { name: full disclosure list feed, url: "https://seclists.org/rss/fulldisclosure.rss", type: RSS, connector: rss, tier: B } | |
| 1524 | + - { name: oss-security list feed, url: "https://seclists.org/rss/oss-sec.rss", type: RSS, connector: rss, tier: A } | |
| 1525 | + | |
| 1526 | + # ───────────────────────── G · Security media & independent researchers (first_party: false) ───────────────────────── | |
| 1527 | + - id: bleepingcomputer | |
| 1528 | + extend: true | |
| 1529 | + country: US | |
| 1530 | + first_party: false | |
| 1531 | + - id: krebs-on-security | |
| 1532 | + extend: true | |
| 1533 | + country: US | |
| 1534 | + first_party: false | |
| 1535 | + - id: the-hacker-news | |
| 1536 | + extend: true | |
| 1537 | + country: IN | |
| 1538 | + first_party: false | |
| 1539 | + - id: the-record | |
| 1540 | + extend: true | |
| 1541 | + country: US | |
| 1542 | + first_party: false | |
| 1543 | + - id: securityweek | |
| 1544 | + name: SecurityWeek | |
| 1545 | + domain: securityweek.com | |
| 1546 | + homepage: https://www.securityweek.com | |
| 1547 | + categories: [news, media, cyber] | |
| 1548 | + tier: A | |
| 1549 | + weight: 0.8 | |
| 1550 | + country: US | |
| 1551 | + first_party: false | |
| 1552 | + llm: false | |
| 1553 | + discover: { rss: false } | |
| 1554 | + sensors: | |
| 1555 | + - { name: feed, url: "https://www.securityweek.com/feed/", type: RSS, connector: rss, tier: A } | |
| 1556 | + - id: dark-reading | |
| 1557 | + name: Dark Reading | |
| 1558 | + domain: darkreading.com | |
| 1559 | + homepage: https://www.darkreading.com | |
| 1560 | + categories: [news, media, cyber] | |
| 1561 | + tier: A | |
| 1562 | + weight: 0.8 | |
| 1563 | + country: US | |
| 1564 | + first_party: false | |
| 1565 | + llm: false | |
| 1566 | + discover: { rss: false } | |
| 1567 | + sensors: | |
| 1568 | + - { name: feed, url: "https://www.darkreading.com/rss.xml", type: RSS, connector: rss, tier: A, config: { maxItems: 50 } } | |
| 1569 | + - id: cyberscoop | |
| 1570 | + name: CyberScoop | |
| 1571 | + domain: cyberscoop.com | |
| 1572 | + homepage: https://cyberscoop.com | |
| 1573 | + categories: [news, media, cyber, politics] | |
| 1574 | + tier: B | |
| 1575 | + weight: 0.8 | |
| 1576 | + country: US | |
| 1577 | + first_party: false | |
| 1578 | + llm: false | |
| 1579 | + discover: { rss: false } | |
| 1580 | + sensors: | |
| 1581 | + - { name: feed, url: "https://cyberscoop.com/feed/", type: RSS, connector: rss, tier: B } | |
| 1582 | + - id: risky-business | |
| 1583 | + name: Risky Business | |
| 1584 | + domain: risky.biz | |
| 1585 | + homepage: https://risky.biz | |
| 1586 | + categories: [news, media, cyber] | |
| 1587 | + tier: B | |
| 1588 | + weight: 0.7 | |
| 1589 | + country: AU | |
| 1590 | + first_party: false | |
| 1591 | + llm: false | |
| 1592 | + aliases: [risky biz, risky business news] | |
| 1593 | + discover: { rss: false } | |
| 1594 | + sensors: | |
| 1595 | + - { name: podcast & newsletter feed, url: "https://risky.biz/rss.xml", type: RSS, connector: rss, tier: B } | |
| 1596 | + - { name: risky business news feed, url: "https://news.risky.biz/rss/", type: RSS, connector: rss, tier: B } | |
| 1597 | + - id: infosecurity-magazine | |
| 1598 | + name: Infosecurity Magazine | |
| 1599 | + domain: infosecurity-magazine.com | |
| 1600 | + homepage: https://www.infosecurity-magazine.com | |
| 1601 | + categories: [news, media, cyber] | |
| 1602 | + tier: B | |
| 1603 | + weight: 0.7 | |
| 1604 | + country: GB | |
| 1605 | + first_party: false | |
| 1606 | + llm: false | |
| 1607 | + discover: { rss: false } | |
| 1608 | + sensors: | |
| 1609 | + - { name: news feed, url: "https://www.infosecurity-magazine.com/rss/news/", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1610 | + - id: help-net-security | |
| 1611 | + name: Help Net Security | |
| 1612 | + domain: helpnetsecurity.com | |
| 1613 | + homepage: https://www.helpnetsecurity.com | |
| 1614 | + categories: [news, media, cyber] | |
| 1615 | + tier: B | |
| 1616 | + weight: 0.6 | |
| 1617 | + country: HR | |
| 1618 | + first_party: false | |
| 1619 | + llm: false | |
| 1620 | + discover: { rss: false } | |
| 1621 | + sensors: | |
| 1622 | + - { name: feed, url: "https://www.helpnetsecurity.com/feed/", type: RSS, connector: rss, tier: B } | |
| 1623 | + - id: security-affairs | |
| 1624 | + name: Security Affairs | |
| 1625 | + domain: securityaffairs.com | |
| 1626 | + homepage: https://securityaffairs.com | |
| 1627 | + categories: [news, media, cyber] | |
| 1628 | + tier: B | |
| 1629 | + weight: 0.6 | |
| 1630 | + country: IT | |
| 1631 | + first_party: false | |
| 1632 | + llm: false | |
| 1633 | + discover: { rss: false } | |
| 1634 | + sensors: | |
| 1635 | + - { name: feed, url: "https://securityaffairs.com/feed", type: RSS, connector: rss, tier: B } | |
| 1636 | + - id: cso-online | |
| 1637 | + name: CSO Online | |
| 1638 | + domain: csoonline.com | |
| 1639 | + homepage: https://www.csoonline.com | |
| 1640 | + categories: [news, media, cyber] | |
| 1641 | + tier: B | |
| 1642 | + weight: 0.7 | |
| 1643 | + country: US | |
| 1644 | + first_party: false | |
| 1645 | + llm: false | |
| 1646 | + aliases: [cso] | |
| 1647 | + discover: { rss: false } | |
| 1648 | + sensors: | |
| 1649 | + - { name: feed, url: "https://www.csoonline.com/feed/", type: RSS, connector: rss, tier: B } | |
| 1650 | + - id: cybersecurity-dive | |
| 1651 | + name: Cybersecurity Dive | |
| 1652 | + domain: cybersecuritydive.com | |
| 1653 | + homepage: https://www.cybersecuritydive.com | |
| 1654 | + categories: [news, media, cyber] | |
| 1655 | + tier: B | |
| 1656 | + weight: 0.7 | |
| 1657 | + country: US | |
| 1658 | + first_party: false | |
| 1659 | + llm: false | |
| 1660 | + discover: { rss: false } | |
| 1661 | + sensors: | |
| 1662 | + - { name: news feed, url: "https://www.cybersecuritydive.com/feeds/news/", type: RSS, connector: rss, tier: B } | |
| 1663 | + - id: databreaches-net | |
| 1664 | + name: DataBreaches.net | |
| 1665 | + domain: databreaches.net | |
| 1666 | + homepage: https://databreaches.net | |
| 1667 | + categories: [news, media, cyber] | |
| 1668 | + tier: B | |
| 1669 | + weight: 0.6 | |
| 1670 | + country: US | |
| 1671 | + first_party: false | |
| 1672 | + llm: false | |
| 1673 | + aliases: [databreaches, dissent doe] | |
| 1674 | + discover: { rss: false } | |
| 1675 | + sensors: | |
| 1676 | + - { name: feed, url: "https://databreaches.net/feed/", type: RSS, connector: rss, tier: B } | |
| 1677 | + - id: schneier-on-security | |
| 1678 | + name: Schneier on Security | |
| 1679 | + domain: schneier.com | |
| 1680 | + homepage: https://www.schneier.com | |
| 1681 | + categories: [media, cyber] | |
| 1682 | + tier: C | |
| 1683 | + weight: 0.7 | |
| 1684 | + country: US | |
| 1685 | + first_party: false | |
| 1686 | + aliases: [bruce schneier] | |
| 1687 | + discover: { rss: false } | |
| 1688 | + sensors: | |
| 1689 | + - { name: feed, url: "https://www.schneier.com/feed/atom/", type: ATOM, connector: rss, tier: C } | |
| 1690 | + - id: troy-hunt | |
| 1691 | + name: Troy Hunt | |
| 1692 | + domain: troyhunt.com | |
| 1693 | + homepage: https://www.troyhunt.com | |
| 1694 | + categories: [media, cyber] | |
| 1695 | + tier: C | |
| 1696 | + weight: 0.6 | |
| 1697 | + country: AU | |
| 1698 | + first_party: false | |
| 1699 | + discover: { rss: false } | |
| 1700 | + sensors: | |
| 1701 | + - { name: feed, url: "https://www.troyhunt.com/rss/", type: RSS, connector: rss, tier: C } | |
| 1702 | + - id: graham-cluley | |
| 1703 | + name: Graham Cluley | |
| 1704 | + domain: grahamcluley.com | |
| 1705 | + homepage: https://grahamcluley.com | |
| 1706 | + categories: [media, cyber] | |
| 1707 | + tier: C | |
| 1708 | + weight: 0.5 | |
| 1709 | + country: GB | |
| 1710 | + first_party: false | |
| 1711 | + aliases: [smashing security] | |
| 1712 | + discover: { rss: false } | |
| 1713 | + sensors: | |
| 1714 | + - { name: feed, url: "https://grahamcluley.com/feed/", type: RSS, connector: rss, tier: C } | |
| 1715 | + - id: heise-security | |
| 1716 | + name: heise Security | |
| 1717 | + domain: heise.de | |
| 1718 | + homepage: https://www.heise.de/security/ | |
| 1719 | + categories: [news, media, cyber] | |
| 1720 | + tier: B | |
| 1721 | + weight: 0.7 | |
| 1722 | + country: DE | |
| 1723 | + language: de | |
| 1724 | + first_party: false | |
| 1725 | + llm: false | |
| 1726 | + aliases: [heise online security, heise] | |
| 1727 | + discover: { rss: false } | |
| 1728 | + sensors: | |
| 1729 | + - { name: security news feed, url: "https://www.heise.de/security/feed.xml", type: ATOM, connector: rss, tier: B } | |
| 1730 | + - { name: security alerts feed, url: "https://www.heise.de/security/Alerts/feed.xml", type: ATOM, connector: rss, tier: A } | |
| 1731 | + - id: golem-security | |
| 1732 | + name: Golem.de Security | |
| 1733 | + domain: golem.de | |
| 1734 | + homepage: https://www.golem.de/specials/security/ | |
| 1735 | + categories: [news, media, cyber] | |
| 1736 | + tier: C | |
| 1737 | + weight: 0.5 | |
| 1738 | + country: DE | |
| 1739 | + language: de | |
| 1740 | + first_party: false | |
| 1741 | + llm: false | |
| 1742 | + discover: { rss: false } | |
| 1743 | + sensors: | |
| 1744 | + - { name: security feed, url: "https://rss.golem.de/rss.php?feed=RSS2.0&ms=security", type: RSS, connector: rss, tier: C } | |
| 1745 | + - id: zataz | |
| 1746 | + name: ZATAZ | |
| 1747 | + domain: zataz.com | |
| 1748 | + homepage: https://www.zataz.com | |
| 1749 | + categories: [news, media, cyber] | |
| 1750 | + tier: C | |
| 1751 | + weight: 0.5 | |
| 1752 | + country: FR | |
| 1753 | + language: fr | |
| 1754 | + first_party: false | |
| 1755 | + llm: false | |
| 1756 | + discover: { rss: false } | |
| 1757 | + sensors: | |
| 1758 | + - { name: feed, url: "https://www.zataz.com/feed/", type: RSS, connector: rss, tier: C } | |
| 1759 | + - id: the-register | |
| 1760 | + extend: true | |
| 1761 | + sensors: | |
| 1762 | + - { name: security headlines, url: "https://www.theregister.com/security/headlines.atom", type: ATOM, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1763 | + - id: ars-technica | |
| 1764 | + extend: true | |
| 1765 | + sensors: | |
| 1766 | + - { name: security feed, url: "https://arstechnica.com/security/feed/", type: RSS, connector: rss, tier: B } | |
| 1767 | + - id: wired | |
| 1768 | + extend: true | |
| 1769 | + sensors: | |
| 1770 | + - { name: security feed, url: "https://www.wired.com/feed/category/security/latest/rss", type: RSS, connector: rss, tier: B } | |
added
config/sources.d/43-finance-markets.yaml
+1278 −0
@@ -0,0 +1,1278 @@ | ||
| 1 | +# config/sources.d/43-finance-markets.yaml — finance depth (authored 2026-09-11): central banks (decisions, data | |
| 2 | +# APIs, release feeds), securities/banking regulators, exchanges & market infrastructure (halts, listings, IPO | |
| 3 | +# calendar, volatility indices), rating agencies & index providers, multilaterals (BIS, IMF, World Bank, OECD), | |
| 4 | +# US Treasury / TreasuryDirect / OFAC, statistics offices (BLS, StatCan, Eurostat, ONS, INSEE, INE, ABS), and | |
| 5 | +# investor relations + SEC EDGAR filings of major financial and Canadian/foreign issuers (earnings, guidance, | |
| 6 | +# dividends, buybacks, M&A, executive changes, capital raises, bankruptcies). Extends organizations declared in | |
| 7 | +# sources.yaml, 11-central-banks-finance.yaml, 30-edgar-filings.yaml and 35-documents-data.yaml; every sensor was | |
| 8 | +# fetched and parsed by apps/engine/src/validate.ts. Blocked or client-rendered endpoints are recorded in `notes:`. | |
| 9 | +# Country codes follow ISO 3166-1 alpha-2 ("EU"/"INT" for supranational bodies); language defaults to English. | |
| 10 | +# Re-checked and found client-rendered/thin on 2026-09-11 (no sensor added): EIOPA filtered press listing, OSC news | |
| 11 | +# listing, LSEG media-centre/press-releases, SGX media-centre, Cboe equities/options notices pages. | |
| 12 | +sources: | |
| 13 | + # ───────────────────────── A · Central banks — decisions, data releases, statistics APIs ───────────────────────── | |
| 14 | + - id: federal-reserve | |
| 15 | + extend: true | |
| 16 | + country: US | |
| 17 | + aliases: [fomc, federal open market committee, the fed, board of governors] | |
| 18 | + notes: "press_* sub-feeds are subsets of press_all (already a sensor) and are not duplicated; FOMC minutes/press-conference/Beige Book pages moved (404); H.4.1 current page is 700 KB and is skipped in favour of the H.4.1 feed + FRED WALCL." | |
| 19 | + sensors: | |
| 20 | + - { name: testimony feed, url: "https://www.federalreserve.gov/feeds/testimony.xml", type: RSS, connector: rss, tier: B } | |
| 21 | + - { name: H.4.1 factors affecting reserve balances feed, url: "https://www.federalreserve.gov/feeds/h41.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 30 } } | |
| 22 | + - { name: H.15 selected interest rates feed, url: "https://www.federalreserve.gov/feeds/h15.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 30 } } | |
| 23 | + - { name: H.10 foreign exchange rates feed, url: "https://www.federalreserve.gov/feeds/h10.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 30 } } | |
| 24 | + - { name: H.8 bank assets and liabilities feed, url: "https://www.federalreserve.gov/feeds/h8.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 30 } } | |
| 25 | + - { name: H.6 money stock feed, url: "https://www.federalreserve.gov/feeds/h6.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 30 } } | |
| 26 | + - { name: H.3 aggregate reserves feed, url: "https://www.federalreserve.gov/feeds/h3.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 30 } } | |
| 27 | + - { name: G.19 consumer credit feed, url: "https://www.federalreserve.gov/feeds/g19.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 30 } } | |
| 28 | + - { name: G.17 industrial production feed, url: "https://www.federalreserve.gov/feeds/g17.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 30 } } | |
| 29 | + - { name: Z.1 financial accounts feed, url: "https://www.federalreserve.gov/feeds/z1.xml", type: RSS, connector: rss, tier: D, config: { maxItems: 30 } } | |
| 30 | + - { name: FEDS working papers feed, url: "https://www.federalreserve.gov/feeds/feds.xml", type: RSS, connector: rss, tier: D } | |
| 31 | + - { name: FEDS notes feed, url: "https://www.federalreserve.gov/feeds/feds_notes.xml", type: RSS, connector: rss, tier: C } | |
| 32 | + - { name: IFDP papers feed, url: "https://www.federalreserve.gov/feeds/ifdp.xml", type: RSS, connector: rss, tier: D } | |
| 33 | + - { name: FOMC calendars and statements, url: "https://www.federalreserve.gov/monetarypolicy/fomccalendars.htm", type: HTML, connector: http, tier: A } | |
| 34 | + - { name: FOMC overview page, url: "https://www.federalreserve.gov/monetarypolicy/fomc.htm", type: HTML, connector: http, tier: B } | |
| 35 | + - id: new-york-fed | |
| 36 | + name: Federal Reserve Bank of New York | |
| 37 | + domain: newyorkfed.org | |
| 38 | + homepage: https://www.newyorkfed.org | |
| 39 | + categories: [finance, central-bank, government, open-data] | |
| 40 | + tier: B | |
| 41 | + weight: 1.2 | |
| 42 | + country: US | |
| 43 | + aliases: [new york fed, ny fed, frbny, federal reserve bank of new york, markets group, sofr] | |
| 44 | + products: | |
| 45 | + - { name: SOFR, type: index, aliases: [secured overnight financing rate] } | |
| 46 | + - { name: EFFR, type: index, aliases: [effective federal funds rate] } | |
| 47 | + discover: { rss: false } | |
| 48 | + notes: "newyorkfed.org RSS paths (/rss/feeds/pressreleases, /speeches) redirect to a 404 page; the Markets Data API (markets.newyorkfed.org) is open. SOMA summary is an ascending 1 200-row series (the jsonlist head cap would keep the oldest rows) — use FRED WALCL instead." | |
| 49 | + sensors: | |
| 50 | + - { name: reference rates (SOFR, EFFR, OBFR, TGCR, BGCR), url: "https://markets.newyorkfed.org/api/rates/all/latest.json", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: refRates, keyField: type, titleTemplate: "{type} {effectiveDate}: {percentRate}", dateField: effectiveDate, compareFields: [effectiveDate, percentRate, average30day, index], urlTemplate: "https://www.newyorkfed.org/markets/reference-rates" } } | |
| 51 | + - { name: secured reference rates, url: "https://markets.newyorkfed.org/api/rates/secured/all/latest.json", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: refRates, keyField: type, titleTemplate: "{type} {effectiveDate}: {percentRate}", dateField: effectiveDate, compareFields: [effectiveDate, percentRate, percentPercentile25, percentPercentile75, volumeInBillions], urlTemplate: "https://www.newyorkfed.org/markets/reference-rates/sofr" } } | |
| 52 | + - { name: unsecured reference rates, url: "https://markets.newyorkfed.org/api/rates/unsecured/all/latest.json", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: refRates, keyField: type, titleTemplate: "{type} {effectiveDate}: {percentRate}", dateField: effectiveDate, compareFields: [effectiveDate, percentRate, percentPercentile25, percentPercentile75, volumeInBillions], urlTemplate: "https://www.newyorkfed.org/markets/reference-rates/effr" } } | |
| 53 | + - { name: SOFR page, url: "https://www.newyorkfed.org/markets/reference-rates/sofr", type: HTML, connector: http, tier: C } | |
| 54 | + - id: fred | |
| 55 | + extend: true | |
| 56 | + country: US | |
| 57 | + sensors: | |
| 58 | + - { name: fed funds daily (DFF), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DFF", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DFF], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 59 | + - { name: EFFR (EFFR), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=EFFR", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [EFFR], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 60 | + - { name: interest on reserve balances (IORB), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=IORB", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [IORB], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 61 | + - { name: prime rate (DPRIME), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DPRIME", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DPRIME], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 62 | + - { name: 3-month treasury yield (DGS3MO), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DGS3MO", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DGS3MO], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 63 | + - { name: 30-year treasury yield (DGS30), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DGS30", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DGS30], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 64 | + - { name: 10-year breakeven inflation (T10YIE), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=T10YIE", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [T10YIE], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 65 | + - { name: high-yield OAS (BAMLH0A0HYM2), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=BAMLH0A0HYM2", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [BAMLH0A0HYM2], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 66 | + - { name: broad dollar index (DTWEXBGS), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DTWEXBGS", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DTWEXBGS], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 67 | + - { name: USD/EUR (DEXUSEU), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DEXUSEU", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DEXUSEU], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 68 | + - { name: JPY/USD (DEXJPUS), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DEXJPUS", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DEXJPUS], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 69 | + - { name: CNY/USD (DEXCHUS), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DEXCHUS", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DEXCHUS], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 70 | + - { name: core CPI (CPILFESL), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=CPILFESL", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [CPILFESL], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 71 | + - { name: core PCE price index (PCEPILFE), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=PCEPILFE", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [PCEPILFE], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 72 | + - { name: overnight reverse repo (RRPONTSYD), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=RRPONTSYD", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [RRPONTSYD], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 73 | + - { name: treasury general account (WTREGEN), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=WTREGEN", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [WTREGEN], dateColumn: observation_date, tail: true, maxRows: 26 } } | |
| 74 | + - { name: total reserves (TOTRESNS), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=TOTRESNS", type: FILE, connector: csv, tier: D, config: { keyColumn: observation_date, compareColumns: [TOTRESNS], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 75 | + - { name: M2 money stock (M2SL), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=M2SL", type: FILE, connector: csv, tier: D, config: { keyColumn: observation_date, compareColumns: [M2SL], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 76 | + - { name: federal debt (GFDEBTN), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=GFDEBTN", type: FILE, connector: csv, tier: D, config: { keyColumn: observation_date, compareColumns: [GFDEBTN], dateColumn: observation_date, tail: true, maxRows: 12 } } | |
| 77 | + - { name: consumer sentiment (UMCSENT), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=UMCSENT", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [UMCSENT], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 78 | + - { name: housing starts (HOUST), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=HOUST", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [HOUST], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 79 | + - { name: retail sales (RSAFS), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=RSAFS", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [RSAFS], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 80 | + - { name: industrial production (INDPRO), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=INDPRO", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [INDPRO], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 81 | + - { name: durable goods orders (DGORDER), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DGORDER", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DGORDER], dateColumn: observation_date, tail: true, maxRows: 24 } } | |
| 82 | + - { name: Brent crude spot (DCOILBRENTEU), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DCOILBRENTEU", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DCOILBRENTEU], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 83 | + - { name: Henry Hub natural gas spot (DHHNGSP), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DHHNGSP", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DHHNGSP], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 84 | + - { name: S&P 500 (SP500), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=SP500", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [SP500], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 85 | + - { name: Nasdaq Composite (NASDAQCOM), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=NASDAQCOM", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [NASDAQCOM], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 86 | + - { name: Dow Jones Industrial Average (DJIA), url: "https://fred.stlouisfed.org/graph/fredgraph.csv?id=DJIA", type: FILE, connector: csv, tier: C, config: { keyColumn: observation_date, compareColumns: [DJIA], dateColumn: observation_date, tail: true, maxRows: 30 } } | |
| 87 | + - id: bank-of-canada | |
| 88 | + extend: true | |
| 89 | + country: CA | |
| 90 | + aliases: [boc, banque du canada, valet api] | |
| 91 | + notes: "Valet CSV has a metadata preamble (JSON used instead); /topic/*/feed/ redirects to HTML; the market-notices feed is empty (dropped). Bond-yield, T-bill and CPI groups are keyed by date with several series compared per row." | |
| 92 | + sensors: | |
| 93 | + - { name: publications feed, url: "https://www.bankofcanada.ca/content_type/publications/feed/", type: RSS, connector: rss, tier: B } | |
| 94 | + - { name: staff analytical notes feed, url: "https://www.bankofcanada.ca/content_type/staff-analytical-notes/feed/", type: RSS, connector: rss, tier: C } | |
| 95 | + - { name: all content feed, url: "https://www.bankofcanada.ca/feed/", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 96 | + - { name: USD/CAD daily rate feed (Valet), url: "https://www.bankofcanada.ca/valet/fx_rss/FXUSDCAD", type: RSS, connector: rss, tier: C } | |
| 97 | + - { name: benchmark bond yields (Valet BOND_YIELDS_ALL), url: "https://www.bankofcanada.ca/valet/observations/group/BOND_YIELDS_ALL/json?recent=5", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: observations, keyField: d, titleTemplate: "Canada benchmark yields {d}: 2y {BD.CDN.2YR.DQ.YLD.v} · 5y {BD.CDN.5YR.DQ.YLD.v} · 10y {BD.CDN.10YR.DQ.YLD.v} · long {BD.CDN.LONG.DQ.YLD.v}", dateField: d, compareFields: [BD.CDN.2YR.DQ.YLD.v, BD.CDN.5YR.DQ.YLD.v, BD.CDN.10YR.DQ.YLD.v, BD.CDN.LONG.DQ.YLD.v, BD.CDN.RRB.DQ.YLD.v], urlTemplate: "https://www.bankofcanada.ca/rates/interest-rates/canadian-bonds/" } } | |
| 98 | + - { name: treasury bill yields (Valet tbill_all), url: "https://www.bankofcanada.ca/valet/observations/group/tbill_all/json?recent=5", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: observations, keyField: d, titleTemplate: "Canada T-bill yields {d}: 1m {V80691342.v} · 3m {V80691344.v} · 6m {V80691345.v} · 1y {V80691346.v}", dateField: d, compareFields: [V80691342.v, V80691344.v, V80691345.v, V80691346.v], urlTemplate: "https://www.bankofcanada.ca/rates/interest-rates/t-bill-yields/" } } | |
| 99 | + - { name: CPI and core measures (Valet CPI_MONTHLY), url: "https://www.bankofcanada.ca/valet/observations/group/CPI_MONTHLY/json?recent=3", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: observations, keyField: d, titleTemplate: "Canada CPI {d}: total {STATIC_TOTALCPICHANGE.v} % · trim {CPI_TRIM.v} · median {CPI_MEDIAN.v} · common {CPI_COMMON.v}", dateField: d, compareFields: [V41690973.v, STATIC_TOTALCPICHANGE.v, CPI_TRIM.v, CPI_MEDIAN.v, CPI_COMMON.v], urlTemplate: "https://www.bankofcanada.ca/rates/price-indexes/cpi/" } } | |
| 100 | + - { name: prime rate (Valet V80691311), url: "https://www.bankofcanada.ca/valet/observations/V80691311/json?recent=6", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: observations, keyField: d, titleTemplate: "Canadian prime rate {d}: {V80691311.v} %", dateField: d, compareFields: [V80691311.v], urlTemplate: "https://www.bankofcanada.ca/rates/daily-digest/" } } | |
| 101 | + - { name: bank rate monthly (Valet V122530), url: "https://www.bankofcanada.ca/valet/observations/V122530/json?recent=6", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: observations, keyField: d, titleTemplate: "Bank rate {d}: {V122530.v} %", dateField: d, compareFields: [V122530.v], urlTemplate: "https://www.bankofcanada.ca/rates/interest-rates/" } } | |
| 102 | + - { name: 5-year conventional mortgage rate (Valet V80691335), url: "https://www.bankofcanada.ca/valet/observations/V80691335/json?recent=6", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: observations, keyField: d, titleTemplate: "5-year conventional mortgage rate {d}: {V80691335.v} %", dateField: d, compareFields: [V80691335.v], urlTemplate: "https://www.bankofcanada.ca/rates/daily-digest/" } } | |
| 103 | + - { name: chartered bank interest rates (Valet group), url: "https://www.bankofcanada.ca/valet/observations/group/chartered_bank_interest/json?recent=3", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: observations, keyField: d, titleTemplate: "Chartered bank rates {d}: prime {V80691311.v} · 5y mortgage {V80691335.v} · 5y GIC {V80691336.v}", dateField: d, compareFields: [V80691311.v, V80691335.v, V80691336.v], urlTemplate: "https://www.bankofcanada.ca/rates/daily-digest/" } } | |
| 104 | + - { name: monetary policy report page, url: "https://www.bankofcanada.ca/publications/mpr/", type: HTML, connector: http, tier: B } | |
| 105 | + - id: ecb | |
| 106 | + extend: true | |
| 107 | + country: EU | |
| 108 | + aliases: [european central bank, banque centrale européenne, europäische zentralbank, governing council] | |
| 109 | + notes: "Only press, pub, blog and wppub RSS exist (speeches/mp/fie/podcast feeds 404). Data Portal (csvdata) series carry a TITLE column." | |
| 110 | + sensors: | |
| 111 | + - { name: publications feed, url: "https://www.ecb.europa.eu/rss/pub.html", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 112 | + - { name: blog feed, url: "https://www.ecb.europa.eu/rss/blog.html", type: RSS, connector: rss, tier: C } | |
| 113 | + - { name: working papers feed, url: "https://www.ecb.europa.eu/rss/wppub.html", type: RSS, connector: rss, tier: D, config: { maxItems: 50 } } | |
| 114 | + - { name: governing council decisions, url: "https://www.ecb.europa.eu/press/govcdec/html/index.en.html", type: HTML, connector: http, tier: A } | |
| 115 | + - { name: monetary policy statements, url: "https://www.ecb.europa.eu/press/press_conference/monetary-policy-statement/html/index.en.html", type: HTML, connector: http, tier: A } | |
| 116 | + - { name: key ECB interest rates, url: "https://www.ecb.europa.eu/stats/policy_and_exchange_rates/key_ecb_interest_rates/html/index.en.html", type: HTML, connector: http, tier: B } | |
| 117 | + - { name: main refinancing rate (FM MRR_FR), url: "https://data-api.ecb.europa.eu/service/data/FM/B.U2.EUR.4F.KR.MRR_FR.LEV?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: B, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 118 | + - { name: marginal lending facility rate (FM MLFR), url: "https://data-api.ecb.europa.eu/service/data/FM/B.U2.EUR.4F.KR.MLFR.LEV?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 119 | + - { name: euro short-term rate €STR (EST), url: "https://data-api.ecb.europa.eu/service/data/EST/B.EU000A2X2A25.WT?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 120 | + - { name: euro area 10-year AAA yield (YC), url: "https://data-api.ecb.europa.eu/service/data/YC/B.U2.EUR.4F.G_N_A.SV_C_YM.SR_10Y?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 121 | + - { name: GBP/EUR reference rate (EXR), url: "https://data-api.ecb.europa.eu/service/data/EXR/D.GBP.EUR.SP00.A?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 122 | + - { name: JPY/EUR reference rate (EXR), url: "https://data-api.ecb.europa.eu/service/data/EXR/D.JPY.EUR.SP00.A?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 123 | + - { name: CHF/EUR reference rate (EXR), url: "https://data-api.ecb.europa.eu/service/data/EXR/D.CHF.EUR.SP00.A?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 124 | + - { name: CAD/EUR reference rate (EXR), url: "https://data-api.ecb.europa.eu/service/data/EXR/D.CAD.EUR.SP00.A?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 125 | + - { name: CNY/EUR reference rate (EXR), url: "https://data-api.ecb.europa.eu/service/data/EXR/D.CNY.EUR.SP00.A?format=csvdata&lastNObservations=10", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 126 | + - { name: core HICP ex energy and food (ICP XEF000), url: "https://data-api.ecb.europa.eu/service/data/ICP/M.U2.N.XEF000.4.ANR?format=csvdata&lastNObservations=6", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 127 | + - { name: M3 annual growth (BSI), url: "https://data-api.ecb.europa.eu/service/data/BSI/M.U2.Y.V.M30.X.I.U2.2300.Z01.A?format=csvdata&lastNObservations=6", type: FILE, connector: csv, tier: D, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 128 | + - { name: bank lending rate to corporations (MIR), url: "https://data-api.ecb.europa.eu/service/data/MIR/M.U2.B.A2C.AM.R.A.2250.EUR.N?format=csvdata&lastNObservations=6", type: FILE, connector: csv, tier: D, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, titleColumn: TITLE } } | |
| 129 | + - id: bank-of-england | |
| 130 | + extend: true | |
| 131 | + country: GB | |
| 132 | + aliases: [boe, monetary policy committee, mpc, bank rate] | |
| 133 | + notes: "rss/prudential-regulation and rss/research 404; the monetary-policy-summary page moved. Bank Rate comes from the IADB CSV export (series IUDBEDR, daily since 2025)." | |
| 134 | + sensors: | |
| 135 | + - { name: statistics feed, url: "https://www.bankofengland.co.uk/rss/statistics", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 136 | + - { name: events feed, url: "https://www.bankofengland.co.uk/rss/events", type: RSS, connector: rss, tier: D, config: { maxItems: 40 } } | |
| 137 | + - { name: bank rate history page, url: "https://www.bankofengland.co.uk/boeapps/database/Bank-Rate.asp", type: HTML, connector: http, tier: B } | |
| 138 | + - { name: bank rate daily (IADB IUDBEDR), url: "https://www.bankofengland.co.uk/boeapps/database/_iadb-fromshowcolumns.asp?csv.x=yes&Datefrom=01/Jan/2025&Dateto=now&SeriesCodes=IUDBEDR&CSVF=TN&UsingCodes=Y&VPD=Y&VFD=N", type: FILE, connector: csv, tier: B, config: { keyColumn: DATE, compareColumns: [IUDBEDR], dateColumn: DATE, tail: true, maxRows: 30 } } | |
| 139 | + - id: bank-of-japan | |
| 140 | + extend: true | |
| 141 | + country: JP | |
| 142 | + aliases: [boj, nichigin, 日本銀行, monetary policy meeting] | |
| 143 | + notes: "Only whatsnew and statistics RSS exist (release/mopo/research 404); the MPM schedule/minutes page is server-rendered but the decisions index is client-rendered (185 chars, dropped)." | |
| 144 | + sensors: | |
| 145 | + - { name: statistics feed, url: "https://www.boj.or.jp/en/rss/statistics.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 146 | + - { name: MPM schedule and minutes, url: "https://www.boj.or.jp/en/mopo/mpmsche_minu/index.htm", type: HTML, connector: http, tier: B } | |
| 147 | + - { name: call rate statistics page, url: "https://www.boj.or.jp/en/statistics/market/short/mutan/index.htm", type: HTML, connector: http, tier: C } | |
| 148 | + - id: snb | |
| 149 | + extend: true | |
| 150 | + country: CH | |
| 151 | + notes: "data.snb.ch cube API: snboffzisa = official interest rates (semicolon CSV with a 2-line preamble)." | |
| 152 | + sensors: | |
| 153 | + - { name: speeches feed, url: "https://www.snb.ch/public/rss/en/speeches", type: RSS, connector: rss, tier: C, config: { maxItems: 40 } } | |
| 154 | + - { name: official interest rates (data.snb.ch snboffzisa), url: "https://data.snb.ch/api/cube/snboffzisa/data/csv/en", type: FILE, connector: csv, tier: B, config: { delimiter: ";", skipRows: 2, keyColumn: [Date, D0], compareColumns: [Value], dateColumn: Date, tail: true, maxRows: 40 } } | |
| 155 | + - id: rba | |
| 156 | + extend: true | |
| 157 | + country: AU | |
| 158 | + notes: "rss-cb-statistics 404. F1.1 money-market CSV has an 8-line metadata preamble; header row is 'Series ID' (FIRMMCRT = cash rate target, monthly)." | |
| 159 | + sensors: | |
| 160 | + - { name: statement on monetary policy feed, url: "https://www.rba.gov.au/rss/rss-cb-smp.xml", type: RSS, connector: rss, tier: B } | |
| 161 | + - { name: financial stability review feed, url: "https://www.rba.gov.au/rss/rss-cb-fsr.xml", type: RSS, connector: rss, tier: C } | |
| 162 | + - { name: cash rate target page, url: "https://www.rba.gov.au/statistics/cash-rate/", type: HTML, connector: http, tier: A } | |
| 163 | + - { name: money market rates monthly (F1.1 CSV), url: "https://www.rba.gov.au/statistics/tables/csv/f1.1-data.csv", type: FILE, connector: csv, tier: C, config: { skipRows: 8, keyColumn: "Series ID", compareColumns: [FIRMMCRT, FIRMMCRI, FIRMMBAB90], dateColumn: "Series ID", tail: true, maxRows: 24 } } | |
| 164 | + - id: riksbank | |
| 165 | + extend: true | |
| 166 | + country: SE | |
| 167 | + notes: "Only press-release and speeches RSS exist (news/monetary-policy/publications feeds 404). SWEA API 'Latest' returns a single object — fingerprinted on the value only." | |
| 168 | + sensors: | |
| 169 | + - { name: policy rate latest (SWEA SECBREPOEFF), url: "https://api.riksbank.se/swea/v1/Observations/Latest/SECBREPOEFF", type: JSON, connector: http, tier: B, config: { jsonPath: value } } | |
| 170 | + - id: norges-bank | |
| 171 | + extend: true | |
| 172 | + country: "NO" | |
| 173 | + notes: "News/publications/monetary-policy RSS paths 404; the Norges Bank Data API (SDMX CSV, semicolon-separated) is open." | |
| 174 | + sensors: | |
| 175 | + - { name: policy rate daily (IR KPRA), url: "https://data.norges-bank.no/api/data/IR/B.KPRA.SD.R?format=csv&lastNObservations=10&locale=en", type: FILE, connector: csv, tier: B, config: { delimiter: ";", keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, tail: true } } | |
| 176 | + - { name: USD/NOK daily (EXR), url: "https://data.norges-bank.no/api/data/EXR/B.USD.NOK.SP?format=csv&lastNObservations=10&locale=en", type: FILE, connector: csv, tier: C, config: { delimiter: ";", keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD, tail: true } } | |
| 177 | + - id: banco-central-do-brasil | |
| 178 | + extend: true | |
| 179 | + country: BR | |
| 180 | + language: pt | |
| 181 | + sensors: | |
| 182 | + - { name: Copom statements api, url: "https://www.bcb.gov.br/api/servico/sitebcb/copom/comunicados?quantidade=10", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: conteudo, keyField: nro_reuniao, titleField: titulo, dateField: dataReferencia, urlTemplate: "https://www.bcb.gov.br/en/publications/copomstatements" } } | |
| 183 | + - id: pboc | |
| 184 | + extend: true | |
| 185 | + country: CN | |
| 186 | + sensors: | |
| 187 | + - { name: speeches, url: "https://www.pbc.gov.cn/en/3688110/3688175/index.html", type: HTML, connector: http, tier: B } | |
| 188 | + - { name: announcements, url: "https://www.pbc.gov.cn/en/3688110/3688181/index.html", type: HTML, connector: http, tier: A } | |
| 189 | + - id: dnb | |
| 190 | + extend: true | |
| 191 | + country: NL | |
| 192 | + sensors: | |
| 193 | + - { name: supervision publications feed, url: "https://www.dnb.nl/en/rss/13039/4613", type: RSS, connector: rss, tier: B } | |
| 194 | + - id: banque-de-france | |
| 195 | + extend: true | |
| 196 | + country: FR | |
| 197 | + sensors: | |
| 198 | + - { name: press releases, url: "https://www.banque-france.fr/en/press-release", type: HTML, connector: http, tier: A } | |
| 199 | + # ───────────────────────── B · US market regulators & EDGAR live streams ───────────────────────── | |
| 200 | + - id: sec | |
| 201 | + extend: true | |
| 202 | + country: US | |
| 203 | + aliases: [securities and exchange commission, u.s. securities and exchange commission, edgar] | |
| 204 | + notes: "Rulemaking, statements, investor-alert and whistleblower RSS paths 404 after the 2025 site redesign; rss/rules/final.xml is gone (proposed.xml remains). EDGAR full-text search (efts) rejects date-range queries (HTTP 500) and sorts by relevance, so it is not usable as a new-filing detector." | |
| 205 | + sensors: | |
| 206 | + - { name: speeches and statements feed, url: "https://www.sec.gov/news/speeches.rss", type: RSS, connector: rss, tier: B } | |
| 207 | + - { name: administrative proceedings feed, url: "https://www.sec.gov/enforcement-litigation/administrative-proceedings/rss", type: RSS, connector: rss, tier: B } | |
| 208 | + - { name: trading suspensions feed, url: "https://www.sec.gov/enforcement-litigation/trading-suspensions/rss", type: RSS, connector: rss, tier: A } | |
| 209 | + - { name: corporation finance updates feed, url: "https://www.sec.gov/rss/divisions/corpfin/cfnew.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 40 } } | |
| 210 | + - { name: proposed rules feed, url: "https://www.sec.gov/rss/rules/proposed.xml", type: RSS, connector: rss, tier: B } | |
| 211 | + - id: edgar-live | |
| 212 | + name: SEC EDGAR — latest filings | |
| 213 | + domain: sec.gov | |
| 214 | + homepage: https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent | |
| 215 | + categories: [filings, finance, open-data] | |
| 216 | + tier: B | |
| 217 | + weight: 1.1 | |
| 218 | + country: US | |
| 219 | + aliases: [edgar latest filings, edgar getcurrent, edgar full index] | |
| 220 | + discover: { rss: false } | |
| 221 | + llm: false | |
| 222 | + notes: "Market-wide EDGAR streams (hundreds of filings a day): heuristics only, capped. The SC 13D 'getcurrent' feed is empty outside filing hours and is therefore not a sensor. Company-level filings live in 30-edgar-filings.yaml and in the `edgar filings` sensors below." | |
| 223 | + sensors: | |
| 224 | + - { name: latest 8-K filings (all issuers), url: "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&type=8-K&count=40&output=atom", type: ATOM, connector: rss, tier: A, config: { maxItems: 40 } } | |
| 225 | + - { name: latest 10-K filings (all issuers), url: "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&type=10-K&count=40&output=atom", type: ATOM, connector: rss, tier: B, config: { maxItems: 40 } } | |
| 226 | + - { name: latest S-1 registration statements (IPOs), url: "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&type=S-1&count=40&output=atom", type: ATOM, connector: rss, tier: A, config: { maxItems: 40 } } | |
| 227 | + - { name: latest 13F-HR holdings reports, url: "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&type=13F-HR&count=40&output=atom", type: ATOM, connector: rss, tier: C, config: { maxItems: 40 } } | |
| 228 | + - { name: inline XBRL filings feed, url: "https://www.sec.gov/Archives/edgar/xbrl-inline.rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 229 | + - id: cftc | |
| 230 | + extend: true | |
| 231 | + country: US | |
| 232 | + aliases: [commodity futures trading commission] | |
| 233 | + notes: "Speeches and letters RSS 404. Commitments-of-Traders text files (dea/newcot/*.txt) have no header row and cannot be keyed by the csv connector." | |
| 234 | + sensors: | |
| 235 | + - { name: enforcement actions feed, url: "https://www.cftc.gov/RSS/RSSENF/rssenf.xml", type: RSS, connector: rss, tier: A } | |
| 236 | + - { name: all site updates feed, url: "https://www.cftc.gov/rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 237 | + - { name: commitments of traders page, url: "https://www.cftc.gov/MarketReports/CommitmentsofTraders/index.htm", type: HTML, connector: http, tier: C } | |
| 238 | + - id: finra | |
| 239 | + extend: true | |
| 240 | + country: US | |
| 241 | + aliases: [financial industry regulatory authority] | |
| 242 | + sensors: | |
| 243 | + - { name: news releases feed, url: "https://www.finra.org/media-center/newsreleases/rss", type: RSS, connector: rss, tier: A } | |
| 244 | + - { name: notices feed, url: "https://www.finra.org/rules-guidance/notices/rss", type: RSS, connector: rss, tier: B } | |
| 245 | + - { name: disciplinary actions online, url: "https://www.finra.org/rules-guidance/oversight-enforcement/finra-disciplinary-actions-online", type: HTML, connector: http, tier: C } | |
| 246 | + - id: nyse-market-data | |
| 247 | + name: NYSE market data (halts & IPO calendar) | |
| 248 | + domain: nyse.com | |
| 249 | + homepage: https://www.nyse.com/trade/trading-halts | |
| 250 | + categories: [finance, exchange, open-data] | |
| 251 | + tier: A | |
| 252 | + country: US | |
| 253 | + aliases: [nyse trading halts, nyse ipo calendar, new york stock exchange halts] | |
| 254 | + discover: { rss: false } | |
| 255 | + llm: false | |
| 256 | + notes: "Data endpoints of nyse.com that the founding `nyse` source does not carry (the source itself has no feed: /news, /market-status and /api/marketstatus 404). Halt rows disappear when trading resumes — removals are expected." | |
| 257 | + sensors: | |
| 258 | + - { name: current trading halts (CSV), url: "https://www.nyse.com/api/trade-halts/current/download", type: FILE, connector: csv, tier: A, config: { keyColumn: ["Halt Date", "Halt Time", Symbol], compareColumns: [Reason, "Resume Date", "NYSE Resume Time"], titleColumn: Name, dateColumn: "Halt Date", maxRows: 500 } } | |
| 259 | + - { name: IPO calendar api, url: "https://www.nyse.com/api/ipo-center/calendar", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: calendarList, keyField: symbol, titleTemplate: "{issuer_nm} ({symbol}) — {deal_status_desc} on {custom_group_exchange_nm}, expected {expected_dt_report}", compareFields: [deal_status_desc, expected_dt_report, current_file_price_range_usd, offer_px_usd, withdrawn_postponed_txt], urlTemplate: "https://www.nyse.com/ipo-center/filings" } } | |
| 260 | + - id: nasdaq-trader | |
| 261 | + name: Nasdaq Trader (market operations) | |
| 262 | + domain: nasdaqtrader.com | |
| 263 | + homepage: https://www.nasdaqtrader.com | |
| 264 | + categories: [finance, exchange, open-data] | |
| 265 | + tier: A | |
| 266 | + country: US | |
| 267 | + aliases: [nasdaqtrader, nasdaq trade halts, nasdaq listed securities, nasdaq equity trader alerts] | |
| 268 | + discover: { rss: false } | |
| 269 | + llm: false | |
| 270 | + notes: "Only tradehalts and currentheadlines RSS exist (equities/uto/dailylist feeds redirect to an error page). Symbol directories are pipe-delimited and ~5 000 rows: a new row = listing, a removed row = delisting." | |
| 271 | + sensors: | |
| 272 | + - { name: trading halts feed, url: "https://www.nasdaqtrader.com/rss.aspx?feed=tradehalts", type: RSS, connector: rss, tier: A, config: { maxItems: 60 } } | |
| 273 | + - { name: equity trader alerts feed, url: "https://www.nasdaqtrader.com/rss.aspx?feed=currentheadlines", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 274 | + - { name: Nasdaq-listed securities directory, url: "https://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt", type: FILE, connector: csv, tier: B, config: { delimiter: "|", keyColumn: Symbol, compareColumns: ["Security Name", "Market Category", "Financial Status", "Test Issue"], titleColumn: "Security Name", maxRows: 7000 } } | |
| 275 | + - { name: other-listed securities directory (NYSE, Arca, Cboe), url: "https://www.nasdaqtrader.com/dynamic/SymDir/otherlisted.txt", type: FILE, connector: csv, tier: C, config: { delimiter: "|", keyColumn: "ACT Symbol", compareColumns: ["Security Name", Exchange, ETF, "Test Issue"], titleColumn: "Security Name", maxRows: 9000 } } | |
| 276 | + - id: cboe-indices | |
| 277 | + extend: true | |
| 278 | + country: US | |
| 279 | + products: | |
| 280 | + - { name: VVIX, type: index, aliases: [vix of vix] } | |
| 281 | + - { name: SKEW, type: index, aliases: [cboe skew index] } | |
| 282 | + sensors: | |
| 283 | + - { name: VIX3M 3-month volatility (CSV), url: "https://cdn.cboe.com/api/global/us_indices/daily_prices/VIX3M_History.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: DATE, compareColumns: [CLOSE], tail: true, maxRows: 30 } } | |
| 284 | + - { name: VIX9D 9-day volatility (CSV), url: "https://cdn.cboe.com/api/global/us_indices/daily_prices/VIX9D_History.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: DATE, compareColumns: [CLOSE], tail: true, maxRows: 30 } } | |
| 285 | + - { name: VVIX (CSV), url: "https://cdn.cboe.com/api/global/us_indices/daily_prices/VVIX_History.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: DATE, compareColumns: [VVIX], tail: true, maxRows: 30 } } | |
| 286 | + - { name: SKEW (CSV), url: "https://cdn.cboe.com/api/global/us_indices/daily_prices/SKEW_History.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: DATE, compareColumns: [SKEW], tail: true, maxRows: 30 } } | |
| 287 | + - { name: VXN Nasdaq-100 volatility (CSV), url: "https://cdn.cboe.com/api/global/us_indices/daily_prices/VXN_History.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: DATE, compareColumns: [CLOSE], tail: true, maxRows: 30 } } | |
| 288 | + - { name: OVX crude oil volatility (CSV), url: "https://cdn.cboe.com/api/global/us_indices/daily_prices/OVX_History.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: DATE, compareColumns: [OVX], tail: true, maxRows: 30 } } | |
| 289 | + - { name: GVZ gold volatility (CSV), url: "https://cdn.cboe.com/api/global/us_indices/daily_prices/GVZ_History.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: DATE, compareColumns: [GVZ], tail: true, maxRows: 30 } } | |
| 290 | + - { name: VXTLT treasury volatility (CSV), url: "https://cdn.cboe.com/api/global/us_indices/daily_prices/VXTLT_History.csv", type: FILE, connector: csv, tier: C, config: { keyColumn: DATE, compareColumns: [VXTLT], tail: true, maxRows: 30 } } | |
| 291 | + # ───────────────────────── C · Exchanges abroad ───────────────────────── | |
| 292 | + - id: asx | |
| 293 | + extend: true | |
| 294 | + country: AU | |
| 295 | + notes: "ASX company announcements API (asx.api.markitdigital.com) is open — used here for ASX Ltd itself." | |
| 296 | + sensors: | |
| 297 | + - { name: ASX Ltd company announcements api, url: "https://asx.api.markitdigital.com/asx-research/1.0/companies/asx/announcements", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: data.items, keyField: documentKey, titleField: headline, dateField: date, compareFields: [isPriceSensitive, announcementType], urlTemplate: "https://www.asx.com.au/markets/trade-our-cash-market/announcements.asx" } } | |
| 298 | + - id: hkex | |
| 299 | + extend: true | |
| 300 | + country: HK | |
| 301 | + sensors: | |
| 302 | + - { name: regulatory announcements, url: "https://www.hkex.com.hk/News/Regulatory-Announcements?sc_lang=en", type: HTML, connector: http, tier: B } | |
| 303 | + - { name: market communications, url: "https://www.hkex.com.hk/News/Market-Communications?sc_lang=en", type: HTML, connector: http, tier: B } | |
| 304 | + - id: deutsche-boerse | |
| 305 | + extend: true | |
| 306 | + country: DE | |
| 307 | + sensors: | |
| 308 | + - { name: Eurex news center, url: "https://www.eurex.com/ex-en/find/news-center", type: HTML, connector: http, tier: B } | |
| 309 | + - id: fca | |
| 310 | + extend: true | |
| 311 | + country: GB | |
| 312 | + notes: "Press-release/news-story sub-feeds 404; the warnings feed (unauthorised firms) is high-volume." | |
| 313 | + sensors: | |
| 314 | + - { name: warnings feed (unauthorised firms), url: "https://www.fca.org.uk/news/warnings/rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 315 | + - { name: publications feed, url: "https://www.fca.org.uk/publications/rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 316 | + - id: amf-france | |
| 317 | + extend: true | |
| 318 | + country: FR | |
| 319 | + sensors: | |
| 320 | + - { name: news feed (actualités), url: "https://www.amf-france.org/en/flux-rss/display/30", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 321 | + - { name: publications feed, url: "https://www.amf-france.org/en/flux-rss/display/25", type: RSS, connector: rss, tier: C, config: { maxItems: 60 } } | |
| 322 | + - { name: public offers feed (sociétés concernées par une offre), url: "https://www.amf-france.org/en/flux-rss/display/29", type: RSS, connector: rss, tier: B } | |
| 323 | + - id: afm | |
| 324 | + name: Autoriteit Financiële Markten | |
| 325 | + domain: afm.nl | |
| 326 | + homepage: https://www.afm.nl/en | |
| 327 | + categories: [finance, regulator, government] | |
| 328 | + tier: B | |
| 329 | + country: NL | |
| 330 | + aliases: [afm, autoriteit financiële markten, dutch authority for the financial markets, netherlands authority for the financial markets] | |
| 331 | + discover: { rss: false } | |
| 332 | + sensors: | |
| 333 | + - { name: news feed, url: "https://www.afm.nl/en/rss", type: RSS, connector: rss, tier: B } | |
| 334 | + - id: cssf | |
| 335 | + name: CSSF Luxembourg | |
| 336 | + domain: cssf.lu | |
| 337 | + homepage: https://www.cssf.lu/en/ | |
| 338 | + categories: [finance, regulator, government] | |
| 339 | + tier: B | |
| 340 | + country: LU | |
| 341 | + aliases: [cssf, commission de surveillance du secteur financier] | |
| 342 | + discover: { rss: false } | |
| 343 | + sensors: | |
| 344 | + - { name: news feed, url: "https://www.cssf.lu/en/feed/", type: RSS, connector: rss, tier: B } | |
| 345 | + - id: sfc-hk | |
| 346 | + extend: true | |
| 347 | + country: HK | |
| 348 | + sensors: | |
| 349 | + - { name: enforcement news feed, url: "https://www.sfc.hk/en/RSS-Feeds/Enforcement-news", type: RSS, connector: rss, tier: A } | |
| 350 | + - id: cdic | |
| 351 | + name: Canada Deposit Insurance Corporation | |
| 352 | + domain: cdic.ca | |
| 353 | + homepage: https://www.cdic.ca | |
| 354 | + categories: [finance, regulator, government] | |
| 355 | + tier: B | |
| 356 | + country: CA | |
| 357 | + aliases: [cdic, canada deposit insurance corporation, sadc, société d'assurance-dépôts du canada] | |
| 358 | + discover: { rss: false } | |
| 359 | + sensors: | |
| 360 | + - { name: news feed, url: "https://www.cdic.ca/feed/", type: RSS, connector: rss, tier: B } | |
| 361 | + - id: fsrao | |
| 362 | + name: Financial Services Regulatory Authority of Ontario | |
| 363 | + domain: fsrao.ca | |
| 364 | + homepage: https://www.fsrao.ca | |
| 365 | + categories: [finance, regulator, government] | |
| 366 | + tier: C | |
| 367 | + country: CA | |
| 368 | + aliases: [fsra, fsrao, financial services regulatory authority of ontario] | |
| 369 | + discover: { rss: false } | |
| 370 | + sensors: | |
| 371 | + - { name: announcements, url: "https://www.fsrao.ca/announcements", type: HTML, connector: http, tier: C } | |
| 372 | + - id: bcsc | |
| 373 | + name: British Columbia Securities Commission | |
| 374 | + domain: bcsc.bc.ca | |
| 375 | + homepage: https://www.bcsc.bc.ca | |
| 376 | + categories: [finance, regulator, government] | |
| 377 | + tier: C | |
| 378 | + country: CA | |
| 379 | + aliases: [bcsc, british columbia securities commission] | |
| 380 | + discover: { rss: false } | |
| 381 | + sensors: | |
| 382 | + - { name: media room, url: "https://www.bcsc.bc.ca/about/media-room", type: HTML, connector: http, tier: C } | |
| 383 | + - id: cfpb | |
| 384 | + extend: true | |
| 385 | + country: US | |
| 386 | + sensors: | |
| 387 | + - { name: enforcement actions feed, url: "https://www.consumerfinance.gov/enforcement/actions/feed/", type: RSS, connector: rss, tier: A } | |
| 388 | + - { name: final rules feed, url: "https://www.consumerfinance.gov/rules-policy/final-rules/feed/", type: RSS, connector: rss, tier: B } | |
| 389 | + - { name: research reports feed, url: "https://www.consumerfinance.gov/data-research/research-reports/feed/", type: RSS, connector: rss, tier: C } | |
| 390 | + - id: occ | |
| 391 | + extend: true | |
| 392 | + country: US | |
| 393 | + aliases: [office of the comptroller of the currency] | |
| 394 | + sensors: | |
| 395 | + - { name: bulletins feed, url: "https://www.occ.gov/rss/occ_bulletins.xml", type: RSS, connector: rss, tier: B } | |
| 396 | + - { name: enforcement actions index, url: "https://www.occ.gov/topics/laws-and-regulations/enforcement-actions/index-enforcement-actions.html", type: HTML, connector: http, tier: B } | |
| 397 | + - id: fdic | |
| 398 | + extend: true | |
| 399 | + country: US | |
| 400 | + aliases: [federal deposit insurance corporation, bankfind] | |
| 401 | + notes: "banklist.csv now serves the HTML page; the BankFind Suite API (banks.data.fdic.gov) provides failures as JSON." | |
| 402 | + sensors: | |
| 403 | + - { name: bank failures api (BankFind), url: "https://banks.data.fdic.gov/api/failures?sort_by=FAILDATE&sort_order=DESC&limit=25&fields=NAME,CITYST,FAILDATE,SAVR,RESTYPE,QBFASSET,QBFDEP,CERT&format=json", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: data, keyField: data.CERT, titleTemplate: "{data.NAME} ({data.CITYST}) failed {data.FAILDATE} — acquirer {data.SAVR}", dateField: data.FAILDATE, compareFields: [data.SAVR, data.RESTYPE, data.QBFASSET, data.QBFDEP], urlTemplate: "https://www.fdic.gov/bank-failures/failed-bank-list" } } | |
| 404 | + - { name: financial institution letters, url: "https://www.fdic.gov/news/financial-institution-letters", type: HTML, connector: http, tier: C } | |
| 405 | + - { name: quarterly banking profile, url: "https://www.fdic.gov/quarterly-banking-profile", type: HTML, connector: http, tier: D } | |
| 406 | + - id: irs | |
| 407 | + extend: true | |
| 408 | + country: US | |
| 409 | + sensors: | |
| 410 | + - { name: news releases for current month, url: "https://www.irs.gov/newsroom/news-releases-for-current-month", type: HTML, connector: http, tier: B } | |
| 411 | + - { name: internal revenue bulletin, url: "https://www.irs.gov/irb", type: HTML, connector: http, tier: C } | |
| 412 | + - id: pcaob | |
| 413 | + name: PCAOB | |
| 414 | + domain: pcaobus.org | |
| 415 | + homepage: https://pcaobus.org | |
| 416 | + categories: [finance, regulator] | |
| 417 | + tier: C | |
| 418 | + country: US | |
| 419 | + aliases: [pcaob, public company accounting oversight board] | |
| 420 | + discover: { rss: false } | |
| 421 | + notes: "NOT COVERED 2026-09-11: no RSS (rss paths 404) and the news-release listing is client-rendered (185 chars of text). Listed for entity linking only." | |
| 422 | + - id: sipc | |
| 423 | + name: SIPC | |
| 424 | + domain: sipc.org | |
| 425 | + homepage: https://www.sipc.org | |
| 426 | + categories: [finance, regulator] | |
| 427 | + tier: C | |
| 428 | + country: US | |
| 429 | + aliases: [sipc, securities investor protection corporation] | |
| 430 | + discover: { rss: false } | |
| 431 | + sensors: | |
| 432 | + - { name: news releases, url: "https://www.sipc.org/news-and-media/news-releases/", type: HTML, connector: http, tier: C } | |
| 433 | + - id: msrb | |
| 434 | + name: MSRB | |
| 435 | + domain: msrb.org | |
| 436 | + homepage: https://www.msrb.org | |
| 437 | + categories: [finance, regulator] | |
| 438 | + tier: C | |
| 439 | + country: US | |
| 440 | + aliases: [msrb, municipal securities rulemaking board, emma] | |
| 441 | + discover: { rss: false } | |
| 442 | + sensors: | |
| 443 | + - { name: press releases, url: "https://www.msrb.org/Press-Releases", type: HTML, connector: http, tier: C } | |
| 444 | + # ───────────────────────── E · Multilaterals & international standard setters ───────────────────────── | |
| 445 | + - id: bis | |
| 446 | + extend: true | |
| 447 | + country: INT | |
| 448 | + notes: "Most /doclist/*.rss feeds (press releases, BCBS, CPMI, papers) 404 — only rss.xml, cbspeeches, bisbulletins and bis_fsi_publs are live. The BIS Stats API (stats.bis.org) serves central-bank policy rates as CSV." | |
| 449 | + sensors: | |
| 450 | + - { name: BIS bulletins feed, url: "https://www.bis.org/doclist/bisbulletins.rss", type: RSS, connector: rss, tier: C } | |
| 451 | + - { name: FSI publications feed, url: "https://www.bis.org/doclist/bis_fsi_publs.rss", type: RSS, connector: rss, tier: D, config: { maxItems: 40 } } | |
| 452 | + - { name: central bank policy rates (WS_CBPOL US/CA/EA/GB/JP), url: "https://stats.bis.org/api/v2/data/dataflow/BIS/WS_CBPOL/1.0/M.US+CA+XM+GB+JP?lastNObservations=3&format=csv", type: FILE, connector: csv, tier: C, config: { keyColumn: [REF_AREA, TIME_PERIOD], compareColumns: [OBS_VALUE], titleColumn: TITLE, dateColumn: TIME_PERIOD } } | |
| 453 | + - id: imf | |
| 454 | + extend: true | |
| 455 | + country: INT | |
| 456 | + aliases: [international monetary fund, world economic outlook, weo] | |
| 457 | + products: | |
| 458 | + - { name: World Economic Outlook, type: product, aliases: [weo] } | |
| 459 | + notes: "imf.org news/publications/RSS paths answer 403 to bots; the DataMapper API (WEO series) is open and is fingerprinted per indicator — any WEO update is one change event." | |
| 460 | + sensors: | |
| 461 | + - { name: WEO real GDP growth (DataMapper NGDP_RPCH), url: "https://www.imf.org/external/datamapper/api/v1/NGDP_RPCH", type: JSON, connector: http, tier: C, config: { jsonPath: values.NGDP_RPCH } } | |
| 462 | + - { name: WEO inflation (DataMapper PCPIPCH), url: "https://www.imf.org/external/datamapper/api/v1/PCPIPCH", type: JSON, connector: http, tier: C, config: { jsonPath: values.PCPIPCH } } | |
| 463 | + - { name: WEO unemployment rate (DataMapper LUR), url: "https://www.imf.org/external/datamapper/api/v1/LUR", type: JSON, connector: http, tier: D, config: { jsonPath: values.LUR } } | |
| 464 | + - { name: WEO gross debt to GDP (DataMapper GGXWDG_NGDP), url: "https://www.imf.org/external/datamapper/api/v1/GGXWDG_NGDP", type: JSON, connector: http, tier: D, config: { jsonPath: values.GGXWDG_NGDP } } | |
| 465 | + - id: world-bank | |
| 466 | + extend: true | |
| 467 | + country: INT | |
| 468 | + notes: "No RSS (news rss paths 404; the search API returns an object keyed by document id, not a list). Indicators API v2 is open but slow — one country per sensor so rows can be keyed by year." | |
| 469 | + sensors: | |
| 470 | + - { name: world GDP growth (API NY.GDP.MKTP.KD.ZG WLD), url: "https://api.worldbank.org/v2/country/WLD/indicator/NY.GDP.MKTP.KD.ZG?format=json&date=2020:2030", type: REST_API, connector: jsonlist, tier: D, config: { itemsPath: "[1]", keyField: date, titleTemplate: "World GDP growth {date}: {value} %", dateField: date, compareFields: [value], urlTemplate: "https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG", timeoutMs: 60000 } } | |
| 471 | + - { name: world inflation (API FP.CPI.TOTL.ZG WLD), url: "https://api.worldbank.org/v2/country/WLD/indicator/FP.CPI.TOTL.ZG?format=json&date=2020:2030", type: REST_API, connector: jsonlist, tier: D, config: { itemsPath: "[1]", keyField: date, titleTemplate: "World inflation {date}: {value} %", dateField: date, compareFields: [value], urlTemplate: "https://data.worldbank.org/indicator/FP.CPI.TOTL.ZG", timeoutMs: 60000 } } | |
| 472 | + - { name: US GDP growth (API NY.GDP.MKTP.KD.ZG USA), url: "https://api.worldbank.org/v2/country/USA/indicator/NY.GDP.MKTP.KD.ZG?format=json&date=2020:2030", type: REST_API, connector: jsonlist, tier: D, config: { itemsPath: "[1]", keyField: date, titleTemplate: "US GDP growth {date}: {value} %", dateField: date, compareFields: [value], urlTemplate: "https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG?locations=US", timeoutMs: 60000 } } | |
| 473 | + - id: oecd | |
| 474 | + extend: true | |
| 475 | + country: INT | |
| 476 | + notes: "oecd.org news/press/RSS answer 403 to bots; the SDMX Data Explorer API (sdmx.oecd.org, csvfilewithlabels) is open." | |
| 477 | + sensors: | |
| 478 | + - { name: CPI inflation G7/OECD (SDMX DF_PRICES_ALL), url: "https://sdmx.oecd.org/public/rest/data/OECD.SDD.TPS,DSD_PRICES@DF_PRICES_ALL,1.0/G7+OECD+USA+CAN+GBR+DEU+FRA+JPN+ITA.M.N.CPI.PA._T.N.GY?lastNObservations=2&format=csvfilewithlabels", type: FILE, connector: csv, tier: C, config: { keyColumn: [REF_AREA, TIME_PERIOD], compareColumns: [OBS_VALUE], titleColumn: "Reference area", dateColumn: TIME_PERIOD } } | |
| 479 | + - { name: unemployment rates (SDMX DF_IALFS_UNE_M), url: "https://sdmx.oecd.org/public/rest/data/OECD.SDD.TPS,DSD_LFS@DF_IALFS_UNE_M,1.0/USA+CAN+GBR+DEU+FRA+JPN+ITA+G7+OECD..._Z.Y._T.Y_GE15..M?lastNObservations=2&format=csvfilewithlabels", type: FILE, connector: csv, tier: C, config: { keyColumn: [REF_AREA, TIME_PERIOD], compareColumns: [OBS_VALUE], titleColumn: "Reference area", dateColumn: TIME_PERIOD } } | |
| 480 | + - { name: composite leading indicators (SDMX DF_CLI), url: "https://sdmx.oecd.org/public/rest/data/OECD.SDD.STES,DSD_STES@DF_CLI,4.1/USA+CAN+GBR+DEU+FRA+JPN.M.LI...AA...H?lastNObservations=2&format=csvfilewithlabels", type: FILE, connector: csv, tier: C, config: { keyColumn: [REF_AREA, TIME_PERIOD], compareColumns: [OBS_VALUE], titleColumn: "Reference area", dateColumn: TIME_PERIOD } } | |
| 481 | + - { name: short-term interest rates (SDMX KEI IRSTCI), url: "https://sdmx.oecd.org/public/rest/data/OECD.SDD.STES,DSD_KEI@DF_KEI,4.0/USA+CAN+GBR+DEU+FRA+JPN.M.IRSTCI.._Z.._Z...?lastNObservations=2&format=csvfilewithlabels", type: FILE, connector: csv, tier: C, config: { keyColumn: [REF_AREA, TIME_PERIOD], compareColumns: [OBS_VALUE], titleColumn: "Reference area", dateColumn: TIME_PERIOD } } | |
| 482 | + # ───────────────────────── F · US Treasury, TreasuryDirect, OFAC ───────────────────────── | |
| 483 | + - id: us-treasury | |
| 484 | + extend: true | |
| 485 | + country: US | |
| 486 | + aliases: [department of the treasury, ofac, office of foreign assets control, fiscal data] | |
| 487 | + notes: "home.treasury.gov RSS paths never answer (connection stalls). The daily yield-curve CSV carries the calendar year in its path (…/2026/all…): roll the segment each January. FiscalData API rows with several records per date (average rates, cash balance, MSPD) cannot be keyed by the jsonlist connector and are skipped." | |
| 488 | + sensors: | |
| 489 | + - { name: daily treasury par yield curve (CSV, current year), url: "https://home.treasury.gov/resource-center/data-chart-center/interest-rates/daily-treasury-rates.csv/2026/all?type=daily_treasury_yield_curve&field_tdr_date_value=2026&page&_format=csv", type: FILE, connector: csv, tier: B, config: { keyColumn: Date, compareColumns: ["1 Mo", "3 Mo", "6 Mo", "1 Yr", "2 Yr", "5 Yr", "10 Yr", "30 Yr"], dateColumn: Date, maxRows: 30 } } | |
| 490 | + - { name: debt to the penny (FiscalData), url: "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v2/accounting/od/debt_to_penny?sort=-record_date&page[size]=10", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: data, keyField: record_date, titleTemplate: "Total public debt outstanding {record_date}: ${tot_pub_debt_out_amt}", dateField: record_date, compareFields: [tot_pub_debt_out_amt, debt_held_public_amt, intragov_hold_amt], urlTemplate: "https://fiscaldata.treasury.gov/datasets/debt-to-the-penny/debt-to-the-penny" } } | |
| 491 | + - { name: OFAC sanctions list latest changes (XML), url: "https://sanctionslistservice.ofac.treas.gov/changes/latest", type: XML, connector: http, tier: A } | |
| 492 | + - { name: OFAC recent actions, url: "https://ofac.treasury.gov/recent-actions", type: HTML, connector: http, tier: A } | |
| 493 | + - id: treasurydirect | |
| 494 | + name: TreasuryDirect (Treasury auctions) | |
| 495 | + domain: treasurydirect.gov | |
| 496 | + homepage: https://www.treasurydirect.gov/auctions/ | |
| 497 | + categories: [finance, government, open-data] | |
| 498 | + tier: B | |
| 499 | + country: US | |
| 500 | + aliases: [treasurydirect, treasury auctions, bureau of the fiscal service auctions] | |
| 501 | + discover: { rss: false } | |
| 502 | + notes: "TA_WS securities web service: announced / auctioned / upcoming auctions as JSON arrays keyed by CUSIP." | |
| 503 | + sensors: | |
| 504 | + - { name: announced auctions api, url: "https://www.treasurydirect.gov/TA_WS/securities/announced?format=json&pagesize=20", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "", keyField: cusip, titleTemplate: "{securityType} {securityTerm} — auction {auctionDate}, offering {offeringAmount}", dateField: announcementDate, compareFields: [auctionDate, issueDate, offeringAmount, reopening], urlTemplate: "https://www.treasurydirect.gov/auctions/announcements-data-results/" } } | |
| 505 | + - { name: auction results api, url: "https://www.treasurydirect.gov/TA_WS/securities/auctioned?format=json&pagesize=20", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "", keyField: cusip, titleTemplate: "{securityType} {securityTerm} auction {auctionDate}: high yield {highYield} · discount rate {highDiscountRate} · bid-to-cover {bidToCoverRatio}", dateField: auctionDate, compareFields: [highYield, highDiscountRate, bidToCoverRatio, totalAccepted, pricePer100], urlTemplate: "https://www.treasurydirect.gov/auctions/announcements-data-results/" } } | |
| 506 | + - { name: upcoming auctions api, url: "https://www.treasurydirect.gov/TA_WS/securities/upcoming?format=json", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "", keyField: cusip, titleTemplate: "Upcoming: {securityType} {securityTerm} on {auctionDate}", dateField: auctionDate, compareFields: [auctionDate, offeringAmount, issueDate], urlTemplate: "https://www.treasurydirect.gov/auctions/upcoming/" } } | |
| 507 | + # ───────────────────────── G · Statistics offices — release feeds & data APIs ───────────────────────── | |
| 508 | + - id: bls | |
| 509 | + extend: true | |
| 510 | + country: US | |
| 511 | + aliases: [bureau of labor statistics, employment situation, jobs report] | |
| 512 | + notes: "Per-release Atom feeds (one per major indicator). The public API v2 returns year+period rows that the jsonlist connector cannot key — FRED series cover the same values." | |
| 513 | + sensors: | |
| 514 | + - { name: employment situation feed, url: "https://www.bls.gov/feed/empsit.rss", type: ATOM, connector: rss, tier: A } | |
| 515 | + - { name: consumer price index feed, url: "https://www.bls.gov/feed/cpi.rss", type: ATOM, connector: rss, tier: A } | |
| 516 | + - { name: producer price index feed, url: "https://www.bls.gov/feed/ppi.rss", type: ATOM, connector: rss, tier: B } | |
| 517 | + - { name: JOLTS feed, url: "https://www.bls.gov/feed/jolts.rss", type: ATOM, connector: rss, tier: B } | |
| 518 | + - { name: employment cost index feed, url: "https://www.bls.gov/feed/eci.rss", type: ATOM, connector: rss, tier: C } | |
| 519 | + - { name: real earnings feed, url: "https://www.bls.gov/feed/realer.rss", type: ATOM, connector: rss, tier: C } | |
| 520 | + - { name: productivity and costs feed, url: "https://www.bls.gov/feed/prod2.rss", type: ATOM, connector: rss, tier: C } | |
| 521 | + - id: statcan | |
| 522 | + extend: true | |
| 523 | + country: CA | |
| 524 | + aliases: [statistics canada, statistique canada, the daily, wds] | |
| 525 | + notes: "Web Data Service vectors from 2025-01 to 2030-12 (≤ 72 monthly rows, safely under the jsonlist head cap); getChangedCubeList needs a YYYY-MM-DD path segment the placeholder engine cannot produce." | |
| 526 | + sensors: | |
| 527 | + - { name: CPI all-items (WDS v41690973), url: "https://www150.statcan.gc.ca/t1/wds/rest/getDataFromVectorByReferencePeriodRange?vectorIds=%2241690973%22&startRefPeriod=2025-01-01&endReferencePeriod=2030-12-31", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "[0].object.vectorDataPoint", keyField: refPer, titleTemplate: "Canada CPI all-items {refPer}: {value}", dateField: releaseTime, compareFields: [value], urlTemplate: "https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1810000401" } } | |
| 528 | + - { name: unemployment rate (WDS v2062815), url: "https://www150.statcan.gc.ca/t1/wds/rest/getDataFromVectorByReferencePeriodRange?vectorIds=%222062815%22&startRefPeriod=2025-01-01&endReferencePeriod=2030-12-31", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "[0].object.vectorDataPoint", keyField: refPer, titleTemplate: "Canada unemployment rate {refPer}: {value} %", dateField: releaseTime, compareFields: [value], urlTemplate: "https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1410028701" } } | |
| 529 | + - { name: employment (WDS v2062811), url: "https://www150.statcan.gc.ca/t1/wds/rest/getDataFromVectorByReferencePeriodRange?vectorIds=%222062811%22&startRefPeriod=2025-01-01&endReferencePeriod=2030-12-31", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "[0].object.vectorDataPoint", keyField: refPer, titleTemplate: "Canada employment {refPer}: {value} thousand", dateField: releaseTime, compareFields: [value], urlTemplate: "https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1410028701" } } | |
| 530 | + - { name: monthly real GDP (WDS v65201210), url: "https://www150.statcan.gc.ca/t1/wds/rest/getDataFromVectorByReferencePeriodRange?vectorIds=%2265201210%22&startRefPeriod=2025-01-01&endReferencePeriod=2030-12-31", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: "[0].object.vectorDataPoint", keyField: refPer, titleTemplate: "Canada monthly GDP {refPer}: {value}", dateField: releaseTime, compareFields: [value], urlTemplate: "https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=3610043401" } } | |
| 531 | + - id: eurostat | |
| 532 | + extend: true | |
| 533 | + country: EU | |
| 534 | + notes: "Catalogue RSS (euro-indicators, news-releases) 404; statistics-update.rss lists every dataset update (capped). SDMX 2.1 CSV works for HICP/GDP; the unemployment SDMX query returns 400 so the JSON-stat statistics API is fingerprinted instead." | |
| 535 | + sensors: | |
| 536 | + - { name: statistics update feed, url: "https://ec.europa.eu/eurostat/api/dissemination/catalogue/rss/en/statistics-update.rss", type: RSS, connector: rss, tier: B, config: { maxItems: 40 } } | |
| 537 | + - { name: HICP annual rate (SDMX prc_hicp_manr), url: "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/prc_hicp_manr/M.RCH_A.CP00.EA+EU+DE+FR+IT+ES?lastNObservations=3&format=SDMX-CSV", type: FILE, connector: csv, tier: B, config: { keyColumn: [geo, TIME_PERIOD], compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD } } | |
| 538 | + - { name: HICP index (SDMX prc_hicp_midx), url: "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/prc_hicp_midx/M.I15.CP00.EA+EU+DE+FR?lastNObservations=3&format=SDMX-CSV", type: FILE, connector: csv, tier: C, config: { keyColumn: [geo, TIME_PERIOD], compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD } } | |
| 539 | + - { name: quarterly GDP growth (SDMX namq_10_gdp), url: "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/namq_10_gdp/Q.CLV_PCH_PRE.SCA.B1GQ.EA20+EU27_2020+DE+FR+IT+ES?lastNObservations=3&format=SDMX-CSV", type: FILE, connector: csv, tier: B, config: { keyColumn: [geo, TIME_PERIOD], compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD } } | |
| 540 | + - { name: unemployment rate (JSON-stat une_rt_m), url: "https://ec.europa.eu/eurostat/api/dissemination/statistics/1.0/data/une_rt_m?geo=EA20&geo=EU27_2020&s_adj=SA&age=TOTAL&unit=PC_ACT&sex=T&lastTimePeriod=3", type: JSON, connector: http, tier: B, config: { jsonPath: value } } | |
| 541 | + - id: ons | |
| 542 | + extend: true | |
| 543 | + country: GB | |
| 544 | + aliases: [office for national statistics, uk statistics] | |
| 545 | + notes: "Time-series /data JSON is an ascending full history (hundreds of rows) — fingerprinted on the months/quarters array rather than keyed (the jsonlist head cap would keep the oldest rows). The generator CSV export has a variable metadata preamble. Bulletin ?rss pages rate-limit (429)." | |
| 546 | + sensors: | |
| 547 | + - { name: CPI annual rate (D7G7), url: "https://www.ons.gov.uk/economy/inflationandpriceindices/timeseries/d7g7/mm23/data", type: JSON, connector: http, tier: B, config: { jsonPath: months } } | |
| 548 | + - { name: CPIH annual rate (L55O), url: "https://www.ons.gov.uk/economy/inflationandpriceindices/timeseries/l55o/mm23/data", type: JSON, connector: http, tier: C, config: { jsonPath: months } } | |
| 549 | + - { name: unemployment rate (MGSX), url: "https://www.ons.gov.uk/employmentandlabourmarket/peoplenotinwork/unemployment/timeseries/mgsx/lms/data", type: JSON, connector: http, tier: B, config: { jsonPath: months } } | |
| 550 | + - { name: average weekly earnings growth (KAC3), url: "https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/earningsandworkinghours/timeseries/kac3/lms/data", type: JSON, connector: http, tier: C, config: { jsonPath: months } } | |
| 551 | + - { name: GDP quarter-on-quarter growth (IHYQ), url: "https://www.ons.gov.uk/economy/grossdomesticproductgdp/timeseries/ihyq/pn2/data", type: JSON, connector: http, tier: B, config: { jsonPath: quarters } } | |
| 552 | + - id: insee | |
| 553 | + extend: true | |
| 554 | + country: FR | |
| 555 | + language: fr | |
| 556 | + notes: "insee.fr RSS paths 404/500; the BDM SDMX API (api.insee.fr) is open without a key for series data — fingerprinted as XML." | |
| 557 | + sensors: | |
| 558 | + - { name: CPI series 001759970 (BDM SDMX), url: "https://api.insee.fr/series/BDM/V1/data/SERIES_BDM/001759970?lastNObservations=3", type: XML, connector: http, tier: C } | |
| 559 | + - id: ine-spain | |
| 560 | + extend: true | |
| 561 | + country: ES | |
| 562 | + notes: "No RSS (all rss paths 404); the Tempus3 JSON API (servicios.ine.es/wstempus) is open." | |
| 563 | + sensors: | |
| 564 | + - { name: CPI monthly change (Tempus IPC206449), url: "https://servicios.ine.es/wstempus/js/EN/DATOS_SERIE/IPC206449?nult=3", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: Data, keyField: Fecha, titleTemplate: "Spain CPI monthly change {Anyo}-{FK_Periodo}: {Valor} %", compareFields: [Valor], urlTemplate: "https://www.ine.es/dyngs/INEbase/en/operacion.htm?c=Estadistica_C&cid=1254736176802&menu=ultiDatos&idp=1254735976607" } } | |
| 565 | + - { name: CPI index (Tempus IPC206446), url: "https://servicios.ine.es/wstempus/js/EN/DATOS_SERIE/IPC206446?nult=3", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: Data, keyField: Fecha, titleTemplate: "Spain CPI index {Anyo}-{FK_Periodo}: {Valor}", compareFields: [Valor], urlTemplate: "https://www.ine.es/dyngs/INEbase/en/operacion.htm?c=Estadistica_C&cid=1254736176802&menu=ultiDatos&idp=1254735976607" } } | |
| 566 | + - id: abs | |
| 567 | + extend: true | |
| 568 | + country: AU | |
| 569 | + notes: "No RSS; the ABS Data API (SDMX csvfile) is open — the CPI dataflow key tried returned 404, labour force works." | |
| 570 | + sensors: | |
| 571 | + - { name: unemployment rate (Data API LF), url: "https://data.api.abs.gov.au/rest/data/ABS,LF,1.0.0/M13.3.1599.20.AUS.M?lastNObservations=3&format=csvfile", type: FILE, connector: csv, tier: C, config: { keyColumn: TIME_PERIOD, compareColumns: [OBS_VALUE], dateColumn: TIME_PERIOD } } | |
| 572 | + # ───────────────────────── H · Investor relations & filings — finance sector ───────────────────────── | |
| 573 | + - id: citigroup | |
| 574 | + extend: true | |
| 575 | + country: US | |
| 576 | + sensors: | |
| 577 | + - { name: press releases, url: "https://www.citigroup.com/global/news/press-release", type: HTML, connector: http, tier: B } | |
| 578 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000831001.json", type: REST_API, connector: edgar, tier: B } | |
| 579 | + - id: charles-schwab | |
| 580 | + extend: true | |
| 581 | + country: US | |
| 582 | + sensors: | |
| 583 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000316709.json", type: REST_API, connector: edgar, tier: B } | |
| 584 | + - id: state-street | |
| 585 | + extend: true | |
| 586 | + country: US | |
| 587 | + sensors: | |
| 588 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000093751.json", type: REST_API, connector: edgar, tier: C } | |
| 589 | + - id: interactive-brokers | |
| 590 | + extend: true | |
| 591 | + country: US | |
| 592 | + sensors: | |
| 593 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001381197.json", type: REST_API, connector: edgar, tier: C } | |
| 594 | + - id: block | |
| 595 | + extend: true | |
| 596 | + country: US | |
| 597 | + sensors: | |
| 598 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001512673.json", type: REST_API, connector: edgar, tier: C } | |
| 599 | + - id: pnc | |
| 600 | + name: PNC Financial Services | |
| 601 | + domain: pnc.com | |
| 602 | + homepage: https://www.pnc.com | |
| 603 | + categories: [finance, banking] | |
| 604 | + tier: B | |
| 605 | + country: US | |
| 606 | + aliases: [pnc, pnc financial services group, pnc bank] | |
| 607 | + discover: { rss: false } | |
| 608 | + sensors: | |
| 609 | + - { name: news releases feed, url: "https://pnc.mediaroom.com/news-releases?pagetemplate=rss", type: RSS, connector: rss, tier: B } | |
| 610 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000713676.json", type: REST_API, connector: edgar, tier: C } | |
| 611 | + - id: us-bancorp | |
| 612 | + name: U.S. Bancorp | |
| 613 | + domain: usbank.com | |
| 614 | + homepage: https://www.usbank.com | |
| 615 | + categories: [finance, banking] | |
| 616 | + tier: B | |
| 617 | + country: US | |
| 618 | + aliases: [u.s. bancorp, us bancorp, u.s. bank, us bank] | |
| 619 | + discover: { rss: false } | |
| 620 | + notes: "ir.usbank.com (Q4) news listing is client-rendered; filings only." | |
| 621 | + sensors: | |
| 622 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000036104.json", type: REST_API, connector: edgar, tier: C } | |
| 623 | + - id: capital-one | |
| 624 | + name: Capital One | |
| 625 | + domain: capitalone.com | |
| 626 | + homepage: https://www.capitalone.com | |
| 627 | + categories: [finance, banking, payments] | |
| 628 | + tier: B | |
| 629 | + country: US | |
| 630 | + aliases: [capital one, capital one financial, discover financial] | |
| 631 | + discover: { rss: false } | |
| 632 | + sensors: | |
| 633 | + - { name: newsroom, url: "https://www.capitalone.com/about/newsroom/", type: HTML, connector: http, tier: B } | |
| 634 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000927628.json", type: REST_API, connector: edgar, tier: C } | |
| 635 | + - id: fannie-mae | |
| 636 | + extend: true | |
| 637 | + country: US | |
| 638 | + sensors: | |
| 639 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000310522.json", type: REST_API, connector: edgar, tier: C } | |
| 640 | + - id: freddie-mac | |
| 641 | + extend: true | |
| 642 | + country: US | |
| 643 | + sensors: | |
| 644 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001026214.json", type: REST_API, connector: edgar, tier: C } | |
| 645 | + - id: blackstone | |
| 646 | + name: Blackstone | |
| 647 | + domain: blackstone.com | |
| 648 | + homepage: https://www.blackstone.com | |
| 649 | + categories: [finance] | |
| 650 | + tier: B | |
| 651 | + weight: 1.1 | |
| 652 | + country: US | |
| 653 | + aliases: [blackstone, blackstone inc, blackstone group, bx] | |
| 654 | + discover: { rss: false } | |
| 655 | + notes: "BLOCKED 2026-09-11: blackstone.com answers 403 to the connector on /news/feed/ and /insights/feed/ (200 to a plain client — UA-based bot management); SEC filings are the open channel." | |
| 656 | + sensors: | |
| 657 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001393818.json", type: REST_API, connector: edgar, tier: B } | |
| 658 | + - id: kkr | |
| 659 | + name: KKR | |
| 660 | + domain: kkr.com | |
| 661 | + homepage: https://www.kkr.com | |
| 662 | + categories: [finance] | |
| 663 | + tier: B | |
| 664 | + country: US | |
| 665 | + aliases: [kkr, kkr & co, kohlberg kravis roberts] | |
| 666 | + discover: { rss: false } | |
| 667 | + notes: "ir.kkr.com press-release page is a 4 KB client-rendered shell; filings only." | |
| 668 | + sensors: | |
| 669 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001404912.json", type: REST_API, connector: edgar, tier: B } | |
| 670 | + - id: apollo-global | |
| 671 | + name: Apollo Global Management | |
| 672 | + domain: apollo.com | |
| 673 | + homepage: https://www.apollo.com | |
| 674 | + categories: [finance] | |
| 675 | + tier: B | |
| 676 | + country: US | |
| 677 | + aliases: [apollo global management, apollo global, apollo, athene] | |
| 678 | + discover: { rss: false } | |
| 679 | + sensors: | |
| 680 | + - { name: press releases, url: "https://www.apollo.com/insights-news/pressreleases", type: HTML, connector: http, tier: B } | |
| 681 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001858681.json", type: REST_API, connector: edgar, tier: B } | |
| 682 | + - id: carlyle | |
| 683 | + name: The Carlyle Group | |
| 684 | + domain: carlyle.com | |
| 685 | + homepage: https://www.carlyle.com | |
| 686 | + categories: [finance] | |
| 687 | + tier: C | |
| 688 | + country: US | |
| 689 | + aliases: [carlyle, carlyle group] | |
| 690 | + discover: { rss: true } | |
| 691 | + sensors: | |
| 692 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001527166.json", type: REST_API, connector: edgar, tier: C } | |
| 693 | + - id: ares-management | |
| 694 | + name: Ares Management | |
| 695 | + domain: aresmgmt.com | |
| 696 | + homepage: https://www.aresmgmt.com | |
| 697 | + categories: [finance] | |
| 698 | + tier: C | |
| 699 | + country: US | |
| 700 | + aliases: [ares management, ares] | |
| 701 | + discover: { rss: true } | |
| 702 | + sensors: | |
| 703 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001176948.json", type: REST_API, connector: edgar, tier: C } | |
| 704 | + - id: t-rowe-price | |
| 705 | + name: T. Rowe Price | |
| 706 | + domain: troweprice.com | |
| 707 | + homepage: https://www.troweprice.com | |
| 708 | + categories: [finance] | |
| 709 | + tier: C | |
| 710 | + country: US | |
| 711 | + aliases: [t. rowe price, t rowe price, troweprice] | |
| 712 | + discover: { rss: true } | |
| 713 | + sensors: | |
| 714 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001113169.json", type: REST_API, connector: edgar, tier: C } | |
| 715 | + - id: franklin-templeton | |
| 716 | + name: Franklin Templeton | |
| 717 | + domain: franklintempleton.com | |
| 718 | + homepage: https://www.franklintempleton.com | |
| 719 | + categories: [finance] | |
| 720 | + tier: C | |
| 721 | + country: US | |
| 722 | + aliases: [franklin templeton, franklin resources] | |
| 723 | + discover: { rss: true } | |
| 724 | + notes: "franklintempleton.com/press-releases is client-rendered (thin); filings only." | |
| 725 | + sensors: | |
| 726 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000038777.json", type: REST_API, connector: edgar, tier: C } | |
| 727 | + - id: invesco | |
| 728 | + name: Invesco | |
| 729 | + domain: invesco.com | |
| 730 | + homepage: https://www.invesco.com/corporate/en/home.html | |
| 731 | + categories: [finance] | |
| 732 | + tier: C | |
| 733 | + country: US | |
| 734 | + aliases: [invesco, invesco ltd] | |
| 735 | + discover: { rss: true } | |
| 736 | + notes: "ir.invesco.com RSS paths redirect to the corporate homepage; filings only." | |
| 737 | + sensors: | |
| 738 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000914208.json", type: REST_API, connector: edgar, tier: C } | |
| 739 | + - id: northern-trust | |
| 740 | + name: Northern Trust | |
| 741 | + domain: northerntrust.com | |
| 742 | + homepage: https://www.northerntrust.com | |
| 743 | + categories: [finance, banking] | |
| 744 | + tier: C | |
| 745 | + country: US | |
| 746 | + aliases: [northern trust] | |
| 747 | + discover: { rss: true } | |
| 748 | + sensors: | |
| 749 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000073124.json", type: REST_API, connector: edgar, tier: C } | |
| 750 | + - id: ameriprise | |
| 751 | + name: Ameriprise Financial | |
| 752 | + domain: ameriprise.com | |
| 753 | + homepage: https://www.ameriprise.com | |
| 754 | + categories: [finance] | |
| 755 | + tier: C | |
| 756 | + country: US | |
| 757 | + aliases: [ameriprise, ameriprise financial] | |
| 758 | + discover: { rss: true } | |
| 759 | + sensors: | |
| 760 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000820027.json", type: REST_API, connector: edgar, tier: C } | |
| 761 | + - id: raymond-james | |
| 762 | + name: Raymond James Financial | |
| 763 | + domain: raymondjames.com | |
| 764 | + homepage: https://www.raymondjames.com | |
| 765 | + categories: [finance] | |
| 766 | + tier: C | |
| 767 | + country: US | |
| 768 | + aliases: [raymond james, raymond james financial] | |
| 769 | + discover: { rss: true } | |
| 770 | + sensors: | |
| 771 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000720005.json", type: REST_API, connector: edgar, tier: C } | |
| 772 | + - id: allstate | |
| 773 | + name: Allstate | |
| 774 | + domain: allstate.com | |
| 775 | + homepage: https://www.allstate.com | |
| 776 | + categories: [finance, insurance] | |
| 777 | + tier: B | |
| 778 | + country: US | |
| 779 | + aliases: [allstate, the allstate corporation] | |
| 780 | + discover: { rss: false } | |
| 781 | + sensors: | |
| 782 | + - { name: newsroom feed, url: "https://www.allstatenewsroom.com/feed/", type: RSS, connector: rss, tier: B } | |
| 783 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000899051.json", type: REST_API, connector: edgar, tier: C } | |
| 784 | + - id: progressive | |
| 785 | + name: Progressive | |
| 786 | + domain: progressive.com | |
| 787 | + homepage: https://www.progressive.com | |
| 788 | + categories: [finance, insurance] | |
| 789 | + tier: C | |
| 790 | + country: US | |
| 791 | + aliases: [progressive, progressive corporation, progressive insurance] | |
| 792 | + discover: { rss: true } | |
| 793 | + sensors: | |
| 794 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000080661.json", type: REST_API, connector: edgar, tier: C } | |
| 795 | + - id: travelers | |
| 796 | + name: Travelers | |
| 797 | + domain: travelers.com | |
| 798 | + homepage: https://www.travelers.com | |
| 799 | + categories: [finance, insurance] | |
| 800 | + tier: C | |
| 801 | + country: US | |
| 802 | + aliases: [travelers, the travelers companies] | |
| 803 | + discover: { rss: true } | |
| 804 | + sensors: | |
| 805 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000086312.json", type: REST_API, connector: edgar, tier: C } | |
| 806 | + - id: hartford | |
| 807 | + name: The Hartford | |
| 808 | + domain: thehartford.com | |
| 809 | + homepage: https://www.thehartford.com | |
| 810 | + categories: [finance, insurance] | |
| 811 | + tier: C | |
| 812 | + country: US | |
| 813 | + aliases: [the hartford, hartford insurance group, hartford financial] | |
| 814 | + discover: { rss: true } | |
| 815 | + sensors: | |
| 816 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000874766.json", type: REST_API, connector: edgar, tier: C } | |
| 817 | + - id: aon | |
| 818 | + name: Aon | |
| 819 | + domain: aon.com | |
| 820 | + homepage: https://www.aon.com | |
| 821 | + categories: [finance, insurance] | |
| 822 | + tier: C | |
| 823 | + country: GB | |
| 824 | + aliases: [aon, aon plc] | |
| 825 | + discover: { rss: true } | |
| 826 | + sensors: | |
| 827 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000315293.json", type: REST_API, connector: edgar, tier: C } | |
| 828 | + - id: chubb | |
| 829 | + extend: true | |
| 830 | + country: CH | |
| 831 | + sensors: | |
| 832 | + - { name: news releases feed, url: "https://news.chubb.com/news-releases?pagetemplate=rss", type: RSS, connector: rss, tier: B } | |
| 833 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000896159.json", type: REST_API, connector: edgar, tier: C } | |
| 834 | + - id: aig | |
| 835 | + extend: true | |
| 836 | + country: US | |
| 837 | + sensors: | |
| 838 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000005272.json", type: REST_API, connector: edgar, tier: C } | |
| 839 | + - id: metlife | |
| 840 | + extend: true | |
| 841 | + country: US | |
| 842 | + sensors: | |
| 843 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001099219.json", type: REST_API, connector: edgar, tier: C } | |
| 844 | + - id: prudential-financial | |
| 845 | + extend: true | |
| 846 | + country: US | |
| 847 | + sensors: | |
| 848 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001137774.json", type: REST_API, connector: edgar, tier: C } | |
| 849 | + # ── Exchanges, rating agencies & index providers as issuers ── | |
| 850 | + - id: moodys | |
| 851 | + extend: true | |
| 852 | + country: US | |
| 853 | + notes: "ratings.moodys.com/rss is an HTML shell and ir.moodys.com (Q4) is blocked — filings are the only open channel." | |
| 854 | + sensors: | |
| 855 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001059556.json", type: REST_API, connector: edgar, tier: B } | |
| 856 | + - id: msci | |
| 857 | + extend: true | |
| 858 | + country: US | |
| 859 | + sensors: | |
| 860 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001408198.json", type: REST_API, connector: edgar, tier: C } | |
| 861 | + - id: ice | |
| 862 | + extend: true | |
| 863 | + country: US | |
| 864 | + sensors: | |
| 865 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001571949.json", type: REST_API, connector: edgar, tier: B } | |
| 866 | + - id: cme-group | |
| 867 | + extend: true | |
| 868 | + country: US | |
| 869 | + sensors: | |
| 870 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001156375.json", type: REST_API, connector: edgar, tier: B } | |
| 871 | + - id: nasdaq | |
| 872 | + extend: true | |
| 873 | + country: US | |
| 874 | + sensors: | |
| 875 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001120193.json", type: REST_API, connector: edgar, tier: C } | |
| 876 | + - id: factset | |
| 877 | + name: FactSet | |
| 878 | + domain: factset.com | |
| 879 | + homepage: https://www.factset.com | |
| 880 | + categories: [finance, enterprise] | |
| 881 | + tier: C | |
| 882 | + country: US | |
| 883 | + aliases: [factset, factset research systems] | |
| 884 | + discover: { rss: false } | |
| 885 | + sensors: | |
| 886 | + - { name: insight blog feed, url: "https://insight.factset.com/rss.xml", type: RSS, connector: rss, tier: C } | |
| 887 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001013237.json", type: REST_API, connector: edgar, tier: C } | |
| 888 | + - id: dow-jones | |
| 889 | + name: Dow Jones | |
| 890 | + domain: dowjones.com | |
| 891 | + homepage: https://www.dowjones.com | |
| 892 | + categories: [finance, media] | |
| 893 | + tier: C | |
| 894 | + country: US | |
| 895 | + aliases: [dow jones, dow jones & company, dow jones newswires] | |
| 896 | + discover: { rss: false } | |
| 897 | + sensors: | |
| 898 | + - { name: press room feed, url: "https://www.dowjones.com/press-room/rss", type: RSS, connector: rss, tier: C } | |
| 899 | + - id: thomson-reuters | |
| 900 | + name: Thomson Reuters | |
| 901 | + domain: thomsonreuters.com | |
| 902 | + homepage: https://www.thomsonreuters.com | |
| 903 | + categories: [finance, media, enterprise] | |
| 904 | + tier: C | |
| 905 | + country: CA | |
| 906 | + aliases: [thomson reuters, thomson reuters corporation, tri] | |
| 907 | + discover: { rss: false } | |
| 908 | + notes: "thomsonreuters.com press-release listing is client-rendered (thin); filings only." | |
| 909 | + sensors: | |
| 910 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001075124.json", type: REST_API, connector: edgar, tier: C } | |
| 911 | + # ── Canadian financials, pension funds & large caps (40-F/6-K filers) ── | |
| 912 | + - id: rbc | |
| 913 | + extend: true | |
| 914 | + country: CA | |
| 915 | + sensors: | |
| 916 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001000275.json", type: REST_API, connector: edgar, tier: B } | |
| 917 | + - id: td-bank | |
| 918 | + extend: true | |
| 919 | + country: CA | |
| 920 | + sensors: | |
| 921 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0000947263.json", type: REST_API, connector: edgar, tier: B } | |
| 922 | + - id: bmo | |
| 923 | + extend: true | |
| 924 | + country: CA | |
| 925 | + sensors: | |
| 926 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0000927971.json", type: REST_API, connector: edgar, tier: B } | |
| 927 | + - id: scotiabank | |
| 928 | + extend: true | |
| 929 | + country: CA | |
| 930 | + sensors: | |
| 931 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0000009631.json", type: REST_API, connector: edgar, tier: B } | |
| 932 | + - id: cibc | |
| 933 | + extend: true | |
| 934 | + country: CA | |
| 935 | + sensors: | |
| 936 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001045520.json", type: REST_API, connector: edgar, tier: B } | |
| 937 | + - id: manulife | |
| 938 | + extend: true | |
| 939 | + country: CA | |
| 940 | + notes: "manulife.com blocks bots; SEC filings (40-F/6-K) are the open channel." | |
| 941 | + sensors: | |
| 942 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001086888.json", type: REST_API, connector: edgar, tier: B } | |
| 943 | + - id: sun-life | |
| 944 | + extend: true | |
| 945 | + country: CA | |
| 946 | + sensors: | |
| 947 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001097362.json", type: REST_API, connector: edgar, tier: C } | |
| 948 | + - id: brookfield | |
| 949 | + extend: true | |
| 950 | + country: CA | |
| 951 | + aliases: [brookfield corporation, brookfield asset management, bam, bn] | |
| 952 | + products: | |
| 953 | + - { name: Brookfield Asset Management, type: product, aliases: [bam] } | |
| 954 | + sensors: | |
| 955 | + - { name: Brookfield Corporation press releases feed, url: "https://bn.brookfield.com/rss/press-releases.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 40 } } | |
| 956 | + - { name: Brookfield Asset Management press releases feed, url: "https://bam.brookfield.com/rss/press-releases.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 40 } } | |
| 957 | + - { name: edgar filings — Brookfield Corporation (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001001085.json", type: REST_API, connector: edgar, tier: C } | |
| 958 | + - { name: edgar filings — Brookfield Asset Management (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001937926.json", type: REST_API, connector: edgar, tier: C } | |
| 959 | + - id: power-corporation | |
| 960 | + name: Power Corporation of Canada | |
| 961 | + domain: powercorporation.com | |
| 962 | + homepage: https://www.powercorporation.com/en/ | |
| 963 | + categories: [finance, insurance] | |
| 964 | + tier: C | |
| 965 | + country: CA | |
| 966 | + aliases: [power corporation, power corporation of canada, power corp, power financial] | |
| 967 | + discover: { rss: false, sitemap: false } | |
| 968 | + notes: "BLOCKED 2026-09-11: /en/news/press-releases/ answers 403 to the connector (no RSS). Listed for entity linking only." | |
| 969 | + - id: bci | |
| 970 | + name: British Columbia Investment Management Corporation | |
| 971 | + domain: bci.ca | |
| 972 | + homepage: https://www.bci.ca | |
| 973 | + categories: [finance] | |
| 974 | + tier: C | |
| 975 | + country: CA | |
| 976 | + aliases: [bci, british columbia investment management corporation, bcimc] | |
| 977 | + discover: { rss: false } | |
| 978 | + sensors: | |
| 979 | + - { name: news feed, url: "https://www.bci.ca/feed/", type: RSS, connector: rss, tier: C } | |
| 980 | + - id: hoopp | |
| 981 | + name: HOOPP | |
| 982 | + domain: hoopp.com | |
| 983 | + homepage: https://hoopp.com | |
| 984 | + categories: [finance] | |
| 985 | + tier: C | |
| 986 | + country: CA | |
| 987 | + aliases: [hoopp, healthcare of ontario pension plan] | |
| 988 | + discover: { rss: false } | |
| 989 | + sensors: | |
| 990 | + - { name: news, url: "https://hoopp.com/news", type: HTML, connector: http, tier: C } | |
| 991 | + - id: aimco | |
| 992 | + name: AIMCo | |
| 993 | + domain: aimco.ca | |
| 994 | + homepage: https://www.aimco.ca | |
| 995 | + categories: [finance] | |
| 996 | + tier: C | |
| 997 | + country: CA | |
| 998 | + aliases: [aimco, alberta investment management corporation] | |
| 999 | + discover: { rss: false } | |
| 1000 | + sensors: | |
| 1001 | + - { name: insights and news, url: "https://www.aimco.ca/insights", type: HTML, connector: http, tier: C } | |
| 1002 | + - id: cpkc | |
| 1003 | + extend: true | |
| 1004 | + country: CA | |
| 1005 | + sensors: | |
| 1006 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0000016875.json", type: REST_API, connector: edgar, tier: C } | |
| 1007 | + - id: cn-rail | |
| 1008 | + extend: true | |
| 1009 | + country: CA | |
| 1010 | + sensors: | |
| 1011 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0000016868.json", type: REST_API, connector: edgar, tier: C } | |
| 1012 | + - id: cnrl | |
| 1013 | + extend: true | |
| 1014 | + country: CA | |
| 1015 | + sensors: | |
| 1016 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001017413.json", type: REST_API, connector: edgar, tier: C } | |
| 1017 | + - id: suncor | |
| 1018 | + extend: true | |
| 1019 | + country: CA | |
| 1020 | + sensors: | |
| 1021 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0000311337.json", type: REST_API, connector: edgar, tier: C } | |
| 1022 | + - id: enbridge | |
| 1023 | + extend: true | |
| 1024 | + country: CA | |
| 1025 | + sensors: | |
| 1026 | + - { name: edgar filings (10-K/8-K), url: "https://data.sec.gov/submissions/CIK0000895728.json", type: REST_API, connector: edgar, tier: C } | |
| 1027 | + - id: tc-energy | |
| 1028 | + extend: true | |
| 1029 | + country: CA | |
| 1030 | + sensors: | |
| 1031 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001232384.json", type: REST_API, connector: edgar, tier: C } | |
| 1032 | + - id: bell-canada | |
| 1033 | + extend: true | |
| 1034 | + country: CA | |
| 1035 | + sensors: | |
| 1036 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0000718940.json", type: REST_API, connector: edgar, tier: C } | |
| 1037 | + - id: nutrien | |
| 1038 | + extend: true | |
| 1039 | + country: CA | |
| 1040 | + sensors: | |
| 1041 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001725964.json", type: REST_API, connector: edgar, tier: C } | |
| 1042 | + - id: cgi | |
| 1043 | + extend: true | |
| 1044 | + country: CA | |
| 1045 | + sensors: | |
| 1046 | + - { name: edgar filings (40-F/6-K), url: "https://data.sec.gov/submissions/CIK0001061574.json", type: REST_API, connector: edgar, tier: C } | |
| 1047 | + # ── Foreign private issuers (20-F/6-K) and other large caps with existing entities ── | |
| 1048 | + - id: meta-ai | |
| 1049 | + extend: true | |
| 1050 | + country: US | |
| 1051 | + aliases: [meta platforms, meta newsroom] | |
| 1052 | + sensors: | |
| 1053 | + - { name: Meta newsroom feed, url: "https://about.fb.com/news/feed/", type: RSS, connector: rss, tier: A, config: { maxItems: 40 } } | |
| 1054 | + - id: warner-bros-discovery | |
| 1055 | + extend: true | |
| 1056 | + country: US | |
| 1057 | + sensors: | |
| 1058 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001437107.json", type: REST_API, connector: edgar, tier: C } | |
| 1059 | + - id: novo-nordisk | |
| 1060 | + extend: true | |
| 1061 | + country: DK | |
| 1062 | + sensors: | |
| 1063 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0000353278.json", type: REST_API, connector: edgar, tier: B } | |
| 1064 | + - id: novartis | |
| 1065 | + extend: true | |
| 1066 | + country: CH | |
| 1067 | + sensors: | |
| 1068 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0001114448.json", type: REST_API, connector: edgar, tier: C } | |
| 1069 | + - id: astrazeneca | |
| 1070 | + extend: true | |
| 1071 | + country: GB | |
| 1072 | + sensors: | |
| 1073 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0000901832.json", type: REST_API, connector: edgar, tier: C } | |
| 1074 | + - id: sap | |
| 1075 | + extend: true | |
| 1076 | + country: DE | |
| 1077 | + sensors: | |
| 1078 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0001000184.json", type: REST_API, connector: edgar, tier: C } | |
| 1079 | + - id: sony | |
| 1080 | + extend: true | |
| 1081 | + country: JP | |
| 1082 | + sensors: | |
| 1083 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0000313838.json", type: REST_API, connector: edgar, tier: C } | |
| 1084 | + - id: hsbc | |
| 1085 | + extend: true | |
| 1086 | + country: GB | |
| 1087 | + sensors: | |
| 1088 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0001089113.json", type: REST_API, connector: edgar, tier: B } | |
| 1089 | + - id: unilever | |
| 1090 | + extend: true | |
| 1091 | + country: GB | |
| 1092 | + sensors: | |
| 1093 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0000217410.json", type: REST_API, connector: edgar, tier: C } | |
| 1094 | + - id: bp | |
| 1095 | + extend: true | |
| 1096 | + country: GB | |
| 1097 | + sensors: | |
| 1098 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0000313807.json", type: REST_API, connector: edgar, tier: C } | |
| 1099 | + - id: shell | |
| 1100 | + extend: true | |
| 1101 | + country: GB | |
| 1102 | + sensors: | |
| 1103 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0001306965.json", type: REST_API, connector: edgar, tier: C } | |
| 1104 | + - id: totalenergies | |
| 1105 | + extend: true | |
| 1106 | + country: FR | |
| 1107 | + sensors: | |
| 1108 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0000879764.json", type: REST_API, connector: edgar, tier: C } | |
| 1109 | + - id: spotify | |
| 1110 | + extend: true | |
| 1111 | + country: SE | |
| 1112 | + sensors: | |
| 1113 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0001639920.json", type: REST_API, connector: edgar, tier: C } | |
| 1114 | + - id: mercadolibre | |
| 1115 | + extend: true | |
| 1116 | + country: UY | |
| 1117 | + sensors: | |
| 1118 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001099590.json", type: REST_API, connector: edgar, tier: C } | |
| 1119 | + - id: pdd-temu | |
| 1120 | + extend: true | |
| 1121 | + country: IE | |
| 1122 | + sensors: | |
| 1123 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0001737806.json", type: REST_API, connector: edgar, tier: C } | |
| 1124 | + - id: jd-com | |
| 1125 | + extend: true | |
| 1126 | + country: CN | |
| 1127 | + sensors: | |
| 1128 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0001549802.json", type: REST_API, connector: edgar, tier: C } | |
| 1129 | + - id: alibaba | |
| 1130 | + extend: true | |
| 1131 | + country: CN | |
| 1132 | + sensors: | |
| 1133 | + - { name: edgar filings (20-F/6-K), url: "https://data.sec.gov/submissions/CIK0001577552.json", type: REST_API, connector: edgar, tier: B } | |
| 1134 | + - id: okta | |
| 1135 | + extend: true | |
| 1136 | + country: US | |
| 1137 | + sensors: | |
| 1138 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001660134.json", type: REST_API, connector: edgar, tier: C } | |
| 1139 | + - id: twilio | |
| 1140 | + extend: true | |
| 1141 | + country: US | |
| 1142 | + sensors: | |
| 1143 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001447669.json", type: REST_API, connector: edgar, tier: C } | |
| 1144 | + - id: reddit | |
| 1145 | + extend: true | |
| 1146 | + country: US | |
| 1147 | + sensors: | |
| 1148 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001713445.json", type: REST_API, connector: edgar, tier: C } | |
| 1149 | + - id: chipotle | |
| 1150 | + extend: true | |
| 1151 | + country: US | |
| 1152 | + sensors: | |
| 1153 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001058090.json", type: REST_API, connector: edgar, tier: C } | |
| 1154 | + - id: yum-brands | |
| 1155 | + extend: true | |
| 1156 | + country: US | |
| 1157 | + sensors: | |
| 1158 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001041061.json", type: REST_API, connector: edgar, tier: C } | |
| 1159 | + - id: dominos | |
| 1160 | + extend: true | |
| 1161 | + country: US | |
| 1162 | + sensors: | |
| 1163 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001286681.json", type: REST_API, connector: edgar, tier: C } | |
| 1164 | + - id: centene | |
| 1165 | + extend: true | |
| 1166 | + country: US | |
| 1167 | + sensors: | |
| 1168 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001071739.json", type: REST_API, connector: edgar, tier: C } | |
| 1169 | + - id: molina-healthcare | |
| 1170 | + extend: true | |
| 1171 | + country: US | |
| 1172 | + sensors: | |
| 1173 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001179929.json", type: REST_API, connector: edgar, tier: C } | |
| 1174 | + - id: hca-healthcare | |
| 1175 | + extend: true | |
| 1176 | + country: US | |
| 1177 | + sensors: | |
| 1178 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000860730.json", type: REST_API, connector: edgar, tier: C } | |
| 1179 | + - id: danaher | |
| 1180 | + extend: true | |
| 1181 | + country: US | |
| 1182 | + sensors: | |
| 1183 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000313616.json", type: REST_API, connector: edgar, tier: C } | |
| 1184 | + - id: agilent | |
| 1185 | + extend: true | |
| 1186 | + country: US | |
| 1187 | + sensors: | |
| 1188 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001090872.json", type: REST_API, connector: edgar, tier: C } | |
| 1189 | + - id: illumina | |
| 1190 | + extend: true | |
| 1191 | + country: US | |
| 1192 | + sensors: | |
| 1193 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001110803.json", type: REST_API, connector: edgar, tier: C } | |
| 1194 | + - id: biogen | |
| 1195 | + extend: true | |
| 1196 | + country: US | |
| 1197 | + sensors: | |
| 1198 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000875045.json", type: REST_API, connector: edgar, tier: C } | |
| 1199 | + - id: waters | |
| 1200 | + extend: true | |
| 1201 | + country: US | |
| 1202 | + sensors: | |
| 1203 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001000697.json", type: REST_API, connector: edgar, tier: C } | |
| 1204 | + - id: bio-rad | |
| 1205 | + extend: true | |
| 1206 | + country: US | |
| 1207 | + sensors: | |
| 1208 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000012208.json", type: REST_API, connector: edgar, tier: C } | |
| 1209 | + - id: baxter | |
| 1210 | + extend: true | |
| 1211 | + country: US | |
| 1212 | + sensors: | |
| 1213 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000010456.json", type: REST_API, connector: edgar, tier: C } | |
| 1214 | + - id: insulet | |
| 1215 | + extend: true | |
| 1216 | + country: US | |
| 1217 | + sensors: | |
| 1218 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001145197.json", type: REST_API, connector: edgar, tier: C } | |
| 1219 | + - id: tandem-diabetes-care | |
| 1220 | + extend: true | |
| 1221 | + country: US | |
| 1222 | + sensors: | |
| 1223 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001438133.json", type: REST_API, connector: edgar, tier: C } | |
| 1224 | + - id: align-technology | |
| 1225 | + extend: true | |
| 1226 | + country: US | |
| 1227 | + sensors: | |
| 1228 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001097149.json", type: REST_API, connector: edgar, tier: C } | |
| 1229 | + - id: dentsply-sirona | |
| 1230 | + extend: true | |
| 1231 | + country: US | |
| 1232 | + sensors: | |
| 1233 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000818479.json", type: REST_API, connector: edgar, tier: C } | |
| 1234 | + - id: wabtec | |
| 1235 | + extend: true | |
| 1236 | + country: US | |
| 1237 | + sensors: | |
| 1238 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000943452.json", type: REST_API, connector: edgar, tier: C } | |
| 1239 | + - id: ups | |
| 1240 | + extend: true | |
| 1241 | + country: US | |
| 1242 | + sensors: | |
| 1243 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001090727.json", type: REST_API, connector: edgar, tier: B } | |
| 1244 | + - id: fedex | |
| 1245 | + extend: true | |
| 1246 | + country: US | |
| 1247 | + sensors: | |
| 1248 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001048911.json", type: REST_API, connector: edgar, tier: B } | |
| 1249 | + - id: csx | |
| 1250 | + extend: true | |
| 1251 | + country: US | |
| 1252 | + sensors: | |
| 1253 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000277948.json", type: REST_API, connector: edgar, tier: C } | |
| 1254 | + - id: norfolk-southern | |
| 1255 | + extend: true | |
| 1256 | + country: US | |
| 1257 | + sensors: | |
| 1258 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000702165.json", type: REST_API, connector: edgar, tier: C } | |
| 1259 | + - id: jb-hunt | |
| 1260 | + extend: true | |
| 1261 | + country: US | |
| 1262 | + sensors: | |
| 1263 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000728535.json", type: REST_API, connector: edgar, tier: C } | |
| 1264 | + - id: xpo | |
| 1265 | + extend: true | |
| 1266 | + country: US | |
| 1267 | + sensors: | |
| 1268 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001166003.json", type: REST_API, connector: edgar, tier: C } | |
| 1269 | + - id: ch-robinson | |
| 1270 | + extend: true | |
| 1271 | + country: US | |
| 1272 | + sensors: | |
| 1273 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0001043277.json", type: REST_API, connector: edgar, tier: C } | |
| 1274 | + - id: ryder | |
| 1275 | + extend: true | |
| 1276 | + country: US | |
| 1277 | + sensors: | |
| 1278 | + - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000085961.json", type: REST_API, connector: edgar, tier: C } | |
added
config/sources.d/44-governments.yaml
+1660 −0
@@ -0,0 +1,2539 @@ | ||
| 1 | +# config/sources.d/44-governments.yaml — governments (2026-09-11): legislation, regulations, gazettes, press | |
| 2 | +# releases, budgets, procurement, open-data catalogues, appointments, sanctions, courts and emergency notices for | |
| 3 | +# Canada (federal + QC/ON/BC/AB/NS/NB/MB/SK/PE/NL/NT), the United States, the European Union, the United Kingdom, | |
| 4 | +# France, Germany, Italy, Spain, Japan, South Korea, Australia, India, Brazil, Mexico and a few international bodies. | |
| 5 | +# Every entry — new or `extend: true` — carries `country:` (ISO alpha-2, EU/INT) and `language:` when not English so | |
| 6 | +# country pages work. Blocked/JS-only endpoints are recorded in `notes:` instead of being pretended. | |
| 7 | +sources: | |
| 8 | + # ───────────────────────────── CANADA — federal ───────────────────────────── | |
| 9 | + - id: canada | |
| 10 | + extend: true | |
| 11 | + country: CA | |
| 12 | + sensors: | |
| 13 | + - { name: statements (all departments), url: "https://api.io.canada.ca/io-server/gc/news/en/v2?type=statements&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=Statements", type: ATOM, connector: rss, tier: A } | |
| 14 | + - { name: speeches (all departments), url: "https://api.io.canada.ca/io-server/gc/news/en/v2?type=speeches&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=Speeches", type: ATOM, connector: rss, tier: B } | |
| 15 | + - { name: backgrounders (all departments), url: "https://api.io.canada.ca/io-server/gc/news/en/v2?type=backgrounders&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=Backgrounders", type: ATOM, connector: rss, tier: B } | |
| 16 | + - id: statcan | |
| 17 | + extend: true | |
| 18 | + country: CA | |
| 19 | + sensors: | |
| 20 | + - { name: news releases (Canada.ca), url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=statisticscanada&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=Statistics%20Canada", type: ATOM, connector: rss, tier: B } | |
| 21 | + - id: finance-canada | |
| 22 | + extend: true | |
| 23 | + country: CA | |
| 24 | + sensors: | |
| 25 | + - { name: consultations, url: "https://www.canada.ca/en/department-finance/programs/consultations.html", type: HTML, connector: http, tier: C } | |
| 26 | + - id: ircc | |
| 27 | + extend: true | |
| 28 | + country: CA | |
| 29 | + sensors: | |
| 30 | + - { name: notices, url: "https://www.canada.ca/en/immigration-refugees-citizenship/news/notices.html", type: HTML, connector: http, tier: B } | |
| 31 | + - { name: ministerial instructions, url: "https://www.canada.ca/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions.html", type: HTML, connector: http, tier: C } | |
| 32 | + - id: global-affairs-canada | |
| 33 | + extend: true | |
| 34 | + country: CA | |
| 35 | + sensors: | |
| 36 | + - { name: current sanctions (regimes), url: "https://www.international.gc.ca/world-monde/international_relations-relations_internationales/sanctions/current-actuelles.aspx?lang=eng", type: HTML, connector: http, tier: C } | |
| 37 | + - { name: travel advisories (all destinations), url: "https://travel.gc.ca/travelling/advisories", type: HTML, connector: http, tier: B } | |
| 38 | + notes: "Consolidated Canadian Autonomous Sanctions List XML (sema-lmes.xml, 2.9 MB) not monitored; travel-advisory JSON (data.international.gc.ca) is an object keyed by ISO code, not a list." | |
| 39 | + - id: canadabuys | |
| 40 | + name: CanadaBuys (PSPC tender notices) | |
| 41 | + domain: canadabuys.canada.ca | |
| 42 | + categories: [government, procurement] | |
| 43 | + tier: B | |
| 44 | + aliases: [canadabuys, achatscanada, buyandsell] | |
| 45 | + llm: false | |
| 46 | + country: CA | |
| 47 | + sensors: | |
| 48 | + - { name: new tender notices (CSV), url: "https://canadabuys.canada.ca/opendata/pub/newTenderNotice-nouvelAvisAppelOffres.csv", type: FILE, connector: csv, tier: B, config: { keyColumn: "referenceNumber-numeroReference", titleColumn: "title-titre-eng", dateColumn: "publicationDate-datePublication", compareColumns: ["amendmentNumber-numeroModification", "tenderStatus-appelOffresStatut-eng", "tenderClosingDate-appelOffresDateCloture"], maxRows: 500, tail: true } } | |
| 49 | + - { name: tender opportunities, url: "https://canadabuys.canada.ca/en/tender-opportunities", type: HTML, connector: http, tier: B } | |
| 50 | + - id: justice-canada | |
| 51 | + name: Department of Justice Canada | |
| 52 | + domain: justice.gc.ca | |
| 53 | + categories: [government, legal] | |
| 54 | + tier: B | |
| 55 | + aliases: [justice canada, department of justice canada] | |
| 56 | + country: CA | |
| 57 | + sensors: | |
| 58 | + - { name: news, url: "https://www.justice.gc.ca/eng/news-nouv/index.html", type: HTML, connector: http, tier: B } | |
| 59 | + notes: "No working Canada.ca news-API department id; laws-lois.justice.gc.ca RSS feeds are gone (404)." | |
| 60 | + - id: privy-council-office | |
| 61 | + name: Privy Council Office (Canada) | |
| 62 | + domain: canada.ca | |
| 63 | + homepage: https://www.canada.ca/en/privy-council.html | |
| 64 | + categories: [government, politics] | |
| 65 | + tier: B | |
| 66 | + aliases: [pco, privy council office, bureau du conseil privé] | |
| 67 | + country: CA | |
| 68 | + sensors: | |
| 69 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=privycouncil&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=PCO", type: ATOM, connector: rss, tier: B } | |
| 70 | + - { name: intergovernmental affairs news, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=intergovernmentalaffairs&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=Intergovernmental%20Affairs", type: ATOM, connector: rss, tier: C } | |
| 71 | + - { name: democratic institutions news, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=democraticinstitutions&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=Democratic%20Institutions", type: ATOM, connector: rss, tier: C } | |
| 72 | + - { name: orders in council, url: "https://orders-in-council.canada.ca/index.php?lang=en", type: HTML, connector: http, tier: C } | |
| 73 | + - id: treasury-board-secretariat | |
| 74 | + name: Treasury Board of Canada Secretariat | |
| 75 | + domain: tbs-sct.canada.ca | |
| 76 | + homepage: https://www.canada.ca/en/treasury-board-secretariat.html | |
| 77 | + categories: [government] | |
| 78 | + tier: B | |
| 79 | + aliases: [tbs, treasury board secretariat, secrétariat du conseil du trésor] | |
| 80 | + country: CA | |
| 81 | + sensors: | |
| 82 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=treasuryboardsecretariat&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=TBS", type: ATOM, connector: rss, tier: B } | |
| 83 | + - id: natural-resources-canada | |
| 84 | + name: Natural Resources Canada | |
| 85 | + domain: natural-resources.canada.ca | |
| 86 | + categories: [government, energy] | |
| 87 | + tier: B | |
| 88 | + aliases: [nrcan, natural resources canada, ressources naturelles canada] | |
| 89 | + country: CA | |
| 90 | + sensors: | |
| 91 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=naturalresourcescanada&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=NRCan", type: ATOM, connector: rss, tier: B } | |
| 92 | + - id: veterans-affairs-canada | |
| 93 | + name: Veterans Affairs Canada | |
| 94 | + domain: veterans.gc.ca | |
| 95 | + categories: [government] | |
| 96 | + tier: C | |
| 97 | + aliases: [vac, veterans affairs canada, anciens combattants canada] | |
| 98 | + country: CA | |
| 99 | + sensors: | |
| 100 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=veteransaffairscanada&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=VAC", type: ATOM, connector: rss, tier: C } | |
| 101 | + - id: crown-indigenous-relations | |
| 102 | + name: Crown-Indigenous Relations and Northern Affairs Canada | |
| 103 | + domain: rcaanc-cirnac.gc.ca | |
| 104 | + categories: [government] | |
| 105 | + tier: B | |
| 106 | + aliases: [cirnac, crown-indigenous relations, relations couronne-autochtones] | |
| 107 | + country: CA | |
| 108 | + sensors: | |
| 109 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=crownindigenousrelationsandnorthernaffairscanada&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=CIRNAC", type: ATOM, connector: rss, tier: B } | |
| 110 | + - id: indigenous-services-canada | |
| 111 | + name: Indigenous Services Canada | |
| 112 | + domain: sac-isc.gc.ca | |
| 113 | + categories: [government] | |
| 114 | + tier: B | |
| 115 | + aliases: [isc, indigenous services canada, services aux autochtones canada] | |
| 116 | + country: CA | |
| 117 | + sensors: | |
| 118 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=indigenousservicescanada&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=ISC", type: ATOM, connector: rss, tier: B } | |
| 119 | + - id: parks-canada | |
| 120 | + name: Parks Canada | |
| 121 | + domain: parks.canada.ca | |
| 122 | + categories: [government, climate] | |
| 123 | + tier: C | |
| 124 | + aliases: [parks canada, parcs canada] | |
| 125 | + country: CA | |
| 126 | + sensors: | |
| 127 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=parkscanada&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=Parks%20Canada", type: ATOM, connector: rss, tier: C } | |
| 128 | + - id: canadian-heritage | |
| 129 | + name: Canadian Heritage | |
| 130 | + domain: canada.ca | |
| 131 | + homepage: https://www.canada.ca/en/canadian-heritage.html | |
| 132 | + categories: [government, media] | |
| 133 | + tier: C | |
| 134 | + aliases: [pch, canadian heritage, patrimoine canadien] | |
| 135 | + country: CA | |
| 136 | + sensors: | |
| 137 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=canadianheritage&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=Canadian%20Heritage", type: ATOM, connector: rss, tier: C } | |
| 138 | + - id: csis-canada | |
| 139 | + name: Canadian Security Intelligence Service | |
| 140 | + domain: canada.ca | |
| 141 | + homepage: https://www.canada.ca/en/security-intelligence-service.html | |
| 142 | + categories: [government, cyber] | |
| 143 | + tier: B | |
| 144 | + aliases: [csis, canadian security intelligence service, scrs] | |
| 145 | + country: CA | |
| 146 | + sensors: | |
| 147 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=canadiansecurityintelligenceservice&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=CSIS", type: ATOM, connector: rss, tier: B } | |
| 148 | + - id: iaac | |
| 149 | + name: Impact Assessment Agency of Canada | |
| 150 | + domain: canada.ca | |
| 151 | + homepage: https://www.canada.ca/en/impact-assessment-agency.html | |
| 152 | + categories: [government, climate, energy] | |
| 153 | + tier: C | |
| 154 | + aliases: [iaac, impact assessment agency, agence d'évaluation d'impact] | |
| 155 | + country: CA | |
| 156 | + sensors: | |
| 157 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=impactassessmentagency&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=IAAC", type: ATOM, connector: rss, tier: C } | |
| 158 | + - id: fcac | |
| 159 | + name: Financial Consumer Agency of Canada | |
| 160 | + domain: canada.ca | |
| 161 | + homepage: https://www.canada.ca/en/financial-consumer-agency.html | |
| 162 | + categories: [government, finance, consumer-safety] | |
| 163 | + tier: C | |
| 164 | + aliases: [fcac, financial consumer agency of canada, acfc] | |
| 165 | + country: CA | |
| 166 | + sensors: | |
| 167 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=financialconsumeragency&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=FCAC", type: ATOM, connector: rss, tier: C } | |
| 168 | + - id: citt | |
| 169 | + name: Canadian International Trade Tribunal | |
| 170 | + domain: citt-tcce.gc.ca | |
| 171 | + categories: [government, legal, commerce] | |
| 172 | + tier: C | |
| 173 | + aliases: [citt, canadian international trade tribunal, tcce] | |
| 174 | + country: CA | |
| 175 | + sensors: | |
| 176 | + - { name: news releases, url: "https://api.io.canada.ca/io-server/gc/news/en/v2?dept=canadianinternationaltradetribunal&sort=publishedDate&orderBy=desc&pick=50&format=atom&atomtitle=CITT", type: ATOM, connector: rss, tier: C } | |
| 177 | + - id: fintrac | |
| 178 | + name: FINTRAC | |
| 179 | + domain: fintrac-canafe.canada.ca | |
| 180 | + categories: [government, finance] | |
| 181 | + tier: B | |
| 182 | + aliases: [fintrac, canafe, financial transactions and reports analysis centre] | |
| 183 | + country: CA | |
| 184 | + sensors: | |
| 185 | + - { name: news and notices, url: "https://fintrac-canafe.canada.ca/rss/rss-eng.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 186 | + - id: canlii | |
| 187 | + name: CanLII | |
| 188 | + domain: canlii.org | |
| 189 | + categories: [legal, open-data] | |
| 190 | + tier: B | |
| 191 | + aliases: [canlii, canadian legal information institute] | |
| 192 | + first_party: false | |
| 193 | + country: CA | |
| 194 | + sensors: | |
| 195 | + - { name: new decisions — Federal Court of Appeal, url: "https://www.canlii.org/en/ca/fca/rss_new.xml", type: RSS, connector: rss, tier: B } | |
| 196 | + - { name: new decisions — Court of Appeal for Ontario, url: "https://www.canlii.org/en/on/onca/rss_new.xml", type: RSS, connector: rss, tier: B } | |
| 197 | + - { name: new decisions — Cour d'appel du Québec, url: "https://www.canlii.org/fr/qc/qcca/rss_new.xml", type: RSS, connector: rss, tier: B } | |
| 198 | + - { name: new decisions — Court of Appeal for British Columbia, url: "https://www.canlii.org/en/bc/bcca/rss_new.xml", type: RSS, connector: rss, tier: B } | |
| 199 | + - { name: new decisions — Court of Appeal of Alberta, url: "https://www.canlii.org/en/ab/abca/rss_new.xml", type: RSS, connector: rss, tier: B } | |
| 200 | + - { name: new decisions — Ontario Superior Court of Justice, url: "https://www.canlii.org/en/on/onsc/rss_new.xml", type: RSS, connector: rss, tier: C } | |
| 201 | + - id: openparliament | |
| 202 | + name: openparliament.ca | |
| 203 | + domain: openparliament.ca | |
| 204 | + categories: [politics, open-data] | |
| 205 | + tier: B | |
| 206 | + aliases: [openparliament] | |
| 207 | + first_party: false | |
| 208 | + country: CA | |
| 209 | + sensors: | |
| 210 | + - { name: bills in the House of Commons, url: "https://openparliament.ca/bills/rss/", type: RSS, connector: rss, tier: B } | |
| 211 | + - { name: House votes (API), url: "https://api.openparliament.ca/votes/?format=json&limit=40", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: objects, keyField: url, titleTemplate: "Vote {number} ({session}) — {result}: {description.en}", dateField: date, urlTemplate: "https://openparliament.ca{key}", compareFields: [result, yea_total, nay_total], maxItems: 40 } } | |
| 212 | + # ───────────────────────────── CANADA — Québec ───────────────────────────── | |
| 213 | + - id: assemblee-nationale-quebec | |
| 214 | + extend: true | |
| 215 | + country: CA | |
| 216 | + language: fr | |
| 217 | + sensors: | |
| 218 | + - { name: projets de loi, url: "https://www.assnat.qc.ca/fr/rss/SyndicationRSS-210.html", type: RSS, connector: rss, tier: B } | |
| 219 | + - { name: actualités, url: "https://www.assnat.qc.ca/fr/rss/SyndicationRSS-214.html", type: RSS, connector: rss, tier: B } | |
| 220 | + - { name: mandats des commissions parlementaires, url: "https://www.assnat.qc.ca/fr/rss/SyndicationRSS-120.html", type: RSS, connector: rss, tier: C } | |
| 221 | + - id: elections-quebec | |
| 222 | + extend: true | |
| 223 | + country: CA | |
| 224 | + language: fr | |
| 225 | + sensors: | |
| 226 | + - { name: actualités (RSS), url: "https://www.electionsquebec.qc.ca/actualites/feed/", type: RSS, connector: rss, tier: B } | |
| 227 | + - id: surete-du-quebec | |
| 228 | + name: Sûreté du Québec | |
| 229 | + domain: sq.gouv.qc.ca | |
| 230 | + categories: [government, politics] | |
| 231 | + tier: B | |
| 232 | + aliases: [sq, sûreté du québec, surete du quebec] | |
| 233 | + country: CA | |
| 234 | + language: fr | |
| 235 | + sensors: | |
| 236 | + - { name: communiqués, url: "https://www.sq.gouv.qc.ca/communiques/feed/", type: RSS, connector: rss, tier: B } | |
| 237 | + - id: ville-de-quebec | |
| 238 | + name: Ville de Québec | |
| 239 | + domain: ville.quebec.qc.ca | |
| 240 | + categories: [government] | |
| 241 | + tier: C | |
| 242 | + aliases: [ville de québec, quebec city] | |
| 243 | + country: CA | |
| 244 | + language: fr | |
| 245 | + sensors: | |
| 246 | + - { name: actualités — affaires urbaines (RSS), url: "https://www.ville.quebec.qc.ca/Rss/rss.aspx?f=gen", type: RSS, connector: rss, tier: C } | |
| 247 | + # ───────────────────────────── CANADA — Ontario ───────────────────────────── | |
| 248 | + - id: ontario | |
| 249 | + extend: true | |
| 250 | + country: CA | |
| 251 | + sensors: | |
| 252 | + - { name: Office of the Premier news, url: "https://news.ontario.ca/opo/en/rss/news.rss", type: RSS, connector: rss, tier: A } | |
| 253 | + - { name: Ministry of Finance news, url: "https://news.ontario.ca/mof/en/rss/news.rss", type: RSS, connector: rss, tier: B } | |
| 254 | + - { name: Ministry of the Attorney General news, url: "https://news.ontario.ca/mag/en/rss/news.rss", type: RSS, connector: rss, tier: B } | |
| 255 | + - { name: Ministry of Environment news, url: "https://news.ontario.ca/mecp/en/rss/news.rss", type: RSS, connector: rss, tier: B } | |
| 256 | + notes: "IPC Ontario (ipc.on.ca) and Ottawa (Incapsula) block bots; Ontario Gazette search is a Drupal page without a feed." | |
| 257 | + - id: oeb | |
| 258 | + name: Ontario Energy Board | |
| 259 | + domain: oeb.ca | |
| 260 | + categories: [government, energy] | |
| 261 | + tier: C | |
| 262 | + aliases: [oeb, ontario energy board] | |
| 263 | + country: CA | |
| 264 | + sensors: | |
| 265 | + - { name: newsroom, url: "https://www.oeb.ca/newsroom", type: HTML, connector: http, tier: C } | |
| 266 | + - id: hamilton | |
| 267 | + name: City of Hamilton | |
| 268 | + domain: hamilton.ca | |
| 269 | + categories: [government] | |
| 270 | + tier: C | |
| 271 | + aliases: [city of hamilton, hamilton ontario] | |
| 272 | + country: CA | |
| 273 | + sensors: | |
| 274 | + - { name: news releases, url: "https://www.hamilton.ca/rss.xml", type: RSS, connector: rss, tier: C } | |
| 275 | + # ───────────────────────────── CANADA — British Columbia ───────────────────────────── | |
| 276 | + - id: british-columbia | |
| 277 | + extend: true | |
| 278 | + country: CA | |
| 279 | + sensors: | |
| 280 | + - { name: Office of the Premier news, url: "https://news.gov.bc.ca/ministries/office-of-the-premier/feed", type: RSS, connector: rss, tier: A } | |
| 281 | + - { name: Ministry of Finance news, url: "https://news.gov.bc.ca/ministries/finance/feed", type: RSS, connector: rss, tier: B } | |
| 282 | + - { name: Emergency Management and Climate Readiness news, url: "https://news.gov.bc.ca/ministries/emergency-management-and-climate-readiness/feed", type: RSS, connector: rss, tier: B } | |
| 283 | + - { name: Attorney General news, url: "https://news.gov.bc.ca/ministries/attorney-general/feed", type: RSS, connector: rss, tier: B } | |
| 284 | + notes: "vancouver.ca (Akamai) and BCUC (JS) are not monitorable." | |
| 285 | + - id: emergency-info-bc | |
| 286 | + name: EmergencyInfoBC | |
| 287 | + domain: emergencyinfobc.gov.bc.ca | |
| 288 | + categories: [government, weather] | |
| 289 | + tier: A | |
| 290 | + aliases: [emergencyinfobc, emergency info bc] | |
| 291 | + country: CA | |
| 292 | + sensors: | |
| 293 | + - { name: emergency notices, url: "https://www.emergencyinfobc.gov.bc.ca/feed/", type: RSS, connector: rss, tier: A } | |
| 294 | + - id: drivebc | |
| 295 | + name: DriveBC (BC Ministry of Transportation Open511) | |
| 296 | + domain: drivebc.ca | |
| 297 | + categories: [government, transport, open-data] | |
| 298 | + tier: A | |
| 299 | + aliases: [drivebc, open511 bc] | |
| 300 | + llm: false | |
| 301 | + country: CA | |
| 302 | + sensors: | |
| 303 | + - { name: road events (Open511 API), url: "https://api.open511.gov.bc.ca/events?format=json&limit=50", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: events, keyField: id, titleTemplate: "{event_type} — {headline}", summaryField: description, dateField: updated, urlField: url, compareFields: [updated, status, severity], maxItems: 50 } } | |
| 304 | + - id: bc-laws | |
| 305 | + name: BC Laws (King's Printer) | |
| 306 | + domain: bclaws.gov.bc.ca | |
| 307 | + categories: [government, legal] | |
| 308 | + tier: C | |
| 309 | + aliases: [bc laws, bclaws] | |
| 310 | + country: CA | |
| 311 | + sensors: | |
| 312 | + - { name: recently added or updated, url: "https://www.bclaws.gov.bc.ca/recent.html", type: HTML, connector: http, tier: C } | |
| 313 | + # ───────────────────────────── CANADA — Alberta ───────────────────────────── | |
| 314 | + - id: auc | |
| 315 | + name: Alberta Utilities Commission | |
| 316 | + domain: auc.ab.ca | |
| 317 | + categories: [government, energy] | |
| 318 | + tier: C | |
| 319 | + aliases: [auc, alberta utilities commission] | |
| 320 | + country: CA | |
| 321 | + sensors: | |
| 322 | + - { name: news, url: "https://www.auc.ab.ca/feed/", type: RSS, connector: rss, tier: C } | |
| 323 | + - id: alberta-auditor-general | |
| 324 | + name: Office of the Auditor General of Alberta | |
| 325 | + domain: oag.ab.ca | |
| 326 | + categories: [government] | |
| 327 | + tier: C | |
| 328 | + aliases: [oag alberta, auditor general of alberta] | |
| 329 | + country: CA | |
| 330 | + sensors: | |
| 331 | + - { name: news, url: "https://www.oag.ab.ca/news/", type: HTML, connector: http, tier: C } | |
| 332 | + - id: oipc-alberta | |
| 333 | + name: Office of the Information and Privacy Commissioner of Alberta | |
| 334 | + domain: oipc.ab.ca | |
| 335 | + categories: [government, web-policy] | |
| 336 | + tier: C | |
| 337 | + aliases: [oipc alberta] | |
| 338 | + country: CA | |
| 339 | + sensors: | |
| 340 | + - { name: news releases, url: "https://www.oipc.ab.ca/news-and-events/news-releases", type: HTML, connector: http, tier: C } | |
| 341 | + - id: edmonton | |
| 342 | + name: City of Edmonton | |
| 343 | + domain: edmonton.ca | |
| 344 | + categories: [government] | |
| 345 | + tier: C | |
| 346 | + aliases: [city of edmonton] | |
| 347 | + country: CA | |
| 348 | + sensors: | |
| 349 | + - { name: news releases, url: "https://www.edmonton.ca/rss.xml", type: RSS, connector: rss, tier: C } | |
| 350 | + - id: nova-scotia | |
| 351 | + extend: true | |
| 352 | + country: CA | |
| 353 | + sensors: | |
| 354 | + - { name: traffic advisories, url: "https://novascotia.ca/news/rss/traffic.asp", type: RSS, connector: rss, tier: B } | |
| 355 | + - { name: open data catalogue (Socrata), url: "https://api.us.socrata.com/api/catalog/v1?domains=data.novascotia.ca&order=updatedAt&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: results, keyField: resource.id, titleField: resource.name, summaryField: resource.description, dateField: resource.updatedAt, urlField: link, compareFields: [resource.updatedAt], maxItems: 50 } } | |
| 356 | + notes: "novascotia.ca/news/rss/rss.asp?dept=<id> department feeds return an empty document." | |
| 357 | + - id: ns-legislature | |
| 358 | + name: Nova Scotia Legislature | |
| 359 | + domain: nslegislature.ca | |
| 360 | + categories: [government, politics] | |
| 361 | + tier: C | |
| 362 | + aliases: [nova scotia house of assembly, ns legislature] | |
| 363 | + country: CA | |
| 364 | + sensors: | |
| 365 | + - { name: bills, url: "https://nslegislature.ca/legislative-business/bills-statutes/bills", type: HTML, connector: http, tier: C } | |
| 366 | + - id: nb-legislature | |
| 367 | + name: Legislative Assembly of New Brunswick | |
| 368 | + domain: legnb.ca | |
| 369 | + categories: [government, politics] | |
| 370 | + tier: C | |
| 371 | + aliases: [legislative assembly of new brunswick, legnb] | |
| 372 | + country: CA | |
| 373 | + sensors: | |
| 374 | + - { name: bills, url: "https://legnb.ca/en/legislation/bills", type: HTML, connector: http, tier: C } | |
| 375 | + - id: manitoba | |
| 376 | + extend: true | |
| 377 | + country: CA | |
| 378 | + sensors: | |
| 379 | + - { name: laws — what's new, url: "https://web2.gov.mb.ca/laws/whats_new.php", type: HTML, connector: http, tier: C } | |
| 380 | + - id: manitoba-legislature | |
| 381 | + name: Legislative Assembly of Manitoba | |
| 382 | + domain: gov.mb.ca | |
| 383 | + homepage: https://www.gov.mb.ca/legislature/ | |
| 384 | + categories: [government, politics] | |
| 385 | + tier: C | |
| 386 | + aliases: [legislative assembly of manitoba] | |
| 387 | + country: CA | |
| 388 | + sensors: | |
| 389 | + - { name: bills (43rd legislature, 2nd session), url: "https://web2.gov.mb.ca/bills/43-2/index.php", type: HTML, connector: http, tier: C } | |
| 390 | + - id: sk-legislature | |
| 391 | + name: Legislative Assembly of Saskatchewan | |
| 392 | + domain: legassembly.sk.ca | |
| 393 | + categories: [government, politics] | |
| 394 | + tier: C | |
| 395 | + aliases: [legislative assembly of saskatchewan] | |
| 396 | + country: CA | |
| 397 | + sensors: | |
| 398 | + - { name: bills, url: "https://www.legassembly.sk.ca/legislative-business/bills/", type: HTML, connector: http, tier: C } | |
| 399 | + - id: prince-edward-island | |
| 400 | + name: Government of Prince Edward Island | |
| 401 | + domain: princeedwardisland.ca | |
| 402 | + categories: [government] | |
| 403 | + tier: B | |
| 404 | + aliases: [pei, prince edward island, government of pei] | |
| 405 | + country: CA | |
| 406 | + sensors: | |
| 407 | + - { name: news releases, url: "https://www.princeedwardisland.ca/en/rss.xml", type: RSS, connector: rss, tier: B } | |
| 408 | + - id: federal-register | |
| 409 | + extend: true | |
| 410 | + country: US | |
| 411 | + sensors: | |
| 412 | + - { name: significant documents, url: "https://www.federalregister.gov/api/v1/documents.json?order=newest&per_page=50&conditions[significant]=1", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: results, keyField: document_number, titleField: title, urlField: html_url, summaryField: abstract, dateField: publication_date, maxItems: 50 } } | |
| 413 | + - { name: FCC documents, url: "https://www.federalregister.gov/api/v1/documents.json?order=newest&per_page=50&conditions[agencies][]=federal-communications-commission", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: results, keyField: document_number, titleField: title, urlField: html_url, summaryField: abstract, dateField: publication_date, maxItems: 50 } } | |
| 414 | + - id: white-house | |
| 415 | + extend: true | |
| 416 | + country: US | |
| 417 | + sensors: | |
| 418 | + - { name: briefings and statements, url: "https://www.whitehouse.gov/briefings-statements/feed/", type: RSS, connector: rss, tier: A } | |
| 419 | + - { name: fact sheets, url: "https://www.whitehouse.gov/fact-sheets/feed/", type: RSS, connector: rss, tier: A } | |
| 420 | + - id: govinfo | |
| 421 | + extend: true | |
| 422 | + country: US | |
| 423 | + sensors: | |
| 424 | + - { name: Federal Register (new items), url: "https://www.govinfo.gov/rss/fr.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 425 | + - { name: Compilation of Presidential Documents, url: "https://www.govinfo.gov/rss/dcpd.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 426 | + - { name: Congressional reports, url: "https://www.govinfo.gov/rss/crpt.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 427 | + - { name: enrolled bills, url: "https://www.govinfo.gov/rss/bills-enr.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 428 | + - id: us-treasury | |
| 429 | + extend: true | |
| 430 | + country: US | |
| 431 | + sensors: | |
| 432 | + - { name: press releases (RSS), url: "https://home.treasury.gov/rss.xml", type: RSS, connector: rss, tier: A } | |
| 433 | + - { name: OFAC documents in the Federal Register, url: "https://www.federalregister.gov/api/v1/documents.json?order=newest&per_page=50&conditions[agencies][]=foreign-assets-control-office", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: results, keyField: document_number, titleField: title, urlField: html_url, summaryField: abstract, dateField: publication_date, maxItems: 50 } } | |
| 434 | + notes: "OFAC SDN CSV (sdn.csv, 5.7 MB, no header row) and sanctionslistservice exports are too large/unkeyed for the csv connector; ofac.treasury.gov/recent-actions is JS-rendered." | |
| 435 | + - id: us-state-department | |
| 436 | + extend: true | |
| 437 | + country: US | |
| 438 | + sensors: | |
| 439 | + - { name: travel advisories, url: "https://travel.state.gov/_res/rss/TAsTWs.xml", type: RSS, connector: rss, tier: A, config: { maxItems: 100 } } | |
| 440 | + - { name: department press briefings, url: "https://www.state.gov/rss-feed/department-press-briefings/feed/", type: RSS, connector: rss, tier: B } | |
| 441 | + - id: us-doj | |
| 442 | + extend: true | |
| 443 | + country: US | |
| 444 | + sensors: | |
| 445 | + - { name: speeches, url: "https://www.justice.gov/news/rss?type=speech", type: RSS, connector: rss, tier: C } | |
| 446 | + - id: ftc | |
| 447 | + extend: true | |
| 448 | + country: US | |
| 449 | + sensors: | |
| 450 | + - { name: competition press releases, url: "https://www.ftc.gov/feeds/press-release-competition.xml", type: RSS, connector: rss, tier: B } | |
| 451 | + - { name: consumer protection press releases, url: "https://www.ftc.gov/feeds/press-release-consumer-protection.xml", type: RSS, connector: rss, tier: B } | |
| 452 | + - id: fema | |
| 453 | + name: Federal Emergency Management Agency | |
| 454 | + domain: fema.gov | |
| 455 | + categories: [government, weather, climate] | |
| 456 | + tier: A | |
| 457 | + weight: 1.2 | |
| 458 | + aliases: [fema, federal emergency management agency] | |
| 459 | + country: US | |
| 460 | + sensors: | |
| 461 | + - { name: disaster declarations (OpenFEMA), url: "https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries?$orderby=declarationDate%20desc&$top=50", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: DisasterDeclarationsSummaries, keyField: id, titleTemplate: "{femaDeclarationString} — {declarationTitle} ({designatedArea}, {state})", dateField: declarationDate, urlTemplate: "https://www.fema.gov/disaster/{disasterNumber}", compareFields: [incidentEndDate, disasterCloseoutDate, lastRefresh], maxItems: 50 } } | |
| 462 | + - { name: press releases, url: "https://www.fema.gov/about/news-multimedia/press-releases", type: HTML, connector: http, tier: B } | |
| 463 | + notes: "fema.gov/feeds/news.rss and fema.gov/rss.xml are WAF-blocked (403)." | |
| 464 | + - id: dhs | |
| 465 | + name: U.S. Department of Homeland Security | |
| 466 | + domain: dhs.gov | |
| 467 | + categories: [government, cyber] | |
| 468 | + tier: A | |
| 469 | + aliases: [dhs, department of homeland security, homeland security] | |
| 470 | + country: US | |
| 471 | + sensors: | |
| 472 | + - { name: news, url: "https://www.dhs.gov/news", type: HTML, connector: http, tier: B } | |
| 473 | + - { name: DHS documents in the Federal Register, url: "https://www.federalregister.gov/api/v1/documents.json?order=newest&per_page=50&conditions[agencies][]=homeland-security-department", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: results, keyField: document_number, titleField: title, urlField: html_url, summaryField: abstract, dateField: publication_date, maxItems: 50 } } | |
| 474 | + notes: "dhs.gov RSS endpoints (news-releases/rss.xml, rss.xml) are gone." | |
| 475 | + - id: bis-commerce | |
| 476 | + name: Bureau of Industry and Security (U.S. Commerce) | |
| 477 | + domain: bis.gov | |
| 478 | + homepage: https://www.bis.gov | |
| 479 | + categories: [government, commerce, semiconductors] | |
| 480 | + tier: A | |
| 481 | + weight: 1.2 | |
| 482 | + aliases: [bis, bureau of industry and security, entity list, export controls] | |
| 483 | + country: US | |
| 484 | + sensors: | |
| 485 | + - { name: BIS rules and notices in the Federal Register, url: "https://www.federalregister.gov/api/v1/documents.json?order=newest&per_page=50&conditions[agencies][]=industry-and-security-bureau", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: results, keyField: document_number, titleField: title, urlField: html_url, summaryField: abstract, dateField: publication_date, maxItems: 50 } } | |
| 486 | + - { name: Federal Register notices (BIS site), url: "https://www.bis.doc.gov/index.php/all-articles/17-regulations/federal-register-notices", type: HTML, connector: http, tier: C } | |
| 487 | + notes: "bis.gov press pages are JS-rendered; commerce.gov returns 403." | |
| 488 | + - id: fbi | |
| 489 | + name: Federal Bureau of Investigation | |
| 490 | + domain: fbi.gov | |
| 491 | + categories: [government, cyber] | |
| 492 | + tier: B | |
| 493 | + aliases: [fbi, federal bureau of investigation] | |
| 494 | + llm: false | |
| 495 | + country: US | |
| 496 | + sensors: | |
| 497 | + - { name: national press releases, url: "https://www.fbi.gov/feeds/national-press-releases/rss.xml", type: ATOM, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 498 | + - id: us-ice | |
| 499 | + name: U.S. Immigration and Customs Enforcement | |
| 500 | + domain: ice.gov | |
| 501 | + categories: [government] | |
| 502 | + tier: B | |
| 503 | + aliases: [ice, immigration and customs enforcement] | |
| 504 | + country: US | |
| 505 | + sensors: | |
| 506 | + - { name: news releases, url: "https://www.ice.gov/rss.xml", type: RSS, connector: rss, tier: B } | |
| 507 | + - id: uscis | |
| 508 | + name: U.S. Citizenship and Immigration Services | |
| 509 | + domain: uscis.gov | |
| 510 | + categories: [government] | |
| 511 | + tier: B | |
| 512 | + aliases: [uscis, citizenship and immigration services] | |
| 513 | + country: US | |
| 514 | + sensors: | |
| 515 | + - { name: news releases, url: "https://www.uscis.gov/news/news-releases", type: HTML, connector: http, tier: B } | |
| 516 | + - { name: alerts, url: "https://www.uscis.gov/news/alerts", type: HTML, connector: http, tier: B } | |
| 517 | + notes: "uscis.gov/rss.xml has not been updated since 2015." | |
| 518 | + - id: us-va | |
| 519 | + name: U.S. Department of Veterans Affairs | |
| 520 | + domain: va.gov | |
| 521 | + categories: [government, health] | |
| 522 | + tier: C | |
| 523 | + aliases: [va, department of veterans affairs, veterans affairs] | |
| 524 | + country: US | |
| 525 | + sensors: | |
| 526 | + - { name: VA news, url: "https://news.va.gov/feed/", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 527 | + - id: sba | |
| 528 | + name: U.S. Small Business Administration | |
| 529 | + domain: sba.gov | |
| 530 | + categories: [government, commerce] | |
| 531 | + tier: C | |
| 532 | + aliases: [sba, small business administration] | |
| 533 | + country: US | |
| 534 | + sensors: | |
| 535 | + - { name: press releases, url: "https://www.sba.gov/rss", type: RSS, connector: rss, tier: C } | |
| 536 | + - id: samhsa | |
| 537 | + name: SAMHSA | |
| 538 | + domain: samhsa.gov | |
| 539 | + categories: [government, health] | |
| 540 | + tier: C | |
| 541 | + aliases: [samhsa, substance abuse and mental health services administration] | |
| 542 | + country: US | |
| 543 | + sensors: | |
| 544 | + - { name: press announcements, url: "https://www.samhsa.gov/newsroom/press-announcements/rss", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 545 | + - id: us-copyright-office | |
| 546 | + name: U.S. Copyright Office | |
| 547 | + domain: copyright.gov | |
| 548 | + categories: [government, legal, media] | |
| 549 | + tier: C | |
| 550 | + aliases: [copyright office, us copyright office] | |
| 551 | + country: US | |
| 552 | + sensors: | |
| 553 | + - { name: NewsNet, url: "https://www.copyright.gov/newsnet/rss.xml", type: RSS, connector: rss, tier: C } | |
| 554 | + - { name: rulemaking, url: "https://www.copyright.gov/rulemaking/", type: HTML, connector: http, tier: C } | |
| 555 | + - id: everycrsreport | |
| 556 | + name: EveryCRSReport (Congressional Research Service reports) | |
| 557 | + domain: everycrsreport.com | |
| 558 | + categories: [politics, research, open-data] | |
| 559 | + tier: B | |
| 560 | + aliases: [everycrsreport, crs reports, congressional research service] | |
| 561 | + first_party: false | |
| 562 | + country: US | |
| 563 | + sensors: | |
| 564 | + - { name: new CRS reports, url: "https://www.everycrsreport.com/rss.xml", type: RSS, connector: rss, tier: B } | |
| 565 | + notes: "crsreports.congress.gov is Akamai-protected (403)." | |
| 566 | + - id: california-open-data | |
| 567 | + name: California Open Data Portal | |
| 568 | + domain: data.ca.gov | |
| 569 | + categories: [government, open-data] | |
| 570 | + tier: C | |
| 571 | + aliases: [california open data, data.ca.gov] | |
| 572 | + llm: false | |
| 573 | + country: US | |
| 574 | + sensors: | |
| 575 | + - { name: dataset catalogue (CKAN), url: "https://data.ca.gov/api/3/action/package_search?sort=metadata_modified+desc&rows=50", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: result.results, keyField: id, titleField: title, summaryField: notes, dateField: metadata_modified, compareFields: [metadata_modified], maxItems: 50, urlTemplate: "https://data.ca.gov/dataset/{name}" } } | |
| 576 | + notes: "catalog.data.gov CKAN API (api/3/action/package_search) now returns 404." | |
| 577 | + - id: new-york-state-open-data | |
| 578 | + name: New York State Open Data | |
| 579 | + domain: data.ny.gov | |
| 580 | + categories: [government, open-data] | |
| 581 | + tier: C | |
| 582 | + aliases: [data.ny.gov, new york state open data] | |
| 583 | + llm: false | |
| 584 | + country: US | |
| 585 | + sensors: | |
| 586 | + - { name: dataset catalogue (Socrata), url: "https://api.us.socrata.com/api/catalog/v1?domains=data.ny.gov&order=updatedAt&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: results, keyField: resource.id, titleField: resource.name, summaryField: resource.description, dateField: resource.updatedAt, urlField: link, compareFields: [resource.updatedAt], maxItems: 50 } } | |
| 587 | + - id: texas-open-data | |
| 588 | + name: Texas Open Data Portal | |
| 589 | + domain: data.texas.gov | |
| 590 | + categories: [government, open-data] | |
| 591 | + tier: C | |
| 592 | + aliases: [data.texas.gov, texas open data] | |
| 593 | + llm: false | |
| 594 | + country: US | |
| 595 | + sensors: | |
| 596 | + - { name: dataset catalogue (Socrata), url: "https://api.us.socrata.com/api/catalog/v1?domains=data.texas.gov&order=updatedAt&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: results, keyField: resource.id, titleField: resource.name, summaryField: resource.description, dateField: resource.updatedAt, urlField: link, compareFields: [resource.updatedAt], maxItems: 50 } } | |
| 597 | + - id: los-angeles-open-data | |
| 598 | + name: Los Angeles Open Data | |
| 599 | + domain: data.lacity.org | |
| 600 | + categories: [government, open-data] | |
| 601 | + tier: C | |
| 602 | + aliases: [data.lacity.org, los angeles open data] | |
| 603 | + llm: false | |
| 604 | + country: US | |
| 605 | + sensors: | |
| 606 | + - { name: dataset catalogue (Socrata), url: "https://api.us.socrata.com/api/catalog/v1?domains=data.lacity.org&order=updatedAt&limit=50", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: results, keyField: resource.id, titleField: resource.name, summaryField: resource.description, dateField: resource.updatedAt, urlField: link, compareFields: [resource.updatedAt], maxItems: 50 } } | |
| 607 | + # ───────────────────────────── EUROPEAN UNION ───────────────────────────── | |
| 608 | + - id: european-commission | |
| 609 | + extend: true | |
| 610 | + country: EU | |
| 611 | + sensors: | |
| 612 | + - { name: digital strategy news, url: "https://digital-strategy.ec.europa.eu/rss.xml", type: RSS, connector: rss, tier: B } | |
| 613 | + notes: "Presscorner RSS accepts only ?language=; documenttype/policyarea filters return 400. Have-your-say (brpapi) initiatives use float-like ids; DG sites (*.ec.europa.eu/rss_en) rate-limit (429)." | |
| 614 | + - id: cjeu | |
| 615 | + extend: true | |
| 616 | + country: EU | |
| 617 | + sensors: | |
| 618 | + - { name: press releases, url: "https://curia.europa.eu/jcms/jcms/Jo2_7052/en/", type: HTML, connector: http, tier: B } | |
| 619 | + - id: efsa | |
| 620 | + extend: true | |
| 621 | + country: EU | |
| 622 | + sensors: | |
| 623 | + - { name: news (RSS), url: "https://www.efsa.europa.eu/en/news/rss", type: RSS, connector: rss, tier: B } | |
| 624 | + - id: eib | |
| 625 | + extend: true | |
| 626 | + country: EU | |
| 627 | + sensors: | |
| 628 | + - { name: press releases (RSS), url: "https://www.eib.org/en/press/all/index.rss", type: RSS, connector: rss, tier: B } | |
| 629 | + - id: eu-sanctions-map | |
| 630 | + name: EU Sanctions Map | |
| 631 | + domain: sanctionsmap.eu | |
| 632 | + categories: [government, international, legal] | |
| 633 | + tier: B | |
| 634 | + weight: 1.2 | |
| 635 | + aliases: [eu sanctions map, sanctionsmap, eu restrictive measures] | |
| 636 | + country: EU | |
| 637 | + sensors: | |
| 638 | + - { name: sanctions regimes (API), url: "https://www.sanctionsmap.eu/api/v1/regime", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: data, keyField: id, titleTemplate: "{specification} ({country.data.title})", compareFields: [amendment, expiration, under_construction], urlTemplate: "https://www.sanctionsmap.eu/#/main/details/{key}", maxItems: 100 } } | |
| 639 | + notes: "The consolidated financial sanctions list (webgate.ec.europa.eu/fsd, XML 3 MB / CSV via token) is not keyed for the csv connector." | |
| 640 | + - id: council-of-the-eu | |
| 641 | + name: Council of the European Union | |
| 642 | + domain: consilium.europa.eu | |
| 643 | + categories: [government, politics, international] | |
| 644 | + tier: A | |
| 645 | + aliases: [council of the eu, european council, consilium] | |
| 646 | + country: EU | |
| 647 | + discover: { rss: true, sitemap: true } | |
| 648 | + notes: "consilium.europa.eu answers 403 to every non-browser request (press releases, RSS); listed for discovery only." | |
| 649 | + # ───────────────────────────── UNITED KINGDOM ───────────────────────────── | |
| 650 | + - id: uk-government | |
| 651 | + extend: true | |
| 652 | + country: GB | |
| 653 | + sensors: | |
| 654 | + - { name: policy papers and consultations, url: "https://www.gov.uk/search/policy-papers-and-consultations.atom", type: ATOM, connector: rss, tier: B } | |
| 655 | + - { name: guidance and regulation, url: "https://www.gov.uk/search/guidance-and-regulation.atom", type: ATOM, connector: rss, tier: C } | |
| 656 | + - { name: transparency and FOI releases, url: "https://www.gov.uk/search/transparency-and-freedom-of-information-releases.atom", type: ATOM, connector: rss, tier: C } | |
| 657 | + notes: "The UK Sanctions List / OFSI consolidated list CSV (16 MB) is too large; northernireland.gov.uk, royal.uk, parliament.uk news and electoralcommission.org.uk return 403." | |
| 658 | + - id: uk-parliament | |
| 659 | + extend: true | |
| 660 | + country: GB | |
| 661 | + sensors: | |
| 662 | + - { name: bills (Bills API, recently updated), url: "https://bills-api.parliament.uk/api/v1/Bills?SortOrder=DateUpdatedDescending&Take=50", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: items, keyField: billId, titleField: shortTitle, dateField: lastUpdate, urlTemplate: "https://bills.parliament.uk/bills/{key}", compareFields: [currentHouse, currentStage.description, isAct, isDefeated, billWithdrawn], maxItems: 50 } } | |
| 663 | + - { name: written ministerial statements (API), url: "https://questions-statements-api.parliament.uk/api/writtenstatements/statements?take=50", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: results, keyField: value.id, titleTemplate: "{value.title} — {value.answeringBodyName} ({value.uin})", summaryField: value.text, dateField: value.dateMade, urlTemplate: "https://questions-statements.parliament.uk/written-statements/detail/{value.dateMade}/{value.uin}", maxItems: 50 } } | |
| 664 | + - { name: Commons divisions (votes API), url: "https://commonsvotes-api.parliament.uk/data/divisions.json/search?take=40", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: "", keyField: DivisionId, titleTemplate: "Division {Number}: {Title} (Ayes {AyeCount} · Noes {NoCount})", dateField: Date, urlTemplate: "https://votes.parliament.uk/Votes/Commons/Division/{key}", compareFields: [AyeCount, NoCount, PublicationUpdated], maxItems: 40 } } | |
| 665 | + - id: legislation-gov-uk | |
| 666 | + extend: true | |
| 667 | + country: GB | |
| 668 | + sensors: | |
| 669 | + - { name: new UK statutory instruments, url: "https://www.legislation.gov.uk/new/uksi/data.feed", type: ATOM, connector: rss, tier: B } | |
| 670 | + - { name: new UK public general acts, url: "https://www.legislation.gov.uk/new/ukpga/data.feed", type: ATOM, connector: rss, tier: B } | |
| 671 | + - id: ico-uk | |
| 672 | + extend: true | |
| 673 | + country: GB | |
| 674 | + sensors: | |
| 675 | + - { name: ICO on GOV.UK, url: "https://www.gov.uk/government/organisations/information-commissioner-s-office.atom", type: ATOM, connector: rss, tier: B } | |
| 676 | + - id: ons | |
| 677 | + extend: true | |
| 678 | + country: GB | |
| 679 | + sensors: | |
| 680 | + - { name: ONS on GOV.UK, url: "https://www.gov.uk/government/organisations/office-for-national-statistics.atom", type: ATOM, connector: rss, tier: B } | |
| 681 | + - id: hm-treasury | |
| 682 | + name: HM Treasury | |
| 683 | + domain: gov.uk | |
| 684 | + homepage: https://www.gov.uk/government/organisations/hm-treasury | |
| 685 | + categories: [government, finance] | |
| 686 | + tier: A | |
| 687 | + weight: 1.2 | |
| 688 | + aliases: [hm treasury, hmt, the treasury] | |
| 689 | + country: GB | |
| 690 | + sensors: | |
| 691 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/hm-treasury.atom", type: ATOM, connector: rss, tier: A } | |
| 692 | + - { name: news and communications, url: "https://www.gov.uk/search/news-and-communications.atom?organisations[]=hm-treasury", type: ATOM, connector: rss, tier: A } | |
| 693 | + - id: uk-cabinet-office | |
| 694 | + name: Cabinet Office | |
| 695 | + domain: gov.uk | |
| 696 | + homepage: https://www.gov.uk/government/organisations/cabinet-office | |
| 697 | + categories: [government, politics] | |
| 698 | + tier: B | |
| 699 | + aliases: [cabinet office] | |
| 700 | + country: GB | |
| 701 | + sensors: | |
| 702 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/cabinet-office.atom", type: ATOM, connector: rss, tier: B } | |
| 703 | + - id: number-10 | |
| 704 | + name: Prime Minister's Office, 10 Downing Street | |
| 705 | + domain: gov.uk | |
| 706 | + homepage: https://www.gov.uk/government/organisations/prime-ministers-office-10-downing-street | |
| 707 | + categories: [government, politics] | |
| 708 | + tier: A | |
| 709 | + weight: 1.3 | |
| 710 | + aliases: [number 10, 10 downing street, downing street, prime minister's office, uk prime minister] | |
| 711 | + country: GB | |
| 712 | + sensors: | |
| 713 | + - { name: announcements, url: "https://www.gov.uk/government/organisations/prime-ministers-office-10-downing-street.atom", type: ATOM, connector: rss, tier: A } | |
| 714 | + - id: dsit | |
| 715 | + name: Department for Science, Innovation and Technology | |
| 716 | + domain: gov.uk | |
| 717 | + homepage: https://www.gov.uk/government/organisations/department-for-science-innovation-and-technology | |
| 718 | + categories: [government, technology, ai] | |
| 719 | + tier: A | |
| 720 | + aliases: [dsit, department for science innovation and technology] | |
| 721 | + country: GB | |
| 722 | + sensors: | |
| 723 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/department-for-science-innovation-and-technology.atom", type: ATOM, connector: rss, tier: A } | |
| 724 | + - id: uk-home-office | |
| 725 | + name: Home Office | |
| 726 | + domain: gov.uk | |
| 727 | + homepage: https://www.gov.uk/government/organisations/home-office | |
| 728 | + categories: [government] | |
| 729 | + tier: B | |
| 730 | + aliases: [home office, uk home office] | |
| 731 | + country: GB | |
| 732 | + sensors: | |
| 733 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/home-office.atom", type: ATOM, connector: rss, tier: B } | |
| 734 | + - id: uk-mod | |
| 735 | + name: Ministry of Defence (UK) | |
| 736 | + domain: gov.uk | |
| 737 | + homepage: https://www.gov.uk/government/organisations/ministry-of-defence | |
| 738 | + categories: [government] | |
| 739 | + tier: B | |
| 740 | + aliases: [mod, ministry of defence, uk mod] | |
| 741 | + country: GB | |
| 742 | + sensors: | |
| 743 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/ministry-of-defence.atom", type: ATOM, connector: rss, tier: B } | |
| 744 | + - id: fcdo | |
| 745 | + name: Foreign, Commonwealth & Development Office | |
| 746 | + domain: gov.uk | |
| 747 | + homepage: https://www.gov.uk/government/organisations/foreign-commonwealth-development-office | |
| 748 | + categories: [government, international] | |
| 749 | + tier: A | |
| 750 | + aliases: [fcdo, foreign office, foreign commonwealth and development office] | |
| 751 | + country: GB | |
| 752 | + sensors: | |
| 753 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/foreign-commonwealth-development-office.atom", type: ATOM, connector: rss, tier: A } | |
| 754 | + - { name: the UK Sanctions List (publication page), url: "https://www.gov.uk/government/publications/the-uk-sanctions-list", type: HTML, connector: http, tier: C } | |
| 755 | + - id: dhsc | |
| 756 | + name: Department of Health and Social Care | |
| 757 | + domain: gov.uk | |
| 758 | + homepage: https://www.gov.uk/government/organisations/department-of-health-and-social-care | |
| 759 | + categories: [government, health] | |
| 760 | + tier: B | |
| 761 | + aliases: [dhsc, department of health and social care] | |
| 762 | + country: GB | |
| 763 | + sensors: | |
| 764 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/department-of-health-and-social-care.atom", type: ATOM, connector: rss, tier: B } | |
| 765 | + - id: uk-dbt | |
| 766 | + name: Department for Business and Trade | |
| 767 | + domain: gov.uk | |
| 768 | + homepage: https://www.gov.uk/government/organisations/department-for-business-and-trade | |
| 769 | + categories: [government, commerce] | |
| 770 | + tier: B | |
| 771 | + aliases: [dbt, department for business and trade] | |
| 772 | + country: GB | |
| 773 | + sensors: | |
| 774 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/department-for-business-and-trade.atom", type: ATOM, connector: rss, tier: B } | |
| 775 | + - id: desnz | |
| 776 | + name: Department for Energy Security and Net Zero | |
| 777 | + domain: gov.uk | |
| 778 | + homepage: https://www.gov.uk/government/organisations/department-for-energy-security-and-net-zero | |
| 779 | + categories: [government, energy, climate] | |
| 780 | + tier: B | |
| 781 | + aliases: [desnz, department for energy security and net zero] | |
| 782 | + country: GB | |
| 783 | + sensors: | |
| 784 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/department-for-energy-security-and-net-zero.atom", type: ATOM, connector: rss, tier: B } | |
| 785 | + - id: uk-moj | |
| 786 | + name: Ministry of Justice (UK) | |
| 787 | + domain: gov.uk | |
| 788 | + homepage: https://www.gov.uk/government/organisations/ministry-of-justice | |
| 789 | + categories: [government, legal] | |
| 790 | + tier: B | |
| 791 | + aliases: [moj, ministry of justice] | |
| 792 | + country: GB | |
| 793 | + sensors: | |
| 794 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/ministry-of-justice.atom", type: ATOM, connector: rss, tier: B } | |
| 795 | + - id: dfe | |
| 796 | + name: Department for Education (UK) | |
| 797 | + domain: gov.uk | |
| 798 | + homepage: https://www.gov.uk/government/organisations/department-for-education | |
| 799 | + categories: [government, education] | |
| 800 | + tier: C | |
| 801 | + aliases: [dfe, department for education] | |
| 802 | + country: GB | |
| 803 | + sensors: | |
| 804 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/department-for-education.atom", type: ATOM, connector: rss, tier: C } | |
| 805 | + - id: dwp | |
| 806 | + name: Department for Work and Pensions | |
| 807 | + domain: gov.uk | |
| 808 | + homepage: https://www.gov.uk/government/organisations/department-for-work-pensions | |
| 809 | + categories: [government, labour] | |
| 810 | + tier: C | |
| 811 | + aliases: [dwp, department for work and pensions] | |
| 812 | + country: GB | |
| 813 | + sensors: | |
| 814 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/department-for-work-pensions.atom", type: ATOM, connector: rss, tier: C } | |
| 815 | + - id: hmrc | |
| 816 | + name: HM Revenue & Customs | |
| 817 | + domain: gov.uk | |
| 818 | + homepage: https://www.gov.uk/government/organisations/hm-revenue-customs | |
| 819 | + categories: [government, finance] | |
| 820 | + tier: B | |
| 821 | + aliases: [hmrc, hm revenue and customs] | |
| 822 | + country: GB | |
| 823 | + sensors: | |
| 824 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/hm-revenue-customs.atom", type: ATOM, connector: rss, tier: B } | |
| 825 | + - id: ofsi | |
| 826 | + name: Office of Financial Sanctions Implementation | |
| 827 | + domain: gov.uk | |
| 828 | + homepage: https://www.gov.uk/government/organisations/office-of-financial-sanctions-implementation | |
| 829 | + categories: [government, finance, international] | |
| 830 | + tier: A | |
| 831 | + aliases: [ofsi, financial sanctions implementation, uk financial sanctions] | |
| 832 | + country: GB | |
| 833 | + sensors: | |
| 834 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/office-of-financial-sanctions-implementation.atom", type: ATOM, connector: rss, tier: A } | |
| 835 | + - { name: consolidated list of financial sanctions targets (page), url: "https://www.gov.uk/government/publications/financial-sanctions-consolidated-list-of-targets", type: HTML, connector: http, tier: C } | |
| 836 | + - id: uk-attorney-general | |
| 837 | + name: Attorney General's Office (UK) | |
| 838 | + domain: gov.uk | |
| 839 | + homepage: https://www.gov.uk/government/organisations/attorney-generals-office | |
| 840 | + categories: [government, legal] | |
| 841 | + tier: C | |
| 842 | + aliases: [attorney general's office, uk attorney general] | |
| 843 | + country: GB | |
| 844 | + sensors: | |
| 845 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/attorney-generals-office.atom", type: ATOM, connector: rss, tier: C } | |
| 846 | + - id: dcms | |
| 847 | + name: Department for Culture, Media and Sport | |
| 848 | + domain: gov.uk | |
| 849 | + homepage: https://www.gov.uk/government/organisations/department-for-culture-media-and-sport | |
| 850 | + categories: [government, media] | |
| 851 | + tier: C | |
| 852 | + aliases: [dcms, culture media and sport] | |
| 853 | + country: GB | |
| 854 | + sensors: | |
| 855 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/department-for-culture-media-and-sport.atom", type: ATOM, connector: rss, tier: C } | |
| 856 | + - id: ukef | |
| 857 | + name: UK Export Finance | |
| 858 | + domain: gov.uk | |
| 859 | + homepage: https://www.gov.uk/government/organisations/uk-export-finance | |
| 860 | + categories: [government, finance, commerce] | |
| 861 | + tier: C | |
| 862 | + aliases: [ukef, uk export finance] | |
| 863 | + country: GB | |
| 864 | + sensors: | |
| 865 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/uk-export-finance.atom", type: ATOM, connector: rss, tier: C } | |
| 866 | + - id: crown-commercial-service | |
| 867 | + name: Crown Commercial Service | |
| 868 | + domain: gov.uk | |
| 869 | + homepage: https://www.gov.uk/government/organisations/crown-commercial-service | |
| 870 | + categories: [government, procurement] | |
| 871 | + tier: C | |
| 872 | + aliases: [ccs, crown commercial service] | |
| 873 | + country: GB | |
| 874 | + sensors: | |
| 875 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/crown-commercial-service.atom", type: ATOM, connector: rss, tier: C } | |
| 876 | + - id: gds | |
| 877 | + name: Government Digital Service | |
| 878 | + domain: gov.uk | |
| 879 | + homepage: https://www.gov.uk/government/organisations/government-digital-service | |
| 880 | + categories: [government, technology] | |
| 881 | + tier: C | |
| 882 | + aliases: [gds, government digital service] | |
| 883 | + country: GB | |
| 884 | + sensors: | |
| 885 | + - { name: announcements and publications, url: "https://www.gov.uk/government/organisations/government-digital-service.atom", type: ATOM, connector: rss, tier: C } | |
| 886 | + - id: find-a-tender | |
| 887 | + name: Find a Tender (UK public procurement notices) | |
| 888 | + domain: find-tender.service.gov.uk | |
| 889 | + categories: [government, procurement, open-data] | |
| 890 | + tier: B | |
| 891 | + aliases: [find a tender, fts, uk tenders] | |
| 892 | + llm: false | |
| 893 | + country: GB | |
| 894 | + sensors: | |
| 895 | + - { name: OCDS release packages (latest notices), url: "https://www.find-tender.service.gov.uk/api/1.0/ocdsReleasePackages?limit=50", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: releases, keyField: id, titleTemplate: "{buyer.name} — {tender.title}", summaryField: tender.description, dateField: date, urlTemplate: "https://www.find-tender.service.gov.uk/Notice/{key}", compareFields: [tender.status, tag], maxItems: 50 } } | |
| 896 | + - id: contracts-finder | |
| 897 | + name: Contracts Finder (UK) | |
| 898 | + domain: contractsfinder.service.gov.uk | |
| 899 | + categories: [government, procurement, open-data] | |
| 900 | + tier: B | |
| 901 | + aliases: [contracts finder] | |
| 902 | + llm: false | |
| 903 | + country: GB | |
| 904 | + sensors: | |
| 905 | + - { name: tender notices published in the last 3 days (OCDS), url: "https://www.contractsfinder.service.gov.uk/Published/Notices/OCDS/Search?publishedFrom={now-3d}&stages=tender&size=50", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: releases, keyField: id, titleTemplate: "{buyer.name} — {tender.title}", summaryField: tender.description, dateField: date, urlTemplate: "https://www.contractsfinder.service.gov.uk/Notice/{key}", compareFields: [tender.status], maxItems: 50, noConditional: true } } | |
| 906 | + - id: the-gazette-uk | |
| 907 | + name: The Gazette (UK official public record) | |
| 908 | + domain: thegazette.co.uk | |
| 909 | + categories: [government, legal] | |
| 910 | + tier: B | |
| 911 | + aliases: [the gazette, london gazette, edinburgh gazette, belfast gazette] | |
| 912 | + llm: false | |
| 913 | + country: GB | |
| 914 | + sensors: | |
| 915 | + - { name: all notices, url: "https://www.thegazette.co.uk/all-notices/notice/data.feed?results-page-size=50", type: ATOM, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 916 | + - { name: insolvency notices, url: "https://www.thegazette.co.uk/insolvency/notice/data.feed", type: ATOM, connector: rss, tier: C } | |
| 917 | + - id: national-archives-uk | |
| 918 | + name: The National Archives (Find Case Law) | |
| 919 | + domain: nationalarchives.gov.uk | |
| 920 | + categories: [government, legal, open-data] | |
| 921 | + tier: B | |
| 922 | + aliases: [national archives, find case law, caselaw.nationalarchives.gov.uk] | |
| 923 | + country: GB | |
| 924 | + sensors: | |
| 925 | + - { name: Find Case Law — latest judgments, url: "https://caselaw.nationalarchives.gov.uk/atom.xml", type: ATOM, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 926 | + - id: nao-uk | |
| 927 | + name: National Audit Office (UK) | |
| 928 | + domain: nao.org.uk | |
| 929 | + categories: [government, finance] | |
| 930 | + tier: C | |
| 931 | + aliases: [nao, national audit office] | |
| 932 | + country: GB | |
| 933 | + sensors: | |
| 934 | + - { name: reports and news, url: "https://www.nao.org.uk/feed/", type: ATOM, connector: rss, tier: C } | |
| 935 | + - id: hse-uk | |
| 936 | + name: Health and Safety Executive | |
| 937 | + domain: hse.gov.uk | |
| 938 | + categories: [government, labour, consumer-safety] | |
| 939 | + tier: C | |
| 940 | + aliases: [hse, health and safety executive] | |
| 941 | + country: GB | |
| 942 | + sensors: | |
| 943 | + - { name: press releases, url: "https://press.hse.gov.uk/feed/", type: RSS, connector: rss, tier: C } | |
| 944 | + - id: scottish-government | |
| 945 | + name: Scottish Government | |
| 946 | + domain: gov.scot | |
| 947 | + categories: [government, politics] | |
| 948 | + tier: B | |
| 949 | + aliases: [scottish government, riaghaltas na h-alba] | |
| 950 | + country: GB | |
| 951 | + sensors: | |
| 952 | + - { name: news, url: "https://www.gov.scot/news/", type: HTML, connector: http, tier: B } | |
| 953 | + notes: "gov.scot publishes no RSS; the GOV.UK organisation feed for the Scottish Government is gone." | |
| 954 | + - id: scottish-parliament | |
| 955 | + name: Scottish Parliament | |
| 956 | + domain: parliament.scot | |
| 957 | + categories: [government, politics] | |
| 958 | + tier: C | |
| 959 | + aliases: [scottish parliament, holyrood] | |
| 960 | + country: GB | |
| 961 | + sensors: | |
| 962 | + - { name: bills, url: "https://www.parliament.scot/bills-and-laws/bills", type: HTML, connector: http, tier: C } | |
| 963 | + - id: welsh-government | |
| 964 | + name: Welsh Government | |
| 965 | + domain: gov.wales | |
| 966 | + categories: [government, politics] | |
| 967 | + tier: B | |
| 968 | + aliases: [welsh government, llywodraeth cymru] | |
| 969 | + country: GB | |
| 970 | + sensors: | |
| 971 | + - { name: announcements, url: "https://www.gov.wales/announcements/rss", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 972 | + - id: senedd | |
| 973 | + name: Senedd Cymru (Welsh Parliament) | |
| 974 | + domain: senedd.wales | |
| 975 | + categories: [government, politics] | |
| 976 | + tier: C | |
| 977 | + aliases: [senedd, welsh parliament] | |
| 978 | + country: GB | |
| 979 | + sensors: | |
| 980 | + - { name: business — what's new, url: "https://business.senedd.wales/mgRss.aspx", type: RSS, connector: rss, tier: C } | |
| 981 | + # ───────────────────────────── FRANCE ───────────────────────────── | |
| 982 | + - id: elysee | |
| 983 | + extend: true | |
| 984 | + country: FR | |
| 985 | + language: fr | |
| 986 | + sensors: | |
| 987 | + - { name: actualités (RSS), url: "https://www.elysee.fr/feed", type: RSS, connector: rss, tier: A, config: { maxItems: 50 } } | |
| 988 | + - id: senat-fr | |
| 989 | + extend: true | |
| 990 | + country: FR | |
| 991 | + language: fr | |
| 992 | + sensors: | |
| 993 | + - { name: rapports, url: "https://www.senat.fr/rss/rapports.rss", type: RSS, connector: rss, tier: C } | |
| 994 | + - id: conseil-etat | |
| 995 | + extend: true | |
| 996 | + country: FR | |
| 997 | + language: fr | |
| 998 | + sensors: | |
| 999 | + - { name: avis, url: "https://www.conseil-etat.fr/rss/avis-rss", type: RSS, connector: rss, tier: C } | |
| 1000 | + - id: cnil | |
| 1001 | + extend: true | |
| 1002 | + country: FR | |
| 1003 | + language: fr | |
| 1004 | + sensors: | |
| 1005 | + - { name: sanctions prononcées, url: "https://www.cnil.fr/fr/les-sanctions-prononcees-par-la-cnil", type: HTML, connector: http, tier: C } | |
| 1006 | + - id: autorite-de-la-concurrence | |
| 1007 | + extend: true | |
| 1008 | + country: FR | |
| 1009 | + language: fr | |
| 1010 | + sensors: | |
| 1011 | + - { name: actualités (page), url: "https://www.autoritedelaconcurrence.fr/fr/actualites", type: HTML, connector: http, tier: B } | |
| 1012 | + - id: gouvernement-fr | |
| 1013 | + name: Gouvernement français (info.gouv.fr) | |
| 1014 | + domain: info.gouv.fr | |
| 1015 | + categories: [government, politics] | |
| 1016 | + tier: A | |
| 1017 | + aliases: [gouvernement, info.gouv.fr, gouvernement.fr, matignon, premier ministre] | |
| 1018 | + country: FR | |
| 1019 | + language: fr | |
| 1020 | + discover: { rss: true, sitemap: true } | |
| 1021 | + notes: "info.gouv.fr / gouvernement.fr, legifrance.gouv.fr (JORF), interieur.gouv.fr, diplomatie.gouv.fr (SPIP backend) and education.gouv.fr all answer 403 to bots; vie-publique.fr is JS-rendered. Listed for discovery only." | |
| 1022 | + - id: ministere-justice-fr | |
| 1023 | + name: Ministère de la Justice (France) | |
| 1024 | + domain: justice.gouv.fr | |
| 1025 | + categories: [government, legal] | |
| 1026 | + tier: B | |
| 1027 | + aliases: [ministère de la justice, place vendôme, garde des sceaux] | |
| 1028 | + country: FR | |
| 1029 | + language: fr | |
| 1030 | + sensors: | |
| 1031 | + - { name: actualités, url: "https://www.justice.gouv.fr/rss.xml", type: RSS, connector: rss, tier: B } | |
| 1032 | + - id: ministere-agriculture-fr | |
| 1033 | + name: Ministère de l'Agriculture et de la Souveraineté alimentaire | |
| 1034 | + domain: agriculture.gouv.fr | |
| 1035 | + categories: [government, agriculture] | |
| 1036 | + tier: C | |
| 1037 | + aliases: [ministère de l'agriculture, agriculture.gouv.fr] | |
| 1038 | + country: FR | |
| 1039 | + language: fr | |
| 1040 | + sensors: | |
| 1041 | + - { name: actualités, url: "https://agriculture.gouv.fr/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1042 | + - id: ministere-travail-fr | |
| 1043 | + name: Ministère du Travail (France) | |
| 1044 | + domain: travail-emploi.gouv.fr | |
| 1045 | + categories: [government, labour] | |
| 1046 | + tier: C | |
| 1047 | + aliases: [ministère du travail, travail-emploi.gouv.fr] | |
| 1048 | + country: FR | |
| 1049 | + language: fr | |
| 1050 | + sensors: | |
| 1051 | + - { name: actualités, url: "https://travail-emploi.gouv.fr/rss.xml", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 1052 | + - id: ministere-economie-fr | |
| 1053 | + name: Ministère de l'Économie, des Finances et de la Souveraineté industrielle | |
| 1054 | + domain: economie.gouv.fr | |
| 1055 | + categories: [government, finance] | |
| 1056 | + tier: B | |
| 1057 | + aliases: [ministère de l'économie, bercy, minefi] | |
| 1058 | + country: FR | |
| 1059 | + language: fr | |
| 1060 | + sensors: | |
| 1061 | + - { name: toutes les actualités, url: "https://www.economie.gouv.fr/rss/toutesactualites", type: RSS, connector: rss, tier: B } | |
| 1062 | + notes: "presse.economie.gouv.fr feeds return 403." | |
| 1063 | + - id: douane-fr | |
| 1064 | + name: Douane française (DGDDI) | |
| 1065 | + domain: douane.gouv.fr | |
| 1066 | + categories: [government, commerce] | |
| 1067 | + tier: C | |
| 1068 | + aliases: [douane, dgddi, douanes françaises] | |
| 1069 | + country: FR | |
| 1070 | + language: fr | |
| 1071 | + sensors: | |
| 1072 | + - { name: actualités, url: "https://www.douane.gouv.fr/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1073 | + - id: anssi | |
| 1074 | + name: ANSSI | |
| 1075 | + domain: cyber.gouv.fr | |
| 1076 | + categories: [government, cyber] | |
| 1077 | + tier: B | |
| 1078 | + aliases: [anssi, agence nationale de la sécurité des systèmes d'information] | |
| 1079 | + country: FR | |
| 1080 | + language: fr | |
| 1081 | + sensors: | |
| 1082 | + - { name: actualités, url: "https://www.cyber.gouv.fr/actualites/rss", type: RSS, connector: rss, tier: B } | |
| 1083 | + - id: arcom | |
| 1084 | + name: Arcom | |
| 1085 | + domain: arcom.fr | |
| 1086 | + categories: [government, media, telecom] | |
| 1087 | + tier: C | |
| 1088 | + aliases: [arcom, autorité de régulation de la communication audiovisuelle et numérique, csa] | |
| 1089 | + country: FR | |
| 1090 | + language: fr | |
| 1091 | + sensors: | |
| 1092 | + - { name: actualités, url: "https://www.arcom.fr/actualites", type: HTML, connector: http, tier: C } | |
| 1093 | + notes: "arcom.fr/rss.xml stopped in 2021." | |
| 1094 | + - id: haut-commissariat-strategie-plan | |
| 1095 | + name: Haut-commissariat à la Stratégie et au Plan | |
| 1096 | + domain: strategie-plan.gouv.fr | |
| 1097 | + categories: [government, research] | |
| 1098 | + tier: C | |
| 1099 | + aliases: [france stratégie, haut-commissariat au plan, hcsp] | |
| 1100 | + country: FR | |
| 1101 | + language: fr | |
| 1102 | + sensors: | |
| 1103 | + - { name: publications et actualités, url: "https://www.strategie-plan.gouv.fr/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1104 | + - id: boamp | |
| 1105 | + name: BOAMP (Bulletin officiel des annonces de marchés publics) | |
| 1106 | + domain: boamp.fr | |
| 1107 | + categories: [government, procurement, open-data] | |
| 1108 | + tier: B | |
| 1109 | + aliases: [boamp, marchés publics, annonces de marchés publics] | |
| 1110 | + llm: false | |
| 1111 | + country: FR | |
| 1112 | + language: fr | |
| 1113 | + sensors: | |
| 1114 | + - { name: dernières annonces (Opendatasoft API), url: "https://boamp-datadila.opendatasoft.com/api/explore/v2.1/catalog/datasets/boamp/records?order_by=dateparution%20desc&limit=50", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: results, keyField: idweb, titleTemplate: "{nomacheteur} — {objet}", dateField: dateparution, urlTemplate: "https://www.boamp.fr/pages/avis/?q=idweb:%22{key}%22", compareFields: [etat, datelimitereponse], maxItems: 50 } } | |
| 1115 | + # ───────────────────────────── GERMANY ───────────────────────────── | |
| 1116 | + - id: bundesregierung | |
| 1117 | + extend: true | |
| 1118 | + country: DE | |
| 1119 | + language: de | |
| 1120 | + sensors: | |
| 1121 | + - { name: Pressemitteilungen, url: "https://www.bundesregierung.de/service/rss/breg-de/1151244/feed.xml", type: RSS, connector: rss, tier: A } | |
| 1122 | + notes: "Bundesrat RSS, recht.bund.de (BGBl) feeds, bundesanzeiger.de and most BM* ministry GSB feeds (BMF, BMI, BMJ, BMWK/BMWE, BMDV, BMEL, BMZ) return 404/400." | |
| 1123 | + - id: bundestag | |
| 1124 | + extend: true | |
| 1125 | + country: DE | |
| 1126 | + language: de | |
| 1127 | + sensors: | |
| 1128 | + - { name: Drucksachen, url: "https://www.bundestag.de/static/appdata/includes/rss/drucksachen.rss", type: RSS, connector: rss, tier: B } | |
| 1129 | + - { name: Plenarprotokolle, url: "https://www.bundestag.de/static/appdata/includes/rss/plenarprotokolle.rss", type: RSS, connector: rss, tier: C } | |
| 1130 | + - id: destatis | |
| 1131 | + extend: true | |
| 1132 | + country: DE | |
| 1133 | + sensors: | |
| 1134 | + - { name: Aktuell (RSS), url: "https://www.destatis.de/SiteGlobals/Functions/RSSFeed/DE/RSSNewsfeed/Aktuell.xml", type: RSS, connector: rss, tier: B } | |
| 1135 | + - id: bmas | |
| 1136 | + name: Bundesministerium für Arbeit und Soziales | |
| 1137 | + domain: bmas.de | |
| 1138 | + categories: [government, labour] | |
| 1139 | + tier: C | |
| 1140 | + aliases: [bmas, arbeitsministerium] | |
| 1141 | + country: DE | |
| 1142 | + language: de | |
| 1143 | + sensors: | |
| 1144 | + - { name: Newsfeed, url: "https://www.bmas.de/SiteGlobals/Functions/RSSFeed/DE/RSSNewsfeed/RSSNewsfeed.xml", type: RSS, connector: rss, tier: C } | |
| 1145 | + - id: bmg | |
| 1146 | + name: Bundesministerium für Gesundheit | |
| 1147 | + domain: bundesgesundheitsministerium.de | |
| 1148 | + categories: [government, health] | |
| 1149 | + tier: B | |
| 1150 | + aliases: [bmg, gesundheitsministerium] | |
| 1151 | + country: DE | |
| 1152 | + language: de | |
| 1153 | + sensors: | |
| 1154 | + - { name: Pressemitteilungen, url: "https://www.bundesgesundheitsministerium.de/pressemitteilungen.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 1155 | + - id: bmukn | |
| 1156 | + name: Bundesministerium für Umwelt, Klimaschutz, Naturschutz und nukleare Sicherheit | |
| 1157 | + domain: bundesumweltministerium.de | |
| 1158 | + categories: [government, climate, energy] | |
| 1159 | + tier: C | |
| 1160 | + aliases: [bmukn, bmuv, umweltministerium] | |
| 1161 | + country: DE | |
| 1162 | + language: de | |
| 1163 | + sensors: | |
| 1164 | + - { name: Meldungen Umwelt, url: "https://www.bundesumweltministerium.de/umwelt.rss", type: RSS, connector: rss, tier: C } | |
| 1165 | + - { name: Meldungen Klimaschutz, url: "https://www.bundesumweltministerium.de/klimaschutz.rss", type: RSS, connector: rss, tier: C } | |
| 1166 | + - id: bbk | |
| 1167 | + name: Bundesamt für Bevölkerungsschutz und Katastrophenhilfe | |
| 1168 | + domain: bbk.bund.de | |
| 1169 | + categories: [government, weather] | |
| 1170 | + tier: A | |
| 1171 | + aliases: [bbk, warnung.bund.de, nina warn-app, mowas] | |
| 1172 | + country: DE | |
| 1173 | + language: de | |
| 1174 | + sensors: | |
| 1175 | + - { name: MoWaS warnings (warnung.bund.de), url: "https://warnung.bund.de/api31/mowas/mapData.json", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: id, titleField: i18nTitle.de, dateField: startDate, urlTemplate: "https://warnung.bund.de/meldungen", compareFields: [version, severity, urgency], maxItems: 100 } } | |
| 1176 | + - id: bayern | |
| 1177 | + name: Bayerische Staatsregierung | |
| 1178 | + domain: bayern.de | |
| 1179 | + categories: [government, politics] | |
| 1180 | + tier: C | |
| 1181 | + aliases: [bayern, freistaat bayern, bayerische staatsregierung] | |
| 1182 | + country: DE | |
| 1183 | + language: de | |
| 1184 | + sensors: | |
| 1185 | + - { name: Meldungen, url: "https://www.bayern.de/feed/", type: RSS, connector: rss, tier: C } | |
| 1186 | + - id: niedersachsen | |
| 1187 | + name: Land Niedersachsen | |
| 1188 | + domain: niedersachsen.de | |
| 1189 | + categories: [government, politics] | |
| 1190 | + tier: C | |
| 1191 | + aliases: [niedersachsen, landesregierung niedersachsen] | |
| 1192 | + country: DE | |
| 1193 | + language: de | |
| 1194 | + sensors: | |
| 1195 | + - { name: Presseinformationen, url: "https://www.niedersachsen.de/rss", type: RSS, connector: rss, tier: C } | |
| 1196 | + - id: baden-wuerttemberg | |
| 1197 | + name: Land Baden-Württemberg | |
| 1198 | + domain: baden-wuerttemberg.de | |
| 1199 | + categories: [government, politics] | |
| 1200 | + tier: C | |
| 1201 | + aliases: [baden-württemberg, landesregierung baden-württemberg] | |
| 1202 | + country: DE | |
| 1203 | + language: de | |
| 1204 | + sensors: | |
| 1205 | + - { name: alle Meldungen, url: "https://www.baden-wuerttemberg.de/de/service/rss/xml/rss-alle-meldungen", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 1206 | + - id: berlin | |
| 1207 | + name: Land Berlin (Senatskanzlei) | |
| 1208 | + domain: berlin.de | |
| 1209 | + categories: [government, politics] | |
| 1210 | + tier: C | |
| 1211 | + aliases: [berlin, senat von berlin, senatskanzlei] | |
| 1212 | + country: DE | |
| 1213 | + language: de | |
| 1214 | + sensors: | |
| 1215 | + - { name: Pressemitteilungen des Regierenden Bürgermeisters, url: "https://www.berlin.de/rbmskzl/aktuelles/pressemitteilungen/index.php/rss", type: RSS, connector: rss, tier: C } | |
| 1216 | + - id: govdata | |
| 1217 | + name: GovData (German open data portal) | |
| 1218 | + domain: govdata.de | |
| 1219 | + categories: [government, open-data] | |
| 1220 | + tier: C | |
| 1221 | + aliases: [govdata, govdata.de] | |
| 1222 | + llm: false | |
| 1223 | + country: DE | |
| 1224 | + language: de | |
| 1225 | + sensors: | |
| 1226 | + - { name: dataset catalogue (CKAN), url: "https://ckan.govdata.de/api/3/action/package_search?sort=metadata_modified+desc&rows=50", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: result.results, keyField: id, titleField: title, summaryField: notes, dateField: metadata_modified, compareFields: [metadata_modified], maxItems: 50, urlTemplate: "https://www.govdata.de/suche/daten/{name}" } } | |
| 1227 | + # ───────────────────────────── ITALY ───────────────────────────── | |
| 1228 | + - id: agcom | |
| 1229 | + name: AGCOM | |
| 1230 | + domain: agcom.it | |
| 1231 | + categories: [government, telecom, media] | |
| 1232 | + tier: C | |
| 1233 | + aliases: [agcom, autorità per le garanzie nelle comunicazioni] | |
| 1234 | + country: IT | |
| 1235 | + language: it | |
| 1236 | + sensors: | |
| 1237 | + - { name: news, url: "https://www.agcom.it/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1238 | + - id: gazzetta-ufficiale | |
| 1239 | + name: Gazzetta Ufficiale della Repubblica Italiana | |
| 1240 | + domain: gazzettaufficiale.it | |
| 1241 | + categories: [government, legal] | |
| 1242 | + tier: B | |
| 1243 | + aliases: [gazzetta ufficiale, gu, gurI] | |
| 1244 | + llm: false | |
| 1245 | + country: IT | |
| 1246 | + language: it | |
| 1247 | + sensors: | |
| 1248 | + - { name: Serie Generale — sommario, url: "https://www.gazzettaufficiale.it/rss/SG.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 60 } } | |
| 1249 | + - { name: 1a Serie Speciale — Corte Costituzionale, url: "https://www.gazzettaufficiale.it/rss/S1.xml", type: RSS, connector: rss, tier: C } | |
| 1250 | + - { name: 3a Serie Speciale — Regioni, url: "https://www.gazzettaufficiale.it/rss/S3.xml", type: RSS, connector: rss, tier: C } | |
| 1251 | + - id: maeci | |
| 1252 | + name: Ministero degli Affari Esteri e della Cooperazione Internazionale | |
| 1253 | + domain: esteri.it | |
| 1254 | + categories: [government, international] | |
| 1255 | + tier: B | |
| 1256 | + aliases: [maeci, farnesina, ministero degli esteri] | |
| 1257 | + country: IT | |
| 1258 | + language: it | |
| 1259 | + sensors: | |
| 1260 | + - { name: sala stampa, url: "https://www.esteri.it/it/feed/", type: RSS, connector: rss, tier: B } | |
| 1261 | + - id: ministero-interno | |
| 1262 | + name: Ministero dell'Interno | |
| 1263 | + domain: interno.gov.it | |
| 1264 | + categories: [government] | |
| 1265 | + tier: C | |
| 1266 | + aliases: [ministero dell'interno, viminale] | |
| 1267 | + country: IT | |
| 1268 | + language: it | |
| 1269 | + sensors: | |
| 1270 | + - { name: notizie, url: "https://www.interno.gov.it/it/rss/news.xml", type: RSS, connector: rss, tier: C } | |
| 1271 | + - id: protezione-civile | |
| 1272 | + name: Dipartimento della Protezione Civile | |
| 1273 | + domain: protezionecivile.gov.it | |
| 1274 | + categories: [government, weather] | |
| 1275 | + tier: A | |
| 1276 | + aliases: [protezione civile, dipartimento della protezione civile, dpc] | |
| 1277 | + country: IT | |
| 1278 | + language: it | |
| 1279 | + sensors: | |
| 1280 | + - { name: comunicati stampa, url: "https://api.protezionecivile.it/default/dpcPortalGenerateRss?categoria=comunicato_stampa", type: RSS, connector: rss, tier: A, config: { maxItems: 50 } } | |
| 1281 | + - { name: bollettino di criticità, url: "https://api.protezionecivile.it/default/dpcPortalGenerateRss?categoria=bollettino_criticita", type: RSS, connector: rss, tier: A } | |
| 1282 | + - { name: notizie, url: "https://api.protezionecivile.it/default/dpcPortalGenerateRss?categoria=notizia", type: RSS, connector: rss, tier: B, config: { maxItems: 50 } } | |
| 1283 | + - id: mef | |
| 1284 | + name: Ministero dell'Economia e delle Finanze | |
| 1285 | + domain: mef.gov.it | |
| 1286 | + categories: [government, finance] | |
| 1287 | + tier: B | |
| 1288 | + aliases: [mef, ministero dell'economia e delle finanze, tesoro] | |
| 1289 | + country: IT | |
| 1290 | + language: it | |
| 1291 | + sensors: | |
| 1292 | + - { name: comunicati stampa, url: "https://www.mef.gov.it/ufficio-stampa/comunicati/index.html", type: HTML, connector: http, tier: B } | |
| 1293 | + - id: dati-gov-it | |
| 1294 | + name: dati.gov.it (Italian open data catalogue) | |
| 1295 | + domain: dati.gov.it | |
| 1296 | + categories: [government, open-data] | |
| 1297 | + tier: C | |
| 1298 | + aliases: [dati.gov.it, dati gov] | |
| 1299 | + llm: false | |
| 1300 | + country: IT | |
| 1301 | + language: it | |
| 1302 | + sensors: | |
| 1303 | + - { name: dataset catalogue (CKAN), url: "https://dati.gov.it/opendata/api/3/action/package_search?sort=metadata_modified+desc&rows=50", type: REST_API, connector: jsonlist, tier: C, config: { itemsPath: result.results, keyField: id, titleField: title, summaryField: notes, dateField: metadata_modified, compareFields: [metadata_modified], maxItems: 50, urlTemplate: "https://dati.gov.it/view-dataset/dataset?id={key}" } } | |
| 1304 | + # ───────────────────────────── SPAIN ───────────────────────────── | |
| 1305 | + - id: spain-government | |
| 1306 | + extend: true | |
| 1307 | + country: ES | |
| 1308 | + language: es | |
| 1309 | + sensors: | |
| 1310 | + - { name: news (English), url: "https://www.lamoncloa.gob.es/lang/en/Paginas/rss.aspx", type: RSS, connector: rss, tier: B } | |
| 1311 | + notes: "Senado, Tribunal Constitucional, Ministerio del Interior and AEMET feeds return 403/400." | |
| 1312 | + - id: boe | |
| 1313 | + name: Boletín Oficial del Estado | |
| 1314 | + domain: boe.es | |
| 1315 | + categories: [government, legal] | |
| 1316 | + tier: A | |
| 1317 | + weight: 1.2 | |
| 1318 | + aliases: [boe, boletín oficial del estado, borme] | |
| 1319 | + llm: false | |
| 1320 | + country: ES | |
| 1321 | + language: es | |
| 1322 | + sensors: | |
| 1323 | + - { name: sumario del día (BOE), url: "https://www.boe.es/rss/boe.php", type: RSS, connector: rss, tier: A, config: { maxItems: 100 } } | |
| 1324 | + - { name: disposiciones generales (sección I), url: "https://www.boe.es/rss/boe.php?s=1", type: RSS, connector: rss, tier: A } | |
| 1325 | + - { name: BORME (registro mercantil), url: "https://www.boe.es/rss/borme.php", type: RSS, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 1326 | + - id: gobierno-vasco | |
| 1327 | + name: Gobierno Vasco / Eusko Jaurlaritza (Irekia) | |
| 1328 | + domain: euskadi.eus | |
| 1329 | + categories: [government, politics] | |
| 1330 | + tier: C | |
| 1331 | + aliases: [gobierno vasco, eusko jaurlaritza, irekia] | |
| 1332 | + country: ES | |
| 1333 | + language: es | |
| 1334 | + sensors: | |
| 1335 | + - { name: noticias (Irekia), url: "https://www.irekia.euskadi.eus/es/news.rss", type: RSS, connector: rss, tier: C } | |
| 1336 | + - id: barcelona | |
| 1337 | + name: Ajuntament de Barcelona | |
| 1338 | + domain: barcelona.cat | |
| 1339 | + categories: [government] | |
| 1340 | + tier: C | |
| 1341 | + aliases: [ajuntament de barcelona, barcelona city council] | |
| 1342 | + country: ES | |
| 1343 | + language: es | |
| 1344 | + sensors: | |
| 1345 | + - { name: servei de premsa, url: "https://ajuntament.barcelona.cat/premsa/rss", type: RSS, connector: rss, tier: C } | |
| 1346 | + - id: aepd | |
| 1347 | + name: Agencia Española de Protección de Datos | |
| 1348 | + domain: aepd.es | |
| 1349 | + categories: [government, web-policy] | |
| 1350 | + tier: C | |
| 1351 | + aliases: [aepd, agencia española de protección de datos] | |
| 1352 | + country: ES | |
| 1353 | + language: es | |
| 1354 | + sensors: | |
| 1355 | + - { name: notas de prensa, url: "https://www.aepd.es/prensa-y-comunicacion/notas-de-prensa", type: HTML, connector: http, tier: C } | |
| 1356 | + - id: congreso | |
| 1357 | + name: Congreso de los Diputados | |
| 1358 | + domain: congreso.es | |
| 1359 | + categories: [government, politics] | |
| 1360 | + tier: B | |
| 1361 | + aliases: [congreso de los diputados, congreso] | |
| 1362 | + country: ES | |
| 1363 | + language: es | |
| 1364 | + sensors: | |
| 1365 | + - { name: notas de prensa, url: "https://www.congreso.es/es/notas-de-prensa", type: HTML, connector: http, tier: B } | |
| 1366 | + - id: cgpj | |
| 1367 | + name: Consejo General del Poder Judicial / Tribunal Supremo | |
| 1368 | + domain: poderjudicial.es | |
| 1369 | + categories: [government, legal] | |
| 1370 | + tier: C | |
| 1371 | + aliases: [cgpj, poder judicial, tribunal supremo] | |
| 1372 | + country: ES | |
| 1373 | + language: es | |
| 1374 | + sensors: | |
| 1375 | + - { name: noticias judiciales — Tribunal Supremo, url: "https://www.poderjudicial.es/cgpj/es/Poder-Judicial/Tribunal-Supremo/Noticias-Judiciales/", type: HTML, connector: http, tier: C } | |
| 1376 | + - id: maec | |
| 1377 | + name: Ministerio de Asuntos Exteriores, Unión Europea y Cooperación | |
| 1378 | + domain: exteriores.gob.es | |
| 1379 | + categories: [government, international] | |
| 1380 | + tier: C | |
| 1381 | + aliases: [maec, ministerio de asuntos exteriores, exteriores] | |
| 1382 | + country: ES | |
| 1383 | + language: es | |
| 1384 | + sensors: | |
| 1385 | + - { name: comunicados, url: "https://www.exteriores.gob.es/es/Comunicacion/Comunicados/Paginas/index.aspx", type: HTML, connector: http, tier: C } | |
| 1386 | + - id: ministerio-defensa-es | |
| 1387 | + name: Ministerio de Defensa (España) | |
| 1388 | + domain: defensa.gob.es | |
| 1389 | + categories: [government] | |
| 1390 | + tier: C | |
| 1391 | + aliases: [ministerio de defensa] | |
| 1392 | + country: ES | |
| 1393 | + language: es | |
| 1394 | + sensors: | |
| 1395 | + - { name: notas de prensa, url: "https://www.defensa.gob.es/gabinete/notasPrensa/index.html", type: HTML, connector: http, tier: C } | |
| 1396 | + # ───────────────────────────── JAPAN ───────────────────────────── | |
| 1397 | + - id: kantei | |
| 1398 | + name: Prime Minister's Office of Japan (Kantei) | |
| 1399 | + domain: kantei.go.jp | |
| 1400 | + categories: [government, politics] | |
| 1401 | + tier: A | |
| 1402 | + weight: 1.2 | |
| 1403 | + aliases: [kantei, prime minister of japan, cabinet of japan, 首相官邸] | |
| 1404 | + country: JP | |
| 1405 | + sensors: | |
| 1406 | + - { name: new information (English), url: "https://japan.kantei.go.jp/index-e2.rdf", type: RSS, connector: rss, tier: A } | |
| 1407 | + - { name: prime minister's actions, url: "https://japan.kantei.go.jp/103/actions/index.html", type: HTML, connector: http, tier: B } | |
| 1408 | + notes: "METI (403/202), MOFA (403), MOD (403), Consumer Affairs Agency (403) and NISC (JS) publish no reachable feeds." | |
| 1409 | + - id: digital-agency-japan | |
| 1410 | + name: Digital Agency (Japan) | |
| 1411 | + domain: digital.go.jp | |
| 1412 | + categories: [government, technology] | |
| 1413 | + tier: B | |
| 1414 | + aliases: [digital agency, デジタル庁] | |
| 1415 | + country: JP | |
| 1416 | + language: ja | |
| 1417 | + sensors: | |
| 1418 | + - { name: news (English), url: "https://www.digital.go.jp/en/rss/news.xml", type: RSS, connector: rss, tier: B } | |
| 1419 | + - { name: 新着・更新, url: "https://www.digital.go.jp/rss/news.xml", type: RSS, connector: rss, tier: B } | |
| 1420 | + - id: e-gov-japan | |
| 1421 | + name: e-Gov public comments (Japan) | |
| 1422 | + domain: e-gov.go.jp | |
| 1423 | + categories: [government, legal] | |
| 1424 | + tier: B | |
| 1425 | + aliases: [e-gov, パブリックコメント, public comment japan] | |
| 1426 | + country: JP | |
| 1427 | + language: ja | |
| 1428 | + sensors: | |
| 1429 | + - { name: 意見募集案件一覧, url: "https://public-comment.e-gov.go.jp/rss/pcm_list.xml", type: RSS, connector: rss, tier: B } | |
| 1430 | + - { name: 結果公示案件一覧, url: "https://public-comment.e-gov.go.jp/rss/pcm_result.xml", type: RSS, connector: rss, tier: C } | |
| 1431 | + - id: jma | |
| 1432 | + extend: true | |
| 1433 | + country: JP | |
| 1434 | + language: ja | |
| 1435 | + sensors: | |
| 1436 | + - { name: earthquake list (bosai JSON), url: "https://www.jma.go.jp/bosai/quake/data/list.json", type: REST_API, connector: jsonlist, tier: A, config: { itemsPath: "", keyField: eid, titleTemplate: "{en_ttl}: {en_anm} M{mag} (max intensity {maxi})", dateField: at, urlTemplate: "https://www.jma.go.jp/bosai/map.html#contents=earthquake_map&lang=en", compareFields: [mag, maxi, ser], maxItems: 50 } } | |
| 1437 | + - { name: 高頻度フィード（警報・注意報・地震・火山）, url: "https://www.data.jma.go.jp/developer/xml/feed/extra.xml", type: ATOM, connector: rss, tier: A, config: { maxItems: 100 } } | |
| 1438 | + - id: jftc | |
| 1439 | + extend: true | |
| 1440 | + country: JP | |
| 1441 | + sensors: | |
| 1442 | + - { name: press releases (English), url: "https://www.jftc.go.jp/en/pressreleases/index.html", type: HTML, connector: http, tier: C } | |
| 1443 | + - id: moe-japan | |
| 1444 | + name: Ministry of the Environment (Japan) | |
| 1445 | + domain: env.go.jp | |
| 1446 | + categories: [government, climate] | |
| 1447 | + tier: C | |
| 1448 | + aliases: [moe japan, 環境省] | |
| 1449 | + country: JP | |
| 1450 | + language: ja | |
| 1451 | + sensors: | |
| 1452 | + - { name: 報道発表資料, url: "https://www.env.go.jp/press/index.html", type: HTML, connector: http, tier: C } | |
| 1453 | + - id: soumu | |
| 1454 | + name: Ministry of Internal Affairs and Communications (Japan) | |
| 1455 | + domain: soumu.go.jp | |
| 1456 | + categories: [government, telecom] | |
| 1457 | + tier: C | |
| 1458 | + aliases: [mic japan, soumu, 総務省] | |
| 1459 | + country: JP | |
| 1460 | + language: ja | |
| 1461 | + sensors: | |
| 1462 | + - { name: 報道資料, url: "https://www.soumu.go.jp/menu_news/s-news/index.html", type: HTML, connector: http, tier: C } | |
| 1463 | + - id: kanpou | |
| 1464 | + name: Kanpō (Official Gazette of Japan) | |
| 1465 | + domain: kanpou.npb.go.jp | |
| 1466 | + categories: [government, legal] | |
| 1467 | + tier: C | |
| 1468 | + aliases: [kanpou, kanpo, 官報] | |
| 1469 | + country: JP | |
| 1470 | + language: ja | |
| 1471 | + sensors: | |
| 1472 | + - { name: 本日の官報, url: "https://kanpou.npb.go.jp/", type: HTML, connector: http, tier: C } | |
| 1473 | + # ───────────────────────────── SOUTH KOREA ───────────────────────────── | |
| 1474 | + - id: korea-policy-briefing | |
| 1475 | + extend: true | |
| 1476 | + country: KR | |
| 1477 | + language: ko | |
| 1478 | + sensors: | |
| 1479 | + - { name: 보도자료 (press releases), url: "https://www.korea.kr/briefing/pressReleaseList.do", type: HTML, connector: http, tier: B } | |
| 1480 | + notes: "korea.kr /rss/*.xml feeds are stale columns from 2017; ministry sites (MOEF, MOFA, MSIT, MOTIE, MOIS, KDCA, PIPC), National Assembly and KMA expose no working RSS without an API key." | |
| 1481 | + - id: pmc-australia | |
| 1482 | + name: Department of the Prime Minister and Cabinet (Australia) | |
| 1483 | + domain: pmc.gov.au | |
| 1484 | + categories: [government, politics] | |
| 1485 | + tier: A | |
| 1486 | + aliases: [pm&c, pmc, ministers' media centre] | |
| 1487 | + country: AU | |
| 1488 | + sensors: | |
| 1489 | + - { name: ministers' media centre, url: "https://ministers.pmc.gov.au/rss.xml", type: RSS, connector: rss, tier: A } | |
| 1490 | + notes: "pmc.gov.au/rss.xml itself returns 403." | |
| 1491 | + - id: australian-treasury | |
| 1492 | + name: Australian Treasury | |
| 1493 | + domain: treasury.gov.au | |
| 1494 | + categories: [government, finance] | |
| 1495 | + tier: B | |
| 1496 | + aliases: [australian treasury, the treasury australia] | |
| 1497 | + country: AU | |
| 1498 | + sensors: | |
| 1499 | + - { name: media releases, url: "https://treasury.gov.au/media-release", type: HTML, connector: http, tier: B } | |
| 1500 | + - id: dfat | |
| 1501 | + name: Department of Foreign Affairs and Trade (Australia) | |
| 1502 | + domain: dfat.gov.au | |
| 1503 | + categories: [government, international] | |
| 1504 | + tier: B | |
| 1505 | + aliases: [dfat, department of foreign affairs and trade] | |
| 1506 | + country: AU | |
| 1507 | + sensors: | |
| 1508 | + - { name: news, url: "https://www.dfat.gov.au/rss.xml", type: RSS, connector: rss, tier: B } | |
| 1509 | + notes: "smartraveller.gov.au is unreachable from bots (connection reset)." | |
| 1510 | + - id: austender | |
| 1511 | + name: AusTender | |
| 1512 | + domain: tenders.gov.au | |
| 1513 | + categories: [government, procurement] | |
| 1514 | + tier: B | |
| 1515 | + aliases: [austender, australian government tenders] | |
| 1516 | + llm: false | |
| 1517 | + country: AU | |
| 1518 | + sensors: | |
| 1519 | + - { name: current approaches to market, url: "https://www.tenders.gov.au/public_data/rss/rss.xml", type: RSS, connector: rss, tier: B, config: { maxItems: 100 } } | |
| 1520 | + - id: australian-parliament | |
| 1521 | + name: Parliament of Australia | |
| 1522 | + domain: aph.gov.au | |
| 1523 | + categories: [government, politics] | |
| 1524 | + tier: B | |
| 1525 | + aliases: [parliament of australia, aph, federal parliament australia] | |
| 1526 | + country: AU | |
| 1527 | + sensors: | |
| 1528 | + - { name: bills before Parliament, url: "https://www.aph.gov.au/Parliamentary_Business/Bills_Legislation/Bills_before_Parliament", type: HTML, connector: http, tier: B } | |
| 1529 | + notes: "ParlInfo RSS (rss.w3p) is behind an Azure WAF that returns 403 after a few requests." | |
| 1530 | + - id: nsw-government | |
| 1531 | + name: NSW Government | |
| 1532 | + domain: nsw.gov.au | |
| 1533 | + categories: [government, politics] | |
| 1534 | + tier: B | |
| 1535 | + aliases: [nsw government, new south wales government] | |
| 1536 | + country: AU | |
| 1537 | + sensors: | |
| 1538 | + - { name: media releases, url: "https://www.nsw.gov.au/media-releases", type: HTML, connector: http, tier: B } | |
| 1539 | + - id: wa-government | |
| 1540 | + name: Government of Western Australia | |
| 1541 | + domain: wa.gov.au | |
| 1542 | + categories: [government, politics] | |
| 1543 | + tier: C | |
| 1544 | + aliases: [wa government, western australian government] | |
| 1545 | + country: AU | |
| 1546 | + sensors: | |
| 1547 | + - { name: media statements, url: "https://www.wa.gov.au/government/media-statements", type: HTML, connector: http, tier: C } | |
| 1548 | + notes: "SA, Tasmania, ACT and NT government sites sit behind Cloudflare challenges." | |
| 1549 | + # ───────────────────────────── INDIA ───────────────────────────── | |
| 1550 | + - id: camara-dos-deputados | |
| 1551 | + name: Câmara dos Deputados | |
| 1552 | + domain: camara.leg.br | |
| 1553 | + categories: [government, politics] | |
| 1554 | + tier: B | |
| 1555 | + aliases: [câmara dos deputados, camara dos deputados, chamber of deputies brazil] | |
| 1556 | + country: BR | |
| 1557 | + language: pt | |
| 1558 | + sensors: | |
| 1559 | + - { name: proposições apresentadas (Dados Abertos), url: "https://dadosabertos.camara.leg.br/api/v2/proposicoes?ordem=DESC&ordenarPor=id&itens=50", type: REST_API, connector: jsonlist, tier: B, config: { itemsPath: dados, keyField: id, titleTemplate: "{siglaTipo} {numero}/{ano} — {ementa}", dateField: dataApresentacao, urlTemplate: "https://www.camara.leg.br/propostas-legislativas/{key}", maxItems: 50 } } | |
| 1560 | + - { name: notícias — Plenário, url: "https://www.camara.leg.br/noticias/rss/dinamico/PLENARIO", type: RSS, connector: rss, tier: B } | |
| 1561 | + - id: senado-federal | |
| 1562 | + name: Senado Federal | |
| 1563 | + domain: senado.leg.br | |
| 1564 | + categories: [government, politics] | |
| 1565 | + tier: B | |
| 1566 | + aliases: [senado federal, senado brasil, federal senate brazil] | |
| 1567 | + country: BR | |
| 1568 | + language: pt | |
| 1569 | + sensors: | |
| 1570 | + - { name: notícias, url: "https://www12.senado.leg.br/noticias/rss.xml", type: ATOM, connector: rss, tier: B } | |
| 1571 | + - id: stf | |
| 1572 | + name: Supremo Tribunal Federal | |
| 1573 | + domain: stf.jus.br | |
| 1574 | + categories: [government, legal] | |
| 1575 | + tier: B | |
| 1576 | + aliases: [stf, supremo tribunal federal, supreme court brazil] | |
| 1577 | + country: BR | |
| 1578 | + language: pt | |
| 1579 | + sensors: | |
| 1580 | + - { name: notícias, url: "https://noticias.stf.jus.br/rss", type: RSS, connector: rss, tier: B } | |
| 1581 | + - id: ministerio-da-defesa | |
| 1582 | + name: Ministério da Defesa (Brasil) | |
| 1583 | + domain: gov.br | |
| 1584 | + homepage: https://www.gov.br/defesa/pt-br | |
| 1585 | + categories: [government] | |
| 1586 | + tier: C | |
| 1587 | + aliases: [ministério da defesa, defesa.gov.br] | |
| 1588 | + country: BR | |
| 1589 | + language: pt | |
| 1590 | + sensors: | |
| 1591 | + - { name: notícias, url: "https://www.gov.br/defesa/pt-br/centrais-de-conteudo/noticias/RSS", type: ATOM, connector: rss, tier: C, config: { maxItems: 50 } } | |
| 1592 | + - id: receita-federal | |
| 1593 | + name: Receita Federal do Brasil | |
| 1594 | + domain: gov.br | |
| 1595 | + homepage: https://www.gov.br/receitafederal/pt-br | |
| 1596 | + categories: [government, finance] | |
| 1597 | + tier: C | |
| 1598 | + aliases: [receita federal, rfb] | |
| 1599 | + country: BR | |
| 1600 | + language: pt | |
| 1601 | + sensors: | |
| 1602 | + - { name: notícias, url: "https://www.gov.br/receitafederal/pt-br/assuntos/noticias/RSS", type: ATOM, connector: rss, tier: C } | |
| 1603 | + - id: mdic | |
| 1604 | + name: Ministério do Desenvolvimento, Indústria, Comércio e Serviços | |
| 1605 | + domain: gov.br | |
| 1606 | + homepage: https://www.gov.br/mdic/pt-br | |
| 1607 | + categories: [government, commerce] | |
| 1608 | + tier: C | |
| 1609 | + aliases: [mdic] | |
| 1610 | + country: BR | |
| 1611 | + language: pt | |
| 1612 | + sensors: | |
| 1613 | + - { name: notícias, url: "https://www.gov.br/mdic/pt-br/assuntos/noticias/RSS", type: ATOM, connector: rss, tier: C } | |
| 1614 | + # ───────────────────────────── MEXICO ───────────────────────────── | |
| 1615 | + - id: ine-mexico | |
| 1616 | + name: Instituto Nacional Electoral | |
| 1617 | + domain: ine.mx | |
| 1618 | + categories: [government, elections] | |
| 1619 | + tier: B | |
| 1620 | + aliases: [ine, instituto nacional electoral, central electoral] | |
| 1621 | + country: MX | |
| 1622 | + language: es | |
| 1623 | + sensors: | |
| 1624 | + - { name: Central Electoral (boletines), url: "https://centralelectoral.ine.mx/feed/", type: RSS, connector: rss, tier: B } | |
| 1625 | + - id: ift-mexico | |
| 1626 | + name: Instituto Federal de Telecomunicaciones / CRT | |
| 1627 | + domain: ift.org.mx | |
| 1628 | + categories: [government, telecom] | |
| 1629 | + tier: C | |
| 1630 | + aliases: [ift, instituto federal de telecomunicaciones, comisión reguladora de telecomunicaciones] | |
| 1631 | + country: MX | |
| 1632 | + language: es | |
| 1633 | + sensors: | |
| 1634 | + - { name: comunicados, url: "https://www.ift.org.mx/rss.xml", type: RSS, connector: rss, tier: C } | |
| 1635 | + - id: scjn | |
| 1636 | + name: Suprema Corte de Justicia de la Nación | |
| 1637 | + domain: scjn.gob.mx | |
| 1638 | + categories: [government, legal] | |
| 1639 | + tier: B | |
| 1640 | + aliases: [scjn, suprema corte de justicia de la nación, supreme court mexico] | |
| 1641 | + country: MX | |
| 1642 | + language: es | |
| 1643 | + sensors: | |
| 1644 | + - { name: comunicados, url: "https://www.scjn.gob.mx/rss.xml", type: RSS, connector: rss, tier: B } | |
| 1645 | + - id: senado-mexico | |
| 1646 | + name: Senado de la República (México) | |
| 1647 | + domain: senado.gob.mx | |
| 1648 | + categories: [government, politics] | |
| 1649 | + tier: C | |
| 1650 | + aliases: [senado de la república, senado mexico] | |
| 1651 | + country: MX | |
| 1652 | + language: es | |
| 1653 | + sensors: | |
| 1654 | + - { name: Gaceta del Senado, url: "https://www.senado.gob.mx/66/gaceta_del_senado", type: HTML, connector: http, tier: C } | |
| 1655 | + - id: united-nations | |
| 1656 | + extend: true | |
| 1657 | + country: INT | |
| 1658 | + sensors: | |
| 1659 | + - { name: Security Council news, url: "https://main.un.org/securitycouncil/en/rss.xml", type: RSS, connector: rss, tier: B } | |
| 1660 | + - id: g20 | |
Diff truncated — file too large.