1<p align="center">2 <img src="docs/brand/websensor-wordmark.svg" alt="WebSensor" width="420">3</p>45<h1 align="center">WebSensor</h1>67<p align="center"><strong>The Web is changing. We are watching.</strong><br>8Real-time Internet change intelligence — <a href="https://www.websensor.io">www.websensor.io</a></p>910<p align="center">11 <img src="docs/brand/websensor-mark.svg" alt="" width="28" valign="middle">12 3 167 organizations · 7 066 active sensors · 16 connector families · 94 event classes · immutable evidence · live WebSocket feed13</p>1415---1617WebSensor continuously watches the public web for meaningful changes, preserves the evidence, connects related18signals into real-world events and surfaces important developments before they become obvious. It is not a19website monitor: it answers **what changed, where, when, exactly what, whether it was public or silent, how20important and unusual it is, who is responsible, which other sources confirm it, whether it is part of a larger21event, how fast the story is spreading and whether WebSensor was early**.2223Built by **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai) · hosted on24[MacLustr](https://www.maclustr.io), an Apple-silicon compute cluster in Québec.2526## Table of contents27281. [Brand](#brand)292. [What WebSensor does](#what-websensor-does)303. [Architecture](#architecture)314. [The pipeline, step by step](#the-pipeline-step-by-step)325. [Scoring model](#scoring-model)336. [Event taxonomy](#event-taxonomy)347. [Source registry](#source-registry)358. [Connectors](#connectors)369. [Data model](#data-model)3710. [Real-time protocol](#real-time-protocol)3811. [REST API](#rest-api)3912. [Frontend](#frontend)4013. [Repository layout](#repository-layout)4114. [Development](#development)4215. [Testing](#testing)4316. [Deployment and operations](#deployment-and-operations)4417. [Security and crawling etiquette](#security-and-crawling-etiquette)4518. [Roadmap](#roadmap)4619. [Credits](#credits)4748## Brand4950<p>51 <img src="docs/brand/websensor-mark.svg" alt="WebSensor mark" width="64">52 53 <img src="apps/web/public/icons/icon-192.png" alt="PWA icon" width="64">54</p>5556The mark is a **pulse in the shape of a W**: the waveform is grey until the moment a change is detected, then57turns signal green and ends in the sensor reading. It is drawn once as SVG and reused everywhere:5859| Asset | Path | Use |60|---|---|---|61| Mark (SVG) | `apps/web/src/app/icon.svg`, `apps/web/public/logo.svg`, `docs/brand/websensor-mark.svg` | favicon (SVG-capable browsers), nav, docs |62| Wordmark (SVG) | `docs/brand/websensor-wordmark.svg` | README, documents |63| `favicon.ico` | `apps/web/src/app/favicon.ico` | 16 / 32 / 48 px, legacy tabs |64| Apple touch icon | `apps/web/src/app/apple-icon.png` | 180 px |65| PWA icons | `apps/web/public/icons/icon-192.png`, `icon-512.png`, `icon-512-maskable.png` | manifest |66| Share image | `apps/web/src/app/opengraph-image.tsx` (+ `twitter-image.tsx`) | 1200 × 630, live counters from the API |67| Event share image | `apps/web/src/app/event/[slug]/opengraph-image.tsx` | per-event card with signal score and badges |68| React component | `apps/web/src/components/brand.tsx` (`Mark`, `PulseBackdrop`, `BRAND` colours) | app and OG images |6970Colours: background `#0a0e15`, ink `#c9d1e0`, signal `#22d3a5`, line `#2d394e`. Type: Inter (UI), JetBrains Mono71(numbers, identifiers, timestamps). The product aesthetic is *Bloomberg Terminal × Linear × Vercel × Datadog ×72threat-intelligence desk*: dense information, one-pixel borders, small badges, precise spacing, dark first.7374## What WebSensor does7576| Question | Where the answer comes from |77|---|---|78| What changed, exactly? | Semantic diff: field-level *before → after* pairs (prices, limits, versions, dates, status words, names) with % deltas, plus unified / side-by-side / semantic diff views |79| Where and when? | Sensor URL, canonical URL, `published_at` (claimed by the source), `observed_from` (previous snapshot), `detected_at`, `processed_at`, all UTC |80| Was it public or silent? | **Silent change** = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs, shutdown, feature removed, leadership, page removed…) · no matching announcement within 12 h · importance ≥ 45 |81| How important? | Importance (8 stored components) → **WebSensor Signal Score** 0–100 with an explainable reason list |82| Is it unusual? | Source activity anomaly vs a 14-day baseline; entity anomaly vs a 30-day baseline (`+1 800 % vs baseline`) |83| Who is responsible? | Entity resolution (organizations, products, models, APIs, regulators) with parent/child relations |84| Which sources confirm it? | Novelty vs the 72 h window, cross-source confirmations, `OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED` |85| Part of a larger event? | Event clusters with propagation timeline, first-party vs external signals, velocity, `breaking / developing / confirmed / watching` state |86| Was WebSensor early? | **Lead time**: first-party detection → first external report |87| What should I pay attention to? | `/breaking`, `/pulse`, `/radar` (weak signals), watchlists, alert rules (web + signed webhooks) |8889## Architecture9091```92 ┌──────────────────────── MacLustr node M4M64b ────────────────────────┐93 Internet ──TLS──▶ BHS64 │ websensor-api :8260 Fastify gateway │94 (OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │95 MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │96 │ websensor-engine scheduler + pipeline, Prometheus :8262 │97 │ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │98 └───────────────────────────────────────────────────────────────────────┘99```100101| Role | Package | Notes |102|---|---|---|103| Shared intelligence | `packages/core` | taxonomy, SSRF policy, hashing, canonical extraction, diff engines, heuristics, semantic diff, scoring, schedule, search syntax, registry schema. `@websensor/core/client` is the browser-safe subset |104| Persistence | `packages/db` | plain-SQL forward migrations (`migrations/*.sql`, advisory-locked) + Drizzle schema |105| Evidence | `packages/store` | content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres |106| Acquisition | `packages/connectors` | connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback |107| Ingestion | `apps/engine` | scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator |108| Gateway | `apps/api` | REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API |109| Product | `apps/web` | Next.js 16 App Router, design system, live feed, drawer, palette, desks |110111Everything is TypeScript ESM in a pnpm workspace (Node ≥ 22.15). One node runs everything today; every role is112separable (several engine processes can run concurrently thanks to `FOR UPDATE SKIP LOCKED` claims).113114## The pipeline, step by step115116Every sensor run goes through `apps/engine/src/pipeline.ts`:1171181. **Schedule** — the scheduler claims due sensors by priority (0 critical … 3 low) with `FOR UPDATE SKIP LOCKED`,119 enforces global concurrency and per-host limits (higher for large public APIs such as `github.com`,120 `data.sec.gov`), applies a per-domain circuit breaker (5 consecutive failures → 10 min pause) and publishes a121 heartbeat to Redis (`ws:engine:status`).1222. **Fetch** — conditional GET (`ETag` / `If-None-Match`, `Last-Modified` / `If-Modified-Since`), manual redirects123 validated per hop by the SSRF policy, HTTP/2 → HTTP/1.1 pinning on NGHTTP2 errors, a second attempt with a124 browser identity on 403/resets, size and time limits, gzip and UTF-16 BOM handling. `304` = cheap check.1253. **Normalize** — the connector turns the observation into a comparable representation: `text` (canonical126 HTML/plain text with navigation, footers, cookie banners, timestamps, tokens and counters stripped), `json`127 (sorted keys, volatile paths ignored) or `list` (keyed items: feed entries, sitemap URLs, incidents, releases,128 filings, records).1294. **Snapshot** — if the canonical hash changed, raw body and canonical form are stored immutably and a130 `snapshots` row records hashes, HTTP metadata, extraction confidence and the source's own timestamp.1315. **Diff** — text (paired modifications), json (path operations) or list (added / removed / modified) with132 connector guards: feed items are "new" only if never seen and < 14 days old, status pages losing an item is a133 resolution, a > 50 % sitemap shrink is a partial response, a 404 becomes `page_removed` only after separated134 confirmations.1356. **Heuristics + semantic class** — `evaluateChange` proposes an event type, magnitude, facts (prices, percents,136 versions, dates) and a signal 0–1 from ~90 typed rules and record-shape priors (KEV records, CVE ids, 8-K137 items, ClinicalTrials statuses…). `classifyChange` labels the change `cosmetic / navigation / timestamp /138 advertisement / boilerplate` (noise) or `meaningful / pricing / policy / product / personnel` (signal) and139 extracts field-level changes. A `changes` row is **always** written; noise classes never become events.1407. **Event candidate** — fingerprint idempotency (sensor + before/after canonical hashes), entity resolution141 (aliases, longest first; ordinary-word aliases such as `first` or `has` only match as exact upper-case142 acronyms), novelty vs the 72 h window (near-duplicates suppressed, redundant third-party reports suppressed),143 source anomaly, preliminary importance (routine firehose batches damped), optional **Claude interpretation**144 above `WS_LLM_MIN_IMPORTANCE` (strict JSON: type, title, summary, why it matters, observed vs inferred,145 severity, confidence, announced?), final importance + confidence + impact, silent-change bar.1468. **Cluster** — attach to an open cluster (shared entity or source, textual similarity within 6 h; same-entity147 cross-source stories within 2 h) or open one; update propagation timeline, first-party/external counts,148 velocity, lead time and breaking state.1499. **Signal score & publish** — `computeSignalScore` with stored reasons; `events` row; entity and source daily150 counters; alert evaluation (web notifications, HMAC-signed webhooks); Redis stream `ws:events` (durable, with151 stream ids) + pub/sub `ws:live`.15210. **Re-schedule** — adaptive interval from tier bounds, recency of change, change frequency, 304 ratio and errors153 (tier S 15 s–5 min · A 1–30 min · B 5 min–2 h · C 30 min–6 h · D 6–24 h).154155Background jobs: connector health rollups (1 min), cluster state ageing (5 min), retention (30 min: raw bodies156of snapshots without an event are dropped after 21 days, canonical forms and hashes kept, event snapshots kept157forever, blobs reference-counted), run log pruning, weekly discovery.158159## Scoring model160161**Importance** (0–100) = 25 % intrinsic severity of the event type · 20 % source authority (tier × weight) ·16215 % entity importance · 15 % novelty · 10 % magnitude · 5 % cross-source confirmation · 5 % user impact ·1635 % unusualness. All eight components are stored on the event.164165**Confidence** (0–100) blends source authenticity, extraction confidence, diff clarity, structured vs free text,166confirmations and LLM/heuristic agreement.167168**Impact** (0–100) weights the event type's user impact, entity importance, magnitude and the largest numeric169delta among extracted fields (`$100 → $500` scores higher than `$100 → $105`).170171**Velocity** (0–100) grows with signals per hour, unique sources and first-party signals inside a cluster.172173**WebSensor Signal Score** (0–100) = 42 % importance · 14 % confidence · 14 % novelty · 12 % velocity ·17412 % impact · 6 % anomaly, then adjusted with explainable reasons: `+ first-party evidence`, `+ N independent175confirmations`, `+ silent change`, `+ tier S source`, `- third-party report`, `- low confidence`,176`- unconfirmed`, `- routine batch from a high-volume feed`, and a hard cap of 20 for noise classes. The reason177list is shown on every event ("Why this score").178179**Breaking state** is not recency: `breaking` needs a strong signal, freshness (< 6 h) and either an180independent confirmation, a tier-S first-party source or signal ≥ 90; `developing` = signals accumulating;181`confirmed` = ≥ 3 sources within 24 h; `watching` otherwise; `closed` after 72 h.182183**Entity rank** = importance × activity (24 h, 7 d) × average signal × confirmation ratio × unique sources ×184acceleration vs the 30-day baseline — never raw event counts.185186## Event taxonomy18718894 event classes grouped for filters and desks:189190| Group | Examples |191|---|---|192| Security | `zero_day`, `active_exploitation`, `vulnerability`, `security_advisory`, `breach`, `supply_chain_attack`, `credential_leak`, `malware_campaign`, `ransomware`, `patch_release` |193| Reliability | `outage`, `incident`, `maintenance`, `infrastructure_change`, `dns_change`, `certificate_change` |194| Product & API | `product_launch`, `model_release`, `software_release`, `service_launch`, `service_shutdown`, `feature_removed`, `API_change`, `availability_change`, `new_region`, `documentation_change` |195| Pricing & terms | `pricing_change`, `terms_change`, `policy_change`, `crawler_policy_change` |196| Corporate | `earnings`, `guidance`, `financial_filing`, `acquisition`, `merger`, `ipo`, `funding`, `capital_raise`, `dividend`, `buyback`, `bankruptcy`, `rating_change`, `insider_transaction`, `leadership_change`, `layoffs` |197| Government & legal | `government_announcement`, `regulatory_filing`, `regulatory_action`, `legislation`, `budget`, `procurement`, `sanction`, `lawsuit`, `court_decision`, `emergency_notice`, `monetary_policy`, `economic_release` |198| Science & health | `scientific_publication`, `retraction`, `dataset_release`, `clinical_trial`, `drug_approval`, `drug_warning`, `recall`, `device_recall`, `outbreak`, `space_mission`, `standard_update` |199| Transport | `accident`, `grounding`, `safety_bulletin`, `route_change`, `production_delay` |200| Sports | `sports_transaction`, `sports_result`, `suspension`, `schedule_change` |201| Web | `announcement`, `page_created`, `page_removed`, `content_change`, `domain_registration_change` |202203Each class carries an intrinsic severity and a `usuallyAnnounced` flag used by silent-change detection. The204full list lives in `packages/core/src/taxonomy.ts`.205206## Source registry207208The registry is declarative YAML, merged in file-name order by `apps/engine/src/seeds.ts`:209210- `config/sources.yaml` — the founding 271 organizations;211- `config/sources.d/10–35` — one fragment per **site class** (open source, central banks & finance, energy /212 climate / weather, telecom & internet infrastructure, retail / consumer / travel, gaming & entertainment,213 universities & research, international organizations / NGOs / standards, sports, crypto, transport / aviation /214 space, consumer safety / food / agriculture, housing / labour / open data, enterprise SaaS, health systems &215 medtech, politics / elections / courts, EDGAR filings, package registries, web posture, OpenAPI, status JSON,216 documents & data);217- `config/sources.d/40–46` — **depth** fragments (AI frontier, cloud infrastructure, cybersecurity, finance &218 markets, governments incl. Canadian provinces, science & health, transport / telecom / sports / world news);219- `config/sources.d/47+` — the long tail (open-source projects, SaaS status pages and changelogs, S&P 500 /220 Nasdaq-100 / TSX 60 EDGAR issuers and 160 US federal agencies via the Federal Register API, cities and regions,221 world governments and regulators, corporate pricing / legal / leadership pages, sports teams, entertainment,222 education, regional media, trade press, think tanks).223224```yaml225sources:226 - id: bank-of-canada # kebab-case, unique across all files227 name: Bank of Canada228 domain: bankofcanada.ca229 categories: [finance, central-bank]230 tier: S # S 15–60 s · A 1–5 min · B 5–30 min · C 30 min–6 h · D 6–24 h231 weight: 1.4232 country: CA # ISO 3166-1 alpha-2, EU, INT233 language: en234 products:235 - { name: Valet API, type: API }236 sensors:237 - { name: press releases, url: "https://www.bankofcanada.ca/…/feed/", type: RSS, connector: rss, tier: A }238 - { name: policy rate, url: "https://www.bankofcanada.ca/valet/observations/V39079/json?recent=3", type: REST_API, connector: jsonlist, tier: A, config: { … } }239 - id: cbc # media report ABOUT others → third-party evidence240 extend: true241 first_party: false242```243244Rules that never bend:245246- **Nothing enters the registry without the validator's OK.** `apps/engine/src/validate.ts` runs every curated247 sensor through its connector (fetch + normalize) and prints `OK / WARN / FAIL`; `WARN` (empty list, thin248 client-rendered page) and `FAIL` (403, 404, parse error) sensors are pruned. Sites blocked by bot management249 are documented in `notes:`, never faked.250- One source = one organization (the entity people search for). `extend: true` adds sensors, products, aliases,251 categories, country, language or first-party flag to a source declared in an earlier file.252- Media and aggregators carry `first_party: false`; the organization's own channels are first-party and weigh253 more. High-volume feeds (news wires, arXiv, NVD, package streams, sports) run with `llm: false`.254- Country and language are inferred from unambiguous TLDs when not declared; scheduling priority is derived from255 tier and category.256257Registry sync (`cli.ts sync`) upserts sources, sensors, organization and product entities, aliases and258`owns` relations; seed sensors removed from YAML are disabled, discovery-created ones are kept. The discovery259engine probes `robots.txt` sitemaps, `<link rel="alternate">` feeds, well-known feed paths and linked status260pages, and promotes only candidates that fetch and parse. Sensors carry a lifecycle status261(`PENDING → VALIDATED → ACTIVE / DEGRADED / DISABLED`). Bulk import (JSON or YAML) is available through the262admin API with a dry-run mode. Authoring guide: `docs/registry/AUTHORING.md`.263264## Connectors265266All connectors implement `fetch(endpoint)` and `normalize(endpoint, observation)`; the shared pipeline does267compare, classify, score and persist (`packages/connectors/src/types.ts`).268269| Connector | Sensor types | Handles |270|---|---|---|271| `http` | HTML, JSON, XML, FILE, HTTP_HEADERS | canonical text extraction (`selector`, `keepChrome`), JSON with `jsonPath` / `ignoreKeys`, HEAD |272| `rss` | RSS, ATOM | RSS 2.0, Atom, RDF, JSON Feed; GitHub / GitLab / YouTube / Blogger / arXiv API feeds |273| `sitemap` | SITEMAP | sitemap index, urlset, news sitemaps (`maxUrls`, `maxChildren`) |274| `statuspage` | STATUSPAGE | Atlassian Statuspage `api/v2/summary.json` |275| `statusjson` | STATUSPAGE | Instatus, incident.io (multi-region subpages), Status.io |276| `github` | GITHUB_RELEASE, GITHUB_REPO | releases / tags / commits Atom, advisories REST |277| `jsonlist` | REST_API, JSON | keyed records from any JSON API (KEV, NVD, Federal Register, ClinicalTrials v2, USGS, HIBP, CKAN, Hugging Face…); `{now-2h}` placeholders, `{key}` / `{field.path}` URL templates |278| `package` | REST_API | npm, PyPI, crates.io, RubyGems, NuGet, Packagist, Hex, Go proxy, Homebrew, Docker Hub version streams |279| `edgar` | REST_API | SEC EDGAR company submissions → filings, 8-K items decoded |280| `openapi` | JSON | API contract fingerprints (operations, parameters, deprecations) |281| `csv` | FILE | open-data rows (`keyColumn`, `tail`, `skipRows`) — FRED, Valet, ECB, ONS, Treasury |282| `pdf` | FILE | text of official PDFs (statements, notices, license agreements) |283| `dns` | DNS | A / AAAA / NS / MX / TXT / CAA / SOA / DMARC via public resolvers |284| `tls` | TLS | certificate identity, chain, ALPN, expiry |285| `headers` | HTTP_HEADERS | security and infrastructure response headers (HSTS, CSP, CDN) |286| `rdap` | JSON | registrar, EPP status, nameservers, expiry via the IANA bootstrap |287| `discovery` | — | candidate feeds / sitemaps / status pages, every candidate fetched and parsed |288| `scrapfly` | — | budgeted anti-bot fallback, only where the source explicitly allows it |289290Per-connector documentation: `docs/connectors/*.md`.291292## Data model293294PostgreSQL 17; migrations in `packages/db/migrations` (`0001_init` … `0006_backfill_states`), all additive.295296```297sources ─┬─ sensors ─┬─ sensor_runs298 │ ├─ snapshots ─┐299 │ └─ changes ───┼─ events ─┬─ event_entities ─ entities ─┬─ entity_aliases300 │ │ ├─ interpretations (versioned) ├─ entity_relations301 │ │ └─ event_clusters (timeline, states)302 ├─ source_entities └─ urls / url_history303 ├─ source_daily entity_daily · metrics_daily · llm_usage · connector_health · discovery_candidates304 └─ (kind = custom, owner_token) watchlists · watchlist_items · alerts · notifications · bookmarks · saved_views305```306307Key columns on `events`: `event_type`, `title`, `summary`, `why_it_matters`, `importance` + `importance_components`,308`confidence`, `novelty`, `signal_score`, `velocity_score`, `impact_score`, `anomaly_score`, `score_reasons`,309`change_class`, `field_changes`, `silent_change`, `evidence_label`, `first_party`, `country`, `language`,310`fingerprint` (unique), `cluster_id`, `published_at` / `observed_from` / `detected_at` / `processed_at` /311`published_to_feed_at`, `detection_latency_ms`, `processing_latency_ms`, a generated `tsvector` for full-text312search. Raw evidence is immutable: snapshots and diffs are never rewritten; re-interpretation adds a row to313`interpretations`.314315## Real-time protocol316317`wss://www.websensor.io/api/v1/live` (protocol 2). One Redis subscriber fans out to every client; the engine318publishes each event once with its stream id.319320```321client → {"subscribe":["events:breaking","entity:org_openai","country:CA"]}322client → {"since":"1789105849242-0"} # after a reconnection: replay up to 500 missed events323client → {"ping":1}324server → {"type":"hello","protocol":2,"channels":[…]}325server → {"type":"event","sid":"1789105849242-0","channels":["events:global","events:ai"],"event":{…}}326server → {"type":"replay_done","since":"…","count":17}327server → {"type":"heartbeat","t":1789105849242} # every 25 s328```329330Channels: `events:global`, `events:breaking`, `events:silent`, `events:first-party`,331`events:<ai|cyber|finance|health|government|science|products|infrastructure|news>`,332`group:<security|reliability|product|commercial|corporate|government|science|transport|sports|web>`,333`country:<CC>`, `state:<breaking|developing>`, `type:<event_type>`, `entity:<id>`, `source:<id>`,334`watchlist:<id>` (server-side matching of entities, sources, keywords, categories, URLs, event types, countries).335The browser client (`apps/web/src/lib/use-live.ts`) reconnects with jittered backoff, detects half-dead sockets336with a 70 s watchdog and replays on reconnection; the feed de-duplicates by event id.337338## REST API339340Public, unauthenticated, 600 requests / minute / IP (`x-ratelimit-*` headers), consistent errors341`{ "error": "…", "detail"?: "…" }`, cursor pagination (`nextCursor` → `cursor`), all timestamps UTC.342Documentation with examples: https://www.websensor.io/api.343344| Endpoint | Purpose |345|---|---|346| `GET /api/v1/events` | filters `after before category entity source domain sensor cluster importance_min confidence_min signal_min event_type group silent_change first_party confirmed country language change_class q limit cursor order=recent\|importance\|signal`; `q` accepts the search syntax `entity:openai type:pricing_change after:7d silent:true importance:>70 country:CA` |347| `GET /api/v1/events/count` · `/events/{id\|slug}` | counts; detail with related events, cluster, change, interpretation versions, snapshots, sensor reliability, page history |348| `GET /api/v1/changes/{id}` · `/snapshots/{id}[?raw=1]` · `/snapshots/compare?a=&b=` · `/sensors/{id}/snapshots` | evidence: diff, canonical or raw body, compare any two snapshots, historical memory of a page |349| `GET /api/v1/breaking` · `/pulse` · `/radar` · `/trending` · `/explore` · `/stats` | intelligence desks and aggregates (cached 5–30 s) |350| `GET /api/v1/clusters` · `/clusters/{id\|slug}` | clusters with propagation timeline, first-party / external signals, lead time |351| `GET /api/v1/entities` · `/entities/rank` · `/entities/{id}` · `/entities/{id}/timeline` | entities, WebSensor ranking, insights (35-day heatmap, baseline, anomaly, velocity, rank) |352| `GET /api/v1/sources` · `/sources/{id}` · `/sensors/{id}` | registry, source quality score, sensor polling metadata (ETag, Last-Modified, intervals) |353| `GET /api/v1/countries[/{code}]` · `/categories/{channel}` · `/domains/{domain}/timeline` · `/urls/history?url=` | country and category desks, domain and URL history |354| `GET /api/v1/search?q=` | events, clusters, entities, sources, URLs with the parsed filters echoed |355| `GET /api/v1/health/connectors` | connector health, throughput, queue depth, engine heartbeat, failing domains |356| `GET /api/v1/feed.rss` | RSS 2.0 with the same filters as `/events` |357| Owner-scoped (`X-WebSensor-Owner`) | `/watchlists`, `/alerts` (web or HMAC-signed webhook channel), `/notifications`, `/bookmarks`, `/views`, `/monitors` (custom public URLs, SSRF-checked, per-owner limit) |358| Admin (`X-WebSensor-Admin`, only when `WS_ADMIN_TOKEN` is set) | `/admin/ops`, `/admin/failures`, sensor and source actions (`run-now`, `enable`, `disable`, patch), `/admin/sensors/test` (connector dry run), `/admin/sensors/bulk`, `/admin/sources/import?dry_run=1` (JSON or YAML) |359360## Frontend361362Next.js 16 (App Router, React 19, Tailwind v4). Server components by default; client components only where363interaction is needed.364365| Route | What it is |366|---|---|367| `/` | hero line, **live system strip** (sources, sensors, checks/min, events 24 h, breaking, silent, status — real values, refreshed every 10 s), live feed, right rail (breaking now, trending, anomalous), clusters, silent changes, infrastructure pulse |368| `/live` | full-width feed; filters persist in the URL (`/live?category=ai&signal_min=70&silent_change=true`); saved views |369| `/breaking` | breaking now · developing · recently confirmed · watching |370| `/silent` | the silent-change desk: previous state, current state, exact diff |371| `/pulse` | what is changing on the Internet right now |372| `/radar` | weak signals, clearly labelled as indicators |373| `/explore` | trending, breaking, silent, newly detected, most active, unusual, clusters, entities, sources, categories, countries |374| `/entity/[id]` | rank, activity, 35-day heatmap, anomaly, tabs overview / live / sources / silent / timeline (1 h → all) / related / metrics |375| `/cluster/[slug]` | web-velocity propagation timeline, first-party vs external, lead time |376| `/event/[slug]` | **what changed** first: field changes, before / after diff, why it matters (analysis), evidence, history, related signals, score transparency, OG image |377| `/source/[id]` · `/sensor/[id]` | source quality, polling metadata, sensors table; sensor history ("how this page looked over time") |378| `/category/[channel]` · `/country/[slug]` | real-time desks per category and per country |379| `/entities` · `/sources` · `/search` | rankings, registry browsing, advanced search |380| `/watchlists` · `/alerts` · `/bookmarks` · `/monitors` | anonymous owner-token tools |381| `/health` · `/ops` · `/api` · `/bot` | observability, token-gated operations dashboard, API docs, crawler policy |382383Interaction model: click any event → **intelligence drawer** (right on desktop, full-screen sheet on mobile);384`⌘K` / `Ctrl+K` command palette (entities, sources, events, clusters, URLs, navigation, commands); three density385modes (compact / normal / comfortable) persisted; pause live; `↑ N new events` without scroll jumps; badges386`BREAKING · DEVELOPING · SILENT · FIRST PARTY · EXTERNAL · CONFIRMED · ANOMALOUS`; skeleton loaders; real 404s;387dark and light themes; audited at 375, 390, 430, 768, 1024, 1440 and 1920 px.388389## Repository layout390391```392apps/393 api/ Fastify gateway: routes.ts (public), routes-user.ts (owner), routes-admin.ts, intel.ts, live.ts, cache.ts394 engine/ scheduler.ts, pipeline.ts, cluster.ts, entities.ts, interpret.ts (Claude), alerts.ts, retention.ts,395 registry.ts, seeds.ts, discovery, metrics.ts, cli.ts, validate.ts396 web/ Next.js app: src/app (routes), src/components (ui.tsx design system, live-feed, event-drawer,397 command-palette, live-strip, brand.tsx…), src/lib (api client, format, feed-filters, use-live, owner)398packages/399 core/ taxonomy · scoring · semantic · heuristics · diff · canonical · search · ssrf · schedule · registry-schema400 db/ schema.ts + migrations/*.sql401 store/ content-addressed blob store402 connectors/ fetcher + 16 connector families + discovery + scrapfly403config/404 sources.yaml, sources.d/*.yaml the registry405docs/406 ARCHITECTURE.md, PRODUCT-BRIEF.md, registry/AUTHORING.md, connectors/*.md, brand/407scripts/ registry generators (EDGAR, Federal Register, web posture, status probe) and pruning helpers408deploy/ mld manifest example and README409tests/ fixtures (RSS before/after, Atlassian summary, KEV, sitemap…)410```411412## Development413414```bash415createdb websensor && cp .env.example .env416pnpm install417pnpm db:migrate # or let the engine migrate on start418npx tsx apps/engine/src/cli.ts sync # registry → database (~6 s for 3 000 sources)419npx tsx apps/engine/src/cli.ts run-due 50 # run 50 due sensors through the pipeline420pnpm dev:api # gateway :8260421pnpm dev:engine # scheduler + pipeline422NEXT_PUBLIC_API_URL=http://localhost:8260 pnpm dev:web # :8261 — open the site here in dev423pnpm test · pnpm typecheck · pnpm --filter @websensor/web build424```425426Useful CLI commands (`apps/engine/src/cli.ts`): `sync`, `discover [sourceId…]`, `probe <domain>`,427`run-once <sensorId>`, `run-due [n]`, `relink-entities [days]` (re-resolve mentioned entities after alias-rule428changes), `refresh-clusters`, `prune-blobs`, `llm-test`. Registry validator:429`node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment.yaml> [--all] [--json report.json]430[--concurrency 6] [--probe <domain>]`; `scripts/prune-fragment.py <fragment> <report.json> [--keep-empty]`431removes failed sensors.432433Environment (see `.env.example`): `DATABASE_URL`, `REDIS_URL`, `BLOB_STORE_DIR`, `WS_FETCH_CONCURRENCY`,434`WS_PER_HOST_CONCURRENCY` (+ `WS_PER_HOST_OVERRIDES`), `WS_MEANINGFUL_SIGNAL`, `WS_SILENT_MIN_IMPORTANCE`,435`WS_RETENTION_*`, `ANTHROPIC_API_KEY` + `WS_LLM_*` (models, daily call budget, thresholds), `SCRAPFLY_API_KEY`,436`WS_ADMIN_TOKEN`, `WS_MONITORS_PER_OWNER`, `WS_USER_AGENT`.437438Dev tip: a copy of production data for UI work is `ssh <node> 'pg_dump websensor --data-only -t events …' |439psql websensor`, then `cli.ts sync`.440441## Testing442443`pnpm test` runs Vitest across the workspace:444445- `packages/core` — diff engines, canonicalization, heuristics, **semantic classifier and field extraction**446 (source-simulator fixtures: footer-year bump, relative-time bump, pricing change with deltas, leadership change,447 site redesign burst, JSON volatile paths, new feed items), new event classes (KEV → active exploitation,448 zero-day, supply chain, guidance, buyback, emergency notice, outbreak, sports), search syntax round-trip,449 impact / velocity / signal / breaking / anomaly scoring, title generation for URL-only and batch items;450- `packages/connectors` — RSS / Atom / JSON Feed, sitemap, Statuspage, KEV, package registries, EDGAR, OpenAPI,451 CSV, DNS/TLS/RDAP normalizers on fixtures;452- `apps/engine` — alert rule matching, country / language inference, priority tiers.453454Type safety is enforced with `pnpm typecheck` (strict TypeScript across all packages); the web app is also455linted with the React Compiler rules.456457## Deployment and operations458459Production runs on the MacLustr cluster, orchestrated by `mld` from the M1M32 gateway:460461```bash462~/Desktop/cluster-skill/mld stage . websensor463~/Desktop/cluster-skill/mld deploy websensor --node M4M64b464```465466Post-sync hooks (fail-fast): `pnpm install --frozen-lockfile` → `createdb` + migrations → registry sync →467`next build`. Processes (PM2): `websensor-engine`, `websensor-api` (:8260, published as468https://www.websensor.io through the MacLustr Tunnel — WireGuard to the OVH gateway BHS64 with Caddy TLS),469`websensor-web` (:8261, loopback). Secrets live only in the manifest on the gateway. After a deploy that changes470alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node.471472Observability: Prometheus metrics (`/api/metrics`, engine `:8262/metrics`: checks by outcome, bytes, changes by473class, events by type, LLM calls and tokens, stage latencies, suppressed candidates, queue depth, inflight),474`/health` (connectors, throughput, 304 ratio, failing domains, slowest sensors), `/ops` (engine heartbeat, queue,475storage, LLM usage, recent errors, connector dry-run, import). Retention keeps the database and blob store bounded476without ever deleting event evidence. Details: `deploy/README.md`, `docs/ARCHITECTURE.md`.477478## Security and crawling etiquette479480- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom481 monitors — passes `assertUrlAllowed()`: private ranges, loopback, link-local, cloud metadata endpoints,482 `.maclustr.io` / `.ts.net`, single-label hosts and NAT64-embedded private IPv4 are blocked, and the HTTP483 dispatcher only connects to validated addresses (DNS-rebinding safe).484- Clear `User-Agent` (`WebSensorBot/0.2 (+https://www.websensor.io/bot; contact@spboucher.ai)`), conditional485 requests, per-host concurrency, adaptive intervals, backoff, timeouts and a per-domain circuit breaker. Official486 APIs, feeds and structured endpoints are preferred; WebSensor does not bypass authentication or anti-bot487 protections (Scrapfly is a budgeted fallback only where a source explicitly allows it).488- Webhooks: HTTPS only, public hosts only, `X-WebSensor-Signature: sha256=<HMAC>`; custom monitors: public489 URLs only, tested before activation, hard per-owner limit, never shown in public feeds.490- Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting491 on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP.492493## Roadmap494495World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel496abstraction is in place); accounts and API keys; embeddings-based similarity; MinIO/S3 blob driver; browser497rendering for the few high-value client-rendered pages; MCP server; more depth fragments.498499## Credits500501WebSensor is designed and built by **Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai).502It runs on [MacLustr](https://www.maclustr.io), an Apple-silicon compute cluster in Québec. Source of truth:503`websensor.git` on spbgit (git.spboucher.ai). AI-generated summaries are labelled as analysis and never replace504the original evidence.505