TypeScript 55.4%
Python 43.2%
SQL 1.2%
1<p align="center">2 <img src="docs/brand/websensor-wordmark.svg" alt="WebSensor" width="420">3</p>45<h1 align="center">WebSensor</h1>67<p align="center"><strong>The Web is changing. We are watching.</strong><br>8Real-time Internet change intelligence — <a href="https://www.websensor.io">www.websensor.io</a></p>910<p align="center">11 <img src="docs/brand/websensor-mark.svg" alt="" width="28" valign="middle">12 5 669 organizations · 10 969 validated sensors · 16 connector families · 94 event classes · immutable evidence · live WebSocket feed13</p>1415---1617WebSensor continuously watches the public web for meaningful changes, preserves the evidence, connects related18signals into real-world events and surfaces important developments before they become obvious. It is not a19website monitor: it answers **what changed, where, when, exactly what, whether it was public or silent, how20important and unusual it is, who is responsible, which other sources confirm it, whether it is part of a larger21event, how fast the story is spreading and whether WebSensor was early**.2223Built by **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai) · hosted on24[MacLustr](https://www.maclustr.io), an Apple-silicon compute cluster in Québec.2526## Table of contents27281. [Brand](#brand)292. [What WebSensor does](#what-websensor-does)303. [Architecture](#architecture)314. [The pipeline, step by step](#the-pipeline-step-by-step)325. [Scoring model](#scoring-model)336. [Event taxonomy](#event-taxonomy)347. [Source registry](#source-registry)358. [Connectors](#connectors)369. [Data model](#data-model)3710. [Real-time protocol](#real-time-protocol)3811. [REST API](#rest-api)3912. [Frontend](#frontend)4013. [Repository layout](#repository-layout)4114. [Development](#development)4215. [Testing](#testing)4316. [Deployment and operations](#deployment-and-operations)4417. [Security and crawling etiquette](#security-and-crawling-etiquette)4518. [Roadmap](#roadmap)4619. [Credits](#credits)4748## Brand4950<p>51 <img src="docs/brand/websensor-mark.svg" alt="WebSensor mark" width="64">52 53 <img src="apps/web/public/icons/icon-192.png" alt="PWA icon" width="64">54</p>5556The mark is a **pulse in the shape of a W**: the waveform is grey until the moment a change is detected, then57turns signal green and ends in the sensor reading. It is drawn once as SVG and reused everywhere:5859| Asset | Path | Use |60|---|---|---|61| Mark (SVG) | `apps/web/src/app/icon.svg`, `apps/web/public/logo.svg`, `docs/brand/websensor-mark.svg` | favicon (SVG-capable browsers), nav, docs |62| Wordmark (SVG) | `docs/brand/websensor-wordmark.svg` | README, documents |63| `favicon.ico` | `apps/web/src/app/favicon.ico` | 16 / 32 / 48 px, legacy tabs |64| Apple touch icon | `apps/web/src/app/apple-icon.png` | 180 px |65| PWA icons | `apps/web/public/icons/icon-192.png`, `icon-512.png`, `icon-512-maskable.png` | manifest |66| Share image | `apps/web/src/app/opengraph-image.tsx` (+ `twitter-image.tsx`) | 1200 × 630, live counters from the API |67| Event share image | `apps/web/src/app/event/[slug]/opengraph-image.tsx` | per-event card with signal score and badges |68| React component | `apps/web/src/components/brand.tsx` (`Mark`, `PulseBackdrop`, `BRAND` colours) | app and OG images |6970Colours: background `#0a0e15`, ink `#c9d1e0`, signal `#22d3a5`, line `#2d394e`. Type: Inter (UI), JetBrains Mono71(numbers, identifiers, timestamps). The product aesthetic is *Bloomberg Terminal × Linear × Vercel × Datadog ×72threat-intelligence desk*: dense information, one-pixel borders, small badges, precise spacing, dark first.7374## What WebSensor does7576| Question | Where the answer comes from |77|---|---|78| What changed, exactly? | Semantic diff: field-level *before → after* pairs (prices, limits, versions, dates, status words, names) with % deltas, plus unified / side-by-side / semantic diff views |79| Where and when? | Sensor URL, canonical URL, `published_at` (claimed by the source), `observed_from` (previous snapshot), `detected_at`, `processed_at`, all UTC |80| Was it public or silent? | **Silent change** = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs, shutdown, feature removed, leadership, page removed…) · no matching announcement within 12 h · importance ≥ 45 |81| How important? | Importance (8 stored components) → **WebSensor Signal Score** 0–100 with an explainable reason list |82| Is it unusual? | Source activity anomaly vs a 14-day baseline; entity anomaly vs a 30-day baseline (`+1 800 % vs baseline`) |83| Who is responsible? | Entity resolution (organizations, products, models, APIs, regulators) with parent/child relations |84| Which sources confirm it? | Novelty vs the 72 h window, cross-source confirmations, `OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED` |85| Part of a larger event? | Event clusters with propagation timeline, first-party vs external signals, velocity, `breaking / developing / confirmed / watching` state |86| Was WebSensor early? | **Lead time**: first-party detection → first external report |87| What should I pay attention to? | `/breaking`, `/pulse`, `/radar` (weak signals), watchlists, alert rules (web + signed webhooks) |8889## Architecture9091```92 ┌──────────────────────── MacLustr node M4M64b ────────────────────────┐93 Internet ──TLS──▶ BHS64 │ websensor-api :8260 Fastify gateway │94 (OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │95 MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │96 │ websensor-engine scheduler + pipeline, Prometheus :8262 │97 │ websensor-factory Source Factory: seeds → discovery → shadow │98 │ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │99 └───────────────────────────────────────────────────────────────────────┘100```101102| Role | Package | Notes |103|---|---|---|104| Shared intelligence | `packages/core` | taxonomy, SSRF policy, hashing, canonical extraction, diff engines, heuristics, semantic diff, scoring, schedule, search syntax, registry schema. `@websensor/core/client` is the browser-safe subset |105| Persistence | `packages/db` | plain-SQL forward migrations (`migrations/*.sql`, advisory-locked) + Drizzle schema |106| Evidence | `packages/store` | content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres |107| Acquisition | `packages/connectors` | connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback |108| Ingestion | `apps/engine` | scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator |109| Expansion | `apps/engine/src/factory` | **Source Factory** (0.3): coverage universes → deep discovery → scored candidates → shadow sensors → evidence-based acceptance; `websensor-factory` process |110| Coverage | `apps/api/src/coverage.ts` | **Global Observation Coverage Score** per sector (`/coverage`, `GET /api/v1/coverage`): how much of the world's high-value public web is observed |111| Gateway | `apps/api` | REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API |112| Product | `apps/web` | Next.js 16 App Router, design system, live feed, drawer, palette, desks |113114Everything is TypeScript ESM in a pnpm workspace (Node ≥ 22.15). One node runs everything today; every role is115separable (several engine processes can run concurrently thanks to `FOR UPDATE SKIP LOCKED` claims).116117## The pipeline, step by step118119Every sensor run goes through `apps/engine/src/pipeline.ts`:1201211. **Schedule** — the scheduler claims due sensors by priority (0 critical … 3 low) with `FOR UPDATE SKIP LOCKED`,122 enforces global concurrency and per-host limits (higher for large public APIs such as `github.com`,123 `data.sec.gov`), applies a per-domain circuit breaker (5 consecutive failures → 10 min pause) and publishes a124 heartbeat to Redis (`ws:engine:status`).1252. **Fetch** — conditional GET (`ETag` / `If-None-Match`, `Last-Modified` / `If-Modified-Since`), manual redirects126 validated per hop by the SSRF policy, HTTP/2 → HTTP/1.1 pinning on NGHTTP2 errors, a second attempt with a127 browser identity on 403/resets, size and time limits, gzip and UTF-16 BOM handling. `304` = cheap check.1283. **Normalize** — the connector turns the observation into a comparable representation: `text` (canonical129 HTML/plain text with navigation, footers, cookie banners, timestamps, tokens and counters stripped), `json`130 (sorted keys, volatile paths ignored) or `list` (keyed items: feed entries, sitemap URLs, incidents, releases,131 filings, records).1324. **Snapshot** — if the canonical hash changed, raw body and canonical form are stored immutably and a133 `snapshots` row records hashes, HTTP metadata, extraction confidence and the source's own timestamp.1345. **Diff** — text (paired modifications), json (path operations) or list (added / removed / modified) with135 connector guards: feed items are "new" only if never seen and < 14 days old, status pages losing an item is a136 resolution, a > 50 % sitemap shrink is a partial response, a 404 becomes `page_removed` only after separated137 confirmations.1386. **Heuristics + semantic class** — `evaluateChange` proposes an event type, magnitude, facts (prices, percents,139 versions, dates) and a signal 0–1 from ~90 typed rules and record-shape priors (KEV records, CVE ids, 8-K140 items, ClinicalTrials statuses…). `classifyChange` labels the change `cosmetic / navigation / timestamp /141 advertisement / boilerplate` (noise) or `meaningful / pricing / policy / product / personnel` (signal) and142 extracts field-level changes. A `changes` row is **always** written; noise classes never become events.1437. **Event candidate** — fingerprint idempotency (sensor + before/after canonical hashes), entity resolution144 (aliases, longest first; ordinary-word aliases such as `first` or `has` only match as exact upper-case145 acronyms), novelty vs the 72 h window (near-duplicates suppressed, redundant third-party reports suppressed),146 source anomaly, preliminary importance (routine firehose batches damped), optional **Claude interpretation**147 above `WS_LLM_MIN_IMPORTANCE` (strict JSON: type, title, summary, why it matters, observed vs inferred,148 severity, confidence, announced?), final importance + confidence + impact, silent-change bar.1498. **Cluster** — attach to an open cluster (shared entity or source, textual similarity within 6 h; same-entity150 cross-source stories within 2 h) or open one; update propagation timeline, first-party/external counts,151 velocity, lead time and breaking state.1529. **Signal score & publish** — `computeSignalScore` with stored reasons; `events` row; entity and source daily153 counters; alert evaluation (web notifications, HMAC-signed webhooks); Redis stream `ws:events` (durable, with154 stream ids) + pub/sub `ws:live`.15510. **Re-schedule** — adaptive interval from tier bounds, recency of change, change frequency, 304 ratio and errors156 (tier S 15 s–5 min · A 1–30 min · B 5 min–2 h · C 30 min–6 h · D 6–24 h).157158Background jobs: connector health rollups (1 min), cluster state ageing (5 min), retention (30 min: raw bodies159of snapshots without an event are dropped after 21 days, canonical forms and hashes kept, event snapshots kept160forever, blobs reference-counted), run log pruning, weekly discovery.161162## Scoring model163164**Importance** (0–100) = 25 % intrinsic severity of the event type · 20 % source authority (tier × weight) ·16515 % entity importance · 15 % novelty · 10 % magnitude · 5 % cross-source confirmation · 5 % user impact ·1665 % unusualness. All eight components are stored on the event.167168**Confidence** (0–100) blends source authenticity, extraction confidence, diff clarity, structured vs free text,169confirmations and LLM/heuristic agreement.170171**Impact** (0–100) weights the event type's user impact, entity importance, magnitude and the largest numeric172delta among extracted fields (`$100 → $500` scores higher than `$100 → $105`).173174**Velocity** (0–100) grows with signals per hour, unique sources and first-party signals inside a cluster.175176**WebSensor Signal Score** (0–100) = 42 % importance · 14 % confidence · 14 % novelty · 12 % velocity ·17712 % impact · 6 % anomaly, then adjusted with explainable reasons: `+ first-party evidence`, `+ N independent178confirmations`, `+ silent change`, `+ tier S source`, `- third-party report`, `- low confidence`,179`- unconfirmed`, `- routine batch from a high-volume feed`, and a hard cap of 20 for noise classes. The reason180list is shown on every event ("Why this score").181182**Breaking state** is not recency: `breaking` needs a strong signal, freshness (< 6 h) and either an183independent confirmation, a tier-S first-party source or signal ≥ 90; `developing` = signals accumulating;184`confirmed` = ≥ 3 sources within 24 h; `watching` otherwise; `closed` after 72 h.185186**Entity rank** = importance × activity (24 h, 7 d) × average signal × confirmation ratio × unique sources ×187acceleration vs the 30-day baseline — never raw event counts.188189## Event taxonomy19019194 event classes grouped for filters and desks:192193| Group | Examples |194|---|---|195| Security | `zero_day`, `active_exploitation`, `vulnerability`, `security_advisory`, `breach`, `supply_chain_attack`, `credential_leak`, `malware_campaign`, `ransomware`, `patch_release` |196| Reliability | `outage`, `incident`, `maintenance`, `infrastructure_change`, `dns_change`, `certificate_change` |197| Product & API | `product_launch`, `model_release`, `software_release`, `service_launch`, `service_shutdown`, `feature_removed`, `API_change`, `availability_change`, `new_region`, `documentation_change` |198| Pricing & terms | `pricing_change`, `terms_change`, `policy_change`, `crawler_policy_change` |199| Corporate | `earnings`, `guidance`, `financial_filing`, `acquisition`, `merger`, `ipo`, `funding`, `capital_raise`, `dividend`, `buyback`, `bankruptcy`, `rating_change`, `insider_transaction`, `leadership_change`, `layoffs` |200| Government & legal | `government_announcement`, `regulatory_filing`, `regulatory_action`, `legislation`, `budget`, `procurement`, `sanction`, `lawsuit`, `court_decision`, `emergency_notice`, `monetary_policy`, `economic_release` |201| Science & health | `scientific_publication`, `retraction`, `dataset_release`, `clinical_trial`, `drug_approval`, `drug_warning`, `recall`, `device_recall`, `outbreak`, `space_mission`, `standard_update` |202| Transport | `accident`, `grounding`, `safety_bulletin`, `route_change`, `production_delay` |203| Sports | `sports_transaction`, `sports_result`, `suspension`, `schedule_change` |204| Web | `announcement`, `page_created`, `page_removed`, `content_change`, `domain_registration_change` |205206Each class carries an intrinsic severity and a `usuallyAnnounced` flag used by silent-change detection. The207full list lives in `packages/core/src/taxonomy.ts`.208209## Source registry210211The registry is declarative YAML, merged in file-name order by `apps/engine/src/seeds.ts`:212213- `config/sources.yaml` — the founding 271 organizations;214- `config/sources.d/10–35` — one fragment per **site class** (open source, central banks & finance, energy /215 climate / weather, telecom & internet infrastructure, retail / consumer / travel, gaming & entertainment,216 universities & research, international organizations / NGOs / standards, sports, crypto, transport / aviation /217 space, consumer safety / food / agriculture, housing / labour / open data, enterprise SaaS, health systems &218 medtech, politics / elections / courts, EDGAR filings, package registries, web posture, OpenAPI, status JSON,219 documents & data);220- `config/sources.d/40–46` — **depth** fragments (AI frontier, cloud infrastructure, cybersecurity, finance &221 markets, governments incl. Canadian provinces, science & health, transport / telecom / sports / world news);222- `config/sources.d/47–55` — **breadth** fragments, wave 2 (2026-09-11): the open-source long tail (482 sensors),223 SaaS status pages and changelogs (923, incl. 208 new status sensors), 424 more EDGAR issuers, 184 US federal224 agencies via the Federal Register API, cities / regions / public bodies (356), world governments and regulators225 in 55 countries (535), the corporate pricing / legal / careers "silent change" surface of ~250 companies (702),226 sports clubs / entertainment / education (396), regional media / trade press / think tanks in 68 countries (638),227 and a small cross-fragment `extend` file. Every sensor was validated live before entering the registry; the228 fragment headers record what was probed and blocked (WAF, JS shells, broken TLS chains).229230```yaml231sources:232 - id: bank-of-canada # kebab-case, unique across all files233 name: Bank of Canada234 domain: bankofcanada.ca235 categories: [finance, central-bank]236 tier: S # S 15–60 s · A 1–5 min · B 5–30 min · C 30 min–6 h · D 6–24 h237 weight: 1.4238 country: CA # ISO 3166-1 alpha-2, EU, INT239 language: en240 products:241 - { name: Valet API, type: API }242 sensors:243 - { name: press releases, url: "https://www.bankofcanada.ca/…/feed/", type: RSS, connector: rss, tier: A }244 - { name: policy rate, url: "https://www.bankofcanada.ca/valet/observations/V39079/json?recent=3", type: REST_API, connector: jsonlist, tier: A, config: { … } }245 - id: cbc # media report ABOUT others → third-party evidence246 extend: true247 first_party: false248```249250Rules that never bend:251252- **Nothing enters the registry without the validator's OK.** `apps/engine/src/validate.ts` runs every curated253 sensor through its connector (fetch + normalize) and prints `OK / WARN / FAIL`; `WARN` (empty list, thin254 client-rendered page) and `FAIL` (403, 404, parse error) sensors are pruned. Sites blocked by bot management255 are documented in `notes:`, never faked.256- One source = one organization (the entity people search for). `extend: true` adds sensors, products, aliases,257 categories, country, language or first-party flag to a source declared in an earlier file.258- Media and aggregators carry `first_party: false`; the organization's own channels are first-party and weigh259 more. High-volume feeds (news wires, arXiv, NVD, package streams, sports) run with `llm: false`.260- Country and language are inferred from unambiguous TLDs when not declared; scheduling priority is derived from261 tier and category.262263Registry sync (`cli.ts sync`) upserts sources, sensors, organization and product entities, aliases and264`owns` relations; seed sensors removed from YAML are disabled, discovery-created ones are kept. The discovery265engine probes `robots.txt` sitemaps, `<link rel="alternate">` feeds, well-known feed paths and linked status266pages, and promotes only candidates that fetch and parse. Sensors carry a lifecycle status267(`PENDING → VALIDATED → ACTIVE / DEGRADED / DISABLED`). Bulk import (JSON or YAML) is available through the268admin API with a dry-run mode. Authoring guide: `docs/registry/AUTHORING.md`.269270## Connectors271272All connectors implement `fetch(endpoint)` and `normalize(endpoint, observation)`; the shared pipeline does273compare, classify, score and persist (`packages/connectors/src/types.ts`).274275| Connector | Sensor types | Handles |276|---|---|---|277| `http` | HTML, JSON, XML, FILE, HTTP_HEADERS | canonical text extraction (`selector`, `keepChrome`), JSON with `jsonPath` / `ignoreKeys`, HEAD |278| `rss` | RSS, ATOM | RSS 2.0, Atom, RDF, JSON Feed; GitHub / GitLab / YouTube / Blogger / arXiv API feeds |279| `sitemap` | SITEMAP | sitemap index, urlset, news sitemaps (`maxUrls`, `maxChildren`) |280| `statuspage` | STATUSPAGE | Atlassian Statuspage `api/v2/summary.json` |281| `statusjson` | STATUSPAGE | Instatus, incident.io (multi-region subpages), Status.io |282| `github` | GITHUB_RELEASE, GITHUB_REPO | releases / tags / commits Atom, advisories REST |283| `jsonlist` | REST_API, JSON | keyed records from any JSON API (KEV, NVD, Federal Register, ClinicalTrials v2, USGS, HIBP, CKAN, Hugging Face…); `{now-2h}` placeholders, `{key}` / `{field.path}` URL templates |284| `package` | REST_API | npm, PyPI, crates.io, RubyGems, NuGet, Packagist, Hex, Go proxy, Homebrew, Docker Hub version streams |285| `edgar` | REST_API | SEC EDGAR company submissions → filings, 8-K items decoded |286| `openapi` | JSON | API contract fingerprints (operations, parameters, deprecations) |287| `csv` | FILE | open-data rows (`keyColumn`, `tail`, `skipRows`) — FRED, Valet, ECB, ONS, Treasury |288| `pdf` | FILE | text of official PDFs (statements, notices, license agreements) |289| `dns` | DNS | A / AAAA / NS / MX / TXT / CAA / SOA / DMARC via public resolvers |290| `tls` | TLS | certificate identity, chain, ALPN, expiry |291| `headers` | HTTP_HEADERS | security and infrastructure response headers (HSTS, CSP, CDN) |292| `rdap` | JSON | registrar, EPP status, nameservers, expiry via the IANA bootstrap |293| `discovery` | — | candidate feeds / sitemaps / status pages, every candidate fetched and parsed |294| `scrapfly` | — | budgeted anti-bot fallback, only where the source explicitly allows it |295296Per-connector documentation: `docs/connectors/*.md`.297298## Data model299300PostgreSQL 17; migrations in `packages/db/migrations` (`0001_init` … `0006_backfill_states`), all additive.301302```303sources ─┬─ sensors ─┬─ sensor_runs304 │ ├─ snapshots ─┐305 │ └─ changes ───┼─ events ─┬─ event_entities ─ entities ─┬─ entity_aliases306 │ │ ├─ interpretations (versioned) ├─ entity_relations307 │ │ └─ event_clusters (timeline, states)308 ├─ source_entities └─ urls / url_history309 ├─ source_daily entity_daily · metrics_daily · llm_usage · connector_health · discovery_candidates310 └─ (kind = custom, owner_token) watchlists · watchlist_items · alerts · notifications · bookmarks · saved_views311```312313Key columns on `events`: `event_type`, `title`, `summary`, `why_it_matters`, `importance` + `importance_components`,314`confidence`, `novelty`, `signal_score`, `velocity_score`, `impact_score`, `anomaly_score`, `score_reasons`,315`change_class`, `field_changes`, `silent_change`, `evidence_label`, `first_party`, `country`, `language`,316`fingerprint` (unique), `cluster_id`, `published_at` / `observed_from` / `detected_at` / `processed_at` /317`published_to_feed_at`, `detection_latency_ms`, `processing_latency_ms`, a generated `tsvector` for full-text318search. Raw evidence is immutable: snapshots and diffs are never rewritten; re-interpretation adds a row to319`interpretations`.320321## Real-time protocol322323`wss://www.websensor.io/api/v1/live` (protocol 2). One Redis subscriber fans out to every client; the engine324publishes each event once with its stream id.325326```327client → {"subscribe":["events:breaking","entity:org_openai","country:CA"]}328client → {"since":"1789105849242-0"} # after a reconnection: replay up to 500 missed events329client → {"ping":1}330server → {"type":"hello","protocol":2,"channels":[…]}331server → {"type":"event","sid":"1789105849242-0","channels":["events:global","events:ai"],"event":{…}}332server → {"type":"replay_done","since":"…","count":17}333server → {"type":"heartbeat","t":1789105849242} # every 25 s334```335336Channels: `events:global`, `events:breaking`, `events:silent`, `events:first-party`,337`events:<ai|cyber|finance|health|government|science|products|infrastructure|news>`,338`group:<security|reliability|product|commercial|corporate|government|science|transport|sports|web>`,339`country:<CC>`, `state:<breaking|developing>`, `type:<event_type>`, `entity:<id>`, `source:<id>`,340`watchlist:<id>` (server-side matching of entities, sources, keywords, categories, URLs, event types, countries).341The browser client (`apps/web/src/lib/use-live.ts`) reconnects with jittered backoff, detects half-dead sockets342with a 70 s watchdog and replays on reconnection; the feed de-duplicates by event id.343344## REST API345346Public, unauthenticated, 600 requests / minute / IP (`x-ratelimit-*` headers), consistent errors347`{ "error": "…", "detail"?: "…" }`, cursor pagination (`nextCursor` → `cursor`), all timestamps UTC.348Documentation with examples: https://www.websensor.io/api.349350| Endpoint | Purpose |351|---|---|352| `GET /api/v1/events` | filters `after before category entity source domain sensor cluster importance_min confidence_min signal_min event_type group silent_change first_party confirmed country language change_class q limit cursor order=recent\|importance\|signal`; `q` accepts the search syntax `entity:openai type:pricing_change after:7d silent:true importance:>70 country:CA` |353| `GET /api/v1/events/count` · `/events/{id\|slug}` | counts; detail with related events, cluster, change, interpretation versions, snapshots, sensor reliability, page history |354| `GET /api/v1/changes/{id}` · `/snapshots/{id}[?raw=1]` · `/snapshots/compare?a=&b=` · `/sensors/{id}/snapshots` | evidence: diff, canonical or raw body, compare any two snapshots, historical memory of a page |355| `GET /api/v1/breaking` · `/pulse` · `/radar` · `/trending` · `/explore` · `/stats` | intelligence desks and aggregates (cached 5–30 s) |356| `GET /api/v1/clusters` · `/clusters/{id\|slug}` | clusters with propagation timeline, first-party / external signals, lead time |357| `GET /api/v1/entities` · `/entities/rank` · `/entities/{id}` · `/entities/{id}/timeline` | entities, WebSensor ranking, insights (35-day heatmap, baseline, anomaly, velocity, rank) |358| `GET /api/v1/sources` · `/sources/{id}` · `/sensors/{id}` | registry, source quality score, sensor polling metadata (ETag, Last-Modified, intervals) |359| `GET /api/v1/countries[/{code}]` · `/categories/{channel}` · `/domains/{domain}/timeline` · `/urls/history?url=` | country and category desks, domain and URL history |360| `GET /api/v1/search?q=` | events, clusters, entities, sources, URLs with the parsed filters echoed |361| `GET /api/v1/health/connectors` | connector health, throughput, queue depth, engine heartbeat, failing domains |362| `GET /api/v1/feed.rss` | RSS 2.0 with the same filters as `/events` |363| Owner-scoped (`X-WebSensor-Owner`) | `/watchlists`, `/alerts` (web or HMAC-signed webhook channel), `/notifications`, `/bookmarks`, `/views`, `/monitors` (custom public URLs, SSRF-checked, per-owner limit) |364| Admin (`X-WebSensor-Admin`, only when `WS_ADMIN_TOKEN` is set) | `/admin/ops`, `/admin/failures`, sensor and source actions (`run-now`, `enable`, `disable`, patch), `/admin/sensors/test` (connector dry run), `/admin/sensors/bulk`, `/admin/sources/import?dry_run=1` (JSON or YAML) |365366## Frontend367368Next.js 16 (App Router, React 19, Tailwind v4). Server components by default; client components only where369interaction is needed.370371| Route | What it is |372|---|---|373| `/` | hero line, **live system strip** (sources, sensors, checks/min, events 24 h, breaking, silent, status — real values, refreshed every 10 s), live feed, right rail (breaking now, trending, anomalous), clusters, silent changes, infrastructure pulse |374| `/live` | full-width feed; filters persist in the URL (`/live?category=ai&signal_min=70&silent_change=true`); saved views |375| `/breaking` | breaking now · developing · recently confirmed · watching |376| `/silent` | the silent-change desk: previous state, current state, exact diff |377| `/pulse` | what is changing on the Internet right now |378| `/radar` | weak signals, clearly labelled as indicators |379| `/explore` | trending, breaking, silent, newly detected, most active, unusual, clusters, entities, sources, categories, countries |380| `/entity/[id]` | rank, activity, 35-day heatmap, anomaly, tabs overview / live / sources / silent / timeline (1 h → all) / related / metrics |381| `/cluster/[slug]` | web-velocity propagation timeline, first-party vs external, lead time |382| `/event/[slug]` | **what changed** first: field changes, before / after diff, why it matters (analysis), evidence, history, related signals, score transparency, OG image |383| `/source/[id]` · `/sensor/[id]` | source quality, polling metadata, sensors table; sensor history ("how this page looked over time") |384| `/category/[channel]` · `/country/[slug]` | real-time desks per category and per country |385| `/entities` · `/sources` · `/search` | rankings, registry browsing, advanced search |386| `/watchlists` · `/alerts` · `/bookmarks` · `/monitors` | anonymous owner-token tools |387| `/health` · `/ops` · `/api` · `/bot` | observability, token-gated operations dashboard, API docs, crawler policy |388389Interaction model: click any event → **intelligence drawer** (right on desktop, full-screen sheet on mobile);390`⌘K` / `Ctrl+K` command palette (entities, sources, events, clusters, URLs, navigation, commands); three density391modes (compact / normal / comfortable) persisted; pause live; `↑ N new events` without scroll jumps; badges392`BREAKING · DEVELOPING · SILENT · FIRST PARTY · EXTERNAL · CONFIRMED · ANOMALOUS`; skeleton loaders; real 404s;393dark and light themes; audited at 375, 390, 430, 768, 1024, 1440 and 1920 px.394395## Repository layout396397```398apps/399 api/ Fastify gateway: routes.ts (public), routes-user.ts (owner), routes-admin.ts, intel.ts, live.ts, cache.ts400 engine/ scheduler.ts, pipeline.ts, cluster.ts, entities.ts, interpret.ts (Claude), alerts.ts, retention.ts,401 registry.ts, seeds.ts, discovery, metrics.ts, cli.ts, validate.ts402 web/ Next.js app: src/app (routes), src/components (ui.tsx design system, live-feed, event-drawer,403 command-palette, live-strip, brand.tsx…), src/lib (api client, format, feed-filters, use-live, owner)404packages/405 core/ taxonomy · scoring · semantic · heuristics · diff · canonical · search · ssrf · schedule · registry-schema406 db/ schema.ts + migrations/*.sql407 store/ content-addressed blob store408 connectors/ fetcher + 16 connector families + discovery + scrapfly409config/410 sources.yaml, sources.d/*.yaml the registry411docs/412 ARCHITECTURE.md, PRODUCT-BRIEF.md, registry/AUTHORING.md, connectors/*.md, brand/413scripts/ registry generators (EDGAR, Federal Register, web posture, status probe) and pruning helpers414deploy/ mld manifest example and README415tests/ fixtures (RSS before/after, Atlassian summary, KEV, sitemap…)416```417418## Development419420```bash421createdb websensor && cp .env.example .env422pnpm install423pnpm db:migrate # or let the engine migrate on start424npx tsx apps/engine/src/cli.ts sync # registry → database (~6 s for 3 000 sources)425npx tsx apps/engine/src/cli.ts run-due 50 # run 50 due sensors through the pipeline426pnpm dev:api # gateway :8260427pnpm dev:engine # scheduler + pipeline428NEXT_PUBLIC_API_URL=http://localhost:8260 pnpm dev:web # :8261 — open the site here in dev429pnpm test · pnpm typecheck · pnpm --filter @websensor/web build430```431432Useful CLI commands (`apps/engine/src/cli.ts`): `sync`, `discover [sourceId…]`, `probe <domain>`,433`run-once <sensorId>`, `run-due [n]`, `relink-entities [days]` (re-resolve mentioned entities after alias-rule434changes), `refresh-clusters`, `prune-blobs`, `llm-test`. Registry validator:435`node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment.yaml> [--all] [--json report.json]436[--concurrency 6] [--probe <domain>]`; `scripts/prune-fragment.py <fragment> <report.json> [--keep-empty]`437removes failed sensors.438439Environment (see `.env.example`): `DATABASE_URL`, `REDIS_URL`, `BLOB_STORE_DIR`, `WS_FETCH_CONCURRENCY`,440`WS_PER_HOST_CONCURRENCY` (+ `WS_PER_HOST_OVERRIDES`), `WS_MEANINGFUL_SIGNAL`, `WS_SILENT_MIN_IMPORTANCE`,441`WS_RETENTION_*`, `ANTHROPIC_API_KEY` + `WS_LLM_*` (models, daily call budget, thresholds), `SCRAPFLY_API_KEY`,442`WS_ADMIN_TOKEN`, `WS_MONITORS_PER_OWNER`, `WS_USER_AGENT`.443444Dev tip: a copy of production data for UI work is `ssh <node> 'pg_dump websensor --data-only -t events …' |445psql websensor`, then `cli.ts sync`.446447## Testing448449`pnpm test` runs Vitest across the workspace:450451- `packages/core` — diff engines, canonicalization, heuristics, **semantic classifier and field extraction**452 (source-simulator fixtures: footer-year bump, relative-time bump, pricing change with deltas, leadership change,453 site redesign burst, JSON volatile paths, new feed items), new event classes (KEV → active exploitation,454 zero-day, supply chain, guidance, buyback, emergency notice, outbreak, sports), search syntax round-trip,455 impact / velocity / signal / breaking / anomaly scoring, title generation for URL-only and batch items;456- `packages/connectors` — RSS / Atom / JSON Feed, sitemap, Statuspage, KEV, package registries, EDGAR, OpenAPI,457 CSV, DNS/TLS/RDAP normalizers on fixtures;458- `apps/engine` — alert rule matching, country / language inference, priority tiers.459460Type safety is enforced with `pnpm typecheck` (strict TypeScript across all packages); the web app is also461linted with the React Compiler rules.462463## Deployment and operations464465Production runs on the MacLustr cluster, orchestrated by `mld` from the M1M32 gateway:466467```bash468~/Desktop/cluster-skill/mld stage . websensor469~/Desktop/cluster-skill/mld deploy websensor --node M4M64b470```471472Post-sync hooks (fail-fast): `pnpm install --frozen-lockfile` → `createdb` + migrations → registry sync →473`next build`. Processes (PM2): `websensor-engine`, `websensor-api` (:8260, published as474https://www.websensor.io through the MacLustr Tunnel — WireGuard to the OVH gateway BHS64 with Caddy TLS),475`websensor-web` (:8261, loopback). Secrets live only in the manifest on the gateway. After a deploy that changes476alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node.477478Observability: Prometheus metrics (`/api/metrics`, engine `:8262/metrics`: checks by outcome, bytes, changes by479class, events by type, LLM calls and tokens, stage latencies, suppressed candidates, queue depth, inflight),480`/health` (connectors, throughput, 304 ratio, failing domains, slowest sensors), `/ops` (engine heartbeat, queue,481storage, LLM usage, recent errors, connector dry-run, import). Retention keeps the database and blob store bounded482without ever deleting event evidence. Details: `deploy/README.md`, `docs/ARCHITECTURE.md`.483484## Security and crawling etiquette485486- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom487 monitors — passes `assertUrlAllowed()`: private ranges, loopback, link-local, cloud metadata endpoints,488 `.maclustr.io` / `.ts.net`, single-label hosts and NAT64-embedded private IPv4 are blocked, and the HTTP489 dispatcher only connects to validated addresses (DNS-rebinding safe).490- Clear `User-Agent` (`WebSensorBot/0.2 (+https://www.websensor.io/bot; contact@spboucher.ai)`), conditional491 requests, per-host concurrency, adaptive intervals, backoff, timeouts and a per-domain circuit breaker. Official492 APIs, feeds and structured endpoints are preferred; WebSensor does not bypass authentication or anti-bot493 protections (Scrapfly is a budgeted fallback only where a source explicitly allows it).494- Webhooks: HTTPS only, public hosts only, `X-WebSensor-Signature: sha256=<HMAC>`; custom monitors: public495 URLs only, tested before activation, hard per-owner limit, never shown in public feeds.496- Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting497 on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP.498499## Source Factory and coverage (0.3, 2026-09-13)500501WebSensor no longer grows by hand-written lists alone. `config/coverage/*.yaml` declares, sector by sector, the502organizations that *should* be observed (listed companies by index, governments of every country by institutional503role, AI, cloud, cybersecurity, markets, science, healthcare, energy, transport, telecom, logistics, open source,504commerce), each with a verified domain, a country, an importance and optional hints (CIK, GitHub organization,505Hugging Face author, status page, explicit URLs). Those universes are the denominator of the **Global Observation506Coverage Score** (`breadth` = importance-weighted share of members with an active sensor, `depth` = sensors per507organization, `score = 100·(0.7·breadth + 0.3·depth)`) and the input of the **Source Factory**:508509```510seed (organization + hints)511 → deep discovery robots · sitemaps · feeds (+ official sub-domains) · navigation-classified pages512 (news · press · IR · changelog · security · pricing · legal · careers · leadership · docs)513 · status-page providers · GitHub org repositories · EDGAR · Hugging Face · OpenAPI · posture514 → validation every candidate fetched and parsed by its connector; wildcard DNS and anti-bot detected;515 duplicates removed by content (shared feed items, identical page text)516 → scoring evidence × page class × organization importance × first-party confidence × change517 frequency − fetch cost, with explainable reasons; caps per organization and per class518 → shadow sensors polled like any sensor, evidence stored, nothing published519 → evaluation reject (errors, pure noise, duplicate of an active sensor) · accept (≥ 5 checks, 24 h) · defer520 → production accepted sensors publish; `cli.ts factory export` graduates them into a registry fragment521```522523`cli.ts expand <domain>` shows what the Factory would find for one organization; `/ops` has the funnel, per-sector524progress and a candidate review table; `/coverage` and `/coverage/<sector>` show the score and every member's state525(observed · shadow · queued · not seeded).526527## Roadmap528529World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel530abstraction is in place); accounts and API keys; embeddings-based similarity; MinIO/S3 blob driver; browser531rendering for the few high-value client-rendered pages; MCP server; more depth fragments.532533## Credits534535WebSensor is designed and built by **Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai).536It runs on [MacLustr](https://www.maclustr.io), an Apple-silicon compute cluster in Québec. Source of truth:537`websensor.git` on spbgit (git.spboucher.ai). AI-generated summaries are labelled as analysis and never replace538the original evidence.539