---
WebSensor continuously watches the public web for meaningful changes, preserves the evidence, connects related
signals into real-world events and surfaces important developments before they become obvious. It is not a
website monitor: it answers **what changed, where, when, exactly what, whether it was public or silent, how
important and unusual it is, who is responsible, which other sources confirm it, whether it is part of a larger
event, how fast the story is spreading and whether WebSensor was early**.
Built by **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai) · hosted on
[MacLustr](https://www.maclustr.io), an Apple-silicon compute cluster in Québec.
## Table of contents
1. [Brand](#brand)
2. [What WebSensor does](#what-websensor-does)
3. [Architecture](#architecture)
4. [The pipeline, step by step](#the-pipeline-step-by-step)
5. [Scoring model](#scoring-model)
6. [Event taxonomy](#event-taxonomy)
7. [Source registry](#source-registry)
8. [Connectors](#connectors)
9. [Data model](#data-model)
10. [Real-time protocol](#real-time-protocol)
11. [REST API](#rest-api)
12. [Frontend](#frontend)
13. [Repository layout](#repository-layout)
14. [Development](#development)
15. [Testing](#testing)
16. [Deployment and operations](#deployment-and-operations)
17. [Security and crawling etiquette](#security-and-crawling-etiquette)
18. [Roadmap](#roadmap)
19. [Credits](#credits)
## Brand
The mark is a **pulse in the shape of a W**: the waveform is grey until the moment a change is detected, then
turns signal green and ends in the sensor reading. It is drawn once as SVG and reused everywhere:
| Asset | Path | Use |
|---|---|---|
| Mark (SVG) | `apps/web/src/app/icon.svg`, `apps/web/public/logo.svg`, `docs/brand/websensor-mark.svg` | favicon (SVG-capable browsers), nav, docs |
| Wordmark (SVG) | `docs/brand/websensor-wordmark.svg` | README, documents |
| `favicon.ico` | `apps/web/src/app/favicon.ico` | 16 / 32 / 48 px, legacy tabs |
| Apple touch icon | `apps/web/src/app/apple-icon.png` | 180 px |
| PWA icons | `apps/web/public/icons/icon-192.png`, `icon-512.png`, `icon-512-maskable.png` | manifest |
| Share image | `apps/web/src/app/opengraph-image.tsx` (+ `twitter-image.tsx`) | 1200 × 630, live counters from the API |
| Event share image | `apps/web/src/app/event/[slug]/opengraph-image.tsx` | per-event card with signal score and badges |
| React component | `apps/web/src/components/brand.tsx` (`Mark`, `PulseBackdrop`, `BRAND` colours) | app and OG images |
Colours: background `#0a0e15`, ink `#c9d1e0`, signal `#22d3a5`, line `#2d394e`. Type: Inter (UI), JetBrains Mono
(numbers, identifiers, timestamps). The product aesthetic is *Bloomberg Terminal × Linear × Vercel × Datadog ×
threat-intelligence desk*: dense information, one-pixel borders, small badges, precise spacing, dark first.
## What WebSensor does
| Question | Where the answer comes from |
|---|---|
| What changed, exactly? | Semantic diff: field-level *before → after* pairs (prices, limits, versions, dates, status words, names) with % deltas, plus unified / side-by-side / semantic diff views |
| Where and when? | Sensor URL, canonical URL, `published_at` (claimed by the source), `observed_from` (previous snapshot), `detected_at`, `processed_at`, all UTC |
| Was it public or silent? | **Silent change** = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs, shutdown, feature removed, leadership, page removed…) · no matching announcement within 12 h · importance ≥ 45 |
| How important? | Importance (8 stored components) → **WebSensor Signal Score** 0–100 with an explainable reason list |
| Is it unusual? | Source activity anomaly vs a 14-day baseline; entity anomaly vs a 30-day baseline (`+1 800 % vs baseline`) |
| Who is responsible? | Entity resolution (organizations, products, models, APIs, regulators) with parent/child relations |
| Which sources confirm it? | Novelty vs the 72 h window, cross-source confirmations, `OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED` |
| Part of a larger event? | Event clusters with propagation timeline, first-party vs external signals, velocity, `breaking / developing / confirmed / watching` state |
| Was WebSensor early? | **Lead time**: first-party detection → first external report |
| What should I pay attention to? | `/breaking`, `/pulse`, `/radar` (weak signals), watchlists, alert rules (web + signed webhooks) |
## Architecture
```
┌──────────────────────── MacLustr node M4M64b ────────────────────────┐
Internet ──TLS──▶ BHS64 │ websensor-api :8260 Fastify gateway │
(OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │
MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │
│ websensor-engine scheduler + pipeline, Prometheus :8262 │
│ websensor-factory Source Factory: seeds → discovery → shadow │
│ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │
└───────────────────────────────────────────────────────────────────────┘
```
| Role | Package | Notes |
|---|---|---|
| Shared intelligence | `packages/core` | taxonomy, SSRF policy, hashing, canonical extraction, diff engines, heuristics, semantic diff, scoring, schedule, search syntax, registry schema. `@websensor/core/client` is the browser-safe subset |
| Persistence | `packages/db` | plain-SQL forward migrations (`migrations/*.sql`, advisory-locked) + Drizzle schema |
| Evidence | `packages/store` | content-addressed zstd blob store (`sha256/ab/cd/.zst`); raw bodies, canonical forms and full diffs never live in Postgres |
| Acquisition | `packages/connectors` | connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback |
| Ingestion | `apps/engine` | scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator |
| Expansion | `apps/engine/src/factory` | **Source Factory** (0.3): coverage universes → deep discovery → scored candidates → shadow sensors → evidence-based acceptance; `websensor-factory` process |
| Coverage | `apps/api/src/coverage.ts` | **Global Observation Coverage Score** per sector (`/coverage`, `GET /api/v1/coverage`): how much of the world's high-value public web is observed |
| Gateway | `apps/api` | REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API |
| Product | `apps/web` | Next.js 16 App Router, design system, live feed, drawer, palette, desks |
Everything is TypeScript ESM in a pnpm workspace (Node ≥ 22.15). One node runs everything today; every role is
separable (several engine processes can run concurrently thanks to `FOR UPDATE SKIP LOCKED` claims).
## The pipeline, step by step
Every sensor run goes through `apps/engine/src/pipeline.ts`:
1. **Schedule** — the scheduler claims due sensors by priority (0 critical … 3 low) with `FOR UPDATE SKIP LOCKED`,
enforces global concurrency and per-host limits (higher for large public APIs such as `github.com`,
`data.sec.gov`), applies a per-domain circuit breaker (5 consecutive failures → 10 min pause) and publishes a
heartbeat to Redis (`ws:engine:status`).
2. **Fetch** — conditional GET (`ETag` / `If-None-Match`, `Last-Modified` / `If-Modified-Since`), manual redirects
validated per hop by the SSRF policy, HTTP/2 → HTTP/1.1 pinning on NGHTTP2 errors, a second attempt with a
browser identity on 403/resets, size and time limits, gzip and UTF-16 BOM handling. `304` = cheap check.
3. **Normalize** — the connector turns the observation into a comparable representation: `text` (canonical
HTML/plain text with navigation, footers, cookie banners, timestamps, tokens and counters stripped), `json`
(sorted keys, volatile paths ignored) or `list` (keyed items: feed entries, sitemap URLs, incidents, releases,
filings, records).
4. **Snapshot** — if the canonical hash changed, raw body and canonical form are stored immutably and a
`snapshots` row records hashes, HTTP metadata, extraction confidence and the source's own timestamp.
5. **Diff** — text (paired modifications), json (path operations) or list (added / removed / modified) with
connector guards: feed items are "new" only if never seen and < 14 days old, status pages losing an item is a
resolution, a > 50 % sitemap shrink is a partial response, a 404 becomes `page_removed` only after separated
confirmations.
6. **Heuristics + semantic class** — `evaluateChange` proposes an event type, magnitude, facts (prices, percents,
versions, dates) and a signal 0–1 from ~90 typed rules and record-shape priors (KEV records, CVE ids, 8-K
items, ClinicalTrials statuses…). `classifyChange` labels the change `cosmetic / navigation / timestamp /
advertisement / boilerplate` (noise) or `meaningful / pricing / policy / product / personnel` (signal) and
extracts field-level changes. A `changes` row is **always** written; noise classes never become events.
7. **Event candidate** — fingerprint idempotency (sensor + before/after canonical hashes), entity resolution
(aliases, longest first; ordinary-word aliases such as `first` or `has` only match as exact upper-case
acronyms), novelty vs the 72 h window (near-duplicates suppressed, redundant third-party reports suppressed),
source anomaly, preliminary importance (routine firehose batches damped), optional **Claude interpretation**
above `WS_LLM_MIN_IMPORTANCE` (strict JSON: type, title, summary, why it matters, observed vs inferred,
severity, confidence, announced?), final importance + confidence + impact, silent-change bar.
8. **Cluster** — attach to an open cluster (shared entity or source, textual similarity within 6 h; same-entity
cross-source stories within 2 h) or open one; update propagation timeline, first-party/external counts,
velocity, lead time and breaking state.
9. **Signal score & publish** — `computeSignalScore` with stored reasons; `events` row; entity and source daily
counters; alert evaluation (web notifications, HMAC-signed webhooks); Redis stream `ws:events` (durable, with
stream ids) + pub/sub `ws:live`.
10. **Re-schedule** — adaptive interval from tier bounds, recency of change, change frequency, 304 ratio and errors
(tier S 15 s–5 min · A 1–30 min · B 5 min–2 h · C 30 min–6 h · D 6–24 h).
Background jobs: connector health rollups (1 min), cluster state ageing (5 min), retention (30 min: raw bodies
of snapshots without an event are dropped after 21 days, canonical forms and hashes kept, event snapshots kept
forever, blobs reference-counted), run log pruning, weekly discovery.
## Scoring model
**Importance** (0–100) = 25 % intrinsic severity of the event type · 20 % source authority (tier × weight) ·
15 % entity importance · 15 % novelty · 10 % magnitude · 5 % cross-source confirmation · 5 % user impact ·
5 % unusualness. All eight components are stored on the event.
**Confidence** (0–100) blends source authenticity, extraction confidence, diff clarity, structured vs free text,
confirmations and LLM/heuristic agreement.
**Impact** (0–100) weights the event type's user impact, entity importance, magnitude and the largest numeric
delta among extracted fields (`$100 → $500` scores higher than `$100 → $105`).
**Velocity** (0–100) grows with signals per hour, unique sources and first-party signals inside a cluster.
**WebSensor Signal Score** (0–100) = 42 % importance · 14 % confidence · 14 % novelty · 12 % velocity ·
12 % impact · 6 % anomaly, then adjusted with explainable reasons: `+ first-party evidence`, `+ N independent
confirmations`, `+ silent change`, `+ tier S source`, `- third-party report`, `- low confidence`,
`- unconfirmed`, `- routine batch from a high-volume feed`, and a hard cap of 20 for noise classes. The reason
list is shown on every event ("Why this score").
**Breaking state** is not recency: `breaking` needs a strong signal, freshness (< 6 h) and either an
independent confirmation, a tier-S first-party source or signal ≥ 90; `developing` = signals accumulating;
`confirmed` = ≥ 3 sources within 24 h; `watching` otherwise; `closed` after 72 h.
**Entity rank** = importance × activity (24 h, 7 d) × average signal × confirmation ratio × unique sources ×
acceleration vs the 30-day baseline — never raw event counts.
## Event taxonomy
94 event classes grouped for filters and desks:
| Group | Examples |
|---|---|
| Security | `zero_day`, `active_exploitation`, `vulnerability`, `security_advisory`, `breach`, `supply_chain_attack`, `credential_leak`, `malware_campaign`, `ransomware`, `patch_release` |
| Reliability | `outage`, `incident`, `maintenance`, `infrastructure_change`, `dns_change`, `certificate_change` |
| Product & API | `product_launch`, `model_release`, `software_release`, `service_launch`, `service_shutdown`, `feature_removed`, `API_change`, `availability_change`, `new_region`, `documentation_change` |
| Pricing & terms | `pricing_change`, `terms_change`, `policy_change`, `crawler_policy_change` |
| Corporate | `earnings`, `guidance`, `financial_filing`, `acquisition`, `merger`, `ipo`, `funding`, `capital_raise`, `dividend`, `buyback`, `bankruptcy`, `rating_change`, `insider_transaction`, `leadership_change`, `layoffs` |
| Government & legal | `government_announcement`, `regulatory_filing`, `regulatory_action`, `legislation`, `budget`, `procurement`, `sanction`, `lawsuit`, `court_decision`, `emergency_notice`, `monetary_policy`, `economic_release` |
| Science & health | `scientific_publication`, `retraction`, `dataset_release`, `clinical_trial`, `drug_approval`, `drug_warning`, `recall`, `device_recall`, `outbreak`, `space_mission`, `standard_update` |
| Transport | `accident`, `grounding`, `safety_bulletin`, `route_change`, `production_delay` |
| Sports | `sports_transaction`, `sports_result`, `suspension`, `schedule_change` |
| Web | `announcement`, `page_created`, `page_removed`, `content_change`, `domain_registration_change` |
Each class carries an intrinsic severity and a `usuallyAnnounced` flag used by silent-change detection. The
full list lives in `packages/core/src/taxonomy.ts`.
## Source registry
The registry is declarative YAML, merged in file-name order by `apps/engine/src/seeds.ts`:
- `config/sources.yaml` — the founding 271 organizations;
- `config/sources.d/10–35` — one fragment per **site class** (open source, central banks & finance, energy /
climate / weather, telecom & internet infrastructure, retail / consumer / travel, gaming & entertainment,
universities & research, international organizations / NGOs / standards, sports, crypto, transport / aviation /
space, consumer safety / food / agriculture, housing / labour / open data, enterprise SaaS, health systems &
medtech, politics / elections / courts, EDGAR filings, package registries, web posture, OpenAPI, status JSON,
documents & data);
- `config/sources.d/40–46` — **depth** fragments (AI frontier, cloud infrastructure, cybersecurity, finance &
markets, governments incl. Canadian provinces, science & health, transport / telecom / sports / world news);
- `config/sources.d/47–55` — **breadth** fragments, wave 2 (2026-09-11): the open-source long tail (482 sensors),
SaaS status pages and changelogs (923, incl. 208 new status sensors), 424 more EDGAR issuers, 184 US federal
agencies via the Federal Register API, cities / regions / public bodies (356), world governments and regulators
in 55 countries (535), the corporate pricing / legal / careers "silent change" surface of ~250 companies (702),
sports clubs / entertainment / education (396), regional media / trade press / think tanks in 68 countries (638),
and a small cross-fragment `extend` file. Every sensor was validated live before entering the registry; the
fragment headers record what was probed and blocked (WAF, JS shells, broken TLS chains).
```yaml
sources:
- id: bank-of-canada # kebab-case, unique across all files
name: Bank of Canada
domain: bankofcanada.ca
categories: [finance, central-bank]
tier: S # S 15–60 s · A 1–5 min · B 5–30 min · C 30 min–6 h · D 6–24 h
weight: 1.4
country: CA # ISO 3166-1 alpha-2, EU, INT
language: en
products:
- { name: Valet API, type: API }
sensors:
- { name: press releases, url: "https://www.bankofcanada.ca/…/feed/", type: RSS, connector: rss, tier: A }
- { name: policy rate, url: "https://www.bankofcanada.ca/valet/observations/V39079/json?recent=3", type: REST_API, connector: jsonlist, tier: A, config: { … } }
- id: cbc # media report ABOUT others → third-party evidence
extend: true
first_party: false
```
Rules that never bend:
- **Nothing enters the registry without the validator's OK.** `apps/engine/src/validate.ts` runs every curated
sensor through its connector (fetch + normalize) and prints `OK / WARN / FAIL`; `WARN` (empty list, thin
client-rendered page) and `FAIL` (403, 404, parse error) sensors are pruned. Sites blocked by bot management
are documented in `notes:`, never faked.
- One source = one organization (the entity people search for). `extend: true` adds sensors, products, aliases,
categories, country, language or first-party flag to a source declared in an earlier file.
- Media and aggregators carry `first_party: false`; the organization's own channels are first-party and weigh
more. High-volume feeds (news wires, arXiv, NVD, package streams, sports) run with `llm: false`.
- Country and language are inferred from unambiguous TLDs when not declared; scheduling priority is derived from
tier and category.
Registry sync (`cli.ts sync`) upserts sources, sensors, organization and product entities, aliases and
`owns` relations; seed sensors removed from YAML are disabled, discovery-created ones are kept. The discovery
engine probes `robots.txt` sitemaps, `` feeds, well-known feed paths and linked status
pages, and promotes only candidates that fetch and parse. Sensors carry a lifecycle status
(`PENDING → VALIDATED → ACTIVE / DEGRADED / DISABLED`). Bulk import (JSON or YAML) is available through the
admin API with a dry-run mode. Authoring guide: `docs/registry/AUTHORING.md`.
## Connectors
All connectors implement `fetch(endpoint)` and `normalize(endpoint, observation)`; the shared pipeline does
compare, classify, score and persist (`packages/connectors/src/types.ts`).
| Connector | Sensor types | Handles |
|---|---|---|
| `http` | HTML, JSON, XML, FILE, HTTP_HEADERS | canonical text extraction (`selector`, `keepChrome`), JSON with `jsonPath` / `ignoreKeys`, HEAD |
| `rss` | RSS, ATOM | RSS 2.0, Atom, RDF, JSON Feed; GitHub / GitLab / YouTube / Blogger / arXiv API feeds |
| `sitemap` | SITEMAP | sitemap index, urlset, news sitemaps (`maxUrls`, `maxChildren`) |
| `statuspage` | STATUSPAGE | Atlassian Statuspage `api/v2/summary.json` |
| `statusjson` | STATUSPAGE | Instatus, incident.io (multi-region subpages), Status.io |
| `github` | GITHUB_RELEASE, GITHUB_REPO | releases / tags / commits Atom, advisories REST |
| `jsonlist` | REST_API, JSON | keyed records from any JSON API (KEV, NVD, Federal Register, ClinicalTrials v2, USGS, HIBP, CKAN, Hugging Face…); `{now-2h}` placeholders, `{key}` / `{field.path}` URL templates |
| `package` | REST_API | npm, PyPI, crates.io, RubyGems, NuGet, Packagist, Hex, Go proxy, Homebrew, Docker Hub version streams |
| `edgar` | REST_API | SEC EDGAR company submissions → filings, 8-K items decoded |
| `openapi` | JSON | API contract fingerprints (operations, parameters, deprecations) |
| `csv` | FILE | open-data rows (`keyColumn`, `tail`, `skipRows`) — FRED, Valet, ECB, ONS, Treasury |
| `pdf` | FILE | text of official PDFs (statements, notices, license agreements) |
| `dns` | DNS | A / AAAA / NS / MX / TXT / CAA / SOA / DMARC via public resolvers |
| `tls` | TLS | certificate identity, chain, ALPN, expiry |
| `headers` | HTTP_HEADERS | security and infrastructure response headers (HSTS, CSP, CDN) |
| `rdap` | JSON | registrar, EPP status, nameservers, expiry via the IANA bootstrap |
| `discovery` | — | candidate feeds / sitemaps / status pages, every candidate fetched and parsed |
| `scrapfly` | — | budgeted anti-bot fallback, only where the source explicitly allows it |
Per-connector documentation: `docs/connectors/*.md`.
## Data model
PostgreSQL 17; migrations in `packages/db/migrations` (`0001_init` … `0006_backfill_states`), all additive.
```
sources ─┬─ sensors ─┬─ sensor_runs
│ ├─ snapshots ─┐
│ └─ changes ───┼─ events ─┬─ event_entities ─ entities ─┬─ entity_aliases
│ │ ├─ interpretations (versioned) ├─ entity_relations
│ │ └─ event_clusters (timeline, states)
├─ source_entities └─ urls / url_history
├─ source_daily entity_daily · metrics_daily · llm_usage · connector_health · discovery_candidates
└─ (kind = custom, owner_token) watchlists · watchlist_items · alerts · notifications · bookmarks · saved_views
```
Key columns on `events`: `event_type`, `title`, `summary`, `why_it_matters`, `importance` + `importance_components`,
`confidence`, `novelty`, `signal_score`, `velocity_score`, `impact_score`, `anomaly_score`, `score_reasons`,
`change_class`, `field_changes`, `silent_change`, `evidence_label`, `first_party`, `country`, `language`,
`fingerprint` (unique), `cluster_id`, `published_at` / `observed_from` / `detected_at` / `processed_at` /
`published_to_feed_at`, `detection_latency_ms`, `processing_latency_ms`, a generated `tsvector` for full-text
search. Raw evidence is immutable: snapshots and diffs are never rewritten; re-interpretation adds a row to
`interpretations`.
## Real-time protocol
`wss://www.websensor.io/api/v1/live` (protocol 2). One Redis subscriber fans out to every client; the engine
publishes each event once with its stream id.
```
client → {"subscribe":["events:breaking","entity:org_openai","country:CA"]}
client → {"since":"1789105849242-0"} # after a reconnection: replay up to 500 missed events
client → {"ping":1}
server → {"type":"hello","protocol":2,"channels":[…]}
server → {"type":"event","sid":"1789105849242-0","channels":["events:global","events:ai"],"event":{…}}
server → {"type":"replay_done","since":"…","count":17}
server → {"type":"heartbeat","t":1789105849242} # every 25 s
```
Channels: `events:global`, `events:breaking`, `events:silent`, `events:first-party`,
`events:`,
`group:`,
`country:`, `state:`, `type:`, `entity:`, `source:`,
`watchlist:` (server-side matching of entities, sources, keywords, categories, URLs, event types, countries).
The browser client (`apps/web/src/lib/use-live.ts`) reconnects with jittered backoff, detects half-dead sockets
with a 70 s watchdog and replays on reconnection; the feed de-duplicates by event id.
## REST API
Public, unauthenticated, 600 requests / minute / IP (`x-ratelimit-*` headers), consistent errors
`{ "error": "…", "detail"?: "…" }`, cursor pagination (`nextCursor` → `cursor`), all timestamps UTC.
Documentation with examples: https://www.websensor.io/api.
| Endpoint | Purpose |
|---|---|
| `GET /api/v1/events` | filters `after before category entity source domain sensor cluster importance_min confidence_min signal_min event_type group silent_change first_party confirmed country language change_class q limit cursor order=recent\|importance\|signal`; `q` accepts the search syntax `entity:openai type:pricing_change after:7d silent:true importance:>70 country:CA` |
| `GET /api/v1/events/count` · `/events/{id\|slug}` | counts; detail with related events, cluster, change, interpretation versions, snapshots, sensor reliability, page history |
| `GET /api/v1/changes/{id}` · `/snapshots/{id}[?raw=1]` · `/snapshots/compare?a=&b=` · `/sensors/{id}/snapshots` | evidence: diff, canonical or raw body, compare any two snapshots, historical memory of a page |
| `GET /api/v1/breaking` · `/pulse` · `/radar` · `/trending` · `/explore` · `/stats` | intelligence desks and aggregates (cached 5–30 s) |
| `GET /api/v1/clusters` · `/clusters/{id\|slug}` | clusters with propagation timeline, first-party / external signals, lead time |
| `GET /api/v1/entities` · `/entities/rank` · `/entities/{id}` · `/entities/{id}/timeline` | entities, WebSensor ranking, insights (35-day heatmap, baseline, anomaly, velocity, rank) |
| `GET /api/v1/sources` · `/sources/{id}` · `/sensors/{id}` | registry, source quality score, sensor polling metadata (ETag, Last-Modified, intervals) |
| `GET /api/v1/countries[/{code}]` · `/categories/{channel}` · `/domains/{domain}/timeline` · `/urls/history?url=` | country and category desks, domain and URL history |
| `GET /api/v1/search?q=` | events, clusters, entities, sources, URLs with the parsed filters echoed |
| `GET /api/v1/health/connectors` | connector health, throughput, queue depth, engine heartbeat, failing domains |
| `GET /api/v1/feed.rss` | RSS 2.0 with the same filters as `/events` |
| Owner-scoped (`X-WebSensor-Owner`) | `/watchlists`, `/alerts` (web or HMAC-signed webhook channel), `/notifications`, `/bookmarks`, `/views`, `/monitors` (custom public URLs, SSRF-checked, per-owner limit) |
| Admin (`X-WebSensor-Admin`, only when `WS_ADMIN_TOKEN` is set) | `/admin/ops`, `/admin/failures`, sensor and source actions (`run-now`, `enable`, `disable`, patch), `/admin/sensors/test` (connector dry run), `/admin/sensors/bulk`, `/admin/sources/import?dry_run=1` (JSON or YAML) |
## Frontend
Next.js 16 (App Router, React 19, Tailwind v4). Server components by default; client components only where
interaction is needed.
| Route | What it is |
|---|---|
| `/` | hero line, **live system strip** (sources, sensors, checks/min, events 24 h, breaking, silent, status — real values, refreshed every 10 s), live feed, right rail (breaking now, trending, anomalous), clusters, silent changes, infrastructure pulse |
| `/live` | full-width feed; filters persist in the URL (`/live?category=ai&signal_min=70&silent_change=true`); saved views |
| `/breaking` | breaking now · developing · recently confirmed · watching |
| `/silent` | the silent-change desk: previous state, current state, exact diff |
| `/pulse` | what is changing on the Internet right now |
| `/radar` | weak signals, clearly labelled as indicators |
| `/explore` | trending, breaking, silent, newly detected, most active, unusual, clusters, entities, sources, categories, countries |
| `/entity/[id]` | rank, activity, 35-day heatmap, anomaly, tabs overview / live / sources / silent / timeline (1 h → all) / related / metrics |
| `/cluster/[slug]` | web-velocity propagation timeline, first-party vs external, lead time |
| `/event/[slug]` | **what changed** first: field changes, before / after diff, why it matters (analysis), evidence, history, related signals, score transparency, OG image |
| `/source/[id]` · `/sensor/[id]` | source quality, polling metadata, sensors table; sensor history ("how this page looked over time") |
| `/category/[channel]` · `/country/[slug]` | real-time desks per category and per country |
| `/entities` · `/sources` · `/search` | rankings, registry browsing, advanced search |
| `/watchlists` · `/alerts` · `/bookmarks` · `/monitors` | anonymous owner-token tools |
| `/health` · `/ops` · `/api` · `/bot` | observability, token-gated operations dashboard, API docs, crawler policy |
Interaction model: click any event → **intelligence drawer** (right on desktop, full-screen sheet on mobile);
`⌘K` / `Ctrl+K` command palette (entities, sources, events, clusters, URLs, navigation, commands); three density
modes (compact / normal / comfortable) persisted; pause live; `↑ N new events` without scroll jumps; badges
`BREAKING · DEVELOPING · SILENT · FIRST PARTY · EXTERNAL · CONFIRMED · ANOMALOUS`; skeleton loaders; real 404s;
dark and light themes; audited at 375, 390, 430, 768, 1024, 1440 and 1920 px.
## Repository layout
```
apps/
api/ Fastify gateway: routes.ts (public), routes-user.ts (owner), routes-admin.ts, intel.ts, live.ts, cache.ts
engine/ scheduler.ts, pipeline.ts, cluster.ts, entities.ts, interpret.ts (Claude), alerts.ts, retention.ts,
registry.ts, seeds.ts, discovery, metrics.ts, cli.ts, validate.ts
web/ Next.js app: src/app (routes), src/components (ui.tsx design system, live-feed, event-drawer,
command-palette, live-strip, brand.tsx…), src/lib (api client, format, feed-filters, use-live, owner)
packages/
core/ taxonomy · scoring · semantic · heuristics · diff · canonical · search · ssrf · schedule · registry-schema
db/ schema.ts + migrations/*.sql
store/ content-addressed blob store
connectors/ fetcher + 16 connector families + discovery + scrapfly
config/
sources.yaml, sources.d/*.yaml the registry
docs/
ARCHITECTURE.md, PRODUCT-BRIEF.md, registry/AUTHORING.md, connectors/*.md, brand/
scripts/ registry generators (EDGAR, Federal Register, web posture, status probe) and pruning helpers
deploy/ mld manifest example and README
tests/ fixtures (RSS before/after, Atlassian summary, KEV, sitemap…)
```
## Development
```bash
createdb websensor && cp .env.example .env
pnpm install
pnpm db:migrate # or let the engine migrate on start
npx tsx apps/engine/src/cli.ts sync # registry → database (~6 s for 3 000 sources)
npx tsx apps/engine/src/cli.ts run-due 50 # run 50 due sensors through the pipeline
pnpm dev:api # gateway :8260
pnpm dev:engine # scheduler + pipeline
NEXT_PUBLIC_API_URL=http://localhost:8260 pnpm dev:web # :8261 — open the site here in dev
pnpm test · pnpm typecheck · pnpm --filter @websensor/web build
```
Useful CLI commands (`apps/engine/src/cli.ts`): `sync`, `discover [sourceId…]`, `probe `,
`run-once `, `run-due [n]`, `relink-entities [days]` (re-resolve mentioned entities after alias-rule
changes), `refresh-clusters`, `prune-blobs`, `llm-test`. Registry validator:
`node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts [--all] [--json report.json]
[--concurrency 6] [--probe ]`; `scripts/prune-fragment.py [--keep-empty]`
removes failed sensors.
Environment (see `.env.example`): `DATABASE_URL`, `REDIS_URL`, `BLOB_STORE_DIR`, `WS_FETCH_CONCURRENCY`,
`WS_PER_HOST_CONCURRENCY` (+ `WS_PER_HOST_OVERRIDES`), `WS_MEANINGFUL_SIGNAL`, `WS_SILENT_MIN_IMPORTANCE`,
`WS_RETENTION_*`, `ANTHROPIC_API_KEY` + `WS_LLM_*` (models, daily call budget, thresholds), `SCRAPFLY_API_KEY`,
`WS_ADMIN_TOKEN`, `WS_MONITORS_PER_OWNER`, `WS_USER_AGENT`.
Dev tip: a copy of production data for UI work is `ssh 'pg_dump websensor --data-only -t events …' |
psql websensor`, then `cli.ts sync`.
## Testing
`pnpm test` runs Vitest across the workspace:
- `packages/core` — diff engines, canonicalization, heuristics, **semantic classifier and field extraction**
(source-simulator fixtures: footer-year bump, relative-time bump, pricing change with deltas, leadership change,
site redesign burst, JSON volatile paths, new feed items), new event classes (KEV → active exploitation,
zero-day, supply chain, guidance, buyback, emergency notice, outbreak, sports), search syntax round-trip,
impact / velocity / signal / breaking / anomaly scoring, title generation for URL-only and batch items;
- `packages/connectors` — RSS / Atom / JSON Feed, sitemap, Statuspage, KEV, package registries, EDGAR, OpenAPI,
CSV, DNS/TLS/RDAP normalizers on fixtures;
- `apps/engine` — alert rule matching, country / language inference, priority tiers.
Type safety is enforced with `pnpm typecheck` (strict TypeScript across all packages); the web app is also
linted with the React Compiler rules.
## Deployment and operations
Production runs on the MacLustr cluster, orchestrated by `mld` from the M1M32 gateway:
```bash
~/Desktop/cluster-skill/mld stage . websensor
~/Desktop/cluster-skill/mld deploy websensor --node M4M64b
```
Post-sync hooks (fail-fast): `pnpm install --frozen-lockfile` → `createdb` + migrations → registry sync →
`next build`. Processes (PM2): `websensor-engine`, `websensor-api` (:8260, published as
https://www.websensor.io through the MacLustr Tunnel — WireGuard to the OVH gateway BHS64 with Caddy TLS),
`websensor-web` (:8261, loopback). Secrets live only in the manifest on the gateway. After a deploy that changes
alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node.
Observability: Prometheus metrics (`/api/metrics`, engine `:8262/metrics`: checks by outcome, bytes, changes by
class, events by type, LLM calls and tokens, stage latencies, suppressed candidates, queue depth, inflight),
`/health` (connectors, throughput, 304 ratio, failing domains, slowest sensors), `/ops` (engine heartbeat, queue,
storage, LLM usage, recent errors, connector dry-run, import). Retention keeps the database and blob store bounded
without ever deleting event evidence. Details: `deploy/README.md`, `docs/ARCHITECTURE.md`.
## Security and crawling etiquette
- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom
monitors — passes `assertUrlAllowed()`: private ranges, loopback, link-local, cloud metadata endpoints,
`.maclustr.io` / `.ts.net`, single-label hosts and NAT64-embedded private IPv4 are blocked, and the HTTP
dispatcher only connects to validated addresses (DNS-rebinding safe).
- Clear `User-Agent` (`WebSensorBot/0.2 (+https://www.websensor.io/bot; contact@spboucher.ai)`), conditional
requests, per-host concurrency, adaptive intervals, backoff, timeouts and a per-domain circuit breaker. Official
APIs, feeds and structured endpoints are preferred; WebSensor does not bypass authentication or anti-bot
protections (Scrapfly is a budgeted fallback only where a source explicitly allows it).
- Webhooks: HTTPS only, public hosts only, `X-WebSensor-Signature: sha256=`; custom monitors: public
URLs only, tested before activation, hard per-owner limit, never shown in public feeds.
- Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting
on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP.
## Source Factory and coverage (0.3, 2026-09-13)
WebSensor no longer grows by hand-written lists alone. `config/coverage/*.yaml` declares, sector by sector, the
organizations that *should* be observed (listed companies by index, governments of every country by institutional
role, AI, cloud, cybersecurity, markets, science, healthcare, energy, transport, telecom, logistics, open source,
commerce), each with a verified domain, a country, an importance and optional hints (CIK, GitHub organization,
Hugging Face author, status page, explicit URLs). Those universes are the denominator of the **Global Observation
Coverage Score** (`breadth` = importance-weighted share of members with an active sensor, `depth` = sensors per
organization, `score = 100·(0.7·breadth + 0.3·depth)`) and the input of the **Source Factory**:
```
seed (organization + hints)
→ deep discovery robots · sitemaps · feeds (+ official sub-domains) · navigation-classified pages
(news · press · IR · changelog · security · pricing · legal · careers · leadership · docs)
· status-page providers · GitHub org repositories · EDGAR · Hugging Face · OpenAPI · posture
→ validation every candidate fetched and parsed by its connector; wildcard DNS and anti-bot detected;
duplicates removed by content (shared feed items, identical page text)
→ scoring evidence × page class × organization importance × first-party confidence × change
frequency − fetch cost, with explainable reasons; caps per organization and per class
→ shadow sensors polled like any sensor, evidence stored, nothing published
→ evaluation reject (errors, pure noise, duplicate of an active sensor) · accept (≥ 5 checks, 24 h) · defer
→ production accepted sensors publish; `cli.ts factory export` graduates them into a registry fragment
```
`cli.ts expand ` shows what the Factory would find for one organization; `/ops` has the funnel, per-sector
progress and a candidate review table; `/coverage` and `/coverage/` show the score and every member's state
(observed · shadow · queued · not seeded).
## Roadmap
World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel
abstraction is in place); accounts and API keys; embeddings-based similarity; MinIO/S3 blob driver; browser
rendering for the few high-value client-rendered pages; MCP server; more depth fragments.
## Credits
WebSensor is designed and built by **Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai).
It runs on [MacLustr](https://www.maclustr.io), an Apple-silicon compute cluster in Québec. Source of truth:
`websensor.git` on spbgit (git.spboucher.ai). AI-generated summaries are labelled as analysis and never replace
the original evidence.
5 669 organizations · 10 969 validated sensors · 16 connector families · 94 event classes · immutable evidence · live WebSocket feed
