SPB Git
spboucher.ai

spb/websensor

Public

docs: detailed README (brand, pipeline, scoring, taxonomy, registry, connectors, data model, real-time protocol, API, frontend, dev, tests, deploy, security) with SVG mark and wordmark

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 11, 2026) parent ac11751 Browse files

3 changed files +528 −45

3 changed files
Unified Split
modified README.md +504 −45
@@ -1,45 +1,504 @@
1−# WebSensor 1+<p align="center">
2− 2+ <img src="docs/brand/websensor-wordmark.svg" alt="WebSensor" width="420">
3−**Detect What Changed. Know Why It Matters.** — https://www.websensor.io 3+</p>
4− 4+
5−WebSensor is a global sensor network for the changing Web. It continuously monitors official public sources 5+<h1 align="center">WebSensor</h1>
6−(newsrooms, status pages, changelogs, regulators, registries, APIs), preserves every observation, detects raw 6+
7−changes, filters noise, interprets meaningful changes into events, links them to entities, scores their 7+<p align="center"><strong>The Web is changing. We are watching.</strong><br>
8−importance and confidence, clusters related events and publishes them live over WebSocket. 8+Real-time Internet change intelligence — <a href="https://www.websensor.io">www.websensor.io</a></p>
9− 9+
10−``` 10+<p align="center">
11−Sources → Sensors/Connectors → Fetch (conditional) → Snapshot (immutable) → Diff → Heuristics 11+ <img src="docs/brand/websensor-mark.svg" alt="" width="28" valign="middle">
12− → Interpretation (Claude, budgeted) → Entities → Novelty/Clustering → Importance → Event 12+ &nbsp;3 167 organizations · 7 066 active sensors · 16 connector families · 94 event classes · immutable evidence · live WebSocket feed
13− → Redis stream → WebSocket gateway → www.websensor.io 13+</p>
14−``` 14+
15− 15+---
16−- **2 681 organizations, 4 052 curated sensors** (2026-09-08), every one fetched and parsed by the registry validator 16+
17− before being committed: the founding 271 (`config/sources.yaml`) plus 22 site-class fragments in 17+WebSensor continuously watches the public web for meaningful changes, preserves the evidence, connects related
18− `config/sources.d/` — open source, central banks & finance, energy/climate/weather, telecom & internet 18+signals into real-world events and surfaces important developments before they become obvious. It is not a
19− infrastructure, retail/consumer/travel, gaming & entertainment, universities & research, international 19+website monitor: it answers **what changed, where, when, exactly what, whether it was public or silent, how
20− organizations/NGOs/standards, sports, crypto, transport/aviation/space, consumer safety/food/agriculture, 20+important and unusual it is, who is responsible, which other sources confirm it, whether it is part of a larger
21− housing/labour/open data, enterprise SaaS, health systems & medtech, politics/elections/courts, SEC EDGAR filings 21+event, how fast the story is spreading and whether WebSensor was early**.
22− (254 issuers), package registries, web posture (robots.txt, TLS, DNS, headers, RDAP), OpenAPI contracts, status 22+
23− JSON pages, documents & data files — plus validated discovery. 23+Built by **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai) · hosted on
24−- **16 connector families**: HTTP (HTML/JSON/HEAD), RSS/Atom/JSON Feed, sitemaps, Statuspage, Instatus/incident.io/ 24+[MacLustr](https://www.maclustr.io), an Apple-silicon compute cluster in Québec.
25− Status.io (`statusjson`), GitHub, JSON APIs (`jsonlist`), package registries (npm, PyPI, crates.io, RubyGems, NuGet, 25+
26− Packagist, Hex, Go proxy, Homebrew, Docker Hub), SEC EDGAR submissions (8-K items decoded), OpenAPI/Swagger 26+## Table of contents
27− contract fingerprints, CSV/TSV time series (FRED, ECB, NOAA…), PDF text (pdf.js), DNS records, TLS certificates, 27+
28− HTTP security headers, RDAP registration — and a budgeted Scrapfly fallback. 28+1. [Brand](#brand)
29−- Authoring guide and connector cheat-sheet: `docs/registry/AUTHORING.md`; validator: 29+2. [What WebSensor does](#what-websensor-does)
30− `node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment.yaml>` (no database needed). 30+3. [Architecture](#architecture)
31−- Every event is auditable: event → change → snapshot A/B → fetch run → sensor → source, with raw evidence. 31+4. [The pipeline, step by step](#the-pipeline-step-by-step)
32−- Observed vs inferred is always labelled; silent (unannounced) changes are flagged. 32+5. [Scoring model](#scoring-model)
33− 33+6. [Event taxonomy](#event-taxonomy)
34−## Quick start 34+7. [Source registry](#source-registry)
35−``` 35+8. [Connectors](#connectors)
36−createdb websensor && cp .env.example .env && pnpm install 36+9. [Data model](#data-model)
37−npx tsx apps/engine/src/cli.ts sync # load the registry 37+10. [Real-time protocol](#real-time-protocol)
38−pnpm dev:api & pnpm dev:web & pnpm dev:engine 38+11. [REST API](#rest-api)
39−open http://localhost:8260 39+12. [Frontend](#frontend)
40−``` 40+13. [Repository layout](#repository-layout)
41− 41+14. [Development](#development)
42−Docs: `CLAUDE.md` (repo guide), `docs/ARCHITECTURE.md`, `docs/connectors/*.md`, `deploy/README.md`, 42+15. [Testing](#testing)
43−API docs at `/api` on the site. Public API: `GET /api/v1/events`, `/api/v1/entities/{id}`, 43+16. [Deployment and operations](#deployment-and-operations)
44−`/api/v1/sources/{id}`, `/api/v1/domains/{domain}/timeline`, `/api/v1/changes/{id}`, `wss://…/api/v1/live`, 44+17. [Security and crawling etiquette](#security-and-crawling-etiquette)
45−`/api/v1/feed.rss`. 45+18. [Roadmap](#roadmap)
46+19. [Credits](#credits)
47+
48+## Brand
49+
50+<p>
51+ <img src="docs/brand/websensor-mark.svg" alt="WebSensor mark" width="64">
52+ &nbsp;&nbsp;
53+ <img src="apps/web/public/icons/icon-192.png" alt="PWA icon" width="64">
54+</p>
55+
56+The mark is a **pulse in the shape of a W**: the waveform is grey until the moment a change is detected, then
57+turns signal green and ends in the sensor reading. It is drawn once as SVG and reused everywhere:
58+
59+| Asset | Path | Use |
60+|---|---|---|
61+| Mark (SVG) | `apps/web/src/app/icon.svg`, `apps/web/public/logo.svg`, `docs/brand/websensor-mark.svg` | favicon (SVG-capable browsers), nav, docs |
62+| Wordmark (SVG) | `docs/brand/websensor-wordmark.svg` | README, documents |
63+| `favicon.ico` | `apps/web/src/app/favicon.ico` | 16 / 32 / 48 px, legacy tabs |
64+| Apple touch icon | `apps/web/src/app/apple-icon.png` | 180 px |
65+| PWA icons | `apps/web/public/icons/icon-192.png`, `icon-512.png`, `icon-512-maskable.png` | manifest |
66+| Share image | `apps/web/src/app/opengraph-image.tsx` (+ `twitter-image.tsx`) | 1200 × 630, live counters from the API |
67+| Event share image | `apps/web/src/app/event/[slug]/opengraph-image.tsx` | per-event card with signal score and badges |
68+| React component | `apps/web/src/components/brand.tsx` (`Mark`, `PulseBackdrop`, `BRAND` colours) | app and OG images |
69+
70+Colours: background `#0a0e15`, ink `#c9d1e0`, signal `#22d3a5`, line `#2d394e`. Type: Inter (UI), JetBrains Mono
71+(numbers, identifiers, timestamps). The product aesthetic is *Bloomberg Terminal × Linear × Vercel × Datadog ×
72+threat-intelligence desk*: dense information, one-pixel borders, small badges, precise spacing, dark first.
73+
74+## What WebSensor does
75+
76+| Question | Where the answer comes from |
77+|---|---|
78+| What changed, exactly? | Semantic diff: field-level *before → after* pairs (prices, limits, versions, dates, status words, names) with % deltas, plus unified / side-by-side / semantic diff views |
79+| Where and when? | Sensor URL, canonical URL, `published_at` (claimed by the source), `observed_from` (previous snapshot), `detected_at`, `processed_at`, all UTC |
80+| Was it public or silent? | **Silent change** = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs, shutdown, feature removed, leadership, page removed…) · no matching announcement within 12 h · importance ≥ 45 |
81+| How important? | Importance (8 stored components) → **WebSensor Signal Score** 0–100 with an explainable reason list |
82+| Is it unusual? | Source activity anomaly vs a 14-day baseline; entity anomaly vs a 30-day baseline (`+1 800 % vs baseline`) |
83+| Who is responsible? | Entity resolution (organizations, products, models, APIs, regulators) with parent/child relations |
84+| Which sources confirm it? | Novelty vs the 72 h window, cross-source confirmations, `OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED` |
85+| Part of a larger event? | Event clusters with propagation timeline, first-party vs external signals, velocity, `breaking / developing / confirmed / watching` state |
86+| Was WebSensor early? | **Lead time**: first-party detection → first external report |
87+| What should I pay attention to? | `/breaking`, `/pulse`, `/radar` (weak signals), watchlists, alert rules (web + signed webhooks) |
88+
89+## Architecture
90+
91+```
92+ ┌──────────────────────── MacLustr node M4M64b ────────────────────────┐
93+ Internet ──TLS──▶ BHS64 │ websensor-api :8260 Fastify gateway │
94+ (OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │
95+ MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │
96+ │ websensor-engine scheduler + pipeline, Prometheus :8262 │
97+ │ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │
98+ └───────────────────────────────────────────────────────────────────────┘
99+```
100+
101+| Role | Package | Notes |
102+|---|---|---|
103+| Shared intelligence | `packages/core` | taxonomy, SSRF policy, hashing, canonical extraction, diff engines, heuristics, semantic diff, scoring, schedule, search syntax, registry schema. `@websensor/core/client` is the browser-safe subset |
104+| Persistence | `packages/db` | plain-SQL forward migrations (`migrations/*.sql`, advisory-locked) + Drizzle schema |
105+| Evidence | `packages/store` | content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres |
106+| Acquisition | `packages/connectors` | connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback |
107+| Ingestion | `apps/engine` | scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator |
108+| Gateway | `apps/api` | REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API |
109+| Product | `apps/web` | Next.js 16 App Router, design system, live feed, drawer, palette, desks |
110+
111+Everything is TypeScript ESM in a pnpm workspace (Node ≥ 22.15). One node runs everything today; every role is
112+separable (several engine processes can run concurrently thanks to `FOR UPDATE SKIP LOCKED` claims).
113+
114+## The pipeline, step by step
115+
116+Every sensor run goes through `apps/engine/src/pipeline.ts`:
117+
118+1. **Schedule** — the scheduler claims due sensors by priority (0 critical … 3 low) with `FOR UPDATE SKIP LOCKED`,
119+ enforces global concurrency and per-host limits (higher for large public APIs such as `github.com`,
120+ `data.sec.gov`), applies a per-domain circuit breaker (5 consecutive failures → 10 min pause) and publishes a
121+ heartbeat to Redis (`ws:engine:status`).
122+2. **Fetch** — conditional GET (`ETag` / `If-None-Match`, `Last-Modified` / `If-Modified-Since`), manual redirects
123+ validated per hop by the SSRF policy, HTTP/2 → HTTP/1.1 pinning on NGHTTP2 errors, a second attempt with a
124+ browser identity on 403/resets, size and time limits, gzip and UTF-16 BOM handling. `304` = cheap check.
125+3. **Normalize** — the connector turns the observation into a comparable representation: `text` (canonical
126+ HTML/plain text with navigation, footers, cookie banners, timestamps, tokens and counters stripped), `json`
127+ (sorted keys, volatile paths ignored) or `list` (keyed items: feed entries, sitemap URLs, incidents, releases,
128+ filings, records).
129+4. **Snapshot** — if the canonical hash changed, raw body and canonical form are stored immutably and a
130+ `snapshots` row records hashes, HTTP metadata, extraction confidence and the source's own timestamp.
131+5. **Diff** — text (paired modifications), json (path operations) or list (added / removed / modified) with
132+ connector guards: feed items are "new" only if never seen and < 14 days old, status pages losing an item is a
133+ resolution, a > 50 % sitemap shrink is a partial response, a 404 becomes `page_removed` only after separated
134+ confirmations.
135+6. **Heuristics + semantic class** — `evaluateChange` proposes an event type, magnitude, facts (prices, percents,
136+ versions, dates) and a signal 0–1 from ~90 typed rules and record-shape priors (KEV records, CVE ids, 8-K
137+ items, ClinicalTrials statuses…). `classifyChange` labels the change `cosmetic / navigation / timestamp /
138+ advertisement / boilerplate` (noise) or `meaningful / pricing / policy / product / personnel` (signal) and
139+ extracts field-level changes. A `changes` row is **always** written; noise classes never become events.
140+7. **Event candidate** — fingerprint idempotency (sensor + before/after canonical hashes), entity resolution
141+ (aliases, longest first; ordinary-word aliases such as `first` or `has` only match as exact upper-case
142+ acronyms), novelty vs the 72 h window (near-duplicates suppressed, redundant third-party reports suppressed),
143+ source anomaly, preliminary importance (routine firehose batches damped), optional **Claude interpretation**
144+ above `WS_LLM_MIN_IMPORTANCE` (strict JSON: type, title, summary, why it matters, observed vs inferred,
145+ severity, confidence, announced?), final importance + confidence + impact, silent-change bar.
146+8. **Cluster** — attach to an open cluster (shared entity or source, textual similarity within 6 h; same-entity
147+ cross-source stories within 2 h) or open one; update propagation timeline, first-party/external counts,
148+ velocity, lead time and breaking state.
149+9. **Signal score & publish** — `computeSignalScore` with stored reasons; `events` row; entity and source daily
150+ counters; alert evaluation (web notifications, HMAC-signed webhooks); Redis stream `ws:events` (durable, with
151+ stream ids) + pub/sub `ws:live`.
152+10. **Re-schedule** — adaptive interval from tier bounds, recency of change, change frequency, 304 ratio and errors
153+ (tier S 15 s–5 min · A 1–30 min · B 5 min–2 h · C 30 min–6 h · D 6–24 h).
154+
155+Background jobs: connector health rollups (1 min), cluster state ageing (5 min), retention (30 min: raw bodies
156+of snapshots without an event are dropped after 21 days, canonical forms and hashes kept, event snapshots kept
157+forever, blobs reference-counted), run log pruning, weekly discovery.
158+
159+## Scoring model
160+
161+**Importance** (0–100) = 25 % intrinsic severity of the event type · 20 % source authority (tier × weight) ·
162+15 % entity importance · 15 % novelty · 10 % magnitude · 5 % cross-source confirmation · 5 % user impact ·
163+5 % unusualness. All eight components are stored on the event.
164+
165+**Confidence** (0–100) blends source authenticity, extraction confidence, diff clarity, structured vs free text,
166+confirmations and LLM/heuristic agreement.
167+
168+**Impact** (0–100) weights the event type's user impact, entity importance, magnitude and the largest numeric
169+delta among extracted fields (`$100 → $500` scores higher than `$100 → $105`).
170+
171+**Velocity** (0–100) grows with signals per hour, unique sources and first-party signals inside a cluster.
172+
173+**WebSensor Signal Score** (0–100) = 42 % importance · 14 % confidence · 14 % novelty · 12 % velocity ·
174+12 % impact · 6 % anomaly, then adjusted with explainable reasons: `+ first-party evidence`, `+ N independent
175+confirmations`, `+ silent change`, `+ tier S source`, `- third-party report`, `- low confidence`,
176+`- unconfirmed`, `- routine batch from a high-volume feed`, and a hard cap of 20 for noise classes. The reason
177+list is shown on every event ("Why this score").
178+
179+**Breaking state** is not recency: `breaking` needs a strong signal, freshness (< 6 h) and either an
180+independent confirmation, a tier-S first-party source or signal ≥ 90; `developing` = signals accumulating;
181+`confirmed` = ≥ 3 sources within 24 h; `watching` otherwise; `closed` after 72 h.
182+
183+**Entity rank** = importance × activity (24 h, 7 d) × average signal × confirmation ratio × unique sources ×
184+acceleration vs the 30-day baseline — never raw event counts.
185+
186+## Event taxonomy
187+
188+94 event classes grouped for filters and desks:
189+
190+| Group | Examples |
191+|---|---|
192+| Security | `zero_day`, `active_exploitation`, `vulnerability`, `security_advisory`, `breach`, `supply_chain_attack`, `credential_leak`, `malware_campaign`, `ransomware`, `patch_release` |
193+| Reliability | `outage`, `incident`, `maintenance`, `infrastructure_change`, `dns_change`, `certificate_change` |
194+| Product & API | `product_launch`, `model_release`, `software_release`, `service_launch`, `service_shutdown`, `feature_removed`, `API_change`, `availability_change`, `new_region`, `documentation_change` |
195+| Pricing & terms | `pricing_change`, `terms_change`, `policy_change`, `crawler_policy_change` |
196+| Corporate | `earnings`, `guidance`, `financial_filing`, `acquisition`, `merger`, `ipo`, `funding`, `capital_raise`, `dividend`, `buyback`, `bankruptcy`, `rating_change`, `insider_transaction`, `leadership_change`, `layoffs` |
197+| Government & legal | `government_announcement`, `regulatory_filing`, `regulatory_action`, `legislation`, `budget`, `procurement`, `sanction`, `lawsuit`, `court_decision`, `emergency_notice`, `monetary_policy`, `economic_release` |
198+| Science & health | `scientific_publication`, `retraction`, `dataset_release`, `clinical_trial`, `drug_approval`, `drug_warning`, `recall`, `device_recall`, `outbreak`, `space_mission`, `standard_update` |
199+| Transport | `accident`, `grounding`, `safety_bulletin`, `route_change`, `production_delay` |
200+| Sports | `sports_transaction`, `sports_result`, `suspension`, `schedule_change` |
201+| Web | `announcement`, `page_created`, `page_removed`, `content_change`, `domain_registration_change` |
202+
203+Each class carries an intrinsic severity and a `usuallyAnnounced` flag used by silent-change detection. The
204+full list lives in `packages/core/src/taxonomy.ts`.
205+
206+## Source registry
207+
208+The registry is declarative YAML, merged in file-name order by `apps/engine/src/seeds.ts`:
209+
210+- `config/sources.yaml` — the founding 271 organizations;
211+- `config/sources.d/10–35` — one fragment per **site class** (open source, central banks & finance, energy /
212+ climate / weather, telecom & internet infrastructure, retail / consumer / travel, gaming & entertainment,
213+ universities & research, international organizations / NGOs / standards, sports, crypto, transport / aviation /
214+ space, consumer safety / food / agriculture, housing / labour / open data, enterprise SaaS, health systems &
215+ medtech, politics / elections / courts, EDGAR filings, package registries, web posture, OpenAPI, status JSON,
216+ documents & data);
217+- `config/sources.d/40–46` — **depth** fragments (AI frontier, cloud infrastructure, cybersecurity, finance &
218+ markets, governments incl. Canadian provinces, science & health, transport / telecom / sports / world news);
219+- `config/sources.d/47+` — the long tail (open-source projects, SaaS status pages and changelogs, S&P 500 /
220+ Nasdaq-100 / TSX 60 EDGAR issuers and 160 US federal agencies via the Federal Register API, cities and regions,
221+ world governments and regulators, corporate pricing / legal / leadership pages, sports teams, entertainment,
222+ education, regional media, trade press, think tanks).
223+
224+```yaml
225+sources:
226+ - id: bank-of-canada # kebab-case, unique across all files
227+ name: Bank of Canada
228+ domain: bankofcanada.ca
229+ categories: [finance, central-bank]
230+ tier: S # S 15–60 s · A 1–5 min · B 5–30 min · C 30 min–6 h · D 6–24 h
231+ weight: 1.4
232+ country: CA # ISO 3166-1 alpha-2, EU, INT
233+ language: en
234+ products:
235+ - { name: Valet API, type: API }
236+ sensors:
237+ - { name: press releases, url: "https://www.bankofcanada.ca/…/feed/", type: RSS, connector: rss, tier: A }
238+ - { name: policy rate, url: "https://www.bankofcanada.ca/valet/observations/V39079/json?recent=3", type: REST_API, connector: jsonlist, tier: A, config: { … } }
239+ - id: cbc # media report ABOUT others → third-party evidence
240+ extend: true
241+ first_party: false
242+```
243+
244+Rules that never bend:
245+
246+- **Nothing enters the registry without the validator's OK.** `apps/engine/src/validate.ts` runs every curated
247+ sensor through its connector (fetch + normalize) and prints `OK / WARN / FAIL`; `WARN` (empty list, thin
248+ client-rendered page) and `FAIL` (403, 404, parse error) sensors are pruned. Sites blocked by bot management
249+ are documented in `notes:`, never faked.
250+- One source = one organization (the entity people search for). `extend: true` adds sensors, products, aliases,
251+ categories, country, language or first-party flag to a source declared in an earlier file.
252+- Media and aggregators carry `first_party: false`; the organization's own channels are first-party and weigh
253+ more. High-volume feeds (news wires, arXiv, NVD, package streams, sports) run with `llm: false`.
254+- Country and language are inferred from unambiguous TLDs when not declared; scheduling priority is derived from
255+ tier and category.
256+
257+Registry sync (`cli.ts sync`) upserts sources, sensors, organization and product entities, aliases and
258+`owns` relations; seed sensors removed from YAML are disabled, discovery-created ones are kept. The discovery
259+engine probes `robots.txt` sitemaps, `<link rel="alternate">` feeds, well-known feed paths and linked status
260+pages, and promotes only candidates that fetch and parse. Sensors carry a lifecycle status
261+(`PENDING → VALIDATED → ACTIVE / DEGRADED / DISABLED`). Bulk import (JSON or YAML) is available through the
262+admin API with a dry-run mode. Authoring guide: `docs/registry/AUTHORING.md`.
263+
264+## Connectors
265+
266+All connectors implement `fetch(endpoint)` and `normalize(endpoint, observation)`; the shared pipeline does
267+compare, classify, score and persist (`packages/connectors/src/types.ts`).
268+
269+| Connector | Sensor types | Handles |
270+|---|---|---|
271+| `http` | HTML, JSON, XML, FILE, HTTP_HEADERS | canonical text extraction (`selector`, `keepChrome`), JSON with `jsonPath` / `ignoreKeys`, HEAD |
272+| `rss` | RSS, ATOM | RSS 2.0, Atom, RDF, JSON Feed; GitHub / GitLab / YouTube / Blogger / arXiv API feeds |
273+| `sitemap` | SITEMAP | sitemap index, urlset, news sitemaps (`maxUrls`, `maxChildren`) |
274+| `statuspage` | STATUSPAGE | Atlassian Statuspage `api/v2/summary.json` |
275+| `statusjson` | STATUSPAGE | Instatus, incident.io (multi-region subpages), Status.io |
276+| `github` | GITHUB_RELEASE, GITHUB_REPO | releases / tags / commits Atom, advisories REST |
277+| `jsonlist` | REST_API, JSON | keyed records from any JSON API (KEV, NVD, Federal Register, ClinicalTrials v2, USGS, HIBP, CKAN, Hugging Face…); `{now-2h}` placeholders, `{key}` / `{field.path}` URL templates |
278+| `package` | REST_API | npm, PyPI, crates.io, RubyGems, NuGet, Packagist, Hex, Go proxy, Homebrew, Docker Hub version streams |
279+| `edgar` | REST_API | SEC EDGAR company submissions → filings, 8-K items decoded |
280+| `openapi` | JSON | API contract fingerprints (operations, parameters, deprecations) |
281+| `csv` | FILE | open-data rows (`keyColumn`, `tail`, `skipRows`) — FRED, Valet, ECB, ONS, Treasury |
282+| `pdf` | FILE | text of official PDFs (statements, notices, license agreements) |
283+| `dns` | DNS | A / AAAA / NS / MX / TXT / CAA / SOA / DMARC via public resolvers |
284+| `tls` | TLS | certificate identity, chain, ALPN, expiry |
285+| `headers` | HTTP_HEADERS | security and infrastructure response headers (HSTS, CSP, CDN) |
286+| `rdap` | JSON | registrar, EPP status, nameservers, expiry via the IANA bootstrap |
287+| `discovery` | — | candidate feeds / sitemaps / status pages, every candidate fetched and parsed |
288+| `scrapfly` | — | budgeted anti-bot fallback, only where the source explicitly allows it |
289+
290+Per-connector documentation: `docs/connectors/*.md`.
291+
292+## Data model
293+
294+PostgreSQL 17; migrations in `packages/db/migrations` (`0001_init` … `0006_backfill_states`), all additive.
295+
296+```
297+sources ─┬─ sensors ─┬─ sensor_runs
298+ │ ├─ snapshots ─┐
299+ │ └─ changes ───┼─ events ─┬─ event_entities ─ entities ─┬─ entity_aliases
300+ │ │ ├─ interpretations (versioned) ├─ entity_relations
301+ │ │ └─ event_clusters (timeline, states)
302+ ├─ source_entities └─ urls / url_history
303+ ├─ source_daily entity_daily · metrics_daily · llm_usage · connector_health · discovery_candidates
304+ └─ (kind = custom, owner_token) watchlists · watchlist_items · alerts · notifications · bookmarks · saved_views
305+```
306+
307+Key columns on `events`: `event_type`, `title`, `summary`, `why_it_matters`, `importance` + `importance_components`,
308+`confidence`, `novelty`, `signal_score`, `velocity_score`, `impact_score`, `anomaly_score`, `score_reasons`,
309+`change_class`, `field_changes`, `silent_change`, `evidence_label`, `first_party`, `country`, `language`,
310+`fingerprint` (unique), `cluster_id`, `published_at` / `observed_from` / `detected_at` / `processed_at` /
311+`published_to_feed_at`, `detection_latency_ms`, `processing_latency_ms`, a generated `tsvector` for full-text
312+search. Raw evidence is immutable: snapshots and diffs are never rewritten; re-interpretation adds a row to
313+`interpretations`.
314+
315+## Real-time protocol
316+
317+`wss://www.websensor.io/api/v1/live` (protocol 2). One Redis subscriber fans out to every client; the engine
318+publishes each event once with its stream id.
319+
320+```
321+client → {"subscribe":["events:breaking","entity:org_openai","country:CA"]}
322+client → {"since":"1789105849242-0"} # after a reconnection: replay up to 500 missed events
323+client → {"ping":1}
324+server → {"type":"hello","protocol":2,"channels":[…]}
325+server → {"type":"event","sid":"1789105849242-0","channels":["events:global","events:ai"],"event":{…}}
326+server → {"type":"replay_done","since":"…","count":17}
327+server → {"type":"heartbeat","t":1789105849242} # every 25 s
328+```
329+
330+Channels: `events:global`, `events:breaking`, `events:silent`, `events:first-party`,
331+`events:<ai|cyber|finance|health|government|science|products|infrastructure|news>`,
332+`group:<security|reliability|product|commercial|corporate|government|science|transport|sports|web>`,
333+`country:<CC>`, `state:<breaking|developing>`, `type:<event_type>`, `entity:<id>`, `source:<id>`,
334+`watchlist:<id>` (server-side matching of entities, sources, keywords, categories, URLs, event types, countries).
335+The browser client (`apps/web/src/lib/use-live.ts`) reconnects with jittered backoff, detects half-dead sockets
336+with a 70 s watchdog and replays on reconnection; the feed de-duplicates by event id.
337+
338+## REST API
339+
340+Public, unauthenticated, 600 requests / minute / IP (`x-ratelimit-*` headers), consistent errors
341+`{ "error": "…", "detail"?: "…" }`, cursor pagination (`nextCursor` → `cursor`), all timestamps UTC.
342+Documentation with examples: https://www.websensor.io/api.
343+
344+| Endpoint | Purpose |
345+|---|---|
346+| `GET /api/v1/events` | filters `after before category entity source domain sensor cluster importance_min confidence_min signal_min event_type group silent_change first_party confirmed country language change_class q limit cursor order=recent\|importance\|signal`; `q` accepts the search syntax `entity:openai type:pricing_change after:7d silent:true importance:>70 country:CA` |
347+| `GET /api/v1/events/count` · `/events/{id\|slug}` | counts; detail with related events, cluster, change, interpretation versions, snapshots, sensor reliability, page history |
348+| `GET /api/v1/changes/{id}` · `/snapshots/{id}[?raw=1]` · `/snapshots/compare?a=&b=` · `/sensors/{id}/snapshots` | evidence: diff, canonical or raw body, compare any two snapshots, historical memory of a page |
349+| `GET /api/v1/breaking` · `/pulse` · `/radar` · `/trending` · `/explore` · `/stats` | intelligence desks and aggregates (cached 5–30 s) |
350+| `GET /api/v1/clusters` · `/clusters/{id\|slug}` | clusters with propagation timeline, first-party / external signals, lead time |
351+| `GET /api/v1/entities` · `/entities/rank` · `/entities/{id}` · `/entities/{id}/timeline` | entities, WebSensor ranking, insights (35-day heatmap, baseline, anomaly, velocity, rank) |
352+| `GET /api/v1/sources` · `/sources/{id}` · `/sensors/{id}` | registry, source quality score, sensor polling metadata (ETag, Last-Modified, intervals) |
353+| `GET /api/v1/countries[/{code}]` · `/categories/{channel}` · `/domains/{domain}/timeline` · `/urls/history?url=` | country and category desks, domain and URL history |
354+| `GET /api/v1/search?q=` | events, clusters, entities, sources, URLs with the parsed filters echoed |
355+| `GET /api/v1/health/connectors` | connector health, throughput, queue depth, engine heartbeat, failing domains |
356+| `GET /api/v1/feed.rss` | RSS 2.0 with the same filters as `/events` |
357+| Owner-scoped (`X-WebSensor-Owner`) | `/watchlists`, `/alerts` (web or HMAC-signed webhook channel), `/notifications`, `/bookmarks`, `/views`, `/monitors` (custom public URLs, SSRF-checked, per-owner limit) |
358+| Admin (`X-WebSensor-Admin`, only when `WS_ADMIN_TOKEN` is set) | `/admin/ops`, `/admin/failures`, sensor and source actions (`run-now`, `enable`, `disable`, patch), `/admin/sensors/test` (connector dry run), `/admin/sensors/bulk`, `/admin/sources/import?dry_run=1` (JSON or YAML) |
359+
360+## Frontend
361+
362+Next.js 16 (App Router, React 19, Tailwind v4). Server components by default; client components only where
363+interaction is needed.
364+
365+| Route | What it is |
366+|---|---|
367+| `/` | hero line, **live system strip** (sources, sensors, checks/min, events 24 h, breaking, silent, status — real values, refreshed every 10 s), live feed, right rail (breaking now, trending, anomalous), clusters, silent changes, infrastructure pulse |
368+| `/live` | full-width feed; filters persist in the URL (`/live?category=ai&signal_min=70&silent_change=true`); saved views |
369+| `/breaking` | breaking now · developing · recently confirmed · watching |
370+| `/silent` | the silent-change desk: previous state, current state, exact diff |
371+| `/pulse` | what is changing on the Internet right now |
372+| `/radar` | weak signals, clearly labelled as indicators |
373+| `/explore` | trending, breaking, silent, newly detected, most active, unusual, clusters, entities, sources, categories, countries |
374+| `/entity/[id]` | rank, activity, 35-day heatmap, anomaly, tabs overview / live / sources / silent / timeline (1 h → all) / related / metrics |
375+| `/cluster/[slug]` | web-velocity propagation timeline, first-party vs external, lead time |
376+| `/event/[slug]` | **what changed** first: field changes, before / after diff, why it matters (analysis), evidence, history, related signals, score transparency, OG image |
377+| `/source/[id]` · `/sensor/[id]` | source quality, polling metadata, sensors table; sensor history ("how this page looked over time") |
378+| `/category/[channel]` · `/country/[slug]` | real-time desks per category and per country |
379+| `/entities` · `/sources` · `/search` | rankings, registry browsing, advanced search |
380+| `/watchlists` · `/alerts` · `/bookmarks` · `/monitors` | anonymous owner-token tools |
381+| `/health` · `/ops` · `/api` · `/bot` | observability, token-gated operations dashboard, API docs, crawler policy |
382+
383+Interaction model: click any event → **intelligence drawer** (right on desktop, full-screen sheet on mobile);
384+`⌘K` / `Ctrl+K` command palette (entities, sources, events, clusters, URLs, navigation, commands); three density
385+modes (compact / normal / comfortable) persisted; pause live; `↑ N new events` without scroll jumps; badges
386+`BREAKING · DEVELOPING · SILENT · FIRST PARTY · EXTERNAL · CONFIRMED · ANOMALOUS`; skeleton loaders; real 404s;
387+dark and light themes; audited at 375, 390, 430, 768, 1024, 1440 and 1920 px.
388+
389+## Repository layout
390+
391+```
392+apps/
393+ api/ Fastify gateway: routes.ts (public), routes-user.ts (owner), routes-admin.ts, intel.ts, live.ts, cache.ts
394+ engine/ scheduler.ts, pipeline.ts, cluster.ts, entities.ts, interpret.ts (Claude), alerts.ts, retention.ts,
395+ registry.ts, seeds.ts, discovery, metrics.ts, cli.ts, validate.ts
396+ web/ Next.js app: src/app (routes), src/components (ui.tsx design system, live-feed, event-drawer,
397+ command-palette, live-strip, brand.tsx…), src/lib (api client, format, feed-filters, use-live, owner)
398+packages/
399+ core/ taxonomy · scoring · semantic · heuristics · diff · canonical · search · ssrf · schedule · registry-schema
400+ db/ schema.ts + migrations/*.sql
401+ store/ content-addressed blob store
402+ connectors/ fetcher + 16 connector families + discovery + scrapfly
403+config/
404+ sources.yaml, sources.d/*.yaml the registry
405+docs/
406+ ARCHITECTURE.md, PRODUCT-BRIEF.md, registry/AUTHORING.md, connectors/*.md, brand/
407+scripts/ registry generators (EDGAR, Federal Register, web posture, status probe) and pruning helpers
408+deploy/ mld manifest example and README
409+tests/ fixtures (RSS before/after, Atlassian summary, KEV, sitemap…)
410+```
411+
412+## Development
413+
414+```bash
415+createdb websensor && cp .env.example .env
416+pnpm install
417+pnpm db:migrate # or let the engine migrate on start
418+npx tsx apps/engine/src/cli.ts sync # registry → database (~6 s for 3 000 sources)
419+npx tsx apps/engine/src/cli.ts run-due 50 # run 50 due sensors through the pipeline
420+pnpm dev:api # gateway :8260
421+pnpm dev:engine # scheduler + pipeline
422+NEXT_PUBLIC_API_URL=http://localhost:8260 pnpm dev:web # :8261 — open the site here in dev
423+pnpm test · pnpm typecheck · pnpm --filter @websensor/web build
424+```
425+
426+Useful CLI commands (`apps/engine/src/cli.ts`): `sync`, `discover [sourceId…]`, `probe <domain>`,
427+`run-once <sensorId>`, `run-due [n]`, `relink-entities [days]` (re-resolve mentioned entities after alias-rule
428+changes), `refresh-clusters`, `prune-blobs`, `llm-test`. Registry validator:
429+`node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment.yaml> [--all] [--json report.json]
430+[--concurrency 6] [--probe <domain>]`; `scripts/prune-fragment.py <fragment> <report.json> [--keep-empty]`
431+removes failed sensors.
432+
433+Environment (see `.env.example`): `DATABASE_URL`, `REDIS_URL`, `BLOB_STORE_DIR`, `WS_FETCH_CONCURRENCY`,
434+`WS_PER_HOST_CONCURRENCY` (+ `WS_PER_HOST_OVERRIDES`), `WS_MEANINGFUL_SIGNAL`, `WS_SILENT_MIN_IMPORTANCE`,
435+`WS_RETENTION_*`, `ANTHROPIC_API_KEY` + `WS_LLM_*` (models, daily call budget, thresholds), `SCRAPFLY_API_KEY`,
436+`WS_ADMIN_TOKEN`, `WS_MONITORS_PER_OWNER`, `WS_USER_AGENT`.
437+
438+Dev tip: a copy of production data for UI work is `ssh <node> 'pg_dump websensor --data-only -t events …' |
439+psql websensor`, then `cli.ts sync`.
440+
441+## Testing
442+
443+`pnpm test` runs Vitest across the workspace:
444+
445+- `packages/core` — diff engines, canonicalization, heuristics, **semantic classifier and field extraction**
446+ (source-simulator fixtures: footer-year bump, relative-time bump, pricing change with deltas, leadership change,
447+ site redesign burst, JSON volatile paths, new feed items), new event classes (KEV → active exploitation,
448+ zero-day, supply chain, guidance, buyback, emergency notice, outbreak, sports), search syntax round-trip,
449+ impact / velocity / signal / breaking / anomaly scoring, title generation for URL-only and batch items;
450+- `packages/connectors` — RSS / Atom / JSON Feed, sitemap, Statuspage, KEV, package registries, EDGAR, OpenAPI,
451+ CSV, DNS/TLS/RDAP normalizers on fixtures;
452+- `apps/engine` — alert rule matching, country / language inference, priority tiers.
453+
454+Type safety is enforced with `pnpm typecheck` (strict TypeScript across all packages); the web app is also
455+linted with the React Compiler rules.
456+
457+## Deployment and operations
458+
459+Production runs on the MacLustr cluster, orchestrated by `mld` from the M1M32 gateway:
460+
461+```bash
462+~/Desktop/cluster-skill/mld stage . websensor
463+~/Desktop/cluster-skill/mld deploy websensor --node M4M64b
464+```
465+
466+Post-sync hooks (fail-fast): `pnpm install --frozen-lockfile` → `createdb` + migrations → registry sync →
467+`next build`. Processes (PM2): `websensor-engine`, `websensor-api` (:8260, published as
468+https://www.websensor.io through the MacLustr Tunnel — WireGuard to the OVH gateway BHS64 with Caddy TLS),
469+`websensor-web` (:8261, loopback). Secrets live only in the manifest on the gateway. After a deploy that changes
470+alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node.
471+
472+Observability: Prometheus metrics (`/api/metrics`, engine `:8262/metrics`: checks by outcome, bytes, changes by
473+class, events by type, LLM calls and tokens, stage latencies, suppressed candidates, queue depth, inflight),
474+`/health` (connectors, throughput, 304 ratio, failing domains, slowest sensors), `/ops` (engine heartbeat, queue,
475+storage, LLM usage, recent errors, connector dry-run, import). Retention keeps the database and blob store bounded
476+without ever deleting event evidence. Details: `deploy/README.md`, `docs/ARCHITECTURE.md`.
477+
478+## Security and crawling etiquette
479+
480+- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom
481+ monitors — passes `assertUrlAllowed()`: private ranges, loopback, link-local, cloud metadata endpoints,
482+ `.maclustr.io` / `.ts.net`, single-label hosts and NAT64-embedded private IPv4 are blocked, and the HTTP
483+ dispatcher only connects to validated addresses (DNS-rebinding safe).
484+- Clear `User-Agent` (`WebSensorBot/0.2 (+https://www.websensor.io/bot; contact@spboucher.ai)`), conditional
485+ requests, per-host concurrency, adaptive intervals, backoff, timeouts and a per-domain circuit breaker. Official
486+ APIs, feeds and structured endpoints are preferred; WebSensor does not bypass authentication or anti-bot
487+ protections (Scrapfly is a budgeted fallback only where a source explicitly allows it).
488+- Webhooks: HTTPS only, public hosts only, `X-WebSensor-Signature: sha256=<HMAC>`; custom monitors: public
489+ URLs only, tested before activation, hard per-owner limit, never shown in public feeds.
490+- Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting
491+ on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP.
492+
493+## Roadmap
494+
495+World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel
496+abstraction is in place); accounts and API keys; embeddings-based similarity; MinIO/S3 blob driver; browser
497+rendering for the few high-value client-rendered pages; MCP server; more depth fragments.
498+
499+## Credits
500+
501+WebSensor is designed and built by **Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai).
502+It runs on [MacLustr](https://www.maclustr.io), an Apple-silicon compute cluster in Québec. Source of truth:
503+`websensor.git` on spbgit (git.spboucher.ai). AI-generated summaries are labelled as analysis and never replace
504+the original evidence.
added docs/brand/websensor-mark.svg +13 −0
@@ -0,0 +1,13 @@
1+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" width="64" height="64">
2+ <title>WebSensor</title>
3+ <rect width="64" height="64" rx="14" fill="#0a0e15"/>
4+ <rect x="0.5" y="0.5" width="63" height="63" rx="13.5" fill="none" stroke="#22d3a5" stroke-opacity="0.18"/>
5+ <!-- baseline of the signal -->
6+ <path d="M8 46H56" stroke="#2d394e" stroke-width="2" stroke-linecap="round"/>
7+ <!-- the pulse: a W-shaped waveform, grey until the detected change, signal green after -->
8+ <path d="M9 24L19 46L31 30L43 46" fill="none" stroke="#c9d1e0" stroke-width="5" stroke-linecap="round" stroke-linejoin="round"/>
9+ <path d="M43 46L54 22" fill="none" stroke="#22d3a5" stroke-width="5" stroke-linecap="round" stroke-linejoin="round"/>
10+ <!-- sensor reading -->
11+ <circle cx="54" cy="22" r="7" fill="#22d3a5" fill-opacity="0.22"/>
12+ <circle cx="54" cy="22" r="4" fill="#22d3a5"/>
13+</svg>
added docs/brand/websensor-wordmark.svg +11 −0
@@ -0,0 +1,11 @@
1+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 420 88" width="420" height="88" role="img" aria-label="WebSensor">
2+ <rect x="0" y="12" width="64" height="64" rx="14" fill="#0a0e15"/>
3+ <rect x="0.5" y="12.5" width="63" height="63" rx="13.5" fill="none" stroke="#22d3a5" stroke-opacity="0.18"/>
4+ <path d="M8 58H56" stroke="#2d394e" stroke-width="2" stroke-linecap="round"/>
5+ <path d="M9 36L19 58L31 42L43 58" fill="none" stroke="#c9d1e0" stroke-width="5" stroke-linecap="round" stroke-linejoin="round"/>
6+ <path d="M43 58L54 34" fill="none" stroke="#22d3a5" stroke-width="5" stroke-linecap="round" stroke-linejoin="round"/>
7+ <circle cx="54" cy="34" r="7" fill="#22d3a5" fill-opacity="0.22"/>
8+ <circle cx="54" cy="34" r="4" fill="#22d3a5"/>
9+ <text x="84" y="58" font-family="Inter, -apple-system, Segoe UI, Helvetica, Arial, sans-serif" font-size="42" font-weight="700" letter-spacing="-1" fill="#e6e8ee">Web<tspan fill="#22d3a5">Sensor</tspan></text>
10+ <text x="86" y="80" font-family="JetBrains Mono, Menlo, Consolas, monospace" font-size="12" letter-spacing="3" fill="#6b7591">REAL-TIME INTERNET CHANGE INTELLIGENCE</text>
11+</svg>
12
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere