SPB Git forge
spboucher.ai ↗

spb/websensor

Public
33commits 1branches 0releases
3.4 MBsize
maindefault branch
10 days agolast push
TypeScript 55.4% Python 43.2% SQL 1.2%
13.5 KB
View source Raw Blame History

# WebSensor — repository guide

WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing Web. It monitors official public sources, detects meaningful changes, classifies them semantically, links entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score, preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through a replayable WebSocket feed. Product brief: docs/PRODUCT-BRIEF.md; architecture: docs/ARCHITECTURE.md. This file is the working guide for the code (v0.3, 2026-09-13: Source Factory + Global Observation Coverage Score).

# Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)

  • packages/core — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups, silent-eligible types, change classes, cluster states, countries), ids, SSRF policy (assertUrlAllowed, safeLookup), hashing, canonical extraction, diff engines (text / json / keyed list), stage-1 heuristics (evaluateChange, describeChange, humanizeUrl), semantic diff (semantic.ts: classifyChange → cosmetic / navigation / timestamp / advertisement / boilerplate vs meaningful / pricing / policy / product / personnel; extractFieldChanges → field-level before → after pairs with % deltas), scoring (scoring.ts: importance components, confidence, impact, velocity, WebSensor Signal Score with explainable reasons, breakingState, entity rank, daily anomaly), adaptive schedule, search syntax (search.ts: entity: type: after:7d silent:true importance:>70 country:CA …), registry seed schema (registry-schema.ts, shared by engine, validator and the admin import endpoint). @websensor/core/client is the browser-safe subset.
  • packages/db — plain SQL migrations (migrations/*.sql, applied by migrate() with an advisory lock) + Drizzle schema. 0005_intelligence.sql added provenance (first_party / country / language / kind / owner_token on sources), sensor lifecycle (status, priority, validated_at), semantic class + field changes on changes, signal/velocity/impact/anomaly scores, fingerprints, score reasons on events, cluster propagation columns, entity_daily, source_daily, bookmarks, saved_views, alert channel config, notification delivery. 0006_backfill_states.sql backfills them. textArray() must be used for = any(...) / && with JS arrays.
  • packages/store — content-addressed blob store (sha256/ab/cd/<hash>.zst). Raw bodies, canonical forms and full diffs live there, never in Postgres.
  • packages/connectors — connector SDK (fetch/normalize), safe fetcher (conditional GET, per-hop SSRF checks, HTTP/2→1.1 pin, browser-UA retry, UTF-16 BOM transcoding, postJson for signed webhooks), 16 connector families (http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap) + discovery + scrapfly fallback. jsonlist.urlTemplate supports {key} (path separators kept) and {field.path}.
  • Source Factory (apps/engine/src/factory/, process apps/engine/src/factory-main.ts = PM2 websensor-factory) — the system grows by itself: coverage universes (config/coverage/*.yaml, one sector per file, members = organizations with a verified domain + hints) and config/factory/seeds/*.yaml become factory_seeds; runFactoryBatch() claims seeds (systemic first) and runs deep discovery (packages/connectors/src/discovery-deep.ts: robots → sitemaps → feeds incl. official sub-domains → navigation-classified pages (news/press/IR/changelog/security/pricing/legal/careers/ leadership/docs) → status-page providers → GitHub org repos → EDGAR (cik) → Hugging Face (hf_author) → OpenAPI → posture; every candidate is fetched and parsed, wildcard DNS detected, feeds/pages de-duplicated by content) → score.ts (evidence × page-class weight × organization importance × first-party confidence × change frequency − fetch cost, explainable reasons; caps per organization/class) → shadow sensors (sensors.status = 'SHADOW', config.factory/shadow/kind/seedId, priority 3: polled, snapshots + changes stored, never published; the pipeline and handleMissing skip event creation) → shadow.ts evaluateShadows() every 15 min (reject: ≥ 50 % errors / all changes noise / duplicate content of an active sensor; accept after ≥ 5 checks and 24 h → ACTIVE with its real priority; defer otherwise, max 120 h). Sources created by the factory carry origin = 'factory' + sector; attached ones reuse the registry source (match by registrable domain, coverageKey()). Candidates and decisions live in discovery_candidates (status candidate / shadow / accepted / rejected / duplicate, reason, score); funnel in factory_daily; heartbeat Redis ws:factory:status. cli.ts expand <domain> [--hints=json] = dry run; cli.ts factory seed|run|evaluate|stats|export|requeue; factory export writes a reviewable YAML fragment (graduating accepted sensors into config/sources.d/ hands them to the registry sync).
  • Coverage (apps/api/src/coverage.ts, GET /api/v1/coverage[/:sector], pages /coverage, /coverage/[sector], homepage rail) — Global Observation Coverage Score: per sector breadth (importance-weighted share of universe members with ≥ 1 active non-shadow sensor on their registrable domain), depth (min(1, sensors/5)), score = 100·(0.7·breadth + 0.3·depth), global = sector-weight-weighted mean. Universes are the denominator (with provenance), never the registry. Validator: apps/engine/src/coverage-validate.ts [file] [--dns]. Guide: docs/registry/COVERAGE.md. Schema shared in packages/core/src/coverage-schema.ts (+ registrableDomain()).
  • apps/engine — scheduler (priority-aware FOR UPDATE SKIP LOCKED claims, global + per-host concurrency, domain circuit breaker, heartbeat to Redis ws:engine:status), pipeline (fetch → normalize → snapshot → diff → heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias rules, novelty, optional Claude interpretation, impact / anomaly / signal scores, silent-change bar, clustering with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook delivery, Redis stream + pub/sub), registry sync (config/sources.yaml + config/sources.d/), discovery, connector health rollups, retention (retention.ts), Prometheus metrics :8262. CLI: src/cli.ts (sync, discover, probe, expand, run-once, run-due, relink-entities [days], prune-blobs, refresh-clusters, llm-test, factory …); src/validate.ts (registry validator); src/coverage-validate.ts.
  • apps/api — Fastify gateway (:8260): REST /api/v1/* (routes.ts public, routes-user.ts owner-scoped: watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, routes-admin.ts behind WS_ADMIN_TOKEN: ops, sensor & source actions, connector test, bulk import), intelligence read-models (intel.ts: breaking desk, pulse, radar, entity insights, rankings, cluster detail, country & category desks), TTL cache (cache.ts), WebSocket /api/v1/live protocol 2 (live.ts: sid on every frame, {"since"} replay from the durable stream, channels events:* group:* country:* state:* type:* entity:* source:* watchlist:*), /api/v1/feed.rss, /api/health|ready|metrics, apex→www redirect, reverse proxy to Next.
  • apps/web — Next.js 16 (:8261 loopback). Design system in components/ui.tsx (Panel, Badge, Score, Tabs, Sparkline, Heatmap, Skeleton…), prefs.tsx (density compact/normal/comfortable, pause), event-drawer.tsx (intelligence panel), command-palette.tsx (⌘K), live-strip.tsx, live-feed.tsx v2 (URL-synced filters, pause, "↑ N new events", replay), field-changes.tsx. Pages: / /live /breaking /silent /pulse /radar /explore /entity/[id] (/company redirects) /cluster/[slug] /source/[id] /sensor/[id] /category/[c] /country/[slug] /event/[slug] (+ OG image) /coverage /coverage/[sector] /bookmarks /watchlists /alerts /monitors /ops (+ Source Factory panel: funnel, sectors, candidates, accept/reject) /health /api.
  • config/sources.yaml (founding registry) + config/sources.d/*.yaml fragments merged in file-name order (extend: true adds to an earlier source). Fragments 10–35 = site classes; 40–46 (2026-09-11) = depth: AI frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport / telecom / sports / news; 47–54 (2026-09-11, wave 2) = breadth: open-source long tail, SaaS status/changelogs, EDGAR issuers, Federal Register agencies, cities/regions/public bodies, world governments & regulators, corporate pricing/legal/careers pages, sports clubs/entertainment/education, media long tail & think tanks (generators scripts/gen-*.py, scripts/prune-fragment.py). Seeds may carry country: (ISO-2, EU, INT), language: and first_party: false (media). Guide: docs/registry/AUTHORING.md. Validator: node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts <fragment> [--all] [--json report.json] — nothing enters the registry without OK.

# Rules

  • Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom monitors — goes through assertUrlAllowed(). Private ranges, metadata endpoints, .maclustr.io/.ts.net, single-label hosts and NAT64-embedded private IPv4 are blocked.
  • Raw evidence is immutable: snapshots and diffs are never rewritten. Retention may drop the RAW body of snapshots that are not referenced by any event (canonical form and hashes stay); event snapshots are kept forever. Re-interpretation creates a new interpretations row.
  • Noise never becomes an event: changes classified cosmetic / navigation / timestamp / advertisement / boilerplate are stored as changes only. Routine batches from firehose feeds are damped. Ingestion is idempotent (events.fingerprint = sensor + before/after canonical hashes).
  • Silent change = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs, shutdown, feature removed, leadership, page removed, crawler policy…) · no matching announcement within 12 h · importance ≥ WS_SILENT_MIN_IMPORTANCE · not a noise class. Never asserted as "unannounced" otherwise.
  • Never label inference as fact: evidence_label OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED; AI text is labelled analysis; score_reasons explain every signal score. First-party evidence outweighs media reports (sources.first_party, media seeds carry first_party: false).
  • Entity aliases that are ordinary words (first, has, who, make…) only match as exact upper-case acronyms in text (AMBIGUOUS_ALIASES in apps/engine/src/entities.ts); run cli.ts relink-entities 7 after changing alias rules.
  • LLM cost control: heuristics first; Claude only above WS_LLM_MIN_IMPORTANCE, daily budget, strict JSON output; llm: false on firehoses (news, arXiv, NVD, package streams). No key → the system still works.
  • Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes page_removed after WS_DELETE_CONFIRMATIONS separated checks.
  • Shadow is not production: status = 'SHADOW' sensors never create events, never count in public stats (stats.sensors, coverage) and never compete with production sensors (priority 3). Nothing the Factory finds is published before evaluateShadows() (or an operator) accepts it. Public source pages show shadow sensors with a SHADOW pill.
  • Coverage universes are lists of real organizations with verified domains (--dns), provenance and retrieval dates — never guessed domains; several files may share a sector key (universes are merged).
  • Admin API is off unless WS_ADMIN_TOKEN is set; custom monitors are limited per owner and never appear in public feeds (sources.kind = 'custom').
  • All timestamps UTC. Ids are prefixed (src_, sen_, snap_, chg_, evt_, clu_, ent_, wl_, alr_…).

# Dev

text
createdb websensor && cp .env.example .env
pnpm install
pnpm db:migrate                        # or let the engine migrate on start
npx tsx apps/engine/src/cli.ts sync    # registry → DB
npx tsx apps/engine/src/cli.ts run-due 50
pnpm dev:api · pnpm dev:engine · pnpm dev:web   # open the site through the gateway (:8260) so /api is same-origin
pnpm test · pnpm typecheck

Run CLI/engine from the repo root. A copy of production data for UI work: ssh M4M64b 'pg_dump websensor --data-only -t events -t …' | psql websensor then cli.ts sync.

# Deploy (MacLustr)

mld stage . websensor && mld deploy websensor --node M4M64b. Manifest M1M32:~/dispatch/apps/websensor.json (secrets incl. WS_ADMIN_TOKEN; gitignored copy deploy/websensor.mld.json). Processes: websensor-api (:8260, public via MacLustr Tunnel www.websensor.io on BHS64), websensor-web (:8261 loopback), websensor-engine (metrics :8262), websensor-factory (Source Factory, heartbeat ws:factory:status; post-sync hook factory seed --mode=hinted re-seeds from the coverage universes at every deploy). Postgres 17 websensor + Redis local. Blob store ~/websensor-data/blobs. After a deploy that changes alias or scoring rules: cli.ts relink-entities 7 and cli.ts refresh-clusters on the node. See deploy/README.md.