TypeScript 55.4%
Python 43.2%
SQL 1.2%
1# WebSensor — repository guide23WebSensor (www.websensor.io) is a real-time web intelligence platform: a global sensor network for the changing4Web. It monitors official public sources, detects meaningful changes, classifies them semantically, links5entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score,6preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through7a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file8is the working guide for the code (v0.3, 2026-09-13: Source Factory + Global Observation Coverage Score).910## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)11- `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups,12 silent-eligible types, change classes, cluster states, countries), ids, SSRF policy (`assertUrlAllowed`,13 `safeLookup`), hashing, canonical extraction, diff engines (text / json / keyed list), stage-1 heuristics14 (`evaluateChange`, `describeChange`, `humanizeUrl`), **semantic diff** (`semantic.ts`: `classifyChange` →15 cosmetic / navigation / timestamp / advertisement / boilerplate vs meaningful / pricing / policy / product /16 personnel; `extractFieldChanges` → field-level before → after pairs with % deltas), scoring (`scoring.ts`:17 importance components, confidence, impact, velocity, **WebSensor Signal Score** with explainable reasons,18 `breakingState`, entity rank, daily anomaly), adaptive schedule, **search syntax** (`search.ts`: `entity:` `type:`19 `after:7d` `silent:true` `importance:>70` `country:CA` …), registry seed schema (`registry-schema.ts`, shared by20 engine, validator and the admin import endpoint). `@websensor/core/client` is the browser-safe subset.21- `packages/db` — plain SQL migrations (`migrations/*.sql`, applied by `migrate()` with an advisory lock) + Drizzle22 schema. `0005_intelligence.sql` added provenance (first_party / country / language / kind / owner_token on23 sources), sensor lifecycle (`status`, `priority`, `validated_at`), semantic class + field changes on changes,24 signal/velocity/impact/anomaly scores, fingerprints, score reasons on events, cluster propagation columns,25 `entity_daily`, `source_daily`, `bookmarks`, `saved_views`, alert channel config, notification delivery.26 `0006_backfill_states.sql` backfills them. **`textArray()`** must be used for `= any(...)` / `&&` with JS arrays.27- `packages/store` — content-addressed blob store (`sha256/ab/cd/<hash>.zst`). Raw bodies, canonical forms and full28 diffs live there, never in Postgres.29- `packages/connectors` — connector SDK (`fetch/normalize`), safe fetcher (conditional GET, per-hop SSRF checks,30 HTTP/2→1.1 pin, browser-UA retry, UTF-16 BOM transcoding, `postJson` for signed webhooks), 16 connector families31 (`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) +32 `discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and33 `{field.path}`.34- **Source Factory** (`apps/engine/src/factory/`, process `apps/engine/src/factory-main.ts` = PM2 `websensor-factory`) —35 the system grows by itself: coverage universes (`config/coverage/*.yaml`, one sector per file, members = organizations36 with a verified domain + `hints`) and `config/factory/seeds/*.yaml` become `factory_seeds`; `runFactoryBatch()` claims37 seeds (systemic first) and runs **deep discovery** (`packages/connectors/src/discovery-deep.ts`: robots → sitemaps →38 feeds incl. official sub-domains → navigation-classified pages (news/press/IR/changelog/security/pricing/legal/careers/39 leadership/docs) → status-page providers → GitHub org repos → EDGAR (cik) → Hugging Face (hf_author) → OpenAPI →40 posture; every candidate is fetched and parsed, wildcard DNS detected, feeds/pages de-duplicated by content) →41 `score.ts` (evidence × page-class weight × organization importance × first-party confidence × change frequency −42 fetch cost, explainable reasons; caps per organization/class) → **shadow sensors** (`sensors.status = 'SHADOW'`,43 `config.factory/shadow/kind/seedId`, priority 3: polled, snapshots + changes stored, **never published**; the pipeline44 and `handleMissing` skip event creation) → `shadow.ts` `evaluateShadows()` every 15 min (reject: ≥ 50 % errors / all45 changes noise / duplicate content of an active sensor; accept after ≥ 5 checks and 24 h → `ACTIVE` with its real46 priority; defer otherwise, max 120 h). Sources created by the factory carry `origin = 'factory'` + `sector`; attached47 ones reuse the registry source (match by registrable domain, `coverageKey()`). Candidates and decisions live in48 `discovery_candidates` (status candidate / shadow / accepted / rejected / duplicate, `reason`, `score`); funnel in49 `factory_daily`; heartbeat Redis `ws:factory:status`. `cli.ts expand <domain> [--hints=json]` = dry run;50 `cli.ts factory seed|run|evaluate|stats|export|requeue`; `factory export` writes a reviewable YAML fragment51 (graduating accepted sensors into `config/sources.d/` hands them to the registry sync).52- **Coverage** (`apps/api/src/coverage.ts`, `GET /api/v1/coverage[/:sector]`, pages `/coverage`, `/coverage/[sector]`,53 homepage rail) — Global Observation Coverage Score: per sector `breadth` (importance-weighted share of universe54 members with ≥ 1 active non-shadow sensor on their registrable domain), `depth` (min(1, sensors/5)),55 `score = 100·(0.7·breadth + 0.3·depth)`, global = sector-weight-weighted mean. Universes are the denominator (with56 `provenance`), never the registry. Validator: `apps/engine/src/coverage-validate.ts [file] [--dns]`. Guide:57 `docs/registry/COVERAGE.md`. Schema shared in `packages/core/src/coverage-schema.ts` (+ `registrableDomain()`).58- `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency,59 **domain circuit breaker**, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff →60 heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias61 rules, novelty, optional Claude interpretation, impact / anomaly / signal scores, silent-change bar, clustering62 with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook63 delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery,64 connector health rollups, **retention** (`retention.ts`), Prometheus metrics :8262.65 CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `expand`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`,66 `refresh-clusters`, `llm-test`, `factory …`); `src/validate.ts` (registry validator); `src/coverage-validate.ts`.67- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped:68 watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts`69 behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models70 (`intel.ts`: breaking desk, pulse, radar, entity insights, rankings, cluster detail, country & category desks),71 TTL cache (`cache.ts`), WebSocket `/api/v1/live` protocol 2 (`live.ts`: `sid` on every frame, `{"since"}` replay72 from the durable stream, channels `events:* group:* country:* state:* type:* entity:* source:* watchlist:*`),73 `/api/v1/feed.rss`, `/api/health|ready|metrics`, apex→www redirect, reverse proxy to Next.74- `apps/web` — Next.js 16 (:8261 loopback). Design system in `components/ui.tsx` (Panel, Badge, Score, Tabs,75 Sparkline, Heatmap, Skeleton…), `prefs.tsx` (density compact/normal/comfortable, pause), `event-drawer.tsx`76 (intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters,77 pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar`78 `/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]`79 `/country/[slug]` `/event/[slug]` (+ OG image) `/coverage` `/coverage/[sector]` `/bookmarks` `/watchlists` `/alerts`80 `/monitors` `/ops` (+ Source Factory panel: funnel, sectors, candidates, accept/reject) `/health` `/api`.81- `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order82 (`extend: true` adds to an earlier source). Fragments 10–35 = site classes; **40–46 (2026-09-11)** = depth: AI83 frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport /84 telecom / sports / news; **47–54 (2026-09-11, wave 2)** = breadth: open-source long tail, SaaS status/changelogs,85 EDGAR issuers, Federal Register agencies, cities/regions/public bodies, world governments & regulators,86 corporate pricing/legal/careers pages, sports clubs/entertainment/education, media long tail & think tanks87 (generators `scripts/gen-*.py`, `scripts/prune-fragment.py`). Seeds may carry `country:` (ISO-2, `EU`, `INT`), `language:` and `first_party: false`88 (media). Guide: `docs/registry/AUTHORING.md`. Validator: `node node_modules/tsx/dist/cli.mjs89 apps/engine/src/validate.ts <fragment> [--all] [--json report.json]` — nothing enters the registry without OK.9091## Rules92- Every URL the engine touches — seeds, discovered candidates, redirects, Scrapfly targets, webhooks, custom93 monitors — goes through `assertUrlAllowed()`. Private ranges, metadata endpoints, `.maclustr.io`/`.ts.net`,94 single-label hosts and NAT64-embedded private IPv4 are blocked.95- Raw evidence is immutable: snapshots and diffs are never rewritten. Retention may drop the RAW body of96 snapshots that are not referenced by any event (canonical form and hashes stay); event snapshots are kept forever.97 Re-interpretation creates a new `interpretations` row.98- Noise never becomes an event: changes classified cosmetic / navigation / timestamp / advertisement /99 boilerplate are stored as changes only. Routine batches from firehose feeds are damped. Ingestion is idempotent100 (`events.fingerprint` = sensor + before/after canonical hashes).101- Silent change = first-party source · silent-eligible type (pricing, terms, policy, API, availability, docs,102 shutdown, feature removed, leadership, page removed, crawler policy…) · no matching announcement within 12 h ·103 importance ≥ `WS_SILENT_MIN_IMPORTANCE` · not a noise class. Never asserted as "unannounced" otherwise.104- Never label inference as fact: `evidence_label` OBSERVED / INFERRED / CONFIRMED / UNCONFIRMED; AI text is105 labelled analysis; `score_reasons` explain every signal score. First-party evidence outweighs media reports106 (`sources.first_party`, media seeds carry `first_party: false`).107- Entity aliases that are ordinary words (`first`, `has`, `who`, `make`…) only match as exact upper-case acronyms108 in text (`AMBIGUOUS_ALIASES` in `apps/engine/src/entities.ts`); run `cli.ts relink-entities 7` after changing109 alias rules.110- LLM cost control: heuristics first; Claude only above `WS_LLM_MIN_IMPORTANCE`, daily budget, strict JSON output;111 `llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works.112- Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes113 `page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks.114- Shadow is not production: `status = 'SHADOW'` sensors never create events, never count in public stats115 (`stats.sensors`, coverage) and never compete with production sensors (priority 3). Nothing the Factory finds is116 published before `evaluateShadows()` (or an operator) accepts it. Public source pages show shadow sensors with a SHADOW pill.117- Coverage universes are lists of real organizations with verified domains (`--dns`), provenance and retrieval dates —118 never guessed domains; several files may share a sector key (universes are merged).119- Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public120 feeds (`sources.kind = 'custom'`).121- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…).122123## Dev124```125createdb websensor && cp .env.example .env126pnpm install127pnpm db:migrate # or let the engine migrate on start128npx tsx apps/engine/src/cli.ts sync # registry → DB129npx tsx apps/engine/src/cli.ts run-due 50130pnpm dev:api · pnpm dev:engine · pnpm dev:web # open the site through the gateway (:8260) so /api is same-origin131pnpm test · pnpm typecheck132```133Run CLI/engine from the repo root. A copy of production data for UI work:134`ssh M4M64b 'pg_dump websensor --data-only -t events -t …' | psql websensor` then `cli.ts sync`.135136## Deploy (MacLustr)137`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json`138(secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260,139public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine`140(metrics :8262), `websensor-factory` (Source Factory, heartbeat `ws:factory:status`; post-sync hook `factory seed --mode=hinted`141re-seeds from the coverage universes at every deploy). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that142changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See143`deploy/README.md`.144