SPB Git
spboucher.ai

spb/websensor

Public

WebSensor 0.3 — Source Factory & Global Observation Coverage Score

- Deep discovery (robots → sitemaps → feeds incl. official sub-domains → navigation-classified pages → status providers → GitHub org → EDGAR → Hugging Face → OpenAPI → posture), every candidate validated, wildcard DNS / anti-bot detection, content de-duplication, expected change frequency (median gap)
- Factory: coverage universes → factory_seeds → scored candidates (explainable) → promotion caps → SHADOW sensors (polled, never published) → evidence-based acceptance/rejection; separate websensor-factory process with Redis heartbeat; CLI expand/factory seed|run|evaluate|stats|export|requeue; YAML export for graduation
- Migration 0007 (sources.origin/sector, factory_seeds, candidate lifecycle columns, factory_daily); pipeline keeps SHADOW status and skips events for shadow sensors
- Coverage: sector universes schema + validator (--dns), registrableDomain(), GET /api/v1/coverage[/:sector], pages /coverage and /coverage/[sector], homepage rail, stats exclude shadow sensors
- Admin API /api/v1/admin/factory* + /ops Source Factory panel (funnel, sectors, candidates, accept/reject)
- Docs: COVERAGE.md, CLAUDE.md, deploy README, ARCHITECTURE, README; manifest gains websensor-factory + factory seed hook; version 0.3.0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 13, 2026) parent 938aab1 Browse files

41 changed files +3,084 −173

41 changed files
Unified Split
modified .env.example +13 −0
@@ -42,3 +42,16 @@ WS_RETENTION_CHANGED_RAW_DAYS=60
42 WS_ADMIN_TOKEN= 42 WS_ADMIN_TOKEN=
43 # Custom URL monitors per anonymous owner (spec §45) 43 # Custom URL monitors per anonymous owner (spec §45)
44 WS_MONITORS_PER_OWNER=5 44 WS_MONITORS_PER_OWNER=5
45+
46+# Source Factory (apps/engine/src/factory-main.ts — separate `websensor-factory` process). Seeds come from
47+# config/coverage/*.yaml (members not yet monitored, or carrying hints) and config/factory/seeds/*.yaml.
48+WS_FACTORY=1
49+WS_FACTORY_CONCURRENCY=6 # organizations discovered in parallel
50+WS_FACTORY_BUDGET=90 # HTTP requests per organization
51+WS_FACTORY_MIN_SCORE=0.35 # candidate score needed to enter shadow monitoring
52+WS_FACTORY_SHADOW_MIN_CHECKS=5 # checks before a shadow sensor can be accepted
53+WS_FACTORY_SHADOW_MIN_HOURS=24
54+WS_FACTORY_SHADOW_MAX_HOURS=120
55+WS_FACTORY_TICK_SECONDS=20
56+WS_COVERAGE_DIR=./config/coverage
57+# GITHUB_TOKEN= # optional: lets the factory rank an organization's repositories via the GitHub API
modified CLAUDE.md +36 −6
@@ -5,7 +5,7 @@ Web. It monitors official public sources, detects meaningful changes, classifies
5 entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score, 5 entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score,
6 preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through 6 preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through
7 a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file 7 a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file
8−is the working guide for the code (v0.2, 2026-09-11). 8+is the working guide for the code (v0.3, 2026-09-13: Source Factory + Global Observation Coverage Score).
9 9
10 ## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15) 10 ## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)
11 - `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups, 11 - `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups,
@@ -31,6 +31,30 @@ is the working guide for the code (v0.2, 2026-09-11).
31 (`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) + 31 (`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) +
32 `discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and 32 `discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and
33 `{field.path}`. 33 `{field.path}`.
34+- **Source Factory** (`apps/engine/src/factory/`, process `apps/engine/src/factory-main.ts` = PM2 `websensor-factory`) —
35+ the system grows by itself: coverage universes (`config/coverage/*.yaml`, one sector per file, members = organizations
36+ with a verified domain + `hints`) and `config/factory/seeds/*.yaml` become `factory_seeds`; `runFactoryBatch()` claims
37+ seeds (systemic first) and runs **deep discovery** (`packages/connectors/src/discovery-deep.ts`: robots → sitemaps →
38+ feeds incl. official sub-domains → navigation-classified pages (news/press/IR/changelog/security/pricing/legal/careers/
39+ leadership/docs) → status-page providers → GitHub org repos → EDGAR (cik) → Hugging Face (hf_author) → OpenAPI →
40+ posture; every candidate is fetched and parsed, wildcard DNS detected, feeds/pages de-duplicated by content) →
41+ `score.ts` (evidence × page-class weight × organization importance × first-party confidence × change frequency −
42+ fetch cost, explainable reasons; caps per organization/class) → **shadow sensors** (`sensors.status = 'SHADOW'`,
43+ `config.factory/shadow/kind/seedId`, priority 3: polled, snapshots + changes stored, **never published**; the pipeline
44+ and `handleMissing` skip event creation) → `shadow.ts` `evaluateShadows()` every 15 min (reject: ≥ 50 % errors / all
45+ changes noise / duplicate content of an active sensor; accept after ≥ 5 checks and 24 h → `ACTIVE` with its real
46+ priority; defer otherwise, max 120 h). Sources created by the factory carry `origin = 'factory'` + `sector`; attached
47+ ones reuse the registry source (match by registrable domain, `coverageKey()`). Candidates and decisions live in
48+ `discovery_candidates` (status candidate / shadow / accepted / rejected / duplicate, `reason`, `score`); funnel in
49+ `factory_daily`; heartbeat Redis `ws:factory:status`. `cli.ts expand <domain> [--hints=json]` = dry run;
50+ `cli.ts factory seed|run|evaluate|stats|export|requeue`; `factory export` writes a reviewable YAML fragment
51+ (graduating accepted sensors into `config/sources.d/` hands them to the registry sync).
52+- **Coverage** (`apps/api/src/coverage.ts`, `GET /api/v1/coverage[/:sector]`, pages `/coverage`, `/coverage/[sector]`,
53+ homepage rail) — Global Observation Coverage Score: per sector `breadth` (importance-weighted share of universe
54+ members with ≥ 1 active non-shadow sensor on their registrable domain), `depth` (min(1, sensors/5)),
55+ `score = 100·(0.7·breadth + 0.3·depth)`, global = sector-weight-weighted mean. Universes are the denominator (with
56+ `provenance`), never the registry. Validator: `apps/engine/src/coverage-validate.ts [file] [--dns]`. Guide:
57+ `docs/registry/COVERAGE.md`. Schema shared in `packages/core/src/coverage-schema.ts` (+ `registrableDomain()`).
34 - `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency, 58 - `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency,
35 **domain circuit breaker**, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff → 59 **domain circuit breaker**, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff →
36 heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias 60 heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias
@@ -38,8 +62,8 @@ is the working guide for the code (v0.2, 2026-09-11).
38 with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook 62 with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook
39 delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery, 63 delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery,
40 connector health rollups, **retention** (`retention.ts`), Prometheus metrics :8262. 64 connector health rollups, **retention** (`retention.ts`), Prometheus metrics :8262.
41− CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`, 65+ CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `expand`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`,
42− `refresh-clusters`, `llm-test`); `src/validate.ts` (registry validator). 66+ `refresh-clusters`, `llm-test`, `factory …`); `src/validate.ts` (registry validator); `src/coverage-validate.ts`.
43 - `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped: 67 - `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped:
44 watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts` 68 watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts`
45 behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models 69 behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models
@@ -52,8 +76,8 @@ is the working guide for the code (v0.2, 2026-09-11).
52 (intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters, 76 (intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters,
53 pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar` 77 pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar`
54 `/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]` 78 `/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]`
55− `/country/[slug]` `/event/[slug]` (+ OG image) `/bookmarks` `/watchlists` `/alerts` `/monitors` `/ops` `/health` 79+ `/country/[slug]` `/event/[slug]` (+ OG image) `/coverage` `/coverage/[sector]` `/bookmarks` `/watchlists` `/alerts`
56− `/api`. 80+ `/monitors` `/ops` (+ Source Factory panel: funnel, sectors, candidates, accept/reject) `/health` `/api`.
57 - `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order 81 - `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order
58 (`extend: true` adds to an earlier source). Fragments 10–35 = site classes; **40–46 (2026-09-11)** = depth: AI 82 (`extend: true` adds to an earlier source). Fragments 10–35 = site classes; **40–46 (2026-09-11)** = depth: AI
59 frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport / 83 frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport /
@@ -87,6 +111,11 @@ is the working guide for the code (v0.2, 2026-09-11).
87 `llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works. 111 `llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works.
88 - Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes 112 - Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes
89 `page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks. 113 `page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks.
114+- Shadow is not production: `status = 'SHADOW'` sensors never create events, never count in public stats
115+ (`stats.sensors`, coverage) and never compete with production sensors (priority 3). Nothing the Factory finds is
116+ published before `evaluateShadows()` (or an operator) accepts it. Public source pages show shadow sensors with a SHADOW pill.
117+- Coverage universes are lists of real organizations with verified domains (`--dns`), provenance and retrieval dates —
118+ never guessed domains; several files may share a sector key (universes are merged).
90 - Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public 119 - Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public
91 feeds (`sources.kind = 'custom'`). 120 feeds (`sources.kind = 'custom'`).
92 - All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…). 121 - All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…).
@@ -108,6 +137,7 @@ Run CLI/engine from the repo root. A copy of production data for UI work:
108 `mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json` 137 `mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json`
109 (secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260, 138 (secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260,
110 public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine` 139 public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine`
111−(metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that 140+(metrics :8262), `websensor-factory` (Source Factory, heartbeat `ws:factory:status`; post-sync hook `factory seed --mode=hinted`
141+re-seeds from the coverage universes at every deploy). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that
112 changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See 142 changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See
113 `deploy/README.md`. 143 `deploy/README.md`.
modified README.md +31 −0
@@ -94,6 +94,7 @@ threat-intelligence desk*: dense information, one-pixel borders, small badges, p
94 (OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │ 94 (OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │
95 MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │ 95 MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │
96 │ websensor-engine scheduler + pipeline, Prometheus :8262 │ 96 │ websensor-engine scheduler + pipeline, Prometheus :8262 │
97+ │ websensor-factory Source Factory: seeds → discovery → shadow │
97 │ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │ 98 │ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │
98 └───────────────────────────────────────────────────────────────────────┘ 99 └───────────────────────────────────────────────────────────────────────┘
99 ``` 100 ```
@@ -105,6 +106,8 @@ threat-intelligence desk*: dense information, one-pixel borders, small badges, p
105 | Evidence | `packages/store` | content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres | 106 | Evidence | `packages/store` | content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres |
106 | Acquisition | `packages/connectors` | connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback | 107 | Acquisition | `packages/connectors` | connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback |
107 | Ingestion | `apps/engine` | scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator | 108 | Ingestion | `apps/engine` | scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator |
109+| Expansion | `apps/engine/src/factory` | **Source Factory** (0.3): coverage universes → deep discovery → scored candidates → shadow sensors → evidence-based acceptance; `websensor-factory` process |
110+| Coverage | `apps/api/src/coverage.ts` | **Global Observation Coverage Score** per sector (`/coverage`, `GET /api/v1/coverage`): how much of the world's high-value public web is observed |
108 | Gateway | `apps/api` | REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API | 111 | Gateway | `apps/api` | REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API |
109 | Product | `apps/web` | Next.js 16 App Router, design system, live feed, drawer, palette, desks | 112 | Product | `apps/web` | Next.js 16 App Router, design system, live feed, drawer, palette, desks |
110 113
@@ -493,6 +496,34 @@ without ever deleting event evidence. Details: `deploy/README.md`, `docs/ARCHITE
493 - Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting 496 - Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting
494 on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP. 497 on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP.
495 498
499+## Source Factory and coverage (0.3, 2026-09-13)
500+
501+WebSensor no longer grows by hand-written lists alone. `config/coverage/*.yaml` declares, sector by sector, the
502+organizations that *should* be observed (listed companies by index, governments of every country by institutional
503+role, AI, cloud, cybersecurity, markets, science, healthcare, energy, transport, telecom, logistics, open source,
504+commerce), each with a verified domain, a country, an importance and optional hints (CIK, GitHub organization,
505+Hugging Face author, status page, explicit URLs). Those universes are the denominator of the **Global Observation
506+Coverage Score** (`breadth` = importance-weighted share of members with an active sensor, `depth` = sensors per
507+organization, `score = 100·(0.7·breadth + 0.3·depth)`) and the input of the **Source Factory**:
508+
509+```
510+seed (organization + hints)
511+ → deep discovery robots · sitemaps · feeds (+ official sub-domains) · navigation-classified pages
512+ (news · press · IR · changelog · security · pricing · legal · careers · leadership · docs)
513+ · status-page providers · GitHub org repositories · EDGAR · Hugging Face · OpenAPI · posture
514+ → validation every candidate fetched and parsed by its connector; wildcard DNS and anti-bot detected;
515+ duplicates removed by content (shared feed items, identical page text)
516+ → scoring evidence × page class × organization importance × first-party confidence × change
517+ frequency − fetch cost, with explainable reasons; caps per organization and per class
518+ → shadow sensors polled like any sensor, evidence stored, nothing published
519+ → evaluation reject (errors, pure noise, duplicate of an active sensor) · accept (≥ 5 checks, 24 h) · defer
520+ → production accepted sensors publish; `cli.ts factory export` graduates them into a registry fragment
521+```
522+
523+`cli.ts expand <domain>` shows what the Factory would find for one organization; `/ops` has the funnel, per-sector
524+progress and a candidate review table; `/coverage` and `/coverage/<sector>` show the score and every member's state
525+(observed · shadow · queued · not seeded).
526+
496 ## Roadmap 527 ## Roadmap
497 528
498 World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel 529 World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel
modified apps/api/src/config.ts +1 −1
@@ -14,5 +14,5 @@ export const config = {
14 adminToken: env.WS_ADMIN_TOKEN ?? "", 14 adminToken: env.WS_ADMIN_TOKEN ?? "",
15 /** Custom URL monitors per anonymous owner (spec §45). */ 15 /** Custom URL monitors per anonymous owner (spec §45). */
16 monitorsPerOwner: Number(env.WS_MONITORS_PER_OWNER ?? 5), 16 monitorsPerOwner: Number(env.WS_MONITORS_PER_OWNER ?? 5),
17− version: "0.2.0", 17+ version: "0.3.0",
18 }; 18 };
added apps/api/src/coverage.ts +236 −0
@@ -0,0 +1,236 @@
1+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
2+import { join } from "node:path";
3+import YAML from "yaml";
4+import { coverageKey, coverageSectorSchema, type CoverageMember, type CoverageSector } from "@websensor/core";
5+import { db, sql } from "@websensor/db";
6+import { cached } from "./cache";
7+
8+/**
9+ * Global Observation Coverage Score — "how much of the world's high-value public web do we observe?"
10+ *
11+ * For every sector the universes in `config/coverage/*.yaml` are the denominator. A member is *covered* when at
12+ * least one active (non-shadow) sensor lives on its registrable domain (or on a host declared in `hints.hosts`).
13+ * breadth = Σ w·covered / Σ w (w = member importance 1–3)
14+ * depth = Σ w·min(1, sensors/5) / Σ w (five sensors per organization = full depth)
15+ * score = 100 · (0.7·breadth + 0.3·depth)
16+ * The global score is the sector-weight-weighted mean. Everything is explainable: the per-member table shows
17+ * which sensors count, what is in shadow and what the Factory is still queuing.
18+ */
19+export interface MemberCoverage {
20+ name: string;
21+ domain: string;
22+ country: string | null;
23+ importance: number;
24+ universe: string;
25+ covered: boolean;
26+ sensors: number;
27+ shadow: number;
28+ source_id: string | null;
29+ seed_status: string | null;
30+ depth: number;
31+}
32+export interface UniverseCoverage {
33+ key: string;
34+ label: string;
35+ provenance: string | null;
36+ members: number;
37+ covered: number;
38+ sensors: number;
39+ breadth: number;
40+ depth: number;
41+ score: number;
42+}
43+export interface SectorCoverage {
44+ sector: string;
45+ label: string;
46+ description: string | null;
47+ weight: number;
48+ members: number;
49+ covered: number;
50+ shadow_members: number;
51+ queued: number;
52+ sensors: number;
53+ breadth: number;
54+ depth: number;
55+ score: number;
56+ universes: UniverseCoverage[];
57+}
58+
59+const COVERAGE_DIR = process.env.WS_COVERAGE_DIR ?? "./config/coverage";
60+
61+let fileCache: { at: number; sectors: CoverageSector[]; issues: string[] } | null = null;
62+export function loadSectors(): { sectors: CoverageSector[]; issues: string[] } {
63+ if (fileCache && Date.now() - fileCache.at < 60_000) return fileCache;
64+ const byKey = new Map<string, CoverageSector>();
65+ const issues: string[] = [];
66+ if (existsSync(COVERAGE_DIR) && statSync(COVERAGE_DIR).isDirectory()) {
67+ for (const f of readdirSync(COVERAGE_DIR).filter((x) => /\.ya?ml$/.test(x) && !x.startsWith("_")).sort()) {
68+ try {
69+ const parsed = coverageSectorSchema.safeParse(YAML.parse(readFileSync(join(COVERAGE_DIR, f), "utf8")));
70+ if (!parsed.success) {
71+ issues.push(`${f}: ${parsed.error.issues[0]?.path.join(".")}: ${parsed.error.issues[0]?.message}`);
72+ continue;
73+ }
74+ const s = parsed.data;
75+ const prev = byKey.get(s.sector);
76+ if (prev) prev.universes.push(...s.universes);
77+ else byKey.set(s.sector, { ...s, universes: [...s.universes] });
78+ } catch (e) {
79+ issues.push(`${f}: ${(e as Error).message}`);
80+ }
81+ }
82+ }
83+ fileCache = { at: Date.now(), sectors: [...byKey.values()], issues };
84+ return fileCache;
85+}
86+
87+interface Observed {
88+ /** registrable domain → { sensors (active), shadow, sourceId } */
89+ byKey: Map<string, { sensors: number; shadow: number; sourceId: string | null }>;
90+ seeds: Map<string, string>;
91+}
92+
93+async function observed(): Promise<Observed> {
94+ const [hosts, srcs, seeds] = await Promise.all([
95+ db.execute<{ host: string; status: string; n: string; source_id: string }>(sql`
96+ select lower(split_part(split_part(url, '/', 3), ':', 1)) as host, case when status = 'SHADOW' then 'SHADOW' else 'ACTIVE' end as status, count(*)::text as n, min(source_id) as source_id
97+ from sensors where enabled and url like 'http%' group by 1, 2`).then((r) => r.rows),
98+ db.execute<{ id: string; domain: string }>(sql`select id, domain from sources where enabled and kind = 'registry'`).then((r) => r.rows),
99+ db.execute<{ domain: string; status: string }>(sql`select domain, status from factory_seeds`).then((r) => r.rows),
100+ ]);
101+ const byKey = new Map<string, { sensors: number; shadow: number; sourceId: string | null }>();
102+ for (const h of hosts) {
103+ if (!h.host) continue;
104+ const k = coverageKey(h.host);
105+ const cur = byKey.get(k) ?? { sensors: 0, shadow: 0, sourceId: null };
106+ if (h.status === "SHADOW") cur.shadow += Number(h.n);
107+ else cur.sensors += Number(h.n);
108+ cur.sourceId = cur.sourceId ?? h.source_id;
109+ byKey.set(k, cur);
110+ }
111+ for (const s of srcs) {
112+ const k = coverageKey(s.domain);
113+ const cur = byKey.get(k) ?? { sensors: 0, shadow: 0, sourceId: null };
114+ cur.sourceId = cur.sourceId ?? s.id;
115+ byKey.set(k, cur);
116+ }
117+ const seedMap = new Map<string, string>();
118+ for (const s of seeds) seedMap.set(coverageKey(s.domain), s.status);
119+ return { byKey, seeds: seedMap };
120+}
121+
122+function memberCoverage(m: CoverageMember, universe: string, obs: Observed): MemberCoverage {
123+ const keys = [coverageKey(m.domain), ...((m.hints?.hosts ?? []).map((h) => coverageKey(h)))];
124+ let sensors = 0;
125+ let shadow = 0;
126+ let sourceId: string | null = null;
127+ for (const k of new Set(keys)) {
128+ const o = obs.byKey.get(k);
129+ if (!o) continue;
130+ sensors += o.sensors;
131+ shadow += o.shadow;
132+ sourceId = sourceId ?? o.sourceId;
133+ }
134+ return { name: m.name, domain: m.domain, country: m.country ?? null, importance: m.importance, universe, covered: sensors > 0, sensors, shadow, source_id: sourceId, seed_status: obs.seeds.get(coverageKey(m.domain)) ?? null, depth: Math.min(1, sensors / 5) };
135+}
136+
137+function aggregate(members: MemberCoverage[]): { breadth: number; depth: number; score: number; covered: number; sensors: number } {
138+ let w = 0;
139+ let cov = 0;
140+ let dep = 0;
141+ let covered = 0;
142+ let sensors = 0;
143+ for (const m of members) {
144+ w += m.importance;
145+ if (m.covered) {
146+ cov += m.importance;
147+ covered++;
148+ }
149+ dep += m.importance * m.depth;
150+ sensors += m.sensors;
151+ }
152+ const breadth = w ? cov / w : 0;
153+ const depth = w ? dep / w : 0;
154+ return { breadth: round(breadth), depth: round(depth), score: round(100 * (0.7 * breadth + 0.3 * depth), 1), covered, sensors };
155+}
156+const round = (x: number, d = 3): number => Math.round(x * 10 ** d) / 10 ** d;
157+
158+export async function coverageSummary(): Promise<Record<string, unknown>> {
159+ return cached("coverage:summary", 5 * 60_000, async () => {
160+ const { sectors, issues } = loadSectors();
161+ const obs = await observed();
162+ const out: SectorCoverage[] = [];
163+ let wsum = 0;
164+ let ssum = 0;
165+ let members = 0;
166+ let covered = 0;
167+ let sensorsTotal = 0;
168+ for (const s of sectors) {
169+ const all: MemberCoverage[] = [];
170+ const universes: UniverseCoverage[] = [];
171+ const seenDomains = new Set<string>();
172+ for (const u of s.universes) {
173+ const ms = u.members.map((m) => memberCoverage(m, u.key, obs));
174+ const a = aggregate(ms);
175+ universes.push({ key: u.key, label: u.label, provenance: u.provenance ?? null, members: ms.length, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score });
176+ for (const m of ms) {
177+ const k = coverageKey(m.domain);
178+ if (seenDomains.has(k)) continue;
179+ seenDomains.add(k);
180+ all.push(m);
181+ }
182+ }
183+ const a = aggregate(all);
184+ const sec: SectorCoverage = { sector: s.sector, label: s.label, description: s.description ?? null, weight: s.weight, members: all.length, covered: a.covered, shadow_members: all.filter((m) => !m.covered && m.shadow > 0).length, queued: all.filter((m) => !m.covered && (m.seed_status === "queued" || m.seed_status === "discovering")).length, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, universes };
185+ out.push(sec);
186+ wsum += s.weight;
187+ ssum += s.weight * a.score;
188+ members += all.length;
189+ covered += a.covered;
190+ sensorsTotal += a.sensors;
191+ }
192+ out.sort((a, b) => b.score - a.score);
193+ const [monitored, factory] = await Promise.all([
194+ db.execute<{ sources: string; sensors: string; shadow: string; countries: string }>(sql`select (select count(*) from sources where enabled and kind = 'registry')::text as sources, (select count(*) from sensors where enabled and status <> 'SHADOW')::text as sensors, (select count(*) from sensors where enabled and status = 'SHADOW')::text as shadow, (select count(distinct country) from sources where country is not null and enabled)::text as countries`).then((r) => r.rows[0]),
195+ db.execute<{ status: string; n: string }>(sql`select status, count(*)::text as n from factory_seeds group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [x.status, Number(x.n)]))),
196+ ]);
197+ return {
198+ global_score: wsum ? round(ssum / wsum, 1) : 0,
199+ members,
200+ covered,
201+ sensors_matched: sensorsTotal,
202+ sectors: out.map(({ universes: _u, ...rest }) => rest),
203+ monitored: { sources: Number(monitored?.sources ?? 0), sensors: Number(monitored?.sensors ?? 0), shadow: Number(monitored?.shadow ?? 0), countries: Number(monitored?.countries ?? 0) },
204+ factory: factory,
205+ method: "breadth = importance-weighted share of members with ≥ 1 active sensor · depth = importance-weighted min(1, sensors/5) · score = 100·(0.7·breadth + 0.3·depth) · global = sector-weight-weighted mean",
206+ issues,
207+ generated_at: new Date().toISOString(),
208+ };
209+ });
210+}
211+
212+export async function coverageSector(sector: string): Promise<Record<string, unknown> | null> {
213+ return cached(`coverage:sector:${sector}`, 5 * 60_000, async () => {
214+ const { sectors } = loadSectors();
215+ const s = sectors.find((x) => x.sector === sector);
216+ if (!s) return null;
217+ const obs = await observed();
218+ const universes = s.universes.map((u) => {
219+ const members = u.members.map((m) => memberCoverage(m, u.key, obs)).sort((a, b) => Number(b.covered) - Number(a.covered) || b.importance - a.importance || b.sensors - a.sensors || a.name.localeCompare(b.name));
220+ const a = aggregate(members);
221+ return { key: u.key, label: u.label, description: u.description ?? null, provenance: u.provenance ?? null, members: members.length, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, items: members };
222+ });
223+ const flat = new Map<string, MemberCoverage>();
224+ for (const u of universes) for (const m of u.items) if (!flat.has(coverageKey(m.domain))) flat.set(coverageKey(m.domain), m);
225+ const a = aggregate([...flat.values()]);
226+ const byCountry = new Map<string, { members: number; covered: number }>();
227+ for (const m of flat.values()) {
228+ const c = m.country ?? "—";
229+ const cur = byCountry.get(c) ?? { members: 0, covered: 0 };
230+ cur.members++;
231+ if (m.covered) cur.covered++;
232+ byCountry.set(c, cur);
233+ }
234+ return { sector: s.sector, label: s.label, description: s.description ?? null, weight: s.weight, categories: s.categories, members: flat.size, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, shadow_members: [...flat.values()].filter((m) => !m.covered && m.shadow > 0).length, queued: [...flat.values()].filter((m) => !m.covered && (m.seed_status === "queued" || m.seed_status === "discovering")).length, universes, by_country: [...byCountry.entries()].map(([country, v]) => ({ country, ...v })).sort((x, y) => y.members - x.members), generated_at: new Date().toISOString() };
235+ });
236+}
modified apps/api/src/live.ts +9 −0
@@ -49,6 +49,15 @@ export async function engineStatus(): Promise<Record<string, unknown> | null> {
49 } 49 }
50 } 50 }
51 51
52+export async function factoryStatus(): Promise<Record<string, unknown> | null> {
53+ try {
54+ const raw = await getCmd().get("ws:factory:status");
55+ return raw ? (JSON.parse(raw) as Record<string, unknown>) : null;
56+ } catch {
57+ return null;
58+ }
59+}
60+
52 function getCmd(): Redis { 61 function getCmd(): Redis {
53 if (!cmd) cmd = new Redis(config.redisUrl, { maxRetriesPerRequest: 2, lazyConnect: false }); 62 if (!cmd) cmd = new Redis(config.redisUrl, { maxRetriesPerRequest: 2, lazyConnect: false });
54 return cmd; 63 return cmd;
modified apps/api/src/queries.ts +5 −3
@@ -127,7 +127,9 @@ export async function stats(): Promise<Record<string, unknown>> {
127 await db.execute<Record<string, unknown>>(sql` 127 await db.execute<Record<string, unknown>>(sql`
128 select 128 select
129 (select count(*) from sources where enabled and kind = 'registry') as sources, 129 (select count(*) from sources where enabled and kind = 'registry') as sources,
130− (select count(*) from sensors where enabled) as sensors, 130+ (select count(*) from sensors where enabled and status <> 'SHADOW') as sensors,
131+ (select count(*) from sensors where enabled and status = 'SHADOW') as sensors_shadow,
132+ (select count(*) from sources where enabled and kind = 'registry' and origin = 'factory') as sources_factory,
131 (select count(*) from entities) as entities, 133 (select count(*) from entities) as entities,
132 (select count(*) from snapshots) as snapshots, 134 (select count(*) from snapshots) as snapshots,
133 (select count(*) from events) as events_total, 135 (select count(*) from events) as events_total,
@@ -144,8 +146,8 @@ export async function stats(): Promise<Record<string, unknown>> {
144 (select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour, 146 (select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour,
145 (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes') as checks_last_5m, 147 (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes') as checks_last_5m,
146 (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified') as not_modified_last_5m, 148 (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified') as not_modified_last_5m,
147− (select count(*) from sensors where health = 'UP' and enabled) as sensors_up, 149+ (select count(*) from sensors where health = 'UP' and enabled and status <> 'SHADOW') as sensors_up,
148− (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled) as sensors_degraded, 150+ (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled and status <> 'SHADOW') as sensors_degraded,
149 (select count(*) from sources where enabled and first_party and kind = 'registry') as sources_first_party, 151 (select count(*) from sources where enabled and first_party and kind = 'registry') as sources_first_party,
150 (select count(distinct country) from sources where country is not null) as countries, 152 (select count(distinct country) from sources where country is not null) as countries,
151 (select max(started_at) from sensor_runs) as last_check_at, 153 (select max(started_at) from sensor_runs) as last_check_at,
modified apps/api/src/routes-admin.ts +76 −1
@@ -8,7 +8,7 @@ import { getConnector, listConnectors, NormalizeError } from "@websensor/connect
8 import { db, sql } from "@websensor/db"; 8 import { db, sql } from "@websensor/db";
9 import { cacheStats, invalidate } from "./cache"; 9 import { cacheStats, invalidate } from "./cache";
10 import { config } from "./config"; 10 import { config } from "./config";
11−import { engineStatus, liveStats } from "./live"; 11+import { engineStatus, factoryStatus, liveStats } from "./live";
12 12
13 /** 13 /**
14 * Internal operations (spec §70–72, §85). Enabled only when WS_ADMIN_TOKEN is set; every request 14 * Internal operations (spec §70–72, §85). Enabled only when WS_ADMIN_TOKEN is set; every request
@@ -193,6 +193,81 @@ export async function registerAdminRoutes(app: FastifyInstance): Promise<void> {
193 return { dry_run: false, imported_sources: valid.filter((v) => !v.extend).length, extended_sources: valid.filter((v) => v.extend).length, sensors, invalid: report.filter((r) => !r.ok).length, report }; 193 return { dry_run: false, imported_sources: valid.filter((v) => !v.extend).length, extended_sources: valid.filter((v) => v.extend).length, sensors, invalid: report.filter((r) => !r.ok).length, report };
194 }); 194 });
195 195
196+ // ---- Source Factory ---------------------------------------------------------------------------------
197+ app.get("/api/v1/admin/factory", async () => {
198+ const [seeds, cands, shadows, daily, sectors, recent, hb, ready] = await Promise.all([
199+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from factory_seeds group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [String(x.status), Number(x.n)]))),
200+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from discovery_candidates group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [String(x.status), Number(x.n)]))),
201+ db.execute<Record<string, unknown>>(sql`select count(*)::int as total, count(*) filter (where health = 'UP')::int as up, count(*) filter (where health <> 'UP')::int as unhealthy, count(*) filter (where total_runs >= 5)::int as ready, coalesce(sum(raw_changes),0)::int as raw_changes from sensors where status = 'SHADOW' and enabled`).then((r) => r.rows[0]),
202+ db.execute<Record<string, unknown>>(sql`select day::text as day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked from factory_daily order by day desc limit 30`).then((r) => r.rows),
203+ db.execute<Record<string, unknown>>(sql`select coalesce(sector, '-') as sector, count(*)::int as seeds, count(*) filter (where status = 'queued')::int as queued, count(*) filter (where status = 'discovering')::int as discovering, count(*) filter (where status in ('discovered','done'))::int as discovered, count(*) filter (where status = 'blocked')::int as blocked, count(*) filter (where status = 'error')::int as errors, coalesce(sum(shadow),0)::int as shadow, coalesce(sum(accepted),0)::int as accepted, coalesce(sum(rejected),0)::int as rejected from factory_seeds group by 1 order by seeds desc`).then((r) => r.rows),
204+ db.execute<Record<string, unknown>>(sql`select id, name, domain, sector, status, candidates, shadow, accepted, rejected, last_error, discovered_at, source_id from factory_seeds where status <> 'queued' order by updated_at desc limit 40`).then((r) => r.rows),
205+ factoryStatus(),
206+ db.execute<Record<string, unknown>>(sql`select kind_class, count(*)::int as n from discovery_candidates where status = 'shadow' group by 1 order by n desc`).then((r) => r.rows),
207+ ]);
208+ return { seeds, candidates: cands, shadow: shadows, shadow_by_kind: ready, daily, sectors, recent_seeds: recent, heartbeat: hb, generated_at: new Date().toISOString() };
209+ });
210+ app.get<{ Querystring: { status?: string; sector?: string; q?: string; limit?: string } }>("/api/v1/admin/factory/seeds", async (req) => {
211+ const lim = Math.min(1000, Number(req.query.limit ?? 200));
212+ const rows = await db.execute<Record<string, unknown>>(sql`select * from factory_seeds where 1=1 ${req.query.status ? sql`and status = ${req.query.status}` : sql``} ${req.query.sector ? sql`and sector = ${req.query.sector}` : sql``} ${req.query.q ? sql`and (name ilike ${"%" + req.query.q + "%"} or domain ilike ${"%" + req.query.q + "%"})` : sql``} order by importance desc, updated_at desc limit ${lim}`);
213+ return { items: rows.rows };
214+ });
215+ /** Bulk seeds `{ seeds: [{ name, domain, categories?, country?, tier?, importance?, aliases?, sector?, universe?, hints? }] }` — queued for the factory process. */
216+ app.post("/api/v1/admin/factory/seeds", async (req) => {
217+ const seedSchema = z.object({ id: z.string().regex(/^[a-z0-9][a-z0-9-]*$/).optional(), name: z.string().min(1), domain: z.string().min(3), categories: z.array(z.string()).default([]), country: z.string().regex(/^[A-Z]{2,3}$/).optional(), language: z.string().optional(), tier: z.enum(["S", "A", "B", "C", "D"]).default("B"), importance: z.number().int().min(1).max(3).default(2), aliases: z.array(z.string()).default([]), first_party: z.boolean().default(true), sector: z.string().optional(), universe: z.string().optional(), hints: z.record(z.string(), z.unknown()).default({}) });
218+ const body = z.object({ seeds: z.array(seedSchema).min(1).max(2000) }).parse(req.body ?? {});
219+ let n = 0;
220+ for (const s of body.seeds) {
221+ const domain = s.domain.toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "");
222+ const id = s.id ?? domain.replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "").slice(0, 60);
223+ await db.execute(sql`insert into factory_seeds (id, name, domain, categories, country, language, tier, importance, aliases, first_party, sector, universe, hints, status)
224+ values (${id}, ${s.name}, ${domain}, ${sql.raw("'{" + s.categories.map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.country ?? null}, ${s.language ?? null}, ${s.tier}, ${s.importance}, ${sql.raw("'{" + s.aliases.map((c) => '"' + c.replace(/["\\]/g, "") + '"').join(",") + "}'::text[]")}, ${s.first_party}, ${s.sector ?? null}, ${s.universe ?? null}, ${JSON.stringify(s.hints)}::jsonb, 'queued')
225+ on conflict (id) do update set name = excluded.name, hints = factory_seeds.hints || excluded.hints, status = 'queued', updated_at = now()`);
226+ n++;
227+ }
228+ invalidate("coverage");
229+ return { ok: true, queued: n };
230+ });
231+ app.post<{ Params: { id: string } }>("/api/v1/admin/factory/seeds/:id/requeue", async (req, reply) => {
232+ const r = await db.execute(sql`update factory_seeds set status = 'queued', updated_at = now() where id = ${req.params.id}`);
233+ if (!r.rowCount) return reply.status(404).send({ error: "not_found" });
234+ return { ok: true };
235+ });
236+ app.get<{ Querystring: { status?: string; seed?: string; source?: string; kind?: string; limit?: string } }>("/api/v1/admin/factory/candidates", async (req) => {
237+ const lim = Math.min(1000, Number(req.query.limit ?? 200));
238+ const rows = await db.execute<Record<string, unknown>>(sql`select c.*, s.health as shadow_health, s.total_runs as shadow_runs, s.raw_changes as shadow_changes from discovery_candidates c left join sensors s on s.id = c.shadow_sensor_id where 1=1 ${req.query.status ? sql`and c.status = ${req.query.status}` : sql``} ${req.query.seed ? sql`and c.seed_id = ${req.query.seed}` : sql``} ${req.query.source ? sql`and c.source_id = ${req.query.source}` : sql``} ${req.query.kind ? sql`and c.kind_class = ${req.query.kind}` : sql``} order by c.updated_at desc limit ${lim}`);
239+ return { items: rows.rows };
240+ });
241+ /** Manual decision on a shadow sensor: accept now (→ ACTIVE) or reject (→ DISABLED). */
242+ app.post<{ Params: { id: string; action: string } }>("/api/v1/admin/factory/shadow/:id/:action", async (req, reply) => {
243+ const s = (await db.execute<Record<string, unknown>>(sql`select id, config, status from sensors where id = ${req.params.id}`)).rows[0];
244+ if (!s) return reply.status(404).send({ error: "not_found" });
245+ const reason = `manual ${req.params.action} via admin API`;
246+ if (req.params.action === "accept") {
247+ const tp = Number(((s.config as Record<string, unknown>).targetPriority as number | undefined) ?? 2);
248+ await db.execute(sql`update sensors set status = 'ACTIVE', enabled = true, priority = ${tp}, config = (config - 'shadow') || jsonb_build_object('acceptedAt', now()::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${req.params.id}`);
249+ await db.execute(sql`update discovery_candidates set status = 'accepted', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${req.params.id}`);
250+ } else if (req.params.action === "reject") {
251+ await db.execute(sql`update sensors set status = 'DISABLED', enabled = false, config = config || jsonb_build_object('rejectedAt', now()::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${req.params.id}`);
252+ await db.execute(sql`update discovery_candidates set status = 'rejected', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${req.params.id}`);
253+ } else return reply.status(400).send({ error: "action must be accept or reject" });
254+ invalidate("");
255+ return { ok: true };
256+ });
257+ /** Promote a plain candidate (never shadowed) straight into shadow monitoring. */
258+ app.post<{ Params: { id: string } }>("/api/v1/admin/factory/candidates/:id/shadow", async (req, reply) => {
259+ const c = (await db.execute<Record<string, unknown>>(sql`select * from discovery_candidates where id = ${req.params.id}`)).rows[0];
260+ if (!c) return reply.status(404).send({ error: "not_found" });
261+ if (!c.connector || !c.type) return reply.status(400).send({ error: "candidate has no connector/type (legacy discovery row)" });
262+ const src = (await db.execute<Record<string, unknown>>(sql`select categories from sources where id = ${String(c.source_id)}`)).rows[0];
263+ const id = `${String(c.source_id)}_${String(c.name ?? c.kind_class ?? "page").toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "")}-${String(c.id).slice(-5)}`.slice(0, 80);
264+ await db.execute(sql`insert into sensors (id, source_id, name, url, type, connector, tier, config, status, priority, validated_at)
265+ values (${id}, ${String(c.source_id)}, ${String(c.name ?? c.kind_class ?? "page")}, ${String(c.url)}, ${String(c.type)}, ${String(c.connector)}, ${String(c.tier ?? "C")}, ${JSON.stringify({ ...((c.config as Record<string, unknown>) ?? {}), factory: true, shadow: true, seed: false, kind: c.kind_class, candidate: c.id, seedId: c.seed_id, shadowSince: new Date().toISOString(), targetPriority: 2, categories: src?.categories ?? [] })}::jsonb, 'SHADOW', 3, now())
266+ on conflict (id) do nothing`);
267+ await db.execute(sql`update discovery_candidates set status = 'shadow', shadow_sensor_id = ${id}, updated_at = now() where id = ${req.params.id}`);
268+ return { ok: true, sensor_id: id };
269+ });
270+
196 app.get("/api/v1/admin/failures", async () => { 271 app.get("/api/v1/admin/failures", async () => {
197 const rows = await db.execute<Record<string, unknown>>(sql`select r.started_at, r.outcome, r.http_status, r.error, r.duration_ms, s.id as sensor_id, s.name, s.source_id, s.url, s.connector from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error','rate_limited') and r.started_at >= now() - interval '24 hours' order by r.started_at desc limit 200`); 272 const rows = await db.execute<Record<string, unknown>>(sql`select r.started_at, r.outcome, r.http_status, r.error, r.duration_ms, s.id as sensor_id, s.name, s.source_id, s.url, s.connector from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error','rate_limited') and r.started_at >= now() - interval '24 hours' order by r.started_at desc limit 200`);
198 return { items: rows.rows }; 273 return { items: rows.rows };
modified apps/api/src/routes.ts +10 −1
@@ -5,6 +5,7 @@ import { db, sql, textArray } from "@websensor/db";
5 import { getBlobStore } from "@websensor/store"; 5 import { getBlobStore } from "@websensor/store";
6 import { cached } from "./cache"; 6 import { cached } from "./cache";
7 import { config } from "./config"; 7 import { config } from "./config";
8+import { coverageSector, coverageSummary } from "./coverage";
8 import { breakingDesk, categoryDesk, clusterDetail, countryDesk, countryList, entityInsights, entityRankings, pulse, radar, sourceSensors } from "./intel"; 9 import { breakingDesk, categoryDesk, clusterDetail, countryDesk, countryList, entityInsights, entityRankings, pulse, radar, sourceSensors } from "./intel";
9 import { engineStatus, liveStats } from "./live"; 10 import { engineStatus, liveStats } from "./live";
10 import { countEvents, getEvent, listEvents, relatedEvents, sensorHistory, sourceActivity, sourceQuality, stats, trending } from "./queries"; 11 import { countEvents, getEvent, listEvents, relatedEvents, sensorHistory, sourceActivity, sourceQuality, stats, trending } from "./queries";
@@ -139,7 +140,8 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {
139 const q = req.query.q; 140 const q = req.query.q;
140 const rows = await db.execute<Record<string, unknown>>(sql` 141 const rows = await db.execute<Record<string, unknown>>(sql`
141 select s.id, s.name, s.domain, s.homepage, s.description, s.categories, s.tier, s.importance_weight, s.enabled, s.notes, s.first_party, s.country, s.language, s.robots_checked_at, 142 select s.id, s.name, s.domain, s.homepage, s.description, s.categories, s.tier, s.importance_weight, s.enabled, s.notes, s.first_party, s.country, s.language, s.robots_checked_at,
142− (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count, 143+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.status <> 'SHADOW')::int as sensor_count,
144+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.status = 'SHADOW')::int as shadow_count, s.origin, s.sector,
143 (select count(*) from events e where e.source_id = s.id)::int as event_count, 145 (select count(*) from events e where e.source_id = s.id)::int as event_count,
144 (select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h, 146 (select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h,
145 (select max(detected_at) from events e where e.source_id = s.id) as last_event_at, 147 (select max(detected_at) from events e where e.source_id = s.id) as last_event_at,
@@ -247,6 +249,13 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {
247 249
248 // ---- Intelligence desks ---------------------------------------------------------------------- 250 // ---- Intelligence desks ----------------------------------------------------------------------
249 app.get("/api/v1/stats", async () => cached("stats", 5_000, stats)); 251 app.get("/api/v1/stats", async () => cached("stats", 5_000, stats));
252+ // ---- Global Observation Coverage Score ------------------------------------------------------
253+ app.get("/api/v1/coverage", async () => coverageSummary());
254+ app.get<{ Params: { sector: string } }>("/api/v1/coverage/:sector", async (req, reply) => {
255+ const r = await coverageSector(req.params.sector.toLowerCase());
256+ if (!r) return reply.status(404).send({ error: "not_found" });
257+ return r;
258+ });
250 app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => { 259 app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => {
251 const hours = Math.min(168, Number(req.query.hours ?? 24)); 260 const hours = Math.min(168, Number(req.query.hours ?? 24));
252 const limit = Math.min(50, Number(req.query.limit ?? 12)); 261 const limit = Math.min(50, Number(req.query.limit ?? 12));
modified apps/engine/src/cli.ts +49 −1
@@ -80,8 +80,56 @@ async function main(): Promise<void> {
80 console.log(JSON.stringify(out, null, 2), `\n${Date.now() - t0} ms`); 80 console.log(JSON.stringify(out, null, 2), `\n${Date.now() - t0} ms`);
81 break; 81 break;
82 } 82 }
83+ case "expand": {
84+ // Deep discovery dry-run (no DB writes): what the Source Factory would find for a domain.
85+ const { discoverOrganization } = await import("@websensor/connectors");
86+ const { scoreCandidate } = await import("./factory/score");
87+ const hintsArg = args.find((a) => a.startsWith("--hints="));
88+ const hints = hintsArg ? (JSON.parse(hintsArg.slice(8)) as Record<string, unknown>) : {};
89+ const r = await discoverOrganization(args[0]!, { hints, probePosture: args.includes("--posture") });
90+ console.log(`${r.domain} → origin ${r.origin} · home ${r.homeStatus}${r.blocked ? " BLOCKED" : ""} · ${r.requests} requests · ${r.durationMs} ms`);
91+ for (const n of r.notes) console.log(` note: ${n}`);
92+ for (const c of r.candidates) {
93+ const s = scoreCandidate(c, { importance: Number(args.find((a) => a.startsWith("--importance="))?.slice(13) ?? 2) });
94+ console.log(`${s.score.toFixed(2)} ${c.kind.padEnd(10)} ${c.connector.padEnd(10)} ${c.suggestedTier} ${String(c.itemCount ?? "").padStart(4)} ${c.url} (${c.evidence})`);
95+ }
96+ if (r.rejected.length) console.log(`rejected: ${r.rejected.map((x) => `${x.url} [${x.reason}]`).join("\n ")}`);
97+ break;
98+ }
99+ case "factory": {
100+ const f = await import("./factory");
101+ const sub = args[0];
102+ const flagVal = (n: string): string | undefined => args.find((a) => a.startsWith(`--${n}=`))?.slice(n.length + 3);
103+ if (sub === "seed") {
104+ const files = await f.seedFromFiles();
105+ const mode = (flagVal("mode") ?? "hinted") as "uncovered" | "hinted" | "all";
106+ const cov = await f.seedFromCoverage({ mode, sectors: flagVal("sector")?.split(","), requeue: args.includes("--requeue") });
107+ console.log(JSON.stringify({ files, coverage: cov }, null, 2));
108+ } else if (sub === "run") {
109+ const n = Number(flagVal("n") ?? 6);
110+ const ids = args.slice(1).filter((a) => !a.startsWith("--"));
111+ const out = await f.runFactoryBatch(n, ids.length ? { seedIds: ids } : {});
112+ for (const o of out) console.log(`${o.status.padEnd(10)} ${o.seedId.padEnd(40)} source=${o.sourceId ?? "-"} candidates=${o.candidates} shadow=${o.shadow} requests=${o.requests} ${o.durationMs} ms${o.notes.length ? " · " + o.notes.join(" · ") : ""}`);
113+ } else if (sub === "evaluate") {
114+ const r = await f.evaluateShadows({ force: args.includes("--force") });
115+ for (const d of r.decisions.filter((x) => x.decision !== "deferred" || args.includes("--verbose"))) console.log(`${d.decision.padEnd(9)} ${d.sensorId.padEnd(60)} ${d.reason}`);
116+ console.log(JSON.stringify({ accepted: r.accepted, rejected: r.rejected, deferred: r.deferred }));
117+ } else if (sub === "stats") {
118+ console.log(JSON.stringify(await f.factoryStats(), null, 2));
119+ } else if (sub === "export") {
120+ process.stdout.write(await f.exportFactoryFragment({ sector: flagVal("sector"), sinceDays: flagVal("since") ? Number(flagVal("since")) : undefined, includeShadow: args.includes("--include-shadow") }));
121+ } else if (sub === "requeue") {
122+ const ids = args.slice(1).filter((a) => !a.startsWith("--"));
123+ const r = await db.execute(sql`update factory_seeds set status = 'queued', updated_at = now() where ${ids.length ? sql`id = any(${sql.raw("array[" + ids.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]")})` : sql`status in ('error','blocked')`}`);
124+ console.log(`requeued ${r.rowCount ?? 0}`);
125+ } else {
126+ console.error("usage: cli.ts factory seed [--mode=uncovered|hinted|all] [--sector=a,b] [--requeue] | run [seedId…] [--n=6] | evaluate [--force] [--verbose] | stats | export [--sector=x] [--since=days] [--include-shadow] | requeue [seedId…]");
127+ process.exitCode = 1;
128+ }
129+ break;
130+ }
83 default: 131 default:
84− console.error("usage: cli.ts sync | discover [sourceId…] | probe <domain> | run-once <sensorId> | run-due [n] | relink-entities [days] | prune-blobs [batch] | refresh-clusters"); 132+ console.error("usage: cli.ts sync | discover [sourceId…] | probe <domain> | expand <domain> [--hints=json] [--posture] | run-once <sensorId> | run-due [n] | relink-entities [days] | prune-blobs [batch] | refresh-clusters | factory <seed|run|evaluate|stats|export|requeue>");
85 process.exitCode = 1; 133 process.exitCode = 1;
86 } 134 }
87 } 135 }
modified apps/engine/src/config.ts +22 −1
@@ -48,7 +48,7 @@ export const config = {
48 retention: { rawDays: Number(env.WS_RETENTION_RAW_DAYS ?? 21), changedRawDays: Number(env.WS_RETENTION_CHANGED_RAW_DAYS ?? 60), enabled: (env.WS_RETENTION ?? "1") !== "0" }, 48 retention: { rawDays: Number(env.WS_RETENTION_RAW_DAYS ?? 21), changedRawDays: Number(env.WS_RETENTION_CHANGED_RAW_DAYS ?? 60), enabled: (env.WS_RETENTION ?? "1") !== "0" },
49 /** silent-change bar: only eligible types with importance ≥ this become "silent" */ 49 /** silent-change bar: only eligible types with importance ≥ this become "silent" */
50 silentMinImportance: Number(env.WS_SILENT_MIN_IMPORTANCE ?? 45), 50 silentMinImportance: Number(env.WS_SILENT_MIN_IMPORTANCE ?? 45),
51− version: "0.2.0", 51+ version: "0.3.0",
52 }; 52 };
53 53
54 export const log = pino({ 54 export const log = pino({
@@ -56,3 +56,24 @@ export const log = pino({
56 ...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}), 56 ...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}),
57 base: { service: "websensor-engine" }, 57 base: { service: "websensor-engine" },
58 }); 58 });
59+
60+/** Source Factory (apps/engine/src/factory) — organization expansion, shadow monitoring, acceptance. */
61+export const factoryConfig = {
62+ enabled: (env.WS_FACTORY ?? "1") !== "0",
63+ /** seeds discovered in parallel */
64+ concurrency: Number(env.WS_FACTORY_CONCURRENCY ?? 6),
65+ /** HTTP requests per organization during discovery */
66+ budget: Number(env.WS_FACTORY_BUDGET ?? 90),
67+ /** minimum candidate score to enter shadow monitoring */
68+ minScore: Number(env.WS_FACTORY_MIN_SCORE ?? 0.35),
69+ /** shadow evaluation: checks needed, minimum and maximum shadow duration */
70+ shadowMinChecks: Number(env.WS_FACTORY_SHADOW_MIN_CHECKS ?? 5),
71+ shadowMinHours: Number(env.WS_FACTORY_SHADOW_MIN_HOURS ?? 24),
72+ shadowMaxHours: Number(env.WS_FACTORY_SHADOW_MAX_HOURS ?? 120),
73+ /** per-organization promotion caps */
74+ caps: { feeds: 4, pages: 6, releases: 4, sitemap: 1, status: 1, models: 1, filings: 1, api: 1, posture: 3, legal: 2, total: 18 },
75+ /** seconds between factory ticks */
76+ tickSeconds: Number(env.WS_FACTORY_TICK_SECONDS ?? 20),
77+ coverageDir: env.WS_COVERAGE_DIR ?? "./config/coverage",
78+ seedsDir: env.WS_FACTORY_SEEDS_DIR ?? "./config/factory/seeds",
79+};
added apps/engine/src/coverage-validate.ts +108 −0
@@ -0,0 +1,108 @@
1+import { promises as dns } from "node:dns";
2+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
3+import { join } from "node:path";
4+import YAML from "yaml";
5+import { coverageKey, coverageSectorSchema, type CoverageSector } from "@websensor/core";
6+
7+/**
8+ * Coverage-universe validator — no database.
9+ *
10+ * node node_modules/tsx/dist/cli.mjs apps/engine/src/coverage-validate.ts [config/coverage/<file>.yaml …] [--dns] [--quiet]
11+ *
12+ * Without arguments every file in config/coverage is checked. Reports schema issues, duplicate domains inside a
13+ * file, duplicate sector keys across files and (with --dns) domains that do not resolve. Exit code 1 on any error.
14+ */
15+const args = process.argv.slice(2);
16+const flag = (n: string): boolean => args.includes(`--${n}`);
17+const files = args.filter((a) => !a.startsWith("--"));
18+
19+export function loadCoverageFiles(dir = process.env.WS_COVERAGE_DIR ?? "./config/coverage"): { file: string; sector: CoverageSector | null; issues: string[] }[] {
20+ const list = files.length ? files : existsSync(dir) && statSync(dir).isDirectory() ? readdirSync(dir).filter((f) => /\.ya?ml$/.test(f) && !f.startsWith("_")).sort().map((f) => join(dir, f)) : [];
21+ return list.map((file) => {
22+ const issues: string[] = [];
23+ let sector: CoverageSector | null = null;
24+ try {
25+ const raw = YAML.parse(readFileSync(file, "utf8"));
26+ const parsed = coverageSectorSchema.safeParse(raw);
27+ if (!parsed.success) {
28+ for (const i of parsed.error.issues.slice(0, 40)) issues.push(`${i.path.join(".")}: ${i.message}`);
29+ if (parsed.error.issues.length > 40) issues.push(`… ${parsed.error.issues.length - 40} more issues`);
30+ } else sector = parsed.data;
31+ } catch (e) {
32+ issues.push(`YAML parse error: ${(e as Error).message}`);
33+ }
34+ if (sector) {
35+ const seen = new Map<string, string>();
36+ const keys = new Set<string>();
37+ for (const u of sector.universes) {
38+ if (keys.has(u.key)) issues.push(`duplicate universe key ${u.key}`);
39+ keys.add(u.key);
40+ if (!u.members.length) issues.push(`universe ${u.key} has no members`);
41+ for (const m of u.members) {
42+ if (!/^[a-z0-9.-]+\.[a-z]{2,}$/.test(m.domain) || m.domain.includes("..")) issues.push(`${u.key}: bad domain "${m.domain}" (${m.name})`);
43+ const k = coverageKey(m.domain);
44+ const prev = seen.get(k);
45+ if (prev && prev !== `${u.key}/${m.name}`) issues.push(`${u.key}: duplicate domain ${m.domain} (${m.name}) — already ${prev}`);
46+ seen.set(k, `${u.key}/${m.name}`);
47+ if (!m.country && !/^(INT|EU)$/.test(String(m.country))) issues.push(`${u.key}: ${m.name} has no country`);
48+ }
49+ }
50+ }
51+ return { file, sector, issues };
52+ });
53+}
54+
55+async function resolves(domain: string): Promise<boolean> {
56+ for (const h of [domain, `www.${domain}`]) {
57+ try {
58+ const r = await dns.lookup(h, { all: true });
59+ if (r.length) return true;
60+ } catch {
61+ // try next
62+ }
63+ }
64+ return false;
65+}
66+
67+async function main(): Promise<void> {
68+ const loaded = loadCoverageFiles();
69+ if (!loaded.length) {
70+ console.error("no coverage files found");
71+ process.exitCode = 1;
72+ return;
73+ }
74+ let errors = 0;
75+ const sectors = new Map<string, string>();
76+ for (const l of loaded) {
77+ const members = l.sector?.universes.reduce((n, u) => n + u.members.length, 0) ?? 0;
78+ const warnIssues = l.issues.filter((i) => / has no country$/.test(i));
79+ const hardIssues = l.issues.filter((i) => !/ has no country$/.test(i));
80+ errors += hardIssues.length;
81+ // Several files may extend the same sector (e.g. corporate indices per region); universes are merged by the loader.
82+ if (l.sector && !sectors.has(l.sector.sector)) sectors.set(l.sector.sector, l.file);
83+ console.log(`${hardIssues.length ? "FAIL" : "OK "} ${l.file} sector=${l.sector?.sector ?? "-"} universes=${l.sector?.universes.length ?? 0} members=${members}${warnIssues.length ? ` warn=${warnIssues.length}` : ""}`);
84+ for (const i of hardIssues) console.log(` ${i}`);
85+ if (!flag("quiet")) for (const i of warnIssues.slice(0, 15)) console.log(` WARN ${i}`);
86+ if (flag("dns") && l.sector) {
87+ const domains = [...new Set(l.sector.universes.flatMap((u) => u.members.map((m) => m.domain)))];
88+ let i = 0;
89+ const bad: string[] = [];
90+ await Promise.all(
91+ Array.from({ length: 16 }, async () => {
92+ while (i < domains.length) {
93+ const d = domains[i++]!;
94+ if (!(await resolves(d))) bad.push(d);
95+ }
96+ }),
97+ );
98+ for (const d of bad.sort()) console.log(` DNS ${d} does not resolve`);
99+ if (bad.length) console.log(` ${bad.length}/${domains.length} domains do not resolve (fix or remove them)`);
100+ errors += bad.length;
101+ }
102+ }
103+ const total = loaded.reduce((n, l) => n + (l.sector?.universes.reduce((m, u) => m + u.members.length, 0) ?? 0), 0);
104+ console.log(`\n${loaded.length} files · ${sectors.size} sectors · ${total} members → ${errors ? `${errors} errors` : "all good"}`);
105+ if (errors) process.exitCode = 1;
106+}
107+
108+if (process.argv[1] && /coverage-validate\.ts$/.test(process.argv[1])) void main();
added apps/engine/src/factory-main.ts +99 −0
@@ -0,0 +1,99 @@
1+import { closeDb, db, migrate, sql } from "@websensor/db";
2+import { closeDispatcher } from "@websensor/connectors";
3+import { config, factoryConfig, log } from "./config";
4+import { evaluateShadows, factoryStats, runFactoryBatch, seedFromCoverage, seedFromFiles } from "./factory";
5+import { closeRedis, getRedis } from "./redis";
6+
7+/**
8+ * Source Factory process (`websensor-factory`): a slow, polite background worker separate from the poller.
9+ * every tick — claim queued seeds (systemic organizations first) and run deep discovery + shadow creation
10+ * every 15 min — evaluate shadow sensors (accept / reject / defer)
11+ * every 24 h — re-seed from the coverage universes (new members become seeds automatically)
12+ * Heartbeat: Redis `ws:factory:status`.
13+ */
14+async function heartbeat(extra: Record<string, unknown>): Promise<void> {
15+ try {
16+ await getRedis().set("ws:factory:status", JSON.stringify({ ...extra, at: new Date().toISOString(), pid: process.pid, version: config.version }), "EX", 120);
17+ } catch {
18+ // best effort
19+ }
20+}
21+
22+async function main(): Promise<void> {
23+ log.info({ env: config.env, version: config.version, concurrency: factoryConfig.concurrency, budget: factoryConfig.budget }, "websensor factory starting");
24+ const applied = await migrate(config.databaseUrl);
25+ if (applied.length) log.info({ applied }, "migrations applied");
26+ if (!factoryConfig.enabled) {
27+ log.warn("WS_FACTORY=0 — factory idle");
28+ setInterval(() => void heartbeat({ idle: true }), 60_000);
29+ return;
30+ }
31+ let stopping = false;
32+ let busy = false;
33+ let processed = 0;
34+ let lastSeed = 0;
35+ let lastEval = 0;
36+
37+ const seed = async (): Promise<void> => {
38+ try {
39+ const a = await seedFromFiles();
40+ const b = await seedFromCoverage({ mode: "hinted" });
41+ log.info({ files: a, coverage: { members: b.members, seeds: b.seeds, inserted: b.inserted, issues: b.issues.length } }, "factory: seeds refreshed");
42+ for (const i of b.issues) log.warn({ issue: i }, "coverage file issue");
43+ } catch (e) {
44+ log.error({ err: (e as Error).message }, "factory: seeding failed");
45+ }
46+ };
47+
48+ const tick = async (): Promise<void> => {
49+ if (stopping || busy) return;
50+ busy = true;
51+ try {
52+ if (Date.now() - lastSeed > 24 * 3600e3) {
53+ lastSeed = Date.now();
54+ await seed();
55+ }
56+ if (Date.now() - lastEval > 15 * 60e3) {
57+ lastEval = Date.now();
58+ await evaluateShadows().catch((e) => log.warn({ err: (e as Error).message }, "factory: shadow evaluation failed"));
59+ }
60+ const out = await runFactoryBatch(factoryConfig.concurrency);
61+ processed += out.length;
62+ const queued = Number((await db.execute<{ n: string }>(sql`select count(*)::text as n from factory_seeds where status = 'queued'`)).rows[0]?.n ?? 0);
63+ await heartbeat({ processed, queued, lastBatch: out.map((o) => ({ seed: o.seedId, status: o.status, candidates: o.candidates, shadow: o.shadow, requests: o.requests, ms: o.durationMs })), concurrency: factoryConfig.concurrency });
64+ if (!out.length && processed % 50 === 0) {
65+ const st = await factoryStats();
66+ log.info({ seeds: st.seeds, candidates: st.candidates, shadow: st.shadow }, "factory: idle");
67+ }
68+ } catch (e) {
69+ log.error({ err: (e as Error).stack ?? (e as Error).message }, "factory tick failed");
70+ } finally {
71+ busy = false;
72+ }
73+ };
74+
75+ await heartbeat({ starting: true });
76+ void tick();
77+ const timer = setInterval(() => void tick(), factoryConfig.tickSeconds * 1000);
78+
79+ const shutdown = async (signal: string): Promise<void> => {
80+ if (stopping) return;
81+ stopping = true;
82+ log.info({ signal }, "factory shutting down");
83+ clearInterval(timer);
84+ const deadline = Date.now() + 60_000;
85+ while (busy && Date.now() < deadline) await new Promise((r) => setTimeout(r, 250));
86+ await closeDispatcher();
87+ await closeRedis();
88+ await closeDb();
89+ process.exit(0);
90+ };
91+ process.on("SIGINT", () => void shutdown("SIGINT"));
92+ process.on("SIGTERM", () => void shutdown("SIGTERM"));
93+ process.on("unhandledRejection", (e) => log.error({ err: e instanceof Error ? e.stack : String(e) }, "unhandled rejection"));
94+}
95+
96+main().catch((e) => {
97+ log.fatal({ err: (e as Error).stack ?? String(e) }, "factory failed to start");
98+ process.exit(1);
99+});
added apps/engine/src/factory/export.ts +46 −0
@@ -0,0 +1,46 @@
1+import YAML from "yaml";
2+import { db, sql } from "@websensor/db";
3+
4+/**
5+ * Export factory-created sources and accepted sensors as a registry fragment (reviewable YAML). Graduating a
6+ * fragment into `config/sources.d/` makes the registry sync own those sensors (`seed: true`) — the Factory never
7+ * re-creates them because their URLs already exist.
8+ */
9+export async function exportFactoryFragment(opts: { sector?: string; sinceDays?: number; includeShadow?: boolean } = {}): Promise<string> {
10+ const rows = await db.execute<Record<string, unknown>>(sql`
11+ select so.id, so.name, so.domain, so.homepage, so.categories, so.tier, so.importance_weight, so.country, so.language, so.first_party, so.origin, so.sector, so.notes,
12+ coalesce(json_agg(json_build_object('name', s.name, 'url', s.url, 'type', s.type, 'connector', s.connector, 'tier', s.tier, 'config', s.config - 'factory' - 'shadow' - 'seed' - 'candidate' - 'seedId' - 'shadowSince' - 'targetPriority' - 'acceptedAt' - 'shadowReport' - 'kind') order by s.name) filter (where s.id is not null), '[]'::json) as sensors
13+ from sources so
14+ left join sensors s on s.source_id = so.id and s.enabled and (s.config->>'factory') = 'true' and (s.status <> 'SHADOW' or ${opts.includeShadow ?? false})
15+ where (so.origin = 'factory' or exists (select 1 from sensors y where y.source_id = so.id and (y.config->>'factory') = 'true'))
16+ ${opts.sector ? sql`and so.sector = ${opts.sector}` : sql``}
17+ ${opts.sinceDays ? sql`and so.updated_at >= now() - make_interval(days => ${opts.sinceDays})` : sql``}
18+ group by so.id order by so.sector, so.id`);
19+ const entities = (await db.execute<{ source_id: string; alias: string }>(sql`select se.source_id, ea.alias from source_entities se join entity_aliases ea on ea.entity_id = se.entity_id where se.entity_id like 'org_%'`)).rows;
20+ const aliasBySource = new Map<string, string[]>();
21+ for (const e of entities) aliasBySource.set(e.source_id, [...(aliasBySource.get(e.source_id) ?? []), e.alias]);
22+ const out: Record<string, unknown>[] = [];
23+ for (const r of rows.rows) {
24+ const sensors = (r.sensors as Record<string, unknown>[]).map((s) => {
25+ const cfg = s.config as Record<string, unknown>;
26+ const o: Record<string, unknown> = { name: s.name, url: s.url, type: s.type, connector: s.connector, tier: s.tier };
27+ if (cfg && Object.keys(cfg).length) o.config = cfg;
28+ return o;
29+ });
30+ if (!sensors.length) continue;
31+ const aliases = (aliasBySource.get(String(r.id)) ?? []).filter((a) => a !== String(r.name).toLowerCase() && a !== String(r.domain).toLowerCase() && a !== String(r.domain).replace(/^www\./, "").toLowerCase());
32+ if (r.origin === "factory") {
33+ const src: Record<string, unknown> = { id: r.id, name: r.name, domain: r.domain, categories: r.categories, tier: r.tier };
34+ if (Number(r.importance_weight) !== 1) src.weight = Number(r.importance_weight);
35+ if (aliases.length) src.aliases = aliases;
36+ if (r.country) src.country = r.country;
37+ if (r.language) src.language = r.language;
38+ if (r.first_party === false) src.first_party = false;
39+ if (r.notes) src.notes = r.notes;
40+ src.sensors = sensors;
41+ out.push(src);
42+ } else out.push({ id: r.id, extend: true, sensors });
43+ }
44+ const header = `# WebSensor — Source Factory export (${new Date().toISOString().slice(0, 10)}${opts.sector ? `, sector ${opts.sector}` : ""}): ${out.length} organizations, ${out.reduce((n, s) => n + (s.sensors as unknown[]).length, 0)} accepted sensors.\n# Generated by \`cli.ts factory export\`; review, then place in config/sources.d/ to let the registry own these sensors.\n`;
45+ return header + YAML.stringify({ sources: out }, { lineWidth: 0, defaultStringType: "PLAIN", defaultKeyType: "PLAIN" });
46+}
added apps/engine/src/factory/factory.test.ts +67 −0
@@ -0,0 +1,67 @@
1+import { describe, expect, it } from "vitest";
2+import { coverageKey, registrableDomain } from "@websensor/core";
3+import type { DeepCandidate } from "@websensor/connectors";
4+import { itemsPerDayOf } from "@websensor/connectors";
5+import { KIND_WEIGHT, scoreCandidate, selectForShadow } from "./score";
6+
7+const cand = (over: Partial<DeepCandidate>): DeepCandidate => ({ url: "https://example.org/feed", type: "RSS", connector: "rss", kind: "news", name: "news feed", config: {}, suggestedTier: "B", evidence: "well-known path", value: 0.8, itemsPerDay: 0.5, firstPartyConfidence: 1, fetchCost: { bytes: 20_000, ms: 400 }, ...over });
8+
9+describe("registrable domain (coverage key)", () => {
10+ it("collapses hosts to the organization's registrable domain", () => {
11+ expect(registrableDomain("ir.apple.com")).toBe("apple.com");
12+ expect(registrableDomain("www.ons.gov.uk")).toBe("ons.gov.uk");
13+ expect(registrableDomain("www.bankofengland.co.uk")).toBe("bankofengland.co.uk");
14+ expect(registrableDomain("statcan.gc.ca")).toBe("statcan.gc.ca");
15+ expect(registrableDomain("open.canada.ca")).toBe("canada.ca");
16+ expect(registrableDomain("boj.or.jp")).toBe("boj.or.jp");
17+ expect(coverageKey("https://WWW.Example.COM/path")).toBe("example.com");
18+ });
19+});
20+
21+describe("factory scoring", () => {
22+ it("weights page classes and organization importance", () => {
23+ const status = scoreCandidate(cand({ kind: "status", connector: "statuspage", value: 0.95, itemsPerDay: null }), { importance: 2 });
24+ const careers = scoreCandidate(cand({ kind: "careers", connector: "http", value: 0.35, itemsPerDay: null }), { importance: 2 });
25+ expect(status.score).toBeGreaterThan(careers.score);
26+ expect(KIND_WEIGHT.status).toBeGreaterThan(KIND_WEIGHT.careers);
27+ const big = scoreCandidate(cand({}), { importance: 3 });
28+ const small = scoreCandidate(cand({}), { importance: 1 });
29+ expect(big.score).toBeGreaterThan(small.score);
30+ expect(big.reasons.some((r) => /importance 3/.test(r))).toBe(true);
31+ });
32+ it("penalises firehoses, dormant feeds and slow endpoints, explainably", () => {
33+ const base = scoreCandidate(cand({}), { importance: 2 }).score;
34+ expect(scoreCandidate(cand({ itemsPerDay: 200 }), { importance: 2 }).score).toBeLessThan(base);
35+ expect(scoreCandidate(cand({ itemsPerDay: 0.001 }), { importance: 2 }).score).toBeLessThan(base);
36+ const slow = scoreCandidate(cand({ fetchCost: { bytes: 100, ms: 12_000 } }), { importance: 2 });
37+ expect(slow.score).toBeLessThan(base);
38+ expect(slow.reasons.some((r) => /slow/.test(r))).toBe(true);
39+ });
40+ it("applies per-organization caps and the minimum score", () => {
41+ const feeds = Array.from({ length: 6 }, (_, i) => scoreCandidate(cand({ url: `https://example.org/f${i}`, value: 0.9 - i * 0.02 }), { importance: 2 }));
42+ const low = scoreCandidate(cand({ url: "https://example.org/careers", kind: "careers", connector: "http", value: 0.2, itemsPerDay: null }), { importance: 1 });
43+ const legal = ["terms", "privacy", "cookies"].map((p) => scoreCandidate(cand({ url: `https://example.org/${p}`, kind: "legal", connector: "http", type: "HTML", value: 0.7, itemsPerDay: null }), { importance: 2 }));
44+ const { selected, skipped } = selectForShadow([...feeds, low, ...legal], { kinds: new Map(), total: 0 });
45+ expect(selected.filter((s) => s.cand.connector === "rss")).toHaveLength(4);
46+ expect(selected.filter((s) => s.cand.kind === "legal")).toHaveLength(2);
47+ expect(skipped.some((s) => s.s === low && /score/.test(s.reason))).toBe(true);
48+ expect(skipped.some((s) => /feeds cap/.test(s.reason))).toBe(true);
49+ });
50+ it("respects sensors the organization already has", () => {
51+ const feed = scoreCandidate(cand({}), { importance: 2 });
52+ const { selected, skipped } = selectForShadow([feed], { kinds: new Map([["news", 4]]), total: 4 });
53+ expect(selected).toHaveLength(0);
54+ expect(skipped[0]?.reason).toMatch(/feeds cap/);
55+ });
56+});
57+
58+describe("expected change frequency", () => {
59+ it("uses the median gap and survives one bogus date", () => {
60+ const day = 86400e3;
61+ const now = Date.UTC(2026, 8, 13);
62+ const dates = [now, now - day, now - 2 * day, now - 3 * day, now - 4 * day, now - 5000 * day];
63+ expect(itemsPerDayOf(dates)).toBe(1);
64+ expect(itemsPerDayOf([now, now - day])).toBeNull();
65+ expect(itemsPerDayOf([now, now - 30 * day, now - 60 * day, now - 90 * day])).toBeCloseTo(0.033, 2);
66+ });
67+});
added apps/engine/src/factory/index.ts +5 −0
@@ -0,0 +1,5 @@
1+export * from "./seeds";
2+export * from "./score";
3+export * from "./run";
4+export * from "./shadow";
5+export * from "./export";
added apps/engine/src/factory/run.ts +288 −0
@@ -0,0 +1,288 @@
1+import { coverageKey, newId, slugify, sourceSchema, type SourceSeed, type Tier } from "@websensor/core";
2+import { discoverOrganization, type CandidateKind, type DeepDiscoveryHints } from "@websensor/connectors";
3+import { db, factorySeeds, sensors, sql, type FactorySeed } from "@websensor/db";
4+import { factoryConfig, log } from "../config";
5+import { priorityFor, upsertSourceRecord } from "../registry";
6+import { scoreCandidate, selectForShadow } from "./score";
7+
8+/**
9+ * Source Factory — one organization at a time:
10+ * seed → deep discovery → (create or attach the source) → score candidates → promotion policy →
11+ * shadow sensors (`status = SHADOW`: observed, evidence stored, nothing published) → later `evaluateShadows()`.
12+ */
13+export interface ProcessOutcome {
14+ seedId: string;
15+ sourceId: string | null;
16+ status: "discovered" | "blocked" | "error";
17+ candidates: number;
18+ shadow: number;
19+ requests: number;
20+ durationMs: number;
21+ notes: string[];
22+}
23+
24+interface SourceRow extends Record<string, unknown> {
25+ id: string;
26+ domain: string;
27+ categories: string[];
28+ tier: string;
29+ enabled: boolean;
30+}
31+
32+let sourceIndex: { at: number; byKey: Map<string, SourceRow>; ids: Set<string> } | null = null;
33+async function sourcesByDomain(force = false): Promise<{ byKey: Map<string, SourceRow>; ids: Set<string> }> {
34+ if (!force && sourceIndex && Date.now() - sourceIndex.at < 5 * 60_000) return sourceIndex;
35+ const rows = await db.execute<SourceRow>(sql`select id, domain, categories, tier, enabled from sources where kind = 'registry'`);
36+ const byKey = new Map<string, SourceRow>();
37+ const ids = new Set<string>();
38+ for (const r of rows.rows) {
39+ ids.add(r.id);
40+ const k = coverageKey(r.domain);
41+ if (!byKey.has(k)) byKey.set(k, r);
42+ }
43+ sourceIndex = { at: Date.now(), byKey, ids };
44+ return sourceIndex;
45+}
46+
47+export async function processSeed(seed: FactorySeed): Promise<ProcessOutcome> {
48+ const started = Date.now();
49+ const notes: string[] = [];
50+ const hints = (seed.hints ?? {}) as DeepDiscoveryHints & { posture?: boolean };
51+ const idx = await sourcesByDomain();
52+ const key = coverageKey(seed.domain);
53+ let source = idx.byKey.get(key) ?? null;
54+
55+ const disc = await discoverOrganization(seed.domain, {
56+ hints,
57+ budget: factoryConfig.budget,
58+ concurrency: 4,
59+ sensorIdForLogs: `factory_${seed.id}`,
60+ probePages: true,
61+ probeSubdomains: true,
62+ probePosture: seed.importance >= 3 || hints.posture === true,
63+ probeOpenApi: true,
64+ });
65+ notes.push(...disc.notes);
66+ // The homepage may redirect to another registrable domain that we already monitor.
67+ if (!source && disc.canonicalDomain !== key) source = idx.byKey.get(disc.canonicalDomain) ?? null;
68+
69+ if (!disc.candidates.length) {
70+ const status = disc.blocked ? "blocked" : "discovered";
71+ await db.execute(sql`update factory_seeds set status = ${status}, candidates = 0, shadow = 0, source_id = ${source?.id ?? null}, discovered_at = now(), last_error = ${disc.blocked ? `homepage ${disc.homeStatus} (anti-bot)` : null}, updated_at = now() where id = ${seed.id}`);
72+ await bumpFactoryDaily({ seeds_processed: 1, requests: disc.requests, blocked: disc.blocked ? 1 : 0 });
73+ return { seedId: seed.id, sourceId: source?.id ?? null, status, candidates: 0, shadow: 0, requests: disc.requests, durationMs: Date.now() - started, notes };
74+ }
75+
76+ // ---- source: attach or create --------------------------------------------------------------------------
77+ let sourceId: string;
78+ if (source) {
79+ sourceId = source.id;
80+ if (!source.enabled) notes.push("source disabled in registry — candidates recorded, no shadow sensors");
81+ } else {
82+ sourceId = idx.ids.has(seed.id) ? `${seed.id}-${slugify(seed.domain.split(".")[0] ?? "x")}`.slice(0, 70) : seed.id;
83+ const seedDoc: SourceSeed = sourceSchema.parse({
84+ id: sourceId,
85+ name: seed.name,
86+ domain: seed.domain,
87+ homepage: seed.homepage ?? `https://${seed.domain}`,
88+ categories: seed.categories,
89+ tier: (seed.tier as Tier) ?? "B",
90+ weight: seed.weight,
91+ aliases: seed.aliases,
92+ country: seed.country ?? undefined,
93+ language: seed.language ?? undefined,
94+ first_party: seed.firstParty,
95+ notes: `Source Factory · ${seed.sector ?? "-"}/${seed.universe ?? "-"} · discovered ${new Date().toISOString().slice(0, 10)}`,
96+ llm: true,
97+ });
98+ await upsertSourceRecord(seedDoc, { origin: "factory", sector: seed.sector });
99+ idx.ids.add(sourceId);
100+ idx.byKey.set(key, { id: sourceId, domain: seed.domain, categories: seed.categories, tier: seed.tier, enabled: true });
101+ source = idx.byKey.get(key)!;
102+ }
103+
104+ // ---- existing sensors (this source + global URL uniqueness) ----------------------------------------------
105+ const existing = await db.execute<{ id: string; url: string; type: string; connector: string; config: Record<string, unknown>; status: string }>(sql`select id, url, type, connector, config, status from sensors where source_id = ${sourceId}`);
106+ const existingUrls = new Set(existing.rows.map((r) => r.url.replace(/\/$/, "")));
107+ const existingKinds = new Map<CandidateKind, number>();
108+ let existingTotal = 0;
109+ for (const r of existing.rows) {
110+ if (r.status === "DISABLED") continue;
111+ existingTotal++;
112+ const k = kindOfExisting(r);
113+ existingKinds.set(k, (existingKinds.get(k) ?? 0) + 1);
114+ }
115+ const urls = disc.candidates.map((c) => c.url);
116+ const global = urls.length ? await db.execute<{ url: string; source_id: string }>(sql`select url, source_id from sensors where url = any(${sql.raw("array[" + urls.map((u) => "'" + u.replace(/'/g, "''") + "'").join(",") + "]::text[]")}) and source_id <> ${sourceId}`) : { rows: [] as { url: string; source_id: string }[] };
117+ const ownedElsewhere = new Map(global.rows.map((r) => [r.url.replace(/\/$/, ""), r.source_id]));
118+
119+ // ---- score + select ---------------------------------------------------------------------------------------
120+ const scored = disc.candidates.map((c) => scoreCandidate(c, { importance: seed.importance, weight: seed.weight }));
121+ const fresh = scored.filter((s) => !existingUrls.has(s.cand.url.replace(/\/$/, "")) && !ownedElsewhere.has(s.cand.url.replace(/\/$/, "")));
122+ const { selected, skipped } = source.enabled ? selectForShadow(fresh, { kinds: existingKinds, total: existingTotal }) : { selected: [], skipped: fresh.map((s) => ({ s, reason: "source disabled" })) };
123+
124+ // ---- persist candidates ------------------------------------------------------------------------------------
125+ const day = new Date().toISOString().slice(0, 10);
126+ let shadowCreated = 0;
127+ for (const s of scored) {
128+ const k = s.cand.url.replace(/\/$/, "");
129+ let status = "candidate";
130+ let reason: string | null = null;
131+ if (existingUrls.has(k)) {
132+ status = "duplicate";
133+ reason = "already a sensor of this source";
134+ } else if (ownedElsewhere.has(k)) {
135+ status = "duplicate";
136+ reason = `already a sensor of ${ownedElsewhere.get(k)}`;
137+ } else if (selected.includes(s)) status = "shadow";
138+ else {
139+ const sk = skipped.find((x) => x.s === s);
140+ status = sk && /score/.test(sk.reason) ? "rejected" : "candidate";
141+ reason = sk?.reason ?? null;
142+ }
143+ const candId = newId("cand");
144+ let shadowSensorId: string | null = null;
145+ if (status === "shadow") {
146+ shadowSensorId = await createShadowSensor(sourceId, source, s.cand, candId, seed);
147+ if (shadowSensorId) shadowCreated++;
148+ else {
149+ status = "duplicate";
150+ reason = "sensor id/url collision";
151+ }
152+ }
153+ await db.execute(sql`
154+ insert into discovery_candidates (id, source_id, url, kind, evidence, score, status, seed_id, connector, type, name, kind_class, config, tier, score_value, reason, shadow_sensor_id, found_at, updated_at)
155+ values (${candId}, ${sourceId}, ${s.cand.url}, ${s.cand.type}, ${s.cand.evidence}, ${JSON.stringify({ value: s.cand.value, score: s.score, reasons: s.reasons, itemCount: s.cand.itemCount ?? null, itemsPerDay: s.cand.itemsPerDay ?? null, firstParty: s.cand.firstPartyConfidence, cost: s.cand.fetchCost, title: s.cand.title ?? null, fingerprint: s.cand.fingerprint ?? null })}::jsonb, ${status}, ${seed.id}, ${s.cand.connector}, ${s.cand.type}, ${s.cand.name}, ${s.cand.kind}, ${JSON.stringify(s.cand.config)}::jsonb, ${s.cand.suggestedTier}, ${s.score}, ${reason}, ${shadowSensorId}, now(), now())
156+ on conflict (source_id, url) do update set evidence = excluded.evidence, score = excluded.score, score_value = excluded.score_value, seed_id = excluded.seed_id, connector = excluded.connector, type = excluded.type, name = excluded.name, kind_class = excluded.kind_class, config = excluded.config, tier = excluded.tier, updated_at = now(),
157+ status = case when discovery_candidates.status in ('accepted','rejected','shadow') then discovery_candidates.status else excluded.status end,
158+ reason = case when discovery_candidates.status in ('accepted','rejected','shadow') then discovery_candidates.reason else excluded.reason end,
159+ shadow_sensor_id = coalesce(discovery_candidates.shadow_sensor_id, excluded.shadow_sensor_id)`);
160+ }
161+ for (const r of disc.rejected.slice(0, 40)) {
162+ await db.execute(sql`insert into discovery_candidates (id, source_id, url, kind, evidence, status, seed_id, reason, kind_class) values (${newId("cand")}, ${sourceId}, ${r.url}, 'HTML', ${r.reason}, 'rejected', ${seed.id}, ${r.reason}, 'other') on conflict (source_id, url) do nothing`);
163+ }
164+
165+ await db.execute(sql`update factory_seeds set status = 'discovered', source_id = ${sourceId}, candidates = ${scored.length}, shadow = shadow + ${shadowCreated}, discovered_at = now(), last_error = null, updated_at = now() where id = ${seed.id}`);
166+ await db.execute(sql`update sources set robots_checked_at = now(), updated_at = now() where id = ${sourceId}`);
167+ await bumpFactoryDaily({ seeds_processed: 1, requests: disc.requests, candidates: scored.length, shadow_created: shadowCreated, blocked: disc.blocked ? 1 : 0 });
168+ log.info({ seed: seed.id, source: sourceId, candidates: scored.length, shadow: shadowCreated, requests: disc.requests, ms: Date.now() - started, day }, "factory: organization discovered");
169+ return { seedId: seed.id, sourceId, status: "discovered", candidates: scored.length, shadow: shadowCreated, requests: disc.requests, durationMs: Date.now() - started, notes };
170+}
171+
172+function kindOfExisting(r: { url: string; type: string; connector: string; config: Record<string, unknown> }): CandidateKind {
173+ const k = (r.config as { kind?: string }).kind as CandidateKind | undefined;
174+ if (k) return k;
175+ if (r.connector === "statuspage" || r.connector === "statusjson") return "status";
176+ if (r.connector === "sitemap") return "sitemap";
177+ if (r.connector === "github") return "releases";
178+ if (r.connector === "edgar") return "filings";
179+ if (r.connector === "dns" || r.connector === "tls" || r.connector === "headers" || r.connector === "rdap") return "posture";
180+ if (r.connector === "openapi") return "api";
181+ if (r.connector === "rss") return /press/i.test(r.url) ? "press" : /blog/i.test(r.url) ? "blog" : /security|advisor/i.test(r.url) ? "security" : /changelog|release/i.test(r.url) ? "changelog" : "news";
182+ if (/pricing|plans/i.test(r.url)) return "pricing";
183+ if (/terms|privacy|legal/i.test(r.url)) return "legal";
184+ if (/career|jobs/i.test(r.url)) return "careers";
185+ if (/changelog|release-notes|whats-new/i.test(r.url)) return "changelog";
186+ if (/security|advisor/i.test(r.url)) return "security";
187+ return "other";
188+}
189+
190+async function createShadowSensor(sourceId: string, source: SourceRow, c: { url: string; type: string; connector: string; name: string; config: Record<string, unknown>; suggestedTier: Tier; kind: CandidateKind }, candId: string, seed: FactorySeed): Promise<string | null> {
191+ let id = `${sourceId}_${slugify(c.name)}`.slice(0, 80);
192+ const clash = (await db.execute<{ url: string }>(sql`select url from sensors where id = ${id}`)).rows[0];
193+ if (clash && clash.url.replace(/\/$/, "") !== c.url.replace(/\/$/, "")) id = `${id}-${newId("x").slice(-5)}`;
194+ const tier = c.suggestedTier;
195+ const priority = 3; // shadows never compete with production sensors
196+ const jitter = Math.floor(Math.random() * 30 * 60e3);
197+ try {
198+ await db
199+ .insert(sensors)
200+ .values({ id, sourceId, name: c.name, url: c.url, type: c.type, connector: c.connector, tier, importanceWeight: 1, config: { ...c.config, factory: true, shadow: true, seed: false, kind: c.kind, candidate: candId, seedId: seed.id, shadowSince: new Date().toISOString(), targetPriority: priorityFor(tier, source.categories) }, status: "SHADOW", priority, validatedAt: new Date(), nextCheckAt: new Date(Date.now() + jitter) })
201+ .onConflictDoNothing();
202+ } catch (e) {
203+ log.warn({ sensor: id, err: (e as Error).message }, "factory: shadow sensor insert failed");
204+ return null;
205+ }
206+ return id;
207+}
208+
209+export async function bumpFactoryDaily(delta: Partial<Record<"seeds_processed" | "requests" | "candidates" | "shadow_created" | "accepted" | "rejected" | "blocked", number>>): Promise<void> {
210+ const day = new Date().toISOString().slice(0, 10);
211+ const d = { seeds_processed: 0, requests: 0, candidates: 0, shadow_created: 0, accepted: 0, rejected: 0, blocked: 0, ...delta };
212+ await db.execute(sql`insert into factory_daily (day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked) values (${day}, ${d.seeds_processed}, ${d.requests}, ${d.candidates}, ${d.shadow_created}, ${d.accepted}, ${d.rejected}, ${d.blocked})
213+ on conflict (day) do update set seeds_processed = factory_daily.seeds_processed + ${d.seeds_processed}, requests = factory_daily.requests + ${d.requests}, candidates = factory_daily.candidates + ${d.candidates}, shadow_created = factory_daily.shadow_created + ${d.shadow_created}, accepted = factory_daily.accepted + ${d.accepted}, rejected = factory_daily.rejected + ${d.rejected}, blocked = factory_daily.blocked + ${d.blocked}`).catch(() => undefined);
214+}
215+
216+/** Claim up to `n` queued seeds (systemic organizations first) and process them with bounded concurrency. */
217+export async function runFactoryBatch(n = factoryConfig.concurrency, opts: { seedIds?: string[] } = {}): Promise<ProcessOutcome[]> {
218+ const claimed = await db.execute<Record<string, unknown>>(sql`
219+ update factory_seeds set status = 'discovering', attempts = attempts + 1, updated_at = now()
220+ where id in (
221+ select id from factory_seeds
222+ where ${opts.seedIds ? sql`id = any(${sql.raw("array[" + opts.seedIds.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]")})` : sql`(status = 'queued' or (status = 'error' and attempts < 3 and updated_at < now() - interval '6 hours') or (status = 'discovering' and updated_at < now() - interval '2 hours'))`}
223+ order by importance desc, created_at asc limit ${n} for update skip locked)
224+ returning *`);
225+ const seeds = claimed.rows.map(normalizeSeed);
226+ const out: ProcessOutcome[] = [];
227+ let i = 0;
228+ await Promise.all(
229+ Array.from({ length: Math.min(factoryConfig.concurrency, seeds.length) }, async () => {
230+ while (i < seeds.length) {
231+ const seed = seeds[i++]!;
232+ try {
233+ out.push(await processSeed(seed));
234+ } catch (e) {
235+ log.error({ seed: seed.id, err: (e as Error).stack ?? (e as Error).message }, "factory: seed failed");
236+ await db.execute(sql`update factory_seeds set status = 'error', last_error = ${(e as Error).message.slice(0, 500)}, updated_at = now() where id = ${seed.id}`).catch(() => undefined);
237+ out.push({ seedId: seed.id, sourceId: null, status: "error", candidates: 0, shadow: 0, requests: 0, durationMs: 0, notes: [(e as Error).message] });
238+ }
239+ }
240+ }),
241+ );
242+ return out;
243+}
244+
245+function normalizeSeed(r: Record<string, unknown>): FactorySeed {
246+ const d = (v: unknown): Date | null => (v ? new Date(v as string) : null);
247+ return {
248+ id: r.id as string,
249+ name: r.name as string,
250+ domain: r.domain as string,
251+ homepage: (r.homepage as string | null) ?? null,
252+ categories: (r.categories as string[]) ?? [],
253+ country: (r.country as string | null) ?? null,
254+ language: (r.language as string | null) ?? null,
255+ tier: (r.tier as string) ?? "B",
256+ weight: Number(r.weight ?? 1),
257+ importance: Number(r.importance ?? 2),
258+ aliases: (r.aliases as string[]) ?? [],
259+ firstParty: Boolean(r.first_party ?? true),
260+ sector: (r.sector as string | null) ?? null,
261+ universe: (r.universe as string | null) ?? null,
262+ hints: (r.hints as Record<string, unknown>) ?? {},
263+ status: r.status as string,
264+ attempts: Number(r.attempts ?? 0),
265+ sourceId: (r.source_id as string | null) ?? null,
266+ candidates: Number(r.candidates ?? 0),
267+ shadow: Number(r.shadow ?? 0),
268+ accepted: Number(r.accepted ?? 0),
269+ rejected: Number(r.rejected ?? 0),
270+ lastError: (r.last_error as string | null) ?? null,
271+ discoveredAt: d(r.discovered_at),
272+ createdAt: d(r.created_at) ?? new Date(),
273+ updatedAt: d(r.updated_at) ?? new Date(),
274+ };
275+}
276+
277+export async function factoryStats(): Promise<Record<string, unknown>> {
278+ const [seeds, cands, shadows, daily, sectors] = await Promise.all([
279+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from factory_seeds group by status`).then((r) => r.rows),
280+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from discovery_candidates group by status`).then((r) => r.rows),
281+ db.execute<Record<string, unknown>>(sql`select count(*)::int as shadow, count(*) filter (where total_runs >= ${factoryConfig.shadowMinChecks})::int as ready, count(*) filter (where health <> 'UP')::int as unhealthy from sensors where status = 'SHADOW'`).then((r) => r.rows[0]),
282+ db.execute<Record<string, unknown>>(sql`select day::text as day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked from factory_daily order by day desc limit 30`).then((r) => r.rows),
283+ db.execute<Record<string, unknown>>(sql`select coalesce(sector, '-') as sector, count(*)::int as seeds, count(*) filter (where status = 'queued')::int as queued, count(*) filter (where status in ('discovered','done'))::int as discovered, count(*) filter (where status = 'blocked')::int as blocked, sum(shadow)::int as shadow, sum(accepted)::int as accepted, sum(rejected)::int as rejected from factory_seeds group by 1 order by seeds desc`).then((r) => r.rows),
284+ ]);
285+ return { seeds: Object.fromEntries(seeds.map((r) => [r.status, r.n])), candidates: Object.fromEntries(cands.map((r) => [r.status, r.n])), shadow: shadows, daily, sectors, config: { concurrency: factoryConfig.concurrency, budget: factoryConfig.budget, minScore: factoryConfig.minScore, shadowMinChecks: factoryConfig.shadowMinChecks, shadowMinHours: factoryConfig.shadowMinHours, shadowMaxHours: factoryConfig.shadowMaxHours, caps: factoryConfig.caps } };
286+}
287+
288+export { factorySeeds };
added apps/engine/src/factory/score.ts +114 −0
@@ -0,0 +1,114 @@
1+import type { CandidateKind, DeepCandidate } from "@websensor/connectors";
2+import { factoryConfig } from "../config";
3+
4+/**
5+ * Candidate scoring (Source Factory, stage "signal/noise evaluation before shadow"). Combines the discovery
6+ * value (evidence quality) with the organization's importance, the class of page, first-party confidence,
7+ * expected change frequency and fetch cost. Explainable: every score carries its reasons.
8+ */
9+export interface ScoredCandidate {
10+ cand: DeepCandidate;
11+ score: number;
12+ reasons: string[];
13+}
14+
15+/** How much a page class is worth for change intelligence (1 = neutral). */
16+export const KIND_WEIGHT: Record<CandidateKind, number> = {
17+ status: 1.15,
18+ security: 1.15,
19+ press: 1.1,
20+ news: 1.05,
21+ filings: 1.1,
22+ changelog: 1.05,
23+ releases: 1.0,
24+ pricing: 1.05,
25+ models: 1.05,
26+ ir: 0.95,
27+ legal: 0.9,
28+ blog: 0.9,
29+ api: 0.9,
30+ leadership: 0.85,
31+ data: 0.85,
32+ sitemap: 0.8,
33+ docs: 0.7,
34+ careers: 0.6,
35+ posture: 0.6,
36+ other: 0.7,
37+};
38+
39+export function scoreCandidate(c: DeepCandidate, seed: { importance: number; weight?: number }): ScoredCandidate {
40+ const reasons: string[] = [];
41+ let score = c.value;
42+ reasons.push(`evidence ${c.value.toFixed(2)} (${c.evidence})`);
43+ const kw = KIND_WEIGHT[c.kind] ?? 0.7;
44+ score *= kw;
45+ if (kw !== 1) reasons.push(`${c.kind} ×${kw.toFixed(2)}`);
46+ const imp = seed.importance === 3 ? 1.1 : seed.importance === 1 ? 0.85 : 1;
47+ score *= imp * (seed.weight ?? 1);
48+ if (imp !== 1) reasons.push(`organization importance ${seed.importance} ×${imp.toFixed(2)}`);
49+ score *= c.firstPartyConfidence;
50+ if (c.firstPartyConfidence < 1) reasons.push(`first-party confidence ${c.firstPartyConfidence}`);
51+ if (c.itemsPerDay !== null && c.itemsPerDay !== undefined) {
52+ if (c.itemsPerDay > 60) {
53+ score *= 0.85;
54+ reasons.push(`firehose ${c.itemsPerDay}/day ×0.85`);
55+ } else if (c.itemsPerDay < 0.02) {
56+ score *= 0.8;
57+ reasons.push(`nearly dormant ${c.itemsPerDay}/day ×0.80`);
58+ } else if (c.itemsPerDay >= 0.2) {
59+ score *= 1.05;
60+ reasons.push(`active ${c.itemsPerDay}/day ×1.05`);
61+ }
62+ }
63+ if (c.fetchCost.ms > 8000) {
64+ score -= 0.08;
65+ reasons.push(`slow (${Math.round(c.fetchCost.ms)} ms) −0.08`);
66+ }
67+ if (c.fetchCost.bytes > 5 * 1024 * 1024) {
68+ score -= 0.08;
69+ reasons.push(`heavy (${Math.round(c.fetchCost.bytes / 1048576)} MB) −0.08`);
70+ }
71+ score = Math.round(Math.max(0, Math.min(1.2, score)) * 1000) / 1000;
72+ return { cand: c, score, reasons };
73+}
74+
75+/** Promotion policy: caps per organization and per class, best score first. Returns the selected candidates and the reason for the others. */
76+export function selectForShadow(scored: ScoredCandidate[], existing: { kinds: Map<CandidateKind, number>; total: number }): { selected: ScoredCandidate[]; skipped: { s: ScoredCandidate; reason: string }[] } {
77+ const caps = factoryConfig.caps;
78+ const groupOf = (k: CandidateKind): keyof typeof caps => (k === "news" || k === "press" || k === "blog" || k === "ir" || k === "changelog" || k === "security" ? "feeds" : k === "pricing" || k === "legal" || k === "careers" || k === "leadership" || k === "docs" || k === "data" || k === "other" ? "pages" : (k as keyof typeof caps));
79+ const used = new Map<string, number>();
80+ for (const [k, n] of existing.kinds) used.set(groupOf(k), (used.get(groupOf(k)) ?? 0) + n);
81+ let total = existing.total;
82+ const selected: ScoredCandidate[] = [];
83+ const skipped: { s: ScoredCandidate; reason: string }[] = [];
84+ const legal = (existing.kinds.get("legal") ?? 0);
85+ let legalUsed = legal;
86+ for (const s of [...scored].sort((a, b) => b.score - a.score)) {
87+ if (s.score < factoryConfig.minScore) {
88+ skipped.push({ s, reason: `score ${s.score} < ${factoryConfig.minScore}` });
89+ continue;
90+ }
91+ if (total >= caps.total) {
92+ skipped.push({ s, reason: `organization cap ${caps.total} reached` });
93+ continue;
94+ }
95+ // Feed-type candidates (rss connector) and HTML pages are capped by group; connectors with their own cap by kind.
96+ const g = s.cand.connector === "rss" ? "feeds" : s.cand.connector === "http" && s.cand.kind !== "posture" ? "pages" : groupOf(s.cand.kind);
97+ const cap = caps[g as keyof typeof caps] ?? 1;
98+ if ((used.get(g) ?? 0) >= cap) {
99+ skipped.push({ s, reason: `${g} cap ${cap} reached` });
100+ continue;
101+ }
102+ if (s.cand.kind === "legal") {
103+ if (legalUsed >= caps.legal) {
104+ skipped.push({ s, reason: `legal cap ${caps.legal} reached` });
105+ continue;
106+ }
107+ legalUsed++;
108+ }
109+ used.set(g, (used.get(g) ?? 0) + 1);
110+ total++;
111+ selected.push(s);
112+ }
113+ return { selected, skipped };
114+}
added apps/engine/src/factory/seeds.ts +168 −0
@@ -0,0 +1,168 @@
1+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
2+import { join } from "node:path";
3+import YAML from "yaml";
4+import { coverageKey, coverageSectorSchema, slugify, type CoverageMember, type CoverageSector } from "@websensor/core";
5+import { db, factorySeeds, sql } from "@websensor/db";
6+import { factoryConfig, log } from "../config";
7+
8+/**
9+ * Factory seeds come from two places: the coverage universes (`config/coverage/*.yaml`, every member that is not
10+ * monitored yet — or that carries hints worth expanding) and ad-hoc seed files (`config/factory/seeds/*.yaml`,
11+ * `{ seeds: [{ name, domain, categories, country, … }] }`) or the admin API.
12+ */
13+export interface SeedInput {
14+ id?: string;
15+ name: string;
16+ domain: string;
17+ homepage?: string;
18+ categories?: string[];
19+ country?: string;
20+ language?: string;
21+ tier?: string;
22+ weight?: number;
23+ importance?: number;
24+ aliases?: string[];
25+ first_party?: boolean;
26+ sector?: string;
27+ universe?: string;
28+ hints?: Record<string, unknown>;
29+}
30+
31+export function loadCoverageSectors(dir = factoryConfig.coverageDir): { sectors: CoverageSector[]; issues: string[] } {
32+ const sectors: CoverageSector[] = [];
33+ const issues: string[] = [];
34+ if (!existsSync(dir) || !statSync(dir).isDirectory()) return { sectors, issues: [`coverage dir ${dir} not found`] };
35+ for (const f of readdirSync(dir).filter((x) => /\.ya?ml$/.test(x) && !x.startsWith("_")).sort()) {
36+ try {
37+ const parsed = coverageSectorSchema.safeParse(YAML.parse(readFileSync(join(dir, f), "utf8")));
38+ if (!parsed.success) {
39+ issues.push(`${f}: ${parsed.error.issues.slice(0, 3).map((i) => `${i.path.join(".")}: ${i.message}`).join("; ")}`);
40+ continue;
41+ }
42+ sectors.push(parsed.data);
43+ } catch (e) {
44+ issues.push(`${f}: ${(e as Error).message}`);
45+ }
46+ }
47+ return { sectors, issues };
48+}
49+
50+/** Merge files that share a sector key. */
51+export function mergeSectors(sectors: CoverageSector[]): CoverageSector[] {
52+ const byKey = new Map<string, CoverageSector>();
53+ for (const s of sectors) {
54+ const prev = byKey.get(s.sector);
55+ if (!prev) byKey.set(s.sector, { ...s, universes: [...s.universes] });
56+ else prev.universes.push(...s.universes);
57+ }
58+ return [...byKey.values()];
59+}
60+
61+export function memberToSeed(sector: CoverageSector, universe: string, m: CoverageMember): SeedInput {
62+ return {
63+ name: m.name,
64+ domain: m.domain,
65+ categories: m.categories ?? sector.categories,
66+ country: m.country,
67+ language: m.language,
68+ tier: m.tier ?? sector.tier,
69+ importance: m.importance,
70+ aliases: m.aliases,
71+ first_party: m.first_party,
72+ sector: sector.sector,
73+ universe,
74+ hints: m.hints as Record<string, unknown>,
75+ };
76+}
77+
78+function hasHints(h: Record<string, unknown> | undefined): boolean {
79+ if (!h) return false;
80+ return Boolean(h.cik || h.github_org || (Array.isArray(h.github_repos) && h.github_repos.length) || h.hf_author || h.status_url || (Array.isArray(h.urls) && h.urls.length) || (Array.isArray(h.hosts) && h.hosts.length));
81+}
82+
83+/** Registrable domains currently monitored (source domains + hosts of enabled non-shadow sensors). */
84+export async function monitoredDomainKeys(): Promise<Set<string>> {
85+ const rows = await db.execute<{ d: string }>(sql`
86+ select domain as d from sources where enabled and kind = 'registry'
87+ union
88+ select lower(split_part(split_part(url, '/', 3), ':', 1)) as d from sensors where enabled and status <> 'SHADOW' and url like 'http%'`);
89+ const out = new Set<string>();
90+ for (const r of rows.rows) if (r.d) out.add(coverageKey(r.d));
91+ return out;
92+}
93+
94+export async function upsertSeeds(inputs: SeedInput[], opts: { requeue?: boolean } = {}): Promise<{ inserted: number; updated: number; skipped: number }> {
95+ let inserted = 0;
96+ let updated = 0;
97+ let skipped = 0;
98+ const existingIds = new Set((await db.execute<{ id: string }>(sql`select id from sources`)).rows.map((r) => r.id));
99+ const existingSeedDomains = new Map((await db.execute<{ id: string; domain: string }>(sql`select id, domain from factory_seeds`)).rows.map((r) => [coverageKey(r.domain), r.id]));
100+ for (const s of inputs) {
101+ const domain = s.domain.toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "");
102+ const key = coverageKey(domain);
103+ let id = s.id ?? existingSeedDomains.get(key) ?? slugify(s.name).slice(0, 60);
104+ if (!id) id = slugify(domain);
105+ // A registry source with the same id but another organization → disambiguate by domain.
106+ if (!s.id && !existingSeedDomains.has(key) && existingIds.has(id)) {
107+ const src = (await db.execute<{ domain: string }>(sql`select domain from sources where id = ${id}`)).rows[0];
108+ if (src && coverageKey(src.domain) !== key) id = `${id}-${slugify(domain.split(".")[0] ?? domain)}`.slice(0, 70);
109+ }
110+ const r = await db.execute<{ inserted: boolean }>(sql`
111+ insert into factory_seeds (id, name, domain, homepage, categories, country, language, tier, weight, importance, aliases, first_party, sector, universe, hints, status)
112+ values (${id}, ${s.name}, ${domain}, ${s.homepage ?? null}, ${sql.raw("'{" + (s.categories ?? []).map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.country ?? null}, ${s.language ?? null}, ${s.tier ?? "B"}, ${s.weight ?? 1}, ${s.importance ?? 2}, ${sql.raw("'{" + (s.aliases ?? []).map((c) => '"' + c.replace(/["\\]/g, "") + '"').join(",") + "}'::text[]")}, ${s.first_party ?? true}, ${s.sector ?? null}, ${s.universe ?? null}, ${JSON.stringify(s.hints ?? {})}::jsonb, 'queued')
113+ on conflict (id) do update set name = excluded.name, domain = excluded.domain, categories = excluded.categories, country = coalesce(excluded.country, factory_seeds.country), language = coalesce(excluded.language, factory_seeds.language), tier = excluded.tier, importance = excluded.importance, aliases = excluded.aliases, sector = coalesce(excluded.sector, factory_seeds.sector), universe = coalesce(excluded.universe, factory_seeds.universe), hints = factory_seeds.hints || excluded.hints, updated_at = now()${opts.requeue ? sql`, status = 'queued'` : sql``}
114+ returning (xmax = 0) as inserted`);
115+ if (r.rows[0]?.inserted) inserted++;
116+ else if (r.rowCount) updated++;
117+ else skipped++;
118+ existingSeedDomains.set(key, id);
119+ }
120+ log.info({ inserted, updated, skipped }, "factory seeds upserted");
121+ return { inserted, updated, skipped };
122+}
123+
124+/**
125+ * Turn coverage universes into seeds. `mode`: `uncovered` = members without any monitored domain; `hinted` = also
126+ * covered members that carry expansion hints; `all` = every member (re-discover everything).
127+ */
128+export async function seedFromCoverage(opts: { mode?: "uncovered" | "hinted" | "all"; sectors?: string[]; requeue?: boolean } = {}): Promise<{ members: number; seeds: number; inserted: number; updated: number; issues: string[] }> {
129+ const mode = opts.mode ?? "hinted";
130+ const { sectors, issues } = loadCoverageSectors();
131+ const monitored = await monitoredDomainKeys();
132+ const inputs: SeedInput[] = [];
133+ const seen = new Set<string>();
134+ let members = 0;
135+ for (const sector of mergeSectors(sectors)) {
136+ if (opts.sectors && !opts.sectors.includes(sector.sector)) continue;
137+ for (const u of sector.universes) {
138+ for (const m of u.members) {
139+ members++;
140+ const key = coverageKey(m.domain);
141+ if (seen.has(key)) continue;
142+ const covered = monitored.has(key);
143+ if (mode === "uncovered" && covered) continue;
144+ if (mode === "hinted" && covered && !hasHints(m.hints as Record<string, unknown>)) continue;
145+ seen.add(key);
146+ inputs.push(memberToSeed(sector, u.key, m));
147+ }
148+ }
149+ }
150+ const r = await upsertSeeds(inputs, { requeue: opts.requeue });
151+ return { members, seeds: inputs.length, inserted: r.inserted, updated: r.updated, issues };
152+}
153+
154+/** `config/factory/seeds/*.yaml` → `{ seeds: [...] }` */
155+export async function seedFromFiles(dir = factoryConfig.seedsDir): Promise<{ files: number; seeds: number }> {
156+ if (!existsSync(dir)) return { files: 0, seeds: 0 };
157+ const inputs: SeedInput[] = [];
158+ let files = 0;
159+ for (const f of readdirSync(dir).filter((x) => /\.ya?ml$/.test(x)).sort()) {
160+ files++;
161+ const raw = YAML.parse(readFileSync(join(dir, f), "utf8")) as { seeds?: SeedInput[] };
162+ for (const s of raw.seeds ?? []) if (s?.name && s?.domain) inputs.push(s);
163+ }
164+ await upsertSeeds(inputs);
165+ return { files, seeds: inputs.length };
166+}
167+
168+export { factorySeeds };
added apps/engine/src/factory/shadow.ts +109 −0
@@ -0,0 +1,109 @@
1+import { NOISE_CLASSES } from "@websensor/core";
2+import { db, sql } from "@websensor/db";
3+import { factoryConfig, log } from "../config";
4+import { bumpFactoryDaily } from "./run";
5+
6+/**
7+ * Shadow evaluation — the Factory's "signal/noise evaluation" stage. A shadow sensor is polled like any other
8+ * sensor (evidence is stored) but never publishes. Once it has enough checks (or enough time has passed) it is:
9+ * rejected — errors (≥ 50 % failed checks, or 3+ consecutive errors), pure noise (changes on most checks,
10+ * none of them meaningful), or a duplicate (same latest canonical content as an active sensor of
11+ * the same source);
12+ * accepted — otherwise, becoming a production sensor with its real priority;
13+ * deferred — not enough evidence yet (until `shadowMaxHours`, then accepted if it never failed).
14+ */
15+export interface ShadowDecision {
16+ sensorId: string;
17+ decision: "accepted" | "rejected" | "deferred";
18+ reason: string;
19+}
20+
21+interface ShadowRow extends Record<string, unknown> {
22+ id: string;
23+ source_id: string;
24+ url: string;
25+ tier: string;
26+ config: Record<string, unknown>;
27+ total_runs: number;
28+ consecutive_errors: number;
29+ raw_changes: number;
30+ health: string;
31+ last_snapshot_id: string | null;
32+ shadow_since: string | null;
33+ categories: string[];
34+ runs: number;
35+ failed: number;
36+ changes: number;
37+ noise_changes: number;
38+ last_canonical: string | null;
39+ dup_sensor: string | null;
40+}
41+
42+export async function evaluateShadows(opts: { limit?: number; force?: boolean } = {}): Promise<{ accepted: number; rejected: number; deferred: number; decisions: ShadowDecision[] }> {
43+ const noise = [...NOISE_CLASSES];
44+ const rows = await db.execute<ShadowRow>(sql`
45+ select s.id, s.source_id, s.url, s.tier, s.config, s.total_runs, s.consecutive_errors, s.raw_changes, s.health, s.last_snapshot_id,
46+ s.config->>'shadowSince' as shadow_since, so.categories,
47+ (select count(*)::int from sensor_runs r where r.sensor_id = s.id) as runs,
48+ (select count(*)::int from sensor_runs r where r.sensor_id = s.id and r.outcome in ('error','parse_error','rate_limited','missing')) as failed,
49+ (select count(*)::int from changes c where c.sensor_id = s.id) as changes,
50+ (select count(*)::int from changes c where c.sensor_id = s.id and c.change_class = any(${sql.raw("array[" + noise.map((n) => "'" + n + "'").join(",") + "]::text[]")})) as noise_changes,
51+ (select sn.canonical_hash from snapshots sn where sn.id = s.last_snapshot_id) as last_canonical,
52+ (select x.id from sensors x join snapshots sx on sx.id = x.last_snapshot_id where x.source_id = s.source_id and x.id <> s.id and x.status <> 'SHADOW' and x.enabled and sx.canonical_hash = (select sn2.canonical_hash from snapshots sn2 where sn2.id = s.last_snapshot_id) limit 1) as dup_sensor
53+ from sensors s join sources so on so.id = s.source_id
54+ where s.status = 'SHADOW' and s.enabled
55+ order by s.total_runs desc
56+ limit ${opts.limit ?? 2000}`);
57+ const decisions: ShadowDecision[] = [];
58+ let accepted = 0;
59+ let rejected = 0;
60+ let deferred = 0;
61+ const now = Date.now();
62+ for (const r of rows.rows) {
63+ const since = r.shadow_since ? new Date(r.shadow_since).getTime() : now;
64+ const hours = (now - since) / 3600e3;
65+ const runs = Math.max(r.runs, r.total_runs);
66+ let decision: ShadowDecision["decision"] = "deferred";
67+ let reason = `${runs} checks · ${hours.toFixed(0)} h`;
68+ if (r.dup_sensor && runs >= 2) {
69+ decision = "rejected";
70+ reason = `duplicate content of ${r.dup_sensor}`;
71+ } else if (r.consecutive_errors >= 3 || (runs >= 4 && r.failed / runs >= 0.5)) {
72+ decision = "rejected";
73+ reason = `errors ${r.failed}/${runs} (consecutive ${r.consecutive_errors})`;
74+ } else if (r.changes >= 4 && r.noise_changes === r.changes && r.changes >= runs * 0.6) {
75+ decision = "rejected";
76+ reason = `noise: ${r.changes} changes on ${runs} checks, all cosmetic/timestamp/ads`;
77+ } else if (opts.force || (runs >= factoryConfig.shadowMinChecks && hours >= factoryConfig.shadowMinHours) || (hours >= factoryConfig.shadowMaxHours && runs >= 2 && r.failed === 0)) {
78+ if (r.health !== "UP" && !opts.force) {
79+ decision = "deferred";
80+ reason = `health ${r.health} — waiting`;
81+ } else {
82+ decision = "accepted";
83+ reason = `${runs} checks · ${r.changes} changes (${r.changes - r.noise_changes} meaningful) · ${r.failed} errors · ${hours.toFixed(0)} h`;
84+ }
85+ } else if (hours >= factoryConfig.shadowMaxHours && runs < 2) {
86+ decision = "rejected";
87+ reason = `never observed successfully in ${hours.toFixed(0)} h`;
88+ }
89+
90+ if (decision === "accepted") {
91+ const targetPriority = Number((r.config as { targetPriority?: number }).targetPriority ?? 2);
92+ await db.execute(sql`update sensors set status = 'ACTIVE', priority = ${targetPriority}, config = (config - 'shadow') || jsonb_build_object('acceptedAt', ${new Date().toISOString()}::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${r.id}`);
93+ await db.execute(sql`update discovery_candidates set status = 'accepted', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${r.id}`);
94+ await db.execute(sql`update factory_seeds set accepted = accepted + 1, updated_at = now() where id = ${(r.config as { seedId?: string }).seedId ?? ""}`);
95+ accepted++;
96+ } else if (decision === "rejected") {
97+ await db.execute(sql`update sensors set status = 'DISABLED', enabled = false, config = config || jsonb_build_object('rejectedAt', ${new Date().toISOString()}::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${r.id}`);
98+ await db.execute(sql`update discovery_candidates set status = 'rejected', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${r.id}`);
99+ await db.execute(sql`update factory_seeds set rejected = rejected + 1, updated_at = now() where id = ${(r.config as { seedId?: string }).seedId ?? ""}`);
100+ rejected++;
101+ } else deferred++;
102+ decisions.push({ sensorId: r.id, decision, reason });
103+ }
104+ if (accepted || rejected) {
105+ await bumpFactoryDaily({ accepted, rejected });
106+ log.info({ accepted, rejected, deferred }, "factory: shadow evaluation");
107+ }
108+ return { accepted, rejected, deferred, decisions };
109+}
modified apps/engine/src/pipeline.ts +9 −2
@@ -54,7 +54,8 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc
54 const isErr = outcome === "error" || outcome === "parse_error" || outcome === "rate_limited"; 54 const isErr = outcome === "error" || outcome === "parse_error" || outcome === "rate_limited";
55 const consecutive = isErr ? sensor.consecutiveErrors + 1 : 0; 55 const consecutive = isErr ? sensor.consecutiveErrors + 1 : 0;
56 const health = outcome === "rate_limited" ? "RATE_LIMITED" : isErr ? (consecutive >= 5 ? "ERROR" : "DEGRADED") : outcome === "missing" ? "DEGRADED" : "UP"; 56 const health = outcome === "rate_limited" ? "RATE_LIMITED" : isErr ? (consecutive >= 5 ? "ERROR" : "DEGRADED") : outcome === "missing" ? "DEGRADED" : "UP";
57− const status = !sensor.enabled ? "DISABLED" : health === "ERROR" || health === "RATE_LIMITED" ? "DEGRADED" : "ACTIVE"; 57+ // Shadow sensors (Source Factory) keep their lifecycle status until evaluateShadows() decides.
58+ const status = !sensor.enabled ? "DISABLED" : sensor.status === "SHADOW" ? "SHADOW" : health === "ERROR" || health === "RATE_LIMITED" ? "DEGRADED" : "ACTIVE";
58 const changes7d = await countChanges7d(sensor.id); 59 const changes7d = await countChanges7d(sensor.id);
59 const events7d = await countEvents7d(sensor.id); 60 const events7d = await countEvents7d(sensor.id);
60 let interval = nextIntervalSeconds({ tier: sensor.tier as Tier, baseIntervalSeconds: sensor.baseIntervalSeconds, lastChangeAt: extra.changed ? new Date() : sensor.lastChangeAt, changes7d, events7d, consecutiveErrors: consecutive, lastWas304: outcome === "not_modified" }); 61 let interval = nextIntervalSeconds({ tier: sensor.tier as Tier, baseIntervalSeconds: sensor.baseIntervalSeconds, lastChangeAt: extra.changed ? new Date() : sensor.lastChangeAt, changes7d, events7d, consecutiveErrors: consecutive, lastWas304: outcome === "not_modified" });
@@ -224,6 +225,12 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc
224 if (isNoise) m.suppressed.inc({ reason: `noise_${semantic.class}` }); 225 if (isNoise) m.suppressed.inc({ reason: `noise_${semantic.class}` });
225 return finish("changed", { snapshotId: snapId, changed: true }); 226 return finish("changed", { snapshotId: snapId, changed: true });
226 } 227 }
228+ // Shadow monitoring (Source Factory): evidence and changes are recorded, nothing is published until the sensor is accepted.
229+ if (sensor.status === "SHADOW") {
230+ m.suppressed.inc({ reason: "shadow" });
231+ await db.update(changes).set({ meaningful: true }).where(sql`id = ${changeId}`);
232+ return finish("changed", { snapshotId: snapId, changed: true });
233+ }
227 234
228 // ---- Event -------------------------------------------------------------------------- 235 // ---- Event --------------------------------------------------------------------------
229 const ev = await createEvent({ sensor, source, obs, norm, prev, snapId, changeId, diff, heuristic, semantic }); 236 const ev = await createEvent({ sensor, source, obs, norm, prev, snapId, changeId, diff, heuristic, semantic });
@@ -444,7 +451,7 @@ async function handleMissing(sensor: Sensor, source: Source, obs: Observation, f
444 const sepOk = !missing.lastAt || obs.fetchedAt.getTime() - new Date(missing.lastAt).getTime() >= config.deletion.minSeparationMin * 60e3; 451 const sepOk = !missing.lastAt || obs.fetchedAt.getTime() - new Date(missing.lastAt).getTime() >= config.deletion.minSeparationMin * 60e3;
445 if (sepOk) missing.count += 1; 452 if (sepOk) missing.count += 1;
446 missing.lastAt = obs.fetchedAt.toISOString(); 453 missing.lastAt = obs.fetchedAt.toISOString();
447− if (sensor.lastSnapshotId && missing.count >= config.deletion.confirmations && !missing.removedEventAt) { 454+ if (sensor.lastSnapshotId && missing.count >= config.deletion.confirmations && !missing.removedEventAt && sensor.status !== "SHADOW") {
448 // Confirmed deletion: independent checks separated in time. 455 // Confirmed deletion: independent checks separated in time.
449 missing.removedEventAt = obs.fetchedAt.toISOString(); 456 missing.removedEventAt = obs.fetchedAt.toISOString();
450 const id = newId("evt"); 457 const id = newId("evt");
modified apps/engine/src/registry.ts +39 −29
@@ -18,38 +18,48 @@ export function loadSeeds(file = config.sourcesFile, dir = config.sourcesDir): S
18 return seeds; 18 return seeds;
19 } 19 }
20 20
21−export async function syncRegistry(seeds = loadSeeds()): Promise<{ sources: number; sensors: number }> { 21+/**
22− let nSensors = 0; 22+ * Upsert one organization (source row + organization entity + aliases + product entities). Shared by the YAML
23− for (const s of seeds) { 23+ * registry sync (`origin = seed`) and the Source Factory (`origin = factory`). Sensors are handled by the caller.
24− // Provenance: media/aggregators are third-party; everything else is the organization's own channel. 24+ */
25− const firstParty = s.first_party ?? !s.categories.some((c) => c === "news" || c === "media"); 25+export async function upsertSourceRecord(s: SourceSeed, opts: { origin?: "seed" | "import" | "factory"; sector?: string | null } = {}): Promise<void> {
26− const country = s.country ?? inferCountry(s.domain, s.categories); 26+ // Provenance: media/aggregators are third-party; everything else is the organization's own channel.
27− const language = s.language ?? inferLanguage(s.domain); 27+ const firstParty = s.first_party ?? !s.categories.some((c) => c === "news" || c === "media");
28− await db 28+ const country = s.country ?? inferCountry(s.domain, s.categories);
29− .insert(sources) 29+ const language = s.language ?? inferLanguage(s.domain);
30− .values({ id: s.id, name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, firstParty, country, language }) 30+ const origin = opts.origin ?? "seed";
31− .onConflictDoUpdate({ target: sources.id, set: { name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, firstParty, country, language, updatedAt: new Date() } }); 31+ const base = { name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, firstParty, country, language };
32+ await db
33+ .insert(sources)
34+ .values({ id: s.id, ...base, origin, sector: opts.sector ?? null })
35+ .onConflictDoUpdate({ target: sources.id, set: origin === "seed" ? { ...base, origin, updatedAt: new Date() } : { ...base, updatedAt: new Date(), ...(opts.sector ? { sector: opts.sector } : {}) } });
32 36
33− // Organization entity + aliases 37+ // Organization entity + aliases
34− const entId = `org_${s.id}`; 38+ const entId = `org_${s.id}`;
39+ await db
40+ .insert(entities)
41+ .values({ id: entId, name: s.name, type: s.entity_type, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, importance: sourceImportanceFromTier(s.tier, s.weight), categories: s.categories })
42+ .onConflictDoUpdate({ target: entities.id, set: { name: s.name, type: s.entity_type, domain: s.domain, description: s.description, importance: sourceImportanceFromTier(s.tier, s.weight), categories: s.categories } });
43+ await db.insert(sourceEntities).values({ sourceId: s.id, entityId: entId }).onConflictDoNothing();
44+ for (const alias of new Set([s.name, s.domain, s.domain.replace(/^www\./, ""), ...s.aliases])) {
45+ await db.insert(entityAliases).values({ alias: alias.toLowerCase(), entityId: entId }).onConflictDoNothing();
46+ }
47+ for (const p of s.products) {
48+ const pid = p.id ?? `prd_${s.id}_${slugify(p.name)}`;
35 await db 49 await db
36 .insert(entities) 50 .insert(entities)
37− .values({ id: entId, name: s.name, type: s.entity_type, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, importance: sourceImportanceFromTier(s.tier, s.weight), categories: s.categories }) 51+ .values({ id: pid, name: p.name, type: p.type, parentId: entId, importance: Math.max(30, sourceImportanceFromTier(s.tier, s.weight) - 10), categories: s.categories, domain: s.domain })
38− .onConflictDoUpdate({ target: entities.id, set: { name: s.name, type: s.entity_type, domain: s.domain, description: s.description, importance: sourceImportanceFromTier(s.tier, s.weight), categories: s.categories } }); 52+ .onConflictDoUpdate({ target: entities.id, set: { name: p.name, type: p.type, parentId: entId } });
39− await db.insert(sourceEntities).values({ sourceId: s.id, entityId: entId }).onConflictDoNothing(); 53+ await db.insert(sourceEntities).values({ sourceId: s.id, entityId: pid }).onConflictDoNothing();
40− for (const alias of new Set([s.name, s.domain, s.domain.replace(/^www\./, ""), ...s.aliases])) { 54+ for (const alias of new Set([p.name, ...p.aliases])) await db.insert(entityAliases).values({ alias: alias.toLowerCase(), entityId: pid }).onConflictDoNothing();
41− await db.insert(entityAliases).values({ alias: alias.toLowerCase(), entityId: entId }).onConflictDoNothing(); 55+ await db.execute(sql`insert into entity_relations (from_id, relation, to_id) values (${entId}, 'owns', ${pid}) on conflict do nothing`);
42− } 56+ }
43− for (const p of s.products) { 57+}
44− const pid = p.id ?? `prd_${s.id}_${slugify(p.name)}`; 58+
45− await db 59+export async function syncRegistry(seeds = loadSeeds()): Promise<{ sources: number; sensors: number }> {
46− .insert(entities) 60+ let nSensors = 0;
47− .values({ id: pid, name: p.name, type: p.type, parentId: entId, importance: Math.max(30, sourceImportanceFromTier(s.tier, s.weight) - 10), categories: s.categories, domain: s.domain }) 61+ for (const s of seeds) {
48− .onConflictDoUpdate({ target: entities.id, set: { name: p.name, type: p.type, parentId: entId } }); 62+ await upsertSourceRecord(s, { origin: "seed" });
49− await db.insert(sourceEntities).values({ sourceId: s.id, entityId: pid }).onConflictDoNothing();
50− for (const alias of new Set([p.name, ...p.aliases])) await db.insert(entityAliases).values({ alias: alias.toLowerCase(), entityId: pid }).onConflictDoNothing();
51− await db.execute(sql`insert into entity_relations (from_id, relation, to_id) values (${entId}, 'owns', ${pid}) on conflict do nothing`);
52− }
53 63
54 const seedIds: string[] = []; 64 const seedIds: string[] = [];
55 for (const sen of s.sensors) { 65 for (const sen of s.sensors) {
added apps/web/src/app/ops/factory-panel.tsx +205 −0
@@ -0,0 +1,205 @@
1+"use client";
2+
3+import Link from "next/link";
4+import { useEffect, useState } from "react";
5+import { Chip, Empty, HealthPill, Panel, Skeleton, Sparkline, Stat, Table, Td } from "@/components/ui";
6+import { fmtInt, relTime, utcDateTime } from "@/lib/format";
7+
8+/** Shape of GET /api/v1/admin/factory (internal). */
9+interface FactoryReport {
10+ seeds: Record<string, number>;
11+ candidates: Record<string, number>;
12+ shadow: { total: number; up: number; unhealthy: number; ready: number; raw_changes: number };
13+ shadow_by_kind: { kind_class: string; n: number }[];
14+ daily: { day: string; seeds_processed: number; requests: number; candidates: number; shadow_created: number; accepted: number; rejected: number; blocked: number }[];
15+ sectors: { sector: string; seeds: number; queued: number; discovering: number; discovered: number; blocked: number; errors: number; shadow: number; accepted: number; rejected: number }[];
16+ recent_seeds: { id: string; name: string; domain: string; sector: string | null; status: string; candidates: number; shadow: number; accepted: number; rejected: number; last_error: string | null; discovered_at: string | null; source_id: string | null }[];
17+ heartbeat: { at?: string; processed?: number; queued?: number; concurrency?: number; idle?: boolean; lastBatch?: { seed: string; status: string; candidates: number; shadow: number; requests: number; ms: number }[] } | null;
18+ generated_at: string;
19+}
20+interface Candidate {
21+ id: string;
22+ source_id: string;
23+ url: string;
24+ name: string | null;
25+ kind_class: string | null;
26+ connector: string | null;
27+ tier: string | null;
28+ status: string;
29+ score_value: number | null;
30+ reason: string | null;
31+ shadow_sensor_id: string | null;
32+ shadow_health: string | null;
33+ shadow_runs: number | null;
34+ shadow_changes: number | null;
35+ updated_at: string;
36+}
37+
38+export function FactoryPanel({ fetcher, tick }: { fetcher: <T>(path: string, init?: RequestInit) => Promise<T>; tick: number }) {
39+ const [d, setD] = useState<FactoryReport | null>(null);
40+ const [err, setErr] = useState<string | null>(null);
41+ const [cands, setCands] = useState<Candidate[] | null>(null);
42+ const [candStatus, setCandStatus] = useState("shadow");
43+ const [busy, setBusy] = useState<string | null>(null);
44+
45+ useEffect(() => {
46+ let cancelled = false;
47+ fetcher<FactoryReport>("/api/v1/admin/factory")
48+ .then((r) => !cancelled && (setD(r), setErr(null)))
49+ .catch((e: Error) => !cancelled && setErr(e.message));
50+ return () => {
51+ cancelled = true;
52+ };
53+ }, [fetcher, tick]);
54+ useEffect(() => {
55+ let cancelled = false;
56+ fetcher<{ items: Candidate[] }>(`/api/v1/admin/factory/candidates?status=${encodeURIComponent(candStatus)}&limit=60`)
57+ .then((r) => !cancelled && setCands(r.items))
58+ .catch(() => !cancelled && setCands([]));
59+ return () => {
60+ cancelled = true;
61+ };
62+ }, [fetcher, tick, candStatus]);
63+
64+ const act = async (path: string): Promise<void> => {
65+ setBusy(path);
66+ try {
67+ await fetcher(path, { method: "POST", body: "{}" });
68+ const r = await fetcher<{ items: Candidate[] }>(`/api/v1/admin/factory/candidates?status=${encodeURIComponent(candStatus)}&limit=60`);
69+ setCands(r.items);
70+ } catch (e) {
71+ setErr((e as Error).message);
72+ } finally {
73+ setBusy(null);
74+ }
75+ };
76+
77+ if (err && !d) return <Panel title="Source Factory"><Empty>{err}</Empty></Panel>;
78+ if (!d) return <Skeleton className="h-40 w-full" />;
79+ const hb = d.heartbeat;
80+ const stale = hb?.at ? Date.now() - new Date(hb.at).getTime() > 5 * 60_000 : true;
81+ const daily = [...d.daily].reverse();
82+ const seedsTotal = Object.values(d.seeds).reduce((a, b) => a + b, 0);
83+ return (
84+ <div className="flex flex-col gap-4">
85+ <div className="panel grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-4 xl:grid-cols-8 xl:divide-y-0">
86+ <Stat label="Factory" value={hb ? (stale ? "STALE" : hb.idle ? "IDLE" : "LIVE") : "NO HEARTBEAT"} tone={hb && !stale ? "signal" : "warn"} hint={hb?.at ? `${relTime(hb.at)} · ${hb.processed ?? 0} processed` : "Redis ws:factory:status empty"} />
87+ <Stat label="Seeds" value={fmtInt(seedsTotal)} hint={`${fmtInt(d.seeds.queued ?? 0)} queued · ${fmtInt(d.seeds.discovering ?? 0)} discovering`} />
88+ <Stat label="Discovered" value={fmtInt((d.seeds.discovered ?? 0) + (d.seeds.done ?? 0))} hint={`${fmtInt(d.seeds.blocked ?? 0)} blocked · ${fmtInt(d.seeds.error ?? 0)} errors`} />
89+ <Stat label="Candidates" value={fmtInt(Object.values(d.candidates).reduce((a, b) => a + b, 0))} hint={`${fmtInt(d.candidates.candidate ?? 0)} waiting · ${fmtInt(d.candidates.duplicate ?? 0)} dup`} />
90+ <Stat label="Shadow" value={fmtInt(d.shadow.total)} tone="info" hint={`${fmtInt(d.shadow.ready)} ready · ${fmtInt(d.shadow.unhealthy)} unhealthy`} />
91+ <Stat label="Accepted" value={fmtInt(d.candidates.accepted ?? 0)} tone="signal" hint="production sensors" />
92+ <Stat label="Rejected" value={fmtInt(d.candidates.rejected ?? 0)} hint="errors · noise · duplicates" />
93+ <Stat label="Requests today" value={fmtInt(daily.at(-1)?.requests ?? 0)} hint={`${fmtInt(daily.at(-1)?.seeds_processed ?? 0)} organizations`} />
94+ </div>
95+
96+ <div className="grid grid-cols-1 gap-4 xl:grid-cols-3">
97+ <Panel title="Funnel · 30 days" dense>
98+ {daily.length ? (
99+ <div className="grid grid-cols-1 gap-3 p-3 sm:grid-cols-2">
100+ {(["seeds_processed", "candidates", "shadow_created", "accepted"] as const).map((k) => (
101+ <div key={k}>
102+ <div className="mb-1 flex items-baseline justify-between"><span className="label">{k.replace(/_/g, " ")}</span><span className="font-mono text-[12px] tabular">{fmtInt(daily.reduce((a, r) => a + Number(r[k] ?? 0), 0))}</span></div>
103+ <Sparkline values={daily.map((r) => Number(r[k] ?? 0))} tone={k === "accepted" ? "signal" : k === "shadow_created" ? "info" : "muted"} responsive />
104+ </div>
105+ ))}
106+ </div>
107+ ) : (
108+ <Empty>No factory activity yet.</Empty>
109+ )}
110+ </Panel>
111+ <Panel title="By sector" dense className="xl:col-span-2">
112+ {d.sectors.length ? (
113+ <div className="max-h-72 overflow-auto">
114+ <Table head={["Sector", "Seeds", "Queued", "Discovered", "Blocked", "Errors", "Shadow", "Accepted", "Rejected"]}>
115+ {d.sectors.map((s) => (
116+ <tr key={s.sector}>
117+ <Td><Link href={`/coverage/${s.sector}`} className="hover:underline">{s.sector}</Link></Td>
118+ <Td mono>{fmtInt(s.seeds)}</Td>
119+ <Td mono className="text-fg-subtle">{fmtInt(s.queued + s.discovering)}</Td>
120+ <Td mono>{fmtInt(s.discovered)}</Td>
121+ <Td mono className={s.blocked ? "text-warn" : "text-fg-subtle"}>{fmtInt(s.blocked)}</Td>
122+ <Td mono className={s.errors ? "text-danger" : "text-fg-subtle"}>{fmtInt(s.errors)}</Td>
123+ <Td mono className="text-info">{fmtInt(s.shadow)}</Td>
124+ <Td mono className="text-signal">{fmtInt(s.accepted)}</Td>
125+ <Td mono className="text-fg-subtle">{fmtInt(s.rejected)}</Td>
126+ </tr>
127+ ))}
128+ </Table>
129+ </div>
130+ ) : (
131+ <Empty>No seeds yet — run <code>cli.ts factory seed</code> or POST /api/v1/admin/factory/seeds.</Empty>
132+ )}
133+ </Panel>
134+ </div>
135+
136+ <div className="grid grid-cols-1 gap-4 xl:grid-cols-2">
137+ <Panel title="Recently processed organizations" dense>
138+ {d.recent_seeds.length ? (
139+ <div className="max-h-80 overflow-auto">
140+ <Table head={["Organization", "Sector", "Status", "Cand.", "Shadow", "Acc.", "Rej.", "When"]}>
141+ {d.recent_seeds.map((s) => (
142+ <tr key={s.id}>
143+ <Td>{s.source_id ? <Link href={`/source/${s.source_id}`} className="hover:underline">{s.name}</Link> : s.name}<div className="font-mono text-[10.5px] text-fg-subtle">{s.domain}{s.last_error ? ` · ${s.last_error}` : ""}</div></Td>
144+ <Td mono className="text-fg-subtle">{s.sector ?? "—"}</Td>
145+ <Td><Chip tone={s.status === "discovered" ? "ok" : s.status === "blocked" ? "warn" : s.status === "error" ? "danger" : "default"}>{s.status}</Chip></Td>
146+ <Td mono>{s.candidates}</Td>
147+ <Td mono className="text-info">{s.shadow}</Td>
148+ <Td mono className="text-signal">{s.accepted}</Td>
149+ <Td mono className="text-fg-subtle">{s.rejected}</Td>
150+ <Td mono className="text-fg-subtle"><span title={s.discovered_at ? utcDateTime(s.discovered_at) : ""}>{s.discovered_at ? relTime(s.discovered_at) : "—"}</span></Td>
151+ </tr>
152+ ))}
153+ </Table>
154+ </div>
155+ ) : (
156+ <Empty>Nothing processed yet.</Empty>
157+ )}
158+ </Panel>
159+ <Panel
160+ title="Candidates"
161+ dense
162+ action={
163+ <span className="flex gap-1">
164+ {["shadow", "candidate", "accepted", "rejected", "duplicate"].map((s) => (
165+ <button key={s} type="button" onClick={() => setCandStatus(s)} className={`rounded px-1.5 py-0.5 font-mono text-[10.5px] ${candStatus === s ? "bg-panel-2 text-fg" : "text-fg-subtle hover:text-fg"}`}>{s}</button>
166+ ))}
167+ </span>
168+ }
169+ >
170+ {cands === null ? (
171+ <Skeleton className="h-40 w-full" />
172+ ) : cands.length === 0 ? (
173+ <Empty>No candidate with status {candStatus}.</Empty>
174+ ) : (
175+ <div className="max-h-80 overflow-auto">
176+ <Table head={["Sensor", "Kind", "Score", "Shadow", "Reason", ""]}>
177+ {cands.map((c) => (
178+ <tr key={c.id}>
179+ <Td><Link href={`/source/${c.source_id}`} className="hover:underline">{c.source_id}</Link><div className="max-w-[28ch] truncate font-mono text-[10.5px] text-fg-subtle" title={c.url}>{c.name ?? ""} · {c.url}</div></Td>
180+ <Td mono className="text-fg-subtle">{c.kind_class ?? "—"}<div>{c.connector} {c.tier}</div></Td>
181+ <Td mono>{c.score_value?.toFixed(2) ?? "—"}</Td>
182+ <Td>{c.shadow_sensor_id ? <span className="flex flex-col gap-0.5"><HealthPill health={c.shadow_health ?? "UP"} /><span className="font-mono text-[10.5px] text-fg-subtle">{c.shadow_runs ?? 0} checks · {c.shadow_changes ?? 0} changes</span></span> : <span className="text-fg-subtle">—</span>}</Td>
183+ <Td className="text-[11px] text-fg-subtle"><span className="block max-w-[24ch] truncate" title={c.reason ?? ""}>{c.reason ?? "—"}</span></Td>
184+ <Td>
185+ {c.status === "shadow" && c.shadow_sensor_id && (
186+ <span className="flex gap-1">
187+ <button type="button" disabled={busy !== null} onClick={() => void act(`/api/v1/admin/factory/shadow/${c.shadow_sensor_id}/accept`)} className="rounded border border-line px-1.5 py-0.5 text-[10.5px] text-signal hover:border-line-strong disabled:opacity-50">accept</button>
188+ <button type="button" disabled={busy !== null} onClick={() => void act(`/api/v1/admin/factory/shadow/${c.shadow_sensor_id}/reject`)} className="rounded border border-line px-1.5 py-0.5 text-[10.5px] text-danger hover:border-line-strong disabled:opacity-50">reject</button>
189+ </span>
190+ )}
191+ {c.status === "candidate" && c.connector && (
192+ <button type="button" disabled={busy !== null} onClick={() => void act(`/api/v1/admin/factory/candidates/${c.id}/shadow`)} className="rounded border border-line px-1.5 py-0.5 text-[10.5px] text-info hover:border-line-strong disabled:opacity-50">shadow</button>
193+ )}
194+ </Td>
195+ </tr>
196+ ))}
197+ </Table>
198+ </div>
199+ )}
200+ </Panel>
201+ </div>
202+ {hb?.lastBatch?.length ? <p className="text-[11px] text-fg-subtle">last batch: {hb.lastBatch.map((b) => `${b.seed} ${b.status} (${b.candidates} cand · ${b.shadow} shadow · ${b.requests} req · ${Math.round(b.ms / 1000)} s)`).join(" · ")}</p> : null}
203+ </div>
204+ );
205+}
modified apps/web/src/app/ops/ops.tsx +6 −1
@@ -2,11 +2,12 @@
2 2
3 import { KeyRound, LogOut, Play, RefreshCw, ShieldAlert } from "lucide-react"; 3 import { KeyRound, LogOut, Play, RefreshCw, ShieldAlert } from "lucide-react";
4 import Link from "next/link"; 4 import Link from "next/link";
5−import { useEffect, useMemo, useState, type ReactNode } from "react"; 5+import { useCallback, useEffect, useMemo, useState, type ReactNode } from "react";
6 import { SENSOR_TYPES, TIERS } from "@websensor/core/client"; 6 import { SENSOR_TYPES, TIERS } from "@websensor/core/client";
7 import { useMounted } from "@/components/theme"; 7 import { useMounted } from "@/components/theme";
8 import { Chip, Empty, HealthPill, Panel, Skeleton, Sparkline, Stat, Table, Td } from "@/components/ui"; 8 import { Chip, Empty, HealthPill, Panel, Skeleton, Sparkline, Stat, Table, Td } from "@/components/ui";
9 import { clientApiBase } from "@/lib/api"; 9 import { clientApiBase } from "@/lib/api";
10+import { FactoryPanel } from "./factory-panel";
10 import { fmtBytes, fmtInt, fmtMs, fmtPct, relTime, untilTime, utcDateTime, utcTime } from "@/lib/format"; 11 import { fmtBytes, fmtInt, fmtMs, fmtPct, relTime, untilTime, utcDateTime, utcTime } from "@/lib/format";
11 12
12 const KEY = "ws_admin"; 13 const KEY = "ws_admin";
@@ -122,6 +123,7 @@ export function Ops() {
122 setRefreshing(true); 123 setRefreshing(true);
123 setTick((x) => x + 1); 124 setTick((x) => x + 1);
124 }; 125 };
126+ const factoryFetcher = useCallback(<T,>(path: string, init: RequestInit = {}): Promise<T> => adminFetch<T>(path, init, token ?? ""), [token]);
125 const submitToken = async (): Promise<void> => { 127 const submitToken = async (): Promise<void> => {
126 const t = input.trim(); 128 const t = input.trim();
127 if (!t) return; 129 if (!t) return;
@@ -187,6 +189,9 @@ export function Ops() {
187 189
188 {data ? <Dashboard d={data} /> : <Skeleton className="h-64 w-full" />} 190 {data ? <Dashboard d={data} /> : <Skeleton className="h-64 w-full" />}
189 191
192+ <h2 className="mt-2 text-[13px] font-semibold tracking-tight">Source Factory <span className="font-normal text-fg-subtle">· discovery → shadow → acceptance</span></h2>
193+ <FactoryPanel fetcher={factoryFetcher} tick={tick} />
194+
190 <Tools connectors={data?.connectors ?? []} onDone={refresh} /> 195 <Tools connectors={data?.connectors ?? []} onDone={refresh} />
191 </div> 196 </div>
192 ); 197 );
modified apps/web/src/app/page.tsx +2 −0
@@ -1,6 +1,7 @@
1 import Link from "next/link"; 1 import Link from "next/link";
2 import { LiveFeed } from "@/components/live-feed"; 2 import { LiveFeed } from "@/components/live-feed";
3 import { LiveStrip } from "@/components/live-strip"; 3 import { LiveStrip } from "@/components/live-strip";
4+import { CoverageRail } from "@/components/coverage-rail";
4 import { AnomalyRail, BreakingRail, ClustersPanel, SilentRail, TrendingPanel } from "@/components/rail"; 5 import { AnomalyRail, BreakingRail, ClustersPanel, SilentRail, TrendingPanel } from "@/components/rail";
5 import { EventRow } from "@/components/event-row"; 6 import { EventRow } from "@/components/event-row";
6 import { Empty, Panel } from "@/components/ui"; 7 import { Empty, Panel } from "@/components/ui";
@@ -39,6 +40,7 @@ export default async function HomePage() {
39 <BreakingRail items={breaking?.breaking_now ?? []} developing={breaking?.developing ?? []} /> 40 <BreakingRail items={breaking?.breaking_now ?? []} developing={breaking?.developing ?? []} />
40 <TrendingPanel items={trending.items} /> 41 <TrendingPanel items={trending.items} />
41 <AnomalyRail items={explore?.unusual_activity ?? []} /> 42 <AnomalyRail items={explore?.unusual_activity ?? []} />
43+ <CoverageRail />
42 </aside> 44 </aside>
43 </div> 45 </div>
44 <div className="mt-4 grid grid-cols-1 gap-4 lg:grid-cols-3 [&>*]:min-w-0"> 46 <div className="mt-4 grid grid-cols-1 gap-4 lg:grid-cols-3 [&>*]:min-w-0">
modified apps/web/src/components/command-palette.tsx +1 −0
@@ -33,6 +33,7 @@ const NAV: { label: string; href: string; hint: string; icon: ReactNode }[] = [
33 { label: "Entities", href: "/entities", hint: "organizations, products, models", icon: <Building2 className="size-3.5" /> }, 33 { label: "Entities", href: "/entities", hint: "organizations, products, models", icon: <Building2 className="size-3.5" /> },
34 { label: "Countries", href: "/country", hint: "national desks", icon: <Globe2 className="size-3.5" /> }, 34 { label: "Countries", href: "/country", hint: "national desks", icon: <Globe2 className="size-3.5" /> },
35 { label: "Sources", href: "/sources", hint: "monitored organizations", icon: <Building2 className="size-3.5" /> }, 35 { label: "Sources", href: "/sources", hint: "monitored organizations", icon: <Building2 className="size-3.5" /> },
36+ { label: "Coverage", href: "/coverage", hint: "global observation coverage score by sector", icon: <Globe2 className="size-3.5" /> },
36 { label: "Watchlists", href: "/watchlists", hint: "your entities, sources, keywords", icon: <Layers className="size-3.5" /> }, 37 { label: "Watchlists", href: "/watchlists", hint: "your entities, sources, keywords", icon: <Layers className="size-3.5" /> },
37 { label: "Alerts", href: "/alerts", hint: "rules · webhooks", icon: <Siren className="size-3.5" /> }, 38 { label: "Alerts", href: "/alerts", hint: "rules · webhooks", icon: <Siren className="size-3.5" /> },
38 { label: "Bookmarks", href: "/bookmarks", hint: "saved events", icon: <Layers className="size-3.5" /> }, 39 { label: "Bookmarks", href: "/bookmarks", hint: "saved events", icon: <Layers className="size-3.5" /> },
added apps/web/src/components/coverage-rail.tsx +27 −0
@@ -0,0 +1,27 @@
1+import Link from "next/link";
2+import { api } from "@/lib/api";
3+import { fmtInt } from "@/lib/format";
4+import { Bar, Empty, Panel } from "./ui";
5+
6+/** Homepage rail: Global Observation Coverage Score by sector (server component). */
7+export async function CoverageRail() {
8+ const c = await api.coverage();
9+ return (
10+ <Panel title={<span>Observation coverage {c ? <span className={`font-mono ${c.global_score >= 60 ? "text-signal" : "text-high"}`}>{c.global_score.toFixed(1)}</span> : null}</span>} dense action={<Link href="/coverage" className="text-[11px] text-fg-subtle hover:text-fg">coverage →</Link>}>
11+ {!c || !c.sectors.length ? (
12+ <Empty>Coverage universes are loading.</Empty>
13+ ) : (
14+ <ul className="px-3 py-1.5 text-[12px]">
15+ {c.sectors.slice(0, 10).map((s) => (
16+ <li key={s.sector} className="grid grid-cols-[7.5rem_1fr_2.5rem] items-center gap-2 py-0.5">
17+ <Link href={`/coverage/${s.sector}`} className="truncate text-fg-muted hover:text-fg" title={`${fmtInt(s.covered)} / ${fmtInt(s.members)} observed`}>{s.label}</Link>
18+ <Bar value={s.score} tone={s.score >= 80 ? "signal" : s.score >= 60 ? "high" : s.score >= 40 ? "mid" : "info"} />
19+ <span className="text-right font-mono tabular text-fg-muted">{Math.round(s.score)}%</span>
20+ </li>
21+ ))}
22+ <li className="pt-1 text-[10.5px] text-fg-subtle">{fmtInt(c.covered)} / {fmtInt(c.members)} organizations · {fmtInt(c.monitored.shadow)} sensors in shadow</li>
23+ </ul>
24+ )}
25+ </Panel>
26+ );
27+}
modified apps/web/src/components/nav.tsx +2 −0
@@ -18,6 +18,7 @@ const NAV = [
18 ["/explore", "Explore"], 18 ["/explore", "Explore"],
19 ["/entities", "Entities"], 19 ["/entities", "Entities"],
20 ["/sources", "Sources"], 20 ["/sources", "Sources"],
21+ ["/coverage", "Coverage"],
21 ["/watchlists", "Watchlists"], 22 ["/watchlists", "Watchlists"],
22 ["/alerts", "Alerts"], 23 ["/alerts", "Alerts"],
23 ] as const; 24 ] as const;
@@ -82,6 +83,7 @@ export function MobileNav() {
82 ["/silent", "Silent changes", LayoutGrid], 83 ["/silent", "Silent changes", LayoutGrid],
83 ["/entities", "Entities", Globe2], 84 ["/entities", "Entities", Globe2],
84 ["/sources", "Sources", Globe2], 85 ["/sources", "Sources", Globe2],
86+ ["/coverage", "Coverage", Globe2],
85 ["/country", "Countries", Globe2], 87 ["/country", "Countries", Globe2],
86 ["/alerts", "Alerts", Bell], 88 ["/alerts", "Alerts", Bell],
87 ["/bookmarks", "Bookmarks", Bookmark], 89 ["/bookmarks", "Bookmarks", Bookmark],
modified apps/web/src/components/ui.tsx +2 −2
@@ -80,10 +80,10 @@ export function EvidenceTag({ label }: { label: string | null | undefined }) {
80 80
81 export function HealthPill({ health }: { health: string | null | undefined }) { 81 export function HealthPill({ health }: { health: string | null | undefined }) {
82 const h = (health ?? "UP").toUpperCase(); 82 const h = (health ?? "UP").toUpperCase();
83− const tone: Tone = h === "UP" || h === "ACTIVE" || h === "VALIDATED" ? "ok" : h === "DEGRADED" || h === "RATE_LIMITED" || h === "PENDING" ? "warn" : h === "ERROR" ? "danger" : "default"; 83+ const tone: Tone = h === "UP" || h === "ACTIVE" || h === "VALIDATED" ? "ok" : h === "DEGRADED" || h === "RATE_LIMITED" || h === "PENDING" ? "warn" : h === "ERROR" ? "danger" : h === "SHADOW" ? "info" : "default";
84 return ( 84 return (
85 <Chip tone={tone} className="font-mono"> 85 <Chip tone={tone} className="font-mono">
86− <span className={`inline-block size-1.5 rounded-full ${tone === "ok" ? "bg-ok" : tone === "danger" ? "bg-danger" : tone === "default" ? "bg-low" : "bg-warn"}`} /> {h} 86+ <span className={`inline-block size-1.5 rounded-full ${tone === "ok" ? "bg-ok" : tone === "danger" ? "bg-danger" : tone === "default" ? "bg-low" : tone === "info" ? "bg-info" : "bg-warn"}`} /> {h}
87 </Chip> 87 </Chip>
88 ); 88 );
89 } 89 }
modified apps/web/src/lib/api.ts +63 −0
@@ -196,7 +196,68 @@ export interface EventsPage {
196 nextCursor: string | null; 196 nextCursor: string | null;
197 } 197 }
198 198
199+export interface CoverageSectorRow {
200+ sector: string;
201+ label: string;
202+ description: string | null;
203+ weight: number;
204+ members: number;
205+ covered: number;
206+ shadow_members: number;
207+ queued: number;
208+ sensors: number;
209+ breadth: number;
210+ depth: number;
211+ score: number;
212+}
213+export interface CoverageSummary {
214+ global_score: number;
215+ members: number;
216+ covered: number;
217+ sensors_matched: number;
218+ sectors: CoverageSectorRow[];
219+ monitored: { sources: number; sensors: number; shadow: number; countries: number };
220+ factory: Record<string, number>;
221+ method: string;
222+ issues: string[];
223+ generated_at: string;
224+}
225+export interface CoverageMemberRow {
226+ name: string;
227+ domain: string;
228+ country: string | null;
229+ importance: number;
230+ universe: string;
231+ covered: boolean;
232+ sensors: number;
233+ shadow: number;
234+ source_id: string | null;
235+ seed_status: string | null;
236+ depth: number;
237+}
238+export interface CoverageUniverseRow {
239+ key: string;
240+ label: string;
241+ description: string | null;
242+ provenance: string | null;
243+ members: number;
244+ covered: number;
245+ sensors: number;
246+ breadth: number;
247+ depth: number;
248+ score: number;
249+ items: CoverageMemberRow[];
250+}
251+export interface CoverageSectorDetail extends CoverageSectorRow {
252+ categories: string[];
253+ universes: CoverageUniverseRow[];
254+ by_country: { country: string; members: number; covered: number }[];
255+ generated_at: string;
256+}
257+
199 export interface Stats { 258 export interface Stats {
259+ sensors_shadow?: number;
260+ sources_factory?: number;
200 checks_per_min?: number; 261 checks_per_min?: number;
201 events_per_min?: number; 262 events_per_min?: number;
202 events_1h?: number; 263 events_1h?: number;
@@ -808,5 +869,7 @@ export const api = {
808 compare: (a: string, b: string) => getJson<CompareResult | null>(`/api/v1/snapshots/compare?a=${encodeURIComponent(a)}&b=${encodeURIComponent(b)}`, null), 869 compare: (a: string, b: string) => getJson<CompareResult | null>(`/api/v1/snapshots/compare?a=${encodeURIComponent(a)}&b=${encodeURIComponent(b)}`, null),
809 change: (id: string) => getJson<{ change: ChangeRow & { diff: DiffSummary; sensor_url?: string; sensor_name?: string; source_id?: string }; unified: string | null } | null>(`/api/v1/changes/${encodeURIComponent(id)}`, null), 870 change: (id: string) => getJson<{ change: ChangeRow & { diff: DiffSummary; sensor_url?: string; sensor_name?: string; source_id?: string }; unified: string | null } | null>(`/api/v1/changes/${encodeURIComponent(id)}`, null),
810 health: () => getJson<HealthReport | null>("/api/v1/health/connectors", null), 871 health: () => getJson<HealthReport | null>("/api/v1/health/connectors", null),
872+ coverage: () => getJson<CoverageSummary | null>("/api/v1/coverage", null, { revalidate: 60 }),
873+ coverageSector: (sector: string) => getJson<CoverageSectorDetail | null>(`/api/v1/coverage/${encodeURIComponent(sector)}`, null, { revalidate: 60 }),
811 search: (q: string) => getJson<SearchResult>(`/api/v1/search?q=${encodeURIComponent(q)}`, { query: q, events: [], entities: [], sources: [], urls: [] }), 874 search: (q: string) => getJson<SearchResult>(`/api/v1/search?q=${encodeURIComponent(q)}`, { query: q, events: [], entities: [], sources: [], urls: [] }),
812 }; 875 };
modified deploy/README.md +17 −1
@@ -1,6 +1,6 @@
1 # Deploying WebSensor on MacLustr 1 # Deploying WebSensor on MacLustr
2 2
3−WebSensor runs as three PM2 processes on one node (M4M64b), published through the MacLustr Tunnel (WireGuard + 3+WebSensor runs as four PM2 processes on one node (M4M64b), published through the MacLustr Tunnel (WireGuard +
4 Caddy on the OVH gateway BHS64, `www.websensor.io`; ngrok was retired on 2026-09-10), orchestrated by `mld` from the 4 Caddy on the OVH gateway BHS64, `www.websensor.io`; ngrok was retired on 2026-09-10), orchestrated by `mld` from the
5 M1M32 gateway. 5 M1M32 gateway.
6 6
@@ -9,6 +9,7 @@ Internet → www.websensor.io → BHS64 (Caddy TLS) → wg1 → websensor-api (:
9 ├── /api/v1/* REST + WebSocket /api/v1/live 9 ├── /api/v1/* REST + WebSocket /api/v1/live
10 └── /* reverse proxy → websensor-web (Next.js :8261, loopback) 10 └── /* reverse proxy → websensor-web (Next.js :8261, loopback)
11 websensor-engine (scheduler + pipeline, Prometheus :8262) → PostgreSQL 17 `websensor` + Redis (loopback) 11 websensor-engine (scheduler + pipeline, Prometheus :8262) → PostgreSQL 17 `websensor` + Redis (loopback)
12+websensor-factory (Source Factory: coverage seeds → deep discovery → shadow sensors → acceptance; heartbeat Redis ws:factory:status)
12 Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by mld (excluded `data/`). 13 Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by mld (excluded `data/`).
13 ``` 14 ```
14 15
@@ -30,6 +31,21 @@ Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by
30 - WebSocket upgrades traverse Caddy without configuration; the API sends a heartbeat every 25 s and clients 31 - WebSocket upgrades traverse Caddy without configuration; the API sends a heartbeat every 25 s and clients
31 replay missed events with `{"since": lastSid}` after a reconnection. 32 replay missed events with `{"since": lastSid}` after a reconnection.
32 33
34+## Source Factory (0.3)
35+- Seeds: `config/coverage/*.yaml` (universes) and `config/factory/seeds/*.yaml`; the post-sync hook runs
36+ `cli.ts factory seed --mode=hinted` (members without a monitored domain + members with hints). `--mode=all --requeue`
37+ re-discovers everything; `factory requeue` re-queues blocked/errored seeds.
38+- Process: `pm2 logs websensor-factory`; `cli.ts factory stats`; `/ops` → Source Factory panel (funnel, sectors,
39+ candidates, manual accept/reject); `GET /api/v1/admin/factory`, `/admin/factory/seeds`, `/admin/factory/candidates`,
40+ `POST /admin/factory/seeds` (bulk), `POST /admin/factory/shadow/:sensorId/accept|reject`.
41+- Tuning (env on the factory process): `WS_FACTORY_CONCURRENCY` (6), `WS_FACTORY_BUDGET` (90 requests/org),
42+ `WS_FACTORY_MIN_SCORE` (0.35), `WS_FACTORY_SHADOW_MIN_CHECKS` (5), `WS_FACTORY_SHADOW_MIN_HOURS` (24),
43+ `WS_FACTORY_SHADOW_MAX_HOURS` (120). `GITHUB_TOKEN` (optional) improves repository ranking.
44+- Shadow sensors are polled by the engine at priority 3; raise `WS_FETCH_CONCURRENCY` on the engine when the shadow
45+ population grows (32 at 0.3). `cli.ts factory evaluate [--force]` runs the decision pass by hand;
46+ `cli.ts factory export --sector=x > config/sources.d/6x-factory-x.yaml` graduates accepted sensors into the registry.
47+- Coverage: `GET /api/v1/coverage`, `/coverage` page; universes validated with `apps/engine/src/coverage-validate.ts --dns`.
48+
33 ## Operations 49 ## Operations
34 - Logs: `mld logs websensor` or `ssh M4M64b pm2 logs websensor-engine`. 50 - Logs: `mld logs websensor` or `ssh M4M64b pm2 logs websensor-engine`.
35 - Registry edits: change `config/sources.yaml`, redeploy (sync runs in post-sync) or 51 - Registry edits: change `config/sources.yaml`, redeploy (sync runs in post-sync) or
modified deploy/websensor.mld.json.example +194 −123
@@ -1,128 +1,199 @@
1 { 1 {
2− "app": "websensor", 2+ "app": "websensor",
3− "label": "WebSensor — Detect What Changed. Know Why It Matters.", 3+ "label": "WebSensor — Detect What Changed. Know Why It Matters.",
4− "domain": "www.websensor.io", 4+ "domain": "www.websensor.io",
5− "port": 8260, 5+ "port": 8260,
6− "health_path": "/api/ready", 6+ "health_path": "/api/ready",
7− "dir": "~/apps/websensor", 7+ "dir": "~/apps/websensor",
8− "extra_paths": [], 8+ "extra_paths": [],
9− "sync_excludes": [ 9+ "sync_excludes": [
10− "node_modules/", 10+ "node_modules/",
11− ".next/", 11+ ".next/",
12− ".git/", 12+ ".git/",
13− ".env", 13+ ".env",
14− ".env.*", 14+ ".env.*",
15− "!.env.example", 15+ "!.env.example",
16− "*.tsbuildinfo", 16+ "*.tsbuildinfo",
17− ".turbo/", 17+ ".turbo/",
18− "logs/", 18+ "logs/",
19− "tmp/", 19+ "tmp/",
20− "data/", 20+ "data/",
21− "coverage/", 21+ "coverage/",
22− ".DS_Store", 22+ ".DS_Store",
23− ".claude/", 23+ ".claude/",
24− "qa/", 24+ "qa/",
25− "apps/web/.next/", 25+ "apps/web/.next/",
26− "apps/web/next-env.d.ts", 26+ "apps/web/next-env.d.ts",
27− "deploy/*.mld.json" 27+ "deploy/*.mld.json"
28+ ],
29+ "requires": {
30+ "runtimes": [
31+ "pm2",
32+ "node",
33+ "pnpm",
34+ "postgresql@17",
35+ "redis"
36+ ],
37+ "ram_gb": 8,
38+ "ports": [
39+ 8260,
40+ 8261,
41+ 8262
42+ ]
43+ },
44+ "ram_mb_observed": 1500,
45+ "size_mb": 40,
46+ "placement": {
47+ "pin": "M4M64b",
48+ "prefer": null,
49+ "avoid": [
50+ "M3U96b",
51+ "M1M32"
28 ], 52 ],
29− "requires": { 53+ "reason": "PostgreSQL 17 + Redis + pnpm présents ; 64 Go RAM ; M3U96a déjà chargé (25 processus PM2)"
30− "runtimes": ["pm2", "ngrok", "node", "pnpm", "postgresql@17", "redis"], 54+ },
31− "ram_gb": 6, 55+ "processes": [
32− "ports": [8260, 8261, 8262] 56+ {
57+ "name": "websensor-engine",
58+ "manager": "pm2",
59+ "script": "/opt/homebrew/bin/node",
60+ "args": [
61+ "node_modules/tsx/dist/cli.mjs",
62+ "apps/engine/src/index.ts"
63+ ],
64+ "interpreter": null,
65+ "cwd": "{{HOME}}/apps/websensor",
66+ "env": {
67+ "NODE_ENV": "production",
68+ "DATABASE_URL": "postgres://localhost:5432/websensor",
69+ "REDIS_URL": "redis://127.0.0.1:6379",
70+ "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
71+ "WS_SOURCES_FILE": "./config/sources.yaml",
72+ "WS_FETCH_CONCURRENCY": "32",
73+ "WS_PER_HOST_CONCURRENCY": "2",
74+ "ENGINE_METRICS_PORT": "8262",
75+ "WS_USER_AGENT": "WebSensorBot/0.3 (+https://www.websensor.io/bot; contact@websensor.io)",
76+ "ANTHROPIC_API_KEY": "<set me>",
77+ "WS_LLM_MODEL_FAST": "claude-haiku-4-5",
78+ "WS_LLM_MODEL_DEEP": "claude-opus-5",
79+ "WS_LLM_DAILY_CALL_BUDGET": "600",
80+ "WS_LLM_MIN_IMPORTANCE": "35",
81+ "WS_LLM_DEEP_MIN_IMPORTANCE": "78",
82+ "SCRAPFLY_API_KEY": "<set me>",
83+ "WS_SCRAPFLY_DAILY_BUDGET": "300",
84+ "LOG_LEVEL": "info",
85+ "WS_RETENTION_RAW_DAYS": "21",
86+ "WS_RETENTION_CHANGED_RAW_DAYS": "60",
87+ "WS_SILENT_MIN_IMPORTANCE": "45"
88+ },
89+ "cron_restart": null,
90+ "autorestart": true,
91+ "max_memory_restart": "2G"
33 }, 92 },
34− "ram_mb_observed": 1500, 93+ {
35− "size_mb": 40, 94+ "name": "websensor-factory",
36− "placement": { 95+ "manager": "pm2",
37− "pin": "M4M64b", 96+ "script": "/opt/homebrew/bin/node",
38− "prefer": null, 97+ "args": [
39− "avoid": ["M3U96b", "M1M32"], 98+ "node_modules/tsx/dist/cli.mjs",
40− "reason": "PostgreSQL 17 + Redis + pnpm présents ; 64 Go RAM ; M3U96a déjà chargé (25 processus PM2)" 99+ "apps/engine/src/factory-main.ts"
100+ ],
101+ "interpreter": null,
102+ "cwd": "{{HOME}}/apps/websensor",
103+ "env": {
104+ "NODE_ENV": "production",
105+ "DATABASE_URL": "postgres://localhost:5432/websensor",
106+ "REDIS_URL": "redis://127.0.0.1:6379",
107+ "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
108+ "WS_SOURCES_FILE": "./config/sources.yaml",
109+ "WS_USER_AGENT": "WebSensorBot/0.3 (+https://www.websensor.io/bot; contact@websensor.io)",
110+ "LOG_LEVEL": "info",
111+ "WS_FACTORY": "1",
112+ "WS_FACTORY_CONCURRENCY": "6",
113+ "WS_FACTORY_BUDGET": "90",
114+ "WS_FACTORY_MIN_SCORE": "0.35",
115+ "WS_FACTORY_SHADOW_MIN_CHECKS": "5",
116+ "WS_FACTORY_SHADOW_MIN_HOURS": "24",
117+ "WS_FACTORY_SHADOW_MAX_HOURS": "120",
118+ "WS_FACTORY_TICK_SECONDS": "20",
119+ "WS_COVERAGE_DIR": "./config/coverage",
120+ "WS_EDGAR_USER_AGENT": "WebSensor (contact@websensor.io)",
121+ "WS_APP_NAME": "websensor-factory"
122+ },
123+ "cron_restart": null,
124+ "autorestart": true,
125+ "max_memory_restart": "1500M"
41 }, 126 },
42− "processes": [ 127+ {
43− { 128+ "name": "websensor-api",
44− "name": "websensor-engine", 129+ "manager": "pm2",
45− "manager": "pm2", 130+ "script": "/opt/homebrew/bin/node",
46− "script": "/opt/homebrew/bin/node", 131+ "args": [
47− "args": ["node_modules/tsx/dist/cli.mjs", "apps/engine/src/index.ts"], 132+ "node_modules/tsx/dist/cli.mjs",
48− "interpreter": null, 133+ "apps/api/src/server.ts"
49− "cwd": "{{HOME}}/apps/websensor", 134+ ],
50− "env": { 135+ "interpreter": null,
51− "NODE_ENV": "production", 136+ "cwd": "{{HOME}}/apps/websensor",
52− "DATABASE_URL": "postgres://localhost:5432/websensor", 137+ "env": {
53− "REDIS_URL": "redis://127.0.0.1:6379", 138+ "NODE_ENV": "production",
54− "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs", 139+ "DATABASE_URL": "postgres://localhost:5432/websensor",
55− "WS_SOURCES_FILE": "./config/sources.yaml", 140+ "REDIS_URL": "redis://127.0.0.1:6379",
56− "WS_FETCH_CONCURRENCY": "16", 141+ "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
57− "WS_PER_HOST_CONCURRENCY": "2", 142+ "API_PORT": "8260",
58− "ENGINE_METRICS_PORT": "8262", 143+ "API_HOST": "0.0.0.0",
59− "WS_USER_AGENT": "WebSensorBot/0.1 (+https://www.websensor.io/bot; contact@websensor.io)", 144+ "WEB_URL": "http://127.0.0.1:8261",
60− "ANTHROPIC_API_KEY": "<secret>", 145+ "PUBLIC_BASE_URL": "https://www.websensor.io",
61− "WS_LLM_MODEL_FAST": "claude-haiku-4-5", 146+ "CANONICAL_HOST": "www.websensor.io",
62− "WS_LLM_MODEL_DEEP": "claude-opus-5", 147+ "LOG_LEVEL": "info",
63− "WS_LLM_DAILY_CALL_BUDGET": "600", 148+ "WS_ADMIN_TOKEN": "<set me>",
64− "WS_LLM_MIN_IMPORTANCE": "35", 149+ "WS_MONITORS_PER_OWNER": "5"
65− "WS_LLM_DEEP_MIN_IMPORTANCE": "78", 150+ },
66− "SCRAPFLY_API_KEY": "<secret>", 151+ "cron_restart": null,
67− "WS_SCRAPFLY_DAILY_BUDGET": "300", 152+ "autorestart": true,
68− "LOG_LEVEL": "info" 153+ "max_memory_restart": "1G"
69− },
70− "cron_restart": null,
71− "autorestart": true,
72− "max_memory_restart": "2G"
73− },
74− {
75− "name": "websensor-api",
76− "manager": "pm2",
77− "script": "/opt/homebrew/bin/node",
78− "args": ["node_modules/tsx/dist/cli.mjs", "apps/api/src/server.ts"],
79− "interpreter": null,
80− "cwd": "{{HOME}}/apps/websensor",
81− "env": {
82− "NODE_ENV": "production",
83− "DATABASE_URL": "postgres://localhost:5432/websensor",
84− "REDIS_URL": "redis://127.0.0.1:6379",
85− "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
86− "API_PORT": "8260",
87− "API_HOST": "0.0.0.0",
88− "WEB_URL": "http://127.0.0.1:8261",
89− "PUBLIC_BASE_URL": "https://www.websensor.io",
90− "CANONICAL_HOST": "www.websensor.io",
91− "LOG_LEVEL": "info"
92− },
93− "cron_restart": null,
94− "autorestart": true,
95− "max_memory_restart": "1G"
96− },
97− {
98− "name": "websensor-web",
99− "manager": "pm2",
100− "script": "/opt/homebrew/bin/node",
101− "args": ["node_modules/next/dist/bin/next", "start", "-p", "8261", "-H", "127.0.0.1"],
102− "interpreter": null,
103− "cwd": "{{HOME}}/apps/websensor/apps/web",
104− "env": {
105− "NODE_ENV": "production",
106− "API_URL": "http://127.0.0.1:8260",
107− "NEXT_PUBLIC_SITE_URL": "https://www.websensor.io",
108− "NEXT_TELEMETRY_DISABLED": "1"
109− },
110− "cron_restart": null,
111− "autorestart": true,
112− "max_memory_restart": "1G"
113− }
114− ],
115− "ngrok": { "name": "websensor-ngrok", "url": "www.websensor.io", "port": 8260 },
116− "launchd": [],
117− "env_overrides": {},
118− "hooks": {
119− "post_sync": [
120− "export PATH=\"/opt/homebrew/bin:$PATH\"; pnpm install --frozen-lockfile --silent && echo ' deps ok'",
121− "export PATH=\"/opt/homebrew/bin:$PATH\"; psql -d postgres -Atc \"select 1 from pg_database where datname='websensor'\" | grep -q 1 || createdb websensor; DATABASE_URL=postgres://localhost:5432/websensor pnpm db:migrate 2>&1 | tail -1 && echo ' db ok'",
122− "export PATH=\"/opt/homebrew/bin:$PATH\"; mkdir -p $HOME/websensor-data/blobs && DATABASE_URL=postgres://localhost:5432/websensor node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync 2>&1 | tail -1 && echo ' registry ok'",
123− "export PATH=\"/opt/homebrew/bin:$PATH\"; export API_URL=http://127.0.0.1:8260 NEXT_PUBLIC_SITE_URL=https://www.websensor.io NEXT_TELEMETRY_DISABLED=1; pnpm --filter @websensor/web build 2>&1 | tail -3 && echo ' web build ok'"
124− ],
125− "post_start": []
126 }, 154 },
127− "notes": "v0.1.0 (2026-09-08) : gateway Fastify :8260 (REST + WS /api/v1/live + proxy → Next :8261), engine (scheduler/pipeline, métriques :8262), 200 sources / ~290 capteurs (config/sources.yaml), Postgres 17 + Redis locaux, blobs zstd dans ~/websensor-data/blobs. Secrets uniquement dans ce manifeste (M1M32)." 155+ {
128−} 156+ "name": "websensor-web",
157+ "manager": "pm2",
158+ "script": "/opt/homebrew/bin/node",
159+ "args": [
160+ "node_modules/next/dist/bin/next",
161+ "start",
162+ "-p",
163+ "8261",
164+ "-H",
165+ "127.0.0.1"
166+ ],
167+ "interpreter": null,
168+ "cwd": "{{HOME}}/apps/websensor/apps/web",
169+ "env": {
170+ "NODE_ENV": "production",
171+ "API_URL": "http://127.0.0.1:8260",
172+ "NEXT_PUBLIC_SITE_URL": "https://www.websensor.io",
173+ "NEXT_TELEMETRY_DISABLED": "1"
174+ },
175+ "cron_restart": null,
176+ "autorestart": true,
177+ "max_memory_restart": "1G"
178+ }
179+ ],
180+ "ngrok": null,
181+ "launchd": [],
182+ "env_overrides": {},
183+ "hooks": {
184+ "post_sync": [
185+ "export PATH=\"/opt/homebrew/bin:$PATH\"; pnpm install --frozen-lockfile --silent && echo ' deps ok'",
186+ "set -o pipefail; export PATH=\"/opt/homebrew/bin:$PATH\"; psql -d postgres -Atc \"select 1 from pg_database where datname='websensor'\" | grep -q 1 || createdb websensor; DATABASE_URL=postgres://localhost:5432/websensor pnpm db:migrate 2>&1 | tail -1 && echo ' db ok'",
187+ "export PATH=\"/opt/homebrew/bin:$PATH\"; mkdir -p $HOME/websensor-data/blobs && DATABASE_URL=postgres://localhost:5432/websensor node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync 2>&1 | tail -1 && echo ' registry ok'",
188+ "export PATH=\"/opt/homebrew/bin:$PATH\"; DATABASE_URL=postgres://localhost:5432/websensor node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts factory seed --mode=hinted 2>&1 | tail -1 && echo ' factory seeds ok'",
189+ "set -o pipefail; export PATH=\"/opt/homebrew/bin:$PATH\"; export API_URL=http://127.0.0.1:8260 NEXT_PUBLIC_SITE_URL=https://www.websensor.io NEXT_TELEMETRY_DISABLED=1; pnpm --filter @websensor/web build 2>&1 | tail -3 && echo ' web build ok'"
190+ ],
191+ "post_start": []
192+ },
193+ "notes": "v0.1.0 (2026-09-08) : gateway Fastify :8260 (REST + WS /api/v1/live + proxy → Next :8261), engine (scheduler/pipeline, métriques :8262), 200 sources / ~290 capteurs (config/sources.yaml), Postgres 17 + Redis locaux, blobs zstd dans ~/websensor-data/blobs. Secrets uniquement dans ce manifeste (M1M32).",
194+ "tunnel": {
195+ "domain": "www.websensor.io",
196+ "gateway": "BHS64",
197+ "note": "MacLustr Tunnel depuis 2026-09-10 ; ngrok retiré"
198+ }
199+}
\ No newline at end of file
modified docs/ARCHITECTURE.md +10 −1
@@ -64,7 +64,16 @@ developing clusters, new coverage), entity insights (35-day heatmap, baseline, d
64 rank), entity rankings (importance × activity × velocity × quality × confirmation), cluster propagation, country and 64 rank), entity rankings (importance × activity × velocity × quality × confirmation), cluster propagation, country and
65 category desks. Aggregates are cached 5–120 s in process (`cache.ts`); the live feed is never cached. 65 category desks. Aggregates are cached 5–120 s in process (`cache.ts`); the live feed is never cached.
66 66
67−## Not in v0.2 (explicitly) 67+## Source Factory (v0.3)
68+Separate process (`factory-main.ts`). Tables: `factory_seeds` (organization + hints + funnel counters),
69+`discovery_candidates` (extended: connector/type/name/kind_class/config/tier/score_value/reason/shadow_sensor_id),
70+`factory_daily`. Sensors gain `status = 'SHADOW'` (pipeline stores snapshots and changes, never events; priority 3);
71+sources gain `origin` (seed | import | factory) and `sector`. Discovery (`packages/connectors/src/discovery-deep.ts`)
72+is budgeted per organization (90 requests) and polite (4 in flight per organization, 6 organizations in parallel).
73+Acceptance is evidence-based (`evaluateShadows`). Coverage (`apps/api/src/coverage.ts`) matches universe members to
74+sensor hosts by registrable domain (`coverageKey`) and is cached 5 min.
75+
76+## Not in v0.3 (explicitly)
68 Browser rendering (Playwright), Firecrawl, OpenSearch (Postgres FTS + search syntax is used), MinIO (fs blob store 77 Browser rendering (Playwright), Firecrawl, OpenSearch (Postgres FTS + search syntax is used), MinIO (fs blob store
69 with the same interface), accounts and email/Slack/Discord/Telegram/push delivery (watchlists, alerts, bookmarks, 78 with the same interface), accounts and email/Slack/Discord/Telegram/push delivery (watchlists, alerts, bookmarks,
70 saved views and custom monitors are keyed by an anonymous owner token; alert channels are web + signed webhooks), 79 saved views and custom monitors are keyed by an anonymous owner token; alert channels are web + signed webhooks),
modified docs/registry/AUTHORING.md +5 −0
@@ -6,6 +6,11 @@ entity; every `sensors:` entry becomes a real, polled endpoint. **Nothing goes i
6 fetched and parsed successfully** — a sensor that returns 403/404, HTML instead of a feed, or an empty list is a 6 fetched and parsed successfully** — a sensor that returns 403/404, HTML instead of a feed, or an empty list is a
7 liability (noise, wasted budget, false "page removed" events), not coverage. 7 liability (noise, wasted budget, false "page removed" events), not coverage.
8 8
9+> **Coverage universes & the Source Factory (2026-09-13).** Organizations that *should* be observed are listed per sector
10+> in `config/coverage/*.yaml` (guide: `COVERAGE.md`). The Factory discovers, validates, shadow-monitors and accepts their
11+> sensors automatically (`sources.origin = 'factory'`, `sensors.config.factory = true`). `cli.ts factory export` turns
12+> accepted sensors into a fragment like the ones below for review and graduation into `config/sources.d/`.
13+
9 ## Fragment format 14 ## Fragment format
10 15
11 ```yaml 16 ```yaml
added docs/registry/COVERAGE.md +67 −0
@@ -0,0 +1,67 @@
1+# Coverage universes — authoring guide
2+
3+`config/coverage/*.yaml` describes **what WebSensor should observe**, sector by sector: the denominator of the
4+Global Observation Coverage Score (`/coverage`, `GET /api/v1/coverage`). The same files feed the **Source
5+Factory**: every member whose registrable domain is not yet monitored becomes an expansion seed, is
6+discovered automatically (robots, sitemaps, feeds, status pages, GitHub, IR/newsroom/pricing/legal pages,
7+EDGAR, Hugging Face…), shadow-monitored, then accepted or rejected on evidence.
8+
9+A universe is a list of **organizations with a correct official domain**. It is *not* a registry fragment:
10+no sensors, no URLs are required (they are welcome as `hints`). The registry stays the source of truth for
11+what is actually polled.
12+
13+## File format — one sector per file (`config/coverage/NN-<sector>.yaml`; several files may share a sector key, their universes are merged — universe keys must then be unique across those files)
14+
15+```yaml
16+sector: energy # kebab-case key, unique across files
17+label: Energy & commodities
18+description: Agencies, grid operators, producers and exchanges whose changes move energy markets.
19+weight: 1.0 # weight of the sector in the global score (1.0 default; 1.2–1.5 systemic)
20+categories: [energy] # default categories for factory-created sources (registry vocabulary)
21+tier: B # default tier for factory-created sources (S A B C D)
22+universes:
23+ - key: agencies
24+ label: Energy agencies & regulators
25+ provenance: "https://en.wikipedia.org/wiki/… (retrieved 2026-09-13)"
26+ members:
27+ - { name: International Energy Agency, domain: iea.org, country: INT, importance: 3, aliases: [iea] }
28+ - { name: ERCOT, domain: ercot.com, country: US, importance: 3, hints: { urls: [{ url: "https://www.ercot.com/news/rss", kind: news }] } }
29+ - key: oil-gas-majors
30+ label: Oil & gas majors
31+ members:
32+ - { name: Shell, domain: shell.com, country: GB, importance: 3, hints: { cik: "1306965", ticker: SHEL } }
33+```
34+
35+### Member fields
36+| field | required | meaning |
37+|---|---|---|
38+| `name` | yes | official English name (ASCII preferred; native name in `aliases`) |
39+| `domain` | yes | bare registrable domain, lower-case, no scheme, no `www.`, no path (`gov.uk` sites: use the registrable domain, e.g. `bankofengland.co.uk`, `ons.gov.uk`) |
40+| `country` | recommended | ISO 3166-1 alpha-2 upper-case; `EU` for EU bodies; `INT` for global bodies. **Quote `"NO"`** (Norway) — YAML reads it as false |
41+| `language` | optional | ISO 639-1 of the site's main language when not English (`"no"` must be quoted) |
42+| `importance` | optional | 1 long tail · 2 notable (default) · 3 systemically important |
43+| `aliases` | optional | lower-case alternative names, tickers, acronyms (entity linking) |
44+| `categories` | optional | override of the sector default |
45+| `tier` | optional | override of the sector default |
46+| `first_party` | optional | `false` only for media/aggregators |
47+| `hints` | optional | `cik`, `ticker`, `exchange`, `github_org`, `github_repos: [owner/name]`, `hf_author`, `status_url`, `hosts: [ir.example.com]`, `urls: [{url, kind, connector?, type?, tier?, config?, name?}]`, `posture: true`, `wikidata: Q…` |
48+| `note` | optional | anything a reviewer should know (blocked by Cloudflare, JS shell…) |
49+
50+`hints.urls[].kind` vocabulary: `news press blog ir filings changelog releases security pricing legal careers docs api status models data other`.
51+
52+## Rules
53+1. **Real organizations, correct domains.** Every domain must resolve (`dig +short <domain>` or `<www.domain>`)
54+ and belong to the organization. Wikidata P856 / the index constituents page / EDGAR `website` are good
55+ sources; never guess a domain from a name.
56+2. **Provenance on every universe** (`provenance:` URL or query + retrieval date). Index constituents change:
57+ record the date.
58+3. **No duplicates inside a file** (same domain twice). The same organization may appear in several sectors
59+ (a bank is both `corporate` and `markets`).
60+4. **Countries everywhere** for governments and companies; `INT` for international bodies.
61+5. **Validate**: `node node_modules/tsx/dist/cli.mjs apps/engine/src/coverage-validate.ts config/coverage/<file>.yaml [--dns]`
62+ (schema, duplicates, domain syntax; `--dns` resolves every domain). Only files that pass are committed.
63+6. Keep entries compact (flow style `{ … }`, one member per line).
64+
65+## Sector keys in use (2026-09-13)
66+`ai` · `cloud` · `cyber` · `markets` · `corporate` · `government` · `science` · `healthcare` · `energy` ·
67+`transport` · `telecom` · `logistics` · `open-source` · `commerce`
added packages/connectors/src/discovery-deep.ts +672 −0
@@ -0,0 +1,672 @@
1+import { promises as dns } from "node:dns";
2+import { canonicalizeHtml, jaccard, registrableDomain, sha256, shingles, type SensorType, type Tier } from "@websensor/core";
3+import { httpFetch } from "./fetcher";
4+import { parseFeed } from "./rss";
5+import { parseSitemap } from "./sitemap";
6+import { detectFlavor } from "./statusjson";
7+
8+/**
9+ * Deep discovery — the acquisition stage of the Source Factory. Starting from an organization (domain +
10+ * optional hints) it walks robots.txt → sitemaps → feeds (`<link rel=alternate>`, well-known paths, official
11+ * sub-domains) → the homepage's own navigation (newsroom, press, investors, changelog, security, pricing,
12+ * legal, careers, docs, status) → known status-page providers → GitHub organization → EDGAR → Hugging Face →
13+ * OpenAPI documents → web posture, and validates EVERY candidate by fetching and parsing it exactly as the
14+ * connector would. Nothing is assumed from a URL pattern.
15+ *
16+ * The result is a ranked list of candidate sensors with the evidence the Factory needs to score them:
17+ * page class, first-party confidence, expected change frequency, fetch cost and a content fingerprint used
18+ * to drop duplicates (two feeds exposing the same items, two pages with the same text).
19+ */
20+export type CandidateKind = "news" | "press" | "blog" | "ir" | "filings" | "changelog" | "releases" | "security" | "pricing" | "legal" | "careers" | "leadership" | "docs" | "api" | "status" | "sitemap" | "models" | "data" | "posture" | "other";
21+
22+export interface DeepCandidate {
23+ url: string;
24+ type: SensorType;
25+ connector: string;
26+ kind: CandidateKind;
27+ name: string;
28+ config: Record<string, unknown>;
29+ suggestedTier: Tier;
30+ evidence: string;
31+ /** rough information value 0–1 (before the Factory's importance-aware scoring) */
32+ value: number;
33+ itemCount?: number;
34+ /** items per day observed in the feed/list window (null when unknown) */
35+ itemsPerDay?: number | null;
36+ /** 1 = the organization's own registrable domain; 0.9 = official channel on a platform (GitHub, HF, status provider) */
37+ firstPartyConfidence: number;
38+ fetchCost: { bytes: number; ms: number };
39+ /** content fingerprint for de-duplication (sorted item keys, or canonical text hash) */
40+ fingerprint?: string;
41+ itemKeys?: string[];
42+ textShingles?: Set<string>;
43+ title?: string;
44+}
45+
46+export interface DeepDiscoveryHints {
47+ cik?: string | number;
48+ github_org?: string;
49+ github_repos?: string[];
50+ hf_author?: string;
51+ status_url?: string;
52+ hosts?: string[];
53+ urls?: { url: string; kind?: string; connector?: string; type?: string; tier?: Tier; config?: Record<string, unknown>; name?: string }[];
54+ posture?: boolean;
55+}
56+
57+export interface DeepDiscoveryOptions {
58+ hints?: DeepDiscoveryHints;
59+ /** maximum number of HTTP requests for this organization (default 90) */
60+ budget?: number;
61+ concurrency?: number;
62+ sensorIdForLogs?: string;
63+ probePages?: boolean;
64+ probeSubdomains?: boolean;
65+ probePosture?: boolean;
66+ probeOpenApi?: boolean;
67+}
68+
69+export interface DeepDiscoveryResult {
70+ domain: string;
71+ /** origin actually used after following the homepage redirect (may be another registrable domain) */
72+ origin: string;
73+ canonicalDomain: string;
74+ homeStatus: number;
75+ /** anti-bot / challenge detected on the homepage */
76+ blocked: boolean;
77+ requests: number;
78+ candidates: DeepCandidate[];
79+ rejected: { url: string; reason: string }[];
80+ notes: string[];
81+ durationMs: number;
82+}
83+
84+// ------------------------------------------------------------------------------------------------
85+// Vocabularies
86+// ------------------------------------------------------------------------------------------------
87+
88+const FEED_PATHS = ["/feed", "/rss", "/rss.xml", "/atom.xml", "/feed.xml", "/index.xml", "/feed/", "/feeds/all.atom.xml", "/blog/feed", "/blog/rss", "/blog/rss.xml", "/blog/feed.xml", "/blog/atom.xml", "/blog/index.xml", "/news/feed", "/news/rss", "/news/rss.xml", "/news/feed.xml", "/news.rss", "/news.xml", "/newsroom/rss", "/newsroom/feed", "/newsroom/rss.xml", "/press/rss", "/press/feed", "/press-releases/rss", "/press-releases/feed", "/rss/news", "/rss/press-releases.xml", "/rss/news-releases.xml", "/rss/pressreleases.xml", "/investors/rss", "/investors/feed", "/en/feed", "/en/rss", "/en/rss.xml", "/en/news/rss", "/changelog/feed", "/changelog.rss", "/changelog/rss.xml", "/changelog.xml", "/releases.atom", "/releases/feed", "/security/feed", "/security/rss", "/advisories/rss", "/advisories.rss", "/updates/feed", "/updates/rss", "/whats-new/feed", "/?feed=rss2", "/feed/atom", "/feeds/posts/default", "/feed.json"];
89+const SITEMAP_PATHS = ["/sitemap.xml", "/sitemap_index.xml", "/sitemap-index.xml", "/sitemaps/sitemap.xml", "/news-sitemap.xml", "/sitemap/news.xml", "/sitemap-news.xml", "/sitemap.txt"];
90+const OPENAPI_PATHS = ["/openapi.json", "/swagger.json", "/api/openapi.json", "/api/swagger.json", "/v1/openapi.json", "/api/v1/openapi.json", "/.well-known/openapi.json", "/swagger/v1/swagger.json", "/api-docs", "/openapi.yaml", "/api/openapi.yaml"];
91+const SUBDOMAIN_KINDS: [string, CandidateKind][] = [["blog", "blog"], ["news", "news"], ["newsroom", "news"], ["press", "press"], ["media", "press"], ["investors", "ir"], ["investor", "ir"], ["ir", "ir"], ["status", "status"], ["docs", "docs"], ["developer", "docs"], ["developers", "docs"], ["security", "security"], ["trust", "security"], ["careers", "careers"], ["jobs", "careers"], ["changelog", "changelog"], ["updates", "changelog"], ["engineering", "blog"], ["research", "blog"], ["about", "other"], ["corporate", "news"], ["api", "api"]];
92+
93+/** Page classes probed from navigation links and well-known paths. `paths` are fallbacks when the navigation has no link. */
94+const PAGE_KINDS: { kind: CandidateKind; label: string; hrefRe: RegExp; textRe: RegExp; paths: string[]; tier: Tier; value: number; max: number }[] = [
95+ { kind: "pricing", label: "pricing", hrefRe: /\/(pricing|plans|plans-and-pricing|tarifs|preise|precios)(\/|$|\?)/i, textRe: /^(pricing|plans|plans & pricing|tarifs|preise|precios)$/i, paths: ["/pricing", "/plans", "/pricing/"], tier: "C", value: 0.75, max: 1 },
96+ { kind: "changelog", label: "changelog", hrefRe: /\/(changelog|change-log|release-notes|releasenotes|whats-new|what-s-new|updates|product-updates)(\/|$|\?)/i, textRe: /^(changelog|change log|release notes|what'?s new|product updates|updates)$/i, paths: ["/changelog", "/release-notes", "/whats-new", "/updates"], tier: "B", value: 0.75, max: 1 },
97+ { kind: "security", label: "security advisories", hrefRe: /\/(security|security-advisories|advisories|psirt|trust|security-bulletins|bulletins|vulnerabilit)/i, textRe: /^(security|security advisories|advisories|trust center|trust|psirt|security bulletins)$/i, paths: ["/security", "/security/advisories", "/trust", "/.well-known/security.txt"], tier: "B", value: 0.7, max: 2 },
98+ { kind: "news", label: "newsroom", hrefRe: /\/(news|newsroom|news-room|media|media-center|media-centre|actualites|actualit%C3%A9s|nouvelles|aktuelles|noticias|notizie)(\/|$|\?)/i, textRe: /^(news|newsroom|news ?room|media|media cent(er|re)|latest news|in the news|actualit[ée]s|nouvelles|aktuelles|noticias)$/i, paths: ["/news", "/newsroom", "/news/", "/en/news", "/media"], tier: "B", value: 0.65, max: 1 },
99+ { kind: "press", label: "press releases", hrefRe: /\/(press|press-releases|pressreleases|press-room|pressroom|press-center|communiques|communiqu%C3%A9s|presse|prensa)(\/|$|\?)/i, textRe: /^(press|press releases|press ?room|press cent(er|re)|communiqu[ée]s( de presse)?|presse|prensa)$/i, paths: ["/press", "/press-releases", "/press/", "/newsroom/press-releases"], tier: "B", value: 0.65, max: 1 },
100+ { kind: "ir", label: "investor relations", hrefRe: /\/(investors|investor-relations|investor|ir|relations-investisseurs|investor-center)(\/|$|\?)/i, textRe: /^(investors|investor relations|investor cent(er|re)|ir|relations investisseurs|shareholders)$/i, paths: ["/investors", "/investor-relations", "/investors/", "/investor"], tier: "C", value: 0.6, max: 1 },
101+ { kind: "legal", label: "terms of service", hrefRe: /\/(terms|terms-of-service|terms-of-use|tos|legal\/terms|conditions|cgu|cgv|legal)(\/|$|\?|\.)/i, textRe: /^(terms|terms of (service|use)|terms & conditions|terms and conditions|legal|conditions g[ée]n[ée]rales|cgu|cgv)$/i, paths: ["/terms", "/legal/terms", "/terms-of-service", "/legal"], tier: "C", value: 0.6, max: 1 },
102+ { kind: "legal", label: "privacy policy", hrefRe: /\/(privacy|privacy-policy|privacypolicy|legal\/privacy|politique-de-confidentialite|datenschutz|privacidad)(\/|$|\?|\.)/i, textRe: /^(privacy|privacy policy|privacy notice|politique de confidentialit[ée]|datenschutz|privacidad)$/i, paths: ["/privacy", "/legal/privacy", "/privacy-policy"], tier: "C", value: 0.55, max: 1 },
103+ { kind: "careers", label: "careers", hrefRe: /\/(careers|jobs|join-us|work-with-us|carrieres|carri%C3%A8res|karriere|empleo)(\/|$|\?)/i, textRe: /^(careers|jobs|join us|work with us|carri[èe]res|karriere|empleo|we'?re hiring)$/i, paths: ["/careers", "/jobs", "/careers/"], tier: "D", value: 0.35, max: 1 },
104+ { kind: "leadership", label: "leadership", hrefRe: /\/(leadership|management|executive-team|our-team|team|board-of-directors|board|about\/leadership|company\/leadership|direction|governance)(\/|$|\?)/i, textRe: /^(leadership|leadership team|management|management team|executive team|our team|board of directors|governance|direction)$/i, paths: ["/leadership", "/about/leadership", "/company/leadership", "/about/team"], tier: "D", value: 0.45, max: 1 },
105+ { kind: "docs", label: "documentation", hrefRe: /\/(docs|documentation|developers|developer|api-docs|reference)(\/|$|\?)/i, textRe: /^(docs|documentation|developers|developer|api|api reference|api docs)$/i, paths: ["/docs", "/documentation", "/developers"], tier: "D", value: 0.4, max: 1 },
106+ { kind: "data", label: "open data", hrefRe: /\/(open-data|opendata|data|statistics|statistiques|datasets|publications)(\/|$|\?)/i, textRe: /^(open data|data|statistics|statistiques|datasets|publications)$/i, paths: [], tier: "C", value: 0.4, max: 1 },
107+];
108+
109+const STATUS_HOST_RE = /^(https?:\/\/)?((status|statuspage|uptime|health|trust|system-status|servicestatus)\.[^/\s"']+|[^/\s"']*\.(statuspage\.io|instatus\.com|betteruptime\.com|betterstack\.com|incident\.io|status\.io|hund\.io|statuspal\.io|freshstatus\.io|status\.page|statusgator\.com))/i;
110+const CHALLENGE_RE = /cf-browser-verification|challenge-platform|__cf_chl|Attention Required!|Just a moment\.\.\.|Access Denied|Request unsuccessful\. Incapsula|akamai.*bot|_Incapsula_Resource|Please enable JavaScript to view|captcha/i;
111+
112+// ------------------------------------------------------------------------------------------------
113+
114+export async function discoverOrganization(domain: string, opts: DeepDiscoveryOptions = {}): Promise<DeepDiscoveryResult> {
115+ const started = Date.now();
116+ const id = opts.sensorIdForLogs ?? `factory_${domain}`;
117+ const budget = opts.budget ?? 90;
118+ const hints = opts.hints ?? {};
119+ const notes: string[] = [];
120+ const rejected: { url: string; reason: string }[] = [];
121+ const found = new Map<string, DeepCandidate>();
122+ let requests = 0;
123+
124+ const add = (c: DeepCandidate): void => {
125+ const k = c.url.replace(/\/$/, "");
126+ const prev = found.get(k);
127+ if (!prev || prev.value < c.value) found.set(k, c);
128+ };
129+ const budgetLeft = (): boolean => requests < budget;
130+ const get = async (url: string, o: { timeoutMs?: number; maxBytes?: number; accept?: string; headers?: Record<string, string>; userAgent?: string } = {}) => {
131+ requests++;
132+ return httpFetch(id, url, { timeoutMs: o.timeoutMs ?? 15_000, maxBytes: o.maxBytes ?? 3 * 1024 * 1024, accept: o.accept, headers: o.headers, userAgent: o.userAgent });
133+ };
134+
135+ // ---- 1. robots.txt --------------------------------------------------------------------------------------
136+ let base = `https://${domain}`;
137+ const sitemapsFromRobots: string[] = [];
138+ let robotsOk = false;
139+ let robotsMentionsAi = false;
140+ const robots = await get(`${base}/robots.txt`, { timeoutMs: 12_000, maxBytes: 512 * 1024 }).catch(() => null);
141+ if (robots?.body && robots.meta.status === 200 && !/^\s*<!doctype html|<html/i.test(robots.body.toString("utf8").slice(0, 300))) {
142+ robotsOk = true;
143+ const text = robots.body.toString("utf8");
144+ for (const line of text.split(/\r?\n/)) {
145+ const m = line.match(/^\s*sitemap:\s*(\S+)/i);
146+ if (m) sitemapsFromRobots.push(m[1]!);
147+ }
148+ robotsMentionsAi = /GPTBot|ClaudeBot|Google-Extended|CCBot|anthropic-ai|PerplexityBot|Bytespider|Applebot-Extended|meta-externalagent/i.test(text);
149+ }
150+
151+ // ---- 2. homepage ------------------------------------------------------------------------------------------
152+ const home = await get(`${base}/`, { timeoutMs: 20_000, maxBytes: 4 * 1024 * 1024 });
153+ const homeStatus = home.meta.status;
154+ let blocked = false;
155+ const homeHtml = home.body && home.meta.status < 400 ? home.body.toString("utf8") : "";
156+ if ([403, 429, 503].includes(homeStatus) || (homeHtml && CHALLENGE_RE.test(homeHtml.slice(0, 20_000)) && homeHtml.length < 60_000)) {
157+ blocked = true;
158+ notes.push(`homepage ${homeStatus}${homeHtml ? " (anti-bot challenge)" : ""}`);
159+ }
160+ let origin = base;
161+ try {
162+ const fu = new URL(home.meta.finalUrl || base);
163+ origin = fu.origin;
164+ if (registrableDomain(fu.hostname) !== registrableDomain(domain)) notes.push(`homepage redirects to ${fu.hostname}`);
165+ } catch {
166+ // keep base
167+ }
168+ base = origin;
169+ const canonicalDomain = registrableDomain(new URL(base).hostname);
170+ const ownHosts = new Set<string>([registrableDomain(domain), canonicalDomain, ...(hints.hosts ?? []).map((h) => registrableDomain(h))]);
171+ const isOwn = (u: string): boolean => {
172+ try {
173+ return ownHosts.has(registrableDomain(new URL(u).hostname));
174+ } catch {
175+ return false;
176+ }
177+ };
178+
179+ const homeCanon = homeHtml ? canonicalizeHtml(homeHtml, home.meta.finalUrl || base) : null;
180+ const homeShingles = homeCanon ? shingles(homeCanon.text) : null;
181+ const alternates: string[] = [];
182+ const statusLinks = new Set<string>();
183+ const githubOrgs = new Set<string>();
184+ const hfAuthors = new Set<string>();
185+ const pageLinks = new Map<CandidateKind, string[]>();
186+ if (homeHtml) {
187+ for (const m of homeHtml.matchAll(/<link[^>]+rel=["']alternate["'][^>]*>/gi)) {
188+ const tag = m[0];
189+ if (!/application\/(rss|atom)\+xml|application\/feed\+json/i.test(tag)) continue;
190+ const href = tag.match(/href=["']([^"']+)["']/i)?.[1];
191+ const abs = href ? safeAbs(href, home.meta.finalUrl || base) : null;
192+ if (abs) alternates.push(abs);
193+ }
194+ // Navigation links: classify by href pattern first, then by anchor text.
195+ const anchors = [...homeHtml.matchAll(/<a\b[^>]*href=["']([^"'#]+)["'][^>]*>([\s\S]{0,300}?)<\/a>/gi)].map((m) => ({ href: safeAbs(m[1]!, home.meta.finalUrl || base), text: m[2]!.replace(/<[^>]+>/g, " ").replace(/\s+/g, " ").trim() }));
196+ for (const a of anchors) {
197+ if (!a.href) continue;
198+ const gh = a.href.match(/^https?:\/\/(?:www\.)?github\.com\/([A-Za-z0-9_.-]+)\/?(?:$|\?)/);
199+ if (gh && !/^(features|about|pricing|topics|login|signup|sponsors|marketplace|explore|apps|orgs|enterprise|solutions|security|team|customer-stories|readme|collections|events|trending|search|settings|notifications|new|organizations|site|contact)$/i.test(gh[1]!)) githubOrgs.add(gh[1]!);
200+ const hf = a.href.match(/^https?:\/\/huggingface\.co\/([A-Za-z0-9_.-]+)\/?$/);
201+ if (hf && !/^(models|datasets|spaces|docs|pricing|papers|blog|join|login|enterprise|posts|tasks|learn|chat)$/i.test(hf[1]!)) hfAuthors.add(hf[1]!);
202+ if (STATUS_HOST_RE.test(a.href)) statusLinks.add(a.href);
203+ if (!isOwn(a.href)) continue;
204+ let path = "";
205+ try {
206+ path = new URL(a.href).pathname;
207+ } catch {
208+ continue;
209+ }
210+ if (path.split("/").filter(Boolean).length > 3) continue; // deep article links are not sections
211+ for (const pk of PAGE_KINDS) {
212+ if (pk.hrefRe.test(path) || pk.textRe.test(a.text)) {
213+ const arr = pageLinks.get(pk.kind) ?? [];
214+ if (!arr.includes(a.href) && arr.length < 4) arr.push(a.href);
215+ pageLinks.set(pk.kind, arr);
216+ }
217+ }
218+ }
219+ }
220+ if (hints.github_org) githubOrgs.add(hints.github_org);
221+ if (hints.hf_author) hfAuthors.add(hints.hf_author);
222+ if (hints.status_url) statusLinks.add(hints.status_url);
223+
224+ // ---- 3. official sub-domains (cheap DNS first) ------------------------------------------------------------
225+ const subHosts: { host: string; kind: CandidateKind }[] = [];
226+ if (opts.probeSubdomains !== false) {
227+ const rd = registrableDomain(domain);
228+ // Wildcard DNS (everything resolves) makes existence checks meaningless: only probe the core labels then.
229+ const wildcard = await resolves(`ws-probe-${Date.now().toString(36)}.${rd}`);
230+ if (wildcard) notes.push("wildcard DNS");
231+ const labels = wildcard ? ["blog", "news", "status", "investors", "docs", "security"] : [...new Set([...SUBDOMAIN_KINDS.map(([l]) => l)])];
232+ await parallel(labels, 8, async (label) => {
233+ const host = `${label}.${rd}`;
234+ if (host === new URL(base).hostname) return;
235+ if (wildcard || (await resolves(host))) subHosts.push({ host, kind: SUBDOMAIN_KINDS.find(([l]) => l === label)![1] });
236+ });
237+ if (wildcard) {
238+ // keep only sub-domains that actually answer with a distinct page
239+ const alive: typeof subHosts = [];
240+ await parallel(subHosts, 4, async (s) => {
241+ if (!budgetLeft()) return;
242+ const o = await get(`https://${s.host}/`, { timeoutMs: 10_000, maxBytes: 512 * 1024 });
243+ const fh = o.meta.finalUrl ? new URL(o.meta.finalUrl).hostname : "";
244+ if (o.meta.status === 200 && fh === s.host) alive.push(s);
245+ });
246+ subHosts.splice(0, subHosts.length, ...alive);
247+ }
248+ for (const h of hints.hosts ?? []) if (!subHosts.some((s) => s.host === h)) subHosts.push({ host: h, kind: "other" });
249+ if (subHosts.length) notes.push(`sub-domains: ${subHosts.map((s) => s.host).join(", ")}`);
250+ }
251+
252+ // ---- 4. feeds ---------------------------------------------------------------------------------------------
253+ const feedUrls = new Set<string>(alternates);
254+ for (const p of FEED_PATHS) feedUrls.add(base + p);
255+ for (const s of subHosts) {
256+ if (s.kind === "status" || s.kind === "api") continue;
257+ for (const p of ["/feed", "/rss", "/rss.xml", "/atom.xml", "/feed.xml", "/index.xml", "/feed/", "/?feed=rss2"]) feedUrls.add(`https://${s.host}${p}`);
258+ }
259+ // Feeds are cheap and high-value: spend up to 55 % of the budget here.
260+ const feedList = [...feedUrls].slice(0, Math.max(10, Math.floor(budget * 0.55)));
261+ const feeds: DeepCandidate[] = [];
262+ await parallel(feedList, opts.concurrency ?? 4, async (url) => {
263+ if (!budgetLeft()) return;
264+ const o = await get(url, { timeoutMs: 12_000, maxBytes: 4 * 1024 * 1024, accept: "application/rss+xml, application/atom+xml, application/feed+json, application/xml;q=0.9, */*;q=0.5" });
265+ if (!o.body || o.meta.status !== 200) return;
266+ const text = o.body.toString("utf8");
267+ if (/^\s*<!doctype html|<html/i.test(text.slice(0, 400))) return;
268+ try {
269+ const f = parseFeed(text, o.meta.finalUrl);
270+ if (!f.items.length) return rejected.push({ url, reason: "empty feed" });
271+ const dates = f.items.map((i) => (i.publishedAt ? new Date(i.publishedAt).getTime() : NaN)).filter((t) => !Number.isNaN(t));
272+ const dated = dates.length / f.items.length;
273+ const itemsPerDay = itemsPerDayOf(dates);
274+ const finalUrl = o.meta.finalUrl || url;
275+ const kind = feedKind(finalUrl, f.title);
276+ const keys = f.items.map((i) => i.key).sort();
277+ const stale = dates.length && Math.max(...dates) < Date.now() - 3 * 365 * 86400e3;
278+ if (stale) return rejected.push({ url, reason: "feed abandoned (> 3 years)" });
279+ feeds.push({
280+ url: finalUrl,
281+ type: f.kind === "atom" ? "ATOM" : "RSS",
282+ connector: "rss",
283+ kind,
284+ name: feedName(kind, finalUrl),
285+ config: {},
286+ suggestedTier: itemsPerDay !== null && itemsPerDay < 0.05 ? "C" : "B",
287+ evidence: `${alternates.includes(url) ? "link[rel=alternate]" : "well-known path"} · ${f.items.length} items${itemsPerDay !== null ? ` · ${itemsPerDay}/day` : ""}`,
288+ value: 0.75 + 0.15 * dated + (kind === "news" || kind === "press" ? 0.1 : kind === "security" || kind === "changelog" ? 0.08 : 0),
289+ itemCount: f.items.length,
290+ itemsPerDay,
291+ firstPartyConfidence: isOwn(finalUrl) ? 1 : 0.7,
292+ fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs },
293+ fingerprint: sha256(keys.join("\n")),
294+ itemKeys: keys,
295+ title: f.title,
296+ });
297+ } catch {
298+ // not a feed
299+ }
300+ });
301+ // De-duplicate feeds sharing ≥ 70 % of their items (rss + atom of the same channel, /feed vs /rss.xml).
302+ feeds.sort((a, b) => b.value - a.value || (b.itemCount ?? 0) - (a.itemCount ?? 0));
303+ const keptFeeds: DeepCandidate[] = [];
304+ for (const f of feeds) {
305+ const dup = keptFeeds.find((k) => k.itemKeys && f.itemKeys && jaccard(new Set(k.itemKeys), new Set(f.itemKeys)) >= 0.7);
306+ if (dup) rejected.push({ url: f.url, reason: `duplicate of ${dup.url}` });
307+ else keptFeeds.push(f);
308+ }
309+ for (const f of keptFeeds) add(f);
310+
311+ // ---- 5. sitemaps -------------------------------------------------------------------------------------------
312+ const sitemapCandidates = [...new Set([...sitemapsFromRobots.filter((u) => !/image|video/i.test(u)).sort((a, b) => Number(/news/i.test(b)) - Number(/news/i.test(a))).slice(0, 6), ...SITEMAP_PATHS.map((p) => base + p)])];
313+ const sitemaps: DeepCandidate[] = [];
314+ await parallel(sitemapCandidates, 3, async (url) => {
315+ if (!budgetLeft() || sitemaps.length >= 3) return;
316+ const o = await get(url, { timeoutMs: 15_000, maxBytes: 20 * 1024 * 1024, accept: "application/xml, text/xml, text/plain;q=0.8, */*;q=0.5" });
317+ if (!o.body || o.meta.status !== 200) return;
318+ const text = o.body.toString("utf8");
319+ if (/^\s*<!doctype html|<html/i.test(text.slice(0, 400))) return;
320+ try {
321+ const s = parseSitemap(text);
322+ const n = s.kind === "sitemapindex" ? s.children.length : s.entries.length;
323+ if (!n) return;
324+ const lastmods = s.entries.filter((e) => e.lastmod).length;
325+ const isNews = /news/i.test(url) || s.entries.some((e) => e.publishedAt);
326+ sitemaps.push({ url: o.meta.finalUrl || url, type: "SITEMAP", connector: "sitemap", kind: "sitemap", name: isNews ? "news sitemap" : "sitemap", config: { maxChildren: 4, maxUrls: 3000 }, suggestedTier: "B", evidence: `${sitemapsFromRobots.includes(url) ? "robots.txt" : "well-known path"} · ${n} ${s.kind === "sitemapindex" ? "children" : "urls"}`, value: 0.45 + (s.entries.length ? 0.3 * (lastmods / s.entries.length) : 0.15) + (isNews ? 0.2 : 0), itemCount: n, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
327+ } catch {
328+ // not a sitemap
329+ }
330+ });
331+ sitemaps.sort((a, b) => b.value - a.value);
332+ if (sitemaps[0]) add(sitemaps[0]);
333+
334+ // ---- 6. status pages ----------------------------------------------------------------------------------------
335+ const statusHosts = new Set<string>();
336+ if (hints.status_url) {
337+ try {
338+ statusHosts.add(new URL(hints.status_url).origin);
339+ } catch {
340+ // ignore
341+ }
342+ }
343+ for (const s of statusLinks) {
344+ try {
345+ statusHosts.add(new URL(s).origin);
346+ } catch {
347+ // ignore
348+ }
349+ }
350+ for (const s of subHosts) if (s.kind === "status") statusHosts.add(`https://${s.host}`);
351+ let statusFound = false;
352+ for (const root of statusHosts) {
353+ if (statusFound || !budgetLeft()) break;
354+ for (const [path, connector] of [["/api/v2/summary.json", "statuspage"], ["/summary.json", "statusjson"], ["/api/v1/summary", "statusjson"]] as const) {
355+ if (statusFound || !budgetLeft()) break;
356+ const o = await get(root + path, { timeoutMs: 12_000, accept: "application/json" });
357+ if (!o.body || o.meta.status !== 200) continue;
358+ const text = o.body.toString("utf8");
359+ try {
360+ const j = JSON.parse(text) as Record<string, unknown>;
361+ if (connector === "statuspage" && Array.isArray(j.incidents) && (j.components || j.status)) {
362+ add({ url: root + path, type: "STATUSPAGE", connector: "statuspage", kind: "status", name: "status", config: {}, suggestedTier: "S", evidence: `Atlassian Statuspage (${new URL(root).host})`, value: 0.95, itemsPerDay: null, firstPartyConfidence: 0.9, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
363+ statusFound = true;
364+ } else if (connector === "statusjson") {
365+ const flavor = detectFlavor(j);
366+ if (flavor) {
367+ add({ url: root + path, type: "STATUSPAGE", connector: "statusjson", kind: "status", name: "status", config: { flavor }, suggestedTier: "S", evidence: `${flavor} status JSON (${new URL(root).host})`, value: 0.93, itemsPerDay: null, firstPartyConfidence: 0.9, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
368+ statusFound = true;
369+ }
370+ }
371+ } catch {
372+ // not JSON
373+ }
374+ }
375+ }
376+
377+ // ---- 7. pages (server-rendered sections) --------------------------------------------------------------------
378+ if (opts.probePages !== false && !blocked) {
379+ const pageJobs: { kind: CandidateKind; label: string; tier: Tier; value: number; urls: string[]; max: number }[] = [];
380+ for (const pk of PAGE_KINDS) {
381+ const linked = pageLinks.get(pk.kind) ?? [];
382+ // feeds already cover news/press/blog for this org → the HTML index page adds little
383+ if ((pk.kind === "news" || pk.kind === "press") && keptFeeds.some((f) => f.kind === pk.kind || f.kind === "news")) continue;
384+ const shallowFirst = (a: string, b: string): number => safePath(a).split("/").length - safePath(b).split("/").length || a.length - b.length;
385+ const urls = [...new Set([...linked.filter((u) => pk.hrefRe.test(safePath(u)) || linked.length <= 2).sort(shallowFirst), ...pk.paths.map((p) => base + p)])].slice(0, 4);
386+ if (!urls.length) continue;
387+ pageJobs.push({ kind: pk.kind, label: pk.label, tier: pk.tier, value: pk.value, urls, max: pk.max });
388+ }
389+ const pageCanon: { url: string; sh: Set<string>; hash: string }[] = [];
390+ const probed = new Set<string>();
391+ await parallel(pageJobs, 3, async (job) => {
392+ let kept = 0;
393+ for (const url of job.urls) {
394+ if (kept >= job.max || !budgetLeft()) break;
395+ const pk = url.replace(/\/$/, "");
396+ if (probed.has(pk) || found.has(pk)) continue;
397+ probed.add(pk);
398+ const o = await get(url, { timeoutMs: 15_000, maxBytes: 3 * 1024 * 1024, accept: "text/html, application/xhtml+xml;q=0.9, */*;q=0.5" });
399+ if (o.meta.status !== 200 || !o.body) continue;
400+ const finalPath = safePath(o.meta.finalUrl);
401+ if (finalPath === "/" || finalPath === "") continue; // redirected home
402+ if (!isOwn(o.meta.finalUrl || url)) continue;
403+ if (!/text\/html|xhtml/i.test(o.meta.contentType ?? "") && /^\s*[{[]/.test(o.body.toString("utf8").slice(0, 50))) continue;
404+ const c = canonicalizeHtml(o.body.toString("utf8"), o.meta.finalUrl || url);
405+ if (c.text.length < 300) {
406+ rejected.push({ url, reason: `thin (${c.text.length} chars)` });
407+ continue;
408+ }
409+ const sh = shingles(c.text);
410+ if (homeCanon && (c.canonicalHash === homeCanon.canonicalHash || (homeShingles && jaccard(sh, homeShingles) > 0.8))) {
411+ rejected.push({ url, reason: "same as homepage" });
412+ continue;
413+ }
414+ const dup = pageCanon.find((p) => p.hash === c.canonicalHash || jaccard(p.sh, sh) > 0.85);
415+ if (dup) {
416+ rejected.push({ url, reason: `duplicate of ${dup.url}` });
417+ continue;
418+ }
419+ pageCanon.push({ url: o.meta.finalUrl || url, sh, hash: c.canonicalHash });
420+ const isSecurityTxt = /security\.txt$/.test(url);
421+ add({ url: o.meta.finalUrl || url, type: "HTML", connector: "http", kind: job.kind, name: isSecurityTxt ? "security.txt" : job.label, config: {}, suggestedTier: isSecurityTxt ? "D" : job.tier, evidence: `${pageLinks.get(job.kind)?.includes(url) ? "linked from homepage" : "well-known path"} · GET 200 · ${c.text.length} chars · distinct from homepage`, value: isSecurityTxt ? 0.3 : job.value, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs }, fingerprint: c.canonicalHash, textShingles: sh, title: c.title ?? undefined });
422+ kept++;
423+ }
424+ });
425+ }
426+
427+ // ---- 8. GitHub organization → repositories' releases ----------------------------------------------------------
428+ const repos = new Set<string>(hints.github_repos ?? []);
429+ for (const org of [...githubOrgs].slice(0, 1)) {
430+ if (repos.size >= 4 || !budgetLeft()) break;
431+ const token = process.env.GITHUB_TOKEN;
432+ if (token) {
433+ const o = await get(`https://api.github.com/search/repositories?q=org:${encodeURIComponent(org)}+fork:false&sort=stars&order=desc&per_page=5`, { accept: "application/vnd.github+json", headers: { authorization: `Bearer ${token}`, "x-github-api-version": "2022-11-28" } });
434+ if (o.body && o.meta.status === 200) {
435+ try {
436+ const j = JSON.parse(o.body.toString("utf8")) as { items?: { full_name: string; archived?: boolean; stargazers_count?: number }[] };
437+ for (const r of j.items ?? []) if (!r.archived && (r.stargazers_count ?? 0) >= 50) repos.add(r.full_name);
438+ } catch {
439+ // ignore
440+ }
441+ }
442+ } else {
443+ // Unauthenticated: the organization page lists pinned / popular repositories server-side.
444+ const o = await get(`https://github.com/${org}`, { timeoutMs: 15_000, maxBytes: 3 * 1024 * 1024 });
445+ if (o.body && o.meta.status === 200) {
446+ const html = o.body.toString("utf8");
447+ const counts = new Map<string, number>();
448+ for (const m of html.matchAll(new RegExp(`href="/${org.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}/([A-Za-z0-9_.-]+)"`, "g"))) {
449+ const r = m[1]!;
450+ if (/^(followers|following|repositories|projects|packages|people|teams|sponsoring|discussions|orgs|\.github)$/i.test(r)) continue;
451+ counts.set(r, (counts.get(r) ?? 0) + 1);
452+ }
453+ const pinned = [...html.matchAll(/class="[^"]*pinned-item-list-item[^"]*"[\s\S]{0,1500}?href="\/([^/"]+)\/([^/"]+)"/g)].map((m) => `${m[1]}/${m[2]}`).filter((f) => f.toLowerCase().startsWith(org.toLowerCase() + "/"));
454+ for (const f of pinned) repos.add(f);
455+ for (const [r] of [...counts.entries()].sort((a, b) => b[1] - a[1]).slice(0, 6)) if (repos.size < 6) repos.add(`${org}/${r}`);
456+ }
457+ }
458+ }
459+ const repoList = [...repos].slice(0, 6);
460+ let releasesKept = 0;
461+ await parallel(repoList, 3, async (full) => {
462+ if (releasesKept >= 4 || !budgetLeft()) return;
463+ for (const kind of ["releases", "tags"] as const) {
464+ const url = `https://github.com/${full}/${kind}.atom`;
465+ const o = await get(url, { timeoutMs: 12_000, accept: "application/atom+xml, application/xml;q=0.9" });
466+ if (!o.body || o.meta.status !== 200) {
467+ if (o.meta.status === 404) return;
468+ continue;
469+ }
470+ try {
471+ const f = parseFeed(o.body.toString("utf8"), url);
472+ if (!f.items.length) continue;
473+ const dates = f.items.map((i) => (i.publishedAt ? new Date(i.publishedAt).getTime() : NaN)).filter((t) => !Number.isNaN(t));
474+ if (dates.length && Math.max(...dates) < Date.now() - 2 * 365 * 86400e3) return rejected.push({ url, reason: "repository inactive (> 2 years)" });
475+ add({ url, type: "GITHUB_RELEASE", connector: "github", kind: "releases", name: `${full.split("/")[1]} ${kind}`, config: { repo: full, kind }, suggestedTier: "B", evidence: `GitHub ${kind}.atom · ${f.items.length} items`, value: kind === "releases" ? 0.8 : 0.65, itemCount: f.items.length, itemsPerDay: null, firstPartyConfidence: 0.9, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs }, fingerprint: sha256(f.items.map((i) => i.key).sort().join("\n")) });
476+ releasesKept++;
477+ return;
478+ } catch {
479+ // not a feed
480+ }
481+ }
482+ });
483+
484+ // ---- 9. Hugging Face models ------------------------------------------------------------------------------------
485+ for (const author of [...hfAuthors].slice(0, 2)) {
486+ if (!budgetLeft()) break;
487+ const url = `https://huggingface.co/api/models?author=${encodeURIComponent(author)}&sort=lastModified&direction=-1&limit=50`;
488+ const o = await get(url, { accept: "application/json" });
489+ if (!o.body || o.meta.status !== 200) continue;
490+ try {
491+ const j = JSON.parse(o.body.toString("utf8")) as unknown[];
492+ if (Array.isArray(j) && j.length) add({ url, type: "REST_API", connector: "jsonlist", kind: "models", name: "hugging face models", config: { keyField: "id", titleField: "id", dateField: "lastModified", compareFields: ["pipeline_tag", "library_name"], maxItems: 50 }, suggestedTier: "B", evidence: `Hugging Face author ${author} · ${j.length} models`, value: 0.85, itemCount: j.length, itemsPerDay: null, firstPartyConfidence: 0.9, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
493+ } catch {
494+ // ignore
495+ }
496+ }
497+
498+ // ---- 10. EDGAR filings -------------------------------------------------------------------------------------------
499+ if (hints.cik !== undefined && budgetLeft()) {
500+ const cik = String(hints.cik).replace(/\D/g, "").padStart(10, "0");
501+ const url = `https://data.sec.gov/submissions/CIK${cik}.json`;
502+ const ua = process.env.WS_EDGAR_USER_AGENT ?? "WebSensor (contact@websensor.io)";
503+ const o = await get(url, { accept: "application/json", userAgent: ua, headers: { "user-agent": ua }, timeoutMs: 30_000, maxBytes: 30 * 1024 * 1024 });
504+ if (o.body && o.meta.status === 200 && /"filings"/.test(o.body.toString("utf8").slice(0, 200_000))) {
505+ add({ url, type: "REST_API", connector: "edgar", kind: "filings", name: "edgar filings", config: { forms: ["8-K", "10-K", "10-Q", "6-K", "20-F", "40-F", "S-1", "SC 13D", "DEF 14A"] }, suggestedTier: "B", evidence: `SEC EDGAR submissions CIK ${cik}`, value: 0.9, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
506+ } else rejected.push({ url, reason: `edgar ${o.meta.status}` });
507+ }
508+
509+ // ---- 11. OpenAPI documents ---------------------------------------------------------------------------------------
510+ if (opts.probeOpenApi !== false && (subHosts.some((s) => s.kind === "docs" || s.kind === "api") || pageLinks.has("docs")) && budgetLeft()) {
511+ const roots = [base, ...subHosts.filter((s) => s.kind === "docs" || s.kind === "api").map((s) => `https://${s.host}`)].slice(0, 3);
512+ let openapiFound = false;
513+ for (const root of roots) {
514+ if (openapiFound) break;
515+ for (const p of OPENAPI_PATHS.slice(0, 7)) {
516+ if (openapiFound || !budgetLeft()) break;
517+ const o = await get(root + p, { timeoutMs: 15_000, maxBytes: 8 * 1024 * 1024, accept: "application/json, application/yaml, */*;q=0.5" });
518+ if (!o.body || o.meta.status !== 200) continue;
519+ const text = o.body.toString("utf8").slice(0, 4000);
520+ if (/^\s*[{[]/.test(text) && /"(openapi|swagger)"\s*:/.test(text)) {
521+ add({ url: root + p, type: "JSON", connector: "openapi", kind: "api", name: "openapi", config: {}, suggestedTier: "C", evidence: `OpenAPI document (${new URL(root).host})`, value: 0.7, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
522+ openapiFound = true;
523+ }
524+ }
525+ }
526+ }
527+
528+ // ---- 12. explicit hint URLs (validated like everything else) --------------------------------------------------------
529+ for (const h of hints.urls ?? []) {
530+ if (!budgetLeft()) break;
531+ const known = h.url.replace(/\/$/, "");
532+ if (found.has(known)) continue;
533+ const o = await get(h.url, { timeoutMs: 15_000, maxBytes: 6 * 1024 * 1024 });
534+ if (!o.body || o.meta.status !== 200) {
535+ rejected.push({ url: h.url, reason: `hint ${o.meta.status || o.error?.code}` });
536+ continue;
537+ }
538+ const text = o.body.toString("utf8");
539+ const kind = (h.kind ?? "other") as CandidateKind;
540+ const tier = h.tier ?? (kind === "status" ? "S" : kind === "pricing" || kind === "legal" || kind === "ir" ? "C" : "B");
541+ const cost = { bytes: o.meta.contentLength, ms: o.meta.durationMs };
542+ if (h.connector) {
543+ add({ url: h.url, type: (h.type ?? (h.connector === "rss" ? "RSS" : h.connector === "jsonlist" || h.connector === "edgar" ? "REST_API" : h.connector === "statuspage" || h.connector === "statusjson" ? "STATUSPAGE" : h.connector === "sitemap" ? "SITEMAP" : "HTML")) as SensorType, connector: h.connector, kind, name: h.name ?? kind, config: h.config ?? {}, suggestedTier: tier, evidence: "explicit hint · GET 200", value: 0.8, itemsPerDay: null, firstPartyConfidence: isOwn(h.url) ? 1 : 0.8, fetchCost: cost });
544+ continue;
545+ }
546+ // Sniff: feed → rss; JSON → http/json; HTML → http
547+ if (!/^\s*<!doctype html|<html/i.test(text.slice(0, 400)) && /<(rss|feed|rdf:RDF)\b/i.test(text.slice(0, 2000))) {
548+ try {
549+ const f = parseFeed(text, o.meta.finalUrl);
550+ if (f.items.length) {
551+ add({ url: o.meta.finalUrl || h.url, type: f.kind === "atom" ? "ATOM" : "RSS", connector: "rss", kind, name: h.name ?? feedName(kind, h.url), config: {}, suggestedTier: tier, evidence: `explicit hint · feed · ${f.items.length} items`, value: 0.85, itemCount: f.items.length, itemsPerDay: null, firstPartyConfidence: isOwn(h.url) ? 1 : 0.8, fetchCost: cost, fingerprint: sha256(f.items.map((i) => i.key).sort().join("\n")), itemKeys: f.items.map((i) => i.key).sort() });
552+ continue;
553+ }
554+ } catch {
555+ // fallthrough
556+ }
557+ }
558+ if (/^\s*[{[]/.test(text.slice(0, 50))) {
559+ add({ url: h.url, type: "JSON", connector: "http", kind, name: h.name ?? kind, config: h.config ?? {}, suggestedTier: tier, evidence: "explicit hint · JSON 200", value: 0.65, itemsPerDay: null, firstPartyConfidence: isOwn(h.url) ? 1 : 0.8, fetchCost: cost });
560+ continue;
561+ }
562+ const c = canonicalizeHtml(text, o.meta.finalUrl || h.url);
563+ if (c.text.length < 300) {
564+ rejected.push({ url: h.url, reason: `hint thin (${c.text.length} chars)` });
565+ continue;
566+ }
567+ add({ url: o.meta.finalUrl || h.url, type: "HTML", connector: "http", kind, name: h.name ?? PAGE_KINDS.find((p) => p.kind === kind)?.label ?? kind, config: h.config ?? {}, suggestedTier: tier, evidence: `explicit hint · GET 200 · ${c.text.length} chars`, value: 0.7, itemsPerDay: null, firstPartyConfidence: isOwn(h.url) ? 1 : 0.8, fetchCost: cost, fingerprint: c.canonicalHash, title: c.title ?? undefined });
568+ }
569+
570+ // ---- 13. web posture (no network here: validated in shadow by the dns/tls connectors) ------------------------------
571+ if (opts.probePosture || hints.posture) {
572+ const rd = registrableDomain(domain);
573+ add({ url: `dns://${rd}`, type: "DNS", connector: "dns", kind: "posture", name: "dns", config: {}, suggestedTier: "D", evidence: "posture", value: 0.3, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: 0, ms: 0 } });
574+ add({ url: `tls://${new URL(base).hostname}`, type: "TLS", connector: "tls", kind: "posture", name: "tls certificate", config: {}, suggestedTier: "D", evidence: "posture", value: 0.3, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: 0, ms: 0 } });
575+ if (robotsOk) add({ url: `https://${domain}/robots.txt`, type: "HTML", connector: "http", kind: "posture", name: "robots.txt", config: {}, suggestedTier: "D", evidence: robotsMentionsAi ? "robots.txt names AI crawlers" : "robots.txt", value: robotsMentionsAi ? 0.45 : 0.3, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: robots?.meta.contentLength ?? 0, ms: robots?.meta.durationMs ?? 0 } });
576+ }
577+
578+ const candidates = [...found.values()].sort((a, b) => b.value - a.value);
579+ // Strip the heavy de-dup helpers before returning (they are not serializable and no longer needed).
580+ for (const c of candidates) {
581+ delete c.textShingles;
582+ if (c.itemKeys && c.itemKeys.length > 60) c.itemKeys = c.itemKeys.slice(0, 60);
583+ }
584+ return { domain, origin: base, canonicalDomain, homeStatus, blocked, requests, candidates, rejected, notes, durationMs: Date.now() - started };
585+}
586+
587+// ------------------------------------------------------------------------------------------------
588+
589+/** Expected publication rate from item dates: the median gap between consecutive items (robust to one bogus date). */
590+export function itemsPerDayOf(dates: number[]): number | null {
591+ const d = [...dates].filter((t) => Number.isFinite(t)).sort((a, b) => b - a).slice(0, 30);
592+ if (d.length < 3) return null;
593+ const gaps = d.slice(1).map((t, i) => (d[i]! - t) / 86400e3).filter((g) => g >= 0).sort((a, b) => a - b);
594+ if (!gaps.length) return null;
595+ const median = gaps[Math.floor(gaps.length / 2)]!;
596+ if (median <= 0) return 50;
597+ return Math.round((1 / median) * 1000) / 1000;
598+}
599+
600+function feedKind(url: string, title?: string): CandidateKind {
601+ const s = `${url} ${title ?? ""}`.toLowerCase();
602+ if (/press|communiqu|pressemitteilung|prensa|news-release|newsrelease/.test(s)) return "press";
603+ if (/investor|\/ir\/|ir\.|sec-filing|filings/.test(s)) return "ir";
604+ if (/security|advisor|psirt|bulletin|vulnerab|cve/.test(s)) return "security";
605+ if (/changelog|release-notes|releasenotes|releases|whats-new|updates/.test(s)) return "changelog";
606+ if (/blog|engineering|research|insights|stories/.test(s)) return "blog";
607+ if (/news|actualit|nouvelles|aktuell|noticias|announcement/.test(s)) return "news";
608+ if (/jobs|careers/.test(s)) return "careers";
609+ return "news";
610+}
611+
612+function feedName(kind: CandidateKind, url: string): string {
613+ switch (kind) {
614+ case "press":
615+ return "press releases feed";
616+ case "ir":
617+ return "investor news feed";
618+ case "security":
619+ return "security feed";
620+ case "changelog":
621+ return "changelog feed";
622+ case "blog":
623+ return /engineering/i.test(url) ? "engineering blog feed" : /research/i.test(url) ? "research feed" : "blog feed";
624+ case "careers":
625+ return "jobs feed";
626+ default:
627+ return "news feed";
628+ }
629+}
630+
631+function safePath(u: string): string {
632+ try {
633+ return new URL(u).pathname.replace(/\/$/, "");
634+ } catch {
635+ return "";
636+ }
637+}
638+
639+function safeAbs(h: string, base: string): string | null {
640+ try {
641+ const u = new URL(h, base);
642+ if (!u.protocol.startsWith("http")) return null;
643+ u.hash = "";
644+ return u.toString();
645+ } catch {
646+ return null;
647+ }
648+}
649+
650+async function resolves(host: string): Promise<boolean> {
651+ try {
652+ const r = await dns.lookup(host, { all: true });
653+ return r.length > 0;
654+ } catch {
655+ return false;
656+ }
657+}
658+
659+async function parallel<T>(items: T[], limit: number, fn: (item: T) => Promise<unknown>): Promise<void> {
660+ let i = 0;
661+ const workers = Array.from({ length: Math.min(limit, items.length) }, async () => {
662+ while (i < items.length) {
663+ const item = items[i++]!;
664+ try {
665+ await fn(item);
666+ } catch {
667+ // best effort
668+ }
669+ }
670+ });
671+ await Promise.all(workers);
672+}
modified packages/connectors/src/index.ts +1 −0
@@ -66,3 +66,4 @@ export function getConnector(key: string): WebSensorConnector {
66 export function listConnectors(): WebSensorConnector[] { 66 export function listConnectors(): WebSensorConnector[] {
67 return Object.values(REGISTRY); 67 return Object.values(REGISTRY);
68 } 68 }
69+export * from "./discovery-deep";
added packages/core/src/coverage-schema.ts +150 −0
@@ -0,0 +1,150 @@
1+import { z } from "zod";
2+import { TIERS } from "./taxonomy";
3+
4+/**
5+ * Coverage universes (`config/coverage/*.yaml`). A universe is the list of organizations WebSensor *should*
6+ * observe for a sector — the denominator of the Global Observation Coverage Score. The same files feed the
7+ * Source Factory: every member without a monitored domain becomes an expansion seed.
8+ *
9+ * One file per sector:
10+ *
11+ * sector: ai
12+ * label: Artificial intelligence
13+ * weight: 1.2 # weight of the sector in the global score (default 1)
14+ * categories: [ai, technology] # default categories applied to members (registry vocabulary)
15+ * tier: B # default tier for factory-created sources
16+ * universes:
17+ * - key: frontier-labs
18+ * label: Frontier labs & model providers
19+ * members:
20+ * - { name: OpenAI, domain: openai.com, country: US, importance: 3, hints: { github_org: openai, hf_author: openai } }
21+ */
22+export const memberHintUrlSchema = z.object({
23+ url: z.string().url(),
24+ /** what the page is: news | press | blog | ir | filings | changelog | releases | security | pricing | legal | careers | docs | api | status | models | data | other */
25+ kind: z.string().default("other"),
26+ connector: z.string().optional(),
27+ type: z.string().optional(),
28+ tier: z.enum(TIERS).optional(),
29+ config: z.record(z.string(), z.unknown()).optional(),
30+ name: z.string().optional(),
31+});
32+
33+export const memberHintsSchema = z.object({
34+ /** SEC CIK (digits, any length) → EDGAR submissions sensor */
35+ cik: z.union([z.string(), z.number()]).optional(),
36+ ticker: z.string().optional(),
37+ exchange: z.string().optional(),
38+ /** GitHub organization → most active repositories' releases */
39+ github_org: z.string().optional(),
40+ /** explicit repositories `owner/name` */
41+ github_repos: z.array(z.string()).optional(),
42+ /** Hugging Face author/organization → model list sensor */
43+ hf_author: z.string().optional(),
44+ /** status page root (Atlassian Statuspage, Instatus, incident.io, Status.io) */
45+ status_url: z.string().url().optional(),
46+ /** additional hosts that belong to the organization (blog.example.org, investors.example.org…) */
47+ hosts: z.array(z.string()).optional(),
48+ /** explicit candidate URLs (validated by the factory before becoming sensors) */
49+ urls: z.array(memberHintUrlSchema).optional(),
50+ /** probe robots/TLS/DNS/headers posture sensors (default: only for importance 3) */
51+ posture: z.boolean().optional(),
52+ /** Wikidata QID, for provenance */
53+ wikidata: z.string().optional(),
54+});
55+
56+export const coverageMemberSchema = z.object({
57+ name: z.string().min(1),
58+ /** bare registrable domain, lower-case, no scheme, no `www.` */
59+ domain: z
60+ .string()
61+ .min(3)
62+ .transform((d) => d.trim().toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "")),
63+ country: z.string().regex(/^[A-Z]{2,3}$/).optional(),
64+ language: z.string().regex(/^[a-z]{2}$/).optional(),
65+ /** 1 = long tail · 2 = notable (default) · 3 = systemically important */
66+ importance: z.union([z.literal(1), z.literal(2), z.literal(3)]).default(2),
67+ aliases: z.array(z.string()).default([]),
68+ categories: z.array(z.string()).optional(),
69+ tier: z.enum(TIERS).optional(),
70+ first_party: z.boolean().optional(),
71+ hints: memberHintsSchema.default({}),
72+ note: z.string().optional(),
73+});
74+export type CoverageMember = z.infer<typeof coverageMemberSchema>;
75+
76+export const coverageUniverseSchema = z.object({
77+ key: z.string().regex(/^[a-z0-9][a-z0-9-]*$/),
78+ label: z.string(),
79+ description: z.string().optional(),
80+ /** provenance of the list (URL of the index constituents page, Wikidata query…) */
81+ provenance: z.string().optional(),
82+ members: z.array(coverageMemberSchema),
83+});
84+export type CoverageUniverse = z.infer<typeof coverageUniverseSchema>;
85+
86+export const coverageSectorSchema = z.object({
87+ sector: z.string().regex(/^[a-z0-9][a-z0-9-]*$/),
88+ label: z.string(),
89+ description: z.string().optional(),
90+ weight: z.number().positive().default(1),
91+ categories: z.array(z.string()).default([]),
92+ tier: z.enum(TIERS).default("B"),
93+ universes: z.array(coverageUniverseSchema),
94+});
95+export type CoverageSector = z.infer<typeof coverageSectorSchema>;
96+
97+/**
98+ * Registrable domain (eTLD+1) with a compact public-suffix list covering the multi-label suffixes that
99+ * matter for official sources (governments, universities, companies). Good enough to match a sensor host
100+ * (`ir.example.com`, `www.gov.uk`) to a universe member; not a full PSL.
101+ */
102+const MULTI_SUFFIX = new Set([
103+ "co.uk", "org.uk", "gov.uk", "ac.uk", "nhs.uk", "police.uk", "ltd.uk", "plc.uk", "me.uk", "net.uk", "sch.uk",
104+ "com.au", "net.au", "org.au", "gov.au", "edu.au", "id.au", "asn.au",
105+ "gc.ca", "qc.ca", "on.ca", "bc.ca", "ab.ca", "mb.ca", "sk.ca", "ns.ca", "nb.ca", "nl.ca", "pe.ca", "nt.ca", "nu.ca", "yk.ca", "gouv.qc.ca",
106+ "co.jp", "ne.jp", "or.jp", "go.jp", "ac.jp", "ad.jp", "lg.jp", "ed.jp",
107+ "co.kr", "go.kr", "or.kr", "ac.kr", "re.kr", "ne.kr",
108+ "com.cn", "gov.cn", "org.cn", "edu.cn", "net.cn", "ac.cn",
109+ "com.hk", "gov.hk", "org.hk", "edu.hk",
110+ "com.tw", "gov.tw", "org.tw", "edu.tw",
111+ "com.sg", "gov.sg", "org.sg", "edu.sg",
112+ "co.in", "gov.in", "nic.in", "org.in", "ac.in", "net.in", "res.in", "edu.in",
113+ "com.br", "gov.br", "org.br", "edu.br", "net.br", "leg.br", "jus.br", "mil.br",
114+ "com.mx", "gob.mx", "org.mx", "edu.mx",
115+ "com.ar", "gob.ar", "gov.ar", "org.ar", "edu.ar",
116+ "gob.cl", "gov.cl", "cl", "gob.pe", "gob.es", "gov.co", "gob.ec", "gob.bo", "gub.uy", "gob.pa", "gob.do", "gob.gt", "gob.hn", "gob.sv", "gob.ni", "gob.cu", "gob.ve", "gob.py",
117+ "co.za", "gov.za", "org.za", "ac.za", "net.za",
118+ "co.nz", "govt.nz", "org.nz", "ac.nz", "net.nz",
119+ "com.tr", "gov.tr", "org.tr", "edu.tr",
120+ "co.il", "gov.il", "org.il", "ac.il", "muni.il",
121+ "gov.ae", "ac.ae", "co.ae",
122+ "gov.sa", "com.sa", "edu.sa", "org.sa",
123+ "gov.qa", "com.qa", "gov.kw", "gov.bh", "gov.om", "gov.jo", "gov.lb", "gov.eg", "com.eg", "gov.ma", "gov.dz", "gov.tn",
124+ "gov.ng", "com.ng", "org.ng", "go.ke", "co.ke", "or.ke", "ac.ke", "gov.gh", "com.gh", "gov.et", "gov.rw", "go.tz", "co.tz", "go.ug", "co.ug", "gov.zm", "gov.zw", "gov.bw", "gov.na", "gov.mz", "gov.sn", "gouv.sn", "gouv.ci", "gouv.bf", "gouv.ml", "gouv.ne", "gouv.tg", "gouv.bj", "gouv.cm", "gov.mu",
125+ "com.my", "gov.my", "org.my", "edu.my", "go.th", "co.th", "or.th", "ac.th", "gov.vn", "com.vn", "edu.vn", "gov.ph", "com.ph", "org.ph", "edu.ph", "go.id", "co.id", "or.id", "ac.id", "gov.bd", "com.bd", "gov.pk", "com.pk", "edu.pk", "gov.lk", "gov.np", "gov.kh", "gov.la", "gov.mm", "gov.bn", "gov.mn",
126+ "gov.kz", "gov.uz", "gov.kg", "gov.tj", "gov.tm", "gov.az", "gov.ge", "gov.am", "gov.by", "gov.ua", "com.ua", "org.ua", "edu.ua", "gov.md", "gov.ru", "com.ru", "org.ru", "edu.ru", "ac.ru",
127+ "gov.pl", "com.pl", "org.pl", "edu.pl", "gov.cz", "gov.sk", "gov.hu", "gov.ro", "gov.bg", "gov.hr", "gov.si", "gov.rs", "gov.ba", "gov.mk", "gov.al", "gov.me", "gov.lv", "gov.lt", "gov.ee", "gov.gr", "gov.cy", "gov.mt", "gov.ie", "gov.pt", "gov.it", "gouv.fr", "gov.lu", "gov.be", "gov.nl", "gov.is", "gov.li",
128+ "com.es", "org.es", "edu.es", "gob.es", "com.pt", "edu.pt", "co.it", "edu.it", "com.gr", "edu.gr", "com.pl", "com.ro", "com.tr",
129+ "europa.eu",
130+ "gov.uk", "parliament.uk", "judiciary.uk", "mod.uk", "nhs.uk", "royal.uk",
131+ "state.gov", "mil", "fed.us", "nsn.us", "k12.us", "lib.us", "dni.us", "isa.us", "kids.us",
132+ "ac.at", "co.at", "gv.at", "or.at", "ac.be", "co.nl", "ac.nz", "ac.uk", "ac.jp", "edu.au", "edu.sg", "edu.hk",
133+ "org.au", "com.au", "co.ke",
134+]);
135+
136+export function registrableDomain(host: string): string {
137+ const h = host.toLowerCase().replace(/\.$/, "").replace(/^www\./, "");
138+ const parts = h.split(".");
139+ if (parts.length <= 2) return h;
140+ const last2 = parts.slice(-2).join(".");
141+ const last3 = parts.slice(-3).join(".");
142+ if (MULTI_SUFFIX.has(last3) && parts.length >= 4) return parts.slice(-4).join(".");
143+ if (MULTI_SUFFIX.has(last2)) return last3;
144+ return last2;
145+}
146+
147+/** Normalize a member/source domain to its comparison key. */
148+export function coverageKey(domain: string): string {
149+ return registrableDomain(domain.trim().toLowerCase().replace(/^https?:\/\//, "").replace(/\/.*$/, ""));
150+}
modified packages/core/src/index.ts +1 −0
@@ -11,3 +11,4 @@ export * from "./types";
11 export * from "./semantic"; 11 export * from "./semantic";
12 export * from "./search"; 12 export * from "./search";
13 export * from "./registry-schema"; 13 export * from "./registry-schema";
14+export * from "./coverage-schema";
added packages/db/migrations/0007_factory.sql +73 −0
@@ -0,0 +1,73 @@
1+-- WebSensor 0.3 — Source Factory & coverage (2026-09-13). Additive only.
2+
3+-- ---- Sources: origin (seed = YAML registry · import = admin import · factory = Source Factory) ----
4+alter table sources add column if not exists origin text not null default 'seed';
5+alter table sources add column if not exists sector text; -- coverage sector that produced the source
6+create index if not exists sources_origin_idx on sources (origin);
7+create index if not exists sources_domain_idx on sources (domain);
8+
9+-- ---- Sensors: SHADOW status (observed, evidence stored, never published) ----------------------
10+create index if not exists sensors_status_idx on sensors (status);
11+create index if not exists sensors_url_host_idx on sensors ((lower(split_part(split_part(url, '/', 3), ':', 1))));
12+
13+-- ---- Factory seeds: organizations to expand -------------------------------------------------
14+create table if not exists factory_seeds (
15+ id text primary key, -- kebab id; becomes the source id when a new source is created
16+ name text not null,
17+ domain text not null,
18+ homepage text,
19+ categories text[] not null default '{}',
20+ country text,
21+ language text,
22+ tier text not null default 'B',
23+ weight real not null default 1,
24+ importance smallint not null default 2, -- 1 long tail · 2 notable · 3 systemic
25+ aliases text[] not null default '{}',
26+ first_party boolean not null default true,
27+ sector text,
28+ universe text,
29+ hints jsonb not null default '{}',
30+ status text not null default 'queued', -- queued | discovering | discovered | blocked | error | done
31+ attempts integer not null default 0,
32+ source_id text, -- existing or created source
33+ candidates integer not null default 0,
34+ shadow integer not null default 0,
35+ accepted integer not null default 0,
36+ rejected integer not null default 0,
37+ last_error text,
38+ discovered_at timestamptz,
39+ created_at timestamptz not null default now(),
40+ updated_at timestamptz not null default now()
41+);
42+create index if not exists factory_seeds_status_idx on factory_seeds (status, importance desc, created_at);
43+create index if not exists factory_seeds_domain_idx on factory_seeds (domain);
44+create index if not exists factory_seeds_sector_idx on factory_seeds (sector);
45+
46+-- ---- Discovery candidates: full lifecycle (candidate → shadow → accepted | rejected | duplicate | blocked) ----
47+alter table discovery_candidates add column if not exists seed_id text;
48+alter table discovery_candidates add column if not exists connector text;
49+alter table discovery_candidates add column if not exists type text;
50+alter table discovery_candidates add column if not exists name text;
51+alter table discovery_candidates add column if not exists kind_class text; -- news press blog ir filings changelog releases security pricing legal careers docs api status sitemap models data posture other
52+alter table discovery_candidates add column if not exists config jsonb not null default '{}';
53+alter table discovery_candidates add column if not exists tier text;
54+alter table discovery_candidates add column if not exists score_value real;
55+alter table discovery_candidates add column if not exists reason text;
56+alter table discovery_candidates add column if not exists shadow_sensor_id text;
57+alter table discovery_candidates add column if not exists decided_at timestamptz;
58+alter table discovery_candidates add column if not exists updated_at timestamptz not null default now();
59+create index if not exists discovery_candidates_status_idx on discovery_candidates (status, score_value desc);
60+create index if not exists discovery_candidates_seed_idx on discovery_candidates (seed_id);
61+create index if not exists discovery_candidates_shadow_idx on discovery_candidates (shadow_sensor_id) where shadow_sensor_id is not null;
62+
63+-- ---- Factory daily counters (funnel) ----------------------------------------------------------
64+create table if not exists factory_daily (
65+ day date primary key,
66+ seeds_processed integer not null default 0,
67+ requests integer not null default 0,
68+ candidates integer not null default 0,
69+ shadow_created integer not null default 0,
70+ accepted integer not null default 0,
71+ rejected integer not null default 0,
72+ blocked integer not null default 0
73+);
modified packages/db/src/schema.ts +46 −0
@@ -25,6 +25,9 @@ export const sources = pgTable("sources", {
25 language: text("language"), 25 language: text("language"),
26 kind: text("kind").notNull().default("registry"), 26 kind: text("kind").notNull().default("registry"),
27 ownerToken: text("owner_token"), 27 ownerToken: text("owner_token"),
28+ /** seed (YAML registry) · import (admin) · factory (Source Factory) */
29+ origin: text("origin").notNull().default("seed"),
30+ sector: text("sector"),
28 createdAt: ts("created_at").notNull().defaultNow(), 31 createdAt: ts("created_at").notNull().defaultNow(),
29 updatedAt: ts("updated_at").notNull().defaultNow(), 32 updatedAt: ts("updated_at").notNull().defaultNow(),
30 }); 33 });
@@ -419,7 +422,50 @@ export const discoveryCandidates = pgTable("discovery_candidates", {
419 score: jsonb("score").$type<Record<string, unknown>>().notNull().default({}), 422 score: jsonb("score").$type<Record<string, unknown>>().notNull().default({}),
420 status: text("status").notNull().default("candidate"), 423 status: text("status").notNull().default("candidate"),
421 foundAt: ts("found_at").notNull().defaultNow(), 424 foundAt: ts("found_at").notNull().defaultNow(),
425+ seedId: text("seed_id"),
426+ connector: text("connector"),
427+ type: text("type"),
428+ name: text("name"),
429+ kindClass: text("kind_class"),
430+ config: jsonb("config").$type<Record<string, unknown>>().notNull().default({}),
431+ tier: text("tier"),
432+ scoreValue: real("score_value"),
433+ reason: text("reason"),
434+ shadowSensorId: text("shadow_sensor_id"),
435+ decidedAt: ts("decided_at"),
436+ updatedAt: ts("updated_at").notNull().defaultNow(),
437+});
438+
439+export const factorySeeds = pgTable("factory_seeds", {
440+ id: text("id").primaryKey(),
441+ name: text("name").notNull(),
442+ domain: text("domain").notNull(),
443+ homepage: text("homepage"),
444+ categories: text("categories").array().notNull().default([]),
445+ country: text("country"),
446+ language: text("language"),
447+ tier: text("tier").notNull().default("B"),
448+ weight: real("weight").notNull().default(1),
449+ importance: smallint("importance").notNull().default(2),
450+ aliases: text("aliases").array().notNull().default([]),
451+ firstParty: boolean("first_party").notNull().default(true),
452+ sector: text("sector"),
453+ universe: text("universe"),
454+ hints: jsonb("hints").$type<Record<string, unknown>>().notNull().default({}),
455+ status: text("status").notNull().default("queued"),
456+ attempts: integer("attempts").notNull().default(0),
457+ sourceId: text("source_id"),
458+ candidates: integer("candidates").notNull().default(0),
459+ shadow: integer("shadow").notNull().default(0),
460+ accepted: integer("accepted").notNull().default(0),
461+ rejected: integer("rejected").notNull().default(0),
462+ lastError: text("last_error"),
463+ discoveredAt: ts("discovered_at"),
464+ createdAt: ts("created_at").notNull().defaultNow(),
465+ updatedAt: ts("updated_at").notNull().defaultNow(),
422 }); 466 });
467+export type FactorySeed = typeof factorySeeds.$inferSelect;
468+export type DiscoveryCandidate = typeof discoveryCandidates.$inferSelect;
423 469
424 export const metricsDaily = pgTable("metrics_daily", { 470 export const metricsDaily = pgTable("metrics_daily", {
425 day: date("day", { mode: "string" }).primaryKey(), 471 day: date("day", { mode: "string" }).primaryKey(),
426 472
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere