WebSensor 0.3 — Source Factory & Global Observation Coverage Score · 03c8dc1 · websensor

WebSensor 0.3 — Source Factory & Global Observation Coverage Score

- Deep discovery (robots → sitemaps → feeds incl. official sub-domains → navigation-classified pages → status providers → GitHub org → EDGAR → Hugging Face → OpenAPI → posture), every candidate validated, wildcard DNS / anti-bot detection, content de-duplication, expected change frequency (median gap)
- Factory: coverage universes → factory_seeds → scored candidates (explainable) → promotion caps → SHADOW sensors (polled, never published) → evidence-based acceptance/rejection; separate websensor-factory process with Redis heartbeat; CLI expand/factory seed|run|evaluate|stats|export|requeue; YAML export for graduation
- Migration 0007 (sources.origin/sector, factory_seeds, candidate lifecycle columns, factory_daily); pipeline keeps SHADOW status and skips events for shadow sensors
- Coverage: sector universes schema + validator (--dns), registrableDomain(), GET /api/v1/coverage[/:sector], pages /coverage and /coverage/[sector], homepage rail, stats exclude shadow sensors
- Admin API /api/v1/admin/factory* + /ops Source Factory panel (funnel, sectors, candidates, accept/reject)
- Docs: COVERAGE.md, CLAUDE.md, deploy README, ARCHITECTURE, README; manifest gains websensor-factory + factory seed hook; version 0.3.0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 1 mo ago (Sep 13, 2026) parent 938aab1 Browse files

41 changed files +3,084 −173

modified .env.example +13 −0

		`@@ -42,3 +42,16 @@ WS_RETENTION_CHANGED_RAW_DAYS=60`
42	42	WS_ADMIN_TOKEN=
43	43	# Custom URL monitors per anonymous owner (spec §45)
44	44	WS_MONITORS_PER_OWNER=5
	45	+
	46	+# Source Factory (apps/engine/src/factory-main.ts — separate `websensor-factory` process). Seeds come from
	47	+# config/coverage/.yaml (members not yet monitored, or carrying hints) and config/factory/seeds/.yaml.
	48	+WS_FACTORY=1
	49	+WS_FACTORY_CONCURRENCY=6 # organizations discovered in parallel
	50	+WS_FACTORY_BUDGET=90 # HTTP requests per organization
	51	+WS_FACTORY_MIN_SCORE=0.35 # candidate score needed to enter shadow monitoring
	52	+WS_FACTORY_SHADOW_MIN_CHECKS=5 # checks before a shadow sensor can be accepted
	53	+WS_FACTORY_SHADOW_MIN_HOURS=24
	54	+WS_FACTORY_SHADOW_MAX_HOURS=120
	55	+WS_FACTORY_TICK_SECONDS=20
	56	+WS_COVERAGE_DIR=./config/coverage
	57	+# GITHUB_TOKEN= # optional: lets the factory rank an organization's repositories via the GitHub API

modified CLAUDE.md +36 −6

		`@@ -5,7 +5,7 @@ Web. It monitors official public sources, detects meaningful changes, classifies`
5	5	entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score,
6	6	preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through
7	7	a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file
8		−is the working guide for the code (v0.2, 2026-09-11).
	8	+is the working guide for the code (v0.3, 2026-09-13: Source Factory + Global Observation Coverage Score).
9	9
10	10	## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)
11	11	- `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups,
		`@@ -31,6 +31,30 @@ is the working guide for the code (v0.2, 2026-09-11).`
31	31	(`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) +
32	32	`discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and
33	33	`{field.path}`.
	34	+- Source Factory (`apps/engine/src/factory/`, process `apps/engine/src/factory-main.ts` = PM2 `websensor-factory`) —
	35	+ the system grows by itself: coverage universes (`config/coverage/*.yaml`, one sector per file, members = organizations
	36	+ with a verified domain + `hints`) and `config/factory/seeds/*.yaml` become `factory_seeds`; `runFactoryBatch()` claims
	37	+ seeds (systemic first) and runs deep discovery (`packages/connectors/src/discovery-deep.ts`: robots → sitemaps →
	38	+ feeds incl. official sub-domains → navigation-classified pages (news/press/IR/changelog/security/pricing/legal/careers/
	39	+ leadership/docs) → status-page providers → GitHub org repos → EDGAR (cik) → Hugging Face (hf_author) → OpenAPI →
	40	+ posture; every candidate is fetched and parsed, wildcard DNS detected, feeds/pages de-duplicated by content) →
	41	+ `score.ts` (evidence × page-class weight × organization importance × first-party confidence × change frequency −
	42	+ fetch cost, explainable reasons; caps per organization/class) → shadow sensors (`sensors.status = 'SHADOW'`,
	43	+ `config.factory/shadow/kind/seedId`, priority 3: polled, snapshots + changes stored, never published; the pipeline
	44	+ and `handleMissing` skip event creation) → `shadow.ts` `evaluateShadows()` every 15 min (reject: ≥ 50 % errors / all
	45	+ changes noise / duplicate content of an active sensor; accept after ≥ 5 checks and 24 h → `ACTIVE` with its real
	46	+ priority; defer otherwise, max 120 h). Sources created by the factory carry `origin = 'factory'` + `sector`; attached
	47	+ ones reuse the registry source (match by registrable domain, `coverageKey()`). Candidates and decisions live in
	48	+ `discovery_candidates` (status candidate / shadow / accepted / rejected / duplicate, `reason`, `score`); funnel in
	49	+ `factory_daily`; heartbeat Redis `ws:factory:status`. `cli.ts expand <domain> [--hints=json]` = dry run;
	50	+ `cli.ts factory seed\|run\|evaluate\|stats\|export\|requeue`; `factory export` writes a reviewable YAML fragment
	51	+ (graduating accepted sensors into `config/sources.d/` hands them to the registry sync).
	52	+- Coverage (`apps/api/src/coverage.ts`, `GET /api/v1/coverage[/:sector]`, pages `/coverage`, `/coverage/[sector]`,
	53	+ homepage rail) — Global Observation Coverage Score: per sector `breadth` (importance-weighted share of universe
	54	+ members with ≥ 1 active non-shadow sensor on their registrable domain), `depth` (min(1, sensors/5)),
	55	+ `score = 100·(0.7·breadth + 0.3·depth)`, global = sector-weight-weighted mean. Universes are the denominator (with
	56	+ `provenance`), never the registry. Validator: `apps/engine/src/coverage-validate.ts [file] [--dns]`. Guide:
	57	+ `docs/registry/COVERAGE.md`. Schema shared in `packages/core/src/coverage-schema.ts` (+ `registrableDomain()`).
34	58	- `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency,
35	59	domain circuit breaker, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff →
36	60	heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias
		`@@ -38,8 +62,8 @@ is the working guide for the code (v0.2, 2026-09-11).`
38	62	with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook
39	63	delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery,
40	64	connector health rollups, retention (`retention.ts`), Prometheus metrics :8262.
41		− CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`,
42		− `refresh-clusters`, `llm-test`); `src/validate.ts` (registry validator).
	65	+ CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `expand`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`,
	66	+ `refresh-clusters`, `llm-test`, `factory …`); `src/validate.ts` (registry validator); `src/coverage-validate.ts`.
43	67	- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped:
44	68	watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts`
45	69	behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models
		`@@ -52,8 +76,8 @@ is the working guide for the code (v0.2, 2026-09-11).`
52	76	(intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters,
53	77	pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar`
54	78	`/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]`
55		− `/country/[slug]` `/event/[slug]` (+ OG image) `/bookmarks` `/watchlists` `/alerts` `/monitors` `/ops` `/health`
56		− `/api`.
	79	+ `/country/[slug]` `/event/[slug]` (+ OG image) `/coverage` `/coverage/[sector]` `/bookmarks` `/watchlists` `/alerts`
	80	+ `/monitors` `/ops` (+ Source Factory panel: funnel, sectors, candidates, accept/reject) `/health` `/api`.
57	81	- `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order
58	82	(`extend: true` adds to an earlier source). Fragments 10–35 = site classes; 40–46 (2026-09-11) = depth: AI
59	83	frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport /
		`@@ -87,6 +111,11 @@ is the working guide for the code (v0.2, 2026-09-11).`
87	111	`llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works.
88	112	- Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes
89	113	`page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks.
	114	+- Shadow is not production: `status = 'SHADOW'` sensors never create events, never count in public stats
	115	+ (`stats.sensors`, coverage) and never compete with production sensors (priority 3). Nothing the Factory finds is
	116	+ published before `evaluateShadows()` (or an operator) accepts it. Public source pages show shadow sensors with a SHADOW pill.
	117	+- Coverage universes are lists of real organizations with verified domains (`--dns`), provenance and retrieval dates —
	118	+ never guessed domains; several files may share a sector key (universes are merged).
90	119	- Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public
91	120	feeds (`sources.kind = 'custom'`).
92	121	- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…).
		`@@ -108,6 +137,7 @@ Run CLI/engine from the repo root. A copy of production data for UI work:`
108	137	`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json`
109	138	(secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260,
110	139	public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine`
111		−(metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that
	140	+(metrics :8262), `websensor-factory` (Source Factory, heartbeat `ws:factory:status`; post-sync hook `factory seed --mode=hinted`
	141	+re-seeds from the coverage universes at every deploy). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that
112	142	changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See
113	143	`deploy/README.md`.

modified README.md +31 −0

		`@@ -94,6 +94,7 @@ threat-intelligence desk*: dense information, one-pixel borders, small badges, p`
94	94	(OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │
95	95	MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │
96	96	│ websensor-engine scheduler + pipeline, Prometheus :8262 │
	97	+ │ websensor-factory Source Factory: seeds → discovery → shadow │
97	98	│ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │
98	99	└───────────────────────────────────────────────────────────────────────┘
99	100	```
		`@@ -105,6 +106,8 @@ threat-intelligence desk*: dense information, one-pixel borders, small badges, p`
105	106	\| Evidence \| `packages/store` \| content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres \|
106	107	\| Acquisition \| `packages/connectors` \| connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback \|
107	108	\| Ingestion \| `apps/engine` \| scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator \|
	109	+\| Expansion \| `apps/engine/src/factory` \| Source Factory (0.3): coverage universes → deep discovery → scored candidates → shadow sensors → evidence-based acceptance; `websensor-factory` process \|
	110	+\| Coverage \| `apps/api/src/coverage.ts` \| Global Observation Coverage Score per sector (`/coverage`, `GET /api/v1/coverage`): how much of the world's high-value public web is observed \|
108	111	\| Gateway \| `apps/api` \| REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API \|
109	112	\| Product \| `apps/web` \| Next.js 16 App Router, design system, live feed, drawer, palette, desks \|
110	113
		@@ -493,6 +496,34 @@ without ever deleting event evidence. Details: `deploy/README.md`, `docs/ARCHITE
493	496	- Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting
494	497	on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP.
495	498
	499	+## Source Factory and coverage (0.3, 2026-09-13)
	500	+
	501	+WebSensor no longer grows by hand-written lists alone. `config/coverage/*.yaml` declares, sector by sector, the
	502	+organizations that should be observed (listed companies by index, governments of every country by institutional
	503	+role, AI, cloud, cybersecurity, markets, science, healthcare, energy, transport, telecom, logistics, open source,
	504	+commerce), each with a verified domain, a country, an importance and optional hints (CIK, GitHub organization,
	505	+Hugging Face author, status page, explicit URLs). Those universes are the denominator of the **Global Observation
	506	+Coverage Score** (`breadth` = importance-weighted share of members with an active sensor, `depth` = sensors per
	507	+organization, `score = 100·(0.7·breadth + 0.3·depth)`) and the input of the Source Factory:
	508	+
	509	+```
	510	+seed (organization + hints)
	511	+ → deep discovery robots · sitemaps · feeds (+ official sub-domains) · navigation-classified pages
	512	+ (news · press · IR · changelog · security · pricing · legal · careers · leadership · docs)
	513	+ · status-page providers · GitHub org repositories · EDGAR · Hugging Face · OpenAPI · posture
	514	+ → validation every candidate fetched and parsed by its connector; wildcard DNS and anti-bot detected;
	515	+ duplicates removed by content (shared feed items, identical page text)
	516	+ → scoring evidence × page class × organization importance × first-party confidence × change
	517	+ frequency − fetch cost, with explainable reasons; caps per organization and per class
	518	+ → shadow sensors polled like any sensor, evidence stored, nothing published
	519	+ → evaluation reject (errors, pure noise, duplicate of an active sensor) · accept (≥ 5 checks, 24 h) · defer
	520	+ → production accepted sensors publish; `cli.ts factory export` graduates them into a registry fragment
	521	+```
	522	+
	523	+`cli.ts expand <domain>` shows what the Factory would find for one organization; `/ops` has the funnel, per-sector
	524	+progress and a candidate review table; `/coverage` and `/coverage/<sector>` show the score and every member's state
	525	+(observed · shadow · queued · not seeded).
	526	+
496	527	## Roadmap
497	528
498	529	World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel

modified apps/api/src/config.ts +1 −1

		`@@ -14,5 +14,5 @@ export const config = {`
14	14	adminToken: env.WS_ADMIN_TOKEN ?? "",
15	15	/** Custom URL monitors per anonymous owner (spec §45). */
16	16	monitorsPerOwner: Number(env.WS_MONITORS_PER_OWNER ?? 5),
17		− version: "0.2.0",
	17	+ version: "0.3.0",
18	18	};

added apps/api/src/coverage.ts +236 −0

		`@@ -0,0 +1,236 @@`
	1	+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
	2	+import { join } from "node:path";
	3	+import YAML from "yaml";
	4	+import { coverageKey, coverageSectorSchema, type CoverageMember, type CoverageSector } from "@websensor/core";
	5	+import { db, sql } from "@websensor/db";
	6	+import { cached } from "./cache";
	7	+
	8	+/**
	9	+ * Global Observation Coverage Score — "how much of the world's high-value public web do we observe?"
	10	+ *
	11	+ * For every sector the universes in `config/coverage/.yaml` are the denominator. A member is covered* when at
	12	+ * least one active (non-shadow) sensor lives on its registrable domain (or on a host declared in `hints.hosts`).
	13	+ * breadth = Σ w·covered / Σ w (w = member importance 1–3)
	14	+ * depth = Σ w·min(1, sensors/5) / Σ w (five sensors per organization = full depth)
	15	+ * score = 100 · (0.7·breadth + 0.3·depth)
	16	+ * The global score is the sector-weight-weighted mean. Everything is explainable: the per-member table shows
	17	+ * which sensors count, what is in shadow and what the Factory is still queuing.
	18	+ */
	19	+export interface MemberCoverage {
	20	+ name: string;
	21	+ domain: string;
	22	+ country: string \| null;
	23	+ importance: number;
	24	+ universe: string;
	25	+ covered: boolean;
	26	+ sensors: number;
	27	+ shadow: number;
	28	+ source_id: string \| null;
	29	+ seed_status: string \| null;
	30	+ depth: number;
	31	+}
	32	+export interface UniverseCoverage {
	33	+ key: string;
	34	+ label: string;
	35	+ provenance: string \| null;
	36	+ members: number;
	37	+ covered: number;
	38	+ sensors: number;
	39	+ breadth: number;
	40	+ depth: number;
	41	+ score: number;
	42	+}
	43	+export interface SectorCoverage {
	44	+ sector: string;
	45	+ label: string;
	46	+ description: string \| null;
	47	+ weight: number;
	48	+ members: number;
	49	+ covered: number;
	50	+ shadow_members: number;
	51	+ queued: number;
	52	+ sensors: number;
	53	+ breadth: number;
	54	+ depth: number;
	55	+ score: number;
	56	+ universes: UniverseCoverage[];
	57	+}
	58	+
	59	+const COVERAGE_DIR = process.env.WS_COVERAGE_DIR ?? "./config/coverage";
	60	+
	61	+let fileCache: { at: number; sectors: CoverageSector[]; issues: string[] } \| null = null;
	62	+export function loadSectors(): { sectors: CoverageSector[]; issues: string[] } {
	63	+ if (fileCache && Date.now() - fileCache.at < 60_000) return fileCache;
	64	+ const byKey = new Map<string, CoverageSector>();
	65	+ const issues: string[] = [];
	66	+ if (existsSync(COVERAGE_DIR) && statSync(COVERAGE_DIR).isDirectory()) {
	67	+ for (const f of readdirSync(COVERAGE_DIR).filter((x) => /\.ya?ml$/.test(x) && !x.startsWith("_")).sort()) {
	68	+ try {
	69	+ const parsed = coverageSectorSchema.safeParse(YAML.parse(readFileSync(join(COVERAGE_DIR, f), "utf8")));
	70	+ if (!parsed.success) {
	71	+ issues.push(`${f}: ${parsed.error.issues[0]?.path.join(".")}: ${parsed.error.issues[0]?.message}`);
	72	+ continue;
	73	+ }
	74	+ const s = parsed.data;
	75	+ const prev = byKey.get(s.sector);
	76	+ if (prev) prev.universes.push(...s.universes);
	77	+ else byKey.set(s.sector, { ...s, universes: [...s.universes] });
	78	+ } catch (e) {
	79	+ issues.push(`${f}: ${(e as Error).message}`);
	80	+ }
	81	+ }
	82	+ }
	83	+ fileCache = { at: Date.now(), sectors: [...byKey.values()], issues };
	84	+ return fileCache;
	85	+}
	86	+
	87	+interface Observed {
	88	+ /** registrable domain → { sensors (active), shadow, sourceId } */
	89	+ byKey: Map<string, { sensors: number; shadow: number; sourceId: string \| null }>;
	90	+ seeds: Map<string, string>;
	91	+}
	92	+
	93	+async function observed(): Promise<Observed> {
	94	+ const [hosts, srcs, seeds] = await Promise.all([
	95	+ db.execute<{ host: string; status: string; n: string; source_id: string }>(sql`
	96	+ select lower(split_part(split_part(url, '/', 3), ':', 1)) as host, case when status = 'SHADOW' then 'SHADOW' else 'ACTIVE' end as status, count(*)::text as n, min(source_id) as source_id
	97	+ from sensors where enabled and url like 'http%' group by 1, 2`).then((r) => r.rows),
	98	+ db.execute<{ id: string; domain: string }>(sql`select id, domain from sources where enabled and kind = 'registry'`).then((r) => r.rows),
	99	+ db.execute<{ domain: string; status: string }>(sql`select domain, status from factory_seeds`).then((r) => r.rows),
	100	+ ]);
	101	+ const byKey = new Map<string, { sensors: number; shadow: number; sourceId: string \| null }>();
	102	+ for (const h of hosts) {
	103	+ if (!h.host) continue;
	104	+ const k = coverageKey(h.host);
	105	+ const cur = byKey.get(k) ?? { sensors: 0, shadow: 0, sourceId: null };
	106	+ if (h.status === "SHADOW") cur.shadow += Number(h.n);
	107	+ else cur.sensors += Number(h.n);
	108	+ cur.sourceId = cur.sourceId ?? h.source_id;
	109	+ byKey.set(k, cur);
	110	+ }
	111	+ for (const s of srcs) {
	112	+ const k = coverageKey(s.domain);
	113	+ const cur = byKey.get(k) ?? { sensors: 0, shadow: 0, sourceId: null };
	114	+ cur.sourceId = cur.sourceId ?? s.id;
	115	+ byKey.set(k, cur);
	116	+ }
	117	+ const seedMap = new Map<string, string>();
	118	+ for (const s of seeds) seedMap.set(coverageKey(s.domain), s.status);
	119	+ return { byKey, seeds: seedMap };
	120	+}
	121	+
	122	+function memberCoverage(m: CoverageMember, universe: string, obs: Observed): MemberCoverage {
	123	+ const keys = [coverageKey(m.domain), ...((m.hints?.hosts ?? []).map((h) => coverageKey(h)))];
	124	+ let sensors = 0;
	125	+ let shadow = 0;
	126	+ let sourceId: string \| null = null;
	127	+ for (const k of new Set(keys)) {
	128	+ const o = obs.byKey.get(k);
	129	+ if (!o) continue;
	130	+ sensors += o.sensors;
	131	+ shadow += o.shadow;
	132	+ sourceId = sourceId ?? o.sourceId;
	133	+ }
	134	+ return { name: m.name, domain: m.domain, country: m.country ?? null, importance: m.importance, universe, covered: sensors > 0, sensors, shadow, source_id: sourceId, seed_status: obs.seeds.get(coverageKey(m.domain)) ?? null, depth: Math.min(1, sensors / 5) };
	135	+}
	136	+
	137	+function aggregate(members: MemberCoverage[]): { breadth: number; depth: number; score: number; covered: number; sensors: number } {
	138	+ let w = 0;
	139	+ let cov = 0;
	140	+ let dep = 0;
	141	+ let covered = 0;
	142	+ let sensors = 0;
	143	+ for (const m of members) {
	144	+ w += m.importance;
	145	+ if (m.covered) {
	146	+ cov += m.importance;
	147	+ covered++;
	148	+ }
	149	+ dep += m.importance * m.depth;
	150	+ sensors += m.sensors;
	151	+ }
	152	+ const breadth = w ? cov / w : 0;
	153	+ const depth = w ? dep / w : 0;
	154	+ return { breadth: round(breadth), depth: round(depth), score: round(100 * (0.7 * breadth + 0.3 * depth), 1), covered, sensors };
	155	+}
	156	+const round = (x: number, d = 3): number => Math.round(x * 10 d) / 10 d;
	157	+
	158	+export async function coverageSummary(): Promise<Record<string, unknown>> {
	159	+ return cached("coverage:summary", 5 * 60_000, async () => {
	160	+ const { sectors, issues } = loadSectors();
	161	+ const obs = await observed();
	162	+ const out: SectorCoverage[] = [];
	163	+ let wsum = 0;
	164	+ let ssum = 0;
	165	+ let members = 0;
	166	+ let covered = 0;
	167	+ let sensorsTotal = 0;
	168	+ for (const s of sectors) {
	169	+ const all: MemberCoverage[] = [];
	170	+ const universes: UniverseCoverage[] = [];
	171	+ const seenDomains = new Set<string>();
	172	+ for (const u of s.universes) {
	173	+ const ms = u.members.map((m) => memberCoverage(m, u.key, obs));
	174	+ const a = aggregate(ms);
	175	+ universes.push({ key: u.key, label: u.label, provenance: u.provenance ?? null, members: ms.length, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score });
	176	+ for (const m of ms) {
	177	+ const k = coverageKey(m.domain);
	178	+ if (seenDomains.has(k)) continue;
	179	+ seenDomains.add(k);
	180	+ all.push(m);
	181	+ }
	182	+ }
	183	+ const a = aggregate(all);
	184	+ const sec: SectorCoverage = { sector: s.sector, label: s.label, description: s.description ?? null, weight: s.weight, members: all.length, covered: a.covered, shadow_members: all.filter((m) => !m.covered && m.shadow > 0).length, queued: all.filter((m) => !m.covered && (m.seed_status === "queued" \|\| m.seed_status === "discovering")).length, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, universes };
	185	+ out.push(sec);
	186	+ wsum += s.weight;
	187	+ ssum += s.weight * a.score;
	188	+ members += all.length;
	189	+ covered += a.covered;
	190	+ sensorsTotal += a.sensors;
	191	+ }
	192	+ out.sort((a, b) => b.score - a.score);
	193	+ const [monitored, factory] = await Promise.all([
	194	+ db.execute<{ sources: string; sensors: string; shadow: string; countries: string }>(sql`select (select count() from sources where enabled and kind = 'registry')::text as sources, (select count() from sensors where enabled and status <> 'SHADOW')::text as sensors, (select count(*) from sensors where enabled and status = 'SHADOW')::text as shadow, (select count(distinct country) from sources where country is not null and enabled)::text as countries`).then((r) => r.rows[0]),
	195	+ db.execute<{ status: string; n: string }>(sql`select status, count(*)::text as n from factory_seeds group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [x.status, Number(x.n)]))),
	196	+ ]);
	197	+ return {
	198	+ global_score: wsum ? round(ssum / wsum, 1) : 0,
	199	+ members,
	200	+ covered,
	201	+ sensors_matched: sensorsTotal,
	202	+ sectors: out.map(({ universes: _u, ...rest }) => rest),
	203	+ monitored: { sources: Number(monitored?.sources ?? 0), sensors: Number(monitored?.sensors ?? 0), shadow: Number(monitored?.shadow ?? 0), countries: Number(monitored?.countries ?? 0) },
	204	+ factory: factory,
	205	+ method: "breadth = importance-weighted share of members with ≥ 1 active sensor · depth = importance-weighted min(1, sensors/5) · score = 100·(0.7·breadth + 0.3·depth) · global = sector-weight-weighted mean",
	206	+ issues,
	207	+ generated_at: new Date().toISOString(),
	208	+ };
	209	+ });
	210	+}
	211	+
	212	+export async function coverageSector(sector: string): Promise<Record<string, unknown> \| null> {
	213	+ return cached(`coverage:sector:${sector}`, 5 * 60_000, async () => {
	214	+ const { sectors } = loadSectors();
	215	+ const s = sectors.find((x) => x.sector === sector);
	216	+ if (!s) return null;
	217	+ const obs = await observed();
	218	+ const universes = s.universes.map((u) => {
	219	+ const members = u.members.map((m) => memberCoverage(m, u.key, obs)).sort((a, b) => Number(b.covered) - Number(a.covered) \|\| b.importance - a.importance \|\| b.sensors - a.sensors \|\| a.name.localeCompare(b.name));
	220	+ const a = aggregate(members);
	221	+ return { key: u.key, label: u.label, description: u.description ?? null, provenance: u.provenance ?? null, members: members.length, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, items: members };
	222	+ });
	223	+ const flat = new Map<string, MemberCoverage>();
	224	+ for (const u of universes) for (const m of u.items) if (!flat.has(coverageKey(m.domain))) flat.set(coverageKey(m.domain), m);
	225	+ const a = aggregate([...flat.values()]);
	226	+ const byCountry = new Map<string, { members: number; covered: number }>();
	227	+ for (const m of flat.values()) {
	228	+ const c = m.country ?? "—";
	229	+ const cur = byCountry.get(c) ?? { members: 0, covered: 0 };
	230	+ cur.members++;
	231	+ if (m.covered) cur.covered++;
	232	+ byCountry.set(c, cur);
	233	+ }
	234	+ return { sector: s.sector, label: s.label, description: s.description ?? null, weight: s.weight, categories: s.categories, members: flat.size, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, shadow_members: [...flat.values()].filter((m) => !m.covered && m.shadow > 0).length, queued: [...flat.values()].filter((m) => !m.covered && (m.seed_status === "queued" \|\| m.seed_status === "discovering")).length, universes, by_country: [...byCountry.entries()].map(([country, v]) => ({ country, ...v })).sort((x, y) => y.members - x.members), generated_at: new Date().toISOString() };
	235	+ });
	236	+}

modified apps/api/src/live.ts +9 −0

		`@@ -49,6 +49,15 @@ export async function engineStatus(): Promise<Record<string, unknown> \| null> {`
49	49	}
50	50	}
51	51
	52	+export async function factoryStatus(): Promise<Record<string, unknown> \| null> {
	53	+ try {
	54	+ const raw = await getCmd().get("ws:factory:status");
	55	+ return raw ? (JSON.parse(raw) as Record<string, unknown>) : null;
	56	+ } catch {
	57	+ return null;
	58	+ }
	59	+}
	60	+
52	61	function getCmd(): Redis {
53	62	if (!cmd) cmd = new Redis(config.redisUrl, { maxRetriesPerRequest: 2, lazyConnect: false });
54	63	return cmd;

modified apps/api/src/queries.ts +5 −3

		`@@ -127,7 +127,9 @@ export async function stats(): Promise<Record<string, unknown>> {`
127	127	await db.execute<Record<string, unknown>>(sql`
128	128	select
129	129	(select count(*) from sources where enabled and kind = 'registry') as sources,
130		− (select count(*) from sensors where enabled) as sensors,
	130	+ (select count(*) from sensors where enabled and status <> 'SHADOW') as sensors,
	131	+ (select count(*) from sensors where enabled and status = 'SHADOW') as sensors_shadow,
	132	+ (select count(*) from sources where enabled and kind = 'registry' and origin = 'factory') as sources_factory,
131	133	(select count(*) from entities) as entities,
132	134	(select count(*) from snapshots) as snapshots,
133	135	(select count(*) from events) as events_total,
		`@@ -144,8 +146,8 @@ export async function stats(): Promise<Record<string, unknown>> {`
144	146	(select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour,
145	147	(select count(*) from sensor_runs where started_at >= now() - interval '5 minutes') as checks_last_5m,
146	148	(select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified') as not_modified_last_5m,
147		− (select count(*) from sensors where health = 'UP' and enabled) as sensors_up,
148		− (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled) as sensors_degraded,
	149	+ (select count(*) from sensors where health = 'UP' and enabled and status <> 'SHADOW') as sensors_up,
	150	+ (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled and status <> 'SHADOW') as sensors_degraded,
149	151	(select count(*) from sources where enabled and first_party and kind = 'registry') as sources_first_party,
150	152	(select count(distinct country) from sources where country is not null) as countries,
151	153	(select max(started_at) from sensor_runs) as last_check_at,

modified apps/api/src/routes-admin.ts +76 −1

		`@@ -8,7 +8,7 @@ import { getConnector, listConnectors, NormalizeError } from "@websensor/connect`
8	8	import { db, sql } from "@websensor/db";
9	9	import { cacheStats, invalidate } from "./cache";
10	10	import { config } from "./config";
11		−import { engineStatus, liveStats } from "./live";
	11	+import { engineStatus, factoryStatus, liveStats } from "./live";
12	12
13	13	/**
14	14	* Internal operations (spec §70–72, §85). Enabled only when WS_ADMIN_TOKEN is set; every request
		`@@ -193,6 +193,81 @@ export async function registerAdminRoutes(app: FastifyInstance): Promise<void> {`
193	193	return { dry_run: false, imported_sources: valid.filter((v) => !v.extend).length, extended_sources: valid.filter((v) => v.extend).length, sensors, invalid: report.filter((r) => !r.ok).length, report };
194	194	});
195	195
	196	+ // ---- Source Factory ---------------------------------------------------------------------------------
	197	+ app.get("/api/v1/admin/factory", async () => {
	198	+ const [seeds, cands, shadows, daily, sectors, recent, hb, ready] = await Promise.all([
	199	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from factory_seeds group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [String(x.status), Number(x.n)]))),
	200	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from discovery_candidates group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [String(x.status), Number(x.n)]))),
	201	+ db.execute<Record<string, unknown>>(sql`select count()::int as total, count() filter (where health = 'UP')::int as up, count() filter (where health <> 'UP')::int as unhealthy, count() filter (where total_runs >= 5)::int as ready, coalesce(sum(raw_changes),0)::int as raw_changes from sensors where status = 'SHADOW' and enabled`).then((r) => r.rows[0]),
	202	+ db.execute<Record<string, unknown>>(sql`select day::text as day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked from factory_daily order by day desc limit 30`).then((r) => r.rows),
	203	+ db.execute<Record<string, unknown>>(sql`select coalesce(sector, '-') as sector, count()::int as seeds, count() filter (where status = 'queued')::int as queued, count() filter (where status = 'discovering')::int as discovering, count() filter (where status in ('discovered','done'))::int as discovered, count() filter (where status = 'blocked')::int as blocked, count() filter (where status = 'error')::int as errors, coalesce(sum(shadow),0)::int as shadow, coalesce(sum(accepted),0)::int as accepted, coalesce(sum(rejected),0)::int as rejected from factory_seeds group by 1 order by seeds desc`).then((r) => r.rows),
	204	+ db.execute<Record<string, unknown>>(sql`select id, name, domain, sector, status, candidates, shadow, accepted, rejected, last_error, discovered_at, source_id from factory_seeds where status <> 'queued' order by updated_at desc limit 40`).then((r) => r.rows),
	205	+ factoryStatus(),
	206	+ db.execute<Record<string, unknown>>(sql`select kind_class, count(*)::int as n from discovery_candidates where status = 'shadow' group by 1 order by n desc`).then((r) => r.rows),
	207	+ ]);
	208	+ return { seeds, candidates: cands, shadow: shadows, shadow_by_kind: ready, daily, sectors, recent_seeds: recent, heartbeat: hb, generated_at: new Date().toISOString() };
	209	+ });
	210	+ app.get<{ Querystring: { status?: string; sector?: string; q?: string; limit?: string } }>("/api/v1/admin/factory/seeds", async (req) => {
	211	+ const lim = Math.min(1000, Number(req.query.limit ?? 200));
	212	+ const rows = await db.execute<Record<string, unknown>>(sql`select * from factory_seeds where 1=1 ${req.query.status ? sql`and status = ${req.query.status}` : sql``} ${req.query.sector ? sql`and sector = ${req.query.sector}` : sql``} ${req.query.q ? sql`and (name ilike ${"%" + req.query.q + "%"} or domain ilike ${"%" + req.query.q + "%"})` : sql``} order by importance desc, updated_at desc limit ${lim}`);
	213	+ return { items: rows.rows };
	214	+ });
	215	+ /** Bulk seeds `{ seeds: [{ name, domain, categories?, country?, tier?, importance?, aliases?, sector?, universe?, hints? }] }` — queued for the factory process. */
	216	+ app.post("/api/v1/admin/factory/seeds", async (req) => {
	217	+ const seedSchema = z.object({ id: z.string().regex(/^[a-z0-9][a-z0-9-]*$/).optional(), name: z.string().min(1), domain: z.string().min(3), categories: z.array(z.string()).default([]), country: z.string().regex(/^[A-Z]{2,3}$/).optional(), language: z.string().optional(), tier: z.enum(["S", "A", "B", "C", "D"]).default("B"), importance: z.number().int().min(1).max(3).default(2), aliases: z.array(z.string()).default([]), first_party: z.boolean().default(true), sector: z.string().optional(), universe: z.string().optional(), hints: z.record(z.string(), z.unknown()).default({}) });
	218	+ const body = z.object({ seeds: z.array(seedSchema).min(1).max(2000) }).parse(req.body ?? {});
	219	+ let n = 0;
	220	+ for (const s of body.seeds) {
	221	+ const domain = s.domain.toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "");
	222	+ const id = s.id ?? domain.replace(/[^a-z0-9]+/g, "-").replace(/^-\|-$/g, "").slice(0, 60);
	223	+ await db.execute(sql`insert into factory_seeds (id, name, domain, categories, country, language, tier, importance, aliases, first_party, sector, universe, hints, status)
	224	+ values (${id}, ${s.name}, ${domain}, ${sql.raw("'{" + s.categories.map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.country ?? null}, ${s.language ?? null}, ${s.tier}, ${s.importance}, ${sql.raw("'{" + s.aliases.map((c) => '"' + c.replace(/["\\]/g, "") + '"').join(",") + "}'::text[]")}, ${s.first_party}, ${s.sector ?? null}, ${s.universe ?? null}, ${JSON.stringify(s.hints)}::jsonb, 'queued')
	225	+ on conflict (id) do update set name = excluded.name, hints = factory_seeds.hints \|\| excluded.hints, status = 'queued', updated_at = now()`);
	226	+ n++;
	227	+ }
	228	+ invalidate("coverage");
	229	+ return { ok: true, queued: n };
	230	+ });
	231	+ app.post<{ Params: { id: string } }>("/api/v1/admin/factory/seeds/:id/requeue", async (req, reply) => {
	232	+ const r = await db.execute(sql`update factory_seeds set status = 'queued', updated_at = now() where id = ${req.params.id}`);
	233	+ if (!r.rowCount) return reply.status(404).send({ error: "not_found" });
	234	+ return { ok: true };
	235	+ });
	236	+ app.get<{ Querystring: { status?: string; seed?: string; source?: string; kind?: string; limit?: string } }>("/api/v1/admin/factory/candidates", async (req) => {
	237	+ const lim = Math.min(1000, Number(req.query.limit ?? 200));
	238	+ const rows = await db.execute<Record<string, unknown>>(sql`select c.*, s.health as shadow_health, s.total_runs as shadow_runs, s.raw_changes as shadow_changes from discovery_candidates c left join sensors s on s.id = c.shadow_sensor_id where 1=1 ${req.query.status ? sql`and c.status = ${req.query.status}` : sql``} ${req.query.seed ? sql`and c.seed_id = ${req.query.seed}` : sql``} ${req.query.source ? sql`and c.source_id = ${req.query.source}` : sql``} ${req.query.kind ? sql`and c.kind_class = ${req.query.kind}` : sql``} order by c.updated_at desc limit ${lim}`);
	239	+ return { items: rows.rows };
	240	+ });
	241	+ /** Manual decision on a shadow sensor: accept now (→ ACTIVE) or reject (→ DISABLED). */
	242	+ app.post<{ Params: { id: string; action: string } }>("/api/v1/admin/factory/shadow/:id/:action", async (req, reply) => {
	243	+ const s = (await db.execute<Record<string, unknown>>(sql`select id, config, status from sensors where id = ${req.params.id}`)).rows[0];
	244	+ if (!s) return reply.status(404).send({ error: "not_found" });
	245	+ const reason = `manual ${req.params.action} via admin API`;
	246	+ if (req.params.action === "accept") {
	247	+ const tp = Number(((s.config as Record<string, unknown>).targetPriority as number \| undefined) ?? 2);
	248	+ await db.execute(sql`update sensors set status = 'ACTIVE', enabled = true, priority = ${tp}, config = (config - 'shadow') \|\| jsonb_build_object('acceptedAt', now()::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${req.params.id}`);
	249	+ await db.execute(sql`update discovery_candidates set status = 'accepted', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${req.params.id}`);
	250	+ } else if (req.params.action === "reject") {
	251	+ await db.execute(sql`update sensors set status = 'DISABLED', enabled = false, config = config \|\| jsonb_build_object('rejectedAt', now()::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${req.params.id}`);
	252	+ await db.execute(sql`update discovery_candidates set status = 'rejected', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${req.params.id}`);
	253	+ } else return reply.status(400).send({ error: "action must be accept or reject" });
	254	+ invalidate("");
	255	+ return { ok: true };
	256	+ });
	257	+ /** Promote a plain candidate (never shadowed) straight into shadow monitoring. */
	258	+ app.post<{ Params: { id: string } }>("/api/v1/admin/factory/candidates/:id/shadow", async (req, reply) => {
	259	+ const c = (await db.execute<Record<string, unknown>>(sql`select * from discovery_candidates where id = ${req.params.id}`)).rows[0];
	260	+ if (!c) return reply.status(404).send({ error: "not_found" });
	261	+ if (!c.connector \|\| !c.type) return reply.status(400).send({ error: "candidate has no connector/type (legacy discovery row)" });
	262	+ const src = (await db.execute<Record<string, unknown>>(sql`select categories from sources where id = ${String(c.source_id)}`)).rows[0];
	263	+ const id = `${String(c.source_id)}_${String(c.name ?? c.kind_class ?? "page").toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-\|-$/g, "")}-${String(c.id).slice(-5)}`.slice(0, 80);
	264	+ await db.execute(sql`insert into sensors (id, source_id, name, url, type, connector, tier, config, status, priority, validated_at)
	265	+ values (${id}, ${String(c.source_id)}, ${String(c.name ?? c.kind_class ?? "page")}, ${String(c.url)}, ${String(c.type)}, ${String(c.connector)}, ${String(c.tier ?? "C")}, ${JSON.stringify({ ...((c.config as Record<string, unknown>) ?? {}), factory: true, shadow: true, seed: false, kind: c.kind_class, candidate: c.id, seedId: c.seed_id, shadowSince: new Date().toISOString(), targetPriority: 2, categories: src?.categories ?? [] })}::jsonb, 'SHADOW', 3, now())
	266	+ on conflict (id) do nothing`);
	267	+ await db.execute(sql`update discovery_candidates set status = 'shadow', shadow_sensor_id = ${id}, updated_at = now() where id = ${req.params.id}`);
	268	+ return { ok: true, sensor_id: id };
	269	+ });
	270	+
196	271	app.get("/api/v1/admin/failures", async () => {
197	272	const rows = await db.execute<Record<string, unknown>>(sql`select r.started_at, r.outcome, r.http_status, r.error, r.duration_ms, s.id as sensor_id, s.name, s.source_id, s.url, s.connector from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error','rate_limited') and r.started_at >= now() - interval '24 hours' order by r.started_at desc limit 200`);
198	273	return { items: rows.rows };

modified apps/api/src/routes.ts +10 −1

		`@@ -5,6 +5,7 @@ import { db, sql, textArray } from "@websensor/db";`
5	5	import { getBlobStore } from "@websensor/store";
6	6	import { cached } from "./cache";
7	7	import { config } from "./config";
	8	+import { coverageSector, coverageSummary } from "./coverage";
8	9	import { breakingDesk, categoryDesk, clusterDetail, countryDesk, countryList, entityInsights, entityRankings, pulse, radar, sourceSensors } from "./intel";
9	10	import { engineStatus, liveStats } from "./live";
10	11	import { countEvents, getEvent, listEvents, relatedEvents, sensorHistory, sourceActivity, sourceQuality, stats, trending } from "./queries";
		`@@ -139,7 +140,8 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {`
139	140	const q = req.query.q;
140	141	const rows = await db.execute<Record<string, unknown>>(sql`
141	142	select s.id, s.name, s.domain, s.homepage, s.description, s.categories, s.tier, s.importance_weight, s.enabled, s.notes, s.first_party, s.country, s.language, s.robots_checked_at,
142		− (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count,
	143	+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.status <> 'SHADOW')::int as sensor_count,
	144	+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.status = 'SHADOW')::int as shadow_count, s.origin, s.sector,
143	145	(select count(*) from events e where e.source_id = s.id)::int as event_count,
144	146	(select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h,
145	147	(select max(detected_at) from events e where e.source_id = s.id) as last_event_at,
		`@@ -247,6 +249,13 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {`
247	249
248	250	// ---- Intelligence desks ----------------------------------------------------------------------
249	251	app.get("/api/v1/stats", async () => cached("stats", 5_000, stats));
	252	+ // ---- Global Observation Coverage Score ------------------------------------------------------
	253	+ app.get("/api/v1/coverage", async () => coverageSummary());
	254	+ app.get<{ Params: { sector: string } }>("/api/v1/coverage/:sector", async (req, reply) => {
	255	+ const r = await coverageSector(req.params.sector.toLowerCase());
	256	+ if (!r) return reply.status(404).send({ error: "not_found" });
	257	+ return r;
	258	+ });
250	259	app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => {
251	260	const hours = Math.min(168, Number(req.query.hours ?? 24));
252	261	const limit = Math.min(50, Number(req.query.limit ?? 12));

modified apps/engine/src/cli.ts +49 −1

		`@@ -80,8 +80,56 @@ async function main(): Promise<void> {`
80	80	console.log(JSON.stringify(out, null, 2), `\n${Date.now() - t0} ms`);
81	81	break;
82	82	}
	83	+ case "expand": {
	84	+ // Deep discovery dry-run (no DB writes): what the Source Factory would find for a domain.
	85	+ const { discoverOrganization } = await import("@websensor/connectors");
	86	+ const { scoreCandidate } = await import("./factory/score");
	87	+ const hintsArg = args.find((a) => a.startsWith("--hints="));
	88	+ const hints = hintsArg ? (JSON.parse(hintsArg.slice(8)) as Record<string, unknown>) : {};
	89	+ const r = await discoverOrganization(args[0]!, { hints, probePosture: args.includes("--posture") });
	90	+ console.log(`${r.domain} → origin ${r.origin} · home ${r.homeStatus}${r.blocked ? " BLOCKED" : ""} · ${r.requests} requests · ${r.durationMs} ms`);
	91	+ for (const n of r.notes) console.log(` note: ${n}`);
	92	+ for (const c of r.candidates) {
	93	+ const s = scoreCandidate(c, { importance: Number(args.find((a) => a.startsWith("--importance="))?.slice(13) ?? 2) });
	94	+ console.log(`${s.score.toFixed(2)} ${c.kind.padEnd(10)} ${c.connector.padEnd(10)} ${c.suggestedTier} ${String(c.itemCount ?? "").padStart(4)} ${c.url} (${c.evidence})`);
	95	+ }
	96	+ if (r.rejected.length) console.log(`rejected: ${r.rejected.map((x) => `${x.url} [${x.reason}]`).join("\n ")}`);
	97	+ break;
	98	+ }
	99	+ case "factory": {
	100	+ const f = await import("./factory");
	101	+ const sub = args[0];
	102	+ const flagVal = (n: string): string \| undefined => args.find((a) => a.startsWith(`--${n}=`))?.slice(n.length + 3);
	103	+ if (sub === "seed") {
	104	+ const files = await f.seedFromFiles();
	105	+ const mode = (flagVal("mode") ?? "hinted") as "uncovered" \| "hinted" \| "all";
	106	+ const cov = await f.seedFromCoverage({ mode, sectors: flagVal("sector")?.split(","), requeue: args.includes("--requeue") });
	107	+ console.log(JSON.stringify({ files, coverage: cov }, null, 2));
	108	+ } else if (sub === "run") {
	109	+ const n = Number(flagVal("n") ?? 6);
	110	+ const ids = args.slice(1).filter((a) => !a.startsWith("--"));
	111	+ const out = await f.runFactoryBatch(n, ids.length ? { seedIds: ids } : {});
	112	+ for (const o of out) console.log(`${o.status.padEnd(10)} ${o.seedId.padEnd(40)} source=${o.sourceId ?? "-"} candidates=${o.candidates} shadow=${o.shadow} requests=${o.requests} ${o.durationMs} ms${o.notes.length ? " · " + o.notes.join(" · ") : ""}`);
	113	+ } else if (sub === "evaluate") {
	114	+ const r = await f.evaluateShadows({ force: args.includes("--force") });
	115	+ for (const d of r.decisions.filter((x) => x.decision !== "deferred" \|\| args.includes("--verbose"))) console.log(`${d.decision.padEnd(9)} ${d.sensorId.padEnd(60)} ${d.reason}`);
	116	+ console.log(JSON.stringify({ accepted: r.accepted, rejected: r.rejected, deferred: r.deferred }));
	117	+ } else if (sub === "stats") {
	118	+ console.log(JSON.stringify(await f.factoryStats(), null, 2));
	119	+ } else if (sub === "export") {
	120	+ process.stdout.write(await f.exportFactoryFragment({ sector: flagVal("sector"), sinceDays: flagVal("since") ? Number(flagVal("since")) : undefined, includeShadow: args.includes("--include-shadow") }));
	121	+ } else if (sub === "requeue") {
	122	+ const ids = args.slice(1).filter((a) => !a.startsWith("--"));
	123	+ const r = await db.execute(sql`update factory_seeds set status = 'queued', updated_at = now() where ${ids.length ? sql`id = any(${sql.raw("array[" + ids.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]")})` : sql`status in ('error','blocked')`}`);
	124	+ console.log(`requeued ${r.rowCount ?? 0}`);
	125	+ } else {
	126	+ console.error("usage: cli.ts factory seed [--mode=uncovered\|hinted\|all] [--sector=a,b] [--requeue] \| run [seedId…] [--n=6] \| evaluate [--force] [--verbose] \| stats \| export [--sector=x] [--since=days] [--include-shadow] \| requeue [seedId…]");
	127	+ process.exitCode = 1;
	128	+ }
	129	+ break;
	130	+ }
83	131	default:
84		− console.error("usage: cli.ts sync \| discover [sourceId…] \| probe <domain> \| run-once <sensorId> \| run-due [n] \| relink-entities [days] \| prune-blobs [batch] \| refresh-clusters");
	132	+ console.error("usage: cli.ts sync \| discover [sourceId…] \| probe <domain> \| expand <domain> [--hints=json] [--posture] \| run-once <sensorId> \| run-due [n] \| relink-entities [days] \| prune-blobs [batch] \| refresh-clusters \| factory <seed\|run\|evaluate\|stats\|export\|requeue>");
85	133	process.exitCode = 1;
86	134	}
87	135	}

modified apps/engine/src/config.ts +22 −1

		`@@ -48,7 +48,7 @@ export const config = {`
48	48	retention: { rawDays: Number(env.WS_RETENTION_RAW_DAYS ?? 21), changedRawDays: Number(env.WS_RETENTION_CHANGED_RAW_DAYS ?? 60), enabled: (env.WS_RETENTION ?? "1") !== "0" },
49	49	/** silent-change bar: only eligible types with importance ≥ this become "silent" */
50	50	silentMinImportance: Number(env.WS_SILENT_MIN_IMPORTANCE ?? 45),
51		− version: "0.2.0",
	51	+ version: "0.3.0",
52	52	};
53	53
54	54	export const log = pino({
		`@@ -56,3 +56,24 @@ export const log = pino({`
56	56	...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}),
57	57	base: { service: "websensor-engine" },
58	58	});
	59	+
	60	+/** Source Factory (apps/engine/src/factory) — organization expansion, shadow monitoring, acceptance. */
	61	+export const factoryConfig = {
	62	+ enabled: (env.WS_FACTORY ?? "1") !== "0",
	63	+ /** seeds discovered in parallel */
	64	+ concurrency: Number(env.WS_FACTORY_CONCURRENCY ?? 6),
	65	+ /** HTTP requests per organization during discovery */
	66	+ budget: Number(env.WS_FACTORY_BUDGET ?? 90),
	67	+ /** minimum candidate score to enter shadow monitoring */
	68	+ minScore: Number(env.WS_FACTORY_MIN_SCORE ?? 0.35),
	69	+ /** shadow evaluation: checks needed, minimum and maximum shadow duration */
	70	+ shadowMinChecks: Number(env.WS_FACTORY_SHADOW_MIN_CHECKS ?? 5),
	71	+ shadowMinHours: Number(env.WS_FACTORY_SHADOW_MIN_HOURS ?? 24),
	72	+ shadowMaxHours: Number(env.WS_FACTORY_SHADOW_MAX_HOURS ?? 120),
	73	+ /** per-organization promotion caps */
	74	+ caps: { feeds: 4, pages: 6, releases: 4, sitemap: 1, status: 1, models: 1, filings: 1, api: 1, posture: 3, legal: 2, total: 18 },
	75	+ /** seconds between factory ticks */
	76	+ tickSeconds: Number(env.WS_FACTORY_TICK_SECONDS ?? 20),
	77	+ coverageDir: env.WS_COVERAGE_DIR ?? "./config/coverage",
	78	+ seedsDir: env.WS_FACTORY_SEEDS_DIR ?? "./config/factory/seeds",
	79	+};

added apps/engine/src/coverage-validate.ts +108 −0

		`@@ -0,0 +1,108 @@`
	1	+import { promises as dns } from "node:dns";
	2	+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
	3	+import { join } from "node:path";
	4	+import YAML from "yaml";
	5	+import { coverageKey, coverageSectorSchema, type CoverageSector } from "@websensor/core";
	6	+
	7	+/**
	8	+ * Coverage-universe validator — no database.
	9	+ *
	10	+ * node node_modules/tsx/dist/cli.mjs apps/engine/src/coverage-validate.ts [config/coverage/<file>.yaml …] [--dns] [--quiet]
	11	+ *
	12	+ * Without arguments every file in config/coverage is checked. Reports schema issues, duplicate domains inside a
	13	+ * file, duplicate sector keys across files and (with --dns) domains that do not resolve. Exit code 1 on any error.
	14	+ */
	15	+const args = process.argv.slice(2);
	16	+const flag = (n: string): boolean => args.includes(`--${n}`);
	17	+const files = args.filter((a) => !a.startsWith("--"));
	18	+
	19	+export function loadCoverageFiles(dir = process.env.WS_COVERAGE_DIR ?? "./config/coverage"): { file: string; sector: CoverageSector \| null; issues: string[] }[] {
	20	+ const list = files.length ? files : existsSync(dir) && statSync(dir).isDirectory() ? readdirSync(dir).filter((f) => /\.ya?ml$/.test(f) && !f.startsWith("_")).sort().map((f) => join(dir, f)) : [];
	21	+ return list.map((file) => {
	22	+ const issues: string[] = [];
	23	+ let sector: CoverageSector \| null = null;
	24	+ try {
	25	+ const raw = YAML.parse(readFileSync(file, "utf8"));
	26	+ const parsed = coverageSectorSchema.safeParse(raw);
	27	+ if (!parsed.success) {
	28	+ for (const i of parsed.error.issues.slice(0, 40)) issues.push(`${i.path.join(".")}: ${i.message}`);
	29	+ if (parsed.error.issues.length > 40) issues.push(`… ${parsed.error.issues.length - 40} more issues`);
	30	+ } else sector = parsed.data;
	31	+ } catch (e) {
	32	+ issues.push(`YAML parse error: ${(e as Error).message}`);
	33	+ }
	34	+ if (sector) {
	35	+ const seen = new Map<string, string>();
	36	+ const keys = new Set<string>();
	37	+ for (const u of sector.universes) {
	38	+ if (keys.has(u.key)) issues.push(`duplicate universe key ${u.key}`);
	39	+ keys.add(u.key);
	40	+ if (!u.members.length) issues.push(`universe ${u.key} has no members`);
	41	+ for (const m of u.members) {
	42	+ if (!/^[a-z0-9.-]+\.[a-z]{2,}$/.test(m.domain) \|\| m.domain.includes("..")) issues.push(`${u.key}: bad domain "${m.domain}" (${m.name})`);
	43	+ const k = coverageKey(m.domain);
	44	+ const prev = seen.get(k);
	45	+ if (prev && prev !== `${u.key}/${m.name}`) issues.push(`${u.key}: duplicate domain ${m.domain} (${m.name}) — already ${prev}`);
	46	+ seen.set(k, `${u.key}/${m.name}`);
	47	+ if (!m.country && !/^(INT\|EU)$/.test(String(m.country))) issues.push(`${u.key}: ${m.name} has no country`);
	48	+ }
	49	+ }
	50	+ }
	51	+ return { file, sector, issues };
	52	+ });
	53	+}
	54	+
	55	+async function resolves(domain: string): Promise<boolean> {
	56	+ for (const h of [domain, `www.${domain}`]) {
	57	+ try {
	58	+ const r = await dns.lookup(h, { all: true });
	59	+ if (r.length) return true;
	60	+ } catch {
	61	+ // try next
	62	+ }
	63	+ }
	64	+ return false;
	65	+}
	66	+
	67	+async function main(): Promise<void> {
	68	+ const loaded = loadCoverageFiles();
	69	+ if (!loaded.length) {
	70	+ console.error("no coverage files found");
	71	+ process.exitCode = 1;
	72	+ return;
	73	+ }
	74	+ let errors = 0;
	75	+ const sectors = new Map<string, string>();
	76	+ for (const l of loaded) {
	77	+ const members = l.sector?.universes.reduce((n, u) => n + u.members.length, 0) ?? 0;
	78	+ const warnIssues = l.issues.filter((i) => / has no country$/.test(i));
	79	+ const hardIssues = l.issues.filter((i) => !/ has no country$/.test(i));
	80	+ errors += hardIssues.length;
	81	+ // Several files may extend the same sector (e.g. corporate indices per region); universes are merged by the loader.
	82	+ if (l.sector && !sectors.has(l.sector.sector)) sectors.set(l.sector.sector, l.file);
	83	+ console.log(`${hardIssues.length ? "FAIL" : "OK "} ${l.file} sector=${l.sector?.sector ?? "-"} universes=${l.sector?.universes.length ?? 0} members=${members}${warnIssues.length ? ` warn=${warnIssues.length}` : ""}`);
	84	+ for (const i of hardIssues) console.log(` ${i}`);
	85	+ if (!flag("quiet")) for (const i of warnIssues.slice(0, 15)) console.log(` WARN ${i}`);
	86	+ if (flag("dns") && l.sector) {
	87	+ const domains = [...new Set(l.sector.universes.flatMap((u) => u.members.map((m) => m.domain)))];
	88	+ let i = 0;
	89	+ const bad: string[] = [];
	90	+ await Promise.all(
	91	+ Array.from({ length: 16 }, async () => {
	92	+ while (i < domains.length) {
	93	+ const d = domains[i++]!;
	94	+ if (!(await resolves(d))) bad.push(d);
	95	+ }
	96	+ }),
	97	+ );
	98	+ for (const d of bad.sort()) console.log(` DNS ${d} does not resolve`);
	99	+ if (bad.length) console.log(` ${bad.length}/${domains.length} domains do not resolve (fix or remove them)`);
	100	+ errors += bad.length;
	101	+ }
	102	+ }
	103	+ const total = loaded.reduce((n, l) => n + (l.sector?.universes.reduce((m, u) => m + u.members.length, 0) ?? 0), 0);
	104	+ console.log(`\n${loaded.length} files · ${sectors.size} sectors · ${total} members → ${errors ? `${errors} errors` : "all good"}`);
	105	+ if (errors) process.exitCode = 1;
	106	+}
	107	+
	108	+if (process.argv[1] && /coverage-validate\.ts$/.test(process.argv[1])) void main();

added apps/engine/src/factory-main.ts +99 −0

		`@@ -0,0 +1,99 @@`
	1	+import { closeDb, db, migrate, sql } from "@websensor/db";
	2	+import { closeDispatcher } from "@websensor/connectors";
	3	+import { config, factoryConfig, log } from "./config";
	4	+import { evaluateShadows, factoryStats, runFactoryBatch, seedFromCoverage, seedFromFiles } from "./factory";
	5	+import { closeRedis, getRedis } from "./redis";
	6	+
	7	+/**
	8	+ * Source Factory process (`websensor-factory`): a slow, polite background worker separate from the poller.
	9	+ * every tick — claim queued seeds (systemic organizations first) and run deep discovery + shadow creation
	10	+ * every 15 min — evaluate shadow sensors (accept / reject / defer)
	11	+ * every 24 h — re-seed from the coverage universes (new members become seeds automatically)
	12	+ * Heartbeat: Redis `ws:factory:status`.
	13	+ */
	14	+async function heartbeat(extra: Record<string, unknown>): Promise<void> {
	15	+ try {
	16	+ await getRedis().set("ws:factory:status", JSON.stringify({ ...extra, at: new Date().toISOString(), pid: process.pid, version: config.version }), "EX", 120);
	17	+ } catch {
	18	+ // best effort
	19	+ }
	20	+}
	21	+
	22	+async function main(): Promise<void> {
	23	+ log.info({ env: config.env, version: config.version, concurrency: factoryConfig.concurrency, budget: factoryConfig.budget }, "websensor factory starting");
	24	+ const applied = await migrate(config.databaseUrl);
	25	+ if (applied.length) log.info({ applied }, "migrations applied");
	26	+ if (!factoryConfig.enabled) {
	27	+ log.warn("WS_FACTORY=0 — factory idle");
	28	+ setInterval(() => void heartbeat({ idle: true }), 60_000);
	29	+ return;
	30	+ }
	31	+ let stopping = false;
	32	+ let busy = false;
	33	+ let processed = 0;
	34	+ let lastSeed = 0;
	35	+ let lastEval = 0;
	36	+
	37	+ const seed = async (): Promise<void> => {
	38	+ try {
	39	+ const a = await seedFromFiles();
	40	+ const b = await seedFromCoverage({ mode: "hinted" });
	41	+ log.info({ files: a, coverage: { members: b.members, seeds: b.seeds, inserted: b.inserted, issues: b.issues.length } }, "factory: seeds refreshed");
	42	+ for (const i of b.issues) log.warn({ issue: i }, "coverage file issue");
	43	+ } catch (e) {
	44	+ log.error({ err: (e as Error).message }, "factory: seeding failed");
	45	+ }
	46	+ };
	47	+
	48	+ const tick = async (): Promise<void> => {
	49	+ if (stopping \|\| busy) return;
	50	+ busy = true;
	51	+ try {
	52	+ if (Date.now() - lastSeed > 24 * 3600e3) {
	53	+ lastSeed = Date.now();
	54	+ await seed();
	55	+ }
	56	+ if (Date.now() - lastEval > 15 * 60e3) {
	57	+ lastEval = Date.now();
	58	+ await evaluateShadows().catch((e) => log.warn({ err: (e as Error).message }, "factory: shadow evaluation failed"));
	59	+ }
	60	+ const out = await runFactoryBatch(factoryConfig.concurrency);
	61	+ processed += out.length;
	62	+ const queued = Number((await db.execute<{ n: string }>(sql`select count(*)::text as n from factory_seeds where status = 'queued'`)).rows[0]?.n ?? 0);
	63	+ await heartbeat({ processed, queued, lastBatch: out.map((o) => ({ seed: o.seedId, status: o.status, candidates: o.candidates, shadow: o.shadow, requests: o.requests, ms: o.durationMs })), concurrency: factoryConfig.concurrency });
	64	+ if (!out.length && processed % 50 === 0) {
	65	+ const st = await factoryStats();
	66	+ log.info({ seeds: st.seeds, candidates: st.candidates, shadow: st.shadow }, "factory: idle");
	67	+ }
	68	+ } catch (e) {
	69	+ log.error({ err: (e as Error).stack ?? (e as Error).message }, "factory tick failed");
	70	+ } finally {
	71	+ busy = false;
	72	+ }
	73	+ };
	74	+
	75	+ await heartbeat({ starting: true });
	76	+ void tick();
	77	+ const timer = setInterval(() => void tick(), factoryConfig.tickSeconds * 1000);
	78	+
	79	+ const shutdown = async (signal: string): Promise<void> => {
	80	+ if (stopping) return;
	81	+ stopping = true;
	82	+ log.info({ signal }, "factory shutting down");
	83	+ clearInterval(timer);
	84	+ const deadline = Date.now() + 60_000;
	85	+ while (busy && Date.now() < deadline) await new Promise((r) => setTimeout(r, 250));
	86	+ await closeDispatcher();
	87	+ await closeRedis();
	88	+ await closeDb();
	89	+ process.exit(0);
	90	+ };
	91	+ process.on("SIGINT", () => void shutdown("SIGINT"));
	92	+ process.on("SIGTERM", () => void shutdown("SIGTERM"));
	93	+ process.on("unhandledRejection", (e) => log.error({ err: e instanceof Error ? e.stack : String(e) }, "unhandled rejection"));
	94	+}
	95	+
	96	+main().catch((e) => {
	97	+ log.fatal({ err: (e as Error).stack ?? String(e) }, "factory failed to start");
	98	+ process.exit(1);
	99	+});

added apps/engine/src/factory/export.ts +46 −0

		`@@ -0,0 +1,46 @@`
	1	+import YAML from "yaml";
	2	+import { db, sql } from "@websensor/db";
	3	+
	4	+/**
	5	+ * Export factory-created sources and accepted sensors as a registry fragment (reviewable YAML). Graduating a
	6	+ * fragment into `config/sources.d/` makes the registry sync own those sensors (`seed: true`) — the Factory never
	7	+ * re-creates them because their URLs already exist.
	8	+ */
	9	+export async function exportFactoryFragment(opts: { sector?: string; sinceDays?: number; includeShadow?: boolean } = {}): Promise<string> {
	10	+ const rows = await db.execute<Record<string, unknown>>(sql`
	11	+ select so.id, so.name, so.domain, so.homepage, so.categories, so.tier, so.importance_weight, so.country, so.language, so.first_party, so.origin, so.sector, so.notes,
	12	+ coalesce(json_agg(json_build_object('name', s.name, 'url', s.url, 'type', s.type, 'connector', s.connector, 'tier', s.tier, 'config', s.config - 'factory' - 'shadow' - 'seed' - 'candidate' - 'seedId' - 'shadowSince' - 'targetPriority' - 'acceptedAt' - 'shadowReport' - 'kind') order by s.name) filter (where s.id is not null), '[]'::json) as sensors
	13	+ from sources so
	14	+ left join sensors s on s.source_id = so.id and s.enabled and (s.config->>'factory') = 'true' and (s.status <> 'SHADOW' or ${opts.includeShadow ?? false})
	15	+ where (so.origin = 'factory' or exists (select 1 from sensors y where y.source_id = so.id and (y.config->>'factory') = 'true'))
	16	+ ${opts.sector ? sql`and so.sector = ${opts.sector}` : sql``}
	17	+ ${opts.sinceDays ? sql`and so.updated_at >= now() - make_interval(days => ${opts.sinceDays})` : sql``}
	18	+ group by so.id order by so.sector, so.id`);
	19	+ const entities = (await db.execute<{ source_id: string; alias: string }>(sql`select se.source_id, ea.alias from source_entities se join entity_aliases ea on ea.entity_id = se.entity_id where se.entity_id like 'org_%'`)).rows;
	20	+ const aliasBySource = new Map<string, string[]>();
	21	+ for (const e of entities) aliasBySource.set(e.source_id, [...(aliasBySource.get(e.source_id) ?? []), e.alias]);
	22	+ const out: Record<string, unknown>[] = [];
	23	+ for (const r of rows.rows) {
	24	+ const sensors = (r.sensors as Record<string, unknown>[]).map((s) => {
	25	+ const cfg = s.config as Record<string, unknown>;
	26	+ const o: Record<string, unknown> = { name: s.name, url: s.url, type: s.type, connector: s.connector, tier: s.tier };
	27	+ if (cfg && Object.keys(cfg).length) o.config = cfg;
	28	+ return o;
	29	+ });
	30	+ if (!sensors.length) continue;
	31	+ const aliases = (aliasBySource.get(String(r.id)) ?? []).filter((a) => a !== String(r.name).toLowerCase() && a !== String(r.domain).toLowerCase() && a !== String(r.domain).replace(/^www\./, "").toLowerCase());
	32	+ if (r.origin === "factory") {
	33	+ const src: Record<string, unknown> = { id: r.id, name: r.name, domain: r.domain, categories: r.categories, tier: r.tier };
	34	+ if (Number(r.importance_weight) !== 1) src.weight = Number(r.importance_weight);
	35	+ if (aliases.length) src.aliases = aliases;
	36	+ if (r.country) src.country = r.country;
	37	+ if (r.language) src.language = r.language;
	38	+ if (r.first_party === false) src.first_party = false;
	39	+ if (r.notes) src.notes = r.notes;
	40	+ src.sensors = sensors;
	41	+ out.push(src);
	42	+ } else out.push({ id: r.id, extend: true, sensors });
	43	+ }
	44	+ const header = `# WebSensor — Source Factory export (${new Date().toISOString().slice(0, 10)}${opts.sector ? `, sector ${opts.sector}` : ""}): ${out.length} organizations, ${out.reduce((n, s) => n + (s.sensors as unknown[]).length, 0)} accepted sensors.\n# Generated by \`cli.ts factory export\`; review, then place in config/sources.d/ to let the registry own these sensors.\n`;
	45	+ return header + YAML.stringify({ sources: out }, { lineWidth: 0, defaultStringType: "PLAIN", defaultKeyType: "PLAIN" });
	46	+}

added apps/engine/src/factory/factory.test.ts +67 −0

		`@@ -0,0 +1,67 @@`
	1	+import { describe, expect, it } from "vitest";
	2	+import { coverageKey, registrableDomain } from "@websensor/core";
	3	+import type { DeepCandidate } from "@websensor/connectors";
	4	+import { itemsPerDayOf } from "@websensor/connectors";
	5	+import { KIND_WEIGHT, scoreCandidate, selectForShadow } from "./score";
	6	+
	7	+const cand = (over: Partial<DeepCandidate>): DeepCandidate => ({ url: "https://example.org/feed", type: "RSS", connector: "rss", kind: "news", name: "news feed", config: {}, suggestedTier: "B", evidence: "well-known path", value: 0.8, itemsPerDay: 0.5, firstPartyConfidence: 1, fetchCost: { bytes: 20_000, ms: 400 }, ...over });
	8	+
	9	+describe("registrable domain (coverage key)", () => {
	10	+ it("collapses hosts to the organization's registrable domain", () => {
	11	+ expect(registrableDomain("ir.apple.com")).toBe("apple.com");
	12	+ expect(registrableDomain("www.ons.gov.uk")).toBe("ons.gov.uk");
	13	+ expect(registrableDomain("www.bankofengland.co.uk")).toBe("bankofengland.co.uk");
	14	+ expect(registrableDomain("statcan.gc.ca")).toBe("statcan.gc.ca");
	15	+ expect(registrableDomain("open.canada.ca")).toBe("canada.ca");
	16	+ expect(registrableDomain("boj.or.jp")).toBe("boj.or.jp");
	17	+ expect(coverageKey("https://WWW.Example.COM/path")).toBe("example.com");
	18	+ });
	19	+});
	20	+
	21	+describe("factory scoring", () => {
	22	+ it("weights page classes and organization importance", () => {
	23	+ const status = scoreCandidate(cand({ kind: "status", connector: "statuspage", value: 0.95, itemsPerDay: null }), { importance: 2 });
	24	+ const careers = scoreCandidate(cand({ kind: "careers", connector: "http", value: 0.35, itemsPerDay: null }), { importance: 2 });
	25	+ expect(status.score).toBeGreaterThan(careers.score);
	26	+ expect(KIND_WEIGHT.status).toBeGreaterThan(KIND_WEIGHT.careers);
	27	+ const big = scoreCandidate(cand({}), { importance: 3 });
	28	+ const small = scoreCandidate(cand({}), { importance: 1 });
	29	+ expect(big.score).toBeGreaterThan(small.score);
	30	+ expect(big.reasons.some((r) => /importance 3/.test(r))).toBe(true);
	31	+ });
	32	+ it("penalises firehoses, dormant feeds and slow endpoints, explainably", () => {
	33	+ const base = scoreCandidate(cand({}), { importance: 2 }).score;
	34	+ expect(scoreCandidate(cand({ itemsPerDay: 200 }), { importance: 2 }).score).toBeLessThan(base);
	35	+ expect(scoreCandidate(cand({ itemsPerDay: 0.001 }), { importance: 2 }).score).toBeLessThan(base);
	36	+ const slow = scoreCandidate(cand({ fetchCost: { bytes: 100, ms: 12_000 } }), { importance: 2 });
	37	+ expect(slow.score).toBeLessThan(base);
	38	+ expect(slow.reasons.some((r) => /slow/.test(r))).toBe(true);
	39	+ });
	40	+ it("applies per-organization caps and the minimum score", () => {
	41	+ const feeds = Array.from({ length: 6 }, (_, i) => scoreCandidate(cand({ url: `https://example.org/f${i}`, value: 0.9 - i * 0.02 }), { importance: 2 }));
	42	+ const low = scoreCandidate(cand({ url: "https://example.org/careers", kind: "careers", connector: "http", value: 0.2, itemsPerDay: null }), { importance: 1 });
	43	+ const legal = ["terms", "privacy", "cookies"].map((p) => scoreCandidate(cand({ url: `https://example.org/${p}`, kind: "legal", connector: "http", type: "HTML", value: 0.7, itemsPerDay: null }), { importance: 2 }));
	44	+ const { selected, skipped } = selectForShadow([...feeds, low, ...legal], { kinds: new Map(), total: 0 });
	45	+ expect(selected.filter((s) => s.cand.connector === "rss")).toHaveLength(4);
	46	+ expect(selected.filter((s) => s.cand.kind === "legal")).toHaveLength(2);
	47	+ expect(skipped.some((s) => s.s === low && /score/.test(s.reason))).toBe(true);
	48	+ expect(skipped.some((s) => /feeds cap/.test(s.reason))).toBe(true);
	49	+ });
	50	+ it("respects sensors the organization already has", () => {
	51	+ const feed = scoreCandidate(cand({}), { importance: 2 });
	52	+ const { selected, skipped } = selectForShadow([feed], { kinds: new Map([["news", 4]]), total: 4 });
	53	+ expect(selected).toHaveLength(0);
	54	+ expect(skipped[0]?.reason).toMatch(/feeds cap/);
	55	+ });
	56	+});
	57	+
	58	+describe("expected change frequency", () => {
	59	+ it("uses the median gap and survives one bogus date", () => {
	60	+ const day = 86400e3;
	61	+ const now = Date.UTC(2026, 8, 13);
	62	+ const dates = [now, now - day, now - 2 * day, now - 3 * day, now - 4 * day, now - 5000 * day];
	63	+ expect(itemsPerDayOf(dates)).toBe(1);
	64	+ expect(itemsPerDayOf([now, now - day])).toBeNull();
	65	+ expect(itemsPerDayOf([now, now - 30 * day, now - 60 * day, now - 90 * day])).toBeCloseTo(0.033, 2);
	66	+ });
	67	+});

added apps/engine/src/factory/index.ts +5 −0

		`@@ -0,0 +1,5 @@`
	1	+export * from "./seeds";
	2	+export * from "./score";
	3	+export * from "./run";
	4	+export * from "./shadow";
	5	+export * from "./export";

added apps/engine/src/factory/run.ts +288 −0

		`@@ -0,0 +1,288 @@`
	1	+import { coverageKey, newId, slugify, sourceSchema, type SourceSeed, type Tier } from "@websensor/core";
	2	+import { discoverOrganization, type CandidateKind, type DeepDiscoveryHints } from "@websensor/connectors";
	3	+import { db, factorySeeds, sensors, sql, type FactorySeed } from "@websensor/db";
	4	+import { factoryConfig, log } from "../config";
	5	+import { priorityFor, upsertSourceRecord } from "../registry";
	6	+import { scoreCandidate, selectForShadow } from "./score";
	7	+
	8	+/**
	9	+ * Source Factory — one organization at a time:
	10	+ * seed → deep discovery → (create or attach the source) → score candidates → promotion policy →
	11	+ * shadow sensors (`status = SHADOW`: observed, evidence stored, nothing published) → later `evaluateShadows()`.
	12	+ */
	13	+export interface ProcessOutcome {
	14	+ seedId: string;
	15	+ sourceId: string \| null;
	16	+ status: "discovered" \| "blocked" \| "error";
	17	+ candidates: number;
	18	+ shadow: number;
	19	+ requests: number;
	20	+ durationMs: number;
	21	+ notes: string[];
	22	+}
	23	+
	24	+interface SourceRow extends Record<string, unknown> {
	25	+ id: string;
	26	+ domain: string;
	27	+ categories: string[];
	28	+ tier: string;
	29	+ enabled: boolean;
	30	+}
	31	+
	32	+let sourceIndex: { at: number; byKey: Map<string, SourceRow>; ids: Set<string> } \| null = null;
	33	+async function sourcesByDomain(force = false): Promise<{ byKey: Map<string, SourceRow>; ids: Set<string> }> {
	34	+ if (!force && sourceIndex && Date.now() - sourceIndex.at < 5 * 60_000) return sourceIndex;
	35	+ const rows = await db.execute<SourceRow>(sql`select id, domain, categories, tier, enabled from sources where kind = 'registry'`);
	36	+ const byKey = new Map<string, SourceRow>();
	37	+ const ids = new Set<string>();
	38	+ for (const r of rows.rows) {
	39	+ ids.add(r.id);
	40	+ const k = coverageKey(r.domain);
	41	+ if (!byKey.has(k)) byKey.set(k, r);
	42	+ }
	43	+ sourceIndex = { at: Date.now(), byKey, ids };
	44	+ return sourceIndex;
	45	+}
	46	+
	47	+export async function processSeed(seed: FactorySeed): Promise<ProcessOutcome> {
	48	+ const started = Date.now();
	49	+ const notes: string[] = [];
	50	+ const hints = (seed.hints ?? {}) as DeepDiscoveryHints & { posture?: boolean };
	51	+ const idx = await sourcesByDomain();
	52	+ const key = coverageKey(seed.domain);
	53	+ let source = idx.byKey.get(key) ?? null;
	54	+
	55	+ const disc = await discoverOrganization(seed.domain, {
	56	+ hints,
	57	+ budget: factoryConfig.budget,
	58	+ concurrency: 4,
	59	+ sensorIdForLogs: `factory_${seed.id}`,
	60	+ probePages: true,
	61	+ probeSubdomains: true,
	62	+ probePosture: seed.importance >= 3 \|\| hints.posture === true,
	63	+ probeOpenApi: true,
	64	+ });
	65	+ notes.push(...disc.notes);
	66	+ // The homepage may redirect to another registrable domain that we already monitor.
	67	+ if (!source && disc.canonicalDomain !== key) source = idx.byKey.get(disc.canonicalDomain) ?? null;
	68	+
	69	+ if (!disc.candidates.length) {
	70	+ const status = disc.blocked ? "blocked" : "discovered";
	71	+ await db.execute(sql`update factory_seeds set status = ${status}, candidates = 0, shadow = 0, source_id = ${source?.id ?? null}, discovered_at = now(), last_error = ${disc.blocked ? `homepage ${disc.homeStatus} (anti-bot)` : null}, updated_at = now() where id = ${seed.id}`);
	72	+ await bumpFactoryDaily({ seeds_processed: 1, requests: disc.requests, blocked: disc.blocked ? 1 : 0 });
	73	+ return { seedId: seed.id, sourceId: source?.id ?? null, status, candidates: 0, shadow: 0, requests: disc.requests, durationMs: Date.now() - started, notes };
	74	+ }
	75	+
	76	+ // ---- source: attach or create --------------------------------------------------------------------------
	77	+ let sourceId: string;
	78	+ if (source) {
	79	+ sourceId = source.id;
	80	+ if (!source.enabled) notes.push("source disabled in registry — candidates recorded, no shadow sensors");
	81	+ } else {
	82	+ sourceId = idx.ids.has(seed.id) ? `${seed.id}-${slugify(seed.domain.split(".")[0] ?? "x")}`.slice(0, 70) : seed.id;
	83	+ const seedDoc: SourceSeed = sourceSchema.parse({
	84	+ id: sourceId,
	85	+ name: seed.name,
	86	+ domain: seed.domain,
	87	+ homepage: seed.homepage ?? `https://${seed.domain}`,
	88	+ categories: seed.categories,
	89	+ tier: (seed.tier as Tier) ?? "B",
	90	+ weight: seed.weight,
	91	+ aliases: seed.aliases,
	92	+ country: seed.country ?? undefined,
	93	+ language: seed.language ?? undefined,
	94	+ first_party: seed.firstParty,
	95	+ notes: `Source Factory · ${seed.sector ?? "-"}/${seed.universe ?? "-"} · discovered ${new Date().toISOString().slice(0, 10)}`,
	96	+ llm: true,
	97	+ });
	98	+ await upsertSourceRecord(seedDoc, { origin: "factory", sector: seed.sector });
	99	+ idx.ids.add(sourceId);
	100	+ idx.byKey.set(key, { id: sourceId, domain: seed.domain, categories: seed.categories, tier: seed.tier, enabled: true });
	101	+ source = idx.byKey.get(key)!;
	102	+ }
	103	+
	104	+ // ---- existing sensors (this source + global URL uniqueness) ----------------------------------------------
	105	+ const existing = await db.execute<{ id: string; url: string; type: string; connector: string; config: Record<string, unknown>; status: string }>(sql`select id, url, type, connector, config, status from sensors where source_id = ${sourceId}`);
	106	+ const existingUrls = new Set(existing.rows.map((r) => r.url.replace(/\/$/, "")));
	107	+ const existingKinds = new Map<CandidateKind, number>();
	108	+ let existingTotal = 0;
	109	+ for (const r of existing.rows) {
	110	+ if (r.status === "DISABLED") continue;
	111	+ existingTotal++;
	112	+ const k = kindOfExisting(r);
	113	+ existingKinds.set(k, (existingKinds.get(k) ?? 0) + 1);
	114	+ }
	115	+ const urls = disc.candidates.map((c) => c.url);
	116	+ const global = urls.length ? await db.execute<{ url: string; source_id: string }>(sql`select url, source_id from sensors where url = any(${sql.raw("array[" + urls.map((u) => "'" + u.replace(/'/g, "''") + "'").join(",") + "]::text[]")}) and source_id <> ${sourceId}`) : { rows: [] as { url: string; source_id: string }[] };
	117	+ const ownedElsewhere = new Map(global.rows.map((r) => [r.url.replace(/\/$/, ""), r.source_id]));
	118	+
	119	+ // ---- score + select ---------------------------------------------------------------------------------------
	120	+ const scored = disc.candidates.map((c) => scoreCandidate(c, { importance: seed.importance, weight: seed.weight }));
	121	+ const fresh = scored.filter((s) => !existingUrls.has(s.cand.url.replace(/\/$/, "")) && !ownedElsewhere.has(s.cand.url.replace(/\/$/, "")));
	122	+ const { selected, skipped } = source.enabled ? selectForShadow(fresh, { kinds: existingKinds, total: existingTotal }) : { selected: [], skipped: fresh.map((s) => ({ s, reason: "source disabled" })) };
	123	+
	124	+ // ---- persist candidates ------------------------------------------------------------------------------------
	125	+ const day = new Date().toISOString().slice(0, 10);
	126	+ let shadowCreated = 0;
	127	+ for (const s of scored) {
	128	+ const k = s.cand.url.replace(/\/$/, "");
	129	+ let status = "candidate";
	130	+ let reason: string \| null = null;
	131	+ if (existingUrls.has(k)) {
	132	+ status = "duplicate";
	133	+ reason = "already a sensor of this source";
	134	+ } else if (ownedElsewhere.has(k)) {
	135	+ status = "duplicate";
	136	+ reason = `already a sensor of ${ownedElsewhere.get(k)}`;
	137	+ } else if (selected.includes(s)) status = "shadow";
	138	+ else {
	139	+ const sk = skipped.find((x) => x.s === s);
	140	+ status = sk && /score/.test(sk.reason) ? "rejected" : "candidate";
	141	+ reason = sk?.reason ?? null;
	142	+ }
	143	+ const candId = newId("cand");
	144	+ let shadowSensorId: string \| null = null;
	145	+ if (status === "shadow") {
	146	+ shadowSensorId = await createShadowSensor(sourceId, source, s.cand, candId, seed);
	147	+ if (shadowSensorId) shadowCreated++;
	148	+ else {
	149	+ status = "duplicate";
	150	+ reason = "sensor id/url collision";
	151	+ }
	152	+ }
	153	+ await db.execute(sql`
	154	+ insert into discovery_candidates (id, source_id, url, kind, evidence, score, status, seed_id, connector, type, name, kind_class, config, tier, score_value, reason, shadow_sensor_id, found_at, updated_at)
	155	+ values (${candId}, ${sourceId}, ${s.cand.url}, ${s.cand.type}, ${s.cand.evidence}, ${JSON.stringify({ value: s.cand.value, score: s.score, reasons: s.reasons, itemCount: s.cand.itemCount ?? null, itemsPerDay: s.cand.itemsPerDay ?? null, firstParty: s.cand.firstPartyConfidence, cost: s.cand.fetchCost, title: s.cand.title ?? null, fingerprint: s.cand.fingerprint ?? null })}::jsonb, ${status}, ${seed.id}, ${s.cand.connector}, ${s.cand.type}, ${s.cand.name}, ${s.cand.kind}, ${JSON.stringify(s.cand.config)}::jsonb, ${s.cand.suggestedTier}, ${s.score}, ${reason}, ${shadowSensorId}, now(), now())
	156	+ on conflict (source_id, url) do update set evidence = excluded.evidence, score = excluded.score, score_value = excluded.score_value, seed_id = excluded.seed_id, connector = excluded.connector, type = excluded.type, name = excluded.name, kind_class = excluded.kind_class, config = excluded.config, tier = excluded.tier, updated_at = now(),
	157	+ status = case when discovery_candidates.status in ('accepted','rejected','shadow') then discovery_candidates.status else excluded.status end,
	158	+ reason = case when discovery_candidates.status in ('accepted','rejected','shadow') then discovery_candidates.reason else excluded.reason end,
	159	+ shadow_sensor_id = coalesce(discovery_candidates.shadow_sensor_id, excluded.shadow_sensor_id)`);
	160	+ }
	161	+ for (const r of disc.rejected.slice(0, 40)) {
	162	+ await db.execute(sql`insert into discovery_candidates (id, source_id, url, kind, evidence, status, seed_id, reason, kind_class) values (${newId("cand")}, ${sourceId}, ${r.url}, 'HTML', ${r.reason}, 'rejected', ${seed.id}, ${r.reason}, 'other') on conflict (source_id, url) do nothing`);
	163	+ }
	164	+
	165	+ await db.execute(sql`update factory_seeds set status = 'discovered', source_id = ${sourceId}, candidates = ${scored.length}, shadow = shadow + ${shadowCreated}, discovered_at = now(), last_error = null, updated_at = now() where id = ${seed.id}`);
	166	+ await db.execute(sql`update sources set robots_checked_at = now(), updated_at = now() where id = ${sourceId}`);
	167	+ await bumpFactoryDaily({ seeds_processed: 1, requests: disc.requests, candidates: scored.length, shadow_created: shadowCreated, blocked: disc.blocked ? 1 : 0 });
	168	+ log.info({ seed: seed.id, source: sourceId, candidates: scored.length, shadow: shadowCreated, requests: disc.requests, ms: Date.now() - started, day }, "factory: organization discovered");
	169	+ return { seedId: seed.id, sourceId, status: "discovered", candidates: scored.length, shadow: shadowCreated, requests: disc.requests, durationMs: Date.now() - started, notes };
	170	+}
	171	+
	172	+function kindOfExisting(r: { url: string; type: string; connector: string; config: Record<string, unknown> }): CandidateKind {
	173	+ const k = (r.config as { kind?: string }).kind as CandidateKind \| undefined;
	174	+ if (k) return k;
	175	+ if (r.connector === "statuspage" \|\| r.connector === "statusjson") return "status";
	176	+ if (r.connector === "sitemap") return "sitemap";
	177	+ if (r.connector === "github") return "releases";
	178	+ if (r.connector === "edgar") return "filings";
	179	+ if (r.connector === "dns" \|\| r.connector === "tls" \|\| r.connector === "headers" \|\| r.connector === "rdap") return "posture";
	180	+ if (r.connector === "openapi") return "api";
	181	+ if (r.connector === "rss") return /press/i.test(r.url) ? "press" : /blog/i.test(r.url) ? "blog" : /security\|advisor/i.test(r.url) ? "security" : /changelog\|release/i.test(r.url) ? "changelog" : "news";
	182	+ if (/pricing\|plans/i.test(r.url)) return "pricing";
	183	+ if (/terms\|privacy\|legal/i.test(r.url)) return "legal";
	184	+ if (/career\|jobs/i.test(r.url)) return "careers";
	185	+ if (/changelog\|release-notes\|whats-new/i.test(r.url)) return "changelog";
	186	+ if (/security\|advisor/i.test(r.url)) return "security";
	187	+ return "other";
	188	+}
	189	+
	190	+async function createShadowSensor(sourceId: string, source: SourceRow, c: { url: string; type: string; connector: string; name: string; config: Record<string, unknown>; suggestedTier: Tier; kind: CandidateKind }, candId: string, seed: FactorySeed): Promise<string \| null> {
	191	+ let id = `${sourceId}_${slugify(c.name)}`.slice(0, 80);
	192	+ const clash = (await db.execute<{ url: string }>(sql`select url from sensors where id = ${id}`)).rows[0];
	193	+ if (clash && clash.url.replace(/\/$/, "") !== c.url.replace(/\/$/, "")) id = `${id}-${newId("x").slice(-5)}`;
	194	+ const tier = c.suggestedTier;
	195	+ const priority = 3; // shadows never compete with production sensors
	196	+ const jitter = Math.floor(Math.random() * 30 * 60e3);
	197	+ try {
	198	+ await db
	199	+ .insert(sensors)
	200	+ .values({ id, sourceId, name: c.name, url: c.url, type: c.type, connector: c.connector, tier, importanceWeight: 1, config: { ...c.config, factory: true, shadow: true, seed: false, kind: c.kind, candidate: candId, seedId: seed.id, shadowSince: new Date().toISOString(), targetPriority: priorityFor(tier, source.categories) }, status: "SHADOW", priority, validatedAt: new Date(), nextCheckAt: new Date(Date.now() + jitter) })
	201	+ .onConflictDoNothing();
	202	+ } catch (e) {
	203	+ log.warn({ sensor: id, err: (e as Error).message }, "factory: shadow sensor insert failed");
	204	+ return null;
	205	+ }
	206	+ return id;
	207	+}
	208	+
	209	+export async function bumpFactoryDaily(delta: Partial<Record<"seeds_processed" \| "requests" \| "candidates" \| "shadow_created" \| "accepted" \| "rejected" \| "blocked", number>>): Promise<void> {
	210	+ const day = new Date().toISOString().slice(0, 10);
	211	+ const d = { seeds_processed: 0, requests: 0, candidates: 0, shadow_created: 0, accepted: 0, rejected: 0, blocked: 0, ...delta };
	212	+ await db.execute(sql`insert into factory_daily (day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked) values (${day}, ${d.seeds_processed}, ${d.requests}, ${d.candidates}, ${d.shadow_created}, ${d.accepted}, ${d.rejected}, ${d.blocked})
	213	+ on conflict (day) do update set seeds_processed = factory_daily.seeds_processed + ${d.seeds_processed}, requests = factory_daily.requests + ${d.requests}, candidates = factory_daily.candidates + ${d.candidates}, shadow_created = factory_daily.shadow_created + ${d.shadow_created}, accepted = factory_daily.accepted + ${d.accepted}, rejected = factory_daily.rejected + ${d.rejected}, blocked = factory_daily.blocked + ${d.blocked}`).catch(() => undefined);
	214	+}
	215	+
	216	+/** Claim up to `n` queued seeds (systemic organizations first) and process them with bounded concurrency. */
	217	+export async function runFactoryBatch(n = factoryConfig.concurrency, opts: { seedIds?: string[] } = {}): Promise<ProcessOutcome[]> {
	218	+ const claimed = await db.execute<Record<string, unknown>>(sql`
	219	+ update factory_seeds set status = 'discovering', attempts = attempts + 1, updated_at = now()
	220	+ where id in (
	221	+ select id from factory_seeds
	222	+ where ${opts.seedIds ? sql`id = any(${sql.raw("array[" + opts.seedIds.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]")})` : sql`(status = 'queued' or (status = 'error' and attempts < 3 and updated_at < now() - interval '6 hours') or (status = 'discovering' and updated_at < now() - interval '2 hours'))`}
	223	+ order by importance desc, created_at asc limit ${n} for update skip locked)
	224	+ returning *`);
	225	+ const seeds = claimed.rows.map(normalizeSeed);
	226	+ const out: ProcessOutcome[] = [];
	227	+ let i = 0;
	228	+ await Promise.all(
	229	+ Array.from({ length: Math.min(factoryConfig.concurrency, seeds.length) }, async () => {
	230	+ while (i < seeds.length) {
	231	+ const seed = seeds[i++]!;
	232	+ try {
	233	+ out.push(await processSeed(seed));
	234	+ } catch (e) {
	235	+ log.error({ seed: seed.id, err: (e as Error).stack ?? (e as Error).message }, "factory: seed failed");
	236	+ await db.execute(sql`update factory_seeds set status = 'error', last_error = ${(e as Error).message.slice(0, 500)}, updated_at = now() where id = ${seed.id}`).catch(() => undefined);
	237	+ out.push({ seedId: seed.id, sourceId: null, status: "error", candidates: 0, shadow: 0, requests: 0, durationMs: 0, notes: [(e as Error).message] });
	238	+ }
	239	+ }
	240	+ }),
	241	+ );
	242	+ return out;
	243	+}
	244	+
	245	+function normalizeSeed(r: Record<string, unknown>): FactorySeed {
	246	+ const d = (v: unknown): Date \| null => (v ? new Date(v as string) : null);
	247	+ return {
	248	+ id: r.id as string,
	249	+ name: r.name as string,
	250	+ domain: r.domain as string,
	251	+ homepage: (r.homepage as string \| null) ?? null,
	252	+ categories: (r.categories as string[]) ?? [],
	253	+ country: (r.country as string \| null) ?? null,
	254	+ language: (r.language as string \| null) ?? null,
	255	+ tier: (r.tier as string) ?? "B",
	256	+ weight: Number(r.weight ?? 1),
	257	+ importance: Number(r.importance ?? 2),
	258	+ aliases: (r.aliases as string[]) ?? [],
	259	+ firstParty: Boolean(r.first_party ?? true),
	260	+ sector: (r.sector as string \| null) ?? null,
	261	+ universe: (r.universe as string \| null) ?? null,
	262	+ hints: (r.hints as Record<string, unknown>) ?? {},
	263	+ status: r.status as string,
	264	+ attempts: Number(r.attempts ?? 0),
	265	+ sourceId: (r.source_id as string \| null) ?? null,
	266	+ candidates: Number(r.candidates ?? 0),
	267	+ shadow: Number(r.shadow ?? 0),
	268	+ accepted: Number(r.accepted ?? 0),
	269	+ rejected: Number(r.rejected ?? 0),
	270	+ lastError: (r.last_error as string \| null) ?? null,
	271	+ discoveredAt: d(r.discovered_at),
	272	+ createdAt: d(r.created_at) ?? new Date(),
	273	+ updatedAt: d(r.updated_at) ?? new Date(),
	274	+ };
	275	+}
	276	+
	277	+export async function factoryStats(): Promise<Record<string, unknown>> {
	278	+ const [seeds, cands, shadows, daily, sectors] = await Promise.all([
	279	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from factory_seeds group by status`).then((r) => r.rows),
	280	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from discovery_candidates group by status`).then((r) => r.rows),
	281	+ db.execute<Record<string, unknown>>(sql`select count()::int as shadow, count() filter (where total_runs >= ${factoryConfig.shadowMinChecks})::int as ready, count(*) filter (where health <> 'UP')::int as unhealthy from sensors where status = 'SHADOW'`).then((r) => r.rows[0]),
	282	+ db.execute<Record<string, unknown>>(sql`select day::text as day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked from factory_daily order by day desc limit 30`).then((r) => r.rows),
	283	+ db.execute<Record<string, unknown>>(sql`select coalesce(sector, '-') as sector, count()::int as seeds, count() filter (where status = 'queued')::int as queued, count() filter (where status in ('discovered','done'))::int as discovered, count() filter (where status = 'blocked')::int as blocked, sum(shadow)::int as shadow, sum(accepted)::int as accepted, sum(rejected)::int as rejected from factory_seeds group by 1 order by seeds desc`).then((r) => r.rows),
	284	+ ]);
	285	+ return { seeds: Object.fromEntries(seeds.map((r) => [r.status, r.n])), candidates: Object.fromEntries(cands.map((r) => [r.status, r.n])), shadow: shadows, daily, sectors, config: { concurrency: factoryConfig.concurrency, budget: factoryConfig.budget, minScore: factoryConfig.minScore, shadowMinChecks: factoryConfig.shadowMinChecks, shadowMinHours: factoryConfig.shadowMinHours, shadowMaxHours: factoryConfig.shadowMaxHours, caps: factoryConfig.caps } };
	286	+}
	287	+
	288	+export { factorySeeds };

added apps/engine/src/factory/score.ts +114 −0

		`@@ -0,0 +1,114 @@`
	1	+import type { CandidateKind, DeepCandidate } from "@websensor/connectors";
	2	+import { factoryConfig } from "../config";
	3	+
	4	+/**
	5	+ * Candidate scoring (Source Factory, stage "signal/noise evaluation before shadow"). Combines the discovery
	6	+ * value (evidence quality) with the organization's importance, the class of page, first-party confidence,
	7	+ * expected change frequency and fetch cost. Explainable: every score carries its reasons.
	8	+ */
	9	+export interface ScoredCandidate {
	10	+ cand: DeepCandidate;
	11	+ score: number;
	12	+ reasons: string[];
	13	+}
	14	+
	15	+/** How much a page class is worth for change intelligence (1 = neutral). */
	16	+export const KIND_WEIGHT: Record<CandidateKind, number> = {
	17	+ status: 1.15,
	18	+ security: 1.15,
	19	+ press: 1.1,
	20	+ news: 1.05,
	21	+ filings: 1.1,
	22	+ changelog: 1.05,
	23	+ releases: 1.0,
	24	+ pricing: 1.05,
	25	+ models: 1.05,
	26	+ ir: 0.95,
	27	+ legal: 0.9,
	28	+ blog: 0.9,
	29	+ api: 0.9,
	30	+ leadership: 0.85,
	31	+ data: 0.85,
	32	+ sitemap: 0.8,
	33	+ docs: 0.7,
	34	+ careers: 0.6,
	35	+ posture: 0.6,
	36	+ other: 0.7,
	37	+};
	38	+
	39	+export function scoreCandidate(c: DeepCandidate, seed: { importance: number; weight?: number }): ScoredCandidate {
	40	+ const reasons: string[] = [];
	41	+ let score = c.value;
	42	+ reasons.push(`evidence ${c.value.toFixed(2)} (${c.evidence})`);
	43	+ const kw = KIND_WEIGHT[c.kind] ?? 0.7;
	44	+ score *= kw;
	45	+ if (kw !== 1) reasons.push(`${c.kind} ×${kw.toFixed(2)}`);
	46	+ const imp = seed.importance === 3 ? 1.1 : seed.importance === 1 ? 0.85 : 1;
	47	+ score = imp (seed.weight ?? 1);
	48	+ if (imp !== 1) reasons.push(`organization importance ${seed.importance} ×${imp.toFixed(2)}`);
	49	+ score *= c.firstPartyConfidence;
	50	+ if (c.firstPartyConfidence < 1) reasons.push(`first-party confidence ${c.firstPartyConfidence}`);
	51	+ if (c.itemsPerDay !== null && c.itemsPerDay !== undefined) {
	52	+ if (c.itemsPerDay > 60) {
	53	+ score *= 0.85;
	54	+ reasons.push(`firehose ${c.itemsPerDay}/day ×0.85`);
	55	+ } else if (c.itemsPerDay < 0.02) {
	56	+ score *= 0.8;
	57	+ reasons.push(`nearly dormant ${c.itemsPerDay}/day ×0.80`);
	58	+ } else if (c.itemsPerDay >= 0.2) {
	59	+ score *= 1.05;
	60	+ reasons.push(`active ${c.itemsPerDay}/day ×1.05`);
	61	+ }
	62	+ }
	63	+ if (c.fetchCost.ms > 8000) {
	64	+ score -= 0.08;
	65	+ reasons.push(`slow (${Math.round(c.fetchCost.ms)} ms) −0.08`);
	66	+ }
	67	+ if (c.fetchCost.bytes > 5 * 1024 * 1024) {
	68	+ score -= 0.08;
	69	+ reasons.push(`heavy (${Math.round(c.fetchCost.bytes / 1048576)} MB) −0.08`);
	70	+ }
	71	+ score = Math.round(Math.max(0, Math.min(1.2, score)) * 1000) / 1000;
	72	+ return { cand: c, score, reasons };
	73	+}
	74	+
	75	+/** Promotion policy: caps per organization and per class, best score first. Returns the selected candidates and the reason for the others. */
	76	+export function selectForShadow(scored: ScoredCandidate[], existing: { kinds: Map<CandidateKind, number>; total: number }): { selected: ScoredCandidate[]; skipped: { s: ScoredCandidate; reason: string }[] } {
	77	+ const caps = factoryConfig.caps;
	78	+ const groupOf = (k: CandidateKind): keyof typeof caps => (k === "news" \|\| k === "press" \|\| k === "blog" \|\| k === "ir" \|\| k === "changelog" \|\| k === "security" ? "feeds" : k === "pricing" \|\| k === "legal" \|\| k === "careers" \|\| k === "leadership" \|\| k === "docs" \|\| k === "data" \|\| k === "other" ? "pages" : (k as keyof typeof caps));
	79	+ const used = new Map<string, number>();
	80	+ for (const [k, n] of existing.kinds) used.set(groupOf(k), (used.get(groupOf(k)) ?? 0) + n);
	81	+ let total = existing.total;
	82	+ const selected: ScoredCandidate[] = [];
	83	+ const skipped: { s: ScoredCandidate; reason: string }[] = [];
	84	+ const legal = (existing.kinds.get("legal") ?? 0);
	85	+ let legalUsed = legal;
	86	+ for (const s of [...scored].sort((a, b) => b.score - a.score)) {
	87	+ if (s.score < factoryConfig.minScore) {
	88	+ skipped.push({ s, reason: `score ${s.score} < ${factoryConfig.minScore}` });
	89	+ continue;
	90	+ }
	91	+ if (total >= caps.total) {
	92	+ skipped.push({ s, reason: `organization cap ${caps.total} reached` });
	93	+ continue;
	94	+ }
	95	+ // Feed-type candidates (rss connector) and HTML pages are capped by group; connectors with their own cap by kind.
	96	+ const g = s.cand.connector === "rss" ? "feeds" : s.cand.connector === "http" && s.cand.kind !== "posture" ? "pages" : groupOf(s.cand.kind);
	97	+ const cap = caps[g as keyof typeof caps] ?? 1;
	98	+ if ((used.get(g) ?? 0) >= cap) {
	99	+ skipped.push({ s, reason: `${g} cap ${cap} reached` });
	100	+ continue;
	101	+ }
	102	+ if (s.cand.kind === "legal") {
	103	+ if (legalUsed >= caps.legal) {
	104	+ skipped.push({ s, reason: `legal cap ${caps.legal} reached` });
	105	+ continue;
	106	+ }
	107	+ legalUsed++;
	108	+ }
	109	+ used.set(g, (used.get(g) ?? 0) + 1);
	110	+ total++;
	111	+ selected.push(s);
	112	+ }
	113	+ return { selected, skipped };
	114	+}

added apps/engine/src/factory/seeds.ts +168 −0

		`@@ -0,0 +1,168 @@`
	1	+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
	2	+import { join } from "node:path";
	3	+import YAML from "yaml";
	4	+import { coverageKey, coverageSectorSchema, slugify, type CoverageMember, type CoverageSector } from "@websensor/core";
	5	+import { db, factorySeeds, sql } from "@websensor/db";
	6	+import { factoryConfig, log } from "../config";
	7	+
	8	+/**
	9	+ * Factory seeds come from two places: the coverage universes (`config/coverage/*.yaml`, every member that is not
	10	+ * monitored yet — or that carries hints worth expanding) and ad-hoc seed files (`config/factory/seeds/*.yaml`,
	11	+ * `{ seeds: [{ name, domain, categories, country, … }] }`) or the admin API.
	12	+ */
	13	+export interface SeedInput {
	14	+ id?: string;
	15	+ name: string;
	16	+ domain: string;
	17	+ homepage?: string;
	18	+ categories?: string[];
	19	+ country?: string;
	20	+ language?: string;
	21	+ tier?: string;
	22	+ weight?: number;
	23	+ importance?: number;
	24	+ aliases?: string[];
	25	+ first_party?: boolean;
	26	+ sector?: string;
	27	+ universe?: string;
	28	+ hints?: Record<string, unknown>;
	29	+}
	30	+
	31	+export function loadCoverageSectors(dir = factoryConfig.coverageDir): { sectors: CoverageSector[]; issues: string[] } {
	32	+ const sectors: CoverageSector[] = [];
	33	+ const issues: string[] = [];
	34	+ if (!existsSync(dir) \|\| !statSync(dir).isDirectory()) return { sectors, issues: [`coverage dir ${dir} not found`] };
	35	+ for (const f of readdirSync(dir).filter((x) => /\.ya?ml$/.test(x) && !x.startsWith("_")).sort()) {
	36	+ try {
	37	+ const parsed = coverageSectorSchema.safeParse(YAML.parse(readFileSync(join(dir, f), "utf8")));
	38	+ if (!parsed.success) {
	39	+ issues.push(`${f}: ${parsed.error.issues.slice(0, 3).map((i) => `${i.path.join(".")}: ${i.message}`).join("; ")}`);
	40	+ continue;
	41	+ }
	42	+ sectors.push(parsed.data);
	43	+ } catch (e) {
	44	+ issues.push(`${f}: ${(e as Error).message}`);
	45	+ }
	46	+ }
	47	+ return { sectors, issues };
	48	+}
	49	+
	50	+/** Merge files that share a sector key. */
	51	+export function mergeSectors(sectors: CoverageSector[]): CoverageSector[] {
	52	+ const byKey = new Map<string, CoverageSector>();
	53	+ for (const s of sectors) {
	54	+ const prev = byKey.get(s.sector);
	55	+ if (!prev) byKey.set(s.sector, { ...s, universes: [...s.universes] });
	56	+ else prev.universes.push(...s.universes);
	57	+ }
	58	+ return [...byKey.values()];
	59	+}
	60	+
	61	+export function memberToSeed(sector: CoverageSector, universe: string, m: CoverageMember): SeedInput {
	62	+ return {
	63	+ name: m.name,
	64	+ domain: m.domain,
	65	+ categories: m.categories ?? sector.categories,
	66	+ country: m.country,
	67	+ language: m.language,
	68	+ tier: m.tier ?? sector.tier,
	69	+ importance: m.importance,
	70	+ aliases: m.aliases,
	71	+ first_party: m.first_party,
	72	+ sector: sector.sector,
	73	+ universe,
	74	+ hints: m.hints as Record<string, unknown>,
	75	+ };
	76	+}
	77	+
	78	+function hasHints(h: Record<string, unknown> \| undefined): boolean {
	79	+ if (!h) return false;
	80	+ return Boolean(h.cik \|\| h.github_org \|\| (Array.isArray(h.github_repos) && h.github_repos.length) \|\| h.hf_author \|\| h.status_url \|\| (Array.isArray(h.urls) && h.urls.length) \|\| (Array.isArray(h.hosts) && h.hosts.length));
	81	+}
	82	+
	83	+/** Registrable domains currently monitored (source domains + hosts of enabled non-shadow sensors). */
	84	+export async function monitoredDomainKeys(): Promise<Set<string>> {
	85	+ const rows = await db.execute<{ d: string }>(sql`
	86	+ select domain as d from sources where enabled and kind = 'registry'
	87	+ union
	88	+ select lower(split_part(split_part(url, '/', 3), ':', 1)) as d from sensors where enabled and status <> 'SHADOW' and url like 'http%'`);
	89	+ const out = new Set<string>();
	90	+ for (const r of rows.rows) if (r.d) out.add(coverageKey(r.d));
	91	+ return out;
	92	+}
	93	+
	94	+export async function upsertSeeds(inputs: SeedInput[], opts: { requeue?: boolean } = {}): Promise<{ inserted: number; updated: number; skipped: number }> {
	95	+ let inserted = 0;
	96	+ let updated = 0;
	97	+ let skipped = 0;
	98	+ const existingIds = new Set((await db.execute<{ id: string }>(sql`select id from sources`)).rows.map((r) => r.id));
	99	+ const existingSeedDomains = new Map((await db.execute<{ id: string; domain: string }>(sql`select id, domain from factory_seeds`)).rows.map((r) => [coverageKey(r.domain), r.id]));
	100	+ for (const s of inputs) {
	101	+ const domain = s.domain.toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "");
	102	+ const key = coverageKey(domain);
	103	+ let id = s.id ?? existingSeedDomains.get(key) ?? slugify(s.name).slice(0, 60);
	104	+ if (!id) id = slugify(domain);
	105	+ // A registry source with the same id but another organization → disambiguate by domain.
	106	+ if (!s.id && !existingSeedDomains.has(key) && existingIds.has(id)) {
	107	+ const src = (await db.execute<{ domain: string }>(sql`select domain from sources where id = ${id}`)).rows[0];
	108	+ if (src && coverageKey(src.domain) !== key) id = `${id}-${slugify(domain.split(".")[0] ?? domain)}`.slice(0, 70);
	109	+ }
	110	+ const r = await db.execute<{ inserted: boolean }>(sql`
	111	+ insert into factory_seeds (id, name, domain, homepage, categories, country, language, tier, weight, importance, aliases, first_party, sector, universe, hints, status)
	112	+ values (${id}, ${s.name}, ${domain}, ${s.homepage ?? null}, ${sql.raw("'{" + (s.categories ?? []).map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.country ?? null}, ${s.language ?? null}, ${s.tier ?? "B"}, ${s.weight ?? 1}, ${s.importance ?? 2}, ${sql.raw("'{" + (s.aliases ?? []).map((c) => '"' + c.replace(/["\\]/g, "") + '"').join(",") + "}'::text[]")}, ${s.first_party ?? true}, ${s.sector ?? null}, ${s.universe ?? null}, ${JSON.stringify(s.hints ?? {})}::jsonb, 'queued')
	113	+ on conflict (id) do update set name = excluded.name, domain = excluded.domain, categories = excluded.categories, country = coalesce(excluded.country, factory_seeds.country), language = coalesce(excluded.language, factory_seeds.language), tier = excluded.tier, importance = excluded.importance, aliases = excluded.aliases, sector = coalesce(excluded.sector, factory_seeds.sector), universe = coalesce(excluded.universe, factory_seeds.universe), hints = factory_seeds.hints \|\| excluded.hints, updated_at = now()${opts.requeue ? sql`, status = 'queued'` : sql``}
	114	+ returning (xmax = 0) as inserted`);
	115	+ if (r.rows[0]?.inserted) inserted++;
	116	+ else if (r.rowCount) updated++;
	117	+ else skipped++;
	118	+ existingSeedDomains.set(key, id);
	119	+ }
	120	+ log.info({ inserted, updated, skipped }, "factory seeds upserted");
	121	+ return { inserted, updated, skipped };
	122	+}
	123	+
	124	+/**
	125	+ * Turn coverage universes into seeds. `mode`: `uncovered` = members without any monitored domain; `hinted` = also
	126	+ * covered members that carry expansion hints; `all` = every member (re-discover everything).
	127	+ */
	128	+export async function seedFromCoverage(opts: { mode?: "uncovered" \| "hinted" \| "all"; sectors?: string[]; requeue?: boolean } = {}): Promise<{ members: number; seeds: number; inserted: number; updated: number; issues: string[] }> {
	129	+ const mode = opts.mode ?? "hinted";
	130	+ const { sectors, issues } = loadCoverageSectors();
	131	+ const monitored = await monitoredDomainKeys();
	132	+ const inputs: SeedInput[] = [];
	133	+ const seen = new Set<string>();
	134	+ let members = 0;
	135	+ for (const sector of mergeSectors(sectors)) {
	136	+ if (opts.sectors && !opts.sectors.includes(sector.sector)) continue;
	137	+ for (const u of sector.universes) {
	138	+ for (const m of u.members) {
	139	+ members++;
	140	+ const key = coverageKey(m.domain);
	141	+ if (seen.has(key)) continue;
	142	+ const covered = monitored.has(key);
	143	+ if (mode === "uncovered" && covered) continue;
	144	+ if (mode === "hinted" && covered && !hasHints(m.hints as Record<string, unknown>)) continue;
	145	+ seen.add(key);
	146	+ inputs.push(memberToSeed(sector, u.key, m));
	147	+ }
	148	+ }
	149	+ }
	150	+ const r = await upsertSeeds(inputs, { requeue: opts.requeue });
	151	+ return { members, seeds: inputs.length, inserted: r.inserted, updated: r.updated, issues };
	152	+}
	153	+
	154	+/** `config/factory/seeds/.yaml` → `{ seeds: [...] }` /
	155	+export async function seedFromFiles(dir = factoryConfig.seedsDir): Promise<{ files: number; seeds: number }> {
	156	+ if (!existsSync(dir)) return { files: 0, seeds: 0 };
	157	+ const inputs: SeedInput[] = [];
	158	+ let files = 0;
	159	+ for (const f of readdirSync(dir).filter((x) => /\.ya?ml$/.test(x)).sort()) {
	160	+ files++;
	161	+ const raw = YAML.parse(readFileSync(join(dir, f), "utf8")) as { seeds?: SeedInput[] };
	162	+ for (const s of raw.seeds ?? []) if (s?.name && s?.domain) inputs.push(s);
	163	+ }
	164	+ await upsertSeeds(inputs);
	165	+ return { files, seeds: inputs.length };
	166	+}
	167	+
	168	+export { factorySeeds };

added apps/engine/src/factory/shadow.ts +109 −0

Large diff (109 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/pipeline.ts +9 −2

Large diff (29 lines) is not loaded by default to keep the page fast.Load diff

modified apps/engine/src/registry.ts +39 −29

Large diff (77 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/app/ops/factory-panel.tsx +205 −0

Large diff (205 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/ops/ops.tsx +6 −1

Large diff (29 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/app/page.tsx +2 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/command-palette.tsx +1 −0

Large diff (7 lines) is not loaded by default to keep the page fast.Load diff

added apps/web/src/components/coverage-rail.tsx +27 −0

Large diff (27 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/nav.tsx +2 −0

Large diff (14 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/components/ui.tsx +2 −2

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

modified apps/web/src/lib/api.ts +63 −0

Large diff (75 lines) is not loaded by default to keep the page fast.Load diff

modified deploy/README.md +17 −1

Large diff (35 lines) is not loaded by default to keep the page fast.Load diff

modified deploy/websensor.mld.json.example +194 −123

Large diff (323 lines) is not loaded by default to keep the page fast.Load diff

modified docs/ARCHITECTURE.md +10 −1

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

modified docs/registry/AUTHORING.md +5 −0

Large diff (11 lines) is not loaded by default to keep the page fast.Load diff

added docs/registry/COVERAGE.md +67 −0

Large diff (67 lines) is not loaded by default to keep the page fast.Load diff

added packages/connectors/src/discovery-deep.ts +672 −0

Large diff (672 lines) is not loaded by default to keep the page fast.Load diff

modified packages/connectors/src/index.ts +1 −0

Large diff (4 lines) is not loaded by default to keep the page fast.Load diff

added packages/core/src/coverage-schema.ts +150 −0

Large diff (150 lines) is not loaded by default to keep the page fast.Load diff

modified packages/core/src/index.ts +1 −0

Large diff (4 lines) is not loaded by default to keep the page fast.Load diff

added packages/db/migrations/0007_factory.sql +73 −0

Large diff (73 lines) is not loaded by default to keep the page fast.Load diff

modified packages/db/src/schema.ts +46 −0

Large diff (60 lines) is not loaded by default to keep the page fast.Load diff

spb/websensor

WebSensor 0.3 — Source Factory & Global Observation Coverage Score

Theme