WebSensor 0.3 — Source Factory & Global Observation Coverage Score · 03c8dc1 · websensor

WebSensor 0.3 — Source Factory & Global Observation Coverage Score

- Deep discovery (robots → sitemaps → feeds incl. official sub-domains → navigation-classified pages → status providers → GitHub org → EDGAR → Hugging Face → OpenAPI → posture), every candidate validated, wildcard DNS / anti-bot detection, content de-duplication, expected change frequency (median gap)
- Factory: coverage universes → factory_seeds → scored candidates (explainable) → promotion caps → SHADOW sensors (polled, never published) → evidence-based acceptance/rejection; separate websensor-factory process with Redis heartbeat; CLI expand/factory seed|run|evaluate|stats|export|requeue; YAML export for graduation
- Migration 0007 (sources.origin/sector, factory_seeds, candidate lifecycle columns, factory_daily); pipeline keeps SHADOW status and skips events for shadow sensors
- Coverage: sector universes schema + validator (--dns), registrableDomain(), GET /api/v1/coverage[/:sector], pages /coverage and /coverage/[sector], homepage rail, stats exclude shadow sensors
- Admin API /api/v1/admin/factory* + /ops Source Factory panel (funnel, sectors, candidates, accept/reject)
- Docs: COVERAGE.md, CLAUDE.md, deploy README, ARCHITECTURE, README; manifest gains websensor-factory + factory seed hook; version 0.3.0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 11 days ago (Sep 13, 2026) parent 938aab1

41 changed files +3,084 −173

modified .env.example +13 −0

		`@@ -42,3 +42,16 @@ WS_RETENTION_CHANGED_RAW_DAYS=60`
42	42	WS_ADMIN_TOKEN=
43	43	# Custom URL monitors per anonymous owner (spec §45)
44	44	WS_MONITORS_PER_OWNER=5
	45	+
	46	+# Source Factory (apps/engine/src/factory-main.ts — separate `websensor-factory` process). Seeds come from
	47	+# config/coverage/.yaml (members not yet monitored, or carrying hints) and config/factory/seeds/.yaml.
	48	+WS_FACTORY=1
	49	+WS_FACTORY_CONCURRENCY=6 # organizations discovered in parallel
	50	+WS_FACTORY_BUDGET=90 # HTTP requests per organization
	51	+WS_FACTORY_MIN_SCORE=0.35 # candidate score needed to enter shadow monitoring
	52	+WS_FACTORY_SHADOW_MIN_CHECKS=5 # checks before a shadow sensor can be accepted
	53	+WS_FACTORY_SHADOW_MIN_HOURS=24
	54	+WS_FACTORY_SHADOW_MAX_HOURS=120
	55	+WS_FACTORY_TICK_SECONDS=20
	56	+WS_COVERAGE_DIR=./config/coverage
	57	+# GITHUB_TOKEN= # optional: lets the factory rank an organization's repositories via the GitHub API

modified CLAUDE.md +36 −6

		`@@ -5,7 +5,7 @@ Web. It monitors official public sources, detects meaningful changes, classifies`
5	5	entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score,
6	6	preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through
7	7	a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file
8		−is the working guide for the code (v0.2, 2026-09-11).
	8	+is the working guide for the code (v0.3, 2026-09-13: Source Factory + Global Observation Coverage Score).
9	9
10	10	## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)
11	11	- `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups,
		`@@ -31,6 +31,30 @@ is the working guide for the code (v0.2, 2026-09-11).`
31	31	(`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) +
32	32	`discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and
33	33	`{field.path}`.
	34	+- Source Factory (`apps/engine/src/factory/`, process `apps/engine/src/factory-main.ts` = PM2 `websensor-factory`) —
	35	+ the system grows by itself: coverage universes (`config/coverage/*.yaml`, one sector per file, members = organizations
	36	+ with a verified domain + `hints`) and `config/factory/seeds/*.yaml` become `factory_seeds`; `runFactoryBatch()` claims
	37	+ seeds (systemic first) and runs deep discovery (`packages/connectors/src/discovery-deep.ts`: robots → sitemaps →
	38	+ feeds incl. official sub-domains → navigation-classified pages (news/press/IR/changelog/security/pricing/legal/careers/
	39	+ leadership/docs) → status-page providers → GitHub org repos → EDGAR (cik) → Hugging Face (hf_author) → OpenAPI →
	40	+ posture; every candidate is fetched and parsed, wildcard DNS detected, feeds/pages de-duplicated by content) →
	41	+ `score.ts` (evidence × page-class weight × organization importance × first-party confidence × change frequency −
	42	+ fetch cost, explainable reasons; caps per organization/class) → shadow sensors (`sensors.status = 'SHADOW'`,
	43	+ `config.factory/shadow/kind/seedId`, priority 3: polled, snapshots + changes stored, never published; the pipeline
	44	+ and `handleMissing` skip event creation) → `shadow.ts` `evaluateShadows()` every 15 min (reject: ≥ 50 % errors / all
	45	+ changes noise / duplicate content of an active sensor; accept after ≥ 5 checks and 24 h → `ACTIVE` with its real
	46	+ priority; defer otherwise, max 120 h). Sources created by the factory carry `origin = 'factory'` + `sector`; attached
	47	+ ones reuse the registry source (match by registrable domain, `coverageKey()`). Candidates and decisions live in
	48	+ `discovery_candidates` (status candidate / shadow / accepted / rejected / duplicate, `reason`, `score`); funnel in
	49	+ `factory_daily`; heartbeat Redis `ws:factory:status`. `cli.ts expand <domain> [--hints=json]` = dry run;
	50	+ `cli.ts factory seed\|run\|evaluate\|stats\|export\|requeue`; `factory export` writes a reviewable YAML fragment
	51	+ (graduating accepted sensors into `config/sources.d/` hands them to the registry sync).
	52	+- Coverage (`apps/api/src/coverage.ts`, `GET /api/v1/coverage[/:sector]`, pages `/coverage`, `/coverage/[sector]`,
	53	+ homepage rail) — Global Observation Coverage Score: per sector `breadth` (importance-weighted share of universe
	54	+ members with ≥ 1 active non-shadow sensor on their registrable domain), `depth` (min(1, sensors/5)),
	55	+ `score = 100·(0.7·breadth + 0.3·depth)`, global = sector-weight-weighted mean. Universes are the denominator (with
	56	+ `provenance`), never the registry. Validator: `apps/engine/src/coverage-validate.ts [file] [--dns]`. Guide:
	57	+ `docs/registry/COVERAGE.md`. Schema shared in `packages/core/src/coverage-schema.ts` (+ `registrableDomain()`).
34	58	- `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency,
35	59	domain circuit breaker, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff →
36	60	heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias
		`@@ -38,8 +62,8 @@ is the working guide for the code (v0.2, 2026-09-11).`
38	62	with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook
39	63	delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery,
40	64	connector health rollups, retention (`retention.ts`), Prometheus metrics :8262.
41		− CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`,
42		− `refresh-clusters`, `llm-test`); `src/validate.ts` (registry validator).
	65	+ CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `expand`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`,
	66	+ `refresh-clusters`, `llm-test`, `factory …`); `src/validate.ts` (registry validator); `src/coverage-validate.ts`.
43	67	- `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped:
44	68	watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts`
45	69	behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models
		`@@ -52,8 +76,8 @@ is the working guide for the code (v0.2, 2026-09-11).`
52	76	(intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters,
53	77	pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar`
54	78	`/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]`
55		− `/country/[slug]` `/event/[slug]` (+ OG image) `/bookmarks` `/watchlists` `/alerts` `/monitors` `/ops` `/health`
56		− `/api`.
	79	+ `/country/[slug]` `/event/[slug]` (+ OG image) `/coverage` `/coverage/[sector]` `/bookmarks` `/watchlists` `/alerts`
	80	+ `/monitors` `/ops` (+ Source Factory panel: funnel, sectors, candidates, accept/reject) `/health` `/api`.
57	81	- `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order
58	82	(`extend: true` adds to an earlier source). Fragments 10–35 = site classes; 40–46 (2026-09-11) = depth: AI
59	83	frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport /
		`@@ -87,6 +111,11 @@ is the working guide for the code (v0.2, 2026-09-11).`
87	111	`llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works.
88	112	- Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes
89	113	`page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks.
	114	+- Shadow is not production: `status = 'SHADOW'` sensors never create events, never count in public stats
	115	+ (`stats.sensors`, coverage) and never compete with production sensors (priority 3). Nothing the Factory finds is
	116	+ published before `evaluateShadows()` (or an operator) accepts it. Public source pages show shadow sensors with a SHADOW pill.
	117	+- Coverage universes are lists of real organizations with verified domains (`--dns`), provenance and retrieval dates —
	118	+ never guessed domains; several files may share a sector key (universes are merged).
90	119	- Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public
91	120	feeds (`sources.kind = 'custom'`).
92	121	- All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…).
		`@@ -108,6 +137,7 @@ Run CLI/engine from the repo root. A copy of production data for UI work:`
108	137	`mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json`
109	138	(secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260,
110	139	public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine`
111		−(metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that
	140	+(metrics :8262), `websensor-factory` (Source Factory, heartbeat `ws:factory:status`; post-sync hook `factory seed --mode=hinted`
	141	+re-seeds from the coverage universes at every deploy). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that
112	142	changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See
113	143	`deploy/README.md`.

modified README.md +31 −0

		`@@ -94,6 +94,7 @@ threat-intelligence desk*: dense information, one-pixel borders, small badges, p`
94	94	(OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │
95	95	MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │
96	96	│ websensor-engine scheduler + pipeline, Prometheus :8262 │
	97	+ │ websensor-factory Source Factory: seeds → discovery → shadow │
97	98	│ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │
98	99	└───────────────────────────────────────────────────────────────────────┘
99	100	```
		`@@ -105,6 +106,8 @@ threat-intelligence desk*: dense information, one-pixel borders, small badges, p`
105	106	\| Evidence \| `packages/store` \| content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres \|
106	107	\| Acquisition \| `packages/connectors` \| connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback \|
107	108	\| Ingestion \| `apps/engine` \| scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator \|
	109	+\| Expansion \| `apps/engine/src/factory` \| Source Factory (0.3): coverage universes → deep discovery → scored candidates → shadow sensors → evidence-based acceptance; `websensor-factory` process \|
	110	+\| Coverage \| `apps/api/src/coverage.ts` \| Global Observation Coverage Score per sector (`/coverage`, `GET /api/v1/coverage`): how much of the world's high-value public web is observed \|
108	111	\| Gateway \| `apps/api` \| REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API \|
109	112	\| Product \| `apps/web` \| Next.js 16 App Router, design system, live feed, drawer, palette, desks \|
110	113
		@@ -493,6 +496,34 @@ without ever deleting event evidence. Details: `deploy/README.md`, `docs/ARCHITE
493	496	- Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting
494	497	on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP.
495	498
	499	+## Source Factory and coverage (0.3, 2026-09-13)
	500	+
	501	+WebSensor no longer grows by hand-written lists alone. `config/coverage/*.yaml` declares, sector by sector, the
	502	+organizations that should be observed (listed companies by index, governments of every country by institutional
	503	+role, AI, cloud, cybersecurity, markets, science, healthcare, energy, transport, telecom, logistics, open source,
	504	+commerce), each with a verified domain, a country, an importance and optional hints (CIK, GitHub organization,
	505	+Hugging Face author, status page, explicit URLs). Those universes are the denominator of the **Global Observation
	506	+Coverage Score** (`breadth` = importance-weighted share of members with an active sensor, `depth` = sensors per
	507	+organization, `score = 100·(0.7·breadth + 0.3·depth)`) and the input of the Source Factory:
	508	+
	509	+```
	510	+seed (organization + hints)
	511	+ → deep discovery robots · sitemaps · feeds (+ official sub-domains) · navigation-classified pages
	512	+ (news · press · IR · changelog · security · pricing · legal · careers · leadership · docs)
	513	+ · status-page providers · GitHub org repositories · EDGAR · Hugging Face · OpenAPI · posture
	514	+ → validation every candidate fetched and parsed by its connector; wildcard DNS and anti-bot detected;
	515	+ duplicates removed by content (shared feed items, identical page text)
	516	+ → scoring evidence × page class × organization importance × first-party confidence × change
	517	+ frequency − fetch cost, with explainable reasons; caps per organization and per class
	518	+ → shadow sensors polled like any sensor, evidence stored, nothing published
	519	+ → evaluation reject (errors, pure noise, duplicate of an active sensor) · accept (≥ 5 checks, 24 h) · defer
	520	+ → production accepted sensors publish; `cli.ts factory export` graduates them into a registry fragment
	521	+```
	522	+
	523	+`cli.ts expand <domain>` shows what the Factory would find for one organization; `/ops` has the funnel, per-sector
	524	+progress and a candidate review table; `/coverage` and `/coverage/<sector>` show the score and every member's state
	525	+(observed · shadow · queued · not seeded).
	526	+
496	527	## Roadmap
497	528
498	529	World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel

modified apps/api/src/config.ts +1 −1

		`@@ -14,5 +14,5 @@ export const config = {`
14	14	adminToken: env.WS_ADMIN_TOKEN ?? "",
15	15	/** Custom URL monitors per anonymous owner (spec §45). */
16	16	monitorsPerOwner: Number(env.WS_MONITORS_PER_OWNER ?? 5),
17		− version: "0.2.0",
	17	+ version: "0.3.0",
18	18	};

added apps/api/src/coverage.ts +236 −0

		`@@ -0,0 +1,236 @@`
	1	+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
	2	+import { join } from "node:path";
	3	+import YAML from "yaml";
	4	+import { coverageKey, coverageSectorSchema, type CoverageMember, type CoverageSector } from "@websensor/core";
	5	+import { db, sql } from "@websensor/db";
	6	+import { cached } from "./cache";
	7	+
	8	+/**
	9	+ * Global Observation Coverage Score — "how much of the world's high-value public web do we observe?"
	10	+ *
	11	+ * For every sector the universes in `config/coverage/.yaml` are the denominator. A member is covered* when at
	12	+ * least one active (non-shadow) sensor lives on its registrable domain (or on a host declared in `hints.hosts`).
	13	+ * breadth = Σ w·covered / Σ w (w = member importance 1–3)
	14	+ * depth = Σ w·min(1, sensors/5) / Σ w (five sensors per organization = full depth)
	15	+ * score = 100 · (0.7·breadth + 0.3·depth)
	16	+ * The global score is the sector-weight-weighted mean. Everything is explainable: the per-member table shows
	17	+ * which sensors count, what is in shadow and what the Factory is still queuing.
	18	+ */
	19	+export interface MemberCoverage {
	20	+ name: string;
	21	+ domain: string;
	22	+ country: string \| null;
	23	+ importance: number;
	24	+ universe: string;
	25	+ covered: boolean;
	26	+ sensors: number;
	27	+ shadow: number;
	28	+ source_id: string \| null;
	29	+ seed_status: string \| null;
	30	+ depth: number;
	31	+}
	32	+export interface UniverseCoverage {
	33	+ key: string;
	34	+ label: string;
	35	+ provenance: string \| null;
	36	+ members: number;
	37	+ covered: number;
	38	+ sensors: number;
	39	+ breadth: number;
	40	+ depth: number;
	41	+ score: number;
	42	+}
	43	+export interface SectorCoverage {
	44	+ sector: string;
	45	+ label: string;
	46	+ description: string \| null;
	47	+ weight: number;
	48	+ members: number;
	49	+ covered: number;
	50	+ shadow_members: number;
	51	+ queued: number;
	52	+ sensors: number;
	53	+ breadth: number;
	54	+ depth: number;
	55	+ score: number;
	56	+ universes: UniverseCoverage[];
	57	+}
	58	+
	59	+const COVERAGE_DIR = process.env.WS_COVERAGE_DIR ?? "./config/coverage";
	60	+
	61	+let fileCache: { at: number; sectors: CoverageSector[]; issues: string[] } \| null = null;
	62	+export function loadSectors(): { sectors: CoverageSector[]; issues: string[] } {
	63	+ if (fileCache && Date.now() - fileCache.at < 60_000) return fileCache;
	64	+ const byKey = new Map<string, CoverageSector>();
	65	+ const issues: string[] = [];
	66	+ if (existsSync(COVERAGE_DIR) && statSync(COVERAGE_DIR).isDirectory()) {
	67	+ for (const f of readdirSync(COVERAGE_DIR).filter((x) => /\.ya?ml$/.test(x) && !x.startsWith("_")).sort()) {
	68	+ try {
	69	+ const parsed = coverageSectorSchema.safeParse(YAML.parse(readFileSync(join(COVERAGE_DIR, f), "utf8")));
	70	+ if (!parsed.success) {
	71	+ issues.push(`${f}: ${parsed.error.issues[0]?.path.join(".")}: ${parsed.error.issues[0]?.message}`);
	72	+ continue;
	73	+ }
	74	+ const s = parsed.data;
	75	+ const prev = byKey.get(s.sector);
	76	+ if (prev) prev.universes.push(...s.universes);
	77	+ else byKey.set(s.sector, { ...s, universes: [...s.universes] });
	78	+ } catch (e) {
	79	+ issues.push(`${f}: ${(e as Error).message}`);
	80	+ }
	81	+ }
	82	+ }
	83	+ fileCache = { at: Date.now(), sectors: [...byKey.values()], issues };
	84	+ return fileCache;
	85	+}
	86	+
	87	+interface Observed {
	88	+ /** registrable domain → { sensors (active), shadow, sourceId } */
	89	+ byKey: Map<string, { sensors: number; shadow: number; sourceId: string \| null }>;
	90	+ seeds: Map<string, string>;
	91	+}
	92	+
	93	+async function observed(): Promise<Observed> {
	94	+ const [hosts, srcs, seeds] = await Promise.all([
	95	+ db.execute<{ host: string; status: string; n: string; source_id: string }>(sql`
	96	+ select lower(split_part(split_part(url, '/', 3), ':', 1)) as host, case when status = 'SHADOW' then 'SHADOW' else 'ACTIVE' end as status, count(*)::text as n, min(source_id) as source_id
	97	+ from sensors where enabled and url like 'http%' group by 1, 2`).then((r) => r.rows),
	98	+ db.execute<{ id: string; domain: string }>(sql`select id, domain from sources where enabled and kind = 'registry'`).then((r) => r.rows),
	99	+ db.execute<{ domain: string; status: string }>(sql`select domain, status from factory_seeds`).then((r) => r.rows),
	100	+ ]);
	101	+ const byKey = new Map<string, { sensors: number; shadow: number; sourceId: string \| null }>();
	102	+ for (const h of hosts) {
	103	+ if (!h.host) continue;
	104	+ const k = coverageKey(h.host);
	105	+ const cur = byKey.get(k) ?? { sensors: 0, shadow: 0, sourceId: null };
	106	+ if (h.status === "SHADOW") cur.shadow += Number(h.n);
	107	+ else cur.sensors += Number(h.n);
	108	+ cur.sourceId = cur.sourceId ?? h.source_id;
	109	+ byKey.set(k, cur);
	110	+ }
	111	+ for (const s of srcs) {
	112	+ const k = coverageKey(s.domain);
	113	+ const cur = byKey.get(k) ?? { sensors: 0, shadow: 0, sourceId: null };
	114	+ cur.sourceId = cur.sourceId ?? s.id;
	115	+ byKey.set(k, cur);
	116	+ }
	117	+ const seedMap = new Map<string, string>();
	118	+ for (const s of seeds) seedMap.set(coverageKey(s.domain), s.status);
	119	+ return { byKey, seeds: seedMap };
	120	+}
	121	+
	122	+function memberCoverage(m: CoverageMember, universe: string, obs: Observed): MemberCoverage {
	123	+ const keys = [coverageKey(m.domain), ...((m.hints?.hosts ?? []).map((h) => coverageKey(h)))];
	124	+ let sensors = 0;
	125	+ let shadow = 0;
	126	+ let sourceId: string \| null = null;
	127	+ for (const k of new Set(keys)) {
	128	+ const o = obs.byKey.get(k);
	129	+ if (!o) continue;
	130	+ sensors += o.sensors;
	131	+ shadow += o.shadow;
	132	+ sourceId = sourceId ?? o.sourceId;
	133	+ }
	134	+ return { name: m.name, domain: m.domain, country: m.country ?? null, importance: m.importance, universe, covered: sensors > 0, sensors, shadow, source_id: sourceId, seed_status: obs.seeds.get(coverageKey(m.domain)) ?? null, depth: Math.min(1, sensors / 5) };
	135	+}
	136	+
	137	+function aggregate(members: MemberCoverage[]): { breadth: number; depth: number; score: number; covered: number; sensors: number } {
	138	+ let w = 0;
	139	+ let cov = 0;
	140	+ let dep = 0;
	141	+ let covered = 0;
	142	+ let sensors = 0;
	143	+ for (const m of members) {
	144	+ w += m.importance;
	145	+ if (m.covered) {
	146	+ cov += m.importance;
	147	+ covered++;
	148	+ }
	149	+ dep += m.importance * m.depth;
	150	+ sensors += m.sensors;
	151	+ }
	152	+ const breadth = w ? cov / w : 0;
	153	+ const depth = w ? dep / w : 0;
	154	+ return { breadth: round(breadth), depth: round(depth), score: round(100 * (0.7 * breadth + 0.3 * depth), 1), covered, sensors };
	155	+}
	156	+const round = (x: number, d = 3): number => Math.round(x * 10 d) / 10 d;
	157	+
	158	+export async function coverageSummary(): Promise<Record<string, unknown>> {
	159	+ return cached("coverage:summary", 5 * 60_000, async () => {
	160	+ const { sectors, issues } = loadSectors();
	161	+ const obs = await observed();
	162	+ const out: SectorCoverage[] = [];
	163	+ let wsum = 0;
	164	+ let ssum = 0;
	165	+ let members = 0;
	166	+ let covered = 0;
	167	+ let sensorsTotal = 0;
	168	+ for (const s of sectors) {
	169	+ const all: MemberCoverage[] = [];
	170	+ const universes: UniverseCoverage[] = [];
	171	+ const seenDomains = new Set<string>();
	172	+ for (const u of s.universes) {
	173	+ const ms = u.members.map((m) => memberCoverage(m, u.key, obs));
	174	+ const a = aggregate(ms);
	175	+ universes.push({ key: u.key, label: u.label, provenance: u.provenance ?? null, members: ms.length, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score });
	176	+ for (const m of ms) {
	177	+ const k = coverageKey(m.domain);
	178	+ if (seenDomains.has(k)) continue;
	179	+ seenDomains.add(k);
	180	+ all.push(m);
	181	+ }
	182	+ }
	183	+ const a = aggregate(all);
	184	+ const sec: SectorCoverage = { sector: s.sector, label: s.label, description: s.description ?? null, weight: s.weight, members: all.length, covered: a.covered, shadow_members: all.filter((m) => !m.covered && m.shadow > 0).length, queued: all.filter((m) => !m.covered && (m.seed_status === "queued" \|\| m.seed_status === "discovering")).length, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, universes };
	185	+ out.push(sec);
	186	+ wsum += s.weight;
	187	+ ssum += s.weight * a.score;
	188	+ members += all.length;
	189	+ covered += a.covered;
	190	+ sensorsTotal += a.sensors;
	191	+ }
	192	+ out.sort((a, b) => b.score - a.score);
	193	+ const [monitored, factory] = await Promise.all([
	194	+ db.execute<{ sources: string; sensors: string; shadow: string; countries: string }>(sql`select (select count() from sources where enabled and kind = 'registry')::text as sources, (select count() from sensors where enabled and status <> 'SHADOW')::text as sensors, (select count(*) from sensors where enabled and status = 'SHADOW')::text as shadow, (select count(distinct country) from sources where country is not null and enabled)::text as countries`).then((r) => r.rows[0]),
	195	+ db.execute<{ status: string; n: string }>(sql`select status, count(*)::text as n from factory_seeds group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [x.status, Number(x.n)]))),
	196	+ ]);
	197	+ return {
	198	+ global_score: wsum ? round(ssum / wsum, 1) : 0,
	199	+ members,
	200	+ covered,
	201	+ sensors_matched: sensorsTotal,
	202	+ sectors: out.map(({ universes: _u, ...rest }) => rest),
	203	+ monitored: { sources: Number(monitored?.sources ?? 0), sensors: Number(monitored?.sensors ?? 0), shadow: Number(monitored?.shadow ?? 0), countries: Number(monitored?.countries ?? 0) },
	204	+ factory: factory,
	205	+ method: "breadth = importance-weighted share of members with ≥ 1 active sensor · depth = importance-weighted min(1, sensors/5) · score = 100·(0.7·breadth + 0.3·depth) · global = sector-weight-weighted mean",
	206	+ issues,
	207	+ generated_at: new Date().toISOString(),
	208	+ };
	209	+ });
	210	+}
	211	+
	212	+export async function coverageSector(sector: string): Promise<Record<string, unknown> \| null> {
	213	+ return cached(`coverage:sector:${sector}`, 5 * 60_000, async () => {
	214	+ const { sectors } = loadSectors();
	215	+ const s = sectors.find((x) => x.sector === sector);
	216	+ if (!s) return null;
	217	+ const obs = await observed();
	218	+ const universes = s.universes.map((u) => {
	219	+ const members = u.members.map((m) => memberCoverage(m, u.key, obs)).sort((a, b) => Number(b.covered) - Number(a.covered) \|\| b.importance - a.importance \|\| b.sensors - a.sensors \|\| a.name.localeCompare(b.name));
	220	+ const a = aggregate(members);
	221	+ return { key: u.key, label: u.label, description: u.description ?? null, provenance: u.provenance ?? null, members: members.length, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, items: members };
	222	+ });
	223	+ const flat = new Map<string, MemberCoverage>();
	224	+ for (const u of universes) for (const m of u.items) if (!flat.has(coverageKey(m.domain))) flat.set(coverageKey(m.domain), m);
	225	+ const a = aggregate([...flat.values()]);
	226	+ const byCountry = new Map<string, { members: number; covered: number }>();
	227	+ for (const m of flat.values()) {
	228	+ const c = m.country ?? "—";
	229	+ const cur = byCountry.get(c) ?? { members: 0, covered: 0 };
	230	+ cur.members++;
	231	+ if (m.covered) cur.covered++;
	232	+ byCountry.set(c, cur);
	233	+ }
	234	+ return { sector: s.sector, label: s.label, description: s.description ?? null, weight: s.weight, categories: s.categories, members: flat.size, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, shadow_members: [...flat.values()].filter((m) => !m.covered && m.shadow > 0).length, queued: [...flat.values()].filter((m) => !m.covered && (m.seed_status === "queued" \|\| m.seed_status === "discovering")).length, universes, by_country: [...byCountry.entries()].map(([country, v]) => ({ country, ...v })).sort((x, y) => y.members - x.members), generated_at: new Date().toISOString() };
	235	+ });
	236	+}

modified apps/api/src/live.ts +9 −0

		`@@ -49,6 +49,15 @@ export async function engineStatus(): Promise<Record<string, unknown> \| null> {`
49	49	}
50	50	}
51	51
	52	+export async function factoryStatus(): Promise<Record<string, unknown> \| null> {
	53	+ try {
	54	+ const raw = await getCmd().get("ws:factory:status");
	55	+ return raw ? (JSON.parse(raw) as Record<string, unknown>) : null;
	56	+ } catch {
	57	+ return null;
	58	+ }
	59	+}
	60	+
52	61	function getCmd(): Redis {
53	62	if (!cmd) cmd = new Redis(config.redisUrl, { maxRetriesPerRequest: 2, lazyConnect: false });
54	63	return cmd;

modified apps/api/src/queries.ts +5 −3

		`@@ -127,7 +127,9 @@ export async function stats(): Promise<Record<string, unknown>> {`
127	127	await db.execute<Record<string, unknown>>(sql`
128	128	select
129	129	(select count(*) from sources where enabled and kind = 'registry') as sources,
130		− (select count(*) from sensors where enabled) as sensors,
	130	+ (select count(*) from sensors where enabled and status <> 'SHADOW') as sensors,
	131	+ (select count(*) from sensors where enabled and status = 'SHADOW') as sensors_shadow,
	132	+ (select count(*) from sources where enabled and kind = 'registry' and origin = 'factory') as sources_factory,
131	133	(select count(*) from entities) as entities,
132	134	(select count(*) from snapshots) as snapshots,
133	135	(select count(*) from events) as events_total,
		`@@ -144,8 +146,8 @@ export async function stats(): Promise<Record<string, unknown>> {`
144	146	(select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour,
145	147	(select count(*) from sensor_runs where started_at >= now() - interval '5 minutes') as checks_last_5m,
146	148	(select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified') as not_modified_last_5m,
147		− (select count(*) from sensors where health = 'UP' and enabled) as sensors_up,
148		− (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled) as sensors_degraded,
	149	+ (select count(*) from sensors where health = 'UP' and enabled and status <> 'SHADOW') as sensors_up,
	150	+ (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled and status <> 'SHADOW') as sensors_degraded,
149	151	(select count(*) from sources where enabled and first_party and kind = 'registry') as sources_first_party,
150	152	(select count(distinct country) from sources where country is not null) as countries,
151	153	(select max(started_at) from sensor_runs) as last_check_at,

modified apps/api/src/routes-admin.ts +76 −1

		`@@ -8,7 +8,7 @@ import { getConnector, listConnectors, NormalizeError } from "@websensor/connect`
8	8	import { db, sql } from "@websensor/db";
9	9	import { cacheStats, invalidate } from "./cache";
10	10	import { config } from "./config";
11		−import { engineStatus, liveStats } from "./live";
	11	+import { engineStatus, factoryStatus, liveStats } from "./live";
12	12
13	13	/**
14	14	* Internal operations (spec §70–72, §85). Enabled only when WS_ADMIN_TOKEN is set; every request
		`@@ -193,6 +193,81 @@ export async function registerAdminRoutes(app: FastifyInstance): Promise<void> {`
193	193	return { dry_run: false, imported_sources: valid.filter((v) => !v.extend).length, extended_sources: valid.filter((v) => v.extend).length, sensors, invalid: report.filter((r) => !r.ok).length, report };
194	194	});
195	195
	196	+ // ---- Source Factory ---------------------------------------------------------------------------------
	197	+ app.get("/api/v1/admin/factory", async () => {
	198	+ const [seeds, cands, shadows, daily, sectors, recent, hb, ready] = await Promise.all([
	199	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from factory_seeds group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [String(x.status), Number(x.n)]))),
	200	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from discovery_candidates group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [String(x.status), Number(x.n)]))),
	201	+ db.execute<Record<string, unknown>>(sql`select count()::int as total, count() filter (where health = 'UP')::int as up, count() filter (where health <> 'UP')::int as unhealthy, count() filter (where total_runs >= 5)::int as ready, coalesce(sum(raw_changes),0)::int as raw_changes from sensors where status = 'SHADOW' and enabled`).then((r) => r.rows[0]),
	202	+ db.execute<Record<string, unknown>>(sql`select day::text as day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked from factory_daily order by day desc limit 30`).then((r) => r.rows),
	203	+ db.execute<Record<string, unknown>>(sql`select coalesce(sector, '-') as sector, count()::int as seeds, count() filter (where status = 'queued')::int as queued, count() filter (where status = 'discovering')::int as discovering, count() filter (where status in ('discovered','done'))::int as discovered, count() filter (where status = 'blocked')::int as blocked, count() filter (where status = 'error')::int as errors, coalesce(sum(shadow),0)::int as shadow, coalesce(sum(accepted),0)::int as accepted, coalesce(sum(rejected),0)::int as rejected from factory_seeds group by 1 order by seeds desc`).then((r) => r.rows),
	204	+ db.execute<Record<string, unknown>>(sql`select id, name, domain, sector, status, candidates, shadow, accepted, rejected, last_error, discovered_at, source_id from factory_seeds where status <> 'queued' order by updated_at desc limit 40`).then((r) => r.rows),
	205	+ factoryStatus(),
	206	+ db.execute<Record<string, unknown>>(sql`select kind_class, count(*)::int as n from discovery_candidates where status = 'shadow' group by 1 order by n desc`).then((r) => r.rows),
	207	+ ]);
	208	+ return { seeds, candidates: cands, shadow: shadows, shadow_by_kind: ready, daily, sectors, recent_seeds: recent, heartbeat: hb, generated_at: new Date().toISOString() };
	209	+ });
	210	+ app.get<{ Querystring: { status?: string; sector?: string; q?: string; limit?: string } }>("/api/v1/admin/factory/seeds", async (req) => {
	211	+ const lim = Math.min(1000, Number(req.query.limit ?? 200));
	212	+ const rows = await db.execute<Record<string, unknown>>(sql`select * from factory_seeds where 1=1 ${req.query.status ? sql`and status = ${req.query.status}` : sql``} ${req.query.sector ? sql`and sector = ${req.query.sector}` : sql``} ${req.query.q ? sql`and (name ilike ${"%" + req.query.q + "%"} or domain ilike ${"%" + req.query.q + "%"})` : sql``} order by importance desc, updated_at desc limit ${lim}`);
	213	+ return { items: rows.rows };
	214	+ });
	215	+ /** Bulk seeds `{ seeds: [{ name, domain, categories?, country?, tier?, importance?, aliases?, sector?, universe?, hints? }] }` — queued for the factory process. */
	216	+ app.post("/api/v1/admin/factory/seeds", async (req) => {
	217	+ const seedSchema = z.object({ id: z.string().regex(/^[a-z0-9][a-z0-9-]*$/).optional(), name: z.string().min(1), domain: z.string().min(3), categories: z.array(z.string()).default([]), country: z.string().regex(/^[A-Z]{2,3}$/).optional(), language: z.string().optional(), tier: z.enum(["S", "A", "B", "C", "D"]).default("B"), importance: z.number().int().min(1).max(3).default(2), aliases: z.array(z.string()).default([]), first_party: z.boolean().default(true), sector: z.string().optional(), universe: z.string().optional(), hints: z.record(z.string(), z.unknown()).default({}) });
	218	+ const body = z.object({ seeds: z.array(seedSchema).min(1).max(2000) }).parse(req.body ?? {});
	219	+ let n = 0;
	220	+ for (const s of body.seeds) {
	221	+ const domain = s.domain.toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "");
	222	+ const id = s.id ?? domain.replace(/[^a-z0-9]+/g, "-").replace(/^-\|-$/g, "").slice(0, 60);
	223	+ await db.execute(sql`insert into factory_seeds (id, name, domain, categories, country, language, tier, importance, aliases, first_party, sector, universe, hints, status)
	224	+ values (${id}, ${s.name}, ${domain}, ${sql.raw("'{" + s.categories.map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.country ?? null}, ${s.language ?? null}, ${s.tier}, ${s.importance}, ${sql.raw("'{" + s.aliases.map((c) => '"' + c.replace(/["\\]/g, "") + '"').join(",") + "}'::text[]")}, ${s.first_party}, ${s.sector ?? null}, ${s.universe ?? null}, ${JSON.stringify(s.hints)}::jsonb, 'queued')
	225	+ on conflict (id) do update set name = excluded.name, hints = factory_seeds.hints \|\| excluded.hints, status = 'queued', updated_at = now()`);
	226	+ n++;
	227	+ }
	228	+ invalidate("coverage");
	229	+ return { ok: true, queued: n };
	230	+ });
	231	+ app.post<{ Params: { id: string } }>("/api/v1/admin/factory/seeds/:id/requeue", async (req, reply) => {
	232	+ const r = await db.execute(sql`update factory_seeds set status = 'queued', updated_at = now() where id = ${req.params.id}`);
	233	+ if (!r.rowCount) return reply.status(404).send({ error: "not_found" });
	234	+ return { ok: true };
	235	+ });
	236	+ app.get<{ Querystring: { status?: string; seed?: string; source?: string; kind?: string; limit?: string } }>("/api/v1/admin/factory/candidates", async (req) => {
	237	+ const lim = Math.min(1000, Number(req.query.limit ?? 200));
	238	+ const rows = await db.execute<Record<string, unknown>>(sql`select c.*, s.health as shadow_health, s.total_runs as shadow_runs, s.raw_changes as shadow_changes from discovery_candidates c left join sensors s on s.id = c.shadow_sensor_id where 1=1 ${req.query.status ? sql`and c.status = ${req.query.status}` : sql``} ${req.query.seed ? sql`and c.seed_id = ${req.query.seed}` : sql``} ${req.query.source ? sql`and c.source_id = ${req.query.source}` : sql``} ${req.query.kind ? sql`and c.kind_class = ${req.query.kind}` : sql``} order by c.updated_at desc limit ${lim}`);
	239	+ return { items: rows.rows };
	240	+ });
	241	+ /** Manual decision on a shadow sensor: accept now (→ ACTIVE) or reject (→ DISABLED). */
	242	+ app.post<{ Params: { id: string; action: string } }>("/api/v1/admin/factory/shadow/:id/:action", async (req, reply) => {
	243	+ const s = (await db.execute<Record<string, unknown>>(sql`select id, config, status from sensors where id = ${req.params.id}`)).rows[0];
	244	+ if (!s) return reply.status(404).send({ error: "not_found" });
	245	+ const reason = `manual ${req.params.action} via admin API`;
	246	+ if (req.params.action === "accept") {
	247	+ const tp = Number(((s.config as Record<string, unknown>).targetPriority as number \| undefined) ?? 2);
	248	+ await db.execute(sql`update sensors set status = 'ACTIVE', enabled = true, priority = ${tp}, config = (config - 'shadow') \|\| jsonb_build_object('acceptedAt', now()::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${req.params.id}`);
	249	+ await db.execute(sql`update discovery_candidates set status = 'accepted', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${req.params.id}`);
	250	+ } else if (req.params.action === "reject") {
	251	+ await db.execute(sql`update sensors set status = 'DISABLED', enabled = false, config = config \|\| jsonb_build_object('rejectedAt', now()::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${req.params.id}`);
	252	+ await db.execute(sql`update discovery_candidates set status = 'rejected', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${req.params.id}`);
	253	+ } else return reply.status(400).send({ error: "action must be accept or reject" });
	254	+ invalidate("");
	255	+ return { ok: true };
	256	+ });
	257	+ /** Promote a plain candidate (never shadowed) straight into shadow monitoring. */
	258	+ app.post<{ Params: { id: string } }>("/api/v1/admin/factory/candidates/:id/shadow", async (req, reply) => {
	259	+ const c = (await db.execute<Record<string, unknown>>(sql`select * from discovery_candidates where id = ${req.params.id}`)).rows[0];
	260	+ if (!c) return reply.status(404).send({ error: "not_found" });
	261	+ if (!c.connector \|\| !c.type) return reply.status(400).send({ error: "candidate has no connector/type (legacy discovery row)" });
	262	+ const src = (await db.execute<Record<string, unknown>>(sql`select categories from sources where id = ${String(c.source_id)}`)).rows[0];
	263	+ const id = `${String(c.source_id)}_${String(c.name ?? c.kind_class ?? "page").toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-\|-$/g, "")}-${String(c.id).slice(-5)}`.slice(0, 80);
	264	+ await db.execute(sql`insert into sensors (id, source_id, name, url, type, connector, tier, config, status, priority, validated_at)
	265	+ values (${id}, ${String(c.source_id)}, ${String(c.name ?? c.kind_class ?? "page")}, ${String(c.url)}, ${String(c.type)}, ${String(c.connector)}, ${String(c.tier ?? "C")}, ${JSON.stringify({ ...((c.config as Record<string, unknown>) ?? {}), factory: true, shadow: true, seed: false, kind: c.kind_class, candidate: c.id, seedId: c.seed_id, shadowSince: new Date().toISOString(), targetPriority: 2, categories: src?.categories ?? [] })}::jsonb, 'SHADOW', 3, now())
	266	+ on conflict (id) do nothing`);
	267	+ await db.execute(sql`update discovery_candidates set status = 'shadow', shadow_sensor_id = ${id}, updated_at = now() where id = ${req.params.id}`);
	268	+ return { ok: true, sensor_id: id };
	269	+ });
	270	+
196	271	app.get("/api/v1/admin/failures", async () => {
197	272	const rows = await db.execute<Record<string, unknown>>(sql`select r.started_at, r.outcome, r.http_status, r.error, r.duration_ms, s.id as sensor_id, s.name, s.source_id, s.url, s.connector from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error','rate_limited') and r.started_at >= now() - interval '24 hours' order by r.started_at desc limit 200`);
198	273	return { items: rows.rows };

modified apps/api/src/routes.ts +10 −1

		`@@ -5,6 +5,7 @@ import { db, sql, textArray } from "@websensor/db";`
5	5	import { getBlobStore } from "@websensor/store";
6	6	import { cached } from "./cache";
7	7	import { config } from "./config";
	8	+import { coverageSector, coverageSummary } from "./coverage";
8	9	import { breakingDesk, categoryDesk, clusterDetail, countryDesk, countryList, entityInsights, entityRankings, pulse, radar, sourceSensors } from "./intel";
9	10	import { engineStatus, liveStats } from "./live";
10	11	import { countEvents, getEvent, listEvents, relatedEvents, sensorHistory, sourceActivity, sourceQuality, stats, trending } from "./queries";
		`@@ -139,7 +140,8 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {`
139	140	const q = req.query.q;
140	141	const rows = await db.execute<Record<string, unknown>>(sql`
141	142	select s.id, s.name, s.domain, s.homepage, s.description, s.categories, s.tier, s.importance_weight, s.enabled, s.notes, s.first_party, s.country, s.language, s.robots_checked_at,
142		− (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count,
	143	+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.status <> 'SHADOW')::int as sensor_count,
	144	+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.status = 'SHADOW')::int as shadow_count, s.origin, s.sector,
143	145	(select count(*) from events e where e.source_id = s.id)::int as event_count,
144	146	(select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h,
145	147	(select max(detected_at) from events e where e.source_id = s.id) as last_event_at,
		`@@ -247,6 +249,13 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {`
247	249
248	250	// ---- Intelligence desks ----------------------------------------------------------------------
249	251	app.get("/api/v1/stats", async () => cached("stats", 5_000, stats));
	252	+ // ---- Global Observation Coverage Score ------------------------------------------------------
	253	+ app.get("/api/v1/coverage", async () => coverageSummary());
	254	+ app.get<{ Params: { sector: string } }>("/api/v1/coverage/:sector", async (req, reply) => {
	255	+ const r = await coverageSector(req.params.sector.toLowerCase());
	256	+ if (!r) return reply.status(404).send({ error: "not_found" });
	257	+ return r;
	258	+ });
250	259	app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => {
251	260	const hours = Math.min(168, Number(req.query.hours ?? 24));
252	261	const limit = Math.min(50, Number(req.query.limit ?? 12));

modified apps/engine/src/cli.ts +49 −1

		`@@ -80,8 +80,56 @@ async function main(): Promise<void> {`
80	80	console.log(JSON.stringify(out, null, 2), `\n${Date.now() - t0} ms`);
81	81	break;
82	82	}
	83	+ case "expand": {
	84	+ // Deep discovery dry-run (no DB writes): what the Source Factory would find for a domain.
	85	+ const { discoverOrganization } = await import("@websensor/connectors");
	86	+ const { scoreCandidate } = await import("./factory/score");
	87	+ const hintsArg = args.find((a) => a.startsWith("--hints="));
	88	+ const hints = hintsArg ? (JSON.parse(hintsArg.slice(8)) as Record<string, unknown>) : {};
	89	+ const r = await discoverOrganization(args[0]!, { hints, probePosture: args.includes("--posture") });
	90	+ console.log(`${r.domain} → origin ${r.origin} · home ${r.homeStatus}${r.blocked ? " BLOCKED" : ""} · ${r.requests} requests · ${r.durationMs} ms`);
	91	+ for (const n of r.notes) console.log(` note: ${n}`);
	92	+ for (const c of r.candidates) {
	93	+ const s = scoreCandidate(c, { importance: Number(args.find((a) => a.startsWith("--importance="))?.slice(13) ?? 2) });
	94	+ console.log(`${s.score.toFixed(2)} ${c.kind.padEnd(10)} ${c.connector.padEnd(10)} ${c.suggestedTier} ${String(c.itemCount ?? "").padStart(4)} ${c.url} (${c.evidence})`);
	95	+ }
	96	+ if (r.rejected.length) console.log(`rejected: ${r.rejected.map((x) => `${x.url} [${x.reason}]`).join("\n ")}`);
	97	+ break;
	98	+ }
	99	+ case "factory": {
	100	+ const f = await import("./factory");
	101	+ const sub = args[0];
	102	+ const flagVal = (n: string): string \| undefined => args.find((a) => a.startsWith(`--${n}=`))?.slice(n.length + 3);
	103	+ if (sub === "seed") {
	104	+ const files = await f.seedFromFiles();
	105	+ const mode = (flagVal("mode") ?? "hinted") as "uncovered" \| "hinted" \| "all";
	106	+ const cov = await f.seedFromCoverage({ mode, sectors: flagVal("sector")?.split(","), requeue: args.includes("--requeue") });
	107	+ console.log(JSON.stringify({ files, coverage: cov }, null, 2));
	108	+ } else if (sub === "run") {
	109	+ const n = Number(flagVal("n") ?? 6);
	110	+ const ids = args.slice(1).filter((a) => !a.startsWith("--"));
	111	+ const out = await f.runFactoryBatch(n, ids.length ? { seedIds: ids } : {});
	112	+ for (const o of out) console.log(`${o.status.padEnd(10)} ${o.seedId.padEnd(40)} source=${o.sourceId ?? "-"} candidates=${o.candidates} shadow=${o.shadow} requests=${o.requests} ${o.durationMs} ms${o.notes.length ? " · " + o.notes.join(" · ") : ""}`);
	113	+ } else if (sub === "evaluate") {
	114	+ const r = await f.evaluateShadows({ force: args.includes("--force") });
	115	+ for (const d of r.decisions.filter((x) => x.decision !== "deferred" \|\| args.includes("--verbose"))) console.log(`${d.decision.padEnd(9)} ${d.sensorId.padEnd(60)} ${d.reason}`);
	116	+ console.log(JSON.stringify({ accepted: r.accepted, rejected: r.rejected, deferred: r.deferred }));
	117	+ } else if (sub === "stats") {
	118	+ console.log(JSON.stringify(await f.factoryStats(), null, 2));
	119	+ } else if (sub === "export") {
	120	+ process.stdout.write(await f.exportFactoryFragment({ sector: flagVal("sector"), sinceDays: flagVal("since") ? Number(flagVal("since")) : undefined, includeShadow: args.includes("--include-shadow") }));
	121	+ } else if (sub === "requeue") {
	122	+ const ids = args.slice(1).filter((a) => !a.startsWith("--"));
	123	+ const r = await db.execute(sql`update factory_seeds set status = 'queued', updated_at = now() where ${ids.length ? sql`id = any(${sql.raw("array[" + ids.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]")})` : sql`status in ('error','blocked')`}`);
	124	+ console.log(`requeued ${r.rowCount ?? 0}`);
	125	+ } else {
	126	+ console.error("usage: cli.ts factory seed [--mode=uncovered\|hinted\|all] [--sector=a,b] [--requeue] \| run [seedId…] [--n=6] \| evaluate [--force] [--verbose] \| stats \| export [--sector=x] [--since=days] [--include-shadow] \| requeue [seedId…]");
	127	+ process.exitCode = 1;
	128	+ }
	129	+ break;
	130	+ }
83	131	default:
84		− console.error("usage: cli.ts sync \| discover [sourceId…] \| probe <domain> \| run-once <sensorId> \| run-due [n] \| relink-entities [days] \| prune-blobs [batch] \| refresh-clusters");
	132	+ console.error("usage: cli.ts sync \| discover [sourceId…] \| probe <domain> \| expand <domain> [--hints=json] [--posture] \| run-once <sensorId> \| run-due [n] \| relink-entities [days] \| prune-blobs [batch] \| refresh-clusters \| factory <seed\|run\|evaluate\|stats\|export\|requeue>");
85	133	process.exitCode = 1;
86	134	}
87	135	}

modified apps/engine/src/config.ts +22 −1

		`@@ -48,7 +48,7 @@ export const config = {`
48	48	retention: { rawDays: Number(env.WS_RETENTION_RAW_DAYS ?? 21), changedRawDays: Number(env.WS_RETENTION_CHANGED_RAW_DAYS ?? 60), enabled: (env.WS_RETENTION ?? "1") !== "0" },
49	49	/** silent-change bar: only eligible types with importance ≥ this become "silent" */
50	50	silentMinImportance: Number(env.WS_SILENT_MIN_IMPORTANCE ?? 45),
51		− version: "0.2.0",
	51	+ version: "0.3.0",
52	52	};
53	53
54	54	export const log = pino({
		`@@ -56,3 +56,24 @@ export const log = pino({`
56	56	...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}),
57	57	base: { service: "websensor-engine" },
58	58	});
	59	+
	60	+/** Source Factory (apps/engine/src/factory) — organization expansion, shadow monitoring, acceptance. */
	61	+export const factoryConfig = {
	62	+ enabled: (env.WS_FACTORY ?? "1") !== "0",
	63	+ /** seeds discovered in parallel */
	64	+ concurrency: Number(env.WS_FACTORY_CONCURRENCY ?? 6),
	65	+ /** HTTP requests per organization during discovery */
	66	+ budget: Number(env.WS_FACTORY_BUDGET ?? 90),
	67	+ /** minimum candidate score to enter shadow monitoring */
	68	+ minScore: Number(env.WS_FACTORY_MIN_SCORE ?? 0.35),
	69	+ /** shadow evaluation: checks needed, minimum and maximum shadow duration */
	70	+ shadowMinChecks: Number(env.WS_FACTORY_SHADOW_MIN_CHECKS ?? 5),
	71	+ shadowMinHours: Number(env.WS_FACTORY_SHADOW_MIN_HOURS ?? 24),
	72	+ shadowMaxHours: Number(env.WS_FACTORY_SHADOW_MAX_HOURS ?? 120),
	73	+ /** per-organization promotion caps */
	74	+ caps: { feeds: 4, pages: 6, releases: 4, sitemap: 1, status: 1, models: 1, filings: 1, api: 1, posture: 3, legal: 2, total: 18 },
	75	+ /** seconds between factory ticks */
	76	+ tickSeconds: Number(env.WS_FACTORY_TICK_SECONDS ?? 20),
	77	+ coverageDir: env.WS_COVERAGE_DIR ?? "./config/coverage",
	78	+ seedsDir: env.WS_FACTORY_SEEDS_DIR ?? "./config/factory/seeds",
	79	+};

added apps/engine/src/coverage-validate.ts +108 −0

		`@@ -0,0 +1,108 @@`
	1	+import { promises as dns } from "node:dns";
	2	+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
	3	+import { join } from "node:path";
	4	+import YAML from "yaml";
	5	+import { coverageKey, coverageSectorSchema, type CoverageSector } from "@websensor/core";
	6	+
	7	+/**
	8	+ * Coverage-universe validator — no database.
	9	+ *
	10	+ * node node_modules/tsx/dist/cli.mjs apps/engine/src/coverage-validate.ts [config/coverage/<file>.yaml …] [--dns] [--quiet]
	11	+ *
	12	+ * Without arguments every file in config/coverage is checked. Reports schema issues, duplicate domains inside a
	13	+ * file, duplicate sector keys across files and (with --dns) domains that do not resolve. Exit code 1 on any error.
	14	+ */
	15	+const args = process.argv.slice(2);
	16	+const flag = (n: string): boolean => args.includes(`--${n}`);
	17	+const files = args.filter((a) => !a.startsWith("--"));
	18	+
	19	+export function loadCoverageFiles(dir = process.env.WS_COVERAGE_DIR ?? "./config/coverage"): { file: string; sector: CoverageSector \| null; issues: string[] }[] {
	20	+ const list = files.length ? files : existsSync(dir) && statSync(dir).isDirectory() ? readdirSync(dir).filter((f) => /\.ya?ml$/.test(f) && !f.startsWith("_")).sort().map((f) => join(dir, f)) : [];
	21	+ return list.map((file) => {
	22	+ const issues: string[] = [];
	23	+ let sector: CoverageSector \| null = null;
	24	+ try {
	25	+ const raw = YAML.parse(readFileSync(file, "utf8"));
	26	+ const parsed = coverageSectorSchema.safeParse(raw);
	27	+ if (!parsed.success) {
	28	+ for (const i of parsed.error.issues.slice(0, 40)) issues.push(`${i.path.join(".")}: ${i.message}`);
	29	+ if (parsed.error.issues.length > 40) issues.push(`… ${parsed.error.issues.length - 40} more issues`);
	30	+ } else sector = parsed.data;
	31	+ } catch (e) {
	32	+ issues.push(`YAML parse error: ${(e as Error).message}`);
	33	+ }
	34	+ if (sector) {
	35	+ const seen = new Map<string, string>();
	36	+ const keys = new Set<string>();
	37	+ for (const u of sector.universes) {
	38	+ if (keys.has(u.key)) issues.push(`duplicate universe key ${u.key}`);
	39	+ keys.add(u.key);
	40	+ if (!u.members.length) issues.push(`universe ${u.key} has no members`);
	41	+ for (const m of u.members) {
	42	+ if (!/^[a-z0-9.-]+\.[a-z]{2,}$/.test(m.domain) \|\| m.domain.includes("..")) issues.push(`${u.key}: bad domain "${m.domain}" (${m.name})`);
	43	+ const k = coverageKey(m.domain);
	44	+ const prev = seen.get(k);
	45	+ if (prev && prev !== `${u.key}/${m.name}`) issues.push(`${u.key}: duplicate domain ${m.domain} (${m.name}) — already ${prev}`);
	46	+ seen.set(k, `${u.key}/${m.name}`);
	47	+ if (!m.country && !/^(INT\|EU)$/.test(String(m.country))) issues.push(`${u.key}: ${m.name} has no country`);
	48	+ }
	49	+ }
	50	+ }
	51	+ return { file, sector, issues };
	52	+ });
	53	+}
	54	+
	55	+async function resolves(domain: string): Promise<boolean> {
	56	+ for (const h of [domain, `www.${domain}`]) {
	57	+ try {
	58	+ const r = await dns.lookup(h, { all: true });
	59	+ if (r.length) return true;
	60	+ } catch {
	61	+ // try next
	62	+ }
	63	+ }
	64	+ return false;
	65	+}
	66	+
	67	+async function main(): Promise<void> {
	68	+ const loaded = loadCoverageFiles();
	69	+ if (!loaded.length) {
	70	+ console.error("no coverage files found");
	71	+ process.exitCode = 1;
	72	+ return;
	73	+ }
	74	+ let errors = 0;
	75	+ const sectors = new Map<string, string>();
	76	+ for (const l of loaded) {
	77	+ const members = l.sector?.universes.reduce((n, u) => n + u.members.length, 0) ?? 0;
	78	+ const warnIssues = l.issues.filter((i) => / has no country$/.test(i));
	79	+ const hardIssues = l.issues.filter((i) => !/ has no country$/.test(i));
	80	+ errors += hardIssues.length;
	81	+ // Several files may extend the same sector (e.g. corporate indices per region); universes are merged by the loader.
	82	+ if (l.sector && !sectors.has(l.sector.sector)) sectors.set(l.sector.sector, l.file);
	83	+ console.log(`${hardIssues.length ? "FAIL" : "OK "} ${l.file} sector=${l.sector?.sector ?? "-"} universes=${l.sector?.universes.length ?? 0} members=${members}${warnIssues.length ? ` warn=${warnIssues.length}` : ""}`);
	84	+ for (const i of hardIssues) console.log(` ${i}`);
	85	+ if (!flag("quiet")) for (const i of warnIssues.slice(0, 15)) console.log(` WARN ${i}`);
	86	+ if (flag("dns") && l.sector) {
	87	+ const domains = [...new Set(l.sector.universes.flatMap((u) => u.members.map((m) => m.domain)))];
	88	+ let i = 0;
	89	+ const bad: string[] = [];
	90	+ await Promise.all(
	91	+ Array.from({ length: 16 }, async () => {
	92	+ while (i < domains.length) {
	93	+ const d = domains[i++]!;
	94	+ if (!(await resolves(d))) bad.push(d);
	95	+ }
	96	+ }),
	97	+ );
	98	+ for (const d of bad.sort()) console.log(` DNS ${d} does not resolve`);
	99	+ if (bad.length) console.log(` ${bad.length}/${domains.length} domains do not resolve (fix or remove them)`);
	100	+ errors += bad.length;
	101	+ }
	102	+ }
	103	+ const total = loaded.reduce((n, l) => n + (l.sector?.universes.reduce((m, u) => m + u.members.length, 0) ?? 0), 0);
	104	+ console.log(`\n${loaded.length} files · ${sectors.size} sectors · ${total} members → ${errors ? `${errors} errors` : "all good"}`);
	105	+ if (errors) process.exitCode = 1;
	106	+}
	107	+
	108	+if (process.argv[1] && /coverage-validate\.ts$/.test(process.argv[1])) void main();

added apps/engine/src/factory-main.ts +99 −0

		`@@ -0,0 +1,99 @@`
	1	+import { closeDb, db, migrate, sql } from "@websensor/db";
	2	+import { closeDispatcher } from "@websensor/connectors";
	3	+import { config, factoryConfig, log } from "./config";
	4	+import { evaluateShadows, factoryStats, runFactoryBatch, seedFromCoverage, seedFromFiles } from "./factory";
	5	+import { closeRedis, getRedis } from "./redis";
	6	+
	7	+/**
	8	+ * Source Factory process (`websensor-factory`): a slow, polite background worker separate from the poller.
	9	+ * every tick — claim queued seeds (systemic organizations first) and run deep discovery + shadow creation
	10	+ * every 15 min — evaluate shadow sensors (accept / reject / defer)
	11	+ * every 24 h — re-seed from the coverage universes (new members become seeds automatically)
	12	+ * Heartbeat: Redis `ws:factory:status`.
	13	+ */
	14	+async function heartbeat(extra: Record<string, unknown>): Promise<void> {
	15	+ try {
	16	+ await getRedis().set("ws:factory:status", JSON.stringify({ ...extra, at: new Date().toISOString(), pid: process.pid, version: config.version }), "EX", 120);
	17	+ } catch {
	18	+ // best effort
	19	+ }
	20	+}
	21	+
	22	+async function main(): Promise<void> {
	23	+ log.info({ env: config.env, version: config.version, concurrency: factoryConfig.concurrency, budget: factoryConfig.budget }, "websensor factory starting");
	24	+ const applied = await migrate(config.databaseUrl);
	25	+ if (applied.length) log.info({ applied }, "migrations applied");
	26	+ if (!factoryConfig.enabled) {
	27	+ log.warn("WS_FACTORY=0 — factory idle");
	28	+ setInterval(() => void heartbeat({ idle: true }), 60_000);
	29	+ return;
	30	+ }
	31	+ let stopping = false;
	32	+ let busy = false;
	33	+ let processed = 0;
	34	+ let lastSeed = 0;
	35	+ let lastEval = 0;
	36	+
	37	+ const seed = async (): Promise<void> => {
	38	+ try {
	39	+ const a = await seedFromFiles();
	40	+ const b = await seedFromCoverage({ mode: "hinted" });
	41	+ log.info({ files: a, coverage: { members: b.members, seeds: b.seeds, inserted: b.inserted, issues: b.issues.length } }, "factory: seeds refreshed");
	42	+ for (const i of b.issues) log.warn({ issue: i }, "coverage file issue");
	43	+ } catch (e) {
	44	+ log.error({ err: (e as Error).message }, "factory: seeding failed");
	45	+ }
	46	+ };
	47	+
	48	+ const tick = async (): Promise<void> => {
	49	+ if (stopping \|\| busy) return;
	50	+ busy = true;
	51	+ try {
	52	+ if (Date.now() - lastSeed > 24 * 3600e3) {
	53	+ lastSeed = Date.now();
	54	+ await seed();
	55	+ }
	56	+ if (Date.now() - lastEval > 15 * 60e3) {
	57	+ lastEval = Date.now();
	58	+ await evaluateShadows().catch((e) => log.warn({ err: (e as Error).message }, "factory: shadow evaluation failed"));
	59	+ }
	60	+ const out = await runFactoryBatch(factoryConfig.concurrency);
	61	+ processed += out.length;
	62	+ const queued = Number((await db.execute<{ n: string }>(sql`select count(*)::text as n from factory_seeds where status = 'queued'`)).rows[0]?.n ?? 0);
	63	+ await heartbeat({ processed, queued, lastBatch: out.map((o) => ({ seed: o.seedId, status: o.status, candidates: o.candidates, shadow: o.shadow, requests: o.requests, ms: o.durationMs })), concurrency: factoryConfig.concurrency });
	64	+ if (!out.length && processed % 50 === 0) {
	65	+ const st = await factoryStats();
	66	+ log.info({ seeds: st.seeds, candidates: st.candidates, shadow: st.shadow }, "factory: idle");
	67	+ }
	68	+ } catch (e) {
	69	+ log.error({ err: (e as Error).stack ?? (e as Error).message }, "factory tick failed");
	70	+ } finally {
	71	+ busy = false;
	72	+ }
	73	+ };
	74	+
	75	+ await heartbeat({ starting: true });
	76	+ void tick();
	77	+ const timer = setInterval(() => void tick(), factoryConfig.tickSeconds * 1000);
	78	+
	79	+ const shutdown = async (signal: string): Promise<void> => {
	80	+ if (stopping) return;
	81	+ stopping = true;
	82	+ log.info({ signal }, "factory shutting down");
	83	+ clearInterval(timer);
	84	+ const deadline = Date.now() + 60_000;
	85	+ while (busy && Date.now() < deadline) await new Promise((r) => setTimeout(r, 250));
	86	+ await closeDispatcher();
	87	+ await closeRedis();
	88	+ await closeDb();
	89	+ process.exit(0);
	90	+ };
	91	+ process.on("SIGINT", () => void shutdown("SIGINT"));
	92	+ process.on("SIGTERM", () => void shutdown("SIGTERM"));
	93	+ process.on("unhandledRejection", (e) => log.error({ err: e instanceof Error ? e.stack : String(e) }, "unhandled rejection"));
	94	+}
	95	+
	96	+main().catch((e) => {
	97	+ log.fatal({ err: (e as Error).stack ?? String(e) }, "factory failed to start");
	98	+ process.exit(1);
	99	+});

added apps/engine/src/factory/export.ts +46 −0

		`@@ -0,0 +1,46 @@`
	1	+import YAML from "yaml";
	2	+import { db, sql } from "@websensor/db";
	3	+
	4	+/**
	5	+ * Export factory-created sources and accepted sensors as a registry fragment (reviewable YAML). Graduating a
	6	+ * fragment into `config/sources.d/` makes the registry sync own those sensors (`seed: true`) — the Factory never
	7	+ * re-creates them because their URLs already exist.
	8	+ */
	9	+export async function exportFactoryFragment(opts: { sector?: string; sinceDays?: number; includeShadow?: boolean } = {}): Promise<string> {
	10	+ const rows = await db.execute<Record<string, unknown>>(sql`
	11	+ select so.id, so.name, so.domain, so.homepage, so.categories, so.tier, so.importance_weight, so.country, so.language, so.first_party, so.origin, so.sector, so.notes,
	12	+ coalesce(json_agg(json_build_object('name', s.name, 'url', s.url, 'type', s.type, 'connector', s.connector, 'tier', s.tier, 'config', s.config - 'factory' - 'shadow' - 'seed' - 'candidate' - 'seedId' - 'shadowSince' - 'targetPriority' - 'acceptedAt' - 'shadowReport' - 'kind') order by s.name) filter (where s.id is not null), '[]'::json) as sensors
	13	+ from sources so
	14	+ left join sensors s on s.source_id = so.id and s.enabled and (s.config->>'factory') = 'true' and (s.status <> 'SHADOW' or ${opts.includeShadow ?? false})
	15	+ where (so.origin = 'factory' or exists (select 1 from sensors y where y.source_id = so.id and (y.config->>'factory') = 'true'))
	16	+ ${opts.sector ? sql`and so.sector = ${opts.sector}` : sql``}
	17	+ ${opts.sinceDays ? sql`and so.updated_at >= now() - make_interval(days => ${opts.sinceDays})` : sql``}
	18	+ group by so.id order by so.sector, so.id`);
	19	+ const entities = (await db.execute<{ source_id: string; alias: string }>(sql`select se.source_id, ea.alias from source_entities se join entity_aliases ea on ea.entity_id = se.entity_id where se.entity_id like 'org_%'`)).rows;
	20	+ const aliasBySource = new Map<string, string[]>();
	21	+ for (const e of entities) aliasBySource.set(e.source_id, [...(aliasBySource.get(e.source_id) ?? []), e.alias]);
	22	+ const out: Record<string, unknown>[] = [];
	23	+ for (const r of rows.rows) {
	24	+ const sensors = (r.sensors as Record<string, unknown>[]).map((s) => {
	25	+ const cfg = s.config as Record<string, unknown>;
	26	+ const o: Record<string, unknown> = { name: s.name, url: s.url, type: s.type, connector: s.connector, tier: s.tier };
	27	+ if (cfg && Object.keys(cfg).length) o.config = cfg;
	28	+ return o;
	29	+ });
	30	+ if (!sensors.length) continue;
	31	+ const aliases = (aliasBySource.get(String(r.id)) ?? []).filter((a) => a !== String(r.name).toLowerCase() && a !== String(r.domain).toLowerCase() && a !== String(r.domain).replace(/^www\./, "").toLowerCase());
	32	+ if (r.origin === "factory") {
	33	+ const src: Record<string, unknown> = { id: r.id, name: r.name, domain: r.domain, categories: r.categories, tier: r.tier };
	34	+ if (Number(r.importance_weight) !== 1) src.weight = Number(r.importance_weight);
	35	+ if (aliases.length) src.aliases = aliases;
	36	+ if (r.country) src.country = r.country;
	37	+ if (r.language) src.language = r.language;
	38	+ if (r.first_party === false) src.first_party = false;
	39	+ if (r.notes) src.notes = r.notes;
	40	+ src.sensors = sensors;
	41	+ out.push(src);
	42	+ } else out.push({ id: r.id, extend: true, sensors });
	43	+ }
	44	+ const header = `# WebSensor — Source Factory export (${new Date().toISOString().slice(0, 10)}${opts.sector ? `, sector ${opts.sector}` : ""}): ${out.length} organizations, ${out.reduce((n, s) => n + (s.sensors as unknown[]).length, 0)} accepted sensors.\n# Generated by \`cli.ts factory export\`; review, then place in config/sources.d/ to let the registry own these sensors.\n`;
	45	+ return header + YAML.stringify({ sources: out }, { lineWidth: 0, defaultStringType: "PLAIN", defaultKeyType: "PLAIN" });
	46	+}

added apps/engine/src/factory/factory.test.ts +67 −0

		`@@ -0,0 +1,67 @@`
	1	+import { describe, expect, it } from "vitest";
	2	+import { coverageKey, registrableDomain } from "@websensor/core";
	3	+import type { DeepCandidate } from "@websensor/connectors";
	4	+import { itemsPerDayOf } from "@websensor/connectors";
	5	+import { KIND_WEIGHT, scoreCandidate, selectForShadow } from "./score";
	6	+
	7	+const cand = (over: Partial<DeepCandidate>): DeepCandidate => ({ url: "https://example.org/feed", type: "RSS", connector: "rss", kind: "news", name: "news feed", config: {}, suggestedTier: "B", evidence: "well-known path", value: 0.8, itemsPerDay: 0.5, firstPartyConfidence: 1, fetchCost: { bytes: 20_000, ms: 400 }, ...over });
	8	+
	9	+describe("registrable domain (coverage key)", () => {
	10	+ it("collapses hosts to the organization's registrable domain", () => {
	11	+ expect(registrableDomain("ir.apple.com")).toBe("apple.com");
	12	+ expect(registrableDomain("www.ons.gov.uk")).toBe("ons.gov.uk");
	13	+ expect(registrableDomain("www.bankofengland.co.uk")).toBe("bankofengland.co.uk");
	14	+ expect(registrableDomain("statcan.gc.ca")).toBe("statcan.gc.ca");
	15	+ expect(registrableDomain("open.canada.ca")).toBe("canada.ca");
	16	+ expect(registrableDomain("boj.or.jp")).toBe("boj.or.jp");
	17	+ expect(coverageKey("https://WWW.Example.COM/path")).toBe("example.com");
	18	+ });
	19	+});
	20	+
	21	+describe("factory scoring", () => {
	22	+ it("weights page classes and organization importance", () => {
	23	+ const status = scoreCandidate(cand({ kind: "status", connector: "statuspage", value: 0.95, itemsPerDay: null }), { importance: 2 });
	24	+ const careers = scoreCandidate(cand({ kind: "careers", connector: "http", value: 0.35, itemsPerDay: null }), { importance: 2 });
	25	+ expect(status.score).toBeGreaterThan(careers.score);
	26	+ expect(KIND_WEIGHT.status).toBeGreaterThan(KIND_WEIGHT.careers);
	27	+ const big = scoreCandidate(cand({}), { importance: 3 });
	28	+ const small = scoreCandidate(cand({}), { importance: 1 });
	29	+ expect(big.score).toBeGreaterThan(small.score);
	30	+ expect(big.reasons.some((r) => /importance 3/.test(r))).toBe(true);
	31	+ });
	32	+ it("penalises firehoses, dormant feeds and slow endpoints, explainably", () => {
	33	+ const base = scoreCandidate(cand({}), { importance: 2 }).score;
	34	+ expect(scoreCandidate(cand({ itemsPerDay: 200 }), { importance: 2 }).score).toBeLessThan(base);
	35	+ expect(scoreCandidate(cand({ itemsPerDay: 0.001 }), { importance: 2 }).score).toBeLessThan(base);
	36	+ const slow = scoreCandidate(cand({ fetchCost: { bytes: 100, ms: 12_000 } }), { importance: 2 });
	37	+ expect(slow.score).toBeLessThan(base);
	38	+ expect(slow.reasons.some((r) => /slow/.test(r))).toBe(true);
	39	+ });
	40	+ it("applies per-organization caps and the minimum score", () => {
	41	+ const feeds = Array.from({ length: 6 }, (_, i) => scoreCandidate(cand({ url: `https://example.org/f${i}`, value: 0.9 - i * 0.02 }), { importance: 2 }));
	42	+ const low = scoreCandidate(cand({ url: "https://example.org/careers", kind: "careers", connector: "http", value: 0.2, itemsPerDay: null }), { importance: 1 });
	43	+ const legal = ["terms", "privacy", "cookies"].map((p) => scoreCandidate(cand({ url: `https://example.org/${p}`, kind: "legal", connector: "http", type: "HTML", value: 0.7, itemsPerDay: null }), { importance: 2 }));
	44	+ const { selected, skipped } = selectForShadow([...feeds, low, ...legal], { kinds: new Map(), total: 0 });
	45	+ expect(selected.filter((s) => s.cand.connector === "rss")).toHaveLength(4);
	46	+ expect(selected.filter((s) => s.cand.kind === "legal")).toHaveLength(2);
	47	+ expect(skipped.some((s) => s.s === low && /score/.test(s.reason))).toBe(true);
	48	+ expect(skipped.some((s) => /feeds cap/.test(s.reason))).toBe(true);
	49	+ });
	50	+ it("respects sensors the organization already has", () => {
	51	+ const feed = scoreCandidate(cand({}), { importance: 2 });
	52	+ const { selected, skipped } = selectForShadow([feed], { kinds: new Map([["news", 4]]), total: 4 });
	53	+ expect(selected).toHaveLength(0);
	54	+ expect(skipped[0]?.reason).toMatch(/feeds cap/);
	55	+ });
	56	+});
	57	+
	58	+describe("expected change frequency", () => {
	59	+ it("uses the median gap and survives one bogus date", () => {
	60	+ const day = 86400e3;
	61	+ const now = Date.UTC(2026, 8, 13);
	62	+ const dates = [now, now - day, now - 2 * day, now - 3 * day, now - 4 * day, now - 5000 * day];
	63	+ expect(itemsPerDayOf(dates)).toBe(1);
	64	+ expect(itemsPerDayOf([now, now - day])).toBeNull();
	65	+ expect(itemsPerDayOf([now, now - 30 * day, now - 60 * day, now - 90 * day])).toBeCloseTo(0.033, 2);
	66	+ });
	67	+});

added apps/engine/src/factory/index.ts +5 −0

		`@@ -0,0 +1,5 @@`
	1	+export * from "./seeds";
	2	+export * from "./score";
	3	+export * from "./run";
	4	+export * from "./shadow";
	5	+export * from "./export";

added apps/engine/src/factory/run.ts +288 −0

		`@@ -0,0 +1,288 @@`
	1	+import { coverageKey, newId, slugify, sourceSchema, type SourceSeed, type Tier } from "@websensor/core";
	2	+import { discoverOrganization, type CandidateKind, type DeepDiscoveryHints } from "@websensor/connectors";
	3	+import { db, factorySeeds, sensors, sql, type FactorySeed } from "@websensor/db";
	4	+import { factoryConfig, log } from "../config";
	5	+import { priorityFor, upsertSourceRecord } from "../registry";
	6	+import { scoreCandidate, selectForShadow } from "./score";
	7	+
	8	+/**
	9	+ * Source Factory — one organization at a time:
	10	+ * seed → deep discovery → (create or attach the source) → score candidates → promotion policy →
	11	+ * shadow sensors (`status = SHADOW`: observed, evidence stored, nothing published) → later `evaluateShadows()`.
	12	+ */
	13	+export interface ProcessOutcome {
	14	+ seedId: string;
	15	+ sourceId: string \| null;
	16	+ status: "discovered" \| "blocked" \| "error";
	17	+ candidates: number;
	18	+ shadow: number;
	19	+ requests: number;
	20	+ durationMs: number;
	21	+ notes: string[];
	22	+}
	23	+
	24	+interface SourceRow extends Record<string, unknown> {
	25	+ id: string;
	26	+ domain: string;
	27	+ categories: string[];
	28	+ tier: string;
	29	+ enabled: boolean;
	30	+}
	31	+
	32	+let sourceIndex: { at: number; byKey: Map<string, SourceRow>; ids: Set<string> } \| null = null;
	33	+async function sourcesByDomain(force = false): Promise<{ byKey: Map<string, SourceRow>; ids: Set<string> }> {
	34	+ if (!force && sourceIndex && Date.now() - sourceIndex.at < 5 * 60_000) return sourceIndex;
	35	+ const rows = await db.execute<SourceRow>(sql`select id, domain, categories, tier, enabled from sources where kind = 'registry'`);
	36	+ const byKey = new Map<string, SourceRow>();
	37	+ const ids = new Set<string>();
	38	+ for (const r of rows.rows) {
	39	+ ids.add(r.id);
	40	+ const k = coverageKey(r.domain);
	41	+ if (!byKey.has(k)) byKey.set(k, r);
	42	+ }
	43	+ sourceIndex = { at: Date.now(), byKey, ids };
	44	+ return sourceIndex;
	45	+}
	46	+
	47	+export async function processSeed(seed: FactorySeed): Promise<ProcessOutcome> {
	48	+ const started = Date.now();
	49	+ const notes: string[] = [];
	50	+ const hints = (seed.hints ?? {}) as DeepDiscoveryHints & { posture?: boolean };
	51	+ const idx = await sourcesByDomain();
	52	+ const key = coverageKey(seed.domain);
	53	+ let source = idx.byKey.get(key) ?? null;
	54	+
	55	+ const disc = await discoverOrganization(seed.domain, {
	56	+ hints,
	57	+ budget: factoryConfig.budget,
	58	+ concurrency: 4,
	59	+ sensorIdForLogs: `factory_${seed.id}`,
	60	+ probePages: true,
	61	+ probeSubdomains: true,
	62	+ probePosture: seed.importance >= 3 \|\| hints.posture === true,
	63	+ probeOpenApi: true,
	64	+ });
	65	+ notes.push(...disc.notes);
	66	+ // The homepage may redirect to another registrable domain that we already monitor.
	67	+ if (!source && disc.canonicalDomain !== key) source = idx.byKey.get(disc.canonicalDomain) ?? null;
	68	+
	69	+ if (!disc.candidates.length) {
	70	+ const status = disc.blocked ? "blocked" : "discovered";
	71	+ await db.execute(sql`update factory_seeds set status = ${status}, candidates = 0, shadow = 0, source_id = ${source?.id ?? null}, discovered_at = now(), last_error = ${disc.blocked ? `homepage ${disc.homeStatus} (anti-bot)` : null}, updated_at = now() where id = ${seed.id}`);
	72	+ await bumpFactoryDaily({ seeds_processed: 1, requests: disc.requests, blocked: disc.blocked ? 1 : 0 });
	73	+ return { seedId: seed.id, sourceId: source?.id ?? null, status, candidates: 0, shadow: 0, requests: disc.requests, durationMs: Date.now() - started, notes };
	74	+ }
	75	+
	76	+ // ---- source: attach or create --------------------------------------------------------------------------
	77	+ let sourceId: string;
	78	+ if (source) {
	79	+ sourceId = source.id;
	80	+ if (!source.enabled) notes.push("source disabled in registry — candidates recorded, no shadow sensors");
	81	+ } else {
	82	+ sourceId = idx.ids.has(seed.id) ? `${seed.id}-${slugify(seed.domain.split(".")[0] ?? "x")}`.slice(0, 70) : seed.id;
	83	+ const seedDoc: SourceSeed = sourceSchema.parse({
	84	+ id: sourceId,
	85	+ name: seed.name,
	86	+ domain: seed.domain,
	87	+ homepage: seed.homepage ?? `https://${seed.domain}`,
	88	+ categories: seed.categories,
	89	+ tier: (seed.tier as Tier) ?? "B",
	90	+ weight: seed.weight,
	91	+ aliases: seed.aliases,
	92	+ country: seed.country ?? undefined,
	93	+ language: seed.language ?? undefined,
	94	+ first_party: seed.firstParty,
	95	+ notes: `Source Factory · ${seed.sector ?? "-"}/${seed.universe ?? "-"} · discovered ${new Date().toISOString().slice(0, 10)}`,
	96	+ llm: true,
	97	+ });
	98	+ await upsertSourceRecord(seedDoc, { origin: "factory", sector: seed.sector });
	99	+ idx.ids.add(sourceId);
	100	+ idx.byKey.set(key, { id: sourceId, domain: seed.domain, categories: seed.categories, tier: seed.tier, enabled: true });
	101	+ source = idx.byKey.get(key)!;
	102	+ }
	103	+
	104	+ // ---- existing sensors (this source + global URL uniqueness) ----------------------------------------------
	105	+ const existing = await db.execute<{ id: string; url: string; type: string; connector: string; config: Record<string, unknown>; status: string }>(sql`select id, url, type, connector, config, status from sensors where source_id = ${sourceId}`);
	106	+ const existingUrls = new Set(existing.rows.map((r) => r.url.replace(/\/$/, "")));
	107	+ const existingKinds = new Map<CandidateKind, number>();
	108	+ let existingTotal = 0;
	109	+ for (const r of existing.rows) {
	110	+ if (r.status === "DISABLED") continue;
	111	+ existingTotal++;
	112	+ const k = kindOfExisting(r);
	113	+ existingKinds.set(k, (existingKinds.get(k) ?? 0) + 1);
	114	+ }
	115	+ const urls = disc.candidates.map((c) => c.url);
	116	+ const global = urls.length ? await db.execute<{ url: string; source_id: string }>(sql`select url, source_id from sensors where url = any(${sql.raw("array[" + urls.map((u) => "'" + u.replace(/'/g, "''") + "'").join(",") + "]::text[]")}) and source_id <> ${sourceId}`) : { rows: [] as { url: string; source_id: string }[] };
	117	+ const ownedElsewhere = new Map(global.rows.map((r) => [r.url.replace(/\/$/, ""), r.source_id]));
	118	+
	119	+ // ---- score + select ---------------------------------------------------------------------------------------
	120	+ const scored = disc.candidates.map((c) => scoreCandidate(c, { importance: seed.importance, weight: seed.weight }));
	121	+ const fresh = scored.filter((s) => !existingUrls.has(s.cand.url.replace(/\/$/, "")) && !ownedElsewhere.has(s.cand.url.replace(/\/$/, "")));
	122	+ const { selected, skipped } = source.enabled ? selectForShadow(fresh, { kinds: existingKinds, total: existingTotal }) : { selected: [], skipped: fresh.map((s) => ({ s, reason: "source disabled" })) };
	123	+
	124	+ // ---- persist candidates ------------------------------------------------------------------------------------
	125	+ const day = new Date().toISOString().slice(0, 10);
	126	+ let shadowCreated = 0;
	127	+ for (const s of scored) {
	128	+ const k = s.cand.url.replace(/\/$/, "");
	129	+ let status = "candidate";
	130	+ let reason: string \| null = null;
	131	+ if (existingUrls.has(k)) {
	132	+ status = "duplicate";
	133	+ reason = "already a sensor of this source";
	134	+ } else if (ownedElsewhere.has(k)) {
	135	+ status = "duplicate";
	136	+ reason = `already a sensor of ${ownedElsewhere.get(k)}`;
	137	+ } else if (selected.includes(s)) status = "shadow";
	138	+ else {
	139	+ const sk = skipped.find((x) => x.s === s);
	140	+ status = sk && /score/.test(sk.reason) ? "rejected" : "candidate";
	141	+ reason = sk?.reason ?? null;
	142	+ }
	143	+ const candId = newId("cand");
	144	+ let shadowSensorId: string \| null = null;
	145	+ if (status === "shadow") {
	146	+ shadowSensorId = await createShadowSensor(sourceId, source, s.cand, candId, seed);
	147	+ if (shadowSensorId) shadowCreated++;
	148	+ else {
	149	+ status = "duplicate";
	150	+ reason = "sensor id/url collision";
	151	+ }
	152	+ }
	153	+ await db.execute(sql`
	154	+ insert into discovery_candidates (id, source_id, url, kind, evidence, score, status, seed_id, connector, type, name, kind_class, config, tier, score_value, reason, shadow_sensor_id, found_at, updated_at)
	155	+ values (${candId}, ${sourceId}, ${s.cand.url}, ${s.cand.type}, ${s.cand.evidence}, ${JSON.stringify({ value: s.cand.value, score: s.score, reasons: s.reasons, itemCount: s.cand.itemCount ?? null, itemsPerDay: s.cand.itemsPerDay ?? null, firstParty: s.cand.firstPartyConfidence, cost: s.cand.fetchCost, title: s.cand.title ?? null, fingerprint: s.cand.fingerprint ?? null })}::jsonb, ${status}, ${seed.id}, ${s.cand.connector}, ${s.cand.type}, ${s.cand.name}, ${s.cand.kind}, ${JSON.stringify(s.cand.config)}::jsonb, ${s.cand.suggestedTier}, ${s.score}, ${reason}, ${shadowSensorId}, now(), now())
	156	+ on conflict (source_id, url) do update set evidence = excluded.evidence, score = excluded.score, score_value = excluded.score_value, seed_id = excluded.seed_id, connector = excluded.connector, type = excluded.type, name = excluded.name, kind_class = excluded.kind_class, config = excluded.config, tier = excluded.tier, updated_at = now(),
	157	+ status = case when discovery_candidates.status in ('accepted','rejected','shadow') then discovery_candidates.status else excluded.status end,
	158	+ reason = case when discovery_candidates.status in ('accepted','rejected','shadow') then discovery_candidates.reason else excluded.reason end,
	159	+ shadow_sensor_id = coalesce(discovery_candidates.shadow_sensor_id, excluded.shadow_sensor_id)`);
	160	+ }
	161	+ for (const r of disc.rejected.slice(0, 40)) {
	162	+ await db.execute(sql`insert into discovery_candidates (id, source_id, url, kind, evidence, status, seed_id, reason, kind_class) values (${newId("cand")}, ${sourceId}, ${r.url}, 'HTML', ${r.reason}, 'rejected', ${seed.id}, ${r.reason}, 'other') on conflict (source_id, url) do nothing`);
	163	+ }
	164	+
	165	+ await db.execute(sql`update factory_seeds set status = 'discovered', source_id = ${sourceId}, candidates = ${scored.length}, shadow = shadow + ${shadowCreated}, discovered_at = now(), last_error = null, updated_at = now() where id = ${seed.id}`);
	166	+ await db.execute(sql`update sources set robots_checked_at = now(), updated_at = now() where id = ${sourceId}`);
	167	+ await bumpFactoryDaily({ seeds_processed: 1, requests: disc.requests, candidates: scored.length, shadow_created: shadowCreated, blocked: disc.blocked ? 1 : 0 });
	168	+ log.info({ seed: seed.id, source: sourceId, candidates: scored.length, shadow: shadowCreated, requests: disc.requests, ms: Date.now() - started, day }, "factory: organization discovered");
	169	+ return { seedId: seed.id, sourceId, status: "discovered", candidates: scored.length, shadow: shadowCreated, requests: disc.requests, durationMs: Date.now() - started, notes };
	170	+}
	171	+
	172	+function kindOfExisting(r: { url: string; type: string; connector: string; config: Record<string, unknown> }): CandidateKind {
	173	+ const k = (r.config as { kind?: string }).kind as CandidateKind \| undefined;
	174	+ if (k) return k;
	175	+ if (r.connector === "statuspage" \|\| r.connector === "statusjson") return "status";
	176	+ if (r.connector === "sitemap") return "sitemap";
	177	+ if (r.connector === "github") return "releases";
	178	+ if (r.connector === "edgar") return "filings";
	179	+ if (r.connector === "dns" \|\| r.connector === "tls" \|\| r.connector === "headers" \|\| r.connector === "rdap") return "posture";
	180	+ if (r.connector === "openapi") return "api";
	181	+ if (r.connector === "rss") return /press/i.test(r.url) ? "press" : /blog/i.test(r.url) ? "blog" : /security\|advisor/i.test(r.url) ? "security" : /changelog\|release/i.test(r.url) ? "changelog" : "news";
	182	+ if (/pricing\|plans/i.test(r.url)) return "pricing";
	183	+ if (/terms\|privacy\|legal/i.test(r.url)) return "legal";
	184	+ if (/career\|jobs/i.test(r.url)) return "careers";
	185	+ if (/changelog\|release-notes\|whats-new/i.test(r.url)) return "changelog";
	186	+ if (/security\|advisor/i.test(r.url)) return "security";
	187	+ return "other";
	188	+}
	189	+
	190	+async function createShadowSensor(sourceId: string, source: SourceRow, c: { url: string; type: string; connector: string; name: string; config: Record<string, unknown>; suggestedTier: Tier; kind: CandidateKind }, candId: string, seed: FactorySeed): Promise<string \| null> {
	191	+ let id = `${sourceId}_${slugify(c.name)}`.slice(0, 80);
	192	+ const clash = (await db.execute<{ url: string }>(sql`select url from sensors where id = ${id}`)).rows[0];
	193	+ if (clash && clash.url.replace(/\/$/, "") !== c.url.replace(/\/$/, "")) id = `${id}-${newId("x").slice(-5)}`;
	194	+ const tier = c.suggestedTier;
	195	+ const priority = 3; // shadows never compete with production sensors
	196	+ const jitter = Math.floor(Math.random() * 30 * 60e3);
	197	+ try {
	198	+ await db
	199	+ .insert(sensors)
	200	+ .values({ id, sourceId, name: c.name, url: c.url, type: c.type, connector: c.connector, tier, importanceWeight: 1, config: { ...c.config, factory: true, shadow: true, seed: false, kind: c.kind, candidate: candId, seedId: seed.id, shadowSince: new Date().toISOString(), targetPriority: priorityFor(tier, source.categories) }, status: "SHADOW", priority, validatedAt: new Date(), nextCheckAt: new Date(Date.now() + jitter) })
	201	+ .onConflictDoNothing();
	202	+ } catch (e) {
	203	+ log.warn({ sensor: id, err: (e as Error).message }, "factory: shadow sensor insert failed");
	204	+ return null;
	205	+ }
	206	+ return id;
	207	+}
	208	+
	209	+export async function bumpFactoryDaily(delta: Partial<Record<"seeds_processed" \| "requests" \| "candidates" \| "shadow_created" \| "accepted" \| "rejected" \| "blocked", number>>): Promise<void> {
	210	+ const day = new Date().toISOString().slice(0, 10);
	211	+ const d = { seeds_processed: 0, requests: 0, candidates: 0, shadow_created: 0, accepted: 0, rejected: 0, blocked: 0, ...delta };
	212	+ await db.execute(sql`insert into factory_daily (day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked) values (${day}, ${d.seeds_processed}, ${d.requests}, ${d.candidates}, ${d.shadow_created}, ${d.accepted}, ${d.rejected}, ${d.blocked})
	213	+ on conflict (day) do update set seeds_processed = factory_daily.seeds_processed + ${d.seeds_processed}, requests = factory_daily.requests + ${d.requests}, candidates = factory_daily.candidates + ${d.candidates}, shadow_created = factory_daily.shadow_created + ${d.shadow_created}, accepted = factory_daily.accepted + ${d.accepted}, rejected = factory_daily.rejected + ${d.rejected}, blocked = factory_daily.blocked + ${d.blocked}`).catch(() => undefined);
	214	+}
	215	+
	216	+/** Claim up to `n` queued seeds (systemic organizations first) and process them with bounded concurrency. */
	217	+export async function runFactoryBatch(n = factoryConfig.concurrency, opts: { seedIds?: string[] } = {}): Promise<ProcessOutcome[]> {
	218	+ const claimed = await db.execute<Record<string, unknown>>(sql`
	219	+ update factory_seeds set status = 'discovering', attempts = attempts + 1, updated_at = now()
	220	+ where id in (
	221	+ select id from factory_seeds
	222	+ where ${opts.seedIds ? sql`id = any(${sql.raw("array[" + opts.seedIds.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]")})` : sql`(status = 'queued' or (status = 'error' and attempts < 3 and updated_at < now() - interval '6 hours') or (status = 'discovering' and updated_at < now() - interval '2 hours'))`}
	223	+ order by importance desc, created_at asc limit ${n} for update skip locked)
	224	+ returning *`);
	225	+ const seeds = claimed.rows.map(normalizeSeed);
	226	+ const out: ProcessOutcome[] = [];
	227	+ let i = 0;
	228	+ await Promise.all(
	229	+ Array.from({ length: Math.min(factoryConfig.concurrency, seeds.length) }, async () => {
	230	+ while (i < seeds.length) {
	231	+ const seed = seeds[i++]!;
	232	+ try {
	233	+ out.push(await processSeed(seed));
	234	+ } catch (e) {
	235	+ log.error({ seed: seed.id, err: (e as Error).stack ?? (e as Error).message }, "factory: seed failed");
	236	+ await db.execute(sql`update factory_seeds set status = 'error', last_error = ${(e as Error).message.slice(0, 500)}, updated_at = now() where id = ${seed.id}`).catch(() => undefined);
	237	+ out.push({ seedId: seed.id, sourceId: null, status: "error", candidates: 0, shadow: 0, requests: 0, durationMs: 0, notes: [(e as Error).message] });
	238	+ }
	239	+ }
	240	+ }),
	241	+ );
	242	+ return out;
	243	+}
	244	+
	245	+function normalizeSeed(r: Record<string, unknown>): FactorySeed {
	246	+ const d = (v: unknown): Date \| null => (v ? new Date(v as string) : null);
	247	+ return {
	248	+ id: r.id as string,
	249	+ name: r.name as string,
	250	+ domain: r.domain as string,
	251	+ homepage: (r.homepage as string \| null) ?? null,
	252	+ categories: (r.categories as string[]) ?? [],
	253	+ country: (r.country as string \| null) ?? null,
	254	+ language: (r.language as string \| null) ?? null,
	255	+ tier: (r.tier as string) ?? "B",
	256	+ weight: Number(r.weight ?? 1),
	257	+ importance: Number(r.importance ?? 2),
	258	+ aliases: (r.aliases as string[]) ?? [],
	259	+ firstParty: Boolean(r.first_party ?? true),
	260	+ sector: (r.sector as string \| null) ?? null,
	261	+ universe: (r.universe as string \| null) ?? null,
	262	+ hints: (r.hints as Record<string, unknown>) ?? {},
	263	+ status: r.status as string,
	264	+ attempts: Number(r.attempts ?? 0),
	265	+ sourceId: (r.source_id as string \| null) ?? null,
	266	+ candidates: Number(r.candidates ?? 0),
	267	+ shadow: Number(r.shadow ?? 0),
	268	+ accepted: Number(r.accepted ?? 0),
	269	+ rejected: Number(r.rejected ?? 0),
	270	+ lastError: (r.last_error as string \| null) ?? null,
	271	+ discoveredAt: d(r.discovered_at),
	272	+ createdAt: d(r.created_at) ?? new Date(),
	273	+ updatedAt: d(r.updated_at) ?? new Date(),
	274	+ };
	275	+}
	276	+
	277	+export async function factoryStats(): Promise<Record<string, unknown>> {
	278	+ const [seeds, cands, shadows, daily, sectors] = await Promise.all([
	279	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from factory_seeds group by status`).then((r) => r.rows),
	280	+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from discovery_candidates group by status`).then((r) => r.rows),
	281	+ db.execute<Record<string, unknown>>(sql`select count()::int as shadow, count() filter (where total_runs >= ${factoryConfig.shadowMinChecks})::int as ready, count(*) filter (where health <> 'UP')::int as unhealthy from sensors where status = 'SHADOW'`).then((r) => r.rows[0]),
	282	+ db.execute<Record<string, unknown>>(sql`select day::text as day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked from factory_daily order by day desc limit 30`).then((r) => r.rows),
	283	+ db.execute<Record<string, unknown>>(sql`select coalesce(sector, '-') as sector, count()::int as seeds, count() filter (where status = 'queued')::int as queued, count() filter (where status in ('discovered','done'))::int as discovered, count() filter (where status = 'blocked')::int as blocked, sum(shadow)::int as shadow, sum(accepted)::int as accepted, sum(rejected)::int as rejected from factory_seeds group by 1 order by seeds desc`).then((r) => r.rows),
	284	+ ]);
	285	+ return { seeds: Object.fromEntries(seeds.map((r) => [r.status, r.n])), candidates: Object.fromEntries(cands.map((r) => [r.status, r.n])), shadow: shadows, daily, sectors, config: { concurrency: factoryConfig.concurrency, budget: factoryConfig.budget, minScore: factoryConfig.minScore, shadowMinChecks: factoryConfig.shadowMinChecks, shadowMinHours: factoryConfig.shadowMinHours, shadowMaxHours: factoryConfig.shadowMaxHours, caps: factoryConfig.caps } };
	286	+}
	287	+
	288	+export { factorySeeds };

added apps/engine/src/factory/score.ts +114 −0

		`@@ -0,0 +1,114 @@`
	1	+import type { CandidateKind, DeepCandidate } from "@websensor/connectors";
	2	+import { factoryConfig } from "../config";
	3	+
	4	+/**
	5	+ * Candidate scoring (Source Factory, stage "signal/noise evaluation before shadow"). Combines the discovery
	6	+ * value (evidence quality) with the organization's importance, the class of page, first-party confidence,
	7	+ * expected change frequency and fetch cost. Explainable: every score carries its reasons.
	8	+ */
	9	+export interface ScoredCandidate {
	10	+ cand: DeepCandidate;
	11	+ score: number;
	12	+ reasons: string[];
	13	+}
	14	+
	15	+/** How much a page class is worth for change intelligence (1 = neutral). */
	16	+export const KIND_WEIGHT: Record<CandidateKind, number> = {
	17	+ status: 1.15,
	18	+ security: 1.15,
	19	+ press: 1.1,
	20	+ news: 1.05,
	21	+ filings: 1.1,
	22	+ changelog: 1.05,
	23	+ releases: 1.0,
	24	+ pricing: 1.05,
	25	+ models: 1.05,
	26	+ ir: 0.95,
	27	+ legal: 0.9,
	28	+ blog: 0.9,
	29	+ api: 0.9,
	30	+ leadership: 0.85,
	31	+ data: 0.85,
	32	+ sitemap: 0.8,
	33	+ docs: 0.7,
	34	+ careers: 0.6,
	35	+ posture: 0.6,
	36	+ other: 0.7,
	37	+};
	38	+
	39	+export function scoreCandidate(c: DeepCandidate, seed: { importance: number; weight?: number }): ScoredCandidate {
	40	+ const reasons: string[] = [];
	41	+ let score = c.value;
	42	+ reasons.push(`evidence ${c.value.toFixed(2)} (${c.evidence})`);
	43	+ const kw = KIND_WEIGHT[c.kind] ?? 0.7;
	44	+ score *= kw;
	45	+ if (kw !== 1) reasons.push(`${c.kind} ×${kw.toFixed(2)}`);
	46	+ const imp = seed.importance === 3 ? 1.1 : seed.importance === 1 ? 0.85 : 1;
	47	+ score = imp (seed.weight ?? 1);
	48	+ if (imp !== 1) reasons.push(`organization importance ${seed.importance} ×${imp.toFixed(2)}`);
	49	+ score *= c.firstPartyConfidence;
	50	+ if (c.firstPartyConfidence < 1) reasons.push(`first-party confidence ${c.firstPartyConfidence}`);
	51	+ if (c.itemsPerDay !== null && c.itemsPerDay !== undefined) {
	52	+ if (c.itemsPerDay > 60) {
	53	+ score *= 0.85;
	54	+ reasons.push(`firehose ${c.itemsPerDay}/day ×0.85`);
	55	+ } else if (c.itemsPerDay < 0.02) {
	56	+ score *= 0.8;
	57	+ reasons.push(`nearly dormant ${c.itemsPerDay}/day ×0.80`);
	58	+ } else if (c.itemsPerDay >= 0.2) {
	59	+ score *= 1.05;
	60	+ reasons.push(`active ${c.itemsPerDay}/day ×1.05`);
	61	+ }
	62	+ }
	63	+ if (c.fetchCost.ms > 8000) {
	64	+ score -= 0.08;
	65	+ reasons.push(`slow (${Math.round(c.fetchCost.ms)} ms) −0.08`);
	66	+ }
	67	+ if (c.fetchCost.bytes > 5 * 1024 * 1024) {
	68	+ score -= 0.08;
	69	+ reasons.push(`heavy (${Math.round(c.fetchCost.bytes / 1048576)} MB) −0.08`);
	70	+ }
	71	+ score = Math.round(Math.max(0, Math.min(1.2, score)) * 1000) / 1000;
	72	+ return { cand: c, score, reasons };
	73	+}
	74	+
	75	+/** Promotion policy: caps per organization and per class, best score first. Returns the selected candidates and the reason for the others. */
	76	+export function selectForShadow(scored: ScoredCandidate[], existing: { kinds: Map<CandidateKind, number>; total: number }): { selected: ScoredCandidate[]; skipped: { s: ScoredCandidate; reason: string }[] } {
	77	+ const caps = factoryConfig.caps;
	78	+ const groupOf = (k: CandidateKind): keyof typeof caps => (k === "news" \|\| k === "press" \|\| k === "blog" \|\| k === "ir" \|\| k === "changelog" \|\| k === "security" ? "feeds" : k === "pricing" \|\| k === "legal" \|\| k === "careers" \|\| k === "leadership" \|\| k === "docs" \|\| k === "data" \|\| k === "other" ? "pages" : (k as keyof typeof caps));
	79	+ const used = new Map<string, number>();
	80	+ for (const [k, n] of existing.kinds) used.set(groupOf(k), (used.get(groupOf(k)) ?? 0) + n);
	81	+ let total = existing.total;
	82	+ const selected: ScoredCandidate[] = [];
	83	+ const skipped: { s: ScoredCandidate; reason: string }[] = [];
	84	+ const legal = (existing.kinds.get("legal") ?? 0);
	85	+ let legalUsed = legal;
	86	+ for (const s of [...scored].sort((a, b) => b.score - a.score)) {
	87	+ if (s.score < factoryConfig.minScore) {
	88	+ skipped.push({ s, reason: `score ${s.score} < ${factoryConfig.minScore}` });
	89	+ continue;
	90	+ }
	91	+ if (total >= caps.total) {
	92	+ skipped.push({ s, reason: `organization cap ${caps.total} reached` });
	93	+ continue;
	94	+ }
	95	+ // Feed-type candidates (rss connector) and HTML pages are capped by group; connectors with their own cap by kind.
	96	+ const g = s.cand.connector === "rss" ? "feeds" : s.cand.connector === "http" && s.cand.kind !== "posture" ? "pages" : groupOf(s.cand.kind);
	97	+ const cap = caps[g as keyof typeof caps] ?? 1;
	98	+ if ((used.get(g) ?? 0) >= cap) {
	99	+ skipped.push({ s, reason: `${g} cap ${cap} reached` });
	100	+ continue;
	101	+ }
	102	+ if (s.cand.kind === "legal") {
	103	+ if (legalUsed >= caps.legal) {
	104	+ skipped.push({ s, reason: `legal cap ${caps.legal} reached` });
	105	+ continue;
	106	+ }
	107	+ legalUsed++;
	108	+ }
	109	+ used.set(g, (used.get(g) ?? 0) + 1);
	110	+ total++;
	111	+ selected.push(s);
	112	+ }
	113	+ return { selected, skipped };
	114	+}

added apps/engine/src/factory/seeds.ts +168 −0

		`@@ -0,0 +1,168 @@`
	1	+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
	2	+import { join } from "node:path";
	3	+import YAML from "yaml";
	4	+import { coverageKey, coverageSectorSchema, slugify, type CoverageMember, type CoverageSector } from "@websensor/core";
	5	+import { db, factorySeeds, sql } from "@websensor/db";
	6	+import { factoryConfig, log } from "../config";
	7	+
	8	+/**
	9	+ * Factory seeds come from two places: the coverage universes (`config/coverage/*.yaml`, every member that is not
	10	+ * monitored yet — or that carries hints worth expanding) and ad-hoc seed files (`config/factory/seeds/*.yaml`,
	11	+ * `{ seeds: [{ name, domain, categories, country, … }] }`) or the admin API.
	12	+ */
	13	+export interface SeedInput {
	14	+ id?: string;
	15	+ name: string;
	16	+ domain: string;
	17	+ homepage?: string;
	18	+ categories?: string[];
	19	+ country?: string;
	20	+ language?: string;
	21	+ tier?: string;
	22	+ weight?: number;
	23	+ importance?: number;
	24	+ aliases?: string[];
	25	+ first_party?: boolean;
	26	+ sector?: string;
	27	+ universe?: string;
	28	+ hints?: Record<string, unknown>;
	29	+}
	30	+
	31	+export function loadCoverageSectors(dir = factoryConfig.coverageDir): { sectors: CoverageSector[]; issues: string[] } {
	32	+ const sectors: CoverageSector[] = [];
	33	+ const issues: string[] = [];
	34	+ if (!existsSync(dir) \|\| !statSync(dir).isDirectory()) return { sectors, issues: [`coverage dir ${dir} not found`] };
	35	+ for (const f of readdirSync(dir).filter((x) => /\.ya?ml$/.test(x) && !x.startsWith("_")).sort()) {
	36	+ try {
	37	+ const parsed = coverageSectorSchema.safeParse(YAML.parse(readFileSync(join(dir, f), "utf8")));
	38	+ if (!parsed.success) {
	39	+ issues.push(`${f}: ${parsed.error.issues.slice(0, 3).map((i) => `${i.path.join(".")}: ${i.message}`).join("; ")}`);
	40	+ continue;
	41	+ }
	42	+ sectors.push(parsed.data);
	43	+ } catch (e) {
	44	+ issues.push(`${f}: ${(e as Error).message}`);
	45	+ }
	46	+ }
	47	+ return { sectors, issues };
	48	+}
	49	+
	50	+/** Merge files that share a sector key. */
	51	+export function mergeSectors(sectors: CoverageSector[]): CoverageSector[] {
	52	+ const byKey = new Map<string, CoverageSector>();
	53	+ for (const s of sectors) {
	54	+ const prev = byKey.get(s.sector);
	55	+ if (!prev) byKey.set(s.sector, { ...s, universes: [...s.universes] });
	56	+ else prev.universes.push(...s.universes);
	57	+ }
	58	+ return [...byKey.values()];
	59	+}
	60	+
	61	+export function memberToSeed(sector: CoverageSector, universe: string, m: CoverageMember): SeedInput {
	62	+ return {
	63	+ name: m.name,
	64	+ domain: m.domain,
	65	+ categories: m.categories ?? sector.categories,
	66	+ country: m.country,
	67	+ language: m.language,
	68	+ tier: m.tier ?? sector.tier,
	69	+ importance: m.importance,
	70	+ aliases: m.aliases,
	71	+ first_party: m.first_party,
	72	+ sector: sector.sector,
	73	+ universe,
	74	+ hints: m.hints as Record<string, unknown>,
	75	+ };
	76	+}
	77	+
	78	+function hasHints(h: Record<string, unknown> \| undefined): boolean {
	79	+ if (!h) return false;
	80	+ return Boolean(h.cik \|\| h.github_org \|\| (Array.isArray(h.github_repos) && h.github_repos.length) \|\| h.hf_author \|\| h.status_url \|\| (Array.isArray(h.urls) && h.urls.length) \|\| (Array.isArray(h.hosts) && h.hosts.length));
	81	+}
	82	+
	83	+/** Registrable domains currently monitored (source domains + hosts of enabled non-shadow sensors). */
	84	+export async function monitoredDomainKeys(): Promise<Set<string>> {
	85	+ const rows = await db.execute<{ d: string }>(sql`
	86	+ select domain as d from sources where enabled and kind = 'registry'
	87	+ union
	88	+ select lower(split_part(split_part(url, '/', 3), ':', 1)) as d from sensors where enabled and status <> 'SHADOW' and url like 'http%'`);
	89	+ const out = new Set<string>();
	90	+ for (const r of rows.rows) if (r.d) out.add(coverageKey(r.d));
	91	+ return out;
	92	+}
	93	+
	94	+export async function upsertSeeds(inputs: SeedInput[], opts: { requeue?: boolean } = {}): Promise<{ inserted: number; updated: number; skipped: number }> {
	95	+ let inserted = 0;
	96	+ let updated = 0;
	97	+ let skipped = 0;
	98	+ const existingIds = new Set((await db.execute<{ id: string }>(sql`select id from sources`)).rows.map((r) => r.id));
	99	+ const existingSeedDomains = new Map((await db.execute<{ id: string; domain: string }>(sql`select id, domain from factory_seeds`)).rows.map((r) => [coverageKey(r.domain), r.id]));
	100	+ for (const s of inputs) {
	101	+ const domain = s.domain.toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "");
	102	+ const key = coverageKey(domain);
	103	+ let id = s.id ?? existingSeedDomains.get(key) ?? slugify(s.name).slice(0, 60);
	104	+ if (!id) id = slugify(domain);
	105	+ // A registry source with the same id but another organization → disambiguate by domain.
	106	+ if (!s.id && !existingSeedDomains.has(key) && existingIds.has(id)) {
	107	+ const src = (await db.execute<{ domain: string }>(sql`select domain from sources where id = ${id}`)).rows[0];
	108	+ if (src && coverageKey(src.domain) !== key) id = `${id}-${slugify(domain.split(".")[0] ?? domain)}`.slice(0, 70);
	109	+ }
	110	+ const r = await db.execute<{ inserted: boolean }>(sql`
	111	+ insert into factory_seeds (id, name, domain, homepage, categories, country, language, tier, weight, importance, aliases, first_party, sector, universe, hints, status)
	112	+ values (${id}, ${s.name}, ${domain}, ${s.homepage ?? null}, ${sql.raw("'{" + (s.categories ?? []).map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.country ?? null}, ${s.language ?? null}, ${s.tier ?? "B"}, ${s.weight ?? 1}, ${s.importance ?? 2}, ${sql.raw("'{" + (s.aliases ?? []).map((c) => '"' + c.replace(/["\\]/g, "") + '"').join(",") + "}'::text[]")}, ${s.first_party ?? true}, ${s.sector ?? null}, ${s.universe ?? null}, ${JSON.stringify(s.hints ?? {})}::jsonb, 'queued')
	113	+ on conflict (id) do update set name = excluded.name, domain = excluded.domain, categories = excluded.categories, country = coalesce(excluded.country, factory_seeds.country), language = coalesce(excluded.language, factory_seeds.language), tier = excluded.tier, importance = excluded.importance, aliases = excluded.aliases, sector = coalesce(excluded.sector, factory_seeds.sector), universe = coalesce(excluded.universe, factory_seeds.universe), hints = factory_seeds.hints \|\| excluded.hints, updated_at = now()${opts.requeue ? sql`, status = 'queued'` : sql``}
	114	+ returning (xmax = 0) as inserted`);
	115	+ if (r.rows[0]?.inserted) inserted++;
	116	+ else if (r.rowCount) updated++;
	117	+ else skipped++;
	118	+ existingSeedDomains.set(key, id);
	119	+ }
	120	+ log.info({ inserted, updated, skipped }, "factory seeds upserted");
	121	+ return { inserted, updated, skipped };
	122	+}
	123	+
	124	+/**
	125	+ * Turn coverage universes into seeds. `mode`: `uncovered` = members without any monitored domain; `hinted` = also
	126	+ * covered members that carry expansion hints; `all` = every member (re-discover everything).
	127	+ */
	128	+export async function seedFromCoverage(opts: { mode?: "uncovered" \| "hinted" \| "all"; sectors?: string[]; requeue?: boolean } = {}): Promise<{ members: number; seeds: number; inserted: number; updated: number; issues: string[] }> {
	129	+ const mode = opts.mode ?? "hinted";
	130	+ const { sectors, issues } = loadCoverageSectors();
	131	+ const monitored = await monitoredDomainKeys();
	132	+ const inputs: SeedInput[] = [];
	133	+ const seen = new Set<string>();
	134	+ let members = 0;
	135	+ for (const sector of mergeSectors(sectors)) {
	136	+ if (opts.sectors && !opts.sectors.includes(sector.sector)) continue;
	137	+ for (const u of sector.universes) {
	138	+ for (const m of u.members) {
	139	+ members++;
	140	+ const key = coverageKey(m.domain);
	141	+ if (seen.has(key)) continue;
	142	+ const covered = monitored.has(key);
	143	+ if (mode === "uncovered" && covered) continue;
	144	+ if (mode === "hinted" && covered && !hasHints(m.hints as Record<string, unknown>)) continue;
	145	+ seen.add(key);
	146	+ inputs.push(memberToSeed(sector, u.key, m));
	147	+ }
	148	+ }
	149	+ }
	150	+ const r = await upsertSeeds(inputs, { requeue: opts.requeue });
	151	+ return { members, seeds: inputs.length, inserted: r.inserted, updated: r.updated, issues };
	152	+}
	153	+
	154	+/** `config/factory/seeds/.yaml` → `{ seeds: [...] }` /
	155	+export async function seedFromFiles(dir = factoryConfig.seedsDir): Promise<{ files: number; seeds: number }> {
	156	+ if (!existsSync(dir)) return { files: 0, seeds: 0 };
	157	+ const inputs: SeedInput[] = [];
	158	+ let files = 0;
	159	+ for (const f of readdirSync(dir).filter((x) => /\.ya?ml$/.test(x)).sort()) {
	160	+ files++;
	161	+ const raw = YAML.parse(readFileSync(join(dir, f), "utf8")) as { seeds?: SeedInput[] };
	162	+ for (const s of raw.seeds ?? []) if (s?.name && s?.domain) inputs.push(s);
	163	+ }
	164	+ await upsertSeeds(inputs);
	165	+ return { files, seeds: inputs.length };
	166	+}
	167	+
	168	+export { factorySeeds };

added apps/engine/src/factory/shadow.ts +109 −0

		`@@ -0,0 +1,109 @@`
	1	+import { NOISE_CLASSES } from "@websensor/core";
	2	+import { db, sql } from "@websensor/db";
	3	+import { factoryConfig, log } from "../config";
	4	+import { bumpFactoryDaily } from "./run";
	5	+
	6	+/**
	7	+ * Shadow evaluation — the Factory's "signal/noise evaluation" stage. A shadow sensor is polled like any other
	8	+ * sensor (evidence is stored) but never publishes. Once it has enough checks (or enough time has passed) it is:
	9	+ * rejected — errors (≥ 50 % failed checks, or 3+ consecutive errors), pure noise (changes on most checks,
	10	+ * none of them meaningful), or a duplicate (same latest canonical content as an active sensor of
	11	+ * the same source);
	12	+ * accepted — otherwise, becoming a production sensor with its real priority;
	13	+ * deferred — not enough evidence yet (until `shadowMaxHours`, then accepted if it never failed).
	14	+ */
	15	+export interface ShadowDecision {
	16	+ sensorId: string;
	17	+ decision: "accepted" \| "rejected" \| "deferred";
	18	+ reason: string;
	19	+}
	20	+
	21	+interface ShadowRow extends Record<string, unknown> {
	22	+ id: string;
	23	+ source_id: string;
	24	+ url: string;
	25	+ tier: string;
	26	+ config: Record<string, unknown>;
	27	+ total_runs: number;
	28	+ consecutive_errors: number;
	29	+ raw_changes: number;
	30	+ health: string;
	31	+ last_snapshot_id: string \| null;
	32	+ shadow_since: string \| null;
	33	+ categories: string[];
	34	+ runs: number;
	35	+ failed: number;
	36	+ changes: number;
	37	+ noise_changes: number;
	38	+ last_canonical: string \| null;
	39	+ dup_sensor: string \| null;
	40	+}
	41	+
	42	+export async function evaluateShadows(opts: { limit?: number; force?: boolean } = {}): Promise<{ accepted: number; rejected: number; deferred: number; decisions: ShadowDecision[] }> {
	43	+ const noise = [...NOISE_CLASSES];
	44	+ const rows = await db.execute<ShadowRow>(sql`
	45	+ select s.id, s.source_id, s.url, s.tier, s.config, s.total_runs, s.consecutive_errors, s.raw_changes, s.health, s.last_snapshot_id,
	46	+ s.config->>'shadowSince' as shadow_since, so.categories,
	47	+ (select count(*)::int from sensor_runs r where r.sensor_id = s.id) as runs,
	48	+ (select count(*)::int from sensor_runs r where r.sensor_id = s.id and r.outcome in ('error','parse_error','rate_limited','missing')) as failed,
	49	+ (select count(*)::int from changes c where c.sensor_id = s.id) as changes,
	50	+ (select count(*)::int from changes c where c.sensor_id = s.id and c.change_class = any(${sql.raw("array[" + noise.map((n) => "'" + n + "'").join(",") + "]::text[]")})) as noise_changes,
	51	+ (select sn.canonical_hash from snapshots sn where sn.id = s.last_snapshot_id) as last_canonical,
	52	+ (select x.id from sensors x join snapshots sx on sx.id = x.last_snapshot_id where x.source_id = s.source_id and x.id <> s.id and x.status <> 'SHADOW' and x.enabled and sx.canonical_hash = (select sn2.canonical_hash from snapshots sn2 where sn2.id = s.last_snapshot_id) limit 1) as dup_sensor
	53	+ from sensors s join sources so on so.id = s.source_id
	54	+ where s.status = 'SHADOW' and s.enabled
	55	+ order by s.total_runs desc
	56	+ limit ${opts.limit ?? 2000}`);
	57	+ const decisions: ShadowDecision[] = [];
	58	+ let accepted = 0;
	59	+ let rejected = 0;
	60	+ let deferred = 0;
	61	+ const now = Date.now();
	62	+ for (const r of rows.rows) {
	63	+ const since = r.shadow_since ? new Date(r.shadow_since).getTime() : now;
	64	+ const hours = (now - since) / 3600e3;
	65	+ const runs = Math.max(r.runs, r.total_runs);
	66	+ let decision: ShadowDecision["decision"] = "deferred";
	67	+ let reason = `${runs} checks · ${hours.toFixed(0)} h`;
	68	+ if (r.dup_sensor && runs >= 2) {
	69	+ decision = "rejected";
	70	+ reason = `duplicate content of ${r.dup_sensor}`;
	71	+ } else if (r.consecutive_errors >= 3 \|\| (runs >= 4 && r.failed / runs >= 0.5)) {
	72	+ decision = "rejected";
	73	+ reason = `errors ${r.failed}/${runs} (consecutive ${r.consecutive_errors})`;
	74	+ } else if (r.changes >= 4 && r.noise_changes === r.changes && r.changes >= runs * 0.6) {
	75	+ decision = "rejected";
	76	+ reason = `noise: ${r.changes} changes on ${runs} checks, all cosmetic/timestamp/ads`;
	77	+ } else if (opts.force \|\| (runs >= factoryConfig.shadowMinChecks && hours >= factoryConfig.shadowMinHours) \|\| (hours >= factoryConfig.shadowMaxHours && runs >= 2 && r.failed === 0)) {
	78	+ if (r.health !== "UP" && !opts.force) {
	79	+ decision = "deferred";
	80	+ reason = `health ${r.health} — waiting`;
	81	+ } else {
	82	+ decision = "accepted";
	83	+ reason = `${runs} checks · ${r.changes} changes (${r.changes - r.noise_changes} meaningful) · ${r.failed} errors · ${hours.toFixed(0)} h`;
	84	+ }
	85	+ } else if (hours >= factoryConfig.shadowMaxHours && runs < 2) {
	86	+ decision = "rejected";
	87	+ reason = `never observed successfully in ${hours.toFixed(0)} h`;
	88	+ }
	89	+
	90	+ if (decision === "accepted") {
	91	+ const targetPriority = Number((r.config as { targetPriority?: number }).targetPriority ?? 2);
	92	+ await db.execute(sql`update sensors set status = 'ACTIVE', priority = ${targetPriority}, config = (config - 'shadow') \|\| jsonb_build_object('acceptedAt', ${new Date().toISOString()}::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${r.id}`);
	93	+ await db.execute(sql`update discovery_candidates set status = 'accepted', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${r.id}`);
	94	+ await db.execute(sql`update factory_seeds set accepted = accepted + 1, updated_at = now() where id = ${(r.config as { seedId?: string }).seedId ?? ""}`);
	95	+ accepted++;
	96	+ } else if (decision === "rejected") {
	97	+ await db.execute(sql`update sensors set status = 'DISABLED', enabled = false, config = config \|\| jsonb_build_object('rejectedAt', ${new Date().toISOString()}::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${r.id}`);
	98	+ await db.execute(sql`update discovery_candidates set status = 'rejected', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${r.id}`);
	99	+ await db.execute(sql`update factory_seeds set rejected = rejected + 1, updated_at = now() where id = ${(r.config as { seedId?: string }).seedId ?? ""}`);
	100	+ rejected++;
	101	+ } else deferred++;
	102	+ decisions.push({ sensorId: r.id, decision, reason });
	103	+ }
	104	+ if (accepted \|\| rejected) {
	105	+ await bumpFactoryDaily({ accepted, rejected });
	106	+ log.info({ accepted, rejected, deferred }, "factory: shadow evaluation");
	107	+ }
	108	+ return { accepted, rejected, deferred, decisions };
	109	+}

modified apps/engine/src/pipeline.ts +9 −2

		`@@ -54,7 +54,8 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc`
54	54	const isErr = outcome === "error" \|\| outcome === "parse_error" \|\| outcome === "rate_limited";
55	55	const consecutive = isErr ? sensor.consecutiveErrors + 1 : 0;
56	56	const health = outcome === "rate_limited" ? "RATE_LIMITED" : isErr ? (consecutive >= 5 ? "ERROR" : "DEGRADED") : outcome === "missing" ? "DEGRADED" : "UP";
57		− const status = !sensor.enabled ? "DISABLED" : health === "ERROR" \|\| health === "RATE_LIMITED" ? "DEGRADED" : "ACTIVE";
	57	+ // Shadow sensors (Source Factory) keep their lifecycle status until evaluateShadows() decides.
	58	+ const status = !sensor.enabled ? "DISABLED" : sensor.status === "SHADOW" ? "SHADOW" : health === "ERROR" \|\| health === "RATE_LIMITED" ? "DEGRADED" : "ACTIVE";
58	59	const changes7d = await countChanges7d(sensor.id);
59	60	const events7d = await countEvents7d(sensor.id);
60	61	let interval = nextIntervalSeconds({ tier: sensor.tier as Tier, baseIntervalSeconds: sensor.baseIntervalSeconds, lastChangeAt: extra.changed ? new Date() : sensor.lastChangeAt, changes7d, events7d, consecutiveErrors: consecutive, lastWas304: outcome === "not_modified" });
		`@@ -224,6 +225,12 @@ export async function runSensor(sensor: Sensor, source: Source): Promise<RunOutc`
224	225	if (isNoise) m.suppressed.inc({ reason: `noise_${semantic.class}` });
225	226	return finish("changed", { snapshotId: snapId, changed: true });
226	227	}
	228	+ // Shadow monitoring (Source Factory): evidence and changes are recorded, nothing is published until the sensor is accepted.
	229	+ if (sensor.status === "SHADOW") {
	230	+ m.suppressed.inc({ reason: "shadow" });
	231	+ await db.update(changes).set({ meaningful: true }).where(sql`id = ${changeId}`);
	232	+ return finish("changed", { snapshotId: snapId, changed: true });
	233	+ }
227	234
228	235	// ---- Event --------------------------------------------------------------------------
229	236	const ev = await createEvent({ sensor, source, obs, norm, prev, snapId, changeId, diff, heuristic, semantic });
		`@@ -444,7 +451,7 @@ async function handleMissing(sensor: Sensor, source: Source, obs: Observation, f`
444	451	const sepOk = !missing.lastAt \|\| obs.fetchedAt.getTime() - new Date(missing.lastAt).getTime() >= config.deletion.minSeparationMin * 60e3;
445	452	if (sepOk) missing.count += 1;
446	453	missing.lastAt = obs.fetchedAt.toISOString();
447		− if (sensor.lastSnapshotId && missing.count >= config.deletion.confirmations && !missing.removedEventAt) {
	454	+ if (sensor.lastSnapshotId && missing.count >= config.deletion.confirmations && !missing.removedEventAt && sensor.status !== "SHADOW") {
448	455	// Confirmed deletion: independent checks separated in time.
449	456	missing.removedEventAt = obs.fetchedAt.toISOString();
450	457	const id = newId("evt");

modified apps/engine/src/registry.ts +39 −29

		`@@ -18,38 +18,48 @@ export function loadSeeds(file = config.sourcesFile, dir = config.sourcesDir): S`
18	18	return seeds;
19	19	}
20	20
21		−export async function syncRegistry(seeds = loadSeeds()): Promise<{ sources: number; sensors: number }> {
22		− let nSensors = 0;
23		− for (const s of seeds) {
24		− // Provenance: media/aggregators are third-party; everything else is the organization's own channel.
25		− const firstParty = s.first_party ?? !s.categories.some((c) => c === "news" \|\| c === "media");
26		− const country = s.country ?? inferCountry(s.domain, s.categories);
27		− const language = s.language ?? inferLanguage(s.domain);
28		− await db
29		− .insert(sources)
30		− .values({ id: s.id, name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, firstParty, country, language })
31		− .onConflictDoUpdate({ target: sources.id, set: { name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, firstParty, country, language, updatedAt: new Date() } });
	21	+/**
	22	+ * Upsert one organization (source row + organization entity + aliases + product entities). Shared by the YAML
	23	+ * registry sync (`origin = seed`) and the Source Factory (`origin = factory`). Sensors are handled by the caller.
	24	+ */
	25	+export async function upsertSourceRecord(s: SourceSeed, opts: { origin?: "seed" \| "import" \| "factory"; sector?: string \| null } = {}): Promise<void> {
	26	+ // Provenance: media/aggregators are third-party; everything else is the organization's own channel.
	27	+ const firstParty = s.first_party ?? !s.categories.some((c) => c === "news" \|\| c === "media");
	28	+ const country = s.country ?? inferCountry(s.domain, s.categories);
	29	+ const language = s.language ?? inferLanguage(s.domain);
	30	+ const origin = opts.origin ?? "seed";
	31	+ const base = { name: s.name, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, categories: s.categories, tier: s.tier, importanceWeight: s.weight, discover: s.discover, fallback: s.fallback, notes: s.notes, enabled: s.enabled, llmEnabled: s.llm, firstParty, country, language };
	32	+ await db
	33	+ .insert(sources)
	34	+ .values({ id: s.id, ...base, origin, sector: opts.sector ?? null })
	35	+ .onConflictDoUpdate({ target: sources.id, set: origin === "seed" ? { ...base, origin, updatedAt: new Date() } : { ...base, updatedAt: new Date(), ...(opts.sector ? { sector: opts.sector } : {}) } });
32	36
33		− // Organization entity + aliases
34		− const entId = `org_${s.id}`;
	37	+ // Organization entity + aliases
	38	+ const entId = `org_${s.id}`;
	39	+ await db
	40	+ .insert(entities)
	41	+ .values({ id: entId, name: s.name, type: s.entity_type, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, importance: sourceImportanceFromTier(s.tier, s.weight), categories: s.categories })
	42	+ .onConflictDoUpdate({ target: entities.id, set: { name: s.name, type: s.entity_type, domain: s.domain, description: s.description, importance: sourceImportanceFromTier(s.tier, s.weight), categories: s.categories } });
	43	+ await db.insert(sourceEntities).values({ sourceId: s.id, entityId: entId }).onConflictDoNothing();
	44	+ for (const alias of new Set([s.name, s.domain, s.domain.replace(/^www\./, ""), ...s.aliases])) {
	45	+ await db.insert(entityAliases).values({ alias: alias.toLowerCase(), entityId: entId }).onConflictDoNothing();
	46	+ }
	47	+ for (const p of s.products) {
	48	+ const pid = p.id ?? `prd_${s.id}_${slugify(p.name)}`;
35	49	await db
36	50	.insert(entities)
37		− .values({ id: entId, name: s.name, type: s.entity_type, domain: s.domain, homepage: s.homepage ?? `https://${s.domain}`, description: s.description, importance: sourceImportanceFromTier(s.tier, s.weight), categories: s.categories })
38		− .onConflictDoUpdate({ target: entities.id, set: { name: s.name, type: s.entity_type, domain: s.domain, description: s.description, importance: sourceImportanceFromTier(s.tier, s.weight), categories: s.categories } });
39		− await db.insert(sourceEntities).values({ sourceId: s.id, entityId: entId }).onConflictDoNothing();
40		− for (const alias of new Set([s.name, s.domain, s.domain.replace(/^www\./, ""), ...s.aliases])) {
41		− await db.insert(entityAliases).values({ alias: alias.toLowerCase(), entityId: entId }).onConflictDoNothing();
42		− }
43		− for (const p of s.products) {
44		− const pid = p.id ?? `prd_${s.id}_${slugify(p.name)}`;
45		− await db
46		− .insert(entities)
47		− .values({ id: pid, name: p.name, type: p.type, parentId: entId, importance: Math.max(30, sourceImportanceFromTier(s.tier, s.weight) - 10), categories: s.categories, domain: s.domain })
48		− .onConflictDoUpdate({ target: entities.id, set: { name: p.name, type: p.type, parentId: entId } });
49		− await db.insert(sourceEntities).values({ sourceId: s.id, entityId: pid }).onConflictDoNothing();
50		− for (const alias of new Set([p.name, ...p.aliases])) await db.insert(entityAliases).values({ alias: alias.toLowerCase(), entityId: pid }).onConflictDoNothing();
51		− await db.execute(sql`insert into entity_relations (from_id, relation, to_id) values (${entId}, 'owns', ${pid}) on conflict do nothing`);
52		− }
	51	+ .values({ id: pid, name: p.name, type: p.type, parentId: entId, importance: Math.max(30, sourceImportanceFromTier(s.tier, s.weight) - 10), categories: s.categories, domain: s.domain })
	52	+ .onConflictDoUpdate({ target: entities.id, set: { name: p.name, type: p.type, parentId: entId } });
	53	+ await db.insert(sourceEntities).values({ sourceId: s.id, entityId: pid }).onConflictDoNothing();
	54	+ for (const alias of new Set([p.name, ...p.aliases])) await db.insert(entityAliases).values({ alias: alias.toLowerCase(), entityId: pid }).onConflictDoNothing();
	55	+ await db.execute(sql`insert into entity_relations (from_id, relation, to_id) values (${entId}, 'owns', ${pid}) on conflict do nothing`);
	56	+ }
	57	+}
	58	+
	59	+export async function syncRegistry(seeds = loadSeeds()): Promise<{ sources: number; sensors: number }> {
	60	+ let nSensors = 0;
	61	+ for (const s of seeds) {
	62	+ await upsertSourceRecord(s, { origin: "seed" });
53	63
54	64	const seedIds: string[] = [];
55	65	for (const sen of s.sensors) {

added apps/web/src/app/ops/factory-panel.tsx +205 −0

		`@@ -0,0 +1,205 @@`
	1	+"use client";
	2	+
	3	+import Link from "next/link";
	4	+import { useEffect, useState } from "react";
	5	+import { Chip, Empty, HealthPill, Panel, Skeleton, Sparkline, Stat, Table, Td } from "@/components/ui";
	6	+import { fmtInt, relTime, utcDateTime } from "@/lib/format";
	7	+
	8	+/** Shape of GET /api/v1/admin/factory (internal). */
	9	+interface FactoryReport {
	10	+ seeds: Record<string, number>;
	11	+ candidates: Record<string, number>;
	12	+ shadow: { total: number; up: number; unhealthy: number; ready: number; raw_changes: number };
	13	+ shadow_by_kind: { kind_class: string; n: number }[];
	14	+ daily: { day: string; seeds_processed: number; requests: number; candidates: number; shadow_created: number; accepted: number; rejected: number; blocked: number }[];
	15	+ sectors: { sector: string; seeds: number; queued: number; discovering: number; discovered: number; blocked: number; errors: number; shadow: number; accepted: number; rejected: number }[];
	16	+ recent_seeds: { id: string; name: string; domain: string; sector: string \| null; status: string; candidates: number; shadow: number; accepted: number; rejected: number; last_error: string \| null; discovered_at: string \| null; source_id: string \| null }[];
	17	+ heartbeat: { at?: string; processed?: number; queued?: number; concurrency?: number; idle?: boolean; lastBatch?: { seed: string; status: string; candidates: number; shadow: number; requests: number; ms: number }[] } \| null;
	18	+ generated_at: string;
	19	+}
	20	+interface Candidate {
	21	+ id: string;
	22	+ source_id: string;
	23	+ url: string;
	24	+ name: string \| null;
	25	+ kind_class: string \| null;
	26	+ connector: string \| null;
	27	+ tier: string \| null;
	28	+ status: string;
	29	+ score_value: number \| null;
	30	+ reason: string \| null;
	31	+ shadow_sensor_id: string \| null;
	32	+ shadow_health: string \| null;
	33	+ shadow_runs: number \| null;
	34	+ shadow_changes: number \| null;
	35	+ updated_at: string;
	36	+}
	37	+
	38	+export function FactoryPanel({ fetcher, tick }: { fetcher: <T>(path: string, init?: RequestInit) => Promise<T>; tick: number }) {
	39	+ const [d, setD] = useState<FactoryReport \| null>(null);
	40	+ const [err, setErr] = useState<string \| null>(null);
	41	+ const [cands, setCands] = useState<Candidate[] \| null>(null);
	42	+ const [candStatus, setCandStatus] = useState("shadow");
	43	+ const [busy, setBusy] = useState<string \| null>(null);
	44	+
	45	+ useEffect(() => {
	46	+ let cancelled = false;
	47	+ fetcher<FactoryReport>("/api/v1/admin/factory")
	48	+ .then((r) => !cancelled && (setD(r), setErr(null)))
	49	+ .catch((e: Error) => !cancelled && setErr(e.message));
	50	+ return () => {
	51	+ cancelled = true;
	52	+ };
	53	+ }, [fetcher, tick]);
	54	+ useEffect(() => {
	55	+ let cancelled = false;
	56	+ fetcher<{ items: Candidate[] }>(`/api/v1/admin/factory/candidates?status=${encodeURIComponent(candStatus)}&limit=60`)
	57	+ .then((r) => !cancelled && setCands(r.items))
	58	+ .catch(() => !cancelled && setCands([]));
	59	+ return () => {
	60	+ cancelled = true;
	61	+ };
	62	+ }, [fetcher, tick, candStatus]);
	63	+
	64	+ const act = async (path: string): Promise<void> => {
	65	+ setBusy(path);
	66	+ try {
	67	+ await fetcher(path, { method: "POST", body: "{}" });
	68	+ const r = await fetcher<{ items: Candidate[] }>(`/api/v1/admin/factory/candidates?status=${encodeURIComponent(candStatus)}&limit=60`);
	69	+ setCands(r.items);
	70	+ } catch (e) {
	71	+ setErr((e as Error).message);
	72	+ } finally {
	73	+ setBusy(null);
	74	+ }
	75	+ };
	76	+
	77	+ if (err && !d) return <Panel title="Source Factory"><Empty>{err}</Empty></Panel>;
	78	+ if (!d) return <Skeleton className="h-40 w-full" />;
	79	+ const hb = d.heartbeat;
	80	+ const stale = hb?.at ? Date.now() - new Date(hb.at).getTime() > 5 * 60_000 : true;
	81	+ const daily = [...d.daily].reverse();
	82	+ const seedsTotal = Object.values(d.seeds).reduce((a, b) => a + b, 0);
	83	+ return (
	84	+ <div className="flex flex-col gap-4">
	85	+ <div className="panel grid grid-cols-2 divide-x divide-y divide-line sm:grid-cols-4 xl:grid-cols-8 xl:divide-y-0">
	86	+ <Stat label="Factory" value={hb ? (stale ? "STALE" : hb.idle ? "IDLE" : "LIVE") : "NO HEARTBEAT"} tone={hb && !stale ? "signal" : "warn"} hint={hb?.at ? `${relTime(hb.at)} · ${hb.processed ?? 0} processed` : "Redis ws:factory:status empty"} />
	87	+ <Stat label="Seeds" value={fmtInt(seedsTotal)} hint={`${fmtInt(d.seeds.queued ?? 0)} queued · ${fmtInt(d.seeds.discovering ?? 0)} discovering`} />
	88	+ <Stat label="Discovered" value={fmtInt((d.seeds.discovered ?? 0) + (d.seeds.done ?? 0))} hint={`${fmtInt(d.seeds.blocked ?? 0)} blocked · ${fmtInt(d.seeds.error ?? 0)} errors`} />
	89	+ <Stat label="Candidates" value={fmtInt(Object.values(d.candidates).reduce((a, b) => a + b, 0))} hint={`${fmtInt(d.candidates.candidate ?? 0)} waiting · ${fmtInt(d.candidates.duplicate ?? 0)} dup`} />
	90	+ <Stat label="Shadow" value={fmtInt(d.shadow.total)} tone="info" hint={`${fmtInt(d.shadow.ready)} ready · ${fmtInt(d.shadow.unhealthy)} unhealthy`} />
	91	+ <Stat label="Accepted" value={fmtInt(d.candidates.accepted ?? 0)} tone="signal" hint="production sensors" />
	92	+ <Stat label="Rejected" value={fmtInt(d.candidates.rejected ?? 0)} hint="errors · noise · duplicates" />
	93	+ <Stat label="Requests today" value={fmtInt(daily.at(-1)?.requests ?? 0)} hint={`${fmtInt(daily.at(-1)?.seeds_processed ?? 0)} organizations`} />
	94	+ </div>
	95	+
	96	+ <div className="grid grid-cols-1 gap-4 xl:grid-cols-3">
	97	+ <Panel title="Funnel · 30 days" dense>
	98	+ {daily.length ? (
	99	+ <div className="grid grid-cols-1 gap-3 p-3 sm:grid-cols-2">
	100	+ {(["seeds_processed", "candidates", "shadow_created", "accepted"] as const).map((k) => (
	101	+ <div key={k}>
	102	+ <div className="mb-1 flex items-baseline justify-between"><span className="label">{k.replace(/_/g, " ")}</span><span className="font-mono text-[12px] tabular">{fmtInt(daily.reduce((a, r) => a + Number(r[k] ?? 0), 0))}</span></div>
	103	+ <Sparkline values={daily.map((r) => Number(r[k] ?? 0))} tone={k === "accepted" ? "signal" : k === "shadow_created" ? "info" : "muted"} responsive />
	104	+ </div>
	105	+ ))}
	106	+ </div>
	107	+ ) : (
	108	+ <Empty>No factory activity yet.</Empty>
	109	+ )}
	110	+ </Panel>
	111	+ <Panel title="By sector" dense className="xl:col-span-2">
	112	+ {d.sectors.length ? (
	113	+ <div className="max-h-72 overflow-auto">
	114	+ <Table head={["Sector", "Seeds", "Queued", "Discovered", "Blocked", "Errors", "Shadow", "Accepted", "Rejected"]}>
	115	+ {d.sectors.map((s) => (
	116	+ <tr key={s.sector}>
	117	+ <Td><Link href={`/coverage/${s.sector}`} className="hover:underline">{s.sector}</Link></Td>
	118	+ <Td mono>{fmtInt(s.seeds)}</Td>
	119	+ <Td mono className="text-fg-subtle">{fmtInt(s.queued + s.discovering)}</Td>
	120	+ <Td mono>{fmtInt(s.discovered)}</Td>
	121	+ <Td mono className={s.blocked ? "text-warn" : "text-fg-subtle"}>{fmtInt(s.blocked)}</Td>
	122	+ <Td mono className={s.errors ? "text-danger" : "text-fg-subtle"}>{fmtInt(s.errors)}</Td>
	123	+ <Td mono className="text-info">{fmtInt(s.shadow)}</Td>
	124	+ <Td mono className="text-signal">{fmtInt(s.accepted)}</Td>
	125	+ <Td mono className="text-fg-subtle">{fmtInt(s.rejected)}</Td>
	126	+ </tr>
	127	+ ))}
	128	+ </Table>
	129	+ </div>
	130	+ ) : (
	131	+ <Empty>No seeds yet — run <code>cli.ts factory seed</code> or POST /api/v1/admin/factory/seeds.</Empty>
	132	+ )}
	133	+ </Panel>
	134	+ </div>
	135	+
	136	+ <div className="grid grid-cols-1 gap-4 xl:grid-cols-2">
	137	+ <Panel title="Recently processed organizations" dense>
	138	+ {d.recent_seeds.length ? (
	139	+ <div className="max-h-80 overflow-auto">
	140	+ <Table head={["Organization", "Sector", "Status", "Cand.", "Shadow", "Acc.", "Rej.", "When"]}>
	141	+ {d.recent_seeds.map((s) => (
	142	+ <tr key={s.id}>
	143	+ <Td>{s.source_id ? <Link href={`/source/${s.source_id}`} className="hover:underline">{s.name}</Link> : s.name}<div className="font-mono text-[10.5px] text-fg-subtle">{s.domain}{s.last_error ? ` · ${s.last_error}` : ""}</div></Td>
	144	+ <Td mono className="text-fg-subtle">{s.sector ?? "—"}</Td>
	145	+ <Td><Chip tone={s.status === "discovered" ? "ok" : s.status === "blocked" ? "warn" : s.status === "error" ? "danger" : "default"}>{s.status}</Chip></Td>
	146	+ <Td mono>{s.candidates}</Td>
	147	+ <Td mono className="text-info">{s.shadow}</Td>
	148	+ <Td mono className="text-signal">{s.accepted}</Td>
	149	+ <Td mono className="text-fg-subtle">{s.rejected}</Td>
	150	+ <Td mono className="text-fg-subtle"><span title={s.discovered_at ? utcDateTime(s.discovered_at) : ""}>{s.discovered_at ? relTime(s.discovered_at) : "—"}</span></Td>
	151	+ </tr>
	152	+ ))}
	153	+ </Table>
	154	+ </div>
	155	+ ) : (
	156	+ <Empty>Nothing processed yet.</Empty>
	157	+ )}
	158	+ </Panel>
	159	+ <Panel
	160	+ title="Candidates"
	161	+ dense
	162	+ action={
	163	+ <span className="flex gap-1">
	164	+ {["shadow", "candidate", "accepted", "rejected", "duplicate"].map((s) => (
	165	+ <button key={s} type="button" onClick={() => setCandStatus(s)} className={`rounded px-1.5 py-0.5 font-mono text-[10.5px] ${candStatus === s ? "bg-panel-2 text-fg" : "text-fg-subtle hover:text-fg"}`}>{s}</button>
	166	+ ))}
	167	+ </span>
	168	+ }
	169	+ >
	170	+ {cands === null ? (
	171	+ <Skeleton className="h-40 w-full" />
	172	+ ) : cands.length === 0 ? (
	173	+ <Empty>No candidate with status {candStatus}.</Empty>
	174	+ ) : (
	175	+ <div className="max-h-80 overflow-auto">
	176	+ <Table head={["Sensor", "Kind", "Score", "Shadow", "Reason", ""]}>
	177	+ {cands.map((c) => (
	178	+ <tr key={c.id}>
	179	+ <Td><Link href={`/source/${c.source_id}`} className="hover:underline">{c.source_id}</Link><div className="max-w-[28ch] truncate font-mono text-[10.5px] text-fg-subtle" title={c.url}>{c.name ?? ""} · {c.url}</div></Td>
	180	+ <Td mono className="text-fg-subtle">{c.kind_class ?? "—"}<div>{c.connector} {c.tier}</div></Td>
	181	+ <Td mono>{c.score_value?.toFixed(2) ?? "—"}</Td>
	182	+ <Td>{c.shadow_sensor_id ? <span className="flex flex-col gap-0.5"><HealthPill health={c.shadow_health ?? "UP"} /><span className="font-mono text-[10.5px] text-fg-subtle">{c.shadow_runs ?? 0} checks · {c.shadow_changes ?? 0} changes</span></span> : <span className="text-fg-subtle">—</span>}</Td>
	183	+ <Td className="text-[11px] text-fg-subtle"><span className="block max-w-[24ch] truncate" title={c.reason ?? ""}>{c.reason ?? "—"}</span></Td>
	184	+ <Td>
	185	+ {c.status === "shadow" && c.shadow_sensor_id && (
	186	+ <span className="flex gap-1">
	187	+ <button type="button" disabled={busy !== null} onClick={() => void act(`/api/v1/admin/factory/shadow/${c.shadow_sensor_id}/accept`)} className="rounded border border-line px-1.5 py-0.5 text-[10.5px] text-signal hover:border-line-strong disabled:opacity-50">accept</button>
	188	+ <button type="button" disabled={busy !== null} onClick={() => void act(`/api/v1/admin/factory/shadow/${c.shadow_sensor_id}/reject`)} className="rounded border border-line px-1.5 py-0.5 text-[10.5px] text-danger hover:border-line-strong disabled:opacity-50">reject</button>
	189	+ </span>
	190	+ )}
	191	+ {c.status === "candidate" && c.connector && (
	192	+ <button type="button" disabled={busy !== null} onClick={() => void act(`/api/v1/admin/factory/candidates/${c.id}/shadow`)} className="rounded border border-line px-1.5 py-0.5 text-[10.5px] text-info hover:border-line-strong disabled:opacity-50">shadow</button>
	193	+ )}
	194	+ </Td>
	195	+ </tr>
	196	+ ))}
	197	+ </Table>
	198	+ </div>
	199	+ )}
	200	+ </Panel>
	201	+ </div>
	202	+ {hb?.lastBatch?.length ? <p className="text-[11px] text-fg-subtle">last batch: {hb.lastBatch.map((b) => `${b.seed} ${b.status} (${b.candidates} cand · ${b.shadow} shadow · ${b.requests} req · ${Math.round(b.ms / 1000)} s)`).join(" · ")}</p> : null}
	203	+ </div>
	204	+ );
	205	+}

modified apps/web/src/app/ops/ops.tsx +6 −1

		`@@ -2,11 +2,12 @@`
2	2
3	3	import { KeyRound, LogOut, Play, RefreshCw, ShieldAlert } from "lucide-react";
4	4	import Link from "next/link";
5		−import { useEffect, useMemo, useState, type ReactNode } from "react";
	5	+import { useCallback, useEffect, useMemo, useState, type ReactNode } from "react";
6	6	import { SENSOR_TYPES, TIERS } from "@websensor/core/client";
7	7	import { useMounted } from "@/components/theme";
8	8	import { Chip, Empty, HealthPill, Panel, Skeleton, Sparkline, Stat, Table, Td } from "@/components/ui";
9	9	import { clientApiBase } from "@/lib/api";
	10	+import { FactoryPanel } from "./factory-panel";
10	11	import { fmtBytes, fmtInt, fmtMs, fmtPct, relTime, untilTime, utcDateTime, utcTime } from "@/lib/format";
11	12
12	13	const KEY = "ws_admin";
		`@@ -122,6 +123,7 @@ export function Ops() {`
122	123	setRefreshing(true);
123	124	setTick((x) => x + 1);
124	125	};
	126	+ const factoryFetcher = useCallback(<T,>(path: string, init: RequestInit = {}): Promise<T> => adminFetch<T>(path, init, token ?? ""), [token]);
125	127	const submitToken = async (): Promise<void> => {
126	128	const t = input.trim();
127	129	if (!t) return;
		`@@ -187,6 +189,9 @@ export function Ops() {`
187	189
188	190	{data ? <Dashboard d={data} /> : <Skeleton className="h-64 w-full" />}
189	191
	192	+ <h2 className="mt-2 text-[13px] font-semibold tracking-tight">Source Factory <span className="font-normal text-fg-subtle">· discovery → shadow → acceptance</span></h2>
	193	+ <FactoryPanel fetcher={factoryFetcher} tick={tick} />
	194	+
190	195	<Tools connectors={data?.connectors ?? []} onDone={refresh} />
191	196	</div>
192	197	);

modified apps/web/src/app/page.tsx +2 −0

		`@@ -1,6 +1,7 @@`
1	1	import Link from "next/link";
2	2	import { LiveFeed } from "@/components/live-feed";
3	3	import { LiveStrip } from "@/components/live-strip";
	4	+import { CoverageRail } from "@/components/coverage-rail";
4	5	import { AnomalyRail, BreakingRail, ClustersPanel, SilentRail, TrendingPanel } from "@/components/rail";
5	6	import { EventRow } from "@/components/event-row";
6	7	import { Empty, Panel } from "@/components/ui";
		`@@ -39,6 +40,7 @@ export default async function HomePage() {`
39	40	<BreakingRail items={breaking?.breaking_now ?? []} developing={breaking?.developing ?? []} />
40	41	<TrendingPanel items={trending.items} />
41	42	<AnomalyRail items={explore?.unusual_activity ?? []} />
	43	+ <CoverageRail />
42	44	</aside>
43	45	</div>
44	46	<div className="mt-4 grid grid-cols-1 gap-4 lg:grid-cols-3 [&>*]:min-w-0">

modified apps/web/src/components/command-palette.tsx +1 −0

		`@@ -33,6 +33,7 @@ const NAV: { label: string; href: string; hint: string; icon: ReactNode }[] = [`
33	33	{ label: "Entities", href: "/entities", hint: "organizations, products, models", icon: <Building2 className="size-3.5" /> },
34	34	{ label: "Countries", href: "/country", hint: "national desks", icon: <Globe2 className="size-3.5" /> },
35	35	{ label: "Sources", href: "/sources", hint: "monitored organizations", icon: <Building2 className="size-3.5" /> },
	36	+ { label: "Coverage", href: "/coverage", hint: "global observation coverage score by sector", icon: <Globe2 className="size-3.5" /> },
36	37	{ label: "Watchlists", href: "/watchlists", hint: "your entities, sources, keywords", icon: <Layers className="size-3.5" /> },
37	38	{ label: "Alerts", href: "/alerts", hint: "rules · webhooks", icon: <Siren className="size-3.5" /> },
38	39	{ label: "Bookmarks", href: "/bookmarks", hint: "saved events", icon: <Layers className="size-3.5" /> },

added apps/web/src/components/coverage-rail.tsx +27 −0

		`@@ -0,0 +1,27 @@`
	1	+import Link from "next/link";
	2	+import { api } from "@/lib/api";
	3	+import { fmtInt } from "@/lib/format";
	4	+import { Bar, Empty, Panel } from "./ui";
	5	+
	6	+/** Homepage rail: Global Observation Coverage Score by sector (server component). */
	7	+export async function CoverageRail() {
	8	+ const c = await api.coverage();
	9	+ return (
	10	+ <Panel title={<span>Observation coverage {c ? <span className={`font-mono ${c.global_score >= 60 ? "text-signal" : "text-high"}`}>{c.global_score.toFixed(1)}</span> : null}</span>} dense action={<Link href="/coverage" className="text-[11px] text-fg-subtle hover:text-fg">coverage →</Link>}>
	11	+ {!c \|\| !c.sectors.length ? (
	12	+ <Empty>Coverage universes are loading.</Empty>
	13	+ ) : (
	14	+ <ul className="px-3 py-1.5 text-[12px]">
	15	+ {c.sectors.slice(0, 10).map((s) => (
	16	+ <li key={s.sector} className="grid grid-cols-[7.5rem_1fr_2.5rem] items-center gap-2 py-0.5">
	17	+ <Link href={`/coverage/${s.sector}`} className="truncate text-fg-muted hover:text-fg" title={`${fmtInt(s.covered)} / ${fmtInt(s.members)} observed`}>{s.label}</Link>
	18	+ <Bar value={s.score} tone={s.score >= 80 ? "signal" : s.score >= 60 ? "high" : s.score >= 40 ? "mid" : "info"} />
	19	+ <span className="text-right font-mono tabular text-fg-muted">{Math.round(s.score)}%</span>
	20	+ </li>
	21	+ ))}
	22	+ <li className="pt-1 text-[10.5px] text-fg-subtle">{fmtInt(c.covered)} / {fmtInt(c.members)} organizations · {fmtInt(c.monitored.shadow)} sensors in shadow</li>
	23	+ </ul>
	24	+ )}
	25	+ </Panel>
	26	+ );
	27	+}

modified apps/web/src/components/nav.tsx +2 −0

		`@@ -18,6 +18,7 @@ const NAV = [`
18	18	["/explore", "Explore"],
19	19	["/entities", "Entities"],
20	20	["/sources", "Sources"],
	21	+ ["/coverage", "Coverage"],
21	22	["/watchlists", "Watchlists"],
22	23	["/alerts", "Alerts"],
23	24	] as const;
		`@@ -82,6 +83,7 @@ export function MobileNav() {`
82	83	["/silent", "Silent changes", LayoutGrid],
83	84	["/entities", "Entities", Globe2],
84	85	["/sources", "Sources", Globe2],
	86	+ ["/coverage", "Coverage", Globe2],
85	87	["/country", "Countries", Globe2],
86	88	["/alerts", "Alerts", Bell],
87	89	["/bookmarks", "Bookmarks", Bookmark],

modified apps/web/src/components/ui.tsx +2 −2

		`@@ -80,10 +80,10 @@ export function EvidenceTag({ label }: { label: string \| null \| undefined }) {`
80	80
81	81	export function HealthPill({ health }: { health: string \| null \| undefined }) {
82	82	const h = (health ?? "UP").toUpperCase();
83		− const tone: Tone = h === "UP" \|\| h === "ACTIVE" \|\| h === "VALIDATED" ? "ok" : h === "DEGRADED" \|\| h === "RATE_LIMITED" \|\| h === "PENDING" ? "warn" : h === "ERROR" ? "danger" : "default";
	83	+ const tone: Tone = h === "UP" \|\| h === "ACTIVE" \|\| h === "VALIDATED" ? "ok" : h === "DEGRADED" \|\| h === "RATE_LIMITED" \|\| h === "PENDING" ? "warn" : h === "ERROR" ? "danger" : h === "SHADOW" ? "info" : "default";
84	84	return (
85	85	<Chip tone={tone} className="font-mono">
86		− <span className={`inline-block size-1.5 rounded-full ${tone === "ok" ? "bg-ok" : tone === "danger" ? "bg-danger" : tone === "default" ? "bg-low" : "bg-warn"}`} /> {h}
	86	+ <span className={`inline-block size-1.5 rounded-full ${tone === "ok" ? "bg-ok" : tone === "danger" ? "bg-danger" : tone === "default" ? "bg-low" : tone === "info" ? "bg-info" : "bg-warn"}`} /> {h}
87	87	</Chip>
88	88	);
89	89	}

modified apps/web/src/lib/api.ts +63 −0

		`@@ -196,7 +196,68 @@ export interface EventsPage {`
196	196	nextCursor: string \| null;
197	197	}
198	198
	199	+export interface CoverageSectorRow {
	200	+ sector: string;
	201	+ label: string;
	202	+ description: string \| null;
	203	+ weight: number;
	204	+ members: number;
	205	+ covered: number;
	206	+ shadow_members: number;
	207	+ queued: number;
	208	+ sensors: number;
	209	+ breadth: number;
	210	+ depth: number;
	211	+ score: number;
	212	+}
	213	+export interface CoverageSummary {
	214	+ global_score: number;
	215	+ members: number;
	216	+ covered: number;
	217	+ sensors_matched: number;
	218	+ sectors: CoverageSectorRow[];
	219	+ monitored: { sources: number; sensors: number; shadow: number; countries: number };
	220	+ factory: Record<string, number>;
	221	+ method: string;
	222	+ issues: string[];
	223	+ generated_at: string;
	224	+}
	225	+export interface CoverageMemberRow {
	226	+ name: string;
	227	+ domain: string;
	228	+ country: string \| null;
	229	+ importance: number;
	230	+ universe: string;
	231	+ covered: boolean;
	232	+ sensors: number;
	233	+ shadow: number;
	234	+ source_id: string \| null;
	235	+ seed_status: string \| null;
	236	+ depth: number;
	237	+}
	238	+export interface CoverageUniverseRow {
	239	+ key: string;
	240	+ label: string;
	241	+ description: string \| null;
	242	+ provenance: string \| null;
	243	+ members: number;
	244	+ covered: number;
	245	+ sensors: number;
	246	+ breadth: number;
	247	+ depth: number;
	248	+ score: number;
	249	+ items: CoverageMemberRow[];
	250	+}
	251	+export interface CoverageSectorDetail extends CoverageSectorRow {
	252	+ categories: string[];
	253	+ universes: CoverageUniverseRow[];
	254	+ by_country: { country: string; members: number; covered: number }[];
	255	+ generated_at: string;
	256	+}
	257	+
199	258	export interface Stats {
	259	+ sensors_shadow?: number;
	260	+ sources_factory?: number;
200	261	checks_per_min?: number;
201	262	events_per_min?: number;
202	263	events_1h?: number;
		`@@ -808,5 +869,7 @@ export const api = {`
808	869	compare: (a: string, b: string) => getJson<CompareResult \| null>(`/api/v1/snapshots/compare?a=${encodeURIComponent(a)}&b=${encodeURIComponent(b)}`, null),
809	870	change: (id: string) => getJson<{ change: ChangeRow & { diff: DiffSummary; sensor_url?: string; sensor_name?: string; source_id?: string }; unified: string \| null } \| null>(`/api/v1/changes/${encodeURIComponent(id)}`, null),
810	871	health: () => getJson<HealthReport \| null>("/api/v1/health/connectors", null),
	872	+ coverage: () => getJson<CoverageSummary \| null>("/api/v1/coverage", null, { revalidate: 60 }),
	873	+ coverageSector: (sector: string) => getJson<CoverageSectorDetail \| null>(`/api/v1/coverage/${encodeURIComponent(sector)}`, null, { revalidate: 60 }),
811	874	search: (q: string) => getJson<SearchResult>(`/api/v1/search?q=${encodeURIComponent(q)}`, { query: q, events: [], entities: [], sources: [], urls: [] }),
812	875	};

modified deploy/README.md +17 −1

		`@@ -1,6 +1,6 @@`
1	1	# Deploying WebSensor on MacLustr
2	2
3		−WebSensor runs as three PM2 processes on one node (M4M64b), published through the MacLustr Tunnel (WireGuard +
	3	+WebSensor runs as four PM2 processes on one node (M4M64b), published through the MacLustr Tunnel (WireGuard +
4	4	Caddy on the OVH gateway BHS64, `www.websensor.io`; ngrok was retired on 2026-09-10), orchestrated by `mld` from the
5	5	M1M32 gateway.
6	6
		`@@ -9,6 +9,7 @@ Internet → www.websensor.io → BHS64 (Caddy TLS) → wg1 → websensor-api (:`
9	9	├── /api/v1/* REST + WebSocket /api/v1/live
10	10	└── /* reverse proxy → websensor-web (Next.js :8261, loopback)
11	11	websensor-engine (scheduler + pipeline, Prometheus :8262) → PostgreSQL 17 `websensor` + Redis (loopback)
	12	+websensor-factory (Source Factory: coverage seeds → deep discovery → shadow sensors → acceptance; heartbeat Redis ws:factory:status)
12	13	Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by mld (excluded `data/`).
13	14	```
14	15
		`@@ -30,6 +31,21 @@ Blob store: ~/websensor-data/blobs (content-addressed, zstd) — never synced by`
30	31	- WebSocket upgrades traverse Caddy without configuration; the API sends a heartbeat every 25 s and clients
31	32	replay missed events with `{"since": lastSid}` after a reconnection.
32	33
	34	+## Source Factory (0.3)
	35	+- Seeds: `config/coverage/.yaml` (universes) and `config/factory/seeds/.yaml`; the post-sync hook runs
	36	+ `cli.ts factory seed --mode=hinted` (members without a monitored domain + members with hints). `--mode=all --requeue`
	37	+ re-discovers everything; `factory requeue` re-queues blocked/errored seeds.
	38	+- Process: `pm2 logs websensor-factory`; `cli.ts factory stats`; `/ops` → Source Factory panel (funnel, sectors,
	39	+ candidates, manual accept/reject); `GET /api/v1/admin/factory`, `/admin/factory/seeds`, `/admin/factory/candidates`,
	40	+ `POST /admin/factory/seeds` (bulk), `POST /admin/factory/shadow/:sensorId/accept\|reject`.
	41	+- Tuning (env on the factory process): `WS_FACTORY_CONCURRENCY` (6), `WS_FACTORY_BUDGET` (90 requests/org),
	42	+ `WS_FACTORY_MIN_SCORE` (0.35), `WS_FACTORY_SHADOW_MIN_CHECKS` (5), `WS_FACTORY_SHADOW_MIN_HOURS` (24),
	43	+ `WS_FACTORY_SHADOW_MAX_HOURS` (120). `GITHUB_TOKEN` (optional) improves repository ranking.
	44	+- Shadow sensors are polled by the engine at priority 3; raise `WS_FETCH_CONCURRENCY` on the engine when the shadow
	45	+ population grows (32 at 0.3). `cli.ts factory evaluate [--force]` runs the decision pass by hand;
	46	+ `cli.ts factory export --sector=x > config/sources.d/6x-factory-x.yaml` graduates accepted sensors into the registry.
	47	+- Coverage: `GET /api/v1/coverage`, `/coverage` page; universes validated with `apps/engine/src/coverage-validate.ts --dns`.
	48	+
33	49	## Operations
34	50	- Logs: `mld logs websensor` or `ssh M4M64b pm2 logs websensor-engine`.
35	51	- Registry edits: change `config/sources.yaml`, redeploy (sync runs in post-sync) or

modified deploy/websensor.mld.json.example +194 −123

		`@@ -1,128 +1,199 @@`
1	1	{
2		− "app": "websensor",
3		− "label": "WebSensor — Detect What Changed. Know Why It Matters.",
4		− "domain": "www.websensor.io",
5		− "port": 8260,
6		− "health_path": "/api/ready",
7		− "dir": "~/apps/websensor",
8		− "extra_paths": [],
9		− "sync_excludes": [
10		− "node_modules/",
11		− ".next/",
12		− ".git/",
13		− ".env",
14		− ".env.*",
15		− "!.env.example",
16		− "*.tsbuildinfo",
17		− ".turbo/",
18		− "logs/",
19		− "tmp/",
20		− "data/",
21		− "coverage/",
22		− ".DS_Store",
23		− ".claude/",
24		− "qa/",
25		− "apps/web/.next/",
26		− "apps/web/next-env.d.ts",
27		− "deploy/*.mld.json"
	2	+ "app": "websensor",
	3	+ "label": "WebSensor — Detect What Changed. Know Why It Matters.",
	4	+ "domain": "www.websensor.io",
	5	+ "port": 8260,
	6	+ "health_path": "/api/ready",
	7	+ "dir": "~/apps/websensor",
	8	+ "extra_paths": [],
	9	+ "sync_excludes": [
	10	+ "node_modules/",
	11	+ ".next/",
	12	+ ".git/",
	13	+ ".env",
	14	+ ".env.*",
	15	+ "!.env.example",
	16	+ "*.tsbuildinfo",
	17	+ ".turbo/",
	18	+ "logs/",
	19	+ "tmp/",
	20	+ "data/",
	21	+ "coverage/",
	22	+ ".DS_Store",
	23	+ ".claude/",
	24	+ "qa/",
	25	+ "apps/web/.next/",
	26	+ "apps/web/next-env.d.ts",
	27	+ "deploy/*.mld.json"
	28	+ ],
	29	+ "requires": {
	30	+ "runtimes": [
	31	+ "pm2",
	32	+ "node",
	33	+ "pnpm",
	34	+ "postgresql@17",
	35	+ "redis"
	36	+ ],
	37	+ "ram_gb": 8,
	38	+ "ports": [
	39	+ 8260,
	40	+ 8261,
	41	+ 8262
	42	+ ]
	43	+ },
	44	+ "ram_mb_observed": 1500,
	45	+ "size_mb": 40,
	46	+ "placement": {
	47	+ "pin": "M4M64b",
	48	+ "prefer": null,
	49	+ "avoid": [
	50	+ "M3U96b",
	51	+ "M1M32"
28	52	],
29		− "requires": {
30		− "runtimes": ["pm2", "ngrok", "node", "pnpm", "postgresql@17", "redis"],
31		− "ram_gb": 6,
32		− "ports": [8260, 8261, 8262]
	53	+ "reason": "PostgreSQL 17 + Redis + pnpm présents ; 64 Go RAM ; M3U96a déjà chargé (25 processus PM2)"
	54	+ },
	55	+ "processes": [
	56	+ {
	57	+ "name": "websensor-engine",
	58	+ "manager": "pm2",
	59	+ "script": "/opt/homebrew/bin/node",
	60	+ "args": [
	61	+ "node_modules/tsx/dist/cli.mjs",
	62	+ "apps/engine/src/index.ts"
	63	+ ],
	64	+ "interpreter": null,
	65	+ "cwd": "{{HOME}}/apps/websensor",
	66	+ "env": {
	67	+ "NODE_ENV": "production",
	68	+ "DATABASE_URL": "postgres://localhost:5432/websensor",
	69	+ "REDIS_URL": "redis://127.0.0.1:6379",
	70	+ "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
	71	+ "WS_SOURCES_FILE": "./config/sources.yaml",
	72	+ "WS_FETCH_CONCURRENCY": "32",
	73	+ "WS_PER_HOST_CONCURRENCY": "2",
	74	+ "ENGINE_METRICS_PORT": "8262",
	75	+ "WS_USER_AGENT": "WebSensorBot/0.3 (+https://www.websensor.io/bot; contact@websensor.io)",
	76	+ "ANTHROPIC_API_KEY": "<set me>",
	77	+ "WS_LLM_MODEL_FAST": "claude-haiku-4-5",
	78	+ "WS_LLM_MODEL_DEEP": "claude-opus-5",
	79	+ "WS_LLM_DAILY_CALL_BUDGET": "600",
	80	+ "WS_LLM_MIN_IMPORTANCE": "35",
	81	+ "WS_LLM_DEEP_MIN_IMPORTANCE": "78",
	82	+ "SCRAPFLY_API_KEY": "<set me>",
	83	+ "WS_SCRAPFLY_DAILY_BUDGET": "300",
	84	+ "LOG_LEVEL": "info",
	85	+ "WS_RETENTION_RAW_DAYS": "21",
	86	+ "WS_RETENTION_CHANGED_RAW_DAYS": "60",
	87	+ "WS_SILENT_MIN_IMPORTANCE": "45"
	88	+ },
	89	+ "cron_restart": null,
	90	+ "autorestart": true,
	91	+ "max_memory_restart": "2G"
33	92	},
34		− "ram_mb_observed": 1500,
35		− "size_mb": 40,
36		− "placement": {
37		− "pin": "M4M64b",
38		− "prefer": null,
39		− "avoid": ["M3U96b", "M1M32"],
40		− "reason": "PostgreSQL 17 + Redis + pnpm présents ; 64 Go RAM ; M3U96a déjà chargé (25 processus PM2)"
	93	+ {
	94	+ "name": "websensor-factory",
	95	+ "manager": "pm2",
	96	+ "script": "/opt/homebrew/bin/node",
	97	+ "args": [
	98	+ "node_modules/tsx/dist/cli.mjs",
	99	+ "apps/engine/src/factory-main.ts"
	100	+ ],
	101	+ "interpreter": null,
	102	+ "cwd": "{{HOME}}/apps/websensor",
	103	+ "env": {
	104	+ "NODE_ENV": "production",
	105	+ "DATABASE_URL": "postgres://localhost:5432/websensor",
	106	+ "REDIS_URL": "redis://127.0.0.1:6379",
	107	+ "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
	108	+ "WS_SOURCES_FILE": "./config/sources.yaml",
	109	+ "WS_USER_AGENT": "WebSensorBot/0.3 (+https://www.websensor.io/bot; contact@websensor.io)",
	110	+ "LOG_LEVEL": "info",
	111	+ "WS_FACTORY": "1",
	112	+ "WS_FACTORY_CONCURRENCY": "6",
	113	+ "WS_FACTORY_BUDGET": "90",
	114	+ "WS_FACTORY_MIN_SCORE": "0.35",
	115	+ "WS_FACTORY_SHADOW_MIN_CHECKS": "5",
	116	+ "WS_FACTORY_SHADOW_MIN_HOURS": "24",
	117	+ "WS_FACTORY_SHADOW_MAX_HOURS": "120",
	118	+ "WS_FACTORY_TICK_SECONDS": "20",
	119	+ "WS_COVERAGE_DIR": "./config/coverage",
	120	+ "WS_EDGAR_USER_AGENT": "WebSensor (contact@websensor.io)",
	121	+ "WS_APP_NAME": "websensor-factory"
	122	+ },
	123	+ "cron_restart": null,
	124	+ "autorestart": true,
	125	+ "max_memory_restart": "1500M"
41	126	},
42		− "processes": [
43		− {
44		− "name": "websensor-engine",
45		− "manager": "pm2",
46		− "script": "/opt/homebrew/bin/node",
47		− "args": ["node_modules/tsx/dist/cli.mjs", "apps/engine/src/index.ts"],
48		− "interpreter": null,
49		− "cwd": "{{HOME}}/apps/websensor",
50		− "env": {
51		− "NODE_ENV": "production",
52		− "DATABASE_URL": "postgres://localhost:5432/websensor",
53		− "REDIS_URL": "redis://127.0.0.1:6379",
54		− "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
55		− "WS_SOURCES_FILE": "./config/sources.yaml",
56		− "WS_FETCH_CONCURRENCY": "16",
57		− "WS_PER_HOST_CONCURRENCY": "2",
58		− "ENGINE_METRICS_PORT": "8262",
59		− "WS_USER_AGENT": "WebSensorBot/0.1 (+https://www.websensor.io/bot; contact@websensor.io)",
60		− "ANTHROPIC_API_KEY": "<secret>",
61		− "WS_LLM_MODEL_FAST": "claude-haiku-4-5",
62		− "WS_LLM_MODEL_DEEP": "claude-opus-5",
63		− "WS_LLM_DAILY_CALL_BUDGET": "600",
64		− "WS_LLM_MIN_IMPORTANCE": "35",
65		− "WS_LLM_DEEP_MIN_IMPORTANCE": "78",
66		− "SCRAPFLY_API_KEY": "<secret>",
67		− "WS_SCRAPFLY_DAILY_BUDGET": "300",
68		− "LOG_LEVEL": "info"
69		− },
70		− "cron_restart": null,
71		− "autorestart": true,
72		− "max_memory_restart": "2G"
73		− },
74		− {
75		− "name": "websensor-api",
76		− "manager": "pm2",
77		− "script": "/opt/homebrew/bin/node",
78		− "args": ["node_modules/tsx/dist/cli.mjs", "apps/api/src/server.ts"],
79		− "interpreter": null,
80		− "cwd": "{{HOME}}/apps/websensor",
81		− "env": {
82		− "NODE_ENV": "production",
83		− "DATABASE_URL": "postgres://localhost:5432/websensor",
84		− "REDIS_URL": "redis://127.0.0.1:6379",
85		− "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
86		− "API_PORT": "8260",
87		− "API_HOST": "0.0.0.0",
88		− "WEB_URL": "http://127.0.0.1:8261",
89		− "PUBLIC_BASE_URL": "https://www.websensor.io",
90		− "CANONICAL_HOST": "www.websensor.io",
91		− "LOG_LEVEL": "info"
92		− },
93		− "cron_restart": null,
94		− "autorestart": true,
95		− "max_memory_restart": "1G"
96		− },
97		− {
98		− "name": "websensor-web",
99		− "manager": "pm2",
100		− "script": "/opt/homebrew/bin/node",
101		− "args": ["node_modules/next/dist/bin/next", "start", "-p", "8261", "-H", "127.0.0.1"],
102		− "interpreter": null,
103		− "cwd": "{{HOME}}/apps/websensor/apps/web",
104		− "env": {
105		− "NODE_ENV": "production",
106		− "API_URL": "http://127.0.0.1:8260",
107		− "NEXT_PUBLIC_SITE_URL": "https://www.websensor.io",
108		− "NEXT_TELEMETRY_DISABLED": "1"
109		− },
110		− "cron_restart": null,
111		− "autorestart": true,
112		− "max_memory_restart": "1G"
113		− }
114		− ],
115		− "ngrok": { "name": "websensor-ngrok", "url": "www.websensor.io", "port": 8260 },
116		− "launchd": [],
117		− "env_overrides": {},
118		− "hooks": {
119		− "post_sync": [
120		− "export PATH=\"/opt/homebrew/bin:$PATH\"; pnpm install --frozen-lockfile --silent && echo ' deps ok'",
121		− "export PATH=\"/opt/homebrew/bin:$PATH\"; psql -d postgres -Atc \"select 1 from pg_database where datname='websensor'\" \| grep -q 1 \|\| createdb websensor; DATABASE_URL=postgres://localhost:5432/websensor pnpm db:migrate 2>&1 \| tail -1 && echo ' db ok'",
122		− "export PATH=\"/opt/homebrew/bin:$PATH\"; mkdir -p $HOME/websensor-data/blobs && DATABASE_URL=postgres://localhost:5432/websensor node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync 2>&1 \| tail -1 && echo ' registry ok'",
123		− "export PATH=\"/opt/homebrew/bin:$PATH\"; export API_URL=http://127.0.0.1:8260 NEXT_PUBLIC_SITE_URL=https://www.websensor.io NEXT_TELEMETRY_DISABLED=1; pnpm --filter @websensor/web build 2>&1 \| tail -3 && echo ' web build ok'"
124		− ],
125		− "post_start": []
	127	+ {
	128	+ "name": "websensor-api",
	129	+ "manager": "pm2",
	130	+ "script": "/opt/homebrew/bin/node",
	131	+ "args": [
	132	+ "node_modules/tsx/dist/cli.mjs",
	133	+ "apps/api/src/server.ts"
	134	+ ],
	135	+ "interpreter": null,
	136	+ "cwd": "{{HOME}}/apps/websensor",
	137	+ "env": {
	138	+ "NODE_ENV": "production",
	139	+ "DATABASE_URL": "postgres://localhost:5432/websensor",
	140	+ "REDIS_URL": "redis://127.0.0.1:6379",
	141	+ "BLOB_STORE_DIR": "{{HOME}}/websensor-data/blobs",
	142	+ "API_PORT": "8260",
	143	+ "API_HOST": "0.0.0.0",
	144	+ "WEB_URL": "http://127.0.0.1:8261",
	145	+ "PUBLIC_BASE_URL": "https://www.websensor.io",
	146	+ "CANONICAL_HOST": "www.websensor.io",
	147	+ "LOG_LEVEL": "info",
	148	+ "WS_ADMIN_TOKEN": "<set me>",
	149	+ "WS_MONITORS_PER_OWNER": "5"
	150	+ },
	151	+ "cron_restart": null,
	152	+ "autorestart": true,
	153	+ "max_memory_restart": "1G"
126	154	},
127		− "notes": "v0.1.0 (2026-09-08) : gateway Fastify :8260 (REST + WS /api/v1/live + proxy → Next :8261), engine (scheduler/pipeline, métriques :8262), 200 sources / ~290 capteurs (config/sources.yaml), Postgres 17 + Redis locaux, blobs zstd dans ~/websensor-data/blobs. Secrets uniquement dans ce manifeste (M1M32)."
128		−}
	155	+ {
	156	+ "name": "websensor-web",
	157	+ "manager": "pm2",
	158	+ "script": "/opt/homebrew/bin/node",
	159	+ "args": [
	160	+ "node_modules/next/dist/bin/next",
	161	+ "start",
	162	+ "-p",
	163	+ "8261",
	164	+ "-H",
	165	+ "127.0.0.1"
	166	+ ],
	167	+ "interpreter": null,
	168	+ "cwd": "{{HOME}}/apps/websensor/apps/web",
	169	+ "env": {
	170	+ "NODE_ENV": "production",
	171	+ "API_URL": "http://127.0.0.1:8260",
	172	+ "NEXT_PUBLIC_SITE_URL": "https://www.websensor.io",
	173	+ "NEXT_TELEMETRY_DISABLED": "1"
	174	+ },
	175	+ "cron_restart": null,
	176	+ "autorestart": true,
	177	+ "max_memory_restart": "1G"
	178	+ }
	179	+ ],
	180	+ "ngrok": null,
	181	+ "launchd": [],
	182	+ "env_overrides": {},
	183	+ "hooks": {
	184	+ "post_sync": [
	185	+ "export PATH=\"/opt/homebrew/bin:$PATH\"; pnpm install --frozen-lockfile --silent && echo ' deps ok'",
	186	+ "set -o pipefail; export PATH=\"/opt/homebrew/bin:$PATH\"; psql -d postgres -Atc \"select 1 from pg_database where datname='websensor'\" \| grep -q 1 \|\| createdb websensor; DATABASE_URL=postgres://localhost:5432/websensor pnpm db:migrate 2>&1 \| tail -1 && echo ' db ok'",
	187	+ "export PATH=\"/opt/homebrew/bin:$PATH\"; mkdir -p $HOME/websensor-data/blobs && DATABASE_URL=postgres://localhost:5432/websensor node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts sync 2>&1 \| tail -1 && echo ' registry ok'",
	188	+ "export PATH=\"/opt/homebrew/bin:$PATH\"; DATABASE_URL=postgres://localhost:5432/websensor node node_modules/tsx/dist/cli.mjs apps/engine/src/cli.ts factory seed --mode=hinted 2>&1 \| tail -1 && echo ' factory seeds ok'",
	189	+ "set -o pipefail; export PATH=\"/opt/homebrew/bin:$PATH\"; export API_URL=http://127.0.0.1:8260 NEXT_PUBLIC_SITE_URL=https://www.websensor.io NEXT_TELEMETRY_DISABLED=1; pnpm --filter @websensor/web build 2>&1 \| tail -3 && echo ' web build ok'"
	190	+ ],
	191	+ "post_start": []
	192	+ },
	193	+ "notes": "v0.1.0 (2026-09-08) : gateway Fastify :8260 (REST + WS /api/v1/live + proxy → Next :8261), engine (scheduler/pipeline, métriques :8262), 200 sources / ~290 capteurs (config/sources.yaml), Postgres 17 + Redis locaux, blobs zstd dans ~/websensor-data/blobs. Secrets uniquement dans ce manifeste (M1M32).",
	194	+ "tunnel": {
	195	+ "domain": "www.websensor.io",
	196	+ "gateway": "BHS64",
	197	+ "note": "MacLustr Tunnel depuis 2026-09-10 ; ngrok retiré"
	198	+ }
	199	+}
		\ No newline at end of file

modified docs/ARCHITECTURE.md +10 −1

		`@@ -64,7 +64,16 @@ developing clusters, new coverage), entity insights (35-day heatmap, baseline, d`
64	64	rank), entity rankings (importance × activity × velocity × quality × confirmation), cluster propagation, country and
65	65	category desks. Aggregates are cached 5–120 s in process (`cache.ts`); the live feed is never cached.
66	66
67		−## Not in v0.2 (explicitly)
	67	+## Source Factory (v0.3)
	68	+Separate process (`factory-main.ts`). Tables: `factory_seeds` (organization + hints + funnel counters),
	69	+`discovery_candidates` (extended: connector/type/name/kind_class/config/tier/score_value/reason/shadow_sensor_id),
	70	+`factory_daily`. Sensors gain `status = 'SHADOW'` (pipeline stores snapshots and changes, never events; priority 3);
	71	+sources gain `origin` (seed \| import \| factory) and `sector`. Discovery (`packages/connectors/src/discovery-deep.ts`)
	72	+is budgeted per organization (90 requests) and polite (4 in flight per organization, 6 organizations in parallel).
	73	+Acceptance is evidence-based (`evaluateShadows`). Coverage (`apps/api/src/coverage.ts`) matches universe members to
	74	+sensor hosts by registrable domain (`coverageKey`) and is cached 5 min.
	75	+
	76	+## Not in v0.3 (explicitly)
68	77	Browser rendering (Playwright), Firecrawl, OpenSearch (Postgres FTS + search syntax is used), MinIO (fs blob store
69	78	with the same interface), accounts and email/Slack/Discord/Telegram/push delivery (watchlists, alerts, bookmarks,
70	79	saved views and custom monitors are keyed by an anonymous owner token; alert channels are web + signed webhooks),

modified docs/registry/AUTHORING.md +5 −0

		@@ -6,6 +6,11 @@ entity; every `sensors:` entry becomes a real, polled endpoint. **Nothing goes i
6	6	fetched and parsed successfully** — a sensor that returns 403/404, HTML instead of a feed, or an empty list is a
7	7	liability (noise, wasted budget, false "page removed" events), not coverage.
8	8
	9	+> Coverage universes & the Source Factory (2026-09-13). Organizations that should be observed are listed per sector
	10	+> in `config/coverage/*.yaml` (guide: `COVERAGE.md`). The Factory discovers, validates, shadow-monitors and accepts their
	11	+> sensors automatically (`sources.origin = 'factory'`, `sensors.config.factory = true`). `cli.ts factory export` turns
	12	+> accepted sensors into a fragment like the ones below for review and graduation into `config/sources.d/`.
	13	+
9	14	## Fragment format
10	15
11	16	```yaml

added docs/registry/COVERAGE.md +67 −0

		`@@ -0,0 +1,67 @@`
	1	+# Coverage universes — authoring guide
	2	+
	3	+`config/coverage/.yaml` describes what WebSensor should observe*, sector by sector: the denominator of the
	4	+Global Observation Coverage Score (`/coverage`, `GET /api/v1/coverage`). The same files feed the **Source
	5	+Factory**: every member whose registrable domain is not yet monitored becomes an expansion seed, is
	6	+discovered automatically (robots, sitemaps, feeds, status pages, GitHub, IR/newsroom/pricing/legal pages,
	7	+EDGAR, Hugging Face…), shadow-monitored, then accepted or rejected on evidence.
	8	+
	9	+A universe is a list of organizations with a correct official domain. It is not a registry fragment:
	10	+no sensors, no URLs are required (they are welcome as `hints`). The registry stays the source of truth for
	11	+what is actually polled.
	12	+
	13	+## File format — one sector per file (`config/coverage/NN-<sector>.yaml`; several files may share a sector key, their universes are merged — universe keys must then be unique across those files)
	14	+
	15	+```yaml
	16	+sector: energy # kebab-case key, unique across files
	17	+label: Energy & commodities
	18	+description: Agencies, grid operators, producers and exchanges whose changes move energy markets.
	19	+weight: 1.0 # weight of the sector in the global score (1.0 default; 1.2–1.5 systemic)
	20	+categories: [energy] # default categories for factory-created sources (registry vocabulary)
	21	+tier: B # default tier for factory-created sources (S A B C D)
	22	+universes:
	23	+ - key: agencies
	24	+ label: Energy agencies & regulators
	25	+ provenance: "https://en.wikipedia.org/wiki/… (retrieved 2026-09-13)"
	26	+ members:
	27	+ - { name: International Energy Agency, domain: iea.org, country: INT, importance: 3, aliases: [iea] }
	28	+ - { name: ERCOT, domain: ercot.com, country: US, importance: 3, hints: { urls: [{ url: "https://www.ercot.com/news/rss", kind: news }] } }
	29	+ - key: oil-gas-majors
	30	+ label: Oil & gas majors
	31	+ members:
	32	+ - { name: Shell, domain: shell.com, country: GB, importance: 3, hints: { cik: "1306965", ticker: SHEL } }
	33	+```
	34	+
	35	+### Member fields
	36	+\| field \| required \| meaning \|
	37	+\|---\|---\|---\|
	38	+\| `name` \| yes \| official English name (ASCII preferred; native name in `aliases`) \|
	39	+\| `domain` \| yes \| bare registrable domain, lower-case, no scheme, no `www.`, no path (`gov.uk` sites: use the registrable domain, e.g. `bankofengland.co.uk`, `ons.gov.uk`) \|
	40	+\| `country` \| recommended \| ISO 3166-1 alpha-2 upper-case; `EU` for EU bodies; `INT` for global bodies. Quote `"NO"` (Norway) — YAML reads it as false \|
	41	+\| `language` \| optional \| ISO 639-1 of the site's main language when not English (`"no"` must be quoted) \|
	42	+\| `importance` \| optional \| 1 long tail · 2 notable (default) · 3 systemically important \|
	43	+\| `aliases` \| optional \| lower-case alternative names, tickers, acronyms (entity linking) \|
	44	+\| `categories` \| optional \| override of the sector default \|
	45	+\| `tier` \| optional \| override of the sector default \|
	46	+\| `first_party` \| optional \| `false` only for media/aggregators \|
	47	+\| `hints` \| optional \| `cik`, `ticker`, `exchange`, `github_org`, `github_repos: [owner/name]`, `hf_author`, `status_url`, `hosts: [ir.example.com]`, `urls: [{url, kind, connector?, type?, tier?, config?, name?}]`, `posture: true`, `wikidata: Q…` \|
	48	+\| `note` \| optional \| anything a reviewer should know (blocked by Cloudflare, JS shell…) \|
	49	+
	50	+`hints.urls[].kind` vocabulary: `news press blog ir filings changelog releases security pricing legal careers docs api status models data other`.
	51	+
	52	+## Rules
	53	+1. Real organizations, correct domains. Every domain must resolve (`dig +short <domain>` or `<www.domain>`)
	54	+ and belong to the organization. Wikidata P856 / the index constituents page / EDGAR `website` are good
	55	+ sources; never guess a domain from a name.
	56	+2. Provenance on every universe (`provenance:` URL or query + retrieval date). Index constituents change:
	57	+ record the date.
	58	+3. No duplicates inside a file (same domain twice). The same organization may appear in several sectors
	59	+ (a bank is both `corporate` and `markets`).
	60	+4. Countries everywhere for governments and companies; `INT` for international bodies.
	61	+5. Validate: `node node_modules/tsx/dist/cli.mjs apps/engine/src/coverage-validate.ts config/coverage/<file>.yaml [--dns]`
	62	+ (schema, duplicates, domain syntax; `--dns` resolves every domain). Only files that pass are committed.
	63	+6. Keep entries compact (flow style `{ … }`, one member per line).
	64	+
	65	+## Sector keys in use (2026-09-13)
	66	+`ai` · `cloud` · `cyber` · `markets` · `corporate` · `government` · `science` · `healthcare` · `energy` ·
	67	+`transport` · `telecom` · `logistics` · `open-source` · `commerce`

added packages/connectors/src/discovery-deep.ts +672 −0

		`@@ -0,0 +1,672 @@`
	1	+import { promises as dns } from "node:dns";
	2	+import { canonicalizeHtml, jaccard, registrableDomain, sha256, shingles, type SensorType, type Tier } from "@websensor/core";
	3	+import { httpFetch } from "./fetcher";
	4	+import { parseFeed } from "./rss";
	5	+import { parseSitemap } from "./sitemap";
	6	+import { detectFlavor } from "./statusjson";
	7	+
	8	+/**
	9	+ * Deep discovery — the acquisition stage of the Source Factory. Starting from an organization (domain +
	10	+ * optional hints) it walks robots.txt → sitemaps → feeds (`<link rel=alternate>`, well-known paths, official
	11	+ * sub-domains) → the homepage's own navigation (newsroom, press, investors, changelog, security, pricing,
	12	+ * legal, careers, docs, status) → known status-page providers → GitHub organization → EDGAR → Hugging Face →
	13	+ * OpenAPI documents → web posture, and validates EVERY candidate by fetching and parsing it exactly as the
	14	+ * connector would. Nothing is assumed from a URL pattern.
	15	+ *
	16	+ * The result is a ranked list of candidate sensors with the evidence the Factory needs to score them:
	17	+ * page class, first-party confidence, expected change frequency, fetch cost and a content fingerprint used
	18	+ * to drop duplicates (two feeds exposing the same items, two pages with the same text).
	19	+ */
	20	+export type CandidateKind = "news" \| "press" \| "blog" \| "ir" \| "filings" \| "changelog" \| "releases" \| "security" \| "pricing" \| "legal" \| "careers" \| "leadership" \| "docs" \| "api" \| "status" \| "sitemap" \| "models" \| "data" \| "posture" \| "other";
	21	+
	22	+export interface DeepCandidate {
	23	+ url: string;
	24	+ type: SensorType;
	25	+ connector: string;
	26	+ kind: CandidateKind;
	27	+ name: string;
	28	+ config: Record<string, unknown>;
	29	+ suggestedTier: Tier;
	30	+ evidence: string;
	31	+ /** rough information value 0–1 (before the Factory's importance-aware scoring) */
	32	+ value: number;
	33	+ itemCount?: number;
	34	+ /** items per day observed in the feed/list window (null when unknown) */
	35	+ itemsPerDay?: number \| null;
	36	+ /** 1 = the organization's own registrable domain; 0.9 = official channel on a platform (GitHub, HF, status provider) */
	37	+ firstPartyConfidence: number;
	38	+ fetchCost: { bytes: number; ms: number };
	39	+ /** content fingerprint for de-duplication (sorted item keys, or canonical text hash) */
	40	+ fingerprint?: string;
	41	+ itemKeys?: string[];
	42	+ textShingles?: Set<string>;
	43	+ title?: string;
	44	+}
	45	+
	46	+export interface DeepDiscoveryHints {
	47	+ cik?: string \| number;
	48	+ github_org?: string;
	49	+ github_repos?: string[];
	50	+ hf_author?: string;
	51	+ status_url?: string;
	52	+ hosts?: string[];
	53	+ urls?: { url: string; kind?: string; connector?: string; type?: string; tier?: Tier; config?: Record<string, unknown>; name?: string }[];
	54	+ posture?: boolean;
	55	+}
	56	+
	57	+export interface DeepDiscoveryOptions {
	58	+ hints?: DeepDiscoveryHints;
	59	+ /** maximum number of HTTP requests for this organization (default 90) */
	60	+ budget?: number;
	61	+ concurrency?: number;
	62	+ sensorIdForLogs?: string;
	63	+ probePages?: boolean;
	64	+ probeSubdomains?: boolean;
	65	+ probePosture?: boolean;
	66	+ probeOpenApi?: boolean;
	67	+}
	68	+
	69	+export interface DeepDiscoveryResult {
	70	+ domain: string;
	71	+ /** origin actually used after following the homepage redirect (may be another registrable domain) */
	72	+ origin: string;
	73	+ canonicalDomain: string;
	74	+ homeStatus: number;
	75	+ /** anti-bot / challenge detected on the homepage */
	76	+ blocked: boolean;
	77	+ requests: number;
	78	+ candidates: DeepCandidate[];
	79	+ rejected: { url: string; reason: string }[];
	80	+ notes: string[];
	81	+ durationMs: number;
	82	+}
	83	+
	84	+// ------------------------------------------------------------------------------------------------
	85	+// Vocabularies
	86	+// ------------------------------------------------------------------------------------------------
	87	+
	88	+const FEED_PATHS = ["/feed", "/rss", "/rss.xml", "/atom.xml", "/feed.xml", "/index.xml", "/feed/", "/feeds/all.atom.xml", "/blog/feed", "/blog/rss", "/blog/rss.xml", "/blog/feed.xml", "/blog/atom.xml", "/blog/index.xml", "/news/feed", "/news/rss", "/news/rss.xml", "/news/feed.xml", "/news.rss", "/news.xml", "/newsroom/rss", "/newsroom/feed", "/newsroom/rss.xml", "/press/rss", "/press/feed", "/press-releases/rss", "/press-releases/feed", "/rss/news", "/rss/press-releases.xml", "/rss/news-releases.xml", "/rss/pressreleases.xml", "/investors/rss", "/investors/feed", "/en/feed", "/en/rss", "/en/rss.xml", "/en/news/rss", "/changelog/feed", "/changelog.rss", "/changelog/rss.xml", "/changelog.xml", "/releases.atom", "/releases/feed", "/security/feed", "/security/rss", "/advisories/rss", "/advisories.rss", "/updates/feed", "/updates/rss", "/whats-new/feed", "/?feed=rss2", "/feed/atom", "/feeds/posts/default", "/feed.json"];
	89	+const SITEMAP_PATHS = ["/sitemap.xml", "/sitemap_index.xml", "/sitemap-index.xml", "/sitemaps/sitemap.xml", "/news-sitemap.xml", "/sitemap/news.xml", "/sitemap-news.xml", "/sitemap.txt"];
	90	+const OPENAPI_PATHS = ["/openapi.json", "/swagger.json", "/api/openapi.json", "/api/swagger.json", "/v1/openapi.json", "/api/v1/openapi.json", "/.well-known/openapi.json", "/swagger/v1/swagger.json", "/api-docs", "/openapi.yaml", "/api/openapi.yaml"];
	91	+const SUBDOMAIN_KINDS: [string, CandidateKind][] = [["blog", "blog"], ["news", "news"], ["newsroom", "news"], ["press", "press"], ["media", "press"], ["investors", "ir"], ["investor", "ir"], ["ir", "ir"], ["status", "status"], ["docs", "docs"], ["developer", "docs"], ["developers", "docs"], ["security", "security"], ["trust", "security"], ["careers", "careers"], ["jobs", "careers"], ["changelog", "changelog"], ["updates", "changelog"], ["engineering", "blog"], ["research", "blog"], ["about", "other"], ["corporate", "news"], ["api", "api"]];
	92	+
	93	+/** Page classes probed from navigation links and well-known paths. `paths` are fallbacks when the navigation has no link. */
	94	+const PAGE_KINDS: { kind: CandidateKind; label: string; hrefRe: RegExp; textRe: RegExp; paths: string[]; tier: Tier; value: number; max: number }[] = [
	95	+ { kind: "pricing", label: "pricing", hrefRe: /\/(pricing\|plans\|plans-and-pricing\|tarifs\|preise\|precios)(\/\|$\|\?)/i, textRe: /^(pricing\|plans\|plans & pricing\|tarifs\|preise\|precios)$/i, paths: ["/pricing", "/plans", "/pricing/"], tier: "C", value: 0.75, max: 1 },
	96	+ { kind: "changelog", label: "changelog", hrefRe: /\/(changelog\|change-log\|release-notes\|releasenotes\|whats-new\|what-s-new\|updates\|product-updates)(\/\|$\|\?)/i, textRe: /^(changelog\|change log\|release notes\|what'?s new\|product updates\|updates)$/i, paths: ["/changelog", "/release-notes", "/whats-new", "/updates"], tier: "B", value: 0.75, max: 1 },
	97	+ { kind: "security", label: "security advisories", hrefRe: /\/(security\|security-advisories\|advisories\|psirt\|trust\|security-bulletins\|bulletins\|vulnerabilit)/i, textRe: /^(security\|security advisories\|advisories\|trust center\|trust\|psirt\|security bulletins)$/i, paths: ["/security", "/security/advisories", "/trust", "/.well-known/security.txt"], tier: "B", value: 0.7, max: 2 },
	98	+ { kind: "news", label: "newsroom", hrefRe: /\/(news\|newsroom\|news-room\|media\|media-center\|media-centre\|actualites\|actualit%C3%A9s\|nouvelles\|aktuelles\|noticias\|notizie)(\/\|$\|\?)/i, textRe: /^(news\|newsroom\|news ?room\|media\|media cent(er\|re)\|latest news\|in the news\|actualit[ée]s\|nouvelles\|aktuelles\|noticias)$/i, paths: ["/news", "/newsroom", "/news/", "/en/news", "/media"], tier: "B", value: 0.65, max: 1 },
	99	+ { kind: "press", label: "press releases", hrefRe: /\/(press\|press-releases\|pressreleases\|press-room\|pressroom\|press-center\|communiques\|communiqu%C3%A9s\|presse\|prensa)(\/\|$\|\?)/i, textRe: /^(press\|press releases\|press ?room\|press cent(er\|re)\|communiqu[ée]s( de presse)?\|presse\|prensa)$/i, paths: ["/press", "/press-releases", "/press/", "/newsroom/press-releases"], tier: "B", value: 0.65, max: 1 },
	100	+ { kind: "ir", label: "investor relations", hrefRe: /\/(investors\|investor-relations\|investor\|ir\|relations-investisseurs\|investor-center)(\/\|$\|\?)/i, textRe: /^(investors\|investor relations\|investor cent(er\|re)\|ir\|relations investisseurs\|shareholders)$/i, paths: ["/investors", "/investor-relations", "/investors/", "/investor"], tier: "C", value: 0.6, max: 1 },
	101	+ { kind: "legal", label: "terms of service", hrefRe: /\/(terms\|terms-of-service\|terms-of-use\|tos\|legal\/terms\|conditions\|cgu\|cgv\|legal)(\/\|$\|\?\|\.)/i, textRe: /^(terms\|terms of (service\|use)\|terms & conditions\|terms and conditions\|legal\|conditions g[ée]n[ée]rales\|cgu\|cgv)$/i, paths: ["/terms", "/legal/terms", "/terms-of-service", "/legal"], tier: "C", value: 0.6, max: 1 },
	102	+ { kind: "legal", label: "privacy policy", hrefRe: /\/(privacy\|privacy-policy\|privacypolicy\|legal\/privacy\|politique-de-confidentialite\|datenschutz\|privacidad)(\/\|$\|\?\|\.)/i, textRe: /^(privacy\|privacy policy\|privacy notice\|politique de confidentialit[ée]\|datenschutz\|privacidad)$/i, paths: ["/privacy", "/legal/privacy", "/privacy-policy"], tier: "C", value: 0.55, max: 1 },
	103	+ { kind: "careers", label: "careers", hrefRe: /\/(careers\|jobs\|join-us\|work-with-us\|carrieres\|carri%C3%A8res\|karriere\|empleo)(\/\|$\|\?)/i, textRe: /^(careers\|jobs\|join us\|work with us\|carri[èe]res\|karriere\|empleo\|we'?re hiring)$/i, paths: ["/careers", "/jobs", "/careers/"], tier: "D", value: 0.35, max: 1 },
	104	+ { kind: "leadership", label: "leadership", hrefRe: /\/(leadership\|management\|executive-team\|our-team\|team\|board-of-directors\|board\|about\/leadership\|company\/leadership\|direction\|governance)(\/\|$\|\?)/i, textRe: /^(leadership\|leadership team\|management\|management team\|executive team\|our team\|board of directors\|governance\|direction)$/i, paths: ["/leadership", "/about/leadership", "/company/leadership", "/about/team"], tier: "D", value: 0.45, max: 1 },
	105	+ { kind: "docs", label: "documentation", hrefRe: /\/(docs\|documentation\|developers\|developer\|api-docs\|reference)(\/\|$\|\?)/i, textRe: /^(docs\|documentation\|developers\|developer\|api\|api reference\|api docs)$/i, paths: ["/docs", "/documentation", "/developers"], tier: "D", value: 0.4, max: 1 },
	106	+ { kind: "data", label: "open data", hrefRe: /\/(open-data\|opendata\|data\|statistics\|statistiques\|datasets\|publications)(\/\|$\|\?)/i, textRe: /^(open data\|data\|statistics\|statistiques\|datasets\|publications)$/i, paths: [], tier: "C", value: 0.4, max: 1 },
	107	+];
	108	+
	109	+const STATUS_HOST_RE = /^(https?:\/\/)?((status\|statuspage\|uptime\|health\|trust\|system-status\|servicestatus)\.[^/\s"']+\|[^/\s"']*\.(statuspage\.io\|instatus\.com\|betteruptime\.com\|betterstack\.com\|incident\.io\|status\.io\|hund\.io\|statuspal\.io\|freshstatus\.io\|status\.page\|statusgator\.com))/i;
	110	+const CHALLENGE_RE = /cf-browser-verification\|challenge-platform\|__cf_chl\|Attention Required!\|Just a moment\.\.\.\|Access Denied\|Request unsuccessful\. Incapsula\|akamai.*bot\|_Incapsula_Resource\|Please enable JavaScript to view\|captcha/i;
	111	+
	112	+// ------------------------------------------------------------------------------------------------
	113	+
	114	+export async function discoverOrganization(domain: string, opts: DeepDiscoveryOptions = {}): Promise<DeepDiscoveryResult> {
	115	+ const started = Date.now();
	116	+ const id = opts.sensorIdForLogs ?? `factory_${domain}`;
	117	+ const budget = opts.budget ?? 90;
	118	+ const hints = opts.hints ?? {};
	119	+ const notes: string[] = [];
	120	+ const rejected: { url: string; reason: string }[] = [];
	121	+ const found = new Map<string, DeepCandidate>();
	122	+ let requests = 0;
	123	+
	124	+ const add = (c: DeepCandidate): void => {
	125	+ const k = c.url.replace(/\/$/, "");
	126	+ const prev = found.get(k);
	127	+ if (!prev \|\| prev.value < c.value) found.set(k, c);
	128	+ };
	129	+ const budgetLeft = (): boolean => requests < budget;
	130	+ const get = async (url: string, o: { timeoutMs?: number; maxBytes?: number; accept?: string; headers?: Record<string, string>; userAgent?: string } = {}) => {
	131	+ requests++;
	132	+ return httpFetch(id, url, { timeoutMs: o.timeoutMs ?? 15_000, maxBytes: o.maxBytes ?? 3 * 1024 * 1024, accept: o.accept, headers: o.headers, userAgent: o.userAgent });
	133	+ };
	134	+
	135	+ // ---- 1. robots.txt --------------------------------------------------------------------------------------
	136	+ let base = `https://${domain}`;
	137	+ const sitemapsFromRobots: string[] = [];
	138	+ let robotsOk = false;
	139	+ let robotsMentionsAi = false;
	140	+ const robots = await get(`${base}/robots.txt`, { timeoutMs: 12_000, maxBytes: 512 * 1024 }).catch(() => null);
	141	+ if (robots?.body && robots.meta.status === 200 && !/^\s*<!doctype html\|<html/i.test(robots.body.toString("utf8").slice(0, 300))) {
	142	+ robotsOk = true;
	143	+ const text = robots.body.toString("utf8");
	144	+ for (const line of text.split(/\r?\n/)) {
	145	+ const m = line.match(/^\ssitemap:\s(\S+)/i);
	146	+ if (m) sitemapsFromRobots.push(m[1]!);
	147	+ }
	148	+ robotsMentionsAi = /GPTBot\|ClaudeBot\|Google-Extended\|CCBot\|anthropic-ai\|PerplexityBot\|Bytespider\|Applebot-Extended\|meta-externalagent/i.test(text);
	149	+ }
	150	+
	151	+ // ---- 2. homepage ------------------------------------------------------------------------------------------
	152	+ const home = await get(`${base}/`, { timeoutMs: 20_000, maxBytes: 4 * 1024 * 1024 });
	153	+ const homeStatus = home.meta.status;
	154	+ let blocked = false;
	155	+ const homeHtml = home.body && home.meta.status < 400 ? home.body.toString("utf8") : "";
	156	+ if ([403, 429, 503].includes(homeStatus) \|\| (homeHtml && CHALLENGE_RE.test(homeHtml.slice(0, 20_000)) && homeHtml.length < 60_000)) {
	157	+ blocked = true;
	158	+ notes.push(`homepage ${homeStatus}${homeHtml ? " (anti-bot challenge)" : ""}`);
	159	+ }
	160	+ let origin = base;
	161	+ try {
	162	+ const fu = new URL(home.meta.finalUrl \|\| base);
	163	+ origin = fu.origin;
	164	+ if (registrableDomain(fu.hostname) !== registrableDomain(domain)) notes.push(`homepage redirects to ${fu.hostname}`);
	165	+ } catch {
	166	+ // keep base
	167	+ }
	168	+ base = origin;
	169	+ const canonicalDomain = registrableDomain(new URL(base).hostname);
	170	+ const ownHosts = new Set<string>([registrableDomain(domain), canonicalDomain, ...(hints.hosts ?? []).map((h) => registrableDomain(h))]);
	171	+ const isOwn = (u: string): boolean => {
	172	+ try {
	173	+ return ownHosts.has(registrableDomain(new URL(u).hostname));
	174	+ } catch {
	175	+ return false;
	176	+ }
	177	+ };
	178	+
	179	+ const homeCanon = homeHtml ? canonicalizeHtml(homeHtml, home.meta.finalUrl \|\| base) : null;
	180	+ const homeShingles = homeCanon ? shingles(homeCanon.text) : null;
	181	+ const alternates: string[] = [];
	182	+ const statusLinks = new Set<string>();
	183	+ const githubOrgs = new Set<string>();
	184	+ const hfAuthors = new Set<string>();
	185	+ const pageLinks = new Map<CandidateKind, string[]>();
	186	+ if (homeHtml) {
	187	+ for (const m of homeHtml.matchAll(/<link[^>]+rel=["']alternate["'][^>]*>/gi)) {
	188	+ const tag = m[0];
	189	+ if (!/application\/(rss\|atom)\+xml\|application\/feed\+json/i.test(tag)) continue;
	190	+ const href = tag.match(/href=["']([^"']+)["']/i)?.[1];
	191	+ const abs = href ? safeAbs(href, home.meta.finalUrl \|\| base) : null;
	192	+ if (abs) alternates.push(abs);
	193	+ }
	194	+ // Navigation links: classify by href pattern first, then by anchor text.
	195	+ const anchors = [...homeHtml.matchAll(/<a\b[^>]href=["']([^"'#]+)["'][^>]>([\s\S]{0,300}?)<\/a>/gi)].map((m) => ({ href: safeAbs(m[1]!, home.meta.finalUrl \|\| base), text: m[2]!.replace(/<[^>]+>/g, " ").replace(/\s+/g, " ").trim() }));
	196	+ for (const a of anchors) {
	197	+ if (!a.href) continue;
	198	+ const gh = a.href.match(/^https?:\/\/(?:www\.)?github\.com\/([A-Za-z0-9_.-]+)\/?(?:$\|\?)/);
	199	+ if (gh && !/^(features\|about\|pricing\|topics\|login\|signup\|sponsors\|marketplace\|explore\|apps\|orgs\|enterprise\|solutions\|security\|team\|customer-stories\|readme\|collections\|events\|trending\|search\|settings\|notifications\|new\|organizations\|site\|contact)$/i.test(gh[1]!)) githubOrgs.add(gh[1]!);
	200	+ const hf = a.href.match(/^https?:\/\/huggingface\.co\/([A-Za-z0-9_.-]+)\/?$/);
	201	+ if (hf && !/^(models\|datasets\|spaces\|docs\|pricing\|papers\|blog\|join\|login\|enterprise\|posts\|tasks\|learn\|chat)$/i.test(hf[1]!)) hfAuthors.add(hf[1]!);
	202	+ if (STATUS_HOST_RE.test(a.href)) statusLinks.add(a.href);
	203	+ if (!isOwn(a.href)) continue;
	204	+ let path = "";
	205	+ try {
	206	+ path = new URL(a.href).pathname;
	207	+ } catch {
	208	+ continue;
	209	+ }
	210	+ if (path.split("/").filter(Boolean).length > 3) continue; // deep article links are not sections
	211	+ for (const pk of PAGE_KINDS) {
	212	+ if (pk.hrefRe.test(path) \|\| pk.textRe.test(a.text)) {
	213	+ const arr = pageLinks.get(pk.kind) ?? [];
	214	+ if (!arr.includes(a.href) && arr.length < 4) arr.push(a.href);
	215	+ pageLinks.set(pk.kind, arr);
	216	+ }
	217	+ }
	218	+ }
	219	+ }
	220	+ if (hints.github_org) githubOrgs.add(hints.github_org);
	221	+ if (hints.hf_author) hfAuthors.add(hints.hf_author);
	222	+ if (hints.status_url) statusLinks.add(hints.status_url);
	223	+
	224	+ // ---- 3. official sub-domains (cheap DNS first) ------------------------------------------------------------
	225	+ const subHosts: { host: string; kind: CandidateKind }[] = [];
	226	+ if (opts.probeSubdomains !== false) {
	227	+ const rd = registrableDomain(domain);
	228	+ // Wildcard DNS (everything resolves) makes existence checks meaningless: only probe the core labels then.
	229	+ const wildcard = await resolves(`ws-probe-${Date.now().toString(36)}.${rd}`);
	230	+ if (wildcard) notes.push("wildcard DNS");
	231	+ const labels = wildcard ? ["blog", "news", "status", "investors", "docs", "security"] : [...new Set([...SUBDOMAIN_KINDS.map(([l]) => l)])];
	232	+ await parallel(labels, 8, async (label) => {
	233	+ const host = `${label}.${rd}`;
	234	+ if (host === new URL(base).hostname) return;
	235	+ if (wildcard \|\| (await resolves(host))) subHosts.push({ host, kind: SUBDOMAIN_KINDS.find(([l]) => l === label)![1] });
	236	+ });
	237	+ if (wildcard) {
	238	+ // keep only sub-domains that actually answer with a distinct page
	239	+ const alive: typeof subHosts = [];
	240	+ await parallel(subHosts, 4, async (s) => {
	241	+ if (!budgetLeft()) return;
	242	+ const o = await get(`https://${s.host}/`, { timeoutMs: 10_000, maxBytes: 512 * 1024 });
	243	+ const fh = o.meta.finalUrl ? new URL(o.meta.finalUrl).hostname : "";
	244	+ if (o.meta.status === 200 && fh === s.host) alive.push(s);
	245	+ });
	246	+ subHosts.splice(0, subHosts.length, ...alive);
	247	+ }
	248	+ for (const h of hints.hosts ?? []) if (!subHosts.some((s) => s.host === h)) subHosts.push({ host: h, kind: "other" });
	249	+ if (subHosts.length) notes.push(`sub-domains: ${subHosts.map((s) => s.host).join(", ")}`);
	250	+ }
	251	+
	252	+ // ---- 4. feeds ---------------------------------------------------------------------------------------------
	253	+ const feedUrls = new Set<string>(alternates);
	254	+ for (const p of FEED_PATHS) feedUrls.add(base + p);
	255	+ for (const s of subHosts) {
	256	+ if (s.kind === "status" \|\| s.kind === "api") continue;
	257	+ for (const p of ["/feed", "/rss", "/rss.xml", "/atom.xml", "/feed.xml", "/index.xml", "/feed/", "/?feed=rss2"]) feedUrls.add(`https://${s.host}${p}`);
	258	+ }
	259	+ // Feeds are cheap and high-value: spend up to 55 % of the budget here.
	260	+ const feedList = [...feedUrls].slice(0, Math.max(10, Math.floor(budget * 0.55)));
	261	+ const feeds: DeepCandidate[] = [];
	262	+ await parallel(feedList, opts.concurrency ?? 4, async (url) => {
	263	+ if (!budgetLeft()) return;
	264	+ const o = await get(url, { timeoutMs: 12_000, maxBytes: 4 * 1024 * 1024, accept: "application/rss+xml, application/atom+xml, application/feed+json, application/xml;q=0.9, /;q=0.5" });
	265	+ if (!o.body \|\| o.meta.status !== 200) return;
	266	+ const text = o.body.toString("utf8");
	267	+ if (/^\s*<!doctype html\|<html/i.test(text.slice(0, 400))) return;
	268	+ try {
	269	+ const f = parseFeed(text, o.meta.finalUrl);
	270	+ if (!f.items.length) return rejected.push({ url, reason: "empty feed" });
	271	+ const dates = f.items.map((i) => (i.publishedAt ? new Date(i.publishedAt).getTime() : NaN)).filter((t) => !Number.isNaN(t));
	272	+ const dated = dates.length / f.items.length;
	273	+ const itemsPerDay = itemsPerDayOf(dates);
	274	+ const finalUrl = o.meta.finalUrl \|\| url;
	275	+ const kind = feedKind(finalUrl, f.title);
	276	+ const keys = f.items.map((i) => i.key).sort();
	277	+ const stale = dates.length && Math.max(...dates) < Date.now() - 3 * 365 * 86400e3;
	278	+ if (stale) return rejected.push({ url, reason: "feed abandoned (> 3 years)" });
	279	+ feeds.push({
	280	+ url: finalUrl,
	281	+ type: f.kind === "atom" ? "ATOM" : "RSS",
	282	+ connector: "rss",
	283	+ kind,
	284	+ name: feedName(kind, finalUrl),
	285	+ config: {},
	286	+ suggestedTier: itemsPerDay !== null && itemsPerDay < 0.05 ? "C" : "B",
	287	+ evidence: `${alternates.includes(url) ? "link[rel=alternate]" : "well-known path"} · ${f.items.length} items${itemsPerDay !== null ? ` · ${itemsPerDay}/day` : ""}`,
	288	+ value: 0.75 + 0.15 * dated + (kind === "news" \|\| kind === "press" ? 0.1 : kind === "security" \|\| kind === "changelog" ? 0.08 : 0),
	289	+ itemCount: f.items.length,
	290	+ itemsPerDay,
	291	+ firstPartyConfidence: isOwn(finalUrl) ? 1 : 0.7,
	292	+ fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs },
	293	+ fingerprint: sha256(keys.join("\n")),
	294	+ itemKeys: keys,
	295	+ title: f.title,
	296	+ });
	297	+ } catch {
	298	+ // not a feed
	299	+ }
	300	+ });
	301	+ // De-duplicate feeds sharing ≥ 70 % of their items (rss + atom of the same channel, /feed vs /rss.xml).
	302	+ feeds.sort((a, b) => b.value - a.value \|\| (b.itemCount ?? 0) - (a.itemCount ?? 0));
	303	+ const keptFeeds: DeepCandidate[] = [];
	304	+ for (const f of feeds) {
	305	+ const dup = keptFeeds.find((k) => k.itemKeys && f.itemKeys && jaccard(new Set(k.itemKeys), new Set(f.itemKeys)) >= 0.7);
	306	+ if (dup) rejected.push({ url: f.url, reason: `duplicate of ${dup.url}` });
	307	+ else keptFeeds.push(f);
	308	+ }
	309	+ for (const f of keptFeeds) add(f);
	310	+
	311	+ // ---- 5. sitemaps -------------------------------------------------------------------------------------------
	312	+ const sitemapCandidates = [...new Set([...sitemapsFromRobots.filter((u) => !/image\|video/i.test(u)).sort((a, b) => Number(/news/i.test(b)) - Number(/news/i.test(a))).slice(0, 6), ...SITEMAP_PATHS.map((p) => base + p)])];
	313	+ const sitemaps: DeepCandidate[] = [];
	314	+ await parallel(sitemapCandidates, 3, async (url) => {
	315	+ if (!budgetLeft() \|\| sitemaps.length >= 3) return;
	316	+ const o = await get(url, { timeoutMs: 15_000, maxBytes: 20 * 1024 * 1024, accept: "application/xml, text/xml, text/plain;q=0.8, /;q=0.5" });
	317	+ if (!o.body \|\| o.meta.status !== 200) return;
	318	+ const text = o.body.toString("utf8");
	319	+ if (/^\s*<!doctype html\|<html/i.test(text.slice(0, 400))) return;
	320	+ try {
	321	+ const s = parseSitemap(text);
	322	+ const n = s.kind === "sitemapindex" ? s.children.length : s.entries.length;
	323	+ if (!n) return;
	324	+ const lastmods = s.entries.filter((e) => e.lastmod).length;
	325	+ const isNews = /news/i.test(url) \|\| s.entries.some((e) => e.publishedAt);
	326	+ sitemaps.push({ url: o.meta.finalUrl \|\| url, type: "SITEMAP", connector: "sitemap", kind: "sitemap", name: isNews ? "news sitemap" : "sitemap", config: { maxChildren: 4, maxUrls: 3000 }, suggestedTier: "B", evidence: `${sitemapsFromRobots.includes(url) ? "robots.txt" : "well-known path"} · ${n} ${s.kind === "sitemapindex" ? "children" : "urls"}`, value: 0.45 + (s.entries.length ? 0.3 * (lastmods / s.entries.length) : 0.15) + (isNews ? 0.2 : 0), itemCount: n, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
	327	+ } catch {
	328	+ // not a sitemap
	329	+ }
	330	+ });
	331	+ sitemaps.sort((a, b) => b.value - a.value);
	332	+ if (sitemaps[0]) add(sitemaps[0]);
	333	+
	334	+ // ---- 6. status pages ----------------------------------------------------------------------------------------
	335	+ const statusHosts = new Set<string>();
	336	+ if (hints.status_url) {
	337	+ try {
	338	+ statusHosts.add(new URL(hints.status_url).origin);
	339	+ } catch {
	340	+ // ignore
	341	+ }
	342	+ }
	343	+ for (const s of statusLinks) {
	344	+ try {
	345	+ statusHosts.add(new URL(s).origin);
	346	+ } catch {
	347	+ // ignore
	348	+ }
	349	+ }
	350	+ for (const s of subHosts) if (s.kind === "status") statusHosts.add(`https://${s.host}`);
	351	+ let statusFound = false;
	352	+ for (const root of statusHosts) {
	353	+ if (statusFound \|\| !budgetLeft()) break;
	354	+ for (const [path, connector] of [["/api/v2/summary.json", "statuspage"], ["/summary.json", "statusjson"], ["/api/v1/summary", "statusjson"]] as const) {
	355	+ if (statusFound \|\| !budgetLeft()) break;
	356	+ const o = await get(root + path, { timeoutMs: 12_000, accept: "application/json" });
	357	+ if (!o.body \|\| o.meta.status !== 200) continue;
	358	+ const text = o.body.toString("utf8");
	359	+ try {
	360	+ const j = JSON.parse(text) as Record<string, unknown>;
	361	+ if (connector === "statuspage" && Array.isArray(j.incidents) && (j.components \|\| j.status)) {
	362	+ add({ url: root + path, type: "STATUSPAGE", connector: "statuspage", kind: "status", name: "status", config: {}, suggestedTier: "S", evidence: `Atlassian Statuspage (${new URL(root).host})`, value: 0.95, itemsPerDay: null, firstPartyConfidence: 0.9, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
	363	+ statusFound = true;
	364	+ } else if (connector === "statusjson") {
	365	+ const flavor = detectFlavor(j);
	366	+ if (flavor) {
	367	+ add({ url: root + path, type: "STATUSPAGE", connector: "statusjson", kind: "status", name: "status", config: { flavor }, suggestedTier: "S", evidence: `${flavor} status JSON (${new URL(root).host})`, value: 0.93, itemsPerDay: null, firstPartyConfidence: 0.9, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
	368	+ statusFound = true;
	369	+ }
	370	+ }
	371	+ } catch {
	372	+ // not JSON
	373	+ }
	374	+ }
	375	+ }
	376	+
	377	+ // ---- 7. pages (server-rendered sections) --------------------------------------------------------------------
	378	+ if (opts.probePages !== false && !blocked) {
	379	+ const pageJobs: { kind: CandidateKind; label: string; tier: Tier; value: number; urls: string[]; max: number }[] = [];
	380	+ for (const pk of PAGE_KINDS) {
	381	+ const linked = pageLinks.get(pk.kind) ?? [];
	382	+ // feeds already cover news/press/blog for this org → the HTML index page adds little
	383	+ if ((pk.kind === "news" \|\| pk.kind === "press") && keptFeeds.some((f) => f.kind === pk.kind \|\| f.kind === "news")) continue;
	384	+ const shallowFirst = (a: string, b: string): number => safePath(a).split("/").length - safePath(b).split("/").length \|\| a.length - b.length;
	385	+ const urls = [...new Set([...linked.filter((u) => pk.hrefRe.test(safePath(u)) \|\| linked.length <= 2).sort(shallowFirst), ...pk.paths.map((p) => base + p)])].slice(0, 4);
	386	+ if (!urls.length) continue;
	387	+ pageJobs.push({ kind: pk.kind, label: pk.label, tier: pk.tier, value: pk.value, urls, max: pk.max });
	388	+ }
	389	+ const pageCanon: { url: string; sh: Set<string>; hash: string }[] = [];
	390	+ const probed = new Set<string>();
	391	+ await parallel(pageJobs, 3, async (job) => {
	392	+ let kept = 0;
	393	+ for (const url of job.urls) {
	394	+ if (kept >= job.max \|\| !budgetLeft()) break;
	395	+ const pk = url.replace(/\/$/, "");
	396	+ if (probed.has(pk) \|\| found.has(pk)) continue;
	397	+ probed.add(pk);
	398	+ const o = await get(url, { timeoutMs: 15_000, maxBytes: 3 * 1024 * 1024, accept: "text/html, application/xhtml+xml;q=0.9, /;q=0.5" });
	399	+ if (o.meta.status !== 200 \|\| !o.body) continue;
	400	+ const finalPath = safePath(o.meta.finalUrl);
	401	+ if (finalPath === "/" \|\| finalPath === "") continue; // redirected home
	402	+ if (!isOwn(o.meta.finalUrl \|\| url)) continue;
	403	+ if (!/text\/html\|xhtml/i.test(o.meta.contentType ?? "") && /^\s*[{[]/.test(o.body.toString("utf8").slice(0, 50))) continue;
	404	+ const c = canonicalizeHtml(o.body.toString("utf8"), o.meta.finalUrl \|\| url);
	405	+ if (c.text.length < 300) {
	406	+ rejected.push({ url, reason: `thin (${c.text.length} chars)` });
	407	+ continue;
	408	+ }
	409	+ const sh = shingles(c.text);
	410	+ if (homeCanon && (c.canonicalHash === homeCanon.canonicalHash \|\| (homeShingles && jaccard(sh, homeShingles) > 0.8))) {
	411	+ rejected.push({ url, reason: "same as homepage" });
	412	+ continue;
	413	+ }
	414	+ const dup = pageCanon.find((p) => p.hash === c.canonicalHash \|\| jaccard(p.sh, sh) > 0.85);
	415	+ if (dup) {
	416	+ rejected.push({ url, reason: `duplicate of ${dup.url}` });
	417	+ continue;
	418	+ }
	419	+ pageCanon.push({ url: o.meta.finalUrl \|\| url, sh, hash: c.canonicalHash });
	420	+ const isSecurityTxt = /security\.txt$/.test(url);
	421	+ add({ url: o.meta.finalUrl \|\| url, type: "HTML", connector: "http", kind: job.kind, name: isSecurityTxt ? "security.txt" : job.label, config: {}, suggestedTier: isSecurityTxt ? "D" : job.tier, evidence: `${pageLinks.get(job.kind)?.includes(url) ? "linked from homepage" : "well-known path"} · GET 200 · ${c.text.length} chars · distinct from homepage`, value: isSecurityTxt ? 0.3 : job.value, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs }, fingerprint: c.canonicalHash, textShingles: sh, title: c.title ?? undefined });
	422	+ kept++;
	423	+ }
	424	+ });
	425	+ }
	426	+
	427	+ // ---- 8. GitHub organization → repositories' releases ----------------------------------------------------------
	428	+ const repos = new Set<string>(hints.github_repos ?? []);
	429	+ for (const org of [...githubOrgs].slice(0, 1)) {
	430	+ if (repos.size >= 4 \|\| !budgetLeft()) break;
	431	+ const token = process.env.GITHUB_TOKEN;
	432	+ if (token) {
	433	+ const o = await get(`https://api.github.com/search/repositories?q=org:${encodeURIComponent(org)}+fork:false&sort=stars&order=desc&per_page=5`, { accept: "application/vnd.github+json", headers: { authorization: `Bearer ${token}`, "x-github-api-version": "2022-11-28" } });
	434	+ if (o.body && o.meta.status === 200) {
	435	+ try {
	436	+ const j = JSON.parse(o.body.toString("utf8")) as { items?: { full_name: string; archived?: boolean; stargazers_count?: number }[] };
	437	+ for (const r of j.items ?? []) if (!r.archived && (r.stargazers_count ?? 0) >= 50) repos.add(r.full_name);
	438	+ } catch {
	439	+ // ignore
	440	+ }
	441	+ }
	442	+ } else {
	443	+ // Unauthenticated: the organization page lists pinned / popular repositories server-side.
	444	+ const o = await get(`https://github.com/${org}`, { timeoutMs: 15_000, maxBytes: 3 * 1024 * 1024 });
	445	+ if (o.body && o.meta.status === 200) {
	446	+ const html = o.body.toString("utf8");
	447	+ const counts = new Map<string, number>();
	448	+ for (const m of html.matchAll(new RegExp(`href="/${org.replace(/[.*+?^${}()\|[\]\\]/g, "\\$&")}/([A-Za-z0-9_.-]+)"`, "g"))) {
	449	+ const r = m[1]!;
	450	+ if (/^(followers\|following\|repositories\|projects\|packages\|people\|teams\|sponsoring\|discussions\|orgs\|\.github)$/i.test(r)) continue;
	451	+ counts.set(r, (counts.get(r) ?? 0) + 1);
	452	+ }
	453	+ const pinned = [...html.matchAll(/class="[^"]pinned-item-list-item[^"]"[\s\S]{0,1500}?href="\/([^/"]+)\/([^/"]+)"/g)].map((m) => `${m[1]}/${m[2]}`).filter((f) => f.toLowerCase().startsWith(org.toLowerCase() + "/"));
	454	+ for (const f of pinned) repos.add(f);
	455	+ for (const [r] of [...counts.entries()].sort((a, b) => b[1] - a[1]).slice(0, 6)) if (repos.size < 6) repos.add(`${org}/${r}`);
	456	+ }
	457	+ }
	458	+ }
	459	+ const repoList = [...repos].slice(0, 6);
	460	+ let releasesKept = 0;
	461	+ await parallel(repoList, 3, async (full) => {
	462	+ if (releasesKept >= 4 \|\| !budgetLeft()) return;
	463	+ for (const kind of ["releases", "tags"] as const) {
	464	+ const url = `https://github.com/${full}/${kind}.atom`;
	465	+ const o = await get(url, { timeoutMs: 12_000, accept: "application/atom+xml, application/xml;q=0.9" });
	466	+ if (!o.body \|\| o.meta.status !== 200) {
	467	+ if (o.meta.status === 404) return;
	468	+ continue;
	469	+ }
	470	+ try {
	471	+ const f = parseFeed(o.body.toString("utf8"), url);
	472	+ if (!f.items.length) continue;
	473	+ const dates = f.items.map((i) => (i.publishedAt ? new Date(i.publishedAt).getTime() : NaN)).filter((t) => !Number.isNaN(t));
	474	+ if (dates.length && Math.max(...dates) < Date.now() - 2 * 365 * 86400e3) return rejected.push({ url, reason: "repository inactive (> 2 years)" });
	475	+ add({ url, type: "GITHUB_RELEASE", connector: "github", kind: "releases", name: `${full.split("/")[1]} ${kind}`, config: { repo: full, kind }, suggestedTier: "B", evidence: `GitHub ${kind}.atom · ${f.items.length} items`, value: kind === "releases" ? 0.8 : 0.65, itemCount: f.items.length, itemsPerDay: null, firstPartyConfidence: 0.9, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs }, fingerprint: sha256(f.items.map((i) => i.key).sort().join("\n")) });
	476	+ releasesKept++;
	477	+ return;
	478	+ } catch {
	479	+ // not a feed
	480	+ }
	481	+ }
	482	+ });
	483	+
	484	+ // ---- 9. Hugging Face models ------------------------------------------------------------------------------------
	485	+ for (const author of [...hfAuthors].slice(0, 2)) {
	486	+ if (!budgetLeft()) break;
	487	+ const url = `https://huggingface.co/api/models?author=${encodeURIComponent(author)}&sort=lastModified&direction=-1&limit=50`;
	488	+ const o = await get(url, { accept: "application/json" });
	489	+ if (!o.body \|\| o.meta.status !== 200) continue;
	490	+ try {
	491	+ const j = JSON.parse(o.body.toString("utf8")) as unknown[];
	492	+ if (Array.isArray(j) && j.length) add({ url, type: "REST_API", connector: "jsonlist", kind: "models", name: "hugging face models", config: { keyField: "id", titleField: "id", dateField: "lastModified", compareFields: ["pipeline_tag", "library_name"], maxItems: 50 }, suggestedTier: "B", evidence: `Hugging Face author ${author} · ${j.length} models`, value: 0.85, itemCount: j.length, itemsPerDay: null, firstPartyConfidence: 0.9, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
	493	+ } catch {
	494	+ // ignore
	495	+ }
	496	+ }
	497	+
	498	+ // ---- 10. EDGAR filings -------------------------------------------------------------------------------------------
	499	+ if (hints.cik !== undefined && budgetLeft()) {
	500	+ const cik = String(hints.cik).replace(/\D/g, "").padStart(10, "0");
	501	+ const url = `https://data.sec.gov/submissions/CIK${cik}.json`;
	502	+ const ua = process.env.WS_EDGAR_USER_AGENT ?? "WebSensor (contact@websensor.io)";
	503	+ const o = await get(url, { accept: "application/json", userAgent: ua, headers: { "user-agent": ua }, timeoutMs: 30_000, maxBytes: 30 * 1024 * 1024 });
	504	+ if (o.body && o.meta.status === 200 && /"filings"/.test(o.body.toString("utf8").slice(0, 200_000))) {
	505	+ add({ url, type: "REST_API", connector: "edgar", kind: "filings", name: "edgar filings", config: { forms: ["8-K", "10-K", "10-Q", "6-K", "20-F", "40-F", "S-1", "SC 13D", "DEF 14A"] }, suggestedTier: "B", evidence: `SEC EDGAR submissions CIK ${cik}`, value: 0.9, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
	506	+ } else rejected.push({ url, reason: `edgar ${o.meta.status}` });
	507	+ }
	508	+
	509	+ // ---- 11. OpenAPI documents ---------------------------------------------------------------------------------------
	510	+ if (opts.probeOpenApi !== false && (subHosts.some((s) => s.kind === "docs" \|\| s.kind === "api") \|\| pageLinks.has("docs")) && budgetLeft()) {
	511	+ const roots = [base, ...subHosts.filter((s) => s.kind === "docs" \|\| s.kind === "api").map((s) => `https://${s.host}`)].slice(0, 3);
	512	+ let openapiFound = false;
	513	+ for (const root of roots) {
	514	+ if (openapiFound) break;
	515	+ for (const p of OPENAPI_PATHS.slice(0, 7)) {
	516	+ if (openapiFound \|\| !budgetLeft()) break;
	517	+ const o = await get(root + p, { timeoutMs: 15_000, maxBytes: 8 * 1024 * 1024, accept: "application/json, application/yaml, /;q=0.5" });
	518	+ if (!o.body \|\| o.meta.status !== 200) continue;
	519	+ const text = o.body.toString("utf8").slice(0, 4000);
	520	+ if (/^\s[{[]/.test(text) && /"(openapi\|swagger)"\s:/.test(text)) {
	521	+ add({ url: root + p, type: "JSON", connector: "openapi", kind: "api", name: "openapi", config: {}, suggestedTier: "C", evidence: `OpenAPI document (${new URL(root).host})`, value: 0.7, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: o.meta.contentLength, ms: o.meta.durationMs } });
	522	+ openapiFound = true;
	523	+ }
	524	+ }
	525	+ }
	526	+ }
	527	+
	528	+ // ---- 12. explicit hint URLs (validated like everything else) --------------------------------------------------------
	529	+ for (const h of hints.urls ?? []) {
	530	+ if (!budgetLeft()) break;
	531	+ const known = h.url.replace(/\/$/, "");
	532	+ if (found.has(known)) continue;
	533	+ const o = await get(h.url, { timeoutMs: 15_000, maxBytes: 6 * 1024 * 1024 });
	534	+ if (!o.body \|\| o.meta.status !== 200) {
	535	+ rejected.push({ url: h.url, reason: `hint ${o.meta.status \|\| o.error?.code}` });
	536	+ continue;
	537	+ }
	538	+ const text = o.body.toString("utf8");
	539	+ const kind = (h.kind ?? "other") as CandidateKind;
	540	+ const tier = h.tier ?? (kind === "status" ? "S" : kind === "pricing" \|\| kind === "legal" \|\| kind === "ir" ? "C" : "B");
	541	+ const cost = { bytes: o.meta.contentLength, ms: o.meta.durationMs };
	542	+ if (h.connector) {
	543	+ add({ url: h.url, type: (h.type ?? (h.connector === "rss" ? "RSS" : h.connector === "jsonlist" \|\| h.connector === "edgar" ? "REST_API" : h.connector === "statuspage" \|\| h.connector === "statusjson" ? "STATUSPAGE" : h.connector === "sitemap" ? "SITEMAP" : "HTML")) as SensorType, connector: h.connector, kind, name: h.name ?? kind, config: h.config ?? {}, suggestedTier: tier, evidence: "explicit hint · GET 200", value: 0.8, itemsPerDay: null, firstPartyConfidence: isOwn(h.url) ? 1 : 0.8, fetchCost: cost });
	544	+ continue;
	545	+ }
	546	+ // Sniff: feed → rss; JSON → http/json; HTML → http
	547	+ if (!/^\s*<!doctype html\|<html/i.test(text.slice(0, 400)) && /<(rss\|feed\|rdf:RDF)\b/i.test(text.slice(0, 2000))) {
	548	+ try {
	549	+ const f = parseFeed(text, o.meta.finalUrl);
	550	+ if (f.items.length) {
	551	+ add({ url: o.meta.finalUrl \|\| h.url, type: f.kind === "atom" ? "ATOM" : "RSS", connector: "rss", kind, name: h.name ?? feedName(kind, h.url), config: {}, suggestedTier: tier, evidence: `explicit hint · feed · ${f.items.length} items`, value: 0.85, itemCount: f.items.length, itemsPerDay: null, firstPartyConfidence: isOwn(h.url) ? 1 : 0.8, fetchCost: cost, fingerprint: sha256(f.items.map((i) => i.key).sort().join("\n")), itemKeys: f.items.map((i) => i.key).sort() });
	552	+ continue;
	553	+ }
	554	+ } catch {
	555	+ // fallthrough
	556	+ }
	557	+ }
	558	+ if (/^\s*[{[]/.test(text.slice(0, 50))) {
	559	+ add({ url: h.url, type: "JSON", connector: "http", kind, name: h.name ?? kind, config: h.config ?? {}, suggestedTier: tier, evidence: "explicit hint · JSON 200", value: 0.65, itemsPerDay: null, firstPartyConfidence: isOwn(h.url) ? 1 : 0.8, fetchCost: cost });
	560	+ continue;
	561	+ }
	562	+ const c = canonicalizeHtml(text, o.meta.finalUrl \|\| h.url);
	563	+ if (c.text.length < 300) {
	564	+ rejected.push({ url: h.url, reason: `hint thin (${c.text.length} chars)` });
	565	+ continue;
	566	+ }
	567	+ add({ url: o.meta.finalUrl \|\| h.url, type: "HTML", connector: "http", kind, name: h.name ?? PAGE_KINDS.find((p) => p.kind === kind)?.label ?? kind, config: h.config ?? {}, suggestedTier: tier, evidence: `explicit hint · GET 200 · ${c.text.length} chars`, value: 0.7, itemsPerDay: null, firstPartyConfidence: isOwn(h.url) ? 1 : 0.8, fetchCost: cost, fingerprint: c.canonicalHash, title: c.title ?? undefined });
	568	+ }
	569	+
	570	+ // ---- 13. web posture (no network here: validated in shadow by the dns/tls connectors) ------------------------------
	571	+ if (opts.probePosture \|\| hints.posture) {
	572	+ const rd = registrableDomain(domain);
	573	+ add({ url: `dns://${rd}`, type: "DNS", connector: "dns", kind: "posture", name: "dns", config: {}, suggestedTier: "D", evidence: "posture", value: 0.3, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: 0, ms: 0 } });
	574	+ add({ url: `tls://${new URL(base).hostname}`, type: "TLS", connector: "tls", kind: "posture", name: "tls certificate", config: {}, suggestedTier: "D", evidence: "posture", value: 0.3, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: 0, ms: 0 } });
	575	+ if (robotsOk) add({ url: `https://${domain}/robots.txt`, type: "HTML", connector: "http", kind: "posture", name: "robots.txt", config: {}, suggestedTier: "D", evidence: robotsMentionsAi ? "robots.txt names AI crawlers" : "robots.txt", value: robotsMentionsAi ? 0.45 : 0.3, itemsPerDay: null, firstPartyConfidence: 1, fetchCost: { bytes: robots?.meta.contentLength ?? 0, ms: robots?.meta.durationMs ?? 0 } });
	576	+ }
	577	+
	578	+ const candidates = [...found.values()].sort((a, b) => b.value - a.value);
	579	+ // Strip the heavy de-dup helpers before returning (they are not serializable and no longer needed).
	580	+ for (const c of candidates) {
	581	+ delete c.textShingles;
	582	+ if (c.itemKeys && c.itemKeys.length > 60) c.itemKeys = c.itemKeys.slice(0, 60);
	583	+ }
	584	+ return { domain, origin: base, canonicalDomain, homeStatus, blocked, requests, candidates, rejected, notes, durationMs: Date.now() - started };
	585	+}
	586	+
	587	+// ------------------------------------------------------------------------------------------------
	588	+
	589	+/** Expected publication rate from item dates: the median gap between consecutive items (robust to one bogus date). */
	590	+export function itemsPerDayOf(dates: number[]): number \| null {
	591	+ const d = [...dates].filter((t) => Number.isFinite(t)).sort((a, b) => b - a).slice(0, 30);
	592	+ if (d.length < 3) return null;
	593	+ const gaps = d.slice(1).map((t, i) => (d[i]! - t) / 86400e3).filter((g) => g >= 0).sort((a, b) => a - b);
	594	+ if (!gaps.length) return null;
	595	+ const median = gaps[Math.floor(gaps.length / 2)]!;
	596	+ if (median <= 0) return 50;
	597	+ return Math.round((1 / median) * 1000) / 1000;
	598	+}
	599	+
	600	+function feedKind(url: string, title?: string): CandidateKind {
	601	+ const s = `${url} ${title ?? ""}`.toLowerCase();
	602	+ if (/press\|communiqu\|pressemitteilung\|prensa\|news-release\|newsrelease/.test(s)) return "press";
	603	+ if (/investor\|\/ir\/\|ir\.\|sec-filing\|filings/.test(s)) return "ir";
	604	+ if (/security\|advisor\|psirt\|bulletin\|vulnerab\|cve/.test(s)) return "security";
	605	+ if (/changelog\|release-notes\|releasenotes\|releases\|whats-new\|updates/.test(s)) return "changelog";
	606	+ if (/blog\|engineering\|research\|insights\|stories/.test(s)) return "blog";
	607	+ if (/news\|actualit\|nouvelles\|aktuell\|noticias\|announcement/.test(s)) return "news";
	608	+ if (/jobs\|careers/.test(s)) return "careers";
	609	+ return "news";
	610	+}
	611	+
	612	+function feedName(kind: CandidateKind, url: string): string {
	613	+ switch (kind) {
	614	+ case "press":
	615	+ return "press releases feed";
	616	+ case "ir":
	617	+ return "investor news feed";
	618	+ case "security":
	619	+ return "security feed";
	620	+ case "changelog":
	621	+ return "changelog feed";
	622	+ case "blog":
	623	+ return /engineering/i.test(url) ? "engineering blog feed" : /research/i.test(url) ? "research feed" : "blog feed";
	624	+ case "careers":
	625	+ return "jobs feed";
	626	+ default:
	627	+ return "news feed";
	628	+ }
	629	+}
	630	+
	631	+function safePath(u: string): string {
	632	+ try {
	633	+ return new URL(u).pathname.replace(/\/$/, "");
	634	+ } catch {
	635	+ return "";
	636	+ }
	637	+}
	638	+
	639	+function safeAbs(h: string, base: string): string \| null {
	640	+ try {
	641	+ const u = new URL(h, base);
	642	+ if (!u.protocol.startsWith("http")) return null;
	643	+ u.hash = "";
	644	+ return u.toString();
	645	+ } catch {
	646	+ return null;
	647	+ }
	648	+}
	649	+
	650	+async function resolves(host: string): Promise<boolean> {
	651	+ try {
	652	+ const r = await dns.lookup(host, { all: true });
	653	+ return r.length > 0;
	654	+ } catch {
	655	+ return false;
	656	+ }
	657	+}
	658	+
	659	+async function parallel<T>(items: T[], limit: number, fn: (item: T) => Promise<unknown>): Promise<void> {
	660	+ let i = 0;
	661	+ const workers = Array.from({ length: Math.min(limit, items.length) }, async () => {
	662	+ while (i < items.length) {
	663	+ const item = items[i++]!;
	664	+ try {
	665	+ await fn(item);
	666	+ } catch {
	667	+ // best effort
	668	+ }
	669	+ }
	670	+ });
	671	+ await Promise.all(workers);
	672	+}

modified packages/connectors/src/index.ts +1 −0

		`@@ -66,3 +66,4 @@ export function getConnector(key: string): WebSensorConnector {`
66	66	export function listConnectors(): WebSensorConnector[] {
67	67	return Object.values(REGISTRY);
68	68	}
	69	+export * from "./discovery-deep";

added packages/core/src/coverage-schema.ts +150 −0

		`@@ -0,0 +1,150 @@`
	1	+import { z } from "zod";
	2	+import { TIERS } from "./taxonomy";
	3	+
	4	+/**
	5	+ * Coverage universes (`config/coverage/.yaml`). A universe is the list of organizations WebSensor should*
	6	+ * observe for a sector — the denominator of the Global Observation Coverage Score. The same files feed the
	7	+ * Source Factory: every member without a monitored domain becomes an expansion seed.
	8	+ *
	9	+ * One file per sector:
	10	+ *
	11	+ * sector: ai
	12	+ * label: Artificial intelligence
	13	+ * weight: 1.2 # weight of the sector in the global score (default 1)
	14	+ * categories: [ai, technology] # default categories applied to members (registry vocabulary)
	15	+ * tier: B # default tier for factory-created sources
	16	+ * universes:
	17	+ * - key: frontier-labs
	18	+ * label: Frontier labs & model providers
	19	+ * members:
	20	+ * - { name: OpenAI, domain: openai.com, country: US, importance: 3, hints: { github_org: openai, hf_author: openai } }
	21	+ */
	22	+export const memberHintUrlSchema = z.object({
	23	+ url: z.string().url(),
	24	+ /** what the page is: news \| press \| blog \| ir \| filings \| changelog \| releases \| security \| pricing \| legal \| careers \| docs \| api \| status \| models \| data \| other */
	25	+ kind: z.string().default("other"),
	26	+ connector: z.string().optional(),
	27	+ type: z.string().optional(),
	28	+ tier: z.enum(TIERS).optional(),
	29	+ config: z.record(z.string(), z.unknown()).optional(),
	30	+ name: z.string().optional(),
	31	+});
	32	+
	33	+export const memberHintsSchema = z.object({
	34	+ /** SEC CIK (digits, any length) → EDGAR submissions sensor */
	35	+ cik: z.union([z.string(), z.number()]).optional(),
	36	+ ticker: z.string().optional(),
	37	+ exchange: z.string().optional(),
	38	+ /** GitHub organization → most active repositories' releases */
	39	+ github_org: z.string().optional(),
	40	+ /** explicit repositories `owner/name` */
	41	+ github_repos: z.array(z.string()).optional(),
	42	+ /** Hugging Face author/organization → model list sensor */
	43	+ hf_author: z.string().optional(),
	44	+ /** status page root (Atlassian Statuspage, Instatus, incident.io, Status.io) */
	45	+ status_url: z.string().url().optional(),
	46	+ /** additional hosts that belong to the organization (blog.example.org, investors.example.org…) */
	47	+ hosts: z.array(z.string()).optional(),
	48	+ /** explicit candidate URLs (validated by the factory before becoming sensors) */
	49	+ urls: z.array(memberHintUrlSchema).optional(),
	50	+ /** probe robots/TLS/DNS/headers posture sensors (default: only for importance 3) */
	51	+ posture: z.boolean().optional(),
	52	+ /** Wikidata QID, for provenance */
	53	+ wikidata: z.string().optional(),
	54	+});
	55	+
	56	+export const coverageMemberSchema = z.object({
	57	+ name: z.string().min(1),
	58	+ /** bare registrable domain, lower-case, no scheme, no `www.` */
	59	+ domain: z
	60	+ .string()
	61	+ .min(3)
	62	+ .transform((d) => d.trim().toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "")),
	63	+ country: z.string().regex(/^[A-Z]{2,3}$/).optional(),
	64	+ language: z.string().regex(/^[a-z]{2}$/).optional(),
	65	+ /** 1 = long tail · 2 = notable (default) · 3 = systemically important */
	66	+ importance: z.union([z.literal(1), z.literal(2), z.literal(3)]).default(2),
	67	+ aliases: z.array(z.string()).default([]),
	68	+ categories: z.array(z.string()).optional(),
	69	+ tier: z.enum(TIERS).optional(),
	70	+ first_party: z.boolean().optional(),
	71	+ hints: memberHintsSchema.default({}),
	72	+ note: z.string().optional(),
	73	+});
	74	+export type CoverageMember = z.infer<typeof coverageMemberSchema>;
	75	+
	76	+export const coverageUniverseSchema = z.object({
	77	+ key: z.string().regex(/^[a-z0-9][a-z0-9-]*$/),
	78	+ label: z.string(),
	79	+ description: z.string().optional(),
	80	+ /** provenance of the list (URL of the index constituents page, Wikidata query…) */
	81	+ provenance: z.string().optional(),
	82	+ members: z.array(coverageMemberSchema),
	83	+});
	84	+export type CoverageUniverse = z.infer<typeof coverageUniverseSchema>;
	85	+
	86	+export const coverageSectorSchema = z.object({
	87	+ sector: z.string().regex(/^[a-z0-9][a-z0-9-]*$/),
	88	+ label: z.string(),
	89	+ description: z.string().optional(),
	90	+ weight: z.number().positive().default(1),
	91	+ categories: z.array(z.string()).default([]),
	92	+ tier: z.enum(TIERS).default("B"),
	93	+ universes: z.array(coverageUniverseSchema),
	94	+});
	95	+export type CoverageSector = z.infer<typeof coverageSectorSchema>;
	96	+
	97	+/**
	98	+ * Registrable domain (eTLD+1) with a compact public-suffix list covering the multi-label suffixes that
	99	+ * matter for official sources (governments, universities, companies). Good enough to match a sensor host
	100	+ * (`ir.example.com`, `www.gov.uk`) to a universe member; not a full PSL.
	101	+ */
	102	+const MULTI_SUFFIX = new Set([
	103	+ "co.uk", "org.uk", "gov.uk", "ac.uk", "nhs.uk", "police.uk", "ltd.uk", "plc.uk", "me.uk", "net.uk", "sch.uk",
	104	+ "com.au", "net.au", "org.au", "gov.au", "edu.au", "id.au", "asn.au",
	105	+ "gc.ca", "qc.ca", "on.ca", "bc.ca", "ab.ca", "mb.ca", "sk.ca", "ns.ca", "nb.ca", "nl.ca", "pe.ca", "nt.ca", "nu.ca", "yk.ca", "gouv.qc.ca",
	106	+ "co.jp", "ne.jp", "or.jp", "go.jp", "ac.jp", "ad.jp", "lg.jp", "ed.jp",
	107	+ "co.kr", "go.kr", "or.kr", "ac.kr", "re.kr", "ne.kr",
	108	+ "com.cn", "gov.cn", "org.cn", "edu.cn", "net.cn", "ac.cn",
	109	+ "com.hk", "gov.hk", "org.hk", "edu.hk",
	110	+ "com.tw", "gov.tw", "org.tw", "edu.tw",
	111	+ "com.sg", "gov.sg", "org.sg", "edu.sg",
	112	+ "co.in", "gov.in", "nic.in", "org.in", "ac.in", "net.in", "res.in", "edu.in",
	113	+ "com.br", "gov.br", "org.br", "edu.br", "net.br", "leg.br", "jus.br", "mil.br",
	114	+ "com.mx", "gob.mx", "org.mx", "edu.mx",
	115	+ "com.ar", "gob.ar", "gov.ar", "org.ar", "edu.ar",
	116	+ "gob.cl", "gov.cl", "cl", "gob.pe", "gob.es", "gov.co", "gob.ec", "gob.bo", "gub.uy", "gob.pa", "gob.do", "gob.gt", "gob.hn", "gob.sv", "gob.ni", "gob.cu", "gob.ve", "gob.py",
	117	+ "co.za", "gov.za", "org.za", "ac.za", "net.za",
	118	+ "co.nz", "govt.nz", "org.nz", "ac.nz", "net.nz",
	119	+ "com.tr", "gov.tr", "org.tr", "edu.tr",
	120	+ "co.il", "gov.il", "org.il", "ac.il", "muni.il",
	121	+ "gov.ae", "ac.ae", "co.ae",
	122	+ "gov.sa", "com.sa", "edu.sa", "org.sa",
	123	+ "gov.qa", "com.qa", "gov.kw", "gov.bh", "gov.om", "gov.jo", "gov.lb", "gov.eg", "com.eg", "gov.ma", "gov.dz", "gov.tn",
	124	+ "gov.ng", "com.ng", "org.ng", "go.ke", "co.ke", "or.ke", "ac.ke", "gov.gh", "com.gh", "gov.et", "gov.rw", "go.tz", "co.tz", "go.ug", "co.ug", "gov.zm", "gov.zw", "gov.bw", "gov.na", "gov.mz", "gov.sn", "gouv.sn", "gouv.ci", "gouv.bf", "gouv.ml", "gouv.ne", "gouv.tg", "gouv.bj", "gouv.cm", "gov.mu",
	125	+ "com.my", "gov.my", "org.my", "edu.my", "go.th", "co.th", "or.th", "ac.th", "gov.vn", "com.vn", "edu.vn", "gov.ph", "com.ph", "org.ph", "edu.ph", "go.id", "co.id", "or.id", "ac.id", "gov.bd", "com.bd", "gov.pk", "com.pk", "edu.pk", "gov.lk", "gov.np", "gov.kh", "gov.la", "gov.mm", "gov.bn", "gov.mn",
	126	+ "gov.kz", "gov.uz", "gov.kg", "gov.tj", "gov.tm", "gov.az", "gov.ge", "gov.am", "gov.by", "gov.ua", "com.ua", "org.ua", "edu.ua", "gov.md", "gov.ru", "com.ru", "org.ru", "edu.ru", "ac.ru",
	127	+ "gov.pl", "com.pl", "org.pl", "edu.pl", "gov.cz", "gov.sk", "gov.hu", "gov.ro", "gov.bg", "gov.hr", "gov.si", "gov.rs", "gov.ba", "gov.mk", "gov.al", "gov.me", "gov.lv", "gov.lt", "gov.ee", "gov.gr", "gov.cy", "gov.mt", "gov.ie", "gov.pt", "gov.it", "gouv.fr", "gov.lu", "gov.be", "gov.nl", "gov.is", "gov.li",
	128	+ "com.es", "org.es", "edu.es", "gob.es", "com.pt", "edu.pt", "co.it", "edu.it", "com.gr", "edu.gr", "com.pl", "com.ro", "com.tr",
	129	+ "europa.eu",
	130	+ "gov.uk", "parliament.uk", "judiciary.uk", "mod.uk", "nhs.uk", "royal.uk",
	131	+ "state.gov", "mil", "fed.us", "nsn.us", "k12.us", "lib.us", "dni.us", "isa.us", "kids.us",
	132	+ "ac.at", "co.at", "gv.at", "or.at", "ac.be", "co.nl", "ac.nz", "ac.uk", "ac.jp", "edu.au", "edu.sg", "edu.hk",
	133	+ "org.au", "com.au", "co.ke",
	134	+]);
	135	+
	136	+export function registrableDomain(host: string): string {
	137	+ const h = host.toLowerCase().replace(/\.$/, "").replace(/^www\./, "");
	138	+ const parts = h.split(".");
	139	+ if (parts.length <= 2) return h;
	140	+ const last2 = parts.slice(-2).join(".");
	141	+ const last3 = parts.slice(-3).join(".");
	142	+ if (MULTI_SUFFIX.has(last3) && parts.length >= 4) return parts.slice(-4).join(".");
	143	+ if (MULTI_SUFFIX.has(last2)) return last3;
	144	+ return last2;
	145	+}
	146	+
	147	+/** Normalize a member/source domain to its comparison key. */
	148	+export function coverageKey(domain: string): string {
	149	+ return registrableDomain(domain.trim().toLowerCase().replace(/^https?:\/\//, "").replace(/\/.*$/, ""));
	150	+}

modified packages/core/src/index.ts +1 −0

		`@@ -11,3 +11,4 @@ export * from "./types";`
11	11	export * from "./semantic";
12	12	export * from "./search";
13	13	export * from "./registry-schema";
	14	+export * from "./coverage-schema";

added packages/db/migrations/0007_factory.sql +73 −0

		`@@ -0,0 +1,73 @@`
	1	+-- WebSensor 0.3 — Source Factory & coverage (2026-09-13). Additive only.
	2	+
	3	+-- ---- Sources: origin (seed = YAML registry · import = admin import · factory = Source Factory) ----
	4	+alter table sources add column if not exists origin text not null default 'seed';
	5	+alter table sources add column if not exists sector text; -- coverage sector that produced the source
	6	+create index if not exists sources_origin_idx on sources (origin);
	7	+create index if not exists sources_domain_idx on sources (domain);
	8	+
	9	+-- ---- Sensors: SHADOW status (observed, evidence stored, never published) ----------------------
	10	+create index if not exists sensors_status_idx on sensors (status);
	11	+create index if not exists sensors_url_host_idx on sensors ((lower(split_part(split_part(url, '/', 3), ':', 1))));
	12	+
	13	+-- ---- Factory seeds: organizations to expand -------------------------------------------------
	14	+create table if not exists factory_seeds (
	15	+ id text primary key, -- kebab id; becomes the source id when a new source is created
	16	+ name text not null,
	17	+ domain text not null,
	18	+ homepage text,
	19	+ categories text[] not null default '{}',
	20	+ country text,
	21	+ language text,
	22	+ tier text not null default 'B',
	23	+ weight real not null default 1,
	24	+ importance smallint not null default 2, -- 1 long tail · 2 notable · 3 systemic
	25	+ aliases text[] not null default '{}',
	26	+ first_party boolean not null default true,
	27	+ sector text,
	28	+ universe text,
	29	+ hints jsonb not null default '{}',
	30	+ status text not null default 'queued', -- queued \| discovering \| discovered \| blocked \| error \| done
	31	+ attempts integer not null default 0,
	32	+ source_id text, -- existing or created source
	33	+ candidates integer not null default 0,
	34	+ shadow integer not null default 0,
	35	+ accepted integer not null default 0,
	36	+ rejected integer not null default 0,
	37	+ last_error text,
	38	+ discovered_at timestamptz,
	39	+ created_at timestamptz not null default now(),
	40	+ updated_at timestamptz not null default now()
	41	+);
	42	+create index if not exists factory_seeds_status_idx on factory_seeds (status, importance desc, created_at);
	43	+create index if not exists factory_seeds_domain_idx on factory_seeds (domain);
	44	+create index if not exists factory_seeds_sector_idx on factory_seeds (sector);
	45	+
	46	+-- ---- Discovery candidates: full lifecycle (candidate → shadow → accepted \| rejected \| duplicate \| blocked) ----
	47	+alter table discovery_candidates add column if not exists seed_id text;
	48	+alter table discovery_candidates add column if not exists connector text;
	49	+alter table discovery_candidates add column if not exists type text;
	50	+alter table discovery_candidates add column if not exists name text;
	51	+alter table discovery_candidates add column if not exists kind_class text; -- news press blog ir filings changelog releases security pricing legal careers docs api status sitemap models data posture other
	52	+alter table discovery_candidates add column if not exists config jsonb not null default '{}';
	53	+alter table discovery_candidates add column if not exists tier text;
	54	+alter table discovery_candidates add column if not exists score_value real;
	55	+alter table discovery_candidates add column if not exists reason text;
	56	+alter table discovery_candidates add column if not exists shadow_sensor_id text;
	57	+alter table discovery_candidates add column if not exists decided_at timestamptz;
	58	+alter table discovery_candidates add column if not exists updated_at timestamptz not null default now();
	59	+create index if not exists discovery_candidates_status_idx on discovery_candidates (status, score_value desc);
	60	+create index if not exists discovery_candidates_seed_idx on discovery_candidates (seed_id);
	61	+create index if not exists discovery_candidates_shadow_idx on discovery_candidates (shadow_sensor_id) where shadow_sensor_id is not null;
	62	+
	63	+-- ---- Factory daily counters (funnel) ----------------------------------------------------------
	64	+create table if not exists factory_daily (
	65	+ day date primary key,
	66	+ seeds_processed integer not null default 0,
	67	+ requests integer not null default 0,
	68	+ candidates integer not null default 0,
	69	+ shadow_created integer not null default 0,
	70	+ accepted integer not null default 0,
	71	+ rejected integer not null default 0,
	72	+ blocked integer not null default 0
	73	+);

modified packages/db/src/schema.ts +46 −0

		`@@ -25,6 +25,9 @@ export const sources = pgTable("sources", {`
25	25	language: text("language"),
26	26	kind: text("kind").notNull().default("registry"),
27	27	ownerToken: text("owner_token"),
	28	+ /** seed (YAML registry) · import (admin) · factory (Source Factory) */
	29	+ origin: text("origin").notNull().default("seed"),
	30	+ sector: text("sector"),
28	31	createdAt: ts("created_at").notNull().defaultNow(),
29	32	updatedAt: ts("updated_at").notNull().defaultNow(),
30	33	});
		`@@ -419,7 +422,50 @@ export const discoveryCandidates = pgTable("discovery_candidates", {`
419	422	score: jsonb("score").$type<Record<string, unknown>>().notNull().default({}),
420	423	status: text("status").notNull().default("candidate"),
421	424	foundAt: ts("found_at").notNull().defaultNow(),
	425	+ seedId: text("seed_id"),
	426	+ connector: text("connector"),
	427	+ type: text("type"),
	428	+ name: text("name"),
	429	+ kindClass: text("kind_class"),
	430	+ config: jsonb("config").$type<Record<string, unknown>>().notNull().default({}),
	431	+ tier: text("tier"),
	432	+ scoreValue: real("score_value"),
	433	+ reason: text("reason"),
	434	+ shadowSensorId: text("shadow_sensor_id"),
	435	+ decidedAt: ts("decided_at"),
	436	+ updatedAt: ts("updated_at").notNull().defaultNow(),
	437	+});
	438	+
	439	+export const factorySeeds = pgTable("factory_seeds", {
	440	+ id: text("id").primaryKey(),
	441	+ name: text("name").notNull(),
	442	+ domain: text("domain").notNull(),
	443	+ homepage: text("homepage"),
	444	+ categories: text("categories").array().notNull().default([]),
	445	+ country: text("country"),
	446	+ language: text("language"),
	447	+ tier: text("tier").notNull().default("B"),
	448	+ weight: real("weight").notNull().default(1),
	449	+ importance: smallint("importance").notNull().default(2),
	450	+ aliases: text("aliases").array().notNull().default([]),
	451	+ firstParty: boolean("first_party").notNull().default(true),
	452	+ sector: text("sector"),
	453	+ universe: text("universe"),
	454	+ hints: jsonb("hints").$type<Record<string, unknown>>().notNull().default({}),
	455	+ status: text("status").notNull().default("queued"),
	456	+ attempts: integer("attempts").notNull().default(0),
	457	+ sourceId: text("source_id"),
	458	+ candidates: integer("candidates").notNull().default(0),
	459	+ shadow: integer("shadow").notNull().default(0),
	460	+ accepted: integer("accepted").notNull().default(0),
	461	+ rejected: integer("rejected").notNull().default(0),
	462	+ lastError: text("last_error"),
	463	+ discoveredAt: ts("discovered_at"),
	464	+ createdAt: ts("created_at").notNull().defaultNow(),
	465	+ updatedAt: ts("updated_at").notNull().defaultNow(),
422	466	});
	467	+export type FactorySeed = typeof factorySeeds.$inferSelect;
	468	+export type DiscoveryCandidate = typeof discoveryCandidates.$inferSelect;
423	469
424	470	export const metricsDaily = pgTable("metrics_daily", {
425	471	day: date("day", { mode: "string" }).primaryKey(),
426	472