SPB Git
spboucher.ai

spb/websensor

Public

WebSensor 0.3 — Source Factory & Global Observation Coverage Score

- Deep discovery (robots → sitemaps → feeds incl. official sub-domains → navigation-classified pages → status providers → GitHub org → EDGAR → Hugging Face → OpenAPI → posture), every candidate validated, wildcard DNS / anti-bot detection, content de-duplication, expected change frequency (median gap)
- Factory: coverage universes → factory_seeds → scored candidates (explainable) → promotion caps → SHADOW sensors (polled, never published) → evidence-based acceptance/rejection; separate websensor-factory process with Redis heartbeat; CLI expand/factory seed|run|evaluate|stats|export|requeue; YAML export for graduation
- Migration 0007 (sources.origin/sector, factory_seeds, candidate lifecycle columns, factory_daily); pipeline keeps SHADOW status and skips events for shadow sensors
- Coverage: sector universes schema + validator (--dns), registrableDomain(), GET /api/v1/coverage[/:sector], pages /coverage and /coverage/[sector], homepage rail, stats exclude shadow sensors
- Admin API /api/v1/admin/factory* + /ops Source Factory panel (funnel, sectors, candidates, accept/reject)
- Docs: COVERAGE.md, CLAUDE.md, deploy README, ARCHITECTURE, README; manifest gains websensor-factory + factory seed hook; version 0.3.0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 13, 2026) parent 938aab1 Browse files

41 changed files +3,084 −173

41 changed files
Load all 41 files Unified Split
modified .env.example +13 −0
@@ -42,3 +42,16 @@ WS_RETENTION_CHANGED_RAW_DAYS=60
42 WS_ADMIN_TOKEN= 42 WS_ADMIN_TOKEN=
43 # Custom URL monitors per anonymous owner (spec §45) 43 # Custom URL monitors per anonymous owner (spec §45)
44 WS_MONITORS_PER_OWNER=5 44 WS_MONITORS_PER_OWNER=5
45+
46+# Source Factory (apps/engine/src/factory-main.ts — separate `websensor-factory` process). Seeds come from
47+# config/coverage/*.yaml (members not yet monitored, or carrying hints) and config/factory/seeds/*.yaml.
48+WS_FACTORY=1
49+WS_FACTORY_CONCURRENCY=6 # organizations discovered in parallel
50+WS_FACTORY_BUDGET=90 # HTTP requests per organization
51+WS_FACTORY_MIN_SCORE=0.35 # candidate score needed to enter shadow monitoring
52+WS_FACTORY_SHADOW_MIN_CHECKS=5 # checks before a shadow sensor can be accepted
53+WS_FACTORY_SHADOW_MIN_HOURS=24
54+WS_FACTORY_SHADOW_MAX_HOURS=120
55+WS_FACTORY_TICK_SECONDS=20
56+WS_COVERAGE_DIR=./config/coverage
57+# GITHUB_TOKEN= # optional: lets the factory rank an organization's repositories via the GitHub API
modified CLAUDE.md +36 −6
@@ -5,7 +5,7 @@ Web. It monitors official public sources, detects meaningful changes, classifies
5 entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score, 5 entities, scores importance / confidence / novelty / impact / velocity / anomaly into a WebSensor Signal Score,
6 preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through 6 preserves immutable evidence, clusters signals into events with propagation timelines, and publishes them through
7 a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file 7 a replayable WebSocket feed. Product brief: `docs/PRODUCT-BRIEF.md`; architecture: `docs/ARCHITECTURE.md`. This file
8−is the working guide for the code (v0.2, 2026-09-11). 8+is the working guide for the code (v0.3, 2026-09-13: Source Factory + Global Observation Coverage Score).
9 9
10 ## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15) 10 ## Layout (pnpm workspace, TypeScript ESM, Node ≥ 22.15)
11 - `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups, 11 - `packages/core` — taxonomy (event types incl. cyber/finance/government/health/transport classes, event groups,
@@ -31,6 +31,30 @@ is the working guide for the code (v0.2, 2026-09-11).
31 (`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) + 31 (`http rss sitemap statuspage statusjson github jsonlist package edgar openapi csv pdf dns tls headers rdap`) +
32 `discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and 32 `discovery` + `scrapfly` fallback. `jsonlist.urlTemplate` supports `{key}` (path separators kept) and
33 `{field.path}`. 33 `{field.path}`.
34+- **Source Factory** (`apps/engine/src/factory/`, process `apps/engine/src/factory-main.ts` = PM2 `websensor-factory`) —
35+ the system grows by itself: coverage universes (`config/coverage/*.yaml`, one sector per file, members = organizations
36+ with a verified domain + `hints`) and `config/factory/seeds/*.yaml` become `factory_seeds`; `runFactoryBatch()` claims
37+ seeds (systemic first) and runs **deep discovery** (`packages/connectors/src/discovery-deep.ts`: robots → sitemaps →
38+ feeds incl. official sub-domains → navigation-classified pages (news/press/IR/changelog/security/pricing/legal/careers/
39+ leadership/docs) → status-page providers → GitHub org repos → EDGAR (cik) → Hugging Face (hf_author) → OpenAPI →
40+ posture; every candidate is fetched and parsed, wildcard DNS detected, feeds/pages de-duplicated by content) →
41+ `score.ts` (evidence × page-class weight × organization importance × first-party confidence × change frequency −
42+ fetch cost, explainable reasons; caps per organization/class) → **shadow sensors** (`sensors.status = 'SHADOW'`,
43+ `config.factory/shadow/kind/seedId`, priority 3: polled, snapshots + changes stored, **never published**; the pipeline
44+ and `handleMissing` skip event creation) → `shadow.ts` `evaluateShadows()` every 15 min (reject: ≥ 50 % errors / all
45+ changes noise / duplicate content of an active sensor; accept after ≥ 5 checks and 24 h → `ACTIVE` with its real
46+ priority; defer otherwise, max 120 h). Sources created by the factory carry `origin = 'factory'` + `sector`; attached
47+ ones reuse the registry source (match by registrable domain, `coverageKey()`). Candidates and decisions live in
48+ `discovery_candidates` (status candidate / shadow / accepted / rejected / duplicate, `reason`, `score`); funnel in
49+ `factory_daily`; heartbeat Redis `ws:factory:status`. `cli.ts expand <domain> [--hints=json]` = dry run;
50+ `cli.ts factory seed|run|evaluate|stats|export|requeue`; `factory export` writes a reviewable YAML fragment
51+ (graduating accepted sensors into `config/sources.d/` hands them to the registry sync).
52+- **Coverage** (`apps/api/src/coverage.ts`, `GET /api/v1/coverage[/:sector]`, pages `/coverage`, `/coverage/[sector]`,
53+ homepage rail) — Global Observation Coverage Score: per sector `breadth` (importance-weighted share of universe
54+ members with ≥ 1 active non-shadow sensor on their registrable domain), `depth` (min(1, sensors/5)),
55+ `score = 100·(0.7·breadth + 0.3·depth)`, global = sector-weight-weighted mean. Universes are the denominator (with
56+ `provenance`), never the registry. Validator: `apps/engine/src/coverage-validate.ts [file] [--dns]`. Guide:
57+ `docs/registry/COVERAGE.md`. Schema shared in `packages/core/src/coverage-schema.ts` (+ `registrableDomain()`).
34 - `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency, 58 - `apps/engine` — scheduler (priority-aware `FOR UPDATE SKIP LOCKED` claims, global + per-host concurrency,
35 **domain circuit breaker**, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff → 59 **domain circuit breaker**, heartbeat to Redis `ws:engine:status`), pipeline (fetch → normalize → snapshot → diff →
36 heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias 60 heuristics + semantic class → change → event: fingerprint idempotency, entity resolution with ambiguous-alias
@@ -38,8 +62,8 @@ is the working guide for the code (v0.2, 2026-09-11).
38 with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook 62 with propagation timeline + lead time + breaking state, entity/source daily counters, alert evaluation + webhook
39 delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery, 63 delivery, Redis stream + pub/sub), registry sync (`config/sources.yaml` + `config/sources.d/`), discovery,
40 connector health rollups, **retention** (`retention.ts`), Prometheus metrics :8262. 64 connector health rollups, **retention** (`retention.ts`), Prometheus metrics :8262.
41− CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`, 65+ CLI: `src/cli.ts` (`sync`, `discover`, `probe`, `expand`, `run-once`, `run-due`, `relink-entities [days]`, `prune-blobs`,
42− `refresh-clusters`, `llm-test`); `src/validate.ts` (registry validator). 66+ `refresh-clusters`, `llm-test`, `factory …`); `src/validate.ts` (registry validator); `src/coverage-validate.ts`.
43 - `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped: 67 - `apps/api` — Fastify gateway (:8260): REST `/api/v1/*` (`routes.ts` public, `routes-user.ts` owner-scoped:
44 watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts` 68 watchlists / alerts (+webhook) / notifications / bookmarks / saved views / custom monitors, `routes-admin.ts`
45 behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models 69 behind `WS_ADMIN_TOKEN`: ops, sensor & source actions, connector test, bulk import), intelligence read-models
@@ -52,8 +76,8 @@ is the working guide for the code (v0.2, 2026-09-11).
52 (intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters, 76 (intelligence panel), `command-palette.tsx` (⌘K), `live-strip.tsx`, `live-feed.tsx` v2 (URL-synced filters,
53 pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar` 77 pause, "↑ N new events", replay), `field-changes.tsx`. Pages: `/` `/live` `/breaking` `/silent` `/pulse` `/radar`
54 `/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]` 78 `/explore` `/entity/[id]` (`/company` redirects) `/cluster/[slug]` `/source/[id]` `/sensor/[id]` `/category/[c]`
55− `/country/[slug]` `/event/[slug]` (+ OG image) `/bookmarks` `/watchlists` `/alerts` `/monitors` `/ops` `/health` 79+ `/country/[slug]` `/event/[slug]` (+ OG image) `/coverage` `/coverage/[sector]` `/bookmarks` `/watchlists` `/alerts`
56− `/api`. 80+ `/monitors` `/ops` (+ Source Factory panel: funnel, sectors, candidates, accept/reject) `/health` `/api`.
57 - `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order 81 - `config/sources.yaml` (founding registry) + `config/sources.d/*.yaml` fragments merged in file-name order
58 (`extend: true` adds to an earlier source). Fragments 10–35 = site classes; **40–46 (2026-09-11)** = depth: AI 82 (`extend: true` adds to an earlier source). Fragments 10–35 = site classes; **40–46 (2026-09-11)** = depth: AI
59 frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport / 83 frontier, cloud infrastructure, cybersecurity, finance & markets, governments, science & health, transport /
@@ -87,6 +111,11 @@ is the working guide for the code (v0.2, 2026-09-11).
87 `llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works. 111 `llm: false` on firehoses (news, arXiv, NVD, package streams). No key → the system still works.
88 - Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes 112 - Feeds: items are "new" only if never seen and < 14 days old; >50 % list shrink = partial response; 404 becomes
89 `page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks. 113 `page_removed` after `WS_DELETE_CONFIRMATIONS` separated checks.
114+- Shadow is not production: `status = 'SHADOW'` sensors never create events, never count in public stats
115+ (`stats.sensors`, coverage) and never compete with production sensors (priority 3). Nothing the Factory finds is
116+ published before `evaluateShadows()` (or an operator) accepts it. Public source pages show shadow sensors with a SHADOW pill.
117+- Coverage universes are lists of real organizations with verified domains (`--dns`), provenance and retrieval dates —
118+ never guessed domains; several files may share a sector key (universes are merged).
90 - Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public 119 - Admin API is off unless `WS_ADMIN_TOKEN` is set; custom monitors are limited per owner and never appear in public
91 feeds (`sources.kind = 'custom'`). 120 feeds (`sources.kind = 'custom'`).
92 - All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…). 121 - All timestamps UTC. Ids are prefixed (`src_`, `sen_`, `snap_`, `chg_`, `evt_`, `clu_`, `ent_`, `wl_`, `alr_`…).
@@ -108,6 +137,7 @@ Run CLI/engine from the repo root. A copy of production data for UI work:
108 `mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json` 137 `mld stage . websensor && mld deploy websensor --node M4M64b`. Manifest `M1M32:~/dispatch/apps/websensor.json`
109 (secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260, 138 (secrets incl. `WS_ADMIN_TOKEN`; gitignored copy `deploy/websensor.mld.json`). Processes: `websensor-api` (:8260,
110 public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine` 139 public via MacLustr Tunnel `www.websensor.io` on BHS64), `websensor-web` (:8261 loopback), `websensor-engine`
111−(metrics :8262). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that 140+(metrics :8262), `websensor-factory` (Source Factory, heartbeat `ws:factory:status`; post-sync hook `factory seed --mode=hinted`
141+re-seeds from the coverage universes at every deploy). Postgres 17 `websensor` + Redis local. Blob store `~/websensor-data/blobs`. After a deploy that
112 changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See 142 changes alias or scoring rules: `cli.ts relink-entities 7` and `cli.ts refresh-clusters` on the node. See
113 `deploy/README.md`. 143 `deploy/README.md`.
modified README.md +31 −0
@@ -94,6 +94,7 @@ threat-intelligence desk*: dense information, one-pixel borders, small badges, p
94 (OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │ 94 (OVH, Caddy, (wg1) │ /api/v1/* REST + WebSocket /api/v1/live (Redis pub/sub fan-out) │
95 MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │ 95 MacLustr Tunnel) │ /* reverse proxy → websensor-web :8261 (Next.js 16) │
96 │ websensor-engine scheduler + pipeline, Prometheus :8262 │ 96 │ websensor-engine scheduler + pipeline, Prometheus :8262 │
97+ │ websensor-factory Source Factory: seeds → discovery → shadow │
97 │ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │ 98 │ PostgreSQL 17 `websensor` · Redis · blob store ~/websensor-data/blobs │
98 └───────────────────────────────────────────────────────────────────────┘ 99 └───────────────────────────────────────────────────────────────────────┘
99 ``` 100 ```
@@ -105,6 +106,8 @@ threat-intelligence desk*: dense information, one-pixel borders, small badges, p
105 | Evidence | `packages/store` | content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres | 106 | Evidence | `packages/store` | content-addressed zstd blob store (`sha256/ab/cd/<hash>.zst`); raw bodies, canonical forms and full diffs never live in Postgres |
106 | Acquisition | `packages/connectors` | connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback | 107 | Acquisition | `packages/connectors` | connector SDK, safe fetcher, 16 connector families, discovery, budgeted Scrapfly fallback |
107 | Ingestion | `apps/engine` | scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator | 108 | Ingestion | `apps/engine` | scheduler, pipeline, registry sync, discovery, health rollups, retention, alerts, CLI, validator |
109+| Expansion | `apps/engine/src/factory` | **Source Factory** (0.3): coverage universes → deep discovery → scored candidates → shadow sensors → evidence-based acceptance; `websensor-factory` process |
110+| Coverage | `apps/api/src/coverage.ts` | **Global Observation Coverage Score** per sector (`/coverage`, `GET /api/v1/coverage`): how much of the world's high-value public web is observed |
108 | Gateway | `apps/api` | REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API | 111 | Gateway | `apps/api` | REST, WebSocket, intelligence read-models, TTL cache, owner-scoped tools, admin API |
109 | Product | `apps/web` | Next.js 16 App Router, design system, live feed, drawer, palette, desks | 112 | Product | `apps/web` | Next.js 16 App Router, design system, live feed, drawer, palette, desks |
110 113
@@ -493,6 +496,34 @@ without ever deleting event evidence. Details: `deploy/README.md`, `docs/ARCHITE
493 - Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting 496 - Admin API disabled unless `WS_ADMIN_TOKEN` is set; owner tools use an anonymous browser token; rate limiting
494 on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP. 497 on every `/api/*` route; strict input validation (zod); snapshots served raw get `nosniff` and a restrictive CSP.
495 498
499+## Source Factory and coverage (0.3, 2026-09-13)
500+
501+WebSensor no longer grows by hand-written lists alone. `config/coverage/*.yaml` declares, sector by sector, the
502+organizations that *should* be observed (listed companies by index, governments of every country by institutional
503+role, AI, cloud, cybersecurity, markets, science, healthcare, energy, transport, telecom, logistics, open source,
504+commerce), each with a verified domain, a country, an importance and optional hints (CIK, GitHub organization,
505+Hugging Face author, status page, explicit URLs). Those universes are the denominator of the **Global Observation
506+Coverage Score** (`breadth` = importance-weighted share of members with an active sensor, `depth` = sensors per
507+organization, `score = 100·(0.7·breadth + 0.3·depth)`) and the input of the **Source Factory**:
508+
509+```
510+seed (organization + hints)
511+ → deep discovery robots · sitemaps · feeds (+ official sub-domains) · navigation-classified pages
512+ (news · press · IR · changelog · security · pricing · legal · careers · leadership · docs)
513+ · status-page providers · GitHub org repositories · EDGAR · Hugging Face · OpenAPI · posture
514+ → validation every candidate fetched and parsed by its connector; wildcard DNS and anti-bot detected;
515+ duplicates removed by content (shared feed items, identical page text)
516+ → scoring evidence × page class × organization importance × first-party confidence × change
517+ frequency − fetch cost, with explainable reasons; caps per organization and per class
518+ → shadow sensors polled like any sensor, evidence stored, nothing published
519+ → evaluation reject (errors, pure noise, duplicate of an active sensor) · accept (≥ 5 checks, 24 h) · defer
520+ → production accepted sensors publish; `cli.ts factory export` graduates them into a registry fragment
521+```
522+
523+`cli.ts expand <domain>` shows what the Factory would find for one organization; `/ops` has the funnel, per-sector
524+progress and a candidate review table; `/coverage` and `/coverage/<sector>` show the score and every member's state
525+(observed · shadow · queued · not seeded).
526+
496 ## Roadmap 527 ## Roadmap
497 528
498 World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel 529 World map for geolocated events; e-mail / Slack / Discord / Telegram / push alert channels (the channel
modified apps/api/src/config.ts +1 −1
@@ -14,5 +14,5 @@ export const config = {
14 adminToken: env.WS_ADMIN_TOKEN ?? "", 14 adminToken: env.WS_ADMIN_TOKEN ?? "",
15 /** Custom URL monitors per anonymous owner (spec §45). */ 15 /** Custom URL monitors per anonymous owner (spec §45). */
16 monitorsPerOwner: Number(env.WS_MONITORS_PER_OWNER ?? 5), 16 monitorsPerOwner: Number(env.WS_MONITORS_PER_OWNER ?? 5),
17− version: "0.2.0", 17+ version: "0.3.0",
18 }; 18 };
added apps/api/src/coverage.ts +236 −0
@@ -0,0 +1,236 @@
1+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
2+import { join } from "node:path";
3+import YAML from "yaml";
4+import { coverageKey, coverageSectorSchema, type CoverageMember, type CoverageSector } from "@websensor/core";
5+import { db, sql } from "@websensor/db";
6+import { cached } from "./cache";
7+
8+/**
9+ * Global Observation Coverage Score — "how much of the world's high-value public web do we observe?"
10+ *
11+ * For every sector the universes in `config/coverage/*.yaml` are the denominator. A member is *covered* when at
12+ * least one active (non-shadow) sensor lives on its registrable domain (or on a host declared in `hints.hosts`).
13+ * breadth = Σ w·covered / Σ w (w = member importance 1–3)
14+ * depth = Σ w·min(1, sensors/5) / Σ w (five sensors per organization = full depth)
15+ * score = 100 · (0.7·breadth + 0.3·depth)
16+ * The global score is the sector-weight-weighted mean. Everything is explainable: the per-member table shows
17+ * which sensors count, what is in shadow and what the Factory is still queuing.
18+ */
19+export interface MemberCoverage {
20+ name: string;
21+ domain: string;
22+ country: string | null;
23+ importance: number;
24+ universe: string;
25+ covered: boolean;
26+ sensors: number;
27+ shadow: number;
28+ source_id: string | null;
29+ seed_status: string | null;
30+ depth: number;
31+}
32+export interface UniverseCoverage {
33+ key: string;
34+ label: string;
35+ provenance: string | null;
36+ members: number;
37+ covered: number;
38+ sensors: number;
39+ breadth: number;
40+ depth: number;
41+ score: number;
42+}
43+export interface SectorCoverage {
44+ sector: string;
45+ label: string;
46+ description: string | null;
47+ weight: number;
48+ members: number;
49+ covered: number;
50+ shadow_members: number;
51+ queued: number;
52+ sensors: number;
53+ breadth: number;
54+ depth: number;
55+ score: number;
56+ universes: UniverseCoverage[];
57+}
58+
59+const COVERAGE_DIR = process.env.WS_COVERAGE_DIR ?? "./config/coverage";
60+
61+let fileCache: { at: number; sectors: CoverageSector[]; issues: string[] } | null = null;
62+export function loadSectors(): { sectors: CoverageSector[]; issues: string[] } {
63+ if (fileCache && Date.now() - fileCache.at < 60_000) return fileCache;
64+ const byKey = new Map<string, CoverageSector>();
65+ const issues: string[] = [];
66+ if (existsSync(COVERAGE_DIR) && statSync(COVERAGE_DIR).isDirectory()) {
67+ for (const f of readdirSync(COVERAGE_DIR).filter((x) => /\.ya?ml$/.test(x) && !x.startsWith("_")).sort()) {
68+ try {
69+ const parsed = coverageSectorSchema.safeParse(YAML.parse(readFileSync(join(COVERAGE_DIR, f), "utf8")));
70+ if (!parsed.success) {
71+ issues.push(`${f}: ${parsed.error.issues[0]?.path.join(".")}: ${parsed.error.issues[0]?.message}`);
72+ continue;
73+ }
74+ const s = parsed.data;
75+ const prev = byKey.get(s.sector);
76+ if (prev) prev.universes.push(...s.universes);
77+ else byKey.set(s.sector, { ...s, universes: [...s.universes] });
78+ } catch (e) {
79+ issues.push(`${f}: ${(e as Error).message}`);
80+ }
81+ }
82+ }
83+ fileCache = { at: Date.now(), sectors: [...byKey.values()], issues };
84+ return fileCache;
85+}
86+
87+interface Observed {
88+ /** registrable domain → { sensors (active), shadow, sourceId } */
89+ byKey: Map<string, { sensors: number; shadow: number; sourceId: string | null }>;
90+ seeds: Map<string, string>;
91+}
92+
93+async function observed(): Promise<Observed> {
94+ const [hosts, srcs, seeds] = await Promise.all([
95+ db.execute<{ host: string; status: string; n: string; source_id: string }>(sql`
96+ select lower(split_part(split_part(url, '/', 3), ':', 1)) as host, case when status = 'SHADOW' then 'SHADOW' else 'ACTIVE' end as status, count(*)::text as n, min(source_id) as source_id
97+ from sensors where enabled and url like 'http%' group by 1, 2`).then((r) => r.rows),
98+ db.execute<{ id: string; domain: string }>(sql`select id, domain from sources where enabled and kind = 'registry'`).then((r) => r.rows),
99+ db.execute<{ domain: string; status: string }>(sql`select domain, status from factory_seeds`).then((r) => r.rows),
100+ ]);
101+ const byKey = new Map<string, { sensors: number; shadow: number; sourceId: string | null }>();
102+ for (const h of hosts) {
103+ if (!h.host) continue;
104+ const k = coverageKey(h.host);
105+ const cur = byKey.get(k) ?? { sensors: 0, shadow: 0, sourceId: null };
106+ if (h.status === "SHADOW") cur.shadow += Number(h.n);
107+ else cur.sensors += Number(h.n);
108+ cur.sourceId = cur.sourceId ?? h.source_id;
109+ byKey.set(k, cur);
110+ }
111+ for (const s of srcs) {
112+ const k = coverageKey(s.domain);
113+ const cur = byKey.get(k) ?? { sensors: 0, shadow: 0, sourceId: null };
114+ cur.sourceId = cur.sourceId ?? s.id;
115+ byKey.set(k, cur);
116+ }
117+ const seedMap = new Map<string, string>();
118+ for (const s of seeds) seedMap.set(coverageKey(s.domain), s.status);
119+ return { byKey, seeds: seedMap };
120+}
121+
122+function memberCoverage(m: CoverageMember, universe: string, obs: Observed): MemberCoverage {
123+ const keys = [coverageKey(m.domain), ...((m.hints?.hosts ?? []).map((h) => coverageKey(h)))];
124+ let sensors = 0;
125+ let shadow = 0;
126+ let sourceId: string | null = null;
127+ for (const k of new Set(keys)) {
128+ const o = obs.byKey.get(k);
129+ if (!o) continue;
130+ sensors += o.sensors;
131+ shadow += o.shadow;
132+ sourceId = sourceId ?? o.sourceId;
133+ }
134+ return { name: m.name, domain: m.domain, country: m.country ?? null, importance: m.importance, universe, covered: sensors > 0, sensors, shadow, source_id: sourceId, seed_status: obs.seeds.get(coverageKey(m.domain)) ?? null, depth: Math.min(1, sensors / 5) };
135+}
136+
137+function aggregate(members: MemberCoverage[]): { breadth: number; depth: number; score: number; covered: number; sensors: number } {
138+ let w = 0;
139+ let cov = 0;
140+ let dep = 0;
141+ let covered = 0;
142+ let sensors = 0;
143+ for (const m of members) {
144+ w += m.importance;
145+ if (m.covered) {
146+ cov += m.importance;
147+ covered++;
148+ }
149+ dep += m.importance * m.depth;
150+ sensors += m.sensors;
151+ }
152+ const breadth = w ? cov / w : 0;
153+ const depth = w ? dep / w : 0;
154+ return { breadth: round(breadth), depth: round(depth), score: round(100 * (0.7 * breadth + 0.3 * depth), 1), covered, sensors };
155+}
156+const round = (x: number, d = 3): number => Math.round(x * 10 ** d) / 10 ** d;
157+
158+export async function coverageSummary(): Promise<Record<string, unknown>> {
159+ return cached("coverage:summary", 5 * 60_000, async () => {
160+ const { sectors, issues } = loadSectors();
161+ const obs = await observed();
162+ const out: SectorCoverage[] = [];
163+ let wsum = 0;
164+ let ssum = 0;
165+ let members = 0;
166+ let covered = 0;
167+ let sensorsTotal = 0;
168+ for (const s of sectors) {
169+ const all: MemberCoverage[] = [];
170+ const universes: UniverseCoverage[] = [];
171+ const seenDomains = new Set<string>();
172+ for (const u of s.universes) {
173+ const ms = u.members.map((m) => memberCoverage(m, u.key, obs));
174+ const a = aggregate(ms);
175+ universes.push({ key: u.key, label: u.label, provenance: u.provenance ?? null, members: ms.length, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score });
176+ for (const m of ms) {
177+ const k = coverageKey(m.domain);
178+ if (seenDomains.has(k)) continue;
179+ seenDomains.add(k);
180+ all.push(m);
181+ }
182+ }
183+ const a = aggregate(all);
184+ const sec: SectorCoverage = { sector: s.sector, label: s.label, description: s.description ?? null, weight: s.weight, members: all.length, covered: a.covered, shadow_members: all.filter((m) => !m.covered && m.shadow > 0).length, queued: all.filter((m) => !m.covered && (m.seed_status === "queued" || m.seed_status === "discovering")).length, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, universes };
185+ out.push(sec);
186+ wsum += s.weight;
187+ ssum += s.weight * a.score;
188+ members += all.length;
189+ covered += a.covered;
190+ sensorsTotal += a.sensors;
191+ }
192+ out.sort((a, b) => b.score - a.score);
193+ const [monitored, factory] = await Promise.all([
194+ db.execute<{ sources: string; sensors: string; shadow: string; countries: string }>(sql`select (select count(*) from sources where enabled and kind = 'registry')::text as sources, (select count(*) from sensors where enabled and status <> 'SHADOW')::text as sensors, (select count(*) from sensors where enabled and status = 'SHADOW')::text as shadow, (select count(distinct country) from sources where country is not null and enabled)::text as countries`).then((r) => r.rows[0]),
195+ db.execute<{ status: string; n: string }>(sql`select status, count(*)::text as n from factory_seeds group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [x.status, Number(x.n)]))),
196+ ]);
197+ return {
198+ global_score: wsum ? round(ssum / wsum, 1) : 0,
199+ members,
200+ covered,
201+ sensors_matched: sensorsTotal,
202+ sectors: out.map(({ universes: _u, ...rest }) => rest),
203+ monitored: { sources: Number(monitored?.sources ?? 0), sensors: Number(monitored?.sensors ?? 0), shadow: Number(monitored?.shadow ?? 0), countries: Number(monitored?.countries ?? 0) },
204+ factory: factory,
205+ method: "breadth = importance-weighted share of members with ≥ 1 active sensor · depth = importance-weighted min(1, sensors/5) · score = 100·(0.7·breadth + 0.3·depth) · global = sector-weight-weighted mean",
206+ issues,
207+ generated_at: new Date().toISOString(),
208+ };
209+ });
210+}
211+
212+export async function coverageSector(sector: string): Promise<Record<string, unknown> | null> {
213+ return cached(`coverage:sector:${sector}`, 5 * 60_000, async () => {
214+ const { sectors } = loadSectors();
215+ const s = sectors.find((x) => x.sector === sector);
216+ if (!s) return null;
217+ const obs = await observed();
218+ const universes = s.universes.map((u) => {
219+ const members = u.members.map((m) => memberCoverage(m, u.key, obs)).sort((a, b) => Number(b.covered) - Number(a.covered) || b.importance - a.importance || b.sensors - a.sensors || a.name.localeCompare(b.name));
220+ const a = aggregate(members);
221+ return { key: u.key, label: u.label, description: u.description ?? null, provenance: u.provenance ?? null, members: members.length, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, items: members };
222+ });
223+ const flat = new Map<string, MemberCoverage>();
224+ for (const u of universes) for (const m of u.items) if (!flat.has(coverageKey(m.domain))) flat.set(coverageKey(m.domain), m);
225+ const a = aggregate([...flat.values()]);
226+ const byCountry = new Map<string, { members: number; covered: number }>();
227+ for (const m of flat.values()) {
228+ const c = m.country ?? "—";
229+ const cur = byCountry.get(c) ?? { members: 0, covered: 0 };
230+ cur.members++;
231+ if (m.covered) cur.covered++;
232+ byCountry.set(c, cur);
233+ }
234+ return { sector: s.sector, label: s.label, description: s.description ?? null, weight: s.weight, categories: s.categories, members: flat.size, covered: a.covered, sensors: a.sensors, breadth: a.breadth, depth: a.depth, score: a.score, shadow_members: [...flat.values()].filter((m) => !m.covered && m.shadow > 0).length, queued: [...flat.values()].filter((m) => !m.covered && (m.seed_status === "queued" || m.seed_status === "discovering")).length, universes, by_country: [...byCountry.entries()].map(([country, v]) => ({ country, ...v })).sort((x, y) => y.members - x.members), generated_at: new Date().toISOString() };
235+ });
236+}
modified apps/api/src/live.ts +9 −0
@@ -49,6 +49,15 @@ export async function engineStatus(): Promise<Record<string, unknown> | null> {
49 } 49 }
50 } 50 }
51 51
52+export async function factoryStatus(): Promise<Record<string, unknown> | null> {
53+ try {
54+ const raw = await getCmd().get("ws:factory:status");
55+ return raw ? (JSON.parse(raw) as Record<string, unknown>) : null;
56+ } catch {
57+ return null;
58+ }
59+}
60+
52 function getCmd(): Redis { 61 function getCmd(): Redis {
53 if (!cmd) cmd = new Redis(config.redisUrl, { maxRetriesPerRequest: 2, lazyConnect: false }); 62 if (!cmd) cmd = new Redis(config.redisUrl, { maxRetriesPerRequest: 2, lazyConnect: false });
54 return cmd; 63 return cmd;
modified apps/api/src/queries.ts +5 −3
@@ -127,7 +127,9 @@ export async function stats(): Promise<Record<string, unknown>> {
127 await db.execute<Record<string, unknown>>(sql` 127 await db.execute<Record<string, unknown>>(sql`
128 select 128 select
129 (select count(*) from sources where enabled and kind = 'registry') as sources, 129 (select count(*) from sources where enabled and kind = 'registry') as sources,
130− (select count(*) from sensors where enabled) as sensors, 130+ (select count(*) from sensors where enabled and status <> 'SHADOW') as sensors,
131+ (select count(*) from sensors where enabled and status = 'SHADOW') as sensors_shadow,
132+ (select count(*) from sources where enabled and kind = 'registry' and origin = 'factory') as sources_factory,
131 (select count(*) from entities) as entities, 133 (select count(*) from entities) as entities,
132 (select count(*) from snapshots) as snapshots, 134 (select count(*) from snapshots) as snapshots,
133 (select count(*) from events) as events_total, 135 (select count(*) from events) as events_total,
@@ -144,8 +146,8 @@ export async function stats(): Promise<Record<string, unknown>> {
144 (select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour, 146 (select count(*) from sensor_runs where started_at >= now() - interval '1 hour') as checks_last_hour,
145 (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes') as checks_last_5m, 147 (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes') as checks_last_5m,
146 (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified') as not_modified_last_5m, 148 (select count(*) from sensor_runs where started_at >= now() - interval '5 minutes' and outcome = 'not_modified') as not_modified_last_5m,
147− (select count(*) from sensors where health = 'UP' and enabled) as sensors_up, 149+ (select count(*) from sensors where health = 'UP' and enabled and status <> 'SHADOW') as sensors_up,
148− (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled) as sensors_degraded, 150+ (select count(*) from sensors where health in ('DEGRADED','ERROR','RATE_LIMITED') and enabled and status <> 'SHADOW') as sensors_degraded,
149 (select count(*) from sources where enabled and first_party and kind = 'registry') as sources_first_party, 151 (select count(*) from sources where enabled and first_party and kind = 'registry') as sources_first_party,
150 (select count(distinct country) from sources where country is not null) as countries, 152 (select count(distinct country) from sources where country is not null) as countries,
151 (select max(started_at) from sensor_runs) as last_check_at, 153 (select max(started_at) from sensor_runs) as last_check_at,
modified apps/api/src/routes-admin.ts +76 −1
@@ -8,7 +8,7 @@ import { getConnector, listConnectors, NormalizeError } from "@websensor/connect
8 import { db, sql } from "@websensor/db"; 8 import { db, sql } from "@websensor/db";
9 import { cacheStats, invalidate } from "./cache"; 9 import { cacheStats, invalidate } from "./cache";
10 import { config } from "./config"; 10 import { config } from "./config";
11−import { engineStatus, liveStats } from "./live"; 11+import { engineStatus, factoryStatus, liveStats } from "./live";
12 12
13 /** 13 /**
14 * Internal operations (spec §70–72, §85). Enabled only when WS_ADMIN_TOKEN is set; every request 14 * Internal operations (spec §70–72, §85). Enabled only when WS_ADMIN_TOKEN is set; every request
@@ -193,6 +193,81 @@ export async function registerAdminRoutes(app: FastifyInstance): Promise<void> {
193 return { dry_run: false, imported_sources: valid.filter((v) => !v.extend).length, extended_sources: valid.filter((v) => v.extend).length, sensors, invalid: report.filter((r) => !r.ok).length, report }; 193 return { dry_run: false, imported_sources: valid.filter((v) => !v.extend).length, extended_sources: valid.filter((v) => v.extend).length, sensors, invalid: report.filter((r) => !r.ok).length, report };
194 }); 194 });
195 195
196+ // ---- Source Factory ---------------------------------------------------------------------------------
197+ app.get("/api/v1/admin/factory", async () => {
198+ const [seeds, cands, shadows, daily, sectors, recent, hb, ready] = await Promise.all([
199+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from factory_seeds group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [String(x.status), Number(x.n)]))),
200+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from discovery_candidates group by status`).then((r) => Object.fromEntries(r.rows.map((x) => [String(x.status), Number(x.n)]))),
201+ db.execute<Record<string, unknown>>(sql`select count(*)::int as total, count(*) filter (where health = 'UP')::int as up, count(*) filter (where health <> 'UP')::int as unhealthy, count(*) filter (where total_runs >= 5)::int as ready, coalesce(sum(raw_changes),0)::int as raw_changes from sensors where status = 'SHADOW' and enabled`).then((r) => r.rows[0]),
202+ db.execute<Record<string, unknown>>(sql`select day::text as day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked from factory_daily order by day desc limit 30`).then((r) => r.rows),
203+ db.execute<Record<string, unknown>>(sql`select coalesce(sector, '-') as sector, count(*)::int as seeds, count(*) filter (where status = 'queued')::int as queued, count(*) filter (where status = 'discovering')::int as discovering, count(*) filter (where status in ('discovered','done'))::int as discovered, count(*) filter (where status = 'blocked')::int as blocked, count(*) filter (where status = 'error')::int as errors, coalesce(sum(shadow),0)::int as shadow, coalesce(sum(accepted),0)::int as accepted, coalesce(sum(rejected),0)::int as rejected from factory_seeds group by 1 order by seeds desc`).then((r) => r.rows),
204+ db.execute<Record<string, unknown>>(sql`select id, name, domain, sector, status, candidates, shadow, accepted, rejected, last_error, discovered_at, source_id from factory_seeds where status <> 'queued' order by updated_at desc limit 40`).then((r) => r.rows),
205+ factoryStatus(),
206+ db.execute<Record<string, unknown>>(sql`select kind_class, count(*)::int as n from discovery_candidates where status = 'shadow' group by 1 order by n desc`).then((r) => r.rows),
207+ ]);
208+ return { seeds, candidates: cands, shadow: shadows, shadow_by_kind: ready, daily, sectors, recent_seeds: recent, heartbeat: hb, generated_at: new Date().toISOString() };
209+ });
210+ app.get<{ Querystring: { status?: string; sector?: string; q?: string; limit?: string } }>("/api/v1/admin/factory/seeds", async (req) => {
211+ const lim = Math.min(1000, Number(req.query.limit ?? 200));
212+ const rows = await db.execute<Record<string, unknown>>(sql`select * from factory_seeds where 1=1 ${req.query.status ? sql`and status = ${req.query.status}` : sql``} ${req.query.sector ? sql`and sector = ${req.query.sector}` : sql``} ${req.query.q ? sql`and (name ilike ${"%" + req.query.q + "%"} or domain ilike ${"%" + req.query.q + "%"})` : sql``} order by importance desc, updated_at desc limit ${lim}`);
213+ return { items: rows.rows };
214+ });
215+ /** Bulk seeds `{ seeds: [{ name, domain, categories?, country?, tier?, importance?, aliases?, sector?, universe?, hints? }] }` — queued for the factory process. */
216+ app.post("/api/v1/admin/factory/seeds", async (req) => {
217+ const seedSchema = z.object({ id: z.string().regex(/^[a-z0-9][a-z0-9-]*$/).optional(), name: z.string().min(1), domain: z.string().min(3), categories: z.array(z.string()).default([]), country: z.string().regex(/^[A-Z]{2,3}$/).optional(), language: z.string().optional(), tier: z.enum(["S", "A", "B", "C", "D"]).default("B"), importance: z.number().int().min(1).max(3).default(2), aliases: z.array(z.string()).default([]), first_party: z.boolean().default(true), sector: z.string().optional(), universe: z.string().optional(), hints: z.record(z.string(), z.unknown()).default({}) });
218+ const body = z.object({ seeds: z.array(seedSchema).min(1).max(2000) }).parse(req.body ?? {});
219+ let n = 0;
220+ for (const s of body.seeds) {
221+ const domain = s.domain.toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "");
222+ const id = s.id ?? domain.replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "").slice(0, 60);
223+ await db.execute(sql`insert into factory_seeds (id, name, domain, categories, country, language, tier, importance, aliases, first_party, sector, universe, hints, status)
224+ values (${id}, ${s.name}, ${domain}, ${sql.raw("'{" + s.categories.map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.country ?? null}, ${s.language ?? null}, ${s.tier}, ${s.importance}, ${sql.raw("'{" + s.aliases.map((c) => '"' + c.replace(/["\\]/g, "") + '"').join(",") + "}'::text[]")}, ${s.first_party}, ${s.sector ?? null}, ${s.universe ?? null}, ${JSON.stringify(s.hints)}::jsonb, 'queued')
225+ on conflict (id) do update set name = excluded.name, hints = factory_seeds.hints || excluded.hints, status = 'queued', updated_at = now()`);
226+ n++;
227+ }
228+ invalidate("coverage");
229+ return { ok: true, queued: n };
230+ });
231+ app.post<{ Params: { id: string } }>("/api/v1/admin/factory/seeds/:id/requeue", async (req, reply) => {
232+ const r = await db.execute(sql`update factory_seeds set status = 'queued', updated_at = now() where id = ${req.params.id}`);
233+ if (!r.rowCount) return reply.status(404).send({ error: "not_found" });
234+ return { ok: true };
235+ });
236+ app.get<{ Querystring: { status?: string; seed?: string; source?: string; kind?: string; limit?: string } }>("/api/v1/admin/factory/candidates", async (req) => {
237+ const lim = Math.min(1000, Number(req.query.limit ?? 200));
238+ const rows = await db.execute<Record<string, unknown>>(sql`select c.*, s.health as shadow_health, s.total_runs as shadow_runs, s.raw_changes as shadow_changes from discovery_candidates c left join sensors s on s.id = c.shadow_sensor_id where 1=1 ${req.query.status ? sql`and c.status = ${req.query.status}` : sql``} ${req.query.seed ? sql`and c.seed_id = ${req.query.seed}` : sql``} ${req.query.source ? sql`and c.source_id = ${req.query.source}` : sql``} ${req.query.kind ? sql`and c.kind_class = ${req.query.kind}` : sql``} order by c.updated_at desc limit ${lim}`);
239+ return { items: rows.rows };
240+ });
241+ /** Manual decision on a shadow sensor: accept now (→ ACTIVE) or reject (→ DISABLED). */
242+ app.post<{ Params: { id: string; action: string } }>("/api/v1/admin/factory/shadow/:id/:action", async (req, reply) => {
243+ const s = (await db.execute<Record<string, unknown>>(sql`select id, config, status from sensors where id = ${req.params.id}`)).rows[0];
244+ if (!s) return reply.status(404).send({ error: "not_found" });
245+ const reason = `manual ${req.params.action} via admin API`;
246+ if (req.params.action === "accept") {
247+ const tp = Number(((s.config as Record<string, unknown>).targetPriority as number | undefined) ?? 2);
248+ await db.execute(sql`update sensors set status = 'ACTIVE', enabled = true, priority = ${tp}, config = (config - 'shadow') || jsonb_build_object('acceptedAt', now()::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${req.params.id}`);
249+ await db.execute(sql`update discovery_candidates set status = 'accepted', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${req.params.id}`);
250+ } else if (req.params.action === "reject") {
251+ await db.execute(sql`update sensors set status = 'DISABLED', enabled = false, config = config || jsonb_build_object('rejectedAt', now()::text, 'shadowReport', ${reason}::text), updated_at = now() where id = ${req.params.id}`);
252+ await db.execute(sql`update discovery_candidates set status = 'rejected', reason = ${reason}, decided_at = now(), updated_at = now() where shadow_sensor_id = ${req.params.id}`);
253+ } else return reply.status(400).send({ error: "action must be accept or reject" });
254+ invalidate("");
255+ return { ok: true };
256+ });
257+ /** Promote a plain candidate (never shadowed) straight into shadow monitoring. */
258+ app.post<{ Params: { id: string } }>("/api/v1/admin/factory/candidates/:id/shadow", async (req, reply) => {
259+ const c = (await db.execute<Record<string, unknown>>(sql`select * from discovery_candidates where id = ${req.params.id}`)).rows[0];
260+ if (!c) return reply.status(404).send({ error: "not_found" });
261+ if (!c.connector || !c.type) return reply.status(400).send({ error: "candidate has no connector/type (legacy discovery row)" });
262+ const src = (await db.execute<Record<string, unknown>>(sql`select categories from sources where id = ${String(c.source_id)}`)).rows[0];
263+ const id = `${String(c.source_id)}_${String(c.name ?? c.kind_class ?? "page").toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "")}-${String(c.id).slice(-5)}`.slice(0, 80);
264+ await db.execute(sql`insert into sensors (id, source_id, name, url, type, connector, tier, config, status, priority, validated_at)
265+ values (${id}, ${String(c.source_id)}, ${String(c.name ?? c.kind_class ?? "page")}, ${String(c.url)}, ${String(c.type)}, ${String(c.connector)}, ${String(c.tier ?? "C")}, ${JSON.stringify({ ...((c.config as Record<string, unknown>) ?? {}), factory: true, shadow: true, seed: false, kind: c.kind_class, candidate: c.id, seedId: c.seed_id, shadowSince: new Date().toISOString(), targetPriority: 2, categories: src?.categories ?? [] })}::jsonb, 'SHADOW', 3, now())
266+ on conflict (id) do nothing`);
267+ await db.execute(sql`update discovery_candidates set status = 'shadow', shadow_sensor_id = ${id}, updated_at = now() where id = ${req.params.id}`);
268+ return { ok: true, sensor_id: id };
269+ });
270+
196 app.get("/api/v1/admin/failures", async () => { 271 app.get("/api/v1/admin/failures", async () => {
197 const rows = await db.execute<Record<string, unknown>>(sql`select r.started_at, r.outcome, r.http_status, r.error, r.duration_ms, s.id as sensor_id, s.name, s.source_id, s.url, s.connector from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error','rate_limited') and r.started_at >= now() - interval '24 hours' order by r.started_at desc limit 200`); 272 const rows = await db.execute<Record<string, unknown>>(sql`select r.started_at, r.outcome, r.http_status, r.error, r.duration_ms, s.id as sensor_id, s.name, s.source_id, s.url, s.connector from sensor_runs r join sensors s on s.id = r.sensor_id where r.outcome in ('error','parse_error','rate_limited') and r.started_at >= now() - interval '24 hours' order by r.started_at desc limit 200`);
198 return { items: rows.rows }; 273 return { items: rows.rows };
modified apps/api/src/routes.ts +10 −1
@@ -5,6 +5,7 @@ import { db, sql, textArray } from "@websensor/db";
5 import { getBlobStore } from "@websensor/store"; 5 import { getBlobStore } from "@websensor/store";
6 import { cached } from "./cache"; 6 import { cached } from "./cache";
7 import { config } from "./config"; 7 import { config } from "./config";
8+import { coverageSector, coverageSummary } from "./coverage";
8 import { breakingDesk, categoryDesk, clusterDetail, countryDesk, countryList, entityInsights, entityRankings, pulse, radar, sourceSensors } from "./intel"; 9 import { breakingDesk, categoryDesk, clusterDetail, countryDesk, countryList, entityInsights, entityRankings, pulse, radar, sourceSensors } from "./intel";
9 import { engineStatus, liveStats } from "./live"; 10 import { engineStatus, liveStats } from "./live";
10 import { countEvents, getEvent, listEvents, relatedEvents, sensorHistory, sourceActivity, sourceQuality, stats, trending } from "./queries"; 11 import { countEvents, getEvent, listEvents, relatedEvents, sensorHistory, sourceActivity, sourceQuality, stats, trending } from "./queries";
@@ -139,7 +140,8 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {
139 const q = req.query.q; 140 const q = req.query.q;
140 const rows = await db.execute<Record<string, unknown>>(sql` 141 const rows = await db.execute<Record<string, unknown>>(sql`
141 select s.id, s.name, s.domain, s.homepage, s.description, s.categories, s.tier, s.importance_weight, s.enabled, s.notes, s.first_party, s.country, s.language, s.robots_checked_at, 142 select s.id, s.name, s.domain, s.homepage, s.description, s.categories, s.tier, s.importance_weight, s.enabled, s.notes, s.first_party, s.country, s.language, s.robots_checked_at,
142− (select count(*) from sensors x where x.source_id = s.id and x.enabled)::int as sensor_count, 143+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.status <> 'SHADOW')::int as sensor_count,
144+ (select count(*) from sensors x where x.source_id = s.id and x.enabled and x.status = 'SHADOW')::int as shadow_count, s.origin, s.sector,
143 (select count(*) from events e where e.source_id = s.id)::int as event_count, 145 (select count(*) from events e where e.source_id = s.id)::int as event_count,
144 (select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h, 146 (select count(*) from events e where e.source_id = s.id and e.detected_at >= now() - interval '24 hours')::int as events_24h,
145 (select max(detected_at) from events e where e.source_id = s.id) as last_event_at, 147 (select max(detected_at) from events e where e.source_id = s.id) as last_event_at,
@@ -247,6 +249,13 @@ export async function registerRoutes(app: FastifyInstance): Promise<void> {
247 249
248 // ---- Intelligence desks ---------------------------------------------------------------------- 250 // ---- Intelligence desks ----------------------------------------------------------------------
249 app.get("/api/v1/stats", async () => cached("stats", 5_000, stats)); 251 app.get("/api/v1/stats", async () => cached("stats", 5_000, stats));
252+ // ---- Global Observation Coverage Score ------------------------------------------------------
253+ app.get("/api/v1/coverage", async () => coverageSummary());
254+ app.get<{ Params: { sector: string } }>("/api/v1/coverage/:sector", async (req, reply) => {
255+ const r = await coverageSector(req.params.sector.toLowerCase());
256+ if (!r) return reply.status(404).send({ error: "not_found" });
257+ return r;
258+ });
250 app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => { 259 app.get<{ Querystring: { hours?: string; limit?: string } }>("/api/v1/trending", async (req) => {
251 const hours = Math.min(168, Number(req.query.hours ?? 24)); 260 const hours = Math.min(168, Number(req.query.hours ?? 24));
252 const limit = Math.min(50, Number(req.query.limit ?? 12)); 261 const limit = Math.min(50, Number(req.query.limit ?? 12));
modified apps/engine/src/cli.ts +49 −1
@@ -80,8 +80,56 @@ async function main(): Promise<void> {
80 console.log(JSON.stringify(out, null, 2), `\n${Date.now() - t0} ms`); 80 console.log(JSON.stringify(out, null, 2), `\n${Date.now() - t0} ms`);
81 break; 81 break;
82 } 82 }
83+ case "expand": {
84+ // Deep discovery dry-run (no DB writes): what the Source Factory would find for a domain.
85+ const { discoverOrganization } = await import("@websensor/connectors");
86+ const { scoreCandidate } = await import("./factory/score");
87+ const hintsArg = args.find((a) => a.startsWith("--hints="));
88+ const hints = hintsArg ? (JSON.parse(hintsArg.slice(8)) as Record<string, unknown>) : {};
89+ const r = await discoverOrganization(args[0]!, { hints, probePosture: args.includes("--posture") });
90+ console.log(`${r.domain} → origin ${r.origin} · home ${r.homeStatus}${r.blocked ? " BLOCKED" : ""} · ${r.requests} requests · ${r.durationMs} ms`);
91+ for (const n of r.notes) console.log(` note: ${n}`);
92+ for (const c of r.candidates) {
93+ const s = scoreCandidate(c, { importance: Number(args.find((a) => a.startsWith("--importance="))?.slice(13) ?? 2) });
94+ console.log(`${s.score.toFixed(2)} ${c.kind.padEnd(10)} ${c.connector.padEnd(10)} ${c.suggestedTier} ${String(c.itemCount ?? "").padStart(4)} ${c.url} (${c.evidence})`);
95+ }
96+ if (r.rejected.length) console.log(`rejected: ${r.rejected.map((x) => `${x.url} [${x.reason}]`).join("\n ")}`);
97+ break;
98+ }
99+ case "factory": {
100+ const f = await import("./factory");
101+ const sub = args[0];
102+ const flagVal = (n: string): string | undefined => args.find((a) => a.startsWith(`--${n}=`))?.slice(n.length + 3);
103+ if (sub === "seed") {
104+ const files = await f.seedFromFiles();
105+ const mode = (flagVal("mode") ?? "hinted") as "uncovered" | "hinted" | "all";
106+ const cov = await f.seedFromCoverage({ mode, sectors: flagVal("sector")?.split(","), requeue: args.includes("--requeue") });
107+ console.log(JSON.stringify({ files, coverage: cov }, null, 2));
108+ } else if (sub === "run") {
109+ const n = Number(flagVal("n") ?? 6);
110+ const ids = args.slice(1).filter((a) => !a.startsWith("--"));
111+ const out = await f.runFactoryBatch(n, ids.length ? { seedIds: ids } : {});
112+ for (const o of out) console.log(`${o.status.padEnd(10)} ${o.seedId.padEnd(40)} source=${o.sourceId ?? "-"} candidates=${o.candidates} shadow=${o.shadow} requests=${o.requests} ${o.durationMs} ms${o.notes.length ? " · " + o.notes.join(" · ") : ""}`);
113+ } else if (sub === "evaluate") {
114+ const r = await f.evaluateShadows({ force: args.includes("--force") });
115+ for (const d of r.decisions.filter((x) => x.decision !== "deferred" || args.includes("--verbose"))) console.log(`${d.decision.padEnd(9)} ${d.sensorId.padEnd(60)} ${d.reason}`);
116+ console.log(JSON.stringify({ accepted: r.accepted, rejected: r.rejected, deferred: r.deferred }));
117+ } else if (sub === "stats") {
118+ console.log(JSON.stringify(await f.factoryStats(), null, 2));
119+ } else if (sub === "export") {
120+ process.stdout.write(await f.exportFactoryFragment({ sector: flagVal("sector"), sinceDays: flagVal("since") ? Number(flagVal("since")) : undefined, includeShadow: args.includes("--include-shadow") }));
121+ } else if (sub === "requeue") {
122+ const ids = args.slice(1).filter((a) => !a.startsWith("--"));
123+ const r = await db.execute(sql`update factory_seeds set status = 'queued', updated_at = now() where ${ids.length ? sql`id = any(${sql.raw("array[" + ids.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]")})` : sql`status in ('error','blocked')`}`);
124+ console.log(`requeued ${r.rowCount ?? 0}`);
125+ } else {
126+ console.error("usage: cli.ts factory seed [--mode=uncovered|hinted|all] [--sector=a,b] [--requeue] | run [seedId…] [--n=6] | evaluate [--force] [--verbose] | stats | export [--sector=x] [--since=days] [--include-shadow] | requeue [seedId…]");
127+ process.exitCode = 1;
128+ }
129+ break;
130+ }
83 default: 131 default:
84− console.error("usage: cli.ts sync | discover [sourceId…] | probe <domain> | run-once <sensorId> | run-due [n] | relink-entities [days] | prune-blobs [batch] | refresh-clusters"); 132+ console.error("usage: cli.ts sync | discover [sourceId…] | probe <domain> | expand <domain> [--hints=json] [--posture] | run-once <sensorId> | run-due [n] | relink-entities [days] | prune-blobs [batch] | refresh-clusters | factory <seed|run|evaluate|stats|export|requeue>");
85 process.exitCode = 1; 133 process.exitCode = 1;
86 } 134 }
87 } 135 }
modified apps/engine/src/config.ts +22 −1
@@ -48,7 +48,7 @@ export const config = {
48 retention: { rawDays: Number(env.WS_RETENTION_RAW_DAYS ?? 21), changedRawDays: Number(env.WS_RETENTION_CHANGED_RAW_DAYS ?? 60), enabled: (env.WS_RETENTION ?? "1") !== "0" }, 48 retention: { rawDays: Number(env.WS_RETENTION_RAW_DAYS ?? 21), changedRawDays: Number(env.WS_RETENTION_CHANGED_RAW_DAYS ?? 60), enabled: (env.WS_RETENTION ?? "1") !== "0" },
49 /** silent-change bar: only eligible types with importance ≥ this become "silent" */ 49 /** silent-change bar: only eligible types with importance ≥ this become "silent" */
50 silentMinImportance: Number(env.WS_SILENT_MIN_IMPORTANCE ?? 45), 50 silentMinImportance: Number(env.WS_SILENT_MIN_IMPORTANCE ?? 45),
51− version: "0.2.0", 51+ version: "0.3.0",
52 }; 52 };
53 53
54 export const log = pino({ 54 export const log = pino({
@@ -56,3 +56,24 @@ export const log = pino({
56 ...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}), 56 ...(config.env !== "production" ? { transport: { target: "pino-pretty", options: { colorize: true, translateTime: "HH:MM:ss" } } } : {}),
57 base: { service: "websensor-engine" }, 57 base: { service: "websensor-engine" },
58 }); 58 });
59+
60+/** Source Factory (apps/engine/src/factory) — organization expansion, shadow monitoring, acceptance. */
61+export const factoryConfig = {
62+ enabled: (env.WS_FACTORY ?? "1") !== "0",
63+ /** seeds discovered in parallel */
64+ concurrency: Number(env.WS_FACTORY_CONCURRENCY ?? 6),
65+ /** HTTP requests per organization during discovery */
66+ budget: Number(env.WS_FACTORY_BUDGET ?? 90),
67+ /** minimum candidate score to enter shadow monitoring */
68+ minScore: Number(env.WS_FACTORY_MIN_SCORE ?? 0.35),
69+ /** shadow evaluation: checks needed, minimum and maximum shadow duration */
70+ shadowMinChecks: Number(env.WS_FACTORY_SHADOW_MIN_CHECKS ?? 5),
71+ shadowMinHours: Number(env.WS_FACTORY_SHADOW_MIN_HOURS ?? 24),
72+ shadowMaxHours: Number(env.WS_FACTORY_SHADOW_MAX_HOURS ?? 120),
73+ /** per-organization promotion caps */
74+ caps: { feeds: 4, pages: 6, releases: 4, sitemap: 1, status: 1, models: 1, filings: 1, api: 1, posture: 3, legal: 2, total: 18 },
75+ /** seconds between factory ticks */
76+ tickSeconds: Number(env.WS_FACTORY_TICK_SECONDS ?? 20),
77+ coverageDir: env.WS_COVERAGE_DIR ?? "./config/coverage",
78+ seedsDir: env.WS_FACTORY_SEEDS_DIR ?? "./config/factory/seeds",
79+};
added apps/engine/src/coverage-validate.ts +108 −0
@@ -0,0 +1,108 @@
1+import { promises as dns } from "node:dns";
2+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
3+import { join } from "node:path";
4+import YAML from "yaml";
5+import { coverageKey, coverageSectorSchema, type CoverageSector } from "@websensor/core";
6+
7+/**
8+ * Coverage-universe validator — no database.
9+ *
10+ * node node_modules/tsx/dist/cli.mjs apps/engine/src/coverage-validate.ts [config/coverage/<file>.yaml …] [--dns] [--quiet]
11+ *
12+ * Without arguments every file in config/coverage is checked. Reports schema issues, duplicate domains inside a
13+ * file, duplicate sector keys across files and (with --dns) domains that do not resolve. Exit code 1 on any error.
14+ */
15+const args = process.argv.slice(2);
16+const flag = (n: string): boolean => args.includes(`--${n}`);
17+const files = args.filter((a) => !a.startsWith("--"));
18+
19+export function loadCoverageFiles(dir = process.env.WS_COVERAGE_DIR ?? "./config/coverage"): { file: string; sector: CoverageSector | null; issues: string[] }[] {
20+ const list = files.length ? files : existsSync(dir) && statSync(dir).isDirectory() ? readdirSync(dir).filter((f) => /\.ya?ml$/.test(f) && !f.startsWith("_")).sort().map((f) => join(dir, f)) : [];
21+ return list.map((file) => {
22+ const issues: string[] = [];
23+ let sector: CoverageSector | null = null;
24+ try {
25+ const raw = YAML.parse(readFileSync(file, "utf8"));
26+ const parsed = coverageSectorSchema.safeParse(raw);
27+ if (!parsed.success) {
28+ for (const i of parsed.error.issues.slice(0, 40)) issues.push(`${i.path.join(".")}: ${i.message}`);
29+ if (parsed.error.issues.length > 40) issues.push(`… ${parsed.error.issues.length - 40} more issues`);
30+ } else sector = parsed.data;
31+ } catch (e) {
32+ issues.push(`YAML parse error: ${(e as Error).message}`);
33+ }
34+ if (sector) {
35+ const seen = new Map<string, string>();
36+ const keys = new Set<string>();
37+ for (const u of sector.universes) {
38+ if (keys.has(u.key)) issues.push(`duplicate universe key ${u.key}`);
39+ keys.add(u.key);
40+ if (!u.members.length) issues.push(`universe ${u.key} has no members`);
41+ for (const m of u.members) {
42+ if (!/^[a-z0-9.-]+\.[a-z]{2,}$/.test(m.domain) || m.domain.includes("..")) issues.push(`${u.key}: bad domain "${m.domain}" (${m.name})`);
43+ const k = coverageKey(m.domain);
44+ const prev = seen.get(k);
45+ if (prev && prev !== `${u.key}/${m.name}`) issues.push(`${u.key}: duplicate domain ${m.domain} (${m.name}) — already ${prev}`);
46+ seen.set(k, `${u.key}/${m.name}`);
47+ if (!m.country && !/^(INT|EU)$/.test(String(m.country))) issues.push(`${u.key}: ${m.name} has no country`);
48+ }
49+ }
50+ }
51+ return { file, sector, issues };
52+ });
53+}
54+
55+async function resolves(domain: string): Promise<boolean> {
56+ for (const h of [domain, `www.${domain}`]) {
57+ try {
58+ const r = await dns.lookup(h, { all: true });
59+ if (r.length) return true;
60+ } catch {
61+ // try next
62+ }
63+ }
64+ return false;
65+}
66+
67+async function main(): Promise<void> {
68+ const loaded = loadCoverageFiles();
69+ if (!loaded.length) {
70+ console.error("no coverage files found");
71+ process.exitCode = 1;
72+ return;
73+ }
74+ let errors = 0;
75+ const sectors = new Map<string, string>();
76+ for (const l of loaded) {
77+ const members = l.sector?.universes.reduce((n, u) => n + u.members.length, 0) ?? 0;
78+ const warnIssues = l.issues.filter((i) => / has no country$/.test(i));
79+ const hardIssues = l.issues.filter((i) => !/ has no country$/.test(i));
80+ errors += hardIssues.length;
81+ // Several files may extend the same sector (e.g. corporate indices per region); universes are merged by the loader.
82+ if (l.sector && !sectors.has(l.sector.sector)) sectors.set(l.sector.sector, l.file);
83+ console.log(`${hardIssues.length ? "FAIL" : "OK "} ${l.file} sector=${l.sector?.sector ?? "-"} universes=${l.sector?.universes.length ?? 0} members=${members}${warnIssues.length ? ` warn=${warnIssues.length}` : ""}`);
84+ for (const i of hardIssues) console.log(` ${i}`);
85+ if (!flag("quiet")) for (const i of warnIssues.slice(0, 15)) console.log(` WARN ${i}`);
86+ if (flag("dns") && l.sector) {
87+ const domains = [...new Set(l.sector.universes.flatMap((u) => u.members.map((m) => m.domain)))];
88+ let i = 0;
89+ const bad: string[] = [];
90+ await Promise.all(
91+ Array.from({ length: 16 }, async () => {
92+ while (i < domains.length) {
93+ const d = domains[i++]!;
94+ if (!(await resolves(d))) bad.push(d);
95+ }
96+ }),
97+ );
98+ for (const d of bad.sort()) console.log(` DNS ${d} does not resolve`);
99+ if (bad.length) console.log(` ${bad.length}/${domains.length} domains do not resolve (fix or remove them)`);
100+ errors += bad.length;
101+ }
102+ }
103+ const total = loaded.reduce((n, l) => n + (l.sector?.universes.reduce((m, u) => m + u.members.length, 0) ?? 0), 0);
104+ console.log(`\n${loaded.length} files · ${sectors.size} sectors · ${total} members → ${errors ? `${errors} errors` : "all good"}`);
105+ if (errors) process.exitCode = 1;
106+}
107+
108+if (process.argv[1] && /coverage-validate\.ts$/.test(process.argv[1])) void main();
added apps/engine/src/factory-main.ts +99 −0
@@ -0,0 +1,99 @@
1+import { closeDb, db, migrate, sql } from "@websensor/db";
2+import { closeDispatcher } from "@websensor/connectors";
3+import { config, factoryConfig, log } from "./config";
4+import { evaluateShadows, factoryStats, runFactoryBatch, seedFromCoverage, seedFromFiles } from "./factory";
5+import { closeRedis, getRedis } from "./redis";
6+
7+/**
8+ * Source Factory process (`websensor-factory`): a slow, polite background worker separate from the poller.
9+ * every tick — claim queued seeds (systemic organizations first) and run deep discovery + shadow creation
10+ * every 15 min — evaluate shadow sensors (accept / reject / defer)
11+ * every 24 h — re-seed from the coverage universes (new members become seeds automatically)
12+ * Heartbeat: Redis `ws:factory:status`.
13+ */
14+async function heartbeat(extra: Record<string, unknown>): Promise<void> {
15+ try {
16+ await getRedis().set("ws:factory:status", JSON.stringify({ ...extra, at: new Date().toISOString(), pid: process.pid, version: config.version }), "EX", 120);
17+ } catch {
18+ // best effort
19+ }
20+}
21+
22+async function main(): Promise<void> {
23+ log.info({ env: config.env, version: config.version, concurrency: factoryConfig.concurrency, budget: factoryConfig.budget }, "websensor factory starting");
24+ const applied = await migrate(config.databaseUrl);
25+ if (applied.length) log.info({ applied }, "migrations applied");
26+ if (!factoryConfig.enabled) {
27+ log.warn("WS_FACTORY=0 — factory idle");
28+ setInterval(() => void heartbeat({ idle: true }), 60_000);
29+ return;
30+ }
31+ let stopping = false;
32+ let busy = false;
33+ let processed = 0;
34+ let lastSeed = 0;
35+ let lastEval = 0;
36+
37+ const seed = async (): Promise<void> => {
38+ try {
39+ const a = await seedFromFiles();
40+ const b = await seedFromCoverage({ mode: "hinted" });
41+ log.info({ files: a, coverage: { members: b.members, seeds: b.seeds, inserted: b.inserted, issues: b.issues.length } }, "factory: seeds refreshed");
42+ for (const i of b.issues) log.warn({ issue: i }, "coverage file issue");
43+ } catch (e) {
44+ log.error({ err: (e as Error).message }, "factory: seeding failed");
45+ }
46+ };
47+
48+ const tick = async (): Promise<void> => {
49+ if (stopping || busy) return;
50+ busy = true;
51+ try {
52+ if (Date.now() - lastSeed > 24 * 3600e3) {
53+ lastSeed = Date.now();
54+ await seed();
55+ }
56+ if (Date.now() - lastEval > 15 * 60e3) {
57+ lastEval = Date.now();
58+ await evaluateShadows().catch((e) => log.warn({ err: (e as Error).message }, "factory: shadow evaluation failed"));
59+ }
60+ const out = await runFactoryBatch(factoryConfig.concurrency);
61+ processed += out.length;
62+ const queued = Number((await db.execute<{ n: string }>(sql`select count(*)::text as n from factory_seeds where status = 'queued'`)).rows[0]?.n ?? 0);
63+ await heartbeat({ processed, queued, lastBatch: out.map((o) => ({ seed: o.seedId, status: o.status, candidates: o.candidates, shadow: o.shadow, requests: o.requests, ms: o.durationMs })), concurrency: factoryConfig.concurrency });
64+ if (!out.length && processed % 50 === 0) {
65+ const st = await factoryStats();
66+ log.info({ seeds: st.seeds, candidates: st.candidates, shadow: st.shadow }, "factory: idle");
67+ }
68+ } catch (e) {
69+ log.error({ err: (e as Error).stack ?? (e as Error).message }, "factory tick failed");
70+ } finally {
71+ busy = false;
72+ }
73+ };
74+
75+ await heartbeat({ starting: true });
76+ void tick();
77+ const timer = setInterval(() => void tick(), factoryConfig.tickSeconds * 1000);
78+
79+ const shutdown = async (signal: string): Promise<void> => {
80+ if (stopping) return;
81+ stopping = true;
82+ log.info({ signal }, "factory shutting down");
83+ clearInterval(timer);
84+ const deadline = Date.now() + 60_000;
85+ while (busy && Date.now() < deadline) await new Promise((r) => setTimeout(r, 250));
86+ await closeDispatcher();
87+ await closeRedis();
88+ await closeDb();
89+ process.exit(0);
90+ };
91+ process.on("SIGINT", () => void shutdown("SIGINT"));
92+ process.on("SIGTERM", () => void shutdown("SIGTERM"));
93+ process.on("unhandledRejection", (e) => log.error({ err: e instanceof Error ? e.stack : String(e) }, "unhandled rejection"));
94+}
95+
96+main().catch((e) => {
97+ log.fatal({ err: (e as Error).stack ?? String(e) }, "factory failed to start");
98+ process.exit(1);
99+});
added apps/engine/src/factory/export.ts +46 −0
@@ -0,0 +1,46 @@
1+import YAML from "yaml";
2+import { db, sql } from "@websensor/db";
3+
4+/**
5+ * Export factory-created sources and accepted sensors as a registry fragment (reviewable YAML). Graduating a
6+ * fragment into `config/sources.d/` makes the registry sync own those sensors (`seed: true`) — the Factory never
7+ * re-creates them because their URLs already exist.
8+ */
9+export async function exportFactoryFragment(opts: { sector?: string; sinceDays?: number; includeShadow?: boolean } = {}): Promise<string> {
10+ const rows = await db.execute<Record<string, unknown>>(sql`
11+ select so.id, so.name, so.domain, so.homepage, so.categories, so.tier, so.importance_weight, so.country, so.language, so.first_party, so.origin, so.sector, so.notes,
12+ coalesce(json_agg(json_build_object('name', s.name, 'url', s.url, 'type', s.type, 'connector', s.connector, 'tier', s.tier, 'config', s.config - 'factory' - 'shadow' - 'seed' - 'candidate' - 'seedId' - 'shadowSince' - 'targetPriority' - 'acceptedAt' - 'shadowReport' - 'kind') order by s.name) filter (where s.id is not null), '[]'::json) as sensors
13+ from sources so
14+ left join sensors s on s.source_id = so.id and s.enabled and (s.config->>'factory') = 'true' and (s.status <> 'SHADOW' or ${opts.includeShadow ?? false})
15+ where (so.origin = 'factory' or exists (select 1 from sensors y where y.source_id = so.id and (y.config->>'factory') = 'true'))
16+ ${opts.sector ? sql`and so.sector = ${opts.sector}` : sql``}
17+ ${opts.sinceDays ? sql`and so.updated_at >= now() - make_interval(days => ${opts.sinceDays})` : sql``}
18+ group by so.id order by so.sector, so.id`);
19+ const entities = (await db.execute<{ source_id: string; alias: string }>(sql`select se.source_id, ea.alias from source_entities se join entity_aliases ea on ea.entity_id = se.entity_id where se.entity_id like 'org_%'`)).rows;
20+ const aliasBySource = new Map<string, string[]>();
21+ for (const e of entities) aliasBySource.set(e.source_id, [...(aliasBySource.get(e.source_id) ?? []), e.alias]);
22+ const out: Record<string, unknown>[] = [];
23+ for (const r of rows.rows) {
24+ const sensors = (r.sensors as Record<string, unknown>[]).map((s) => {
25+ const cfg = s.config as Record<string, unknown>;
26+ const o: Record<string, unknown> = { name: s.name, url: s.url, type: s.type, connector: s.connector, tier: s.tier };
27+ if (cfg && Object.keys(cfg).length) o.config = cfg;
28+ return o;
29+ });
30+ if (!sensors.length) continue;
31+ const aliases = (aliasBySource.get(String(r.id)) ?? []).filter((a) => a !== String(r.name).toLowerCase() && a !== String(r.domain).toLowerCase() && a !== String(r.domain).replace(/^www\./, "").toLowerCase());
32+ if (r.origin === "factory") {
33+ const src: Record<string, unknown> = { id: r.id, name: r.name, domain: r.domain, categories: r.categories, tier: r.tier };
34+ if (Number(r.importance_weight) !== 1) src.weight = Number(r.importance_weight);
35+ if (aliases.length) src.aliases = aliases;
36+ if (r.country) src.country = r.country;
37+ if (r.language) src.language = r.language;
38+ if (r.first_party === false) src.first_party = false;
39+ if (r.notes) src.notes = r.notes;
40+ src.sensors = sensors;
41+ out.push(src);
42+ } else out.push({ id: r.id, extend: true, sensors });
43+ }
44+ const header = `# WebSensor — Source Factory export (${new Date().toISOString().slice(0, 10)}${opts.sector ? `, sector ${opts.sector}` : ""}): ${out.length} organizations, ${out.reduce((n, s) => n + (s.sensors as unknown[]).length, 0)} accepted sensors.\n# Generated by \`cli.ts factory export\`; review, then place in config/sources.d/ to let the registry own these sensors.\n`;
45+ return header + YAML.stringify({ sources: out }, { lineWidth: 0, defaultStringType: "PLAIN", defaultKeyType: "PLAIN" });
46+}
added apps/engine/src/factory/factory.test.ts +67 −0
@@ -0,0 +1,67 @@
1+import { describe, expect, it } from "vitest";
2+import { coverageKey, registrableDomain } from "@websensor/core";
3+import type { DeepCandidate } from "@websensor/connectors";
4+import { itemsPerDayOf } from "@websensor/connectors";
5+import { KIND_WEIGHT, scoreCandidate, selectForShadow } from "./score";
6+
7+const cand = (over: Partial<DeepCandidate>): DeepCandidate => ({ url: "https://example.org/feed", type: "RSS", connector: "rss", kind: "news", name: "news feed", config: {}, suggestedTier: "B", evidence: "well-known path", value: 0.8, itemsPerDay: 0.5, firstPartyConfidence: 1, fetchCost: { bytes: 20_000, ms: 400 }, ...over });
8+
9+describe("registrable domain (coverage key)", () => {
10+ it("collapses hosts to the organization's registrable domain", () => {
11+ expect(registrableDomain("ir.apple.com")).toBe("apple.com");
12+ expect(registrableDomain("www.ons.gov.uk")).toBe("ons.gov.uk");
13+ expect(registrableDomain("www.bankofengland.co.uk")).toBe("bankofengland.co.uk");
14+ expect(registrableDomain("statcan.gc.ca")).toBe("statcan.gc.ca");
15+ expect(registrableDomain("open.canada.ca")).toBe("canada.ca");
16+ expect(registrableDomain("boj.or.jp")).toBe("boj.or.jp");
17+ expect(coverageKey("https://WWW.Example.COM/path")).toBe("example.com");
18+ });
19+});
20+
21+describe("factory scoring", () => {
22+ it("weights page classes and organization importance", () => {
23+ const status = scoreCandidate(cand({ kind: "status", connector: "statuspage", value: 0.95, itemsPerDay: null }), { importance: 2 });
24+ const careers = scoreCandidate(cand({ kind: "careers", connector: "http", value: 0.35, itemsPerDay: null }), { importance: 2 });
25+ expect(status.score).toBeGreaterThan(careers.score);
26+ expect(KIND_WEIGHT.status).toBeGreaterThan(KIND_WEIGHT.careers);
27+ const big = scoreCandidate(cand({}), { importance: 3 });
28+ const small = scoreCandidate(cand({}), { importance: 1 });
29+ expect(big.score).toBeGreaterThan(small.score);
30+ expect(big.reasons.some((r) => /importance 3/.test(r))).toBe(true);
31+ });
32+ it("penalises firehoses, dormant feeds and slow endpoints, explainably", () => {
33+ const base = scoreCandidate(cand({}), { importance: 2 }).score;
34+ expect(scoreCandidate(cand({ itemsPerDay: 200 }), { importance: 2 }).score).toBeLessThan(base);
35+ expect(scoreCandidate(cand({ itemsPerDay: 0.001 }), { importance: 2 }).score).toBeLessThan(base);
36+ const slow = scoreCandidate(cand({ fetchCost: { bytes: 100, ms: 12_000 } }), { importance: 2 });
37+ expect(slow.score).toBeLessThan(base);
38+ expect(slow.reasons.some((r) => /slow/.test(r))).toBe(true);
39+ });
40+ it("applies per-organization caps and the minimum score", () => {
41+ const feeds = Array.from({ length: 6 }, (_, i) => scoreCandidate(cand({ url: `https://example.org/f${i}`, value: 0.9 - i * 0.02 }), { importance: 2 }));
42+ const low = scoreCandidate(cand({ url: "https://example.org/careers", kind: "careers", connector: "http", value: 0.2, itemsPerDay: null }), { importance: 1 });
43+ const legal = ["terms", "privacy", "cookies"].map((p) => scoreCandidate(cand({ url: `https://example.org/${p}`, kind: "legal", connector: "http", type: "HTML", value: 0.7, itemsPerDay: null }), { importance: 2 }));
44+ const { selected, skipped } = selectForShadow([...feeds, low, ...legal], { kinds: new Map(), total: 0 });
45+ expect(selected.filter((s) => s.cand.connector === "rss")).toHaveLength(4);
46+ expect(selected.filter((s) => s.cand.kind === "legal")).toHaveLength(2);
47+ expect(skipped.some((s) => s.s === low && /score/.test(s.reason))).toBe(true);
48+ expect(skipped.some((s) => /feeds cap/.test(s.reason))).toBe(true);
49+ });
50+ it("respects sensors the organization already has", () => {
51+ const feed = scoreCandidate(cand({}), { importance: 2 });
52+ const { selected, skipped } = selectForShadow([feed], { kinds: new Map([["news", 4]]), total: 4 });
53+ expect(selected).toHaveLength(0);
54+ expect(skipped[0]?.reason).toMatch(/feeds cap/);
55+ });
56+});
57+
58+describe("expected change frequency", () => {
59+ it("uses the median gap and survives one bogus date", () => {
60+ const day = 86400e3;
61+ const now = Date.UTC(2026, 8, 13);
62+ const dates = [now, now - day, now - 2 * day, now - 3 * day, now - 4 * day, now - 5000 * day];
63+ expect(itemsPerDayOf(dates)).toBe(1);
64+ expect(itemsPerDayOf([now, now - day])).toBeNull();
65+ expect(itemsPerDayOf([now, now - 30 * day, now - 60 * day, now - 90 * day])).toBeCloseTo(0.033, 2);
66+ });
67+});
added apps/engine/src/factory/index.ts +5 −0
@@ -0,0 +1,5 @@
1+export * from "./seeds";
2+export * from "./score";
3+export * from "./run";
4+export * from "./shadow";
5+export * from "./export";
added apps/engine/src/factory/run.ts +288 −0
@@ -0,0 +1,288 @@
1+import { coverageKey, newId, slugify, sourceSchema, type SourceSeed, type Tier } from "@websensor/core";
2+import { discoverOrganization, type CandidateKind, type DeepDiscoveryHints } from "@websensor/connectors";
3+import { db, factorySeeds, sensors, sql, type FactorySeed } from "@websensor/db";
4+import { factoryConfig, log } from "../config";
5+import { priorityFor, upsertSourceRecord } from "../registry";
6+import { scoreCandidate, selectForShadow } from "./score";
7+
8+/**
9+ * Source Factory — one organization at a time:
10+ * seed → deep discovery → (create or attach the source) → score candidates → promotion policy →
11+ * shadow sensors (`status = SHADOW`: observed, evidence stored, nothing published) → later `evaluateShadows()`.
12+ */
13+export interface ProcessOutcome {
14+ seedId: string;
15+ sourceId: string | null;
16+ status: "discovered" | "blocked" | "error";
17+ candidates: number;
18+ shadow: number;
19+ requests: number;
20+ durationMs: number;
21+ notes: string[];
22+}
23+
24+interface SourceRow extends Record<string, unknown> {
25+ id: string;
26+ domain: string;
27+ categories: string[];
28+ tier: string;
29+ enabled: boolean;
30+}
31+
32+let sourceIndex: { at: number; byKey: Map<string, SourceRow>; ids: Set<string> } | null = null;
33+async function sourcesByDomain(force = false): Promise<{ byKey: Map<string, SourceRow>; ids: Set<string> }> {
34+ if (!force && sourceIndex && Date.now() - sourceIndex.at < 5 * 60_000) return sourceIndex;
35+ const rows = await db.execute<SourceRow>(sql`select id, domain, categories, tier, enabled from sources where kind = 'registry'`);
36+ const byKey = new Map<string, SourceRow>();
37+ const ids = new Set<string>();
38+ for (const r of rows.rows) {
39+ ids.add(r.id);
40+ const k = coverageKey(r.domain);
41+ if (!byKey.has(k)) byKey.set(k, r);
42+ }
43+ sourceIndex = { at: Date.now(), byKey, ids };
44+ return sourceIndex;
45+}
46+
47+export async function processSeed(seed: FactorySeed): Promise<ProcessOutcome> {
48+ const started = Date.now();
49+ const notes: string[] = [];
50+ const hints = (seed.hints ?? {}) as DeepDiscoveryHints & { posture?: boolean };
51+ const idx = await sourcesByDomain();
52+ const key = coverageKey(seed.domain);
53+ let source = idx.byKey.get(key) ?? null;
54+
55+ const disc = await discoverOrganization(seed.domain, {
56+ hints,
57+ budget: factoryConfig.budget,
58+ concurrency: 4,
59+ sensorIdForLogs: `factory_${seed.id}`,
60+ probePages: true,
61+ probeSubdomains: true,
62+ probePosture: seed.importance >= 3 || hints.posture === true,
63+ probeOpenApi: true,
64+ });
65+ notes.push(...disc.notes);
66+ // The homepage may redirect to another registrable domain that we already monitor.
67+ if (!source && disc.canonicalDomain !== key) source = idx.byKey.get(disc.canonicalDomain) ?? null;
68+
69+ if (!disc.candidates.length) {
70+ const status = disc.blocked ? "blocked" : "discovered";
71+ await db.execute(sql`update factory_seeds set status = ${status}, candidates = 0, shadow = 0, source_id = ${source?.id ?? null}, discovered_at = now(), last_error = ${disc.blocked ? `homepage ${disc.homeStatus} (anti-bot)` : null}, updated_at = now() where id = ${seed.id}`);
72+ await bumpFactoryDaily({ seeds_processed: 1, requests: disc.requests, blocked: disc.blocked ? 1 : 0 });
73+ return { seedId: seed.id, sourceId: source?.id ?? null, status, candidates: 0, shadow: 0, requests: disc.requests, durationMs: Date.now() - started, notes };
74+ }
75+
76+ // ---- source: attach or create --------------------------------------------------------------------------
77+ let sourceId: string;
78+ if (source) {
79+ sourceId = source.id;
80+ if (!source.enabled) notes.push("source disabled in registry — candidates recorded, no shadow sensors");
81+ } else {
82+ sourceId = idx.ids.has(seed.id) ? `${seed.id}-${slugify(seed.domain.split(".")[0] ?? "x")}`.slice(0, 70) : seed.id;
83+ const seedDoc: SourceSeed = sourceSchema.parse({
84+ id: sourceId,
85+ name: seed.name,
86+ domain: seed.domain,
87+ homepage: seed.homepage ?? `https://${seed.domain}`,
88+ categories: seed.categories,
89+ tier: (seed.tier as Tier) ?? "B",
90+ weight: seed.weight,
91+ aliases: seed.aliases,
92+ country: seed.country ?? undefined,
93+ language: seed.language ?? undefined,
94+ first_party: seed.firstParty,
95+ notes: `Source Factory · ${seed.sector ?? "-"}/${seed.universe ?? "-"} · discovered ${new Date().toISOString().slice(0, 10)}`,
96+ llm: true,
97+ });
98+ await upsertSourceRecord(seedDoc, { origin: "factory", sector: seed.sector });
99+ idx.ids.add(sourceId);
100+ idx.byKey.set(key, { id: sourceId, domain: seed.domain, categories: seed.categories, tier: seed.tier, enabled: true });
101+ source = idx.byKey.get(key)!;
102+ }
103+
104+ // ---- existing sensors (this source + global URL uniqueness) ----------------------------------------------
105+ const existing = await db.execute<{ id: string; url: string; type: string; connector: string; config: Record<string, unknown>; status: string }>(sql`select id, url, type, connector, config, status from sensors where source_id = ${sourceId}`);
106+ const existingUrls = new Set(existing.rows.map((r) => r.url.replace(/\/$/, "")));
107+ const existingKinds = new Map<CandidateKind, number>();
108+ let existingTotal = 0;
109+ for (const r of existing.rows) {
110+ if (r.status === "DISABLED") continue;
111+ existingTotal++;
112+ const k = kindOfExisting(r);
113+ existingKinds.set(k, (existingKinds.get(k) ?? 0) + 1);
114+ }
115+ const urls = disc.candidates.map((c) => c.url);
116+ const global = urls.length ? await db.execute<{ url: string; source_id: string }>(sql`select url, source_id from sensors where url = any(${sql.raw("array[" + urls.map((u) => "'" + u.replace(/'/g, "''") + "'").join(",") + "]::text[]")}) and source_id <> ${sourceId}`) : { rows: [] as { url: string; source_id: string }[] };
117+ const ownedElsewhere = new Map(global.rows.map((r) => [r.url.replace(/\/$/, ""), r.source_id]));
118+
119+ // ---- score + select ---------------------------------------------------------------------------------------
120+ const scored = disc.candidates.map((c) => scoreCandidate(c, { importance: seed.importance, weight: seed.weight }));
121+ const fresh = scored.filter((s) => !existingUrls.has(s.cand.url.replace(/\/$/, "")) && !ownedElsewhere.has(s.cand.url.replace(/\/$/, "")));
122+ const { selected, skipped } = source.enabled ? selectForShadow(fresh, { kinds: existingKinds, total: existingTotal }) : { selected: [], skipped: fresh.map((s) => ({ s, reason: "source disabled" })) };
123+
124+ // ---- persist candidates ------------------------------------------------------------------------------------
125+ const day = new Date().toISOString().slice(0, 10);
126+ let shadowCreated = 0;
127+ for (const s of scored) {
128+ const k = s.cand.url.replace(/\/$/, "");
129+ let status = "candidate";
130+ let reason: string | null = null;
131+ if (existingUrls.has(k)) {
132+ status = "duplicate";
133+ reason = "already a sensor of this source";
134+ } else if (ownedElsewhere.has(k)) {
135+ status = "duplicate";
136+ reason = `already a sensor of ${ownedElsewhere.get(k)}`;
137+ } else if (selected.includes(s)) status = "shadow";
138+ else {
139+ const sk = skipped.find((x) => x.s === s);
140+ status = sk && /score/.test(sk.reason) ? "rejected" : "candidate";
141+ reason = sk?.reason ?? null;
142+ }
143+ const candId = newId("cand");
144+ let shadowSensorId: string | null = null;
145+ if (status === "shadow") {
146+ shadowSensorId = await createShadowSensor(sourceId, source, s.cand, candId, seed);
147+ if (shadowSensorId) shadowCreated++;
148+ else {
149+ status = "duplicate";
150+ reason = "sensor id/url collision";
151+ }
152+ }
153+ await db.execute(sql`
154+ insert into discovery_candidates (id, source_id, url, kind, evidence, score, status, seed_id, connector, type, name, kind_class, config, tier, score_value, reason, shadow_sensor_id, found_at, updated_at)
155+ values (${candId}, ${sourceId}, ${s.cand.url}, ${s.cand.type}, ${s.cand.evidence}, ${JSON.stringify({ value: s.cand.value, score: s.score, reasons: s.reasons, itemCount: s.cand.itemCount ?? null, itemsPerDay: s.cand.itemsPerDay ?? null, firstParty: s.cand.firstPartyConfidence, cost: s.cand.fetchCost, title: s.cand.title ?? null, fingerprint: s.cand.fingerprint ?? null })}::jsonb, ${status}, ${seed.id}, ${s.cand.connector}, ${s.cand.type}, ${s.cand.name}, ${s.cand.kind}, ${JSON.stringify(s.cand.config)}::jsonb, ${s.cand.suggestedTier}, ${s.score}, ${reason}, ${shadowSensorId}, now(), now())
156+ on conflict (source_id, url) do update set evidence = excluded.evidence, score = excluded.score, score_value = excluded.score_value, seed_id = excluded.seed_id, connector = excluded.connector, type = excluded.type, name = excluded.name, kind_class = excluded.kind_class, config = excluded.config, tier = excluded.tier, updated_at = now(),
157+ status = case when discovery_candidates.status in ('accepted','rejected','shadow') then discovery_candidates.status else excluded.status end,
158+ reason = case when discovery_candidates.status in ('accepted','rejected','shadow') then discovery_candidates.reason else excluded.reason end,
159+ shadow_sensor_id = coalesce(discovery_candidates.shadow_sensor_id, excluded.shadow_sensor_id)`);
160+ }
161+ for (const r of disc.rejected.slice(0, 40)) {
162+ await db.execute(sql`insert into discovery_candidates (id, source_id, url, kind, evidence, status, seed_id, reason, kind_class) values (${newId("cand")}, ${sourceId}, ${r.url}, 'HTML', ${r.reason}, 'rejected', ${seed.id}, ${r.reason}, 'other') on conflict (source_id, url) do nothing`);
163+ }
164+
165+ await db.execute(sql`update factory_seeds set status = 'discovered', source_id = ${sourceId}, candidates = ${scored.length}, shadow = shadow + ${shadowCreated}, discovered_at = now(), last_error = null, updated_at = now() where id = ${seed.id}`);
166+ await db.execute(sql`update sources set robots_checked_at = now(), updated_at = now() where id = ${sourceId}`);
167+ await bumpFactoryDaily({ seeds_processed: 1, requests: disc.requests, candidates: scored.length, shadow_created: shadowCreated, blocked: disc.blocked ? 1 : 0 });
168+ log.info({ seed: seed.id, source: sourceId, candidates: scored.length, shadow: shadowCreated, requests: disc.requests, ms: Date.now() - started, day }, "factory: organization discovered");
169+ return { seedId: seed.id, sourceId, status: "discovered", candidates: scored.length, shadow: shadowCreated, requests: disc.requests, durationMs: Date.now() - started, notes };
170+}
171+
172+function kindOfExisting(r: { url: string; type: string; connector: string; config: Record<string, unknown> }): CandidateKind {
173+ const k = (r.config as { kind?: string }).kind as CandidateKind | undefined;
174+ if (k) return k;
175+ if (r.connector === "statuspage" || r.connector === "statusjson") return "status";
176+ if (r.connector === "sitemap") return "sitemap";
177+ if (r.connector === "github") return "releases";
178+ if (r.connector === "edgar") return "filings";
179+ if (r.connector === "dns" || r.connector === "tls" || r.connector === "headers" || r.connector === "rdap") return "posture";
180+ if (r.connector === "openapi") return "api";
181+ if (r.connector === "rss") return /press/i.test(r.url) ? "press" : /blog/i.test(r.url) ? "blog" : /security|advisor/i.test(r.url) ? "security" : /changelog|release/i.test(r.url) ? "changelog" : "news";
182+ if (/pricing|plans/i.test(r.url)) return "pricing";
183+ if (/terms|privacy|legal/i.test(r.url)) return "legal";
184+ if (/career|jobs/i.test(r.url)) return "careers";
185+ if (/changelog|release-notes|whats-new/i.test(r.url)) return "changelog";
186+ if (/security|advisor/i.test(r.url)) return "security";
187+ return "other";
188+}
189+
190+async function createShadowSensor(sourceId: string, source: SourceRow, c: { url: string; type: string; connector: string; name: string; config: Record<string, unknown>; suggestedTier: Tier; kind: CandidateKind }, candId: string, seed: FactorySeed): Promise<string | null> {
191+ let id = `${sourceId}_${slugify(c.name)}`.slice(0, 80);
192+ const clash = (await db.execute<{ url: string }>(sql`select url from sensors where id = ${id}`)).rows[0];
193+ if (clash && clash.url.replace(/\/$/, "") !== c.url.replace(/\/$/, "")) id = `${id}-${newId("x").slice(-5)}`;
194+ const tier = c.suggestedTier;
195+ const priority = 3; // shadows never compete with production sensors
196+ const jitter = Math.floor(Math.random() * 30 * 60e3);
197+ try {
198+ await db
199+ .insert(sensors)
200+ .values({ id, sourceId, name: c.name, url: c.url, type: c.type, connector: c.connector, tier, importanceWeight: 1, config: { ...c.config, factory: true, shadow: true, seed: false, kind: c.kind, candidate: candId, seedId: seed.id, shadowSince: new Date().toISOString(), targetPriority: priorityFor(tier, source.categories) }, status: "SHADOW", priority, validatedAt: new Date(), nextCheckAt: new Date(Date.now() + jitter) })
201+ .onConflictDoNothing();
202+ } catch (e) {
203+ log.warn({ sensor: id, err: (e as Error).message }, "factory: shadow sensor insert failed");
204+ return null;
205+ }
206+ return id;
207+}
208+
209+export async function bumpFactoryDaily(delta: Partial<Record<"seeds_processed" | "requests" | "candidates" | "shadow_created" | "accepted" | "rejected" | "blocked", number>>): Promise<void> {
210+ const day = new Date().toISOString().slice(0, 10);
211+ const d = { seeds_processed: 0, requests: 0, candidates: 0, shadow_created: 0, accepted: 0, rejected: 0, blocked: 0, ...delta };
212+ await db.execute(sql`insert into factory_daily (day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked) values (${day}, ${d.seeds_processed}, ${d.requests}, ${d.candidates}, ${d.shadow_created}, ${d.accepted}, ${d.rejected}, ${d.blocked})
213+ on conflict (day) do update set seeds_processed = factory_daily.seeds_processed + ${d.seeds_processed}, requests = factory_daily.requests + ${d.requests}, candidates = factory_daily.candidates + ${d.candidates}, shadow_created = factory_daily.shadow_created + ${d.shadow_created}, accepted = factory_daily.accepted + ${d.accepted}, rejected = factory_daily.rejected + ${d.rejected}, blocked = factory_daily.blocked + ${d.blocked}`).catch(() => undefined);
214+}
215+
216+/** Claim up to `n` queued seeds (systemic organizations first) and process them with bounded concurrency. */
217+export async function runFactoryBatch(n = factoryConfig.concurrency, opts: { seedIds?: string[] } = {}): Promise<ProcessOutcome[]> {
218+ const claimed = await db.execute<Record<string, unknown>>(sql`
219+ update factory_seeds set status = 'discovering', attempts = attempts + 1, updated_at = now()
220+ where id in (
221+ select id from factory_seeds
222+ where ${opts.seedIds ? sql`id = any(${sql.raw("array[" + opts.seedIds.map((i) => "'" + i.replace(/'/g, "''") + "'").join(",") + "]::text[]")})` : sql`(status = 'queued' or (status = 'error' and attempts < 3 and updated_at < now() - interval '6 hours') or (status = 'discovering' and updated_at < now() - interval '2 hours'))`}
223+ order by importance desc, created_at asc limit ${n} for update skip locked)
224+ returning *`);
225+ const seeds = claimed.rows.map(normalizeSeed);
226+ const out: ProcessOutcome[] = [];
227+ let i = 0;
228+ await Promise.all(
229+ Array.from({ length: Math.min(factoryConfig.concurrency, seeds.length) }, async () => {
230+ while (i < seeds.length) {
231+ const seed = seeds[i++]!;
232+ try {
233+ out.push(await processSeed(seed));
234+ } catch (e) {
235+ log.error({ seed: seed.id, err: (e as Error).stack ?? (e as Error).message }, "factory: seed failed");
236+ await db.execute(sql`update factory_seeds set status = 'error', last_error = ${(e as Error).message.slice(0, 500)}, updated_at = now() where id = ${seed.id}`).catch(() => undefined);
237+ out.push({ seedId: seed.id, sourceId: null, status: "error", candidates: 0, shadow: 0, requests: 0, durationMs: 0, notes: [(e as Error).message] });
238+ }
239+ }
240+ }),
241+ );
242+ return out;
243+}
244+
245+function normalizeSeed(r: Record<string, unknown>): FactorySeed {
246+ const d = (v: unknown): Date | null => (v ? new Date(v as string) : null);
247+ return {
248+ id: r.id as string,
249+ name: r.name as string,
250+ domain: r.domain as string,
251+ homepage: (r.homepage as string | null) ?? null,
252+ categories: (r.categories as string[]) ?? [],
253+ country: (r.country as string | null) ?? null,
254+ language: (r.language as string | null) ?? null,
255+ tier: (r.tier as string) ?? "B",
256+ weight: Number(r.weight ?? 1),
257+ importance: Number(r.importance ?? 2),
258+ aliases: (r.aliases as string[]) ?? [],
259+ firstParty: Boolean(r.first_party ?? true),
260+ sector: (r.sector as string | null) ?? null,
261+ universe: (r.universe as string | null) ?? null,
262+ hints: (r.hints as Record<string, unknown>) ?? {},
263+ status: r.status as string,
264+ attempts: Number(r.attempts ?? 0),
265+ sourceId: (r.source_id as string | null) ?? null,
266+ candidates: Number(r.candidates ?? 0),
267+ shadow: Number(r.shadow ?? 0),
268+ accepted: Number(r.accepted ?? 0),
269+ rejected: Number(r.rejected ?? 0),
270+ lastError: (r.last_error as string | null) ?? null,
271+ discoveredAt: d(r.discovered_at),
272+ createdAt: d(r.created_at) ?? new Date(),
273+ updatedAt: d(r.updated_at) ?? new Date(),
274+ };
275+}
276+
277+export async function factoryStats(): Promise<Record<string, unknown>> {
278+ const [seeds, cands, shadows, daily, sectors] = await Promise.all([
279+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from factory_seeds group by status`).then((r) => r.rows),
280+ db.execute<Record<string, unknown>>(sql`select status, count(*)::int as n from discovery_candidates group by status`).then((r) => r.rows),
281+ db.execute<Record<string, unknown>>(sql`select count(*)::int as shadow, count(*) filter (where total_runs >= ${factoryConfig.shadowMinChecks})::int as ready, count(*) filter (where health <> 'UP')::int as unhealthy from sensors where status = 'SHADOW'`).then((r) => r.rows[0]),
282+ db.execute<Record<string, unknown>>(sql`select day::text as day, seeds_processed, requests, candidates, shadow_created, accepted, rejected, blocked from factory_daily order by day desc limit 30`).then((r) => r.rows),
283+ db.execute<Record<string, unknown>>(sql`select coalesce(sector, '-') as sector, count(*)::int as seeds, count(*) filter (where status = 'queued')::int as queued, count(*) filter (where status in ('discovered','done'))::int as discovered, count(*) filter (where status = 'blocked')::int as blocked, sum(shadow)::int as shadow, sum(accepted)::int as accepted, sum(rejected)::int as rejected from factory_seeds group by 1 order by seeds desc`).then((r) => r.rows),
284+ ]);
285+ return { seeds: Object.fromEntries(seeds.map((r) => [r.status, r.n])), candidates: Object.fromEntries(cands.map((r) => [r.status, r.n])), shadow: shadows, daily, sectors, config: { concurrency: factoryConfig.concurrency, budget: factoryConfig.budget, minScore: factoryConfig.minScore, shadowMinChecks: factoryConfig.shadowMinChecks, shadowMinHours: factoryConfig.shadowMinHours, shadowMaxHours: factoryConfig.shadowMaxHours, caps: factoryConfig.caps } };
286+}
287+
288+export { factorySeeds };
added apps/engine/src/factory/score.ts +114 −0
@@ -0,0 +1,114 @@
1+import type { CandidateKind, DeepCandidate } from "@websensor/connectors";
2+import { factoryConfig } from "../config";
3+
4+/**
5+ * Candidate scoring (Source Factory, stage "signal/noise evaluation before shadow"). Combines the discovery
6+ * value (evidence quality) with the organization's importance, the class of page, first-party confidence,
7+ * expected change frequency and fetch cost. Explainable: every score carries its reasons.
8+ */
9+export interface ScoredCandidate {
10+ cand: DeepCandidate;
11+ score: number;
12+ reasons: string[];
13+}
14+
15+/** How much a page class is worth for change intelligence (1 = neutral). */
16+export const KIND_WEIGHT: Record<CandidateKind, number> = {
17+ status: 1.15,
18+ security: 1.15,
19+ press: 1.1,
20+ news: 1.05,
21+ filings: 1.1,
22+ changelog: 1.05,
23+ releases: 1.0,
24+ pricing: 1.05,
25+ models: 1.05,
26+ ir: 0.95,
27+ legal: 0.9,
28+ blog: 0.9,
29+ api: 0.9,
30+ leadership: 0.85,
31+ data: 0.85,
32+ sitemap: 0.8,
33+ docs: 0.7,
34+ careers: 0.6,
35+ posture: 0.6,
36+ other: 0.7,
37+};
38+
39+export function scoreCandidate(c: DeepCandidate, seed: { importance: number; weight?: number }): ScoredCandidate {
40+ const reasons: string[] = [];
41+ let score = c.value;
42+ reasons.push(`evidence ${c.value.toFixed(2)} (${c.evidence})`);
43+ const kw = KIND_WEIGHT[c.kind] ?? 0.7;
44+ score *= kw;
45+ if (kw !== 1) reasons.push(`${c.kind} ×${kw.toFixed(2)}`);
46+ const imp = seed.importance === 3 ? 1.1 : seed.importance === 1 ? 0.85 : 1;
47+ score *= imp * (seed.weight ?? 1);
48+ if (imp !== 1) reasons.push(`organization importance ${seed.importance} ×${imp.toFixed(2)}`);
49+ score *= c.firstPartyConfidence;
50+ if (c.firstPartyConfidence < 1) reasons.push(`first-party confidence ${c.firstPartyConfidence}`);
51+ if (c.itemsPerDay !== null && c.itemsPerDay !== undefined) {
52+ if (c.itemsPerDay > 60) {
53+ score *= 0.85;
54+ reasons.push(`firehose ${c.itemsPerDay}/day ×0.85`);
55+ } else if (c.itemsPerDay < 0.02) {
56+ score *= 0.8;
57+ reasons.push(`nearly dormant ${c.itemsPerDay}/day ×0.80`);
58+ } else if (c.itemsPerDay >= 0.2) {
59+ score *= 1.05;
60+ reasons.push(`active ${c.itemsPerDay}/day ×1.05`);
61+ }
62+ }
63+ if (c.fetchCost.ms > 8000) {
64+ score -= 0.08;
65+ reasons.push(`slow (${Math.round(c.fetchCost.ms)} ms) −0.08`);
66+ }
67+ if (c.fetchCost.bytes > 5 * 1024 * 1024) {
68+ score -= 0.08;
69+ reasons.push(`heavy (${Math.round(c.fetchCost.bytes / 1048576)} MB) −0.08`);
70+ }
71+ score = Math.round(Math.max(0, Math.min(1.2, score)) * 1000) / 1000;
72+ return { cand: c, score, reasons };
73+}
74+
75+/** Promotion policy: caps per organization and per class, best score first. Returns the selected candidates and the reason for the others. */
76+export function selectForShadow(scored: ScoredCandidate[], existing: { kinds: Map<CandidateKind, number>; total: number }): { selected: ScoredCandidate[]; skipped: { s: ScoredCandidate; reason: string }[] } {
77+ const caps = factoryConfig.caps;
78+ const groupOf = (k: CandidateKind): keyof typeof caps => (k === "news" || k === "press" || k === "blog" || k === "ir" || k === "changelog" || k === "security" ? "feeds" : k === "pricing" || k === "legal" || k === "careers" || k === "leadership" || k === "docs" || k === "data" || k === "other" ? "pages" : (k as keyof typeof caps));
79+ const used = new Map<string, number>();
80+ for (const [k, n] of existing.kinds) used.set(groupOf(k), (used.get(groupOf(k)) ?? 0) + n);
81+ let total = existing.total;
82+ const selected: ScoredCandidate[] = [];
83+ const skipped: { s: ScoredCandidate; reason: string }[] = [];
84+ const legal = (existing.kinds.get("legal") ?? 0);
85+ let legalUsed = legal;
86+ for (const s of [...scored].sort((a, b) => b.score - a.score)) {
87+ if (s.score < factoryConfig.minScore) {
88+ skipped.push({ s, reason: `score ${s.score} < ${factoryConfig.minScore}` });
89+ continue;
90+ }
91+ if (total >= caps.total) {
92+ skipped.push({ s, reason: `organization cap ${caps.total} reached` });
93+ continue;
94+ }
95+ // Feed-type candidates (rss connector) and HTML pages are capped by group; connectors with their own cap by kind.
96+ const g = s.cand.connector === "rss" ? "feeds" : s.cand.connector === "http" && s.cand.kind !== "posture" ? "pages" : groupOf(s.cand.kind);
97+ const cap = caps[g as keyof typeof caps] ?? 1;
98+ if ((used.get(g) ?? 0) >= cap) {
99+ skipped.push({ s, reason: `${g} cap ${cap} reached` });
100+ continue;
101+ }
102+ if (s.cand.kind === "legal") {
103+ if (legalUsed >= caps.legal) {
104+ skipped.push({ s, reason: `legal cap ${caps.legal} reached` });
105+ continue;
106+ }
107+ legalUsed++;
108+ }
109+ used.set(g, (used.get(g) ?? 0) + 1);
110+ total++;
111+ selected.push(s);
112+ }
113+ return { selected, skipped };
114+}
added apps/engine/src/factory/seeds.ts +168 −0
@@ -0,0 +1,168 @@
1+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
2+import { join } from "node:path";
3+import YAML from "yaml";
4+import { coverageKey, coverageSectorSchema, slugify, type CoverageMember, type CoverageSector } from "@websensor/core";
5+import { db, factorySeeds, sql } from "@websensor/db";
6+import { factoryConfig, log } from "../config";
7+
8+/**
9+ * Factory seeds come from two places: the coverage universes (`config/coverage/*.yaml`, every member that is not
10+ * monitored yet — or that carries hints worth expanding) and ad-hoc seed files (`config/factory/seeds/*.yaml`,
11+ * `{ seeds: [{ name, domain, categories, country, … }] }`) or the admin API.
12+ */
13+export interface SeedInput {
14+ id?: string;
15+ name: string;
16+ domain: string;
17+ homepage?: string;
18+ categories?: string[];
19+ country?: string;
20+ language?: string;
21+ tier?: string;
22+ weight?: number;
23+ importance?: number;
24+ aliases?: string[];
25+ first_party?: boolean;
26+ sector?: string;
27+ universe?: string;
28+ hints?: Record<string, unknown>;
29+}
30+
31+export function loadCoverageSectors(dir = factoryConfig.coverageDir): { sectors: CoverageSector[]; issues: string[] } {
32+ const sectors: CoverageSector[] = [];
33+ const issues: string[] = [];
34+ if (!existsSync(dir) || !statSync(dir).isDirectory()) return { sectors, issues: [`coverage dir ${dir} not found`] };
35+ for (const f of readdirSync(dir).filter((x) => /\.ya?ml$/.test(x) && !x.startsWith("_")).sort()) {
36+ try {
37+ const parsed = coverageSectorSchema.safeParse(YAML.parse(readFileSync(join(dir, f), "utf8")));
38+ if (!parsed.success) {
39+ issues.push(`${f}: ${parsed.error.issues.slice(0, 3).map((i) => `${i.path.join(".")}: ${i.message}`).join("; ")}`);
40+ continue;
41+ }
42+ sectors.push(parsed.data);
43+ } catch (e) {
44+ issues.push(`${f}: ${(e as Error).message}`);
45+ }
46+ }
47+ return { sectors, issues };
48+}
49+
50+/** Merge files that share a sector key. */
51+export function mergeSectors(sectors: CoverageSector[]): CoverageSector[] {
52+ const byKey = new Map<string, CoverageSector>();
53+ for (const s of sectors) {
54+ const prev = byKey.get(s.sector);
55+ if (!prev) byKey.set(s.sector, { ...s, universes: [...s.universes] });
56+ else prev.universes.push(...s.universes);
57+ }
58+ return [...byKey.values()];
59+}
60+
61+export function memberToSeed(sector: CoverageSector, universe: string, m: CoverageMember): SeedInput {
62+ return {
63+ name: m.name,
64+ domain: m.domain,
65+ categories: m.categories ?? sector.categories,
66+ country: m.country,
67+ language: m.language,
68+ tier: m.tier ?? sector.tier,
69+ importance: m.importance,
70+ aliases: m.aliases,
71+ first_party: m.first_party,
72+ sector: sector.sector,
73+ universe,
74+ hints: m.hints as Record<string, unknown>,
75+ };
76+}
77+
78+function hasHints(h: Record<string, unknown> | undefined): boolean {
79+ if (!h) return false;
80+ return Boolean(h.cik || h.github_org || (Array.isArray(h.github_repos) && h.github_repos.length) || h.hf_author || h.status_url || (Array.isArray(h.urls) && h.urls.length) || (Array.isArray(h.hosts) && h.hosts.length));
81+}
82+
83+/** Registrable domains currently monitored (source domains + hosts of enabled non-shadow sensors). */
84+export async function monitoredDomainKeys(): Promise<Set<string>> {
85+ const rows = await db.execute<{ d: string }>(sql`
86+ select domain as d from sources where enabled and kind = 'registry'
87+ union
88+ select lower(split_part(split_part(url, '/', 3), ':', 1)) as d from sensors where enabled and status <> 'SHADOW' and url like 'http%'`);
89+ const out = new Set<string>();
90+ for (const r of rows.rows) if (r.d) out.add(coverageKey(r.d));
91+ return out;
92+}
93+
94+export async function upsertSeeds(inputs: SeedInput[], opts: { requeue?: boolean } = {}): Promise<{ inserted: number; updated: number; skipped: number }> {
95+ let inserted = 0;
96+ let updated = 0;
97+ let skipped = 0;
98+ const existingIds = new Set((await db.execute<{ id: string }>(sql`select id from sources`)).rows.map((r) => r.id));
99+ const existingSeedDomains = new Map((await db.execute<{ id: string; domain: string }>(sql`select id, domain from factory_seeds`)).rows.map((r) => [coverageKey(r.domain), r.id]));
100+ for (const s of inputs) {
101+ const domain = s.domain.toLowerCase().replace(/^https?:\/\//, "").replace(/^www\./, "").replace(/\/.*$/, "");
102+ const key = coverageKey(domain);
103+ let id = s.id ?? existingSeedDomains.get(key) ?? slugify(s.name).slice(0, 60);
104+ if (!id) id = slugify(domain);
105+ // A registry source with the same id but another organization → disambiguate by domain.
106+ if (!s.id && !existingSeedDomains.has(key) && existingIds.has(id)) {
107+ const src = (await db.execute<{ domain: string }>(sql`select domain from sources where id = ${id}`)).rows[0];
108+ if (src && coverageKey(src.domain) !== key) id = `${id}-${slugify(domain.split(".")[0] ?? domain)}`.slice(0, 70);
109+ }
110+ const r = await db.execute<{ inserted: boolean }>(sql`
111+ insert into factory_seeds (id, name, domain, homepage, categories, country, language, tier, weight, importance, aliases, first_party, sector, universe, hints, status)
112+ values (${id}, ${s.name}, ${domain}, ${s.homepage ?? null}, ${sql.raw("'{" + (s.categories ?? []).map((c) => '"' + c.replace(/"/g, "") + '"').join(",") + "}'::text[]")}, ${s.country ?? null}, ${s.language ?? null}, ${s.tier ?? "B"}, ${s.weight ?? 1}, ${s.importance ?? 2}, ${sql.raw("'{" + (s.aliases ?? []).map((c) => '"' + c.replace(/["\\]/g, "") + '"').join(",") + "}'::text[]")}, ${s.first_party ?? true}, ${s.sector ?? null}, ${s.universe ?? null}, ${JSON.stringify(s.hints ?? {})}::jsonb, 'queued')
113+ on conflict (id) do update set name = excluded.name, domain = excluded.domain, categories = excluded.categories, country = coalesce(excluded.country, factory_seeds.country), language = coalesce(excluded.language, factory_seeds.language), tier = excluded.tier, importance = excluded.importance, aliases = excluded.aliases, sector = coalesce(excluded.sector, factory_seeds.sector), universe = coalesce(excluded.universe, factory_seeds.universe), hints = factory_seeds.hints || excluded.hints, updated_at = now()${opts.requeue ? sql`, status = 'queued'` : sql``}
114+ returning (xmax = 0) as inserted`);
115+ if (r.rows[0]?.inserted) inserted++;
116+ else if (r.rowCount) updated++;
117+ else skipped++;
118+ existingSeedDomains.set(key, id);
119+ }
120+ log.info({ inserted, updated, skipped }, "factory seeds upserted");
121+ return { inserted, updated, skipped };
122+}
123+
124+/**
125+ * Turn coverage universes into seeds. `mode`: `uncovered` = members without any monitored domain; `hinted` = also
126+ * covered members that carry expansion hints; `all` = every member (re-discover everything).
127+ */
128+export async function seedFromCoverage(opts: { mode?: "uncovered" | "hinted" | "all"; sectors?: string[]; requeue?: boolean } = {}): Promise<{ members: number; seeds: number; inserted: number; updated: number; issues: string[] }> {
129+ const mode = opts.mode ?? "hinted";
130+ const { sectors, issues } = loadCoverageSectors();
131+ const monitored = await monitoredDomainKeys();
132+ const inputs: SeedInput[] = [];
133+ const seen = new Set<string>();
134+ let members = 0;
135+ for (const sector of mergeSectors(sectors)) {
136+ if (opts.sectors && !opts.sectors.includes(sector.sector)) continue;
137+ for (const u of sector.universes) {
138+ for (const m of u.members) {
139+ members++;
140+ const key = coverageKey(m.domain);
141+ if (seen.has(key)) continue;
142+ const covered = monitored.has(key);
143+ if (mode === "uncovered" && covered) continue;
144+ if (mode === "hinted" && covered && !hasHints(m.hints as Record<string, unknown>)) continue;
145+ seen.add(key);
146+ inputs.push(memberToSeed(sector, u.key, m));
147+ }
148+ }
149+ }
150+ const r = await upsertSeeds(inputs, { requeue: opts.requeue });
151+ return { members, seeds: inputs.length, inserted: r.inserted, updated: r.updated, issues };
152+}
153+
154+/** `config/factory/seeds/*.yaml` → `{ seeds: [...] }` */
155+export async function seedFromFiles(dir = factoryConfig.seedsDir): Promise<{ files: number; seeds: number }> {
156+ if (!existsSync(dir)) return { files: 0, seeds: 0 };
157+ const inputs: SeedInput[] = [];
158+ let files = 0;
159+ for (const f of readdirSync(dir).filter((x) => /\.ya?ml$/.test(x)).sort()) {
160+ files++;
161+ const raw = YAML.parse(readFileSync(join(dir, f), "utf8")) as { seeds?: SeedInput[] };
162+ for (const s of raw.seeds ?? []) if (s?.name && s?.domain) inputs.push(s);
163+ }
164+ await upsertSeeds(inputs);
165+ return { files, seeds: inputs.length };
166+}
167+
168+export { factorySeeds };
added apps/engine/src/factory/shadow.ts +109 −0
Large diff (109 lines) is not loaded by default to keep the page fast.Load diff
modified apps/engine/src/pipeline.ts +9 −2
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
modified apps/engine/src/registry.ts +39 −29
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/app/ops/factory-panel.tsx +205 −0
Large diff (205 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/ops/ops.tsx +6 −1
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/app/page.tsx +2 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/command-palette.tsx +1 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added apps/web/src/components/coverage-rail.tsx +27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/nav.tsx +2 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/components/ui.tsx +2 −2
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
modified apps/web/src/lib/api.ts +63 −0
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
modified deploy/README.md +17 −1
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
modified deploy/websensor.mld.json.example +194 −123
Large diff (323 lines) is not loaded by default to keep the page fast.Load diff
modified docs/ARCHITECTURE.md +10 −1
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
modified docs/registry/AUTHORING.md +5 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added docs/registry/COVERAGE.md +67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
added packages/connectors/src/discovery-deep.ts +672 −0
Large diff (672 lines) is not loaded by default to keep the page fast.Load diff
modified packages/connectors/src/index.ts +1 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added packages/core/src/coverage-schema.ts +150 −0
Large diff (150 lines) is not loaded by default to keep the page fast.Load diff
modified packages/core/src/index.ts +1 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added packages/db/migrations/0007_factory.sql +73 −0
Large diff (73 lines) is not loaded by default to keep the page fast.Load diff
modified packages/db/src/schema.ts +46 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere