spb/websensor

1# Coverage universes — authoring guide23`config/coverage/*.yaml` describes **what WebSensor should observe**, sector by sector: the denominator of the4Global Observation Coverage Score (`/coverage`, `GET /api/v1/coverage`). The same files feed the **Source5Factory**: every member whose registrable domain is not yet monitored becomes an expansion seed, is6discovered automatically (robots, sitemaps, feeds, status pages, GitHub, IR/newsroom/pricing/legal pages,7EDGAR, Hugging Face…), shadow-monitored, then accepted or rejected on evidence.89A universe is a list of **organizations with a correct official domain**. It is *not* a registry fragment:10no sensors, no URLs are required (they are welcome as `hints`). The registry stays the source of truth for11what is actually polled.1213## File format — one sector per file (`config/coverage/NN-<sector>.yaml`; several files may share a sector key, their universes are merged — universe keys must then be unique across those files)1415```yaml16sector: energy                     # kebab-case key, unique across files17label: Energy & commodities18description: Agencies, grid operators, producers and exchanges whose changes move energy markets.19weight: 1.0                        # weight of the sector in the global score (1.0 default; 1.2–1.5 systemic)20categories: [energy]               # default categories for factory-created sources (registry vocabulary)21tier: B                            # default tier for factory-created sources (S A B C D)22universes:23  - key: agencies24    label: Energy agencies & regulators25    provenance: "https://en.wikipedia.org/wiki/…  (retrieved 2026-09-13)"26    members:27      - { name: International Energy Agency, domain: iea.org, country: INT, importance: 3, aliases: [iea] }28      - { name: ERCOT, domain: ercot.com, country: US, importance: 3, hints: { urls: [{ url: "https://www.ercot.com/news/rss", kind: news }] } }29  - key: oil-gas-majors30    label: Oil & gas majors31    members:32      - { name: Shell, domain: shell.com, country: GB, importance: 3, hints: { cik: "1306965", ticker: SHEL } }33```3435### Member fields36| field | required | meaning |37|---|---|---|38| `name` | yes | official English name (ASCII preferred; native name in `aliases`) |39| `domain` | yes | bare registrable domain, lower-case, no scheme, no `www.`, no path (`gov.uk` sites: use the registrable domain, e.g. `bankofengland.co.uk`, `ons.gov.uk`) |40| `country` | recommended | ISO 3166-1 alpha-2 upper-case; `EU` for EU bodies; `INT` for global bodies. **Quote `"NO"`** (Norway) — YAML reads it as false |41| `language` | optional | ISO 639-1 of the site's main language when not English (`"no"` must be quoted) |42| `importance` | optional | 1 long tail · 2 notable (default) · 3 systemically important |43| `aliases` | optional | lower-case alternative names, tickers, acronyms (entity linking) |44| `categories` | optional | override of the sector default |45| `tier` | optional | override of the sector default |46| `first_party` | optional | `false` only for media/aggregators |47| `hints` | optional | `cik`, `ticker`, `exchange`, `github_org`, `github_repos: [owner/name]`, `hf_author`, `status_url`, `hosts: [ir.example.com]`, `urls: [{url, kind, connector?, type?, tier?, config?, name?}]`, `posture: true`, `wikidata: Q…` |48| `note` | optional | anything a reviewer should know (blocked by Cloudflare, JS shell…) |4950`hints.urls[].kind` vocabulary: `news press blog ir filings changelog releases security pricing legal careers docs api status models data other`.5152## Rules531. **Real organizations, correct domains.** Every domain must resolve (`dig +short <domain>` or `<www.domain>`)54   and belong to the organization. Wikidata P856 / the index constituents page / EDGAR `website` are good55   sources; never guess a domain from a name.562. **Provenance on every universe** (`provenance:` URL or query + retrieval date). Index constituents change:57   record the date.583. **No duplicates inside a file** (same domain twice). The same organization may appear in several sectors59   (a bank is both `corporate` and `markets`).604. **Countries everywhere** for governments and companies; `INT` for international bodies.615. **Validate**: `node node_modules/tsx/dist/cli.mjs apps/engine/src/coverage-validate.ts config/coverage/<file>.yaml [--dns]`62   (schema, duplicates, domain syntax; `--dns` resolves every domain). Only files that pass are committed.636. Keep entries compact (flow style `{ … }`, one member per line).6465## Sector keys in use (2026-09-13)66`ai` · `cloud` · `cyber` · `markets` · `corporate` · `government` · `science` · `healthcare` · `energy` ·67`transport` · `telecom` · `logistics` · `open-source` · `commerce`68