SPB Git forge
spboucher.ai ↗

spb/websensor

Public
33commits 1branches 0releases
3.4 MBsize
maindefault branch
10 days agolast push
TypeScript 55.4% Python 43.2% SQL 1.2%
4.6 KB
View source Raw Blame History

# Coverage universes — authoring guide

config/coverage/*.yaml describes what WebSensor should observe, sector by sector: the denominator of the Global Observation Coverage Score (/coverage, GET /api/v1/coverage). The same files feed the Source Factory: every member whose registrable domain is not yet monitored becomes an expansion seed, is discovered automatically (robots, sitemaps, feeds, status pages, GitHub, IR/newsroom/pricing/legal pages, EDGAR, Hugging Face…), shadow-monitored, then accepted or rejected on evidence.

A universe is a list of organizations with a correct official domain. It is not a registry fragment: no sensors, no URLs are required (they are welcome as hints). The registry stays the source of truth for what is actually polled.

# File format — one sector per file (config/coverage/NN-<sector>.yaml; several files may share a sector key, their universes are merged — universe keys must then be unique across those files)

yaml
sector: energy                     # kebab-case key, unique across files
label: Energy & commodities
description: Agencies, grid operators, producers and exchanges whose changes move energy markets.
weight: 1.0                        # weight of the sector in the global score (1.0 default; 1.2–1.5 systemic)
categories: [energy]               # default categories for factory-created sources (registry vocabulary)
tier: B                            # default tier for factory-created sources (S A B C D)
universes:
  - key: agencies
    label: Energy agencies & regulators
    provenance: "https://en.wikipedia.org/wiki/…  (retrieved 2026-09-13)"
    members:
      - { name: International Energy Agency, domain: iea.org, country: INT, importance: 3, aliases: [iea] }
      - { name: ERCOT, domain: ercot.com, country: US, importance: 3, hints: { urls: [{ url: "https://www.ercot.com/news/rss", kind: news }] } }
  - key: oil-gas-majors
    label: Oil & gas majors
    members:
      - { name: Shell, domain: shell.com, country: GB, importance: 3, hints: { cik: "1306965", ticker: SHEL } }

# Member fields

field required meaning
name yes official English name (ASCII preferred; native name in aliases)
domain yes bare registrable domain, lower-case, no scheme, no www., no path (gov.uk sites: use the registrable domain, e.g. bankofengland.co.uk, ons.gov.uk)
country recommended ISO 3166-1 alpha-2 upper-case; EU for EU bodies; INT for global bodies. Quote "NO" (Norway) — YAML reads it as false
language optional ISO 639-1 of the site's main language when not English ("no" must be quoted)
importance optional 1 long tail · 2 notable (default) · 3 systemically important
aliases optional lower-case alternative names, tickers, acronyms (entity linking)
categories optional override of the sector default
tier optional override of the sector default
first_party optional false only for media/aggregators
hints optional cik, ticker, exchange, github_org, github_repos: [owner/name], hf_author, status_url, hosts: [ir.example.com], urls: [{url, kind, connector?, type?, tier?, config?, name?}], posture: true, wikidata: Q…
note optional anything a reviewer should know (blocked by Cloudflare, JS shell…)

hints.urls[].kind vocabulary: news press blog ir filings changelog releases security pricing legal careers docs api status models data other.

# Rules

  1. Real organizations, correct domains. Every domain must resolve (dig +short <domain> or <www.domain>) and belong to the organization. Wikidata P856 / the index constituents page / EDGAR website are good sources; never guess a domain from a name.
  2. Provenance on every universe (provenance: URL or query + retrieval date). Index constituents change: record the date.
  3. No duplicates inside a file (same domain twice). The same organization may appear in several sectors (a bank is both corporate and markets).
  4. Countries everywhere for governments and companies; INT for international bodies.
  5. Validate: node node_modules/tsx/dist/cli.mjs apps/engine/src/coverage-validate.ts config/coverage/<file>.yaml [--dns] (schema, duplicates, domain syntax; --dns resolves every domain). Only files that pass are committed.
  6. Keep entries compact (flow style { … }, one member per line).

# Sector keys in use (2026-09-13)

ai · cloud · cyber · markets · corporate · government · science · healthcare · energy · transport · telecom · logistics · open-source · commerce