# Source registry — authoring guide

The registry is `config/sources.yaml` (founding file, 271 organizations) plus fragments in
`config/sources.d/*.yaml`, merged in file-name order by `apps/engine/src/seeds.ts`. Every organization becomes an
entity; every `sensors:` entry becomes a real, polled endpoint. **Nothing goes in the registry that has not been
fetched and parsed successfully** — a sensor that returns 403/404, HTML instead of a feed, or an empty list is a
liability (noise, wasted budget, false "page removed" events), not coverage.

> **Coverage universes & the Source Factory (2026-09-13).** Organizations that *should* be observed are listed per sector
> in `config/coverage/*.yaml` (guide: `COVERAGE.md`). The Factory discovers, validates, shadow-monitors and accepts their
> sensors automatically (`sources.origin = 'factory'`, `sensors.config.factory = true`). `cli.ts factory export` turns
> accepted sensors into a fragment like the ones below for review and graduation into `config/sources.d/`.

## Fragment format

```yaml
# config/sources.d/12-energy-climate.yaml — one site class per file, header comment says what and when.
sources:
  - id: iea                      # kebab-case, unique across ALL files (see the id list below)
    name: International Energy Agency
    domain: iea.org              # bare domain, no scheme
    homepage: https://www.iea.org   # optional, defaults to https://<domain>
    categories: [energy, government]
    tier: B                      # S 15–60 s · A 1–5 min · B 5–30 min · C 30 min–6 h · D 6–24 h
    weight: 1.1                  # importance multiplier (default 1; 1.2–1.5 for globally important orgs)
    aliases: [iea]               # lowercase alternative names for entity linking
    products:                    # optional child entities (product, AI_model, API, software, service, index…)
      - { name: World Energy Outlook, type: product, aliases: [weo] }
    discover: { rss: true, sitemap: false, status: false, pages: false }   # only ALLOWS discovery to probe
    llm: true                    # false for high-volume feeds (news wires, recalls firehoses, package streams)
    country: FR                  # optional ISO 3166-1 alpha-2 (upper-case); EU for EU bodies, INT for global bodies
    language: fr                 # optional ISO 639-1 when the content is not English
    first_party: false           # ONLY for media / aggregators reporting about others; omit for an organization's own channels
    notes: "why this source matters / what was blocked (optional)"
    sensors:
      - { name: news feed, url: "https://www.iea.org/news.rss", type: RSS, connector: rss, tier: B }
      - { name: status, url: "https://status.example.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S }
```

To add sensors to an organization that already exists in an earlier file, use `extend: true` (only `id` is
required; `sensors`, `products`, `aliases`, `categories`, `fallback`, `notes`, `country`, `language`, `first_party`
are merged — `llm` cannot be changed by an extend, edit the declaring file):

```yaml
  - id: apple
    extend: true
    sensors:
      - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000320193.json", type: REST_API, connector: edgar, tier: B }
```

## Connector cheat-sheet (type → connector)

| connector | type | what it is for | config |
|---|---|---|---|
| `rss` | RSS / ATOM | RSS 2.0, Atom, RDF, JSON Feed. GitHub `releases.atom`, GitLab `…/-/tags?format=atom`, YouTube `feeds/videos.xml?channel_id=`, Blogger `feeds/posts/summary?max-results=25`, arXiv API Atom | `maxItems` |
| `sitemap` | SITEMAP | sitemap index / urlset / news sitemap (one per source, tier A/B) | `maxUrls`, `maxChildren` |
| `statuspage` | STATUSPAGE | Atlassian Statuspage **`/api/v2/summary.json`** only (tier S) | — |
| `statusjson` | STATUSPAGE | Instatus `/summary.json`, Status.io `/1.0/status/<id>`, incident.io `/api/v1/summary` (tier S) | `flavor: instatus\|statusio\|incidentio` (auto-detected) |
| `github` | GITHUB_RELEASE | `https://github.com/<owner>/<repo>/releases.atom` (or `/tags.atom`, `/commits/<branch>.atom`) | `{ repo: owner/name, kind: releases\|tags\|commits\|advisories }` |
| `jsonlist` | REST_API / JSON | any official JSON API returning an array of records (recalls, alerts, incidents, datasets) | `itemsPath`, `keyField`, `titleField`/`titleTemplate`, `urlField`/`urlTemplate`, `summaryField`, `dateField`, `compareFields`, `maxItems`, `headers`, `noConditional`; URL placeholders `{now}`, `{now-2h}`, `{now-30d}` |
| `http` | HTML / JSON / XML / FILE | a server-rendered page whose text changes matter (pricing, changelog, advisories index, release notes, robots.txt, security.txt, llms.txt) | `selector`, `keepChrome`, `jsonPath`, `ignoreKeys`, `headers`, `method: HEAD` |
| `package` | REST_API | package registries: npm, PyPI, crates.io, RubyGems, NuGet, Packagist, Hex, Go proxy, Homebrew, Docker Hub tags | `{ registry: npm\|pypi\|crates\|rubygems\|nuget\|packagist\|hex\|goproxy\|homebrew\|dockerhub, name }` |
| `edgar` | REST_API | SEC EDGAR company submissions (`https://data.sec.gov/submissions/CIK##########.json`) → filings list | `forms: [8-K, 10-K, 10-Q, S-1, 6-K, 4]` (default: all but 4/144) |
| `openapi` | JSON | an OpenAPI/Swagger document (JSON or YAML) → operations list, API-contract changes | — |
| `csv` | FILE | CSV/TSV open-data file → keyed rows | `keyColumn`, `compareColumns`, `delimiter`, `maxRows` |
| `pdf` | FILE | a PDF whose text matters (central-bank statement, official notice) | `maxPages` |
| `dns` | DNS | `dns://example.com` → A/AAAA/NS/MX/TXT/CAA records | `records` |
| `tls` | TLS | `tls://example.com` → certificate issuer, SANs, validity | — |
| `headers` | HTTP_HEADERS | `https://…` → security/infra response headers (HSTS, CSP, server, CDN) | — |
| `rdap` | JSON | `https://rdap.org/domain/example.com` (or a registry RDAP URL) → registrar, status, nameservers, expiry | — |

Prefer, in this order: official API → feed → status page → server-rendered HTML page. Never add a sensor whose
only content is client-rendered (the validator reports it as `thin`).

## Rules
1. **Validate before you write.** From the repo root:
   `node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts config/sources.d/<file>.yaml`
   prints OK / WARN / FAIL per sensor. Only **OK** sensors stay. `WARN empty list` on an RSS feed = drop it.
   `--probe <domain>` runs the discovery engine (robots sitemaps, `<link rel=alternate>`, well-known feed paths,
   linked status pages) without a database — the fastest way to find a real feed.
2. **403 / 429 / Akamai / Cloudflare challenge = not a sensor** (unless `fallback: { scrapfly: true }` is set on the
   source *and* the tier is C or D — reserved for a handful of high-value pages). Put blocked sites in `notes:`
   rather than pretending.
3. **One source = one organization** (the entity people search for), 1–4 sensors. A source without any validated
   sensor may still be listed with `discover: { rss: true, sitemap: true }` if the organization matters, but
   prefer to find a real endpoint.
4. **Tiers**: S only for status pages and security advisories of infrastructure people depend on; A for the news
   feed of a globally important organization; B default; C for pricing/legal/policy pages; D for slow reference
   pages (robots.txt, TLS, DNS, RDAP).
5. **Volume**: feeds publishing > 50 items/day (wire services, recall firehoses, package version streams) get
   `llm: false` on the source (heuristics only) and a `maxItems` cap.
6. **Ids** are kebab-case and unique across all files. Do not redefine an existing id; use `extend: true`.
7. Keep entries compact (flow style `{ … }` for sensors and products, one per line) so files stay reviewable.
8. Categories are free strings but prefer the existing vocabulary: ai, cloud, developer, cyber, consumer-tech,
   semiconductors, finance, government, statistics, health, pharma, science, space, automotive, commerce, payments,
   crypto, enterprise, internet, standards, technology, news, media, politics — plus the class vocabulary added on
   2026-09-08: energy, climate, weather, telecom, retail, travel, gaming, entertainment, education, research,
   ngo, international, sports, transport, logistics, aviation, food, agriculture, consumer-safety, real-estate,
   labour, open-data, legal, elections, open-source, packages, filings, web-policy, infrastructure.

## Existing source ids (founding file — do not redefine, `extend: true` to add sensors)
openai anthropic deepmind google-ai google-developers-ai microsoft-ai azure-ai meta-ai mistral huggingface cohere
xai stability-ai replicate together-ai fireworks-ai groq cerebras perplexity nvidia aws azure google-cloud
cloudflare cloudflare-radar digitalocean akamai fastly vercel netlify heroku fly-io render railway oracle-cloud
ibm-cloud ovhcloud hetzner vultr linode github gitlab docker kubernetes nodejs python rust go php ruby postgresql
mysql redis mongodb elastic hashicorp grafana sentry supabase firebase cisa nvd nist cert-cc mitre
microsoft-security apple-security google-security-blog project-zero cisco-security palo-alto-networks fortinet
crowdstrike mandiant rapid7 tenable sophos trend-micro malwarebytes hibp apple samsung google microsoft sony lg
dell hp lenovo asus acer intel amd qualcomm arm tsmc micron western-digital seagate raspberry-pi sec
federal-reserve us-treasury bank-of-canada ecb bank-of-england bank-of-japan finra cftc nasdaq nyse tmx
sedar-plus fdic occ bis imf world-bank oecd fred canada statcan quebec federal-register white-house congress bls
census bea eurostat fda health-canada ema clinicaltrials pubmed nih cdc who mayo-clinic cleveland-clinic pfizer
moderna astrazeneca roche novartis merck eli-lilly novo-nordisk gsk sanofi arxiv nature science nasa esa noaa
usgs cern nsf mit stanford harvard berkeley caltech max-planck tesla ford gm toyota honda bmw mercedes-benz
volkswagen stellantis hyundai nhtsa transport-canada rivian lucid waymo stripe shopify paypal block visa
mastercard coinbase kraken salesforce servicenow reddit wikipedia wikimedia mozilla chromium w3c icann
internet-society ietf letsencrypt uk-government european-commission european-parliament bundesregierung
pib-india united-nations nato us-state-department us-doj ftc us-dod us-doe gao cbo ontario british-columbia
alberta finance-canada cra ircc global-affairs-canada ised eccc dnd-canada public-safety-canada
prime-minister-canada bbc-news nytimes the-guardian cnbc cnn al-jazeera france24 le-monde le-figaro radio-canada
cbc la-presse le-devoir journal-de-montreal tva-nouvelles 24-heures globe-and-mail national-post global-news
the-logic wsj financial-times the-economist axios npr washington-post scmp dw euronews der-spiegel el-pais semafor
google-news techcrunch the-verge ars-technica wired hacker-news the-register bleepingcomputer krebs-on-security
the-hacker-news the-record 404-media mit-technology-review

## Fragment files (2026-09-08)
| file | site class |
|---|---|
| `10-open-source.yaml` | open-source foundations, languages, frameworks, Linux/BSD distributions, dev tools |
| `11-central-banks-finance.yaml` | central banks worldwide, financial regulators, exchanges, rating agencies, major banks & asset managers |
| `12-energy-climate-weather.yaml` | energy agencies, grid operators, oil & gas, renewables, weather/hazard and climate bodies |
| `13-telecom-internet-infra.yaml` | telcos, network vendors, CDNs, CAs, RIRs, DNS providers, browsers |
| `14-retail-consumer-travel.yaml` | retail, consumer brands, food & beverage, hospitality, travel platforms, delivery |
| `15-gaming-entertainment.yaml` | gaming platforms/publishers/engines, streaming, music, studios |
| `16-universities-research.yaml` | universities, research institutes, journals, preprint servers, funders |
| `17-international-orgs-ngos-standards.yaml` | UN agencies, NGOs, standards bodies, think tanks |
| `18-sports.yaml` | leagues, federations, Olympic movement, anti-doping |
| `19-crypto-web3.yaml` | chains, exchanges, stablecoins, wallets, protocols |
| `20-transport-logistics-aviation-space.yaml` | carriers, postal, ports, airlines, aviation regulators, space companies |
| `21-consumer-safety-food-agri.yaml` | recall APIs, consumer-protection and food/agriculture agencies |
| `22-housing-labour-opendata.yaml` | housing data, labour, open-data portals |
| `23-enterprise-saas.yaml` | SaaS / B2B software with status pages |
| `24-health-systems-medtech.yaml` | health agencies abroad, hospital systems, insurers, medtech |
| `25-politics-elections-courts.yaml` | courts, election bodies, legislatures |
| `30-edgar-filings.yaml` | SEC EDGAR submissions for ~100 issuers (`edgar`) |
| `31-packages.yaml` | package registries (`package`) |
| `32-web-posture.yaml` | robots.txt / ai policy, security.txt, TLS, DNS, headers, RDAP, CT (`http`, `tls`, `dns`, `headers`, `rdap`, `jsonlist`) |
| `33-openapi.yaml` | public OpenAPI documents (`openapi`) |
| `34-status-json.yaml` | Instatus / Status.io / incident.io pages (`statusjson`) |
| `35-documents-data.yaml` | PDFs and CSV open data (`pdf`, `csv`) |

## Depth fragments (2026-09-11) — one file per domain, many sensors per organization
| file | coverage |
|---|---|
| `40-ai-frontier.yaml` | AI labs, model providers, inference clouds, dev tools: HF model-list APIs, GitHub SDK releases, pricing/docs/changelog pages, llms.txt, status pages, OpenRouter model list |
| `41-cloud-infrastructure.yaml` | clouds, PaaS, databases, CDNs, RIRs, CAs, observability: status pages (incl. OVHcloud's six Statuspage instances, GCP incidents JSON), release-note feeds, pricing/region/deprecation pages, OpenAPI specs, public pricing APIs |
| `42-cybersecurity.yaml` | CISA KEV + advisories, NVD 2.0 window, EPSS, GitHub Advisory DB (per ecosystem), EUVD, national CERTs, vendor PSIRTs, distro security, threat-intel research, ZDI, ransomware.live, security media (`first_party: false`) |
| `43-finance-markets.yaml` | central banks (statements, key rates, Valet/SDW/FRED CSV series), regulators, exchanges (halts, IPO calendar, symbol directories), rating agencies, multilaterals, Treasury/OFAC, statistics agencies, 100 EDGAR issuers, IR feeds |
| `44-governments.yaml` | Canada + provinces, US, EU, UK, FR, DE, IT, ES, JP, KR, AU, IN, BR, MX: news APIs, gazettes, legislation, procurement, CKAN portals, sanctions, emergency alerts (`country:` on every entry) |
| `45-science-health.yaml` | arXiv categories, bioRxiv/medRxiv, Crossref/OpenAlex retractions, journals, agencies (NASA/ESA/CERN/NSF), CDC/WHO/FDA openFDA/MHRA, ClinicalTrials.gov v2 (stopped trials, sponsors), Launch Library, science media |
| `46-transport-telecom-sports-news.yaml` | FAA/NTSB/TSB/EASA, airlines, rail, ports, NHTSA; CRTC/FCC/Ofcom, carriers, vendors; official leagues (NHL/NBA/MLB/NFL/FIA…, `llm: false`); world news media section feeds (`first_party: false`, `country`, `language`) |

## Breadth fragments, wave 2 (2026-09-11) — thousands more organizations
| file | coverage |
|---|---|
| `47-open-source-long-tail.yaml` | ~300 open-source projects and developer-tool vendors: languages, frameworks, build tooling, data/infra, observability, containers/IaC, desktop apps, CMS/e-commerce, ML tooling, OS/desktop environments, foundations (GitHub `releases.atom`, `tags.atom` + `kind: tags` for tag-only repos, org blogs); generated by `scripts/gen-open-source-long-tail.py` from the `-spec*.py` files |
| `48-saas-status-changelogs.yaml` | SaaS status pages (Statuspage / Instatus / Status.io / incident.io) and product changelogs of widely used tools; probe by `scripts/gen-status-probe.py` |
| `49-edgar-issuers.yaml` | ~420 additional SEC EDGAR issuers (Russell 1000 depth) via `scripts/gen-edgar2.py` |
| `49b-us-agencies-fedreg.yaml` | Federal Register documents per US federal agency (`jsonlist` on the Federal Register API, `scripts/gen-fedreg.py`) |
| `50-cities-regions-public-bodies.yaml` | sub-national public bodies: Canadian municipalities, US states and major cities, provinces/Länder/regions in EU, UK, AU; transit agencies, ports, utilities, health authorities (`country:` everywhere) |
| `51-world-governments-regulators.yaml` | rest-of-world national governments, ministries, central banks and regulators (Africa, Middle East, Asia-Pacific, Latin America, smaller EU states); `scripts/gen-world-gov.py` + `gen_world_gov_data*.py` |
| `52-corporate-pricing-legal-careers.yaml` | the "silent change" surface of ~250 major companies: pricing, terms of service, privacy policies, SLA, careers and leadership pages (`scripts/gen-corporate-pages.py`) |
| `53-sports-teams-entertainment-education.yaml` | official club sites of NHL / MLB / NFL / MLS / NBA / CFL and European football, federations (`llm: false`); studios, labels, publishers, museums, festivals; universities, school boards, accreditors, funders |
| `54-media-long-tail-trade-press-think-tanks.yaml` | regional and long-tail news media, trade press per industry, think tanks and research institutes in 68 countries (`first_party: false`, `country`, `language`) |
| `55-cross-fragment-extends.yaml` | `extend: true` sensors whose organization is declared in a later fragment than the one that found them (an extend can only reference an id declared in an earlier file) |

Gotchas learned while authoring 47–54: `country: NO` (Norway) and `language: no` parse as YAML booleans — quote
them (`"NO"`, `"no"`); `prune-fragment.py --keep-empty` leaves a dangling `sensors:` (null) when a source loses
its last sensor, which the seed schema rejects — remove the key by hand; the validator loads the whole registry,
so a parse error in *another* fragment fails your run too; many national government sites pass `curl` but fail
Node with "unable to verify the first certificate" (incomplete TLS chain) — probe with the validator, not curl;
sensor names in non-Latin scripts yield empty or duplicate sensor ids — give them ASCII English names; an
`extend: true` can only target an id declared in an *earlier* file (see `55-cross-fragment-extends.yaml`);
Q4/Notified IR feeds, most club sites (JS shells) and
WordPress sites with feeds disabled are unmonitorable — record them in the fragment header.

Gotchas learned while authoring 40–46: Hugging Face `owner/model` ids need `{key}` untouched in `urlTemplate`
(handled); AWS Health JSON is UTF-16 with BOM (handled by the fetcher); nature.com throttles per IP (validate
journal feeds at concurrency 1); the arXiv API asks for ≥ 3 s between requests (never tier A/S); Q4/Notified IR
feeds and most Cloudflare/Akamai newsrooms are blocked — record them in `notes:`; Reuters/AP have no public feeds.