# Source registry — authoring guide

The registry is config/sources.yaml (founding file, 271 organizations) plus fragments in config/sources.d/*.yaml, merged in file-name order by apps/engine/src/seeds.ts. Every organization becomes an entity; every sensors: entry becomes a real, polled endpoint. Nothing goes in the registry that has not been fetched and parsed successfully — a sensor that returns 403/404, HTML instead of a feed, or an empty list is a liability (noise, wasted budget, false "page removed" events), not coverage.

Coverage universes & the Source Factory (2026-09-13). Organizations that should be observed are listed per sector in config/coverage/*.yaml (guide: COVERAGE.md). The Factory discovers, validates, shadow-monitors and accepts their sensors automatically (sources.origin = 'factory', sensors.config.factory = true). cli.ts factory export turns accepted sensors into a fragment like the ones below for review and graduation into config/sources.d/.

# Fragment format

# config/sources.d/12-energy-climate.yaml — one site class per file, header comment says what and when.
sources:
  - id: iea                      # kebab-case, unique across ALL files (see the id list below)
    name: International Energy Agency
    domain: iea.org              # bare domain, no scheme
    homepage: https://www.iea.org   # optional, defaults to https://<domain>
    categories: [energy, government]
    tier: B                      # S 15–60 s · A 1–5 min · B 5–30 min · C 30 min–6 h · D 6–24 h
    weight: 1.1                  # importance multiplier (default 1; 1.2–1.5 for globally important orgs)
    aliases: [iea]               # lowercase alternative names for entity linking
    products:                    # optional child entities (product, AI_model, API, software, service, index…)
      - { name: World Energy Outlook, type: product, aliases: [weo] }
    discover: { rss: true, sitemap: false, status: false, pages: false }   # only ALLOWS discovery to probe
    llm: true                    # false for high-volume feeds (news wires, recalls firehoses, package streams)
    country: FR                  # optional ISO 3166-1 alpha-2 (upper-case); EU for EU bodies, INT for global bodies
    language: fr                 # optional ISO 639-1 when the content is not English
    first_party: false           # ONLY for media / aggregators reporting about others; omit for an organization's own channels
    notes: "why this source matters / what was blocked (optional)"
    sensors:
      - { name: news feed, url: "https://www.iea.org/news.rss", type: RSS, connector: rss, tier: B }
      - { name: status, url: "https://status.example.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S }

To add sensors to an organization that already exists in an earlier file, use extend: true (only id is required; sensors, products, aliases, categories, fallback, notes, country, language, first_party are merged — llm cannot be changed by an extend, edit the declaring file):

  - id: apple
    extend: true
    sensors:
      - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000320193.json", type: REST_API, connector: edgar, tier: B }

# Connector cheat-sheet (type → connector)

Prefer, in this order: official API → feed → status page → server-rendered HTML page. Never add a sensor whose only content is client-rendered (the validator reports it as thin).

# Rules

1. Validate before you write. From the repo root: node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts config/sources.d/<file>.yaml prints OK / WARN / FAIL per sensor. Only OK sensors stay. WARN empty list on an RSS feed = drop it. --probe <domain> runs the discovery engine (robots sitemaps, <link rel=alternate>, well-known feed paths, linked status pages) without a database — the fastest way to find a real feed.
2. 403 / 429 / Akamai / Cloudflare challenge = not a sensor (unless fallback: { scrapfly: true } is set on the source and the tier is C or D — reserved for a handful of high-value pages). Put blocked sites in notes: rather than pretending.
3. One source = one organization (the entity people search for), 1–4 sensors. A source without any validated sensor may still be listed with discover: { rss: true, sitemap: true } if the organization matters, but prefer to find a real endpoint.
4. Tiers: S only for status pages and security advisories of infrastructure people depend on; A for the news feed of a globally important organization; B default; C for pricing/legal/policy pages; D for slow reference pages (robots.txt, TLS, DNS, RDAP).
5. Volume: feeds publishing > 50 items/day (wire services, recall firehoses, package version streams) get llm: false on the source (heuristics only) and a maxItems cap.
6. Ids are kebab-case and unique across all files. Do not redefine an existing id; use extend: true.
7. Keep entries compact (flow style { … } for sensors and products, one per line) so files stay reviewable.
8. Categories are free strings but prefer the existing vocabulary: ai, cloud, developer, cyber, consumer-tech, semiconductors, finance, government, statistics, health, pharma, science, space, automotive, commerce, payments, crypto, enterprise, internet, standards, technology, news, media, politics — plus the class vocabulary added on 2026-09-08: energy, climate, weather, telecom, retail, travel, gaming, entertainment, education, research, ngo, international, sports, transport, logistics, aviation, food, agriculture, consumer-safety, real-estate, labour, open-data, legal, elections, open-source, packages, filings, web-policy, infrastructure.

# Existing source ids (founding file — do not redefine, extend: true to add sensors)

openai anthropic deepmind google-ai google-developers-ai microsoft-ai azure-ai meta-ai mistral huggingface cohere xai stability-ai replicate together-ai fireworks-ai groq cerebras perplexity nvidia aws azure google-cloud cloudflare cloudflare-radar digitalocean akamai fastly vercel netlify heroku fly-io render railway oracle-cloud ibm-cloud ovhcloud hetzner vultr linode github gitlab docker kubernetes nodejs python rust go php ruby postgresql mysql redis mongodb elastic hashicorp grafana sentry supabase firebase cisa nvd nist cert-cc mitre microsoft-security apple-security google-security-blog project-zero cisco-security palo-alto-networks fortinet crowdstrike mandiant rapid7 tenable sophos trend-micro malwarebytes hibp apple samsung google microsoft sony lg dell hp lenovo asus acer intel amd qualcomm arm tsmc micron western-digital seagate raspberry-pi sec federal-reserve us-treasury bank-of-canada ecb bank-of-england bank-of-japan finra cftc nasdaq nyse tmx sedar-plus fdic occ bis imf world-bank oecd fred canada statcan quebec federal-register white-house congress bls census bea eurostat fda health-canada ema clinicaltrials pubmed nih cdc who mayo-clinic cleveland-clinic pfizer moderna astrazeneca roche novartis merck eli-lilly novo-nordisk gsk sanofi arxiv nature science nasa esa noaa usgs cern nsf mit stanford harvard berkeley caltech max-planck tesla ford gm toyota honda bmw mercedes-benz volkswagen stellantis hyundai nhtsa transport-canada rivian lucid waymo stripe shopify paypal block visa mastercard coinbase kraken salesforce servicenow reddit wikipedia wikimedia mozilla chromium w3c icann internet-society ietf letsencrypt uk-government european-commission european-parliament bundesregierung pib-india united-nations nato us-state-department us-doj ftc us-dod us-doe gao cbo ontario british-columbia alberta finance-canada cra ircc global-affairs-canada ised eccc dnd-canada public-safety-canada prime-minister-canada bbc-news nytimes the-guardian cnbc cnn al-jazeera france24 le-monde le-figaro radio-canada cbc la-presse le-devoir journal-de-montreal tva-nouvelles 24-heures globe-and-mail national-post global-news the-logic wsj financial-times the-economist axios npr washington-post scmp dw euronews der-spiegel el-pais semafor google-news techcrunch the-verge ars-technica wired hacker-news the-register bleepingcomputer krebs-on-security the-hacker-news the-record 404-media mit-technology-review

# Fragment files (2026-09-08)

# Depth fragments (2026-09-11) — one file per domain, many sensors per organization

# Breadth fragments, wave 2 (2026-09-11) — thousands more organizations

Gotchas learned while authoring 47–54: country: NO (Norway) and language: no parse as YAML booleans — quote them ("NO", "no"); prune-fragment.py --keep-empty leaves a dangling sensors: (null) when a source loses its last sensor, which the seed schema rejects — remove the key by hand; the validator loads the whole registry, so a parse error in another fragment fails your run too; many national government sites pass curl but fail Node with "unable to verify the first certificate" (incomplete TLS chain) — probe with the validator, not curl; sensor names in non-Latin scripts yield empty or duplicate sensor ids — give them ASCII English names; an extend: true can only target an id declared in an earlier file (see 55-cross-fragment-extends.yaml); Q4/Notified IR feeds, most club sites (JS shells) and WordPress sites with feeds disabled are unmonitorable — record them in the fragment header.

Gotchas learned while authoring 40–46: Hugging Face owner/model ids need {key} untouched in urlTemplate (handled); AWS Health JSON is UTF-16 with BOM (handled by the fetcher); nature.com throttles per IP (validate journal feeds at concurrency 1); the arXiv API asks for ≥ 3 s between requests (never tier A/S); Q4/Notified IR feeds and most Cloudflare/Akamai newsrooms are blocked — record them in notes:; Reuters/AP have no public feeds.