TypeScript 55.4%
Python 43.2%
SQL 1.2%
1# Source registry — authoring guide23The registry is `config/sources.yaml` (founding file, 271 organizations) plus fragments in4`config/sources.d/*.yaml`, merged in file-name order by `apps/engine/src/seeds.ts`. Every organization becomes an5entity; every `sensors:` entry becomes a real, polled endpoint. **Nothing goes in the registry that has not been6fetched and parsed successfully** — a sensor that returns 403/404, HTML instead of a feed, or an empty list is a7liability (noise, wasted budget, false "page removed" events), not coverage.89> **Coverage universes & the Source Factory (2026-09-13).** Organizations that *should* be observed are listed per sector10> in `config/coverage/*.yaml` (guide: `COVERAGE.md`). The Factory discovers, validates, shadow-monitors and accepts their11> sensors automatically (`sources.origin = 'factory'`, `sensors.config.factory = true`). `cli.ts factory export` turns12> accepted sensors into a fragment like the ones below for review and graduation into `config/sources.d/`.1314## Fragment format1516```yaml17# config/sources.d/12-energy-climate.yaml — one site class per file, header comment says what and when.18sources:19 - id: iea # kebab-case, unique across ALL files (see the id list below)20 name: International Energy Agency21 domain: iea.org # bare domain, no scheme22 homepage: https://www.iea.org # optional, defaults to https://<domain>23 categories: [energy, government]24 tier: B # S 15–60 s · A 1–5 min · B 5–30 min · C 30 min–6 h · D 6–24 h25 weight: 1.1 # importance multiplier (default 1; 1.2–1.5 for globally important orgs)26 aliases: [iea] # lowercase alternative names for entity linking27 products: # optional child entities (product, AI_model, API, software, service, index…)28 - { name: World Energy Outlook, type: product, aliases: [weo] }29 discover: { rss: true, sitemap: false, status: false, pages: false } # only ALLOWS discovery to probe30 llm: true # false for high-volume feeds (news wires, recalls firehoses, package streams)31 country: FR # optional ISO 3166-1 alpha-2 (upper-case); EU for EU bodies, INT for global bodies32 language: fr # optional ISO 639-1 when the content is not English33 first_party: false # ONLY for media / aggregators reporting about others; omit for an organization's own channels34 notes: "why this source matters / what was blocked (optional)"35 sensors:36 - { name: news feed, url: "https://www.iea.org/news.rss", type: RSS, connector: rss, tier: B }37 - { name: status, url: "https://status.example.com/api/v2/summary.json", type: STATUSPAGE, connector: statuspage, tier: S }38```3940To add sensors to an organization that already exists in an earlier file, use `extend: true` (only `id` is41required; `sensors`, `products`, `aliases`, `categories`, `fallback`, `notes`, `country`, `language`, `first_party`42are merged — `llm` cannot be changed by an extend, edit the declaring file):4344```yaml45 - id: apple46 extend: true47 sensors:48 - { name: edgar filings, url: "https://data.sec.gov/submissions/CIK0000320193.json", type: REST_API, connector: edgar, tier: B }49```5051## Connector cheat-sheet (type → connector)5253| connector | type | what it is for | config |54|---|---|---|---|55| `rss` | RSS / ATOM | RSS 2.0, Atom, RDF, JSON Feed. GitHub `releases.atom`, GitLab `…/-/tags?format=atom`, YouTube `feeds/videos.xml?channel_id=`, Blogger `feeds/posts/summary?max-results=25`, arXiv API Atom | `maxItems` |56| `sitemap` | SITEMAP | sitemap index / urlset / news sitemap (one per source, tier A/B) | `maxUrls`, `maxChildren` |57| `statuspage` | STATUSPAGE | Atlassian Statuspage **`/api/v2/summary.json`** only (tier S) | — |58| `statusjson` | STATUSPAGE | Instatus `/summary.json`, Status.io `/1.0/status/<id>`, incident.io `/api/v1/summary` (tier S) | `flavor: instatus\|statusio\|incidentio` (auto-detected) |59| `github` | GITHUB_RELEASE | `https://github.com/<owner>/<repo>/releases.atom` (or `/tags.atom`, `/commits/<branch>.atom`) | `{ repo: owner/name, kind: releases\|tags\|commits\|advisories }` |60| `jsonlist` | REST_API / JSON | any official JSON API returning an array of records (recalls, alerts, incidents, datasets) | `itemsPath`, `keyField`, `titleField`/`titleTemplate`, `urlField`/`urlTemplate`, `summaryField`, `dateField`, `compareFields`, `maxItems`, `headers`, `noConditional`; URL placeholders `{now}`, `{now-2h}`, `{now-30d}` |61| `http` | HTML / JSON / XML / FILE | a server-rendered page whose text changes matter (pricing, changelog, advisories index, release notes, robots.txt, security.txt, llms.txt) | `selector`, `keepChrome`, `jsonPath`, `ignoreKeys`, `headers`, `method: HEAD` |62| `package` | REST_API | package registries: npm, PyPI, crates.io, RubyGems, NuGet, Packagist, Hex, Go proxy, Homebrew, Docker Hub tags | `{ registry: npm\|pypi\|crates\|rubygems\|nuget\|packagist\|hex\|goproxy\|homebrew\|dockerhub, name }` |63| `edgar` | REST_API | SEC EDGAR company submissions (`https://data.sec.gov/submissions/CIK##########.json`) → filings list | `forms: [8-K, 10-K, 10-Q, S-1, 6-K, 4]` (default: all but 4/144) |64| `openapi` | JSON | an OpenAPI/Swagger document (JSON or YAML) → operations list, API-contract changes | — |65| `csv` | FILE | CSV/TSV open-data file → keyed rows | `keyColumn`, `compareColumns`, `delimiter`, `maxRows` |66| `pdf` | FILE | a PDF whose text matters (central-bank statement, official notice) | `maxPages` |67| `dns` | DNS | `dns://example.com` → A/AAAA/NS/MX/TXT/CAA records | `records` |68| `tls` | TLS | `tls://example.com` → certificate issuer, SANs, validity | — |69| `headers` | HTTP_HEADERS | `https://…` → security/infra response headers (HSTS, CSP, server, CDN) | — |70| `rdap` | JSON | `https://rdap.org/domain/example.com` (or a registry RDAP URL) → registrar, status, nameservers, expiry | — |7172Prefer, in this order: official API → feed → status page → server-rendered HTML page. Never add a sensor whose73only content is client-rendered (the validator reports it as `thin`).7475## Rules761. **Validate before you write.** From the repo root:77 `node node_modules/tsx/dist/cli.mjs apps/engine/src/validate.ts config/sources.d/<file>.yaml`78 prints OK / WARN / FAIL per sensor. Only **OK** sensors stay. `WARN empty list` on an RSS feed = drop it.79 `--probe <domain>` runs the discovery engine (robots sitemaps, `<link rel=alternate>`, well-known feed paths,80 linked status pages) without a database — the fastest way to find a real feed.812. **403 / 429 / Akamai / Cloudflare challenge = not a sensor** (unless `fallback: { scrapfly: true }` is set on the82 source *and* the tier is C or D — reserved for a handful of high-value pages). Put blocked sites in `notes:`83 rather than pretending.843. **One source = one organization** (the entity people search for), 1–4 sensors. A source without any validated85 sensor may still be listed with `discover: { rss: true, sitemap: true }` if the organization matters, but86 prefer to find a real endpoint.874. **Tiers**: S only for status pages and security advisories of infrastructure people depend on; A for the news88 feed of a globally important organization; B default; C for pricing/legal/policy pages; D for slow reference89 pages (robots.txt, TLS, DNS, RDAP).905. **Volume**: feeds publishing > 50 items/day (wire services, recall firehoses, package version streams) get91 `llm: false` on the source (heuristics only) and a `maxItems` cap.926. **Ids** are kebab-case and unique across all files. Do not redefine an existing id; use `extend: true`.937. Keep entries compact (flow style `{ … }` for sensors and products, one per line) so files stay reviewable.948. Categories are free strings but prefer the existing vocabulary: ai, cloud, developer, cyber, consumer-tech,95 semiconductors, finance, government, statistics, health, pharma, science, space, automotive, commerce, payments,96 crypto, enterprise, internet, standards, technology, news, media, politics — plus the class vocabulary added on97 2026-09-08: energy, climate, weather, telecom, retail, travel, gaming, entertainment, education, research,98 ngo, international, sports, transport, logistics, aviation, food, agriculture, consumer-safety, real-estate,99 labour, open-data, legal, elections, open-source, packages, filings, web-policy, infrastructure.100101## Existing source ids (founding file — do not redefine, `extend: true` to add sensors)102openai anthropic deepmind google-ai google-developers-ai microsoft-ai azure-ai meta-ai mistral huggingface cohere103xai stability-ai replicate together-ai fireworks-ai groq cerebras perplexity nvidia aws azure google-cloud104cloudflare cloudflare-radar digitalocean akamai fastly vercel netlify heroku fly-io render railway oracle-cloud105ibm-cloud ovhcloud hetzner vultr linode github gitlab docker kubernetes nodejs python rust go php ruby postgresql106mysql redis mongodb elastic hashicorp grafana sentry supabase firebase cisa nvd nist cert-cc mitre107microsoft-security apple-security google-security-blog project-zero cisco-security palo-alto-networks fortinet108crowdstrike mandiant rapid7 tenable sophos trend-micro malwarebytes hibp apple samsung google microsoft sony lg109dell hp lenovo asus acer intel amd qualcomm arm tsmc micron western-digital seagate raspberry-pi sec110federal-reserve us-treasury bank-of-canada ecb bank-of-england bank-of-japan finra cftc nasdaq nyse tmx111sedar-plus fdic occ bis imf world-bank oecd fred canada statcan quebec federal-register white-house congress bls112census bea eurostat fda health-canada ema clinicaltrials pubmed nih cdc who mayo-clinic cleveland-clinic pfizer113moderna astrazeneca roche novartis merck eli-lilly novo-nordisk gsk sanofi arxiv nature science nasa esa noaa114usgs cern nsf mit stanford harvard berkeley caltech max-planck tesla ford gm toyota honda bmw mercedes-benz115volkswagen stellantis hyundai nhtsa transport-canada rivian lucid waymo stripe shopify paypal block visa116mastercard coinbase kraken salesforce servicenow reddit wikipedia wikimedia mozilla chromium w3c icann117internet-society ietf letsencrypt uk-government european-commission european-parliament bundesregierung118pib-india united-nations nato us-state-department us-doj ftc us-dod us-doe gao cbo ontario british-columbia119alberta finance-canada cra ircc global-affairs-canada ised eccc dnd-canada public-safety-canada120prime-minister-canada bbc-news nytimes the-guardian cnbc cnn al-jazeera france24 le-monde le-figaro radio-canada121cbc la-presse le-devoir journal-de-montreal tva-nouvelles 24-heures globe-and-mail national-post global-news122the-logic wsj financial-times the-economist axios npr washington-post scmp dw euronews der-spiegel el-pais semafor123google-news techcrunch the-verge ars-technica wired hacker-news the-register bleepingcomputer krebs-on-security124the-hacker-news the-record 404-media mit-technology-review125126## Fragment files (2026-09-08)127| file | site class |128|---|---|129| `10-open-source.yaml` | open-source foundations, languages, frameworks, Linux/BSD distributions, dev tools |130| `11-central-banks-finance.yaml` | central banks worldwide, financial regulators, exchanges, rating agencies, major banks & asset managers |131| `12-energy-climate-weather.yaml` | energy agencies, grid operators, oil & gas, renewables, weather/hazard and climate bodies |132| `13-telecom-internet-infra.yaml` | telcos, network vendors, CDNs, CAs, RIRs, DNS providers, browsers |133| `14-retail-consumer-travel.yaml` | retail, consumer brands, food & beverage, hospitality, travel platforms, delivery |134| `15-gaming-entertainment.yaml` | gaming platforms/publishers/engines, streaming, music, studios |135| `16-universities-research.yaml` | universities, research institutes, journals, preprint servers, funders |136| `17-international-orgs-ngos-standards.yaml` | UN agencies, NGOs, standards bodies, think tanks |137| `18-sports.yaml` | leagues, federations, Olympic movement, anti-doping |138| `19-crypto-web3.yaml` | chains, exchanges, stablecoins, wallets, protocols |139| `20-transport-logistics-aviation-space.yaml` | carriers, postal, ports, airlines, aviation regulators, space companies |140| `21-consumer-safety-food-agri.yaml` | recall APIs, consumer-protection and food/agriculture agencies |141| `22-housing-labour-opendata.yaml` | housing data, labour, open-data portals |142| `23-enterprise-saas.yaml` | SaaS / B2B software with status pages |143| `24-health-systems-medtech.yaml` | health agencies abroad, hospital systems, insurers, medtech |144| `25-politics-elections-courts.yaml` | courts, election bodies, legislatures |145| `30-edgar-filings.yaml` | SEC EDGAR submissions for ~100 issuers (`edgar`) |146| `31-packages.yaml` | package registries (`package`) |147| `32-web-posture.yaml` | robots.txt / ai policy, security.txt, TLS, DNS, headers, RDAP, CT (`http`, `tls`, `dns`, `headers`, `rdap`, `jsonlist`) |148| `33-openapi.yaml` | public OpenAPI documents (`openapi`) |149| `34-status-json.yaml` | Instatus / Status.io / incident.io pages (`statusjson`) |150| `35-documents-data.yaml` | PDFs and CSV open data (`pdf`, `csv`) |151152## Depth fragments (2026-09-11) — one file per domain, many sensors per organization153| file | coverage |154|---|---|155| `40-ai-frontier.yaml` | AI labs, model providers, inference clouds, dev tools: HF model-list APIs, GitHub SDK releases, pricing/docs/changelog pages, llms.txt, status pages, OpenRouter model list |156| `41-cloud-infrastructure.yaml` | clouds, PaaS, databases, CDNs, RIRs, CAs, observability: status pages (incl. OVHcloud's six Statuspage instances, GCP incidents JSON), release-note feeds, pricing/region/deprecation pages, OpenAPI specs, public pricing APIs |157| `42-cybersecurity.yaml` | CISA KEV + advisories, NVD 2.0 window, EPSS, GitHub Advisory DB (per ecosystem), EUVD, national CERTs, vendor PSIRTs, distro security, threat-intel research, ZDI, ransomware.live, security media (`first_party: false`) |158| `43-finance-markets.yaml` | central banks (statements, key rates, Valet/SDW/FRED CSV series), regulators, exchanges (halts, IPO calendar, symbol directories), rating agencies, multilaterals, Treasury/OFAC, statistics agencies, 100 EDGAR issuers, IR feeds |159| `44-governments.yaml` | Canada + provinces, US, EU, UK, FR, DE, IT, ES, JP, KR, AU, IN, BR, MX: news APIs, gazettes, legislation, procurement, CKAN portals, sanctions, emergency alerts (`country:` on every entry) |160| `45-science-health.yaml` | arXiv categories, bioRxiv/medRxiv, Crossref/OpenAlex retractions, journals, agencies (NASA/ESA/CERN/NSF), CDC/WHO/FDA openFDA/MHRA, ClinicalTrials.gov v2 (stopped trials, sponsors), Launch Library, science media |161| `46-transport-telecom-sports-news.yaml` | FAA/NTSB/TSB/EASA, airlines, rail, ports, NHTSA; CRTC/FCC/Ofcom, carriers, vendors; official leagues (NHL/NBA/MLB/NFL/FIA…, `llm: false`); world news media section feeds (`first_party: false`, `country`, `language`) |162163## Breadth fragments, wave 2 (2026-09-11) — thousands more organizations164| file | coverage |165|---|---|166| `47-open-source-long-tail.yaml` | ~300 open-source projects and developer-tool vendors: languages, frameworks, build tooling, data/infra, observability, containers/IaC, desktop apps, CMS/e-commerce, ML tooling, OS/desktop environments, foundations (GitHub `releases.atom`, `tags.atom` + `kind: tags` for tag-only repos, org blogs); generated by `scripts/gen-open-source-long-tail.py` from the `-spec*.py` files |167| `48-saas-status-changelogs.yaml` | SaaS status pages (Statuspage / Instatus / Status.io / incident.io) and product changelogs of widely used tools; probe by `scripts/gen-status-probe.py` |168| `49-edgar-issuers.yaml` | ~420 additional SEC EDGAR issuers (Russell 1000 depth) via `scripts/gen-edgar2.py` |169| `49b-us-agencies-fedreg.yaml` | Federal Register documents per US federal agency (`jsonlist` on the Federal Register API, `scripts/gen-fedreg.py`) |170| `50-cities-regions-public-bodies.yaml` | sub-national public bodies: Canadian municipalities, US states and major cities, provinces/Länder/regions in EU, UK, AU; transit agencies, ports, utilities, health authorities (`country:` everywhere) |171| `51-world-governments-regulators.yaml` | rest-of-world national governments, ministries, central banks and regulators (Africa, Middle East, Asia-Pacific, Latin America, smaller EU states); `scripts/gen-world-gov.py` + `gen_world_gov_data*.py` |172| `52-corporate-pricing-legal-careers.yaml` | the "silent change" surface of ~250 major companies: pricing, terms of service, privacy policies, SLA, careers and leadership pages (`scripts/gen-corporate-pages.py`) |173| `53-sports-teams-entertainment-education.yaml` | official club sites of NHL / MLB / NFL / MLS / NBA / CFL and European football, federations (`llm: false`); studios, labels, publishers, museums, festivals; universities, school boards, accreditors, funders |174| `54-media-long-tail-trade-press-think-tanks.yaml` | regional and long-tail news media, trade press per industry, think tanks and research institutes in 68 countries (`first_party: false`, `country`, `language`) |175| `56-ai-model-watch.yaml` | **AI Model Watch** (2026-09-13): 654 validated sensors on 272 organizations — Hugging Face model/dataset lists for 183 + 65 authors, provider model catalogues / pricing / rate-limit / deprecation docs (OpenAI, Anthropic, Vertex, Azure OpenAI, Bedrock, Mistral, Cohere, DeepSeek, Moonshot, Alibaba, OCI, watsonx, Together, Groq, Cerebras, Replicate, Baseten…), NVIDIA NIM and Vercel AI Gateway model APIs, leaderboards (HELM, AlpacaEval, BFCL, Aider, MLPerf, ARC Prize, SWE-bench…), SDK/runtime/agent-framework releases; `hf-hub-firehose` and `hf-community-quantizers` are `llm: false` |176| `57-packages-top.yaml` | top packages per registry (PyPI 1 200 · npm 1 200 · crates 297 · RubyGems 98 · NuGet 99 · Docker Hub 91 · Go 150), `package` connector, tier D; generator `scripts/gen-packages-top.py` |177| `58-status-pages-wave3.yaml` | 530 more JSON status pages (498 Statuspage, 32 Instatus) on 530 organizations in ~40 countries; probe `scripts/gen-status-probe3.py` (`gate` step drops squatted / abandoned `<brand>.statuspage.io` pages) |178| `55-cross-fragment-extends.yaml` | `extend: true` sensors whose organization is declared in a later fragment than the one that found them (an extend can only reference an id declared in an earlier file) |179180Gotchas learned while authoring 47–54: `country: NO` (Norway) and `language: no` parse as YAML booleans — quote181them (`"NO"`, `"no"`); `prune-fragment.py --keep-empty` leaves a dangling `sensors:` (null) when a source loses182its last sensor, which the seed schema rejects — remove the key by hand; the validator loads the whole registry,183so a parse error in *another* fragment fails your run too; many national government sites pass `curl` but fail184Node with "unable to verify the first certificate" (incomplete TLS chain) — probe with the validator, not curl;185sensor names in non-Latin scripts yield empty or duplicate sensor ids — give them ASCII English names; an186`extend: true` can only target an id declared in an *earlier* file (see `55-cross-fragment-extends.yaml`);187Q4/Notified IR feeds, most club sites (JS shells) and188WordPress sites with feeds disabled are unmonitorable — record them in the fragment header.189190Gotchas learned while authoring 40–46: Hugging Face `owner/model` ids need `{key}` untouched in `urlTemplate`191(handled); AWS Health JSON is UTF-16 with BOM (handled by the fetcher); nature.com throttles per IP (validate192journal feeds at concurrency 1); the arXiv API asks for ≥ 3 s between requests (never tier A/S); Q4/Notified IR193feeds and most Cloudflare/Akamai newsrooms are blocked — record them in `notes:`; Reuters/AP have no public feeds.194