1# Source registry — every domain AI Atlas crawls, with trust tier (1 official primary … 4 unverified), crawl policy and owner org.2# `key` is stable; `organization` refers to registry/organizations.yaml keys. Rate limits are requests/minute per domain.3sources:4 # ---------------------------------------------------------------- major AI labs (tier 1, official)5 - key: openai.com6 name: OpenAI — news & developer platform7 domain: openai.com8 organization: openai9 tier: 110 kind: website11 category: lab12 base_url: https://openai.com13 rate_limit_per_min: 2014 crawl_interval_s: 180015 priority: 016 - key: platform.openai.com17 name: OpenAI Platform docs18 domain: platform.openai.com19 organization: openai20 tier: 121 kind: docs22 category: lab23 base_url: https://platform.openai.com24 rate_limit_per_min: 1225 crawl_interval_s: 360026 priority: 027 - key: anthropic.com28 name: Anthropic — news29 domain: anthropic.com30 organization: anthropic31 tier: 132 kind: website33 category: lab34 base_url: https://www.anthropic.com35 rate_limit_per_min: 2036 crawl_interval_s: 180037 priority: 038 - key: docs.claude.com39 name: Claude Developer Platform docs40 domain: docs.claude.com41 organization: anthropic42 tier: 143 kind: docs44 category: lab45 base_url: https://docs.claude.com46 rate_limit_per_min: 2047 crawl_interval_s: 360048 priority: 049 - key: deepmind.google50 name: Google DeepMind — blog51 domain: deepmind.google52 organization: google-deepmind53 tier: 154 kind: website55 category: lab56 base_url: https://deepmind.google57 rate_limit_per_min: 2058 crawl_interval_s: 360059 priority: 060 - key: ai.google.dev61 name: Google AI for Developers (Gemini API docs)62 domain: ai.google.dev63 organization: google64 tier: 165 kind: docs66 category: lab67 base_url: https://ai.google.dev68 rate_limit_per_min: 2069 crawl_interval_s: 360070 priority: 071 - key: ai.meta.com72 name: Meta AI — blog & research73 domain: ai.meta.com74 organization: meta-ai75 tier: 176 kind: website77 category: lab78 base_url: https://ai.meta.com79 rate_limit_per_min: 1580 crawl_interval_s: 360081 priority: 082 - key: llama.com83 name: Llama — official site84 domain: llama.com85 organization: meta-ai86 tier: 187 kind: website88 category: lab89 base_url: https://www.llama.com90 rate_limit_per_min: 1591 crawl_interval_s: 720092 priority: 093 - key: microsoft.com/research94 name: Microsoft Research — publications & blog95 domain: microsoft.com96 organization: microsoft97 tier: 198 kind: feed99 category: lab100 base_url: https://www.microsoft.com/en-us/research101 rate_limit_per_min: 15102 crawl_interval_s: 7200103 priority: 1104 - key: blogs.nvidia.com105 name: NVIDIA — blog & developer blog106 domain: nvidia.com107 organization: nvidia108 tier: 1109 kind: feed110 category: hardware111 base_url: https://blogs.nvidia.com112 rate_limit_per_min: 15113 crawl_interval_s: 7200114 priority: 1115 - key: machinelearning.apple.com116 name: Apple Machine Learning Research117 domain: machinelearning.apple.com118 organization: apple119 tier: 1120 kind: feed121 category: lab122 base_url: https://machinelearning.apple.com123 rate_limit_per_min: 15124 crawl_interval_s: 21600125 priority: 1126 - key: mistral.ai127 name: Mistral AI — news, models & pricing128 domain: mistral.ai129 organization: mistral130 tier: 1131 kind: website132 category: lab133 base_url: https://mistral.ai134 rate_limit_per_min: 15135 crawl_interval_s: 3600136 priority: 0137 - key: docs.mistral.ai138 name: Mistral AI docs139 domain: docs.mistral.ai140 organization: mistral141 tier: 1142 kind: docs143 category: lab144 base_url: https://docs.mistral.ai145 rate_limit_per_min: 15146 crawl_interval_s: 3600147 priority: 0148 - key: qwenlm.github.io149 name: Qwen — official blog150 domain: qwenlm.github.io151 organization: qwen152 tier: 1153 kind: feed154 category: lab155 base_url: https://qwenlm.github.io156 rate_limit_per_min: 15157 crawl_interval_s: 3600158 priority: 0159 - key: deepseek.com160 name: DeepSeek — site & API docs161 domain: deepseek.com162 organization: deepseek163 tier: 1164 kind: docs165 category: lab166 base_url: https://api-docs.deepseek.com167 rate_limit_per_min: 15168 crawl_interval_s: 3600169 priority: 0170 - key: cohere.com171 name: Cohere — docs & blog172 domain: cohere.com173 organization: cohere174 tier: 1175 kind: docs176 category: lab177 base_url: https://docs.cohere.com178 rate_limit_per_min: 15179 crawl_interval_s: 7200180 priority: 1181 - key: x.ai182 name: xAI — docs183 domain: x.ai184 organization: xai185 tier: 1186 kind: docs187 category: lab188 base_url: https://docs.x.ai189 rate_limit_per_min: 12190 crawl_interval_s: 7200191 priority: 1192 # ---------------------------------------------------------------- model & code ecosystems193 - key: huggingface.co194 name: Hugging Face Hub (public pages, model cards, papers)195 domain: huggingface.co196 organization: huggingface197 tier: 2198 kind: repository199 category: hub200 base_url: https://huggingface.co201 rate_limit_per_min: 30202 crawl_interval_s: 3600203 priority: 0204 notes: Model cards are tier-1 statements by their authors; hub metadata (downloads, likes) is community signal.205 - key: github.com206 name: GitHub public repositories (HTML, releases.atom, raw files)207 domain: github.com208 organization: github209 tier: 2210 kind: repository211 category: code212 base_url: https://github.com213 rate_limit_per_min: 20214 crawl_interval_s: 3600215 priority: 0216 - key: pypi.org217 name: PyPI public package metadata218 domain: pypi.org219 organization: python-software-foundation220 tier: 2221 kind: registry222 category: code223 base_url: https://pypi.org224 rate_limit_per_min: 30225 crawl_interval_s: 21600226 priority: 1227 # ---------------------------------------------------------------- research228 - key: arxiv.org229 name: arXiv (Atom API + RSS)230 domain: arxiv.org231 organization: arxiv232 tier: 1233 kind: feed234 category: research235 base_url: https://export.arxiv.org236 rate_limit_per_min: 4237 crawl_interval_s: 21600238 priority: 0239 notes: arXiv asks for ≤ 1 request / 3 s on the export API.240 - key: openreview.net241 name: OpenReview242 domain: openreview.net243 organization: openreview244 tier: 1245 kind: registry246 category: research247 base_url: https://api2.openreview.net248 rate_limit_per_min: 10249 crawl_interval_s: 86400250 priority: 1251 # ---------------------------------------------------------------- providers (pricing pages, model lists)252 - key: openrouter.ai253 name: OpenRouter public model & pricing listing254 domain: openrouter.ai255 organization: openrouter256 tier: 2257 kind: registry258 category: provider259 base_url: https://openrouter.ai260 rate_limit_per_min: 10261 crawl_interval_s: 3600262 priority: 0263 notes: Aggregated secondary pricing across providers; tier 2 by design — official provider pages win conflicts.264 - key: groq.com265 name: Groq — docs & pricing266 domain: groq.com267 organization: groq268 tier: 1269 kind: docs270 category: provider271 base_url: https://console.groq.com272 rate_limit_per_min: 12273 crawl_interval_s: 7200274 priority: 1275 - key: together.ai276 name: Together AI — pricing277 domain: together.ai278 organization: together-ai279 tier: 1280 kind: website281 category: provider282 base_url: https://www.together.ai283 rate_limit_per_min: 12284 crawl_interval_s: 7200285 priority: 1286 - key: fireworks.ai287 name: Fireworks AI — pricing288 domain: fireworks.ai289 organization: fireworks-ai290 tier: 1291 kind: website292 category: provider293 base_url: https://fireworks.ai294 rate_limit_per_min: 12295 crawl_interval_s: 7200296 priority: 1297 # ---------------------------------------------------------------- benchmarks / leaderboards298 - key: aider.chat299 name: Aider polyglot leaderboard (data file on GitHub)300 domain: aider.chat301 organization: aider302 tier: 2303 kind: leaderboard304 category: benchmark305 base_url: https://aider.chat/docs/leaderboards/306 rate_limit_per_min: 10307 crawl_interval_s: 21600308 priority: 1309 - key: swebench.com310 name: SWE-bench leaderboards311 domain: swebench.com312 organization: swe-bench313 tier: 2314 kind: leaderboard315 category: benchmark316 base_url: https://www.swebench.com317 rate_limit_per_min: 10318 crawl_interval_s: 21600319 priority: 1320 - key: livebench.ai321 name: LiveBench322 domain: livebench.ai323 organization: livebench324 tier: 2325 kind: leaderboard326 category: benchmark327 base_url: https://livebench.ai328 rate_limit_per_min: 10329 crawl_interval_s: 43200330 priority: 2331 - key: artificialanalysis.ai332 name: Artificial Analysis333 domain: artificialanalysis.ai334 organization: artificial-analysis335 tier: 2336 kind: leaderboard337 category: benchmark338 base_url: https://artificialanalysis.ai339 rate_limit_per_min: 6340 crawl_interval_s: 43200341 priority: 2342 # ---------------------------------------------------------------- hardware manufacturers343 - key: apple.com344 name: Apple — Mac tech specs345 domain: apple.com346 organization: apple347 tier: 1348 kind: website349 category: hardware350 base_url: https://www.apple.com351 rate_limit_per_min: 10352 crawl_interval_s: 86400353 priority: 1354 - key: nvidia.com355 name: NVIDIA — data center product pages356 domain: nvidia.com357 organization: nvidia358 tier: 1359 kind: website360 category: hardware361 base_url: https://www.nvidia.com362 rate_limit_per_min: 10363 crawl_interval_s: 86400364 priority: 1365 - key: amd.com366 name: AMD — Instinct accelerators367 domain: amd.com368 organization: amd369 tier: 1370 kind: website371 category: hardware372 base_url: https://www.amd.com373 rate_limit_per_min: 6374 crawl_interval_s: 86400375 priority: 2376 # ---------------------------------------------------------------- curated registries (AI Atlas itself)377 - key: ai-atlas.registry378 name: AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)379 domain: ai-atlas.co380 organization: ai-atlas381 tier: 2382 kind: registry383 category: registry384 base_url: https://www.ai-atlas.co/sources385 rate_limit_per_min: 1000386 crawl_interval_s: 86400387 priority: 1388