HTML 77.2%
TypeScript 10.5%
Python 9.6%
JavaScript 2.5%
1# Source registry — every domain AI Atlas crawls, with trust tier (1 official primary … 4 unverified), crawl policy and owner org.2# `key` is stable; `organization` refers to registry/organizations.yaml keys. Rate limits are requests/minute per domain.3sources:4 # ---------------------------------------------------------------- major AI labs (tier 1, official)5 - key: openai.com6 name: OpenAI — news & developer platform7 domain: openai.com8 organization: openai9 tier: 110 kind: website11 category: lab12 base_url: https://openai.com13 rate_limit_per_min: 2014 crawl_interval_s: 180015 priority: 016 - key: platform.openai.com17 name: OpenAI Platform docs18 domain: platform.openai.com19 organization: openai20 tier: 121 kind: docs22 category: lab23 base_url: https://platform.openai.com24 rate_limit_per_min: 1225 crawl_interval_s: 360026 priority: 027 - key: anthropic.com28 name: Anthropic — news29 domain: anthropic.com30 organization: anthropic31 tier: 132 kind: website33 category: lab34 base_url: https://www.anthropic.com35 rate_limit_per_min: 2036 crawl_interval_s: 180037 priority: 038 - key: docs.claude.com39 name: Claude Developer Platform docs40 domain: docs.claude.com41 organization: anthropic42 tier: 143 kind: docs44 category: lab45 base_url: https://docs.claude.com46 rate_limit_per_min: 2047 crawl_interval_s: 360048 priority: 049 - key: deepmind.google50 name: Google DeepMind — blog51 domain: deepmind.google52 organization: google-deepmind53 tier: 154 kind: website55 category: lab56 base_url: https://deepmind.google57 rate_limit_per_min: 2058 crawl_interval_s: 360059 priority: 060 - key: ai.google.dev61 name: Google AI for Developers (Gemini API docs)62 domain: ai.google.dev63 organization: google64 tier: 165 kind: docs66 category: lab67 base_url: https://ai.google.dev68 rate_limit_per_min: 2069 crawl_interval_s: 360070 priority: 071 - key: ai.meta.com72 name: Meta AI — blog & research73 domain: ai.meta.com74 organization: meta-ai75 tier: 176 kind: website77 category: lab78 base_url: https://ai.meta.com79 rate_limit_per_min: 1580 crawl_interval_s: 360081 priority: 082 - key: llama.com83 name: Llama — official site84 domain: llama.com85 organization: meta-ai86 tier: 187 kind: website88 category: lab89 base_url: https://www.llama.com90 rate_limit_per_min: 1591 crawl_interval_s: 720092 priority: 093 - key: microsoft.com/research94 name: Microsoft Research — publications & blog95 domain: microsoft.com96 organization: microsoft97 tier: 198 kind: feed99 category: lab100 base_url: https://www.microsoft.com/en-us/research101 rate_limit_per_min: 15102 crawl_interval_s: 7200103 priority: 1104 - key: blogs.nvidia.com105 name: NVIDIA — blog & developer blog106 domain: nvidia.com107 organization: nvidia108 tier: 1109 kind: feed110 category: hardware111 base_url: https://blogs.nvidia.com112 rate_limit_per_min: 15113 crawl_interval_s: 7200114 priority: 1115 - key: machinelearning.apple.com116 name: Apple Machine Learning Research117 domain: machinelearning.apple.com118 organization: apple119 tier: 1120 kind: feed121 category: lab122 base_url: https://machinelearning.apple.com123 rate_limit_per_min: 15124 crawl_interval_s: 21600125 priority: 1126 - key: mistral.ai127 name: Mistral AI — news, models & pricing128 domain: mistral.ai129 organization: mistral130 tier: 1131 kind: website132 category: lab133 base_url: https://mistral.ai134 rate_limit_per_min: 15135 crawl_interval_s: 3600136 priority: 0137 - key: docs.mistral.ai138 name: Mistral AI docs139 domain: docs.mistral.ai140 organization: mistral141 tier: 1142 kind: docs143 category: lab144 base_url: https://docs.mistral.ai145 rate_limit_per_min: 15146 crawl_interval_s: 3600147 priority: 0148 - key: qwenlm.github.io149 name: Qwen — official blog150 domain: qwenlm.github.io151 organization: qwen152 tier: 1153 kind: feed154 category: lab155 base_url: https://qwenlm.github.io156 rate_limit_per_min: 15157 crawl_interval_s: 3600158 priority: 0159 - key: deepseek.com160 name: DeepSeek — site & API docs161 domain: deepseek.com162 organization: deepseek163 tier: 1164 kind: docs165 category: lab166 base_url: https://api-docs.deepseek.com167 rate_limit_per_min: 15168 crawl_interval_s: 3600169 priority: 0170 - key: cohere.com171 name: Cohere — docs & blog172 domain: cohere.com173 organization: cohere174 tier: 1175 kind: docs176 category: lab177 base_url: https://docs.cohere.com178 rate_limit_per_min: 15179 crawl_interval_s: 7200180 priority: 1181 - key: x.ai182 name: xAI — docs183 domain: x.ai184 organization: xai185 tier: 1186 kind: docs187 category: lab188 base_url: https://docs.x.ai189 rate_limit_per_min: 12190 crawl_interval_s: 7200191 priority: 1192 # ---------------------------------------------------------------- model & code ecosystems193 - key: huggingface.co194 name: Hugging Face Hub (public pages, model cards, papers)195 domain: huggingface.co196 organization: huggingface197 tier: 2198 kind: repository199 category: hub200 base_url: https://huggingface.co201 rate_limit_per_min: 30202 crawl_interval_s: 3600203 priority: 0204 notes: Model cards are tier-1 statements by their authors; hub metadata (downloads, likes) is community signal.205 - key: github.com206 name: GitHub public repositories (HTML, releases.atom, raw files)207 domain: github.com208 organization: github209 tier: 2210 kind: repository211 category: code212 base_url: https://github.com213 rate_limit_per_min: 20214 crawl_interval_s: 3600215 priority: 0216 - key: pypi.org217 name: PyPI public package metadata218 domain: pypi.org219 organization: python-software-foundation220 tier: 2221 kind: registry222 category: code223 base_url: https://pypi.org224 rate_limit_per_min: 30225 crawl_interval_s: 21600226 priority: 1227 # ---------------------------------------------------------------- research228 - key: arxiv.org229 name: arXiv (Atom API + RSS)230 domain: arxiv.org231 organization: arxiv232 tier: 1233 kind: feed234 category: research235 base_url: https://export.arxiv.org236 rate_limit_per_min: 4237 crawl_interval_s: 21600238 priority: 0239 robots_policy: documented-exception240 notes: export.arxiv.org/robots.txt disallows generic crawlers but the arXiv API Terms of Use designate the host for programmatic access (≤ 1 request / 3 s); we use 4/min.241 - key: openreview.net242 name: OpenReview243 domain: openreview.net244 organization: openreview245 tier: 1246 kind: registry247 category: research248 base_url: https://api2.openreview.net249 rate_limit_per_min: 10250 crawl_interval_s: 86400251 priority: 1252 # ---------------------------------------------------------------- providers (pricing pages, model lists)253 - key: openrouter.ai254 name: OpenRouter public model & pricing listing255 domain: openrouter.ai256 organization: openrouter257 tier: 2258 kind: registry259 category: provider260 base_url: https://openrouter.ai261 rate_limit_per_min: 10262 crawl_interval_s: 3600263 priority: 0264 notes: Aggregated secondary pricing across providers; tier 2 by design — official provider pages win conflicts.265 - key: groq.com266 name: Groq — docs & pricing267 domain: groq.com268 organization: groq269 tier: 1270 kind: docs271 category: provider272 base_url: https://console.groq.com273 rate_limit_per_min: 12274 crawl_interval_s: 7200275 priority: 1276 - key: together.ai277 name: Together AI — pricing278 domain: together.ai279 organization: together-ai280 tier: 1281 kind: website282 category: provider283 base_url: https://www.together.ai284 rate_limit_per_min: 12285 crawl_interval_s: 7200286 priority: 1287 - key: fireworks.ai288 name: Fireworks AI — pricing289 domain: fireworks.ai290 organization: fireworks-ai291 tier: 1292 kind: website293 category: provider294 base_url: https://fireworks.ai295 rate_limit_per_min: 12296 crawl_interval_s: 7200297 priority: 1298 # ---------------------------------------------------------------- benchmarks / leaderboards299 - key: aider.chat300 name: Aider polyglot leaderboard (data file on GitHub)301 domain: aider.chat302 organization: aider303 tier: 2304 kind: leaderboard305 category: benchmark306 base_url: https://aider.chat/docs/leaderboards/307 rate_limit_per_min: 10308 crawl_interval_s: 21600309 priority: 1310 - key: swebench.com311 name: SWE-bench leaderboards312 domain: swebench.com313 organization: swe-bench314 tier: 2315 kind: leaderboard316 category: benchmark317 base_url: https://www.swebench.com318 rate_limit_per_min: 10319 crawl_interval_s: 21600320 priority: 1321 - key: livebench.ai322 name: LiveBench323 domain: livebench.ai324 organization: livebench325 tier: 2326 kind: leaderboard327 category: benchmark328 base_url: https://livebench.ai329 rate_limit_per_min: 10330 crawl_interval_s: 43200331 priority: 2332 - key: artificialanalysis.ai333 name: Artificial Analysis334 domain: artificialanalysis.ai335 organization: artificial-analysis336 tier: 2337 kind: leaderboard338 category: benchmark339 base_url: https://artificialanalysis.ai340 rate_limit_per_min: 6341 crawl_interval_s: 43200342 priority: 2343 # ---------------------------------------------------------------- hardware manufacturers344 - key: apple.com345 name: Apple — Mac tech specs346 domain: apple.com347 organization: apple348 tier: 1349 kind: website350 category: hardware351 base_url: https://www.apple.com352 rate_limit_per_min: 10353 crawl_interval_s: 86400354 priority: 1355 - key: nvidia.com356 name: NVIDIA — data center product pages357 domain: nvidia.com358 organization: nvidia359 tier: 1360 kind: website361 category: hardware362 base_url: https://www.nvidia.com363 rate_limit_per_min: 10364 crawl_interval_s: 86400365 priority: 1366 - key: amd.com367 name: AMD — Instinct accelerators368 domain: amd.com369 organization: amd370 tier: 1371 kind: website372 category: hardware373 base_url: https://www.amd.com374 rate_limit_per_min: 6375 crawl_interval_s: 86400376 priority: 2377 # ---------------------------------------------------------------- curated registries (AI Atlas itself)378 - key: ai-atlas.registry379 name: AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)380 domain: ai-atlas.co381 organization: ai-atlas382 tier: 2383 kind: registry384 category: registry385 base_url: https://www.ai-atlas.co/sources386 rate_limit_per_min: 1000387 crawl_interval_s: 86400388 priority: 1389