Connectors + registry on the canonical ontology (Stream B): artifacts, effort variants as configurations, benchmark families, identity helper
Hugging Face: repositories are models or artifacts (quantization / conversion / packaging) with `canonical` → base model (base_model tag or
analysed name, family org), `family` hints, canonical licence (+license_raw), `access: gated|open` instead of openness=restricted,
weights_available + derived openness, modalities from pipeline tags, parameter_count kept on artifacts only; converters `publish`, never `develop`.
Leaderboards: Artificial Analysis / LiveBench / aider / SWE-bench fold effort variants ("(xhigh)", "-thinking-64k-high-effort", non-reasoning)
into the base model with the setting in config; LiveBench categories become `livebench-<category>` benchmarks; aider well-formed rate →
`aider-polyglot-well-formed`; AA GPQA → gpqa-diamond, τ² → Telecom; trust_level / variant / run_group on every result; bookkeeping out of config;
free-text tags are aliases, identifiers only for trusted pinned vendor ids. New shared `connectors/_identity.py` (provider prefixes, first-party
patterns, effort stripping with size-tier guard, family hints).
Registry: benchmarks gain family / variant / version / family_head / canonical metric / bounds / higher_is_better / harness / comparability_note +
14 entries (gpqa-diamond, aime-2024, swe-bench-pro, arc-agi-2, terminal-bench-2, mmteb, livebench ×7, aider-polyglot-well-formed); seed writes them
and `variant_of` relations, org_kind from `kind`. Organisations: kind + country fields, upstage / stepfun / kwaipilot / kuaishou / thinking-machines /
lg-ai-research / xiaomi / inclusionai entries with hub aliases. Repositories: canonical kinds, agents (agent_kind) and tools get their own entity types.
OpenRouter: prices booked on the OpenRouter provider only (features.upstream_provider), `created` → openrouter_listed_at (no release_date),
canonical modalities, identity helper. Provider pricing: fireworks_model_id (+ legacy scheme), org from the id, family hints.
Labs: no hard-coded Anthropic modalities/tool constants (table-driven), Google pdf → document + google_model_id, canonical licence/status/
capabilities/api_aliases helpers in labs/_common.py, family_model hints everywhere. arXiv/OpenReview: researchers only with an identifier
(ORCID / OpenReview profile). GitHub/PyPI: canonical kind (+kind_raw), licence keys. docs/CONNECTORS.md rewritten on the canonical vocabulary.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
33 changed files +2,331 −442
modified
docs/CONNECTORS.md
+106 −21
@@ -32,8 +32,7 @@ class ExampleConnector(BaseConnector): | ||
| 32 | 32 | facts.entities.append(org) |
| 33 | 33 | if target.key == "models" and parsed.html: |
| 34 | 34 | for row in parsed.html.tables[0]["rows"]: ... |
| 35 | − model = facts.entity("model", name, identifiers={"example_model_id": api_id}, organization=org) | |
| 36 | − facts.relate(org, "develops", model) | |
| 35 | + model = model_ref(facts, name, org, api_id=api_id, provider_key="example") # from connectors/labs/_common.py: identity + family hint | |
| 37 | 36 | facts.claim(model, "context_length", 128000, unit="tokens") |
| 38 | 37 | facts.price(model=model, provider=provider_ref("example"), input_per_mtok=1.0, output_per_mtok=4.0) |
| 39 | 38 | facts.follow(detail_url, doc_type="model_page", entity=model, needs_llm=True, meta={"llm_task": "model_passport"}) |
@@ -56,7 +55,9 @@ CONNECTORS = [ExampleConnector] | ||
| 56 | 55 | |
| 57 | 56 | Helpers: `aiatlas.sdk.extract.numbers` (`parse_param_count("70B")`, `parse_active_params("235B-A22B")`, `parse_context_length("128K")`, |
| 58 | 57 | `parse_money_per_mtok("$3 / 1M tokens")`, `parse_percent`), `aiatlas.sdk.extract.dates` (`parse_datetime`, `parse_date` with precision), |
| 59 | −`connectors/labs/_common.py` (`announcement_events`, `transpose_feature_table`, `kv_tables`, `clean_cell`, `money`, `tokens`, `month_year`, `parse_retirement`, `model_ref`). | |
| 58 | +`connectors/labs/_common.py` (`announcement_events`, `transpose_feature_table`, `kv_tables`, `clean_cell`, `money`, `tokens`, `month_year`, `parse_retirement`, | |
| 59 | +`model_ref`, `claim_license`, `claim_status`, `claim_modalities`, `claim_api_aliases`, `normalize_capabilities`), | |
| 60 | +`connectors/_identity.py` (see below). | |
| 60 | 61 | |
| 61 | 62 | ## Rules |
| 62 | 63 | |
@@ -64,34 +65,118 @@ Helpers: `aiatlas.sdk.extract.numbers` (`parse_param_count("70B")`, `parse_activ | ||
| 64 | 65 | `pypi.org/pypi/<pkg>/json`, arXiv Atom) are fine — they are public documents, not commercial APIs. Never require an API key. |
| 65 | 66 | Set `Target(escalate=True)` only for pages known to block bots *and* only if a key is configured; otherwise let the document be `blocked`. |
| 66 | 67 | 2. **Never fabricate.** Only emit a claim when the page states it. Unknown → no claim. Don't infer parameter counts from names unless the |
| 67 | − name literally contains them (`Qwen3-235B-A22B` → 235e9 / 22e9 is fine; "Large" is not). | |
| 68 | + name literally contains them (`Qwen3-235B-A22B` → 235e9 / 22e9 is fine; "Large" is not). No hard-coded "all models of this lab do X" constants: | |
| 69 | + read the comparison table, else omit. | |
| 68 | 70 | 3. **Identifiers make resolution deterministic.** Give models the provider's API id (`{provider}_model_id`), HF repo (`hf_repo`), arXiv id |
| 69 | 71 | (`arxiv`), GitHub repo (`github_repo`), PyPI name (`pypi`). Organizations always come from `org_ref(<registry key>)` (add missing orgs to |
| 70 | − `registry/organizations.yaml` with a `source_url`). | |
| 71 | −4. **Properties are shared vocabulary** (see below). Add new ones sparingly; prefix metrics with `metric.` (downloads, likes, stars) so they | |
| 72 | − never generate events. | |
| 72 | + `registry/organizations.yaml` with a `source_url`) — **never guess a developer from a display name**; the `_identity` helper only maps | |
| 73 | + first-party prefixes and family words (`claude-` → Anthropic, `gemini-` → Google, `openrouter/x-ai/…` → xAI). | |
| 74 | +4. **Properties are shared vocabulary** (see below) and **values are canonical**: run licences through `ontology.normalize_license`, statuses through | |
| 75 | + `normalize_status`, modalities through `normalize_modalities`, capabilities through `_common.normalize_capabilities`; keep the source label in | |
| 76 | + `<property>_raw` when it differed. Prefix metrics with `metric.` (downloads, likes, stars) so they never generate events. | |
| 73 | 77 | 5. **Events**: NEW_*, PRICE_CHANGED, CONTEXT_CHANGED… are emitted automatically by the writer. Emit `ANNOUNCEMENT`/`RELEASE` events yourself for |
| 74 | 78 | feed items (`announcement_events`) with `effective_at` = publication date and a `dedupe_key` (URL). |
| 75 | 79 | 6. **Follow-ups** (`facts.follow`) let a listing discover detail pages; keep `max_targets` reasonable (`ctx.max_targets`, default 2000). |
| 76 | 80 | 7. **Every connector has a fixture test**: save real responses under `tests/fixtures/<connector>/…` and assert extracted facts |
| 77 | 81 | (see `tests/test_anthropic.py`). Run connectors with `--file key=path` to use fixtures instead of the network. |
| 78 | 82 | 8. Respect `rate_per_min` from `registry/sources.yaml`; arXiv ≤ 4/min; Hugging Face ≤ 30/min; GitHub ≤ 20/min. |
| 83 | +9. **Results carry provenance**: every `ResultObs` sets `trust_level` (`ontology.benchmarks.trust_level(source_key, config)`), `variant` | |
| 84 | + (GPQA Diamond, SWE-bench Verified, a LiveBench category…) and `run_group` (the evaluation run the row belongs to: LiveBench release, | |
| 85 | + aider run date, AA index version). Bookkeeping (cost, wall time, command lines, harness versions) stays out of `config`. | |
| 86 | + | |
| 87 | +## Identity: model families, models, artifacts, configurations | |
| 88 | + | |
| 89 | +The canonical hierarchy is **model_family → model → artifact → deployment** (`aiatlas.ontology.models`). Connectors describe it with hints on | |
| 90 | +`EntityRef`; the writer materialises `entities.family_id`, `canonical_id`, `artifact_kind`, `identity_confidence` and the relations | |
| 91 | +`member_of_family` / `artifact_of`. | |
| 92 | + | |
| 93 | +| hint | meaning | who sets it | | |
| 94 | +|---|---|---| | |
| 95 | +| `family=EntityRef("model_family", "Qwen3.6", organization=org)` | the versioned family the model belongs to (`_identity.family_ref(name, org)` or the lab's own label via `model_ref(family=…)`) | every model ref with a detectable family | | |
| 96 | +| `canonical=EntityRef("model", …)` | for an **artifact**: the model it packages; for an alias entity: the canonical entity | Hugging Face (quantisations, conversions, mirrors) | | |
| 97 | +| `artifact_kind` | `checkpoint` \| `quantization` \| `conversion` \| `packaging` | Hugging Face | | |
| 98 | +| `identity_confidence` | `high` (vendor id / official repo) · `medium` (leaderboard label, analysed base name) · `low` | everyone | | |
| 99 | + | |
| 100 | +**Entity types**: `model` (a release: Qwen3-8B, Claude Opus 5), `model_family` (Qwen3, Claude), `artifact` (`bartowski/Qwen3.8-27B-GGUF`, | |
| 101 | +`zai-org/GLM-5-FP8`, `mlx-community/Kimi-K2.5` — never a model of its own; keeps the full repo id as name, `parameter_count` = the packaged size), | |
| 102 | +`agent` (coding/browser agents from the repositories registry, attribute `agent_kind`), `tool` (applications, MCP servers), `framework` | |
| 103 | +(libraries, SDKs, inference engines… attribute `kind` canonical), `repository` (model-code drops), `researcher` (**only with an identifier**: | |
| 104 | +ORCID or OpenReview profile id — never from a bare author name; authors stay a claim on the paper). | |
| 105 | + | |
| 106 | +**Effort variants are configurations, not models.** `claude-opus-5-xhigh`, "GPT-5.5 (xhigh)", `deepseek-v3-1-reasoning`, | |
| 107 | +`…-thinking-64k-high-effort`, "Mistral Small 4 (Non-reasoning)" all point at the base model; the setting goes into `ResultObs.config` | |
| 108 | +(`reasoning_effort`, `reasoning: on|off|adaptive`, `thinking_budget`) and the evaluator's slug into `config.aa_slug` / `livebench_model_id`. | |
| 109 | +Size tiers and official products are not efforts: `mistral-medium`, `qwen3.7-max`, `sonar-reasoning`, `kimi-k2-thinking`, `grok-4-1-fast` stay models | |
| 110 | +(`_identity.strip_effort` only strips an ambiguous suffix when the stem still carries a version digit). | |
| 111 | + | |
| 112 | +### `connectors/_identity.py` | |
| 113 | + | |
| 114 | +`model_identity(api_id, trusted=False)` analyses an API id from a third-party source (`anthropic/claude-3-7-sonnet-20250219`, | |
| 115 | +`gemini/gemini-2.5-pro-preview-05-06`, `openrouter/x-ai/grok-4`, `gpt-4o-2024-08-06`, `Qwen/Qwen2.5-Coder-32B-Instruct`, | |
| 116 | +`fireworks_ai/accounts/fireworks/models/qwq-32b`) and returns the developer organisation key, resolver-friendly aliases (raw id, id without | |
| 117 | +provider prefix, id without effort suffixes), the effort configuration and — **only when `trusted=True`** (the string really is the vendor's own | |
| 118 | +API id, e.g. the `--model` argument aider passed) — the vendor identifier (`anthropic_model_id`, `gemini_model_id`, `openai_model_id`, | |
| 119 | +`xai_model_id`, `deepseek_model_id`, `mistral_model_id`, `cohere_model_id`, `hf_repo`, `openrouter`, `fireworks_model_id`). Free-text tags | |
| 120 | +(SWE-bench "Model:" tags, LiveBench ids, OpenRouter slugs) never become identifiers: the resolver refuses an alias match when the entity already | |
| 121 | +carries a *different* value for the same scheme, so a wrong id would split an entity instead of linking it. Rolling ids | |
| 122 | +(`deepseek/deepseek-chat`, `codestral-latest`) are aliases only even when trusted — they name whichever snapshot served that day. | |
| 123 | +`model_ref_from_api_id(facts, api_id, name=…, trusted=…)` builds the deduplicated model `EntityRef` (+ family hint) and returns the effort config. | |
| 124 | +`family_ref`, `split_effort_label`, `strip_effort`, `org_key_for_vendor`, `org_ref_in` are the building blocks. | |
| 79 | 125 | |
| 80 | 126 | ## Property vocabulary (entities.attributes) |
| 81 | 127 | |
| 82 | −**model**: `family, version, release_date (ISO, may be YYYY-MM), status (active|preview|deprecated|retired|announced|limited-availability), openness | |
| 83 | −(open-weights|open-source|proprietary|restricted), license, architecture, parameter_count (int), active_parameter_count, is_moe, modalities | |
| 84 | −(list of text|image|audio|video|code|embedding), modalities_input, modalities_output, context_length (tokens), max_output_tokens, knowledge_cutoff | |
| 85 | −(YYYY-MM), training_data_cutoff, languages, tool_calling, structured_output, reasoning, vision, audio, fine_tuning_available, tokenizer, | |
| 86 | −api_model_id, api_alias, official_url, model_card_url, paper_url, repository_url, hf_repo, pipeline_tag, base_model, quantization, quant_format | |
| 87 | −(gguf|mlx|awq|gptq|fp8), file_size_gb, deprecation_date, retirement_date, retirement_tentative, metric.downloads, metric.likes` | |
| 88 | − | |
| 89 | −**company / organization / lab**: `country (ISO-2), headquarters, founded, website, domains, hf_org, github_org, org_kind, legal_name, founders, leadership, employee_count` | |
| 128 | +Canonical enums live in `src/aiatlas/ontology/` — connectors write canonical values and keep the source label in `<property>_raw`. | |
| 129 | + | |
| 130 | +**model**: `family` (label of the `model_family` hint), `version, release_date (ISO, may be YYYY-MM), status` (`ontology.taxonomy.MODEL_STATUSES`: | |
| 131 | +`announced | preview | active | limited-availability | deprecated | retired | archived | unknown`), `openness` (`ontology.openness.OPENNESS_CATEGORIES`: | |
| 132 | +`open-source | open-weights | restricted-weights | proprietary | unknown` — **derived** with `derive_openness` from `weights_available` + the licence, | |
| 133 | +asserted directly only when the source literally states it), `weights_available` (bool), `access` (`gated | open`, Hugging Face gating — access, | |
| 134 | +not a licence property), `gated_mode`, `license` (`ontology.licenses` key: `Apache-2.0`, `MIT`, `Llama-3.1-Community`, `Mistral-Research`, | |
| 135 | +`Gemma-Terms`… — SPDX id when one exists) + `license_raw`, `architecture, parameter_count (int), active_parameter_count, is_moe, modalities / | |
| 136 | +modalities_input / modalities_output` (`ontology.taxonomy.MODALITIES`: `text | image | audio | video | document | code | embedding | 3d | structured | action`, | |
| 137 | +sorted), `context_length (tokens), max_output_tokens, knowledge_cutoff (YYYY-MM), training_data_cutoff, languages, capabilities` (canonical slugs: | |
| 138 | +`function_calling, structured_output, reasoning, vision, audio_input, audio_output, image_generation, video_generation, code_execution, | |
| 139 | +search_grounding, caching, batch, fine_tuning, streaming, live_api, computer_use, file_search, url_context, mcp…` via `_common.normalize_capabilities`) | |
| 140 | ++ `capabilities_raw`, `tool_calling, structured_output, reasoning, vision, audio, fine_tuning_available, tokenizer, api_model_id, api_aliases` (always a | |
| 141 | +list; `api_alias` = the first one, kept for compatibility), `official_url, model_card_url, paper_url, repository_url, hf_repo, pipeline_tag, base_model, | |
| 142 | +quantization, quant_format (gguf|mlx|awq|gptq|fp8|…), artifact_kind, is_quantized, quantized_by, file_size_gb, deprecation_date, retirement_date, | |
| 143 | +retirement_tentative, openrouter_listed_at` (listing date — never `release_date`), `aa_release_date / aa_openness / aa_context_window` (second-hand | |
| 144 | +facts from Artificial Analysis), `metric.downloads, metric.likes` | |
| 145 | + | |
| 146 | +**company / organization / lab**: `country (ISO-2), headquarters, founded, website, domains, hf_org, github_org, org_kind` | |
| 147 | +(`ontology.taxonomy.ORG_KINDS`: `company | lab | university | nonprofit | government | community | consortium | individual`), `legal_name, founders, leadership, employee_count` | |
| 90 | 148 | |
| 91 | 149 | **provider**: `website, pricing_url, docs_url, regions, features` · **paper**: `authors, published_at, updated_at, abstract, arxiv_id, doi, categories, primary_category, pdf_url, code_url, venue` |
| 92 | −· **benchmark**: `category, task, metric, unit, creator, website, paper, known_limitations, methodology` · **hardware**: `kind, architecture, release_date, memory_gb, | |
| 93 | −memory_type, memory_bandwidth_gbs, tdp_watts, runtimes, manufacturer, spec_url, price_usd, compute_fp16_tflops` · **framework / repository**: `repository_url, latest_version, | |
| 94 | −latest_release_at, license, language, description, topics, pypi, metric.stars, metric.forks` · **dataset**: `license, modality, size, publisher, task, hf_repo` | |
| 95 | − | |
| 96 | −Relations: `develops, owns, operates, available_through, evaluated_on, described_by, derived_from, fine_tuned_from, quantized_from, distilled_from, | |
| 97 | −merged_from, superseded_by, runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates, published_by`. | |
| 150 | +· **researcher**: `openreview_profile_url` (+ identifiers `orcid` / `openreview_profile`) | |
| 151 | +· **benchmark** (registry `registry/benchmarks.yaml` + `.d/`, see below): `family, variant, version, family_head, category, task, metric` (canonical: | |
| 152 | +`accuracy | pass@1 | pass^1 | resolved | pass_rate_2 | percent_cases_well_formed | global_average | average score | mean score | index | elo …`), | |
| 153 | +`metric_label, metric_raw, unit, metric_min, metric_max, higher_is_better, harness, comparability_note, creator, website, paper, known_limitations, methodology` | |
| 154 | +· **hardware**: `kind` (`ontology.taxonomy.HARDWARE_KINDS`), `architecture, release_date, memory_gb, memory_type, memory_bandwidth_gbs, tdp_watts, runtimes, manufacturer, spec_url, | |
| 155 | +price_usd, compute_fp16_tflops` · **framework / tool / agent / repository**: `kind` (`ontology.taxonomy.FRAMEWORK_KINDS`: `training-framework | inference-engine | | |
| 156 | +serving-engine | library | runtime | agent-framework | orchestration | evaluation-harness | sdk | tool | application | agent | mcp-server | vector-database | | |
| 157 | +observability | data-tooling`) + `kind_raw`, `agent_kind` (`coding | browser | research`), `repository_url, latest_version, latest_release_at, license (+ license_raw), | |
| 158 | +language, description, topics, pypi, metric.stars, metric.forks` · **dataset**: `license, modality, size, publisher, task, hf_repo` | |
| 159 | + | |
| 160 | +Relations: `develops, published_by` (artifact → its publishing org; converters never `develop`), `owns, operates, available_through, evaluated_on, | |
| 161 | +described_by, derived_from, fine_tuned_from, quantized_from, distilled_from, merged_from, superseded_by, variant_of` (benchmark variant → family head), | |
| 162 | +`runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates`. The writer adds `member_of_family` and | |
| 163 | +`artifact_of` from the hints. | |
| 164 | + | |
| 165 | +## Prices | |
| 166 | + | |
| 167 | +`PriceObs(model, provider, provider_model_id, …)` — one row per model × provider × provider id. Aggregator prices belong to the aggregator: | |
| 168 | +OpenRouter rows are booked on `provider_ref("openrouter")` with `features.upstream_provider` = the routed lab's registry provider key (never on | |
| 169 | +the lab's own provider entity, which would create two "current" prices). Variants (`:free`, `:thinking`, `:nitro`) are separate rows of the same | |
| 170 | +model (`features.variant`). Identifier schemes for provider catalogues: `openrouter`, `groq_model_id`, `together_ai_model_slug`, | |
| 171 | +`fireworks_model_id` (the historical `fireworks-ai_model_id` is still emitted alongside). | |
| 172 | + | |
| 173 | +## Benchmark registry (`registry/benchmarks.yaml` + `registry/benchmarks.d/*.yaml`) | |
| 174 | + | |
| 175 | +Every entry declares `family`, `variant`, optional `version`, `family_head: true` on the representative member, canonical `metric` | |
| 176 | +(`ontology.benchmarks.normalize_metric`) with `metric_min` / `metric_max` / `higher_is_better`, `harness` and `comparability_note`, plus the | |
| 177 | +aliases evaluators use ("GPQA Diamond", "HLE", "SWE-Bench Verified", "τ²-Bench Telecom"…). `aia seed` writes them as claims and a `variant_of` | |
| 178 | +relation from each member to its family head. One benchmark entity per measured thing: LiveBench categories are `livebench-<category>` entities | |
| 179 | +(family `livebench`), aider's well-formed rate is `aider-polyglot-well-formed`, Artificial Analysis' GPQA results land on `gpqa-diamond`. | |
| 180 | +Connectors address benchmarks with `benchmark_ref(key)` (identifier `registry_benchmark`, `slug_hint` = key). Slugs are only hints: when an | |
| 181 | +organisation already owns the slug (`livebench` the org), the resolver appends a collision suffix (`livebench-2` in production) — identity is | |
| 182 | +the `registry_benchmark` identifier, and existing production slugs never change. | |
modified
registry/benchmarks.d/ecosystem.yaml
+110 −10
@@ -1,81 +1,181 @@ | ||
| 1 | −# Benchmarks reported by the leaderboard connectors (SWE-bench boards, Artificial Analysis component evaluations). Fragment of registry/benchmarks.yaml. | |
| 1 | +# Benchmarks reported by the leaderboard connectors (SWE-bench boards, Artificial Analysis component evaluations). Fragment of registry/benchmarks.yaml | |
| 2 | +# (same canonical fields: family / variant / version / family_head / metric / metric_min / metric_max / higher_is_better / harness / comparability_note). | |
| 2 | 3 | benchmarks: |
| 3 | − - key: swe-bench-lite | |
| 4 | − name: SWE-bench Lite | |
| 4 | + - key: swe-bench-full | |
| 5 | + name: SWE-bench (full test split) | |
| 6 | + aliases: [SWE-bench Test, SWE-bench Full, SWE-bench (full), swebench full] | |
| 7 | + family: swe-bench | |
| 8 | + variant: full | |
| 9 | + family_head: true | |
| 5 | 10 | category: coding |
| 6 | − task: resolve real GitHub issues (300-instance subset) | |
| 11 | + task: resolve real GitHub issues (2,294 instances) | |
| 7 | 12 | metric: resolved |
| 13 | + metric_label: "% resolved" | |
| 8 | 14 | unit: "%" |
| 15 | + metric_min: 0 | |
| 16 | + metric_max: 100 | |
| 17 | + higher_is_better: true | |
| 18 | + harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness | |
| 19 | + comparability_note: Scaffold-dependent; compare within one scaffold and attempt regime only. | |
| 9 | 20 | website: https://www.swebench.com |
| 10 | 21 | paper: https://arxiv.org/abs/2310.06770 |
| 11 | − known_limitations: Scaffold/agent dependent; results are not comparable across harnesses. | |
| 12 | 22 | source_url: https://www.swebench.com/ |
| 13 | − - key: swe-bench-full | |
| 14 | − name: SWE-bench (full test split) | |
| 15 | − aliases: [SWE-bench Test, SWE-bench Full] | |
| 23 | + - key: swe-bench-lite | |
| 24 | + name: SWE-bench Lite | |
| 25 | + aliases: [SWE-Bench Lite, swebench lite] | |
| 26 | + family: swe-bench | |
| 27 | + variant: Lite | |
| 16 | 28 | category: coding |
| 17 | − task: resolve real GitHub issues (2,294 instances) | |
| 29 | + task: resolve real GitHub issues (300-instance subset) | |
| 18 | 30 | metric: resolved |
| 31 | + metric_label: "% resolved" | |
| 19 | 32 | unit: "%" |
| 33 | + metric_min: 0 | |
| 34 | + metric_max: 100 | |
| 35 | + higher_is_better: true | |
| 36 | + harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness | |
| 37 | + comparability_note: Scaffold-dependent; Lite is an easier subset — never compare with Verified or full scores. | |
| 20 | 38 | website: https://www.swebench.com |
| 21 | 39 | paper: https://arxiv.org/abs/2310.06770 |
| 40 | + known_limitations: Scaffold/agent dependent; results are not comparable across harnesses. | |
| 22 | 41 | source_url: https://www.swebench.com/ |
| 23 | 42 | - key: swe-bench-multimodal |
| 24 | 43 | name: SWE-bench Multimodal |
| 44 | + aliases: [SWE-Bench Multimodal, SWE-bench MM] | |
| 45 | + family: swe-bench | |
| 46 | + variant: Multimodal | |
| 25 | 47 | category: coding |
| 26 | 48 | task: resolve visual JavaScript issues (screenshots + code) |
| 27 | 49 | metric: resolved |
| 50 | + metric_label: "% resolved" | |
| 28 | 51 | unit: "%" |
| 52 | + metric_min: 0 | |
| 53 | + metric_max: 100 | |
| 54 | + higher_is_better: true | |
| 55 | + harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness | |
| 56 | + comparability_note: Requires image input; scaffold-dependent. | |
| 29 | 57 | website: https://www.swebench.com/multimodal |
| 30 | 58 | source_url: https://www.swebench.com/ |
| 31 | 59 | - key: swe-bench-multilingual |
| 32 | 60 | name: SWE-bench Multilingual |
| 61 | + aliases: [SWE-Bench Multilingual] | |
| 62 | + family: swe-bench | |
| 63 | + variant: Multilingual | |
| 33 | 64 | category: coding |
| 34 | 65 | task: resolve GitHub issues across 9 programming languages |
| 35 | 66 | metric: resolved |
| 67 | + metric_label: "% resolved" | |
| 36 | 68 | unit: "%" |
| 69 | + metric_min: 0 | |
| 70 | + metric_max: 100 | |
| 71 | + higher_is_better: true | |
| 72 | + harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness | |
| 73 | + comparability_note: Scaffold-dependent; different instance set from Verified. | |
| 37 | 74 | website: https://www.swebench.com/multilingual |
| 38 | 75 | source_url: https://www.swebench.com/ |
| 76 | + - key: swe-bench-pro | |
| 77 | + name: SWE-Bench Pro | |
| 78 | + aliases: [SWE-bench Pro, SWE Bench Pro, swebench pro] | |
| 79 | + family: swe-bench | |
| 80 | + variant: Pro | |
| 81 | + category: coding | |
| 82 | + task: long-horizon, enterprise-grade software engineering tasks (public set) curated by Scale AI | |
| 83 | + metric: resolved | |
| 84 | + metric_label: "% resolved" | |
| 85 | + unit: "%" | |
| 86 | + metric_min: 0 | |
| 87 | + metric_max: 100 | |
| 88 | + higher_is_better: true | |
| 89 | + harness: Scale AI evaluation with a fixed scaffold per leaderboard column | |
| 90 | + comparability_note: Public vs commercial (held-out) sets are different leaderboards; not comparable with SWE-bench Verified. | |
| 91 | + website: https://scale.com/leaderboard/swe_bench_pro_public | |
| 92 | + source_url: https://scale.com/leaderboard/swe_bench_pro_public | |
| 39 | 93 | - key: mmmu-pro |
| 40 | 94 | name: MMMU-Pro |
| 95 | + aliases: [MMMU Pro] | |
| 96 | + family: mmmu | |
| 97 | + variant: Pro | |
| 41 | 98 | category: multimodal |
| 42 | 99 | task: robust multimodal understanding (10-option, vision-only variants) |
| 43 | 100 | metric: accuracy |
| 101 | + metric_label: accuracy | |
| 44 | 102 | unit: "%" |
| 103 | + metric_min: 0 | |
| 104 | + metric_max: 100 | |
| 105 | + higher_is_better: true | |
| 106 | + harness: Artificial Analysis runs its own evaluation; labs self-report | |
| 107 | + comparability_note: Standard (10-option) vs vision-only settings are different numbers. | |
| 45 | 108 | paper: https://arxiv.org/abs/2409.02813 |
| 46 | 109 | source_url: https://arxiv.org/abs/2409.02813 |
| 47 | 110 | - key: livecodebench |
| 48 | 111 | name: LiveCodeBench |
| 112 | + aliases: [LCB, Live Code Bench] | |
| 113 | + family: livecodebench | |
| 114 | + variant: rolling | |
| 115 | + family_head: true | |
| 49 | 116 | category: coding |
| 50 | 117 | task: contamination-free competitive programming problems |
| 51 | 118 | metric: pass@1 |
| 119 | + metric_label: pass@1 | |
| 52 | 120 | unit: "%" |
| 121 | + metric_min: 0 | |
| 122 | + metric_max: 100 | |
| 123 | + higher_is_better: true | |
| 124 | + harness: Artificial Analysis runs its own evaluation; labs self-report on a chosen date window | |
| 125 | + comparability_note: The problem window (release dates) differs between reporters — scores from different windows are not comparable. | |
| 53 | 126 | website: https://livecodebench.github.io |
| 54 | 127 | paper: https://arxiv.org/abs/2403.07974 |
| 55 | 128 | source_url: https://arxiv.org/abs/2403.07974 |
| 56 | 129 | - key: scicode |
| 57 | 130 | name: SciCode |
| 131 | + aliases: [SciCode benchmark] | |
| 132 | + family: scicode | |
| 133 | + variant: main | |
| 134 | + family_head: true | |
| 58 | 135 | category: coding |
| 59 | 136 | task: research-level scientific coding problems |
| 60 | 137 | metric: accuracy |
| 138 | + metric_label: accuracy | |
| 61 | 139 | unit: "%" |
| 140 | + metric_min: 0 | |
| 141 | + metric_max: 100 | |
| 142 | + higher_is_better: true | |
| 143 | + harness: Artificial Analysis runs its own evaluation | |
| 144 | + comparability_note: Sub-problem vs main-problem accuracy are different numbers; with/without background prompts. | |
| 62 | 145 | website: https://scicode-bench.github.io |
| 63 | 146 | paper: https://arxiv.org/abs/2407.13168 |
| 64 | 147 | source_url: https://arxiv.org/abs/2407.13168 |
| 65 | 148 | - key: ifbench |
| 66 | 149 | name: IFBench |
| 150 | + aliases: [IF-Bench] | |
| 151 | + family: ifeval | |
| 152 | + variant: IFBench | |
| 67 | 153 | category: instruction-following |
| 68 | 154 | task: precise instruction following with novel constraints |
| 69 | 155 | metric: accuracy |
| 156 | + metric_label: accuracy | |
| 70 | 157 | unit: "%" |
| 158 | + metric_min: 0 | |
| 159 | + metric_max: 100 | |
| 160 | + higher_is_better: true | |
| 161 | + harness: Artificial Analysis runs its own evaluation | |
| 162 | + comparability_note: Different constraint set from IFEval; not comparable with IFEval scores. | |
| 71 | 163 | paper: https://arxiv.org/abs/2507.02833 |
| 72 | 164 | source_url: https://arxiv.org/abs/2507.02833 |
| 73 | 165 | - key: tau2-bench |
| 74 | 166 | name: τ²-bench |
| 75 | − aliases: [tau2-bench, TAU2-bench, tau-squared bench] | |
| 167 | + aliases: [tau2-bench, TAU2-bench, tau-squared bench, τ²-Bench, τ2-bench, τ²-Bench Telecom, tau2 bench telecom] | |
| 168 | + family: tau-bench | |
| 169 | + variant: τ² | |
| 76 | 170 | category: agentic |
| 77 | 171 | task: dual-control tool-agent-user interaction (telecom, retail, airline) |
| 78 | 172 | metric: pass^1 |
| 173 | + metric_label: pass^1 | |
| 79 | 174 | unit: "%" |
| 175 | + metric_min: 0 | |
| 176 | + metric_max: 100 | |
| 177 | + higher_is_better: true | |
| 178 | + harness: simulated user (LLM) + dual-control tool environment; Artificial Analysis reports the Telecom domain | |
| 179 | + comparability_note: Domain (config.variant = Telecom / Retail / Airline) and the user-simulator model must match. | |
| 80 | 180 | paper: https://arxiv.org/abs/2506.07982 |
| 81 | 181 | source_url: https://arxiv.org/abs/2506.07982 |
modified
registry/benchmarks.yaml
+423 −20
@@ -1,12 +1,29 @@ | ||
| 1 | 1 | # Benchmark definitions (entities of type `benchmark`). Results come from connectors; definitions are curated with sources. |
| 2 | +# | |
| 3 | +# Canonical fields (see aiatlas.ontology.benchmarks): | |
| 4 | +# family benchmark family key (swe-bench, gpqa, livebench…); `variant` names the member, `version` the dataset revision when one exists | |
| 5 | +# family_head true on the representative entry of a family — every other member gets a `variant_of` relation to it (aia seed) | |
| 6 | +# metric canonical metric key from ontology.benchmarks.METRICS (accuracy, pass@1, resolved, elo, index…); `metric_label` = human label | |
| 7 | +# higher_is_better score direction · metric_min / metric_max = bounds of the reported metric (null = unbounded) | |
| 8 | +# harness who runs the evaluation / with what scaffold; comparability_note = what makes two scores on this board (not) comparable | |
| 9 | +# aliases labels used by Artificial Analysis, LiveBench, model cards, papers… (the API resolves benchmarks by alias) | |
| 2 | 10 | benchmarks: |
| 3 | 11 | - key: mmlu |
| 4 | 12 | name: MMLU |
| 5 | − aliases: [Massive Multitask Language Understanding] | |
| 13 | + aliases: [Massive Multitask Language Understanding, MMLU 5-shot, MMLU (5-shot)] | |
| 14 | + family: mmlu | |
| 15 | + variant: original | |
| 16 | + family_head: true | |
| 6 | 17 | category: knowledge |
| 7 | 18 | task: 57-subject multiple-choice questions |
| 8 | 19 | metric: accuracy |
| 20 | + metric_label: accuracy | |
| 9 | 21 | unit: "%" |
| 22 | + metric_min: 0 | |
| 23 | + metric_max: 100 | |
| 24 | + higher_is_better: true | |
| 25 | + harness: self-reported by labs (shot count and prompt format vary) | |
| 26 | + comparability_note: Widespread training-data contamination; 0-shot vs 5-shot and chain-of-thought prompting change scores by several points. | |
| 10 | 27 | creator: Hendrycks et al. |
| 11 | 28 | website: https://github.com/hendrycks/test |
| 12 | 29 | paper: https://arxiv.org/abs/2009.03300 |
@@ -14,160 +31,546 @@ benchmarks: | ||
| 14 | 31 | source_url: https://arxiv.org/abs/2009.03300 |
| 15 | 32 | - key: mmlu-pro |
| 16 | 33 | name: MMLU-Pro |
| 34 | + aliases: [MMLU Pro, MMLU-Pro (0-shot CoT)] | |
| 35 | + family: mmlu | |
| 36 | + variant: Pro | |
| 17 | 37 | category: knowledge |
| 18 | 38 | task: harder, 10-option MMLU variant |
| 19 | 39 | metric: accuracy |
| 40 | + metric_label: accuracy | |
| 20 | 41 | unit: "%" |
| 42 | + metric_min: 0 | |
| 43 | + metric_max: 100 | |
| 44 | + higher_is_better: true | |
| 45 | + harness: self-reported by labs or independent evaluators | |
| 46 | + comparability_note: 10-option questions; chain-of-thought is the standard regime, direct-answer scores are lower and not comparable. | |
| 21 | 47 | website: https://github.com/TIGER-AI-Lab/MMLU-Pro |
| 22 | 48 | paper: https://arxiv.org/abs/2406.01574 |
| 23 | 49 | source_url: https://arxiv.org/abs/2406.01574 |
| 24 | 50 | - key: gpqa |
| 25 | 51 | name: GPQA |
| 26 | − aliases: [GPQA Diamond] | |
| 52 | + aliases: [GPQA main, Graduate-Level Google-Proof Q&A] | |
| 53 | + family: gpqa | |
| 54 | + variant: main | |
| 55 | + family_head: true | |
| 27 | 56 | category: reasoning |
| 28 | − task: graduate-level science questions | |
| 57 | + task: graduate-level science questions (main set, 448 questions) | |
| 29 | 58 | metric: accuracy |
| 59 | + metric_label: accuracy | |
| 30 | 60 | unit: "%" |
| 61 | + metric_min: 0 | |
| 62 | + metric_max: 100 | |
| 63 | + higher_is_better: true | |
| 64 | + harness: self-reported by labs or independent evaluators | |
| 65 | + comparability_note: Main and Diamond subsets are different question sets; reasoning effort and sampling (pass@1 vs majority vote) change scores. | |
| 66 | + website: https://github.com/idavidrein/gpqa | |
| 67 | + paper: https://arxiv.org/abs/2311.12022 | |
| 68 | + source_url: https://arxiv.org/abs/2311.12022 | |
| 69 | + - key: gpqa-diamond | |
| 70 | + name: GPQA Diamond | |
| 71 | + aliases: [GPQA-Diamond, gpqa_diamond, GPQA (Diamond), GPQA Diamond (0-shot)] | |
| 72 | + family: gpqa | |
| 73 | + variant: Diamond | |
| 74 | + category: reasoning | |
| 75 | + task: graduate-level science questions — the 198-question Diamond subset (expert-validated, non-expert-failed) | |
| 76 | + metric: accuracy | |
| 77 | + metric_label: accuracy | |
| 78 | + unit: "%" | |
| 79 | + metric_min: 0 | |
| 80 | + metric_max: 100 | |
| 81 | + higher_is_better: true | |
| 82 | + harness: Artificial Analysis runs its own evaluation; labs self-report with varying sampling | |
| 83 | + comparability_note: 198 questions → ±3 pt noise; reasoning effort, thinking budget and repeated sampling strongly affect scores. | |
| 31 | 84 | website: https://github.com/idavidrein/gpqa |
| 32 | 85 | paper: https://arxiv.org/abs/2311.12022 |
| 33 | 86 | source_url: https://arxiv.org/abs/2311.12022 |
| 34 | 87 | - key: humaneval |
| 35 | 88 | name: HumanEval |
| 89 | + aliases: [HumanEval pass@1, OpenAI HumanEval] | |
| 90 | + family: humaneval | |
| 91 | + variant: original | |
| 92 | + family_head: true | |
| 36 | 93 | category: coding |
| 37 | 94 | task: Python function synthesis from docstrings |
| 38 | 95 | metric: pass@1 |
| 96 | + metric_label: pass@1 | |
| 39 | 97 | unit: "%" |
| 98 | + metric_min: 0 | |
| 99 | + metric_max: 100 | |
| 100 | + higher_is_better: true | |
| 101 | + harness: self-reported by labs | |
| 102 | + comparability_note: Saturated; pass@1 vs pass@k and prompt wrapping differ across reporters. | |
| 40 | 103 | website: https://github.com/openai/human-eval |
| 41 | 104 | paper: https://arxiv.org/abs/2107.03374 |
| 42 | 105 | known_limitations: Saturated; small (164 problems). |
| 43 | 106 | source_url: https://arxiv.org/abs/2107.03374 |
| 44 | 107 | - key: swe-bench-verified |
| 45 | 108 | name: SWE-bench Verified |
| 46 | − aliases: [SWE-bench] | |
| 109 | + aliases: [SWE-bench, SWE-Bench Verified, SWE-bench-Verified, swebench verified, SWE Bench Verified] | |
| 110 | + family: swe-bench | |
| 111 | + variant: Verified | |
| 47 | 112 | category: coding |
| 48 | 113 | task: resolve real GitHub issues (500 human-validated instances) |
| 49 | 114 | metric: resolved |
| 115 | + metric_label: "% resolved" | |
| 50 | 116 | unit: "%" |
| 117 | + metric_min: 0 | |
| 118 | + metric_max: 100 | |
| 119 | + higher_is_better: true | |
| 120 | + harness: submitter's agent scaffold (config.system); evaluation by the SWE-bench harness, submissions optionally checked by the SWE-bench team | |
| 121 | + comparability_note: Scores depend on the agent scaffold as much as on the model; compare only within one scaffold (config.system) and one attempt regime. | |
| 51 | 122 | website: https://www.swebench.com |
| 52 | 123 | paper: https://arxiv.org/abs/2310.06770 |
| 53 | 124 | known_limitations: Scaffold/agent dependent; results are not comparable across harnesses. |
| 54 | 125 | source_url: https://www.swebench.com/ |
| 55 | 126 | - key: aider-polyglot |
| 56 | 127 | name: Aider polyglot |
| 57 | − aliases: [Aider polyglot coding leaderboard] | |
| 128 | + aliases: [Aider polyglot coding leaderboard, Aider Polyglot, aider polyglot benchmark, Aider] | |
| 129 | + family: aider-polyglot | |
| 130 | + variant: polyglot | |
| 131 | + family_head: true | |
| 58 | 132 | category: coding |
| 59 | 133 | task: 225 Exercism exercises in 6 languages, edit-format aware |
| 60 | − metric: pass rate (2 attempts) | |
| 134 | + metric: pass_rate_2 | |
| 135 | + metric_label: pass rate (2 attempts) | |
| 61 | 136 | unit: "%" |
| 137 | + metric_min: 0 | |
| 138 | + metric_max: 100 | |
| 139 | + higher_is_better: true | |
| 140 | + harness: aider (config.edit_format = whole | diff | diff-fenced | architect), second attempt after test feedback | |
| 141 | + comparability_note: Edit format and aider version change results; the well-formed rate is a separate benchmark entry. | |
| 62 | 142 | website: https://aider.chat/docs/leaderboards/ |
| 63 | 143 | known_limitations: Depends on aider's edit format and prompting; cost column depends on provider pricing. |
| 64 | 144 | source_url: https://aider.chat/docs/leaderboards/ |
| 145 | + - key: aider-polyglot-well-formed | |
| 146 | + name: Aider polyglot — well-formed responses | |
| 147 | + aliases: [Aider well-formed rate, percent_cases_well_formed, Aider polyglot well-formed] | |
| 148 | + family: aider-polyglot | |
| 149 | + variant: well-formed | |
| 150 | + category: coding | |
| 151 | + task: share of aider polyglot cases where every edit was syntactically well formed (parsable edit blocks) | |
| 152 | + metric: percent_cases_well_formed | |
| 153 | + metric_label: "% cases well formed" | |
| 154 | + unit: "%" | |
| 155 | + metric_min: 0 | |
| 156 | + metric_max: 100 | |
| 157 | + higher_is_better: true | |
| 158 | + harness: aider (config.edit_format), same runs as aider-polyglot | |
| 159 | + comparability_note: Measures edit-format compliance, not correctness; only comparable within one edit format. | |
| 160 | + website: https://aider.chat/docs/leaderboards/ | |
| 161 | + source_url: https://aider.chat/docs/leaderboards/ | |
| 65 | 162 | - key: livebench |
| 66 | 163 | name: LiveBench |
| 164 | + aliases: [LiveBench global average, LiveBench Global, LiveBench overall] | |
| 165 | + family: livebench | |
| 166 | + variant: global | |
| 167 | + family_head: true | |
| 67 | 168 | category: general |
| 68 | − task: contamination-limited, monthly refreshed questions across 6 categories | |
| 69 | − metric: average score | |
| 169 | + task: contamination-limited, monthly refreshed questions across 7 categories — overall mean of the category averages | |
| 170 | + metric: global_average | |
| 171 | + metric_label: global average | |
| 70 | 172 | unit: "%" |
| 173 | + metric_min: 0 | |
| 174 | + metric_max: 100 | |
| 175 | + higher_is_better: true | |
| 176 | + harness: LiveBench official runs; config.release = question-set release date | |
| 177 | + comparability_note: Each release is a new question set — compare scores within one release (run group) only; effort variants are configurations. | |
| 71 | 178 | website: https://livebench.ai |
| 72 | 179 | paper: https://arxiv.org/abs/2406.19314 |
| 73 | 180 | source_url: https://livebench.ai/ |
| 181 | + - key: livebench-reasoning | |
| 182 | + name: LiveBench Reasoning | |
| 183 | + aliases: [LiveBench — Reasoning, LiveBench reasoning average] | |
| 184 | + family: livebench | |
| 185 | + variant: Reasoning | |
| 186 | + category: reasoning | |
| 187 | + task: LiveBench reasoning category — mean of its subtasks (web of lies, zebra puzzles, spatial…) | |
| 188 | + metric: average score | |
| 189 | + metric_label: category average | |
| 190 | + unit: "%" | |
| 191 | + metric_min: 0 | |
| 192 | + metric_max: 100 | |
| 193 | + higher_is_better: true | |
| 194 | + harness: LiveBench official runs | |
| 195 | + comparability_note: Compare within one release only; subtask mix changes between releases. | |
| 196 | + website: https://livebench.ai | |
| 197 | + source_url: https://livebench.ai/ | |
| 198 | + - key: livebench-coding | |
| 199 | + name: LiveBench Coding | |
| 200 | + aliases: [LiveBench — Coding, LiveBench coding average] | |
| 201 | + family: livebench | |
| 202 | + variant: Coding | |
| 203 | + category: coding | |
| 204 | + task: LiveBench coding category — mean of its subtasks (LCB generation, code completion) | |
| 205 | + metric: average score | |
| 206 | + metric_label: category average | |
| 207 | + unit: "%" | |
| 208 | + metric_min: 0 | |
| 209 | + metric_max: 100 | |
| 210 | + higher_is_better: true | |
| 211 | + harness: LiveBench official runs | |
| 212 | + comparability_note: Compare within one release only; subtask mix changes between releases. | |
| 213 | + website: https://livebench.ai | |
| 214 | + source_url: https://livebench.ai/ | |
| 215 | + - key: livebench-agentic-coding | |
| 216 | + name: LiveBench Agentic Coding | |
| 217 | + aliases: [LiveBench — Agentic Coding, LiveBench agentic coding average] | |
| 218 | + family: livebench | |
| 219 | + variant: Agentic Coding | |
| 220 | + category: coding | |
| 221 | + task: LiveBench agentic coding category — mean of its subtasks | |
| 222 | + metric: average score | |
| 223 | + metric_label: category average | |
| 224 | + unit: "%" | |
| 225 | + metric_min: 0 | |
| 226 | + metric_max: 100 | |
| 227 | + higher_is_better: true | |
| 228 | + harness: LiveBench official runs | |
| 229 | + comparability_note: Compare within one release only; subtask mix changes between releases. | |
| 230 | + website: https://livebench.ai | |
| 231 | + source_url: https://livebench.ai/ | |
| 232 | + - key: livebench-mathematics | |
| 233 | + name: LiveBench Mathematics | |
| 234 | + aliases: [LiveBench — Mathematics, LiveBench math average, LiveBench Math] | |
| 235 | + family: livebench | |
| 236 | + variant: Mathematics | |
| 237 | + category: math | |
| 238 | + task: LiveBench mathematics category — mean of its subtasks (competition math, AMPS hard, olympiad) | |
| 239 | + metric: average score | |
| 240 | + metric_label: category average | |
| 241 | + unit: "%" | |
| 242 | + metric_min: 0 | |
| 243 | + metric_max: 100 | |
| 244 | + higher_is_better: true | |
| 245 | + harness: LiveBench official runs | |
| 246 | + comparability_note: Compare within one release only; subtask mix changes between releases. | |
| 247 | + website: https://livebench.ai | |
| 248 | + source_url: https://livebench.ai/ | |
| 249 | + - key: livebench-data-analysis | |
| 250 | + name: LiveBench Data Analysis | |
| 251 | + aliases: [LiveBench — Data Analysis, LiveBench data analysis average] | |
| 252 | + family: livebench | |
| 253 | + variant: Data Analysis | |
| 254 | + category: general | |
| 255 | + task: LiveBench data analysis category — mean of its subtasks (table reformatting, join, column type) | |
| 256 | + metric: average score | |
| 257 | + metric_label: category average | |
| 258 | + unit: "%" | |
| 259 | + metric_min: 0 | |
| 260 | + metric_max: 100 | |
| 261 | + higher_is_better: true | |
| 262 | + harness: LiveBench official runs | |
| 263 | + comparability_note: Compare within one release only; subtask mix changes between releases. | |
| 264 | + website: https://livebench.ai | |
| 265 | + source_url: https://livebench.ai/ | |
| 266 | + - key: livebench-language | |
| 267 | + name: LiveBench Language | |
| 268 | + aliases: [LiveBench — Language, LiveBench language average] | |
| 269 | + family: livebench | |
| 270 | + variant: Language | |
| 271 | + category: general | |
| 272 | + task: LiveBench language category — mean of its subtasks (typos, connections, plot unscrambling) | |
| 273 | + metric: average score | |
| 274 | + metric_label: category average | |
| 275 | + unit: "%" | |
| 276 | + metric_min: 0 | |
| 277 | + metric_max: 100 | |
| 278 | + higher_is_better: true | |
| 279 | + harness: LiveBench official runs | |
| 280 | + comparability_note: Compare within one release only; subtask mix changes between releases. | |
| 281 | + website: https://livebench.ai | |
| 282 | + source_url: https://livebench.ai/ | |
| 283 | + - key: livebench-if | |
| 284 | + name: LiveBench Instruction Following | |
| 285 | + aliases: [LiveBench — IF, LiveBench IF, LiveBench instruction following average] | |
| 286 | + family: livebench | |
| 287 | + variant: IF | |
| 288 | + category: instruction-following | |
| 289 | + task: LiveBench instruction-following category — mean of its subtasks (paraphrase, simplify, story generation, summarize) | |
| 290 | + metric: average score | |
| 291 | + metric_label: category average | |
| 292 | + unit: "%" | |
| 293 | + metric_min: 0 | |
| 294 | + metric_max: 100 | |
| 295 | + higher_is_better: true | |
| 296 | + harness: LiveBench official runs | |
| 297 | + comparability_note: Compare within one release only; subtask mix changes between releases. | |
| 298 | + website: https://livebench.ai | |
| 299 | + source_url: https://livebench.ai/ | |
| 74 | 300 | - key: math-500 |
| 75 | 301 | name: MATH-500 |
| 76 | − aliases: [MATH] | |
| 302 | + aliases: [MATH, MATH 500, MATH500] | |
| 303 | + family: math | |
| 304 | + variant: "500" | |
| 305 | + family_head: true | |
| 77 | 306 | category: math |
| 78 | − task: competition mathematics | |
| 307 | + task: competition mathematics (500-problem subset of MATH used since the PRM800K paper) | |
| 79 | 308 | metric: accuracy |
| 309 | + metric_label: accuracy | |
| 80 | 310 | unit: "%" |
| 311 | + metric_min: 0 | |
| 312 | + metric_max: 100 | |
| 313 | + higher_is_better: true | |
| 314 | + harness: self-reported by labs or independent evaluators | |
| 315 | + comparability_note: Saturated at the frontier; sampling regime (pass@1 vs majority vote) must match. | |
| 81 | 316 | paper: https://arxiv.org/abs/2103.03874 |
| 82 | 317 | source_url: https://arxiv.org/abs/2103.03874 |
| 318 | + - key: aime-2024 | |
| 319 | + name: AIME 2024 | |
| 320 | + aliases: [AIME24, AIME 24, AIME I/II 2024] | |
| 321 | + family: aime | |
| 322 | + variant: "2024" | |
| 323 | + version: "2024" | |
| 324 | + category: math | |
| 325 | + task: American Invitational Mathematics Examination 2024 problems (30 problems, I + II) | |
| 326 | + metric: accuracy | |
| 327 | + metric_label: accuracy | |
| 328 | + unit: "%" | |
| 329 | + metric_min: 0 | |
| 330 | + metric_max: 100 | |
| 331 | + higher_is_better: true | |
| 332 | + harness: self-reported by labs or independent evaluators | |
| 333 | + comparability_note: 30 problems → high variance; pass@1 averaged over samples vs majority voting (cons@64) are not comparable; 2024 problems appear in training data of later models. | |
| 334 | + source_url: https://maa.org/maa-invitational-competitions/ | |
| 83 | 335 | - key: aime-2025 |
| 84 | 336 | name: AIME 2025 |
| 85 | − aliases: [AIME] | |
| 337 | + aliases: [AIME, AIME25, AIME 25, AIME I/II 2025] | |
| 338 | + family: aime | |
| 339 | + variant: "2025" | |
| 340 | + version: "2025" | |
| 341 | + family_head: true | |
| 86 | 342 | category: math |
| 87 | − task: American Invitational Mathematics Examination problems | |
| 343 | + task: American Invitational Mathematics Examination 2025 problems (30 problems, I + II) | |
| 88 | 344 | metric: accuracy |
| 345 | + metric_label: accuracy | |
| 89 | 346 | unit: "%" |
| 347 | + metric_min: 0 | |
| 348 | + metric_max: 100 | |
| 349 | + higher_is_better: true | |
| 350 | + harness: Artificial Analysis runs its own evaluation; labs self-report with varying sampling | |
| 351 | + comparability_note: 30 problems → high variance; pass@1 averaged over samples vs majority voting are not comparable; reasoning effort matters. | |
| 90 | 352 | known_limitations: 30 problems per year; high variance; often reported with majority voting. |
| 91 | 353 | source_url: https://maa.org/maa-invitational-competitions/ |
| 92 | 354 | - key: arc-agi |
| 93 | 355 | name: ARC-AGI |
| 94 | − aliases: [ARC-AGI-1, ARC-AGI-2, Abstraction and Reasoning Corpus] | |
| 356 | + aliases: [ARC-AGI-1, ARC AGI 1, Abstraction and Reasoning Corpus, ARC-AGI v1] | |
| 357 | + family: arc-agi | |
| 358 | + variant: "1" | |
| 359 | + version: "1" | |
| 360 | + family_head: true | |
| 95 | 361 | category: reasoning |
| 96 | − task: novel visual abstraction puzzles | |
| 362 | + task: novel visual abstraction puzzles (ARC-AGI-1 semi-private / public evaluation sets) | |
| 97 | 363 | metric: accuracy |
| 364 | + metric_label: accuracy | |
| 98 | 365 | unit: "%" |
| 366 | + metric_min: 0 | |
| 367 | + metric_max: 100 | |
| 368 | + higher_is_better: true | |
| 369 | + harness: ARC Prize Foundation verified runs (semi-private set) or self-reported public-set runs | |
| 370 | + comparability_note: Public, semi-private and private evaluation sets differ; the ARC Prize reports cost per task alongside the score. | |
| 99 | 371 | website: https://arcprize.org |
| 100 | 372 | source_url: https://arcprize.org/ |
| 373 | + - key: arc-agi-2 | |
| 374 | + name: ARC-AGI-2 | |
| 375 | + aliases: [ARC AGI 2, ARC-AGI v2, ARC-AGI 2] | |
| 376 | + family: arc-agi | |
| 377 | + variant: "2" | |
| 378 | + version: "2" | |
| 379 | + category: reasoning | |
| 380 | + task: harder abstraction puzzles released in 2025 (ARC-AGI-2), designed to resist brute-force program search | |
| 381 | + metric: accuracy | |
| 382 | + metric_label: accuracy | |
| 383 | + unit: "%" | |
| 384 | + metric_min: 0 | |
| 385 | + metric_max: 100 | |
| 386 | + higher_is_better: true | |
| 387 | + harness: ARC Prize Foundation verified runs (semi-private set) or self-reported public-set runs | |
| 388 | + comparability_note: Not comparable with ARC-AGI-1 scores; evaluation set (public / semi-private) and cost budget must match. | |
| 389 | + website: https://arcprize.org/arc-agi/2/ | |
| 390 | + source_url: https://arcprize.org/arc-agi/2/ | |
| 101 | 391 | - key: humanitys-last-exam |
| 102 | 392 | name: Humanity's Last Exam |
| 103 | − aliases: [HLE] | |
| 393 | + aliases: [HLE, Humanity’s Last Exam, HLE (no tools)] | |
| 394 | + family: humanitys-last-exam | |
| 395 | + variant: full | |
| 396 | + family_head: true | |
| 104 | 397 | category: knowledge |
| 105 | 398 | task: expert-written frontier questions |
| 106 | 399 | metric: accuracy |
| 400 | + metric_label: accuracy | |
| 107 | 401 | unit: "%" |
| 402 | + metric_min: 0 | |
| 403 | + metric_max: 100 | |
| 404 | + higher_is_better: true | |
| 405 | + harness: Artificial Analysis and labs; text-only subset vs full multimodal set, with or without tools | |
| 406 | + comparability_note: Tool use (search) and the text-only subset change scores substantially; check config before comparing. | |
| 108 | 407 | website: https://lastexam.ai |
| 109 | 408 | paper: https://arxiv.org/abs/2501.14249 |
| 110 | 409 | source_url: https://lastexam.ai/ |
| 111 | 410 | - key: lmarena-text |
| 112 | 411 | name: LMArena text leaderboard |
| 113 | − aliases: [Chatbot Arena, LMSYS Chatbot Arena, Arena Elo] | |
| 412 | + aliases: [Chatbot Arena, LMSYS Chatbot Arena, Arena Elo, LMArena, Arena Score] | |
| 413 | + family: lmarena | |
| 414 | + variant: text | |
| 415 | + family_head: true | |
| 114 | 416 | category: preference |
| 115 | 417 | task: crowdsourced pairwise human preference |
| 116 | − metric: Elo / Bradley–Terry score | |
| 418 | + metric: elo | |
| 419 | + metric_label: Elo / Bradley–Terry score | |
| 117 | 420 | unit: "" |
| 421 | + metric_min: 0 | |
| 422 | + metric_max: null | |
| 423 | + higher_is_better: true | |
| 424 | + harness: LMArena live human votes (Bradley–Terry model) | |
| 425 | + comparability_note: Relative score that drifts as the model pool changes; only a snapshot of the same date is comparable; style control changes rankings. | |
| 118 | 426 | website: https://lmarena.ai |
| 119 | 427 | known_limitations: Style bias; sampling of prompts by users. |
| 120 | 428 | source_url: https://lmarena.ai/ |
| 121 | 429 | - key: mmmu |
| 122 | 430 | name: MMMU |
| 431 | + aliases: [MMMU (val), Massive Multi-discipline Multimodal Understanding] | |
| 432 | + family: mmmu | |
| 433 | + variant: original | |
| 434 | + family_head: true | |
| 123 | 435 | category: multimodal |
| 124 | 436 | task: college-level multimodal understanding |
| 125 | 437 | metric: accuracy |
| 438 | + metric_label: accuracy | |
| 126 | 439 | unit: "%" |
| 440 | + metric_min: 0 | |
| 441 | + metric_max: 100 | |
| 442 | + higher_is_better: true | |
| 443 | + harness: self-reported by labs (validation split) | |
| 444 | + comparability_note: Validation vs test split and chain-of-thought prompting differ across reporters. | |
| 127 | 445 | website: https://mmmu-benchmark.github.io |
| 128 | 446 | paper: https://arxiv.org/abs/2311.16502 |
| 129 | 447 | source_url: https://arxiv.org/abs/2311.16502 |
| 130 | 448 | - key: tau-bench |
| 131 | 449 | name: τ-bench |
| 132 | − aliases: [tau-bench, TAU-bench] | |
| 450 | + aliases: [tau-bench, TAU-bench, tau bench, τ-Bench] | |
| 451 | + family: tau-bench | |
| 452 | + variant: v1 | |
| 453 | + family_head: true | |
| 133 | 454 | category: agentic |
| 134 | 455 | task: tool-agent-user interaction in retail/airline domains |
| 135 | 456 | metric: pass^1 |
| 457 | + metric_label: pass^1 | |
| 136 | 458 | unit: "%" |
| 459 | + metric_min: 0 | |
| 460 | + metric_max: 100 | |
| 461 | + higher_is_better: true | |
| 462 | + harness: simulated user (LLM) + tool environment; pass^k over k trials | |
| 463 | + comparability_note: Domain (retail / airline), pass^k regime and the user-simulator model must match. | |
| 137 | 464 | paper: https://arxiv.org/abs/2406.12045 |
| 138 | 465 | source_url: https://arxiv.org/abs/2406.12045 |
| 139 | 466 | - key: terminal-bench |
| 140 | 467 | name: Terminal-Bench |
| 468 | + aliases: [TerminalBench, Terminal Bench, Terminal-Bench 1.0] | |
| 469 | + family: terminal-bench | |
| 470 | + variant: "1.0" | |
| 471 | + version: "1.0" | |
| 472 | + family_head: true | |
| 141 | 473 | category: agentic |
| 142 | 474 | task: terminal tasks solved by agents |
| 143 | 475 | metric: accuracy |
| 476 | + metric_label: accuracy | |
| 144 | 477 | unit: "%" |
| 478 | + metric_min: 0 | |
| 479 | + metric_max: 100 | |
| 480 | + higher_is_better: true | |
| 481 | + harness: Terminus agent (official) or the submitter's agent; Artificial Analysis reports its own harness versions (config.variant) | |
| 482 | + comparability_note: Agent scaffold and harness version change results; config.variant (v2.1 / v4.0 / hard) must match. | |
| 483 | + website: https://www.tbench.ai | |
| 484 | + source_url: https://www.tbench.ai/ | |
| 485 | + - key: terminal-bench-2 | |
| 486 | + name: Terminal-Bench 2.0 | |
| 487 | + aliases: [Terminal Bench 2, TerminalBench 2.0, Terminal-Bench 2, TB2] | |
| 488 | + family: terminal-bench | |
| 489 | + variant: "2.0" | |
| 490 | + version: "2.0" | |
| 491 | + category: agentic | |
| 492 | + task: revised, harder set of terminal tasks (Terminal-Bench 2.0) | |
| 493 | + metric: accuracy | |
| 494 | + metric_label: accuracy | |
| 495 | + unit: "%" | |
| 496 | + metric_min: 0 | |
| 497 | + metric_max: 100 | |
| 498 | + higher_is_better: true | |
| 499 | + harness: Terminus 2 agent (official) or the submitter's agent | |
| 500 | + comparability_note: Not comparable with Terminal-Bench 1.0; agent scaffold must match. | |
| 145 | 501 | website: https://www.tbench.ai |
| 146 | 502 | source_url: https://www.tbench.ai/ |
| 147 | 503 | - key: ifeval |
| 148 | 504 | name: IFEval |
| 505 | + aliases: [IF-Eval, Instruction-Following Eval, IFEval strict] | |
| 506 | + family: ifeval | |
| 507 | + variant: original | |
| 508 | + family_head: true | |
| 149 | 509 | category: instruction-following |
| 150 | 510 | task: verifiable instruction following |
| 151 | − metric: prompt-level strict accuracy | |
| 511 | + metric: accuracy | |
| 512 | + metric_label: prompt-level strict accuracy | |
| 152 | 513 | unit: "%" |
| 514 | + metric_min: 0 | |
| 515 | + metric_max: 100 | |
| 516 | + higher_is_better: true | |
| 517 | + harness: self-reported by labs | |
| 518 | + comparability_note: Prompt-level vs instruction-level and strict vs loose accuracy are four different numbers. | |
| 153 | 519 | paper: https://arxiv.org/abs/2311.07911 |
| 154 | 520 | source_url: https://arxiv.org/abs/2311.07911 |
| 155 | 521 | - key: mteb |
| 156 | 522 | name: MTEB |
| 157 | − aliases: [Massive Text Embedding Benchmark] | |
| 523 | + aliases: [Massive Text Embedding Benchmark, MTEB (English), MTEB v1] | |
| 524 | + family: mteb | |
| 525 | + variant: v1 | |
| 526 | + family_head: true | |
| 158 | 527 | category: embeddings |
| 159 | 528 | task: embedding tasks across retrieval, classification, clustering… |
| 160 | 529 | metric: mean score |
| 530 | + metric_label: mean score | |
| 161 | 531 | unit: "" |
| 532 | + metric_min: 0 | |
| 533 | + metric_max: 100 | |
| 534 | + higher_is_better: true | |
| 535 | + harness: MTEB library, official leaderboard | |
| 536 | + comparability_note: Task set (English v1, v2, multilingual) and aggregation (mean of tasks vs mean of task types) must match. | |
| 162 | 537 | website: https://huggingface.co/spaces/mteb/leaderboard |
| 163 | 538 | paper: https://arxiv.org/abs/2210.07316 |
| 164 | 539 | source_url: https://arxiv.org/abs/2210.07316 |
| 540 | + - key: mmteb | |
| 541 | + name: MMTEB | |
| 542 | + aliases: [Massive Multilingual Text Embedding Benchmark, MTEB multilingual, MTEB (Multilingual)] | |
| 543 | + family: mteb | |
| 544 | + variant: MMTEB | |
| 545 | + category: embeddings | |
| 546 | + task: multilingual extension of MTEB (500+ tasks across 250+ languages) | |
| 547 | + metric: mean score | |
| 548 | + metric_label: mean score | |
| 549 | + unit: "" | |
| 550 | + metric_min: 0 | |
| 551 | + metric_max: 100 | |
| 552 | + higher_is_better: true | |
| 553 | + harness: MTEB library, official leaderboard | |
| 554 | + comparability_note: Not comparable with English-only MTEB; the leaderboard's Borda rank and task-type mean are different aggregations. | |
| 555 | + website: https://huggingface.co/spaces/mteb/leaderboard | |
| 556 | + paper: https://arxiv.org/abs/2502.13595 | |
| 557 | + source_url: https://arxiv.org/abs/2502.13595 | |
| 165 | 558 | - key: artificial-analysis-intelligence-index |
| 166 | 559 | name: Artificial Analysis Intelligence Index |
| 560 | + aliases: [AA Intelligence Index, Intelligence Index, Artificial Analysis Index] | |
| 561 | + family: artificial-analysis-intelligence-index | |
| 562 | + variant: index | |
| 563 | + family_head: true | |
| 167 | 564 | category: composite |
| 168 | 565 | task: composite of several evaluations run by Artificial Analysis |
| 169 | 566 | metric: index |
| 567 | + metric_label: index | |
| 170 | 568 | unit: "" |
| 569 | + metric_min: 0 | |
| 570 | + metric_max: 100 | |
| 571 | + higher_is_better: true | |
| 572 | + harness: Artificial Analysis internal runs; config.version = index methodology version | |
| 573 | + comparability_note: Methodology versions change the component set — compare within one version (run group) only; reasoning-effort variants are configurations. | |
| 171 | 574 | website: https://artificialanalysis.ai |
| 172 | 575 | known_limitations: Proprietary composite; methodology versions change. |
| 173 | 576 | source_url: https://artificialanalysis.ai/methodology |
modified
registry/organizations.d/ecosystem.yaml
+29 −1
@@ -1,9 +1,10 @@ | ||
| 1 | 1 | # Ecosystem organizations referenced by the code/hub connectors (GitHub seed list, PyPI). Fragment of registry/organizations.yaml. |
| 2 | −# Only identity fields we can read from the organization's own pages; unknown fields are omitted. | |
| 2 | +# Only identity fields we can read from the organization's own pages; unknown fields are omitted. `kind` = canonical org kind (see organizations.yaml). | |
| 3 | 3 | organizations: |
| 4 | 4 | - key: eleutherai |
| 5 | 5 | name: EleutherAI |
| 6 | 6 | type: lab |
| 7 | + kind: nonprofit | |
| 7 | 8 | aliases: [Eleuther AI] |
| 8 | 9 | domains: [eleuther.ai] |
| 9 | 10 | hf_org: EleutherAI |
@@ -13,6 +14,7 @@ organizations: | ||
| 13 | 14 | - key: stanford-crfm |
| 14 | 15 | name: Stanford CRFM |
| 15 | 16 | type: university |
| 17 | + kind: university | |
| 16 | 18 | aliases: [Center for Research on Foundation Models, Stanford Center for Research on Foundation Models] |
| 17 | 19 | domains: [crfm.stanford.edu] |
| 18 | 20 | hf_org: stanford-crfm |
@@ -23,6 +25,7 @@ organizations: | ||
| 23 | 25 | - key: berriai |
| 24 | 26 | name: BerriAI |
| 25 | 27 | type: company |
| 28 | + kind: company | |
| 26 | 29 | aliases: [LiteLLM, Berri AI] |
| 27 | 30 | domains: [litellm.ai, docs.litellm.ai] |
| 28 | 31 | github_org: BerriAI |
@@ -31,6 +34,7 @@ organizations: | ||
| 31 | 34 | - key: chroma |
| 32 | 35 | name: Chroma |
| 33 | 36 | type: company |
| 37 | + kind: company | |
| 34 | 38 | aliases: [Chroma DB, ChromaDB, trychroma] |
| 35 | 39 | domains: [trychroma.com] |
| 36 | 40 | github_org: chroma-core |
@@ -39,6 +43,7 @@ organizations: | ||
| 39 | 43 | - key: qdrant |
| 40 | 44 | name: Qdrant |
| 41 | 45 | type: company |
| 46 | + kind: company | |
| 42 | 47 | domains: [qdrant.tech] |
| 43 | 48 | github_org: qdrant |
| 44 | 49 | website: https://qdrant.tech |
@@ -46,6 +51,7 @@ organizations: | ||
| 46 | 51 | - key: zilliz |
| 47 | 52 | name: Zilliz |
| 48 | 53 | type: company |
| 54 | + kind: company | |
| 49 | 55 | aliases: [Milvus] |
| 50 | 56 | domains: [zilliz.com, milvus.io] |
| 51 | 57 | github_org: milvus-io |
@@ -54,6 +60,7 @@ organizations: | ||
| 54 | 60 | - key: weaviate |
| 55 | 61 | name: Weaviate |
| 56 | 62 | type: company |
| 63 | + kind: company | |
| 57 | 64 | domains: [weaviate.io] |
| 58 | 65 | github_org: weaviate |
| 59 | 66 | website: https://weaviate.io |
@@ -61,6 +68,7 @@ organizations: | ||
| 61 | 68 | - key: lancedb |
| 62 | 69 | name: LanceDB |
| 63 | 70 | type: company |
| 71 | + kind: company | |
| 64 | 72 | domains: [lancedb.com] |
| 65 | 73 | github_org: lancedb |
| 66 | 74 | website: https://lancedb.com |
@@ -68,6 +76,7 @@ organizations: | ||
| 68 | 76 | - key: all-hands-ai |
| 69 | 77 | name: All Hands AI |
| 70 | 78 | type: company |
| 79 | + kind: company | |
| 71 | 80 | aliases: [OpenHands, All-Hands-AI] |
| 72 | 81 | domains: [all-hands.dev] |
| 73 | 82 | github_org: OpenHands |
@@ -76,6 +85,7 @@ organizations: | ||
| 76 | 85 | - key: crewai |
| 77 | 86 | name: CrewAI |
| 78 | 87 | type: company |
| 88 | + kind: company | |
| 79 | 89 | aliases: [crewAI Inc.] |
| 80 | 90 | domains: [crewai.com] |
| 81 | 91 | github_org: crewAIInc |
@@ -84,6 +94,7 @@ organizations: | ||
| 84 | 94 | - key: pydantic |
| 85 | 95 | name: Pydantic |
| 86 | 96 | type: company |
| 97 | + kind: company | |
| 87 | 98 | aliases: [Pydantic Services] |
| 88 | 99 | domains: [pydantic.dev] |
| 89 | 100 | github_org: pydantic |
@@ -92,6 +103,7 @@ organizations: | ||
| 92 | 103 | - key: browser-use |
| 93 | 104 | name: Browser Use |
| 94 | 105 | type: company |
| 106 | + kind: company | |
| 95 | 107 | aliases: [browser-use] |
| 96 | 108 | domains: [browser-use.com] |
| 97 | 109 | github_org: browser-use |
@@ -100,12 +112,14 @@ organizations: | ||
| 100 | 112 | - key: exo-labs |
| 101 | 113 | name: EXO Labs |
| 102 | 114 | type: company |
| 115 | + kind: company | |
| 103 | 116 | aliases: [exo, exo-explore] |
| 104 | 117 | github_org: exo-explore |
| 105 | 118 | source_url: https://github.com/exo-explore |
| 106 | 119 | - key: lm-studio |
| 107 | 120 | name: LM Studio |
| 108 | 121 | type: company |
| 122 | + kind: company | |
| 109 | 123 | aliases: [Element Labs, lmstudio] |
| 110 | 124 | domains: [lmstudio.ai] |
| 111 | 125 | github_org: lmstudio-ai |
@@ -114,6 +128,7 @@ organizations: | ||
| 114 | 128 | - key: comfy-org |
| 115 | 129 | name: Comfy Org |
| 116 | 130 | type: organization |
| 131 | + kind: community | |
| 117 | 132 | aliases: [ComfyUI, comfyanonymous] |
| 118 | 133 | domains: [comfy.org] |
| 119 | 134 | github_org: Comfy-Org |
@@ -130,6 +145,7 @@ organizations: | ||
| 130 | 145 | - key: internlm |
| 131 | 146 | name: InternLM (Shanghai AI Laboratory) |
| 132 | 147 | type: lab |
| 148 | + kind: lab | |
| 133 | 149 | aliases: [InternLM, Shanghai AI Lab, Shanghai Artificial Intelligence Laboratory] |
| 134 | 150 | hf_org: internlm |
| 135 | 151 | github_org: InternLM |
@@ -138,12 +154,14 @@ organizations: | ||
| 138 | 154 | - key: axolotl-ai |
| 139 | 155 | name: Axolotl AI |
| 140 | 156 | type: organization |
| 157 | + kind: community | |
| 141 | 158 | aliases: [axolotl] |
| 142 | 159 | github_org: axolotl-ai-cloud |
| 143 | 160 | source_url: https://github.com/axolotl-ai-cloud |
| 144 | 161 | - key: vibrant-labs |
| 145 | 162 | name: Vibrant Labs (Ragas) |
| 146 | 163 | type: company |
| 164 | + kind: company | |
| 147 | 165 | aliases: [Ragas, explodinggradients, Exploding Gradients, vibrantlabsai] |
| 148 | 166 | domains: [ragas.io] |
| 149 | 167 | github_org: vibrantlabsai |
@@ -152,6 +170,7 @@ organizations: | ||
| 152 | 170 | - key: langfuse |
| 153 | 171 | name: Langfuse |
| 154 | 172 | type: company |
| 173 | + kind: company | |
| 155 | 174 | domains: [langfuse.com] |
| 156 | 175 | github_org: langfuse |
| 157 | 176 | website: https://langfuse.com |
@@ -159,6 +178,7 @@ organizations: | ||
| 159 | 178 | - key: arize |
| 160 | 179 | name: Arize AI |
| 161 | 180 | type: company |
| 181 | + kind: company | |
| 162 | 182 | aliases: [Arize, Arize Phoenix] |
| 163 | 183 | domains: [arize.com] |
| 164 | 184 | github_org: Arize-ai |
@@ -167,6 +187,7 @@ organizations: | ||
| 167 | 187 | - key: anyscale |
| 168 | 188 | name: Anyscale |
| 169 | 189 | type: company |
| 190 | + kind: company | |
| 170 | 191 | aliases: [Ray, Ray project] |
| 171 | 192 | domains: [anyscale.com, ray.io] |
| 172 | 193 | github_org: ray-project |
@@ -175,6 +196,7 @@ organizations: | ||
| 175 | 196 | - key: kubeflow |
| 176 | 197 | name: Kubeflow |
| 177 | 198 | type: organization |
| 199 | + kind: community | |
| 178 | 200 | domains: [kubeflow.org] |
| 179 | 201 | github_org: kubeflow |
| 180 | 202 | website: https://www.kubeflow.org |
@@ -182,6 +204,7 @@ organizations: | ||
| 182 | 204 | - key: bentoml |
| 183 | 205 | name: BentoML |
| 184 | 206 | type: company |
| 207 | + kind: company | |
| 185 | 208 | domains: [bentoml.com] |
| 186 | 209 | github_org: bentoml |
| 187 | 210 | website: https://www.bentoml.com |
@@ -189,6 +212,7 @@ organizations: | ||
| 189 | 212 | - key: lightning-ai |
| 190 | 213 | name: Lightning AI |
| 191 | 214 | type: company |
| 215 | + kind: company | |
| 192 | 216 | aliases: [PyTorch Lightning, Grid.ai] |
| 193 | 217 | domains: [lightning.ai] |
| 194 | 218 | github_org: Lightning-AI |
@@ -197,18 +221,21 @@ organizations: | ||
| 197 | 221 | - key: dao-ailab |
| 198 | 222 | name: Dao AI Lab |
| 199 | 223 | type: lab |
| 224 | + kind: lab | |
| 200 | 225 | aliases: [Dao-AILab, FlashAttention] |
| 201 | 226 | github_org: Dao-AILab |
| 202 | 227 | source_url: https://github.com/Dao-AILab |
| 203 | 228 | - key: turboderp |
| 204 | 229 | name: turboderp |
| 205 | 230 | type: organization |
| 231 | + kind: individual | |
| 206 | 232 | aliases: [ExLlama, turboderp-org] |
| 207 | 233 | github_org: turboderp-org |
| 208 | 234 | source_url: https://github.com/turboderp-org |
| 209 | 235 | - key: model-context-protocol |
| 210 | 236 | name: Model Context Protocol |
| 211 | 237 | type: organization |
| 238 | + kind: community | |
| 212 | 239 | aliases: [MCP, modelcontextprotocol] |
| 213 | 240 | domains: [modelcontextprotocol.io] |
| 214 | 241 | github_org: modelcontextprotocol |
@@ -217,6 +244,7 @@ organizations: | ||
| 217 | 244 | - key: canopy-labs |
| 218 | 245 | name: Canopy Labs |
| 219 | 246 | type: company |
| 247 | + kind: company | |
| 220 | 248 | aliases: [canopylabs] |
| 221 | 249 | hf_org: canopylabs |
| 222 | 250 | source_url: https://huggingface.co/canopylabs |
modified
registry/organizations.yaml
+169 −9
@@ -1,11 +1,17 @@ | ||
| 1 | 1 | # Canonical organizations (companies, labs, universities, foundations). Deterministic aliases and identifiers feed entity resolution; |
| 2 | 2 | # every factual attribute carries the page it was read from (`source_url`). Fields not known are omitted — never guessed. |
| 3 | +# | |
| 4 | +# `type` = entity type (company | lab | organization | university) — kept for slugs and back-compat. | |
| 5 | +# `kind` = canonical organisation kind (aiatlas.ontology.taxonomy.ORG_KINDS: company | lab | university | nonprofit | government | community | | |
| 6 | +# consortium | individual) → attribute `org_kind`. Omitted when we are not sure; the seed then falls back to the type's default (none for `organization`). | |
| 7 | +# `country` = ISO-2 of the legal seat as stated on the organisation's own site; omitted when not stated there. | |
| 3 | 8 | organizations: |
| 4 | 9 | - key: openai |
| 5 | 10 | name: OpenAI |
| 6 | 11 | type: company |
| 12 | + kind: company | |
| 7 | 13 | aliases: [OpenAI Inc., OpenAI, L.P., OpenAI OpCo] |
| 8 | − domains: [openai.com, platform.openai.com, chatgpt.com] | |
| 14 | + domains: [openai.com, platform.openai.com, chatgpt.com, developers.openai.com] | |
| 9 | 15 | hf_org: openai |
| 10 | 16 | github_org: openai |
| 11 | 17 | country: US |
@@ -16,8 +22,9 @@ organizations: | ||
| 16 | 22 | - key: anthropic |
| 17 | 23 | name: Anthropic |
| 18 | 24 | type: company |
| 25 | + kind: company | |
| 19 | 26 | aliases: [Anthropic PBC] |
| 20 | − domains: [anthropic.com, claude.com, docs.claude.com, claude.ai] | |
| 27 | + domains: [anthropic.com, claude.com, docs.claude.com, claude.ai, platform.claude.com] | |
| 21 | 28 | hf_org: Anthropic |
| 22 | 29 | github_org: anthropics |
| 23 | 30 | country: US |
@@ -28,6 +35,7 @@ organizations: | ||
| 28 | 35 | - key: google |
| 29 | 36 | name: Google |
| 30 | 37 | type: company |
| 38 | + kind: company | |
| 31 | 39 | aliases: [Google LLC, Alphabet, Google AI, Google Research] |
| 32 | 40 | domains: [google.com, ai.google, ai.google.dev, research.google, blog.google, cloud.google.com] |
| 33 | 41 | hf_org: google |
@@ -39,6 +47,7 @@ organizations: | ||
| 39 | 47 | - key: google-deepmind |
| 40 | 48 | name: Google DeepMind |
| 41 | 49 | type: lab |
| 50 | + kind: lab | |
| 42 | 51 | aliases: [DeepMind, DeepMind Technologies, Google Brain] |
| 43 | 52 | domains: [deepmind.google, deepmind.com] |
| 44 | 53 | github_org: google-deepmind |
@@ -51,6 +60,7 @@ organizations: | ||
| 51 | 60 | - key: meta |
| 52 | 61 | name: Meta Platforms |
| 53 | 62 | type: company |
| 63 | + kind: company | |
| 54 | 64 | aliases: [Meta, Facebook, Facebook Inc.] |
| 55 | 65 | domains: [meta.com, about.fb.com] |
| 56 | 66 | country: US |
@@ -60,7 +70,8 @@ organizations: | ||
| 60 | 70 | - key: meta-ai |
| 61 | 71 | name: Meta AI |
| 62 | 72 | type: lab |
| 63 | − aliases: [FAIR, Facebook AI Research, Meta FAIR, Meta AI Research] | |
| 73 | + kind: lab | |
| 74 | + aliases: [FAIR, Facebook AI Research, Meta FAIR, Meta AI Research, Meta Superintelligence Labs] | |
| 64 | 75 | domains: [ai.meta.com, llama.com, ai.facebook.com] |
| 65 | 76 | hf_org: meta-llama |
| 66 | 77 | github_org: facebookresearch |
@@ -71,6 +82,7 @@ organizations: | ||
| 71 | 82 | - key: microsoft |
| 72 | 83 | name: Microsoft |
| 73 | 84 | type: company |
| 85 | + kind: company | |
| 74 | 86 | aliases: [Microsoft Corporation, Microsoft Research, Microsoft AI, Azure AI] |
| 75 | 87 | domains: [microsoft.com, azure.microsoft.com, learn.microsoft.com] |
| 76 | 88 | hf_org: microsoft |
@@ -83,6 +95,7 @@ organizations: | ||
| 83 | 95 | - key: nvidia |
| 84 | 96 | name: NVIDIA |
| 85 | 97 | type: company |
| 98 | + kind: company | |
| 86 | 99 | aliases: [NVIDIA Corporation, Nvidia, NVIDIA AI, NVIDIA Research] |
| 87 | 100 | domains: [nvidia.com, developer.nvidia.com, blogs.nvidia.com, build.nvidia.com] |
| 88 | 101 | hf_org: nvidia |
@@ -95,6 +108,7 @@ organizations: | ||
| 95 | 108 | - key: apple |
| 96 | 109 | name: Apple |
| 97 | 110 | type: company |
| 111 | + kind: company | |
| 98 | 112 | aliases: [Apple Inc., Apple Machine Learning Research] |
| 99 | 113 | domains: [apple.com, machinelearning.apple.com] |
| 100 | 114 | hf_org: apple |
@@ -107,6 +121,7 @@ organizations: | ||
| 107 | 121 | - key: mistral |
| 108 | 122 | name: Mistral AI |
| 109 | 123 | type: company |
| 124 | + kind: company | |
| 110 | 125 | aliases: [Mistral, MistralAI] |
| 111 | 126 | domains: [mistral.ai, docs.mistral.ai, console.mistral.ai] |
| 112 | 127 | hf_org: mistralai |
@@ -119,6 +134,7 @@ organizations: | ||
| 119 | 134 | - key: alibaba |
| 120 | 135 | name: Alibaba Group |
| 121 | 136 | type: company |
| 137 | + kind: company | |
| 122 | 138 | aliases: [Alibaba, Alibaba Cloud, Aliyun] |
| 123 | 139 | domains: [alibabacloud.com, alibaba.com] |
| 124 | 140 | country: CN |
@@ -128,6 +144,7 @@ organizations: | ||
| 128 | 144 | - key: qwen |
| 129 | 145 | name: Qwen |
| 130 | 146 | type: lab |
| 147 | + kind: lab | |
| 131 | 148 | aliases: [Qwen Team, Alibaba Qwen, Tongyi Qianwen, QwenLM] |
| 132 | 149 | domains: [qwenlm.github.io, qwen.ai, chat.qwen.ai] |
| 133 | 150 | hf_org: Qwen |
@@ -139,6 +156,7 @@ organizations: | ||
| 139 | 156 | - key: deepseek |
| 140 | 157 | name: DeepSeek |
| 141 | 158 | type: company |
| 159 | + kind: company | |
| 142 | 160 | aliases: [DeepSeek AI, DeepSeek-AI, Hangzhou DeepSeek Artificial Intelligence] |
| 143 | 161 | domains: [deepseek.com, api-docs.deepseek.com, chat.deepseek.com] |
| 144 | 162 | hf_org: deepseek-ai |
@@ -151,7 +169,8 @@ organizations: | ||
| 151 | 169 | - key: cohere |
| 152 | 170 | name: Cohere |
| 153 | 171 | type: company |
| 154 | − aliases: [Cohere Inc., Cohere For AI, C4AI] | |
| 172 | + kind: company | |
| 173 | + aliases: [Cohere Inc., Cohere For AI, C4AI, Cohere Labs] | |
| 155 | 174 | domains: [cohere.com, docs.cohere.com, cohere.ai] |
| 156 | 175 | hf_org: CohereLabs |
| 157 | 176 | github_org: cohere-ai |
@@ -163,6 +182,7 @@ organizations: | ||
| 163 | 182 | - key: xai |
| 164 | 183 | name: xAI |
| 165 | 184 | type: company |
| 185 | + kind: company | |
| 166 | 186 | aliases: [x.ai, xAI Corp] |
| 167 | 187 | domains: [x.ai, docs.x.ai] |
| 168 | 188 | hf_org: xai-org |
@@ -174,6 +194,7 @@ organizations: | ||
| 174 | 194 | - key: huggingface |
| 175 | 195 | name: Hugging Face |
| 176 | 196 | type: company |
| 197 | + kind: company | |
| 177 | 198 | aliases: [HuggingFace, HF] |
| 178 | 199 | domains: [huggingface.co, hf.co] |
| 179 | 200 | hf_org: HuggingFaceTB |
@@ -186,6 +207,7 @@ organizations: | ||
| 186 | 207 | - key: stability-ai |
| 187 | 208 | name: Stability AI |
| 188 | 209 | type: company |
| 210 | + kind: company | |
| 189 | 211 | aliases: [StabilityAI] |
| 190 | 212 | domains: [stability.ai] |
| 191 | 213 | hf_org: stabilityai |
@@ -196,6 +218,7 @@ organizations: | ||
| 196 | 218 | - key: ai21 |
| 197 | 219 | name: AI21 Labs |
| 198 | 220 | type: company |
| 221 | + kind: company | |
| 199 | 222 | aliases: [AI21] |
| 200 | 223 | domains: [ai21.com] |
| 201 | 224 | hf_org: ai21labs |
@@ -207,6 +230,7 @@ organizations: | ||
| 207 | 230 | - key: together-ai |
| 208 | 231 | name: Together AI |
| 209 | 232 | type: company |
| 233 | + kind: company | |
| 210 | 234 | aliases: [Together, Together Computer] |
| 211 | 235 | domains: [together.ai, together.xyz] |
| 212 | 236 | hf_org: togethercomputer |
@@ -217,8 +241,9 @@ organizations: | ||
| 217 | 241 | - key: fireworks-ai |
| 218 | 242 | name: Fireworks AI |
| 219 | 243 | type: company |
| 244 | + kind: company | |
| 220 | 245 | aliases: [Fireworks] |
| 221 | − domains: [fireworks.ai] | |
| 246 | + domains: [fireworks.ai, docs.fireworks.ai] | |
| 222 | 247 | hf_org: fireworks-ai |
| 223 | 248 | github_org: fw-ai |
| 224 | 249 | country: US |
@@ -227,6 +252,7 @@ organizations: | ||
| 227 | 252 | - key: groq |
| 228 | 253 | name: Groq |
| 229 | 254 | type: company |
| 255 | + kind: company | |
| 230 | 256 | aliases: [Groq Inc., GroqCloud] |
| 231 | 257 | domains: [groq.com, console.groq.com] |
| 232 | 258 | github_org: groq |
@@ -237,6 +263,7 @@ organizations: | ||
| 237 | 263 | - key: cerebras |
| 238 | 264 | name: Cerebras Systems |
| 239 | 265 | type: company |
| 266 | + kind: company | |
| 240 | 267 | aliases: [Cerebras] |
| 241 | 268 | domains: [cerebras.ai, cerebras.net] |
| 242 | 269 | hf_org: cerebras |
@@ -247,6 +274,7 @@ organizations: | ||
| 247 | 274 | - key: sambanova |
| 248 | 275 | name: SambaNova Systems |
| 249 | 276 | type: company |
| 277 | + kind: company | |
| 250 | 278 | aliases: [SambaNova] |
| 251 | 279 | domains: [sambanova.ai] |
| 252 | 280 | country: US |
@@ -255,6 +283,7 @@ organizations: | ||
| 255 | 283 | - key: perplexity |
| 256 | 284 | name: Perplexity AI |
| 257 | 285 | type: company |
| 286 | + kind: company | |
| 258 | 287 | aliases: [Perplexity] |
| 259 | 288 | domains: [perplexity.ai] |
| 260 | 289 | hf_org: perplexity-ai |
@@ -264,6 +293,7 @@ organizations: | ||
| 264 | 293 | - key: openrouter |
| 265 | 294 | name: OpenRouter |
| 266 | 295 | type: company |
| 296 | + kind: company | |
| 267 | 297 | aliases: [OpenRouter.ai] |
| 268 | 298 | domains: [openrouter.ai] |
| 269 | 299 | country: US |
@@ -272,12 +302,14 @@ organizations: | ||
| 272 | 302 | - key: deepinfra |
| 273 | 303 | name: DeepInfra |
| 274 | 304 | type: company |
| 305 | + kind: company | |
| 275 | 306 | domains: [deepinfra.com] |
| 276 | 307 | website: https://deepinfra.com |
| 277 | 308 | source_url: https://deepinfra.com/about |
| 278 | 309 | - key: replicate |
| 279 | 310 | name: Replicate |
| 280 | 311 | type: company |
| 312 | + kind: company | |
| 281 | 313 | domains: [replicate.com] |
| 282 | 314 | github_org: replicate |
| 283 | 315 | country: US |
@@ -286,6 +318,7 @@ organizations: | ||
| 286 | 318 | - key: cloudflare |
| 287 | 319 | name: Cloudflare |
| 288 | 320 | type: company |
| 321 | + kind: company | |
| 289 | 322 | aliases: [Cloudflare Workers AI] |
| 290 | 323 | domains: [cloudflare.com, developers.cloudflare.com] |
| 291 | 324 | country: US |
@@ -294,7 +327,8 @@ organizations: | ||
| 294 | 327 | - key: amazon |
| 295 | 328 | name: Amazon Web Services |
| 296 | 329 | type: company |
| 297 | − aliases: [AWS, Amazon Bedrock, Amazon, Amazon.com] | |
| 330 | + kind: company | |
| 331 | + aliases: [AWS, Amazon Bedrock, Amazon, Amazon.com, Amazon AGI] | |
| 298 | 332 | domains: [aws.amazon.com, amazon.com] |
| 299 | 333 | hf_org: amazon |
| 300 | 334 | github_org: aws |
@@ -304,6 +338,7 @@ organizations: | ||
| 304 | 338 | - key: elevenlabs |
| 305 | 339 | name: ElevenLabs |
| 306 | 340 | type: company |
| 341 | + kind: company | |
| 307 | 342 | domains: [elevenlabs.io] |
| 308 | 343 | country: US |
| 309 | 344 | website: https://elevenlabs.io |
@@ -311,6 +346,7 @@ organizations: | ||
| 311 | 346 | - key: runway |
| 312 | 347 | name: Runway |
| 313 | 348 | type: company |
| 349 | + kind: company | |
| 314 | 350 | aliases: [Runway ML, RunwayML] |
| 315 | 351 | domains: [runwayml.com] |
| 316 | 352 | country: US |
@@ -319,6 +355,7 @@ organizations: | ||
| 319 | 355 | - key: amd |
| 320 | 356 | name: AMD |
| 321 | 357 | type: company |
| 358 | + kind: company | |
| 322 | 359 | aliases: [Advanced Micro Devices] |
| 323 | 360 | domains: [amd.com] |
| 324 | 361 | hf_org: amd |
@@ -330,6 +367,7 @@ organizations: | ||
| 330 | 367 | - key: intel |
| 331 | 368 | name: Intel |
| 332 | 369 | type: company |
| 370 | + kind: company | |
| 333 | 371 | aliases: [Intel Corporation] |
| 334 | 372 | domains: [intel.com] |
| 335 | 373 | hf_org: Intel |
@@ -340,6 +378,7 @@ organizations: | ||
| 340 | 378 | - key: arxiv |
| 341 | 379 | name: arXiv |
| 342 | 380 | type: organization |
| 381 | + kind: nonprofit | |
| 343 | 382 | aliases: [arXiv.org, Cornell arXiv] |
| 344 | 383 | domains: [arxiv.org, export.arxiv.org, rss.arxiv.org] |
| 345 | 384 | website: https://arxiv.org |
@@ -347,12 +386,14 @@ organizations: | ||
| 347 | 386 | - key: openreview |
| 348 | 387 | name: OpenReview |
| 349 | 388 | type: organization |
| 389 | + kind: nonprofit | |
| 350 | 390 | domains: [openreview.net] |
| 351 | 391 | website: https://openreview.net |
| 352 | 392 | source_url: https://openreview.net/about |
| 353 | 393 | - key: github |
| 354 | 394 | name: GitHub |
| 355 | 395 | type: company |
| 396 | + kind: company | |
| 356 | 397 | aliases: [GitHub Inc.] |
| 357 | 398 | domains: [github.com, raw.githubusercontent.com] |
| 358 | 399 | parent: microsoft |
@@ -362,13 +403,16 @@ organizations: | ||
| 362 | 403 | - key: python-software-foundation |
| 363 | 404 | name: Python Software Foundation |
| 364 | 405 | type: organization |
| 406 | + kind: nonprofit | |
| 365 | 407 | aliases: [PSF, PyPI] |
| 366 | 408 | domains: [pypi.org, python.org] |
| 409 | + country: US | |
| 367 | 410 | website: https://www.python.org/psf/ |
| 368 | 411 | source_url: https://www.python.org/psf/about/ |
| 369 | 412 | - key: aider |
| 370 | 413 | name: Aider |
| 371 | 414 | type: organization |
| 415 | + kind: community | |
| 372 | 416 | aliases: [Aider AI] |
| 373 | 417 | domains: [aider.chat] |
| 374 | 418 | github_org: Aider-AI |
@@ -377,6 +421,7 @@ organizations: | ||
| 377 | 421 | - key: swe-bench |
| 378 | 422 | name: SWE-bench |
| 379 | 423 | type: organization |
| 424 | + kind: community | |
| 380 | 425 | aliases: [SWE-bench team, Princeton SWE-bench] |
| 381 | 426 | domains: [swebench.com] |
| 382 | 427 | github_org: SWE-bench |
@@ -385,6 +430,7 @@ organizations: | ||
| 385 | 430 | - key: livebench |
| 386 | 431 | name: LiveBench |
| 387 | 432 | type: organization |
| 433 | + kind: community | |
| 388 | 434 | domains: [livebench.ai] |
| 389 | 435 | github_org: LiveBench |
| 390 | 436 | website: https://livebench.ai |
@@ -392,6 +438,7 @@ organizations: | ||
| 392 | 438 | - key: artificial-analysis |
| 393 | 439 | name: Artificial Analysis |
| 394 | 440 | type: company |
| 441 | + kind: company | |
| 395 | 442 | domains: [artificialanalysis.ai] |
| 396 | 443 | website: https://artificialanalysis.ai |
| 397 | 444 | source_url: https://artificialanalysis.ai/ |
@@ -404,6 +451,7 @@ organizations: | ||
| 404 | 451 | - key: pytorch-foundation |
| 405 | 452 | name: PyTorch Foundation |
| 406 | 453 | type: organization |
| 454 | + kind: nonprofit | |
| 407 | 455 | aliases: [PyTorch, Linux Foundation PyTorch] |
| 408 | 456 | domains: [pytorch.org] |
| 409 | 457 | github_org: pytorch |
@@ -412,6 +460,7 @@ organizations: | ||
| 412 | 460 | - key: ggml |
| 413 | 461 | name: ggml.ai |
| 414 | 462 | type: organization |
| 463 | + kind: company | |
| 415 | 464 | aliases: [ggml, ggml-org, llama.cpp project] |
| 416 | 465 | domains: [ggml.ai] |
| 417 | 466 | github_org: ggml-org |
@@ -420,6 +469,7 @@ organizations: | ||
| 420 | 469 | - key: vllm |
| 421 | 470 | name: vLLM project |
| 422 | 471 | type: organization |
| 472 | + kind: community | |
| 423 | 473 | aliases: [vLLM] |
| 424 | 474 | domains: [vllm.ai, docs.vllm.ai] |
| 425 | 475 | github_org: vllm-project |
@@ -428,6 +478,7 @@ organizations: | ||
| 428 | 478 | - key: ollama |
| 429 | 479 | name: Ollama |
| 430 | 480 | type: company |
| 481 | + kind: company | |
| 431 | 482 | domains: [ollama.com] |
| 432 | 483 | github_org: ollama |
| 433 | 484 | website: https://ollama.com |
@@ -435,6 +486,7 @@ organizations: | ||
| 435 | 486 | - key: langchain |
| 436 | 487 | name: LangChain |
| 437 | 488 | type: company |
| 489 | + kind: company | |
| 438 | 490 | aliases: [LangChain Inc., LangGraph] |
| 439 | 491 | domains: [langchain.com] |
| 440 | 492 | github_org: langchain-ai |
@@ -444,6 +496,7 @@ organizations: | ||
| 444 | 496 | - key: llamaindex |
| 445 | 497 | name: LlamaIndex |
| 446 | 498 | type: company |
| 499 | + kind: company | |
| 447 | 500 | aliases: [Llama Index, run-llama] |
| 448 | 501 | domains: [llamaindex.ai] |
| 449 | 502 | github_org: run-llama |
@@ -452,6 +505,7 @@ organizations: | ||
| 452 | 505 | - key: sglang |
| 453 | 506 | name: SGLang project |
| 454 | 507 | type: organization |
| 508 | + kind: community | |
| 455 | 509 | aliases: [SGLang, LMSYS] |
| 456 | 510 | domains: [sglang.ai, lmsys.org] |
| 457 | 511 | github_org: sgl-project |
@@ -460,6 +514,8 @@ organizations: | ||
| 460 | 514 | - key: unsloth |
| 461 | 515 | name: Unsloth |
| 462 | 516 | type: company |
| 517 | + kind: company | |
| 518 | + aliases: [Unsloth AI] | |
| 463 | 519 | domains: [unsloth.ai] |
| 464 | 520 | hf_org: unsloth |
| 465 | 521 | github_org: unslothai |
@@ -468,6 +524,7 @@ organizations: | ||
| 468 | 524 | - key: bartowski |
| 469 | 525 | name: bartowski |
| 470 | 526 | type: organization |
| 527 | + kind: individual | |
| 471 | 528 | aliases: [Bartowski quantizations] |
| 472 | 529 | hf_org: bartowski |
| 473 | 530 | website: https://huggingface.co/bartowski |
@@ -475,6 +532,7 @@ organizations: | ||
| 475 | 532 | - key: mlx-community |
| 476 | 533 | name: MLX Community |
| 477 | 534 | type: organization |
| 535 | + kind: community | |
| 478 | 536 | aliases: [mlx-community] |
| 479 | 537 | hf_org: mlx-community |
| 480 | 538 | github_org: ml-explore |
@@ -483,6 +541,7 @@ organizations: | ||
| 483 | 541 | - key: allenai |
| 484 | 542 | name: Allen Institute for AI |
| 485 | 543 | type: lab |
| 544 | + kind: nonprofit | |
| 486 | 545 | aliases: [AI2, AllenAI, Ai2] |
| 487 | 546 | domains: [allenai.org] |
| 488 | 547 | hf_org: allenai |
@@ -494,6 +553,8 @@ organizations: | ||
| 494 | 553 | - key: nous-research |
| 495 | 554 | name: Nous Research |
| 496 | 555 | type: company |
| 556 | + kind: company | |
| 557 | + aliases: [Nous, NousResearch] | |
| 497 | 558 | domains: [nousresearch.com] |
| 498 | 559 | hf_org: NousResearch |
| 499 | 560 | github_org: NousResearch |
@@ -502,7 +563,8 @@ organizations: | ||
| 502 | 563 | - key: zhipu |
| 503 | 564 | name: Z.ai (Zhipu AI) |
| 504 | 565 | type: company |
| 505 | − aliases: [Zhipu AI, Zhipu, THUDM, Z.ai, ChatGLM] | |
| 566 | + kind: company | |
| 567 | + aliases: [Zhipu AI, Zhipu, THUDM, Z.ai, ChatGLM, Z AI] | |
| 506 | 568 | domains: [z.ai, zhipuai.cn, bigmodel.cn] |
| 507 | 569 | hf_org: zai-org |
| 508 | 570 | github_org: zai-org |
@@ -513,7 +575,8 @@ organizations: | ||
| 513 | 575 | - key: moonshot |
| 514 | 576 | name: Moonshot AI |
| 515 | 577 | type: company |
| 516 | − aliases: [Moonshot, Kimi] | |
| 578 | + kind: company | |
| 579 | + aliases: [Moonshot, Kimi, MoonshotAI] | |
| 517 | 580 | domains: [moonshot.cn, moonshot.ai, kimi.com] |
| 518 | 581 | hf_org: moonshotai |
| 519 | 582 | github_org: MoonshotAI |
@@ -524,7 +587,8 @@ organizations: | ||
| 524 | 587 | - key: minimax |
| 525 | 588 | name: MiniMax |
| 526 | 589 | type: company |
| 527 | − aliases: [MiniMax AI] | |
| 590 | + kind: company | |
| 591 | + aliases: [MiniMax AI, MiniMaxAI] | |
| 528 | 592 | domains: [minimax.io, minimaxi.com] |
| 529 | 593 | hf_org: MiniMaxAI |
| 530 | 594 | github_org: MiniMax-AI |
@@ -534,6 +598,7 @@ organizations: | ||
| 534 | 598 | - key: bytedance |
| 535 | 599 | name: ByteDance |
| 536 | 600 | type: company |
| 601 | + kind: company | |
| 537 | 602 | aliases: [ByteDance Seed, Seed, Doubao] |
| 538 | 603 | domains: [bytedance.com, seed.bytedance.com] |
| 539 | 604 | hf_org: ByteDance-Seed |
@@ -544,6 +609,7 @@ organizations: | ||
| 544 | 609 | - key: baidu |
| 545 | 610 | name: Baidu |
| 546 | 611 | type: company |
| 612 | + kind: company | |
| 547 | 613 | aliases: [Baidu ERNIE, ERNIE] |
| 548 | 614 | domains: [baidu.com, ernie.baidu.com] |
| 549 | 615 | hf_org: baidu |
@@ -554,6 +620,7 @@ organizations: | ||
| 554 | 620 | - key: tencent |
| 555 | 621 | name: Tencent |
| 556 | 622 | type: company |
| 623 | + kind: company | |
| 557 | 624 | aliases: [Tencent Hunyuan, Hunyuan] |
| 558 | 625 | domains: [tencent.com, hunyuan.tencent.com] |
| 559 | 626 | hf_org: tencent |
@@ -564,6 +631,7 @@ organizations: | ||
| 564 | 631 | - key: ibm |
| 565 | 632 | name: IBM |
| 566 | 633 | type: company |
| 634 | + kind: company | |
| 567 | 635 | aliases: [IBM Research, IBM Granite] |
| 568 | 636 | domains: [ibm.com, research.ibm.com] |
| 569 | 637 | hf_org: ibm-granite |
@@ -574,6 +642,7 @@ organizations: | ||
| 574 | 642 | - key: databricks |
| 575 | 643 | name: Databricks |
| 576 | 644 | type: company |
| 645 | + kind: company | |
| 577 | 646 | aliases: [Mosaic, MosaicML] |
| 578 | 647 | domains: [databricks.com] |
| 579 | 648 | hf_org: databricks |
@@ -584,6 +653,7 @@ organizations: | ||
| 584 | 653 | - key: salesforce |
| 585 | 654 | name: Salesforce |
| 586 | 655 | type: company |
| 656 | + kind: company | |
| 587 | 657 | aliases: [Salesforce AI Research] |
| 588 | 658 | domains: [salesforce.com] |
| 589 | 659 | hf_org: Salesforce |
@@ -594,6 +664,7 @@ organizations: | ||
| 594 | 664 | - key: black-forest-labs |
| 595 | 665 | name: Black Forest Labs |
| 596 | 666 | type: company |
| 667 | + kind: company | |
| 597 | 668 | aliases: [BFL, FLUX] |
| 598 | 669 | domains: [blackforestlabs.ai, bfl.ai] |
| 599 | 670 | hf_org: black-forest-labs |
@@ -604,6 +675,8 @@ organizations: | ||
| 604 | 675 | - key: liquid-ai |
| 605 | 676 | name: Liquid AI |
| 606 | 677 | type: company |
| 678 | + kind: company | |
| 679 | + aliases: [Liquid] | |
| 607 | 680 | domains: [liquid.ai] |
| 608 | 681 | hf_org: LiquidAI |
| 609 | 682 | country: US |
@@ -612,8 +685,95 @@ organizations: | ||
| 612 | 685 | - key: ml-explore |
| 613 | 686 | name: Apple ML Explore (MLX) |
| 614 | 687 | type: lab |
| 688 | + kind: lab | |
| 615 | 689 | aliases: [MLX, ml-explore] |
| 616 | 690 | github_org: ml-explore |
| 617 | 691 | parent: apple |
| 692 | + country: US | |
| 618 | 693 | website: https://ml-explore.github.io/mlx |
| 619 | 694 | source_url: https://github.com/ml-explore/mlx |
| 695 | + # ---------------------------------------------------------------- labs that appear on leaderboards (Artificial Analysis / LiveBench creators) | |
| 696 | + - key: upstage | |
| 697 | + name: Upstage | |
| 698 | + type: company | |
| 699 | + kind: company | |
| 700 | + aliases: [Upstage AI, upstage] | |
| 701 | + domains: [upstage.ai, console.upstage.ai] | |
| 702 | + hf_org: upstage | |
| 703 | + github_org: UpstageAI | |
| 704 | + country: KR | |
| 705 | + website: https://www.upstage.ai | |
| 706 | + source_url: https://www.upstage.ai/ | |
| 707 | + - key: stepfun | |
| 708 | + name: StepFun | |
| 709 | + type: company | |
| 710 | + kind: company | |
| 711 | + aliases: [Step Fun, stepfun-ai, 阶跃星辰, StepFun AI] | |
| 712 | + domains: [stepfun.com, platform.stepfun.com] | |
| 713 | + hf_org: stepfun-ai | |
| 714 | + github_org: stepfun-ai | |
| 715 | + country: CN | |
| 716 | + website: https://www.stepfun.com | |
| 717 | + source_url: https://www.stepfun.com/ | |
| 718 | + - key: kwaipilot | |
| 719 | + name: Kwaipilot | |
| 720 | + type: company # entity type of the existing production row (hf_org Kwaipilot); the canonical kind is `lab` | |
| 721 | + kind: lab | |
| 722 | + aliases: [KwaiPilot, Kuaishou Kwaipilot, KAT] | |
| 723 | + hf_org: Kwaipilot | |
| 724 | + github_org: Kwaipilot | |
| 725 | + parent: kuaishou | |
| 726 | + country: CN | |
| 727 | + website: https://github.com/Kwaipilot | |
| 728 | + source_url: https://github.com/Kwaipilot | |
| 729 | + - key: kuaishou | |
| 730 | + name: Kuaishou | |
| 731 | + type: company | |
| 732 | + kind: company | |
| 733 | + aliases: [Kuaishou Technology, Kling AI, Kwai] | |
| 734 | + domains: [kuaishou.com, klingai.com] | |
| 735 | + country: CN | |
| 736 | + website: https://www.kuaishou.com | |
| 737 | + source_url: https://ir.kuaishou.com/ | |
| 738 | + - key: thinking-machines | |
| 739 | + name: Thinking Machines | |
| 740 | + type: company | |
| 741 | + kind: company | |
| 742 | + aliases: [Thinking Machines Lab, thinkingmachines, Thinking Machines Lab Inc.] | |
| 743 | + domains: [thinkingmachines.ai] | |
| 744 | + github_org: thinking-machines-lab # no hf_org: the hub organisation is not stated on thinkingmachines.ai (a wrong value would split the entity) | |
| 745 | + country: US | |
| 746 | + website: https://thinkingmachines.ai | |
| 747 | + source_url: https://thinkingmachines.ai/ | |
| 748 | + - key: lg-ai-research | |
| 749 | + name: LG AI Research | |
| 750 | + type: lab | |
| 751 | + kind: lab | |
| 752 | + aliases: [LGAI, LG AI, EXAONE, LGAI-EXAONE] | |
| 753 | + domains: [lgresearch.ai] | |
| 754 | + hf_org: LGAI-EXAONE | |
| 755 | + github_org: LG-AI-EXAONE | |
| 756 | + country: KR | |
| 757 | + website: https://www.lgresearch.ai | |
| 758 | + source_url: https://www.lgresearch.ai/ | |
| 759 | + - key: xiaomi | |
| 760 | + name: Xiaomi | |
| 761 | + type: company | |
| 762 | + kind: company | |
| 763 | + aliases: [Xiaomi MiMo, MiMo, Xiaomi Corporation] | |
| 764 | + domains: [mi.com, xiaomimimo.com] | |
| 765 | + hf_org: XiaomiMiMo | |
| 766 | + github_org: XiaomiMiMo | |
| 767 | + country: CN | |
| 768 | + website: https://www.mi.com | |
| 769 | + source_url: https://www.mi.com/global/about/ | |
| 770 | + - key: inclusionai | |
| 771 | + name: inclusionAI (Ant Group) | |
| 772 | + type: company # entity type of the existing production row (AA creator / OpenRouter vendor); the canonical kind is `lab` | |
| 773 | + kind: lab | |
| 774 | + aliases: [inclusionAI, InclusionAI, Ant Group AGI, Ling] | |
| 775 | + hf_org: inclusionAI | |
| 776 | + github_org: inclusionAI | |
| 777 | + country: CN | |
| 778 | + website: https://github.com/inclusionAI | |
| 779 | + source_url: https://github.com/inclusionAI | |
modified
registry/repositories.yaml
+61 −58
Large diff (140 lines) is not loaded by default to keep the page fast.Load diff
added
src/aiatlas/connectors/_identity.py
+387 −0
Large diff (387 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/benchmarks/leaderboards.py
+178 −77
Large diff (424 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/code/github.py
+44 −9
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/code/pypi.py
+6 −5
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/hub/huggingface.py
+228 −56
Large diff (406 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/labs/_common.py
+108 −3
Large diff (147 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/labs/anthropic.py
+39 −9
Large diff (92 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/labs/cohere.py
+21 −8
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/labs/google.py
+14 −14
Large diff (80 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/labs/mistral.py
+30 −11
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/labs/openai.py
+16 −10
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/providers/openrouter.py
+38 −39
Large diff (161 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/providers/provider_pricing.py
+27 −16
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/research/arxiv.py
+28 −8
Large diff (58 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/connectors/research/openreview.py
+12 −5
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
modified
src/aiatlas/registry/seed.py
+47 −6
Large diff (76 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_arxiv.py
+3 −2
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_cohere.py
+7 −2
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_github_pypi.py
+11 −2
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_google.py
+10 −6
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_huggingface.py
+42 −10
Large diff (110 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_leaderboards.py
+94 −12
Large diff (145 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_mistral.py
+12 −5
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_openai.py
+4 −1
Large diff (15 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_openreview.py
+8 −1
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_openrouter.py
+13 −4
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
modified
tests/test_provider_pricing.py
+6 −2
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff