SPB Git forge
spboucher.ai ↗

spb/ai-atlas

Public
41commits 1branches 0releases
4.6 MBsize
maindefault branch
12 days agolast push
HTML 77.2% TypeScript 10.5% Python 9.6% JavaScript 2.5%

Canonical upgrade foundation: ontology package (licences, openness, taxonomy, model naming, benchmarks, anomalies), migration 0003, Facts hierarchy hints, implementation audit

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed 12 days ago (Sep 12, 2026) parent c64e7b8

12 changed files +1,624 −1

added apps/web/qa/_shot_tmp.mjs +12 −0
@@ -0,0 +1,12 @@
1 +import { chromium } from 'playwright';
2 +const b = await chromium.launch();
3 +for (const [w,h,tag] of [[1440,900,'desk'],[390,844,'mob']]) {
4 + const ctx = await b.newContext({ viewport: { width: w, height: h }, colorScheme: 'dark' });
5 + const p = await ctx.newPage();
6 + for (const path of ['/', '/models/claude-fable-5.1', '/benchmarks/gpqa']) {
7 + await p.goto('https://www.ai-atlas.co' + path, { waitUntil: 'networkidle', timeout: 60000 }).catch(()=>{});
8 + await p.screenshot({ path: `/tmp/aia-${tag}-${path.replace(/[^a-z0-9]+/gi,'_')||'home'}.png`, fullPage: false });
9 + }
10 + await ctx.close();
11 +}
12 +await b.close();
added docs/UPGRADE-2026-09-AUDIT.md +137 −0
@@ -0,0 +1,137 @@
1 +# AI Atlas — implementation audit for the canonical upgrade (2026-09-12)
2 +
3 +Written before the "temporal knowledge graph + intelligence terminal" upgrade. Numbers are from the production database
4 +(M2M32c, dump `aiatlas-prod.dump` taken 2026-09-11 23:47 EDT, 133 MB) and from reading every module of the repository.
5 +
6 +## 1. Existing architecture (keep)
7 +
8 +FastAPI + SQLAlchemy Core/asyncpg + Alembic (forward-only SQL) · Redis cache · APScheduler scheduler (`aia schedule`) running connectors
9 +**sequentially** · Postgres job queue (SKIP LOCKED) · LLM gateway → llm-api.io (deterministic first, LLM claims one tier lower) ·
10 +Next 16 App Router (server components, Tailwind v4 tokens, hand-rolled d3 SVG charts, Playwright QA borrowed from `~/Desktop/uqo-eval`).
11 +Deployment: `mld` manifest `deploy/ai-atlas.mld.json`, PM2 `ai-atlas-api :8321 / ai-atlas-scheduler / ai-atlas-web :8320`, MacLustr Tunnel.
12 +
13 +## 2. Existing schema (0001 + 0002)
14 +
15 +`entities` (type, canonical_name, slug, organization_id, attributes/provenance/quality/counts jsonb, first/last_seen, merged_into, search) ·
16 +`entity_aliases` · `entity_identifiers (scheme,value) unique` · `sources` · `connectors` · `connector_runs` · `connector_errors` ·
17 +`documents` · `snapshots (run_id, raw_path, text_path, structured, diff, parser_version)` · `claims (temporal: valid_from/valid_to, status
18 +current|superseded|conflicting|retracted, tier, confidence, extractor)` · `relations (live uniq on subject,predicate,object)` ·
19 +`change_events (observed_at, effective_at, importance, dedupe_key)` · `benchmark_results (config jsonb, dedupe_key, valid_to)` ·
20 +`prices (live uniq per model×provider×provider_model_id)` · `jobs` · `llm_jobs` · `review_queue` · `entity_embeddings` · `stats_snapshots` ·
21 +`page_views` · `api_keys` · `metric_definitions` · `domains`.
22 +
23 +Missing for this upgrade: run/batch id on facts, backfill/occurrence semantics on events, canonical hierarchy columns, benchmark
24 +comparability key and trust level, persisted resolution decisions, anomaly flags, admin audit log, quarantine.
25 +
26 +## 3. Entity types in the data (prod, live rows)
27 +
28 +researcher 1 899 · model 1 880 · paper 1 060 · dataset 146 · company 119 · library 67 · framework 66 · hardware 42 · organization 36 ·
29 +benchmark 27 · provider 27 · repository 10 · lab 8 · university 1 · tool 0 (public nav shows Tools with "0 total").
30 +`ids.ENTITY_TYPES` declares 33 types; `quantization`, `license`, `release`, `agent`, `runtime`, `tool` are declared but never produced.
31 +
32 +## 4. Claim architecture (keep)
33 +
34 +`FactWriter.write_claim`: no current → insert; same → confirm; different & source tier ≤ current → supersede + event for material
35 +properties; different & worse tier → `conflicting` + review item. Soft properties never event. `metric.*` never event. Prices and
36 +results append-only with close/open. **Loophole**: `same_source` lets an LLM claim (tier+1) supersede a deterministic claim from the same
37 +URL (`writer.py:141-145`). **Gap**: no `run_id` on facts; `EntityRef.first_seen_hint` never read (`resolution.py:131`).
38 +
39 +## 5. Connector architecture (keep)
40 +
41 +`BaseConnector.run()` → discover → fetch (conditional) → archive → snapshot (+diff) → parse → extract → `FactWriter` → follow-ups →
42 +optional `llm_extract` job. Breakage detection (`expected_min_records`, only on full re-extraction), circuit breaker, adaptive interval,
43 +reprocess-from-archive. **Bugs**: single-URL reprocess builds key-less targets → no-op (`connector.py:188`); `Fetcher` has **no SSRF guard**
44 +(no private-IP/scheme/redirect validation, follow-ups come from crawled content); rate limit read from class attr, not registry.
45 +
46 +## 6. Current model resolution
47 +
48 +`Resolver`: identifiers (exact scheme+value) → normalized alias within type (org-disambiguated) → slug → create. `normalize_alias`
49 +erases dots/hyphens/spaces (`Qwen3-8B` ≡ `Qwen 38B`). Identifier schemes are per source (`anthropic_model_id`, `gemini_model_id`,
50 +`fireworks-ai_model_id`…) so cross-source identity rests on aliases. Consequences measured in prod:
51 +
52 +* **399 models exist only through an `artificial_analysis` identifier**, 204 of them are *effort/thinking variants* of another model
53 + (`claude-fable-5-1-xhigh`, `gpt-6-astra-high`, `deepseek-v4-pro-0424-non-reasoning`) — evaluation configurations, not models.
54 +* **157 Hugging Face repos flagged `is_quantized`** (`quant_format` gguf 64, fp8 41, onnx 27, mlx 23, awq 2) plus ~150 conversions/re-packagings
55 + (`amd/…-MXFP4`, `zai-org/GLM-5-FP8`, `mlx-community/Kimi-K2.5`) are typed `model`; they dominate "largest models" (Kimi K2.5 appears 4×).
56 +* 72 models exist only through OpenRouter; OpenRouter `release_date` = listing date; routed prices booked on the lab's provider entity with
57 + a different `provider_model_id` → two concurrent "current" prices for one model×provider.
58 +* 19 exact normalized-name duplicates across slugs (`gemini-2.5-flash-lite` / `gemini-2-5-flash-lite`, `deepseek-v3-{2,3,4}`…); 80 pending
59 + `merge_candidate` items; 140 pending `conflict` items (status 34, modalities_input 30, context_length 26).
60 +* 1 899 researchers created **by name only** (first 3 arXiv authors, no identifier, no affiliation) — homonyms merge, variants split.
61 +* 70 models without organization; 853 without release date; 1 205 without parameter count; only 115 carry `family`.
62 +
63 +## 7. Current benchmark pipeline
64 +
65 +Four boards (Artificial Analysis 4 003 rows, LiveBench 456, SWE-bench 323, aider 138) → 4 920 live rows on 15 of 27 registered benchmarks
66 +(12 registry benchmarks — MMLU, HumanEval, AIME 2025, MATH-500, MTEB, LMArena… — have **zero** results: no connector feeds them).
67 +Leaderboards are one row per live result: LiveBench mixes `global_average` with 7 `category:*` metrics in one board; aider mixes
68 +`pass_rate_2` and `percent_cases_well_formed`; every LiveBench release / aider run date creates new rows that are never retired;
69 +`/compare` picks the last-observed metric per model. Models are resolved by free text (aider, SWE-bench) or AA slug, so AA effort variants
70 +become separate leaderboard entries (GPQA: 614 results = 614 "models"). No trust level, no comparability key, no metric bounds.
71 +The registry lacks family / variant / direction / metric range. The API only resolves benchmarks by exact slug (no aliases).
72 +
73 +## 8. Current event pipeline
74 +
75 +12 196 events, **all observed on 2026-09-11** (initial corpus): NEW_MODEL 1 880 (importance 3), BENCHMARK_RESULT 4 920, ANNOUNCEMENT 1 764,
76 +NEW_PAPER 1 060, RELEASE 692, PROVIDER_LISTED 613… `/changes`, `/changes/daily` and `/stats.change_events_24h` key on `observed_at`; only
77 +`/timeline` uses `coalesce(effective_at, observed_at)`. 8 713 events have no `effective_at`. There is no backfill flag, no
78 +occurred/observed/recorded distinction, no semantic dedupe of one release across sources. Homepage says "+12,196 · 24 h".
79 +
80 +## 9. Taxonomy problems found (prod values)
81 +
82 +* license: `apache-2.0` 344 · `Apache 2.0` 9 · `Modified MIT` · `CC BY-NC 4.0` vs `cc-by-nc-4.0` · `other` 157 · Llama variants `llama3`, `llama3.1`… (no SPDX, no permissions)
83 +* openness: `open-weights` 971 · empty 478 · `proprietary` 348 · `restricted` 82 (= HF *gated*, not a licence property) · `open-source` 1 (Cohere prose)
84 +* modalities: `["Text"]` vs `["text"]`; Google emits `pdf`; Anthropic values are hard-coded constants; `capabilities` uses OpenAI slugs and Google labels
85 +* status: `limited-availability` vs schema `available`; `archived` undocumented; `groq_status` side channel
86 +* org kinds: `company|organization|lab|university` as entity types, with duplicates (`Kwaipilot` company + `kwaipilot-2` organization, `Upstage`/`upstage-2`, `StepFun`/`stepfun-ai`, `Thinking Machines`/`thinkingmachines`)
87 +* identifiers: `gemini_model_id` (provider key is `google-gemini-api`), `fireworks-ai_model_id` (hyphen), platform ids stored as claims
88 +* hardware `kind`: `soc|gpu|system|computer`; `memory_gb` int vs list
89 +* benchmarks: `metric` free text (`pass rate (2 attempts)`), `unit` `%` or empty
90 +
91 +## 10. Duplicate / hierarchy problems
92 +
93 +Model = release = checkpoint = quantization = provider alias today. Needed hierarchy: **model_family → model → artifact → deployment**.
94 +Artifact kinds: `checkpoint` (official weights repo), `quantization`, `conversion` (BF16/FP8 repack, MLX, ONNX), `packaging`.
95 +Evaluation-effort variants fold into the canonical model as **result configuration** (`reasoning_effort`, `reasoning`), never entities.
96 +
97 +## 11. Components to preserve
98 +
99 +Archive (`sdk/archive.py`), `FactWriter` temporal rules, `Resolver` precedence, connector SDK and all 27 connectors + 91 fixture tests,
100 +LLM gateway cascade and accounting, `/asof` `/history` `/diff` `/compare` `/prices/index` `/hardware/fit` endpoints and their semantics,
101 +`merge_entities`, quality methodology, design tokens, `KeyValue`/`DataTable`/`Tabs`/`ProvenanceInline`, compare tray, admin console,
102 +sitemap shards, OG frame, slugs, ids, all public URLs.
103 +
104 +## 12. Schema migration 0003 (`0003_canonical_ontology`)
105 +
106 +* `entities`: `family_id`, `canonical_id` (artifact → model, folded variant → model), `artifact_kind`, `identity_confidence`; new types
107 + `model_family` (prefix `family`), `artifact` (prefix `artifact`); `license` entities created from the ontology (`uses_license` relation).
108 +* facts: `run_id` on `claims`, `relations`, `prices`, `benchmark_results`, `change_events` (+ indexes) for batch rollback.
109 +* `change_events`: `recorded_at`, `is_backfill`, generated `occurred_at = coalesce(effective_at, observed_at)`, `group_key` (semantic
110 + dedupe of one release across documents), indexes on `(is_backfill, occurred_at)`.
111 +* `benchmark_results`: `trust_level`, `config_key` (comparability), `variant`, `run_group`, `is_current` maintained by the writer
112 + (one current row per model × benchmark × metric × config_key; older runs closed with `valid_to`).
113 +* new tables: `resolution_decisions`, `anomalies`, `admin_audit_log`, `quarantined_runs`, `taxonomy_mappings`, `watch_digests` (none), `model_families` view.
114 +* benchmark registry gains `family, variant, version, direction, metric_min, metric_max, harness, comparability` fields (claims).
115 +
116 +## 13. Backward compatibility
117 +
118 +* Every existing slug and id keeps resolving: artifacts stay reachable at `/models/<slug>` (API accepts `artifact` on the models mount,
119 + web 301s to `/artifacts/<slug>`); folded effort variants resolve through `merged_into` (API follows it, web 301s to the canonical model).
120 +* `/models` excludes artifacts and merged rows by default (`include=artifacts` restores the old universe); documented in `docs/API.md`
121 + as v1.1 with a changelog. No v2 needed: no field is removed or retyped; new fields are additive.
122 +* Raw snapshots are never modified; taxonomy normalisation writes canonical attributes and keeps `*_raw` when the source label differs.
123 +* Counters get explicit definitions ("Models = canonical model releases; artifacts and folded variants excluded").
124 +
125 +## 14. Implementation sequence
126 +
127 +1. Ontology package (`src/aiatlas/ontology/`) + migration 0003 + tests.
128 +2. SDK: writer/resolver use the ontology; `run_id` propagation; `first_seen_hint`; same-source loophole; SSRF guard; reprocess fix; quarantine.
129 +3. `aia canonicalize` (dry-run + apply): artifacts, effort variants, families, licences, taxonomy, org duplicates, event backfill flags,
130 + benchmark result comparability/trust; `aia anomalies`; `aia audit-data`.
131 +4. Connectors: HF → artifacts, AA → configs, leaderboards metrics split, benchmark registry families, OpenRouter fixes, provider ids.
132 +5. API 1.1: families/artifacts/deployments/frontier/pulse/pareto/finder/cost/matrix/history/comparability/time-machine/claims/admin quality
133 + & resolution & extraction debugger & audit; counters; backfill-aware feeds; search compiler v2; security fixes.
134 +6. Web: brand, shell (nav + ⌘K commands + evidence drawer + density), homepage 3.0, model page 3.0, compare 3.0, benchmarks 2.0,
135 + frontier, prices terminal, calculator, run-locally, find-a-model, open, graph, timeline 2.0, time machine, diff, pulse, families,
136 + watchlist, admin workbenches, mobile, OG per type, SEO.
137 +7. Tests, QA sweeps, final data/benchmark/event/counter audits, deploy, memory.
added migrations/versions/0003_canonical_ontology.py +188 −0
@@ -0,0 +1,188 @@
1 +"""Canonical ontology: model hierarchy columns (family / canonical / artifact kind), run ids on every fact for batch rollback,
2 +event semantics (recorded_at, occurred_at, is_backfill, group_key), benchmark comparability (config_key, trust_level, variant,
3 +run_group, is_current), persisted entity-resolution decisions, anomaly flags, admin audit log, quarantined runs, taxonomy mappings.
4 +
5 +Additive only — no column is dropped or retyped; existing rows keep their values.
6 +
7 +Revision ID: 0003
8 +"""
9 +from __future__ import annotations
10 +
11 +from alembic import op
12 +
13 +revision = "0003"
14 +down_revision = "0002"
15 +branch_labels = None
16 +depends_on = None
17 +
18 +SQL = r"""
19 +-- ---------------------------------------------------------------------------------------------- entity hierarchy
20 +alter table entities add column if not exists family_id text references entities(id);
21 +-- artifact → model; folded variant → model
22 +alter table entities add column if not exists canonical_id text references entities(id);
23 +-- checkpoint|quantization|conversion|packaging (artifacts only)
24 +alter table entities add column if not exists artifact_kind text;
25 +-- high|medium|low (how sure we are this row is one real thing)
26 +alter table entities add column if not exists identity_confidence text not null default 'high';
27 +create index if not exists entities_family_idx on entities (family_id) where family_id is not null;
28 +create index if not exists entities_canonical_idx on entities (canonical_id) where canonical_id is not null;
29 +create index if not exists entities_type_slug_idx on entities (entity_type, slug);
30 +create index if not exists entities_type_status_idx on entities (entity_type, status) where merged_into is null;
31 +
32 +-- ---------------------------------------------------------------------------------------------- run ids on facts (batch rollback)
33 +alter table claims add column if not exists run_id text;
34 +alter table relations add column if not exists run_id text;
35 +alter table prices add column if not exists run_id text;
36 +alter table benchmark_results add column if not exists run_id text;
37 +alter table change_events add column if not exists run_id text;
38 +create index if not exists claims_run_idx on claims (run_id) where run_id is not null;
39 +create index if not exists relations_run_idx on relations (run_id) where run_id is not null;
40 +create index if not exists prices_run_idx on prices (run_id) where run_id is not null;
41 +create index if not exists benchmark_results_run_idx on benchmark_results (run_id) where run_id is not null;
42 +create index if not exists change_events_run_idx on change_events (run_id) where run_id is not null;
43 +-- source label when the canonical value differs
44 +alter table claims add column if not exists value_raw text;
45 +
46 +-- ---------------------------------------------------------------------------------------------- event semantics
47 +alter table change_events add column if not exists recorded_at timestamptz not null default now();
48 +alter table change_events add column if not exists is_backfill boolean not null default false;
49 +-- semantic group (one release across documents)
50 +alter table change_events add column if not exists group_key text;
51 +alter table change_events add column if not exists occurred_at timestamptz generated always as (coalesce(effective_at, observed_at)) stored;
52 +create index if not exists change_events_live_idx on change_events (occurred_at desc) where is_backfill = false and event_type <> 'DOCUMENT_CHANGED';
53 +create index if not exists change_events_occurred_idx on change_events (occurred_at desc);
54 +create index if not exists change_events_group_idx on change_events (group_key) where group_key is not null;
55 +create index if not exists change_events_live_importance_idx on change_events (importance desc, occurred_at desc) where is_backfill = false;
56 +
57 +-- ---------------------------------------------------------------------------------------------- benchmark comparability & trust
58 +alter table benchmark_results add column if not exists config_key text;
59 +alter table benchmark_results add column if not exists trust_level text;
60 +alter table benchmark_results add column if not exists variant text;
61 +alter table benchmark_results add column if not exists run_group text;
62 +alter table benchmark_results add column if not exists is_current boolean not null default true;
63 +alter table benchmark_results add column if not exists extractor text not null default 'deterministic';
64 +create index if not exists benchmark_results_current2_idx on benchmark_results (benchmark_id, config_key, is_current) where valid_to is null;
65 +create index if not exists benchmark_results_model_metric_idx on benchmark_results (model_id, benchmark_id, metric) where valid_to is null;
66 +
67 +-- ---------------------------------------------------------------------------------------------- persisted resolution decisions
68 +create table if not exists resolution_decisions (
69 + id text primary key,
70 + a_id text not null references entities(id) on delete cascade,
71 + b_id text not null references entities(id) on delete cascade,
72 + decision text not null, -- merge|alias|variant_of|family_member|keep_separate|defer
73 + actor text not null default 'admin',
74 + note text,
75 + payload jsonb not null default '{}'::jsonb,
76 + applied boolean not null default false,
77 + created_at timestamptz not null default now(),
78 + unique (a_id, b_id, decision)
79 +);
80 +create index if not exists resolution_decisions_pair_idx on resolution_decisions (a_id, b_id);
81 +create index if not exists resolution_decisions_b_idx on resolution_decisions (b_id);
82 +
83 +-- ---------------------------------------------------------------------------------------------- anomaly flags (never deletes)
84 +create table if not exists anomalies (
85 + id text primary key,
86 + entity_id text references entities(id) on delete cascade,
87 + check_name text not null,
88 + severity text not null, -- critical|warning|info
89 + message text not null,
90 + value jsonb,
91 + detail jsonb not null default '{}'::jsonb,
92 + status text not null default 'open', -- open|resolved|ignored|fixed
93 + first_seen_at timestamptz not null default now(),
94 + last_seen_at timestamptz not null default now(),
95 + resolved_at timestamptz,
96 + resolution text,
97 + dedupe_key text unique
98 +);
99 +create index if not exists anomalies_status_idx on anomalies (status, severity, last_seen_at desc);
100 +create index if not exists anomalies_entity_idx on anomalies (entity_id);
101 +
102 +-- ---------------------------------------------------------------------------------------------- admin audit log
103 +create table if not exists admin_audit_log (
104 + id bigserial primary key,
105 + actor text not null default 'admin',
106 + action text not null,
107 + target text,
108 + payload jsonb not null default '{}'::jsonb,
109 + ip text,
110 + created_at timestamptz not null default now()
111 +);
112 +create index if not exists admin_audit_log_created_idx on admin_audit_log (created_at desc);
113 +
114 +-- ---------------------------------------------------------------------------------------------- quarantined runs (held facts awaiting review)
115 +create table if not exists quarantined_runs (
116 + id text primary key,
117 + run_id text,
118 + connector_name text not null,
119 + reason text not null,
120 + stats jsonb not null default '{}'::jsonb, -- baseline vs observed counts
121 + facts jsonb not null default '[]'::jsonb, -- serialised Facts objects, written on release
122 + status text not null default 'pending', -- pending|released|discarded
123 + created_at timestamptz not null default now(),
124 + resolved_at timestamptz,
125 + resolved_by text
126 +);
127 +create index if not exists quarantined_runs_status_idx on quarantined_runs (status, created_at desc);
128 +
129 +-- ---------------------------------------------------------------------------------------------- taxonomy mappings observed (raw → canonical)
130 +create table if not exists taxonomy_mappings (
131 + domain text not null, -- license|openness|modality|status|org_kind|hardware_kind|framework_kind|metric
132 + raw text not null,
133 + canonical text,
134 + count integer not null default 1,
135 + first_seen_at timestamptz not null default now(),
136 + last_seen_at timestamptz not null default now(),
137 + primary key (domain, raw)
138 +);
139 +
140 +-- ---------------------------------------------------------------------------------------------- connector run baselines (anomaly detector)
141 +alter table connector_runs add column if not exists quarantined boolean not null default false;
142 +alter table connector_runs add column if not exists baseline jsonb;
143 +-- rolling medians of records/prices/results
144 +alter table connectors add column if not exists baseline jsonb;
145 +
146 +-- ---------------------------------------------------------------------------------------------- search vector: include aliases of family
147 +create or replace function entities_search_update() returns trigger language plpgsql as $$
148 +begin
149 + new.search :=
150 + setweight(to_tsvector('simple', coalesce(new.canonical_name, '')), 'A') ||
151 + setweight(to_tsvector('simple', coalesce(new.attributes->>'family', '')), 'B') ||
152 + setweight(to_tsvector('simple', coalesce(new.attributes->>'api_model_id', '')), 'B') ||
153 + setweight(to_tsvector('simple', coalesce(new.entity_type, '')), 'C') ||
154 + setweight(to_tsvector('english', left(coalesce(new.description, ''), 4000)), 'C');
155 + new.updated_at := now();
156 + return new;
157 +end $$;
158 +"""
159 +
160 +
161 +def upgrade() -> None:
162 + for statement in _split(SQL):
163 + op.execute(statement)
164 +
165 +
166 +def downgrade() -> None:
167 + raise RuntimeError("AI Atlas migrations are forward-only: historical data is never disposable")
168 +
169 +
170 +def _split(sql: str) -> list[str]:
171 + out: list[str] = []
172 + buf: list[str] = []
173 + in_dollar = False
174 + for line in sql.splitlines():
175 + stripped = line.strip()
176 + if stripped.count("$$") % 2 == 1:
177 + in_dollar = not in_dollar
178 + buf.append(line)
179 + if not in_dollar and stripped.endswith(";"):
180 + body = [ln for ln in buf if not ln.strip().startswith("--") or in_dollar]
181 + stmt = "\n".join(body).strip()
182 + if stmt:
183 + out.append(stmt)
184 + buf = []
185 + tail = "\n".join(ln for ln in buf if not ln.strip().startswith("--")).strip()
186 + if tail:
187 + out.append(tail)
188 + return out
modified src/aiatlas/ids.py +2 −0
@@ -8,6 +8,8 @@ from ulid import ULID
8 8
9 9 PREFIXES: dict[str, str] = {
10 10 "model": "model",
11 + "model_family": "family", # Llama 4, Qwen3.6, Claude — groups model releases
12 + "artifact": "artifact", # checkpoint / quantisation / conversion of a model (never a model of its own)
11 13 "company": "company",
12 14 "organization": "org",
13 15 "researcher": "person",
added src/aiatlas/ontology/__init__.py +50 −0
@@ -0,0 +1,50 @@
1 +"""AI Atlas ontology — canonical vocabularies and deterministic normalisers.
2 +
3 +Everything here is pure Python (no database, no network): connectors, the writer, the API and the QA checks all import the same
4 +definitions so that "Apache 2.0", "apache-2.0" and "Apache-2.0" are one licence, "Text" and "text" one modality, and
5 +`unsloth/Qwen3.6-35B-A3B-GGUF` an *artifact* of the model `Qwen3.6-35B-A3B` rather than a new foundation model.
6 +
7 +Modules
8 + licenses SPDX-anchored licence ontology (permissions, restrictions, category) + `normalize_license`
9 + openness openness dimensions → carefully labelled category + `normalize_openness`
10 + taxonomy small enums (modalities, statuses, organization kinds, hardware kinds, framework kinds) + normalisers
11 + models model naming: quantisation/precision detection, evaluation-effort variants, family inference, base-name keys
12 + benchmarks benchmark families / variants / metric ranges + comparability of two result configurations
13 + anomalies deterministic sanity checks on entity attributes, prices and results (flags, never deletes)
14 +"""
15 +from aiatlas.ontology.licenses import LICENSES, LicenseInfo, normalize_license
16 +from aiatlas.ontology.openness import OPENNESS_CATEGORIES, derive_openness, normalize_openness, openness_dimensions
17 +from aiatlas.ontology.taxonomy import (
18 + FRAMEWORK_KINDS,
19 + HARDWARE_KINDS,
20 + MODALITIES,
21 + MODEL_STATUSES,
22 + ORG_KINDS,
23 + normalize_framework_kind,
24 + normalize_hardware_kind,
25 + normalize_modalities,
26 + normalize_modality,
27 + normalize_org_kind,
28 + normalize_status,
29 +)
30 +
31 +__all__ = [
32 + "FRAMEWORK_KINDS",
33 + "HARDWARE_KINDS",
34 + "LICENSES",
35 + "MODALITIES",
36 + "MODEL_STATUSES",
37 + "OPENNESS_CATEGORIES",
38 + "ORG_KINDS",
39 + "LicenseInfo",
40 + "derive_openness",
41 + "normalize_framework_kind",
42 + "normalize_hardware_kind",
43 + "normalize_license",
44 + "normalize_modalities",
45 + "normalize_modality",
46 + "normalize_openness",
47 + "normalize_org_kind",
48 + "normalize_status",
49 + "openness_dimensions",
50 +]
added src/aiatlas/ontology/anomalies.py +178 −0
@@ -0,0 +1,178 @@
1 +"""Deterministic sanity checks. Each check returns `Anomaly` records — flags with evidence, never deletions or silent fixes.
2 +
3 +Severity: `critical` (value is physically impossible or contradicts itself), `warning` (implausible, needs a look), `info` (worth
4 +knowing, e.g. a duplicate candidate). `dedupe_key` keeps one open row per (entity, check)."""
5 +from __future__ import annotations
6 +
7 +from dataclasses import dataclass, field
8 +from datetime import UTC, date, datetime
9 +from typing import Any
10 +
11 +from aiatlas.ontology.benchmarks import metric_bounds
12 +
13 +MAX_PARAMS = 10e12 # 10T
14 +MAX_CONTEXT = 100_000_000 # 100M tokens
15 +MAX_OUTPUT = 100_000_000
16 +MAX_PRICE_PER_MTOK = 5_000.0 # USD per 1M tokens
17 +MAX_MEMORY_GB = 100_000
18 +MIN_RELEASE = date(2010, 1, 1)
19 +
20 +
21 +@dataclass
22 +class Anomaly:
23 + check: str
24 + severity: str
25 + message: str
26 + entity_id: str | None = None
27 + value: Any = None
28 + detail: dict[str, Any] = field(default_factory=dict)
29 +
30 + @property
31 + def dedupe_key(self) -> str:
32 + return f"{self.check}:{self.entity_id or self.detail.get('key', '')}"
33 +
34 +
35 +def _num(v: Any) -> float | None:
36 + if isinstance(v, bool):
37 + return None
38 + if isinstance(v, (int, float)):
39 + return float(v)
40 + if isinstance(v, str):
41 + try:
42 + return float(v.replace(",", ""))
43 + except ValueError:
44 + return None
45 + return None
46 +
47 +
48 +def _date(v: Any) -> date | None:
49 + if isinstance(v, datetime):
50 + return v.date()
51 + if isinstance(v, date):
52 + return v
53 + if isinstance(v, str) and len(v) >= 4 and v[:4].isdigit():
54 + try:
55 + parts = v[:10].split("-")
56 + y = int(parts[0]); m = int(parts[1]) if len(parts) > 1 and parts[1].isdigit() else 1; d = int(parts[2]) if len(parts) > 2 and parts[2][:2].isdigit() else 1
57 + return date(y, max(1, min(12, m)), max(1, min(28, d)))
58 + except ValueError:
59 + return None
60 + return None
61 +
62 +
63 +def check_model(entity_id: str, name: str, attrs: dict[str, Any], *, today: date | None = None) -> list[Anomaly]:
64 + today = today or datetime.now(UTC).date()
65 + out: list[Anomaly] = []
66 + params = _num(attrs.get("parameter_count"))
67 + active = _num(attrs.get("active_parameter_count"))
68 + ctx = _num(attrs.get("context_length"))
69 + max_out = _num(attrs.get("max_output_tokens"))
70 + rel = _date(attrs.get("release_date"))
71 + dep = _date(attrs.get("deprecation_date"))
72 + ret = _date(attrs.get("retirement_date"))
73 + cutoff = _date(attrs.get("knowledge_cutoff"))
74 +
75 + if params is not None and params > MAX_PARAMS:
76 + out.append(Anomaly("params_too_large", "critical", f"{name}: parameter_count {params:.3g} exceeds {MAX_PARAMS:.0e}", entity_id, params))
77 + if params is not None and params < 1e5:
78 + out.append(Anomaly("params_too_small", "warning", f"{name}: parameter_count {params:.3g} below 100K", entity_id, params))
79 + if params is not None and active is not None and active > params:
80 + out.append(Anomaly("active_gt_total", "critical", f"{name}: active parameters {active:.3g} > total {params:.3g}", entity_id, active,
81 + {"parameter_count": params, "active_parameter_count": active}))
82 + if ctx is not None and ctx > MAX_CONTEXT:
83 + out.append(Anomaly("context_too_large", "critical", f"{name}: context_length {ctx:.0f} exceeds 100M tokens", entity_id, ctx))
84 + if ctx is not None and ctx < 256:
85 + out.append(Anomaly("context_too_small", "warning", f"{name}: context_length {ctx:.0f} below 256 tokens", entity_id, ctx))
86 + if max_out is not None and ctx is not None and max_out > ctx:
87 + out.append(Anomaly("max_output_gt_context", "warning", f"{name}: max_output_tokens {max_out:.0f} > context_length {ctx:.0f}", entity_id, max_out,
88 + {"context_length": ctx}))
89 + if rel and rel > today:
90 + out.append(Anomaly("release_in_future", "critical", f"{name}: release_date {rel.isoformat()} is in the future", entity_id, rel.isoformat()))
91 + if rel and rel < MIN_RELEASE:
92 + out.append(Anomaly("release_too_old", "warning", f"{name}: release_date {rel.isoformat()} before 2010", entity_id, rel.isoformat()))
93 + if rel and dep and dep < rel:
94 + out.append(Anomaly("deprecated_before_release", "critical", f"{name}: deprecation_date {dep} before release_date {rel}", entity_id, dep.isoformat(),
95 + {"release_date": rel.isoformat()}))
96 + if rel and ret and ret < rel:
97 + out.append(Anomaly("retired_before_release", "critical", f"{name}: retirement_date {ret} before release_date {rel}", entity_id, ret.isoformat(),
98 + {"release_date": rel.isoformat()}))
99 + if dep and ret and ret < dep:
100 + out.append(Anomaly("retired_before_deprecated", "warning", f"{name}: retirement_date {ret} before deprecation_date {dep}", entity_id, ret.isoformat()))
101 + if cutoff and rel and cutoff > rel:
102 + out.append(Anomaly("cutoff_after_release", "warning", f"{name}: knowledge_cutoff {cutoff} after release_date {rel}", entity_id, cutoff.isoformat()))
103 + status = str(attrs.get("status") or "").lower()
104 + if status in ("deprecated", "retired") and rel and rel > today:
105 + out.append(Anomaly("deprecated_future_release", "critical", f"{name}: status {status} but release in the future", entity_id, status))
106 + if attrs.get("openness") in ("open-weights", "open-source") and not any(attrs.get(k) for k in ("hf_repo", "repository_url", "model_card_url", "weights_url")):
107 + out.append(Anomaly("open_without_weights_url", "info", f"{name}: labelled {attrs.get('openness')} but no weights location recorded", entity_id, attrs.get("openness")))
108 + return out
109 +
110 +
111 +def check_price(row: dict[str, Any]) -> list[Anomaly]:
112 + out: list[Anomaly] = []
113 + key = f"{row.get('model_id')}:{row.get('provider_id')}:{row.get('provider_model_id') or ''}"
114 + label = f"{row.get('model_name') or row.get('model_id')} @ {row.get('provider_name') or row.get('provider_id')}"
115 + inp, outp = _num(row.get("input_per_mtok")), _num(row.get("output_per_mtok"))
116 + for k, v in (("input_per_mtok", inp), ("output_per_mtok", outp), ("cached_input_per_mtok", _num(row.get("cached_input_per_mtok"))),
117 + ("batch_input_per_mtok", _num(row.get("batch_input_per_mtok"))), ("batch_output_per_mtok", _num(row.get("batch_output_per_mtok")))):
118 + if v is not None and v < 0:
119 + out.append(Anomaly("negative_price", "critical", f"{label}: {k} is negative ({v})", row.get("model_id"), v, {"key": key, "field": k, "price_id": row.get("id")}))
120 + if v is not None and v > MAX_PRICE_PER_MTOK:
121 + out.append(Anomaly("price_too_high", "warning", f"{label}: {k} = ${v:g} per 1M tokens", row.get("model_id"), v, {"key": key, "field": k, "price_id": row.get("id")}))
122 + if inp is not None and outp is not None and outp == 0 and inp > 0 and not (row.get("features") or {}).get("free"):
123 + out.append(Anomaly("zero_output_price", "warning", f"{label}: output price is 0 while input is ${inp:g}", row.get("model_id"), 0, {"key": key, "price_id": row.get("id")}))
124 + if inp is not None and outp is not None and inp > 0 and outp > 0 and inp > outp * 4:
125 + out.append(Anomaly("input_gt_output_price", "info", f"{label}: input ${inp:g} is more than 4× output ${outp:g}", row.get("model_id"), inp, {"key": key, "price_id": row.get("id")}))
126 + cached = _num(row.get("cached_input_per_mtok"))
127 + if cached is not None and inp is not None and cached > inp:
128 + out.append(Anomaly("cached_gt_input_price", "warning", f"{label}: cached input ${cached:g} > input ${inp:g}", row.get("model_id"), cached, {"key": key, "price_id": row.get("id")}))
129 + return out
130 +
131 +
132 +def check_price_movement(old: dict[str, Any], new: dict[str, Any]) -> list[Anomaly]:
133 + out: list[Anomaly] = []
134 + for k in ("input_per_mtok", "output_per_mtok"):
135 + a, b = _num(old.get(k)), _num(new.get(k))
136 + if a and b and a > 0 and b > 0 and (b / a > 100 or a / b > 100):
137 + out.append(Anomaly("price_jump_100x", "critical", f"{k} moved {a:g} → {b:g} (>100×)", new.get("model_id"), b,
138 + {"key": f"{new.get('model_id')}:{new.get('provider_id')}:{k}", "old": a, "new": b}))
139 + return out
140 +
141 +
142 +def check_result(row: dict[str, Any]) -> list[Anomaly]:
143 + out: list[Anomaly] = []
144 + score = _num(row.get("score"))
145 + lo, hi = metric_bounds(row.get("metric"), row.get("unit"))
146 + label = f"{row.get('model_name') or row.get('model_id')} on {row.get('benchmark_name') or row.get('benchmark_id')}"
147 + if score is not None and hi is not None and score > hi + 1e-9:
148 + out.append(Anomaly("score_above_max", "critical", f"{label}: {score:g} > metric maximum {hi:g}", row.get("model_id"), score,
149 + {"key": row.get("id"), "result_id": row.get("id"), "metric": row.get("metric")}))
150 + if score is not None and lo is not None and score < lo - 1e-9:
151 + out.append(Anomaly("score_below_min", "critical", f"{label}: {score:g} < metric minimum {lo:g}", row.get("model_id"), score,
152 + {"key": row.get("id"), "result_id": row.get("id"), "metric": row.get("metric")}))
153 + ev = _date(row.get("evaluated_at"))
154 + rel = _date(row.get("model_release_date"))
155 + if ev and rel and ev < rel and (rel - ev).days > 45:
156 + out.append(Anomaly("evaluated_before_release", "warning", f"{label}: evaluated {ev} before model release {rel}", row.get("model_id"), ev.isoformat(),
157 + {"key": row.get("id"), "result_id": row.get("id")}))
158 + return out
159 +
160 +
161 +def check_hardware(entity_id: str, name: str, attrs: dict[str, Any]) -> list[Anomaly]:
162 + out: list[Anomaly] = []
163 + mem = attrs.get("memory_gb")
164 + mems = mem if isinstance(mem, list) else [mem]
165 + for m in mems:
166 + v = _num(m)
167 + if v is not None and (v <= 0 or v > MAX_MEMORY_GB):
168 + out.append(Anomaly("memory_implausible", "critical", f"{name}: memory_gb {v:g} implausible", entity_id, v))
169 + bw = _num(attrs.get("memory_bandwidth_gbs"))
170 + if bw is not None and (bw <= 0 or bw > 100_000):
171 + out.append(Anomaly("bandwidth_implausible", "warning", f"{name}: memory_bandwidth_gbs {bw:g} implausible", entity_id, bw))
172 + tdp = _num(attrs.get("tdp_watts"))
173 + if tdp is not None and (tdp <= 0 or tdp > 200_000):
174 + out.append(Anomaly("tdp_implausible", "warning", f"{name}: tdp_watts {tdp:g} implausible", entity_id, tdp))
175 + return out
176 +
177 +
178 +__all__ = ["Anomaly", "check_hardware", "check_model", "check_price", "check_price_movement", "check_result"]
added src/aiatlas/ontology/benchmarks.py +219 −0
@@ -0,0 +1,219 @@
1 +"""Benchmark ontology — families, variants, metric bounds, trust levels and the comparability of two result configurations.
2 +
3 +Two scores are only *directly comparable* when they measure the same benchmark variant with the same metric under configurations that
4 +do not change the task: same dataset revision/variant, same evaluator or harness class, same scaffold when the benchmark is agentic,
5 +same shot/pass regime. Reasoning effort, sampling temperature and judge differences make them *partially comparable*. Different
6 +variants (SWE-bench Verified vs Lite), different metrics, or different pass regimes are *not directly comparable*.
7 +"""
8 +from __future__ import annotations
9 +
10 +import hashlib
11 +import json
12 +import re
13 +from typing import Any
14 +
15 +COMPARABLE, PARTIAL, NOT_COMPARABLE = "comparable", "partially-comparable", "not-comparable"
16 +
17 +# Benchmark family → known variants (registry keys). Used to build `variant_of` relations and the family attribute; the registry YAML
18 +# carries the authoritative per-entry `family`/`variant` fields, this table is the fallback for benchmarks created from other sources.
19 +FAMILIES: dict[str, dict[str, Any]] = {
20 + "swe-bench": {"label": "SWE-bench", "variants": {"swe-bench-full": "full", "swe-bench-verified": "Verified", "swe-bench-lite": "Lite",
21 + "swe-bench-multimodal": "Multimodal", "swe-bench-multilingual": "Multilingual", "swe-bench-pro": "Pro"}},
22 + "aime": {"label": "AIME", "variants": {"aime-2024": "2024", "aime-2025": "2025", "aime-2026": "2026"}},
23 + "mmlu": {"label": "MMLU", "variants": {"mmlu": "original", "mmlu-pro": "Pro", "mmlu-redux": "Redux", "mmmlu": "multilingual"}},
24 + "mmmu": {"label": "MMMU", "variants": {"mmmu": "original", "mmmu-pro": "Pro"}},
25 + "gpqa": {"label": "GPQA", "variants": {"gpqa": "main", "gpqa-diamond": "Diamond"}},
26 + "tau-bench": {"label": "τ-bench", "variants": {"tau-bench": "v1", "tau2-bench": "τ²"}},
27 + "livebench": {"label": "LiveBench", "variants": {"livebench": "global", "livebench-2": "global"}},
28 + "arc-agi": {"label": "ARC-AGI", "variants": {"arc-agi": "1", "arc-agi-2": "2", "arc-agi-3": "3"}},
29 + "humaneval": {"label": "HumanEval", "variants": {"humaneval": "original", "humaneval-plus": "Plus", "mbpp": None}},
30 + "livecodebench": {"label": "LiveCodeBench", "variants": {"livecodebench": "rolling"}},
31 + "math": {"label": "MATH", "variants": {"math": "full", "math-500": "500"}},
32 + "terminal-bench": {"label": "Terminal-Bench", "variants": {"terminal-bench": "1.0", "terminal-bench-2": "2.0"}},
33 + "ifeval": {"label": "IFEval", "variants": {"ifeval": "original", "ifbench": "IFBench"}},
34 + "artificial-analysis-intelligence-index": {"label": "Artificial Analysis Intelligence Index", "variants": {"artificial-analysis-intelligence-index": "index"}},
35 + "humanitys-last-exam": {"label": "Humanity's Last Exam", "variants": {"humanitys-last-exam": "full"}},
36 + "scicode": {"label": "SciCode", "variants": {"scicode": "main"}},
37 + "aider-polyglot": {"label": "Aider polyglot", "variants": {"aider-polyglot": "polyglot"}},
38 + "lmarena": {"label": "LMArena", "variants": {"lmarena-text": "text", "lmarena-vision": "vision", "lmarena-webdev": "webdev"}},
39 + "mteb": {"label": "MTEB", "variants": {"mteb": "v1", "mteb-v2": "v2", "mmteb": "MMTEB"}},
40 +}
41 +_KEY_TO_FAMILY: dict[str, tuple[str, str | None]] = {k: (fam, v) for fam, spec in FAMILIES.items() for k, v in spec["variants"].items()}
42 +
43 +
44 +def family_of(benchmark_key: str) -> tuple[str | None, str | None]:
45 + """(family key, variant label) for a registry key / slug."""
46 + if benchmark_key in _KEY_TO_FAMILY:
47 + return _KEY_TO_FAMILY[benchmark_key]
48 + for fam in FAMILIES:
49 + if benchmark_key.startswith(fam):
50 + return fam, benchmark_key[len(fam):].strip("-") or None
51 + return None, None
52 +
53 +
54 +# ---------------------------------------------------------------------------------------------- metrics
55 +METRICS: dict[str, dict[str, Any]] = {
56 + # canonical metric → bounds and direction. `None` bound = unbounded.
57 + "accuracy": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
58 + "pass@1": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
59 + "pass^1": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
60 + "pass^k": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
61 + "resolved": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
62 + "pass_rate_2": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
63 + "percent_cases_well_formed": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
64 + "global_average": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
65 + "average score": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
66 + "mean score": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
67 + "index": {"min": 0, "max": 100, "higher_is_better": True, "unit": ""},
68 + "elo": {"min": 0, "max": None, "higher_is_better": True, "unit": ""},
69 + "score": {"min": None, "max": None, "higher_is_better": True, "unit": ""},
70 + "win_rate": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
71 + "f1": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
72 + "ndcg": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
73 + "perplexity": {"min": 0, "max": None, "higher_is_better": False, "unit": ""},
74 + "latency_ms": {"min": 0, "max": None, "higher_is_better": False, "unit": "ms"},
75 + "cost_usd": {"min": 0, "max": None, "higher_is_better": False, "unit": "USD"},
76 +}
77 +_METRIC_ALIASES = {
78 + "acc": "accuracy", "exact match": "accuracy", "em": "accuracy", "pass@1": "pass@1", "pass at 1": "pass@1", "pass^1": "pass^1", "pass1": "pass^1",
79 + "pass rate (2 attempts)": "pass_rate_2", "pass rate": "pass_rate_2", "percent resolved": "resolved", "% resolved": "resolved", "resolved rate": "resolved",
80 + "elo / bradley–terry score": "elo", "elo / bradley-terry score": "elo", "arena score": "elo", "bradley-terry": "elo", "mean score": "mean score",
81 + "average score": "average score", "global average": "global_average", "prompt-level strict accuracy": "accuracy", "strict accuracy": "accuracy",
82 + "intelligence index": "index", "composite": "index", "ppl": "perplexity",
83 +}
84 +
85 +
86 +def normalize_metric(raw: str | None) -> str | None:
87 + if not raw:
88 + return None
89 + s = raw.strip().lower()
90 + if s.startswith("category:"):
91 + return raw.strip() # LiveBench per-category averages keep their label; they are a separate metric each
92 + if s in METRICS:
93 + return s
94 + return _METRIC_ALIASES.get(s) or (s if re.fullmatch(r"[a-z0-9_@^%.\- ]+", s) else None)
95 +
96 +
97 +def metric_bounds(metric: str | None, unit: str | None = None) -> tuple[float | None, float | None]:
98 + m = normalize_metric(metric)
99 + if m and m.startswith("category:"):
100 + return 0, 100
101 + spec = METRICS.get(m or "")
102 + if spec:
103 + return spec["min"], spec["max"]
104 + if unit == "%":
105 + return 0, 100
106 + return None, None
107 +
108 +
109 +# ---------------------------------------------------------------------------------------------- trust levels
110 +TRUST_LEVELS = ("official-model-card", "official-benchmark", "peer-reviewed", "independent-evaluator", "community", "unverified")
111 +TRUST_LABELS = {
112 + "official-model-card": "Official model card / technical report (self-reported)",
113 + "official-benchmark": "Official benchmark leaderboard (submissions checked by the benchmark owner)",
114 + "peer-reviewed": "Peer-reviewed paper",
115 + "independent-evaluator": "Independent third-party evaluator",
116 + "community": "Community-run leaderboard or submission",
117 + "unverified": "Unverified / unknown provenance",
118 +}
119 +# source key (domain) → trust level of results it publishes
120 +SOURCE_TRUST: dict[str, str] = {
121 + "swebench.com": "official-benchmark", "aider.chat": "official-benchmark", "livebench.ai": "official-benchmark", "artificialanalysis.ai": "independent-evaluator",
122 + "lmarena.ai": "independent-evaluator", "scale.com": "independent-evaluator", "epoch.ai": "independent-evaluator", "vals.ai": "independent-evaluator",
123 + "huggingface.co": "community", "github.com": "community", "arxiv.org": "peer-reviewed", "openreview.net": "peer-reviewed",
124 +}
125 +OFFICIAL_LAB_SOURCES = {"docs.claude.com", "platform.openai.com", "ai.google.dev", "docs.mistral.ai", "api-docs.deepseek.com", "docs.cohere.com",
126 + "docs.x.ai", "ai.meta.com", "llama.com", "qwenlm.github.io", "developer.nvidia.com", "machinelearning.apple.com"}
127 +
128 +
129 +def trust_level(source_key: str | None, config: dict[str, Any] | None = None, *, extractor: str = "deterministic") -> str:
130 + cfg = config or {}
131 + if source_key in SOURCE_TRUST:
132 + level = SOURCE_TRUST[source_key]
133 + if level == "official-benchmark" and cfg.get("checked_by_swebench") is False:
134 + return "community"
135 + if level == "official-benchmark" and str(cfg.get("submission", "")).lower() in ("self-reported", "self reported", "unverified"):
136 + return "community"
137 + return level
138 + if source_key in OFFICIAL_LAB_SOURCES or cfg.get("self_reported") or cfg.get("source_kind") == "model_card":
139 + return "official-model-card"
140 + if extractor == "llm":
141 + return "unverified"
142 + return "unverified"
143 +
144 +
145 +# ---------------------------------------------------------------------------------------------- comparability
146 +# Config keys that change the *task* (must match for full comparability).
147 +TASK_KEYS = ("variant", "board", "dataset_revision", "harness", "evaluator", "index_version", "release", "version", "subset", "split", "shots",
148 + "pass_count", "attempts", "language", "scaffold", "agent", "system")
149 +# Config keys that change the *conditions* (mismatch → partially comparable).
150 +CONDITION_KEYS = ("reasoning_effort", "reasoning", "thinking_budget", "temperature", "judge", "tools", "tool_use", "max_tokens", "context_length",
151 + "sampling", "aggregation", "edit_format", "model_tag")
152 +# Keys that are pure bookkeeping (never affect comparability).
153 +IGNORED_KEYS = {"aa_slug", "livebench_model_id", "api_model_id", "date", "submission", "checked_by_swebench", "open_source_system", "system_org",
154 + "total_cost_usd", "cost_per_instance_usd", "dirname", "command", "versions", "test_cases", "seconds_per_case", "estimated",
155 + "livebench_hf_link", "source_kind", "self_reported", "subtasks", "notes", "url"}
156 +
157 +
158 +def _clean(v: Any) -> Any:
159 + if isinstance(v, str):
160 + return v.strip().lower()
161 + return v
162 +
163 +
164 +def config_key(config: dict[str, Any] | None, metric: str | None = None) -> str:
165 + """Stable hash of the comparability-relevant part of a result configuration (task keys + metric)."""
166 + cfg = config or {}
167 + core = {k: _clean(cfg[k]) for k in TASK_KEYS if k in cfg and cfg[k] not in (None, "", [], {})}
168 + core["metric"] = normalize_metric(metric) or ""
169 + return hashlib.sha1(json.dumps(core, sort_keys=True, default=str).encode()).hexdigest()[:12]
170 +
171 +
172 +def comparability(a_cfg: dict[str, Any] | None, b_cfg: dict[str, Any] | None, a_metric: str | None = None, b_metric: str | None = None,
173 + *, same_benchmark: bool = True) -> tuple[str, list[str]]:
174 + """Return (level, reasons). `same_benchmark` False → not comparable outright."""
175 + reasons: list[str] = []
176 + if not same_benchmark:
177 + return NOT_COMPARABLE, ["different benchmark variants"]
178 + ma, mb = normalize_metric(a_metric), normalize_metric(b_metric)
179 + if ma != mb and (ma or mb):
180 + return NOT_COMPARABLE, [f"different metrics ({a_metric} vs {b_metric})"]
181 + a, b = a_cfg or {}, b_cfg or {}
182 + for k in TASK_KEYS:
183 + if k in a or k in b:
184 + va, vb = _clean(a.get(k)), _clean(b.get(k))
185 + if va != vb and va not in (None, "") and vb not in (None, ""):
186 + reasons.append(f"{k}: {a.get(k)} vs {b.get(k)}")
187 + if reasons:
188 + return NOT_COMPARABLE, reasons
189 + for k in CONDITION_KEYS:
190 + if k in a or k in b:
191 + va, vb = _clean(a.get(k)), _clean(b.get(k))
192 + if va != vb:
193 + reasons.append(f"{k}: {a.get(k) if k in a else 'unspecified'} vs {b.get(k) if k in b else 'unspecified'}")
194 + if reasons:
195 + return PARTIAL, reasons
196 + return COMPARABLE, ["same variant, metric and evaluation conditions"]
197 +
198 +
199 +def variant_from_config(config: dict[str, Any] | None) -> str | None:
200 + cfg = config or {}
201 + for k in ("variant", "board", "subset", "split"):
202 + v = cfg.get(k)
203 + if isinstance(v, str) and v.strip():
204 + return v.strip()
205 + return None
206 +
207 +
208 +def run_group_from_config(config: dict[str, Any] | None) -> str | None:
209 + """The 'run' a result belongs to (a LiveBench release, an aider run date, an AA index version…). One current row per run group."""
210 + cfg = config or {}
211 + for k in ("release", "index_version", "version", "date", "dataset_revision"):
212 + v = cfg.get(k)
213 + if v not in (None, ""):
214 + return str(v)
215 + return None
216 +
217 +
218 +__all__ = ["COMPARABLE", "CONDITION_KEYS", "FAMILIES", "IGNORED_KEYS", "METRICS", "NOT_COMPARABLE", "PARTIAL", "TASK_KEYS", "TRUST_LABELS", "TRUST_LEVELS",
219 + "comparability", "config_key", "family_of", "metric_bounds", "normalize_metric", "run_group_from_config", "trust_level", "variant_from_config"]
added src/aiatlas/ontology/licenses.py +276 −0
@@ -0,0 +1,276 @@
1 +"""Licence ontology.
2 +
3 +A licence is identified by its SPDX id when one exists (`Apache-2.0`, `MIT`, `CC-BY-NC-4.0`) and by a stable AI Atlas key otherwise
4 +(`llama-3.1-community`, `gemma-terms`, `openrail-m`). Every entry states, as booleans (or `None` when the text is ambiguous), what the
5 +licence permits — so the UI can say "commercial use allowed / derivatives allowed / redistribution allowed / acceptable-use restrictions"
6 +instead of the meaningless "open source" for a model whose weights merely download.
7 +
8 +`normalize_license(raw)` maps the strings observed in the wild (Hugging Face `license:` tags, docs pages, GitHub) to a canonical key.
9 +Unknown strings return `None` — the raw label is kept separately by the writer (`license_raw`); nothing is guessed.
10 +"""
11 +from __future__ import annotations
12 +
13 +import re
14 +from dataclasses import dataclass, field
15 +
16 +# Categories are about *what the text allows*, not about ideology.
17 +CATEGORIES = ("permissive", "copyleft", "creative-commons", "responsible-ai", "community", "research-only", "proprietary", "unknown")
18 +
19 +
20 +@dataclass(frozen=True)
21 +class LicenseInfo:
22 + key: str # canonical key (SPDX id when applicable)
23 + label: str # display name
24 + category: str # one of CATEGORIES
25 + spdx: str | None = None # SPDX identifier when the licence is in the SPDX list
26 + url: str | None = None # canonical text
27 + commercial_use: bool | None = None # may be used commercially
28 + redistribution: bool | None = None # weights/code may be redistributed
29 + derivatives: bool | None = None # fine-tunes / modifications allowed
30 + hosting_restrictions: bool | None = None # restrictions on hosting the model as a service for third parties (incl. MAU caps)
31 + attribution: bool | None = None # attribution / notice required
32 + acceptable_use: bool | None = None # an acceptable-use policy restricts what the model may do
33 + osi_approved: bool = False # OSI-approved open-source licence (code) — never true for custom model licences
34 + weights_downloadable: bool = True # False for proprietary API-only terms
35 + aliases: tuple[str, ...] = field(default_factory=tuple)
36 +
37 + def as_dict(self) -> dict[str, object]:
38 + return {
39 + "key": self.key, "label": self.label, "category": self.category, "spdx": self.spdx, "url": self.url,
40 + "commercial_use": self.commercial_use, "redistribution": self.redistribution, "derivatives": self.derivatives,
41 + "hosting_restrictions": self.hosting_restrictions, "attribution": self.attribution, "acceptable_use": self.acceptable_use,
42 + "osi_approved": self.osi_approved, "weights_downloadable": self.weights_downloadable,
43 + }
44 +
45 +
46 +def _permissive(key: str, label: str, spdx: str, url: str, **kw: object) -> LicenseInfo:
47 + base: dict[str, object] = dict(key=key, label=label, category="permissive", spdx=spdx, url=url, commercial_use=True, redistribution=True,
48 + derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False, osi_approved=True)
49 + base.update(kw)
50 + return LicenseInfo(**base) # type: ignore[arg-type]
51 +
52 +
53 +LICENSES: dict[str, LicenseInfo] = {}
54 +
55 +
56 +def _add(info: LicenseInfo) -> None:
57 + LICENSES[info.key] = info
58 +
59 +
60 +# ---------------------------------------------------------------------------------------------- permissive / copyleft (OSI)
61 +_add(_permissive("Apache-2.0", "Apache License 2.0", "Apache-2.0", "https://www.apache.org/licenses/LICENSE-2.0",
62 + aliases=("apache 2.0", "apache-2", "apache2", "apache license 2.0", "apache license, version 2.0", "apache")))
63 +_add(_permissive("MIT", "MIT License", "MIT", "https://opensource.org/license/mit", aliases=("mit license", "the mit license")))
64 +_add(LicenseInfo(key="MIT-Modified", label="Modified MIT License", category="permissive", spdx=None, commercial_use=True, redistribution=True,
65 + derivatives=True, hosting_restrictions=None, attribution=True, acceptable_use=None, aliases=("modified mit", "mit-modified")))
66 +_add(_permissive("BSD-3-Clause", "BSD 3-Clause License", "BSD-3-Clause", "https://opensource.org/license/bsd-3-clause",
67 + aliases=("bsd-3", "bsd 3-clause", "bsd3", "new bsd", "modified bsd", "bsd")))
68 +_add(_permissive("BSD-2-Clause", "BSD 2-Clause License", "BSD-2-Clause", "https://opensource.org/license/bsd-2-clause", aliases=("bsd-2", "simplified bsd")))
69 +_add(_permissive("ISC", "ISC License", "ISC", "https://opensource.org/license/isc-license-txt"))
70 +_add(_permissive("Unlicense", "The Unlicense", "Unlicense", "https://unlicense.org", attribution=False, aliases=("unlicense",)))
71 +_add(_permissive("0BSD", "Zero-Clause BSD", "0BSD", "https://opensource.org/license/0bsd", attribution=False))
72 +_add(LicenseInfo(key="GPL-3.0", label="GNU GPL v3", category="copyleft", spdx="GPL-3.0-only", url="https://www.gnu.org/licenses/gpl-3.0.html",
73 + commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False,
74 + osi_approved=True, aliases=("gpl-3.0-only", "gpl-3.0-or-later", "gplv3", "gpl3", "gpl-3")))
75 +_add(LicenseInfo(key="GPL-2.0", label="GNU GPL v2", category="copyleft", spdx="GPL-2.0-only", url="https://www.gnu.org/licenses/old-licenses/gpl-2.0.html",
76 + commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False,
77 + osi_approved=True, aliases=("gpl-2.0-only", "gpl-2.0-or-later", "gplv2", "gpl2", "gpl-2")))
78 +_add(LicenseInfo(key="LGPL-3.0", label="GNU LGPL v3", category="copyleft", spdx="LGPL-3.0-only", url="https://www.gnu.org/licenses/lgpl-3.0.html",
79 + commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False,
80 + osi_approved=True, aliases=("lgpl-3.0-only", "lgpl-3.0-or-later", "lgplv3", "lgpl")))
81 +_add(LicenseInfo(key="AGPL-3.0", label="GNU AGPL v3", category="copyleft", spdx="AGPL-3.0-only", url="https://www.gnu.org/licenses/agpl-3.0.html",
82 + commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=True, attribution=True, acceptable_use=False,
83 + osi_approved=True, aliases=("agpl-3.0-only", "agpl-3.0-or-later", "agplv3", "agpl")))
84 +_add(LicenseInfo(key="MPL-2.0", label="Mozilla Public License 2.0", category="copyleft", spdx="MPL-2.0", url="https://www.mozilla.org/MPL/2.0/",
85 + commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False,
86 + osi_approved=True, aliases=("mpl", "mozilla public license 2.0")))
87 +_add(LicenseInfo(key="EPL-2.0", label="Eclipse Public License 2.0", category="copyleft", spdx="EPL-2.0", url="https://www.eclipse.org/legal/epl-2.0/",
88 + commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False, osi_approved=True))
89 +
90 +# ---------------------------------------------------------------------------------------------- Creative Commons
91 +def _cc(key: str, label: str, url: str, *, nc: bool, nd: bool, sa: bool, aliases: tuple[str, ...]) -> LicenseInfo:
92 + return LicenseInfo(key=key, label=label, category="creative-commons", spdx=key, url=url, commercial_use=not nc, redistribution=True,
93 + derivatives=not nd, hosting_restrictions=nc or None, attribution=True, acceptable_use=False, osi_approved=False, aliases=aliases)
94 +
95 +
96 +_add(_cc("CC-BY-4.0", "Creative Commons Attribution 4.0", "https://creativecommons.org/licenses/by/4.0/", nc=False, nd=False, sa=False,
97 + aliases=("cc by 4.0", "cc-by", "cc by", "cc-by-4", "creative commons attribution 4.0")))
98 +_add(_cc("CC-BY-SA-4.0", "Creative Commons Attribution-ShareAlike 4.0", "https://creativecommons.org/licenses/by-sa/4.0/", nc=False, nd=False, sa=True,
99 + aliases=("cc by-sa 4.0", "cc-by-sa", "cc by-sa")))
100 +_add(_cc("CC-BY-NC-4.0", "Creative Commons Attribution-NonCommercial 4.0", "https://creativecommons.org/licenses/by-nc/4.0/", nc=True, nd=False, sa=False,
101 + aliases=("cc by-nc 4.0", "cc-by-nc", "cc by-nc", "cc-by-nc-4", "creative commons attribution non commercial 4.0")))
102 +_add(_cc("CC-BY-NC-SA-4.0", "Creative Commons Attribution-NonCommercial-ShareAlike 4.0", "https://creativecommons.org/licenses/by-nc-sa/4.0/", nc=True, nd=False, sa=True,
103 + aliases=("cc by-nc-sa 4.0", "cc-by-nc-sa", "cc by-nc-sa")))
104 +_add(_cc("CC-BY-NC-ND-4.0", "Creative Commons Attribution-NonCommercial-NoDerivatives 4.0", "https://creativecommons.org/licenses/by-nc-nd/4.0/", nc=True, nd=True, sa=False,
105 + aliases=("cc by-nc-nd 4.0", "cc-by-nc-nd")))
106 +_add(_cc("CC-BY-ND-4.0", "Creative Commons Attribution-NoDerivatives 4.0", "https://creativecommons.org/licenses/by-nd/4.0/", nc=False, nd=True, sa=False,
107 + aliases=("cc by-nd 4.0", "cc-by-nd")))
108 +_add(LicenseInfo(key="CC0-1.0", label="Creative Commons Zero 1.0 (public domain dedication)", category="creative-commons", spdx="CC0-1.0",
109 + url="https://creativecommons.org/publicdomain/zero/1.0/", commercial_use=True, redistribution=True, derivatives=True,
110 + hosting_restrictions=False, attribution=False, acceptable_use=False, aliases=("cc0", "cc0 1.0", "public domain")))
111 +_add(LicenseInfo(key="ODC-By-1.0", label="Open Data Commons Attribution", category="creative-commons", spdx="ODC-By-1.0", url="https://opendatacommons.org/licenses/by/1-0/",
112 + commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False, aliases=("odc-by",)))
113 +_add(LicenseInfo(key="PDDL-1.0", label="Open Data Commons Public Domain Dedication", category="creative-commons", spdx="PDDL-1.0", url="https://opendatacommons.org/licenses/pddl/",
114 + commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=False, acceptable_use=False, aliases=("pddl",)))
115 +
116 +# ---------------------------------------------------------------------------------------------- responsible-AI (RAIL) family
117 +def _rail(key: str, label: str, url: str, aliases: tuple[str, ...]) -> LicenseInfo:
118 + return LicenseInfo(key=key, label=label, category="responsible-ai", spdx=None, url=url, commercial_use=True, redistribution=True, derivatives=True,
119 + hosting_restrictions=False, attribution=True, acceptable_use=True, osi_approved=False, aliases=aliases)
120 +
121 +
122 +_add(_rail("OpenRAIL-M", "Open RAIL-M", "https://www.licenses.ai/ai-licenses", ("openrail", "openrail-m", "open rail-m", "creativeml-openrail-m", "creativeml openrail-m")))
123 +_add(_rail("OpenRAIL++-M", "Open RAIL++-M (Stable Diffusion XL)", "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md", ("openrail++", "openrail++-m")))
124 +_add(_rail("BigScience-BLOOM-RAIL-1.0", "BigScience BLOOM RAIL 1.0", "https://huggingface.co/spaces/bigscience/license", ("bigscience-bloom-rail-1.0", "bloom-rail", "bigscience-openrail-m")))
125 +_add(_rail("BigCode-OpenRAIL-M", "BigCode Open RAIL-M", "https://www.bigcode-project.org/docs/pages/bigcode-openrail/", ("bigcode-openrail-m",)))
126 +
127 +# ---------------------------------------------------------------------------------------------- community / custom model licences (weights download, restrictions apply)
128 +def _community(key: str, label: str, url: str, aliases: tuple[str, ...], *, commercial: bool | None = True, hosting: bool | None = True,
129 + derivatives: bool | None = True, aup: bool | None = True) -> LicenseInfo:
130 + return LicenseInfo(key=key, label=label, category="community", spdx=None, url=url, commercial_use=commercial, redistribution=True, derivatives=derivatives,
131 + hosting_restrictions=hosting, attribution=True, acceptable_use=aup, osi_approved=False, aliases=aliases)
132 +
133 +
134 +_add(_community("Llama-2-Community", "Llama 2 Community License", "https://ai.meta.com/llama/license/", ("llama2", "llama-2", "llama 2 community license", "llama2 community license")))
135 +_add(_community("Llama-3-Community", "Llama 3 Community License", "https://llama.meta.com/llama3/license/", ("llama3", "llama-3", "llama 3 community license", "meta llama 3 community license")))
136 +_add(_community("Llama-3.1-Community", "Llama 3.1 Community License", "https://llama.meta.com/llama3_1/license/", ("llama3.1", "llama-3.1", "llama 3.1 community license")))
137 +_add(_community("Llama-3.2-Community", "Llama 3.2 Community License", "https://www.llama.com/llama3_2/license/", ("llama3.2", "llama-3.2", "llama 3.2 community license")))
138 +_add(_community("Llama-3.3-Community", "Llama 3.3 Community License", "https://www.llama.com/llama3_3/license/", ("llama3.3", "llama-3.3", "llama 3.3 community license")))
139 +_add(_community("Llama-4-Community", "Llama 4 Community License", "https://www.llama.com/llama4/license/", ("llama4", "llama-4", "llama 4 community license")))
140 +_add(_community("Gemma-Terms", "Gemma Terms of Use", "https://ai.google.dev/gemma/terms", ("gemma", "gemma terms of use", "gemma license", "gemma-terms")))
141 +_add(_community("Mistral-Research", "Mistral AI Research License", "https://mistral.ai/licenses/MRL-0.1.md", ("mrl", "mistral research license", "mistral ai research license", "mrl-0.1"),
142 + commercial=False, hosting=True, aup=True))
143 +_add(_community("Mistral-Non-Production", "Mistral AI Non-Production License", "https://mistral.ai/licenses/MNPL-0.1.md", ("mnpl", "mnpl-0.1", "mistral ai non-production license"),
144 + commercial=False, hosting=True, aup=True))
145 +_add(_community("Qwen-Research", "Qwen Research License", "https://huggingface.co/Qwen/Qwen2.5-3B/blob/main/LICENSE", ("qwen research license", "qwen-research"), commercial=False))
146 +_add(_community("Tongyi-Qianwen", "Tongyi Qianwen License", "https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT", ("tongyi-qianwen", "tongyi qianwen license", "qianwen"),
147 + commercial=True, hosting=True))
148 +_add(_community("DeepSeek-Model", "DeepSeek Model License", "https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/LICENSE-MODEL", ("deepseek", "deepseek license", "deepseek model license"), hosting=False))
149 +_add(_community("NVIDIA-Open-Model", "NVIDIA Open Model License", "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/",
150 + ("nvidia open model license", "nvidia-open-model-license", "nvidia open model"), hosting=False))
151 +_add(_community("NVIDIA-Community-Model", "NVIDIA Community Model License", "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-community-models-license/",
152 + ("nvidia community model license", "nvidia-community-model-license"), hosting=None))
153 +_add(_community("Apple-AMLR", "Apple Sample Code / ML Research License", "https://github.com/apple/ml-fastvlm/blob/main/LICENSE_MODEL", ("apple-amlr", "apple-ascl", "apple ml research license"),
154 + commercial=False, hosting=True, derivatives=True))
155 +_add(_community("Cohere-CC-BY-NC", "Cohere Labs Acceptable Use (CC-BY-NC)", "https://cohere.com/c4ai-cc-by-nc-license", ("cohere", "c4ai", "cc-by-nc-4.0 + acceptable use"), commercial=False))
156 +_add(_community("Stability-Community", "Stability AI Community License", "https://stability.ai/community-license-agreement", ("stabilityai-ai-community", "stability ai community license", "stabilityai-community"),
157 + commercial=True, hosting=True))
158 +_add(_community("Stability-Non-Commercial-Research", "Stability AI Non-Commercial Research Community License", "https://stability.ai/license",
159 + ("stabilityai-nc-research-community", "stability ai non-commercial research community"), commercial=False))
160 +_add(_community("Falcon-LLM", "Falcon LLM License (TII)", "https://falconllm.tii.ae/falcon-terms-and-conditions.html", ("falcon-llm-license", "tii falcon license", "falcon"), hosting=None))
161 +_add(_community("Yi", "Yi Series Models Community License", "https://github.com/01-ai/Yi/blob/main/MODEL_LICENSE_AGREEMENT.txt", ("yi-license", "yi license", "yi"), commercial=True))
162 +_add(_community("ChatGLM", "ChatGLM / GLM Model License", "https://github.com/THUDM/ChatGLM3/blob/main/MODEL_LICENSE", ("glm-4", "chatglm", "glm license", "zhipu"), commercial=True))
163 +_add(_community("Phi", "Microsoft Phi (MIT with terms)", "https://huggingface.co/microsoft/phi-4/blob/main/LICENSE", ("phi",), commercial=True, hosting=False))
164 +_add(_community("LFM-Open-1.0", "LFM Open License v1.0 (Liquid AI)", "https://www.liquid.ai/lfm-license", ("lfm1.0", "lfm-1.0", "lfm open license"), commercial=True, hosting=True))
165 +_add(_community("Jamba-Open-Model", "Jamba Open Model License (AI21)", "https://www.ai21.com/jamba-open-model-license/", ("jamba open model license", "jamba"), commercial=True))
166 +_add(_community("Kimi-Modified-MIT", "Kimi Modified MIT License (Moonshot)", "https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/LICENSE", ("kimi modified mit", "moonshot modified mit"),
167 + commercial=True, hosting=True, aup=False))
168 +_add(_community("SEED-Model", "ByteDance Seed Model License", "https://github.com/ByteDance-Seed", ("seed license", "bytedance seed"), commercial=None))
169 +_add(_community("Hunyuan-Community", "Tencent Hunyuan Community License", "https://github.com/Tencent/Tencent-Hunyuan-Large/blob/main/License.docx", ("tencent-hunyuan-community", "hunyuan", "tencent hunyuan community license"),
170 + commercial=True, hosting=True))
171 +_add(_community("IBM-Granite", "IBM Granite (Apache-2.0 with usage terms)", "https://www.ibm.com/granite", ("granite",), commercial=True, hosting=False, aup=True))
172 +_add(_community("Custom-Community", "Custom community licence (see source)", "", ("custom community", "community license", "other-community"), commercial=None, hosting=None, derivatives=None, aup=None))
173 +
174 +# ---------------------------------------------------------------------------------------------- research-only / non-commercial custom
175 +_add(LicenseInfo(key="Research-Only", label="Research / non-commercial licence (custom)", category="research-only", spdx=None, commercial_use=False, redistribution=None,
176 + derivatives=None, hosting_restrictions=True, attribution=True, acceptable_use=True, aliases=("research", "research only", "research-only", "non-commercial", "noncommercial",
177 + "academic", "academic use only", "non commercial")))
178 +_add(LicenseInfo(key="Meta-Research", label="Meta Research License", category="research-only", spdx=None, url="https://ai.meta.com/resources/models-and-libraries/",
179 + commercial_use=False, redistribution=False, derivatives=True, hosting_restrictions=True, attribution=True, acceptable_use=True,
180 + aliases=("meta research license", "fair noncommercial research license", "fair-noncommercial")))
181 +
182 +# ---------------------------------------------------------------------------------------------- proprietary
183 +_add(LicenseInfo(key="Proprietary", label="Proprietary (API / terms of service)", category="proprietary", spdx=None, commercial_use=None, redistribution=False,
184 + derivatives=False, hosting_restrictions=True, attribution=None, acceptable_use=True, weights_downloadable=False,
185 + aliases=("proprietary", "closed", "closed source", "commercial", "terms of service", "tos", "api terms", "license: proprietary")))
186 +_add(LicenseInfo(key="Other", label="Other (unclassified licence)", category="unknown", spdx=None, aliases=("other", "unknown", "custom", "see model card", "license: other")))
187 +
188 +
189 +# ---------------------------------------------------------------------------------------------- normalisation
190 +_ALIAS_INDEX: dict[str, str] = {}
191 +for _info in LICENSES.values():
192 + _ALIAS_INDEX[_info.key.lower()] = _info.key
193 + if _info.spdx:
194 + _ALIAS_INDEX[_info.spdx.lower()] = _info.key
195 + for _a in _info.aliases:
196 + _ALIAS_INDEX[_a.lower()] = _info.key
197 +
198 +_STRIP = re.compile(r"\s+|[_]")
199 +_PATTERNS: list[tuple[re.Pattern[str], str]] = [
200 + (re.compile(r"^apache[\s\-]*(license)?[\s\-,]*(version)?[\s\-]*2(\.0)?$", re.I), "Apache-2.0"),
201 + (re.compile(r"^llama[\s\-]*4", re.I), "Llama-4-Community"),
202 + (re.compile(r"^llama[\s\-]*3[.\-]3", re.I), "Llama-3.3-Community"),
203 + (re.compile(r"^llama[\s\-]*3[.\-]2", re.I), "Llama-3.2-Community"),
204 + (re.compile(r"^llama[\s\-]*3[.\-]1", re.I), "Llama-3.1-Community"),
205 + (re.compile(r"^(meta[\s\-]*)?llama[\s\-]*3", re.I), "Llama-3-Community"),
206 + (re.compile(r"^(meta[\s\-]*)?llama[\s\-]*2", re.I), "Llama-2-Community"),
207 + (re.compile(r"^gemma", re.I), "Gemma-Terms"),
208 + (re.compile(r"^cc[\s\-]*by[\s\-]*nc[\s\-]*sa", re.I), "CC-BY-NC-SA-4.0"),
209 + (re.compile(r"^cc[\s\-]*by[\s\-]*nc[\s\-]*nd", re.I), "CC-BY-NC-ND-4.0"),
210 + (re.compile(r"^cc[\s\-]*by[\s\-]*nc", re.I), "CC-BY-NC-4.0"),
211 + (re.compile(r"^cc[\s\-]*by[\s\-]*sa", re.I), "CC-BY-SA-4.0"),
212 + (re.compile(r"^cc[\s\-]*by[\s\-]*nd", re.I), "CC-BY-ND-4.0"),
213 + (re.compile(r"^cc[\s\-]*by", re.I), "CC-BY-4.0"),
214 + (re.compile(r"^cc0", re.I), "CC0-1.0"),
215 + (re.compile(r"^(creativeml[\s\-]*)?openrail\+\+", re.I), "OpenRAIL++-M"),
216 + (re.compile(r"^(creativeml[\s\-]*)?openrail", re.I), "OpenRAIL-M"),
217 + (re.compile(r"^bigscience", re.I), "BigScience-BLOOM-RAIL-1.0"),
218 + (re.compile(r"^bigcode", re.I), "BigCode-OpenRAIL-M"),
219 + (re.compile(r"^(gpl|gnu general public license)[\s\-]*v?3", re.I), "GPL-3.0"),
220 + (re.compile(r"^(gpl|gnu general public license)[\s\-]*v?2", re.I), "GPL-2.0"),
221 + (re.compile(r"^agpl", re.I), "AGPL-3.0"),
222 + (re.compile(r"^lgpl", re.I), "LGPL-3.0"),
223 + (re.compile(r"^bsd[\s\-]*3", re.I), "BSD-3-Clause"),
224 + (re.compile(r"^bsd[\s\-]*2", re.I), "BSD-2-Clause"),
225 + (re.compile(r"^mit\b", re.I), "MIT"),
226 + (re.compile(r"^mistral.*research", re.I), "Mistral-Research"),
227 + (re.compile(r"^mistral.*non[\s\-]*production", re.I), "Mistral-Non-Production"),
228 + (re.compile(r"^nvidia.*open.*model", re.I), "NVIDIA-Open-Model"),
229 + (re.compile(r"^nvidia.*community", re.I), "NVIDIA-Community-Model"),
230 + (re.compile(r"^deepseek", re.I), "DeepSeek-Model"),
231 + (re.compile(r"^qwen.*research", re.I), "Qwen-Research"),
232 + (re.compile(r"^tongyi", re.I), "Tongyi-Qianwen"),
233 + (re.compile(r"^stability.*(nc|non[\s\-]*commercial)", re.I), "Stability-Non-Commercial-Research"),
234 + (re.compile(r"^stability", re.I), "Stability-Community"),
235 + (re.compile(r"^apple", re.I), "Apple-AMLR"),
236 + (re.compile(r"^(tencent[\s\-]*)?hunyuan", re.I), "Hunyuan-Community"),
237 + (re.compile(r"^lfm", re.I), "LFM-Open-1.0"),
238 + (re.compile(r"^jamba", re.I), "Jamba-Open-Model"),
239 + (re.compile(r"^falcon", re.I), "Falcon-LLM"),
240 + (re.compile(r"^yi\b", re.I), "Yi"),
241 + (re.compile(r"(research|academic|non[\s\-]*commercial|noncommercial)", re.I), "Research-Only"),
242 + (re.compile(r"(proprietary|closed|terms of service)", re.I), "Proprietary"),
243 +]
244 +
245 +
246 +def normalize_license(raw: str | None) -> str | None:
247 + """Map an observed licence label to a canonical key, or `None` when it cannot be classified (keep the raw string separately)."""
248 + if raw is None:
249 + return None
250 + s = str(raw).strip()
251 + if not s:
252 + return None
253 + low = s.lower().strip().strip(".")
254 + low = re.sub(r"^license:\s*", "", low)
255 + if low in _ALIAS_INDEX:
256 + return _ALIAS_INDEX[low]
257 + compact = _STRIP.sub("-", low).replace("--", "-")
258 + if compact in _ALIAS_INDEX:
259 + return _ALIAS_INDEX[compact]
260 + for pattern, key in _PATTERNS:
261 + if pattern.search(low):
262 + return key
263 + return None
264 +
265 +
266 +def license_info(key_or_raw: str | None) -> LicenseInfo | None:
267 + key = key_or_raw if key_or_raw in LICENSES else normalize_license(key_or_raw)
268 + return LICENSES.get(key) if key else None
269 +
270 +
271 +def license_label(key_or_raw: str | None) -> str | None:
272 + info = license_info(key_or_raw)
273 + return info.label if info else (str(key_or_raw) if key_or_raw else None)
274 +
275 +
276 +__all__ = ["CATEGORIES", "LICENSES", "LicenseInfo", "license_info", "license_label", "normalize_license"]
added src/aiatlas/ontology/models.py +271 −0
@@ -0,0 +1,271 @@
1 +"""Model identity ontology — the rules that separate a MODEL (release) from its ARTIFACTS (checkpoints, conversions, quantisations),
2 +its EVALUATION VARIANTS (reasoning-effort settings of the same weights) and its FAMILY.
3 +
4 + MODEL FAMILY Llama 4 · Qwen3.6 · Claude · Gemini
5 + └ MODEL Llama 4 Maverick · Qwen3.6-35B-A3B · Claude Fable 5.1
6 + └ ARTIFACT meta-llama/Llama-4-Maverick-17B-128E-Instruct (official checkpoint) · unsloth/…-GGUF (third-party quantisation)
7 + └ DEPLOYMENT OpenRouter meta-llama/llama-4-maverick · Together meta-llama/Llama-4-Maverick (= prices rows)
8 +
9 +Everything here is deterministic string analysis. It never *asserts* a relation on its own: the resolution service uses these
10 +signals together with identifiers, organisations, `base_model` metadata and dates, and parks anything ambiguous in the review queue.
11 +"""
12 +from __future__ import annotations
13 +
14 +import re
15 +from dataclasses import dataclass, field
16 +
17 +# ---------------------------------------------------------------------------------------------- quantisation / precision formats
18 +QUANT_FORMATS: dict[str, str] = {
19 + # token (lowercase, matched on word boundaries) → canonical format
20 + "gguf": "gguf", "ggml": "gguf", "q2_k": "gguf", "q3_k_m": "gguf", "q4_k_m": "gguf", "q4_k_s": "gguf", "q5_k_m": "gguf", "q6_k": "gguf", "q8_0": "gguf",
21 + "q4_0": "gguf", "q4_1": "gguf", "q5_0": "gguf", "iq4_xs": "gguf", "iq3_m": "gguf", "iq2_m": "gguf", "ud-q4_k_xl": "gguf",
22 + "awq": "awq", "w4a16": "awq", "gptq": "gptq", "exl2": "exl2", "exl3": "exl3", "mlx": "mlx", "onnx": "onnx", "openvino": "openvino", "coreml": "coreml",
23 + "tensorrt": "tensorrt", "trt": "tensorrt", "trt-llm": "tensorrt", "bnb": "bnb", "bitsandbytes": "bnb", "nf4": "bnb", "4bit": "int4", "8bit": "int8",
24 + "int4": "int4", "int8": "int8", "w8a8": "int8", "w8a16": "int8", "fp8": "fp8", "fp8-kv": "fp8", "e4m3": "fp8", "nvfp4": "nvfp4", "mxfp4": "mxfp4",
25 + "mxfp8": "mxfp8", "fp4": "fp4", "quark": "quark", "quantized": "quantized", "quant": "quantized", "qat": "qat", "hqq": "hqq", "aqlm": "aqlm", "eetq": "eetq",
26 + "smoothquant": "smoothquant", "compressed-tensors": "compressed-tensors", "marlin": "gptq", "gptq-int4": "gptq", "autoround": "autoround",
27 + "2bit": "int2", "3bit": "int3", "5bit": "int5", "6bit": "int6", "2-bit": "int2", "3-bit": "int3", "4-bit": "int4", "5-bit": "int5", "6-bit": "int6", "8-bit": "int8",
28 +}
29 +# Full-precision dtype tokens: a "-BF16" or "-FP16" repo is a *conversion/packaging* of the same weights, not a quantisation.
30 +PRECISION_FORMATS = {"bf16", "fp16", "fp32", "f16", "f32", "float16", "bfloat16", "half"}
31 +ARTIFACT_PACKAGING = {"safetensors", "pytorch", "pth", "ckpt", "jax", "flax", "tf", "tflite", "litert", "gguf", "mlx", "onnx", "coreml", "openvino", "tensorrt"}
32 +
33 +# Third-party organisations that (almost) only publish conversions/quantisations of other people's models.
34 +CONVERTER_ORGS = {
35 + "unsloth", "bartowski", "mlx-community", "thebloke", "lmstudio-community", "qwen-community", "mradermacher", "quantfactory", "nousresearch-quant",
36 + "mistral-community", "turboderp", "casperhansen", "neuralmagic", "redhatai", "amd", "intel", "nvidia-quant", "ggml-org", "second-state",
37 + "mlx-vision", "cortexso", "gaianet", "bunnycore", "dranger003", "ubergarm", "anthracite-org", "modelcloud", "jinaai-quant", "ai-forever-quant",
38 + "lmstudio", "ollama", "ggerganov", "mlc-ai", "onnx-community", "onnxmodelzoo", "kaitchup", "thedrummer-quant",
39 +}
40 +
41 +# ---------------------------------------------------------------------------------------------- evaluation-effort variants (same weights, different setting)
42 +# Suffixes appended by evaluators (Artificial Analysis, LiveBench…) to a model slug to denote a *configuration* of the model.
43 +EFFORT_SUFFIXES: dict[str, dict[str, str]] = {
44 + "xhigh": {"reasoning_effort": "xhigh"}, "x-high": {"reasoning_effort": "xhigh"}, "extra-high": {"reasoning_effort": "xhigh"},
45 + "high": {"reasoning_effort": "high"}, "medium": {"reasoning_effort": "medium"}, "low": {"reasoning_effort": "low"}, "minimal": {"reasoning_effort": "minimal"},
46 + "max": {"reasoning_effort": "max"}, "max-effort": {"reasoning_effort": "max"}, "high-effort": {"reasoning_effort": "high"}, "low-effort": {"reasoning_effort": "low"},
47 + "medium-effort": {"reasoning_effort": "medium"}, "xhigh-effort": {"reasoning_effort": "xhigh"},
48 + "thinking": {"reasoning": "on"}, "reasoning": {"reasoning": "on"}, "think": {"reasoning": "on"}, "thinking-on": {"reasoning": "on"},
49 + "non-reasoning": {"reasoning": "off"}, "no-reasoning": {"reasoning": "off"}, "non-thinking": {"reasoning": "off"}, "nothink": {"reasoning": "off"},
50 + "no-think": {"reasoning": "off"}, "instant": {"reasoning": "off"}, "thinking-off": {"reasoning": "off"}, "fast": {"reasoning": "off"},
51 + "adaptive": {"reasoning": "adaptive"}, "adaptive-reasoning": {"reasoning": "adaptive"},
52 + "thinking-16k": {"reasoning": "on", "thinking_budget": "16k"}, "thinking-32k": {"reasoning": "on", "thinking_budget": "32k"},
53 + "thinking-64k": {"reasoning": "on", "thinking_budget": "64k"}, "thinking-128k": {"reasoning": "on", "thinking_budget": "128k"},
54 + "thinking-8k": {"reasoning": "on", "thinking_budget": "8k"}, "thinking-4k": {"reasoning": "on", "thinking_budget": "4k"}, "thinking-1k": {"reasoning": "on", "thinking_budget": "1k"},
55 +}
56 +# order matters: try the longest compound suffixes first
57 +_EFFORT_ORDERED = sorted(EFFORT_SUFFIXES, key=len, reverse=True)
58 +_EFFORT_RE = re.compile(r"[-_ ](" + "|".join(re.escape(s) for s in _EFFORT_ORDERED) + r")$", re.I)
59 +# Words that are *part of a model name*, never an effort suffix, when they precede the suffix (e.g. "Kimi K2 Thinking" is a distinct release).
60 +OFFICIAL_THINKING_RELEASES = {"kimi-k2-thinking", "qwen3-235b-a22b-thinking-2507", "qwen3-30b-a3b-thinking-2507", "qwen3-4b-thinking-2507", "glm-4.5-air-thinking",
61 + "gemini-2.5-flash-thinking", "grok-3-mini-thinking", "gemini-2-0-flash-thinking-exp-1219", "gemini-2-0-flash-thinking-exp-01-21"}
62 +
63 +# ---------------------------------------------------------------------------------------------- name analysis
64 +_SIZE_RE = re.compile(r"(?<![a-z0-9])(\d+(?:\.\d+)?)\s?([bmt])(?![a-z])", re.I) # 70B 3.8B 235B 1.5T 350M
65 +_ACTIVE_RE = re.compile(r"(?<![a-z0-9])(\d+(?:\.\d+)?)\s?[bmt]\s?-?a(\d+(?:\.\d+)?)\s?([bmt])(?![a-z])", re.I) # 35B-A3B
66 +_DATE_RE = re.compile(r"(?<!\d)(20\d{2})[-_.]?(0[1-9]|1[0-2])[-_.]?(0[1-9]|[12]\d|3[01])(?!\d)") # 20240620
67 +_MMDD_RE = re.compile(r"(?<![0-9])(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])(?![0-9])") # 0905 / 0324 (release snapshots)
68 +_YYMM_RE = re.compile(r"(?<![0-9])(2[3-9])(0[1-9]|1[0-2])(?![0-9])") # 2507 (Qwen-style)
69 +_TRAILING_VARIANT_WORDS = {"instruct", "it", "chat", "base", "sft", "dpo", "rl", "hf", "preview", "exp", "experimental", "latest", "beta", "alpha", "v1", "v2", "v3",
70 + "mtp", "distill", "distilled", "uncensored", "abliterated", "heretic", "merge"}
71 +_TOKEN_SPLIT = re.compile(r"[\s_/]+|(?<=[a-z])(?=[A-Z][a-z])")
72 +
73 +
74 +@dataclass
75 +class NameAnalysis:
76 + raw: str
77 + repo_org: str | None = None # "unsloth" in "unsloth/Qwen3.6-35B-A3B-GGUF"
78 + repo_name: str | None = None # "Qwen3.6-35B-A3B-GGUF"
79 + base_key: str = "" # normalised key with quant/precision/effort tokens removed: "qwen3.6-35b-a3b"
80 + quant_formats: list[str] = field(default_factory=list) # ["gguf"]
81 + precision: str | None = None # "bf16" when only a precision tag is present
82 + is_quantized: bool = False
83 + is_conversion: bool = False # packaging/precision conversion without quantisation (BF16 repack, MLX fp16, ONNX)
84 + from_converter_org: bool = False
85 + effort: dict[str, str] = field(default_factory=dict) # {"reasoning_effort": "xhigh"} when an evaluator suffix was stripped
86 + effort_suffix: str | None = None
87 + parameter_count: int | None = None
88 + active_parameter_count: int | None = None
89 + snapshot_date: str | None = None # "2024-06-20" / "2025-09" style date embedded in the name
90 + family_hint: str | None = None # "Qwen3.6", "Llama 4", "Claude", "Gemini 2.5"
91 +
92 + @property
93 + def is_artifact(self) -> bool:
94 + return self.is_quantized or self.is_conversion
95 +
96 + @property
97 + def is_effort_variant(self) -> bool:
98 + return bool(self.effort)
99 +
100 +
101 +def _to_count(num: str, unit: str) -> int:
102 + mult = {"m": 1e6, "b": 1e9, "t": 1e12}[unit.lower()]
103 + return int(round(float(num) * mult))
104 +
105 +
106 +def analyze_model_name(raw: str) -> NameAnalysis:
107 + """Deterministic analysis of a model / repository name."""
108 + a = NameAnalysis(raw=raw.strip())
109 + name = a.raw
110 + if "/" in name and " " not in name.split("/")[0]:
111 + org, _, rest = name.partition("/")
112 + a.repo_org, a.repo_name = org.strip(), rest.strip()
113 + name = rest
114 + a.from_converter_org = org.strip().lower() in CONVERTER_ORGS
115 + low = re.sub(r"[\s_]+", "-", name.lower().strip())
116 + low = re.sub(r"[()\[\]]+", "-", low).strip("-")
117 + low = re.sub(r"-{2,}", "-", low)
118 +
119 + # evaluator effort suffix (only when the remaining stem is not itself an official "Thinking" release)
120 + m = _EFFORT_RE.search(low)
121 + if m and low not in OFFICIAL_THINKING_RELEASES:
122 + suffix = m.group(1).lower()
123 + a.effort = dict(EFFORT_SUFFIXES[suffix])
124 + a.effort_suffix = suffix
125 + low = low[: m.start()]
126 +
127 + # sizes
128 + am = _ACTIVE_RE.search(low)
129 + if am:
130 + a.parameter_count = _to_count(am.group(1), low[am.start(1) + len(am.group(1)):].strip()[0])
131 + a.active_parameter_count = _to_count(am.group(2), am.group(3))
132 + else:
133 + sizes = _SIZE_RE.findall(low)
134 + if sizes:
135 + counts = [_to_count(n, u) for n, u in sizes]
136 + a.parameter_count = max(counts)
137 +
138 + # dates embedded in the name
139 + dm = _DATE_RE.search(low)
140 + if dm:
141 + a.snapshot_date = f"{dm.group(1)}-{dm.group(2)}-{dm.group(3)}"
142 + else:
143 + ym = _YYMM_RE.search(low)
144 + if ym and not _SIZE_RE.search(ym.group(0)):
145 + a.snapshot_date = f"20{ym.group(1)}-{ym.group(2)}"
146 +
147 + # quantisation / precision tokens
148 + tokens = [t for t in re.split(r"[-_\s./()\[\]]+", low) if t]
149 + quant: list[str] = []
150 + precision = None
151 + kept: list[str] = []
152 + for t in tokens:
153 + if t in QUANT_FORMATS:
154 + quant.append(QUANT_FORMATS[t])
155 + continue
156 + if t in PRECISION_FORMATS:
157 + precision = t
158 + continue
159 + if re.fullmatch(r"(w\d+a\d+|q\d(_[a-z0-9]+)*|iq\d(_[a-z0-9]+)*|\d-?bit|int\d|fp\d|nvfp\d|mxfp\d)", t):
160 + quant.append(QUANT_FORMATS.get(t, "quantized"))
161 + continue
162 + kept.append(t)
163 + # "GGUF" / "MLX" packaging counts as artifact even without a bit-width; "MLX-8bit" is quantised
164 + a.quant_formats = sorted(set(quant))
165 + a.precision = precision
166 + a.is_quantized = any(q not in ("onnx", "coreml", "openvino", "tensorrt", "mlx", "gguf") for q in a.quant_formats) or "gguf" in a.quant_formats
167 + a.is_conversion = (not a.is_quantized) and (bool(a.quant_formats) or precision is not None or (a.from_converter_org and bool(a.repo_org)))
168 + if "mlx" in a.quant_formats and any(q.startswith("int") for q in a.quant_formats):
169 + a.is_quantized = True
170 +
171 + a.base_key = "-".join(kept).strip("-")
172 + a.family_hint = family_hint(name)
173 + return a
174 +
175 +
176 +# ---------------------------------------------------------------------------------------------- family inference
177 +_FAMILY_PATTERNS: list[tuple[re.Pattern[str], str]] = [
178 + (re.compile(r"\bclaude\b", re.I), "Claude"),
179 + (re.compile(r"\bgpt[- ]?(oss)\b", re.I), "gpt-oss"),
180 + (re.compile(r"\b(chat)?gpt[- ]?\d", re.I), "GPT"),
181 + (re.compile(r"\bo[1-9](-| |$|mini|pro)", re.I), "OpenAI o-series"),
182 + (re.compile(r"\bgemini\b", re.I), "Gemini"),
183 + (re.compile(r"\bgemma(?=\d|\b)", re.I), "Gemma"),
184 + (re.compile(r"\bpalm\b", re.I), "PaLM"),
185 + (re.compile(r"\bllama(?=\d|\b)", re.I), "Llama"),
186 + (re.compile(r"\bmistral\b", re.I), "Mistral"),
187 + (re.compile(r"\bmixtral\b", re.I), "Mixtral"),
188 + (re.compile(r"\bministral\b", re.I), "Ministral"),
189 + (re.compile(r"\b(codestral|devstral|magistral|pixtral|voxtral)\b", re.I), None), # own families, name = family
190 + (re.compile(r"\bqwen(?=\d|\b)|\bqwq\b|\bqvq\b", re.I), "Qwen"),
191 + (re.compile(r"\bdeepseek\b", re.I), "DeepSeek"),
192 + (re.compile(r"\bkimi\b", re.I), "Kimi"),
193 + (re.compile(r"\bglm(?=\d|\b)|\bchatglm\b", re.I), "GLM"),
194 + (re.compile(r"\bgrok\b", re.I), "Grok"),
195 + (re.compile(r"\bcommand\b", re.I), "Command"),
196 + (re.compile(r"\baya\b", re.I), "Aya"),
197 + (re.compile(r"\bphi(?=\d|\b)", re.I), "Phi"),
198 + (re.compile(r"\bnemotron\b", re.I), "Nemotron"),
199 + (re.compile(r"\bgranite\b", re.I), "Granite"),
200 + (re.compile(r"\bolmo(?=\d|\b)", re.I), "OLMo"),
201 + (re.compile(r"\bmolmo\b", re.I), "Molmo"),
202 + (re.compile(r"\bfalcon\b", re.I), "Falcon"),
203 + (re.compile(r"\byi\b", re.I), "Yi"),
204 + (re.compile(r"\bminimax\b", re.I), "MiniMax"),
205 + (re.compile(r"\bhunyuan\b", re.I), "Hunyuan"),
206 + (re.compile(r"\bernie\b", re.I), "ERNIE"),
207 + (re.compile(r"\bseed\b", re.I), "Seed"),
208 + (re.compile(r"\bdoubao\b", re.I), "Doubao"),
209 + (re.compile(r"\bstep\b", re.I), "Step"),
210 + (re.compile(r"\binternlm(?=\d|\b)|\binternvl(?=\d|\b)", re.I), "InternLM"),
211 + (re.compile(r"\bjamba\b", re.I), "Jamba"),
212 + (re.compile(r"\blfm(?=\d|\b)", re.I), "LFM"),
213 + (re.compile(r"\bexaone(?=\d|\b)", re.I), "EXAONE"),
214 + (re.compile(r"\bsolar\b", re.I), "Solar"),
215 + (re.compile(r"\bnova\b", re.I), "Nova"),
216 + (re.compile(r"\btitan\b", re.I), "Titan"),
217 + (re.compile(r"\bsonar\b", re.I), "Sonar"),
218 + (re.compile(r"\bstable[- ]?diffusion\b|\bsdxl\b|\bsd3\b", re.I), "Stable Diffusion"),
219 + (re.compile(r"\bflux\b", re.I), "FLUX"),
220 + (re.compile(r"\bwhisper\b", re.I), "Whisper"),
221 + (re.compile(r"\bdall[- ]?e\b", re.I), "DALL·E"),
222 + (re.compile(r"\bsora\b", re.I), "Sora"),
223 + (re.compile(r"\bveo\b", re.I), "Veo"),
224 + (re.compile(r"\bimagen\b", re.I), "Imagen"),
225 + (re.compile(r"\bcogito\b", re.I), "Cogito"),
226 + (re.compile(r"\bhermes\b", re.I), "Hermes"),
227 + (re.compile(r"\bsmol(lm|vlm)\b", re.I), "SmolLM"),
228 + (re.compile(r"\bbert\b", re.I), "BERT"),
229 + (re.compile(r"\bt5\b", re.I), "T5"),
230 + (re.compile(r"\bclip\b", re.I), "CLIP"),
231 + (re.compile(r"\bembed(ding)?\b", re.I), None),
232 + (re.compile(r"\brerank\b", re.I), None),
233 +]
234 +_FAMILY_VERSION_RE = re.compile(r"^(?P<fam>[A-Za-z][A-Za-z·\-]*?)[\s\-]?(?P<ver>\d+(?:\.\d+)?)", re.I)
235 +
236 +
237 +def family_hint(name: str) -> str | None:
238 + """Family label without a version ("Qwen", "Llama", "Claude"). Versioned families ("Llama 3.1") are `family_release_hint`."""
239 + n = name.split("/")[-1]
240 + for pat, fam in _FAMILY_PATTERNS:
241 + if pat.search(n):
242 + if fam is None:
243 + m = pat.search(n)
244 + return m.group(0).title() if m else None
245 + return fam
246 + return None
247 +
248 +
249 +def family_release_hint(name: str) -> str | None:
250 + """Versioned family ("Llama 3.1", "Qwen3.6", "Gemini 2.5", "Claude 4") when the name carries a version right after the family word."""
251 + fam = family_hint(name)
252 + if not fam:
253 + return None
254 + n = name.split("/")[-1]
255 + m = re.search(re.escape(fam.split(" ")[0]) + r"[\s\-]?(\d+(?:\.\d+)?)", n, re.I)
256 + if m:
257 + joined = fam in ("Qwen", "GLM", "Phi", "Yi", "Step") or re.search(re.escape(fam) + r"\d", n, re.I)
258 + return f"{fam}{m.group(1)}" if joined else f"{fam} {m.group(1)}"
259 + return fam
260 +
261 +
262 +def variant_key(name: str) -> str:
263 + """Grouping key for near-duplicates: analysed base key + parameter count, ignoring org prefixes, quant/precision/effort tokens,
264 + separators and trailing 'instruct/chat/it' words. `Qwen3.6-35B-A3B`, `unsloth/Qwen3.6-35B-A3B-GGUF`, `Qwen3.6 35B A3B FP8` → same key."""
265 + a = analyze_model_name(name)
266 + toks = [t for t in a.base_key.split("-") if t and t not in _TRAILING_VARIANT_WORDS]
267 + return "-".join(toks)
268 +
269 +
270 +__all__ = ["ARTIFACT_PACKAGING", "CONVERTER_ORGS", "EFFORT_SUFFIXES", "NameAnalysis", "PRECISION_FORMATS", "QUANT_FORMATS", "analyze_model_name",
271 + "family_hint", "family_release_hint", "variant_key"]
added src/aiatlas/ontology/openness.py +115 −0
@@ -0,0 +1,115 @@
1 +"""Openness 2.0 — measurable dimensions first, a carefully labelled category second.
2 +
3 +Dimensions (each `True | False | None` = unknown):
4 + weights_available the weights can be downloaded (a checkpoint exists on a public hub / official download)
5 + source_code_available inference / model code is published
6 + training_code_available the training pipeline is published
7 + training_data_disclosed the composition of the training data is documented
8 + dataset_available the training data itself (or most of it) is downloadable
9 + commercial_use_allowed licence permits commercial use
10 + redistribution_allowed licence permits redistributing the weights
11 + derivatives_allowed licence permits fine-tunes / modifications
12 +
13 +Categories (`OPENNESS_CATEGORIES`) are derived, never asserted directly by a connector unless the source literally states it:
14 + open-source weights + code under an OSI-approved (or equivalently permissive) licence, commercial use and derivatives allowed
15 + open-weights weights downloadable under a permissive or Creative Commons licence allowing commercial use (code may be missing)
16 + restricted-weights weights downloadable but the licence restricts commercial use, hosting, derivatives or field of use
17 + proprietary weights not available (API / product only)
18 + unknown not enough evidence
19 +
20 +The historical single-value vocabulary (`open-weights | open-source | proprietary | restricted`) stays valid as input to `normalize_openness`.
21 +"""
22 +from __future__ import annotations
23 +
24 +from typing import Any
25 +
26 +from aiatlas.ontology.licenses import LICENSES, normalize_license
27 +
28 +OPENNESS_CATEGORIES = ("open-source", "open-weights", "restricted-weights", "proprietary", "unknown")
29 +
30 +OPENNESS_DIMENSIONS = (
31 + "weights_available", "source_code_available", "training_code_available", "training_data_disclosed", "dataset_available",
32 + "commercial_use_allowed", "redistribution_allowed", "derivatives_allowed",
33 +)
34 +
35 +_LEGACY = {
36 + "open-weights": "open-weights", "open_weights": "open-weights", "open weights": "open-weights", "openweights": "open-weights", "open": "open-weights",
37 + "open-source": "open-source", "open_source": "open-source", "open source": "open-source", "opensource": "open-source", "oss": "open-source",
38 + "proprietary": "proprietary", "closed": "proprietary", "closed-source": "proprietary", "closed_source": "proprietary", "closed source": "proprietary",
39 + "api-only": "proprietary", "api only": "proprietary", "commercial": "proprietary",
40 + "restricted": "restricted-weights", "restricted-weights": "restricted-weights", "restricted_weights": "restricted-weights", "gated": "restricted-weights",
41 + "research-only": "restricted-weights", "non-commercial": "restricted-weights",
42 + "unknown": "unknown", "": "unknown",
43 +}
44 +
45 +
46 +def normalize_openness(raw: Any) -> str | None:
47 + """Legacy single label → canonical category. Unknown strings → None (the writer keeps the raw label)."""
48 + if raw is None:
49 + return None
50 + s = str(raw).strip().lower()
51 + return _LEGACY.get(s)
52 +
53 +
54 +def openness_dimensions(*, weights_available: bool | None = None, license_key: str | None = None, license_raw: str | None = None,
55 + source_code_available: bool | None = None, training_code_available: bool | None = None,
56 + training_data_disclosed: bool | None = None, dataset_available: bool | None = None) -> dict[str, bool | None]:
57 + """Fill the licence-derived dimensions from the licence ontology; everything else is passed through (unknown stays None)."""
58 + key = license_key or normalize_license(license_raw)
59 + info = LICENSES.get(key) if key else None
60 + dims: dict[str, bool | None] = {
61 + "weights_available": weights_available,
62 + "source_code_available": source_code_available,
63 + "training_code_available": training_code_available,
64 + "training_data_disclosed": training_data_disclosed,
65 + "dataset_available": dataset_available,
66 + "commercial_use_allowed": info.commercial_use if info else None,
67 + "redistribution_allowed": info.redistribution if info else None,
68 + "derivatives_allowed": info.derivatives if info else None,
69 + }
70 + if info and not info.weights_downloadable and weights_available is None:
71 + dims["weights_available"] = False
72 + return dims
73 +
74 +
75 +def derive_openness(dims: dict[str, bool | None], *, license_key: str | None = None) -> str:
76 + """Category from dimensions (+ licence category). Conservative: a custom community licence is never 'open-source'."""
77 + info = LICENSES.get(license_key) if license_key else None
78 + weights = dims.get("weights_available")
79 + if weights is False:
80 + return "proprietary"
81 + if weights is None:
82 + return "unknown"
83 + commercial = dims.get("commercial_use_allowed")
84 + derivatives = dims.get("derivatives_allowed")
85 + redistribution = dims.get("redistribution_allowed")
86 + restricted_by_license = bool(info and (info.category in ("community", "research-only", "responsible-ai") or info.hosting_restrictions))
87 + if commercial is False or derivatives is False or redistribution is False or restricted_by_license:
88 + return "restricted-weights"
89 + if info and info.osi_approved and dims.get("source_code_available") is True and commercial is True and derivatives is True:
90 + return "open-source"
91 + if info and info.category in ("permissive", "creative-commons") and commercial is True and redistribution is True:
92 + return "open-weights"
93 + if info is None:
94 + # weights downloadable, licence unknown → we know the weights exist, not the terms
95 + return "open-weights" if commercial is None else "restricted-weights"
96 + return "open-weights"
97 +
98 +
99 +OPENNESS_LABELS = {
100 + "open-source": "Open source",
101 + "open-weights": "Open weights",
102 + "restricted-weights": "Restricted weights",
103 + "proprietary": "Closed / proprietary",
104 + "unknown": "Unknown",
105 +}
106 +
107 +OPENNESS_DEFINITIONS = {
108 + "open-source": "Weights and code published under an OSI-approved licence that allows commercial use and derivatives.",
109 + "open-weights": "Weights downloadable under a permissive or Creative Commons licence allowing commercial use; code or data may be missing.",
110 + "restricted-weights": "Weights downloadable, but the licence restricts commercial use, hosting, derivatives or field of use (community, research and RAIL licences).",
111 + "proprietary": "Weights are not available; the model is reachable only through an API or a product.",
112 + "unknown": "Not enough sourced evidence to classify.",
113 +}
114 +
115 +__all__ = ["OPENNESS_CATEGORIES", "OPENNESS_DEFINITIONS", "OPENNESS_DIMENSIONS", "OPENNESS_LABELS", "derive_openness", "normalize_openness", "openness_dimensions"]
added src/aiatlas/ontology/taxonomy.py +166 −0
@@ -0,0 +1,166 @@
1 +"""Small canonical enums and their normalisers. Raw source strings are never mutated in snapshots — only the materialised attribute
2 +is canonical; connectors and the writer call these before writing a claim."""
3 +from __future__ import annotations
4 +
5 +import re
6 +from typing import Any
7 +
8 +# ---------------------------------------------------------------------------------------------- modalities
9 +MODALITIES = ("text", "image", "audio", "video", "document", "code", "embedding", "3d", "structured", "action")
10 +_MODALITY_ALIASES = {
11 + "text": "text", "texts": "text", "language": "text", "nlp": "text", "chat": "text", "textual": "text",
12 + "image": "image", "images": "image", "vision": "image", "visual": "image", "picture": "image", "photo": "image", "img": "image",
13 + "audio": "audio", "speech": "audio", "voice": "audio", "sound": "audio", "music": "audio",
14 + "video": "video", "videos": "video",
15 + "pdf": "document", "document": "document", "documents": "document", "file": "document", "files": "document", "doc": "document",
16 + "code": "code", "coding": "code", "programming": "code",
17 + "embedding": "embedding", "embeddings": "embedding", "vector": "embedding",
18 + "3d": "3d", "mesh": "3d", "point cloud": "3d",
19 + "structured": "structured", "json": "structured", "table": "structured", "tabular": "structured", "time-series": "structured", "time series": "structured",
20 + "action": "action", "actions": "action", "computer-use": "action", "computer use": "action", "tool": "action", "robotics": "action",
21 + "multimodal": None, "multi-modal": None, "omni": None, "any": None, # not a modality — a property of the set
22 +}
23 +
24 +
25 +def normalize_modality(raw: Any) -> str | None:
26 + if raw is None:
27 + return None
28 + s = str(raw).strip().lower().replace("_", "-")
29 + if s in _MODALITY_ALIASES:
30 + return _MODALITY_ALIASES[s]
31 + s2 = s.replace("-", " ")
32 + return _MODALITY_ALIASES.get(s2)
33 +
34 +
35 +def normalize_modalities(raw: Any) -> list[str]:
36 + """Any iterable / comma string → sorted canonical unique list (unknown values dropped)."""
37 + if raw is None:
38 + return []
39 + items = raw if isinstance(raw, (list, tuple, set)) else re.split(r"[,/;+&]|\band\b|→|->", str(raw))
40 + out: set[str] = set()
41 + for it in items:
42 + m = normalize_modality(it)
43 + if m:
44 + out.add(m)
45 + return sorted(out)
46 +
47 +
48 +# ---------------------------------------------------------------------------------------------- model / entity status
49 +MODEL_STATUSES = ("announced", "preview", "active", "limited-availability", "deprecated", "retired", "archived", "unknown")
50 +_STATUS_ALIASES = {
51 + "active": "active", "available": "active", "ga": "active", "general availability": "active", "generally available": "active", "live": "active",
52 + "released": "active", "stable": "active", "current": "active", "supported": "active", "production": "active", "online": "active",
53 + "preview": "preview", "beta": "preview", "alpha": "preview", "experimental": "preview", "public preview": "preview", "early access": "preview",
54 + "research preview": "preview", "exp": "preview",
55 + "limited-availability": "limited-availability", "limited availability": "limited-availability", "limited": "limited-availability",
56 + "waitlist": "limited-availability", "private preview": "limited-availability", "invite-only": "limited-availability", "gated": "limited-availability",
57 + "deprecated": "deprecated", "legacy": "deprecated", "sunset": "deprecated", "sunsetting": "deprecated", "end-of-life announced": "deprecated",
58 + "deprecation": "deprecated",
59 + "retired": "retired", "discontinued": "retired", "shut down": "retired", "shutdown": "retired", "removed": "retired", "end of life": "retired",
60 + "eol": "retired", "decommissioned": "retired", "offline": "retired", "delisted": "retired",
61 + "announced": "announced", "upcoming": "announced", "coming soon": "announced", "planned": "announced", "unreleased": "announced",
62 + "archived": "archived",
63 + "unknown": "unknown", "": "unknown",
64 +}
65 +
66 +
67 +def normalize_status(raw: Any) -> str | None:
68 + if raw is None:
69 + return None
70 + s = str(raw).strip().lower().replace("_", "-")
71 + if s in _STATUS_ALIASES:
72 + return _STATUS_ALIASES[s]
73 + return _STATUS_ALIASES.get(s.replace("-", " "))
74 +
75 +
76 +# ---------------------------------------------------------------------------------------------- organization kinds
77 +ORG_KINDS = ("company", "lab", "university", "nonprofit", "government", "community", "consortium", "individual")
78 +_ORG_KIND_ALIASES = {
79 + "company": "company", "corporation": "company", "startup": "company", "enterprise": "company", "vendor": "company", "business": "company",
80 + "lab": "lab", "laboratory": "lab", "research lab": "lab", "research-lab": "lab", "research institute": "lab", "institute": "lab", "research": "lab",
81 + "university": "university", "academic": "university", "school": "university", "college": "university",
82 + "nonprofit": "nonprofit", "non-profit": "nonprofit", "non profit": "nonprofit", "foundation": "nonprofit", "ngo": "nonprofit",
83 + "government": "government", "public": "government", "agency": "government", "state": "government",
84 + "community": "community", "open-source community": "community", "collective": "community", "hf-community": "community", "group": "community",
85 + "consortium": "consortium", "alliance": "consortium", "coalition": "consortium",
86 + "individual": "individual", "person": "individual", "independent": "individual", "user": "individual",
87 +}
88 +# entity_type → default org kind (the entity_type stays; org_kind is the canonical sub-kind)
89 +ORG_TYPE_DEFAULT_KIND = {"company": "company", "lab": "lab", "university": "university", "organization": None}
90 +
91 +
92 +def normalize_org_kind(raw: Any) -> str | None:
93 + if raw is None:
94 + return None
95 + s = str(raw).strip().lower().replace("_", "-")
96 + return _ORG_KIND_ALIASES.get(s) or _ORG_KIND_ALIASES.get(s.replace("-", " "))
97 +
98 +
99 +# ---------------------------------------------------------------------------------------------- hardware kinds
100 +HARDWARE_KINDS = ("gpu", "accelerator", "npu", "cpu", "soc", "system", "server", "workstation", "cloud-instance", "rack")
101 +_HW_ALIASES = {
102 + "gpu": "gpu", "graphics card": "gpu", "graphics": "gpu", "dgpu": "gpu",
103 + "accelerator": "accelerator", "tpu": "accelerator", "asic": "accelerator", "ai accelerator": "accelerator", "lpu": "accelerator", "wse": "accelerator",
104 + "npu": "npu", "neural engine": "npu",
105 + "cpu": "cpu", "processor": "cpu",
106 + "soc": "soc", "system on chip": "soc", "system-on-chip": "soc", "apple silicon": "soc", "chip": "soc",
107 + "system": "system", "computer": "system", "desktop": "system", "laptop": "system", "mini": "system", "mac": "system",
108 + "server": "server", "node": "server", "dgx": "server", "appliance": "server",
109 + "workstation": "workstation",
110 + "cloud-instance": "cloud-instance", "cloud instance": "cloud-instance", "instance": "cloud-instance", "vm": "cloud-instance",
111 + "rack": "rack", "superpod": "rack", "pod": "rack", "cluster": "rack", "nvl72": "rack",
112 +}
113 +
114 +
115 +def normalize_hardware_kind(raw: Any) -> str | None:
116 + if raw is None:
117 + return None
118 + s = str(raw).strip().lower().replace("_", "-")
119 + return _HW_ALIASES.get(s) or _HW_ALIASES.get(s.replace("-", " "))
120 +
121 +
122 +# ---------------------------------------------------------------------------------------------- framework / library / tool kinds
123 +FRAMEWORK_KINDS = ("training-framework", "inference-engine", "serving-engine", "library", "runtime", "agent-framework", "orchestration",
124 + "evaluation-harness", "sdk", "tool", "application", "agent", "mcp-server", "vector-database", "observability", "data-tooling")
125 +_FW_ALIASES = {
126 + "framework": "training-framework", "training framework": "training-framework", "training-framework": "training-framework", "deep learning framework": "training-framework",
127 + "inference engine": "inference-engine", "inference-engine": "inference-engine", "inference": "inference-engine", "engine": "inference-engine",
128 + "serving": "serving-engine", "serving engine": "serving-engine", "serving-engine": "serving-engine", "server": "serving-engine",
129 + "library": "library", "lib": "library", "package": "library",
130 + "runtime": "runtime",
131 + "agent framework": "agent-framework", "agent-framework": "agent-framework", "agents": "agent-framework",
132 + "orchestration": "orchestration", "workflow": "orchestration", "pipeline": "orchestration",
133 + "evaluation": "evaluation-harness", "eval": "evaluation-harness", "evaluation harness": "evaluation-harness", "evaluation-harness": "evaluation-harness", "benchmarking": "evaluation-harness",
134 + "sdk": "sdk", "client": "sdk", "api client": "sdk",
135 + "tool": "tool", "cli": "tool", "utility": "tool",
136 + "application": "application", "app": "application", "product": "application", "ui": "application", "desktop app": "application",
137 + "agent": "agent", "coding agent": "agent", "assistant": "agent",
138 + "mcp": "mcp-server", "mcp server": "mcp-server", "mcp-server": "mcp-server", "mcp_server": "mcp-server",
139 + "vector database": "vector-database", "vector-database": "vector-database", "vector db": "vector-database", "vector store": "vector-database",
140 + "observability": "observability", "tracing": "observability", "monitoring": "observability", "experiment tracking": "observability",
141 + "data": "data-tooling", "data tooling": "data-tooling", "data-tooling": "data-tooling", "datasets": "data-tooling", "tokenizer": "data-tooling",
142 +}
143 +
144 +
145 +def normalize_framework_kind(raw: Any) -> str | None:
146 + if raw is None:
147 + return None
148 + s = str(raw).strip().lower()
149 + return _FW_ALIASES.get(s) or _FW_ALIASES.get(s.replace("_", "-")) or _FW_ALIASES.get(s.replace("-", " "))
150 +
151 +
152 +# ---------------------------------------------------------------------------------------------- generic helper
153 +def canonical_enum(value: Any, normaliser) -> tuple[Any, str | None]:
154 + """Return (canonical_or_original, raw_if_changed). Lets the writer store `x` canonical and `x_raw` when the source label differed."""
155 + if value is None:
156 + return None, None
157 + canon = normaliser(value)
158 + if canon is None:
159 + return value, None
160 + return canon, (str(value) if str(value) != canon else None)
161 +
162 +
163 +__all__ = [
164 + "FRAMEWORK_KINDS", "HARDWARE_KINDS", "MODALITIES", "MODEL_STATUSES", "ORG_KINDS", "ORG_TYPE_DEFAULT_KIND", "canonical_enum",
165 + "normalize_framework_kind", "normalize_hardware_kind", "normalize_modalities", "normalize_modality", "normalize_org_kind", "normalize_status",
166 +]
modified src/aiatlas/sdk/facts.py +10 −1
@@ -21,6 +21,11 @@ class EntityRef:
21 21 status: str | None = None
22 22 attributes: dict[str, Any] = field(default_factory=dict) # convenience: each item becomes a Claim
23 23 first_seen_hint: datetime | None = None # e.g. release date, to backdate `first_seen_at` for historical backfill
24 + # canonical hierarchy hints (model_family → model → artifact): the writer materialises `entities.family_id` / `canonical_id`
25 + family: EntityRef | None = None # model → its model_family ("Qwen3.6", "Llama 4", "Claude")
26 + canonical: EntityRef | None = None # artifact → the model it packages/quantises; alias entity → canonical entity
27 + artifact_kind: str | None = None # artifacts only: checkpoint|quantization|conversion|packaging
28 + identity_confidence: str | None = None # high|medium|low — how sure the connector is that this is one real thing
24 29 id: str | None = field(default=None, compare=False)
25 30
26 31 def key(self) -> str:
@@ -104,6 +109,9 @@ class ResultObs:
104 109 evaluated_at: datetime | None = None
105 110 source_url: str | None = None
106 111 confidence: str | None = None
112 + trust_level: str | None = None # ontology.benchmarks.TRUST_LEVELS; derived from the source when None
113 + variant: str | None = None # benchmark variant label (GPQA Diamond, Verified…); derived from config when None
114 + run_group: str | None = None # evaluation run/release the row belongs to; derived from config when None
107 115
108 116
109 117 @dataclass
@@ -196,6 +204,7 @@ MATERIAL_PROPERTIES: dict[str, tuple[str, int]] = {
196 204 "active_parameter_count": ("PARAMETERS_CHANGED", 1),
197 205 "weights_availability": ("OPENNESS_CHANGED", 3),
198 206 "openness": ("OPENNESS_CHANGED", 3),
207 + "license_key": ("LICENSE_CHANGED", 2),
199 208 "release_date": ("RELEASE_DATE_CHANGED", 1),
200 209 "knowledge_cutoff": ("KNOWLEDGE_CUTOFF_CHANGED", 1),
201 210 "latest_version": ("VERSION_RELEASED", 2),
@@ -211,7 +220,7 @@ MATERIAL_PROPERTIES: dict[str, tuple[str, int]] = {
211 220 NOISY_PREFIXES = ("metric.", "stats.", "counts.")
212 221
213 222 EVENT_CATEGORY_BY_TYPE: dict[str, str] = {
214 − "model": "model", "company": "company", "organization": "company", "lab": "company", "university": "company",
223 + "model": "model", "model_family": "model", "artifact": "model", "company": "company", "organization": "company", "lab": "company", "university": "company",
215 224 "paper": "paper", "dataset": "dataset", "benchmark": "benchmark", "provider": "provider", "framework": "framework",
216 225 "library": "framework", "repository": "repository", "tool": "tool", "agent": "tool", "hardware": "hardware",
217 226 "runtime": "framework", "quantization": "model", "regulation": "regulation", "incident": "incident", "release": "release",
218 227