SPB Git
spboucher.ai

spb/ai-atlas

Public

Canonical upgrade foundation: ontology package (licences, openness, taxonomy, model naming, benchmarks, anomalies), migration 0003, Facts hierarchy hints, implementation audit

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 12, 2026) parent c64e7b8 Browse files

12 changed files +1,624 −1

12 changed files
Load all 12 files Unified Split
added apps/web/qa/_shot_tmp.mjs +12 −0
@@ -0,0 +1,12 @@
1+import { chromium } from 'playwright';
2+const b = await chromium.launch();
3+for (const [w,h,tag] of [[1440,900,'desk'],[390,844,'mob']]) {
4+ const ctx = await b.newContext({ viewport: { width: w, height: h }, colorScheme: 'dark' });
5+ const p = await ctx.newPage();
6+ for (const path of ['/', '/models/claude-fable-5.1', '/benchmarks/gpqa']) {
7+ await p.goto('https://www.ai-atlas.co' + path, { waitUntil: 'networkidle', timeout: 60000 }).catch(()=>{});
8+ await p.screenshot({ path: `/tmp/aia-${tag}-${path.replace(/[^a-z0-9]+/gi,'_')||'home'}.png`, fullPage: false });
9+ }
10+ await ctx.close();
11+}
12+await b.close();
added docs/UPGRADE-2026-09-AUDIT.md +137 −0
@@ -0,0 +1,137 @@
1+# AI Atlas — implementation audit for the canonical upgrade (2026-09-12)
2+
3+Written before the "temporal knowledge graph + intelligence terminal" upgrade. Numbers are from the production database
4+(M2M32c, dump `aiatlas-prod.dump` taken 2026-09-11 23:47 EDT, 133 MB) and from reading every module of the repository.
5+
6+## 1. Existing architecture (keep)
7+
8+FastAPI + SQLAlchemy Core/asyncpg + Alembic (forward-only SQL) · Redis cache · APScheduler scheduler (`aia schedule`) running connectors
9+**sequentially** · Postgres job queue (SKIP LOCKED) · LLM gateway → llm-api.io (deterministic first, LLM claims one tier lower) ·
10+Next 16 App Router (server components, Tailwind v4 tokens, hand-rolled d3 SVG charts, Playwright QA borrowed from `~/Desktop/uqo-eval`).
11+Deployment: `mld` manifest `deploy/ai-atlas.mld.json`, PM2 `ai-atlas-api :8321 / ai-atlas-scheduler / ai-atlas-web :8320`, MacLustr Tunnel.
12+
13+## 2. Existing schema (0001 + 0002)
14+
15+`entities` (type, canonical_name, slug, organization_id, attributes/provenance/quality/counts jsonb, first/last_seen, merged_into, search) ·
16+`entity_aliases` · `entity_identifiers (scheme,value) unique` · `sources` · `connectors` · `connector_runs` · `connector_errors` ·
17+`documents` · `snapshots (run_id, raw_path, text_path, structured, diff, parser_version)` · `claims (temporal: valid_from/valid_to, status
18+current|superseded|conflicting|retracted, tier, confidence, extractor)` · `relations (live uniq on subject,predicate,object)` ·
19+`change_events (observed_at, effective_at, importance, dedupe_key)` · `benchmark_results (config jsonb, dedupe_key, valid_to)` ·
20+`prices (live uniq per model×provider×provider_model_id)` · `jobs` · `llm_jobs` · `review_queue` · `entity_embeddings` · `stats_snapshots` ·
21+`page_views` · `api_keys` · `metric_definitions` · `domains`.
22+
23+Missing for this upgrade: run/batch id on facts, backfill/occurrence semantics on events, canonical hierarchy columns, benchmark
24+comparability key and trust level, persisted resolution decisions, anomaly flags, admin audit log, quarantine.
25+
26+## 3. Entity types in the data (prod, live rows)
27+
28+researcher 1 899 · model 1 880 · paper 1 060 · dataset 146 · company 119 · library 67 · framework 66 · hardware 42 · organization 36 ·
29+benchmark 27 · provider 27 · repository 10 · lab 8 · university 1 · tool 0 (public nav shows Tools with "0 total").
30+`ids.ENTITY_TYPES` declares 33 types; `quantization`, `license`, `release`, `agent`, `runtime`, `tool` are declared but never produced.
31+
32+## 4. Claim architecture (keep)
33+
34+`FactWriter.write_claim`: no current → insert; same → confirm; different & source tier ≤ current → supersede + event for material
35+properties; different & worse tier → `conflicting` + review item. Soft properties never event. `metric.*` never event. Prices and
36+results append-only with close/open. **Loophole**: `same_source` lets an LLM claim (tier+1) supersede a deterministic claim from the same
37+URL (`writer.py:141-145`). **Gap**: no `run_id` on facts; `EntityRef.first_seen_hint` never read (`resolution.py:131`).
38+
39+## 5. Connector architecture (keep)
40+
41+`BaseConnector.run()` → discover → fetch (conditional) → archive → snapshot (+diff) → parse → extract → `FactWriter` → follow-ups →
42+optional `llm_extract` job. Breakage detection (`expected_min_records`, only on full re-extraction), circuit breaker, adaptive interval,
43+reprocess-from-archive. **Bugs**: single-URL reprocess builds key-less targets → no-op (`connector.py:188`); `Fetcher` has **no SSRF guard**
44+(no private-IP/scheme/redirect validation, follow-ups come from crawled content); rate limit read from class attr, not registry.
45+
46+## 6. Current model resolution
47+
48+`Resolver`: identifiers (exact scheme+value) → normalized alias within type (org-disambiguated) → slug → create. `normalize_alias`
49+erases dots/hyphens/spaces (`Qwen3-8B` ≡ `Qwen 38B`). Identifier schemes are per source (`anthropic_model_id`, `gemini_model_id`,
50+`fireworks-ai_model_id`…) so cross-source identity rests on aliases. Consequences measured in prod:
51+
52+* **399 models exist only through an `artificial_analysis` identifier**, 204 of them are *effort/thinking variants* of another model
53+ (`claude-fable-5-1-xhigh`, `gpt-6-astra-high`, `deepseek-v4-pro-0424-non-reasoning`) — evaluation configurations, not models.
54+* **157 Hugging Face repos flagged `is_quantized`** (`quant_format` gguf 64, fp8 41, onnx 27, mlx 23, awq 2) plus ~150 conversions/re-packagings
55+ (`amd/…-MXFP4`, `zai-org/GLM-5-FP8`, `mlx-community/Kimi-K2.5`) are typed `model`; they dominate "largest models" (Kimi K2.5 appears 4×).
56+* 72 models exist only through OpenRouter; OpenRouter `release_date` = listing date; routed prices booked on the lab's provider entity with
57+ a different `provider_model_id` → two concurrent "current" prices for one model×provider.
58+* 19 exact normalized-name duplicates across slugs (`gemini-2.5-flash-lite` / `gemini-2-5-flash-lite`, `deepseek-v3-{2,3,4}`…); 80 pending
59+ `merge_candidate` items; 140 pending `conflict` items (status 34, modalities_input 30, context_length 26).
60+* 1 899 researchers created **by name only** (first 3 arXiv authors, no identifier, no affiliation) — homonyms merge, variants split.
61+* 70 models without organization; 853 without release date; 1 205 without parameter count; only 115 carry `family`.
62+
63+## 7. Current benchmark pipeline
64+
65+Four boards (Artificial Analysis 4 003 rows, LiveBench 456, SWE-bench 323, aider 138) → 4 920 live rows on 15 of 27 registered benchmarks
66+(12 registry benchmarks — MMLU, HumanEval, AIME 2025, MATH-500, MTEB, LMArena… — have **zero** results: no connector feeds them).
67+Leaderboards are one row per live result: LiveBench mixes `global_average` with 7 `category:*` metrics in one board; aider mixes
68+`pass_rate_2` and `percent_cases_well_formed`; every LiveBench release / aider run date creates new rows that are never retired;
69+`/compare` picks the last-observed metric per model. Models are resolved by free text (aider, SWE-bench) or AA slug, so AA effort variants
70+become separate leaderboard entries (GPQA: 614 results = 614 "models"). No trust level, no comparability key, no metric bounds.
71+The registry lacks family / variant / direction / metric range. The API only resolves benchmarks by exact slug (no aliases).
72+
73+## 8. Current event pipeline
74+
75+12 196 events, **all observed on 2026-09-11** (initial corpus): NEW_MODEL 1 880 (importance 3), BENCHMARK_RESULT 4 920, ANNOUNCEMENT 1 764,
76+NEW_PAPER 1 060, RELEASE 692, PROVIDER_LISTED 613… `/changes`, `/changes/daily` and `/stats.change_events_24h` key on `observed_at`; only
77+`/timeline` uses `coalesce(effective_at, observed_at)`. 8 713 events have no `effective_at`. There is no backfill flag, no
78+occurred/observed/recorded distinction, no semantic dedupe of one release across sources. Homepage says "+12,196 · 24 h".
79+
80+## 9. Taxonomy problems found (prod values)
81+
82+* license: `apache-2.0` 344 · `Apache 2.0` 9 · `Modified MIT` · `CC BY-NC 4.0` vs `cc-by-nc-4.0` · `other` 157 · Llama variants `llama3`, `llama3.1`… (no SPDX, no permissions)
83+* openness: `open-weights` 971 · empty 478 · `proprietary` 348 · `restricted` 82 (= HF *gated*, not a licence property) · `open-source` 1 (Cohere prose)
84+* modalities: `["Text"]` vs `["text"]`; Google emits `pdf`; Anthropic values are hard-coded constants; `capabilities` uses OpenAI slugs and Google labels
85+* status: `limited-availability` vs schema `available`; `archived` undocumented; `groq_status` side channel
86+* org kinds: `company|organization|lab|university` as entity types, with duplicates (`Kwaipilot` company + `kwaipilot-2` organization, `Upstage`/`upstage-2`, `StepFun`/`stepfun-ai`, `Thinking Machines`/`thinkingmachines`)
87+* identifiers: `gemini_model_id` (provider key is `google-gemini-api`), `fireworks-ai_model_id` (hyphen), platform ids stored as claims
88+* hardware `kind`: `soc|gpu|system|computer`; `memory_gb` int vs list
89+* benchmarks: `metric` free text (`pass rate (2 attempts)`), `unit` `%` or empty
90+
91+## 10. Duplicate / hierarchy problems
92+
93+Model = release = checkpoint = quantization = provider alias today. Needed hierarchy: **model_family → model → artifact → deployment**.
94+Artifact kinds: `checkpoint` (official weights repo), `quantization`, `conversion` (BF16/FP8 repack, MLX, ONNX), `packaging`.
95+Evaluation-effort variants fold into the canonical model as **result configuration** (`reasoning_effort`, `reasoning`), never entities.
96+
97+## 11. Components to preserve
98+
99+Archive (`sdk/archive.py`), `FactWriter` temporal rules, `Resolver` precedence, connector SDK and all 27 connectors + 91 fixture tests,
100+LLM gateway cascade and accounting, `/asof` `/history` `/diff` `/compare` `/prices/index` `/hardware/fit` endpoints and their semantics,
101+`merge_entities`, quality methodology, design tokens, `KeyValue`/`DataTable`/`Tabs`/`ProvenanceInline`, compare tray, admin console,
102+sitemap shards, OG frame, slugs, ids, all public URLs.
103+
104+## 12. Schema migration 0003 (`0003_canonical_ontology`)
105+
106+* `entities`: `family_id`, `canonical_id` (artifact → model, folded variant → model), `artifact_kind`, `identity_confidence`; new types
107+ `model_family` (prefix `family`), `artifact` (prefix `artifact`); `license` entities created from the ontology (`uses_license` relation).
108+* facts: `run_id` on `claims`, `relations`, `prices`, `benchmark_results`, `change_events` (+ indexes) for batch rollback.
109+* `change_events`: `recorded_at`, `is_backfill`, generated `occurred_at = coalesce(effective_at, observed_at)`, `group_key` (semantic
110+ dedupe of one release across documents), indexes on `(is_backfill, occurred_at)`.
111+* `benchmark_results`: `trust_level`, `config_key` (comparability), `variant`, `run_group`, `is_current` maintained by the writer
112+ (one current row per model × benchmark × metric × config_key; older runs closed with `valid_to`).
113+* new tables: `resolution_decisions`, `anomalies`, `admin_audit_log`, `quarantined_runs`, `taxonomy_mappings`, `watch_digests` (none), `model_families` view.
114+* benchmark registry gains `family, variant, version, direction, metric_min, metric_max, harness, comparability` fields (claims).
115+
116+## 13. Backward compatibility
117+
118+* Every existing slug and id keeps resolving: artifacts stay reachable at `/models/<slug>` (API accepts `artifact` on the models mount,
119+ web 301s to `/artifacts/<slug>`); folded effort variants resolve through `merged_into` (API follows it, web 301s to the canonical model).
120+* `/models` excludes artifacts and merged rows by default (`include=artifacts` restores the old universe); documented in `docs/API.md`
121+ as v1.1 with a changelog. No v2 needed: no field is removed or retyped; new fields are additive.
122+* Raw snapshots are never modified; taxonomy normalisation writes canonical attributes and keeps `*_raw` when the source label differs.
123+* Counters get explicit definitions ("Models = canonical model releases; artifacts and folded variants excluded").
124+
125+## 14. Implementation sequence
126+
127+1. Ontology package (`src/aiatlas/ontology/`) + migration 0003 + tests.
128+2. SDK: writer/resolver use the ontology; `run_id` propagation; `first_seen_hint`; same-source loophole; SSRF guard; reprocess fix; quarantine.
129+3. `aia canonicalize` (dry-run + apply): artifacts, effort variants, families, licences, taxonomy, org duplicates, event backfill flags,
130+ benchmark result comparability/trust; `aia anomalies`; `aia audit-data`.
131+4. Connectors: HF → artifacts, AA → configs, leaderboards metrics split, benchmark registry families, OpenRouter fixes, provider ids.
132+5. API 1.1: families/artifacts/deployments/frontier/pulse/pareto/finder/cost/matrix/history/comparability/time-machine/claims/admin quality
133+ & resolution & extraction debugger & audit; counters; backfill-aware feeds; search compiler v2; security fixes.
134+6. Web: brand, shell (nav + ⌘K commands + evidence drawer + density), homepage 3.0, model page 3.0, compare 3.0, benchmarks 2.0,
135+ frontier, prices terminal, calculator, run-locally, find-a-model, open, graph, timeline 2.0, time machine, diff, pulse, families,
136+ watchlist, admin workbenches, mobile, OG per type, SEO.
137+7. Tests, QA sweeps, final data/benchmark/event/counter audits, deploy, memory.
added migrations/versions/0003_canonical_ontology.py +188 −0
@@ -0,0 +1,188 @@
1+"""Canonical ontology: model hierarchy columns (family / canonical / artifact kind), run ids on every fact for batch rollback,
2+event semantics (recorded_at, occurred_at, is_backfill, group_key), benchmark comparability (config_key, trust_level, variant,
3+run_group, is_current), persisted entity-resolution decisions, anomaly flags, admin audit log, quarantined runs, taxonomy mappings.
4+
5+Additive only — no column is dropped or retyped; existing rows keep their values.
6+
7+Revision ID: 0003
8+"""
9+from __future__ import annotations
10+
11+from alembic import op
12+
13+revision = "0003"
14+down_revision = "0002"
15+branch_labels = None
16+depends_on = None
17+
18+SQL = r"""
19+-- ---------------------------------------------------------------------------------------------- entity hierarchy
20+alter table entities add column if not exists family_id text references entities(id);
21+-- artifact → model; folded variant → model
22+alter table entities add column if not exists canonical_id text references entities(id);
23+-- checkpoint|quantization|conversion|packaging (artifacts only)
24+alter table entities add column if not exists artifact_kind text;
25+-- high|medium|low (how sure we are this row is one real thing)
26+alter table entities add column if not exists identity_confidence text not null default 'high';
27+create index if not exists entities_family_idx on entities (family_id) where family_id is not null;
28+create index if not exists entities_canonical_idx on entities (canonical_id) where canonical_id is not null;
29+create index if not exists entities_type_slug_idx on entities (entity_type, slug);
30+create index if not exists entities_type_status_idx on entities (entity_type, status) where merged_into is null;
31+
32+-- ---------------------------------------------------------------------------------------------- run ids on facts (batch rollback)
33+alter table claims add column if not exists run_id text;
34+alter table relations add column if not exists run_id text;
35+alter table prices add column if not exists run_id text;
36+alter table benchmark_results add column if not exists run_id text;
37+alter table change_events add column if not exists run_id text;
38+create index if not exists claims_run_idx on claims (run_id) where run_id is not null;
39+create index if not exists relations_run_idx on relations (run_id) where run_id is not null;
40+create index if not exists prices_run_idx on prices (run_id) where run_id is not null;
41+create index if not exists benchmark_results_run_idx on benchmark_results (run_id) where run_id is not null;
42+create index if not exists change_events_run_idx on change_events (run_id) where run_id is not null;
43+-- source label when the canonical value differs
44+alter table claims add column if not exists value_raw text;
45+
46+-- ---------------------------------------------------------------------------------------------- event semantics
47+alter table change_events add column if not exists recorded_at timestamptz not null default now();
48+alter table change_events add column if not exists is_backfill boolean not null default false;
49+-- semantic group (one release across documents)
50+alter table change_events add column if not exists group_key text;
51+alter table change_events add column if not exists occurred_at timestamptz generated always as (coalesce(effective_at, observed_at)) stored;
52+create index if not exists change_events_live_idx on change_events (occurred_at desc) where is_backfill = false and event_type <> 'DOCUMENT_CHANGED';
53+create index if not exists change_events_occurred_idx on change_events (occurred_at desc);
54+create index if not exists change_events_group_idx on change_events (group_key) where group_key is not null;
55+create index if not exists change_events_live_importance_idx on change_events (importance desc, occurred_at desc) where is_backfill = false;
56+
57+-- ---------------------------------------------------------------------------------------------- benchmark comparability & trust
58+alter table benchmark_results add column if not exists config_key text;
59+alter table benchmark_results add column if not exists trust_level text;
60+alter table benchmark_results add column if not exists variant text;
61+alter table benchmark_results add column if not exists run_group text;
62+alter table benchmark_results add column if not exists is_current boolean not null default true;
63+alter table benchmark_results add column if not exists extractor text not null default 'deterministic';
64+create index if not exists benchmark_results_current2_idx on benchmark_results (benchmark_id, config_key, is_current) where valid_to is null;
65+create index if not exists benchmark_results_model_metric_idx on benchmark_results (model_id, benchmark_id, metric) where valid_to is null;
66+
67+-- ---------------------------------------------------------------------------------------------- persisted resolution decisions
68+create table if not exists resolution_decisions (
69+ id text primary key,
70+ a_id text not null references entities(id) on delete cascade,
71+ b_id text not null references entities(id) on delete cascade,
72+ decision text not null, -- merge|alias|variant_of|family_member|keep_separate|defer
73+ actor text not null default 'admin',
74+ note text,
75+ payload jsonb not null default '{}'::jsonb,
76+ applied boolean not null default false,
77+ created_at timestamptz not null default now(),
78+ unique (a_id, b_id, decision)
79+);
80+create index if not exists resolution_decisions_pair_idx on resolution_decisions (a_id, b_id);
81+create index if not exists resolution_decisions_b_idx on resolution_decisions (b_id);
82+
83+-- ---------------------------------------------------------------------------------------------- anomaly flags (never deletes)
84+create table if not exists anomalies (
85+ id text primary key,
86+ entity_id text references entities(id) on delete cascade,
87+ check_name text not null,
88+ severity text not null, -- critical|warning|info
89+ message text not null,
90+ value jsonb,
91+ detail jsonb not null default '{}'::jsonb,
92+ status text not null default 'open', -- open|resolved|ignored|fixed
93+ first_seen_at timestamptz not null default now(),
94+ last_seen_at timestamptz not null default now(),
95+ resolved_at timestamptz,
96+ resolution text,
97+ dedupe_key text unique
98+);
99+create index if not exists anomalies_status_idx on anomalies (status, severity, last_seen_at desc);
100+create index if not exists anomalies_entity_idx on anomalies (entity_id);
101+
102+-- ---------------------------------------------------------------------------------------------- admin audit log
103+create table if not exists admin_audit_log (
104+ id bigserial primary key,
105+ actor text not null default 'admin',
106+ action text not null,
107+ target text,
108+ payload jsonb not null default '{}'::jsonb,
109+ ip text,
110+ created_at timestamptz not null default now()
111+);
112+create index if not exists admin_audit_log_created_idx on admin_audit_log (created_at desc);
113+
114+-- ---------------------------------------------------------------------------------------------- quarantined runs (held facts awaiting review)
115+create table if not exists quarantined_runs (
116+ id text primary key,
117+ run_id text,
118+ connector_name text not null,
119+ reason text not null,
120+ stats jsonb not null default '{}'::jsonb, -- baseline vs observed counts
121+ facts jsonb not null default '[]'::jsonb, -- serialised Facts objects, written on release
122+ status text not null default 'pending', -- pending|released|discarded
123+ created_at timestamptz not null default now(),
124+ resolved_at timestamptz,
125+ resolved_by text
126+);
127+create index if not exists quarantined_runs_status_idx on quarantined_runs (status, created_at desc);
128+
129+-- ---------------------------------------------------------------------------------------------- taxonomy mappings observed (raw → canonical)
130+create table if not exists taxonomy_mappings (
131+ domain text not null, -- license|openness|modality|status|org_kind|hardware_kind|framework_kind|metric
132+ raw text not null,
133+ canonical text,
134+ count integer not null default 1,
135+ first_seen_at timestamptz not null default now(),
136+ last_seen_at timestamptz not null default now(),
137+ primary key (domain, raw)
138+);
139+
140+-- ---------------------------------------------------------------------------------------------- connector run baselines (anomaly detector)
141+alter table connector_runs add column if not exists quarantined boolean not null default false;
142+alter table connector_runs add column if not exists baseline jsonb;
143+-- rolling medians of records/prices/results
144+alter table connectors add column if not exists baseline jsonb;
145+
146+-- ---------------------------------------------------------------------------------------------- search vector: include aliases of family
147+create or replace function entities_search_update() returns trigger language plpgsql as $$
148+begin
149+ new.search :=
150+ setweight(to_tsvector('simple', coalesce(new.canonical_name, '')), 'A') ||
151+ setweight(to_tsvector('simple', coalesce(new.attributes->>'family', '')), 'B') ||
152+ setweight(to_tsvector('simple', coalesce(new.attributes->>'api_model_id', '')), 'B') ||
153+ setweight(to_tsvector('simple', coalesce(new.entity_type, '')), 'C') ||
154+ setweight(to_tsvector('english', left(coalesce(new.description, ''), 4000)), 'C');
155+ new.updated_at := now();
156+ return new;
157+end $$;
158+"""
159+
160+
161+def upgrade() -> None:
162+ for statement in _split(SQL):
163+ op.execute(statement)
164+
165+
166+def downgrade() -> None:
167+ raise RuntimeError("AI Atlas migrations are forward-only: historical data is never disposable")
168+
169+
170+def _split(sql: str) -> list[str]:
171+ out: list[str] = []
172+ buf: list[str] = []
173+ in_dollar = False
174+ for line in sql.splitlines():
175+ stripped = line.strip()
176+ if stripped.count("$$") % 2 == 1:
177+ in_dollar = not in_dollar
178+ buf.append(line)
179+ if not in_dollar and stripped.endswith(";"):
180+ body = [ln for ln in buf if not ln.strip().startswith("--") or in_dollar]
181+ stmt = "\n".join(body).strip()
182+ if stmt:
183+ out.append(stmt)
184+ buf = []
185+ tail = "\n".join(ln for ln in buf if not ln.strip().startswith("--")).strip()
186+ if tail:
187+ out.append(tail)
188+ return out
modified src/aiatlas/ids.py +2 −0
@@ -8,6 +8,8 @@ from ulid import ULID
8 8
9 PREFIXES: dict[str, str] = { 9 PREFIXES: dict[str, str] = {
10 "model": "model", 10 "model": "model",
11+ "model_family": "family", # Llama 4, Qwen3.6, Claude — groups model releases
12+ "artifact": "artifact", # checkpoint / quantisation / conversion of a model (never a model of its own)
11 "company": "company", 13 "company": "company",
12 "organization": "org", 14 "organization": "org",
13 "researcher": "person", 15 "researcher": "person",
added src/aiatlas/ontology/__init__.py +50 −0
@@ -0,0 +1,50 @@
1+"""AI Atlas ontology — canonical vocabularies and deterministic normalisers.
2+
3+Everything here is pure Python (no database, no network): connectors, the writer, the API and the QA checks all import the same
4+definitions so that "Apache 2.0", "apache-2.0" and "Apache-2.0" are one licence, "Text" and "text" one modality, and
5+`unsloth/Qwen3.6-35B-A3B-GGUF` an *artifact* of the model `Qwen3.6-35B-A3B` rather than a new foundation model.
6+
7+Modules
8+ licenses SPDX-anchored licence ontology (permissions, restrictions, category) + `normalize_license`
9+ openness openness dimensions → carefully labelled category + `normalize_openness`
10+ taxonomy small enums (modalities, statuses, organization kinds, hardware kinds, framework kinds) + normalisers
11+ models model naming: quantisation/precision detection, evaluation-effort variants, family inference, base-name keys
12+ benchmarks benchmark families / variants / metric ranges + comparability of two result configurations
13+ anomalies deterministic sanity checks on entity attributes, prices and results (flags, never deletes)
14+"""
15+from aiatlas.ontology.licenses import LICENSES, LicenseInfo, normalize_license
16+from aiatlas.ontology.openness import OPENNESS_CATEGORIES, derive_openness, normalize_openness, openness_dimensions
17+from aiatlas.ontology.taxonomy import (
18+ FRAMEWORK_KINDS,
19+ HARDWARE_KINDS,
20+ MODALITIES,
21+ MODEL_STATUSES,
22+ ORG_KINDS,
23+ normalize_framework_kind,
24+ normalize_hardware_kind,
25+ normalize_modalities,
26+ normalize_modality,
27+ normalize_org_kind,
28+ normalize_status,
29+)
30+
31+__all__ = [
32+ "FRAMEWORK_KINDS",
33+ "HARDWARE_KINDS",
34+ "LICENSES",
35+ "MODALITIES",
36+ "MODEL_STATUSES",
37+ "OPENNESS_CATEGORIES",
38+ "ORG_KINDS",
39+ "LicenseInfo",
40+ "derive_openness",
41+ "normalize_framework_kind",
42+ "normalize_hardware_kind",
43+ "normalize_license",
44+ "normalize_modalities",
45+ "normalize_modality",
46+ "normalize_openness",
47+ "normalize_org_kind",
48+ "normalize_status",
49+ "openness_dimensions",
50+]
added src/aiatlas/ontology/anomalies.py +178 −0
@@ -0,0 +1,178 @@
1+"""Deterministic sanity checks. Each check returns `Anomaly` records — flags with evidence, never deletions or silent fixes.
2+
3+Severity: `critical` (value is physically impossible or contradicts itself), `warning` (implausible, needs a look), `info` (worth
4+knowing, e.g. a duplicate candidate). `dedupe_key` keeps one open row per (entity, check)."""
5+from __future__ import annotations
6+
7+from dataclasses import dataclass, field
8+from datetime import UTC, date, datetime
9+from typing import Any
10+
11+from aiatlas.ontology.benchmarks import metric_bounds
12+
13+MAX_PARAMS = 10e12 # 10T
14+MAX_CONTEXT = 100_000_000 # 100M tokens
15+MAX_OUTPUT = 100_000_000
16+MAX_PRICE_PER_MTOK = 5_000.0 # USD per 1M tokens
17+MAX_MEMORY_GB = 100_000
18+MIN_RELEASE = date(2010, 1, 1)
19+
20+
21+@dataclass
22+class Anomaly:
23+ check: str
24+ severity: str
25+ message: str
26+ entity_id: str | None = None
27+ value: Any = None
28+ detail: dict[str, Any] = field(default_factory=dict)
29+
30+ @property
31+ def dedupe_key(self) -> str:
32+ return f"{self.check}:{self.entity_id or self.detail.get('key', '')}"
33+
34+
35+def _num(v: Any) -> float | None:
36+ if isinstance(v, bool):
37+ return None
38+ if isinstance(v, (int, float)):
39+ return float(v)
40+ if isinstance(v, str):
41+ try:
42+ return float(v.replace(",", ""))
43+ except ValueError:
44+ return None
45+ return None
46+
47+
48+def _date(v: Any) -> date | None:
49+ if isinstance(v, datetime):
50+ return v.date()
51+ if isinstance(v, date):
52+ return v
53+ if isinstance(v, str) and len(v) >= 4 and v[:4].isdigit():
54+ try:
55+ parts = v[:10].split("-")
56+ y = int(parts[0]); m = int(parts[1]) if len(parts) > 1 and parts[1].isdigit() else 1; d = int(parts[2]) if len(parts) > 2 and parts[2][:2].isdigit() else 1
57+ return date(y, max(1, min(12, m)), max(1, min(28, d)))
58+ except ValueError:
59+ return None
60+ return None
61+
62+
63+def check_model(entity_id: str, name: str, attrs: dict[str, Any], *, today: date | None = None) -> list[Anomaly]:
64+ today = today or datetime.now(UTC).date()
65+ out: list[Anomaly] = []
66+ params = _num(attrs.get("parameter_count"))
67+ active = _num(attrs.get("active_parameter_count"))
68+ ctx = _num(attrs.get("context_length"))
69+ max_out = _num(attrs.get("max_output_tokens"))
70+ rel = _date(attrs.get("release_date"))
71+ dep = _date(attrs.get("deprecation_date"))
72+ ret = _date(attrs.get("retirement_date"))
73+ cutoff = _date(attrs.get("knowledge_cutoff"))
74+
75+ if params is not None and params > MAX_PARAMS:
76+ out.append(Anomaly("params_too_large", "critical", f"{name}: parameter_count {params:.3g} exceeds {MAX_PARAMS:.0e}", entity_id, params))
77+ if params is not None and params < 1e5:
78+ out.append(Anomaly("params_too_small", "warning", f"{name}: parameter_count {params:.3g} below 100K", entity_id, params))
79+ if params is not None and active is not None and active > params:
80+ out.append(Anomaly("active_gt_total", "critical", f"{name}: active parameters {active:.3g} > total {params:.3g}", entity_id, active,
81+ {"parameter_count": params, "active_parameter_count": active}))
82+ if ctx is not None and ctx > MAX_CONTEXT:
83+ out.append(Anomaly("context_too_large", "critical", f"{name}: context_length {ctx:.0f} exceeds 100M tokens", entity_id, ctx))
84+ if ctx is not None and ctx < 256:
85+ out.append(Anomaly("context_too_small", "warning", f"{name}: context_length {ctx:.0f} below 256 tokens", entity_id, ctx))
86+ if max_out is not None and ctx is not None and max_out > ctx:
87+ out.append(Anomaly("max_output_gt_context", "warning", f"{name}: max_output_tokens {max_out:.0f} > context_length {ctx:.0f}", entity_id, max_out,
88+ {"context_length": ctx}))
89+ if rel and rel > today:
90+ out.append(Anomaly("release_in_future", "critical", f"{name}: release_date {rel.isoformat()} is in the future", entity_id, rel.isoformat()))
91+ if rel and rel < MIN_RELEASE:
92+ out.append(Anomaly("release_too_old", "warning", f"{name}: release_date {rel.isoformat()} before 2010", entity_id, rel.isoformat()))
93+ if rel and dep and dep < rel:
94+ out.append(Anomaly("deprecated_before_release", "critical", f"{name}: deprecation_date {dep} before release_date {rel}", entity_id, dep.isoformat(),
95+ {"release_date": rel.isoformat()}))
96+ if rel and ret and ret < rel:
97+ out.append(Anomaly("retired_before_release", "critical", f"{name}: retirement_date {ret} before release_date {rel}", entity_id, ret.isoformat(),
98+ {"release_date": rel.isoformat()}))
99+ if dep and ret and ret < dep:
100+ out.append(Anomaly("retired_before_deprecated", "warning", f"{name}: retirement_date {ret} before deprecation_date {dep}", entity_id, ret.isoformat()))
101+ if cutoff and rel and cutoff > rel:
102+ out.append(Anomaly("cutoff_after_release", "warning", f"{name}: knowledge_cutoff {cutoff} after release_date {rel}", entity_id, cutoff.isoformat()))
103+ status = str(attrs.get("status") or "").lower()
104+ if status in ("deprecated", "retired") and rel and rel > today:
105+ out.append(Anomaly("deprecated_future_release", "critical", f"{name}: status {status} but release in the future", entity_id, status))
106+ if attrs.get("openness") in ("open-weights", "open-source") and not any(attrs.get(k) for k in ("hf_repo", "repository_url", "model_card_url", "weights_url")):
107+ out.append(Anomaly("open_without_weights_url", "info", f"{name}: labelled {attrs.get('openness')} but no weights location recorded", entity_id, attrs.get("openness")))
108+ return out
109+
110+
111+def check_price(row: dict[str, Any]) -> list[Anomaly]:
112+ out: list[Anomaly] = []
113+ key = f"{row.get('model_id')}:{row.get('provider_id')}:{row.get('provider_model_id') or ''}"
114+ label = f"{row.get('model_name') or row.get('model_id')} @ {row.get('provider_name') or row.get('provider_id')}"
115+ inp, outp = _num(row.get("input_per_mtok")), _num(row.get("output_per_mtok"))
116+ for k, v in (("input_per_mtok", inp), ("output_per_mtok", outp), ("cached_input_per_mtok", _num(row.get("cached_input_per_mtok"))),
117+ ("batch_input_per_mtok", _num(row.get("batch_input_per_mtok"))), ("batch_output_per_mtok", _num(row.get("batch_output_per_mtok")))):
118+ if v is not None and v < 0:
119+ out.append(Anomaly("negative_price", "critical", f"{label}: {k} is negative ({v})", row.get("model_id"), v, {"key": key, "field": k, "price_id": row.get("id")}))
120+ if v is not None and v > MAX_PRICE_PER_MTOK:
121+ out.append(Anomaly("price_too_high", "warning", f"{label}: {k} = ${v:g} per 1M tokens", row.get("model_id"), v, {"key": key, "field": k, "price_id": row.get("id")}))
122+ if inp is not None and outp is not None and outp == 0 and inp > 0 and not (row.get("features") or {}).get("free"):
123+ out.append(Anomaly("zero_output_price", "warning", f"{label}: output price is 0 while input is ${inp:g}", row.get("model_id"), 0, {"key": key, "price_id": row.get("id")}))
124+ if inp is not None and outp is not None and inp > 0 and outp > 0 and inp > outp * 4:
125+ out.append(Anomaly("input_gt_output_price", "info", f"{label}: input ${inp:g} is more than 4× output ${outp:g}", row.get("model_id"), inp, {"key": key, "price_id": row.get("id")}))
126+ cached = _num(row.get("cached_input_per_mtok"))
127+ if cached is not None and inp is not None and cached > inp:
128+ out.append(Anomaly("cached_gt_input_price", "warning", f"{label}: cached input ${cached:g} > input ${inp:g}", row.get("model_id"), cached, {"key": key, "price_id": row.get("id")}))
129+ return out
130+
131+
132+def check_price_movement(old: dict[str, Any], new: dict[str, Any]) -> list[Anomaly]:
133+ out: list[Anomaly] = []
134+ for k in ("input_per_mtok", "output_per_mtok"):
135+ a, b = _num(old.get(k)), _num(new.get(k))
136+ if a and b and a > 0 and b > 0 and (b / a > 100 or a / b > 100):
137+ out.append(Anomaly("price_jump_100x", "critical", f"{k} moved {a:g} → {b:g} (>100×)", new.get("model_id"), b,
138+ {"key": f"{new.get('model_id')}:{new.get('provider_id')}:{k}", "old": a, "new": b}))
139+ return out
140+
141+
142+def check_result(row: dict[str, Any]) -> list[Anomaly]:
143+ out: list[Anomaly] = []
144+ score = _num(row.get("score"))
145+ lo, hi = metric_bounds(row.get("metric"), row.get("unit"))
146+ label = f"{row.get('model_name') or row.get('model_id')} on {row.get('benchmark_name') or row.get('benchmark_id')}"
147+ if score is not None and hi is not None and score > hi + 1e-9:
148+ out.append(Anomaly("score_above_max", "critical", f"{label}: {score:g} > metric maximum {hi:g}", row.get("model_id"), score,
149+ {"key": row.get("id"), "result_id": row.get("id"), "metric": row.get("metric")}))
150+ if score is not None and lo is not None and score < lo - 1e-9:
151+ out.append(Anomaly("score_below_min", "critical", f"{label}: {score:g} < metric minimum {lo:g}", row.get("model_id"), score,
152+ {"key": row.get("id"), "result_id": row.get("id"), "metric": row.get("metric")}))
153+ ev = _date(row.get("evaluated_at"))
154+ rel = _date(row.get("model_release_date"))
155+ if ev and rel and ev < rel and (rel - ev).days > 45:
156+ out.append(Anomaly("evaluated_before_release", "warning", f"{label}: evaluated {ev} before model release {rel}", row.get("model_id"), ev.isoformat(),
157+ {"key": row.get("id"), "result_id": row.get("id")}))
158+ return out
159+
160+
161+def check_hardware(entity_id: str, name: str, attrs: dict[str, Any]) -> list[Anomaly]:
162+ out: list[Anomaly] = []
163+ mem = attrs.get("memory_gb")
164+ mems = mem if isinstance(mem, list) else [mem]
165+ for m in mems:
166+ v = _num(m)
167+ if v is not None and (v <= 0 or v > MAX_MEMORY_GB):
168+ out.append(Anomaly("memory_implausible", "critical", f"{name}: memory_gb {v:g} implausible", entity_id, v))
169+ bw = _num(attrs.get("memory_bandwidth_gbs"))
170+ if bw is not None and (bw <= 0 or bw > 100_000):
171+ out.append(Anomaly("bandwidth_implausible", "warning", f"{name}: memory_bandwidth_gbs {bw:g} implausible", entity_id, bw))
172+ tdp = _num(attrs.get("tdp_watts"))
173+ if tdp is not None and (tdp <= 0 or tdp > 200_000):
174+ out.append(Anomaly("tdp_implausible", "warning", f"{name}: tdp_watts {tdp:g} implausible", entity_id, tdp))
175+ return out
176+
177+
178+__all__ = ["Anomaly", "check_hardware", "check_model", "check_price", "check_price_movement", "check_result"]
added src/aiatlas/ontology/benchmarks.py +219 −0
@@ -0,0 +1,219 @@
1+"""Benchmark ontology — families, variants, metric bounds, trust levels and the comparability of two result configurations.
2+
3+Two scores are only *directly comparable* when they measure the same benchmark variant with the same metric under configurations that
4+do not change the task: same dataset revision/variant, same evaluator or harness class, same scaffold when the benchmark is agentic,
5+same shot/pass regime. Reasoning effort, sampling temperature and judge differences make them *partially comparable*. Different
6+variants (SWE-bench Verified vs Lite), different metrics, or different pass regimes are *not directly comparable*.
7+"""
8+from __future__ import annotations
9+
10+import hashlib
11+import json
12+import re
13+from typing import Any
14+
15+COMPARABLE, PARTIAL, NOT_COMPARABLE = "comparable", "partially-comparable", "not-comparable"
16+
17+# Benchmark family → known variants (registry keys). Used to build `variant_of` relations and the family attribute; the registry YAML
18+# carries the authoritative per-entry `family`/`variant` fields, this table is the fallback for benchmarks created from other sources.
19+FAMILIES: dict[str, dict[str, Any]] = {
20+ "swe-bench": {"label": "SWE-bench", "variants": {"swe-bench-full": "full", "swe-bench-verified": "Verified", "swe-bench-lite": "Lite",
21+ "swe-bench-multimodal": "Multimodal", "swe-bench-multilingual": "Multilingual", "swe-bench-pro": "Pro"}},
22+ "aime": {"label": "AIME", "variants": {"aime-2024": "2024", "aime-2025": "2025", "aime-2026": "2026"}},
23+ "mmlu": {"label": "MMLU", "variants": {"mmlu": "original", "mmlu-pro": "Pro", "mmlu-redux": "Redux", "mmmlu": "multilingual"}},
24+ "mmmu": {"label": "MMMU", "variants": {"mmmu": "original", "mmmu-pro": "Pro"}},
25+ "gpqa": {"label": "GPQA", "variants": {"gpqa": "main", "gpqa-diamond": "Diamond"}},
26+ "tau-bench": {"label": "τ-bench", "variants": {"tau-bench": "v1", "tau2-bench": "τ²"}},
27+ "livebench": {"label": "LiveBench", "variants": {"livebench": "global", "livebench-2": "global"}},
28+ "arc-agi": {"label": "ARC-AGI", "variants": {"arc-agi": "1", "arc-agi-2": "2", "arc-agi-3": "3"}},
29+ "humaneval": {"label": "HumanEval", "variants": {"humaneval": "original", "humaneval-plus": "Plus", "mbpp": None}},
30+ "livecodebench": {"label": "LiveCodeBench", "variants": {"livecodebench": "rolling"}},
31+ "math": {"label": "MATH", "variants": {"math": "full", "math-500": "500"}},
32+ "terminal-bench": {"label": "Terminal-Bench", "variants": {"terminal-bench": "1.0", "terminal-bench-2": "2.0"}},
33+ "ifeval": {"label": "IFEval", "variants": {"ifeval": "original", "ifbench": "IFBench"}},
34+ "artificial-analysis-intelligence-index": {"label": "Artificial Analysis Intelligence Index", "variants": {"artificial-analysis-intelligence-index": "index"}},
35+ "humanitys-last-exam": {"label": "Humanity's Last Exam", "variants": {"humanitys-last-exam": "full"}},
36+ "scicode": {"label": "SciCode", "variants": {"scicode": "main"}},
37+ "aider-polyglot": {"label": "Aider polyglot", "variants": {"aider-polyglot": "polyglot"}},
38+ "lmarena": {"label": "LMArena", "variants": {"lmarena-text": "text", "lmarena-vision": "vision", "lmarena-webdev": "webdev"}},
39+ "mteb": {"label": "MTEB", "variants": {"mteb": "v1", "mteb-v2": "v2", "mmteb": "MMTEB"}},
40+}
41+_KEY_TO_FAMILY: dict[str, tuple[str, str | None]] = {k: (fam, v) for fam, spec in FAMILIES.items() for k, v in spec["variants"].items()}
42+
43+
44+def family_of(benchmark_key: str) -> tuple[str | None, str | None]:
45+ """(family key, variant label) for a registry key / slug."""
46+ if benchmark_key in _KEY_TO_FAMILY:
47+ return _KEY_TO_FAMILY[benchmark_key]
48+ for fam in FAMILIES:
49+ if benchmark_key.startswith(fam):
50+ return fam, benchmark_key[len(fam):].strip("-") or None
51+ return None, None
52+
53+
54+# ---------------------------------------------------------------------------------------------- metrics
55+METRICS: dict[str, dict[str, Any]] = {
56+ # canonical metric → bounds and direction. `None` bound = unbounded.
57+ "accuracy": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
58+ "pass@1": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
59+ "pass^1": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
60+ "pass^k": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
61+ "resolved": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
62+ "pass_rate_2": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
63+ "percent_cases_well_formed": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
64+ "global_average": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
65+ "average score": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
66+ "mean score": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
67+ "index": {"min": 0, "max": 100, "higher_is_better": True, "unit": ""},
68+ "elo": {"min": 0, "max": None, "higher_is_better": True, "unit": ""},
69+ "score": {"min": None, "max": None, "higher_is_better": True, "unit": ""},
70+ "win_rate": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
71+ "f1": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
72+ "ndcg": {"min": 0, "max": 100, "higher_is_better": True, "unit": "%"},
73+ "perplexity": {"min": 0, "max": None, "higher_is_better": False, "unit": ""},
74+ "latency_ms": {"min": 0, "max": None, "higher_is_better": False, "unit": "ms"},
75+ "cost_usd": {"min": 0, "max": None, "higher_is_better": False, "unit": "USD"},
76+}
77+_METRIC_ALIASES = {
78+ "acc": "accuracy", "exact match": "accuracy", "em": "accuracy", "pass@1": "pass@1", "pass at 1": "pass@1", "pass^1": "pass^1", "pass1": "pass^1",
79+ "pass rate (2 attempts)": "pass_rate_2", "pass rate": "pass_rate_2", "percent resolved": "resolved", "% resolved": "resolved", "resolved rate": "resolved",
80+ "elo / bradley–terry score": "elo", "elo / bradley-terry score": "elo", "arena score": "elo", "bradley-terry": "elo", "mean score": "mean score",
81+ "average score": "average score", "global average": "global_average", "prompt-level strict accuracy": "accuracy", "strict accuracy": "accuracy",
82+ "intelligence index": "index", "composite": "index", "ppl": "perplexity",
83+}
84+
85+
86+def normalize_metric(raw: str | None) -> str | None:
87+ if not raw:
88+ return None
89+ s = raw.strip().lower()
90+ if s.startswith("category:"):
91+ return raw.strip() # LiveBench per-category averages keep their label; they are a separate metric each
92+ if s in METRICS:
93+ return s
94+ return _METRIC_ALIASES.get(s) or (s if re.fullmatch(r"[a-z0-9_@^%.\- ]+", s) else None)
95+
96+
97+def metric_bounds(metric: str | None, unit: str | None = None) -> tuple[float | None, float | None]:
98+ m = normalize_metric(metric)
99+ if m and m.startswith("category:"):
100+ return 0, 100
101+ spec = METRICS.get(m or "")
102+ if spec:
103+ return spec["min"], spec["max"]
104+ if unit == "%":
105+ return 0, 100
106+ return None, None
107+
108+
109+# ---------------------------------------------------------------------------------------------- trust levels
110+TRUST_LEVELS = ("official-model-card", "official-benchmark", "peer-reviewed", "independent-evaluator", "community", "unverified")
111+TRUST_LABELS = {
112+ "official-model-card": "Official model card / technical report (self-reported)",
113+ "official-benchmark": "Official benchmark leaderboard (submissions checked by the benchmark owner)",
114+ "peer-reviewed": "Peer-reviewed paper",
115+ "independent-evaluator": "Independent third-party evaluator",
116+ "community": "Community-run leaderboard or submission",
117+ "unverified": "Unverified / unknown provenance",
118+}
119+# source key (domain) → trust level of results it publishes
120+SOURCE_TRUST: dict[str, str] = {
121+ "swebench.com": "official-benchmark", "aider.chat": "official-benchmark", "livebench.ai": "official-benchmark", "artificialanalysis.ai": "independent-evaluator",
122+ "lmarena.ai": "independent-evaluator", "scale.com": "independent-evaluator", "epoch.ai": "independent-evaluator", "vals.ai": "independent-evaluator",
123+ "huggingface.co": "community", "github.com": "community", "arxiv.org": "peer-reviewed", "openreview.net": "peer-reviewed",
124+}
125+OFFICIAL_LAB_SOURCES = {"docs.claude.com", "platform.openai.com", "ai.google.dev", "docs.mistral.ai", "api-docs.deepseek.com", "docs.cohere.com",
126+ "docs.x.ai", "ai.meta.com", "llama.com", "qwenlm.github.io", "developer.nvidia.com", "machinelearning.apple.com"}
127+
128+
129+def trust_level(source_key: str | None, config: dict[str, Any] | None = None, *, extractor: str = "deterministic") -> str:
130+ cfg = config or {}
131+ if source_key in SOURCE_TRUST:
132+ level = SOURCE_TRUST[source_key]
133+ if level == "official-benchmark" and cfg.get("checked_by_swebench") is False:
134+ return "community"
135+ if level == "official-benchmark" and str(cfg.get("submission", "")).lower() in ("self-reported", "self reported", "unverified"):
136+ return "community"
137+ return level
138+ if source_key in OFFICIAL_LAB_SOURCES or cfg.get("self_reported") or cfg.get("source_kind") == "model_card":
139+ return "official-model-card"
140+ if extractor == "llm":
141+ return "unverified"
142+ return "unverified"
143+
144+
145+# ---------------------------------------------------------------------------------------------- comparability
146+# Config keys that change the *task* (must match for full comparability).
147+TASK_KEYS = ("variant", "board", "dataset_revision", "harness", "evaluator", "index_version", "release", "version", "subset", "split", "shots",
148+ "pass_count", "attempts", "language", "scaffold", "agent", "system")
149+# Config keys that change the *conditions* (mismatch → partially comparable).
150+CONDITION_KEYS = ("reasoning_effort", "reasoning", "thinking_budget", "temperature", "judge", "tools", "tool_use", "max_tokens", "context_length",
151+ "sampling", "aggregation", "edit_format", "model_tag")
152+# Keys that are pure bookkeeping (never affect comparability).
153+IGNORED_KEYS = {"aa_slug", "livebench_model_id", "api_model_id", "date", "submission", "checked_by_swebench", "open_source_system", "system_org",
154+ "total_cost_usd", "cost_per_instance_usd", "dirname", "command", "versions", "test_cases", "seconds_per_case", "estimated",
155+ "livebench_hf_link", "source_kind", "self_reported", "subtasks", "notes", "url"}
156+
157+
158+def _clean(v: Any) -> Any:
159+ if isinstance(v, str):
160+ return v.strip().lower()
161+ return v
162+
163+
164+def config_key(config: dict[str, Any] | None, metric: str | None = None) -> str:
165+ """Stable hash of the comparability-relevant part of a result configuration (task keys + metric)."""
166+ cfg = config or {}
167+ core = {k: _clean(cfg[k]) for k in TASK_KEYS if k in cfg and cfg[k] not in (None, "", [], {})}
168+ core["metric"] = normalize_metric(metric) or ""
169+ return hashlib.sha1(json.dumps(core, sort_keys=True, default=str).encode()).hexdigest()[:12]
170+
171+
172+def comparability(a_cfg: dict[str, Any] | None, b_cfg: dict[str, Any] | None, a_metric: str | None = None, b_metric: str | None = None,
173+ *, same_benchmark: bool = True) -> tuple[str, list[str]]:
174+ """Return (level, reasons). `same_benchmark` False → not comparable outright."""
175+ reasons: list[str] = []
176+ if not same_benchmark:
177+ return NOT_COMPARABLE, ["different benchmark variants"]
178+ ma, mb = normalize_metric(a_metric), normalize_metric(b_metric)
179+ if ma != mb and (ma or mb):
180+ return NOT_COMPARABLE, [f"different metrics ({a_metric} vs {b_metric})"]
181+ a, b = a_cfg or {}, b_cfg or {}
182+ for k in TASK_KEYS:
183+ if k in a or k in b:
184+ va, vb = _clean(a.get(k)), _clean(b.get(k))
185+ if va != vb and va not in (None, "") and vb not in (None, ""):
186+ reasons.append(f"{k}: {a.get(k)} vs {b.get(k)}")
187+ if reasons:
188+ return NOT_COMPARABLE, reasons
189+ for k in CONDITION_KEYS:
190+ if k in a or k in b:
191+ va, vb = _clean(a.get(k)), _clean(b.get(k))
192+ if va != vb:
193+ reasons.append(f"{k}: {a.get(k) if k in a else 'unspecified'} vs {b.get(k) if k in b else 'unspecified'}")
194+ if reasons:
195+ return PARTIAL, reasons
196+ return COMPARABLE, ["same variant, metric and evaluation conditions"]
197+
198+
199+def variant_from_config(config: dict[str, Any] | None) -> str | None:
200+ cfg = config or {}
201+ for k in ("variant", "board", "subset", "split"):
202+ v = cfg.get(k)
203+ if isinstance(v, str) and v.strip():
204+ return v.strip()
205+ return None
206+
207+
208+def run_group_from_config(config: dict[str, Any] | None) -> str | None:
209+ """The 'run' a result belongs to (a LiveBench release, an aider run date, an AA index version…). One current row per run group."""
210+ cfg = config or {}
211+ for k in ("release", "index_version", "version", "date", "dataset_revision"):
212+ v = cfg.get(k)
213+ if v not in (None, ""):
214+ return str(v)
215+ return None
216+
217+
218+__all__ = ["COMPARABLE", "CONDITION_KEYS", "FAMILIES", "IGNORED_KEYS", "METRICS", "NOT_COMPARABLE", "PARTIAL", "TASK_KEYS", "TRUST_LABELS", "TRUST_LEVELS",
219+ "comparability", "config_key", "family_of", "metric_bounds", "normalize_metric", "run_group_from_config", "trust_level", "variant_from_config"]
added src/aiatlas/ontology/licenses.py +276 −0
@@ -0,0 +1,276 @@
1+"""Licence ontology.
2+
3+A licence is identified by its SPDX id when one exists (`Apache-2.0`, `MIT`, `CC-BY-NC-4.0`) and by a stable AI Atlas key otherwise
4+(`llama-3.1-community`, `gemma-terms`, `openrail-m`). Every entry states, as booleans (or `None` when the text is ambiguous), what the
5+licence permits — so the UI can say "commercial use allowed / derivatives allowed / redistribution allowed / acceptable-use restrictions"
6+instead of the meaningless "open source" for a model whose weights merely download.
7+
8+`normalize_license(raw)` maps the strings observed in the wild (Hugging Face `license:` tags, docs pages, GitHub) to a canonical key.
9+Unknown strings return `None` — the raw label is kept separately by the writer (`license_raw`); nothing is guessed.
10+"""
11+from __future__ import annotations
12+
13+import re
14+from dataclasses import dataclass, field
15+
16+# Categories are about *what the text allows*, not about ideology.
17+CATEGORIES = ("permissive", "copyleft", "creative-commons", "responsible-ai", "community", "research-only", "proprietary", "unknown")
18+
19+
20+@dataclass(frozen=True)
21+class LicenseInfo:
22+ key: str # canonical key (SPDX id when applicable)
23+ label: str # display name
24+ category: str # one of CATEGORIES
25+ spdx: str | None = None # SPDX identifier when the licence is in the SPDX list
26+ url: str | None = None # canonical text
27+ commercial_use: bool | None = None # may be used commercially
28+ redistribution: bool | None = None # weights/code may be redistributed
29+ derivatives: bool | None = None # fine-tunes / modifications allowed
30+ hosting_restrictions: bool | None = None # restrictions on hosting the model as a service for third parties (incl. MAU caps)
31+ attribution: bool | None = None # attribution / notice required
32+ acceptable_use: bool | None = None # an acceptable-use policy restricts what the model may do
33+ osi_approved: bool = False # OSI-approved open-source licence (code) — never true for custom model licences
34+ weights_downloadable: bool = True # False for proprietary API-only terms
35+ aliases: tuple[str, ...] = field(default_factory=tuple)
36+
37+ def as_dict(self) -> dict[str, object]:
38+ return {
39+ "key": self.key, "label": self.label, "category": self.category, "spdx": self.spdx, "url": self.url,
40+ "commercial_use": self.commercial_use, "redistribution": self.redistribution, "derivatives": self.derivatives,
41+ "hosting_restrictions": self.hosting_restrictions, "attribution": self.attribution, "acceptable_use": self.acceptable_use,
42+ "osi_approved": self.osi_approved, "weights_downloadable": self.weights_downloadable,
43+ }
44+
45+
46+def _permissive(key: str, label: str, spdx: str, url: str, **kw: object) -> LicenseInfo:
47+ base: dict[str, object] = dict(key=key, label=label, category="permissive", spdx=spdx, url=url, commercial_use=True, redistribution=True,
48+ derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False, osi_approved=True)
49+ base.update(kw)
50+ return LicenseInfo(**base) # type: ignore[arg-type]
51+
52+
53+LICENSES: dict[str, LicenseInfo] = {}
54+
55+
56+def _add(info: LicenseInfo) -> None:
57+ LICENSES[info.key] = info
58+
59+
60+# ---------------------------------------------------------------------------------------------- permissive / copyleft (OSI)
61+_add(_permissive("Apache-2.0", "Apache License 2.0", "Apache-2.0", "https://www.apache.org/licenses/LICENSE-2.0",
62+ aliases=("apache 2.0", "apache-2", "apache2", "apache license 2.0", "apache license, version 2.0", "apache")))
63+_add(_permissive("MIT", "MIT License", "MIT", "https://opensource.org/license/mit", aliases=("mit license", "the mit license")))
64+_add(LicenseInfo(key="MIT-Modified", label="Modified MIT License", category="permissive", spdx=None, commercial_use=True, redistribution=True,
65+ derivatives=True, hosting_restrictions=None, attribution=True, acceptable_use=None, aliases=("modified mit", "mit-modified")))
66+_add(_permissive("BSD-3-Clause", "BSD 3-Clause License", "BSD-3-Clause", "https://opensource.org/license/bsd-3-clause",
67+ aliases=("bsd-3", "bsd 3-clause", "bsd3", "new bsd", "modified bsd", "bsd")))
68+_add(_permissive("BSD-2-Clause", "BSD 2-Clause License", "BSD-2-Clause", "https://opensource.org/license/bsd-2-clause", aliases=("bsd-2", "simplified bsd")))
69+_add(_permissive("ISC", "ISC License", "ISC", "https://opensource.org/license/isc-license-txt"))
70+_add(_permissive("Unlicense", "The Unlicense", "Unlicense", "https://unlicense.org", attribution=False, aliases=("unlicense",)))
71+_add(_permissive("0BSD", "Zero-Clause BSD", "0BSD", "https://opensource.org/license/0bsd", attribution=False))
72+_add(LicenseInfo(key="GPL-3.0", label="GNU GPL v3", category="copyleft", spdx="GPL-3.0-only", url="https://www.gnu.org/licenses/gpl-3.0.html",
73+ commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False,
74+ osi_approved=True, aliases=("gpl-3.0-only", "gpl-3.0-or-later", "gplv3", "gpl3", "gpl-3")))
75+_add(LicenseInfo(key="GPL-2.0", label="GNU GPL v2", category="copyleft", spdx="GPL-2.0-only", url="https://www.gnu.org/licenses/old-licenses/gpl-2.0.html",
76+ commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False,
77+ osi_approved=True, aliases=("gpl-2.0-only", "gpl-2.0-or-later", "gplv2", "gpl2", "gpl-2")))
78+_add(LicenseInfo(key="LGPL-3.0", label="GNU LGPL v3", category="copyleft", spdx="LGPL-3.0-only", url="https://www.gnu.org/licenses/lgpl-3.0.html",
79+ commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False,
80+ osi_approved=True, aliases=("lgpl-3.0-only", "lgpl-3.0-or-later", "lgplv3", "lgpl")))
81+_add(LicenseInfo(key="AGPL-3.0", label="GNU AGPL v3", category="copyleft", spdx="AGPL-3.0-only", url="https://www.gnu.org/licenses/agpl-3.0.html",
82+ commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=True, attribution=True, acceptable_use=False,
83+ osi_approved=True, aliases=("agpl-3.0-only", "agpl-3.0-or-later", "agplv3", "agpl")))
84+_add(LicenseInfo(key="MPL-2.0", label="Mozilla Public License 2.0", category="copyleft", spdx="MPL-2.0", url="https://www.mozilla.org/MPL/2.0/",
85+ commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False,
86+ osi_approved=True, aliases=("mpl", "mozilla public license 2.0")))
87+_add(LicenseInfo(key="EPL-2.0", label="Eclipse Public License 2.0", category="copyleft", spdx="EPL-2.0", url="https://www.eclipse.org/legal/epl-2.0/",
88+ commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False, osi_approved=True))
89+
90+# ---------------------------------------------------------------------------------------------- Creative Commons
91+def _cc(key: str, label: str, url: str, *, nc: bool, nd: bool, sa: bool, aliases: tuple[str, ...]) -> LicenseInfo:
92+ return LicenseInfo(key=key, label=label, category="creative-commons", spdx=key, url=url, commercial_use=not nc, redistribution=True,
93+ derivatives=not nd, hosting_restrictions=nc or None, attribution=True, acceptable_use=False, osi_approved=False, aliases=aliases)
94+
95+
96+_add(_cc("CC-BY-4.0", "Creative Commons Attribution 4.0", "https://creativecommons.org/licenses/by/4.0/", nc=False, nd=False, sa=False,
97+ aliases=("cc by 4.0", "cc-by", "cc by", "cc-by-4", "creative commons attribution 4.0")))
98+_add(_cc("CC-BY-SA-4.0", "Creative Commons Attribution-ShareAlike 4.0", "https://creativecommons.org/licenses/by-sa/4.0/", nc=False, nd=False, sa=True,
99+ aliases=("cc by-sa 4.0", "cc-by-sa", "cc by-sa")))
100+_add(_cc("CC-BY-NC-4.0", "Creative Commons Attribution-NonCommercial 4.0", "https://creativecommons.org/licenses/by-nc/4.0/", nc=True, nd=False, sa=False,
101+ aliases=("cc by-nc 4.0", "cc-by-nc", "cc by-nc", "cc-by-nc-4", "creative commons attribution non commercial 4.0")))
102+_add(_cc("CC-BY-NC-SA-4.0", "Creative Commons Attribution-NonCommercial-ShareAlike 4.0", "https://creativecommons.org/licenses/by-nc-sa/4.0/", nc=True, nd=False, sa=True,
103+ aliases=("cc by-nc-sa 4.0", "cc-by-nc-sa", "cc by-nc-sa")))
104+_add(_cc("CC-BY-NC-ND-4.0", "Creative Commons Attribution-NonCommercial-NoDerivatives 4.0", "https://creativecommons.org/licenses/by-nc-nd/4.0/", nc=True, nd=True, sa=False,
105+ aliases=("cc by-nc-nd 4.0", "cc-by-nc-nd")))
106+_add(_cc("CC-BY-ND-4.0", "Creative Commons Attribution-NoDerivatives 4.0", "https://creativecommons.org/licenses/by-nd/4.0/", nc=False, nd=True, sa=False,
107+ aliases=("cc by-nd 4.0", "cc-by-nd")))
108+_add(LicenseInfo(key="CC0-1.0", label="Creative Commons Zero 1.0 (public domain dedication)", category="creative-commons", spdx="CC0-1.0",
109+ url="https://creativecommons.org/publicdomain/zero/1.0/", commercial_use=True, redistribution=True, derivatives=True,
110+ hosting_restrictions=False, attribution=False, acceptable_use=False, aliases=("cc0", "cc0 1.0", "public domain")))
111+_add(LicenseInfo(key="ODC-By-1.0", label="Open Data Commons Attribution", category="creative-commons", spdx="ODC-By-1.0", url="https://opendatacommons.org/licenses/by/1-0/",
112+ commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=True, acceptable_use=False, aliases=("odc-by",)))
113+_add(LicenseInfo(key="PDDL-1.0", label="Open Data Commons Public Domain Dedication", category="creative-commons", spdx="PDDL-1.0", url="https://opendatacommons.org/licenses/pddl/",
114+ commercial_use=True, redistribution=True, derivatives=True, hosting_restrictions=False, attribution=False, acceptable_use=False, aliases=("pddl",)))
115+
116+# ---------------------------------------------------------------------------------------------- responsible-AI (RAIL) family
117+def _rail(key: str, label: str, url: str, aliases: tuple[str, ...]) -> LicenseInfo:
118+ return LicenseInfo(key=key, label=label, category="responsible-ai", spdx=None, url=url, commercial_use=True, redistribution=True, derivatives=True,
119+ hosting_restrictions=False, attribution=True, acceptable_use=True, osi_approved=False, aliases=aliases)
120+
121+
122+_add(_rail("OpenRAIL-M", "Open RAIL-M", "https://www.licenses.ai/ai-licenses", ("openrail", "openrail-m", "open rail-m", "creativeml-openrail-m", "creativeml openrail-m")))
123+_add(_rail("OpenRAIL++-M", "Open RAIL++-M (Stable Diffusion XL)", "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md", ("openrail++", "openrail++-m")))
124+_add(_rail("BigScience-BLOOM-RAIL-1.0", "BigScience BLOOM RAIL 1.0", "https://huggingface.co/spaces/bigscience/license", ("bigscience-bloom-rail-1.0", "bloom-rail", "bigscience-openrail-m")))
125+_add(_rail("BigCode-OpenRAIL-M", "BigCode Open RAIL-M", "https://www.bigcode-project.org/docs/pages/bigcode-openrail/", ("bigcode-openrail-m",)))
126+
127+# ---------------------------------------------------------------------------------------------- community / custom model licences (weights download, restrictions apply)
128+def _community(key: str, label: str, url: str, aliases: tuple[str, ...], *, commercial: bool | None = True, hosting: bool | None = True,
129+ derivatives: bool | None = True, aup: bool | None = True) -> LicenseInfo:
130+ return LicenseInfo(key=key, label=label, category="community", spdx=None, url=url, commercial_use=commercial, redistribution=True, derivatives=derivatives,
131+ hosting_restrictions=hosting, attribution=True, acceptable_use=aup, osi_approved=False, aliases=aliases)
132+
133+
134+_add(_community("Llama-2-Community", "Llama 2 Community License", "https://ai.meta.com/llama/license/", ("llama2", "llama-2", "llama 2 community license", "llama2 community license")))
135+_add(_community("Llama-3-Community", "Llama 3 Community License", "https://llama.meta.com/llama3/license/", ("llama3", "llama-3", "llama 3 community license", "meta llama 3 community license")))
136+_add(_community("Llama-3.1-Community", "Llama 3.1 Community License", "https://llama.meta.com/llama3_1/license/", ("llama3.1", "llama-3.1", "llama 3.1 community license")))
137+_add(_community("Llama-3.2-Community", "Llama 3.2 Community License", "https://www.llama.com/llama3_2/license/", ("llama3.2", "llama-3.2", "llama 3.2 community license")))
138+_add(_community("Llama-3.3-Community", "Llama 3.3 Community License", "https://www.llama.com/llama3_3/license/", ("llama3.3", "llama-3.3", "llama 3.3 community license")))
139+_add(_community("Llama-4-Community", "Llama 4 Community License", "https://www.llama.com/llama4/license/", ("llama4", "llama-4", "llama 4 community license")))
140+_add(_community("Gemma-Terms", "Gemma Terms of Use", "https://ai.google.dev/gemma/terms", ("gemma", "gemma terms of use", "gemma license", "gemma-terms")))
141+_add(_community("Mistral-Research", "Mistral AI Research License", "https://mistral.ai/licenses/MRL-0.1.md", ("mrl", "mistral research license", "mistral ai research license", "mrl-0.1"),
142+ commercial=False, hosting=True, aup=True))
143+_add(_community("Mistral-Non-Production", "Mistral AI Non-Production License", "https://mistral.ai/licenses/MNPL-0.1.md", ("mnpl", "mnpl-0.1", "mistral ai non-production license"),
144+ commercial=False, hosting=True, aup=True))
145+_add(_community("Qwen-Research", "Qwen Research License", "https://huggingface.co/Qwen/Qwen2.5-3B/blob/main/LICENSE", ("qwen research license", "qwen-research"), commercial=False))
146+_add(_community("Tongyi-Qianwen", "Tongyi Qianwen License", "https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT", ("tongyi-qianwen", "tongyi qianwen license", "qianwen"),
147+ commercial=True, hosting=True))
148+_add(_community("DeepSeek-Model", "DeepSeek Model License", "https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/LICENSE-MODEL", ("deepseek", "deepseek license", "deepseek model license"), hosting=False))
149+_add(_community("NVIDIA-Open-Model", "NVIDIA Open Model License", "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/",
150+ ("nvidia open model license", "nvidia-open-model-license", "nvidia open model"), hosting=False))
151+_add(_community("NVIDIA-Community-Model", "NVIDIA Community Model License", "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-community-models-license/",
152+ ("nvidia community model license", "nvidia-community-model-license"), hosting=None))
153+_add(_community("Apple-AMLR", "Apple Sample Code / ML Research License", "https://github.com/apple/ml-fastvlm/blob/main/LICENSE_MODEL", ("apple-amlr", "apple-ascl", "apple ml research license"),
154+ commercial=False, hosting=True, derivatives=True))
155+_add(_community("Cohere-CC-BY-NC", "Cohere Labs Acceptable Use (CC-BY-NC)", "https://cohere.com/c4ai-cc-by-nc-license", ("cohere", "c4ai", "cc-by-nc-4.0 + acceptable use"), commercial=False))
156+_add(_community("Stability-Community", "Stability AI Community License", "https://stability.ai/community-license-agreement", ("stabilityai-ai-community", "stability ai community license", "stabilityai-community"),
157+ commercial=True, hosting=True))
158+_add(_community("Stability-Non-Commercial-Research", "Stability AI Non-Commercial Research Community License", "https://stability.ai/license",
159+ ("stabilityai-nc-research-community", "stability ai non-commercial research community"), commercial=False))
160+_add(_community("Falcon-LLM", "Falcon LLM License (TII)", "https://falconllm.tii.ae/falcon-terms-and-conditions.html", ("falcon-llm-license", "tii falcon license", "falcon"), hosting=None))
161+_add(_community("Yi", "Yi Series Models Community License", "https://github.com/01-ai/Yi/blob/main/MODEL_LICENSE_AGREEMENT.txt", ("yi-license", "yi license", "yi"), commercial=True))
162+_add(_community("ChatGLM", "ChatGLM / GLM Model License", "https://github.com/THUDM/ChatGLM3/blob/main/MODEL_LICENSE", ("glm-4", "chatglm", "glm license", "zhipu"), commercial=True))
163+_add(_community("Phi", "Microsoft Phi (MIT with terms)", "https://huggingface.co/microsoft/phi-4/blob/main/LICENSE", ("phi",), commercial=True, hosting=False))
164+_add(_community("LFM-Open-1.0", "LFM Open License v1.0 (Liquid AI)", "https://www.liquid.ai/lfm-license", ("lfm1.0", "lfm-1.0", "lfm open license"), commercial=True, hosting=True))
165+_add(_community("Jamba-Open-Model", "Jamba Open Model License (AI21)", "https://www.ai21.com/jamba-open-model-license/", ("jamba open model license", "jamba"), commercial=True))
166+_add(_community("Kimi-Modified-MIT", "Kimi Modified MIT License (Moonshot)", "https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/LICENSE", ("kimi modified mit", "moonshot modified mit"),
167+ commercial=True, hosting=True, aup=False))
168+_add(_community("SEED-Model", "ByteDance Seed Model License", "https://github.com/ByteDance-Seed", ("seed license", "bytedance seed"), commercial=None))
169+_add(_community("Hunyuan-Community", "Tencent Hunyuan Community License", "https://github.com/Tencent/Tencent-Hunyuan-Large/blob/main/License.docx", ("tencent-hunyuan-community", "hunyuan", "tencent hunyuan community license"),
170+ commercial=True, hosting=True))
171+_add(_community("IBM-Granite", "IBM Granite (Apache-2.0 with usage terms)", "https://www.ibm.com/granite", ("granite",), commercial=True, hosting=False, aup=True))
172+_add(_community("Custom-Community", "Custom community licence (see source)", "", ("custom community", "community license", "other-community"), commercial=None, hosting=None, derivatives=None, aup=None))
173+
174+# ---------------------------------------------------------------------------------------------- research-only / non-commercial custom
175+_add(LicenseInfo(key="Research-Only", label="Research / non-commercial licence (custom)", category="research-only", spdx=None, commercial_use=False, redistribution=None,
176+ derivatives=None, hosting_restrictions=True, attribution=True, acceptable_use=True, aliases=("research", "research only", "research-only", "non-commercial", "noncommercial",
177+ "academic", "academic use only", "non commercial")))
178+_add(LicenseInfo(key="Meta-Research", label="Meta Research License", category="research-only", spdx=None, url="https://ai.meta.com/resources/models-and-libraries/",
179+ commercial_use=False, redistribution=False, derivatives=True, hosting_restrictions=True, attribution=True, acceptable_use=True,
180+ aliases=("meta research license", "fair noncommercial research license", "fair-noncommercial")))
181+
182+# ---------------------------------------------------------------------------------------------- proprietary
183+_add(LicenseInfo(key="Proprietary", label="Proprietary (API / terms of service)", category="proprietary", spdx=None, commercial_use=None, redistribution=False,
184+ derivatives=False, hosting_restrictions=True, attribution=None, acceptable_use=True, weights_downloadable=False,
185+ aliases=("proprietary", "closed", "closed source", "commercial", "terms of service", "tos", "api terms", "license: proprietary")))
186+_add(LicenseInfo(key="Other", label="Other (unclassified licence)", category="unknown", spdx=None, aliases=("other", "unknown", "custom", "see model card", "license: other")))
187+
188+
189+# ---------------------------------------------------------------------------------------------- normalisation
190+_ALIAS_INDEX: dict[str, str] = {}
191+for _info in LICENSES.values():
192+ _ALIAS_INDEX[_info.key.lower()] = _info.key
193+ if _info.spdx:
194+ _ALIAS_INDEX[_info.spdx.lower()] = _info.key
195+ for _a in _info.aliases:
196+ _ALIAS_INDEX[_a.lower()] = _info.key
197+
198+_STRIP = re.compile(r"\s+|[_]")
199+_PATTERNS: list[tuple[re.Pattern[str], str]] = [
200+ (re.compile(r"^apache[\s\-]*(license)?[\s\-,]*(version)?[\s\-]*2(\.0)?$", re.I), "Apache-2.0"),
201+ (re.compile(r"^llama[\s\-]*4", re.I), "Llama-4-Community"),
202+ (re.compile(r"^llama[\s\-]*3[.\-]3", re.I), "Llama-3.3-Community"),
203+ (re.compile(r"^llama[\s\-]*3[.\-]2", re.I), "Llama-3.2-Community"),
204+ (re.compile(r"^llama[\s\-]*3[.\-]1", re.I), "Llama-3.1-Community"),
205+ (re.compile(r"^(meta[\s\-]*)?llama[\s\-]*3", re.I), "Llama-3-Community"),
206+ (re.compile(r"^(meta[\s\-]*)?llama[\s\-]*2", re.I), "Llama-2-Community"),
207+ (re.compile(r"^gemma", re.I), "Gemma-Terms"),
208+ (re.compile(r"^cc[\s\-]*by[\s\-]*nc[\s\-]*sa", re.I), "CC-BY-NC-SA-4.0"),
209+ (re.compile(r"^cc[\s\-]*by[\s\-]*nc[\s\-]*nd", re.I), "CC-BY-NC-ND-4.0"),
210+ (re.compile(r"^cc[\s\-]*by[\s\-]*nc", re.I), "CC-BY-NC-4.0"),
211+ (re.compile(r"^cc[\s\-]*by[\s\-]*sa", re.I), "CC-BY-SA-4.0"),
212+ (re.compile(r"^cc[\s\-]*by[\s\-]*nd", re.I), "CC-BY-ND-4.0"),
213+ (re.compile(r"^cc[\s\-]*by", re.I), "CC-BY-4.0"),
214+ (re.compile(r"^cc0", re.I), "CC0-1.0"),
215+ (re.compile(r"^(creativeml[\s\-]*)?openrail\+\+", re.I), "OpenRAIL++-M"),
216+ (re.compile(r"^(creativeml[\s\-]*)?openrail", re.I), "OpenRAIL-M"),
217+ (re.compile(r"^bigscience", re.I), "BigScience-BLOOM-RAIL-1.0"),
218+ (re.compile(r"^bigcode", re.I), "BigCode-OpenRAIL-M"),
219+ (re.compile(r"^(gpl|gnu general public license)[\s\-]*v?3", re.I), "GPL-3.0"),
220+ (re.compile(r"^(gpl|gnu general public license)[\s\-]*v?2", re.I), "GPL-2.0"),
221+ (re.compile(r"^agpl", re.I), "AGPL-3.0"),
222+ (re.compile(r"^lgpl", re.I), "LGPL-3.0"),
223+ (re.compile(r"^bsd[\s\-]*3", re.I), "BSD-3-Clause"),
224+ (re.compile(r"^bsd[\s\-]*2", re.I), "BSD-2-Clause"),
225+ (re.compile(r"^mit\b", re.I), "MIT"),
226+ (re.compile(r"^mistral.*research", re.I), "Mistral-Research"),
227+ (re.compile(r"^mistral.*non[\s\-]*production", re.I), "Mistral-Non-Production"),
228+ (re.compile(r"^nvidia.*open.*model", re.I), "NVIDIA-Open-Model"),
229+ (re.compile(r"^nvidia.*community", re.I), "NVIDIA-Community-Model"),
230+ (re.compile(r"^deepseek", re.I), "DeepSeek-Model"),
231+ (re.compile(r"^qwen.*research", re.I), "Qwen-Research"),
232+ (re.compile(r"^tongyi", re.I), "Tongyi-Qianwen"),
233+ (re.compile(r"^stability.*(nc|non[\s\-]*commercial)", re.I), "Stability-Non-Commercial-Research"),
234+ (re.compile(r"^stability", re.I), "Stability-Community"),
235+ (re.compile(r"^apple", re.I), "Apple-AMLR"),
236+ (re.compile(r"^(tencent[\s\-]*)?hunyuan", re.I), "Hunyuan-Community"),
237+ (re.compile(r"^lfm", re.I), "LFM-Open-1.0"),
238+ (re.compile(r"^jamba", re.I), "Jamba-Open-Model"),
239+ (re.compile(r"^falcon", re.I), "Falcon-LLM"),
240+ (re.compile(r"^yi\b", re.I), "Yi"),
241+ (re.compile(r"(research|academic|non[\s\-]*commercial|noncommercial)", re.I), "Research-Only"),
242+ (re.compile(r"(proprietary|closed|terms of service)", re.I), "Proprietary"),
243+]
244+
245+
246+def normalize_license(raw: str | None) -> str | None:
247+ """Map an observed licence label to a canonical key, or `None` when it cannot be classified (keep the raw string separately)."""
248+ if raw is None:
249+ return None
250+ s = str(raw).strip()
251+ if not s:
252+ return None
253+ low = s.lower().strip().strip(".")
254+ low = re.sub(r"^license:\s*", "", low)
255+ if low in _ALIAS_INDEX:
256+ return _ALIAS_INDEX[low]
257+ compact = _STRIP.sub("-", low).replace("--", "-")
258+ if compact in _ALIAS_INDEX:
259+ return _ALIAS_INDEX[compact]
260+ for pattern, key in _PATTERNS:
261+ if pattern.search(low):
262+ return key
263+ return None
264+
265+
266+def license_info(key_or_raw: str | None) -> LicenseInfo | None:
267+ key = key_or_raw if key_or_raw in LICENSES else normalize_license(key_or_raw)
268+ return LICENSES.get(key) if key else None
269+
270+
271+def license_label(key_or_raw: str | None) -> str | None:
272+ info = license_info(key_or_raw)
273+ return info.label if info else (str(key_or_raw) if key_or_raw else None)
274+
275+
276+__all__ = ["CATEGORIES", "LICENSES", "LicenseInfo", "license_info", "license_label", "normalize_license"]
added src/aiatlas/ontology/models.py +271 −0
@@ -0,0 +1,271 @@
1+"""Model identity ontology — the rules that separate a MODEL (release) from its ARTIFACTS (checkpoints, conversions, quantisations),
2+its EVALUATION VARIANTS (reasoning-effort settings of the same weights) and its FAMILY.
3+
4+ MODEL FAMILY Llama 4 · Qwen3.6 · Claude · Gemini
5+ └ MODEL Llama 4 Maverick · Qwen3.6-35B-A3B · Claude Fable 5.1
6+ └ ARTIFACT meta-llama/Llama-4-Maverick-17B-128E-Instruct (official checkpoint) · unsloth/…-GGUF (third-party quantisation)
7+ └ DEPLOYMENT OpenRouter meta-llama/llama-4-maverick · Together meta-llama/Llama-4-Maverick (= prices rows)
8+
9+Everything here is deterministic string analysis. It never *asserts* a relation on its own: the resolution service uses these
10+signals together with identifiers, organisations, `base_model` metadata and dates, and parks anything ambiguous in the review queue.
11+"""
12+from __future__ import annotations
13+
14+import re
15+from dataclasses import dataclass, field
16+
17+# ---------------------------------------------------------------------------------------------- quantisation / precision formats
18+QUANT_FORMATS: dict[str, str] = {
19+ # token (lowercase, matched on word boundaries) → canonical format
20+ "gguf": "gguf", "ggml": "gguf", "q2_k": "gguf", "q3_k_m": "gguf", "q4_k_m": "gguf", "q4_k_s": "gguf", "q5_k_m": "gguf", "q6_k": "gguf", "q8_0": "gguf",
21+ "q4_0": "gguf", "q4_1": "gguf", "q5_0": "gguf", "iq4_xs": "gguf", "iq3_m": "gguf", "iq2_m": "gguf", "ud-q4_k_xl": "gguf",
22+ "awq": "awq", "w4a16": "awq", "gptq": "gptq", "exl2": "exl2", "exl3": "exl3", "mlx": "mlx", "onnx": "onnx", "openvino": "openvino", "coreml": "coreml",
23+ "tensorrt": "tensorrt", "trt": "tensorrt", "trt-llm": "tensorrt", "bnb": "bnb", "bitsandbytes": "bnb", "nf4": "bnb", "4bit": "int4", "8bit": "int8",
24+ "int4": "int4", "int8": "int8", "w8a8": "int8", "w8a16": "int8", "fp8": "fp8", "fp8-kv": "fp8", "e4m3": "fp8", "nvfp4": "nvfp4", "mxfp4": "mxfp4",
25+ "mxfp8": "mxfp8", "fp4": "fp4", "quark": "quark", "quantized": "quantized", "quant": "quantized", "qat": "qat", "hqq": "hqq", "aqlm": "aqlm", "eetq": "eetq",
26+ "smoothquant": "smoothquant", "compressed-tensors": "compressed-tensors", "marlin": "gptq", "gptq-int4": "gptq", "autoround": "autoround",
27+ "2bit": "int2", "3bit": "int3", "5bit": "int5", "6bit": "int6", "2-bit": "int2", "3-bit": "int3", "4-bit": "int4", "5-bit": "int5", "6-bit": "int6", "8-bit": "int8",
28+}
29+# Full-precision dtype tokens: a "-BF16" or "-FP16" repo is a *conversion/packaging* of the same weights, not a quantisation.
30+PRECISION_FORMATS = {"bf16", "fp16", "fp32", "f16", "f32", "float16", "bfloat16", "half"}
31+ARTIFACT_PACKAGING = {"safetensors", "pytorch", "pth", "ckpt", "jax", "flax", "tf", "tflite", "litert", "gguf", "mlx", "onnx", "coreml", "openvino", "tensorrt"}
32+
33+# Third-party organisations that (almost) only publish conversions/quantisations of other people's models.
34+CONVERTER_ORGS = {
35+ "unsloth", "bartowski", "mlx-community", "thebloke", "lmstudio-community", "qwen-community", "mradermacher", "quantfactory", "nousresearch-quant",
36+ "mistral-community", "turboderp", "casperhansen", "neuralmagic", "redhatai", "amd", "intel", "nvidia-quant", "ggml-org", "second-state",
37+ "mlx-vision", "cortexso", "gaianet", "bunnycore", "dranger003", "ubergarm", "anthracite-org", "modelcloud", "jinaai-quant", "ai-forever-quant",
38+ "lmstudio", "ollama", "ggerganov", "mlc-ai", "onnx-community", "onnxmodelzoo", "kaitchup", "thedrummer-quant",
39+}
40+
41+# ---------------------------------------------------------------------------------------------- evaluation-effort variants (same weights, different setting)
42+# Suffixes appended by evaluators (Artificial Analysis, LiveBench…) to a model slug to denote a *configuration* of the model.
43+EFFORT_SUFFIXES: dict[str, dict[str, str]] = {
44+ "xhigh": {"reasoning_effort": "xhigh"}, "x-high": {"reasoning_effort": "xhigh"}, "extra-high": {"reasoning_effort": "xhigh"},
45+ "high": {"reasoning_effort": "high"}, "medium": {"reasoning_effort": "medium"}, "low": {"reasoning_effort": "low"}, "minimal": {"reasoning_effort": "minimal"},
46+ "max": {"reasoning_effort": "max"}, "max-effort": {"reasoning_effort": "max"}, "high-effort": {"reasoning_effort": "high"}, "low-effort": {"reasoning_effort": "low"},
47+ "medium-effort": {"reasoning_effort": "medium"}, "xhigh-effort": {"reasoning_effort": "xhigh"},
48+ "thinking": {"reasoning": "on"}, "reasoning": {"reasoning": "on"}, "think": {"reasoning": "on"}, "thinking-on": {"reasoning": "on"},
49+ "non-reasoning": {"reasoning": "off"}, "no-reasoning": {"reasoning": "off"}, "non-thinking": {"reasoning": "off"}, "nothink": {"reasoning": "off"},
50+ "no-think": {"reasoning": "off"}, "instant": {"reasoning": "off"}, "thinking-off": {"reasoning": "off"}, "fast": {"reasoning": "off"},
51+ "adaptive": {"reasoning": "adaptive"}, "adaptive-reasoning": {"reasoning": "adaptive"},
52+ "thinking-16k": {"reasoning": "on", "thinking_budget": "16k"}, "thinking-32k": {"reasoning": "on", "thinking_budget": "32k"},
53+ "thinking-64k": {"reasoning": "on", "thinking_budget": "64k"}, "thinking-128k": {"reasoning": "on", "thinking_budget": "128k"},
54+ "thinking-8k": {"reasoning": "on", "thinking_budget": "8k"}, "thinking-4k": {"reasoning": "on", "thinking_budget": "4k"}, "thinking-1k": {"reasoning": "on", "thinking_budget": "1k"},
55+}
56+# order matters: try the longest compound suffixes first
57+_EFFORT_ORDERED = sorted(EFFORT_SUFFIXES, key=len, reverse=True)
58+_EFFORT_RE = re.compile(r"[-_ ](" + "|".join(re.escape(s) for s in _EFFORT_ORDERED) + r")$", re.I)
59+# Words that are *part of a model name*, never an effort suffix, when they precede the suffix (e.g. "Kimi K2 Thinking" is a distinct release).
60+OFFICIAL_THINKING_RELEASES = {"kimi-k2-thinking", "qwen3-235b-a22b-thinking-2507", "qwen3-30b-a3b-thinking-2507", "qwen3-4b-thinking-2507", "glm-4.5-air-thinking",
61+ "gemini-2.5-flash-thinking", "grok-3-mini-thinking", "gemini-2-0-flash-thinking-exp-1219", "gemini-2-0-flash-thinking-exp-01-21"}
62+
63+# ---------------------------------------------------------------------------------------------- name analysis
64+_SIZE_RE = re.compile(r"(?<![a-z0-9])(\d+(?:\.\d+)?)\s?([bmt])(?![a-z])", re.I) # 70B 3.8B 235B 1.5T 350M
65+_ACTIVE_RE = re.compile(r"(?<![a-z0-9])(\d+(?:\.\d+)?)\s?[bmt]\s?-?a(\d+(?:\.\d+)?)\s?([bmt])(?![a-z])", re.I) # 35B-A3B
66+_DATE_RE = re.compile(r"(?<!\d)(20\d{2})[-_.]?(0[1-9]|1[0-2])[-_.]?(0[1-9]|[12]\d|3[01])(?!\d)") # 20240620
67+_MMDD_RE = re.compile(r"(?<![0-9])(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])(?![0-9])") # 0905 / 0324 (release snapshots)
68+_YYMM_RE = re.compile(r"(?<![0-9])(2[3-9])(0[1-9]|1[0-2])(?![0-9])") # 2507 (Qwen-style)
69+_TRAILING_VARIANT_WORDS = {"instruct", "it", "chat", "base", "sft", "dpo", "rl", "hf", "preview", "exp", "experimental", "latest", "beta", "alpha", "v1", "v2", "v3",
70+ "mtp", "distill", "distilled", "uncensored", "abliterated", "heretic", "merge"}
71+_TOKEN_SPLIT = re.compile(r"[\s_/]+|(?<=[a-z])(?=[A-Z][a-z])")
72+
73+
74+@dataclass
75+class NameAnalysis:
76+ raw: str
77+ repo_org: str | None = None # "unsloth" in "unsloth/Qwen3.6-35B-A3B-GGUF"
78+ repo_name: str | None = None # "Qwen3.6-35B-A3B-GGUF"
79+ base_key: str = "" # normalised key with quant/precision/effort tokens removed: "qwen3.6-35b-a3b"
80+ quant_formats: list[str] = field(default_factory=list) # ["gguf"]
81+ precision: str | None = None # "bf16" when only a precision tag is present
82+ is_quantized: bool = False
83+ is_conversion: bool = False # packaging/precision conversion without quantisation (BF16 repack, MLX fp16, ONNX)
84+ from_converter_org: bool = False
85+ effort: dict[str, str] = field(default_factory=dict) # {"reasoning_effort": "xhigh"} when an evaluator suffix was stripped
86+ effort_suffix: str | None = None
87+ parameter_count: int | None = None
88+ active_parameter_count: int | None = None
89+ snapshot_date: str | None = None # "2024-06-20" / "2025-09" style date embedded in the name
90+ family_hint: str | None = None # "Qwen3.6", "Llama 4", "Claude", "Gemini 2.5"
91+
92+ @property
93+ def is_artifact(self) -> bool:
94+ return self.is_quantized or self.is_conversion
95+
96+ @property
97+ def is_effort_variant(self) -> bool:
98+ return bool(self.effort)
99+
100+
101+def _to_count(num: str, unit: str) -> int:
102+ mult = {"m": 1e6, "b": 1e9, "t": 1e12}[unit.lower()]
103+ return int(round(float(num) * mult))
104+
105+
106+def analyze_model_name(raw: str) -> NameAnalysis:
107+ """Deterministic analysis of a model / repository name."""
108+ a = NameAnalysis(raw=raw.strip())
109+ name = a.raw
110+ if "/" in name and " " not in name.split("/")[0]:
111+ org, _, rest = name.partition("/")
112+ a.repo_org, a.repo_name = org.strip(), rest.strip()
113+ name = rest
114+ a.from_converter_org = org.strip().lower() in CONVERTER_ORGS
115+ low = re.sub(r"[\s_]+", "-", name.lower().strip())
116+ low = re.sub(r"[()\[\]]+", "-", low).strip("-")
117+ low = re.sub(r"-{2,}", "-", low)
118+
119+ # evaluator effort suffix (only when the remaining stem is not itself an official "Thinking" release)
120+ m = _EFFORT_RE.search(low)
121+ if m and low not in OFFICIAL_THINKING_RELEASES:
122+ suffix = m.group(1).lower()
123+ a.effort = dict(EFFORT_SUFFIXES[suffix])
124+ a.effort_suffix = suffix
125+ low = low[: m.start()]
126+
127+ # sizes
128+ am = _ACTIVE_RE.search(low)
129+ if am:
130+ a.parameter_count = _to_count(am.group(1), low[am.start(1) + len(am.group(1)):].strip()[0])
131+ a.active_parameter_count = _to_count(am.group(2), am.group(3))
132+ else:
133+ sizes = _SIZE_RE.findall(low)
134+ if sizes:
135+ counts = [_to_count(n, u) for n, u in sizes]
136+ a.parameter_count = max(counts)
137+
138+ # dates embedded in the name
139+ dm = _DATE_RE.search(low)
140+ if dm:
141+ a.snapshot_date = f"{dm.group(1)}-{dm.group(2)}-{dm.group(3)}"
142+ else:
143+ ym = _YYMM_RE.search(low)
144+ if ym and not _SIZE_RE.search(ym.group(0)):
145+ a.snapshot_date = f"20{ym.group(1)}-{ym.group(2)}"
146+
147+ # quantisation / precision tokens
148+ tokens = [t for t in re.split(r"[-_\s./()\[\]]+", low) if t]
149+ quant: list[str] = []
150+ precision = None
151+ kept: list[str] = []
152+ for t in tokens:
153+ if t in QUANT_FORMATS:
154+ quant.append(QUANT_FORMATS[t])
155+ continue
156+ if t in PRECISION_FORMATS:
157+ precision = t
158+ continue
159+ if re.fullmatch(r"(w\d+a\d+|q\d(_[a-z0-9]+)*|iq\d(_[a-z0-9]+)*|\d-?bit|int\d|fp\d|nvfp\d|mxfp\d)", t):
160+ quant.append(QUANT_FORMATS.get(t, "quantized"))
161+ continue
162+ kept.append(t)
163+ # "GGUF" / "MLX" packaging counts as artifact even without a bit-width; "MLX-8bit" is quantised
164+ a.quant_formats = sorted(set(quant))
165+ a.precision = precision
166+ a.is_quantized = any(q not in ("onnx", "coreml", "openvino", "tensorrt", "mlx", "gguf") for q in a.quant_formats) or "gguf" in a.quant_formats
167+ a.is_conversion = (not a.is_quantized) and (bool(a.quant_formats) or precision is not None or (a.from_converter_org and bool(a.repo_org)))
168+ if "mlx" in a.quant_formats and any(q.startswith("int") for q in a.quant_formats):
169+ a.is_quantized = True
170+
171+ a.base_key = "-".join(kept).strip("-")
172+ a.family_hint = family_hint(name)
173+ return a
174+
175+
176+# ---------------------------------------------------------------------------------------------- family inference
177+_FAMILY_PATTERNS: list[tuple[re.Pattern[str], str]] = [
178+ (re.compile(r"\bclaude\b", re.I), "Claude"),
179+ (re.compile(r"\bgpt[- ]?(oss)\b", re.I), "gpt-oss"),
180+ (re.compile(r"\b(chat)?gpt[- ]?\d", re.I), "GPT"),
181+ (re.compile(r"\bo[1-9](-| |$|mini|pro)", re.I), "OpenAI o-series"),
182+ (re.compile(r"\bgemini\b", re.I), "Gemini"),
183+ (re.compile(r"\bgemma(?=\d|\b)", re.I), "Gemma"),
184+ (re.compile(r"\bpalm\b", re.I), "PaLM"),
185+ (re.compile(r"\bllama(?=\d|\b)", re.I), "Llama"),
186+ (re.compile(r"\bmistral\b", re.I), "Mistral"),
187+ (re.compile(r"\bmixtral\b", re.I), "Mixtral"),
188+ (re.compile(r"\bministral\b", re.I), "Ministral"),
189+ (re.compile(r"\b(codestral|devstral|magistral|pixtral|voxtral)\b", re.I), None), # own families, name = family
190+ (re.compile(r"\bqwen(?=\d|\b)|\bqwq\b|\bqvq\b", re.I), "Qwen"),
191+ (re.compile(r"\bdeepseek\b", re.I), "DeepSeek"),
192+ (re.compile(r"\bkimi\b", re.I), "Kimi"),
193+ (re.compile(r"\bglm(?=\d|\b)|\bchatglm\b", re.I), "GLM"),
194+ (re.compile(r"\bgrok\b", re.I), "Grok"),
195+ (re.compile(r"\bcommand\b", re.I), "Command"),
196+ (re.compile(r"\baya\b", re.I), "Aya"),
197+ (re.compile(r"\bphi(?=\d|\b)", re.I), "Phi"),
198+ (re.compile(r"\bnemotron\b", re.I), "Nemotron"),
199+ (re.compile(r"\bgranite\b", re.I), "Granite"),
200+ (re.compile(r"\bolmo(?=\d|\b)", re.I), "OLMo"),
201+ (re.compile(r"\bmolmo\b", re.I), "Molmo"),
202+ (re.compile(r"\bfalcon\b", re.I), "Falcon"),
203+ (re.compile(r"\byi\b", re.I), "Yi"),
204+ (re.compile(r"\bminimax\b", re.I), "MiniMax"),
205+ (re.compile(r"\bhunyuan\b", re.I), "Hunyuan"),
206+ (re.compile(r"\bernie\b", re.I), "ERNIE"),
207+ (re.compile(r"\bseed\b", re.I), "Seed"),
208+ (re.compile(r"\bdoubao\b", re.I), "Doubao"),
209+ (re.compile(r"\bstep\b", re.I), "Step"),
210+ (re.compile(r"\binternlm(?=\d|\b)|\binternvl(?=\d|\b)", re.I), "InternLM"),
211+ (re.compile(r"\bjamba\b", re.I), "Jamba"),
212+ (re.compile(r"\blfm(?=\d|\b)", re.I), "LFM"),
213+ (re.compile(r"\bexaone(?=\d|\b)", re.I), "EXAONE"),
214+ (re.compile(r"\bsolar\b", re.I), "Solar"),
215+ (re.compile(r"\bnova\b", re.I), "Nova"),
216+ (re.compile(r"\btitan\b", re.I), "Titan"),
217+ (re.compile(r"\bsonar\b", re.I), "Sonar"),
218+ (re.compile(r"\bstable[- ]?diffusion\b|\bsdxl\b|\bsd3\b", re.I), "Stable Diffusion"),
219+ (re.compile(r"\bflux\b", re.I), "FLUX"),
220+ (re.compile(r"\bwhisper\b", re.I), "Whisper"),
221+ (re.compile(r"\bdall[- ]?e\b", re.I), "DALL·E"),
222+ (re.compile(r"\bsora\b", re.I), "Sora"),
223+ (re.compile(r"\bveo\b", re.I), "Veo"),
224+ (re.compile(r"\bimagen\b", re.I), "Imagen"),
225+ (re.compile(r"\bcogito\b", re.I), "Cogito"),
226+ (re.compile(r"\bhermes\b", re.I), "Hermes"),
227+ (re.compile(r"\bsmol(lm|vlm)\b", re.I), "SmolLM"),
228+ (re.compile(r"\bbert\b", re.I), "BERT"),
229+ (re.compile(r"\bt5\b", re.I), "T5"),
230+ (re.compile(r"\bclip\b", re.I), "CLIP"),
231+ (re.compile(r"\bembed(ding)?\b", re.I), None),
232+ (re.compile(r"\brerank\b", re.I), None),
233+]
234+_FAMILY_VERSION_RE = re.compile(r"^(?P<fam>[A-Za-z][A-Za-z·\-]*?)[\s\-]?(?P<ver>\d+(?:\.\d+)?)", re.I)
235+
236+
237+def family_hint(name: str) -> str | None:
238+ """Family label without a version ("Qwen", "Llama", "Claude"). Versioned families ("Llama 3.1") are `family_release_hint`."""
239+ n = name.split("/")[-1]
240+ for pat, fam in _FAMILY_PATTERNS:
241+ if pat.search(n):
242+ if fam is None:
243+ m = pat.search(n)
244+ return m.group(0).title() if m else None
245+ return fam
246+ return None
247+
248+
249+def family_release_hint(name: str) -> str | None:
250+ """Versioned family ("Llama 3.1", "Qwen3.6", "Gemini 2.5", "Claude 4") when the name carries a version right after the family word."""
251+ fam = family_hint(name)
252+ if not fam:
253+ return None
254+ n = name.split("/")[-1]
255+ m = re.search(re.escape(fam.split(" ")[0]) + r"[\s\-]?(\d+(?:\.\d+)?)", n, re.I)
256+ if m:
257+ joined = fam in ("Qwen", "GLM", "Phi", "Yi", "Step") or re.search(re.escape(fam) + r"\d", n, re.I)
258+ return f"{fam}{m.group(1)}" if joined else f"{fam} {m.group(1)}"
259+ return fam
260+
261+
262+def variant_key(name: str) -> str:
263+ """Grouping key for near-duplicates: analysed base key + parameter count, ignoring org prefixes, quant/precision/effort tokens,
264+ separators and trailing 'instruct/chat/it' words. `Qwen3.6-35B-A3B`, `unsloth/Qwen3.6-35B-A3B-GGUF`, `Qwen3.6 35B A3B FP8` → same key."""
265+ a = analyze_model_name(name)
266+ toks = [t for t in a.base_key.split("-") if t and t not in _TRAILING_VARIANT_WORDS]
267+ return "-".join(toks)
268+
269+
270+__all__ = ["ARTIFACT_PACKAGING", "CONVERTER_ORGS", "EFFORT_SUFFIXES", "NameAnalysis", "PRECISION_FORMATS", "QUANT_FORMATS", "analyze_model_name",
271+ "family_hint", "family_release_hint", "variant_key"]
added src/aiatlas/ontology/openness.py +115 −0
@@ -0,0 +1,115 @@
1+"""Openness 2.0 — measurable dimensions first, a carefully labelled category second.
2+
3+Dimensions (each `True | False | None` = unknown):
4+ weights_available the weights can be downloaded (a checkpoint exists on a public hub / official download)
5+ source_code_available inference / model code is published
6+ training_code_available the training pipeline is published
7+ training_data_disclosed the composition of the training data is documented
8+ dataset_available the training data itself (or most of it) is downloadable
9+ commercial_use_allowed licence permits commercial use
10+ redistribution_allowed licence permits redistributing the weights
11+ derivatives_allowed licence permits fine-tunes / modifications
12+
13+Categories (`OPENNESS_CATEGORIES`) are derived, never asserted directly by a connector unless the source literally states it:
14+ open-source weights + code under an OSI-approved (or equivalently permissive) licence, commercial use and derivatives allowed
15+ open-weights weights downloadable under a permissive or Creative Commons licence allowing commercial use (code may be missing)
16+ restricted-weights weights downloadable but the licence restricts commercial use, hosting, derivatives or field of use
17+ proprietary weights not available (API / product only)
18+ unknown not enough evidence
19+
20+The historical single-value vocabulary (`open-weights | open-source | proprietary | restricted`) stays valid as input to `normalize_openness`.
21+"""
22+from __future__ import annotations
23+
24+from typing import Any
25+
26+from aiatlas.ontology.licenses import LICENSES, normalize_license
27+
28+OPENNESS_CATEGORIES = ("open-source", "open-weights", "restricted-weights", "proprietary", "unknown")
29+
30+OPENNESS_DIMENSIONS = (
31+ "weights_available", "source_code_available", "training_code_available", "training_data_disclosed", "dataset_available",
32+ "commercial_use_allowed", "redistribution_allowed", "derivatives_allowed",
33+)
34+
35+_LEGACY = {
36+ "open-weights": "open-weights", "open_weights": "open-weights", "open weights": "open-weights", "openweights": "open-weights", "open": "open-weights",
37+ "open-source": "open-source", "open_source": "open-source", "open source": "open-source", "opensource": "open-source", "oss": "open-source",
38+ "proprietary": "proprietary", "closed": "proprietary", "closed-source": "proprietary", "closed_source": "proprietary", "closed source": "proprietary",
39+ "api-only": "proprietary", "api only": "proprietary", "commercial": "proprietary",
40+ "restricted": "restricted-weights", "restricted-weights": "restricted-weights", "restricted_weights": "restricted-weights", "gated": "restricted-weights",
41+ "research-only": "restricted-weights", "non-commercial": "restricted-weights",
42+ "unknown": "unknown", "": "unknown",
43+}
44+
45+
46+def normalize_openness(raw: Any) -> str | None:
47+ """Legacy single label → canonical category. Unknown strings → None (the writer keeps the raw label)."""
48+ if raw is None:
49+ return None
50+ s = str(raw).strip().lower()
51+ return _LEGACY.get(s)
52+
53+
54+def openness_dimensions(*, weights_available: bool | None = None, license_key: str | None = None, license_raw: str | None = None,
55+ source_code_available: bool | None = None, training_code_available: bool | None = None,
56+ training_data_disclosed: bool | None = None, dataset_available: bool | None = None) -> dict[str, bool | None]:
57+ """Fill the licence-derived dimensions from the licence ontology; everything else is passed through (unknown stays None)."""
58+ key = license_key or normalize_license(license_raw)
59+ info = LICENSES.get(key) if key else None
60+ dims: dict[str, bool | None] = {
61+ "weights_available": weights_available,
62+ "source_code_available": source_code_available,
63+ "training_code_available": training_code_available,
64+ "training_data_disclosed": training_data_disclosed,
65+ "dataset_available": dataset_available,
66+ "commercial_use_allowed": info.commercial_use if info else None,
67+ "redistribution_allowed": info.redistribution if info else None,
68+ "derivatives_allowed": info.derivatives if info else None,
69+ }
70+ if info and not info.weights_downloadable and weights_available is None:
71+ dims["weights_available"] = False
72+ return dims
73+
74+
75+def derive_openness(dims: dict[str, bool | None], *, license_key: str | None = None) -> str:
76+ """Category from dimensions (+ licence category). Conservative: a custom community licence is never 'open-source'."""
77+ info = LICENSES.get(license_key) if license_key else None
78+ weights = dims.get("weights_available")
79+ if weights is False:
80+ return "proprietary"
81+ if weights is None:
82+ return "unknown"
83+ commercial = dims.get("commercial_use_allowed")
84+ derivatives = dims.get("derivatives_allowed")
85+ redistribution = dims.get("redistribution_allowed")
86+ restricted_by_license = bool(info and (info.category in ("community", "research-only", "responsible-ai") or info.hosting_restrictions))
87+ if commercial is False or derivatives is False or redistribution is False or restricted_by_license:
88+ return "restricted-weights"
89+ if info and info.osi_approved and dims.get("source_code_available") is True and commercial is True and derivatives is True:
90+ return "open-source"
91+ if info and info.category in ("permissive", "creative-commons") and commercial is True and redistribution is True:
92+ return "open-weights"
93+ if info is None:
94+ # weights downloadable, licence unknown → we know the weights exist, not the terms
95+ return "open-weights" if commercial is None else "restricted-weights"
96+ return "open-weights"
97+
98+
99+OPENNESS_LABELS = {
100+ "open-source": "Open source",
101+ "open-weights": "Open weights",
102+ "restricted-weights": "Restricted weights",
103+ "proprietary": "Closed / proprietary",
104+ "unknown": "Unknown",
105+}
106+
107+OPENNESS_DEFINITIONS = {
108+ "open-source": "Weights and code published under an OSI-approved licence that allows commercial use and derivatives.",
109+ "open-weights": "Weights downloadable under a permissive or Creative Commons licence allowing commercial use; code or data may be missing.",
110+ "restricted-weights": "Weights downloadable, but the licence restricts commercial use, hosting, derivatives or field of use (community, research and RAIL licences).",
111+ "proprietary": "Weights are not available; the model is reachable only through an API or a product.",
112+ "unknown": "Not enough sourced evidence to classify.",
113+}
114+
115+__all__ = ["OPENNESS_CATEGORIES", "OPENNESS_DEFINITIONS", "OPENNESS_DIMENSIONS", "OPENNESS_LABELS", "derive_openness", "normalize_openness", "openness_dimensions"]
added src/aiatlas/ontology/taxonomy.py +166 −0
@@ -0,0 +1,166 @@
1+"""Small canonical enums and their normalisers. Raw source strings are never mutated in snapshots — only the materialised attribute
2+is canonical; connectors and the writer call these before writing a claim."""
3+from __future__ import annotations
4+
5+import re
6+from typing import Any
7+
8+# ---------------------------------------------------------------------------------------------- modalities
9+MODALITIES = ("text", "image", "audio", "video", "document", "code", "embedding", "3d", "structured", "action")
10+_MODALITY_ALIASES = {
11+ "text": "text", "texts": "text", "language": "text", "nlp": "text", "chat": "text", "textual": "text",
12+ "image": "image", "images": "image", "vision": "image", "visual": "image", "picture": "image", "photo": "image", "img": "image",
13+ "audio": "audio", "speech": "audio", "voice": "audio", "sound": "audio", "music": "audio",
14+ "video": "video", "videos": "video",
15+ "pdf": "document", "document": "document", "documents": "document", "file": "document", "files": "document", "doc": "document",
16+ "code": "code", "coding": "code", "programming": "code",
17+ "embedding": "embedding", "embeddings": "embedding", "vector": "embedding",
18+ "3d": "3d", "mesh": "3d", "point cloud": "3d",
19+ "structured": "structured", "json": "structured", "table": "structured", "tabular": "structured", "time-series": "structured", "time series": "structured",
20+ "action": "action", "actions": "action", "computer-use": "action", "computer use": "action", "tool": "action", "robotics": "action",
21+ "multimodal": None, "multi-modal": None, "omni": None, "any": None, # not a modality — a property of the set
22+}
23+
24+
25+def normalize_modality(raw: Any) -> str | None:
26+ if raw is None:
27+ return None
28+ s = str(raw).strip().lower().replace("_", "-")
29+ if s in _MODALITY_ALIASES:
30+ return _MODALITY_ALIASES[s]
31+ s2 = s.replace("-", " ")
32+ return _MODALITY_ALIASES.get(s2)
33+
34+
35+def normalize_modalities(raw: Any) -> list[str]:
36+ """Any iterable / comma string → sorted canonical unique list (unknown values dropped)."""
37+ if raw is None:
38+ return []
39+ items = raw if isinstance(raw, (list, tuple, set)) else re.split(r"[,/;+&]|\band\b|→|->", str(raw))
40+ out: set[str] = set()
41+ for it in items:
42+ m = normalize_modality(it)
43+ if m:
44+ out.add(m)
45+ return sorted(out)
46+
47+
48+# ---------------------------------------------------------------------------------------------- model / entity status
49+MODEL_STATUSES = ("announced", "preview", "active", "limited-availability", "deprecated", "retired", "archived", "unknown")
50+_STATUS_ALIASES = {
51+ "active": "active", "available": "active", "ga": "active", "general availability": "active", "generally available": "active", "live": "active",
52+ "released": "active", "stable": "active", "current": "active", "supported": "active", "production": "active", "online": "active",
53+ "preview": "preview", "beta": "preview", "alpha": "preview", "experimental": "preview", "public preview": "preview", "early access": "preview",
54+ "research preview": "preview", "exp": "preview",
55+ "limited-availability": "limited-availability", "limited availability": "limited-availability", "limited": "limited-availability",
56+ "waitlist": "limited-availability", "private preview": "limited-availability", "invite-only": "limited-availability", "gated": "limited-availability",
57+ "deprecated": "deprecated", "legacy": "deprecated", "sunset": "deprecated", "sunsetting": "deprecated", "end-of-life announced": "deprecated",
58+ "deprecation": "deprecated",
59+ "retired": "retired", "discontinued": "retired", "shut down": "retired", "shutdown": "retired", "removed": "retired", "end of life": "retired",
60+ "eol": "retired", "decommissioned": "retired", "offline": "retired", "delisted": "retired",
61+ "announced": "announced", "upcoming": "announced", "coming soon": "announced", "planned": "announced", "unreleased": "announced",
62+ "archived": "archived",
63+ "unknown": "unknown", "": "unknown",
64+}
65+
66+
67+def normalize_status(raw: Any) -> str | None:
68+ if raw is None:
69+ return None
70+ s = str(raw).strip().lower().replace("_", "-")
71+ if s in _STATUS_ALIASES:
72+ return _STATUS_ALIASES[s]
73+ return _STATUS_ALIASES.get(s.replace("-", " "))
74+
75+
76+# ---------------------------------------------------------------------------------------------- organization kinds
77+ORG_KINDS = ("company", "lab", "university", "nonprofit", "government", "community", "consortium", "individual")
78+_ORG_KIND_ALIASES = {
79+ "company": "company", "corporation": "company", "startup": "company", "enterprise": "company", "vendor": "company", "business": "company",
80+ "lab": "lab", "laboratory": "lab", "research lab": "lab", "research-lab": "lab", "research institute": "lab", "institute": "lab", "research": "lab",
81+ "university": "university", "academic": "university", "school": "university", "college": "university",
82+ "nonprofit": "nonprofit", "non-profit": "nonprofit", "non profit": "nonprofit", "foundation": "nonprofit", "ngo": "nonprofit",
83+ "government": "government", "public": "government", "agency": "government", "state": "government",
84+ "community": "community", "open-source community": "community", "collective": "community", "hf-community": "community", "group": "community",
85+ "consortium": "consortium", "alliance": "consortium", "coalition": "consortium",
86+ "individual": "individual", "person": "individual", "independent": "individual", "user": "individual",
87+}
88+# entity_type → default org kind (the entity_type stays; org_kind is the canonical sub-kind)
89+ORG_TYPE_DEFAULT_KIND = {"company": "company", "lab": "lab", "university": "university", "organization": None}
90+
91+
92+def normalize_org_kind(raw: Any) -> str | None:
93+ if raw is None:
94+ return None
95+ s = str(raw).strip().lower().replace("_", "-")
96+ return _ORG_KIND_ALIASES.get(s) or _ORG_KIND_ALIASES.get(s.replace("-", " "))
97+
98+
99+# ---------------------------------------------------------------------------------------------- hardware kinds
100+HARDWARE_KINDS = ("gpu", "accelerator", "npu", "cpu", "soc", "system", "server", "workstation", "cloud-instance", "rack")
101+_HW_ALIASES = {
102+ "gpu": "gpu", "graphics card": "gpu", "graphics": "gpu", "dgpu": "gpu",
103+ "accelerator": "accelerator", "tpu": "accelerator", "asic": "accelerator", "ai accelerator": "accelerator", "lpu": "accelerator", "wse": "accelerator",
104+ "npu": "npu", "neural engine": "npu",
105+ "cpu": "cpu", "processor": "cpu",
106+ "soc": "soc", "system on chip": "soc", "system-on-chip": "soc", "apple silicon": "soc", "chip": "soc",
107+ "system": "system", "computer": "system", "desktop": "system", "laptop": "system", "mini": "system", "mac": "system",
108+ "server": "server", "node": "server", "dgx": "server", "appliance": "server",
109+ "workstation": "workstation",
110+ "cloud-instance": "cloud-instance", "cloud instance": "cloud-instance", "instance": "cloud-instance", "vm": "cloud-instance",
111+ "rack": "rack", "superpod": "rack", "pod": "rack", "cluster": "rack", "nvl72": "rack",
112+}
113+
114+
115+def normalize_hardware_kind(raw: Any) -> str | None:
116+ if raw is None:
117+ return None
118+ s = str(raw).strip().lower().replace("_", "-")
119+ return _HW_ALIASES.get(s) or _HW_ALIASES.get(s.replace("-", " "))
120+
121+
122+# ---------------------------------------------------------------------------------------------- framework / library / tool kinds
123+FRAMEWORK_KINDS = ("training-framework", "inference-engine", "serving-engine", "library", "runtime", "agent-framework", "orchestration",
124+ "evaluation-harness", "sdk", "tool", "application", "agent", "mcp-server", "vector-database", "observability", "data-tooling")
125+_FW_ALIASES = {
126+ "framework": "training-framework", "training framework": "training-framework", "training-framework": "training-framework", "deep learning framework": "training-framework",
127+ "inference engine": "inference-engine", "inference-engine": "inference-engine", "inference": "inference-engine", "engine": "inference-engine",
128+ "serving": "serving-engine", "serving engine": "serving-engine", "serving-engine": "serving-engine", "server": "serving-engine",
129+ "library": "library", "lib": "library", "package": "library",
130+ "runtime": "runtime",
131+ "agent framework": "agent-framework", "agent-framework": "agent-framework", "agents": "agent-framework",
132+ "orchestration": "orchestration", "workflow": "orchestration", "pipeline": "orchestration",
133+ "evaluation": "evaluation-harness", "eval": "evaluation-harness", "evaluation harness": "evaluation-harness", "evaluation-harness": "evaluation-harness", "benchmarking": "evaluation-harness",
134+ "sdk": "sdk", "client": "sdk", "api client": "sdk",
135+ "tool": "tool", "cli": "tool", "utility": "tool",
136+ "application": "application", "app": "application", "product": "application", "ui": "application", "desktop app": "application",
137+ "agent": "agent", "coding agent": "agent", "assistant": "agent",
138+ "mcp": "mcp-server", "mcp server": "mcp-server", "mcp-server": "mcp-server", "mcp_server": "mcp-server",
139+ "vector database": "vector-database", "vector-database": "vector-database", "vector db": "vector-database", "vector store": "vector-database",
140+ "observability": "observability", "tracing": "observability", "monitoring": "observability", "experiment tracking": "observability",
141+ "data": "data-tooling", "data tooling": "data-tooling", "data-tooling": "data-tooling", "datasets": "data-tooling", "tokenizer": "data-tooling",
142+}
143+
144+
145+def normalize_framework_kind(raw: Any) -> str | None:
146+ if raw is None:
147+ return None
148+ s = str(raw).strip().lower()
149+ return _FW_ALIASES.get(s) or _FW_ALIASES.get(s.replace("_", "-")) or _FW_ALIASES.get(s.replace("-", " "))
150+
151+
152+# ---------------------------------------------------------------------------------------------- generic helper
153+def canonical_enum(value: Any, normaliser) -> tuple[Any, str | None]:
154+ """Return (canonical_or_original, raw_if_changed). Lets the writer store `x` canonical and `x_raw` when the source label differed."""
155+ if value is None:
156+ return None, None
157+ canon = normaliser(value)
158+ if canon is None:
159+ return value, None
160+ return canon, (str(value) if str(value) != canon else None)
161+
162+
163+__all__ = [
164+ "FRAMEWORK_KINDS", "HARDWARE_KINDS", "MODALITIES", "MODEL_STATUSES", "ORG_KINDS", "ORG_TYPE_DEFAULT_KIND", "canonical_enum",
165+ "normalize_framework_kind", "normalize_hardware_kind", "normalize_modalities", "normalize_modality", "normalize_org_kind", "normalize_status",
166+]
modified src/aiatlas/sdk/facts.py +10 −1
Large diff (36 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere