HTML 77.2%
TypeScript 10.5%
Python 9.6%
JavaScript 2.5%
1# Canonicalization — `aia canonicalize`23The canonical upgrade (2026-09) turns the flat "everything is a model" corpus into one hierarchy and one vocabulary:45```6model_family Llama 4 · Qwen3.6 · Claude · GPT 5.47 └ model Llama 4 Maverick · Qwen3.6-35B-A3B · Claude Fable 5.1 ← the unit counted as "a model"8 └ artifact unsloth/Qwen3.6-35B-A3B-GGUF (quantization) · zai-org/GLM-5-FP8 (conversion) · mlx-community/… (packaging)9 └ deployment OpenRouter qwen/qwen3.6-35b-a3b · Together … ← `prices` rows, never entities10```1112Evaluation-effort variants (`gpt-5-4-mini-medium`, `claude-opus-5-xhigh`, `deepseek-v4-pro-non-reasoning`) are **result configurations**13(`reasoning_effort`, `reasoning`, `thinking_budget`) of the canonical model, never entities of their own.1415Two layers implement it:1617* **at write time** — `FactWriter` / `Resolver` (src/aiatlas/sdk) apply the ontology to every incoming fact, so new data is born canonical;18* **on the existing corpus** — `aia canonicalize` (src/aiatlas/services/canonical.py) replays the same rules over what is already stored.1920## Running it2122```bash23.venv/bin/aia canonicalize # dry-run: plan + counts + examples, writes nothing24.venv/bin/aia canonicalize --apply # execute the plan (one transaction per step)25.venv/bin/aia canonicalize --step results --step events --apply26.venv/bin/aia canonicalize --json # machine-readable report27.venv/bin/aia anomalies [--severity critical] [--refresh]28.venv/bin/aia quarantine list | release <id> | discard <id>29```3031Steps always run in this order (later steps depend on earlier ones):32`duplicates → variants → artifacts → families → licenses → taxonomy → results → events → anomalies`.3334**Idempotence is a hard requirement**: a second `--apply` immediately after the first must report `0 change(s)`. Every step computes its plan35from reads, and `apply` executes exactly that plan; nothing depends on "what changed last time". Re-run it after every connector wave.3637## What is never done3839* **No row is ever deleted.** Merged duplicates and folded variants stay as rows with `status='merged'`, `merged_into=<survivor>`; their40 slugs and ids keep resolving (API follows `merged_into`, web 301s). Artifacts keep their own row, slug, aliases, identifiers, claims,41 results, prices and events.42* **Raw snapshots and the archive are never touched.** Canonical values live in `claims.value` / `entities.attributes`; the source label is43 kept next to them (`claims.value_raw`, `attributes.<prop>_raw`) whenever it differed.44* **No value is guessed.** Unknown licence / modality / status strings are kept as-is and recorded in `taxonomy_mappings` with45 `canonical = NULL` (the backlog is visible; nothing is dropped). Ambiguous identities go to `review_queue`, never to a merge.46* **Derived facts never outrank sourced facts.** Everything the engine infers is written by a tier-2 `derived` writer (source47 `ai-atlas.registry`): the ordinary writer rules apply, so a tier-1 explicit statement (`openness` from a lab page) is kept and the derived48 disagreement is stored once as `conflicting` (without a review item — derived disagreements are expected and inspectable).49* **Observation truth is preserved.** `first_seen_at` is when *we* first saw an entity; a release date earlier than that is a fact about the50 model (`attributes.release_date`), not about our observation. `/diff`, `/changes` and the counters stay observation-based; timelines use51 `occurred_at = coalesce(effective_at, observed_at)`.5253## Steps5455### duplicates56Exact normalised-name duplicates (`ids.normalize_alias`) within a type — models, providers, benchmarks, datasets, frameworks, libraries,57hardware — and across the organisation group (`company | organization | lab | university`, also joined by a shared `hf_org` / `github_org`).58* identifiers **do not conflict** (no scheme with two different values) → merge the weaker row into the survivor. Survivor = curated59 org type (`company/lab/university` beat the hub's generic `organization`), then most current claims, then oldest `first_seen_at`.60 Recorded as a `merge` decision in `resolution_decisions` and in `admin_audit_log`.61* identifiers **conflict** (`gemini-2-5-flash-lite` AA vs `gemini-2.5-flash-lite` Google id + a *different* AA slug) → `review_queue`62 `merge_candidate` only.63* same name but two organisations (non-org types) → review only. Pairs with a `keep_separate` decision are skipped.64* single-letter organisations (`w`) → `junk_entity` review item; `org_kind` defaults from the entity type (`ORG_TYPE_DEFAULT_KIND`).6566### variants67Models whose name carries an evaluator effort suffix (`ontology.models.analyze_model_name().is_effort_variant`: `-xhigh/-high/-medium/-low/68-minimal`, `-thinking/-reasoning`, `-non-reasoning/-no-think`, `-32k-thinking`, `-high-effort`, up to three chained) **and** that are known69only through evaluator identifiers (`artificial_analysis`, `livebench_model_id`…), have no `hf_repo` and no tier-1 claim. Anything with an70official/hub/provider identifier is a real model that happens to end in "Thinking" (Qwen3-VL-8B-Thinking has its own weights) and is left alone.71`-max`, `-fast`, `-instant` are model tiers (Qwen3-Max, GPT-5.1-Codex-Max, Grok 4.1 Fast), never effort suffixes.7273Base resolution (`Resolver.resolve_variant`): same `artificial_analysis` identifier / alias / slug as the base name, or the same74`variant_key` through the precomputed index; never onto another effort variant; several bases → prefer the variant's organisation, else review.75* resolved → every `benchmark_results.config` of the variant gains the effort dict + `aa_variant_slug`, then `merge_entities(variant →76 canonical)` moves results (dedupe keys recomputed, collisions close the older row), prices, relations, events, aliases and identifiers. The77 variant's AA identifier now points at the canonical model, so the next AA run resolves it directly; the writer adds the effort dict to78 the config of any result whose model ref is an effort variant (`effort_config`), so folded results stay distinguishable and comparable79 (`reasoning_effort` is a *condition* key: same `config_key`, partially comparable).80* unresolved → `identity_confidence='medium'`, `attributes.evaluation_variant_of_hint`, `variant_candidate` review item.8182### artifacts83A model row is an **artifact** when its name / `hf_repo` carries a quantisation or precision token (`GGUF`, `FP8`, `MXFP4`, `AWQ`, `BF16`,84`MLX`, `ONNX`…), when its repository lives under a converter organisation (`ontology.models.CONVERTER_ORGS`: unsloth, bartowski,85mlx-community…), or when its hub repo attributes say `is_quantized` / `quant_format` (repo attributes only count with an `hf_repo`: a86provider's serving precision — OpenRouter `fp8` — says nothing about identity).8788* **Official checkpoints stay models.** A repo under the developer's own organisation (`FAMILY_ORGS` publisher or the entity's own org,89 never a converter org) is the model itself even in native FP8 (`deepseek-ai/DeepSeek-R1`, `nvidia/…-Nemotron-3-Ultra-…-BF16`). It only90 becomes an artifact when the same organisation *also* has the plain model entity **and** nobody else knows the tagged repo under its own91 identity; when evaluators/providers do, the pair is a `merge_candidate` for review. No checkpoint entities are created.92* `artifact_kind`: `quantization` (bit-width / GGUF / AWQ…), `conversion` (BF16/FP8 repack, MLX, ONNX), else `packaging`.93* `canonical_id`: the `quantized_from` object (never `derived_from` / `fine_tuned_from` — a fine-tune is a new model), else the unique model with94 the same `variant_key` (same organisation preferred, then the family's official publisher); none → artifact with `canonical_id = NULL`,95 `identity_confidence='low'`, `unresolved_artifact` review item.96* Writes: `entity_type='artifact'`, `artifact_kind`, `canonical_id`, relation `artifact_of`, a `variant_of` decision. Slug, aliases,97 identifiers, claims, results, prices, events untouched. `model` and `artifact` are lookup-compatible types in the resolver, so the same98 hub repo keeps resolving to the same row.99100### families101For every model with a family hint: family = versioned label when a version follows the family word (`Llama 3.1`, `Qwen3.6`, `Gemini 2.5`,102`Claude 4`, `GPT 5.4` — hyphenated slug versions `gpt-5-4` count; `38B`/`235B` are sizes, not versions), else the base family (`Claude`,103`Kimi`). One level only; the family entity carries `attributes.family_root` (`Llama`) for grouping.104Slug = `slugify(label)`; on collision with any other entity (the model `gpt-5.5` itself) the organisation slug is prefixed, then `family-`.105Organisation = the family's official publisher when known, else the model's. Writes: `entities.family_id`, relation `member_of_family`,106`attributes.family` only when the model had none (a sourced `family` claim is never overwritten).107108### licenses109`normalize_license(attributes.license)` → canonical key (SPDX id when one exists: `Apache-2.0`, `MIT`, `CC-BY-NC-4.0`; AI Atlas key110otherwise: `Llama-3.1-Community`, `Gemma-Terms`). Creates one `license` entity per observed key (slug = key lower-cased, attributes =111`LicenseInfo.as_dict()`: commercial use, redistribution, derivatives, hosting restrictions, acceptable-use policy, OSI approval…), relation112`uses_license`, `attributes.license_key`. Unclassified strings stay in `license` (mapping recorded with `canonical = NULL`).113114Openness 2.0 as **derived claims** (tier 2, `extractor='derived'`): `weights_available` (true with an `hf_repo` / model card / weights URL115or an open-ish openness label; false when proprietary; unknown → nothing written), `commercial_use_allowed`, `redistribution_allowed`,116`derivatives_allowed` from the licence, and the category `openness` = `derive_openness(dims)` (`open-source | open-weights | restricted-weights117| proprietary`). HF "gated" is a distribution mechanism, not a licence property: a gated Apache-2.0 repo is `open-weights`. A tier-1118`openness` statement that disagrees is kept current; the derived value is stored once as `conflicting`.119120### taxonomy121Current claims (and claim-less attributes) for `license`, `openness`, `modalities`, `modalities_input`, `modalities_output`, `status`, hardware122and framework `kind`, `org_kind` are **re-encoded in place**: same claim row, same source, same tier — only the encoding changes123(`apache-2.0` → `Apache-2.0`, `restricted` → `restricted-weights`, `["Text","pdf"]` → `["document","text"]`, `computer` → `system`). The124source label goes to `claims.value_raw` / `attributes.<prop>_raw`. This is *not* a new assertion, so no change event is emitted. Every125raw → canonical pair (and every unknown raw value) lands in `taxonomy_mappings`.126127The writer applies the same normalisation to incoming facts; when a source re-states the same licence in another spelling the current claim is128re-encoded and confirmed — never a `LICENSE_CHANGED` event.129130### results131Back-fills `config_key` (comparability hash of the *task* keys + metric — variant, board, harness, evaluator, shots, scaffold, system…),132`trust_level` (`ontology.benchmarks.trust_level(source_key, config)`: official-benchmark / independent-evaluator / official-model-card /133community / peer-reviewed / unverified), `variant`, `run_group` (release, index version, dataset revision, run date) and `extractor`.134Then **one current row per (model, benchmark, metric, config_key)**: rows from an older run group than the latest observed one get135`is_current = false`, `valid_to = <newer observation>`, so leaderboards show the latest LiveBench release / aider run / AA index version136instead of piling up every run. Run keys are *not* part of `config_key` (two releases of the same task are partially comparable); condition keys137(`reasoning_effort`, temperature, judge…) are not either, so folded effort variants share a key and are compared as conditions.138Scores outside the metric bounds are stored with `confidence='low'` and flagged (`score_above_max` / `score_below_min`).139140### events141Three clocks: `occurred_at = coalesce(effective_at, observed_at)` (when it happened), `observed_at` (when a connector saw it), `recorded_at`142(when the row was written). `is_backfill = true` when the event was observed before the connector's second successful run (the initial143corpus is history, not news), when `effective_at` is more than 3 days before `observed_at`, or for `NEW_*` events whose entity's144`release_date` / `published_at` predates observation by more than 3 days. `group_key = release:<entity>:<yyyy-mm>` for `RELEASE`,145`ANNOUNCEMENT`, `NEW_MODEL`, `VERSION_RELEASED` so one release seen in several documents groups. Live counters ("+N · 24 h", `/changes`,146pulse) count non-backfill events only; timelines use everything. The step prints live-event counts for the last 24 h before/after.147Importance is deterministic (`services.events.importance_for`): frontier or open-weight releases 3, price moves ≥ 50 % 3 / ≥ 20 % 2 / else 1,148context ≥ 5× 3, deprecations 3, leader changes 2, metadata corrections 0, artifact events 0, family events 1; tier > 2 sources lose one point.149150### anomalies151Runs `ontology.anomalies` over live models/artifacts, hardware, live prices and current results; upserts `anomalies` rows by `dedupe_key`152(reopened when a check fires again; operator states `ignored` / `fixed` are kept), and auto-resolves open flags whose check no longer fires.153`aia anomalies` lists them. Flags never change the data.154155## Write-time counterparts (SDK)156157* `FactWriter(run_id=…)`: every claim, relation, price, result and event carries the connector run id (batch inspection/rollback).158* Same-source loophole closed: a claim supersedes only when `tier <= current.tier` **or** same source **and** same extractor — an LLM claim159 never overwrites a deterministic one from the same URL (it is stored as `conflicting`, once).160* `Resolver`: `keep_separate` decisions block alias resolution/merges; aliases whose normalisation collapses digit separators161 (`Qwen3-8B` ≡ `Qwen 38B` → `qwen38b`) additionally require the same `variant_key`; `first_seen_hint` back-dates `first_seen_at`162 (`min(now, hint)`); hierarchy hints (`EntityRef.family` / `.canonical` / `.artifact_kind` / `.identity_confidence`) are materialised.163* Quarantine: tier ≥ 2 connectors (hubs, leaderboards, registries) hold their facts until the run ends and compare the run with the164 connector's rolling baseline (median of the last 5 full-extraction runs): new entities > max(50, 1.3× baseline), price rows > 3×, or — on a165 full re-extraction — entity references < 0.8× or results < 0.5× → the whole run goes to `quarantined_runs` (`status='pending'`, serialised166 Facts), `connector_runs.status='quarantined'`, a `quarantine` review item; nothing is written. `aia quarantine release <id>` writes the167 facts exactly as the connector would have; `discard` drops them (snapshots stay archived). A connector's first run is never quarantined —168 it sets the baseline.169* SSRF guard (`sdk/fetch.py`): every URL and every redirect hop (max 5) is resolved and refused when it targets a non-http(s) scheme,170 `localhost`/`.local`/`.internal`, RFC1918, loopback, link-local (cloud metadata), CGNAT, IPv6 loopback/link-local/ULA or unspecified addresses.171172## Counting rules (for the API / UI)173174* **Models** = `entity_type = 'model' and merged_into is null` (artifacts and folded variants excluded; `include=artifacts` restores the old175 universe). **Artifacts** = `entity_type = 'artifact'`. **Families** = `entity_type = 'model_family'`.176* **Current benchmark results** = `valid_to is null and is_current`. **Live events** = `is_backfill = false`.177