spb/ai-atlas

Hugging Face: repositories are models or artifacts (quantization / conversion / packaging) with `canonical` → base model (base_model tag or
analysed name, family org), `family` hints, canonical licence (+license_raw), `access: gated|open` instead of openness=restricted,
weights_available + derived openness, modalities from pipeline tags, parameter_count kept on artifacts only; converters `publish`, never `develop`.
Leaderboards: Artificial Analysis / LiveBench / aider / SWE-bench fold effort variants ("(xhigh)", "-thinking-64k-high-effort", non-reasoning)
into the base model with the setting in config; LiveBench categories become `livebench-<category>` benchmarks; aider well-formed rate →
`aider-polyglot-well-formed`; AA GPQA → gpqa-diamond, τ² → Telecom; trust_level / variant / run_group on every result; bookkeeping out of config;
free-text tags are aliases, identifiers only for trusted pinned vendor ids. New shared `connectors/_identity.py` (provider prefixes, first-party
patterns, effort stripping with size-tier guard, family hints).
Registry: benchmarks gain family / variant / version / family_head / canonical metric / bounds / higher_is_better / harness / comparability_note +
14 entries (gpqa-diamond, aime-2024, swe-bench-pro, arc-agi-2, terminal-bench-2, mmteb, livebench ×7, aider-polyglot-well-formed); seed writes them
and `variant_of` relations, org_kind from `kind`. Organisations: kind + country fields, upstage / stepfun / kwaipilot / kuaishou / thinking-machines /
lg-ai-research / xiaomi / inclusionai entries with hub aliases. Repositories: canonical kinds, agents (agent_kind) and tools get their own entity types.
OpenRouter: prices booked on the OpenRouter provider only (features.upstream_provider), `created` → openrouter_listed_at (no release_date),
canonical modalities, identity helper. Provider pricing: fireworks_model_id (+ legacy scheme), org from the id, family hints.
Labs: no hard-coded Anthropic modalities/tool constants (table-driven), Google pdf → document + google_model_id, canonical licence/status/
capabilities/api_aliases helpers in labs/_common.py, family_model hints everywhere. arXiv/OpenReview: researchers only with an identifier
(ORCID / OpenReview profile). GitHub/PyPI: canonical kind (+kind_raw), licence keys. docs/CONNECTORS.md rewritten on the canonical vocabulary.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Every step plans from reads and reports counts + examples; --apply executes the plan through merge_entities / upsert_relation / a tier-2
derived FactWriter. No deletes, snapshots untouched, a second --apply is a no-op. Optional entity scope for targeted runs and tests.
docs/CANONICALIZATION.md documents every rule, what is never done, counting rules and how to re-run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

- fetch: validate_destination() on every request and redirect hop (max 5, manual loop): non-http(s), localhost/.local/.internal, RFC1918,
  loopback, link-local (metadata), CGNAT, IPv6 loopback/link-local/ULA, unspecified → FetchError("blocked destination")
- facts: facts_to_json / facts_from_json (tagged datetimes) for quarantined runs
- writer: run_id/source_key/is_first_run; a claim supersedes only when tier <= current or same source AND same extractor (LLM never
  overwrites deterministic from the same URL); licence/openness/modalities/status/kind/org_kind normalised via the ontology with value_raw
  + taxonomy_mappings, same assertion in another spelling re-encoded in place (no event); license_key claim; results get config_key,
  trust_level, variant, run_group, extractor, effort config for variant refs, one current row per (model, benchmark, metric, config_key),
  out-of-range scores → confidence low + anomaly; events get recorded_at, is_backfill (classify_backfill), group_key, deterministic
  importance; family/canonical hints materialised (family_id, canonical_id, artifact_kind, member_of_family, artifact_of); NEW_ARTIFACT 0,
  NEW_MODEL_FAMILY 1; conflicting claims recorded once; derived writers never open review items and mark their events backfill
- resolver: model/artifact compatible lookups; keep_separate decisions honoured; digit-collapsing aliases require the same variant_key;
  first_seen_hint → first_seen_at = min(now, hint); resolve_variant() folds evaluator-only effort variants onto the canonical model
- connector: run_id/source_key/first-run flag into the writer; --url reprocess rebuilds Targets from documents.meta._target (key, doc_type,
  meta, needs_llm, entity persisted at first fetch); quarantine for tier ≥ 2 connectors (rolling median baseline of the last 5 full runs:
  new entities > max(50, 1.3×), prices > 3×, entities < 0.8× / results < 0.5× on full extraction) → quarantined_runs + review item, nothing written
- services/events: classify_backfill, group_key_for, importance_for · services/anomalies: record / run_checks / list_anomalies
- merge: modes merge|alias|variant|family_member, keep_separate refusal, org-type and model/artifact pairs, dedupe/config_key recompute with
  collision handling, one current result per key, embeddings/sources/domains/review_queue/family_id/canonical_id re-pointed,
  resolution_decisions + admin_audit_log rows
- handlers: LLM extraction writer carries the snapshot's run_id

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

- models: FAMILY_ORGS / official_orgs / is_official_org; effort_config() and base_name() shared by the writer and the canonicalizer;
  effort suffixes chain ("-thinking-64k-high-effort"), LiveBench budgets ("32k thinking", "(no thinking)"); "max"/"fast"/"instant" are model
  tiers (Qwen3-Max, GPT-5.1-Codex-Max, Grok 4.1 Fast), no longer effort markers; family_release_hint reads "gpt-5-4" as 5.4 and never a size (38B)
- taxonomy: normalize_property(entity_type, prop, value) → (canonical, value_raw, mappings); unknown values kept, mapped to NULL
- benchmarks: RUN_KEYS (release, index_version, version, dataset_revision) leave config_key so a newer run replaces the older one; they are
  condition keys for comparability (partially comparable)
- tests/test_ontology.py: licence table, openness derivation, taxonomy, 22 model names, comparability trio, metric bounds/trust

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>