# AI Atlas — implementation audit for the canonical upgrade (2026-09-12)

Written before the "temporal knowledge graph + intelligence terminal" upgrade. Numbers are from the production database
(M2M32c, dump `aiatlas-prod.dump` taken 2026-09-11 23:47 EDT, 133 MB) and from reading every module of the repository.

## 1. Existing architecture (keep)

FastAPI + SQLAlchemy Core/asyncpg + Alembic (forward-only SQL) · Redis cache · APScheduler scheduler (`aia schedule`) running connectors
**sequentially** · Postgres job queue (SKIP LOCKED) · LLM gateway → llm-api.io (deterministic first, LLM claims one tier lower) ·
Next 16 App Router (server components, Tailwind v4 tokens, hand-rolled d3 SVG charts, Playwright QA borrowed from `~/Desktop/uqo-eval`).
Deployment: `mld` manifest `deploy/ai-atlas.mld.json`, PM2 `ai-atlas-api :8321 / ai-atlas-scheduler / ai-atlas-web :8320`, MacLustr Tunnel.

## 2. Existing schema (0001 + 0002)

`entities` (type, canonical_name, slug, organization_id, attributes/provenance/quality/counts jsonb, first/last_seen, merged_into, search) ·
`entity_aliases` · `entity_identifiers (scheme,value) unique` · `sources` · `connectors` · `connector_runs` · `connector_errors` ·
`documents` · `snapshots (run_id, raw_path, text_path, structured, diff, parser_version)` · `claims (temporal: valid_from/valid_to, status
current|superseded|conflicting|retracted, tier, confidence, extractor)` · `relations (live uniq on subject,predicate,object)` ·
`change_events (observed_at, effective_at, importance, dedupe_key)` · `benchmark_results (config jsonb, dedupe_key, valid_to)` ·
`prices (live uniq per model×provider×provider_model_id)` · `jobs` · `llm_jobs` · `review_queue` · `entity_embeddings` · `stats_snapshots` ·
`page_views` · `api_keys` · `metric_definitions` · `domains`.

Missing for this upgrade: run/batch id on facts, backfill/occurrence semantics on events, canonical hierarchy columns, benchmark
comparability key and trust level, persisted resolution decisions, anomaly flags, admin audit log, quarantine.

## 3. Entity types in the data (prod, live rows)

researcher 1 899 · model 1 880 · paper 1 060 · dataset 146 · company 119 · library 67 · framework 66 · hardware 42 · organization 36 ·
benchmark 27 · provider 27 · repository 10 · lab 8 · university 1 · tool 0 (public nav shows Tools with "0 total").
`ids.ENTITY_TYPES` declares 33 types; `quantization`, `license`, `release`, `agent`, `runtime`, `tool` are declared but never produced.

## 4. Claim architecture (keep)

`FactWriter.write_claim`: no current → insert; same → confirm; different & source tier ≤ current → supersede + event for material
properties; different & worse tier → `conflicting` + review item. Soft properties never event. `metric.*` never event. Prices and
results append-only with close/open. **Loophole**: `same_source` lets an LLM claim (tier+1) supersede a deterministic claim from the same
URL (`writer.py:141-145`). **Gap**: no `run_id` on facts; `EntityRef.first_seen_hint` never read (`resolution.py:131`).

## 5. Connector architecture (keep)

`BaseConnector.run()` → discover → fetch (conditional) → archive → snapshot (+diff) → parse → extract → `FactWriter` → follow-ups →
optional `llm_extract` job. Breakage detection (`expected_min_records`, only on full re-extraction), circuit breaker, adaptive interval,
reprocess-from-archive. **Bugs**: single-URL reprocess builds key-less targets → no-op (`connector.py:188`); `Fetcher` has **no SSRF guard**
(no private-IP/scheme/redirect validation, follow-ups come from crawled content); rate limit read from class attr, not registry.

## 6. Current model resolution

`Resolver`: identifiers (exact scheme+value) → normalized alias within type (org-disambiguated) → slug → create. `normalize_alias`
erases dots/hyphens/spaces (`Qwen3-8B` ≡ `Qwen 38B`). Identifier schemes are per source (`anthropic_model_id`, `gemini_model_id`,
`fireworks-ai_model_id`…) so cross-source identity rests on aliases. Consequences measured in prod:

* **399 models exist only through an `artificial_analysis` identifier**, 204 of them are *effort/thinking variants* of another model
  (`claude-fable-5-1-xhigh`, `gpt-6-astra-high`, `deepseek-v4-pro-0424-non-reasoning`) — evaluation configurations, not models.
* **157 Hugging Face repos flagged `is_quantized`** (`quant_format` gguf 64, fp8 41, onnx 27, mlx 23, awq 2) plus ~150 conversions/re-packagings
  (`amd/…-MXFP4`, `zai-org/GLM-5-FP8`, `mlx-community/Kimi-K2.5`) are typed `model`; they dominate "largest models" (Kimi K2.5 appears 4×).
* 72 models exist only through OpenRouter; OpenRouter `release_date` = listing date; routed prices booked on the lab's provider entity with
  a different `provider_model_id` → two concurrent "current" prices for one model×provider.
* 19 exact normalized-name duplicates across slugs (`gemini-2.5-flash-lite` / `gemini-2-5-flash-lite`, `deepseek-v3-{2,3,4}`…); 80 pending
  `merge_candidate` items; 140 pending `conflict` items (status 34, modalities_input 30, context_length 26).
* 1 899 researchers created **by name only** (first 3 arXiv authors, no identifier, no affiliation) — homonyms merge, variants split.
* 70 models without organization; 853 without release date; 1 205 without parameter count; only 115 carry `family`.

## 7. Current benchmark pipeline

Four boards (Artificial Analysis 4 003 rows, LiveBench 456, SWE-bench 323, aider 138) → 4 920 live rows on 15 of 27 registered benchmarks
(12 registry benchmarks — MMLU, HumanEval, AIME 2025, MATH-500, MTEB, LMArena… — have **zero** results: no connector feeds them).
Leaderboards are one row per live result: LiveBench mixes `global_average` with 7 `category:*` metrics in one board; aider mixes
`pass_rate_2` and `percent_cases_well_formed`; every LiveBench release / aider run date creates new rows that are never retired;
`/compare` picks the last-observed metric per model. Models are resolved by free text (aider, SWE-bench) or AA slug, so AA effort variants
become separate leaderboard entries (GPQA: 614 results = 614 "models"). No trust level, no comparability key, no metric bounds.
The registry lacks family / variant / direction / metric range. The API only resolves benchmarks by exact slug (no aliases).

## 8. Current event pipeline

12 196 events, **all observed on 2026-09-11** (initial corpus): NEW_MODEL 1 880 (importance 3), BENCHMARK_RESULT 4 920, ANNOUNCEMENT 1 764,
NEW_PAPER 1 060, RELEASE 692, PROVIDER_LISTED 613… `/changes`, `/changes/daily` and `/stats.change_events_24h` key on `observed_at`; only
`/timeline` uses `coalesce(effective_at, observed_at)`. 8 713 events have no `effective_at`. There is no backfill flag, no
occurred/observed/recorded distinction, no semantic dedupe of one release across sources. Homepage says "+12,196 · 24 h".

## 9. Taxonomy problems found (prod values)

* license: `apache-2.0` 344 · `Apache 2.0` 9 · `Modified MIT` · `CC BY-NC 4.0` vs `cc-by-nc-4.0` · `other` 157 · Llama variants `llama3`, `llama3.1`… (no SPDX, no permissions)
* openness: `open-weights` 971 · empty 478 · `proprietary` 348 · `restricted` 82 (= HF *gated*, not a licence property) · `open-source` 1 (Cohere prose)
* modalities: `["Text"]` vs `["text"]`; Google emits `pdf`; Anthropic values are hard-coded constants; `capabilities` uses OpenAI slugs and Google labels
* status: `limited-availability` vs schema `available`; `archived` undocumented; `groq_status` side channel
* org kinds: `company|organization|lab|university` as entity types, with duplicates (`Kwaipilot` company + `kwaipilot-2` organization, `Upstage`/`upstage-2`, `StepFun`/`stepfun-ai`, `Thinking Machines`/`thinkingmachines`)
* identifiers: `gemini_model_id` (provider key is `google-gemini-api`), `fireworks-ai_model_id` (hyphen), platform ids stored as claims
* hardware `kind`: `soc|gpu|system|computer`; `memory_gb` int vs list
* benchmarks: `metric` free text (`pass rate (2 attempts)`), `unit` `%` or empty

## 10. Duplicate / hierarchy problems

Model = release = checkpoint = quantization = provider alias today. Needed hierarchy: **model_family → model → artifact → deployment**.
Artifact kinds: `checkpoint` (official weights repo), `quantization`, `conversion` (BF16/FP8 repack, MLX, ONNX), `packaging`.
Evaluation-effort variants fold into the canonical model as **result configuration** (`reasoning_effort`, `reasoning`), never entities.

## 11. Components to preserve

Archive (`sdk/archive.py`), `FactWriter` temporal rules, `Resolver` precedence, connector SDK and all 27 connectors + 91 fixture tests,
LLM gateway cascade and accounting, `/asof` `/history` `/diff` `/compare` `/prices/index` `/hardware/fit` endpoints and their semantics,
`merge_entities`, quality methodology, design tokens, `KeyValue`/`DataTable`/`Tabs`/`ProvenanceInline`, compare tray, admin console,
sitemap shards, OG frame, slugs, ids, all public URLs.

## 12. Schema migration 0003 (`0003_canonical_ontology`)

* `entities`: `family_id`, `canonical_id` (artifact → model, folded variant → model), `artifact_kind`, `identity_confidence`; new types
  `model_family` (prefix `family`), `artifact` (prefix `artifact`); `license` entities created from the ontology (`uses_license` relation).
* facts: `run_id` on `claims`, `relations`, `prices`, `benchmark_results`, `change_events` (+ indexes) for batch rollback.
* `change_events`: `recorded_at`, `is_backfill`, generated `occurred_at = coalesce(effective_at, observed_at)`, `group_key` (semantic
  dedupe of one release across documents), indexes on `(is_backfill, occurred_at)`.
* `benchmark_results`: `trust_level`, `config_key` (comparability), `variant`, `run_group`, `is_current` maintained by the writer
  (one current row per model × benchmark × metric × config_key; older runs closed with `valid_to`).
* new tables: `resolution_decisions`, `anomalies`, `admin_audit_log`, `quarantined_runs`, `taxonomy_mappings`, `watch_digests` (none), `model_families` view.
* benchmark registry gains `family, variant, version, direction, metric_min, metric_max, harness, comparability` fields (claims).

## 13. Backward compatibility

* Every existing slug and id keeps resolving: artifacts stay reachable at `/models/<slug>` (API accepts `artifact` on the models mount,
  web 301s to `/artifacts/<slug>`); folded effort variants resolve through `merged_into` (API follows it, web 301s to the canonical model).
* `/models` excludes artifacts and merged rows by default (`include=artifacts` restores the old universe); documented in `docs/API.md`
  as v1.1 with a changelog. No v2 needed: no field is removed or retyped; new fields are additive.
* Raw snapshots are never modified; taxonomy normalisation writes canonical attributes and keeps `*_raw` when the source label differs.
* Counters get explicit definitions ("Models = canonical model releases; artifacts and folded variants excluded").

## 14. Implementation sequence

1. Ontology package (`src/aiatlas/ontology/`) + migration 0003 + tests.
2. SDK: writer/resolver use the ontology; `run_id` propagation; `first_seen_hint`; same-source loophole; SSRF guard; reprocess fix; quarantine.
3. `aia canonicalize` (dry-run + apply): artifacts, effort variants, families, licences, taxonomy, org duplicates, event backfill flags,
   benchmark result comparability/trust; `aia anomalies`; `aia audit-data`.
4. Connectors: HF → artifacts, AA → configs, leaderboards metrics split, benchmark registry families, OpenRouter fixes, provider ids.
5. API 1.1: families/artifacts/deployments/frontier/pulse/pareto/finder/cost/matrix/history/comparability/time-machine/claims/admin quality
   & resolution & extraction debugger & audit; counters; backfill-aware feeds; search compiler v2; security fixes.
6. Web: brand, shell (nav + ⌘K commands + evidence drawer + density), homepage 3.0, model page 3.0, compare 3.0, benchmarks 2.0,
   frontier, prices terminal, calculator, run-locally, find-a-model, open, graph, timeline 2.0, time machine, diff, pulse, families,
   watchlist, admin workbenches, mobile, OG per type, SEO.
7. Tests, QA sweeps, final data/benchmark/event/counter audits, deploy, memory.