# AI Atlas — implementation audit for the canonical upgrade (2026-09-12)

Written before the "temporal knowledge graph + intelligence terminal" upgrade. Numbers are from the production database (M2M32c, dump aiatlas-prod.dump taken 2026-09-11 23:47 EDT, 133 MB) and from reading every module of the repository.

# 1. Existing architecture (keep)

FastAPI + SQLAlchemy Core/asyncpg + Alembic (forward-only SQL) · Redis cache · APScheduler scheduler (aia schedule) running connectors sequentially · Postgres job queue (SKIP LOCKED) · LLM gateway → llm-api.io (deterministic first, LLM claims one tier lower) · Next 16 App Router (server components, Tailwind v4 tokens, hand-rolled d3 SVG charts, Playwright QA borrowed from ~/Desktop/uqo-eval). Deployment: mld manifest deploy/ai-atlas.mld.json, PM2 ai-atlas-api :8321 / ai-atlas-scheduler / ai-atlas-web :8320, MacLustr Tunnel.

# 2. Existing schema (0001 + 0002)

entities (type, canonical_name, slug, organization_id, attributes/provenance/quality/counts jsonb, first/last_seen, merged_into, search) · entity_aliases · entity_identifiers (scheme,value) unique · sources · connectors · connector_runs · connector_errors · documents · snapshots (run_id, raw_path, text_path, structured, diff, parser_version) · claims (temporal: valid_from/valid_to, status current|superseded|conflicting|retracted, tier, confidence, extractor) · relations (live uniq on subject,predicate,object) · change_events (observed_at, effective_at, importance, dedupe_key) · benchmark_results (config jsonb, dedupe_key, valid_to) · prices (live uniq per model×provider×provider_model_id) · jobs · llm_jobs · review_queue · entity_embeddings · stats_snapshots · page_views · api_keys · metric_definitions · domains.

Missing for this upgrade: run/batch id on facts, backfill/occurrence semantics on events, canonical hierarchy columns, benchmark comparability key and trust level, persisted resolution decisions, anomaly flags, admin audit log, quarantine.

# 3. Entity types in the data (prod, live rows)

researcher 1 899 · model 1 880 · paper 1 060 · dataset 146 · company 119 · library 67 · framework 66 · hardware 42 · organization 36 · benchmark 27 · provider 27 · repository 10 · lab 8 · university 1 · tool 0 (public nav shows Tools with "0 total"). ids.ENTITY_TYPES declares 33 types; quantization, license, release, agent, runtime, tool are declared but never produced.

# 4. Claim architecture (keep)

FactWriter.write_claim: no current → insert; same → confirm; different & source tier ≤ current → supersede + event for material properties; different & worse tier → conflicting + review item. Soft properties never event. metric.* never event. Prices and results append-only with close/open. Loophole: same_source lets an LLM claim (tier+1) supersede a deterministic claim from the same URL (writer.py:141-145). Gap: no run_id on facts; EntityRef.first_seen_hint never read (resolution.py:131).

# 5. Connector architecture (keep)

BaseConnector.run() → discover → fetch (conditional) → archive → snapshot (+diff) → parse → extract → FactWriter → follow-ups → optional llm_extract job. Breakage detection (expected_min_records, only on full re-extraction), circuit breaker, adaptive interval, reprocess-from-archive. Bugs: single-URL reprocess builds key-less targets → no-op (connector.py:188); Fetcher has no SSRF guard (no private-IP/scheme/redirect validation, follow-ups come from crawled content); rate limit read from class attr, not registry.

# 6. Current model resolution

Resolver: identifiers (exact scheme+value) → normalized alias within type (org-disambiguated) → slug → create. normalize_alias erases dots/hyphens/spaces (Qwen3-8B ≡ Qwen 38B). Identifier schemes are per source (anthropic_model_id, gemini_model_id, fireworks-ai_model_id…) so cross-source identity rests on aliases. Consequences measured in prod:

399 models exist only through an artificial_analysis identifier, 204 of them are effort/thinking variants of another model (claude-fable-5-1-xhigh, gpt-6-astra-high, deepseek-v4-pro-0424-non-reasoning) — evaluation configurations, not models.
157 Hugging Face repos flagged is_quantized (quant_format gguf 64, fp8 41, onnx 27, mlx 23, awq 2) plus ~150 conversions/re-packagings (amd/…-MXFP4, zai-org/GLM-5-FP8, mlx-community/Kimi-K2.5) are typed model; they dominate "largest models" (Kimi K2.5 appears 4×).
72 models exist only through OpenRouter; OpenRouter release_date = listing date; routed prices booked on the lab's provider entity with a different provider_model_id → two concurrent "current" prices for one model×provider.
19 exact normalized-name duplicates across slugs (gemini-2.5-flash-lite / gemini-2-5-flash-lite, deepseek-v3-{2,3,4}…); 80 pending merge_candidate items; 140 pending conflict items (status 34, modalities_input 30, context_length 26).
1 899 researchers created by name only (first 3 arXiv authors, no identifier, no affiliation) — homonyms merge, variants split.
70 models without organization; 853 without release date; 1 205 without parameter count; only 115 carry family.

# 7. Current benchmark pipeline

Four boards (Artificial Analysis 4 003 rows, LiveBench 456, SWE-bench 323, aider 138) → 4 920 live rows on 15 of 27 registered benchmarks (12 registry benchmarks — MMLU, HumanEval, AIME 2025, MATH-500, MTEB, LMArena… — have zero results: no connector feeds them). Leaderboards are one row per live result: LiveBench mixes global_average with 7 category:* metrics in one board; aider mixes pass_rate_2 and percent_cases_well_formed; every LiveBench release / aider run date creates new rows that are never retired; /compare picks the last-observed metric per model. Models are resolved by free text (aider, SWE-bench) or AA slug, so AA effort variants become separate leaderboard entries (GPQA: 614 results = 614 "models"). No trust level, no comparability key, no metric bounds. The registry lacks family / variant / direction / metric range. The API only resolves benchmarks by exact slug (no aliases).

# 8. Current event pipeline

12 196 events, all observed on 2026-09-11 (initial corpus): NEW_MODEL 1 880 (importance 3), BENCHMARK_RESULT 4 920, ANNOUNCEMENT 1 764, NEW_PAPER 1 060, RELEASE 692, PROVIDER_LISTED 613… /changes, /changes/daily and /stats.change_events_24h key on observed_at; only /timeline uses coalesce(effective_at, observed_at). 8 713 events have no effective_at. There is no backfill flag, no occurred/observed/recorded distinction, no semantic dedupe of one release across sources. Homepage says "+12,196 · 24 h".

# 9. Taxonomy problems found (prod values)

license: apache-2.0 344 · Apache 2.0 9 · Modified MIT · CC BY-NC 4.0 vs cc-by-nc-4.0 · other 157 · Llama variants llama3, llama3.1… (no SPDX, no permissions)
openness: open-weights 971 · empty 478 · proprietary 348 · restricted 82 (= HF gated, not a licence property) · open-source 1 (Cohere prose)
modalities: ["Text"] vs ["text"]; Google emits pdf; Anthropic values are hard-coded constants; capabilities uses OpenAI slugs and Google labels
status: limited-availability vs schema available; archived undocumented; groq_status side channel
org kinds: company|organization|lab|university as entity types, with duplicates (Kwaipilot company + kwaipilot-2 organization, Upstage/upstage-2, StepFun/stepfun-ai, Thinking Machines/thinkingmachines)
identifiers: gemini_model_id (provider key is google-gemini-api), fireworks-ai_model_id (hyphen), platform ids stored as claims
hardware kind: soc|gpu|system|computer; memory_gb int vs list
benchmarks: metric free text (pass rate (2 attempts)), unit % or empty

# 10. Duplicate / hierarchy problems

Model = release = checkpoint = quantization = provider alias today. Needed hierarchy: model_family → model → artifact → deployment. Artifact kinds: checkpoint (official weights repo), quantization, conversion (BF16/FP8 repack, MLX, ONNX), packaging. Evaluation-effort variants fold into the canonical model as result configuration (reasoning_effort, reasoning), never entities.

# 11. Components to preserve

Archive (sdk/archive.py), FactWriter temporal rules, Resolver precedence, connector SDK and all 27 connectors + 91 fixture tests, LLM gateway cascade and accounting, /asof /history /diff /compare /prices/index /hardware/fit endpoints and their semantics, merge_entities, quality methodology, design tokens, KeyValue/DataTable/Tabs/ProvenanceInline, compare tray, admin console, sitemap shards, OG frame, slugs, ids, all public URLs.

# 12. Schema migration 0003 (`0003_canonical_ontology`)

entities: family_id, canonical_id (artifact → model, folded variant → model), artifact_kind, identity_confidence; new types model_family (prefix family), artifact (prefix artifact); license entities created from the ontology (uses_license relation).
facts: run_id on claims, relations, prices, benchmark_results, change_events (+ indexes) for batch rollback.
change_events: recorded_at, is_backfill, generated occurred_at = coalesce(effective_at, observed_at), group_key (semantic dedupe of one release across documents), indexes on (is_backfill, occurred_at).
benchmark_results: trust_level, config_key (comparability), variant, run_group, is_current maintained by the writer (one current row per model × benchmark × metric × config_key; older runs closed with valid_to).
new tables: resolution_decisions, anomalies, admin_audit_log, quarantined_runs, taxonomy_mappings, watch_digests (none), model_families view.
benchmark registry gains family, variant, version, direction, metric_min, metric_max, harness, comparability fields (claims).

# 13. Backward compatibility

Every existing slug and id keeps resolving: artifacts stay reachable at /models/<slug> (API accepts artifact on the models mount, web 301s to /artifacts/<slug>); folded effort variants resolve through merged_into (API follows it, web 301s to the canonical model).
/models excludes artifacts and merged rows by default (include=artifacts restores the old universe); documented in docs/API.md as v1.1 with a changelog. No v2 needed: no field is removed or retyped; new fields are additive.
Raw snapshots are never modified; taxonomy normalisation writes canonical attributes and keeps *_raw when the source label differs.
Counters get explicit definitions ("Models = canonical model releases; artifacts and folded variants excluded").

# 14. Implementation sequence

Ontology package (src/aiatlas/ontology/) + migration 0003 + tests.
SDK: writer/resolver use the ontology; run_id propagation; first_seen_hint; same-source loophole; SSRF guard; reprocess fix; quarantine.
aia canonicalize (dry-run + apply): artifacts, effort variants, families, licences, taxonomy, org duplicates, event backfill flags, benchmark result comparability/trust; aia anomalies; aia audit-data.
Connectors: HF → artifacts, AA → configs, leaderboards metrics split, benchmark registry families, OpenRouter fixes, provider ids.
API 1.1: families/artifacts/deployments/frontier/pulse/pareto/finder/cost/matrix/history/comparability/time-machine/claims/admin quality & resolution & extraction debugger & audit; counters; backfill-aware feeds; search compiler v2; security fixes.
Web: brand, shell (nav + ⌘K commands + evidence drawer + density), homepage 3.0, model page 3.0, compare 3.0, benchmarks 2.0, frontier, prices terminal, calculator, run-locally, find-a-model, open, graph, timeline 2.0, time machine, diff, pulse, families, watchlist, admin workbenches, mobile, OG per type, SEO.
Tests, QA sweeps, final data/benchmark/event/counter audits, deploy, memory.