HTML 77.2%
TypeScript 10.5%
Python 9.6%
JavaScript 2.5%
1# AI Atlas — implementation audit for the canonical upgrade (2026-09-12)23Written before the "temporal knowledge graph + intelligence terminal" upgrade. Numbers are from the production database4(M2M32c, dump `aiatlas-prod.dump` taken 2026-09-11 23:47 EDT, 133 MB) and from reading every module of the repository.56## 1. Existing architecture (keep)78FastAPI + SQLAlchemy Core/asyncpg + Alembic (forward-only SQL) · Redis cache · APScheduler scheduler (`aia schedule`) running connectors9**sequentially** · Postgres job queue (SKIP LOCKED) · LLM gateway → llm-api.io (deterministic first, LLM claims one tier lower) ·10Next 16 App Router (server components, Tailwind v4 tokens, hand-rolled d3 SVG charts, Playwright QA borrowed from `~/Desktop/uqo-eval`).11Deployment: `mld` manifest `deploy/ai-atlas.mld.json`, PM2 `ai-atlas-api :8321 / ai-atlas-scheduler / ai-atlas-web :8320`, MacLustr Tunnel.1213## 2. Existing schema (0001 + 0002)1415`entities` (type, canonical_name, slug, organization_id, attributes/provenance/quality/counts jsonb, first/last_seen, merged_into, search) ·16`entity_aliases` · `entity_identifiers (scheme,value) unique` · `sources` · `connectors` · `connector_runs` · `connector_errors` ·17`documents` · `snapshots (run_id, raw_path, text_path, structured, diff, parser_version)` · `claims (temporal: valid_from/valid_to, status18current|superseded|conflicting|retracted, tier, confidence, extractor)` · `relations (live uniq on subject,predicate,object)` ·19`change_events (observed_at, effective_at, importance, dedupe_key)` · `benchmark_results (config jsonb, dedupe_key, valid_to)` ·20`prices (live uniq per model×provider×provider_model_id)` · `jobs` · `llm_jobs` · `review_queue` · `entity_embeddings` · `stats_snapshots` ·21`page_views` · `api_keys` · `metric_definitions` · `domains`.2223Missing for this upgrade: run/batch id on facts, backfill/occurrence semantics on events, canonical hierarchy columns, benchmark24comparability key and trust level, persisted resolution decisions, anomaly flags, admin audit log, quarantine.2526## 3. Entity types in the data (prod, live rows)2728researcher 1 899 · model 1 880 · paper 1 060 · dataset 146 · company 119 · library 67 · framework 66 · hardware 42 · organization 36 ·29benchmark 27 · provider 27 · repository 10 · lab 8 · university 1 · tool 0 (public nav shows Tools with "0 total").30`ids.ENTITY_TYPES` declares 33 types; `quantization`, `license`, `release`, `agent`, `runtime`, `tool` are declared but never produced.3132## 4. Claim architecture (keep)3334`FactWriter.write_claim`: no current → insert; same → confirm; different & source tier ≤ current → supersede + event for material35properties; different & worse tier → `conflicting` + review item. Soft properties never event. `metric.*` never event. Prices and36results append-only with close/open. **Loophole**: `same_source` lets an LLM claim (tier+1) supersede a deterministic claim from the same37URL (`writer.py:141-145`). **Gap**: no `run_id` on facts; `EntityRef.first_seen_hint` never read (`resolution.py:131`).3839## 5. Connector architecture (keep)4041`BaseConnector.run()` → discover → fetch (conditional) → archive → snapshot (+diff) → parse → extract → `FactWriter` → follow-ups →42optional `llm_extract` job. Breakage detection (`expected_min_records`, only on full re-extraction), circuit breaker, adaptive interval,43reprocess-from-archive. **Bugs**: single-URL reprocess builds key-less targets → no-op (`connector.py:188`); `Fetcher` has **no SSRF guard**44(no private-IP/scheme/redirect validation, follow-ups come from crawled content); rate limit read from class attr, not registry.4546## 6. Current model resolution4748`Resolver`: identifiers (exact scheme+value) → normalized alias within type (org-disambiguated) → slug → create. `normalize_alias`49erases dots/hyphens/spaces (`Qwen3-8B` ≡ `Qwen 38B`). Identifier schemes are per source (`anthropic_model_id`, `gemini_model_id`,50`fireworks-ai_model_id`…) so cross-source identity rests on aliases. Consequences measured in prod:5152* **399 models exist only through an `artificial_analysis` identifier**, 204 of them are *effort/thinking variants* of another model53 (`claude-fable-5-1-xhigh`, `gpt-6-astra-high`, `deepseek-v4-pro-0424-non-reasoning`) — evaluation configurations, not models.54* **157 Hugging Face repos flagged `is_quantized`** (`quant_format` gguf 64, fp8 41, onnx 27, mlx 23, awq 2) plus ~150 conversions/re-packagings55 (`amd/…-MXFP4`, `zai-org/GLM-5-FP8`, `mlx-community/Kimi-K2.5`) are typed `model`; they dominate "largest models" (Kimi K2.5 appears 4×).56* 72 models exist only through OpenRouter; OpenRouter `release_date` = listing date; routed prices booked on the lab's provider entity with57 a different `provider_model_id` → two concurrent "current" prices for one model×provider.58* 19 exact normalized-name duplicates across slugs (`gemini-2.5-flash-lite` / `gemini-2-5-flash-lite`, `deepseek-v3-{2,3,4}`…); 80 pending59 `merge_candidate` items; 140 pending `conflict` items (status 34, modalities_input 30, context_length 26).60* 1 899 researchers created **by name only** (first 3 arXiv authors, no identifier, no affiliation) — homonyms merge, variants split.61* 70 models without organization; 853 without release date; 1 205 without parameter count; only 115 carry `family`.6263## 7. Current benchmark pipeline6465Four boards (Artificial Analysis 4 003 rows, LiveBench 456, SWE-bench 323, aider 138) → 4 920 live rows on 15 of 27 registered benchmarks66(12 registry benchmarks — MMLU, HumanEval, AIME 2025, MATH-500, MTEB, LMArena… — have **zero** results: no connector feeds them).67Leaderboards are one row per live result: LiveBench mixes `global_average` with 7 `category:*` metrics in one board; aider mixes68`pass_rate_2` and `percent_cases_well_formed`; every LiveBench release / aider run date creates new rows that are never retired;69`/compare` picks the last-observed metric per model. Models are resolved by free text (aider, SWE-bench) or AA slug, so AA effort variants70become separate leaderboard entries (GPQA: 614 results = 614 "models"). No trust level, no comparability key, no metric bounds.71The registry lacks family / variant / direction / metric range. The API only resolves benchmarks by exact slug (no aliases).7273## 8. Current event pipeline747512 196 events, **all observed on 2026-09-11** (initial corpus): NEW_MODEL 1 880 (importance 3), BENCHMARK_RESULT 4 920, ANNOUNCEMENT 1 764,76NEW_PAPER 1 060, RELEASE 692, PROVIDER_LISTED 613… `/changes`, `/changes/daily` and `/stats.change_events_24h` key on `observed_at`; only77`/timeline` uses `coalesce(effective_at, observed_at)`. 8 713 events have no `effective_at`. There is no backfill flag, no78occurred/observed/recorded distinction, no semantic dedupe of one release across sources. Homepage says "+12,196 · 24 h".7980## 9. Taxonomy problems found (prod values)8182* license: `apache-2.0` 344 · `Apache 2.0` 9 · `Modified MIT` · `CC BY-NC 4.0` vs `cc-by-nc-4.0` · `other` 157 · Llama variants `llama3`, `llama3.1`… (no SPDX, no permissions)83* openness: `open-weights` 971 · empty 478 · `proprietary` 348 · `restricted` 82 (= HF *gated*, not a licence property) · `open-source` 1 (Cohere prose)84* modalities: `["Text"]` vs `["text"]`; Google emits `pdf`; Anthropic values are hard-coded constants; `capabilities` uses OpenAI slugs and Google labels85* status: `limited-availability` vs schema `available`; `archived` undocumented; `groq_status` side channel86* org kinds: `company|organization|lab|university` as entity types, with duplicates (`Kwaipilot` company + `kwaipilot-2` organization, `Upstage`/`upstage-2`, `StepFun`/`stepfun-ai`, `Thinking Machines`/`thinkingmachines`)87* identifiers: `gemini_model_id` (provider key is `google-gemini-api`), `fireworks-ai_model_id` (hyphen), platform ids stored as claims88* hardware `kind`: `soc|gpu|system|computer`; `memory_gb` int vs list89* benchmarks: `metric` free text (`pass rate (2 attempts)`), `unit` `%` or empty9091## 10. Duplicate / hierarchy problems9293Model = release = checkpoint = quantization = provider alias today. Needed hierarchy: **model_family → model → artifact → deployment**.94Artifact kinds: `checkpoint` (official weights repo), `quantization`, `conversion` (BF16/FP8 repack, MLX, ONNX), `packaging`.95Evaluation-effort variants fold into the canonical model as **result configuration** (`reasoning_effort`, `reasoning`), never entities.9697## 11. Components to preserve9899Archive (`sdk/archive.py`), `FactWriter` temporal rules, `Resolver` precedence, connector SDK and all 27 connectors + 91 fixture tests,100LLM gateway cascade and accounting, `/asof` `/history` `/diff` `/compare` `/prices/index` `/hardware/fit` endpoints and their semantics,101`merge_entities`, quality methodology, design tokens, `KeyValue`/`DataTable`/`Tabs`/`ProvenanceInline`, compare tray, admin console,102sitemap shards, OG frame, slugs, ids, all public URLs.103104## 12. Schema migration 0003 (`0003_canonical_ontology`)105106* `entities`: `family_id`, `canonical_id` (artifact → model, folded variant → model), `artifact_kind`, `identity_confidence`; new types107 `model_family` (prefix `family`), `artifact` (prefix `artifact`); `license` entities created from the ontology (`uses_license` relation).108* facts: `run_id` on `claims`, `relations`, `prices`, `benchmark_results`, `change_events` (+ indexes) for batch rollback.109* `change_events`: `recorded_at`, `is_backfill`, generated `occurred_at = coalesce(effective_at, observed_at)`, `group_key` (semantic110 dedupe of one release across documents), indexes on `(is_backfill, occurred_at)`.111* `benchmark_results`: `trust_level`, `config_key` (comparability), `variant`, `run_group`, `is_current` maintained by the writer112 (one current row per model × benchmark × metric × config_key; older runs closed with `valid_to`).113* new tables: `resolution_decisions`, `anomalies`, `admin_audit_log`, `quarantined_runs`, `taxonomy_mappings`, `watch_digests` (none), `model_families` view.114* benchmark registry gains `family, variant, version, direction, metric_min, metric_max, harness, comparability` fields (claims).115116## 13. Backward compatibility117118* Every existing slug and id keeps resolving: artifacts stay reachable at `/models/<slug>` (API accepts `artifact` on the models mount,119 web 301s to `/artifacts/<slug>`); folded effort variants resolve through `merged_into` (API follows it, web 301s to the canonical model).120* `/models` excludes artifacts and merged rows by default (`include=artifacts` restores the old universe); documented in `docs/API.md`121 as v1.1 with a changelog. No v2 needed: no field is removed or retyped; new fields are additive.122* Raw snapshots are never modified; taxonomy normalisation writes canonical attributes and keeps `*_raw` when the source label differs.123* Counters get explicit definitions ("Models = canonical model releases; artifacts and folded variants excluded").124125## 14. Implementation sequence1261271. Ontology package (`src/aiatlas/ontology/`) + migration 0003 + tests.1282. SDK: writer/resolver use the ontology; `run_id` propagation; `first_seen_hint`; same-source loophole; SSRF guard; reprocess fix; quarantine.1293. `aia canonicalize` (dry-run + apply): artifacts, effort variants, families, licences, taxonomy, org duplicates, event backfill flags,130 benchmark result comparability/trust; `aia anomalies`; `aia audit-data`.1314. Connectors: HF → artifacts, AA → configs, leaderboards metrics split, benchmark registry families, OpenRouter fixes, provider ids.1325. API 1.1: families/artifacts/deployments/frontier/pulse/pareto/finder/cost/matrix/history/comparability/time-machine/claims/admin quality133 & resolution & extraction debugger & audit; counters; backfill-aware feeds; search compiler v2; security fixes.1346. Web: brand, shell (nav + ⌘K commands + evidence drawer + density), homepage 3.0, model page 3.0, compare 3.0, benchmarks 2.0,135 frontier, prices terminal, calculator, run-locally, find-a-model, open, graph, timeline 2.0, time machine, diff, pulse, families,136 watchlist, admin workbenches, mobile, OG per type, SEO.1377. Tests, QA sweeps, final data/benchmark/event/counter audits, deploy, memory.138