API 1.1 (Stream C): canonical model universe, comparability-grouped leaderboards, deployments & cost, intelligence surfaces, admin workbenches
Existing routes (additive, v1 shapes kept): /models canonical universe + include=artifacts, family_id join, canonical licence
matching, families/licenses/trust facets, /models/{slug} accepts artifacts and reports redirected_from; EntityDetail adds family,
artifacts, deployments, identity, licence, openness, version_history, grouped benchmarks and runs in <= 4 connection groups; /stats
adds organizations_total, artifacts, model_families, change_events_live_24h and counter definitions; /changes, /changes/daily
("Today in AI 2.0" with group_key folding), /timeline, entity timelines and /diff default to is_backfill=false on occurred_at
(include_backfill, date_field restore v1); /benchmarks exposes family/variant/metric/direction/groups/trust_mix and resolves aliases;
/compare groups benchmark dimensions by (benchmark, metric, config_key) with comparability, diff_only and mode; /prices adds
family/org/modality filters and sort=cheapest_frontier; /prices/index becomes the AI Price Index; /providers adds distributions,
30-day churn, organizations_covered, features_supported; /search compiler v2 (compiled/sort/residual/unrecognised, org verified
against aliases, guarded casts, FTS fallback when pgvector is absent); /methodology, /trending?kind=.
New routes: /benchmarks/{slug}/leaderboard, /benchmarks/{slug}/frontier, /benchmarks/matrix, /models/{a}/diff/{b}, /deployments,
/cost, /cost/context, /frontier, /pareto, /pulse, /open, /find-a-model, /run-locally, /hardware/{slug}/fit, /families,
/families/{slug}, /graph/explore, /time-machine, /claims/{id}, /entities/{slug}/claims, /entities/{slug}/provenance/{property},
/licenses, /licenses/{key}; admin: /admin/quality, /admin/entity-resolution (GET + POST decisions via merge_entities mode= when
available), /admin/anomalies, /admin/extractions/{snapshot_id}, /admin/quarantine (501 until services.canonical ships),
/admin/audit, /admin/runs/{run_id}/rollback, /admin/cache/flush?after_run=1.
Services: frontier.py (comparability groups, one-row-per-model leaderboards, frontier model set, leader_at), finder.py, pareto.py,
cost.py, hardware_fit.py (architecture-aware KV cache, observed artifact sizes, multi-GPU), search.py v2, stats.py definitions.
Security/perf: rate_limit("admin") before require_admin, failed-auth limiter 10/min/IP, x-forwarded-for only with AIA_TRUST_PROXY,
set_limit GUC leak removed, weak ETag + Cache-Control + 304 on public GETs, admin audit log on every admin call. EXPLAIN ANALYZE on the
production copy: every read path < 20 ms, no index migration needed.
Docs: docs/API.md rewritten as the 1.1 contract (changelog, 1.1 additions, semantics changes, deprecations: none).
Tests: tests/test_api_v11.py (contract) and tests/test_api_services.py (pareto, cost, compiler, hardware fit); 44 API tests pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
34 changed files +4,815 −476
modified
docs/API.md
+175 −105
@@ -1,10 +1,24 @@ | |||||
| 1 | −# AI Atlas API — contract (v1) | 1 | +# AI Atlas API — contract (API 1.1, mounted at `/api/v1`) | ||
| 2 | 2 | ||||
| 3 | Base: `/api/v1` (FastAPI, `src/aiatlas/api`). JSON, UTF-8, ISO-8601 UTC timestamps, numbers as numbers (Postgres aggregates may arrive as | 3 | Base: `/api/v1` (FastAPI, `src/aiatlas/api`). JSON, UTF-8, ISO-8601 UTC timestamps, numbers as numbers (Postgres aggregates may arrive as | ||
| 4 | −strings — the web layer normalises). Public routes are cached in Redis (`aia:api:*`, 60–600 s) and rate-limited per IP on search. | 4 | +strings — the web layer normalises). Public routes are cached in Redis (`aia:api:*`, 60–600 s), carry a weak `ETag` + `Cache-Control: public, | ||
| 5 | −Admin routes require header `x-aia-admin-token: <AIA_ADMIN_TOKEN>`. OpenAPI at `/api/v1/docs`. Errors: `{"detail": "..."}` with 400/404/429/503. | 5 | +max-age=60, stale-while-revalidate=300` and answer `304` to a matching `If-None-Match`; search is rate-limited per IP. Every response carries | ||
| 6 | +`x-api-version: 1.1`. Admin routes require header `x-aia-admin-token: <AIA_ADMIN_TOKEN>` and are rate-limited **before** the token check | ||||
| 7 | +(240/min per IP; 10 failed authentications per minute per IP → 429). OpenAPI at `/api/v1/docs`. Errors: `{"detail": "..."}` with 400/404/409/422/429/501/503. | ||||
| 6 | Never expose credentials, internal hostnames, raw archive content or `raw_path`s. | 8 | Never expose credentials, internal hostnames, raw archive content or `raw_path`s. | ||
| 7 | 9 | ||||
| 10 | +**1.1 is additive over v1.** No v1 route, field or type was removed or retyped (see "Deprecations": none). New fields and routes are listed | ||||
| 11 | +below; three defaults changed semantics (see "Semantics changes") and each has a parameter that restores the v1 behaviour. Every number | ||||
| 12 | +that is derived or estimated comes with a `methodology`, `note`, `definition` or `estimated: true` string next to it; when data is missing the | ||||
| 13 | +value is `null`/empty with a note — nothing is fabricated. | ||||
| 14 | + | ||||
| 15 | +## Changelog | ||||
| 16 | + | ||||
| 17 | +| version | date | changes | | ||||
| 18 | +|---|---|---| | ||||
| 19 | +| **1.1** | 2026-09-12 | Canonical model universe (`/models` excludes artifacts and folded variants; `include=artifacts`), model detail blocks (family, artifacts, deployments, identity, licence, openness, version_history, grouped benchmarks, `redirected_from`), backfill-aware feeds keyed on `occurred_at`, "Today in AI 2.0", comparability groups + one-row-per-model leaderboards, `/benchmarks/matrix`, `/benchmarks/{slug}/frontier`, grouped `/compare` + `diff_only` + `/models/{a}/diff/{b}`, `Deployment` shape + `/deployments` + `/cost`, AI Price Index, provider aggregates, `/frontier`, `/pareto`, `/pulse`, `/open`, `/find-a-model`, `/run-locally`, `/hardware/{slug}/fit`, `/families`, `/graph/explore`, `/time-machine`, `/diff` sections, search compiler v2 (`compiled`), `/claims/{id}`, `/entities/{slug}/claims`, `/entities/{slug}/provenance/{property}`, `/licenses`, richer `/methodology`, `/trending?kind=`, `/stats` counter definitions, admin workbenches (quality, entity-resolution, anomalies, extractions, quarantine, audit, rollback), ETag/304, admin rate limit before auth, failed-auth limiter, trust-proxy, `set_limit` GUC leak fixed, detail fan-out ≤ 4 connections. | | ||||
| 20 | +| 1.0 | 2026-09-11 | Initial contract (table "v1 routes" below). | | ||||
| 21 | + | ||||
| 8 | ## Shared shapes | 22 | ## Shared shapes | ||
| 9 | 23 | ||||
| 10 | ```ts | 24 | ```ts | ||
@@ -19,132 +33,188 @@ type Provenance = Record<string, { source_id: string | null; source_name?: strin | |||||
| 19 | confidence: string; extractor: string; unit?: string }> | 33 | confidence: string; extractor: string; unit?: string }> | ||
| 20 | type ChangeEvent = { id: string; event_type: string; category: string; property: string | null; old_value: unknown; new_value: unknown; | 34 | type ChangeEvent = { id: string; event_type: string; category: string; property: string | null; old_value: unknown; new_value: unknown; | ||
| 21 | summary: string; importance: 0 | 1 | 2 | 3; observed_at: string; effective_at: string | null; source_url: string | null; connector_name: string | null; | 35 | summary: string; importance: 0 | 1 | 2 | 3; observed_at: string; effective_at: string | null; source_url: string | null; connector_name: string | null; | ||
| 22 | − entity: EntitySummary | null; meta: Record<string, unknown> } | 36 | + entity: EntitySummary | null; meta: Record<string, unknown>; | ||
| 37 | + occurred_at?: string; is_backfill?: boolean; group_key?: string | null } // 1.1 (feeds) | ||||
| 23 | type Price = { id: string; model: EntitySummary; provider: EntitySummary; provider_model_id: string | null; input_per_mtok: Num; output_per_mtok: Num; | 38 | type Price = { id: string; model: EntitySummary; provider: EntitySummary; provider_model_id: string | null; input_per_mtok: Num; output_per_mtok: Num; | ||
| 24 | cached_input_per_mtok: Num; cache_write_per_mtok: Num; batch_input_per_mtok: Num; batch_output_per_mtok: Num; per_image: Num; currency: string; | 39 | cached_input_per_mtok: Num; cache_write_per_mtok: Num; batch_input_per_mtok: Num; batch_output_per_mtok: Num; per_image: Num; currency: string; | ||
| 25 | context_length: Num; max_output_tokens: Num; features: Record<string, unknown>; observed_at: string; valid_from: string; valid_to: string | null; source_url: string | null; tier: number } | 40 | context_length: Num; max_output_tokens: Num; features: Record<string, unknown>; observed_at: string; valid_from: string; valid_to: string | null; source_url: string | null; tier: number } | ||
| 26 | type BenchmarkResult = { id: string; model: EntitySummary; benchmark: EntitySummary; score: number; metric: string | null; unit: string | null; | 41 | type BenchmarkResult = { id: string; model: EntitySummary; benchmark: EntitySummary; score: number; metric: string | null; unit: string | null; | ||
| 27 | − higher_is_better: boolean; config: Record<string, unknown>; evaluated_at: string | null; observed_at: string; source_url: string | null; tier: number; confidence: string } | 42 | + higher_is_better: boolean; config: Record<string, unknown>; evaluated_at: string | null; observed_at: string; source_url: string | null; tier: number; confidence: string; | ||
| 43 | + valid_to: string | null; config_key?: string | null; trust_level?: string | null } // 1.1 adds config_key/trust_level where returned | ||||
| 28 | type Claim = { id: string; property: string; value: unknown; unit: string | null; tier: number; confidence: string; status: string; extractor: string; | 44 | type Claim = { id: string; property: string; value: unknown; unit: string | null; tier: number; confidence: string; status: string; extractor: string; | ||
| 29 | observed_at: string; effective_at: string | null; valid_from: string; valid_to: string | null; source_url: string | null; source_name: string | null } | 45 | observed_at: string; effective_at: string | null; valid_from: string; valid_to: string | null; source_url: string | null; source_name: string | null } | ||
| 30 | type SourceRef = { source_id: string | null; source_name: string | null; domain: string | null; tier: number | null; url: string; doc_type: string; last_observed_at: string | null; snapshots: number } | 46 | type SourceRef = { source_id: string | null; source_name: string | null; domain: string | null; tier: number | null; url: string; doc_type: string; last_observed_at: string | null; snapshots: number } | ||
| 47 | + | ||||
| 48 | +// ---- 1.1 shapes | ||||
| 49 | +type Deployment = { id: string; model: EntitySummary; provider: EntitySummary; provider_model_id: string | null; context_length: Num; max_output_tokens: Num; | ||||
| 50 | + prices: { input: Num; cached_input: Num; cache_write: Num; output: Num; batch_input: Num; batch_output: Num; per_image: Num; per_request: Num; currency: string; | ||||
| 51 | + unit: "USD per 1M tokens"; native_units: Record<string, unknown> }; // provider-specific priced features kept verbatim (flex_*, priority_*, *_per_1k_requests…) | ||||
| 52 | + features: Record<string, unknown>; status: "active" | "delisted"; observed_at: string; valid_from: string; valid_to: string | null; source_url: string | null; tier: number } | ||||
| 53 | +type Group = { metric: string; config_key: string; label: string; n: number; model_count: number; config: Record<string, unknown>; higher_is_better: boolean; trust_mix: Record<string, number> } | ||||
| 54 | +type LeaderboardRow = { rank: number; model: ModelRef; score: number; metric: string; unit: string | null; higher_is_better: boolean; delta_rank: number | null; previous_rank: number | null; | ||||
| 55 | + trust_level: string; trust_label: string; config: Record<string, unknown>; config_key: string; comparability: "comparable" | "partially-comparable" | "not-comparable"; | ||||
| 56 | + comparability_reasons: string[]; evaluated_at: string | null; observed_at: string; source_url: string | null; tier: number; result_id: string; n_rows: number } | ||||
| 57 | +type ModelRef = { id: string; slug: string; name: string; entity_type: string; organization: Org; attributes: Record<string, unknown> } // openness/params/context/release/modalities/license/family | ||||
| 58 | +type LicenseInfo = { key: string; label: string; category: string; spdx: string | null; url: string | null; commercial_use: boolean | null; redistribution: boolean | null; | ||||
| 59 | + derivatives: boolean | null; hosting_restrictions: boolean | null; attribution: boolean | null; acceptable_use: boolean | null; osi_approved: boolean; weights_downloadable: boolean } | ||||
| 60 | +type Fit = { quantization: string; estimated: true; fits: boolean; estimated_memory_gb: number; headroom_gb: number; parameter_count?: number; | ||||
| 61 | + breakdown: { weights_gb: number; weights_source: "observed" | "estimated"; overhead_gb: number; kv_cache_gb: number; kv_cache_method: "architecture" | "heuristic"; reserved_gb: number; context: number; batch: number }; | ||||
| 62 | + device: { memory_gb: number; gpu_count: number; total_memory_gb: number }; note: string; multi_gpu_note?: string } | ||||
| 31 | ``` | 63 | ``` | ||
| 32 | 64 | ||||
| 33 | −## Public routes | 65 | +## v1 routes (unchanged unless noted "1.1:") | ||
| 34 | 66 | ||||
| 35 | | route | returns | | 67 | | route | returns | | ||
| 36 | |---|---| | 68 | |---|---| | ||
| 37 | −| `GET /health` | `{ status: "ok"|"degraded", version, db: bool, redis: bool, llm: { available, reachable? }, time }` | | 69 | +| `GET /health` | `{ status: "ok"|"degraded", version, api_version: "1.1", db: bool, redis: bool, llm: { available, reachable? }, time }` | | ||
| 38 | −| `GET /stats` | `{ entities: Record<type, number>, entities_total, sources, connectors, connectors_enabled, documents, snapshots, claims, claims_current, relations, change_events, change_events_24h, change_events_7d, benchmark_results, prices_current, prices_total, review_pending, llm_jobs, llm_tokens, last_snapshot_at, last_event_at, first_entity_at, archive: { raw_bytes, raw_files, text_bytes, text_files }, computed_at }` — **always live from the DB** | | 70 | +| `GET /stats` | `{ entities: Record<type, number>, entities_total, sources, connectors, connectors_enabled, documents, snapshots, claims, claims_current, relations, change_events, change_events_24h, change_events_7d, benchmark_results, prices_current, prices_total, review_pending, llm_jobs, llm_tokens, last_snapshot_at, last_event_at, first_entity_at, archive: {…}, computed_at }` — **1.1:** `entities.model` = canonical models; `entities.artifact`, `entities.model_family` (zero-filled); adds `organizations_total`, `artifacts`, `model_families`, `change_events_live_24h`, `change_events_live_7d`, `definitions: Record<key, string>` ("How counted"). Cached 60 s. | | ||
| 39 | | `GET /stats/history?days=90` | `{ items: { day: string; counts: object }[] }` | | 71 | | `GET /stats/history?days=90` | `{ items: { day: string; counts: object }[] }` | | ||
| 40 | −| `GET /search?q=&type=&limit=30&offset=0` | `{ query: CompiledQuery, items: (EntitySummary & { rank: number })[], total: number }` — uses `services.search.compile_query` + `search_entities` (+ embedding when the gateway is up) | | 72 | +| `GET /search?q=&type=&limit=30&offset=0` | `{ query: CompiledQuery, items: (EntitySummary & { rank: number })[], total: number }` — **1.1:** compiler v2 (below) | | ||
| 41 | | `GET /search/suggest?q=` | `{ items: { id, entity_type, slug, name, organization_name }[] }` (≤ 8, prefix, fast) | | 73 | | `GET /search/suggest?q=` | `{ items: { id, entity_type, slug, name, organization_name }[] }` (≤ 8, prefix, fast) | | ||
| 42 | −| `GET /entities/{slug_or_id}` | `EntityDetail` (below). Also mounted as `/models/{slug}`, `/companies/{slug}`, `/papers/{slug}`, `/providers/{slug}`, `/benchmarks/{slug}`, `/hardware/{slug}`, `/frameworks/{slug}`, `/datasets/{slug}`, `/tools/{slug}` (404 if the type doesn't match) | | 74 | +| `GET /entities/{slug_or_id}` | `EntityDetail` (below). Also mounted as `/models/{slug}` (**1.1:** accepts `model` AND `artifact`), `/companies/{slug}`, `/papers/{slug}`, `/providers/{slug}`, `/benchmarks/{slug}` (**1.1:** slug, alias or id), `/hardware/{slug}`, `/frameworks/{slug}`, `/datasets/{slug}`, `/tools/{slug}` (404 if the type doesn't match) | | ||
| 43 | −| `GET /entities/{slug}/timeline?limit=50&before=` | `{ items: ChangeEvent[] }` (entity's own events + events of entities it develops for companies) | | 75 | +| `GET /entities/{slug}/timeline?limit=50&before=&include_backfill=0&date_field=occurred` | `{ items: ChangeEvent[], next_before, date_field, include_backfill }` — **1.1:** defaults `is_backfill=false`, ordered by `occurred_at`; cursor = `occurred_at` (or `observed_at` with `date_field=observed`) | | ||
| 44 | | `GET /entities/{slug}/history?property=` | `{ items: Claim[] }` — full claim history (all statuses), newest first | | 76 | | `GET /entities/{slug}/history?property=` | `{ items: Claim[] }` — full claim history (all statuses), newest first | | ||
| 45 | −| `GET /entities/{slug}/asof?date=YYYY-MM-DD` | `{ existed: bool, first_seen_at, date, attributes: Record<string, unknown>, claims: Claim[] }` — state as known at that date | | 77 | +| `GET /entities/{slug}/asof?date=YYYY-MM-DD` | `{ existed: bool, first_seen_at, date, attributes, claims: Claim[] }` | | ||
| 46 | −| `GET /entities/{slug}/graph?depth=1&limit=80` | `{ nodes: { id, slug, name, entity_type, organization_name? }[], edges: { source, target, predicate }[] }` (depth ≤ 2, cap nodes) | | 78 | +| `GET /entities/{slug}/graph?depth=1&limit=80` | `{ root, nodes: { id, slug, name, entity_type, organization_name }[], edges: { source, target, predicate }[] }` | | ||
| 47 | | `GET /entities/{slug}/sources` | `{ items: SourceRef[] }` | | 79 | | `GET /entities/{slug}/sources` | `{ items: SourceRef[] }` | | ||
| 48 | −| `GET /entities/{slug}/related?limit=` | `{ items: EntitySummary[] }` (same org / same family / shared relations) | | 80 | +| `GET /entities/{slug}/related?limit=` | `{ items: EntitySummary[] }` (same org / same family (`family_id` or label) / shared relations) | | ||
| 49 | −| `GET /models?q=&org=&family=&openness=&modality=&status=&min_params=&max_params=&min_context=&year_from=&year_to=&license=&sort=(updated|name|params|context|release|quality|downloads)&order=&limit=&offset=` | `Page<EntitySummary>` + `facets` when `facets=1`: `{ organizations: {slug,name,count}[], openness: {value,count}[], modalities, families, years, licenses, status }` | | 81 | +| `GET /models?q=&org=&family=&openness=&modality=&status=&min_params=&max_params=&min_context=&year_from=&year_to=&license=&sort=&order=&limit=&offset=&facets=1` | `Page<EntitySummary & { identity_confidence, family?: {id,slug,name}, canonical?: EntitySummary, artifact_kind? }> & { universe }` — **1.1:** default universe = `entity_type='model' and merged_into is null`; `include=artifacts`; `family=` matches `model_family` slug/id/name via `entities.family_id` (falls back to `attributes.family`); `license=` matches the canonical key (`attributes.license_key`) or any raw label the ontology maps to it; `reasoning=0|1`, `trust=high,medium,low` (identity_confidence), `sort=cheapest`. Facets add `families` (`{value,label,canonical,count}`), canonical `licenses` (`{value,label,category,count,raw_labels,raw?}`), `trust`, `definitions`. | | ||
| 50 | −| `GET /companies?q=&country=&kind=&sort=(models|name|updated|quality)&limit=&offset=` | `Page<EntitySummary & { model_count: number; paper_count: number }>` (+`facets`: countries, kinds) | | 82 | +| `GET /companies?q=&country=&kind=&sort=&limit=&offset=&facets=1` | `Page<EntitySummary & { model_count, paper_count }>` — covers `company|organization|lab|university`; `total` = `/stats.organizations_total` | | ||
| 51 | −| `GET /papers?q=&category=&org=&since=&until=&sort=(published|updated)&limit=&offset=` | `Page<EntitySummary>` | | 83 | +| `GET /papers?…` | `Page<EntitySummary>` | | ||
| 52 | −| `GET /providers` | `{ items: (EntitySummary & { model_count: number; price_count: number; min_input_per_mtok: Num; min_output_per_mtok: Num })[] }` | | 84 | +| `GET /providers` | `{ items: (EntitySummary & { model_count, price_count, min_input_per_mtok, min_output_per_mtok, input_price_distribution: {min,p25,median,p75,max,n}|null, output_price_distribution, models_added_30d, models_removed_30d, price_changes_30d, organizations_covered, features_supported: string[], feature_keys: string[] })[], note }` — **1.1** aggregates | | ||
| 53 | −| `GET /prices?model=&provider=&sort=(input|output|model|provider|observed)&limit=&offset=¤t=1` | `Page<Price>` | | 85 | +| `GET /prices?model=&provider=&sort=&limit=&offset=¤t=1&family=&org=&modality=` | `Page<Price>` — **1.1:** `family`, `org`, `modality` filters; `sort=cheapest_frontier` (frontier models only, cheapest output first, + `methodology`) | | ||
| 54 | −| `GET /prices/history?model=&provider=` | `{ items: Price[] }` (all rows incl. closed, oldest first) | | 86 | +| `GET /prices/history?model=&provider=` | `{ items: Price[] }` | | ||
| 55 | −| `GET /prices/index?days=180` | `{ series: { day: string; median_input: Num; median_output: Num; min_input: Num; models: number }[]; movers: ChangeEvent[] }` computed from `prices` history | | 87 | +| `GET /prices/index?days=180` | **AI Price Index (1.1)** `{ days, series: { day, median_input, median_output, median_frontier_output, median_frontier_input, median_open_output, median_embedding_input, min_input, max_input, min_frontier_output, models, offers, sample: { models, offers, frontier_models, frontier_offers, open_models, embedding_models } }[], movers: ChangeEvent[], cheapest_frontier: { model, provider, output, input, context_length, price_id }|null, cheapest_frontier_1m_context, distribution: { metric, unit, buckets: { from, to, label, offers }[], offers }, new_listings_30d, delistings_30d, price_changes_30d, frontier: { composition, methodology }, methodology, note }` — v1 keys kept | | ||
| 56 | −| `GET /benchmarks` | `{ items: (EntitySummary & { result_count: number; model_count: number; top: { model: EntitySummary; score: number } | null })[] }` | | 88 | +| `GET /benchmarks?category=` | `{ items: { id, entity_type, slug, name, category, family, variant, metric, unit, direction, attributes, result_count, model_count, leader: LeaderboardRow|null, second, top: {model, score}|null, primary_group: Group|null, groups: Group[], trust_mix, trust_labels }[], total, note }` — **1.1:** `result_count` = current rows, `model_count` = distinct canonical models, `leader` from the primary metric group | | ||
| 57 | −| `GET /benchmarks/{slug}/results?limit=&offset=&config=` | `Page<BenchmarkResult>` sorted by score (respecting `higher_is_better`), current rows only unless `history=1` | | 89 | +| `GET /benchmarks/{slug}` | `EntityDetail` + `family, variant, metric, direction, category, groups, primary_group, result_count, model_count, leaderboard (top 25), trust_mix` | | ||
| 58 | −| `GET /benchmarks/{slug}/history?model=` | `{ items: BenchmarkResult[] }` over time | | 90 | +| `GET /benchmarks/{slug}/results?limit=&offset=&config=&history=&metric=&config_key=` | `Page<BenchmarkResult>` — v1 behaviour (one row per result) + `metric` / `config_key` filters | | ||
| 59 | −| `GET /hardware?kind=&manufacturer=&min_memory=&sort=` | `Page<EntitySummary>` | | 91 | +| `GET /benchmarks/{slug}/history?model=` | `{ benchmark, items: BenchmarkResult[] }` | | ||
| 60 | −| `GET /hardware/fit?memory_gb=&quant=(4bit|8bit|fp16)&context=8192&limit=` | `{ inputs, assumptions: string[], items: { model: EntitySummary; parameter_count: number; estimated_memory_gb: number; fits: boolean; headroom_gb: number; quantization: string; note: string }[] }` — **ESTIMATED**: bytes/param (4bit 0.5+overhead 1.15, 8bit 1.0, fp16 2.0) + KV cache estimate; label as estimated | | 92 | +| `GET /hardware?…` | `Page<EntitySummary>` + facets | | ||
| 61 | −| `GET /explore/types` | `{ items: { entity_type, count, label }[] }` | | 93 | +| `GET /hardware/fit?memory_gb=&quant=&context=&limit=&openness=` | v1 estimate (unchanged) | | ||
| 62 | −| `GET /explore/{type}?q=&org=&sort=&limit=&offset=` | `Page<EntitySummary>` generic listing for any type (datasets, frameworks, tools, repositories, regulation…) | | 94 | +| `GET /explore/types` · `GET /explore/{type}` | unchanged (`artifact`, `model_family` appear as types once present) | | ||
| 63 | −| `GET /changes?category=&type=&entity_type=&importance_min=&since=&until=&q=&limit=50&before=<observed_at cursor>` | `Page<ChangeEvent>` newest first; `type` = comma list of event types; excludes `DOCUMENT_CHANGED` unless `include_documents=1` | | 95 | +| `GET /changes?category=&type=&entity_type=&importance_min=&since=&until=&q=&entity=&limit=50&before=&offset=&include_documents=0&include_backfill=0&date_field=occurred` | `Page<ChangeEvent> & { next_before, date_field, include_backfill }` — **1.1:** defaults `is_backfill=false`, `occurred_at`; cursor/filters follow `date_field` | | ||
| 64 | −| `GET /changes/daily?date=YYYY-MM-DD` | `{ date, counts: Record<category, number>, sections: { category, label, items: ChangeEvent[] }[], new_models: EntitySummary[] }` ("What changed in AI today", DB-generated only) | | 96 | +| `GET /changes/daily?date=&per_section=30&include_backfill=0&date_field=occurred` | `{ date, counts, total, sections (v1 by category), today: { key, label, items: (ChangeEvent & { sources: n, documents: string[], grouped_events, event_ids? })[], total }[], new_models, labels, backfill_excluded, previous_day, next_day, note }` — **Today in AI 2.0** sections: `MAJOR_RELEASES, OPEN_WEIGHT_RELEASES, PRICE_MOVES, BENCHMARK_MOVES, MODEL_CHANGES, RESEARCH, DEPRECATIONS, PROVIDER_CHANGES, HARDWARE`; events sharing a `group_key` are folded into one item | | ||
| 65 | −| `GET /changes/categories?days=7` | `{ items: { category, event_type, count }[] }` | | 97 | +| `GET /changes/categories?days=7&include_backfill=0` | `{ items: { category, event_type, count }[] }` | | ||
| 66 | −| `GET /timeline?entity=&year=&category=&limit=` | `{ items: { month: string; events: ChangeEvent[] }[] }` grouped by month; global when no entity | | 98 | +| `GET /timeline?entity=&year=&category=&importance_min=&limit=&include_backfill=0&date_field=occurred` | `{ items: { month, count, events }[], total, date_field, include_backfill }` | | ||
| 67 | −| `GET /compare?ids=slug,slug,…` (2–6) | `{ entity_type, dimensions: { key, label, unit?, kind: "number"|"text"|"list"|"bool"|"date" }[], items: { entity: EntitySummary; values: Record<key, unknown>; provenance: Provenance; prices?: Price[]; results?: BenchmarkResult[] }[] }` — dimensions per type: model (params, active params, context, max output, openness, license, modalities, release date, knowledge cutoff, best price in/out, benchmark scores shared by all), provider (models, min prices, features), hardware (memory, bandwidth, tdp, runtimes), framework (version, license, stars), company (country, founded, models, papers) | | 99 | +| `GET /compare?ids=a,b,…&diff_only=0&mode=` | `{ entity_type, dimensions, items, comparability: Record<dimKey, { level, reasons, trust }>, diff_only, note }` — **1.1:** benchmark dimensions keyed `bench:<benchmark>:<metric>:<config_key>` (only groups where EVERY model has a current result; dimension carries `comparability`, `trust_levels`), model dims add `reasoning`, `tool_calling`; `diff_only=1` keeps differing dimensions; `mode=models|providers|hardware|companies|frameworks|benchmarks` asserts the type (400 otherwise) | | ||
| 68 | −| `GET /diff?a=YYYY-MM-DD&b=YYYY-MM-DD&scope=(all|models|org:<slug>|family:<name>)` | `{ a, b, new_entities: EntitySummary[], gone_entities: EntitySummary[], property_changes: ChangeEvent[], price_changes: ChangeEvent[], benchmark_changes: ChangeEvent[], counts }` | | 100 | +| `GET /diff?a=&b=&scope=&limit=&include=&include_backfill=0` | `{ a, b, scope, new_entities, gone_entities, property_changes, price_changes, benchmark_changes, new_benchmark_leaders: { benchmark, at_a, at_b }[], provider_changes, hardware_changes, context_changes, retired_models, counts, include_artifacts, include_backfill, note }` — **1.1:** events on `occurred_at`, not back-filled; `new_entities` excludes artifacts unless `include=artifacts` | | ||
| 69 | −| `GET /sources` | `{ items: { key, name, domain, tier, kind, category, organization: Org, enabled, documents, last_crawled_at, connectors: { name, label, health, last_success_at, interval_seconds }[] }[] }` (public transparency page) | | 101 | +| `GET /sources` | unchanged | | ||
| 70 | −| `GET /methodology` | `{ metrics: metric_definitions[], confidence_levels, tiers, event_types, extractors }` | | 102 | +| `GET /methodology` | v1 keys + **1.1:** `openness: { categories, labels, definitions, dimensions }`, `trust_levels`, `comparability` (rules, task/condition/ignored keys), `counters` (how counted), `anomaly_checks`, `event_semantics`, `hardware_fit`, `frontier`, `find_a_model`, `licence_categories` | | ||
| 71 | −| `GET /trending?days=7&limit=12` | `{ items: (EntitySummary & { views: number })[] }` from `page_views` | | 103 | +| `GET /trending?days=7&limit=12&type=&kind=views` | `{ days, kind, items, definition }` — **1.1:** `kind=views` (v1, `views`) · `most_changed` · `new_listings` · `new_results` (each `events`, `last_event_at`); separate lists, never merged | | ||
| 72 | −| `POST /views` `{ path }` | `{ ok: true }` (beacon; 1 req/s/IP) | | 104 | +| `POST /views` · `GET /sitemap` · `GET /api-keys/me` | unchanged | | ||
| 73 | −| `GET /sitemap?type=&limit=5000&offset=0` | `{ items: { slug, entity_type, updated_at }[], total }` | | ||||
| 74 | −| `GET /api-keys/me` (header `x-api-key`) | `{ label, plan, rate_per_min, usage_count }` — developer keys (later); public routes work without a key | | ||||
| 75 | 105 | ||||
| 76 | ### `EntityDetail` | 106 | ### `EntityDetail` | ||
| 77 | 107 | ||||
| 78 | ```ts | 108 | ```ts | ||
| 79 | EntitySummary & { | 109 | EntitySummary & { | ||
| 80 | − attributes: Record<string, unknown> // full | 110 | + attributes: Record<string, unknown>; provenance: Provenance; aliases: string[]; identifiers: { scheme, value }[] | ||
| 81 | − provenance: Provenance | 111 | + relations: { predicate, direction: "out"|"in", items: EntitySummary[], total }[]; sources: SourceRef[]; timeline: ChangeEvent[] // timeline: occurred_at order, not back-filled | ||
| 82 | − aliases: string[]; identifiers: { scheme: string; value: string }[] | 112 | + redirected_from?: { slug, id, entity_type } // 1.1: the caller asked for a merged/folded row — web should 301 to this detail's slug | ||
| 83 | − relations: { predicate: string; direction: "out" | "in"; items: EntitySummary[]; total: number }[] // grouped, ≤ 24 per group | 113 | + // model | ||
| 84 | − sources: SourceRef[] // documents describing this entity (deduped by URL) | 114 | + prices?: Price[]; price_history?: Price[]; results?: BenchmarkResult[]; lineage?: { ancestors, descendants, quantizations }; providers?: EntitySummary[] | ||
| 85 | − timeline: ChangeEvent[] // latest 30 | 115 | + hardware_fit?: {…}[]; hardware_fit_assumptions?: string[]; papers?: EntitySummary[]; repositories?: EntitySummary[] | ||
| 86 | − quality: {...}; counts: {...} | 116 | + family?: EntitySummary | { id: null, name, canonical: false, note } | null // 1.1 | ||
| 87 | − // type-specific blocks (present only when relevant) | 117 | + artifacts?: { items: { kind, items: (EntitySummary & { artifact_kind })[], count }[], total } // 1.1, grouped by artifact_kind | ||
| 88 | − prices?: Price[] // current rows across providers (model) or for this provider (provider) | 118 | + deployments?: Deployment[] // 1.1 | ||
| 89 | − price_history?: Price[] // closed + current, oldest first (model) | 119 | + identity?: { canonical_model: true, official_checkpoints: string[], official_artifacts, third_party_artifacts, provider_deployments, folded_variants, api_aliases: string[], note } | ||
| 90 | − results?: BenchmarkResult[] // model: its results · benchmark: leaderboard top 100 | 120 | + licence?: LicenseInfo & { raw, url_observed } | { key: null, raw, note } // 1.1 | ||
| 91 | − lineage?: { ancestors: EntitySummary[]; descendants: EntitySummary[]; quantizations: EntitySummary[] } // model | 121 | + openness?: { category, raw, label, definition, dimensions: Record<dim, boolean|null>, note } // 1.1 | ||
| 92 | − providers?: EntitySummary[] // model | 122 | + version_history?: { property, transitions: { from, to, valid_from, valid_to, effective_at, source_url, tier, claim_id, status }[], current }[] // context_length, max_output_tokens, status, knowledge_cutoff, license, openness, parameter_count | ||
| 93 | − hardware_fit?: { hardware: EntitySummary; quantization: string; estimated_memory_gb: number; fits: boolean }[] // model, ESTIMATED | 123 | + benchmarks?: { items: { id, slug, name, category, metrics: { metric, groups: { config_key, comparability_group, n_rows, higher_is_better, best: {…trust_level…}, trust_levels }[] }[] }[], total_rows, note } | ||
| 94 | − models?: Page<EntitySummary> // company / provider / hardware(runnable) | 124 | + family_id?: string | null; identity_confidence?: "high"|"medium"|"low" | ||
| 95 | − papers?: EntitySummary[] // company / model | 125 | + // artifact (1.1) | ||
| 96 | − repositories?: EntitySummary[] // company / model / framework | 126 | + canonical?: EntitySummary | null; artifact_kind?: "checkpoint"|"quantization"|"conversion"|"packaging"|null | ||
| 127 | + // company / provider / hardware / framework / model_family | ||||
| 128 | + models?: Page<EntitySummary>; papers?; repositories? | ||||
| 97 | } | 129 | } | ||
| 98 | ``` | 130 | ``` | ||
| 99 | 131 | ||||
| 100 | −## Admin routes (`x-aia-admin-token`) | 132 | +## 1.1 additions | ||
| 133 | + | ||||
| 134 | +| route | returns | | ||||
| 135 | +|---|---| | ||||
| 136 | +| `GET /models/{a}/diff/{b}` | `{ a, b, dimensions: (Dimension & { a, b, delta: { absolute, percent } | { added, removed } | null })[], comparability, note }` — differing dimensions only | | ||||
| 137 | +| `GET /entities/{slug}/claims?property=&status=current|all|superseded|conflicting|retracted&limit=&offset=` | `{ entity, items: (Claim & { snapshot_id, run_id, value_raw })[], total, limit, offset, status }` | | ||||
| 138 | +| `GET /entities/{slug}/provenance/{property}` | evidence drawer: `{ entity, property, value, value_raw, unit, source: { id, name, domain, url }, tier, confidence, extractor, extractor_version, observed_at, effective_at, valid_since, claim_id, run_id, snapshot_id, snapshot: { id, observed_at, document_url, title, archived }, conflicts: Claim[], history_count, note }` (404 when no claim and no attribute) | | ||||
| 139 | +| `GET /claims/{id}` | `{ claim, entity, property, chain: { previous: Claim[], superseding: Claim[], conflicting: Claim[], history_count }, source: { id, name, domain, tier, url, snapshot_id, observed_at }, extractor: { name, version, confidence }, run_id, evidence: { snapshot_id, document_url, archived, snapshot_observed_at, document_title, doc_type }, note }` | | ||||
| 140 | +| `GET /benchmarks/{slug}/leaderboard?metric=&config_key=&trust=&org=&since=&until=&comparable_only=0&limit=100&offset=0` | `{ benchmark, group: Group, groups: Group[], items: LeaderboardRow[], total, limit, offset, comparable_only, filters, history_available, methodology }` — ONE row per canonical model (best current row in the group; default group = primary metric, most-populated config); `delta_rank` vs the ranking of the closed rows of the same group when history exists | | ||||
| 141 | +| `GET /benchmarks/{slug}/frontier?metric=&config_key=` | `{ benchmark, series: { group: Group, primary: bool, points: { date, model: ModelRef, score, trust_level, config, result_id }[], current_leader }[], generated_at, methodology }` — leader history per group (a point each time a new best appears, ordered by `coalesce(evaluated_at, observed_at)`, closed rows included) | | ||||
| 142 | +| `GET /benchmarks/matrix?benchmarks=a,b,c&models=…&org=&family=&limit=60&comparable_only=0&min_cells=3` | `{ columns: { id, slug, name, category, metric, config_key, group_label, higher_is_better, n_models }[], rows: { model: {…}, cells: Record<benchmarkId, { score, rank, trust_level, config_key, comparability, result_id } | null>, n_cells, mean_rank }[], total_rows, comparable_only, min_cells, methodology }` — default columns = 12 benchmarks with most current results, default rows = models with ≥ `min_cells` cells (`mean_rank` is a sort key, not a score) | | ||||
| 143 | +| `GET /deployments?model=&provider=&org=¤t=1&sort=valid_from|output|input|model|provider&limit=50&offset=&before=<valid_from cursor>` | `Page<Deployment> & { next_before, current }` | | ||||
| 144 | +| `GET /cost?model=&provider=&input_tokens=1000&output_tokens=500&requests_per_day=1000&cached_share=0..1&batch=0|1` | `{ model, inputs, items: { deployment: Deployment, cost: { per_request, daily, monthly, annual, effective_input_per_mtok, effective_output_per_mtok, per_request_fee, inputs, notes: string[] } }[], total, currency, methodology, note }` — cached/batch prices used only when published (else a `notes` entry) | | ||||
| 145 | +| `GET /cost/context?tokens=1000000&limit=50&model=&org=` | `{ tokens, items: { deployment, context_length, context_source: "offer"|"model attribute", cost_usd }[], total, currency, methodology }` — offers whose context ≥ tokens, cheapest first | | ||||
| 146 | +| `GET /frontier?limit=12` | `{ latest_major_models: ChangeEvent[], benchmark_frontier: { benchmark, group, leader, second, gap }[], price_frontier: { cheapest_output: Deployment, cheapest_output_1m_context, frontier_models, composition }, context_frontier: { model, context_length }[], open_weight_frontier: { items: { model, best_rank, best_rank_on, parameter_count, context_length, ranks }[], dimensions, note }, efficiency_frontier: { quality: { benchmark, group }, x, points: { id, model, x, y, rank, trust_level, pareto }[], frontier: string[] }, agentic_frontier: { benchmark, group, leaders: LeaderboardRow[] }[], multimodal_frontier: { model, modalities, top10_on }[], recent_frontier_movements: (ChangeEvent & { percent_change? })[], generated_at, methodology }` | | ||||
| 147 | +| `GET /pareto?benchmark=&x=output_price|input_price|parameter_count|context_length|memory_estimate&y=score&metric=&config_key=&org=&family=&openness=` | `{ benchmark, group, groups, x: { key, label }, y, points: { id, model, x, y, rank, trust_level, config, provider?, estimated?, pareto }[], frontier: string[], methodology }` — `x=latency` → 400 (not stored, never estimated) | | ||||
| 148 | +| `GET /pulse?days=7` | `{ days, since, until, counters: Record<key, { value, definition, median_percent?, items? }>, note }` — keys: `new_models, new_open_weight_models, new_artifacts, new_papers, provider_listings, provider_delistings, price_changes, new_models_1m_context, new_benchmark_leaders, documents_changed, sources_observed, events_total`; all `is_backfill=false`, `occurred_at` in window | | ||||
| 149 | +| `GET /open?sort=release|params|context|rank|name|downloads&license=&min_params=&max_params=&min_context=&modality=&days=&openness=&limit=&offset=` | `Page<{ model, licence: LicenseInfo & { raw } | { key: null, raw, note }, dimensions, best_results: { benchmark, rank }[], best_rank, hardware_fit: { "4bit_64gb": Fit, "8bit_128gb": Fit, estimated: true }, providers, cheapest_output_per_mtok }> & { summary: { by_category, by_license_top, new_30d }, note }` — universe = downloadable weights (open-weights, open-source, restricted-weights) | | ||||
| 150 | +| `GET /find-a-model?use_case=coding|reasoning|agentic|long_context|vision|low_cost|local|embeddings|chat&deployment=local|api|any&memory_gb=&quant=&context_min=&license=commercial|any&openness=&max_input_price=&max_output_price=&modalities=&limit=30` | `{ matches: { model, why: string[], observed: {…}, estimated_fit?: Fit, deployments?: Deployment[] }[], total, filters_applied, rules, note }` — deterministic rules (see `/methodology.find_a_model`), sorted by satisfied criteria → best benchmark rank → release date; **no composite score** | | ||||
| 151 | +| `GET /run-locally?memory_gb=&gpu_count=1|2|4|8&quant=&context=8192&batch=1&platform=apple|nvidia|amd|any&use_case=&openness=&limit=` | `{ inputs, estimated: true, assumptions, counts, items: { model, fit: Fit, artifacts: { artifact, quant_format, file_size_gb, weights_source: "observed"|"estimated", fit: Fit }[], artifact_count }[], note }` — artifacts with an observed `file_size_gb` use it as the weights size | | ||||
| 152 | +| `GET /hardware/{slug}/fit?quant=&context=&memory_gb=&gpu_count=&limit=&openness=` | `{ hardware, memory_options_gb, inputs, estimated: true, assumptions, runtimes, counts, items: ({ model } & Fit)[] }` | | ||||
| 153 | +| `GET /families?q=&org=&sort=models|name|last_release&limit=&offset=` | `Page<{ id, slug, name, canonical: bool, entity_type: "model_family", organization, model_count, first_release, last_release, param_range, modalities, licenses, benchmark_best }> & { note }` — `model_family` entities + legacy `attributes.family` labels (`canonical: false`) | | ||||
| 154 | +| `GET /families/{slug}?limit=` | `{ id, slug, name, canonical, summary, …aggregates, members: { model, key_facts, benchmark_ranks }[], artifacts_count, providers, lineage: { source, target, predicate }[], timeline, note }` — accepts a family label pre-canonicalisation | | ||||
| 155 | +| `GET /graph/explore?node=&mode=lineage|research|company|benchmark|dataset|provider|hardware&depth=1|2&limit=150` | `{ root, mode, depth, predicates, nodes: { id, slug, name, entity_type, org, org_slug, level, artifact_kind, attributes }[], edges: { source, target, predicate, attributes, tier }[], truncated, counts }` — never more than `limit` nodes | | ||||
| 156 | +| `GET /time-machine?date=YYYY-MM-DD&scope=models|prices|benchmarks|hardware|all&limit=50` | `{ date, scope, first_entity_at, reconstructed, note, models?: { items: { model, attributes_as_of, observed_then, reconstructed }[], total }, prices?: { items: Deployment[], total }, benchmarks?: { leaders: { benchmark, leader }[] }, hardware?: { items } }` — before `first_entity_at` the state is reconstructed from claims/release dates (`reconstructed: true`) | | ||||
| 157 | +| `GET /licenses` · `GET /licenses/{key}?limit=` | `{ items: (LicenseInfo & { aliases, models })[], total, categories, unclassified: { raw, models }[], note }` · `LicenseInfo & { aliases, models: Page<EntitySummary> }` | | ||||
| 158 | + | ||||
| 159 | +### Search compiler v2 (`/search`) | ||||
| 160 | + | ||||
| 161 | +`query` = `CompiledQuery & { compiled: { filter, label, value, source_span }[], sort, residual, unrecognised: string[], semantic, version: 2, note? }`. | ||||
| 162 | +Recognised: `open|open-weights|open source`, `proprietary|closed`, `reasoning|thinking models`, `vision|multimodal|image|audio|video|embedding(s)`, | ||||
| 163 | +`under/below/cheaper than $X/M [input|output]`, `fit(s) in|runs on N GB`, `on my mac`, `context ≥ 1M|1M context|long-context`, `since|after 2025`, | ||||
| 164 | +`in 2026`, `released this year|last year`, `last N days|weeks|months`, `by <Org>|from <Org>|<Org> models…` (verified against organization names | ||||
| 165 | +and aliases — an unknown name is demoted to free text and listed in `unrecognised`), `over|≥|more than 100B`, `under 30B`, `between 7B and 70B`, | ||||
| 166 | +`apache|mit|bsd|cc-by|gpl|llama|gemma license`, `commercial use`, `papers …`, `benchmark <name>`, `provider <name>`, sort hints `cheapest|newest|largest|smallest|best`. | ||||
| 167 | +`memory_gb` becomes an ESTIMATED parameter bound (4-bit, 8K context) — noted in `query.note`. All numeric casts in SQL are regex-guarded. | ||||
| 168 | + | ||||
| 169 | +## Semantics changes (1.1) | ||||
| 170 | + | ||||
| 171 | +1. **Models universe.** `/models`, `/hardware/fit`, `/open`, `/run-locally`, leaderboards, matrix, price index and `/stats.entities.model` count | ||||
| 172 | + **canonical model releases**: `entity_type = 'model' and merged_into is null`. Artifacts (`entity_type = 'artifact'`: checkpoints, quantisations, | ||||
| 173 | + conversions, packagings) and folded evaluation variants (rows with `merged_into`) are excluded. `include=artifacts` on `/models` restores the | ||||
| 174 | + pre-1.1 universe. Every old slug keeps resolving: `/models/<artifact slug>` returns the artifact detail (`canonical`, `artifact_kind`); | ||||
| 175 | + `/models/<folded variant slug>` returns the canonical model with `redirected_from`. | ||||
| 176 | +2. **Event date fields.** `/changes`, `/changes/daily`, `/changes/categories`, `/timeline`, `/entities/{slug}/timeline`, `/diff`, `/pulse`, | ||||
| 177 | + `/trending?kind≠views` use `occurred_at = coalesce(effective_at, observed_at)` and exclude `is_backfill = true` rows. `include_backfill=1` and | ||||
| 178 | + `date_field=observed` restore the v1 behaviour; cursors follow the chosen field. `/stats.change_events_24h` keeps the v1 meaning | ||||
| 179 | + (observed) and `change_events_live_24h` is the new "what happened today" counter. | ||||
| 180 | +3. **Leaderboard grouping.** Results are organised in comparability groups (canonical metric × `config_key`, the hash of task-defining | ||||
| 181 | + configuration keys). `/benchmarks` `leader`/`top`, `/benchmarks/{slug}/leaderboard`, `/benchmarks/matrix`, `/compare` benchmark dimensions, | ||||
| 182 | + `/pareto`, `/frontier` and `/find-a-model` ranks are computed per group with one row per canonical model. `/benchmarks/{slug}/results` keeps the | ||||
| 183 | + v1 one-row-per-result listing. Rows written before migration 0003 (NULL `config_key`/`trust_level`) are recomputed on read from `config` and | ||||
| 184 | + the source key, so the API behaves identically before and after canonicalisation. | ||||
| 185 | + | ||||
| 186 | +Also: `/companies.total == /stats.organizations_total`; `/benchmarks/{slug}` resolves aliases (`GPQA Diamond`); `/entities/{slug}/related` | ||||
| 187 | +uses `family_id` when present; `x-forwarded-for` is honoured only when `settings.trust_proxy` / `AIA_TRUST_PROXY=1`. | ||||
| 188 | + | ||||
| 189 | +## Deprecations | ||||
| 190 | + | ||||
| 191 | +None. No v1 route, parameter, field or type was removed or retyped in 1.1. | ||||
| 192 | + | ||||
| 193 | +## Admin routes (`x-aia-admin-token`; rate limit 240/min per IP evaluated BEFORE the token; 10 failed auths/min per IP → 429; every call except the polling GETs `overview`, `infrastructure`, `llm/health`, `audit` writes an `admin_audit_log` row `{ actor: 'admin', action: '<METHOD> <path>', target, payload: { method, path, query?, body? }, ip }`) | ||||
| 101 | 194 | ||||
| 102 | | route | returns / action | | 195 | | route | returns / action | | ||
| 103 | |---|---| | 196 | |---|---| | ||
| 104 | −| `GET /admin/overview` | `{ stats, queue: Record<kind, Record<status, n>>, heartbeats, connectors: { ok, degraded, failing, disabled }, review_pending, recent_errors: n, llm: { jobs_24h, tokens_24h, by_stage } , archive }` | | 197 | +| `GET /admin/overview` · `GET /admin/connectors` · `POST /admin/connectors/{name}/run` · `PATCH /admin/connectors/{name}` · `GET /admin/runs` · `GET /admin/errors` | v1 (unchanged) | | ||
| 105 | −| `GET /admin/connectors` | full `connectors` rows + last run + docs/snapshots counts + source tier | | 198 | +| `GET /admin/documents…` · `GET /admin/documents/{id}` · `GET /admin/snapshots/{id}` · `GET /admin/jobs` · `POST /admin/jobs/{id}/retry` · `POST /admin/jobs/requeue-dead` · `GET /admin/llm-jobs` · `GET /admin/llm/health` · `POST /admin/llm/enqueue` · `POST /admin/reprocess` | v1 (unchanged) | | ||
| 106 | −| `POST /admin/connectors/{name}/run` `{ force?: bool }` | enqueue an immediate run (Redis flag `aia:run-now:<name>` consumed by the scheduler tick) → `{ queued: true }` | | 199 | +| `GET /admin/review` · `POST /admin/review/{id}` · `GET /admin/entities/duplicates?type=&limit=&threshold=` · `POST /admin/entities/merge` · `POST /admin/entities/{id}/claims/{claim_id}/retract` | v1 — **1.1:** `duplicates` no longer calls `set_limit()` (GUC leak): the `%` operator uses the index at the default 0.3 threshold and `similarity() > :threshold` applies the exact bound | | ||
| 107 | −| `PATCH /admin/connectors/{name}` `{ enabled?, interval_seconds?, priority? }` | update | | 200 | +| `GET /admin/infrastructure` · `POST /admin/stats/recompute` · `POST /admin/quality/recompute` | v1 (unchanged) | | ||
| 108 | −| `GET /admin/runs?connector=&limit=` · `GET /admin/errors?connector=&limit=` | rows | | 201 | +| `POST /admin/cache/flush?prefix=&after_run=0` | `{ flushed, prefix }`; `after_run=1` → `{ flushed, by_prefix }` over the read paths a connector run invalidates (`/api/v1/stats`, `/changes*`, `/benchmarks*`, `/prices*`, `/models*`, `/deployments`, `/frontier`, `/pulse`, `/open`, `/families`, `/timeline`, `/diff`, `facets:`, `frontier:`). The scheduler calls `services.cache.cache_invalidate(prefix)` with the same prefixes after each run (Stream A). | | ||
| 109 | −| `GET /admin/documents?connector=&status=&q=&limit=&offset=` · `GET /admin/documents/{id}` | documents (+ snapshots list); `GET /admin/snapshots/{id}` → metadata + `structured` + `diff` + first 20 kB of cleaned text (never raw HTML) | | 202 | +| **`GET /admin/quality`** | data-health dashboard: `duplicate_candidates { pending_decisions, review_merge_candidates }`, `taxonomy_violations { unmapped_taxonomy_rows, openness_unknown_vocab, status_unknown_vocab, license_unclassified }`, `impossible_values { count, by_check, sample }`, `conflicting_t1_claims`, `models_without_organization`, `models_without_release_source`, `models_without_parameters`, `orphan_benchmark_results` (model merged or artifact), `benchmarks_without_results`, `unresolved_provider_deployments` (provider_model_id matching no identifier), `quantisations_typed_as_models` (ontology name analysis says artifact), `stale_sources` (last success > 3× interval), `empty_public_categories`, `quarantined_runs_pending` — each `{ count, sample }` — plus `review_queue_priority: { kind, id, reasons: string[], … }[]` (frontier models, benchmark leaders, largest params/context claims, price anomalies, major orgs, duplicates of frontier models first) | | ||
| 110 | −| `GET /admin/jobs?status=&kind=` · `POST /admin/jobs/{id}/retry` · `POST /admin/jobs/requeue-dead` | queue | | 203 | +| **`GET /admin/entity-resolution?type=model&status=pending|decided|all&limit=&threshold=0.8`** | `{ items: { a: Side, b: Side, signals: { kind: review_merge_candidate|trigram_similarity|same_variant_key, … }[], similarity, same_variant_key, same_organization, decision, hint }[], total, decisions }` — `Side` = names, org, family, params, release date, architecture, hf_repo, identifiers, aliases, relations/sources/claims/prices/results counts, `variant_key`, `name_analysis` | | ||
| 111 | −| `GET /admin/llm-jobs?limit=` · `GET /admin/llm/health` | LLM factory accounting | | 204 | +| **`POST /admin/entity-resolution/{a}/{b}`** `{ decision: merge|alias|variant_of|family_member|keep_separate|defer, note? }` | persists `resolution_decisions` (upsert on a, b, decision) and applies: `merge_entities(conn, a, b, mode=<decision>)` when the service signature accepts `mode` (checked with `inspect.signature`), otherwise plain merge for `merge`, `canonical_id` + merge for `variant_of`, alias row for `alias`, `family_id` for `family_member` (b must be a `model_family`); matching pending `merge_candidate` review items are approved → `{ ok, a, b, decision, applied, effect }` | | ||
| 112 | −| `GET /admin/review?status=pending&kind=` · `POST /admin/review/{id}` `{ action: "approve"|"reject"|"edit", resolution?: object }` | review queue; approving a `merge_candidate` calls `merge_entities(source, target)` (aliases/identifiers/claims/relations/events moved, `merged_into` set) | | 205 | +| **`GET /admin/anomalies?status=open|all&severity=&check=&limit=&offset=`** · **`POST /admin/anomalies/{id}`** `{ status: resolved|ignored|open, note? }` | anomaly flags (never deleted) | | ||
| 113 | −| `GET /admin/entities/duplicates?type=&limit=` | candidate pairs by normalized name similarity (`pg_trgm`) within a type | | 206 | +| **`GET /admin/extractions/{snapshot_id}?text_limit=`** | extraction debugger: snapshot metadata (no paths), `text` (≤ 20 kB cleaned text), `structured`, `diff`, `previous_snapshot`, `claims`, `relations`, `results`, `prices`, `events`, `llm_jobs`, `entity_candidates`, `spans: { claim_id, property, value, found, offset?, match?, context?, tried? }[]`, `spans_found` — value location is best effort, not-found is reported | | ||
| 114 | −| `POST /admin/entities/merge` `{ source_id, target_id }` · `POST /admin/entities/{id}/claims/{claim_id}/retract` | curation | | 207 | +| **`GET /admin/quarantine?status=pending|all`** · **`POST /admin/quarantine/{id}`** `{ action: release|discard, note? }` | held facts; the POST calls `services.canonical.release_quarantine/discard_quarantine` (imported lazily) → 501 with a note while Stream A has not shipped them | | ||
| 115 | −| `POST /admin/reprocess` `{ connector, url? }` | enqueue `reprocess_snapshot` | | 208 | +| **`GET /admin/audit?limit=&offset=&action=`** | `{ items: { id, actor, action, target, payload, ip, created_at }[], total }` | | ||
| 116 | −| `POST /admin/llm/enqueue` `{ limit?, task? }` | queue `llm_extract` for pending snapshots | | 209 | +| **`POST /admin/runs/{run_id}/rollback`** | retracts claims (`status = retracted`), closes relations / prices / results (`valid_to`, `is_current = false`), flags events `is_backfill = true` + `meta.rolled_back`, re-materialises attributes from the best remaining claim — only rows with that `run_id`; deletes nothing → `{ ok, run_id, connector, counts, note }` | | ||
| 117 | −| `GET /admin/infrastructure` | heartbeats, archive size, DB size (`pg_database_size`), table sizes, node hostname, uptime | | 210 | + | ||
| 118 | −| `POST /admin/cache/flush` | `{ flushed: n }` | | 211 | +## Implementation notes (`src/aiatlas/api`) | ||
| 119 | −| `POST /admin/stats/recompute` · `POST /admin/quality/recompute` | maintenance | | 212 | + | ||
| 120 | − | 213 | +- **Validation errors**: parameters rejected by FastAPI/pydantic typing or bounds return **422** with `{ detail, errors }`; semantic errors raised by the routes return **400**. | ||
| 121 | −## Implementation notes (2026-09-11, `src/aiatlas/api`) | 214 | +- **Caching**: public GETs are cached in Redis (`aia:api:<path>?<sorted query>`, 60–600 s) and served with a weak ETag (`W/"sha1(body)"`) computed on the uncompressed body inside GZip; `/stats` stays at 60 s. Cache invalidation: `services.cache.cache_invalidate(prefix)`; `POST /admin/cache/flush?after_run=1`. | ||
| 122 | − | 215 | +- **Connections**: `EntityDetail` runs its blocks in ≤ 4 concurrent groups (one pooled connection each, sequential inside a group) instead of one connection per block. | ||
| 123 | −Deviations and additions relative to the table above — everything else is implemented as written. | 216 | +- **Performance (2026-09-12, prod copy, EXPLAIN ANALYZE)**: leaderboard load 8 ms · matrix / all groups 8 ms · models listing 0.5 ms (+ count 1.2 ms, facets ≤ 4 ms) · changes feed 0.8 ms (count 3 ms) · daily sections 0.4 ms · price index 180 d 9 ms · frontier composition 1.2 ms · stats 7 ms · deployments cursor 0.4 ms · providers 2.5 ms · free-text search 16 ms. No query exceeded 200 ms, so **no migration 0004 was needed** (the 0003 partial indexes on `(is_backfill, occurred_at)`, `(benchmark_id, config_key, is_current)`, `family_id`, `canonical_id` are used). The admin trigram self-join (`/admin/entities/duplicates`, `/admin/entity-resolution`) takes ~120 ms in SQL + per-pair side-by-side lookups; it is admin-only and not cached. | ||
| 124 | − | 217 | +- **Search**: `query.semantic` says whether an embedding was used; on embedding timeout (2 s) the API backs off to FTS-only for 5 minutes. | ||
| 125 | −- **Validation errors**: parameters rejected by FastAPI/pydantic typing or bounds (`limit=999`, non-integer `depth`…) return **422** with | 218 | +- **`hardware_fit`** (model detail) is omitted when `parameter_count` is unknown; every fit payload carries `estimated: true`. | ||
| 126 | − `{ detail: "<field>: <message>", errors: [...] }`; semantic errors raised by the routes (bad `sort`, bad date, wrong `scope`, `ids` count) return **400**. | 219 | +- **Rate limits** are in-process sliding windows (search 60/min, views 1/s, admin 240/min, admin-auth-failed 10/min per IP) — fine for the single uvicorn worker behind Next.js. | ||
| 127 | −- **`/companies`** (listing, alias, `/compare` type `company`) covers the organization-like types `company | organization | lab | university`. | 220 | +- **Migrations**: `0002_api_indexes` (read paths), `0003_canonical_ontology` (hierarchy, event semantics, comparability, admin tables) — both required by 1.1. | ||
| 128 | − `/frameworks/{slug}` also accepts `library | runtime`; `/tools/{slug}` accepts `tool | agent | application | product | mcp_server`. | ||||
| 129 | −- **`/explore/{type}`** accepts plural forms (`/explore/models`, `/explore/companies`) and adds `entity_type` + `label` to the page. | ||||
| 130 | −- **`EntitySummary.description`** is truncated at 280 characters in summaries (full text in `EntityDetail`). | ||||
| 131 | −- **Cursor feeds** (`/changes`, `/entities/{slug}/timeline`) add `next_before` (the `observed_at` of the last item, or `null` at the end). | ||||
| 132 | − `/changes` also honours `offset` and `entity=` for convenience; `total` is capped at 10 000. | ||||
| 133 | −- **`/changes/daily`** adds `total`, `labels`, `previous_day`, `next_day`; sections are capped at `per_section` (default 30) items each. | ||||
| 134 | −- **`/timeline`** month groups carry `count`; the response includes `total` (events returned). | ||||
| 135 | −- **`/search`**: `query` carries `semantic: bool` (embedding used or not). When the embedding endpoint times out (2 s) the API backs off to | ||||
| 136 | − FTS-only for 5 minutes (`aia:api:search:embed-degraded`) so the LLM never adds latency to consecutive queries. | ||||
| 137 | −- **`hardware_fit`** (model detail) is *omitted* when `parameter_count` is unknown (nothing is estimated from thin air); when present the | ||||
| 138 | − detail also carries `hardware_fit_assumptions: string[]`. `/hardware/fit` adds `estimated: true`, `counts`, optional `openness=` filter. | ||||
| 139 | −- **`/prices/index`** series rows add `max_input` and `offers`; response adds `days` and `note`. `/prices/history` requires `model` or `provider`. | ||||
| 140 | −- **`/benchmarks/{slug}/results`** and `/history` include the `benchmark` summary; result rows include `valid_to`. | ||||
| 141 | −- **`/diff`** adds `scope` (resolved description) and extra `counts` (`events`, `price_rows_opened/closed`, `claims_superseded`, `entities_at_a/b`). | ||||
| 142 | −- **`/sources`** items add `snapshots`, `claims`, `base_url`, `robots_policy`, `priority`, `notes`; response adds `tiers` legend. | ||||
| 143 | −- **`/methodology`** adds `quality_version`, `expected_fields`, `principles`; `event_types` are live counts from `change_events`. | ||||
| 144 | −- **`/trending`** accepts `type=`; `/sitemap` `limit` ≤ 5000 (the one listing not capped at 200). | ||||
| 145 | −- **Rate limits** are in-process sliding windows (search 60/min, views 1/s, admin 240/min per IP) — fine for the single uvicorn worker behind Next.js. | ||||
| 146 | −- **Admin**: `GET /admin/review?status=all` lists every status; approving a `conflict` with `resolution.keep_claim_id` promotes that claim; | ||||
| 147 | − `PATCH /admin/connectors/{name}` also resets `health`/`circuit_open_until` when toggling; `/admin/connectors` flags `run_now_pending` and | ||||
| 148 | − connectors present in code but not in the table; `/admin/snapshots/{id}` never returns `raw_path`/`text_path` (`has_raw`/`has_text` booleans instead). | ||||
| 149 | −- **Migration `0002_api_indexes`** adds partial indexes for the read paths (type + release_date/family/openness/updated/first_seen, importance-ordered | ||||
| 150 | − events, current leaderboards/prices, page views, review kinds). | ||||
modified
src/aiatlas/api/common.py
+151 −32
@@ -5,6 +5,8 @@ from __future__ import annotations | |||||
| 5 | 5 | ||||
| 6 | import hmac | 6 | import hmac | ||
| 7 | import inspect | 7 | import inspect | ||
| 8 | +import logging | ||||
| 9 | +import os | ||||
| 8 | import time | 10 | import time | ||
| 9 | from collections import defaultdict, deque | 11 | from collections import defaultdict, deque | ||
| 10 | from collections.abc import Awaitable, Callable | 12 | from collections.abc import Awaitable, Callable | ||
@@ -20,9 +22,11 @@ from fastapi.responses import JSONResponse | |||||
| 20 | from sqlalchemy.ext.asyncio import AsyncConnection | 22 | from sqlalchemy.ext.asyncio import AsyncConnection | ||
| 21 | 23 | ||||
| 22 | from aiatlas.config import settings | 24 | from aiatlas.config import settings | ||
| 23 | −from aiatlas.db import fetch_one | 25 | +from aiatlas.db import execute, fetch_one, jsonb, transaction | ||
| 24 | from aiatlas.services import cache | 26 | from aiatlas.services import cache | ||
| 25 | 27 | ||||
| 28 | +log = logging.getLogger("aiatlas.api") | ||||
| 29 | + | ||||
| 26 | # ------------------------------------------------------------------------------------------------------------------ errors & JSON | 30 | # ------------------------------------------------------------------------------------------------------------------ errors & JSON | ||
| 27 | 31 | ||||
| 28 | 32 | ||||
@@ -167,48 +171,84 @@ def cached(ttl_s: int) -> Callable[[Callable[..., Awaitable[Any]]], Callable[... | |||||
| 167 | return deco | 171 | return deco | ||
| 168 | 172 | ||||
| 169 | 173 | ||||
| 170 | −# ------------------------------------------------------------------------------------------------------------------ admin auth | ||||
| 171 | − | ||||
| 172 | − | ||||
| 173 | −def require_admin(request: Request) -> None: | ||||
| 174 | − expected = settings.admin_token | ||||
| 175 | − if not expected: | ||||
| 176 | − raise ApiError(503, "admin API disabled: AIA_ADMIN_TOKEN is not configured") | ||||
| 177 | − token = request.headers.get("x-aia-admin-token") or "" | ||||
| 178 | − if not hmac.compare_digest(token, expected): | ||||
| 179 | − raise ApiError(401, "invalid or missing x-aia-admin-token") | ||||
| 180 | − | ||||
| 181 | − | ||||
| 182 | # ------------------------------------------------------------------------------------------------------------------ rate limiting | 174 | # ------------------------------------------------------------------------------------------------------------------ rate limiting | ||
| 183 | 175 | ||||
| 184 | −_LIMITS: dict[str, tuple[int, float]] = {"search": (60, 60.0), "views": (1, 1.0), "admin": (240, 60.0), "default": (600, 60.0)} | 176 | +_LIMITS: dict[str, tuple[int, float]] = {"search": (60, 60.0), "views": (1, 1.0), "admin": (240, 60.0), "admin-auth-failed": (10, 60.0), "default": (600, 60.0)} | ||
| 185 | _hits: dict[tuple[str, str], deque[float]] = defaultdict(deque) | 177 | _hits: dict[tuple[str, str], deque[float]] = defaultdict(deque) | ||
| 186 | 178 | ||||
| 187 | 179 | ||||
| 180 | +def trust_proxy() -> bool: | ||||
| 181 | + """`x-forwarded-for` is only honoured behind a trusted reverse proxy (`settings.trust_proxy` or env `AIA_TRUST_PROXY=1`).""" | ||||
| 182 | + v = getattr(settings, "trust_proxy", None) | ||||
| 183 | + if v is None: | ||||
| 184 | + v = os.environ.get("AIA_TRUST_PROXY", "") | ||||
| 185 | + return str(v).strip().lower() in ("1", "true", "yes", "on") | ||||
| 186 | + | ||||
| 187 | + | ||||
| 188 | def client_ip(request: Request) -> str: | 188 | def client_ip(request: Request) -> str: | ||
| 189 | − fwd = request.headers.get("x-forwarded-for") | 189 | + if trust_proxy(): | ||
| 190 | − if fwd: | 190 | + fwd = request.headers.get("x-forwarded-for") | ||
| 191 | − return fwd.split(",")[0].strip() | 191 | + if fwd: | ||
| 192 | + return fwd.split(",")[0].strip() | ||||
| 192 | return request.client.host if request.client else "unknown" | 193 | return request.client.host if request.client else "unknown" | ||
| 193 | 194 | ||||
| 194 | 195 | ||||
| 196 | +def _bucket_hit(ip: str, bucket: str, *, record: bool = True) -> bool: | ||||
| 197 | + """Sliding-window counter. Returns True when the call is allowed (and records it), False when the bucket is full.""" | ||||
| 198 | + n, window = _LIMITS.get(bucket, _LIMITS["default"]) | ||||
| 199 | + now = time.monotonic() | ||||
| 200 | + q = _hits[(ip, bucket)] | ||||
| 201 | + while q and q[0] <= now - window: | ||||
| 202 | + q.popleft() | ||||
| 203 | + if len(q) >= n: | ||||
| 204 | + return False | ||||
| 205 | + if record: | ||||
| 206 | + q.append(now) | ||||
| 207 | + if len(_hits) > 20000: | ||||
| 208 | + for k in [k for k, v in _hits.items() if not v or v[-1] < now - 120]: | ||||
| 209 | + _hits.pop(k, None) | ||||
| 210 | + return True | ||||
| 211 | + | ||||
| 212 | + | ||||
| 195 | def rate_limit(bucket: str) -> Callable[[Request], None]: | 213 | def rate_limit(bucket: str) -> Callable[[Request], None]: | ||
| 196 | def dep(request: Request) -> None: | 214 | def dep(request: Request) -> None: | ||
| 197 | n, window = _LIMITS.get(bucket, _LIMITS["default"]) | 215 | n, window = _LIMITS.get(bucket, _LIMITS["default"]) | ||
| 198 | − now = time.monotonic() | 216 | + if not _bucket_hit(client_ip(request), bucket): | ||
| 199 | − q = _hits[(client_ip(request), bucket)] | ||||
| 200 | − while q and q[0] <= now - window: | ||||
| 201 | − q.popleft() | ||||
| 202 | − if len(q) >= n: | ||||
| 203 | raise ApiError(429, f"rate limit exceeded for {bucket}: {n} per {int(window)} s") | 217 | raise ApiError(429, f"rate limit exceeded for {bucket}: {n} per {int(window)} s") | ||
| 204 | − q.append(now) | ||||
| 205 | − if len(_hits) > 20000: | ||||
| 206 | − for k in [k for k, v in _hits.items() if not v or v[-1] < now - 120]: | ||||
| 207 | − _hits.pop(k, None) | ||||
| 208 | 218 | ||||
| 209 | return dep | 219 | return dep | ||
| 210 | 220 | ||||
| 211 | 221 | ||||
| 222 | +# ------------------------------------------------------------------------------------------------------------------ admin auth | ||||
| 223 | + | ||||
| 224 | + | ||||
| 225 | +def require_admin(request: Request) -> None: | ||||
| 226 | + """Token check. Failed attempts are counted per IP (10/min → 429) so the token cannot be brute-forced; must run AFTER `rate_limit('admin')`.""" | ||||
| 227 | + ip = client_ip(request) | ||||
| 228 | + if not _bucket_hit(ip, "admin-auth-failed", record=False): | ||||
| 229 | + raise ApiError(429, "too many failed admin authentications; retry in a minute") | ||||
| 230 | + expected = settings.admin_token | ||||
| 231 | + if not expected: | ||||
| 232 | + raise ApiError(503, "admin API disabled: AIA_ADMIN_TOKEN is not configured") | ||||
| 233 | + token = request.headers.get("x-aia-admin-token") or "" | ||||
| 234 | + if not hmac.compare_digest(token, expected): | ||||
| 235 | + _bucket_hit(ip, "admin-auth-failed") | ||||
| 236 | + raise ApiError(401, "invalid or missing x-aia-admin-token") | ||||
| 237 | + | ||||
| 238 | + | ||||
| 239 | +ADMIN_DEPENDENCIES = [Depends(rate_limit("admin")), Depends(require_admin)] # order matters: limit first, then auth | ||||
| 240 | + | ||||
| 241 | + | ||||
| 242 | +async def audit(action: str, target: str | None = None, payload: dict[str, Any] | None = None, ip: str | None = None, *, actor: str = "admin") -> None: | ||||
| 243 | + """One row in `admin_audit_log` per admin action (best effort — never breaks the request).""" | ||||
| 244 | + try: | ||||
| 245 | + async with transaction() as conn: | ||||
| 246 | + await execute(conn, "insert into admin_audit_log (actor, action, target, payload, ip) values (:a, :ac, :t, cast(:p as jsonb), :ip)", | ||||
| 247 | + a=actor, ac=action, t=target, p=jsonb(payload or {}), ip=ip) | ||||
| 248 | + except Exception: # noqa: BLE001 | ||||
| 249 | + log.warning("audit log write failed", extra={"action": action, "target": target}) | ||||
| 250 | + | ||||
| 251 | + | ||||
| 212 | # ------------------------------------------------------------------------------------------------------------------ entity serialisers | 252 | # ------------------------------------------------------------------------------------------------------------------ entity serialisers | ||
| 213 | 253 | ||||
| 214 | ENTITY_FIELDS = ("id", "entity_type", "canonical_name", "slug", "description", "status", "organization_id", "attributes", "quality", "counts", | 254 | ENTITY_FIELDS = ("id", "entity_type", "canonical_name", "slug", "description", "status", "organization_id", "attributes", "quality", "counts", | ||
@@ -249,7 +289,36 @@ COMPANY_TYPES = ("company", "organization", "lab", "university") | |||||
| 249 | TYPE_LABELS = {"model": "Models", "company": "Companies", "organization": "Organizations", "lab": "Labs", "university": "Universities", | 289 | TYPE_LABELS = {"model": "Models", "company": "Companies", "organization": "Organizations", "lab": "Labs", "university": "Universities", | ||
| 250 | "provider": "Providers", "paper": "Papers", "benchmark": "Benchmarks", "hardware": "Hardware", "framework": "Frameworks", | 290 | "provider": "Providers", "paper": "Papers", "benchmark": "Benchmarks", "hardware": "Hardware", "framework": "Frameworks", | ||
| 251 | "dataset": "Datasets", "tool": "Tools", "repository": "Repositories", "regulation": "Regulation", "incident": "Incidents", | 291 | "dataset": "Datasets", "tool": "Tools", "repository": "Repositories", "regulation": "Regulation", "incident": "Incidents", | ||
| 252 | − "researcher": "Researchers", "agent": "Agents", "product": "Products", "runtime": "Runtimes", "conference": "Conferences"} | 292 | + "researcher": "Researchers", "agent": "Agents", "product": "Products", "runtime": "Runtimes", "conference": "Conferences", | ||
| 293 | + "artifact": "Artifacts", "model_family": "Model families", "license": "Licenses"} | ||||
| 294 | + | ||||
| 295 | +# Canonical model universe (API 1.1): model releases only — artifacts (checkpoints, quantisations, conversions) are `entity_type = 'artifact'`, | ||||
| 296 | +# folded evaluation variants carry `merged_into`. `include=artifacts` on /models restores the pre-1.1 universe. | ||||
| 297 | +MODEL_UNIVERSE = "e.entity_type = 'model' and e.merged_into is null" | ||||
| 298 | +MODEL_OR_ARTIFACT_UNIVERSE = "e.entity_type in ('model', 'artifact') and e.merged_into is null" | ||||
| 299 | +OPEN_CATEGORIES = ("open-weights", "open-source") # "open-*" | ||||
| 300 | +DOWNLOADABLE_CATEGORIES = ("open-weights", "open-source", "restricted-weights", "restricted") # weights can be downloaded (terms may restrict) | ||||
| 301 | +ARTIFACT_KINDS = ("checkpoint", "quantization", "conversion", "packaging") | ||||
| 302 | + | ||||
| 303 | + | ||||
| 304 | +def openness_values(raw: str | None) -> list[str]: | ||||
| 305 | + """Expand the `openness=` filter vocabulary: `open` → open-weights + open-source (+ legacy `open`); `restricted` ↔ `restricted-weights`.""" | ||||
| 306 | + vals = [v.strip() for v in (raw or "").split(",") if v.strip()] | ||||
| 307 | + out: list[str] = [] | ||||
| 308 | + for v in vals: | ||||
| 309 | + if v == "open": | ||||
| 310 | + out += [*OPEN_CATEGORIES, "open"] | ||||
| 311 | + elif v in ("restricted", "restricted-weights"): | ||||
| 312 | + out += ["restricted", "restricted-weights"] | ||||
| 313 | + elif v == "downloadable": | ||||
| 314 | + out += list(DOWNLOADABLE_CATEGORIES) | ||||
| 315 | + else: | ||||
| 316 | + out.append(v) | ||||
| 317 | + return list(dict.fromkeys(out)) | ||||
| 318 | + | ||||
| 319 | + | ||||
| 320 | +def is_open(openness: Any) -> bool: | ||||
| 321 | + return str(openness or "") in OPEN_CATEGORIES | ||||
| 253 | 322 | ||||
| 254 | 323 | ||||
| 255 | def summary_attributes(entity_type: str, attrs: dict[str, Any] | None) -> dict[str, Any]: | 324 | def summary_attributes(entity_type: str, attrs: dict[str, Any] | None) -> dict[str, Any]: | ||
@@ -371,6 +440,24 @@ PRICE_COLS = ("p.id, p.provider_model_id, p.input_per_mtok, p.output_per_mtok, p | |||||
| 371 | "p.valid_to, p.source_url, p.tier, " + entity_cols("m", "m_") + ", " + entity_cols("pv", "p_")) | 440 | "p.valid_to, p.source_url, p.tier, " + entity_cols("m", "m_") + ", " + entity_cols("pv", "p_")) | ||
| 372 | PRICE_FROM = "prices p " + entity_join("m", "p.model_id") + " " + entity_join("pv", "p.provider_id") | 441 | PRICE_FROM = "prices p " + entity_join("m", "p.model_id") + " " + entity_join("pv", "p.provider_id") | ||
| 373 | 442 | ||||
| 443 | +_NATIVE_UNIT_SUFFIXES = ("_per_mtok", "_per_1k_requests", "_per_mtok_hour", "_per_image", "_per_request", "_per_second", "_per_minute", "_per_hour", "_per_char") | ||||
| 444 | + | ||||
| 445 | + | ||||
| 446 | +def deployment_row(row: dict[str, Any]) -> dict[str, Any]: | ||||
| 447 | + """`Deployment` (API 1.1): one model × provider × provider_model_id offer. Prices in USD per 1M tokens unless the key says otherwise; | ||||
| 448 | + provider-specific priced features (`flex_input_per_mtok`, `search_grounding_per_1k_requests`…) are kept verbatim under `native_units`.""" | ||||
| 449 | + feats = dict(row.get("features") or {}) | ||||
| 450 | + native = {k: v for k, v in feats.items() if any(k.endswith(s) for s in _NATIVE_UNIT_SUFFIXES) or k == "per_request"} | ||||
| 451 | + other = {k: v for k, v in feats.items() if k not in native} | ||||
| 452 | + return {"id": row["id"], "model": entity_summary(row, "m_"), "provider": entity_summary(row, "p_"), "provider_model_id": row.get("provider_model_id"), | ||||
| 453 | + "context_length": row.get("context_length"), "max_output_tokens": row.get("max_output_tokens"), | ||||
| 454 | + "prices": {"input": row.get("input_per_mtok"), "cached_input": row.get("cached_input_per_mtok"), "cache_write": row.get("cache_write_per_mtok"), | ||||
| 455 | + "output": row.get("output_per_mtok"), "batch_input": row.get("batch_input_per_mtok"), "batch_output": row.get("batch_output_per_mtok"), | ||||
| 456 | + "per_image": row.get("per_image"), "per_request": row.get("per_request"), "currency": row.get("currency") or "USD", "unit": "USD per 1M tokens", | ||||
| 457 | + "native_units": native}, | ||||
| 458 | + "features": other, "status": "active" if row.get("valid_to") is None else "delisted", "observed_at": row.get("observed_at"), | ||||
| 459 | + "valid_from": row.get("valid_from"), "valid_to": row.get("valid_to"), "source_url": row.get("source_url"), "tier": row.get("tier")} | ||||
| 460 | + | ||||
| 374 | 461 | ||||
| 375 | def result_row(row: dict[str, Any]) -> dict[str, Any]: | 462 | def result_row(row: dict[str, Any]) -> dict[str, Any]: | ||
| 376 | return {"id": row["id"], "model": entity_summary(row, "m_"), "benchmark": entity_summary(row, "b_"), "score": row.get("score"), "metric": row.get("metric"), | 463 | return {"id": row["id"], "model": entity_summary(row, "m_"), "benchmark": entity_summary(row, "b_"), "score": row.get("score"), "metric": row.get("metric"), | ||
@@ -400,24 +487,44 @@ CLAIM_FROM = "claims c left join sources s on s.id = c.source_id" | |||||
| 400 | # ------------------------------------------------------------------------------------------------------------------ entity resolution | 487 | # ------------------------------------------------------------------------------------------------------------------ entity resolution | ||
| 401 | 488 | ||||
| 402 | 489 | ||||
| 403 | −async def resolve_entity(conn: AsyncConnection, slug_or_id: str, types: tuple[str, ...] | None = None) -> dict[str, Any]: | 490 | +ENTITY_EXTRA_COLS = "e.provenance, e.merged_into, e.family_id, e.canonical_id, e.artifact_kind, e.identity_confidence" | ||
| 404 | − """Slug first, then id; follows `merged_into`; 404 when absent or when the type does not match the mounted alias.""" | 491 | + | ||
| 492 | + | ||||
| 493 | +async def resolve_entity(conn: AsyncConnection, slug_or_id: str, types: tuple[str, ...] | None = None, *, aliases: bool = False) -> dict[str, Any]: | ||||
| 494 | + """Slug first, then id (then alias when `aliases=True`); follows `merged_into` and records the hop in `redirected_from` | ||||
| 495 | + (`{slug, id}` of the row the caller asked for) so the web layer can 301; 404 when absent or when the type does not match the mounted alias.""" | ||||
| 405 | key = slug_or_id.strip() | 496 | key = slug_or_id.strip() | ||
| 406 | if not key or len(key) > 200: | 497 | if not key or len(key) > 200: | ||
| 407 | raise ApiError(404, "entity not found") | 498 | raise ApiError(404, "entity not found") | ||
| 408 | − row = await fetch_one(conn, f"select {ENTITY_COLS}, e.provenance, e.merged_into from {ENTITY_FROM} where e.slug = :k or e.id = :k " | 499 | + sel = f"select {ENTITY_COLS}, {ENTITY_EXTRA_COLS} from {ENTITY_FROM}" | ||
| 409 | − f"order by case when e.slug = :k then 0 else 1 end limit 1", k=key) | 500 | + row = await fetch_one(conn, f"{sel} where e.slug = :k or e.id = :k order by case when e.slug = :k then 0 else 1 end limit 1", k=key) | ||
| 501 | + if not row and aliases: | ||||
| 502 | + row = await fetch_one(conn, f"{sel} where e.id = (select a.entity_id from entity_aliases a join entities x on x.id = a.entity_id " | ||||
| 503 | + f"where (a.alias_norm = :n or lower(a.alias) = lower(:k)) {'and x.entity_type = any(cast(:types as text[]))' if types else ''} " | ||||
| 504 | + f"order by x.merged_into is not null, x.updated_at desc limit 1)", k=key, n=_alias_norm(key), types=list(types or ())) | ||||
| 505 | + asked = row | ||||
| 410 | hops = 0 | 506 | hops = 0 | ||
| 411 | while row and row.get("merged_into") and hops < 5: | 507 | while row and row.get("merged_into") and hops < 5: | ||
| 412 | − row = await fetch_one(conn, f"select {ENTITY_COLS}, e.provenance, e.merged_into from {ENTITY_FROM} where e.id = :k", k=row["merged_into"]) | 508 | + row = await fetch_one(conn, f"{sel} where e.id = :k", k=row["merged_into"]) | ||
| 413 | hops += 1 | 509 | hops += 1 | ||
| 414 | if not row: | 510 | if not row: | ||
| 415 | raise ApiError(404, "entity not found") | 511 | raise ApiError(404, "entity not found") | ||
| 416 | if types and row["entity_type"] not in types: | 512 | if types and row["entity_type"] not in types: | ||
| 417 | raise ApiError(404, f"entity {key!r} is a {row['entity_type']}, not one of {', '.join(types)}") | 513 | raise ApiError(404, f"entity {key!r} is a {row['entity_type']}, not one of {', '.join(types)}") | ||
| 514 | + if asked is not None and asked["id"] != row["id"]: | ||||
| 515 | + row["redirected_from"] = {"slug": asked["slug"], "id": asked["id"], "entity_type": asked["entity_type"]} | ||||
| 418 | return row | 516 | return row | ||
| 419 | 517 | ||||
| 420 | 518 | ||||
| 519 | +def _alias_norm(s: str) -> str: | ||||
| 520 | + try: | ||||
| 521 | + from aiatlas.ids import normalize_alias | ||||
| 522 | + | ||||
| 523 | + return normalize_alias(s) | ||||
| 524 | + except Exception: # noqa: BLE001 | ||||
| 525 | + return "".join(ch for ch in s.lower() if ch.isalnum()) | ||||
| 526 | + | ||||
| 527 | + | ||||
| 421 | async def resolve_id(conn: AsyncConnection, slug_or_id: str | None, types: tuple[str, ...] | None = None) -> str | None: | 528 | async def resolve_id(conn: AsyncConnection, slug_or_id: str | None, types: tuple[str, ...] | None = None) -> str | None: | ||
| 422 | if not slug_or_id: | 529 | if not slug_or_id: | ||
| 423 | return None | 530 | return None | ||
@@ -425,15 +532,22 @@ async def resolve_id(conn: AsyncConnection, slug_or_id: str | None, types: tuple | |||||
| 425 | 532 | ||||
| 426 | 533 | ||||
| 427 | __all__ = [ | 534 | __all__ = [ | ||
| 535 | + "ADMIN_DEPENDENCIES", | ||||
| 536 | + "ARTIFACT_KINDS", | ||||
| 428 | "CLAIM_COLS", | 537 | "CLAIM_COLS", | ||
| 429 | "CLAIM_FROM", | 538 | "CLAIM_FROM", | ||
| 430 | "COMPANY_TYPES", | 539 | "COMPANY_TYPES", | ||
| 540 | + "DOWNLOADABLE_CATEGORIES", | ||||
| 431 | "ENTITY_COLS", | 541 | "ENTITY_COLS", | ||
| 542 | + "ENTITY_EXTRA_COLS", | ||||
| 432 | "ENTITY_FROM", | 543 | "ENTITY_FROM", | ||
| 433 | "EVENT_COLS", | 544 | "EVENT_COLS", | ||
| 434 | "EVENT_FROM", | 545 | "EVENT_FROM", | ||
| 435 | "EVENT_TYPE_LABELS", | 546 | "EVENT_TYPE_LABELS", | ||
| 436 | "MAIN_ENTITY_TYPES", | 547 | "MAIN_ENTITY_TYPES", | ||
| 548 | + "MODEL_OR_ARTIFACT_UNIVERSE", | ||||
| 549 | + "MODEL_UNIVERSE", | ||||
| 550 | + "OPEN_CATEGORIES", | ||||
| 437 | "PAGINATION", | 551 | "PAGINATION", | ||
| 438 | "PRICE_COLS", | 552 | "PRICE_COLS", | ||
| 439 | "PRICE_FROM", | 553 | "PRICE_FROM", | ||
@@ -446,6 +560,7 @@ __all__ = [ | |||||
| 446 | "AtlasJSONResponse", | 560 | "AtlasJSONResponse", | ||
| 447 | "Pagination", | 561 | "Pagination", | ||
| 448 | "attr_num", | 562 | "attr_num", | ||
| 563 | + "audit", | ||||
| 449 | "cache_key", | 564 | "cache_key", | ||
| 450 | "cached", | 565 | "cached", | ||
| 451 | "change_event", | 566 | "change_event", | ||
@@ -453,6 +568,7 @@ __all__ = [ | |||||
| 453 | "client_ip", | 568 | "client_ip", | ||
| 454 | "csv", | 569 | "csv", | ||
| 455 | "day_bounds", | 570 | "day_bounds", | ||
| 571 | + "deployment_row", | ||||
| 456 | "dumps", | 572 | "dumps", | ||
| 457 | "enrich_provenance", | 573 | "enrich_provenance", | ||
| 458 | "entity_cols", | 574 | "entity_cols", | ||
@@ -461,9 +577,11 @@ __all__ = [ | |||||
| 461 | "event_type_importance", | 577 | "event_type_importance", | ||
| 462 | "event_type_label", | 578 | "event_type_label", | ||
| 463 | "flip_order", | 579 | "flip_order", | ||
| 580 | + "is_open", | ||||
| 464 | "normalize", | 581 | "normalize", | ||
| 465 | "normalize_status", | 582 | "normalize_status", | ||
| 466 | "num_expr", | 583 | "num_expr", | ||
| 584 | + "openness_values", | ||||
| 467 | "org_of", | 585 | "org_of", | ||
| 468 | "page", | 586 | "page", | ||
| 469 | "parse_date", | 587 | "parse_date", | ||
@@ -474,4 +592,5 @@ __all__ = [ | |||||
| 474 | "resolve_entity", | 592 | "resolve_entity", | ||
| 475 | "resolve_id", | 593 | "resolve_id", | ||
| 476 | "result_row", | 594 | "result_row", | ||
| 595 | + "trust_proxy", | ||||
| 477 | ] | 596 | ] | ||
modified
src/aiatlas/api/detail.py
+274 −40
@@ -1,12 +1,20 @@ | |||||
| 1 | −"""`EntityDetail` builder (docs/API.md): shared by `/entities/{slug}` and the type-scoped aliases.""" | 1 | +"""`EntityDetail` builder (docs/API.md): shared by `/entities/{slug}` and the type-scoped aliases. | ||
| 2 | + | ||||
| 3 | +API 1.1: blocks run in at most FOUR concurrent groups (one pooled connection each, sequential inside a group) instead of one connection per | ||||
| 4 | +block; model details gain `family`, `artifacts`, `deployments`, `identity`, `licence`, `openness`, `version_history` and grouped `benchmarks`; | ||||
| 5 | +artifact details carry `canonical` + `artifact_kind`; a resolved `merged_into` hop is reported as `redirected_from`.""" | ||||
| 2 | from __future__ import annotations | 6 | from __future__ import annotations | ||
| 3 | 7 | ||||
| 4 | import asyncio | 8 | import asyncio | ||
| 9 | +from collections import defaultdict | ||||
| 5 | from typing import Any | 10 | from typing import Any | ||
| 6 | 11 | ||||
| 7 | from sqlalchemy.ext.asyncio import AsyncConnection | 12 | from sqlalchemy.ext.asyncio import AsyncConnection | ||
| 8 | 13 | ||||
| 9 | from aiatlas.api.common import ( | 14 | from aiatlas.api.common import ( | ||
| 15 | + ARTIFACT_KINDS, | ||||
| 16 | + CLAIM_COLS, | ||||
| 17 | + CLAIM_FROM, | ||||
| 10 | COMPANY_TYPES, | 18 | COMPANY_TYPES, | ||
| 11 | ENTITY_COLS, | 19 | ENTITY_COLS, | ||
| 12 | ENTITY_FROM, | 20 | ENTITY_FROM, | ||
@@ -18,16 +26,23 @@ from aiatlas.api.common import ( | |||||
| 18 | RESULT_FROM, | 26 | RESULT_FROM, | ||
| 19 | RESULT_ORDER, | 27 | RESULT_ORDER, | ||
| 20 | change_event, | 28 | change_event, | ||
| 29 | + deployment_row, | ||||
| 21 | enrich_provenance, | 30 | enrich_provenance, | ||
| 22 | entity_summary, | 31 | entity_summary, | ||
| 23 | price_row, | 32 | price_row, | ||
| 24 | result_row, | 33 | result_row, | ||
| 25 | ) | 34 | ) | ||
| 26 | −from aiatlas.db import connection, fetch_all | 35 | +from aiatlas.db import connection, fetch_all, fetch_one | ||
| 36 | +from aiatlas.ontology.benchmarks import TRUST_LABELS | ||||
| 37 | +from aiatlas.ontology.licenses import LICENSES, normalize_license | ||||
| 38 | +from aiatlas.ontology.openness import OPENNESS_DEFINITIONS, OPENNESS_LABELS, normalize_openness, openness_dimensions | ||||
| 27 | from aiatlas.services import hardware_fit as hf | 39 | from aiatlas.services import hardware_fit as hf | ||
| 40 | +from aiatlas.services.frontier import config_summary, enrich, group_label | ||||
| 28 | 41 | ||||
| 29 | LINEAGE_PREDICATES = ("derived_from", "fine_tuned_from", "distilled_from", "merged_from", "quantized_from") | 42 | LINEAGE_PREDICATES = ("derived_from", "fine_tuned_from", "distilled_from", "merged_from", "quantized_from") | ||
| 30 | RELATION_GROUP_LIMIT = 24 | 43 | RELATION_GROUP_LIMIT = 24 | ||
| 44 | +VERSIONED_PROPERTIES = ("context_length", "max_output_tokens", "status", "knowledge_cutoff", "license", "openness", "parameter_count") | ||||
| 45 | +MAX_CONCURRENT_GROUPS = 4 | ||||
| 31 | 46 | ||||
| 32 | 47 | ||||
| 33 | async def relations_grouped(conn: AsyncConnection, entity_id: str) -> list[dict[str, Any]]: | 48 | async def relations_grouped(conn: AsyncConnection, entity_id: str) -> list[dict[str, Any]]: | ||
@@ -68,17 +83,30 @@ def _scope_sql(is_org: bool) -> str: | |||||
| 68 | return "ev.entity_id = :id" | 83 | return "ev.entity_id = :id" | ||
| 69 | 84 | ||||
| 70 | 85 | ||||
| 86 | +def event_date_col(date_field: str) -> str: | ||||
| 87 | + return "ev.observed_at" if date_field == "observed" else "ev.occurred_at" | ||||
| 88 | + | ||||
| 89 | + | ||||
| 71 | async def timeline_of(conn: AsyncConnection, entity_id: str, entity_type: str, *, limit: int = 30, before: Any = None, | 90 | async def timeline_of(conn: AsyncConnection, entity_id: str, entity_type: str, *, limit: int = 30, before: Any = None, | ||
| 72 | − include_documents: bool = False) -> list[dict[str, Any]]: | 91 | + include_documents: bool = False, include_backfill: bool = False, date_field: str = "occurred") -> list[dict[str, Any]]: | ||
| 92 | + col = event_date_col(date_field) | ||||
| 73 | where = [_scope_sql(entity_type in COMPANY_TYPES)] | 93 | where = [_scope_sql(entity_type in COMPANY_TYPES)] | ||
| 74 | params: dict[str, Any] = {"id": entity_id, "lim": limit} | 94 | params: dict[str, Any] = {"id": entity_id, "lim": limit} | ||
| 75 | if before is not None: | 95 | if before is not None: | ||
| 76 | − where.append("ev.observed_at < :before") | 96 | + where.append(f"{col} < :before") | ||
| 77 | params["before"] = before | 97 | params["before"] = before | ||
| 78 | if not include_documents: | 98 | if not include_documents: | ||
| 79 | where.append("ev.event_type <> 'DOCUMENT_CHANGED'") | 99 | where.append("ev.event_type <> 'DOCUMENT_CHANGED'") | ||
| 80 | − rows = await fetch_all(conn, f"select {EVENT_COLS} from {EVENT_FROM} where {' and '.join(where)} order by ev.observed_at desc, ev.id desc limit :lim", **params) | 100 | + if not include_backfill: | ||
| 81 | − return [change_event(r) for r in rows] | 101 | + where.append("ev.is_backfill = false") | ||
| 102 | + rows = await fetch_all(conn, f"select {EVENT_COLS}, ev.occurred_at, ev.is_backfill, ev.group_key from {EVENT_FROM} where {' and '.join(where)} " | ||||
| 103 | + f"order by {col} desc, ev.id desc limit :lim", **params) | ||||
| 104 | + out = [] | ||||
| 105 | + for r in rows: | ||||
| 106 | + ev = change_event(r) | ||||
| 107 | + ev["occurred_at"], ev["is_backfill"], ev["group_key"] = r.get("occurred_at"), r.get("is_backfill"), r.get("group_key") | ||||
| 108 | + out.append(ev) | ||||
| 109 | + return out | ||||
| 82 | 110 | ||||
| 83 | 111 | ||||
| 84 | async def prices_of_model(conn: AsyncConnection, model_id: str, *, current_only: bool) -> list[dict[str, Any]]: | 112 | async def prices_of_model(conn: AsyncConnection, model_id: str, *, current_only: bool) -> list[dict[str, Any]]: | ||
@@ -88,6 +116,11 @@ async def prices_of_model(conn: AsyncConnection, model_id: str, *, current_only: | |||||
| 88 | return [price_row(r) for r in rows] | 116 | return [price_row(r) for r in rows] | ||
| 89 | 117 | ||||
| 90 | 118 | ||||
| 119 | +async def deployments_of_model(conn: AsyncConnection, model_id: str) -> list[dict[str, Any]]: | ||||
| 120 | + rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.model_id = :id and p.valid_to is null order by p.output_per_mtok nulls last, pv.canonical_name limit 200", id=model_id) | ||||
| 121 | + return [deployment_row(r) for r in rows] | ||||
| 122 | + | ||||
| 123 | + | ||||
| 91 | async def prices_of_provider(conn: AsyncConnection, provider_id: str) -> list[dict[str, Any]]: | 124 | async def prices_of_provider(conn: AsyncConnection, provider_id: str) -> list[dict[str, Any]]: | ||
| 92 | rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.provider_id = :id and p.valid_to is null order by m.canonical_name limit 500", id=provider_id) | 125 | rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.provider_id = :id and p.valid_to is null order by m.canonical_name limit 500", id=provider_id) | ||
| 93 | return [price_row(r) for r in rows] | 126 | return [price_row(r) for r in rows] | ||
@@ -98,7 +131,43 @@ async def results_of_model(conn: AsyncConnection, model_id: str) -> list[dict[st | |||||
| 98 | return [result_row(r) for r in rows] | 131 | return [result_row(r) for r in rows] | ||
| 99 | 132 | ||||
| 100 | 133 | ||||
| 101 | −async def leaderboard(conn: AsyncConnection, benchmark_id: str, *, limit: int = 100, offset: int = 0, config: str | None = None, history: bool = False) -> list[dict[str, Any]]: | 134 | +async def benchmarks_of_model(conn: AsyncConnection, model_id: str) -> dict[str, Any]: | ||
| 135 | + """Current results grouped by benchmark → metric → (config group): best row, n_rows, trust level, comparability group label.""" | ||||
| 136 | + rows = await fetch_all(conn, """ | ||||
| 137 | + select r.id, r.model_id, r.benchmark_id, r.score, r.metric, r.unit, r.higher_is_better, r.config, r.evaluated_at, r.observed_at, r.source_url, r.tier, r.confidence, | ||||
| 138 | + r.config_key, r.trust_level, r.extractor, s.key as source_key, b.slug as benchmark_slug, b.canonical_name as benchmark_name, b.attributes->>'category' as category | ||||
| 139 | + from benchmark_results r join entities b on b.id = r.benchmark_id left join sources s on s.id = r.source_id | ||||
| 140 | + where r.model_id = :id and r.valid_to is null and r.is_current order by b.canonical_name, r.observed_at desc limit 500""", id=model_id) | ||||
| 141 | + for r in rows: | ||||
| 142 | + r["model_name"] = None | ||||
| 143 | + enrich(r) | ||||
| 144 | + by_bench: dict[str, dict[str, Any]] = {} | ||||
| 145 | + for r in rows: | ||||
| 146 | + b = by_bench.setdefault(r["benchmark_id"], {"benchmark": {"id": r["benchmark_id"], "slug": r["benchmark_slug"], "name": r["benchmark_name"], "category": r["category"]}, "metrics": {}}) | ||||
| 147 | + m = b["metrics"].setdefault(r["metric_canonical"], {"metric": r["metric_canonical"], "groups": {}}) | ||||
| 148 | + g = m["groups"].setdefault(r["config_key"], {"config_key": r["config_key"], "label": group_label(r["metric_canonical"], r.get("config")), "rows": []}) | ||||
| 149 | + g["rows"].append(r) | ||||
| 150 | + items = [] | ||||
| 151 | + for b in by_bench.values(): | ||||
| 152 | + metrics = [] | ||||
| 153 | + for m in b["metrics"].values(): | ||||
| 154 | + groups = [] | ||||
| 155 | + for g in m["groups"].values(): | ||||
| 156 | + hib = all(x.get("higher_is_better", True) for x in g["rows"]) | ||||
| 157 | + best = max(g["rows"], key=lambda x: x["score"]) if hib else min(g["rows"], key=lambda x: x["score"]) | ||||
| 158 | + groups.append({"config_key": g["config_key"], "comparability_group": g["label"], "n_rows": len(g["rows"]), "higher_is_better": hib, | ||||
| 159 | + "best": {"score": best["score"], "unit": best.get("unit"), "trust_level": best["trust_level"], "trust_label": TRUST_LABELS.get(best["trust_level"], best["trust_level"]), | ||||
| 160 | + "config": config_summary(best.get("config")), "evaluated_at": best.get("evaluated_at"), "observed_at": best["observed_at"], | ||||
| 161 | + "source_url": best.get("source_url"), "tier": best.get("tier"), "result_id": best["id"]}, | ||||
| 162 | + "trust_levels": sorted({x["trust_level"] for x in g["rows"]})}) | ||||
| 163 | + metrics.append({"metric": m["metric"], "groups": groups}) | ||||
| 164 | + items.append({**b["benchmark"], "metrics": metrics}) | ||||
| 165 | + return {"items": items, "total_rows": len(rows), "note": "Current rows only, grouped by benchmark → canonical metric → comparability group (task configuration). " | ||||
| 166 | + "Effort variants folded into this model appear as rows of the same group."} | ||||
| 167 | + | ||||
| 168 | + | ||||
| 169 | +async def leaderboard(conn: AsyncConnection, benchmark_id: str, *, limit: int = 100, offset: int = 0, config: str | None = None, history: bool = False, | ||||
| 170 | + metric: str | None = None, config_key: str | None = None) -> list[dict[str, Any]]: | ||||
| 102 | where = ["r.benchmark_id = :id"] | 171 | where = ["r.benchmark_id = :id"] | ||
| 103 | params: dict[str, Any] = {"id": benchmark_id, "lim": limit, "off": offset} | 172 | params: dict[str, Any] = {"id": benchmark_id, "lim": limit, "off": offset} | ||
| 104 | if not history: | 173 | if not history: | ||
@@ -106,8 +175,20 @@ async def leaderboard(conn: AsyncConnection, benchmark_id: str, *, limit: int = | |||||
| 106 | if config: | 175 | if config: | ||
| 107 | where.append("r.config::text ilike :cfg") | 176 | where.append("r.config::text ilike :cfg") | ||
| 108 | params["cfg"] = f"%{config}%" | 177 | params["cfg"] = f"%{config}%" | ||
| 109 | − rows = await fetch_all(conn, f"select {RESULT_COLS} from {RESULT_FROM} where {' and '.join(where)} order by {RESULT_ORDER} limit :lim offset :off", **params) | 178 | + if metric: | ||
| 110 | − return [result_row(r) for r in rows] | 179 | + where.append("lower(r.metric) = lower(:metric)") | ||
| 180 | + params["metric"] = metric | ||||
| 181 | + if config_key: | ||||
| 182 | + where.append("r.config_key = :ck") | ||||
| 183 | + params["ck"] = config_key | ||||
| 184 | + rows = await fetch_all(conn, f"select {RESULT_COLS}, r.config_key, r.trust_level from {RESULT_FROM} where {' and '.join(where)} order by {RESULT_ORDER} limit :lim offset :off", **params) | ||||
| 185 | + out = [] | ||||
| 186 | + for r in rows: | ||||
| 187 | + item = result_row(r) | ||||
| 188 | + item["config_key"] = r.get("config_key") | ||||
| 189 | + item["trust_level"] = r.get("trust_level") | ||||
| 190 | + out.append(item) | ||||
| 191 | + return out | ||||
| 111 | 192 | ||||
| 112 | 193 | ||||
| 113 | async def lineage_of(conn: AsyncConnection, model_id: str) -> dict[str, list[dict[str, Any]]]: | 194 | async def lineage_of(conn: AsyncConnection, model_id: str) -> dict[str, list[dict[str, Any]]]: | ||
@@ -126,7 +207,7 @@ async def lineage_of(conn: AsyncConnection, model_id: str) -> dict[str, list[dic | |||||
| 126 | union | 207 | union | ||
| 127 | select r.subject_id, down.depth + 1 from down join relations r on r.object_id = down.id and r.valid_to is null and r.predicate = any(cast(:preds as text[])) where down.depth < 3) | 208 | select r.subject_id, down.depth + 1 from down join relations r on r.object_id = down.id and r.valid_to is null and r.predicate = any(cast(:preds as text[])) where down.depth < 3) | ||
| 128 | select distinct on (e.id) down.depth, {ENTITY_COLS} from down join entities e on e.id = down.id left join entities eo on eo.id = e.organization_id | 209 | select distinct on (e.id) down.depth, {ENTITY_COLS} from down join entities e on e.id = down.id left join entities eo on eo.id = e.organization_id | ||
| 129 | − where e.id <> :id and e.merged_into is null order by e.id, down.depth""", id=model_id, preds=desc_preds) | 210 | + where e.id <> :id and e.merged_into is null and e.entity_type <> 'artifact' order by e.id, down.depth""", id=model_id, preds=desc_preds) | ||
| 130 | quants = await fetch_all(conn, f"""select {ENTITY_COLS} from relations r join entities e on e.id = r.subject_id left join entities eo on eo.id = e.organization_id | 211 | quants = await fetch_all(conn, f"""select {ENTITY_COLS} from relations r join entities e on e.id = r.subject_id left join entities eo on eo.id = e.organization_id | ||
| 131 | where r.object_id = :id and r.predicate = 'quantized_from' and r.valid_to is null and e.merged_into is null | 212 | where r.object_id = :id and r.predicate = 'quantized_from' and r.valid_to is null and e.merged_into is null | ||
| 132 | order by e.canonical_name limit 100""", id=model_id) | 213 | order by e.canonical_name limit 100""", id=model_id) | ||
@@ -135,6 +216,111 @@ async def lineage_of(conn: AsyncConnection, model_id: str) -> dict[str, list[dic | |||||
| 135 | "quantizations": [entity_summary(r) for r in quants]} | 216 | "quantizations": [entity_summary(r) for r in quants]} | ||
| 136 | 217 | ||||
| 137 | 218 | ||||
| 219 | +async def artifacts_of_model(conn: AsyncConnection, model_id: str) -> dict[str, Any]: | ||||
| 220 | + """Artifacts (entity_type 'artifact' with canonical_id = model, or `artifact_of` relation) grouped by kind.""" | ||||
| 221 | + rows = await fetch_all(conn, f""" | ||||
| 222 | + with ids as (select e.id, e.artifact_kind from entities e where e.canonical_id = :id and e.entity_type = 'artifact' and e.merged_into is null | ||||
| 223 | + union select r.subject_id, null from relations r join entities x on x.id = r.subject_id where r.object_id = :id and r.predicate in ('artifact_of','quantized_from') | ||||
| 224 | + and r.valid_to is null and x.entity_type = 'artifact' and x.merged_into is null) | ||||
| 225 | + select distinct on (e.id) coalesce(e.artifact_kind, ids.artifact_kind) as kind, {ENTITY_COLS} from ids join entities e on e.id = ids.id left join entities eo on eo.id = e.organization_id | ||||
| 226 | + order by e.id limit 300""", id=model_id) | ||||
| 227 | + groups: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 228 | + for r in rows: | ||||
| 229 | + kind = r["kind"] if r["kind"] in ARTIFACT_KINDS else "other" | ||||
| 230 | + s = entity_summary(r) or {} | ||||
| 231 | + s["artifact_kind"] = r["kind"] | ||||
| 232 | + groups[kind].append(s) | ||||
| 233 | + ordered = [k for k in (*ARTIFACT_KINDS, "other") if k in groups] | ||||
| 234 | + return {"items": [{"kind": k, "items": sorted(groups[k], key=lambda x: x["name"] or ""), "count": len(groups[k])} for k in ordered], "total": len(rows)} | ||||
| 235 | + | ||||
| 236 | + | ||||
| 237 | +async def family_of_model(conn: AsyncConnection, row: dict[str, Any]) -> dict[str, Any] | None: | ||||
| 238 | + fid = row.get("family_id") | ||||
| 239 | + if fid: | ||||
| 240 | + fam = await fetch_one(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.id = :id", id=fid) | ||||
| 241 | + if fam: | ||||
| 242 | + return entity_summary(fam) | ||||
| 243 | + label = (row.get("attributes") or {}).get("family") | ||||
| 244 | + if label: | ||||
| 245 | + fam = await fetch_one(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.entity_type = 'model_family' and e.merged_into is null and e.canonical_name ilike :n limit 1", n=label) | ||||
| 246 | + return entity_summary(fam) if fam else {"id": None, "entity_type": "model_family", "slug": None, "name": label, "canonical": False, | ||||
| 247 | + "note": "family label from attributes; no model_family entity yet"} | ||||
| 248 | + return None | ||||
| 249 | + | ||||
| 250 | + | ||||
| 251 | +async def canonical_of_artifact(conn: AsyncConnection, row: dict[str, Any]) -> dict[str, Any] | None: | ||||
| 252 | + cid = row.get("canonical_id") | ||||
| 253 | + if not cid: | ||||
| 254 | + rel = await fetch_one(conn, "select object_id from relations where subject_id = :id and predicate in ('artifact_of','quantized_from') and valid_to is null order by predicate limit 1", id=row["id"]) | ||||
| 255 | + cid = rel["object_id"] if rel else None | ||||
| 256 | + if not cid: | ||||
| 257 | + return None | ||||
| 258 | + can = await fetch_one(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.id = :id", id=cid) | ||||
| 259 | + return entity_summary(can) if can else None | ||||
| 260 | + | ||||
| 261 | + | ||||
| 262 | +async def identity_of_model(conn: AsyncConnection, model_id: str, identifiers: list[dict[str, str]]) -> dict[str, Any]: | ||||
| 263 | + counts = await fetch_one(conn, """ | ||||
| 264 | + select (select count(*) from entities a where a.canonical_id = :id and a.entity_type = 'artifact' and a.merged_into is null and coalesce(a.artifact_kind, '') <> 'checkpoint') as third_party, | ||||
| 265 | + (select count(*) from entities a where a.canonical_id = :id and a.entity_type = 'artifact' and a.merged_into is null and a.artifact_kind = 'checkpoint') as official_artifacts, | ||||
| 266 | + (select count(distinct p.provider_id) from prices p where p.model_id = :id and p.valid_to is null) as providers, | ||||
| 267 | + (select count(*) from entities v where v.merged_into = :id) as folded_variants""", id=model_id) | ||||
| 268 | + hf_repos = [i["value"] for i in identifiers if i["scheme"] == "hf_repo"] | ||||
| 269 | + api_ids = sorted({i["value"] for i in identifiers if i["scheme"].endswith("_model_id") or i["scheme"] in ("openrouter", "api_model_id", "artificial_analysis")}) | ||||
| 270 | + c = counts or {} | ||||
| 271 | + return {"canonical_model": True, "official_checkpoints": hf_repos, "official_artifacts": int(c.get("official_artifacts") or 0), "third_party_artifacts": int(c.get("third_party") or 0), | ||||
| 272 | + "provider_deployments": int(c.get("providers") or 0), "folded_variants": int(c.get("folded_variants") or 0), "api_aliases": api_ids, | ||||
| 273 | + "note": "official_checkpoints = hf_repo identifiers carried by the model itself; artifacts are separate entities pointing here through canonical_id."} | ||||
| 274 | + | ||||
| 275 | + | ||||
| 276 | +def licence_block(attrs: dict[str, Any]) -> dict[str, Any] | None: | ||||
| 277 | + key = attrs.get("license_key") or normalize_license(attrs.get("license")) | ||||
| 278 | + info = LICENSES.get(key) if key else None | ||||
| 279 | + if not info: | ||||
| 280 | + return {"key": None, "raw": attrs.get("license") or attrs.get("license_raw"), "note": "licence label not classified in the ontology"} if attrs.get("license") else None | ||||
| 281 | + return {**info.as_dict(), "raw": attrs.get("license_raw") or attrs.get("license"), "url_observed": attrs.get("license_url")} | ||||
| 282 | + | ||||
| 283 | + | ||||
| 284 | +def openness_block(attrs: dict[str, Any]) -> dict[str, Any] | None: | ||||
| 285 | + raw = attrs.get("openness") | ||||
| 286 | + cat = normalize_openness(raw) if raw else None | ||||
| 287 | + dims = attrs.get("openness_dimensions") | ||||
| 288 | + if not isinstance(dims, dict): | ||||
| 289 | + key = attrs.get("license_key") or normalize_license(attrs.get("license")) | ||||
| 290 | + weights = True if cat in ("open-weights", "open-source", "restricted-weights") else False if cat == "proprietary" else None | ||||
| 291 | + dims = openness_dimensions(weights_available=weights, license_key=key, license_raw=attrs.get("license")) | ||||
| 292 | + if not cat and not raw: | ||||
| 293 | + return None | ||||
| 294 | + return {"category": cat or "unknown", "raw": raw, "label": OPENNESS_LABELS.get(cat or "unknown"), "definition": OPENNESS_DEFINITIONS.get(cat or "unknown"), "dimensions": dims, | ||||
| 295 | + "note": "dimensions marked null are unknown, not false"} | ||||
| 296 | + | ||||
| 297 | + | ||||
| 298 | +async def version_history_of(conn: AsyncConnection, entity_id: str) -> list[dict[str, Any]]: | ||||
| 299 | + rows = await fetch_all(conn, f"select {CLAIM_COLS} from {CLAIM_FROM} where c.entity_id = :id and c.property = any(cast(:props as text[])) and c.status <> 'retracted' " | ||||
| 300 | + f"order by c.property, c.valid_from asc, c.observed_at asc limit 2000", id=entity_id, props=list(VERSIONED_PROPERTIES)) | ||||
| 301 | + by_prop: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 302 | + for r in rows: | ||||
| 303 | + by_prop[r["property"]].append(r) | ||||
| 304 | + out = [] | ||||
| 305 | + for prop in VERSIONED_PROPERTIES: | ||||
| 306 | + claims = by_prop.get(prop) | ||||
| 307 | + if not claims: | ||||
| 308 | + continue | ||||
| 309 | + transitions = [] | ||||
| 310 | + prev: Any = None | ||||
| 311 | + for c in claims: | ||||
| 312 | + if c["status"] == "conflicting": | ||||
| 313 | + continue | ||||
| 314 | + if c["value"] == prev and transitions: | ||||
| 315 | + transitions[-1]["valid_to"] = c["valid_to"] or transitions[-1]["valid_to"] | ||||
| 316 | + continue | ||||
| 317 | + transitions.append({"from": prev, "to": c["value"], "valid_from": c["valid_from"], "valid_to": c["valid_to"], "effective_at": c["effective_at"], | ||||
| 318 | + "source_url": c["source_url"], "tier": c["tier"], "claim_id": c["id"], "status": c["status"]}) | ||||
| 319 | + prev = c["value"] | ||||
| 320 | + out.append({"property": prop, "transitions": transitions, "current": transitions[-1]["to"] if transitions else None}) | ||||
| 321 | + return out | ||||
| 322 | + | ||||
| 323 | + | ||||
| 138 | async def providers_of_model(conn: AsyncConnection, model_id: str) -> list[dict[str, Any]]: | 324 | async def providers_of_model(conn: AsyncConnection, model_id: str) -> list[dict[str, Any]]: | ||
| 139 | rows = await fetch_all(conn, f""" | 325 | rows = await fetch_all(conn, f""" | ||
| 140 | with ids as (select r.object_id as id from relations r where r.subject_id = :id and r.predicate = 'available_through' and r.valid_to is null | 326 | with ids as (select r.object_id as id from relations r where r.subject_id = :id and r.predicate = 'available_through' and r.valid_to is null | ||
@@ -173,15 +359,20 @@ async def related_by_type(conn: AsyncConnection, entity_id: str, etype: str, *, | |||||
| 173 | return [entity_summary(r) for r in rows], int(rows[0]["total"]) if rows else 0 | 359 | return [entity_summary(r) for r in rows], int(rows[0]["total"]) if rows else 0 | ||
| 174 | 360 | ||||
| 175 | 361 | ||||
| 176 | −async def entity_detail(row: dict[str, Any]) -> dict[str, Any]: | 362 | +async def _run_group(tasks: list[tuple[str, Any, tuple[Any, ...], dict[str, Any]]]) -> dict[str, Any]: | ||
| 177 | − """Full detail. Blocks run on separate pooled connections concurrently (read-only, so no transaction needed).""" | 363 | + """Run the blocks of one group sequentially on a single pooled connection.""" | ||
| 178 | − eid, etype = row["id"], row["entity_type"] | 364 | + out: dict[str, Any] = {} | ||
| 365 | + async with connection() as conn: | ||||
| 366 | + for name, fn, args, kw in tasks: | ||||
| 367 | + out[name] = await fn(conn, *args, **kw) | ||||
| 368 | + return out | ||||
| 179 | 369 | ||||
| 180 | − async def one(fn: Any, *args: Any, **kw: Any) -> Any: | ||||
| 181 | − async with connection() as conn: | ||||
| 182 | − return await fn(conn, *args, **kw) | ||||
| 183 | 370 | ||||
| 371 | +async def entity_detail(row: dict[str, Any]) -> dict[str, Any]: | ||||
| 372 | + """Full detail. Blocks are spread over ≤ 4 concurrent groups (one connection each).""" | ||||
| 373 | + eid, etype = row["id"], row["entity_type"] | ||||
| 184 | provenance = dict(row.get("provenance") or {}) | 374 | provenance = dict(row.get("provenance") or {}) | ||
| 375 | + attrs = row.get("attributes") or {} | ||||
| 185 | 376 | ||||
| 186 | async def base(conn: AsyncConnection) -> dict[str, Any]: | 377 | async def base(conn: AsyncConnection) -> dict[str, Any]: | ||
| 187 | aliases = await fetch_all(conn, "select alias from entity_aliases where entity_id = :id order by kind, alias limit 200", id=eid) | 378 | aliases = await fetch_all(conn, "select alias from entity_aliases where entity_id = :id order by kind, alias limit 200", id=eid) | ||
@@ -189,36 +380,42 @@ async def entity_detail(row: dict[str, Any]) -> dict[str, Any]: | |||||
| 189 | await enrich_provenance(conn, provenance) | 380 | await enrich_provenance(conn, provenance) | ||
| 190 | return {"aliases": [a["alias"] for a in aliases], "identifiers": [{"scheme": i["scheme"], "value": i["value"]} for i in idents]} | 381 | return {"aliases": [a["alias"] for a in aliases], "identifiers": [{"scheme": i["scheme"], "value": i["value"]} for i in idents]} | ||
| 191 | 382 | ||||
| 192 | − tasks: dict[str, Any] = {"base": one(base), "relations": one(relations_grouped, eid), "sources": one(sources_of, eid), "timeline": one(timeline_of, eid, etype)} | 383 | + T = lambda name, fn, *args, **kw: (name, fn, args, kw) | ||
| 384 | + groups: list[list[tuple[str, Any, tuple[Any, ...], dict[str, Any]]]] = [ | ||||
| 385 | + [T("base", base), T("relations", relations_grouped, eid)], | ||||
| 386 | + [T("sources", sources_of, eid), T("timeline", timeline_of, eid, etype)], | ||||
| 387 | + ] | ||||
| 193 | if etype == "model": | 388 | if etype == "model": | ||
| 194 | − tasks["prices"] = one(prices_of_model, eid, current_only=True) | 389 | + groups[0] += [T("prices", prices_of_model, eid, current_only=True), T("deployments", deployments_of_model, eid), T("providers", providers_of_model, eid)] | ||
| 195 | − tasks["price_history"] = one(prices_of_model, eid, current_only=False) | 390 | + groups[1] += [T("price_history", prices_of_model, eid, current_only=False), T("family", family_of_model, row), T("artifacts", artifacts_of_model, eid)] | ||
| 196 | − tasks["results"] = one(results_of_model, eid) | 391 | + groups.append([T("results", results_of_model, eid), T("benchmarks", benchmarks_of_model, eid), T("version_history", version_history_of, eid)]) | ||
| 197 | − tasks["lineage"] = one(lineage_of, eid) | 392 | + groups.append([T("lineage", lineage_of, eid), T("hardware_fit", hardware_fit_of_model, attrs), T("papers", related_by_type, eid, "paper", limit=24), | ||
| 198 | − tasks["providers"] = one(providers_of_model, eid) | 393 | + T("repositories", related_by_type, eid, "repository", limit=24)]) | ||
| 199 | − tasks["hardware_fit"] = one(hardware_fit_of_model, row["attributes"] or {}) | 394 | + elif etype == "artifact": | ||
| 200 | − tasks["papers"] = one(related_by_type, eid, "paper", limit=24) | 395 | + groups[0] += [T("canonical", canonical_of_artifact, row), T("prices", prices_of_model, eid, current_only=True)] | ||
| 201 | − tasks["repositories"] = one(related_by_type, eid, "repository", limit=24) | 396 | + groups[1] += [T("lineage", lineage_of, eid), T("hardware_fit", hardware_fit_of_model, attrs)] | ||
| 202 | elif etype in COMPANY_TYPES: | 397 | elif etype in COMPANY_TYPES: | ||
| 203 | − tasks["models"] = one(related_by_type, eid, "model", limit=50, include_org_children=True) | 398 | + groups[0] += [T("models", related_by_type, eid, "model", limit=50, include_org_children=True)] | ||
| 204 | − tasks["papers"] = one(related_by_type, eid, "paper", limit=24, include_org_children=True) | 399 | + groups[1] += [T("papers", related_by_type, eid, "paper", limit=24, include_org_children=True), T("repositories", related_by_type, eid, "repository", limit=24, include_org_children=True)] | ||
| 205 | − tasks["repositories"] = one(related_by_type, eid, "repository", limit=24, include_org_children=True) | ||||
| 206 | elif etype == "provider": | 400 | elif etype == "provider": | ||
| 207 | − tasks["prices"] = one(prices_of_provider, eid) | 401 | + groups[0] += [T("prices", prices_of_provider, eid)] | ||
| 208 | − tasks["models"] = one(_provider_models, eid) | 402 | + groups[1] += [T("models", _provider_models, eid)] | ||
| 209 | elif etype == "benchmark": | 403 | elif etype == "benchmark": | ||
| 210 | − tasks["results"] = one(leaderboard, eid, limit=100) | 404 | + groups[0] += [T("results", leaderboard, eid, limit=100)] | ||
| 211 | elif etype == "hardware": | 405 | elif etype == "hardware": | ||
| 212 | − tasks["models"] = one(related_by_type, eid, "model", limit=50) | 406 | + groups[0] += [T("models", related_by_type, eid, "model", limit=50)] | ||
| 213 | − elif etype == "framework": | 407 | + elif etype in ("framework", "library", "runtime"): | ||
| 214 | − tasks["repositories"] = one(related_by_type, eid, "repository", limit=24) | 408 | + groups[0] += [T("repositories", related_by_type, eid, "repository", limit=24)] | ||
| 409 | + elif etype == "model_family": | ||||
| 410 | + groups[0] += [T("models", _family_models, eid)] | ||||
| 215 | 411 | ||||
| 216 | − keys = list(tasks) | 412 | + results = await asyncio.gather(*(_run_group(g) for g in groups[:MAX_CONCURRENT_GROUPS])) | ||
| 217 | − values = await asyncio.gather(*(tasks[k] for k in keys)) | 413 | + blocks: dict[str, Any] = {} | ||
| 218 | − blocks = dict(zip(keys, values, strict=True)) | 414 | + for r in results: | ||
| 415 | + blocks.update(r) | ||||
| 219 | 416 | ||||
| 220 | detail = entity_summary(row) or {} | 417 | detail = entity_summary(row) or {} | ||
| 221 | − detail["attributes"] = row.get("attributes") or {} | 418 | + detail["attributes"] = attrs | ||
| 222 | detail["provenance"] = provenance | 419 | detail["provenance"] = provenance | ||
| 223 | detail.update(blocks.pop("base")) | 420 | detail.update(blocks.pop("base")) | ||
| 224 | detail["relations"] = blocks.pop("relations") | 421 | detail["relations"] = blocks.pop("relations") | ||
@@ -232,11 +429,33 @@ async def entity_detail(row: dict[str, Any]) -> dict[str, Any]: | |||||
| 232 | detail[k] = v[0] | 429 | detail[k] = v[0] | ||
| 233 | elif v is not None: | 430 | elif v is not None: | ||
| 234 | detail[k] = v | 431 | detail[k] = v | ||
| 235 | − if etype == "model" and "hardware_fit" in detail: | 432 | + if etype == "model": | ||
| 236 | − detail["hardware_fit_assumptions"] = hf.ASSUMPTIONS | 433 | + if "hardware_fit" in detail: | ||
| 434 | + detail["hardware_fit_assumptions"] = hf.ASSUMPTIONS | ||||
| 435 | + detail["identity"] = await _with_conn(identity_of_model, eid, detail.get("identifiers") or []) | ||||
| 436 | + lic = licence_block(attrs) | ||||
| 437 | + if lic is not None: | ||||
| 438 | + detail["licence"] = lic | ||||
| 439 | + opn = openness_block(attrs) | ||||
| 440 | + if opn is not None: | ||||
| 441 | + detail["openness"] = opn | ||||
| 442 | + detail["family_id"] = row.get("family_id") | ||||
| 443 | + detail["identity_confidence"] = row.get("identity_confidence") | ||||
| 444 | + if etype == "artifact": | ||||
| 445 | + detail["artifact_kind"] = row.get("artifact_kind") | ||||
| 446 | + detail.setdefault("canonical", None) | ||||
| 447 | + if "hardware_fit" in detail: | ||||
| 448 | + detail["hardware_fit_assumptions"] = hf.ASSUMPTIONS | ||||
| 449 | + if row.get("redirected_from"): | ||||
| 450 | + detail["redirected_from"] = row["redirected_from"] | ||||
| 237 | return detail | 451 | return detail | ||
| 238 | 452 | ||||
| 239 | 453 | ||||
| 454 | +async def _with_conn(fn: Any, *args: Any) -> Any: | ||||
| 455 | + async with connection() as conn: | ||||
| 456 | + return await fn(conn, *args) | ||||
| 457 | + | ||||
| 458 | + | ||||
| 240 | async def _provider_models(conn: AsyncConnection, provider_id: str) -> tuple[list[dict[str, Any]], int]: | 459 | async def _provider_models(conn: AsyncConnection, provider_id: str) -> tuple[list[dict[str, Any]], int]: | ||
| 241 | rows = await fetch_all(conn, f""" | 460 | rows = await fetch_all(conn, f""" | ||
| 242 | with ids as (select p.model_id as id from prices p where p.provider_id = :id and p.valid_to is null | 461 | with ids as (select p.model_id as id from prices p where p.provider_id = :id and p.valid_to is null | ||
@@ -246,11 +465,25 @@ async def _provider_models(conn: AsyncConnection, provider_id: str) -> tuple[lis | |||||
| 246 | return [entity_summary(r) for r in rows], int(rows[0]["total"]) if rows else 0 | 465 | return [entity_summary(r) for r in rows], int(rows[0]["total"]) if rows else 0 | ||
| 247 | 466 | ||||
| 248 | 467 | ||||
| 468 | +async def _family_models(conn: AsyncConnection, family_id: str) -> tuple[list[dict[str, Any]], int]: | ||||
| 469 | + rows = await fetch_all(conn, f"""select count(*) over () as total, {ENTITY_COLS} from {ENTITY_FROM} where e.family_id = :id and e.entity_type = 'model' and e.merged_into is null | ||||
| 470 | + order by e.attributes->>'release_date' desc nulls last, e.canonical_name limit 50""", id=family_id) | ||||
| 471 | + return [entity_summary(r) for r in rows], int(rows[0]["total"]) if rows else 0 | ||||
| 472 | + | ||||
| 473 | + | ||||
| 249 | __all__ = [ | 474 | __all__ = [ | ||
| 475 | + "VERSIONED_PROPERTIES", | ||||
| 476 | + "artifacts_of_model", | ||||
| 477 | + "benchmarks_of_model", | ||||
| 478 | + "deployments_of_model", | ||||
| 250 | "entity_detail", | 479 | "entity_detail", | ||
| 480 | + "event_date_col", | ||||
| 481 | + "family_of_model", | ||||
| 251 | "hardware_fit_of_model", | 482 | "hardware_fit_of_model", | ||
| 252 | "leaderboard", | 483 | "leaderboard", | ||
| 484 | + "licence_block", | ||||
| 253 | "lineage_of", | 485 | "lineage_of", | ||
| 486 | + "openness_block", | ||||
| 254 | "prices_of_model", | 487 | "prices_of_model", | ||
| 255 | "prices_of_provider", | 488 | "prices_of_provider", | ||
| 256 | "providers_of_model", | 489 | "providers_of_model", | ||
@@ -259,4 +492,5 @@ __all__ = [ | |||||
| 259 | "results_of_model", | 492 | "results_of_model", | ||
| 260 | "sources_of", | 493 | "sources_of", | ||
| 261 | "timeline_of", | 494 | "timeline_of", | ||
| 495 | + "version_history_of", | ||||
| 262 | ] | 496 | ] | ||
added
src/aiatlas/api/etag.py
+103 −0
@@ -0,0 +1,103 @@ | |||||
| 1 | +"""Weak ETag + Cache-Control on public GET responses (API 1.1). | ||||
| 2 | + | ||||
| 3 | +Pure ASGI middleware so it can sit *inside* GZip (the hash is computed on the uncompressed JSON body — gzip output embeds a | ||||
| 4 | +timestamp and would never be stable). `If-None-Match` matching the body hash → `304 Not Modified` with an empty body. | ||||
| 5 | +Admin, docs and non-200 responses are left untouched.""" | ||||
| 6 | +from __future__ import annotations | ||||
| 7 | + | ||||
| 8 | +import hashlib | ||||
| 9 | +from typing import Any | ||||
| 10 | + | ||||
| 11 | +from starlette.types import ASGIApp, Message, Receive, Scope, Send | ||||
| 12 | + | ||||
| 13 | +PUBLIC_PREFIX = "/api/v1/" | ||||
| 14 | +SKIP_PREFIXES = ("/api/v1/admin", "/api/v1/docs", "/api/v1/openapi.json", "/api/v1/api-keys") | ||||
| 15 | +CACHE_CONTROL = "public, max-age=60, stale-while-revalidate=300" | ||||
| 16 | +MAX_BUFFER = 8 * 1024 * 1024 # bodies above this are passed through unhashed | ||||
| 17 | + | ||||
| 18 | + | ||||
| 19 | +def weak_etag(body: bytes) -> str: | ||||
| 20 | + return 'W/"' + hashlib.sha1(body).hexdigest() + '"' | ||||
| 21 | + | ||||
| 22 | + | ||||
| 23 | +def _etags_match(header: str | None, etag: str) -> bool: | ||||
| 24 | + if not header: | ||||
| 25 | + return False | ||||
| 26 | + if header.strip() == "*": | ||||
| 27 | + return True | ||||
| 28 | + wanted = {t.strip() for t in header.split(",")} | ||||
| 29 | + strong = etag.removeprefix("W/") | ||||
| 30 | + return etag in wanted or strong in wanted or any((t.removeprefix("W/")) == strong for t in wanted) | ||||
| 31 | + | ||||
| 32 | + | ||||
| 33 | +class ETagMiddleware: | ||||
| 34 | + def __init__(self, app: ASGIApp) -> None: | ||||
| 35 | + self.app = app | ||||
| 36 | + | ||||
| 37 | + async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: | ||||
| 38 | + if scope["type"] != "http" or scope.get("method") != "GET": | ||||
| 39 | + await self.app(scope, receive, send) | ||||
| 40 | + return | ||||
| 41 | + path: str = scope.get("path", "") | ||||
| 42 | + if not path.startswith(PUBLIC_PREFIX) or path.startswith(SKIP_PREFIXES): | ||||
| 43 | + await self.app(scope, receive, send) | ||||
| 44 | + return | ||||
| 45 | + headers = {k.decode("latin-1").lower(): v.decode("latin-1") for k, v in scope.get("headers", [])} | ||||
| 46 | + inm = headers.get("if-none-match") | ||||
| 47 | + | ||||
| 48 | + start: Message | None = None | ||||
| 49 | + chunks: list[bytes] = [] | ||||
| 50 | + passthrough = False | ||||
| 51 | + size = 0 | ||||
| 52 | + | ||||
| 53 | + async def send_wrapper(message: Message) -> None: | ||||
| 54 | + nonlocal start, passthrough, size | ||||
| 55 | + if passthrough: | ||||
| 56 | + await send(message) | ||||
| 57 | + return | ||||
| 58 | + if message["type"] == "http.response.start": | ||||
| 59 | + start = message | ||||
| 60 | + if message.get("status") != 200: | ||||
| 61 | + passthrough = True | ||||
| 62 | + await send(message) | ||||
| 63 | + return | ||||
| 64 | + if message["type"] == "http.response.body": | ||||
| 65 | + body = message.get("body", b"") | ||||
| 66 | + size += len(body) | ||||
| 67 | + if size > MAX_BUFFER: | ||||
| 68 | + passthrough = True | ||||
| 69 | + assert start is not None | ||||
| 70 | + await send(start) | ||||
| 71 | + for c in chunks: | ||||
| 72 | + await send({"type": "http.response.body", "body": c, "more_body": True}) | ||||
| 73 | + await send(message) | ||||
| 74 | + return | ||||
| 75 | + chunks.append(body) | ||||
| 76 | + if message.get("more_body"): | ||||
| 77 | + return | ||||
| 78 | + assert start is not None | ||||
| 79 | + full = b"".join(chunks) | ||||
| 80 | + etag = weak_etag(full) | ||||
| 81 | + raw_headers: list[tuple[bytes, bytes]] = [(k, v) for k, v in start.get("headers", []) if k.lower() not in (b"etag", b"cache-control")] | ||||
| 82 | + raw_headers.append((b"etag", etag.encode("latin-1"))) | ||||
| 83 | + raw_headers.append((b"cache-control", CACHE_CONTROL.encode("latin-1"))) | ||||
| 84 | + if _etags_match(inm, etag): | ||||
| 85 | + keep = {b"etag", b"cache-control", b"vary", b"x-content-type-options", b"referrer-policy", b"access-control-allow-origin"} | ||||
| 86 | + hdrs = [(k, v) for k, v in raw_headers if k.lower() in keep] | ||||
| 87 | + await send({"type": "http.response.start", "status": 304, "headers": hdrs}) | ||||
| 88 | + await send({"type": "http.response.body", "body": b"", "more_body": False}) | ||||
| 89 | + return | ||||
| 90 | + await send({"type": "http.response.start", "status": 200, "headers": raw_headers}) | ||||
| 91 | + await send({"type": "http.response.body", "body": full, "more_body": False}) | ||||
| 92 | + return | ||||
| 93 | + await send(message) | ||||
| 94 | + | ||||
| 95 | + await self.app(scope, receive, send_wrapper) | ||||
| 96 | + | ||||
| 97 | + | ||||
| 98 | +def etag_of(value: Any) -> str: | ||||
| 99 | + """Helper for tests and callers holding an already-rendered body.""" | ||||
| 100 | + return weak_etag(value if isinstance(value, bytes) else str(value).encode()) | ||||
| 101 | + | ||||
| 102 | + | ||||
| 103 | +__all__ = ["CACHE_CONTROL", "ETagMiddleware", "etag_of", "weak_etag"] | ||||
modified
src/aiatlas/api/main.py
+20 −7
@@ -1,4 +1,4 @@ | |||||
| 1 | −"""FastAPI application — `/api/v1`. Loopback-only in production (Next.js proxies `/api/v1/*`); docs at `/api/v1/docs`.""" | 1 | +"""FastAPI application — `/api/v1` (API 1.1). Loopback-only in production (Next.js proxies `/api/v1/*`); docs at `/api/v1/docs`.""" | ||
| 2 | from __future__ import annotations | 2 | from __future__ import annotations | ||
| 3 | 3 | ||||
| 4 | import asyncio | 4 | import asyncio | ||
@@ -15,16 +15,23 @@ from starlette.exceptions import HTTPException as StarletteHTTPException | |||||
| 15 | 15 | ||||
| 16 | import aiatlas | 16 | import aiatlas | ||
| 17 | from aiatlas.api.common import AtlasJSONResponse | 17 | from aiatlas.api.common import AtlasJSONResponse | ||
| 18 | +from aiatlas.api.etag import ETagMiddleware | ||||
| 18 | from aiatlas.api.routers import ( | 19 | from aiatlas.api.routers import ( | ||
| 19 | admin, | 20 | admin, | ||
| 21 | + admin_quality, | ||||
| 20 | benchmarks, | 22 | benchmarks, | ||
| 21 | changes, | 23 | changes, | ||
| 24 | + claims, | ||||
| 22 | companies, | 25 | companies, | ||
| 23 | compare, | 26 | compare, | ||
| 27 | + deployments, | ||||
| 24 | diff, | 28 | diff, | ||
| 25 | entities, | 29 | entities, | ||
| 26 | explore, | 30 | explore, | ||
| 31 | + families, | ||||
| 32 | + graph, | ||||
| 27 | hardware, | 33 | hardware, | ||
| 34 | + intelligence, | ||||
| 28 | misc, | 35 | misc, | ||
| 29 | models, | 36 | models, | ||
| 30 | papers, | 37 | papers, | ||
@@ -42,13 +49,14 @@ from aiatlas.services import cache | |||||
| 42 | from aiatlas.services.llm import gateway | 49 | from aiatlas.services.llm import gateway | ||
| 43 | 50 | ||||
| 44 | log = logging.getLogger("aiatlas.api") | 51 | log = logging.getLogger("aiatlas.api") | ||
| 52 | +API_VERSION = "1.1" | ||||
| 45 | 53 | ||||
| 46 | 54 | ||||
| 47 | @asynccontextmanager | 55 | @asynccontextmanager | ||
| 48 | async def lifespan(app: FastAPI): # type: ignore[no-untyped-def] | 56 | async def lifespan(app: FastAPI): # type: ignore[no-untyped-def] | ||
| 49 | setup_logging(service="aia-api") | 57 | setup_logging(service="aia-api") | ||
| 50 | settings.ensure_dirs() | 58 | settings.ensure_dirs() | ||
| 51 | − log.info("api started", extra={"version": aiatlas.__version__, "env": settings.app_env, "port": settings.api_port}) | 59 | + log.info("api started", extra={"version": aiatlas.__version__, "api": API_VERSION, "env": settings.app_env, "port": settings.api_port}) | ||
| 52 | yield | 60 | yield | ||
| 53 | await cache.close() | 61 | await cache.close() | ||
| 54 | await dispose() | 62 | await dispose() | ||
@@ -57,12 +65,14 @@ async def lifespan(app: FastAPI): # type: ignore[no-untyped-def] | |||||
| 57 | app = FastAPI(title="AI Atlas API", version=aiatlas.__version__, lifespan=lifespan, default_response_class=AtlasJSONResponse, | 65 | app = FastAPI(title="AI Atlas API", version=aiatlas.__version__, lifespan=lifespan, default_response_class=AtlasJSONResponse, | ||
| 58 | docs_url="/api/v1/docs", redoc_url=None, openapi_url="/api/v1/openapi.json", | 66 | docs_url="/api/v1/docs", redoc_url=None, openapi_url="/api/v1/openapi.json", | ||
| 59 | description="The global intelligence layer for artificial intelligence: models, companies, papers, providers, prices, benchmarks, hardware — " | 67 | description="The global intelligence layer for artificial intelligence: models, companies, papers, providers, prices, benchmarks, hardware — " | ||
| 60 | − "with provenance and history on every fact. Contract: docs/API.md.") | 68 | + "with provenance and history on every fact. Contract: docs/API.md (API 1.1, additive over v1).") | ||
| 61 | 69 | ||||
| 70 | +# Middleware order: the LAST added is the OUTERMOST. ETag must be inside GZip (hash of the uncompressed body), so it is added first. | ||||
| 71 | +app.add_middleware(ETagMiddleware) | ||||
| 62 | app.add_middleware(GZipMiddleware, minimum_size=1024) | 72 | app.add_middleware(GZipMiddleware, minimum_size=1024) | ||
| 63 | _origins = {settings.site_url, "https://www.ai-atlas.co", "https://ai-atlas.co", "http://localhost:8320", "http://127.0.0.1:8320", "http://localhost:8330", "http://127.0.0.1:8330"} | 73 | _origins = {settings.site_url, "https://www.ai-atlas.co", "https://ai-atlas.co", "http://localhost:8320", "http://127.0.0.1:8320", "http://localhost:8330", "http://127.0.0.1:8330"} | ||
| 64 | app.add_middleware(CORSMiddleware, allow_origins=sorted(o for o in _origins if o), allow_methods=["GET", "POST", "PATCH", "OPTIONS"], | 74 | app.add_middleware(CORSMiddleware, allow_origins=sorted(o for o in _origins if o), allow_methods=["GET", "POST", "PATCH", "OPTIONS"], | ||
| 65 | − allow_headers=["*"], max_age=600) | 75 | + allow_headers=["*"], expose_headers=["etag", "cache-control", "x-api-version"], max_age=600) | ||
| 66 | 76 | ||||
| 67 | 77 | ||||
| 68 | @app.middleware("http") | 78 | @app.middleware("http") | ||
@@ -74,6 +84,7 @@ async def security_headers(request: Request, call_next): # type: ignore[no-unty | |||||
| 74 | return AtlasJSONResponse({"detail": "internal server error"}, status_code=500) | 84 | return AtlasJSONResponse({"detail": "internal server error"}, status_code=500) | ||
| 75 | response.headers["x-content-type-options"] = "nosniff" | 85 | response.headers["x-content-type-options"] = "nosniff" | ||
| 76 | response.headers["referrer-policy"] = "strict-origin-when-cross-origin" | 86 | response.headers["referrer-policy"] = "strict-origin-when-cross-origin" | ||
| 87 | + response.headers["x-api-version"] = API_VERSION | ||||
| 77 | return response | 88 | return response | ||
| 78 | 89 | ||||
| 79 | 90 | ||||
@@ -117,7 +128,7 @@ async def _health() -> dict[str, Any]: | |||||
| 117 | cached = {"reachable": False} | 128 | cached = {"reachable": False} | ||
| 118 | await cache.cache_set("health:llm", cached, 300) | 129 | await cache.cache_set("health:llm", cached, 300) | ||
| 119 | llm.update(cached) | 130 | llm.update(cached) | ||
| 120 | − return {"status": "ok" if db_ok else "degraded", "version": aiatlas.__version__, "db": db_ok, "redis": redis_ok, "llm": llm, "time": datetime.now(UTC)} | 131 | + return {"status": "ok" if db_ok else "degraded", "version": aiatlas.__version__, "api_version": API_VERSION, "db": db_ok, "redis": redis_ok, "llm": llm, "time": datetime.now(UTC)} | ||
| 121 | 132 | ||||
| 122 | 133 | ||||
| 123 | @app.get("/health", tags=["health"]) | 134 | @app.get("/health", tags=["health"]) | ||
@@ -131,7 +142,9 @@ async def health_v1() -> dict[str, Any]: | |||||
| 131 | 142 | ||||
| 132 | 143 | ||||
| 133 | # Order matters only where a literal path and a `{param}` path share a prefix — literal routes live in the same router and are declared first. | 144 | # Order matters only where a literal path and a `{param}` path share a prefix — literal routes live in the same router and are declared first. | ||
| 134 | −for r in (stats, search, models, companies, papers, providers, prices, benchmarks, hardware, explore, changes, timeline, compare, diff, sources, misc, entities, admin): | 145 | +# `intelligence`, `families`, `graph`, `claims`, `deployments` and `misc` mount literal paths under /api/v1 and must precede `entities` (/{slug_or_id}). | ||
| 146 | +for r in (stats, search, models, companies, papers, providers, prices, deployments, benchmarks, hardware, explore, changes, timeline, compare, diff, sources, | ||||
| 147 | + intelligence, families, graph, claims, misc, entities, admin, admin_quality): | ||||
| 135 | app.include_router(r.router) | 148 | app.include_router(r.router) | ||
| 136 | 149 | ||||
| 137 | −__all__ = ["app"] | 150 | +__all__ = ["API_VERSION", "app"] | ||
modified
src/aiatlas/api/routers/admin.py
+41 −6
@@ -8,10 +8,10 @@ import time | |||||
| 8 | from datetime import UTC, datetime | 8 | from datetime import UTC, datetime | ||
| 9 | from typing import Any | 9 | from typing import Any | ||
| 10 | 10 | ||||
| 11 | −from fastapi import APIRouter, Depends, Query | 11 | +from fastapi import APIRouter, Depends, Query, Request | ||
| 12 | from pydantic import BaseModel, Field | 12 | from pydantic import BaseModel, Field | ||
| 13 | 13 | ||||
| 14 | −from aiatlas.api.common import PAGINATION, ApiError, Pagination, page, rate_limit, require_admin | 14 | +from aiatlas.api.common import ADMIN_DEPENDENCIES, PAGINATION, ApiError, Pagination, audit, client_ip, page | ||
| 15 | from aiatlas.config import settings | 15 | from aiatlas.config import settings | ||
| 16 | from aiatlas.connectors import registry as connector_registry | 16 | from aiatlas.connectors import registry as connector_registry | ||
| 17 | from aiatlas.db import connection, execute, fetch_all, fetch_one, fetch_val, jsonb, transaction | 17 | from aiatlas.db import connection, execute, fetch_all, fetch_one, fetch_val, jsonb, transaction | ||
@@ -23,7 +23,32 @@ from aiatlas.services.llm import gateway | |||||
| 23 | from aiatlas.services.merge import merge_entities | 23 | from aiatlas.services.merge import merge_entities | ||
| 24 | from aiatlas.services.stats import live_counts | 24 | from aiatlas.services.stats import live_counts | ||
| 25 | 25 | ||||
| 26 | −router = APIRouter(prefix="/api/v1/admin", tags=["admin"], dependencies=[Depends(require_admin), Depends(rate_limit("admin"))]) | 26 | +SKIP_AUDIT_GET = ("/api/v1/admin/overview", "/api/v1/admin/infrastructure", "/api/v1/admin/llm/health", "/api/v1/admin/audit") | ||
| 27 | + | ||||
| 28 | + | ||||
| 29 | +async def audit_request(request: Request) -> None: | ||||
| 30 | + """Every admin call leaves a row in `admin_audit_log` (except the dashboard polling GETs). Runs AFTER the rate limit and the token check.""" | ||||
| 31 | + if request.method == "GET" and request.url.path in SKIP_AUDIT_GET: | ||||
| 32 | + return | ||||
| 33 | + payload: dict[str, Any] = {"method": request.method, "path": request.url.path} | ||||
| 34 | + if request.query_params: | ||||
| 35 | + payload["query"] = dict(request.query_params) | ||||
| 36 | + if request.method in ("POST", "PATCH", "PUT", "DELETE"): | ||||
| 37 | + try: | ||||
| 38 | + body = await request.body() | ||||
| 39 | + if body: | ||||
| 40 | + import orjson | ||||
| 41 | + | ||||
| 42 | + payload["body"] = orjson.loads(body) if len(body) < 64 * 1024 else {"truncated": True, "bytes": len(body)} | ||||
| 43 | + except Exception: # noqa: BLE001 | ||||
| 44 | + payload["body"] = {"unparsed": True} | ||||
| 45 | + target = request.path_params.get("name") or request.path_params.get("review_id") or request.path_params.get("job_id") or request.path_params.get("snap_id") \ | ||||
| 46 | + or request.path_params.get("doc_id") or request.path_params.get("entity_id") or request.path_params.get("run_id") or request.path_params.get("anomaly_id") \ | ||||
| 47 | + or request.path_params.get("quarantine_id") or request.path_params.get("a") | ||||
| 48 | + await audit(f"{request.method} {request.url.path}", str(target) if target else None, payload, client_ip(request)) | ||||
| 49 | + | ||||
| 50 | + | ||||
| 51 | +router = APIRouter(prefix="/api/v1/admin", tags=["admin"], dependencies=[*ADMIN_DEPENDENCIES, Depends(audit_request)]) | ||||
| 27 | _STARTED = time.time() | 52 | _STARTED = time.time() | ||
| 28 | SNAPSHOT_TEXT_LIMIT = 20 * 1024 | 53 | SNAPSHOT_TEXT_LIMIT = 20 * 1024 | ||
| 29 | HIDDEN = ("raw_path", "text_path") | 54 | HIDDEN = ("raw_path", "text_path") | ||
@@ -422,7 +447,8 @@ async def _accept_claim(conn: Any, claim_id: str) -> dict[str, Any]: | |||||
| 422 | async def duplicates(type: str | None = Query(None, alias="type"), limit: int = Query(100, ge=1, le=500), threshold: float = Query(0.8, ge=0.3, le=1.0)) -> dict[str, Any]: | 447 | async def duplicates(type: str | None = Query(None, alias="type"), limit: int = Query(100, ge=1, le=500), threshold: float = Query(0.8, ge=0.3, le=1.0)) -> dict[str, Any]: | ||
| 423 | where = "a.entity_type = :t" if type else "true" | 448 | where = "a.entity_type = :t" if type else "true" | ||
| 424 | async with connection() as conn: | 449 | async with connection() as conn: | ||
| 425 | − await execute(conn, "select set_limit(:th)", th=threshold) | 450 | + # `%` uses the pg_trgm GIN index at the session default threshold (0.3, a superset of any threshold ≥ 0.3); the exact bound is applied | ||
| 451 | + # with similarity() > :th — no set_limit(), which used to leak a GUC change into the pooled connection. | ||||
| 426 | rows = await fetch_all(conn, f""" | 452 | rows = await fetch_all(conn, f""" | ||
| 427 | select a.id as a_id, a.slug as a_slug, a.canonical_name as a_name, a.entity_type, a.first_seen_at as a_first_seen_at, oa.canonical_name as a_org, | 453 | select a.id as a_id, a.slug as a_slug, a.canonical_name as a_name, a.entity_type, a.first_seen_at as a_first_seen_at, oa.canonical_name as a_org, | ||
| 428 | b.id as b_id, b.slug as b_slug, b.canonical_name as b_name, b.first_seen_at as b_first_seen_at, ob.canonical_name as b_org, | 454 | b.id as b_id, b.slug as b_slug, b.canonical_name as b_name, b.first_seen_at as b_first_seen_at, ob.canonical_name as b_org, | ||
@@ -518,9 +544,18 @@ async def infrastructure() -> dict[str, Any]: | |||||
| 518 | "llm": {"available": gateway.available, "engine": gateway.engine.name}, "scheduler_tick_s": settings.scheduler_tick_s} | 544 | "llm": {"available": gateway.available, "engine": gateway.engine.name}, "scheduler_tick_s": settings.scheduler_tick_s} | ||
| 519 | 545 | ||||
| 520 | 546 | ||||
| 547 | +CACHE_PREFIXES_AFTER_RUN = ("/api/v1/stats", "/api/v1/changes", "/api/v1/benchmarks", "/api/v1/prices", "/api/v1/models", "/api/v1/deployments", "/api/v1/frontier", | ||||
| 548 | + "/api/v1/pulse", "/api/v1/open", "/api/v1/families", "/api/v1/timeline", "/api/v1/diff", "facets:", "frontier:") | ||||
| 549 | + | ||||
| 550 | + | ||||
| 521 | @router.post("/cache/flush") | 551 | @router.post("/cache/flush") | ||
| 522 | −async def cache_flush(prefix: str = "") -> dict[str, Any]: | 552 | +async def cache_flush(prefix: str = "", after_run: int = Query(0, ge=0, le=1)) -> dict[str, Any]: | ||
| 523 | − return {"flushed": await cache.cache_invalidate(prefix)} | 553 | + """Flush `aia:api:<prefix>*`. `after_run=1` flushes the read paths a connector run invalidates (the scheduler calls `services.cache.cache_invalidate` | ||
| 554 | + with these prefixes after every run — see docs/API.md).""" | ||||
| 555 | + if after_run: | ||||
| 556 | + flushed = {p: await cache.cache_invalidate(p) for p in CACHE_PREFIXES_AFTER_RUN} | ||||
| 557 | + return {"flushed": sum(flushed.values()), "by_prefix": flushed} | ||||
| 558 | + return {"flushed": await cache.cache_invalidate(prefix), "prefix": prefix or "*"} | ||||
| 524 | 559 | ||||
| 525 | 560 | ||||
| 526 | @router.post("/stats/recompute") | 561 | @router.post("/stats/recompute") | ||
added
src/aiatlas/api/routers/admin_quality.py
+474 −0
@@ -0,0 +1,474 @@ | |||||
| 1 | +"""Admin workbenches (API 1.1, `x-aia-admin-token`): data-health dashboard, entity resolution, anomalies, extraction debugger, quarantine, | ||||
| 2 | +audit log, run rollback. Every call is audited (router dependency shared with `admin.py`). Nothing here deletes data.""" | ||||
| 3 | +from __future__ import annotations | ||||
| 4 | + | ||||
| 5 | +import inspect | ||||
| 6 | +import re | ||||
| 7 | +from collections import defaultdict | ||||
| 8 | +from typing import Any | ||||
| 9 | + | ||||
| 10 | +from fastapi import APIRouter, Depends, Query, Request | ||||
| 11 | +from pydantic import BaseModel, Field | ||||
| 12 | + | ||||
| 13 | +from aiatlas.api.common import ADMIN_DEPENDENCIES, MAIN_ENTITY_TYPES, STATUS_VOCAB, ApiError, audit, client_ip | ||||
| 14 | +from aiatlas.api.routers.admin import SNAPSHOT_TEXT_LIMIT, audit_request | ||||
| 15 | +from aiatlas.db import connection, execute, fetch_all, fetch_one, fetch_val, jsonb, transaction | ||||
| 16 | +from aiatlas.ids import new_id | ||||
| 17 | +from aiatlas.ontology.licenses import normalize_license | ||||
| 18 | +from aiatlas.ontology.models import analyze_model_name, variant_key | ||||
| 19 | +from aiatlas.ontology.openness import OPENNESS_CATEGORIES | ||||
| 20 | +from aiatlas.sdk import archive | ||||
| 21 | +from aiatlas.services import cache | ||||
| 22 | +from aiatlas.services import merge as merge_service | ||||
| 23 | +from aiatlas.services.frontier import all_primary_groups, frontier_model_ids, rank_rows | ||||
| 24 | + | ||||
| 25 | +router = APIRouter(prefix="/api/v1/admin", tags=["admin-quality"], dependencies=[*ADMIN_DEPENDENCIES, Depends(audit_request)]) | ||||
| 26 | +SAMPLE = 8 | ||||
| 27 | +DECISIONS = ("merge", "alias", "variant_of", "family_member", "keep_separate", "defer") | ||||
| 28 | + | ||||
| 29 | + | ||||
| 30 | +async def _count_sample(conn: Any, sql_from_where: str, select: str, order: str = "1", **params: Any) -> dict[str, Any]: | ||||
| 31 | + count = await fetch_val(conn, f"select count(*) {sql_from_where}", **params) | ||||
| 32 | + sample = await fetch_all(conn, f"select {select} {sql_from_where} order by {order} limit {SAMPLE}", **params) | ||||
| 33 | + return {"count": int(count or 0), "sample": sample} | ||||
| 34 | + | ||||
| 35 | + | ||||
| 36 | +# ------------------------------------------------------------------------------------------------------------------ /admin/quality | ||||
| 37 | + | ||||
| 38 | + | ||||
| 39 | +@router.get("/quality") | ||||
| 40 | +async def quality_dashboard() -> dict[str, Any]: | ||||
| 41 | + out: dict[str, Any] = {} | ||||
| 42 | + ent = "e.id, e.slug, e.canonical_name as name, e.entity_type" | ||||
| 43 | + async with connection() as conn: | ||||
| 44 | + out["duplicate_candidates"] = { | ||||
| 45 | + "pending_decisions": await _count_sample(conn, "from resolution_decisions d where d.decision = 'defer' and not d.applied", "d.id, d.a_id, d.b_id, d.note, d.created_at", "d.created_at desc"), | ||||
| 46 | + "review_merge_candidates": await _count_sample(conn, "from review_queue r where r.kind = 'merge_candidate' and r.status = 'pending'", "r.id, r.entity_ids, r.reason, r.created_at", "r.created_at desc"), | ||||
| 47 | + } | ||||
| 48 | + tax_unmapped = await _count_sample(conn, "from taxonomy_mappings t where t.canonical is null", "t.domain, t.raw, t.count, t.last_seen_at", "t.count desc") | ||||
| 49 | + bad_open = await _count_sample(conn, "from entities e where e.entity_type = 'model' and e.merged_into is null and coalesce(e.attributes->>'openness', '') <> '' and e.attributes->>'openness' <> all(cast(:cats as text[]))", | ||||
| 50 | + f"{ent}, e.attributes->>'openness' as value", "e.updated_at desc", cats=list(OPENNESS_CATEGORIES)) | ||||
| 51 | + bad_status = await _count_sample(conn, "from entities e where e.entity_type = 'model' and e.merged_into is null and e.status <> all(cast(:st as text[]))", f"{ent}, e.status as value", "e.updated_at desc", | ||||
| 52 | + st=[*STATUS_VOCAB, "merged", "archived"]) | ||||
| 53 | + lic_rows = await fetch_all(conn, "select e.attributes->>'license' as raw, count(*) as n from entities e where e.entity_type = 'model' and e.merged_into is null and e.attributes ? 'license' and not e.attributes ? 'license_key' group by 1") | ||||
| 54 | + bad_lic = [{"raw": r["raw"], "models": int(r["n"])} for r in lic_rows if r["raw"] and normalize_license(r["raw"]) is None] | ||||
| 55 | + out["taxonomy_violations"] = {"unmapped_taxonomy_rows": tax_unmapped, "openness_unknown_vocab": bad_open, "status_unknown_vocab": bad_status, | ||||
| 56 | + "license_unclassified": {"count": sum(x["models"] for x in bad_lic), "sample": sorted(bad_lic, key=lambda x: -x["models"])[:SAMPLE]}} | ||||
| 57 | + by_check = await fetch_all(conn, "select check_name, severity, count(*) as n from anomalies where status = 'open' group by 1, 2 order by 3 desc") | ||||
| 58 | + out["impossible_values"] = {"count": sum(int(r["n"]) for r in by_check), "by_check": [{**r, "n": int(r["n"])} for r in by_check], | ||||
| 59 | + "sample": await fetch_all(conn, "select a.id, a.check_name, a.severity, a.message, a.entity_id, e.slug, a.value, a.last_seen_at from anomalies a left join entities e on e.id = a.entity_id where a.status = 'open' order by case a.severity when 'critical' then 0 when 'warning' then 1 else 2 end, a.last_seen_at desc limit 12")} | ||||
| 60 | + out["conflicting_t1_claims"] = await _count_sample(conn, "from claims c join entities e on e.id = c.entity_id where c.status = 'conflicting' and c.tier = 1", | ||||
| 61 | + f"c.id as claim_id, c.property, c.value, c.source_url, {ent}", "c.observed_at desc") | ||||
| 62 | + base_models = "from entities e where e.entity_type = 'model' and e.merged_into is null" | ||||
| 63 | + out["models_without_organization"] = await _count_sample(conn, f"{base_models} and e.organization_id is null", ent, "e.updated_at desc") | ||||
| 64 | + out["models_without_release_source"] = await _count_sample(conn, f"{base_models} and not exists (select 1 from claims c where c.entity_id = e.id and c.property = 'release_date' and c.status = 'current')", ent, "e.updated_at desc") | ||||
| 65 | + out["models_without_parameters"] = await _count_sample(conn, f"{base_models} and not e.attributes ? 'parameter_count'", ent, "e.updated_at desc") | ||||
| 66 | + out["orphan_benchmark_results"] = await _count_sample(conn, "from benchmark_results r join entities m on m.id = r.model_id where r.valid_to is null and (m.merged_into is not null or m.entity_type <> 'model')", | ||||
| 67 | + "r.id as result_id, r.benchmark_id, m.id, m.slug, m.canonical_name as name, m.entity_type, m.merged_into", "r.observed_at desc") | ||||
| 68 | + out["benchmarks_without_results"] = await _count_sample(conn, "from entities e where e.entity_type = 'benchmark' and e.merged_into is null and not exists (select 1 from benchmark_results r where r.benchmark_id = e.id and r.valid_to is null)", | ||||
| 69 | + ent, "e.canonical_name") | ||||
| 70 | + out["unresolved_provider_deployments"] = await _count_sample(conn, """from prices p join entities m on m.id = p.model_id join entities pv on pv.id = p.provider_id where p.valid_to is null and p.provider_model_id is not null | ||||
| 71 | + and not exists (select 1 from entity_identifiers i where i.entity_id = p.model_id and i.value = p.provider_model_id)""", | ||||
| 72 | + "p.id as price_id, p.provider_model_id, m.slug as model, pv.slug as provider", "p.observed_at desc") | ||||
| 73 | + names = await fetch_all(conn, "select e.id, e.slug, e.canonical_name, e.attributes->>'hf_repo' as hf_repo, e.attributes->>'is_quantized' as is_q, e.attributes->>'quant_format' as fmt from entities e where e.entity_type = 'model' and e.merged_into is null") | ||||
| 74 | + quants = [] | ||||
| 75 | + for r in names: | ||||
| 76 | + a = analyze_model_name(r["hf_repo"] or r["canonical_name"]) | ||||
| 77 | + if a.is_artifact or r["is_q"] == "true": | ||||
| 78 | + quants.append({"id": r["id"], "slug": r["slug"], "name": r["canonical_name"], "hf_repo": r["hf_repo"], "quant_formats": a.quant_formats or ([r["fmt"]] if r["fmt"] else []), | ||||
| 79 | + "kind": "quantization" if (a.is_quantized or r["is_q"] == "true") else "conversion"}) | ||||
| 80 | + out["quantisations_typed_as_models"] = {"count": len(quants), "sample": quants[:SAMPLE]} | ||||
| 81 | + out["stale_sources"] = await _count_sample(conn, "from connectors c where c.enabled and c.last_success_at is not null and c.last_success_at < now() - make_interval(secs => 3 * c.interval_seconds)", | ||||
| 82 | + "c.name, c.health, c.last_success_at, c.interval_seconds, c.consecutive_failures", "c.last_success_at") | ||||
| 83 | + counts = {r["entity_type"]: int(r["n"]) for r in await fetch_all(conn, "select entity_type, count(*) as n from entities where merged_into is null group by 1")} | ||||
| 84 | + out["empty_public_categories"] = {"count": sum(1 for t in MAIN_ENTITY_TYPES if not counts.get(t)), "sample": [t for t in MAIN_ENTITY_TYPES if not counts.get(t)]} | ||||
| 85 | + out["quarantined_runs_pending"] = await _count_sample(conn, "from quarantined_runs q where q.status = 'pending'", "q.id, q.run_id, q.connector_name, q.reason, q.stats, q.created_at", "q.created_at desc") | ||||
| 86 | + out["review_queue_priority"] = await _review_priority(conn) | ||||
| 87 | + out["note"] = "Counts are live. Nothing is deleted by these checks; they point at review actions (/admin/entity-resolution, /admin/anomalies, /admin/quarantine)." | ||||
| 88 | + return out | ||||
| 89 | + | ||||
| 90 | + | ||||
| 91 | +async def _review_priority(conn: Any) -> list[dict[str, Any]]: | ||||
| 92 | + """Homepage-visible items first: frontier models, largest params/context claims, price anomalies, benchmark leaders, major orgs, duplicate canonical models.""" | ||||
| 93 | + frontier, _ = await frontier_model_ids(conn) | ||||
| 94 | + groups = await all_primary_groups(conn) | ||||
| 95 | + leaders = {rank_rows(g["rows"], g["higher_is_better"])[0]["model_id"] for g in groups.values() if g["rows"]} | ||||
| 96 | + big = {r["id"] for r in await fetch_all(conn, """select id from entities where entity_type = 'model' and merged_into is null and attributes->>'parameter_count' ~ '^[0-9.]+$' | ||||
| 97 | + order by (attributes->>'parameter_count')::double precision desc limit 20""")} | ||||
| 98 | + big |= {r["id"] for r in await fetch_all(conn, """select id from entities where entity_type = 'model' and merged_into is null and attributes->>'context_length' ~ '^[0-9]+$' | ||||
| 99 | + order by (attributes->>'context_length')::bigint desc limit 20""")} | ||||
| 100 | + major_orgs = {r["organization_id"] for r in await fetch_all(conn, "select organization_id from entities where entity_type = 'model' and merged_into is null and organization_id is not null group by 1 having count(*) >= 10")} | ||||
| 101 | + anomalies = await fetch_all(conn, """select a.id, a.check_name, a.severity, a.message, a.entity_id, e.slug, e.organization_id, e.entity_type from anomalies a left join entities e on e.id = a.entity_id | ||||
| 102 | + where a.status = 'open' order by a.last_seen_at desc limit 2000""") | ||||
| 103 | + merges = await fetch_all(conn, "select r.id, r.entity_ids, r.reason from review_queue r where r.kind = 'merge_candidate' and r.status = 'pending' limit 500") | ||||
| 104 | + items = [] | ||||
| 105 | + for a in anomalies: | ||||
| 106 | + reasons = [] | ||||
| 107 | + eid = a["entity_id"] | ||||
| 108 | + if eid in frontier: | ||||
| 109 | + reasons.append("frontier model") | ||||
| 110 | + if eid in leaders: | ||||
| 111 | + reasons.append("benchmark leader") | ||||
| 112 | + if eid in big: | ||||
| 113 | + reasons.append("among the largest params/context claims") | ||||
| 114 | + if a["check_name"] in ("negative_price", "price_too_high", "zero_output_price", "price_jump_100x", "cached_gt_input_price"): | ||||
| 115 | + reasons.append("price anomaly") | ||||
| 116 | + if a["organization_id"] in major_orgs: | ||||
| 117 | + reasons.append("major organization") | ||||
| 118 | + if reasons or a["severity"] == "critical": | ||||
| 119 | + items.append({"kind": "anomaly", "id": a["id"], "check": a["check_name"], "severity": a["severity"], "message": a["message"], "entity_id": eid, "slug": a["slug"], "reasons": reasons or ["critical severity"], | ||||
| 120 | + "_p": (0 if reasons else 1, {"critical": 0, "warning": 1}.get(a["severity"], 2))}) | ||||
| 121 | + for m in merges: | ||||
| 122 | + ids = list(m["entity_ids"] or []) | ||||
| 123 | + hits = [i for i in ids if i in frontier or i in leaders] | ||||
| 124 | + if hits: | ||||
| 125 | + items.append({"kind": "merge_candidate", "id": m["id"], "entity_ids": ids, "reason": m["reason"], "reasons": ["duplicate involving a frontier model / benchmark leader"], "_p": (0, 0)}) | ||||
| 126 | + items.sort(key=lambda x: x["_p"]) | ||||
| 127 | + for x in items: | ||||
| 128 | + x.pop("_p") | ||||
| 129 | + return items[:50] | ||||
| 130 | + | ||||
| 131 | + | ||||
| 132 | +# ------------------------------------------------------------------------------------------------------------------ /admin/entity-resolution | ||||
| 133 | + | ||||
| 134 | + | ||||
| 135 | +async def _side(conn: Any, eid: str) -> dict[str, Any] | None: | ||||
| 136 | + row = await fetch_one(conn, """select e.id, e.slug, e.canonical_name, e.entity_type, e.attributes, e.first_seen_at, e.merged_into, e.family_id, e.canonical_id, e.identity_confidence, | ||||
| 137 | + o.slug as org_slug, o.canonical_name as org_name, f.canonical_name as family_name, | ||||
| 138 | + (select count(*) from relations r where (r.subject_id = e.id or r.object_id = e.id) and r.valid_to is null) as relations, | ||||
| 139 | + (select count(distinct c.source_id) from claims c where c.entity_id = e.id and c.status = 'current') as sources, | ||||
| 140 | + (select count(*) from claims c where c.entity_id = e.id and c.status = 'current') as claims, | ||||
| 141 | + (select count(*) from prices p where p.model_id = e.id and p.valid_to is null) as prices, | ||||
| 142 | + (select count(*) from benchmark_results r where r.model_id = e.id and r.valid_to is null) as results, | ||||
| 143 | + (select jsonb_agg(jsonb_build_object('scheme', i.scheme, 'value', i.value) order by i.scheme) from entity_identifiers i where i.entity_id = e.id) as identifiers, | ||||
| 144 | + (select jsonb_agg(a.alias order by a.alias) from entity_aliases a where a.entity_id = e.id) as aliases | ||||
| 145 | + from entities e left join entities o on o.id = e.organization_id left join entities f on f.id = e.family_id where e.id = :id or e.slug = :id limit 1""", id=eid) | ||||
| 146 | + if not row: | ||||
| 147 | + return None | ||||
| 148 | + a = row["attributes"] or {} | ||||
| 149 | + return {"id": row["id"], "slug": row["slug"], "name": row["canonical_name"], "entity_type": row["entity_type"], "organization": row["org_name"], "organization_slug": row["org_slug"], | ||||
| 150 | + "family": row["family_name"] or a.get("family"), "parameter_count": a.get("parameter_count"), "active_parameter_count": a.get("active_parameter_count"), "release_date": a.get("release_date"), | ||||
| 151 | + "architecture": a.get("architecture"), "model_type": a.get("model_type"), "hf_repo": a.get("hf_repo"), "openness": a.get("openness"), "context_length": a.get("context_length"), | ||||
| 152 | + "identifiers": row["identifiers"] or [], "aliases": row["aliases"] or [], "relations": int(row["relations"] or 0), "sources": int(row["sources"] or 0), "claims": int(row["claims"] or 0), | ||||
| 153 | + "prices": int(row["prices"] or 0), "results": int(row["results"] or 0), "first_seen_at": row["first_seen_at"], "merged_into": row["merged_into"], "canonical_id": row["canonical_id"], | ||||
| 154 | + "identity_confidence": row["identity_confidence"], "variant_key": variant_key(a.get("hf_repo") or row["canonical_name"]), "name_analysis": _analysis(a.get("hf_repo") or row["canonical_name"])} | ||||
| 155 | + | ||||
| 156 | + | ||||
| 157 | +def _analysis(name: str) -> dict[str, Any]: | ||||
| 158 | + a = analyze_model_name(name) | ||||
| 159 | + return {"base_key": a.base_key, "quant_formats": a.quant_formats, "precision": a.precision, "is_artifact": a.is_artifact, "effort": a.effort, "family_hint": a.family_hint, | ||||
| 160 | + "parameter_count": a.parameter_count, "snapshot_date": a.snapshot_date} | ||||
| 161 | + | ||||
| 162 | + | ||||
| 163 | +@router.get("/entity-resolution") | ||||
| 164 | +async def entity_resolution(type: str | None = Query("model", alias="type"), status: str = Query("pending"), limit: int = Query(50, ge=1, le=200), threshold: float = Query(0.8, ge=0.3, le=1.0)) -> dict[str, Any]: | ||||
| 165 | + """Candidate pairs: review merge_candidates + trigram similarity + same `variant_key` (ontology.models) — side by side.""" | ||||
| 166 | + pairs: dict[tuple[str, str], dict[str, Any]] = {} | ||||
| 167 | + async with connection() as conn: | ||||
| 168 | + decided = {(d["a_id"], d["b_id"]): d for d in await fetch_all(conn, "select a_id, b_id, decision, applied, created_at, note from resolution_decisions")} | ||||
| 169 | + if status in ("pending", "all"): | ||||
| 170 | + for r in await fetch_all(conn, "select r.id, r.entity_ids, r.reason, r.payload from review_queue r where r.kind = 'merge_candidate' and r.status = 'pending' order by r.created_at desc limit :lim", lim=limit * 2): | ||||
| 171 | + ids = list(r["entity_ids"] or []) | ||||
| 172 | + if len(ids) >= 2: | ||||
| 173 | + pairs.setdefault(tuple(sorted(ids[:2])), {"sources": [], "similarity": None})["sources"].append({"kind": "review_merge_candidate", "review_id": r["id"], "reason": r["reason"]}) | ||||
| 174 | + where = "a.entity_type = :t" if type else "a.entity_type in ('model','company','organization','lab','provider','benchmark','researcher')" | ||||
| 175 | + for r in await fetch_all(conn, f"""select a.id as a_id, b.id as b_id, similarity(a.canonical_name, b.canonical_name) as sim from entities a join entities b | ||||
| 176 | + on b.entity_type = a.entity_type and b.id > a.id and a.canonical_name % b.canonical_name | ||||
| 177 | + where {where} and a.merged_into is null and b.merged_into is null and similarity(a.canonical_name, b.canonical_name) > :th order by sim desc limit :lim""", | ||||
| 178 | + t=type, th=threshold, lim=limit * 2): | ||||
| 179 | + p = pairs.setdefault(tuple(sorted((r["a_id"], r["b_id"]))), {"sources": [], "similarity": None}) | ||||
| 180 | + p["similarity"] = round(float(r["sim"]), 3) | ||||
| 181 | + p["sources"].append({"kind": "trigram_similarity", "similarity": round(float(r["sim"]), 3)}) | ||||
| 182 | + if type in (None, "model"): | ||||
| 183 | + names = await fetch_all(conn, "select id, canonical_name, attributes->>'hf_repo' as hf from entities where entity_type = 'model' and merged_into is null") | ||||
| 184 | + by_key: dict[str, list[str]] = defaultdict(list) | ||||
| 185 | + for n in names: | ||||
| 186 | + k = variant_key(n["hf"] or n["canonical_name"]) | ||||
| 187 | + if k: | ||||
| 188 | + by_key[k].append(n["id"]) | ||||
| 189 | + for k, ids in by_key.items(): | ||||
| 190 | + if 2 <= len(ids) <= 6: | ||||
| 191 | + for i in range(len(ids)): | ||||
| 192 | + for j in range(i + 1, len(ids)): | ||||
| 193 | + pairs.setdefault(tuple(sorted((ids[i], ids[j]))), {"sources": [], "similarity": None})["sources"].append({"kind": "same_variant_key", "variant_key": k}) | ||||
| 194 | + items = [] | ||||
| 195 | + for (a, b), p in pairs.items(): | ||||
| 196 | + d = decided.get((a, b)) or decided.get((b, a)) | ||||
| 197 | + if status == "pending" and d and d["decision"] != "defer": | ||||
| 198 | + continue | ||||
| 199 | + if status == "decided" and not d: | ||||
| 200 | + continue | ||||
| 201 | + sa, sb = await _side(conn, a), await _side(conn, b) | ||||
| 202 | + if not sa or not sb: | ||||
| 203 | + continue | ||||
| 204 | + items.append({"a": sa, "b": sb, "signals": p["sources"], "similarity": p["similarity"], "same_variant_key": sa["variant_key"] == sb["variant_key"] and bool(sa["variant_key"]), | ||||
| 205 | + "same_organization": sa["organization_slug"] == sb["organization_slug"] and sa["organization_slug"] is not None, "decision": d, | ||||
| 206 | + "hint": ("artifact of the other" if sa["name_analysis"]["is_artifact"] != sb["name_analysis"]["is_artifact"] else | ||||
| 207 | + "effort variant" if (sa["name_analysis"]["effort"] or sb["name_analysis"]["effort"]) else "possible duplicate")}) | ||||
| 208 | + if len(items) >= limit: | ||||
| 209 | + break | ||||
| 210 | + items.sort(key=lambda x: (-len(x["signals"]), -(x["similarity"] or 0))) | ||||
| 211 | + return {"items": items, "total": len(items), "threshold": threshold, "status": status, "decisions": list(DECISIONS), | ||||
| 212 | + "note": "Signals are independent (review queue, trigram similarity, shared variant_key); nothing is merged until POST /admin/entity-resolution/{a}/{b}."} | ||||
| 213 | + | ||||
| 214 | + | ||||
| 215 | +class ResolutionBody(BaseModel): | ||||
| 216 | + decision: str = Field(..., pattern="^(merge|alias|variant_of|family_member|keep_separate|defer)$") | ||||
| 217 | + note: str | None = Field(None, max_length=2000) | ||||
| 218 | + | ||||
| 219 | + | ||||
| 220 | +@router.post("/entity-resolution/{a}/{b}") | ||||
| 221 | +async def resolve_pair(a: str, b: str, body: ResolutionBody, request: Request) -> dict[str, Any]: | ||||
| 222 | + """Persist the decision, then apply it (merge_entities with `mode=` when the service supports it). `a` is folded INTO `b`.""" | ||||
| 223 | + applied = False | ||||
| 224 | + effect: dict[str, Any] | None = None | ||||
| 225 | + async with transaction() as conn: | ||||
| 226 | + ea = await fetch_one(conn, "select id, entity_type, canonical_name, merged_into from entities where id = :k or slug = :k limit 1", k=a) | ||||
| 227 | + eb = await fetch_one(conn, "select id, entity_type, canonical_name, merged_into from entities where id = :k or slug = :k limit 1", k=b) | ||||
| 228 | + if not ea or not eb: | ||||
| 229 | + raise ApiError(404, "entity a or b not found") | ||||
| 230 | + if ea["id"] == eb["id"]: | ||||
| 231 | + raise ApiError(400, "a and b are the same entity") | ||||
| 232 | + did = new_id("review") | ||||
| 233 | + sig = inspect.signature(merge_service.merge_entities) | ||||
| 234 | + supports_mode = "mode" in sig.parameters | ||||
| 235 | + try: | ||||
| 236 | + if body.decision == "merge": | ||||
| 237 | + effect = await (merge_service.merge_entities(conn, ea["id"], eb["id"], mode="merge") if supports_mode else merge_service.merge_entities(conn, ea["id"], eb["id"])) | ||||
| 238 | + applied = True | ||||
| 239 | + elif body.decision == "variant_of": | ||||
| 240 | + if supports_mode: | ||||
| 241 | + effect = await merge_service.merge_entities(conn, ea["id"], eb["id"], mode="variant_of") | ||||
| 242 | + else: | ||||
| 243 | + await execute(conn, "update entities set canonical_id = :b where id = :a", a=ea["id"], b=eb["id"]) | ||||
| 244 | + effect = await merge_service.merge_entities(conn, ea["id"], eb["id"]) | ||||
| 245 | + effect["fallback"] = "canonical_id set then folded with merge_entities (service has no mode= yet)" | ||||
| 246 | + applied = True | ||||
| 247 | + elif body.decision == "alias": | ||||
| 248 | + if supports_mode: | ||||
| 249 | + effect = await merge_service.merge_entities(conn, ea["id"], eb["id"], mode="alias") | ||||
| 250 | + else: | ||||
| 251 | + from aiatlas.ids import normalize_alias | ||||
| 252 | + | ||||
| 253 | + await execute(conn, "insert into entity_aliases (entity_id, alias, alias_norm, kind) values (:t, :al, :n, 'alias') on conflict (entity_id, alias_norm) do nothing", | ||||
| 254 | + t=eb["id"], al=ea["canonical_name"], n=normalize_alias(ea["canonical_name"])) | ||||
| 255 | + effect = {"alias_added": ea["canonical_name"], "to": eb["id"], "fallback": "alias row only (service has no mode= yet); entities kept separate"} | ||||
| 256 | + applied = True | ||||
| 257 | + elif body.decision == "family_member": | ||||
| 258 | + if supports_mode: | ||||
| 259 | + effect = await merge_service.merge_entities(conn, ea["id"], eb["id"], mode="family_member") | ||||
| 260 | + applied = True | ||||
| 261 | + elif eb["entity_type"] == "model_family": | ||||
| 262 | + await execute(conn, "update entities set family_id = :b, updated_at = now() where id = :a", a=ea["id"], b=eb["id"]) | ||||
| 263 | + effect = {"family_id": eb["id"]} | ||||
| 264 | + applied = True | ||||
| 265 | + else: | ||||
| 266 | + raise ApiError(400, "family_member requires b to be a model_family entity") | ||||
| 267 | + except (ValueError, LookupError) as exc: | ||||
| 268 | + raise ApiError(400, str(exc)) from exc | ||||
| 269 | + await execute(conn, """insert into resolution_decisions (id, a_id, b_id, decision, actor, note, payload, applied) values (:id, :a, :b, :d, 'admin', :n, cast(:p as jsonb), :ap) | ||||
| 270 | + on conflict (a_id, b_id, decision) do update set note = excluded.note, payload = excluded.payload, applied = excluded.applied, created_at = now()""", | ||||
| 271 | + id=did, a=ea["id"], b=eb["id"], d=body.decision, n=body.note, p=jsonb({"effect": effect, "mode_supported": supports_mode}), ap=applied) | ||||
| 272 | + if applied: | ||||
| 273 | + await execute(conn, "update review_queue set status = 'approved', resolved_at = now(), resolution = cast(:r as jsonb) where kind = 'merge_candidate' and status = 'pending' and entity_ids @> cast(:ids as text[])", | ||||
| 274 | + r=jsonb({"via": "entity-resolution", "decision": body.decision}), ids=[ea["id"], eb["id"]]) | ||||
| 275 | + if applied: | ||||
| 276 | + await cache.cache_invalidate() | ||||
| 277 | + await audit("entity-resolution", f"{ea['id']}→{eb['id']}", {"decision": body.decision, "applied": applied, "effect": effect}, client_ip(request)) | ||||
| 278 | + return {"ok": True, "a": ea["id"], "b": eb["id"], "decision": body.decision, "applied": applied, "effect": effect} | ||||
| 279 | + | ||||
| 280 | + | ||||
| 281 | +# ------------------------------------------------------------------------------------------------------------------ /admin/anomalies | ||||
| 282 | + | ||||
| 283 | + | ||||
| 284 | +@router.get("/anomalies") | ||||
| 285 | +async def anomalies(status: str = Query("open"), severity: str | None = None, check: str | None = None, limit: int = Query(100, ge=1, le=1000), offset: int = Query(0, ge=0)) -> dict[str, Any]: | ||||
| 286 | + where = ["true"] if status == "all" else ["a.status = :st"] | ||||
| 287 | + params: dict[str, Any] = {"st": status, "lim": limit, "off": offset} | ||||
| 288 | + if severity: | ||||
| 289 | + where.append("a.severity = :sev") | ||||
| 290 | + params["sev"] = severity | ||||
| 291 | + if check: | ||||
| 292 | + where.append("a.check_name = :chk") | ||||
| 293 | + params["chk"] = check | ||||
| 294 | + w = " and ".join(where) | ||||
| 295 | + async with connection() as conn: | ||||
| 296 | + rows = await fetch_all(conn, f"""select a.*, e.slug, e.canonical_name as entity_name, e.entity_type from anomalies a left join entities e on e.id = a.entity_id where {w} | ||||
| 297 | + order by case a.severity when 'critical' then 0 when 'warning' then 1 else 2 end, a.last_seen_at desc limit :lim offset :off""", **params) | ||||
| 298 | + total = await fetch_val(conn, f"select count(*) from anomalies a where {w}", **{k: v for k, v in params.items() if k not in ("lim", "off")}) | ||||
| 299 | + by = await fetch_all(conn, "select check_name, severity, status, count(*) as n from anomalies group by 1, 2, 3 order by 4 desc") | ||||
| 300 | + return {"items": rows, "total": int(total or 0), "limit": limit, "offset": offset, "by_check": [{**r, "n": int(r["n"])} for r in by]} | ||||
| 301 | + | ||||
| 302 | + | ||||
| 303 | +class AnomalyBody(BaseModel): | ||||
| 304 | + status: str = Field(..., pattern="^(resolved|ignored|open)$") | ||||
| 305 | + note: str | None = Field(None, max_length=2000) | ||||
| 306 | + | ||||
| 307 | + | ||||
| 308 | +@router.post("/anomalies/{anomaly_id}") | ||||
| 309 | +async def anomaly_action(anomaly_id: str, body: AnomalyBody) -> dict[str, Any]: | ||||
| 310 | + async with transaction() as conn: | ||||
| 311 | + row = await fetch_one(conn, "update anomalies set status = :st, resolution = :n, resolved_at = case when :st = 'open' then null else now() end where id = :id returning id, check_name, status", | ||||
| 312 | + st=body.status, n=body.note, id=anomaly_id) | ||||
| 313 | + if not row: | ||||
| 314 | + raise ApiError(404, "anomaly not found") | ||||
| 315 | + return {"ok": True, **row} | ||||
| 316 | + | ||||
| 317 | + | ||||
| 318 | +# ------------------------------------------------------------------------------------------------------------------ /admin/extractions/{snapshot_id} | ||||
| 319 | + | ||||
| 320 | +_NUM_FORMATS = (lambda v: str(v), lambda v: f"{int(v):,}" if float(v).is_integer() else f"{v:,}", lambda v: f"{int(v)}" if float(v).is_integer() else f"{v}", | ||||
| 321 | + lambda v: f"{v / 1e9:g}B" if abs(v) >= 1e9 else f"{v / 1e6:g}M" if abs(v) >= 1e6 else f"{v / 1e3:g}K" if abs(v) >= 1e3 else str(v), | ||||
| 322 | + lambda v: f"{v / 1e9:.1f}B" if abs(v) >= 1e9 else f"{v / 1e6:.1f}M" if abs(v) >= 1e6 else f"{v / 1e3:.0f}K" if abs(v) >= 1e3 else str(v), | ||||
| 323 | + lambda v: f"{v / 1e3:g}k" if abs(v) >= 1e3 else str(v), lambda v: f"${v:g}", lambda v: f"${v:.2f}") | ||||
| 324 | + | ||||
| 325 | + | ||||
| 326 | +def _locate(text: str, value: Any) -> dict[str, Any]: | ||||
| 327 | + cands: list[str] = [] | ||||
| 328 | + if isinstance(value, bool): | ||||
| 329 | + cands = [str(value).lower()] | ||||
| 330 | + elif isinstance(value, (int, float)): | ||||
| 331 | + for f in _NUM_FORMATS: | ||||
| 332 | + try: | ||||
| 333 | + cands.append(f(float(value))) | ||||
| 334 | + except Exception: # noqa: BLE001 | ||||
| 335 | + pass | ||||
| 336 | + elif isinstance(value, str) and value.strip(): | ||||
| 337 | + cands = [value.strip()] | ||||
| 338 | + for c in dict.fromkeys(cands): | ||||
| 339 | + m = re.search(re.escape(c), text, flags=re.IGNORECASE) | ||||
| 340 | + if m: | ||||
| 341 | + s, e = max(0, m.start() - 80), min(len(text), m.end() + 80) | ||||
| 342 | + return {"found": True, "offset": m.start(), "match": text[m.start():m.end()], "context": text[s:e]} | ||||
| 343 | + return {"found": False, "tried": cands[:6]} | ||||
| 344 | + | ||||
| 345 | + | ||||
| 346 | +@router.get("/extractions/{snap_id}") | ||||
| 347 | +async def extraction_debugger(snap_id: str, text_limit: int = Query(SNAPSHOT_TEXT_LIMIT, ge=0, le=SNAPSHOT_TEXT_LIMIT)) -> dict[str, Any]: | ||||
| 348 | + async with connection() as conn: | ||||
| 349 | + snap = await fetch_one(conn, """select s.id, s.document_id, s.run_id, s.url, s.final_url, s.observed_at, s.http_status, s.content_type, s.content_hash, s.byte_size, s.text_hash, s.structured, s.diff, | ||||
| 350 | + s.parser_version, s.connector_version, s.transport, s.changed, s.processing_status, s.raw_path is not null as has_raw, s.text_path, d.url as document_url, | ||||
| 351 | + d.doc_type, d.connector_name, d.entity_id, d.title, e.slug as entity_slug, e.canonical_name as entity_name | ||||
| 352 | + from snapshots s join documents d on d.id = s.document_id left join entities e on e.id = d.entity_id where s.id = :id""", id=snap_id) | ||||
| 353 | + if not snap: | ||||
| 354 | + raise ApiError(404, "snapshot not found") | ||||
| 355 | + prev = await fetch_one(conn, "select id, observed_at, content_hash from snapshots where document_id = :d and observed_at < :t and changed order by observed_at desc limit 1", d=snap["document_id"], t=snap["observed_at"]) | ||||
| 356 | + claims = await fetch_all(conn, "select c.id, c.entity_id, e.slug as entity_slug, c.property, c.value, c.value_raw, c.unit, c.status, c.confidence, c.extractor, c.tier, c.observed_at from claims c left join entities e on e.id = c.entity_id where c.snapshot_id = :id order by e.slug, c.property limit 1000", id=snap_id) | ||||
| 357 | + relations = await fetch_all(conn, "select r.id, r.subject_id, r.predicate, r.object_id, r.attributes, r.valid_to from relations r where r.snapshot_id = :id limit 500", id=snap_id) | ||||
| 358 | + results = await fetch_all(conn, "select r.id, r.model_id, r.benchmark_id, r.score, r.metric, r.config, r.config_key, r.trust_level, r.valid_to from benchmark_results r where r.snapshot_id = :id limit 500", id=snap_id) | ||||
| 359 | + prices = await fetch_all(conn, "select p.id, p.model_id, p.provider_id, p.provider_model_id, p.input_per_mtok, p.output_per_mtok, p.valid_from, p.valid_to from prices p where p.snapshot_id = :id limit 500", id=snap_id) | ||||
| 360 | + events = await fetch_all(conn, "select id, entity_id, event_type, category, property, summary, importance, occurred_at, is_backfill, group_key from change_events where snapshot_id = :id order by occurred_at desc limit 300", id=snap_id) | ||||
| 361 | + llm = await fetch_all(conn, "select id, task_type, stage, model, status, input_tokens, output_tokens, duration_ms, error, created_at from llm_jobs where snapshot_id = :id order by created_at desc limit 50", id=snap_id) | ||||
| 362 | + touched = {c["entity_id"] for c in claims} | {r["subject_id"] for r in relations} | {r["object_id"] for r in relations} | {r["model_id"] for r in results} | {p["model_id"] for p in prices} | ||||
| 363 | + touched.discard(None) | ||||
| 364 | + cands = await fetch_all(conn, "select id, slug, canonical_name, entity_type, merged_into, identity_confidence from entities where id = any(cast(:ids as text[])) order by canonical_name", ids=list(touched)) if touched else [] | ||||
| 365 | + text, text_error, text_len = None, None, 0 | ||||
| 366 | + if snap.get("text_path"): | ||||
| 367 | + try: | ||||
| 368 | + full = archive.load_text(snap["text_path"]) | ||||
| 369 | + text_len = len(full) | ||||
| 370 | + text = full[:text_limit] | ||||
| 371 | + except OSError as exc: | ||||
| 372 | + text_error = f"cleaned text unavailable ({exc.__class__.__name__})" | ||||
| 373 | + full = "" | ||||
| 374 | + else: | ||||
| 375 | + full = "" | ||||
| 376 | + spans = [] | ||||
| 377 | + for c in claims: | ||||
| 378 | + v = c["value"] | ||||
| 379 | + if isinstance(v, (int, float, str, bool)) and not (isinstance(v, str) and len(v) > 200): | ||||
| 380 | + spans.append({"claim_id": c["id"], "property": c["property"], "value": v, **(_locate(full, v) if full else {"found": False, "reason": "no cleaned text"})}) | ||||
| 381 | + out = {k: v for k, v in snap.items() if k != "text_path"} | ||||
| 382 | + out.update({"has_text": bool(snap.get("text_path")), "text": text, "text_chars": text_len, "text_truncated": text_len > text_limit, "text_error": text_error, | ||||
| 383 | + "previous_snapshot": prev, "diff": snap.get("diff"), "claims": claims, "relations": relations, "results": results, "prices": prices, "events": events, "llm_jobs": llm, | ||||
| 384 | + "entity_candidates": cands, "spans": spans, "spans_found": sum(1 for s in spans if s.get("found")), | ||||
| 385 | + "note": "spans are a best-effort textual search of each claim value in the cleaned text (several number formats tried); not-found is reported honestly, never inferred."}) | ||||
| 386 | + return out | ||||
| 387 | + | ||||
| 388 | + | ||||
| 389 | +# ------------------------------------------------------------------------------------------------------------------ /admin/quarantine | ||||
| 390 | + | ||||
| 391 | + | ||||
| 392 | +@router.get("/quarantine") | ||||
| 393 | +async def quarantine(status: str = Query("pending"), limit: int = Query(50, ge=1, le=500), offset: int = Query(0, ge=0)) -> dict[str, Any]: | ||||
| 394 | + where = "true" if status == "all" else "q.status = :st" | ||||
| 395 | + async with connection() as conn: | ||||
| 396 | + rows = await fetch_all(conn, f"select q.id, q.run_id, q.connector_name, q.reason, q.stats, q.status, q.created_at, q.resolved_at, q.resolved_by, jsonb_array_length(q.facts) as facts from quarantined_runs q where {where} order by q.created_at desc limit :lim offset :off", | ||||
| 397 | + st=status, lim=limit, off=offset) | ||||
| 398 | + total = await fetch_val(conn, f"select count(*) from quarantined_runs q where {where}", st=status) | ||||
| 399 | + return {"items": rows, "total": int(total or 0), "limit": limit, "offset": offset} | ||||
| 400 | + | ||||
| 401 | + | ||||
| 402 | +class QuarantineBody(BaseModel): | ||||
| 403 | + action: str = Field(..., pattern="^(release|discard)$") | ||||
| 404 | + note: str | None = Field(None, max_length=2000) | ||||
| 405 | + | ||||
| 406 | + | ||||
| 407 | +@router.post("/quarantine/{quarantine_id}") | ||||
| 408 | +async def quarantine_action(quarantine_id: str, body: QuarantineBody) -> dict[str, Any]: | ||||
| 409 | + try: | ||||
| 410 | + from aiatlas.services import canonical | ||||
| 411 | + except ImportError: | ||||
| 412 | + canonical = None # type: ignore[assignment] | ||||
| 413 | + fn = getattr(canonical, f"{body.action}_quarantine", None) if canonical else None | ||||
| 414 | + if fn is None: | ||||
| 415 | + raise ApiError(501, f"quarantine {body.action} is not available yet: services.canonical.{body.action}_quarantine is missing (Stream A)") | ||||
| 416 | + async with transaction() as conn: | ||||
| 417 | + row = await fetch_one(conn, "select * from quarantined_runs where id = :id for update", id=quarantine_id) | ||||
| 418 | + if not row: | ||||
| 419 | + raise ApiError(404, "quarantined run not found") | ||||
| 420 | + if row["status"] != "pending": | ||||
| 421 | + raise ApiError(409, f"quarantined run already {row['status']}") | ||||
| 422 | + result = await fn(conn, quarantine_id) if inspect.iscoroutinefunction(fn) else fn(conn, quarantine_id) | ||||
| 423 | + await execute(conn, "update quarantined_runs set status = :st, resolved_at = now(), resolved_by = 'admin' where id = :id and status = 'pending'", | ||||
| 424 | + st="released" if body.action == "release" else "discarded", id=quarantine_id) | ||||
| 425 | + await cache.cache_invalidate() | ||||
| 426 | + return {"ok": True, "id": quarantine_id, "action": body.action, "result": result} | ||||
| 427 | + | ||||
| 428 | + | ||||
| 429 | +# ------------------------------------------------------------------------------------------------------------------ /admin/audit · rollback | ||||
| 430 | + | ||||
| 431 | + | ||||
| 432 | +@router.get("/audit") | ||||
| 433 | +async def audit_log(limit: int = Query(100, ge=1, le=1000), offset: int = Query(0, ge=0), action: str | None = None) -> dict[str, Any]: | ||||
| 434 | + where = "action ilike :a" if action else "true" | ||||
| 435 | + async with connection() as conn: | ||||
| 436 | + rows = await fetch_all(conn, f"select id, actor, action, target, payload, ip, created_at from admin_audit_log where {where} order by id desc limit :lim offset :off", a=f"%{action}%" if action else None, lim=limit, off=offset) | ||||
| 437 | + total = await fetch_val(conn, f"select count(*) from admin_audit_log where {where}", a=f"%{action}%" if action else None) | ||||
| 438 | + return {"items": rows, "total": int(total or 0), "limit": limit, "offset": offset} | ||||
| 439 | + | ||||
| 440 | + | ||||
| 441 | +@router.post("/runs/{run_id}/rollback") | ||||
| 442 | +async def rollback_run(run_id: str, request: Request) -> dict[str, Any]: | ||||
| 443 | + """Undo one connector run WITHOUT deleting: retract its claims, close its relations/prices/results, flag its events as back-fill (meta.rolled_back).""" | ||||
| 444 | + async with transaction() as conn: | ||||
| 445 | + run = await fetch_one(conn, "select id, connector_name, status from connector_runs where id = :id", id=run_id) | ||||
| 446 | + n: dict[str, int] = {} | ||||
| 447 | + n["claims_retracted"] = int(await fetch_val(conn, "with u as (update claims set status = 'retracted', valid_to = coalesce(valid_to, now()) where run_id = :r and status <> 'retracted' returning 1) select count(*) from u", r=run_id) or 0) | ||||
| 448 | + n["relations_closed"] = int(await fetch_val(conn, "with u as (update relations set valid_to = now() where run_id = :r and valid_to is null returning 1) select count(*) from u", r=run_id) or 0) | ||||
| 449 | + n["prices_closed"] = int(await fetch_val(conn, "with u as (update prices set valid_to = now() where run_id = :r and valid_to is null returning 1) select count(*) from u", r=run_id) or 0) | ||||
| 450 | + n["results_closed"] = int(await fetch_val(conn, "with u as (update benchmark_results set valid_to = now(), is_current = false where run_id = :r and valid_to is null returning 1) select count(*) from u", r=run_id) or 0) | ||||
| 451 | + n["events_flagged"] = int(await fetch_val(conn, """with u as (update change_events set is_backfill = true, meta = meta || '{"rolled_back": true}'::jsonb where run_id = :r | ||||
| 452 | + and not coalesce((meta->>'rolled_back')::boolean, false) returning 1) select count(*) from u""", r=run_id) or 0) | ||||
| 453 | + # re-materialise attributes whose current claim was retracted: fall back to the best remaining claim | ||||
| 454 | + affected = await fetch_all(conn, "select distinct entity_id, property from claims where run_id = :r and status = 'retracted'", r=run_id) | ||||
| 455 | + restored = 0 | ||||
| 456 | + for a in affected: | ||||
| 457 | + prev = await fetch_one(conn, "select id, value from claims where entity_id = :e and property = :p and status in ('superseded','current') order by tier, valid_from desc limit 1", e=a["entity_id"], p=a["property"]) | ||||
| 458 | + if prev: | ||||
| 459 | + await execute(conn, "update claims set status = 'current', valid_to = null where id = :id", id=prev["id"]) | ||||
| 460 | + await execute(conn, "update entities set attributes = attributes || jsonb_build_object(:p, cast(:v as jsonb)), updated_at = now() where id = :e", p=a["property"], v=jsonb(prev["value"]), e=a["entity_id"]) | ||||
| 461 | + restored += 1 | ||||
| 462 | + else: | ||||
| 463 | + await execute(conn, "update entities set attributes = attributes - :p, provenance = provenance - :p, updated_at = now() where id = :e", p=a["property"], e=a["entity_id"]) | ||||
| 464 | + n["attributes_restored"] = restored | ||||
| 465 | + if run: | ||||
| 466 | + await execute(conn, "update connector_runs set meta = meta || cast(:m as jsonb) where id = :id", m=jsonb({"rolled_back": True, "rollback_counts": n}), id=run_id) | ||||
| 467 | + if not run and not any(n.values()): | ||||
| 468 | + raise ApiError(404, f"no run or facts found for run_id {run_id!r}") | ||||
| 469 | + await cache.cache_invalidate() | ||||
| 470 | + await audit("rollback", run_id, {"counts": n, "connector": run["connector_name"] if run else None}, client_ip(request)) | ||||
| 471 | + return {"ok": True, "run_id": run_id, "connector": run["connector_name"] if run else None, "counts": n, "note": "nothing deleted: claims retracted, live rows closed with valid_to, events flagged is_backfill + meta.rolled_back"} | ||||
| 472 | + | ||||
| 473 | + | ||||
| 474 | +__all__ = ["DECISIONS", "router"] | ||||
modified
src/aiatlas/api/routers/benchmarks.py
+223 −30
@@ -1,83 +1,276 @@ | |||||
| 1 | −"""/benchmarks listing, /benchmarks/{slug}, /benchmarks/{slug}/results (leaderboard), /benchmarks/{slug}/history.""" | 1 | +"""/benchmarks · /benchmarks/{slug} · /benchmarks/{slug}/results · /benchmarks/{slug}/leaderboard · /benchmarks/{slug}/history · | ||
| 2 | +/benchmarks/{slug}/frontier · /benchmarks/matrix. | ||||
| 3 | + | ||||
| 4 | +API 1.1: results are organised in comparability GROUPS (canonical metric × config_key); leaderboards are ONE row per canonical model (best | ||||
| 5 | +current row inside the chosen group); benchmarks resolve by slug, alias or id; every benchmark exposes `family`, `variant`, `metric`, | ||||
| 6 | +`direction`, `groups`, `trust_mix`.""" | ||||
| 2 | from __future__ import annotations | 7 | from __future__ import annotations | ||
| 3 | 8 | ||||
| 9 | +from collections import defaultdict | ||||
| 10 | +from datetime import UTC, datetime | ||||
| 4 | from typing import Any | 11 | from typing import Any | ||
| 5 | 12 | ||||
| 6 | from fastapi import APIRouter, Query, Request | 13 | from fastapi import APIRouter, Query, Request | ||
| 7 | 14 | ||||
| 8 | from aiatlas.api.common import ( | 15 | from aiatlas.api.common import ( | ||
| 9 | − ENTITY_COLS, | ||||
| 10 | − ENTITY_FROM, | ||||
| 11 | PAGINATION, | 16 | PAGINATION, | ||
| 12 | RESULT_COLS, | 17 | RESULT_COLS, | ||
| 13 | RESULT_FROM, | 18 | RESULT_FROM, | ||
| 19 | + ApiError, | ||||
| 14 | Pagination, | 20 | Pagination, | ||
| 15 | cached, | 21 | cached, | ||
| 16 | − entity_cols, | 22 | + csv, | ||
| 17 | entity_summary, | 23 | entity_summary, | ||
| 18 | page, | 24 | page, | ||
| 25 | + parse_ts, | ||||
| 19 | resolve_entity, | 26 | resolve_entity, | ||
| 20 | resolve_id, | 27 | resolve_id, | ||
| 21 | result_row, | 28 | result_row, | ||
| 22 | ) | 29 | ) | ||
| 23 | −from aiatlas.api.detail import leaderboard | 30 | +from aiatlas.api.detail import leaderboard as legacy_leaderboard | ||
| 24 | from aiatlas.api.routers.entities import detail_for_type | 31 | from aiatlas.api.routers.entities import detail_for_type | ||
| 25 | from aiatlas.db import connection, fetch_all, fetch_val | 32 | from aiatlas.db import connection, fetch_all, fetch_val | ||
| 33 | +from aiatlas.ontology.benchmarks import TRUST_LABELS, comparability | ||||
| 34 | +from aiatlas.services.frontier import ( | ||||
| 35 | + all_primary_groups, | ||||
| 36 | + benchmark_meta, | ||||
| 37 | + frontier_series, | ||||
| 38 | + group_rows, | ||||
| 39 | + group_summary, | ||||
| 40 | + leaderboard_rows, | ||||
| 41 | + load_results, | ||||
| 42 | + primary_group, | ||||
| 43 | + rank_rows, | ||||
| 44 | +) | ||||
| 26 | 45 | ||||
| 27 | router = APIRouter(prefix="/api/v1/benchmarks", tags=["benchmarks"]) | 46 | router = APIRouter(prefix="/api/v1/benchmarks", tags=["benchmarks"]) | ||
| 47 | +GROUPING_NOTE = ("Rows are grouped by comparability group = canonical metric × config_key (hash of the task-defining configuration keys: variant, " | ||||
| 48 | + "evaluator, harness, shots, pass regime…). A leaderboard shows one row per canonical model: its best current row inside the group. " | ||||
| 49 | + "Reasoning effort, temperature or judge differences keep rows in the same group but mark them partially comparable.") | ||||
| 50 | + | ||||
| 51 | + | ||||
| 52 | +async def _resolve_benchmark(conn: Any, slug: str) -> dict[str, Any]: | ||||
| 53 | + return await resolve_entity(conn, slug, ("benchmark",), aliases=True) | ||||
| 54 | + | ||||
| 55 | + | ||||
| 56 | +def _pick_group(groups: list[dict[str, Any]], attrs: dict[str, Any] | None, *, metric: str | None, config_key: str | None) -> dict[str, Any] | None: | ||||
| 57 | + cands = groups | ||||
| 58 | + if metric: | ||||
| 59 | + m = metric.strip().lower() | ||||
| 60 | + cands = [g for g in cands if g["metric"] == m or (g["metric"] or "").lower() == m] | ||||
| 61 | + if config_key: | ||||
| 62 | + cands = [g for g in cands if g["config_key"] == config_key] | ||||
| 63 | + if metric or config_key: | ||||
| 64 | + return max(cands, key=lambda g: (g["model_count"], g["n"])) if cands else None | ||||
| 65 | + return primary_group(attrs, groups) | ||||
| 66 | + | ||||
| 67 | + | ||||
| 68 | +# ------------------------------------------------------------------------------------------------------------------ listing | ||||
| 28 | 69 | ||||
| 29 | 70 | ||||
| 30 | @router.get("") | 71 | @router.get("") | ||
| 31 | @cached(300) | 72 | @cached(300) | ||
| 32 | async def list_benchmarks(request: Request, category: str | None = None) -> dict[str, Any]: | 73 | async def list_benchmarks(request: Request, category: str | None = None) -> dict[str, Any]: | ||
| 33 | − where = "e.entity_type = 'benchmark' and e.merged_into is null" + (" and e.attributes->>'category' ilike :cat" if category else "") | ||||
| 34 | async with connection() as conn: | 74 | async with connection() as conn: | ||
| 35 | − rows = await fetch_all(conn, f""" | 75 | + meta = await benchmark_meta(conn) | ||
| 36 | − select {ENTITY_COLS}, | 76 | + rows = await load_results(conn) | ||
| 37 | − (select count(*) from benchmark_results r where r.benchmark_id = e.id and r.valid_to is null) as result_count, | 77 | + by_bench: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||
| 38 | − (select count(distinct r.model_id) from benchmark_results r where r.benchmark_id = e.id and r.valid_to is null) as model_count, | ||||
| 39 | − top.score as top_score, {entity_cols("tm", "t_")} | ||||
| 40 | − from {ENTITY_FROM} | ||||
| 41 | − left join lateral (select r.model_id, r.score from benchmark_results r where r.benchmark_id = e.id and r.valid_to is null | ||||
| 42 | − order by case when r.higher_is_better then -r.score else r.score end limit 1) top on true | ||||
| 43 | − left join entities tm on tm.id = top.model_id left join entities tmo on tmo.id = tm.organization_id | ||||
| 44 | − where {where} order by result_count desc, e.canonical_name""", cat=category) | ||||
| 45 | − items = [] | ||||
| 46 | for r in rows: | 78 | for r in rows: | ||
| 47 | − top_model = entity_summary(r, "t_") | 79 | + by_bench[r["benchmark_id"]].append(r) | ||
| 48 | − items.append({**(entity_summary(r) or {}), "result_count": int(r["result_count"] or 0), "model_count": int(r["model_count"] or 0), | 80 | + items = [] | ||
| 49 | − "top": {"model": top_model, "score": r["top_score"]} if top_model else None}) | 81 | + for bid, m in meta.items(): | ||
| 50 | − return {"items": items} | 82 | + if category and (m.get("category") or "").lower() != category.lower(): | ||
| 83 | + continue | ||||
| 84 | + brows = by_bench.get(bid, []) | ||||
| 85 | + groups = list(group_rows(brows).values()) | ||||
| 86 | + pg = primary_group(m["attributes"], groups) | ||||
| 87 | + leader = None | ||||
| 88 | + second = None | ||||
| 89 | + if pg: | ||||
| 90 | + ranked = rank_rows(pg["rows"], pg["higher_is_better"]) | ||||
| 91 | + if ranked: | ||||
| 92 | + lb = leaderboard_rows(pg) | ||||
| 93 | + leader = lb[0] if lb else None | ||||
| 94 | + second = lb[1] if len(lb) > 1 else None | ||||
| 95 | + trust_mix: dict[str, int] = defaultdict(int) | ||||
| 96 | + for r in brows: | ||||
| 97 | + trust_mix[r["trust_level"]] += 1 | ||||
| 98 | + items.append({ | ||||
| 99 | + "id": bid, "entity_type": "benchmark", "slug": m["slug"], "name": m["name"], "category": m.get("category"), "family": m.get("family"), "variant": m.get("variant"), | ||||
| 100 | + "metric": m.get("metric"), "unit": m.get("unit"), "direction": m.get("direction"), "attributes": m["attributes"], | ||||
| 101 | + "result_count": len(brows), "model_count": len({r["model_id"] for r in brows}), | ||||
| 102 | + "leader": leader, "second": second, "top": {"model": leader["model"], "score": leader["score"]} if leader else None, | ||||
| 103 | + "primary_group": group_summary(pg) if pg else None, "groups": sorted((group_summary(g) for g in groups), key=lambda g: (-g["model_count"], g["label"])), | ||||
| 104 | + "trust_mix": dict(trust_mix), "trust_labels": {k: TRUST_LABELS.get(k, k) for k in trust_mix}, | ||||
| 105 | + }) | ||||
| 106 | + items.sort(key=lambda x: (-x["result_count"], x["name"])) | ||||
| 107 | + return {"items": items, "total": len(items), "note": GROUPING_NOTE} | ||||
| 108 | + | ||||
| 109 | + | ||||
| 110 | +@router.get("/matrix") | ||||
| 111 | +@cached(300) | ||||
| 112 | +async def matrix(request: Request, benchmarks: str | None = None, models: str | None = None, org: str | None = None, family: str | None = None, | ||||
| 113 | + limit: int = Query(60, ge=1, le=300), comparable_only: int = Query(0, ge=0, le=1), min_cells: int = Query(3, ge=1, le=30)) -> dict[str, Any]: | ||||
| 114 | + """Rows = canonical models, columns = benchmarks, cell = best current score in the primary comparability group (+ trust, config_key).""" | ||||
| 115 | + async with connection() as conn: | ||||
| 116 | + groups = await all_primary_groups(conn) | ||||
| 117 | + meta = await benchmark_meta(conn) | ||||
| 118 | + wanted_b: list[str] | None = None | ||||
| 119 | + if benchmarks: | ||||
| 120 | + wanted_b = [] | ||||
| 121 | + for key in csv(benchmarks): | ||||
| 122 | + b = await _resolve_benchmark(conn, key) | ||||
| 123 | + wanted_b.append(b["id"]) | ||||
| 124 | + wanted_m: set[str] | None = None | ||||
| 125 | + if models: | ||||
| 126 | + wanted_m = {await resolve_id(conn, k, ("model",)) or "" for k in csv(models)} | ||||
| 127 | + if org: | ||||
| 128 | + rows = await fetch_all(conn, "select e.id from entities e join entities o on o.id = e.organization_id where e.entity_type = 'model' and e.merged_into is null and (o.slug = :o or o.id = :o or o.canonical_name ilike :o)", o=org) | ||||
| 129 | + wanted_m = (wanted_m or set()) | {r["id"] for r in rows} | ||||
| 130 | + if family: | ||||
| 131 | + rows = await fetch_all(conn, "select e.id from entities e left join entities f on f.id = e.family_id where e.entity_type = 'model' and e.merged_into is null and (f.slug = :f or f.id = :f or e.attributes->>'family' ilike :f)", f=family) | ||||
| 132 | + wanted_m = (wanted_m or set()) | {r["id"] for r in rows} | ||||
| 133 | + if wanted_b is None: | ||||
| 134 | + wanted_b = [bid for bid, _ in sorted(groups.items(), key=lambda kv: -kv[1]["n"])[:12]] | ||||
| 135 | + columns = [] | ||||
| 136 | + cells: dict[str, dict[str, dict[str, Any]]] = defaultdict(dict) | ||||
| 137 | + model_ref: dict[str, dict[str, Any]] = {} | ||||
| 138 | + for bid in wanted_b: | ||||
| 139 | + g = groups.get(bid) | ||||
| 140 | + m = meta.get(bid) or {"id": bid, "slug": None, "name": None} | ||||
| 141 | + columns.append({"id": bid, "slug": m.get("slug"), "name": m.get("name"), "category": m.get("category"), "metric": g["metric"] if g else m.get("metric"), | ||||
| 142 | + "config_key": g["config_key"] if g else None, "group_label": g["label"] if g else None, "higher_is_better": g["higher_is_better"] if g else None, | ||||
| 143 | + "n_models": g["model_count"] if g else 0}) | ||||
| 144 | + if not g: | ||||
| 145 | + continue | ||||
| 146 | + leader_cfg = None | ||||
| 147 | + ranked = rank_rows(g["rows"], g["higher_is_better"]) | ||||
| 148 | + if ranked: | ||||
| 149 | + leader_cfg = ranked[0] | ||||
| 150 | + for r in ranked: | ||||
| 151 | + if wanted_m is not None and r["model_id"] not in wanted_m: | ||||
| 152 | + continue | ||||
| 153 | + level, _ = comparability(leader_cfg["config"] if leader_cfg else None, r.get("config"), leader_cfg["metric"] if leader_cfg else None, r.get("metric")) if leader_cfg else ("comparable", []) | ||||
| 154 | + if comparable_only and level != "comparable": | ||||
| 155 | + continue | ||||
| 156 | + cells[r["model_id"]][bid] = {"score": r["score"], "rank": r["rank"], "trust_level": r["trust_level"], "config_key": r["config_key"], "comparability": level, | ||||
| 157 | + "result_id": r["id"]} | ||||
| 158 | + model_ref.setdefault(r["model_id"], {"id": r["model_id"], "slug": r["model_slug"], "name": r["model_name"], "organization": r.get("org_name"), "organization_slug": r.get("org_slug"), | ||||
| 159 | + "openness": (r.get("model_attrs") or {}).get("openness"), "release_date": (r.get("model_attrs") or {}).get("release_date")}) | ||||
| 160 | + rows_out = [] | ||||
| 161 | + for mid, c in cells.items(): | ||||
| 162 | + if wanted_m is None and len(c) < min_cells: | ||||
| 163 | + continue | ||||
| 164 | + rows_out.append({"model": model_ref[mid], "cells": {bid: c.get(bid) for bid in wanted_b}, "n_cells": len(c), | ||||
| 165 | + "mean_rank": round(sum(x["rank"] for x in c.values()) / len(c), 2)}) | ||||
| 166 | + rows_out.sort(key=lambda x: (-x["n_cells"], x["mean_rank"], x["model"]["name"] or "")) | ||||
| 167 | + return {"columns": columns, "rows": rows_out[:limit], "total_rows": len(rows_out), "comparable_only": bool(comparable_only), "min_cells": min_cells if wanted_m is None else None, | ||||
| 168 | + "methodology": GROUPING_NOTE + " Each cell is the model's best current row in the benchmark's primary group; `mean_rank` is only a sort key, not a composite score."} | ||||
| 169 | + | ||||
| 170 | + | ||||
| 171 | +# ------------------------------------------------------------------------------------------------------------------ one benchmark | ||||
| 51 | 172 | ||||
| 52 | 173 | ||||
| 53 | @router.get("/{slug}") | 174 | @router.get("/{slug}") | ||
| 54 | @cached(300) | 175 | @cached(300) | ||
| 55 | async def get_benchmark(request: Request, slug: str) -> dict[str, Any]: | 176 | async def get_benchmark(request: Request, slug: str) -> dict[str, Any]: | ||
| 56 | − return await detail_for_type(slug, ("benchmark",)) | 177 | + async with connection() as conn: | ||
| 178 | + bench = await _resolve_benchmark(conn, slug) | ||||
| 179 | + rows = await load_results(conn, benchmark_ids=[bench["id"]]) | ||||
| 180 | + meta = (await benchmark_meta(conn, [bench["id"]])).get(bench["id"], {}) | ||||
| 181 | + detail = await detail_for_type(bench["slug"], ("benchmark",)) | ||||
| 182 | + groups = list(group_rows(rows).values()) | ||||
| 183 | + pg = primary_group(bench.get("attributes"), groups) | ||||
| 184 | + detail.update({"family": meta.get("family"), "variant": meta.get("variant"), "metric": meta.get("metric"), "direction": meta.get("direction"), "category": meta.get("category"), | ||||
| 185 | + "groups": sorted((group_summary(g) for g in groups), key=lambda g: (-g["model_count"], g["label"])), "primary_group": group_summary(pg) if pg else None, | ||||
| 186 | + "result_count": len(rows), "model_count": len({r["model_id"] for r in rows}), "leaderboard": leaderboard_rows(pg)[:25] if pg else [], | ||||
| 187 | + "trust_mix": dict(defaultdict(int, {k: sum(1 for r in rows if r["trust_level"] == k) for k in {r["trust_level"] for r in rows}}))}) | ||||
| 188 | + return detail | ||||
| 57 | 189 | ||||
| 58 | 190 | ||||
| 59 | @router.get("/{slug}/results") | 191 | @router.get("/{slug}/results") | ||
| 60 | @cached(300) | 192 | @cached(300) | ||
| 61 | −async def benchmark_results(request: Request, slug: str, p: Pagination = PAGINATION, config: str | None = Query(None, max_length=200), history: int = Query(0, ge=0, le=1)) -> dict[str, Any]: | 193 | +async def benchmark_results(request: Request, slug: str, p: Pagination = PAGINATION, config: str | None = Query(None, max_length=200), history: int = Query(0, ge=0, le=1), | ||
| 194 | + metric: str | None = Query(None, max_length=80), config_key: str | None = Query(None, max_length=40)) -> dict[str, Any]: | ||||
| 195 | + """v1 behaviour (one row per result, sorted by score) + `metric=` and `config_key=` filters.""" | ||||
| 62 | async with connection() as conn: | 196 | async with connection() as conn: | ||
| 63 | − bench = await resolve_entity(conn, slug, ("benchmark",)) | 197 | + bench = await _resolve_benchmark(conn, slug) | ||
| 64 | − items = await leaderboard(conn, bench["id"], limit=p.limit, offset=p.offset, config=config, history=bool(history)) | 198 | + items = await legacy_leaderboard(conn, bench["id"], limit=p.limit, offset=p.offset, config=config, history=bool(history), metric=metric, config_key=config_key) | ||
| 65 | − where = "r.benchmark_id = :id" + ("" if history else " and r.valid_to is null") + (" and r.config::text ilike :cfg" if config else "") | 199 | + where = "r.benchmark_id = :id" + ("" if history else " and r.valid_to is null") + (" and r.config::text ilike :cfg" if config else "") \ | ||
| 66 | − total = await fetch_val(conn, f"select count(*) from benchmark_results r where {where}", id=bench["id"], cfg=f"%{config}%" if config else None) | 200 | + + (" and lower(r.metric) = lower(:metric)" if metric else "") + (" and r.config_key = :ck" if config_key else "") | ||
| 201 | + total = await fetch_val(conn, f"select count(*) from benchmark_results r where {where}", id=bench["id"], cfg=f"%{config}%" if config else None, metric=metric, ck=config_key) | ||||
| 67 | out = page(items, int(total or 0), p) | 202 | out = page(items, int(total or 0), p) | ||
| 68 | out["benchmark"] = entity_summary(bench) | 203 | out["benchmark"] = entity_summary(bench) | ||
| 69 | return out | 204 | return out | ||
| 70 | 205 | ||||
| 71 | 206 | ||||
| 207 | +@router.get("/{slug}/leaderboard") | ||||
| 208 | +@cached(300) | ||||
| 209 | +async def benchmark_leaderboard(request: Request, slug: str, metric: str | None = None, config_key: str | None = None, trust: str | None = None, org: str | None = None, | ||||
| 210 | + since: str | None = None, until: str | None = None, comparable_only: int = Query(0, ge=0, le=1), limit: int = Query(100, ge=1, le=1000), | ||||
| 211 | + offset: int = Query(0, ge=0)) -> dict[str, Any]: | ||||
| 212 | + """ONE row per canonical model — best current row inside the chosen comparability group (default: primary group).""" | ||||
| 213 | + since_ts, until_ts = parse_ts(since, "since"), parse_ts(until, "until") | ||||
| 214 | + async with connection() as conn: | ||||
| 215 | + bench = await _resolve_benchmark(conn, slug) | ||||
| 216 | + rows = await load_results(conn, benchmark_ids=[bench["id"]]) | ||||
| 217 | + closed = await load_results(conn, benchmark_ids=[bench["id"]], current_only=False) | ||||
| 218 | + org_id = await resolve_id(conn, org) if org else None | ||||
| 219 | + groups = list(group_rows(rows).values()) | ||||
| 220 | + g = _pick_group(groups, bench.get("attributes"), metric=metric, config_key=config_key) | ||||
| 221 | + if not g: | ||||
| 222 | + return {"benchmark": entity_summary(bench), "group": None, "groups": [group_summary(x) for x in groups], "items": [], "total": 0, | ||||
| 223 | + "note": "no current results in the requested group" if (metric or config_key) else "no current results for this benchmark"} | ||||
| 224 | + trust_set = set(csv(trust)) if trust else None | ||||
| 225 | + sel = [r for r in g["rows"] if (trust_set is None or r["trust_level"] in trust_set) and (org_id is None or r.get("organization_id") == org_id) | ||||
| 226 | + and (since_ts is None or (r.get("evaluated_at") or r["observed_at"]) >= since_ts) and (until_ts is None or (r.get("evaluated_at") or r["observed_at"]) <= until_ts)] | ||||
| 227 | + history_rows = [r for r in closed if r.get("valid_to") is not None and r["metric_canonical"] == g["metric"] and r["config_key"] == g["config_key"]] | ||||
| 228 | + sub = {**g, "rows": sel} | ||||
| 229 | + items = leaderboard_rows(sub, history_rows=history_rows or None, comparable_only=bool(comparable_only)) | ||||
| 230 | + return {"benchmark": entity_summary(bench), "group": group_summary(g), "groups": sorted((group_summary(x) for x in groups), key=lambda x: (-x["model_count"], x["label"])), | ||||
| 231 | + "items": items[offset:offset + limit], "total": len(items), "limit": limit, "offset": offset, "comparable_only": bool(comparable_only), | ||||
| 232 | + "filters": {k: v for k, v in {"metric": metric, "config_key": config_key, "trust": trust, "org": org, "since": since, "until": until}.items() if v}, | ||||
| 233 | + "history_available": bool(history_rows), "methodology": GROUPING_NOTE + " delta_rank compares with the ranking built from the closed (previous) rows of the same group."} | ||||
| 234 | + | ||||
| 235 | + | ||||
| 236 | +@router.get("/{slug}/frontier") | ||||
| 237 | +@cached(300) | ||||
| 238 | +async def benchmark_frontier(request: Request, slug: str, metric: str | None = None, config_key: str | None = None) -> dict[str, Any]: | ||||
| 239 | + """History of the leader per comparability group: a point each time a new best score appears (ordered by coalesce(evaluated_at, observed_at)).""" | ||||
| 240 | + async with connection() as conn: | ||||
| 241 | + bench = await _resolve_benchmark(conn, slug) | ||||
| 242 | + rows = await load_results(conn, benchmark_ids=[bench["id"]], current_only=False) | ||||
| 243 | + groups = list(group_rows(rows).values()) | ||||
| 244 | + if metric or config_key: | ||||
| 245 | + groups = [g for g in groups if (not metric or g["metric"] == metric.strip().lower()) and (not config_key or g["config_key"] == config_key)] | ||||
| 246 | + pg = primary_group(bench.get("attributes"), groups) | ||||
| 247 | + series = [] | ||||
| 248 | + for g in sorted(groups, key=lambda x: (-x["model_count"], x["label"])): | ||||
| 249 | + pts = frontier_series(g["rows"], g["higher_is_better"]) | ||||
| 250 | + series.append({"group": group_summary(g), "primary": pg is not None and g["config_key"] == pg["config_key"] and g["metric"] == pg["metric"], "points": pts, | ||||
| 251 | + "current_leader": pts[-1] if pts else None}) | ||||
| 252 | + return {"benchmark": entity_summary(bench), "series": series, "generated_at": datetime.now(UTC), | ||||
| 253 | + "methodology": "Includes closed rows (history). A point is emitted whenever a result beats every earlier result of the same group, " | ||||
| 254 | + "ordered by evaluated_at when the source publishes it, else observed_at."} | ||||
| 255 | + | ||||
| 256 | + | ||||
| 72 | @router.get("/{slug}/history") | 257 | @router.get("/{slug}/history") | ||
| 73 | @cached(300) | 258 | @cached(300) | ||
| 74 | async def benchmark_history(request: Request, slug: str, model: str | None = None, limit: int = Query(1000, ge=1, le=5000)) -> dict[str, Any]: | 259 | async def benchmark_history(request: Request, slug: str, model: str | None = None, limit: int = Query(1000, ge=1, le=5000)) -> dict[str, Any]: | ||
| 75 | async with connection() as conn: | 260 | async with connection() as conn: | ||
| 76 | − bench = await resolve_entity(conn, slug, ("benchmark",)) | 261 | + bench = await _resolve_benchmark(conn, slug) | ||
| 77 | where = ["r.benchmark_id = :id"] | 262 | where = ["r.benchmark_id = :id"] | ||
| 78 | params: dict[str, Any] = {"id": bench["id"], "lim": limit} | 263 | params: dict[str, Any] = {"id": bench["id"], "lim": limit} | ||
| 79 | if model: | 264 | if model: | ||
| 80 | where.append("r.model_id = :model") | 265 | where.append("r.model_id = :model") | ||
| 81 | params["model"] = await resolve_id(conn, model) | 266 | params["model"] = await resolve_id(conn, model) | ||
| 82 | − rows = await fetch_all(conn, f"select {RESULT_COLS} from {RESULT_FROM} where {' and '.join(where)} order by r.observed_at asc, r.id limit :lim", **params) | 267 | + rows = await fetch_all(conn, f"select {RESULT_COLS}, r.config_key, r.trust_level from {RESULT_FROM} where {' and '.join(where)} order by r.observed_at asc, r.id limit :lim", **params) | ||
| 83 | − return {"benchmark": entity_summary(bench), "items": [result_row(r) for r in rows]} | 268 | + items = [] | ||
| 269 | + for r in rows: | ||||
| 270 | + it = result_row(r) | ||||
| 271 | + it["config_key"], it["trust_level"] = r.get("config_key"), r.get("trust_level") | ||||
| 272 | + items.append(it) | ||||
| 273 | + return {"benchmark": entity_summary(bench), "items": items} | ||||
| 274 | + | ||||
| 275 | + | ||||
| 276 | +__all__ = ["ApiError", "router"] | ||||
modified
src/aiatlas/api/routers/changes.py
+99 −24
@@ -1,4 +1,8 @@ | |||||
| 1 | −"""/changes (cursor feed) · /changes/daily ("What changed in AI today") · /changes/categories.""" | 1 | +"""/changes (cursor feed) · /changes/daily ("Today in AI 2.0") · /changes/categories. | ||
| 2 | + | ||||
| 3 | +API 1.1: feeds default to `is_backfill = false` and are keyed on `occurred_at` (= coalesce(effective_at, observed_at)); `include_backfill=1` | ||||
| 4 | +restores the historical corpus, `date_field=observed` restores the v1 ordering. `/changes/daily` groups the events that share a `group_key` | ||||
| 5 | +(one release across several documents) into one item with `sources: n` and `documents: [urls]`.""" | ||||
| 2 | from __future__ import annotations | 6 | from __future__ import annotations | ||
| 3 | 7 | ||||
| 4 | from datetime import UTC, datetime | 8 | from datetime import UTC, datetime | ||
@@ -11,6 +15,7 @@ from aiatlas.api.common import ( | |||||
| 11 | ENTITY_FROM, | 15 | ENTITY_FROM, | ||
| 12 | EVENT_COLS, | 16 | EVENT_COLS, | ||
| 13 | EVENT_FROM, | 17 | EVENT_FROM, | ||
| 18 | + OPEN_CATEGORIES, | ||||
| 14 | ApiError, | 19 | ApiError, | ||
| 15 | cached, | 20 | cached, | ||
| 16 | change_event, | 21 | change_event, | ||
@@ -32,11 +37,28 @@ CATEGORY_LABELS = {"model": "Models", "price": "Pricing", "benchmark": "Benchmar | |||||
| 32 | CATEGORY_ORDER = list(CATEGORY_LABELS) | 37 | CATEGORY_ORDER = list(CATEGORY_LABELS) | ||
| 33 | TOTAL_CAP = 10_000 | 38 | TOTAL_CAP = 10_000 | ||
| 34 | 39 | ||||
| 40 | +# "Today in AI 2.0" sections — (key, label, SQL predicate on ev/e) | ||||
| 41 | +SECTIONS: list[tuple[str, str, str]] = [ | ||||
| 42 | + ("MAJOR_RELEASES", "Major releases", "ev.event_type in ('NEW_MODEL','RELEASE') and ev.importance >= 2 and e.entity_type = 'model'"), | ||||
| 43 | + ("OPEN_WEIGHT_RELEASES", "Open-weight releases", "ev.event_type = 'NEW_MODEL' and e.entity_type = 'model' and e.attributes->>'openness' in (" + ", ".join(f"'{c}'" for c in OPEN_CATEGORIES) + ")"), | ||||
| 44 | + ("PRICE_MOVES", "Price moves", "ev.category = 'price' or ev.event_type = 'PRICE_CHANGED'"), | ||||
| 45 | + ("BENCHMARK_MOVES", "Benchmark moves", "ev.event_type in ('BENCHMARK_UPDATED','BENCHMARK_LEADER_CHANGED','NEW_BENCHMARK_LEADER')"), | ||||
| 46 | + ("MODEL_CHANGES", "Model changes", "ev.event_type in ('CONTEXT_CHANGED','MAX_OUTPUT_CHANGED','STATUS_CHANGED','CAPABILITIES_CHANGED','PARAMETERS_CHANGED','KNOWLEDGE_CUTOFF_CHANGED','LICENSE_CHANGED','OPENNESS_CHANGED')"), | ||||
| 47 | + ("RESEARCH", "Research", "ev.event_type = 'NEW_PAPER'"), | ||||
| 48 | + ("DEPRECATIONS", "Deprecations & retirements", "ev.event_type in ('DEPRECATION_ANNOUNCED','RETIREMENT_ANNOUNCED') or (ev.event_type = 'STATUS_CHANGED' and ev.new_value::text ~* 'deprecated|retired')"), | ||||
| 49 | + ("PROVIDER_CHANGES", "Provider listings", "ev.event_type in ('PROVIDER_LISTED','PROVIDER_DELISTED','NEW_PROVIDER')"), | ||||
| 50 | + ("HARDWARE", "Hardware", "ev.category = 'hardware' or ev.event_type = 'NEW_HARDWARE'"), | ||||
| 51 | +] | ||||
| 52 | + | ||||
| 35 | 53 | ||||
| 36 | def _filters(*, category: str | None, types: list[str], entity_type: str | None, importance_min: int | None, since: datetime | None, until: datetime | None, | 54 | def _filters(*, category: str | None, types: list[str], entity_type: str | None, importance_min: int | None, since: datetime | None, until: datetime | None, | ||
| 37 | − q: str | None, include_documents: bool, entity_id: str | None = None, before: datetime | None = None) -> tuple[list[str], dict[str, Any]]: | 55 | + q: str | None, include_documents: bool, entity_id: str | None = None, before: datetime | None = None, include_backfill: bool = False, | ||
| 56 | + date_field: str = "occurred") -> tuple[list[str], dict[str, Any]]: | ||||
| 57 | + col = "ev.observed_at" if date_field == "observed" else "ev.occurred_at" | ||||
| 38 | where: list[str] = [] | 58 | where: list[str] = [] | ||
| 39 | p: dict[str, Any] = {} | 59 | p: dict[str, Any] = {} | ||
| 60 | + if not include_backfill: | ||||
| 61 | + where.append("ev.is_backfill = false") | ||||
| 40 | if category: | 62 | if category: | ||
| 41 | where.append("ev.category = any(cast(:cats as text[]))") | 63 | where.append("ev.category = any(cast(:cats as text[]))") | ||
| 42 | p["cats"] = csv(category) | 64 | p["cats"] = csv(category) | ||
@@ -52,13 +74,13 @@ def _filters(*, category: str | None, types: list[str], entity_type: str | None, | |||||
| 52 | where.append("ev.importance >= :imp") | 74 | where.append("ev.importance >= :imp") | ||
| 53 | p["imp"] = importance_min | 75 | p["imp"] = importance_min | ||
| 54 | if since is not None: | 76 | if since is not None: | ||
| 55 | − where.append("ev.observed_at >= :since") | 77 | + where.append(f"{col} >= :since") | ||
| 56 | p["since"] = since | 78 | p["since"] = since | ||
| 57 | if until is not None: | 79 | if until is not None: | ||
| 58 | − where.append("ev.observed_at <= :until") | 80 | + where.append(f"{col} <= :until") | ||
| 59 | p["until"] = until | 81 | p["until"] = until | ||
| 60 | if before is not None: | 82 | if before is not None: | ||
| 61 | − where.append("ev.observed_at < :before") | 83 | + where.append(f"{col} < :before") | ||
| 62 | p["before"] = before | 84 | p["before"] = before | ||
| 63 | if q: | 85 | if q: | ||
| 64 | where.append("(ev.summary ilike :qlike or e.canonical_name ilike :qlike)") | 86 | where.append("(ev.summary ilike :qlike or e.canonical_name ilike :qlike)") | ||
@@ -69,59 +91,112 @@ def _filters(*, category: str | None, types: list[str], entity_type: str | None, | |||||
| 69 | return where or ["true"], p | 91 | return where or ["true"], p | ||
| 70 | 92 | ||||
| 71 | 93 | ||||
| 94 | +def _event(r: dict[str, Any]) -> dict[str, Any]: | ||||
| 95 | + ev = change_event(r) | ||||
| 96 | + ev["occurred_at"] = r.get("occurred_at") | ||||
| 97 | + ev["is_backfill"] = r.get("is_backfill") | ||||
| 98 | + ev["group_key"] = r.get("group_key") | ||||
| 99 | + return ev | ||||
| 100 | + | ||||
| 101 | + | ||||
| 72 | @router.get("") | 102 | @router.get("") | ||
| 73 | @cached(60) | 103 | @cached(60) | ||
| 74 | async def list_changes(request: Request, category: str | None = None, type: str | None = Query(None, alias="type"), entity_type: str | None = None, | 104 | async def list_changes(request: Request, category: str | None = None, type: str | None = Query(None, alias="type"), entity_type: str | None = None, | ||
| 75 | importance_min: int | None = Query(None, ge=0, le=3), since: str | None = None, until: str | None = None, q: str | None = Query(None, max_length=200), | 105 | importance_min: int | None = Query(None, ge=0, le=3), since: str | None = None, until: str | None = None, q: str | None = Query(None, max_length=200), | ||
| 76 | entity: str | None = None, limit: int = Query(50, ge=1, le=200), before: str | None = None, offset: int = Query(0, ge=0, le=10000), | 106 | entity: str | None = None, limit: int = Query(50, ge=1, le=200), before: str | None = None, offset: int = Query(0, ge=0, le=10000), | ||
| 77 | − include_documents: int = Query(0, ge=0, le=1)) -> dict[str, Any]: | 107 | + include_documents: int = Query(0, ge=0, le=1), include_backfill: int = Query(0, ge=0, le=1), | ||
| 108 | + date_field: str = Query("occurred", pattern="^(occurred|observed)$")) -> dict[str, Any]: | ||||
| 78 | before_ts, since_ts, until_ts = parse_ts(before, "before"), parse_ts(since, "since"), parse_ts(until, "until") | 109 | before_ts, since_ts, until_ts = parse_ts(before, "before"), parse_ts(since, "since"), parse_ts(until, "until") | ||
| 110 | + col = "ev.observed_at" if date_field == "observed" else "ev.occurred_at" | ||||
| 79 | async with connection() as conn: | 111 | async with connection() as conn: | ||
| 80 | eid = await resolve_id(conn, entity) if entity else None | 112 | eid = await resolve_id(conn, entity) if entity else None | ||
| 81 | where, params = _filters(category=category, types=csv(type), entity_type=entity_type, importance_min=importance_min, since=since_ts, until=until_ts, q=q, | 113 | where, params = _filters(category=category, types=csv(type), entity_type=entity_type, importance_min=importance_min, since=since_ts, until=until_ts, q=q, | ||
| 82 | − include_documents=bool(include_documents), entity_id=eid, before=before_ts) | 114 | + include_documents=bool(include_documents), entity_id=eid, before=before_ts, include_backfill=bool(include_backfill), date_field=date_field) | ||
| 83 | where_sql = " and ".join(where) | 115 | where_sql = " and ".join(where) | ||
| 84 | − rows = await fetch_all(conn, f"select {EVENT_COLS} from {EVENT_FROM} where {where_sql} order by ev.observed_at desc, ev.id desc limit :lim offset :off", | 116 | + rows = await fetch_all(conn, f"select {EVENT_COLS}, ev.occurred_at, ev.is_backfill, ev.group_key from {EVENT_FROM} where {where_sql} order by {col} desc, ev.id desc limit :lim offset :off", | ||
| 85 | lim=limit, off=offset, **params) | 117 | lim=limit, off=offset, **params) | ||
| 86 | total = await fetch_val(conn, f"select count(*) from (select 1 from {EVENT_FROM} where {where_sql} limit {TOTAL_CAP}) t", **params) | 118 | total = await fetch_val(conn, f"select count(*) from (select 1 from {EVENT_FROM} where {where_sql} limit {TOTAL_CAP}) t", **params) | ||
| 87 | − items = [change_event(r) for r in rows] | 119 | + items = [_event(r) for r in rows] | ||
| 88 | − return {"items": items, "total": int(total or 0), "limit": limit, "offset": offset, "next_before": items[-1]["observed_at"] if len(items) == limit else None} | 120 | + cursor_key = "observed_at" if date_field == "observed" else "occurred_at" | ||
| 121 | + return {"items": items, "total": int(total or 0), "limit": limit, "offset": offset, "next_before": items[-1][cursor_key] if len(items) == limit else None, | ||||
| 122 | + "date_field": date_field, "include_backfill": bool(include_backfill)} | ||||
| 123 | + | ||||
| 124 | + | ||||
| 125 | +def _group(items: list[dict[str, Any]]) -> list[dict[str, Any]]: | ||||
| 126 | + """Fold events that share a `group_key` (one release across documents) into one item: the most important event + sources/documents.""" | ||||
| 127 | + out: list[dict[str, Any]] = [] | ||||
| 128 | + by_key: dict[str, dict[str, Any]] = {} | ||||
| 129 | + for ev in items: | ||||
| 130 | + k = ev.get("group_key") | ||||
| 131 | + if not k: | ||||
| 132 | + out.append({**ev, "sources": 1, "documents": [ev["source_url"]] if ev.get("source_url") else [], "grouped_events": 1}) | ||||
| 133 | + continue | ||||
| 134 | + g = by_key.get(k) | ||||
| 135 | + if g is None: | ||||
| 136 | + g = by_key[k] = {**ev, "sources": 0, "documents": [], "grouped_events": 0, "event_ids": []} | ||||
| 137 | + out.append(g) | ||||
| 138 | + g["grouped_events"] += 1 | ||||
| 139 | + g["event_ids"].append(ev["id"]) | ||||
| 140 | + if ev.get("source_url") and ev["source_url"] not in g["documents"]: | ||||
| 141 | + g["documents"].append(ev["source_url"]) | ||||
| 142 | + if (ev.get("importance") or 0) > (g.get("importance") or 0): | ||||
| 143 | + for f in ("id", "event_type", "summary", "importance", "new_value", "old_value", "property", "entity"): | ||||
| 144 | + g[f] = ev.get(f) | ||||
| 145 | + for g in out: | ||||
| 146 | + g["sources"] = max(1, len(g["documents"])) | ||||
| 147 | + return out | ||||
| 89 | 148 | ||||
| 90 | 149 | ||||
| 91 | @router.get("/daily") | 150 | @router.get("/daily") | ||
| 92 | @cached(120) | 151 | @cached(120) | ||
| 93 | −async def changes_daily(request: Request, date: str | None = None, per_section: int = Query(30, ge=1, le=100)) -> dict[str, Any]: | 152 | +async def changes_daily(request: Request, date: str | None = None, per_section: int = Query(30, ge=1, le=100), include_backfill: int = Query(0, ge=0, le=1), | ||
| 153 | + date_field: str = Query("occurred", pattern="^(occurred|observed)$")) -> dict[str, Any]: | ||||
| 94 | d = parse_date(date, "date") or datetime.now(UTC).date() | 154 | d = parse_date(date, "date") or datetime.now(UTC).date() | ||
| 95 | start, end = day_bounds(d) | 155 | start, end = day_bounds(d) | ||
| 156 | + col = "ev.observed_at" if date_field == "observed" else "ev.occurred_at" | ||||
| 157 | + bf = "" if include_backfill else " and ev.is_backfill = false" | ||||
| 158 | + base = f"{col} >= :s and {col} <= :e and ev.event_type <> 'DOCUMENT_CHANGED'{bf}" | ||||
| 96 | async with connection() as conn: | 159 | async with connection() as conn: | ||
| 97 | − rows = await fetch_all(conn, f"""select {EVENT_COLS} from ( | 160 | + rows = await fetch_all(conn, f"""select {EVENT_COLS}, ev.occurred_at, ev.is_backfill, ev.group_key from ( | ||
| 98 | − select ev.*, row_number() over (partition by ev.category order by ev.importance desc, ev.observed_at desc) as rn | 161 | + select ev.*, row_number() over (partition by ev.category order by ev.importance desc, ev.occurred_at desc) as rn | ||
| 99 | − from change_events ev where ev.observed_at >= :s and ev.observed_at <= :e and ev.event_type <> 'DOCUMENT_CHANGED') ev | 162 | + from change_events ev where {base}) ev | ||
| 100 | left join entities e on e.id = ev.entity_id left join entities eo on eo.id = e.organization_id | 163 | left join entities e on e.id = ev.entity_id left join entities eo on eo.id = e.organization_id | ||
| 101 | where ev.rn <= :n order by ev.category, ev.rn""", s=start, e=end, n=per_section) | 164 | where ev.rn <= :n order by ev.category, ev.rn""", s=start, e=end, n=per_section) | ||
| 102 | − counts = await fetch_all(conn, "select category, count(*) as n from change_events where observed_at >= :s and observed_at <= :e and event_type <> 'DOCUMENT_CHANGED' group by 1", s=start, e=end) | 165 | + counts = await fetch_all(conn, f"select ev.category, count(*) as n from change_events ev where {base} group by 1", s=start, e=end) | ||
| 166 | + backfill_excluded = 0 if include_backfill else int(await fetch_val(conn, f"select count(*) from change_events ev where {col} >= :s and {col} <= :e and ev.is_backfill and ev.event_type <> 'DOCUMENT_CHANGED'", s=start, e=end) or 0) | ||||
| 103 | new_models = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and e.first_seen_at >= :s and e.first_seen_at <= :e " | 167 | new_models = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and e.first_seen_at >= :s and e.first_seen_at <= :e " | ||
| 104 | f"order by e.first_seen_at desc limit 100", s=start, e=end) | 168 | f"order by e.first_seen_at desc limit 100", s=start, e=end) | ||
| 105 | − prev = await fetch_val(conn, "select max(observed_at)::date from change_events where observed_at < :s and event_type <> 'DOCUMENT_CHANGED'", s=start) | 169 | + prev = await fetch_val(conn, f"select max(ev.occurred_at)::date from change_events ev where ev.occurred_at < :s and ev.event_type <> 'DOCUMENT_CHANGED'{bf}", s=start) | ||
| 106 | − nxt = await fetch_val(conn, "select min(observed_at)::date from change_events where observed_at > :e and event_type <> 'DOCUMENT_CHANGED'", e=end) | 170 | + nxt = await fetch_val(conn, f"select min(ev.occurred_at)::date from change_events ev where ev.occurred_at > :e and ev.event_type <> 'DOCUMENT_CHANGED'{bf}", e=end) | ||
| 171 | + sections2: list[dict[str, Any]] = [] | ||||
| 172 | + for key, label, pred in SECTIONS: | ||||
| 173 | + srows = await fetch_all(conn, f"""select {EVENT_COLS}, ev.occurred_at, ev.is_backfill, ev.group_key from {EVENT_FROM} where {base} and ({pred}) | ||||
| 174 | + order by ev.importance desc, ev.occurred_at desc, ev.id limit :n""", s=start, e=end, n=per_section * 3) | ||||
| 175 | + items = _group([_event(r) for r in srows])[:per_section] | ||||
| 176 | + total_s = await fetch_val(conn, f"select count(*) from {EVENT_FROM} where {base} and ({pred})", s=start, e=end) | ||||
| 177 | + if items: | ||||
| 178 | + sections2.append({"key": key, "label": label, "items": items, "total": int(total_s or 0)}) | ||||
| 107 | by_cat: dict[str, list[dict[str, Any]]] = {} | 179 | by_cat: dict[str, list[dict[str, Any]]] = {} | ||
| 108 | for r in rows: | 180 | for r in rows: | ||
| 109 | − by_cat.setdefault(r["category"], []).append(change_event(r)) | 181 | + by_cat.setdefault(r["category"], []).append(_event(r)) | ||
| 110 | order = {c: i for i, c in enumerate(CATEGORY_ORDER)} | 182 | order = {c: i for i, c in enumerate(CATEGORY_ORDER)} | ||
| 111 | sections = [{"category": c, "label": CATEGORY_LABELS.get(c, c.title()), "items": items} for c, items in sorted(by_cat.items(), key=lambda kv: (order.get(kv[0], 99), kv[0]))] | 183 | sections = [{"category": c, "label": CATEGORY_LABELS.get(c, c.title()), "items": items} for c, items in sorted(by_cat.items(), key=lambda kv: (order.get(kv[0], 99), kv[0]))] | ||
| 112 | return {"date": d.isoformat(), "counts": {r["category"]: int(r["n"]) for r in counts}, "total": sum(int(r["n"]) for r in counts), "sections": sections, | 184 | return {"date": d.isoformat(), "counts": {r["category"]: int(r["n"]) for r in counts}, "total": sum(int(r["n"]) for r in counts), "sections": sections, | ||
| 113 | − "new_models": [entity_summary(r) for r in new_models], "labels": CATEGORY_LABELS, | 185 | + "today": sections2, "new_models": [entity_summary(r) for r in new_models], "labels": CATEGORY_LABELS, "backfill_excluded": backfill_excluded, | ||
| 114 | − "previous_day": prev.isoformat() if prev else None, "next_day": nxt.isoformat() if nxt else None} | 186 | + "date_field": date_field, "previous_day": prev.isoformat() if prev else None, "next_day": nxt.isoformat() if nxt else None, | ||
| 187 | + "note": "Events that occurred on this UTC day (effective date when known, else observation date), excluding back-filled history and source-document " | ||||
| 188 | + "changes. `today` groups events sharing a group_key (one release seen in several documents)."} | ||||
| 115 | 189 | ||||
| 116 | 190 | ||||
| 117 | @router.get("/categories") | 191 | @router.get("/categories") | ||
| 118 | @cached(300) | 192 | @cached(300) | ||
| 119 | −async def changes_categories(request: Request, days: int = Query(7, ge=1, le=365)) -> dict[str, Any]: | 193 | +async def changes_categories(request: Request, days: int = Query(7, ge=1, le=365), include_backfill: int = Query(0, ge=0, le=1)) -> dict[str, Any]: | ||
| 194 | + bf = "" if include_backfill else " and is_backfill = false" | ||||
| 120 | async with connection() as conn: | 195 | async with connection() as conn: | ||
| 121 | − rows = await fetch_all(conn, """select category, event_type, count(*) as count from change_events where observed_at > now() - make_interval(days => :d) | 196 | + rows = await fetch_all(conn, f"""select category, event_type, count(*) as count from change_events where occurred_at > now() - make_interval(days => :d) | ||
| 122 | − and event_type <> 'DOCUMENT_CHANGED' group by 1, 2 order by 3 desc, 1, 2""", d=days) | 197 | + and event_type <> 'DOCUMENT_CHANGED'{bf} group by 1, 2 order by 3 desc, 1, 2""", d=days) | ||
| 123 | return {"days": days, "items": [{"category": r["category"], "label": CATEGORY_LABELS.get(r["category"], r["category"].title()), "event_type": r["event_type"], | 198 | return {"days": days, "items": [{"category": r["category"], "label": CATEGORY_LABELS.get(r["category"], r["category"].title()), "event_type": r["event_type"], | ||
| 124 | "event_label": event_type_label(r["event_type"]), "count": int(r["count"])} for r in rows]} | 199 | "event_label": event_type_label(r["event_type"]), "count": int(r["count"])} for r in rows]} | ||
| 125 | 200 | ||||
| 126 | 201 | ||||
| 127 | −__all__ = ["CATEGORY_LABELS", "ApiError", "router"] | 202 | +__all__ = ["CATEGORY_LABELS", "SECTIONS", "ApiError", "router"] | ||
added
src/aiatlas/api/routers/claims.py
+51 −0
@@ -0,0 +1,51 @@ | |||||
| 1 | +"""/claims/{id} — claim lifecycle (API 1.1): the claim, its entity, the chain of superseded / superseding / conflicting siblings, source, extractor, | ||||
| 2 | +run id and evidence pointer (snapshot id + document URL + archived flag; never raw content).""" | ||||
| 3 | +from __future__ import annotations | ||||
| 4 | + | ||||
| 5 | +from typing import Any | ||||
| 6 | + | ||||
| 7 | +from fastapi import APIRouter, Request | ||||
| 8 | + | ||||
| 9 | +from aiatlas.api.common import ( | ||||
| 10 | + CLAIM_COLS, | ||||
| 11 | + CLAIM_FROM, | ||||
| 12 | + ENTITY_COLS, | ||||
| 13 | + ENTITY_FROM, | ||||
| 14 | + ApiError, | ||||
| 15 | + cached, | ||||
| 16 | + claim_row, | ||||
| 17 | + entity_summary, | ||||
| 18 | +) | ||||
| 19 | +from aiatlas.db import connection, fetch_all, fetch_one | ||||
| 20 | + | ||||
| 21 | +router = APIRouter(prefix="/api/v1/claims", tags=["claims"]) | ||||
| 22 | + | ||||
| 23 | + | ||||
| 24 | +@router.get("/{claim_id}") | ||||
| 25 | +@cached(120) | ||||
| 26 | +async def claim_detail(request: Request, claim_id: str) -> dict[str, Any]: | ||||
| 27 | + async with connection() as conn: | ||||
| 28 | + c = await fetch_one(conn, f"select {CLAIM_COLS}, c.entity_id, c.snapshot_id, c.run_id, c.extractor_version, c.value_raw, c.value_text, c.value_num, c.source_id, " | ||||
| 29 | + f"s.domain as source_domain, s.tier as source_tier from {CLAIM_FROM} where c.id = :id", id=claim_id) | ||||
| 30 | + if not c: | ||||
| 31 | + raise ApiError(404, "claim not found") | ||||
| 32 | + ent = await fetch_one(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.id = :id", id=c["entity_id"]) | ||||
| 33 | + siblings = await fetch_all(conn, f"select {CLAIM_COLS}, c.snapshot_id, c.run_id from {CLAIM_FROM} where c.entity_id = :e and c.property = :p and c.id <> :id order by c.valid_from asc, c.observed_at asc limit 200", | ||||
| 34 | + e=c["entity_id"], p=c["property"], id=claim_id) | ||||
| 35 | + snap = None | ||||
| 36 | + if c.get("snapshot_id"): | ||||
| 37 | + snap = await fetch_one(conn, """select s.id, s.observed_at, s.http_status, s.content_type, s.parser_version, s.raw_path is not null as archived, s.text_path is not null as has_text, | ||||
| 38 | + d.url as document_url, d.title as document_title, d.doc_type from snapshots s join documents d on d.id = s.document_id where s.id = :id""", id=c["snapshot_id"]) | ||||
| 39 | + vf = c["valid_from"] | ||||
| 40 | + previous = [claim_row(s) for s in siblings if s["status"] == "superseded" and (s["valid_to"] is None or s["valid_to"] <= vf)] | ||||
| 41 | + superseding = [claim_row(s) for s in siblings if s["valid_from"] >= vf and s["status"] in ("current", "superseded") and c["status"] != "current"] | ||||
| 42 | + conflicting = [claim_row(s) for s in siblings if s["status"] == "conflicting"] | ||||
| 43 | + return {"claim": {**claim_row(c), "value_raw": c.get("value_raw"), "run_id": c.get("run_id"), "extractor_version": c.get("extractor_version")}, | ||||
| 44 | + "entity": entity_summary(ent) if ent else {"id": c["entity_id"]}, "property": c["property"], | ||||
| 45 | + "chain": {"previous": previous[-5:], "superseding": superseding[:5], "conflicting": conflicting, "history_count": len(siblings) + 1}, | ||||
| 46 | + "source": {"id": c.get("source_id"), "name": c.get("source_name"), "domain": c.get("source_domain"), "tier": c.get("tier"), "url": c.get("source_url"), | ||||
| 47 | + "snapshot_id": c.get("snapshot_id"), "observed_at": c.get("observed_at")}, | ||||
| 48 | + "extractor": {"name": c.get("extractor"), "version": c.get("extractor_version"), "confidence": c.get("confidence")}, "run_id": c.get("run_id"), | ||||
| 49 | + "evidence": {"snapshot_id": c.get("snapshot_id"), "document_url": (snap or {}).get("document_url") or c.get("source_url"), "archived": bool((snap or {}).get("archived")), | ||||
| 50 | + "snapshot_observed_at": (snap or {}).get("observed_at"), "document_title": (snap or {}).get("document_title"), "doc_type": (snap or {}).get("doc_type")}, | ||||
| 51 | + "note": "Evidence is a pointer to the archived snapshot; raw content is available to administrators only (/admin/extractions/{snapshot_id})."} | ||||
modified
src/aiatlas/api/routers/compare.py
+74 −26
@@ -1,4 +1,8 @@ | |||||
| 1 | −"""/compare?ids=a,b,… — side-by-side of 2–6 entities of the same type with per-type dimension sets and provenance.""" | 1 | +"""/compare?ids=a,b,… — side-by-side of 2–6 entities of the same type with per-type dimension sets and provenance. | ||
| 2 | + | ||||
| 3 | +API 1.1: benchmark dimensions are keyed by (benchmark, canonical metric, config_key) and only appear when EVERY compared model has a current | ||||
| 4 | +result in that comparability group; `comparability` labels each benchmark dimension; `diff_only=1` keeps only differing dimensions; | ||||
| 5 | +`mode=models|providers|hardware|companies|frameworks` asserts the expected type.""" | ||||
| 2 | from __future__ import annotations | 6 | from __future__ import annotations | ||
| 3 | 7 | ||||
| 4 | from typing import Any | 8 | from typing import Any | ||
@@ -9,8 +13,6 @@ from aiatlas.api.common import ( | |||||
| 9 | COMPANY_TYPES, | 13 | COMPANY_TYPES, | ||
| 10 | PRICE_COLS, | 14 | PRICE_COLS, | ||
| 11 | PRICE_FROM, | 15 | PRICE_FROM, | ||
| 12 | − RESULT_COLS, | ||||
| 13 | − RESULT_FROM, | ||||
| 14 | ApiError, | 16 | ApiError, | ||
| 15 | cached, | 17 | cached, | ||
| 16 | csv, | 18 | csv, | ||
@@ -18,9 +20,10 @@ from aiatlas.api.common import ( | |||||
| 18 | entity_summary, | 20 | entity_summary, | ||
| 19 | price_row, | 21 | price_row, | ||
| 20 | resolve_entity, | 22 | resolve_entity, | ||
| 21 | − result_row, | ||||
| 22 | ) | 23 | ) | ||
| 23 | from aiatlas.db import connection, fetch_all | 24 | from aiatlas.db import connection, fetch_all | ||
| 25 | +from aiatlas.ontology.benchmarks import TRUST_LABELS, comparability | ||||
| 26 | +from aiatlas.services.frontier import best_per_model, config_summary, group_rows, load_results | ||||
| 24 | 27 | ||||
| 25 | router = APIRouter(prefix="/api/v1/compare", tags=["compare"]) | 28 | router = APIRouter(prefix="/api/v1/compare", tags=["compare"]) | ||
| 26 | 29 | ||||
@@ -30,7 +33,7 @@ DIMENSIONS: dict[str, list[dict[str, Any]]] = { | |||||
| 30 | "model": [D("parameter_count", "Parameters", "number", "params"), D("active_parameter_count", "Active parameters", "number", "params"), | 33 | "model": [D("parameter_count", "Parameters", "number", "params"), D("active_parameter_count", "Active parameters", "number", "params"), | ||
| 31 | D("context_length", "Context window", "number", "tokens"), D("max_output_tokens", "Max output", "number", "tokens"), D("openness", "Openness"), | 34 | D("context_length", "Context window", "number", "tokens"), D("max_output_tokens", "Max output", "number", "tokens"), D("openness", "Openness"), | ||
| 32 | D("license", "License"), D("modalities", "Modalities", "list"), D("release_date", "Release date", "date"), D("knowledge_cutoff", "Knowledge cutoff", "date"), | 35 | D("license", "License"), D("modalities", "Modalities", "list"), D("release_date", "Release date", "date"), D("knowledge_cutoff", "Knowledge cutoff", "date"), | ||
| 33 | − D("status", "Status"), D("family", "Family"), D("architecture", "Architecture"), | 36 | + D("status", "Status"), D("family", "Family"), D("architecture", "Architecture"), D("reasoning", "Reasoning", "bool"), D("tool_calling", "Tool calling", "bool"), | ||
| 34 | D("best_input_per_mtok", "Best input price", "number", "USD / 1M tokens", "prices"), D("best_output_per_mtok", "Best output price", "number", "USD / 1M tokens", "prices"), | 37 | D("best_input_per_mtok", "Best input price", "number", "USD / 1M tokens", "prices"), D("best_output_per_mtok", "Best output price", "number", "USD / 1M tokens", "prices"), | ||
| 35 | D("provider_count", "Providers", "number", None, "prices")], | 38 | D("provider_count", "Providers", "number", None, "prices")], | ||
| 36 | "provider": [D("website", "Website"), D("pricing_url", "Pricing page"), D("model_count", "Models priced", "number", None, "prices"), | 39 | "provider": [D("website", "Website"), D("pricing_url", "Pricing page"), D("model_count", "Models priced", "number", None, "prices"), | ||
@@ -44,50 +47,77 @@ DIMENSIONS: dict[str, list[dict[str, Any]]] = { | |||||
| 44 | D("model_count", "Models", "number", None, "graph"), D("paper_count", "Papers", "number", None, "graph")], | 47 | D("model_count", "Models", "number", None, "graph"), D("paper_count", "Papers", "number", None, "graph")], | ||
| 45 | "benchmark": [D("category", "Category"), D("metric", "Metric"), D("unit", "Unit"), D("task", "Task"), D("result_count", "Results", "number", None, "results")], | 48 | "benchmark": [D("category", "Category"), D("metric", "Metric"), D("unit", "Unit"), D("task", "Task"), D("result_count", "Results", "number", None, "results")], | ||
| 46 | } | 49 | } | ||
| 50 | +MODES = {"models": "model", "providers": "provider", "hardware": "hardware", "companies": "company", "frameworks": "framework", "benchmarks": "benchmark"} | ||||
| 51 | +TYPE_ALIASES = {"library": "framework", "runtime": "framework", "artifact": "model"} | ||||
| 47 | 52 | ||||
| 48 | 53 | ||||
| 49 | −@router.get("") | 54 | +def _norm(v: Any) -> Any: | ||
| 50 | −@cached(300) | 55 | + if isinstance(v, list): | ||
| 51 | −async def compare(request: Request, ids: str = Query(..., description="2–6 slugs or ids, comma-separated")) -> dict[str, Any]: | 56 | + return sorted(str(x) for x in v) | ||
| 52 | − keys = csv(ids) | 57 | + if isinstance(v, float) and v.is_integer(): | ||
| 58 | + return int(v) | ||||
| 59 | + return v | ||||
| 60 | + | ||||
| 61 | + | ||||
| 62 | +async def compare_entities(keys: list[str], *, diff_only: bool = False, mode: str | None = None) -> dict[str, Any]: | ||||
| 53 | if not 2 <= len(keys) <= 6: | 63 | if not 2 <= len(keys) <= 6: | ||
| 54 | raise ApiError(400, "ids must list between 2 and 6 entities") | 64 | raise ApiError(400, "ids must list between 2 and 6 entities") | ||
| 65 | + if mode and mode not in MODES: | ||||
| 66 | + raise ApiError(400, f"mode must be one of {', '.join(MODES)}") | ||||
| 55 | async with connection() as conn: | 67 | async with connection() as conn: | ||
| 56 | rows = [await resolve_entity(conn, k) for k in keys] | 68 | rows = [await resolve_entity(conn, k) for k in keys] | ||
| 57 | − types = {r["entity_type"] for r in rows} | 69 | + types = {TYPE_ALIASES.get(r["entity_type"], r["entity_type"]) for r in rows} | ||
| 58 | − etype = rows[0]["entity_type"] | 70 | + etype = TYPE_ALIASES.get(rows[0]["entity_type"], rows[0]["entity_type"]) | ||
| 59 | if etype in COMPANY_TYPES: | 71 | if etype in COMPANY_TYPES: | ||
| 60 | etype = "company" | 72 | etype = "company" | ||
| 61 | types = {"company"} | 73 | types = {"company"} | ||
| 62 | if len(types) != 1: | 74 | if len(types) != 1: | ||
| 63 | raise ApiError(400, f"all entities must share one type, got {', '.join(sorted(types))}") | 75 | raise ApiError(400, f"all entities must share one type, got {', '.join(sorted(types))}") | ||
| 76 | + if mode and MODES[mode] != etype: | ||||
| 77 | + raise ApiError(400, f"mode={mode} expects {MODES[mode]} entities, got {etype}") | ||||
| 64 | dims = [dict(d) for d in DIMENSIONS.get(etype, [])] | 78 | dims = [dict(d) for d in DIMENSIONS.get(etype, [])] | ||
| 65 | eids = [r["id"] for r in rows] | 79 | eids = [r["id"] for r in rows] | ||
| 66 | items: list[dict[str, Any]] = [] | 80 | items: list[dict[str, Any]] = [] | ||
| 67 | extra: dict[str, dict[str, Any]] = {eid: {} for eid in eids} | 81 | extra: dict[str, dict[str, Any]] = {eid: {} for eid in eids} | ||
| 68 | prices_by: dict[str, list[dict[str, Any]]] = {eid: [] for eid in eids} | 82 | prices_by: dict[str, list[dict[str, Any]]] = {eid: [] for eid in eids} | ||
| 69 | results_by: dict[str, list[dict[str, Any]]] = {eid: [] for eid in eids} | 83 | results_by: dict[str, list[dict[str, Any]]] = {eid: [] for eid in eids} | ||
| 84 | + comp: dict[str, dict[str, Any]] = {} | ||||
| 70 | if etype == "model": | 85 | if etype == "model": | ||
| 71 | for p in await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.model_id = any(cast(:ids as text[])) and p.valid_to is null order by p.input_per_mtok nulls last", ids=eids): | 86 | for p in await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.model_id = any(cast(:ids as text[])) and p.valid_to is null order by p.input_per_mtok nulls last", ids=eids): | ||
| 72 | prices_by[p["m_id"]].append(price_row(p)) | 87 | prices_by[p["m_id"]].append(price_row(p)) | ||
| 73 | for eid, plist in prices_by.items(): | 88 | for eid, plist in prices_by.items(): | ||
| 74 | − ins = [x["input_per_mtok"] for x in plist if x["input_per_mtok"] is not None] | 89 | + ins = [x["input_per_mtok"] for x in plist if x["input_per_mtok"] is not None and x["input_per_mtok"] > 0] | ||
| 75 | − outs = [x["output_per_mtok"] for x in plist if x["output_per_mtok"] is not None] | 90 | + outs = [x["output_per_mtok"] for x in plist if x["output_per_mtok"] is not None and x["output_per_mtok"] > 0] | ||
| 76 | extra[eid] = {"best_input_per_mtok": min(ins) if ins else None, "best_output_per_mtok": min(outs) if outs else None, | 91 | extra[eid] = {"best_input_per_mtok": min(ins) if ins else None, "best_output_per_mtok": min(outs) if outs else None, | ||
| 77 | "provider_count": len({x["provider"]["id"] for x in plist if x["provider"]})} | 92 | "provider_count": len({x["provider"]["id"] for x in plist if x["provider"]})} | ||
| 78 | − res = await fetch_all(conn, f"select {RESULT_COLS} from {RESULT_FROM} where r.model_id = any(cast(:ids as text[])) and r.valid_to is null order by b.canonical_name, r.observed_at desc", ids=eids) | 93 | + res = await load_results(conn, model_ids=eids, canonical_models_only=False) | ||
| 79 | − per_bench: dict[str, dict[str, float]] = {} | ||||
| 80 | − bench_meta: dict[str, dict[str, Any]] = {} | ||||
| 81 | for r in res: | 94 | for r in res: | ||
| 82 | − results_by[r["m_id"]].append(result_row(r)) | 95 | + results_by[r["model_id"]].append({"id": r["id"], "benchmark": {"id": r["benchmark_id"], "slug": r["benchmark_slug"], "name": r["benchmark_name"]}, "score": r["score"], | ||
| 83 | − per_bench.setdefault(r["b_id"], {}).setdefault(r["m_id"], r["score"]) | 96 | + "metric": r["metric_canonical"], "unit": r.get("unit"), "higher_is_better": r.get("higher_is_better"), "config": config_summary(r.get("config")), | ||
| 84 | − bench_meta.setdefault(r["b_id"], {"slug": r["b_slug"], "name": r["b_canonical_name"], "unit": r.get("unit") or r.get("b_attributes", {}).get("unit"), "higher_is_better": r["higher_is_better"]}) | 97 | + "config_key": r["config_key"], "trust_level": r["trust_level"], "evaluated_at": r.get("evaluated_at"), "observed_at": r["observed_at"], | ||
| 85 | − for bid, scores in per_bench.items(): | 98 | + "source_url": r.get("source_url"), "tier": r.get("tier")}) | ||
| 86 | − if all(eid in scores for eid in eids): | 99 | + for g in sorted(group_rows(res).values(), key=lambda g: (g["rows"][0]["benchmark_name"] or "", g["label"])): | ||
| 87 | − meta = bench_meta[bid] | 100 | + if g["models"] < set(eids) and not set(eids) <= g["models"]: | ||
| 88 | − dims.append({"key": f"bench:{meta['slug']}", "label": meta["name"], "kind": "number", "unit": meta["unit"], "source": "results", "higher_is_better": meta["higher_is_better"]}) | 101 | + continue | ||
| 89 | − for eid in eids: | 102 | + best = {r["model_id"]: r for r in best_per_model(g["rows"], g["higher_is_better"])} | ||
| 90 | − extra[eid][f"bench:{meta['slug']}"] = scores[eid] | 103 | + if not all(eid in best for eid in eids): | ||
| 104 | + continue | ||||
| 105 | + first = g["rows"][0] | ||||
| 106 | + key = f"bench:{first['benchmark_slug']}:{g['metric']}:{g['config_key']}" | ||||
| 107 | + ref = best[eids[0]] | ||||
| 108 | + worst_level, reasons = "comparable", [] | ||||
| 109 | + for eid in eids[1:]: | ||||
| 110 | + level, why = comparability(ref.get("config"), best[eid].get("config"), ref.get("metric"), best[eid].get("metric")) | ||||
| 111 | + if level == "not-comparable" or (level == "partially-comparable" and worst_level == "comparable"): | ||||
| 112 | + worst_level = level | ||||
| 113 | + reasons += [w for w in why if w not in reasons and level != "comparable"] | ||||
| 114 | + dims.append({"key": key, "label": f"{first['benchmark_name']} · {g['label']}", "kind": "number", "unit": first.get("unit"), "source": "results", | ||||
| 115 | + "higher_is_better": g["higher_is_better"], "benchmark": first["benchmark_slug"], "metric": g["metric"], "config_key": g["config_key"], | ||||
| 116 | + "comparability": worst_level, "trust_levels": sorted({best[e]["trust_level"] for e in eids})}) | ||||
| 117 | + comp[key] = {"level": worst_level, "reasons": reasons or ["same variant, metric and evaluation conditions"], | ||||
| 118 | + "trust": {e: {"level": best[e]["trust_level"], "label": TRUST_LABELS.get(best[e]["trust_level"], best[e]["trust_level"])} for e in eids}} | ||||
| 119 | + for eid in eids: | ||||
| 120 | + extra[eid][key] = best[eid]["score"] | ||||
| 91 | elif etype == "provider": | 121 | elif etype == "provider": | ||
| 92 | agg = await fetch_all(conn, """select provider_id, count(distinct model_id) as model_count, min(nullif(input_per_mtok, 0)) as min_input_per_mtok, | 122 | agg = await fetch_all(conn, """select provider_id, count(distinct model_id) as model_count, min(nullif(input_per_mtok, 0)) as min_input_per_mtok, | ||
| 93 | min(nullif(output_per_mtok, 0)) as min_output_per_mtok, jsonb_agg(distinct k.key) filter (where k.key is not null) as features | 123 | min(nullif(output_per_mtok, 0)) as min_output_per_mtok, jsonb_agg(distinct k.key) filter (where k.key is not null) as features | ||
@@ -115,4 +145,22 @@ async def compare(request: Request, ids: str = Query(..., description="2–6 slu | |||||
| 115 | item["prices"] = prices_by[r["id"]] | 145 | item["prices"] = prices_by[r["id"]] | ||
| 116 | item["results"] = results_by[r["id"]] | 146 | item["results"] = results_by[r["id"]] | ||
| 117 | items.append(item) | 147 | items.append(item) | ||
| 118 | − return {"entity_type": etype, "dimensions": dims, "items": items} | 148 | + if diff_only: | ||
| 149 | + keep = [d for d in dims if len({repr(_norm(it["values"].get(d["key"]))) for it in items}) > 1] | ||||
| 150 | + kept = {d["key"] for d in keep} | ||||
| 151 | + dims = keep | ||||
| 152 | + for it in items: | ||||
| 153 | + it["values"] = {k: v for k, v in it["values"].items() if k in kept} | ||||
| 154 | + it["provenance"] = {k: v for k, v in it["provenance"].items() if k in kept} | ||||
| 155 | + return {"entity_type": etype, "dimensions": dims, "items": items, "comparability": comp, "diff_only": diff_only, | ||||
| 156 | + "note": "Benchmark dimensions appear only when every compared model has a current result in the same comparability group (benchmark × metric × config_key)."} | ||||
| 157 | + | ||||
| 158 | + | ||||
| 159 | +@router.get("") | ||||
| 160 | +@cached(300) | ||||
| 161 | +async def compare(request: Request, ids: str = Query(..., description="2–6 slugs or ids, comma-separated"), diff_only: int = Query(0, ge=0, le=1), | ||||
| 162 | + mode: str | None = Query(None, description="models|providers|hardware|companies|frameworks|benchmarks")) -> dict[str, Any]: | ||||
| 163 | + return await compare_entities(csv(ids), diff_only=bool(diff_only), mode=mode) | ||||
| 164 | + | ||||
| 165 | + | ||||
| 166 | +__all__ = ["DIMENSIONS", "compare_entities", "router"] | ||||
added
src/aiatlas/api/routers/deployments.py
+111 −0
@@ -0,0 +1,111 @@ | |||||
| 1 | +"""/deployments (cursor page of `Deployment` rows) · /cost (per request / day / month / year for every current deployment) · | ||||
| 2 | +/cost/context (cost of one fully populated context).""" | ||||
| 3 | +from __future__ import annotations | ||||
| 4 | + | ||||
| 5 | +from typing import Any | ||||
| 6 | + | ||||
| 7 | +from fastapi import APIRouter, Query, Request | ||||
| 8 | + | ||||
| 9 | +from aiatlas.api.common import ( | ||||
| 10 | + PRICE_COLS, | ||||
| 11 | + PRICE_FROM, | ||||
| 12 | + ApiError, | ||||
| 13 | + cached, | ||||
| 14 | + deployment_row, | ||||
| 15 | + parse_ts, | ||||
| 16 | + resolve_entity, | ||||
| 17 | + resolve_id, | ||||
| 18 | +) | ||||
| 19 | +from aiatlas.db import connection, fetch_all, fetch_val | ||||
| 20 | +from aiatlas.services.cost import compute_cost, context_fill_cost | ||||
| 21 | + | ||||
| 22 | +router = APIRouter(prefix="/api/v1", tags=["deployments"]) | ||||
| 23 | +TOTAL_CAP = 10_000 | ||||
| 24 | + | ||||
| 25 | + | ||||
| 26 | +@router.get("/deployments") | ||||
| 27 | +@cached(300) | ||||
| 28 | +async def list_deployments(request: Request, model: str | None = None, provider: str | None = None, current: int = Query(1, ge=0, le=1), limit: int = Query(50, ge=1, le=200), | ||||
| 29 | + offset: int = Query(0, ge=0, le=10000), before: str | None = Query(None, description="cursor: valid_from of the last item"), | ||||
| 30 | + org: str | None = None, sort: str = Query("valid_from", pattern="^(valid_from|output|input|model|provider)$")) -> dict[str, Any]: | ||||
| 31 | + where = ["m.entity_type in ('model','artifact')", "m.merged_into is null"] | ||||
| 32 | + params: dict[str, Any] = {"lim": limit, "off": offset} | ||||
| 33 | + if current: | ||||
| 34 | + where.append("p.valid_to is null") | ||||
| 35 | + before_ts = parse_ts(before, "before") | ||||
| 36 | + if before_ts is not None: | ||||
| 37 | + where.append("p.valid_from < :before") | ||||
| 38 | + params["before"] = before_ts | ||||
| 39 | + async with connection() as conn: | ||||
| 40 | + if model: | ||||
| 41 | + where.append("p.model_id = :model") | ||||
| 42 | + params["model"] = await resolve_id(conn, model) | ||||
| 43 | + if provider: | ||||
| 44 | + where.append("p.provider_id = :provider") | ||||
| 45 | + params["provider"] = await resolve_id(conn, provider) | ||||
| 46 | + if org: | ||||
| 47 | + where.append("(mo.slug = :org or mo.id = :org or mo.canonical_name ilike :org)") | ||||
| 48 | + params["org"] = org | ||||
| 49 | + order = {"valid_from": "p.valid_from desc, p.id desc", "output": "p.output_per_mtok asc nulls last, p.id", "input": "p.input_per_mtok asc nulls last, p.id", | ||||
| 50 | + "model": "m.canonical_name asc, pv.canonical_name asc, p.id", "provider": "pv.canonical_name asc, m.canonical_name asc, p.id"}[sort] | ||||
| 51 | + where_sql = " and ".join(where) | ||||
| 52 | + rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {where_sql} order by {order} limit :lim offset :off", **params) | ||||
| 53 | + total = await fetch_val(conn, f"select count(*) from (select 1 from {PRICE_FROM} where {where_sql} limit {TOTAL_CAP}) t", **{k: v for k, v in params.items() if k not in ('lim', 'off')}) | ||||
| 54 | + items = [deployment_row(r) for r in rows] | ||||
| 55 | + return {"items": items, "total": int(total or 0), "limit": limit, "offset": offset, | ||||
| 56 | + "next_before": items[-1]["valid_from"] if len(items) == limit and sort == "valid_from" else None, "current": bool(current)} | ||||
| 57 | + | ||||
| 58 | + | ||||
| 59 | +@router.get("/cost") | ||||
| 60 | +@cached(300) | ||||
| 61 | +async def cost(request: Request, model: str = Query(..., description="model slug or id"), provider: str | None = None, input_tokens: int = Query(1000, ge=0, le=100_000_000), | ||||
| 62 | + output_tokens: int = Query(500, ge=0, le=100_000_000), requests_per_day: float = Query(1000, ge=0, le=1e9), cached_share: float = Query(0.0, ge=0, le=1), | ||||
| 63 | + batch: int = Query(0, ge=0, le=1)) -> dict[str, Any]: | ||||
| 64 | + async with connection() as conn: | ||||
| 65 | + m = await resolve_entity(conn, model, ("model", "artifact")) | ||||
| 66 | + where = ["p.model_id = :mid", "p.valid_to is null"] | ||||
| 67 | + params: dict[str, Any] = {"mid": m["id"]} | ||||
| 68 | + if provider: | ||||
| 69 | + where.append("p.provider_id = :pid") | ||||
| 70 | + params["pid"] = await resolve_id(conn, provider, ("provider",)) | ||||
| 71 | + rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {' and '.join(where)} order by p.output_per_mtok asc nulls last, pv.canonical_name limit 200", **params) | ||||
| 72 | + items = [] | ||||
| 73 | + for r in rows: | ||||
| 74 | + d = deployment_row(r) | ||||
| 75 | + c = compute_cost(d["prices"], input_tokens=input_tokens, output_tokens=output_tokens, requests_per_day=requests_per_day, cached_share=cached_share, batch=bool(batch)) | ||||
| 76 | + items.append({"deployment": d, "cost": c}) | ||||
| 77 | + items.sort(key=lambda x: (x["cost"]["per_request"] is None, x["cost"]["per_request"] or 0)) | ||||
| 78 | + return {"model": {"id": m["id"], "slug": m["slug"], "name": m["canonical_name"]}, "inputs": {"input_tokens": input_tokens, "output_tokens": output_tokens, "requests_per_day": requests_per_day, | ||||
| 79 | + "cached_share": cached_share, "batch": bool(batch)}, | ||||
| 80 | + "items": items, "total": len(items), "currency": "USD", | ||||
| 81 | + "methodology": "per_request = input_tokens × effective input price / 1e6 + output_tokens × output price / 1e6 (+ per-request fee when published); " | ||||
| 82 | + "effective input = (1 − cached_share) × input + cached_share × cached input (falls back to the standard price with a note); " | ||||
| 83 | + "batch uses batch prices when published; daily = per_request × requests_per_day; monthly = daily × 30; annual = daily × 365. Only current offers.", | ||||
| 84 | + "note": None if items else "no current deployment for this model"} | ||||
| 85 | + | ||||
| 86 | + | ||||
| 87 | +@router.get("/cost/context") | ||||
| 88 | +@cached(300) | ||||
| 89 | +async def cost_context(request: Request, tokens: int = Query(1_000_000, ge=1, le=100_000_000), limit: int = Query(50, ge=1, le=500), model: str | None = None, | ||||
| 90 | + org: str | None = None) -> dict[str, Any]: | ||||
| 91 | + """Cost of ONE fully populated context of `tokens` input tokens per current deployment whose context window is ≥ tokens, cheapest first.""" | ||||
| 92 | + where = ["p.valid_to is null", "p.input_per_mtok > 0", "m.entity_type in ('model','artifact')", "m.merged_into is null", | ||||
| 93 | + "coalesce(p.context_length, case when m.attributes->>'context_length' ~ '^[0-9]+$' then (m.attributes->>'context_length')::bigint end) >= :tokens"] | ||||
| 94 | + params: dict[str, Any] = {"tokens": tokens, "lim": limit} | ||||
| 95 | + async with connection() as conn: | ||||
| 96 | + if model: | ||||
| 97 | + where.append("p.model_id = :mid") | ||||
| 98 | + params["mid"] = await resolve_id(conn, model) | ||||
| 99 | + if org: | ||||
| 100 | + where.append("(mo.slug = :org or mo.id = :org or mo.canonical_name ilike :org)") | ||||
| 101 | + params["org"] = org | ||||
| 102 | + rows = await fetch_all(conn, f"""select {PRICE_COLS}, coalesce(p.context_length, case when m.attributes->>'context_length' ~ '^[0-9]+$' then (m.attributes->>'context_length')::bigint end) as ctx | ||||
| 103 | + from {PRICE_FROM} where {' and '.join(where)} order by p.input_per_mtok asc, m.canonical_name limit :lim""", **params) | ||||
| 104 | + items = [{"deployment": deployment_row(r), "context_length": r["ctx"], "context_source": "offer" if r.get("context_length") else "model attribute", | ||||
| 105 | + "cost_usd": context_fill_cost(r["input_per_mtok"], tokens)} for r in rows] | ||||
| 106 | + return {"tokens": tokens, "items": items, "total": len(items), "currency": "USD", | ||||
| 107 | + "methodology": "cost = input price (USD per 1M tokens) × tokens / 1e6 for every current offer whose context window (offer's, else the model's attribute) is at least `tokens`. " | ||||
| 108 | + "Long-context surcharges published as native units are not applied."} | ||||
| 109 | + | ||||
| 110 | + | ||||
| 111 | +__all__ = ["ApiError", "router"] | ||||
modified
src/aiatlas/api/routers/diff.py
+55 −16
@@ -1,4 +1,6 @@ | |||||
| 1 | −"""/diff?a=&b=&scope= — what changed between two dates: new / gone entities, property, price and benchmark changes.""" | 1 | +"""/diff?a=&b=&scope= — what changed between two dates: new / gone entities, property, price and benchmark changes, new benchmark leaders, | ||
| 2 | +provider listings, hardware, context changes, retired models. API 1.1: events use `occurred_at` and `is_backfill = false`; `new_entities` | ||||
| 3 | +excludes artifacts unless `include=artifacts`.""" | ||||
| 2 | from __future__ import annotations | 4 | from __future__ import annotations | ||
| 3 | 5 | ||||
| 4 | from datetime import UTC, datetime | 6 | from datetime import UTC, datetime | ||
@@ -19,9 +21,12 @@ from aiatlas.api.common import ( | |||||
| 19 | parse_date, | 21 | parse_date, | ||
| 20 | ) | 22 | ) | ||
| 21 | from aiatlas.db import connection, fetch_all, fetch_one, fetch_val | 23 | from aiatlas.db import connection, fetch_all, fetch_one, fetch_val | ||
| 24 | +from aiatlas.services.frontier import benchmark_meta, leader_at, load_results | ||||
| 22 | 25 | ||||
| 23 | router = APIRouter(prefix="/api/v1/diff", tags=["diff"]) | 26 | router = APIRouter(prefix="/api/v1/diff", tags=["diff"]) | ||
| 24 | GONE_TYPES = ("RETIREMENT_ANNOUNCED", "DEPRECATION_ANNOUNCED", "STATUS_CHANGED", "ENTITY_MERGED", "MODEL_RETIRED", "MODEL_DEPRECATED") | 27 | GONE_TYPES = ("RETIREMENT_ANNOUNCED", "DEPRECATION_ANNOUNCED", "STATUS_CHANGED", "ENTITY_MERGED", "MODEL_RETIRED", "MODEL_DEPRECATED") | ||
| 28 | +RETIRED_PRED = "(ev.event_type in ('RETIREMENT_ANNOUNCED','DEPRECATION_ANNOUNCED','MODEL_RETIRED','MODEL_DEPRECATED') or (ev.event_type = 'STATUS_CHANGED' and ev.new_value::text ~* 'retired|deprecated'))" | ||||
| 29 | +_leaders_at = leader_at | ||||
| 25 | 30 | ||||
| 26 | 31 | ||||
| 27 | async def _scope(conn: Any, scope: str) -> tuple[str, str, dict[str, Any], dict[str, Any]]: | 32 | async def _scope(conn: Any, scope: str) -> tuple[str, str, dict[str, Any], dict[str, Any]]: | ||
@@ -39,7 +44,7 @@ async def _scope(conn: Any, scope: str) -> tuple[str, str, dict[str, Any], dict[ | |||||
| 39 | frag = "(e.organization_id = :org or e.id = :org)" | 44 | frag = "(e.organization_id = :org or e.id = :org)" | ||
| 40 | return frag, frag, {"org": org["id"]}, {"kind": "org", "organization": {"id": org["id"], "slug": org["slug"], "name": org["canonical_name"]}} | 45 | return frag, frag, {"org": org["id"]}, {"kind": "org", "organization": {"id": org["id"], "slug": org["slug"], "name": org["canonical_name"]}} | ||
| 41 | if s.startswith("family:"): | 46 | if s.startswith("family:"): | ||
| 42 | − frag = "e.attributes->>'family' ilike :family" | 47 | + frag = "(e.attributes->>'family' ilike :family or exists (select 1 from entities f where f.id = e.family_id and (f.slug = :family or f.canonical_name ilike :family)))" | ||
| 43 | return frag, frag, {"family": s[7:]}, {"kind": "family", "family": s[7:]} | 48 | return frag, frag, {"family": s[7:]}, {"kind": "family", "family": s[7:]} | ||
| 44 | raise ApiError(400, "scope must be all | models | org:<slug> | family:<name>") | 49 | raise ApiError(400, "scope must be all | models | org:<slug> | family:<name>") | ||
| 45 | 50 | ||||
@@ -47,39 +52,73 @@ async def _scope(conn: Any, scope: str) -> tuple[str, str, dict[str, Any], dict[ | |||||
| 47 | @router.get("") | 52 | @router.get("") | ||
| 48 | @cached(300) | 53 | @cached(300) | ||
| 49 | async def diff(request: Request, a: str = Query(..., description="YYYY-MM-DD"), b: str = Query(..., description="YYYY-MM-DD"), scope: str = "all", | 54 | async def diff(request: Request, a: str = Query(..., description="YYYY-MM-DD"), b: str = Query(..., description="YYYY-MM-DD"), scope: str = "all", | ||
| 50 | − limit: int = Query(200, ge=1, le=1000)) -> dict[str, Any]: | 55 | + limit: int = Query(200, ge=1, le=1000), include: str | None = None, include_backfill: int = Query(0, ge=0, le=1)) -> dict[str, Any]: | ||
| 51 | da, db = parse_date(a, "a"), parse_date(b, "b") | 56 | da, db = parse_date(a, "a"), parse_date(b, "b") | ||
| 52 | assert da is not None and db is not None | 57 | assert da is not None and db is not None | ||
| 53 | if da > db: | 58 | if da > db: | ||
| 54 | da, db = db, da | 59 | da, db = db, da | ||
| 55 | start, end = datetime.combine(da, dtime.max, UTC), datetime.combine(db, dtime.max, UTC) | 60 | start, end = datetime.combine(da, dtime.max, UTC), datetime.combine(db, dtime.max, UTC) | ||
| 61 | + include_artifacts = "artifacts" in {x.strip() for x in (include or "").split(",")} | ||||
| 62 | + bf = "" if include_backfill else " and ev.is_backfill = false" | ||||
| 63 | + win = f"ev.occurred_at > :s and ev.occurred_at <= :e{bf}" | ||||
| 56 | async with connection() as conn: | 64 | async with connection() as conn: | ||
| 57 | ent_frag, ev_frag, params, scope_desc = await _scope(conn, scope) | 65 | ent_frag, ev_frag, params, scope_desc = await _scope(conn, scope) | ||
| 58 | − new_entities = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.merged_into is null and {ent_frag} and e.first_seen_at > :s and e.first_seen_at <= :e " | 66 | + art = "" if include_artifacts else " and e.entity_type <> 'artifact'" | ||
| 67 | + new_entities = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.merged_into is null and {ent_frag}{art} and e.first_seen_at > :s and e.first_seen_at <= :e " | ||||
| 59 | f"order by e.first_seen_at desc limit :lim", s=start, e=end, lim=limit, **params) | 68 | f"order by e.first_seen_at desc limit :lim", s=start, e=end, lim=limit, **params) | ||
| 60 | gone = await fetch_all(conn, f"""select distinct on (e.id) {ENTITY_COLS} from change_events ev join entities e on e.id = ev.entity_id left join entities eo on eo.id = e.organization_id | 69 | gone = await fetch_all(conn, f"""select distinct on (e.id) {ENTITY_COLS} from change_events ev join entities e on e.id = ev.entity_id left join entities eo on eo.id = e.organization_id | ||
| 61 | − where {ev_frag} and ev.observed_at > :s and ev.observed_at <= :e and (ev.event_type = any(cast(:gt as text[])) or e.merged_into is not null) | 70 | + where {ev_frag} and {win} and (ev.event_type = any(cast(:gt as text[])) or e.merged_into is not null) | ||
| 62 | and (e.status in ('retired','deprecated','merged') or ev.event_type in ('RETIREMENT_ANNOUNCED','DEPRECATION_ANNOUNCED')) | 71 | and (e.status in ('retired','deprecated','merged') or ev.event_type in ('RETIREMENT_ANNOUNCED','DEPRECATION_ANNOUNCED')) | ||
| 63 | order by e.id limit :lim""", s=start, e=end, gt=list(GONE_TYPES), lim=limit, **params) | 72 | order by e.id limit :lim""", s=start, e=end, gt=list(GONE_TYPES), lim=limit, **params) | ||
| 64 | 73 | ||||
| 65 | async def events(cond: str) -> list[dict[str, Any]]: | 74 | async def events(cond: str) -> list[dict[str, Any]]: | ||
| 66 | − rows = await fetch_all(conn, f"select {EVENT_COLS} from {EVENT_FROM} where {ev_frag} and ev.observed_at > :s and ev.observed_at <= :e and {cond} " | 75 | + rows = await fetch_all(conn, f"select {EVENT_COLS}, ev.occurred_at, ev.is_backfill from {EVENT_FROM} where {ev_frag} and {win} and {cond} " | ||
| 67 | − f"order by ev.importance desc, ev.observed_at desc limit :lim", s=start, e=end, lim=limit, **params) | 76 | + f"order by ev.importance desc, ev.occurred_at desc limit :lim", s=start, e=end, lim=limit, **params) | ||
| 68 | − return [change_event(r) for r in rows] | 77 | + out = [] | ||
| 78 | + for r in rows: | ||||
| 79 | + ev = change_event(r) | ||||
| 80 | + ev["occurred_at"] = r.get("occurred_at") | ||||
| 81 | + out.append(ev) | ||||
| 82 | + return out | ||||
| 69 | 83 | ||||
| 70 | prop_changes = await events("ev.property is not null and ev.category not in ('price','benchmark') and ev.event_type <> 'DOCUMENT_CHANGED'") | 84 | prop_changes = await events("ev.property is not null and ev.category not in ('price','benchmark') and ev.event_type <> 'DOCUMENT_CHANGED'") | ||
| 71 | price_changes = await events("ev.category = 'price'") | 85 | price_changes = await events("ev.category = 'price'") | ||
| 72 | bench_changes = await events("ev.category = 'benchmark'") | 86 | bench_changes = await events("ev.category = 'benchmark'") | ||
| 73 | − counts = await fetch_one(conn, f"""select (select count(*) from entities e where e.merged_into is null and {ent_frag} and e.first_seen_at > :s and e.first_seen_at <= :e) as new_entities, | 87 | + provider_changes = await events("ev.event_type in ('PROVIDER_LISTED','PROVIDER_DELISTED','NEW_PROVIDER')") | ||
| 74 | − (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and ev.observed_at > :s and ev.observed_at <= :e and ev.event_type <> 'DOCUMENT_CHANGED') as events, | 88 | + hardware_changes = await events("(ev.category = 'hardware' or ev.event_type = 'NEW_HARDWARE')") | ||
| 75 | − (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and ev.observed_at > :s and ev.observed_at <= :e and ev.category = 'price') as price_changes, | 89 | + context_changes = await events("ev.event_type in ('CONTEXT_CHANGED','MAX_OUTPUT_CHANGED')") | ||
| 76 | − (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and ev.observed_at > :s and ev.observed_at <= :e and ev.category = 'benchmark') as benchmark_changes, | 90 | + retired = await events(RETIRED_PRED) | ||
| 91 | + counts = await fetch_one(conn, f"""select (select count(*) from entities e where e.merged_into is null and {ent_frag}{art} and e.first_seen_at > :s and e.first_seen_at <= :e) as new_entities, | ||||
| 92 | + (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and {win} and ev.event_type <> 'DOCUMENT_CHANGED') as events, | ||||
| 93 | + (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and {win} and ev.category = 'price') as price_changes, | ||||
| 94 | + (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and {win} and ev.category = 'benchmark') as benchmark_changes, | ||||
| 95 | + (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and {win} and ev.event_type in ('PROVIDER_LISTED','PROVIDER_DELISTED','NEW_PROVIDER')) as provider_changes, | ||||
| 96 | + (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and {win} and ev.event_type in ('CONTEXT_CHANGED','MAX_OUTPUT_CHANGED')) as context_changes, | ||||
| 97 | + (select count(*) from change_events ev left join entities e on e.id = ev.entity_id where {ev_frag} and {win} and {RETIRED_PRED}) as retired_models, | ||||
| 77 | (select count(*) from prices p join entities e on e.id = p.model_id where {ent_frag} and p.valid_from > :s and p.valid_from <= :e) as price_rows_opened, | 98 | (select count(*) from prices p join entities e on e.id = p.model_id where {ent_frag} and p.valid_from > :s and p.valid_from <= :e) as price_rows_opened, | ||
| 78 | (select count(*) from prices p join entities e on e.id = p.model_id where {ent_frag} and p.valid_to > :s and p.valid_to <= :e) as price_rows_closed, | 99 | (select count(*) from prices p join entities e on e.id = p.model_id where {ent_frag} and p.valid_to > :s and p.valid_to <= :e) as price_rows_closed, | ||
| 79 | (select count(*) from claims c join entities e on e.id = c.entity_id where {ent_frag} and c.valid_to > :s and c.valid_to <= :e) as claims_superseded""", | 100 | (select count(*) from claims c join entities e on e.id = c.entity_id where {ent_frag} and c.valid_to > :s and c.valid_to <= :e) as claims_superseded""", | ||
| 80 | s=start, e=end, **params) | 101 | s=start, e=end, **params) | ||
| 81 | − entities_at_a = await fetch_val(conn, f"select count(*) from entities e where e.merged_into is null and {ent_frag} and e.first_seen_at <= :s", s=start, **params) | 102 | + entities_at_a = await fetch_val(conn, f"select count(*) from entities e where e.merged_into is null and {ent_frag}{art} and e.first_seen_at <= :s", s=start, **params) | ||
| 82 | − entities_at_b = await fetch_val(conn, f"select count(*) from entities e where e.merged_into is null and {ent_frag} and e.first_seen_at <= :e", e=end, **params) | 103 | + entities_at_b = await fetch_val(conn, f"select count(*) from entities e where e.merged_into is null and {ent_frag}{art} and e.first_seen_at <= :e", e=end, **params) | ||
| 104 | + # new benchmark leaders: leader of each benchmark's primary group at a vs at b (from results observed by each date) | ||||
| 105 | + meta = await benchmark_meta(conn) | ||||
| 106 | + all_rows = await load_results(conn, current_only=False) | ||||
| 107 | + by_bench: dict[str, list[dict[str, Any]]] = {} | ||||
| 108 | + for r in all_rows: | ||||
| 109 | + by_bench.setdefault(r["benchmark_id"], []).append(r) | ||||
| 110 | + new_leaders = [] | ||||
| 111 | + for bid, rows in by_bench.items(): | ||||
| 112 | + m = meta.get(bid, {}) | ||||
| 113 | + la, lb = _leaders_at(rows, start, m.get("attributes")), _leaders_at(rows, end, m.get("attributes")) | ||||
| 114 | + if lb and (la is None or la["model"]["id"] != lb["model"]["id"]): | ||||
| 115 | + new_leaders.append({"benchmark": {"id": bid, "slug": m.get("slug"), "name": m.get("name"), "category": m.get("category")}, "at_a": la, "at_b": lb}) | ||||
| 116 | + new_leaders.sort(key=lambda x: x["benchmark"]["name"] or "") | ||||
| 83 | return {"a": da.isoformat(), "b": db.isoformat(), "scope": scope_desc, "new_entities": [entity_summary(r) for r in new_entities], "gone_entities": [entity_summary(r) for r in gone], | 117 | return {"a": da.isoformat(), "b": db.isoformat(), "scope": scope_desc, "new_entities": [entity_summary(r) for r in new_entities], "gone_entities": [entity_summary(r) for r in gone], | ||
| 84 | − "property_changes": prop_changes, "price_changes": price_changes, "benchmark_changes": bench_changes, | 118 | + "property_changes": prop_changes, "price_changes": price_changes, "benchmark_changes": bench_changes, "new_benchmark_leaders": new_leaders, | ||
| 85 | − "counts": {**{k: int(v or 0) for k, v in (counts or {}).items()}, "gone_entities": len(gone), "entities_at_a": int(entities_at_a or 0), "entities_at_b": int(entities_at_b or 0)}} | 119 | + "provider_changes": provider_changes, "hardware_changes": hardware_changes, "context_changes": context_changes, "retired_models": retired, | ||
| 120 | + "counts": {**{k: int(v or 0) for k, v in (counts or {}).items()}, "gone_entities": len(gone), "new_benchmark_leaders": len(new_leaders), | ||||
| 121 | + "entities_at_a": int(entities_at_a or 0), "entities_at_b": int(entities_at_b or 0)}, | ||||
| 122 | + "include_artifacts": include_artifacts, "include_backfill": bool(include_backfill), | ||||
| 123 | + "note": "Events are keyed on occurred_at (effective date when known) and exclude back-filled history unless include_backfill=1; new_entities uses first_seen_at. " | ||||
| 124 | + "new_benchmark_leaders compares the primary-group leader computed from results observed by each date."} | ||||
modified
src/aiatlas/api/routers/entities.py
+66 −6
@@ -1,4 +1,4 @@ | |||||
| 1 | −"""/entities/{slug_or_id} and its sub-resources (timeline, history, asof, graph, sources, related).""" | 1 | +"""/entities/{slug_or_id} and its sub-resources (timeline, history, asof, graph, sources, related, claims, provenance).""" | ||
| 2 | from __future__ import annotations | 2 | from __future__ import annotations | ||
| 3 | 3 | ||||
| 4 | from datetime import UTC, datetime | 4 | from datetime import UTC, datetime | ||
@@ -21,7 +21,7 @@ from aiatlas.api.common import ( | |||||
| 21 | resolve_entity, | 21 | resolve_entity, | ||
| 22 | ) | 22 | ) | ||
| 23 | from aiatlas.api.detail import entity_detail, sources_of, timeline_of | 23 | from aiatlas.api.detail import entity_detail, sources_of, timeline_of | ||
| 24 | −from aiatlas.db import connection, fetch_all, fetch_one | 24 | +from aiatlas.db import connection, fetch_all, fetch_one, fetch_val | ||
| 25 | 25 | ||||
| 26 | router = APIRouter(prefix="/api/v1/entities", tags=["entities"]) | 26 | router = APIRouter(prefix="/api/v1/entities", tags=["entities"]) | ||
| 27 | 27 | ||||
@@ -37,12 +37,15 @@ async def get_entity(request: Request, slug_or_id: str) -> dict[str, Any]: | |||||
| 37 | @router.get("/{slug_or_id}/timeline") | 37 | @router.get("/{slug_or_id}/timeline") | ||
| 38 | @cached(60) | 38 | @cached(60) | ||
| 39 | async def entity_timeline(request: Request, slug_or_id: str, limit: int = Query(50, ge=1, le=200), before: str | None = None, | 39 | async def entity_timeline(request: Request, slug_or_id: str, limit: int = Query(50, ge=1, le=200), before: str | None = None, | ||
| 40 | − include_documents: int = Query(0, ge=0, le=1)) -> dict[str, Any]: | 40 | + include_documents: int = Query(0, ge=0, le=1), include_backfill: int = Query(0, ge=0, le=1), | ||
| 41 | + date_field: str = Query("occurred", pattern="^(occurred|observed)$")) -> dict[str, Any]: | ||||
| 41 | before_ts = parse_ts(before, "before") | 42 | before_ts = parse_ts(before, "before") | ||
| 42 | async with connection() as conn: | 43 | async with connection() as conn: | ||
| 43 | row = await resolve_entity(conn, slug_or_id) | 44 | row = await resolve_entity(conn, slug_or_id) | ||
| 44 | − items = await timeline_of(conn, row["id"], row["entity_type"], limit=limit, before=before_ts, include_documents=bool(include_documents)) | 45 | + items = await timeline_of(conn, row["id"], row["entity_type"], limit=limit, before=before_ts, include_documents=bool(include_documents), | ||
| 45 | − return {"items": items, "next_before": items[-1]["observed_at"] if len(items) == limit else None} | 46 | + include_backfill=bool(include_backfill), date_field=date_field) | ||
| 47 | + cursor_key = "observed_at" if date_field == "observed" else "occurred_at" | ||||
| 48 | + return {"items": items, "next_before": items[-1][cursor_key] if len(items) == limit else None, "date_field": date_field, "include_backfill": bool(include_backfill)} | ||||
| 46 | 49 | ||||
| 47 | 50 | ||||
| 48 | @router.get("/{slug_or_id}/history") | 51 | @router.get("/{slug_or_id}/history") | ||
@@ -56,6 +59,61 @@ async def entity_history(request: Request, slug_or_id: str, property: str | None | |||||
| 56 | return {"items": [claim_row(r) for r in rows]} | 59 | return {"items": [claim_row(r) for r in rows]} | ||
| 57 | 60 | ||||
| 58 | 61 | ||||
| 62 | +@router.get("/{slug_or_id}/claims") | ||||
| 63 | +@cached(120) | ||||
| 64 | +async def entity_claims(request: Request, slug_or_id: str, property: str | None = Query(None, max_length=120), status: str | None = Query(None, max_length=40), | ||||
| 65 | + limit: int = Query(200, ge=1, le=2000), offset: int = Query(0, ge=0)) -> dict[str, Any]: | ||||
| 66 | + """Public claim list (current by default; `status=all|superseded|conflicting|retracted`), newest first, with claim ids for `/claims/{id}`.""" | ||||
| 67 | + async with connection() as conn: | ||||
| 68 | + row = await resolve_entity(conn, slug_or_id) | ||||
| 69 | + where = ["c.entity_id = :id"] | ||||
| 70 | + params: dict[str, Any] = {"id": row["id"], "lim": limit, "off": offset} | ||||
| 71 | + if property: | ||||
| 72 | + where.append("c.property = :p") | ||||
| 73 | + params["p"] = property | ||||
| 74 | + st = status or "current" | ||||
| 75 | + if st != "all": | ||||
| 76 | + where.append("c.status = :st") | ||||
| 77 | + params["st"] = st | ||||
| 78 | + rows = await fetch_all(conn, f"select {CLAIM_COLS}, c.entity_id, c.snapshot_id, c.run_id, c.extractor_version, c.value_raw from {CLAIM_FROM} where {' and '.join(where)} " | ||||
| 79 | + f"order by c.property, c.valid_from desc limit :lim offset :off", **params) | ||||
| 80 | + total = await fetch_val(conn, f"select count(*) from claims c where {' and '.join(where)}", **{k: v for k, v in params.items() if k not in ('lim', 'off')}) | ||||
| 81 | + items = [{**claim_row(r), "snapshot_id": r.get("snapshot_id"), "run_id": r.get("run_id"), "value_raw": r.get("value_raw")} for r in rows] | ||||
| 82 | + return {"entity": entity_summary(row), "items": items, "total": int(total or 0), "limit": limit, "offset": offset, "status": st} | ||||
| 83 | + | ||||
| 84 | + | ||||
| 85 | +@router.get("/{slug_or_id}/provenance/{property}") | ||||
| 86 | +@cached(120) | ||||
| 87 | +async def entity_provenance(request: Request, slug_or_id: str, property: str) -> dict[str, Any]: | ||||
| 88 | + """Evidence-drawer payload for one property: current value, source, tier, extractor, confidence, conflicts, history count, snapshot id (no raw content).""" | ||||
| 89 | + async with connection() as conn: | ||||
| 90 | + row = await resolve_entity(conn, slug_or_id) | ||||
| 91 | + cur = await fetch_one(conn, f"select {CLAIM_COLS}, c.snapshot_id, c.run_id, c.extractor_version, c.value_raw, s.tier as source_tier, s.domain as source_domain, " | ||||
| 92 | + f"s.id as source_id from {CLAIM_FROM} where c.entity_id = :id and c.property = :p and c.status = 'current' order by c.tier, c.valid_from desc limit 1", | ||||
| 93 | + id=row["id"], p=property) | ||||
| 94 | + conflicts = await fetch_all(conn, f"select {CLAIM_COLS}, c.snapshot_id from {CLAIM_FROM} where c.entity_id = :id and c.property = :p and c.status = 'conflicting' " | ||||
| 95 | + f"order by c.tier, c.observed_at desc limit 20", id=row["id"], p=property) | ||||
| 96 | + history = await fetch_val(conn, "select count(*) from claims c where c.entity_id = :id and c.property = :p", id=row["id"], p=property) | ||||
| 97 | + snap = None | ||||
| 98 | + if cur and cur.get("snapshot_id"): | ||||
| 99 | + snap = await fetch_one(conn, "select s.id, s.observed_at, d.url as document_url, d.title, s.raw_path is not null as archived from snapshots s join documents d on d.id = s.document_id where s.id = :id", | ||||
| 100 | + id=cur["snapshot_id"]) | ||||
| 101 | + attrs = row.get("attributes") or {} | ||||
| 102 | + prov = (row.get("provenance") or {}).get(property) or {} | ||||
| 103 | + if not cur and property not in attrs: | ||||
| 104 | + raise ApiError(404, f"no claim for property {property!r}") | ||||
| 105 | + return {"entity": {"id": row["id"], "slug": row["slug"], "name": row["canonical_name"], "entity_type": row["entity_type"]}, "property": property, | ||||
| 106 | + "value": cur["value"] if cur else attrs.get(property), "value_raw": cur.get("value_raw") if cur else None, "unit": (cur or {}).get("unit") or prov.get("unit"), | ||||
| 107 | + "source": {"id": (cur or {}).get("source_id") or prov.get("source_id"), "name": (cur or {}).get("source_name") or prov.get("source_name"), | ||||
| 108 | + "domain": (cur or {}).get("source_domain"), "url": (cur or {}).get("source_url") or prov.get("url")}, | ||||
| 109 | + "tier": (cur or {}).get("tier") or prov.get("tier"), "confidence": (cur or {}).get("confidence") or prov.get("confidence"), | ||||
| 110 | + "extractor": (cur or {}).get("extractor") or prov.get("extractor"), "extractor_version": (cur or {}).get("extractor_version"), | ||||
| 111 | + "observed_at": (cur or {}).get("observed_at") or prov.get("observed_at"), "effective_at": (cur or {}).get("effective_at"), | ||||
| 112 | + "valid_since": (cur or {}).get("valid_from"), "claim_id": (cur or {}).get("id"), "run_id": (cur or {}).get("run_id"), | ||||
| 113 | + "snapshot_id": (cur or {}).get("snapshot_id"), "snapshot": snap, "conflicts": [claim_row(c) for c in conflicts], "history_count": int(history or 0), | ||||
| 114 | + "note": None if cur else "value materialised in attributes without a current claim row (curated or inherited)"} | ||||
| 115 | + | ||||
| 116 | + | ||||
| 59 | @router.get("/{slug_or_id}/asof") | 117 | @router.get("/{slug_or_id}/asof") | ||
| 60 | @cached(300) | 118 | @cached(300) | ||
| 61 | async def entity_asof(request: Request, slug_or_id: str, date: str = Query(..., description="YYYY-MM-DD")) -> dict[str, Any]: | 119 | async def entity_asof(request: Request, slug_or_id: str, date: str = Query(..., description="YYYY-MM-DD")) -> dict[str, Any]: | ||
@@ -126,6 +184,8 @@ async def entity_related(request: Request, slug_or_id: str, limit: int = Query(1 | |||||
| 126 | union all | 184 | union all | ||
| 127 | select e.id, 4 from entities e where :fam <> '' and e.entity_type = :t and e.attributes->>'family' = :fam and e.id <> :id | 185 | select e.id, 4 from entities e where :fam <> '' and e.entity_type = :t and e.attributes->>'family' = :fam and e.id <> :id | ||
| 128 | union all | 186 | union all | ||
| 187 | + select e.id, 5 from entities e where :fid <> '' and e.entity_type = :t and e.family_id = :fid and e.id <> :id | ||||
| 188 | + union all | ||||
| 129 | select r2.subject_id, 2 from relations r1 join relations r2 on r2.object_id = r1.object_id and r2.predicate = r1.predicate | 189 | select r2.subject_id, 2 from relations r1 join relations r2 on r2.object_id = r1.object_id and r2.predicate = r1.predicate | ||
| 130 | where r1.subject_id = :id and r1.valid_to is null and r2.valid_to is null and r2.subject_id <> :id | 190 | where r1.subject_id = :id and r1.valid_to is null and r2.valid_to is null and r2.subject_id <> :id | ||
| 131 | union all | 191 | union all | ||
@@ -134,7 +194,7 @@ async def entity_related(request: Request, slug_or_id: str, limit: int = Query(1 | |||||
| 134 | scored as (select id, sum(w) as w from cand group by id) | 194 | scored as (select id, sum(w) as w from cand group by id) | ||
| 135 | select {ENTITY_COLS} from scored join entities e on e.id = scored.id left join entities eo on eo.id = e.organization_id | 195 | select {ENTITY_COLS} from scored join entities e on e.id = scored.id left join entities eo on eo.id = e.organization_id | ||
| 136 | where e.merged_into is null order by scored.w desc, coalesce((e.quality->>'score')::float, 0) desc, e.updated_at desc limit :lim""", | 196 | where e.merged_into is null order by scored.w desc, coalesce((e.quality->>'score')::float, 0) desc, e.updated_at desc limit :lim""", | ||
| 137 | − id=row["id"], t=row["entity_type"], org=row.get("organization_id") or "", fam=str(fam or ""), lim=limit) | 197 | + id=row["id"], t=row["entity_type"], org=row.get("organization_id") or "", fam=str(fam or ""), fid=row.get("family_id") or "", lim=limit) | ||
| 138 | return {"items": [entity_summary(r) for r in rows]} | 198 | return {"items": [entity_summary(r) for r in rows]} | ||
| 139 | 199 | ||||
| 140 | 200 | ||||
added
src/aiatlas/api/routers/families.py
+145 −0
@@ -0,0 +1,145 @@ | |||||
| 1 | +"""/families · /families/{slug} — model families (API 1.1). Canonical `model_family` entities first; pre-canonicalisation data is served | ||||
| 2 | +from the legacy `attributes.family` label (items flagged `canonical: false`).""" | ||||
| 3 | +from __future__ import annotations | ||||
| 4 | + | ||||
| 5 | +from collections import defaultdict | ||||
| 6 | +from typing import Any | ||||
| 7 | + | ||||
| 8 | +from fastapi import APIRouter, Query, Request | ||||
| 9 | + | ||||
| 10 | +from aiatlas.api.common import ENTITY_COLS, ENTITY_FROM, ApiError, cached, entity_summary | ||||
| 11 | +from aiatlas.api.detail import LINEAGE_PREDICATES | ||||
| 12 | +from aiatlas.db import connection, fetch_all, fetch_one, fetch_val | ||||
| 13 | +from aiatlas.ontology.licenses import LICENSES, normalize_license | ||||
| 14 | +from aiatlas.services.frontier import all_primary_groups, rank_rows | ||||
| 15 | + | ||||
| 16 | +router = APIRouter(prefix="/api/v1/families", tags=["families"]) | ||||
| 17 | +MEMBER_SQL = "((f.id is not null and e.family_id = f.id) or (f.id is null and e.attributes->>'family' = f.label))" | ||||
| 18 | + | ||||
| 19 | + | ||||
| 20 | +def _f(v: Any) -> float | None: | ||||
| 21 | + if v is None or isinstance(v, bool): | ||||
| 22 | + return None | ||||
| 23 | + try: | ||||
| 24 | + return float(v) | ||||
| 25 | + except (TypeError, ValueError): | ||||
| 26 | + return None | ||||
| 27 | + | ||||
| 28 | + | ||||
| 29 | +def _aggregate(members: list[dict[str, Any]], ranks: dict[str, dict[str, int]]) -> dict[str, Any]: | ||||
| 30 | + params = [p for p in (_f((m["attributes"] or {}).get("parameter_count")) for m in members) if p] | ||||
| 31 | + dates = sorted(str((m["attributes"] or {}).get("release_date") or "")[:10] for m in members if (m["attributes"] or {}).get("release_date")) | ||||
| 32 | + mods: set[str] = set() | ||||
| 33 | + lics: dict[str, int] = defaultdict(int) | ||||
| 34 | + best: dict[str, tuple[int, str]] = {} | ||||
| 35 | + for m in members: | ||||
| 36 | + a = m["attributes"] or {} | ||||
| 37 | + for k in ("modalities", "modalities_input", "modalities_output"): | ||||
| 38 | + if isinstance(a.get(k), list): | ||||
| 39 | + mods |= {str(x).lower() for x in a[k]} | ||||
| 40 | + key = a.get("license_key") or normalize_license(a.get("license")) | ||||
| 41 | + if key or a.get("license"): | ||||
| 42 | + lics[key or str(a.get("license"))] += 1 | ||||
| 43 | + for b, rk in ranks.get(m["id"], {}).items(): | ||||
| 44 | + if b not in best or rk < best[b][0]: | ||||
| 45 | + best[b] = (rk, m["slug"]) | ||||
| 46 | + return {"model_count": len(members), "first_release": dates[0] if dates else None, "last_release": dates[-1] if dates else None, | ||||
| 47 | + "param_range": {"min": min(params), "max": max(params)} if params else None, "modalities": sorted(mods), | ||||
| 48 | + "licenses": [{"key": k, "label": LICENSES[k].label if k in LICENSES else k, "models": n} for k, n in sorted(lics.items(), key=lambda kv: -kv[1])], | ||||
| 49 | + "benchmark_best": {b: {"rank": rk, "model": slug} for b, (rk, slug) in sorted(best.items())}} | ||||
| 50 | + | ||||
| 51 | + | ||||
| 52 | +async def _families(conn: Any) -> list[dict[str, Any]]: | ||||
| 53 | + """Canonical model_family entities + legacy labels not yet backed by an entity.""" | ||||
| 54 | + fam = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.entity_type = 'model_family' and e.merged_into is null order by e.canonical_name") | ||||
| 55 | + labels = await fetch_all(conn, """select e.attributes->>'family' as label, min(e.organization_id) as organization_id, count(*) as n from entities e | ||||
| 56 | + where e.entity_type = 'model' and e.merged_into is null and e.family_id is null and e.attributes ? 'family' group by 1 order by 1""") | ||||
| 57 | + out = [{"id": r["id"], "slug": r["slug"], "name": r["canonical_name"], "label": r["canonical_name"], "canonical": True, "summary": entity_summary(r), "organization_id": r["organization_id"]} for r in fam] | ||||
| 58 | + names = {x["name"].lower() for x in out} | ||||
| 59 | + for r in labels: | ||||
| 60 | + if r["label"] and r["label"].lower() not in names: | ||||
| 61 | + out.append({"id": None, "slug": r["label"].lower().replace(" ", "-").replace(".", "-"), "name": r["label"], "label": r["label"], "canonical": False, "summary": None, "organization_id": r["organization_id"]}) | ||||
| 62 | + return out | ||||
| 63 | + | ||||
| 64 | + | ||||
| 65 | +@router.get("") | ||||
| 66 | +@cached(300) | ||||
| 67 | +async def list_families(request: Request, q: str | None = Query(None, max_length=120), org: str | None = None, limit: int = Query(50, ge=1, le=200), offset: int = Query(0, ge=0), | ||||
| 68 | + sort: str = Query("models", pattern="^(models|name|last_release)$")) -> dict[str, Any]: | ||||
| 69 | + async with connection() as conn: | ||||
| 70 | + fams = await _families(conn) | ||||
| 71 | + members = await fetch_all(conn, f"""select e.id, e.slug, e.canonical_name, e.attributes, e.family_id, e.attributes->>'family' as label, e.organization_id, eo.slug as org_slug, eo.canonical_name as org_name | ||||
| 72 | + from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and (e.family_id is not null or e.attributes ? 'family')""") | ||||
| 73 | + groups = await all_primary_groups(conn) | ||||
| 74 | + ranks: dict[str, dict[str, int]] = defaultdict(dict) | ||||
| 75 | + for g in groups.values(): | ||||
| 76 | + for r in rank_rows(g["rows"], g["higher_is_better"]): | ||||
| 77 | + ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"] | ||||
| 78 | + by_fid: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 79 | + by_label: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 80 | + for m in members: | ||||
| 81 | + if m["family_id"]: | ||||
| 82 | + by_fid[m["family_id"]].append(m) | ||||
| 83 | + elif m["label"]: | ||||
| 84 | + by_label[m["label"].lower()].append(m) | ||||
| 85 | + items = [] | ||||
| 86 | + for f in fams: | ||||
| 87 | + mem = by_fid.get(f["id"], []) if f["id"] else by_label.get(f["name"].lower(), []) | ||||
| 88 | + if not mem and not f["canonical"]: | ||||
| 89 | + continue | ||||
| 90 | + if q and q.lower() not in f["name"].lower(): | ||||
| 91 | + continue | ||||
| 92 | + orgs = defaultdict(int) | ||||
| 93 | + for m in mem: | ||||
| 94 | + if m["organization_id"]: | ||||
| 95 | + orgs[(m["organization_id"], m["org_slug"], m["org_name"])] += 1 | ||||
| 96 | + top_org = max(orgs.items(), key=lambda kv: kv[1])[0] if orgs else None | ||||
| 97 | + if org and not (top_org and (top_org[1] == org or top_org[0] == org or (top_org[2] or "").lower() == org.lower())): | ||||
| 98 | + continue | ||||
| 99 | + items.append({"id": f["id"], "slug": f["slug"], "name": f["name"], "canonical": f["canonical"], "entity_type": "model_family", | ||||
| 100 | + "organization": {"id": top_org[0], "slug": top_org[1], "name": top_org[2]} if top_org else None, **_aggregate(mem, ranks)}) | ||||
| 101 | + key = {"models": lambda x: (-x["model_count"], x["name"]), "name": lambda x: x["name"].lower(), "last_release": lambda x: (x["last_release"] or "", x["name"])}[sort] | ||||
| 102 | + items.sort(key=key, reverse=(sort == "last_release")) | ||||
| 103 | + return {"items": items[offset:offset + limit], "total": len(items), "limit": limit, "offset": offset, | ||||
| 104 | + "note": "canonical: true = model_family entity; false = legacy attributes.family label awaiting canonicalisation. benchmark_best = best rank of any member in each benchmark's primary group."} | ||||
| 105 | + | ||||
| 106 | + | ||||
| 107 | +@router.get("/{slug}") | ||||
| 108 | +@cached(300) | ||||
| 109 | +async def family_detail(request: Request, slug: str, limit: int = Query(100, ge=1, le=500)) -> dict[str, Any]: | ||||
| 110 | + async with connection() as conn: | ||||
| 111 | + fam = await fetch_one(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.entity_type = 'model_family' and e.merged_into is null and (e.slug = :s or e.id = :s or e.canonical_name ilike :s) limit 1", s=slug) | ||||
| 112 | + label = fam["canonical_name"] if fam else slug | ||||
| 113 | + if fam: | ||||
| 114 | + members = await fetch_all(conn, f"select {ENTITY_COLS}, e.family_id from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and (e.family_id = :fid or e.attributes->>'family' ilike :label) " | ||||
| 115 | + f"order by e.attributes->>'release_date' asc nulls last, e.canonical_name limit :lim", fid=fam["id"], label=label, lim=limit) | ||||
| 116 | + else: | ||||
| 117 | + members = await fetch_all(conn, f"select {ENTITY_COLS}, e.family_id from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and e.family_id is null " | ||||
| 118 | + f"and (e.attributes->>'family' ilike :label or replace(replace(lower(e.attributes->>'family'), ' ', '-'), '.', '-') = lower(:slug)) " | ||||
| 119 | + f"order by e.attributes->>'release_date' asc nulls last, e.canonical_name limit :lim", label=label, slug=slug, lim=limit) | ||||
| 120 | + if not fam and not members: | ||||
| 121 | + raise ApiError(404, f"family {slug!r} not found") | ||||
| 122 | + ids = [m["id"] for m in members] | ||||
| 123 | + arts = await fetch_val(conn, "select count(*) from entities where entity_type = 'artifact' and merged_into is null and canonical_id = any(cast(:ids as text[]))", ids=ids) if ids else 0 | ||||
| 124 | + providers = await fetch_all(conn, f"select distinct {ENTITY_COLS} from prices p join entities e on e.id = p.provider_id left join entities eo on eo.id = e.organization_id where p.model_id = any(cast(:ids as text[])) and p.valid_to is null order by e.canonical_name", ids=ids) if ids else [] | ||||
| 125 | + edges = await fetch_all(conn, "select r.subject_id, r.predicate, r.object_id from relations r where r.valid_to is null and r.predicate = any(cast(:p as text[])) and r.subject_id = any(cast(:ids as text[])) and r.object_id = any(cast(:ids as text[]))", | ||||
| 126 | + p=[*LINEAGE_PREDICATES, "superseded_by"], ids=ids) if ids else [] | ||||
| 127 | + events = await fetch_all(conn, """select ev.entity_id, ev.event_type, ev.summary, ev.occurred_at, ev.importance from change_events ev where ev.entity_id = any(cast(:ids as text[])) | ||||
| 128 | + and ev.event_type in ('NEW_MODEL','RELEASE','DEPRECATION_ANNOUNCED','RETIREMENT_ANNOUNCED','STATUS_CHANGED') order by ev.occurred_at asc limit 500""", ids=ids) if ids else [] | ||||
| 129 | + groups = await all_primary_groups(conn) | ||||
| 130 | + ranks: dict[str, dict[str, int]] = defaultdict(dict) | ||||
| 131 | + for g in groups.values(): | ||||
| 132 | + for r in rank_rows(g["rows"], g["higher_is_better"]): | ||||
| 133 | + ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"] | ||||
| 134 | + mem_summ = [] | ||||
| 135 | + for m in members: | ||||
| 136 | + a = m["attributes"] or {} | ||||
| 137 | + mem_summ.append({"model": entity_summary(m), "key_facts": {k: a.get(k) for k in ("release_date", "parameter_count", "active_parameter_count", "context_length", "openness", "license", "modalities", "status") if a.get(k) not in (None, "", [])}, | ||||
| 138 | + "benchmark_ranks": dict(sorted(ranks.get(m["id"], {}).items()))}) | ||||
| 139 | + timeline = [{"date": str((m["attributes"] or {}).get("release_date") or "")[:10] or None, "kind": "release", "model": {"id": m["id"], "slug": m["slug"], "name": m["canonical_name"]}} for m in members] | ||||
| 140 | + timeline += [{"date": e["occurred_at"], "kind": e["event_type"], "summary": e["summary"], "model_id": e["entity_id"]} for e in events if e["event_type"] != "NEW_MODEL"] | ||||
| 141 | + timeline.sort(key=lambda x: str(x["date"] or "")) | ||||
| 142 | + return {"id": fam["id"] if fam else None, "slug": fam["slug"] if fam else slug, "name": label, "canonical": bool(fam), "entity_type": "model_family", | ||||
| 143 | + "summary": entity_summary(fam) if fam else None, **_aggregate(members, ranks), "members": mem_summ, "artifacts_count": int(arts or 0), | ||||
| 144 | + "providers": [entity_summary(p) for p in providers], "lineage": [{"source": e["subject_id"], "target": e["object_id"], "predicate": e["predicate"]} for e in edges], | ||||
| 145 | + "timeline": timeline, "note": None if fam else "served from the legacy attributes.family label (no model_family entity yet)"} | ||||
added
src/aiatlas/api/routers/graph.py
+78 −0
@@ -0,0 +1,78 @@ | |||||
| 1 | +"""/graph/explore — neighbourhood exploration filtered by predicate sets per mode (API 1.1). `/entities/{slug}/graph` (v1) is unchanged.""" | ||||
| 2 | +from __future__ import annotations | ||||
| 3 | + | ||||
| 4 | +from typing import Any | ||||
| 5 | + | ||||
| 6 | +from fastapi import APIRouter, Query, Request | ||||
| 7 | + | ||||
| 8 | +from aiatlas.api.common import ApiError, cached, resolve_entity | ||||
| 9 | +from aiatlas.db import connection, fetch_all | ||||
| 10 | + | ||||
| 11 | +router = APIRouter(prefix="/api/v1/graph", tags=["graph"]) | ||||
| 12 | + | ||||
| 13 | +MODES: dict[str, tuple[str, ...]] = { | ||||
| 14 | + "lineage": ("artifact_of", "quantized_from", "fine_tuned_from", "distilled_from", "merged_from", "derived_from", "superseded_by", "member_of_family", "develops"), | ||||
| 15 | + "research": ("authored", "described_by", "published_by", "uses_dataset", "evaluates_on", "evaluated_on"), | ||||
| 16 | + "company": ("develops", "owns", "operates", "manufactures", "member_of_family"), | ||||
| 17 | + "benchmark": ("evaluated_on", "evaluates_on", "variant_of"), | ||||
| 18 | + "dataset": ("uses_dataset", "derived_from", "subset_of"), | ||||
| 19 | + "provider": ("available_through",), | ||||
| 20 | + "hardware": ("runs_on", "uses", "manufactures"), | ||||
| 21 | +} | ||||
| 22 | +KEY_ATTRS = ("parameter_count", "context_length", "release_date", "openness", "published_at", "category", "memory_gb", "kind") | ||||
| 23 | + | ||||
| 24 | + | ||||
| 25 | +@router.get("/explore") | ||||
| 26 | +@cached(300) | ||||
| 27 | +async def explore(request: Request, node: str = Query(...), mode: str = Query("lineage"), depth: int = Query(1, ge=1, le=2), limit: int = Query(150, ge=2, le=150)) -> dict[str, Any]: | ||||
| 28 | + if mode not in MODES: | ||||
| 29 | + raise ApiError(400, f"mode must be one of {', '.join(MODES)}") | ||||
| 30 | + preds = list(MODES[mode]) | ||||
| 31 | + async with connection() as conn: | ||||
| 32 | + root_row = await resolve_entity(conn, node) | ||||
| 33 | + root = root_row["id"] | ||||
| 34 | + seen: dict[str, int] = {root: 0} | ||||
| 35 | + frontier = [root] | ||||
| 36 | + edges: dict[tuple[str, str, str], dict[str, Any]] = {} | ||||
| 37 | + truncated = False | ||||
| 38 | + for level in range(1, depth + 1): | ||||
| 39 | + if not frontier: | ||||
| 40 | + break | ||||
| 41 | + rows = await fetch_all(conn, """select r.subject_id, r.predicate, r.object_id, r.attributes, r.tier from relations r | ||||
| 42 | + where r.valid_to is null and r.predicate = any(cast(:preds as text[])) | ||||
| 43 | + and (r.subject_id = any(cast(:ids as text[])) or r.object_id = any(cast(:ids as text[]))) | ||||
| 44 | + order by r.tier, r.observed_at desc limit :lim""", preds=preds, ids=frontier, lim=limit * 4) | ||||
| 45 | + nxt: list[str] = [] | ||||
| 46 | + for r in rows: | ||||
| 47 | + other = r["object_id"] if r["subject_id"] in seen else r["subject_id"] | ||||
| 48 | + if other not in seen: | ||||
| 49 | + if len(seen) >= limit: | ||||
| 50 | + truncated = True | ||||
| 51 | + continue | ||||
| 52 | + seen[other] = level | ||||
| 53 | + nxt.append(other) | ||||
| 54 | + edges[(r["subject_id"], r["object_id"], r["predicate"])] = {"source": r["subject_id"], "target": r["object_id"], "predicate": r["predicate"], "attributes": r["attributes"] or {}, "tier": r["tier"]} | ||||
| 55 | + frontier = nxt | ||||
| 56 | + # organisation edges for the company mode come from entities.organization_id as well | ||||
| 57 | + if mode == "company": | ||||
| 58 | + org_rows = await fetch_all(conn, "select id, organization_id from entities where organization_id is not null and merged_into is null and (id = any(cast(:ids as text[])) or organization_id = any(cast(:ids as text[]))) limit :lim", | ||||
| 59 | + ids=list(seen), lim=limit * 4) | ||||
| 60 | + for r in org_rows: | ||||
| 61 | + for x in (r["id"], r["organization_id"]): | ||||
| 62 | + if x not in seen: | ||||
| 63 | + if len(seen) >= limit: | ||||
| 64 | + truncated = True | ||||
| 65 | + break | ||||
| 66 | + seen[x] = depth | ||||
| 67 | + edges.setdefault((r["organization_id"], r["id"], "develops"), {"source": r["organization_id"], "target": r["id"], "predicate": "develops", "attributes": {"from": "organization_id"}, "tier": None}) | ||||
| 68 | + nodes = await fetch_all(conn, """select e.id, e.slug, e.canonical_name, e.entity_type, e.attributes, e.merged_into, e.artifact_kind, o.canonical_name as org_name, o.slug as org_slug | ||||
| 69 | + from entities e left join entities o on o.id = e.organization_id where e.id = any(cast(:ids as text[]))""", ids=list(seen)) | ||||
| 70 | + node_ids = {n["id"] for n in nodes} | ||||
| 71 | + out_edges = [e for e in edges.values() if e["source"] in node_ids and e["target"] in node_ids] | ||||
| 72 | + counts: dict[str, int] = {} | ||||
| 73 | + for n in nodes: | ||||
| 74 | + counts[n["entity_type"]] = counts.get(n["entity_type"], 0) + 1 | ||||
| 75 | + return {"root": root, "mode": mode, "depth": depth, "predicates": preds, | ||||
| 76 | + "nodes": [{"id": n["id"], "slug": n["slug"], "name": n["canonical_name"], "entity_type": n["entity_type"], "org": n["org_name"], "org_slug": n["org_slug"], "level": seen.get(n["id"], 0), | ||||
| 77 | + "artifact_kind": n["artifact_kind"], "attributes": {k: (n["attributes"] or {}).get(k) for k in KEY_ATTRS if (n["attributes"] or {}).get(k) not in (None, "", [])}} for n in nodes], | ||||
| 78 | + "edges": out_edges, "truncated": truncated, "counts": {"nodes": len(nodes), "edges": len(out_edges), "by_type": counts}} | ||||
modified
src/aiatlas/api/routers/hardware.py
+30 −0
@@ -86,6 +86,36 @@ async def hardware_fit(request: Request, memory_gb: float = Query(..., gt=0, le= | |||||
| 86 | "counts": {"fits": sum(1 for i in items if i["fits"]), "evaluated": len(items)}, "items": items[:limit]} | 86 | "counts": {"fits": sum(1 for i in items if i["fits"]), "evaluated": len(items)}, "items": items[:limit]} | ||
| 87 | 87 | ||||
| 88 | 88 | ||||
| 89 | +@router.get("/{slug}/fit") | ||||
| 90 | +@cached(300) | ||||
| 91 | +async def hardware_entity_fit(request: Request, slug: str, quant: str = Query("4bit"), context: int = Query(8192, ge=0, le=10_000_000), memory_gb: float | None = Query(None, gt=0), | ||||
| 92 | + gpu_count: int = Query(1, ge=1, le=8), limit: int = Query(100, ge=1, le=500), openness: str | None = None) -> dict[str, Any]: | ||||
| 93 | + """ESTIMATED fit of canonical models on ONE hardware entity (largest memory configuration unless `memory_gb` picks one).""" | ||||
| 94 | + from aiatlas.api.common import openness_values, resolve_entity | ||||
| 95 | + | ||||
| 96 | + q = hf.normalize_quant(quant) | ||||
| 97 | + async with connection() as conn: | ||||
| 98 | + hw = await resolve_entity(conn, slug, ("hardware",)) | ||||
| 99 | + options = hf.hardware_memory_options(hw.get("attributes")) | ||||
| 100 | + if not options: | ||||
| 101 | + return {"hardware": entity_summary(hw), "memory_options_gb": [], "items": [], "estimated": True, "note": "no memory_gb recorded for this hardware — nothing is estimated"} | ||||
| 102 | + mem = memory_gb if memory_gb is not None else max(options) | ||||
| 103 | + where = "e.entity_type = 'model' and e.merged_into is null and e.attributes ? 'parameter_count'" | ||||
| 104 | + params: dict[str, Any] = {} | ||||
| 105 | + if openness: | ||||
| 106 | + where += " and e.attributes->>'openness' = any(cast(:openness as text[]))" | ||||
| 107 | + params["openness"] = openness_values(openness) | ||||
| 108 | + rows = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where {where} order by {attr_num('parameter_count')} desc nulls last limit 3000", **params) | ||||
| 109 | + items = [] | ||||
| 110 | + for r in rows: | ||||
| 111 | + f = hf.fit_detailed(r["attributes"], mem, quant=q, context=context, gpu_count=gpu_count) | ||||
| 112 | + if f: | ||||
| 113 | + items.append({"model": entity_summary(r), **f}) | ||||
| 114 | + items.sort(key=lambda x: (not x["fits"], -(x.get("parameter_count") or 0) if x["fits"] else (x.get("parameter_count") or 0))) | ||||
| 115 | + return {"hardware": entity_summary(hw), "memory_options_gb": options, "inputs": {"memory_gb": mem, "quant": q, "context": context, "gpu_count": gpu_count}, "estimated": True, | ||||
| 116 | + "assumptions": hf.ASSUMPTIONS, "runtimes": (hw.get("attributes") or {}).get("runtimes"), "counts": {"fits": sum(1 for i in items if i["fits"]), "evaluated": len(items)}, "items": items[:limit]} | ||||
| 117 | + | ||||
| 118 | + | ||||
| 89 | @router.get("/{slug}") | 119 | @router.get("/{slug}") | ||
| 90 | @cached(300) | 120 | @cached(300) | ||
| 91 | async def get_hardware(request: Request, slug: str) -> dict[str, Any]: | 121 | async def get_hardware(request: Request, slug: str) -> dict[str, Any]: | ||
added
src/aiatlas/api/routers/intelligence.py
+565 −0
@@ -0,0 +1,565 @@ | |||||
| 1 | +"""Intelligence surfaces (API 1.1) — deterministic, no LLM, no composite scores: | ||||
| 2 | +/frontier · /pareto · /pulse · /open · /find-a-model · /run-locally · /time-machine.""" | ||||
| 3 | +from __future__ import annotations | ||||
| 4 | + | ||||
| 5 | +import statistics | ||||
| 6 | +from collections import defaultdict | ||||
| 7 | +from datetime import UTC, datetime, timedelta | ||||
| 8 | +from datetime import time as dtime | ||||
| 9 | +from typing import Any | ||||
| 10 | + | ||||
| 11 | +from fastapi import APIRouter, Query, Request | ||||
| 12 | + | ||||
| 13 | +from aiatlas.api.common import ( | ||||
| 14 | + CLAIM_COLS, | ||||
| 15 | + CLAIM_FROM, | ||||
| 16 | + DOWNLOADABLE_CATEGORIES, | ||||
| 17 | + ENTITY_COLS, | ||||
| 18 | + ENTITY_FROM, | ||||
| 19 | + EVENT_COLS, | ||||
| 20 | + EVENT_FROM, | ||||
| 21 | + OPEN_CATEGORIES, | ||||
| 22 | + PRICE_COLS, | ||||
| 23 | + PRICE_FROM, | ||||
| 24 | + ApiError, | ||||
| 25 | + cached, | ||||
| 26 | + change_event, | ||||
| 27 | + csv, | ||||
| 28 | + deployment_row, | ||||
| 29 | + entity_summary, | ||||
| 30 | + num_expr, | ||||
| 31 | + openness_values, | ||||
| 32 | + parse_date, | ||||
| 33 | + resolve_entity, | ||||
| 34 | + resolve_id, | ||||
| 35 | +) | ||||
| 36 | +from aiatlas.db import connection, fetch_all, fetch_one, fetch_val | ||||
| 37 | +from aiatlas.ontology.licenses import LICENSES, normalize_license | ||||
| 38 | +from aiatlas.services import hardware_fit as hf | ||||
| 39 | +from aiatlas.services.finder import USE_CASES, find_models | ||||
| 40 | +from aiatlas.services.frontier import ( | ||||
| 41 | + FRONTIER_METHODOLOGY, | ||||
| 42 | + all_primary_groups, | ||||
| 43 | + benchmark_meta, | ||||
| 44 | + frontier_model_ids, | ||||
| 45 | + group_rows, | ||||
| 46 | + group_summary, | ||||
| 47 | + leader_at, | ||||
| 48 | + leaderboard_rows, | ||||
| 49 | + load_results, | ||||
| 50 | + primary_group, | ||||
| 51 | + rank_rows, | ||||
| 52 | +) | ||||
| 53 | +from aiatlas.services.pareto import pareto_frontier | ||||
| 54 | + | ||||
| 55 | +router = APIRouter(prefix="/api/v1", tags=["intelligence"]) | ||||
| 56 | +PARAMS = num_expr("e.attributes->>'parameter_count'") | ||||
| 57 | +CONTEXT = num_expr("e.attributes->>'context_length'") | ||||
| 58 | +QUALITY_BENCHMARKS = ("artificial-analysis-intelligence-index", "gpqa") | ||||
| 59 | +PARETO_X = {"output_price": "cheapest current output price (USD / 1M tokens)", "input_price": "cheapest current input price (USD / 1M tokens)", | ||||
| 60 | + "parameter_count": "total parameters", "context_length": "context window (tokens)", "memory_estimate": "ESTIMATED memory at 4-bit, 8K context (GB)"} | ||||
| 61 | + | ||||
| 62 | + | ||||
| 63 | +def _f(v: Any) -> float | None: | ||||
| 64 | + if v is None or isinstance(v, bool): | ||||
| 65 | + return None | ||||
| 66 | + try: | ||||
| 67 | + return float(v) | ||||
| 68 | + except (TypeError, ValueError): | ||||
| 69 | + return None | ||||
| 70 | + | ||||
| 71 | + | ||||
| 72 | +def _mods(attrs: dict[str, Any]) -> set[str]: | ||||
| 73 | + out: set[str] = set() | ||||
| 74 | + for k in ("modalities", "modalities_input", "modalities_output"): | ||||
| 75 | + v = attrs.get(k) | ||||
| 76 | + if isinstance(v, list): | ||||
| 77 | + out |= {str(x).lower() for x in v} | ||||
| 78 | + return {("image" if m == "vision" else "document" if m == "pdf" else m) for m in out} | ||||
| 79 | + | ||||
| 80 | + | ||||
| 81 | +async def _cheapest_prices(conn: Any, model_ids: list[str] | None = None) -> dict[str, dict[str, Any]]: | ||||
| 82 | + where = "p.valid_to is null" + (" and p.model_id = any(cast(:ids as text[]))" if model_ids is not None else "") | ||||
| 83 | + rows = await fetch_all(conn, f"""select p.model_id, min(p.output_per_mtok) filter (where p.output_per_mtok > 0) as min_output, min(p.input_per_mtok) filter (where p.input_per_mtok > 0) as min_input, | ||||
| 84 | + count(distinct p.provider_id) as providers, max(p.context_length) as max_ctx | ||||
| 85 | + from prices p where {where} group by 1""", ids=model_ids) | ||||
| 86 | + return {r["model_id"]: r for r in rows} | ||||
| 87 | + | ||||
| 88 | + | ||||
| 89 | +async def _cheapest_offer(conn: Any, model_id: str, field: str = "output") -> dict[str, Any] | None: | ||||
| 90 | + col = "p.output_per_mtok" if field == "output" else "p.input_per_mtok" | ||||
| 91 | + row = await fetch_one(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.model_id = :id and p.valid_to is null and {col} > 0 order by {col} asc limit 1", id=model_id) | ||||
| 92 | + return deployment_row(row) if row else None | ||||
| 93 | + | ||||
| 94 | + | ||||
| 95 | +# ------------------------------------------------------------------------------------------------------------------ /frontier | ||||
| 96 | + | ||||
| 97 | + | ||||
| 98 | +@router.get("/frontier") | ||||
| 99 | +@cached(300) | ||||
| 100 | +async def frontier(request: Request, limit: int = Query(12, ge=1, le=50)) -> dict[str, Any]: | ||||
| 101 | + now = datetime.now(UTC) | ||||
| 102 | + async with connection() as conn: | ||||
| 103 | + groups = await all_primary_groups(conn) | ||||
| 104 | + fids, composition = await frontier_model_ids(conn) | ||||
| 105 | + major = await fetch_all(conn, f""" | ||||
| 106 | + select distinct on (e.id) {EVENT_COLS}, ev.occurred_at, ev.is_backfill from {EVENT_FROM} left join sources s on s.id = ev.source_id | ||||
| 107 | + where ev.event_type in ('NEW_MODEL','RELEASE') and ev.importance >= 3 and e.entity_type = 'model' and e.merged_into is null and coalesce(s.tier, 2) <= 2 | ||||
| 108 | + and (ev.is_backfill = false or e.attributes->>'release_date' >= :since) | ||||
| 109 | + order by e.id, ev.occurred_at desc""", since=(now - timedelta(days=60)).date().isoformat()) | ||||
| 110 | + major.sort(key=lambda r: (str((r.get("e_attributes") or {}).get("release_date") or ""), r["occurred_at"]), reverse=True) | ||||
| 111 | + cheapest = await fetch_one(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.valid_to is null and p.output_per_mtok > 0 and p.model_id = any(cast(:ids as text[])) order by p.output_per_mtok asc limit 1", ids=sorted(fids)) | ||||
| 112 | + cheapest_1m = await fetch_one(conn, f"""select {PRICE_COLS} from {PRICE_FROM} where p.valid_to is null and p.output_per_mtok > 0 and p.model_id = any(cast(:ids as text[])) | ||||
| 113 | + and coalesce(p.context_length, case when m.attributes->>'context_length' ~ '^[0-9]+$' then (m.attributes->>'context_length')::bigint end) >= 1000000 | ||||
| 114 | + order by p.output_per_mtok asc limit 1""", ids=sorted(fids)) | ||||
| 115 | + ctx_rows = await fetch_all(conn, f"select {ENTITY_COLS}, {CONTEXT} as ctx from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and {CONTEXT} is not null order by {CONTEXT} desc, e.canonical_name limit :lim", lim=limit) | ||||
| 116 | + open_rows = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and e.attributes->>'openness' = any(cast(:o as text[]))", o=list(OPEN_CATEGORIES)) | ||||
| 117 | + movements = await fetch_all(conn, f"""select {EVENT_COLS}, ev.occurred_at from {EVENT_FROM} where ev.is_backfill = false and ev.occurred_at > now() - interval '30 days' | ||||
| 118 | + and (ev.event_type in ('BENCHMARK_UPDATED','BENCHMARK_LEADER_CHANGED','NEW_BENCHMARK_LEADER') or ev.event_type = 'PRICE_CHANGED') | ||||
| 119 | + order by ev.occurred_at desc limit 500""") | ||||
| 120 | + all_prices = await _cheapest_prices(conn) | ||||
| 121 | + # ranks per model across primary groups | ||||
| 122 | + ranks: dict[str, dict[str, int]] = defaultdict(dict) | ||||
| 123 | + for bid, g in groups.items(): | ||||
| 124 | + for r in rank_rows(g["rows"], g["higher_is_better"]): | ||||
| 125 | + ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"] | ||||
| 126 | + bench_frontier = [] | ||||
| 127 | + agentic = [] | ||||
| 128 | + for bid, g in sorted(groups.items(), key=lambda kv: -kv[1]["n"]): | ||||
| 129 | + lb = leaderboard_rows(g) | ||||
| 130 | + if g["n"] >= 20 and lb: | ||||
| 131 | + leader, second = lb[0], (lb[1] if len(lb) > 1 else None) | ||||
| 132 | + bench_frontier.append({"benchmark": {k: g["benchmark"].get(k) for k in ("id", "slug", "name", "category")}, "group": group_summary(g), "leader": leader, "second": second, | ||||
| 133 | + "gap": round(leader["score"] - second["score"], 3) if second else None}) | ||||
| 134 | + if (g["benchmark"].get("category") or "").lower() == "agentic" and lb: | ||||
| 135 | + agentic.append({"benchmark": {k: g["benchmark"].get(k) for k in ("id", "slug", "name")}, "group": group_summary(g), "leaders": lb[:5]}) | ||||
| 136 | + open_frontier = [] | ||||
| 137 | + for r in open_rows: | ||||
| 138 | + attrs = r["attributes"] or {} | ||||
| 139 | + rk = ranks.get(r["id"], {}) | ||||
| 140 | + best = min(rk.values()) if rk else None | ||||
| 141 | + open_frontier.append({"model": entity_summary(r), "best_rank": best, "best_rank_on": min(rk, key=rk.get) if rk else None, "parameter_count": _f(attrs.get("parameter_count")), | ||||
| 142 | + "context_length": _f(attrs.get("context_length")), "ranks": dict(sorted(rk.items())), "_k": (best if best is not None else 10_000, -(_f(attrs.get("parameter_count")) or 0))}) | ||||
| 143 | + open_frontier.sort(key=lambda x: x["_k"]) | ||||
| 144 | + for x in open_frontier: | ||||
| 145 | + x.pop("_k") | ||||
| 146 | + # efficiency frontier: quality (index or GPQA primary group) vs cheapest output price | ||||
| 147 | + quality_group = None | ||||
| 148 | + for slug in QUALITY_BENCHMARKS: | ||||
| 149 | + quality_group = next((g for g in groups.values() if g["benchmark"]["slug"] == slug), None) | ||||
| 150 | + if quality_group: | ||||
| 151 | + break | ||||
| 152 | + eff_points = [] | ||||
| 153 | + if quality_group: | ||||
| 154 | + for r in rank_rows(quality_group["rows"], quality_group["higher_is_better"]): | ||||
| 155 | + pr = all_prices.get(r["model_id"]) | ||||
| 156 | + if pr and pr["min_output"] is not None: | ||||
| 157 | + eff_points.append({"id": r["model_id"], "model": {"id": r["model_id"], "slug": r["model_slug"], "name": r["model_name"]}, "x": float(pr["min_output"]), "y": float(r["score"]), | ||||
| 158 | + "rank": r["rank"], "trust_level": r["trust_level"]}) | ||||
| 159 | + eff_front = set(pareto_frontier(eff_points, maximize_y=quality_group["higher_is_better"] if quality_group else True)) | ||||
| 160 | + multimodal = [] | ||||
| 161 | + for mid, rk in ranks.items(): | ||||
| 162 | + if min(rk.values()) > 10: | ||||
| 163 | + continue | ||||
| 164 | + row = next((r for g in groups.values() for r in g["rows"] if r["model_id"] == mid), None) | ||||
| 165 | + if not row: | ||||
| 166 | + continue | ||||
| 167 | + mods = _mods(row.get("model_attrs") or {}) | ||||
| 168 | + if len(mods) >= 3: | ||||
| 169 | + multimodal.append({"model": {"id": mid, "slug": row["model_slug"], "name": row["model_name"]}, "modalities": sorted(mods), "top10_on": sorted(b for b, k in rk.items() if k <= 10)}) | ||||
| 170 | + moves = [] | ||||
| 171 | + for r in movements: | ||||
| 172 | + ev = change_event(r) | ||||
| 173 | + if ev["event_type"] == "PRICE_CHANGED": | ||||
| 174 | + o, n = _f(ev.get("old_value")), _f(ev.get("new_value")) | ||||
| 175 | + if o and n is not None and abs(n - o) / o >= 0.20: | ||||
| 176 | + ev["percent_change"] = round((n - o) / o * 100, 1) | ||||
| 177 | + moves.append(ev) | ||||
| 178 | + else: | ||||
| 179 | + moves.append(ev) | ||||
| 180 | + return { | ||||
| 181 | + "latest_major_models": [change_event(r) | {"occurred_at": r["occurred_at"], "is_backfill": r["is_backfill"]} for r in major[:limit]], | ||||
| 182 | + "benchmark_frontier": bench_frontier, | ||||
| 183 | + "price_frontier": {"cheapest_output": _deploy_or_none(cheapest), "cheapest_output_1m_context": _deploy_or_none(cheapest_1m), "frontier_models": len(fids), "composition": composition}, | ||||
| 184 | + "context_frontier": [{"model": entity_summary(r), "context_length": r["ctx"]} for r in ctx_rows], | ||||
| 185 | + "open_weight_frontier": {"items": open_frontier[:limit], "dimensions": ["best_rank", "parameter_count", "context_length"], "note": "sorted by best benchmark rank then parameters; no composite"}, | ||||
| 186 | + "efficiency_frontier": {"quality": {"benchmark": quality_group["benchmark"]["slug"], "group": group_summary(quality_group)} if quality_group else None, "x": "cheapest current output price (USD / 1M tokens)", | ||||
| 187 | + "points": [{**p, "pareto": p["id"] in eff_front} for p in sorted(eff_points, key=lambda p: p["x"])], "frontier": sorted(eff_front)}, | ||||
| 188 | + "agentic_frontier": agentic, | ||||
| 189 | + "multimodal_frontier": sorted(multimodal, key=lambda x: (-len(x["top10_on"]), x["model"]["name"] or ""))[:limit], | ||||
| 190 | + "recent_frontier_movements": moves[:50], | ||||
| 191 | + "generated_at": now, | ||||
| 192 | + "methodology": FRONTIER_METHODOLOGY + " benchmark_frontier lists the primary comparability group of every benchmark with ≥ 20 current results; efficiency_frontier is the " | ||||
| 193 | + "Pareto set (maximise quality score, minimise cheapest current output price); recent movements are non-backfill benchmark events and price moves ≥ 20% in 30 days. " | ||||
| 194 | + "Nothing here is a composite ranking.", | ||||
| 195 | + } | ||||
| 196 | + | ||||
| 197 | + | ||||
| 198 | +def _deploy_or_none(row: dict[str, Any] | None) -> dict[str, Any] | None: | ||||
| 199 | + return deployment_row(row) if row else None | ||||
| 200 | + | ||||
| 201 | + | ||||
| 202 | +# ------------------------------------------------------------------------------------------------------------------ /pareto | ||||
| 203 | + | ||||
| 204 | + | ||||
| 205 | +@router.get("/pareto") | ||||
| 206 | +@cached(300) | ||||
| 207 | +async def pareto(request: Request, benchmark: str = Query(...), x: str = Query("output_price"), y: str = Query("score"), metric: str | None = None, config_key: str | None = None, | ||||
| 208 | + org: str | None = None, family: str | None = None, openness: str | None = None) -> dict[str, Any]: | ||||
| 209 | + if x == "latency": | ||||
| 210 | + raise ApiError(400, "x=latency is not available: AI Atlas does not store latency measurements (nothing is estimated for it)") | ||||
| 211 | + if x not in PARETO_X: | ||||
| 212 | + raise ApiError(400, f"x must be one of {', '.join(PARETO_X)}") | ||||
| 213 | + if y != "score": | ||||
| 214 | + raise ApiError(400, "y must be 'score'") | ||||
| 215 | + async with connection() as conn: | ||||
| 216 | + bench = await resolve_entity(conn, benchmark, ("benchmark",), aliases=True) | ||||
| 217 | + rows = await load_results(conn, benchmark_ids=[bench["id"]]) | ||||
| 218 | + groups = list(group_rows(rows).values()) | ||||
| 219 | + g = None | ||||
| 220 | + if metric or config_key: | ||||
| 221 | + cands = [gg for gg in groups if (not metric or gg["metric"] == metric.lower()) and (not config_key or gg["config_key"] == config_key)] | ||||
| 222 | + g = max(cands, key=lambda gg: gg["model_count"]) if cands else None | ||||
| 223 | + else: | ||||
| 224 | + g = primary_group(bench.get("attributes"), groups) | ||||
| 225 | + if not g: | ||||
| 226 | + return {"benchmark": entity_summary(bench), "points": [], "frontier": [], "groups": [group_summary(x) for x in groups], "note": "no current results in the requested group"} | ||||
| 227 | + ranked = rank_rows(g["rows"], g["higher_is_better"]) | ||||
| 228 | + ids = [r["model_id"] for r in ranked] | ||||
| 229 | + prices = await _cheapest_prices(conn, ids) if x in ("output_price", "input_price") else {} | ||||
| 230 | + offers: dict[str, dict[str, Any]] = {} | ||||
| 231 | + if x in ("output_price", "input_price"): | ||||
| 232 | + col = "p.output_per_mtok" if x == "output_price" else "p.input_per_mtok" | ||||
| 233 | + for pr in await fetch_all(conn, f"select distinct on (p.model_id) {PRICE_COLS} from {PRICE_FROM} where p.valid_to is null and {col} > 0 and p.model_id = any(cast(:ids as text[])) order by p.model_id, {col} asc", ids=ids): | ||||
| 234 | + offers[pr["m_id"]] = deployment_row(pr) | ||||
| 235 | + org_id = await resolve_id(conn, org) if org else None | ||||
| 236 | + fam_ids: set[str] | None = None | ||||
| 237 | + if family: | ||||
| 238 | + fr = await fetch_all(conn, "select e.id from entities e left join entities f on f.id = e.family_id where e.entity_type = 'model' and (f.slug = :f or f.id = :f or e.attributes->>'family' ilike :f)", f=family) | ||||
| 239 | + fam_ids = {r["id"] for r in fr} | ||||
| 240 | + open_vals = set(openness_values(openness)) if openness else None | ||||
| 241 | + points = [] | ||||
| 242 | + for r in ranked: | ||||
| 243 | + attrs = r.get("model_attrs") or {} | ||||
| 244 | + if org_id and r.get("organization_id") != org_id: | ||||
| 245 | + continue | ||||
| 246 | + if fam_ids is not None and r["model_id"] not in fam_ids: | ||||
| 247 | + continue | ||||
| 248 | + if open_vals and str(attrs.get("openness") or "") not in open_vals: | ||||
| 249 | + continue | ||||
| 250 | + xv: float | None | ||||
| 251 | + provider = None | ||||
| 252 | + if x == "output_price": | ||||
| 253 | + pr = prices.get(r["model_id"]) | ||||
| 254 | + xv = _f(pr["min_output"]) if pr else None | ||||
| 255 | + provider = (offers.get(r["model_id"]) or {}).get("provider") | ||||
| 256 | + elif x == "input_price": | ||||
| 257 | + pr = prices.get(r["model_id"]) | ||||
| 258 | + xv = _f(pr["min_input"]) if pr else None | ||||
| 259 | + provider = (offers.get(r["model_id"]) or {}).get("provider") | ||||
| 260 | + elif x == "parameter_count": | ||||
| 261 | + xv = _f(attrs.get("parameter_count")) | ||||
| 262 | + elif x == "context_length": | ||||
| 263 | + xv = _f(attrs.get("context_length")) | ||||
| 264 | + else: | ||||
| 265 | + pc = hf.parameter_count(attrs) | ||||
| 266 | + xv = hf.estimate_memory_gb(pc, "4bit", 8192) if pc else None | ||||
| 267 | + if xv is None: | ||||
| 268 | + continue | ||||
| 269 | + points.append({"id": r["model_id"], "model": {"id": r["model_id"], "slug": r["model_slug"], "name": r["model_name"], "organization": r.get("org_name"), "openness": attrs.get("openness")}, | ||||
| 270 | + "x": xv, "y": float(r["score"]), "rank": r["rank"], "trust_level": r["trust_level"], "config": {k: v for k, v in (r.get("config") or {}).items() if k in ("reasoning_effort", "reasoning", "variant", "evaluator")}, | ||||
| 271 | + **({"provider": provider} if provider else {}), **({"estimated": True} if x == "memory_estimate" else {})}) | ||||
| 272 | + front = pareto_frontier(points, maximize_y=g["higher_is_better"]) | ||||
| 273 | + fset = set(front) | ||||
| 274 | + for p in points: | ||||
| 275 | + p["pareto"] = p["id"] in fset | ||||
| 276 | + return {"benchmark": entity_summary(bench), "group": group_summary(g), "groups": [group_summary(gg) for gg in groups], "x": {"key": x, "label": PARETO_X[x]}, "y": {"key": "score", "label": f"{g['metric']} on {bench['canonical_name']}"}, | ||||
| 277 | + "points": sorted(points, key=lambda p: p["x"]), "frontier": front, | ||||
| 278 | + "methodology": f"Points are the best current row per canonical model in comparability group '{g['label']}'. Price = cheapest current offer across providers " | ||||
| 279 | + f"(the provider shown). Pareto frontier maximises the score and minimises x; exact ties are all kept. memory_estimate is an estimate (see /methodology)."} | ||||
| 280 | + | ||||
| 281 | + | ||||
| 282 | +# ------------------------------------------------------------------------------------------------------------------ /pulse | ||||
| 283 | + | ||||
| 284 | + | ||||
| 285 | +@router.get("/pulse") | ||||
| 286 | +@cached(120) | ||||
| 287 | +async def pulse(request: Request, days: int = Query(7, ge=1, le=90)) -> dict[str, Any]: | ||||
| 288 | + now = datetime.now(UTC) | ||||
| 289 | + since = now - timedelta(days=days) | ||||
| 290 | + async with connection() as conn: | ||||
| 291 | + c = await fetch_one(conn, f""" | ||||
| 292 | + with ev as (select ev.*, e.entity_type as et, e.attributes as attrs from change_events ev left join entities e on e.id = ev.entity_id | ||||
| 293 | + where ev.is_backfill = false and ev.occurred_at > :since) | ||||
| 294 | + select (select count(distinct entity_id) from ev where event_type = 'NEW_MODEL' and et = 'model') as new_models, | ||||
| 295 | + (select count(distinct entity_id) from ev where event_type = 'NEW_MODEL' and et = 'model' and attrs->>'openness' in ('open-weights','open-source')) as new_open_models, | ||||
| 296 | + (select count(*) from entities where entity_type = 'artifact' and merged_into is null and first_seen_at > :since) as new_artifacts, | ||||
| 297 | + (select count(distinct entity_id) from ev where event_type = 'NEW_PAPER') as new_papers, | ||||
| 298 | + (select count(*) from ev where event_type = 'PROVIDER_LISTED') as provider_listings, | ||||
| 299 | + (select count(*) from ev where event_type = 'PROVIDER_DELISTED') as provider_delistings, | ||||
| 300 | + (select count(*) from ev where event_type = 'PRICE_CHANGED') as price_changes, | ||||
| 301 | + (select count(distinct entity_id) from ev where event_type = 'NEW_MODEL' and et = 'model' and {num_expr("attrs->>'context_length'")} >= 1000000) as new_models_1m_context, | ||||
| 302 | + (select count(*) from ev where event_type = 'DOCUMENT_CHANGED') as documents_changed, | ||||
| 303 | + (select count(distinct d.source_id) from snapshots s join documents d on d.id = s.document_id where s.observed_at > :since) as sources_observed, | ||||
| 304 | + (select count(*) from ev where event_type <> 'DOCUMENT_CHANGED') as events_total""", since=since) | ||||
| 305 | + pct = await fetch_all(conn, "select old_value, new_value from change_events where is_backfill = false and occurred_at > :since and event_type = 'PRICE_CHANGED'", since=since) | ||||
| 306 | + rows = await load_results(conn, current_only=False) | ||||
| 307 | + meta = await benchmark_meta(conn) | ||||
| 308 | + moves = [] | ||||
| 309 | + for r in pct: | ||||
| 310 | + o, n = _f(r["old_value"]), _f(r["new_value"]) | ||||
| 311 | + if o and n is not None: | ||||
| 312 | + moves.append((n - o) / o * 100) | ||||
| 313 | + by_bench: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 314 | + for r in rows: | ||||
| 315 | + by_bench[r["benchmark_id"]].append(r) | ||||
| 316 | + new_leaders = [] | ||||
| 317 | + for bid, brows in by_bench.items(): | ||||
| 318 | + m = meta.get(bid, {}) | ||||
| 319 | + la, lb = leader_at(brows, since, m.get("attributes")), leader_at(brows, now, m.get("attributes")) | ||||
| 320 | + if lb and (la is None or la["model"]["id"] != lb["model"]["id"]): | ||||
| 321 | + new_leaders.append({"benchmark": {"id": bid, "slug": m.get("slug"), "name": m.get("name")}, "previous": la, "current": lb}) | ||||
| 322 | + c = c or {} | ||||
| 323 | + counters = { | ||||
| 324 | + "new_models": {"value": int(c.get("new_models") or 0), "definition": "canonical models with a NEW_MODEL event that occurred in the window (not back-filled)"}, | ||||
| 325 | + "new_open_weight_models": {"value": int(c.get("new_open_models") or 0), "definition": "subset of new_models with openness open-weights / open-source"}, | ||||
| 326 | + "new_artifacts": {"value": int(c.get("new_artifacts") or 0), "definition": "artifact entities (checkpoints, quantisations, conversions) first seen in the window"}, | ||||
| 327 | + "new_papers": {"value": int(c.get("new_papers") or 0), "definition": "papers with a NEW_PAPER event that occurred in the window"}, | ||||
| 328 | + "provider_listings": {"value": int(c.get("provider_listings") or 0), "definition": "PROVIDER_LISTED events in the window"}, | ||||
| 329 | + "provider_delistings": {"value": int(c.get("provider_delistings") or 0), "definition": "PROVIDER_DELISTED events in the window"}, | ||||
| 330 | + "price_changes": {"value": int(c.get("price_changes") or 0), "median_percent": round(statistics.median(moves), 2) if moves else None, | ||||
| 331 | + "definition": "PRICE_CHANGED events in the window; median % change computed from the events' old/new numeric values when both are present"}, | ||||
| 332 | + "new_models_1m_context": {"value": int(c.get("new_models_1m_context") or 0), "definition": "new_models whose context_length is at least 1 000 000 tokens"}, | ||||
| 333 | + "new_benchmark_leaders": {"value": len(new_leaders), "items": new_leaders, "definition": "benchmarks whose primary-group leader (computed from results observed by each date) changed over the window"}, | ||||
| 334 | + "documents_changed": {"value": int(c.get("documents_changed") or 0), "definition": "DOCUMENT_CHANGED events in the window"}, | ||||
| 335 | + "sources_observed": {"value": int(c.get("sources_observed") or 0), "definition": "distinct sources with at least one snapshot taken in the window"}, | ||||
| 336 | + "events_total": {"value": int(c.get("events_total") or 0), "definition": "all non-backfill events (excluding source-document changes) that occurred in the window"}, | ||||
| 337 | + } | ||||
| 338 | + return {"days": days, "since": since, "until": now, "counters": counters, | ||||
| 339 | + "note": "Deterministic counters over events that OCCURRED in the window and are not back-filled history; each counter carries its own definition."} | ||||
| 340 | + | ||||
| 341 | + | ||||
| 342 | +# ------------------------------------------------------------------------------------------------------------------ /open | ||||
| 343 | + | ||||
| 344 | + | ||||
| 345 | +@router.get("/open") | ||||
| 346 | +@cached(300) | ||||
| 347 | +async def open_models(request: Request, sort: str = Query("release", pattern="^(release|params|context|rank|name|downloads)$"), license: str | None = None, | ||||
| 348 | + min_params: float | None = Query(None, ge=0), max_params: float | None = Query(None, ge=0), min_context: int | None = Query(None, ge=0), | ||||
| 349 | + modality: str | None = None, days: int | None = Query(None, ge=1, le=3650), limit: int = Query(50, ge=1, le=200), offset: int = Query(0, ge=0), | ||||
| 350 | + openness: str | None = None) -> dict[str, Any]: | ||||
| 351 | + """Open-weight / open-source / restricted-weights canonical models with licence permissions, observed dimensions, best results, estimated hardware fit.""" | ||||
| 352 | + where = ["e.entity_type = 'model'", "e.merged_into is null", "e.attributes->>'openness' = any(cast(:o as text[]))"] | ||||
| 353 | + params: dict[str, Any] = {"o": openness_values(openness) if openness else list(DOWNLOADABLE_CATEGORIES)} | ||||
| 354 | + if license: | ||||
| 355 | + from aiatlas.api.routers.models import license_match_sql, license_params | ||||
| 356 | + | ||||
| 357 | + where.append(license_match_sql()) | ||||
| 358 | + params.update(license_params(license)) | ||||
| 359 | + if min_params is not None: | ||||
| 360 | + where.append(f"{PARAMS} >= :minp") | ||||
| 361 | + params["minp"] = float(min_params) | ||||
| 362 | + if max_params is not None: | ||||
| 363 | + where.append(f"{PARAMS} <= :maxp") | ||||
| 364 | + params["maxp"] = float(max_params) | ||||
| 365 | + if min_context is not None: | ||||
| 366 | + where.append(f"{CONTEXT} >= :minc") | ||||
| 367 | + params["minc"] = float(min_context) | ||||
| 368 | + if modality: | ||||
| 369 | + where.append("(e.attributes->'modalities' ? :mod or e.attributes->'modalities_input' ? :mod or e.attributes->'modalities_output' ? :mod)") | ||||
| 370 | + params["mod"] = modality | ||||
| 371 | + if days: | ||||
| 372 | + where.append("(e.attributes->>'release_date' >= :since or e.first_seen_at > now() - make_interval(days => :days))") | ||||
| 373 | + params["since"] = (datetime.now(UTC) - timedelta(days=days)).date().isoformat() | ||||
| 374 | + params["days"] = days | ||||
| 375 | + order = {"release": "e.attributes->>'release_date' desc nulls last", "params": f"{PARAMS} desc nulls last", "context": f"{CONTEXT} desc nulls last", "name": "e.canonical_name asc", | ||||
| 376 | + "downloads": num_expr("e.attributes->>'metric.downloads'") + " desc nulls last", "rank": "e.canonical_name asc"}[sort] | ||||
| 377 | + where_sql = " and ".join(where) | ||||
| 378 | + async with connection() as conn: | ||||
| 379 | + rows = await fetch_all(conn, f"select {ENTITY_COLS}, e.family_id from {ENTITY_FROM} where {where_sql} order by {order}, e.id limit :lim offset :off", lim=limit if sort != "rank" else 2000, off=0 if sort == "rank" else offset, **params) | ||||
| 380 | + total = await fetch_val(conn, f"select count(*) from {ENTITY_FROM} where {where_sql}", **params) | ||||
| 381 | + groups = await all_primary_groups(conn) | ||||
| 382 | + prices = await _cheapest_prices(conn, [r["id"] for r in rows]) | ||||
| 383 | + summary = await fetch_one(conn, """select jsonb_object_agg(k, n) as by_cat from (select attributes->>'openness' as k, count(*) as n from entities | ||||
| 384 | + where entity_type = 'model' and merged_into is null and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted') group by 1) x""") | ||||
| 385 | + lic_rows = await fetch_all(conn, """select coalesce(attributes->>'license_key', attributes->>'license') as raw, count(*) as n from entities where entity_type = 'model' and merged_into is null | ||||
| 386 | + and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted') and (attributes ? 'license' or attributes ? 'license_key') group by 1""") | ||||
| 387 | + new_30d = await fetch_val(conn, """select count(*) from entities where entity_type = 'model' and merged_into is null and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted') | ||||
| 388 | + and (attributes->>'release_date' >= to_char((now() at time zone 'UTC') - interval '30 days', 'YYYY-MM-DD') or first_seen_at > now() - interval '30 days')""") | ||||
| 389 | + ranks: dict[str, dict[str, int]] = defaultdict(dict) | ||||
| 390 | + for g in groups.values(): | ||||
| 391 | + for r in rank_rows(g["rows"], g["higher_is_better"]): | ||||
| 392 | + ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"] | ||||
| 393 | + items = [] | ||||
| 394 | + for r in rows: | ||||
| 395 | + attrs = r["attributes"] or {} | ||||
| 396 | + key = attrs.get("license_key") or normalize_license(attrs.get("license")) | ||||
| 397 | + info = LICENSES.get(key) if key else None | ||||
| 398 | + rk = ranks.get(r["id"], {}) | ||||
| 399 | + best = sorted(rk.items(), key=lambda kv: kv[1])[:3] | ||||
| 400 | + pr = prices.get(r["id"]) | ||||
| 401 | + items.append({"model": entity_summary(r), "licence": {**info.as_dict(), "raw": attrs.get("license")} if info else {"key": None, "raw": attrs.get("license"), "note": "not classified"}, | ||||
| 402 | + "dimensions": {"parameter_count": _f(attrs.get("parameter_count")), "active_parameter_count": _f(attrs.get("active_parameter_count")), "context_length": _f(attrs.get("context_length")), | ||||
| 403 | + "modalities": sorted(_mods(attrs)), "release_date": attrs.get("release_date"), "openness": attrs.get("openness"), "downloads": _f(attrs.get("metric.downloads"))}, | ||||
| 404 | + "best_results": [{"benchmark": b, "rank": k} for b, k in best], "best_rank": best[0][1] if best else None, | ||||
| 405 | + "hardware_fit": {"4bit_64gb": hf.fit_detailed(attrs, 64, quant="4bit", context=8192), "8bit_128gb": hf.fit_detailed(attrs, 128, quant="8bit", context=8192), "estimated": True}, | ||||
| 406 | + "providers": int(pr["providers"]) if pr else 0, "cheapest_output_per_mtok": _f(pr["min_output"]) if pr else None}) | ||||
| 407 | + if sort == "rank": | ||||
| 408 | + items.sort(key=lambda x: (x["best_rank"] if x["best_rank"] is not None else 10_000, x["model"]["name"] or "")) | ||||
| 409 | + items = items[offset:offset + limit] | ||||
| 410 | + lic_counts: dict[str, int] = defaultdict(int) | ||||
| 411 | + for lr in lic_rows: | ||||
| 412 | + k = lr["raw"] if lr["raw"] in LICENSES else normalize_license(lr["raw"]) | ||||
| 413 | + lic_counts[k or f"raw:{lr['raw']}"] += int(lr["n"]) | ||||
| 414 | + return {"items": items, "total": int(total or 0), "limit": limit, "offset": offset, | ||||
| 415 | + "summary": {"by_category": (summary or {}).get("by_cat") or {}, "by_license_top": [{"key": k, "label": LICENSES[k].label if k in LICENSES else k, "models": n} for k, n in sorted(lic_counts.items(), key=lambda kv: -kv[1])[:12]], | ||||
| 416 | + "new_30d": int(new_30d or 0)}, | ||||
| 417 | + "note": "Universe = canonical models whose weights can be downloaded (open-weights, open-source, restricted-weights). hardware_fit values are ESTIMATES (see /methodology); " | ||||
| 418 | + "best_results are ranks inside each benchmark's primary comparability group — no composite score."} | ||||
| 419 | + | ||||
| 420 | + | ||||
| 421 | +# ------------------------------------------------------------------------------------------------------------------ /find-a-model | ||||
| 422 | + | ||||
| 423 | + | ||||
| 424 | +@router.get("/find-a-model") | ||||
| 425 | +@cached(300) | ||||
| 426 | +async def find_a_model(request: Request, use_case: str | None = Query(None), deployment: str = Query("any", pattern="^(local|api|any)$"), memory_gb: float | None = Query(None, gt=0, le=100000), | ||||
| 427 | + quant: str = Query("4bit"), context_min: int | None = Query(None, ge=0), license: str = Query("any", pattern="^(commercial|any)$"), openness: str | None = None, | ||||
| 428 | + max_input_price: float | None = Query(None, ge=0), max_output_price: float | None = Query(None, ge=0), modalities: str | None = None, | ||||
| 429 | + limit: int = Query(30, ge=1, le=100)) -> dict[str, Any]: | ||||
| 430 | + if use_case and use_case not in USE_CASES: | ||||
| 431 | + raise ApiError(400, f"use_case must be one of {', '.join(USE_CASES)}") | ||||
| 432 | + async with connection() as conn: | ||||
| 433 | + out = await find_models(conn, use_case=use_case, deployment=deployment, memory_gb=memory_gb, quant=quant, context_min=context_min, license=license, | ||||
| 434 | + openness=openness_values(openness) if openness else None, max_input_price=max_input_price, max_output_price=max_output_price, | ||||
| 435 | + modalities=csv(modalities), limit=limit) | ||||
| 436 | + if deployment != "local": | ||||
| 437 | + ids = [m["model"]["id"] for m in out["matches"]] | ||||
| 438 | + rows = await fetch_all(conn, f"""select * from (select {PRICE_COLS}, row_number() over (partition by p.model_id order by p.output_per_mtok asc nulls last) as rn | ||||
| 439 | + from {PRICE_FROM} where p.valid_to is null and p.model_id = any(cast(:ids as text[]))) x where rn <= 5""", ids=ids) | ||||
| 440 | + by: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 441 | + for r in rows: | ||||
| 442 | + by[r["m_id"]].append(deployment_row(r)) | ||||
| 443 | + for m in out["matches"]: | ||||
| 444 | + m["deployments"] = by.get(m["model"]["id"], []) | ||||
| 445 | + return out | ||||
| 446 | + | ||||
| 447 | + | ||||
| 448 | +# ------------------------------------------------------------------------------------------------------------------ /run-locally | ||||
| 449 | + | ||||
| 450 | + | ||||
| 451 | +@router.get("/run-locally") | ||||
| 452 | +@cached(300) | ||||
| 453 | +async def run_locally(request: Request, memory_gb: float = Query(..., gt=0, le=100000), gpu_count: int = Query(1, ge=1, le=8), quant: str = Query("4bit"), context: int = Query(8192, ge=0, le=10_000_000), | ||||
| 454 | + batch: int = Query(1, ge=1, le=256), platform: str = Query("any", pattern="^(apple|nvidia|amd|any)$"), use_case: str | None = None, | ||||
| 455 | + limit: int = Query(60, ge=1, le=300), openness: str | None = None) -> dict[str, Any]: | ||||
| 456 | + """Canonical models (and their compatible artifacts) whose ESTIMATED footprint fits `memory_gb × gpu_count`.""" | ||||
| 457 | + if gpu_count not in (1, 2, 4, 8): | ||||
| 458 | + raise ApiError(400, "gpu_count must be 1, 2, 4 or 8") | ||||
| 459 | + q = hf.normalize_quant(quant) | ||||
| 460 | + fmt_pref = {"apple": ("mlx", "gguf"), "nvidia": ("gguf", "awq", "gptq", "fp8", "nvfp4", "int4", "int8"), "amd": ("gguf", "mxfp4", "int8"), "any": ()}[platform] | ||||
| 461 | + where = ["e.entity_type = 'model'", "e.merged_into is null", "e.attributes ? 'parameter_count'", "e.attributes->>'openness' = any(cast(:o as text[]))"] | ||||
| 462 | + params: dict[str, Any] = {"o": openness_values(openness) if openness else list(DOWNLOADABLE_CATEGORIES)} | ||||
| 463 | + async with connection() as conn: | ||||
| 464 | + rows = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where {' and '.join(where)} order by {PARAMS} desc nulls last limit 3000", **params) | ||||
| 465 | + arts = await fetch_all(conn, f""" | ||||
| 466 | + select a.canonical_id as model_id, {ENTITY_COLS} from entities e join lateral (select e.canonical_id) a on true left join entities eo on eo.id = e.organization_id | ||||
| 467 | + where e.entity_type = 'artifact' and e.merged_into is null and e.canonical_id is not null | ||||
| 468 | + union all | ||||
| 469 | + select r.object_id as model_id, {ENTITY_COLS} from relations r join entities e on e.id = r.subject_id left join entities eo on eo.id = e.organization_id | ||||
| 470 | + where r.predicate = 'quantized_from' and r.valid_to is null and e.merged_into is null and (e.attributes->>'is_quantized' = 'true' or e.attributes ? 'quant_format')""") | ||||
| 471 | + groups = await all_primary_groups(conn) if use_case else {} | ||||
| 472 | + cat_models: set[str] | None = None | ||||
| 473 | + if use_case: | ||||
| 474 | + cat = {"coding": "coding", "reasoning": "reasoning", "agentic": "agentic", "vision": "multimodal", "math": "math"}.get(use_case, use_case) | ||||
| 475 | + cat_models = {r["model_id"] for g in groups.values() if (g["benchmark"].get("category") or "").lower() == cat for r in g["rows"]} | ||||
| 476 | + art_by: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 477 | + for a in arts: | ||||
| 478 | + art_by[a["model_id"]].append(a) | ||||
| 479 | + items = [] | ||||
| 480 | + for r in rows: | ||||
| 481 | + if cat_models is not None and r["id"] not in cat_models: | ||||
| 482 | + continue | ||||
| 483 | + attrs = r["attributes"] or {} | ||||
| 484 | + f = hf.fit_detailed(attrs, memory_gb, quant=q, context=context, batch=batch, gpu_count=gpu_count) | ||||
| 485 | + if not f: | ||||
| 486 | + continue | ||||
| 487 | + compatible = [] | ||||
| 488 | + for a in art_by.get(r["id"], []): | ||||
| 489 | + aa = a["attributes"] or {} | ||||
| 490 | + fmt = str(aa.get("quant_format") or "").lower() | ||||
| 491 | + if fmt_pref and fmt and fmt not in fmt_pref: | ||||
| 492 | + continue | ||||
| 493 | + size = hf.file_size_gb(aa) | ||||
| 494 | + af = hf.fit_detailed({**attrs, **{k: v for k, v in aa.items() if k in ("num_hidden_layers", "num_key_value_heads", "num_attention_heads", "head_dim", "hidden_size")}}, memory_gb, | ||||
| 495 | + quant=q, context=context, batch=batch, gpu_count=gpu_count, observed_size_gb=size) | ||||
| 496 | + compatible.append({"artifact": entity_summary(a), "quant_format": fmt or None, "file_size_gb": size, "weights_source": "observed" if size else "estimated", "fit": af}) | ||||
| 497 | + compatible.sort(key=lambda x: (not (x["fit"] or {}).get("fits", False), x["file_size_gb"] or 1e9)) | ||||
| 498 | + items.append({"model": entity_summary(r), "fit": f, "artifacts": compatible[:8], "artifact_count": len(art_by.get(r["id"], []))}) | ||||
| 499 | + fits = [i for i in items if i["fit"]["fits"]] | ||||
| 500 | + fits.sort(key=lambda i: -(i["fit"].get("parameter_count") or 0)) | ||||
| 501 | + return {"inputs": {"memory_gb": memory_gb, "gpu_count": gpu_count, "total_memory_gb": memory_gb * gpu_count, "quant": q, "context": context, "batch": batch, "platform": platform, "use_case": use_case}, | ||||
| 502 | + "estimated": True, "assumptions": hf.ASSUMPTIONS, "counts": {"fits": len(fits), "evaluated": len(items)}, "items": fits[:limit], | ||||
| 503 | + "note": "Every figure is an ESTIMATE: weights = params × bytes/param (× 1.15 overhead) unless an artifact's observed file size is available; KV cache uses architecture metadata when known, " | ||||
| 504 | + + ("else 0.5 GB per 8K tokens × batch. Multi-GPU sums device memory and ignores interconnect." if gpu_count > 1 else "else 0.5 GB per 8K tokens × batch.")} | ||||
| 505 | + | ||||
| 506 | + | ||||
| 507 | +# ------------------------------------------------------------------------------------------------------------------ /time-machine | ||||
| 508 | + | ||||
| 509 | + | ||||
| 510 | +@router.get("/time-machine") | ||||
| 511 | +@cached(300) | ||||
| 512 | +async def time_machine(request: Request, date: str = Query(..., description="YYYY-MM-DD"), scope: str = Query("models", pattern="^(models|prices|benchmarks|hardware|all)$"), | ||||
| 513 | + limit: int = Query(50, ge=1, le=300)) -> dict[str, Any]: | ||||
| 514 | + d = parse_date(date, "date") | ||||
| 515 | + assert d is not None | ||||
| 516 | + at = datetime.combine(d, dtime.max, UTC) | ||||
| 517 | + out: dict[str, Any] = {"date": d.isoformat(), "scope": scope} | ||||
| 518 | + async with connection() as conn: | ||||
| 519 | + first = await fetch_val(conn, "select min(first_seen_at) from entities") | ||||
| 520 | + reconstructed = first is not None and at < first | ||||
| 521 | + out["first_entity_at"] = first | ||||
| 522 | + out["reconstructed"] = bool(reconstructed) | ||||
| 523 | + out["note"] = (f"AI Atlas observation history starts at {first.isoformat() if first else 'n/a'}. " + | ||||
| 524 | + ("This date is earlier: the state is RECONSTRUCTED from claims with effective dates and from release dates — not from direct observation." if reconstructed | ||||
| 525 | + else "The state is taken from claims, prices and results as they were known at the end of that UTC day.")) | ||||
| 526 | + if scope in ("models", "all"): | ||||
| 527 | + rows = await fetch_all(conn, f"""select {ENTITY_COLS}, (e.first_seen_at <= :at) as observed_then from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null | ||||
| 528 | + and (e.first_seen_at <= :at or (e.attributes->>'release_date' ~ '^\\d{{4}}' and left(e.attributes->>'release_date', 10) <= :d)) | ||||
| 529 | + order by e.attributes->>'release_date' desc nulls last, e.canonical_name limit :lim""", at=at, d=d.isoformat(), lim=limit) | ||||
| 530 | + ids = [r["id"] for r in rows] | ||||
| 531 | + claims = await fetch_all(conn, f"""select distinct on (c.entity_id, c.property) c.entity_id, {CLAIM_COLS} from {CLAIM_FROM} | ||||
| 532 | + where c.entity_id = any(cast(:ids as text[])) and c.status <> 'retracted' and coalesce(c.effective_at, c.valid_from) <= :at and (c.valid_to is null or c.valid_to > :at) | ||||
| 533 | + order by c.entity_id, c.property, c.tier, c.valid_from desc""", ids=ids, at=at) if ids else [] | ||||
| 534 | + by: dict[str, dict[str, Any]] = defaultdict(dict) | ||||
| 535 | + for c in claims: | ||||
| 536 | + by[c["entity_id"]][c["property"]] = c["value"] | ||||
| 537 | + total = await fetch_val(conn, """select count(*) from entities e where e.entity_type = 'model' and e.merged_into is null | ||||
| 538 | + and (e.first_seen_at <= :at or (e.attributes->>'release_date' ~ '^\\d{4}' and left(e.attributes->>'release_date', 10) <= :d))""", at=at, d=d.isoformat()) | ||||
| 539 | + out["models"] = {"items": [{"model": entity_summary(r), "attributes_as_of": by.get(r["id"], {}), "observed_then": bool(r["observed_then"]), | ||||
| 540 | + "reconstructed": not r["observed_then"]} for r in rows], "total": int(total or 0), "limit": limit} | ||||
| 541 | + if scope in ("prices", "all"): | ||||
| 542 | + prs = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.valid_from <= :at and (p.valid_to is null or p.valid_to > :at) order by p.output_per_mtok nulls last limit :lim", at=at, lim=limit) | ||||
| 543 | + n = await fetch_val(conn, "select count(*) from prices p where p.valid_from <= :at and (p.valid_to is null or p.valid_to > :at)", at=at) | ||||
| 544 | + out["prices"] = {"items": [deployment_row(p) for p in prs], "total": int(n or 0), "note": "offers whose validity interval covers the date (price rows are append-only)"} | ||||
| 545 | + if scope in ("benchmarks", "all"): | ||||
| 546 | + rows_r = await load_results(conn, current_only=False) | ||||
| 547 | + meta = await benchmark_meta(conn) | ||||
| 548 | + by_b: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 549 | + for r in rows_r: | ||||
| 550 | + by_b[r["benchmark_id"]].append(r) | ||||
| 551 | + leaders = [] | ||||
| 552 | + for bid, brows in by_b.items(): | ||||
| 553 | + la = leader_at(brows, at, meta.get(bid, {}).get("attributes")) | ||||
| 554 | + if la: | ||||
| 555 | + leaders.append({"benchmark": {"id": bid, "slug": meta.get(bid, {}).get("slug"), "name": meta.get(bid, {}).get("name")}, "leader": la}) | ||||
| 556 | + out["benchmarks"] = {"leaders": sorted(leaders, key=lambda x: x["benchmark"]["name"] or ""), "note": "leaders from results observed by the date (evaluation dates are not used: a result is known only once observed)"} | ||||
| 557 | + if scope in ("hardware", "all"): | ||||
| 558 | + hw = await fetch_all(conn, f"""select {ENTITY_COLS}, (e.first_seen_at <= :at) as observed_then from {ENTITY_FROM} where e.entity_type = 'hardware' and e.merged_into is null | ||||
| 559 | + and (e.first_seen_at <= :at or (e.attributes->>'release_date' ~ '^\\d{{4}}' and left(e.attributes->>'release_date', 7) <= :d)) | ||||
| 560 | + order by e.attributes->>'release_date' desc nulls last limit :lim""", at=at, d=d.isoformat()[:7], lim=limit) | ||||
| 561 | + out["hardware"] = {"items": [{"hardware": entity_summary(r), "reconstructed": not r["observed_then"]} for r in hw]} | ||||
| 562 | + return out | ||||
| 563 | + | ||||
| 564 | + | ||||
| 565 | +__all__ = ["PARETO_X", "router"] | ||||
modified
src/aiatlas/api/routers/misc.py
+127 −9
@@ -1,4 +1,4 @@ | |||||
| 1 | −"""/methodology · /trending · POST /views · /sitemap · /api-keys/me.""" | 1 | +"""/methodology · /trending · POST /views · /sitemap · /api-keys/me · /licenses.""" | ||
| 2 | from __future__ import annotations | 2 | from __future__ import annotations | ||
| 3 | 3 | ||||
| 4 | import hashlib | 4 | import hashlib | ||
@@ -10,6 +10,7 @@ from pydantic import BaseModel, Field | |||||
| 10 | 10 | ||||
| 11 | from aiatlas.api.common import ( | 11 | from aiatlas.api.common import ( | ||
| 12 | ENTITY_COLS, | 12 | ENTITY_COLS, | ||
| 13 | + ENTITY_FROM, | ||||
| 13 | EVENT_TYPE_LABELS, | 14 | EVENT_TYPE_LABELS, | ||
| 14 | STATUS_VOCAB, | 15 | STATUS_VOCAB, | ||
| 15 | ApiError, | 16 | ApiError, | ||
@@ -21,7 +22,15 @@ from aiatlas.api.common import ( | |||||
| 21 | ) | 22 | ) | ||
| 22 | from aiatlas.db import connection, execute, fetch_all, fetch_one, transaction | 23 | from aiatlas.db import connection, execute, fetch_all, fetch_one, transaction | ||
| 23 | from aiatlas.ids import ENTITY_TYPES | 24 | from aiatlas.ids import ENTITY_TYPES | ||
| 25 | +from aiatlas.ontology import anomalies as anomaly_checks | ||||
| 26 | +from aiatlas.ontology.benchmarks import CONDITION_KEYS, IGNORED_KEYS, TASK_KEYS, TRUST_LABELS, TRUST_LEVELS | ||||
| 27 | +from aiatlas.ontology.licenses import CATEGORIES, LICENSES, normalize_license | ||||
| 28 | +from aiatlas.ontology.openness import OPENNESS_CATEGORIES, OPENNESS_DEFINITIONS, OPENNESS_DIMENSIONS, OPENNESS_LABELS | ||||
| 29 | +from aiatlas.services import hardware_fit as hf | ||||
| 30 | +from aiatlas.services.finder import RULES as FINDER_RULES | ||||
| 31 | +from aiatlas.services.frontier import FRONTIER_METHODOLOGY | ||||
| 24 | from aiatlas.services.quality import EXPECTED_FIELDS, QUALITY_VERSION | 32 | from aiatlas.services.quality import EXPECTED_FIELDS, QUALITY_VERSION | ||
| 33 | +from aiatlas.services.stats import DEFINITIONS as COUNTER_DEFINITIONS | ||||
| 25 | 34 | ||||
| 26 | router = APIRouter(prefix="/api/v1", tags=["misc"]) | 35 | router = APIRouter(prefix="/api/v1", tags=["misc"]) | ||
| 27 | 36 | ||||
@@ -43,6 +52,42 @@ EXTRACTORS = [ | |||||
| 43 | {"key": "curated", "description": "Hand-maintained registries shipped with the code (organizations, providers, benchmarks, hardware)."}, | 52 | {"key": "curated", "description": "Hand-maintained registries shipped with the code (organizations, providers, benchmarks, hardware)."}, | ||
| 44 | {"key": "llm", "description": "Local LLM extraction validated against a strict JSON schema; medium/low confidence, never overrides deterministic tier-1 claims."}, | 53 | {"key": "llm", "description": "Local LLM extraction validated against a strict JSON schema; medium/low confidence, never overrides deterministic tier-1 claims."}, | ||
| 45 | ] | 54 | ] | ||
| 55 | +COMPARABILITY_RULES = { | ||||
| 56 | + "comparable": "Same benchmark variant, same canonical metric, same task-defining configuration (variant, evaluator/harness, shots, pass regime, scaffold…) and same conditions.", | ||||
| 57 | + "partially-comparable": "Same task, but conditions differ (reasoning effort, thinking budget, temperature, judge, tool use, max tokens…).", | ||||
| 58 | + "not-comparable": "Different benchmark variant, different metric, or a task-defining configuration key differs.", | ||||
| 59 | + "task_keys": list(TASK_KEYS), "condition_keys": list(CONDITION_KEYS), "ignored_keys": sorted(IGNORED_KEYS), | ||||
| 60 | + "group": "A comparability group is (benchmark, canonical metric, config_key) where config_key = sha1 of the task keys + metric (12 hex chars).", | ||||
| 61 | + "leaderboard": "One row per canonical model: its best current row inside the chosen group; effort variants folded into a model share its rows.", | ||||
| 62 | +} | ||||
| 63 | +ANOMALY_CHECKS = [ | ||||
| 64 | + {"check": "params_too_large", "severity": "critical", "description": f"parameter_count above {anomaly_checks.MAX_PARAMS:.0e}"}, | ||||
| 65 | + {"check": "params_too_small", "severity": "warning", "description": "parameter_count below 100K"}, | ||||
| 66 | + {"check": "active_gt_total", "severity": "critical", "description": "active parameters exceed total parameters"}, | ||||
| 67 | + {"check": "context_too_large", "severity": "critical", "description": "context_length above 100M tokens"}, | ||||
| 68 | + {"check": "context_too_small", "severity": "warning", "description": "context_length below 256 tokens"}, | ||||
| 69 | + {"check": "max_output_gt_context", "severity": "warning", "description": "max_output_tokens exceeds context_length"}, | ||||
| 70 | + {"check": "release_in_future", "severity": "critical", "description": "release_date after today"}, | ||||
| 71 | + {"check": "release_too_old", "severity": "warning", "description": "release_date before 2010"}, | ||||
| 72 | + {"check": "deprecated_before_release", "severity": "critical", "description": "deprecation_date before release_date"}, | ||||
| 73 | + {"check": "retired_before_release", "severity": "critical", "description": "retirement_date before release_date"}, | ||||
| 74 | + {"check": "retired_before_deprecated", "severity": "warning", "description": "retirement_date before deprecation_date"}, | ||||
| 75 | + {"check": "cutoff_after_release", "severity": "warning", "description": "knowledge_cutoff after release_date"}, | ||||
| 76 | + {"check": "deprecated_future_release", "severity": "critical", "description": "deprecated/retired status with a future release date"}, | ||||
| 77 | + {"check": "open_without_weights_url", "severity": "info", "description": "labelled open but no weights location recorded"}, | ||||
| 78 | + {"check": "negative_price", "severity": "critical", "description": "a price field is negative"}, | ||||
| 79 | + {"check": "price_too_high", "severity": "warning", "description": f"a price exceeds ${anomaly_checks.MAX_PRICE_PER_MTOK:g} per 1M tokens"}, | ||||
| 80 | + {"check": "zero_output_price", "severity": "warning", "description": "output price 0 while input is positive and the offer is not free"}, | ||||
| 81 | + {"check": "input_gt_output_price", "severity": "info", "description": "input price more than 4× the output price"}, | ||||
| 82 | + {"check": "cached_gt_input_price", "severity": "warning", "description": "cached input price above the input price"}, | ||||
| 83 | + {"check": "price_jump_100x", "severity": "critical", "description": "a price moved by more than 100× between two observations"}, | ||||
| 84 | + {"check": "score_above_max", "severity": "critical", "description": "benchmark score above the metric maximum"}, | ||||
| 85 | + {"check": "score_below_min", "severity": "critical", "description": "benchmark score below the metric minimum"}, | ||||
| 86 | + {"check": "evaluated_before_release", "severity": "warning", "description": "result evaluated more than 45 days before the model's release"}, | ||||
| 87 | + {"check": "memory_implausible", "severity": "critical", "description": "hardware memory_gb ≤ 0 or above 100 000"}, | ||||
| 88 | + {"check": "bandwidth_implausible", "severity": "warning", "description": "memory bandwidth ≤ 0 or above 100 000 GB/s"}, | ||||
| 89 | + {"check": "tdp_implausible", "severity": "warning", "description": "TDP ≤ 0 or above 200 000 W"}, | ||||
| 90 | +] | ||||
| 46 | 91 | ||||
| 47 | 92 | ||||
| 48 | @router.get("/methodology") | 93 | @router.get("/methodology") | ||
@@ -57,21 +102,91 @@ async def methodology(request: Request) -> dict[str, Any]: | |||||
| 57 | for t, (lbl, imp) in EVENT_TYPE_LABELS.items() if t not in seen] | 102 | for t, (lbl, imp) in EVENT_TYPE_LABELS.items() if t not in seen] | ||
| 58 | return {"metrics": metrics, "quality_version": QUALITY_VERSION, "expected_fields": EXPECTED_FIELDS, "confidence_levels": CONFIDENCE_LEVELS, "tiers": TIERS, | 103 | return {"metrics": metrics, "quality_version": QUALITY_VERSION, "expected_fields": EXPECTED_FIELDS, "confidence_levels": CONFIDENCE_LEVELS, "tiers": TIERS, | ||
| 59 | "event_types": types, "status_vocabulary": list(STATUS_VOCAB), "extractors": EXTRACTORS, | 104 | "event_types": types, "status_vocabulary": list(STATUS_VOCAB), "extractors": EXTRACTORS, | ||
| 105 | + "openness": {"categories": list(OPENNESS_CATEGORIES), "labels": OPENNESS_LABELS, "definitions": OPENNESS_DEFINITIONS, "dimensions": list(OPENNESS_DIMENSIONS), | ||||
| 106 | + "note": "Categories are derived from measurable dimensions and the licence ontology; a custom community licence is never 'open-source'."}, | ||||
| 107 | + "trust_levels": [{"key": k, "label": TRUST_LABELS[k]} for k in TRUST_LEVELS], | ||||
| 108 | + "comparability": COMPARABILITY_RULES, | ||||
| 109 | + "counters": COUNTER_DEFINITIONS, | ||||
| 110 | + "anomaly_checks": ANOMALY_CHECKS, | ||||
| 111 | + "event_semantics": {"occurred_at": "coalesce(effective_at, observed_at) — when the change happened (effective date when a source states it)", | ||||
| 112 | + "observed_at": "when AI Atlas first saw the change", "recorded_at": "when the row was written", | ||||
| 113 | + "is_backfill": "true for history imported when a source is first crawled (never shown as 'today' in feeds)", | ||||
| 114 | + "group_key": "one release / announcement seen through several documents shares a group_key"}, | ||||
| 115 | + "hardware_fit": {"assumptions": hf.ASSUMPTIONS, "bytes_per_param": hf.BYTES_PER_PARAM, "reserved_gb": hf.RESERVED_GB}, | ||||
| 116 | + "frontier": FRONTIER_METHODOLOGY, | ||||
| 117 | + "find_a_model": FINDER_RULES, | ||||
| 118 | + "licence_categories": list(CATEGORIES), | ||||
| 60 | "principles": ["Never fabricate: missing data is reported as unavailable.", "Every fact carries provenance (source, snapshot, URL, tier, confidence, extractor).", | 119 | "principles": ["Never fabricate: missing data is reported as unavailable.", "Every fact carries provenance (source, snapshot, URL, tier, confidence, extractor).", | ||
| 61 | "History is append-only: claims, prices and benchmark results are never overwritten.", | 120 | "History is append-only: claims, prices and benchmark results are never overwritten.", | ||
| 62 | − "Conflicts between sources are stored side by side and flagged for review.", "Live counters and feeds are computed from the database."]} | 121 | + "Conflicts between sources are stored side by side and flagged for review.", "Live counters and feeds are computed from the database.", | ||
| 122 | + "No composite 'best model' score: rankings are per benchmark comparability group; finders return the observed dimensions."]} | ||||
| 63 | 123 | ||||
| 64 | 124 | ||||
| 65 | @router.get("/trending") | 125 | @router.get("/trending") | ||
| 66 | @cached(300) | 126 | @cached(300) | ||
| 67 | −async def trending(request: Request, days: int = Query(7, ge=1, le=90), limit: int = Query(12, ge=1, le=60), type: str | None = Query(None, alias="type")) -> dict[str, Any]: | 127 | +async def trending(request: Request, days: int = Query(7, ge=1, le=90), limit: int = Query(12, ge=1, le=60), type: str | None = Query(None, alias="type"), | ||
| 128 | + kind: str = Query("views", pattern="^(views|most_changed|new_listings|new_results)$")) -> dict[str, Any]: | ||||
| 129 | + """`kind=views` (page views, v1) · `most_changed` (events per entity) · `new_listings` (PROVIDER_LISTED) · `new_results` (benchmark result events) — separate lists, never merged.""" | ||||
| 130 | + async with connection() as conn: | ||||
| 131 | + if kind == "views": | ||||
| 132 | + rows = await fetch_all(conn, f""" | ||||
| 133 | + with v as (select split_part(regexp_replace(path, '[?#].*$', ''), '/', 3) as slug, sum(views) as views from page_views | ||||
| 134 | + where day >= ((now() at time zone 'UTC') - make_interval(days => :d))::date and path ~ '^/[a-z-]+/[^/?#]+' group by 1) | ||||
| 135 | + select v.views as n, {ENTITY_COLS} from v join entities e on e.slug = v.slug left join entities eo on eo.id = e.organization_id | ||||
| 136 | + where e.merged_into is null {"and e.entity_type = :t" if type else ""} order by v.views desc, e.updated_at desc limit :lim""", d=days, lim=limit, t=type) | ||||
| 137 | + items = [{**(entity_summary(r) or {}), "views": int(r["n"] or 0)} for r in rows] | ||||
| 138 | + definition = "Page views recorded by the site beacon in the window." | ||||
| 139 | + else: | ||||
| 140 | + cond = {"most_changed": "ev.event_type <> 'DOCUMENT_CHANGED'", "new_listings": "ev.event_type in ('PROVIDER_LISTED','PROVIDER_DELISTED')", | ||||
| 141 | + "new_results": "ev.event_type in ('BENCHMARK_RESULT','BENCHMARK_UPDATED')"}[kind] | ||||
| 142 | + rows = await fetch_all(conn, f""" | ||||
| 143 | + with v as (select ev.entity_id, count(*) as n, max(ev.occurred_at) as last_at from change_events ev | ||||
| 144 | + where ev.is_backfill = false and ev.occurred_at > now() - make_interval(days => :d) and {cond} group by 1) | ||||
| 145 | + select v.n, v.last_at, {ENTITY_COLS} from v join entities e on e.id = v.entity_id left join entities eo on eo.id = e.organization_id | ||||
| 146 | + where e.merged_into is null {"and e.entity_type = :t" if type else ""} order by v.n desc, v.last_at desc limit :lim""", d=days, lim=limit, t=type) | ||||
| 147 | + items = [{**(entity_summary(r) or {}), "events": int(r["n"] or 0), "last_event_at": r["last_at"]} for r in rows] | ||||
| 148 | + definition = {"most_changed": "Entities with the most non-backfill events (any type except source-document changes) that occurred in the window.", | ||||
| 149 | + "new_listings": "Entities with the most provider listing / delisting events in the window.", | ||||
| 150 | + "new_results": "Entities with the most new or updated benchmark result events in the window."}[kind] | ||||
| 151 | + return {"days": days, "kind": kind, "items": items, "definition": definition} | ||||
| 152 | + | ||||
| 153 | + | ||||
| 154 | +@router.get("/licenses") | ||||
| 155 | +@cached(600) | ||||
| 156 | +async def licenses(request: Request) -> dict[str, Any]: | ||||
| 157 | + """Licence ontology + how many canonical models use each key (canonical `license_key` or a raw label the ontology maps to it).""" | ||||
| 68 | async with connection() as conn: | 158 | async with connection() as conn: | ||
| 69 | − rows = await fetch_all(conn, f""" | 159 | + rows = await fetch_all(conn, """select coalesce(e.attributes->>'license_key', e.attributes->>'license') as raw, count(*) as n from entities e | ||
| 70 | − with v as (select split_part(regexp_replace(path, '[?#].*$', ''), '/', 3) as slug, sum(views) as views from page_views | 160 | + where e.entity_type = 'model' and e.merged_into is null and (e.attributes ? 'license' or e.attributes ? 'license_key') group by 1""") | ||
| 71 | − where day >= ((now() at time zone 'UTC') - make_interval(days => :d))::date and path ~ '^/[a-z-]+/[^/?#]+' group by 1) | 161 | + counts: dict[str, int] = {} | ||
| 72 | − select v.views, {ENTITY_COLS} from v join entities e on e.slug = v.slug left join entities eo on eo.id = e.organization_id | 162 | + unclassified: dict[str, int] = {} | ||
| 73 | − where e.merged_into is null {"and e.entity_type = :t" if type else ""} order by v.views desc, e.updated_at desc limit :lim""", d=days, lim=limit, t=type) | 163 | + for r in rows: | ||
| 74 | − return {"days": days, "items": [{**(entity_summary(r) or {}), "views": int(r["views"] or 0)} for r in rows]} | 164 | + key = r["raw"] if r["raw"] in LICENSES else normalize_license(r["raw"]) | ||
| 165 | + if key: | ||||
| 166 | + counts[key] = counts.get(key, 0) + int(r["n"]) | ||||
| 167 | + elif r["raw"]: | ||||
| 168 | + unclassified[r["raw"]] = unclassified.get(r["raw"], 0) + int(r["n"]) | ||||
| 169 | + items = [{**info.as_dict(), "aliases": list(info.aliases), "models": counts.get(key, 0)} for key, info in LICENSES.items()] | ||||
| 170 | + items.sort(key=lambda x: (-x["models"], x["label"])) | ||||
| 171 | + return {"items": items, "total": len(items), "categories": list(CATEGORIES), "unclassified": [{"raw": k, "models": v} for k, v in sorted(unclassified.items(), key=lambda kv: -kv[1])], | ||||
| 172 | + "note": "Permissions are read from the licence text (null = the text is ambiguous). Counts cover canonical models only."} | ||||
| 173 | + | ||||
| 174 | + | ||||
| 175 | +@router.get("/licenses/{key}") | ||||
| 176 | +@cached(600) | ||||
| 177 | +async def license_detail(request: Request, key: str, limit: int = Query(50, ge=1, le=200)) -> dict[str, Any]: | ||||
| 178 | + canon = key if key in LICENSES else normalize_license(key) | ||||
| 179 | + info = LICENSES.get(canon) if canon else None | ||||
| 180 | + if not info: | ||||
| 181 | + raise ApiError(404, f"unknown licence {key!r}") | ||||
| 182 | + raw = sorted({info.key.lower(), *(a.lower() for a in info.aliases), *([info.spdx.lower()] if info.spdx else [])}) | ||||
| 183 | + async with connection() as conn: | ||||
| 184 | + rows = await fetch_all(conn, f"""select {ENTITY_COLS} from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null | ||||
| 185 | + and (e.attributes->>'license_key' = :k or lower(e.attributes->>'license') = any(cast(:raw as text[]))) | ||||
| 186 | + order by e.attributes->>'release_date' desc nulls last, e.canonical_name limit :lim""", k=info.key, raw=raw, lim=limit) | ||||
| 187 | + total = await fetch_one(conn, """select count(*) as n from entities e where e.entity_type = 'model' and e.merged_into is null | ||||
| 188 | + and (e.attributes->>'license_key' = :k or lower(e.attributes->>'license') = any(cast(:raw as text[])))""", k=info.key, raw=raw) | ||||
| 189 | + return {**info.as_dict(), "aliases": list(info.aliases), "models": {"items": [entity_summary(r) for r in rows], "total": int((total or {}).get("n") or 0), "limit": limit, "offset": 0}} | ||||
| 75 | 190 | ||||
| 76 | 191 | ||||
| 77 | class ViewBeacon(BaseModel): | 192 | class ViewBeacon(BaseModel): | ||
@@ -113,3 +228,6 @@ async def api_key_me(request: Request) -> dict[str, Any]: | |||||
| 113 | if not row: | 228 | if not row: | ||
| 114 | raise ApiError(401, "unknown or disabled API key") | 229 | raise ApiError(401, "unknown or disabled API key") | ||
| 115 | return row | 230 | return row | ||
| 231 | + | ||||
| 232 | + | ||||
| 233 | +__all__ = ["ANOMALY_CHECKS", "COMPARABILITY_RULES", "router"] | ||||
modified
src/aiatlas/api/routers/models.py
+130 −23
@@ -1,4 +1,7 @@ | |||||
| 1 | −"""/models listing (filters on `entities.attributes`, facets) and /models/{slug} alias.""" | 1 | +"""/models listing (canonical universe, filters on `entities.attributes`, facets), /models/{slug} alias, /models/{a}/diff/{b}. | ||
| 2 | + | ||||
| 3 | +API 1.1: the default universe is CANONICAL MODELS (`entity_type = 'model' and merged_into is null`). `include=artifacts` adds | ||||
| 4 | +checkpoints / quantisations / conversions as rows with `entity_type: 'artifact'` and a `canonical` summary.""" | ||||
| 2 | from __future__ import annotations | 5 | from __future__ import annotations | ||
| 3 | 6 | ||||
| 4 | import asyncio | 7 | import asyncio | ||
@@ -9,18 +12,24 @@ from fastapi import APIRouter, Query, Request | |||||
| 9 | from aiatlas.api.common import ( | 12 | from aiatlas.api.common import ( | ||
| 10 | ENTITY_COLS, | 13 | ENTITY_COLS, | ||
| 11 | ENTITY_FROM, | 14 | ENTITY_FROM, | ||
| 15 | + MODEL_OR_ARTIFACT_UNIVERSE, | ||||
| 16 | + MODEL_UNIVERSE, | ||||
| 12 | PAGINATION, | 17 | PAGINATION, | ||
| 13 | ApiError, | 18 | ApiError, | ||
| 14 | Pagination, | 19 | Pagination, | ||
| 20 | + attr_num, | ||||
| 15 | cached, | 21 | cached, | ||
| 22 | + entity_cols, | ||||
| 16 | entity_summary, | 23 | entity_summary, | ||
| 17 | flip_order, | 24 | flip_order, | ||
| 18 | normalize, | 25 | normalize, | ||
| 19 | num_expr, | 26 | num_expr, | ||
| 27 | + openness_values, | ||||
| 20 | page, | 28 | page, | ||
| 21 | ) | 29 | ) | ||
| 22 | from aiatlas.api.routers.entities import detail_for_type | 30 | from aiatlas.api.routers.entities import detail_for_type | ||
| 23 | from aiatlas.db import connection, fetch_all, fetch_val | 31 | from aiatlas.db import connection, fetch_all, fetch_val | ||
| 32 | +from aiatlas.ontology.licenses import LICENSES, normalize_license | ||||
| 24 | from aiatlas.services import cache | 33 | from aiatlas.services import cache | ||
| 25 | 34 | ||||
| 26 | router = APIRouter(prefix="/api/v1/models", tags=["models"]) | 35 | router = APIRouter(prefix="/api/v1/models", tags=["models"]) | ||
@@ -32,13 +41,32 @@ SORTS = { | |||||
| 32 | "updated": "e.updated_at desc", "name": "e.canonical_name asc", "params": f"{PARAMS} desc nulls last", "context": f"{CONTEXT} desc nulls last", | 41 | "updated": "e.updated_at desc", "name": "e.canonical_name asc", "params": f"{PARAMS} desc nulls last", "context": f"{CONTEXT} desc nulls last", | ||
| 33 | "release": "e.attributes->>'release_date' desc nulls last", "quality": "coalesce((e.quality->>'score')::float, 0) desc", "downloads": f"{DOWNLOADS} desc nulls last", | 42 | "release": "e.attributes->>'release_date' desc nulls last", "quality": "coalesce((e.quality->>'score')::float, 0) desc", "downloads": f"{DOWNLOADS} desc nulls last", | ||
| 34 | "first_seen": "e.first_seen_at desc", | 43 | "first_seen": "e.first_seen_at desc", | ||
| 44 | + "cheapest": "(select min(p.output_per_mtok) from prices p where p.model_id = e.id and p.valid_to is null and p.output_per_mtok > 0) asc nulls last", | ||||
| 35 | } | 45 | } | ||
| 46 | +FAMILY_JOIN = "left join entities fam on fam.id = e.family_id" | ||||
| 47 | +CANONICAL_JOIN = "left join entities can on can.id = e.canonical_id left join entities cano on cano.id = can.organization_id" | ||||
| 48 | + | ||||
| 49 | + | ||||
| 50 | +def license_match_sql(param: str = "license") -> str: | ||||
| 51 | + """Match a canonical licence key (`attributes.license_key`) OR any raw label the ontology maps to it OR a plain ilike on the raw label.""" | ||||
| 52 | + return f"(e.attributes->>'license_key' = :{param} or lower(e.attributes->>'license') = any(cast(:{param}_raw as text[])) or e.attributes->>'license' ilike :{param})" | ||||
| 53 | + | ||||
| 54 | + | ||||
| 55 | +def license_params(value: str) -> dict[str, Any]: | ||||
| 56 | + key = value if value in LICENSES else (normalize_license(value) or value) | ||||
| 57 | + info = LICENSES.get(key) | ||||
| 58 | + raw = {key.lower(), value.lower()} | ||||
| 59 | + if info: | ||||
| 60 | + raw |= {a.lower() for a in info.aliases} | ||||
| 61 | + if info.spdx: | ||||
| 62 | + raw.add(info.spdx.lower()) | ||||
| 63 | + return {"license": key, "license_raw": sorted(raw)} | ||||
| 36 | 64 | ||||
| 37 | 65 | ||||
| 38 | def model_filters(*, q: str | None, org: str | None, family: str | None, openness: str | None, modality: str | None, status: str | None, | 66 | def model_filters(*, q: str | None, org: str | None, family: str | None, openness: str | None, modality: str | None, status: str | None, | ||
| 39 | min_params: float | None, max_params: float | None, min_context: int | None, year_from: int | None, year_to: int | None, | 67 | min_params: float | None, max_params: float | None, min_context: int | None, year_from: int | None, year_to: int | None, | ||
| 40 | − license: str | None) -> tuple[list[str], dict[str, Any]]: | 68 | + license: str | None, include_artifacts: bool = False, reasoning: bool | None = None, trust: str | None = None) -> tuple[list[str], dict[str, Any]]: | ||
| 41 | − where = ["e.entity_type = 'model'", "e.merged_into is null"] | 69 | + where = [MODEL_OR_ARTIFACT_UNIVERSE if include_artifacts else MODEL_UNIVERSE] | ||
| 42 | p: dict[str, Any] = {} | 70 | p: dict[str, Any] = {} | ||
| 43 | if q: | 71 | if q: | ||
| 44 | where.append("(e.canonical_name ilike :qlike or e.search @@ plainto_tsquery('simple', :q) or exists (select 1 from entity_aliases a where a.entity_id = e.id and a.alias ilike :qlike))") | 72 | where.append("(e.canonical_name ilike :qlike or e.search @@ plainto_tsquery('simple', :q) or exists (select 1 from entity_aliases a where a.entity_id = e.id and a.alias ilike :qlike))") | ||
@@ -47,14 +75,12 @@ def model_filters(*, q: str | None, org: str | None, family: str | None, opennes | |||||
| 47 | where.append("(eo.slug = :org or eo.id = :org or eo.canonical_name ilike :org)") | 75 | where.append("(eo.slug = :org or eo.id = :org or eo.canonical_name ilike :org)") | ||
| 48 | p["org"] = org | 76 | p["org"] = org | ||
| 49 | if family: | 77 | if family: | ||
| 50 | − where.append("e.attributes->>'family' ilike :family") | 78 | + # family slug / id (model_family entity via family_id) — falls back to the legacy `attributes.family` label | ||
| 79 | + where.append("(fam.slug = :family or fam.id = :family or fam.canonical_name ilike :family or e.attributes->>'family' ilike :family)") | ||||
| 51 | p["family"] = family | 80 | p["family"] = family | ||
| 52 | if openness: | 81 | if openness: | ||
| 53 | − vals = [v.strip() for v in openness.split(",") if v.strip()] | ||||
| 54 | − if "open" in vals: | ||||
| 55 | − vals += ["open-weights", "open-source"] | ||||
| 56 | where.append("e.attributes->>'openness' = any(cast(:openness as text[]))") | 82 | where.append("e.attributes->>'openness' = any(cast(:openness as text[]))") | ||
| 57 | − p["openness"] = vals | 83 | + p["openness"] = openness_values(openness) | ||
| 58 | if modality: | 84 | if modality: | ||
| 59 | where.append("(e.attributes->'modalities' ? :modality or e.attributes->'modalities_input' ? :modality or e.attributes->'modalities_output' ? :modality)") | 85 | where.append("(e.attributes->'modalities' ? :modality or e.attributes->'modalities_input' ? :modality or e.attributes->'modalities_output' ? :modality)") | ||
| 60 | p["modality"] = modality | 86 | p["modality"] = modality | ||
@@ -77,31 +103,61 @@ def model_filters(*, q: str | None, org: str | None, family: str | None, opennes | |||||
| 77 | where.append("left(e.attributes->>'release_date', 4) <= :yt") | 103 | where.append("left(e.attributes->>'release_date', 4) <= :yt") | ||
| 78 | p["yt"] = str(year_to) | 104 | p["yt"] = str(year_to) | ||
| 79 | if license: | 105 | if license: | ||
| 80 | − where.append("e.attributes->>'license' ilike :license") | 106 | + where.append(license_match_sql()) | ||
| 81 | − p["license"] = license | 107 | + p.update(license_params(license)) | ||
| 108 | + if reasoning is not None: | ||||
| 109 | + where.append("e.attributes->>'reasoning' = :reasoning") | ||||
| 110 | + p["reasoning"] = "true" if reasoning else "false" | ||||
| 111 | + if trust: | ||||
| 112 | + where.append("e.identity_confidence = any(cast(:trust as text[]))") | ||||
| 113 | + p["trust"] = [v.strip() for v in trust.split(",") if v.strip()] | ||||
| 82 | return where, p | 114 | return where, p | ||
| 83 | 115 | ||||
| 84 | 116 | ||||
| 117 | +def _canonical_licenses(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: | ||||
| 118 | + """Fold raw licence labels into canonical keys (Apache 2.0 / apache-2.0 → Apache-2.0); unknown labels are kept raw under `raw: true`.""" | ||||
| 119 | + agg: dict[str, dict[str, Any]] = {} | ||||
| 120 | + for r in rows: | ||||
| 121 | + raw = r["value"] | ||||
| 122 | + if raw in (None, ""): | ||||
| 123 | + continue | ||||
| 124 | + key = normalize_license(raw) | ||||
| 125 | + k = key or str(raw) | ||||
| 126 | + item = agg.setdefault(k, {"value": k, "label": LICENSES[key].label if key else str(raw), "category": LICENSES[key].category if key else "unknown", | ||||
| 127 | + "count": 0, "raw_labels": [], **({} if key else {"raw": True})}) | ||||
| 128 | + item["count"] += int(r["count"]) | ||||
| 129 | + if str(raw) != k: | ||||
| 130 | + item["raw_labels"].append(str(raw)) | ||||
| 131 | + return sorted(agg.values(), key=lambda x: (-x["count"], x["value"])) | ||||
| 132 | + | ||||
| 133 | + | ||||
| 85 | async def model_facets(where_sql: str, params: dict[str, Any]) -> dict[str, Any]: | 134 | async def model_facets(where_sql: str, params: dict[str, Any]) -> dict[str, Any]: | ||
| 86 | async def run(sql: str) -> list[dict[str, Any]]: | 135 | async def run(sql: str) -> list[dict[str, Any]]: | ||
| 87 | async with connection() as conn: | 136 | async with connection() as conn: | ||
| 88 | return await fetch_all(conn, sql, **params) | 137 | return await fetch_all(conn, sql, **params) | ||
| 89 | 138 | ||||
| 90 | − base = f"from {ENTITY_FROM} where {where_sql}" | 139 | + base = f"from {ENTITY_FROM} {FAMILY_JOIN} where {where_sql}" | ||
| 91 | − mods_from = (f"from {ENTITY_FROM} cross join lateral jsonb_array_elements_text(case when jsonb_typeof(e.attributes->'modalities') = 'array' " | 140 | + mods_from = (f"from {ENTITY_FROM} {FAMILY_JOIN} cross join lateral jsonb_array_elements_text(case when jsonb_typeof(e.attributes->'modalities') = 'array' " | ||
| 92 | f"then e.attributes->'modalities' else '[]'::jsonb end) m where {where_sql}") | 141 | f"then e.attributes->'modalities' else '[]'::jsonb end) m where {where_sql}") | ||
| 93 | − orgs, openness, mods, fams, years, lics, status = await asyncio.gather( | 142 | + orgs, openness, mods, fams, years, lics, status, trust = await asyncio.gather( | ||
| 94 | run(f"select eo.slug, eo.canonical_name as name, count(*) as count {base} and eo.id is not null group by 1, 2 order by 3 desc, 2 limit 60"), | 143 | run(f"select eo.slug, eo.canonical_name as name, count(*) as count {base} and eo.id is not null group by 1, 2 order by 3 desc, 2 limit 60"), | ||
| 95 | run(f"select e.attributes->>'openness' as value, count(*) as count {base} and e.attributes ? 'openness' group by 1 order by 2 desc"), | 144 | run(f"select e.attributes->>'openness' as value, count(*) as count {base} and e.attributes ? 'openness' group by 1 order by 2 desc"), | ||
| 96 | run(f"select m.value, count(*) as count {mods_from} group by 1 order by 2 desc limit 30"), | 145 | run(f"select m.value, count(*) as count {mods_from} group by 1 order by 2 desc limit 30"), | ||
| 97 | − run(f"select e.attributes->>'family' as value, count(*) as count {base} and e.attributes ? 'family' group by 1 order by 2 desc, 1 limit 60"), | 146 | + run(f"select coalesce(fam.slug, e.attributes->>'family') as value, coalesce(fam.canonical_name, e.attributes->>'family') as label, fam.id is not null as canonical, " | ||
| 147 | + f"count(*) as count {base} and (fam.id is not null or e.attributes ? 'family') group by 1, 2, 3 order by 4 desc, 2 limit 60"), | ||||
| 98 | run(f"select left(e.attributes->>'release_date', 4) as value, count(*) as count {base} and e.attributes ? 'release_date' group by 1 order by 1 desc limit 30"), | 148 | run(f"select left(e.attributes->>'release_date', 4) as value, count(*) as count {base} and e.attributes ? 'release_date' group by 1 order by 1 desc limit 30"), | ||
| 99 | − run(f"select e.attributes->>'license' as value, count(*) as count {base} and e.attributes ? 'license' group by 1 order by 2 desc, 1 limit 40"), | 149 | + run(f"select coalesce(e.attributes->>'license_key', e.attributes->>'license') as value, count(*) as count {base} and (e.attributes ? 'license' or e.attributes ? 'license_key') group by 1 order by 2 desc, 1 limit 80"), | ||
| 100 | run(f"select e.status as value, count(*) as count {base} group by 1 order by 2 desc"), | 150 | run(f"select e.status as value, count(*) as count {base} group by 1 order by 2 desc"), | ||
| 151 | + run(f"select e.identity_confidence as value, count(*) as count {base} group by 1 order by 2 desc"), | ||||
| 101 | ) | 152 | ) | ||
| 102 | conv = lambda rows: [{"value": r["value"], "count": int(r["count"])} for r in rows if r["value"] not in (None, "")] | 153 | conv = lambda rows: [{"value": r["value"], "count": int(r["count"])} for r in rows if r["value"] not in (None, "")] | ||
| 103 | return {"organizations": [{"slug": r["slug"], "name": r["name"], "count": int(r["count"])} for r in orgs], "openness": conv(openness), "modalities": conv(mods), | 154 | return {"organizations": [{"slug": r["slug"], "name": r["name"], "count": int(r["count"])} for r in orgs], "openness": conv(openness), "modalities": conv(mods), | ||
| 104 | − "families": conv(fams), "years": conv(years), "licenses": conv(lics), "status": conv(status)} | 155 | + "families": [{"value": r["value"], "label": r["label"], "canonical": bool(r["canonical"]), "count": int(r["count"])} for r in fams if r["value"]], | ||
| 156 | + "years": conv(years), "licenses": _canonical_licenses(lics), "status": conv(status), | ||||
| 157 | + "trust": [{**x, "label": {"high": "Identity confirmed", "medium": "Identity probable", "low": "Identity uncertain"}.get(x["value"], x["value"])} for x in conv(trust)], | ||||
| 158 | + "definitions": {"trust": "identity_confidence of the row: how sure AI Atlas is that this entry is one real model release (high | medium | low)", | ||||
| 159 | + "licenses": "canonical licence keys from the ontology; raw labels that could not be classified are flagged raw: true", | ||||
| 160 | + "families": "model_family entities (canonical: true) or legacy attribute labels (canonical: false)"}} | ||||
| 105 | 161 | ||||
| 106 | 162 | ||||
| 107 | @router.get("") | 163 | @router.get("") | ||
@@ -110,25 +166,41 @@ async def list_models(request: Request, p: Pagination = PAGINATION, q: str | Non | |||||
| 110 | openness: str | None = None, modality: str | None = None, status: str | None = None, min_params: float | None = Query(None, ge=0), | 166 | openness: str | None = None, modality: str | None = None, status: str | None = None, min_params: float | None = Query(None, ge=0), | ||
| 111 | max_params: float | None = Query(None, ge=0), min_context: int | None = Query(None, ge=0), year_from: int | None = Query(None, ge=1950, le=2100), | 167 | max_params: float | None = Query(None, ge=0), min_context: int | None = Query(None, ge=0), year_from: int | None = Query(None, ge=1950, le=2100), | ||
| 112 | year_to: int | None = Query(None, ge=1950, le=2100), license: str | None = None, sort: str = "updated", order: str = "", | 168 | year_to: int | None = Query(None, ge=1950, le=2100), license: str | None = None, sort: str = "updated", order: str = "", | ||
| 113 | − facets: int = Query(0, ge=0, le=1)) -> dict[str, Any]: | 169 | + facets: int = Query(0, ge=0, le=1), include: str | None = Query(None, description="`artifacts` restores the pre-1.1 universe (models + artifacts)"), | ||
| 170 | + reasoning: int | None = Query(None, ge=0, le=1), trust: str | None = Query(None, description="identity_confidence: high,medium,low")) -> dict[str, Any]: | ||||
| 114 | if sort not in SORTS: | 171 | if sort not in SORTS: | ||
| 115 | raise ApiError(400, f"sort must be one of {', '.join(SORTS)}") | 172 | raise ApiError(400, f"sort must be one of {', '.join(SORTS)}") | ||
| 173 | + include_artifacts = "artifacts" in {x.strip() for x in (include or "").split(",")} | ||||
| 116 | where, params = model_filters(q=q, org=org, family=family, openness=openness, modality=modality, status=status, min_params=min_params, max_params=max_params, | 174 | where, params = model_filters(q=q, org=org, family=family, openness=openness, modality=modality, status=status, min_params=min_params, max_params=max_params, | ||
| 117 | − min_context=min_context, year_from=year_from, year_to=year_to, license=license) | 175 | + min_context=min_context, year_from=year_from, year_to=year_to, license=license, include_artifacts=include_artifacts, | ||
| 176 | + reasoning=None if reasoning is None else bool(reasoning), trust=trust) | ||||
| 118 | order_sql = flip_order(SORTS[sort], order) | 177 | order_sql = flip_order(SORTS[sort], order) | ||
| 119 | where_sql = " and ".join(where) | 178 | where_sql = " and ".join(where) | ||
| 120 | async with connection() as conn: | 179 | async with connection() as conn: | ||
| 121 | − rows = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where {where_sql} order by {order_sql}, e.id limit :lim offset :off", | 180 | + rows = await fetch_all(conn, f"""select {ENTITY_COLS}, e.family_id, e.canonical_id, e.artifact_kind, e.identity_confidence, | ||
| 181 | + fam.slug as family_slug, fam.canonical_name as family_name, {entity_cols("can", "c_")} | ||||
| 182 | + from {ENTITY_FROM} {FAMILY_JOIN} {CANONICAL_JOIN} where {where_sql} order by {order_sql}, e.id limit :lim offset :off""", | ||||
| 122 | lim=p.limit, off=p.offset, **params) | 183 | lim=p.limit, off=p.offset, **params) | ||
| 123 | − total = await fetch_val(conn, f"select count(*) from {ENTITY_FROM} where {where_sql}", **params) | 184 | + total = await fetch_val(conn, f"select count(*) from {ENTITY_FROM} {FAMILY_JOIN} where {where_sql}", **params) | ||
| 124 | − out = page([entity_summary(r) for r in rows], int(total or 0), p) | 185 | + items = [] | ||
| 186 | + for r in rows: | ||||
| 187 | + s = entity_summary(r) or {} | ||||
| 188 | + if r.get("family_id"): | ||||
| 189 | + s["family"] = {"id": r["family_id"], "slug": r["family_slug"], "name": r["family_name"]} | ||||
| 190 | + if r["entity_type"] == "artifact": | ||||
| 191 | + s["artifact_kind"] = r.get("artifact_kind") | ||||
| 192 | + s["canonical"] = entity_summary(r, "c_") | ||||
| 193 | + s["identity_confidence"] = r.get("identity_confidence") | ||||
| 194 | + items.append(s) | ||||
| 195 | + out = page(items, int(total or 0), p) | ||||
| 196 | + out["universe"] = "models+artifacts" if include_artifacts else "canonical models" | ||||
| 125 | if facets: | 197 | if facets: | ||
| 126 | out["facets"] = await _facets_cached(where_sql, params) | 198 | out["facets"] = await _facets_cached(where_sql, params) | ||
| 127 | return out | 199 | return out | ||
| 128 | 200 | ||||
| 129 | 201 | ||||
| 130 | async def _facets_cached(where_sql: str, params: dict[str, Any]) -> dict[str, Any]: | 202 | async def _facets_cached(where_sql: str, params: dict[str, Any]) -> dict[str, Any]: | ||
| 131 | − key = "facets:models:" + where_sql + ":" + repr(sorted(params.items())) | 203 | + key = "facets:models:v2:" + where_sql + ":" + repr(sorted(params.items())) | ||
| 132 | hit = await cache.cache_get(key) | 204 | hit = await cache.cache_get(key) | ||
| 133 | if hit is not None: | 205 | if hit is not None: | ||
| 134 | return hit | 206 | return hit | ||
@@ -140,4 +212,39 @@ async def _facets_cached(where_sql: str, params: dict[str, Any]) -> dict[str, An | |||||
| 140 | @router.get("/{slug}") | 212 | @router.get("/{slug}") | ||
| 141 | @cached(300) | 213 | @cached(300) | ||
| 142 | async def get_model(request: Request, slug: str) -> dict[str, Any]: | 214 | async def get_model(request: Request, slug: str) -> dict[str, Any]: | ||
| 143 | − return await detail_for_type(slug, ("model",)) | 215 | + """Accepts canonical models AND artifacts (an artifact detail carries `canonical` + `artifact_kind`); folded variants follow `merged_into` | ||
| 216 | + and report `redirected_from` so the web layer can 301.""" | ||||
| 217 | + return await detail_for_type(slug, ("model", "artifact")) | ||||
| 218 | + | ||||
| 219 | + | ||||
| 220 | +@router.get("/{a}/diff/{b}") | ||||
| 221 | +@cached(300) | ||||
| 222 | +async def model_diff(request: Request, a: str, b: str) -> dict[str, Any]: | ||||
| 223 | + """Only the dimensions where two models differ, with a delta (numeric % change, list added/removed).""" | ||||
| 224 | + from aiatlas.api.routers.compare import compare_entities | ||||
| 225 | + | ||||
| 226 | + body = await compare_entities([a, b], diff_only=True) | ||||
| 227 | + items = body["items"] | ||||
| 228 | + if len(items) != 2: | ||||
| 229 | + raise ApiError(400, "two models are required") | ||||
| 230 | + va, vb = items[0]["values"], items[1]["values"] | ||||
| 231 | + dims = [] | ||||
| 232 | + for d in body["dimensions"]: | ||||
| 233 | + x, y = va.get(d["key"]), vb.get(d["key"]) | ||||
| 234 | + dims.append({**d, "a": x, "b": y, "delta": _delta(x, y, d.get("kind"))}) | ||||
| 235 | + return {"a": items[0]["entity"], "b": items[1]["entity"], "dimensions": dims, "comparability": body.get("comparability"), | ||||
| 236 | + "note": "Only dimensions with differing observed values; numeric delta = (b − a) / a; lists show added/removed elements."} | ||||
| 237 | + | ||||
| 238 | + | ||||
| 239 | +def _delta(x: Any, y: Any, kind: str | None) -> dict[str, Any] | None: | ||||
| 240 | + if isinstance(x, list) or isinstance(y, list): | ||||
| 241 | + sx, sy = {str(v) for v in (x or [])}, {str(v) for v in (y or [])} | ||||
| 242 | + return {"added": sorted(sy - sx), "removed": sorted(sx - sy)} | ||||
| 243 | + try: | ||||
| 244 | + fx, fy = float(x), float(y) | ||||
| 245 | + except (TypeError, ValueError): | ||||
| 246 | + return None | ||||
| 247 | + return {"absolute": fy - fx, "percent": round((fy - fx) / fx * 100, 2) if fx else None} | ||||
| 248 | + | ||||
| 249 | + | ||||
| 250 | +__all__ = ["CONTEXT", "PARAMS", "attr_num", "license_match_sql", "license_params", "model_filters", "router"] | ||||
modified
src/aiatlas/api/routers/prices.py
+100 −17
@@ -1,4 +1,7 @@ | |||||
| 1 | −"""/prices · /prices/history · /prices/index — append-only pricing table (`valid_to` closes a row).""" | 1 | +"""/prices · /prices/history · /prices/index — append-only pricing table (`valid_to` closes a row). | ||
| 2 | + | ||||
| 3 | +API 1.1: `/prices` gains `family=`, `org=`, `modality=` filters and `sort=cheapest_frontier`; `/prices/index` becomes the AI Price Index | ||||
| 4 | +(daily medians for all / frontier / open / embedding offers, cheapest frontier offer, current distribution, listings and delistings).""" | ||||
| 2 | from __future__ import annotations | 5 | from __future__ import annotations | ||
| 3 | 6 | ||||
| 4 | from typing import Any | 7 | from typing import Any | ||
@@ -8,6 +11,7 @@ from fastapi import APIRouter, Query, Request | |||||
| 8 | from aiatlas.api.common import ( | 11 | from aiatlas.api.common import ( | ||
| 9 | EVENT_COLS, | 12 | EVENT_COLS, | ||
| 10 | EVENT_FROM, | 13 | EVENT_FROM, | ||
| 14 | + OPEN_CATEGORIES, | ||||
| 11 | PAGINATION, | 15 | PAGINATION, | ||
| 12 | PRICE_COLS, | 16 | PRICE_COLS, | ||
| 13 | PRICE_FROM, | 17 | PRICE_FROM, | ||
@@ -19,17 +23,19 @@ from aiatlas.api.common import ( | |||||
| 19 | price_row, | 23 | price_row, | ||
| 20 | resolve_id, | 24 | resolve_id, | ||
| 21 | ) | 25 | ) | ||
| 22 | −from aiatlas.db import connection, fetch_all, fetch_val | 26 | +from aiatlas.db import connection, fetch_all, fetch_one, fetch_val | ||
| 27 | +from aiatlas.services.frontier import FRONTIER_METHODOLOGY, frontier_model_ids | ||||
| 23 | 28 | ||||
| 24 | router = APIRouter(prefix="/api/v1/prices", tags=["prices"]) | 29 | router = APIRouter(prefix="/api/v1/prices", tags=["prices"]) | ||
| 25 | SORTS = {"input": "p.input_per_mtok asc nulls last", "output": "p.output_per_mtok asc nulls last", "model": "m.canonical_name asc", "provider": "pv.canonical_name asc", | 30 | SORTS = {"input": "p.input_per_mtok asc nulls last", "output": "p.output_per_mtok asc nulls last", "model": "m.canonical_name asc", "provider": "pv.canonical_name asc", | ||
| 26 | − "observed": "p.observed_at desc", "valid_from": "p.valid_from desc"} | 31 | + "observed": "p.observed_at desc", "valid_from": "p.valid_from desc", "cheapest_frontier": "p.output_per_mtok asc nulls last"} | ||
| 32 | +BUCKETS = [(0, 0.1), (0.1, 0.5), (0.5, 1), (1, 2), (2, 5), (5, 10), (10, 20), (20, 50), (50, None)] | ||||
| 27 | 33 | ||||
| 28 | 34 | ||||
| 29 | @router.get("") | 35 | @router.get("") | ||
| 30 | @cached(300) | 36 | @cached(300) | ||
| 31 | async def list_prices(request: Request, p: Pagination = PAGINATION, model: str | None = None, provider: str | None = None, sort: str = "model", order: str = "", | 37 | async def list_prices(request: Request, p: Pagination = PAGINATION, model: str | None = None, provider: str | None = None, sort: str = "model", order: str = "", | ||
| 32 | − current: int = Query(1, ge=0, le=1)) -> dict[str, Any]: | 38 | + current: int = Query(1, ge=0, le=1), family: str | None = None, org: str | None = None, modality: str | None = None) -> dict[str, Any]: | ||
| 33 | if sort not in SORTS: | 39 | if sort not in SORTS: | ||
| 34 | raise ApiError(400, f"sort must be one of {', '.join(SORTS)}") | 40 | raise ApiError(400, f"sort must be one of {', '.join(SORTS)}") | ||
| 35 | order_sql = SORTS[sort] | 41 | order_sql = SORTS[sort] | ||
@@ -38,7 +44,9 @@ async def list_prices(request: Request, p: Pagination = PAGINATION, model: str | | |||||
| 38 | elif order == "asc" and " desc" in order_sql: | 44 | elif order == "asc" and " desc" in order_sql: | ||
| 39 | order_sql = order_sql.replace(" desc", " asc") | 45 | order_sql = order_sql.replace(" desc", " asc") | ||
| 40 | where = ["p.valid_to is null"] if current else ["true"] | 46 | where = ["p.valid_to is null"] if current else ["true"] | ||
| 47 | + where.append("m.entity_type in ('model','artifact')") | ||||
| 41 | params: dict[str, Any] = {} | 48 | params: dict[str, Any] = {} | ||
| 49 | + frontier_note = None | ||||
| 42 | async with connection() as conn: | 50 | async with connection() as conn: | ||
| 43 | if model: | 51 | if model: | ||
| 44 | where.append("p.model_id = :model") | 52 | where.append("p.model_id = :model") | ||
@@ -46,10 +54,28 @@ async def list_prices(request: Request, p: Pagination = PAGINATION, model: str | | |||||
| 46 | if provider: | 54 | if provider: | ||
| 47 | where.append("p.provider_id = :provider") | 55 | where.append("p.provider_id = :provider") | ||
| 48 | params["provider"] = await resolve_id(conn, provider) | 56 | params["provider"] = await resolve_id(conn, provider) | ||
| 57 | + if family: | ||||
| 58 | + where.append("(exists (select 1 from entities f where f.id = m.family_id and (f.slug = :family or f.id = :family or f.canonical_name ilike :family)) or m.attributes->>'family' ilike :family)") | ||||
| 59 | + params["family"] = family | ||||
| 60 | + if org: | ||||
| 61 | + where.append("(mo.slug = :org or mo.id = :org or mo.canonical_name ilike :org)") | ||||
| 62 | + params["org"] = org | ||||
| 63 | + if modality: | ||||
| 64 | + where.append("(m.attributes->'modalities' ? :modality or m.attributes->'modalities_input' ? :modality or m.attributes->'modalities_output' ? :modality)") | ||||
| 65 | + params["modality"] = modality | ||||
| 66 | + if sort == "cheapest_frontier": | ||||
| 67 | + ids, _ = await frontier_model_ids(conn) | ||||
| 68 | + where.append("p.model_id = any(cast(:frontier as text[]))") | ||||
| 69 | + where.append("p.output_per_mtok > 0") | ||||
| 70 | + params["frontier"] = sorted(ids) | ||||
| 71 | + frontier_note = FRONTIER_METHODOLOGY | ||||
| 49 | where_sql = " and ".join(where) | 72 | where_sql = " and ".join(where) | ||
| 50 | rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {where_sql} order by {order_sql}, p.id limit :lim offset :off", lim=p.limit, off=p.offset, **params) | 73 | rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {where_sql} order by {order_sql}, p.id limit :lim offset :off", lim=p.limit, off=p.offset, **params) | ||
| 51 | − total = await fetch_val(conn, f"select count(*) from prices p where {where_sql}", **params) | 74 | + total = await fetch_val(conn, f"select count(*) from {PRICE_FROM} where {where_sql}", **params) | ||
| 52 | − return page([price_row(r) for r in rows], int(total or 0), p) | 75 | + out = page([price_row(r) for r in rows], int(total or 0), p) | ||
| 76 | + if frontier_note: | ||||
| 77 | + out["methodology"] = frontier_note | ||||
| 78 | + return out | ||||
| 53 | 79 | ||||
| 54 | 80 | ||||
| 55 | @router.get("/history") | 81 | @router.get("/history") | ||
@@ -73,18 +99,75 @@ async def price_history(request: Request, model: str | None = None, provider: st | |||||
| 73 | @router.get("/index") | 99 | @router.get("/index") | ||
| 74 | @cached(600) | 100 | @cached(600) | ||
| 75 | async def price_index(request: Request, days: int = Query(180, ge=7, le=1825)) -> dict[str, Any]: | 101 | async def price_index(request: Request, days: int = Query(180, ge=7, le=1825)) -> dict[str, Any]: | ||
| 102 | + """AI Price Index: daily medians of live USD-per-1M-token prices, by universe (all offers / frontier / open-weight / embeddings).""" | ||||
| 76 | async with connection() as conn: | 103 | async with connection() as conn: | ||
| 104 | + frontier, composition = await frontier_model_ids(conn) | ||||
| 105 | + open_ids = [r["id"] for r in await fetch_all(conn, "select id from entities where entity_type = 'model' and merged_into is null and attributes->>'openness' = any(cast(:o as text[]))", o=list(OPEN_CATEGORIES))] | ||||
| 106 | + emb_ids = [r["id"] for r in await fetch_all(conn, """select id from entities where entity_type = 'model' and merged_into is null and (attributes->'modalities' ? 'embedding' | ||||
| 107 | + or attributes->'modalities_output' ? 'embedding' or attributes->>'pipeline_tag' ilike '%embedding%')""")] | ||||
| 77 | series = await fetch_all(conn, """ | 108 | series = await fetch_all(conn, """ | ||
| 78 | with days as (select generate_series(((now() at time zone 'UTC') - make_interval(days => :d))::date::timestamp, (now() at time zone 'UTC')::date::timestamp, interval '1 day')::date as day), | 109 | with days as (select generate_series(((now() at time zone 'UTC') - make_interval(days => :d))::date::timestamp, (now() at time zone 'UTC')::date::timestamp, interval '1 day')::date as day), | ||
| 79 | − live as (select p.model_id, p.input_per_mtok, p.output_per_mtok, p.valid_from, p.valid_to from prices p | 110 | + live as (select p.model_id, p.provider_id, p.input_per_mtok, p.output_per_mtok, p.valid_from, p.valid_to, | ||
| 80 | − where p.input_per_mtok is not null and p.input_per_mtok > 0 and p.valid_from < now()) | 111 | + p.model_id = any(cast(:frontier as text[])) as is_frontier, p.model_id = any(cast(:open as text[])) as is_open, p.model_id = any(cast(:emb as text[])) as is_emb | ||
| 81 | − select d.day, percentile_cont(0.5) within group (order by l.input_per_mtok) as median_input, | 112 | + from prices p join entities m on m.id = p.model_id | ||
| 82 | − percentile_cont(0.5) within group (order by l.output_per_mtok) as median_output, | 113 | + where m.entity_type in ('model','artifact') and m.merged_into is null and ((p.input_per_mtok is not null and p.input_per_mtok > 0) or (p.output_per_mtok is not null and p.output_per_mtok > 0)) and p.valid_from < now()) | ||
| 83 | − min(l.input_per_mtok) as min_input, max(l.input_per_mtok) as max_input, count(distinct l.model_id) as models, count(l.model_id) as offers | 114 | + select d.day, | ||
| 115 | + percentile_cont(0.5) within group (order by l.input_per_mtok) filter (where l.input_per_mtok > 0) as median_input, | ||||
| 116 | + percentile_cont(0.5) within group (order by l.output_per_mtok) filter (where l.output_per_mtok > 0) as median_output, | ||||
| 117 | + percentile_cont(0.5) within group (order by l.output_per_mtok) filter (where l.output_per_mtok > 0 and l.is_frontier) as median_frontier_output, | ||||
| 118 | + percentile_cont(0.5) within group (order by l.input_per_mtok) filter (where l.input_per_mtok > 0 and l.is_frontier) as median_frontier_input, | ||||
| 119 | + percentile_cont(0.5) within group (order by l.output_per_mtok) filter (where l.output_per_mtok > 0 and l.is_open) as median_open_output, | ||||
| 120 | + percentile_cont(0.5) within group (order by l.input_per_mtok) filter (where l.input_per_mtok > 0 and l.is_emb) as median_embedding_input, | ||||
| 121 | + min(l.input_per_mtok) filter (where l.input_per_mtok > 0) as min_input, max(l.input_per_mtok) as max_input, | ||||
| 122 | + min(l.output_per_mtok) filter (where l.output_per_mtok > 0 and l.is_frontier) as min_frontier_output, | ||||
| 123 | + count(distinct l.model_id) as models, count(l.model_id) as offers, | ||||
| 124 | + count(distinct l.model_id) filter (where l.is_frontier) as frontier_models, count(l.model_id) filter (where l.is_frontier) as frontier_offers, | ||||
| 125 | + count(distinct l.model_id) filter (where l.is_open) as open_models, count(distinct l.model_id) filter (where l.is_emb) as embedding_models | ||||
| 84 | from days d left join live l on l.valid_from < (d.day + 1)::timestamp at time zone 'UTC' and (l.valid_to is null or l.valid_to >= (d.day + 1)::timestamp at time zone 'UTC') | 126 | from days d left join live l on l.valid_from < (d.day + 1)::timestamp at time zone 'UTC' and (l.valid_to is null or l.valid_to >= (d.day + 1)::timestamp at time zone 'UTC') | ||
| 85 | − group by d.day order by d.day""", d=days) | 127 | + group by d.day order by d.day""", d=days, frontier=sorted(frontier), open=open_ids, emb=emb_ids) | ||
| 86 | − movers = await fetch_all(conn, f"select {EVENT_COLS} from {EVENT_FROM} where ev.category = 'price' and ev.observed_at > now() - make_interval(days => :d) " | 128 | + movers = await fetch_all(conn, f"select {EVENT_COLS} from {EVENT_FROM} where ev.category = 'price' and ev.is_backfill = false and ev.occurred_at > now() - make_interval(days => :d) " | ||
| 87 | − f"order by ev.importance desc, ev.observed_at desc limit 30", d=days) | 129 | + f"order by ev.importance desc, ev.occurred_at desc limit 30", d=days) | ||
| 88 | − return {"days": days, "series": [{"day": r["day"].isoformat(), "median_input": r["median_input"], "median_output": r["median_output"], "min_input": r["min_input"], | 130 | + cheapest = await fetch_one(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.valid_to is null and p.output_per_mtok > 0 and p.model_id = any(cast(:ids as text[])) " | ||
| 89 | − "max_input": r["max_input"], "models": int(r["models"] or 0), "offers": int(r["offers"] or 0)} for r in series], | 131 | + f"order by p.output_per_mtok asc, p.input_per_mtok asc nulls last limit 1", ids=sorted(frontier)) | ||
| 90 | − "movers": [change_event(r) for r in movers], "note": "Daily medians of live USD-per-1M-token input/output prices across all provider offers valid at the end of each UTC day."} | 132 | + cheapest_1m = await fetch_one(conn, f"""select {PRICE_COLS} from {PRICE_FROM} where p.valid_to is null and p.output_per_mtok > 0 and p.model_id = any(cast(:ids as text[])) | ||
| 133 | + and coalesce(p.context_length, case when m.attributes->>'context_length' ~ '^[0-9]+$' then (m.attributes->>'context_length')::bigint end) >= 1000000 | ||||
| 134 | + order by p.output_per_mtok asc limit 1""", ids=sorted(frontier)) | ||||
| 135 | + current_outputs = await fetch_all(conn, "select p.output_per_mtok as v from prices p join entities m on m.id = p.model_id where p.valid_to is null and p.output_per_mtok > 0 and m.merged_into is null") | ||||
| 136 | + listings = await fetch_one(conn, """ | ||||
| 137 | + select (select count(*) from prices p where p.valid_from > now() - interval '30 days' | ||||
| 138 | + and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_from < p.valid_from)) as new_listings_30d, | ||||
| 139 | + (select count(*) from prices p where p.valid_to > now() - interval '30 days' | ||||
| 140 | + and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_to is null)) as delistings_30d, | ||||
| 141 | + (select count(*) from change_events where category = 'price' and is_backfill = false and occurred_at > now() - interval '30 days') as price_changes_30d""") | ||||
| 142 | + dist = [] | ||||
| 143 | + vals = [float(r["v"]) for r in current_outputs] | ||||
| 144 | + for lo, hi in BUCKETS: | ||||
| 145 | + n = sum(1 for v in vals if v >= lo and (hi is None or v < hi)) | ||||
| 146 | + dist.append({"from": lo, "to": hi, "label": f"${lo:g}–${hi:g}" if hi is not None else f"≥ ${lo:g}", "offers": n}) | ||||
| 147 | + out_series = [] | ||||
| 148 | + for r in series: | ||||
| 149 | + out_series.append({"day": r["day"].isoformat(), "median_input": r["median_input"], "median_output": r["median_output"], "median_frontier_output": r["median_frontier_output"], | ||||
| 150 | + "median_frontier_input": r["median_frontier_input"], "median_open_output": r["median_open_output"], "median_embedding_input": r["median_embedding_input"], | ||||
| 151 | + "min_input": r["min_input"], "max_input": r["max_input"], "min_frontier_output": r["min_frontier_output"], "models": int(r["models"] or 0), "offers": int(r["offers"] or 0), | ||||
| 152 | + "sample": {"models": int(r["models"] or 0), "offers": int(r["offers"] or 0), "frontier_models": int(r["frontier_models"] or 0), "frontier_offers": int(r["frontier_offers"] or 0), | ||||
| 153 | + "open_models": int(r["open_models"] or 0), "embedding_models": int(r["embedding_models"] or 0)}}) | ||||
| 154 | + return {"days": days, "series": out_series, "movers": [change_event(r) for r in movers], | ||||
| 155 | + "cheapest_frontier": _cheap(cheapest), "cheapest_frontier_1m_context": _cheap(cheapest_1m), | ||||
| 156 | + "distribution": {"metric": "output_per_mtok", "unit": "USD per 1M tokens", "buckets": dist, "offers": len(vals)}, | ||||
| 157 | + "new_listings_30d": int((listings or {}).get("new_listings_30d") or 0), "delistings_30d": int((listings or {}).get("delistings_30d") or 0), | ||||
| 158 | + "price_changes_30d": int((listings or {}).get("price_changes_30d") or 0), | ||||
| 159 | + "frontier": {"composition": composition, "methodology": FRONTIER_METHODOLOGY}, | ||||
| 160 | + "methodology": "Daily medians of live USD-per-1M-token prices across every provider offer valid at the end of each UTC day (offers with a zero or missing price are excluded " | ||||
| 161 | + "from that median). `median_open_output` covers models with openness open-weights/open-source; `median_embedding_input` covers models whose modalities include " | ||||
| 162 | + "embedding. Sample sizes are returned per day; a null median means no offer in that universe on that day. " + FRONTIER_METHODOLOGY, | ||||
| 163 | + "note": "Daily medians of live USD-per-1M-token input/output prices across all provider offers valid at the end of each UTC day."} | ||||
| 164 | + | ||||
| 165 | + | ||||
| 166 | +def _cheap(row: dict[str, Any] | None) -> dict[str, Any] | None: | ||||
| 167 | + if not row: | ||||
| 168 | + return None | ||||
| 169 | + pr = price_row(row) | ||||
| 170 | + return {"model": pr["model"], "provider": pr["provider"], "output": pr["output_per_mtok"], "input": pr["input_per_mtok"], "context_length": pr["context_length"], "price_id": pr["id"]} | ||||
| 171 | + | ||||
| 172 | + | ||||
| 173 | +__all__ = ["BUCKETS", "router"] | ||||
modified
src/aiatlas/api/routers/providers.py
+45 −4
@@ -1,4 +1,5 @@ | |||||
| 1 | −"""/providers listing (with pricing aggregates) and /providers/{slug} alias.""" | 1 | +"""/providers listing (with pricing aggregates) and /providers/{slug} alias. API 1.1 adds price distributions, 30-day listing churn, | ||
| 2 | +organizations covered and the union of priced feature keys.""" | ||||
| 2 | from __future__ import annotations | 3 | from __future__ import annotations | ||
| 3 | 4 | ||||
| 4 | from typing import Any | 5 | from typing import Any | ||
@@ -10,6 +11,11 @@ from aiatlas.api.routers.entities import detail_for_type | |||||
| 10 | from aiatlas.db import connection, fetch_all | 11 | from aiatlas.db import connection, fetch_all | ||
| 11 | 12 | ||||
| 12 | router = APIRouter(prefix="/api/v1/providers", tags=["providers"]) | 13 | router = APIRouter(prefix="/api/v1/providers", tags=["providers"]) | ||
| 14 | +FEATURE_KEYS = {"batch": ("batch_input_per_mtok", "batch_output_per_mtok", "batch_enabled", "batch"), "cached": ("cached_input_per_mtok", "cache_write_per_mtok", "cache_write_1h_per_mtok", "input_cache_write_1h", "prompt_caching"), | ||||
| 15 | + "fine_tuning": ("fine_tuning", "fine_tuning_input_per_mtok", "fine_tuning_training_per_mtok"), "flex": ("flex_input_per_mtok", "flex_output_per_mtok"), | ||||
| 16 | + "priority": ("priority", "priority_input_per_mtok", "priority_output_per_mtok"), "long_context": ("long_context_input_per_mtok", "long_context_output_per_mtok"), | ||||
| 17 | + "audio": ("audio", "audio_input_per_mtok", "audio_output_per_mtok"), "image": ("image_input_per_mtok", "image_output", "image_output_per_mtok", "per_image"), | ||||
| 18 | + "web_search": ("web_search", "search_grounding_per_1k_requests"), "free_tier": ("free_tier", "free"), "reasoning": ("internal_reasoning",), "serverless": ("serverless",)} | ||||
| 13 | 19 | ||||
| 14 | 20 | ||||
| 15 | @router.get("") | 21 | @router.get("") | ||
@@ -22,11 +28,46 @@ async def list_providers(request: Request) -> dict[str, Any]: | |||||
| 22 | (select count(*) from relations r where r.object_id = e.id and r.predicate = 'available_through' and r.valid_to is null) as listed_models, | 28 | (select count(*) from relations r where r.object_id = e.id and r.predicate = 'available_through' and r.valid_to is null) as listed_models, | ||
| 23 | (select count(*) from prices p where p.provider_id = e.id and p.valid_to is null) as price_count, | 29 | (select count(*) from prices p where p.provider_id = e.id and p.valid_to is null) as price_count, | ||
| 24 | (select min(p.input_per_mtok) from prices p where p.provider_id = e.id and p.valid_to is null and p.input_per_mtok > 0) as min_input_per_mtok, | 30 | (select min(p.input_per_mtok) from prices p where p.provider_id = e.id and p.valid_to is null and p.input_per_mtok > 0) as min_input_per_mtok, | ||
| 25 | − (select min(p.output_per_mtok) from prices p where p.provider_id = e.id and p.valid_to is null and p.output_per_mtok > 0) as min_output_per_mtok | 31 | + (select min(p.output_per_mtok) from prices p where p.provider_id = e.id and p.valid_to is null and p.output_per_mtok > 0) as min_output_per_mtok, | ||
| 32 | + (select jsonb_build_object('min', min(x.v), 'p25', percentile_cont(0.25) within group (order by x.v), 'median', percentile_cont(0.5) within group (order by x.v), | ||||
| 33 | + 'p75', percentile_cont(0.75) within group (order by x.v), 'max', max(x.v), 'n', count(*)) | ||||
| 34 | + from (select p.input_per_mtok as v from prices p where p.provider_id = e.id and p.valid_to is null and p.input_per_mtok > 0) x) as input_dist, | ||||
| 35 | + (select jsonb_build_object('min', min(x.v), 'p25', percentile_cont(0.25) within group (order by x.v), 'median', percentile_cont(0.5) within group (order by x.v), | ||||
| 36 | + 'p75', percentile_cont(0.75) within group (order by x.v), 'max', max(x.v), 'n', count(*)) | ||||
| 37 | + from (select p.output_per_mtok as v from prices p where p.provider_id = e.id and p.valid_to is null and p.output_per_mtok > 0) x) as output_dist, | ||||
| 38 | + (select count(distinct p.model_id) from prices p where p.provider_id = e.id and p.valid_from > now() - interval '30 days' | ||||
| 39 | + and not exists (select 1 from prices q where q.provider_id = e.id and q.model_id = p.model_id and q.valid_from <= now() - interval '30 days')) as models_added_30d, | ||||
| 40 | + (select count(distinct p.model_id) from prices p where p.provider_id = e.id and p.valid_to > now() - interval '30 days' | ||||
| 41 | + and not exists (select 1 from prices q where q.provider_id = e.id and q.model_id = p.model_id and q.valid_to is null)) as models_removed_30d, | ||||
| 42 | + (select count(*) from change_events ev join prices p on p.model_id = ev.entity_id where ev.category = 'price' and ev.is_backfill = false | ||||
| 43 | + and ev.occurred_at > now() - interval '30 days' and p.provider_id = e.id and p.valid_to is null) as price_changes_30d, | ||||
| 44 | + (select count(distinct m.organization_id) from prices p join entities m on m.id = p.model_id where p.provider_id = e.id and p.valid_to is null and m.organization_id is not null) as organizations_covered, | ||||
| 45 | + (select coalesce(jsonb_agg(distinct k.key), '[]'::jsonb) from prices p, jsonb_object_keys(p.features) k(key) where p.provider_id = e.id and p.valid_to is null) as feature_keys, | ||||
| 46 | + (select count(*) from prices p where p.provider_id = e.id and p.valid_to is null and p.cached_input_per_mtok is not null) as with_cached, | ||||
| 47 | + (select count(*) from prices p where p.provider_id = e.id and p.valid_to is null and p.batch_input_per_mtok is not null) as with_batch | ||||
| 26 | from {ENTITY_FROM} where e.entity_type = 'provider' and e.merged_into is null | 48 | from {ENTITY_FROM} where e.entity_type = 'provider' and e.merged_into is null | ||
| 27 | order by price_count desc, listed_models desc, e.canonical_name""") | 49 | order by price_count desc, listed_models desc, e.canonical_name""") | ||
| 28 | − return {"items": [{**(entity_summary(r) or {}), "model_count": int(max(r["priced_models"] or 0, r["listed_models"] or 0)), "price_count": int(r["price_count"] or 0), | 50 | + items = [] | ||
| 29 | − "min_input_per_mtok": r["min_input_per_mtok"], "min_output_per_mtok": r["min_output_per_mtok"]} for r in rows]} | 51 | + for r in rows: | ||
| 52 | + keys = set(r["feature_keys"] or []) | ||||
| 53 | + supported = sorted(f for f, ks in FEATURE_KEYS.items() if keys & set(ks)) | ||||
| 54 | + if int(r["with_cached"] or 0) and "cached" not in supported: | ||||
| 55 | + supported.append("cached") | ||||
| 56 | + if int(r["with_batch"] or 0) and "batch" not in supported: | ||||
| 57 | + supported.append("batch") | ||||
| 58 | + items.append({**(entity_summary(r) or {}), "model_count": int(max(r["priced_models"] or 0, r["listed_models"] or 0)), "price_count": int(r["price_count"] or 0), | ||||
| 59 | + "min_input_per_mtok": r["min_input_per_mtok"], "min_output_per_mtok": r["min_output_per_mtok"], | ||||
| 60 | + "input_price_distribution": _dist(r["input_dist"]), "output_price_distribution": _dist(r["output_dist"]), | ||||
| 61 | + "models_added_30d": int(r["models_added_30d"] or 0), "models_removed_30d": int(r["models_removed_30d"] or 0), "price_changes_30d": int(r["price_changes_30d"] or 0), | ||||
| 62 | + "organizations_covered": int(r["organizations_covered"] or 0), "features_supported": sorted(set(supported)), "feature_keys": sorted(keys)}) | ||||
| 63 | + return {"items": items, "note": "Distributions are over live offers with a positive price (USD per 1M tokens). models_added/removed_30d count model listings first opened / last closed " | ||||
| 64 | + "in the window; price_changes_30d counts price events (not back-filled) on models the provider currently lists."} | ||||
| 65 | + | ||||
| 66 | + | ||||
| 67 | +def _dist(v: Any) -> dict[str, Any] | None: | ||||
| 68 | + if not isinstance(v, dict) or not v.get("n"): | ||||
| 69 | + return None | ||||
| 70 | + return {k: v.get(k) for k in ("min", "p25", "median", "p75", "max", "n")} | ||||
| 30 | 71 | ||||
| 31 | 72 | ||||
| 32 | @router.get("/{slug}") | 73 | @router.get("/{slug}") | ||
modified
src/aiatlas/api/routers/search.py
+53 −46
@@ -1,12 +1,12 @@ | |||||
| 1 | −"""/search · /search/suggest — natural-language compiler + FTS/trigram (+ embeddings when the gateway is reachable).""" | 1 | +"""/search · /search/suggest — natural-language compiler v2 + FTS/trigram (+ embeddings when the gateway is reachable).""" | ||
| 2 | from __future__ import annotations | 2 | from __future__ import annotations | ||
| 3 | 3 | ||||
| 4 | import asyncio | 4 | import asyncio | ||
| 5 | −import re | ||||
| 6 | import time | 5 | import time | ||
| 7 | from typing import Any | 6 | from typing import Any | ||
| 8 | 7 | ||||
| 9 | from fastapi import APIRouter, Depends, Query, Request | 8 | from fastapi import APIRouter, Depends, Query, Request | ||
| 9 | +from sqlalchemy.exc import DBAPIError | ||||
| 10 | 10 | ||||
| 11 | from aiatlas.api.common import ENTITY_COLS, ENTITY_FROM, ApiError, cached, entity_summary, rate_limit | 11 | from aiatlas.api.common import ENTITY_COLS, ENTITY_FROM, ApiError, cached, entity_summary, rate_limit | ||
| 12 | from aiatlas.db import connection, fetch_all, fetch_val | 12 | from aiatlas.db import connection, fetch_all, fetch_val | ||
@@ -14,7 +14,7 @@ from aiatlas.services import cache | |||||
| 14 | from aiatlas.services.embeddings import embed_query | 14 | from aiatlas.services.embeddings import embed_query | ||
| 15 | from aiatlas.services.llm import gateway | 15 | from aiatlas.services.llm import gateway | ||
| 16 | from aiatlas.services.search import Query as SearchQuery | 16 | from aiatlas.services.search import Query as SearchQuery | ||
| 17 | −from aiatlas.services.search import compile_query, search_entities, suggest | 17 | +from aiatlas.services.search import compile_query, search_entities, suggest, verify_organization, where_for | ||
| 18 | 18 | ||||
| 19 | router = APIRouter(prefix="/api/v1/search", tags=["search"]) | 19 | router = APIRouter(prefix="/api/v1/search", tags=["search"]) | ||
| 20 | TOTAL_CAP = 10_000 | 20 | TOTAL_CAP = 10_000 | ||
@@ -24,38 +24,15 @@ EMBED_DEGRADED_KEY = "search:embed-degraded" | |||||
| 24 | 24 | ||||
| 25 | 25 | ||||
| 26 | def _count_sql(q: SearchQuery) -> tuple[str, dict[str, Any]]: | 26 | def _count_sql(q: SearchQuery) -> tuple[str, dict[str, Any]]: | ||
| 27 | − """Same filters as `search_entities`, without ranking — capped estimate.""" | 27 | + """Same filters as `search_entities`, without ranking — capped estimate. All casts are regex-guarded in `where_for`.""" | ||
| 28 | − where = ["e.merged_into is null"] | 28 | + import re | ||
| 29 | − params: dict[str, Any] = {} | 29 | + | ||
| 30 | − if q.entity_type: | 30 | + where, params = where_for(q) | ||
| 31 | − where.append("e.entity_type = :etype") | 31 | + text = (q.residual or "").strip() or ("" if q.has_structure else q.text) | ||
| 32 | − params["etype"] = q.entity_type | 32 | + if q.benchmark and q.entity_type == "benchmark": | ||
| 33 | − if q.openness == "open": | 33 | + text = q.benchmark | ||
| 34 | − where.append("(e.attributes->>'openness' in ('open-weights','open-source','open') or e.attributes->>'weights_availability' = 'open')") | 34 | + if q.provider and q.entity_type == "provider": | ||
| 35 | − elif q.openness == "proprietary": | 35 | + text = q.provider | ||
| 36 | − where.append("e.attributes->>'openness' in ('proprietary','closed')") | ||||
| 37 | − if q.year_from: | ||||
| 38 | − where.append("left(e.attributes->>'release_date', 4) >= :yf") | ||||
| 39 | − params["yf"] = str(q.year_from) | ||||
| 40 | − if q.year_to: | ||||
| 41 | − where.append("left(e.attributes->>'release_date', 4) <= :yt") | ||||
| 42 | − params["yt"] = str(q.year_to) | ||||
| 43 | − if q.params_min: | ||||
| 44 | − where.append("(e.attributes->>'parameter_count')::double precision >= :pmin") | ||||
| 45 | − params["pmin"] = float(q.params_min) | ||||
| 46 | − if q.params_max: | ||||
| 47 | − where.append("(e.attributes->>'parameter_count')::double precision <= :pmax") | ||||
| 48 | − params["pmax"] = float(q.params_max) | ||||
| 49 | − if q.context_min: | ||||
| 50 | − where.append("(e.attributes->>'context_length')::double precision >= :cmin") | ||||
| 51 | − params["cmin"] = float(q.context_min) | ||||
| 52 | − for i, mod in enumerate(q.modalities): | ||||
| 53 | − where.append(f"e.attributes->'modalities' ? :mod{i}") | ||||
| 54 | − params[f"mod{i}"] = mod | ||||
| 55 | − if q.organization: | ||||
| 56 | − where.append("exists (select 1 from entities o where o.id = e.organization_id and o.canonical_name ilike :org)") | ||||
| 57 | − params["org"] = f"%{q.organization}%" | ||||
| 58 | − text = (q.filters.get("residual") or "").strip() or ("" if any([q.entity_type, q.openness, q.year_from, q.params_min, q.context_min, q.modalities]) else q.text) | ||||
| 59 | if text: | 36 | if text: | ||
| 60 | params["q"] = text | 37 | params["q"] = text | ||
| 61 | params["qlike"] = f"%{text}%" | 38 | params["qlike"] = f"%{text}%" | ||
@@ -74,17 +51,44 @@ async def search(request: Request, q: str = Query("", max_length=300), type: str | |||||
| 74 | compiled.entity_type = type | 51 | compiled.entity_type = type | ||
| 75 | if not q.strip() and not type: | 52 | if not q.strip() and not type: | ||
| 76 | raise ApiError(400, "q is required") | 53 | raise ApiError(400, "q is required") | ||
| 77 | − embedding = None | ||||
| 78 | − residual = (compiled.filters.get("residual") or "").strip() | ||||
| 79 | − if gateway.available and residual and not await cache.cache_get(EMBED_DEGRADED_KEY): | ||||
| 80 | − try: | ||||
| 81 | − embedding = await asyncio.wait_for(embed_query(residual), timeout=EMBED_TIMEOUT_S) | ||||
| 82 | − except Exception: # noqa: BLE001 — never wait on the LLM | ||||
| 83 | − embedding = None | ||||
| 84 | − if embedding is None: # back off: FTS-only for a while instead of paying the timeout on every query | ||||
| 85 | − await cache.cache_set(EMBED_DEGRADED_KEY, {"since": time.time()}, EMBED_BACKOFF_S) | ||||
| 86 | async with connection() as conn: | 54 | async with connection() as conn: | ||
| 87 | − hits = await search_entities(conn, compiled, limit=limit, offset=offset, embedding=embedding) | 55 | + org_ok = None | ||
| 56 | + if compiled.organization: | ||||
| 57 | + org_ok = await verify_organization(conn, compiled.organization) | ||||
| 58 | + if org_ok is None: | ||||
| 59 | + # not an organization we know: demote to free text and say so | ||||
| 60 | + name = compiled.organization | ||||
| 61 | + compiled.organization = None | ||||
| 62 | + compiled.compiled = [c for c in compiled.compiled if c["filter"] != "organization"] | ||||
| 63 | + compiled.residual = " ".join(x for x in [compiled.residual, name] if x) | ||||
| 64 | + compiled.filters["residual"] = compiled.residual | ||||
| 65 | + compiled.unrecognised = sorted(set(compiled.unrecognised) | {name}) | ||||
| 66 | + else: | ||||
| 67 | + compiled.organization = org_ok["canonical_name"] | ||||
| 68 | + for c in compiled.compiled: | ||||
| 69 | + if c["filter"] == "organization": | ||||
| 70 | + c["value"] = {"id": org_ok["id"], "slug": org_ok["slug"], "name": org_ok["canonical_name"]} | ||||
| 71 | + c["label"] = f"Organization: {org_ok['canonical_name']}" | ||||
| 72 | + embedding = None | ||||
| 73 | + residual = (compiled.residual or "").strip() | ||||
| 74 | + if gateway.available and residual and not await cache.cache_get(EMBED_DEGRADED_KEY): | ||||
| 75 | + try: | ||||
| 76 | + embedding = await asyncio.wait_for(embed_query(residual), timeout=EMBED_TIMEOUT_S) | ||||
| 77 | + except Exception: # noqa: BLE001 — never wait on the LLM | ||||
| 78 | + embedding = None | ||||
| 79 | + if embedding is None: # back off: FTS-only for a while instead of paying the timeout on every query | ||||
| 80 | + await cache.cache_set(EMBED_DEGRADED_KEY, {"since": time.time()}, EMBED_BACKOFF_S) | ||||
| 81 | + if embedding is not None: | ||||
| 82 | + # the vector join needs `entity_embeddings` (pgvector); when the table is missing the statement aborts the connection's | ||||
| 83 | + # transaction, so the FTS-only fallback runs on a fresh pooled connection and embeddings are backed off for a while | ||||
| 84 | + try: | ||||
| 85 | + async with connection() as vconn: | ||||
| 86 | + hits = await search_entities(vconn, compiled, limit=limit, offset=offset, embedding=embedding) | ||||
| 87 | + except DBAPIError: | ||||
| 88 | + embedding = None | ||||
| 89 | + await cache.cache_set(EMBED_DEGRADED_KEY, {"since": time.time(), "reason": "entity_embeddings unavailable"}, EMBED_BACKOFF_S) | ||||
| 90 | + if embedding is None: | ||||
| 91 | + hits = await search_entities(conn, compiled, limit=limit, offset=offset, embedding=None) | ||||
| 88 | ids = [h["id"] for h in hits] | 92 | ids = [h["id"] for h in hits] | ||
| 89 | full = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.id = any(cast(:ids as text[]))", ids=ids) if ids else [] | 93 | full = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where e.id = any(cast(:ids as text[]))", ids=ids) if ids else [] | ||
| 90 | count_sql, count_params = _count_sql(compiled) | 94 | count_sql, count_params = _count_sql(compiled) | ||
@@ -97,7 +101,10 @@ async def search(request: Request, q: str = Query("", max_length=300), type: str | |||||
| 97 | if s: | 101 | if s: | ||
| 98 | s["rank"] = float(h["rank"] or 0) | 102 | s["rank"] = float(h["rank"] or 0) | ||
| 99 | items.append(s) | 103 | items.append(s) | ||
| 100 | − return {"query": {**compiled.as_dict(), "semantic": embedding is not None}, "items": items, "total": int(total or 0), "limit": limit, "offset": offset} | 104 | + query = {**compiled.as_dict(), "semantic": embedding is not None, "version": 2} | ||
| 105 | + if compiled.memory_gb is not None: | ||||
| 106 | + query["note"] = "memory_gb is converted to an ESTIMATED parameter bound (4-bit weights, 8K context, 2 GB reserved) — see /methodology hardware_fit" | ||||
| 107 | + return {"query": query, "items": items, "total": int(total or 0), "limit": limit, "offset": offset} | ||||
| 101 | 108 | ||||
| 102 | 109 | ||||
| 103 | @router.get("/suggest", dependencies=[Depends(rate_limit("search"))]) | 110 | @router.get("/suggest", dependencies=[Depends(rate_limit("search"))]) | ||
modified
src/aiatlas/api/routers/stats.py
+1 −1
@@ -20,7 +20,7 @@ router = APIRouter(prefix="/api/v1/stats", tags=["stats"]) | |||||
| 20 | async def stats(request: Request) -> dict[str, Any]: | 20 | async def stats(request: Request) -> dict[str, Any]: | ||
| 21 | async with connection() as conn: | 21 | async with connection() as conn: | ||
| 22 | counts = await live_counts(conn) | 22 | counts = await live_counts(conn) | ||
| 23 | − counts["entities"] = {**{t: 0 for t in MAIN_ENTITY_TYPES}, **counts["entities"]} # zero-filled: the homepage never renders a missing key | 23 | + counts["entities"] = {**{t: 0 for t in (*MAIN_ENTITY_TYPES, "artifact", "model_family")}, **counts["entities"]} # zero-filled: the homepage never renders a missing key | ||
| 24 | counts["archive"] = await asyncio.to_thread(archive_size) | 24 | counts["archive"] = await asyncio.to_thread(archive_size) | ||
| 25 | counts["computed_at"] = datetime.now(UTC) | 25 | counts["computed_at"] = datetime.now(UTC) | ||
| 26 | return counts | 26 | return counts | ||
modified
src/aiatlas/api/routers/timeline.py
+15 −7
@@ -1,4 +1,4 @@ | |||||
| 1 | −"""/timeline — events grouped by month (global, or one entity + what it develops).""" | 1 | +"""/timeline — events grouped by month (global, or one entity + what it develops). API 1.1: `occurred_at`, `is_backfill = false` by default.""" | ||
| 2 | from __future__ import annotations | 2 | from __future__ import annotations | ||
| 3 | 3 | ||||
| 4 | from typing import Any | 4 | from typing import Any | ||
@@ -14,9 +14,13 @@ router = APIRouter(prefix="/api/v1/timeline", tags=["timeline"]) | |||||
| 14 | @router.get("") | 14 | @router.get("") | ||
| 15 | @cached(120) | 15 | @cached(120) | ||
| 16 | async def timeline(request: Request, entity: str | None = None, year: int | None = Query(None, ge=1950, le=2100), category: str | None = None, | 16 | async def timeline(request: Request, entity: str | None = None, year: int | None = Query(None, ge=1950, le=2100), category: str | None = None, | ||
| 17 | − importance_min: int | None = Query(None, ge=0, le=3), limit: int = Query(200, ge=1, le=1000)) -> dict[str, Any]: | 17 | + importance_min: int | None = Query(None, ge=0, le=3), limit: int = Query(200, ge=1, le=1000), include_backfill: int = Query(0, ge=0, le=1), | ||
| 18 | + date_field: str = Query("occurred", pattern="^(occurred|observed)$")) -> dict[str, Any]: | ||||
| 19 | + col = "ev.observed_at" if date_field == "observed" else "ev.occurred_at" | ||||
| 18 | where = ["ev.event_type <> 'DOCUMENT_CHANGED'"] | 20 | where = ["ev.event_type <> 'DOCUMENT_CHANGED'"] | ||
| 19 | params: dict[str, Any] = {"lim": limit} | 21 | params: dict[str, Any] = {"lim": limit} | ||
| 22 | + if not include_backfill: | ||||
| 23 | + where.append("ev.is_backfill = false") | ||||
| 20 | async with connection() as conn: | 24 | async with connection() as conn: | ||
| 21 | if entity: | 25 | if entity: | ||
| 22 | row = await resolve_entity(conn, entity) | 26 | row = await resolve_entity(conn, entity) | ||
@@ -27,7 +31,7 @@ async def timeline(request: Request, entity: str | None = None, year: int | None | |||||
| 27 | where.append("ev.entity_id = :eid") | 31 | where.append("ev.entity_id = :eid") | ||
| 28 | params["eid"] = row["id"] | 32 | params["eid"] = row["id"] | ||
| 29 | if year: | 33 | if year: | ||
| 30 | − where.append("extract(year from coalesce(ev.effective_at, ev.observed_at)) = :year") | 34 | + where.append(f"extract(year from {col}) = :year") | ||
| 31 | params["year"] = year | 35 | params["year"] = year | ||
| 32 | if category: | 36 | if category: | ||
| 33 | where.append("ev.category = any(cast(:cats as text[]))") | 37 | where.append("ev.category = any(cast(:cats as text[]))") | ||
@@ -35,9 +39,13 @@ async def timeline(request: Request, entity: str | None = None, year: int | None | |||||
| 35 | if importance_min is not None: | 39 | if importance_min is not None: | ||
| 36 | where.append("ev.importance >= :imp") | 40 | where.append("ev.importance >= :imp") | ||
| 37 | params["imp"] = importance_min | 41 | params["imp"] = importance_min | ||
| 38 | − rows = await fetch_all(conn, f"select to_char(coalesce(ev.effective_at, ev.observed_at) at time zone 'UTC', 'YYYY-MM') as month, {EVENT_COLS} from {EVENT_FROM} " | 42 | + rows = await fetch_all(conn, f"select to_char({col} at time zone 'UTC', 'YYYY-MM') as month, {EVENT_COLS}, ev.occurred_at, ev.is_backfill, ev.group_key from {EVENT_FROM} " | ||
| 39 | − f"where {' and '.join(where)} order by coalesce(ev.effective_at, ev.observed_at) desc, ev.id desc limit :lim", **params) | 43 | + f"where {' and '.join(where)} order by {col} desc, ev.id desc limit :lim", **params) | ||
| 40 | groups: dict[str, list[dict[str, Any]]] = {} | 44 | groups: dict[str, list[dict[str, Any]]] = {} | ||
| 41 | for r in rows: | 45 | for r in rows: | ||
| 42 | − groups.setdefault(r["month"], []).append(change_event(r)) | 46 | + ev = change_event(r) | ||
| 43 | − return {"items": [{"month": m, "count": len(evs), "events": evs} for m, evs in groups.items()], "total": len(rows)} | 47 | + ev["occurred_at"] = r.get("occurred_at") | ||
| 48 | + ev["is_backfill"] = r.get("is_backfill") | ||||
| 49 | + groups.setdefault(r["month"], []).append(ev) | ||||
| 50 | + return {"items": [{"month": m, "count": len(evs), "events": evs} for m, evs in groups.items()], "total": len(rows), "date_field": date_field, | ||||
| 51 | + "include_backfill": bool(include_backfill)} | ||||
added
src/aiatlas/services/cost.py
+75 −0
@@ -0,0 +1,75 @@ | |||||
| 1 | +"""Cost arithmetic for API deployments (pure, deterministic). Prices are USD per 1M tokens; nothing is assumed when a price is missing — | ||||
| 2 | +the affected component is `null` and a `note` says why. | ||||
| 3 | + | ||||
| 4 | + per_request = input_tokens × effective_input / 1e6 + output_tokens × effective_output / 1e6 (+ per_request fee when published) | ||||
| 5 | + effective_input = (1 − cached_share) × input + cached_share × cached_input (cached_input falls back to input with a note) | ||||
| 6 | + batch = batch_input / batch_output when published (else the standard prices, with a note) | ||||
| 7 | + daily = per_request × requests_per_day · monthly = daily × 30 · annual = daily × 365 | ||||
| 8 | +""" | ||||
| 9 | +from __future__ import annotations | ||||
| 10 | + | ||||
| 11 | +from typing import Any | ||||
| 12 | + | ||||
| 13 | +DAYS_PER_MONTH = 30 | ||||
| 14 | +DAYS_PER_YEAR = 365 | ||||
| 15 | + | ||||
| 16 | + | ||||
| 17 | +def _f(v: Any) -> float | None: | ||||
| 18 | + if v is None or isinstance(v, bool): | ||||
| 19 | + return None | ||||
| 20 | + try: | ||||
| 21 | + return float(v) | ||||
| 22 | + except (TypeError, ValueError): | ||||
| 23 | + return None | ||||
| 24 | + | ||||
| 25 | + | ||||
| 26 | +def compute_cost(prices: dict[str, Any], *, input_tokens: int, output_tokens: int, requests_per_day: float = 1.0, cached_share: float = 0.0, | ||||
| 27 | + batch: bool = False) -> dict[str, Any]: | ||||
| 28 | + """`prices` uses the Deployment `prices` keys (input, output, cached_input, batch_input, batch_output, per_request).""" | ||||
| 29 | + notes: list[str] = [] | ||||
| 30 | + cached_share = min(1.0, max(0.0, float(cached_share or 0.0))) | ||||
| 31 | + inp, outp = _f(prices.get("input")), _f(prices.get("output")) | ||||
| 32 | + if batch: | ||||
| 33 | + bi, bo = _f(prices.get("batch_input")), _f(prices.get("batch_output")) | ||||
| 34 | + if bi is None or bo is None: | ||||
| 35 | + notes.append("batch prices not published for this deployment; standard prices used") | ||||
| 36 | + inp = bi if bi is not None else inp | ||||
| 37 | + outp = bo if bo is not None else outp | ||||
| 38 | + eff_input = inp | ||||
| 39 | + if cached_share > 0 and inp is not None: | ||||
| 40 | + cached = _f(prices.get("cached_input")) | ||||
| 41 | + if cached is None: | ||||
| 42 | + notes.append("cached input price not published; cached share billed at the standard input price") | ||||
| 43 | + cached = inp | ||||
| 44 | + eff_input = (1 - cached_share) * inp + cached_share * cached | ||||
| 45 | + fee = _f(prices.get("per_request")) or 0.0 | ||||
| 46 | + if inp is None: | ||||
| 47 | + notes.append("input price unavailable") | ||||
| 48 | + if outp is None: | ||||
| 49 | + notes.append("output price unavailable") | ||||
| 50 | + if eff_input is None or outp is None: | ||||
| 51 | + per_request = None | ||||
| 52 | + else: | ||||
| 53 | + per_request = input_tokens * eff_input / 1e6 + output_tokens * outp / 1e6 + fee | ||||
| 54 | + daily = per_request * requests_per_day if per_request is not None else None | ||||
| 55 | + return { | ||||
| 56 | + "per_request": _round(per_request, 8), "daily": _round(daily, 6), | ||||
| 57 | + "monthly": _round(daily * DAYS_PER_MONTH, 4) if daily is not None else None, | ||||
| 58 | + "annual": _round(daily * DAYS_PER_YEAR, 4) if daily is not None else None, | ||||
| 59 | + "effective_input_per_mtok": _round(eff_input, 6), "effective_output_per_mtok": _round(outp, 6), "per_request_fee": fee or None, | ||||
| 60 | + "inputs": {"input_tokens": input_tokens, "output_tokens": output_tokens, "requests_per_day": requests_per_day, "cached_share": cached_share, "batch": batch}, | ||||
| 61 | + "notes": notes, | ||||
| 62 | + } | ||||
| 63 | + | ||||
| 64 | + | ||||
| 65 | +def context_fill_cost(input_per_mtok: Any, tokens: int) -> float | None: | ||||
| 66 | + """Cost of one fully populated context of `tokens` input tokens.""" | ||||
| 67 | + p = _f(input_per_mtok) | ||||
| 68 | + return _round(p * tokens / 1e6, 6) if p is not None else None | ||||
| 69 | + | ||||
| 70 | + | ||||
| 71 | +def _round(v: float | None, nd: int) -> float | None: | ||||
| 72 | + return round(v, nd) if v is not None else None | ||||
| 73 | + | ||||
| 74 | + | ||||
| 75 | +__all__ = ["DAYS_PER_MONTH", "DAYS_PER_YEAR", "compute_cost", "context_fill_cost"] | ||||
added
src/aiatlas/services/finder.py
+255 −0
@@ -0,0 +1,255 @@ | |||||
| 1 | +"""Find-a-model (API 1.1): deterministic matching of canonical models against use-case criteria. No LLM, no composite "winner" score — | ||||
| 2 | +each match lists the observed facts (`why`) that satisfied each criterion and the observed dimensions; sorting is by number of satisfied | ||||
| 3 | +criteria, then by the best benchmark rank the model holds anywhere, then by release date. | ||||
| 4 | + | ||||
| 5 | +Rules (documented in `RULES`, returned to the client as `filters_applied`): | ||||
| 6 | + coding current result on a coding-category benchmark, or capabilities/modalities mention code | ||||
| 7 | + reasoning attributes.reasoning is true, or a current result on a reasoning/math benchmark | ||||
| 8 | + agentic current result on an agentic benchmark, or tool_calling is true | ||||
| 9 | + long_context context_length ≥ context_min (default 128 000) | ||||
| 10 | + vision modalities (any direction) include image | ||||
| 11 | + low_cost cheapest current output price ≤ max_output_price, or in the bottom quartile of all cheapest output prices | ||||
| 12 | + local weights downloadable (open-weights / open-source / restricted-weights) and the estimated footprint fits memory_gb | ||||
| 13 | + embeddings modalities include embedding (or pipeline tag says so) | ||||
| 14 | + chat text modality (or unspecified) and reachable through ≥ 1 current deployment or downloadable weights | ||||
| 15 | +""" | ||||
| 16 | +from __future__ import annotations | ||||
| 17 | + | ||||
| 18 | +from collections import defaultdict | ||||
| 19 | +from typing import Any | ||||
| 20 | + | ||||
| 21 | +from sqlalchemy.ext.asyncio import AsyncConnection | ||||
| 22 | + | ||||
| 23 | +from aiatlas.db import fetch_all | ||||
| 24 | +from aiatlas.ontology.licenses import LICENSES, normalize_license | ||||
| 25 | +from aiatlas.services import hardware_fit as hf | ||||
| 26 | +from aiatlas.services.frontier import all_primary_groups, rank_rows | ||||
| 27 | + | ||||
| 28 | +USE_CASES = ("coding", "reasoning", "agentic", "long_context", "vision", "low_cost", "local", "embeddings", "chat") | ||||
| 29 | +DOWNLOADABLE = ("open-weights", "open-source", "restricted-weights", "restricted") | ||||
| 30 | +CODING_CATEGORIES = ("coding",) | ||||
| 31 | +REASONING_CATEGORIES = ("reasoning", "math") | ||||
| 32 | +AGENTIC_CATEGORIES = ("agentic",) | ||||
| 33 | +RULES = { | ||||
| 34 | + "coding": "current result on a coding-category benchmark, or capabilities/modalities mention code", | ||||
| 35 | + "reasoning": "attributes.reasoning = true, or a current result on a reasoning/math benchmark", | ||||
| 36 | + "agentic": "current result on an agentic benchmark, or tool_calling = true", | ||||
| 37 | + "long_context": "context_length ≥ context_min (default 128 000 tokens)", | ||||
| 38 | + "vision": "modalities include image", | ||||
| 39 | + "low_cost": "cheapest current output price ≤ max_output_price, or in the bottom quartile of all models' cheapest output prices", | ||||
| 40 | + "local": "weights downloadable and the ESTIMATED footprint (services.hardware_fit) fits memory_gb at the requested quantisation", | ||||
| 41 | + "embeddings": "modalities include embedding", | ||||
| 42 | + "chat": "text modality and reachable (≥ 1 current deployment or downloadable weights)", | ||||
| 43 | + "deployment": "local → downloadable weights · api → ≥ 1 current deployment · any → no constraint", | ||||
| 44 | + "license": "commercial → licence key known in the ontology with commercial_use = true", | ||||
| 45 | + "openness": "attributes.openness in the requested categories", | ||||
| 46 | + "max_input_price / max_output_price": "cheapest current provider price (USD per 1M tokens) ≤ the bound", | ||||
| 47 | + "modalities": "every requested modality present in modalities / modalities_input / modalities_output", | ||||
| 48 | +} | ||||
| 49 | + | ||||
| 50 | + | ||||
| 51 | +def _modalities(attrs: dict[str, Any]) -> set[str]: | ||||
| 52 | + out: set[str] = set() | ||||
| 53 | + for k in ("modalities", "modalities_input", "modalities_output"): | ||||
| 54 | + v = attrs.get(k) | ||||
| 55 | + if isinstance(v, list): | ||||
| 56 | + out |= {str(x).strip().lower() for x in v} | ||||
| 57 | + elif isinstance(v, str): | ||||
| 58 | + out |= {x.strip().lower() for x in v.split(",") if x.strip()} | ||||
| 59 | + if attrs.get("vision") is True: | ||||
| 60 | + out.add("image") | ||||
| 61 | + if attrs.get("audio") is True: | ||||
| 62 | + out.add("audio") | ||||
| 63 | + return {("image" if m in ("vision", "images") else "document" if m == "pdf" else m) for m in out} | ||||
| 64 | + | ||||
| 65 | + | ||||
| 66 | +def _num(v: Any) -> float | None: | ||||
| 67 | + if isinstance(v, bool) or v is None: | ||||
| 68 | + return None | ||||
| 69 | + try: | ||||
| 70 | + return float(v) | ||||
| 71 | + except (TypeError, ValueError): | ||||
| 72 | + return None | ||||
| 73 | + | ||||
| 74 | + | ||||
| 75 | +def _mentions_code(attrs: dict[str, Any]) -> bool: | ||||
| 76 | + caps = attrs.get("capabilities") | ||||
| 77 | + text = " ".join(str(c) for c in caps).lower() if isinstance(caps, list) else str(caps or "").lower() | ||||
| 78 | + return "code" in text or "code" in _modalities(attrs) or str(attrs.get("pipeline_tag") or "").lower().startswith("text-generation") and "coder" in str(attrs.get("family") or "").lower() | ||||
| 79 | + | ||||
| 80 | + | ||||
| 81 | +async def find_models(conn: AsyncConnection, *, use_case: str | None, deployment: str = "any", memory_gb: float | None = None, quant: str = "4bit", | ||||
| 82 | + context_min: int | None = None, license: str = "any", openness: list[str] | None = None, max_input_price: float | None = None, | ||||
| 83 | + max_output_price: float | None = None, modalities: list[str] | None = None, limit: int = 30) -> dict[str, Any]: | ||||
| 84 | + models = await fetch_all(conn, """ | ||||
| 85 | + select e.id, e.slug, e.canonical_name, e.attributes, e.organization_id, o.slug as org_slug, o.canonical_name as org_name, e.first_seen_at | ||||
| 86 | + from entities e left join entities o on o.id = e.organization_id where e.entity_type = 'model' and e.merged_into is null""") | ||||
| 87 | + prices = await fetch_all(conn, """ | ||||
| 88 | + select p.model_id, min(p.input_per_mtok) filter (where p.input_per_mtok > 0) as min_input, min(p.output_per_mtok) filter (where p.output_per_mtok > 0) as min_output, | ||||
| 89 | + count(distinct p.provider_id) as providers | ||||
| 90 | + from prices p where p.valid_to is null group by 1""") | ||||
| 91 | + price_by = {p["model_id"]: p for p in prices} | ||||
| 92 | + groups = await all_primary_groups(conn) | ||||
| 93 | + ranks: dict[str, dict[str, int]] = defaultdict(dict) # model_id → {benchmark slug: rank} | ||||
| 94 | + categories: dict[str, set[str]] = defaultdict(set) # model_id → benchmark categories with a current result | ||||
| 95 | + for g in groups.values(): | ||||
| 96 | + cat = (g["benchmark"].get("category") or "").lower() | ||||
| 97 | + for r in rank_rows(g["rows"], g["higher_is_better"]): | ||||
| 98 | + ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"] | ||||
| 99 | + if cat: | ||||
| 100 | + categories[r["model_id"]].add(cat) | ||||
| 101 | + outputs = sorted(float(p["min_output"]) for p in prices if p["min_output"] is not None) | ||||
| 102 | + bottom_quartile = outputs[len(outputs) // 4] if outputs else None | ||||
| 103 | + | ||||
| 104 | + ctx_min = context_min or 128_000 | ||||
| 105 | + quant = hf.normalize_quant(quant) | ||||
| 106 | + want_mods = {m.strip().lower() for m in (modalities or []) if m.strip()} | ||||
| 107 | + open_set = set(openness or []) | ||||
| 108 | + matches: list[dict[str, Any]] = [] | ||||
| 109 | + for m in models: | ||||
| 110 | + attrs = m["attributes"] or {} | ||||
| 111 | + why: list[str] = [] | ||||
| 112 | + failed = False | ||||
| 113 | + mods = _modalities(attrs) | ||||
| 114 | + pr = price_by.get(m["id"]) | ||||
| 115 | + cheapest_out = _num(pr["min_output"]) if pr else None | ||||
| 116 | + cheapest_in = _num(pr["min_input"]) if pr else None | ||||
| 117 | + providers = int(pr["providers"]) if pr else 0 | ||||
| 118 | + opn = str(attrs.get("openness") or "") | ||||
| 119 | + downloadable = opn in DOWNLOADABLE | ||||
| 120 | + ctx = _num(attrs.get("context_length")) | ||||
| 121 | + cats = categories.get(m["id"], set()) | ||||
| 122 | + estimated_fit: dict[str, Any] | None = None | ||||
| 123 | + | ||||
| 124 | + # --- hard filters | ||||
| 125 | + if deployment == "local" and not downloadable: | ||||
| 126 | + continue | ||||
| 127 | + if deployment == "api" and providers == 0: | ||||
| 128 | + continue | ||||
| 129 | + if open_set and opn not in open_set: | ||||
| 130 | + continue | ||||
| 131 | + if license == "commercial": | ||||
| 132 | + key = attrs.get("license_key") or normalize_license(attrs.get("license")) | ||||
| 133 | + info = LICENSES.get(key) if key else None | ||||
| 134 | + if not info or info.commercial_use is not True: | ||||
| 135 | + continue | ||||
| 136 | + why.append(f"licence {info.label} allows commercial use") | ||||
| 137 | + if max_input_price is not None: | ||||
| 138 | + if cheapest_in is None or cheapest_in > max_input_price: | ||||
| 139 | + continue | ||||
| 140 | + why.append(f"cheapest input ${cheapest_in:g}/M ≤ ${max_input_price:g}") | ||||
| 141 | + if max_output_price is not None and use_case != "low_cost": | ||||
| 142 | + if cheapest_out is None or cheapest_out > max_output_price: | ||||
| 143 | + continue | ||||
| 144 | + why.append(f"cheapest output ${cheapest_out:g}/M ≤ ${max_output_price:g}") | ||||
| 145 | + if want_mods and not want_mods <= mods: | ||||
| 146 | + continue | ||||
| 147 | + if want_mods: | ||||
| 148 | + why.append("modalities include " + ", ".join(sorted(want_mods))) | ||||
| 149 | + if memory_gb is not None and (use_case == "local" or deployment == "local"): | ||||
| 150 | + estimated_fit = hf.fit_detailed(attrs, memory_gb, quant=quant, context=min(int(ctx or 8192), 8192)) | ||||
| 151 | + if estimated_fit is None or not estimated_fit["fits"]: | ||||
| 152 | + continue | ||||
| 153 | + why.append(f"estimated {estimated_fit['estimated_memory_gb']} GB at {quant} fits {memory_gb:g} GB (estimate)") | ||||
| 154 | + | ||||
| 155 | + # --- use-case rule (one criterion; `why` records the observed fact) | ||||
| 156 | + if use_case == "coding": | ||||
| 157 | + if cats & set(CODING_CATEGORIES): | ||||
| 158 | + why.append("current result on a coding benchmark: " + ", ".join(sorted(b for b in ranks[m["id"]] if groups_cat(groups, b) == "coding")[:4])) | ||||
| 159 | + elif _mentions_code(attrs): | ||||
| 160 | + why.append("capabilities/modalities mention code") | ||||
| 161 | + else: | ||||
| 162 | + failed = True | ||||
| 163 | + elif use_case == "reasoning": | ||||
| 164 | + if attrs.get("reasoning") is True: | ||||
| 165 | + why.append("attributes.reasoning = true") | ||||
| 166 | + elif cats & set(REASONING_CATEGORIES): | ||||
| 167 | + why.append("current result on a reasoning/math benchmark") | ||||
| 168 | + else: | ||||
| 169 | + failed = True | ||||
| 170 | + elif use_case == "agentic": | ||||
| 171 | + if cats & set(AGENTIC_CATEGORIES): | ||||
| 172 | + why.append("current result on an agentic benchmark") | ||||
| 173 | + elif attrs.get("tool_calling") is True: | ||||
| 174 | + why.append("tool_calling = true") | ||||
| 175 | + else: | ||||
| 176 | + failed = True | ||||
| 177 | + elif use_case == "long_context": | ||||
| 178 | + if ctx is not None and ctx >= ctx_min: | ||||
| 179 | + why.append(f"context_length {int(ctx):,} ≥ {ctx_min:,}") | ||||
| 180 | + else: | ||||
| 181 | + failed = True | ||||
| 182 | + elif use_case == "vision": | ||||
| 183 | + if "image" in mods: | ||||
| 184 | + why.append("modalities include image") | ||||
| 185 | + else: | ||||
| 186 | + failed = True | ||||
| 187 | + elif use_case == "low_cost": | ||||
| 188 | + bound = max_output_price if max_output_price is not None else bottom_quartile | ||||
| 189 | + if cheapest_out is not None and bound is not None and cheapest_out <= bound: | ||||
| 190 | + why.append(f"cheapest output ${cheapest_out:g}/M ≤ ${bound:g}/M " + ("(bound)" if max_output_price is not None else "(bottom quartile of observed prices)")) | ||||
| 191 | + else: | ||||
| 192 | + failed = True | ||||
| 193 | + elif use_case == "local": | ||||
| 194 | + if not downloadable: | ||||
| 195 | + failed = True | ||||
| 196 | + else: | ||||
| 197 | + why.append(f"openness {opn}: weights downloadable") | ||||
| 198 | + if memory_gb is None: | ||||
| 199 | + estimated_fit = hf.fit_detailed(attrs, 64.0, quant=quant, context=8192) | ||||
| 200 | + if estimated_fit is not None: | ||||
| 201 | + why.append(f"estimated {estimated_fit['estimated_memory_gb']} GB at {quant} (reference 64 GB device, estimate)") | ||||
| 202 | + elif use_case == "embeddings": | ||||
| 203 | + if "embedding" in mods or "embedding" in str(attrs.get("pipeline_tag") or "").lower(): | ||||
| 204 | + why.append("modalities include embedding") | ||||
| 205 | + else: | ||||
| 206 | + failed = True | ||||
| 207 | + elif use_case == "chat": | ||||
| 208 | + if (not mods or "text" in mods) and (providers > 0 or downloadable): | ||||
| 209 | + why.append("text modality and reachable (" + (f"{providers} providers" if providers else "downloadable weights") + ")") | ||||
| 210 | + else: | ||||
| 211 | + failed = True | ||||
| 212 | + if failed: | ||||
| 213 | + continue | ||||
| 214 | + if context_min is not None and use_case != "long_context": | ||||
| 215 | + if ctx is None or ctx < context_min: | ||||
| 216 | + continue | ||||
| 217 | + why.append(f"context_length {int(ctx):,} ≥ {context_min:,}") | ||||
| 218 | + best_rank = min(ranks[m["id"]].values()) if ranks.get(m["id"]) else None | ||||
| 219 | + matches.append({ | ||||
| 220 | + "model": {"id": m["id"], "slug": m["slug"], "name": m["canonical_name"], "entity_type": "model", | ||||
| 221 | + "organization": {"id": m["organization_id"], "slug": m["org_slug"], "name": m["org_name"]} if m["organization_id"] else None}, | ||||
| 222 | + "why": why, | ||||
| 223 | + "observed": {"context_length": ctx, "parameter_count": _num(attrs.get("parameter_count")), "openness": opn or None, "license": attrs.get("license"), | ||||
| 224 | + "license_key": attrs.get("license_key") or normalize_license(attrs.get("license")), "modalities": sorted(mods), "reasoning": attrs.get("reasoning"), | ||||
| 225 | + "tool_calling": attrs.get("tool_calling"), "release_date": attrs.get("release_date"), "cheapest_input_per_mtok": cheapest_in, | ||||
| 226 | + "cheapest_output_per_mtok": cheapest_out, "providers": providers, "benchmark_ranks": dict(sorted(ranks.get(m["id"], {}).items())), "best_rank": best_rank}, | ||||
| 227 | + **({"estimated_fit": estimated_fit} if estimated_fit else {}), | ||||
| 228 | + "_sort": (-len(why), best_rank if best_rank is not None else 10_000, -(_date_ord(attrs.get("release_date")))), | ||||
| 229 | + }) | ||||
| 230 | + matches.sort(key=lambda x: x["_sort"]) | ||||
| 231 | + for x in matches: | ||||
| 232 | + x.pop("_sort", None) | ||||
| 233 | + return {"matches": matches[:limit], "total": len(matches), | ||||
| 234 | + "filters_applied": {k: v for k, v in {"use_case": use_case, "deployment": deployment, "memory_gb": memory_gb, "quant": quant if memory_gb is not None or use_case == "local" else None, | ||||
| 235 | + "context_min": context_min, "license": license, "openness": openness or None, "max_input_price": max_input_price, | ||||
| 236 | + "max_output_price": max_output_price, "modalities": sorted(want_mods) or None}.items() if v not in (None, "any")}, | ||||
| 237 | + "rules": {k: RULES[k] for k in RULES if k == use_case or k not in USE_CASES}, | ||||
| 238 | + "note": "Deterministic filters over observed attributes, current prices and current benchmark results of canonical models; sorted by number of satisfied " | ||||
| 239 | + "criteria, then best benchmark rank, then release date. No composite score. Hardware fit is an estimate."} | ||||
| 240 | + | ||||
| 241 | + | ||||
| 242 | +def groups_cat(groups: dict[str, dict[str, Any]], slug: str) -> str: | ||||
| 243 | + for g in groups.values(): | ||||
| 244 | + if g["benchmark"]["slug"] == slug: | ||||
| 245 | + return (g["benchmark"].get("category") or "").lower() | ||||
| 246 | + return "" | ||||
| 247 | + | ||||
| 248 | + | ||||
| 249 | +def _date_ord(v: Any) -> int: | ||||
| 250 | + s = str(v or "") | ||||
| 251 | + digits = "".join(ch for ch in s[:10] if ch.isdigit()) | ||||
| 252 | + return int(digits.ljust(8, "0")[:8]) if digits else 0 | ||||
| 253 | + | ||||
| 254 | + | ||||
| 255 | +__all__ = ["RULES", "USE_CASES", "find_models"] | ||||
added
src/aiatlas/services/frontier.py
+303 −0
@@ -0,0 +1,303 @@ | |||||
| 1 | +"""Benchmark comparability groups, per-model leaderboards and the "frontier" model set (API 1.1). Deterministic, no LLM. | ||||
| 2 | + | ||||
| 3 | +A *group* is (benchmark, canonical metric, config_key): the comparability-relevant part of the result configuration hashed by | ||||
| 4 | +`ontology.benchmarks.config_key` (task keys + metric). Rows written before migration 0003 carry a NULL `config_key` / `trust_level`; | ||||
| 5 | +both are recomputed here from `config` and the source key so the API behaves the same before and after canonicalisation. | ||||
| 6 | + | ||||
| 7 | +Leaderboards are ONE row per canonical model: the best current row of that model inside the chosen group. Effort variants folded into | ||||
| 8 | +a model land in the same group (reasoning effort is a *condition*, not a task key) and are summarised in `config`.""" | ||||
| 9 | +from __future__ import annotations | ||||
| 10 | + | ||||
| 11 | +from collections import Counter, defaultdict | ||||
| 12 | +from datetime import UTC, datetime, timedelta | ||||
| 13 | +from typing import Any | ||||
| 14 | + | ||||
| 15 | +from sqlalchemy.ext.asyncio import AsyncConnection | ||||
| 16 | + | ||||
| 17 | +from aiatlas.db import fetch_all | ||||
| 18 | +from aiatlas.ontology.benchmarks import ( | ||||
| 19 | + CONDITION_KEYS, | ||||
| 20 | + METRICS, | ||||
| 21 | + TASK_KEYS, | ||||
| 22 | + TRUST_LABELS, | ||||
| 23 | + comparability, | ||||
| 24 | + config_key, | ||||
| 25 | + family_of, | ||||
| 26 | + normalize_metric, | ||||
| 27 | + trust_level, | ||||
| 28 | +) | ||||
| 29 | +from aiatlas.services import cache | ||||
| 30 | + | ||||
| 31 | +RESULT_SELECT = """ | ||||
| 32 | + select r.id, r.model_id, r.benchmark_id, r.score, r.metric, r.unit, r.higher_is_better, r.config, r.evaluated_at, r.observed_at, r.source_url, r.tier, | ||||
| 33 | + r.confidence, r.valid_to, r.config_key, r.trust_level, r.variant, r.run_group, r.is_current, r.extractor, s.key as source_key, | ||||
| 34 | + m.slug as model_slug, m.canonical_name as model_name, m.entity_type as model_type, m.merged_into as model_merged_into, m.organization_id, | ||||
| 35 | + m.attributes as model_attrs, m.family_id, mo.slug as org_slug, mo.canonical_name as org_name, | ||||
| 36 | + b.slug as benchmark_slug, b.canonical_name as benchmark_name, b.attributes as benchmark_attrs | ||||
| 37 | + from benchmark_results r | ||||
| 38 | + join entities m on m.id = r.model_id left join entities mo on mo.id = m.organization_id | ||||
| 39 | + join entities b on b.id = r.benchmark_id | ||||
| 40 | + left join sources s on s.id = r.source_id | ||||
| 41 | +""" | ||||
| 42 | +CURRENT = "r.valid_to is null and r.is_current" | ||||
| 43 | +CANONICAL_MODEL = "m.entity_type = 'model' and m.merged_into is null" | ||||
| 44 | +FRONTIER_CACHE_KEY = "frontier:model-ids:v1" | ||||
| 45 | +FRONTIER_TTL_S = 600 | ||||
| 46 | +FRONTIER_METHODOLOGY = ("Frontier models = canonical models released in the last 12 months by organizations with at least 3 canonical models, " | ||||
| 47 | + "OR holding a top-10 rank in the primary comparability group of at least one benchmark. Artifacts and folded variants are excluded. " | ||||
| 48 | + "No composite score is used to pick them.") | ||||
| 49 | + | ||||
| 50 | + | ||||
| 51 | +# ------------------------------------------------------------------------------------------------------------------ row enrichment | ||||
| 52 | + | ||||
| 53 | + | ||||
| 54 | +def enrich(row: dict[str, Any]) -> dict[str, Any]: | ||||
| 55 | + """Fill `config_key` / `trust_level` / `metric_canonical` when the writer has not (pre-0003 rows).""" | ||||
| 56 | + cfg = row.get("config") or {} | ||||
| 57 | + metric = normalize_metric(row.get("metric")) or (row.get("metric") or "").strip().lower() or "score" | ||||
| 58 | + row["metric_canonical"] = metric | ||||
| 59 | + if not row.get("config_key"): | ||||
| 60 | + row["config_key"] = config_key(cfg, row.get("metric")) | ||||
| 61 | + if not row.get("trust_level"): | ||||
| 62 | + row["trust_level"] = trust_level(row.get("source_key"), cfg, extractor=row.get("extractor") or "deterministic") | ||||
| 63 | + return row | ||||
| 64 | + | ||||
| 65 | + | ||||
| 66 | +def group_label(metric: str, cfg: dict[str, Any] | None) -> str: | ||||
| 67 | + parts = [f"{k}={cfg[k]}" for k in TASK_KEYS if cfg and cfg.get(k) not in (None, "", [], {})] | ||||
| 68 | + return metric + (" · " + " · ".join(str(p) for p in parts) if parts else "") | ||||
| 69 | + | ||||
| 70 | + | ||||
| 71 | +def config_summary(cfg: dict[str, Any] | None) -> dict[str, Any]: | ||||
| 72 | + """Task + condition keys only (bookkeeping keys such as aa_slug are dropped).""" | ||||
| 73 | + return {k: cfg[k] for k in (*TASK_KEYS, *CONDITION_KEYS) if cfg and cfg.get(k) not in (None, "", [], {})} | ||||
| 74 | + | ||||
| 75 | + | ||||
| 76 | +def direction_of(rows: list[dict[str, Any]], metric: str) -> bool: | ||||
| 77 | + votes = Counter(bool(r.get("higher_is_better")) for r in rows if r.get("higher_is_better") is not None) | ||||
| 78 | + if votes: | ||||
| 79 | + return votes.most_common(1)[0][0] | ||||
| 80 | + spec = METRICS.get(metric) | ||||
| 81 | + return bool(spec["higher_is_better"]) if spec else True | ||||
| 82 | + | ||||
| 83 | + | ||||
| 84 | +# ------------------------------------------------------------------------------------------------------------------ loading | ||||
| 85 | + | ||||
| 86 | + | ||||
| 87 | +async def load_results(conn: AsyncConnection, *, benchmark_ids: list[str] | None = None, model_ids: list[str] | None = None, current_only: bool = True, | ||||
| 88 | + canonical_models_only: bool = True, limit: int = 50_000) -> list[dict[str, Any]]: | ||||
| 89 | + where = [] | ||||
| 90 | + params: dict[str, Any] = {"lim": limit} | ||||
| 91 | + if benchmark_ids is not None: | ||||
| 92 | + where.append("r.benchmark_id = any(cast(:bids as text[]))") | ||||
| 93 | + params["bids"] = benchmark_ids | ||||
| 94 | + if model_ids is not None: | ||||
| 95 | + where.append("r.model_id = any(cast(:mids as text[]))") | ||||
| 96 | + params["mids"] = model_ids | ||||
| 97 | + if current_only: | ||||
| 98 | + where.append(CURRENT) | ||||
| 99 | + if canonical_models_only: | ||||
| 100 | + where.append(CANONICAL_MODEL) | ||||
| 101 | + sql = RESULT_SELECT + " where " + (" and ".join(where) or "true") + " order by r.benchmark_id, r.observed_at desc limit :lim" | ||||
| 102 | + return [enrich(r) for r in await fetch_all(conn, sql, **params)] | ||||
| 103 | + | ||||
| 104 | + | ||||
| 105 | +# ------------------------------------------------------------------------------------------------------------------ grouping | ||||
| 106 | + | ||||
| 107 | + | ||||
| 108 | +def group_rows(rows: list[dict[str, Any]]) -> dict[tuple[str, str, str], dict[str, Any]]: | ||||
| 109 | + """{(benchmark_id, metric, config_key): {rows, metric, config_key, label, n, models, higher_is_better, representative config}}.""" | ||||
| 110 | + groups: dict[tuple[str, str, str], dict[str, Any]] = {} | ||||
| 111 | + for r in rows: | ||||
| 112 | + k = (r["benchmark_id"], r["metric_canonical"], r["config_key"]) | ||||
| 113 | + g = groups.get(k) | ||||
| 114 | + if g is None: | ||||
| 115 | + g = groups[k] = {"benchmark_id": r["benchmark_id"], "metric": r["metric_canonical"], "config_key": r["config_key"], "rows": [], "models": set(), | ||||
| 116 | + "config": {k2: r["config"][k2] for k2 in TASK_KEYS if (r.get("config") or {}).get(k2) not in (None, "", [], {})}} | ||||
| 117 | + g["rows"].append(r) | ||||
| 118 | + g["models"].add(r["model_id"]) | ||||
| 119 | + for g in groups.values(): | ||||
| 120 | + g["n"] = len(g["rows"]) | ||||
| 121 | + g["model_count"] = len(g["models"]) | ||||
| 122 | + g["label"] = group_label(g["metric"], g["config"]) | ||||
| 123 | + g["higher_is_better"] = direction_of(g["rows"], g["metric"]) | ||||
| 124 | + g["trust_mix"] = dict(Counter(r["trust_level"] for r in g["rows"])) | ||||
| 125 | + return groups | ||||
| 126 | + | ||||
| 127 | + | ||||
| 128 | +def primary_metric(benchmark_attrs: dict[str, Any] | None, groups: list[dict[str, Any]]) -> str | None: | ||||
| 129 | + """Registry metric when it has current rows, else the most populated metric that is not a LiveBench per-category average.""" | ||||
| 130 | + declared = normalize_metric((benchmark_attrs or {}).get("metric")) | ||||
| 131 | + present = Counter() | ||||
| 132 | + for g in groups: | ||||
| 133 | + present[g["metric"]] += g["n"] | ||||
| 134 | + if declared and present.get(declared): | ||||
| 135 | + return declared | ||||
| 136 | + ranked = [m for m, _ in present.most_common() if not m.startswith("category:")] or [m for m, _ in present.most_common()] | ||||
| 137 | + return ranked[0] if ranked else declared | ||||
| 138 | + | ||||
| 139 | + | ||||
| 140 | +def primary_group(benchmark_attrs: dict[str, Any] | None, groups: list[dict[str, Any]]) -> dict[str, Any] | None: | ||||
| 141 | + metric = primary_metric(benchmark_attrs, groups) | ||||
| 142 | + cands = [g for g in groups if g["metric"] == metric] | ||||
| 143 | + if not cands: | ||||
| 144 | + return None | ||||
| 145 | + return max(cands, key=lambda g: (g["model_count"], g["n"], g["config_key"])) | ||||
| 146 | + | ||||
| 147 | + | ||||
| 148 | +def best_per_model(rows: list[dict[str, Any]], higher_is_better: bool) -> list[dict[str, Any]]: | ||||
| 149 | + best: dict[str, dict[str, Any]] = {} | ||||
| 150 | + for r in rows: | ||||
| 151 | + cur = best.get(r["model_id"]) | ||||
| 152 | + if cur is None or (r["score"] > cur["score"] if higher_is_better else r["score"] < cur["score"]) or \ | ||||
| 153 | + (r["score"] == cur["score"] and (r.get("evaluated_at") or r["observed_at"]) > (cur.get("evaluated_at") or cur["observed_at"])): | ||||
| 154 | + best[r["model_id"]] = r | ||||
| 155 | + return sorted(best.values(), key=lambda r: (-r["score"] if higher_is_better else r["score"], r["model_name"] or "")) | ||||
| 156 | + | ||||
| 157 | + | ||||
| 158 | +def rank_rows(rows: list[dict[str, Any]], higher_is_better: bool) -> list[dict[str, Any]]: | ||||
| 159 | + """Competition ranking (1, 2, 2, 4) over best-per-model rows.""" | ||||
| 160 | + ranked = best_per_model(rows, higher_is_better) | ||||
| 161 | + out: list[dict[str, Any]] = [] | ||||
| 162 | + prev_score, prev_rank = None, 0 | ||||
| 163 | + for i, r in enumerate(ranked, start=1): | ||||
| 164 | + rank = prev_rank if prev_score is not None and r["score"] == prev_score else i | ||||
| 165 | + prev_score, prev_rank = r["score"], rank | ||||
| 166 | + out.append({**r, "rank": rank}) | ||||
| 167 | + return out | ||||
| 168 | + | ||||
| 169 | + | ||||
| 170 | +def leaderboard_rows(group: dict[str, Any], *, history_rows: list[dict[str, Any]] | None = None, comparable_only: bool = False) -> list[dict[str, Any]]: | ||||
| 171 | + """Public leaderboard rows for one group. `history_rows` (closed rows of the same group) give the previous rank per model.""" | ||||
| 172 | + hib = group["higher_is_better"] | ||||
| 173 | + ranked = rank_rows(group["rows"], hib) | ||||
| 174 | + leader = ranked[0] if ranked else None | ||||
| 175 | + prev_rank: dict[str, int] = {} | ||||
| 176 | + if history_rows: | ||||
| 177 | + prev_rank = {r["model_id"]: r["rank"] for r in rank_rows(history_rows, hib)} | ||||
| 178 | + out: list[dict[str, Any]] = [] | ||||
| 179 | + for r in ranked: | ||||
| 180 | + level, reasons = comparability(leader["config"] if leader else None, r.get("config"), leader["metric"] if leader else None, r.get("metric")) if leader else ("comparable", []) | ||||
| 181 | + if comparable_only and level != "comparable": | ||||
| 182 | + continue | ||||
| 183 | + prev = prev_rank.get(r["model_id"]) | ||||
| 184 | + out.append({ | ||||
| 185 | + "rank": r["rank"], "model": _model_ref(r), "score": r["score"], "metric": r["metric_canonical"], "unit": r.get("unit"), "higher_is_better": hib, | ||||
| 186 | + "delta_rank": (prev - r["rank"]) if prev is not None else None, "previous_rank": prev, | ||||
| 187 | + "trust_level": r["trust_level"], "trust_label": TRUST_LABELS.get(r["trust_level"], r["trust_level"]), "config": config_summary(r.get("config")), | ||||
| 188 | + "config_key": r["config_key"], "comparability": level, "comparability_reasons": reasons, "evaluated_at": r.get("evaluated_at"), "observed_at": r["observed_at"], | ||||
| 189 | + "source_url": r.get("source_url"), "tier": r.get("tier"), "result_id": r["id"], "n_rows": sum(1 for x in group["rows"] if x["model_id"] == r["model_id"]), | ||||
| 190 | + }) | ||||
| 191 | + return out | ||||
| 192 | + | ||||
| 193 | + | ||||
| 194 | +def _model_ref(r: dict[str, Any]) -> dict[str, Any]: | ||||
| 195 | + attrs = r.get("model_attrs") or {} | ||||
| 196 | + return {"id": r["model_id"], "slug": r["model_slug"], "name": r["model_name"], "entity_type": r.get("model_type", "model"), | ||||
| 197 | + "organization": {"id": r.get("organization_id"), "slug": r.get("org_slug"), "name": r.get("org_name")} if r.get("organization_id") else None, | ||||
| 198 | + "attributes": {k: attrs[k] for k in ("openness", "parameter_count", "context_length", "release_date", "modalities", "license", "family") if attrs.get(k) not in (None, "", [])}} | ||||
| 199 | + | ||||
| 200 | + | ||||
| 201 | +def group_summary(g: dict[str, Any]) -> dict[str, Any]: | ||||
| 202 | + return {"metric": g["metric"], "config_key": g["config_key"], "label": g["label"], "n": g["n"], "model_count": g["model_count"], "config": g["config"], | ||||
| 203 | + "higher_is_better": g["higher_is_better"], "trust_mix": g["trust_mix"]} | ||||
| 204 | + | ||||
| 205 | + | ||||
| 206 | +def frontier_series(rows: list[dict[str, Any]], higher_is_better: bool) -> list[dict[str, Any]]: | ||||
| 207 | + """History of the leader: each time a new best score appears (ordered by coalesce(evaluated_at, observed_at)).""" | ||||
| 208 | + ordered = sorted(rows, key=lambda r: ((r.get("evaluated_at") or r["observed_at"]), r["observed_at"], r["id"])) | ||||
| 209 | + best: float | None = None | ||||
| 210 | + out: list[dict[str, Any]] = [] | ||||
| 211 | + for r in ordered: | ||||
| 212 | + s = float(r["score"]) | ||||
| 213 | + better = best is None or (s > best if higher_is_better else s < best) | ||||
| 214 | + if better: | ||||
| 215 | + best = s | ||||
| 216 | + out.append({"date": (r.get("evaluated_at") or r["observed_at"]), "model": _model_ref(r), "score": s, "trust_level": r["trust_level"], | ||||
| 217 | + "config": config_summary(r.get("config")), "result_id": r["id"]}) | ||||
| 218 | + return out | ||||
| 219 | + | ||||
| 220 | + | ||||
| 221 | +def leader_at(rows: list[dict[str, Any]], at: datetime, benchmark_attrs: dict[str, Any] | None) -> dict[str, Any] | None: | ||||
| 222 | + """Leader of the primary group as it was known at `at`: rows observed at or before `at` and not closed before `at`.""" | ||||
| 223 | + visible = [r for r in rows if r["observed_at"] <= at and (r.get("valid_to") is None or r["valid_to"] > at)] | ||||
| 224 | + if not visible: | ||||
| 225 | + return None | ||||
| 226 | + groups = list(group_rows(visible).values()) | ||||
| 227 | + pg = primary_group(benchmark_attrs, groups) | ||||
| 228 | + if not pg: | ||||
| 229 | + return None | ||||
| 230 | + ranked = rank_rows(pg["rows"], pg["higher_is_better"]) | ||||
| 231 | + if not ranked: | ||||
| 232 | + return None | ||||
| 233 | + top = ranked[0] | ||||
| 234 | + return {"model": _model_ref(top), "score": top["score"], "metric": pg["metric"], "config_key": pg["config_key"], "group_label": pg["label"], | ||||
| 235 | + "trust_level": top["trust_level"], "n_models": pg["model_count"], "as_of": at} | ||||
| 236 | + | ||||
| 237 | + | ||||
| 238 | +# ------------------------------------------------------------------------------------------------------------------ benchmark catalogue | ||||
| 239 | + | ||||
| 240 | + | ||||
| 241 | +async def benchmark_meta(conn: AsyncConnection, ids: list[str] | None = None) -> dict[str, dict[str, Any]]: | ||||
| 242 | + where = "e.entity_type = 'benchmark' and e.merged_into is null" + (" and e.id = any(cast(:ids as text[]))" if ids is not None else "") | ||||
| 243 | + rows = await fetch_all(conn, f"select e.id, e.slug, e.canonical_name, e.attributes from entities e where {where}", ids=ids) | ||||
| 244 | + out: dict[str, dict[str, Any]] = {} | ||||
| 245 | + for r in rows: | ||||
| 246 | + attrs = r["attributes"] or {} | ||||
| 247 | + fam, variant = family_of(r["slug"]) | ||||
| 248 | + out[r["id"]] = {"id": r["id"], "slug": r["slug"], "name": r["canonical_name"], "attributes": attrs, "category": attrs.get("category"), | ||||
| 249 | + "family": attrs.get("family") or fam, "variant": attrs.get("variant") or variant, | ||||
| 250 | + "metric": normalize_metric(attrs.get("metric")) or attrs.get("metric"), "unit": attrs.get("unit"), | ||||
| 251 | + "direction": attrs.get("direction") or ("higher" if METRICS.get(normalize_metric(attrs.get("metric")) or "", {}).get("higher_is_better", True) else "lower")} | ||||
| 252 | + return out | ||||
| 253 | + | ||||
| 254 | + | ||||
| 255 | +async def all_primary_groups(conn: AsyncConnection, *, min_results: int = 1) -> dict[str, dict[str, Any]]: | ||||
| 256 | + """{benchmark_id: primary group (with rows)} across every benchmark, from current rows of canonical models.""" | ||||
| 257 | + meta = await benchmark_meta(conn) | ||||
| 258 | + rows = await load_results(conn) | ||||
| 259 | + by_bench: dict[str, list[dict[str, Any]]] = defaultdict(list) | ||||
| 260 | + for r in rows: | ||||
| 261 | + by_bench[r["benchmark_id"]].append(r) | ||||
| 262 | + out: dict[str, dict[str, Any]] = {} | ||||
| 263 | + for bid, brows in by_bench.items(): | ||||
| 264 | + groups = list(group_rows(brows).values()) | ||||
| 265 | + pg = primary_group(meta.get(bid, {}).get("attributes"), groups) | ||||
| 266 | + if pg and pg["n"] >= min_results: | ||||
| 267 | + pg = dict(pg) | ||||
| 268 | + pg["benchmark"] = meta.get(bid) or {"id": bid, "slug": brows[0]["benchmark_slug"], "name": brows[0]["benchmark_name"]} | ||||
| 269 | + pg["all_groups"] = groups | ||||
| 270 | + out[bid] = pg | ||||
| 271 | + return out | ||||
| 272 | + | ||||
| 273 | + | ||||
| 274 | +# ------------------------------------------------------------------------------------------------------------------ frontier model set | ||||
| 275 | + | ||||
| 276 | + | ||||
| 277 | +async def frontier_model_ids(conn: AsyncConnection, *, use_cache: bool = True) -> tuple[set[str], dict[str, Any]]: | ||||
| 278 | + """Frontier composition (see FRONTIER_METHODOLOGY). Returns (ids, sample counts).""" | ||||
| 279 | + if use_cache: | ||||
| 280 | + hit = await cache.cache_get(FRONTIER_CACHE_KEY) | ||||
| 281 | + if hit: | ||||
| 282 | + return set(hit["ids"]), hit["composition"] | ||||
| 283 | + since = (datetime.now(UTC) - timedelta(days=365)).date().isoformat() | ||||
| 284 | + recent = await fetch_all(conn, """ | ||||
| 285 | + select e.id from entities e where e.entity_type = 'model' and e.merged_into is null and e.attributes->>'release_date' >= :since | ||||
| 286 | + and e.organization_id in (select organization_id from entities where entity_type = 'model' and merged_into is null and organization_id is not null | ||||
| 287 | + group by organization_id having count(*) >= 3)""", since=since) | ||||
| 288 | + recent_ids = {r["id"] for r in recent} | ||||
| 289 | + top10: set[str] = set() | ||||
| 290 | + groups = await all_primary_groups(conn) | ||||
| 291 | + for g in groups.values(): | ||||
| 292 | + for r in rank_rows(g["rows"], g["higher_is_better"]): | ||||
| 293 | + if r["rank"] <= 10: | ||||
| 294 | + top10.add(r["model_id"]) | ||||
| 295 | + ids = recent_ids | top10 | ||||
| 296 | + composition = {"recent_by_active_orgs": len(recent_ids), "top10_on_a_benchmark": len(top10), "total": len(ids), "since": since} | ||||
| 297 | + await cache.cache_set(FRONTIER_CACHE_KEY, {"ids": sorted(ids), "composition": composition}, FRONTIER_TTL_S) | ||||
| 298 | + return ids, composition | ||||
| 299 | + | ||||
| 300 | + | ||||
| 301 | +__all__ = ["CANONICAL_MODEL", "CURRENT", "FRONTIER_METHODOLOGY", "RESULT_SELECT", "all_primary_groups", "benchmark_meta", "best_per_model", "config_summary", | ||||
| 302 | + "direction_of", "enrich", "frontier_model_ids", "frontier_series", "group_label", "group_rows", "group_summary", "leader_at", "leaderboard_rows", | ||||
| 303 | + "load_results", "primary_group", "primary_metric", "rank_rows"] | ||||
modified
src/aiatlas/services/hardware_fit.py
+107 −12
@@ -1,25 +1,38 @@ | |||||
| 1 | """ESTIMATED memory footprint of a model on a piece of hardware. Transparent formula, always labelled as an estimate: | 1 | """ESTIMATED memory footprint of a model on a piece of hardware. Transparent formula, always labelled as an estimate: | ||
| 2 | 2 | ||||
| 3 | weights = parameter_count × bytes_per_param × 1.15 (runtime overhead: activations, buffers, fragmentation) | 3 | weights = parameter_count × bytes_per_param × 1.15 (runtime overhead: activations, buffers, fragmentation) | ||
| 4 | − kv_cache = 0.5 GB per 8 192 tokens of context (coarse, architecture-agnostic) | 4 | + — or the OBSERVED `file_size_gb` of an artifact when one is available (flagged `weights_source: observed`) | ||
| 5 | − fits = estimated_memory_gb ≤ hardware memory − 2 GB (OS / framework headroom) | 5 | + kv_cache = per layer: 2 (K and V) × kv_heads × head_dim × bytes × context × batch — when the architecture metadata is known | ||
| 6 | + (`num_hidden_layers`, `num_key_value_heads` | `num_attention_heads`, `head_dim` | `hidden_size`); otherwise the documented | ||||
| 7 | + heuristic 0.5 GB per 8 192 tokens (× batch), architecture-agnostic | ||||
| 8 | + fits = estimated_memory_gb ≤ hardware memory − 2 GB (OS / framework headroom); multi-GPU = sum of device memory, interconnect ignored | ||||
| 6 | """ | 9 | """ | ||
| 7 | from __future__ import annotations | 10 | from __future__ import annotations | ||
| 8 | 11 | ||||
| 9 | from typing import Any | 12 | from typing import Any | ||
| 10 | 13 | ||||
| 11 | −BYTES_PER_PARAM = {"4bit": 0.5, "8bit": 1.0, "fp16": 2.0} | 14 | +BYTES_PER_PARAM = {"4bit": 0.5, "8bit": 1.0, "fp16": 2.0, "bf16": 2.0, "fp8": 1.0, "int4": 0.5, "int8": 1.0, "fp32": 4.0} | ||
| 15 | +KV_BYTES = {"fp16": 2.0, "bf16": 2.0, "fp8": 1.0, "int8": 1.0, "8bit": 1.0, "4bit": 2.0, "int4": 2.0, "fp32": 4.0} # KV cache dtype (4-bit weights usually keep fp16 KV) | ||||
| 12 | OVERHEAD = 1.15 | 16 | OVERHEAD = 1.15 | ||
| 13 | KV_GB_PER_8K = 0.5 | 17 | KV_GB_PER_8K = 0.5 | ||
| 14 | RESERVED_GB = 2.0 | 18 | RESERVED_GB = 2.0 | ||
| 15 | ASSUMPTIONS = [ | 19 | ASSUMPTIONS = [ | ||
| 16 | − "Estimated, not measured: weights = parameters × bytes/param × 1.15 runtime overhead.", | 20 | + "Estimated, not measured: weights = parameters × bytes/param × 1.15 runtime overhead (or the observed artifact file size when one is recorded).", | ||
| 17 | "bytes/param: 4bit = 0.5, 8bit = 1.0, fp16 = 2.0 (uniform quantization, no per-layer exceptions).", | 21 | "bytes/param: 4bit = 0.5, 8bit = 1.0, fp16 = 2.0 (uniform quantization, no per-layer exceptions).", | ||
| 18 | − "KV cache approximated at 0.5 GB per 8 192 tokens of context, independent of architecture (GQA/MLA models need less).", | 22 | + "KV cache: 2 × layers × kv_heads × head_dim × 2 bytes × context × batch when the architecture is known; otherwise 0.5 GB per 8 192 tokens " | ||
| 23 | + "(× batch), independent of architecture (GQA/MLA models need less).", | ||||
| 19 | "A model 'fits' when the estimate is at most the device memory minus 2 GB reserved for the OS and framework.", | 24 | "A model 'fits' when the estimate is at most the device memory minus 2 GB reserved for the OS and framework.", | ||
| 20 | "Mixture-of-experts models are estimated on total parameters (all experts must be resident); active parameters are ignored.", | 25 | "Mixture-of-experts models are estimated on total parameters (all experts must be resident); active parameters are ignored.", | ||
| 21 | "Device memory uses the largest configuration when several are listed (e.g. Apple silicon tiers).", | 26 | "Device memory uses the largest configuration when several are listed (e.g. Apple silicon tiers).", | ||
| 27 | + "Multi-GPU: device memories are summed; interconnect bandwidth, tensor-parallel replication and pipeline bubbles are not modelled.", | ||||
| 22 | ] | 28 | ] | ||
| 29 | +QUANT_ALIASES = {"q4": "4bit", "int4": "4bit", "nf4": "4bit", "gguf-q4": "4bit", "mlx-4bit": "4bit", "q8": "8bit", "int8": "8bit", "fp8": "8bit", | ||||
| 30 | + "f16": "fp16", "half": "fp16", "bf16": "fp16", "fp16": "fp16", "4bit": "4bit", "8bit": "8bit", "fp32": "fp32", "f32": "fp32"} | ||||
| 31 | + | ||||
| 32 | + | ||||
| 33 | +def normalize_quant(q: str | None) -> str: | ||||
| 34 | + s = (q or "4bit").strip().lower() | ||||
| 35 | + return QUANT_ALIASES.get(s, s if s in BYTES_PER_PARAM else "4bit") | ||||
| 23 | 36 | ||||
| 24 | 37 | ||||
| 25 | def estimate_memory_gb(parameter_count: float, quant: str = "4bit", context: int = 8192) -> float: | 38 | def estimate_memory_gb(parameter_count: float, quant: str = "4bit", context: int = 8192) -> float: | ||
@@ -45,24 +58,106 @@ def hardware_memory_gb(attrs: dict[str, Any] | None) -> float | None: | |||||
| 45 | return None | 58 | return None | ||
| 46 | 59 | ||||
| 47 | 60 | ||||
| 48 | −def parameter_count(attrs: dict[str, Any] | None) -> float | None: | 61 | +def hardware_memory_options(attrs: dict[str, Any] | None) -> list[float]: | ||
| 49 | − v = (attrs or {}).get("parameter_count") | 62 | + v = (attrs or {}).get("memory_gb") | ||
| 50 | − if isinstance(v, (int, float)) and not isinstance(v, bool) and v > 0: | 63 | + if isinstance(v, list): | ||
| 64 | + return sorted(float(x) for x in v if isinstance(x, (int, float)) and not isinstance(x, bool)) | ||||
| 65 | + m = hardware_memory_gb(attrs) | ||||
| 66 | + return [m] if m is not None else [] | ||||
| 67 | + | ||||
| 68 | + | ||||
| 69 | +def _num(v: Any) -> float | None: | ||||
| 70 | + if isinstance(v, bool): | ||||
| 71 | + return None | ||||
| 72 | + if isinstance(v, (int, float)): | ||||
| 51 | return float(v) | 73 | return float(v) | ||
| 52 | if isinstance(v, str): | 74 | if isinstance(v, str): | ||
| 53 | try: | 75 | try: | ||
| 54 | − f = float(v) | 76 | + return float(v.replace(",", "")) | ||
| 55 | − return f if f > 0 else None | ||||
| 56 | except ValueError: | 77 | except ValueError: | ||
| 57 | return None | 78 | return None | ||
| 58 | return None | 79 | return None | ||
| 59 | 80 | ||||
| 60 | 81 | ||||
| 82 | +def parameter_count(attrs: dict[str, Any] | None) -> float | None: | ||||
| 83 | + v = _num((attrs or {}).get("parameter_count")) | ||||
| 84 | + return v if v is not None and v > 0 else None | ||||
| 85 | + | ||||
| 86 | + | ||||
| 87 | +def file_size_gb(attrs: dict[str, Any] | None) -> float | None: | ||||
| 88 | + """Observed weights size (Hugging Face repo file total) when recorded.""" | ||||
| 89 | + v = _num((attrs or {}).get("file_size_gb")) | ||||
| 90 | + return v if v is not None and v > 0 else None | ||||
| 91 | + | ||||
| 92 | + | ||||
| 93 | +def architecture(attrs: dict[str, Any] | None) -> dict[str, float] | None: | ||||
| 94 | + """{layers, kv_heads, head_dim} from Hugging Face-style config keys when all three can be derived, else None.""" | ||||
| 95 | + a = attrs or {} | ||||
| 96 | + layers = _num(a.get("num_hidden_layers") or a.get("n_layer") or a.get("num_layers")) | ||||
| 97 | + heads = _num(a.get("num_attention_heads") or a.get("n_head")) | ||||
| 98 | + kv_heads = _num(a.get("num_key_value_heads")) or heads | ||||
| 99 | + head_dim = _num(a.get("head_dim")) | ||||
| 100 | + hidden = _num(a.get("hidden_size") or a.get("n_embd")) | ||||
| 101 | + if head_dim is None and hidden and heads: | ||||
| 102 | + head_dim = hidden / heads | ||||
| 103 | + if layers and kv_heads and head_dim: | ||||
| 104 | + return {"layers": layers, "kv_heads": kv_heads, "head_dim": head_dim} | ||||
| 105 | + return None | ||||
| 106 | + | ||||
| 107 | + | ||||
| 108 | +def kv_cache_gb(context: int, *, batch: int = 1, quant: str = "4bit", arch: dict[str, float] | None = None) -> tuple[float, str]: | ||||
| 109 | + """(GB, method). `method` is 'architecture' when computed from layers × kv_heads × head_dim, else 'heuristic'.""" | ||||
| 110 | + ctx = max(0.0, float(context)) | ||||
| 111 | + b = max(1, int(batch)) | ||||
| 112 | + if arch: | ||||
| 113 | + bytes_ = KV_BYTES.get(quant, 2.0) | ||||
| 114 | + gb = 2.0 * arch["layers"] * arch["kv_heads"] * arch["head_dim"] * bytes_ * ctx * b / 1e9 | ||||
| 115 | + return round(gb, 3), "architecture" | ||||
| 116 | + return round(KV_GB_PER_8K * ctx / 8192.0 * b, 3), "heuristic" | ||||
| 117 | + | ||||
| 118 | + | ||||
| 61 | def fit(parameter_count_: float, memory_gb: float, quant: str = "4bit", context: int = 8192) -> dict[str, Any]: | 119 | def fit(parameter_count_: float, memory_gb: float, quant: str = "4bit", context: int = 8192) -> dict[str, Any]: | ||
| 62 | est = estimate_memory_gb(parameter_count_, quant, context) | 120 | est = estimate_memory_gb(parameter_count_, quant, context) | ||
| 63 | headroom = round(memory_gb - RESERVED_GB - est, 2) | 121 | headroom = round(memory_gb - RESERVED_GB - est, 2) | ||
| 64 | − return {"quantization": quant, "estimated_memory_gb": est, "fits": headroom >= 0, "headroom_gb": headroom, | 122 | + return {"quantization": quant, "estimated_memory_gb": est, "fits": headroom >= 0, "headroom_gb": headroom, "estimated": True, | ||
| 65 | "note": f"estimated: {parameter_count_ / 1e9:.1f}B params × {BYTES_PER_PARAM.get(quant, 0.5)} B × 1.15 + KV cache for {context} tokens"} | 123 | "note": f"estimated: {parameter_count_ / 1e9:.1f}B params × {BYTES_PER_PARAM.get(quant, 0.5)} B × 1.15 + KV cache for {context} tokens"} | ||
| 66 | 124 | ||||
| 67 | 125 | ||||
| 68 | −__all__ = ["ASSUMPTIONS", "BYTES_PER_PARAM", "estimate_memory_gb", "fit", "hardware_memory_gb", "parameter_count"] | 126 | +def fit_detailed(attrs: dict[str, Any] | None, memory_gb: float, *, quant: str = "4bit", context: int = 8192, batch: int = 1, gpu_count: int = 1, | ||
| 127 | + observed_size_gb: float | None = None) -> dict[str, Any] | None: | ||||
| 128 | + """Full breakdown (weights / KV cache / overhead / headroom). `observed_size_gb` (artifact file size) replaces the parameter-based weights | ||||
| 129 | + estimate when given. Returns None when neither parameters nor an observed size are known — nothing is estimated from thin air.""" | ||||
| 130 | + quant = normalize_quant(quant) | ||||
| 131 | + params = parameter_count(attrs) | ||||
| 132 | + observed = observed_size_gb if observed_size_gb else None | ||||
| 133 | + if params is None and observed is None: | ||||
| 134 | + return None | ||||
| 135 | + if observed is not None: | ||||
| 136 | + weights_raw = observed | ||||
| 137 | + weights_source = "observed" | ||||
| 138 | + else: | ||||
| 139 | + assert params is not None | ||||
| 140 | + weights_raw = params * BYTES_PER_PARAM[quant] / 1e9 | ||||
| 141 | + weights_source = "estimated" | ||||
| 142 | + overhead = round(weights_raw * (OVERHEAD - 1), 2) | ||||
| 143 | + kv, kv_method = kv_cache_gb(context, batch=batch, quant=quant, arch=architecture(attrs)) | ||||
| 144 | + total = round(weights_raw + overhead + kv, 2) | ||||
| 145 | + total_memory = float(memory_gb) * max(1, int(gpu_count)) | ||||
| 146 | + headroom = round(total_memory - RESERVED_GB - total, 2) | ||||
| 147 | + out: dict[str, Any] = { | ||||
| 148 | + "quantization": quant, "estimated": True, "fits": headroom >= 0, "estimated_memory_gb": total, "headroom_gb": headroom, | ||||
| 149 | + "breakdown": {"weights_gb": round(weights_raw, 2), "weights_source": weights_source, "overhead_gb": overhead, "kv_cache_gb": kv, "kv_cache_method": kv_method, | ||||
| 150 | + "reserved_gb": RESERVED_GB, "context": int(context), "batch": int(batch)}, | ||||
| 151 | + "device": {"memory_gb": float(memory_gb), "gpu_count": int(gpu_count), "total_memory_gb": total_memory}, | ||||
| 152 | + "note": (f"{'observed' if weights_source == 'observed' else 'estimated'} weights {weights_raw:.1f} GB + 15% overhead + KV cache {kv:.2f} GB ({kv_method}) " | ||||
| 153 | + f"for {context} tokens × batch {batch}; {RESERVED_GB:.0f} GB reserved for the OS/framework"), | ||||
| 154 | + } | ||||
| 155 | + if params is not None: | ||||
| 156 | + out["parameter_count"] = params | ||||
| 157 | + if gpu_count > 1: | ||||
| 158 | + out["multi_gpu_note"] = "Device memories summed; interconnect bandwidth, tensor-parallel replication and pipeline bubbles are not modelled." | ||||
| 159 | + return out | ||||
| 160 | + | ||||
| 161 | + | ||||
| 162 | +__all__ = ["ASSUMPTIONS", "BYTES_PER_PARAM", "KV_BYTES", "RESERVED_GB", "architecture", "estimate_memory_gb", "file_size_gb", "fit", "fit_detailed", | ||||
| 163 | + "hardware_memory_gb", "hardware_memory_options", "kv_cache_gb", "normalize_quant", "parameter_count"] | ||||
added
src/aiatlas/services/pareto.py
+44 −0
@@ -0,0 +1,44 @@ | |||||
| 1 | +"""Pareto frontier (maximise y, minimise x) over labelled points. Pure Python, deterministic. | ||||
| 2 | + | ||||
| 3 | +A point is *Pareto-efficient* when no other point has both a lower-or-equal x and a higher-or-equal y with at least one strict inequality. | ||||
| 4 | +Exact ties (same x and same y) are kept together on the frontier — we never break a tie by choosing one model over another.""" | ||||
| 5 | +from __future__ import annotations | ||||
| 6 | + | ||||
| 7 | +from typing import Any | ||||
| 8 | + | ||||
| 9 | + | ||||
| 10 | +def pareto_frontier(points: list[dict[str, Any]], *, x: str = "x", y: str = "y", key: str = "id", maximize_y: bool = True) -> list[Any]: | ||||
| 11 | + """Return the `key`s of the Pareto-efficient points, ordered by increasing x. Points with a missing x or y are ignored.""" | ||||
| 12 | + valid = [p for p in points if p.get(x) is not None and p.get(y) is not None] | ||||
| 13 | + if not valid: | ||||
| 14 | + return [] | ||||
| 15 | + sign = 1.0 if maximize_y else -1.0 | ||||
| 16 | + # sort by x ascending, then by y descending (best first) so a sweep with the running best y finds the frontier | ||||
| 17 | + ordered = sorted(valid, key=lambda p: (float(p[x]), -sign * float(p[y]))) | ||||
| 18 | + frontier: list[Any] = [] | ||||
| 19 | + best_y: float | None = None | ||||
| 20 | + best_x: float | None = None | ||||
| 21 | + for p in ordered: | ||||
| 22 | + px, py = float(p[x]), sign * float(p[y]) | ||||
| 23 | + if best_y is None or py > best_y: | ||||
| 24 | + frontier.append(p[key]) | ||||
| 25 | + best_y, best_x = py, px | ||||
| 26 | + elif py == best_y and best_x is not None and px == best_x: | ||||
| 27 | + frontier.append(p[key]) # exact tie: keep both | ||||
| 28 | + return frontier | ||||
| 29 | + | ||||
| 30 | + | ||||
| 31 | +def is_dominated(p: dict[str, Any], others: list[dict[str, Any]], *, x: str = "x", y: str = "y", maximize_y: bool = True) -> bool: | ||||
| 32 | + """True when some other point is at least as good on both axes and strictly better on one.""" | ||||
| 33 | + sign = 1.0 if maximize_y else -1.0 | ||||
| 34 | + px, py = float(p[x]), sign * float(p[y]) | ||||
| 35 | + for o in others: | ||||
| 36 | + if o is p or o.get(x) is None or o.get(y) is None: | ||||
| 37 | + continue | ||||
| 38 | + ox, oy = float(o[x]), sign * float(o[y]) | ||||
| 39 | + if ox <= px and oy >= py and (ox < px or oy > py): | ||||
| 40 | + return True | ||||
| 41 | + return False | ||||
| 42 | + | ||||
| 43 | + | ||||
| 44 | +__all__ = ["is_dominated", "pareto_frontier"] | ||||
modified
src/aiatlas/services/search.py
+369 −61
@@ -1,28 +1,49 @@ | |||||
| 1 | −"""Search: Postgres FTS + trigram (+ pgvector when embeddings exist) and a deterministic natural-language → filter compiler. | 1 | +"""Search: Postgres FTS + trigram (+ pgvector when embeddings exist) and a deterministic natural-language → filter compiler (v2). | ||
| 2 | 2 | ||||
| 3 | − "open models released in 2026 with more than 100B parameters and 128k context" | 3 | + "open models over 100B released this year" → type model · openness open · params ≥ 1e11 · year_from = year_to = <this year> | ||
| 4 | − → {entity_type: model, openness: open, year_from: 2026, params_min: 1e11, context_min: 128000} | 4 | + "cheapest models with 1M context" → type model · context ≥ 1 000 000 · sort cheapest | ||
| 5 | −""" | 5 | + "reasoning models under $1/M tokens" → type model · reasoning · max_output_price 1.0 | ||
| 6 | + "models that fit in 64GB" → type model · memory_gb 64 (→ ESTIMATED params bound at 4-bit) | ||||
| 7 | + "Anthropic models released since 2025" → type model · organization Anthropic (verified by the API against org aliases) · year_from 2025 | ||||
| 8 | + "open vision models with Apache license" → type model · openness open · modality image · license Apache-2.0 | ||||
| 9 | + "papers introducing MoE models" → type paper · residual "introducing MoE" | ||||
| 10 | + | ||||
| 11 | +Every recognised fragment is listed in `compiled` (filter, label, value, source_span) so the UI can show "Compiled as …"; the rest of the | ||||
| 12 | +text is `residual` (used for full-text search) and its words are echoed in `unrecognised`.""" | ||||
| 6 | from __future__ import annotations | 13 | from __future__ import annotations | ||
| 7 | 14 | ||||
| 8 | import re | 15 | import re | ||
| 9 | from dataclasses import dataclass, field | 16 | from dataclasses import dataclass, field | ||
| 17 | +from datetime import UTC, datetime | ||||
| 10 | from typing import Any | 18 | from typing import Any | ||
| 11 | 19 | ||||
| 12 | from sqlalchemy.ext.asyncio import AsyncConnection | 20 | from sqlalchemy.ext.asyncio import AsyncConnection | ||
| 13 | 21 | ||||
| 14 | from aiatlas.db import fetch_all | 22 | from aiatlas.db import fetch_all | ||
| 23 | +from aiatlas.ontology.licenses import normalize_license | ||||
| 15 | from aiatlas.sdk.extract.numbers import parse_context_length, parse_param_count | 24 | from aiatlas.sdk.extract.numbers import parse_context_length, parse_param_count | ||
| 25 | +from aiatlas.services import hardware_fit as hf | ||||
| 16 | 26 | ||||
| 17 | TYPE_WORDS = { | 27 | TYPE_WORDS = { | ||
| 18 | − "model": ("model", "models", "llm", "llms", "language model"), "company": ("company", "companies", "startup", "startups", "lab", "labs", "organization"), | 28 | + "model": ("model", "models", "llm", "llms", "language model", "language models"), | ||
| 19 | − "paper": ("paper", "papers", "research", "publication", "preprint", "arxiv"), "provider": ("provider", "providers", "inference provider", "api provider"), | 29 | + "company": ("company", "companies", "startup", "startups", "lab", "labs", "organization", "organizations", "organisation"), | ||
| 20 | − "benchmark": ("benchmark", "benchmarks", "leaderboard", "eval", "evals"), "hardware": ("gpu", "gpus", "chip", "chips", "hardware", "accelerator", "npu", "tpu"), | 30 | + "paper": ("paper", "papers", "research", "publication", "publications", "preprint", "preprints", "arxiv"), | ||
| 21 | − "framework": ("framework", "frameworks", "library", "libraries", "runtime", "runtimes"), "dataset": ("dataset", "datasets"), | 31 | + "provider": ("provider", "providers", "inference provider", "api provider", "inference providers"), | ||
| 22 | − "tool": ("tool", "tools", "agent", "agents", "app", "application"), "repository": ("repo", "repos", "repository", "repositories"), | 32 | + "benchmark": ("benchmark", "benchmarks", "leaderboard", "leaderboards", "eval", "evals"), | ||
| 33 | + "hardware": ("gpu", "gpus", "chip", "chips", "hardware", "accelerator", "accelerators", "npu", "tpu"), | ||||
| 34 | + "framework": ("framework", "frameworks", "library", "libraries", "runtime", "runtimes"), | ||||
| 35 | + "dataset": ("dataset", "datasets"), | ||||
| 36 | + "tool": ("tool", "tools", "agent", "agents", "app", "application"), | ||||
| 37 | + "repository": ("repo", "repos", "repository", "repositories"), | ||||
| 23 | } | 38 | } | ||
| 24 | −MODALITY_WORDS = {"vision": "image", "multimodal": "multimodal", "image": "image", "audio": "audio", "speech": "audio", "video": "video", "code": "code", | 39 | +MODALITY_WORDS = {"vision": "image", "multimodal": "image", "image": "image", "images": "image", "audio": "audio", "speech": "audio", "voice": "audio", | ||
| 25 | − "coding": "code", "embedding": "embedding", "embeddings": "embedding"} | 40 | + "video": "video", "embedding": "embedding", "embeddings": "embedding"} | ||
| 41 | +LICENSE_WORDS = {"apache": "Apache-2.0", "apache 2.0": "Apache-2.0", "apache-2.0": "Apache-2.0", "mit": "MIT", "mit license": "MIT", "bsd": "BSD-3-Clause", | ||||
| 42 | + "cc-by": "CC-BY-4.0", "cc by": "CC-BY-4.0", "gpl": "GPL-3.0", "llama license": "Llama-3.1-Community", "gemma license": "Gemma-Terms"} | ||||
| 43 | +SORT_WORDS = {"cheapest": "cheapest", "cheap": "cheapest", "lowest price": "cheapest", "newest": "newest", "latest": "newest", "most recent": "newest", | ||||
| 44 | + "recent": "newest", "largest": "largest", "biggest": "largest", "smallest": "smallest", "best": "best"} | ||||
| 45 | +STOP = {"the", "a", "an", "with", "and", "or", "of", "for", "that", "which", "in", "on", "to", "by", "from", "at", "my", "me", "i", "are", "is", "show", "find", "list", "all", "any"} | ||||
| 46 | +_SIZE = r"(\d+(?:\.\d+)?)\s*([bBmMtT])\b(?:\s*param(?:eter)?s?)?" | ||||
| 26 | 47 | ||||
| 27 | 48 | ||||
| 28 | @dataclass | 49 | @dataclass | ||
@@ -38,90 +59,365 @@ class Query: | |||||
| 38 | modalities: list[str] = field(default_factory=list) | 59 | modalities: list[str] = field(default_factory=list) | ||
| 39 | organization: str | None = None | 60 | organization: str | None = None | ||
| 40 | filters: dict[str, Any] = field(default_factory=dict) | 61 | filters: dict[str, Any] = field(default_factory=dict) | ||
| 62 | + # v2 | ||||
| 63 | + reasoning: bool | None = None | ||||
| 64 | + max_input_price: float | None = None | ||||
| 65 | + max_output_price: float | None = None | ||||
| 66 | + memory_gb: float | None = None | ||||
| 67 | + license_key: str | None = None | ||||
| 68 | + commercial_use: bool | None = None | ||||
| 69 | + days_back: int | None = None | ||||
| 70 | + benchmark: str | None = None | ||||
| 71 | + provider: str | None = None | ||||
| 72 | + sort: str | None = None | ||||
| 73 | + compiled: list[dict[str, Any]] = field(default_factory=list) | ||||
| 74 | + residual: str = "" | ||||
| 75 | + unrecognised: list[str] = field(default_factory=list) | ||||
| 41 | 76 | ||||
| 42 | def as_dict(self) -> dict[str, Any]: | 77 | def as_dict(self) -> dict[str, Any]: | ||
| 43 | − return {k: v for k, v in self.__dict__.items() if v not in (None, "", [], {})} | 78 | + out = {k: v for k, v in self.__dict__.items() if v not in (None, "", [], {}) and k not in ("compiled", "residual", "unrecognised")} | ||
| 79 | + out["compiled"] = self.compiled | ||||
| 80 | + out["residual"] = self.residual | ||||
| 81 | + out["unrecognised"] = self.unrecognised | ||||
| 82 | + out["filters"] = {**self.filters, "residual": self.residual} | ||||
| 83 | + return out | ||||
| 84 | + | ||||
| 85 | + @property | ||||
| 86 | + def has_structure(self) -> bool: | ||||
| 87 | + return any([self.entity_type, self.openness, self.year_from, self.year_to, self.params_min, self.params_max, self.context_min, self.modalities, | ||||
| 88 | + self.organization, self.reasoning, self.max_input_price, self.max_output_price, self.memory_gb, self.license_key, self.commercial_use, | ||||
| 89 | + self.days_back, self.benchmark, self.provider]) | ||||
| 90 | + | ||||
| 91 | + | ||||
| 92 | +class _Compiler: | ||||
| 93 | + def __init__(self, text: str): | ||||
| 94 | + self.text = text | ||||
| 95 | + self.low = text.lower() | ||||
| 96 | + self.consumed: list[tuple[int, int]] = [] | ||||
| 97 | + self.q = Query(text=text) | ||||
| 98 | + | ||||
| 99 | + def take(self, m: re.Match[str], filter_: str, label: str, value: Any) -> None: | ||||
| 100 | + self.consumed.append((m.start(), m.end())) | ||||
| 101 | + self.q.compiled.append({"filter": filter_, "label": label, "value": value, "source_span": self.text[m.start():m.end()]}) | ||||
| 102 | + | ||||
| 103 | + def search(self, pattern: str) -> re.Match[str] | None: | ||||
| 104 | + for m in re.finditer(pattern, self.low, flags=re.IGNORECASE): | ||||
| 105 | + if not any(s <= m.start() < e or s < m.end() <= e for s, e in self.consumed): | ||||
| 106 | + return m | ||||
| 107 | + return None | ||||
| 108 | + | ||||
| 109 | + def residual(self) -> str: | ||||
| 110 | + chars = list(self.text) | ||||
| 111 | + for s, e in self.consumed: | ||||
| 112 | + for i in range(s, e): | ||||
| 113 | + chars[i] = " " | ||||
| 114 | + return " ".join("".join(chars).split()) | ||||
| 44 | 115 | ||||
| 45 | 116 | ||||
| 46 | def compile_query(q: str) -> Query: | 117 | def compile_query(q: str) -> Query: | ||
| 47 | − s = q.strip() | 118 | + c = _Compiler(q.strip()) | ||
| 48 | − low = f" {s.lower()} " | 119 | + out = c.q | ||
| 49 | − out = Query(text=s) | 120 | + year_now = datetime.now(UTC).year | ||
| 50 | − for etype, words in TYPE_WORDS.items(): | 121 | + | ||
| 51 | − if any(f" {w} " in low for w in words): | 122 | + # --- explicit "benchmark <name>" / "provider <name>" | ||
| 52 | − out.entity_type = etype | 123 | + m = c.search(r"\bbenchmark\s+([a-z0-9][\w.\- ]{1,40}?)(?=\s+(?:leaderboard|results|scores|models)\b|$)") | ||
| 53 | − break | 124 | + if m: | ||
| 54 | − if re.search(r"\b(open[- ]?(weight|weights|source)|open models?)\b", low): | 125 | + out.benchmark = m.group(1).strip() | ||
| 126 | + out.entity_type = "benchmark" | ||||
| 127 | + c.take(m, "benchmark", "Benchmark", out.benchmark) | ||||
| 128 | + m = c.search(r"\bprovider\s+([a-z0-9][\w.\- ]{1,40}?)(?=\s+(?:prices?|models)\b|$)") | ||||
| 129 | + if m: | ||||
| 130 | + out.provider = m.group(1).strip() | ||||
| 131 | + out.entity_type = "provider" | ||||
| 132 | + c.take(m, "provider", "Provider", out.provider) | ||||
| 133 | + | ||||
| 134 | + # --- price bounds | ||||
| 135 | + m = c.search(r"(?:under|below|cheaper than|less than|at most|<|≤)\s*\$?\s*(\d+(?:\.\d+)?)\s*(?:usd)?\s*(?:/|per)\s*(?:1\s*)?m(?:illion)?(?:\s*(?:tok(?:ens?)?|output(?:\s*tokens?)?|input(?:\s*tokens?)?))?") | ||||
| 136 | + if m: | ||||
| 137 | + which = "input" if "input" in m.group(0) else "output" | ||||
| 138 | + val = float(m.group(1)) | ||||
| 139 | + if which == "input": | ||||
| 140 | + out.max_input_price = val | ||||
| 141 | + else: | ||||
| 142 | + out.max_output_price = val | ||||
| 143 | + c.take(m, f"max_{which}_price", f"Cheapest {which} price ≤ ${val:g} / 1M tokens", val) | ||||
| 144 | + m = c.search(r"\$\s*(\d+(?:\.\d+)?)\s*(?:/|per)\s*(?:1\s*)?m(?:illion)?\b(?:\s*tok(?:ens?)?)?") | ||||
| 145 | + if m and out.max_output_price is None and out.max_input_price is None: | ||||
| 146 | + out.max_output_price = float(m.group(1)) | ||||
| 147 | + c.take(m, "max_output_price", f"Cheapest output price ≤ ${out.max_output_price:g} / 1M tokens", out.max_output_price) | ||||
| 148 | + | ||||
| 149 | + # --- memory / hardware fit | ||||
| 150 | + m = c.search(r"(?:fits?\s+(?:in|on)|runs?\s+on|run\s+locally\s+on|on\s+my)\s*(?:a\s+|an\s+)?(?:mac(?:book)?(?:\s+\w+)?\s+with\s+)?(\d+(?:\.\d+)?)\s*gb\b(?:\s*(?:of\s+)?(?:ram|memory|vram|unified memory))?") | ||||
| 151 | + if not m: | ||||
| 152 | + m = c.search(r"\b(\d+(?:\.\d+)?)\s*gb\s*(?:of\s+)?(?:ram|memory|vram)\b") | ||||
| 153 | + if m: | ||||
| 154 | + out.memory_gb = float(m.group(1)) | ||||
| 155 | + c.take(m, "memory_gb", f"Fits in {out.memory_gb:g} GB (estimated at 4-bit, 8K context)", out.memory_gb) | ||||
| 156 | + elif c.search(r"\bon my mac\b"): | ||||
| 157 | + m2 = c.search(r"\bon my mac\b") | ||||
| 158 | + assert m2 is not None | ||||
| 55 | out.openness = "open" | 159 | out.openness = "open" | ||
| 56 | − elif re.search(r"\b(proprietary|closed)\b", low): | 160 | + c.take(m2, "openness", "Runs locally → open weights", "open") | ||
| 57 | − out.openness = "proprietary" | 161 | + | ||
| 58 | − m = re.search(r"\b(?:released|launched|published|from|since|after|in)\s+(20\d\d)\b", low) | 162 | + # --- context window | ||
| 163 | + m = c.search(r"(?:context(?:\s+window)?\s*(?:of\s+)?(?:>=|≥|>|over|above|at least|min(?:imum)?)?\s*(\d+(?:\.\d+)?\s*[kKmM])\b(?:\s*tokens?)?)") | ||||
| 164 | + if not m: | ||||
| 165 | + m = c.search(r"(?:with\s+)?(\d+(?:\.\d+)?\s*[kKmM])\b\s*(?:\+\s*)?(?:tokens?\s+)?(?:of\s+)?context(?:\s+window)?") | ||||
| 166 | + if not m: | ||||
| 167 | + m = c.search(r"\blong[\s-]context\b") | ||||
| 168 | + if m: | ||||
| 169 | + out.context_min = 128_000 | ||||
| 170 | + c.take(m, "context_min", "Context ≥ 128K tokens", 128_000) | ||||
| 171 | + m = None | ||||
| 172 | + if m: | ||||
| 173 | + out.context_min = parse_context_length(m.group(1)) | ||||
| 174 | + c.take(m, "context_min", f"Context ≥ {out.context_min:,} tokens", out.context_min) | ||||
| 175 | + | ||||
| 176 | + # --- parameter counts | ||||
| 177 | + m = c.search(r"\bbetween\s+" + _SIZE + r"\s+and\s+" + _SIZE) | ||||
| 178 | + if m: | ||||
| 179 | + out.params_min = parse_param_count(m.group(1) + m.group(2) + " params") | ||||
| 180 | + out.params_max = parse_param_count(m.group(3) + m.group(4) + " params") | ||||
| 181 | + c.take(m, "params_range", f"Parameters between {m.group(1)}{m.group(2).upper()} and {m.group(3)}{m.group(4).upper()}", [out.params_min, out.params_max]) | ||||
| 182 | + m = c.search(r"(?:more than|over|above|at least|>=|≥|>|larger than|bigger than)\s*" + _SIZE) | ||||
| 183 | + if m: | ||||
| 184 | + out.params_min = parse_param_count(m.group(1) + m.group(2) + " params") | ||||
| 185 | + c.take(m, "params_min", f"Parameters ≥ {m.group(1)}{m.group(2).upper()}", out.params_min) | ||||
| 186 | + m = c.search(r"(?:less than|under|below|at most|<=|≤|<|smaller than)\s*" + _SIZE) | ||||
| 187 | + if m: | ||||
| 188 | + out.params_max = parse_param_count(m.group(1) + m.group(2) + " params") | ||||
| 189 | + c.take(m, "params_max", f"Parameters ≤ {m.group(1)}{m.group(2).upper()}", out.params_max) | ||||
| 190 | + m = c.search(r"\b" + _SIZE + r"\s*(?:models?|parameters?)\b") | ||||
| 191 | + if m and out.params_min is None and out.params_max is None: | ||||
| 192 | + n = parse_param_count(m.group(1) + m.group(2) + " params") | ||||
| 193 | + if n: | ||||
| 194 | + out.params_min, out.params_max = int(n * 0.85), int(n * 1.15) | ||||
| 195 | + c.take(m, "params_about", f"About {m.group(1)}{m.group(2).upper()} parameters (±15%)", n) | ||||
| 196 | + | ||||
| 197 | + # --- dates | ||||
| 198 | + m = c.search(r"\b(?:(?:released|launched|published|announced)\s+)?(?:in\s+the\s+)?(?:last|past)\s+(\d+)\s+(day|week|month|year)s?\b") | ||||
| 199 | + if m: | ||||
| 200 | + n, unit = int(m.group(1)), m.group(2) | ||||
| 201 | + out.days_back = n * {"day": 1, "week": 7, "month": 30, "year": 365}[unit] | ||||
| 202 | + c.take(m, "days_back", f"Released in the last {n} {unit}{'s' if n > 1 else ''}", out.days_back) | ||||
| 203 | + m = c.search(r"\b(?:released|launched|published|announced)?\s*(?:this|the current)\s+year\b") | ||||
| 204 | + if m: | ||||
| 205 | + out.year_from = out.year_to = year_now | ||||
| 206 | + c.take(m, "year", f"Released in {year_now}", year_now) | ||||
| 207 | + m = c.search(r"\b(?:released|launched|published|announced)?\s*(?:last|previous)\s+year\b") | ||||
| 208 | + if m: | ||||
| 209 | + out.year_from = out.year_to = year_now - 1 | ||||
| 210 | + c.take(m, "year", f"Released in {year_now - 1}", year_now - 1) | ||||
| 211 | + m = c.search(r"\b(?:(?:released|launched|published|announced)\s+)?(?:since|after|from)\s+(20\d\d)\b") | ||||
| 59 | if m: | 212 | if m: | ||
| 60 | out.year_from = int(m.group(1)) | 213 | out.year_from = int(m.group(1)) | ||
| 61 | − if re.search(r"\bin\s+" + m.group(1), low): | 214 | + c.take(m, "year_from", f"Released since {out.year_from}", out.year_from) | ||
| 62 | − out.year_to = out.year_from | 215 | + m = c.search(r"\b(?:(?:released|launched|published|announced)\s+)?(?:before|until|up to)\s+(20\d\d)\b") | ||
| 63 | − m = re.search(r"\b(?:before|until)\s+(20\d\d)\b", low) | ||||
| 64 | if m: | 216 | if m: | ||
| 65 | out.year_to = int(m.group(1)) | 217 | out.year_to = int(m.group(1)) | ||
| 66 | − m = re.search(r"(?:more than|over|above|>|at least|≥)\s*(\d+(?:\.\d+)?\s*[bBmMtT])\b(?:\s*param)?", s) | 218 | + c.take(m, "year_to", f"Released before {out.year_to}", out.year_to) | ||
| 219 | + m = c.search(r"\b(?:(?:released|launched|published|announced)\s+)?in\s+(20\d\d)\b") | ||||
| 67 | if m: | 220 | if m: | ||
| 68 | − out.params_min = parse_param_count(m.group(1) + " params") | 221 | + out.year_from = out.year_to = int(m.group(1)) | ||
| 69 | − m = re.search(r"(?:less than|under|below|<|at most|≤)\s*(\d+(?:\.\d+)?\s*[bBmMtT])\b", s) | 222 | + c.take(m, "year", f"Released in {m.group(1)}", int(m.group(1))) | ||
| 223 | + m = c.search(r"\b(20\d\d)\b") | ||||
| 224 | + if m and out.year_from is None and out.year_to is None: | ||||
| 225 | + out.year_from = out.year_to = int(m.group(1)) | ||||
| 226 | + c.take(m, "year", f"Released in {m.group(1)}", int(m.group(1))) | ||||
| 227 | + | ||||
| 228 | + # --- licence | ||||
| 229 | + m = c.search(r"\b(apache(?:[\s-]*2(?:\.0)?)?|mit|bsd|cc[\s-]by|gpl|llama|gemma)\s*(?:licen[cs]e[d]?)\b") | ||||
| 70 | if m: | 230 | if m: | ||
| 71 | − out.params_max = parse_param_count(m.group(1) + " params") | 231 | + raw = m.group(1).strip() | ||
| 72 | − m = re.search(r"(?:context|ctx)\s*(?:window\s*)?(?:>|over|above|of at least|at least|≥)?\s*(\d+\s*[kKmM])", s) or \ | 232 | + key = normalize_license(raw) or LICENSE_WORDS.get(raw.lower()) | ||
| 73 | − re.search(r"(\d+\s*[kKmM])\s*(?:\+\s*)?(?:token\s*)?context", s) | 233 | + if key: | ||
| 234 | + out.license_key = key | ||||
| 235 | + c.take(m, "license", f"Licence {key}", key) | ||||
| 236 | + m = c.search(r"\b(?:for\s+)?commercial(?:\s+use|ly usable|-use)?\b(?:\s+allowed|\s+ok)?") | ||||
| 74 | if m: | 237 | if m: | ||
| 75 | − out.context_min = parse_context_length(m.group(1)) | 238 | + out.commercial_use = True | ||
| 239 | + c.take(m, "commercial_use", "Licence allows commercial use", True) | ||||
| 240 | + | ||||
| 241 | + # --- openness | ||||
| 242 | + m = c.search(r"\b(open[- ]?(?:weights?|source|sourced)|open)\b(?=\s+(?:models?|llms?|vision|reasoning|coding|multimodal|\w+\s+models?)\b)") or c.search(r"\bopen[- ]?(?:weights?|source|sourced)\b") | ||||
| 243 | + if m: | ||||
| 244 | + out.openness = "open" | ||||
| 245 | + c.take(m, "openness", "Open weights / open source", "open") | ||||
| 246 | + else: | ||||
| 247 | + m = c.search(r"\b(proprietary|closed(?:[- ]source)?|api[- ]only)\b") | ||||
| 248 | + if m: | ||||
| 249 | + out.openness = "proprietary" | ||||
| 250 | + c.take(m, "openness", "Proprietary (API only)", "proprietary") | ||||
| 251 | + | ||||
| 252 | + # --- reasoning | ||||
| 253 | + m = c.search(r"\b(reasoning|thinking)\b(?=\s+(?:models?|llms?)\b)") or c.search(r"\bwith\s+(reasoning|thinking)\b") | ||||
| 254 | + if m: | ||||
| 255 | + out.reasoning = True | ||||
| 256 | + c.take(m, "reasoning", "Reasoning / thinking models", True) | ||||
| 257 | + | ||||
| 258 | + # --- modalities | ||||
| 76 | for w, mod in MODALITY_WORDS.items(): | 259 | for w, mod in MODALITY_WORDS.items(): | ||
| 77 | − if f" {w} " in low and mod not in out.modalities: | 260 | + m = c.search(rf"\b{w}\b") | ||
| 261 | + if m and mod not in out.modalities: | ||||
| 78 | out.modalities.append(mod) | 262 | out.modalities.append(mod) | ||
| 79 | − m = re.search(r"\b(?:by|from)\s+([A-Z][\w.&-]+(?:\s+[A-Z][\w.&-]+)?)", s) | 263 | + c.take(m, "modality", f"Modality: {mod}", mod) | ||
| 80 | − if m: | 264 | + | ||
| 81 | − out.organization = m.group(1) | 265 | + # --- entity type: the earliest surviving type word wins ("papers introducing MoE models" → paper) | ||
| 82 | − # residual free-text (remove the structured bits) | 266 | + if out.entity_type is None: | ||
| 83 | − residual = re.sub(r"\b(more than|over|above|less than|under|below|at least|at most|released|launched|published|since|after|before|until|with|and|in|from|by|the|a|an|context|window|params?|parameters)\b", " ", s, flags=re.IGNORECASE) | 267 | + best: tuple[int, str, re.Match[str]] | None = None | ||
| 84 | − residual = re.sub(r"\d+(?:\.\d+)?\s*[bBmMkKtT]\b|\b20\d\d\b|[<>≤≥]", " ", residual) | 268 | + for etype, words in TYPE_WORDS.items(): | ||
| 85 | − for words in TYPE_WORDS.values(): | 269 | + for w in sorted(words, key=len, reverse=True): | ||
| 86 | − for w in words: | 270 | + m = c.search(rf"\b{re.escape(w)}\b") | ||
| 87 | − residual = re.sub(rf"\b{re.escape(w)}\b", " ", residual, flags=re.IGNORECASE) | 271 | + if m and (best is None or m.start() < best[0]): | ||
| 88 | − residual = re.sub(r"\b(open[- ]?(weight|weights|source)|proprietary|closed)\b", " ", residual, flags=re.IGNORECASE) | 272 | + best = (m.start(), etype, m) | ||
| 89 | − out.filters["residual"] = " ".join(residual.split()) | 273 | + break | ||
| 274 | + if best: | ||||
| 275 | + out.entity_type = best[1] | ||||
| 276 | + c.take(best[2], "entity_type", f"Type: {best[1]}", best[1]) | ||||
| 277 | + if out.entity_type is None and (out.openness or out.params_min or out.params_max or out.context_min or out.reasoning or out.memory_gb or out.max_output_price or out.max_input_price): | ||||
| 278 | + out.entity_type = "model" | ||||
| 279 | + out.compiled.append({"filter": "entity_type", "label": "Type: model (implied)", "value": "model", "source_span": None}) | ||||
| 280 | + | ||||
| 281 | + # --- sort hints | ||||
| 282 | + for w, s in sorted(SORT_WORDS.items(), key=lambda kv: -len(kv[0])): | ||||
| 283 | + m = c.search(rf"\b{re.escape(w)}\b") | ||||
| 284 | + if m: | ||||
| 285 | + out.sort = s | ||||
| 286 | + c.take(m, "sort", f"Sort: {s}", s) | ||||
| 287 | + break | ||||
| 288 | + | ||||
| 289 | + # --- organization: "by <Org>" / "from <Org>" / "<Org> models" (verified against the organization alias table by the API layer) | ||||
| 290 | + m = c.search(r"\b(?:by|from)\s+([A-Za-z][\w.&-]+(?:\s+[A-Z][\w.&-]+)?)") | ||||
| 291 | + if m and m.group(1).lower() not in STOP and m.group(1).lower() not in {w for ws in TYPE_WORDS.values() for w in ws}: | ||||
| 292 | + out.organization = q[m.start(1):m.end(1)] | ||||
| 293 | + c.take(m, "organization", f"Organization: {out.organization}", out.organization) | ||||
| 294 | + else: | ||||
| 295 | + # "<Org> models …" only at the very start of the query — a capitalised word in the middle ("introducing MoE models") is free text | ||||
| 296 | + m = re.match(r"\s*(?:(?:all|the|show|list|find)\s+)?([A-Z][\w.&-]*(?:\s+[A-Z][\w.&-]*)?)\s+(?:models?|llms?|papers?|releases?)\b", q) | ||||
| 297 | + if m and not any(s <= m.start(1) < e for s, e in c.consumed) and m.group(1).lower() not in STOP and m.group(1).lower() not in {w for ws in TYPE_WORDS.values() for w in ws} \ | ||||
| 298 | + and m.group(1).lower() not in ("open", "reasoning", "vision", "proprietary", "multimodal", "cheapest", "newest", "largest", "best", "new", "moe", "small", "large"): | ||||
| 299 | + out.organization = m.group(1) | ||||
| 300 | + c.consumed.append((m.start(1), m.end(1))) | ||||
| 301 | + out.compiled.append({"filter": "organization", "label": f"Organization: {out.organization}", "value": out.organization, "source_span": m.group(1)}) | ||||
| 302 | + | ||||
| 303 | + residual = c.residual() | ||||
| 304 | + residual = re.sub(r"\b(" + "|".join(sorted(STOP, key=len, reverse=True)) + r")\b", " ", residual, flags=re.IGNORECASE) | ||||
| 305 | + residual = re.sub(r"[<>≤≥$]", " ", residual) | ||||
| 306 | + out.residual = " ".join(residual.split()) | ||||
| 307 | + out.filters["residual"] = out.residual | ||||
| 308 | + out.unrecognised = [w for w in re.findall(r"[\w.\-]+", out.residual) if len(w) > 1] if out.has_structure else [] | ||||
| 90 | return out | 309 | return out | ||
| 91 | 310 | ||||
| 92 | 311 | ||||
| 93 | −async def search_entities(conn: AsyncConnection, q: Query, *, limit: int = 30, offset: int = 0, embedding: list[float] | None = None) -> list[dict[str, Any]]: | 312 | +def params_bound_from_memory(memory_gb: float, quant: str = "4bit") -> int: | ||
| 313 | + """ESTIMATED largest parameter count that fits `memory_gb` at 8K context (inverse of services.hardware_fit).""" | ||||
| 314 | + kv = hf.KV_GB_PER_8K | ||||
| 315 | + avail = max(0.0, memory_gb - hf.RESERVED_GB - kv) | ||||
| 316 | + return int(avail * 1e9 / (hf.BYTES_PER_PARAM.get(quant, 0.5) * hf.OVERHEAD)) | ||||
| 317 | + | ||||
| 318 | + | ||||
| 319 | +# ------------------------------------------------------------------------------------------------------------------ SQL | ||||
| 320 | + | ||||
| 321 | + | ||||
| 322 | +def _num(path: str) -> str: | ||||
| 323 | + return f"(case when {path} ~ '^-?[0-9]+(\\.[0-9]+)?$' then ({path})::double precision end)" | ||||
| 324 | + | ||||
| 325 | + | ||||
| 326 | +def where_for(q: Query) -> tuple[list[str], dict[str, Any]]: | ||||
| 327 | + """Filters shared by `search_entities` and the count query. Every numeric cast is regex-guarded.""" | ||||
| 94 | where = ["e.merged_into is null"] | 328 | where = ["e.merged_into is null"] | ||
| 95 | − params: dict[str, Any] = {"limit": limit, "offset": offset} | 329 | + params: dict[str, Any] = {} | ||
| 96 | if q.entity_type: | 330 | if q.entity_type: | ||
| 97 | − where.append("e.entity_type = :etype") | 331 | + where.append("e.entity_type = :etype" if q.entity_type != "model" else "e.entity_type = 'model'") | ||
| 98 | params["etype"] = q.entity_type | 332 | params["etype"] = q.entity_type | ||
| 99 | if q.openness == "open": | 333 | if q.openness == "open": | ||
| 100 | where.append("(e.attributes->>'openness' in ('open-weights','open-source','open') or e.attributes->>'weights_availability' = 'open')") | 334 | where.append("(e.attributes->>'openness' in ('open-weights','open-source','open') or e.attributes->>'weights_availability' = 'open')") | ||
| 101 | elif q.openness == "proprietary": | 335 | elif q.openness == "proprietary": | ||
| 102 | where.append("e.attributes->>'openness' in ('proprietary','closed')") | 336 | where.append("e.attributes->>'openness' in ('proprietary','closed')") | ||
| 337 | + date_col = "coalesce(e.attributes->>'release_date', e.attributes->>'published_at')" | ||||
| 103 | if q.year_from: | 338 | if q.year_from: | ||
| 104 | − where.append("left(e.attributes->>'release_date', 4) >= :yf") | 339 | + where.append(f"left({date_col}, 4) >= :yf") | ||
| 105 | params["yf"] = str(q.year_from) | 340 | params["yf"] = str(q.year_from) | ||
| 106 | if q.year_to: | 341 | if q.year_to: | ||
| 107 | − where.append("left(e.attributes->>'release_date', 4) <= :yt") | 342 | + where.append(f"left({date_col}, 4) <= :yt") | ||
| 108 | params["yt"] = str(q.year_to) | 343 | params["yt"] = str(q.year_to) | ||
| 344 | + if q.days_back: | ||||
| 345 | + where.append(f"({date_col} >= to_char((now() at time zone 'UTC') - make_interval(days => :days), 'YYYY-MM-DD') or e.first_seen_at >= now() - make_interval(days => :days))") | ||||
| 346 | + params["days"] = int(q.days_back) | ||||
| 347 | + pmax = q.params_max | ||||
| 348 | + if q.memory_gb is not None: | ||||
| 349 | + bound = params_bound_from_memory(q.memory_gb) | ||||
| 350 | + pmax = min(pmax, bound) if pmax else bound | ||||
| 351 | + where.append("e.attributes ? 'parameter_count'") | ||||
| 109 | if q.params_min: | 352 | if q.params_min: | ||
| 110 | − where.append("(e.attributes->>'parameter_count')::double precision >= :pmin") | 353 | + where.append(f"{_num('e.attributes->>' + chr(39) + 'parameter_count' + chr(39))} >= :pmin") | ||
| 111 | params["pmin"] = float(q.params_min) | 354 | params["pmin"] = float(q.params_min) | ||
| 112 | − if q.params_max: | 355 | + if pmax: | ||
| 113 | − where.append("(e.attributes->>'parameter_count')::double precision <= :pmax") | 356 | + where.append(f"{_num('e.attributes->>' + chr(39) + 'parameter_count' + chr(39))} <= :pmax") | ||
| 114 | − params["pmax"] = float(q.params_max) | 357 | + params["pmax"] = float(pmax) | ||
| 115 | if q.context_min: | 358 | if q.context_min: | ||
| 116 | − where.append("(e.attributes->>'context_length')::double precision >= :cmin") | 359 | + where.append(f"{_num('e.attributes->>' + chr(39) + 'context_length' + chr(39))} >= :cmin") | ||
| 117 | params["cmin"] = float(q.context_min) | 360 | params["cmin"] = float(q.context_min) | ||
| 118 | for i, mod in enumerate(q.modalities): | 361 | for i, mod in enumerate(q.modalities): | ||
| 119 | − where.append(f"e.attributes->'modalities' ? :mod{i}") | 362 | + where.append(f"(e.attributes->'modalities' ? :mod{i} or e.attributes->'modalities_input' ? :mod{i} or e.attributes->'modalities_output' ? :mod{i}" | ||
| 363 | + + (" or e.attributes->>'vision' = 'true'" if mod == "image" else "") + ")") | ||||
| 120 | params[f"mod{i}"] = mod | 364 | params[f"mod{i}"] = mod | ||
| 365 | + if q.reasoning: | ||||
| 366 | + where.append("e.attributes->>'reasoning' = 'true'") | ||||
| 367 | + if q.max_output_price is not None: | ||||
| 368 | + where.append("exists (select 1 from prices p where p.model_id = e.id and p.valid_to is null and p.output_per_mtok > 0 and p.output_per_mtok <= :maxout)") | ||||
| 369 | + params["maxout"] = float(q.max_output_price) | ||||
| 370 | + if q.max_input_price is not None: | ||||
| 371 | + where.append("exists (select 1 from prices p where p.model_id = e.id and p.valid_to is null and p.input_per_mtok > 0 and p.input_per_mtok <= :maxin)") | ||||
| 372 | + params["maxin"] = float(q.max_input_price) | ||||
| 373 | + if q.license_key: | ||||
| 374 | + where.append("(e.attributes->>'license_key' = :lic or lower(e.attributes->>'license') = any(cast(:lic_raw as text[])))") | ||||
| 375 | + params["lic"] = q.license_key | ||||
| 376 | + params["lic_raw"] = _raw_license_labels(q.license_key) | ||||
| 377 | + if q.commercial_use: | ||||
| 378 | + keys = _commercial_keys() | ||||
| 379 | + where.append("(e.attributes->>'license_key' = any(cast(:ckeys as text[])) or lower(e.attributes->>'license') = any(cast(:craw as text[])))") | ||||
| 380 | + params["ckeys"] = keys | ||||
| 381 | + params["craw"] = [lbl for k in keys for lbl in _raw_license_labels(k)] | ||||
| 121 | if q.organization: | 382 | if q.organization: | ||
| 122 | − where.append("exists (select 1 from entities o where o.id = e.organization_id and o.canonical_name ilike :org)") | 383 | + where.append("exists (select 1 from entities o where o.id = e.organization_id and (o.canonical_name ilike :org or o.slug = :orgslug " | ||
| 123 | − params["org"] = f"%{q.organization}%" | 384 | + "or exists (select 1 from entity_aliases a where a.entity_id = o.id and a.alias ilike :org)))") | ||
| 124 | − text = (q.filters.get("residual") or "").strip() or ("" if any([q.entity_type, q.openness, q.year_from, q.params_min, q.context_min, q.modalities]) else q.text) | 385 | + params["org"] = q.organization | ||
| 386 | + params["orgslug"] = q.organization.lower().replace(" ", "-") | ||||
| 387 | + return where, params | ||||
| 388 | + | ||||
| 389 | + | ||||
| 390 | +def _raw_license_labels(key: str) -> list[str]: | ||||
| 391 | + from aiatlas.ontology.licenses import LICENSES | ||||
| 392 | + | ||||
| 393 | + info = LICENSES.get(key) | ||||
| 394 | + if not info: | ||||
| 395 | + return [key.lower()] | ||||
| 396 | + return sorted({key.lower(), *(a.lower() for a in info.aliases), *( [info.spdx.lower()] if info.spdx else [])}) | ||||
| 397 | + | ||||
| 398 | + | ||||
| 399 | +def _commercial_keys() -> list[str]: | ||||
| 400 | + from aiatlas.ontology.licenses import LICENSES | ||||
| 401 | + | ||||
| 402 | + return sorted(k for k, v in LICENSES.items() if v.commercial_use is True and v.weights_downloadable) | ||||
| 403 | + | ||||
| 404 | + | ||||
| 405 | +SORT_SQL = { | ||||
| 406 | + "cheapest": "(select min(p.output_per_mtok) from prices p where p.model_id = e.id and p.valid_to is null and p.output_per_mtok > 0) asc nulls last", | ||||
| 407 | + "newest": "coalesce(e.attributes->>'release_date', e.attributes->>'published_at', '') desc", | ||||
| 408 | + "largest": _num("e.attributes->>'parameter_count'") + " desc nulls last", | ||||
| 409 | + "smallest": _num("e.attributes->>'parameter_count'") + " asc nulls last", | ||||
| 410 | +} | ||||
| 411 | + | ||||
| 412 | + | ||||
| 413 | +async def search_entities(conn: AsyncConnection, q: Query, *, limit: int = 30, offset: int = 0, embedding: list[float] | None = None) -> list[dict[str, Any]]: | ||||
| 414 | + where, params = where_for(q) | ||||
| 415 | + params.update({"limit": limit, "offset": offset}) | ||||
| 416 | + text = (q.residual or q.filters.get("residual") or "").strip() or ("" if q.has_structure else q.text) | ||||
| 417 | + if q.benchmark and q.entity_type == "benchmark": | ||||
| 418 | + text = q.benchmark | ||||
| 419 | + if q.provider and q.entity_type == "provider": | ||||
| 420 | + text = q.provider | ||||
| 125 | rank = "coalesce((e.quality->>'score')::float, 0) / 100.0" | 421 | rank = "coalesce((e.quality->>'score')::float, 0) / 100.0" | ||
| 126 | if text: | 422 | if text: | ||
| 127 | params["q"] = text | 423 | params["q"] = text | ||
@@ -138,13 +434,25 @@ async def search_entities(conn: AsyncConnection, q: Query, *, limit: int = 30, o | |||||
| 138 | join = "left join entity_embeddings x on x.entity_id = e.id" | 434 | join = "left join entity_embeddings x on x.entity_id = e.id" | ||
| 139 | else: | 435 | else: | ||
| 140 | join = "" | 436 | join = "" | ||
| 437 | + order = SORT_SQL.get(q.sort or "", None) | ||||
| 438 | + order_sql = f"{order}, rank desc" if order and q.entity_type in (None, "model") else "rank desc, e.updated_at desc" | ||||
| 141 | sql = f"""select e.id, e.entity_type, e.canonical_name, e.slug, left(e.description, 240) as description, e.status, e.attributes, e.quality, | 439 | sql = f"""select e.id, e.entity_type, e.canonical_name, e.slug, left(e.description, 240) as description, e.status, e.attributes, e.quality, | ||
| 142 | o.canonical_name as organization_name, o.slug as organization_slug, {rank} as rank | 440 | o.canonical_name as organization_name, o.slug as organization_slug, {rank} as rank | ||
| 143 | from entities e left join entities o on o.id = e.organization_id {join} | 441 | from entities e left join entities o on o.id = e.organization_id {join} | ||
| 144 | − where {' and '.join(where)} order by rank desc, e.updated_at desc limit :limit offset :offset""" | 442 | + where {' and '.join(where)} order by {order_sql} limit :limit offset :offset""" | ||
| 145 | return await fetch_all(conn, sql, **params) | 443 | return await fetch_all(conn, sql, **params) | ||
| 146 | 444 | ||||
| 147 | 445 | ||||
| 446 | +async def verify_organization(conn: AsyncConnection, name: str | None) -> dict[str, Any] | None: | ||||
| 447 | + """The compiler only *proposes* an organization; it counts only when it matches an organization-like entity or one of its aliases.""" | ||||
| 448 | + if not name: | ||||
| 449 | + return None | ||||
| 450 | + rows = await fetch_all(conn, """select e.id, e.slug, e.canonical_name from entities e where e.entity_type in ('company','organization','lab','university') and e.merged_into is null | ||||
| 451 | + and (e.canonical_name ilike :n or e.slug = :s or exists (select 1 from entity_aliases a where a.entity_id = e.id and a.alias ilike :n)) | ||||
| 452 | + order by length(e.canonical_name) limit 1""", n=name, s=name.lower().replace(" ", "-")) | ||||
| 453 | + return rows[0] if rows else None | ||||
| 454 | + | ||||
| 455 | + | ||||
| 148 | async def suggest(conn: AsyncConnection, prefix: str, *, limit: int = 8) -> list[dict[str, Any]]: | 456 | async def suggest(conn: AsyncConnection, prefix: str, *, limit: int = 8) -> list[dict[str, Any]]: | ||
| 149 | return await fetch_all(conn, """select e.id, e.entity_type, e.canonical_name, e.slug, o.canonical_name as organization_name | 457 | return await fetch_all(conn, """select e.id, e.entity_type, e.canonical_name, e.slug, o.canonical_name as organization_name | ||
| 150 | from entities e left join entities o on o.id = e.organization_id | 458 | from entities e left join entities o on o.id = e.organization_id | ||
@@ -153,4 +461,4 @@ async def suggest(conn: AsyncConnection, prefix: str, *, limit: int = 8) -> list | |||||
| 153 | coalesce((e.quality->>'score')::float, 0) desc, length(e.canonical_name) limit :n""", p=f"{prefix}%", n=limit) | 461 | coalesce((e.quality->>'score')::float, 0) desc, length(e.canonical_name) limit :n""", p=f"{prefix}%", n=limit) | ||
| 154 | 462 | ||||
| 155 | 463 | ||||
| 156 | −__all__ = ["Query", "compile_query", "search_entities", "suggest"] | 464 | +__all__ = ["Query", "compile_query", "params_bound_from_memory", "search_entities", "suggest", "verify_organization", "where_for"] | ||
modified
src/aiatlas/services/stats.py
+36 −4
@@ -1,4 +1,8 @@ | |||||
| 1 | −"""Live counters and aggregate statistics — always computed from the database, never hardcoded.""" | 1 | +"""Live counters and aggregate statistics — always computed from the database, never hardcoded. | ||
| 2 | + | ||||
| 3 | +API 1.1 semantics: `entities.model` counts CANONICAL model releases (entity_type 'model', not merged) — artifacts (checkpoints, quantisations, | ||||
| 4 | +conversions) are `entities.artifact`, folded evaluation variants are excluded through `merged_into`. `organizations_total` is the number | ||||
| 5 | +shown by /companies (company + organization + lab + university). `definitions` says how every counter is counted.""" | ||||
| 2 | from __future__ import annotations | 6 | from __future__ import annotations | ||
| 3 | 7 | ||||
| 4 | from typing import Any | 8 | from typing import Any | ||
@@ -8,6 +12,25 @@ from sqlalchemy.ext.asyncio import AsyncConnection | |||||
| 8 | from aiatlas.db import execute, fetch_all, fetch_one, jsonb, transaction | 12 | from aiatlas.db import execute, fetch_all, fetch_one, jsonb, transaction | ||
| 9 | from aiatlas.sdk.archive import archive_size | 13 | from aiatlas.sdk.archive import archive_size | ||
| 10 | 14 | ||||
| 15 | +ORG_TYPES = ("company", "organization", "lab", "university") | ||||
| 16 | + | ||||
| 17 | +DEFINITIONS: dict[str, str] = { | ||||
| 18 | + "models": "Canonical model releases (artifacts, quantisations, conversions and folded evaluation variants excluded; merged duplicates excluded).", | ||||
| 19 | + "artifacts": "Checkpoints, quantisations, conversions and packagings of a canonical model (entity_type 'artifact').", | ||||
| 20 | + "model_families": "Model families (Llama 4, Qwen3.6, Claude…) grouping canonical releases.", | ||||
| 21 | + "organizations_total": "Companies + organizations + labs + universities, merged duplicates excluded — the same universe as /companies.", | ||||
| 22 | + "entities_total": "All live entities of every type (merged duplicates excluded).", | ||||
| 23 | + "change_events": "All change events ever recorded, including the initial back-filled corpus.", | ||||
| 24 | + "change_events_24h": "Events OBSERVED in the last 24 hours (legacy counter — includes back-filled history when a connector first runs).", | ||||
| 25 | + "change_events_live_24h": "Events that OCCURRED in the last 24 hours, excluding back-fill and source-document changes — what actually happened today.", | ||||
| 26 | + "change_events_7d": "Events observed in the last 7 days (legacy counter).", | ||||
| 27 | + "benchmark_results": "Current benchmark result rows (one per model × benchmark × metric × configuration run).", | ||||
| 28 | + "prices_current": "Live provider price offers (model × provider × provider model id).", | ||||
| 29 | + "claims_current": "Current temporal claims (one per entity × property).", | ||||
| 30 | + "relations": "Live relations in the knowledge graph.", | ||||
| 31 | + "sources": "Enabled sources (websites, registries, leaderboards) crawled by AI Atlas connectors.", | ||||
| 32 | +} | ||||
| 33 | + | ||||
| 11 | 34 | ||||
| 12 | async def live_counts(conn: AsyncConnection) -> dict[str, Any]: | 35 | async def live_counts(conn: AsyncConnection) -> dict[str, Any]: | ||
| 13 | by_type = await fetch_all(conn, "select entity_type, count(*) as n from entities where merged_into is null group by 1") | 36 | by_type = await fetch_all(conn, "select entity_type, count(*) as n from entities where merged_into is null group by 1") | ||
@@ -22,8 +45,12 @@ async def live_counts(conn: AsyncConnection) -> dict[str, Any]: | |||||
| 22 | (select count(*) from relations where valid_to is null) as relations, | 45 | (select count(*) from relations where valid_to is null) as relations, | ||
| 23 | (select count(*) from change_events) as change_events, | 46 | (select count(*) from change_events) as change_events, | ||
| 24 | (select count(*) from change_events where observed_at > now() - interval '24 hours') as change_events_24h, | 47 | (select count(*) from change_events where observed_at > now() - interval '24 hours') as change_events_24h, | ||
| 48 | + (select count(*) from change_events where is_backfill = false and event_type <> 'DOCUMENT_CHANGED' | ||||
| 49 | + and occurred_at > now() - interval '24 hours') as change_events_live_24h, | ||||
| 50 | + (select count(*) from change_events where is_backfill = false and event_type <> 'DOCUMENT_CHANGED' | ||||
| 51 | + and occurred_at > now() - interval '7 days') as change_events_live_7d, | ||||
| 25 | (select count(*) from change_events where observed_at > now() - interval '7 days') as change_events_7d, | 52 | (select count(*) from change_events where observed_at > now() - interval '7 days') as change_events_7d, | ||
| 26 | − (select count(*) from benchmark_results where valid_to is null) as benchmark_results, | 53 | + (select count(*) from benchmark_results where valid_to is null and is_current) as benchmark_results, | ||
| 27 | (select count(*) from prices where valid_to is null) as prices_current, | 54 | (select count(*) from prices where valid_to is null) as prices_current, | ||
| 28 | (select count(*) from prices) as prices_total, | 55 | (select count(*) from prices) as prices_total, | ||
| 29 | (select count(*) from jobs where status = 'queued') as jobs_queued, | 56 | (select count(*) from jobs where status = 'queued') as jobs_queued, | ||
@@ -34,7 +61,12 @@ async def live_counts(conn: AsyncConnection) -> dict[str, Any]: | |||||
| 34 | (select max(observed_at) from snapshots) as last_snapshot_at, | 61 | (select max(observed_at) from snapshots) as last_snapshot_at, | ||
| 35 | (select max(observed_at) from change_events) as last_event_at, | 62 | (select max(observed_at) from change_events) as last_event_at, | ||
| 36 | (select min(first_seen_at) from entities) as first_entity_at""") | 63 | (select min(first_seen_at) from entities) as first_entity_at""") | ||
| 37 | − return {"entities": counts, "entities_total": sum(counts.values()), **{k: (int(v) if isinstance(v, int) else v) for k, v in (infra or {}).items()}} | 64 | + out = {"entities": counts, "entities_total": sum(counts.values()), **{k: (int(v) if isinstance(v, int) else v) for k, v in (infra or {}).items()}} | ||
| 65 | + out["organizations_total"] = sum(counts.get(t, 0) for t in ORG_TYPES) | ||||
| 66 | + out["artifacts"] = counts.get("artifact", 0) | ||||
| 67 | + out["model_families"] = counts.get("model_family", 0) | ||||
| 68 | + out["definitions"] = DEFINITIONS | ||||
| 69 | + return out | ||||
| 38 | 70 | ||||
| 39 | 71 | ||||
| 40 | async def compute_stats() -> dict[str, Any]: | 72 | async def compute_stats() -> dict[str, Any]: | ||
@@ -50,4 +82,4 @@ async def history(conn: AsyncConnection, days: int = 90) -> list[dict[str, Any]] | |||||
| 50 | from stats_snapshots where computed_at > now() - make_interval(days => :d) order by 1 desc, computed_at desc""", d=days) | 82 | from stats_snapshots where computed_at > now() - make_interval(days => :d) order by 1 desc, computed_at desc""", d=days) | ||
| 51 | 83 | ||||
| 52 | 84 | ||||
| 53 | −__all__ = ["compute_stats", "history", "live_counts"] | 85 | +__all__ = ["DEFINITIONS", "ORG_TYPES", "compute_stats", "history", "live_counts"] | ||
added
tests/test_api_services.py
+136 −0
@@ -0,0 +1,136 @@ | |||||
| 1 | +"""Unit tests for the pure API 1.1 services: Pareto frontier, cost arithmetic, search compiler v2, hardware fit breakdown. No database.""" | ||||
| 2 | +from __future__ import annotations | ||||
| 3 | + | ||||
| 4 | +from datetime import UTC, datetime | ||||
| 5 | + | ||||
| 6 | +import pytest | ||||
| 7 | + | ||||
| 8 | +from aiatlas.services import hardware_fit as hf | ||||
| 9 | +from aiatlas.services.cost import compute_cost, context_fill_cost | ||||
| 10 | +from aiatlas.services.pareto import is_dominated, pareto_frontier | ||||
| 11 | +from aiatlas.services.search import compile_query, params_bound_from_memory | ||||
| 12 | + | ||||
| 13 | +# ------------------------------------------------------------------------------------------------------------------ pareto | ||||
| 14 | + | ||||
| 15 | + | ||||
| 16 | +def test_pareto_frontier_is_efficient_and_keeps_ties() -> None: | ||||
| 17 | + pts = [ | ||||
| 18 | + {"id": "a", "x": 1.0, "y": 50}, # cheap, low quality → efficient | ||||
| 19 | + {"id": "b", "x": 2.0, "y": 70}, # efficient | ||||
| 20 | + {"id": "c", "x": 3.0, "y": 65}, # dominated by b (more expensive, lower) | ||||
| 21 | + {"id": "d", "x": 5.0, "y": 90}, # efficient | ||||
| 22 | + {"id": "e", "x": 5.0, "y": 90}, # exact tie with d → kept | ||||
| 23 | + {"id": "f", "x": 6.0, "y": 90}, # dominated by d (same y, more expensive) | ||||
| 24 | + {"id": "g", "x": None, "y": 99}, # ignored | ||||
| 25 | + ] | ||||
| 26 | + front = pareto_frontier(pts) | ||||
| 27 | + assert front == ["a", "b", "d", "e"] | ||||
| 28 | + valid = [p for p in pts if p["x"] is not None] | ||||
| 29 | + for p in valid: | ||||
| 30 | + assert is_dominated(p, valid) == (p["id"] not in front), p["id"] | ||||
| 31 | + | ||||
| 32 | + | ||||
| 33 | +def test_pareto_minimise_y() -> None: | ||||
| 34 | + pts = [{"id": "a", "x": 1, "y": 10}, {"id": "b", "x": 2, "y": 5}, {"id": "c", "x": 3, "y": 7}] | ||||
| 35 | + assert pareto_frontier(pts, maximize_y=False) == ["a", "b"] | ||||
| 36 | + | ||||
| 37 | + | ||||
| 38 | +# ------------------------------------------------------------------------------------------------------------------ cost | ||||
| 39 | + | ||||
| 40 | + | ||||
| 41 | +def test_cost_arithmetic_standard() -> None: | ||||
| 42 | + c = compute_cost({"input": 3.0, "output": 15.0}, input_tokens=1000, output_tokens=500, requests_per_day=1000) | ||||
| 43 | + assert c["per_request"] == pytest.approx(1000 * 3 / 1e6 + 500 * 15 / 1e6) | ||||
| 44 | + assert c["daily"] == pytest.approx(c["per_request"] * 1000) | ||||
| 45 | + assert c["monthly"] == pytest.approx(c["daily"] * 30) and c["annual"] == pytest.approx(c["daily"] * 365) | ||||
| 46 | + assert c["notes"] == [] | ||||
| 47 | + | ||||
| 48 | + | ||||
| 49 | +def test_cost_cached_and_batch_with_notes() -> None: | ||||
| 50 | + c = compute_cost({"input": 3.0, "output": 15.0, "cached_input": 0.3}, input_tokens=1_000_000, output_tokens=0, cached_share=0.5) | ||||
| 51 | + assert c["effective_input_per_mtok"] == pytest.approx(1.65) and c["per_request"] == pytest.approx(1.65) | ||||
| 52 | + c2 = compute_cost({"input": 3.0, "output": 15.0}, input_tokens=1_000_000, output_tokens=0, cached_share=0.5) | ||||
| 53 | + assert c2["per_request"] == pytest.approx(3.0) and any("cached input price not published" in n for n in c2["notes"]) | ||||
| 54 | + c3 = compute_cost({"input": 3.0, "output": 15.0, "batch_input": 1.5, "batch_output": 7.5}, input_tokens=1_000_000, output_tokens=1_000_000, batch=True) | ||||
| 55 | + assert c3["per_request"] == pytest.approx(9.0) | ||||
| 56 | + c4 = compute_cost({"input": 3.0, "output": 15.0}, input_tokens=1_000_000, output_tokens=1_000_000, batch=True) | ||||
| 57 | + assert c4["per_request"] == pytest.approx(18.0) and any("batch prices not published" in n for n in c4["notes"]) | ||||
| 58 | + missing = compute_cost({"input": None, "output": 15.0}, input_tokens=10, output_tokens=10) | ||||
| 59 | + assert missing["per_request"] is None and "input price unavailable" in missing["notes"] | ||||
| 60 | + assert context_fill_cost(2.0, 1_000_000) == 2.0 and context_fill_cost(None, 10) is None | ||||
| 61 | + | ||||
| 62 | + | ||||
| 63 | +# ------------------------------------------------------------------------------------------------------------------ search compiler v2 | ||||
| 64 | + | ||||
| 65 | +YEAR = datetime.now(UTC).year | ||||
| 66 | + | ||||
| 67 | + | ||||
| 68 | +def _d(q: str) -> dict: | ||||
| 69 | + d = compile_query(q).as_dict() | ||||
| 70 | + d.pop("compiled"), d.pop("text"), d.pop("filters") | ||||
| 71 | + return d | ||||
| 72 | + | ||||
| 73 | + | ||||
| 74 | +def test_compile_open_models_over_100b_this_year() -> None: | ||||
| 75 | + d = _d("open models over 100B released this year") | ||||
| 76 | + assert d["entity_type"] == "model" and d["openness"] == "open" and d["params_min"] == 100_000_000_000 | ||||
| 77 | + assert d["year_from"] == YEAR and d["year_to"] == YEAR and d["residual"] == "" | ||||
| 78 | + | ||||
| 79 | + | ||||
| 80 | +def test_compile_cheapest_1m_context() -> None: | ||||
| 81 | + d = _d("cheapest models with 1M context") | ||||
| 82 | + assert d["entity_type"] == "model" and d["context_min"] == 1_000_000 and d["sort"] == "cheapest" and d["residual"] == "" | ||||
| 83 | + | ||||
| 84 | + | ||||
| 85 | +def test_compile_reasoning_under_price() -> None: | ||||
| 86 | + d = _d("reasoning models under $1/M tokens") | ||||
| 87 | + assert d["reasoning"] is True and d["max_output_price"] == 1.0 and d["entity_type"] == "model" | ||||
| 88 | + | ||||
| 89 | + | ||||
| 90 | +def test_compile_fits_in_memory() -> None: | ||||
| 91 | + d = _d("models that fit in 64GB") | ||||
| 92 | + assert d["memory_gb"] == 64.0 and d["entity_type"] == "model" | ||||
| 93 | + assert 0 < params_bound_from_memory(64.0) < 200e9 | ||||
| 94 | + | ||||
| 95 | + | ||||
| 96 | +def test_compile_org_since_year() -> None: | ||||
| 97 | + d = _d("Anthropic models released since 2025") | ||||
| 98 | + assert d["organization"] == "Anthropic" and d["year_from"] == 2025 and "year_to" not in d and d["entity_type"] == "model" | ||||
| 99 | + | ||||
| 100 | + | ||||
| 101 | +def test_compile_open_vision_apache() -> None: | ||||
| 102 | + d = _d("open vision models with Apache license") | ||||
| 103 | + assert d["openness"] == "open" and d["modalities"] == ["image"] and d["license_key"] == "Apache-2.0" | ||||
| 104 | + | ||||
| 105 | + | ||||
| 106 | +def test_compile_papers_free_text() -> None: | ||||
| 107 | + d = _d("papers introducing MoE models") | ||||
| 108 | + assert d["entity_type"] == "paper" and "organization" not in d and "MoE" in d["residual"] | ||||
| 109 | + | ||||
| 110 | + | ||||
| 111 | +def test_compile_misc_rules() -> None: | ||||
| 112 | + assert _d("models between 7B and 70B")["params_min"] == 7_000_000_000 and _d("models between 7B and 70B")["params_max"] == 70_000_000_000 | ||||
| 113 | + assert _d("benchmark gpqa")["benchmark"] == "gpqa" and _d("provider groq")["provider"] == "groq" | ||||
| 114 | + d = _d("open source models with commercial use released in the last 30 days") | ||||
| 115 | + assert d["commercial_use"] is True and d["days_back"] == 30 and d["residual"] == "" | ||||
| 116 | + assert _d("largest proprietary models")["sort"] == "largest" and _d("largest proprietary models")["openness"] == "proprietary" | ||||
| 117 | + plain = compile_query("claude") | ||||
| 118 | + assert plain.residual == "claude" and plain.unrecognised == [] and not plain.has_structure | ||||
| 119 | + comp = compile_query("open models over 100B").compiled | ||||
| 120 | + assert {c["filter"] for c in comp} >= {"openness", "params_min", "entity_type"} and all("label" in c and "source_span" in c for c in comp) | ||||
| 121 | + | ||||
| 122 | + | ||||
| 123 | +# ------------------------------------------------------------------------------------------------------------------ hardware fit | ||||
| 124 | + | ||||
| 125 | + | ||||
| 126 | +def test_fit_detailed_observed_vs_estimated_and_kv() -> None: | ||||
| 127 | + est = hf.fit_detailed({"parameter_count": 70e9}, 64, quant="4bit", context=8192) | ||||
| 128 | + assert est and est["estimated"] is True and est["breakdown"]["weights_source"] == "estimated" and est["breakdown"]["kv_cache_method"] == "heuristic" | ||||
| 129 | + obs = hf.fit_detailed({"parameter_count": 70e9}, 64, quant="4bit", context=8192, observed_size_gb=40.0) | ||||
| 130 | + assert obs and obs["breakdown"]["weights_gb"] == 40.0 and obs["breakdown"]["weights_source"] == "observed" | ||||
| 131 | + arch = hf.fit_detailed({"parameter_count": 8e9, "num_hidden_layers": 32, "num_key_value_heads": 8, "head_dim": 128}, 24, quant="4bit", context=8192) | ||||
| 132 | + assert arch and arch["breakdown"]["kv_cache_method"] == "architecture" and arch["breakdown"]["kv_cache_gb"] == pytest.approx(2 * 32 * 8 * 128 * 2 * 8192 / 1e9, rel=1e-3) | ||||
| 133 | + multi = hf.fit_detailed({"parameter_count": 400e9}, 80, quant="8bit", gpu_count=8) | ||||
| 134 | + assert multi and multi["device"]["total_memory_gb"] == 640 and "multi_gpu_note" in multi | ||||
| 135 | + assert hf.fit_detailed({}, 64) is None # nothing estimated from thin air | ||||
| 136 | + assert hf.fit(70e9, 64, "4bit", 8192)["estimated"] is True # v1 helper unchanged | ||||
added
tests/test_api_v11.py
+284 −0
@@ -0,0 +1,284 @@ | |||||
| 1 | +"""API 1.1 contract tests against the live local database (docs/API.md). Read-only. Counts are never asserted as fixed numbers.""" | ||||
| 2 | +from __future__ import annotations | ||||
| 3 | + | ||||
| 4 | +from collections.abc import AsyncIterator | ||||
| 5 | + | ||||
| 6 | +import pytest | ||||
| 7 | +from httpx import ASGITransport, AsyncClient | ||||
| 8 | + | ||||
| 9 | +from aiatlas import db | ||||
| 10 | +from aiatlas.api.main import app | ||||
| 11 | +from aiatlas.config import settings | ||||
| 12 | +from aiatlas.services import cache | ||||
| 13 | + | ||||
| 14 | +ADMIN = {"x-aia-admin-token": settings.admin_token or "dev-admin-token"} | ||||
| 15 | +MODEL, MODEL_B, BENCH = "claude-opus-5", "claude-sonnet-5", "gpqa" | ||||
| 16 | + | ||||
| 17 | + | ||||
| 18 | +@pytest.fixture | ||||
| 19 | +async def client() -> AsyncIterator[AsyncClient]: | ||||
| 20 | + await cache.cache_invalidate() | ||||
| 21 | + async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test", timeout=120) as c: | ||||
| 22 | + yield c | ||||
| 23 | + await cache.close() | ||||
| 24 | + await db.dispose() | ||||
| 25 | + | ||||
| 26 | + | ||||
| 27 | +async def test_models_canonical_universe_and_artifacts(client: AsyncClient) -> None: | ||||
| 28 | + r = await client.get("/api/v1/models", params={"limit": 200}) | ||||
| 29 | + assert r.status_code == 200 | ||||
| 30 | + body = r.json() | ||||
| 31 | + assert body["universe"] == "canonical models" and body["items"] | ||||
| 32 | + assert all(it["entity_type"] == "model" for it in body["items"]) | ||||
| 33 | + assert all("identity_confidence" in it for it in body["items"]) | ||||
| 34 | + r2 = await client.get("/api/v1/models", params={"limit": 200, "include": "artifacts"}) | ||||
| 35 | + assert r2.status_code == 200 and r2.json()["universe"] == "models+artifacts" and r2.json()["total"] >= body["total"] | ||||
| 36 | + for it in r2.json()["items"]: | ||||
| 37 | + if it["entity_type"] == "artifact": | ||||
| 38 | + assert "canonical" in it and "artifact_kind" in it | ||||
| 39 | + facets = (await client.get("/api/v1/models", params={"facets": 1, "limit": 1})).json()["facets"] | ||||
| 40 | + assert {"families", "licenses", "trust", "definitions"} <= set(facets) | ||||
| 41 | + assert all({"value", "label", "category", "count"} <= set(x) for x in facets["licenses"]) | ||||
| 42 | + lic = await client.get("/api/v1/models", params={"license": "Apache 2.0", "limit": 5}) # raw label → canonical key | ||||
| 43 | + assert lic.status_code == 200 | ||||
| 44 | + assert (await client.get("/api/v1/models", params={"family": "Claude", "limit": 2})).status_code == 200 | ||||
| 45 | + | ||||
| 46 | + | ||||
| 47 | +async def test_model_detail_v11_blocks(client: AsyncClient) -> None: | ||||
| 48 | + d = (await client.get(f"/api/v1/models/{MODEL}")).json() | ||||
| 49 | + for block in ("family", "artifacts", "deployments", "identity", "openness", "version_history", "benchmarks"): | ||||
| 50 | + assert block in d, block | ||||
| 51 | + assert {"items", "total"} <= set(d["artifacts"]) and {"canonical_model", "official_checkpoints", "third_party_artifacts", "provider_deployments", "api_aliases"} <= set(d["identity"]) | ||||
| 52 | + assert d["openness"]["category"] in ("open-source", "open-weights", "restricted-weights", "proprietary", "unknown") and "dimensions" in d["openness"] | ||||
| 53 | + assert all({"property", "transitions"} <= set(v) for v in d["version_history"]) | ||||
| 54 | + if d["deployments"]: | ||||
| 55 | + dep = d["deployments"][0] | ||||
| 56 | + assert {"model", "provider", "prices", "features", "status", "valid_from"} <= set(dep) and {"input", "output", "native_units"} <= set(dep["prices"]) | ||||
| 57 | + assert {"items", "total_rows", "note"} <= set(d["benchmarks"]) | ||||
| 58 | + for b in d["benchmarks"]["items"]: | ||||
| 59 | + for m in b["metrics"]: | ||||
| 60 | + for g in m["groups"]: | ||||
| 61 | + assert {"config_key", "comparability_group", "n_rows", "best"} <= set(g) and "trust_level" in g["best"] | ||||
| 62 | + | ||||
| 63 | + | ||||
| 64 | +async def test_leaderboard_one_row_per_model(client: AsyncClient) -> None: | ||||
| 65 | + r = await client.get(f"/api/v1/benchmarks/{BENCH}/leaderboard", params={"limit": 1000}) | ||||
| 66 | + assert r.status_code == 200 | ||||
| 67 | + body = r.json() | ||||
| 68 | + assert body["group"] and {"metric", "config_key", "label", "n", "model_count"} <= set(body["group"]) and body["groups"] | ||||
| 69 | + ids = [it["model"]["id"] for it in body["items"]] | ||||
| 70 | + assert ids and len(ids) == len(set(ids)), "one row per model" | ||||
| 71 | + ranks = [it["rank"] for it in body["items"]] | ||||
| 72 | + assert ranks[0] == 1 and ranks == sorted(ranks) | ||||
| 73 | + first = body["items"][0] | ||||
| 74 | + assert {"rank", "model", "score", "trust_level", "config", "config_key", "comparability", "comparability_reasons", "delta_rank", "n_rows"} <= set(first) | ||||
| 75 | + assert first["comparability"] in ("comparable", "partially-comparable", "not-comparable") | ||||
| 76 | + only = (await client.get(f"/api/v1/benchmarks/{BENCH}/leaderboard", params={"comparable_only": 1, "limit": 50})).json() | ||||
| 77 | + assert all(it["comparability"] == "comparable" for it in only["items"]) | ||||
| 78 | + alias = await client.get("/api/v1/benchmarks/GPQA Diamond/leaderboard", params={"limit": 3}) # alias resolution | ||||
| 79 | + assert alias.status_code == 200 and alias.json()["benchmark"]["slug"] == BENCH | ||||
| 80 | + listing = (await client.get("/api/v1/benchmarks")).json() | ||||
| 81 | + b = next(x for x in listing["items"] if x["slug"] == BENCH) | ||||
| 82 | + assert {"family", "variant", "metric", "direction", "result_count", "model_count", "leader", "groups", "trust_mix", "category"} <= set(b) | ||||
| 83 | + res = (await client.get(f"/api/v1/benchmarks/{BENCH}/results", params={"limit": 3, "metric": "accuracy"})).json() | ||||
| 84 | + assert res["items"] and all(it["metric"].lower() == "accuracy" for it in res["items"]) | ||||
| 85 | + fr = (await client.get(f"/api/v1/benchmarks/{BENCH}/frontier")).json() | ||||
| 86 | + assert "series" in fr and all({"group", "points", "current_leader"} <= set(s) for s in fr["series"]) | ||||
| 87 | + | ||||
| 88 | + | ||||
| 89 | +async def test_matrix_shape(client: AsyncClient) -> None: | ||||
| 90 | + body = (await client.get("/api/v1/benchmarks/matrix", params={"limit": 5})).json() | ||||
| 91 | + assert {"columns", "rows", "total_rows", "methodology", "min_cells"} <= set(body) | ||||
| 92 | + assert 1 <= len(body["columns"]) <= 12 and all({"id", "slug", "metric", "config_key", "group_label"} <= set(c) for c in body["columns"]) | ||||
| 93 | + col_ids = [c["id"] for c in body["columns"]] | ||||
| 94 | + for row in body["rows"]: | ||||
| 95 | + assert set(row["cells"]) == set(col_ids) and row["n_cells"] >= 3 and {"model", "mean_rank"} <= set(row) | ||||
| 96 | + for cell in row["cells"].values(): | ||||
| 97 | + if cell: | ||||
| 98 | + assert {"score", "rank", "trust_level", "config_key", "comparability"} <= set(cell) | ||||
| 99 | + sub = (await client.get("/api/v1/benchmarks/matrix", params={"benchmarks": f"{BENCH},swe-bench-verified", "org": "anthropic"})).json() | ||||
| 100 | + assert [c["slug"] for c in sub["columns"]] == [BENCH, "swe-bench-verified"] | ||||
| 101 | + | ||||
| 102 | + | ||||
| 103 | +async def test_changes_default_excludes_backfill(client: AsyncClient) -> None: | ||||
| 104 | + body = (await client.get("/api/v1/changes", params={"limit": 20})).json() | ||||
| 105 | + assert body["date_field"] == "occurred" and body["include_backfill"] is False | ||||
| 106 | + assert all(e["is_backfill"] is False and "occurred_at" in e for e in body["items"]) | ||||
| 107 | + if body["items"] and body["next_before"]: | ||||
| 108 | + assert body["next_before"] == body["items"][-1]["occurred_at"] | ||||
| 109 | + obs = (await client.get("/api/v1/changes", params={"limit": 3, "include_backfill": 1, "date_field": "observed"})).json() | ||||
| 110 | + assert obs["date_field"] == "observed" and obs["total"] >= body["total"] or obs["total"] == 10000 | ||||
| 111 | + daily = (await client.get("/api/v1/changes/daily")).json() | ||||
| 112 | + assert {"date", "counts", "sections", "new_models", "today", "backfill_excluded"} <= set(daily) | ||||
| 113 | + for s in daily["today"]: | ||||
| 114 | + assert {"key", "label", "items", "total"} <= set(s) and all({"sources", "documents", "grouped_events"} <= set(i) for i in s["items"]) | ||||
| 115 | + tl = (await client.get(f"/api/v1/entities/{MODEL}/timeline", params={"limit": 3})).json() | ||||
| 116 | + assert tl["date_field"] == "occurred" and all(e["is_backfill"] is False for e in tl["items"]) | ||||
| 117 | + gt = (await client.get("/api/v1/timeline", params={"limit": 5})).json() | ||||
| 118 | + assert gt["include_backfill"] is False | ||||
| 119 | + | ||||
| 120 | + | ||||
| 121 | +async def test_stats_definitions(client: AsyncClient) -> None: | ||||
| 122 | + s = (await client.get("/api/v1/stats")).json() | ||||
| 123 | + assert {"organizations_total", "artifacts", "model_families", "definitions", "change_events_live_24h", "change_events_24h"} <= set(s) | ||||
| 124 | + assert s["organizations_total"] == sum(s["entities"].get(t, 0) for t in ("company", "organization", "lab", "university")) | ||||
| 125 | + companies = (await client.get("/api/v1/companies", params={"limit": 1})).json() | ||||
| 126 | + assert companies["total"] == s["organizations_total"] | ||||
| 127 | + assert "models" in s["definitions"] and "artifact" in s["entities"] and "model_family" in s["entities"] | ||||
| 128 | + | ||||
| 129 | + | ||||
| 130 | +async def test_frontier_shape(client: AsyncClient) -> None: | ||||
| 131 | + body = (await client.get("/api/v1/frontier")).json() | ||||
| 132 | + for key in ("latest_major_models", "benchmark_frontier", "price_frontier", "context_frontier", "open_weight_frontier", "efficiency_frontier", "agentic_frontier", | ||||
| 133 | + "multimodal_frontier", "recent_frontier_movements", "methodology"): | ||||
| 134 | + assert key in body, key | ||||
| 135 | + for b in body["benchmark_frontier"]: | ||||
| 136 | + assert {"benchmark", "group", "leader", "second", "gap"} <= set(b) and b["group"]["n"] >= 20 | ||||
| 137 | + assert {"cheapest_output", "cheapest_output_1m_context", "frontier_models", "composition"} <= set(body["price_frontier"]) | ||||
| 138 | + assert {"points", "frontier"} <= set(body["efficiency_frontier"]) | ||||
| 139 | + assert "dimensions" in body["open_weight_frontier"] and "score" not in str(body["open_weight_frontier"]["dimensions"]) | ||||
| 140 | + | ||||
| 141 | + | ||||
| 142 | +async def test_pareto_frontier_ids_are_efficient(client: AsyncClient) -> None: | ||||
| 143 | + body = (await client.get("/api/v1/pareto", params={"benchmark": BENCH})).json() | ||||
| 144 | + assert {"group", "points", "frontier", "methodology", "x", "y"} <= set(body) | ||||
| 145 | + pts = {p["id"]: p for p in body["points"]} | ||||
| 146 | + front = set(body["frontier"]) | ||||
| 147 | + assert front <= set(pts) | ||||
| 148 | + hib = body["group"]["higher_is_better"] | ||||
| 149 | + for fid in front: # no other point is at least as good on both axes and strictly better on one | ||||
| 150 | + p = pts[fid] | ||||
| 151 | + for o in pts.values(): | ||||
| 152 | + if o["id"] == fid: | ||||
| 153 | + continue | ||||
| 154 | + better_y = o["y"] > p["y"] if hib else o["y"] < p["y"] | ||||
| 155 | + ge_y = o["y"] >= p["y"] if hib else o["y"] <= p["y"] | ||||
| 156 | + assert not (o["x"] <= p["x"] and ge_y and (o["x"] < p["x"] or better_y)), (fid, o["id"]) | ||||
| 157 | + assert all(p["pareto"] == (p["id"] in front) for p in body["points"]) | ||||
| 158 | + assert (await client.get("/api/v1/pareto", params={"benchmark": BENCH, "x": "latency"})).status_code == 400 | ||||
| 159 | + | ||||
| 160 | + | ||||
| 161 | +async def test_cost_routes(client: AsyncClient) -> None: | ||||
| 162 | + body = (await client.get("/api/v1/cost", params={"model": MODEL, "input_tokens": 1000, "output_tokens": 500, "requests_per_day": 100, "cached_share": 0.5})).json() | ||||
| 163 | + assert {"items", "methodology", "inputs"} <= set(body) | ||||
| 164 | + for it in body["items"]: | ||||
| 165 | + c, p = it["cost"], it["deployment"]["prices"] | ||||
| 166 | + if c["per_request"] is not None and p["input"] is not None and p["output"] is not None: | ||||
| 167 | + assert c["daily"] == pytest.approx(c["per_request"] * 100) and c["monthly"] == pytest.approx(c["daily"] * 30) | ||||
| 168 | + ctx = (await client.get("/api/v1/cost/context", params={"tokens": 100000, "limit": 5})).json() | ||||
| 169 | + assert ctx["items"] and all(it["context_length"] >= 100000 and it["cost_usd"] == pytest.approx(it["deployment"]["prices"]["input"] * 0.1) for it in ctx["items"]) | ||||
| 170 | + deps = (await client.get("/api/v1/deployments", params={"model": MODEL})).json() | ||||
| 171 | + assert deps["items"] and all(d["status"] == "active" for d in deps["items"]) | ||||
| 172 | + | ||||
| 173 | + | ||||
| 174 | +async def test_find_a_model_returns_why(client: AsyncClient) -> None: | ||||
| 175 | + body = (await client.get("/api/v1/find-a-model", params={"use_case": "coding", "limit": 5})).json() | ||||
| 176 | + assert {"matches", "filters_applied", "note", "rules"} <= set(body) and body["matches"] | ||||
| 177 | + for m in body["matches"]: | ||||
| 178 | + assert m["why"] and {"model", "observed"} <= set(m) and "best_rank" in m["observed"] | ||||
| 179 | + assert "score" not in m # no composite winner score | ||||
| 180 | + local = (await client.get("/api/v1/find-a-model", params={"use_case": "local", "memory_gb": 64, "limit": 3})).json() | ||||
| 181 | + assert all("estimated_fit" in m and m["estimated_fit"]["estimated"] is True for m in local["matches"]) | ||||
| 182 | + assert (await client.get("/api/v1/find-a-model", params={"use_case": "nope"})).status_code == 400 | ||||
| 183 | + | ||||
| 184 | + | ||||
| 185 | +async def test_open_run_locally_families_graph(client: AsyncClient) -> None: | ||||
| 186 | + op = (await client.get("/api/v1/open", params={"limit": 3})).json() | ||||
| 187 | + assert {"items", "summary", "note"} <= set(op) and all({"licence", "dimensions", "best_results", "hardware_fit", "providers"} <= set(i) for i in op["items"]) | ||||
| 188 | + rl = (await client.get("/api/v1/run-locally", params={"memory_gb": 64, "limit": 3})).json() | ||||
| 189 | + assert rl["estimated"] is True and all(i["fit"]["fits"] and "breakdown" in i["fit"] for i in rl["items"]) | ||||
| 190 | + fams = (await client.get("/api/v1/families", params={"limit": 3})).json() | ||||
| 191 | + assert fams["items"] and all({"name", "model_count", "canonical", "licenses", "benchmark_best"} <= set(f) for f in fams["items"]) | ||||
| 192 | + fd = await client.get(f"/api/v1/families/{fams['items'][0]['slug']}") | ||||
| 193 | + assert fd.status_code == 200 and {"members", "timeline", "lineage", "artifacts_count"} <= set(fd.json()) | ||||
| 194 | + g = (await client.get("/api/v1/graph/explore", params={"node": MODEL, "mode": "company", "depth": 2, "limit": 20})).json() | ||||
| 195 | + assert {"nodes", "edges", "truncated", "counts"} <= set(g) and len(g["nodes"]) <= 20 | ||||
| 196 | + assert (await client.get("/api/v1/graph/explore", params={"node": MODEL, "mode": "nope"})).status_code == 400 | ||||
| 197 | + | ||||
| 198 | + | ||||
| 199 | +async def test_search_compiler_v2_response(client: AsyncClient) -> None: | ||||
| 200 | + body = (await client.get("/api/v1/search", params={"q": "Anthropic models released since 2025"})).json() | ||||
| 201 | + q = body["query"] | ||||
| 202 | + assert q["version"] == 2 and {"compiled", "residual", "unrecognised"} <= set(q) | ||||
| 203 | + assert q["organization"] == "Anthropic" and q["year_from"] == 2025 | ||||
| 204 | + assert any(c["filter"] == "organization" and isinstance(c["value"], dict) for c in q["compiled"]) | ||||
| 205 | + assert body["items"] and all(it["organization"]["slug"] == "anthropic" for it in body["items"]) | ||||
| 206 | + unknown = (await client.get("/api/v1/search", params={"q": "Foobarbaz models released since 2025"})).json()["query"] | ||||
| 207 | + assert "organization" not in unknown and "Foobarbaz" in unknown["unrecognised"] | ||||
| 208 | + | ||||
| 209 | + | ||||
| 210 | +async def test_etag_304(client: AsyncClient) -> None: | ||||
| 211 | + r1 = await client.get("/api/v1/stats") | ||||
| 212 | + etag = r1.headers.get("etag") | ||||
| 213 | + assert etag and etag.startswith('W/"') and "max-age" in r1.headers.get("cache-control", "") | ||||
| 214 | + r2 = await client.get("/api/v1/stats", headers={"if-none-match": etag}) | ||||
| 215 | + assert r2.status_code == 304 and r2.content == b"" and r2.headers.get("etag") == etag | ||||
| 216 | + admin = await client.get("/api/v1/admin/overview", headers=ADMIN) | ||||
| 217 | + assert "etag" not in admin.headers # admin responses are never cacheable | ||||
| 218 | + | ||||
| 219 | + | ||||
| 220 | +async def test_admin_rate_limit_before_auth(client: AsyncClient) -> None: | ||||
| 221 | + from aiatlas.api.common import _hits | ||||
| 222 | + | ||||
| 223 | + _hits.clear() | ||||
| 224 | + codes = [(await client.get("/api/v1/admin/overview", headers={"x-aia-admin-token": "wrong"})).status_code for _ in range(11)] | ||||
| 225 | + assert codes[:10] == [401] * 10 and codes[10] == 429, codes # 10 failed auths per minute per IP, then 429 before any token check | ||||
| 226 | + _hits.clear() | ||||
| 227 | + assert (await client.get("/api/v1/admin/overview", headers=ADMIN)).status_code == 200 | ||||
| 228 | + | ||||
| 229 | + | ||||
| 230 | +async def test_admin_workbenches_and_audit(client: AsyncClient) -> None: | ||||
| 231 | + from aiatlas.api.common import _hits | ||||
| 232 | + | ||||
| 233 | + _hits.clear() | ||||
| 234 | + q = (await client.get("/api/v1/admin/quality", headers=ADMIN)).json() | ||||
| 235 | + for key in ("duplicate_candidates", "taxonomy_violations", "impossible_values", "conflicting_t1_claims", "models_without_organization", "orphan_benchmark_results", | ||||
| 236 | + "unresolved_provider_deployments", "quantisations_typed_as_models", "stale_sources", "empty_public_categories", "quarantined_runs_pending", "review_queue_priority"): | ||||
| 237 | + assert key in q, key | ||||
| 238 | + assert {"count", "sample"} <= set(q["models_without_organization"]) | ||||
| 239 | + er = (await client.get("/api/v1/admin/entity-resolution", params={"limit": 3}, headers=ADMIN)).json() | ||||
| 240 | + assert {"items", "decisions"} <= set(er) and all({"a", "b", "signals", "hint"} <= set(i) for i in er["items"]) | ||||
| 241 | + for path in ("/api/v1/admin/anomalies", "/api/v1/admin/quarantine", "/api/v1/admin/audit?limit=5"): | ||||
| 242 | + assert (await client.get(path, headers=ADMIN)).status_code == 200, path | ||||
| 243 | + audit = (await client.get("/api/v1/admin/audit?limit=5", headers=ADMIN)).json() | ||||
| 244 | + assert audit["items"] and audit["items"][0]["actor"] == "admin" and audit["items"][0]["action"].startswith("GET /api/v1/admin/") | ||||
| 245 | + assert (await client.post("/api/v1/admin/quarantine/nope", headers=ADMIN, json={"action": "release"})).status_code in (404, 501) | ||||
| 246 | + assert (await client.post("/api/v1/admin/anomalies/nope", headers=ADMIN, json={"status": "resolved"})).status_code == 404 | ||||
| 247 | + | ||||
| 248 | + | ||||
| 249 | +async def test_claims_provenance_licenses_misc(client: AsyncClient) -> None: | ||||
| 250 | + cl = (await client.get(f"/api/v1/entities/{MODEL}/claims", params={"limit": 2})).json() | ||||
| 251 | + assert cl["items"] and cl["status"] == "current" | ||||
| 252 | + detail = (await client.get(f"/api/v1/claims/{cl['items'][0]['id']}")).json() | ||||
| 253 | + assert {"claim", "entity", "chain", "source", "extractor", "evidence"} <= set(detail) and {"snapshot_id", "document_url", "archived"} <= set(detail["evidence"]) | ||||
| 254 | + prov = (await client.get(f"/api/v1/entities/{MODEL}/provenance/context_length")).json() | ||||
| 255 | + assert {"value", "source", "tier", "observed_at", "claim_id", "conflicts", "history_count", "snapshot_id"} <= set(prov) | ||||
| 256 | + assert (await client.get(f"/api/v1/entities/{MODEL}/provenance/not_a_property")).status_code == 404 | ||||
| 257 | + lic = (await client.get("/api/v1/licenses")).json() | ||||
| 258 | + assert lic["items"] and all({"key", "commercial_use", "models"} <= set(i) for i in lic["items"]) | ||||
| 259 | + assert (await client.get("/api/v1/licenses/apache-2.0")).json()["key"] == "Apache-2.0" | ||||
| 260 | + meth = (await client.get("/api/v1/methodology")).json() | ||||
| 261 | + assert {"openness", "trust_levels", "comparability", "counters", "anomaly_checks", "event_semantics"} <= set(meth) | ||||
| 262 | + tr = (await client.get("/api/v1/trending", params={"kind": "most_changed"})).json() | ||||
| 263 | + assert tr["kind"] == "most_changed" and "definition" in tr | ||||
| 264 | + cmp_ = (await client.get("/api/v1/compare", params={"ids": f"{MODEL},{MODEL_B}", "diff_only": 1, "mode": "models"})).json() | ||||
| 265 | + assert cmp_["diff_only"] is True and "comparability" in cmp_ | ||||
| 266 | + for d in cmp_["dimensions"]: | ||||
| 267 | + vals = [repr(sorted(map(str, it["values"][d["key"]])) if isinstance(it["values"][d["key"]], list) else it["values"][d["key"]]) for it in cmp_["items"]] | ||||
| 268 | + assert len(set(vals)) > 1, d["key"] | ||||
| 269 | + diff = (await client.get(f"/api/v1/models/{MODEL}/diff/{MODEL_B}")).json() | ||||
| 270 | + assert all("delta" in d for d in diff["dimensions"]) | ||||
| 271 | + assert (await client.get("/api/v1/compare", params={"ids": f"{MODEL},{MODEL_B}", "mode": "providers"})).status_code == 400 | ||||
| 272 | + df = (await client.get("/api/v1/diff", params={"a": "2026-01-01", "b": "2026-12-31", "limit": 5})).json() | ||||
| 273 | + assert {"new_benchmark_leaders", "provider_changes", "hardware_changes", "context_changes", "retired_models"} <= set(df) and df["include_artifacts"] is False | ||||
| 274 | + tm = (await client.get("/api/v1/time-machine", params={"date": "2025-06-01", "scope": "models", "limit": 3})).json() | ||||
| 275 | + assert {"reconstructed", "first_entity_at", "note", "models"} <= set(tm) and all("attributes_as_of" in m for m in tm["models"]["items"]) | ||||
| 276 | + pulse = (await client.get("/api/v1/pulse", params={"days": 7})).json() | ||||
| 277 | + assert all({"value", "definition"} <= set(v) for v in pulse["counters"].values()) | ||||
| 278 | + idx = (await client.get("/api/v1/prices/index", params={"days": 14})).json() | ||||
| 279 | + assert {"series", "cheapest_frontier", "distribution", "new_listings_30d", "delistings_30d", "methodology"} <= set(idx) | ||||
| 280 | + assert all({"median_frontier_output", "median_open_output", "median_embedding_input", "sample"} <= set(s) for s in idx["series"]) | ||||
| 281 | + prov_list = (await client.get("/api/v1/providers")).json()["items"] | ||||
| 282 | + assert all({"input_price_distribution", "output_price_distribution", "models_added_30d", "models_removed_30d", "price_changes_30d", "organizations_covered", "features_supported"} <= set(p) for p in prov_list) | ||||
| 283 | + hw = (await client.get("/api/v1/hardware/apple-m3-ultra/fit", params={"limit": 3})).json() | ||||
| 284 | + assert hw["estimated"] is True and hw["memory_options_gb"] | ||||
| 285 | |||||