SPB Git forge

spb/ai-atlas

Public
41commits 1branches 0releases
4.6 MBsize
maindefault branch
12 days agolast push
HTML 77.2% TypeScript 10.5% Python 9.6% JavaScript 2.5%

API 1.1 follow-ups: per-database cache namespace (frontier×price fix), pulse/index items, provider deployments, best_price, matrix/pareto fields

Root cause of the null cheapest_frontier: the Redis API cache was not scoped to the database, so the dev API (:8331) and the
production-copy API (:8332), which share redis db 5, served each other's cached bodies and the cached frontier id set —
prod joined dev ids against its prices (0 rows) while `sample.frontier_offers` came from a body cached by the other process.
Cache keys are now `aia:api:<sha1(DATABASE_URL)[:8]>:…` and the frontier id set is re-validated against live canonical models
on every cache hit. Regression test asserts cheapest_frontier / sort=cheapest_frontier / price_frontier are non-null whenever
frontier offers exist.

Also: PRICE_CHANGED old/new values are {input,output} dicts → price_delta() (pulse median_percent, pulse/index/frontier items
with % and provider); /prices/index 30-day counters → {count, items[], definition}; /providers/{slug} deployments + removed
(90 d); /deployments status=active|delisted|all (current=0 ⇒ all); framework/library kind (normalised) + dataset summary
attributes + license_key derived on read; /open new_30d by release_date (first_seen_at only when unknown); /models best_price
(one lateral join); /families benchmark_best with scores (list + members); /benchmarks/matrix since/until on release_date and
observed_at/evaluated_at per cell; /pareto points carry context_length, parameter_count, release_date. docs/API.md updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed 12 days ago (Sep 12, 2026) parent 16922fb

13 changed files +280 −71

modified docs/API.md +13 −10
@@ -78,20 +78,20 @@ type Fit = { quantization: string; estimated: true; fits: boolean; estimated_mem
78 78 | `GET /entities/{slug}/graph?depth=1&limit=80` | `{ root, nodes: { id, slug, name, entity_type, organization_name }[], edges: { source, target, predicate }[] }` |
79 79 | `GET /entities/{slug}/sources` | `{ items: SourceRef[] }` |
80 80 | `GET /entities/{slug}/related?limit=` | `{ items: EntitySummary[] }` (same org / same family (`family_id` or label) / shared relations) |
81 −| `GET /models?q=&org=&family=&openness=&modality=&status=&min_params=&max_params=&min_context=&year_from=&year_to=&license=&sort=&order=&limit=&offset=&facets=1` | `Page<EntitySummary & { identity_confidence, family?: {id,slug,name}, canonical?: EntitySummary, artifact_kind? }> & { universe }` — **1.1:** default universe = `entity_type='model' and merged_into is null`; `include=artifacts`; `family=` matches `model_family` slug/id/name via `entities.family_id` (falls back to `attributes.family`); `license=` matches the canonical key (`attributes.license_key`) or any raw label the ontology maps to it; `reasoning=0|1`, `trust=high,medium,low` (identity_confidence), `sort=cheapest`. Facets add `families` (`{value,label,canonical,count}`), canonical `licenses` (`{value,label,category,count,raw_labels,raw?}`), `trust`, `definitions`. |
81 +| `GET /models?q=&org=&family=&openness=&modality=&status=&min_params=&max_params=&min_context=&year_from=&year_to=&license=&sort=&order=&limit=&offset=&facets=1` | `Page<EntitySummary & { identity_confidence, best_price: { input_per_mtok, output_per_mtok, unit, provider: Org, providers, note } | null, family?: {id,slug,name}, canonical?: EntitySummary, artifact_kind? }> & { universe }` (`best_price` = cheapest current offer by output price, one lateral join; summary attributes gain `license_key` derived on read, `reasoning`, `tool_calling`) — **1.1:** default universe = `entity_type='model' and merged_into is null`; `include=artifacts`; `family=` matches `model_family` slug/id/name via `entities.family_id` (falls back to `attributes.family`); `license=` matches the canonical key (`attributes.license_key`) or any raw label the ontology maps to it; `reasoning=0|1`, `trust=high,medium,low` (identity_confidence), `sort=cheapest`. Facets add `families` (`{value,label,canonical,count}`), canonical `licenses` (`{value,label,category,count,raw_labels,raw?}`), `trust`, `definitions`. |
82 82 | `GET /companies?q=&country=&kind=&sort=&limit=&offset=&facets=1` | `Page<EntitySummary & { model_count, paper_count }>` — covers `company|organization|lab|university`; `total` = `/stats.organizations_total` |
83 83 | `GET /papers?…` | `Page<EntitySummary>` |
84 84 | `GET /providers` | `{ items: (EntitySummary & { model_count, price_count, min_input_per_mtok, min_output_per_mtok, input_price_distribution: {min,p25,median,p75,max,n}|null, output_price_distribution, models_added_30d, models_removed_30d, price_changes_30d, organizations_covered, features_supported: string[], feature_keys: string[] })[], note }` — **1.1** aggregates |
85 85 | `GET /prices?model=&provider=&sort=&limit=&offset=&current=1&family=&org=&modality=` | `Page<Price>` — **1.1:** `family`, `org`, `modality` filters; `sort=cheapest_frontier` (frontier models only, cheapest output first, + `methodology`) |
86 86 | `GET /prices/history?model=&provider=` | `{ items: Price[] }` |
87 −| `GET /prices/index?days=180` | **AI Price Index (1.1)** `{ days, series: { day, median_input, median_output, median_frontier_output, median_frontier_input, median_open_output, median_embedding_input, min_input, max_input, min_frontier_output, models, offers, sample: { models, offers, frontier_models, frontier_offers, open_models, embedding_models } }[], movers: ChangeEvent[], cheapest_frontier: { model, provider, output, input, context_length, price_id }|null, cheapest_frontier_1m_context, distribution: { metric, unit, buckets: { from, to, label, offers }[], offers }, new_listings_30d, delistings_30d, price_changes_30d, frontier: { composition, methodology }, methodology, note }` — v1 keys kept |
87 +| `GET /prices/index?days=180` | **AI Price Index (1.1)** `{ days, series: { day, median_input, median_output, median_frontier_output, median_frontier_input, median_open_output, median_embedding_input, min_input, max_input, min_frontier_output, models, offers, sample: { models, offers, frontier_models, frontier_offers, open_models, embedding_models } }[], movers: ChangeEvent[], cheapest_frontier: { model, provider, output, input, context_length, price_id }|null, cheapest_frontier_1m_context, distribution: { metric, unit, buckets: { from, to, label, offers }[], offers }, new_listings_30d: { count, items: Deployment[], definition }, delistings_30d: { count, items: Deployment[], definition }, price_changes_30d: { count, items: (ChangeEvent & { percent_change, price_delta, provider })[], definition }, frontier: { composition, methodology }, methodology, note }` — v1 keys kept; the three 30-day counters became objects (read `.count`), items capped at 50 |
88 88 | `GET /benchmarks?category=` | `{ items: { id, entity_type, slug, name, category, family, variant, metric, unit, direction, attributes, result_count, model_count, leader: LeaderboardRow|null, second, top: {model, score}|null, primary_group: Group|null, groups: Group[], trust_mix, trust_labels }[], total, note }` — **1.1:** `result_count` = current rows, `model_count` = distinct canonical models, `leader` from the primary metric group |
89 89 | `GET /benchmarks/{slug}` | `EntityDetail` + `family, variant, metric, direction, category, groups, primary_group, result_count, model_count, leaderboard (top 25), trust_mix` |
90 90 | `GET /benchmarks/{slug}/results?limit=&offset=&config=&history=&metric=&config_key=` | `Page<BenchmarkResult>` — v1 behaviour (one row per result) + `metric` / `config_key` filters |
91 91 | `GET /benchmarks/{slug}/history?model=` | `{ benchmark, items: BenchmarkResult[] }` |
92 92 | `GET /hardware?…` | `Page<EntitySummary>` + facets |
93 93 | `GET /hardware/fit?memory_gb=&quant=&context=&limit=&openness=` | v1 estimate (unchanged) |
94 −| `GET /explore/types` · `GET /explore/{type}` | unchanged (`artifact`, `model_family` appear as types once present) |
94 +| `GET /explore/types` · `GET /explore/{type}` | unchanged (`artifact`, `model_family` appear as types once present); **1.1:** framework/library/runtime/tool items expose `kind` (canonical framework kind, normalised on read, `kind_raw` when the source label differed) and dataset summaries surface `publisher, size, access, hf_repo, modality, license_key…` when recorded |
95 95 | `GET /changes?category=&type=&entity_type=&importance_min=&since=&until=&q=&entity=&limit=50&before=&offset=&include_documents=0&include_backfill=0&date_field=occurred` | `Page<ChangeEvent> & { next_before, date_field, include_backfill }` — **1.1:** defaults `is_backfill=false`, `occurred_at`; cursor/filters follow `date_field` |
96 96 | `GET /changes/daily?date=&per_section=30&include_backfill=0&date_field=occurred` | `{ date, counts, total, sections (v1 by category), today: { key, label, items: (ChangeEvent & { sources: n, documents: string[], grouped_events, event_ids? })[], total }[], new_models, labels, backfill_excluded, previous_day, next_day, note }` — **Today in AI 2.0** sections: `MAJOR_RELEASES, OPEN_WEIGHT_RELEASES, PRICE_MOVES, BENCHMARK_MOVES, MODEL_CHANGES, RESEARCH, DEPRECATIONS, PROVIDER_CHANGES, HARDWARE`; events sharing a `group_key` are folded into one item |
97 97 | `GET /changes/categories?days=7&include_backfill=0` | `{ items: { category, event_type, count }[] }` |
@@ -124,6 +124,9 @@ EntitySummary & {
124 124 family_id?: string | null; identity_confidence?: "high"|"medium"|"low"
125 125 // artifact (1.1)
126 126 canonical?: EntitySummary | null; artifact_kind?: "checkpoint"|"quantization"|"conversion"|"packaging"|null
127 + // provider (1.1)
128 + deployments?: Deployment[] // current offers of this provider, cheapest output first
129 + removed?: Deployment[] // offers closed in the last 90 days (status delisted)
127 130 // company / provider / hardware / framework / model_family
128 131 models?: Page<EntitySummary>; papers?; repositories?
129 132 }
@@ -139,18 +142,18 @@ EntitySummary & {
139 142 | `GET /claims/{id}` | `{ claim, entity, property, chain: { previous: Claim[], superseding: Claim[], conflicting: Claim[], history_count }, source: { id, name, domain, tier, url, snapshot_id, observed_at }, extractor: { name, version, confidence }, run_id, evidence: { snapshot_id, document_url, archived, snapshot_observed_at, document_title, doc_type }, note }` |
140 143 | `GET /benchmarks/{slug}/leaderboard?metric=&config_key=&trust=&org=&since=&until=&comparable_only=0&limit=100&offset=0` | `{ benchmark, group: Group, groups: Group[], items: LeaderboardRow[], total, limit, offset, comparable_only, filters, history_available, methodology }` — ONE row per canonical model (best current row in the group; default group = primary metric, most-populated config); `delta_rank` vs the ranking of the closed rows of the same group when history exists |
141 144 | `GET /benchmarks/{slug}/frontier?metric=&config_key=` | `{ benchmark, series: { group: Group, primary: bool, points: { date, model: ModelRef, score, trust_level, config, result_id }[], current_leader }[], generated_at, methodology }` — leader history per group (a point each time a new best appears, ordered by `coalesce(evaluated_at, observed_at)`, closed rows included) |
142 −| `GET /benchmarks/matrix?benchmarks=a,b,c&models=…&org=&family=&limit=60&comparable_only=0&min_cells=3` | `{ columns: { id, slug, name, category, metric, config_key, group_label, higher_is_better, n_models }[], rows: { model: {…}, cells: Record<benchmarkId, { score, rank, trust_level, config_key, comparability, result_id } | null>, n_cells, mean_rank }[], total_rows, comparable_only, min_cells, methodology }` — default columns = 12 benchmarks with most current results, default rows = models with ≥ `min_cells` cells (`mean_rank` is a sort key, not a score) |
143 −| `GET /deployments?model=&provider=&org=&current=1&sort=valid_from|output|input|model|provider&limit=50&offset=&before=<valid_from cursor>` | `Page<Deployment> & { next_before, current }` |
145 +| `GET /benchmarks/matrix?benchmarks=a,b,c&models=…&org=&family=&since=&until=&limit=60&comparable_only=0&min_cells=3` | `{ columns: { id, slug, name, category, metric, config_key, group_label, higher_is_better, n_models }[], rows: { model: {…}, cells: Record<benchmarkId, { score, rank, trust_level, config_key, comparability, result_id, observed_at, evaluated_at } | null>, n_cells, mean_rank }[], total_rows, comparable_only, min_cells, filters, methodology }` (`since`/`until` bound the model `release_date`) — default columns = 12 benchmarks with most current results, default rows = models with ≥ `min_cells` cells (`mean_rank` is a sort key, not a score) |
146 +| `GET /deployments?model=&provider=&org=&current=1&status=active|delisted|all&sort=valid_from|output|input|model|provider&limit=50&offset=&before=<valid_from cursor>` | `Page<Deployment> & { next_before, current, status }` — `current=0` ⇒ `status=all` (active + delisted rows); `status=delisted` returns closed rows only (newest `valid_to` first) |
144 147 | `GET /cost?model=&provider=&input_tokens=1000&output_tokens=500&requests_per_day=1000&cached_share=0..1&batch=0|1` | `{ model, inputs, items: { deployment: Deployment, cost: { per_request, daily, monthly, annual, effective_input_per_mtok, effective_output_per_mtok, per_request_fee, inputs, notes: string[] } }[], total, currency, methodology, note }` — cached/batch prices used only when published (else a `notes` entry) |
145 148 | `GET /cost/context?tokens=1000000&limit=50&model=&org=` | `{ tokens, items: { deployment, context_length, context_source: "offer"|"model attribute", cost_usd }[], total, currency, methodology }` — offers whose context ≥ tokens, cheapest first |
146 149 | `GET /frontier?limit=12` | `{ latest_major_models: ChangeEvent[], benchmark_frontier: { benchmark, group, leader, second, gap }[], price_frontier: { cheapest_output: Deployment, cheapest_output_1m_context, frontier_models, composition }, context_frontier: { model, context_length }[], open_weight_frontier: { items: { model, best_rank, best_rank_on, parameter_count, context_length, ranks }[], dimensions, note }, efficiency_frontier: { quality: { benchmark, group }, x, points: { id, model, x, y, rank, trust_level, pareto }[], frontier: string[] }, agentic_frontier: { benchmark, group, leaders: LeaderboardRow[] }[], multimodal_frontier: { model, modalities, top10_on }[], recent_frontier_movements: (ChangeEvent & { percent_change? })[], generated_at, methodology }` |
147 −| `GET /pareto?benchmark=&x=output_price|input_price|parameter_count|context_length|memory_estimate&y=score&metric=&config_key=&org=&family=&openness=` | `{ benchmark, group, groups, x: { key, label }, y, points: { id, model, x, y, rank, trust_level, config, provider?, estimated?, pareto }[], frontier: string[], methodology }` — `x=latency` → 400 (not stored, never estimated) |
148 −| `GET /pulse?days=7` | `{ days, since, until, counters: Record<key, { value, definition, median_percent?, items? }>, note }` — keys: `new_models, new_open_weight_models, new_artifacts, new_papers, provider_listings, provider_delistings, price_changes, new_models_1m_context, new_benchmark_leaders, documents_changed, sources_observed, events_total`; all `is_backfill=false`, `occurred_at` in window |
149 −| `GET /open?sort=release|params|context|rank|name|downloads&license=&min_params=&max_params=&min_context=&modality=&days=&openness=&limit=&offset=` | `Page<{ model, licence: LicenseInfo & { raw } | { key: null, raw, note }, dimensions, best_results: { benchmark, rank }[], best_rank, hardware_fit: { "4bit_64gb": Fit, "8bit_128gb": Fit, estimated: true }, providers, cheapest_output_per_mtok }> & { summary: { by_category, by_license_top, new_30d }, note }` — universe = downloadable weights (open-weights, open-source, restricted-weights) |
150 +| `GET /pareto?benchmark=&x=output_price|input_price|parameter_count|context_length|memory_estimate&y=score&metric=&config_key=&org=&family=&openness=` | `{ benchmark, group, groups, x: { key, label }, y, points: { id, model, x, y, rank, trust_level, config, context_length, parameter_count, release_date, provider?, estimated?, pareto }[], frontier: string[], methodology }` — `x=latency` → 400 (not stored, never estimated) |
151 +| `GET /pulse?days=7` | `{ days, since, until, counters: Record<key, { value, definition, median_percent?, items? }>, note }` — keys: `new_models, new_open_weight_models, new_artifacts, new_papers, provider_listings, provider_delistings, price_changes (items: { id, summary, model, provider, occurred_at, percent_change, delta }[], median_percent), new_models_1m_context (items: EntitySummary[]), new_benchmark_leaders (items), documents_changed, sources_observed, events_total`; all `is_backfill=false`, `occurred_at` in window. Price deltas are read from the events' `{input_per_mtok, output_per_mtok}` old/new values (output first) |
152 +| `GET /open?sort=release|params|context|rank|name|downloads&license=&min_params=&max_params=&min_context=&modality=&days=&openness=&limit=&offset=` | `Page<{ model, licence: LicenseInfo & { raw } | { key: null, raw, note }, dimensions, best_results: { benchmark, rank }[], best_rank, hardware_fit: { "4bit_64gb": Fit, "8bit_128gb": Fit, estimated: true }, providers, cheapest_output_per_mtok }> & { summary: { by_category, by_license_top, new_30d, new_30d_definition }, note }` (`new_30d` = `release_date` in the last 30 days, `first_seen_at` only when no release date is known; `licence.key` is derived on read from the raw label when `license_key` is missing) — universe = downloadable weights (open-weights, open-source, restricted-weights) |
150 153 | `GET /find-a-model?use_case=coding|reasoning|agentic|long_context|vision|low_cost|local|embeddings|chat&deployment=local|api|any&memory_gb=&quant=&context_min=&license=commercial|any&openness=&max_input_price=&max_output_price=&modalities=&limit=30` | `{ matches: { model, why: string[], observed: {…}, estimated_fit?: Fit, deployments?: Deployment[] }[], total, filters_applied, rules, note }` — deterministic rules (see `/methodology.find_a_model`), sorted by satisfied criteria → best benchmark rank → release date; **no composite score** |
151 154 | `GET /run-locally?memory_gb=&gpu_count=1|2|4|8&quant=&context=8192&batch=1&platform=apple|nvidia|amd|any&use_case=&openness=&limit=` | `{ inputs, estimated: true, assumptions, counts, items: { model, fit: Fit, artifacts: { artifact, quant_format, file_size_gb, weights_source: "observed"|"estimated", fit: Fit }[], artifact_count }[], note }` — artifacts with an observed `file_size_gb` use it as the weights size |
152 155 | `GET /hardware/{slug}/fit?quant=&context=&memory_gb=&gpu_count=&limit=&openness=` | `{ hardware, memory_options_gb, inputs, estimated: true, assumptions, runtimes, counts, items: ({ model } & Fit)[] }` |
153 −| `GET /families?q=&org=&sort=models|name|last_release&limit=&offset=` | `Page<{ id, slug, name, canonical: bool, entity_type: "model_family", organization, model_count, first_release, last_release, param_range, modalities, licenses, benchmark_best }> & { note }` — `model_family` entities + legacy `attributes.family` labels (`canonical: false`) |
156 +| `GET /families?q=&org=&sort=models|name|last_release&limit=&offset=` | `Page<{ id, slug, name, canonical: bool, entity_type: "model_family", organization, model_count, first_release, last_release, param_range, modalities, licenses, benchmark_best: Record<benchmark, { rank, score, metric, config_key, model, model_name }> }> & { note }` (members in `/families/{slug}` carry `benchmark_ranks` and `benchmark_best: { benchmark, rank, score, metric, config_key }[]`) — `model_family` entities + legacy `attributes.family` labels (`canonical: false`) |
154 157 | `GET /families/{slug}?limit=` | `{ id, slug, name, canonical, summary, …aggregates, members: { model, key_facts, benchmark_ranks }[], artifacts_count, providers, lineage: { source, target, predicate }[], timeline, note }` — accepts a family label pre-canonicalisation |
155 158 | `GET /graph/explore?node=&mode=lineage|research|company|benchmark|dataset|provider|hardware&depth=1|2&limit=150` | `{ root, mode, depth, predicates, nodes: { id, slug, name, entity_type, org, org_slug, level, artifact_kind, attributes }[], edges: { source, target, predicate, attributes, tier }[], truncated, counts }` — never more than `limit` nodes |
156 159 | `GET /time-machine?date=YYYY-MM-DD&scope=models|prices|benchmarks|hardware|all&limit=50` | `{ date, scope, first_entity_at, reconstructed, note, models?: { items: { model, attributes_as_of, observed_then, reconstructed }[], total }, prices?: { items: Deployment[], total }, benchmarks?: { leaders: { benchmark, leader }[] }, hardware?: { items } }` — before `first_entity_at` the state is reconstructed from claims/release dates (`reconstructed: true`) |
@@ -211,7 +214,7 @@ None. No v1 route, parameter, field or type was removed or retyped in 1.1.
211 214 ## Implementation notes (`src/aiatlas/api`)
212 215
213 216 - **Validation errors**: parameters rejected by FastAPI/pydantic typing or bounds return **422** with `{ detail, errors }`; semantic errors raised by the routes return **400**.
214 −- **Caching**: public GETs are cached in Redis (`aia:api:<path>?<sorted query>`, 60–600 s) and served with a weak ETag (`W/"sha1(body)"`) computed on the uncompressed body inside GZip; `/stats` stays at 60 s. Cache invalidation: `services.cache.cache_invalidate(prefix)`; `POST /admin/cache/flush?after_run=1`.
217 +- **Caching**: public GETs are cached in Redis (`aia:api:<ns>:<path>?<sorted query>`, 60–600 s) where `<ns>` = 8-hex sha1 of `DATABASE_URL` — two API processes on different databases (dev :8331, prod copy :8332) sharing one Redis db never read each other's bodies or derived id sets (the frontier id set is also re-validated against live canonical models on every cache hit) — and served with a weak ETag (`W/"sha1(body)"`) computed on the uncompressed body inside GZip; `/stats` stays at 60 s. Cache invalidation: `services.cache.cache_invalidate(prefix)`; `POST /admin/cache/flush?after_run=1`.
215 218 - **Connections**: `EntityDetail` runs its blocks in ≤ 4 concurrent groups (one pooled connection each, sequential inside a group) instead of one connection per block.
216 219 - **Performance (2026-09-12, prod copy, EXPLAIN ANALYZE)**: leaderboard load 8 ms · matrix / all groups 8 ms · models listing 0.5 ms (+ count 1.2 ms, facets ≤ 4 ms) · changes feed 0.8 ms (count 3 ms) · daily sections 0.4 ms · price index 180 d 9 ms · frontier composition 1.2 ms · stats 7 ms · deployments cursor 0.4 ms · providers 2.5 ms · free-text search 16 ms. No query exceeded 200 ms, so **no migration 0004 was needed** (the 0003 partial indexes on `(is_backfill, occurred_at)`, `(benchmark_id, config_key, is_current)`, `family_id`, `canonical_id` are used). The admin trigram self-join (`/admin/entities/duplicates`, `/admin/entity-resolution`) takes ~120 ms in SQL + per-pair side-by-side lookups; it is admin-only and not cached.
217 220 - **Search**: `query.semantic` says whether an embedding was used; on embedding timeout (2 s) the API backs off to FTS-only for 5 minutes.
modified src/aiatlas/api/common.py +25 −4
@@ -271,8 +271,9 @@ ENTITY_COLS = entity_cols("e")
271 271 ENTITY_FROM = "entities e left join entities eo on eo.id = e.organization_id"
272 272
273 273 SUMMARY_ATTRS: dict[str, tuple[str, ...]] = {
274 − "model": ("family", "openness", "license", "parameter_count", "active_parameter_count", "context_length", "max_output_tokens", "modalities",
275 − "release_date", "status", "knowledge_cutoff", "hf_repo", "api_model_id"),
274 + "model": ("family", "openness", "license", "license_key", "parameter_count", "active_parameter_count", "context_length", "max_output_tokens", "modalities",
275 + "release_date", "status", "knowledge_cutoff", "hf_repo", "api_model_id", "reasoning", "tool_calling"),
276 + "artifact": ("openness", "license", "license_key", "parameter_count", "quant_format", "file_size_gb", "weights_dtype", "hf_repo", "release_date", "base_model"),
276 277 "company": ("country", "founded", "website", "org_kind"),
277 278 "organization": ("country", "founded", "website", "org_kind"),
278 279 "lab": ("country", "founded", "website", "org_kind"),
@@ -280,11 +281,23 @@ SUMMARY_ATTRS: dict[str, tuple[str, ...]] = {
280 281 "provider": ("website", "pricing_url"),
281 282 "benchmark": ("category", "metric", "unit"),
282 283 "hardware": ("kind", "memory_gb", "memory_bandwidth_gbs", "release_date", "manufacturer"),
283 − "framework": ("latest_version", "latest_release_at", "license", "language", "metric.stars"),
284 + "framework": ("kind", "latest_version", "latest_release_at", "license", "license_key", "language", "metric.stars", "repository_url", "homepage"),
285 + "library": ("kind", "latest_version", "latest_release_at", "license", "license_key", "language", "metric.stars", "repository_url", "homepage"),
286 + "runtime": ("kind", "latest_version", "latest_release_at", "license", "license_key", "language", "metric.stars", "repository_url"),
284 287 "repository": ("latest_version", "latest_release_at", "license", "language", "metric.stars"),
285 288 "paper": ("authors", "published_at", "arxiv_id", "primary_category"),
286 − "dataset": ("license", "modality"),
289 + "dataset": ("license", "license_key", "modality", "modalities", "publisher", "size", "access", "hf_repo", "hf_dataset", "task", "languages", "release_date"),
287 290 }
291 +
292 +
293 +def _fill_license_key(etype: str, attrs: dict[str, Any], out: dict[str, Any]) -> None:
294 + """Canonical `license_key` derived on read when the writer has not stored it yet (raw label → ontology key; unknown labels stay absent)."""
295 + if "license_key" not in out and attrs.get("license"):
296 + from aiatlas.ontology.licenses import normalize_license
297 +
298 + key = normalize_license(str(attrs["license"]))
299 + if key:
300 + out["license_key"] = key
288 301 COMPANY_TYPES = ("company", "organization", "lab", "university")
289 302 TYPE_LABELS = {"model": "Models", "company": "Companies", "organization": "Organizations", "lab": "Labs", "university": "Universities",
290 303 "provider": "Providers", "paper": "Papers", "benchmark": "Benchmarks", "hardware": "Hardware", "framework": "Frameworks",
@@ -331,6 +344,14 @@ def summary_attributes(entity_type: str, attrs: dict[str, Any] | None) -> dict[s
331 344 if k in attrs and attrs[k] not in (None, "", [], {}):
332 345 v = attrs[k]
333 346 out[k] = v[:5] if k == "authors" and isinstance(v, list) else v
347 + if "license_key" in keys:
348 + _fill_license_key(entity_type, attrs, out)
349 + if "kind" in keys and out.get("kind") and entity_type in ("framework", "library", "runtime"):
350 + from aiatlas.ontology.taxonomy import normalize_framework_kind
351 +
352 + canon = normalize_framework_kind(out["kind"])
353 + if canon and canon != out["kind"]:
354 + out["kind_raw"], out["kind"] = out["kind"], canon
334 355 return out
335 356
336 357
modified src/aiatlas/api/detail.py +14 −2
@@ -126,6 +126,18 @@ async def prices_of_provider(conn: AsyncConnection, provider_id: str) -> list[di
126 126 return [price_row(r) for r in rows]
127 127
128 128
129 +async def deployments_of_provider(conn: AsyncConnection, provider_id: str) -> list[dict[str, Any]]:
130 + """Current offers of this provider, cheapest output first."""
131 + rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.provider_id = :id and p.valid_to is null order by p.output_per_mtok asc nulls last, m.canonical_name limit 500", id=provider_id)
132 + return [deployment_row(r) for r in rows]
133 +
134 +
135 +async def removed_deployments_of_provider(conn: AsyncConnection, provider_id: str, *, days: int = 90) -> list[dict[str, Any]]:
136 + """Offers this provider closed in the last `days` days (status delisted)."""
137 + rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.provider_id = :id and p.valid_to > now() - make_interval(days => :d) order by p.valid_to desc limit 200", id=provider_id, d=days)
138 + return [deployment_row(r) for r in rows]
139 +
140 +
129 141 async def results_of_model(conn: AsyncConnection, model_id: str) -> list[dict[str, Any]]:
130 142 rows = await fetch_all(conn, f"select {RESULT_COLS} from {RESULT_FROM} where r.model_id = :id and r.valid_to is null order by b.canonical_name, r.observed_at desc limit 300", id=model_id)
131 143 return [result_row(r) for r in rows]
@@ -398,8 +410,8 @@ async def entity_detail(row: dict[str, Any]) -> dict[str, Any]:
398 410 groups[0] += [T("models", related_by_type, eid, "model", limit=50, include_org_children=True)]
399 411 groups[1] += [T("papers", related_by_type, eid, "paper", limit=24, include_org_children=True), T("repositories", related_by_type, eid, "repository", limit=24, include_org_children=True)]
400 412 elif etype == "provider":
401 − groups[0] += [T("prices", prices_of_provider, eid)]
402 − groups[1] += [T("models", _provider_models, eid)]
413 + groups[0] += [T("prices", prices_of_provider, eid), T("deployments", deployments_of_provider, eid)]
414 + groups[1] += [T("models", _provider_models, eid), T("removed", removed_deployments_of_provider, eid)]
403 415 elif etype == "benchmark":
404 416 groups[0] += [T("results", leaderboard, eid, limit=100)]
405 417 elif etype == "hardware":
modified src/aiatlas/api/routers/benchmarks.py +11 −3
@@ -110,8 +110,12 @@ async def list_benchmarks(request: Request, category: str | None = None) -> dict
110 110 @router.get("/matrix")
111 111 @cached(300)
112 112 async def matrix(request: Request, benchmarks: str | None = None, models: str | None = None, org: str | None = None, family: str | None = None,
113 − limit: int = Query(60, ge=1, le=300), comparable_only: int = Query(0, ge=0, le=1), min_cells: int = Query(3, ge=1, le=30)) -> dict[str, Any]:
114 − """Rows = canonical models, columns = benchmarks, cell = best current score in the primary comparability group (+ trust, config_key)."""
113 + limit: int = Query(60, ge=1, le=300), comparable_only: int = Query(0, ge=0, le=1), min_cells: int = Query(3, ge=1, le=30),
114 + since: str | None = Query(None, description="model release_date ≥ YYYY-MM-DD"), until: str | None = Query(None, description="model release_date ≤ YYYY-MM-DD")) -> dict[str, Any]:
115 + """Rows = canonical models, columns = benchmarks, cell = best current score in the primary comparability group (+ trust, config_key, observed_at)."""
116 + from aiatlas.api.common import parse_date
117 +
118 + since_d, until_d = parse_date(since, "since"), parse_date(until, "until")
115 119 async with connection() as conn:
116 120 groups = await all_primary_groups(conn)
117 121 meta = await benchmark_meta(conn)
@@ -150,11 +154,14 @@ async def matrix(request: Request, benchmarks: str | None = None, models: str |
150 154 for r in ranked:
151 155 if wanted_m is not None and r["model_id"] not in wanted_m:
152 156 continue
157 + rel = str((r.get("model_attrs") or {}).get("release_date") or "")[:10]
158 + if (since_d or until_d) and (not rel or (since_d and rel < since_d.isoformat()) or (until_d and rel > until_d.isoformat())):
159 + continue
153 160 level, _ = comparability(leader_cfg["config"] if leader_cfg else None, r.get("config"), leader_cfg["metric"] if leader_cfg else None, r.get("metric")) if leader_cfg else ("comparable", [])
154 161 if comparable_only and level != "comparable":
155 162 continue
156 163 cells[r["model_id"]][bid] = {"score": r["score"], "rank": r["rank"], "trust_level": r["trust_level"], "config_key": r["config_key"], "comparability": level,
157 − "result_id": r["id"]}
164 + "result_id": r["id"], "observed_at": r["observed_at"], "evaluated_at": r.get("evaluated_at")}
158 165 model_ref.setdefault(r["model_id"], {"id": r["model_id"], "slug": r["model_slug"], "name": r["model_name"], "organization": r.get("org_name"), "organization_slug": r.get("org_slug"),
159 166 "openness": (r.get("model_attrs") or {}).get("openness"), "release_date": (r.get("model_attrs") or {}).get("release_date")})
160 167 rows_out = []
@@ -165,6 +172,7 @@ async def matrix(request: Request, benchmarks: str | None = None, models: str |
165 172 "mean_rank": round(sum(x["rank"] for x in c.values()) / len(c), 2)})
166 173 rows_out.sort(key=lambda x: (-x["n_cells"], x["mean_rank"], x["model"]["name"] or ""))
167 174 return {"columns": columns, "rows": rows_out[:limit], "total_rows": len(rows_out), "comparable_only": bool(comparable_only), "min_cells": min_cells if wanted_m is None else None,
175 + "filters": {k: v for k, v in {"since": since, "until": until, "org": org, "family": family, "models": models}.items() if v},
168 176 "methodology": GROUPING_NOTE + " Each cell is the model's best current row in the benchmark's primary group; `mean_rank` is only a sort key, not a composite score."}
169 177
170 178
modified src/aiatlas/api/routers/deployments.py +13 −6
@@ -27,11 +27,18 @@ TOTAL_CAP = 10_000
27 27 @cached(300)
28 28 async def list_deployments(request: Request, model: str | None = None, provider: str | None = None, current: int = Query(1, ge=0, le=1), limit: int = Query(50, ge=1, le=200),
29 29 offset: int = Query(0, ge=0, le=10000), before: str | None = Query(None, description="cursor: valid_from of the last item"),
30 − org: str | None = None, sort: str = Query("valid_from", pattern="^(valid_from|output|input|model|provider)$")) -> dict[str, Any]:
31 − where = ["m.entity_type in ('model','artifact')", "m.merged_into is null"]
30 + org: str | None = None, sort: str = Query("valid_from", pattern="^(valid_from|output|input|model|provider)$"),
31 + status: str | None = Query(None, pattern="^(active|delisted|all)$", description="active = current rows (default when current=1), delisted = closed rows only, all = both")) -> dict[str, Any]:
32 + where = ["m.entity_type in ('model','artifact')"]
32 33 params: dict[str, Any] = {"lim": limit, "off": offset}
33 − if current:
34 + status = status or ("active" if current else "all")
35 + if status == "active":
34 36 where.append("p.valid_to is null")
37 + where.append("m.merged_into is null")
38 + elif status == "delisted":
39 + where.append("p.valid_to is not null")
40 + else:
41 + where.append("(p.valid_to is not null or m.merged_into is null)") # closed rows of since-merged models stay visible as history
35 42 before_ts = parse_ts(before, "before")
36 43 if before_ts is not None:
37 44 where.append("p.valid_from < :before")
@@ -46,14 +53,14 @@ async def list_deployments(request: Request, model: str | None = None, provider:
46 53 if org:
47 54 where.append("(mo.slug = :org or mo.id = :org or mo.canonical_name ilike :org)")
48 55 params["org"] = org
49 − order = {"valid_from": "p.valid_from desc, p.id desc", "output": "p.output_per_mtok asc nulls last, p.id", "input": "p.input_per_mtok asc nulls last, p.id",
50 − "model": "m.canonical_name asc, pv.canonical_name asc, p.id", "provider": "pv.canonical_name asc, m.canonical_name asc, p.id"}[sort]
56 + order = {"valid_from": ("p.valid_to desc nulls last, p.id desc" if status == "delisted" else "p.valid_from desc, p.id desc"), "output": "p.output_per_mtok asc nulls last, p.id",
57 + "input": "p.input_per_mtok asc nulls last, p.id", "model": "m.canonical_name asc, pv.canonical_name asc, p.id", "provider": "pv.canonical_name asc, m.canonical_name asc, p.id"}[sort]
51 58 where_sql = " and ".join(where)
52 59 rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {where_sql} order by {order} limit :lim offset :off", **params)
53 60 total = await fetch_val(conn, f"select count(*) from (select 1 from {PRICE_FROM} where {where_sql} limit {TOTAL_CAP}) t", **{k: v for k, v in params.items() if k not in ('lim', 'off')})
54 61 items = [deployment_row(r) for r in rows]
55 62 return {"items": items, "total": int(total or 0), "limit": limit, "offset": offset,
56 − "next_before": items[-1]["valid_from"] if len(items) == limit and sort == "valid_from" else None, "current": bool(current)}
63 + "next_before": items[-1]["valid_from"] if len(items) == limit and sort == "valid_from" and status != "delisted" else None, "current": bool(current), "status": status}
57 64
58 65
59 66 @router.get("/cost")
modified src/aiatlas/api/routers/explore.py +7 −1
@@ -60,7 +60,13 @@ async def explore_type(request: Request, type: str, p: Pagination = PAGINATION,
60 60 async with connection() as conn:
61 61 rows = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where {where_sql} order by {SORTS[sort]}, e.id limit :lim offset :off", lim=p.limit, off=p.offset, **params)
62 62 total = await fetch_val(conn, f"select count(*) from {ENTITY_FROM} where {where_sql}", **params)
63 − out = page([entity_summary(r) for r in rows], int(total or 0), p)
63 + items = []
64 + for r in rows:
65 + s = entity_summary(r) or {}
66 + if etype in ("framework", "library", "runtime", "tool", "agent"):
67 + s["kind"] = (s.get("attributes") or {}).get("kind") # canonical framework kind (normalised on read), null when the source did not say
68 + items.append(s)
69 + out = page(items, int(total or 0), p)
64 70 out["entity_type"] = etype
65 71 out["label"] = TYPE_LABELS.get(etype, etype.title())
66 72 return out
modified src/aiatlas/api/routers/families.py +20 −14
@@ -26,12 +26,22 @@ def _f(v: Any) -> float | None:
26 26 return None
27 27
28 28
29 −def _aggregate(members: list[dict[str, Any]], ranks: dict[str, dict[str, int]]) -> dict[str, Any]:
29 +def _ranks(groups: dict[str, dict[str, Any]]) -> dict[str, dict[str, dict[str, Any]]]:
30 + """{model_id: {benchmark slug: {rank, score, metric, config_key}}} over every benchmark's primary group."""
31 + out: dict[str, dict[str, dict[str, Any]]] = defaultdict(dict)
32 + for g in groups.values():
33 + for r in rank_rows(g["rows"], g["higher_is_better"]):
34 + out[r["model_id"]][g["benchmark"]["slug"]] = {"rank": r["rank"], "score": r["score"], "metric": g["metric"], "config_key": g["config_key"],
35 + "higher_is_better": g["higher_is_better"], "benchmark": g["benchmark"]["slug"]}
36 + return out
37 +
38 +
39 +def _aggregate(members: list[dict[str, Any]], ranks: dict[str, dict[str, dict[str, Any]]]) -> dict[str, Any]:
30 40 params = [p for p in (_f((m["attributes"] or {}).get("parameter_count")) for m in members) if p]
31 41 dates = sorted(str((m["attributes"] or {}).get("release_date") or "")[:10] for m in members if (m["attributes"] or {}).get("release_date"))
32 42 mods: set[str] = set()
33 43 lics: dict[str, int] = defaultdict(int)
34 − best: dict[str, tuple[int, str]] = {}
44 + best: dict[str, dict[str, Any]] = {}
35 45 for m in members:
36 46 a = m["attributes"] or {}
37 47 for k in ("modalities", "modalities_input", "modalities_output"):
@@ -41,12 +51,12 @@ def _aggregate(members: list[dict[str, Any]], ranks: dict[str, dict[str, int]])
41 51 if key or a.get("license"):
42 52 lics[key or str(a.get("license"))] += 1
43 53 for b, rk in ranks.get(m["id"], {}).items():
44 − if b not in best or rk < best[b][0]:
45 − best[b] = (rk, m["slug"])
54 + if b not in best or rk["rank"] < best[b]["rank"]:
55 + best[b] = {**rk, "model": m["slug"], "model_name": m["canonical_name"]}
46 56 return {"model_count": len(members), "first_release": dates[0] if dates else None, "last_release": dates[-1] if dates else None,
47 57 "param_range": {"min": min(params), "max": max(params)} if params else None, "modalities": sorted(mods),
48 58 "licenses": [{"key": k, "label": LICENSES[k].label if k in LICENSES else k, "models": n} for k, n in sorted(lics.items(), key=lambda kv: -kv[1])],
49 − "benchmark_best": {b: {"rank": rk, "model": slug} for b, (rk, slug) in sorted(best.items())}}
59 + "benchmark_best": dict(sorted(best.items()))}
50 60
51 61
52 62 async def _families(conn: Any) -> list[dict[str, Any]]:
@@ -71,10 +81,7 @@ async def list_families(request: Request, q: str | None = Query(None, max_length
71 81 members = await fetch_all(conn, f"""select e.id, e.slug, e.canonical_name, e.attributes, e.family_id, e.attributes->>'family' as label, e.organization_id, eo.slug as org_slug, eo.canonical_name as org_name
72 82 from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and (e.family_id is not null or e.attributes ? 'family')""")
73 83 groups = await all_primary_groups(conn)
74 − ranks: dict[str, dict[str, int]] = defaultdict(dict)
75 − for g in groups.values():
76 − for r in rank_rows(g["rows"], g["higher_is_better"]):
77 − ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"]
84 + ranks = _ranks(groups)
78 85 by_fid: dict[str, list[dict[str, Any]]] = defaultdict(list)
79 86 by_label: dict[str, list[dict[str, Any]]] = defaultdict(list)
80 87 for m in members:
@@ -127,15 +134,14 @@ async def family_detail(request: Request, slug: str, limit: int = Query(100, ge=
127 134 events = await fetch_all(conn, """select ev.entity_id, ev.event_type, ev.summary, ev.occurred_at, ev.importance from change_events ev where ev.entity_id = any(cast(:ids as text[]))
128 135 and ev.event_type in ('NEW_MODEL','RELEASE','DEPRECATION_ANNOUNCED','RETIREMENT_ANNOUNCED','STATUS_CHANGED') order by ev.occurred_at asc limit 500""", ids=ids) if ids else []
129 136 groups = await all_primary_groups(conn)
130 − ranks: dict[str, dict[str, int]] = defaultdict(dict)
131 − for g in groups.values():
132 − for r in rank_rows(g["rows"], g["higher_is_better"]):
133 − ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"]
137 + ranks = _ranks(groups)
134 138 mem_summ = []
135 139 for m in members:
136 140 a = m["attributes"] or {}
141 + rk = ranks.get(m["id"], {})
137 142 mem_summ.append({"model": entity_summary(m), "key_facts": {k: a.get(k) for k in ("release_date", "parameter_count", "active_parameter_count", "context_length", "openness", "license", "modalities", "status") if a.get(k) not in (None, "", [])},
138 − "benchmark_ranks": dict(sorted(ranks.get(m["id"], {}).items()))})
143 + "benchmark_ranks": {b: v["rank"] for b, v in sorted(rk.items())},
144 + "benchmark_best": [{"benchmark": b, "rank": v["rank"], "score": v["score"], "metric": v["metric"], "config_key": v["config_key"]} for b, v in sorted(rk.items(), key=lambda kv: kv[1]["rank"])]})
139 145 timeline = [{"date": str((m["attributes"] or {}).get("release_date") or "")[:10] or None, "kind": "release", "model": {"id": m["id"], "slug": m["slug"], "name": m["canonical_name"]}} for m in members]
140 146 timeline += [{"date": e["occurred_at"], "kind": e["event_type"], "summary": e["summary"], "model_id": e["entity_id"]} for e in events if e["event_type"] != "NEW_MODEL"]
141 147 timeline.sort(key=lambda x: str(x["date"] or ""))
modified src/aiatlas/api/routers/intelligence.py +43 −13
@@ -69,6 +69,23 @@ def _f(v: Any) -> float | None:
69 69 return None
70 70
71 71
72 +def price_delta(old: Any, new: Any) -> dict[str, Any] | None:
73 + """PRICE_CHANGED events carry `{input_per_mtok, output_per_mtok}` dicts (or a bare number). Returns % changes (output first) or None."""
74 + def pick(v: Any, k: str) -> float | None:
75 + return _f(v.get(k)) if isinstance(v, dict) else (_f(v) if k == "output_per_mtok" else None)
76 +
77 + out: dict[str, Any] = {}
78 + for k, label in (("output_per_mtok", "output"), ("input_per_mtok", "input")):
79 + o, n = pick(old, k), pick(new, k)
80 + if o and n is not None:
81 + out[f"{label}_percent"] = round((n - o) / o * 100, 2)
82 + out[f"{label}_from"], out[f"{label}_to"] = o, n
83 + if not out:
84 + return None
85 + out["percent"] = out.get("output_percent", out.get("input_percent"))
86 + return out
87 +
88 +
72 89 def _mods(attrs: dict[str, Any]) -> set[str]:
73 90 out: set[str] = set()
74 91 for k in ("modalities", "modalities_input", "modalities_output"):
@@ -171,9 +188,11 @@ async def frontier(request: Request, limit: int = Query(12, ge=1, le=50)) -> dic
171 188 for r in movements:
172 189 ev = change_event(r)
173 190 if ev["event_type"] == "PRICE_CHANGED":
174 − o, n = _f(ev.get("old_value")), _f(ev.get("new_value"))
175 − if o and n is not None and abs(n - o) / o >= 0.20:
176 − ev["percent_change"] = round((n - o) / o * 100, 1)
191 + d = price_delta(ev.get("old_value"), ev.get("new_value"))
192 + if d and d["percent"] is not None and abs(d["percent"]) >= 20:
193 + ev["percent_change"] = d["percent"]
194 + ev["price_delta"] = d
195 + ev["provider"] = (ev.get("meta") or {}).get("provider")
177 196 moves.append(ev)
178 197 else:
179 198 moves.append(ev)
@@ -268,6 +287,7 @@ async def pareto(request: Request, benchmark: str = Query(...), x: str = Query("
268 287 continue
269 288 points.append({"id": r["model_id"], "model": {"id": r["model_id"], "slug": r["model_slug"], "name": r["model_name"], "organization": r.get("org_name"), "openness": attrs.get("openness")},
270 289 "x": xv, "y": float(r["score"]), "rank": r["rank"], "trust_level": r["trust_level"], "config": {k: v for k, v in (r.get("config") or {}).items() if k in ("reasoning_effort", "reasoning", "variant", "evaluator")},
290 + "context_length": _f(attrs.get("context_length")), "parameter_count": _f(attrs.get("parameter_count")), "release_date": attrs.get("release_date"),
271 291 **({"provider": provider} if provider else {}), **({"estimated": True} if x == "memory_estimate" else {})})
272 292 front = pareto_frontier(points, maximize_y=g["higher_is_better"])
273 293 fset = set(front)
@@ -302,14 +322,21 @@ async def pulse(request: Request, days: int = Query(7, ge=1, le=90)) -> dict[str
302 322 (select count(*) from ev where event_type = 'DOCUMENT_CHANGED') as documents_changed,
303 323 (select count(distinct d.source_id) from snapshots s join documents d on d.id = s.document_id where s.observed_at > :since) as sources_observed,
304 324 (select count(*) from ev where event_type <> 'DOCUMENT_CHANGED') as events_total""", since=since)
305 − pct = await fetch_all(conn, "select old_value, new_value from change_events where is_backfill = false and occurred_at > :since and event_type = 'PRICE_CHANGED'", since=since)
325 + pct = await fetch_all(conn, f"select {EVENT_COLS}, ev.occurred_at from {EVENT_FROM} where ev.is_backfill = false and ev.occurred_at > :since and ev.event_type = 'PRICE_CHANGED' "
326 + f"order by ev.occurred_at desc limit 500", since=since)
327 + ctx_models = await fetch_all(conn, f"""select distinct on (e.id) {ENTITY_COLS} from {EVENT_FROM} where ev.is_backfill = false and ev.occurred_at > :since and ev.event_type = 'NEW_MODEL'
328 + and e.entity_type = 'model' and e.merged_into is null and {CONTEXT} >= 1000000 order by e.id limit 100""", since=since)
306 329 rows = await load_results(conn, current_only=False)
307 330 meta = await benchmark_meta(conn)
308 − moves = []
331 + moves: list[float] = []
332 + price_items: list[dict[str, Any]] = []
309 333 for r in pct:
310 − o, n = _f(r["old_value"]), _f(r["new_value"])
311 − if o and n is not None:
312 − moves.append((n - o) / o * 100)
334 + ev = change_event(r)
335 + d = price_delta(ev.get("old_value"), ev.get("new_value"))
336 + if d and d["percent"] is not None:
337 + moves.append(float(d["percent"]))
338 + price_items.append({"id": ev["id"], "summary": ev["summary"], "model": ev["entity"], "provider": (ev.get("meta") or {}).get("provider"), "provider_id": (ev.get("meta") or {}).get("provider_id"),
339 + "occurred_at": r.get("occurred_at"), "percent_change": d["percent"] if d else None, "delta": d, "source_url": ev.get("source_url")})
313 340 by_bench: dict[str, list[dict[str, Any]]] = defaultdict(list)
314 341 for r in rows:
315 342 by_bench[r["benchmark_id"]].append(r)
@@ -327,9 +354,10 @@ async def pulse(request: Request, days: int = Query(7, ge=1, le=90)) -> dict[str
327 354 "new_papers": {"value": int(c.get("new_papers") or 0), "definition": "papers with a NEW_PAPER event that occurred in the window"},
328 355 "provider_listings": {"value": int(c.get("provider_listings") or 0), "definition": "PROVIDER_LISTED events in the window"},
329 356 "provider_delistings": {"value": int(c.get("provider_delistings") or 0), "definition": "PROVIDER_DELISTED events in the window"},
330 − "price_changes": {"value": int(c.get("price_changes") or 0), "median_percent": round(statistics.median(moves), 2) if moves else None,
331 − "definition": "PRICE_CHANGED events in the window; median % change computed from the events' old/new numeric values when both are present"},
332 − "new_models_1m_context": {"value": int(c.get("new_models_1m_context") or 0), "definition": "new_models whose context_length is at least 1 000 000 tokens"},
357 + "price_changes": {"value": int(c.get("price_changes") or 0), "median_percent": round(statistics.median(moves), 2) if moves else None, "items": price_items[:50],
358 + "definition": "PRICE_CHANGED events in the window; % change = output price (else input) new vs old from the event's old/new values; median over events with both values"},
359 + "new_models_1m_context": {"value": int(c.get("new_models_1m_context") or 0), "items": [entity_summary(m) for m in ctx_models],
360 + "definition": "new_models whose context_length is at least 1 000 000 tokens"},
333 361 "new_benchmark_leaders": {"value": len(new_leaders), "items": new_leaders, "definition": "benchmarks whose primary-group leader (computed from results observed by each date) changed over the window"},
334 362 "documents_changed": {"value": int(c.get("documents_changed") or 0), "definition": "DOCUMENT_CHANGED events in the window"},
335 363 "sources_observed": {"value": int(c.get("sources_observed") or 0), "definition": "distinct sources with at least one snapshot taken in the window"},
@@ -384,8 +412,10 @@ async def open_models(request: Request, sort: str = Query("release", pattern="^(
384 412 where entity_type = 'model' and merged_into is null and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted') group by 1) x""")
385 413 lic_rows = await fetch_all(conn, """select coalesce(attributes->>'license_key', attributes->>'license') as raw, count(*) as n from entities where entity_type = 'model' and merged_into is null
386 414 and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted') and (attributes ? 'license' or attributes ? 'license_key') group by 1""")
415 + # released in the last 30 days (release_date when known; first_seen_at only for models without any release date)
387 416 new_30d = await fetch_val(conn, """select count(*) from entities where entity_type = 'model' and merged_into is null and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted')
388 − and (attributes->>'release_date' >= to_char((now() at time zone 'UTC') - interval '30 days', 'YYYY-MM-DD') or first_seen_at > now() - interval '30 days')""")
417 + and case when attributes->>'release_date' ~ '^\\d{4}' then left(attributes->>'release_date', 10) >= to_char((now() at time zone 'UTC') - interval '30 days', 'YYYY-MM-DD')
418 + else first_seen_at > now() - interval '30 days' end""")
389 419 ranks: dict[str, dict[str, int]] = defaultdict(dict)
390 420 for g in groups.values():
391 421 for r in rank_rows(g["rows"], g["higher_is_better"]):
@@ -413,7 +443,7 @@ async def open_models(request: Request, sort: str = Query("release", pattern="^(
413 443 lic_counts[k or f"raw:{lr['raw']}"] += int(lr["n"])
414 444 return {"items": items, "total": int(total or 0), "limit": limit, "offset": offset,
415 445 "summary": {"by_category": (summary or {}).get("by_cat") or {}, "by_license_top": [{"key": k, "label": LICENSES[k].label if k in LICENSES else k, "models": n} for k, n in sorted(lic_counts.items(), key=lambda kv: -kv[1])[:12]],
416 − "new_30d": int(new_30d or 0)},
446 + "new_30d": int(new_30d or 0), "new_30d_definition": "release_date within the last 30 days (first_seen_at only when no release date is known)"},
417 447 "note": "Universe = canonical models whose weights can be downloaded (open-weights, open-source, restricted-weights). hardware_fit values are ESTIMATES (see /methodology); "
418 448 "best_results are ranks inside each benchmark's primary comparability group — no composite score."}
419 449
modified src/aiatlas/api/routers/models.py +12 −2
@@ -178,8 +178,15 @@ async def list_models(request: Request, p: Pagination = PAGINATION, q: str | Non
178 178 where_sql = " and ".join(where)
179 179 async with connection() as conn:
180 180 rows = await fetch_all(conn, f"""select {ENTITY_COLS}, e.family_id, e.canonical_id, e.artifact_kind, e.identity_confidence,
181 − fam.slug as family_slug, fam.canonical_name as family_name, {entity_cols("can", "c_")}
182 − from {ENTITY_FROM} {FAMILY_JOIN} {CANONICAL_JOIN} where {where_sql} order by {order_sql}, e.id limit :lim offset :off""",
181 + fam.slug as family_slug, fam.canonical_name as family_name, {entity_cols("can", "c_")},
182 + bp.input_per_mtok as bp_input, bp.output_per_mtok as bp_output, bp.provider_slug as bp_provider_slug, bp.provider_name as bp_provider_name,
183 + bp.provider_id as bp_provider_id, bp.providers as bp_providers
184 + from {ENTITY_FROM} {FAMILY_JOIN} {CANONICAL_JOIN}
185 + left join lateral (select p.input_per_mtok, p.output_per_mtok, p.provider_id, pv.slug as provider_slug, pv.canonical_name as provider_name,
186 + (select count(distinct q.provider_id) from prices q where q.model_id = e.id and q.valid_to is null) as providers
187 + from prices p join entities pv on pv.id = p.provider_id
188 + where p.model_id = e.id and p.valid_to is null and p.output_per_mtok > 0 order by p.output_per_mtok asc, p.input_per_mtok asc nulls last limit 1) bp on true
189 + where {where_sql} order by {order_sql}, e.id limit :lim offset :off""",
183 190 lim=p.limit, off=p.offset, **params)
184 191 total = await fetch_val(conn, f"select count(*) from {ENTITY_FROM} {FAMILY_JOIN} where {where_sql}", **params)
185 192 items = []
@@ -191,6 +198,9 @@ async def list_models(request: Request, p: Pagination = PAGINATION, q: str | Non
191 198 s["artifact_kind"] = r.get("artifact_kind")
192 199 s["canonical"] = entity_summary(r, "c_")
193 200 s["identity_confidence"] = r.get("identity_confidence")
201 + s["best_price"] = ({"input_per_mtok": r["bp_input"], "output_per_mtok": r["bp_output"], "unit": "USD per 1M tokens",
202 + "provider": {"id": r["bp_provider_id"], "slug": r["bp_provider_slug"], "name": r["bp_provider_name"]}, "providers": int(r["bp_providers"] or 0),
203 + "note": "cheapest current offer by output price"} if r.get("bp_output") is not None else None)
194 204 items.append(s)
195 205 out = page(items, int(total or 0), p)
196 206 out["universe"] = "models+artifacts" if include_artifacts else "canonical models"
modified src/aiatlas/api/routers/prices.py +26 −8
@@ -19,6 +19,7 @@ from aiatlas.api.common import (
19 19 Pagination,
20 20 cached,
21 21 change_event,
22 + deployment_row,
22 23 page,
23 24 price_row,
24 25 resolve_id,
@@ -133,12 +134,16 @@ async def price_index(request: Request, days: int = Query(180, ge=7, le=1825)) -
133 134 and coalesce(p.context_length, case when m.attributes->>'context_length' ~ '^[0-9]+$' then (m.attributes->>'context_length')::bigint end) >= 1000000
134 135 order by p.output_per_mtok asc limit 1""", ids=sorted(frontier))
135 136 current_outputs = await fetch_all(conn, "select p.output_per_mtok as v from prices p join entities m on m.id = p.model_id where p.valid_to is null and p.output_per_mtok > 0 and m.merged_into is null")
136 − listings = await fetch_one(conn, """
137 − select (select count(*) from prices p where p.valid_from > now() - interval '30 days'
138 − and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_from < p.valid_from)) as new_listings_30d,
139 − (select count(*) from prices p where p.valid_to > now() - interval '30 days'
140 − and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_to is null)) as delistings_30d,
141 − (select count(*) from change_events where category = 'price' and is_backfill = false and occurred_at > now() - interval '30 days') as price_changes_30d""")
137 + new_where = ("p.valid_from > now() - interval '30 days' and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_from < p.valid_from) "
138 + "and m.entity_type in ('model','artifact') and m.merged_into is null")
139 + gone_where = "p.valid_to > now() - interval '30 days' and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_to is null)"
140 + new_rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {new_where} order by p.valid_from desc limit 50")
141 + new_n = await fetch_val(conn, f"select count(*) from {PRICE_FROM} where {new_where}")
142 + gone_rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {gone_where} order by p.valid_to desc limit 50")
143 + gone_n = await fetch_val(conn, f"select count(*) from {PRICE_FROM} where {gone_where}")
144 + chg_rows = await fetch_all(conn, f"select {EVENT_COLS}, ev.occurred_at from {EVENT_FROM} where ev.category = 'price' and ev.is_backfill = false and ev.occurred_at > now() - interval '30 days' "
145 + f"order by ev.occurred_at desc limit 50")
146 + chg_n = await fetch_val(conn, "select count(*) from change_events where category = 'price' and is_backfill = false and occurred_at > now() - interval '30 days'")
142 147 dist = []
143 148 vals = [float(r["v"]) for r in current_outputs]
144 149 for lo, hi in BUCKETS:
@@ -154,8 +159,9 @@ async def price_index(request: Request, days: int = Query(180, ge=7, le=1825)) -
154 159 return {"days": days, "series": out_series, "movers": [change_event(r) for r in movers],
155 160 "cheapest_frontier": _cheap(cheapest), "cheapest_frontier_1m_context": _cheap(cheapest_1m),
156 161 "distribution": {"metric": "output_per_mtok", "unit": "USD per 1M tokens", "buckets": dist, "offers": len(vals)},
157 − "new_listings_30d": int((listings or {}).get("new_listings_30d") or 0), "delistings_30d": int((listings or {}).get("delistings_30d") or 0),
158 − "price_changes_30d": int((listings or {}).get("price_changes_30d") or 0),
162 + "new_listings_30d": {"count": int(new_n or 0), "items": [deployment_row(r) for r in new_rows], "definition": "offers whose first row for (model, provider) opened in the last 30 days"},
163 + "delistings_30d": {"count": int(gone_n or 0), "items": [deployment_row(r) for r in gone_rows], "definition": "offers closed in the last 30 days with no live successor for (model, provider)"},
164 + "price_changes_30d": {"count": int(chg_n or 0), "items": [_price_event(r) for r in chg_rows], "definition": "price events (not back-filled) that occurred in the last 30 days"},
159 165 "frontier": {"composition": composition, "methodology": FRONTIER_METHODOLOGY},
160 166 "methodology": "Daily medians of live USD-per-1M-token prices across every provider offer valid at the end of each UTC day (offers with a zero or missing price are excluded "
161 167 "from that median). `median_open_output` covers models with openness open-weights/open-source; `median_embedding_input` covers models whose modalities include "
@@ -163,6 +169,18 @@ async def price_index(request: Request, days: int = Query(180, ge=7, le=1825)) -
163 169 "note": "Daily medians of live USD-per-1M-token input/output prices across all provider offers valid at the end of each UTC day."}
164 170
165 171
172 +def _price_event(r: dict[str, Any]) -> dict[str, Any]:
173 + from aiatlas.api.routers.intelligence import price_delta
174 +
175 + ev = change_event(r)
176 + d = price_delta(ev.get("old_value"), ev.get("new_value"))
177 + ev["occurred_at"] = r.get("occurred_at")
178 + ev["percent_change"] = d["percent"] if d else None
179 + ev["price_delta"] = d
180 + ev["provider"] = (ev.get("meta") or {}).get("provider")
181 + return ev
182 +
183 +
166 184 def _cheap(row: dict[str, Any] | None) -> dict[str, Any] | None:
167 185 if not row:
168 186 return None
modified src/aiatlas/services/cache.py +24 −5
@@ -1,7 +1,12 @@
1 −"""Redis helpers: API cache (prefix `aia:api:`), distributed locks (`aia:lock:`), heartbeats."""
1 +"""Redis helpers: API cache (prefix `aia:api:<db namespace>:`), distributed locks (`aia:lock:`), heartbeats.
2 +
3 +The API cache is namespaced by a hash of `DATABASE_URL`: two API processes pointing at different databases (dev on :8331, a production
4 +copy on :8332) but sharing one Redis db must never serve each other's bodies or derived id sets (that is how `/frontier` once reported
5 +frontier ids from another database and found no prices for them)."""
2 6 from __future__ import annotations
3 7
4 8 import asyncio
9 +import hashlib
5 10 import logging
6 11 from collections.abc import AsyncIterator
7 12 from contextlib import asynccontextmanager
@@ -14,6 +19,7 @@ from aiatlas.config import settings
14 19
15 20 log = logging.getLogger(__name__)
16 21 _redis: Redis | None = None
22 +_NS: str | None = None
17 23
18 24
19 25 def redis() -> Redis:
@@ -23,9 +29,21 @@ def redis() -> Redis:
23 29 return _redis
24 30
25 31
32 +def namespace() -> str:
33 + """8-hex namespace derived from the database URL (host, port, database name) — same DB ⇒ same cache, different DB ⇒ disjoint keys."""
34 + global _NS
35 + if _NS is None:
36 + _NS = hashlib.sha1(settings.database_url.encode()).hexdigest()[:8]
37 + return _NS
38 +
39 +
40 +def api_key(key: str) -> str:
41 + return f"aia:api:{namespace()}:{key}"
42 +
43 +
26 44 async def cache_get(key: str) -> Any | None:
27 45 try:
28 − raw = await redis().get(f"aia:api:{key}")
46 + raw = await redis().get(api_key(key))
29 47 return orjson.loads(raw) if raw else None
30 48 except Exception: # noqa: BLE001
31 49 return None
@@ -33,15 +51,16 @@ async def cache_get(key: str) -> Any | None:
33 51
34 52 async def cache_set(key: str, value: Any, ttl_s: int = 300) -> None:
35 53 try:
36 − await redis().set(f"aia:api:{key}", orjson.dumps(value, default=str), ex=ttl_s)
54 + await redis().set(api_key(key), orjson.dumps(value, default=str), ex=ttl_s)
37 55 except Exception: # noqa: BLE001
38 56 pass
39 57
40 58
41 59 async def cache_invalidate(prefix: str = "") -> int:
60 + """Delete `aia:api:<ns>:<prefix>*` — only this database's cache."""
42 61 n = 0
43 62 try:
44 − async for key in redis().scan_iter(match=f"aia:api:{prefix}*", count=500):
63 + async for key in redis().scan_iter(match=f"{api_key(prefix)}*", count=500):
45 64 await redis().delete(key)
46 65 n += 1
47 66 except Exception: # noqa: BLE001
@@ -96,4 +115,4 @@ async def close() -> None:
96 115 await asyncio.sleep(0)
97 116
98 117
99 −__all__ = ["cache_get", "cache_invalidate", "cache_set", "close", "heartbeat", "heartbeats", "lock", "redis"]
118 +__all__ = ["api_key", "cache_get", "cache_invalidate", "cache_set", "close", "heartbeat", "heartbeats", "lock", "namespace", "redis"]
modified src/aiatlas/services/frontier.py +7 −3
@@ -14,7 +14,7 @@ from typing import Any
14 14
15 15 from sqlalchemy.ext.asyncio import AsyncConnection
16 16
17 −from aiatlas.db import fetch_all
17 +from aiatlas.db import fetch_all, fetch_val
18 18 from aiatlas.ontology.benchmarks import (
19 19 CONDITION_KEYS,
20 20 METRICS,
@@ -278,8 +278,12 @@ async def frontier_model_ids(conn: AsyncConnection, *, use_cache: bool = True) -
278 278 """Frontier composition (see FRONTIER_METHODOLOGY). Returns (ids, sample counts)."""
279 279 if use_cache:
280 280 hit = await cache.cache_get(FRONTIER_CACHE_KEY)
281 − if hit:
282 − return set(hit["ids"]), hit["composition"]
281 + if hit and hit.get("ids"):
282 + # guard: the cached set must still be live canonical models of THIS database (cache keys are namespaced per DB, but a
283 + # canonicalisation run can merge ids away between refreshes) — otherwise recompute
284 + live = await fetch_val(conn, "select count(*) from entities where id = any(cast(:ids as text[])) and entity_type = 'model' and merged_into is null", ids=hit["ids"])
285 + if int(live or 0) == len(hit["ids"]):
286 + return set(hit["ids"]), hit["composition"]
283 287 since = (datetime.now(UTC) - timedelta(days=365)).date().isoformat()
284 288 recent = await fetch_all(conn, """
285 289 select e.id from entities e where e.entity_type = 'model' and e.merged_into is null and e.attributes->>'release_date' >= :since
modified tests/test_api_v11.py +65 −0
@@ -207,6 +207,71 @@ async def test_search_compiler_v2_response(client: AsyncClient) -> None:
207 207 assert "organization" not in unknown and "Foobarbaz" in unknown["unrecognised"]
208 208
209 209
210 +async def test_frontier_price_join_and_cache_namespace(client: AsyncClient) -> None:
211 + """Regression: frontier ids and prices come from the SAME database — cheapest_frontier must exist whenever frontier offers exist."""
212 + from aiatlas.services import cache
213 +
214 + assert cache.api_key("x").startswith(f"aia:api:{cache.namespace()}:") and len(cache.namespace()) == 8
215 + idx = (await client.get("/api/v1/prices/index", params={"days": 7})).json()
216 + offers_today = idx["series"][-1]["sample"]["frontier_offers"]
217 + if offers_today > 0:
218 + assert idx["cheapest_frontier"] is not None and idx["cheapest_frontier"]["output"] > 0
219 + cheapest = (await client.get("/api/v1/prices", params={"sort": "cheapest_frontier", "limit": 5})).json()
220 + assert cheapest["items"] and cheapest["items"][0]["output_per_mtok"] > 0
221 + fr = (await client.get("/api/v1/frontier")).json()
222 + assert fr["price_frontier"]["cheapest_output"] is not None and fr["price_frontier"]["frontier_models"] > 0
223 + for key in ("new_listings_30d", "delistings_30d", "price_changes_30d"):
224 + assert {"count", "items", "definition"} <= set(idx[key]) and len(idx[key]["items"]) <= 50
225 +
226 +
227 +async def test_pulse_items_and_price_delta(client: AsyncClient) -> None:
228 + from aiatlas.api.routers.intelligence import price_delta
229 +
230 + d = price_delta({"input_per_mtok": 1.0, "output_per_mtok": 4.0}, {"input_per_mtok": 0.5, "output_per_mtok": 5.0})
231 + assert d["output_percent"] == 25.0 and d["input_percent"] == -50.0 and d["percent"] == 25.0
232 + assert price_delta(None, {"output_per_mtok": 1}) is None and price_delta(2.0, 3.0)["percent"] == 50.0
233 + body = (await client.get("/api/v1/pulse", params={"days": 90})).json()
234 + pc = body["counters"]["price_changes"]
235 + assert "items" in pc and "median_percent" in pc and isinstance(body["counters"]["new_models_1m_context"]["items"], list)
236 + if pc["value"]:
237 + assert pc["items"] and {"summary", "percent_change", "provider", "occurred_at"} <= set(pc["items"][0])
238 +
239 +
240 +async def test_provider_detail_deployments_and_listing_status(client: AsyncClient) -> None:
241 + providers = (await client.get("/api/v1/providers")).json()["items"]
242 + slug = providers[0]["slug"]
243 + d = (await client.get(f"/api/v1/providers/{slug}")).json()
244 + assert "deployments" in d and "removed" in d
245 + outs = [x["prices"]["output"] for x in d["deployments"] if x["prices"]["output"] is not None]
246 + assert outs == sorted(outs) and all(x["status"] == "active" for x in d["deployments"]) and all(x["status"] == "delisted" for x in d["removed"])
247 + gone = (await client.get("/api/v1/deployments", params={"current": 0, "status": "delisted", "limit": 5})).json()
248 + assert gone["status"] == "delisted" and all(x["status"] == "delisted" for x in gone["items"])
249 + both = (await client.get("/api/v1/deployments", params={"current": 0, "limit": 200})).json()
250 + assert both["status"] == "all" and both["total"] >= gone["total"]
251 +
252 +
253 +async def test_models_best_price_and_family_scores_and_matrix_cells(client: AsyncClient) -> None:
254 + rows = (await client.get("/api/v1/models", params={"limit": 50, "sort": "cheapest"})).json()["items"]
255 + priced = [r for r in rows if r.get("best_price")]
256 + assert priced and {"input_per_mtok", "output_per_mtok", "provider", "providers"} <= set(priced[0]["best_price"]) and priced[0]["best_price"]["provider"]["slug"]
257 + fams = (await client.get("/api/v1/families", params={"limit": 3})).json()["items"]
258 + for b in fams[0]["benchmark_best"].values():
259 + assert {"rank", "score", "metric", "model"} <= set(b)
260 + fd = (await client.get(f"/api/v1/families/{fams[0]['slug']}")).json()
261 + assert all("benchmark_best" in m and all({"benchmark", "rank", "score"} <= set(x) for x in m["benchmark_best"]) for m in fd["members"])
262 + mx = (await client.get("/api/v1/benchmarks/matrix", params={"limit": 3, "since": "2024-01-01"})).json()
263 + assert mx["filters"]["since"] == "2024-01-01"
264 + for row in mx["rows"]:
265 + assert row["model"]["release_date"] and row["model"]["release_date"][:10] >= "2024-01-01"
266 + assert all("observed_at" in c for c in row["cells"].values() if c)
267 + pts = (await client.get("/api/v1/pareto", params={"benchmark": BENCH})).json()["points"]
268 + assert pts and {"context_length", "parameter_count"} <= set(pts[0])
269 + fw = (await client.get("/api/v1/explore/framework", params={"limit": 20})).json()
270 + assert all("kind" in it for it in fw["items"])
271 + op = (await client.get("/api/v1/open", params={"limit": 1})).json()
272 + assert op["summary"]["new_30d"] <= op["total"] and "new_30d_definition" in op["summary"]
273 +
274 +
210 275 async def test_etag_304(client: AsyncClient) -> None:
211 276 r1 = await client.get("/api/v1/stats")
212 277 etag = r1.headers.get("etag")
213 278