API 1.1 follow-ups: per-database cache namespace (frontier×price fix), pulse/index items, provider deployments, best_price, matrix/pareto fields
Root cause of the null cheapest_frontier: the Redis API cache was not scoped to the database, so the dev API (:8331) and the
production-copy API (:8332), which share redis db 5, served each other's cached bodies and the cached frontier id set —
prod joined dev ids against its prices (0 rows) while `sample.frontier_offers` came from a body cached by the other process.
Cache keys are now `aia:api:<sha1(DATABASE_URL)[:8]>:…` and the frontier id set is re-validated against live canonical models
on every cache hit. Regression test asserts cheapest_frontier / sort=cheapest_frontier / price_frontier are non-null whenever
frontier offers exist.
Also: PRICE_CHANGED old/new values are {input,output} dicts → price_delta() (pulse median_percent, pulse/index/frontier items
with % and provider); /prices/index 30-day counters → {count, items[], definition}; /providers/{slug} deployments + removed
(90 d); /deployments status=active|delisted|all (current=0 ⇒ all); framework/library kind (normalised) + dataset summary
attributes + license_key derived on read; /open new_30d by release_date (first_seen_at only when unknown); /models best_price
(one lateral join); /families benchmark_best with scores (list + members); /benchmarks/matrix since/until on release_date and
observed_at/evaluated_at per cell; /pareto points carry context_length, parameter_count, release_date. docs/API.md updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
13 changed files +280 −71
modified
docs/API.md
+13 −10
@@ -78,20 +78,20 @@ type Fit = { quantization: string; estimated: true; fits: boolean; estimated_mem | ||
| 78 | 78 | | `GET /entities/{slug}/graph?depth=1&limit=80` | `{ root, nodes: { id, slug, name, entity_type, organization_name }[], edges: { source, target, predicate }[] }` | |
| 79 | 79 | | `GET /entities/{slug}/sources` | `{ items: SourceRef[] }` | |
| 80 | 80 | | `GET /entities/{slug}/related?limit=` | `{ items: EntitySummary[] }` (same org / same family (`family_id` or label) / shared relations) | |
| 81 | −| `GET /models?q=&org=&family=&openness=&modality=&status=&min_params=&max_params=&min_context=&year_from=&year_to=&license=&sort=&order=&limit=&offset=&facets=1` | `Page<EntitySummary & { identity_confidence, family?: {id,slug,name}, canonical?: EntitySummary, artifact_kind? }> & { universe }` — **1.1:** default universe = `entity_type='model' and merged_into is null`; `include=artifacts`; `family=` matches `model_family` slug/id/name via `entities.family_id` (falls back to `attributes.family`); `license=` matches the canonical key (`attributes.license_key`) or any raw label the ontology maps to it; `reasoning=0|1`, `trust=high,medium,low` (identity_confidence), `sort=cheapest`. Facets add `families` (`{value,label,canonical,count}`), canonical `licenses` (`{value,label,category,count,raw_labels,raw?}`), `trust`, `definitions`. | | |
| 81 | +| `GET /models?q=&org=&family=&openness=&modality=&status=&min_params=&max_params=&min_context=&year_from=&year_to=&license=&sort=&order=&limit=&offset=&facets=1` | `Page<EntitySummary & { identity_confidence, best_price: { input_per_mtok, output_per_mtok, unit, provider: Org, providers, note } | null, family?: {id,slug,name}, canonical?: EntitySummary, artifact_kind? }> & { universe }` (`best_price` = cheapest current offer by output price, one lateral join; summary attributes gain `license_key` derived on read, `reasoning`, `tool_calling`) — **1.1:** default universe = `entity_type='model' and merged_into is null`; `include=artifacts`; `family=` matches `model_family` slug/id/name via `entities.family_id` (falls back to `attributes.family`); `license=` matches the canonical key (`attributes.license_key`) or any raw label the ontology maps to it; `reasoning=0|1`, `trust=high,medium,low` (identity_confidence), `sort=cheapest`. Facets add `families` (`{value,label,canonical,count}`), canonical `licenses` (`{value,label,category,count,raw_labels,raw?}`), `trust`, `definitions`. | | |
| 82 | 82 | | `GET /companies?q=&country=&kind=&sort=&limit=&offset=&facets=1` | `Page<EntitySummary & { model_count, paper_count }>` — covers `company|organization|lab|university`; `total` = `/stats.organizations_total` | |
| 83 | 83 | | `GET /papers?…` | `Page<EntitySummary>` | |
| 84 | 84 | | `GET /providers` | `{ items: (EntitySummary & { model_count, price_count, min_input_per_mtok, min_output_per_mtok, input_price_distribution: {min,p25,median,p75,max,n}|null, output_price_distribution, models_added_30d, models_removed_30d, price_changes_30d, organizations_covered, features_supported: string[], feature_keys: string[] })[], note }` — **1.1** aggregates | |
| 85 | 85 | | `GET /prices?model=&provider=&sort=&limit=&offset=¤t=1&family=&org=&modality=` | `Page<Price>` — **1.1:** `family`, `org`, `modality` filters; `sort=cheapest_frontier` (frontier models only, cheapest output first, + `methodology`) | |
| 86 | 86 | | `GET /prices/history?model=&provider=` | `{ items: Price[] }` | |
| 87 | −| `GET /prices/index?days=180` | **AI Price Index (1.1)** `{ days, series: { day, median_input, median_output, median_frontier_output, median_frontier_input, median_open_output, median_embedding_input, min_input, max_input, min_frontier_output, models, offers, sample: { models, offers, frontier_models, frontier_offers, open_models, embedding_models } }[], movers: ChangeEvent[], cheapest_frontier: { model, provider, output, input, context_length, price_id }|null, cheapest_frontier_1m_context, distribution: { metric, unit, buckets: { from, to, label, offers }[], offers }, new_listings_30d, delistings_30d, price_changes_30d, frontier: { composition, methodology }, methodology, note }` — v1 keys kept | | |
| 87 | +| `GET /prices/index?days=180` | **AI Price Index (1.1)** `{ days, series: { day, median_input, median_output, median_frontier_output, median_frontier_input, median_open_output, median_embedding_input, min_input, max_input, min_frontier_output, models, offers, sample: { models, offers, frontier_models, frontier_offers, open_models, embedding_models } }[], movers: ChangeEvent[], cheapest_frontier: { model, provider, output, input, context_length, price_id }|null, cheapest_frontier_1m_context, distribution: { metric, unit, buckets: { from, to, label, offers }[], offers }, new_listings_30d: { count, items: Deployment[], definition }, delistings_30d: { count, items: Deployment[], definition }, price_changes_30d: { count, items: (ChangeEvent & { percent_change, price_delta, provider })[], definition }, frontier: { composition, methodology }, methodology, note }` — v1 keys kept; the three 30-day counters became objects (read `.count`), items capped at 50 | | |
| 88 | 88 | | `GET /benchmarks?category=` | `{ items: { id, entity_type, slug, name, category, family, variant, metric, unit, direction, attributes, result_count, model_count, leader: LeaderboardRow|null, second, top: {model, score}|null, primary_group: Group|null, groups: Group[], trust_mix, trust_labels }[], total, note }` — **1.1:** `result_count` = current rows, `model_count` = distinct canonical models, `leader` from the primary metric group | |
| 89 | 89 | | `GET /benchmarks/{slug}` | `EntityDetail` + `family, variant, metric, direction, category, groups, primary_group, result_count, model_count, leaderboard (top 25), trust_mix` | |
| 90 | 90 | | `GET /benchmarks/{slug}/results?limit=&offset=&config=&history=&metric=&config_key=` | `Page<BenchmarkResult>` — v1 behaviour (one row per result) + `metric` / `config_key` filters | |
| 91 | 91 | | `GET /benchmarks/{slug}/history?model=` | `{ benchmark, items: BenchmarkResult[] }` | |
| 92 | 92 | | `GET /hardware?…` | `Page<EntitySummary>` + facets | |
| 93 | 93 | | `GET /hardware/fit?memory_gb=&quant=&context=&limit=&openness=` | v1 estimate (unchanged) | |
| 94 | −| `GET /explore/types` · `GET /explore/{type}` | unchanged (`artifact`, `model_family` appear as types once present) | | |
| 94 | +| `GET /explore/types` · `GET /explore/{type}` | unchanged (`artifact`, `model_family` appear as types once present); **1.1:** framework/library/runtime/tool items expose `kind` (canonical framework kind, normalised on read, `kind_raw` when the source label differed) and dataset summaries surface `publisher, size, access, hf_repo, modality, license_key…` when recorded | | |
| 95 | 95 | | `GET /changes?category=&type=&entity_type=&importance_min=&since=&until=&q=&entity=&limit=50&before=&offset=&include_documents=0&include_backfill=0&date_field=occurred` | `Page<ChangeEvent> & { next_before, date_field, include_backfill }` — **1.1:** defaults `is_backfill=false`, `occurred_at`; cursor/filters follow `date_field` | |
| 96 | 96 | | `GET /changes/daily?date=&per_section=30&include_backfill=0&date_field=occurred` | `{ date, counts, total, sections (v1 by category), today: { key, label, items: (ChangeEvent & { sources: n, documents: string[], grouped_events, event_ids? })[], total }[], new_models, labels, backfill_excluded, previous_day, next_day, note }` — **Today in AI 2.0** sections: `MAJOR_RELEASES, OPEN_WEIGHT_RELEASES, PRICE_MOVES, BENCHMARK_MOVES, MODEL_CHANGES, RESEARCH, DEPRECATIONS, PROVIDER_CHANGES, HARDWARE`; events sharing a `group_key` are folded into one item | |
| 97 | 97 | | `GET /changes/categories?days=7&include_backfill=0` | `{ items: { category, event_type, count }[] }` | |
@@ -124,6 +124,9 @@ EntitySummary & { | ||
| 124 | 124 | family_id?: string | null; identity_confidence?: "high"|"medium"|"low" |
| 125 | 125 | // artifact (1.1) |
| 126 | 126 | canonical?: EntitySummary | null; artifact_kind?: "checkpoint"|"quantization"|"conversion"|"packaging"|null |
| 127 | + // provider (1.1) | |
| 128 | + deployments?: Deployment[] // current offers of this provider, cheapest output first | |
| 129 | + removed?: Deployment[] // offers closed in the last 90 days (status delisted) | |
| 127 | 130 | // company / provider / hardware / framework / model_family |
| 128 | 131 | models?: Page<EntitySummary>; papers?; repositories? |
| 129 | 132 | } |
@@ -139,18 +142,18 @@ EntitySummary & { | ||
| 139 | 142 | | `GET /claims/{id}` | `{ claim, entity, property, chain: { previous: Claim[], superseding: Claim[], conflicting: Claim[], history_count }, source: { id, name, domain, tier, url, snapshot_id, observed_at }, extractor: { name, version, confidence }, run_id, evidence: { snapshot_id, document_url, archived, snapshot_observed_at, document_title, doc_type }, note }` | |
| 140 | 143 | | `GET /benchmarks/{slug}/leaderboard?metric=&config_key=&trust=&org=&since=&until=&comparable_only=0&limit=100&offset=0` | `{ benchmark, group: Group, groups: Group[], items: LeaderboardRow[], total, limit, offset, comparable_only, filters, history_available, methodology }` — ONE row per canonical model (best current row in the group; default group = primary metric, most-populated config); `delta_rank` vs the ranking of the closed rows of the same group when history exists | |
| 141 | 144 | | `GET /benchmarks/{slug}/frontier?metric=&config_key=` | `{ benchmark, series: { group: Group, primary: bool, points: { date, model: ModelRef, score, trust_level, config, result_id }[], current_leader }[], generated_at, methodology }` — leader history per group (a point each time a new best appears, ordered by `coalesce(evaluated_at, observed_at)`, closed rows included) | |
| 142 | −| `GET /benchmarks/matrix?benchmarks=a,b,c&models=…&org=&family=&limit=60&comparable_only=0&min_cells=3` | `{ columns: { id, slug, name, category, metric, config_key, group_label, higher_is_better, n_models }[], rows: { model: {…}, cells: Record<benchmarkId, { score, rank, trust_level, config_key, comparability, result_id } | null>, n_cells, mean_rank }[], total_rows, comparable_only, min_cells, methodology }` — default columns = 12 benchmarks with most current results, default rows = models with ≥ `min_cells` cells (`mean_rank` is a sort key, not a score) | | |
| 143 | −| `GET /deployments?model=&provider=&org=¤t=1&sort=valid_from|output|input|model|provider&limit=50&offset=&before=<valid_from cursor>` | `Page<Deployment> & { next_before, current }` | | |
| 145 | +| `GET /benchmarks/matrix?benchmarks=a,b,c&models=…&org=&family=&since=&until=&limit=60&comparable_only=0&min_cells=3` | `{ columns: { id, slug, name, category, metric, config_key, group_label, higher_is_better, n_models }[], rows: { model: {…}, cells: Record<benchmarkId, { score, rank, trust_level, config_key, comparability, result_id, observed_at, evaluated_at } | null>, n_cells, mean_rank }[], total_rows, comparable_only, min_cells, filters, methodology }` (`since`/`until` bound the model `release_date`) — default columns = 12 benchmarks with most current results, default rows = models with ≥ `min_cells` cells (`mean_rank` is a sort key, not a score) | | |
| 146 | +| `GET /deployments?model=&provider=&org=¤t=1&status=active|delisted|all&sort=valid_from|output|input|model|provider&limit=50&offset=&before=<valid_from cursor>` | `Page<Deployment> & { next_before, current, status }` — `current=0` ⇒ `status=all` (active + delisted rows); `status=delisted` returns closed rows only (newest `valid_to` first) | | |
| 144 | 147 | | `GET /cost?model=&provider=&input_tokens=1000&output_tokens=500&requests_per_day=1000&cached_share=0..1&batch=0|1` | `{ model, inputs, items: { deployment: Deployment, cost: { per_request, daily, monthly, annual, effective_input_per_mtok, effective_output_per_mtok, per_request_fee, inputs, notes: string[] } }[], total, currency, methodology, note }` — cached/batch prices used only when published (else a `notes` entry) | |
| 145 | 148 | | `GET /cost/context?tokens=1000000&limit=50&model=&org=` | `{ tokens, items: { deployment, context_length, context_source: "offer"|"model attribute", cost_usd }[], total, currency, methodology }` — offers whose context ≥ tokens, cheapest first | |
| 146 | 149 | | `GET /frontier?limit=12` | `{ latest_major_models: ChangeEvent[], benchmark_frontier: { benchmark, group, leader, second, gap }[], price_frontier: { cheapest_output: Deployment, cheapest_output_1m_context, frontier_models, composition }, context_frontier: { model, context_length }[], open_weight_frontier: { items: { model, best_rank, best_rank_on, parameter_count, context_length, ranks }[], dimensions, note }, efficiency_frontier: { quality: { benchmark, group }, x, points: { id, model, x, y, rank, trust_level, pareto }[], frontier: string[] }, agentic_frontier: { benchmark, group, leaders: LeaderboardRow[] }[], multimodal_frontier: { model, modalities, top10_on }[], recent_frontier_movements: (ChangeEvent & { percent_change? })[], generated_at, methodology }` | |
| 147 | −| `GET /pareto?benchmark=&x=output_price|input_price|parameter_count|context_length|memory_estimate&y=score&metric=&config_key=&org=&family=&openness=` | `{ benchmark, group, groups, x: { key, label }, y, points: { id, model, x, y, rank, trust_level, config, provider?, estimated?, pareto }[], frontier: string[], methodology }` — `x=latency` → 400 (not stored, never estimated) | | |
| 148 | −| `GET /pulse?days=7` | `{ days, since, until, counters: Record<key, { value, definition, median_percent?, items? }>, note }` — keys: `new_models, new_open_weight_models, new_artifacts, new_papers, provider_listings, provider_delistings, price_changes, new_models_1m_context, new_benchmark_leaders, documents_changed, sources_observed, events_total`; all `is_backfill=false`, `occurred_at` in window | | |
| 149 | −| `GET /open?sort=release|params|context|rank|name|downloads&license=&min_params=&max_params=&min_context=&modality=&days=&openness=&limit=&offset=` | `Page<{ model, licence: LicenseInfo & { raw } | { key: null, raw, note }, dimensions, best_results: { benchmark, rank }[], best_rank, hardware_fit: { "4bit_64gb": Fit, "8bit_128gb": Fit, estimated: true }, providers, cheapest_output_per_mtok }> & { summary: { by_category, by_license_top, new_30d }, note }` — universe = downloadable weights (open-weights, open-source, restricted-weights) | | |
| 150 | +| `GET /pareto?benchmark=&x=output_price|input_price|parameter_count|context_length|memory_estimate&y=score&metric=&config_key=&org=&family=&openness=` | `{ benchmark, group, groups, x: { key, label }, y, points: { id, model, x, y, rank, trust_level, config, context_length, parameter_count, release_date, provider?, estimated?, pareto }[], frontier: string[], methodology }` — `x=latency` → 400 (not stored, never estimated) | | |
| 151 | +| `GET /pulse?days=7` | `{ days, since, until, counters: Record<key, { value, definition, median_percent?, items? }>, note }` — keys: `new_models, new_open_weight_models, new_artifacts, new_papers, provider_listings, provider_delistings, price_changes (items: { id, summary, model, provider, occurred_at, percent_change, delta }[], median_percent), new_models_1m_context (items: EntitySummary[]), new_benchmark_leaders (items), documents_changed, sources_observed, events_total`; all `is_backfill=false`, `occurred_at` in window. Price deltas are read from the events' `{input_per_mtok, output_per_mtok}` old/new values (output first) | | |
| 152 | +| `GET /open?sort=release|params|context|rank|name|downloads&license=&min_params=&max_params=&min_context=&modality=&days=&openness=&limit=&offset=` | `Page<{ model, licence: LicenseInfo & { raw } | { key: null, raw, note }, dimensions, best_results: { benchmark, rank }[], best_rank, hardware_fit: { "4bit_64gb": Fit, "8bit_128gb": Fit, estimated: true }, providers, cheapest_output_per_mtok }> & { summary: { by_category, by_license_top, new_30d, new_30d_definition }, note }` (`new_30d` = `release_date` in the last 30 days, `first_seen_at` only when no release date is known; `licence.key` is derived on read from the raw label when `license_key` is missing) — universe = downloadable weights (open-weights, open-source, restricted-weights) | | |
| 150 | 153 | | `GET /find-a-model?use_case=coding|reasoning|agentic|long_context|vision|low_cost|local|embeddings|chat&deployment=local|api|any&memory_gb=&quant=&context_min=&license=commercial|any&openness=&max_input_price=&max_output_price=&modalities=&limit=30` | `{ matches: { model, why: string[], observed: {…}, estimated_fit?: Fit, deployments?: Deployment[] }[], total, filters_applied, rules, note }` — deterministic rules (see `/methodology.find_a_model`), sorted by satisfied criteria → best benchmark rank → release date; **no composite score** | |
| 151 | 154 | | `GET /run-locally?memory_gb=&gpu_count=1|2|4|8&quant=&context=8192&batch=1&platform=apple|nvidia|amd|any&use_case=&openness=&limit=` | `{ inputs, estimated: true, assumptions, counts, items: { model, fit: Fit, artifacts: { artifact, quant_format, file_size_gb, weights_source: "observed"|"estimated", fit: Fit }[], artifact_count }[], note }` — artifacts with an observed `file_size_gb` use it as the weights size | |
| 152 | 155 | | `GET /hardware/{slug}/fit?quant=&context=&memory_gb=&gpu_count=&limit=&openness=` | `{ hardware, memory_options_gb, inputs, estimated: true, assumptions, runtimes, counts, items: ({ model } & Fit)[] }` | |
| 153 | −| `GET /families?q=&org=&sort=models|name|last_release&limit=&offset=` | `Page<{ id, slug, name, canonical: bool, entity_type: "model_family", organization, model_count, first_release, last_release, param_range, modalities, licenses, benchmark_best }> & { note }` — `model_family` entities + legacy `attributes.family` labels (`canonical: false`) | | |
| 156 | +| `GET /families?q=&org=&sort=models|name|last_release&limit=&offset=` | `Page<{ id, slug, name, canonical: bool, entity_type: "model_family", organization, model_count, first_release, last_release, param_range, modalities, licenses, benchmark_best: Record<benchmark, { rank, score, metric, config_key, model, model_name }> }> & { note }` (members in `/families/{slug}` carry `benchmark_ranks` and `benchmark_best: { benchmark, rank, score, metric, config_key }[]`) — `model_family` entities + legacy `attributes.family` labels (`canonical: false`) | | |
| 154 | 157 | | `GET /families/{slug}?limit=` | `{ id, slug, name, canonical, summary, …aggregates, members: { model, key_facts, benchmark_ranks }[], artifacts_count, providers, lineage: { source, target, predicate }[], timeline, note }` — accepts a family label pre-canonicalisation | |
| 155 | 158 | | `GET /graph/explore?node=&mode=lineage|research|company|benchmark|dataset|provider|hardware&depth=1|2&limit=150` | `{ root, mode, depth, predicates, nodes: { id, slug, name, entity_type, org, org_slug, level, artifact_kind, attributes }[], edges: { source, target, predicate, attributes, tier }[], truncated, counts }` — never more than `limit` nodes | |
| 156 | 159 | | `GET /time-machine?date=YYYY-MM-DD&scope=models|prices|benchmarks|hardware|all&limit=50` | `{ date, scope, first_entity_at, reconstructed, note, models?: { items: { model, attributes_as_of, observed_then, reconstructed }[], total }, prices?: { items: Deployment[], total }, benchmarks?: { leaders: { benchmark, leader }[] }, hardware?: { items } }` — before `first_entity_at` the state is reconstructed from claims/release dates (`reconstructed: true`) | |
@@ -211,7 +214,7 @@ None. No v1 route, parameter, field or type was removed or retyped in 1.1. | ||
| 211 | 214 | ## Implementation notes (`src/aiatlas/api`) |
| 212 | 215 | |
| 213 | 216 | - **Validation errors**: parameters rejected by FastAPI/pydantic typing or bounds return **422** with `{ detail, errors }`; semantic errors raised by the routes return **400**. |
| 214 | −- **Caching**: public GETs are cached in Redis (`aia:api:<path>?<sorted query>`, 60–600 s) and served with a weak ETag (`W/"sha1(body)"`) computed on the uncompressed body inside GZip; `/stats` stays at 60 s. Cache invalidation: `services.cache.cache_invalidate(prefix)`; `POST /admin/cache/flush?after_run=1`. | |
| 217 | +- **Caching**: public GETs are cached in Redis (`aia:api:<ns>:<path>?<sorted query>`, 60–600 s) where `<ns>` = 8-hex sha1 of `DATABASE_URL` — two API processes on different databases (dev :8331, prod copy :8332) sharing one Redis db never read each other's bodies or derived id sets (the frontier id set is also re-validated against live canonical models on every cache hit) — and served with a weak ETag (`W/"sha1(body)"`) computed on the uncompressed body inside GZip; `/stats` stays at 60 s. Cache invalidation: `services.cache.cache_invalidate(prefix)`; `POST /admin/cache/flush?after_run=1`. | |
| 215 | 218 | - **Connections**: `EntityDetail` runs its blocks in ≤ 4 concurrent groups (one pooled connection each, sequential inside a group) instead of one connection per block. |
| 216 | 219 | - **Performance (2026-09-12, prod copy, EXPLAIN ANALYZE)**: leaderboard load 8 ms · matrix / all groups 8 ms · models listing 0.5 ms (+ count 1.2 ms, facets ≤ 4 ms) · changes feed 0.8 ms (count 3 ms) · daily sections 0.4 ms · price index 180 d 9 ms · frontier composition 1.2 ms · stats 7 ms · deployments cursor 0.4 ms · providers 2.5 ms · free-text search 16 ms. No query exceeded 200 ms, so **no migration 0004 was needed** (the 0003 partial indexes on `(is_backfill, occurred_at)`, `(benchmark_id, config_key, is_current)`, `family_id`, `canonical_id` are used). The admin trigram self-join (`/admin/entities/duplicates`, `/admin/entity-resolution`) takes ~120 ms in SQL + per-pair side-by-side lookups; it is admin-only and not cached. |
| 217 | 220 | - **Search**: `query.semantic` says whether an embedding was used; on embedding timeout (2 s) the API backs off to FTS-only for 5 minutes. |
modified
src/aiatlas/api/common.py
+25 −4
@@ -271,8 +271,9 @@ ENTITY_COLS = entity_cols("e") | ||
| 271 | 271 | ENTITY_FROM = "entities e left join entities eo on eo.id = e.organization_id" |
| 272 | 272 | |
| 273 | 273 | SUMMARY_ATTRS: dict[str, tuple[str, ...]] = { |
| 274 | − "model": ("family", "openness", "license", "parameter_count", "active_parameter_count", "context_length", "max_output_tokens", "modalities", | |
| 275 | − "release_date", "status", "knowledge_cutoff", "hf_repo", "api_model_id"), | |
| 274 | + "model": ("family", "openness", "license", "license_key", "parameter_count", "active_parameter_count", "context_length", "max_output_tokens", "modalities", | |
| 275 | + "release_date", "status", "knowledge_cutoff", "hf_repo", "api_model_id", "reasoning", "tool_calling"), | |
| 276 | + "artifact": ("openness", "license", "license_key", "parameter_count", "quant_format", "file_size_gb", "weights_dtype", "hf_repo", "release_date", "base_model"), | |
| 276 | 277 | "company": ("country", "founded", "website", "org_kind"), |
| 277 | 278 | "organization": ("country", "founded", "website", "org_kind"), |
| 278 | 279 | "lab": ("country", "founded", "website", "org_kind"), |
@@ -280,11 +281,23 @@ SUMMARY_ATTRS: dict[str, tuple[str, ...]] = { | ||
| 280 | 281 | "provider": ("website", "pricing_url"), |
| 281 | 282 | "benchmark": ("category", "metric", "unit"), |
| 282 | 283 | "hardware": ("kind", "memory_gb", "memory_bandwidth_gbs", "release_date", "manufacturer"), |
| 283 | − "framework": ("latest_version", "latest_release_at", "license", "language", "metric.stars"), | |
| 284 | + "framework": ("kind", "latest_version", "latest_release_at", "license", "license_key", "language", "metric.stars", "repository_url", "homepage"), | |
| 285 | + "library": ("kind", "latest_version", "latest_release_at", "license", "license_key", "language", "metric.stars", "repository_url", "homepage"), | |
| 286 | + "runtime": ("kind", "latest_version", "latest_release_at", "license", "license_key", "language", "metric.stars", "repository_url"), | |
| 284 | 287 | "repository": ("latest_version", "latest_release_at", "license", "language", "metric.stars"), |
| 285 | 288 | "paper": ("authors", "published_at", "arxiv_id", "primary_category"), |
| 286 | − "dataset": ("license", "modality"), | |
| 289 | + "dataset": ("license", "license_key", "modality", "modalities", "publisher", "size", "access", "hf_repo", "hf_dataset", "task", "languages", "release_date"), | |
| 287 | 290 | } |
| 291 | + | |
| 292 | + | |
| 293 | +def _fill_license_key(etype: str, attrs: dict[str, Any], out: dict[str, Any]) -> None: | |
| 294 | + """Canonical `license_key` derived on read when the writer has not stored it yet (raw label → ontology key; unknown labels stay absent).""" | |
| 295 | + if "license_key" not in out and attrs.get("license"): | |
| 296 | + from aiatlas.ontology.licenses import normalize_license | |
| 297 | + | |
| 298 | + key = normalize_license(str(attrs["license"])) | |
| 299 | + if key: | |
| 300 | + out["license_key"] = key | |
| 288 | 301 | COMPANY_TYPES = ("company", "organization", "lab", "university") |
| 289 | 302 | TYPE_LABELS = {"model": "Models", "company": "Companies", "organization": "Organizations", "lab": "Labs", "university": "Universities", |
| 290 | 303 | "provider": "Providers", "paper": "Papers", "benchmark": "Benchmarks", "hardware": "Hardware", "framework": "Frameworks", |
@@ -331,6 +344,14 @@ def summary_attributes(entity_type: str, attrs: dict[str, Any] | None) -> dict[s | ||
| 331 | 344 | if k in attrs and attrs[k] not in (None, "", [], {}): |
| 332 | 345 | v = attrs[k] |
| 333 | 346 | out[k] = v[:5] if k == "authors" and isinstance(v, list) else v |
| 347 | + if "license_key" in keys: | |
| 348 | + _fill_license_key(entity_type, attrs, out) | |
| 349 | + if "kind" in keys and out.get("kind") and entity_type in ("framework", "library", "runtime"): | |
| 350 | + from aiatlas.ontology.taxonomy import normalize_framework_kind | |
| 351 | + | |
| 352 | + canon = normalize_framework_kind(out["kind"]) | |
| 353 | + if canon and canon != out["kind"]: | |
| 354 | + out["kind_raw"], out["kind"] = out["kind"], canon | |
| 334 | 355 | return out |
| 335 | 356 | |
| 336 | 357 | |
modified
src/aiatlas/api/detail.py
+14 −2
@@ -126,6 +126,18 @@ async def prices_of_provider(conn: AsyncConnection, provider_id: str) -> list[di | ||
| 126 | 126 | return [price_row(r) for r in rows] |
| 127 | 127 | |
| 128 | 128 | |
| 129 | +async def deployments_of_provider(conn: AsyncConnection, provider_id: str) -> list[dict[str, Any]]: | |
| 130 | + """Current offers of this provider, cheapest output first.""" | |
| 131 | + rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.provider_id = :id and p.valid_to is null order by p.output_per_mtok asc nulls last, m.canonical_name limit 500", id=provider_id) | |
| 132 | + return [deployment_row(r) for r in rows] | |
| 133 | + | |
| 134 | + | |
| 135 | +async def removed_deployments_of_provider(conn: AsyncConnection, provider_id: str, *, days: int = 90) -> list[dict[str, Any]]: | |
| 136 | + """Offers this provider closed in the last `days` days (status delisted).""" | |
| 137 | + rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where p.provider_id = :id and p.valid_to > now() - make_interval(days => :d) order by p.valid_to desc limit 200", id=provider_id, d=days) | |
| 138 | + return [deployment_row(r) for r in rows] | |
| 139 | + | |
| 140 | + | |
| 129 | 141 | async def results_of_model(conn: AsyncConnection, model_id: str) -> list[dict[str, Any]]: |
| 130 | 142 | rows = await fetch_all(conn, f"select {RESULT_COLS} from {RESULT_FROM} where r.model_id = :id and r.valid_to is null order by b.canonical_name, r.observed_at desc limit 300", id=model_id) |
| 131 | 143 | return [result_row(r) for r in rows] |
@@ -398,8 +410,8 @@ async def entity_detail(row: dict[str, Any]) -> dict[str, Any]: | ||
| 398 | 410 | groups[0] += [T("models", related_by_type, eid, "model", limit=50, include_org_children=True)] |
| 399 | 411 | groups[1] += [T("papers", related_by_type, eid, "paper", limit=24, include_org_children=True), T("repositories", related_by_type, eid, "repository", limit=24, include_org_children=True)] |
| 400 | 412 | elif etype == "provider": |
| 401 | − groups[0] += [T("prices", prices_of_provider, eid)] | |
| 402 | − groups[1] += [T("models", _provider_models, eid)] | |
| 413 | + groups[0] += [T("prices", prices_of_provider, eid), T("deployments", deployments_of_provider, eid)] | |
| 414 | + groups[1] += [T("models", _provider_models, eid), T("removed", removed_deployments_of_provider, eid)] | |
| 403 | 415 | elif etype == "benchmark": |
| 404 | 416 | groups[0] += [T("results", leaderboard, eid, limit=100)] |
| 405 | 417 | elif etype == "hardware": |
modified
src/aiatlas/api/routers/benchmarks.py
+11 −3
@@ -110,8 +110,12 @@ async def list_benchmarks(request: Request, category: str | None = None) -> dict | ||
| 110 | 110 | @router.get("/matrix") |
| 111 | 111 | @cached(300) |
| 112 | 112 | async def matrix(request: Request, benchmarks: str | None = None, models: str | None = None, org: str | None = None, family: str | None = None, |
| 113 | − limit: int = Query(60, ge=1, le=300), comparable_only: int = Query(0, ge=0, le=1), min_cells: int = Query(3, ge=1, le=30)) -> dict[str, Any]: | |
| 114 | − """Rows = canonical models, columns = benchmarks, cell = best current score in the primary comparability group (+ trust, config_key).""" | |
| 113 | + limit: int = Query(60, ge=1, le=300), comparable_only: int = Query(0, ge=0, le=1), min_cells: int = Query(3, ge=1, le=30), | |
| 114 | + since: str | None = Query(None, description="model release_date ≥ YYYY-MM-DD"), until: str | None = Query(None, description="model release_date ≤ YYYY-MM-DD")) -> dict[str, Any]: | |
| 115 | + """Rows = canonical models, columns = benchmarks, cell = best current score in the primary comparability group (+ trust, config_key, observed_at).""" | |
| 116 | + from aiatlas.api.common import parse_date | |
| 117 | + | |
| 118 | + since_d, until_d = parse_date(since, "since"), parse_date(until, "until") | |
| 115 | 119 | async with connection() as conn: |
| 116 | 120 | groups = await all_primary_groups(conn) |
| 117 | 121 | meta = await benchmark_meta(conn) |
@@ -150,11 +154,14 @@ async def matrix(request: Request, benchmarks: str | None = None, models: str | | ||
| 150 | 154 | for r in ranked: |
| 151 | 155 | if wanted_m is not None and r["model_id"] not in wanted_m: |
| 152 | 156 | continue |
| 157 | + rel = str((r.get("model_attrs") or {}).get("release_date") or "")[:10] | |
| 158 | + if (since_d or until_d) and (not rel or (since_d and rel < since_d.isoformat()) or (until_d and rel > until_d.isoformat())): | |
| 159 | + continue | |
| 153 | 160 | level, _ = comparability(leader_cfg["config"] if leader_cfg else None, r.get("config"), leader_cfg["metric"] if leader_cfg else None, r.get("metric")) if leader_cfg else ("comparable", []) |
| 154 | 161 | if comparable_only and level != "comparable": |
| 155 | 162 | continue |
| 156 | 163 | cells[r["model_id"]][bid] = {"score": r["score"], "rank": r["rank"], "trust_level": r["trust_level"], "config_key": r["config_key"], "comparability": level, |
| 157 | − "result_id": r["id"]} | |
| 164 | + "result_id": r["id"], "observed_at": r["observed_at"], "evaluated_at": r.get("evaluated_at")} | |
| 158 | 165 | model_ref.setdefault(r["model_id"], {"id": r["model_id"], "slug": r["model_slug"], "name": r["model_name"], "organization": r.get("org_name"), "organization_slug": r.get("org_slug"), |
| 159 | 166 | "openness": (r.get("model_attrs") or {}).get("openness"), "release_date": (r.get("model_attrs") or {}).get("release_date")}) |
| 160 | 167 | rows_out = [] |
@@ -165,6 +172,7 @@ async def matrix(request: Request, benchmarks: str | None = None, models: str | | ||
| 165 | 172 | "mean_rank": round(sum(x["rank"] for x in c.values()) / len(c), 2)}) |
| 166 | 173 | rows_out.sort(key=lambda x: (-x["n_cells"], x["mean_rank"], x["model"]["name"] or "")) |
| 167 | 174 | return {"columns": columns, "rows": rows_out[:limit], "total_rows": len(rows_out), "comparable_only": bool(comparable_only), "min_cells": min_cells if wanted_m is None else None, |
| 175 | + "filters": {k: v for k, v in {"since": since, "until": until, "org": org, "family": family, "models": models}.items() if v}, | |
| 168 | 176 | "methodology": GROUPING_NOTE + " Each cell is the model's best current row in the benchmark's primary group; `mean_rank` is only a sort key, not a composite score."} |
| 169 | 177 | |
| 170 | 178 | |
modified
src/aiatlas/api/routers/deployments.py
+13 −6
@@ -27,11 +27,18 @@ TOTAL_CAP = 10_000 | ||
| 27 | 27 | @cached(300) |
| 28 | 28 | async def list_deployments(request: Request, model: str | None = None, provider: str | None = None, current: int = Query(1, ge=0, le=1), limit: int = Query(50, ge=1, le=200), |
| 29 | 29 | offset: int = Query(0, ge=0, le=10000), before: str | None = Query(None, description="cursor: valid_from of the last item"), |
| 30 | − org: str | None = None, sort: str = Query("valid_from", pattern="^(valid_from|output|input|model|provider)$")) -> dict[str, Any]: | |
| 31 | − where = ["m.entity_type in ('model','artifact')", "m.merged_into is null"] | |
| 30 | + org: str | None = None, sort: str = Query("valid_from", pattern="^(valid_from|output|input|model|provider)$"), | |
| 31 | + status: str | None = Query(None, pattern="^(active|delisted|all)$", description="active = current rows (default when current=1), delisted = closed rows only, all = both")) -> dict[str, Any]: | |
| 32 | + where = ["m.entity_type in ('model','artifact')"] | |
| 32 | 33 | params: dict[str, Any] = {"lim": limit, "off": offset} |
| 33 | − if current: | |
| 34 | + status = status or ("active" if current else "all") | |
| 35 | + if status == "active": | |
| 34 | 36 | where.append("p.valid_to is null") |
| 37 | + where.append("m.merged_into is null") | |
| 38 | + elif status == "delisted": | |
| 39 | + where.append("p.valid_to is not null") | |
| 40 | + else: | |
| 41 | + where.append("(p.valid_to is not null or m.merged_into is null)") # closed rows of since-merged models stay visible as history | |
| 35 | 42 | before_ts = parse_ts(before, "before") |
| 36 | 43 | if before_ts is not None: |
| 37 | 44 | where.append("p.valid_from < :before") |
@@ -46,14 +53,14 @@ async def list_deployments(request: Request, model: str | None = None, provider: | ||
| 46 | 53 | if org: |
| 47 | 54 | where.append("(mo.slug = :org or mo.id = :org or mo.canonical_name ilike :org)") |
| 48 | 55 | params["org"] = org |
| 49 | − order = {"valid_from": "p.valid_from desc, p.id desc", "output": "p.output_per_mtok asc nulls last, p.id", "input": "p.input_per_mtok asc nulls last, p.id", | |
| 50 | − "model": "m.canonical_name asc, pv.canonical_name asc, p.id", "provider": "pv.canonical_name asc, m.canonical_name asc, p.id"}[sort] | |
| 56 | + order = {"valid_from": ("p.valid_to desc nulls last, p.id desc" if status == "delisted" else "p.valid_from desc, p.id desc"), "output": "p.output_per_mtok asc nulls last, p.id", | |
| 57 | + "input": "p.input_per_mtok asc nulls last, p.id", "model": "m.canonical_name asc, pv.canonical_name asc, p.id", "provider": "pv.canonical_name asc, m.canonical_name asc, p.id"}[sort] | |
| 51 | 58 | where_sql = " and ".join(where) |
| 52 | 59 | rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {where_sql} order by {order} limit :lim offset :off", **params) |
| 53 | 60 | total = await fetch_val(conn, f"select count(*) from (select 1 from {PRICE_FROM} where {where_sql} limit {TOTAL_CAP}) t", **{k: v for k, v in params.items() if k not in ('lim', 'off')}) |
| 54 | 61 | items = [deployment_row(r) for r in rows] |
| 55 | 62 | return {"items": items, "total": int(total or 0), "limit": limit, "offset": offset, |
| 56 | − "next_before": items[-1]["valid_from"] if len(items) == limit and sort == "valid_from" else None, "current": bool(current)} | |
| 63 | + "next_before": items[-1]["valid_from"] if len(items) == limit and sort == "valid_from" and status != "delisted" else None, "current": bool(current), "status": status} | |
| 57 | 64 | |
| 58 | 65 | |
| 59 | 66 | @router.get("/cost") |
modified
src/aiatlas/api/routers/explore.py
+7 −1
@@ -60,7 +60,13 @@ async def explore_type(request: Request, type: str, p: Pagination = PAGINATION, | ||
| 60 | 60 | async with connection() as conn: |
| 61 | 61 | rows = await fetch_all(conn, f"select {ENTITY_COLS} from {ENTITY_FROM} where {where_sql} order by {SORTS[sort]}, e.id limit :lim offset :off", lim=p.limit, off=p.offset, **params) |
| 62 | 62 | total = await fetch_val(conn, f"select count(*) from {ENTITY_FROM} where {where_sql}", **params) |
| 63 | − out = page([entity_summary(r) for r in rows], int(total or 0), p) | |
| 63 | + items = [] | |
| 64 | + for r in rows: | |
| 65 | + s = entity_summary(r) or {} | |
| 66 | + if etype in ("framework", "library", "runtime", "tool", "agent"): | |
| 67 | + s["kind"] = (s.get("attributes") or {}).get("kind") # canonical framework kind (normalised on read), null when the source did not say | |
| 68 | + items.append(s) | |
| 69 | + out = page(items, int(total or 0), p) | |
| 64 | 70 | out["entity_type"] = etype |
| 65 | 71 | out["label"] = TYPE_LABELS.get(etype, etype.title()) |
| 66 | 72 | return out |
modified
src/aiatlas/api/routers/families.py
+20 −14
@@ -26,12 +26,22 @@ def _f(v: Any) -> float | None: | ||
| 26 | 26 | return None |
| 27 | 27 | |
| 28 | 28 | |
| 29 | −def _aggregate(members: list[dict[str, Any]], ranks: dict[str, dict[str, int]]) -> dict[str, Any]: | |
| 29 | +def _ranks(groups: dict[str, dict[str, Any]]) -> dict[str, dict[str, dict[str, Any]]]: | |
| 30 | + """{model_id: {benchmark slug: {rank, score, metric, config_key}}} over every benchmark's primary group.""" | |
| 31 | + out: dict[str, dict[str, dict[str, Any]]] = defaultdict(dict) | |
| 32 | + for g in groups.values(): | |
| 33 | + for r in rank_rows(g["rows"], g["higher_is_better"]): | |
| 34 | + out[r["model_id"]][g["benchmark"]["slug"]] = {"rank": r["rank"], "score": r["score"], "metric": g["metric"], "config_key": g["config_key"], | |
| 35 | + "higher_is_better": g["higher_is_better"], "benchmark": g["benchmark"]["slug"]} | |
| 36 | + return out | |
| 37 | + | |
| 38 | + | |
| 39 | +def _aggregate(members: list[dict[str, Any]], ranks: dict[str, dict[str, dict[str, Any]]]) -> dict[str, Any]: | |
| 30 | 40 | params = [p for p in (_f((m["attributes"] or {}).get("parameter_count")) for m in members) if p] |
| 31 | 41 | dates = sorted(str((m["attributes"] or {}).get("release_date") or "")[:10] for m in members if (m["attributes"] or {}).get("release_date")) |
| 32 | 42 | mods: set[str] = set() |
| 33 | 43 | lics: dict[str, int] = defaultdict(int) |
| 34 | − best: dict[str, tuple[int, str]] = {} | |
| 44 | + best: dict[str, dict[str, Any]] = {} | |
| 35 | 45 | for m in members: |
| 36 | 46 | a = m["attributes"] or {} |
| 37 | 47 | for k in ("modalities", "modalities_input", "modalities_output"): |
@@ -41,12 +51,12 @@ def _aggregate(members: list[dict[str, Any]], ranks: dict[str, dict[str, int]]) | ||
| 41 | 51 | if key or a.get("license"): |
| 42 | 52 | lics[key or str(a.get("license"))] += 1 |
| 43 | 53 | for b, rk in ranks.get(m["id"], {}).items(): |
| 44 | − if b not in best or rk < best[b][0]: | |
| 45 | − best[b] = (rk, m["slug"]) | |
| 54 | + if b not in best or rk["rank"] < best[b]["rank"]: | |
| 55 | + best[b] = {**rk, "model": m["slug"], "model_name": m["canonical_name"]} | |
| 46 | 56 | return {"model_count": len(members), "first_release": dates[0] if dates else None, "last_release": dates[-1] if dates else None, |
| 47 | 57 | "param_range": {"min": min(params), "max": max(params)} if params else None, "modalities": sorted(mods), |
| 48 | 58 | "licenses": [{"key": k, "label": LICENSES[k].label if k in LICENSES else k, "models": n} for k, n in sorted(lics.items(), key=lambda kv: -kv[1])], |
| 49 | − "benchmark_best": {b: {"rank": rk, "model": slug} for b, (rk, slug) in sorted(best.items())}} | |
| 59 | + "benchmark_best": dict(sorted(best.items()))} | |
| 50 | 60 | |
| 51 | 61 | |
| 52 | 62 | async def _families(conn: Any) -> list[dict[str, Any]]: |
@@ -71,10 +81,7 @@ async def list_families(request: Request, q: str | None = Query(None, max_length | ||
| 71 | 81 | members = await fetch_all(conn, f"""select e.id, e.slug, e.canonical_name, e.attributes, e.family_id, e.attributes->>'family' as label, e.organization_id, eo.slug as org_slug, eo.canonical_name as org_name |
| 72 | 82 | from {ENTITY_FROM} where e.entity_type = 'model' and e.merged_into is null and (e.family_id is not null or e.attributes ? 'family')""") |
| 73 | 83 | groups = await all_primary_groups(conn) |
| 74 | − ranks: dict[str, dict[str, int]] = defaultdict(dict) | |
| 75 | − for g in groups.values(): | |
| 76 | − for r in rank_rows(g["rows"], g["higher_is_better"]): | |
| 77 | − ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"] | |
| 84 | + ranks = _ranks(groups) | |
| 78 | 85 | by_fid: dict[str, list[dict[str, Any]]] = defaultdict(list) |
| 79 | 86 | by_label: dict[str, list[dict[str, Any]]] = defaultdict(list) |
| 80 | 87 | for m in members: |
@@ -127,15 +134,14 @@ async def family_detail(request: Request, slug: str, limit: int = Query(100, ge= | ||
| 127 | 134 | events = await fetch_all(conn, """select ev.entity_id, ev.event_type, ev.summary, ev.occurred_at, ev.importance from change_events ev where ev.entity_id = any(cast(:ids as text[])) |
| 128 | 135 | and ev.event_type in ('NEW_MODEL','RELEASE','DEPRECATION_ANNOUNCED','RETIREMENT_ANNOUNCED','STATUS_CHANGED') order by ev.occurred_at asc limit 500""", ids=ids) if ids else [] |
| 129 | 136 | groups = await all_primary_groups(conn) |
| 130 | − ranks: dict[str, dict[str, int]] = defaultdict(dict) | |
| 131 | − for g in groups.values(): | |
| 132 | − for r in rank_rows(g["rows"], g["higher_is_better"]): | |
| 133 | − ranks[r["model_id"]][g["benchmark"]["slug"]] = r["rank"] | |
| 137 | + ranks = _ranks(groups) | |
| 134 | 138 | mem_summ = [] |
| 135 | 139 | for m in members: |
| 136 | 140 | a = m["attributes"] or {} |
| 141 | + rk = ranks.get(m["id"], {}) | |
| 137 | 142 | mem_summ.append({"model": entity_summary(m), "key_facts": {k: a.get(k) for k in ("release_date", "parameter_count", "active_parameter_count", "context_length", "openness", "license", "modalities", "status") if a.get(k) not in (None, "", [])}, |
| 138 | − "benchmark_ranks": dict(sorted(ranks.get(m["id"], {}).items()))}) | |
| 143 | + "benchmark_ranks": {b: v["rank"] for b, v in sorted(rk.items())}, | |
| 144 | + "benchmark_best": [{"benchmark": b, "rank": v["rank"], "score": v["score"], "metric": v["metric"], "config_key": v["config_key"]} for b, v in sorted(rk.items(), key=lambda kv: kv[1]["rank"])]}) | |
| 139 | 145 | timeline = [{"date": str((m["attributes"] or {}).get("release_date") or "")[:10] or None, "kind": "release", "model": {"id": m["id"], "slug": m["slug"], "name": m["canonical_name"]}} for m in members] |
| 140 | 146 | timeline += [{"date": e["occurred_at"], "kind": e["event_type"], "summary": e["summary"], "model_id": e["entity_id"]} for e in events if e["event_type"] != "NEW_MODEL"] |
| 141 | 147 | timeline.sort(key=lambda x: str(x["date"] or "")) |
modified
src/aiatlas/api/routers/intelligence.py
+43 −13
@@ -69,6 +69,23 @@ def _f(v: Any) -> float | None: | ||
| 69 | 69 | return None |
| 70 | 70 | |
| 71 | 71 | |
| 72 | +def price_delta(old: Any, new: Any) -> dict[str, Any] | None: | |
| 73 | + """PRICE_CHANGED events carry `{input_per_mtok, output_per_mtok}` dicts (or a bare number). Returns % changes (output first) or None.""" | |
| 74 | + def pick(v: Any, k: str) -> float | None: | |
| 75 | + return _f(v.get(k)) if isinstance(v, dict) else (_f(v) if k == "output_per_mtok" else None) | |
| 76 | + | |
| 77 | + out: dict[str, Any] = {} | |
| 78 | + for k, label in (("output_per_mtok", "output"), ("input_per_mtok", "input")): | |
| 79 | + o, n = pick(old, k), pick(new, k) | |
| 80 | + if o and n is not None: | |
| 81 | + out[f"{label}_percent"] = round((n - o) / o * 100, 2) | |
| 82 | + out[f"{label}_from"], out[f"{label}_to"] = o, n | |
| 83 | + if not out: | |
| 84 | + return None | |
| 85 | + out["percent"] = out.get("output_percent", out.get("input_percent")) | |
| 86 | + return out | |
| 87 | + | |
| 88 | + | |
| 72 | 89 | def _mods(attrs: dict[str, Any]) -> set[str]: |
| 73 | 90 | out: set[str] = set() |
| 74 | 91 | for k in ("modalities", "modalities_input", "modalities_output"): |
@@ -171,9 +188,11 @@ async def frontier(request: Request, limit: int = Query(12, ge=1, le=50)) -> dic | ||
| 171 | 188 | for r in movements: |
| 172 | 189 | ev = change_event(r) |
| 173 | 190 | if ev["event_type"] == "PRICE_CHANGED": |
| 174 | − o, n = _f(ev.get("old_value")), _f(ev.get("new_value")) | |
| 175 | − if o and n is not None and abs(n - o) / o >= 0.20: | |
| 176 | − ev["percent_change"] = round((n - o) / o * 100, 1) | |
| 191 | + d = price_delta(ev.get("old_value"), ev.get("new_value")) | |
| 192 | + if d and d["percent"] is not None and abs(d["percent"]) >= 20: | |
| 193 | + ev["percent_change"] = d["percent"] | |
| 194 | + ev["price_delta"] = d | |
| 195 | + ev["provider"] = (ev.get("meta") or {}).get("provider") | |
| 177 | 196 | moves.append(ev) |
| 178 | 197 | else: |
| 179 | 198 | moves.append(ev) |
@@ -268,6 +287,7 @@ async def pareto(request: Request, benchmark: str = Query(...), x: str = Query(" | ||
| 268 | 287 | continue |
| 269 | 288 | points.append({"id": r["model_id"], "model": {"id": r["model_id"], "slug": r["model_slug"], "name": r["model_name"], "organization": r.get("org_name"), "openness": attrs.get("openness")}, |
| 270 | 289 | "x": xv, "y": float(r["score"]), "rank": r["rank"], "trust_level": r["trust_level"], "config": {k: v for k, v in (r.get("config") or {}).items() if k in ("reasoning_effort", "reasoning", "variant", "evaluator")}, |
| 290 | + "context_length": _f(attrs.get("context_length")), "parameter_count": _f(attrs.get("parameter_count")), "release_date": attrs.get("release_date"), | |
| 271 | 291 | **({"provider": provider} if provider else {}), **({"estimated": True} if x == "memory_estimate" else {})}) |
| 272 | 292 | front = pareto_frontier(points, maximize_y=g["higher_is_better"]) |
| 273 | 293 | fset = set(front) |
@@ -302,14 +322,21 @@ async def pulse(request: Request, days: int = Query(7, ge=1, le=90)) -> dict[str | ||
| 302 | 322 | (select count(*) from ev where event_type = 'DOCUMENT_CHANGED') as documents_changed, |
| 303 | 323 | (select count(distinct d.source_id) from snapshots s join documents d on d.id = s.document_id where s.observed_at > :since) as sources_observed, |
| 304 | 324 | (select count(*) from ev where event_type <> 'DOCUMENT_CHANGED') as events_total""", since=since) |
| 305 | − pct = await fetch_all(conn, "select old_value, new_value from change_events where is_backfill = false and occurred_at > :since and event_type = 'PRICE_CHANGED'", since=since) | |
| 325 | + pct = await fetch_all(conn, f"select {EVENT_COLS}, ev.occurred_at from {EVENT_FROM} where ev.is_backfill = false and ev.occurred_at > :since and ev.event_type = 'PRICE_CHANGED' " | |
| 326 | + f"order by ev.occurred_at desc limit 500", since=since) | |
| 327 | + ctx_models = await fetch_all(conn, f"""select distinct on (e.id) {ENTITY_COLS} from {EVENT_FROM} where ev.is_backfill = false and ev.occurred_at > :since and ev.event_type = 'NEW_MODEL' | |
| 328 | + and e.entity_type = 'model' and e.merged_into is null and {CONTEXT} >= 1000000 order by e.id limit 100""", since=since) | |
| 306 | 329 | rows = await load_results(conn, current_only=False) |
| 307 | 330 | meta = await benchmark_meta(conn) |
| 308 | − moves = [] | |
| 331 | + moves: list[float] = [] | |
| 332 | + price_items: list[dict[str, Any]] = [] | |
| 309 | 333 | for r in pct: |
| 310 | − o, n = _f(r["old_value"]), _f(r["new_value"]) | |
| 311 | − if o and n is not None: | |
| 312 | − moves.append((n - o) / o * 100) | |
| 334 | + ev = change_event(r) | |
| 335 | + d = price_delta(ev.get("old_value"), ev.get("new_value")) | |
| 336 | + if d and d["percent"] is not None: | |
| 337 | + moves.append(float(d["percent"])) | |
| 338 | + price_items.append({"id": ev["id"], "summary": ev["summary"], "model": ev["entity"], "provider": (ev.get("meta") or {}).get("provider"), "provider_id": (ev.get("meta") or {}).get("provider_id"), | |
| 339 | + "occurred_at": r.get("occurred_at"), "percent_change": d["percent"] if d else None, "delta": d, "source_url": ev.get("source_url")}) | |
| 313 | 340 | by_bench: dict[str, list[dict[str, Any]]] = defaultdict(list) |
| 314 | 341 | for r in rows: |
| 315 | 342 | by_bench[r["benchmark_id"]].append(r) |
@@ -327,9 +354,10 @@ async def pulse(request: Request, days: int = Query(7, ge=1, le=90)) -> dict[str | ||
| 327 | 354 | "new_papers": {"value": int(c.get("new_papers") or 0), "definition": "papers with a NEW_PAPER event that occurred in the window"}, |
| 328 | 355 | "provider_listings": {"value": int(c.get("provider_listings") or 0), "definition": "PROVIDER_LISTED events in the window"}, |
| 329 | 356 | "provider_delistings": {"value": int(c.get("provider_delistings") or 0), "definition": "PROVIDER_DELISTED events in the window"}, |
| 330 | − "price_changes": {"value": int(c.get("price_changes") or 0), "median_percent": round(statistics.median(moves), 2) if moves else None, | |
| 331 | − "definition": "PRICE_CHANGED events in the window; median % change computed from the events' old/new numeric values when both are present"}, | |
| 332 | − "new_models_1m_context": {"value": int(c.get("new_models_1m_context") or 0), "definition": "new_models whose context_length is at least 1 000 000 tokens"}, | |
| 357 | + "price_changes": {"value": int(c.get("price_changes") or 0), "median_percent": round(statistics.median(moves), 2) if moves else None, "items": price_items[:50], | |
| 358 | + "definition": "PRICE_CHANGED events in the window; % change = output price (else input) new vs old from the event's old/new values; median over events with both values"}, | |
| 359 | + "new_models_1m_context": {"value": int(c.get("new_models_1m_context") or 0), "items": [entity_summary(m) for m in ctx_models], | |
| 360 | + "definition": "new_models whose context_length is at least 1 000 000 tokens"}, | |
| 333 | 361 | "new_benchmark_leaders": {"value": len(new_leaders), "items": new_leaders, "definition": "benchmarks whose primary-group leader (computed from results observed by each date) changed over the window"}, |
| 334 | 362 | "documents_changed": {"value": int(c.get("documents_changed") or 0), "definition": "DOCUMENT_CHANGED events in the window"}, |
| 335 | 363 | "sources_observed": {"value": int(c.get("sources_observed") or 0), "definition": "distinct sources with at least one snapshot taken in the window"}, |
@@ -384,8 +412,10 @@ async def open_models(request: Request, sort: str = Query("release", pattern="^( | ||
| 384 | 412 | where entity_type = 'model' and merged_into is null and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted') group by 1) x""") |
| 385 | 413 | lic_rows = await fetch_all(conn, """select coalesce(attributes->>'license_key', attributes->>'license') as raw, count(*) as n from entities where entity_type = 'model' and merged_into is null |
| 386 | 414 | and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted') and (attributes ? 'license' or attributes ? 'license_key') group by 1""") |
| 415 | + # released in the last 30 days (release_date when known; first_seen_at only for models without any release date) | |
| 387 | 416 | new_30d = await fetch_val(conn, """select count(*) from entities where entity_type = 'model' and merged_into is null and attributes->>'openness' in ('open-weights','open-source','restricted-weights','restricted') |
| 388 | − and (attributes->>'release_date' >= to_char((now() at time zone 'UTC') - interval '30 days', 'YYYY-MM-DD') or first_seen_at > now() - interval '30 days')""") | |
| 417 | + and case when attributes->>'release_date' ~ '^\\d{4}' then left(attributes->>'release_date', 10) >= to_char((now() at time zone 'UTC') - interval '30 days', 'YYYY-MM-DD') | |
| 418 | + else first_seen_at > now() - interval '30 days' end""") | |
| 389 | 419 | ranks: dict[str, dict[str, int]] = defaultdict(dict) |
| 390 | 420 | for g in groups.values(): |
| 391 | 421 | for r in rank_rows(g["rows"], g["higher_is_better"]): |
@@ -413,7 +443,7 @@ async def open_models(request: Request, sort: str = Query("release", pattern="^( | ||
| 413 | 443 | lic_counts[k or f"raw:{lr['raw']}"] += int(lr["n"]) |
| 414 | 444 | return {"items": items, "total": int(total or 0), "limit": limit, "offset": offset, |
| 415 | 445 | "summary": {"by_category": (summary or {}).get("by_cat") or {}, "by_license_top": [{"key": k, "label": LICENSES[k].label if k in LICENSES else k, "models": n} for k, n in sorted(lic_counts.items(), key=lambda kv: -kv[1])[:12]], |
| 416 | − "new_30d": int(new_30d or 0)}, | |
| 446 | + "new_30d": int(new_30d or 0), "new_30d_definition": "release_date within the last 30 days (first_seen_at only when no release date is known)"}, | |
| 417 | 447 | "note": "Universe = canonical models whose weights can be downloaded (open-weights, open-source, restricted-weights). hardware_fit values are ESTIMATES (see /methodology); " |
| 418 | 448 | "best_results are ranks inside each benchmark's primary comparability group — no composite score."} |
| 419 | 449 | |
modified
src/aiatlas/api/routers/models.py
+12 −2
@@ -178,8 +178,15 @@ async def list_models(request: Request, p: Pagination = PAGINATION, q: str | Non | ||
| 178 | 178 | where_sql = " and ".join(where) |
| 179 | 179 | async with connection() as conn: |
| 180 | 180 | rows = await fetch_all(conn, f"""select {ENTITY_COLS}, e.family_id, e.canonical_id, e.artifact_kind, e.identity_confidence, |
| 181 | − fam.slug as family_slug, fam.canonical_name as family_name, {entity_cols("can", "c_")} | |
| 182 | − from {ENTITY_FROM} {FAMILY_JOIN} {CANONICAL_JOIN} where {where_sql} order by {order_sql}, e.id limit :lim offset :off""", | |
| 181 | + fam.slug as family_slug, fam.canonical_name as family_name, {entity_cols("can", "c_")}, | |
| 182 | + bp.input_per_mtok as bp_input, bp.output_per_mtok as bp_output, bp.provider_slug as bp_provider_slug, bp.provider_name as bp_provider_name, | |
| 183 | + bp.provider_id as bp_provider_id, bp.providers as bp_providers | |
| 184 | + from {ENTITY_FROM} {FAMILY_JOIN} {CANONICAL_JOIN} | |
| 185 | + left join lateral (select p.input_per_mtok, p.output_per_mtok, p.provider_id, pv.slug as provider_slug, pv.canonical_name as provider_name, | |
| 186 | + (select count(distinct q.provider_id) from prices q where q.model_id = e.id and q.valid_to is null) as providers | |
| 187 | + from prices p join entities pv on pv.id = p.provider_id | |
| 188 | + where p.model_id = e.id and p.valid_to is null and p.output_per_mtok > 0 order by p.output_per_mtok asc, p.input_per_mtok asc nulls last limit 1) bp on true | |
| 189 | + where {where_sql} order by {order_sql}, e.id limit :lim offset :off""", | |
| 183 | 190 | lim=p.limit, off=p.offset, **params) |
| 184 | 191 | total = await fetch_val(conn, f"select count(*) from {ENTITY_FROM} {FAMILY_JOIN} where {where_sql}", **params) |
| 185 | 192 | items = [] |
@@ -191,6 +198,9 @@ async def list_models(request: Request, p: Pagination = PAGINATION, q: str | Non | ||
| 191 | 198 | s["artifact_kind"] = r.get("artifact_kind") |
| 192 | 199 | s["canonical"] = entity_summary(r, "c_") |
| 193 | 200 | s["identity_confidence"] = r.get("identity_confidence") |
| 201 | + s["best_price"] = ({"input_per_mtok": r["bp_input"], "output_per_mtok": r["bp_output"], "unit": "USD per 1M tokens", | |
| 202 | + "provider": {"id": r["bp_provider_id"], "slug": r["bp_provider_slug"], "name": r["bp_provider_name"]}, "providers": int(r["bp_providers"] or 0), | |
| 203 | + "note": "cheapest current offer by output price"} if r.get("bp_output") is not None else None) | |
| 194 | 204 | items.append(s) |
| 195 | 205 | out = page(items, int(total or 0), p) |
| 196 | 206 | out["universe"] = "models+artifacts" if include_artifacts else "canonical models" |
modified
src/aiatlas/api/routers/prices.py
+26 −8
@@ -19,6 +19,7 @@ from aiatlas.api.common import ( | ||
| 19 | 19 | Pagination, |
| 20 | 20 | cached, |
| 21 | 21 | change_event, |
| 22 | + deployment_row, | |
| 22 | 23 | page, |
| 23 | 24 | price_row, |
| 24 | 25 | resolve_id, |
@@ -133,12 +134,16 @@ async def price_index(request: Request, days: int = Query(180, ge=7, le=1825)) - | ||
| 133 | 134 | and coalesce(p.context_length, case when m.attributes->>'context_length' ~ '^[0-9]+$' then (m.attributes->>'context_length')::bigint end) >= 1000000 |
| 134 | 135 | order by p.output_per_mtok asc limit 1""", ids=sorted(frontier)) |
| 135 | 136 | current_outputs = await fetch_all(conn, "select p.output_per_mtok as v from prices p join entities m on m.id = p.model_id where p.valid_to is null and p.output_per_mtok > 0 and m.merged_into is null") |
| 136 | − listings = await fetch_one(conn, """ | |
| 137 | − select (select count(*) from prices p where p.valid_from > now() - interval '30 days' | |
| 138 | − and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_from < p.valid_from)) as new_listings_30d, | |
| 139 | − (select count(*) from prices p where p.valid_to > now() - interval '30 days' | |
| 140 | − and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_to is null)) as delistings_30d, | |
| 141 | − (select count(*) from change_events where category = 'price' and is_backfill = false and occurred_at > now() - interval '30 days') as price_changes_30d""") | |
| 137 | + new_where = ("p.valid_from > now() - interval '30 days' and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_from < p.valid_from) " | |
| 138 | + "and m.entity_type in ('model','artifact') and m.merged_into is null") | |
| 139 | + gone_where = "p.valid_to > now() - interval '30 days' and not exists (select 1 from prices q where q.model_id = p.model_id and q.provider_id = p.provider_id and q.valid_to is null)" | |
| 140 | + new_rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {new_where} order by p.valid_from desc limit 50") | |
| 141 | + new_n = await fetch_val(conn, f"select count(*) from {PRICE_FROM} where {new_where}") | |
| 142 | + gone_rows = await fetch_all(conn, f"select {PRICE_COLS} from {PRICE_FROM} where {gone_where} order by p.valid_to desc limit 50") | |
| 143 | + gone_n = await fetch_val(conn, f"select count(*) from {PRICE_FROM} where {gone_where}") | |
| 144 | + chg_rows = await fetch_all(conn, f"select {EVENT_COLS}, ev.occurred_at from {EVENT_FROM} where ev.category = 'price' and ev.is_backfill = false and ev.occurred_at > now() - interval '30 days' " | |
| 145 | + f"order by ev.occurred_at desc limit 50") | |
| 146 | + chg_n = await fetch_val(conn, "select count(*) from change_events where category = 'price' and is_backfill = false and occurred_at > now() - interval '30 days'") | |
| 142 | 147 | dist = [] |
| 143 | 148 | vals = [float(r["v"]) for r in current_outputs] |
| 144 | 149 | for lo, hi in BUCKETS: |
@@ -154,8 +159,9 @@ async def price_index(request: Request, days: int = Query(180, ge=7, le=1825)) - | ||
| 154 | 159 | return {"days": days, "series": out_series, "movers": [change_event(r) for r in movers], |
| 155 | 160 | "cheapest_frontier": _cheap(cheapest), "cheapest_frontier_1m_context": _cheap(cheapest_1m), |
| 156 | 161 | "distribution": {"metric": "output_per_mtok", "unit": "USD per 1M tokens", "buckets": dist, "offers": len(vals)}, |
| 157 | − "new_listings_30d": int((listings or {}).get("new_listings_30d") or 0), "delistings_30d": int((listings or {}).get("delistings_30d") or 0), | |
| 158 | − "price_changes_30d": int((listings or {}).get("price_changes_30d") or 0), | |
| 162 | + "new_listings_30d": {"count": int(new_n or 0), "items": [deployment_row(r) for r in new_rows], "definition": "offers whose first row for (model, provider) opened in the last 30 days"}, | |
| 163 | + "delistings_30d": {"count": int(gone_n or 0), "items": [deployment_row(r) for r in gone_rows], "definition": "offers closed in the last 30 days with no live successor for (model, provider)"}, | |
| 164 | + "price_changes_30d": {"count": int(chg_n or 0), "items": [_price_event(r) for r in chg_rows], "definition": "price events (not back-filled) that occurred in the last 30 days"}, | |
| 159 | 165 | "frontier": {"composition": composition, "methodology": FRONTIER_METHODOLOGY}, |
| 160 | 166 | "methodology": "Daily medians of live USD-per-1M-token prices across every provider offer valid at the end of each UTC day (offers with a zero or missing price are excluded " |
| 161 | 167 | "from that median). `median_open_output` covers models with openness open-weights/open-source; `median_embedding_input` covers models whose modalities include " |
@@ -163,6 +169,18 @@ async def price_index(request: Request, days: int = Query(180, ge=7, le=1825)) - | ||
| 163 | 169 | "note": "Daily medians of live USD-per-1M-token input/output prices across all provider offers valid at the end of each UTC day."} |
| 164 | 170 | |
| 165 | 171 | |
| 172 | +def _price_event(r: dict[str, Any]) -> dict[str, Any]: | |
| 173 | + from aiatlas.api.routers.intelligence import price_delta | |
| 174 | + | |
| 175 | + ev = change_event(r) | |
| 176 | + d = price_delta(ev.get("old_value"), ev.get("new_value")) | |
| 177 | + ev["occurred_at"] = r.get("occurred_at") | |
| 178 | + ev["percent_change"] = d["percent"] if d else None | |
| 179 | + ev["price_delta"] = d | |
| 180 | + ev["provider"] = (ev.get("meta") or {}).get("provider") | |
| 181 | + return ev | |
| 182 | + | |
| 183 | + | |
| 166 | 184 | def _cheap(row: dict[str, Any] | None) -> dict[str, Any] | None: |
| 167 | 185 | if not row: |
| 168 | 186 | return None |
modified
src/aiatlas/services/cache.py
+24 −5
@@ -1,7 +1,12 @@ | ||
| 1 | −"""Redis helpers: API cache (prefix `aia:api:`), distributed locks (`aia:lock:`), heartbeats.""" | |
| 1 | +"""Redis helpers: API cache (prefix `aia:api:<db namespace>:`), distributed locks (`aia:lock:`), heartbeats. | |
| 2 | + | |
| 3 | +The API cache is namespaced by a hash of `DATABASE_URL`: two API processes pointing at different databases (dev on :8331, a production | |
| 4 | +copy on :8332) but sharing one Redis db must never serve each other's bodies or derived id sets (that is how `/frontier` once reported | |
| 5 | +frontier ids from another database and found no prices for them).""" | |
| 2 | 6 | from __future__ import annotations |
| 3 | 7 | |
| 4 | 8 | import asyncio |
| 9 | +import hashlib | |
| 5 | 10 | import logging |
| 6 | 11 | from collections.abc import AsyncIterator |
| 7 | 12 | from contextlib import asynccontextmanager |
@@ -14,6 +19,7 @@ from aiatlas.config import settings | ||
| 14 | 19 | |
| 15 | 20 | log = logging.getLogger(__name__) |
| 16 | 21 | _redis: Redis | None = None |
| 22 | +_NS: str | None = None | |
| 17 | 23 | |
| 18 | 24 | |
| 19 | 25 | def redis() -> Redis: |
@@ -23,9 +29,21 @@ def redis() -> Redis: | ||
| 23 | 29 | return _redis |
| 24 | 30 | |
| 25 | 31 | |
| 32 | +def namespace() -> str: | |
| 33 | + """8-hex namespace derived from the database URL (host, port, database name) — same DB ⇒ same cache, different DB ⇒ disjoint keys.""" | |
| 34 | + global _NS | |
| 35 | + if _NS is None: | |
| 36 | + _NS = hashlib.sha1(settings.database_url.encode()).hexdigest()[:8] | |
| 37 | + return _NS | |
| 38 | + | |
| 39 | + | |
| 40 | +def api_key(key: str) -> str: | |
| 41 | + return f"aia:api:{namespace()}:{key}" | |
| 42 | + | |
| 43 | + | |
| 26 | 44 | async def cache_get(key: str) -> Any | None: |
| 27 | 45 | try: |
| 28 | − raw = await redis().get(f"aia:api:{key}") | |
| 46 | + raw = await redis().get(api_key(key)) | |
| 29 | 47 | return orjson.loads(raw) if raw else None |
| 30 | 48 | except Exception: # noqa: BLE001 |
| 31 | 49 | return None |
@@ -33,15 +51,16 @@ async def cache_get(key: str) -> Any | None: | ||
| 33 | 51 | |
| 34 | 52 | async def cache_set(key: str, value: Any, ttl_s: int = 300) -> None: |
| 35 | 53 | try: |
| 36 | − await redis().set(f"aia:api:{key}", orjson.dumps(value, default=str), ex=ttl_s) | |
| 54 | + await redis().set(api_key(key), orjson.dumps(value, default=str), ex=ttl_s) | |
| 37 | 55 | except Exception: # noqa: BLE001 |
| 38 | 56 | pass |
| 39 | 57 | |
| 40 | 58 | |
| 41 | 59 | async def cache_invalidate(prefix: str = "") -> int: |
| 60 | + """Delete `aia:api:<ns>:<prefix>*` — only this database's cache.""" | |
| 42 | 61 | n = 0 |
| 43 | 62 | try: |
| 44 | − async for key in redis().scan_iter(match=f"aia:api:{prefix}*", count=500): | |
| 63 | + async for key in redis().scan_iter(match=f"{api_key(prefix)}*", count=500): | |
| 45 | 64 | await redis().delete(key) |
| 46 | 65 | n += 1 |
| 47 | 66 | except Exception: # noqa: BLE001 |
@@ -96,4 +115,4 @@ async def close() -> None: | ||
| 96 | 115 | await asyncio.sleep(0) |
| 97 | 116 | |
| 98 | 117 | |
| 99 | −__all__ = ["cache_get", "cache_invalidate", "cache_set", "close", "heartbeat", "heartbeats", "lock", "redis"] | |
| 118 | +__all__ = ["api_key", "cache_get", "cache_invalidate", "cache_set", "close", "heartbeat", "heartbeats", "lock", "namespace", "redis"] | |
modified
src/aiatlas/services/frontier.py
+7 −3
@@ -14,7 +14,7 @@ from typing import Any | ||
| 14 | 14 | |
| 15 | 15 | from sqlalchemy.ext.asyncio import AsyncConnection |
| 16 | 16 | |
| 17 | −from aiatlas.db import fetch_all | |
| 17 | +from aiatlas.db import fetch_all, fetch_val | |
| 18 | 18 | from aiatlas.ontology.benchmarks import ( |
| 19 | 19 | CONDITION_KEYS, |
| 20 | 20 | METRICS, |
@@ -278,8 +278,12 @@ async def frontier_model_ids(conn: AsyncConnection, *, use_cache: bool = True) - | ||
| 278 | 278 | """Frontier composition (see FRONTIER_METHODOLOGY). Returns (ids, sample counts).""" |
| 279 | 279 | if use_cache: |
| 280 | 280 | hit = await cache.cache_get(FRONTIER_CACHE_KEY) |
| 281 | − if hit: | |
| 282 | − return set(hit["ids"]), hit["composition"] | |
| 281 | + if hit and hit.get("ids"): | |
| 282 | + # guard: the cached set must still be live canonical models of THIS database (cache keys are namespaced per DB, but a | |
| 283 | + # canonicalisation run can merge ids away between refreshes) — otherwise recompute | |
| 284 | + live = await fetch_val(conn, "select count(*) from entities where id = any(cast(:ids as text[])) and entity_type = 'model' and merged_into is null", ids=hit["ids"]) | |
| 285 | + if int(live or 0) == len(hit["ids"]): | |
| 286 | + return set(hit["ids"]), hit["composition"] | |
| 283 | 287 | since = (datetime.now(UTC) - timedelta(days=365)).date().isoformat() |
| 284 | 288 | recent = await fetch_all(conn, """ |
| 285 | 289 | select e.id from entities e where e.entity_type = 'model' and e.merged_into is null and e.attributes->>'release_date' >= :since |
modified
tests/test_api_v11.py
+65 −0
@@ -207,6 +207,71 @@ async def test_search_compiler_v2_response(client: AsyncClient) -> None: | ||
| 207 | 207 | assert "organization" not in unknown and "Foobarbaz" in unknown["unrecognised"] |
| 208 | 208 | |
| 209 | 209 | |
| 210 | +async def test_frontier_price_join_and_cache_namespace(client: AsyncClient) -> None: | |
| 211 | + """Regression: frontier ids and prices come from the SAME database — cheapest_frontier must exist whenever frontier offers exist.""" | |
| 212 | + from aiatlas.services import cache | |
| 213 | + | |
| 214 | + assert cache.api_key("x").startswith(f"aia:api:{cache.namespace()}:") and len(cache.namespace()) == 8 | |
| 215 | + idx = (await client.get("/api/v1/prices/index", params={"days": 7})).json() | |
| 216 | + offers_today = idx["series"][-1]["sample"]["frontier_offers"] | |
| 217 | + if offers_today > 0: | |
| 218 | + assert idx["cheapest_frontier"] is not None and idx["cheapest_frontier"]["output"] > 0 | |
| 219 | + cheapest = (await client.get("/api/v1/prices", params={"sort": "cheapest_frontier", "limit": 5})).json() | |
| 220 | + assert cheapest["items"] and cheapest["items"][0]["output_per_mtok"] > 0 | |
| 221 | + fr = (await client.get("/api/v1/frontier")).json() | |
| 222 | + assert fr["price_frontier"]["cheapest_output"] is not None and fr["price_frontier"]["frontier_models"] > 0 | |
| 223 | + for key in ("new_listings_30d", "delistings_30d", "price_changes_30d"): | |
| 224 | + assert {"count", "items", "definition"} <= set(idx[key]) and len(idx[key]["items"]) <= 50 | |
| 225 | + | |
| 226 | + | |
| 227 | +async def test_pulse_items_and_price_delta(client: AsyncClient) -> None: | |
| 228 | + from aiatlas.api.routers.intelligence import price_delta | |
| 229 | + | |
| 230 | + d = price_delta({"input_per_mtok": 1.0, "output_per_mtok": 4.0}, {"input_per_mtok": 0.5, "output_per_mtok": 5.0}) | |
| 231 | + assert d["output_percent"] == 25.0 and d["input_percent"] == -50.0 and d["percent"] == 25.0 | |
| 232 | + assert price_delta(None, {"output_per_mtok": 1}) is None and price_delta(2.0, 3.0)["percent"] == 50.0 | |
| 233 | + body = (await client.get("/api/v1/pulse", params={"days": 90})).json() | |
| 234 | + pc = body["counters"]["price_changes"] | |
| 235 | + assert "items" in pc and "median_percent" in pc and isinstance(body["counters"]["new_models_1m_context"]["items"], list) | |
| 236 | + if pc["value"]: | |
| 237 | + assert pc["items"] and {"summary", "percent_change", "provider", "occurred_at"} <= set(pc["items"][0]) | |
| 238 | + | |
| 239 | + | |
| 240 | +async def test_provider_detail_deployments_and_listing_status(client: AsyncClient) -> None: | |
| 241 | + providers = (await client.get("/api/v1/providers")).json()["items"] | |
| 242 | + slug = providers[0]["slug"] | |
| 243 | + d = (await client.get(f"/api/v1/providers/{slug}")).json() | |
| 244 | + assert "deployments" in d and "removed" in d | |
| 245 | + outs = [x["prices"]["output"] for x in d["deployments"] if x["prices"]["output"] is not None] | |
| 246 | + assert outs == sorted(outs) and all(x["status"] == "active" for x in d["deployments"]) and all(x["status"] == "delisted" for x in d["removed"]) | |
| 247 | + gone = (await client.get("/api/v1/deployments", params={"current": 0, "status": "delisted", "limit": 5})).json() | |
| 248 | + assert gone["status"] == "delisted" and all(x["status"] == "delisted" for x in gone["items"]) | |
| 249 | + both = (await client.get("/api/v1/deployments", params={"current": 0, "limit": 200})).json() | |
| 250 | + assert both["status"] == "all" and both["total"] >= gone["total"] | |
| 251 | + | |
| 252 | + | |
| 253 | +async def test_models_best_price_and_family_scores_and_matrix_cells(client: AsyncClient) -> None: | |
| 254 | + rows = (await client.get("/api/v1/models", params={"limit": 50, "sort": "cheapest"})).json()["items"] | |
| 255 | + priced = [r for r in rows if r.get("best_price")] | |
| 256 | + assert priced and {"input_per_mtok", "output_per_mtok", "provider", "providers"} <= set(priced[0]["best_price"]) and priced[0]["best_price"]["provider"]["slug"] | |
| 257 | + fams = (await client.get("/api/v1/families", params={"limit": 3})).json()["items"] | |
| 258 | + for b in fams[0]["benchmark_best"].values(): | |
| 259 | + assert {"rank", "score", "metric", "model"} <= set(b) | |
| 260 | + fd = (await client.get(f"/api/v1/families/{fams[0]['slug']}")).json() | |
| 261 | + assert all("benchmark_best" in m and all({"benchmark", "rank", "score"} <= set(x) for x in m["benchmark_best"]) for m in fd["members"]) | |
| 262 | + mx = (await client.get("/api/v1/benchmarks/matrix", params={"limit": 3, "since": "2024-01-01"})).json() | |
| 263 | + assert mx["filters"]["since"] == "2024-01-01" | |
| 264 | + for row in mx["rows"]: | |
| 265 | + assert row["model"]["release_date"] and row["model"]["release_date"][:10] >= "2024-01-01" | |
| 266 | + assert all("observed_at" in c for c in row["cells"].values() if c) | |
| 267 | + pts = (await client.get("/api/v1/pareto", params={"benchmark": BENCH})).json()["points"] | |
| 268 | + assert pts and {"context_length", "parameter_count"} <= set(pts[0]) | |
| 269 | + fw = (await client.get("/api/v1/explore/framework", params={"limit": 20})).json() | |
| 270 | + assert all("kind" in it for it in fw["items"]) | |
| 271 | + op = (await client.get("/api/v1/open", params={"limit": 1})).json() | |
| 272 | + assert op["summary"]["new_30d"] <= op["total"] and "new_30d_definition" in op["summary"] | |
| 273 | + | |
| 274 | + | |
| 210 | 275 | async def test_etag_304(client: AsyncClient) -> None: |
| 211 | 276 | r1 = await client.get("/api/v1/stats") |
| 212 | 277 | etag = r1.headers.get("etag") |
| 213 | 278 | |