AI Atlas foundation: schema, connector SDK, archive, temporal writer, LLM gateway, scheduler, registry, CLI, Anthropic connector

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 1 mo ago (Sep 11, 2026) Browse files

72 changed files +8,918 −0

added .env.example +25 −0

++# AI Atlas — local development. Copy to .env (never commit .env).
++APP_ENV=development
++AIA_SITE_URL=http://localhost:8320
++DATABASE_URL=postgresql+asyncpg://aiatlas:aiatlas@127.0.0.1:5432/aiatlas
++REDIS_URL=redis://127.0.0.1:6379/5
++AIA_DATA_DIR=./data
++AIA_API_HOST=127.0.0.1
++AIA_API_PORT=8321
++AIA_ADMIN_TOKEN=dev-admin-token
++AIA_LOG_JSON=0
++# Crawler identity (robots.txt is honoured; contact address is public)
++AIA_USER_AGENT="AIAtlasBot/0.1 (+https://www.ai-atlas.co/bot; contact@spboucher.ai)"
++# Local LLM factory (OpenAI-compatible; MacLustr llm-api.io). Optional — deterministic extraction works without it.
++AIA_LLM_BASE_URL=https://www.llm-api.io/v1
++AIA_LLM_API_KEY=
++AIA_LLM_SMALL_MODEL=qwen3-4b-instruct-2507-4bit
++AIA_LLM_MEDIUM_MODEL=qwen3.6-35b-a3b-4bit
++AIA_LLM_LARGE_MODEL=qwen3.8-27b-4bit
++AIA_EMBEDDING_MODEL=qwen3-embedding-0.6b-4bit
++# Optional escalation transports (never required)
++SCRAPFLY_API_KEY=
++FIRECRAWL_API_KEY=
++# Web
++API_URL=http://127.0.0.1:8321
++NEXT_PUBLIC_SITE_URL=http://localhost:8320

added .gitignore +22 −0

++.env
++.env.*
++!.env.example
++.venv/
++__pycache__/
++*.pyc
++/data/
++node_modules/
++.next/
++tmp/
++.DS_Store
++deploy/.admin-token
++deploy/.llm-key
++deploy/rendered/
++*.tsbuildinfo
++apps/web/next-env.d.ts
++apps/web/qa/screens/
++apps/web/AGENTS.md
++apps/web/CLAUDE.md
++.pytest_cache/
++.ruff_cache/
++*.egg-info/

added CLAUDE.md +52 −0

++# AI Atlas — project guide (condensed from the founding spec; the full architecture is in docs/ARCHITECTURE.md)
++
++**Mission**: the most comprehensive, structured, searchable and continuously updated map of the global AI ecosystem — Bloomberg × Wikipedia ×
++Crunchbase × Hugging Face × Papers With Code, purpose-built for AI. Not a tools directory, not a news aggregator, not an API wrapper.
++**The dataset is the product; the website and the API are interfaces to it.**
++
++## Non-negotiables
++
++1. No critical dependence on external data APIs — direct crawling first; Scrapfly/Firecrawl/browser are optional escalation only.
++2. Store provenance (source, snapshot, URL, tier, confidence, extractor) on every fact; store history (temporal claims, append-only prices/results, raw snapshots).
++3. Local LLM (MacLustr llm-api.io via the gateway) for large-scale extraction; deterministic extraction always runs first.
++4. **Never fabricate data.** Missing → "Unavailable". Conflicts → both claims stored and flagged; never a compromise value.
++5. Primary sources first (tier 1 official > 2 quality secondary > 3 community > 4 unverified). Connectors are owned IP with fixture tests.
++6. Live counters, stats, feeds and timelines come from the database — never hardcoded.
++7. Public information stays public (SEO); login only for watchlists/alerts/API keys later.
++8. Respectful crawling: robots.txt, per-domain rate limits, conditional requests, identified UA, no bypassing access controls, no private data.
++
++## Stack
++
++- Backend `src/aiatlas/` — Python 3.12, FastAPI, SQLAlchemy Core + asyncpg, Alembic (SQL, forward-only), Redis, APScheduler, selectolax, feedparser, pydantic.
++  CLI `aia` (`.venv/bin/aia --help`). Connector SDK in `sdk/` (docs/CONNECTORS.md). API contract in docs/API.md.
++- Database: Postgres 17 + pgvector + pg_trgm. Internal ids = prefixed ULIDs (`model_…`); slugs stable and unique.
++- Frontend `apps/web/` — Next.js 16 (App Router, React 19, TypeScript strict, Tailwind v4). Guide: docs/FRONTEND.md. Dev on :8320, API on :8321.
++- Data outside the repo: `AIA_DATA_DIR` (`./data` in dev, `~/ai-atlas-data` in prod): raw/, text/, backups/, logs/, cache/, seed/.
++
++## Local development
++
++```bash
++uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev]"
++cp .env.example .env          # local Postgres `aiatlas` (role aiatlas/aiatlas), Redis db 5
++.venv/bin/aia migrate && .venv/bin/aia seed
++.venv/bin/aia run anthropic   # one connector, live · `--file key=path` uses a fixture · `aia crawl --priority 0` runs all P0 connectors
++.venv/bin/aia api --reload    # http://127.0.0.1:8321/api/v1/docs
++pnpm install && pnpm dev:web  # http://localhost:8320
++.venv/bin/pytest -q           # fixtures only; `-m live` hits the network
++```
++
++## Production (MacLustr)
++
++Node **M2M32c** (dedicated), deployed by `mld` — manifest `deploy/ai-atlas.mld.json` → `M1M32:~/dispatch/apps/ai-atlas.json`. PM2 processes
++`ai-atlas-api` (uvicorn 127.0.0.1:8321), `ai-atlas-scheduler` (`aia schedule`), `ai-atlas-web` (next start :8320). Public route through the
++MacLustr Tunnel: `https://www.ai-atlas.co → M2M32c:8320`. See docs/DEPLOY.md. Secrets live only in the rendered manifest (`deploy/.admin-token`,
++`deploy/.llm-key`, git-ignored). LLM factory: `AIA_LLM_BASE_URL=https://www.llm-api.io/v1` (private MacLustr server on M1M64).
++
++## Conventions
++
++- Properties and relations use the shared vocabulary in docs/CONNECTORS.md; metrics use the `metric.` prefix (no events).
++- Every connector: `source_key` registered in `registry/sources.yaml`, organization in `registry/organizations.yaml`, fixture test in `tests/`.
++- Parser improved? bump `parser_version` and `aia reprocess <connector>` — never re-crawl for a code change.
++- Frontend: mobile-first (390/430/768/1440), DOM order = visual order, ≥ 44 px targets, no horizontal overflow, dark + light, premium
++  institutional design (no generic SaaS cards). Every page: real data, loading + error + empty states, source attribution, `generateMetadata`.
++- Units: tokens, USD per 1M tokens, GB, GB/s, W, ISO dates (UTC). Openness vocabulary: open-weights | open-source | proprietary | restricted.

added README.md +39 −0

++# AI Atlas
++
++**www.ai-atlas.co — Explore the entire AI ecosystem.**
++
++AI Atlas is a continuously updated, historically versioned, machine-readable map of the global artificial intelligence ecosystem: models,
++companies, research, providers and prices, benchmarks, hardware, frameworks, datasets, tools — connected in one temporal knowledge graph,
++with provenance on every fact.
++
++- **First-party connectors** read official pages, docs, feeds, model cards and public files directly. No commercial data API is required.
++- **Raw historical archive**: every changed page is stored (content-addressed, compressed) and can be reprocessed when parsers improve.
++- **Temporal knowledge graph**: claims carry `valid_from`/`valid_to`, prices and benchmark results are append-only, every change is an event.
++- **Local LLM factory**: large-scale extraction runs on MacLustr's own inference servers through an engine-agnostic gateway.
++- **Interfaces**: the website, the public API (`/api/v1`), and — later — datasets and alerts.
++
++## Layout
++
++```
++registry/            curated YAML (sources, organizations, providers, benchmarks, hardware) — every entry cites its source
++src/aiatlas/         Python platform: sdk/ (connector SDK), connectors/, services/ (jobs, scheduler, LLM gateway, search, quality), api/, cli.py
++migrations/          forward-only SQL migrations
++apps/web/            Next.js 16 website
++tests/               pytest with saved fixtures
++deploy/              MacLustr (mld) manifest and scripts
++docs/                ARCHITECTURE.md · CONNECTORS.md · API.md · FRONTEND.md · DEPLOY.md
++```
++
++## Quick start
++
++```bash
++uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev]"
++cp .env.example .env && createdb -O aiatlas aiatlas
++.venv/bin/aia migrate && .venv/bin/aia seed
++.venv/bin/aia crawl --priority 0            # initial corpus: labs, Hugging Face, arXiv, providers, benchmarks…
++.venv/bin/aia api                           # http://127.0.0.1:8321/api/v1/docs
++pnpm install && pnpm dev:web                # http://localhost:8320
++```
++
++© 2026 Simon-Pierre Boucher · contact@spboucher.ai · hosted on [MacLustr](https://www.maclustr.io). AI Atlas links to original sources and
++never republishes documents in full; extracted facts are provided as-is with their provenance.

added alembic.ini +31 −0

++[alembic]
++script_location = migrations
++prepend_sys_path = src
++path_separator = os
++
++[loggers]
++keys = root,alembic
++
++[handlers]
++keys = console
++
++[formatters]
++keys = generic
++
++[logger_root]
++level = WARN
++handlers = console
++
++[logger_alembic]
++level = INFO
++handlers =
++qualname = alembic
++
++[handler_console]
++class = StreamHandler
++args = (sys.stderr,)
++level = NOTSET
++formatter = generic
++
++[formatter_generic]
++format = %(levelname)-5.5s [%(name)s] %(message)s

added docs/API.md +119 −0

++# AI Atlas API — contract (v1)
++
++Base: `/api/v1` (FastAPI, `src/aiatlas/api`). JSON, UTF-8, ISO-8601 UTC timestamps, numbers as numbers (Postgres aggregates may arrive as
++strings — the web layer normalises). Public routes are cached in Redis (`aia:api:*`, 60–600 s) and rate-limited per IP on search.
++Admin routes require header `x-aia-admin-token: <AIA_ADMIN_TOKEN>`. OpenAPI at `/api/v1/docs`. Errors: `{"detail": "..."}` with 400/404/429/503.
++Never expose credentials, internal hostnames, raw archive content or `raw_path`s.
++
++## Shared shapes
++
++```ts
++type Num = number | string | null
++type Org = { id: string; slug: string; name: string } | null
++type EntitySummary = { id: string; entity_type: string; slug: string; name: string; description: string | null; status: string;
++  organization: Org; attributes: Record<string, unknown>; quality: { score?: number; completeness?: number; primary_source_ratio?: number;
++  freshness?: number; source_count?: number; conflicts?: number }; counts: { relations?: number; events?: number; claims?: number };
++  first_seen_at: string; last_seen_at: string; updated_at: string }
++type Page<T> = { items: T[]; total: number; limit: number; offset: number }
++type Provenance = Record<string, { source_id: string | null; source_name?: string; url: string | null; observed_at: string; tier: number;
++  confidence: string; extractor: string; unit?: string }>
++type ChangeEvent = { id: string; event_type: string; category: string; property: string | null; old_value: unknown; new_value: unknown;
++  summary: string; importance: 0 | 1 | 2 | 3; observed_at: string; effective_at: string | null; source_url: string | null; connector_name: string | null;
++  entity: EntitySummary | null; meta: Record<string, unknown> }
++type Price = { id: string; model: EntitySummary; provider: EntitySummary; provider_model_id: string | null; input_per_mtok: Num; output_per_mtok: Num;
++  cached_input_per_mtok: Num; cache_write_per_mtok: Num; batch_input_per_mtok: Num; batch_output_per_mtok: Num; per_image: Num; currency: string;
++  context_length: Num; max_output_tokens: Num; features: Record<string, unknown>; observed_at: string; valid_from: string; valid_to: string | null; source_url: string | null; tier: number }
++type BenchmarkResult = { id: string; model: EntitySummary; benchmark: EntitySummary; score: number; metric: string | null; unit: string | null;
++  higher_is_better: boolean; config: Record<string, unknown>; evaluated_at: string | null; observed_at: string; source_url: string | null; tier: number; confidence: string }
++type Claim = { id: string; property: string; value: unknown; unit: string | null; tier: number; confidence: string; status: string; extractor: string;
++  observed_at: string; effective_at: string | null; valid_from: string; valid_to: string | null; source_url: string | null; source_name: string | null }
++type SourceRef = { source_id: string | null; source_name: string | null; domain: string | null; tier: number | null; url: string; doc_type: string; last_observed_at: string | null; snapshots: number }
++```
++
++## Public routes
++
++| route | returns |
++|---|---|
++| `GET /health` | `{ status: "ok"|"degraded", version, db: bool, redis: bool, llm: { available, reachable? }, time }` |
++| `GET /stats` | `{ entities: Record<type, number>, entities_total, sources, connectors, connectors_enabled, documents, snapshots, claims, claims_current, relations, change_events, change_events_24h, change_events_7d, benchmark_results, prices_current, prices_total, review_pending, llm_jobs, llm_tokens, last_snapshot_at, last_event_at, first_entity_at, archive: { raw_bytes, raw_files, text_bytes, text_files }, computed_at }` — **always live from the DB** |
++| `GET /stats/history?days=90` | `{ items: { day: string; counts: object }[] }` |
++| `GET /search?q=&type=&limit=30&offset=0` | `{ query: CompiledQuery, items: (EntitySummary & { rank: number })[], total: number }` — uses `services.search.compile_query` + `search_entities` (+ embedding when the gateway is up) |
++| `GET /search/suggest?q=` | `{ items: { id, entity_type, slug, name, organization_name }[] }` (≤ 8, prefix, fast) |
++| `GET /entities/{slug_or_id}` | `EntityDetail` (below). Also mounted as `/models/{slug}`, `/companies/{slug}`, `/papers/{slug}`, `/providers/{slug}`, `/benchmarks/{slug}`, `/hardware/{slug}`, `/frameworks/{slug}`, `/datasets/{slug}`, `/tools/{slug}` (404 if the type doesn't match) |
++| `GET /entities/{slug}/timeline?limit=50&before=` | `{ items: ChangeEvent[] }` (entity's own events + events of entities it develops for companies) |
++| `GET /entities/{slug}/history?property=` | `{ items: Claim[] }` — full claim history (all statuses), newest first |
++| `GET /entities/{slug}/asof?date=YYYY-MM-DD` | `{ existed: bool, first_seen_at, date, attributes: Record<string, unknown>, claims: Claim[] }` — state as known at that date |
++| `GET /entities/{slug}/graph?depth=1&limit=80` | `{ nodes: { id, slug, name, entity_type, organization_name? }[], edges: { source, target, predicate }[] }` (depth ≤ 2, cap nodes) |
++| `GET /entities/{slug}/sources` | `{ items: SourceRef[] }` |
++| `GET /entities/{slug}/related?limit=` | `{ items: EntitySummary[] }` (same org / same family / shared relations) |
++| `GET /models?q=&org=&family=&openness=&modality=&status=&min_params=&max_params=&min_context=&year_from=&year_to=&license=&sort=(updated|name|params|context|release|quality|downloads)&order=&limit=&offset=` | `Page<EntitySummary>` + `facets` when `facets=1`: `{ organizations: {slug,name,count}[], openness: {value,count}[], modalities, families, years, licenses, status }` |
++| `GET /companies?q=&country=&kind=&sort=(models|name|updated|quality)&limit=&offset=` | `Page<EntitySummary & { model_count: number; paper_count: number }>` (+`facets`: countries, kinds) |
++| `GET /papers?q=&category=&org=&since=&until=&sort=(published|updated)&limit=&offset=` | `Page<EntitySummary>` |
++| `GET /providers` | `{ items: (EntitySummary & { model_count: number; price_count: number; min_input_per_mtok: Num; min_output_per_mtok: Num })[] }` |
++| `GET /prices?model=&provider=&sort=(input|output|model|provider|observed)&limit=&offset=&current=1` | `Page<Price>` |
++| `GET /prices/history?model=&provider=` | `{ items: Price[] }` (all rows incl. closed, oldest first) |
++| `GET /prices/index?days=180` | `{ series: { day: string; median_input: Num; median_output: Num; min_input: Num; models: number }[]; movers: ChangeEvent[] }` computed from `prices` history |
++| `GET /benchmarks` | `{ items: (EntitySummary & { result_count: number; model_count: number; top: { model: EntitySummary; score: number } | null })[] }` |
++| `GET /benchmarks/{slug}/results?limit=&offset=&config=` | `Page<BenchmarkResult>` sorted by score (respecting `higher_is_better`), current rows only unless `history=1` |
++| `GET /benchmarks/{slug}/history?model=` | `{ items: BenchmarkResult[] }` over time |
++| `GET /hardware?kind=&manufacturer=&min_memory=&sort=` | `Page<EntitySummary>` |
++| `GET /hardware/fit?memory_gb=&quant=(4bit|8bit|fp16)&context=8192&limit=` | `{ inputs, assumptions: string[], items: { model: EntitySummary; parameter_count: number; estimated_memory_gb: number; fits: boolean; headroom_gb: number; quantization: string; note: string }[] }` — **ESTIMATED**: bytes/param (4bit 0.5+overhead 1.15, 8bit 1.0, fp16 2.0) + KV cache estimate; label as estimated |
++| `GET /explore/types` | `{ items: { entity_type, count, label }[] }` |
++| `GET /explore/{type}?q=&org=&sort=&limit=&offset=` | `Page<EntitySummary>` generic listing for any type (datasets, frameworks, tools, repositories, regulation…) |
++| `GET /changes?category=&type=&entity_type=&importance_min=&since=&until=&q=&limit=50&before=<observed_at cursor>` | `Page<ChangeEvent>` newest first; `type` = comma list of event types; excludes `DOCUMENT_CHANGED` unless `include_documents=1` |
++| `GET /changes/daily?date=YYYY-MM-DD` | `{ date, counts: Record<category, number>, sections: { category, label, items: ChangeEvent[] }[], new_models: EntitySummary[] }` ("What changed in AI today", DB-generated only) |
++| `GET /changes/categories?days=7` | `{ items: { category, event_type, count }[] }` |
++| `GET /timeline?entity=&year=&category=&limit=` | `{ items: { month: string; events: ChangeEvent[] }[] }` grouped by month; global when no entity |
++| `GET /compare?ids=slug,slug,…` (2–6) | `{ entity_type, dimensions: { key, label, unit?, kind: "number"|"text"|"list"|"bool"|"date" }[], items: { entity: EntitySummary; values: Record<key, unknown>; provenance: Provenance; prices?: Price[]; results?: BenchmarkResult[] }[] }` — dimensions per type: model (params, active params, context, max output, openness, license, modalities, release date, knowledge cutoff, best price in/out, benchmark scores shared by all), provider (models, min prices, features), hardware (memory, bandwidth, tdp, runtimes), framework (version, license, stars), company (country, founded, models, papers) |
++| `GET /diff?a=YYYY-MM-DD&b=YYYY-MM-DD&scope=(all|models|org:<slug>|family:<name>)` | `{ a, b, new_entities: EntitySummary[], gone_entities: EntitySummary[], property_changes: ChangeEvent[], price_changes: ChangeEvent[], benchmark_changes: ChangeEvent[], counts }` |
++| `GET /sources` | `{ items: { key, name, domain, tier, kind, category, organization: Org, enabled, documents, last_crawled_at, connectors: { name, label, health, last_success_at, interval_seconds }[] }[] }` (public transparency page) |
++| `GET /methodology` | `{ metrics: metric_definitions[], confidence_levels, tiers, event_types, extractors }` |
++| `GET /trending?days=7&limit=12` | `{ items: (EntitySummary & { views: number })[] }` from `page_views` |
++| `POST /views` `{ path }` | `{ ok: true }` (beacon; 1 req/s/IP) |
++| `GET /sitemap?type=&limit=5000&offset=0` | `{ items: { slug, entity_type, updated_at }[], total }` |
++| `GET /api-keys/me` (header `x-api-key`) | `{ label, plan, rate_per_min, usage_count }` — developer keys (later); public routes work without a key |
++
++### `EntityDetail`
++
++```ts
++EntitySummary & {
++  attributes: Record<string, unknown>            // full
++  provenance: Provenance
++  aliases: string[]; identifiers: { scheme: string; value: string }[]
++  relations: { predicate: string; direction: "out" | "in"; items: EntitySummary[]; total: number }[]   // grouped, ≤ 24 per group
++  sources: SourceRef[]                           // documents describing this entity (deduped by URL)
++  timeline: ChangeEvent[]                         // latest 30
++  quality: {...}; counts: {...}
++  // type-specific blocks (present only when relevant)
++  prices?: Price[]                                // current rows across providers (model) or for this provider (provider)
++  price_history?: Price[]                         // closed + current, oldest first (model)
++  results?: BenchmarkResult[]                     // model: its results · benchmark: leaderboard top 100
++  lineage?: { ancestors: EntitySummary[]; descendants: EntitySummary[]; quantizations: EntitySummary[] }   // model
++  providers?: EntitySummary[]                     // model
++  hardware_fit?: { hardware: EntitySummary; quantization: string; estimated_memory_gb: number; fits: boolean }[]   // model, ESTIMATED
++  models?: Page<EntitySummary>                    // company / provider / hardware(runnable)
++  papers?: EntitySummary[]                        // company / model
++  repositories?: EntitySummary[]                  // company / model / framework
++}
++```
++
++## Admin routes (`x-aia-admin-token`)
++
++| route | returns / action |
++|---|---|
++| `GET /admin/overview` | `{ stats, queue: Record<kind, Record<status, n>>, heartbeats, connectors: { ok, degraded, failing, disabled }, review_pending, recent_errors: n, llm: { jobs_24h, tokens_24h, by_stage } , archive }` |
++| `GET /admin/connectors` | full `connectors` rows + last run + docs/snapshots counts + source tier |
++| `POST /admin/connectors/{name}/run` `{ force?: bool }` | enqueue an immediate run (Redis flag `aia:run-now:<name>` consumed by the scheduler tick) → `{ queued: true }` |
++| `PATCH /admin/connectors/{name}` `{ enabled?, interval_seconds?, priority? }` | update |
++| `GET /admin/runs?connector=&limit=` · `GET /admin/errors?connector=&limit=` | rows |
++| `GET /admin/documents?connector=&status=&q=&limit=&offset=` · `GET /admin/documents/{id}` | documents (+ snapshots list); `GET /admin/snapshots/{id}` → metadata + `structured` + `diff` + first 20 kB of cleaned text (never raw HTML) |
++| `GET /admin/jobs?status=&kind=` · `POST /admin/jobs/{id}/retry` · `POST /admin/jobs/requeue-dead` | queue |
++| `GET /admin/llm-jobs?limit=` · `GET /admin/llm/health` | LLM factory accounting |
++| `GET /admin/review?status=pending&kind=` · `POST /admin/review/{id}` `{ action: "approve"|"reject"|"edit", resolution?: object }` | review queue; approving a `merge_candidate` calls `merge_entities(source, target)` (aliases/identifiers/claims/relations/events moved, `merged_into` set) |
++| `GET /admin/entities/duplicates?type=&limit=` | candidate pairs by normalized name similarity (`pg_trgm`) within a type |
++| `POST /admin/entities/merge` `{ source_id, target_id }` · `POST /admin/entities/{id}/claims/{claim_id}/retract` | curation |
++| `POST /admin/reprocess` `{ connector, url? }` | enqueue `reprocess_snapshot` |
++| `POST /admin/llm/enqueue` `{ limit?, task? }` | queue `llm_extract` for pending snapshots |
++| `GET /admin/infrastructure` | heartbeats, archive size, DB size (`pg_database_size`), table sizes, node hostname, uptime |
++| `POST /admin/cache/flush` | `{ flushed: n }` |
++| `POST /admin/stats/recompute` · `POST /admin/quality/recompute` | maintenance |

added docs/ARCHITECTURE.md +142 −0

++# AI Atlas — Technical architecture
++
++> The global intelligence layer for artificial intelligence: first-party connectors → raw historical archive → deterministic and
++> local-LLM extraction → temporal knowledge graph → search / compare / timeline → www.ai-atlas.co + API.
++
++## 1. Principles that shape the code
++
++1. **No critical dependence on third-party data APIs.** Every source is read directly (HTML, embedded JSON, JSON-LD, RSS/Atom, sitemaps,
++   Markdown docs, raw Git files, public JSON/CSV files). Scrapfly / Firecrawl / a headless browser are *optional escalation transports*
++   (`Fetcher(escalate=True)`), never the database.
++2. **Provenance first.** Every claim, relation, price, benchmark result and event stores `source_id`, `snapshot_id`, `source_url`, `tier`,
++   `confidence`, `extractor` and `observed_at`. Entity pages can always answer "where does this number come from?".
++3. **History is never disposable.** Claims are temporal (`valid_from`/`valid_to`, statuses `current|superseded|conflicting|retracted`),
++   prices and benchmark results are append-only, raw snapshots are content-addressed and kept forever, migrations are forward-only.
++4. **Deterministic before LLM.** Stage 1 (DOM, JSON-LD, tables, Markdown, feeds) runs on every document. The local LLM factory
++   (MacLustr llm-api.io through an OpenAI-compatible gateway) only sees documents flagged `needs_llm`, through versioned pydantic schemas,
++   with full token accounting.
++5. **Never fabricate.** Missing = missing. Conflicts between sources are stored side by side, flagged and queued for review — never averaged.
++
++## 2. Topology
++
++```
++                     ┌──────────────── aia schedule (PM2 ai-atlas-scheduler) ─────────────────┐
++ public AI web ──►   │ connectors (registry) → Fetcher (direct/escalate) → archive (raw+text) │
++                     │   → parse → extract (Facts) → FactWriter (resolve, version, diff)      │
++                     │   → jobs (llm_extract, embed, reprocess) → worker → LLM gateway         │
++                     └──────────────────────────────────────────────────────────────────────────┘
++                                                   │  Postgres 17 (+pgvector, pg_trgm) · Redis (cache/locks) · AIA_DATA_DIR
++                     ┌──────────────────────────────┴───────────────────────────────────────────┐
++                     │ FastAPI /api/v1 (PM2 ai-atlas-api :8321)  ◄──  Next.js 16 (PM2 ai-atlas-web :8320)  ◄── MacLustr Tunnel (BHS64 Caddy)
++                     └──────────────────────────────────────────────────────────────────────────┘
++```
++
++Production node: **M2M32c** (Mac Studio M2, 12 c / 32 GB, dedicated), Postgres 17 + pgvector + Redis via Homebrew, data in
++`~/ai-atlas-data/` (raw, text, backups, logs). Public route `https://www.ai-atlas.co → M2M32c:8320` on the MacLustr Tunnel gateway BHS64.
++Workers can run on any node with database access (`aia worker`), which is how the LLM factory scales across the cluster.
++
++## 3. Repository layout
++
++```
++registry/                 curated YAML: sources.yaml (domains, tiers, crawl policy), organizations.yaml (aliases, domains, hf/github orgs),
++                          providers.yaml, benchmarks.yaml, hardware.yaml — every entry carries its source_url
++migrations/versions/      forward-only SQL (0001_initial.py)
++src/aiatlas/
++  config.py               settings (AIA_*), ids.py (prefixed ULIDs, slugify, normalize_alias), logging.py, db/ (SQLAlchemy Core + asyncpg)
++  sdk/                    connector SDK
++    fetch.py              Fetcher: robots.txt, per-domain rate limit, ETag/If-Modified-Since, retries/backoff, escalation chain
++    archive.py            content-addressed gzip store (raw/, text/) with dedupe
++    extract/              html (selectolax; meta, JSON-LD, embedded JSON, tables, links, text), feeds, sitemap, markdown (front matter, tables),
++                          numbers (70B, 128K, $3/1M), dates (partial precision)
++    facts.py              EntityRef, Claim, Relation, Event, PriceObs, ResultObs, Target, Facts (+ MATERIAL_PROPERTIES → event types)
++    resolution.py         Resolver: identifiers → aliases (org-disambiguated) → slug → create; ambiguity → review queue
++    writer.py             FactWriter: temporal claims, attributes/provenance materialisation, conflicts, prices, results, events
++    connector.py          BaseConnector.run(): documents, snapshots, hash change detection, structural diff, DOCUMENT_CHANGED,
++                          breakage detection, circuit breaker, adaptive interval, follow-up targets, --file overrides, reprocess mode
++  connectors/             one package per group (labs/, hub/, research/, code/, providers/, benchmarks/, hardware/); CONNECTORS = [cls]
++  schemas/extraction.py   pydantic schemas: ModelPassport, PricingExtraction, CompanyPassport, PaperPassport, BenchmarkResultExtraction, HardwareSpec, …
++  services/
++    jobs.py               Postgres queue (SKIP LOCKED), handlers.py (llm_extract, embed_entity, reprocess_snapshot, recompute_quality)
++    llm/gateway.py        LLMGateway: stage cascade small→medium→large, OpenAI-compatible engine, JSON validation, llm_jobs accounting
++    embeddings.py         local embeddings → pgvector · search.py FTS+trigram(+vector) and NL→filters compiler
++    scheduler.py          APScheduler: due connectors (Redis lock), worker, hourly stats/quality/embeddings, nightly pg_dump
++    quality.py            transparency scores (documented in metric_definitions) · stats.py live counters · backup.py · cache.py
++  api/                    FastAPI application (see docs/API.md)
++  cli.py                  `aia`: migrate seed connectors run reprocess crawl status stats quality schedule worker api backup llm embed search review
++apps/web/                 Next.js 16 site (see docs/FRONTEND.md)
++tests/                    pytest with saved fixtures (tests/fixtures/<connector>/…); live tests behind `-m live`
++deploy/                   mld manifest (ai-atlas.mld.json), render-manifest.sh, first-run.sh — see docs/DEPLOY.md
++```
++
++## 4. Data model (Postgres)
++
++| table | role |
++|---|---|
++| `entities` | unified graph node: `entity_type`, `canonical_name`, `slug`, `organization_id`, `attributes` (current value per property), `provenance` (per property), `quality`, `counts`, `first_seen_at`, `search` tsvector |
++| `entity_aliases`, `entity_identifiers` | resolution keys (normalized aliases; `(scheme, value)` unique: hf_repo, github_repo, arxiv, doi, anthropic_model_id, openrouter, pypi, domain…) |
++| `sources`, `connectors`, `connector_runs`, `connector_errors` | source registry with tiers; connector state (health, adaptive interval, circuit, parser_version, expected_min_records) |
++| `documents` | one row per URL: validators (etag/last-modified), `content_hash`, counters, `entity_id`, `needs_llm` |
++| `snapshots` | one row per *changed* fetch: `raw_path`, `text_path`, `structured` (deterministic summary), `diff`, `transport`, `parser_version`, `processing_status` |
++| `claims` | temporal facts: `(entity_id, property, value jsonb, unit, tier, confidence, status, valid_from, valid_to, snapshot_id, source_url, extractor)` |
++| `relations` | typed edges with validity: `develops, owns, operates, available_through, evaluated_on, described_by, derived_from, fine_tuned_from, quantized_from, superseded_by, runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates` |
++| `change_events` | the event engine: `NEW_MODEL, MODEL_UPDATED, PRICE_CHANGED, PROVIDER_LISTED, CONTEXT_CHANGED, STATUS_CHANGED, LICENSE_CHANGED, DEPRECATION_ANNOUNCED, RETIREMENT_ANNOUNCED, BENCHMARK_RESULT, BENCHMARK_UPDATED, NEW_PAPER, ANNOUNCEMENT, RELEASE, VERSION_RELEASED, DOCUMENT_CHANGED, …` with `category`, `importance 0–3`, `dedupe_key` |
++| `prices` | append-only provider pricing (`valid_to` closes a row): per-1M-token input/output/cached/cache-write/batch, per image/request, context, features |
++| `benchmark_results` | append-only results with `config` (harness, prompting, judge…) — never compared blindly |
++| `jobs`, `llm_jobs` | work queue; LLM cost accounting (stage, model, node, tokens, duration, status) |
++| `review_queue` | merge candidates, conflicts, blocked sources, parser breakage, unusual changes |
++| `entity_embeddings` | pgvector(1024) local embeddings |
++| `domains`, `stats_snapshots`, `page_views`, `api_keys`, `metric_definitions` | trust graph, counters history, trending, developer keys, documented metrics |
++
++Identifiers: prefixed ULIDs (`model_01J…`, `company_…`, `paper_…`, `snap_…`, `evt_…`). Slugs are stable, readable and unique.
++
++### Temporal rules (FactWriter)
++
++```
++no current claim         → insert current, set attribute, NEW_<TYPE> event when the entity is new
++same value               → confirm (observed_at)
++different, source ≥ tier → supersede (valid_to = observed), insert current, CHANGE event for material properties
++different, worse source  → store as `conflicting`, mark current `conflicted`, review-queue item — never overwrite
++soft text (description…) → first statement wins until its own source changes; never an event
++metric.* / stats.*       → time series without events
++```
++
++Historical mode = `claims` filtered by `valid_from <= date < coalesce(valid_to, ∞)`; "did it exist?" = `entities.first_seen_at <= date`.
++Diff A→B = new/removed entities by `first_seen_at`, claims superseded between A and B, price rows opened/closed, results observed.
++
++## 5. Connector lifecycle
++
++`discover()` returns `Target`s (URL, doc_type, optional entity hint, `needs_llm`, `key` for fixtures). For each target `run()`:
++fetch (conditional) → unchanged? stop · changed → archive raw + text → snapshot row (+ structural diff vs previous) → `parse()` →
++`extract()` → `FactWriter.write(facts)` → follow-up targets → optional `llm_extract` job. Blocked (401/403/451/robots) → document
++`blocked` + review item; 404/410 twice → `gone` (never deletes entities). Fewer records than `expected_min_records` → run `suspect`
+++ review item. Intervals adapt: ×0.7 on change (≥ min), ×1.5 after 3 unchanged runs (≤ max). Three failures open the circuit 30 min.
++
++Parser improvements: bump `parser_version`, then `aia reprocess <connector>` replays the latest stored snapshot of every document — no crawl.
++
++## 6. LLM factory
++
++`LLMGateway.extract(task_type, document, schema, stage)` → strict JSON validated by a pydantic schema; on schema errors the next stage
++model is tried (small → medium → large); transport failures do not escalate. Every call is recorded in `llm_jobs` (tokens, duration,
++node, status, output). `facts_from_llm()` maps outputs onto `Facts` with `extractor='llm'` and medium/low confidence, so deterministic
++tier-1 claims always win conflicts. Embeddings use the same gateway (`/v1/embeddings`, Qwen3-Embedding 1024-d) into pgvector.
++Without a configured gateway the platform degrades gracefully: deterministic extraction, FTS search, no embeddings.
++
++## 7. Quality, search, events
++
++* `quality.score` = 100·(0.25 completeness + 0.25 primary-source ratio + 0.20 freshness + 0.15 agreement + 0.15 source diversity),
++  versioned and explained in `metric_definitions` and `/methodology`. It measures how well AI Atlas knows an entity, not the entity.
++* Search: tsvector (name A, family B, type/description C) + `pg_trgm` similarity + aliases (+ cosine on embeddings when present).
++  `compile_query()` turns "open models released in 2026 with more than 100B parameters and 128k context" into structured filters.
++* Events power the homepage feed, entity timelines, `/changes`, the daily digest and, later, watchlists/alerts.
++
++## 8. Security & compliance
++
++Admin routes require `x-aia-admin-token`. The API binds to loopback; Next.js rewrites `/api/v1/*` to it. No credentials or internal
++addresses in responses. Crawling honours robots.txt, identifies itself (`AIAtlasBot/0.1 … contact@spboucher.ai`), rate-limits per domain,
++uses conditional requests, never bypasses access controls; blocked sources go to the review queue. Raw archive content is never exposed
++publicly — only extracted facts with links to the original page.
++
++## 9. Backups & durability
++
++Nightly `pg_dump -Fc` into `AIA_DATA_DIR/backups` (30 kept), off-node rsync (`scripts/backup-offnode.sh`), raw archive is append-only and
++content-addressed (rsync-friendly). The dataset is the product: restore = `pg_restore` + point `DATABASE_URL`.

added docs/CONNECTORS.md +97 −0

++# Writing a connector
++
++A connector is a Python class under `src/aiatlas/connectors/<group>/<name>.py`, listed in that module's `CONNECTORS = [Cls]`.
++The registry auto-discovers it; `aia seed` registers it in the database; `aia run <name>` runs it; the scheduler runs it on its interval.
++
++## Skeleton
++
++```python
++from aiatlas.registry import org_ref, provider_ref
++from aiatlas.sdk.connector import BaseConnector, Parsed, RunContext
++from aiatlas.sdk.facts import Facts, Target
++from aiatlas.sdk.fetch import FetchResult
++
++class ExampleConnector(BaseConnector):
++    name = "example"                       # unique, snake_case, stable (used in documents/claims/events)
++    label = "Example Lab — models & news"
++    description = "Official docs and blog of Example Lab."
++    source_key = "example.com"             # must exist in registry/sources.yaml (tier, rate limit, org)
++    version = "1"; parser_version = "1"    # bump parser_version when extraction improves → `aia reprocess example`
++    interval_seconds = 3600; min_interval_seconds = 1800; max_interval_seconds = 86400
++    rate_per_min = 15; tier = 1; priority = 0 | 1 | 2
++    expected_min_records = 5               # breakage detection: fewer entities/prices/results ⇒ run "suspect", nothing deleted
++    concurrency = 2
++
++    async def discover(self, ctx: RunContext) -> list[Target]:
++        return [Target(url="https://example.com/models", doc_type="model_docs", key="models", min_bytes=2000),
++                Target(url="https://example.com/blog/rss.xml", doc_type="feed", key="feed")]
++
++    async def extract(self, ctx: RunContext, target: Target, res: FetchResult, parsed: Parsed) -> Facts:
++        facts = Facts()
++        org = org_ref("example")            # registry organization → deterministic identifiers
++        facts.entities.append(org)
++        if target.key == "models" and parsed.html:
++            for row in parsed.html.tables[0]["rows"]: ...
++                model = facts.entity("model", name, identifiers={"example_model_id": api_id}, organization=org)
++                facts.relate(org, "develops", model)
++                facts.claim(model, "context_length", 128000, unit="tokens")
++                facts.price(model=model, provider=provider_ref("example"), input_per_mtok=1.0, output_per_mtok=4.0)
++                facts.follow(detail_url, doc_type="model_page", entity=model, needs_llm=True, meta={"llm_task": "model_passport"})
++        elif target.key == "feed" and parsed.kind == "feed":
++            announcement_events(facts, org, parsed.feed_items, source_name="example.com/blog")   # from connectors/labs/_common.py
++        return facts
++
++CONNECTORS = [ExampleConnector]
++```
++
++## What `Parsed` gives you
++
++| `parsed.kind` | fields |
++|---|---|
++| `html` | `parsed.html: HtmlDoc` — `title, canonical, description, meta, og, json_ld, embedded_json (__NEXT_DATA__, data-props…), headings, tables [{headers, rows}], links [(href, text)], text`, `css()`, `links_matching()` ; `find_in_json(obj, key)` helper |
++| `markdown` | `parsed.markdown: MarkdownDoc` — `front_matter, headings, tables, links, text, section(pattern)` |
++| `feed` | `parsed.feed_items: list[FeedItem]` (`url, title, summary, published_at, categories, authors`) |
++| `json` | `parsed.json` |
++| `pdf` / `text` / `xml` | `parsed.text` |
++
++Helpers: `aiatlas.sdk.extract.numbers` (`parse_param_count("70B")`, `parse_active_params("235B-A22B")`, `parse_context_length("128K")`,
++`parse_money_per_mtok("$3 / 1M tokens")`, `parse_percent`), `aiatlas.sdk.extract.dates` (`parse_datetime`, `parse_date` with precision),
++`connectors/labs/_common.py` (`announcement_events`, `transpose_feature_table`, `kv_tables`, `clean_cell`, `money`, `tokens`, `month_year`, `parse_retirement`, `model_ref`).
++
++## Rules
++
++1. **Direct mode first.** Use official pages, feeds, sitemaps, Markdown/raw files, embedded JSON. Public JSON files (e.g. `openrouter.ai/api/v1/models`,
++   `pypi.org/pypi/<pkg>/json`, arXiv Atom) are fine — they are public documents, not commercial APIs. Never require an API key.
++   Set `Target(escalate=True)` only for pages known to block bots *and* only if a key is configured; otherwise let the document be `blocked`.
++2. **Never fabricate.** Only emit a claim when the page states it. Unknown → no claim. Don't infer parameter counts from names unless the
++   name literally contains them (`Qwen3-235B-A22B` → 235e9 / 22e9 is fine; "Large" is not).
++3. **Identifiers make resolution deterministic.** Give models the provider's API id (`{provider}_model_id`), HF repo (`hf_repo`), arXiv id
++   (`arxiv`), GitHub repo (`github_repo`), PyPI name (`pypi`). Organizations always come from `org_ref(<registry key>)` (add missing orgs to
++   `registry/organizations.yaml` with a `source_url`).
++4. **Properties are shared vocabulary** (see below). Add new ones sparingly; prefix metrics with `metric.` (downloads, likes, stars) so they
++   never generate events.
++5. **Events**: NEW_*, PRICE_CHANGED, CONTEXT_CHANGED… are emitted automatically by the writer. Emit `ANNOUNCEMENT`/`RELEASE` events yourself for
++   feed items (`announcement_events`) with `effective_at` = publication date and a `dedupe_key` (URL).
++6. **Follow-ups** (`facts.follow`) let a listing discover detail pages; keep `max_targets` reasonable (`ctx.max_targets`, default 2000).
++7. **Every connector has a fixture test**: save real responses under `tests/fixtures/<connector>/…` and assert extracted facts
++   (see `tests/test_anthropic.py`). Run connectors with `--file key=path` to use fixtures instead of the network.
++8. Respect `rate_per_min` from `registry/sources.yaml`; arXiv ≤ 4/min; Hugging Face ≤ 30/min; GitHub ≤ 20/min.
++
++## Property vocabulary (entities.attributes)
++
++**model**: `family, version, release_date (ISO, may be YYYY-MM), status (active|preview|deprecated|retired|announced|limited-availability), openness
++(open-weights|open-source|proprietary|restricted), license, architecture, parameter_count (int), active_parameter_count, is_moe, modalities
++(list of text|image|audio|video|code|embedding), modalities_input, modalities_output, context_length (tokens), max_output_tokens, knowledge_cutoff
++(YYYY-MM), training_data_cutoff, languages, tool_calling, structured_output, reasoning, vision, audio, fine_tuning_available, tokenizer,
++api_model_id, api_alias, official_url, model_card_url, paper_url, repository_url, hf_repo, pipeline_tag, base_model, quantization, quant_format
++(gguf|mlx|awq|gptq|fp8), file_size_gb, deprecation_date, retirement_date, retirement_tentative, metric.downloads, metric.likes`
++
++**company / organization / lab**: `country (ISO-2), headquarters, founded, website, domains, hf_org, github_org, org_kind, legal_name, founders, leadership, employee_count`
++
++**provider**: `website, pricing_url, docs_url, regions, features` · **paper**: `authors, published_at, updated_at, abstract, arxiv_id, doi, categories, primary_category, pdf_url, code_url, venue`
++· **benchmark**: `category, task, metric, unit, creator, website, paper, known_limitations, methodology` · **hardware**: `kind, architecture, release_date, memory_gb,
++memory_type, memory_bandwidth_gbs, tdp_watts, runtimes, manufacturer, spec_url, price_usd, compute_fp16_tflops` · **framework / repository**: `repository_url, latest_version,
++latest_release_at, license, language, description, topics, pypi, metric.stars, metric.forks` · **dataset**: `license, modality, size, publisher, task, hf_repo`
++
++Relations: `develops, owns, operates, available_through, evaluated_on, described_by, derived_from, fine_tuned_from, quantized_from, distilled_from,
++merged_from, superseded_by, runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates, published_by`.

added migrations/env.py +45 −0

++"""Alembic environment (async engine). Migrations are plain SQL executed through `op.execute`."""
++from __future__ import annotations
++
++import asyncio
++from logging.config import fileConfig
++
++from alembic import context
++from sqlalchemy.ext.asyncio import create_async_engine
++
++from aiatlas.config import settings
++
++config = context.config
++if config.config_file_name is not None:
++    fileConfig(config.config_file_name)
++
++target_metadata = None
++
++
++def run_migrations_offline() -> None:
++    context.configure(url=settings.sync_database_url, literal_binds=True, dialect_opts={"paramstyle": "named"})
++    with context.begin_transaction():
++        context.run_migrations()
++
++
++def do_run_migrations(connection) -> None:  # type: ignore[no-untyped-def]
++    context.configure(connection=connection, target_metadata=target_metadata, transaction_per_migration=True)
++    with context.begin_transaction():
++        context.run_migrations()
++
++
++async def run_async_migrations() -> None:
++    engine = create_async_engine(settings.database_url, poolclass=None)
++    async with engine.connect() as connection:
++        await connection.run_sync(do_run_migrations)
++    await engine.dispose()
++
++
++def run_migrations_online() -> None:
++    asyncio.run(run_async_migrations())
++
++
++if context.is_offline_mode():
++    run_migrations_offline()
++else:
++    run_migrations_online()

added migrations/script.py.mako +21 −0

++"""${message}
++
++Revision ID: ${up_revision}
++Revises: ${down_revision | comma,n}
++"""
++from __future__ import annotations
++
++from alembic import op
++
++revision = ${repr(up_revision)}
++down_revision = ${repr(down_revision)}
++branch_labels = ${repr(branch_labels)}
++depends_on = ${repr(depends_on)}
++
++
++def upgrade() -> None:
++    ${upgrades if upgrades else "pass"}
++
++
++def downgrade() -> None:
++    raise RuntimeError("forward-only")

added migrations/versions/0001_initial.py +502 −0

++"""AI Atlas initial schema — entities, aliases, identifiers, sources, connectors, documents, snapshots, claims, relations,
++change events, benchmark results, prices, jobs, LLM jobs, review queue, stats.
++
++Revision ID: 0001
++"""
++from __future__ import annotations
++
++from alembic import op
++
++revision = "0001"
++down_revision = None
++branch_labels = None
++depends_on = None
++
++SQL = r"""
++create extension if not exists pg_trgm;
++create extension if not exists "uuid-ossp";
++do $$ begin
++  create extension if not exists vector;
++exception when others then
++  raise notice 'pgvector unavailable: %', sqlerrm;
++end $$;
++
++-- ---------------------------------------------------------------------------------------------- reference: sources & connectors
++create table if not exists sources (
++  id                text primary key,
++  key               text not null unique,               -- e.g. 'openai.com', 'huggingface.co', 'arxiv.org'
++  name              text not null,
++  domain            text not null,
++  organization_id   text,                               -- entity id of the owning organization (domain trust graph)
++  tier              smallint not null default 2,        -- 1 official primary, 2 quality secondary, 3 community, 4 unverified
++  kind              text not null default 'website',    -- website|docs|feed|repository|registry|dataset|leaderboard|regulator
++  category          text not null default 'lab',
++  base_url          text,
++  robots_policy     text not null default 'respect',    -- respect|documented-exception
++  rate_limit_per_min integer not null default 30,
++  crawl_interval_s  integer not null default 86400,
++  enabled           boolean not null default true,
++  priority          smallint not null default 2,        -- 0 = P0
++  notes             text,
++  meta              jsonb not null default '{}'::jsonb,
++  created_at        timestamptz not null default now(),
++  updated_at        timestamptz not null default now()
++);
++
++create table if not exists connectors (
++  name                 text primary key,
++  source_id            text references sources(id),
++  label                text not null,
++  description          text,
++  enabled              boolean not null default true,
++  priority             smallint not null default 2,
++  interval_seconds     integer not null default 3600,
++  min_interval_seconds integer not null default 900,
++  max_interval_seconds integer not null default 604800,
++  parser_version       text not null default '1',
++  rate_limit_per_min   integer not null default 30,
++  owner                text not null default 'ai-atlas',
++  expected_min_records integer not null default 0,      -- breakage detection: fewer records than this = suspected failure
++  last_attempt_at      timestamptz,
++  last_success_at      timestamptz,
++  last_change_at       timestamptz,
++  next_run_at          timestamptz,
++  last_duration_ms     integer,
++  consecutive_failures integer not null default 0,
++  consecutive_unchanged integer not null default 0,
++  circuit_open_until   timestamptz,
++  health               text not null default 'unknown', -- ok|degraded|failing|disabled|unknown
++  meta                 jsonb not null default '{}'::jsonb,
++  created_at           timestamptz not null default now(),
++  updated_at           timestamptz not null default now()
++);
++
++create table if not exists connector_runs (
++  id               text primary key,
++  connector_name   text not null references connectors(name),
++  started_at       timestamptz not null,
++  finished_at      timestamptz,
++  status           text not null,                       -- running|success|unchanged|failed|skipped|suspect
++  duration_ms      integer,
++  docs_discovered  integer not null default 0,
++  docs_fetched     integer not null default 0,
++  docs_changed     integer not null default 0,
++  docs_unchanged   integer not null default 0,
++  docs_failed      integer not null default 0,
++  entities_created integer not null default 0,
++  entities_updated integer not null default 0,
++  claims_written   integer not null default 0,
++  relations_written integer not null default 0,
++  events_emitted   integer not null default 0,
++  error            text,
++  meta             jsonb not null default '{}'::jsonb
++);
++create index if not exists connector_runs_name_idx on connector_runs (connector_name, started_at desc);
++
++create table if not exists connector_errors (
++  id             bigserial primary key,
++  connector_name text not null,
++  run_id         text,
++  url            text,
++  error_type     text not null,
++  message        text not null,
++  created_at     timestamptz not null default now()
++);
++create index if not exists connector_errors_name_idx on connector_errors (connector_name, created_at desc);
++
++-- ---------------------------------------------------------------------------------------------- entities (unified graph)
++create table if not exists entities (
++  id              text primary key,
++  entity_type     text not null,
++  canonical_name  text not null,
++  slug            text not null unique,
++  description     text,
++  status          text not null default 'active',     -- active|deprecated|retired|announced|unknown|merged
++  organization_id text references entities(id),
++  attributes      jsonb not null default '{}'::jsonb, -- current value per property (materialised from claims)
++  provenance      jsonb not null default '{}'::jsonb, -- property -> {source, snapshot_id, url, observed_at, tier, confidence}
++  quality         jsonb not null default '{}'::jsonb, -- source_count, primary_source_ratio, freshness, completeness, agreement, score
++  counts          jsonb not null default '{}'::jsonb, -- cached relation / claim / event counts
++  first_seen_at   timestamptz not null default now(),
++  last_seen_at    timestamptz not null default now(),
++  merged_into     text references entities(id),
++  search          tsvector,
++  created_at      timestamptz not null default now(),
++  updated_at      timestamptz not null default now()
++);
++create index if not exists entities_type_idx on entities (entity_type, canonical_name);
++create index if not exists entities_org_idx on entities (organization_id);
++create index if not exists entities_search_idx on entities using gin (search);
++create index if not exists entities_name_trgm_idx on entities using gin (canonical_name gin_trgm_ops);
++create index if not exists entities_attrs_idx on entities using gin (attributes jsonb_path_ops);
++create index if not exists entities_updated_idx on entities (updated_at desc);
++create index if not exists entities_first_seen_idx on entities (first_seen_at desc);
++
++create table if not exists entity_aliases (
++  id          bigserial primary key,
++  entity_id   text not null references entities(id) on delete cascade,
++  alias       text not null,
++  alias_norm  text not null,
++  kind        text not null default 'alias',            -- alias|former_name|abbreviation|localized|typo
++  snapshot_id text,
++  created_at  timestamptz not null default now(),
++  unique (entity_id, alias_norm)
++);
++create index if not exists entity_aliases_norm_idx on entity_aliases (alias_norm);
++
++create table if not exists entity_identifiers (
++  id          bigserial primary key,
++  entity_id   text not null references entities(id) on delete cascade,
++  scheme      text not null,                            -- hf_repo|github_repo|arxiv|doi|openrouter|domain|pypi|wikidata|url|provider_model_id
++  value       text not null,
++  snapshot_id text,
++  created_at  timestamptz not null default now(),
++  unique (scheme, value)
++);
++create index if not exists entity_identifiers_entity_idx on entity_identifiers (entity_id);
++
++-- ---------------------------------------------------------------------------------------------- documents & snapshots (raw archive)
++create table if not exists documents (
++  id               text primary key,
++  source_id        text references sources(id),
++  connector_name   text references connectors(name),
++  url              text not null unique,
++  canonical_url    text,
++  doc_type         text not null default 'page',       -- page|feed|feed_item|model_card|docs|pricing|release|paper|pdf|json|sitemap|repo|readme|leaderboard
++  title            text,
++  entity_id        text references entities(id),
++  status           text not null default 'active',     -- active|gone|blocked|error
++  first_seen_at    timestamptz not null default now(),
++  last_fetched_at  timestamptz,
++  last_changed_at  timestamptz,
++  last_status      integer,
++  etag             text,
++  last_modified    text,
++  content_hash     text,
++  fetch_count      integer not null default 0,
++  change_count     integer not null default 0,
++  fail_count       integer not null default 0,
++  fetch_interval_s integer,
++  next_fetch_at    timestamptz,
++  priority         smallint not null default 2,
++  needs_llm        boolean not null default false,
++  meta             jsonb not null default '{}'::jsonb
++);
++create index if not exists documents_connector_idx on documents (connector_name, last_fetched_at desc);
++create index if not exists documents_entity_idx on documents (entity_id);
++create index if not exists documents_next_idx on documents (next_fetch_at) where status = 'active';
++
++create table if not exists snapshots (
++  id                text primary key,
++  document_id       text not null references documents(id) on delete cascade,
++  run_id            text,
++  url               text not null,
++  final_url         text,
++  observed_at       timestamptz not null default now(),
++  http_status       integer,
++  headers           jsonb not null default '{}'::jsonb,
++  content_type      text,
++  content_hash      text not null,
++  byte_size         integer not null default 0,
++  raw_path          text,                              -- content-addressed gzip under AIA_DATA_DIR/raw
++  text_path         text,                              -- cleaned text (gzip) under AIA_DATA_DIR/text
++  text_hash         text,
++  structured        jsonb,                             -- deterministic extraction (json-ld, meta, tables, embedded json)
++  parser_version    text not null default '1',
++  connector_version text not null default '1',
++  transport         text not null default 'direct',    -- direct|browser|scrapfly|firecrawl|file|git
++  changed           boolean not null default true,
++  diff              jsonb,                             -- structural diff vs previous changed snapshot
++  processing_status text not null default 'stored',    -- stored|extracted|llm_pending|llm_done|failed
++  created_at        timestamptz not null default now()
++);
++create index if not exists snapshots_doc_idx on snapshots (document_id, observed_at desc);
++create index if not exists snapshots_hash_idx on snapshots (content_hash);
++create index if not exists snapshots_status_idx on snapshots (processing_status) where processing_status in ('stored','llm_pending');
++
++-- ---------------------------------------------------------------------------------------------- temporal facts
++create table if not exists claims (
++  id                text primary key,
++  entity_id         text not null references entities(id) on delete cascade,
++  property          text not null,
++  value             jsonb not null,
++  value_text        text,
++  value_num         double precision,
++  unit              text,
++  source_id         text references sources(id),
++  snapshot_id       text references snapshots(id),
++  source_url        text,
++  tier              smallint not null default 2,
++  confidence        text not null default 'medium',     -- verified|high|medium|low|conflicted
++  status            text not null default 'current',    -- current|superseded|conflicting|retracted
++  extractor         text not null default 'deterministic',
++  extractor_version text not null default '1',
++  observed_at       timestamptz not null default now(),
++  effective_at      timestamptz,
++  valid_from        timestamptz not null default now(),
++  valid_to          timestamptz,
++  created_at        timestamptz not null default now()
++);
++create index if not exists claims_entity_prop_idx on claims (entity_id, property, valid_from desc);
++create index if not exists claims_current_idx on claims (entity_id, property) where status = 'current';
++create index if not exists claims_snapshot_idx on claims (snapshot_id);
++create index if not exists claims_observed_idx on claims (observed_at desc);
++
++create table if not exists relations (
++  id          text primary key,
++  subject_id  text not null references entities(id) on delete cascade,
++  predicate   text not null,
++  object_id   text not null references entities(id) on delete cascade,
++  attributes  jsonb not null default '{}'::jsonb,
++  source_id   text references sources(id),
++  snapshot_id text references snapshots(id),
++  source_url  text,
++  tier        smallint not null default 2,
++  confidence  text not null default 'medium',
++  observed_at timestamptz not null default now(),
++  valid_from  timestamptz not null default now(),
++  valid_to    timestamptz,
++  created_at  timestamptz not null default now()
++);
++create unique index if not exists relations_live_uniq on relations (subject_id, predicate, object_id) where valid_to is null;
++create index if not exists relations_subject_idx on relations (subject_id, predicate);
++create index if not exists relations_object_idx on relations (object_id, predicate);
++
++create table if not exists change_events (
++  id             text primary key,
++  entity_id      text references entities(id) on delete cascade,
++  event_type     text not null,                          -- NEW_MODEL|MODEL_UPDATED|PRICE_CHANGED|CONTEXT_CHANGED|NEW_PAPER|RELEASE|BENCHMARK_RESULT|NEW_PROVIDER|DOCUMENT_CHANGED|…
++  category       text not null default 'update',         -- model|price|benchmark|paper|release|company|hardware|framework|provider|dataset|regulation|incident|repository
++  property       text,
++  old_value      jsonb,
++  new_value      jsonb,
++  summary        text,
++  importance     smallint not null default 2,            -- 0 minor … 3 major
++  observed_at    timestamptz not null default now(),
++  effective_at   timestamptz,
++  source_id      text references sources(id),
++  snapshot_id    text references snapshots(id),
++  source_url     text,
++  connector_name text,
++  dedupe_key     text unique,
++  meta           jsonb not null default '{}'::jsonb
++);
++create index if not exists change_events_observed_idx on change_events (observed_at desc);
++create index if not exists change_events_entity_idx on change_events (entity_id, observed_at desc);
++create index if not exists change_events_type_idx on change_events (event_type, observed_at desc);
++create index if not exists change_events_category_idx on change_events (category, observed_at desc);
++
++-- ---------------------------------------------------------------------------------------------- domain tables
++create table if not exists benchmark_results (
++  id            text primary key,
++  model_id      text not null references entities(id) on delete cascade,
++  benchmark_id  text not null references entities(id) on delete cascade,
++  score         double precision not null,
++  metric        text,
++  unit          text,
++  higher_is_better boolean not null default true,
++  config        jsonb not null default '{}'::jsonb,     -- prompting, judge, tool use, sampling, harness, model variant string
++  evaluated_at  timestamptz,
++  observed_at   timestamptz not null default now(),
++  source_id     text references sources(id),
++  snapshot_id   text references snapshots(id),
++  source_url    text,
++  tier          smallint not null default 2,
++  confidence    text not null default 'medium',
++  dedupe_key    text unique,
++  valid_to      timestamptz
++);
++create index if not exists benchmark_results_model_idx on benchmark_results (model_id, benchmark_id, observed_at desc);
++create index if not exists benchmark_results_bench_idx on benchmark_results (benchmark_id, score desc);
++
++create table if not exists prices (
++  id                     text primary key,
++  model_id               text not null references entities(id) on delete cascade,
++  provider_id            text not null references entities(id) on delete cascade,
++  provider_model_id      text,
++  input_per_mtok         double precision,
++  output_per_mtok        double precision,
++  cached_input_per_mtok  double precision,
++  cache_write_per_mtok   double precision,
++  batch_input_per_mtok   double precision,
++  batch_output_per_mtok  double precision,
++  per_image              double precision,
++  per_request            double precision,
++  currency               text not null default 'USD',
++  context_length         integer,
++  max_output_tokens      integer,
++  features               jsonb not null default '{}'::jsonb,
++  observed_at            timestamptz not null default now(),
++  valid_from             timestamptz not null default now(),
++  valid_to               timestamptz,
++  source_id              text references sources(id),
++  snapshot_id            text references snapshots(id),
++  source_url             text,
++  tier                   smallint not null default 2,
++  meta                   jsonb not null default '{}'::jsonb
++);
++create unique index if not exists prices_live_uniq on prices (model_id, provider_id, coalesce(provider_model_id, '')) where valid_to is null;
++create index if not exists prices_provider_idx on prices (provider_id, valid_from desc);
++create index if not exists prices_model_idx on prices (model_id, valid_from desc);
++
++create table if not exists domains (
++  domain          text primary key,
++  organization_id text references entities(id),
++  trust_tier      smallint not null default 3,
++  category        text,
++  notes           text,
++  created_at      timestamptz not null default now()
++);
++
++-- ---------------------------------------------------------------------------------------------- work queues
++create table if not exists jobs (
++  id           text primary key,
++  kind         text not null,                            -- fetch_document|llm_extract|embed_entity|reprocess_snapshot|recompute_quality|resolve_entity
++  payload      jsonb not null default '{}'::jsonb,
++  priority     smallint not null default 5,              -- 0 highest
++  status       text not null default 'queued',           -- queued|running|done|failed|dead
++  attempts     integer not null default 0,
++  max_attempts integer not null default 3,
++  run_after    timestamptz not null default now(),
++  locked_by    text,
++  locked_at    timestamptz,
++  started_at   timestamptz,
++  finished_at  timestamptz,
++  error        text,
++  batch_id     text,
++  dedupe_key   text,
++  created_at   timestamptz not null default now()
++);
++create index if not exists jobs_pick_idx on jobs (status, priority, run_after) where status = 'queued';
++create unique index if not exists jobs_dedupe_idx on jobs (dedupe_key) where status in ('queued','running');
++create index if not exists jobs_batch_idx on jobs (batch_id);
++
++create table if not exists llm_jobs (
++  id             text primary key,
++  job_id         text,
++  task_type      text not null,
++  stage          text not null,                          -- small|medium|large
++  engine         text not null,
++  model          text not null,
++  node           text,
++  schema_name    text,
++  snapshot_id    text references snapshots(id),
++  entity_id      text references entities(id),
++  input_tokens   integer,
++  output_tokens  integer,
++  duration_ms    integer,
++  status         text not null,                          -- ok|failed|invalid_json|schema_error
++  output         jsonb,
++  error          text,
++  created_at     timestamptz not null default now()
++);
++create index if not exists llm_jobs_created_idx on llm_jobs (created_at desc);
++
++create table if not exists review_queue (
++  id          text primary key,
++  kind        text not null,                             -- merge_candidate|conflict|blocked_source|parser_breakage|unusual_change|new_source
++  entity_ids  text[] not null default '{}',
++  payload     jsonb not null default '{}'::jsonb,
++  reason      text not null,
++  status      text not null default 'pending',           -- pending|approved|rejected|edited
++  resolution  jsonb,
++  created_at  timestamptz not null default now(),
++  resolved_at timestamptz,
++  dedupe_key  text unique
++);
++create index if not exists review_queue_status_idx on review_queue (status, created_at desc);
++
++-- ---------------------------------------------------------------------------------------------- embeddings (optional; local models)
++do $$ begin
++  if exists (select 1 from pg_extension where extname = 'vector') then
++    execute 'create table if not exists entity_embeddings (
++      entity_id text primary key references entities(id) on delete cascade,
++      model text not null,
++      embedding vector(1024) not null,
++      text_hash text not null,
++      created_at timestamptz not null default now())';
++  end if;
++end $$;
++
++-- ---------------------------------------------------------------------------------------------- stats, views, api keys
++create table if not exists stats_snapshots (
++  id          bigserial primary key,
++  computed_at timestamptz not null default now(),
++  counts      jsonb not null
++);
++
++create table if not exists page_views (
++  path  text not null,
++  day   date not null,
++  views integer not null default 0,
++  primary key (path, day)
++);
++
++create table if not exists api_keys (
++  id          text primary key,
++  key_hash    text not null unique,
++  label       text not null,
++  owner_email text,
++  plan        text not null default 'developer',
++  rate_per_min integer not null default 60,
++  enabled     boolean not null default true,
++  created_at  timestamptz not null default now(),
++  last_used_at timestamptz,
++  usage_count bigint not null default 0
++);
++
++create table if not exists metric_definitions (
++  key         text primary key,
++  label       text not null,
++  version     text not null,
++  description text not null,
++  formula     text,
++  created_at  timestamptz not null default now()
++);
++
++-- ---------------------------------------------------------------------------------------------- search vector maintenance
++create or replace function entities_search_update() returns trigger language plpgsql as $$
++begin
++  new.search :=
++    setweight(to_tsvector('simple', coalesce(new.canonical_name, '')), 'A') ||
++    setweight(to_tsvector('simple', coalesce(new.attributes->>'family', '')), 'B') ||
++    setweight(to_tsvector('simple', coalesce(new.entity_type, '')), 'C') ||
++    setweight(to_tsvector('english', left(coalesce(new.description, ''), 4000)), 'C');
++  new.updated_at := now();
++  return new;
++end $$;
++drop trigger if exists entities_search_trg on entities;
++create trigger entities_search_trg before insert or update of canonical_name, description, attributes, entity_type on entities
++  for each row execute function entities_search_update();
++"""
++
++
++def upgrade() -> None:
++    for statement in _split(SQL):
++        op.execute(statement)
++
++
++def downgrade() -> None:
++    raise RuntimeError("AI Atlas migrations are forward-only: historical data is never disposable")
++
++
++def _split(sql: str) -> list[str]:
++    """Split on semicolons outside `$$ … $$` blocks."""
++    out: list[str] = []
++    buf: list[str] = []
++    in_dollar = False
++    for line in sql.splitlines():
++        stripped = line.strip()
++        if stripped.count("$$") % 2 == 1:
++            in_dollar = not in_dollar
++        buf.append(line)
++        if not in_dollar and stripped.endswith(";"):
++            body = [ln for ln in buf if not ln.strip().startswith("--") or in_dollar]
++            stmt = "\n".join(body).strip()
++            if stmt:
++                out.append(stmt)
++            buf = []
++    tail = "\n".join(ln for ln in buf if not ln.strip().startswith("--")).strip()
++    if tail:
++        out.append(tail)
++    return out

added package.json +15 −0

++{
++  "name": "ai-atlas",
++  "version": "0.1.0",
++  "private": true,
++  "description": "AI Atlas — explore the entire AI ecosystem. Web workspace (backend is Python, see pyproject.toml).",
++  "packageManager": "pnpm@11.1.2",
++  "engines": { "node": ">=22" },
++  "scripts": {
++    "dev:web": "pnpm --filter @ai-atlas/web run dev",
++    "build": "pnpm --filter @ai-atlas/web run build",
++    "start:web": "pnpm --filter @ai-atlas/web run start",
++    "typecheck": "pnpm -r run typecheck",
++    "qa": "pnpm --filter @ai-atlas/web run qa"
++  }
++}

added pnpm-workspace.yaml +2 −0

		`@@ -0,0 +1,2 @@`
1	+packages:
2	+ - apps/*

added pyproject.toml +55 −0

++[project]
++name = "aiatlas"
++version = "0.1.0"
++description = "AI Atlas — the global intelligence layer for artificial intelligence: first-party connectors, raw historical archive, temporal knowledge graph, local LLM extraction factory and public API"
++requires-python = ">=3.12"
++dependencies = [
++  "fastapi>=0.115",
++  "uvicorn[standard]>=0.30",
++  "pydantic>=2.8",
++  "pydantic-settings>=2.4",
++  "sqlalchemy[asyncio]>=2.0.35",
++  "asyncpg>=0.30",
++  "alembic>=1.13",
++  "httpx[http2]>=0.27",
++  "orjson>=3.10",
++  "typer>=0.12",
++  "rich>=13",
++  "redis>=5.1",
++  "apscheduler>=3.10,<4",
++  "python-ulid>=2.7",
++  "pyyaml>=6",
++  "python-dateutil>=2.9",
++  "tenacity>=9",
++  "python-slugify>=8",
++  "selectolax>=0.3.21",
++  "feedparser>=6.0.11",
++  "markdown-it-py>=3.0",
++  "pypdf>=5.0",
++  "rapidfuzz>=3.9",
++  "pgvector>=0.3",
++  "numpy>=2",
++]
++
++[project.optional-dependencies]
++dev = ["pytest>=8", "pytest-asyncio>=0.24", "ruff>=0.6", "respx>=0.21"]
++browser = ["playwright>=1.47"]
++
++[project.scripts]
++aia = "aiatlas.cli:app"
++
++[build-system]
++requires = ["hatchling"]
++build-backend = "hatchling.build"
++
++[tool.hatch.build.targets.wheel]
++packages = ["src/aiatlas"]
++
++[tool.ruff]
++line-length = 120
++target-version = "py312"
++
++[tool.pytest.ini_options]
++testpaths = ["tests"]
++asyncio_mode = "auto"
++markers = ["live: hits real external endpoints (skipped unless -m live)"]

added registry/benchmarks.d/.gitkeep +0 −0

added registry/benchmarks.yaml +173 −0

++# Benchmark definitions (entities of type `benchmark`). Results come from connectors; definitions are curated with sources.
++benchmarks:
++  - key: mmlu
++    name: MMLU
++    aliases: [Massive Multitask Language Understanding]
++    category: knowledge
++    task: 57-subject multiple-choice questions
++    metric: accuracy
++    unit: "%"
++    creator: Hendrycks et al.
++    website: https://github.com/hendrycks/test
++    paper: https://arxiv.org/abs/2009.03300
++    known_limitations: Widespread training-data contamination; saturated at the frontier.
++    source_url: https://arxiv.org/abs/2009.03300
++  - key: mmlu-pro
++    name: MMLU-Pro
++    category: knowledge
++    task: harder, 10-option MMLU variant
++    metric: accuracy
++    unit: "%"
++    website: https://github.com/TIGER-AI-Lab/MMLU-Pro
++    paper: https://arxiv.org/abs/2406.01574
++    source_url: https://arxiv.org/abs/2406.01574
++  - key: gpqa
++    name: GPQA
++    aliases: [GPQA Diamond]
++    category: reasoning
++    task: graduate-level science questions
++    metric: accuracy
++    unit: "%"
++    website: https://github.com/idavidrein/gpqa
++    paper: https://arxiv.org/abs/2311.12022
++    source_url: https://arxiv.org/abs/2311.12022
++  - key: humaneval
++    name: HumanEval
++    category: coding
++    task: Python function synthesis from docstrings
++    metric: pass@1
++    unit: "%"
++    website: https://github.com/openai/human-eval
++    paper: https://arxiv.org/abs/2107.03374
++    known_limitations: Saturated; small (164 problems).
++    source_url: https://arxiv.org/abs/2107.03374
++  - key: swe-bench-verified
++    name: SWE-bench Verified
++    aliases: [SWE-bench]
++    category: coding
++    task: resolve real GitHub issues (500 human-validated instances)
++    metric: resolved
++    unit: "%"
++    website: https://www.swebench.com
++    paper: https://arxiv.org/abs/2310.06770
++    known_limitations: Scaffold/agent dependent; results are not comparable across harnesses.
++    source_url: https://www.swebench.com/
++  - key: aider-polyglot
++    name: Aider polyglot
++    aliases: [Aider polyglot coding leaderboard]
++    category: coding
++    task: 225 Exercism exercises in 6 languages, edit-format aware
++    metric: pass rate (2 attempts)
++    unit: "%"
++    website: https://aider.chat/docs/leaderboards/
++    known_limitations: Depends on aider's edit format and prompting; cost column depends on provider pricing.
++    source_url: https://aider.chat/docs/leaderboards/
++  - key: livebench
++    name: LiveBench
++    category: general
++    task: contamination-limited, monthly refreshed questions across 6 categories
++    metric: average score
++    unit: "%"
++    website: https://livebench.ai
++    paper: https://arxiv.org/abs/2406.19314
++    source_url: https://livebench.ai/
++  - key: math-500
++    name: MATH-500
++    aliases: [MATH]
++    category: math
++    task: competition mathematics
++    metric: accuracy
++    unit: "%"
++    paper: https://arxiv.org/abs/2103.03874
++    source_url: https://arxiv.org/abs/2103.03874
++  - key: aime-2025
++    name: AIME 2025
++    aliases: [AIME]
++    category: math
++    task: American Invitational Mathematics Examination problems
++    metric: accuracy
++    unit: "%"
++    known_limitations: 30 problems per year; high variance; often reported with majority voting.
++    source_url: https://maa.org/maa-invitational-competitions/
++  - key: arc-agi
++    name: ARC-AGI
++    aliases: [ARC-AGI-1, ARC-AGI-2, Abstraction and Reasoning Corpus]
++    category: reasoning
++    task: novel visual abstraction puzzles
++    metric: accuracy
++    unit: "%"
++    website: https://arcprize.org
++    source_url: https://arcprize.org/
++  - key: humanitys-last-exam
++    name: Humanity's Last Exam
++    aliases: [HLE]
++    category: knowledge
++    task: expert-written frontier questions
++    metric: accuracy
++    unit: "%"
++    website: https://lastexam.ai
++    paper: https://arxiv.org/abs/2501.14249
++    source_url: https://lastexam.ai/
++  - key: lmarena-text
++    name: LMArena text leaderboard
++    aliases: [Chatbot Arena, LMSYS Chatbot Arena, Arena Elo]
++    category: preference
++    task: crowdsourced pairwise human preference
++    metric: Elo / Bradley–Terry score
++    unit: ""
++    website: https://lmarena.ai
++    known_limitations: Style bias; sampling of prompts by users.
++    source_url: https://lmarena.ai/
++  - key: mmmu
++    name: MMMU
++    category: multimodal
++    task: college-level multimodal understanding
++    metric: accuracy
++    unit: "%"
++    website: https://mmmu-benchmark.github.io
++    paper: https://arxiv.org/abs/2311.16502
++    source_url: https://arxiv.org/abs/2311.16502
++  - key: tau-bench
++    name: τ-bench
++    aliases: [tau-bench, TAU-bench]
++    category: agentic
++    task: tool-agent-user interaction in retail/airline domains
++    metric: pass^1
++    unit: "%"
++    paper: https://arxiv.org/abs/2406.12045
++    source_url: https://arxiv.org/abs/2406.12045
++  - key: terminal-bench
++    name: Terminal-Bench
++    category: agentic
++    task: terminal tasks solved by agents
++    metric: accuracy
++    unit: "%"
++    website: https://www.tbench.ai
++    source_url: https://www.tbench.ai/
++  - key: ifeval
++    name: IFEval
++    category: instruction-following
++    task: verifiable instruction following
++    metric: prompt-level strict accuracy
++    unit: "%"
++    paper: https://arxiv.org/abs/2311.07911
++    source_url: https://arxiv.org/abs/2311.07911
++  - key: mteb
++    name: MTEB
++    aliases: [Massive Text Embedding Benchmark]
++    category: embeddings
++    task: embedding tasks across retrieval, classification, clustering…
++    metric: mean score
++    unit: ""
++    website: https://huggingface.co/spaces/mteb/leaderboard
++    paper: https://arxiv.org/abs/2210.07316
++    source_url: https://arxiv.org/abs/2210.07316
++  - key: artificial-analysis-intelligence-index
++    name: Artificial Analysis Intelligence Index
++    category: composite
++    task: composite of several evaluations run by Artificial Analysis
++    metric: index
++    unit: ""
++    website: https://artificialanalysis.ai
++    known_limitations: Proprietary composite; methodology versions change.
++    source_url: https://artificialanalysis.ai/methodology

added registry/hardware.d/.gitkeep +0 −0

added registry/hardware.yaml +259 −0

++# Curated hardware registry (entities of type `hardware`), used by the hardware compatibility engine.
++# Only manufacturer-published figures; each entry cites the spec page it was read from. Fields not published are omitted.
++# memory_gb may be a list when the product ships with several configurations.
++hardware:
++  # ---------------------------------------------------------------- Apple Silicon (unified memory)
++  - key: apple-m1
++    name: Apple M1
++    manufacturer: apple
++    kind: soc
++    architecture: Apple Silicon (Firestorm/Icestorm)
++    release_date: "2020-11"
++    memory_gb: [8, 16]
++    memory_bandwidth_gbs: 68.25
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/111883
++  - key: apple-m1-pro
++    name: Apple M1 Pro
++    manufacturer: apple
++    kind: soc
++    release_date: "2021-10"
++    memory_gb: [16, 32]
++    memory_bandwidth_gbs: 200
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/111902
++  - key: apple-m1-max
++    name: Apple M1 Max
++    manufacturer: apple
++    kind: soc
++    release_date: "2021-10"
++    memory_gb: [32, 64]
++    memory_bandwidth_gbs: 400
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/111902
++  - key: apple-m1-ultra
++    name: Apple M1 Ultra
++    manufacturer: apple
++    kind: soc
++    release_date: "2022-03"
++    memory_gb: [64, 128]
++    memory_bandwidth_gbs: 800
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/111900
++  - key: apple-m2
++    name: Apple M2
++    manufacturer: apple
++    kind: soc
++    release_date: "2022-06"
++    memory_gb: [8, 16, 24]
++    memory_bandwidth_gbs: 100
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/111837
++  - key: apple-m2-pro
++    name: Apple M2 Pro
++    manufacturer: apple
++    kind: soc
++    release_date: "2023-01"
++    memory_gb: [16, 32]
++    memory_bandwidth_gbs: 200
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/111837
++  - key: apple-m2-max
++    name: Apple M2 Max
++    manufacturer: apple
++    kind: soc
++    release_date: "2023-01"
++    memory_gb: [32, 64, 96]
++    memory_bandwidth_gbs: 400
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/111835
++  - key: apple-m2-ultra
++    name: Apple M2 Ultra
++    manufacturer: apple
++    kind: soc
++    release_date: "2023-06"
++    memory_gb: [64, 128, 192]
++    memory_bandwidth_gbs: 800
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/111835
++  - key: apple-m3
++    name: Apple M3
++    manufacturer: apple
++    kind: soc
++    release_date: "2023-10"
++    memory_gb: [8, 16, 24]
++    memory_bandwidth_gbs: 100
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/117735
++  - key: apple-m3-pro
++    name: Apple M3 Pro
++    manufacturer: apple
++    kind: soc
++    release_date: "2023-10"
++    memory_gb: [18, 36]
++    memory_bandwidth_gbs: 150
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/117736
++  - key: apple-m3-max
++    name: Apple M3 Max
++    manufacturer: apple
++    kind: soc
++    release_date: "2023-10"
++    memory_gb: [36, 48, 64, 96, 128]
++    memory_bandwidth_gbs: 400
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://support.apple.com/en-us/117736
++  - key: apple-m3-ultra
++    name: Apple M3 Ultra
++    manufacturer: apple
++    kind: soc
++    release_date: "2025-03"
++    memory_gb: [96, 256, 512]
++    memory_bandwidth_gbs: 819
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://www.apple.com/mac-studio/specs/
++  - key: apple-m4
++    name: Apple M4
++    manufacturer: apple
++    kind: soc
++    release_date: "2024-05"
++    memory_gb: [16, 24, 32]
++    memory_bandwidth_gbs: 120
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://www.apple.com/mac-mini/specs/
++  - key: apple-m4-pro
++    name: Apple M4 Pro
++    manufacturer: apple
++    kind: soc
++    release_date: "2024-10"
++    memory_gb: [24, 48, 64]
++    memory_bandwidth_gbs: 273
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://www.apple.com/mac-mini/specs/
++  - key: apple-m4-max
++    name: Apple M4 Max
++    manufacturer: apple
++    kind: soc
++    release_date: "2024-10"
++    memory_gb: [36, 48, 64, 128]
++    memory_bandwidth_gbs: 546
++    runtimes: [mlx, llama.cpp, coreml]
++    source_url: https://www.apple.com/mac-studio/specs/
++  # ---------------------------------------------------------------- NVIDIA data center
++  - key: nvidia-a100-80gb
++    name: NVIDIA A100 80GB
++    manufacturer: nvidia
++    kind: gpu
++    architecture: Ampere
++    release_date: "2020-11"
++    memory_gb: 80
++    memory_type: HBM2e
++    memory_bandwidth_gbs: 2039
++    tdp_watts: 400
++    runtimes: [cuda, tensorrt-llm, vllm, sglang, llama.cpp]
++    source_url: https://www.nvidia.com/en-us/data-center/a100/
++  - key: nvidia-h100-sxm
++    name: NVIDIA H100 SXM
++    manufacturer: nvidia
++    kind: gpu
++    architecture: Hopper
++    release_date: "2022-09"
++    memory_gb: 80
++    memory_type: HBM3
++    memory_bandwidth_gbs: 3350
++    tdp_watts: 700
++    runtimes: [cuda, tensorrt-llm, vllm, sglang, llama.cpp]
++    source_url: https://www.nvidia.com/en-us/data-center/h100/
++  - key: nvidia-h200
++    name: NVIDIA H200
++    manufacturer: nvidia
++    kind: gpu
++    architecture: Hopper
++    release_date: "2024"
++    memory_gb: 141
++    memory_type: HBM3e
++    memory_bandwidth_gbs: 4800
++    tdp_watts: 700
++    runtimes: [cuda, tensorrt-llm, vllm, sglang, llama.cpp]
++    source_url: https://www.nvidia.com/en-us/data-center/h200/
++  - key: nvidia-b200
++    name: NVIDIA B200
++    manufacturer: nvidia
++    kind: gpu
++    architecture: Blackwell
++    release_date: "2024-03"
++    memory_gb: 180
++    memory_type: HBM3e
++    runtimes: [cuda, tensorrt-llm, vllm, sglang]
++    source_url: https://www.nvidia.com/en-us/data-center/dgx-b200/
++  - key: nvidia-rtx-4090
++    name: NVIDIA GeForce RTX 4090
++    manufacturer: nvidia
++    kind: gpu
++    architecture: Ada Lovelace
++    release_date: "2022-10"
++    memory_gb: 24
++    memory_type: GDDR6X
++    memory_bandwidth_gbs: 1008
++    tdp_watts: 450
++    runtimes: [cuda, vllm, llama.cpp, exllama]
++    source_url: https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/
++  - key: nvidia-rtx-5090
++    name: NVIDIA GeForce RTX 5090
++    manufacturer: nvidia
++    kind: gpu
++    architecture: Blackwell
++    release_date: "2025-01"
++    memory_gb: 32
++    memory_type: GDDR7
++    memory_bandwidth_gbs: 1792
++    tdp_watts: 575
++    runtimes: [cuda, vllm, llama.cpp, exllama]
++    source_url: https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
++  - key: nvidia-rtx-3090
++    name: NVIDIA GeForce RTX 3090
++    manufacturer: nvidia
++    kind: gpu
++    architecture: Ampere
++    release_date: "2020-09"
++    memory_gb: 24
++    memory_type: GDDR6X
++    memory_bandwidth_gbs: 936
++    tdp_watts: 350
++    runtimes: [cuda, vllm, llama.cpp, exllama]
++    source_url: https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/
++  - key: nvidia-dgx-spark
++    name: NVIDIA DGX Spark
++    manufacturer: nvidia
++    kind: system
++    architecture: Grace Blackwell (GB10)
++    release_date: "2025"
++    memory_gb: 128
++    memory_type: LPDDR5x
++    memory_bandwidth_gbs: 273
++    runtimes: [cuda, tensorrt-llm, vllm, llama.cpp]
++    source_url: https://www.nvidia.com/en-us/products/workstations/dgx-spark/
++  # ---------------------------------------------------------------- AMD
++  - key: amd-instinct-mi300x
++    name: AMD Instinct MI300X
++    manufacturer: amd
++    kind: gpu
++    architecture: CDNA 3
++    release_date: "2023-12"
++    memory_gb: 192
++    memory_type: HBM3
++    memory_bandwidth_gbs: 5300
++    tdp_watts: 750
++    runtimes: [rocm, vllm, sglang, llama.cpp]
++    source_url: https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html
++  - key: amd-instinct-mi325x
++    name: AMD Instinct MI325X
++    manufacturer: amd
++    kind: gpu
++    architecture: CDNA 3
++    release_date: "2024-10"
++    memory_gb: 256
++    memory_type: HBM3e
++    memory_bandwidth_gbs: 6000
++    runtimes: [rocm, vllm, sglang]
++    source_url: https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html

added registry/organizations.d/.gitkeep +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added registry/organizations.yaml +619 −0

Large diff (619 lines) is not loaded by default to keep the page fast.Load diff

added registry/providers.d/.gitkeep +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added registry/providers.yaml +176 −0

Large diff (176 lines) is not loaded by default to keep the page fast.Load diff

added registry/sources.d/.gitkeep +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added registry/sources.yaml +387 −0

Large diff (387 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/__init__.py +3 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/cli.py +295 −0

Large diff (295 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/config.py +94 −0

Large diff (94 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/connectors/__init__.py +46 −0

Large diff (46 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/connectors/labs/__init__.py +2 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/connectors/labs/_common.py +153 −0

Large diff (153 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/connectors/labs/anthropic.py +312 −0

Large diff (312 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/db/__init__.py +75 −0

Large diff (75 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/ids.py +93 −0

Large diff (93 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/logging.py +56 −0

Large diff (56 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/registry/__init__.py +103 −0

Large diff (103 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/registry/seed.py +168 −0

Large diff (168 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/schemas/__init__.py +17 −0

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/schemas/extraction.py +174 −0

Large diff (174 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/__init__.py +7 −0

Large diff (7 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/archive.py +74 −0

Large diff (74 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/connector.py +483 −0

Large diff (483 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/__init__.py +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/dates.py +71 −0

Large diff (71 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/feeds.py +76 −0

Large diff (76 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/html.py +236 −0

Large diff (236 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/markdown.py +103 −0

Large diff (103 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/numbers.py +106 −0

Large diff (106 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/sitemap.py +42 −0

Large diff (42 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/facts.py +222 −0

Large diff (222 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/fetch.py +329 −0

Large diff (329 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/resolution.py +169 −0

Large diff (169 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/writer.py +357 −0

Large diff (357 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/__init__.py +1 −0

Large diff (1 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/backup.py +39 −0

Large diff (39 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/cache.py +99 −0

Large diff (99 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/embeddings.py +83 −0

Large diff (83 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/handlers.py +228 −0

Large diff (228 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/jobs.py +124 −0

Large diff (124 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/llm/__init__.py +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/llm/gateway.py +231 −0

Large diff (231 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/quality.py +72 −0

Large diff (72 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/scheduler.py +119 −0

Large diff (119 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/search.py +156 −0

Large diff (156 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/stats.py +53 −0

Large diff (53 lines) is not loaded by default to keep the page fast.Load diff

added tests/__init__.py +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added tests/conftest.py +47 −0

Large diff (47 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/anthropic/model-deprecations.md +223 −0

Large diff (223 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/anthropic/models-overview.md +110 −0

Large diff (110 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/anthropic/news.html +1 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/anthropic/pricing.md +526 −0

Large diff (526 lines) is not loaded by default to keep the page fast.Load diff

added tests/test_anthropic.py +63 −0

Large diff (63 lines) is not loaded by default to keep the page fast.Load diff

added tests/test_extract.py +76 −0

Large diff (77 lines) is not loaded by default to keep the page fast.Load diff

spb/ai-atlas

AI Atlas foundation: schema, connector SDK, archive, temporal writer, LLM gateway, scheduler, registry, CLI, Anthropic connector

Theme