spb/ai-atlas

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

added .env.example +25 −0

		@@ -0,0 +1,25 @@
	1	+# AI Atlas — local development. Copy to .env (never commit .env).
	2	+APP_ENV=development
	3	+AIA_SITE_URL=http://localhost:8320
	4	+DATABASE_URL=postgresql+asyncpg://aiatlas:aiatlas@127.0.0.1:5432/aiatlas
	5	+REDIS_URL=redis://127.0.0.1:6379/5
	6	+AIA_DATA_DIR=./data
	7	+AIA_API_HOST=127.0.0.1
	8	+AIA_API_PORT=8321
	9	+AIA_ADMIN_TOKEN=dev-admin-token
	10	+AIA_LOG_JSON=0
	11	+# Crawler identity (robots.txt is honoured; contact address is public)
	12	+AIA_USER_AGENT="AIAtlasBot/0.1 (+https://www.ai-atlas.co/bot; contact@spboucher.ai)"
	13	+# Local LLM factory (OpenAI-compatible; MacLustr llm-api.io). Optional — deterministic extraction works without it.
	14	+AIA_LLM_BASE_URL=https://www.llm-api.io/v1
	15	+AIA_LLM_API_KEY=
	16	+AIA_LLM_SMALL_MODEL=qwen3-4b-instruct-2507-4bit
	17	+AIA_LLM_MEDIUM_MODEL=qwen3.6-35b-a3b-4bit
	18	+AIA_LLM_LARGE_MODEL=qwen3.8-27b-4bit
	19	+AIA_EMBEDDING_MODEL=qwen3-embedding-0.6b-4bit
	20	+# Optional escalation transports (never required)
	21	+SCRAPFLY_API_KEY=
	22	+FIRECRAWL_API_KEY=
	23	+# Web
	24	+API_URL=http://127.0.0.1:8321
	25	+NEXT_PUBLIC_SITE_URL=http://localhost:8320

added .gitignore +22 −0

		@@ -0,0 +1,22 @@
	1	+.env
	2	+.env.*
	3	+!.env.example
	4	+.venv/
	5	+__pycache__/
	6	+*.pyc
	7	+/data/
	8	+node_modules/
	9	+.next/
	10	+tmp/
	11	+.DS_Store
	12	+deploy/.admin-token
	13	+deploy/.llm-key
	14	+deploy/rendered/
	15	+*.tsbuildinfo
	16	+apps/web/next-env.d.ts
	17	+apps/web/qa/screens/
	18	+apps/web/AGENTS.md
	19	+apps/web/CLAUDE.md
	20	+.pytest_cache/
	21	+.ruff_cache/
	22	+*.egg-info/

added CLAUDE.md +52 −0

		@@ -0,0 +1,52 @@
	1	+# AI Atlas — project guide (condensed from the founding spec; the full architecture is in docs/ARCHITECTURE.md)
	2	+
	3	+**Mission**: the most comprehensive, structured, searchable and continuously updated map of the global AI ecosystem — Bloomberg × Wikipedia ×
	4	+Crunchbase × Hugging Face × Papers With Code, purpose-built for AI. Not a tools directory, not a news aggregator, not an API wrapper.
	5	+**The dataset is the product; the website and the API are interfaces to it.**
	6	+
	7	+## Non-negotiables
	8	+
	9	+1. No critical dependence on external data APIs — direct crawling first; Scrapfly/Firecrawl/browser are optional escalation only.
	10	+2. Store provenance (source, snapshot, URL, tier, confidence, extractor) on every fact; store history (temporal claims, append-only prices/results, raw snapshots).
	11	+3. Local LLM (MacLustr llm-api.io via the gateway) for large-scale extraction; deterministic extraction always runs first.
	12	+4. **Never fabricate data.** Missing → "Unavailable". Conflicts → both claims stored and flagged; never a compromise value.
	13	+5. Primary sources first (tier 1 official > 2 quality secondary > 3 community > 4 unverified). Connectors are owned IP with fixture tests.
	14	+6. Live counters, stats, feeds and timelines come from the database — never hardcoded.
	15	+7. Public information stays public (SEO); login only for watchlists/alerts/API keys later.
	16	+8. Respectful crawling: robots.txt, per-domain rate limits, conditional requests, identified UA, no bypassing access controls, no private data.
	17	+
	18	+## Stack
	19	+
	20	+- Backend `src/aiatlas/` — Python 3.12, FastAPI, SQLAlchemy Core + asyncpg, Alembic (SQL, forward-only), Redis, APScheduler, selectolax, feedparser, pydantic.
	21	+ CLI `aia` (`.venv/bin/aia --help`). Connector SDK in `sdk/` (docs/CONNECTORS.md). API contract in docs/API.md.
	22	+- Database: Postgres 17 + pgvector + pg_trgm. Internal ids = prefixed ULIDs (`model_…`); slugs stable and unique.
	23	+- Frontend `apps/web/` — Next.js 16 (App Router, React 19, TypeScript strict, Tailwind v4). Guide: docs/FRONTEND.md. Dev on :8320, API on :8321.
	24	+- Data outside the repo: `AIA_DATA_DIR` (`./data` in dev, `~/ai-atlas-data` in prod): raw/, text/, backups/, logs/, cache/, seed/.
	25	+
	26	+## Local development
	27	+
	28	+```bash
	29	+uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev]"
	30	+cp .env.example .env # local Postgres `aiatlas` (role aiatlas/aiatlas), Redis db 5
	31	+.venv/bin/aia migrate && .venv/bin/aia seed
	32	+.venv/bin/aia run anthropic # one connector, live · `--file key=path` uses a fixture · `aia crawl --priority 0` runs all P0 connectors
	33	+.venv/bin/aia api --reload # http://127.0.0.1:8321/api/v1/docs
	34	+pnpm install && pnpm dev:web # http://localhost:8320
	35	+.venv/bin/pytest -q # fixtures only; `-m live` hits the network
	36	+```
	37	+
	38	+## Production (MacLustr)
	39	+
	40	+Node **M2M32c** (dedicated), deployed by `mld` — manifest `deploy/ai-atlas.mld.json` → `M1M32:~/dispatch/apps/ai-atlas.json`. PM2 processes
	41	+`ai-atlas-api` (uvicorn 127.0.0.1:8321), `ai-atlas-scheduler` (`aia schedule`), `ai-atlas-web` (next start :8320). Public route through the
	42	+MacLustr Tunnel: `https://www.ai-atlas.co → M2M32c:8320`. See docs/DEPLOY.md. Secrets live only in the rendered manifest (`deploy/.admin-token`,
	43	+`deploy/.llm-key`, git-ignored). LLM factory: `AIA_LLM_BASE_URL=https://www.llm-api.io/v1` (private MacLustr server on M1M64).
	44	+
	45	+## Conventions
	46	+
	47	+- Properties and relations use the shared vocabulary in docs/CONNECTORS.md; metrics use the `metric.` prefix (no events).
	48	+- Every connector: `source_key` registered in `registry/sources.yaml`, organization in `registry/organizations.yaml`, fixture test in `tests/`.
	49	+- Parser improved? bump `parser_version` and `aia reprocess <connector>` — never re-crawl for a code change.
	50	+- Frontend: mobile-first (390/430/768/1440), DOM order = visual order, ≥ 44 px targets, no horizontal overflow, dark + light, premium
	51	+ institutional design (no generic SaaS cards). Every page: real data, loading + error + empty states, source attribution, `generateMetadata`.
	52	+- Units: tokens, USD per 1M tokens, GB, GB/s, W, ISO dates (UTC). Openness vocabulary: open-weights | open-source | proprietary | restricted.

added README.md +39 −0

		@@ -0,0 +1,39 @@
	1	+# AI Atlas
	2	+
	3	+**www.ai-atlas.co — Explore the entire AI ecosystem.**
	4	+
	5	+AI Atlas is a continuously updated, historically versioned, machine-readable map of the global artificial intelligence ecosystem: models,
	6	+companies, research, providers and prices, benchmarks, hardware, frameworks, datasets, tools — connected in one temporal knowledge graph,
	7	+with provenance on every fact.
	8	+
	9	+- **First-party connectors** read official pages, docs, feeds, model cards and public files directly. No commercial data API is required.
	10	+- **Raw historical archive**: every changed page is stored (content-addressed, compressed) and can be reprocessed when parsers improve.
	11	+- **Temporal knowledge graph**: claims carry `valid_from`/`valid_to`, prices and benchmark results are append-only, every change is an event.
	12	+- **Local LLM factory**: large-scale extraction runs on MacLustr's own inference servers through an engine-agnostic gateway.
	13	+- **Interfaces**: the website, the public API (`/api/v1`), and — later — datasets and alerts.
	14	+
	15	+## Layout
	16	+
	17	+```
	18	+registry/ curated YAML (sources, organizations, providers, benchmarks, hardware) — every entry cites its source
	19	+src/aiatlas/ Python platform: sdk/ (connector SDK), connectors/, services/ (jobs, scheduler, LLM gateway, search, quality), api/, cli.py
	20	+migrations/ forward-only SQL migrations
	21	+apps/web/ Next.js 16 website
	22	+tests/ pytest with saved fixtures
	23	+deploy/ MacLustr (mld) manifest and scripts
	24	+docs/ ARCHITECTURE.md · CONNECTORS.md · API.md · FRONTEND.md · DEPLOY.md
	25	+```
	26	+
	27	+## Quick start
	28	+
	29	+```bash
	30	+uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev]"
	31	+cp .env.example .env && createdb -O aiatlas aiatlas
	32	+.venv/bin/aia migrate && .venv/bin/aia seed
	33	+.venv/bin/aia crawl --priority 0 # initial corpus: labs, Hugging Face, arXiv, providers, benchmarks…
	34	+.venv/bin/aia api # http://127.0.0.1:8321/api/v1/docs
	35	+pnpm install && pnpm dev:web # http://localhost:8320
	36	+```
	37	+
	38	+© 2026 Simon-Pierre Boucher · contact@spboucher.ai · hosted on [MacLustr](https://www.maclustr.io). AI Atlas links to original sources and
	39	+never republishes documents in full; extracted facts are provided as-is with their provenance.

added alembic.ini +31 −0

		@@ -0,0 +1,31 @@
	1	+[alembic]
	2	+script_location = migrations
	3	+prepend_sys_path = src
	4	+path_separator = os
	5	+
	6	+[loggers]
	7	+keys = root,alembic
	8	+
	9	+[handlers]
	10	+keys = console
	11	+
	12	+[formatters]
	13	+keys = generic
	14	+
	15	+[logger_root]
	16	+level = WARN
	17	+handlers = console
	18	+
	19	+[logger_alembic]
	20	+level = INFO
	21	+handlers =
	22	+qualname = alembic
	23	+
	24	+[handler_console]
	25	+class = StreamHandler
	26	+args = (sys.stderr,)
	27	+level = NOTSET
	28	+formatter = generic
	29	+
	30	+[formatter_generic]
	31	+format = %(levelname)-5.5s [%(name)s] %(message)s

added docs/API.md +119 −0

		@@ -0,0 +1,119 @@
	1	+# AI Atlas API — contract (v1)
	2	+
	3	+Base: `/api/v1` (FastAPI, `src/aiatlas/api`). JSON, UTF-8, ISO-8601 UTC timestamps, numbers as numbers (Postgres aggregates may arrive as
	4	+strings — the web layer normalises). Public routes are cached in Redis (`aia:api:*`, 60–600 s) and rate-limited per IP on search.
	5	+Admin routes require header `x-aia-admin-token: <AIA_ADMIN_TOKEN>`. OpenAPI at `/api/v1/docs`. Errors: `{"detail": "..."}` with 400/404/429/503.
	6	+Never expose credentials, internal hostnames, raw archive content or `raw_path`s.
	7	+
	8	+## Shared shapes
	9	+
	10	+```ts
	11	+type Num = number | string | null
	12	+type Org = { id: string; slug: string; name: string } | null
	13	+type EntitySummary = { id: string; entity_type: string; slug: string; name: string; description: string | null; status: string;
	14	+ organization: Org; attributes: Record<string, unknown>; quality: { score?: number; completeness?: number; primary_source_ratio?: number;
	15	+ freshness?: number; source_count?: number; conflicts?: number }; counts: { relations?: number; events?: number; claims?: number };
	16	+ first_seen_at: string; last_seen_at: string; updated_at: string }
	17	+type Page<T> = { items: T[]; total: number; limit: number; offset: number }
	18	+type Provenance = Record<string, { source_id: string | null; source_name?: string; url: string | null; observed_at: string; tier: number;
	19	+ confidence: string; extractor: string; unit?: string }>
	20	+type ChangeEvent = { id: string; event_type: string; category: string; property: string | null; old_value: unknown; new_value: unknown;
	21	+ summary: string; importance: 0 | 1 | 2 | 3; observed_at: string; effective_at: string | null; source_url: string | null; connector_name: string | null;
	22	+ entity: EntitySummary | null; meta: Record<string, unknown> }
	23	+type Price = { id: string; model: EntitySummary; provider: EntitySummary; provider_model_id: string | null; input_per_mtok: Num; output_per_mtok: Num;
	24	+ cached_input_per_mtok: Num; cache_write_per_mtok: Num; batch_input_per_mtok: Num; batch_output_per_mtok: Num; per_image: Num; currency: string;
	25	+ context_length: Num; max_output_tokens: Num; features: Record<string, unknown>; observed_at: string; valid_from: string; valid_to: string | null; source_url: string | null; tier: number }
	26	+type BenchmarkResult = { id: string; model: EntitySummary; benchmark: EntitySummary; score: number; metric: string | null; unit: string | null;
	27	+ higher_is_better: boolean; config: Record<string, unknown>; evaluated_at: string | null; observed_at: string; source_url: string | null; tier: number; confidence: string }
	28	+type Claim = { id: string; property: string; value: unknown; unit: string | null; tier: number; confidence: string; status: string; extractor: string;
	29	+ observed_at: string; effective_at: string | null; valid_from: string; valid_to: string | null; source_url: string | null; source_name: string | null }
	30	+type SourceRef = { source_id: string | null; source_name: string | null; domain: string | null; tier: number | null; url: string; doc_type: string; last_observed_at: string | null; snapshots: number }
	31	+```
	32	+
	33	+## Public routes
	34	+
	35	+| route | returns |
	36	+|---|---|
	37	+| `GET /health` | `{ status: "ok"|"degraded", version, db: bool, redis: bool, llm: { available, reachable? }, time }` |
	38	+| `GET /stats` | `{ entities: Record<type, number>, entities_total, sources, connectors, connectors_enabled, documents, snapshots, claims, claims_current, relations, change_events, change_events_24h, change_events_7d, benchmark_results, prices_current, prices_total, review_pending, llm_jobs, llm_tokens, last_snapshot_at, last_event_at, first_entity_at, archive: { raw_bytes, raw_files, text_bytes, text_files }, computed_at }` — **always live from the DB** |
	39	+| `GET /stats/history?days=90` | `{ items: { day: string; counts: object }[] }` |
	40	+| `GET /search?q=&type=&limit=30&offset=0` | `{ query: CompiledQuery, items: (EntitySummary & { rank: number })[], total: number }` — uses `services.search.compile_query` + `search_entities` (+ embedding when the gateway is up) |
	41	+| `GET /search/suggest?q=` | `{ items: { id, entity_type, slug, name, organization_name }[] }` (≤ 8, prefix, fast) |
	42	+| `GET /entities/{slug_or_id}` | `EntityDetail` (below). Also mounted as `/models/{slug}`, `/companies/{slug}`, `/papers/{slug}`, `/providers/{slug}`, `/benchmarks/{slug}`, `/hardware/{slug}`, `/frameworks/{slug}`, `/datasets/{slug}`, `/tools/{slug}` (404 if the type doesn't match) |
	43	+| `GET /entities/{slug}/timeline?limit=50&before=` | `{ items: ChangeEvent[] }` (entity's own events + events of entities it develops for companies) |
	44	+| `GET /entities/{slug}/history?property=` | `{ items: Claim[] }` — full claim history (all statuses), newest first |
	45	+| `GET /entities/{slug}/asof?date=YYYY-MM-DD` | `{ existed: bool, first_seen_at, date, attributes: Record<string, unknown>, claims: Claim[] }` — state as known at that date |
	46	+| `GET /entities/{slug}/graph?depth=1&limit=80` | `{ nodes: { id, slug, name, entity_type, organization_name? }[], edges: { source, target, predicate }[] }` (depth ≤ 2, cap nodes) |
	47	+| `GET /entities/{slug}/sources` | `{ items: SourceRef[] }` |
	48	+| `GET /entities/{slug}/related?limit=` | `{ items: EntitySummary[] }` (same org / same family / shared relations) |
	49	+| `GET /models?q=&org=&family=&openness=&modality=&status=&min_params=&max_params=&min_context=&year_from=&year_to=&license=&sort=(updated|name|params|context|release|quality|downloads)&order=&limit=&offset=` | `Page<EntitySummary>` + `facets` when `facets=1`: `{ organizations: {slug,name,count}[], openness: {value,count}[], modalities, families, years, licenses, status }` |
	50	+| `GET /companies?q=&country=&kind=&sort=(models|name|updated|quality)&limit=&offset=` | `Page<EntitySummary & { model_count: number; paper_count: number }>` (+`facets`: countries, kinds) |
	51	+| `GET /papers?q=&category=&org=&since=&until=&sort=(published|updated)&limit=&offset=` | `Page<EntitySummary>` |
	52	+| `GET /providers` | `{ items: (EntitySummary & { model_count: number; price_count: number; min_input_per_mtok: Num; min_output_per_mtok: Num })[] }` |
	53	+| `GET /prices?model=&provider=&sort=(input|output|model|provider|observed)&limit=&offset=&current=1` | `Page<Price>` |
	54	+| `GET /prices/history?model=&provider=` | `{ items: Price[] }` (all rows incl. closed, oldest first) |
	55	+| `GET /prices/index?days=180` | `{ series: { day: string; median_input: Num; median_output: Num; min_input: Num; models: number }[]; movers: ChangeEvent[] }` computed from `prices` history |
	56	+| `GET /benchmarks` | `{ items: (EntitySummary & { result_count: number; model_count: number; top: { model: EntitySummary; score: number } | null })[] }` |
	57	+| `GET /benchmarks/{slug}/results?limit=&offset=&config=` | `Page<BenchmarkResult>` sorted by score (respecting `higher_is_better`), current rows only unless `history=1` |
	58	+| `GET /benchmarks/{slug}/history?model=` | `{ items: BenchmarkResult[] }` over time |
	59	+| `GET /hardware?kind=&manufacturer=&min_memory=&sort=` | `Page<EntitySummary>` |
	60	+| `GET /hardware/fit?memory_gb=&quant=(4bit|8bit|fp16)&context=8192&limit=` | `{ inputs, assumptions: string[], items: { model: EntitySummary; parameter_count: number; estimated_memory_gb: number; fits: boolean; headroom_gb: number; quantization: string; note: string }[] }` — **ESTIMATED**: bytes/param (4bit 0.5+overhead 1.15, 8bit 1.0, fp16 2.0) + KV cache estimate; label as estimated |
	61	+| `GET /explore/types` | `{ items: { entity_type, count, label }[] }` |
	62	+| `GET /explore/{type}?q=&org=&sort=&limit=&offset=` | `Page<EntitySummary>` generic listing for any type (datasets, frameworks, tools, repositories, regulation…) |
	63	+| `GET /changes?category=&type=&entity_type=&importance_min=&since=&until=&q=&limit=50&before=<observed_at cursor>` | `Page<ChangeEvent>` newest first; `type` = comma list of event types; excludes `DOCUMENT_CHANGED` unless `include_documents=1` |
	64	+| `GET /changes/daily?date=YYYY-MM-DD` | `{ date, counts: Record<category, number>, sections: { category, label, items: ChangeEvent[] }[], new_models: EntitySummary[] }` ("What changed in AI today", DB-generated only) |
	65	+| `GET /changes/categories?days=7` | `{ items: { category, event_type, count }[] }` |
	66	+| `GET /timeline?entity=&year=&category=&limit=` | `{ items: { month: string; events: ChangeEvent[] }[] }` grouped by month; global when no entity |
	67	+| `GET /compare?ids=slug,slug,…` (2–6) | `{ entity_type, dimensions: { key, label, unit?, kind: "number"|"text"|"list"|"bool"|"date" }[], items: { entity: EntitySummary; values: Record<key, unknown>; provenance: Provenance; prices?: Price[]; results?: BenchmarkResult[] }[] }` — dimensions per type: model (params, active params, context, max output, openness, license, modalities, release date, knowledge cutoff, best price in/out, benchmark scores shared by all), provider (models, min prices, features), hardware (memory, bandwidth, tdp, runtimes), framework (version, license, stars), company (country, founded, models, papers) |
	68	+| `GET /diff?a=YYYY-MM-DD&b=YYYY-MM-DD&scope=(all|models|org:<slug>|family:<name>)` | `{ a, b, new_entities: EntitySummary[], gone_entities: EntitySummary[], property_changes: ChangeEvent[], price_changes: ChangeEvent[], benchmark_changes: ChangeEvent[], counts }` |
	69	+| `GET /sources` | `{ items: { key, name, domain, tier, kind, category, organization: Org, enabled, documents, last_crawled_at, connectors: { name, label, health, last_success_at, interval_seconds }[] }[] }` (public transparency page) |
	70	+| `GET /methodology` | `{ metrics: metric_definitions[], confidence_levels, tiers, event_types, extractors }` |
	71	+| `GET /trending?days=7&limit=12` | `{ items: (EntitySummary & { views: number })[] }` from `page_views` |
	72	+| `POST /views` `{ path }` | `{ ok: true }` (beacon; 1 req/s/IP) |
	73	+| `GET /sitemap?type=&limit=5000&offset=0` | `{ items: { slug, entity_type, updated_at }[], total }` |
	74	+| `GET /api-keys/me` (header `x-api-key`) | `{ label, plan, rate_per_min, usage_count }` — developer keys (later); public routes work without a key |
	75	+
	76	+### `EntityDetail`
	77	+
	78	+```ts
	79	+EntitySummary & {
	80	+ attributes: Record<string, unknown> // full
	81	+ provenance: Provenance
	82	+ aliases: string[]; identifiers: { scheme: string; value: string }[]
	83	+ relations: { predicate: string; direction: "out" | "in"; items: EntitySummary[]; total: number }[] // grouped, ≤ 24 per group
	84	+ sources: SourceRef[] // documents describing this entity (deduped by URL)
	85	+ timeline: ChangeEvent[] // latest 30
	86	+ quality: {...}; counts: {...}
	87	+ // type-specific blocks (present only when relevant)
	88	+ prices?: Price[] // current rows across providers (model) or for this provider (provider)
	89	+ price_history?: Price[] // closed + current, oldest first (model)
	90	+ results?: BenchmarkResult[] // model: its results · benchmark: leaderboard top 100
	91	+ lineage?: { ancestors: EntitySummary[]; descendants: EntitySummary[]; quantizations: EntitySummary[] } // model
	92	+ providers?: EntitySummary[] // model
	93	+ hardware_fit?: { hardware: EntitySummary; quantization: string; estimated_memory_gb: number; fits: boolean }[] // model, ESTIMATED
	94	+ models?: Page<EntitySummary> // company / provider / hardware(runnable)
	95	+ papers?: EntitySummary[] // company / model
	96	+ repositories?: EntitySummary[] // company / model / framework
	97	+}
	98	+```
	99	+
	100	+## Admin routes (`x-aia-admin-token`)
	101	+
	102	+| route | returns / action |
	103	+|---|---|
	104	+| `GET /admin/overview` | `{ stats, queue: Record<kind, Record<status, n>>, heartbeats, connectors: { ok, degraded, failing, disabled }, review_pending, recent_errors: n, llm: { jobs_24h, tokens_24h, by_stage } , archive }` |
	105	+| `GET /admin/connectors` | full `connectors` rows + last run + docs/snapshots counts + source tier |
	106	+| `POST /admin/connectors/{name}/run` `{ force?: bool }` | enqueue an immediate run (Redis flag `aia:run-now:<name>` consumed by the scheduler tick) → `{ queued: true }` |
	107	+| `PATCH /admin/connectors/{name}` `{ enabled?, interval_seconds?, priority? }` | update |
	108	+| `GET /admin/runs?connector=&limit=` · `GET /admin/errors?connector=&limit=` | rows |
	109	+| `GET /admin/documents?connector=&status=&q=&limit=&offset=` · `GET /admin/documents/{id}` | documents (+ snapshots list); `GET /admin/snapshots/{id}` → metadata + `structured` + `diff` + first 20 kB of cleaned text (never raw HTML) |
	110	+| `GET /admin/jobs?status=&kind=` · `POST /admin/jobs/{id}/retry` · `POST /admin/jobs/requeue-dead` | queue |
	111	+| `GET /admin/llm-jobs?limit=` · `GET /admin/llm/health` | LLM factory accounting |
	112	+| `GET /admin/review?status=pending&kind=` · `POST /admin/review/{id}` `{ action: "approve"|"reject"|"edit", resolution?: object }` | review queue; approving a `merge_candidate` calls `merge_entities(source, target)` (aliases/identifiers/claims/relations/events moved, `merged_into` set) |
	113	+| `GET /admin/entities/duplicates?type=&limit=` | candidate pairs by normalized name similarity (`pg_trgm`) within a type |
	114	+| `POST /admin/entities/merge` `{ source_id, target_id }` · `POST /admin/entities/{id}/claims/{claim_id}/retract` | curation |
	115	+| `POST /admin/reprocess` `{ connector, url? }` | enqueue `reprocess_snapshot` |
	116	+| `POST /admin/llm/enqueue` `{ limit?, task? }` | queue `llm_extract` for pending snapshots |
	117	+| `GET /admin/infrastructure` | heartbeats, archive size, DB size (`pg_database_size`), table sizes, node hostname, uptime |
	118	+| `POST /admin/cache/flush` | `{ flushed: n }` |
	119	+| `POST /admin/stats/recompute` · `POST /admin/quality/recompute` | maintenance |

added docs/ARCHITECTURE.md +142 −0

		@@ -0,0 +1,142 @@
	1	+# AI Atlas — Technical architecture
	2	+
	3	+> The global intelligence layer for artificial intelligence: first-party connectors → raw historical archive → deterministic and
	4	+> local-LLM extraction → temporal knowledge graph → search / compare / timeline → www.ai-atlas.co + API.
	5	+
	6	+## 1. Principles that shape the code
	7	+
	8	+1. **No critical dependence on third-party data APIs.** Every source is read directly (HTML, embedded JSON, JSON-LD, RSS/Atom, sitemaps,
	9	+ Markdown docs, raw Git files, public JSON/CSV files). Scrapfly / Firecrawl / a headless browser are *optional escalation transports*
	10	+ (`Fetcher(escalate=True)`), never the database.
	11	+2. **Provenance first.** Every claim, relation, price, benchmark result and event stores `source_id`, `snapshot_id`, `source_url`, `tier`,
	12	+ `confidence`, `extractor` and `observed_at`. Entity pages can always answer "where does this number come from?".
	13	+3. **History is never disposable.** Claims are temporal (`valid_from`/`valid_to`, statuses `current|superseded|conflicting|retracted`),
	14	+ prices and benchmark results are append-only, raw snapshots are content-addressed and kept forever, migrations are forward-only.
	15	+4. **Deterministic before LLM.** Stage 1 (DOM, JSON-LD, tables, Markdown, feeds) runs on every document. The local LLM factory
	16	+ (MacLustr llm-api.io through an OpenAI-compatible gateway) only sees documents flagged `needs_llm`, through versioned pydantic schemas,
	17	+ with full token accounting.
	18	+5. **Never fabricate.** Missing = missing. Conflicts between sources are stored side by side, flagged and queued for review — never averaged.
	19	+
	20	+## 2. Topology
	21	+
	22	+```
	23	+ ┌──────────────── aia schedule (PM2 ai-atlas-scheduler) ─────────────────┐
	24	+ public AI web ──► │ connectors (registry) → Fetcher (direct/escalate) → archive (raw+text) │
	25	+ │ → parse → extract (Facts) → FactWriter (resolve, version, diff) │
	26	+ │ → jobs (llm_extract, embed, reprocess) → worker → LLM gateway │
	27	+ └──────────────────────────────────────────────────────────────────────────┘
	28	+ │ Postgres 17 (+pgvector, pg_trgm) · Redis (cache/locks) · AIA_DATA_DIR
	29	+ ┌──────────────────────────────┴───────────────────────────────────────────┐
	30	+ │ FastAPI /api/v1 (PM2 ai-atlas-api :8321) ◄── Next.js 16 (PM2 ai-atlas-web :8320) ◄── MacLustr Tunnel (BHS64 Caddy)
	31	+ └──────────────────────────────────────────────────────────────────────────┘
	32	+```
	33	+
	34	+Production node: **M2M32c** (Mac Studio M2, 12 c / 32 GB, dedicated), Postgres 17 + pgvector + Redis via Homebrew, data in
	35	+`~/ai-atlas-data/` (raw, text, backups, logs). Public route `https://www.ai-atlas.co → M2M32c:8320` on the MacLustr Tunnel gateway BHS64.
	36	+Workers can run on any node with database access (`aia worker`), which is how the LLM factory scales across the cluster.
	37	+
	38	+## 3. Repository layout
	39	+
	40	+```
	41	+registry/ curated YAML: sources.yaml (domains, tiers, crawl policy), organizations.yaml (aliases, domains, hf/github orgs),
	42	+ providers.yaml, benchmarks.yaml, hardware.yaml — every entry carries its source_url
	43	+migrations/versions/ forward-only SQL (0001_initial.py)
	44	+src/aiatlas/
	45	+ config.py settings (AIA_*), ids.py (prefixed ULIDs, slugify, normalize_alias), logging.py, db/ (SQLAlchemy Core + asyncpg)
	46	+ sdk/ connector SDK
	47	+ fetch.py Fetcher: robots.txt, per-domain rate limit, ETag/If-Modified-Since, retries/backoff, escalation chain
	48	+ archive.py content-addressed gzip store (raw/, text/) with dedupe
	49	+ extract/ html (selectolax; meta, JSON-LD, embedded JSON, tables, links, text), feeds, sitemap, markdown (front matter, tables),
	50	+ numbers (70B, 128K, $3/1M), dates (partial precision)
	51	+ facts.py EntityRef, Claim, Relation, Event, PriceObs, ResultObs, Target, Facts (+ MATERIAL_PROPERTIES → event types)
	52	+ resolution.py Resolver: identifiers → aliases (org-disambiguated) → slug → create; ambiguity → review queue
	53	+ writer.py FactWriter: temporal claims, attributes/provenance materialisation, conflicts, prices, results, events
	54	+ connector.py BaseConnector.run(): documents, snapshots, hash change detection, structural diff, DOCUMENT_CHANGED,
	55	+ breakage detection, circuit breaker, adaptive interval, follow-up targets, --file overrides, reprocess mode
	56	+ connectors/ one package per group (labs/, hub/, research/, code/, providers/, benchmarks/, hardware/); CONNECTORS = [cls]
	57	+ schemas/extraction.py pydantic schemas: ModelPassport, PricingExtraction, CompanyPassport, PaperPassport, BenchmarkResultExtraction, HardwareSpec, …
	58	+ services/
	59	+ jobs.py Postgres queue (SKIP LOCKED), handlers.py (llm_extract, embed_entity, reprocess_snapshot, recompute_quality)
	60	+ llm/gateway.py LLMGateway: stage cascade small→medium→large, OpenAI-compatible engine, JSON validation, llm_jobs accounting
	61	+ embeddings.py local embeddings → pgvector · search.py FTS+trigram(+vector) and NL→filters compiler
	62	+ scheduler.py APScheduler: due connectors (Redis lock), worker, hourly stats/quality/embeddings, nightly pg_dump
	63	+ quality.py transparency scores (documented in metric_definitions) · stats.py live counters · backup.py · cache.py
	64	+ api/ FastAPI application (see docs/API.md)
	65	+ cli.py `aia`: migrate seed connectors run reprocess crawl status stats quality schedule worker api backup llm embed search review
	66	+apps/web/ Next.js 16 site (see docs/FRONTEND.md)
	67	+tests/ pytest with saved fixtures (tests/fixtures/<connector>/…); live tests behind `-m live`
	68	+deploy/ mld manifest (ai-atlas.mld.json), render-manifest.sh, first-run.sh — see docs/DEPLOY.md
	69	+```
	70	+
	71	+## 4. Data model (Postgres)
	72	+
	73	+| table | role |
	74	+|---|---|
	75	+| `entities` | unified graph node: `entity_type`, `canonical_name`, `slug`, `organization_id`, `attributes` (current value per property), `provenance` (per property), `quality`, `counts`, `first_seen_at`, `search` tsvector |
	76	+| `entity_aliases`, `entity_identifiers` | resolution keys (normalized aliases; `(scheme, value)` unique: hf_repo, github_repo, arxiv, doi, anthropic_model_id, openrouter, pypi, domain…) |
	77	+| `sources`, `connectors`, `connector_runs`, `connector_errors` | source registry with tiers; connector state (health, adaptive interval, circuit, parser_version, expected_min_records) |
	78	+| `documents` | one row per URL: validators (etag/last-modified), `content_hash`, counters, `entity_id`, `needs_llm` |
	79	+| `snapshots` | one row per *changed* fetch: `raw_path`, `text_path`, `structured` (deterministic summary), `diff`, `transport`, `parser_version`, `processing_status` |
	80	+| `claims` | temporal facts: `(entity_id, property, value jsonb, unit, tier, confidence, status, valid_from, valid_to, snapshot_id, source_url, extractor)` |
	81	+| `relations` | typed edges with validity: `develops, owns, operates, available_through, evaluated_on, described_by, derived_from, fine_tuned_from, quantized_from, superseded_by, runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates` |
	82	+| `change_events` | the event engine: `NEW_MODEL, MODEL_UPDATED, PRICE_CHANGED, PROVIDER_LISTED, CONTEXT_CHANGED, STATUS_CHANGED, LICENSE_CHANGED, DEPRECATION_ANNOUNCED, RETIREMENT_ANNOUNCED, BENCHMARK_RESULT, BENCHMARK_UPDATED, NEW_PAPER, ANNOUNCEMENT, RELEASE, VERSION_RELEASED, DOCUMENT_CHANGED, …` with `category`, `importance 0–3`, `dedupe_key` |
	83	+| `prices` | append-only provider pricing (`valid_to` closes a row): per-1M-token input/output/cached/cache-write/batch, per image/request, context, features |
	84	+| `benchmark_results` | append-only results with `config` (harness, prompting, judge…) — never compared blindly |
	85	+| `jobs`, `llm_jobs` | work queue; LLM cost accounting (stage, model, node, tokens, duration, status) |
	86	+| `review_queue` | merge candidates, conflicts, blocked sources, parser breakage, unusual changes |
	87	+| `entity_embeddings` | pgvector(1024) local embeddings |
	88	+| `domains`, `stats_snapshots`, `page_views`, `api_keys`, `metric_definitions` | trust graph, counters history, trending, developer keys, documented metrics |
	89	+
	90	+Identifiers: prefixed ULIDs (`model_01J…`, `company_…`, `paper_…`, `snap_…`, `evt_…`). Slugs are stable, readable and unique.
	91	+
	92	+### Temporal rules (FactWriter)
	93	+
	94	+```
	95	+no current claim → insert current, set attribute, NEW_<TYPE> event when the entity is new
	96	+same value → confirm (observed_at)
	97	+different, source ≥ tier → supersede (valid_to = observed), insert current, CHANGE event for material properties
	98	+different, worse source → store as `conflicting`, mark current `conflicted`, review-queue item — never overwrite
	99	+soft text (description…) → first statement wins until its own source changes; never an event
	100	+metric.* / stats.* → time series without events
	101	+```
	102	+
	103	+Historical mode = `claims` filtered by `valid_from <= date < coalesce(valid_to, ∞)`; "did it exist?" = `entities.first_seen_at <= date`.
	104	+Diff A→B = new/removed entities by `first_seen_at`, claims superseded between A and B, price rows opened/closed, results observed.
	105	+
	106	+## 5. Connector lifecycle
	107	+
	108	+`discover()` returns `Target`s (URL, doc_type, optional entity hint, `needs_llm`, `key` for fixtures). For each target `run()`:
	109	+fetch (conditional) → unchanged? stop · changed → archive raw + text → snapshot row (+ structural diff vs previous) → `parse()` →
	110	+`extract()` → `FactWriter.write(facts)` → follow-up targets → optional `llm_extract` job. Blocked (401/403/451/robots) → document
	111	+`blocked` + review item; 404/410 twice → `gone` (never deletes entities). Fewer records than `expected_min_records` → run `suspect`
	112	++ review item. Intervals adapt: ×0.7 on change (≥ min), ×1.5 after 3 unchanged runs (≤ max). Three failures open the circuit 30 min.
	113	+
	114	+Parser improvements: bump `parser_version`, then `aia reprocess <connector>` replays the latest stored snapshot of every document — no crawl.
	115	+
	116	+## 6. LLM factory
	117	+
	118	+`LLMGateway.extract(task_type, document, schema, stage)` → strict JSON validated by a pydantic schema; on schema errors the next stage
	119	+model is tried (small → medium → large); transport failures do not escalate. Every call is recorded in `llm_jobs` (tokens, duration,
	120	+node, status, output). `facts_from_llm()` maps outputs onto `Facts` with `extractor='llm'` and medium/low confidence, so deterministic
	121	+tier-1 claims always win conflicts. Embeddings use the same gateway (`/v1/embeddings`, Qwen3-Embedding 1024-d) into pgvector.
	122	+Without a configured gateway the platform degrades gracefully: deterministic extraction, FTS search, no embeddings.
	123	+
	124	+## 7. Quality, search, events
	125	+
	126	+* `quality.score` = 100·(0.25 completeness + 0.25 primary-source ratio + 0.20 freshness + 0.15 agreement + 0.15 source diversity),
	127	+ versioned and explained in `metric_definitions` and `/methodology`. It measures how well AI Atlas knows an entity, not the entity.
	128	+* Search: tsvector (name A, family B, type/description C) + `pg_trgm` similarity + aliases (+ cosine on embeddings when present).
	129	+ `compile_query()` turns "open models released in 2026 with more than 100B parameters and 128k context" into structured filters.
	130	+* Events power the homepage feed, entity timelines, `/changes`, the daily digest and, later, watchlists/alerts.
	131	+
	132	+## 8. Security & compliance
	133	+
	134	+Admin routes require `x-aia-admin-token`. The API binds to loopback; Next.js rewrites `/api/v1/*` to it. No credentials or internal
	135	+addresses in responses. Crawling honours robots.txt, identifies itself (`AIAtlasBot/0.1 … contact@spboucher.ai`), rate-limits per domain,
	136	+uses conditional requests, never bypasses access controls; blocked sources go to the review queue. Raw archive content is never exposed
	137	+publicly — only extracted facts with links to the original page.
	138	+
	139	+## 9. Backups & durability
	140	+
	141	+Nightly `pg_dump -Fc` into `AIA_DATA_DIR/backups` (30 kept), off-node rsync (`scripts/backup-offnode.sh`), raw archive is append-only and
	142	+content-addressed (rsync-friendly). The dataset is the product: restore = `pg_restore` + point `DATABASE_URL`.

added docs/CONNECTORS.md +97 −0

		@@ -0,0 +1,97 @@
	1	+# Writing a connector
	2	+
	3	+A connector is a Python class under `src/aiatlas/connectors/<group>/<name>.py`, listed in that module's `CONNECTORS = [Cls]`.
	4	+The registry auto-discovers it; `aia seed` registers it in the database; `aia run <name>` runs it; the scheduler runs it on its interval.
	5	+
	6	+## Skeleton
	7	+
	8	+```python
	9	+from aiatlas.registry import org_ref, provider_ref
	10	+from aiatlas.sdk.connector import BaseConnector, Parsed, RunContext
	11	+from aiatlas.sdk.facts import Facts, Target
	12	+from aiatlas.sdk.fetch import FetchResult
	13	+
	14	+class ExampleConnector(BaseConnector):
	15	+ name = "example" # unique, snake_case, stable (used in documents/claims/events)
	16	+ label = "Example Lab — models & news"
	17	+ description = "Official docs and blog of Example Lab."
	18	+ source_key = "example.com" # must exist in registry/sources.yaml (tier, rate limit, org)
	19	+ version = "1"; parser_version = "1" # bump parser_version when extraction improves → `aia reprocess example`
	20	+ interval_seconds = 3600; min_interval_seconds = 1800; max_interval_seconds = 86400
	21	+ rate_per_min = 15; tier = 1; priority = 0 | 1 | 2
	22	+ expected_min_records = 5 # breakage detection: fewer entities/prices/results ⇒ run "suspect", nothing deleted
	23	+ concurrency = 2
	24	+
	25	+ async def discover(self, ctx: RunContext) -> list[Target]:
	26	+ return [Target(url="https://example.com/models", doc_type="model_docs", key="models", min_bytes=2000),
	27	+ Target(url="https://example.com/blog/rss.xml", doc_type="feed", key="feed")]
	28	+
	29	+ async def extract(self, ctx: RunContext, target: Target, res: FetchResult, parsed: Parsed) -> Facts:
	30	+ facts = Facts()
	31	+ org = org_ref("example") # registry organization → deterministic identifiers
	32	+ facts.entities.append(org)
	33	+ if target.key == "models" and parsed.html:
	34	+ for row in parsed.html.tables[0]["rows"]: ...
	35	+ model = facts.entity("model", name, identifiers={"example_model_id": api_id}, organization=org)
	36	+ facts.relate(org, "develops", model)
	37	+ facts.claim(model, "context_length", 128000, unit="tokens")
	38	+ facts.price(model=model, provider=provider_ref("example"), input_per_mtok=1.0, output_per_mtok=4.0)
	39	+ facts.follow(detail_url, doc_type="model_page", entity=model, needs_llm=True, meta={"llm_task": "model_passport"})
	40	+ elif target.key == "feed" and parsed.kind == "feed":
	41	+ announcement_events(facts, org, parsed.feed_items, source_name="example.com/blog") # from connectors/labs/_common.py
	42	+ return facts
	43	+
	44	+CONNECTORS = [ExampleConnector]
	45	+```
	46	+
	47	+## What `Parsed` gives you
	48	+
	49	+| `parsed.kind` | fields |
	50	+|---|---|
	51	+| `html` | `parsed.html: HtmlDoc` — `title, canonical, description, meta, og, json_ld, embedded_json (__NEXT_DATA__, data-props…), headings, tables [{headers, rows}], links [(href, text)], text`, `css()`, `links_matching()` ; `find_in_json(obj, key)` helper |
	52	+| `markdown` | `parsed.markdown: MarkdownDoc` — `front_matter, headings, tables, links, text, section(pattern)` |
	53	+| `feed` | `parsed.feed_items: list[FeedItem]` (`url, title, summary, published_at, categories, authors`) |
	54	+| `json` | `parsed.json` |
	55	+| `pdf` / `text` / `xml` | `parsed.text` |
	56	+
	57	+Helpers: `aiatlas.sdk.extract.numbers` (`parse_param_count("70B")`, `parse_active_params("235B-A22B")`, `parse_context_length("128K")`,
	58	+`parse_money_per_mtok("$3 / 1M tokens")`, `parse_percent`), `aiatlas.sdk.extract.dates` (`parse_datetime`, `parse_date` with precision),
	59	+`connectors/labs/_common.py` (`announcement_events`, `transpose_feature_table`, `kv_tables`, `clean_cell`, `money`, `tokens`, `month_year`, `parse_retirement`, `model_ref`).
	60	+
	61	+## Rules
	62	+
	63	+1. **Direct mode first.** Use official pages, feeds, sitemaps, Markdown/raw files, embedded JSON. Public JSON files (e.g. `openrouter.ai/api/v1/models`,
	64	+ `pypi.org/pypi/<pkg>/json`, arXiv Atom) are fine — they are public documents, not commercial APIs. Never require an API key.
	65	+ Set `Target(escalate=True)` only for pages known to block bots *and* only if a key is configured; otherwise let the document be `blocked`.
	66	+2. **Never fabricate.** Only emit a claim when the page states it. Unknown → no claim. Don't infer parameter counts from names unless the
	67	+ name literally contains them (`Qwen3-235B-A22B` → 235e9 / 22e9 is fine; "Large" is not).
	68	+3. **Identifiers make resolution deterministic.** Give models the provider's API id (`{provider}_model_id`), HF repo (`hf_repo`), arXiv id
	69	+ (`arxiv`), GitHub repo (`github_repo`), PyPI name (`pypi`). Organizations always come from `org_ref(<registry key>)` (add missing orgs to
	70	+ `registry/organizations.yaml` with a `source_url`).
	71	+4. **Properties are shared vocabulary** (see below). Add new ones sparingly; prefix metrics with `metric.` (downloads, likes, stars) so they
	72	+ never generate events.
	73	+5. **Events**: NEW_*, PRICE_CHANGED, CONTEXT_CHANGED… are emitted automatically by the writer. Emit `ANNOUNCEMENT`/`RELEASE` events yourself for
	74	+ feed items (`announcement_events`) with `effective_at` = publication date and a `dedupe_key` (URL).
	75	+6. **Follow-ups** (`facts.follow`) let a listing discover detail pages; keep `max_targets` reasonable (`ctx.max_targets`, default 2000).
	76	+7. **Every connector has a fixture test**: save real responses under `tests/fixtures/<connector>/…` and assert extracted facts
	77	+ (see `tests/test_anthropic.py`). Run connectors with `--file key=path` to use fixtures instead of the network.
	78	+8. Respect `rate_per_min` from `registry/sources.yaml`; arXiv ≤ 4/min; Hugging Face ≤ 30/min; GitHub ≤ 20/min.
	79	+
	80	+## Property vocabulary (entities.attributes)
	81	+
	82	+**model**: `family, version, release_date (ISO, may be YYYY-MM), status (active|preview|deprecated|retired|announced|limited-availability), openness
	83	+(open-weights|open-source|proprietary|restricted), license, architecture, parameter_count (int), active_parameter_count, is_moe, modalities
	84	+(list of text|image|audio|video|code|embedding), modalities_input, modalities_output, context_length (tokens), max_output_tokens, knowledge_cutoff
	85	+(YYYY-MM), training_data_cutoff, languages, tool_calling, structured_output, reasoning, vision, audio, fine_tuning_available, tokenizer,
	86	+api_model_id, api_alias, official_url, model_card_url, paper_url, repository_url, hf_repo, pipeline_tag, base_model, quantization, quant_format
	87	+(gguf|mlx|awq|gptq|fp8), file_size_gb, deprecation_date, retirement_date, retirement_tentative, metric.downloads, metric.likes`
	88	+
	89	+**company / organization / lab**: `country (ISO-2), headquarters, founded, website, domains, hf_org, github_org, org_kind, legal_name, founders, leadership, employee_count`
	90	+
	91	+**provider**: `website, pricing_url, docs_url, regions, features` · **paper**: `authors, published_at, updated_at, abstract, arxiv_id, doi, categories, primary_category, pdf_url, code_url, venue`
	92	+· **benchmark**: `category, task, metric, unit, creator, website, paper, known_limitations, methodology` · **hardware**: `kind, architecture, release_date, memory_gb,
	93	+memory_type, memory_bandwidth_gbs, tdp_watts, runtimes, manufacturer, spec_url, price_usd, compute_fp16_tflops` · **framework / repository**: `repository_url, latest_version,
	94	+latest_release_at, license, language, description, topics, pypi, metric.stars, metric.forks` · **dataset**: `license, modality, size, publisher, task, hf_repo`
	95	+
	96	+Relations: `develops, owns, operates, available_through, evaluated_on, described_by, derived_from, fine_tuned_from, quantized_from, distilled_from,
	97	+merged_from, superseded_by, runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates, published_by`.

added migrations/env.py +45 −0

		@@ -0,0 +1,45 @@
	1	+"""Alembic environment (async engine). Migrations are plain SQL executed through `op.execute`."""
	2	+from __future__ import annotations
	3	+
	4	+import asyncio
	5	+from logging.config import fileConfig
	6	+
	7	+from alembic import context
	8	+from sqlalchemy.ext.asyncio import create_async_engine
	9	+
	10	+from aiatlas.config import settings
	11	+
	12	+config = context.config
	13	+if config.config_file_name is not None:
	14	+ fileConfig(config.config_file_name)
	15	+
	16	+target_metadata = None
	17	+
	18	+
	19	+def run_migrations_offline() -> None:
	20	+ context.configure(url=settings.sync_database_url, literal_binds=True, dialect_opts={"paramstyle": "named"})
	21	+ with context.begin_transaction():
	22	+ context.run_migrations()
	23	+
	24	+
	25	+def do_run_migrations(connection) -> None: # type: ignore[no-untyped-def]
	26	+ context.configure(connection=connection, target_metadata=target_metadata, transaction_per_migration=True)
	27	+ with context.begin_transaction():
	28	+ context.run_migrations()
	29	+
	30	+
	31	+async def run_async_migrations() -> None:
	32	+ engine = create_async_engine(settings.database_url, poolclass=None)
	33	+ async with engine.connect() as connection:
	34	+ await connection.run_sync(do_run_migrations)
	35	+ await engine.dispose()
	36	+
	37	+
	38	+def run_migrations_online() -> None:
	39	+ asyncio.run(run_async_migrations())
	40	+
	41	+
	42	+if context.is_offline_mode():
	43	+ run_migrations_offline()
	44	+else:
	45	+ run_migrations_online()

added migrations/script.py.mako +21 −0

		@@ -0,0 +1,21 @@
	1	+"""${message}
	2	+
	3	+Revision ID: ${up_revision}
	4	+Revises: ${down_revision | comma,n}
	5	+"""
	6	+from __future__ import annotations
	7	+
	8	+from alembic import op
	9	+
	10	+revision = ${repr(up_revision)}
	11	+down_revision = ${repr(down_revision)}
	12	+branch_labels = ${repr(branch_labels)}
	13	+depends_on = ${repr(depends_on)}
	14	+
	15	+
	16	+def upgrade() -> None:
	17	+ ${upgrades if upgrades else "pass"}
	18	+
	19	+
	20	+def downgrade() -> None:
	21	+ raise RuntimeError("forward-only")

added migrations/versions/0001_initial.py +502 −0

		@@ -0,0 +1,502 @@
	1	+"""AI Atlas initial schema — entities, aliases, identifiers, sources, connectors, documents, snapshots, claims, relations,
	2	+change events, benchmark results, prices, jobs, LLM jobs, review queue, stats.
	3	+
	4	+Revision ID: 0001
	5	+"""
	6	+from __future__ import annotations
	7	+
	8	+from alembic import op
	9	+
	10	+revision = "0001"
	11	+down_revision = None
	12	+branch_labels = None
	13	+depends_on = None
	14	+
	15	+SQL = r"""
	16	+create extension if not exists pg_trgm;
	17	+create extension if not exists "uuid-ossp";
	18	+do $$ begin
	19	+ create extension if not exists vector;
	20	+exception when others then
	21	+ raise notice 'pgvector unavailable: %', sqlerrm;
	22	+end $$;
	23	+
	24	+-- ---------------------------------------------------------------------------------------------- reference: sources & connectors
	25	+create table if not exists sources (
	26	+ id text primary key,
	27	+ key text not null unique, -- e.g. 'openai.com', 'huggingface.co', 'arxiv.org'
	28	+ name text not null,
	29	+ domain text not null,
	30	+ organization_id text, -- entity id of the owning organization (domain trust graph)
	31	+ tier smallint not null default 2, -- 1 official primary, 2 quality secondary, 3 community, 4 unverified
	32	+ kind text not null default 'website', -- website|docs|feed|repository|registry|dataset|leaderboard|regulator
	33	+ category text not null default 'lab',
	34	+ base_url text,
	35	+ robots_policy text not null default 'respect', -- respect|documented-exception
	36	+ rate_limit_per_min integer not null default 30,
	37	+ crawl_interval_s integer not null default 86400,
	38	+ enabled boolean not null default true,
	39	+ priority smallint not null default 2, -- 0 = P0
	40	+ notes text,
	41	+ meta jsonb not null default '{}'::jsonb,
	42	+ created_at timestamptz not null default now(),
	43	+ updated_at timestamptz not null default now()
	44	+);
	45	+
	46	+create table if not exists connectors (
	47	+ name text primary key,
	48	+ source_id text references sources(id),
	49	+ label text not null,
	50	+ description text,
	51	+ enabled boolean not null default true,
	52	+ priority smallint not null default 2,
	53	+ interval_seconds integer not null default 3600,
	54	+ min_interval_seconds integer not null default 900,
	55	+ max_interval_seconds integer not null default 604800,
	56	+ parser_version text not null default '1',
	57	+ rate_limit_per_min integer not null default 30,
	58	+ owner text not null default 'ai-atlas',
	59	+ expected_min_records integer not null default 0, -- breakage detection: fewer records than this = suspected failure
	60	+ last_attempt_at timestamptz,
	61	+ last_success_at timestamptz,
	62	+ last_change_at timestamptz,
	63	+ next_run_at timestamptz,
	64	+ last_duration_ms integer,
	65	+ consecutive_failures integer not null default 0,
	66	+ consecutive_unchanged integer not null default 0,
	67	+ circuit_open_until timestamptz,
	68	+ health text not null default 'unknown', -- ok|degraded|failing|disabled|unknown
	69	+ meta jsonb not null default '{}'::jsonb,
	70	+ created_at timestamptz not null default now(),
	71	+ updated_at timestamptz not null default now()
	72	+);
	73	+
	74	+create table if not exists connector_runs (
	75	+ id text primary key,
	76	+ connector_name text not null references connectors(name),
	77	+ started_at timestamptz not null,
	78	+ finished_at timestamptz,
	79	+ status text not null, -- running|success|unchanged|failed|skipped|suspect
	80	+ duration_ms integer,
	81	+ docs_discovered integer not null default 0,
	82	+ docs_fetched integer not null default 0,
	83	+ docs_changed integer not null default 0,
	84	+ docs_unchanged integer not null default 0,
	85	+ docs_failed integer not null default 0,
	86	+ entities_created integer not null default 0,
	87	+ entities_updated integer not null default 0,
	88	+ claims_written integer not null default 0,
	89	+ relations_written integer not null default 0,
	90	+ events_emitted integer not null default 0,
	91	+ error text,
	92	+ meta jsonb not null default '{}'::jsonb
	93	+);
	94	+create index if not exists connector_runs_name_idx on connector_runs (connector_name, started_at desc);
	95	+
	96	+create table if not exists connector_errors (
	97	+ id bigserial primary key,
	98	+ connector_name text not null,
	99	+ run_id text,
	100	+ url text,
	101	+ error_type text not null,
	102	+ message text not null,
	103	+ created_at timestamptz not null default now()
	104	+);
	105	+create index if not exists connector_errors_name_idx on connector_errors (connector_name, created_at desc);
	106	+
	107	+-- ---------------------------------------------------------------------------------------------- entities (unified graph)
	108	+create table if not exists entities (
	109	+ id text primary key,
	110	+ entity_type text not null,
	111	+ canonical_name text not null,
	112	+ slug text not null unique,
	113	+ description text,
	114	+ status text not null default 'active', -- active|deprecated|retired|announced|unknown|merged
	115	+ organization_id text references entities(id),
	116	+ attributes jsonb not null default '{}'::jsonb, -- current value per property (materialised from claims)
	117	+ provenance jsonb not null default '{}'::jsonb, -- property -> {source, snapshot_id, url, observed_at, tier, confidence}
	118	+ quality jsonb not null default '{}'::jsonb, -- source_count, primary_source_ratio, freshness, completeness, agreement, score
	119	+ counts jsonb not null default '{}'::jsonb, -- cached relation / claim / event counts
	120	+ first_seen_at timestamptz not null default now(),
	121	+ last_seen_at timestamptz not null default now(),
	122	+ merged_into text references entities(id),
	123	+ search tsvector,
	124	+ created_at timestamptz not null default now(),
	125	+ updated_at timestamptz not null default now()
	126	+);
	127	+create index if not exists entities_type_idx on entities (entity_type, canonical_name);
	128	+create index if not exists entities_org_idx on entities (organization_id);
	129	+create index if not exists entities_search_idx on entities using gin (search);
	130	+create index if not exists entities_name_trgm_idx on entities using gin (canonical_name gin_trgm_ops);
	131	+create index if not exists entities_attrs_idx on entities using gin (attributes jsonb_path_ops);
	132	+create index if not exists entities_updated_idx on entities (updated_at desc);
	133	+create index if not exists entities_first_seen_idx on entities (first_seen_at desc);
	134	+
	135	+create table if not exists entity_aliases (
	136	+ id bigserial primary key,
	137	+ entity_id text not null references entities(id) on delete cascade,
	138	+ alias text not null,
	139	+ alias_norm text not null,
	140	+ kind text not null default 'alias', -- alias|former_name|abbreviation|localized|typo
	141	+ snapshot_id text,
	142	+ created_at timestamptz not null default now(),
	143	+ unique (entity_id, alias_norm)
	144	+);
	145	+create index if not exists entity_aliases_norm_idx on entity_aliases (alias_norm);
	146	+
	147	+create table if not exists entity_identifiers (
	148	+ id bigserial primary key,
	149	+ entity_id text not null references entities(id) on delete cascade,
	150	+ scheme text not null, -- hf_repo|github_repo|arxiv|doi|openrouter|domain|pypi|wikidata|url|provider_model_id
	151	+ value text not null,
	152	+ snapshot_id text,
	153	+ created_at timestamptz not null default now(),
	154	+ unique (scheme, value)
	155	+);
	156	+create index if not exists entity_identifiers_entity_idx on entity_identifiers (entity_id);
	157	+
	158	+-- ---------------------------------------------------------------------------------------------- documents & snapshots (raw archive)
	159	+create table if not exists documents (
	160	+ id text primary key,
	161	+ source_id text references sources(id),
	162	+ connector_name text references connectors(name),
	163	+ url text not null unique,
	164	+ canonical_url text,
	165	+ doc_type text not null default 'page', -- page|feed|feed_item|model_card|docs|pricing|release|paper|pdf|json|sitemap|repo|readme|leaderboard
	166	+ title text,
	167	+ entity_id text references entities(id),
	168	+ status text not null default 'active', -- active|gone|blocked|error
	169	+ first_seen_at timestamptz not null default now(),
	170	+ last_fetched_at timestamptz,
	171	+ last_changed_at timestamptz,
	172	+ last_status integer,
	173	+ etag text,
	174	+ last_modified text,
	175	+ content_hash text,
	176	+ fetch_count integer not null default 0,
	177	+ change_count integer not null default 0,
	178	+ fail_count integer not null default 0,
	179	+ fetch_interval_s integer,
	180	+ next_fetch_at timestamptz,
	181	+ priority smallint not null default 2,
	182	+ needs_llm boolean not null default false,
	183	+ meta jsonb not null default '{}'::jsonb
	184	+);
	185	+create index if not exists documents_connector_idx on documents (connector_name, last_fetched_at desc);
	186	+create index if not exists documents_entity_idx on documents (entity_id);
	187	+create index if not exists documents_next_idx on documents (next_fetch_at) where status = 'active';
	188	+
	189	+create table if not exists snapshots (
	190	+ id text primary key,
	191	+ document_id text not null references documents(id) on delete cascade,
	192	+ run_id text,
	193	+ url text not null,
	194	+ final_url text,
	195	+ observed_at timestamptz not null default now(),
	196	+ http_status integer,
	197	+ headers jsonb not null default '{}'::jsonb,
	198	+ content_type text,
	199	+ content_hash text not null,
	200	+ byte_size integer not null default 0,
	201	+ raw_path text, -- content-addressed gzip under AIA_DATA_DIR/raw
	202	+ text_path text, -- cleaned text (gzip) under AIA_DATA_DIR/text
	203	+ text_hash text,
	204	+ structured jsonb, -- deterministic extraction (json-ld, meta, tables, embedded json)
	205	+ parser_version text not null default '1',
	206	+ connector_version text not null default '1',
	207	+ transport text not null default 'direct', -- direct|browser|scrapfly|firecrawl|file|git
	208	+ changed boolean not null default true,
	209	+ diff jsonb, -- structural diff vs previous changed snapshot
	210	+ processing_status text not null default 'stored', -- stored|extracted|llm_pending|llm_done|failed
	211	+ created_at timestamptz not null default now()
	212	+);
	213	+create index if not exists snapshots_doc_idx on snapshots (document_id, observed_at desc);
	214	+create index if not exists snapshots_hash_idx on snapshots (content_hash);
	215	+create index if not exists snapshots_status_idx on snapshots (processing_status) where processing_status in ('stored','llm_pending');
	216	+
	217	+-- ---------------------------------------------------------------------------------------------- temporal facts
	218	+create table if not exists claims (
	219	+ id text primary key,
	220	+ entity_id text not null references entities(id) on delete cascade,
	221	+ property text not null,
	222	+ value jsonb not null,
	223	+ value_text text,
	224	+ value_num double precision,
	225	+ unit text,
	226	+ source_id text references sources(id),
	227	+ snapshot_id text references snapshots(id),
	228	+ source_url text,
	229	+ tier smallint not null default 2,
	230	+ confidence text not null default 'medium', -- verified|high|medium|low|conflicted
	231	+ status text not null default 'current', -- current|superseded|conflicting|retracted
	232	+ extractor text not null default 'deterministic',
	233	+ extractor_version text not null default '1',
	234	+ observed_at timestamptz not null default now(),
	235	+ effective_at timestamptz,
	236	+ valid_from timestamptz not null default now(),
	237	+ valid_to timestamptz,
	238	+ created_at timestamptz not null default now()
	239	+);
	240	+create index if not exists claims_entity_prop_idx on claims (entity_id, property, valid_from desc);
	241	+create index if not exists claims_current_idx on claims (entity_id, property) where status = 'current';
	242	+create index if not exists claims_snapshot_idx on claims (snapshot_id);
	243	+create index if not exists claims_observed_idx on claims (observed_at desc);
	244	+
	245	+create table if not exists relations (
	246	+ id text primary key,
	247	+ subject_id text not null references entities(id) on delete cascade,
	248	+ predicate text not null,
	249	+ object_id text not null references entities(id) on delete cascade,
	250	+ attributes jsonb not null default '{}'::jsonb,
	251	+ source_id text references sources(id),
	252	+ snapshot_id text references snapshots(id),
	253	+ source_url text,
	254	+ tier smallint not null default 2,
	255	+ confidence text not null default 'medium',
	256	+ observed_at timestamptz not null default now(),
	257	+ valid_from timestamptz not null default now(),
	258	+ valid_to timestamptz,
	259	+ created_at timestamptz not null default now()
	260	+);
	261	+create unique index if not exists relations_live_uniq on relations (subject_id, predicate, object_id) where valid_to is null;
	262	+create index if not exists relations_subject_idx on relations (subject_id, predicate);
	263	+create index if not exists relations_object_idx on relations (object_id, predicate);
	264	+
	265	+create table if not exists change_events (
	266	+ id text primary key,
	267	+ entity_id text references entities(id) on delete cascade,
	268	+ event_type text not null, -- NEW_MODEL|MODEL_UPDATED|PRICE_CHANGED|CONTEXT_CHANGED|NEW_PAPER|RELEASE|BENCHMARK_RESULT|NEW_PROVIDER|DOCUMENT_CHANGED|…
	269	+ category text not null default 'update', -- model|price|benchmark|paper|release|company|hardware|framework|provider|dataset|regulation|incident|repository
	270	+ property text,
	271	+ old_value jsonb,
	272	+ new_value jsonb,
	273	+ summary text,
	274	+ importance smallint not null default 2, -- 0 minor … 3 major
	275	+ observed_at timestamptz not null default now(),
	276	+ effective_at timestamptz,
	277	+ source_id text references sources(id),
	278	+ snapshot_id text references snapshots(id),
	279	+ source_url text,
	280	+ connector_name text,
	281	+ dedupe_key text unique,
	282	+ meta jsonb not null default '{}'::jsonb
	283	+);
	284	+create index if not exists change_events_observed_idx on change_events (observed_at desc);
	285	+create index if not exists change_events_entity_idx on change_events (entity_id, observed_at desc);
	286	+create index if not exists change_events_type_idx on change_events (event_type, observed_at desc);
	287	+create index if not exists change_events_category_idx on change_events (category, observed_at desc);
	288	+
	289	+-- ---------------------------------------------------------------------------------------------- domain tables
	290	+create table if not exists benchmark_results (
	291	+ id text primary key,
	292	+ model_id text not null references entities(id) on delete cascade,
	293	+ benchmark_id text not null references entities(id) on delete cascade,
	294	+ score double precision not null,
	295	+ metric text,
	296	+ unit text,
	297	+ higher_is_better boolean not null default true,
	298	+ config jsonb not null default '{}'::jsonb, -- prompting, judge, tool use, sampling, harness, model variant string
	299	+ evaluated_at timestamptz,
	300	+ observed_at timestamptz not null default now(),
	301	+ source_id text references sources(id),
	302	+ snapshot_id text references snapshots(id),
	303	+ source_url text,
	304	+ tier smallint not null default 2,
	305	+ confidence text not null default 'medium',
	306	+ dedupe_key text unique,
	307	+ valid_to timestamptz
	308	+);
	309	+create index if not exists benchmark_results_model_idx on benchmark_results (model_id, benchmark_id, observed_at desc);
	310	+create index if not exists benchmark_results_bench_idx on benchmark_results (benchmark_id, score desc);
	311	+
	312	+create table if not exists prices (
	313	+ id text primary key,
	314	+ model_id text not null references entities(id) on delete cascade,
	315	+ provider_id text not null references entities(id) on delete cascade,
	316	+ provider_model_id text,
	317	+ input_per_mtok double precision,
	318	+ output_per_mtok double precision,
	319	+ cached_input_per_mtok double precision,
	320	+ cache_write_per_mtok double precision,
	321	+ batch_input_per_mtok double precision,
	322	+ batch_output_per_mtok double precision,
	323	+ per_image double precision,
	324	+ per_request double precision,
	325	+ currency text not null default 'USD',
	326	+ context_length integer,
	327	+ max_output_tokens integer,
	328	+ features jsonb not null default '{}'::jsonb,
	329	+ observed_at timestamptz not null default now(),
	330	+ valid_from timestamptz not null default now(),
	331	+ valid_to timestamptz,
	332	+ source_id text references sources(id),
	333	+ snapshot_id text references snapshots(id),
	334	+ source_url text,
	335	+ tier smallint not null default 2,
	336	+ meta jsonb not null default '{}'::jsonb
	337	+);
	338	+create unique index if not exists prices_live_uniq on prices (model_id, provider_id, coalesce(provider_model_id, '')) where valid_to is null;
	339	+create index if not exists prices_provider_idx on prices (provider_id, valid_from desc);
	340	+create index if not exists prices_model_idx on prices (model_id, valid_from desc);
	341	+
	342	+create table if not exists domains (
	343	+ domain text primary key,
	344	+ organization_id text references entities(id),
	345	+ trust_tier smallint not null default 3,
	346	+ category text,
	347	+ notes text,
	348	+ created_at timestamptz not null default now()
	349	+);
	350	+
	351	+-- ---------------------------------------------------------------------------------------------- work queues
	352	+create table if not exists jobs (
	353	+ id text primary key,
	354	+ kind text not null, -- fetch_document|llm_extract|embed_entity|reprocess_snapshot|recompute_quality|resolve_entity
	355	+ payload jsonb not null default '{}'::jsonb,
	356	+ priority smallint not null default 5, -- 0 highest
	357	+ status text not null default 'queued', -- queued|running|done|failed|dead
	358	+ attempts integer not null default 0,
	359	+ max_attempts integer not null default 3,
	360	+ run_after timestamptz not null default now(),
	361	+ locked_by text,
	362	+ locked_at timestamptz,
	363	+ started_at timestamptz,
	364	+ finished_at timestamptz,
	365	+ error text,
	366	+ batch_id text,
	367	+ dedupe_key text,
	368	+ created_at timestamptz not null default now()
	369	+);
	370	+create index if not exists jobs_pick_idx on jobs (status, priority, run_after) where status = 'queued';
	371	+create unique index if not exists jobs_dedupe_idx on jobs (dedupe_key) where status in ('queued','running');
	372	+create index if not exists jobs_batch_idx on jobs (batch_id);
	373	+
	374	+create table if not exists llm_jobs (
	375	+ id text primary key,
	376	+ job_id text,
	377	+ task_type text not null,
	378	+ stage text not null, -- small|medium|large
	379	+ engine text not null,
	380	+ model text not null,
	381	+ node text,
	382	+ schema_name text,
	383	+ snapshot_id text references snapshots(id),
	384	+ entity_id text references entities(id),
	385	+ input_tokens integer,
	386	+ output_tokens integer,
	387	+ duration_ms integer,
	388	+ status text not null, -- ok|failed|invalid_json|schema_error
	389	+ output jsonb,
	390	+ error text,
	391	+ created_at timestamptz not null default now()
	392	+);
	393	+create index if not exists llm_jobs_created_idx on llm_jobs (created_at desc);
	394	+
	395	+create table if not exists review_queue (
	396	+ id text primary key,
	397	+ kind text not null, -- merge_candidate|conflict|blocked_source|parser_breakage|unusual_change|new_source
	398	+ entity_ids text[] not null default '{}',
	399	+ payload jsonb not null default '{}'::jsonb,
	400	+ reason text not null,
	401	+ status text not null default 'pending', -- pending|approved|rejected|edited
	402	+ resolution jsonb,
	403	+ created_at timestamptz not null default now(),
	404	+ resolved_at timestamptz,
	405	+ dedupe_key text unique
	406	+);
	407	+create index if not exists review_queue_status_idx on review_queue (status, created_at desc);
	408	+
	409	+-- ---------------------------------------------------------------------------------------------- embeddings (optional; local models)
	410	+do $$ begin
	411	+ if exists (select 1 from pg_extension where extname = 'vector') then
	412	+ execute 'create table if not exists entity_embeddings (
	413	+ entity_id text primary key references entities(id) on delete cascade,
	414	+ model text not null,
	415	+ embedding vector(1024) not null,
	416	+ text_hash text not null,
	417	+ created_at timestamptz not null default now())';
	418	+ end if;
	419	+end $$;
	420	+
	421	+-- ---------------------------------------------------------------------------------------------- stats, views, api keys
	422	+create table if not exists stats_snapshots (
	423	+ id bigserial primary key,
	424	+ computed_at timestamptz not null default now(),
	425	+ counts jsonb not null
	426	+);
	427	+
	428	+create table if not exists page_views (
	429	+ path text not null,
	430	+ day date not null,
	431	+ views integer not null default 0,
	432	+ primary key (path, day)
	433	+);
	434	+
	435	+create table if not exists api_keys (
	436	+ id text primary key,
	437	+ key_hash text not null unique,
	438	+ label text not null,
	439	+ owner_email text,
	440	+ plan text not null default 'developer',
	441	+ rate_per_min integer not null default 60,
	442	+ enabled boolean not null default true,
	443	+ created_at timestamptz not null default now(),
	444	+ last_used_at timestamptz,
	445	+ usage_count bigint not null default 0
	446	+);
	447	+
	448	+create table if not exists metric_definitions (
	449	+ key text primary key,
	450	+ label text not null,
	451	+ version text not null,
	452	+ description text not null,
	453	+ formula text,
	454	+ created_at timestamptz not null default now()
	455	+);
	456	+
	457	+-- ---------------------------------------------------------------------------------------------- search vector maintenance
	458	+create or replace function entities_search_update() returns trigger language plpgsql as $$
	459	+begin
	460	+ new.search :=
	461	+ setweight(to_tsvector('simple', coalesce(new.canonical_name, '')), 'A') ||
	462	+ setweight(to_tsvector('simple', coalesce(new.attributes->>'family', '')), 'B') ||
	463	+ setweight(to_tsvector('simple', coalesce(new.entity_type, '')), 'C') ||
	464	+ setweight(to_tsvector('english', left(coalesce(new.description, ''), 4000)), 'C');
	465	+ new.updated_at := now();
	466	+ return new;
	467	+end $$;
	468	+drop trigger if exists entities_search_trg on entities;
	469	+create trigger entities_search_trg before insert or update of canonical_name, description, attributes, entity_type on entities
	470	+ for each row execute function entities_search_update();
	471	+"""
	472	+
	473	+
	474	+def upgrade() -> None:
	475	+ for statement in _split(SQL):
	476	+ op.execute(statement)
	477	+
	478	+
	479	+def downgrade() -> None:
	480	+ raise RuntimeError("AI Atlas migrations are forward-only: historical data is never disposable")
	481	+
	482	+
	483	+def _split(sql: str) -> list[str]:
	484	+ """Split on semicolons outside `$$ … $$` blocks."""
	485	+ out: list[str] = []
	486	+ buf: list[str] = []
	487	+ in_dollar = False
	488	+ for line in sql.splitlines():
	489	+ stripped = line.strip()
	490	+ if stripped.count("$$") % 2 == 1:
	491	+ in_dollar = not in_dollar
	492	+ buf.append(line)
	493	+ if not in_dollar and stripped.endswith(";"):
	494	+ body = [ln for ln in buf if not ln.strip().startswith("--") or in_dollar]
	495	+ stmt = "\n".join(body).strip()
	496	+ if stmt:
	497	+ out.append(stmt)
	498	+ buf = []
	499	+ tail = "\n".join(ln for ln in buf if not ln.strip().startswith("--")).strip()
	500	+ if tail:
	501	+ out.append(tail)
	502	+ return out

added package.json +15 −0

		@@ -0,0 +1,15 @@
	1	+{
	2	+ "name": "ai-atlas",
	3	+ "version": "0.1.0",
	4	+ "private": true,
	5	+ "description": "AI Atlas — explore the entire AI ecosystem. Web workspace (backend is Python, see pyproject.toml).",
	6	+ "packageManager": "pnpm@11.1.2",
	7	+ "engines": { "node": ">=22" },
	8	+ "scripts": {
	9	+ "dev:web": "pnpm --filter @ai-atlas/web run dev",
	10	+ "build": "pnpm --filter @ai-atlas/web run build",
	11	+ "start:web": "pnpm --filter @ai-atlas/web run start",
	12	+ "typecheck": "pnpm -r run typecheck",
	13	+ "qa": "pnpm --filter @ai-atlas/web run qa"
	14	+ }
	15	+}

added pnpm-workspace.yaml +2 −0

		@@ -0,0 +1,2 @@
	1	+packages:
	2	+ - apps/*

added pyproject.toml +55 −0

		@@ -0,0 +1,55 @@
	1	+[project]
	2	+name = "aiatlas"
	3	+version = "0.1.0"
	4	+description = "AI Atlas — the global intelligence layer for artificial intelligence: first-party connectors, raw historical archive, temporal knowledge graph, local LLM extraction factory and public API"
	5	+requires-python = ">=3.12"
	6	+dependencies = [
	7	+ "fastapi>=0.115",
	8	+ "uvicorn[standard]>=0.30",
	9	+ "pydantic>=2.8",
	10	+ "pydantic-settings>=2.4",
	11	+ "sqlalchemy[asyncio]>=2.0.35",
	12	+ "asyncpg>=0.30",
	13	+ "alembic>=1.13",
	14	+ "httpx[http2]>=0.27",
	15	+ "orjson>=3.10",
	16	+ "typer>=0.12",
	17	+ "rich>=13",
	18	+ "redis>=5.1",
	19	+ "apscheduler>=3.10,<4",
	20	+ "python-ulid>=2.7",
	21	+ "pyyaml>=6",
	22	+ "python-dateutil>=2.9",
	23	+ "tenacity>=9",
	24	+ "python-slugify>=8",
	25	+ "selectolax>=0.3.21",
	26	+ "feedparser>=6.0.11",
	27	+ "markdown-it-py>=3.0",
	28	+ "pypdf>=5.0",
	29	+ "rapidfuzz>=3.9",
	30	+ "pgvector>=0.3",
	31	+ "numpy>=2",
	32	+]
	33	+
	34	+[project.optional-dependencies]
	35	+dev = ["pytest>=8", "pytest-asyncio>=0.24", "ruff>=0.6", "respx>=0.21"]
	36	+browser = ["playwright>=1.47"]
	37	+
	38	+[project.scripts]
	39	+aia = "aiatlas.cli:app"
	40	+
	41	+[build-system]
	42	+requires = ["hatchling"]
	43	+build-backend = "hatchling.build"
	44	+
	45	+[tool.hatch.build.targets.wheel]
	46	+packages = ["src/aiatlas"]
	47	+
	48	+[tool.ruff]
	49	+line-length = 120
	50	+target-version = "py312"
	51	+
	52	+[tool.pytest.ini_options]
	53	+testpaths = ["tests"]
	54	+asyncio_mode = "auto"
	55	+markers = ["live: hits real external endpoints (skipped unless -m live)"]

added registry/benchmarks.d/.gitkeep +0 −0

added registry/benchmarks.yaml +173 −0

		@@ -0,0 +1,173 @@
	1	+# Benchmark definitions (entities of type `benchmark`). Results come from connectors; definitions are curated with sources.
	2	+benchmarks:
	3	+ - key: mmlu
	4	+ name: MMLU
	5	+ aliases: [Massive Multitask Language Understanding]
	6	+ category: knowledge
	7	+ task: 57-subject multiple-choice questions
	8	+ metric: accuracy
	9	+ unit: "%"
	10	+ creator: Hendrycks et al.
	11	+ website: https://github.com/hendrycks/test
	12	+ paper: https://arxiv.org/abs/2009.03300
	13	+ known_limitations: Widespread training-data contamination; saturated at the frontier.
	14	+ source_url: https://arxiv.org/abs/2009.03300
	15	+ - key: mmlu-pro
	16	+ name: MMLU-Pro
	17	+ category: knowledge
	18	+ task: harder, 10-option MMLU variant
	19	+ metric: accuracy
	20	+ unit: "%"
	21	+ website: https://github.com/TIGER-AI-Lab/MMLU-Pro
	22	+ paper: https://arxiv.org/abs/2406.01574
	23	+ source_url: https://arxiv.org/abs/2406.01574
	24	+ - key: gpqa
	25	+ name: GPQA
	26	+ aliases: [GPQA Diamond]
	27	+ category: reasoning
	28	+ task: graduate-level science questions
	29	+ metric: accuracy
	30	+ unit: "%"
	31	+ website: https://github.com/idavidrein/gpqa
	32	+ paper: https://arxiv.org/abs/2311.12022
	33	+ source_url: https://arxiv.org/abs/2311.12022
	34	+ - key: humaneval
	35	+ name: HumanEval
	36	+ category: coding
	37	+ task: Python function synthesis from docstrings
	38	+ metric: pass@1
	39	+ unit: "%"
	40	+ website: https://github.com/openai/human-eval
	41	+ paper: https://arxiv.org/abs/2107.03374
	42	+ known_limitations: Saturated; small (164 problems).
	43	+ source_url: https://arxiv.org/abs/2107.03374
	44	+ - key: swe-bench-verified
	45	+ name: SWE-bench Verified
	46	+ aliases: [SWE-bench]
	47	+ category: coding
	48	+ task: resolve real GitHub issues (500 human-validated instances)
	49	+ metric: resolved
	50	+ unit: "%"
	51	+ website: https://www.swebench.com
	52	+ paper: https://arxiv.org/abs/2310.06770
	53	+ known_limitations: Scaffold/agent dependent; results are not comparable across harnesses.
	54	+ source_url: https://www.swebench.com/
	55	+ - key: aider-polyglot
	56	+ name: Aider polyglot
	57	+ aliases: [Aider polyglot coding leaderboard]
	58	+ category: coding
	59	+ task: 225 Exercism exercises in 6 languages, edit-format aware
	60	+ metric: pass rate (2 attempts)
	61	+ unit: "%"
	62	+ website: https://aider.chat/docs/leaderboards/
	63	+ known_limitations: Depends on aider's edit format and prompting; cost column depends on provider pricing.
	64	+ source_url: https://aider.chat/docs/leaderboards/
	65	+ - key: livebench
	66	+ name: LiveBench
	67	+ category: general
	68	+ task: contamination-limited, monthly refreshed questions across 6 categories
	69	+ metric: average score
	70	+ unit: "%"
	71	+ website: https://livebench.ai
	72	+ paper: https://arxiv.org/abs/2406.19314
	73	+ source_url: https://livebench.ai/
	74	+ - key: math-500
	75	+ name: MATH-500
	76	+ aliases: [MATH]
	77	+ category: math
	78	+ task: competition mathematics
	79	+ metric: accuracy
	80	+ unit: "%"
	81	+ paper: https://arxiv.org/abs/2103.03874
	82	+ source_url: https://arxiv.org/abs/2103.03874
	83	+ - key: aime-2025
	84	+ name: AIME 2025
	85	+ aliases: [AIME]
	86	+ category: math
	87	+ task: American Invitational Mathematics Examination problems
	88	+ metric: accuracy
	89	+ unit: "%"
	90	+ known_limitations: 30 problems per year; high variance; often reported with majority voting.
	91	+ source_url: https://maa.org/maa-invitational-competitions/
	92	+ - key: arc-agi
	93	+ name: ARC-AGI
	94	+ aliases: [ARC-AGI-1, ARC-AGI-2, Abstraction and Reasoning Corpus]
	95	+ category: reasoning
	96	+ task: novel visual abstraction puzzles
	97	+ metric: accuracy
	98	+ unit: "%"
	99	+ website: https://arcprize.org
	100	+ source_url: https://arcprize.org/
	101	+ - key: humanitys-last-exam
	102	+ name: Humanity's Last Exam
	103	+ aliases: [HLE]
	104	+ category: knowledge
	105	+ task: expert-written frontier questions
	106	+ metric: accuracy
	107	+ unit: "%"
	108	+ website: https://lastexam.ai
	109	+ paper: https://arxiv.org/abs/2501.14249
	110	+ source_url: https://lastexam.ai/
	111	+ - key: lmarena-text
	112	+ name: LMArena text leaderboard
	113	+ aliases: [Chatbot Arena, LMSYS Chatbot Arena, Arena Elo]
	114	+ category: preference
	115	+ task: crowdsourced pairwise human preference
	116	+ metric: Elo / Bradley–Terry score
	117	+ unit: ""
	118	+ website: https://lmarena.ai
	119	+ known_limitations: Style bias; sampling of prompts by users.
	120	+ source_url: https://lmarena.ai/
	121	+ - key: mmmu
	122	+ name: MMMU
	123	+ category: multimodal
	124	+ task: college-level multimodal understanding
	125	+ metric: accuracy
	126	+ unit: "%"
	127	+ website: https://mmmu-benchmark.github.io
	128	+ paper: https://arxiv.org/abs/2311.16502
	129	+ source_url: https://arxiv.org/abs/2311.16502
	130	+ - key: tau-bench
	131	+ name: τ-bench
	132	+ aliases: [tau-bench, TAU-bench]
	133	+ category: agentic
	134	+ task: tool-agent-user interaction in retail/airline domains
	135	+ metric: pass^1
	136	+ unit: "%"
	137	+ paper: https://arxiv.org/abs/2406.12045
	138	+ source_url: https://arxiv.org/abs/2406.12045
	139	+ - key: terminal-bench
	140	+ name: Terminal-Bench
	141	+ category: agentic
	142	+ task: terminal tasks solved by agents
	143	+ metric: accuracy
	144	+ unit: "%"
	145	+ website: https://www.tbench.ai
	146	+ source_url: https://www.tbench.ai/
	147	+ - key: ifeval
	148	+ name: IFEval
	149	+ category: instruction-following
	150	+ task: verifiable instruction following
	151	+ metric: prompt-level strict accuracy
	152	+ unit: "%"
	153	+ paper: https://arxiv.org/abs/2311.07911
	154	+ source_url: https://arxiv.org/abs/2311.07911
	155	+ - key: mteb
	156	+ name: MTEB
	157	+ aliases: [Massive Text Embedding Benchmark]
	158	+ category: embeddings
	159	+ task: embedding tasks across retrieval, classification, clustering…
	160	+ metric: mean score
	161	+ unit: ""
	162	+ website: https://huggingface.co/spaces/mteb/leaderboard
	163	+ paper: https://arxiv.org/abs/2210.07316
	164	+ source_url: https://arxiv.org/abs/2210.07316
	165	+ - key: artificial-analysis-intelligence-index
	166	+ name: Artificial Analysis Intelligence Index
	167	+ category: composite
	168	+ task: composite of several evaluations run by Artificial Analysis
	169	+ metric: index
	170	+ unit: ""
	171	+ website: https://artificialanalysis.ai
	172	+ known_limitations: Proprietary composite; methodology versions change.
	173	+ source_url: https://artificialanalysis.ai/methodology

added registry/hardware.d/.gitkeep +0 −0

added registry/hardware.yaml +259 −0

		@@ -0,0 +1,259 @@
	1	+# Curated hardware registry (entities of type `hardware`), used by the hardware compatibility engine.
	2	+# Only manufacturer-published figures; each entry cites the spec page it was read from. Fields not published are omitted.
	3	+# memory_gb may be a list when the product ships with several configurations.
	4	+hardware:
	5	+ # ---------------------------------------------------------------- Apple Silicon (unified memory)
	6	+ - key: apple-m1
	7	+ name: Apple M1
	8	+ manufacturer: apple
	9	+ kind: soc
	10	+ architecture: Apple Silicon (Firestorm/Icestorm)
	11	+ release_date: "2020-11"
	12	+ memory_gb: [8, 16]
	13	+ memory_bandwidth_gbs: 68.25
	14	+ runtimes: [mlx, llama.cpp, coreml]
	15	+ source_url: https://support.apple.com/en-us/111883
	16	+ - key: apple-m1-pro
	17	+ name: Apple M1 Pro
	18	+ manufacturer: apple
	19	+ kind: soc
	20	+ release_date: "2021-10"
	21	+ memory_gb: [16, 32]
	22	+ memory_bandwidth_gbs: 200
	23	+ runtimes: [mlx, llama.cpp, coreml]
	24	+ source_url: https://support.apple.com/en-us/111902
	25	+ - key: apple-m1-max
	26	+ name: Apple M1 Max
	27	+ manufacturer: apple
	28	+ kind: soc
	29	+ release_date: "2021-10"
	30	+ memory_gb: [32, 64]
	31	+ memory_bandwidth_gbs: 400
	32	+ runtimes: [mlx, llama.cpp, coreml]
	33	+ source_url: https://support.apple.com/en-us/111902
	34	+ - key: apple-m1-ultra
	35	+ name: Apple M1 Ultra
	36	+ manufacturer: apple
	37	+ kind: soc
	38	+ release_date: "2022-03"
	39	+ memory_gb: [64, 128]
	40	+ memory_bandwidth_gbs: 800
	41	+ runtimes: [mlx, llama.cpp, coreml]
	42	+ source_url: https://support.apple.com/en-us/111900
	43	+ - key: apple-m2
	44	+ name: Apple M2
	45	+ manufacturer: apple
	46	+ kind: soc
	47	+ release_date: "2022-06"
	48	+ memory_gb: [8, 16, 24]
	49	+ memory_bandwidth_gbs: 100
	50	+ runtimes: [mlx, llama.cpp, coreml]
	51	+ source_url: https://support.apple.com/en-us/111837
	52	+ - key: apple-m2-pro
	53	+ name: Apple M2 Pro
	54	+ manufacturer: apple
	55	+ kind: soc
	56	+ release_date: "2023-01"
	57	+ memory_gb: [16, 32]
	58	+ memory_bandwidth_gbs: 200
	59	+ runtimes: [mlx, llama.cpp, coreml]
	60	+ source_url: https://support.apple.com/en-us/111837
	61	+ - key: apple-m2-max
	62	+ name: Apple M2 Max
	63	+ manufacturer: apple
	64	+ kind: soc
	65	+ release_date: "2023-01"
	66	+ memory_gb: [32, 64, 96]
	67	+ memory_bandwidth_gbs: 400
	68	+ runtimes: [mlx, llama.cpp, coreml]
	69	+ source_url: https://support.apple.com/en-us/111835
	70	+ - key: apple-m2-ultra
	71	+ name: Apple M2 Ultra
	72	+ manufacturer: apple
	73	+ kind: soc
	74	+ release_date: "2023-06"
	75	+ memory_gb: [64, 128, 192]
	76	+ memory_bandwidth_gbs: 800
	77	+ runtimes: [mlx, llama.cpp, coreml]
	78	+ source_url: https://support.apple.com/en-us/111835
	79	+ - key: apple-m3
	80	+ name: Apple M3
	81	+ manufacturer: apple
	82	+ kind: soc
	83	+ release_date: "2023-10"
	84	+ memory_gb: [8, 16, 24]
	85	+ memory_bandwidth_gbs: 100
	86	+ runtimes: [mlx, llama.cpp, coreml]
	87	+ source_url: https://support.apple.com/en-us/117735
	88	+ - key: apple-m3-pro
	89	+ name: Apple M3 Pro
	90	+ manufacturer: apple
	91	+ kind: soc
	92	+ release_date: "2023-10"
	93	+ memory_gb: [18, 36]
	94	+ memory_bandwidth_gbs: 150
	95	+ runtimes: [mlx, llama.cpp, coreml]
	96	+ source_url: https://support.apple.com/en-us/117736
	97	+ - key: apple-m3-max
	98	+ name: Apple M3 Max
	99	+ manufacturer: apple
	100	+ kind: soc
	101	+ release_date: "2023-10"
	102	+ memory_gb: [36, 48, 64, 96, 128]
	103	+ memory_bandwidth_gbs: 400
	104	+ runtimes: [mlx, llama.cpp, coreml]
	105	+ source_url: https://support.apple.com/en-us/117736
	106	+ - key: apple-m3-ultra
	107	+ name: Apple M3 Ultra
	108	+ manufacturer: apple
	109	+ kind: soc
	110	+ release_date: "2025-03"
	111	+ memory_gb: [96, 256, 512]
	112	+ memory_bandwidth_gbs: 819
	113	+ runtimes: [mlx, llama.cpp, coreml]
	114	+ source_url: https://www.apple.com/mac-studio/specs/
	115	+ - key: apple-m4
	116	+ name: Apple M4
	117	+ manufacturer: apple
	118	+ kind: soc
	119	+ release_date: "2024-05"
	120	+ memory_gb: [16, 24, 32]
	121	+ memory_bandwidth_gbs: 120
	122	+ runtimes: [mlx, llama.cpp, coreml]
	123	+ source_url: https://www.apple.com/mac-mini/specs/
	124	+ - key: apple-m4-pro
	125	+ name: Apple M4 Pro
	126	+ manufacturer: apple
	127	+ kind: soc
	128	+ release_date: "2024-10"
	129	+ memory_gb: [24, 48, 64]
	130	+ memory_bandwidth_gbs: 273
	131	+ runtimes: [mlx, llama.cpp, coreml]
	132	+ source_url: https://www.apple.com/mac-mini/specs/
	133	+ - key: apple-m4-max
	134	+ name: Apple M4 Max
	135	+ manufacturer: apple
	136	+ kind: soc
	137	+ release_date: "2024-10"
	138	+ memory_gb: [36, 48, 64, 128]
	139	+ memory_bandwidth_gbs: 546
	140	+ runtimes: [mlx, llama.cpp, coreml]
	141	+ source_url: https://www.apple.com/mac-studio/specs/
	142	+ # ---------------------------------------------------------------- NVIDIA data center
	143	+ - key: nvidia-a100-80gb
	144	+ name: NVIDIA A100 80GB
	145	+ manufacturer: nvidia
	146	+ kind: gpu
	147	+ architecture: Ampere
	148	+ release_date: "2020-11"
	149	+ memory_gb: 80
	150	+ memory_type: HBM2e
	151	+ memory_bandwidth_gbs: 2039
	152	+ tdp_watts: 400
	153	+ runtimes: [cuda, tensorrt-llm, vllm, sglang, llama.cpp]
	154	+ source_url: https://www.nvidia.com/en-us/data-center/a100/
	155	+ - key: nvidia-h100-sxm
	156	+ name: NVIDIA H100 SXM
	157	+ manufacturer: nvidia
	158	+ kind: gpu
	159	+ architecture: Hopper
	160	+ release_date: "2022-09"
	161	+ memory_gb: 80
	162	+ memory_type: HBM3
	163	+ memory_bandwidth_gbs: 3350
	164	+ tdp_watts: 700
	165	+ runtimes: [cuda, tensorrt-llm, vllm, sglang, llama.cpp]
	166	+ source_url: https://www.nvidia.com/en-us/data-center/h100/
	167	+ - key: nvidia-h200
	168	+ name: NVIDIA H200
	169	+ manufacturer: nvidia
	170	+ kind: gpu
	171	+ architecture: Hopper
	172	+ release_date: "2024"
	173	+ memory_gb: 141
	174	+ memory_type: HBM3e
	175	+ memory_bandwidth_gbs: 4800
	176	+ tdp_watts: 700
	177	+ runtimes: [cuda, tensorrt-llm, vllm, sglang, llama.cpp]
	178	+ source_url: https://www.nvidia.com/en-us/data-center/h200/
	179	+ - key: nvidia-b200
	180	+ name: NVIDIA B200
	181	+ manufacturer: nvidia
	182	+ kind: gpu
	183	+ architecture: Blackwell
	184	+ release_date: "2024-03"
	185	+ memory_gb: 180
	186	+ memory_type: HBM3e
	187	+ runtimes: [cuda, tensorrt-llm, vllm, sglang]
	188	+ source_url: https://www.nvidia.com/en-us/data-center/dgx-b200/
	189	+ - key: nvidia-rtx-4090
	190	+ name: NVIDIA GeForce RTX 4090
	191	+ manufacturer: nvidia
	192	+ kind: gpu
	193	+ architecture: Ada Lovelace
	194	+ release_date: "2022-10"
	195	+ memory_gb: 24
	196	+ memory_type: GDDR6X
	197	+ memory_bandwidth_gbs: 1008
	198	+ tdp_watts: 450
	199	+ runtimes: [cuda, vllm, llama.cpp, exllama]
	200	+ source_url: https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/
	201	+ - key: nvidia-rtx-5090
	202	+ name: NVIDIA GeForce RTX 5090
	203	+ manufacturer: nvidia
	204	+ kind: gpu
	205	+ architecture: Blackwell
	206	+ release_date: "2025-01"
	207	+ memory_gb: 32
	208	+ memory_type: GDDR7
	209	+ memory_bandwidth_gbs: 1792
	210	+ tdp_watts: 575
	211	+ runtimes: [cuda, vllm, llama.cpp, exllama]
	212	+ source_url: https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
	213	+ - key: nvidia-rtx-3090
	214	+ name: NVIDIA GeForce RTX 3090
	215	+ manufacturer: nvidia
	216	+ kind: gpu
	217	+ architecture: Ampere
	218	+ release_date: "2020-09"
	219	+ memory_gb: 24
	220	+ memory_type: GDDR6X
	221	+ memory_bandwidth_gbs: 936
	222	+ tdp_watts: 350
	223	+ runtimes: [cuda, vllm, llama.cpp, exllama]
	224	+ source_url: https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/
	225	+ - key: nvidia-dgx-spark
	226	+ name: NVIDIA DGX Spark
	227	+ manufacturer: nvidia
	228	+ kind: system
	229	+ architecture: Grace Blackwell (GB10)
	230	+ release_date: "2025"
	231	+ memory_gb: 128
	232	+ memory_type: LPDDR5x
	233	+ memory_bandwidth_gbs: 273
	234	+ runtimes: [cuda, tensorrt-llm, vllm, llama.cpp]
	235	+ source_url: https://www.nvidia.com/en-us/products/workstations/dgx-spark/
	236	+ # ---------------------------------------------------------------- AMD
	237	+ - key: amd-instinct-mi300x
	238	+ name: AMD Instinct MI300X
	239	+ manufacturer: amd
	240	+ kind: gpu
	241	+ architecture: CDNA 3
	242	+ release_date: "2023-12"
	243	+ memory_gb: 192
	244	+ memory_type: HBM3
	245	+ memory_bandwidth_gbs: 5300
	246	+ tdp_watts: 750
	247	+ runtimes: [rocm, vllm, sglang, llama.cpp]
	248	+ source_url: https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html
	249	+ - key: amd-instinct-mi325x
	250	+ name: AMD Instinct MI325X
	251	+ manufacturer: amd
	252	+ kind: gpu
	253	+ architecture: CDNA 3
	254	+ release_date: "2024-10"
	255	+ memory_gb: 256
	256	+ memory_type: HBM3e
	257	+ memory_bandwidth_gbs: 6000
	258	+ runtimes: [rocm, vllm, sglang]
	259	+ source_url: https://www.amd.com/en/products/accelerators/instinct/mi300/mi325x.html

added registry/organizations.d/.gitkeep +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added registry/organizations.yaml +619 −0

Large diff (619 lines) is not loaded by default to keep the page fast.Load diff

added registry/providers.d/.gitkeep +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added registry/providers.yaml +176 −0

Large diff (176 lines) is not loaded by default to keep the page fast.Load diff

added registry/sources.d/.gitkeep +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added registry/sources.yaml +387 −0

Large diff (387 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/__init__.py +3 −0

Large diff (3 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/cli.py +295 −0

Large diff (295 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/config.py +94 −0

Large diff (94 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/connectors/__init__.py +46 −0

Large diff (46 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/connectors/labs/__init__.py +2 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/connectors/labs/_common.py +153 −0

Large diff (153 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/connectors/labs/anthropic.py +312 −0

Large diff (312 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/db/__init__.py +75 −0

Large diff (75 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/ids.py +93 −0

Large diff (93 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/logging.py +56 −0

Large diff (56 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/registry/__init__.py +103 −0

Large diff (103 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/registry/seed.py +168 −0

Large diff (168 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/schemas/__init__.py +17 −0

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/schemas/extraction.py +174 −0

Large diff (174 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/__init__.py +7 −0

Large diff (7 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/archive.py +74 −0

Large diff (74 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/connector.py +483 −0

Large diff (483 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/__init__.py +12 −0

Large diff (12 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/dates.py +71 −0

Large diff (71 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/feeds.py +76 −0

Large diff (76 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/html.py +236 −0

Large diff (236 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/markdown.py +103 −0

Large diff (103 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/numbers.py +106 −0

Large diff (106 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/extract/sitemap.py +42 −0

Large diff (42 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/facts.py +222 −0

Large diff (222 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/fetch.py +329 −0

Large diff (329 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/resolution.py +169 −0

Large diff (169 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/sdk/writer.py +357 −0

Large diff (357 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/__init__.py +1 −0

Large diff (1 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/backup.py +39 −0

Large diff (39 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/cache.py +99 −0

Large diff (99 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/embeddings.py +83 −0

Large diff (83 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/handlers.py +228 −0

Large diff (228 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/jobs.py +124 −0

Large diff (124 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/llm/__init__.py +8 −0

Large diff (8 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/llm/gateway.py +231 −0

Large diff (231 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/quality.py +72 −0

Large diff (72 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/scheduler.py +119 −0

Large diff (119 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/search.py +156 −0

Large diff (156 lines) is not loaded by default to keep the page fast.Load diff

added src/aiatlas/services/stats.py +53 −0

Large diff (53 lines) is not loaded by default to keep the page fast.Load diff

added tests/__init__.py +0 −0

Large diff (0 lines) is not loaded by default to keep the page fast.Load diff

added tests/conftest.py +47 −0

Large diff (47 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/anthropic/model-deprecations.md +223 −0

Large diff (223 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/anthropic/models-overview.md +110 −0

Large diff (110 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/anthropic/news.html +1 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added tests/fixtures/anthropic/pricing.md +526 −0

Large diff (526 lines) is not loaded by default to keep the page fast.Load diff

added tests/test_anthropic.py +63 −0

Large diff (63 lines) is not loaded by default to keep the page fast.Load diff

added tests/test_extract.py +76 −0

Large diff (77 lines) is not loaded by default to keep the page fast.Load diff