HTML 77.2%
TypeScript 10.5%
Python 9.6%
JavaScript 2.5%
1# AI Atlas — Technical architecture23> The global intelligence layer for artificial intelligence: first-party connectors → raw historical archive → deterministic and4> local-LLM extraction → temporal knowledge graph → search / compare / timeline → www.ai-atlas.co + API.56## 1. Principles that shape the code781. **No critical dependence on third-party data APIs.** Every source is read directly (HTML, embedded JSON, JSON-LD, RSS/Atom, sitemaps,9 Markdown docs, raw Git files, public JSON/CSV files). Scrapfly / Firecrawl / a headless browser are *optional escalation transports*10 (`Fetcher(escalate=True)`), never the database.112. **Provenance first.** Every claim, relation, price, benchmark result and event stores `source_id`, `snapshot_id`, `source_url`, `tier`,12 `confidence`, `extractor` and `observed_at`. Entity pages can always answer "where does this number come from?".133. **History is never disposable.** Claims are temporal (`valid_from`/`valid_to`, statuses `current|superseded|conflicting|retracted`),14 prices and benchmark results are append-only, raw snapshots are content-addressed and kept forever, migrations are forward-only.154. **Deterministic before LLM.** Stage 1 (DOM, JSON-LD, tables, Markdown, feeds) runs on every document. The local LLM factory16 (MacLustr llm-api.io through an OpenAI-compatible gateway) only sees documents flagged `needs_llm`, through versioned pydantic schemas,17 with full token accounting.185. **Never fabricate.** Missing = missing. Conflicts between sources are stored side by side, flagged and queued for review — never averaged.1920## 2. Topology2122```23 ┌──────────────── aia schedule (PM2 ai-atlas-scheduler) ─────────────────┐24 public AI web ──► │ connectors (registry) → Fetcher (direct/escalate) → archive (raw+text) │25 │ → parse → extract (Facts) → FactWriter (resolve, version, diff) │26 │ → jobs (llm_extract, embed, reprocess) → worker → LLM gateway │27 └──────────────────────────────────────────────────────────────────────────┘28 │ Postgres 17 (+pgvector, pg_trgm) · Redis (cache/locks) · AIA_DATA_DIR29 ┌──────────────────────────────┴───────────────────────────────────────────┐30 │ FastAPI /api/v1 (PM2 ai-atlas-api :8321) ◄── Next.js 16 (PM2 ai-atlas-web :8320) ◄── MacLustr Tunnel (BHS64 Caddy)31 └──────────────────────────────────────────────────────────────────────────┘32```3334Production node: **M2M32c** (Mac Studio M2, 12 c / 32 GB, dedicated), Postgres 17 + pgvector + Redis via Homebrew, data in35`~/ai-atlas-data/` (raw, text, backups, logs). Public route `https://www.ai-atlas.co → M2M32c:8320` on the MacLustr Tunnel gateway BHS64.36Workers can run on any node with database access (`aia worker`), which is how the LLM factory scales across the cluster.3738## 3. Repository layout3940```41registry/ curated YAML: sources.yaml (domains, tiers, crawl policy), organizations.yaml (aliases, domains, hf/github orgs),42 providers.yaml, benchmarks.yaml, hardware.yaml — every entry carries its source_url43migrations/versions/ forward-only SQL (0001_initial.py)44src/aiatlas/45 config.py settings (AIA_*), ids.py (prefixed ULIDs, slugify, normalize_alias), logging.py, db/ (SQLAlchemy Core + asyncpg)46 sdk/ connector SDK47 fetch.py Fetcher: robots.txt, per-domain rate limit, ETag/If-Modified-Since, retries/backoff, escalation chain48 archive.py content-addressed gzip store (raw/, text/) with dedupe49 extract/ html (selectolax; meta, JSON-LD, embedded JSON, tables, links, text), feeds, sitemap, markdown (front matter, tables),50 numbers (70B, 128K, $3/1M), dates (partial precision)51 facts.py EntityRef, Claim, Relation, Event, PriceObs, ResultObs, Target, Facts (+ MATERIAL_PROPERTIES → event types)52 resolution.py Resolver: identifiers → aliases (org-disambiguated) → slug → create; ambiguity → review queue53 writer.py FactWriter: temporal claims, attributes/provenance materialisation, conflicts, prices, results, events54 connector.py BaseConnector.run(): documents, snapshots, hash change detection, structural diff, DOCUMENT_CHANGED,55 breakage detection, circuit breaker, adaptive interval, follow-up targets, --file overrides, reprocess mode56 connectors/ one package per group (labs/, hub/, research/, code/, providers/, benchmarks/, hardware/); CONNECTORS = [cls]57 schemas/extraction.py pydantic schemas: ModelPassport, PricingExtraction, CompanyPassport, PaperPassport, BenchmarkResultExtraction, HardwareSpec, …58 services/59 jobs.py Postgres queue (SKIP LOCKED), handlers.py (llm_extract, embed_entity, reprocess_snapshot, recompute_quality)60 llm/gateway.py LLMGateway: stage cascade small→medium→large, OpenAI-compatible engine, JSON validation, llm_jobs accounting61 embeddings.py local embeddings → pgvector · search.py FTS+trigram(+vector) and NL→filters compiler62 scheduler.py APScheduler: due connectors (Redis lock), worker, hourly stats/quality/embeddings, nightly pg_dump63 quality.py transparency scores (documented in metric_definitions) · stats.py live counters · backup.py · cache.py64 api/ FastAPI application (see docs/API.md)65 cli.py `aia`: migrate seed connectors run reprocess crawl status stats quality schedule worker api backup llm embed search review66apps/web/ Next.js 16 site (see docs/FRONTEND.md)67tests/ pytest with saved fixtures (tests/fixtures/<connector>/…); live tests behind `-m live`68deploy/ mld manifest (ai-atlas.mld.json), render-manifest.sh, first-run.sh — see docs/DEPLOY.md69```7071## 4. Data model (Postgres)7273| table | role |74|---|---|75| `entities` | unified graph node: `entity_type`, `canonical_name`, `slug`, `organization_id`, `attributes` (current value per property), `provenance` (per property), `quality`, `counts`, `first_seen_at`, `search` tsvector |76| `entity_aliases`, `entity_identifiers` | resolution keys (normalized aliases; `(scheme, value)` unique: hf_repo, github_repo, arxiv, doi, anthropic_model_id, openrouter, pypi, domain…) |77| `sources`, `connectors`, `connector_runs`, `connector_errors` | source registry with tiers; connector state (health, adaptive interval, circuit, parser_version, expected_min_records) |78| `documents` | one row per URL: validators (etag/last-modified), `content_hash`, counters, `entity_id`, `needs_llm` |79| `snapshots` | one row per *changed* fetch: `raw_path`, `text_path`, `structured` (deterministic summary), `diff`, `transport`, `parser_version`, `processing_status` |80| `claims` | temporal facts: `(entity_id, property, value jsonb, unit, tier, confidence, status, valid_from, valid_to, snapshot_id, source_url, extractor)` |81| `relations` | typed edges with validity: `develops, owns, operates, available_through, evaluated_on, described_by, derived_from, fine_tuned_from, quantized_from, superseded_by, runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates` |82| `change_events` | the event engine: `NEW_MODEL, MODEL_UPDATED, PRICE_CHANGED, PROVIDER_LISTED, CONTEXT_CHANGED, STATUS_CHANGED, LICENSE_CHANGED, DEPRECATION_ANNOUNCED, RETIREMENT_ANNOUNCED, BENCHMARK_RESULT, BENCHMARK_UPDATED, NEW_PAPER, ANNOUNCEMENT, RELEASE, VERSION_RELEASED, DOCUMENT_CHANGED, …` with `category`, `importance 0–3`, `dedupe_key` |83| `prices` | append-only provider pricing (`valid_to` closes a row): per-1M-token input/output/cached/cache-write/batch, per image/request, context, features |84| `benchmark_results` | append-only results with `config` (harness, prompting, judge…) — never compared blindly |85| `jobs`, `llm_jobs` | work queue; LLM cost accounting (stage, model, node, tokens, duration, status) |86| `review_queue` | merge candidates, conflicts, blocked sources, parser breakage, unusual changes |87| `entity_embeddings` | pgvector(1024) local embeddings |88| `domains`, `stats_snapshots`, `page_views`, `api_keys`, `metric_definitions` | trust graph, counters history, trending, developer keys, documented metrics |8990Identifiers: prefixed ULIDs (`model_01J…`, `company_…`, `paper_…`, `snap_…`, `evt_…`). Slugs are stable, readable and unique.9192### Temporal rules (FactWriter)9394```95no current claim → insert current, set attribute, NEW_<TYPE> event when the entity is new96same value → confirm (observed_at)97different, source ≥ tier → supersede (valid_to = observed), insert current, CHANGE event for material properties98different, worse source → store as `conflicting`, mark current `conflicted`, review-queue item — never overwrite99soft text (description…) → first statement wins until its own source changes; never an event100metric.* / stats.* → time series without events101```102103Historical mode = `claims` filtered by `valid_from <= date < coalesce(valid_to, ∞)`; "did it exist?" = `entities.first_seen_at <= date`.104Diff A→B = new/removed entities by `first_seen_at`, claims superseded between A and B, price rows opened/closed, results observed.105106## 5. Connector lifecycle107108`discover()` returns `Target`s (URL, doc_type, optional entity hint, `needs_llm`, `key` for fixtures). For each target `run()`:109fetch (conditional) → unchanged? stop · changed → archive raw + text → snapshot row (+ structural diff vs previous) → `parse()` →110`extract()` → `FactWriter.write(facts)` → follow-up targets → optional `llm_extract` job. Blocked (401/403/451/robots) → document111`blocked` + review item; 404/410 twice → `gone` (never deletes entities). Fewer records than `expected_min_records` → run `suspect`112+ review item. Intervals adapt: ×0.7 on change (≥ min), ×1.5 after 3 unchanged runs (≤ max). Three failures open the circuit 30 min.113114Parser improvements: bump `parser_version`, then `aia reprocess <connector>` replays the latest stored snapshot of every document — no crawl.115116## 6. LLM factory117118`LLMGateway.extract(task_type, document, schema, stage)` → strict JSON validated by a pydantic schema; on schema errors the next stage119model is tried (small → medium → large); transport failures do not escalate. Every call is recorded in `llm_jobs` (tokens, duration,120node, status, output). `facts_from_llm()` maps outputs onto `Facts` with `extractor='llm'` and medium/low confidence, so deterministic121tier-1 claims always win conflicts. Embeddings use the same gateway (`/v1/embeddings`, Qwen3-Embedding 1024-d) into pgvector.122Without a configured gateway the platform degrades gracefully: deterministic extraction, FTS search, no embeddings.123124## 7. Quality, search, events125126* `quality.score` = 100·(0.25 completeness + 0.25 primary-source ratio + 0.20 freshness + 0.15 agreement + 0.15 source diversity),127 versioned and explained in `metric_definitions` and `/methodology`. It measures how well AI Atlas knows an entity, not the entity.128* Search: tsvector (name A, family B, type/description C) + `pg_trgm` similarity + aliases (+ cosine on embeddings when present).129 `compile_query()` turns "open models released in 2026 with more than 100B parameters and 128k context" into structured filters.130* Events power the homepage feed, entity timelines, `/changes`, the daily digest and, later, watchlists/alerts.131132## 8. Security & compliance133134Admin routes require `x-aia-admin-token`. The API binds to loopback; Next.js rewrites `/api/v1/*` to it. No credentials or internal135addresses in responses. Crawling honours robots.txt, identifies itself (`AIAtlasBot/0.1 … contact@spboucher.ai`), rate-limits per domain,136uses conditional requests, never bypasses access controls; blocked sources go to the review queue. Raw archive content is never exposed137publicly — only extracted facts with links to the original page.138139## 9. Backups & durability140141Nightly `pg_dump -Fc` into `AIA_DATA_DIR/backups` (30 kept), off-node rsync (`scripts/backup-offnode.sh`), raw archive is append-only and142content-addressed (rsync-friendly). The dataset is the product: restore = `pg_restore` + point `DATABASE_URL`.143