# AI Atlas — Technical architecture

The global intelligence layer for artificial intelligence: first-party connectors → raw historical archive → deterministic and local-LLM extraction → temporal knowledge graph → search / compare / timeline → www.ai-atlas.co + API.

# 1. Principles that shape the code

1. No critical dependence on third-party data APIs. Every source is read directly (HTML, embedded JSON, JSON-LD, RSS/Atom, sitemaps, Markdown docs, raw Git files, public JSON/CSV files). Scrapfly / Firecrawl / a headless browser are optional escalation transports (Fetcher(escalate=True)), never the database.
2. Provenance first. Every claim, relation, price, benchmark result and event stores source_id, snapshot_id, source_url, tier, confidence, extractor and observed_at. Entity pages can always answer "where does this number come from?".
3. History is never disposable. Claims are temporal (valid_from/valid_to, statuses current|superseded|conflicting|retracted), prices and benchmark results are append-only, raw snapshots are content-addressed and kept forever, migrations are forward-only.
4. Deterministic before LLM. Stage 1 (DOM, JSON-LD, tables, Markdown, feeds) runs on every document. The local LLM factory (MacLustr llm-api.io through an OpenAI-compatible gateway) only sees documents flagged needs_llm, through versioned pydantic schemas, with full token accounting.
5. Never fabricate. Missing = missing. Conflicts between sources are stored side by side, flagged and queued for review — never averaged.

# 2. Topology

                     ┌──────────────── aia schedule (PM2 ai-atlas-scheduler) ─────────────────┐
 public AI web ──►   │ connectors (registry) → Fetcher (direct/escalate) → archive (raw+text) │
                     │   → parse → extract (Facts) → FactWriter (resolve, version, diff)      │
                     │   → jobs (llm_extract, embed, reprocess) → worker → LLM gateway         │
                     └──────────────────────────────────────────────────────────────────────────┘
                                                   │  Postgres 17 (+pgvector, pg_trgm) · Redis (cache/locks) · AIA_DATA_DIR
                     ┌──────────────────────────────┴───────────────────────────────────────────┐
                     │ FastAPI /api/v1 (PM2 ai-atlas-api :8321)  ◄──  Next.js 16 (PM2 ai-atlas-web :8320)  ◄── MacLustr Tunnel (BHS64 Caddy)
                     └──────────────────────────────────────────────────────────────────────────┘

Production node: M2M32c (Mac Studio M2, 12 c / 32 GB, dedicated), Postgres 17 + pgvector + Redis via Homebrew, data in ~/ai-atlas-data/ (raw, text, backups, logs). Public route https://www.ai-atlas.co → M2M32c:8320 on the MacLustr Tunnel gateway BHS64. Workers can run on any node with database access (aia worker), which is how the LLM factory scales across the cluster.

# 3. Repository layout

registry/                 curated YAML: sources.yaml (domains, tiers, crawl policy), organizations.yaml (aliases, domains, hf/github orgs),
                          providers.yaml, benchmarks.yaml, hardware.yaml — every entry carries its source_url
migrations/versions/      forward-only SQL (0001_initial.py)
src/aiatlas/
  config.py               settings (AIA_*), ids.py (prefixed ULIDs, slugify, normalize_alias), logging.py, db/ (SQLAlchemy Core + asyncpg)
  sdk/                    connector SDK
    fetch.py              Fetcher: robots.txt, per-domain rate limit, ETag/If-Modified-Since, retries/backoff, escalation chain
    archive.py            content-addressed gzip store (raw/, text/) with dedupe
    extract/              html (selectolax; meta, JSON-LD, embedded JSON, tables, links, text), feeds, sitemap, markdown (front matter, tables),
                          numbers (70B, 128K, $3/1M), dates (partial precision)
    facts.py              EntityRef, Claim, Relation, Event, PriceObs, ResultObs, Target, Facts (+ MATERIAL_PROPERTIES → event types)
    resolution.py         Resolver: identifiers → aliases (org-disambiguated) → slug → create; ambiguity → review queue
    writer.py             FactWriter: temporal claims, attributes/provenance materialisation, conflicts, prices, results, events
    connector.py          BaseConnector.run(): documents, snapshots, hash change detection, structural diff, DOCUMENT_CHANGED,
                          breakage detection, circuit breaker, adaptive interval, follow-up targets, --file overrides, reprocess mode
  connectors/             one package per group (labs/, hub/, research/, code/, providers/, benchmarks/, hardware/); CONNECTORS = [cls]
  schemas/extraction.py   pydantic schemas: ModelPassport, PricingExtraction, CompanyPassport, PaperPassport, BenchmarkResultExtraction, HardwareSpec, …
  services/
    jobs.py               Postgres queue (SKIP LOCKED), handlers.py (llm_extract, embed_entity, reprocess_snapshot, recompute_quality)
    llm/gateway.py        LLMGateway: stage cascade small→medium→large, OpenAI-compatible engine, JSON validation, llm_jobs accounting
    embeddings.py         local embeddings → pgvector · search.py FTS+trigram(+vector) and NL→filters compiler
    scheduler.py          APScheduler: due connectors (Redis lock), worker, hourly stats/quality/embeddings, nightly pg_dump
    quality.py            transparency scores (documented in metric_definitions) · stats.py live counters · backup.py · cache.py
  api/                    FastAPI application (see docs/API.md)
  cli.py                  `aia`: migrate seed connectors run reprocess crawl status stats quality schedule worker api backup llm embed search review
apps/web/                 Next.js 16 site (see docs/FRONTEND.md)
tests/                    pytest with saved fixtures (tests/fixtures/<connector>/…); live tests behind `-m live`
deploy/                   mld manifest (ai-atlas.mld.json), render-manifest.sh, first-run.sh — see docs/DEPLOY.md

# 4. Data model (Postgres)

Identifiers: prefixed ULIDs (model_01J…, company_…, paper_…, snap_…, evt_…). Slugs are stable, readable and unique.

# Temporal rules (FactWriter)

no current claim         → insert current, set attribute, NEW_<TYPE> event when the entity is new
same value               → confirm (observed_at)
different, source ≥ tier → supersede (valid_to = observed), insert current, CHANGE event for material properties
different, worse source  → store as `conflicting`, mark current `conflicted`, review-queue item — never overwrite
soft text (description…) → first statement wins until its own source changes; never an event
metric.* / stats.*       → time series without events

Historical mode = claims filtered by valid_from <= date < coalesce(valid_to, ∞); "did it exist?" = entities.first_seen_at <= date. Diff A→B = new/removed entities by first_seen_at, claims superseded between A and B, price rows opened/closed, results observed.

# 5. Connector lifecycle

discover() returns Targets (URL, doc_type, optional entity hint, needs_llm, key for fixtures). For each target run(): fetch (conditional) → unchanged? stop · changed → archive raw + text → snapshot row (+ structural diff vs previous) → parse() → extract() → FactWriter.write(facts) → follow-up targets → optional llm_extract job. Blocked (401/403/451/robots) → document blocked + review item; 404/410 twice → gone (never deletes entities). Fewer records than expected_min_records → run suspect

• review item. Intervals adapt: ×0.7 on change (≥ min), ×1.5 after 3 unchanged runs (≤ max). Three failures open the circuit 30 min.

Parser improvements: bump parser_version, then aia reprocess <connector> replays the latest stored snapshot of every document — no crawl.

# 6. LLM factory

LLMGateway.extract(task_type, document, schema, stage) → strict JSON validated by a pydantic schema; on schema errors the next stage model is tried (small → medium → large); transport failures do not escalate. Every call is recorded in llm_jobs (tokens, duration, node, status, output). facts_from_llm() maps outputs onto Facts with extractor='llm' and medium/low confidence, so deterministic tier-1 claims always win conflicts. Embeddings use the same gateway (/v1/embeddings, Qwen3-Embedding 1024-d) into pgvector. Without a configured gateway the platform degrades gracefully: deterministic extraction, FTS search, no embeddings.

# 7. Quality, search, events

• quality.score = 100·(0.25 completeness + 0.25 primary-source ratio + 0.20 freshness + 0.15 agreement + 0.15 source diversity), versioned and explained in metric_definitions and /methodology. It measures how well AI Atlas knows an entity, not the entity.
• Search: tsvector (name A, family B, type/description C) + pg_trgm similarity + aliases (+ cosine on embeddings when present). compile_query() turns "open models released in 2026 with more than 100B parameters and 128k context" into structured filters.
• Events power the homepage feed, entity timelines, /changes, the daily digest and, later, watchlists/alerts.

# 8. Security & compliance

Admin routes require x-aia-admin-token. The API binds to loopback; Next.js rewrites /api/v1/* to it. No credentials or internal addresses in responses. Crawling honours robots.txt, identifies itself (AIAtlasBot/0.1 … contact@spboucher.ai), rate-limits per domain, uses conditional requests, never bypasses access controls; blocked sources go to the review queue. Raw archive content is never exposed publicly — only extracted facts with links to the original page.

# 9. Backups & durability

Nightly pg_dump -Fc into AIA_DATA_DIR/backups (30 kept), off-node rsync (scripts/backup-offnode.sh), raw archive is append-only and content-addressed (rsync-friendly). The dataset is the product: restore = pg_restore + point DATABASE_URL.