# AI Atlas — Technical architecture

> The global intelligence layer for artificial intelligence: first-party connectors → raw historical archive → deterministic and
> local-LLM extraction → temporal knowledge graph → search / compare / timeline → www.ai-atlas.co + API.

## 1. Principles that shape the code

1. **No critical dependence on third-party data APIs.** Every source is read directly (HTML, embedded JSON, JSON-LD, RSS/Atom, sitemaps,
   Markdown docs, raw Git files, public JSON/CSV files). Scrapfly / Firecrawl / a headless browser are *optional escalation transports*
   (`Fetcher(escalate=True)`), never the database.
2. **Provenance first.** Every claim, relation, price, benchmark result and event stores `source_id`, `snapshot_id`, `source_url`, `tier`,
   `confidence`, `extractor` and `observed_at`. Entity pages can always answer "where does this number come from?".
3. **History is never disposable.** Claims are temporal (`valid_from`/`valid_to`, statuses `current|superseded|conflicting|retracted`),
   prices and benchmark results are append-only, raw snapshots are content-addressed and kept forever, migrations are forward-only.
4. **Deterministic before LLM.** Stage 1 (DOM, JSON-LD, tables, Markdown, feeds) runs on every document. The local LLM factory
   (MacLustr llm-api.io through an OpenAI-compatible gateway) only sees documents flagged `needs_llm`, through versioned pydantic schemas,
   with full token accounting.
5. **Never fabricate.** Missing = missing. Conflicts between sources are stored side by side, flagged and queued for review — never averaged.

## 2. Topology

```
                     ┌──────────────── aia schedule (PM2 ai-atlas-scheduler) ─────────────────┐
 public AI web ──►   │ connectors (registry) → Fetcher (direct/escalate) → archive (raw+text) │
                     │   → parse → extract (Facts) → FactWriter (resolve, version, diff)      │
                     │   → jobs (llm_extract, embed, reprocess) → worker → LLM gateway         │
                     └──────────────────────────────────────────────────────────────────────────┘
                                                   │  Postgres 17 (+pgvector, pg_trgm) · Redis (cache/locks) · AIA_DATA_DIR
                     ┌──────────────────────────────┴───────────────────────────────────────────┐
                     │ FastAPI /api/v1 (PM2 ai-atlas-api :8321)  ◄──  Next.js 16 (PM2 ai-atlas-web :8320)  ◄── MacLustr Tunnel (BHS64 Caddy)
                     └──────────────────────────────────────────────────────────────────────────┘
```

Production node: **M2M32c** (Mac Studio M2, 12 c / 32 GB, dedicated), Postgres 17 + pgvector + Redis via Homebrew, data in
`~/ai-atlas-data/` (raw, text, backups, logs). Public route `https://www.ai-atlas.co → M2M32c:8320` on the MacLustr Tunnel gateway BHS64.
Workers can run on any node with database access (`aia worker`), which is how the LLM factory scales across the cluster.

## 3. Repository layout

```
registry/                 curated YAML: sources.yaml (domains, tiers, crawl policy), organizations.yaml (aliases, domains, hf/github orgs),
                          providers.yaml, benchmarks.yaml, hardware.yaml — every entry carries its source_url
migrations/versions/      forward-only SQL (0001_initial.py)
src/aiatlas/
  config.py               settings (AIA_*), ids.py (prefixed ULIDs, slugify, normalize_alias), logging.py, db/ (SQLAlchemy Core + asyncpg)
  sdk/                    connector SDK
    fetch.py              Fetcher: robots.txt, per-domain rate limit, ETag/If-Modified-Since, retries/backoff, escalation chain
    archive.py            content-addressed gzip store (raw/, text/) with dedupe
    extract/              html (selectolax; meta, JSON-LD, embedded JSON, tables, links, text), feeds, sitemap, markdown (front matter, tables),
                          numbers (70B, 128K, $3/1M), dates (partial precision)
    facts.py              EntityRef, Claim, Relation, Event, PriceObs, ResultObs, Target, Facts (+ MATERIAL_PROPERTIES → event types)
    resolution.py         Resolver: identifiers → aliases (org-disambiguated) → slug → create; ambiguity → review queue
    writer.py             FactWriter: temporal claims, attributes/provenance materialisation, conflicts, prices, results, events
    connector.py          BaseConnector.run(): documents, snapshots, hash change detection, structural diff, DOCUMENT_CHANGED,
                          breakage detection, circuit breaker, adaptive interval, follow-up targets, --file overrides, reprocess mode
  connectors/             one package per group (labs/, hub/, research/, code/, providers/, benchmarks/, hardware/); CONNECTORS = [cls]
  schemas/extraction.py   pydantic schemas: ModelPassport, PricingExtraction, CompanyPassport, PaperPassport, BenchmarkResultExtraction, HardwareSpec, …
  services/
    jobs.py               Postgres queue (SKIP LOCKED), handlers.py (llm_extract, embed_entity, reprocess_snapshot, recompute_quality)
    llm/gateway.py        LLMGateway: stage cascade small→medium→large, OpenAI-compatible engine, JSON validation, llm_jobs accounting
    embeddings.py         local embeddings → pgvector · search.py FTS+trigram(+vector) and NL→filters compiler
    scheduler.py          APScheduler: due connectors (Redis lock), worker, hourly stats/quality/embeddings, nightly pg_dump
    quality.py            transparency scores (documented in metric_definitions) · stats.py live counters · backup.py · cache.py
  api/                    FastAPI application (see docs/API.md)
  cli.py                  `aia`: migrate seed connectors run reprocess crawl status stats quality schedule worker api backup llm embed search review
apps/web/                 Next.js 16 site (see docs/FRONTEND.md)
tests/                    pytest with saved fixtures (tests/fixtures/<connector>/…); live tests behind `-m live`
deploy/                   mld manifest (ai-atlas.mld.json), render-manifest.sh, first-run.sh — see docs/DEPLOY.md
```

## 4. Data model (Postgres)

| table | role |
|---|---|
| `entities` | unified graph node: `entity_type`, `canonical_name`, `slug`, `organization_id`, `attributes` (current value per property), `provenance` (per property), `quality`, `counts`, `first_seen_at`, `search` tsvector |
| `entity_aliases`, `entity_identifiers` | resolution keys (normalized aliases; `(scheme, value)` unique: hf_repo, github_repo, arxiv, doi, anthropic_model_id, openrouter, pypi, domain…) |
| `sources`, `connectors`, `connector_runs`, `connector_errors` | source registry with tiers; connector state (health, adaptive interval, circuit, parser_version, expected_min_records) |
| `documents` | one row per URL: validators (etag/last-modified), `content_hash`, counters, `entity_id`, `needs_llm` |
| `snapshots` | one row per *changed* fetch: `raw_path`, `text_path`, `structured` (deterministic summary), `diff`, `transport`, `parser_version`, `processing_status` |
| `claims` | temporal facts: `(entity_id, property, value jsonb, unit, tier, confidence, status, valid_from, valid_to, snapshot_id, source_url, extractor)` |
| `relations` | typed edges with validity: `develops, owns, operates, available_through, evaluated_on, described_by, derived_from, fine_tuned_from, quantized_from, superseded_by, runs_on, uses, manufactures, funded_by, acquired, authored, works_at, uses_dataset, evaluates_on, integrates` |
| `change_events` | the event engine: `NEW_MODEL, MODEL_UPDATED, PRICE_CHANGED, PROVIDER_LISTED, CONTEXT_CHANGED, STATUS_CHANGED, LICENSE_CHANGED, DEPRECATION_ANNOUNCED, RETIREMENT_ANNOUNCED, BENCHMARK_RESULT, BENCHMARK_UPDATED, NEW_PAPER, ANNOUNCEMENT, RELEASE, VERSION_RELEASED, DOCUMENT_CHANGED, …` with `category`, `importance 0–3`, `dedupe_key` |
| `prices` | append-only provider pricing (`valid_to` closes a row): per-1M-token input/output/cached/cache-write/batch, per image/request, context, features |
| `benchmark_results` | append-only results with `config` (harness, prompting, judge…) — never compared blindly |
| `jobs`, `llm_jobs` | work queue; LLM cost accounting (stage, model, node, tokens, duration, status) |
| `review_queue` | merge candidates, conflicts, blocked sources, parser breakage, unusual changes |
| `entity_embeddings` | pgvector(1024) local embeddings |
| `domains`, `stats_snapshots`, `page_views`, `api_keys`, `metric_definitions` | trust graph, counters history, trending, developer keys, documented metrics |

Identifiers: prefixed ULIDs (`model_01J…`, `company_…`, `paper_…`, `snap_…`, `evt_…`). Slugs are stable, readable and unique.

### Temporal rules (FactWriter)

```
no current claim         → insert current, set attribute, NEW_<TYPE> event when the entity is new
same value               → confirm (observed_at)
different, source ≥ tier → supersede (valid_to = observed), insert current, CHANGE event for material properties
different, worse source  → store as `conflicting`, mark current `conflicted`, review-queue item — never overwrite
soft text (description…) → first statement wins until its own source changes; never an event
metric.* / stats.*       → time series without events
```

Historical mode = `claims` filtered by `valid_from <= date < coalesce(valid_to, ∞)`; "did it exist?" = `entities.first_seen_at <= date`.
Diff A→B = new/removed entities by `first_seen_at`, claims superseded between A and B, price rows opened/closed, results observed.

## 5. Connector lifecycle

`discover()` returns `Target`s (URL, doc_type, optional entity hint, `needs_llm`, `key` for fixtures). For each target `run()`:
fetch (conditional) → unchanged? stop · changed → archive raw + text → snapshot row (+ structural diff vs previous) → `parse()` →
`extract()` → `FactWriter.write(facts)` → follow-up targets → optional `llm_extract` job. Blocked (401/403/451/robots) → document
`blocked` + review item; 404/410 twice → `gone` (never deletes entities). Fewer records than `expected_min_records` → run `suspect`
+ review item. Intervals adapt: ×0.7 on change (≥ min), ×1.5 after 3 unchanged runs (≤ max). Three failures open the circuit 30 min.

Parser improvements: bump `parser_version`, then `aia reprocess <connector>` replays the latest stored snapshot of every document — no crawl.

## 6. LLM factory

`LLMGateway.extract(task_type, document, schema, stage)` → strict JSON validated by a pydantic schema; on schema errors the next stage
model is tried (small → medium → large); transport failures do not escalate. Every call is recorded in `llm_jobs` (tokens, duration,
node, status, output). `facts_from_llm()` maps outputs onto `Facts` with `extractor='llm'` and medium/low confidence, so deterministic
tier-1 claims always win conflicts. Embeddings use the same gateway (`/v1/embeddings`, Qwen3-Embedding 1024-d) into pgvector.
Without a configured gateway the platform degrades gracefully: deterministic extraction, FTS search, no embeddings.

## 7. Quality, search, events

* `quality.score` = 100·(0.25 completeness + 0.25 primary-source ratio + 0.20 freshness + 0.15 agreement + 0.15 source diversity),
  versioned and explained in `metric_definitions` and `/methodology`. It measures how well AI Atlas knows an entity, not the entity.
* Search: tsvector (name A, family B, type/description C) + `pg_trgm` similarity + aliases (+ cosine on embeddings when present).
  `compile_query()` turns "open models released in 2026 with more than 100B parameters and 128k context" into structured filters.
* Events power the homepage feed, entity timelines, `/changes`, the daily digest and, later, watchlists/alerts.

## 8. Security & compliance

Admin routes require `x-aia-admin-token`. The API binds to loopback; Next.js rewrites `/api/v1/*` to it. No credentials or internal
addresses in responses. Crawling honours robots.txt, identifies itself (`AIAtlasBot/0.1 … contact@spboucher.ai`), rate-limits per domain,
uses conditional requests, never bypasses access controls; blocked sources go to the review queue. Raw archive content is never exposed
publicly — only extracted facts with links to the original page.

## 9. Backups & durability

Nightly `pg_dump -Fc` into `AIA_DATA_DIR/backups` (30 kept), off-node rsync (`scripts/backup-offnode.sh`), raw archive is append-only and
content-addressed (rsync-friendly). The dataset is the product: restore = `pg_restore` + point `DATABASE_URL`.