# AI Atlas — project guide (condensed from the founding spec; the full architecture is in docs/ARCHITECTURE.md)

Mission: the most comprehensive, structured, searchable and continuously updated map of the global AI ecosystem — Bloomberg × Wikipedia × Crunchbase × Hugging Face × Papers With Code, purpose-built for AI. Not a tools directory, not a news aggregator, not an API wrapper. The dataset is the product; the website and the API are interfaces to it.

# Non-negotiables

No critical dependence on external data APIs — direct crawling first; Scrapfly/Firecrawl/browser are optional escalation only.
Store provenance (source, snapshot, URL, tier, confidence, extractor) on every fact; store history (temporal claims, append-only prices/results, raw snapshots).
Local LLM (MacLustr llm-api.io via the gateway) for large-scale extraction; deterministic extraction always runs first.
Never fabricate data. Missing → "Unavailable". Conflicts → both claims stored and flagged; never a compromise value.
Primary sources first (tier 1 official > 2 quality secondary > 3 community > 4 unverified). Connectors are owned IP with fixture tests.
Live counters, stats, feeds and timelines come from the database — never hardcoded.
Public information stays public (SEO); login only for watchlists/alerts/API keys later.
Respectful crawling: robots.txt, per-domain rate limits, conditional requests, identified UA, no bypassing access controls, no private data.

# Stack

Backend src/aiatlas/ — Python 3.12, FastAPI, SQLAlchemy Core + asyncpg, Alembic (SQL, forward-only), Redis, APScheduler, selectolax, feedparser, pydantic. CLI aia (.venv/bin/aia --help). Connector SDK in sdk/ (docs/CONNECTORS.md). API contract in docs/API.md.
Database: Postgres 17 + pgvector + pg_trgm. Internal ids = prefixed ULIDs (model_…); slugs stable and unique.
Frontend apps/web/ — Next.js 16 (App Router, React 19, TypeScript strict, Tailwind v4). Guide: docs/FRONTEND.md. Dev on :8320, API on :8321.
Data outside the repo: AIA_DATA_DIR (./data in dev, ~/ai-atlas-data in prod): raw/, text/, backups/, logs/, cache/, seed/.

# Local development

bash

uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev]"
cp .env.example .env          # local Postgres `aiatlas` (role aiatlas/aiatlas), Redis db 5
.venv/bin/aia migrate && .venv/bin/aia seed
.venv/bin/aia run anthropic   # one connector, live · `--file key=path` uses a fixture · `aia crawl --priority 0` runs all P0 connectors
.venv/bin/aia api --reload    # http://127.0.0.1:8321/api/v1/docs
pnpm install && pnpm dev:web  # http://localhost:8320
.venv/bin/pytest -q           # fixtures only; `-m live` hits the network

# Production (MacLustr)

Node M2M32c (dedicated), deployed by mld — manifest deploy/ai-atlas.mld.json → M1M32:~/dispatch/apps/ai-atlas.json. PM2 processes ai-atlas-api (uvicorn 127.0.0.1:8321), ai-atlas-scheduler (aia schedule), ai-atlas-web (next start :8320). Public route through the MacLustr Tunnel: https://www.ai-atlas.co → M2M32c:8320. See docs/DEPLOY.md. Secrets live only in the rendered manifest (deploy/.admin-token, deploy/.llm-key, git-ignored). LLM factory: AIA_LLM_BASE_URL=https://www.llm-api.io/v1 (private MacLustr server on M1M64).

# Conventions

Properties and relations use the shared vocabulary in docs/CONNECTORS.md; metrics use the metric. prefix (no events).
Every connector: source_key registered in registry/sources.yaml, organization in registry/organizations.yaml, fixture test in tests/.
Parser improved? bump parser_version and aia reprocess <connector> — never re-crawl for a code change.
Frontend: mobile-first (390/430/768/1440), DOM order = visual order, ≥ 44 px targets, no horizontal overflow, dark + light, premium institutional design (no generic SaaS cards). Every page: real data, loading + error + empty states, source attribution, generateMetadata.
Units: tokens, USD per 1M tokens, GB, GB/s, W, ISO dates (UTC). Openness vocabulary: open-weights | open-source | proprietary | restricted.