# AI Atlas — project guide (condensed from the founding spec; the full architecture is in docs/ARCHITECTURE.md)

**Mission**: the most comprehensive, structured, searchable and continuously updated map of the global AI ecosystem — Bloomberg × Wikipedia ×
Crunchbase × Hugging Face × Papers With Code, purpose-built for AI. Not a tools directory, not a news aggregator, not an API wrapper.
**The dataset is the product; the website and the API are interfaces to it.**

## Non-negotiables

1. No critical dependence on external data APIs — direct crawling first; Scrapfly/Firecrawl/browser are optional escalation only.
2. Store provenance (source, snapshot, URL, tier, confidence, extractor) on every fact; store history (temporal claims, append-only prices/results, raw snapshots).
3. Local LLM (MacLustr llm-api.io via the gateway) for large-scale extraction; deterministic extraction always runs first.
4. **Never fabricate data.** Missing → "Unavailable". Conflicts → both claims stored and flagged; never a compromise value.
5. Primary sources first (tier 1 official > 2 quality secondary > 3 community > 4 unverified). Connectors are owned IP with fixture tests.
6. Live counters, stats, feeds and timelines come from the database — never hardcoded.
7. Public information stays public (SEO); login only for watchlists/alerts/API keys later.
8. Respectful crawling: robots.txt, per-domain rate limits, conditional requests, identified UA, no bypassing access controls, no private data.

## Stack

- Backend `src/aiatlas/` — Python 3.12, FastAPI, SQLAlchemy Core + asyncpg, Alembic (SQL, forward-only), Redis, APScheduler, selectolax, feedparser, pydantic.
  CLI `aia` (`.venv/bin/aia --help`). Connector SDK in `sdk/` (docs/CONNECTORS.md). API contract in docs/API.md.
- Database: Postgres 17 + pgvector + pg_trgm. Internal ids = prefixed ULIDs (`model_…`); slugs stable and unique.
- Frontend `apps/web/` — Next.js 16 (App Router, React 19, TypeScript strict, Tailwind v4). Guide: docs/FRONTEND.md. Dev on :8320, API on :8321.
- Data outside the repo: `AIA_DATA_DIR` (`./data` in dev, `~/ai-atlas-data` in prod): raw/, text/, backups/, logs/, cache/, seed/.

## Local development

```bash
uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev]"
cp .env.example .env          # local Postgres `aiatlas` (role aiatlas/aiatlas), Redis db 5
.venv/bin/aia migrate && .venv/bin/aia seed
.venv/bin/aia run anthropic   # one connector, live · `--file key=path` uses a fixture · `aia crawl --priority 0` runs all P0 connectors
.venv/bin/aia api --reload    # http://127.0.0.1:8321/api/v1/docs
pnpm install && pnpm dev:web  # http://localhost:8320
.venv/bin/pytest -q           # fixtures only; `-m live` hits the network
```

## Production (MacLustr)

Node **M2M32c** (dedicated), deployed by `mld` — manifest `deploy/ai-atlas.mld.json` → `M1M32:~/dispatch/apps/ai-atlas.json`. PM2 processes
`ai-atlas-api` (uvicorn 127.0.0.1:8321), `ai-atlas-scheduler` (`aia schedule`), `ai-atlas-web` (next start :8320). Public route through the
MacLustr Tunnel: `https://www.ai-atlas.co → M2M32c:8320`. See docs/DEPLOY.md. Secrets live only in the rendered manifest (`deploy/.admin-token`,
`deploy/.llm-key`, git-ignored). LLM factory: `AIA_LLM_BASE_URL=https://www.llm-api.io/v1` (private MacLustr server on M1M64).

## Conventions

- Properties and relations use the shared vocabulary in docs/CONNECTORS.md; metrics use the `metric.` prefix (no events).
- Every connector: `source_key` registered in `registry/sources.yaml`, organization in `registry/organizations.yaml`, fixture test in `tests/`.
- Parser improved? bump `parser_version` and `aia reprocess <connector>` — never re-crawl for a code change.
- Frontend: mobile-first (390/430/768/1440), DOM order = visual order, ≥ 44 px targets, no horizontal overflow, dark + light, premium
  institutional design (no generic SaaS cards). Every page: real data, loading + error + empty states, source attribution, `generateMetadata`.
- Units: tokens, USD per 1M tokens, GB, GB/s, W, ISO dates (UTC). Openness vocabulary: open-weights | open-source | proprietary | restricted.