spb/ai-atlas

1# AI Atlas — project guide (condensed from the founding spec; the full architecture is in docs/ARCHITECTURE.md)23**Mission**: the most comprehensive, structured, searchable and continuously updated map of the global AI ecosystem — Bloomberg × Wikipedia ×4Crunchbase × Hugging Face × Papers With Code, purpose-built for AI. Not a tools directory, not a news aggregator, not an API wrapper.5**The dataset is the product; the website and the API are interfaces to it.**67## Non-negotiables891. No critical dependence on external data APIs — direct crawling first; Scrapfly/Firecrawl/browser are optional escalation only.102. Store provenance (source, snapshot, URL, tier, confidence, extractor) on every fact; store history (temporal claims, append-only prices/results, raw snapshots).113. Local LLM (MacLustr llm-api.io via the gateway) for large-scale extraction; deterministic extraction always runs first.124. **Never fabricate data.** Missing → "Unavailable". Conflicts → both claims stored and flagged; never a compromise value.135. Primary sources first (tier 1 official > 2 quality secondary > 3 community > 4 unverified). Connectors are owned IP with fixture tests.146. Live counters, stats, feeds and timelines come from the database — never hardcoded.157. Public information stays public (SEO); login only for watchlists/alerts/API keys later.168. Respectful crawling: robots.txt, per-domain rate limits, conditional requests, identified UA, no bypassing access controls, no private data.1718## Stack1920- Backend `src/aiatlas/` — Python 3.12, FastAPI, SQLAlchemy Core + asyncpg, Alembic (SQL, forward-only), Redis, APScheduler, selectolax, feedparser, pydantic.21  CLI `aia` (`.venv/bin/aia --help`). Connector SDK in `sdk/` (docs/CONNECTORS.md). API contract in docs/API.md.22- Database: Postgres 17 + pgvector + pg_trgm. Internal ids = prefixed ULIDs (`model_…`); slugs stable and unique.23- Frontend `apps/web/` — Next.js 16 (App Router, React 19, TypeScript strict, Tailwind v4). Guide: docs/FRONTEND.md. Dev on :8320, API on :8321.24- Data outside the repo: `AIA_DATA_DIR` (`./data` in dev, `~/ai-atlas-data` in prod): raw/, text/, backups/, logs/, cache/, seed/.2526## Local development2728```bash29uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev]"30cp .env.example .env          # local Postgres `aiatlas` (role aiatlas/aiatlas), Redis db 531.venv/bin/aia migrate && .venv/bin/aia seed32.venv/bin/aia run anthropic   # one connector, live · `--file key=path` uses a fixture · `aia crawl --priority 0` runs all P0 connectors33.venv/bin/aia api --reload    # http://127.0.0.1:8321/api/v1/docs34pnpm install && pnpm dev:web  # http://localhost:832035.venv/bin/pytest -q           # fixtures only; `-m live` hits the network36```3738## Production (MacLustr)3940Node **M2M32c** (dedicated), deployed by `mld` — manifest `deploy/ai-atlas.mld.json` → `M1M32:~/dispatch/apps/ai-atlas.json`. PM2 processes41`ai-atlas-api` (uvicorn 127.0.0.1:8321), `ai-atlas-scheduler` (`aia schedule`), `ai-atlas-web` (next start :8320). Public route through the42MacLustr Tunnel: `https://www.ai-atlas.co → M2M32c:8320`. See docs/DEPLOY.md. Secrets live only in the rendered manifest (`deploy/.admin-token`,43`deploy/.llm-key`, git-ignored). LLM factory: `AIA_LLM_BASE_URL=https://www.llm-api.io/v1` (private MacLustr server on M1M64).4445## Conventions4647- Properties and relations use the shared vocabulary in docs/CONNECTORS.md; metrics use the `metric.` prefix (no events).48- Every connector: `source_key` registered in `registry/sources.yaml`, organization in `registry/organizations.yaml`, fixture test in `tests/`.49- Parser improved? bump `parser_version` and `aia reprocess <connector>` — never re-crawl for a code change.50- Frontend: mobile-first (390/430/768/1440), DOM order = visual order, ≥ 44 px targets, no horizontal overflow, dark + light, premium51  institutional design (no generic SaaS cards). Every page: real data, loading + error + empty states, source attribution, `generateMetadata`.52- Units: tokens, USD per 1M tokens, GB, GB/s, W, ISO dates (UTC). Openness vocabulary: open-weights | open-source | proprietary | restricted.53