spb/company-atlas

1# CLAUDE.md — Company Atlas (repo guide)23**Product:** Company Atlas — *The Live Atlas of Global Companies*. A distributed public-web sensor network attached to companies:4observations → snapshots → changes → structured events → metrics → intelligence. The accumulated history is the product.5Full product specification (200 sections, non-negotiable principles): **`docs/PRODUCT-SPEC.md`**. Read it before changing behaviour.67**Domains:** canonical `https://www.company-atlas.co` today (DNS A → MacLustr Tunnel 51.161.112.61). `www.company-atlas.com` is the8spec's primary hostname but its nameservers are at Vercel, not GoDaddy — once its A records point to the gateway, add it as a tunnel9route/redirect and flip `CA_SITE_URL`. Apex → www 308.1011## Stack12- Python 3.12 `src/companyatlas` (FastAPI, SQLAlchemy Core + asyncpg, Alembic SQL forward-only, httpx, selectolax, feedparser, zstd),13  CLI **`catlas`** (commands auto-discovered from `companyatlas/commands/*.py`).14- Postgres 17 only (no Redis): queue = `queue_jobs` with `SKIP LOCKED`; caches in-process (`api/common.TTLCache`).15- Objects: content-addressed zstd store `CA_DATA_DIR/objects/ab/cd/<sha256>.zst` (`archive.py`).16- Web: `apps/web` Next 16 + React 19 + Tailwind v4 (Geist), SSR, rewrites `/api/v1/*` → FastAPI loopback. Ports: prod web **8360** / api **8361**;17  dev web **8370** / api **8371** (see `.env.example`).18- LLM enrichment: OpenAI-compatible `CA_LLM_BASE_URL` (MacLustr llm-api.io, key "company-atlas" in `deploy/.llm-key`, git-ignored). Optional; deterministic first.1920## Layout21```22src/companyatlas/   config.py taxonomy.py ids.py urls.py fetch.py archive.py db/ logging.py cli.py23                    sdk/ (models, normalize, diff, connector)  connectors/  services/ (discovery, pipeline, scheduler, events, llm, metrics…)24                    api/ (main, common, routers/*)  commands/ (CLI groups)  registry/ (seed loader)25migrations/versions/0001_initial.py   registry/ (companies ndjson, industries.yaml, countries.csv)   prompts/   fixtures/   tests/26apps/web/   docs/ (PRODUCT-SPEC, ARCHITECTURE, API, DATA-MODEL, CONNECTORS, SCORING, DEPLOY, OPERATIONS)   deploy/ (mld manifest, scripts)27```28Architecture, ownership map and boundaries: `docs/ARCHITECTURE.md`. API contract: `docs/API.md`.2930## Operating rules (spec §182)311. Inspect existing architecture first; preserve working functionality; avoid rewrites; reuse the SDK.322. Never fabricate data. No inference presented as fact: careful language (*detected*, *no longer listed*, *signal*), confidence labels.333. Historical-first: never overwrite or delete history; new versions, `status` columns, forward-only migrations.344. Raw vs interpreted stay separate (objects ↔ snapshots ↔ changes ↔ events). Reprocessing must never require re-fetching.355. LLMs are enrichment, not the crawler: deterministic fetch → normalize → hash → diff → significance → LLM only if useful, budgeted.366. Only `fetch.Fetcher` talks to the network: SSRF guard, robots, per-domain rate/concurrency, size and redirect caps. Never bypass37   authentication or challenges; never collect private data.387. No magic numbers: tunables in `config.Settings` / `taxonomy.py`; bump formula/prompt/connector versions when behaviour changes.398. Mobile is first-class; dark and light both designed; dense, readable, terminal-grade UI.409. Tests use fixtures (`fixtures/`), never the live web (`-m live` opt-in). Run `.venv/bin/pytest -q`, `.venv/bin/ruff check src tests`,41   `pnpm typecheck` after meaningful changes.4210. Secrets only in env / git-ignored files (`deploy/.admin-token`, `deploy/.llm-key`); the rendered manifest lives on M1M32.4344## Dev quickstart45```bash46uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e '.[dev]'47createdb -O companyatlas companyatlas   # role companyatlas/companyatlas, extensions pg_trgm + uuid-ossp48cp .env.example .env && .venv/bin/catlas migrate && .venv/bin/catlas seed49.venv/bin/catlas onboard --limit 50      # discovery + sensors for the first companies50.venv/bin/catlas schedule                # scheduler + workers (Ctrl-C to stop)51.venv/bin/catlas api                     # http://127.0.0.1:8371/api/v1/docs52pnpm install && pnpm dev:web             # http://localhost:837053```5455## Deploy (see docs/DEPLOY.md)56**Since 2026-09-14: OVH server BHS128b** (`ssh BHS128b`, Ubuntu 24.04), outside `mld` — dir `~/apps/company-atlas` (uv Python 3.12 venv),57data `~/company-atlas-data`, Postgres 17 native (`companyatlas`), PM2 under systemd (`~/apps/pm2.company-atlas.config.cjs`, generated from58the mld manifest kept in `~/apps/.manifests/`): `company-atlas-api` / `company-atlas-scheduler` / `company-atlas-web`. Route59`https://www.company-atlas.co → wg1 10.67.0.62:8360`. Release = rsync + `uv pip install -e .` + `catlas migrate` + web build + `pm2 restart`60(exact command in docs/DEPLOY.md). Former home: M2U64 via `mld` (`deploy/render-manifest.sh --push` → `mld stage` → `mld deploy`), cold copy kept.61