spb/company-atlas

1# Deploying Company Atlas on MacLustr23> **Since 2026-09-14 production runs on the OVH server BHS128b** (`ssh BHS128b`, ubuntu@51.161.112.85, Ubuntu 24.04, 16 threads / 128 GB),4> outside `mld`: code in `~/apps/company-atlas` (venv `.venv` = uv Python 3.12, `uv pip install -e .`), Postgres 17 native (db/role5> `companyatlas`/`companyatlas`, pg_trgm + uuid-ossp, en_US.UTF-8), objects in `~/company-atlas-data`, PM2 under systemd (`pm2-ubuntu`) with6> `~/apps/pm2.company-atlas.config.cjs` generated from the mld manifest by `~/apps/.manifests/gen-pm2.py` (secrets, 0600). Tunnel route:7> `mlt add www.company-atlas.co BHS128b:8360` (BHS64 Caddy → wg1 10.67.0.62); firewall `ufw allow in on wg1 from 10.67.0.1 to any port 8360 proto tcp`.8> **Release on BHS128b**: `rsync -az --exclude .venv --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data . BHS128b:apps/company-atlas/`9> then `ssh BHS128b 'cd apps/company-atlas && ~/.local/bin/uv pip install --python .venv/bin/python -e . && DATABASE_URL=postgresql+asyncpg://companyatlas:companyatlas@127.0.0.1:5432/companyatlas CA_DATA_DIR=$HOME/company-atlas-data .venv/bin/catlas migrate && pnpm install --frozen-lockfile && (cd apps/web && API_URL=http://127.0.0.1:8361 NEXT_PUBLIC_SITE_URL=https://www.company-atlas.co pnpm build) && pm2 restart company-atlas-api company-atlas-scheduler company-atlas-web && pm2 save'`.10> Known: `company-atlas-scheduler` segfaults now and then inside `lexbor` (selectolax native parser) during enrichment — PM2 restarts it11> (`PYTHONFAULTHANDLER=1` set in the PM2 env to capture the Python stack); `/api/v1/index` and `/api/v1/pulse` return 50012> (`TypeError: float() argument … 'dict'` in `api/aggregates.py` `by_country`) — pre-existing on M2U64 too.13> The M2U64 copy (code, Postgres `companyatlas`, `~/company-atlas-data`) is kept as a cold copy, retired from `mld` (`--keep-dir`).1415Production formerly ran on **M2U64** (Mac Studio M2 Ultra, 24 c / 64 GB) behind the MacLustr Tunnel (BHS64 Caddy → WireGuard wg1 10.67.0.12).16Everything goes through the gateway M1M32 and `mld` (`~/Desktop/cluster-skill/mld`).1718## Topology1920```21Internet → DNS (A www.company-atlas.co → 51.161.112.61)22        → BHS64 Caddy (TLS Let's Encrypt)  →  WireGuard wg1  →  M2U64:8360 (Next.js, PM2 company-atlas-web)23                                                                  └─ rewrite /api/v1/* → 127.0.0.1:8361 (FastAPI, 2 uvicorn workers, PM2 company-atlas-api)24                                                                  └─ PM2 company-atlas-scheduler (`catlas schedule`: due sensors + discovery queue + periodic tasks)25                                                                  └─ Postgres 17 Homebrew (db companyatlas) · ~/company-atlas-data (objects zstd, logs, backups)26        LLM enrichment: https://www.llm-api.io/v1 (MacLustr llm-api on M1M64, key "company-atlas") — optional, budgeted27```2829Node prerequisites (done 2026-09-12): node 25 + pnpm + pm2 (Homebrew), `uv` + Python 3.12 (`~/.local/bin/uv`), `postgresql@17` running,30role/db `companyatlas` (password `companyatlas`, localhost only), extensions `pg_trgm` + `uuid-ossp`, `~/company-atlas-data`.31No Redis is required.3233## Domains3435| Host | State | Action |36|---|---|---|37| `www.company-atlas.co` | A → 51.161.112.61 (GoDaddy) — **canonical today** | route posed by `mld deploy` |38| `company-atlas.co` | no A record | add `A @ → 51.161.112.61`, then put `company-atlas.co` in `tunnel.redirects` |39| `www.company-atlas.com` / `company-atlas.com` | nameservers at **Vercel** (`ns1.vercel-dns.com`), A → Vercel IPs | point A records (or NS) at the gateway; then add both to `tunnel.redirects` (308 → www.company-atlas.co) or make `.com` canonical: `CA_SITE_URL`, `NEXT_PUBLIC_SITE_URL`, `tunnel.domain` |4041Caddy only issues certificates for hosts whose authoritative DNS points at the gateway — do not add a route before the DNS is in place.4243## Release procedure (from the laptop)4445```bash46cd ~/Desktop/Projets/apps-web/company-atlas47.venv/bin/pytest -q && .venv/bin/ruff check src tests && pnpm -r typecheck     # 1. tests + typecheck48deploy/render-manifest.sh --push                                                # 2. manifest with CA_ADMIN_TOKEN + CA_LLM_API_KEY → M1M32:~/dispatch/apps/company-atlas.json49~/Desktop/cluster-skill/mld stage ~/Desktop/Projets/apps-web/company-atlas company-atlas   # 3. laptop → gateway staging (sync_excludes)50~/Desktop/cluster-skill/mld deploy company-atlas --node M2U64                   # 4. rsync → node, hooks (venv, migrate, seed, pnpm build), PM2, health, tunnel route, registry51ssh M2U64 'cd ~/apps/company-atlas && nohup bash deploy/first-run.sh > ~/company-atlas-data/logs/first-run.log 2>&1 &'   # 5. first time: mass onboarding52curl -sI https://www.company-atlas.co | head -1                                 # 6. public check53```5455`mld deploy` re-points `https://www.company-atlas.co → M2U64:8360` on BHS64 and runs the public health check. `mld heal` (every 5 min on56M1M32) restarts anything missing — note it also *undoes* a manual `pm2 stop`: to pause a process, remove it from the manifest.5758## Operations5960```bash61~/Desktop/cluster-skill/mld status | grep company-atlas62~/Desktop/cluster-skill/mld logs company-atlas63ssh M2U64 'cd ~/apps/company-atlas && export PATH=/opt/homebrew/opt/postgresql@17/bin:$PATH CA_LOG_JSON=0 DATABASE_URL=postgresql+asyncpg://companyatlas:companyatlas@127.0.0.1:5432/companyatlas CA_DATA_DIR=$HOME/company-atlas-data; .venv/bin/catlas status'   # heartbeat, queue, failures64ssh M2U64 '… .venv/bin/catlas stats'                                            # dataset counters65ssh M2U64 '… .venv/bin/catlas sensors --status failing --limit 50'66ssh M2U64 '… .venv/bin/catlas discover https://example.com --dry-run'67ssh M2U64 '… .venv/bin/catlas backup'                                           # pg_dump → ~/company-atlas-data/backups (nightly 04:35 too)68scripts/backup-offnode.sh                                                       # second copy (dumps + objects) → M1M32:~/backups/company-atlas69curl -s https://www.company-atlas.co/api/v1/stats | python3 -m json.tool | head70```7172Admin console: `https://www.company-atlas.co/admin` (token = `deploy/.admin-token`). Environment variables: `.env.example`.7374## Scaling7576- More crawl throughput: raise `CA_FETCH_CONCURRENCY` (scheduler env in the manifest) or run a second `catlas schedule --no-onboarding`77  on another node with `DATABASE_URL` pointing at M2U64 (Postgres must then listen on the LAN and the object store must be shared or78  per-node — observations reference object keys, so a per-node store needs a shared path; prefer scaling on the same node first).79- LLM enrichment: any OpenAI-compatible server (`CA_LLM_BASE_URL`); budget `CA_LLM_DAILY_BUDGET`.80- Storage: object store growth ≈ changed snapshots × ~15 kB (zstd). 373 GB free on M2U64 at launch.81