spb/company-atlas

1# Company Atlas — operations23## API45The API is one FastAPI process (`catlas api`, uvicorn) bound to loopback; Next.js proxies `/api/v1/*` to it. It is stateless: every cache is6in-process (`api/common.TTLCache`, cleared with `POST /api/v1/admin/cache/clear` or on restart), so several workers may run behind PM2 and7the rate limiter buckets are per process (effective limits scale with the number of workers — keep `--workers 1` unless traffic requires8more, or terminate rate limiting at the tunnel).910### Environment variables (read by `config.Settings`, prefix `CA_`)1112| Variable | Default | Used by the API for |13|---|---|---|14| `DATABASE_URL` | `postgresql+asyncpg://companyatlas:companyatlas@127.0.0.1:5432/companyatlas` | every query (pool 8 + 8 overflow per process) |15| `CA_API_HOST` / `CA_API_PORT` | `127.0.0.1` / `8371` (prod `8361`) | bind address of `catlas api` |16| `CA_ADMIN_TOKEN` | *(empty → every `/admin/*` call is 401)* | `X-CA-Admin-Token` (also bypasses rate limits) |17| `CA_SITE_URL` | `https://www.company-atlas.co` | CORS allow-list (plus the localhost dev/prod ports) |18| `CA_DATA_DIR` | `./data` | object store read by `/snapshots/{id}` (text/blocks) and `/stats.archive` |19| `CA_SEO_MIN_EVENTS` / `CA_SEO_MIN_SENSORS` | `1` / `3` | `/sitemap` inclusion rule (`companies.indexed` or ≥ events **and** ≥ active sensors) |20| `CA_MIN_INTERVAL_S` / `CA_MAX_INTERVAL_S` | `900` / `604800` | clamp for `POST /admin/sensors/{id}/set_interval` |21| `CA_LLM_DAILY_BUDGET`, `CA_LLM_BASE_URL`, `CA_LLM_ENABLED` | `1500`, *(empty)*, `true` | `/admin/overview.llm.budget_left`, `/ask` LLM refinement availability |22| `CA_NOISE_THRESHOLD` … `CA_CRITICAL_THRESHOLD` | `0.20 / 0.40 / 0.65 / 0.85` | `/methodology.significance_bands`, `/admin/costs` meaningful-event denominator |23| `CA_LOG_JSON` | `true` | JSON logs (`service=ca-api`) |2425Run: `.venv/bin/catlas api` (dev, port 8371) · `.venv/bin/catlas api --port 8361` (prod under PM2 `company-atlas-api`).26Health: `GET /health` → `{status, db, version, api_version, uptime_s}`; `GET /ready` → `{ready}` (DB reachable). Docs: `/api/v1/docs`,27schema `/api/v1/openapi.json`. The scheduler's heartbeat (`settings_kv['scheduler:heartbeat']`) surfaces as `/system.scheduler_last_tick_at`.2829### API keys and rate-limit tiers3031Anonymous traffic is limited to 120 requests/min per client IP (first hop of `X-Forwarded-For` — Caddy on the MacLustr Tunnel sets it).32Keys raise the tier (`authenticated` 600/min, `paid` 3 000/min, `internal` unlimited). Keys are stored as sha256 hashes in `api_keys`;33the raw key is printed **once**:3435```bash36.venv/bin/catlas api-key create "acme research" --tier paid     # prints ca_paid_… once; store it in the customer's vault37.venv/bin/catlas api-key list [--include-revoked]                # id, prefix, tier, last used, request count38.venv/bin/catlas api-key revoke key_01…                          # soft revoke (row kept, `revoked_at` set); cached for ≤ 60 s in running API processes39```4041Clients send `X-CA-API-Key: <key>`; responses carry `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Tier`; a `429` carries42`Retry-After`. Usage counters (`last_used_at`, `request_count`) are flushed every ~30 s per process. There is no self-service key issuance —43create keys from the M2U64 shell (or any host with `DATABASE_URL`).4445### Caches and freshness4647| Endpoint | TTL | Notes |48|---|---|---|49| `/pulse`, `/stats` | 60 s | homepage / counters; `archive` stats refresh every 10 min (`settings_kv['archive:stats']` if the ops job writes it, else a directory walk in a thread) |50| `/system` | 30 s | public health |51| `/rankings` | 120 s | per (kind, window, country, industry, limit) |52| `/industries*`, `/countries*`, `/index`, `/map`, `/trends`, `/events/types`, `/events/summary`, `/stats/history` | 300 s | aggregate producers in `api/aggregates.py` |53| `/sitemap` | 600 s | 5 000 entries per page |54| `/live*`, owner, admin | none (`no-store`) | |5556After bulk backfills or manual corrections: `curl -X POST -H "X-CA-Admin-Token: $CA_ADMIN_TOKEN" localhost:8371/api/v1/admin/cache/clear`57(optionally `?prefix=pulse`). Retract/restore of events clears the caches automatically.5859### Pending SQL6061`migrations/pending/api.sql` adds two trigram GIN indexes (`people.name`, `products.name`) used by `/search?types=people,products`. Apply62with `psql -f` (uses `CREATE INDEX CONCURRENTLY`, so outside a transaction) or fold into the next numbered Alembic migration. The endpoints63work without them (sequential scans on small tables).6465### Troubleshooting6667- `503`/`degraded` on `/health` → Postgres unreachable; the API keeps serving cached aggregates until they expire.68- `429` for the web app itself → the Next.js server is one IP; give it an `internal` key (`API_KEY` env in `apps/web`) or terminate limits upstream.69- `/snapshots/{id}` returns `text: null` → the object key is missing from `CA_DATA_DIR/objects` (different data dir or pruned by retention);70  metadata still renders and `/snapshots/{a}/diff/{b}` falls back to the stored `changes.diff` when present.71- `/ask` engine is always `deterministic` → `CA_LLM_BASE_URL`/`CA_LLM_API_KEY` unset or the small model unreachable (the deterministic parser is72  the designed fallback; nothing is fabricated either way).73