# Company Atlas — architecture & module ownership

One Python package (src/companyatlas, CLI catlas) runs every logical service of the spec as processes/commands, plus a Next 16 web app (apps/web). Postgres 17 is the only stateful dependency (entities, history, queue via SKIP LOCKED, metrics); raw objects live in a content-addressed zstd store on disk (CA_DATA_DIR/objects). No Redis. Workers are stateless and idempotent: any node with DATABASE_URL + the data dir (or its own object store) can run catlas schedule / catlas worker.

text

registry/ (seed companies, industries, countries)  ──catlas seed──▶  companies
                                                                        │  onboarding queue
                                                                        ▼
                                              services/discovery.py  (homepage → robots → sitemaps → nav → ATS/feeds → classify → sensors)
                                                                        │
              scheduler (services/scheduler.py) claims due sensors ─────┤  adaptive intervals, domain budgets, priority
                                                                        ▼
                    services/pipeline.py  run_sensor():  fetch (fetch.py) → observation → connector.extract → Extraction
                                                          → normalize/fingerprint (sdk/normalize.py) → snapshot (+ objects)
                                                          → sdk/diff.py block diff + significance → changes row (+ structured_delta)
                                                          → entity reconciliation (jobs/people/products/plans/locations/news tables)
                                                                        │ changes.status = 'pending'
                                                                        ▼
                    services/events.py  process_changes(): deterministic events from structured deltas + diff → events (+ clusters, dedupe)
                                        → LLM enrichment jobs when useful (services/llm/*, prompts/ versioned, budgeted)
                                                                        ▼
                    services/metrics.py  hourly scores (activity, hiring momentum, product velocity, AI adoption, geo, developer, CCI),
                                        daily aggregates (company_daily, global_daily, coverage-normalised index), baselines → anomaly,
                                        signals, trends
                                                                        ▼
                    api/ (FastAPI, docs/API.md)  ◀── apps/web (Next 16, SSR + SSE live feed)  ◀── Caddy (MacLustr Tunnel) ◀── www.company-atlas.co

# Ownership map (who writes what)

Area	Modules	CLI (`companyatlas/commands/*.py`, `register(app)`)
Kernel (done)	`config.py`, `taxonomy.py`, `ids.py`, `urls.py`, `fetch.py`, `archive.py`, `db/`, `logging.py`, `sdk/models.py`, migration 0001, `api/main.py`, `api/common.py`, `cli.py`	`migrate`, `api`, `version`
Crawl core	`sdk/normalize.py`, `sdk/diff.py`, `sdk/connector.py` (+ registry), `connectors/*`, `services/discovery.py`, `services/pipeline.py`, `services/scheduler.py`, `services/repair.py`	`crawl.py`: `onboard`, `discover`, `run-sensor`, `schedule`, `sensors`, `repair`, `connectors`
Intelligence	`services/events.py`, `services/clustering.py`, `services/llm/{gateway,enrich,schemas}.py`, `prompts/*`, `services/metrics.py`, `services/signals.py`, `services/trends.py`, `services/alerts.py`, `services/digest.py`	`intel.py`: `process-changes`, `enrich`, `metrics`, `daily`, `signals`, `alerts`
Seeds	`registry/` data files, `companyatlas/registry/seed.py`, `scripts/seed_wikidata.py`, `scripts/seed_edgar.py`	`seed.py`: `seed`, `import-companies`
Profile enrichment	`services/enrichment.py` (Wikidata entity + Wikipedia summary + homepage facts + grounded LLM text → `companies.source_meta.profile`, column back-fills with provenance, `people`, `company_relationships`; periodic `company-enrichment`), `prompts/company-profile/`	`enrichment.py`: `enrich-companies`, `profile`
API	`api/routers/*.py` (auto-included; `ORDER` for precedence), `api/sse.py`, `api/ratelimit.py`	—
Web	`apps/web`	—
Ops	`deploy/.mld.json`, `deploy/render-manifest.sh`, `deploy/first-run.sh`, `scripts/backup.sh`	`ops.py`: `stats`, `status`, `backup`, `retention`

# Boundaries (contracts)

Crawl → Intelligence: changes rows with status='pending', diff (bounded JSON, BlockDiff.to_json()), structured_delta (shape documented in sdk/models.py), significance, kind. Entity tables already reconciled (first_seen/last_seen/removed_at).
Intelligence → API: events, event_clusters, event_sources, metrics_current, metric_series, company_daily, global_daily, baselines, signals, trends, alert_deliveries.
API → Web: docs/API.md. The web never touches the database.
Seeds → Crawl: companies with onboarding_status='pending' + queue_jobs(kind='discover').

# Conventions

IDs: ids.new_id(kind) (prefixed ULIDs). Slugs from ids.slugify. Alias keys from ids.normalize_alias.
SQL: plain text via db.fetch_all/fetch_one/execute (SQLAlchemy Core, asyncpg). Cast ambiguous binds (cast(:x as text)), arrays as any(cast(:ids as text[])), real datetime/date objects as params, jsonb(value) + cast(:p as jsonb).
Timestamps UTC. Never delete history; use status columns. Migrations forward-only, additive, in migrations/versions/000N_*.py.
Politeness: only fetch.Fetcher talks to the network (SSRF guard, robots, per-domain governor, size caps). Never bypass challenges.
Language: "detected", "observed", "no longer listed", "appears", "signal", "inferred". Never "fired", "laid off", "shut down".
Config: tunables in config.Settings / taxonomy.py; formula versions bump when weights change.
Tests: fixtures in fixtures/, never live network in CI (@pytest.mark.live for opt-in checks).
Logging: logging.getLogger(__name__) with extra={…} (JSON in prod).