# Company Atlas — architecture & module ownership

One Python package (`src/companyatlas`, CLI `catlas`) runs every logical service of the spec as processes/commands, plus a Next 16 web app
(`apps/web`). Postgres 17 is the only stateful dependency (entities, history, queue via `SKIP LOCKED`, metrics); raw objects live in a
content-addressed zstd store on disk (`CA_DATA_DIR/objects`). No Redis. Workers are stateless and idempotent: any node with
`DATABASE_URL` + the data dir (or its own object store) can run `catlas schedule` / `catlas worker`.

```
registry/ (seed companies, industries, countries)  ──catlas seed──▶  companies
                                                                        │  onboarding queue
                                                                        ▼
                                              services/discovery.py  (homepage → robots → sitemaps → nav → ATS/feeds → classify → sensors)
                                                                        │
              scheduler (services/scheduler.py) claims due sensors ─────┤  adaptive intervals, domain budgets, priority
                                                                        ▼
                    services/pipeline.py  run_sensor():  fetch (fetch.py) → observation → connector.extract → Extraction
                                                          → normalize/fingerprint (sdk/normalize.py) → snapshot (+ objects)
                                                          → sdk/diff.py block diff + significance → changes row (+ structured_delta)
                                                          → entity reconciliation (jobs/people/products/plans/locations/news tables)
                                                                        │ changes.status = 'pending'
                                                                        ▼
                    services/events.py  process_changes(): deterministic events from structured deltas + diff → events (+ clusters, dedupe)
                                        → LLM enrichment jobs when useful (services/llm/*, prompts/ versioned, budgeted)
                                                                        ▼
                    services/metrics.py  hourly scores (activity, hiring momentum, product velocity, AI adoption, geo, developer, CCI),
                                        daily aggregates (company_daily, global_daily, coverage-normalised index), baselines → anomaly,
                                        signals, trends
                                                                        ▼
                    api/ (FastAPI, docs/API.md)  ◀── apps/web (Next 16, SSR + SSE live feed)  ◀── Caddy (MacLustr Tunnel) ◀── www.company-atlas.co
```

## Ownership map (who writes what)

| Area | Modules | CLI (`companyatlas/commands/*.py`, `register(app)`) |
|---|---|---|
| Kernel (done) | `config.py`, `taxonomy.py`, `ids.py`, `urls.py`, `fetch.py`, `archive.py`, `db/`, `logging.py`, `sdk/models.py`, migration 0001, `api/main.py`, `api/common.py`, `cli.py` | `migrate`, `api`, `version` |
| Crawl core | `sdk/normalize.py`, `sdk/diff.py`, `sdk/connector.py` (+ registry), `connectors/*`, `services/discovery.py`, `services/pipeline.py`, `services/scheduler.py`, `services/repair.py` | `crawl.py`: `onboard`, `discover`, `run-sensor`, `schedule`, `sensors`, `repair`, `connectors` |
| Intelligence | `services/events.py`, `services/clustering.py`, `services/llm/{gateway,enrich,schemas}.py`, `prompts/*`, `services/metrics.py`, `services/signals.py`, `services/trends.py`, `services/alerts.py`, `services/digest.py` | `intel.py`: `process-changes`, `enrich`, `metrics`, `daily`, `signals`, `alerts` |
| Seeds | `registry/` data files, `companyatlas/registry/seed.py`, `scripts/seed_wikidata.py`, `scripts/seed_edgar.py` | `seed.py`: `seed`, `import-companies` |
| Profile enrichment | `services/enrichment.py` (Wikidata entity + Wikipedia summary + homepage facts + grounded LLM text → `companies.source_meta.profile`, column back-fills with provenance, `people`, `company_relationships`; periodic `company-enrichment`), `prompts/company-profile/` | `enrichment.py`: `enrich-companies`, `profile` |
| API | `api/routers/*.py` (auto-included; `ORDER` for precedence), `api/sse.py`, `api/ratelimit.py` | — |
| Web | `apps/web` | — |
| Ops | `deploy/*.mld.json`, `deploy/render-manifest.sh`, `deploy/first-run.sh`, `scripts/backup*.sh` | `ops.py`: `stats`, `status`, `backup`, `retention` |

## Boundaries (contracts)
- **Crawl → Intelligence**: `changes` rows with `status='pending'`, `diff` (bounded JSON, `BlockDiff.to_json()`), `structured_delta`
  (shape documented in `sdk/models.py`), `significance`, `kind`. Entity tables already reconciled (first_seen/last_seen/removed_at).
- **Intelligence → API**: `events`, `event_clusters`, `event_sources`, `metrics_current`, `metric_series`, `company_daily`, `global_daily`,
  `baselines`, `signals`, `trends`, `alert_deliveries`.
- **API → Web**: `docs/API.md`. The web never touches the database.
- **Seeds → Crawl**: `companies` with `onboarding_status='pending'` + `queue_jobs(kind='discover')`.

## Conventions
- IDs: `ids.new_id(kind)` (prefixed ULIDs). Slugs from `ids.slugify`. Alias keys from `ids.normalize_alias`.
- SQL: plain text via `db.fetch_all/fetch_one/execute` (SQLAlchemy Core, asyncpg). Cast ambiguous binds (`cast(:x as text)`),
  arrays as `any(cast(:ids as text[]))`, real `datetime`/`date` objects as params, `jsonb(value)` + `cast(:p as jsonb)`.
- Timestamps UTC. Never delete history; use `status` columns. Migrations forward-only, additive, in `migrations/versions/000N_*.py`.
- Politeness: only `fetch.Fetcher` talks to the network (SSRF guard, robots, per-domain governor, size caps). Never bypass challenges.
- Language: "detected", "observed", "no longer listed", "appears", "signal", "inferred". Never "fired", "laid off", "shut down".
- Config: tunables in `config.Settings` / `taxonomy.py`; formula versions bump when weights change.
- Tests: fixtures in `fixtures/`, never live network in CI (`@pytest.mark.live` for opt-in checks).
- Logging: `logging.getLogger(__name__)` with `extra={…}` (JSON in prod).