spb/company-atlas

1# Company Atlas — architecture & module ownership23One Python package (`src/companyatlas`, CLI `catlas`) runs every logical service of the spec as processes/commands, plus a Next 16 web app4(`apps/web`). Postgres 17 is the only stateful dependency (entities, history, queue via `SKIP LOCKED`, metrics); raw objects live in a5content-addressed zstd store on disk (`CA_DATA_DIR/objects`). No Redis. Workers are stateless and idempotent: any node with6`DATABASE_URL` + the data dir (or its own object store) can run `catlas schedule` / `catlas worker`.78```9registry/ (seed companies, industries, countries)  ──catlas seed──▶  companies10                                                                        │  onboarding queue11                                                                        ▼12                                              services/discovery.py  (homepage → robots → sitemaps → nav → ATS/feeds → classify → sensors)13                                                                        │14              scheduler (services/scheduler.py) claims due sensors ─────┤  adaptive intervals, domain budgets, priority15                                                                        ▼16                    services/pipeline.py  run_sensor():  fetch (fetch.py) → observation → connector.extract → Extraction17                                                          → normalize/fingerprint (sdk/normalize.py) → snapshot (+ objects)18                                                          → sdk/diff.py block diff + significance → changes row (+ structured_delta)19                                                          → entity reconciliation (jobs/people/products/plans/locations/news tables)20                                                                        │ changes.status = 'pending'21                                                                        ▼22                    services/events.py  process_changes(): deterministic events from structured deltas + diff → events (+ clusters, dedupe)23                                        → LLM enrichment jobs when useful (services/llm/*, prompts/ versioned, budgeted)24                                                                        ▼25                    services/metrics.py  hourly scores (activity, hiring momentum, product velocity, AI adoption, geo, developer, CCI),26                                        daily aggregates (company_daily, global_daily, coverage-normalised index), baselines → anomaly,27                                        signals, trends28                                                                        ▼29                    api/ (FastAPI, docs/API.md)  ◀── apps/web (Next 16, SSR + SSE live feed)  ◀── Caddy (MacLustr Tunnel) ◀── www.company-atlas.co30```3132## Ownership map (who writes what)3334| Area | Modules | CLI (`companyatlas/commands/*.py`, `register(app)`) |35|---|---|---|36| Kernel (done) | `config.py`, `taxonomy.py`, `ids.py`, `urls.py`, `fetch.py`, `archive.py`, `db/`, `logging.py`, `sdk/models.py`, migration 0001, `api/main.py`, `api/common.py`, `cli.py` | `migrate`, `api`, `version` |37| Crawl core | `sdk/normalize.py`, `sdk/diff.py`, `sdk/connector.py` (+ registry), `connectors/*`, `services/discovery.py`, `services/pipeline.py`, `services/scheduler.py`, `services/repair.py` | `crawl.py`: `onboard`, `discover`, `run-sensor`, `schedule`, `sensors`, `repair`, `connectors` |38| Intelligence | `services/events.py`, `services/clustering.py`, `services/llm/{gateway,enrich,schemas}.py`, `prompts/*`, `services/metrics.py`, `services/signals.py`, `services/trends.py`, `services/alerts.py`, `services/digest.py` | `intel.py`: `process-changes`, `enrich`, `metrics`, `daily`, `signals`, `alerts` |39| Seeds | `registry/` data files, `companyatlas/registry/seed.py`, `scripts/seed_wikidata.py`, `scripts/seed_edgar.py` | `seed.py`: `seed`, `import-companies` |40| Profile enrichment | `services/enrichment.py` (Wikidata entity + Wikipedia summary + homepage facts + grounded LLM text → `companies.source_meta.profile`, column back-fills with provenance, `people`, `company_relationships`; periodic `company-enrichment`), `prompts/company-profile/` | `enrichment.py`: `enrich-companies`, `profile` |41| API | `api/routers/*.py` (auto-included; `ORDER` for precedence), `api/sse.py`, `api/ratelimit.py` | — |42| Web | `apps/web` | — |43| Ops | `deploy/*.mld.json`, `deploy/render-manifest.sh`, `deploy/first-run.sh`, `scripts/backup*.sh` | `ops.py`: `stats`, `status`, `backup`, `retention` |4445## Boundaries (contracts)46- **Crawl → Intelligence**: `changes` rows with `status='pending'`, `diff` (bounded JSON, `BlockDiff.to_json()`), `structured_delta`47  (shape documented in `sdk/models.py`), `significance`, `kind`. Entity tables already reconciled (first_seen/last_seen/removed_at).48- **Intelligence → API**: `events`, `event_clusters`, `event_sources`, `metrics_current`, `metric_series`, `company_daily`, `global_daily`,49  `baselines`, `signals`, `trends`, `alert_deliveries`.50- **API → Web**: `docs/API.md`. The web never touches the database.51- **Seeds → Crawl**: `companies` with `onboarding_status='pending'` + `queue_jobs(kind='discover')`.5253## Conventions54- IDs: `ids.new_id(kind)` (prefixed ULIDs). Slugs from `ids.slugify`. Alias keys from `ids.normalize_alias`.55- SQL: plain text via `db.fetch_all/fetch_one/execute` (SQLAlchemy Core, asyncpg). Cast ambiguous binds (`cast(:x as text)`),56  arrays as `any(cast(:ids as text[]))`, real `datetime`/`date` objects as params, `jsonb(value)` + `cast(:p as jsonb)`.57- Timestamps UTC. Never delete history; use `status` columns. Migrations forward-only, additive, in `migrations/versions/000N_*.py`.58- Politeness: only `fetch.Fetcher` talks to the network (SSRF guard, robots, per-domain governor, size caps). Never bypass challenges.59- Language: "detected", "observed", "no longer listed", "appears", "signal", "inferred". Never "fired", "laid off", "shut down".60- Config: tunables in `config.Settings` / `taxonomy.py`; formula versions bump when weights change.61- Tests: fixtures in `fixtures/`, never live network in CI (`@pytest.mark.live` for opt-in checks).62- Logging: `logging.getLogger(__name__)` with `extra={…}` (JSON in prod).63