# Seeds — the initial company universe

The seed registry is the diversified starting universe of Company Atlas (spec §99: 5,000+ companies across sectors and regions). It is
plain data committed in `registry/` plus an idempotent loader; the crawl core takes over from `companies.onboarding_status = 'pending'`
and the `queue_jobs(kind='discover')` rows the loader creates (boundary "Seeds → Crawl" in `docs/ARCHITECTURE.md`).

```
registry/industries.yaml              taxonomy (51 industries, 23 top-level + children, keywords for label mapping)
registry/countries.csv                250 ISO-3166-1 alpha-2 codes (+ XK), UN M49 region/subregion, centroids
registry/companies/wikidata-*.ndjson  one JSON object per company, one file per UN region
registry/companies/README.md          counts by tier / region / country / industry (regenerated by the harvester)
scripts/seed_countries.py             regenerates countries.csv
scripts/seed_wikidata.py              Wikidata harvester (candidates → select → details → assemble)
scripts/seed_edgar.py                 SEC EDGAR enrichment (CIK, ticker, exchange, SIC industry hint)
src/companyatlas/registry/industries.py   taxonomy loader, map_industry(labels) → slugs, map_sic(code)
src/companyatlas/registry/seed.py         loader: seed(), import_companies(), add_company()
src/companyatlas/commands/seed.py         catlas seed | import-companies | company-add | registry-stats
```

## Sources

| Source | What we take | Politeness |
|---|---|---|
| **Wikidata** (SPARQL, `query.wikidata.org`) | companies by class (P31), official website P856, country P17→P297, labels/aliases/descriptions, official name P1448, industry P452, inception P571, employees P1128 (+P585), HQ P159 (+P625, P131, P17), coordinates P625, ticker P249 / exchange P414, LEI P1278, CIK P5531, parent P749, logo P154, sitelink count | UA `CompanyAtlasBot/0.1 (contact@spboucher.ai)`, one query at a time, ≥ 2 s apart, 60 s server timeout, retries with exponential backoff, every result cached in `data/seed/wikidata/sparql/<sha256>.json` (git-ignored) |
| **SEC EDGAR** (`company_tickers.json`, `data.sec.gov/submissions`) | CIK, tickers, exchanges, SIC + description, business address | required UA, ≤ 5 req/s, cached in `data/seed/edgar/` |
| **ISO 3166 / UN M49** (lukes/ISO-3166-Countries-with-Regional-Codes) + Wikidata P625 | country names, regions, centroids | one-off; the CSV is committed |

Facts are copied, never inferred: a company only enters the registry with an official website *stated on Wikidata*; nothing is guessed
from names. Provenance is kept per company (`source`, `harvested_at`, `sitelinks`, raw `industry_labels`, `notes`) and lands in
`companies.source_meta`.

## Harvest procedure (`scripts/seed_wikidata.py`)

```bash
.venv/bin/python scripts/seed_wikidata.py all --target 8000    # ≈ 45–60 min the first time; every stage resumes from cache
.venv/bin/python scripts/seed_wikidata.py candidates|select|details|assemble   # single stage
.venv/bin/python scripts/seed_edgar.py                         # optional enrichment, rewrites the NDJSON in place
.venv/bin/catlas registry-stats                                # counts without a database
```

1. **candidates** — `?item wdt:P31 wd:<class>` for 30 company classes (business, enterprise, public company, company, privately held
   company, corporation, bank, airline, automobile/aerospace manufacturer, software/technology company, video-game developer/publisher,
   publisher, record label, telecom, brewery, retail chain, railway, shipyard, …). The subclass tree of *business* (`P279*`) is far too
   broad to page, so classes are queried directly and the big ones are chunked by **sitelink bands** (≥ 80, 50–79, 35–49, 25–34, 18–24,
   12–17, 8–11, 5–7); a band that times out is split in two automatically. Minimum 5 sitelinks.
   Then **country boosts** (sitelinks ≥ 2, country-first scan, only for minimum-coverage countries still short after the class stage;
   a timeout is tolerated and reported) and **industry boosts** (P452 label lists per top-level industry, sitelinks ≥ 2, 600 each).
   Output `data/seed/candidates.json`.
   **dissolved** — a separate `P576` lookup over every candidate (300 QIDs per query; a `NOT EXISTS` filter inside the big scans
   is what makes them time out); defunct companies are excluded at selection. Output `data/seed/dissolved.json`.
2. **select** — websites normalised to `https://<host>` (paths stripped); generic hosts dropped (social networks, blogs, app stores,
   site builders, code forges, Wikipedia …; for platform roots such as `google.com`/`apple.com` only the bare/www host counts);
   duplicates by registrable domain keep the highest-sitelink item (the loser is recorded as a `related_domain_conflict` note on the
   winner and in `data/seed/dropped.json`). Then the diversified selection below. Output `data/seed/selected.json` (+ a reserve of
   boost candidates per top-level industry).
3. **details** — five light queries per batch of 100 QIDs (labels/description/aliases via the label service; inception/coordinates/
   LEI/CIK/logo; HQ + parent with end-time qualifiers; legal names + industry labels; tickers/exchanges + employee observations).
   A `GROUP BY` with many `SAMPLE()` aggregates makes Blazegraph throw `StackOverflowError`, hence plain queries reduced in Python.
   Output `data/seed/details.json` (incremental).
4. **assemble** — industry mapping, importance, tiers, parent/domain conflicts, per-region NDJSON + README.

Gotchas learnt on the endpoint: a query that runs past 60 s may come back as **HTTP 200 with a truncated JSON body followed by a Java
stack trace** (the gateway even caches it) — the client treats an unparsable body as a timeout; some literals contain raw control
characters (`json.loads(strict=False)`); starting a query from `P17` of a large country never finishes; `FILTER NOT EXISTS` over tens of
thousands of bindings is what pushes a scan over the limit (fetch as OPTIONAL and filter client-side instead).

## Selection and diversification rules

Candidates are ordered by sitelinks (desc) and selected until `--target` (8,000) with:

* **Caps** — United States ≤ 40 % of the target, any other country ≤ 12 %, unknown country ≤ 3 %; narrow Wikidata classes that are
  over-represented among high-sitelink items are capped too (airlines 5 %, video-game developers 4 %, record labels / publishers /
  film production / railway companies 3 %, shipyards / breweries / book and game publishers 2 %).
* **Minimum coverage** (taken first, best-ranked companies of that country, sitelinks ≥ 2): ≥ 150 for CA, GB, DE, FR, JP, KR, IN, AU;
  ≥ 60 for BR, MX, AE, SA, ZA, NG, SG, ID, NL, SE, CH, ES, IT, CN, TW, HK — when Wikidata has that many companies with a website.
* **Industry floor** — every top-level industry gets ≥ 60 companies where the data allows, topped up from the industry-boost reserve.
* **One company per registrable domain**; a subsidiary that shares its parent's domain is dropped (the parent keeps it).

The resulting counts are in `registry/companies/README.md` and `catlas registry-stats`; `tests/test_registry_files.py` enforces the
caps (with a small tolerance for the industry top-up), the minimums for the large economies and the tier sizes.

## Importance and tiers

```
s_sitelinks = min(1, ln(1 + sitelinks) / ln(1 + 300))         # 300 sitelinks ≈ the most-linked companies
s_employees = min(1, log10(1 + employees) / 6)                # 1,000,000 employees → 1
importance  = 0.6·s_sitelinks + 0.2·s_employees + 0.1·public_company + 0.1·has_ticker     (clamped to 0.02–1)
tier        = rank by importance: 1–150 → 1 (global) · 151–950 → 2 (major) · 951–3450 → 3 (notable) · rest → 4 (long tail)
```

Importance drives crawl priority only (spec §113); it is refreshed on every `catlas seed`, nothing else about an existing row is.

## NDJSON schema (one object per line)

```
wikidata_id, display_name, legal_name, aliases[≤8], website (https://host), canonical_domain, country (ISO-2 | null), hq_city, hq_region,
lat, lon, industries[] (taxonomy slugs), industry_labels[] (raw P452 labels, + "SIC …" after EDGAR), founded_year, employees,
public_company, ticker, exchange, lei, sec_cik, parent {wikidata_id, name} | null, logo_url (Commons Special:FilePath), description,
sitelinks, importance (0–1), tier (1–4), source ("wikidata"), harvested_at, notes[] (optional: related_domain_conflict),
source_edgar {cik, enriched_at} (optional)
```

## Loading (`catlas seed`)

`seed(conn, companies=True, limit=None, files=None)` in `companyatlas.registry.seed`:

1. upserts `industries` (slug, name, parent, description, keywords, sort order) and `countries`;
2. for each registry line: match an existing company by `wikidata_id`, then by `canonical_domain`;
   * new → `id = ids.new_id('company')`, slug `ids.slugify(display_name)` de-duplicated as `-<country>` then `-2`, `-3`…,
     `onboarding_status='pending'`, provenance in `source_meta`;
   * existing → only **null** columns are filled; `importance`, `tier` and `source_meta` provenance are refreshed; names, website,
     history and anything already set are never overwritten;
3. `company_aliases` (display name → `brand`, legal name → `legal`, ticker → `ticker`, others → `alias`; key `ids.normalize_alias`),
   `domains` primary row, `company_relationships` PARENT_OF / SUBSIDIARY_OF (confidence 0.8, provenance `{source, property: P749}`)
   when the parent is also in the registry;
4. `queue_jobs(kind='discover', key='discover:<company_id>', priority=importance)` for new pending companies (`on conflict do nothing`);
5. `settings_kv['seed:last_run']` with the counters.

Running it twice yields `companies_new = 0`. `--limit N` seeds the first N lines (files are ordered tier 1 → 4 inside each region);
`--no-companies` only refreshes the reference tables; `--file` loads specific NDJSON files.

## Adding companies

```bash
.venv/bin/catlas company-add https://www.example.com --name "Example Corp" --country CA --industry software
.venv/bin/catlas import-companies my-companies.csv        # columns: website (required), display_name, country, industries ("a;b"), aliases, …
.venv/bin/catlas import-companies my-companies.ndjson     # same keys as the registry schema
```

Imports go through the same upsert (idempotent, same slug/alias/queue rules) with `source_meta.source = "manual"` (`--source` to change).
To add companies to the *committed* universe, append lines to a `registry/companies/*.ndjson` file (or a new `manual-*.ndjson`) with at
least `website`; run `pytest tests/test_registry_files.py` (validity, no duplicate domain, caps) then `catlas seed`.

## Tests

```bash
.venv/bin/pytest -q tests/test_industry_map.py tests/test_seed_loader.py tests/test_registry_files.py
```

`test_seed_loader.py` needs the local database (`catlas migrate`); it seeds a fixture twice under `ztest-` slugs and cleans up after itself.

## Harvest of 2026-09-12 (current registry)

* 24,448 candidates from 30 classes + boosts; 2,402 excluded as dissolved (P576); 8,086 companies assembled (target 8,000 + industry
  top-up); 2,637 public, 2,323 with a ticker, 1,121 with a SEC CIK after EDGAR (242 new matches, 321 rows enriched, 29 industries from SIC).
* Country caps respected (US 22.4 %, next Japan 7.4 %); all 150-minimums met (CA 191, GB 536, DE 557, FR 175, JP 602, KR 156, IN 152,
  AU 154); 60-minimums met except BR 59, MX 59, ID 58 (one or two companies short after dissolution/domain de-duplication);
  every top-level industry ≥ 60. 257 companies (3.2 %) have no country on Wikidata; 384 (4.7 %) have no industry mapping.
* Country-first boosts for AU and MX were cut by the endpoint's time limit (tolerated); KR/GB/DE/FR/JP/CA/IN never needed one.
* `catlas seed` on the local DB: 8,075 new + 11 matched to companies the crawl agent had already created; 20,777 aliases, 2,582
  PARENT_OF/SUBSIDIARY_OF rows, 8,075 `discover` jobs; second run 0 new. About 18 s per run.

## Known gaps

Structural limits:

* Coverage follows Wikidata: companies without an official website there are absent; a few small markets end up one or two short of
  their 60 minimum; country-first boost queries for larger countries time out on the endpoint (they only matter below 5 sitelinks).
* Industry slugs come from P452 labels and, failing that, the boost query / description keywords; ~5 % of companies have none
  (the LLM `classify_industry` job and the crawl fill them later). The class mix over-represents media/entertainment and transport
  even after the class caps (Wikidata has many well-linked airlines, labels, publishers and game studios).
* EDGAR only fills what is missing; ~1,800 US companies but only ~1,100 CIKs because name matching is exact on the normalised name
  (no fuzzy guessing, spec §6 "do not merge on similar names").
* `assemble` regenerates the NDJSON from the Wikidata cache and therefore drops EDGAR fields — always run `seed_edgar.py` after it.
* `hq_region` is the P131 parent of the HQ item and is sometimes the country itself.
* Websites are forced to `https://`; discovery handles redirects and http-only fallbacks.