# Seeds — the initial company universe

The seed registry is the diversified starting universe of Company Atlas (spec §99: 5,000+ companies across sectors and regions). It is plain data committed in registry/ plus an idempotent loader; the crawl core takes over from companies.onboarding_status = 'pending' and the queue_jobs(kind='discover') rows the loader creates (boundary "Seeds → Crawl" in docs/ARCHITECTURE.md).

registry/industries.yaml              taxonomy (51 industries, 23 top-level + children, keywords for label mapping)
registry/countries.csv                250 ISO-3166-1 alpha-2 codes (+ XK), UN M49 region/subregion, centroids
registry/companies/wikidata-*.ndjson  one JSON object per company, one file per UN region
registry/companies/README.md          counts by tier / region / country / industry (regenerated by the harvester)
scripts/seed_countries.py             regenerates countries.csv
scripts/seed_wikidata.py              Wikidata harvester (candidates → select → details → assemble)
scripts/seed_edgar.py                 SEC EDGAR enrichment (CIK, ticker, exchange, SIC industry hint)
src/companyatlas/registry/industries.py   taxonomy loader, map_industry(labels) → slugs, map_sic(code)
src/companyatlas/registry/seed.py         loader: seed(), import_companies(), add_company()
src/companyatlas/commands/seed.py         catlas seed | import-companies | company-add | registry-stats

# Sources

Facts are copied, never inferred: a company only enters the registry with an official website stated on Wikidata; nothing is guessed from names. Provenance is kept per company (source, harvested_at, sitelinks, raw industry_labels, notes) and lands in companies.source_meta.

# Harvest procedure (scripts/seed_wikidata.py)

.venv/bin/python scripts/seed_wikidata.py all --target 8000    # ≈ 45–60 min the first time; every stage resumes from cache
.venv/bin/python scripts/seed_wikidata.py candidates|select|details|assemble   # single stage
.venv/bin/python scripts/seed_edgar.py                         # optional enrichment, rewrites the NDJSON in place
.venv/bin/catlas registry-stats                                # counts without a database

1. candidates — ?item wdt:P31 wd:<class> for 30 company classes (business, enterprise, public company, company, privately held company, corporation, bank, airline, automobile/aerospace manufacturer, software/technology company, video-game developer/publisher, publisher, record label, telecom, brewery, retail chain, railway, shipyard, …). The subclass tree of business (P279*) is far too broad to page, so classes are queried directly and the big ones are chunked by sitelink bands (≥ 80, 50–79, 35–49, 25–34, 18–24, 12–17, 8–11, 5–7); a band that times out is split in two automatically. Minimum 5 sitelinks. Then country boosts (sitelinks ≥ 2, country-first scan, only for minimum-coverage countries still short after the class stage; a timeout is tolerated and reported) and industry boosts (P452 label lists per top-level industry, sitelinks ≥ 2, 600 each). Output data/seed/candidates.json. dissolved — a separate P576 lookup over every candidate (300 QIDs per query; a NOT EXISTS filter inside the big scans is what makes them time out); defunct companies are excluded at selection. Output data/seed/dissolved.json.
2. select — websites normalised to https://<host> (paths stripped); generic hosts dropped (social networks, blogs, app stores, site builders, code forges, Wikipedia …; for platform roots such as google.com/apple.com only the bare/www host counts); duplicates by registrable domain keep the highest-sitelink item (the loser is recorded as a related_domain_conflict note on the winner and in data/seed/dropped.json). Then the diversified selection below. Output data/seed/selected.json (+ a reserve of boost candidates per top-level industry).
3. details — five light queries per batch of 100 QIDs (labels/description/aliases via the label service; inception/coordinates/ LEI/CIK/logo; HQ + parent with end-time qualifiers; legal names + industry labels; tickers/exchanges + employee observations). A GROUP BY with many SAMPLE() aggregates makes Blazegraph throw StackOverflowError, hence plain queries reduced in Python. Output data/seed/details.json (incremental).
4. assemble — industry mapping, importance, tiers, parent/domain conflicts, per-region NDJSON + README.

Gotchas learnt on the endpoint: a query that runs past 60 s may come back as HTTP 200 with a truncated JSON body followed by a Java stack trace (the gateway even caches it) — the client treats an unparsable body as a timeout; some literals contain raw control characters (json.loads(strict=False)); starting a query from P17 of a large country never finishes; FILTER NOT EXISTS over tens of thousands of bindings is what pushes a scan over the limit (fetch as OPTIONAL and filter client-side instead).

# Selection and diversification rules

Candidates are ordered by sitelinks (desc) and selected until --target (8,000) with:

• Caps — United States ≤ 40 % of the target, any other country ≤ 12 %, unknown country ≤ 3 %; narrow Wikidata classes that are over-represented among high-sitelink items are capped too (airlines 5 %, video-game developers 4 %, record labels / publishers / film production / railway companies 3 %, shipyards / breweries / book and game publishers 2 %).
• Minimum coverage (taken first, best-ranked companies of that country, sitelinks ≥ 2): ≥ 150 for CA, GB, DE, FR, JP, KR, IN, AU; ≥ 60 for BR, MX, AE, SA, ZA, NG, SG, ID, NL, SE, CH, ES, IT, CN, TW, HK — when Wikidata has that many companies with a website.
• Industry floor — every top-level industry gets ≥ 60 companies where the data allows, topped up from the industry-boost reserve.
• One company per registrable domain; a subsidiary that shares its parent's domain is dropped (the parent keeps it).

The resulting counts are in registry/companies/README.md and catlas registry-stats; tests/test_registry_files.py enforces the caps (with a small tolerance for the industry top-up), the minimums for the large economies and the tier sizes.

# Importance and tiers

s_sitelinks = min(1, ln(1 + sitelinks) / ln(1 + 300))         # 300 sitelinks ≈ the most-linked companies
s_employees = min(1, log10(1 + employees) / 6)                # 1,000,000 employees → 1
importance  = 0.6·s_sitelinks + 0.2·s_employees + 0.1·public_company + 0.1·has_ticker     (clamped to 0.02–1)
tier        = rank by importance: 1–150 → 1 (global) · 151–950 → 2 (major) · 951–3450 → 3 (notable) · rest → 4 (long tail)

Importance drives crawl priority only (spec §113); it is refreshed on every catlas seed, nothing else about an existing row is.

# NDJSON schema (one object per line)

wikidata_id, display_name, legal_name, aliases[≤8], website (https://host), canonical_domain, country (ISO-2 | null), hq_city, hq_region,
lat, lon, industries[] (taxonomy slugs), industry_labels[] (raw P452 labels, + "SIC …" after EDGAR), founded_year, employees,
public_company, ticker, exchange, lei, sec_cik, parent {wikidata_id, name} | null, logo_url (Commons Special:FilePath), description,
sitelinks, importance (0–1), tier (1–4), source ("wikidata"), harvested_at, notes[] (optional: related_domain_conflict),
source_edgar {cik, enriched_at} (optional)

# Loading (catlas seed)

seed(conn, companies=True, limit=None, files=None) in companyatlas.registry.seed:

1. upserts industries (slug, name, parent, description, keywords, sort order) and countries;
2. for each registry line: match an existing company by wikidata_id, then by canonical_domain;
   • new → id = ids.new_id('company'), slug ids.slugify(display_name) de-duplicated as -<country> then -2, -3…, onboarding_status='pending', provenance in source_meta;
   • existing → only null columns are filled; importance, tier and source_meta provenance are refreshed; names, website, history and anything already set are never overwritten;
3. company_aliases (display name → brand, legal name → legal, ticker → ticker, others → alias; key ids.normalize_alias), domains primary row, company_relationships PARENT_OF / SUBSIDIARY_OF (confidence 0.8, provenance {source, property: P749}) when the parent is also in the registry;
4. queue_jobs(kind='discover', key='discover:<company_id>', priority=importance) for new pending companies (on conflict do nothing);
5. settings_kv['seed:last_run'] with the counters.

Running it twice yields companies_new = 0. --limit N seeds the first N lines (files are ordered tier 1 → 4 inside each region); --no-companies only refreshes the reference tables; --file loads specific NDJSON files.

# Adding companies

.venv/bin/catlas company-add https://www.example.com --name "Example Corp" --country CA --industry software
.venv/bin/catlas import-companies my-companies.csv        # columns: website (required), display_name, country, industries ("a;b"), aliases, …
.venv/bin/catlas import-companies my-companies.ndjson     # same keys as the registry schema

Imports go through the same upsert (idempotent, same slug/alias/queue rules) with source_meta.source = "manual" (--source to change). To add companies to the committed universe, append lines to a registry/companies/*.ndjson file (or a new manual-*.ndjson) with at least website; run pytest tests/test_registry_files.py (validity, no duplicate domain, caps) then catlas seed.

# Tests

.venv/bin/pytest -q tests/test_industry_map.py tests/test_seed_loader.py tests/test_registry_files.py

test_seed_loader.py needs the local database (catlas migrate); it seeds a fixture twice under ztest- slugs and cleans up after itself.

# Harvest of 2026-09-12 (current registry)

• 24,448 candidates from 30 classes + boosts; 2,402 excluded as dissolved (P576); 8,086 companies assembled (target 8,000 + industry top-up); 2,637 public, 2,323 with a ticker, 1,121 with a SEC CIK after EDGAR (242 new matches, 321 rows enriched, 29 industries from SIC).
• Country caps respected (US 22.4 %, next Japan 7.4 %); all 150-minimums met (CA 191, GB 536, DE 557, FR 175, JP 602, KR 156, IN 152, AU 154); 60-minimums met except BR 59, MX 59, ID 58 (one or two companies short after dissolution/domain de-duplication); every top-level industry ≥ 60. 257 companies (3.2 %) have no country on Wikidata; 384 (4.7 %) have no industry mapping.
• Country-first boosts for AU and MX were cut by the endpoint's time limit (tolerated); KR/GB/DE/FR/JP/CA/IN never needed one.
• catlas seed on the local DB: 8,075 new + 11 matched to companies the crawl agent had already created; 20,777 aliases, 2,582 PARENT_OF/SUBSIDIARY_OF rows, 8,075 discover jobs; second run 0 new. About 18 s per run.

# Known gaps

Structural limits:

• Coverage follows Wikidata: companies without an official website there are absent; a few small markets end up one or two short of their 60 minimum; country-first boost queries for larger countries time out on the endpoint (they only matter below 5 sitelinks).
• Industry slugs come from P452 labels and, failing that, the boost query / description keywords; ~5 % of companies have none (the LLM classify_industry job and the crawl fill them later). The class mix over-represents media/entertainment and transport even after the class caps (Wikidata has many well-linked airlines, labels, publishers and game studios).
• EDGAR only fills what is missing; ~1,800 US companies but only ~1,100 CIKs because name matching is exact on the normalised name (no fuzzy guessing, spec §6 "do not merge on similar names").
• assemble regenerates the NDJSON from the Wikidata cache and therefore drops EDGAR fields — always run seed_edgar.py after it.
• hq_region is the P131 parent of the HQ item and is sometimes the country itself.
• Websites are forced to https://; discovery handles redirects and http-only fallbacks.