# Seeds — the initial company universe

The seed registry is the diversified starting universe of Company Atlas (spec §99: 5,000+ companies across sectors and regions; since 2026-09-13 a reference atlas of ≥ 30,000 companies, every listed company first). It is plain data committed in registry/ plus an idempotent loader; the crawl core takes over from companies.onboarding_status = 'pending' and the queue_jobs(kind='discover') rows the loader creates (boundary "Seeds → Crawl" in docs/ARCHITECTURE.md).

registry/industries.yaml              taxonomy (51 industries, 23 top-level + children, keywords for label mapping)
registry/countries.csv                250 ISO-3166-1 alpha-2 codes (+ XK), UN M49 region/subregion, centroids
registry/companies/wikidata-*.ndjson  one JSON object per company, one file per UN region
registry/companies/README.md          counts by tier / region / country / industry (regenerated by the harvester)
scripts/seed_countries.py             regenerates countries.csv
scripts/seed_wikidata.py              Wikidata harvester (candidates → select → details → assemble)
scripts/seed_edgar.py                 SEC EDGAR enrichment (CIK, ticker, exchange, SIC industry hint)
src/companyatlas/registry/industries.py   taxonomy loader, map_industry(labels) → slugs, map_sic(code)
src/companyatlas/registry/seed.py         loader: seed(), import_companies(), add_company()
src/companyatlas/commands/seed.py         catlas seed | import-companies | company-add | registry-stats

# Sources

Facts are copied, never inferred: a company only enters the registry with an official website stated on Wikidata; nothing is guessed from names. Provenance is kept per company (source, harvested_at, sitelinks, raw industry_labels, notes) and lands in companies.source_meta. Per-stage counters live in data/seed/harvest-stats.json and are rendered as the harvest report at the end of registry/companies/README.md (seed_edgar.py appends its own line).

# Harvest procedure (scripts/seed_wikidata.py)

nohup .venv/bin/python scripts/seed_wikidata.py all --target 30000 > data/seed/harvest.log 2>&1 &   # ≈ 1.5–2 h the first time,
                                                                # minutes when the cache is warm; every stage resumes from cache
.venv/bin/python scripts/seed_wikidata.py candidates|dissolved|select|details|assemble   # single stage
.venv/bin/python scripts/seed_edgar.py                         # enrichment, rewrites the NDJSON in place — always after assemble
.venv/bin/catlas registry-stats                                # counts without a database

1. candidates — four band families, every row needing a truthy official website (P856):
   • classes × sitelink bands — ?item wdt:P31 wd:<class> for 44 company classes (business, enterprise, public company, company, privately held company, corporation, joint-stock company, kabushiki / kōkai gaisha, holding company, financial institution, credit institution, insurance company, REIT, subsidiary, state-owned enterprise, multinational, bank, airline, automobile / commercial-vehicle / aerospace manufacturer, software / technology company, video-game developer / publisher, publishing house, book publisher, record label, film production company, telecom, brewery, retail / restaurant / hotel / gas-station chain, railway, transport company, law firm, printing / pharmaceutical company, concern, conglomerate, shipyard). The subclass tree of business (P279*) is far too broad to page, so classes are queried directly; the big ones are chunked by sitelink bands (≥ 80, 50–79, 35–49, 25–34, 18–24, 12–17, 8–11, 5–7, 3–4), the others in two queries (≥ 5 and 3–4). A band that times out is split in two, then by the last QID digit. The class list is checked against the English Wikidata labels — the 2026-09-12 list had six wrong labels and three classes that were not companies (charitable organization, service on Internet, online service), now removed.
   • listed companies — one SELECT ?ex (COUNT …) GROUP BY ?ex lists the ~250 stock exchanges used as P414 values; then every exchange is scanned (p:P414 statements with their P582 end-time qualifier, any P31 class), large exchanges alone, small ones grouped to ≈ 1,500 statements per query. An item whose listings have all ended counts as not listed (delisted) but stays a candidate through its other bands. Plus the few dozen items carrying P249 directly.
   • large companies — per class, P1128 ≥ 500 employees and P2139 revenue present, any sitelink count (class-restricted on purpose: 70 % of the items with a revenue on Wikidata are nonprofits, universities, hospitals or municipalities).
   • boosts — country-first scans (sitelinks ≥ 2, only for minimum-coverage countries still short after the bands above; a timeout is tolerated and reported) and industry boosts (P452 label lists per top-level industry, sitelinks ≥ 2, 600 each). Output data/seed/candidates.json (per item: sitelinks, websites, countries, classes, bands, listed, exchanges, employee hint). dissolved — a separate P576 | P582 lookup over every candidate (300 QIDs per query; a NOT EXISTS filter inside the big scans is what makes them time out); defunct companies are excluded at selection. Output data/seed/dissolved.json.
2. select — websites normalised to https://<host> (paths stripped); generic hosts dropped (social networks, blogs, app stores, site builders, code forges, Wikipedia …; for platform roots such as google.com/apple.com only the bare/www host counts); duplicates by registrable domain keep the highest-sitelink item (the loser is recorded as a related_domain_conflict note on the winner and in data/seed/dropped.json). Then the diversified selection below. Output data/seed/selected.json (+ a reserve of boost candidates per top-level industry).
3. details — five light queries per batch of 100 QIDs (labels/description/aliases via the label service; inception/coordinates/ LEI/CIK/logo; HQ + parent with end-time qualifiers; legal names + industry labels; tickers/exchanges + employee observations), then the full P31 class list of every item (300 QIDs per query) for the non-company exclusion below. A GROUP BY with many SAMPLE() aggregates makes Blazegraph throw StackOverflowError, hence plain queries reduced in Python. Output data/seed/details.json (incremental).
4. assemble — non-company exclusion, industry mapping, importance, tiers, parent/domain conflicts, per-region NDJSON + README (tables + the harvest report rendered from data/seed/harvest-stats.json, which every stage updates).

# Non-company exclusion (assemble)

Wikidata's company classes also carry museums, universities, charities, football clubs or YouTube channels (an item may be business and museum). An item is dropped, and counted in the README harvest report, when:

• one of its P31 classes is in NON_COMPANY_CLASSES (scripts/seed_wikidata.py): museums / libraries / archives, universities / schools / colleges, government agencies / ministries, NGOs / charities / foundations / nonprofits / associations / think tanks, religious organisations, political parties, labor unions, hospitals, sports clubs and teams, YouTube channels — unless it has a current stock-exchange listing or a ticker (listed clubs such as Manchester United or Juventus, listed hospital groups stay);
• or its description matches museum|university|school|ministry|agency|charity|foundation|association|club|church|channel without a company word (company, manufacturer, retailer, bank, chain, brand, … advertising/travel/news/… agency) and it has no ticker, exchange, employee count or revenue.

The dropped items are listed in data/seed/dropped.json with their reason (non_company_class:<QID> / non_company_description). Because the rule runs after selection, select slightly above the goal (--target 31500 for 30,000) and check the assembled count.

Gotchas learnt on the endpoint: a query that runs past 60 s may come back as HTTP 200 with a truncated JSON body followed by a Java stack trace (the gateway even caches it) — the client treats an unparsable body as a timeout; some literals contain raw control characters (json.loads(strict=False)); starting a query from P17 of a large country never finishes; FILTER NOT EXISTS over tens of thousands of bindings is what pushes a scan over the limit (fetch as OPTIONAL and filter client-side instead).

# Selection and diversification rules

Candidates are ordered by sitelinks (desc) and selected until --target (30,000) in passes — country minimums, then every currently listed company, then the rest by sitelinks ≥ 3, then (only if the target is still short) sitelinks ≥ 2 — with:

• Eligibility — listed companies and large companies (employees / revenue bands) qualify at any sitelink count (many small caps on the Indonesian, Thai, Malaysian or Japanese exchanges have no Wikipedia article at all); class-band and boost candidates need ≥ 3 (≥ 2 in the fallback pass).
• Caps — United States ≤ 35 % of the target, any other country ≤ 12 %, unknown country ≤ 3 %; narrow Wikidata classes that are over-represented among high-sitelink items are capped too (airlines 5 %, video-game developers 4 %, record labels / publishers / film production / railway companies 3 %, shipyards / breweries / book and game publishers 2 %).
• Minimum coverage (taken first, best-ranked companies of that country): ≥ 150 for CA, GB, DE, FR, JP, KR, IN, AU; ≥ 60 for BR, MX, AE, SA, ZA, NG, SG, ID, NL, SE, CH, ES, IT, CN, TW, HK — when Wikidata has that many companies with a website.
• Industry floor — every top-level industry gets ≥ 60 companies where the data allows, topped up from the industry-boost reserve.
• One company per registrable domain; a subsidiary that shares its parent's domain is dropped (the parent keeps it).

The resulting counts are in registry/companies/README.md and catlas registry-stats; tests/test_registry_files.py enforces the caps (with a small tolerance for the industry top-up), the minimums for the large economies and the tier sizes.

# Importance and tiers

s_sitelinks = min(1, ln(1 + sitelinks) / ln(1 + 300))         # 300 sitelinks ≈ the most-linked companies
s_employees = min(1, log10(1 + employees) / 6)                # 1,000,000 employees → 1
importance  = 0.6·s_sitelinks + 0.2·s_employees + 0.1·public_company + 0.1·has_ticker     (clamped to 0.02–1)
tier        = quantile of the importance rank: top 1 % → 1 (global, ≈ 300 at 30 k) · next 6.67 % → 2 (major, ≈ 2,000) ·
              next 26.67 % → 3 (notable, ≈ 8,000) · rest → 4 (long tail)          (TIER_SHARES in scripts/seed_wikidata.py)

public_company is true for the public company class, for any current stock-exchange listing (P414 without an end time) and for any ticker; a listing without a ticker qualifier keeps the exchange with ticker = null. Importance drives crawl priority only (spec §113); it is refreshed on every catlas seed, nothing else about an existing row is.

# NDJSON schema (one object per line)

wikidata_id, display_name, legal_name, aliases[≤8], website (https://host), canonical_domain, country (ISO-2 | null), hq_city, hq_region,
lat, lon, industries[] (taxonomy slugs), industry_labels[] (raw P452 labels, + "SIC …" after EDGAR), founded_year, employees,
public_company, ticker, exchange, lei, sec_cik, parent {wikidata_id, name} | null, logo_url (Commons Special:FilePath), description,
sitelinks, importance (0–1), tier (1–4), source ("wikidata"), harvested_at, notes[] (optional: related_domain_conflict),
source_edgar {cik, enriched_at} (optional)

# Loading (catlas seed)

seed(conn, companies=True, limit=None, files=None) in companyatlas.registry.seed:

1. upserts industries (slug, name, parent, description, keywords, sort order) and countries;
2. for each registry line: match an existing company by wikidata_id, then by canonical_domain;
   • new → id = ids.new_id('company'), slug ids.slugify(display_name) de-duplicated as -<country> then -2, -3…, onboarding_status='pending', provenance in source_meta;
   • existing → only null columns are filled; importance, tier and source_meta provenance are refreshed; names, website, history and anything already set are never overwritten;
3. company_aliases (display name → brand, legal name → legal, ticker → ticker, others → alias; key ids.normalize_alias), domains primary row, company_relationships PARENT_OF / SUBSIDIARY_OF (confidence 0.8, provenance {source, property: P749}) when the parent is also in the registry;
4. queue_jobs(kind='discover', key='discover:<company_id>', priority=importance) for new pending companies (on conflict do nothing);
5. settings_kv['seed:last_run'] with the counters.

Running it twice yields companies_new = 0. --limit N seeds the first N lines (files are ordered tier 1 → 4 inside each region); --no-companies only refreshes the reference tables; --file loads specific NDJSON files.

# Adding companies

.venv/bin/catlas company-add https://www.example.com --name "Example Corp" --country CA --industry software
.venv/bin/catlas import-companies my-companies.csv        # columns: website (required), display_name, country, industries ("a;b"), aliases, …
.venv/bin/catlas import-companies my-companies.ndjson     # same keys as the registry schema

Imports go through the same upsert (idempotent, same slug/alias/queue rules) with source_meta.source = "manual" (--source to change). To add companies to the committed universe, append lines to a registry/companies/*.ndjson file (or a new manual-*.ndjson) with at least website; run pytest tests/test_registry_files.py (validity, no duplicate domain, caps) then catlas seed.

# Tests

.venv/bin/pytest -q tests/test_industry_map.py tests/test_seed_loader.py tests/test_registry_files.py

test_seed_loader.py needs the local database (catlas migrate); it seeds a fixture twice under ztest- slugs and cleans up after itself.

# Harvest of 2026-09-13 (current registry, 30 k scale-up)

• 51,286 candidates with a website from 44 P31 classes × sitelink bands ≥ 3 (39,684 first seen there), 250 stock exchanges scanned in 13 P414 queries (13,541 currently listed items; 7,664 first seen there), employees ≥ 500 / revenue bands per class (1,723 + 1,748), industry boosts 354, country boosts 103. 854 live SPARQL queries + 418 cached, 5,306 s (≈ 1 h 30, including ~10 min of 429 back-off); the whole candidate stage is now cached, so a re-run costs only the new detail batches.
• Exclusions: 3,580 dissolved (P576 / P582); 89 generic / invalid websites; 1,639 duplicate registrable domains (1,458 winners carry a related_domain_conflict note); 652 non-companies removed at assemble (515 by P31 class — 170 nonprofit organisations, 40 government agencies, 30 universities, 27 museums, 20 learned societies, …; 137 by description).
• Selection: target 33,000 (a margin for the non-company rule) in passes — country minimums 2,160, 11,964 listed companies, 18,876 by sitelinks ≥ 3 (the ≥ 2 fallback was not needed) → 32,368 companies assembled, 13,209 public (40.8 %), 12,781 with an exchange, 10,969 with a ticker, 4,364 with an employee count; 206 countries. Sitelink profile: 17,402 ≥ 5, 8,048 at 3–4, 1,323 at 2, 5,595 at 0–1 (listed / large companies without a Wikipedia article).
• Caps respected (US 26.5 %, next Japan 10.4 %, Germany 7.1 %); every country minimum met (JP 3,382, DE 2,310, GB 1,548, CN 1,374, KR 846, CA 723, …); every top-level industry ≥ 60 without any top-up. 990 companies (3.1 %) have no country on Wikidata; 5,294 (16.4 %) have no industry mapping before EDGAR (see Known gaps).
• Tiers (quantiles): 324 / 2,159 / 8,633 / 21,252.
• Exchanges with the most companies: Tokyo 1,956, Nasdaq 1,569, NYSE 1,562, Bursa Malaysia 780, Indonesia 630, Hong Kong 620, Tel Aviv 532, London 477, Korea Exchange 368.
• EDGAR: 920 companies newly matched to a CIK, 2,543 already had one → 3,463 with a SEC CIK (6,130 with an LEI); 1,319 rows enriched, 202 industries from SIC (5,092 = 15.7 % still without an industry mapping), 816 submissions read (~9 min). After EDGAR: 13,311 public (41.1 %), 12,892 with an exchange, 11,120 with a ticker.
• catlas seed on the local DB: 24,871 new companies (pending, one discover job each) + 7,497 matched to rows of the 2026-09-12 seed by wikidata_id / domain; 69,882 aliases, 32,368 primary domains, 4,644 PARENT_OF / SUBSIDIARY_OF rows; second run 0 new, 32,368 updated (importance / tier / provenance only). About 18 s per run. The 589 rows of the previous seed that are no longer in the registry (mislabelled classes, non-company rule) stay in the database untouched — the loader never deletes.

# Known gaps

Structural limits:

• Coverage follows Wikidata: companies without an official website there are absent; country-first boost queries for larger countries time out on the endpoint (they no longer matter: every minimum is met from the bands).
• Industry slugs come from P452 labels and, failing that, the boost query / description / class keywords; ~16 % of companies have none — almost all of them listed small caps (Tokyo, Bursa Malaysia, Korea Exchange, Tel Aviv, Shenzhen …) whose Wikidata item has no P452 and a bare "public company" description; the LLM classify_industry job and the crawl fill them later. Only ~360 carry a P452 label the taxonomy does not know (registry/industries.yaml keywords were extended for the commonest ones on 2026-09-13).
• ~5,600 companies have 0–1 sitelinks (no Wikipedia article): names may be the Wikidata label in the local script or, when the item has no English label at all, the registrable domain (e.g. sinoair.com); discovery and the crawl replace those with the site's own name.
• The non-company exclusion is a class + description heuristic: a listed sports club or a company whose item carries a stray nonprofit class is kept when it has a listing / ticker; unlisted cooperatives typed only as nonprofit organization are dropped.
• EDGAR only fills what is missing; name matching is exact on the normalised name (no fuzzy guessing, spec §6 "do not merge on similar names"), so a large share of the ~8,500 US companies keep no CIK.
• assemble regenerates the NDJSON from the Wikidata cache and therefore drops EDGAR fields — always run seed_edgar.py after it.
• hq_region is the P131 parent of the HQ item and is sometimes the country itself.
• Websites are forced to https://; discovery handles redirects and http-only fallbacks.