spb/company-atlas

1# Seeds — the initial company universe23The seed registry is the diversified starting universe of Company Atlas (spec §99: 5,000+ companies across sectors and regions; since42026-09-13 a reference atlas of **≥ 30,000 companies, every listed company first**). It is plain data committed in `registry/` plus an5idempotent loader; the crawl core takes over from `companies.onboarding_status = 'pending'` and the `queue_jobs(kind='discover')` rows6the loader creates (boundary "Seeds → Crawl" in `docs/ARCHITECTURE.md`).78```9registry/industries.yaml              taxonomy (51 industries, 23 top-level + children, keywords for label mapping)10registry/countries.csv                250 ISO-3166-1 alpha-2 codes (+ XK), UN M49 region/subregion, centroids11registry/companies/wikidata-*.ndjson  one JSON object per company, one file per UN region12registry/companies/README.md          counts by tier / region / country / industry (regenerated by the harvester)13scripts/seed_countries.py             regenerates countries.csv14scripts/seed_wikidata.py              Wikidata harvester (candidates → select → details → assemble)15scripts/seed_edgar.py                 SEC EDGAR enrichment (CIK, ticker, exchange, SIC industry hint)16src/companyatlas/registry/industries.py   taxonomy loader, map_industry(labels) → slugs, map_sic(code)17src/companyatlas/registry/seed.py         loader: seed(), import_companies(), add_company()18src/companyatlas/commands/seed.py         catlas seed | import-companies | company-add | registry-stats19```2021## Sources2223| Source | What we take | Politeness |24|---|---|---|25| **Wikidata** (SPARQL, `query.wikidata.org`) | companies by class (P31) × sitelink bands, every company listed on a stock exchange (P414, per exchange; P249), companies with ≥ 500 employees (P1128) or a revenue (P2139) per class; official website P856, country P17→P297, labels/aliases/descriptions, official name P1448, industry P452, inception P571, employees P1128 (+P585), HQ P159 (+P625, P131, P17), coordinates P625, ticker P249 / exchange P414, LEI P1278, CIK P5531, parent P749, logo P154, dissolution P576 / P582, sitelink count | UA `CompanyAtlasBot/0.1 (contact@spboucher.ai)`, one query at a time, ≥ 2 s apart, 60 s server timeout, retries with exponential backoff (429 honours `Retry-After`), every result cached in `data/seed/wikidata/sparql/<sha256>.json` (git-ignored) |26| **SEC EDGAR** (`company_tickers.json`, `data.sec.gov/submissions`) | CIK, tickers, exchanges, SIC + description, business address | required UA, ≤ 5 req/s, cached in `data/seed/edgar/` |27| **ISO 3166 / UN M49** (lukes/ISO-3166-Countries-with-Regional-Codes) + Wikidata P625 | country names, regions, centroids | one-off; the CSV is committed |2829Facts are copied, never inferred: a company only enters the registry with an official website *stated on Wikidata*; nothing is guessed30from names. Provenance is kept per company (`source`, `harvested_at`, `sitelinks`, raw `industry_labels`, `notes`) and lands in31`companies.source_meta`. Per-stage counters live in `data/seed/harvest-stats.json` and are rendered as the harvest report at the end of32`registry/companies/README.md` (`seed_edgar.py` appends its own line).3334## Harvest procedure (`scripts/seed_wikidata.py`)3536```bash37nohup .venv/bin/python scripts/seed_wikidata.py all --target 30000 > data/seed/harvest.log 2>&1 &   # ≈ 1.5–2 h the first time,38                                                                # minutes when the cache is warm; every stage resumes from cache39.venv/bin/python scripts/seed_wikidata.py candidates|dissolved|select|details|assemble   # single stage40.venv/bin/python scripts/seed_edgar.py                         # enrichment, rewrites the NDJSON in place — always after assemble41.venv/bin/catlas registry-stats                                # counts without a database42```43441. **candidates** — four band families, every row needing a truthy official website (P856):45   * **classes × sitelink bands** — `?item wdt:P31 wd:<class>` for 44 company classes (business, enterprise, public company, company,46     privately held company, corporation, joint-stock company, kabushiki / kōkai gaisha, holding company, financial institution, credit47     institution, insurance company, REIT, subsidiary, state-owned enterprise, multinational, bank, airline, automobile / commercial-vehicle /48     aerospace manufacturer, software / technology company, video-game developer / publisher, publishing house, book publisher, record49     label, film production company, telecom, brewery, retail / restaurant / hotel / gas-station chain, railway, transport company, law50     firm, printing / pharmaceutical company, concern, conglomerate, shipyard). The subclass tree of *business* (`P279*`) is far too51     broad to page, so classes are queried directly; the big ones are chunked by **sitelink bands** (≥ 80, 50–79, 35–49, 25–34, 18–24,52     12–17, 8–11, 5–7, **3–4**), the others in two queries (≥ 5 and 3–4). A band that times out is split in two, then by the last QID53     digit. The class list is checked against the English Wikidata labels — the 2026-09-12 list had six wrong labels and three classes54     that were not companies (charitable organization, service on Internet, online service), now removed.55   * **listed companies** — one `SELECT ?ex (COUNT …) GROUP BY ?ex` lists the ~250 stock exchanges used as P414 values; then every56     exchange is scanned (`p:P414` statements with their P582 end-time qualifier, any P31 class), large exchanges alone, small ones grouped57     to ≈ 1,500 statements per query. An item whose listings have all ended counts as *not listed* (delisted) but stays a candidate through58     its other bands. Plus the few dozen items carrying P249 directly.59   * **large companies** — per class, `P1128 ≥ 500` employees and `P2139` revenue present, any sitelink count (class-restricted on60     purpose: 70 % of the items with a revenue on Wikidata are nonprofits, universities, hospitals or municipalities).61   * **boosts** — country-first scans (sitelinks ≥ 2, only for minimum-coverage countries still short after the bands above; a timeout is62     tolerated and reported) and industry boosts (P452 label lists per top-level industry, sitelinks ≥ 2, 600 each).63   Output `data/seed/candidates.json` (per item: sitelinks, websites, countries, classes, bands, `listed`, exchanges, employee hint).64   **dissolved** — a separate `P576 | P582` lookup over every candidate (300 QIDs per query; a `NOT EXISTS` filter inside the big scans65   is what makes them time out); defunct companies are excluded at selection. Output `data/seed/dissolved.json`.662. **select** — websites normalised to `https://<host>` (paths stripped); generic hosts dropped (social networks, blogs, app stores,67   site builders, code forges, Wikipedia …; for platform roots such as `google.com`/`apple.com` only the bare/www host counts);68   duplicates by registrable domain keep the highest-sitelink item (the loser is recorded as a `related_domain_conflict` note on the69   winner and in `data/seed/dropped.json`). Then the diversified selection below. Output `data/seed/selected.json` (+ a reserve of70   boost candidates per top-level industry).713. **details** — five light queries per batch of 100 QIDs (labels/description/aliases via the label service; inception/coordinates/72   LEI/CIK/logo; HQ + parent with end-time qualifiers; legal names + industry labels; tickers/exchanges + employee observations), then73   the full **P31 class list** of every item (300 QIDs per query) for the non-company exclusion below.74   A `GROUP BY` with many `SAMPLE()` aggregates makes Blazegraph throw `StackOverflowError`, hence plain queries reduced in Python.75   Output `data/seed/details.json` (incremental).764. **assemble** — **non-company exclusion**, industry mapping, importance, tiers, parent/domain conflicts, per-region NDJSON + README77   (tables + the harvest report rendered from `data/seed/harvest-stats.json`, which every stage updates).7879### Non-company exclusion (assemble)8081Wikidata's company classes also carry museums, universities, charities, football clubs or YouTube channels (an item may be `business`82*and* `museum`). An item is dropped, and counted in the README harvest report, when:8384* one of its P31 classes is in `NON_COMPANY_CLASSES` (`scripts/seed_wikidata.py`): museums / libraries / archives, universities / schools /85  colleges, government agencies / ministries, NGOs / charities / foundations / nonprofits / associations / think tanks, religious86  organisations, political parties, labor unions, hospitals, sports clubs and teams, YouTube channels — **unless it has a current87  stock-exchange listing or a ticker** (listed clubs such as Manchester United or Juventus, listed hospital groups stay);88* or its description matches `museum|university|school|ministry|agency|charity|foundation|association|club|church|channel` without a89  company word (`company, manufacturer, retailer, bank, chain, brand, … advertising/travel/news/… agency`) **and** it has no ticker, exchange,90  employee count or revenue.9192The dropped items are listed in `data/seed/dropped.json` with their reason (`non_company_class:<QID>` / `non_company_description`).93Because the rule runs after selection, select slightly above the goal (`--target 31500` for 30,000) and check the assembled count.9495Gotchas learnt on the endpoint: a query that runs past 60 s may come back as **HTTP 200 with a truncated JSON body followed by a Java96stack trace** (the gateway even caches it) — the client treats an unparsable body as a timeout; some literals contain raw control97characters (`json.loads(strict=False)`); starting a query from `P17` of a large country never finishes; `FILTER NOT EXISTS` over tens of98thousands of bindings is what pushes a scan over the limit (fetch as OPTIONAL and filter client-side instead).99100## Selection and diversification rules101102Candidates are ordered by sitelinks (desc) and selected until `--target` (30,000) in passes — country minimums, then **every currently103listed company**, then the rest by sitelinks ≥ 3, then (only if the target is still short) sitelinks ≥ 2 — with:104105* **Eligibility** — listed companies and large companies (employees / revenue bands) qualify at any sitelink count (many small caps on106  the Indonesian, Thai, Malaysian or Japanese exchanges have no Wikipedia article at all); class-band and boost candidates need ≥ 3107  (≥ 2 in the fallback pass).108* **Caps** — United States ≤ 35 % of the target, any other country ≤ 12 %, unknown country ≤ 3 %; narrow Wikidata classes that are109  over-represented among high-sitelink items are capped too (airlines 5 %, video-game developers 4 %, record labels / publishers /110  film production / railway companies 3 %, shipyards / breweries / book and game publishers 2 %).111* **Minimum coverage** (taken first, best-ranked companies of that country): ≥ 150 for CA, GB, DE, FR, JP, KR, IN, AU;112  ≥ 60 for BR, MX, AE, SA, ZA, NG, SG, ID, NL, SE, CH, ES, IT, CN, TW, HK — when Wikidata has that many companies with a website.113* **Industry floor** — every top-level industry gets ≥ 60 companies where the data allows, topped up from the industry-boost reserve.114* **One company per registrable domain**; a subsidiary that shares its parent's domain is dropped (the parent keeps it).115116The resulting counts are in `registry/companies/README.md` and `catlas registry-stats`; `tests/test_registry_files.py` enforces the117caps (with a small tolerance for the industry top-up), the minimums for the large economies and the tier sizes.118119## Importance and tiers120121```122s_sitelinks = min(1, ln(1 + sitelinks) / ln(1 + 300))         # 300 sitelinks ≈ the most-linked companies123s_employees = min(1, log10(1 + employees) / 6)                # 1,000,000 employees → 1124importance  = 0.6·s_sitelinks + 0.2·s_employees + 0.1·public_company + 0.1·has_ticker     (clamped to 0.02–1)125tier        = quantile of the importance rank: top 1 % → 1 (global, ≈ 300 at 30 k) · next 6.67 % → 2 (major, ≈ 2,000) ·126              next 26.67 % → 3 (notable, ≈ 8,000) · rest → 4 (long tail)          (TIER_SHARES in scripts/seed_wikidata.py)127```128129`public_company` is true for the *public company* class, for any current stock-exchange listing (P414 without an end time) and for any130ticker; a listing without a ticker qualifier keeps the exchange with `ticker = null`. Importance drives crawl priority only (spec §113);131it is refreshed on every `catlas seed`, nothing else about an existing row is.132133## NDJSON schema (one object per line)134135```136wikidata_id, display_name, legal_name, aliases[≤8], website (https://host), canonical_domain, country (ISO-2 | null), hq_city, hq_region,137lat, lon, industries[] (taxonomy slugs), industry_labels[] (raw P452 labels, + "SIC …" after EDGAR), founded_year, employees,138public_company, ticker, exchange, lei, sec_cik, parent {wikidata_id, name} | null, logo_url (Commons Special:FilePath), description,139sitelinks, importance (0–1), tier (1–4), source ("wikidata"), harvested_at, notes[] (optional: related_domain_conflict),140source_edgar {cik, enriched_at} (optional)141```142143## Loading (`catlas seed`)144145`seed(conn, companies=True, limit=None, files=None)` in `companyatlas.registry.seed`:1461471. upserts `industries` (slug, name, parent, description, keywords, sort order) and `countries`;1482. for each registry line: match an existing company by `wikidata_id`, then by `canonical_domain`;149   * new → `id = ids.new_id('company')`, slug `ids.slugify(display_name)` de-duplicated as `-<country>` then `-2`, `-3`…,150     `onboarding_status='pending'`, provenance in `source_meta`;151   * existing → only **null** columns are filled; `importance`, `tier` and `source_meta` provenance are refreshed; names, website,152     history and anything already set are never overwritten;1533. `company_aliases` (display name → `brand`, legal name → `legal`, ticker → `ticker`, others → `alias`; key `ids.normalize_alias`),154   `domains` primary row, `company_relationships` PARENT_OF / SUBSIDIARY_OF (confidence 0.8, provenance `{source, property: P749}`)155   when the parent is also in the registry;1564. `queue_jobs(kind='discover', key='discover:<company_id>', priority=importance)` for new pending companies (`on conflict do nothing`);1575. `settings_kv['seed:last_run']` with the counters.158159Running it twice yields `companies_new = 0`. `--limit N` seeds the first N lines (files are ordered tier 1 → 4 inside each region);160`--no-companies` only refreshes the reference tables; `--file` loads specific NDJSON files.161162## Adding companies163164```bash165.venv/bin/catlas company-add https://www.example.com --name "Example Corp" --country CA --industry software166.venv/bin/catlas import-companies my-companies.csv        # columns: website (required), display_name, country, industries ("a;b"), aliases, …167.venv/bin/catlas import-companies my-companies.ndjson     # same keys as the registry schema168```169170Imports go through the same upsert (idempotent, same slug/alias/queue rules) with `source_meta.source = "manual"` (`--source` to change).171To add companies to the *committed* universe, append lines to a `registry/companies/*.ndjson` file (or a new `manual-*.ndjson`) with at172least `website`; run `pytest tests/test_registry_files.py` (validity, no duplicate domain, caps) then `catlas seed`.173174## Tests175176```bash177.venv/bin/pytest -q tests/test_industry_map.py tests/test_seed_loader.py tests/test_registry_files.py178```179180`test_seed_loader.py` needs the local database (`catlas migrate`); it seeds a fixture twice under `ztest-` slugs and cleans up after itself.181182## Harvest of 2026-09-13 (current registry, 30 k scale-up)183184* **51,286 candidates** with a website from 44 P31 classes × sitelink bands ≥ 3 (39,684 first seen there), 250 stock exchanges scanned in185  13 P414 queries (13,541 currently listed items; 7,664 first seen there), employees ≥ 500 / revenue bands per class (1,723 + 1,748),186  industry boosts 354, country boosts 103. 854 live SPARQL queries + 418 cached, 5,306 s (≈ 1 h 30, including ~10 min of `429`187  back-off); the whole candidate stage is now cached, so a re-run costs only the new detail batches.188* Exclusions: 3,580 dissolved (P576 / P582); 89 generic / invalid websites; 1,639 duplicate registrable domains (1,458 winners carry a189  `related_domain_conflict` note); **652 non-companies** removed at assemble (515 by P31 class — 170 nonprofit organisations, 40 government190  agencies, 30 universities, 27 museums, 20 learned societies, …; 137 by description).191* Selection: target 33,000 (a margin for the non-company rule) in passes — country minimums 2,160, **11,964 listed companies**, 18,876 by192  sitelinks ≥ 3 (the ≥ 2 fallback was not needed) → **32,368 companies** assembled, 13,209 public (40.8 %), 12,781 with an exchange,193  10,969 with a ticker, 4,364 with an employee count; 206 countries. Sitelink profile: 17,402 ≥ 5, 8,048 at 3–4, 1,323 at 2, 5,595 at194  0–1 (listed / large companies without a Wikipedia article).195* Caps respected (US 26.5 %, next Japan 10.4 %, Germany 7.1 %); every country minimum met (JP 3,382, DE 2,310, GB 1,548, CN 1,374,196  KR 846, CA 723, …); every top-level industry ≥ 60 without any top-up. 990 companies (3.1 %) have no country on Wikidata;197  5,294 (16.4 %) have no industry mapping before EDGAR (see Known gaps).198* Tiers (quantiles): 324 / 2,159 / 8,633 / 21,252.199* Exchanges with the most companies: Tokyo 1,956, Nasdaq 1,569, NYSE 1,562, Bursa Malaysia 780, Indonesia 630, Hong Kong 620,200  Tel Aviv 532, London 477, Korea Exchange 368.201* EDGAR: 920 companies newly matched to a CIK, 2,543 already had one → 3,463 with a SEC CIK (6,130 with an LEI); 1,319 rows enriched,202  202 industries from SIC (5,092 = 15.7 % still without an industry mapping), 816 submissions read (~9 min). After EDGAR: 13,311 public203  (41.1 %), 12,892 with an exchange, 11,120 with a ticker.204* `catlas seed` on the local DB: 24,871 new companies (`pending`, one `discover` job each) + 7,497 matched to rows of the 2026-09-12205  seed by `wikidata_id` / domain; 69,882 aliases, 32,368 primary domains, 4,644 PARENT_OF / SUBSIDIARY_OF rows; second run 0 new,206  32,368 updated (importance / tier / provenance only). About 18 s per run. The 589 rows of the previous seed that are no longer in the207  registry (mislabelled classes, non-company rule) stay in the database untouched — the loader never deletes.208209## Known gaps210211Structural limits:212213* Coverage follows Wikidata: companies without an official website there are absent; country-first boost queries for larger countries214  time out on the endpoint (they no longer matter: every minimum is met from the bands).215* Industry slugs come from P452 labels and, failing that, the boost query / description / class keywords; **~16 % of companies have216  none** — almost all of them listed small caps (Tokyo, Bursa Malaysia, Korea Exchange, Tel Aviv, Shenzhen …) whose Wikidata item has no217  P452 and a bare "public company" description; the LLM `classify_industry` job and the crawl fill them later. Only ~360 carry a P452218  label the taxonomy does not know (`registry/industries.yaml` keywords were extended for the commonest ones on 2026-09-13).219* ~5,600 companies have 0–1 sitelinks (no Wikipedia article): names may be the Wikidata label in the local script or, when the item has220  no English label at all, the registrable domain (e.g. `sinoair.com`); discovery and the crawl replace those with the site's own name.221* The non-company exclusion is a class + description heuristic: a listed sports club or a company whose item carries a stray nonprofit222  class is kept when it has a listing / ticker; unlisted cooperatives typed only as `nonprofit organization` are dropped.223* EDGAR only fills what is missing; name matching is exact on the normalised name (no fuzzy guessing, spec §6 "do not merge on similar224  names"), so a large share of the ~8,500 US companies keep no CIK.225* `assemble` regenerates the NDJSON from the Wikidata cache and therefore drops EDGAR fields — always run `seed_edgar.py` after it.226* `hq_region` is the P131 parent of the HQ item and is sometimes the country itself.227* Websites are forced to `https://`; discovery handles redirects and http-only fallbacks.228