SPB Git
spboucher.ai

spb/company-atlas

Public

seeds: industries taxonomy, countries, Wikidata/EDGAR harvesters, idempotent catlas seed

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 12, 2026) parent b8a5c1a Browse files

12 changed files +2,972 −0

12 changed files
Load all 12 files Unified Split
added docs/SEEDS.md +147 −0
@@ -0,0 +1,147 @@
1 +# Seeds — the initial company universe
2 +
3 +The seed registry is the diversified starting universe of Company Atlas (spec §99: 5,000+ companies across sectors and regions). It is
4 +plain data committed in `registry/` plus an idempotent loader; the crawl core takes over from `companies.onboarding_status = 'pending'`
5 +and the `queue_jobs(kind='discover')` rows the loader creates (boundary "Seeds → Crawl" in `docs/ARCHITECTURE.md`).
6 +
7 +```
8 +registry/industries.yaml taxonomy (51 industries, 23 top-level + children, keywords for label mapping)
9 +registry/countries.csv 250 ISO-3166-1 alpha-2 codes (+ XK), UN M49 region/subregion, centroids
10 +registry/companies/wikidata-*.ndjson one JSON object per company, one file per UN region
11 +registry/companies/README.md counts by tier / region / country / industry (regenerated by the harvester)
12 +scripts/seed_countries.py regenerates countries.csv
13 +scripts/seed_wikidata.py Wikidata harvester (candidates → select → details → assemble)
14 +scripts/seed_edgar.py SEC EDGAR enrichment (CIK, ticker, exchange, SIC industry hint)
15 +src/companyatlas/registry/industries.py taxonomy loader, map_industry(labels) → slugs, map_sic(code)
16 +src/companyatlas/registry/seed.py loader: seed(), import_companies(), add_company()
17 +src/companyatlas/commands/seed.py catlas seed | import-companies | company-add | registry-stats
18 +```
19 +
20 +## Sources
21 +
22 +| Source | What we take | Politeness |
23 +|---|---|---|
24 +| **Wikidata** (SPARQL, `query.wikidata.org`) | companies by class (P31), official website P856, country P17→P297, labels/aliases/descriptions, official name P1448, industry P452, inception P571, employees P1128 (+P585), HQ P159 (+P625, P131, P17), coordinates P625, ticker P249 / exchange P414, LEI P1278, CIK P5531, parent P749, logo P154, sitelink count | UA `CompanyAtlasBot/0.1 (contact@spboucher.ai)`, one query at a time, ≥ 2 s apart, 60 s server timeout, retries with exponential backoff, every result cached in `data/seed/wikidata/sparql/<sha256>.json` (git-ignored) |
25 +| **SEC EDGAR** (`company_tickers.json`, `data.sec.gov/submissions`) | CIK, tickers, exchanges, SIC + description, business address | required UA, ≤ 5 req/s, cached in `data/seed/edgar/` |
26 +| **ISO 3166 / UN M49** (lukes/ISO-3166-Countries-with-Regional-Codes) + Wikidata P625 | country names, regions, centroids | one-off; the CSV is committed |
27 +
28 +Facts are copied, never inferred: a company only enters the registry with an official website *stated on Wikidata*; nothing is guessed
29 +from names. Provenance is kept per company (`source`, `harvested_at`, `sitelinks`, raw `industry_labels`, `notes`) and lands in
30 +`companies.source_meta`.
31 +
32 +## Harvest procedure (`scripts/seed_wikidata.py`)
33 +
34 +```bash
35 +.venv/bin/python scripts/seed_wikidata.py all --target 8000 # ≈ 45–60 min the first time; every stage resumes from cache
36 +.venv/bin/python scripts/seed_wikidata.py candidates|select|details|assemble # single stage
37 +.venv/bin/python scripts/seed_edgar.py # optional enrichment, rewrites the NDJSON in place
38 +.venv/bin/catlas registry-stats # counts without a database
39 +```
40 +
41 +1. **candidates** — `?item wdt:P31 wd:<class>` for 30 company classes (business, enterprise, public company, company, privately held
42 + company, corporation, bank, airline, automobile/aerospace manufacturer, software/technology company, video-game developer/publisher,
43 + publisher, record label, telecom, brewery, retail chain, railway, shipyard, …). The subclass tree of *business* (`P279*`) is far too
44 + broad to page, so classes are queried directly and the big ones are chunked by **sitelink bands** (≥ 80, 50–79, 35–49, 25–34, 18–24,
45 + 12–17, 8–11, 5–7); a band that times out is split in two automatically. Minimum 5 sitelinks. Dissolved items (P576) are dropped.
46 + Then **country boosts** (sitelinks ≥ 2, country-first scan, only for minimum-coverage countries still short after the class stage;
47 + a timeout is tolerated and reported) and **industry boosts** (P452 label lists per top-level industry, sitelinks ≥ 2, 600 each).
48 + Output `data/seed/candidates.json`.
49 +2. **select** — websites normalised to `https://<host>` (paths stripped); generic hosts dropped (social networks, blogs, app stores,
50 + site builders, code forges, Wikipedia …; for platform roots such as `google.com`/`apple.com` only the bare/www host counts);
51 + duplicates by registrable domain keep the highest-sitelink item (the loser is recorded as a `related_domain_conflict` note on the
52 + winner and in `data/seed/dropped.json`). Then the diversified selection below. Output `data/seed/selected.json` (+ a reserve of
53 + boost candidates per top-level industry).
54 +3. **details** — five light queries per batch of 100 QIDs (labels/description/aliases via the label service; inception/coordinates/
55 + LEI/CIK/logo; HQ + parent with end-time qualifiers; legal names + industry labels; tickers/exchanges + employee observations).
56 + A `GROUP BY` with many `SAMPLE()` aggregates makes Blazegraph throw `StackOverflowError`, hence plain queries reduced in Python.
57 + Output `data/seed/details.json` (incremental).
58 +4. **assemble** — industry mapping, importance, tiers, parent/domain conflicts, per-region NDJSON + README.
59 +
60 +Gotchas learnt on the endpoint: a query that runs past 60 s may come back as **HTTP 200 with a truncated JSON body followed by a Java
61 +stack trace** (the gateway even caches it) — the client treats an unparsable body as a timeout; some literals contain raw control
62 +characters (`json.loads(strict=False)`); starting a query from `P17` of a large country never finishes; `FILTER NOT EXISTS` over tens of
63 +thousands of bindings is what pushes a scan over the limit (fetch as OPTIONAL and filter client-side instead).
64 +
65 +## Selection and diversification rules
66 +
67 +Candidates are ordered by sitelinks (desc) and selected until `--target` (8,000) with:
68 +
69 +* **Caps** — United States ≤ 40 % of the target, any other country ≤ 12 %, unknown country ≤ 3 %.
70 +* **Minimum coverage** (taken first, best-ranked companies of that country, sitelinks ≥ 2): ≥ 150 for CA, GB, DE, FR, JP, KR, IN, AU;
71 + ≥ 60 for BR, MX, AE, SA, ZA, NG, SG, ID, NL, SE, CH, ES, IT, CN, TW, HK — when Wikidata has that many companies with a website.
72 +* **Industry floor** — every top-level industry gets ≥ 60 companies where the data allows, topped up from the industry-boost reserve.
73 +* **One company per registrable domain**; a subsidiary that shares its parent's domain is dropped (the parent keeps it).
74 +
75 +The resulting counts are in `registry/companies/README.md` and `catlas registry-stats`; `tests/test_registry_files.py` enforces the
76 +caps (with a small tolerance for the industry top-up), the minimums for the large economies and the tier sizes.
77 +
78 +## Importance and tiers
79 +
80 +```
81 +s_sitelinks = min(1, ln(1 + sitelinks) / ln(1 + 300)) # 300 sitelinks ≈ the most-linked companies
82 +s_employees = min(1, log10(1 + employees) / 6) # 1,000,000 employees → 1
83 +importance = 0.6·s_sitelinks + 0.2·s_employees + 0.1·public_company + 0.1·has_ticker (clamped to 0.02–1)
84 +tier = rank by importance: 1–150 → 1 (global) · 151–950 → 2 (major) · 951–3450 → 3 (notable) · rest → 4 (long tail)
85 +```
86 +
87 +Importance drives crawl priority only (spec §113); it is refreshed on every `catlas seed`, nothing else about an existing row is.
88 +
89 +## NDJSON schema (one object per line)
90 +
91 +```
92 +wikidata_id, display_name, legal_name, aliases[≤8], website (https://host), canonical_domain, country (ISO-2 | null), hq_city, hq_region,
93 +lat, lon, industries[] (taxonomy slugs), industry_labels[] (raw P452 labels, + "SIC …" after EDGAR), founded_year, employees,
94 +public_company, ticker, exchange, lei, sec_cik, parent {wikidata_id, name} | null, logo_url (Commons Special:FilePath), description,
95 +sitelinks, importance (0–1), tier (1–4), source ("wikidata"), harvested_at, notes[] (optional: related_domain_conflict),
96 +source_edgar {cik, enriched_at} (optional)
97 +```
98 +
99 +## Loading (`catlas seed`)
100 +
101 +`seed(conn, companies=True, limit=None, files=None)` in `companyatlas.registry.seed`:
102 +
103 +1. upserts `industries` (slug, name, parent, description, keywords, sort order) and `countries`;
104 +2. for each registry line: match an existing company by `wikidata_id`, then by `canonical_domain`;
105 + * new → `id = ids.new_id('company')`, slug `ids.slugify(display_name)` de-duplicated as `-<country>` then `-2`, `-3`…,
106 + `onboarding_status='pending'`, provenance in `source_meta`;
107 + * existing → only **null** columns are filled; `importance`, `tier` and `source_meta` provenance are refreshed; names, website,
108 + history and anything already set are never overwritten;
109 +3. `company_aliases` (display name → `brand`, legal name → `legal`, ticker → `ticker`, others → `alias`; key `ids.normalize_alias`),
110 + `domains` primary row, `company_relationships` PARENT_OF / SUBSIDIARY_OF (confidence 0.8, provenance `{source, property: P749}`)
111 + when the parent is also in the registry;
112 +4. `queue_jobs(kind='discover', key='discover:<company_id>', priority=importance)` for new pending companies (`on conflict do nothing`);
113 +5. `settings_kv['seed:last_run']` with the counters.
114 +
115 +Running it twice yields `companies_new = 0`. `--limit N` seeds the first N lines (files are ordered tier 1 → 4 inside each region);
116 +`--no-companies` only refreshes the reference tables; `--file` loads specific NDJSON files.
117 +
118 +## Adding companies
119 +
120 +```bash
121 +.venv/bin/catlas company-add https://www.example.com --name "Example Corp" --country CA --industry software
122 +.venv/bin/catlas import-companies my-companies.csv # columns: website (required), display_name, country, industries ("a;b"), aliases, …
123 +.venv/bin/catlas import-companies my-companies.ndjson # same keys as the registry schema
124 +```
125 +
126 +Imports go through the same upsert (idempotent, same slug/alias/queue rules) with `source_meta.source = "manual"` (`--source` to change).
127 +To add companies to the *committed* universe, append lines to a `registry/companies/*.ndjson` file (or a new `manual-*.ndjson`) with at
128 +least `website`; run `pytest tests/test_registry_files.py` (validity, no duplicate domain, caps) then `catlas seed`.
129 +
130 +## Tests
131 +
132 +```bash
133 +.venv/bin/pytest -q tests/test_industry_map.py tests/test_seed_loader.py tests/test_registry_files.py
134 +```
135 +
136 +`test_seed_loader.py` needs the local database (`catlas migrate`); it seeds a fixture twice under `ztest-` slugs and cleans up after itself.
137 +
138 +## Known gaps
139 +
140 +See the "Harvest report" section of `registry/companies/README.md` for the run-specific numbers. Structural limits:
141 +
142 +* Coverage follows Wikidata: companies without an official website there are absent; small markets (e.g. Nigeria, Saudi Arabia)
143 + may fall short of their minimum; South Korea's country boost times out on the endpoint (KR relies on the class stage).
144 +* Industry slugs come from P452 labels and, failing that, the boost query / description keywords; ~10 % of companies have none
145 + (the LLM `classify_industry` job and the crawl fill them later).
146 +* `hq_region` is the P131 parent of the HQ item and is sometimes the country itself.
147 +* Websites are forced to `https://`; discovery handles redirects and http-only fallbacks.
added registry/countries.csv +251 −0
@@ -0,0 +1,251 @@
1 +code,name,region,subregion,lat,lon
2 +AD,Andorra,Europe,Southern Europe,42.558,1.555
3 +AE,United Arab Emirates,Asia,Western Asia,23.4,53.8
4 +AF,Afghanistan,Asia,Southern Asia,33.9,67.7
5 +AG,Antigua and Barbuda,Americas,Latin America and the Caribbean,17.117,-61.85
6 +AI,Anguilla,Americas,Latin America and the Caribbean,18.227,-63.049
7 +AL,Albania,Europe,Southern Europe,41.2,20.2
8 +AM,Armenia,Asia,Western Asia,40.1,45.0
9 +AO,Angola,Africa,Sub-Saharan Africa,-11.2,17.9
10 +AQ,Antarctica,Antarctica,Antarctica,-82.9,135.0
11 +AR,Argentina,Americas,Latin America and the Caribbean,-38.4,-63.6
12 +AS,American Samoa,Oceania,Polynesia,-14.296,-170.708
13 +AT,Austria,Europe,Western Europe,47.5,14.6
14 +AU,Australia,Oceania,Australia and New Zealand,-25.3,133.8
15 +AW,Aruba,Americas,Latin America and the Caribbean,12.515,-69.975
16 +AX,Åland Islands,Europe,Northern Europe,60.25,20.0
17 +AZ,Azerbaijan,Asia,Western Asia,40.1,47.6
18 +BA,Bosnia and Herzegovina,Europe,Southern Europe,43.9,17.7
19 +BB,Barbados,Americas,Latin America and the Caribbean,13.17,-59.553
20 +BD,Bangladesh,Asia,Southern Asia,23.7,90.4
21 +BE,Belgium,Europe,Western Europe,50.5,4.5
22 +BF,Burkina Faso,Africa,Sub-Saharan Africa,12.267,-2.067
23 +BG,Bulgaria,Europe,Eastern Europe,42.7,25.5
24 +BH,Bahrain,Asia,Western Asia,26.0,50.6
25 +BI,Burundi,Africa,Sub-Saharan Africa,-3.667,29.817
26 +BJ,Benin,Africa,Sub-Saharan Africa,8.833,2.183
27 +BL,Saint Barthélemy,Americas,Latin America and the Caribbean,17.898,-62.834
28 +BM,Bermuda,Americas,Northern America,32.32,-64.74
29 +BN,Brunei,Asia,South-eastern Asia,4.5,114.7
30 +BO,Bolivia,Americas,Latin America and the Caribbean,-16.3,-63.6
31 +BQ,Caribbean Netherlands,Americas,Latin America and the Caribbean,12.183,-68.233
32 +BR,Brazil,Americas,Latin America and the Caribbean,-14.2,-51.9
33 +BS,Bahamas,Americas,Latin America and the Caribbean,25.0,-77.4
34 +BT,Bhutan,Asia,Southern Asia,27.5,90.4
35 +BV,Bouvet Island,Americas,Latin America and the Caribbean,-54.42,3.36
36 +BW,Botswana,Africa,Sub-Saharan Africa,-22.2,23.7
37 +BY,Belarus,Europe,Eastern Europe,53.7,27.95
38 +BZ,Belize,Americas,Latin America and the Caribbean,17.067,-88.7
39 +CA,Canada,Americas,Northern America,56.1,-106.3
40 +CC,Cocos,Oceania,Australia and New Zealand,-12.117,96.895
41 +CD,DR Congo,Africa,Sub-Saharan Africa,-4.0,21.8
42 +CF,Central African Republic,Africa,Sub-Saharan Africa,6.7,20.9
43 +CG,Republic of the Congo,Africa,Sub-Saharan Africa,-0.75,15.383
44 +CH,Switzerland,Europe,Western Europe,46.8,8.2
45 +CI,Côte d'Ivoire,Africa,Sub-Saharan Africa,8.0,-6.0
46 +CK,Cook Islands,Oceania,Polynesia,-21.233,-159.783
47 +CL,Chile,Americas,Latin America and the Caribbean,-35.7,-71.5
48 +CM,Cameroon,Africa,Sub-Saharan Africa,5.133,12.65
49 +CN,China,Asia,Eastern Asia,35.9,104.2
50 +CO,Colombia,Americas,Latin America and the Caribbean,4.6,-74.3
51 +CR,Costa Rica,Americas,Latin America and the Caribbean,9.7,-83.8
52 +CU,Cuba,Americas,Latin America and the Caribbean,21.5,-77.8
53 +CV,Cabo Verde,Africa,Sub-Saharan Africa,16.0,-24.0
54 +CW,Curaçao,Americas,Latin America and the Caribbean,12.2,-69.0
55 +CX,Christmas Island,Oceania,Australia and New Zealand,-10.49,105.627
56 +CY,Cyprus,Asia,Western Asia,35.1,33.4
57 +CZ,Czechia,Europe,Eastern Europe,49.8,15.5
58 +DE,Germany,Europe,Western Europe,51.2,10.4
59 +DJ,Djibouti,Africa,Sub-Saharan Africa,11.8,42.433
60 +DK,Denmark,Europe,Northern Europe,56.0,10.0
61 +DM,Dominica,Americas,Latin America and the Caribbean,15.417,-61.333
62 +DO,Dominican Republic,Americas,Latin America and the Caribbean,18.8,-70.2
63 +DZ,Algeria,Africa,Northern Africa,28.0,1.7
64 +EC,Ecuador,Americas,Latin America and the Caribbean,-1.8,-78.2
65 +EE,Estonia,Europe,Northern Europe,58.6,25.0
66 +EG,Egypt,Africa,Northern Africa,26.8,30.8
67 +EH,Western Sahara,Africa,Northern Africa,25.0,-13.0
68 +ER,Eritrea,Africa,Sub-Saharan Africa,15.483,38.25
69 +ES,Spain,Europe,Southern Europe,40.5,-3.7
70 +ET,Ethiopia,Africa,Sub-Saharan Africa,9.1,40.5
71 +FI,Finland,Europe,Northern Europe,61.9,25.7
72 +FJ,Fiji,Oceania,Melanesia,-17.7,178.1
73 +FK,Falkland Islands,Americas,Latin America and the Caribbean,-51.73,-59.22
74 +FM,Micronesia,Oceania,Micronesia,7.4,150.6
75 +FO,Faroe Islands,Europe,Northern Europe,61.97,-6.844
76 +FR,France,Europe,Western Europe,46.6,2.2
77 +GA,Gabon,Africa,Sub-Saharan Africa,-0.683,11.5
78 +GB,United Kingdom,Europe,Northern Europe,54.0,-2.5
79 +GD,Grenada,Americas,Latin America and the Caribbean,12.1,-61.7
80 +GE,Georgia,Asia,Western Asia,42.3,43.4
81 +GF,French Guiana,Americas,Latin America and the Caribbean,4.0,-53.0
82 +GG,Guernsey,Europe,Northern Europe,49.45,-2.583
83 +GH,Ghana,Africa,Sub-Saharan Africa,7.947,-1.023
84 +GI,Gibraltar,Europe,Southern Europe,36.14,-5.35
85 +GL,Greenland,Americas,Northern America,71.7,-42.6
86 +GM,Gambia,Africa,Sub-Saharan Africa,13.5,-15.5
87 +GN,Guinea,Africa,Sub-Saharan Africa,10.0,-11.0
88 +GP,Guadeloupe,Americas,Latin America and the Caribbean,16.25,-61.5
89 +GQ,Equatorial Guinea,Africa,Sub-Saharan Africa,1.5,10.0
90 +GR,Greece,Europe,Southern Europe,39.1,21.8
91 +GS,South Georgia and the South Sandwich Islands,Americas,Latin America and the Caribbean,-54.25,-36.75
92 +GT,Guatemala,Americas,Latin America and the Caribbean,15.8,-90.2
93 +GU,Guam,Oceania,Micronesia,13.5,144.8
94 +GW,Guinea-Bissau,Africa,Sub-Saharan Africa,12.0,-15.0
95 +GY,Guyana,Americas,Latin America and the Caribbean,5.733,-59.317
96 +HK,Hong Kong,Asia,Eastern Asia,22.3,114.2
97 +HM,Heard Island and McDonald Islands,Oceania,Australia and New Zealand,-53.078,73.507
98 +HN,Honduras,Americas,Latin America and the Caribbean,15.2,-86.2
99 +HR,Croatia,Europe,Southern Europe,45.1,15.2
100 +HT,Haiti,Americas,Latin America and the Caribbean,19.0,-72.8
101 +HU,Hungary,Europe,Eastern Europe,47.2,19.5
102 +ID,Indonesia,Asia,South-eastern Asia,-2.5,118.0
103 +IE,Ireland,Europe,Northern Europe,53.4,-8.2
104 +IL,Israel,Asia,Western Asia,31.0,34.9
105 +IM,Isle of Man,Europe,Northern Europe,54.235,-4.525
106 +IN,India,Asia,Southern Asia,20.6,79.0
107 +IO,British Indian Ocean Territory,Africa,Sub-Saharan Africa,-6.0,71.5
108 +IQ,Iraq,Asia,Western Asia,33.2,43.7
109 +IR,Iran,Asia,Southern Asia,32.4,53.7
110 +IS,Iceland,Europe,Northern Europe,64.96,-19.0
111 +IT,Italy,Europe,Southern Europe,41.9,12.6
112 +JE,Jersey,Europe,Northern Europe,49.19,-2.11
113 +JM,Jamaica,Americas,Latin America and the Caribbean,18.18,-77.4
114 +JO,Jordan,Asia,Western Asia,30.6,36.2
115 +JP,Japan,Asia,Eastern Asia,36.2,138.3
116 +KE,Kenya,Africa,Sub-Saharan Africa,-0.02,37.9
117 +KG,Kyrgyzstan,Asia,Central Asia,41.2,74.8
118 +KH,Cambodia,Asia,South-eastern Asia,12.6,105.0
119 +KI,Kiribati,Oceania,Micronesia,-3.4,-168.7
120 +KM,Comoros,Africa,Sub-Saharan Africa,-12.3,43.7
121 +KN,Saint Kitts and Nevis,Americas,Latin America and the Caribbean,17.272,-62.667
122 +KP,North Korea,Asia,Eastern Asia,40.3,127.5
123 +KR,South Korea,Asia,Eastern Asia,35.9,127.8
124 +KW,Kuwait,Asia,Western Asia,29.3,47.5
125 +KY,Cayman Islands,Americas,Latin America and the Caribbean,19.5,-80.5
126 +KZ,Kazakhstan,Asia,Central Asia,48.0,66.9
127 +LA,Laos,Asia,South-eastern Asia,19.9,102.5
128 +LB,Lebanon,Asia,Western Asia,33.9,35.9
129 +LC,Saint Lucia,Americas,Latin America and the Caribbean,13.883,-60.967
130 +LI,Liechtenstein,Europe,Western Europe,47.145,9.554
131 +LK,Sri Lanka,Asia,Southern Asia,7.9,80.8
132 +LR,Liberia,Africa,Sub-Saharan Africa,6.533,-9.75
133 +LS,Lesotho,Africa,Sub-Saharan Africa,-29.55,28.25
134 +LT,Lithuania,Europe,Northern Europe,55.2,23.9
135 +LU,Luxembourg,Europe,Western Europe,49.8,6.1
136 +LV,Latvia,Europe,Northern Europe,56.9,24.6
137 +LY,Libya,Africa,Northern Africa,26.3,17.2
138 +MA,Morocco,Africa,Northern Africa,31.8,-7.1
139 +MC,Monaco,Europe,Western Europe,43.731,7.42
140 +MD,Moldova,Europe,Eastern Europe,47.4,28.4
141 +ME,Montenegro,Europe,Southern Europe,42.7,19.4
142 +MF,Saint Martin,Americas,Latin America and the Caribbean,18.075,-63.06
143 +MG,Madagascar,Africa,Sub-Saharan Africa,-18.8,46.9
144 +MH,Marshall Islands,Oceania,Micronesia,9.82,169.29
145 +MK,North Macedonia,Europe,Southern Europe,41.6,21.7
146 +ML,Mali,Africa,Sub-Saharan Africa,17.6,-4.0
147 +MM,Myanmar,Asia,South-eastern Asia,21.9,95.96
148 +MN,Mongolia,Asia,Eastern Asia,46.9,103.8
149 +MO,Macao,Asia,Eastern Asia,22.2,113.55
150 +MP,Northern Mariana Islands,Oceania,Micronesia,16.705,145.78
151 +MQ,Martinique,Americas,Latin America and the Caribbean,14.65,-61.015
152 +MR,Mauritania,Africa,Sub-Saharan Africa,21.0,-10.9
153 +MS,Montserrat,Americas,Latin America and the Caribbean,16.75,-62.2
154 +MT,Malta,Europe,Southern Europe,35.9,14.4
155 +MU,Mauritius,Africa,Sub-Saharan Africa,-20.3,57.6
156 +MV,Maldives,Asia,Southern Asia,3.2,73.2
157 +MW,Malawi,Africa,Sub-Saharan Africa,-13.0,34.0
158 +MX,Mexico,Americas,Latin America and the Caribbean,23.6,-102.6
159 +MY,Malaysia,Asia,South-eastern Asia,4.2,108.0
160 +MZ,Mozambique,Africa,Sub-Saharan Africa,-18.7,35.5
161 +NA,Namibia,Africa,Sub-Saharan Africa,-23.0,17.0
162 +NC,New Caledonia,Oceania,Melanesia,-21.25,165.3
163 +NE,Niger,Africa,Sub-Saharan Africa,17.6,8.1
164 +NF,Norfolk Island,Oceania,Australia and New Zealand,-29.033,167.95
165 +NG,Nigeria,Africa,Sub-Saharan Africa,9.1,8.7
166 +NI,Nicaragua,Americas,Latin America and the Caribbean,12.9,-85.2
167 +NL,Netherlands,Europe,Western Europe,52.2,5.3
168 +NO,Norway,Europe,Northern Europe,64.6,12.7
169 +NP,Nepal,Asia,Southern Asia,28.4,84.1
170 +NR,Nauru,Oceania,Micronesia,-0.527,166.935
171 +NU,Niue,Oceania,Polynesia,-19.05,-169.917
172 +NZ,New Zealand,Oceania,Australia and New Zealand,-41.5,172.8
173 +OM,Oman,Asia,Western Asia,21.5,55.9
174 +PA,Panama,Americas,Latin America and the Caribbean,8.5,-80.8
175 +PE,Peru,Americas,Latin America and the Caribbean,-9.2,-75.0
176 +PF,French Polynesia,Oceania,Polynesia,-17.7,-149.4
177 +PG,Papua New Guinea,Oceania,Melanesia,-6.3,143.96
178 +PH,Philippines,Asia,South-eastern Asia,12.9,121.8
179 +PK,Pakistan,Asia,Southern Asia,30.4,69.3
180 +PL,Poland,Europe,Eastern Europe,51.9,19.1
181 +PM,Saint Pierre and Miquelon,Americas,Northern America,46.85,-56.3
182 +PN,Pitcairn,Oceania,Polynesia,-25.068,-130.105
183 +PR,Puerto Rico,Americas,Latin America and the Caribbean,18.25,-66.5
184 +PS,Palestine,Asia,Western Asia,31.9,35.2
185 +PT,Portugal,Europe,Southern Europe,39.4,-8.2
186 +PW,Palau,Oceania,Micronesia,7.467,134.55
187 +PY,Paraguay,Americas,Latin America and the Caribbean,-23.4,-58.4
188 +QA,Qatar,Asia,Western Asia,25.4,51.2
189 +RE,Réunion,Africa,Sub-Saharan Africa,-21.114,55.532
190 +RO,Romania,Europe,Eastern Europe,45.9,25.0
191 +RS,Serbia,Europe,Southern Europe,44.0,21.0
192 +RU,Russia,Europe,Eastern Europe,61.5,105.3
193 +RW,Rwanda,Africa,Sub-Saharan Africa,-2.0,30.0
194 +SA,Saudi Arabia,Asia,Western Asia,23.9,45.1
195 +SB,Solomon Islands,Oceania,Melanesia,-9.6,160.2
196 +SC,Seychelles,Africa,Sub-Saharan Africa,-4.7,55.5
197 +SD,Sudan,Africa,Northern Africa,12.9,30.2
198 +SE,Sweden,Europe,Northern Europe,60.1,18.6
199 +SG,Singapore,Asia,South-eastern Asia,1.35,103.8
200 +SH,Saint Helena,Africa,Sub-Saharan Africa,-15.925,-5.718
201 +SI,Slovenia,Europe,Southern Europe,46.2,15.0
202 +SJ,Svalbard and Jan Mayen,Europe,Northern Europe,77.6,23.7
203 +SK,Slovakia,Europe,Eastern Europe,48.7,19.7
204 +SL,Sierra Leone,Africa,Sub-Saharan Africa,8.5,-12.1
205 +SM,San Marino,Europe,Southern Europe,43.933,12.467
206 +SN,Senegal,Africa,Sub-Saharan Africa,14.367,-14.283
207 +SO,Somalia,Africa,Sub-Saharan Africa,6.0,47.0
208 +SR,Suriname,Americas,Latin America and the Caribbean,4.0,-56.0
209 +SS,South Sudan,Africa,Sub-Saharan Africa,7.0,30.0
210 +ST,Sao Tome and Principe,Africa,Sub-Saharan Africa,0.317,6.6
211 +SV,El Salvador,Americas,Latin America and the Caribbean,13.669,-88.866
212 +SX,Sint Maarten,Americas,Latin America and the Caribbean,18.032,-63.068
213 +SY,Syria,Asia,Western Asia,34.8,39.0
214 +SZ,Eswatini,Africa,Sub-Saharan Africa,-26.483,31.433
215 +TC,Turks and Caicos Islands,Americas,Latin America and the Caribbean,21.78,-71.8
216 +TD,Chad,Africa,Sub-Saharan Africa,15.5,18.7
217 +TF,French Southern Territories,Africa,Sub-Saharan Africa,-49.3,69.3
218 +TG,Togo,Africa,Sub-Saharan Africa,8.25,1.183
219 +TH,Thailand,Asia,South-eastern Asia,15.9,100.9
220 +TJ,Tajikistan,Asia,Central Asia,38.9,71.3
221 +TK,Tokelau,Oceania,Polynesia,-9.167,-171.833
222 +TL,Timor-Leste,Asia,South-eastern Asia,-8.9,125.7
223 +TM,Turkmenistan,Asia,Central Asia,38.97,59.6
224 +TN,Tunisia,Africa,Northern Africa,34.0,10.0
225 +TO,Tonga,Oceania,Polynesia,-21.2,-175.2
226 +TR,Türkiye,Asia,Western Asia,39.0,35.2
227 +TT,Trinidad and Tobago,Americas,Latin America and the Caribbean,10.667,-61.517
228 +TV,Tuvalu,Oceania,Polynesia,-7.475,178.006
229 +TW,Taiwan,Asia,Eastern Asia,23.7,121.0
230 +TZ,Tanzania,Africa,Sub-Saharan Africa,-6.4,34.9
231 +UA,Ukraine,Europe,Eastern Europe,48.4,31.2
232 +UG,Uganda,Africa,Sub-Saharan Africa,1.28,32.39
233 +UM,U.S. Minor Outlying Islands,Oceania,Micronesia,19.3,166.6
234 +US,United States,Americas,Northern America,39.8,-98.6
235 +UY,Uruguay,Americas,Latin America and the Caribbean,-32.5,-55.8
236 +UZ,Uzbekistan,Asia,Central Asia,41.4,64.6
237 +VA,Vatican City,Europe,Southern Europe,41.904,12.453
238 +VC,Saint Vincent and the Grenadines,Americas,Latin America and the Caribbean,13.014,-61.23
239 +VE,Venezuela,Americas,Latin America and the Caribbean,6.4,-66.6
240 +VG,British Virgin Islands,Americas,Latin America and the Caribbean,18.445,-64.54
241 +VI,U.S. Virgin Islands,Americas,Latin America and the Caribbean,18.333,-64.833
242 +VN,Vietnam,Asia,South-eastern Asia,14.1,108.3
243 +VU,Vanuatu,Oceania,Melanesia,-15.4,166.96
244 +WF,Wallis and Futuna,Oceania,Polynesia,-14.302,-178.109
245 +WS,Samoa,Oceania,Polynesia,-13.8,-172.1
246 +XK,Kosovo,Europe,Southern Europe,42.6,20.9
247 +YE,Yemen,Asia,Western Asia,15.6,48.5
248 +YT,Mayotte,Africa,Sub-Saharan Africa,-12.843,45.138
249 +ZA,South Africa,Africa,Sub-Saharan Africa,-30.6,22.9
250 +ZM,Zambia,Africa,Sub-Saharan Africa,-14.0,28.0
251 +ZW,Zimbabwe,Africa,Sub-Saharan Africa,-19.0,30.0
added registry/industries.yaml +429 −0
@@ -0,0 +1,429 @@
1 +# Company Atlas industry taxonomy (spec §6, §99). Two levels: top-level sectors and their children (`parent`).
2 +# `keywords` map Wikidata industry labels (P452), SIC descriptions and free text to slugs — see companyatlas.registry.industries.map_industry.
3 +# Matching is case-insensitive; a keyword matches as a whole phrase inside the label (word boundaries). Longer keywords win.
4 +# Order = sort_order in the database (children follow their parent).
5 +industries:
6 + # ------------------------------------------------------------------------------------------------ technology
7 + - slug: technology
8 + name: Technology
9 + description: Information technology, hardware, electronics and computing platforms.
10 + keywords: [technology, information technology, tech, computer hardware, hardware, electronics, consumer electronics, electronics industry,
11 + computer industry, computing, computer, information and communications technology, ict, high tech, technology company,
12 + electronic engineering, electrical equipment, computer peripheral, computer storage, data storage, optics, photonics,
13 + information technology consulting, it services, it service, information service, digital]
14 + - slug: software
15 + name: Software
16 + parent: technology
17 + description: Software products, SaaS, enterprise applications and developer tools.
18 + keywords: [software, software industry, software development, software engineering, saas, software as a service, enterprise software,
19 + application software, operating system, computer software, software publisher, developer tools, database, video game engine,
20 + open source, open-source software, cloud software, business software, erp, crm, productivity software]
21 + - slug: artificial-intelligence
22 + name: Artificial Intelligence
23 + parent: technology
24 + description: Machine learning, AI research, foundation models and AI applications.
25 + keywords: [artificial intelligence, machine learning, deep learning, ai, natural language processing, computer vision, generative artificial
26 + intelligence, large language model, ai research, neural network, robotics process automation, data science, speech recognition]
27 + - slug: semiconductors
28 + name: Semiconductors
29 + parent: technology
30 + description: Chip design, foundries, semiconductor equipment and materials.
31 + keywords: [semiconductor, semiconductors, semiconductor industry, integrated circuit, microprocessor, chip, chipmaker, fabless,
32 + semiconductor device fabrication, semiconductor equipment, memory chip, microelectronics, electronic component, sensor]
33 + - slug: cloud-infrastructure
34 + name: Cloud & Infrastructure
35 + parent: technology
36 + description: Cloud computing, hosting, data centres, networking and infrastructure software.
37 + keywords: [cloud computing, cloud, web hosting, hosting, data center, data centre, infrastructure as a service, platform as a service,
38 + content delivery network, cdn, networking hardware, computer network, network equipment, server, virtualization, devops, colocation]
39 + - slug: cybersecurity
40 + name: Cybersecurity
41 + parent: technology
42 + description: Information security products and services.
43 + keywords: [cybersecurity, cyber security, computer security, information security, network security, antivirus, security software,
44 + identity management, encryption, it security, cyber defense]
45 + - slug: internet
46 + name: Internet
47 + parent: technology
48 + description: Consumer internet, search, social media, online platforms and marketplaces.
49 + keywords: [internet, internet industry, world wide web, web, social media, social network, social networking service, search engine,
50 + online service, online platform, web portal, online community, streaming service, internet service provider, dot-com, online,
51 + web search, digital media, mobile app, mobile application, online dating, online advertising, advertising technology, adtech,
52 + website, online video platform, sharing economy, internet company]
53 + - slug: e-commerce
54 + name: E-commerce
55 + parent: technology
56 + description: Online retail and digital marketplaces.
57 + keywords: [e-commerce, ecommerce, electronic commerce, online retail, online retailer, online shopping, online marketplace, marketplace,
58 + online store, internet retail, mail order, direct-to-consumer, dtc, online auction]
59 + # ------------------------------------------------------------------------------------------------ finance
60 + - slug: financial-services
61 + name: Financial Services
62 + description: Banks, insurers, asset managers, exchanges and other financial institutions.
63 + keywords: [financial services, finance, financial, financial industry, financial sector, financial institution, financial technology holding,
64 + stock exchange, securities, brokerage, broker, investment, investment banking, credit, lending, loan, credit card, credit union,
65 + trust company, financial holding, capital markets, exchange, trading, securities trading, consumer finance, microfinance,
66 + leasing, factoring, wealth, financial market, fund, mortgage]
67 + - slug: fintech
68 + name: Fintech
69 + parent: financial-services
70 + description: Technology-driven financial services, neobanks, crypto and trading apps.
71 + keywords: [fintech, financial technology, neobank, cryptocurrency, crypto, blockchain, digital bank, online bank, digital payments platform,
72 + robo-advisor, crowdfunding, peer-to-peer lending, buy now pay later, bnpl, trading platform, online brokerage, digital currency,
73 + cryptocurrency exchange, decentralized finance, defi]
74 + - slug: payments
75 + name: Payments
76 + parent: financial-services
77 + description: Payment networks, processors, wallets and money transfer.
78 + keywords: [payment, payments, payment system, payment processing, payment processor, payment service provider, mobile payment,
79 + digital wallet, e-wallet, money transfer, remittance, payment card, card network, merchant acquiring, point of sale, pos, acquirer]
80 + - slug: banking
81 + name: Banking
82 + parent: financial-services
83 + description: Commercial, retail, private and investment banks.
84 + keywords: [banking, bank, commercial bank, retail banking, investment bank, private banking, savings bank, cooperative bank, central bank,
85 + universal bank, banking industry, merchant bank, building society, development bank, bank holding company, mortgage bank,
86 + online banking, islamic banking, postal savings]
87 + - slug: insurance
88 + name: Insurance
89 + parent: financial-services
90 + description: Life, health, property and casualty insurance and reinsurance.
91 + keywords: [insurance, insurer, reinsurance, reinsurer, life insurance, health insurance, property insurance, casualty insurance,
92 + insurance industry, insurance company, mutual insurance, assurance, underwriting, annuity, pension insurance, insurance broker]
93 + - slug: asset-management
94 + name: Asset Management
95 + parent: financial-services
96 + description: Investment management, private equity, venture capital, hedge funds and pensions.
97 + keywords: [asset management, investment management, fund management, private equity, venture capital, hedge fund, wealth management,
98 + pension fund, sovereign wealth fund, mutual fund, investment fund, investment company, exchange-traded fund, etf, investment firm,
99 + investment trust, financial advisory, holding company, family office, real estate investment trust, reit, alternative investment,
100 + private investment, investor]
101 + # ------------------------------------------------------------------------------------------------ real estate & construction
102 + - slug: real-estate
103 + name: Real Estate
104 + description: Property development, ownership, brokerage and services.
105 + keywords: [real estate, real estate industry, real estate development, property development, property developer, property management,
106 + real estate investment, real estate company, commercial real estate, residential real estate, housing, property, realty,
107 + real estate brokerage, proptech, coworking, self storage, shopping center, shopping mall, land development, home builder,
108 + homebuilding, property investment, estate agent]
109 + - slug: construction
110 + name: Construction
111 + description: Contractors, engineering & construction and building products.
112 + keywords: [construction, construction industry, civil engineering, building construction, general contractor, engineering and construction,
113 + infrastructure construction, building materials, building products, cement, concrete, construction materials, contractor,
114 + architecture, architectural firm, engineering firm, engineering, structural engineering, construction equipment, homebuilder,
115 + plumbing, hvac, roofing, building, real estate construction]
116 + # ------------------------------------------------------------------------------------------------ consumer
117 + - slug: retail
118 + name: Retail
119 + description: Store-based retailers, supermarkets, department stores and wholesale.
120 + keywords: [retail, retailing, retailer, retail industry, retail trade, supermarket, grocery store, grocery, hypermarket, department store,
121 + discount store, convenience store, chain store, retail chain, wholesale, wholesaling, distribution, distributor, drugstore,
122 + pharmacy chain, pharmacy, home improvement, hardware store, bookstore, bookseller, toy store, furniture store, warehouse club,
123 + sporting goods store, electronics retailer, variety store, duty-free, cash and carry, consumer goods distribution]
124 + - slug: consumer-goods
125 + name: Consumer Goods
126 + description: Household, personal care, cosmetics, toys, appliances and other consumer products.
127 + keywords: [consumer goods, consumer products, fast-moving consumer goods, fmcg, personal care, cosmetics, cosmetics industry, beauty,
128 + household goods, household products, home appliance, appliances, toys, toy industry, toy, furniture, furniture industry,
129 + luxury goods, luxury, jewelry, jewellery, watchmaking, watch, eyewear, sporting goods, sports equipment, tobacco, tobacco industry,
130 + consumer durables, cleaning products, hygiene, fragrance, perfume, stationery, bicycle, bicycle industry, pet food, pet products,
131 + kitchenware, tableware, home furnishings, mattress, musical instrument, consumer electronics manufacturer, camera]
132 + - slug: food-beverage
133 + name: Food & Beverage
134 + parent: consumer-goods
135 + description: Food processing, beverages, brewing, distilling and restaurants.
136 + keywords: [food, food industry, food processing, food manufacturing, beverage, beverages, beverage industry, drink, brewery, brewing, beer,
137 + distillery, distilling, spirits, wine, winery, winemaking, soft drink, dairy, dairy industry, meat, meat processing, confectionery,
138 + chocolate, snack, bakery, coffee, tea, restaurant, restaurant chain, fast food, foodservice, food service, quick service restaurant,
139 + catering, bottling, sugar, seafood, food retail, agri-food, agrifood, food and drink, nutrition, infant formula, frozen food,
140 + packaged food, condiment, cereal]
141 + - slug: apparel
142 + name: Apparel & Fashion
143 + parent: consumer-goods
144 + description: Clothing, footwear, fashion houses and textiles.
145 + keywords: [apparel, clothing, clothing industry, fashion, fashion industry, fashion house, footwear, shoe, shoes, textile, textile industry,
146 + textiles, garment, sportswear, luxury fashion, haute couture, denim, lingerie, accessories, leather goods, fashion design,
147 + fashion brand, clothing brand, ready-to-wear, outdoor clothing, uniform]
148 + # ------------------------------------------------------------------------------------------------ energy & resources
149 + - slug: energy
150 + name: Energy
151 + description: Energy production, trading and services.
152 + keywords: [energy, energy industry, energy sector, power generation, electricity generation, electric power, energy company, power,
153 + power plant, nuclear power, nuclear energy, nuclear, energy trading, energy services, coal, coal mining, uranium, lng,
154 + liquefied natural gas, district heating, heating, fuel, biofuel, hydrogen, energy storage, battery, batteries, battery manufacturer,
155 + power engineering, oilfield services]
156 + - slug: oil-gas
157 + name: Oil & Gas
158 + parent: energy
159 + description: Exploration, production, refining, pipelines and fuel distribution.
160 + keywords: [oil, oil and gas, oil and gas industry, petroleum, petroleum industry, oil industry, natural gas, gas, oil refining, refining,
161 + refinery, oil exploration, exploration and production, upstream, downstream, midstream, pipeline, pipeline transport, oil company,
162 + petrochemical, gas station, filling station, fuel retail, oilfield, drilling, offshore drilling, gas distribution, gas utility]
163 + - slug: renewables
164 + name: Renewable Energy
165 + parent: energy
166 + description: Solar, wind, hydro, geothermal and clean-energy technology.
167 + keywords: [renewable energy, renewables, solar, solar energy, solar power, photovoltaic, photovoltaics, wind, wind power, wind energy,
168 + wind turbine, hydroelectric, hydroelectricity, hydropower, geothermal, geothermal energy, clean energy, cleantech, clean technology,
169 + green energy, bioenergy, biomass, tidal power, energy transition, electric vehicle charging, ev charging, solar panel, fuel cell]
170 + - slug: utilities
171 + name: Utilities
172 + description: Electricity, gas and water utilities and grid operators.
173 + keywords: [utility, utilities, public utility, electric utility, electricity, electricity distribution, electricity transmission, power grid,
174 + grid operator, transmission system operator, distribution system operator, water utility, water supply, water, wastewater,
175 + sewerage, waste management, waste, recycling, waste collection, environmental services, sanitation, multi-utility, water industry,
176 + electricity retailer, energy supplier, electricity supplier, gas supplier, natural gas distribution, public service]
177 + - slug: mining
178 + name: Mining & Metals
179 + description: Mining, metals production and mineral extraction.
180 + keywords: [mining, mining industry, mine, metal, metals, metallurgy, steel, steelmaking, steel industry, iron ore, iron, aluminium, aluminum,
181 + copper, gold, gold mining, silver, nickel, zinc, lithium, rare earth, precious metals, mineral, minerals, mineral exploration,
182 + quarrying, quarry, coal, diamond, diamond mining, potash, phosphate, metal production, smelting, steel producer, iron and steel,
183 + non-ferrous metals, ferroalloy, mining company, extraction]
184 + - slug: chemicals
185 + name: Chemicals
186 + description: Basic, specialty and agricultural chemicals, plastics and coatings.
187 + keywords: [chemical, chemicals, chemical industry, chemistry, specialty chemicals, speciality chemicals, petrochemicals, plastics, plastic,
188 + polymer, polymers, coatings, paint, paints, fertilizer, fertiliser, agrochemical, agrochemicals, pesticide, industrial gases,
189 + industrial gas, adhesives, rubber, synthetic fiber, fibers, resin, explosives, chemical manufacturer, chemical engineering,
190 + cosmetics ingredients, flavors and fragrances, flavours, catalysts, pharmaceutical chemicals, fine chemicals]
191 + - slug: materials
192 + name: Materials
193 + description: Glass, paper, packaging, ceramics, wood and other basic materials.
194 + keywords: [materials, materials science, glass, glass industry, glassmaking, paper, paper industry, pulp and paper, pulp, packaging,
195 + packaging industry, container, containers, ceramics, ceramic, wood, wood industry, timber, lumber, forestry, forest products,
196 + forest industry, sawmill, cardboard, corrugated, building material, insulation, composite materials, composites, carbon fiber,
197 + abrasives, refractory, tiles, stone, gravel, asphalt, textile fiber, nonwovens, basic materials, raw materials]
198 + # ------------------------------------------------------------------------------------------------ industrials
199 + - slug: manufacturing
200 + name: Manufacturing
201 + description: Diversified and general manufacturing, industrial conglomerates.
202 + keywords: [manufacturing, manufacturer, manufacturing industry, industrial, industry, industrial conglomerate, conglomerate, engineering,
203 + industrial engineering, fabrication, metal fabrication, metalworking, industrial products, industrial goods, diversified industrials,
204 + electrical engineering, electrical industry, electrical equipment manufacturer, electric motor, lighting, lighting industry,
205 + tools, hand tools, power tools, fasteners, bearings, valves, pumps, pump, compressor, industrial automation, automation,
206 + contract manufacturing, electronics manufacturing services, ems, precision engineering, 3d printing, additive manufacturing,
207 + printing, printing industry, packaging machinery, textile machinery, industrial supplies, heavy industry,
208 + shipbuilding, shipyard, boatbuilding]
209 + - slug: industrial-machinery
210 + name: Industrial Machinery
211 + parent: manufacturing
212 + description: Machine tools, heavy equipment, agricultural and construction machinery.
213 + keywords: [machinery, industrial machinery, machine tool, machine tools, machine, machines, heavy equipment, construction equipment,
214 + construction machinery, agricultural machinery, farm equipment, mechanical engineering, mining equipment, engines, engine,
215 + engine manufacturer, turbine, turbines, elevator, elevators, escalator, crane, cranes, forklift, material handling, hydraulics,
216 + pneumatics, packaging equipment, printing press, sewing machine, machine building, plant engineering, process equipment,
217 + industrial equipment, equipment manufacturer, diesel engine, gas turbine, tractor, tractors, agricultural equipment, robots]
218 + - slug: robotics
219 + name: Robotics
220 + parent: manufacturing
221 + description: Industrial and service robots, drones and autonomous systems.
222 + keywords: [robotics, robot, robots, industrial robot, industrial robots, service robot, autonomous vehicle, autonomous systems,
223 + unmanned aerial vehicle, drone, drones, automation technology, mechatronics, humanoid robot, autonomous robot, robotic surgery,
224 + warehouse automation, robotic process, cobot, exoskeleton, self-driving]
225 + - slug: automotive
226 + name: Automotive
227 + description: Vehicle manufacturers, suppliers, dealers and mobility.
228 + keywords: [automotive, automotive industry, automobile, automobiles, automobile manufacturer, car, cars, car manufacturer, motor vehicle,
229 + motor vehicles, vehicle, vehicles, vehicle manufacturer, auto parts, automotive parts, automotive supplier, auto industry, truck,
230 + trucks, truck manufacturer, bus manufacturer, bus, motorcycle, motorcycles, motorcycle manufacturer, electric vehicle,
231 + electric vehicles, ev, car dealership, car dealer, car rental, tire, tyre, tires, tyres, automotive engineering, coachbuilder,
232 + commercial vehicle, commercial vehicles, scooter, automaker, mobility, ride-hailing, ridesharing, car sharing, carsharing,
233 + automotive technology, racing, motorsport, caravan, recreational vehicle]
234 + - slug: aerospace-defense
235 + name: Aerospace & Defense
236 + description: Aircraft, space systems, defence contractors and related suppliers.
237 + keywords: [aerospace, aerospace industry, aerospace manufacturer, aircraft, aircraft manufacturer, aviation industry, aviation, space,
238 + space industry, spaceflight, spacecraft, satellite, satellites, satellite manufacturer, launch vehicle, rocket, rockets, aeronautics,
239 + aeronautical, avionics, helicopter, helicopters, jet engine, aircraft engine, aerospace engineering, aircraft component,
240 + aircraft leasing, space technology, spaceport, aerospace and defense, aerospace and defence, unmanned aircraft, aircraft maintenance,
241 + mro, aircraft parts, general aviation, business jet]
242 + - slug: defense
243 + name: Defense
244 + parent: aerospace-defense
245 + description: Defence contractors, arms manufacturers and military technology.
246 + keywords: [defense, defence, defense industry, defence industry, arms industry, arms manufacturer, weapons, weapon, weapons manufacturer,
247 + military, military technology, military industry, armament, armaments, defense contractor, defence contractor, munitions,
248 + ammunition, firearms, firearms manufacturer, small arms, armored vehicle, armoured vehicle, naval shipbuilding, warship, missile,
249 + missiles, security and defense, military equipment, military aircraft, tank, radar, electronic warfare, defense electronics]
250 + # ------------------------------------------------------------------------------------------------ transport
251 + - slug: transportation
252 + name: Transportation
253 + description: Passenger and freight transport, rail, road, transit and infrastructure operators.
254 + keywords: [transportation, transport, transport industry, transportation industry, transport company, public transport, public transit,
255 + transit, rail transport, railway, railways, rail, railroad, railroads, railway company, rail freight, passenger rail, high-speed rail,
256 + metro, subway, tram, bus company, bus operator, coach operator, road transport, trucking, haulage, taxi, ferry, ferries,
257 + toll road, highway, motorway operator, airport, airports, airport operator, port, ports, port operator, seaport, infrastructure,
258 + transport infrastructure, rolling stock, railway equipment, locomotive, locomotives, railcar, train manufacturer, ride sharing,
259 + mobility services, car rental company, vehicle rental, parking, bike sharing, transportation company, transit authority]
260 + - slug: logistics
261 + name: Logistics
262 + parent: transportation
263 + description: Freight forwarding, parcel delivery, supply chain, warehousing and postal services.
264 + keywords: [logistics, logistics industry, logistics company, freight, freight forwarding, freight forwarder, freight transport, courier,
265 + courier service, parcel, parcel delivery, package delivery, express delivery, delivery, delivery service, postal service,
266 + postal, post, mail, supply chain, supply chain management, warehousing, warehouse, third-party logistics, 3pl, cargo,
267 + air cargo, cold chain, distribution logistics, last mile, last-mile delivery, food delivery, moving company, relocation,
268 + shipping and logistics, transportation and logistics, intermodal, freight railroad]
269 + - slug: airlines
270 + name: Airlines
271 + parent: transportation
272 + description: Passenger and cargo airlines, low-cost carriers and air charter.
273 + keywords: [airline, airlines, airline industry, air transport, air transportation, aviation company, low-cost airline, low-cost carrier,
274 + air carrier, flag carrier, regional airline, charter airline, cargo airline, air travel, passenger airline, air taxi, helicopter operator,
275 + commercial aviation, airline holding, air freight airline, airline alliance]
276 + - slug: shipping
277 + name: Shipping & Maritime
278 + parent: transportation
279 + description: Ocean shipping, container lines, tankers, ports and maritime services.
280 + keywords: [shipping, shipping industry, shipping company, maritime, maritime transport, maritime industry, sea transport, ocean shipping,
281 + container shipping, container line, shipping line, tanker, tankers, dry bulk, bulk shipping, ship owner, shipowner, ship management,
282 + cruise line, cruise, cruise ship, ferry operator, ocean freight, marine transport, marine services, offshore vessels, seafaring,
283 + ship operator, navigation, water transport, inland shipping, tugboat, towage, marine, maritime services, ship chartering]
284 + # ------------------------------------------------------------------------------------------------ telecom & media
285 + - slug: telecommunications
286 + name: Telecommunications
287 + description: Telecom operators, mobile networks, broadband, satellite communications and telecom equipment.
288 + keywords: [telecommunications, telecommunication, telecom, telecoms, telecommunications industry, telecommunications company,
289 + mobile network operator, mobile operator, mobile telephony, mobile network, wireless, wireless carrier, cellular, broadband,
290 + fiber optics, fibre, fixed-line, landline, telephone, telephony, voip, internet access, internet service provider, isp,
291 + cable operator, cable television operator, cable company, satellite communications, satellite communication, satellite operator,
292 + telecommunications equipment, telecom equipment, network infrastructure, tower company, telecommunications tower, 5g,
293 + unified communications, messaging, telecommunications services, phone company, communications, communication services]
294 + - slug: media
295 + name: Media
296 + description: Publishing, news, broadcasting, advertising and marketing services.
297 + keywords: [media, mass media, media industry, media company, publishing, publisher, publishing industry, book publishing, book publisher,
298 + newspaper, newspapers, newspaper publisher, magazine, magazines, periodical, press, news, news agency, news media, journalism,
299 + broadcasting, broadcaster, television, television broadcasting, television network, tv, radio, radio broadcasting, radio network,
300 + advertising, advertising agency, advertising industry, marketing, marketing agency, public relations, pr, digital marketing,
301 + media conglomerate, printing and publishing, information services, data provider, market research, communications agency,
302 + educational publishing, academic publishing, scientific publishing, comics publisher, trade publisher, outdoor advertising,
303 + billboard, media group, creative agency, content, podcast, stock photography, photography]
304 + - slug: entertainment
305 + name: Entertainment
306 + parent: media
307 + description: Film, television production, music, streaming, live events and theme parks.
308 + keywords: [entertainment, entertainment industry, entertainment company, film, film industry, film production, film production company,
309 + motion picture, motion pictures, movie, movies, cinema, cinema chain, movie theater, film studio, film distribution, television
310 + production, tv production, production company, animation, animation studio, music, music industry, record label, recording,
311 + music publishing, music streaming, streaming, streaming media, video streaming, video on demand, live entertainment, concert,
312 + concert promoter, live events, event promotion, theme park, amusement park, theme parks, circus, theatre, theater, performing arts,
313 + talent agency, sports entertainment, professional wrestling, esports, ticketing, casino, casinos, gambling, betting, sports betting,
314 + lottery, lotteries, sports club, football club, sports team, sports franchise, visual effects, vfx, sports league, dance, opera]
315 + - slug: gaming
316 + name: Video Games
317 + parent: media
318 + description: Video game developers, publishers, platforms and interactive entertainment.
319 + keywords: [video game, video games, video game industry, video game developer, video game publisher, game developer, game publisher,
320 + game development, gaming, games, computer game, computer games, mobile game, mobile games, mobile gaming, online game,
321 + online games, interactive entertainment, game studio, game engine, game console, video game console, arcade game, esports company,
322 + game platform, board game, tabletop game, toy and game, games industry, indie game, virtual reality, augmented reality, metaverse]
323 + # ------------------------------------------------------------------------------------------------ health
324 + - slug: healthcare
325 + name: Healthcare
326 + description: Hospitals, health services, health insurance providers and care delivery.
327 + keywords: [healthcare, health care, health, healthcare industry, health care industry, hospital, hospitals, hospital operator, clinic,
328 + clinics, medical, medicine, medical services, medical care, health services, health system, health network, nursing, nursing home,
329 + elderly care, senior living, home care, home health, dental, dentistry, dental care, veterinary, veterinary medicine, animal health,
330 + telehealth, telemedicine, digital health, health technology, healthtech, medical laboratory, laboratory, diagnostics laboratory,
331 + clinical laboratory, medical imaging, radiology, mental health, behavioral health, rehabilitation, physiotherapy, optometry,
332 + optical retail, health insurance provider, managed care, pharmacy benefit, pharmacy benefit management, healthcare services,
333 + dialysis, fertility, hospice, ambulance, emergency medical services, medical tourism, wellness, fitness, gym, health club,
334 + life sciences, life science, health and wellness, medical research, contract research organization, cro, medical distribution,
335 + pharmaceutical distribution, drug distribution, medical supplies, nutrition science]
336 + - slug: biotechnology
337 + name: Biotechnology
338 + parent: healthcare
339 + description: Biotech, genomics, cell and gene therapy and life-science tools.
340 + keywords: [biotechnology, biotech, biotechnology industry, biotechnology company, genomics, genetics, genetic engineering, gene therapy,
341 + cell therapy, synthetic biology, bioinformatics, biopharmaceutical, biopharma, biologics, immunotherapy, antibody, antibodies,
342 + molecular diagnostics, dna sequencing, sequencing, proteomics, microbiome, agricultural biotechnology, industrial biotechnology,
343 + bioengineering, biomedical, biomedical engineering, stem cell, regenerative medicine, life sciences tools, laboratory equipment,
344 + laboratory instruments, scientific instruments, research tools, vaccine development, crispr, mrna]
345 + - slug: pharmaceuticals
346 + name: Pharmaceuticals
347 + parent: healthcare
348 + description: Drug discovery, development, manufacturing and generics.
349 + keywords: [pharmaceutical, pharmaceuticals, pharmaceutical industry, pharma, pharmaceutical company, drug, drugs, drug manufacturer,
350 + drug development, drug discovery, generic drugs, generics, generic pharmaceuticals, medication, medications, medicines, vaccine,
351 + vaccines, over-the-counter, otc drugs, pharmaceutical manufacturing, contract manufacturing organization, cdmo, cmo, api manufacturing,
352 + active pharmaceutical ingredient, specialty pharma, oncology, ophthalmology, dermatology, nutraceutical, nutraceuticals,
353 + dietary supplement, supplements, animal pharmaceuticals, consumer health, pharmaceutical research, clinical trials, clinical research,
354 + pharmacology, homeopathy, traditional medicine, ayurveda]
355 + - slug: medical-devices
356 + name: Medical Devices
357 + parent: healthcare
358 + description: Medical equipment, devices, diagnostics and healthcare technology.
359 + keywords: [medical device, medical devices, medical equipment, medical technology, medtech, medical instruments, medical instrument,
360 + surgical instruments, surgical equipment, surgical, surgical robotics, diagnostic equipment, diagnostics, in vitro diagnostics,
361 + imaging equipment, medical imaging equipment, implants, implant, orthopedic, orthopedics, orthopaedic, prosthetics, prosthetic,
362 + hearing aid, hearing aids, hearing, dental equipment, dental products, cardiovascular devices, pacemaker, stent, insulin pump,
363 + glucose monitoring, respiratory equipment, ventilator, wheelchair, mobility aids, medical supplies manufacturer, lab equipment,
364 + health monitoring, wearable medical, contact lenses, eyeglasses, ophthalmic, endoscopy, infusion, dialysis equipment, sterilization,
365 + medical consumables, medical software, healthcare it, electronic health record, ehr, hospital equipment]
366 + # ------------------------------------------------------------------------------------------------ services
367 + - slug: hospitality
368 + name: Hospitality
369 + description: Hotels, resorts, restaurants chains, casinos and leisure venues.
370 + keywords: [hospitality, hospitality industry, hotel, hotels, hotel chain, hotel industry, hotel group, hotel operator, resort, resorts,
371 + lodging, accommodation, accommodations, motel, hostel, inn, bed and breakfast, vacation rental, timeshare, casino resort,
372 + restaurant group, restaurant operator, restaurant industry, bar, pub, pub chain, café, cafe, coffee shop, coffeehouse, coffee chain,
373 + food and hospitality, leisure, leisure industry, spa, wellness resort, ski resort, golf, golf course, club, nightclub,
374 + event venue, convention center, catering services, food service industry, contract catering, hotel management, franchise restaurants]
375 + - slug: travel
376 + name: Travel & Tourism
377 + description: Travel agencies, online travel, tour operators, cruise and tourism services.
378 + keywords: [travel, travel industry, tourism, tourism industry, travel agency, travel agencies, online travel agency, ota, online travel,
379 + tour operator, tour operators, travel technology, travel services, travel booking, booking, reservation system, global distribution
380 + system, gds, tourist, tourism company, cruise tourism, travel management, corporate travel, airline booking, holiday, holidays,
381 + vacation, vacations, package holidays, tourism board, travel retail, travel insurance, adventure travel, ecotourism, expedition,
382 + travel guide, travel publishing, luggage, travel accessories, destination management, theme travel, hotel booking, lodging platform]
383 + - slug: education
384 + name: Education
385 + description: Education providers, edtech, training, publishing and testing.
386 + keywords: [education, education industry, educational, educational services, edtech, education technology, educational technology,
387 + e-learning, online learning, online education, distance learning, school, schools, private school, school operator, university,
388 + universities, higher education, college, colleges, vocational training, training, professional training, corporate training,
389 + tutoring, test preparation, language school, language learning, language education, coding bootcamp, bootcamp, learning,
390 + learning platform, mooc, educational publisher, textbook, textbooks, early childhood education, childcare, daycare, nursery,
391 + education services, education management, for-profit education, exam, assessment, certification, driving school, music school,
392 + educational software, learning management system, lms, academic, academy, research institute, think tank]
393 + - slug: professional-services
394 + name: Professional Services
395 + description: Accounting, legal, staffing, outsourcing, engineering services and other B2B services.
396 + keywords: [professional services, business services, services, service industry, accounting, accountancy, accounting firm, audit, auditing,
397 + tax, tax services, legal services, law firm, law, legal, lawyer, lawyers, attorneys, staffing, staffing agency, recruitment,
398 + recruiting, human resources, hr, hr services, employment agency, temporary staffing, outsourcing, business process outsourcing,
399 + bpo, call center, customer service, facilities management, facility management, facility services, security services, security
400 + guard, private security, cleaning services, engineering services, architecture and engineering, design, design agency, industrial
401 + design, testing, inspection, certification services, tic, quality assurance, translation, translation services, market research
402 + firm, data analytics, business intelligence, credit rating agency, credit rating, rating agency, credit bureau, background check,
403 + payroll, payroll services, professional employer organization, peo, event management, exhibition, trade fair, trade shows,
404 + conference, printing services, document management, notary, patent, intellectual property services, real estate services,
405 + property services, consulting engineering, surveying, environmental consulting, government contractor, government services,
406 + defense services, aerospace services, research and development, r&d, contract research, laboratory testing, investigation]
407 + - slug: consulting
408 + name: Consulting
409 + parent: professional-services
410 + description: Management, strategy, IT and specialist consulting firms.
411 + keywords: [consulting, consultancy, consultancies, consulting firm, management consulting, management consultancy, strategy consulting,
412 + strategic consulting, it consulting, information technology consulting, technology consulting, business consulting, consulting
413 + services, advisory, advisory services, financial advisory services, economic consulting, engineering consulting, environmental
414 + consultancy, hr consulting, human resources consulting, executive search, headhunting, actuarial consulting, actuarial, risk
415 + consulting, systems integration, systems integrator, digital transformation, technology services, it outsourcing, professional
416 + consulting, tax advisory, transaction advisory, restructuring, public affairs, lobbying, political consulting, polling, research
417 + consultancy, design consultancy, brand consultancy, communications consultancy]
418 + # ------------------------------------------------------------------------------------------------ agriculture
419 + - slug: agriculture
420 + name: Agriculture
421 + description: Farming, agribusiness, seeds, forestry, fishing and agricultural inputs.
422 + keywords: [agriculture, agricultural, agriculture industry, agribusiness, agri-business, farming, farm, farms, farmer, crop, crops, crop
423 + production, seed, seeds, seed company, grain, grains, grain trading, agricultural trading, commodity trading, commodities,
424 + agricultural cooperative, cooperative, livestock, cattle, poultry, poultry farming, pig farming, aquaculture, fish farming, fishing,
425 + fisheries, fishery, forestry industry, plantation, plantations, palm oil, rubber plantation, horticulture, floriculture, flowers,
426 + greenhouse, vertical farming, agtech, agricultural technology, agricultural chemicals, animal feed, feed, animal nutrition,
427 + agricultural products, food production, sugar cane, coffee growing, cocoa, tobacco farming, tobacco growing, dairy farming,
428 + egg production, organic farming, viticulture, vineyard, orchard, fruit, vegetables, fruit and vegetables, agricultural machinery
429 + dealer, irrigation, land management, cotton, wool, timber production, cannabis, hemp, cannabis industry]
added scripts/seed_countries.py +126 −0
@@ -0,0 +1,126 @@
1 +#!/usr/bin/env python
2 +"""Generate `registry/countries.csv` (ISO-3166-1 alpha-2, name, UN M49 region/subregion, centroid lat/lon).
3 +
4 +Sources (fetched once at generation time; the CSV is committed so nothing is downloaded at runtime):
5 + * ISO 3166 + UN M49 regions: https://github.com/lukes/ISO-3166-Countries-with-Regional-Codes (all/all.csv, public domain-ish data)
6 + * Centroids: Wikidata `P297` (ISO alpha-2) → `P625` (coordinate location) via the SPARQL endpoint; a few manual overrides fix items whose
7 + coordinate is a capital or an outlying island rather than the country centroid.
8 +
9 +Usage: .venv/bin/python scripts/seed_countries.py [--out registry/countries.csv]
10 +"""
11 +from __future__ import annotations
12 +
13 +import argparse
14 +import csv
15 +import io
16 +import sys
17 +import time
18 +from pathlib import Path
19 +
20 +import httpx
21 +
22 +ROOT = Path(__file__).resolve().parents[1]
23 +UA = "CompanyAtlasBot/0.1 (contact@spboucher.ai)"
24 +ISO_URL = "https://raw.githubusercontent.com/lukes/ISO-3166-Countries-with-Regional-Codes/master/all/all.csv"
25 +SPARQL = "https://query.wikidata.org/sparql"
26 +NAME_OVERRIDES = {
27 + "US": "United States", "GB": "United Kingdom", "RU": "Russia", "KR": "South Korea", "KP": "North Korea", "IR": "Iran", "VN": "Vietnam",
28 + "TW": "Taiwan", "BO": "Bolivia", "VE": "Venezuela", "TZ": "Tanzania", "MD": "Moldova", "SY": "Syria", "LA": "Laos", "BN": "Brunei",
29 + "CZ": "Czechia", "FM": "Micronesia", "PS": "Palestine", "CD": "DR Congo", "CG": "Republic of the Congo", "CI": "Côte d'Ivoire",
30 + "VA": "Vatican City", "FK": "Falkland Islands", "VG": "British Virgin Islands", "VI": "U.S. Virgin Islands", "TR": "Türkiye",
31 + "NL": "Netherlands", "BQ": "Caribbean Netherlands", "SH": "Saint Helena", "MO": "Macao", "HK": "Hong Kong", "AE": "United Arab Emirates",
32 + "TF": "French Southern Territories", "UM": "U.S. Minor Outlying Islands", "GS": "South Georgia and the South Sandwich Islands",
33 +}
34 +# UN M49 does not assign a region to Taiwan (and Antarctica has none); keep the atlas usable for filtering.
35 +REGION_OVERRIDES = {"TW": ("Asia", "Eastern Asia"), "AQ": ("Antarctica", "Antarctica")}
36 +# Centroid overrides (lat, lon) where Wikidata's P625 is a capital/point rather than a usable centroid, or is missing.
37 +CENTROID_OVERRIDES = {
38 + "US": (39.8, -98.6), "CA": (56.1, -106.3), "RU": (61.5, 105.3), "FR": (46.6, 2.2), "NO": (64.6, 12.7), "DK": (56.0, 10.0),
39 + "NL": (52.2, 5.3), "GB": (54.0, -2.5), "AU": (-25.3, 133.8), "NZ": (-41.5, 172.8), "CL": (-35.7, -71.5), "BR": (-14.2, -51.9),
40 + "ID": (-2.5, 118.0), "JP": (36.2, 138.3), "CN": (35.9, 104.2), "IN": (20.6, 79.0), "KI": (-3.4, -168.7), "FM": (7.4, 150.6),
41 + "PT": (39.4, -8.2), "ES": (40.5, -3.7), "EC": (-1.8, -78.2), "UM": (19.3, 166.6), "TF": (-49.3, 69.3), "AQ": (-82.9, 135.0),
42 + "ZA": (-30.6, 22.9), "AR": (-38.4, -63.6), "MX": (23.6, -102.6), "DE": (51.2, 10.4), "IT": (41.9, 12.6), "TW": (23.7, 121.0),
43 + "HK": (22.3, 114.2), "SG": (1.35, 103.8), "AE": (23.4, 53.8), "SA": (23.9, 45.1), "NG": (9.1, 8.7), "SE": (60.1, 18.6), "CH": (46.8, 8.2),
44 + "KR": (35.9, 127.8), "IL": (31.0, 34.9), "PS": (31.9, 35.2), "EG": (26.8, 30.8), "TR": (39.0, 35.2), "IR": (32.4, 53.7), "PK": (30.4, 69.3),
45 + "GL": (71.7, -42.6), "SJ": (77.6, 23.7), "MY": (4.2, 108.0), "PH": (12.9, 121.8), "VN": (14.1, 108.3), "TH": (15.9, 100.9),
46 + "UA": (48.4, 31.2), "PL": (51.9, 19.1), "KZ": (48.0, 66.9), "MN": (46.9, 103.8), "IE": (53.4, -8.2), "IS": (64.96, -19.0), "FI": (61.9, 25.7),
47 + "GR": (39.1, 21.8), "MA": (31.8, -7.1), "DZ": (28.0, 1.7), "LY": (26.3, 17.2), "SD": (12.9, 30.2), "ET": (9.1, 40.5), "KE": (-0.02, 37.9),
48 + "CD": (-4.0, 21.8), "AO": (-11.2, 17.9), "MZ": (-18.7, 35.5), "MG": (-18.8, 46.9), "TZ": (-6.4, 34.9), "ML": (17.6, -4.0), "NE": (17.6, 8.1),
49 + "TD": (15.5, 18.7), "MR": (21.0, -10.9), "PE": (-9.2, -75.0), "CO": (4.6, -74.3), "VE": (6.4, -66.6), "BO": (-16.3, -63.6), "PY": (-23.4, -58.4),
50 + "UY": (-32.5, -55.8), "CU": (21.5, -77.8), "GT": (15.8, -90.2), "HN": (15.2, -86.2), "NI": (12.9, -85.2), "PA": (8.5, -80.8), "CR": (9.7, -83.8),
51 + "AF": (33.9, 67.7), "IQ": (33.2, 43.7), "SY": (34.8, 39.0), "JO": (30.6, 36.2), "OM": (21.5, 55.9), "YE": (15.6, 48.5), "UZ": (41.4, 64.6),
52 + "TM": (38.97, 59.6), "NP": (28.4, 84.1), "BD": (23.7, 90.4), "MM": (21.9, 95.96), "LK": (7.9, 80.8), "KH": (12.6, 105.0), "LA": (19.9, 102.5),
53 + "PG": (-6.3, 143.96), "SB": (-9.6, 160.2), "VU": (-15.4, 166.96), "FJ": (-17.7, 178.1), "TO": (-21.2, -175.2), "WS": (-13.8, -172.1),
54 + "CV": (16.0, -24.0), "MU": (-20.3, 57.6), "SC": (-4.7, 55.5), "MV": (3.2, 73.2), "BH": (26.0, 50.6), "QA": (25.4, 51.2), "KW": (29.3, 47.5),
55 + "LB": (33.9, 35.9), "CY": (35.1, 33.4), "MT": (35.9, 14.4), "LU": (49.8, 6.1), "BE": (50.5, 4.5), "AT": (47.5, 14.6), "CZ": (49.8, 15.5),
56 + "SK": (48.7, 19.7), "HU": (47.2, 19.5), "RO": (45.9, 25.0), "BG": (42.7, 25.5), "RS": (44.0, 21.0), "HR": (45.1, 15.2), "SI": (46.2, 15.0),
57 + "BA": (43.9, 17.7), "ME": (42.7, 19.4), "MK": (41.6, 21.7), "AL": (41.2, 20.2), "XK": (42.6, 20.9), "EE": (58.6, 25.0), "LV": (56.9, 24.6),
58 + "LT": (55.2, 23.9), "BY": (53.7, 27.95), "MD": (47.4, 28.4), "GE": (42.3, 43.4), "AM": (40.1, 45.0), "AZ": (40.1, 47.6), "KG": (41.2, 74.8),
59 + "TJ": (38.9, 71.3), "BT": (27.5, 90.4), "TL": (-8.9, 125.7), "BN": (4.5, 114.7), "KP": (40.3, 127.5), "MO": (22.2, 113.55),
60 +}
61 +
62 +
63 +def fetch_iso() -> list[dict[str, str]]:
64 + r = httpx.get(ISO_URL, headers={"User-Agent": UA}, timeout=60, follow_redirects=True)
65 + r.raise_for_status()
66 + return list(csv.DictReader(io.StringIO(r.text)))
67 +
68 +
69 +def fetch_centroids() -> dict[str, tuple[float, float]]:
70 + query = """
71 + SELECT ?iso ?coord WHERE {
72 + ?c wdt:P297 ?iso ; wdt:P625 ?coord .
73 + FILTER NOT EXISTS { ?c wdt:P576 ?dissolved }
74 + }"""
75 + for attempt in range(4):
76 + r = httpx.get(SPARQL, params={"query": query, "format": "json"}, headers={"User-Agent": UA, "Accept": "application/sparql-results+json"},
77 + timeout=90)
78 + if r.status_code == 200:
79 + break
80 + time.sleep(5 * (attempt + 1))
81 + r.raise_for_status()
82 + out: dict[str, tuple[float, float]] = {}
83 + for b in r.json()["results"]["bindings"]:
84 + iso = b["iso"]["value"].upper()
85 + val = b["coord"]["value"] # Point(lon lat)
86 + try:
87 + lon, lat = val.removeprefix("Point(").removesuffix(")").split()
88 + out.setdefault(iso, (round(float(lat), 3), round(float(lon), 3)))
89 + except ValueError:
90 + continue
91 + return out
92 +
93 +
94 +def main() -> int:
95 + ap = argparse.ArgumentParser()
96 + ap.add_argument("--out", default=str(ROOT / "registry" / "countries.csv"))
97 + args = ap.parse_args()
98 + iso = fetch_iso()
99 + time.sleep(2)
100 + centroids = fetch_centroids()
101 + rows = []
102 + for r in iso:
103 + code = r["alpha-2"].strip().upper()
104 + if len(code) != 2:
105 + continue
106 + name = NAME_OVERRIDES.get(code) or r["name"].split(" (")[0].split(",")[0].strip()
107 + lat, lon = CENTROID_OVERRIDES.get(code) or centroids.get(code) or ("", "")
108 + region, subregion = REGION_OVERRIDES.get(code) or (r.get("region") or "", r.get("sub-region") or "")
109 + rows.append({"code": code, "name": name, "region": region, "subregion": subregion, "lat": lat, "lon": lon})
110 + # Kosovo is user-assigned (XK) and not in ISO 3166-1; several data sources (and Wikidata companies) use it.
111 + if not any(x["code"] == "XK" for x in rows):
112 + rows.append({"code": "XK", "name": "Kosovo", "region": "Europe", "subregion": "Southern Europe", "lat": 42.6, "lon": 20.9})
113 + rows.sort(key=lambda x: x["code"])
114 + missing = [x["code"] for x in rows if x["lat"] == ""]
115 + out = Path(args.out)
116 + out.parent.mkdir(parents=True, exist_ok=True)
117 + with out.open("w", newline="", encoding="utf-8") as f:
118 + w = csv.DictWriter(f, fieldnames=["code", "name", "region", "subregion", "lat", "lon"])
119 + w.writeheader()
120 + w.writerows(rows)
121 + print(f"wrote {len(rows)} countries → {out} (missing centroids: {missing or 'none'})")
122 + return 0
123 +
124 +
125 +if __name__ == "__main__":
126 + sys.exit(main())
added scripts/seed_edgar.py +181 −0
@@ -0,0 +1,181 @@
1 +#!/usr/bin/env python
2 +"""SEC EDGAR enrichment for the seed registry (docs/SEEDS.md).
3 +
4 +Matches registry companies to `company_tickers.json` (ticker first, then normalised name) and fills *missing* `sec_cik`, `ticker`,
5 +`exchange`, `hq_city`/`hq_region` and an industry hint (SIC → taxonomy slug) from the submissions JSON. Existing values are never
6 +overwritten. Polite: SEC-required User-Agent, ≤ 5 requests/s, responses cached under data/seed/edgar/.
7 +
8 +Usage: .venv/bin/python scripts/seed_edgar.py [--max-submissions N] [--dry-run]
9 +"""
10 +from __future__ import annotations
11 +
12 +import argparse
13 +import json
14 +import logging
15 +import sys
16 +import time
17 +from collections import Counter, defaultdict
18 +from pathlib import Path
19 +from typing import Any
20 +
21 +import httpx
22 +
23 +ROOT = Path(__file__).resolve().parents[1]
24 +sys.path.insert(0, str(ROOT / "src"))
25 +
26 +from companyatlas.ids import normalize_alias
27 +from companyatlas.registry.industries import map_sic
28 +
29 +log = logging.getLogger("seed_edgar")
30 +USER_AGENT = "CompanyAtlasBot/0.1 contact@spboucher.ai"
31 +TICKERS_URL = "https://www.sec.gov/files/company_tickers.json"
32 +SUBMISSIONS_URL = "https://data.sec.gov/submissions/CIK{cik:010d}.json"
33 +MIN_INTERVAL_S = 0.21 # ≤ 5 requests per second (SEC fair-access policy is 10/s)
34 +CACHE_DIR = ROOT / "data" / "seed" / "edgar"
35 +OUT_DIR = ROOT / "registry" / "companies"
36 +US_STATES = {"AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE", "FL", "GA", "HI", "ID", "IL", "IN", "IA", "KS", "KY", "LA", "ME", "MD", "MA", "MI",
37 + "MN", "MS", "MO", "MT", "NE", "NV", "NH", "NJ", "NM", "NY", "NC", "ND", "OH", "OK", "OR", "PA", "RI", "SC", "SD", "TN", "TX", "UT",
38 + "VT", "VA", "WA", "WV", "WI", "WY", "DC", "PR"}
39 +
40 +
41 +class Edgar:
42 + def __init__(self) -> None:
43 + self.client = httpx.Client(headers={"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"}, timeout=30, follow_redirects=True)
44 + self.last = 0.0
45 + self.requests = 0
46 + CACHE_DIR.mkdir(parents=True, exist_ok=True)
47 +
48 + def get_json(self, url: str, cache_name: str) -> Any | None:
49 + path = CACHE_DIR / cache_name
50 + if path.exists():
51 + return json.loads(path.read_text(encoding="utf-8"))
52 + for attempt in range(4):
53 + wait = MIN_INTERVAL_S - (time.monotonic() - self.last)
54 + if wait > 0:
55 + time.sleep(wait)
56 + try:
57 + r = self.client.get(url)
58 + except httpx.HTTPError as e:
59 + log.warning("edgar %s: %s", url, e)
60 + time.sleep(2 * (attempt + 1))
61 + continue
62 + finally:
63 + self.last = time.monotonic()
64 + self.requests += 1
65 + if r.status_code == 200:
66 + path.write_text(r.text, encoding="utf-8")
67 + return r.json()
68 + if r.status_code == 404:
69 + path.write_text("null", encoding="utf-8")
70 + return None
71 + log.warning("edgar %s → HTTP %d, backing off", url, r.status_code)
72 + time.sleep(5 * (attempt + 1))
73 + return None
74 +
75 +
76 +def load_registry() -> dict[Path, list[dict[str, Any]]]:
77 + out: dict[Path, list[dict[str, Any]]] = {}
78 + for path in sorted(OUT_DIR.glob("wikidata-*.ndjson")):
79 + with path.open(encoding="utf-8") as f:
80 + out[path] = [json.loads(line) for line in f if line.strip()]
81 + return out
82 +
83 +
84 +def main() -> int:
85 + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
86 + ap.add_argument("--max-submissions", type=int, default=4000, help="cap on submissions JSON fetches")
87 + ap.add_argument("--dry-run", action="store_true")
88 + ap.add_argument("-v", "--verbose", action="store_true")
89 + args = ap.parse_args()
90 + logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
91 + logging.getLogger("httpx").setLevel(logging.WARNING)
92 +
93 + edgar = Edgar()
94 + tickers = edgar.get_json(TICKERS_URL, "company_tickers.json")
95 + if not tickers:
96 + log.error("could not fetch %s", TICKERS_URL)
97 + return 1
98 + by_ticker: dict[str, list[dict[str, Any]]] = defaultdict(list)
99 + by_name: dict[str, set[int]] = defaultdict(set)
100 + for e in tickers.values():
101 + by_ticker[e["ticker"].upper()].append(e)
102 + by_name[normalize_alias(e["title"])].add(int(e["cik_str"]))
103 + log.info("edgar tickers: %d entries, %d names", len(tickers), len(by_name))
104 +
105 + files = load_registry()
106 + stats: Counter[str] = Counter()
107 + submissions_fetched = 0
108 + for path, rows in files.items():
109 + for r in rows:
110 + cik: int | None = int(r["sec_cik"]) if r.get("sec_cik") and str(r["sec_cik"]).isdigit() else None
111 + if cik is None:
112 + cands: set[int] = set()
113 + if r.get("ticker") and r["ticker"].upper() in by_ticker:
114 + for e in by_ticker[r["ticker"].upper()]:
115 + if normalize_alias(e["title"])[:6] == normalize_alias(r["display_name"])[:6] or r.get("country") == "US":
116 + cands.add(int(e["cik_str"]))
117 + if not cands:
118 + names = [r["display_name"], r.get("legal_name") or ""] + list(r.get("aliases") or [])
119 + for nm in names:
120 + if nm and normalize_alias(nm) in by_name:
121 + cands |= by_name[normalize_alias(nm)]
122 + if len(cands) > 1 and r.get("country") != "US":
123 + cands = set() # ambiguous non-US name: do not guess
124 + if len(cands) == 1:
125 + cik = cands.pop()
126 + stats["matched_name_or_ticker"] += 1
127 + elif len(cands) > 1:
128 + stats["ambiguous"] += 1
129 + continue
130 + else:
131 + continue
132 + else:
133 + stats["had_cik"] += 1
134 + r.setdefault("industry_labels", [])
135 + changed = False
136 + if not r.get("sec_cik"):
137 + r["sec_cik"] = str(cik)
138 + changed = True
139 + needs_sub = not r.get("exchange") or not r.get("industries") or not r.get("hq_city") or not r.get("ticker")
140 + if needs_sub and submissions_fetched < args.max_submissions:
141 + sub = edgar.get_json(SUBMISSIONS_URL.format(cik=cik), f"CIK{cik:010d}.json")
142 + submissions_fetched += 1
143 + if sub:
144 + if not r.get("ticker") and sub.get("tickers"):
145 + r["ticker"] = sub["tickers"][0]
146 + changed = True
147 + if not r.get("exchange") and sub.get("exchanges"):
148 + r["exchange"] = next((x for x in sub["exchanges"] if x), None)
149 + changed = changed or bool(r["exchange"])
150 + if r.get("ticker"):
151 + r["public_company"] = True
152 + sic = sub.get("sic")
153 + slug = map_sic(sic)
154 + if sic and sub.get("sicDescription") and f"SIC {sic} {sub['sicDescription']}" not in r["industry_labels"]:
155 + r["industry_labels"].append(f"SIC {sic} {sub['sicDescription']}")
156 + changed = True
157 + if slug and not r.get("industries"):
158 + r["industries"] = [slug]
159 + stats["industry_from_sic"] += 1
160 + changed = True
161 + biz = (sub.get("addresses") or {}).get("business") or {}
162 + if not r.get("hq_city") and biz.get("city"):
163 + r["hq_city"] = biz["city"].title()
164 + changed = True
165 + if not r.get("hq_region") and biz.get("stateOrCountry") in US_STATES:
166 + r["hq_region"] = biz["stateOrCountry"]
167 + changed = True
168 + if changed:
169 + r["source_edgar"] = {"cik": str(cik), "enriched_at": time.strftime("%Y-%m-%dT%H:%M:%S+00:00", time.gmtime())}
170 + stats["rows_enriched"] += 1
171 + if not args.dry_run:
172 + with path.open("w", encoding="utf-8") as f:
173 + for r in rows:
174 + f.write(json.dumps(r, ensure_ascii=False) + "\n")
175 + log.info("edgar done: %s (%d HTTP requests, %d submissions)", dict(stats), edgar.requests, submissions_fetched)
176 + (ROOT / "data" / "seed" / "edgar-report.json").write_text(json.dumps({"stats": stats, "submissions": submissions_fetched}, indent=1))
177 + return 0
178 +
179 +
180 +if __name__ == "__main__":
181 + sys.exit(main())
added scripts/seed_wikidata.py +846 −0
@@ -0,0 +1,846 @@
1 +#!/usr/bin/env python
2 +"""Polite Wikidata harvester for the Company Atlas seed registry (docs/SEEDS.md).
3 +
4 +Stages (each resumable; every SPARQL result is cached under data/seed/wikidata/sparql/<sha256>.json):
5 + candidates company-class × sitelink-band queries + per-country and per-industry boost queries → data/seed/candidates.json
6 + select normalise websites, drop generic hosts / duplicate domains, diversify (US ≤ 40 %, others ≤ 12 %, country minimums)
7 + → data/seed/selected.json
8 + details batched detail queries (labels, legal names, industries, HQ, coordinates, tickers, LEI, CIK, parent, logo, employees)
9 + → data/seed/details.json
10 + assemble importance + tiers + industry mapping + parent/domain conflicts → registry/companies/wikidata-<region>.ndjson + README.md
11 + all the four stages in sequence (default)
12 +
13 +Politeness: User-Agent CompanyAtlasBot/0.1, one query at a time, ≥ 2 s between queries, 60 s server timeout, retries with backoff.
14 +"""
15 +from __future__ import annotations
16 +
17 +import argparse
18 +import csv
19 +import hashlib
20 +import json
21 +import logging
22 +import math
23 +import re
24 +import sys
25 +import time
26 +from collections import Counter, defaultdict
27 +from datetime import UTC, datetime
28 +from pathlib import Path
29 +from typing import Any
30 +from urllib.parse import quote, urlparse
31 +
32 +import httpx
33 +
34 +ROOT = Path(__file__).resolve().parents[1]
35 +sys.path.insert(0, str(ROOT / "src"))
36 +
37 +from companyatlas.registry.industries import load_industries, map_industry, top_level_of, top_level_slugs
38 +from companyatlas.urls import registrable_domain
39 +
40 +log = logging.getLogger("seed_wikidata")
41 +
42 +SPARQL_URL = "https://query.wikidata.org/sparql"
43 +USER_AGENT = "CompanyAtlasBot/0.1 (contact@spboucher.ai)"
44 +MIN_INTERVAL_S = 2.0
45 +TIMEOUT_S = 60.0
46 +MAX_TRIES = 6
47 +DATA_DIR = ROOT / "data" / "seed"
48 +CACHE_DIR = DATA_DIR / "wikidata" / "sparql"
49 +REGISTRY_DIR = ROOT / "registry"
50 +OUT_DIR = REGISTRY_DIR / "companies"
51 +COUNTRIES_CSV = REGISTRY_DIR / "countries.csv"
52 +
53 +# Wikidata classes queried with wdt:P31 (no P279* — the subclass tree of "business" is too broad to page). Banded classes are large.
54 +CLASSES: dict[str, str] = {
55 + "Q4830453": "business", "Q6881511": "enterprise", "Q891723": "public company", "Q783794": "company", "Q1589009": "privately held company",
56 + "Q167037": "corporation", "Q18388277": "technology company", "Q1058914": "software company", "Q22687": "bank", "Q46970": "airline",
57 + "Q786820": "automobile manufacturer", "Q6500733": "pharmaceutical company", "Q210167": "video game developer",
58 + "Q1137109": "video game publisher", "Q1762059": "film production company", "Q18127": "record label", "Q2085381": "publisher",
59 + "Q613142": "law firm", "Q740752": "transport company", "Q2401749": "telecommunications company", "Q131734": "brewery",
60 + "Q507619": "retail chain", "Q936518": "aerospace manufacturer", "Q206361": "conglomerate", "Q249556": "railway company",
61 + "Q2005696": "real estate company", "Q708676": "shipyard", "Q1668024": "internet service", "Q1320047": "book publisher",
62 + "Q19967801": "online service",
63 +}
64 +BANDED_CLASSES = {"Q4830453", "Q6881511", "Q891723", "Q46970"}
65 +# Country boosts run only when the class stage yielded fewer than BOOST_MARGIN × minimum candidates for that country.
66 +BOOST_MARGIN = 2.0
67 +SITELINK_BANDS: list[tuple[int, int | None]] = [(80, None), (50, 80), (35, 50), (25, 35), (18, 25), (12, 18), (8, 12), (5, 8)]
68 +MIN_SITELINKS = 5
69 +BOOST_MIN_SITELINKS = 2
70 +BOOST_LIMIT = 2500
71 +INDUSTRY_BOOST_LIMIT = 600
72 +
73 +# Diversification (spec: "seed diversified companies across US, Canada, Europe, UK, Japan, South Korea, India, Australia, LatAm, ME, Africa, SEA")
74 +US_CAP_SHARE = 0.40
75 +OTHER_CAP_SHARE = 0.12
76 +UNKNOWN_COUNTRY_SHARE = 0.03
77 +COUNTRY_MINIMUMS: dict[str, int] = {
78 + **dict.fromkeys(["CA", "GB", "DE", "FR", "JP", "KR", "IN", "AU"], 150),
79 + **dict.fromkeys(["BR", "MX", "AE", "SA", "ZA", "NG", "SG", "ID", "NL", "SE", "CH", "ES", "IT", "CN", "TW", "HK"], 60),
80 +}
81 +INDUSTRY_MINIMUM = 60
82 +INDUSTRY_RESERVE = 150 # extra candidates per top-level industry fetched in the detail stage to satisfy INDUSTRY_MINIMUM
83 +# English labels of Wikidata items commonly used as P452 (industry) values, per top-level industry, for the industry-boost queries.
84 +INDUSTRY_BOOST_LABELS: dict[str, list[str]] = {
85 + "technology": ["information technology", "electronics industry", "consumer electronics", "computer hardware", "electronics"],
86 + "financial-services": ["financial services", "insurance", "banking", "asset management", "investment banking", "payment system"],
87 + "real-estate": ["real estate", "real estate development", "property management", "real estate industry"],
88 + "construction": ["construction", "construction industry", "civil engineering", "building construction", "engineering"],
89 + "retail": ["retail", "retailing", "supermarket", "department store", "wholesale", "grocery store"],
90 + "consumer-goods": ["consumer goods", "cosmetics industry", "toy industry", "furniture industry", "fast-moving consumer goods",
91 + "household goods", "luxury goods", "personal care", "home appliance"],
92 + "energy": ["energy industry", "energy", "electric power industry", "nuclear power", "oil industry", "petroleum industry",
93 + "renewable energy", "solar energy", "wind power"],
94 + "utilities": ["public utility", "electric utility", "water industry", "waste management", "electricity generation",
95 + "electric power distribution", "water supply"],
96 + "mining": ["mining", "mining industry", "steel industry", "metallurgy", "gold mining", "steelmaking", "metal industry"],
97 + "chemicals": ["chemical industry", "chemicals", "petrochemical industry", "specialty chemicals", "plastics industry", "fertilizer"],
98 + "materials": ["glass industry", "paper industry", "packaging industry", "pulp and paper industry", "forestry", "building material",
99 + "cement industry", "wood industry"],
100 + "manufacturing": ["manufacturing", "mechanical engineering", "industrial machinery", "machine industry", "electrical engineering",
101 + "machine tool", "industrial engineering", "heavy industry", "shipbuilding"],
102 + "automotive": ["automotive industry", "automobile", "automotive", "motor vehicle", "auto parts"],
103 + "aerospace-defense": ["aerospace industry", "arms industry", "defense industry", "aerospace", "aviation industry", "space industry",
104 + "defence industry"],
105 + "transportation": ["transport", "rail transport", "public transport", "logistics", "transportation", "freight transport", "shipping",
106 + "maritime transport", "railway"],
107 + "telecommunications": ["telecommunications industry", "telecommunications", "telecommunication", "mobile telephony",
108 + "internet service provider"],
109 + "media": ["mass media", "publishing", "advertising", "broadcasting", "entertainment industry", "film industry", "video game industry",
110 + "music industry", "media industry", "newspaper"],
111 + "healthcare": ["health care industry", "health care", "medical technology", "hospital", "pharmaceutical industry", "biotechnology",
112 + "medical device", "healthcare industry", "medical equipment"],
113 + "hospitality": ["hospitality industry", "hotel industry", "hotel", "restaurant", "hospitality", "catering", "restaurant chain"],
114 + "travel": ["tourism", "travel agency", "travel industry", "travel", "tourism industry", "online travel agency", "tour operator"],
115 + "education": ["education", "educational technology", "higher education", "e-learning", "education industry", "educational services"],
116 + "professional-services": ["professional services", "consulting", "management consulting", "accounting", "outsourcing", "staffing",
117 + "business services", "legal services", "information technology consulting", "human resources"],
118 + "agriculture": ["agriculture", "agribusiness", "agricultural industry", "forestry", "fishing industry", "food industry", "farming",
119 + "aquaculture", "agricultural machinery"],
120 +}
121 +
122 +# Hosts that are never a company's own website (social profiles, blogs, stores, code forges, encyclopaedias, site builders …).
123 +GENERIC_DOMAINS = {
124 + "facebook.com", "fb.com", "linkedin.com", "twitter.com", "x.com", "instagram.com", "youtube.com", "youtu.be", "wikipedia.org",
125 + "wikimedia.org", "wikidata.org", "blogspot.com", "blogspot.co.uk", "wordpress.com", "tumblr.com", "medium.com", "github.com", "gitlab.com",
126 + "sourceforge.net", "archive.org", "tiktok.com", "vk.com", "weibo.com", "t.me", "telegram.me", "bit.ly", "wix.com", "wixsite.com",
127 + "weebly.com", "squarespace.com", "webnode.com", "jimdo.com", "jimdosite.com", "godaddysites.com", "carrd.co", "notion.site", "substack.com",
128 + "patreon.com", "itch.io", "steampowered.com", "bandcamp.com", "soundcloud.com", "imdb.com", "myspace.com", "flickr.com", "pinterest.com",
129 + "twitch.tv", "discord.gg", "discord.com", "bilibili.com", "tistory.com", "ameblo.jp", "fc2.com", "livejournal.com", "geocities.com",
130 + "angelfire.com", "tripod.com", "netlify.app", "vercel.app", "herokuapp.com", "github.io", "pages.dev", "web.app", "firebaseapp.com",
131 + "glitch.me", "strikingly.com", "yolasite.com", "webs.com", "linktr.ee", "about.me", "crunchbase.com", "bloomberg.com", "sec.gov",
132 + "yelp.com", "tripadvisor.com", "foursquare.com", "goo.gl", "ow.ly", "tinyurl.com", "wa.me", "whatsapp.com", "line.me", "kakao.com",
133 + "spotify.com", "deezer.com", "vimeo.com", "dailymotion.com", "behance.net", "dribbble.com", "etsy.com", "ebay.com", "aliexpress.com",
134 + "taobao.com", "tmall.com", "rakuten.co.jp", "shopee.com", "mercadolibre.com", "google.co.uk", "googleusercontent.com", "gstatic.com",
135 + "webflow.io", "mystrikingly.com", "site123.me", "simplesite.com", "ucoz.ru", "narod.ru", "hatenablog.com", "note.com", "wixstatic.com",
136 + "shopify.com", "myshopify.com", "bigcartel.com", "storenvy.com", "yahoo.co.jp", "yahoo.com", "aol.com", "cargo.site", "format.com",
137 + "blogger.com", "mixi.jp", "naver.me", "cafe24.com", "modoo.at", "over-blog.com", "canalblog.com", "skyrock.com", "free.fr", "orange.fr",
138 + "wanadoo.fr", "pagesperso-orange.fr", "t-online.de", "web.de", "gmx.de", "chello.at", "bplaced.net", "beepworld.de", "npage.de",
139 + "altervista.org", "xoom.it", "libero.it", "interfree.it", "terra.com.br", "uol.com.br", "ig.com.br", "sapo.pt", "webcindario.com",
140 + "iespana.es", "galeon.com", "hpage.com", "wordpress.org", "js.org", "readthedocs.io", "gitbook.io", "gumroad.com", "ko-fi.com",
141 + "onlyfans.com", "reddit.com", "quora.com", "scribd.com", "issuu.com", "slideshare.net", "docs.google.com", "drive.google.com",
142 + "sites.google.com", "play.google.com", "apps.apple.com", "itunes.apple.com", "amazon.com", "amazon.co.uk", "amazon.de", "amazon.co.jp",
143 + "amazon.fr", "amazon.ca", "amazon.in", "amazon.com.br", "amzn.to", "microsoft.com", "apple.com", "google.com", "naver.com", "daum.net",
144 + "qq.com", "163.com", "sina.com.cn", "baidu.com", "sohu.com", "douyin.com", "kuaishou.com", "zhihu.com", "xiaohongshu.com",
145 +}
146 +# Registrable domains that are themselves seed companies: only the bare/www host counts as that company's site (not sub-brands/store pages).
147 +PLATFORM_ROOTS = {"google.com": "www.google.com", "apple.com": "www.apple.com", "amazon.com": "www.amazon.com", "microsoft.com": "www.microsoft.com",
148 + "naver.com": "www.naver.com", "yahoo.com": "www.yahoo.com", "qq.com": "www.qq.com", "baidu.com": "www.baidu.com",
149 + "163.com": "www.163.com", "sohu.com": "www.sohu.com", "sina.com.cn": "www.sina.com.cn", "daum.net": "www.daum.net",
150 + "kakao.com": "www.kakao.com", "shopify.com": "www.shopify.com", "spotify.com": "www.spotify.com", "reddit.com": "www.reddit.com",
151 + "ebay.com": "www.ebay.com", "etsy.com": "www.etsy.com", "yelp.com": "www.yelp.com", "tripadvisor.com": "www.tripadvisor.com",
152 + "linkedin.com": "www.linkedin.com", "facebook.com": "www.facebook.com", "instagram.com": "www.instagram.com",
153 + "youtube.com": "www.youtube.com", "twitter.com": "twitter.com", "x.com": "x.com", "tiktok.com": "www.tiktok.com",
154 + "github.com": "github.com", "gitlab.com": "gitlab.com", "medium.com": "medium.com", "substack.com": "substack.com",
155 + "patreon.com": "www.patreon.com", "twitch.tv": "www.twitch.tv", "pinterest.com": "www.pinterest.com", "vimeo.com": "vimeo.com",
156 + "soundcloud.com": "soundcloud.com", "bandcamp.com": "bandcamp.com", "imdb.com": "www.imdb.com", "crunchbase.com": "www.crunchbase.com",
157 + "bloomberg.com": "www.bloomberg.com", "wix.com": "www.wix.com", "squarespace.com": "www.squarespace.com", "weebly.com": "www.weebly.com",
158 + "godaddy.com": "www.godaddy.com", "wordpress.com": "wordpress.com", "tumblr.com": "www.tumblr.com", "flickr.com": "www.flickr.com",
159 + "quora.com": "www.quora.com", "scribd.com": "www.scribd.com", "issuu.com": "issuu.com", "discord.com": "discord.com",
160 + "telegram.org": "telegram.org", "whatsapp.com": "www.whatsapp.com", "line.me": "line.me", "bilibili.com": "www.bilibili.com",
161 + "weibo.com": "weibo.com", "vk.com": "vk.com", "zhihu.com": "www.zhihu.com", "aliexpress.com": "www.aliexpress.com",
162 + "taobao.com": "www.taobao.com", "tmall.com": "www.tmall.com", "rakuten.co.jp": "www.rakuten.co.jp", "shopee.com": "shopee.com",
163 + "mercadolibre.com": "www.mercadolibre.com", "archive.org": "archive.org", "sourceforge.net": "sourceforge.net", "itch.io": "itch.io",
164 + "steampowered.com": "store.steampowered.com", "deezer.com": "www.deezer.com", "dailymotion.com": "www.dailymotion.com",
165 + "behance.net": "www.behance.net", "dribbble.com": "dribbble.com", "notion.so": "www.notion.so", "gumroad.com": "gumroad.com",
166 + "onlyfans.com": "onlyfans.com", "yahoo.co.jp": "www.yahoo.co.jp", "aol.com": "www.aol.com", "free.fr": "www.free.fr",
167 + "orange.fr": "www.orange.fr", "t-online.de": "www.t-online.de", "web.de": "web.de", "gmx.de": "www.gmx.de", "uol.com.br": "www.uol.com.br",
168 + "terra.com.br": "www.terra.com.br", "sapo.pt": "www.sapo.pt", "libero.it": "www.libero.it", "douyin.com": "www.douyin.com",
169 + "kuaishou.com": "www.kuaishou.com", "xiaohongshu.com": "www.xiaohongshu.com", "note.com": "note.com", "cafe24.com": "www.cafe24.com",
170 + "hatenablog.com": "hatenablog.com", "mixi.jp": "mixi.jp", "myspace.com": "myspace.com", "livejournal.com": "www.livejournal.com",
171 + "webflow.com": "webflow.com", "carrd.co": "carrd.co", "linktr.ee": "linktr.ee", "about.me": "about.me", "netlify.com": "www.netlify.com",
172 + "vercel.com": "vercel.com", "heroku.com": "www.heroku.com", "glitch.com": "glitch.com", "readthedocs.org": "readthedocs.org",
173 + "gitbook.com": "www.gitbook.com", "ko-fi.com": "ko-fi.com", "foursquare.com": "foursquare.com", "slideshare.net": "www.slideshare.net"}
174 +
175 +
176 +# ------------------------------------------------------------------------------------------------------------ SPARQL client
177 +class Sparql:
178 + def __init__(self, *, refresh: bool = False) -> None:
179 + self.client = httpx.Client(headers={"User-Agent": USER_AGENT, "Accept": "application/sparql-results+json"}, timeout=TIMEOUT_S + 15,
180 + follow_redirects=True)
181 + self.last_call = 0.0
182 + self.refresh = refresh
183 + self.queries = 0
184 + self.cached = 0
185 + CACHE_DIR.mkdir(parents=True, exist_ok=True)
186 +
187 + def query(self, sparql: str, *, label: str = "") -> list[dict[str, str]]:
188 + key = hashlib.sha256(sparql.encode("utf-8")).hexdigest()
189 + path = CACHE_DIR / f"{key}.json"
190 + if path.exists() and not self.refresh:
191 + self.cached += 1
192 + return json.loads(path.read_text(encoding="utf-8"))["bindings"]
193 + delay = 5.0
194 + last_err = ""
195 + gateway_timeouts = 0
196 + for attempt in range(1, MAX_TRIES + 1):
197 + wait = MIN_INTERVAL_S - (time.monotonic() - self.last_call)
198 + if wait > 0:
199 + time.sleep(wait)
200 + t0 = time.monotonic()
201 + try:
202 + r = self.client.get(SPARQL_URL, params={"query": sparql, "format": "json"})
203 + self.last_call = time.monotonic()
204 + self.queries += 1
205 + if r.status_code == 200:
206 + try:
207 + # strict=False: a handful of Wikidata literals contain raw control characters.
208 + payload = json.loads(r.text, strict=False)
209 + except json.JSONDecodeError:
210 + # The endpoint streams results and, on a server-side timeout, appends a Java stack trace to a *200* body
211 + # (which the gateway may even cache). Treat a truncated body as a timeout so callers can split or skip.
212 + if "TimeoutException" in r.text or "SPARQL-QUERY" in r.text:
213 + raise TimeoutError(f"server timeout (truncated body) {label}") from None
214 + last_err = "truncated/invalid JSON body"
215 + continue
216 + rows = [{k: v["value"] for k, v in b.items()} for b in payload["results"]["bindings"]]
217 + path.write_text(json.dumps({"label": label, "fetched_at": datetime.now(UTC).isoformat(), "query": sparql, "bindings": rows},
218 + ensure_ascii=False), encoding="utf-8")
219 + log.info("sparql ok %s rows=%d %.1fs", label, len(rows), self.last_call - t0)
220 + return rows
221 + last_err = f"HTTP {r.status_code}: {r.text[:160]!r}"
222 + if r.status_code == 504:
223 + # Gateway timeout = the query ran past the 60 s server limit. One retry (load varies), then let the caller split it.
224 + gateway_timeouts += 1
225 + if gateway_timeouts >= 2:
226 + raise TimeoutError(f"gateway timeout {label}")
227 + if r.status_code == 429:
228 + retry_after = r.headers.get("Retry-After")
229 + delay = max(delay, float(retry_after)) if retry_after and retry_after.isdigit() else max(delay, 30.0)
230 + if r.status_code == 400:
231 + raise RuntimeError(f"bad query {label}: {r.text[:500]}")
232 + if r.status_code == 500 and "TimeoutException" in r.text:
233 + raise TimeoutError(f"server timeout {label}")
234 + except (httpx.TimeoutException, httpx.TransportError) as e:
235 + self.last_call = time.monotonic()
236 + last_err = f"{type(e).__name__}: {e}"
237 + log.warning("sparql retry %d/%d %s (%s) sleeping %.0fs", attempt, MAX_TRIES, label, last_err, delay)
238 + time.sleep(delay)
239 + delay = min(delay * 2, 120.0)
240 + raise RuntimeError(f"sparql failed {label}: {last_err}")
241 +
242 +
243 +def values_clause(var: str, qids: list[str]) -> str:
244 + return f"VALUES ?{var} {{ {' '.join('wd:' + q for q in qids)} }}"
245 +
246 +
247 +def class_values() -> str:
248 + return values_clause("cls", list(CLASSES))
249 +
250 +
251 +# Truthy website (best rank, deprecated excluded). Dissolution is fetched as an OPTIONAL and filtered client-side: `FILTER NOT EXISTS`
252 +# sub-selects over tens of thousands of bindings are what pushed the big class scans past the 60 s server limit.
253 +WEBSITE_BLOCK = """
254 + ?item wdt:P856 ?web .
255 + OPTIONAL { ?item wdt:P576 ?dissolved }
256 + OPTIONAL { ?item wdt:P17 ?c . ?c wdt:P297 ?iso }
257 +"""
258 +
259 +
260 +def q_class_band(cls: str, lo: int, hi: int | None) -> str:
261 + band = f"?sl >= {lo}" + (f" && ?sl < {hi}" if hi else "")
262 + return f"""SELECT ?item ?sl ?web ?iso WHERE {{
263 + ?item wdt:P31 wd:{cls} ; wikibase:sitelinks ?sl .
264 + FILTER({band})
265 + {WEBSITE_BLOCK}
266 +}}"""
267 +
268 +
269 +def q_country_qids(isos: list[str]) -> str:
270 + vals = " ".join(json.dumps(x) for x in isos)
271 + return f"""SELECT ?iso ?country WHERE {{ VALUES ?iso {{ {vals} }} ?country wdt:P297 ?iso . FILTER NOT EXISTS {{ ?country wdt:P576 [] }} }}"""
272 +
273 +
274 +def q_country_boost(iso: str, country_qid: str) -> str:
275 + """Country-first scan (P17 → website → class) with the optimizer pinned. Cheap for countries with few items in Wikidata; the
276 + caller only issues it for countries still short of their minimum after the class stage and tolerates a timeout."""
277 + return f"""SELECT ?item ?sl ?web ?iso WHERE {{
278 + hint:Query hint:optimizer "None" .
279 + ?item wdt:P17 wd:{country_qid} .
280 + ?item wdt:P856 ?web .
281 + ?item wdt:P31 ?cls .
282 + {class_values()}
283 + ?item wikibase:sitelinks ?sl . FILTER(?sl >= {BOOST_MIN_SITELINKS})
284 + OPTIONAL {{ ?item wdt:P576 ?dissolved }}
285 + BIND("{iso}" AS ?iso)
286 +}}"""
287 +
288 +
289 +def q_industry_boost(labels: list[str]) -> str:
290 + vals = " ".join(json.dumps(x) + "@en" for x in labels)
291 + return f"""SELECT ?item ?sl ?web ?iso ?indLabel WHERE {{
292 + hint:Query hint:optimizer "None" .
293 + VALUES ?indLabel {{ {vals} }}
294 + ?ind rdfs:label ?indLabel .
295 + ?item wdt:P452 ?ind .
296 + ?item wikibase:sitelinks ?sl .
297 + FILTER(?sl >= {BOOST_MIN_SITELINKS})
298 + ?item wdt:P31 ?cls .
299 + {class_values()}
300 + {WEBSITE_BLOCK}
301 +}} ORDER BY DESC(?sl) LIMIT {INDUSTRY_BOOST_LIMIT}"""
302 +
303 +
304 +def q_labels(qids: list[str]) -> str:
305 + return f"""SELECT ?item ?itemLabel ?itemDescription ?itemAltLabel WHERE {{
306 + {values_clause("item", qids)}
307 + SERVICE wikibase:label {{ bd:serviceParam wikibase:language "en". }}
308 +}}"""
309 +
310 +
311 +# Plain (non-aggregated) detail queries, reduced client-side: a GROUP BY with a dozen SAMPLE() aggregates over nested OPTIONALs makes
312 +# Blazegraph throw StackOverflowError.
313 +def q_misc(qids: list[str]) -> str:
314 + return f"""SELECT ?item ?inception ?coord ?lei ?cik ?logo WHERE {{
315 + {values_clause("item", qids)}
316 + OPTIONAL {{ ?item wdt:P571 ?inception }}
317 + OPTIONAL {{ ?item wdt:P625 ?coord }}
318 + OPTIONAL {{ ?item wdt:P1278 ?lei }}
319 + OPTIONAL {{ ?item wdt:P5531 ?cik }}
320 + OPTIONAL {{ ?item wdt:P154 ?logo }}
321 +}}"""
322 +
323 +
324 +def q_hq_parent(qids: list[str]) -> str:
325 + return f"""SELECT ?item ?hq ?hqEnd ?hqLabel ?hqcoord ?hqRegionLabel ?hqiso ?parent ?parentEnd ?parentLabel WHERE {{
326 + {values_clause("item", qids)}
327 + OPTIONAL {{
328 + ?item p:P159 ?hqs . ?hqs ps:P159 ?hq .
329 + OPTIONAL {{ ?hqs pq:P582 ?hqEnd }}
330 + OPTIONAL {{ ?hq rdfs:label ?hqLabel FILTER(LANG(?hqLabel) = "en") }}
331 + OPTIONAL {{ ?hq wdt:P625 ?hqcoord }}
332 + OPTIONAL {{ ?hq wdt:P131 ?hqRegion . ?hqRegion rdfs:label ?hqRegionLabel FILTER(LANG(?hqRegionLabel) = "en") }}
333 + OPTIONAL {{ ?hq wdt:P17 ?hqc . ?hqc wdt:P297 ?hqiso }}
334 + }}
335 + OPTIONAL {{
336 + ?item p:P749 ?ps . ?ps ps:P749 ?parent .
337 + OPTIONAL {{ ?ps pq:P582 ?parentEnd }}
338 + OPTIONAL {{ ?parent rdfs:label ?parentLabel FILTER(LANG(?parentLabel) = "en") }}
339 + }}
340 +}}"""
341 +
342 +
343 +def q_names_industries(qids: list[str]) -> str:
344 + return f"""SELECT ?item ?legal ?indLabel WHERE {{
345 + {values_clause("item", qids)}
346 + OPTIONAL {{ ?item wdt:P1448 ?legal }}
347 + OPTIONAL {{ ?item wdt:P452 ?ind . ?ind rdfs:label ?indLabel FILTER(LANG(?indLabel) = "en") }}
348 +}}"""
349 +
350 +
351 +def q_tickers_employees(qids: list[str]) -> str:
352 + return f"""SELECT ?item ?ticker ?exchangeLabel ?ticker2 ?employees ?empDate WHERE {{
353 + {values_clause("item", qids)}
354 + OPTIONAL {{
355 + ?item p:P414 ?exs . ?exs ps:P414 ?exchange . FILTER NOT EXISTS {{ ?exs pq:P582 [] }}
356 + OPTIONAL {{ ?exs pq:P249 ?ticker }}
357 + OPTIONAL {{ ?exchange rdfs:label ?exchangeLabel FILTER(LANG(?exchangeLabel) = "en") }}
358 + }}
359 + OPTIONAL {{ ?item wdt:P249 ?ticker2 }}
360 + OPTIONAL {{ ?item p:P1128 ?es . ?es ps:P1128 ?employees . OPTIONAL {{ ?es pq:P585 ?empDate }} }}
361 +}}"""
362 +
363 +
364 +# ------------------------------------------------------------------------------------------------------------ helpers
365 +def qid_of(uri: str) -> str:
366 + return uri.rsplit("/", 1)[-1]
367 +
368 +
369 +def load_countries() -> dict[str, dict[str, str]]:
370 + with COUNTRIES_CSV.open(encoding="utf-8") as f:
371 + return {r["code"]: r for r in csv.DictReader(f)}
372 +
373 +
374 +def normalise_website(url: str) -> tuple[str, str] | None:
375 + """→ (website `https://host`, registrable domain) or None when unusable/generic."""
376 + url = (url or "").strip()
377 + if not url:
378 + return None
379 + if "://" not in url:
380 + url = "https://" + url
381 + try:
382 + p = urlparse(url)
383 + except ValueError:
384 + return None
385 + host = (p.hostname or "").lower().strip().rstrip(".")
386 + if not host or "." not in host or re.fullmatch(r"[\d.]+", host) or ":" in host:
387 + return None
388 + if p.scheme not in ("http", "https"):
389 + return None
390 + try:
391 + host.encode("idna")
392 + except UnicodeError:
393 + return None
394 + dom = registrable_domain(host)
395 + if not dom or "." not in dom:
396 + return None
397 + path = (p.path or "/").rstrip("/")
398 + if dom in PLATFORM_ROOTS:
399 + canonical_host = PLATFORM_ROOTS[dom]
400 + if host not in (canonical_host, dom, "www." + dom) or path:
401 + return None
402 + return f"https://{canonical_host}", dom
403 + if dom in GENERIC_DOMAINS or host in GENERIC_DOMAINS:
404 + return None
405 + if host.endswith((".blogspot.com", ".wordpress.com", ".github.io", ".wixsite.com", ".weebly.com", ".tumblr.com")):
406 + return None
407 + return f"https://{host}", dom
408 +
409 +
410 +def parse_point(value: str | None) -> tuple[float, float] | None:
411 + if not value or not value.startswith("Point("):
412 + return None
413 + try:
414 + lon, lat = value[6:-1].split()
415 + return round(float(lat), 5), round(float(lon), 5)
416 + except ValueError:
417 + return None
418 +
419 +
420 +def parse_year(value: str | None) -> int | None:
421 + if not value:
422 + return None
423 + m = re.match(r"^(-?\d{1,4})-", value)
424 + if not m:
425 + return None
426 + y = int(m.group(1))
427 + return y if 1000 <= y <= datetime.now(UTC).year else None
428 +
429 +
430 +def parse_int(value: str | None) -> int | None:
431 + try:
432 + return int(float(value)) if value not in (None, "") else None
433 + except ValueError:
434 + return None
435 +
436 +
437 +def commons_url(filename: str | None) -> str | None:
438 + if not filename:
439 + return None
440 + name = filename.rsplit("/", 1)[-1].replace(" ", "_")
441 + return f"https://commons.wikimedia.org/wiki/Special:FilePath/{quote(name)}"
442 +
443 +
444 +def dump_json(path: Path, data: Any) -> None:
445 + path.parent.mkdir(parents=True, exist_ok=True)
446 + path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
447 +
448 +
449 +def load_json(path: Path) -> Any:
450 + return json.loads(path.read_text(encoding="utf-8"))
451 +
452 +
453 +# ------------------------------------------------------------------------------------------------------------ stage: candidates
454 +def stage_candidates(sp: Sparql, *, refresh: bool = False) -> dict[str, dict[str, Any]]:
455 + out_path = DATA_DIR / "candidates.json"
456 + cands: dict[str, dict[str, Any]] = {}
457 + dissolved: set[str] = set()
458 +
459 + def add(row: dict[str, str], *, cls: str | None = None, boost_country: str | None = None, boost_industry: str | None = None) -> None:
460 + if row.get("dissolved"):
461 + dissolved.add(qid_of(row["item"]))
462 + return
463 + qid = qid_of(row["item"])
464 + c = cands.setdefault(qid, {"sitelinks": 0, "websites": [], "isos": [], "classes": [], "boost_countries": [], "boost_industries": []})
465 + c["sitelinks"] = max(c["sitelinks"], parse_int(row.get("sl")) or 0)
466 + if row.get("web") and row["web"] not in c["websites"]:
467 + c["websites"].append(row["web"])
468 + iso = (row.get("iso") or "").upper()
469 + if iso and iso not in c["isos"]:
470 + c["isos"].append(iso)
471 + if cls and cls not in c["classes"]:
472 + c["classes"].append(cls)
473 + if boost_country and boost_country not in c["boost_countries"]:
474 + c["boost_countries"].append(boost_country)
475 + if boost_industry and boost_industry not in c["boost_industries"]:
476 + c["boost_industries"].append(boost_industry)
477 +
478 + def run_band(cls: str, lo: int, hi: int | None) -> None:
479 + label = f"class {cls} {CLASSES[cls]} sl[{lo},{hi or '∞'})"
480 + try:
481 + rows = sp.query(q_class_band(cls, lo, hi), label=label)
482 + except TimeoutError:
483 + if hi is None:
484 + hi = 400
485 + if hi - lo <= 1:
486 + raise
487 + mid = (lo + hi) // 2
488 + log.warning("splitting band %s → [%d,%d) [%d,%d)", label, lo, mid, mid, hi)
489 + run_band(cls, lo, mid)
490 + run_band(cls, mid, hi)
491 + return
492 + for r in rows:
493 + add(r, cls=cls)
494 +
495 + for cls in CLASSES:
496 + if cls in BANDED_CLASSES:
497 + for lo, hi in SITELINK_BANDS:
498 + run_band(cls, lo, hi)
499 + else:
500 + run_band(cls, MIN_SITELINKS, None)
501 + log.info("candidates so far: %d", len(cands))
502 + iso_to_qid = {r["iso"]: qid_of(r["country"]) for r in sp.query(q_country_qids(list(COUNTRY_MINIMUMS)), label="country qids")}
503 + per_country: Counter[str] = Counter(c["isos"][0] for c in cands.values() if c["isos"])
504 + for iso, minimum in COUNTRY_MINIMUMS.items():
505 + if per_country[iso] >= minimum * BOOST_MARGIN or iso not in iso_to_qid:
506 + log.info("country %s: %d candidates (min %d) — no boost", iso, per_country[iso], minimum)
507 + continue
508 + try:
509 + rows = sp.query(q_country_boost(iso, iso_to_qid[iso]), label=f"country boost {iso}")
510 + except (RuntimeError, TimeoutError) as e:
511 + log.warning("country boost %s failed (%s) — known gap, see README", iso, e)
512 + continue
513 + for r in rows:
514 + add(r, boost_country=iso)
515 + for slug, labels in INDUSTRY_BOOST_LABELS.items():
516 + for r in sp.query(q_industry_boost(labels), label=f"industry boost {slug}"):
517 + add(r, boost_industry=slug)
518 + for qid in dissolved:
519 + cands.pop(qid, None)
520 + dump_json(out_path, cands)
521 + log.info("candidates: %d (%d dissolved dropped) → %s", len(cands), len(dissolved), out_path)
522 + return cands
523 +
524 +
525 +# ------------------------------------------------------------------------------------------------------------ stage: select
526 +def prefilter(cands: dict[str, dict[str, Any]], countries: dict[str, dict[str, str]]) -> tuple[list[dict[str, Any]], dict[str, int]]:
527 + """Normalise websites, pick one country, drop generic hosts and duplicate registrable domains (highest sitelinks wins)."""
528 + stats: Counter[str] = Counter()
529 + rows: list[dict[str, Any]] = []
530 + for qid, c in cands.items():
531 + picked = None
532 + for w in c["websites"]:
533 + picked = normalise_website(w)
534 + if picked:
535 + break
536 + if not picked:
537 + stats["dropped_generic_or_invalid_website"] += 1
538 + continue
539 + website, dom = picked
540 + isos = [i for i in c["isos"] if i in countries]
541 + country = isos[0] if isos else None
542 + rows.append({"wikidata_id": qid, "sitelinks": c["sitelinks"], "website": website, "canonical_domain": dom, "country": country,
543 + "country_candidates": isos, "classes": c["classes"], "boost_industries": c["boost_industries"]})
544 + rows.sort(key=lambda r: (-r["sitelinks"], int(r["wikidata_id"][1:])))
545 + by_domain: dict[str, dict[str, Any]] = {}
546 + for r in rows:
547 + keeper = by_domain.get(r["canonical_domain"])
548 + if keeper is None:
549 + by_domain[r["canonical_domain"]] = r
550 + else:
551 + keeper.setdefault("domain_conflicts", []).append(r["wikidata_id"])
552 + stats["dropped_duplicate_domain"] += 1
553 + kept = list(by_domain.values())
554 + stats["kept"] = len(kept)
555 + return kept, dict(stats)
556 +
557 +
558 +def select_diversified(rows: list[dict[str, Any]], *, target: int, countries: dict[str, dict[str, str]]) -> list[dict[str, Any]]:
559 + """Sitelinks-ordered selection with country caps and minimum coverage. Rows must be sorted by sitelinks desc."""
560 + caps: dict[str | None, int] = defaultdict(lambda: int(target * OTHER_CAP_SHARE))
561 + caps["US"] = int(target * US_CAP_SHARE)
562 + caps[None] = int(target * UNKNOWN_COUNTRY_SHARE)
563 + chosen: dict[str, dict[str, Any]] = {}
564 + per_country: Counter[str | None] = Counter()
565 +
566 + def take(r: dict[str, Any]) -> None:
567 + chosen[r["wikidata_id"]] = r
568 + per_country[r["country"]] += 1
569 +
570 + by_country: dict[str | None, list[dict[str, Any]]] = defaultdict(list)
571 + for r in rows:
572 + by_country[r["country"]].append(r)
573 + for iso, minimum in COUNTRY_MINIMUMS.items():
574 + for r in by_country.get(iso, [])[:minimum]:
575 + take(r)
576 + for r in rows:
577 + if len(chosen) >= target:
578 + break
579 + if r["wikidata_id"] in chosen or r["sitelinks"] < MIN_SITELINKS:
580 + continue
581 + if per_country[r["country"]] >= caps[r["country"]]:
582 + continue
583 + take(r)
584 + return list(chosen.values())
585 +
586 +
587 +def stage_select(*, target: int) -> list[dict[str, Any]]:
588 + countries = load_countries()
589 + cands = load_json(DATA_DIR / "candidates.json")
590 + kept, stats = prefilter(cands, countries)
591 + log.info("prefilter: %s", stats)
592 + selected = select_diversified(kept, target=target, countries=countries)
593 + # Pool for the industry guarantee: best boost candidates per industry, fetched in the details stage too (≤ INDUSTRY_RESERVE each).
594 + chosen_ids = {r["wikidata_id"] for r in selected}
595 + reserve: list[dict[str, Any]] = []
596 + per_ind: Counter[str] = Counter()
597 + for r in kept:
598 + if r["wikidata_id"] in chosen_ids:
599 + continue
600 + for slug in r["boost_industries"]:
601 + if per_ind[slug] < INDUSTRY_RESERVE:
602 + per_ind[slug] += 1
603 + reserve.append(r)
604 + break
605 + dump_json(DATA_DIR / "selected.json", {"selected": selected, "reserve": reserve, "stats": stats, "target": target})
606 + log.info("selected %d (+%d reserve for industry minimums) → %s", len(selected), len(reserve), DATA_DIR / "selected.json")
607 + return selected
608 +
609 +
610 +# ------------------------------------------------------------------------------------------------------------ stage: details
611 +def stage_details(sp: Sparql, *, batch: int = 100) -> dict[str, dict[str, Any]]:
612 + sel = load_json(DATA_DIR / "selected.json")
613 + qids = [r["wikidata_id"] for r in sel["selected"]] + [r["wikidata_id"] for r in sel["reserve"]]
614 + out_path = DATA_DIR / "details.json"
615 + details: dict[str, dict[str, Any]] = load_json(out_path) if out_path.exists() else {}
616 + todo = [q for q in qids if q not in details]
617 + log.info("details: %d to fetch (%d cached)", len(todo), len(qids) - len(todo))
618 + for i in range(0, len(todo), batch):
619 + chunk = todo[i:i + batch]
620 + tag = f"details {i + len(chunk)}/{len(todo)}"
621 + d: dict[str, dict[str, Any]] = {q: {"legal_names": [], "industry_labels": [], "tickers": [], "employees_obs": []} for q in chunk}
622 + for r in sp.query(q_labels(chunk), label=f"{tag} labels"):
623 + q = qid_of(r["item"])
624 + d[q]["label"] = r.get("itemLabel")
625 + d[q]["description"] = r.get("itemDescription")
626 + d[q]["alt_labels"] = [a.strip() for a in (r.get("itemAltLabel") or "").split(",") if a.strip()]
627 + for r in sp.query(q_misc(chunk), label=f"{tag} misc"):
628 + m = d[qid_of(r["item"])]
629 + for key in ("inception", "coord", "lei", "cik", "logo"):
630 + if r.get(key) and (not m.get(key) or (key == "inception" and r[key] < m[key])):
631 + m[key] = r[key]
632 + grouped: dict[str, list[dict[str, str]]] = defaultdict(list)
633 + for r in sp.query(q_hq_parent(chunk), label=f"{tag} hq/parent"):
634 + grouped[qid_of(r["item"])].append(r)
635 + for q, rs in grouped.items():
636 + hqs = [r for r in rs if r.get("hq")]
637 + current = [r for r in hqs if not r.get("hqEnd")] or hqs
638 + if current:
639 + r = current[0]
640 + d[q].update({"hq": qid_of(r["hq"]), "hq_label": r.get("hqLabel"), "hq_coord": r.get("hqcoord"),
641 + "hq_region": r.get("hqRegionLabel"), "hq_iso": r.get("hqiso")})
642 + parents = [r for r in rs if r.get("parent") and not r.get("parentEnd")]
643 + if parents:
644 + d[q].update({"parent": qid_of(parents[0]["parent"]), "parent_label": parents[0].get("parentLabel")})
645 + for r in sp.query(q_names_industries(chunk), label=f"{tag} names/industries"):
646 + q = qid_of(r["item"])
647 + if r.get("legal") and r["legal"] not in d[q]["legal_names"]:
648 + d[q]["legal_names"].append(r["legal"])
649 + if r.get("indLabel") and r["indLabel"] not in d[q]["industry_labels"]:
650 + d[q]["industry_labels"].append(r["indLabel"])
651 + for r in sp.query(q_tickers_employees(chunk), label=f"{tag} tickers/employees"):
652 + q = qid_of(r["item"])
653 + t = r.get("ticker") or r.get("ticker2")
654 + if t:
655 + pair = [t, r.get("exchangeLabel")]
656 + if pair not in d[q]["tickers"]:
657 + d[q]["tickers"].append(pair)
658 + if r.get("employees"):
659 + obs = [r["employees"], r.get("empDate")]
660 + if obs not in d[q]["employees_obs"]:
661 + d[q]["employees_obs"].append(obs)
662 + details.update(d)
663 + dump_json(out_path, details)
664 + log.info("details: %d entries → %s", len(details), out_path)
665 + return details
666 +
667 +
668 +# ------------------------------------------------------------------------------------------------------------ stage: assemble
669 +def importance_score(sitelinks: int, employees: int | None, public: bool, ticker: str | None) -> float:
670 + s_sl = min(1.0, math.log1p(max(sitelinks, 0)) / math.log1p(300))
671 + s_emp = min(1.0, math.log10((employees or 0) + 1) / 6) if employees else 0.0
672 + score = 0.6 * s_sl + 0.2 * s_emp + 0.1 * (1 if public else 0) + 0.1 * (1 if ticker else 0)
673 + return round(min(1.0, max(0.02, score)), 4)
674 +
675 +
676 +def assign_tiers(rows: list[dict[str, Any]]) -> None:
677 + rows.sort(key=lambda r: (-r["importance"], -r["sitelinks"], r["wikidata_id"]))
678 + for i, r in enumerate(rows):
679 + r["tier"] = 1 if i < 150 else 2 if i < 950 else 3 if i < 3450 else 4
680 +
681 +
682 +def build_row(sel: dict[str, Any], d: dict[str, Any], harvested_at: str) -> dict[str, Any]:
683 + label = d.get("label") or ""
684 + if not label or re.fullmatch(r"Q\d+", label):
685 + label = (d.get("legal_names") or [sel["canonical_domain"]])[0]
686 + legal = next((x for x in d.get("legal_names", []) if re.search(r"[A-Za-z]", x)), None) or (d.get("legal_names") or [None])[0]
687 + aliases = [a for a in d.get("alt_labels", []) if a and a != label and len(a) <= 80][:8]
688 + emp = None
689 + obs = d.get("employees_obs") or []
690 + if obs:
691 + obs = sorted(obs, key=lambda o: (o[1] or ""), reverse=True)
692 + emp = parse_int(obs[0][0])
693 + tickers = d.get("tickers") or []
694 + ticker, exchange = (tickers[0][0], tickers[0][1]) if tickers else (None, None)
695 + public = "Q891723" in sel.get("classes", []) or bool(ticker) or bool(exchange)
696 + coord = parse_point(d.get("coord")) or parse_point(d.get("hq_coord"))
697 + country = sel["country"]
698 + if d.get("hq_iso") and d["hq_iso"] in (sel.get("country_candidates") or []):
699 + country = d["hq_iso"]
700 + industry_labels = d.get("industry_labels") or []
701 + slugs = map_industry(industry_labels)
702 + if not slugs and sel.get("boost_industries"):
703 + slugs = [sel["boost_industries"][0]]
704 + if not slugs:
705 + slugs = map_industry([label, d.get("description") or ""] + [CLASSES.get(c, "") for c in sel.get("classes", [])], limit=2)
706 + return {
707 + "wikidata_id": sel["wikidata_id"], "display_name": label.strip(), "legal_name": legal, "aliases": aliases, "website": sel["website"],
708 + "canonical_domain": sel["canonical_domain"], "country": country, "hq_city": d.get("hq_label"), "hq_region": d.get("hq_region"),
709 + "lat": coord[0] if coord else None, "lon": coord[1] if coord else None, "industries": slugs, "industry_labels": industry_labels[:8],
710 + "founded_year": parse_year(d.get("inception")), "employees": emp, "public_company": public, "ticker": ticker, "exchange": exchange,
711 + "lei": d.get("lei"), "sec_cik": str(d["cik"]).lstrip("0") or None if d.get("cik") else None,
712 + "parent": {"wikidata_id": d["parent"], "name": d.get("parent_label")} if d.get("parent") else None,
713 + "logo_url": commons_url(d.get("logo")), "description": (d.get("description") or None), "sitelinks": sel["sitelinks"],
714 + "importance": importance_score(sel["sitelinks"], emp, public, ticker), "tier": 4, "source": "wikidata", "harvested_at": harvested_at,
715 + "domain_conflicts": sel.get("domain_conflicts") or [],
716 + }
717 +
718 +
719 +def stage_assemble() -> list[dict[str, Any]]:
720 + countries = load_countries()
721 + sel = load_json(DATA_DIR / "selected.json")
722 + details = load_json(DATA_DIR / "details.json")
723 + harvested_at = datetime.now(UTC).replace(microsecond=0).isoformat()
724 + selected = [r for r in sel["selected"] if r["wikidata_id"] in details]
725 + reserve = [r for r in sel["reserve"] if r["wikidata_id"] in details]
726 + rows = [build_row(s, details[s["wikidata_id"]], harvested_at) for s in selected]
727 + # Industry minimums: top up from the reserve where a top-level industry is below the floor.
728 + counts: Counter[str] = Counter()
729 + for r in rows:
730 + for top in {top_level_of(s) for s in r["industries"]}:
731 + counts[top] += 1
732 + chosen = {r["wikidata_id"] for r in rows}
733 + reserve_rows = [build_row(s, details[s["wikidata_id"]], harvested_at) for s in reserve]
734 + reserve_rows.sort(key=lambda r: -r["sitelinks"])
735 + for top in top_level_slugs():
736 + for rr in reserve_rows:
737 + if counts[top] >= INDUSTRY_MINIMUM:
738 + break
739 + if rr["wikidata_id"] in chosen or top not in {top_level_of(s) for s in rr["industries"]}:
740 + continue
741 + rows.append(rr)
742 + chosen.add(rr["wikidata_id"])
743 + for t in {top_level_of(s) for s in rr["industries"]}:
744 + counts[t] += 1
745 + # Parent/child sharing a domain: parent wins (regardless of sitelinks); duplicate domains (should not happen after prefilter) drop the lower.
746 + by_domain: dict[str, dict[str, Any]] = {}
747 + for r in sorted(rows, key=lambda r: -r["sitelinks"]):
748 + by_domain.setdefault(r["canonical_domain"], r)
749 + ids = {r["wikidata_id"] for r in by_domain.values()}
750 + dropped: list[dict[str, Any]] = []
751 + for r in list(by_domain.values()):
752 + p = r.get("parent")
753 + if p and p["wikidata_id"] in ids and p["wikidata_id"] != r["wikidata_id"]:
754 + parent_row = next((x for x in by_domain.values() if x["wikidata_id"] == p["wikidata_id"]), None)
755 + if parent_row and parent_row["canonical_domain"] == r["canonical_domain"]:
756 + dropped.append({"wikidata_id": r["wikidata_id"], "reason": "shares_parent_domain", "parent": p["wikidata_id"]})
757 + for r in rows:
758 + if r["wikidata_id"] not in ids:
759 + dropped.append({"wikidata_id": r["wikidata_id"], "reason": "duplicate_domain", "domain": r["canonical_domain"]})
760 + drop_ids = {x["wikidata_id"] for x in dropped}
761 + final = [r for r in by_domain.values() if r["wikidata_id"] not in drop_ids]
762 + for r in final:
763 + r["notes"] = [{"related_domain_conflict": q} for q in r.pop("domain_conflicts", [])] or []
764 + if not r["notes"]:
765 + r.pop("notes")
766 + assign_tiers(final)
767 + # Write per region.
768 + OUT_DIR.mkdir(parents=True, exist_ok=True)
769 + for old in OUT_DIR.glob("wikidata-*.ndjson"):
770 + old.unlink()
771 + by_region: dict[str, list[dict[str, Any]]] = defaultdict(list)
772 + for r in final:
773 + region = (countries.get(r["country"] or "", {}).get("region") or "other").lower().replace(" ", "-")
774 + by_region[region].append(r)
775 + for region, items in sorted(by_region.items()):
776 + items.sort(key=lambda r: (r["tier"], -r["importance"], r["wikidata_id"]))
777 + with (OUT_DIR / f"wikidata-{region}.ndjson").open("w", encoding="utf-8") as f:
778 + for r in items:
779 + f.write(json.dumps(r, ensure_ascii=False) + "\n")
780 + dump_json(DATA_DIR / "dropped.json", dropped)
781 + write_readme(final, countries)
782 + log.info("assembled %d companies into %d region files (%d dropped) → %s", len(final), len(by_region), len(dropped), OUT_DIR)
783 + return final
784 +
785 +
786 +def write_readme(rows: list[dict[str, Any]], countries: dict[str, dict[str, str]]) -> None:
787 + names = {i.slug: i.name for i in load_industries()}
788 + by_country = Counter(r["country"] or "??" for r in rows)
789 + by_tier = Counter(r["tier"] for r in rows)
790 + by_top: Counter[str] = Counter()
791 + by_ind: Counter[str] = Counter()
792 + for r in rows:
793 + for s in r["industries"]:
794 + by_ind[s] += 1
795 + for t in {top_level_of(s) for s in r["industries"]}:
796 + by_top[t] += 1
797 + no_ind = sum(1 for r in rows if not r["industries"])
798 + public = sum(1 for r in rows if r["public_company"])
799 + region = Counter((countries.get(r["country"] or "", {}).get("region") or "other") for r in rows)
800 + generated = rows[0]["harvested_at"] if rows else ""
801 + intro = (f"Generated by `scripts/seed_wikidata.py` on {generated}. **{len(rows)} companies**, {public} public, {no_ind} without an "
802 + f"industry mapping, {len(by_country)} countries.")
803 + files_note = ("Files: one NDJSON per UN region (`wikidata-<region>.ndjson`), one JSON object per line — see `docs/SEEDS.md` for the "
804 + "schema, the diversification rules and how to add companies.")
805 + lines = [
806 + "# Seed registry — Wikidata harvest", "", intro, "", files_note, "",
807 + "## Tiers", "", "| Tier | Companies |", "|---|---|",
808 + *[f"| {t} ({ {1: 'global', 2: 'major', 3: 'notable', 4: 'long tail'}[t] }) | {by_tier[t]} |" for t in sorted(by_tier)], "",
809 + "## Regions", "", "| Region | Companies |", "|---|---|", *[f"| {k} | {v} |" for k, v in region.most_common()], "",
810 + "## Countries", "", "| Code | Country | Companies | Share |", "|---|---|---|---|",
811 + *[f"| {c} | {countries.get(c, {}).get('name', 'unknown')} | {n} | {100 * n / len(rows):.1f} % |" for c, n in by_country.most_common()], "",
812 + "## Top-level industries (a company counts once per top-level sector)", "", "| Industry | Companies |", "|---|---|",
813 + *[f"| {names.get(k, k)} | {v} |" for k, v in by_top.most_common()], "",
814 + "## All industries", "", "| Slug | Companies |", "|---|---|", *[f"| {k} | {v} |" for k, v in by_ind.most_common()], "",
815 + ]
816 + (OUT_DIR / "README.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
817 +
818 +
819 +# ------------------------------------------------------------------------------------------------------------ main
820 +def main() -> int:
821 + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
822 + ap.add_argument("stage", nargs="?", default="all", choices=["candidates", "select", "details", "assemble", "all"])
823 + ap.add_argument("--target", type=int, default=8000, help="companies to select before detail fetching (default 8000)")
824 + ap.add_argument("--batch", type=int, default=100, help="QIDs per detail query")
825 + ap.add_argument("--refresh", action="store_true", help="ignore the SPARQL cache")
826 + ap.add_argument("-v", "--verbose", action="store_true")
827 + args = ap.parse_args()
828 + logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
829 + logging.getLogger("httpx").setLevel(logging.WARNING)
830 + DATA_DIR.mkdir(parents=True, exist_ok=True)
831 + sp = Sparql(refresh=args.refresh)
832 + t0 = time.monotonic()
833 + if args.stage in ("candidates", "all"):
834 + stage_candidates(sp)
835 + if args.stage in ("select", "all"):
836 + stage_select(target=args.target)
837 + if args.stage in ("details", "all"):
838 + stage_details(sp, batch=args.batch)
839 + if args.stage in ("assemble", "all"):
840 + stage_assemble()
841 + log.info("done in %.0fs (%d live queries, %d cached)", time.monotonic() - t0, sp.queries, sp.cached)
842 + return 0
843 +
844 +
845 +if __name__ == "__main__":
846 + sys.exit(main())
added src/companyatlas/commands/seed.py +100 −0
Large diff (100 lines) is not loaded by default to keep the page fast.Load diff
added src/companyatlas/registry/industries.py +202 −0
Large diff (202 lines) is not loaded by default to keep the page fast.Load diff
added src/companyatlas/registry/seed.py +367 −0
Large diff (367 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_industry_map.py +115 −0
Large diff (115 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_registry_files.py +95 −0
Large diff (95 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_seed_loader.py +113 −0
Large diff (114 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere