seeds: industries taxonomy, countries, Wikidata/EDGAR harvesters, idempotent catlas seed

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 12 days ago (Sep 12, 2026) parent b8a5c1a

12 changed files +2,972 −0

added docs/SEEDS.md +147 −0

		`@@ -0,0 +1,147 @@`
	1	+# Seeds — the initial company universe
	2	+
	3	+The seed registry is the diversified starting universe of Company Atlas (spec §99: 5,000+ companies across sectors and regions). It is
	4	+plain data committed in `registry/` plus an idempotent loader; the crawl core takes over from `companies.onboarding_status = 'pending'`
	5	+and the `queue_jobs(kind='discover')` rows the loader creates (boundary "Seeds → Crawl" in `docs/ARCHITECTURE.md`).
	6	+
	7	+```
	8	+registry/industries.yaml taxonomy (51 industries, 23 top-level + children, keywords for label mapping)
	9	+registry/countries.csv 250 ISO-3166-1 alpha-2 codes (+ XK), UN M49 region/subregion, centroids
	10	+registry/companies/wikidata-*.ndjson one JSON object per company, one file per UN region
	11	+registry/companies/README.md counts by tier / region / country / industry (regenerated by the harvester)
	12	+scripts/seed_countries.py regenerates countries.csv
	13	+scripts/seed_wikidata.py Wikidata harvester (candidates → select → details → assemble)
	14	+scripts/seed_edgar.py SEC EDGAR enrichment (CIK, ticker, exchange, SIC industry hint)
	15	+src/companyatlas/registry/industries.py taxonomy loader, map_industry(labels) → slugs, map_sic(code)
	16	+src/companyatlas/registry/seed.py loader: seed(), import_companies(), add_company()
	17	+src/companyatlas/commands/seed.py catlas seed \| import-companies \| company-add \| registry-stats
	18	+```
	19	+
	20	+## Sources
	21	+
	22	+\| Source \| What we take \| Politeness \|
	23	+\|---\|---\|---\|
	24	+\| Wikidata (SPARQL, `query.wikidata.org`) \| companies by class (P31), official website P856, country P17→P297, labels/aliases/descriptions, official name P1448, industry P452, inception P571, employees P1128 (+P585), HQ P159 (+P625, P131, P17), coordinates P625, ticker P249 / exchange P414, LEI P1278, CIK P5531, parent P749, logo P154, sitelink count \| UA `CompanyAtlasBot/0.1 (contact@spboucher.ai)`, one query at a time, ≥ 2 s apart, 60 s server timeout, retries with exponential backoff, every result cached in `data/seed/wikidata/sparql/<sha256>.json` (git-ignored) \|
	25	+\| SEC EDGAR (`company_tickers.json`, `data.sec.gov/submissions`) \| CIK, tickers, exchanges, SIC + description, business address \| required UA, ≤ 5 req/s, cached in `data/seed/edgar/` \|
	26	+\| ISO 3166 / UN M49 (lukes/ISO-3166-Countries-with-Regional-Codes) + Wikidata P625 \| country names, regions, centroids \| one-off; the CSV is committed \|
	27	+
	28	+Facts are copied, never inferred: a company only enters the registry with an official website stated on Wikidata; nothing is guessed
	29	+from names. Provenance is kept per company (`source`, `harvested_at`, `sitelinks`, raw `industry_labels`, `notes`) and lands in
	30	+`companies.source_meta`.
	31	+
	32	+## Harvest procedure (`scripts/seed_wikidata.py`)
	33	+
	34	+```bash
	35	+.venv/bin/python scripts/seed_wikidata.py all --target 8000 # ≈ 45–60 min the first time; every stage resumes from cache
	36	+.venv/bin/python scripts/seed_wikidata.py candidates\|select\|details\|assemble # single stage
	37	+.venv/bin/python scripts/seed_edgar.py # optional enrichment, rewrites the NDJSON in place
	38	+.venv/bin/catlas registry-stats # counts without a database
	39	+```
	40	+
	41	+1. candidates — `?item wdt:P31 wd:<class>` for 30 company classes (business, enterprise, public company, company, privately held
	42	+ company, corporation, bank, airline, automobile/aerospace manufacturer, software/technology company, video-game developer/publisher,
	43	+ publisher, record label, telecom, brewery, retail chain, railway, shipyard, …). The subclass tree of business (`P279*`) is far too
	44	+ broad to page, so classes are queried directly and the big ones are chunked by sitelink bands (≥ 80, 50–79, 35–49, 25–34, 18–24,
	45	+ 12–17, 8–11, 5–7); a band that times out is split in two automatically. Minimum 5 sitelinks. Dissolved items (P576) are dropped.
	46	+ Then country boosts (sitelinks ≥ 2, country-first scan, only for minimum-coverage countries still short after the class stage;
	47	+ a timeout is tolerated and reported) and industry boosts (P452 label lists per top-level industry, sitelinks ≥ 2, 600 each).
	48	+ Output `data/seed/candidates.json`.
	49	+2. select — websites normalised to `https://<host>` (paths stripped); generic hosts dropped (social networks, blogs, app stores,
	50	+ site builders, code forges, Wikipedia …; for platform roots such as `google.com`/`apple.com` only the bare/www host counts);
	51	+ duplicates by registrable domain keep the highest-sitelink item (the loser is recorded as a `related_domain_conflict` note on the
	52	+ winner and in `data/seed/dropped.json`). Then the diversified selection below. Output `data/seed/selected.json` (+ a reserve of
	53	+ boost candidates per top-level industry).
	54	+3. details — five light queries per batch of 100 QIDs (labels/description/aliases via the label service; inception/coordinates/
	55	+ LEI/CIK/logo; HQ + parent with end-time qualifiers; legal names + industry labels; tickers/exchanges + employee observations).
	56	+ A `GROUP BY` with many `SAMPLE()` aggregates makes Blazegraph throw `StackOverflowError`, hence plain queries reduced in Python.
	57	+ Output `data/seed/details.json` (incremental).
	58	+4. assemble — industry mapping, importance, tiers, parent/domain conflicts, per-region NDJSON + README.
	59	+
	60	+Gotchas learnt on the endpoint: a query that runs past 60 s may come back as **HTTP 200 with a truncated JSON body followed by a Java
	61	+stack trace** (the gateway even caches it) — the client treats an unparsable body as a timeout; some literals contain raw control
	62	+characters (`json.loads(strict=False)`); starting a query from `P17` of a large country never finishes; `FILTER NOT EXISTS` over tens of
	63	+thousands of bindings is what pushes a scan over the limit (fetch as OPTIONAL and filter client-side instead).
	64	+
	65	+## Selection and diversification rules
	66	+
	67	+Candidates are ordered by sitelinks (desc) and selected until `--target` (8,000) with:
	68	+
	69	+* Caps — United States ≤ 40 % of the target, any other country ≤ 12 %, unknown country ≤ 3 %.
	70	+* Minimum coverage (taken first, best-ranked companies of that country, sitelinks ≥ 2): ≥ 150 for CA, GB, DE, FR, JP, KR, IN, AU;
	71	+ ≥ 60 for BR, MX, AE, SA, ZA, NG, SG, ID, NL, SE, CH, ES, IT, CN, TW, HK — when Wikidata has that many companies with a website.
	72	+* Industry floor — every top-level industry gets ≥ 60 companies where the data allows, topped up from the industry-boost reserve.
	73	+* One company per registrable domain; a subsidiary that shares its parent's domain is dropped (the parent keeps it).
	74	+
	75	+The resulting counts are in `registry/companies/README.md` and `catlas registry-stats`; `tests/test_registry_files.py` enforces the
	76	+caps (with a small tolerance for the industry top-up), the minimums for the large economies and the tier sizes.
	77	+
	78	+## Importance and tiers
	79	+
	80	+```
	81	+s_sitelinks = min(1, ln(1 + sitelinks) / ln(1 + 300)) # 300 sitelinks ≈ the most-linked companies
	82	+s_employees = min(1, log10(1 + employees) / 6) # 1,000,000 employees → 1
	83	+importance = 0.6·s_sitelinks + 0.2·s_employees + 0.1·public_company + 0.1·has_ticker (clamped to 0.02–1)
	84	+tier = rank by importance: 1–150 → 1 (global) · 151–950 → 2 (major) · 951–3450 → 3 (notable) · rest → 4 (long tail)
	85	+```
	86	+
	87	+Importance drives crawl priority only (spec §113); it is refreshed on every `catlas seed`, nothing else about an existing row is.
	88	+
	89	+## NDJSON schema (one object per line)
	90	+
	91	+```
	92	+wikidata_id, display_name, legal_name, aliases[≤8], website (https://host), canonical_domain, country (ISO-2 \| null), hq_city, hq_region,
	93	+lat, lon, industries[] (taxonomy slugs), industry_labels[] (raw P452 labels, + "SIC …" after EDGAR), founded_year, employees,
	94	+public_company, ticker, exchange, lei, sec_cik, parent {wikidata_id, name} \| null, logo_url (Commons Special:FilePath), description,
	95	+sitelinks, importance (0–1), tier (1–4), source ("wikidata"), harvested_at, notes[] (optional: related_domain_conflict),
	96	+source_edgar {cik, enriched_at} (optional)
	97	+```
	98	+
	99	+## Loading (`catlas seed`)
	100	+
	101	+`seed(conn, companies=True, limit=None, files=None)` in `companyatlas.registry.seed`:
	102	+
	103	+1. upserts `industries` (slug, name, parent, description, keywords, sort order) and `countries`;
	104	+2. for each registry line: match an existing company by `wikidata_id`, then by `canonical_domain`;
	105	+ * new → `id = ids.new_id('company')`, slug `ids.slugify(display_name)` de-duplicated as `-<country>` then `-2`, `-3`…,
	106	+ `onboarding_status='pending'`, provenance in `source_meta`;
	107	+ * existing → only null columns are filled; `importance`, `tier` and `source_meta` provenance are refreshed; names, website,
	108	+ history and anything already set are never overwritten;
	109	+3. `company_aliases` (display name → `brand`, legal name → `legal`, ticker → `ticker`, others → `alias`; key `ids.normalize_alias`),
	110	+ `domains` primary row, `company_relationships` PARENT_OF / SUBSIDIARY_OF (confidence 0.8, provenance `{source, property: P749}`)
	111	+ when the parent is also in the registry;
	112	+4. `queue_jobs(kind='discover', key='discover:<company_id>', priority=importance)` for new pending companies (`on conflict do nothing`);
	113	+5. `settings_kv['seed:last_run']` with the counters.
	114	+
	115	+Running it twice yields `companies_new = 0`. `--limit N` seeds the first N lines (files are ordered tier 1 → 4 inside each region);
	116	+`--no-companies` only refreshes the reference tables; `--file` loads specific NDJSON files.
	117	+
	118	+## Adding companies
	119	+
	120	+```bash
	121	+.venv/bin/catlas company-add https://www.example.com --name "Example Corp" --country CA --industry software
	122	+.venv/bin/catlas import-companies my-companies.csv # columns: website (required), display_name, country, industries ("a;b"), aliases, …
	123	+.venv/bin/catlas import-companies my-companies.ndjson # same keys as the registry schema
	124	+```
	125	+
	126	+Imports go through the same upsert (idempotent, same slug/alias/queue rules) with `source_meta.source = "manual"` (`--source` to change).
	127	+To add companies to the committed universe, append lines to a `registry/companies/.ndjson` file (or a new `manual-.ndjson`) with at
	128	+least `website`; run `pytest tests/test_registry_files.py` (validity, no duplicate domain, caps) then `catlas seed`.
	129	+
	130	+## Tests
	131	+
	132	+```bash
	133	+.venv/bin/pytest -q tests/test_industry_map.py tests/test_seed_loader.py tests/test_registry_files.py
	134	+```
	135	+
	136	+`test_seed_loader.py` needs the local database (`catlas migrate`); it seeds a fixture twice under `ztest-` slugs and cleans up after itself.
	137	+
	138	+## Known gaps
	139	+
	140	+See the "Harvest report" section of `registry/companies/README.md` for the run-specific numbers. Structural limits:
	141	+
	142	+* Coverage follows Wikidata: companies without an official website there are absent; small markets (e.g. Nigeria, Saudi Arabia)
	143	+ may fall short of their minimum; South Korea's country boost times out on the endpoint (KR relies on the class stage).
	144	+* Industry slugs come from P452 labels and, failing that, the boost query / description keywords; ~10 % of companies have none
	145	+ (the LLM `classify_industry` job and the crawl fill them later).
	146	+* `hq_region` is the P131 parent of the HQ item and is sometimes the country itself.
	147	+* Websites are forced to `https://`; discovery handles redirects and http-only fallbacks.

added registry/countries.csv +251 −0

		`@@ -0,0 +1,251 @@`
	1	+code,name,region,subregion,lat,lon
	2	+AD,Andorra,Europe,Southern Europe,42.558,1.555
	3	+AE,United Arab Emirates,Asia,Western Asia,23.4,53.8
	4	+AF,Afghanistan,Asia,Southern Asia,33.9,67.7
	5	+AG,Antigua and Barbuda,Americas,Latin America and the Caribbean,17.117,-61.85
	6	+AI,Anguilla,Americas,Latin America and the Caribbean,18.227,-63.049
	7	+AL,Albania,Europe,Southern Europe,41.2,20.2
	8	+AM,Armenia,Asia,Western Asia,40.1,45.0
	9	+AO,Angola,Africa,Sub-Saharan Africa,-11.2,17.9
	10	+AQ,Antarctica,Antarctica,Antarctica,-82.9,135.0
	11	+AR,Argentina,Americas,Latin America and the Caribbean,-38.4,-63.6
	12	+AS,American Samoa,Oceania,Polynesia,-14.296,-170.708
	13	+AT,Austria,Europe,Western Europe,47.5,14.6
	14	+AU,Australia,Oceania,Australia and New Zealand,-25.3,133.8
	15	+AW,Aruba,Americas,Latin America and the Caribbean,12.515,-69.975
	16	+AX,Åland Islands,Europe,Northern Europe,60.25,20.0
	17	+AZ,Azerbaijan,Asia,Western Asia,40.1,47.6
	18	+BA,Bosnia and Herzegovina,Europe,Southern Europe,43.9,17.7
	19	+BB,Barbados,Americas,Latin America and the Caribbean,13.17,-59.553
	20	+BD,Bangladesh,Asia,Southern Asia,23.7,90.4
	21	+BE,Belgium,Europe,Western Europe,50.5,4.5
	22	+BF,Burkina Faso,Africa,Sub-Saharan Africa,12.267,-2.067
	23	+BG,Bulgaria,Europe,Eastern Europe,42.7,25.5
	24	+BH,Bahrain,Asia,Western Asia,26.0,50.6
	25	+BI,Burundi,Africa,Sub-Saharan Africa,-3.667,29.817
	26	+BJ,Benin,Africa,Sub-Saharan Africa,8.833,2.183
	27	+BL,Saint Barthélemy,Americas,Latin America and the Caribbean,17.898,-62.834
	28	+BM,Bermuda,Americas,Northern America,32.32,-64.74
	29	+BN,Brunei,Asia,South-eastern Asia,4.5,114.7
	30	+BO,Bolivia,Americas,Latin America and the Caribbean,-16.3,-63.6
	31	+BQ,Caribbean Netherlands,Americas,Latin America and the Caribbean,12.183,-68.233
	32	+BR,Brazil,Americas,Latin America and the Caribbean,-14.2,-51.9
	33	+BS,Bahamas,Americas,Latin America and the Caribbean,25.0,-77.4
	34	+BT,Bhutan,Asia,Southern Asia,27.5,90.4
	35	+BV,Bouvet Island,Americas,Latin America and the Caribbean,-54.42,3.36
	36	+BW,Botswana,Africa,Sub-Saharan Africa,-22.2,23.7
	37	+BY,Belarus,Europe,Eastern Europe,53.7,27.95
	38	+BZ,Belize,Americas,Latin America and the Caribbean,17.067,-88.7
	39	+CA,Canada,Americas,Northern America,56.1,-106.3
	40	+CC,Cocos,Oceania,Australia and New Zealand,-12.117,96.895
	41	+CD,DR Congo,Africa,Sub-Saharan Africa,-4.0,21.8
	42	+CF,Central African Republic,Africa,Sub-Saharan Africa,6.7,20.9
	43	+CG,Republic of the Congo,Africa,Sub-Saharan Africa,-0.75,15.383
	44	+CH,Switzerland,Europe,Western Europe,46.8,8.2
	45	+CI,Côte d'Ivoire,Africa,Sub-Saharan Africa,8.0,-6.0
	46	+CK,Cook Islands,Oceania,Polynesia,-21.233,-159.783
	47	+CL,Chile,Americas,Latin America and the Caribbean,-35.7,-71.5
	48	+CM,Cameroon,Africa,Sub-Saharan Africa,5.133,12.65
	49	+CN,China,Asia,Eastern Asia,35.9,104.2
	50	+CO,Colombia,Americas,Latin America and the Caribbean,4.6,-74.3
	51	+CR,Costa Rica,Americas,Latin America and the Caribbean,9.7,-83.8
	52	+CU,Cuba,Americas,Latin America and the Caribbean,21.5,-77.8
	53	+CV,Cabo Verde,Africa,Sub-Saharan Africa,16.0,-24.0
	54	+CW,Curaçao,Americas,Latin America and the Caribbean,12.2,-69.0
	55	+CX,Christmas Island,Oceania,Australia and New Zealand,-10.49,105.627
	56	+CY,Cyprus,Asia,Western Asia,35.1,33.4
	57	+CZ,Czechia,Europe,Eastern Europe,49.8,15.5
	58	+DE,Germany,Europe,Western Europe,51.2,10.4
	59	+DJ,Djibouti,Africa,Sub-Saharan Africa,11.8,42.433
	60	+DK,Denmark,Europe,Northern Europe,56.0,10.0
	61	+DM,Dominica,Americas,Latin America and the Caribbean,15.417,-61.333
	62	+DO,Dominican Republic,Americas,Latin America and the Caribbean,18.8,-70.2
	63	+DZ,Algeria,Africa,Northern Africa,28.0,1.7
	64	+EC,Ecuador,Americas,Latin America and the Caribbean,-1.8,-78.2
	65	+EE,Estonia,Europe,Northern Europe,58.6,25.0
	66	+EG,Egypt,Africa,Northern Africa,26.8,30.8
	67	+EH,Western Sahara,Africa,Northern Africa,25.0,-13.0
	68	+ER,Eritrea,Africa,Sub-Saharan Africa,15.483,38.25
	69	+ES,Spain,Europe,Southern Europe,40.5,-3.7
	70	+ET,Ethiopia,Africa,Sub-Saharan Africa,9.1,40.5
	71	+FI,Finland,Europe,Northern Europe,61.9,25.7
	72	+FJ,Fiji,Oceania,Melanesia,-17.7,178.1
	73	+FK,Falkland Islands,Americas,Latin America and the Caribbean,-51.73,-59.22
	74	+FM,Micronesia,Oceania,Micronesia,7.4,150.6
	75	+FO,Faroe Islands,Europe,Northern Europe,61.97,-6.844
	76	+FR,France,Europe,Western Europe,46.6,2.2
	77	+GA,Gabon,Africa,Sub-Saharan Africa,-0.683,11.5
	78	+GB,United Kingdom,Europe,Northern Europe,54.0,-2.5
	79	+GD,Grenada,Americas,Latin America and the Caribbean,12.1,-61.7
	80	+GE,Georgia,Asia,Western Asia,42.3,43.4
	81	+GF,French Guiana,Americas,Latin America and the Caribbean,4.0,-53.0
	82	+GG,Guernsey,Europe,Northern Europe,49.45,-2.583
	83	+GH,Ghana,Africa,Sub-Saharan Africa,7.947,-1.023
	84	+GI,Gibraltar,Europe,Southern Europe,36.14,-5.35
	85	+GL,Greenland,Americas,Northern America,71.7,-42.6
	86	+GM,Gambia,Africa,Sub-Saharan Africa,13.5,-15.5
	87	+GN,Guinea,Africa,Sub-Saharan Africa,10.0,-11.0
	88	+GP,Guadeloupe,Americas,Latin America and the Caribbean,16.25,-61.5
	89	+GQ,Equatorial Guinea,Africa,Sub-Saharan Africa,1.5,10.0
	90	+GR,Greece,Europe,Southern Europe,39.1,21.8
	91	+GS,South Georgia and the South Sandwich Islands,Americas,Latin America and the Caribbean,-54.25,-36.75
	92	+GT,Guatemala,Americas,Latin America and the Caribbean,15.8,-90.2
	93	+GU,Guam,Oceania,Micronesia,13.5,144.8
	94	+GW,Guinea-Bissau,Africa,Sub-Saharan Africa,12.0,-15.0
	95	+GY,Guyana,Americas,Latin America and the Caribbean,5.733,-59.317
	96	+HK,Hong Kong,Asia,Eastern Asia,22.3,114.2
	97	+HM,Heard Island and McDonald Islands,Oceania,Australia and New Zealand,-53.078,73.507
	98	+HN,Honduras,Americas,Latin America and the Caribbean,15.2,-86.2
	99	+HR,Croatia,Europe,Southern Europe,45.1,15.2
	100	+HT,Haiti,Americas,Latin America and the Caribbean,19.0,-72.8
	101	+HU,Hungary,Europe,Eastern Europe,47.2,19.5
	102	+ID,Indonesia,Asia,South-eastern Asia,-2.5,118.0
	103	+IE,Ireland,Europe,Northern Europe,53.4,-8.2
	104	+IL,Israel,Asia,Western Asia,31.0,34.9
	105	+IM,Isle of Man,Europe,Northern Europe,54.235,-4.525
	106	+IN,India,Asia,Southern Asia,20.6,79.0
	107	+IO,British Indian Ocean Territory,Africa,Sub-Saharan Africa,-6.0,71.5
	108	+IQ,Iraq,Asia,Western Asia,33.2,43.7
	109	+IR,Iran,Asia,Southern Asia,32.4,53.7
	110	+IS,Iceland,Europe,Northern Europe,64.96,-19.0
	111	+IT,Italy,Europe,Southern Europe,41.9,12.6
	112	+JE,Jersey,Europe,Northern Europe,49.19,-2.11
	113	+JM,Jamaica,Americas,Latin America and the Caribbean,18.18,-77.4
	114	+JO,Jordan,Asia,Western Asia,30.6,36.2
	115	+JP,Japan,Asia,Eastern Asia,36.2,138.3
	116	+KE,Kenya,Africa,Sub-Saharan Africa,-0.02,37.9
	117	+KG,Kyrgyzstan,Asia,Central Asia,41.2,74.8
	118	+KH,Cambodia,Asia,South-eastern Asia,12.6,105.0
	119	+KI,Kiribati,Oceania,Micronesia,-3.4,-168.7
	120	+KM,Comoros,Africa,Sub-Saharan Africa,-12.3,43.7
	121	+KN,Saint Kitts and Nevis,Americas,Latin America and the Caribbean,17.272,-62.667
	122	+KP,North Korea,Asia,Eastern Asia,40.3,127.5
	123	+KR,South Korea,Asia,Eastern Asia,35.9,127.8
	124	+KW,Kuwait,Asia,Western Asia,29.3,47.5
	125	+KY,Cayman Islands,Americas,Latin America and the Caribbean,19.5,-80.5
	126	+KZ,Kazakhstan,Asia,Central Asia,48.0,66.9
	127	+LA,Laos,Asia,South-eastern Asia,19.9,102.5
	128	+LB,Lebanon,Asia,Western Asia,33.9,35.9
	129	+LC,Saint Lucia,Americas,Latin America and the Caribbean,13.883,-60.967
	130	+LI,Liechtenstein,Europe,Western Europe,47.145,9.554
	131	+LK,Sri Lanka,Asia,Southern Asia,7.9,80.8
	132	+LR,Liberia,Africa,Sub-Saharan Africa,6.533,-9.75
	133	+LS,Lesotho,Africa,Sub-Saharan Africa,-29.55,28.25
	134	+LT,Lithuania,Europe,Northern Europe,55.2,23.9
	135	+LU,Luxembourg,Europe,Western Europe,49.8,6.1
	136	+LV,Latvia,Europe,Northern Europe,56.9,24.6
	137	+LY,Libya,Africa,Northern Africa,26.3,17.2
	138	+MA,Morocco,Africa,Northern Africa,31.8,-7.1
	139	+MC,Monaco,Europe,Western Europe,43.731,7.42
	140	+MD,Moldova,Europe,Eastern Europe,47.4,28.4
	141	+ME,Montenegro,Europe,Southern Europe,42.7,19.4
	142	+MF,Saint Martin,Americas,Latin America and the Caribbean,18.075,-63.06
	143	+MG,Madagascar,Africa,Sub-Saharan Africa,-18.8,46.9
	144	+MH,Marshall Islands,Oceania,Micronesia,9.82,169.29
	145	+MK,North Macedonia,Europe,Southern Europe,41.6,21.7
	146	+ML,Mali,Africa,Sub-Saharan Africa,17.6,-4.0
	147	+MM,Myanmar,Asia,South-eastern Asia,21.9,95.96
	148	+MN,Mongolia,Asia,Eastern Asia,46.9,103.8
	149	+MO,Macao,Asia,Eastern Asia,22.2,113.55
	150	+MP,Northern Mariana Islands,Oceania,Micronesia,16.705,145.78
	151	+MQ,Martinique,Americas,Latin America and the Caribbean,14.65,-61.015
	152	+MR,Mauritania,Africa,Sub-Saharan Africa,21.0,-10.9
	153	+MS,Montserrat,Americas,Latin America and the Caribbean,16.75,-62.2
	154	+MT,Malta,Europe,Southern Europe,35.9,14.4
	155	+MU,Mauritius,Africa,Sub-Saharan Africa,-20.3,57.6
	156	+MV,Maldives,Asia,Southern Asia,3.2,73.2
	157	+MW,Malawi,Africa,Sub-Saharan Africa,-13.0,34.0
	158	+MX,Mexico,Americas,Latin America and the Caribbean,23.6,-102.6
	159	+MY,Malaysia,Asia,South-eastern Asia,4.2,108.0
	160	+MZ,Mozambique,Africa,Sub-Saharan Africa,-18.7,35.5
	161	+NA,Namibia,Africa,Sub-Saharan Africa,-23.0,17.0
	162	+NC,New Caledonia,Oceania,Melanesia,-21.25,165.3
	163	+NE,Niger,Africa,Sub-Saharan Africa,17.6,8.1
	164	+NF,Norfolk Island,Oceania,Australia and New Zealand,-29.033,167.95
	165	+NG,Nigeria,Africa,Sub-Saharan Africa,9.1,8.7
	166	+NI,Nicaragua,Americas,Latin America and the Caribbean,12.9,-85.2
	167	+NL,Netherlands,Europe,Western Europe,52.2,5.3
	168	+NO,Norway,Europe,Northern Europe,64.6,12.7
	169	+NP,Nepal,Asia,Southern Asia,28.4,84.1
	170	+NR,Nauru,Oceania,Micronesia,-0.527,166.935
	171	+NU,Niue,Oceania,Polynesia,-19.05,-169.917
	172	+NZ,New Zealand,Oceania,Australia and New Zealand,-41.5,172.8
	173	+OM,Oman,Asia,Western Asia,21.5,55.9
	174	+PA,Panama,Americas,Latin America and the Caribbean,8.5,-80.8
	175	+PE,Peru,Americas,Latin America and the Caribbean,-9.2,-75.0
	176	+PF,French Polynesia,Oceania,Polynesia,-17.7,-149.4
	177	+PG,Papua New Guinea,Oceania,Melanesia,-6.3,143.96
	178	+PH,Philippines,Asia,South-eastern Asia,12.9,121.8
	179	+PK,Pakistan,Asia,Southern Asia,30.4,69.3
	180	+PL,Poland,Europe,Eastern Europe,51.9,19.1
	181	+PM,Saint Pierre and Miquelon,Americas,Northern America,46.85,-56.3
	182	+PN,Pitcairn,Oceania,Polynesia,-25.068,-130.105
	183	+PR,Puerto Rico,Americas,Latin America and the Caribbean,18.25,-66.5
	184	+PS,Palestine,Asia,Western Asia,31.9,35.2
	185	+PT,Portugal,Europe,Southern Europe,39.4,-8.2
	186	+PW,Palau,Oceania,Micronesia,7.467,134.55
	187	+PY,Paraguay,Americas,Latin America and the Caribbean,-23.4,-58.4
	188	+QA,Qatar,Asia,Western Asia,25.4,51.2
	189	+RE,Réunion,Africa,Sub-Saharan Africa,-21.114,55.532
	190	+RO,Romania,Europe,Eastern Europe,45.9,25.0
	191	+RS,Serbia,Europe,Southern Europe,44.0,21.0
	192	+RU,Russia,Europe,Eastern Europe,61.5,105.3
	193	+RW,Rwanda,Africa,Sub-Saharan Africa,-2.0,30.0
	194	+SA,Saudi Arabia,Asia,Western Asia,23.9,45.1
	195	+SB,Solomon Islands,Oceania,Melanesia,-9.6,160.2
	196	+SC,Seychelles,Africa,Sub-Saharan Africa,-4.7,55.5
	197	+SD,Sudan,Africa,Northern Africa,12.9,30.2
	198	+SE,Sweden,Europe,Northern Europe,60.1,18.6
	199	+SG,Singapore,Asia,South-eastern Asia,1.35,103.8
	200	+SH,Saint Helena,Africa,Sub-Saharan Africa,-15.925,-5.718
	201	+SI,Slovenia,Europe,Southern Europe,46.2,15.0
	202	+SJ,Svalbard and Jan Mayen,Europe,Northern Europe,77.6,23.7
	203	+SK,Slovakia,Europe,Eastern Europe,48.7,19.7
	204	+SL,Sierra Leone,Africa,Sub-Saharan Africa,8.5,-12.1
	205	+SM,San Marino,Europe,Southern Europe,43.933,12.467
	206	+SN,Senegal,Africa,Sub-Saharan Africa,14.367,-14.283
	207	+SO,Somalia,Africa,Sub-Saharan Africa,6.0,47.0
	208	+SR,Suriname,Americas,Latin America and the Caribbean,4.0,-56.0
	209	+SS,South Sudan,Africa,Sub-Saharan Africa,7.0,30.0
	210	+ST,Sao Tome and Principe,Africa,Sub-Saharan Africa,0.317,6.6
	211	+SV,El Salvador,Americas,Latin America and the Caribbean,13.669,-88.866
	212	+SX,Sint Maarten,Americas,Latin America and the Caribbean,18.032,-63.068
	213	+SY,Syria,Asia,Western Asia,34.8,39.0
	214	+SZ,Eswatini,Africa,Sub-Saharan Africa,-26.483,31.433
	215	+TC,Turks and Caicos Islands,Americas,Latin America and the Caribbean,21.78,-71.8
	216	+TD,Chad,Africa,Sub-Saharan Africa,15.5,18.7
	217	+TF,French Southern Territories,Africa,Sub-Saharan Africa,-49.3,69.3
	218	+TG,Togo,Africa,Sub-Saharan Africa,8.25,1.183
	219	+TH,Thailand,Asia,South-eastern Asia,15.9,100.9
	220	+TJ,Tajikistan,Asia,Central Asia,38.9,71.3
	221	+TK,Tokelau,Oceania,Polynesia,-9.167,-171.833
	222	+TL,Timor-Leste,Asia,South-eastern Asia,-8.9,125.7
	223	+TM,Turkmenistan,Asia,Central Asia,38.97,59.6
	224	+TN,Tunisia,Africa,Northern Africa,34.0,10.0
	225	+TO,Tonga,Oceania,Polynesia,-21.2,-175.2
	226	+TR,Türkiye,Asia,Western Asia,39.0,35.2
	227	+TT,Trinidad and Tobago,Americas,Latin America and the Caribbean,10.667,-61.517
	228	+TV,Tuvalu,Oceania,Polynesia,-7.475,178.006
	229	+TW,Taiwan,Asia,Eastern Asia,23.7,121.0
	230	+TZ,Tanzania,Africa,Sub-Saharan Africa,-6.4,34.9
	231	+UA,Ukraine,Europe,Eastern Europe,48.4,31.2
	232	+UG,Uganda,Africa,Sub-Saharan Africa,1.28,32.39
	233	+UM,U.S. Minor Outlying Islands,Oceania,Micronesia,19.3,166.6
	234	+US,United States,Americas,Northern America,39.8,-98.6
	235	+UY,Uruguay,Americas,Latin America and the Caribbean,-32.5,-55.8
	236	+UZ,Uzbekistan,Asia,Central Asia,41.4,64.6
	237	+VA,Vatican City,Europe,Southern Europe,41.904,12.453
	238	+VC,Saint Vincent and the Grenadines,Americas,Latin America and the Caribbean,13.014,-61.23
	239	+VE,Venezuela,Americas,Latin America and the Caribbean,6.4,-66.6
	240	+VG,British Virgin Islands,Americas,Latin America and the Caribbean,18.445,-64.54
	241	+VI,U.S. Virgin Islands,Americas,Latin America and the Caribbean,18.333,-64.833
	242	+VN,Vietnam,Asia,South-eastern Asia,14.1,108.3
	243	+VU,Vanuatu,Oceania,Melanesia,-15.4,166.96
	244	+WF,Wallis and Futuna,Oceania,Polynesia,-14.302,-178.109
	245	+WS,Samoa,Oceania,Polynesia,-13.8,-172.1
	246	+XK,Kosovo,Europe,Southern Europe,42.6,20.9
	247	+YE,Yemen,Asia,Western Asia,15.6,48.5
	248	+YT,Mayotte,Africa,Sub-Saharan Africa,-12.843,45.138
	249	+ZA,South Africa,Africa,Sub-Saharan Africa,-30.6,22.9
	250	+ZM,Zambia,Africa,Sub-Saharan Africa,-14.0,28.0
	251	+ZW,Zimbabwe,Africa,Sub-Saharan Africa,-19.0,30.0

added registry/industries.yaml +429 −0

		`@@ -0,0 +1,429 @@`
	1	+# Company Atlas industry taxonomy (spec §6, §99). Two levels: top-level sectors and their children (`parent`).
	2	+# `keywords` map Wikidata industry labels (P452), SIC descriptions and free text to slugs — see companyatlas.registry.industries.map_industry.
	3	+# Matching is case-insensitive; a keyword matches as a whole phrase inside the label (word boundaries). Longer keywords win.
	4	+# Order = sort_order in the database (children follow their parent).
	5	+industries:
	6	+ # ------------------------------------------------------------------------------------------------ technology
	7	+ - slug: technology
	8	+ name: Technology
	9	+ description: Information technology, hardware, electronics and computing platforms.
	10	+ keywords: [technology, information technology, tech, computer hardware, hardware, electronics, consumer electronics, electronics industry,
	11	+ computer industry, computing, computer, information and communications technology, ict, high tech, technology company,
	12	+ electronic engineering, electrical equipment, computer peripheral, computer storage, data storage, optics, photonics,
	13	+ information technology consulting, it services, it service, information service, digital]
	14	+ - slug: software
	15	+ name: Software
	16	+ parent: technology
	17	+ description: Software products, SaaS, enterprise applications and developer tools.
	18	+ keywords: [software, software industry, software development, software engineering, saas, software as a service, enterprise software,
	19	+ application software, operating system, computer software, software publisher, developer tools, database, video game engine,
	20	+ open source, open-source software, cloud software, business software, erp, crm, productivity software]
	21	+ - slug: artificial-intelligence
	22	+ name: Artificial Intelligence
	23	+ parent: technology
	24	+ description: Machine learning, AI research, foundation models and AI applications.
	25	+ keywords: [artificial intelligence, machine learning, deep learning, ai, natural language processing, computer vision, generative artificial
	26	+ intelligence, large language model, ai research, neural network, robotics process automation, data science, speech recognition]
	27	+ - slug: semiconductors
	28	+ name: Semiconductors
	29	+ parent: technology
	30	+ description: Chip design, foundries, semiconductor equipment and materials.
	31	+ keywords: [semiconductor, semiconductors, semiconductor industry, integrated circuit, microprocessor, chip, chipmaker, fabless,
	32	+ semiconductor device fabrication, semiconductor equipment, memory chip, microelectronics, electronic component, sensor]
	33	+ - slug: cloud-infrastructure
	34	+ name: Cloud & Infrastructure
	35	+ parent: technology
	36	+ description: Cloud computing, hosting, data centres, networking and infrastructure software.
	37	+ keywords: [cloud computing, cloud, web hosting, hosting, data center, data centre, infrastructure as a service, platform as a service,
	38	+ content delivery network, cdn, networking hardware, computer network, network equipment, server, virtualization, devops, colocation]
	39	+ - slug: cybersecurity
	40	+ name: Cybersecurity
	41	+ parent: technology
	42	+ description: Information security products and services.
	43	+ keywords: [cybersecurity, cyber security, computer security, information security, network security, antivirus, security software,
	44	+ identity management, encryption, it security, cyber defense]
	45	+ - slug: internet
	46	+ name: Internet
	47	+ parent: technology
	48	+ description: Consumer internet, search, social media, online platforms and marketplaces.
	49	+ keywords: [internet, internet industry, world wide web, web, social media, social network, social networking service, search engine,
	50	+ online service, online platform, web portal, online community, streaming service, internet service provider, dot-com, online,
	51	+ web search, digital media, mobile app, mobile application, online dating, online advertising, advertising technology, adtech,
	52	+ website, online video platform, sharing economy, internet company]
	53	+ - slug: e-commerce
	54	+ name: E-commerce
	55	+ parent: technology
	56	+ description: Online retail and digital marketplaces.
	57	+ keywords: [e-commerce, ecommerce, electronic commerce, online retail, online retailer, online shopping, online marketplace, marketplace,
	58	+ online store, internet retail, mail order, direct-to-consumer, dtc, online auction]
	59	+ # ------------------------------------------------------------------------------------------------ finance
	60	+ - slug: financial-services
	61	+ name: Financial Services
	62	+ description: Banks, insurers, asset managers, exchanges and other financial institutions.
	63	+ keywords: [financial services, finance, financial, financial industry, financial sector, financial institution, financial technology holding,
	64	+ stock exchange, securities, brokerage, broker, investment, investment banking, credit, lending, loan, credit card, credit union,
	65	+ trust company, financial holding, capital markets, exchange, trading, securities trading, consumer finance, microfinance,
	66	+ leasing, factoring, wealth, financial market, fund, mortgage]
	67	+ - slug: fintech
	68	+ name: Fintech
	69	+ parent: financial-services
	70	+ description: Technology-driven financial services, neobanks, crypto and trading apps.
	71	+ keywords: [fintech, financial technology, neobank, cryptocurrency, crypto, blockchain, digital bank, online bank, digital payments platform,
	72	+ robo-advisor, crowdfunding, peer-to-peer lending, buy now pay later, bnpl, trading platform, online brokerage, digital currency,
	73	+ cryptocurrency exchange, decentralized finance, defi]
	74	+ - slug: payments
	75	+ name: Payments
	76	+ parent: financial-services
	77	+ description: Payment networks, processors, wallets and money transfer.
	78	+ keywords: [payment, payments, payment system, payment processing, payment processor, payment service provider, mobile payment,
	79	+ digital wallet, e-wallet, money transfer, remittance, payment card, card network, merchant acquiring, point of sale, pos, acquirer]
	80	+ - slug: banking
	81	+ name: Banking
	82	+ parent: financial-services
	83	+ description: Commercial, retail, private and investment banks.
	84	+ keywords: [banking, bank, commercial bank, retail banking, investment bank, private banking, savings bank, cooperative bank, central bank,
	85	+ universal bank, banking industry, merchant bank, building society, development bank, bank holding company, mortgage bank,
	86	+ online banking, islamic banking, postal savings]
	87	+ - slug: insurance
	88	+ name: Insurance
	89	+ parent: financial-services
	90	+ description: Life, health, property and casualty insurance and reinsurance.
	91	+ keywords: [insurance, insurer, reinsurance, reinsurer, life insurance, health insurance, property insurance, casualty insurance,
	92	+ insurance industry, insurance company, mutual insurance, assurance, underwriting, annuity, pension insurance, insurance broker]
	93	+ - slug: asset-management
	94	+ name: Asset Management
	95	+ parent: financial-services
	96	+ description: Investment management, private equity, venture capital, hedge funds and pensions.
	97	+ keywords: [asset management, investment management, fund management, private equity, venture capital, hedge fund, wealth management,
	98	+ pension fund, sovereign wealth fund, mutual fund, investment fund, investment company, exchange-traded fund, etf, investment firm,
	99	+ investment trust, financial advisory, holding company, family office, real estate investment trust, reit, alternative investment,
	100	+ private investment, investor]
	101	+ # ------------------------------------------------------------------------------------------------ real estate & construction
	102	+ - slug: real-estate
	103	+ name: Real Estate
	104	+ description: Property development, ownership, brokerage and services.
	105	+ keywords: [real estate, real estate industry, real estate development, property development, property developer, property management,
	106	+ real estate investment, real estate company, commercial real estate, residential real estate, housing, property, realty,
	107	+ real estate brokerage, proptech, coworking, self storage, shopping center, shopping mall, land development, home builder,
	108	+ homebuilding, property investment, estate agent]
	109	+ - slug: construction
	110	+ name: Construction
	111	+ description: Contractors, engineering & construction and building products.
	112	+ keywords: [construction, construction industry, civil engineering, building construction, general contractor, engineering and construction,
	113	+ infrastructure construction, building materials, building products, cement, concrete, construction materials, contractor,
	114	+ architecture, architectural firm, engineering firm, engineering, structural engineering, construction equipment, homebuilder,
	115	+ plumbing, hvac, roofing, building, real estate construction]
	116	+ # ------------------------------------------------------------------------------------------------ consumer
	117	+ - slug: retail
	118	+ name: Retail
	119	+ description: Store-based retailers, supermarkets, department stores and wholesale.
	120	+ keywords: [retail, retailing, retailer, retail industry, retail trade, supermarket, grocery store, grocery, hypermarket, department store,
	121	+ discount store, convenience store, chain store, retail chain, wholesale, wholesaling, distribution, distributor, drugstore,
	122	+ pharmacy chain, pharmacy, home improvement, hardware store, bookstore, bookseller, toy store, furniture store, warehouse club,
	123	+ sporting goods store, electronics retailer, variety store, duty-free, cash and carry, consumer goods distribution]
	124	+ - slug: consumer-goods
	125	+ name: Consumer Goods
	126	+ description: Household, personal care, cosmetics, toys, appliances and other consumer products.
	127	+ keywords: [consumer goods, consumer products, fast-moving consumer goods, fmcg, personal care, cosmetics, cosmetics industry, beauty,
	128	+ household goods, household products, home appliance, appliances, toys, toy industry, toy, furniture, furniture industry,
	129	+ luxury goods, luxury, jewelry, jewellery, watchmaking, watch, eyewear, sporting goods, sports equipment, tobacco, tobacco industry,
	130	+ consumer durables, cleaning products, hygiene, fragrance, perfume, stationery, bicycle, bicycle industry, pet food, pet products,
	131	+ kitchenware, tableware, home furnishings, mattress, musical instrument, consumer electronics manufacturer, camera]
	132	+ - slug: food-beverage
	133	+ name: Food & Beverage
	134	+ parent: consumer-goods
	135	+ description: Food processing, beverages, brewing, distilling and restaurants.
	136	+ keywords: [food, food industry, food processing, food manufacturing, beverage, beverages, beverage industry, drink, brewery, brewing, beer,
	137	+ distillery, distilling, spirits, wine, winery, winemaking, soft drink, dairy, dairy industry, meat, meat processing, confectionery,
	138	+ chocolate, snack, bakery, coffee, tea, restaurant, restaurant chain, fast food, foodservice, food service, quick service restaurant,
	139	+ catering, bottling, sugar, seafood, food retail, agri-food, agrifood, food and drink, nutrition, infant formula, frozen food,
	140	+ packaged food, condiment, cereal]
	141	+ - slug: apparel
	142	+ name: Apparel & Fashion
	143	+ parent: consumer-goods
	144	+ description: Clothing, footwear, fashion houses and textiles.
	145	+ keywords: [apparel, clothing, clothing industry, fashion, fashion industry, fashion house, footwear, shoe, shoes, textile, textile industry,
	146	+ textiles, garment, sportswear, luxury fashion, haute couture, denim, lingerie, accessories, leather goods, fashion design,
	147	+ fashion brand, clothing brand, ready-to-wear, outdoor clothing, uniform]
	148	+ # ------------------------------------------------------------------------------------------------ energy & resources
	149	+ - slug: energy
	150	+ name: Energy
	151	+ description: Energy production, trading and services.
	152	+ keywords: [energy, energy industry, energy sector, power generation, electricity generation, electric power, energy company, power,
	153	+ power plant, nuclear power, nuclear energy, nuclear, energy trading, energy services, coal, coal mining, uranium, lng,
	154	+ liquefied natural gas, district heating, heating, fuel, biofuel, hydrogen, energy storage, battery, batteries, battery manufacturer,
	155	+ power engineering, oilfield services]
	156	+ - slug: oil-gas
	157	+ name: Oil & Gas
	158	+ parent: energy
	159	+ description: Exploration, production, refining, pipelines and fuel distribution.
	160	+ keywords: [oil, oil and gas, oil and gas industry, petroleum, petroleum industry, oil industry, natural gas, gas, oil refining, refining,
	161	+ refinery, oil exploration, exploration and production, upstream, downstream, midstream, pipeline, pipeline transport, oil company,
	162	+ petrochemical, gas station, filling station, fuel retail, oilfield, drilling, offshore drilling, gas distribution, gas utility]
	163	+ - slug: renewables
	164	+ name: Renewable Energy
	165	+ parent: energy
	166	+ description: Solar, wind, hydro, geothermal and clean-energy technology.
	167	+ keywords: [renewable energy, renewables, solar, solar energy, solar power, photovoltaic, photovoltaics, wind, wind power, wind energy,
	168	+ wind turbine, hydroelectric, hydroelectricity, hydropower, geothermal, geothermal energy, clean energy, cleantech, clean technology,
	169	+ green energy, bioenergy, biomass, tidal power, energy transition, electric vehicle charging, ev charging, solar panel, fuel cell]
	170	+ - slug: utilities
	171	+ name: Utilities
	172	+ description: Electricity, gas and water utilities and grid operators.
	173	+ keywords: [utility, utilities, public utility, electric utility, electricity, electricity distribution, electricity transmission, power grid,
	174	+ grid operator, transmission system operator, distribution system operator, water utility, water supply, water, wastewater,
	175	+ sewerage, waste management, waste, recycling, waste collection, environmental services, sanitation, multi-utility, water industry,
	176	+ electricity retailer, energy supplier, electricity supplier, gas supplier, natural gas distribution, public service]
	177	+ - slug: mining
	178	+ name: Mining & Metals
	179	+ description: Mining, metals production and mineral extraction.
	180	+ keywords: [mining, mining industry, mine, metal, metals, metallurgy, steel, steelmaking, steel industry, iron ore, iron, aluminium, aluminum,
	181	+ copper, gold, gold mining, silver, nickel, zinc, lithium, rare earth, precious metals, mineral, minerals, mineral exploration,
	182	+ quarrying, quarry, coal, diamond, diamond mining, potash, phosphate, metal production, smelting, steel producer, iron and steel,
	183	+ non-ferrous metals, ferroalloy, mining company, extraction]
	184	+ - slug: chemicals
	185	+ name: Chemicals
	186	+ description: Basic, specialty and agricultural chemicals, plastics and coatings.
	187	+ keywords: [chemical, chemicals, chemical industry, chemistry, specialty chemicals, speciality chemicals, petrochemicals, plastics, plastic,
	188	+ polymer, polymers, coatings, paint, paints, fertilizer, fertiliser, agrochemical, agrochemicals, pesticide, industrial gases,
	189	+ industrial gas, adhesives, rubber, synthetic fiber, fibers, resin, explosives, chemical manufacturer, chemical engineering,
	190	+ cosmetics ingredients, flavors and fragrances, flavours, catalysts, pharmaceutical chemicals, fine chemicals]
	191	+ - slug: materials
	192	+ name: Materials
	193	+ description: Glass, paper, packaging, ceramics, wood and other basic materials.
	194	+ keywords: [materials, materials science, glass, glass industry, glassmaking, paper, paper industry, pulp and paper, pulp, packaging,
	195	+ packaging industry, container, containers, ceramics, ceramic, wood, wood industry, timber, lumber, forestry, forest products,
	196	+ forest industry, sawmill, cardboard, corrugated, building material, insulation, composite materials, composites, carbon fiber,
	197	+ abrasives, refractory, tiles, stone, gravel, asphalt, textile fiber, nonwovens, basic materials, raw materials]
	198	+ # ------------------------------------------------------------------------------------------------ industrials
	199	+ - slug: manufacturing
	200	+ name: Manufacturing
	201	+ description: Diversified and general manufacturing, industrial conglomerates.
	202	+ keywords: [manufacturing, manufacturer, manufacturing industry, industrial, industry, industrial conglomerate, conglomerate, engineering,
	203	+ industrial engineering, fabrication, metal fabrication, metalworking, industrial products, industrial goods, diversified industrials,
	204	+ electrical engineering, electrical industry, electrical equipment manufacturer, electric motor, lighting, lighting industry,
	205	+ tools, hand tools, power tools, fasteners, bearings, valves, pumps, pump, compressor, industrial automation, automation,
	206	+ contract manufacturing, electronics manufacturing services, ems, precision engineering, 3d printing, additive manufacturing,
	207	+ printing, printing industry, packaging machinery, textile machinery, industrial supplies, heavy industry,
	208	+ shipbuilding, shipyard, boatbuilding]
	209	+ - slug: industrial-machinery
	210	+ name: Industrial Machinery
	211	+ parent: manufacturing
	212	+ description: Machine tools, heavy equipment, agricultural and construction machinery.
	213	+ keywords: [machinery, industrial machinery, machine tool, machine tools, machine, machines, heavy equipment, construction equipment,
	214	+ construction machinery, agricultural machinery, farm equipment, mechanical engineering, mining equipment, engines, engine,
	215	+ engine manufacturer, turbine, turbines, elevator, elevators, escalator, crane, cranes, forklift, material handling, hydraulics,
	216	+ pneumatics, packaging equipment, printing press, sewing machine, machine building, plant engineering, process equipment,
	217	+ industrial equipment, equipment manufacturer, diesel engine, gas turbine, tractor, tractors, agricultural equipment, robots]
	218	+ - slug: robotics
	219	+ name: Robotics
	220	+ parent: manufacturing
	221	+ description: Industrial and service robots, drones and autonomous systems.
	222	+ keywords: [robotics, robot, robots, industrial robot, industrial robots, service robot, autonomous vehicle, autonomous systems,
	223	+ unmanned aerial vehicle, drone, drones, automation technology, mechatronics, humanoid robot, autonomous robot, robotic surgery,
	224	+ warehouse automation, robotic process, cobot, exoskeleton, self-driving]
	225	+ - slug: automotive
	226	+ name: Automotive
	227	+ description: Vehicle manufacturers, suppliers, dealers and mobility.
	228	+ keywords: [automotive, automotive industry, automobile, automobiles, automobile manufacturer, car, cars, car manufacturer, motor vehicle,
	229	+ motor vehicles, vehicle, vehicles, vehicle manufacturer, auto parts, automotive parts, automotive supplier, auto industry, truck,
	230	+ trucks, truck manufacturer, bus manufacturer, bus, motorcycle, motorcycles, motorcycle manufacturer, electric vehicle,
	231	+ electric vehicles, ev, car dealership, car dealer, car rental, tire, tyre, tires, tyres, automotive engineering, coachbuilder,
	232	+ commercial vehicle, commercial vehicles, scooter, automaker, mobility, ride-hailing, ridesharing, car sharing, carsharing,
	233	+ automotive technology, racing, motorsport, caravan, recreational vehicle]
	234	+ - slug: aerospace-defense
	235	+ name: Aerospace & Defense
	236	+ description: Aircraft, space systems, defence contractors and related suppliers.
	237	+ keywords: [aerospace, aerospace industry, aerospace manufacturer, aircraft, aircraft manufacturer, aviation industry, aviation, space,
	238	+ space industry, spaceflight, spacecraft, satellite, satellites, satellite manufacturer, launch vehicle, rocket, rockets, aeronautics,
	239	+ aeronautical, avionics, helicopter, helicopters, jet engine, aircraft engine, aerospace engineering, aircraft component,
	240	+ aircraft leasing, space technology, spaceport, aerospace and defense, aerospace and defence, unmanned aircraft, aircraft maintenance,
	241	+ mro, aircraft parts, general aviation, business jet]
	242	+ - slug: defense
	243	+ name: Defense
	244	+ parent: aerospace-defense
	245	+ description: Defence contractors, arms manufacturers and military technology.
	246	+ keywords: [defense, defence, defense industry, defence industry, arms industry, arms manufacturer, weapons, weapon, weapons manufacturer,
	247	+ military, military technology, military industry, armament, armaments, defense contractor, defence contractor, munitions,
	248	+ ammunition, firearms, firearms manufacturer, small arms, armored vehicle, armoured vehicle, naval shipbuilding, warship, missile,
	249	+ missiles, security and defense, military equipment, military aircraft, tank, radar, electronic warfare, defense electronics]
	250	+ # ------------------------------------------------------------------------------------------------ transport
	251	+ - slug: transportation
	252	+ name: Transportation
	253	+ description: Passenger and freight transport, rail, road, transit and infrastructure operators.
	254	+ keywords: [transportation, transport, transport industry, transportation industry, transport company, public transport, public transit,
	255	+ transit, rail transport, railway, railways, rail, railroad, railroads, railway company, rail freight, passenger rail, high-speed rail,
	256	+ metro, subway, tram, bus company, bus operator, coach operator, road transport, trucking, haulage, taxi, ferry, ferries,
	257	+ toll road, highway, motorway operator, airport, airports, airport operator, port, ports, port operator, seaport, infrastructure,
	258	+ transport infrastructure, rolling stock, railway equipment, locomotive, locomotives, railcar, train manufacturer, ride sharing,
	259	+ mobility services, car rental company, vehicle rental, parking, bike sharing, transportation company, transit authority]
	260	+ - slug: logistics
	261	+ name: Logistics
	262	+ parent: transportation
	263	+ description: Freight forwarding, parcel delivery, supply chain, warehousing and postal services.
	264	+ keywords: [logistics, logistics industry, logistics company, freight, freight forwarding, freight forwarder, freight transport, courier,
	265	+ courier service, parcel, parcel delivery, package delivery, express delivery, delivery, delivery service, postal service,
	266	+ postal, post, mail, supply chain, supply chain management, warehousing, warehouse, third-party logistics, 3pl, cargo,
	267	+ air cargo, cold chain, distribution logistics, last mile, last-mile delivery, food delivery, moving company, relocation,
	268	+ shipping and logistics, transportation and logistics, intermodal, freight railroad]
	269	+ - slug: airlines
	270	+ name: Airlines
	271	+ parent: transportation
	272	+ description: Passenger and cargo airlines, low-cost carriers and air charter.
	273	+ keywords: [airline, airlines, airline industry, air transport, air transportation, aviation company, low-cost airline, low-cost carrier,
	274	+ air carrier, flag carrier, regional airline, charter airline, cargo airline, air travel, passenger airline, air taxi, helicopter operator,
	275	+ commercial aviation, airline holding, air freight airline, airline alliance]
	276	+ - slug: shipping
	277	+ name: Shipping & Maritime
	278	+ parent: transportation
	279	+ description: Ocean shipping, container lines, tankers, ports and maritime services.
	280	+ keywords: [shipping, shipping industry, shipping company, maritime, maritime transport, maritime industry, sea transport, ocean shipping,
	281	+ container shipping, container line, shipping line, tanker, tankers, dry bulk, bulk shipping, ship owner, shipowner, ship management,
	282	+ cruise line, cruise, cruise ship, ferry operator, ocean freight, marine transport, marine services, offshore vessels, seafaring,
	283	+ ship operator, navigation, water transport, inland shipping, tugboat, towage, marine, maritime services, ship chartering]
	284	+ # ------------------------------------------------------------------------------------------------ telecom & media
	285	+ - slug: telecommunications
	286	+ name: Telecommunications
	287	+ description: Telecom operators, mobile networks, broadband, satellite communications and telecom equipment.
	288	+ keywords: [telecommunications, telecommunication, telecom, telecoms, telecommunications industry, telecommunications company,
	289	+ mobile network operator, mobile operator, mobile telephony, mobile network, wireless, wireless carrier, cellular, broadband,
	290	+ fiber optics, fibre, fixed-line, landline, telephone, telephony, voip, internet access, internet service provider, isp,
	291	+ cable operator, cable television operator, cable company, satellite communications, satellite communication, satellite operator,
	292	+ telecommunications equipment, telecom equipment, network infrastructure, tower company, telecommunications tower, 5g,
	293	+ unified communications, messaging, telecommunications services, phone company, communications, communication services]
	294	+ - slug: media
	295	+ name: Media
	296	+ description: Publishing, news, broadcasting, advertising and marketing services.
	297	+ keywords: [media, mass media, media industry, media company, publishing, publisher, publishing industry, book publishing, book publisher,
	298	+ newspaper, newspapers, newspaper publisher, magazine, magazines, periodical, press, news, news agency, news media, journalism,
	299	+ broadcasting, broadcaster, television, television broadcasting, television network, tv, radio, radio broadcasting, radio network,
	300	+ advertising, advertising agency, advertising industry, marketing, marketing agency, public relations, pr, digital marketing,
	301	+ media conglomerate, printing and publishing, information services, data provider, market research, communications agency,
	302	+ educational publishing, academic publishing, scientific publishing, comics publisher, trade publisher, outdoor advertising,
	303	+ billboard, media group, creative agency, content, podcast, stock photography, photography]
	304	+ - slug: entertainment
	305	+ name: Entertainment
	306	+ parent: media
	307	+ description: Film, television production, music, streaming, live events and theme parks.
	308	+ keywords: [entertainment, entertainment industry, entertainment company, film, film industry, film production, film production company,
	309	+ motion picture, motion pictures, movie, movies, cinema, cinema chain, movie theater, film studio, film distribution, television
	310	+ production, tv production, production company, animation, animation studio, music, music industry, record label, recording,
	311	+ music publishing, music streaming, streaming, streaming media, video streaming, video on demand, live entertainment, concert,
	312	+ concert promoter, live events, event promotion, theme park, amusement park, theme parks, circus, theatre, theater, performing arts,
	313	+ talent agency, sports entertainment, professional wrestling, esports, ticketing, casino, casinos, gambling, betting, sports betting,
	314	+ lottery, lotteries, sports club, football club, sports team, sports franchise, visual effects, vfx, sports league, dance, opera]
	315	+ - slug: gaming
	316	+ name: Video Games
	317	+ parent: media
	318	+ description: Video game developers, publishers, platforms and interactive entertainment.
	319	+ keywords: [video game, video games, video game industry, video game developer, video game publisher, game developer, game publisher,
	320	+ game development, gaming, games, computer game, computer games, mobile game, mobile games, mobile gaming, online game,
	321	+ online games, interactive entertainment, game studio, game engine, game console, video game console, arcade game, esports company,
	322	+ game platform, board game, tabletop game, toy and game, games industry, indie game, virtual reality, augmented reality, metaverse]
	323	+ # ------------------------------------------------------------------------------------------------ health
	324	+ - slug: healthcare
	325	+ name: Healthcare
	326	+ description: Hospitals, health services, health insurance providers and care delivery.
	327	+ keywords: [healthcare, health care, health, healthcare industry, health care industry, hospital, hospitals, hospital operator, clinic,
	328	+ clinics, medical, medicine, medical services, medical care, health services, health system, health network, nursing, nursing home,
	329	+ elderly care, senior living, home care, home health, dental, dentistry, dental care, veterinary, veterinary medicine, animal health,
	330	+ telehealth, telemedicine, digital health, health technology, healthtech, medical laboratory, laboratory, diagnostics laboratory,
	331	+ clinical laboratory, medical imaging, radiology, mental health, behavioral health, rehabilitation, physiotherapy, optometry,
	332	+ optical retail, health insurance provider, managed care, pharmacy benefit, pharmacy benefit management, healthcare services,
	333	+ dialysis, fertility, hospice, ambulance, emergency medical services, medical tourism, wellness, fitness, gym, health club,
	334	+ life sciences, life science, health and wellness, medical research, contract research organization, cro, medical distribution,
	335	+ pharmaceutical distribution, drug distribution, medical supplies, nutrition science]
	336	+ - slug: biotechnology
	337	+ name: Biotechnology
	338	+ parent: healthcare
	339	+ description: Biotech, genomics, cell and gene therapy and life-science tools.
	340	+ keywords: [biotechnology, biotech, biotechnology industry, biotechnology company, genomics, genetics, genetic engineering, gene therapy,
	341	+ cell therapy, synthetic biology, bioinformatics, biopharmaceutical, biopharma, biologics, immunotherapy, antibody, antibodies,
	342	+ molecular diagnostics, dna sequencing, sequencing, proteomics, microbiome, agricultural biotechnology, industrial biotechnology,
	343	+ bioengineering, biomedical, biomedical engineering, stem cell, regenerative medicine, life sciences tools, laboratory equipment,
	344	+ laboratory instruments, scientific instruments, research tools, vaccine development, crispr, mrna]
	345	+ - slug: pharmaceuticals
	346	+ name: Pharmaceuticals
	347	+ parent: healthcare
	348	+ description: Drug discovery, development, manufacturing and generics.
	349	+ keywords: [pharmaceutical, pharmaceuticals, pharmaceutical industry, pharma, pharmaceutical company, drug, drugs, drug manufacturer,
	350	+ drug development, drug discovery, generic drugs, generics, generic pharmaceuticals, medication, medications, medicines, vaccine,
	351	+ vaccines, over-the-counter, otc drugs, pharmaceutical manufacturing, contract manufacturing organization, cdmo, cmo, api manufacturing,
	352	+ active pharmaceutical ingredient, specialty pharma, oncology, ophthalmology, dermatology, nutraceutical, nutraceuticals,
	353	+ dietary supplement, supplements, animal pharmaceuticals, consumer health, pharmaceutical research, clinical trials, clinical research,
	354	+ pharmacology, homeopathy, traditional medicine, ayurveda]
	355	+ - slug: medical-devices
	356	+ name: Medical Devices
	357	+ parent: healthcare
	358	+ description: Medical equipment, devices, diagnostics and healthcare technology.
	359	+ keywords: [medical device, medical devices, medical equipment, medical technology, medtech, medical instruments, medical instrument,
	360	+ surgical instruments, surgical equipment, surgical, surgical robotics, diagnostic equipment, diagnostics, in vitro diagnostics,
	361	+ imaging equipment, medical imaging equipment, implants, implant, orthopedic, orthopedics, orthopaedic, prosthetics, prosthetic,
	362	+ hearing aid, hearing aids, hearing, dental equipment, dental products, cardiovascular devices, pacemaker, stent, insulin pump,
	363	+ glucose monitoring, respiratory equipment, ventilator, wheelchair, mobility aids, medical supplies manufacturer, lab equipment,
	364	+ health monitoring, wearable medical, contact lenses, eyeglasses, ophthalmic, endoscopy, infusion, dialysis equipment, sterilization,
	365	+ medical consumables, medical software, healthcare it, electronic health record, ehr, hospital equipment]
	366	+ # ------------------------------------------------------------------------------------------------ services
	367	+ - slug: hospitality
	368	+ name: Hospitality
	369	+ description: Hotels, resorts, restaurants chains, casinos and leisure venues.
	370	+ keywords: [hospitality, hospitality industry, hotel, hotels, hotel chain, hotel industry, hotel group, hotel operator, resort, resorts,
	371	+ lodging, accommodation, accommodations, motel, hostel, inn, bed and breakfast, vacation rental, timeshare, casino resort,
	372	+ restaurant group, restaurant operator, restaurant industry, bar, pub, pub chain, café, cafe, coffee shop, coffeehouse, coffee chain,
	373	+ food and hospitality, leisure, leisure industry, spa, wellness resort, ski resort, golf, golf course, club, nightclub,
	374	+ event venue, convention center, catering services, food service industry, contract catering, hotel management, franchise restaurants]
	375	+ - slug: travel
	376	+ name: Travel & Tourism
	377	+ description: Travel agencies, online travel, tour operators, cruise and tourism services.
	378	+ keywords: [travel, travel industry, tourism, tourism industry, travel agency, travel agencies, online travel agency, ota, online travel,
	379	+ tour operator, tour operators, travel technology, travel services, travel booking, booking, reservation system, global distribution
	380	+ system, gds, tourist, tourism company, cruise tourism, travel management, corporate travel, airline booking, holiday, holidays,
	381	+ vacation, vacations, package holidays, tourism board, travel retail, travel insurance, adventure travel, ecotourism, expedition,
	382	+ travel guide, travel publishing, luggage, travel accessories, destination management, theme travel, hotel booking, lodging platform]
	383	+ - slug: education
	384	+ name: Education
	385	+ description: Education providers, edtech, training, publishing and testing.
	386	+ keywords: [education, education industry, educational, educational services, edtech, education technology, educational technology,
	387	+ e-learning, online learning, online education, distance learning, school, schools, private school, school operator, university,
	388	+ universities, higher education, college, colleges, vocational training, training, professional training, corporate training,
	389	+ tutoring, test preparation, language school, language learning, language education, coding bootcamp, bootcamp, learning,
	390	+ learning platform, mooc, educational publisher, textbook, textbooks, early childhood education, childcare, daycare, nursery,
	391	+ education services, education management, for-profit education, exam, assessment, certification, driving school, music school,
	392	+ educational software, learning management system, lms, academic, academy, research institute, think tank]
	393	+ - slug: professional-services
	394	+ name: Professional Services
	395	+ description: Accounting, legal, staffing, outsourcing, engineering services and other B2B services.
	396	+ keywords: [professional services, business services, services, service industry, accounting, accountancy, accounting firm, audit, auditing,
	397	+ tax, tax services, legal services, law firm, law, legal, lawyer, lawyers, attorneys, staffing, staffing agency, recruitment,
	398	+ recruiting, human resources, hr, hr services, employment agency, temporary staffing, outsourcing, business process outsourcing,
	399	+ bpo, call center, customer service, facilities management, facility management, facility services, security services, security
	400	+ guard, private security, cleaning services, engineering services, architecture and engineering, design, design agency, industrial
	401	+ design, testing, inspection, certification services, tic, quality assurance, translation, translation services, market research
	402	+ firm, data analytics, business intelligence, credit rating agency, credit rating, rating agency, credit bureau, background check,
	403	+ payroll, payroll services, professional employer organization, peo, event management, exhibition, trade fair, trade shows,
	404	+ conference, printing services, document management, notary, patent, intellectual property services, real estate services,
	405	+ property services, consulting engineering, surveying, environmental consulting, government contractor, government services,
	406	+ defense services, aerospace services, research and development, r&d, contract research, laboratory testing, investigation]
	407	+ - slug: consulting
	408	+ name: Consulting
	409	+ parent: professional-services
	410	+ description: Management, strategy, IT and specialist consulting firms.
	411	+ keywords: [consulting, consultancy, consultancies, consulting firm, management consulting, management consultancy, strategy consulting,
	412	+ strategic consulting, it consulting, information technology consulting, technology consulting, business consulting, consulting
	413	+ services, advisory, advisory services, financial advisory services, economic consulting, engineering consulting, environmental
	414	+ consultancy, hr consulting, human resources consulting, executive search, headhunting, actuarial consulting, actuarial, risk
	415	+ consulting, systems integration, systems integrator, digital transformation, technology services, it outsourcing, professional
	416	+ consulting, tax advisory, transaction advisory, restructuring, public affairs, lobbying, political consulting, polling, research
	417	+ consultancy, design consultancy, brand consultancy, communications consultancy]
	418	+ # ------------------------------------------------------------------------------------------------ agriculture
	419	+ - slug: agriculture
	420	+ name: Agriculture
	421	+ description: Farming, agribusiness, seeds, forestry, fishing and agricultural inputs.
	422	+ keywords: [agriculture, agricultural, agriculture industry, agribusiness, agri-business, farming, farm, farms, farmer, crop, crops, crop
	423	+ production, seed, seeds, seed company, grain, grains, grain trading, agricultural trading, commodity trading, commodities,
	424	+ agricultural cooperative, cooperative, livestock, cattle, poultry, poultry farming, pig farming, aquaculture, fish farming, fishing,
	425	+ fisheries, fishery, forestry industry, plantation, plantations, palm oil, rubber plantation, horticulture, floriculture, flowers,
	426	+ greenhouse, vertical farming, agtech, agricultural technology, agricultural chemicals, animal feed, feed, animal nutrition,
	427	+ agricultural products, food production, sugar cane, coffee growing, cocoa, tobacco farming, tobacco growing, dairy farming,
	428	+ egg production, organic farming, viticulture, vineyard, orchard, fruit, vegetables, fruit and vegetables, agricultural machinery
	429	+ dealer, irrigation, land management, cotton, wool, timber production, cannabis, hemp, cannabis industry]

added scripts/seed_countries.py +126 −0

		`@@ -0,0 +1,126 @@`
	1	+#!/usr/bin/env python
	2	+"""Generate `registry/countries.csv` (ISO-3166-1 alpha-2, name, UN M49 region/subregion, centroid lat/lon).
	3	+
	4	+Sources (fetched once at generation time; the CSV is committed so nothing is downloaded at runtime):
	5	+ * ISO 3166 + UN M49 regions: https://github.com/lukes/ISO-3166-Countries-with-Regional-Codes (all/all.csv, public domain-ish data)
	6	+ * Centroids: Wikidata `P297` (ISO alpha-2) → `P625` (coordinate location) via the SPARQL endpoint; a few manual overrides fix items whose
	7	+ coordinate is a capital or an outlying island rather than the country centroid.
	8	+
	9	+Usage: .venv/bin/python scripts/seed_countries.py [--out registry/countries.csv]
	10	+"""
	11	+from __future__ import annotations
	12	+
	13	+import argparse
	14	+import csv
	15	+import io
	16	+import sys
	17	+import time
	18	+from pathlib import Path
	19	+
	20	+import httpx
	21	+
	22	+ROOT = Path(__file__).resolve().parents[1]
	23	+UA = "CompanyAtlasBot/0.1 (contact@spboucher.ai)"
	24	+ISO_URL = "https://raw.githubusercontent.com/lukes/ISO-3166-Countries-with-Regional-Codes/master/all/all.csv"
	25	+SPARQL = "https://query.wikidata.org/sparql"
	26	+NAME_OVERRIDES = {
	27	+ "US": "United States", "GB": "United Kingdom", "RU": "Russia", "KR": "South Korea", "KP": "North Korea", "IR": "Iran", "VN": "Vietnam",
	28	+ "TW": "Taiwan", "BO": "Bolivia", "VE": "Venezuela", "TZ": "Tanzania", "MD": "Moldova", "SY": "Syria", "LA": "Laos", "BN": "Brunei",
	29	+ "CZ": "Czechia", "FM": "Micronesia", "PS": "Palestine", "CD": "DR Congo", "CG": "Republic of the Congo", "CI": "Côte d'Ivoire",
	30	+ "VA": "Vatican City", "FK": "Falkland Islands", "VG": "British Virgin Islands", "VI": "U.S. Virgin Islands", "TR": "Türkiye",
	31	+ "NL": "Netherlands", "BQ": "Caribbean Netherlands", "SH": "Saint Helena", "MO": "Macao", "HK": "Hong Kong", "AE": "United Arab Emirates",
	32	+ "TF": "French Southern Territories", "UM": "U.S. Minor Outlying Islands", "GS": "South Georgia and the South Sandwich Islands",
	33	+}
	34	+# UN M49 does not assign a region to Taiwan (and Antarctica has none); keep the atlas usable for filtering.
	35	+REGION_OVERRIDES = {"TW": ("Asia", "Eastern Asia"), "AQ": ("Antarctica", "Antarctica")}
	36	+# Centroid overrides (lat, lon) where Wikidata's P625 is a capital/point rather than a usable centroid, or is missing.
	37	+CENTROID_OVERRIDES = {
	38	+ "US": (39.8, -98.6), "CA": (56.1, -106.3), "RU": (61.5, 105.3), "FR": (46.6, 2.2), "NO": (64.6, 12.7), "DK": (56.0, 10.0),
	39	+ "NL": (52.2, 5.3), "GB": (54.0, -2.5), "AU": (-25.3, 133.8), "NZ": (-41.5, 172.8), "CL": (-35.7, -71.5), "BR": (-14.2, -51.9),
	40	+ "ID": (-2.5, 118.0), "JP": (36.2, 138.3), "CN": (35.9, 104.2), "IN": (20.6, 79.0), "KI": (-3.4, -168.7), "FM": (7.4, 150.6),
	41	+ "PT": (39.4, -8.2), "ES": (40.5, -3.7), "EC": (-1.8, -78.2), "UM": (19.3, 166.6), "TF": (-49.3, 69.3), "AQ": (-82.9, 135.0),
	42	+ "ZA": (-30.6, 22.9), "AR": (-38.4, -63.6), "MX": (23.6, -102.6), "DE": (51.2, 10.4), "IT": (41.9, 12.6), "TW": (23.7, 121.0),
	43	+ "HK": (22.3, 114.2), "SG": (1.35, 103.8), "AE": (23.4, 53.8), "SA": (23.9, 45.1), "NG": (9.1, 8.7), "SE": (60.1, 18.6), "CH": (46.8, 8.2),
	44	+ "KR": (35.9, 127.8), "IL": (31.0, 34.9), "PS": (31.9, 35.2), "EG": (26.8, 30.8), "TR": (39.0, 35.2), "IR": (32.4, 53.7), "PK": (30.4, 69.3),
	45	+ "GL": (71.7, -42.6), "SJ": (77.6, 23.7), "MY": (4.2, 108.0), "PH": (12.9, 121.8), "VN": (14.1, 108.3), "TH": (15.9, 100.9),
	46	+ "UA": (48.4, 31.2), "PL": (51.9, 19.1), "KZ": (48.0, 66.9), "MN": (46.9, 103.8), "IE": (53.4, -8.2), "IS": (64.96, -19.0), "FI": (61.9, 25.7),
	47	+ "GR": (39.1, 21.8), "MA": (31.8, -7.1), "DZ": (28.0, 1.7), "LY": (26.3, 17.2), "SD": (12.9, 30.2), "ET": (9.1, 40.5), "KE": (-0.02, 37.9),
	48	+ "CD": (-4.0, 21.8), "AO": (-11.2, 17.9), "MZ": (-18.7, 35.5), "MG": (-18.8, 46.9), "TZ": (-6.4, 34.9), "ML": (17.6, -4.0), "NE": (17.6, 8.1),
	49	+ "TD": (15.5, 18.7), "MR": (21.0, -10.9), "PE": (-9.2, -75.0), "CO": (4.6, -74.3), "VE": (6.4, -66.6), "BO": (-16.3, -63.6), "PY": (-23.4, -58.4),
	50	+ "UY": (-32.5, -55.8), "CU": (21.5, -77.8), "GT": (15.8, -90.2), "HN": (15.2, -86.2), "NI": (12.9, -85.2), "PA": (8.5, -80.8), "CR": (9.7, -83.8),
	51	+ "AF": (33.9, 67.7), "IQ": (33.2, 43.7), "SY": (34.8, 39.0), "JO": (30.6, 36.2), "OM": (21.5, 55.9), "YE": (15.6, 48.5), "UZ": (41.4, 64.6),
	52	+ "TM": (38.97, 59.6), "NP": (28.4, 84.1), "BD": (23.7, 90.4), "MM": (21.9, 95.96), "LK": (7.9, 80.8), "KH": (12.6, 105.0), "LA": (19.9, 102.5),
	53	+ "PG": (-6.3, 143.96), "SB": (-9.6, 160.2), "VU": (-15.4, 166.96), "FJ": (-17.7, 178.1), "TO": (-21.2, -175.2), "WS": (-13.8, -172.1),
	54	+ "CV": (16.0, -24.0), "MU": (-20.3, 57.6), "SC": (-4.7, 55.5), "MV": (3.2, 73.2), "BH": (26.0, 50.6), "QA": (25.4, 51.2), "KW": (29.3, 47.5),
	55	+ "LB": (33.9, 35.9), "CY": (35.1, 33.4), "MT": (35.9, 14.4), "LU": (49.8, 6.1), "BE": (50.5, 4.5), "AT": (47.5, 14.6), "CZ": (49.8, 15.5),
	56	+ "SK": (48.7, 19.7), "HU": (47.2, 19.5), "RO": (45.9, 25.0), "BG": (42.7, 25.5), "RS": (44.0, 21.0), "HR": (45.1, 15.2), "SI": (46.2, 15.0),
	57	+ "BA": (43.9, 17.7), "ME": (42.7, 19.4), "MK": (41.6, 21.7), "AL": (41.2, 20.2), "XK": (42.6, 20.9), "EE": (58.6, 25.0), "LV": (56.9, 24.6),
	58	+ "LT": (55.2, 23.9), "BY": (53.7, 27.95), "MD": (47.4, 28.4), "GE": (42.3, 43.4), "AM": (40.1, 45.0), "AZ": (40.1, 47.6), "KG": (41.2, 74.8),
	59	+ "TJ": (38.9, 71.3), "BT": (27.5, 90.4), "TL": (-8.9, 125.7), "BN": (4.5, 114.7), "KP": (40.3, 127.5), "MO": (22.2, 113.55),
	60	+}
	61	+
	62	+
	63	+def fetch_iso() -> list[dict[str, str]]:
	64	+ r = httpx.get(ISO_URL, headers={"User-Agent": UA}, timeout=60, follow_redirects=True)
	65	+ r.raise_for_status()
	66	+ return list(csv.DictReader(io.StringIO(r.text)))
	67	+
	68	+
	69	+def fetch_centroids() -> dict[str, tuple[float, float]]:
	70	+ query = """
	71	+ SELECT ?iso ?coord WHERE {
	72	+ ?c wdt:P297 ?iso ; wdt:P625 ?coord .
	73	+ FILTER NOT EXISTS { ?c wdt:P576 ?dissolved }
	74	+ }"""
	75	+ for attempt in range(4):
	76	+ r = httpx.get(SPARQL, params={"query": query, "format": "json"}, headers={"User-Agent": UA, "Accept": "application/sparql-results+json"},
	77	+ timeout=90)
	78	+ if r.status_code == 200:
	79	+ break
	80	+ time.sleep(5 * (attempt + 1))
	81	+ r.raise_for_status()
	82	+ out: dict[str, tuple[float, float]] = {}
	83	+ for b in r.json()["results"]["bindings"]:
	84	+ iso = b["iso"]["value"].upper()
	85	+ val = b["coord"]["value"] # Point(lon lat)
	86	+ try:
	87	+ lon, lat = val.removeprefix("Point(").removesuffix(")").split()
	88	+ out.setdefault(iso, (round(float(lat), 3), round(float(lon), 3)))
	89	+ except ValueError:
	90	+ continue
	91	+ return out
	92	+
	93	+
	94	+def main() -> int:
	95	+ ap = argparse.ArgumentParser()
	96	+ ap.add_argument("--out", default=str(ROOT / "registry" / "countries.csv"))
	97	+ args = ap.parse_args()
	98	+ iso = fetch_iso()
	99	+ time.sleep(2)
	100	+ centroids = fetch_centroids()
	101	+ rows = []
	102	+ for r in iso:
	103	+ code = r["alpha-2"].strip().upper()
	104	+ if len(code) != 2:
	105	+ continue
	106	+ name = NAME_OVERRIDES.get(code) or r["name"].split(" (")[0].split(",")[0].strip()
	107	+ lat, lon = CENTROID_OVERRIDES.get(code) or centroids.get(code) or ("", "")
	108	+ region, subregion = REGION_OVERRIDES.get(code) or (r.get("region") or "", r.get("sub-region") or "")
	109	+ rows.append({"code": code, "name": name, "region": region, "subregion": subregion, "lat": lat, "lon": lon})
	110	+ # Kosovo is user-assigned (XK) and not in ISO 3166-1; several data sources (and Wikidata companies) use it.
	111	+ if not any(x["code"] == "XK" for x in rows):
	112	+ rows.append({"code": "XK", "name": "Kosovo", "region": "Europe", "subregion": "Southern Europe", "lat": 42.6, "lon": 20.9})
	113	+ rows.sort(key=lambda x: x["code"])
	114	+ missing = [x["code"] for x in rows if x["lat"] == ""]
	115	+ out = Path(args.out)
	116	+ out.parent.mkdir(parents=True, exist_ok=True)
	117	+ with out.open("w", newline="", encoding="utf-8") as f:
	118	+ w = csv.DictWriter(f, fieldnames=["code", "name", "region", "subregion", "lat", "lon"])
	119	+ w.writeheader()
	120	+ w.writerows(rows)
	121	+ print(f"wrote {len(rows)} countries → {out} (missing centroids: {missing or 'none'})")
	122	+ return 0
	123	+
	124	+
	125	+if __name__ == "__main__":
	126	+ sys.exit(main())

added scripts/seed_edgar.py +181 −0

		`@@ -0,0 +1,181 @@`
	1	+#!/usr/bin/env python
	2	+"""SEC EDGAR enrichment for the seed registry (docs/SEEDS.md).
	3	+
	4	+Matches registry companies to `company_tickers.json` (ticker first, then normalised name) and fills missing `sec_cik`, `ticker`,
	5	+`exchange`, `hq_city`/`hq_region` and an industry hint (SIC → taxonomy slug) from the submissions JSON. Existing values are never
	6	+overwritten. Polite: SEC-required User-Agent, ≤ 5 requests/s, responses cached under data/seed/edgar/.
	7	+
	8	+Usage: .venv/bin/python scripts/seed_edgar.py [--max-submissions N] [--dry-run]
	9	+"""
	10	+from __future__ import annotations
	11	+
	12	+import argparse
	13	+import json
	14	+import logging
	15	+import sys
	16	+import time
	17	+from collections import Counter, defaultdict
	18	+from pathlib import Path
	19	+from typing import Any
	20	+
	21	+import httpx
	22	+
	23	+ROOT = Path(__file__).resolve().parents[1]
	24	+sys.path.insert(0, str(ROOT / "src"))
	25	+
	26	+from companyatlas.ids import normalize_alias
	27	+from companyatlas.registry.industries import map_sic
	28	+
	29	+log = logging.getLogger("seed_edgar")
	30	+USER_AGENT = "CompanyAtlasBot/0.1 contact@spboucher.ai"
	31	+TICKERS_URL = "https://www.sec.gov/files/company_tickers.json"
	32	+SUBMISSIONS_URL = "https://data.sec.gov/submissions/CIK{cik:010d}.json"
	33	+MIN_INTERVAL_S = 0.21 # ≤ 5 requests per second (SEC fair-access policy is 10/s)
	34	+CACHE_DIR = ROOT / "data" / "seed" / "edgar"
	35	+OUT_DIR = ROOT / "registry" / "companies"
	36	+US_STATES = {"AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE", "FL", "GA", "HI", "ID", "IL", "IN", "IA", "KS", "KY", "LA", "ME", "MD", "MA", "MI",
	37	+ "MN", "MS", "MO", "MT", "NE", "NV", "NH", "NJ", "NM", "NY", "NC", "ND", "OH", "OK", "OR", "PA", "RI", "SC", "SD", "TN", "TX", "UT",
	38	+ "VT", "VA", "WA", "WV", "WI", "WY", "DC", "PR"}
	39	+
	40	+
	41	+class Edgar:
	42	+ def __init__(self) -> None:
	43	+ self.client = httpx.Client(headers={"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"}, timeout=30, follow_redirects=True)
	44	+ self.last = 0.0
	45	+ self.requests = 0
	46	+ CACHE_DIR.mkdir(parents=True, exist_ok=True)
	47	+
	48	+ def get_json(self, url: str, cache_name: str) -> Any \| None:
	49	+ path = CACHE_DIR / cache_name
	50	+ if path.exists():
	51	+ return json.loads(path.read_text(encoding="utf-8"))
	52	+ for attempt in range(4):
	53	+ wait = MIN_INTERVAL_S - (time.monotonic() - self.last)
	54	+ if wait > 0:
	55	+ time.sleep(wait)
	56	+ try:
	57	+ r = self.client.get(url)
	58	+ except httpx.HTTPError as e:
	59	+ log.warning("edgar %s: %s", url, e)
	60	+ time.sleep(2 * (attempt + 1))
	61	+ continue
	62	+ finally:
	63	+ self.last = time.monotonic()
	64	+ self.requests += 1
	65	+ if r.status_code == 200:
	66	+ path.write_text(r.text, encoding="utf-8")
	67	+ return r.json()
	68	+ if r.status_code == 404:
	69	+ path.write_text("null", encoding="utf-8")
	70	+ return None
	71	+ log.warning("edgar %s → HTTP %d, backing off", url, r.status_code)
	72	+ time.sleep(5 * (attempt + 1))
	73	+ return None
	74	+
	75	+
	76	+def load_registry() -> dict[Path, list[dict[str, Any]]]:
	77	+ out: dict[Path, list[dict[str, Any]]] = {}
	78	+ for path in sorted(OUT_DIR.glob("wikidata-*.ndjson")):
	79	+ with path.open(encoding="utf-8") as f:
	80	+ out[path] = [json.loads(line) for line in f if line.strip()]
	81	+ return out
	82	+
	83	+
	84	+def main() -> int:
	85	+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
	86	+ ap.add_argument("--max-submissions", type=int, default=4000, help="cap on submissions JSON fetches")
	87	+ ap.add_argument("--dry-run", action="store_true")
	88	+ ap.add_argument("-v", "--verbose", action="store_true")
	89	+ args = ap.parse_args()
	90	+ logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
	91	+ logging.getLogger("httpx").setLevel(logging.WARNING)
	92	+
	93	+ edgar = Edgar()
	94	+ tickers = edgar.get_json(TICKERS_URL, "company_tickers.json")
	95	+ if not tickers:
	96	+ log.error("could not fetch %s", TICKERS_URL)
	97	+ return 1
	98	+ by_ticker: dict[str, list[dict[str, Any]]] = defaultdict(list)
	99	+ by_name: dict[str, set[int]] = defaultdict(set)
	100	+ for e in tickers.values():
	101	+ by_ticker[e["ticker"].upper()].append(e)
	102	+ by_name[normalize_alias(e["title"])].add(int(e["cik_str"]))
	103	+ log.info("edgar tickers: %d entries, %d names", len(tickers), len(by_name))
	104	+
	105	+ files = load_registry()
	106	+ stats: Counter[str] = Counter()
	107	+ submissions_fetched = 0
	108	+ for path, rows in files.items():
	109	+ for r in rows:
	110	+ cik: int \| None = int(r["sec_cik"]) if r.get("sec_cik") and str(r["sec_cik"]).isdigit() else None
	111	+ if cik is None:
	112	+ cands: set[int] = set()
	113	+ if r.get("ticker") and r["ticker"].upper() in by_ticker:
	114	+ for e in by_ticker[r["ticker"].upper()]:
	115	+ if normalize_alias(e["title"])[:6] == normalize_alias(r["display_name"])[:6] or r.get("country") == "US":
	116	+ cands.add(int(e["cik_str"]))
	117	+ if not cands:
	118	+ names = [r["display_name"], r.get("legal_name") or ""] + list(r.get("aliases") or [])
	119	+ for nm in names:
	120	+ if nm and normalize_alias(nm) in by_name:
	121	+ cands \|= by_name[normalize_alias(nm)]
	122	+ if len(cands) > 1 and r.get("country") != "US":
	123	+ cands = set() # ambiguous non-US name: do not guess
	124	+ if len(cands) == 1:
	125	+ cik = cands.pop()
	126	+ stats["matched_name_or_ticker"] += 1
	127	+ elif len(cands) > 1:
	128	+ stats["ambiguous"] += 1
	129	+ continue
	130	+ else:
	131	+ continue
	132	+ else:
	133	+ stats["had_cik"] += 1
	134	+ r.setdefault("industry_labels", [])
	135	+ changed = False
	136	+ if not r.get("sec_cik"):
	137	+ r["sec_cik"] = str(cik)
	138	+ changed = True
	139	+ needs_sub = not r.get("exchange") or not r.get("industries") or not r.get("hq_city") or not r.get("ticker")
	140	+ if needs_sub and submissions_fetched < args.max_submissions:
	141	+ sub = edgar.get_json(SUBMISSIONS_URL.format(cik=cik), f"CIK{cik:010d}.json")
	142	+ submissions_fetched += 1
	143	+ if sub:
	144	+ if not r.get("ticker") and sub.get("tickers"):
	145	+ r["ticker"] = sub["tickers"][0]
	146	+ changed = True
	147	+ if not r.get("exchange") and sub.get("exchanges"):
	148	+ r["exchange"] = next((x for x in sub["exchanges"] if x), None)
	149	+ changed = changed or bool(r["exchange"])
	150	+ if r.get("ticker"):
	151	+ r["public_company"] = True
	152	+ sic = sub.get("sic")
	153	+ slug = map_sic(sic)
	154	+ if sic and sub.get("sicDescription") and f"SIC {sic} {sub['sicDescription']}" not in r["industry_labels"]:
	155	+ r["industry_labels"].append(f"SIC {sic} {sub['sicDescription']}")
	156	+ changed = True
	157	+ if slug and not r.get("industries"):
	158	+ r["industries"] = [slug]
	159	+ stats["industry_from_sic"] += 1
	160	+ changed = True
	161	+ biz = (sub.get("addresses") or {}).get("business") or {}
	162	+ if not r.get("hq_city") and biz.get("city"):
	163	+ r["hq_city"] = biz["city"].title()
	164	+ changed = True
	165	+ if not r.get("hq_region") and biz.get("stateOrCountry") in US_STATES:
	166	+ r["hq_region"] = biz["stateOrCountry"]
	167	+ changed = True
	168	+ if changed:
	169	+ r["source_edgar"] = {"cik": str(cik), "enriched_at": time.strftime("%Y-%m-%dT%H:%M:%S+00:00", time.gmtime())}
	170	+ stats["rows_enriched"] += 1
	171	+ if not args.dry_run:
	172	+ with path.open("w", encoding="utf-8") as f:
	173	+ for r in rows:
	174	+ f.write(json.dumps(r, ensure_ascii=False) + "\n")
	175	+ log.info("edgar done: %s (%d HTTP requests, %d submissions)", dict(stats), edgar.requests, submissions_fetched)
	176	+ (ROOT / "data" / "seed" / "edgar-report.json").write_text(json.dumps({"stats": stats, "submissions": submissions_fetched}, indent=1))
	177	+ return 0
	178	+
	179	+
	180	+if __name__ == "__main__":
	181	+ sys.exit(main())

added scripts/seed_wikidata.py +846 −0

		`@@ -0,0 +1,846 @@`
	1	+#!/usr/bin/env python
	2	+"""Polite Wikidata harvester for the Company Atlas seed registry (docs/SEEDS.md).
	3	+
	4	+Stages (each resumable; every SPARQL result is cached under data/seed/wikidata/sparql/<sha256>.json):
	5	+ candidates company-class × sitelink-band queries + per-country and per-industry boost queries → data/seed/candidates.json
	6	+ select normalise websites, drop generic hosts / duplicate domains, diversify (US ≤ 40 %, others ≤ 12 %, country minimums)
	7	+ → data/seed/selected.json
	8	+ details batched detail queries (labels, legal names, industries, HQ, coordinates, tickers, LEI, CIK, parent, logo, employees)
	9	+ → data/seed/details.json
	10	+ assemble importance + tiers + industry mapping + parent/domain conflicts → registry/companies/wikidata-<region>.ndjson + README.md
	11	+ all the four stages in sequence (default)
	12	+
	13	+Politeness: User-Agent CompanyAtlasBot/0.1, one query at a time, ≥ 2 s between queries, 60 s server timeout, retries with backoff.
	14	+"""
	15	+from __future__ import annotations
	16	+
	17	+import argparse
	18	+import csv
	19	+import hashlib
	20	+import json
	21	+import logging
	22	+import math
	23	+import re
	24	+import sys
	25	+import time
	26	+from collections import Counter, defaultdict
	27	+from datetime import UTC, datetime
	28	+from pathlib import Path
	29	+from typing import Any
	30	+from urllib.parse import quote, urlparse
	31	+
	32	+import httpx
	33	+
	34	+ROOT = Path(__file__).resolve().parents[1]
	35	+sys.path.insert(0, str(ROOT / "src"))
	36	+
	37	+from companyatlas.registry.industries import load_industries, map_industry, top_level_of, top_level_slugs
	38	+from companyatlas.urls import registrable_domain
	39	+
	40	+log = logging.getLogger("seed_wikidata")
	41	+
	42	+SPARQL_URL = "https://query.wikidata.org/sparql"
	43	+USER_AGENT = "CompanyAtlasBot/0.1 (contact@spboucher.ai)"
	44	+MIN_INTERVAL_S = 2.0
	45	+TIMEOUT_S = 60.0
	46	+MAX_TRIES = 6
	47	+DATA_DIR = ROOT / "data" / "seed"
	48	+CACHE_DIR = DATA_DIR / "wikidata" / "sparql"
	49	+REGISTRY_DIR = ROOT / "registry"
	50	+OUT_DIR = REGISTRY_DIR / "companies"
	51	+COUNTRIES_CSV = REGISTRY_DIR / "countries.csv"
	52	+
	53	+# Wikidata classes queried with wdt:P31 (no P279* — the subclass tree of "business" is too broad to page). Banded classes are large.
	54	+CLASSES: dict[str, str] = {
	55	+ "Q4830453": "business", "Q6881511": "enterprise", "Q891723": "public company", "Q783794": "company", "Q1589009": "privately held company",
	56	+ "Q167037": "corporation", "Q18388277": "technology company", "Q1058914": "software company", "Q22687": "bank", "Q46970": "airline",
	57	+ "Q786820": "automobile manufacturer", "Q6500733": "pharmaceutical company", "Q210167": "video game developer",
	58	+ "Q1137109": "video game publisher", "Q1762059": "film production company", "Q18127": "record label", "Q2085381": "publisher",
	59	+ "Q613142": "law firm", "Q740752": "transport company", "Q2401749": "telecommunications company", "Q131734": "brewery",
	60	+ "Q507619": "retail chain", "Q936518": "aerospace manufacturer", "Q206361": "conglomerate", "Q249556": "railway company",
	61	+ "Q2005696": "real estate company", "Q708676": "shipyard", "Q1668024": "internet service", "Q1320047": "book publisher",
	62	+ "Q19967801": "online service",
	63	+}
	64	+BANDED_CLASSES = {"Q4830453", "Q6881511", "Q891723", "Q46970"}
	65	+# Country boosts run only when the class stage yielded fewer than BOOST_MARGIN × minimum candidates for that country.
	66	+BOOST_MARGIN = 2.0
	67	+SITELINK_BANDS: list[tuple[int, int \| None]] = [(80, None), (50, 80), (35, 50), (25, 35), (18, 25), (12, 18), (8, 12), (5, 8)]
	68	+MIN_SITELINKS = 5
	69	+BOOST_MIN_SITELINKS = 2
	70	+BOOST_LIMIT = 2500
	71	+INDUSTRY_BOOST_LIMIT = 600
	72	+
	73	+# Diversification (spec: "seed diversified companies across US, Canada, Europe, UK, Japan, South Korea, India, Australia, LatAm, ME, Africa, SEA")
	74	+US_CAP_SHARE = 0.40
	75	+OTHER_CAP_SHARE = 0.12
	76	+UNKNOWN_COUNTRY_SHARE = 0.03
	77	+COUNTRY_MINIMUMS: dict[str, int] = {
	78	+ **dict.fromkeys(["CA", "GB", "DE", "FR", "JP", "KR", "IN", "AU"], 150),
	79	+ **dict.fromkeys(["BR", "MX", "AE", "SA", "ZA", "NG", "SG", "ID", "NL", "SE", "CH", "ES", "IT", "CN", "TW", "HK"], 60),
	80	+}
	81	+INDUSTRY_MINIMUM = 60
	82	+INDUSTRY_RESERVE = 150 # extra candidates per top-level industry fetched in the detail stage to satisfy INDUSTRY_MINIMUM
	83	+# English labels of Wikidata items commonly used as P452 (industry) values, per top-level industry, for the industry-boost queries.
	84	+INDUSTRY_BOOST_LABELS: dict[str, list[str]] = {
	85	+ "technology": ["information technology", "electronics industry", "consumer electronics", "computer hardware", "electronics"],
	86	+ "financial-services": ["financial services", "insurance", "banking", "asset management", "investment banking", "payment system"],
	87	+ "real-estate": ["real estate", "real estate development", "property management", "real estate industry"],
	88	+ "construction": ["construction", "construction industry", "civil engineering", "building construction", "engineering"],
	89	+ "retail": ["retail", "retailing", "supermarket", "department store", "wholesale", "grocery store"],
	90	+ "consumer-goods": ["consumer goods", "cosmetics industry", "toy industry", "furniture industry", "fast-moving consumer goods",
	91	+ "household goods", "luxury goods", "personal care", "home appliance"],
	92	+ "energy": ["energy industry", "energy", "electric power industry", "nuclear power", "oil industry", "petroleum industry",
	93	+ "renewable energy", "solar energy", "wind power"],
	94	+ "utilities": ["public utility", "electric utility", "water industry", "waste management", "electricity generation",
	95	+ "electric power distribution", "water supply"],
	96	+ "mining": ["mining", "mining industry", "steel industry", "metallurgy", "gold mining", "steelmaking", "metal industry"],
	97	+ "chemicals": ["chemical industry", "chemicals", "petrochemical industry", "specialty chemicals", "plastics industry", "fertilizer"],
	98	+ "materials": ["glass industry", "paper industry", "packaging industry", "pulp and paper industry", "forestry", "building material",
	99	+ "cement industry", "wood industry"],
	100	+ "manufacturing": ["manufacturing", "mechanical engineering", "industrial machinery", "machine industry", "electrical engineering",
	101	+ "machine tool", "industrial engineering", "heavy industry", "shipbuilding"],
	102	+ "automotive": ["automotive industry", "automobile", "automotive", "motor vehicle", "auto parts"],
	103	+ "aerospace-defense": ["aerospace industry", "arms industry", "defense industry", "aerospace", "aviation industry", "space industry",
	104	+ "defence industry"],
	105	+ "transportation": ["transport", "rail transport", "public transport", "logistics", "transportation", "freight transport", "shipping",
	106	+ "maritime transport", "railway"],
	107	+ "telecommunications": ["telecommunications industry", "telecommunications", "telecommunication", "mobile telephony",
	108	+ "internet service provider"],
	109	+ "media": ["mass media", "publishing", "advertising", "broadcasting", "entertainment industry", "film industry", "video game industry",
	110	+ "music industry", "media industry", "newspaper"],
	111	+ "healthcare": ["health care industry", "health care", "medical technology", "hospital", "pharmaceutical industry", "biotechnology",
	112	+ "medical device", "healthcare industry", "medical equipment"],
	113	+ "hospitality": ["hospitality industry", "hotel industry", "hotel", "restaurant", "hospitality", "catering", "restaurant chain"],
	114	+ "travel": ["tourism", "travel agency", "travel industry", "travel", "tourism industry", "online travel agency", "tour operator"],
	115	+ "education": ["education", "educational technology", "higher education", "e-learning", "education industry", "educational services"],
	116	+ "professional-services": ["professional services", "consulting", "management consulting", "accounting", "outsourcing", "staffing",
	117	+ "business services", "legal services", "information technology consulting", "human resources"],
	118	+ "agriculture": ["agriculture", "agribusiness", "agricultural industry", "forestry", "fishing industry", "food industry", "farming",
	119	+ "aquaculture", "agricultural machinery"],
	120	+}
	121	+
	122	+# Hosts that are never a company's own website (social profiles, blogs, stores, code forges, encyclopaedias, site builders …).
	123	+GENERIC_DOMAINS = {
	124	+ "facebook.com", "fb.com", "linkedin.com", "twitter.com", "x.com", "instagram.com", "youtube.com", "youtu.be", "wikipedia.org",
	125	+ "wikimedia.org", "wikidata.org", "blogspot.com", "blogspot.co.uk", "wordpress.com", "tumblr.com", "medium.com", "github.com", "gitlab.com",
	126	+ "sourceforge.net", "archive.org", "tiktok.com", "vk.com", "weibo.com", "t.me", "telegram.me", "bit.ly", "wix.com", "wixsite.com",
	127	+ "weebly.com", "squarespace.com", "webnode.com", "jimdo.com", "jimdosite.com", "godaddysites.com", "carrd.co", "notion.site", "substack.com",
	128	+ "patreon.com", "itch.io", "steampowered.com", "bandcamp.com", "soundcloud.com", "imdb.com", "myspace.com", "flickr.com", "pinterest.com",
	129	+ "twitch.tv", "discord.gg", "discord.com", "bilibili.com", "tistory.com", "ameblo.jp", "fc2.com", "livejournal.com", "geocities.com",
	130	+ "angelfire.com", "tripod.com", "netlify.app", "vercel.app", "herokuapp.com", "github.io", "pages.dev", "web.app", "firebaseapp.com",
	131	+ "glitch.me", "strikingly.com", "yolasite.com", "webs.com", "linktr.ee", "about.me", "crunchbase.com", "bloomberg.com", "sec.gov",
	132	+ "yelp.com", "tripadvisor.com", "foursquare.com", "goo.gl", "ow.ly", "tinyurl.com", "wa.me", "whatsapp.com", "line.me", "kakao.com",
	133	+ "spotify.com", "deezer.com", "vimeo.com", "dailymotion.com", "behance.net", "dribbble.com", "etsy.com", "ebay.com", "aliexpress.com",
	134	+ "taobao.com", "tmall.com", "rakuten.co.jp", "shopee.com", "mercadolibre.com", "google.co.uk", "googleusercontent.com", "gstatic.com",
	135	+ "webflow.io", "mystrikingly.com", "site123.me", "simplesite.com", "ucoz.ru", "narod.ru", "hatenablog.com", "note.com", "wixstatic.com",
	136	+ "shopify.com", "myshopify.com", "bigcartel.com", "storenvy.com", "yahoo.co.jp", "yahoo.com", "aol.com", "cargo.site", "format.com",
	137	+ "blogger.com", "mixi.jp", "naver.me", "cafe24.com", "modoo.at", "over-blog.com", "canalblog.com", "skyrock.com", "free.fr", "orange.fr",
	138	+ "wanadoo.fr", "pagesperso-orange.fr", "t-online.de", "web.de", "gmx.de", "chello.at", "bplaced.net", "beepworld.de", "npage.de",
	139	+ "altervista.org", "xoom.it", "libero.it", "interfree.it", "terra.com.br", "uol.com.br", "ig.com.br", "sapo.pt", "webcindario.com",
	140	+ "iespana.es", "galeon.com", "hpage.com", "wordpress.org", "js.org", "readthedocs.io", "gitbook.io", "gumroad.com", "ko-fi.com",
	141	+ "onlyfans.com", "reddit.com", "quora.com", "scribd.com", "issuu.com", "slideshare.net", "docs.google.com", "drive.google.com",
	142	+ "sites.google.com", "play.google.com", "apps.apple.com", "itunes.apple.com", "amazon.com", "amazon.co.uk", "amazon.de", "amazon.co.jp",
	143	+ "amazon.fr", "amazon.ca", "amazon.in", "amazon.com.br", "amzn.to", "microsoft.com", "apple.com", "google.com", "naver.com", "daum.net",
	144	+ "qq.com", "163.com", "sina.com.cn", "baidu.com", "sohu.com", "douyin.com", "kuaishou.com", "zhihu.com", "xiaohongshu.com",
	145	+}
	146	+# Registrable domains that are themselves seed companies: only the bare/www host counts as that company's site (not sub-brands/store pages).
	147	+PLATFORM_ROOTS = {"google.com": "www.google.com", "apple.com": "www.apple.com", "amazon.com": "www.amazon.com", "microsoft.com": "www.microsoft.com",
	148	+ "naver.com": "www.naver.com", "yahoo.com": "www.yahoo.com", "qq.com": "www.qq.com", "baidu.com": "www.baidu.com",
	149	+ "163.com": "www.163.com", "sohu.com": "www.sohu.com", "sina.com.cn": "www.sina.com.cn", "daum.net": "www.daum.net",
	150	+ "kakao.com": "www.kakao.com", "shopify.com": "www.shopify.com", "spotify.com": "www.spotify.com", "reddit.com": "www.reddit.com",
	151	+ "ebay.com": "www.ebay.com", "etsy.com": "www.etsy.com", "yelp.com": "www.yelp.com", "tripadvisor.com": "www.tripadvisor.com",
	152	+ "linkedin.com": "www.linkedin.com", "facebook.com": "www.facebook.com", "instagram.com": "www.instagram.com",
	153	+ "youtube.com": "www.youtube.com", "twitter.com": "twitter.com", "x.com": "x.com", "tiktok.com": "www.tiktok.com",
	154	+ "github.com": "github.com", "gitlab.com": "gitlab.com", "medium.com": "medium.com", "substack.com": "substack.com",
	155	+ "patreon.com": "www.patreon.com", "twitch.tv": "www.twitch.tv", "pinterest.com": "www.pinterest.com", "vimeo.com": "vimeo.com",
	156	+ "soundcloud.com": "soundcloud.com", "bandcamp.com": "bandcamp.com", "imdb.com": "www.imdb.com", "crunchbase.com": "www.crunchbase.com",
	157	+ "bloomberg.com": "www.bloomberg.com", "wix.com": "www.wix.com", "squarespace.com": "www.squarespace.com", "weebly.com": "www.weebly.com",
	158	+ "godaddy.com": "www.godaddy.com", "wordpress.com": "wordpress.com", "tumblr.com": "www.tumblr.com", "flickr.com": "www.flickr.com",
	159	+ "quora.com": "www.quora.com", "scribd.com": "www.scribd.com", "issuu.com": "issuu.com", "discord.com": "discord.com",
	160	+ "telegram.org": "telegram.org", "whatsapp.com": "www.whatsapp.com", "line.me": "line.me", "bilibili.com": "www.bilibili.com",
	161	+ "weibo.com": "weibo.com", "vk.com": "vk.com", "zhihu.com": "www.zhihu.com", "aliexpress.com": "www.aliexpress.com",
	162	+ "taobao.com": "www.taobao.com", "tmall.com": "www.tmall.com", "rakuten.co.jp": "www.rakuten.co.jp", "shopee.com": "shopee.com",
	163	+ "mercadolibre.com": "www.mercadolibre.com", "archive.org": "archive.org", "sourceforge.net": "sourceforge.net", "itch.io": "itch.io",
	164	+ "steampowered.com": "store.steampowered.com", "deezer.com": "www.deezer.com", "dailymotion.com": "www.dailymotion.com",
	165	+ "behance.net": "www.behance.net", "dribbble.com": "dribbble.com", "notion.so": "www.notion.so", "gumroad.com": "gumroad.com",
	166	+ "onlyfans.com": "onlyfans.com", "yahoo.co.jp": "www.yahoo.co.jp", "aol.com": "www.aol.com", "free.fr": "www.free.fr",
	167	+ "orange.fr": "www.orange.fr", "t-online.de": "www.t-online.de", "web.de": "web.de", "gmx.de": "www.gmx.de", "uol.com.br": "www.uol.com.br",
	168	+ "terra.com.br": "www.terra.com.br", "sapo.pt": "www.sapo.pt", "libero.it": "www.libero.it", "douyin.com": "www.douyin.com",
	169	+ "kuaishou.com": "www.kuaishou.com", "xiaohongshu.com": "www.xiaohongshu.com", "note.com": "note.com", "cafe24.com": "www.cafe24.com",
	170	+ "hatenablog.com": "hatenablog.com", "mixi.jp": "mixi.jp", "myspace.com": "myspace.com", "livejournal.com": "www.livejournal.com",
	171	+ "webflow.com": "webflow.com", "carrd.co": "carrd.co", "linktr.ee": "linktr.ee", "about.me": "about.me", "netlify.com": "www.netlify.com",
	172	+ "vercel.com": "vercel.com", "heroku.com": "www.heroku.com", "glitch.com": "glitch.com", "readthedocs.org": "readthedocs.org",
	173	+ "gitbook.com": "www.gitbook.com", "ko-fi.com": "ko-fi.com", "foursquare.com": "foursquare.com", "slideshare.net": "www.slideshare.net"}
	174	+
	175	+
	176	+# ------------------------------------------------------------------------------------------------------------ SPARQL client
	177	+class Sparql:
	178	+ def __init__(self, *, refresh: bool = False) -> None:
	179	+ self.client = httpx.Client(headers={"User-Agent": USER_AGENT, "Accept": "application/sparql-results+json"}, timeout=TIMEOUT_S + 15,
	180	+ follow_redirects=True)
	181	+ self.last_call = 0.0
	182	+ self.refresh = refresh
	183	+ self.queries = 0
	184	+ self.cached = 0
	185	+ CACHE_DIR.mkdir(parents=True, exist_ok=True)
	186	+
	187	+ def query(self, sparql: str, *, label: str = "") -> list[dict[str, str]]:
	188	+ key = hashlib.sha256(sparql.encode("utf-8")).hexdigest()
	189	+ path = CACHE_DIR / f"{key}.json"
	190	+ if path.exists() and not self.refresh:
	191	+ self.cached += 1
	192	+ return json.loads(path.read_text(encoding="utf-8"))["bindings"]
	193	+ delay = 5.0
	194	+ last_err = ""
	195	+ gateway_timeouts = 0
	196	+ for attempt in range(1, MAX_TRIES + 1):
	197	+ wait = MIN_INTERVAL_S - (time.monotonic() - self.last_call)
	198	+ if wait > 0:
	199	+ time.sleep(wait)
	200	+ t0 = time.monotonic()
	201	+ try:
	202	+ r = self.client.get(SPARQL_URL, params={"query": sparql, "format": "json"})
	203	+ self.last_call = time.monotonic()
	204	+ self.queries += 1
	205	+ if r.status_code == 200:
	206	+ try:
	207	+ # strict=False: a handful of Wikidata literals contain raw control characters.
	208	+ payload = json.loads(r.text, strict=False)
	209	+ except json.JSONDecodeError:
	210	+ # The endpoint streams results and, on a server-side timeout, appends a Java stack trace to a 200 body
	211	+ # (which the gateway may even cache). Treat a truncated body as a timeout so callers can split or skip.
	212	+ if "TimeoutException" in r.text or "SPARQL-QUERY" in r.text:
	213	+ raise TimeoutError(f"server timeout (truncated body) {label}") from None
	214	+ last_err = "truncated/invalid JSON body"
	215	+ continue
	216	+ rows = [{k: v["value"] for k, v in b.items()} for b in payload["results"]["bindings"]]
	217	+ path.write_text(json.dumps({"label": label, "fetched_at": datetime.now(UTC).isoformat(), "query": sparql, "bindings": rows},
	218	+ ensure_ascii=False), encoding="utf-8")
	219	+ log.info("sparql ok %s rows=%d %.1fs", label, len(rows), self.last_call - t0)
	220	+ return rows
	221	+ last_err = f"HTTP {r.status_code}: {r.text[:160]!r}"
	222	+ if r.status_code == 504:
	223	+ # Gateway timeout = the query ran past the 60 s server limit. One retry (load varies), then let the caller split it.
	224	+ gateway_timeouts += 1
	225	+ if gateway_timeouts >= 2:
	226	+ raise TimeoutError(f"gateway timeout {label}")
	227	+ if r.status_code == 429:
	228	+ retry_after = r.headers.get("Retry-After")
	229	+ delay = max(delay, float(retry_after)) if retry_after and retry_after.isdigit() else max(delay, 30.0)
	230	+ if r.status_code == 400:
	231	+ raise RuntimeError(f"bad query {label}: {r.text[:500]}")
	232	+ if r.status_code == 500 and "TimeoutException" in r.text:
	233	+ raise TimeoutError(f"server timeout {label}")
	234	+ except (httpx.TimeoutException, httpx.TransportError) as e:
	235	+ self.last_call = time.monotonic()
	236	+ last_err = f"{type(e).__name__}: {e}"
	237	+ log.warning("sparql retry %d/%d %s (%s) sleeping %.0fs", attempt, MAX_TRIES, label, last_err, delay)
	238	+ time.sleep(delay)
	239	+ delay = min(delay * 2, 120.0)
	240	+ raise RuntimeError(f"sparql failed {label}: {last_err}")
	241	+
	242	+
	243	+def values_clause(var: str, qids: list[str]) -> str:
	244	+ return f"VALUES ?{var} {{ {' '.join('wd:' + q for q in qids)} }}"
	245	+
	246	+
	247	+def class_values() -> str:
	248	+ return values_clause("cls", list(CLASSES))
	249	+
	250	+
	251	+# Truthy website (best rank, deprecated excluded). Dissolution is fetched as an OPTIONAL and filtered client-side: `FILTER NOT EXISTS`
	252	+# sub-selects over tens of thousands of bindings are what pushed the big class scans past the 60 s server limit.
	253	+WEBSITE_BLOCK = """
	254	+ ?item wdt:P856 ?web .
	255	+ OPTIONAL { ?item wdt:P576 ?dissolved }
	256	+ OPTIONAL { ?item wdt:P17 ?c . ?c wdt:P297 ?iso }
	257	+"""
	258	+
	259	+
	260	+def q_class_band(cls: str, lo: int, hi: int \| None) -> str:
	261	+ band = f"?sl >= {lo}" + (f" && ?sl < {hi}" if hi else "")
	262	+ return f"""SELECT ?item ?sl ?web ?iso WHERE {{
	263	+ ?item wdt:P31 wd:{cls} ; wikibase:sitelinks ?sl .
	264	+ FILTER({band})
	265	+ {WEBSITE_BLOCK}
	266	+}}"""
	267	+
	268	+
	269	+def q_country_qids(isos: list[str]) -> str:
	270	+ vals = " ".join(json.dumps(x) for x in isos)
	271	+ return f"""SELECT ?iso ?country WHERE {{ VALUES ?iso {{ {vals} }} ?country wdt:P297 ?iso . FILTER NOT EXISTS {{ ?country wdt:P576 [] }} }}"""
	272	+
	273	+
	274	+def q_country_boost(iso: str, country_qid: str) -> str:
	275	+ """Country-first scan (P17 → website → class) with the optimizer pinned. Cheap for countries with few items in Wikidata; the
	276	+ caller only issues it for countries still short of their minimum after the class stage and tolerates a timeout."""
	277	+ return f"""SELECT ?item ?sl ?web ?iso WHERE {{
	278	+ hint:Query hint:optimizer "None" .
	279	+ ?item wdt:P17 wd:{country_qid} .
	280	+ ?item wdt:P856 ?web .
	281	+ ?item wdt:P31 ?cls .
	282	+ {class_values()}
	283	+ ?item wikibase:sitelinks ?sl . FILTER(?sl >= {BOOST_MIN_SITELINKS})
	284	+ OPTIONAL {{ ?item wdt:P576 ?dissolved }}
	285	+ BIND("{iso}" AS ?iso)
	286	+}}"""
	287	+
	288	+
	289	+def q_industry_boost(labels: list[str]) -> str:
	290	+ vals = " ".join(json.dumps(x) + "@en" for x in labels)
	291	+ return f"""SELECT ?item ?sl ?web ?iso ?indLabel WHERE {{
	292	+ hint:Query hint:optimizer "None" .
	293	+ VALUES ?indLabel {{ {vals} }}
	294	+ ?ind rdfs:label ?indLabel .
	295	+ ?item wdt:P452 ?ind .
	296	+ ?item wikibase:sitelinks ?sl .
	297	+ FILTER(?sl >= {BOOST_MIN_SITELINKS})
	298	+ ?item wdt:P31 ?cls .
	299	+ {class_values()}
	300	+ {WEBSITE_BLOCK}
	301	+}} ORDER BY DESC(?sl) LIMIT {INDUSTRY_BOOST_LIMIT}"""
	302	+
	303	+
	304	+def q_labels(qids: list[str]) -> str:
	305	+ return f"""SELECT ?item ?itemLabel ?itemDescription ?itemAltLabel WHERE {{
	306	+ {values_clause("item", qids)}
	307	+ SERVICE wikibase:label {{ bd:serviceParam wikibase:language "en". }}
	308	+}}"""
	309	+
	310	+
	311	+# Plain (non-aggregated) detail queries, reduced client-side: a GROUP BY with a dozen SAMPLE() aggregates over nested OPTIONALs makes
	312	+# Blazegraph throw StackOverflowError.
	313	+def q_misc(qids: list[str]) -> str:
	314	+ return f"""SELECT ?item ?inception ?coord ?lei ?cik ?logo WHERE {{
	315	+ {values_clause("item", qids)}
	316	+ OPTIONAL {{ ?item wdt:P571 ?inception }}
	317	+ OPTIONAL {{ ?item wdt:P625 ?coord }}
	318	+ OPTIONAL {{ ?item wdt:P1278 ?lei }}
	319	+ OPTIONAL {{ ?item wdt:P5531 ?cik }}
	320	+ OPTIONAL {{ ?item wdt:P154 ?logo }}
	321	+}}"""
	322	+
	323	+
	324	+def q_hq_parent(qids: list[str]) -> str:
	325	+ return f"""SELECT ?item ?hq ?hqEnd ?hqLabel ?hqcoord ?hqRegionLabel ?hqiso ?parent ?parentEnd ?parentLabel WHERE {{
	326	+ {values_clause("item", qids)}
	327	+ OPTIONAL {{
	328	+ ?item p:P159 ?hqs . ?hqs ps:P159 ?hq .
	329	+ OPTIONAL {{ ?hqs pq:P582 ?hqEnd }}
	330	+ OPTIONAL {{ ?hq rdfs:label ?hqLabel FILTER(LANG(?hqLabel) = "en") }}
	331	+ OPTIONAL {{ ?hq wdt:P625 ?hqcoord }}
	332	+ OPTIONAL {{ ?hq wdt:P131 ?hqRegion . ?hqRegion rdfs:label ?hqRegionLabel FILTER(LANG(?hqRegionLabel) = "en") }}
	333	+ OPTIONAL {{ ?hq wdt:P17 ?hqc . ?hqc wdt:P297 ?hqiso }}
	334	+ }}
	335	+ OPTIONAL {{
	336	+ ?item p:P749 ?ps . ?ps ps:P749 ?parent .
	337	+ OPTIONAL {{ ?ps pq:P582 ?parentEnd }}
	338	+ OPTIONAL {{ ?parent rdfs:label ?parentLabel FILTER(LANG(?parentLabel) = "en") }}
	339	+ }}
	340	+}}"""
	341	+
	342	+
	343	+def q_names_industries(qids: list[str]) -> str:
	344	+ return f"""SELECT ?item ?legal ?indLabel WHERE {{
	345	+ {values_clause("item", qids)}
	346	+ OPTIONAL {{ ?item wdt:P1448 ?legal }}
	347	+ OPTIONAL {{ ?item wdt:P452 ?ind . ?ind rdfs:label ?indLabel FILTER(LANG(?indLabel) = "en") }}
	348	+}}"""
	349	+
	350	+
	351	+def q_tickers_employees(qids: list[str]) -> str:
	352	+ return f"""SELECT ?item ?ticker ?exchangeLabel ?ticker2 ?employees ?empDate WHERE {{
	353	+ {values_clause("item", qids)}
	354	+ OPTIONAL {{
	355	+ ?item p:P414 ?exs . ?exs ps:P414 ?exchange . FILTER NOT EXISTS {{ ?exs pq:P582 [] }}
	356	+ OPTIONAL {{ ?exs pq:P249 ?ticker }}
	357	+ OPTIONAL {{ ?exchange rdfs:label ?exchangeLabel FILTER(LANG(?exchangeLabel) = "en") }}
	358	+ }}
	359	+ OPTIONAL {{ ?item wdt:P249 ?ticker2 }}
	360	+ OPTIONAL {{ ?item p:P1128 ?es . ?es ps:P1128 ?employees . OPTIONAL {{ ?es pq:P585 ?empDate }} }}
	361	+}}"""
	362	+
	363	+
	364	+# ------------------------------------------------------------------------------------------------------------ helpers
	365	+def qid_of(uri: str) -> str:
	366	+ return uri.rsplit("/", 1)[-1]
	367	+
	368	+
	369	+def load_countries() -> dict[str, dict[str, str]]:
	370	+ with COUNTRIES_CSV.open(encoding="utf-8") as f:
	371	+ return {r["code"]: r for r in csv.DictReader(f)}
	372	+
	373	+
	374	+def normalise_website(url: str) -> tuple[str, str] \| None:
	375	+ """→ (website `https://host`, registrable domain) or None when unusable/generic."""
	376	+ url = (url or "").strip()
	377	+ if not url:
	378	+ return None
	379	+ if "://" not in url:
	380	+ url = "https://" + url
	381	+ try:
	382	+ p = urlparse(url)
	383	+ except ValueError:
	384	+ return None
	385	+ host = (p.hostname or "").lower().strip().rstrip(".")
	386	+ if not host or "." not in host or re.fullmatch(r"[\d.]+", host) or ":" in host:
	387	+ return None
	388	+ if p.scheme not in ("http", "https"):
	389	+ return None
	390	+ try:
	391	+ host.encode("idna")
	392	+ except UnicodeError:
	393	+ return None
	394	+ dom = registrable_domain(host)
	395	+ if not dom or "." not in dom:
	396	+ return None
	397	+ path = (p.path or "/").rstrip("/")
	398	+ if dom in PLATFORM_ROOTS:
	399	+ canonical_host = PLATFORM_ROOTS[dom]
	400	+ if host not in (canonical_host, dom, "www." + dom) or path:
	401	+ return None
	402	+ return f"https://{canonical_host}", dom
	403	+ if dom in GENERIC_DOMAINS or host in GENERIC_DOMAINS:
	404	+ return None
	405	+ if host.endswith((".blogspot.com", ".wordpress.com", ".github.io", ".wixsite.com", ".weebly.com", ".tumblr.com")):
	406	+ return None
	407	+ return f"https://{host}", dom
	408	+
	409	+
	410	+def parse_point(value: str \| None) -> tuple[float, float] \| None:
	411	+ if not value or not value.startswith("Point("):
	412	+ return None
	413	+ try:
	414	+ lon, lat = value[6:-1].split()
	415	+ return round(float(lat), 5), round(float(lon), 5)
	416	+ except ValueError:
	417	+ return None
	418	+
	419	+
	420	+def parse_year(value: str \| None) -> int \| None:
	421	+ if not value:
	422	+ return None
	423	+ m = re.match(r"^(-?\d{1,4})-", value)
	424	+ if not m:
	425	+ return None
	426	+ y = int(m.group(1))
	427	+ return y if 1000 <= y <= datetime.now(UTC).year else None
	428	+
	429	+
	430	+def parse_int(value: str \| None) -> int \| None:
	431	+ try:
	432	+ return int(float(value)) if value not in (None, "") else None
	433	+ except ValueError:
	434	+ return None
	435	+
	436	+
	437	+def commons_url(filename: str \| None) -> str \| None:
	438	+ if not filename:
	439	+ return None
	440	+ name = filename.rsplit("/", 1)[-1].replace(" ", "_")
	441	+ return f"https://commons.wikimedia.org/wiki/Special:FilePath/{quote(name)}"
	442	+
	443	+
	444	+def dump_json(path: Path, data: Any) -> None:
	445	+ path.parent.mkdir(parents=True, exist_ok=True)
	446	+ path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
	447	+
	448	+
	449	+def load_json(path: Path) -> Any:
	450	+ return json.loads(path.read_text(encoding="utf-8"))
	451	+
	452	+
	453	+# ------------------------------------------------------------------------------------------------------------ stage: candidates
	454	+def stage_candidates(sp: Sparql, *, refresh: bool = False) -> dict[str, dict[str, Any]]:
	455	+ out_path = DATA_DIR / "candidates.json"
	456	+ cands: dict[str, dict[str, Any]] = {}
	457	+ dissolved: set[str] = set()
	458	+
	459	+ def add(row: dict[str, str], *, cls: str \| None = None, boost_country: str \| None = None, boost_industry: str \| None = None) -> None:
	460	+ if row.get("dissolved"):
	461	+ dissolved.add(qid_of(row["item"]))
	462	+ return
	463	+ qid = qid_of(row["item"])
	464	+ c = cands.setdefault(qid, {"sitelinks": 0, "websites": [], "isos": [], "classes": [], "boost_countries": [], "boost_industries": []})
	465	+ c["sitelinks"] = max(c["sitelinks"], parse_int(row.get("sl")) or 0)
	466	+ if row.get("web") and row["web"] not in c["websites"]:
	467	+ c["websites"].append(row["web"])
	468	+ iso = (row.get("iso") or "").upper()
	469	+ if iso and iso not in c["isos"]:
	470	+ c["isos"].append(iso)
	471	+ if cls and cls not in c["classes"]:
	472	+ c["classes"].append(cls)
	473	+ if boost_country and boost_country not in c["boost_countries"]:
	474	+ c["boost_countries"].append(boost_country)
	475	+ if boost_industry and boost_industry not in c["boost_industries"]:
	476	+ c["boost_industries"].append(boost_industry)
	477	+
	478	+ def run_band(cls: str, lo: int, hi: int \| None) -> None:
	479	+ label = f"class {cls} {CLASSES[cls]} sl[{lo},{hi or '∞'})"
	480	+ try:
	481	+ rows = sp.query(q_class_band(cls, lo, hi), label=label)
	482	+ except TimeoutError:
	483	+ if hi is None:
	484	+ hi = 400
	485	+ if hi - lo <= 1:
	486	+ raise
	487	+ mid = (lo + hi) // 2
	488	+ log.warning("splitting band %s → [%d,%d) [%d,%d)", label, lo, mid, mid, hi)
	489	+ run_band(cls, lo, mid)
	490	+ run_band(cls, mid, hi)
	491	+ return
	492	+ for r in rows:
	493	+ add(r, cls=cls)
	494	+
	495	+ for cls in CLASSES:
	496	+ if cls in BANDED_CLASSES:
	497	+ for lo, hi in SITELINK_BANDS:
	498	+ run_band(cls, lo, hi)
	499	+ else:
	500	+ run_band(cls, MIN_SITELINKS, None)
	501	+ log.info("candidates so far: %d", len(cands))
	502	+ iso_to_qid = {r["iso"]: qid_of(r["country"]) for r in sp.query(q_country_qids(list(COUNTRY_MINIMUMS)), label="country qids")}
	503	+ per_country: Counter[str] = Counter(c["isos"][0] for c in cands.values() if c["isos"])
	504	+ for iso, minimum in COUNTRY_MINIMUMS.items():
	505	+ if per_country[iso] >= minimum * BOOST_MARGIN or iso not in iso_to_qid:
	506	+ log.info("country %s: %d candidates (min %d) — no boost", iso, per_country[iso], minimum)
	507	+ continue
	508	+ try:
	509	+ rows = sp.query(q_country_boost(iso, iso_to_qid[iso]), label=f"country boost {iso}")
	510	+ except (RuntimeError, TimeoutError) as e:
	511	+ log.warning("country boost %s failed (%s) — known gap, see README", iso, e)
	512	+ continue
	513	+ for r in rows:
	514	+ add(r, boost_country=iso)
	515	+ for slug, labels in INDUSTRY_BOOST_LABELS.items():
	516	+ for r in sp.query(q_industry_boost(labels), label=f"industry boost {slug}"):
	517	+ add(r, boost_industry=slug)
	518	+ for qid in dissolved:
	519	+ cands.pop(qid, None)
	520	+ dump_json(out_path, cands)
	521	+ log.info("candidates: %d (%d dissolved dropped) → %s", len(cands), len(dissolved), out_path)
	522	+ return cands
	523	+
	524	+
	525	+# ------------------------------------------------------------------------------------------------------------ stage: select
	526	+def prefilter(cands: dict[str, dict[str, Any]], countries: dict[str, dict[str, str]]) -> tuple[list[dict[str, Any]], dict[str, int]]:
	527	+ """Normalise websites, pick one country, drop generic hosts and duplicate registrable domains (highest sitelinks wins)."""
	528	+ stats: Counter[str] = Counter()
	529	+ rows: list[dict[str, Any]] = []
	530	+ for qid, c in cands.items():
	531	+ picked = None
	532	+ for w in c["websites"]:
	533	+ picked = normalise_website(w)
	534	+ if picked:
	535	+ break
	536	+ if not picked:
	537	+ stats["dropped_generic_or_invalid_website"] += 1
	538	+ continue
	539	+ website, dom = picked
	540	+ isos = [i for i in c["isos"] if i in countries]
	541	+ country = isos[0] if isos else None
	542	+ rows.append({"wikidata_id": qid, "sitelinks": c["sitelinks"], "website": website, "canonical_domain": dom, "country": country,
	543	+ "country_candidates": isos, "classes": c["classes"], "boost_industries": c["boost_industries"]})
	544	+ rows.sort(key=lambda r: (-r["sitelinks"], int(r["wikidata_id"][1:])))
	545	+ by_domain: dict[str, dict[str, Any]] = {}
	546	+ for r in rows:
	547	+ keeper = by_domain.get(r["canonical_domain"])
	548	+ if keeper is None:
	549	+ by_domain[r["canonical_domain"]] = r
	550	+ else:
	551	+ keeper.setdefault("domain_conflicts", []).append(r["wikidata_id"])
	552	+ stats["dropped_duplicate_domain"] += 1
	553	+ kept = list(by_domain.values())
	554	+ stats["kept"] = len(kept)
	555	+ return kept, dict(stats)
	556	+
	557	+
	558	+def select_diversified(rows: list[dict[str, Any]], *, target: int, countries: dict[str, dict[str, str]]) -> list[dict[str, Any]]:
	559	+ """Sitelinks-ordered selection with country caps and minimum coverage. Rows must be sorted by sitelinks desc."""
	560	+ caps: dict[str \| None, int] = defaultdict(lambda: int(target * OTHER_CAP_SHARE))
	561	+ caps["US"] = int(target * US_CAP_SHARE)
	562	+ caps[None] = int(target * UNKNOWN_COUNTRY_SHARE)
	563	+ chosen: dict[str, dict[str, Any]] = {}
	564	+ per_country: Counter[str \| None] = Counter()
	565	+
	566	+ def take(r: dict[str, Any]) -> None:
	567	+ chosen[r["wikidata_id"]] = r
	568	+ per_country[r["country"]] += 1
	569	+
	570	+ by_country: dict[str \| None, list[dict[str, Any]]] = defaultdict(list)
	571	+ for r in rows:
	572	+ by_country[r["country"]].append(r)
	573	+ for iso, minimum in COUNTRY_MINIMUMS.items():
	574	+ for r in by_country.get(iso, [])[:minimum]:
	575	+ take(r)
	576	+ for r in rows:
	577	+ if len(chosen) >= target:
	578	+ break
	579	+ if r["wikidata_id"] in chosen or r["sitelinks"] < MIN_SITELINKS:
	580	+ continue
	581	+ if per_country[r["country"]] >= caps[r["country"]]:
	582	+ continue
	583	+ take(r)
	584	+ return list(chosen.values())
	585	+
	586	+
	587	+def stage_select(*, target: int) -> list[dict[str, Any]]:
	588	+ countries = load_countries()
	589	+ cands = load_json(DATA_DIR / "candidates.json")
	590	+ kept, stats = prefilter(cands, countries)
	591	+ log.info("prefilter: %s", stats)
	592	+ selected = select_diversified(kept, target=target, countries=countries)
	593	+ # Pool for the industry guarantee: best boost candidates per industry, fetched in the details stage too (≤ INDUSTRY_RESERVE each).
	594	+ chosen_ids = {r["wikidata_id"] for r in selected}
	595	+ reserve: list[dict[str, Any]] = []
	596	+ per_ind: Counter[str] = Counter()
	597	+ for r in kept:
	598	+ if r["wikidata_id"] in chosen_ids:
	599	+ continue
	600	+ for slug in r["boost_industries"]:
	601	+ if per_ind[slug] < INDUSTRY_RESERVE:
	602	+ per_ind[slug] += 1
	603	+ reserve.append(r)
	604	+ break
	605	+ dump_json(DATA_DIR / "selected.json", {"selected": selected, "reserve": reserve, "stats": stats, "target": target})
	606	+ log.info("selected %d (+%d reserve for industry minimums) → %s", len(selected), len(reserve), DATA_DIR / "selected.json")
	607	+ return selected
	608	+
	609	+
	610	+# ------------------------------------------------------------------------------------------------------------ stage: details
	611	+def stage_details(sp: Sparql, *, batch: int = 100) -> dict[str, dict[str, Any]]:
	612	+ sel = load_json(DATA_DIR / "selected.json")
	613	+ qids = [r["wikidata_id"] for r in sel["selected"]] + [r["wikidata_id"] for r in sel["reserve"]]
	614	+ out_path = DATA_DIR / "details.json"
	615	+ details: dict[str, dict[str, Any]] = load_json(out_path) if out_path.exists() else {}
	616	+ todo = [q for q in qids if q not in details]
	617	+ log.info("details: %d to fetch (%d cached)", len(todo), len(qids) - len(todo))
	618	+ for i in range(0, len(todo), batch):
	619	+ chunk = todo[i:i + batch]
	620	+ tag = f"details {i + len(chunk)}/{len(todo)}"
	621	+ d: dict[str, dict[str, Any]] = {q: {"legal_names": [], "industry_labels": [], "tickers": [], "employees_obs": []} for q in chunk}
	622	+ for r in sp.query(q_labels(chunk), label=f"{tag} labels"):
	623	+ q = qid_of(r["item"])
	624	+ d[q]["label"] = r.get("itemLabel")
	625	+ d[q]["description"] = r.get("itemDescription")
	626	+ d[q]["alt_labels"] = [a.strip() for a in (r.get("itemAltLabel") or "").split(",") if a.strip()]
	627	+ for r in sp.query(q_misc(chunk), label=f"{tag} misc"):
	628	+ m = d[qid_of(r["item"])]
	629	+ for key in ("inception", "coord", "lei", "cik", "logo"):
	630	+ if r.get(key) and (not m.get(key) or (key == "inception" and r[key] < m[key])):
	631	+ m[key] = r[key]
	632	+ grouped: dict[str, list[dict[str, str]]] = defaultdict(list)
	633	+ for r in sp.query(q_hq_parent(chunk), label=f"{tag} hq/parent"):
	634	+ grouped[qid_of(r["item"])].append(r)
	635	+ for q, rs in grouped.items():
	636	+ hqs = [r for r in rs if r.get("hq")]
	637	+ current = [r for r in hqs if not r.get("hqEnd")] or hqs
	638	+ if current:
	639	+ r = current[0]
	640	+ d[q].update({"hq": qid_of(r["hq"]), "hq_label": r.get("hqLabel"), "hq_coord": r.get("hqcoord"),
	641	+ "hq_region": r.get("hqRegionLabel"), "hq_iso": r.get("hqiso")})
	642	+ parents = [r for r in rs if r.get("parent") and not r.get("parentEnd")]
	643	+ if parents:
	644	+ d[q].update({"parent": qid_of(parents[0]["parent"]), "parent_label": parents[0].get("parentLabel")})
	645	+ for r in sp.query(q_names_industries(chunk), label=f"{tag} names/industries"):
	646	+ q = qid_of(r["item"])
	647	+ if r.get("legal") and r["legal"] not in d[q]["legal_names"]:
	648	+ d[q]["legal_names"].append(r["legal"])
	649	+ if r.get("indLabel") and r["indLabel"] not in d[q]["industry_labels"]:
	650	+ d[q]["industry_labels"].append(r["indLabel"])
	651	+ for r in sp.query(q_tickers_employees(chunk), label=f"{tag} tickers/employees"):
	652	+ q = qid_of(r["item"])
	653	+ t = r.get("ticker") or r.get("ticker2")
	654	+ if t:
	655	+ pair = [t, r.get("exchangeLabel")]
	656	+ if pair not in d[q]["tickers"]:
	657	+ d[q]["tickers"].append(pair)
	658	+ if r.get("employees"):
	659	+ obs = [r["employees"], r.get("empDate")]
	660	+ if obs not in d[q]["employees_obs"]:
	661	+ d[q]["employees_obs"].append(obs)
	662	+ details.update(d)
	663	+ dump_json(out_path, details)
	664	+ log.info("details: %d entries → %s", len(details), out_path)
	665	+ return details
	666	+
	667	+
	668	+# ------------------------------------------------------------------------------------------------------------ stage: assemble
	669	+def importance_score(sitelinks: int, employees: int \| None, public: bool, ticker: str \| None) -> float:
	670	+ s_sl = min(1.0, math.log1p(max(sitelinks, 0)) / math.log1p(300))
	671	+ s_emp = min(1.0, math.log10((employees or 0) + 1) / 6) if employees else 0.0
	672	+ score = 0.6 * s_sl + 0.2 * s_emp + 0.1 * (1 if public else 0) + 0.1 * (1 if ticker else 0)
	673	+ return round(min(1.0, max(0.02, score)), 4)
	674	+
	675	+
	676	+def assign_tiers(rows: list[dict[str, Any]]) -> None:
	677	+ rows.sort(key=lambda r: (-r["importance"], -r["sitelinks"], r["wikidata_id"]))
	678	+ for i, r in enumerate(rows):
	679	+ r["tier"] = 1 if i < 150 else 2 if i < 950 else 3 if i < 3450 else 4
	680	+
	681	+
	682	+def build_row(sel: dict[str, Any], d: dict[str, Any], harvested_at: str) -> dict[str, Any]:
	683	+ label = d.get("label") or ""
	684	+ if not label or re.fullmatch(r"Q\d+", label):
	685	+ label = (d.get("legal_names") or [sel["canonical_domain"]])[0]
	686	+ legal = next((x for x in d.get("legal_names", []) if re.search(r"[A-Za-z]", x)), None) or (d.get("legal_names") or [None])[0]
	687	+ aliases = [a for a in d.get("alt_labels", []) if a and a != label and len(a) <= 80][:8]
	688	+ emp = None
	689	+ obs = d.get("employees_obs") or []
	690	+ if obs:
	691	+ obs = sorted(obs, key=lambda o: (o[1] or ""), reverse=True)
	692	+ emp = parse_int(obs[0][0])
	693	+ tickers = d.get("tickers") or []
	694	+ ticker, exchange = (tickers[0][0], tickers[0][1]) if tickers else (None, None)
	695	+ public = "Q891723" in sel.get("classes", []) or bool(ticker) or bool(exchange)
	696	+ coord = parse_point(d.get("coord")) or parse_point(d.get("hq_coord"))
	697	+ country = sel["country"]
	698	+ if d.get("hq_iso") and d["hq_iso"] in (sel.get("country_candidates") or []):
	699	+ country = d["hq_iso"]
	700	+ industry_labels = d.get("industry_labels") or []
	701	+ slugs = map_industry(industry_labels)
	702	+ if not slugs and sel.get("boost_industries"):
	703	+ slugs = [sel["boost_industries"][0]]
	704	+ if not slugs:
	705	+ slugs = map_industry([label, d.get("description") or ""] + [CLASSES.get(c, "") for c in sel.get("classes", [])], limit=2)
	706	+ return {
	707	+ "wikidata_id": sel["wikidata_id"], "display_name": label.strip(), "legal_name": legal, "aliases": aliases, "website": sel["website"],
	708	+ "canonical_domain": sel["canonical_domain"], "country": country, "hq_city": d.get("hq_label"), "hq_region": d.get("hq_region"),
	709	+ "lat": coord[0] if coord else None, "lon": coord[1] if coord else None, "industries": slugs, "industry_labels": industry_labels[:8],
	710	+ "founded_year": parse_year(d.get("inception")), "employees": emp, "public_company": public, "ticker": ticker, "exchange": exchange,
	711	+ "lei": d.get("lei"), "sec_cik": str(d["cik"]).lstrip("0") or None if d.get("cik") else None,
	712	+ "parent": {"wikidata_id": d["parent"], "name": d.get("parent_label")} if d.get("parent") else None,
	713	+ "logo_url": commons_url(d.get("logo")), "description": (d.get("description") or None), "sitelinks": sel["sitelinks"],
	714	+ "importance": importance_score(sel["sitelinks"], emp, public, ticker), "tier": 4, "source": "wikidata", "harvested_at": harvested_at,
	715	+ "domain_conflicts": sel.get("domain_conflicts") or [],
	716	+ }
	717	+
	718	+
	719	+def stage_assemble() -> list[dict[str, Any]]:
	720	+ countries = load_countries()
	721	+ sel = load_json(DATA_DIR / "selected.json")
	722	+ details = load_json(DATA_DIR / "details.json")
	723	+ harvested_at = datetime.now(UTC).replace(microsecond=0).isoformat()
	724	+ selected = [r for r in sel["selected"] if r["wikidata_id"] in details]
	725	+ reserve = [r for r in sel["reserve"] if r["wikidata_id"] in details]
	726	+ rows = [build_row(s, details[s["wikidata_id"]], harvested_at) for s in selected]
	727	+ # Industry minimums: top up from the reserve where a top-level industry is below the floor.
	728	+ counts: Counter[str] = Counter()
	729	+ for r in rows:
	730	+ for top in {top_level_of(s) for s in r["industries"]}:
	731	+ counts[top] += 1
	732	+ chosen = {r["wikidata_id"] for r in rows}
	733	+ reserve_rows = [build_row(s, details[s["wikidata_id"]], harvested_at) for s in reserve]
	734	+ reserve_rows.sort(key=lambda r: -r["sitelinks"])
	735	+ for top in top_level_slugs():
	736	+ for rr in reserve_rows:
	737	+ if counts[top] >= INDUSTRY_MINIMUM:
	738	+ break
	739	+ if rr["wikidata_id"] in chosen or top not in {top_level_of(s) for s in rr["industries"]}:
	740	+ continue
	741	+ rows.append(rr)
	742	+ chosen.add(rr["wikidata_id"])
	743	+ for t in {top_level_of(s) for s in rr["industries"]}:
	744	+ counts[t] += 1
	745	+ # Parent/child sharing a domain: parent wins (regardless of sitelinks); duplicate domains (should not happen after prefilter) drop the lower.
	746	+ by_domain: dict[str, dict[str, Any]] = {}
	747	+ for r in sorted(rows, key=lambda r: -r["sitelinks"]):
	748	+ by_domain.setdefault(r["canonical_domain"], r)
	749	+ ids = {r["wikidata_id"] for r in by_domain.values()}
	750	+ dropped: list[dict[str, Any]] = []
	751	+ for r in list(by_domain.values()):
	752	+ p = r.get("parent")
	753	+ if p and p["wikidata_id"] in ids and p["wikidata_id"] != r["wikidata_id"]:
	754	+ parent_row = next((x for x in by_domain.values() if x["wikidata_id"] == p["wikidata_id"]), None)
	755	+ if parent_row and parent_row["canonical_domain"] == r["canonical_domain"]:
	756	+ dropped.append({"wikidata_id": r["wikidata_id"], "reason": "shares_parent_domain", "parent": p["wikidata_id"]})
	757	+ for r in rows:
	758	+ if r["wikidata_id"] not in ids:
	759	+ dropped.append({"wikidata_id": r["wikidata_id"], "reason": "duplicate_domain", "domain": r["canonical_domain"]})
	760	+ drop_ids = {x["wikidata_id"] for x in dropped}
	761	+ final = [r for r in by_domain.values() if r["wikidata_id"] not in drop_ids]
	762	+ for r in final:
	763	+ r["notes"] = [{"related_domain_conflict": q} for q in r.pop("domain_conflicts", [])] or []
	764	+ if not r["notes"]:
	765	+ r.pop("notes")
	766	+ assign_tiers(final)
	767	+ # Write per region.
	768	+ OUT_DIR.mkdir(parents=True, exist_ok=True)
	769	+ for old in OUT_DIR.glob("wikidata-*.ndjson"):
	770	+ old.unlink()
	771	+ by_region: dict[str, list[dict[str, Any]]] = defaultdict(list)
	772	+ for r in final:
	773	+ region = (countries.get(r["country"] or "", {}).get("region") or "other").lower().replace(" ", "-")
	774	+ by_region[region].append(r)
	775	+ for region, items in sorted(by_region.items()):
	776	+ items.sort(key=lambda r: (r["tier"], -r["importance"], r["wikidata_id"]))
	777	+ with (OUT_DIR / f"wikidata-{region}.ndjson").open("w", encoding="utf-8") as f:
	778	+ for r in items:
	779	+ f.write(json.dumps(r, ensure_ascii=False) + "\n")
	780	+ dump_json(DATA_DIR / "dropped.json", dropped)
	781	+ write_readme(final, countries)
	782	+ log.info("assembled %d companies into %d region files (%d dropped) → %s", len(final), len(by_region), len(dropped), OUT_DIR)
	783	+ return final
	784	+
	785	+
	786	+def write_readme(rows: list[dict[str, Any]], countries: dict[str, dict[str, str]]) -> None:
	787	+ names = {i.slug: i.name for i in load_industries()}
	788	+ by_country = Counter(r["country"] or "??" for r in rows)
	789	+ by_tier = Counter(r["tier"] for r in rows)
	790	+ by_top: Counter[str] = Counter()
	791	+ by_ind: Counter[str] = Counter()
	792	+ for r in rows:
	793	+ for s in r["industries"]:
	794	+ by_ind[s] += 1
	795	+ for t in {top_level_of(s) for s in r["industries"]}:
	796	+ by_top[t] += 1
	797	+ no_ind = sum(1 for r in rows if not r["industries"])
	798	+ public = sum(1 for r in rows if r["public_company"])
	799	+ region = Counter((countries.get(r["country"] or "", {}).get("region") or "other") for r in rows)
	800	+ generated = rows[0]["harvested_at"] if rows else ""
	801	+ intro = (f"Generated by `scripts/seed_wikidata.py` on {generated}. {len(rows)} companies, {public} public, {no_ind} without an "
	802	+ f"industry mapping, {len(by_country)} countries.")
	803	+ files_note = ("Files: one NDJSON per UN region (`wikidata-<region>.ndjson`), one JSON object per line — see `docs/SEEDS.md` for the "
	804	+ "schema, the diversification rules and how to add companies.")
	805	+ lines = [
	806	+ "# Seed registry — Wikidata harvest", "", intro, "", files_note, "",
	807	+ "## Tiers", "", "\| Tier \| Companies \|", "\|---\|---\|",
	808	+ *[f"\| {t} ({ {1: 'global', 2: 'major', 3: 'notable', 4: 'long tail'}[t] }) \| {by_tier[t]} \|" for t in sorted(by_tier)], "",
	809	+ "## Regions", "", "\| Region \| Companies \|", "\|---\|---\|", *[f"\| {k} \| {v} \|" for k, v in region.most_common()], "",
	810	+ "## Countries", "", "\| Code \| Country \| Companies \| Share \|", "\|---\|---\|---\|---\|",
	811	+ [f"\| {c} \| {countries.get(c, {}).get('name', 'unknown')} \| {n} \| {100 n / len(rows):.1f} % \|" for c, n in by_country.most_common()], "",
	812	+ "## Top-level industries (a company counts once per top-level sector)", "", "\| Industry \| Companies \|", "\|---\|---\|",
	813	+ *[f"\| {names.get(k, k)} \| {v} \|" for k, v in by_top.most_common()], "",
	814	+ "## All industries", "", "\| Slug \| Companies \|", "\|---\|---\|", *[f"\| {k} \| {v} \|" for k, v in by_ind.most_common()], "",
	815	+ ]
	816	+ (OUT_DIR / "README.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
	817	+
	818	+
	819	+# ------------------------------------------------------------------------------------------------------------ main
	820	+def main() -> int:
	821	+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
	822	+ ap.add_argument("stage", nargs="?", default="all", choices=["candidates", "select", "details", "assemble", "all"])
	823	+ ap.add_argument("--target", type=int, default=8000, help="companies to select before detail fetching (default 8000)")
	824	+ ap.add_argument("--batch", type=int, default=100, help="QIDs per detail query")
	825	+ ap.add_argument("--refresh", action="store_true", help="ignore the SPARQL cache")
	826	+ ap.add_argument("-v", "--verbose", action="store_true")
	827	+ args = ap.parse_args()
	828	+ logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
	829	+ logging.getLogger("httpx").setLevel(logging.WARNING)
	830	+ DATA_DIR.mkdir(parents=True, exist_ok=True)
	831	+ sp = Sparql(refresh=args.refresh)
	832	+ t0 = time.monotonic()
	833	+ if args.stage in ("candidates", "all"):
	834	+ stage_candidates(sp)
	835	+ if args.stage in ("select", "all"):
	836	+ stage_select(target=args.target)
	837	+ if args.stage in ("details", "all"):
	838	+ stage_details(sp, batch=args.batch)
	839	+ if args.stage in ("assemble", "all"):
	840	+ stage_assemble()
	841	+ log.info("done in %.0fs (%d live queries, %d cached)", time.monotonic() - t0, sp.queries, sp.cached)
	842	+ return 0
	843	+
	844	+
	845	+if __name__ == "__main__":
	846	+ sys.exit(main())

added src/companyatlas/commands/seed.py +100 −0

		`@@ -0,0 +1,100 @@`
	1	+"""`catlas` seed commands: `seed`, `import-companies`, `company-add`, `registry-stats` (docs/SEEDS.md)."""
	2	+from __future__ import annotations
	3	+
	4	+from collections import Counter
	5	+from pathlib import Path
	6	+from typing import Annotated
	7	+
	8	+import typer
	9	+from rich.table import Table
	10	+
	11	+
	12	+def _print_counters(counters: dict[str, int]) -> None:
	13	+ from companyatlas.cli import out
	14	+
	15	+ table = Table(title="seed", show_header=False)
	16	+ for k, v in counters.items():
	17	+ table.add_row(k, str(v))
	18	+ out.print(table)
	19	+
	20	+
	21	+def register(app: typer.Typer) -> None:
	22	+ @app.command("seed")
	23	+ def seed_cmd(
	24	+ no_companies: Annotated[bool, typer.Option("--no-companies", help="only industries + countries")] = False,
	25	+ limit: Annotated[int \| None, typer.Option("--limit", help="stop after N registry rows")] = None,
	26	+ file: Annotated[list[Path] \| None, typer.Option("--file", help="NDJSON file(s) instead of registry/companies/*.ndjson")] = None,
	27	+ ) -> None:
	28	+ """Load the seed registry (idempotent): industries, countries, companies + discover queue."""
	29	+ from companyatlas.cli import run_async
	30	+ from companyatlas.db import transaction
	31	+ from companyatlas.registry.seed import seed
	32	+
	33	+ async def go() -> dict[str, int]:
	34	+ async with transaction() as conn:
	35	+ return await seed(conn, companies=not no_companies, limit=limit, files=file or None)
	36	+
	37	+ _print_counters(run_async(go()))
	38	+
	39	+ @app.command("import-companies")
	40	+ def import_cmd(path: Annotated[Path, typer.Argument(help=".ndjson / .jsonl / .json / .csv with a `website` column")],
	41	+ source: Annotated[str, typer.Option("--source")] = "manual") -> None:
	42	+ """Import companies from a manual file (`website` required; `display_name` derived from the domain when missing)."""
	43	+ from companyatlas.cli import run_async
	44	+ from companyatlas.db import transaction
	45	+ from companyatlas.registry.seed import import_companies, read_rows_file
	46	+
	47	+ rows = read_rows_file(path)
	48	+
	49	+ async def go() -> dict[str, int]:
	50	+ async with transaction() as conn:
	51	+ return await import_companies(conn, rows, source=source)
	52	+
	53	+ _print_counters(run_async(go()))
	54	+
	55	+ @app.command("company-add")
	56	+ def add_cmd(website: Annotated[str, typer.Argument()],
	57	+ name: Annotated[str \| None, typer.Option("--name")] = None,
	58	+ country: Annotated[str \| None, typer.Option("--country", help="ISO-2")] = None,
	59	+ industry: Annotated[list[str] \| None, typer.Option("--industry", help="taxonomy slug (repeatable)")] = None) -> None:
	60	+ """Add one company by website and queue its discovery."""
	61	+ from companyatlas.cli import out, run_async
	62	+ from companyatlas.db import transaction
	63	+ from companyatlas.registry.seed import add_company
	64	+
	65	+ async def go() -> dict:
	66	+ async with transaction() as conn:
	67	+ return await add_company(conn, website, display_name=name, country=country, industries=industry or [])
	68	+
	69	+ row = run_async(go())
	70	+ counters = row.pop("counters", {})
	71	+ out.print(row)
	72	+ _print_counters(counters)
	73	+
	74	+ @app.command("registry-stats")
	75	+ def stats_cmd(top: Annotated[int, typer.Option("--top", help="rows per table")] = 30) -> None:
	76	+ """Counts by country / industry / tier from the registry NDJSON files (no database)."""
	77	+ from companyatlas.cli import out
	78	+ from companyatlas.registry.industries import industry_index, top_level_of
	79	+ from companyatlas.registry.seed import load_countries_file, load_registry_rows, registry_files
	80	+
	81	+ rows = load_registry_rows()
	82	+ countries = {c["code"]: c for c in load_countries_file()}
	83	+ names = {slug: ind.name for slug, ind in industry_index().items()}
	84	+ out.print(f"[bold]{len(rows)}[/] companies in {len(registry_files())} file(s); "
	85	+ f"{sum(1 for r in rows if r.get('public_company'))} public; {sum(1 for r in rows if not r.get('industries'))} without industry")
	86	+
	87	+ def table(title: str, counter: Counter, label=lambda k: k) -> None: # type: ignore[no-untyped-def]
	88	+ t = Table(title=title)
	89	+ t.add_column("key")
	90	+ t.add_column("companies", justify="right")
	91	+ t.add_column("share", justify="right")
	92	+ for k, v in counter.most_common(top):
	93	+ t.add_row(str(label(k)), str(v), f"{100 * v / max(1, len(rows)):.1f} %")
	94	+ out.print(t)
	95	+
	96	+ table("tiers", Counter(r.get("tier") for r in rows))
	97	+ table("regions", Counter(countries.get(r.get("country") or "", {}).get("region") or "unknown" for r in rows))
	98	+ table("countries", Counter(r.get("country") or "??" for r in rows), lambda k: f"{k} {countries.get(k, {}).get('name', '')}".strip())
	99	+ table("top-level industries", Counter(t for r in rows for t in {top_level_of(s) for s in r.get("industries", [])}), lambda k: names.get(k, k))
	100	+ table("industries", Counter(s for r in rows for s in r.get("industries", [])), lambda k: names.get(k, k))

added src/companyatlas/registry/industries.py +202 −0

		`@@ -0,0 +1,202 @@`
	1	+"""Industry taxonomy (registry/industries.yaml) and deterministic mapping of free-text industry labels to our slugs.
	2	+
	3	+`map_industry(labels)` turns Wikidata P452 labels ("software industry", "banking"), SIC descriptions or any free text into an ordered,
	4	+de-duplicated list of taxonomy slugs. Matching is lexical only (no LLM): exact keyword match first, then the longest keyword phrase found
	5	+inside the label on word boundaries. `map_sic(code)` maps a US SIC code (SEC EDGAR) to a slug.
	6	+"""
	7	+from __future__ import annotations
	8	+
	9	+import re
	10	+from dataclasses import dataclass, field
	11	+from functools import lru_cache
	12	+from pathlib import Path
	13	+
	14	+import yaml
	15	+
	16	+REGISTRY_DIR = Path(__file__).resolve().parents[3] / "registry"
	17	+INDUSTRIES_FILE = REGISTRY_DIR / "industries.yaml"
	18	+
	19	+
	20	+@dataclass(frozen=True)
	21	+class Industry:
	22	+ slug: str
	23	+ name: str
	24	+ parent: str \| None
	25	+ description: str
	26	+ keywords: tuple[str, ...]
	27	+ sort_order: int = 100
	28	+ children: tuple[str, ...] = field(default_factory=tuple)
	29	+
	30	+
	31	+def _norm(text: str) -> str:
	32	+ text = text.lower().replace("_", " ").replace("&", " and ").replace("/", " ")
	33	+ text = re.sub(r"[^\w\s\-']", " ", text)
	34	+ return re.sub(r"\s+", " ", text).strip()
	35	+
	36	+
	37	+@lru_cache
	38	+def load_industries(path: Path = INDUSTRIES_FILE) -> tuple[Industry, ...]:
	39	+ raw = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
	40	+ rows = raw.get("industries") or []
	41	+ children: dict[str, list[str]] = {}
	42	+ for r in rows:
	43	+ if r.get("parent"):
	44	+ children.setdefault(r["parent"], []).append(r["slug"])
	45	+ out: list[Industry] = []
	46	+ for i, r in enumerate(rows):
	47	+ out.append(Industry(slug=r["slug"], name=r["name"], parent=r.get("parent"), description=(r.get("description") or "").strip(),
	48	+ keywords=tuple(_norm(k) for k in (r.get("keywords") or []) if k), sort_order=(i + 1) * 10,
	49	+ children=tuple(children.get(r["slug"], ()))))
	50	+ return tuple(out)
	51	+
	52	+
	53	+@lru_cache
	54	+def industry_index() -> dict[str, Industry]:
	55	+ return {i.slug: i for i in load_industries()}
	56	+
	57	+
	58	+def top_level_slugs() -> list[str]:
	59	+ return [i.slug for i in load_industries() if i.parent is None]
	60	+
	61	+
	62	+def top_level_of(slug: str) -> str:
	63	+ ind = industry_index().get(slug)
	64	+ if ind is None:
	65	+ return slug
	66	+ return ind.parent or ind.slug
	67	+
	68	+
	69	+def is_valid_slug(slug: str) -> bool:
	70	+ return slug in industry_index()
	71	+
	72	+
	73	+@lru_cache
	74	+def _keyword_table() -> tuple[dict[str, tuple[str, ...]], tuple[tuple[str, re.Pattern[str], tuple[str, ...]], ...]]:
	75	+ """(exact keyword → slugs, [(keyword, boundary regex, slugs)] sorted by keyword length desc)."""
	76	+ exact: dict[str, list[str]] = {}
	77	+ for ind in load_industries():
	78	+ for kw in ind.keywords:
	79	+ exact.setdefault(kw, [])
	80	+ if ind.slug not in exact[kw]:
	81	+ exact[kw].append(ind.slug)
	82	+ phrases = []
	83	+ for kw, slugs in sorted(exact.items(), key=lambda kv: (-len(kv[0]), kv[0])):
	84	+ pat = re.compile(r"(?<![\w\-])" + re.escape(kw) + r"(?![\w\-])")
	85	+ phrases.append((kw, pat, tuple(slugs)))
	86	+ return {k: tuple(v) for k, v in exact.items()}, tuple(phrases)
	87	+
	88	+
	89	+def map_label(label: str) -> list[str]:
	90	+ """Slugs for a single label. Exact keyword match, else the longest keyword phrase(s) contained in the label."""
	91	+ text = _norm(label)
	92	+ if not text:
	93	+ return []
	94	+ exact, phrases = _keyword_table()
	95	+ if text in exact:
	96	+ return list(exact[text])
	97	+ for variant in (text.removesuffix(" industry"), text.removesuffix(" company"), text.removesuffix(" sector"), text.removesuffix("s")):
	98	+ if variant != text and variant in exact:
	99	+ return list(exact[variant])
	100	+ best_len = 0
	101	+ found: list[str] = []
	102	+ for kw, pat, slugs in phrases:
	103	+ if best_len and len(kw) < best_len:
	104	+ break
	105	+ if pat.search(text):
	106	+ best_len = len(kw)
	107	+ for s in slugs:
	108	+ if s not in found:
	109	+ found.append(s)
	110	+ return found
	111	+
	112	+
	113	+def map_industry(labels: str \| list[str] \| tuple[str, ...] \| None, *, limit: int = 4) -> list[str]:
	114	+ """Ordered, de-duplicated taxonomy slugs for one or many labels (first label's matches come first)."""
	115	+ if not labels:
	116	+ return []
	117	+ if isinstance(labels, str):
	118	+ labels = [labels]
	119	+ out: list[str] = []
	120	+ for label in labels:
	121	+ for slug in map_label(label):
	122	+ if slug not in out:
	123	+ out.append(slug)
	124	+ return out[:limit]
	125	+
	126	+
	127	+# SIC (Standard Industrial Classification, as used by SEC EDGAR) → slug. Specific codes first, then ranges (inclusive).
	128	+_SIC_EXACT: dict[int, str] = {
	129	+ 1311: "oil-gas", 1381: "oil-gas", 1382: "oil-gas", 1389: "oil-gas", 2111: "consumer-goods", 2834: "pharmaceuticals", 2835: "pharmaceuticals",
	130	+ 2836: "biotechnology", 2833: "pharmaceuticals", 2911: "oil-gas", 3571: "technology", 3572: "cloud-infrastructure", 3576: "cloud-infrastructure",
	131	+ 3577: "technology", 3578: "technology", 3661: "telecommunications", 3663: "telecommunications", 3669: "telecommunications",
	132	+ 3674: "semiconductors", 3672: "semiconductors", 3679: "technology", 3630: "consumer-goods", 3634: "consumer-goods", 3651: "consumer-goods",
	133	+ 3711: "automotive", 3713: "automotive", 3714: "automotive", 3715: "automotive", 3716: "automotive", 3751: "automotive",
	134	+ 3720: "aerospace-defense", 3721: "aerospace-defense", 3724: "aerospace-defense", 3728: "aerospace-defense", 3760: "aerospace-defense",
	135	+ 3812: "aerospace-defense", 3730: "manufacturing", 3743: "transportation", 3841: "medical-devices", 3842: "medical-devices",
	136	+ 3843: "medical-devices", 3844: "medical-devices", 3845: "medical-devices", 3851: "medical-devices", 3942: "consumer-goods",
	137	+ 3944: "consumer-goods", 3949: "consumer-goods", 4011: "transportation", 4013: "transportation", 4210: "logistics", 4213: "logistics",
	138	+ 4400: "shipping", 4412: "shipping", 4512: "airlines", 4513: "logistics", 4522: "airlines", 4610: "oil-gas", 4700: "logistics",
	139	+ 4731: "logistics", 4812: "telecommunications", 4813: "telecommunications", 4822: "telecommunications", 4832: "media", 4833: "media",
	140	+ 4841: "media", 4899: "telecommunications", 4911: "utilities", 4922: "utilities", 4923: "utilities", 4924: "utilities", 4931: "utilities",
	141	+ 4932: "utilities", 4941: "utilities", 4950: "utilities", 4953: "utilities", 4955: "utilities", 4991: "renewables", 5812: "food-beverage",
	142	+ 5912: "retail", 5961: "e-commerce", 6021: "banking", 6022: "banking", 6029: "banking", 6035: "banking", 6036: "banking",
	143	+ 6099: "payments", 6111: "financial-services", 6141: "financial-services", 6153: "financial-services", 6159: "financial-services",
	144	+ 6162: "financial-services", 6163: "financial-services", 6172: "financial-services", 6189: "financial-services", 6199: "financial-services",
	145	+ 6200: "financial-services", 6211: "financial-services", 6221: "financial-services", 6282: "asset-management", 6311: "insurance",
	146	+ 6321: "insurance", 6324: "insurance", 6331: "insurance", 6351: "insurance", 6361: "insurance", 6399: "insurance", 6411: "insurance",
	147	+ 6500: "real-estate", 6510: "real-estate", 6512: "real-estate", 6513: "real-estate", 6519: "real-estate", 6531: "real-estate",
	148	+ 6552: "real-estate", 6770: "financial-services", 6792: "financial-services", 6794: "professional-services", 6795: "mining",
	149	+ 6798: "real-estate", 6799: "asset-management", 7011: "hospitality", 7200: "professional-services", 7310: "media", 7311: "media",
	150	+ 7320: "professional-services", 7330: "professional-services", 7331: "media", 7350: "professional-services", 7359: "professional-services",
	151	+ 7361: "professional-services", 7363: "professional-services", 7370: "software", 7371: "software", 7372: "software", 7373: "software",
	152	+ 7374: "cloud-infrastructure", 7377: "cloud-infrastructure", 7380: "professional-services", 7381: "professional-services",
	153	+ 7384: "professional-services", 7385: "telecommunications", 7389: "professional-services", 7500: "automotive", 7510: "automotive",
	154	+ 7812: "entertainment", 7819: "entertainment", 7822: "entertainment", 7829: "entertainment", 7830: "entertainment", 7841: "entertainment",
	155	+ 7900: "entertainment", 7948: "entertainment", 7990: "entertainment", 7997: "hospitality", 8000: "healthcare", 8011: "healthcare",
	156	+ 8050: "healthcare", 8051: "healthcare", 8060: "healthcare", 8062: "healthcare", 8071: "healthcare", 8082: "healthcare", 8090: "healthcare",
	157	+ 8093: "healthcare", 8111: "professional-services", 8200: "education", 8300: "healthcare", 8351: "education", 8600: "professional-services",
	158	+ 8700: "professional-services", 8711: "professional-services", 8721: "professional-services", 8731: "biotechnology",
	159	+ 8734: "professional-services", 8741: "professional-services", 8742: "consulting", 8744: "professional-services", 8748: "consulting",
	160	+ 8880: "financial-services", 8888: "financial-services", 8900: "professional-services", 9995: "financial-services",
	161	+}
	162	+_SIC_RANGES: tuple[tuple[int, int, str], ...] = (
	163	+ (100, 999, "agriculture"), (1000, 1299, "mining"), (1300, 1399, "oil-gas"), (1400, 1499, "mining"), (1500, 1799, "construction"),
	164	+ (2000, 2099, "food-beverage"), (2100, 2199, "consumer-goods"), (2200, 2399, "apparel"), (2400, 2499, "materials"), (2500, 2599, "consumer-goods"),
	165	+ (2600, 2699, "materials"), (2700, 2799, "media"), (2800, 2829, "chemicals"), (2830, 2839, "pharmaceuticals"), (2840, 2899, "chemicals"),
	166	+ (2900, 2999, "oil-gas"), (3000, 3099, "chemicals"), (3100, 3199, "apparel"), (3200, 3299, "materials"), (3300, 3399, "mining"),
	167	+ (3400, 3499, "manufacturing"), (3500, 3569, "industrial-machinery"), (3570, 3579, "technology"), (3580, 3599, "industrial-machinery"),
	168	+ (3600, 3629, "manufacturing"), (3630, 3639, "consumer-goods"), (3640, 3659, "manufacturing"), (3660, 3669, "telecommunications"),
	169	+ (3670, 3679, "semiconductors"), (3680, 3699, "technology"), (3700, 3719, "automotive"), (3720, 3729, "aerospace-defense"),
	170	+ (3730, 3739, "manufacturing"), (3740, 3749, "transportation"), (3750, 3759, "automotive"), (3760, 3769, "aerospace-defense"),
	171	+ (3770, 3799, "manufacturing"), (3800, 3839, "technology"), (3840, 3859, "medical-devices"), (3860, 3899, "technology"),
	172	+ (3900, 3999, "consumer-goods"), (4000, 4099, "transportation"), (4100, 4199, "transportation"), (4200, 4299, "logistics"),
	173	+ (4300, 4399, "logistics"), (4400, 4499, "shipping"), (4500, 4599, "airlines"), (4600, 4699, "oil-gas"), (4700, 4799, "logistics"),
	174	+ (4800, 4829, "telecommunications"), (4830, 4849, "media"), (4850, 4899, "telecommunications"), (4900, 4999, "utilities"),
	175	+ (5000, 5199, "retail"), (5200, 5799, "retail"), (5800, 5899, "food-beverage"), (5900, 5999, "retail"), (6000, 6099, "banking"),
	176	+ (6100, 6199, "financial-services"), (6200, 6299, "financial-services"), (6300, 6499, "insurance"), (6500, 6599, "real-estate"),
	177	+ (6700, 6799, "asset-management"), (7000, 7099, "hospitality"), (7200, 7299, "professional-services"), (7300, 7369, "professional-services"),
	178	+ (7370, 7379, "software"), (7380, 7399, "professional-services"), (7500, 7599, "automotive"), (7600, 7699, "professional-services"),
	179	+ (7800, 7899, "entertainment"), (7900, 7999, "entertainment"), (8000, 8099, "healthcare"), (8100, 8199, "professional-services"),
	180	+ (8200, 8299, "education"), (8300, 8399, "healthcare"), (8400, 8499, "entertainment"), (8600, 8699, "professional-services"),
	181	+ (8700, 8799, "professional-services"), (8800, 8999, "professional-services"),
	182	+)
	183	+
	184	+
	185	+def map_sic(code: int \| str \| None) -> str \| None:
	186	+ """US SIC code → taxonomy slug (None when unknown/blank)."""
	187	+ if code in (None, ""):
	188	+ return None
	189	+ try:
	190	+ n = int(str(code).strip())
	191	+ except ValueError:
	192	+ return None
	193	+ if n in _SIC_EXACT:
	194	+ return _SIC_EXACT[n]
	195	+ for lo, hi, slug in _SIC_RANGES:
	196	+ if lo <= n <= hi:
	197	+ return slug
	198	+ return None
	199	+
	200	+
	201	+__all__ = ["INDUSTRIES_FILE", "REGISTRY_DIR", "Industry", "industry_index", "is_valid_slug", "load_industries", "map_industry",
	202	+ "map_label", "map_sic", "top_level_of", "top_level_slugs"]

added src/companyatlas/registry/seed.py +367 −0

		`@@ -0,0 +1,367 @@`
	1	+"""Idempotent seed loader: registry files → `industries`, `countries`, `companies` (+ aliases, domains, relationships, discover queue).
	2	+
	3	+Contract with the crawl core (docs/ARCHITECTURE.md, "Seeds → Crawl"): new companies get `onboarding_status='pending'` and a
	4	+`queue_jobs(kind='discover', key='discover:<company_id>')` row. Re-running never overwrites a non-null value except
	5	+`importance`, `tier` and the provenance in `source_meta` (spec: factual provenance, historical-first).
	6	+"""
	7	+from __future__ import annotations
	8	+
	9	+import csv
	10	+import json
	11	+import logging
	12	+from collections.abc import Iterable
	13	+from datetime import UTC, datetime
	14	+from pathlib import Path
	15	+from typing import Any
	16	+
	17	+from sqlalchemy.ext.asyncio import AsyncConnection
	18	+
	19	+from companyatlas import ids
	20	+from companyatlas.db import execute, execute_many, fetch_all, jsonb
	21	+from companyatlas.registry.industries import REGISTRY_DIR, is_valid_slug, load_industries
	22	+from companyatlas.urls import registrable_domain
	23	+
	24	+log = logging.getLogger(__name__)
	25	+
	26	+COMPANIES_DIR = REGISTRY_DIR / "companies"
	27	+COUNTRIES_FILE = REGISTRY_DIR / "countries.csv"
	28	+DEFAULT_IMPORTANCE = 0.2
	29	+DEFAULT_TIER = 4
	30	+GENERIC_TLD_NAMES = {"www", "web", "site", "home", "online", "official"}
	31	+
	32	+
	33	+# ------------------------------------------------------------------------------------------------------------ registry files
	34	+def registry_files() -> list[Path]:
	35	+ return sorted(COMPANIES_DIR.glob("*.ndjson"))
	36	+
	37	+
	38	+def load_registry_rows(files: Iterable[Path] \| None = None) -> list[dict[str, Any]]:
	39	+ rows: list[dict[str, Any]] = []
	40	+ for path in files or registry_files():
	41	+ with Path(path).open(encoding="utf-8") as f:
	42	+ for n, line in enumerate(f, 1):
	43	+ line = line.strip()
	44	+ if not line:
	45	+ continue
	46	+ try:
	47	+ rows.append(json.loads(line))
	48	+ except json.JSONDecodeError as e:
	49	+ raise ValueError(f"{path}:{n}: invalid JSON ({e})") from e
	50	+ return rows
	51	+
	52	+
	53	+def load_countries_file(path: Path = COUNTRIES_FILE) -> list[dict[str, Any]]:
	54	+ with path.open(encoding="utf-8") as f:
	55	+ out = []
	56	+ for r in csv.DictReader(f):
	57	+ out.append({"code": r["code"].strip().upper(), "name": r["name"], "region": r.get("region") or None,
	58	+ "subregion": r.get("subregion") or None, "lat": float(r["lat"]) if r.get("lat") else None,
	59	+ "lon": float(r["lon"]) if r.get("lon") else None})
	60	+ return out
	61	+
	62	+
	63	+def read_rows_file(path: Path) -> list[dict[str, Any]]:
	64	+ """Manual import: `.ndjson`/`.jsonl`/`.json` (list) or `.csv` with a header (`website` required)."""
	65	+ path = Path(path)
	66	+ if path.suffix.lower() == ".csv":
	67	+ with path.open(encoding="utf-8-sig") as f:
	68	+ rows = []
	69	+ for r in csv.DictReader(f):
	70	+ row = {k.strip(): (v.strip() if isinstance(v, str) else v) for k, v in r.items() if k}
	71	+ for key in ("industries", "aliases"):
	72	+ if isinstance(row.get(key), str):
	73	+ row[key] = [x.strip() for x in row[key].replace(";", ",").split(",") if x.strip()]
	74	+ rows.append(row)
	75	+ return rows
	76	+ if path.suffix.lower() == ".json":
	77	+ data = json.loads(path.read_text(encoding="utf-8"))
	78	+ return list(data if isinstance(data, list) else data.get("companies", []))
	79	+ return load_registry_rows([path])
	80	+
	81	+
	82	+# ------------------------------------------------------------------------------------------------------------ normalisation
	83	+def name_from_domain(domain: str) -> str:
	84	+ label = domain.split(".")[0]
	85	+ if label in GENERIC_TLD_NAMES and domain.count(".") >= 2:
	86	+ label = domain.split(".")[1]
	87	+ return label.replace("-", " ").replace("_", " ").title()
	88	+
	89	+
	90	+def normalise_row(row: dict[str, Any], *, source: str) -> dict[str, Any] \| None:
	91	+ website = (row.get("website") or "").strip()
	92	+ if not website:
	93	+ return None
	94	+ if "://" not in website:
	95	+ website = "https://" + website
	96	+ if not website.startswith(("http://", "https://")):
	97	+ return None
	98	+ domain = (row.get("canonical_domain") or "").strip().lower() or registrable_domain(website)
	99	+ if not domain or "." not in domain:
	100	+ return None
	101	+ display = (row.get("display_name") or "").strip() or name_from_domain(domain)
	102	+ industries = [s for s in (row.get("industries") or []) if isinstance(s, str) and is_valid_slug(s)]
	103	+ if row.get("industry") and is_valid_slug(row["industry"]) and row["industry"] not in industries:
	104	+ industries.insert(0, row["industry"])
	105	+ country = (row.get("country") or "").strip().upper() or None
	106	+ parent = row.get("parent") if isinstance(row.get("parent"), dict) else None
	107	+ try:
	108	+ importance = float(row.get("importance", DEFAULT_IMPORTANCE))
	109	+ except (TypeError, ValueError):
	110	+ importance = DEFAULT_IMPORTANCE
	111	+ try:
	112	+ tier = int(row.get("tier", DEFAULT_TIER))
	113	+ except (TypeError, ValueError):
	114	+ tier = DEFAULT_TIER
	115	+ meta = {
	116	+ "source": row.get("source") or source,
	117	+ "harvested_at": row.get("harvested_at"),
	118	+ "sitelinks": row.get("sitelinks"),
	119	+ "industry_labels": row.get("industry_labels") or [],
	120	+ "parent_wikidata_id": parent.get("wikidata_id") if parent else None,
	121	+ "parent_name": parent.get("name") if parent else None,
	122	+ "notes": row.get("notes") or [],
	123	+ "seeded_at": datetime.now(UTC).replace(microsecond=0).isoformat(),
	124	+ }
	125	+ meta = {k: v for k, v in meta.items() if v not in (None, [], "")}
	126	+ return {
	127	+ "wikidata_id": (row.get("wikidata_id") or None), "display_name": display[:200], "legal_name": (row.get("legal_name") or None),
	128	+ "aliases": [a for a in (row.get("aliases") or []) if isinstance(a, str) and a.strip()][:12], "website": website,
	129	+ "canonical_domain": domain, "country": country, "hq_city": row.get("hq_city") or None, "hq_region": row.get("hq_region") or None,
	130	+ "industries": industries, "industry_primary": industries[0] if industries else None, "founded_year": row.get("founded_year"),
	131	+ "employees": row.get("employees"), "public_company": bool(row.get("public_company")), "ticker": row.get("ticker") or None,
	132	+ "exchange": row.get("exchange") or None, "lei": row.get("lei") or None, "sec_cik": str(row["sec_cik"]) if row.get("sec_cik") else None,
	133	+ "logo_url": row.get("logo_url") or None, "description": (row.get("description") or None), "importance": max(0.0, min(1.0, importance)),
	134	+ "tier": min(4, max(1, tier)), "company_type": "public" if row.get("public_company") else None, "source_meta": meta,
	135	+ "parent_wikidata_id": parent.get("wikidata_id") if parent else None,
	136	+ }
	137	+
	138	+
	139	+def unique_slug(base: str, country: str \| None, taken: set[str]) -> str:
	140	+ if base not in taken:
	141	+ return base
	142	+ if country and f"{base}-{country.lower()}" not in taken:
	143	+ return f"{base}-{country.lower()}"
	144	+ n = 2
	145	+ while f"{base}-{n}" in taken:
	146	+ n += 1
	147	+ return f"{base}-{n}"
	148	+
	149	+
	150	+# ------------------------------------------------------------------------------------------------------------ reference tables
	151	+async def upsert_industries(conn: AsyncConnection) -> int:
	152	+ rows = [{"slug": i.slug, "name": i.name, "parent": i.parent, "description": i.description, "keywords": list(i.keywords),
	153	+ "sort_order": i.sort_order} for i in load_industries()]
	154	+ # Parents first (FK on parent_slug); insert parent_slug in a second pass so ordering inside the file does not matter.
	155	+ await execute_many(conn, """
	156	+ insert into industries (slug, name, description, keywords, sort_order) values (:slug, :name, :description, cast(:keywords as text[]), :sort_order)
	157	+ on conflict (slug) do update set name = excluded.name, description = excluded.description, keywords = excluded.keywords,
	158	+ sort_order = excluded.sort_order""", rows)
	159	+ await execute_many(conn, "update industries set parent_slug = :parent where slug = :slug", [r for r in rows if r["parent"]])
	160	+ return len(rows)
	161	+
	162	+
	163	+async def upsert_countries(conn: AsyncConnection) -> int:
	164	+ rows = load_countries_file()
	165	+ await execute_many(conn, """
	166	+ insert into countries (code, name, region, subregion, lat, lon) values (:code, :name, :region, :subregion, :lat, :lon)
	167	+ on conflict (code) do update set name = excluded.name, region = excluded.region, subregion = excluded.subregion,
	168	+ lat = excluded.lat, lon = excluded.lon""", rows)
	169	+ return len(rows)
	170	+
	171	+
	172	+# ------------------------------------------------------------------------------------------------------------ companies
	173	+class _Index:
	174	+ """In-memory view of existing companies for one loader run (avoids a lookup per row)."""
	175	+
	176	+ def __init__(self, by_wikidata: dict[str, str], by_domain: dict[str, str], slugs: set[str], countries: set[str]) -> None:
	177	+ self.by_wikidata = by_wikidata
	178	+ self.by_domain = by_domain
	179	+ self.slugs = slugs
	180	+ self.countries = countries
	181	+
	182	+ @classmethod
	183	+ async def load(cls, conn: AsyncConnection) -> _Index:
	184	+ rows = await fetch_all(conn, "select id, slug, canonical_domain, wikidata_id from companies")
	185	+ countries = {r["code"] for r in await fetch_all(conn, "select code from countries")}
	186	+ return cls({r["wikidata_id"]: r["id"] for r in rows if r["wikidata_id"]}, {r["canonical_domain"]: r["id"] for r in rows},
	187	+ {r["slug"] for r in rows}, countries)
	188	+
	189	+
	190	+def _alias_rows(company_id: str, row: dict[str, Any], source: str) -> list[dict[str, Any]]:
	191	+ seen: set[str] = set()
	192	+ out: list[dict[str, Any]] = []
	193	+ for alias, kind in ([(row["display_name"], "brand"), (row.get("legal_name"), "legal"), (row.get("ticker"), "ticker")]
	194	+ + [(a, "alias") for a in row.get("aliases", [])]):
	195	+ if not alias:
	196	+ continue
	197	+ norm = ids.normalize_alias(alias)
	198	+ if not norm or norm in seen:
	199	+ continue
	200	+ seen.add(norm)
	201	+ out.append({"company_id": company_id, "alias": alias[:200], "alias_norm": norm[:200], "kind": kind, "source": source})
	202	+ return out
	203	+
	204	+
	205	+async def _insert_company(conn: AsyncConnection, row: dict[str, Any], idx: _Index) -> str:
	206	+ company_id = ids.new_id("company")
	207	+ slug = unique_slug(ids.slugify(row["display_name"]), row["country"], idx.slugs)
	208	+ idx.slugs.add(slug)
	209	+ await execute(conn, """
	210	+ insert into companies (id, slug, legal_name, display_name, canonical_domain, website, description, industries, industry_primary, country,
	211	+ hq_city, hq_region, founded_year, company_type, public_company, ticker, exchange, employees, wikidata_id, lei,
	212	+ sec_cik, logo_url, onboarding_status, importance, tier, source_meta)
	213	+ values (:id, :slug, :legal_name, :display_name, :canonical_domain, :website, :description, cast(:industries as text[]), :industry_primary,
	214	+ :country, :hq_city, :hq_region, :founded_year, :company_type, :public_company, :ticker, :exchange, :employees, :wikidata_id, :lei,
	215	+ :sec_cik, :logo_url, 'pending', :importance, :tier, cast(:source_meta as jsonb))""",
	216	+ id=company_id, slug=slug, legal_name=row["legal_name"], display_name=row["display_name"], canonical_domain=row["canonical_domain"],
	217	+ website=row["website"], description=row["description"], industries=row["industries"], industry_primary=row["industry_primary"],
	218	+ country=row["country"], hq_city=row["hq_city"], hq_region=row["hq_region"], founded_year=row["founded_year"],
	219	+ company_type=row["company_type"], public_company=row["public_company"], ticker=row["ticker"], exchange=row["exchange"],
	220	+ employees=row["employees"], wikidata_id=row["wikidata_id"], lei=row["lei"], sec_cik=row["sec_cik"], logo_url=row["logo_url"],
	221	+ importance=row["importance"], tier=row["tier"], source_meta=jsonb(row["source_meta"]))
	222	+ idx.by_domain[row["canonical_domain"]] = company_id
	223	+ if row["wikidata_id"]:
	224	+ idx.by_wikidata[row["wikidata_id"]] = company_id
	225	+ return company_id
	226	+
	227	+
	228	+async def _update_company(conn: AsyncConnection, company_id: str, row: dict[str, Any]) -> None:
	229	+ """Fill only null columns; refresh importance/tier and merge provenance. Never touches names, website or history."""
	230	+ await execute(conn, """
	231	+ update companies set
	232	+ legal_name = coalesce(legal_name, :legal_name), description = coalesce(description, :description),
	233	+ industries = case when cardinality(industries) = 0 then cast(:industries as text[]) else industries end,
	234	+ industry_primary = coalesce(industry_primary, :industry_primary), country = coalesce(country, :country),
	235	+ hq_city = coalesce(hq_city, :hq_city), hq_region = coalesce(hq_region, :hq_region), founded_year = coalesce(founded_year, :founded_year),
	236	+ company_type = coalesce(company_type, :company_type), public_company = public_company or :public_company,
	237	+ ticker = coalesce(ticker, :ticker), exchange = coalesce(exchange, :exchange), employees = coalesce(employees, :employees),
	238	+ wikidata_id = coalesce(wikidata_id, :wikidata_id), lei = coalesce(lei, :lei), sec_cik = coalesce(sec_cik, :sec_cik),
	239	+ logo_url = coalesce(logo_url, :logo_url), importance = :importance, tier = :tier,
	240	+ source_meta = source_meta \|\| cast(:source_meta as jsonb), updated_at = now()
	241	+ where id = :id""",
	242	+ id=company_id, legal_name=row["legal_name"], description=row["description"], industries=row["industries"],
	243	+ industry_primary=row["industry_primary"], country=row["country"], hq_city=row["hq_city"], hq_region=row["hq_region"],
	244	+ founded_year=row["founded_year"], company_type=row["company_type"], public_company=row["public_company"], ticker=row["ticker"],
	245	+ exchange=row["exchange"], employees=row["employees"], wikidata_id=row["wikidata_id"], lei=row["lei"], sec_cik=row["sec_cik"],
	246	+ logo_url=row["logo_url"], importance=row["importance"], tier=row["tier"], source_meta=jsonb(row["source_meta"]))
	247	+
	248	+
	249	+async def _link_parents(conn: AsyncConnection, pairs: list[tuple[str, str]], source: str) -> int:
	250	+ """(child_id, parent_id) → PARENT_OF + SUBSIDIARY_OF when missing. Confidence 0.8 (Wikidata P749, not verified on the web)."""
	251	+ if not pairs:
	252	+ return 0
	253	+ existing = await fetch_all(conn, """
	254	+ select from_company_id, to_company_id, kind from company_relationships
	255	+ where kind in ('PARENT_OF', 'SUBSIDIARY_OF') and from_company_id = any(cast(:ids as text[]))""",
	256	+ ids=sorted({c for c, _ in pairs} \| {p for _, p in pairs}))
	257	+ have = {(r["from_company_id"], r["to_company_id"], r["kind"]) for r in existing}
	258	+ rows = []
	259	+ for child, parent in pairs:
	260	+ for frm, to, kind in ((parent, child, "PARENT_OF"), (child, parent, "SUBSIDIARY_OF")):
	261	+ if (frm, to, kind) not in have:
	262	+ have.add((frm, to, kind))
	263	+ rows.append({"id": ids.new_id("relationship"), "frm": frm, "to": to, "kind": kind,
	264	+ "prov": jsonb({"source": source, "property": "P749"})})
	265	+ await execute_many(conn, """
	266	+ insert into company_relationships (id, from_company_id, to_company_id, kind, confidence, provenance)
	267	+ values (:id, :frm, :to, :kind, 0.8, cast(:prov as jsonb))""", rows)
	268	+ return len(rows)
	269	+
	270	+
	271	+async def upsert_companies(conn: AsyncConnection, rows: Iterable[dict[str, Any]], *, source: str = "registry", limit: int \| None = None,
	272	+ enqueue: bool = True) -> dict[str, int]:
	273	+ idx = await _Index.load(conn)
	274	+ counters: dict[str, int] = {"companies_seen": 0, "companies_new": 0, "companies_updated": 0, "companies_skipped": 0, "aliases": 0,
	275	+ "domains": 0, "relationships": 0, "queue_jobs": 0}
	276	+ parents: list[tuple[str, str]] = [] # (child_id, parent_wikidata_id)
	277	+ new_ids: list[tuple[str, float]] = []
	278	+ seen_domains: set[str] = set()
	279	+ for raw in rows:
	280	+ if limit is not None and counters["companies_seen"] >= limit:
	281	+ break
	282	+ counters["companies_seen"] += 1
	283	+ row = normalise_row(raw, source=source)
	284	+ if row is None or row["canonical_domain"] in seen_domains:
	285	+ counters["companies_skipped"] += 1
	286	+ continue
	287	+ seen_domains.add(row["canonical_domain"])
	288	+ if row["country"] and row["country"] not in idx.countries:
	289	+ row["source_meta"]["unknown_country"] = row["country"]
	290	+ row["country"] = None
	291	+ company_id = idx.by_wikidata.get(row["wikidata_id"] or "") or idx.by_domain.get(row["canonical_domain"])
	292	+ if company_id:
	293	+ await _update_company(conn, company_id, row)
	294	+ counters["companies_updated"] += 1
	295	+ else:
	296	+ company_id = await _insert_company(conn, row, idx)
	297	+ counters["companies_new"] += 1
	298	+ new_ids.append((company_id, row["importance"]))
	299	+ aliases = _alias_rows(company_id, row, source)
	300	+ await execute_many(conn, """
	301	+ insert into company_aliases (company_id, alias, alias_norm, kind, source) values (:company_id, :alias, :alias_norm, :kind, :source)
	302	+ on conflict (company_id, alias_norm) do nothing""", aliases)
	303	+ counters["aliases"] += len(aliases)
	304	+ await execute(conn, """
	305	+ insert into domains (id, company_id, domain, kind) values (:id, :company_id, :domain, 'primary')
	306	+ on conflict (domain, company_id) do nothing""", id=ids.new_id("domain"), company_id=company_id, domain=row["canonical_domain"])
	307	+ counters["domains"] += 1
	308	+ if row["parent_wikidata_id"] and row["parent_wikidata_id"] != row["wikidata_id"]:
	309	+ parents.append((company_id, row["parent_wikidata_id"]))
	310	+ pairs = [(child, idx.by_wikidata[pq]) for child, pq in parents if pq in idx.by_wikidata and idx.by_wikidata[pq] != child]
	311	+ counters["relationships"] = await _link_parents(conn, pairs, source)
	312	+ if enqueue:
	313	+ counters["queue_jobs"] = await enqueue_discovery(conn, [cid for cid, _ in new_ids])
	314	+ return counters
	315	+
	316	+
	317	+async def enqueue_discovery(conn: AsyncConnection, company_ids: list[str] \| None = None) -> int:
	318	+ """`discover` jobs for pending companies (all pending when `company_ids` is None). Idempotent on the job key."""
	319	+ if company_ids is None:
	320	+ pending = await fetch_all(conn, "select id, importance from companies where onboarding_status = 'pending'")
	321	+ elif company_ids:
	322	+ pending = await fetch_all(conn, "select id, importance from companies where id = any(cast(:ids as text[])) and onboarding_status = 'pending'",
	323	+ ids=company_ids)
	324	+ else:
	325	+ return 0
	326	+ rows = [{"id": ids.new_id("queue_job"), "key": f"discover:{r['id']}", "priority": float(r["importance"]),
	327	+ "payload": jsonb({"company_id": r["id"]})} for r in pending]
	328	+ await execute_many(conn, """
	329	+ insert into queue_jobs (id, kind, key, payload, priority) values (:id, 'discover', :key, cast(:payload as jsonb), :priority)
	330	+ on conflict (key) do nothing""", rows)
	331	+ return len(rows)
	332	+
	333	+
	334	+# ------------------------------------------------------------------------------------------------------------ public API
	335	+async def seed(conn: AsyncConnection, *, companies: bool = True, limit: int \| None = None, files: Iterable[Path] \| None = None) -> dict[str, int]:
	336	+ """Upsert industries + countries, then (optionally) every company of the registry NDJSON files. Idempotent."""
	337	+ started = datetime.now(UTC)
	338	+ counters: dict[str, int] = {"industries": await upsert_industries(conn), "countries": await upsert_countries(conn)}
	339	+ if companies:
	340	+ rows = load_registry_rows(files)
	341	+ counters.update(await upsert_companies(conn, rows, source="wikidata", limit=limit))
	342	+ await execute(conn, """
	343	+ insert into settings_kv (key, value, updated_at) values ('seed:last_run', cast(:v as jsonb), now())
	344	+ on conflict (key) do update set value = excluded.value, updated_at = now()""",
	345	+ v=jsonb({"started_at": started.isoformat(), "finished_at": datetime.now(UTC).isoformat(), "counters": counters}))
	346	+ log.info("seed done", extra=counters)
	347	+ return counters
	348	+
	349	+
	350	+async def import_companies(conn: AsyncConnection, rows: Iterable[dict[str, Any]], *, source: str = "manual") -> dict[str, int]:
	351	+ """Manual CSV/NDJSON rows (`website` required; `display_name` derived from the domain when missing)."""
	352	+ await upsert_industries(conn)
	353	+ await upsert_countries(conn)
	354	+ return await upsert_companies(conn, rows, source=source)
	355	+
	356	+
	357	+async def add_company(conn: AsyncConnection, website: str, **fields: Any) -> dict[str, Any]:
	358	+ """Add (or top up) a single company by website. Returns the stored row (id, slug, canonical_domain, onboarding_status)."""
	359	+ counters = await import_companies(conn, [{"website": website, **fields}], source=fields.pop("source", "manual"))
	360	+ domain = registrable_domain(website if "://" in website else "https://" + website)
	361	+ row = await fetch_all(conn, "select id, slug, display_name, canonical_domain, onboarding_status from companies where canonical_domain = :d",
	362	+ d=domain)
	363	+ return {**(row[0] if row else {}), "counters": counters}
	364	+
	365	+
	366	+__all__ = ["COMPANIES_DIR", "add_company", "enqueue_discovery", "import_companies", "load_registry_rows", "normalise_row", "read_rows_file",
	367	+ "registry_files", "seed", "unique_slug", "upsert_companies", "upsert_countries", "upsert_industries"]

added tests/test_industry_map.py +115 −0

		`@@ -0,0 +1,115 @@`
	1	+"""Industry taxonomy file + deterministic label → slug mapping."""
	2	+from __future__ import annotations
	3	+
	4	+import pytest
	5	+
	6	+from companyatlas.registry.industries import (
	7	+ industry_index,
	8	+ is_valid_slug,
	9	+ load_industries,
	10	+ map_industry,
	11	+ map_label,
	12	+ map_sic,
	13	+ top_level_of,
	14	+ top_level_slugs,
	15	+)
	16	+
	17	+EXPECTED = {
	18	+ "software industry": ["software"],
	19	+ "Software": ["software"],
	20	+ "banking": ["banking"],
	21	+ "bank": ["banking"],
	22	+ "video game industry": ["gaming"],
	23	+ "automotive industry": ["automotive"],
	24	+ "insurance": ["insurance"],
	25	+ "pharmaceutical industry": ["pharmaceuticals"],
	26	+ "semiconductor industry": ["semiconductors"],
	27	+ "e-commerce": ["e-commerce"],
	28	+ "retail": ["retail"],
	29	+ "telecommunications industry": ["telecommunications"],
	30	+ "airline": ["airlines"],
	31	+ "film industry": ["entertainment"],
	32	+ "financial services": ["financial-services"],
	33	+ "oil and gas industry": ["oil-gas"],
	34	+ "renewable energy": ["renewables"],
	35	+ "biotechnology": ["biotechnology"],
	36	+ "hospitality industry": ["hospitality"],
	37	+ "aerospace manufacturer": ["aerospace-defense"],
	38	+ "arms industry": ["defense"],
	39	+ "food industry": ["food-beverage"],
	40	+ "fashion": ["apparel"],
	41	+ "mass media": ["media"],
	42	+ "real estate": ["real-estate"],
	43	+ "construction industry": ["construction"],
	44	+ "steel industry": ["mining"],
	45	+ "chemical industry": ["chemicals"],
	46	+ "cloud computing": ["cloud-infrastructure"],
	47	+ "artificial intelligence": ["artificial-intelligence"],
	48	+ "payment system": ["payments"],
	49	+ "asset management": ["asset-management"],
	50	+ "management consulting": ["consulting"],
	51	+ "agriculture": ["agriculture"],
	52	+ "robotics": ["robotics"],
	53	+ "machinery industry": ["industrial-machinery"],
	54	+ "public utility": ["utilities"],
	55	+ "glass industry": ["materials"],
	56	+ "health care industry": ["healthcare"],
	57	+ "medical device": ["medical-devices"],
	58	+ "toy industry": ["consumer-goods"],
	59	+ "logistics": ["logistics"],
	60	+ "shipping": ["shipping"],
	61	+ "education": ["education"],
	62	+ "tourism": ["travel"],
	63	+ "think tank": ["education"], # longest keyword wins over "tank" (defense)
	64	+ "cybersecurity": ["cybersecurity"],
	65	+ "social media": ["internet"],
	66	+}
	67	+
	68	+
	69	+@pytest.mark.parametrize(("label", "slugs"), sorted(EXPECTED.items()))
	70	+def test_map_label(label: str, slugs: list[str]) -> None:
	71	+ assert map_industry(label) == slugs
	72	+
	73	+
	74	+def test_unknown_and_empty() -> None:
	75	+ assert map_industry("") == []
	76	+ assert map_industry(None) == []
	77	+ assert map_industry("zzz nothing here") == []
	78	+ assert map_label("post-production") == [] # hyphenated words are not split into keywords
	79	+
	80	+
	81	+def test_multi_label_order_and_dedupe() -> None:
	82	+ assert map_industry(["banking", "insurance", "banking", "software industry"]) == ["banking", "insurance", "software"]
	83	+ assert map_industry(["a", "b", "c", "d", "e"] + ["banking"] * 3, limit=1) == ["banking"]
	84	+
	85	+
	86	+def test_taxonomy_integrity() -> None:
	87	+ inds = load_industries()
	88	+ slugs = [i.slug for i in inds]
	89	+ assert len(slugs) == len(set(slugs)) >= 45
	90	+ for i in inds:
	91	+ assert i.name and i.description
	92	+ assert len(i.keywords) >= 3, i.slug
	93	+ assert i.parent is None or i.parent in slugs
	94	+ assert i.parent is None or industry_index()[i.parent].parent is None, "max two levels"
	95	+ required = {"technology", "software", "artificial-intelligence", "semiconductors", "cloud-infrastructure", "cybersecurity", "internet",
	96	+ "e-commerce", "fintech", "payments", "banking", "insurance", "asset-management", "real-estate", "construction", "retail",
	97	+ "consumer-goods", "food-beverage", "apparel", "energy", "oil-gas", "renewables", "utilities", "mining", "chemicals", "materials",
	98	+ "manufacturing", "industrial-machinery", "automotive", "aerospace-defense", "transportation", "logistics", "airlines", "shipping",
	99	+ "telecommunications", "media", "entertainment", "gaming", "healthcare", "biotechnology", "pharmaceuticals", "medical-devices",
	100	+ "hospitality", "travel", "education", "professional-services", "consulting", "agriculture", "robotics", "defense"}
	101	+ assert required <= set(slugs)
	102	+ assert "technology" in top_level_slugs() and "software" not in top_level_slugs()
	103	+ assert top_level_of("software") == "technology" and top_level_of("technology") == "technology"
	104	+ assert is_valid_slug("banking") and not is_valid_slug("banks")
	105	+
	106	+
	107	+def test_map_sic() -> None:
	108	+ assert map_sic(7372) == "software"
	109	+ assert map_sic("3674") == "semiconductors"
	110	+ assert map_sic(6021) == "banking"
	111	+ assert map_sic(2836) == "biotechnology"
	112	+ assert map_sic(4512) == "airlines"
	113	+ assert map_sic(1311) == "oil-gas"
	114	+ assert map_sic(9999) is None
	115	+ assert map_sic(None) is None and map_sic("") is None and map_sic("abc") is None

added tests/test_registry_files.py +95 −0

		`@@ -0,0 +1,95 @@`
	1	+"""Every committed registry line is loadable and consistent; the universe is large and diversified (docs/SEEDS.md)."""
	2	+from __future__ import annotations
	3	+
	4	+import csv
	5	+from collections import Counter
	6	+
	7	+import pytest
	8	+
	9	+from companyatlas.registry.industries import REGISTRY_DIR, is_valid_slug, load_industries, top_level_of, top_level_slugs
	10	+from companyatlas.registry.seed import load_registry_rows, registry_files
	11	+from companyatlas.urls import registrable_domain
	12	+
	13	+MIN_TOTAL = 6000
	14	+US_CAP = 0.42 # 40 % target + tolerance for the industry top-up
	15	+OTHER_CAP = 0.13
	16	+REQUIRED = {"wikidata_id", "display_name", "website", "canonical_domain", "country", "industries", "importance", "tier", "source", "harvested_at"}
	17	+
	18	+
	19	+@pytest.fixture(scope="module")
	20	+def rows() -> list[dict]:
	21	+ files = registry_files()
	22	+ assert files, "registry/companies/*.ndjson missing — run scripts/seed_wikidata.py"
	23	+ return load_registry_rows(files)
	24	+
	25	+
	26	+@pytest.fixture(scope="module")
	27	+def countries() -> dict[str, dict]:
	28	+ with (REGISTRY_DIR / "countries.csv").open(encoding="utf-8") as f:
	29	+ return {r["code"]: r for r in csv.DictReader(f)}
	30	+
	31	+
	32	+def test_countries_csv(countries: dict[str, dict]) -> None:
	33	+ assert len(countries) >= 245
	34	+ for code, r in countries.items():
	35	+ assert len(code) == 2 and code.isupper() and r["name"]
	36	+ assert -90 <= float(r["lat"]) <= 90 and -180 <= float(r["lon"]) <= 180
	37	+ for must in ("US", "CA", "GB", "DE", "FR", "JP", "KR", "IN", "AU", "BR", "NG", "TW", "HK", "XK"):
	38	+ assert must in countries
	39	+ assert countries["TW"]["region"] == "Asia"
	40	+
	41	+
	42	+def test_every_line_is_valid(rows: list[dict], countries: dict[str, dict]) -> None:
	43	+ assert len(rows) >= MIN_TOTAL, f"only {len(rows)} companies"
	44	+ slugs = {i.slug for i in load_industries()}
	45	+ domains: Counter[str] = Counter()
	46	+ qids: Counter[str] = Counter()
	47	+ for r in rows:
	48	+ assert REQUIRED <= set(r), r.get("wikidata_id")
	49	+ assert r["wikidata_id"].startswith("Q") and r["wikidata_id"][1:].isdigit()
	50	+ assert r["display_name"].strip()
	51	+ assert r["website"].startswith(("https://", "http://")), r["website"]
	52	+ assert "/" not in r["website"].split("://", 1)[1], "website must be scheme + host only"
	53	+ assert r["canonical_domain"] == registrable_domain(r["website"])
	54	+ assert r["country"] is None or r["country"] in countries, r["country"]
	55	+ assert all(is_valid_slug(s) for s in r["industries"]), r["industries"]
	56	+ assert all(s in slugs for s in r["industries"])
	57	+ assert 0.0 <= r["importance"] <= 1.0 and r["tier"] in (1, 2, 3, 4)
	58	+ assert r["source"] == "wikidata" and r["sitelinks"] >= 2
	59	+ assert r["founded_year"] is None or 1000 <= r["founded_year"] <= 2026
	60	+ assert r["lat"] is None or -90 <= r["lat"] <= 90
	61	+ assert r["lon"] is None or -180 <= r["lon"] <= 180
	62	+ assert r["parent"] is None or set(r["parent"]) == {"wikidata_id", "name"}
	63	+ domains[r["canonical_domain"]] += 1
	64	+ qids[r["wikidata_id"]] += 1
	65	+ assert not [d for d, n in domains.items() if n > 1], "duplicate domains"
	66	+ assert not [q for q, n in qids.items() if n > 1], "duplicate wikidata ids"
	67	+
	68	+
	69	+def test_diversification(rows: list[dict]) -> None:
	70	+ n = len(rows)
	71	+ by_country = Counter(r["country"] for r in rows)
	72	+ assert by_country["US"] / n <= US_CAP, by_country["US"] / n
	73	+ for code, count in by_country.items():
	74	+ if code not in ("US", None):
	75	+ assert count / n <= OTHER_CAP, (code, count / n)
	76	+ assert by_country[None] / n <= 0.05
	77	+ for code, minimum in {"CA": 150, "GB": 150, "DE": 150, "FR": 150, "JP": 150, "IN": 150, "AU": 150}.items():
	78	+ assert by_country[code] >= minimum, (code, by_country[code])
	79	+ tiers = Counter(r["tier"] for r in rows)
	80	+ assert tiers[1] == 150 and tiers[2] == 800 and tiers[3] == 2500 and tiers[4] == n - 3450
	81	+ by_top: Counter[str] = Counter()
	82	+ for r in rows:
	83	+ for t in {top_level_of(s) for s in r["industries"]}:
	84	+ by_top[t] += 1
	85	+ covered = [t for t in top_level_slugs() if by_top[t] >= 60]
	86	+ assert len(covered) >= len(top_level_slugs()) - 2, {t: by_top[t] for t in top_level_slugs() if by_top[t] < 60}
	87	+ assert sum(1 for r in rows if r["industries"]) / n >= 0.85
	88	+
	89	+
	90	+def test_files_are_split_by_region(rows: list[dict], countries: dict[str, dict]) -> None:
	91	+ for path in registry_files():
	92	+ region = path.stem.removeprefix("wikidata-")
	93	+ for r in load_registry_rows([path]):
	94	+ actual = (countries.get(r["country"] or "", {}).get("region") or "other").lower().replace(" ", "-")
	95	+ assert actual == region, (path.name, r["wikidata_id"], actual)

added tests/test_seed_loader.py +113 −0

		`@@ -0,0 +1,113 @@`
	1	+"""Seed loader against the local database: idempotent upserts, aliases, relationships, discover queue (rows use `ztest-` slugs)."""
	2	+from __future__ import annotations
	3	+
	4	+import json
	5	+from pathlib import Path
	6	+
	7	+import pytest
	8	+
	9	+from companyatlas.db import dispose, fetch_all, fetch_one, fetch_val, transaction
	10	+from companyatlas.registry.seed import add_company, normalise_row, seed, unique_slug
	11	+
	12	+FIXTURE = [
	13	+ {"wikidata_id": "Q900000001", "display_name": "ZTest Alpha Corp", "legal_name": "ZTest Alpha Corporation", "aliases": ["ZTest Alpha", "Alpha Systems"],
	14	+ "website": "https://www.ztest-alpha.com/", "canonical_domain": "ztest-alpha.com", "country": "CA", "hq_city": "Montréal",
	15	+ "industries": ["software", "cloud-infrastructure"], "industry_labels": ["software industry"], "founded_year": 1999, "employees": 1200,
	16	+ "public_company": True, "ticker": "ZTA", "exchange": "Toronto Stock Exchange", "parent": None, "sitelinks": 40, "importance": 0.8, "tier": 1,
	17	+ "source": "wikidata", "harvested_at": "2026-09-12T00:00:00+00:00"},
	18	+ {"wikidata_id": "Q900000002", "display_name": "ZTest Beta", "website": "https://ztest-beta.com", "country": "US", "industries": ["fintech"],
	19	+ "parent": {"wikidata_id": "Q900000001", "name": "ZTest Alpha Corp"}, "importance": 0.4, "tier": 3, "source": "wikidata"},
	20	+ {"website": "ztest-gamma.com", "country": "QZ"}, # no name, unknown country → derived / nulled
	21	+ {"wikidata_id": "Q900000004", "display_name": "ZTest Alpha Duplicate", "website": "https://shop.ztest-alpha.com/x"}, # same domain → skipped
	22	+]
	23	+
	24	+
	25	+async def _cleanup() -> None:
	26	+ async with transaction() as conn:
	27	+ ids = [r["id"] for r in await fetch_all(conn, "select id from companies where slug like 'ztest-%'")]
	28	+ if ids:
	29	+ await conn.exec_driver_sql("delete from queue_jobs where key = any($1::text[])", ([f"discover:{i}" for i in ids],))
	30	+ await conn.exec_driver_sql("delete from companies where id = any($1::text[])", (ids,))
	31	+
	32	+
	33	+@pytest.fixture
	34	+def fixture_file(tmp_path: Path) -> Path:
	35	+ p = tmp_path / "ztest.ndjson"
	36	+ p.write_text("\n".join(json.dumps(r) for r in FIXTURE) + "\n", encoding="utf-8")
	37	+ return p
	38	+
	39	+
	40	+async def test_seed_twice_is_idempotent(fixture_file: Path) -> None:
	41	+ await _cleanup()
	42	+ try:
	43	+ async with transaction() as conn:
	44	+ first = await seed(conn, files=[fixture_file])
	45	+ assert first["industries"] >= 45 and first["countries"] >= 240
	46	+ assert first["companies_seen"] == 4 and first["companies_new"] == 3 and first["companies_skipped"] == 1
	47	+ assert first["relationships"] == 2 and first["queue_jobs"] == 3
	48	+
	49	+ async with transaction() as conn:
	50	+ rows = await fetch_all(conn, "select * from companies where slug like 'ztest-%' order by slug")
	51	+ assert [r["slug"] for r in rows] == ["ztest-alpha-corp", "ztest-beta", "ztest-gamma"]
	52	+ alpha, beta, gamma = rows
	53	+ assert alpha["onboarding_status"] == "pending" and alpha["country"] == "CA" and alpha["industry_primary"] == "software"
	54	+ assert alpha["public_company"] is True and alpha["ticker"] == "ZTA" and alpha["tier"] == 1 and abs(alpha["importance"] - 0.8) < 1e-6
	55	+ assert alpha["source_meta"]["source"] == "wikidata" and alpha["source_meta"]["sitelinks"] == 40
	56	+ assert gamma["display_name"] == "Ztest Gamma" and gamma["country"] is None and gamma["source_meta"]["unknown_country"] == "QZ"
	57	+ aliases = await fetch_all(conn, "select alias_norm, kind from company_aliases where company_id = :id order by alias_norm", id=alpha["id"])
	58	+ # "ZTest Alpha Corp", "ZTest Alpha Corporation" and "ZTest Alpha" collapse to one key (legal suffixes dropped by normalize_alias)
	59	+ assert {a["alias_norm"] for a in aliases} == {"ztestalpha", "zta", "alphasystems"}
	60	+ assert {a["kind"] for a in aliases} >= {"brand", "ticker"}
	61	+ assert await fetch_val(conn, "select count(*) from domains where company_id = :id and kind = 'primary'", id=alpha["id"]) == 1
	62	+ rels = await fetch_all(conn, "select from_company_id, to_company_id, kind, confidence from company_relationships "
	63	+ "where from_company_id in (:a, :b) order by kind", a=alpha["id"], b=beta["id"])
	64	+ assert [(r["kind"], r["from_company_id"] == alpha["id"]) for r in rels] == [("PARENT_OF", True), ("SUBSIDIARY_OF", False)]
	65	+ assert all(abs(r["confidence"] - 0.8) < 1e-6 for r in rels)
	66	+ job = await fetch_one(conn, "select kind, status, priority, payload from queue_jobs where key = :k", k=f"discover:{alpha['id']}")
	67	+ assert job and job["kind"] == "discover" and job["status"] == "pending" and abs(job["priority"] - 0.8) < 1e-6
	68	+ assert job["payload"]["company_id"] == alpha["id"]
	69	+ assert (await fetch_one(conn, "select value from settings_kv where key = 'seed:last_run'"))["value"]["counters"]["companies_new"] == 3
	70	+
	71	+ # Second run: nothing new, non-null values untouched even if the file changed, importance refreshed.
	72	+ changed = [dict(r) for r in FIXTURE]
	73	+ changed[0]["display_name"] = "RENAMED"
	74	+ changed[0]["legal_name"] = "Other Legal"
	75	+ changed[0]["importance"] = 0.5
	76	+ changed[0]["hq_region"] = "Quebec" # was null → filled
	77	+ fixture_file.write_text("\n".join(json.dumps(r) for r in changed) + "\n", encoding="utf-8")
	78	+ async with transaction() as conn:
	79	+ second = await seed(conn, files=[fixture_file])
	80	+ assert second["companies_new"] == 0 and second["companies_updated"] == 3 and second["queue_jobs"] == 0 and second["relationships"] == 0
	81	+ async with transaction() as conn:
	82	+ alpha2 = await fetch_one(conn, "select * from companies where wikidata_id = 'Q900000001'")
	83	+ assert alpha2["display_name"] == "ZTest Alpha Corp" and alpha2["legal_name"] == "ZTest Alpha Corporation"
	84	+ assert alpha2["hq_region"] == "Quebec" and abs(alpha2["importance"] - 0.5) < 1e-6
	85	+ assert await fetch_val(conn, "select count(*) from companies where slug like 'ztest-%'") == 3
	86	+ assert await fetch_val(conn, "select count(*) from company_relationships r join companies c on c.id = r.from_company_id "
	87	+ "where c.slug like 'ztest-%'") == 2
	88	+ assert await fetch_val(conn, "select count(*) from queue_jobs where key like 'discover:%' and payload->>'company_id' in "
	89	+ "(select id from companies where slug like 'ztest-%')") == 3
	90	+
	91	+ # add_company on an existing domain tops up; a new one is created pending with a job.
	92	+ async with transaction() as conn:
	93	+ res = await add_company(conn, "ztest-alpha.com", display_name="ignored")
	94	+ assert res["slug"] == "ztest-alpha-corp" and res["counters"]["companies_new"] == 0
	95	+ res = await add_company(conn, "https://ztest-delta.com/about", display_name="ZTest Delta", country="FR", industries=["banking"])
	96	+ assert res["slug"] == "ztest-delta" and res["onboarding_status"] == "pending" and res["counters"]["queue_jobs"] == 1
	97	+ finally:
	98	+ await _cleanup()
	99	+ await dispose()
	100	+
	101	+
	102	+def test_normalise_and_slug_helpers() -> None:
	103	+ assert normalise_row({"website": ""}, source="x") is None
	104	+ assert normalise_row({"website": "ftp://nope"}, source="x") is None
	105	+ row = normalise_row({"website": "example.org", "industries": ["software", "not-a-slug"], "industry": "banking", "tier": 9, "importance": 3},
	106	+ source="manual")
	107	+ assert row is not None
	108	+ assert row["website"] == "https://example.org" and row["canonical_domain"] == "example.org" and row["display_name"] == "Example"
	109	+ assert row["industries"] == ["banking", "software"] and row["tier"] == 4 and row["importance"] == 1.0
	110	+ assert unique_slug("acme", "US", set()) == "acme"
	111	+ assert unique_slug("acme", "US", {"acme"}) == "acme-us"
	112	+ assert unique_slug("acme", "US", {"acme", "acme-us"}) == "acme-2"
	113	+ assert unique_slug("acme", None, {"acme", "acme-2"}) == "acme-3"
	114