SPB Git
spboucher.ai

spb/company-atlas

Public

company profile enrichment: Wikidata facts/executives/structure, Wikipedia description, homepage JSON-LD/logo/socials, grounded LLM text; API profile/facts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 13, 2026) parent 82b8305 Browse files

17 changed files +4,166 −14

17 changed files
Load all 17 files Unified Split
modified docs/API.md +47 −0
@@ -236,6 +236,53 @@ ever stored hashed. Admin endpoints answer `401 { "detail": "admin token require
236 duplicate | noise | misclassified) feeding `/admin/quality.calibration`; retract/restore append an audit trail under `payload._audit` and 236 duplicate | noise | misclassified) feeding `/admin/quality.calibration`; retract/restore append an audit trail under `payload._audit` and
237 never delete; `GET /admin/storage` reports the object store. `/admin/overview` also returns `companies_by_onboarding`, `scheduler_heartbeat`. 237 never delete; `GET /admin/storage` reports the object store. `/admin/overview` also returns `companies_by_onboarding`, `scheduler_heartbeat`.
238 238
239+## Profile & facts (additive, 2026-09-13 — `services/enrichment.py`)
240+
241+Companies are enriched from **Wikidata** (entity), **Wikipedia** (lead summary, CC BY-SA 4.0), the **homepage** (meta description,
242+JSON-LD `Organization`, icons) and, only when no Wikipedia text exists, a **grounded LLM description** built from the company's own pages.
243+Every accepted value keeps its provenance; a weaker source never overwrites a better one (description: wikipedia > llm > homepage > wikidata;
244+other facts: wikidata > homepage > registry > wikipedia). Nothing is inferred — absent facts are `null`.
245+
246+```ts
247+type Provenance = { field: string; source: 'wikidata'|'wikipedia'|'homepage'|'llm'|'registry'; url: string | null; retrieved_at: string };
248+type Money = { value: number; currency: string; year: number }; // ISO 4217; the fiscal year the figure refers to
249+
250+type CompanyProfile = {
251+ description: string | null; description_source: 'wikipedia'|'wikidata'|'homepage'|'llm'|null; description_url: string | null;
252+ description_license: string | null; // 'CC BY-SA 4.0' for Wikipedia text (attribution required: show description_url)
253+ description_attribution: string | null; // e.g. 'Text from Wikipedia (en), CC BY-SA 4.0' · "Generated from the company's public pages"
254+ logo_url: string | null; icon_url: string | null; founded_year: number | null; legal_form: string | null; legal_name: string | null;
255+ employees: number | null; employees_year: number | null; revenue: Money | null; net_income: Money | null; total_assets: Money | null;
256+ hq: { city: string | null; region: string | null; country: string | null; address: string | null; lat: number | null; lon: number | null };
257+ ticker: string | null; exchange: string | null; isin: string | null; lei: string | null; sec_cik: string | null; public_company: boolean;
258+ wikipedia_url: string | null; wikidata_url: string | null; official_website: string | null; phone: string | null;
259+ products: string[]; industries: string[]; industry_labels: string[];
260+ socials: { linkedin?: string; x?: string; youtube?: string; facebook?: string; instagram?: string; github?: string; tiktok?: string; crunchbase?: string };
261+ enriched_at: string | null; sources: Provenance[]; version: 'profile-v1';
262+};
263+type Fact = { key: 'founded'|'headquarters'|'employees'|'revenue'|'net_income'|'total_assets'|'legal_form'|'listing'|'isin'|'lei'|'sec_cik'|'website'|'wikipedia';
264+ label: string; value: string; raw: unknown; source: string | null; url: string | null; retrieved_at: string | null };
265+```
266+
267+- **`CompanyCard.profile?: CompanyProfile`** — present on every card (lists, rankings, detail…) once the company has been enriched
268+ (`companies.source_meta.profile`); absent before. `sources[]` holds one entry per populated field.
269+- **`GET /companies/{slug}`** adds `profile: CompanyProfile | null` and `facts: Fact[]` (display-ready strings such as `"47,756 (2013)"`,
270+ `"USD 305.6 B (2023)"`, `"Mountain View, California, US"`; `raw` carries the underlying value). `relationships` rows now carry
271+ `provenance: {source, property, qid?, retrieved_at?}`, `first_seen_at`, `last_seen_at`, and `kind` ∈ `PARENT_OF | SUBSIDIARY_OF | OWNED_BY |
272+ OWNER_OF | ACQUIRED_BY | ACQUIRED | PARTNER_OF | COMPETITOR_OF | INVESTOR_IN | BRAND_OF` (Wikidata rows: confidence 0.85, `valid_from` /
273+ `valid_to` from the start/end qualifiers; `company` is set when the counterpart is in the atlas, else only `to_name`). Current
274+ relationships are listed before ended ones (limit 200).
275+- **`GET /companies/{slug}/people`** rows add `source: 'wikidata' | 'page'` (derived from `source_url`; Wikidata rows have
276+ `title` = role such as `Chief Executive Officer`, `Chairperson`, `Founder`, or the position label for key people) and the payload adds
277+ `sources: string[]`. A person whose Wikidata role ended is `no_longer_listed` (never "left"); page-sourced rows are never overridden by Wikidata.
278+- Columns back-filled by enrichment (`description`, `logo_url`, `hq_city`, `hq_region`, `country`, `founded_year`, `employees`, `ticker`,
279+ `exchange`, `legal_name`, `lei`, `sec_cik`, `industries`) record their source in `companies.source_meta.provenance[column]`
280+ (`{source, url, retrieved_at, previous?}`), so `CompanyCard.description` may now be a Wikipedia paragraph — show the attribution from
281+ `profile.description_attribution` / `profile.description_url` when `profile.description_source === 'wikipedia'`.
282+- CLI: `catlas enrich-companies [--limit N] [--company slug…] [--source wikidata|wikipedia|homepage|llm|all] [--concurrency N] [--no-llm]`,
283+ `catlas profile <slug> [--json]`. Periodic task `company-enrichment` (every 10 min, batch `CA_ENRICH_BATCH` = 300; never-enriched first,
284+ active companies first, then profiles older than `CA_ENRICH_REFRESH_DAYS` = 30).
285+
239 ## Conventions for implementers 286 ## Conventions for implementers
240 - Every list endpoint is bounded (`per_page ≤ 200`, `limit ≤ 500`), uses indexed predicates, and returns `Cache-Control: public, max-age=60` for public aggregates (`/pulse`, `/stats`, `/rankings`, `/industries`, `/countries`) and `no-store` for `/live*`, owner and admin routes. 287 - Every list endpoint is bounded (`per_page ≤ 200`, `limit ≤ 500`), uses indexed predicates, and returns `Cache-Control: public, max-age=60` for public aggregates (`/pulse`, `/stats`, `/rankings`, `/industries`, `/countries`) and `no-store` for `/live*`, owner and admin routes.
241 - Company lookups accept slug **or** id. Unknown → 404 `{detail: "company not found"}`. 288 - Company lookups accept slug **or** id. Unknown → 404 `{detail: "company not found"}`.
modified docs/ARCHITECTURE.md +1 −0
@@ -37,6 +37,7 @@ registry/ (seed companies, industries, countries) ──catlas seed──▶ c
37 | Crawl core | `sdk/normalize.py`, `sdk/diff.py`, `sdk/connector.py` (+ registry), `connectors/*`, `services/discovery.py`, `services/pipeline.py`, `services/scheduler.py`, `services/repair.py` | `crawl.py`: `onboard`, `discover`, `run-sensor`, `schedule`, `sensors`, `repair`, `connectors` | 37 | Crawl core | `sdk/normalize.py`, `sdk/diff.py`, `sdk/connector.py` (+ registry), `connectors/*`, `services/discovery.py`, `services/pipeline.py`, `services/scheduler.py`, `services/repair.py` | `crawl.py`: `onboard`, `discover`, `run-sensor`, `schedule`, `sensors`, `repair`, `connectors` |
38 | Intelligence | `services/events.py`, `services/clustering.py`, `services/llm/{gateway,enrich,schemas}.py`, `prompts/*`, `services/metrics.py`, `services/signals.py`, `services/trends.py`, `services/alerts.py`, `services/digest.py` | `intel.py`: `process-changes`, `enrich`, `metrics`, `daily`, `signals`, `alerts` | 38 | Intelligence | `services/events.py`, `services/clustering.py`, `services/llm/{gateway,enrich,schemas}.py`, `prompts/*`, `services/metrics.py`, `services/signals.py`, `services/trends.py`, `services/alerts.py`, `services/digest.py` | `intel.py`: `process-changes`, `enrich`, `metrics`, `daily`, `signals`, `alerts` |
39 | Seeds | `registry/` data files, `companyatlas/registry/seed.py`, `scripts/seed_wikidata.py`, `scripts/seed_edgar.py` | `seed.py`: `seed`, `import-companies` | 39 | Seeds | `registry/` data files, `companyatlas/registry/seed.py`, `scripts/seed_wikidata.py`, `scripts/seed_edgar.py` | `seed.py`: `seed`, `import-companies` |
40+| Profile enrichment | `services/enrichment.py` (Wikidata entity + Wikipedia summary + homepage facts + grounded LLM text → `companies.source_meta.profile`, column back-fills with provenance, `people`, `company_relationships`; periodic `company-enrichment`), `prompts/company-profile/` | `enrichment.py`: `enrich-companies`, `profile` |
40 | API | `api/routers/*.py` (auto-included; `ORDER` for precedence), `api/sse.py`, `api/ratelimit.py` | — | 41 | API | `api/routers/*.py` (auto-included; `ORDER` for precedence), `api/sse.py`, `api/ratelimit.py` | — |
41 | Web | `apps/web` | — | 42 | Web | `apps/web` | — |
42 | Ops | `deploy/*.mld.json`, `deploy/render-manifest.sh`, `deploy/first-run.sh`, `scripts/backup*.sh` | `ops.py`: `stats`, `status`, `backup`, `retention` | 43 | Ops | `deploy/*.mld.json`, `deploy/render-manifest.sh`, `deploy/first-run.sh`, `scripts/backup*.sh` | `ops.py`: `stats`, `status`, `backup`, `retention` |
added fixtures/enrichment/homepage.html +54 −0
@@ -0,0 +1,54 @@
1+<!DOCTYPE html>
2+<html lang="en">
3+<head>
4+ <meta charset="utf-8">
5+ <title>Example Robotics — autonomous warehouse robots</title>
6+ <meta name="description" content="Example Robotics designs and builds autonomous mobile robots for warehouses and distribution centres in North America and Europe.">
7+ <meta property="og:description" content="Autonomous mobile robots for warehouses.">
8+ <meta property="og:image" content="/static/og-card.png">
9+ <link rel="icon" href="/favicon.ico" sizes="32x32">
10+ <link rel="apple-touch-icon" href="/static/apple-touch-icon.png" sizes="180x180">
11+ <script type="application/ld+json">
12+ {
13+ "@context": "https://schema.org",
14+ "@type": "Organization",
15+ "name": "Example Robotics",
16+ "legalName": "Example Robotics Inc.",
17+ "url": "https://www.example-robotics.test/",
18+ "logo": "https://www.example-robotics.test/static/logo.svg",
19+ "foundingDate": "2014-03-01",
20+ "numberOfEmployees": {"@type": "QuantitativeValue", "value": 420},
21+ "telephone": "+1 514-555-0100",
22+ "address": {
23+ "@type": "PostalAddress",
24+ "streetAddress": "1200 Rue Example",
25+ "addressLocality": "Montréal",
26+ "addressRegion": "Quebec",
27+ "postalCode": "H2X 1Y4",
28+ "addressCountry": "CA"
29+ },
30+ "sameAs": [
31+ "https://www.linkedin.com/company/example-robotics",
32+ "https://twitter.com/examplerobotics",
33+ "https://github.com/example-robotics",
34+ "https://en.wikipedia.org/wiki/Example_Robotics"
35+ ]
36+ }
37+ </script>
38+</head>
39+<body>
40+ <header><nav><a href="/">Home</a> <a href="/products">Products</a> <a href="/about">About</a> <a href="/careers">Careers</a></nav></header>
41+ <main>
42+ <h1>Autonomous mobile robots for the modern warehouse</h1>
43+ <p>Example Robotics designs and builds autonomous mobile robots for warehouses and distribution centres. Our fleet software coordinates
44+ hundreds of robots per site, integrates with existing warehouse management systems and is deployed with customers in Canada, the
45+ United States, Germany and the Netherlands.</p>
46+ <p>Founded in Montréal in 2014, the company operates two manufacturing sites and employs 420 people across engineering, operations and
47+ customer support. Example Robotics serves retailers, third-party logistics providers and manufacturers that need flexible automation
48+ without fixed conveyors.</p>
49+ <h2>Products</h2>
50+ <ul><li>Carry 200 — goods-to-person robot</li><li>Lift 1000 — pallet mover</li><li>Fleet Manager — orchestration software</li></ul>
51+ </main>
52+ <footer><p>© Example Robotics Inc. · 1200 Rue Example, Montréal, Quebec H2X 1Y4 · +1 514-555-0100</p></footer>
53+</body>
54+</html>
added fixtures/enrichment/wikidata-Q95.json +1862 −0
Large diff (1,863 lines) is not loaded by default to keep the page fast.Load diff
added fixtures/enrichment/wikidata-labels.json +134 −0
Large diff (135 lines) is not loaded by default to keep the page fast.Load diff
added fixtures/enrichment/wikipedia-summary.json +27 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added prompts/company-profile/v1.md +17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
modified src/companyatlas/api/queries.py +1 −1
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified src/companyatlas/api/routers/companies.py +11 −9
Large diff (43 lines) is not loaded by default to keep the page fast.Load diff
modified src/companyatlas/api/serializers.py +17 −2
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added src/companyatlas/commands/enrichment.py +99 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
modified src/companyatlas/config.py +18 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added src/companyatlas/services/enrichment.py +1415 −0
Large diff (1,415 lines) is not loaded by default to keep the page fast.Load diff
modified src/companyatlas/services/llm/schemas.py +15 −2
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
modified src/companyatlas/services/periodic.py +1 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_api_profile.py +69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_enrichment.py +378 −0
Large diff (379 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere