company profile enrichment: Wikidata facts/executives/structure, Wikipedia description, homepage JSON-LD/logo/socials, grounded LLM text; API profile/facts
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
17 changed files +4,166 −14
modified
docs/API.md
+47 −0
@@ -236,6 +236,53 @@ ever stored hashed. Admin endpoints answer `401 { "detail": "admin token require | ||
| 236 | 236 | duplicate | noise | misclassified) feeding `/admin/quality.calibration`; retract/restore append an audit trail under `payload._audit` and |
| 237 | 237 | never delete; `GET /admin/storage` reports the object store. `/admin/overview` also returns `companies_by_onboarding`, `scheduler_heartbeat`. |
| 238 | 238 | |
| 239 | +## Profile & facts (additive, 2026-09-13 — `services/enrichment.py`) | |
| 240 | + | |
| 241 | +Companies are enriched from **Wikidata** (entity), **Wikipedia** (lead summary, CC BY-SA 4.0), the **homepage** (meta description, | |
| 242 | +JSON-LD `Organization`, icons) and, only when no Wikipedia text exists, a **grounded LLM description** built from the company's own pages. | |
| 243 | +Every accepted value keeps its provenance; a weaker source never overwrites a better one (description: wikipedia > llm > homepage > wikidata; | |
| 244 | +other facts: wikidata > homepage > registry > wikipedia). Nothing is inferred — absent facts are `null`. | |
| 245 | + | |
| 246 | +```ts | |
| 247 | +type Provenance = { field: string; source: 'wikidata'|'wikipedia'|'homepage'|'llm'|'registry'; url: string | null; retrieved_at: string }; | |
| 248 | +type Money = { value: number; currency: string; year: number }; // ISO 4217; the fiscal year the figure refers to | |
| 249 | + | |
| 250 | +type CompanyProfile = { | |
| 251 | + description: string | null; description_source: 'wikipedia'|'wikidata'|'homepage'|'llm'|null; description_url: string | null; | |
| 252 | + description_license: string | null; // 'CC BY-SA 4.0' for Wikipedia text (attribution required: show description_url) | |
| 253 | + description_attribution: string | null; // e.g. 'Text from Wikipedia (en), CC BY-SA 4.0' · "Generated from the company's public pages" | |
| 254 | + logo_url: string | null; icon_url: string | null; founded_year: number | null; legal_form: string | null; legal_name: string | null; | |
| 255 | + employees: number | null; employees_year: number | null; revenue: Money | null; net_income: Money | null; total_assets: Money | null; | |
| 256 | + hq: { city: string | null; region: string | null; country: string | null; address: string | null; lat: number | null; lon: number | null }; | |
| 257 | + ticker: string | null; exchange: string | null; isin: string | null; lei: string | null; sec_cik: string | null; public_company: boolean; | |
| 258 | + wikipedia_url: string | null; wikidata_url: string | null; official_website: string | null; phone: string | null; | |
| 259 | + products: string[]; industries: string[]; industry_labels: string[]; | |
| 260 | + socials: { linkedin?: string; x?: string; youtube?: string; facebook?: string; instagram?: string; github?: string; tiktok?: string; crunchbase?: string }; | |
| 261 | + enriched_at: string | null; sources: Provenance[]; version: 'profile-v1'; | |
| 262 | +}; | |
| 263 | +type Fact = { key: 'founded'|'headquarters'|'employees'|'revenue'|'net_income'|'total_assets'|'legal_form'|'listing'|'isin'|'lei'|'sec_cik'|'website'|'wikipedia'; | |
| 264 | + label: string; value: string; raw: unknown; source: string | null; url: string | null; retrieved_at: string | null }; | |
| 265 | +``` | |
| 266 | + | |
| 267 | +- **`CompanyCard.profile?: CompanyProfile`** — present on every card (lists, rankings, detail…) once the company has been enriched | |
| 268 | + (`companies.source_meta.profile`); absent before. `sources[]` holds one entry per populated field. | |
| 269 | +- **`GET /companies/{slug}`** adds `profile: CompanyProfile | null` and `facts: Fact[]` (display-ready strings such as `"47,756 (2013)"`, | |
| 270 | + `"USD 305.6 B (2023)"`, `"Mountain View, California, US"`; `raw` carries the underlying value). `relationships` rows now carry | |
| 271 | + `provenance: {source, property, qid?, retrieved_at?}`, `first_seen_at`, `last_seen_at`, and `kind` ∈ `PARENT_OF | SUBSIDIARY_OF | OWNED_BY | | |
| 272 | + OWNER_OF | ACQUIRED_BY | ACQUIRED | PARTNER_OF | COMPETITOR_OF | INVESTOR_IN | BRAND_OF` (Wikidata rows: confidence 0.85, `valid_from` / | |
| 273 | + `valid_to` from the start/end qualifiers; `company` is set when the counterpart is in the atlas, else only `to_name`). Current | |
| 274 | + relationships are listed before ended ones (limit 200). | |
| 275 | +- **`GET /companies/{slug}/people`** rows add `source: 'wikidata' | 'page'` (derived from `source_url`; Wikidata rows have | |
| 276 | + `title` = role such as `Chief Executive Officer`, `Chairperson`, `Founder`, or the position label for key people) and the payload adds | |
| 277 | + `sources: string[]`. A person whose Wikidata role ended is `no_longer_listed` (never "left"); page-sourced rows are never overridden by Wikidata. | |
| 278 | +- Columns back-filled by enrichment (`description`, `logo_url`, `hq_city`, `hq_region`, `country`, `founded_year`, `employees`, `ticker`, | |
| 279 | + `exchange`, `legal_name`, `lei`, `sec_cik`, `industries`) record their source in `companies.source_meta.provenance[column]` | |
| 280 | + (`{source, url, retrieved_at, previous?}`), so `CompanyCard.description` may now be a Wikipedia paragraph — show the attribution from | |
| 281 | + `profile.description_attribution` / `profile.description_url` when `profile.description_source === 'wikipedia'`. | |
| 282 | +- CLI: `catlas enrich-companies [--limit N] [--company slug…] [--source wikidata|wikipedia|homepage|llm|all] [--concurrency N] [--no-llm]`, | |
| 283 | + `catlas profile <slug> [--json]`. Periodic task `company-enrichment` (every 10 min, batch `CA_ENRICH_BATCH` = 300; never-enriched first, | |
| 284 | + active companies first, then profiles older than `CA_ENRICH_REFRESH_DAYS` = 30). | |
| 285 | + | |
| 239 | 286 | ## Conventions for implementers |
| 240 | 287 | - Every list endpoint is bounded (`per_page ≤ 200`, `limit ≤ 500`), uses indexed predicates, and returns `Cache-Control: public, max-age=60` for public aggregates (`/pulse`, `/stats`, `/rankings`, `/industries`, `/countries`) and `no-store` for `/live*`, owner and admin routes. |
| 241 | 288 | - Company lookups accept slug **or** id. Unknown → 404 `{detail: "company not found"}`. |
modified
docs/ARCHITECTURE.md
+1 −0
@@ -37,6 +37,7 @@ registry/ (seed companies, industries, countries) ──catlas seed──▶ c | ||
| 37 | 37 | | Crawl core | `sdk/normalize.py`, `sdk/diff.py`, `sdk/connector.py` (+ registry), `connectors/*`, `services/discovery.py`, `services/pipeline.py`, `services/scheduler.py`, `services/repair.py` | `crawl.py`: `onboard`, `discover`, `run-sensor`, `schedule`, `sensors`, `repair`, `connectors` | |
| 38 | 38 | | Intelligence | `services/events.py`, `services/clustering.py`, `services/llm/{gateway,enrich,schemas}.py`, `prompts/*`, `services/metrics.py`, `services/signals.py`, `services/trends.py`, `services/alerts.py`, `services/digest.py` | `intel.py`: `process-changes`, `enrich`, `metrics`, `daily`, `signals`, `alerts` | |
| 39 | 39 | | Seeds | `registry/` data files, `companyatlas/registry/seed.py`, `scripts/seed_wikidata.py`, `scripts/seed_edgar.py` | `seed.py`: `seed`, `import-companies` | |
| 40 | +| Profile enrichment | `services/enrichment.py` (Wikidata entity + Wikipedia summary + homepage facts + grounded LLM text → `companies.source_meta.profile`, column back-fills with provenance, `people`, `company_relationships`; periodic `company-enrichment`), `prompts/company-profile/` | `enrichment.py`: `enrich-companies`, `profile` | | |
| 40 | 41 | | API | `api/routers/*.py` (auto-included; `ORDER` for precedence), `api/sse.py`, `api/ratelimit.py` | — | |
| 41 | 42 | | Web | `apps/web` | — | |
| 42 | 43 | | Ops | `deploy/*.mld.json`, `deploy/render-manifest.sh`, `deploy/first-run.sh`, `scripts/backup*.sh` | `ops.py`: `stats`, `status`, `backup`, `retention` | |
added
fixtures/enrichment/homepage.html
+54 −0
@@ -0,0 +1,54 @@ | ||
| 1 | +<!DOCTYPE html> | |
| 2 | +<html lang="en"> | |
| 3 | +<head> | |
| 4 | + <meta charset="utf-8"> | |
| 5 | + <title>Example Robotics — autonomous warehouse robots</title> | |
| 6 | + <meta name="description" content="Example Robotics designs and builds autonomous mobile robots for warehouses and distribution centres in North America and Europe."> | |
| 7 | + <meta property="og:description" content="Autonomous mobile robots for warehouses."> | |
| 8 | + <meta property="og:image" content="/static/og-card.png"> | |
| 9 | + <link rel="icon" href="/favicon.ico" sizes="32x32"> | |
| 10 | + <link rel="apple-touch-icon" href="/static/apple-touch-icon.png" sizes="180x180"> | |
| 11 | + <script type="application/ld+json"> | |
| 12 | + { | |
| 13 | + "@context": "https://schema.org", | |
| 14 | + "@type": "Organization", | |
| 15 | + "name": "Example Robotics", | |
| 16 | + "legalName": "Example Robotics Inc.", | |
| 17 | + "url": "https://www.example-robotics.test/", | |
| 18 | + "logo": "https://www.example-robotics.test/static/logo.svg", | |
| 19 | + "foundingDate": "2014-03-01", | |
| 20 | + "numberOfEmployees": {"@type": "QuantitativeValue", "value": 420}, | |
| 21 | + "telephone": "+1 514-555-0100", | |
| 22 | + "address": { | |
| 23 | + "@type": "PostalAddress", | |
| 24 | + "streetAddress": "1200 Rue Example", | |
| 25 | + "addressLocality": "Montréal", | |
| 26 | + "addressRegion": "Quebec", | |
| 27 | + "postalCode": "H2X 1Y4", | |
| 28 | + "addressCountry": "CA" | |
| 29 | + }, | |
| 30 | + "sameAs": [ | |
| 31 | + "https://www.linkedin.com/company/example-robotics", | |
| 32 | + "https://twitter.com/examplerobotics", | |
| 33 | + "https://github.com/example-robotics", | |
| 34 | + "https://en.wikipedia.org/wiki/Example_Robotics" | |
| 35 | + ] | |
| 36 | + } | |
| 37 | + </script> | |
| 38 | +</head> | |
| 39 | +<body> | |
| 40 | + <header><nav><a href="/">Home</a> <a href="/products">Products</a> <a href="/about">About</a> <a href="/careers">Careers</a></nav></header> | |
| 41 | + <main> | |
| 42 | + <h1>Autonomous mobile robots for the modern warehouse</h1> | |
| 43 | + <p>Example Robotics designs and builds autonomous mobile robots for warehouses and distribution centres. Our fleet software coordinates | |
| 44 | + hundreds of robots per site, integrates with existing warehouse management systems and is deployed with customers in Canada, the | |
| 45 | + United States, Germany and the Netherlands.</p> | |
| 46 | + <p>Founded in Montréal in 2014, the company operates two manufacturing sites and employs 420 people across engineering, operations and | |
| 47 | + customer support. Example Robotics serves retailers, third-party logistics providers and manufacturers that need flexible automation | |
| 48 | + without fixed conveyors.</p> | |
| 49 | + <h2>Products</h2> | |
| 50 | + <ul><li>Carry 200 — goods-to-person robot</li><li>Lift 1000 — pallet mover</li><li>Fleet Manager — orchestration software</li></ul> | |
| 51 | + </main> | |
| 52 | + <footer><p>© Example Robotics Inc. · 1200 Rue Example, Montréal, Quebec H2X 1Y4 · +1 514-555-0100</p></footer> | |
| 53 | +</body> | |
| 54 | +</html> | |
added
fixtures/enrichment/wikidata-Q95.json
+1862 −0
Large diff (1,863 lines) is not loaded by default to keep the page fast.Load diff
added
fixtures/enrichment/wikidata-labels.json
+134 −0
Large diff (135 lines) is not loaded by default to keep the page fast.Load diff
added
fixtures/enrichment/wikipedia-summary.json
+27 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added
prompts/company-profile/v1.md
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
modified
src/companyatlas/api/queries.py
+1 −1
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified
src/companyatlas/api/routers/companies.py
+11 −9
Large diff (43 lines) is not loaded by default to keep the page fast.Load diff
modified
src/companyatlas/api/serializers.py
+17 −2
Large diff (45 lines) is not loaded by default to keep the page fast.Load diff
added
src/companyatlas/commands/enrichment.py
+99 −0
Large diff (99 lines) is not loaded by default to keep the page fast.Load diff
modified
src/companyatlas/config.py
+18 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
src/companyatlas/services/enrichment.py
+1415 −0
Large diff (1,415 lines) is not loaded by default to keep the page fast.Load diff
modified
src/companyatlas/services/llm/schemas.py
+15 −2
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
modified
src/companyatlas/services/periodic.py
+1 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
tests/test_api_profile.py
+69 −0
Large diff (69 lines) is not loaded by default to keep the page fast.Load diff
added
tests/test_enrichment.py
+378 −0
Large diff (379 lines) is not loaded by default to keep the page fast.Load diff