extraction precision: multilingual validators for jobs/people/locations/plans/products/news, extractor fixes, fixtures, audit script
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
15 changed files +1,723 −97
modified
docs/CONNECTORS.md
+36 −0
@@ -44,6 +44,42 @@ hybrid, `seniority_guess`, `employment_type_norm`. The pipeline sets `is_ai` (`t | ||
| 44 | 44 | Job identity = `job_fingerprint(title, location_text, external_id | url)`; structured boards also give each job a stable block |
| 45 | 45 | key so the block diff mirrors the job delta. |
| 46 | 46 | |
| 47 | +## Precision rules (`connectors/_precision.py`, `precision-v1`, 2026-09-13) | |
| 48 | + | |
| 49 | +Production samples showed the generic HTML connector capturing navigation / call-to-action anchors as jobs (`Meer info`, `Early careers | |
| 50 | +bij …`), cookie-consent categories as locations (`Performance & Analytics | Allows use of behavioural data…`), swapped person cards | |
| 51 | +(`Chair Emeritus | Warner Bros. Discovery`), marketing headings as plans (`Win your market with … | Talk to sales`) and truncated | |
| 52 | +`price_text`. The precision module holds deterministic, multilingual (EN/FR/DE/NL/ES/IT/PT/JA) validators and normalisers that | |
| 53 | +`generic_html.py` applies while extracting and `services/pipeline._drop_corrupt_entities` re-applies as the last line of defence for | |
| 54 | +every connector (job rules only for `generic-html-v1` — structured boards are trusted). Every rule returns a `Verdict(ok, reason)`, so | |
| 55 | +`scripts/audit_extractions.py` can explain rejections on stored rows. Limits are module constants (no magic numbers). | |
| 56 | + | |
| 57 | +| entity | rejected | required | normalised | | |
| 58 | +|---|---|---|---| | |
| 59 | +| jobs (HTML) | CTA / nav stoplist (`read more`, `apply now`, `meer info`, `en savoir plus`, `mehr erfahren`, `alle stellen`, `ver más`, `scopri di più`, `詳細を見る` …), category prefixes (`early careers`, `careers at …`, `why …`), single-word titles, titles ending with an ellipsis, prose-like titles (≥ 4 words with function words) without role vocabulary, > 140 chars | role vocabulary (engineer, manager, analyst, nurse, driver, `ingénieur`, `entwickler`, `verpleegkundige`, `ingeniero`, `エンジニア` … or a format marker `(m/w/d)`, `Vollzeit`), **or** a job-like URL (`/job/`, `/jobs/<slug>`, `/careers/x/y`, `/vacature`, `/stelle`, `/offre`, `/emploi`, `/empleo`, `/lavoro`, numeric id, ATS hosts), **or** an explicit location / department cell | `(m/w/d)`, `(h/f)`, `(all genders)`, `- Apply now` and bracketed ids stripped from the display title; the id becomes `external_id` (so it stays in the fingerprint). Bare known cities (`München`) count as a location cell | | |
| 60 | +| people | names that are role titles or generic labels (`Contact`, `Team`, `Board of Directors`, `Leadership`, `お問い合わせ`), digits, < 2 or > 5 tokens | name = 2–5 capitalised tokens (particles / initials allowed) or a short CJK name without role words | name ↔ title **swapped** when the card put the title first; link-label titles (`Contact`, `Read more`, `Bio`, `LinkedIn`, `プロフィール`) and bio sentences → `title=None`; `role_category` / `is_executive` recomputed after the swap (JA titles: 代表取締役社長 → ceo, 社外取締役 → board …) | | |
| 61 | +| locations | cookie / consent / nav vocabulary (`performance`, `analytics`, `marketing`, `necessary`, `preferences`, `cookies`, `contact us`, `careers`, `privacy`, `newsletter`, `datenschutz`, `voorkeuren` …), names > 80 chars or > 8 words, sentences | at least one of: ISO country, known city (≈ 400 major cities + exonyms, n-gram match), postal code (UK/NL/CA/numeric), street address (EN/FR/DE/NL/ES/IT patterns), explicit kind (`headquarters`, `factory`, `warehouse`, `lab`, `store` … or a kind word in the name) | `city` cleared unless it is a plausible city (`Level 12` is not); `Oman | Hormuz Grand Hotel` → name = venue, country kept; a known city parsed as region (`3089 JH Rotterdam`) is promoted to `city`; city-states keep `city` | | |
| 62 | +| pricing plans | names > 40 chars or > 5 words, terminal punctuation, `?`, `|`, eyebrows (`Most popular`, `New`), generic headings (`Plans`, `Pricing`, `Tarifs`, `Preise`), marketing verbs (`win`, `get`, `discover`, `unlock`, `découvrez`, `entdecken` …), plans without a parsed price / `contact_sales` / free tier | — | `price_text` = the price phrase only (`$13/day`, `Starting at €333 / month`, `US$ 1,199 per user / month`, `Talk to sales`, `Free`) — never a cut sentence; the plan name is the first valid label before the price line (eyebrows skipped); a tier literally called `Free` is a valid name | | |
| 63 | +| products | nav words (`Overview`, `Products`, `Solutions`, `All products`, `Learn more`, `Produits`, `製品一覧`), > 80 chars / > 10 words, sentences and slogans | — | `®` / `™` kept | | |
| 64 | +| news | pagination (`Next`, `Older posts »`, `Page 2`, `3`), category labels (`Press releases`, `Blog posts`, `Actualités`, `ニュース`), `Read more` | ≥ 3 words or ≥ 15 chars **unless** `published_at` or a `/YYYY/MM/` URL confirms an article | the title is the first non-date line of the card | | |
| 65 | + | |
| 66 | +Normalizer changes made for these rules (`sdk/normalize.py`): a `.job-list`-style wrapper whose only child is a linked / card-like | |
| 67 | +`<ul>` or `<table>` is a container (its rows become `job_listing` blocks, inheriting the wrapper's kind) while a feature `<ul>` inside a | |
| 68 | +pricing card is not; card text has one line per direct child so inline `<a>title</a><span>city</span>` cells do not merge. Stripping | |
| 69 | +gender markers changes the display title (and therefore the fingerprint) of affected HTML jobs once — expect a one-off | |
| 70 | +removed + added pair for `(m/w/d)`-style listings on the first crawl after deployment. | |
| 71 | + | |
| 72 | +Fixtures reproducing the production noise: `generic_html/careers_cta_noise.html` (NL careers page: CTA anchors, single word, ellipsis | |
| 73 | ++ 5 real listings incl. `(m/v/x)`, `[12933]`, `- Apply now`), `careers_de.html` (German `(m/w/d)` listings + `Alle Stellen` / `Mehr | |
| 74 | +erfahren`), `leadership_swapped.html` (title-first card, `Contact` link, `Chair Emeritus | Warner Bros. Discovery`, `Board of Directors`), | |
| 75 | +`leadership_ja.html` (Japanese officers page), `locations_cookie_noise.html` (consent categories inside a locations page, `Contact Us | KR`, | |
| 76 | +`Oman | Hormuz Grand Hotel`, Dutch address), `pricing_marketing_noise.html` (marketing hero + `Talk to sales`, `Worry-free roaming.` + | |
| 77 | +prose price, eyebrow, real Free / Starter / Team / Business / Enterprise), `products_news_noise.html`. Tests: `tests/test_extraction_precision.py` | |
| 78 | +(noise gone, every real item still extracted). Audit / purge of stored rows: `scripts/audit_extractions.py` — samples HTML-sourced rows per | |
| 79 | +table, prints a precision estimate (Wilson 95 % interval) and the rejections by reason; `--all-sources` widens the read-only audit to | |
| 80 | +enrichment rows (Wikidata people); `--purge` deletes failing **HTML-sourced** rows of people / products / pricing_plans / locations / jobs only | |
| 81 | +(never ATS jobs, enrichment rows or news). Run it before and after deploying to measure the effect on production. | |
| 82 | + | |
| 47 | 83 | ## Adding a connector family |
| 48 | 84 | |
| 49 | 85 | 1. Create `src/companyatlas/connectors/<family>.py`; subclass `Connector` (or `AtsConnector` for JSON/XML boards — implement |
added
fixtures/connectors/generic_html/careers_cta_noise.html
+30 −0
@@ -0,0 +1,30 @@ | ||
| 1 | +<!doctype html> | |
| 2 | +<html lang="nl"> | |
| 3 | +<head><title>Jobs bij Telenet group</title></head> | |
| 4 | +<body> | |
| 5 | +<nav class="navbar"><a href="/">Home</a><a href="/nl/jobs">Jobs</a><a href="/nl/over-ons">Over ons</a></nav> | |
| 6 | +<main> | |
| 7 | + <section class="hero"><h1>Werken bij Telenet group</h1><p>Ontdek onze openstaande vacatures.</p></section> | |
| 8 | + <section class="job-teasers"> | |
| 9 | + <h2>Ontdek</h2> | |
| 10 | + <div class="job-card"><a href="/nl/jobs/starters"><h3>Voor starters die willen gáán</h3></a><p>Start je carrière bij ons.</p></div> | |
| 11 | + <div class="job-card"><a href="/nl/jobs/early-careers"><h3>Early careers bij Telenet group</h3></a><p>Stages en graduate programma's.</p></div> | |
| 12 | + <div class="job-card"><a href="/nl/jobs/why-us"><h3>Waarom werken bij ons?</h3></a><a href="/nl/jobs/why-us">Meer info</a></div> | |
| 13 | + <div class="job-card"><a href="/nl/jobs/search"><h3>Alle vacatures</h3></a><a href="/nl/jobs/search">Bekijk alle vacatures →</a></div> | |
| 14 | + </section> | |
| 15 | + <section class="job-list"> | |
| 16 | + <h2>Openstaande vacatures</h2> | |
| 17 | + <ul> | |
| 18 | + <li class="job-item"><a href="/nl/jobs/12841-senior-backend-software-engineer">Senior Backend Software Engineer - Infrastructure</a><span class="location">London, United Kingdom</span><span class="dept">Engineering</span></li> | |
| 19 | + <li class="job-item"><a href="/nl/jobs/12902-verpleegkundige">Verpleegkundige spoedgevallen (m/v/x)</a><span class="location">Gent, België</span><span class="dept">Zorg</span></li> | |
| 20 | + <li class="job-item"><a href="/nl/jobs/12933-data-analyst">Data Analyst (m/w/d) [12933]</a><span class="location">Berlin, Deutschland</span><span class="dept">Data</span></li> | |
| 21 | + <li class="job-item"><a href="/nl/jobs/12950-accountmanager">Accountmanager KMO - Apply now</a><span class="location">Mechelen, België</span><span class="dept">Sales</span></li> | |
| 22 | + <li class="job-item"><a href="/nl/jobs/12977-technicien">Technicien réseau (h/f)</a><span class="location">Bruxelles, Belgique</span><span class="dept">Network</span></li> | |
| 23 | + <li class="job-item"><a href="/nl/jobs/12990-sales">Sales</a><span class="location"></span></li> | |
| 24 | + <li class="job-item"><a href="/nl/jobs/search?page=2">Meer info</a><span class="location"></span></li> | |
| 25 | + <li class="job-item"><a href="/nl/jobs/13001">Wij zoeken mensen die het verschil willen maken voor onze klanten…</a><span class="location">Mechelen</span></li> | |
| 26 | + </ul> | |
| 27 | + </section> | |
| 28 | +</main> | |
| 29 | +<footer><p>© 2026 Telenet group</p></footer> | |
| 30 | +</body></html> | |
added
fixtures/connectors/generic_html/careers_de.html
+26 −0
@@ -0,0 +1,26 @@ | ||
| 1 | +<!doctype html> | |
| 2 | +<html lang="de"> | |
| 3 | +<head><title>Karriere bei Muster AG</title></head> | |
| 4 | +<body> | |
| 5 | +<nav class="navbar"><a href="/">Start</a><a href="/karriere">Karriere</a><a href="/unternehmen">Unternehmen</a></nav> | |
| 6 | +<main> | |
| 7 | + <section class="hero"><h1>Karriere bei Muster AG</h1><p>Gestalten Sie mit uns die Zukunft.</p></section> | |
| 8 | + <section class="job-teasers"> | |
| 9 | + <div class="job-card"><a href="/karriere/stellen"><h3>Alle Stellen</h3></a><a href="/karriere/stellen">Mehr erfahren</a></div> | |
| 10 | + <div class="job-card"><a href="/karriere/warum-wir"><h3>Warum Muster AG?</h3></a><a href="/karriere/warum-wir">Mehr erfahren →</a></div> | |
| 11 | + <div class="job-card"><a href="/karriere/bewerbung"><h3>Jetzt bewerben</h3></a></div> | |
| 12 | + </section> | |
| 13 | + <section class="job-list"> | |
| 14 | + <h2>Offene Stellen</h2> | |
| 15 | + <ul> | |
| 16 | + <li class="job-item"><a href="/karriere/stellen/4471-softwareentwickler">Softwareentwickler Backend (m/w/d)</a><span>München</span><span>IT</span></li> | |
| 17 | + <li class="job-item"><a href="/karriere/stellen/4488-projektleiter">Projektleiter Anlagenbau (m/w/d)</a><span>Stuttgart</span><span>Engineering</span></li> | |
| 18 | + <li class="job-item"><a href="/karriere/stellen/4490-ausbildung">Ausbildung zum Industriemechaniker (m/w/d) 2027</a><span>Hamburg</span><span>Ausbildung</span></li> | |
| 19 | + <li class="job-item"><a href="/karriere/stellen/4501-werkstudent">Werkstudent Marketing (all genders)</a><span>Berlin</span><span>Marketing</span></li> | |
| 20 | + <li class="job-item"><a href="/karriere/stellen/4510-pflege">Pflegefachkraft Intensivstation (w/m/d)</a><span>Köln</span><span>Pflege</span></li> | |
| 21 | + <li class="job-item"><a href="/karriere/stellen">Weiterlesen</a><span></span></li> | |
| 22 | + </ul> | |
| 23 | + </section> | |
| 24 | +</main> | |
| 25 | +<footer><p>© 2026 Muster AG</p></footer> | |
| 26 | +</body></html> | |
added
fixtures/connectors/generic_html/leadership_ja.html
+19 −0
@@ -0,0 +1,19 @@ | ||
| 1 | +<!doctype html> | |
| 2 | +<html lang="ja"> | |
| 3 | +<head><title>役員紹介 | 株式会社サンプル</title></head> | |
| 4 | +<body> | |
| 5 | +<nav class="navbar"><a href="/">ホーム</a><a href="/company/officers">役員紹介</a><a href="/contact">お問い合わせ</a></nav> | |
| 6 | +<main id="main"> | |
| 7 | + <h1>役員紹介</h1> | |
| 8 | + <section class="team-grid"> | |
| 9 | + <h2>取締役</h2> | |
| 10 | + <div class="team-member"><h3>山田 太郎</h3><p class="role">代表取締役社長</p></div> | |
| 11 | + <div class="team-member"><h3>佐藤 花子</h3><p class="role">取締役 最高財務責任者（CFO）</p></div> | |
| 12 | + <div class="team-member"><h3>鈴木 一郎</h3><p class="role">社外取締役</p></div> | |
| 13 | + <div class="team-member"><h3>高橋 美咲</h3><p class="role">執行役員 技術本部長</p><a href="/company/officers/takahashi">プロフィール</a></div> | |
| 14 | + <div class="team-member"><h3>お問い合わせ</h3><p class="role">詳細を見る</p></div> | |
| 15 | + <div class="team-member"><h3>株式会社サンプル</h3><p class="role">会社概要</p></div> | |
| 16 | + </section> | |
| 17 | +</main> | |
| 18 | +<footer><p>© 2026 株式会社サンプル</p></footer> | |
| 19 | +</body></html> | |
added
fixtures/connectors/generic_html/leadership_swapped.html
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +<!doctype html> | |
| 2 | +<html lang="en"> | |
| 3 | +<head><title>Leadership | Warner Example</title></head> | |
| 4 | +<body> | |
| 5 | +<nav class="navbar"><a href="/">Home</a><a href="/leadership">Leadership</a><a href="/contact">Contact</a></nav> | |
| 6 | +<main id="main"> | |
| 7 | + <h1>Our leadership</h1> | |
| 8 | + <section class="team-grid"> | |
| 9 | + <h2>Executive team</h2> | |
| 10 | + <div class="team-member"><p class="role">Chief Executive Officer</p><h3>Jane Doe</h3><p>Jane leads the company since 2019.</p></div> | |
| 11 | + <div class="team-member"><h3>Brian Jacobson</h3><p class="role">Chief Financial Officer</p><a href="/contact/brian">Contact</a></div> | |
| 12 | + <div class="team-member"><h3>Priya Natarajan</h3><a href="/contact/priya">Contact</a></div> | |
| 13 | + <div class="team-member"><h3>Chair Emeritus</h3><p class="role">Warner Bros. Discovery</p></div> | |
| 14 | + <div class="team-member"><h3>Board of Directors</h3><p class="role">Meet the people who govern the company</p></div> | |
| 15 | + <div class="team-member"><h3>Leadership</h3><p class="role">Read more</p></div> | |
| 16 | + <div class="team-member"><h3>Contact</h3><p class="role">Media relations</p></div> | |
| 17 | + <div class="team-member"><h3>Marc van der Berg</h3><p class="role">Chief Operating Officer</p><a href="/bio/marc">Read bio</a></div> | |
| 18 | + <div class="team-member"><p class="role">Chair of the Board</p><h3>Samuel Adebayo</h3></div> | |
| 19 | + </section> | |
| 20 | +</main> | |
| 21 | +<footer><p>© 2026 Warner Example</p></footer> | |
| 22 | +</body></html> | |
added
fixtures/connectors/generic_html/locations_cookie_noise.html
+30 −0
@@ -0,0 +1,30 @@ | ||
| 1 | +<!doctype html> | |
| 2 | +<html lang="en"> | |
| 3 | +<head><title>Our locations — Hormuz Example Group</title></head> | |
| 4 | +<body> | |
| 5 | +<header><nav><a href="/">Home</a><a href="/locations">Locations</a><a href="/contact-us">Contact Us</a></nav></header> | |
| 6 | +<div class="preferences-modal" role="dialog"> | |
| 7 | + <h2>Manage your preferences</h2> | |
| 8 | + <table class="preferences-table"> | |
| 9 | + <tr class="preference location"><td>Strictly Necessary</td><td>These are required for the website to function and cannot be switched off.</td></tr> | |
| 10 | + <tr class="preference location"><td>Performance & Analytics</td><td>Allows use of behavioural data to optimise performance. Third-party providers store aggregated data.</td></tr> | |
| 11 | + <tr class="preference location"><td>Functional</td><td>Enables enhanced functionality such as videos and live chat. Providers may store your settings.</td></tr> | |
| 12 | + <tr class="preference location"><td>Marketing</td><td>Used by advertising partners to build a profile of your interests and show relevant ads on other sites. Data is stored by the store.</td></tr> | |
| 13 | + </table> | |
| 14 | +</div> | |
| 15 | +<main> | |
| 16 | + <h1>Where to find us</h1> | |
| 17 | + <section class="offices"> | |
| 18 | + <h2>Offices</h2> | |
| 19 | + <div class="office-card"><h3>Oman</h3><p>Hormuz Grand Hotel</p><p>Oman</p></div> | |
| 20 | + <div class="office-card"><h3>Contact Us</h3><p>Contact Us</p><p>Seoul, KR</p></div> | |
| 21 | + <div class="office-card"><h3>Dubai Office</h3><p>Level 12, Emirates Towers<br>Dubai, United Arab Emirates</p></div> | |
| 22 | + <div class="office-card"><h3>London</h3><p>1 Finsbury Avenue, London EC2M 2PF, United Kingdom</p></div> | |
| 23 | + <div class="office-card"><h3>Singapore</h3><p>Singapore</p></div> | |
| 24 | + <div class="office-card"><h3>Rotterdam Warehouse</h3><p>Waalhaven Zuidzijde 21, 3089 JH Rotterdam, Nederland</p></div> | |
| 25 | + <div class="office-card"><h3>Careers</h3><p>Join our team in one of our offices around the world.</p><p>Seoul, KR</p></div> | |
| 26 | + <div class="office-card"><h3>Sign up for our newsletter</h3><p>Receive updates from our offices around the world.</p><p>Paris, France</p></div> | |
| 27 | + </section> | |
| 28 | +</main> | |
| 29 | +<footer><p>© 2026 Hormuz Example Group</p></footer> | |
| 30 | +</body></html> | |
added
fixtures/connectors/generic_html/pricing_marketing_noise.html
+56 −0
@@ -0,0 +1,56 @@ | ||
| 1 | +<!doctype html> | |
| 2 | +<html lang="en"> | |
| 3 | +<head><title>Pricing — Similar Example</title></head> | |
| 4 | +<body> | |
| 5 | +<header class="site-header"><nav><a href="/">Home</a><a href="/pricing">Pricing</a></nav></header> | |
| 6 | +<main> | |
| 7 | + <section class="hero pricing-card"> | |
| 8 | + <h1>Win your market with Similar Example for businesses</h1> | |
| 9 | + <p>Get the insights you need to outperform the competition.</p> | |
| 10 | + <a href="/contact-sales">Talk to sales</a> | |
| 11 | + </section> | |
| 12 | + <section class="pricing-grid"> | |
| 13 | + <h2>Plans</h2> | |
| 14 | + <div class="pricing-card"> | |
| 15 | + <h3>Free</h3> | |
| 16 | + <p class="price">$0 forever</p> | |
| 17 | + <ul><li>1 user</li><li>Basic reports</li></ul> | |
| 18 | + </div> | |
| 19 | + <div class="pricing-card"> | |
| 20 | + <h3>Starter</h3> | |
| 21 | + <p class="price">$125 per month, billed annually</p> | |
| 22 | + <ul><li>1 user</li><li>3 months of data</li></ul> | |
| 23 | + <a href="/signup?plan=starter">Start now</a> | |
| 24 | + </div> | |
| 25 | + <div class="pricing-card"> | |
| 26 | + <h3>Team</h3> | |
| 27 | + <p class="price">Starting at €333 / month</p> | |
| 28 | + <ul><li>3 users</li><li>15 months of data</li></ul> | |
| 29 | + </div> | |
| 30 | + <div class="pricing-card"> | |
| 31 | + <p class="eyebrow">Most popular</p> | |
| 32 | + <h3>Business</h3> | |
| 33 | + <p class="price">US$ 1,199 per user / month</p> | |
| 34 | + <ul><li>5 users</li><li>Custom data</li></ul> | |
| 35 | + </div> | |
| 36 | + <div class="pricing-card"> | |
| 37 | + <h3>Enterprise</h3> | |
| 38 | + <p class="price">Talk to sales</p> | |
| 39 | + <ul><li>Unlimited users</li><li>Dedicated success manager</li></ul> | |
| 40 | + </div> | |
| 41 | + <div class="pricing-card"> | |
| 42 | + <h3>Worry-free roaming.</h3> | |
| 43 | + <p>Stay connected in the U.S. ($13/day) and over 200 international destinations with our roaming add-on for travellers.</p> | |
| 44 | + </div> | |
| 45 | + <div class="pricing-card"> | |
| 46 | + <h3>Unlock the full potential of your data</h3> | |
| 47 | + <p class="price">$49 per month</p> | |
| 48 | + </div> | |
| 49 | + <div class="pricing-card"> | |
| 50 | + <h3>Add-on: extra seats</h3> | |
| 51 | + <p>Bring the whole team along, more information on request.</p> | |
| 52 | + </div> | |
| 53 | + </section> | |
| 54 | +</main> | |
| 55 | +<footer class="site-footer"><p>© 2026 Similar Example</p></footer> | |
| 56 | +</body></html> | |
added
fixtures/connectors/generic_html/products_news_noise.html
+29 −0
@@ -0,0 +1,29 @@ | ||
| 1 | +<!doctype html> | |
| 2 | +<html lang="en"> | |
| 3 | +<head><title>Products & News — Acme Example</title></head> | |
| 4 | +<body> | |
| 5 | +<nav class="navbar"><a href="/">Home</a><a href="/products">Products</a><a href="/news">News</a></nav> | |
| 6 | +<main> | |
| 7 | + <section class="products"> | |
| 8 | + <h2>Products</h2> | |
| 9 | + <div class="product-card"><h3>Overview</h3><p>Everything we make, in one place.</p><a href="/products">All products</a></div> | |
| 10 | + <div class="product-card"><h3>Atlas Metrics™</h3><p>High-cardinality metrics at any scale.</p><a href="/products/metrics">Learn more</a></div> | |
| 11 | + <div class="product-card"><h3>Atlas Logs®</h3><p>Search petabytes in seconds.</p><a href="/products/logs">Learn more</a></div> | |
| 12 | + <div class="product-card"><h3>Discover how Atlas helps teams ship faster.</h3><p>Slogan card</p><a href="/why">Learn more</a></div> | |
| 13 | + <div class="product-card"><h3>Learn more</h3><a href="/products/more">Learn more</a></div> | |
| 14 | + <div class="product-card"><h3>Solutions</h3><a href="/solutions">Solutions</a></div> | |
| 15 | + <div class="product-card"><h3>Atlas Traces</h3><p>Distributed tracing without sampling.</p><a href="/products/traces">Learn more</a></div> | |
| 16 | + </section> | |
| 17 | + <section class="news-list"> | |
| 18 | + <h2>Latest news</h2> | |
| 19 | + <article class="news-item"><time datetime="2026-09-10">September 10, 2026</time><h3><a href="/news/2026/09/acme-launches-atlas-ai">Acme launches Atlas AI, an assistant for cloud operations</a></h3></article> | |
| 20 | + <article class="news-item"><time datetime="2026-08-28">August 28, 2026</time><h3><a href="/news/2026/08/q2-results">Q2 results</a></h3></article> | |
| 21 | + <article class="news-item"><h3><a href="/news/press-releases">Press releases</a></h3></article> | |
| 22 | + <article class="news-item"><h3><a href="/news/category/blog">Blog posts</a></h3></article> | |
| 23 | + <article class="news-item"><h3><a href="/news/2026/07/acme-partners-with-bigco">Acme and BigCo announce strategic partnership</a></h3></article> | |
| 24 | + <article class="news-item"><h3><a href="/news?page=2">Older posts »</a></h3></article> | |
| 25 | + <article class="news-item"><h3><a href="/news/all">Read more news</a></h3></article> | |
| 26 | + </section> | |
| 27 | +</main> | |
| 28 | +<footer><p>© 2026 Acme Example</p></footer> | |
| 29 | +</body></html> | |
added
scripts/audit_extractions.py
+242 −0
@@ -0,0 +1,242 @@ | ||
| 1 | +"""Audit stored typed entities against the precision rules (`connectors/_precision`) — read-only by default. | |
| 2 | + | |
| 3 | +For each table it samples N rows (seeded, reproducible), applies the same validators the connector and the pipeline now apply, and | |
| 4 | +prints a precision estimate (share of rows the rules accept, with a 95 % interval) plus the rows that would now be rejected, grouped | |
| 5 | +by reason. Run it before and after a deploy to measure the effect on production. | |
| 6 | + | |
| 7 | + DATABASE_URL=… .venv/bin/python scripts/audit_extractions.py # sample 200 rows per table | |
| 8 | + .venv/bin/python scripts/audit_extractions.py --sample 500 --table people,locations --show 40 | |
| 9 | + .venv/bin/python scripts/audit_extractions.py --purge # delete rows failing the rules (see below) | |
| 10 | + | |
| 11 | +Scope: by default only rows attached to a `generic-html-v1` sensor are audited (the rows the rules were written for); `--all-sources` | |
| 12 | +widens the read-only audit to enrichment-sourced rows (e.g. Wikidata people) as a recall check on the validators. | |
| 13 | +`--purge` scans the *whole* table (not a sample) and deletes HTML-sourced rows that fail the validators, for `people`, `products`, | |
| 14 | +`pricing_plans`, `locations` and `jobs` (`sensors.connector_id = 'generic-html-v1'`); it never touches ATS jobs, enrichment rows or `news_items`. | |
| 15 | +People whose only defect is a stoplisted title ("Contact", "Read more") are kept with `title = null`. Rows the rules would *rename* | |
| 16 | +(swapped person cards, country-as-name locations) are reported, not rewritten: the next crawl re-extracts them correctly and the | |
| 17 | +reconciliation marks the old row `no_longer_listed`. Events derived from purged rows are history and stay untouched. | |
| 18 | +""" | |
| 19 | +from __future__ import annotations | |
| 20 | + | |
| 21 | +import argparse | |
| 22 | +import asyncio | |
| 23 | +import math | |
| 24 | +from collections import Counter, defaultdict | |
| 25 | +from collections.abc import Callable, Sequence | |
| 26 | +from dataclasses import dataclass, field | |
| 27 | +from typing import Any | |
| 28 | + | |
| 29 | +from companyatlas.connectors import _precision as P | |
| 30 | +from companyatlas.db import dispose, execute, fetch_all, transaction | |
| 31 | +from companyatlas.sdk.models import ExtractedLocation, ExtractedPlan | |
| 32 | + | |
| 33 | +DEFAULT_SAMPLE = 200 | |
| 34 | +DEFAULT_SHOW = 25 | |
| 35 | +DEFAULT_SEED = 0.42 | |
| 36 | +Z_95 = 1.96 | |
| 37 | +LABEL_WIDTH = 70 | |
| 38 | +HTML_CONNECTOR = "generic-html-v1" | |
| 39 | +PURGE_BATCH = 500 | |
| 40 | + | |
| 41 | + | |
| 42 | +@dataclass(slots=True) | |
| 43 | +class Outcome: | |
| 44 | + """What the rules say about one stored row.""" | |
| 45 | + status: str # accept | reject | normalise | |
| 46 | + reason: str | None = None | |
| 47 | + fix: str | None = None # normalise: what would change (for the report) — "title → null" is applied under --purge | |
| 48 | + | |
| 49 | + | |
| 50 | +@dataclass(slots=True) | |
| 51 | +class TableSpec: | |
| 52 | + name: str | |
| 53 | + label_col: str | |
| 54 | + select_sql: str # must select id + the columns `judge` reads; `{where}` and `{limit}` are filled in | |
| 55 | + judge: Callable[[dict[str, Any]], Outcome] | |
| 56 | + purgeable: bool | |
| 57 | + label: Callable[[dict[str, Any]], str] | |
| 58 | + summary_cols: Sequence[str] = field(default_factory=tuple) | |
| 59 | + | |
| 60 | + | |
| 61 | +# ------------------------------------------------------------------------------------------------------------ judges (one per table) | |
| 62 | + | |
| 63 | + | |
| 64 | +def _judge_person(row: dict[str, Any]) -> Outcome: | |
| 65 | + fixed = P.normalize_person(row["name"] or "", row.get("title")) | |
| 66 | + if fixed is None: | |
| 67 | + return Outcome("reject", P.person_verdict(row["name"] or "", row.get("title")).reason) | |
| 68 | + name, title = fixed | |
| 69 | + if name != (row["name"] or "").strip(): | |
| 70 | + return Outcome("normalise", "name/title swapped", fix=f"name → {name!r}, title → {title!r}") | |
| 71 | + if (title or None) != (row.get("title") or None): | |
| 72 | + return Outcome("normalise", "title is a link label", fix="title → null") | |
| 73 | + return Outcome("accept") | |
| 74 | + | |
| 75 | + | |
| 76 | +def _judge_product(row: dict[str, Any]) -> Outcome: | |
| 77 | + v = P.product_verdict(row["name"] or "") | |
| 78 | + return Outcome("accept") if v.ok else Outcome("reject", v.reason) | |
| 79 | + | |
| 80 | + | |
| 81 | +def _judge_plan(row: dict[str, Any]) -> Outcome: | |
| 82 | + plan = ExtractedPlan(plan_name=row["plan_name"] or "", price=float(row["price"]) if row.get("price") is not None else None, | |
| 83 | + price_text=row.get("price_text"), contact_sales=bool(row.get("contact_sales"))) | |
| 84 | + v = P.plan_verdict(plan) | |
| 85 | + return Outcome("accept") if v.ok else Outcome("reject", v.reason) | |
| 86 | + | |
| 87 | + | |
| 88 | +def _judge_location(row: dict[str, Any]) -> Outcome: | |
| 89 | + loc = ExtractedLocation(name=row["name"] or "", kind=row.get("kind") or "office", city=row.get("city"), region=row.get("region"), | |
| 90 | + country=row.get("country"), address_text=None) | |
| 91 | + v = P.location_verdict(loc) | |
| 92 | + if not v.ok: | |
| 93 | + return Outcome("reject", v.reason) | |
| 94 | + fixed = P.normalize_location(loc) | |
| 95 | + if fixed is not None and fixed.name != loc.name: | |
| 96 | + return Outcome("normalise", "country used as the name", fix=f"name → {fixed.name!r}, city → {fixed.city!r}") | |
| 97 | + if fixed is not None and fixed.city != loc.city: | |
| 98 | + return Outcome("normalise", "city cell is not a city", fix=f"city → {fixed.city!r}") | |
| 99 | + return Outcome("accept") | |
| 100 | + | |
| 101 | + | |
| 102 | +def _judge_job(row: dict[str, Any]) -> Outcome: | |
| 103 | + v = P.job_verdict(row["title"] or "", url=row.get("url"), location=row.get("location_text"), department=row.get("department")) | |
| 104 | + if not v.ok: | |
| 105 | + return Outcome("reject", v.reason) | |
| 106 | + clean, _ = P.clean_job_title(row["title"] or "") | |
| 107 | + if clean and clean != (row["title"] or "").strip(): | |
| 108 | + return Outcome("normalise", "display title carries a marker / id", fix=f"title → {clean!r}") | |
| 109 | + return Outcome("accept") | |
| 110 | + | |
| 111 | + | |
| 112 | +def _judge_news(row: dict[str, Any]) -> Outcome: | |
| 113 | + v = P.news_verdict(row["title"] or "", url=row.get("url"), published_at=row.get("published_at")) | |
| 114 | + return Outcome("accept") if v.ok else Outcome("reject", v.reason) | |
| 115 | + | |
| 116 | + | |
| 117 | +def _lbl(*parts: Any) -> str: | |
| 118 | + return " | ".join(str(p) for p in parts if p not in (None, "")) | |
| 119 | + | |
| 120 | + | |
| 121 | +# Every select joins `sensors` so rows can be scoped to the HTML connector (`{where}`): enrichment-sourced rows (Wikidata people…) | |
| 122 | +# have no HTML sensor and are only ever *audited* with --all-sources, never purged. | |
| 123 | +TABLES: dict[str, TableSpec] = { | |
| 124 | + "people": TableSpec("people", "name", "select t.id, t.name, t.title from people t left join sensors s on s.id = t.sensor_id {where} {limit}", | |
| 125 | + _judge_person, True, lambda r: _lbl(r["name"], r.get("title"))), | |
| 126 | + "products": TableSpec("products", "name", "select t.id, t.name from products t left join sensors s on s.id = t.sensor_id {where} {limit}", | |
| 127 | + _judge_product, True, lambda r: _lbl(r["name"])), | |
| 128 | + "pricing_plans": TableSpec("pricing_plans", "plan_name", | |
| 129 | + "select t.id, t.plan_name, t.price, t.price_text, t.contact_sales from pricing_plans t left join sensors s on s.id = t.sensor_id " | |
| 130 | + "{where} {limit}", _judge_plan, True, lambda r: _lbl(r["plan_name"], r.get("price_text"))), | |
| 131 | + "locations": TableSpec("locations", "name", | |
| 132 | + "select t.id, t.name, t.kind, t.city, t.region, t.country from locations t left join sensors s on s.id = t.sensor_id {where} {limit}", | |
| 133 | + _judge_location, True, lambda r: _lbl(r["name"], r.get("city"), r.get("country"), r.get("kind"))), | |
| 134 | + "jobs": TableSpec("jobs", "title", | |
| 135 | + "select t.id, t.title, t.url, t.location_text, t.department from jobs t left join sensors s on s.id = t.sensor_id {where} {limit}", | |
| 136 | + _judge_job, True, lambda r: _lbl(r["title"], r.get("location_text"))), | |
| 137 | + "news_items": TableSpec("news_items", "title", | |
| 138 | + "select t.id, t.title, t.url, t.published_at from news_items t left join sensors s on s.id = t.sensor_id {where} {limit}", | |
| 139 | + _judge_news, False, lambda r: _lbl(r["title"], r.get("url"))), | |
| 140 | +} | |
| 141 | +PURGEABLE = [t for t, spec in TABLES.items() if spec.purgeable] | |
| 142 | +HTML_ONLY_WHERE = f"where s.connector_id = '{HTML_CONNECTOR}'" | |
| 143 | + | |
| 144 | + | |
| 145 | +# ------------------------------------------------------------------------------------------------------------ reporting | |
| 146 | + | |
| 147 | + | |
| 148 | +def _interval(k: int, n: int) -> tuple[float, float]: | |
| 149 | + """Wilson 95 % interval for a proportion k/n.""" | |
| 150 | + if n == 0: | |
| 151 | + return 0.0, 0.0 | |
| 152 | + p = k / n | |
| 153 | + denom = 1 + Z_95 ** 2 / n | |
| 154 | + centre = (p + Z_95 ** 2 / (2 * n)) / denom | |
| 155 | + half = Z_95 * math.sqrt(p * (1 - p) / n + Z_95 ** 2 / (4 * n * n)) / denom | |
| 156 | + return max(0.0, centre - half), min(1.0, centre + half) | |
| 157 | + | |
| 158 | + | |
| 159 | +def _sql(spec: TableSpec, *, sample: int | None, html_only: bool) -> str: | |
| 160 | + where = HTML_ONLY_WHERE if (html_only or spec.name == "jobs") else "" # jobs: ATS boards are always out of scope | |
| 161 | + limit = f"order by random() limit {int(sample)}" if sample else "" | |
| 162 | + return spec.select_sql.format(where=where, limit=limit) | |
| 163 | + | |
| 164 | + | |
| 165 | +async def _audit_table(conn: Any, spec: TableSpec, *, sample: int | None, show: int, html_only: bool) -> tuple[list[str], list[str]]: | |
| 166 | + rows = await fetch_all(conn, _sql(spec, sample=sample, html_only=html_only)) | |
| 167 | + outcomes = [(r, spec.judge(r)) for r in rows] | |
| 168 | + accepted = sum(1 for _, o in outcomes if o.status == "accept") | |
| 169 | + normalised = [(r, o) for r, o in outcomes if o.status == "normalise"] | |
| 170 | + rejected = [(r, o) for r, o in outcomes if o.status == "reject"] | |
| 171 | + n = len(rows) | |
| 172 | + lo, hi = _interval(accepted + len(normalised), n) | |
| 173 | + print(f"\n== {spec.name}: sampled {n} · accepted {accepted} · normalised {len(normalised)} · rejected {len(rejected)}") | |
| 174 | + if n: | |
| 175 | + print(f" precision estimate (accepted + normalisable) = {(accepted + len(normalised)) / n:.1%} [{lo:.1%}, {hi:.1%}]" | |
| 176 | + f" rejected share = {len(rejected) / n:.1%}") | |
| 177 | + reasons: Counter[str] = Counter(o.reason or "?" for _, o in rejected) | |
| 178 | + by_reason: dict[str, list[str]] = defaultdict(list) | |
| 179 | + for r, o in rejected: | |
| 180 | + by_reason[o.reason or "?"].append(spec.label(r)) | |
| 181 | + for reason, count in reasons.most_common(): | |
| 182 | + print(f" - {reason}: {count}") | |
| 183 | + for lbl in by_reason[reason][:show]: | |
| 184 | + print(f" ✗ {lbl[:LABEL_WIDTH]}") | |
| 185 | + if normalised: | |
| 186 | + print(f" ~ would be normalised ({len(normalised)}):") | |
| 187 | + for r, o in normalised[:show]: | |
| 188 | + print(f" ~ {spec.label(r)[:LABEL_WIDTH]} → {o.fix}") | |
| 189 | + return [r["id"] for r, _ in rejected], [r["id"] for r, o in normalised if o.fix == "title → null"] | |
| 190 | + | |
| 191 | + | |
| 192 | +async def _purge(conn: Any, spec: TableSpec, reject_ids: list[str], null_title_ids: list[str]) -> None: | |
| 193 | + for i in range(0, len(reject_ids), PURGE_BATCH): | |
| 194 | + await execute(conn, f"delete from {spec.name} where id = any(cast(:ids as text[]))", ids=reject_ids[i:i + PURGE_BATCH]) | |
| 195 | + if spec.name == "people": | |
| 196 | + for i in range(0, len(null_title_ids), PURGE_BATCH): | |
| 197 | + await execute(conn, "update people set title = null where id = any(cast(:ids as text[]))", ids=null_title_ids[i:i + PURGE_BATCH]) | |
| 198 | + print(f" purged {len(reject_ids)} row(s) from {spec.name}" + (f", cleared {len(null_title_ids)} title(s)" if spec.name == "people" else "")) | |
| 199 | + | |
| 200 | + | |
| 201 | +async def main(args: argparse.Namespace) -> None: | |
| 202 | + names = [t.strip() for t in args.table.split(",")] if args.table else list(TABLES) | |
| 203 | + unknown = [t for t in names if t not in TABLES] | |
| 204 | + if unknown: | |
| 205 | + raise SystemExit(f"unknown table(s): {unknown}; choose from {list(TABLES)}") | |
| 206 | + if args.purge: | |
| 207 | + names = [t for t in names if TABLES[t].purgeable] | |
| 208 | + print(f"PURGE mode: full scan of {names}; rows failing the precision rules will be deleted (news and ATS jobs are never touched)") | |
| 209 | + sample = None if args.purge else args.sample | |
| 210 | + html_only = args.purge or not args.all_sources # a purge is always scoped to HTML-sourced rows | |
| 211 | + print(f"precision rules {P.PRECISION_VERSION} · sample={'all' if sample is None else sample} · seed={args.seed} · " | |
| 212 | + f"scope={'HTML-sourced rows (generic-html-v1)' if html_only else 'all sources'}") | |
| 213 | + async with transaction() as conn: | |
| 214 | + await execute(conn, "select setseed(:s)", s=args.seed) | |
| 215 | + for name in names: | |
| 216 | + spec = TABLES[name] | |
| 217 | + reject_ids, null_title_ids = await _audit_table(conn, spec, sample=sample, show=args.show, html_only=html_only) | |
| 218 | + if args.purge: | |
| 219 | + await _purge(conn, spec, reject_ids, null_title_ids) | |
| 220 | + print("\ndone" if args.purge else "\nread-only audit done (pass --purge to delete the rejected rows of the purgeable tables)") | |
| 221 | + | |
| 222 | + | |
| 223 | +def _parse() -> argparse.Namespace: | |
| 224 | + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) | |
| 225 | + ap.add_argument("--sample", type=int, default=DEFAULT_SAMPLE, help=f"rows sampled per table (default {DEFAULT_SAMPLE})") | |
| 226 | + ap.add_argument("--table", type=str, default="", help=f"comma-separated subset of {list(TABLES)}") | |
| 227 | + ap.add_argument("--show", type=int, default=DEFAULT_SHOW, help="examples printed per reason") | |
| 228 | + ap.add_argument("--seed", type=float, default=DEFAULT_SEED, help="Postgres setseed() value in [-1, 1] for reproducible samples") | |
| 229 | + ap.add_argument("--purge", action="store_true", help=f"delete rows failing the rules in {PURGEABLE} (full scan, not a sample, HTML-sourced only)") | |
| 230 | + ap.add_argument("--all-sources", action="store_true", help="read-only audit over every source (enrichment rows included); ignored with --purge") | |
| 231 | + return ap.parse_args() | |
| 232 | + | |
| 233 | + | |
| 234 | +async def _run() -> None: | |
| 235 | + try: | |
| 236 | + await main(_parse()) | |
| 237 | + finally: | |
| 238 | + await dispose() | |
| 239 | + | |
| 240 | + | |
| 241 | +if __name__ == "__main__": | |
| 242 | + asyncio.run(_run()) | |
added
src/companyatlas/connectors/_precision.py
+764 −0
@@ -0,0 +1,764 @@ | ||
| 1 | +"""Precision rules for typed extractions (spec §107, "never fabricate"): deterministic validators and normalisers that reject the | |
| 2 | +navigation / call-to-action / cookie-consent / marketing noise the generic HTML connector picks up on real corporate pages, and | |
| 3 | +repair the classic confusions (person name ↔ title swapped, country used as a location name, truncated price text). | |
| 4 | + | |
| 5 | +Shared by `connectors/generic_html.py` (applied while extracting) and `services/pipeline._drop_corrupt_entities` (last line of | |
| 6 | +defence for every connector) and by `scripts/audit_extractions.py` (measurement / purge on stored rows). Every rule is a small pure | |
| 7 | +function returning a `Verdict`, so the same code explains *why* a row is rejected. Multilingual where cheap (EN/FR/DE/NL/ES/IT/PT/JA). | |
| 8 | +No network, no LLM. | |
| 9 | +""" | |
| 10 | +from __future__ import annotations | |
| 11 | + | |
| 12 | +import re | |
| 13 | +import unicodedata | |
| 14 | +from collections.abc import Iterable | |
| 15 | +from dataclasses import dataclass | |
| 16 | +from datetime import datetime | |
| 17 | +from typing import Any | |
| 18 | + | |
| 19 | +from companyatlas.connectors._util import POSTAL_RE, country_code, date_from_url, parse_location | |
| 20 | +from companyatlas.sdk.models import ( | |
| 21 | + ExtractedJob, | |
| 22 | + ExtractedLocation, | |
| 23 | + ExtractedNewsItem, | |
| 24 | + ExtractedPerson, | |
| 25 | + ExtractedPlan, | |
| 26 | + ExtractedProduct, | |
| 27 | + Extraction, | |
| 28 | +) | |
| 29 | +from companyatlas.sdk.normalize import normalize_whitespace | |
| 30 | + | |
| 31 | +PRECISION_VERSION = "precision-v1" | |
| 32 | + | |
| 33 | +# ------------------------------------------------------------------------------------------------------------ limits (no magic numbers) | |
| 34 | + | |
| 35 | +MAX_JOB_TITLE_CHARS = 140 | |
| 36 | +MIN_JOB_TITLE_CHARS = 4 | |
| 37 | +MIN_JOB_TITLE_WORDS = 2 | |
| 38 | +PROSE_MIN_WORDS = 4 # a "title" with ≥ 4 words and function words reads like a sentence | |
| 39 | +PERSON_NAME_MIN_TOKENS, PERSON_NAME_MAX_TOKENS = 2, 5 | |
| 40 | +MIN_PERSON_NAME_CHARS, MAX_PERSON_NAME_CHARS = 4, 60 | |
| 41 | +MAX_PERSON_TITLE_CHARS = 100 | |
| 42 | +SENTENCE_TITLE_MIN_WORDS = 6 # a person "title" of ≥ 6 words ending with a full stop is a bio sentence | |
| 43 | +MAX_LOCATION_NAME_CHARS = 80 | |
| 44 | +MAX_LOCATION_NAME_WORDS = 8 | |
| 45 | +MAX_CITY_CHARS, MAX_CITY_WORDS = 40, 4 | |
| 46 | +VENUE_MIN_WORDS = 3 # "Hormuz Grand Hotel": ≥ 3 words in the city slot is a venue, not a city | |
| 47 | +MAX_PLAN_NAME_CHARS, MAX_PLAN_NAME_WORDS = 40, 5 | |
| 48 | +MAX_PRICE_TEXT_CHARS = 60 | |
| 49 | +MAX_PRODUCT_NAME_CHARS, MAX_PRODUCT_NAME_WORDS = 80, 10 | |
| 50 | +NEWS_MIN_WORDS, NEWS_MIN_CHARS = 3, 15 | |
| 51 | +NEWS_MIN_CJK_CHARS = 6 # CJK titles have no word boundaries: 6 characters already carry a headline | |
| 52 | +MAX_DEPARTMENT_CHARS, MAX_DEPARTMENT_WORDS = 40, 4 | |
| 53 | +CITY_NGRAM_MAX = 3 | |
| 54 | +YEAR_RE = re.compile(r"^(?:19|20)\d{2}$") | |
| 55 | + | |
| 56 | + | |
| 57 | +@dataclass(slots=True, frozen=True) | |
| 58 | +class Verdict: | |
| 59 | + ok: bool | |
| 60 | + reason: str | None = None | |
| 61 | + | |
| 62 | + @staticmethod | |
| 63 | + def accept() -> Verdict: | |
| 64 | + return Verdict(True, None) | |
| 65 | + | |
| 66 | + @staticmethod | |
| 67 | + def reject(reason: str) -> Verdict: | |
| 68 | + return Verdict(False, reason) | |
| 69 | + | |
| 70 | + | |
| 71 | +# ------------------------------------------------------------------------------------------------------------ text helpers | |
| 72 | + | |
| 73 | +_WORD_RE = re.compile(r"[^\W_]+", re.UNICODE) | |
| 74 | +_CJK_RE = re.compile(r"[-ヿ㐀-鿿]") | |
| 75 | +TERMINAL_PUNCT = ".!?:;," | |
| 76 | +TRAILING_MARKS_RE = re.compile(r"[\s→➔➡›»>\-–—|·•]+$") # arrows, chevrons, dashes, bullets | |
| 77 | +ELLIPSIS_RE = re.compile(r"(?:\.\.\.|…)\s*$") | |
| 78 | +PROSE_FUNCTION_WORDS_RE = re.compile( | |
| 79 | + r"\b(?:who|that|which|want|wants|with|for|your|you|we|our|are|is|the|die|dat|willen|voor|met|jouw|onze|wij|zijn|qui|que|pour|avec|votre|nous|" | |
| 80 | + r"sont|est|les|der|das|mit|für|und|ihre|wir|sind|para|con|nuestro|somos|per|che|nostro|siamo|com|nosso|somos)\b", re.IGNORECASE) | |
| 81 | +SENTENCE_VERB_RE = re.compile( | |
| 82 | + r"\b(?:allows?|enables?|helps?|uses?|provides?|lets?|makes?|gives?|ensures?|collects?|stores?|can|will|would|should|permet|permettent|utilise|" | |
| 83 | + r"erlaubt|ermöglicht|verwendet|gebruikt|permite|consente|permitem)\b", re.IGNORECASE) | |
| 84 | +MARKETING_VERB_RE = re.compile( | |
| 85 | + r"^(?:win|get|start|discover|unlock|boost|grow|try|choose|compare|see|learn|find|build|join|save|upgrade|buy|sign up|request|book|schedule|" | |
| 86 | + r"contact|talk|let'?s|ready|why|how|what|explore|meet|introducing|welcome|enjoy|make|take|achieve|transform|power|scale|drive|accelerate|" | |
| 87 | + r"supercharge|level up|bring|stay|experience|gagnez|obtenez|découvrez|commencez|essayez|choisissez|comparez|rejoignez|débloquez|profitez|" | |
| 88 | + r"gewinnen|holen|starten|entdecken|testen|wählen|vergleichen|jetzt|ontdek|start|kies|vergelijk|probeer|gana|obtén|empieza|descubre|prueba|" | |
| 89 | + r"elige|compara|scopri|inizia|prova|scegli|confronta|ottieni|ganhe|obtenha|comece|descubra|experimente|escolha)\b", re.IGNORECASE) | |
| 90 | + | |
| 91 | + | |
| 92 | +def word_count(text: str) -> int: | |
| 93 | + return len(_WORD_RE.findall(text)) | |
| 94 | + | |
| 95 | + | |
| 96 | +def key_of(text: str | None) -> str: | |
| 97 | + """Accent-insensitive, lower-case, single-spaced key (keeps CJK) used for stoplists and the city table.""" | |
| 98 | + if not text: | |
| 99 | + return "" | |
| 100 | + nfkd = unicodedata.normalize("NFKD", text) | |
| 101 | + stripped = "".join(ch for ch in nfkd if not unicodedata.combining(ch)) | |
| 102 | + return " ".join(_WORD_RE.findall(stripped.lower())) | |
| 103 | + | |
| 104 | + | |
| 105 | +def strip_trailing_marks(text: str) -> str: | |
| 106 | + return TRAILING_MARKS_RE.sub("", normalize_whitespace(text)).strip() | |
| 107 | + | |
| 108 | + | |
| 109 | +def has_terminal_punct(text: str) -> bool: | |
| 110 | + t = text.rstrip("®™ ") | |
| 111 | + return bool(t) and t[-1] in TERMINAL_PUNCT | |
| 112 | + | |
| 113 | + | |
| 114 | +def is_prose(text: str) -> bool: | |
| 115 | + """A label that reads like a sentence: several words with function words, a sentence verb, or a full stop at the end.""" | |
| 116 | + words = word_count(text) | |
| 117 | + if words < PROSE_MIN_WORDS: | |
| 118 | + return False | |
| 119 | + if text.rstrip().endswith((".", "!", "?")): | |
| 120 | + return True | |
| 121 | + return bool(SENTENCE_VERB_RE.search(text)) or len(PROSE_FUNCTION_WORDS_RE.findall(text)) >= 2 | |
| 122 | + | |
| 123 | + | |
| 124 | +def is_slogan(text: str) -> bool: | |
| 125 | + return bool(MARKETING_VERB_RE.match(text)) and word_count(text) >= PROSE_MIN_WORDS | |
| 126 | + | |
| 127 | + | |
| 128 | +def _phrase_re(phrases: Iterable[str]) -> re.Pattern[str]: | |
| 129 | + alts = sorted({p.strip().lower() for p in phrases if p.strip()}, key=len, reverse=True) | |
| 130 | + return re.compile(r"^(?:" + "|".join(re.escape(a) for a in alts) + r")$", re.IGNORECASE) | |
| 131 | + | |
| 132 | + | |
| 133 | +# ------------------------------------------------------------------------------------------------------------ role vocabulary (people) | |
| 134 | + | |
| 135 | +ROLE_RULES: list[tuple[str, re.Pattern[str]]] = [ | |
| 136 | + ("founder", re.compile(r"\b(co-?founder|founder|fondat(eur|rice)|gründer(in)?|fondator[ei]|fundador(a)?|oprichter|medeoprichter)\b", re.IGNORECASE)), | |
| 137 | + ("ceo", re.compile(r"\b(chief executive( officer)?|ceo|pdg|président-directeur|geschäftsführer(in)?|managing director|directeur général|" | |
| 138 | + r"algemeen directeur|consejero delegado|amministratore delegato|diretor executivo)\b", re.IGNORECASE)), | |
| 139 | + ("cfo", re.compile(r"\b(chief financial( officer)?|cfo|finanzvorstand|directeur financier|financieel directeur|director financiero|direttore finanziario)\b", re.IGNORECASE)), | |
| 140 | + ("cto", re.compile(r"\b(chief technology( officer)?|chief technical( officer)?|cto|directeur technique)\b", re.IGNORECASE)), | |
| 141 | + ("coo", re.compile(r"\b(chief operating( officer)?|coo|directeur des opérations)\b", re.IGNORECASE)), | |
| 142 | + ("chair", re.compile(r"\b(chair(man|woman|person)?|executive chair|présidente? du conseil|vorsitzende[rn]?|voorzitter|presidente del consejo|" | |
| 143 | + r"presidente del consiglio)\b", re.IGNORECASE)), | |
| 144 | + ("president", re.compile(r"\b(?<!vice )(?<!vice-)president(?! of)|président(?!e? du conseil)\b", re.IGNORECASE)), | |
| 145 | + ("board", re.compile(r"\b(board member|member of the (supervisory |advisory )?board|non-executive director|independent director|" | |
| 146 | + r"administrat(eur|rice)|aufsichtsrat|conseil d'administration|director(?= \(board)|trustee|bestuurslid|consejer[oa]|consigliere)\b", re.IGNORECASE)), | |
| 147 | + ("vp", re.compile(r"\b(vice[- ]president|vp|evp|svp|avp|vice-président(e)?)\b", re.IGNORECASE)), | |
| 148 | + ("head", re.compile(r"\b(head of|head,|general manager|gm|leiter(in)?|directeur|directrice|director|managing partner|partner|responsable|hoofd|" | |
| 149 | + r"direttore|diretor(a)?|directora)\b", re.IGNORECASE)), | |
| 150 | +] | |
| 151 | +# Japanese titles have no word boundaries: plain substring rules, most specific first. | |
| 152 | +ROLE_RULES_JA: list[tuple[str, re.Pattern[str]]] = [ | |
| 153 | + ("founder", re.compile(r"創業者|共同創業者|ファウンダー")), | |
| 154 | + ("cfo", re.compile(r"最高財務責任者|CFO")), | |
| 155 | + ("cto", re.compile(r"最高技術責任者|CTO")), | |
| 156 | + ("coo", re.compile(r"最高執行責任者|COO")), | |
| 157 | + ("ceo", re.compile(r"代表取締役|最高経営責任者|社長|CEO")), | |
| 158 | + ("chair", re.compile(r"取締役会長|会長|議長")), | |
| 159 | + ("vp", re.compile(r"副社長|執行役員|バイスプレジデント")), | |
| 160 | + ("board", re.compile(r"社外取締役|取締役|監査役|理事")), | |
| 161 | + ("head", re.compile(r"本部長|事業部長|部長|統括|責任者|マネージャー|ディレクター")), | |
| 162 | +] | |
| 163 | +CHIEF_RE = re.compile(r"\b(chief\b|c[a-z]{1,3}o\b)|最高.{1,8}責任者", re.IGNORECASE) | |
| 164 | +EXEC_CATEGORIES = frozenset({"ceo", "cfo", "cto", "coo", "founder", "president", "chair"}) | |
| 165 | +TITLE_WORD_RE = re.compile(r"\b(officer|manager|engineer|lead|counsel|scientist|architect|analyst|controller|secretary|treasurer|advisor|adviser|" | |
| 166 | + r"strategist|evangelist|designer|emeritus|emerita|executive|principal|fellow|associate|specialist|consultant|" | |
| 167 | + r"ingénieur|responsable|chargée?|gérant(e)?|associée?|manager|leiter(in)?|mitglied|vorstand|directeur|directrice|" | |
| 168 | + r"adviseur|bestuurder|gerente|socio|sócio|membro|miembro)\b", re.IGNORECASE) | |
| 169 | +TITLE_WORD_JA_RE = re.compile(r"取締役|執行役員|部長|社長|会長|役員|担当|責任者|マネージャー|ディレクター|エンジニア|顧問|監査役") | |
| 170 | + | |
| 171 | + | |
| 172 | +def role_category(title: str | None) -> tuple[str, bool]: | |
| 173 | + """(category, is_executive) for a person's title — 'other' when nothing in the vocabulary matches.""" | |
| 174 | + if not title: | |
| 175 | + return "other", False | |
| 176 | + for cat, pat in ROLE_RULES: | |
| 177 | + if pat.search(title): | |
| 178 | + return cat, cat in EXEC_CATEGORIES or bool(CHIEF_RE.search(title)) | |
| 179 | + for cat, pat in ROLE_RULES_JA: | |
| 180 | + if pat.search(title): | |
| 181 | + return cat, cat in EXEC_CATEGORIES or bool(CHIEF_RE.search(title)) | |
| 182 | + return "other", bool(CHIEF_RE.search(title)) | |
| 183 | + | |
| 184 | + | |
| 185 | +def looks_like_role_title(text: str) -> bool: | |
| 186 | + t = normalize_whitespace(text) | |
| 187 | + if not (2 <= len(t) <= MAX_PERSON_TITLE_CHARS): | |
| 188 | + return False | |
| 189 | + return role_category(t)[0] != "other" or bool(TITLE_WORD_RE.search(t)) or bool(TITLE_WORD_JA_RE.search(t)) or bool(CHIEF_RE.search(t)) | |
| 190 | + | |
| 191 | + | |
| 192 | +# ------------------------------------------------------------------------------------------------------------ people | |
| 193 | + | |
| 194 | +_UP, _LOW = r"A-ZÀ-ÝĀ-Ž", r"a-zà-ÿā-ž'’" | |
| 195 | +NAME_TOKEN = (rf"(?:[{_UP}](?:['’][{_UP}])?[{_LOW}]*(?:[{_UP}][{_LOW}]+)?(?:-[{_UP}{_LOW}][{_LOW}]*)*|" # Jane · O'Neill · McGrath · García-López | |
| 196 | + rf"[{_UP}]\.|" # initial | |
| 197 | + rf"(?:de|van|von|der|den|ter|ten|da|di|do|dos|das|le|la|los|las|du|del|della|bin|ibn|al|el|y|e|of|op)\b|" | |
| 198 | + rf"(?:de|van|von|da|di|le|la|du|d|l)[{_UP}][{_LOW}]+)") # deSouza · DiCaprio · LeBlanc | |
| 199 | +NAME_SUFFIX = r"(?:,?\s+(?:Jr\.?|Sr\.?|II|III|IV|PhD|MD|MBA|CPA|Esq\.?))?" | |
| 200 | +NAME_RE = re.compile(rf"^(?:(?:Dr|Prof|Mr|Mrs|Ms|Sir|Dame|Hon)\.?\s+)?{NAME_TOKEN}(?:\s+{NAME_TOKEN}){{{PERSON_NAME_MIN_TOKENS - 1},{PERSON_NAME_MAX_TOKENS - 1}}}" | |
| 201 | + rf"{NAME_SUFFIX}$") | |
| 202 | +NAME_SUFFIX_TAIL_RE = re.compile(r"^\s*(?:Jr\.?|Sr\.?|II|III|IV|PhD|MD|MBA|CPA|Esq\.?)\s*$", re.IGNORECASE) | |
| 203 | +ABBREVIATION_END_RE = re.compile(r"\b(?:Jr|Sr|[A-Z])\.$") | |
| 204 | +JA_NAME_RE = re.compile(r"^[一-鿿-ヿ]{1,5}(?:[\s ・]+[一-鿿-ヿ]{1,5})?$") | |
| 205 | +JA_NOT_NAME_RE = re.compile(r"代表|取締役|社長|会長|役員|部長|執行|責任者|最高|担当|マネージャー|ディレクター|チーム|株式会社|有限会社|会社|事業|本部|営業|経営|" | |
| 206 | + r"お問い合わせ|プロフィール|詳細|一覧|紹介|概要|採用|ニュース|ホーム") | |
| 207 | +NOT_NAME_RE = re.compile( | |
| 208 | + r"\b(team|our|meet|leadership|board|executive|management|contact|about|officer|director|directors|president|manager|head|chief|founder|" | |
| 209 | + r"partner|partners|group|company|global|senior|vice|read|more|learn|view|profile|bio|linkedin|email|join|careers|news|press|the|and|of|for|" | |
| 210 | + r"at|in|on|to|with|chair|chairman|emeritus|emerita|vp|advisors?|advisory|committee|members?|staff|people|employees|équipe|direction|" | |
| 211 | + r"conseil|kontakt|vorstand|geschäftsführung|équipe de direction|equipo|dirección|squadra|direzione|diretoria|equipe)\b", re.IGNORECASE) | |
| 212 | +GENERIC_PERSON_NAMES_RE = _phrase_re([ | |
| 213 | + "contact", "contact us", "team", "our team", "the team", "meet the team", "board", "board of directors", "directors", "leadership", "our leadership", | |
| 214 | + "leadership team", "senior leadership", "executive team", "executive committee", "management", "management team", "our people", "advisors", | |
| 215 | + "advisory board", "about us", "kontakt", "unser team", "vorstand", "geschäftsführung", "aufsichtsrat", "équipe", "notre équipe", "direction", | |
| 216 | + "conseil d'administration", "comité de direction", "equipo", "nuestro equipo", "dirección", "consejo", "squadra", "direzione", "equipe", | |
| 217 | + "diretoria", "役員", "経営陣", "チーム", "お問い合わせ", "会社概要", | |
| 218 | +]) | |
| 219 | +PERSON_TITLE_STOPLIST_RE = _phrase_re([ | |
| 220 | + "contact", "contact us", "read more", "read bio", "view bio", "full bio", "bio", "biography", "profile", "view profile", "linkedin", "twitter", "x", | |
| 221 | + "email", "e-mail", "phone", "download", "download photo", "download bio", "website", "more", "learn more", "details", "en savoir plus", "voir le profil", | |
| 222 | + "profil", "kontakt", "lebenslauf", "mehr erfahren", "biografie", "biografía", "ver perfil", "leggi la biografia", "meer info", "lees meer", | |
| 223 | + "プロフィール", "詳細を見る", "詳細", | |
| 224 | +]) | |
| 225 | + | |
| 226 | + | |
| 227 | +def _strip_name(text: str) -> str: | |
| 228 | + """Trim punctuation around a name but keep a final abbreviation dot ("Edgar S. Woolard, Jr.").""" | |
| 229 | + t = normalize_whitespace(text).strip(" ,:;-–—|") | |
| 230 | + return t if ABBREVIATION_END_RE.search(t) else t.rstrip(".").strip() | |
| 231 | + | |
| 232 | + | |
| 233 | +def looks_like_person_name(text: str) -> bool: | |
| 234 | + """2–5 capitalised tokens (particles, initials, Mc/Mac, glued particles allowed), no digits, no role vocabulary; a peerage or | |
| 235 | + post-nominal tail after a comma ("…, Baron Trotman") is ignored; or a short CJK name without role words.""" | |
| 236 | + t = _strip_name(text) | |
| 237 | + if not t or any(ch.isdigit() for ch in t): | |
| 238 | + return False | |
| 239 | + if _CJK_RE.search(t): | |
| 240 | + return bool(JA_NAME_RE.match(t)) and not JA_NOT_NAME_RE.search(t) and 2 <= len(t.replace(" ", "").replace(" ", "")) <= 10 | |
| 241 | + head, _, tail = t.partition(",") | |
| 242 | + core = t if (not tail or NAME_SUFFIX_TAIL_RE.match(tail)) else head.strip() | |
| 243 | + if not (MIN_PERSON_NAME_CHARS <= len(core) <= MAX_PERSON_NAME_CHARS): | |
| 244 | + return False | |
| 245 | + if NOT_NAME_RE.search(core) or GENERIC_PERSON_NAMES_RE.match(core): | |
| 246 | + return False | |
| 247 | + return bool(NAME_RE.match(core)) | |
| 248 | + | |
| 249 | + | |
| 250 | +def clean_person_title(title: str | None) -> str | None: | |
| 251 | + """Stoplisted link labels (Contact, Read more, LinkedIn…), bio sentences and over-long strings are not titles.""" | |
| 252 | + if not title: | |
| 253 | + return None | |
| 254 | + t = strip_trailing_marks(title) | |
| 255 | + if not t or PERSON_TITLE_STOPLIST_RE.match(t) or len(t) > MAX_PERSON_TITLE_CHARS: | |
| 256 | + return None | |
| 257 | + if word_count(t) >= SENTENCE_TITLE_MIN_WORDS and t.endswith((".", "!", "?")): | |
| 258 | + return None | |
| 259 | + if is_prose(t) and not looks_like_role_title(t): | |
| 260 | + return None | |
| 261 | + return t.rstrip(".") if not t.endswith("...") else t | |
| 262 | + | |
| 263 | + | |
| 264 | +def normalize_person(name: str, title: str | None) -> tuple[str, str | None] | None: | |
| 265 | + """Validated (name, title) — swapped when the card put the title first — or None when the row is not a person.""" | |
| 266 | + n = _strip_name(name) | |
| 267 | + t = normalize_whitespace(title).strip(" ,:;-–—|") if title else None # keep a final "." — it marks a bio sentence | |
| 268 | + if not looks_like_person_name(n): | |
| 269 | + if t and looks_like_person_name(t) and looks_like_role_title(n): | |
| 270 | + n, t = t, n | |
| 271 | + else: | |
| 272 | + return None | |
| 273 | + return n, clean_person_title(t) | |
| 274 | + | |
| 275 | + | |
| 276 | +def person_verdict(name: str, title: str | None) -> Verdict: | |
| 277 | + fixed = normalize_person(name, title) | |
| 278 | + if fixed is None: | |
| 279 | + if looks_like_role_title(name): | |
| 280 | + return Verdict.reject("name is a role title") | |
| 281 | + if GENERIC_PERSON_NAMES_RE.match(_strip_name(name)): | |
| 282 | + return Verdict.reject("generic label, not a person") | |
| 283 | + return Verdict.reject("not a person name") | |
| 284 | + return Verdict.accept() | |
| 285 | + | |
| 286 | + | |
| 287 | +# ------------------------------------------------------------------------------------------------------------ jobs | |
| 288 | + | |
| 289 | +JOB_ROLE_RE = re.compile( | |
| 290 | + r"\b(?:engineer|ingénieur|ingenieur|ingeniero|ingegnere|engenheiro|developer|développeur|entwickler|desarrollador|sviluppatore|desenvolvedor|" | |
| 291 | + r"ontwikkelaar|programmer|programmeur|manager|gestionnaire|gerente|analyst|analyste|analista|designer|director|directeur|directrice|direktor|" | |
| 292 | + r"diretor|direttore|specialist|spécialiste|spezialist|especialista|specialista|intern|internship|stagiaire|praktikant|praktikum|stagiair|stagista|" | |
| 293 | + r"tirocinio|tirocinante|becario|estagiário|estágio|alternan(?:t|ce)|apprenti|apprentice|auszubildende[rn]?|ausbildung|azubi|werkstudent|" | |
| 294 | + r"sales|consultant|consulente|consultor|berater|adviseur|technician|technicien|techniker|técnico|tecnico|monteur|nurse|infirmi(?:er|ère)|" | |
| 295 | + r"krankenpfleger|verpleegkundige|enfermer[oa]|infermier[ea]|pflegefachkraft|pflegekraft|driver|chauffeur|fahrer|conductor|autista|motorista|officer|" | |
| 296 | + r"lead|head|associate|coordinator|coordinat(?:eur|rice)|koordinator|coördinator|coordinador|coordinatore|coordenador|architect|architecte|" | |
| 297 | + r"architekt|arquitecto|architetto|arquiteto|scientist|scientifique|wissenschaftler|científico|scienziato|cientista|accountant|comptable|" | |
| 298 | + r"buchhalter|contable|contabile|contador|operator|opérateur|operador|operatore|mechanic|mécanicien|mechaniker|mecánico|meccanico|mecânico|" | |
| 299 | + r"representative|représentant|assistant|assistent|asistente|assistente|administrator|administrateur|administrador|amministratore|advisor|" | |
| 300 | + r"adviser|counsel|attorney|lawyer|avocat|jurist|recruiter|controller|supervisor|superviseur|planner|buyer|acheteur|einkäufer|trainee|graduate|" | |
| 301 | + r"student|étudiant|studierende|estudiante|responsable|chargée?|chef de|leiter(?:in)?|mitarbeiter(?:in)?|referent(?:in)?|sachbearbeiter(?:in)?|" | |
| 302 | + r"kaufmann|kauffrau|fachkraft|medewerker|teamleider|projectleider|projektleiter|accountmanager|jefe|addetto|impiegato|responsabile|executive|" | |
| 303 | + r"principal|partner|paralegal|pharmacist|physician|therapist|teacher|professor|researcher|chercheur|forscher|electrician|électricien|elektriker|" | |
| 304 | + r"plumber|welder|soudeur|schweißer|machinist|cook|chef|barista|cashier|clerk|agent|guard|cleaner|handler|installer|inspector|auditor|actuary|" | |
| 305 | + r"underwriter|banker|trader|economist|strategist|producer|editor|writer|rédacteur|redakteur|translator|marketer|merchandiser|dispatcher|steward|" | |
| 306 | + r"pilot|captain|technologist|veterinarian|dentist|dietitian|paramedic|caregiver|educator|industriemechaniker|softwareentwickler|elektroniker|" | |
| 307 | + r"mechatroniker|pfleger|erzieher(?:in)?|verkäufer(?:in)?|vendeur|vendeuse|verkoper|commercial|comercial|venditore|vendedor)s?\b|" | |
| 308 | + r"エンジニア|マネージャー|マネジャー|デザイナー|ディレクター|コンサルタント|スペシャリスト|アナリスト|セールス|営業|開発|正社員|契約社員|インターン|職|担当", re.IGNORECASE) | |
| 309 | +JOB_FORMAT_RE = re.compile(r"\(\s*(?:[mfwhvdx]|all genders|alle geschlechter|tous genres)(?:\s*/\s*[mfwhvdx*])*\s*\)|\b(?:full[- ]?time|part[- ]?time|vollzeit|" | |
| 310 | + r"teilzeit|temps plein|temps partiel|fulltime|parttime|cdi|cdd|freelance|\d{2,3}\s?%)\b", re.IGNORECASE) | |
| 311 | +GENDER_MARK_RE = re.compile(r"\s*\(\s*(?:[mfwhvdx](?:\s*/\s*[mfwhvdx*]){1,3}|all genders|alle geschlechter|tous genres|m/f/d|h/f)\s*\)", re.IGNORECASE) | |
| 312 | +APPLY_SUFFIX_RE = re.compile(r"\s*[-–—|·]\s*(?:apply(?: now)?|postuler|jetzt bewerben|bewerben|solliciteer(?: nu)?|candidati|candidatar|aplicar)\s*$", re.IGNORECASE) | |
| 313 | +BRACKET_ID_RE = re.compile(r"\s*[\[(]\s*(?:#\s*|(?:job\s*id|ref\.?|id|req)(?:\s*[:#]\s*|\s+))?([A-Z]{0,4}[-_]?\d{3,})\s*[\])]", re.IGNORECASE) | |
| 314 | +JOB_STOP_EXACT_RE = _phrase_re([ | |
| 315 | + "read more", "learn more", "more info", "more information", "more", "info", "details", "view details", "view job", "view all", "view all jobs", | |
| 316 | + "view all openings", "view openings", "see all", "see all jobs", "see more", "see open roles", "search jobs", "search", "job search", "all jobs", | |
| 317 | + "all openings", "all positions", "open positions", "open roles", "openings", "jobs", "careers", "career", "apply", "apply now", "apply here", | |
| 318 | + "join us", "join our team", "join the team", "find out more", "discover more", "explore", "explore roles", "back", "back to top", "load more", | |
| 319 | + "show more", "next", "previous", "home", "filter", "filters", "sort by", "share", "save", "print", "benefits", "culture", "our culture", | |
| 320 | + "our values", "diversity", "faq", "hiring process", "talent community", "talent network", "job alerts", "job alert", "sign up for job alerts", | |
| 321 | + "meer info", "meer informatie", "lees meer", "solliciteer", "solliciteer nu", "alle vacatures", "bekijk alle vacatures", "bekijk vacature", | |
| 322 | + "vacatures", "ontdek meer", "meer weten", "en savoir plus", "lire la suite", "postuler", "postulez", "voir toutes les offres", "toutes les offres", | |
| 323 | + "voir l'offre", "voir plus", "découvrir", "nos offres", "offres d'emploi", "mehr erfahren", "mehr infos", "jetzt bewerben", "bewerben", | |
| 324 | + "alle stellen", "alle stellenangebote", "stellenangebote", "alle jobs", "offene stellen", "weiterlesen", "mehr anzeigen", "zur stelle", "karriere", | |
| 325 | + "ver más", "leer más", "aplicar", "aplica ahora", "postúlate", "ver todas las ofertas", "todas las ofertas", "más información", "saber más", | |
| 326 | + "empleo", "ofertas de empleo", "scopri di più", "leggi tutto", "candidati", "candidati ora", "tutte le posizioni", "tutte le offerte", | |
| 327 | + "maggiori informazioni", "posizioni aperte", "lavora con noi", "saiba mais", "ler mais", "candidatar", "candidate-se", "ver todas as vagas", | |
| 328 | + "todas as vagas", "mais informações", "vagas", "詳細を見る", "もっと見る", "応募する", "エントリー", "募集一覧", "採用情報", "募集要項", | |
| 329 | +]) | |
| 330 | +JOB_STOP_PREFIX_RE = re.compile(r"^(?:early careers?|graduate programm?e|graduates?\b|students?\b|starters\b|careers? (?:at|bij|chez|bei)\b|jobs? (?:at|bij|chez|bei)\b|" | |
| 331 | + r"working at\b|life at\b|why (?:join|work|us)\b|join us\b|about us\b|who we are\b|what we do\b|meet our\b|our (?:story|teams?|culture|" | |
| 332 | + r"values|benefits|people)\b|werken bij\b|waarom\b|pourquoi\b|warum\b|karriere bei\b|travailler chez\b|arbeiten bei\b|" | |
| 333 | + r"trabajar en\b|lavorare in\b|trabalhar na\b)", re.IGNORECASE) | |
| 334 | +JOB_URL_STRONG_RE = re.compile( | |
| 335 | + r"(?:/jobs?/[^/?#]+|/careers?/[^/?#]+/[^/?#]+|/positions?/|/openings?/|/vacanc(?:y|ies)/[^/?#]+|/vacatures?/[^/?#]+|/stellen(?:angebote?)?/[^/?#]+|" | |
| 336 | + r"/stelle/|/offres?(?:-d-?emploi)?/[^/?#]+|/emplois?/[^/?#]+|/empleos?/[^/?#]+|/lavoro/[^/?#]+|/vagas?/[^/?#]+|/opportunit(?:y|ies)/[^/?#]+|" | |
| 337 | + r"/\d{4,}(?:[/-]|$)|[?&](?:gh_jid|jobid|job_id|jid|reqid|req_id|id)=\d+|lever\.co/|greenhouse\.io/|ashbyhq\.com/|myworkdayjobs\.com/|" | |
| 338 | + r"smartrecruiters\.com/|workable\.com/|recruitee\.com/|personio\.(?:de|com)/|teamtailor\.com/|bamboohr\.com/|jobvite\.com/|icims\.com/|" | |
| 339 | + r"taleo\.net/|successfactors\.(?:com|eu)/|breezy\.hr/|applytojob\.com/|eightfold\.ai/|phenompeople\.com/|avature\.net/|csod\.com/|oraclecloud\.com/)", | |
| 340 | + re.IGNORECASE) | |
| 341 | +KNOWN_LOCATION_WORDS_RE = re.compile(r"\b(?:remote|hybrid|on-?site|télétravail|homeoffice|home office|thuiswerken|worldwide|anywhere)\b", re.IGNORECASE) | |
| 342 | + | |
| 343 | + | |
| 344 | +def clean_job_title(title: str) -> tuple[str, str | None]: | |
| 345 | + """Display title without gender markers, "- Apply" suffixes and bracketed ids; the id found (if any) is returned for the fingerprint.""" | |
| 346 | + t = strip_trailing_marks(title) | |
| 347 | + found_id: str | None = None | |
| 348 | + m = BRACKET_ID_RE.search(t) | |
| 349 | + if m: | |
| 350 | + found_id = m.group(1) | |
| 351 | + t = (t[:m.start()] + " " + t[m.end():]).strip() | |
| 352 | + t = GENDER_MARK_RE.sub("", t) | |
| 353 | + t = APPLY_SUFFIX_RE.sub("", t) | |
| 354 | + return normalize_whitespace(t).strip(" -–—|·,"), found_id | |
| 355 | + | |
| 356 | + | |
| 357 | +def job_url_is_joblike(url: str | None) -> bool: | |
| 358 | + return bool(url) and bool(JOB_URL_STRONG_RE.search(url or "")) | |
| 359 | + | |
| 360 | + | |
| 361 | +def states_job_location(text: str | None) -> bool: | |
| 362 | + if not text or len(text) > MAX_LOCATION_NAME_CHARS: | |
| 363 | + return False | |
| 364 | + p = parse_location(text) | |
| 365 | + return bool(p["city"] or p["country"] or p["remote"] or p["region"]) or is_known_city(text) or bool(KNOWN_LOCATION_WORDS_RE.search(text)) | |
| 366 | + | |
| 367 | + | |
| 368 | +def _department_like(text: str | None) -> bool: | |
| 369 | + return bool(text) and len(text or "") <= MAX_DEPARTMENT_CHARS and word_count(text or "") <= MAX_DEPARTMENT_WORDS and not any(ch.isdigit() for ch in text or "") | |
| 370 | + | |
| 371 | + | |
| 372 | +def job_verdict(title: str, *, url: str | None = None, location: str | None = None, department: str | None = None) -> Verdict: | |
| 373 | + """CTA / navigation anchors are not jobs; a job needs role vocabulary, a job-like URL or an explicit location/department cell.""" | |
| 374 | + raw = normalize_whitespace(title) | |
| 375 | + if ELLIPSIS_RE.search(raw): | |
| 376 | + return Verdict.reject("truncated title (ellipsis)") | |
| 377 | + t, _ = clean_job_title(raw) | |
| 378 | + if not (MIN_JOB_TITLE_CHARS <= len(t) <= MAX_JOB_TITLE_CHARS): | |
| 379 | + return Verdict.reject("title length") | |
| 380 | + if JOB_STOP_EXACT_RE.match(t) or JOB_STOP_PREFIX_RE.match(t): | |
| 381 | + return Verdict.reject("call-to-action / navigation label") | |
| 382 | + if word_count(t) < MIN_JOB_TITLE_WORDS and not _CJK_RE.search(t): | |
| 383 | + return Verdict.reject("single-word title") | |
| 384 | + if JOB_ROLE_RE.search(t) or JOB_FORMAT_RE.search(raw): | |
| 385 | + return Verdict.accept() | |
| 386 | + if is_prose(t): | |
| 387 | + return Verdict.reject("reads like a sentence") | |
| 388 | + if job_url_is_joblike(url) or states_job_location(location) or _department_like(department): | |
| 389 | + return Verdict.accept() | |
| 390 | + return Verdict.reject("no job-like signal (vocabulary, url, location)") | |
| 391 | + | |
| 392 | + | |
| 393 | +def refine_job(job: ExtractedJob) -> ExtractedJob | None: | |
| 394 | + """Apply `job_verdict`, clean the display title and keep a bracketed id as `external_id` (so the fingerprint keeps it).""" | |
| 395 | + if not job_verdict(job.title, url=job.url, location=job.location_text, department=job.department).ok: | |
| 396 | + return None | |
| 397 | + clean, found_id = clean_job_title(job.title) | |
| 398 | + if clean: | |
| 399 | + job.title = clean | |
| 400 | + if found_id and not job.external_id: | |
| 401 | + job.external_id = found_id | |
| 402 | + return job | |
| 403 | + | |
| 404 | + | |
| 405 | +# ------------------------------------------------------------------------------------------------------------ locations | |
| 406 | + | |
| 407 | +NAV_COOKIE_RE = re.compile( | |
| 408 | + r"\b(?:performance|analytics|analytical|marketing|functional|functionality|necessary|essential|preferences?|strictly|targeting|advertising|" | |
| 409 | + r"advertisement|statistics?|statistik(?:en)?|statistiques?|statistieken|estadisticas?|statistiche|tracking|cookies?|consent|privacy|privacybeleid|" | |
| 410 | + r"datenschutz|confidentialite|privacidad|gdpr|contact us|contact|contactez|contacto|contatti|kontakt|careers?|jobs?|sitemap|terms|legal|" | |
| 411 | + r"mentions legales|impressum|accessibility|newsletter|subscribe|login|log in|sign in|sign up|register|search|menu|language|select|filter|" | |
| 412 | + r"view all|see all|all locations|find (?:a|an|your)|more|back|home|about us|faq|help|support|settings|einstellungen|notwendig|funktional|" | |
| 413 | + r"necessaires?|fonctionnels?|preferences|noodzakelijk|functioneel|voorkeuren|necesarias?|funcionales?|preferencias|necessari|funzionali|" | |
| 414 | + r"preferenze|publicidad|pubblicita|werbung|publicite|social media|session|third party|unclassified|uncategori[sz]ed|others?)\b", re.IGNORECASE) | |
| 415 | +LOCATION_KIND_WORD_RE = re.compile( | |
| 416 | + r"\b(?:headquarters|head office|hq|office|offices|bureau|büro|kantoor|oficina|ufficio|escritório|store|shop|boutique|showroom|plant|factory|usine|" | |
| 417 | + r"werk|fabrik|fabriek|fábrica|fabbrica|warehouse|entrepôt|lager|magazijn|almacén|lab|laboratory|laboratoire|labor|campus|branch|agence|filiale|" | |
| 418 | + r"succursale|niederlassung|vestiging|sede|siège|hauptsitz|data ?cent(?:er|re)|distribution cent(?:er|re))\b|本社|支社|営業所|工場|オフィス|拠点", re.IGNORECASE) | |
| 419 | +VENUE_RE = re.compile(r"\b(?:hotel|tower|towers|building|centre|center|plaza|mall|park|campus|house|hall|street|avenue|road|floor|level|suite|" | |
| 420 | + r"hôtel|gebäude|gebouw|edificio|torre|palazzo)\b", re.IGNORECASE) | |
| 421 | +STREET_RE = re.compile( | |
| 422 | + r"\b\d{1,5}[a-z]?\s+[^\n,]{2,40}\b(?:street|st\.?|avenue|ave\.?|road|rd\.?|boulevard|blvd\.?|drive|dr\.?|lane|ln\.?|way|place|pl\.?|square|plaza|" | |
| 423 | + r"court|ct\.?|parkway|highway|straße|strasse|str\.|allee|platz|weg|rue|avenida|calle|via|piazza)\b" | |
| 424 | + r"|\b(?:rue|avenue|boulevard|via|calle|avenida|carrer|rua|praça)\s+[^\n,]{2,40}\s\d{1,5}\b" | |
| 425 | + r"|\b(?:[^\s,\d]+\s){0,2}[^\s,]{3,40}(?:straat|weg|laan|plein|kade|zijde|gracht|singel|markt|dijk|straße|strasse|str\.|allee|platz|gasse|ring|damm|ufer|" | |
| 426 | + r"vej|gade|gatan|vägen|katu|tie)\s+\d{1,5}[a-z]?\b", re.IGNORECASE) | |
| 427 | + | |
| 428 | +# ~400 major cities and business hubs with common exonyms/endonyms; keys are accent-insensitive (see `key_of`). | |
| 429 | +_CITIES = """ | |
| 430 | +new york, los angeles, chicago, houston, phoenix, philadelphia, san antonio, san diego, dallas, san jose, austin, jacksonville, fort worth, columbus, | |
| 431 | +charlotte, san francisco, indianapolis, seattle, denver, washington, boston, nashville, detroit, portland, las vegas, memphis, louisville, baltimore, | |
| 432 | +milwaukee, albuquerque, tucson, fresno, sacramento, kansas city, atlanta, miami, oakland, minneapolis, cleveland, raleigh, omaha, tampa, orlando, | |
| 433 | +pittsburgh, cincinnati, st. louis, salt lake city, richmond, new orleans, buffalo, hartford, providence, durham, boulder, palo alto, mountain view, | |
| 434 | +menlo park, redmond, bellevue, cambridge, princeton, stamford, irvine, santa clara, sunnyvale, cupertino, redwood city, san mateo, arlington, reston, | |
| 435 | +mclean, plano, scottsdale, tempe, ann arbor, madison, des moines, boise, honolulu, anchorage, charleston, savannah, oklahoma city, tulsa, el paso, | |
| 436 | +long beach, colorado springs, newark, jersey city, brooklyn, manhattan, santa monica, pasadena, burbank, anaheim, riverside, henderson, reno, spokane, | |
| 437 | +tacoma, rochester, syracuse, albany, wilmington, trenton, grand rapids, toledo, dayton, lexington, knoxville, chattanooga, huntsville, baton rouge, | |
| 438 | +little rock, tallahassee, st. petersburg, fort lauderdale, west palm beach, boca raton, greenville, columbia, greensboro, norfolk, alexandria, bethesda, | |
| 439 | +rockville, toronto, montréal, montreal, vancouver, calgary, edmonton, ottawa, winnipeg, québec, quebec city, hamilton, kitchener, waterloo, halifax, | |
| 440 | +victoria, saskatoon, regina, mississauga, brampton, markham, vaughan, burnaby, surrey, laval, gatineau, kelowna, oakville, burlington, | |
| 441 | +mexico city, ciudad de méxico, guadalajara, monterrey, puebla, tijuana, querétaro, cancún, mérida, são paulo, sao paulo, rio de janeiro, brasília, brasilia, | |
| 442 | +belo horizonte, curitiba, porto alegre, salvador, recife, fortaleza, campinas, florianópolis, manaus, buenos aires, córdoba, rosario, mendoza, santiago, | |
| 443 | +valparaíso, lima, bogotá, bogota, medellín, cali, cartagena, quito, guayaquil, caracas, montevideo, asunción, la paz, san josé, panama city, | |
| 444 | +ciudad de panamá, guatemala city, san salvador, tegucigalpa, managua, santo domingo, san juan, havana, kingston, nassau, | |
| 445 | +london, manchester, birmingham, leeds, glasgow, edinburgh, liverpool, bristol, sheffield, newcastle, nottingham, cardiff, belfast, leicester, coventry, | |
| 446 | +oxford, reading, milton keynes, brighton, southampton, portsmouth, aberdeen, dundee, york, bath, exeter, plymouth, norwich, swindon, slough, watford, | |
| 447 | +guildford, basingstoke, warrington, derby, sunderland, hull, bradford, dublin, cork, galway, limerick, waterford, | |
| 448 | +paris, marseille, lyon, toulouse, nice, nantes, strasbourg, montpellier, bordeaux, lille, rennes, reims, le havre, saint-étienne, toulon, grenoble, | |
| 449 | +dijon, angers, nîmes, villeurbanne, clermont-ferrand, le mans, aix-en-provence, brest, tours, amiens, limoges, annecy, perpignan, metz, besançon, | |
| 450 | +orléans, rouen, mulhouse, caen, nancy, boulogne-billancourt, issy-les-moulineaux, la défense, courbevoie, neuilly-sur-seine, levallois-perret, nanterre, | |
| 451 | +puteaux, saint-denis, versailles, sophia antipolis, cannes, monaco, | |
| 452 | +berlin, hamburg, münchen, munich, köln, cologne, frankfurt, frankfurt am main, stuttgart, düsseldorf, dortmund, essen, leipzig, bremen, dresden, hannover, | |
| 453 | +hanover, nürnberg, nuremberg, duisburg, bochum, wuppertal, bielefeld, bonn, münster, karlsruhe, mannheim, augsburg, wiesbaden, gelsenkirchen, | |
| 454 | +mönchengladbach, braunschweig, chemnitz, kiel, aachen, halle, magdeburg, freiburg, krefeld, lübeck, mainz, erfurt, oberhausen, rostock, kassel, hagen, | |
| 455 | +saarbrücken, potsdam, ludwigshafen, oldenburg, leverkusen, heidelberg, darmstadt, regensburg, ingolstadt, würzburg, ulm, wolfsburg, göttingen, | |
| 456 | +paderborn, heilbronn, erlangen, jena, walldorf, böblingen, sindelfingen, neckarsulm, herzogenaurach, gütersloh, | |
| 457 | +wien, vienna, graz, linz, salzburg, innsbruck, zürich, zurich, genève, geneva, genf, basel, bâle, bern, berne, lausanne, winterthur, luzern, lucerne, | |
| 458 | +st. gallen, lugano, zug, baar, vevey, neuchâtel, | |
| 459 | +amsterdam, rotterdam, den haag, the hague, utrecht, eindhoven, groningen, tilburg, almere, breda, nijmegen, arnhem, haarlem, amersfoort, enschede, | |
| 460 | +apeldoorn, 's-hertogenbosch, leiden, delft, maastricht, zwolle, hilversum, hoofddorp, schiphol, amstelveen, veldhoven, brussels, bruxelles, brussel, | |
| 461 | +antwerp, antwerpen, anvers, ghent, gent, gand, charleroi, liège, luik, bruges, brugge, namur, leuven, louvain, mechelen, mons, hasselt, kortrijk, | |
| 462 | +zaventem, diegem, luxembourg, esch-sur-alzette, | |
| 463 | +stockholm, göteborg, gothenburg, malmö, uppsala, västerås, linköping, lund, helsingborg, örebro, oslo, bergen, trondheim, stavanger, drammen, | |
| 464 | +copenhagen, københavn, aarhus, århus, odense, aalborg, helsinki, helsingfors, espoo, tampere, vantaa, oulu, turku, reykjavík, reykjavik, tallinn, riga, | |
| 465 | +vilnius, kaunas, | |
| 466 | +madrid, barcelona, valencia, sevilla, seville, zaragoza, málaga, murcia, palma, las palmas, bilbao, alicante, valladolid, vigo, gijón, a coruña, | |
| 467 | +granada, vitoria-gasteiz, san sebastián, donostia, pamplona, santander, lisboa, lisbon, porto, oporto, braga, coimbra, faro, funchal, roma, rome, | |
| 468 | +milano, milan, napoli, naples, torino, turin, palermo, genova, genoa, bologna, firenze, florence, bari, catania, venezia, venice, verona, messina, | |
| 469 | +padova, padua, trieste, brescia, parma, modena, reggio emilia, perugia, bergamo, vicenza, monza, athens, athina, thessaloniki, piraeus, nicosia, | |
| 470 | +limassol, valletta, ljubljana, zagreb, split, belgrade, beograd, sarajevo, skopje, tirana, podgorica, | |
| 471 | +warsaw, warszawa, kraków, krakow, cracow, łódź, lodz, wrocław, wroclaw, poznań, poznan, gdańsk, gdansk, szczecin, katowice, lublin, bydgoszcz, prague, | |
| 472 | +praha, brno, ostrava, plzeň, bratislava, košice, budapest, debrecen, szeged, bucharest, bucurești, cluj-napoca, timișoara, iași, sofia, plovdiv, | |
| 473 | +varna, kyiv, kiev, kharkiv, lviv, odesa, odessa, dnipro, minsk, moscow, moskva, saint petersburg, st. petersburg, novosibirsk, yekaterinburg, kazan, | |
| 474 | +chișinău, tbilisi, yerevan, baku, | |
| 475 | +istanbul, ankara, izmir, bursa, antalya, tel aviv, jerusalem, haifa, herzliya, petah tikva, beersheba, dubai, abu dhabi, sharjah, doha, riyadh, jeddah, | |
| 476 | +dammam, kuwait city, manama, muscat, amman, beirut, baghdad, tehran, cairo, giza, casablanca, rabat, marrakech, tangier, tunis, algiers, lagos, abuja, | |
| 477 | +nairobi, mombasa, accra, addis ababa, dar es salaam, kampala, kigali, johannesburg, cape town, durban, pretoria, gqeberha, luanda, maputo, lusaka, | |
| 478 | +harare, dakar, abidjan, kinshasa, douala, yaoundé, windhoek, gaborone, antananarivo, port louis, khartoum, | |
| 479 | +tokyo, 東京, osaka, 大阪, yokohama, 横浜, nagoya, 名古屋, sapporo, 札幌, fukuoka, 福岡, kobe, 神戸, kyoto, 京都, kawasaki, 川崎, saitama, hiroshima, 広島, | |
| 480 | +sendai, 仙台, chiba, 千葉, seoul, 서울, busan, 부산, incheon, 인천, daegu, daejeon, gwangju, suwon, pangyo, seongnam, beijing, 北京, peking, shanghai, 上海, | |
| 481 | +guangzhou, 广州, shenzhen, 深圳, chengdu, 成都, hangzhou, 杭州, wuhan, 武汉, xi'an, 西安, chongqing, 重庆, tianjin, 天津, nanjing, 南京, suzhou, 苏州, | |
| 482 | +qingdao, 青岛, dalian, 大连, xiamen, 厦门, shenyang, changsha, zhengzhou, dongguan, ningbo, hong kong, 香港, macau, macao, 澳门, taipei, 台北, 臺北, | |
| 483 | +taichung, kaohsiung, hsinchu, tainan, singapore, kuala lumpur, penang, george town, johor bahru, cyberjaya, petaling jaya, bangkok, กรุงเทพ, | |
| 484 | +chiang mai, jakarta, surabaya, bandung, denpasar, medan, manila, makati, quezon city, cebu, taguig, bonifacio global city, pasig, hanoi, hà nội, | |
| 485 | +ho chi minh city, hồ chí minh, saigon, da nang, phnom penh, yangon, vientiane, dhaka, chittagong, colombo, kathmandu, karachi, lahore, islamabad, | |
| 486 | +rawalpindi, faisalabad, mumbai, bombay, delhi, new delhi, bengaluru, bangalore, hyderabad, chennai, madras, kolkata, calcutta, pune, ahmedabad, jaipur, | |
| 487 | +surat, lucknow, kanpur, nagpur, indore, thane, bhopal, visakhapatnam, vadodara, coimbatore, kochi, cochin, gurgaon, gurugram, noida, chandigarh, | |
| 488 | +mysore, mysuru, thiruvananthapuram, trivandrum, bhubaneswar, mohali, navi mumbai, sydney, melbourne, brisbane, perth, adelaide, canberra, | |
| 489 | +gold coast, hobart, darwin, wollongong, geelong, auckland, wellington, christchurch, dunedin, tauranga, ulaanbaatar, almaty, astana, tashkent, bishkek | |
| 490 | +""" | |
| 491 | +CITY_KEYS: frozenset[str] = frozenset(key_of(c) for c in _CITIES.replace("\n", ",").split(",") if c.strip()) | |
| 492 | + | |
| 493 | + | |
| 494 | +def is_known_city(text: str | None) -> bool: | |
| 495 | + """Whole string or any 1–3-word n-gram of a short string names a city in the table.""" | |
| 496 | + if not text: | |
| 497 | + return False | |
| 498 | + k = key_of(text) | |
| 499 | + if not k: | |
| 500 | + return False | |
| 501 | + if k in CITY_KEYS: | |
| 502 | + return True | |
| 503 | + if _CJK_RE.search(text): | |
| 504 | + return any(city in k for city in CITY_KEYS if _CJK_RE.search(city)) | |
| 505 | + toks = k.split() | |
| 506 | + if len(toks) > MAX_LOCATION_NAME_WORDS: | |
| 507 | + return False | |
| 508 | + for n in range(1, min(CITY_NGRAM_MAX, len(toks)) + 1): | |
| 509 | + for i in range(len(toks) - n + 1): | |
| 510 | + if " ".join(toks[i:i + n]) in CITY_KEYS: | |
| 511 | + return True | |
| 512 | + return False | |
| 513 | + | |
| 514 | + | |
| 515 | +CITY_DISTRICT_RE = re.compile(r"\s\d{1,2}$") # "Dublin 2", "Paris 8" — a postal district after a known city | |
| 516 | + | |
| 517 | + | |
| 518 | +def has_postal_code(text: str | None) -> bool: | |
| 519 | + if not text: | |
| 520 | + return False | |
| 521 | + return any(not YEAR_RE.match(m.group(0)) for m in POSTAL_RE.finditer(text)) | |
| 522 | + | |
| 523 | + | |
| 524 | +def looks_like_city_value(text: str | None) -> bool: | |
| 525 | + """A plausible `city` cell: short, capitalised words, no digits, no nav/cookie vocabulary, no sentence.""" | |
| 526 | + if not text: | |
| 527 | + return False | |
| 528 | + t = normalize_whitespace(text) | |
| 529 | + if not (2 <= len(t) <= MAX_CITY_CHARS) or word_count(t) > MAX_CITY_WORDS: | |
| 530 | + return False | |
| 531 | + if any(ch.isdigit() for ch in t) and not (CITY_DISTRICT_RE.search(t) and is_known_city(CITY_DISTRICT_RE.sub("", t))): | |
| 532 | + return False # "Level 12" no · "Dublin 2" (postal district) yes | |
| 533 | + if NAV_COOKIE_RE.search(key_of(t)) or is_prose(t): | |
| 534 | + return False | |
| 535 | + return t[0].isupper() or bool(_CJK_RE.search(t[0])) | |
| 536 | + | |
| 537 | + | |
| 538 | +def _looks_like_venue(text: str) -> bool: | |
| 539 | + return word_count(text) >= VENUE_MIN_WORDS or bool(VENUE_RE.search(text)) | |
| 540 | + | |
| 541 | + | |
| 542 | +def location_signal(*, name: str, city: str | None, region: str | None, country: str | None, kind: str | None, address: str | None) -> str | None: | |
| 543 | + """The evidence that makes this a real place, or None: country / known city / postal code / street address / explicit kind label.""" | |
| 544 | + if country: | |
| 545 | + return "country" | |
| 546 | + if is_known_city(city) or is_known_city(region) or is_known_city(name): | |
| 547 | + return "known city" | |
| 548 | + if has_postal_code(address) or (has_postal_code(name) and "," in name): | |
| 549 | + return "postal code" | |
| 550 | + if STREET_RE.search(address or "") or STREET_RE.search(name): | |
| 551 | + return "street address" | |
| 552 | + if (kind and kind not in ("office", "other")) or LOCATION_KIND_WORD_RE.search(name): | |
| 553 | + return "kind label" | |
| 554 | + return None | |
| 555 | + | |
| 556 | + | |
| 557 | +def normalize_location(loc: ExtractedLocation) -> ExtractedLocation | None: | |
| 558 | + """Validated copy of `loc` (city cell cleaned, country-as-name repaired) or None when it is not a place.""" | |
| 559 | + name = strip_trailing_marks(loc.name or "") | |
| 560 | + if not name or len(name) > MAX_LOCATION_NAME_CHARS or word_count(name) > MAX_LOCATION_NAME_WORDS: | |
| 561 | + return None | |
| 562 | + if NAV_COOKIE_RE.search(key_of(name)) or is_prose(name): | |
| 563 | + return None | |
| 564 | + city = loc.city if looks_like_city_value(loc.city) else None | |
| 565 | + region = loc.region | |
| 566 | + country = loc.country | |
| 567 | + name_country = country_code(name) | |
| 568 | + if name_country and city and key_of(city) == key_of(name) and not is_known_city(name): | |
| 569 | + city = None # "Oman | Oman": a country is not its own city (Singapore is) | |
| 570 | + if name_country and loc.city and (city is None or _looks_like_venue(loc.city)) and not NAV_COOKIE_RE.search(key_of(loc.city)) and not is_prose(loc.city) \ | |
| 571 | + and len(loc.city) <= MAX_LOCATION_NAME_CHARS and key_of(loc.city) != key_of(name): | |
| 572 | + name, city, country = normalize_whitespace(loc.city), None, country or name_country # "Oman | Hormuz Grand Hotel" → venue named, country kept | |
| 573 | + if region and is_known_city(region) and looks_like_city_value(region) and (city is None or not is_known_city(city)): | |
| 574 | + city, region = region, None # "3089 JH Rotterdam" parsed as a region | |
| 575 | + if location_signal(name=name, city=city, region=region, country=country, kind=loc.kind, address=loc.address_text) is None: | |
| 576 | + return None | |
| 577 | + return ExtractedLocation(name=name, kind=loc.kind or "office", city=city, region=region, country=country, address_text=loc.address_text) | |
| 578 | + | |
| 579 | + | |
| 580 | +def location_verdict(loc: ExtractedLocation) -> Verdict: | |
| 581 | + name = strip_trailing_marks(loc.name or "") | |
| 582 | + if not name: | |
| 583 | + return Verdict.reject("empty name") | |
| 584 | + if len(name) > MAX_LOCATION_NAME_CHARS or word_count(name) > MAX_LOCATION_NAME_WORDS: | |
| 585 | + return Verdict.reject("name too long") | |
| 586 | + if NAV_COOKIE_RE.search(key_of(name)): | |
| 587 | + return Verdict.reject("cookie/consent/navigation vocabulary") | |
| 588 | + if is_prose(name): | |
| 589 | + return Verdict.reject("name is a sentence") | |
| 590 | + fixed = normalize_location(loc) | |
| 591 | + if fixed is None: | |
| 592 | + return Verdict.reject("no place evidence (country, city, postal code, street, kind)") | |
| 593 | + return Verdict.accept() | |
| 594 | + | |
| 595 | + | |
| 596 | +# ------------------------------------------------------------------------------------------------------------ pricing plans | |
| 597 | + | |
| 598 | +PLAN_EYEBROW_RE = _phrase_re(["most popular", "popular", "best value", "recommended", "new", "beta", "coming soon", "limited offer", "best seller", | |
| 599 | + "le plus populaire", "beliebt", "am beliebtesten", "meest gekozen", "más popular", "più popolare", "mais popular", "人気", | |
| 600 | + # generic section headings that are not tiers | |
| 601 | + "plans", "plan", "pricing", "prices", "price", "plans & pricing", "plans and pricing", "our plans", "compare plans", | |
| 602 | + "choose your plan", "tarifs", "nos tarifs", "abonnements", "preise", "unsere preise", "tarife", "prijzen", "abonnementen", | |
| 603 | + "precios", "planes", "prezzi", "piani", "preços", "planos", "料金", "料金プラン", "プラン"]) | |
| 604 | +PRICE_TOKEN_RE = re.compile( | |
| 605 | + r"(?:(?:US\$|CA\$|C\$|A\$|NZ\$|HK\$|S\$|MX\$|R\$|USD|EUR|GBP|CAD|AUD|CHF|JPY|INR|SEK|NOK|DKK|PLN|BRL|CNY|RMB|SGD|[$€£¥₹₩₺])\s?" | |
| 606 | + r"\d{1,3}(?:[,.\s]\d{3})*(?:[.,]\d{1,2})?|\d{1,3}(?:[,.\s]\d{3})*(?:[.,]\d{1,2})?\s?(?:€|£|USD|EUR|GBP|CHF|kr|zł|元|\$))", re.IGNORECASE) | |
| 607 | +PRICE_LEAD_RE = re.compile(r"(?:from|starting at|starts at|as low as|à partir de|ab|desde|a partire da|vanaf|only|just)\s*$", re.IGNORECASE) | |
| 608 | +PRICE_UNIT = (r"(?:month|mo|year|yr|annum|week|wk|day|hour|hr|user|seat|member|license|licence|agent|editor|contact|1,?000|1k|GB|TB|request|" | |
| 609 | + r"transaction|call|minute|mois|an|année|monat|jahr|nutzer|benutzer|utilisateur|usuario|utente|mese|anno|mes|año|maand|jaar|gebruiker|" | |
| 610 | + r"dag|tag|giorno|día|dia|jour)") | |
| 611 | +PRICE_TAIL_RE = re.compile(rf"^\s*(?:(?:/|per|a|an|each|every|par|pro|pour|por|al|je)\s*{PRICE_UNIT}\b(?:\s*(?:/|per|par|pro|pour|por)\s*{PRICE_UNIT}\b)?|" | |
| 612 | + rf"(?:monthly|annually|yearly|mensuel|annuel|monatlich|jährlich|one[- ]time|lifetime|forever))?" | |
| 613 | + r"(?:,?\s*billed (?:monthly|annually|yearly))?", re.IGNORECASE) | |
| 614 | +CONTACT_PHRASE_RE = re.compile(r"\b(contact (?:us|sales)|talk to (?:us|sales|an expert)|custom pricing|custom quote|get a quote|request (?:a )?(?:quote|demo)|" | |
| 615 | + r"let'?s talk|on request|upon request|sur devis|nous contacter|contactez-nous|auf anfrage|individuell|tailored|bespoke|call us)\b", | |
| 616 | + re.IGNORECASE) | |
| 617 | +FREE_TOKEN_RE = re.compile(r"^(free|gratuit|kostenlos|gratis|\$\s?0(?:\.00)?|0\s?€|€\s?0)\b", re.IGNORECASE) | |
| 618 | + | |
| 619 | + | |
| 620 | +def price_text_from(text: str) -> str | None: | |
| 621 | + """The price phrase stated in `text` — currency + amount + period/unit, or the contact-sales / free wording — never a cut sentence.""" | |
| 622 | + t = normalize_whitespace(text) | |
| 623 | + m = PRICE_TOKEN_RE.search(t) | |
| 624 | + if m: | |
| 625 | + lead = PRICE_LEAD_RE.search(t[:m.start()]) | |
| 626 | + start = lead.start() if lead else m.start() | |
| 627 | + tail = PRICE_TAIL_RE.match(t[m.end():]) | |
| 628 | + end = m.end() + (tail.end() if tail else 0) | |
| 629 | + return t[start:end].strip(" ,")[:MAX_PRICE_TEXT_CHARS] | |
| 630 | + c = CONTACT_PHRASE_RE.search(t) | |
| 631 | + if c: | |
| 632 | + phrase = c.group(1) | |
| 633 | + return phrase[0].upper() + phrase[1:] | |
| 634 | + f = FREE_TOKEN_RE.match(t) | |
| 635 | + if f: | |
| 636 | + return f.group(1) | |
| 637 | + return None | |
| 638 | + | |
| 639 | + | |
| 640 | +def plan_name_ok(name: str | None) -> bool: | |
| 641 | + if not name: | |
| 642 | + return False | |
| 643 | + n = normalize_whitespace(name) | |
| 644 | + if not (1 <= len(n) <= MAX_PLAN_NAME_CHARS) or word_count(n) > MAX_PLAN_NAME_WORDS: | |
| 645 | + return False | |
| 646 | + if has_terminal_punct(n) or "|" in n or "?" in n or PLAN_EYEBROW_RE.match(n) or re.search(r"\d+\s?%", n): | |
| 647 | + return False | |
| 648 | + if MARKETING_VERB_RE.match(n) and word_count(n) > 1: | |
| 649 | + return False | |
| 650 | + return not PRICE_TOKEN_RE.search(n) | |
| 651 | + | |
| 652 | + | |
| 653 | +def plan_verdict(plan: ExtractedPlan) -> Verdict: | |
| 654 | + if not plan_name_ok(plan.plan_name): | |
| 655 | + return Verdict.reject("plan name is a heading / marketing phrase") | |
| 656 | + if plan.price is None and not plan.contact_sales and not (plan.price_text and FREE_TOKEN_RE.match(plan.price_text)): | |
| 657 | + return Verdict.reject("no price, no contact-sales, no free tier") | |
| 658 | + if plan.price_text and price_text_from(plan.price_text) is None: | |
| 659 | + return Verdict.reject("price text does not state a price") | |
| 660 | + return Verdict.accept() | |
| 661 | + | |
| 662 | + | |
| 663 | +# ------------------------------------------------------------------------------------------------------------ products | |
| 664 | + | |
| 665 | +PRODUCT_NAV_RE = _phrase_re([ | |
| 666 | + "overview", "products", "product", "all products", "our products", "solutions", "solution", "all solutions", "services", "our services", "features", | |
| 667 | + "pricing", "resources", "support", "docs", "documentation", "blog", "contact", "contact us", "home", "learn more", "read more", "view all", "see all", | |
| 668 | + "explore", "more", "get started", "sign up", "log in", "download", "compare", "industries", "platform", "use cases", "customers", "partners", | |
| 669 | + "produits", "nos produits", "tous les produits", "en savoir plus", "produkte", "alle produkte", "unsere produkte", "lösungen", "mehr erfahren", | |
| 670 | + "producten", "alle producten", "oplossingen", "meer info", "productos", "todos los productos", "soluciones", "ver más", "prodotti", "tutti i prodotti", | |
| 671 | + "soluzioni", "scopri di più", "produtos", "todos os produtos", "soluções", "saiba mais", "製品", "製品一覧", "ソリューション", "サービス", "詳細", "もっと見る", | |
| 672 | +]) | |
| 673 | + | |
| 674 | + | |
| 675 | +def product_verdict(name: str) -> Verdict: | |
| 676 | + n = strip_trailing_marks(name) | |
| 677 | + if not (2 <= len(n) <= MAX_PRODUCT_NAME_CHARS): | |
| 678 | + return Verdict.reject("name length") | |
| 679 | + if word_count(n) > MAX_PRODUCT_NAME_WORDS: | |
| 680 | + return Verdict.reject("name too long") | |
| 681 | + if PRODUCT_NAV_RE.match(n.rstrip("®™ ")): | |
| 682 | + return Verdict.reject("navigation label") | |
| 683 | + if has_terminal_punct(n) and word_count(n) >= 3: | |
| 684 | + return Verdict.reject("name is a sentence") | |
| 685 | + if is_slogan(n) or is_prose(n): | |
| 686 | + return Verdict.reject("marketing slogan") | |
| 687 | + return Verdict.accept() | |
| 688 | + | |
| 689 | + | |
| 690 | +# ------------------------------------------------------------------------------------------------------------ news | |
| 691 | + | |
| 692 | +NEWS_NOISE_RE = _phrase_re([ | |
| 693 | + "read more", "read more news", "learn more", "more", "more news", "continue reading", "view all", "see all", "all news", "all posts", "all articles", | |
| 694 | + "next", "previous", "older", "newer", "older posts", "newer posts", "load more", "show more", "page", "news", "press", "press releases", "press release", | |
| 695 | + "blog", "blog posts", "articles", "insights", "events", "media", "in the news", "case studies", "webinars", "whitepapers", "podcasts", "videos", | |
| 696 | + "newsroom", "latest news", "category", "categories", "archive", "archives", "tags", "share", "rss", "subscribe", "newsletter", "actualités", | |
| 697 | + "toutes les actualités", "communiqués de presse", "lire la suite", "en savoir plus", "alle news", "pressemitteilungen", "aktuelles", "weiterlesen", | |
| 698 | + "mehr erfahren", "nieuws", "alle berichten", "persberichten", "lees meer", "noticias", "todas las noticias", "notas de prensa", "leer más", "notizie", | |
| 699 | + "comunicati stampa", "leggi tutto", "notícias", "ler mais", "ニュース", "お知らせ", "プレスリリース", "一覧", "もっと見る", | |
| 700 | +]) | |
| 701 | +PAGINATION_RE = re.compile(r"^(?:page\s*)?\d{1,4}$|^(?:[«»‹›<>]|\.\.\.|…)+$", re.IGNORECASE) | |
| 702 | + | |
| 703 | + | |
| 704 | +def news_verdict(title: str, *, url: str | None = None, published_at: datetime | None = None) -> Verdict: | |
| 705 | + t = strip_trailing_marks(title) | |
| 706 | + if not t or PAGINATION_RE.match(t): | |
| 707 | + return Verdict.reject("pagination") | |
| 708 | + if NEWS_NOISE_RE.match(t): | |
| 709 | + return Verdict.reject("navigation / category label") | |
| 710 | + confirmed = published_at is not None or date_from_url(url) is not None | |
| 711 | + long_enough = word_count(t) >= NEWS_MIN_WORDS or len(t) >= NEWS_MIN_CHARS or (bool(_CJK_RE.search(t)) and len(t) >= NEWS_MIN_CJK_CHARS) | |
| 712 | + if not confirmed and not long_enough: | |
| 713 | + return Verdict.reject("too short without a date") | |
| 714 | + return Verdict.accept() | |
| 715 | + | |
| 716 | + | |
| 717 | +# ------------------------------------------------------------------------------------------------------------ whole extraction | |
| 718 | + | |
| 719 | +HTML_JOB_CONNECTORS = frozenset({"generic-html-v1"}) | |
| 720 | + | |
| 721 | + | |
| 722 | +def apply_precision(ex: Extraction, *, html_jobs: bool) -> dict[str, int]: | |
| 723 | + """Filter/normalise every typed list in place; returns how many items each list lost. Job rules only for HTML-scraped listings.""" | |
| 724 | + dropped: dict[str, int] = {} | |
| 725 | + if html_jobs: | |
| 726 | + kept_jobs = [j for j in (refine_job(j) for j in ex.jobs) if j is not None] | |
| 727 | + dropped["jobs"] = len(ex.jobs) - len(kept_jobs) | |
| 728 | + ex.jobs = kept_jobs | |
| 729 | + people: list[ExtractedPerson] = [] | |
| 730 | + for p in ex.people: | |
| 731 | + fixed = normalize_person(p.name, p.title) | |
| 732 | + if fixed is None: | |
| 733 | + continue | |
| 734 | + p.name, p.title = fixed | |
| 735 | + p.role_category, p.is_executive = role_category(p.title) | |
| 736 | + people.append(p) | |
| 737 | + dropped["people"] = len(ex.people) - len(people) | |
| 738 | + ex.people = people | |
| 739 | + products: list[ExtractedProduct] = [p for p in ex.products if product_verdict(p.name).ok] | |
| 740 | + dropped["products"] = len(ex.products) - len(products) | |
| 741 | + ex.products = products | |
| 742 | + plans: list[ExtractedPlan] = [p for p in ex.plans if plan_verdict(p).ok] | |
| 743 | + dropped["plans"] = len(ex.plans) - len(plans) | |
| 744 | + ex.plans = plans | |
| 745 | + locations: list[ExtractedLocation] = [loc for loc in (normalize_location(loc) for loc in ex.locations) if loc is not None] | |
| 746 | + dropped["locations"] = len(ex.locations) - len(locations) | |
| 747 | + ex.locations = locations | |
| 748 | + news: list[ExtractedNewsItem] = [n for n in ex.news if news_verdict(n.title, url=n.url, published_at=n.published_at).ok] | |
| 749 | + dropped["news"] = len(ex.news) - len(news) | |
| 750 | + ex.news = news | |
| 751 | + return {k: v for k, v in dropped.items() if v} | |
| 752 | + | |
| 753 | + | |
| 754 | +def describe(verdict: Verdict) -> dict[str, Any]: | |
| 755 | + return {"ok": verdict.ok, "reason": verdict.reason} | |
| 756 | + | |
| 757 | + | |
| 758 | +__all__ = [ | |
| 759 | + "CHIEF_RE", "CITY_KEYS", "EXEC_CATEGORIES", "HTML_JOB_CONNECTORS", "NAV_COOKIE_RE", "PRECISION_VERSION", "ROLE_RULES", "STREET_RE", "Verdict", | |
| 760 | + "apply_precision", "clean_job_title", "clean_person_title", "describe", "has_postal_code", "is_known_city", "is_prose", "is_slogan", "job_url_is_joblike", | |
| 761 | + "job_verdict", "key_of", "location_signal", "location_verdict", "looks_like_city_value", "looks_like_person_name", "looks_like_role_title", "news_verdict", | |
| 762 | + "normalize_location", "normalize_person", "person_verdict", "plan_name_ok", "plan_verdict", "price_text_from", "product_verdict", "refine_job", | |
| 763 | + "role_category", "states_job_location", "strip_trailing_marks", "word_count", | |
| 764 | +] | |
modified
src/companyatlas/connectors/_util.py
+1 −1
@@ -66,7 +66,7 @@ CA_PROVINCE_CODES_SAFE = {"ON", "QC", "BC", "AB", "MB", "SK", "NS", "NB"} | ||
| 66 | 66 | AMBIGUOUS_CODES = (set(US_STATES.values()) | set(CA_PROVINCES.values())) & ISO2 |
| 67 | 67 | REMOTE_RE = re.compile(r"\b(remote|work from home|wfh|anywhere|distributed|télétravail|homeoffice|home office)\b", re.IGNORECASE) |
| 68 | 68 | HYBRID_RE = re.compile(r"\bhybrid\b", re.IGNORECASE) |
| 69 | −POSTAL_RE = re.compile(r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b|\b\d{4,6}(?:-\d{4})?\b|\b[A-Z]\d[A-Z]\s*\d[A-Z]\d\b") | |
| 69 | +POSTAL_RE = re.compile(r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b|\b\d{4}\s?[A-Z]{2}\b|\b\d{4,6}(?:-\d{4})?\b|\b[A-Z]\d[A-Z]\s*\d[A-Z]\d\b") # UK · NL · numeric · CA | |
| 70 | 70 | |
| 71 | 71 | |
| 72 | 72 | def country_code(text: str | None) -> str | None: |
modified
src/companyatlas/connectors/generic_html.py
+103 −81
@@ -8,6 +8,7 @@ metadata, JSON-LD and links; this module adds *surface-aware* typed extraction: | ||
| 8 | 8 | |
| 9 | 9 | Structured data (JSON-LD / microdata) is used first, DOM heuristics second, and nothing is ever invented: no address, no country, no |
| 10 | 10 | date that the page does not state. Every surface also yields `discovered` URLs (classified links) for the discovery feedback loop. |
| 11 | +Precision rules (`connectors/_precision.py`) reject navigation / CTA / cookie-consent / marketing noise at the point of extraction. | |
| 11 | 12 | """ |
| 12 | 13 | from __future__ import annotations |
| 13 | 14 | |
@@ -19,6 +20,22 @@ from urllib.parse import urlparse | ||
| 19 | 20 | from selectolax.lexbor import LexborHTMLParser |
| 20 | 21 | |
| 21 | 22 | from companyatlas.config import settings |
| 23 | +from companyatlas.connectors._precision import ( | |
| 24 | + STREET_RE, | |
| 25 | + is_known_city, | |
| 26 | + looks_like_person_name, | |
| 27 | + looks_like_role_title, | |
| 28 | + news_verdict, | |
| 29 | + normalize_location, | |
| 30 | + normalize_person, | |
| 31 | + plan_name_ok, | |
| 32 | + plan_verdict, | |
| 33 | + price_text_from, | |
| 34 | + product_verdict, | |
| 35 | + refine_job, | |
| 36 | + role_category, | |
| 37 | + states_job_location, | |
| 38 | +) | |
| 22 | 39 | from companyatlas.connectors._util import ( |
| 23 | 40 | country_code, |
| 24 | 41 | date_from_text, |
@@ -57,52 +74,22 @@ PRODUCT_SURFACES = {Surface.PRODUCTS, Surface.SERVICES, Surface.SOLUTIONS} | ||
| 57 | 74 | |
| 58 | 75 | # ------------------------------------------------------------------------------------------------------------ people |
| 59 | 76 | |
| 60 | −ROLE_RULES: list[tuple[str, re.Pattern[str]]] = [ | |
| 61 | − ("founder", re.compile(r"\b(co-?founder|founder|fondat(eur|rice)|gründer(in)?)\b", re.IGNORECASE)), | |
| 62 | − ("ceo", re.compile(r"\b(chief executive( officer)?|ceo|pdg|président-directeur|geschäftsführer(in)?|managing director|directeur général)\b", re.IGNORECASE)), | |
| 63 | − ("cfo", re.compile(r"\b(chief financial( officer)?|cfo|finanzvorstand|directeur financier)\b", re.IGNORECASE)), | |
| 64 | − ("cto", re.compile(r"\b(chief technology( officer)?|chief technical( officer)?|cto)\b", re.IGNORECASE)), | |
| 65 | − ("coo", re.compile(r"\b(chief operating( officer)?|coo)\b", re.IGNORECASE)), | |
| 66 | − ("chair", re.compile(r"\b(chair(man|woman|person)?|executive chair|présidente? du conseil|vorsitzende[rn]?)\b", re.IGNORECASE)), | |
| 67 | − ("president", re.compile(r"\b(?<!vice )(?<!vice-)president(?! of)|président(?!e? du conseil)\b", re.IGNORECASE)), | |
| 68 | − ("board", re.compile(r"\b(board member|member of the (supervisory |advisory )?board|non-executive director|independent director|" | |
| 69 | − r"administrat(eur|rice)|aufsichtsrat|conseil d'administration|director(?= \(board)|trustee)\b", re.IGNORECASE)), | |
| 70 | − ("vp", re.compile(r"\b(vice[- ]president|vp|evp|svp|avp)\b", re.IGNORECASE)), | |
| 71 | − ("head", re.compile(r"\b(head of|head,|general manager|gm|leiter(in)?|directeur|directrice|director|managing partner|partner)\b", re.IGNORECASE)), | |
| 72 | −] | |
| 73 | −CHIEF_RE = re.compile(r"\b(chief\b|c[a-z]{1,3}o\b)", re.IGNORECASE) | |
| 74 | −EXEC_CATEGORIES = {"ceo", "cfo", "cto", "coo", "founder", "president", "chair"} | |
| 75 | −NAME_TOKEN = r"(?:[A-ZÀ-ÝĀ-Ž](?:['’][A-ZÀ-ÝĀ-Ž])?[a-zà-ÿā-ž'’]+(?:-[A-ZÀ-ÝĀ-Ža-zà-ÿā-ž][a-zà-ÿā-ž'’]*)*|[A-ZÀ-ÝĀ-Ž]\.|(?:de|van|von|der|den|da|di|le|la|du|del|bin|al)\b)" | |
| 76 | −NAME_RE = re.compile(rf"^(?:(?:Dr|Prof|Mr|Mrs|Ms|Sir|Dame|Hon)\.?\s+)?{NAME_TOKEN}(?:\s+{NAME_TOKEN}){{1,4}}(?:,?\s+(?:Jr\.?|Sr\.?|II|III|IV|PhD|MD|MBA|CPA|Esq\.?))?$") | |
| 77 | −NOT_NAME_RE = re.compile(r"\b(team|our|meet|leadership|board|executive|management|contact|about|officer|director|president|manager|head|chief|" | |
| 78 | − r"founder|partner|group|company|global|senior|vice|read|more|learn|view|profile|bio|linkedin|email|join|careers|" | |
| 79 | − r"news|press|the|and|of|for|at|in|on|to|with)\b", re.IGNORECASE) | |
| 80 | − | |
| 81 | − | |
| 82 | −def role_category(title: str | None) -> tuple[str, bool]: | |
| 83 | − if not title: | |
| 84 | − return "other", False | |
| 85 | − for cat, pat in ROLE_RULES: | |
| 86 | − if pat.search(title): | |
| 87 | − return cat, cat in EXEC_CATEGORIES or bool(CHIEF_RE.search(title)) | |
| 88 | − return "other", bool(CHIEF_RE.search(title)) | |
| 89 | − | |
| 90 | − | |
| 91 | −def looks_like_name(text: str) -> bool: | |
| 92 | − t = normalize_whitespace(text).strip(" .,:;-–—|") | |
| 93 | − if not (4 <= len(t) <= 60) or any(ch.isdigit() for ch in t): | |
| 94 | − return False | |
| 95 | − if NOT_NAME_RE.search(t): | |
| 96 | − return False | |
| 97 | − return bool(NAME_RE.match(t)) | |
| 98 | − | |
| 99 | − | |
| 100 | −def looks_like_title(text: str) -> bool: | |
| 101 | − t = normalize_whitespace(text) | |
| 102 | − if not (2 <= len(t) <= 140): | |
| 103 | − return False | |
| 104 | − return role_category(t)[0] != "other" or bool(re.search(r"\b(officer|manager|engineer|lead|counsel|scientist|architect|analyst|controller|" | |
| 105 | − r"secretary|treasurer|advisor|adviser|strategist|evangelist|designer)\b", t, re.IGNORECASE)) | |
| 77 | +# Role vocabulary, name shape and title cleaning live in `_precision` (shared with the pipeline); re-exported here for callers/tests. | |
| 78 | +looks_like_name = looks_like_person_name | |
| 79 | +looks_like_title = looks_like_role_title | |
| 80 | +PERSON_CARD_SCAN_LINES = 4 # a person card states the name within its first lines | |
| 81 | +PERSON_TITLE_WINDOW = 3 # …and the title within the next few | |
| 82 | + | |
| 83 | + | |
| 84 | +def _card_person(lines: list[str]) -> tuple[str, str | None] | None: | |
| 85 | + """(name, title) from a card's lines: the title may precede the name (swapped cards) or follow it; role-like lines win over prose.""" | |
| 86 | + name_idx = next((i for i, ln in enumerate(lines[:PERSON_CARD_SCAN_LINES]) if looks_like_name(ln)), None) | |
| 87 | + if name_idx is None: | |
| 88 | + return None | |
| 89 | + before = [x for x in lines[:name_idx] if looks_like_title(x)] | |
| 90 | + after = [x for x in lines[name_idx + 1:name_idx + 1 + PERSON_TITLE_WINDOW] if not looks_like_name(x)] | |
| 91 | + title = (before[-1] if before else None) or next((x for x in after if looks_like_title(x)), None) or (after[0] if after else None) | |
| 92 | + return lines[name_idx], title | |
| 106 | 93 | |
| 107 | 94 | |
| 108 | 95 | def extract_people(page: NormalizedPage) -> list[ExtractedPerson]: |
@@ -110,27 +97,28 @@ def extract_people(page: NormalizedPage) -> list[ExtractedPerson]: | ||
| 110 | 97 | seen: set[str] = set() |
| 111 | 98 | |
| 112 | 99 | def add(name: str, title: str | None, url: str | None = None) -> None: |
| 113 | − name = normalize_whitespace(name).strip(" .,:;-–—|") | |
| 100 | + fixed = normalize_person(name, title) | |
| 101 | + if fixed is None: | |
| 102 | + return | |
| 103 | + name, title = fixed | |
| 114 | 104 | key = norm_name(name) |
| 115 | 105 | if not key or key in seen or len(out) >= MAX_PEOPLE: |
| 116 | 106 | return |
| 117 | 107 | seen.add(key) |
| 118 | 108 | cat, is_exec = role_category(title) |
| 119 | − out.append(ExtractedPerson(name=name[:120], title=(normalize_whitespace(title)[:160] if title else None), role_category=cat, is_executive=is_exec, url=url)) | |
| 109 | + out.append(ExtractedPerson(name=name[:120], title=(title[:160] if title else None), role_category=cat, is_executive=is_exec, url=url)) | |
| 120 | 110 | |
| 121 | 111 | for p in page.jsonld.get("persons", []) + page.microdata.get("persons", []): |
| 122 | 112 | name = text_of(p.get("name")) |
| 123 | − if name and looks_like_name(name): | |
| 113 | + if name: | |
| 124 | 114 | add(name, text_of(p.get("jobTitle")) or text_of(p.get("title")), text_of(p.get("url")) if isinstance(p.get("url"), str) else None) |
| 125 | 115 | for b in page.blocks: |
| 126 | 116 | if b.kind != "person": |
| 127 | 117 | continue |
| 128 | 118 | lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()] |
| 129 | − for i, ln in enumerate(lines[:4]): | |
| 130 | − if looks_like_name(ln): | |
| 131 | − title = next((x for x in lines[i + 1:i + 4] if not looks_like_name(x) and len(x) <= 140), None) | |
| 132 | − add(ln, title, b.attrs.get("href")) | |
| 133 | − break | |
| 119 | + found = _card_person(lines) | |
| 120 | + if found is not None: | |
| 121 | + add(found[0], found[1], b.attrs.get("href")) | |
| 134 | 122 | if len(out) < 2: # fallback: heading = name, next short block = title |
| 135 | 123 | blocks = page.blocks |
| 136 | 124 | for i, b in enumerate(blocks): |
@@ -167,11 +155,11 @@ def parse_price(text: str) -> dict[str, Any] | None: | ||
| 167 | 155 | """→ {price, currency, billing_period, unit, price_text, contact_sales} or None when the text states no price.""" |
| 168 | 156 | t = normalize_whitespace(text) |
| 169 | 157 | if CONTACT_RE.search(t) and not PRICE_RE.search(t): |
| 170 | − return {"price": None, "currency": None, "billing_period": "contact", "unit": None, "price_text": t[:80], "contact_sales": True} | |
| 158 | + return {"price": None, "currency": None, "billing_period": "contact", "unit": None, "price_text": price_text_from(t) or t[:80], "contact_sales": True} | |
| 171 | 159 | m = PRICE_RE.search(t) |
| 172 | 160 | if m is None: |
| 173 | 161 | if FREE_RE.match(t): |
| 174 | − return {"price": 0.0, "currency": None, "billing_period": None, "unit": None, "price_text": t[:80], "contact_sales": False} | |
| 162 | + return {"price": 0.0, "currency": None, "billing_period": None, "unit": None, "price_text": price_text_from(t) or t[:80], "contact_sales": False} | |
| 175 | 163 | return None |
| 176 | 164 | cur = m.group("cur") or m.group("cur2") |
| 177 | 165 | amt = m.group("amt") or m.group("amt2") |
@@ -213,10 +201,28 @@ def parse_price(text: str) -> dict[str, Any] | None: | ||
| 213 | 201 | elif (u2 in YEAR_UNITS) and not period: |
| 214 | 202 | period = "year" |
| 215 | 203 | return {"price": price, "currency": CURRENCY_SYMBOLS.get(cur, cur.upper() if cur and cur.isalpha() else None), "billing_period": period, "unit": unit, |
| 216 | − "price_text": t[max(0, m.start() - 12):m.end() + 40].strip()[:80], "contact_sales": bool(CONTACT_RE.search(t))} | |
| 204 | + "price_text": price_text_from(t) or t[m.start():m.end()].strip()[:80], "contact_sales": bool(CONTACT_RE.search(t))} | |
| 217 | 205 | |
| 218 | 206 | |
| 219 | −def _plan_from_lines(name: str, lines: list[str]) -> ExtractedPlan | None: | |
| 207 | +PLAN_NAME_SCAN_LINES = 3 # the plan name is one of the first lines before the price (eyebrows / marketing headings are skipped) | |
| 208 | + | |
| 209 | + | |
| 210 | +def _plan_name_from(lines: list[str], path: str) -> tuple[str | None, list[str]]: | |
| 211 | + """(plan name, remaining lines): the first valid short label before the price line (eyebrows such as "Most popular" are skipped). | |
| 212 | + Otherwise the heading just above the card (its path tail) names the tier — unless that heading is generic ("Plans", "Pricing") or a | |
| 213 | + marketing sentence, in which case the card yields no plan.""" | |
| 214 | + for i, ln in enumerate(lines[:PLAN_NAME_SCAN_LINES]): | |
| 215 | + if plan_name_ok(ln): # "Free" is a valid tier name even though it also reads as a price | |
| 216 | + return ln, lines[i + 1:] | |
| 217 | + if PRICE_RE.search(ln) or parse_price(ln) is not None: | |
| 218 | + break | |
| 219 | + tail = path.split(" > ")[-1] if path else "" | |
| 220 | + return (tail if plan_name_ok(tail) else None), lines | |
| 221 | + | |
| 222 | + | |
| 223 | +def _plan_from_lines(name: str | None, lines: list[str]) -> ExtractedPlan | None: | |
| 224 | + if not name: | |
| 225 | + return None | |
| 220 | 226 | price_info = None |
| 221 | 227 | features: list[str] = [] |
| 222 | 228 | for ln in lines: |
@@ -227,6 +233,8 @@ def _plan_from_lines(name: str, lines: list[str]) -> ExtractedPlan | None: | ||
| 227 | 233 | continue |
| 228 | 234 | if 2 <= len(ln) <= 140 and not parse_price(ln) and len(features) < 25: |
| 229 | 235 | features.append(ln) |
| 236 | + if price_info is None and FREE_RE.match(name): | |
| 237 | + price_info = parse_price(name) # a tier literally called "Free" with no separate price line | |
| 230 | 238 | if price_info is None: |
| 231 | 239 | return None |
| 232 | 240 | return ExtractedPlan(plan_name=name[:80], price=price_info["price"], price_text=price_info["price_text"], currency=price_info["currency"], |
@@ -238,7 +246,7 @@ def extract_plans(page: NormalizedPage) -> list[ExtractedPlan]: | ||
| 238 | 246 | seen: set[str] = set() |
| 239 | 247 | |
| 240 | 248 | def add(plan: ExtractedPlan | None) -> None: |
| 241 | − if plan is None: | |
| 249 | + if plan is None or not plan_verdict(plan).ok: | |
| 242 | 250 | return |
| 243 | 251 | key = norm_name(plan.plan_name) |
| 244 | 252 | if not key or key in seen or len(out) >= MAX_PLANS: |
@@ -252,14 +260,14 @@ def extract_plans(page: NormalizedPage) -> list[ExtractedPlan]: | ||
| 252 | 260 | lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()] |
| 253 | 261 | if not lines: |
| 254 | 262 | continue |
| 255 | − name = lines[0] if parse_price(lines[0]) is None and len(lines[0]) <= 60 else (b.path.split(" > ")[-1] if b.path else "Plan") | |
| 256 | − add(_plan_from_lines(name, lines[1:] if lines[0] == name else lines)) | |
| 263 | + name, rest = _plan_name_from(lines, b.path) | |
| 264 | + add(_plan_from_lines(name, rest)) | |
| 257 | 265 | if not out: # fallback: heading followed by a price before the next heading (common in hand-rolled pricing tables) |
| 258 | 266 | blocks = page.blocks |
| 259 | 267 | i = 0 |
| 260 | 268 | while i < len(blocks): |
| 261 | 269 | b = blocks[i] |
| 262 | − if b.kind == "heading" and int(b.attrs.get("level", 2)) >= 2 and len(b.text) <= 60: | |
| 270 | + if b.kind == "heading" and int(b.attrs.get("level", 2)) >= 2 and plan_name_ok(b.text): | |
| 263 | 271 | j = i + 1 |
| 264 | 272 | lines: list[str] = [] |
| 265 | 273 | while j < len(blocks) and blocks[j].kind != "heading" and j - i <= 10: |
@@ -288,10 +296,8 @@ LOCATION_KIND_RULES: list[tuple[str, re.Pattern[str]]] = [ | ||
| 288 | 296 | ("data_center", re.compile(r"\b(data ?cent(er|re)|datacenter|server farm)\b", re.IGNORECASE)), |
| 289 | 297 | ("store", re.compile(r"\b(store|shop|boutique|showroom|outlet|dealer(ship)?|branch|agence|filiale)\b", re.IGNORECASE)), |
| 290 | 298 | ] |
| 291 | −STREET_RE = re.compile(r"\b\d{1,5}[a-z]?\s+[^\n,]{2,40}\b(street|st\.?|avenue|ave\.?|road|rd\.?|boulevard|blvd\.?|drive|dr\.?|lane|ln\.?|way|place|pl\.?|" | |
| 292 | − r"square|plaza|court|ct\.?|parkway|highway|straße|strasse|str\.|allee|platz|weg|rue|avenue|boulevard|avenida|calle|via|piazza)\b" | |
| 293 | − r"|\b(rue|avenue|boulevard|via|calle|avenida)\s+[^\n,]{2,40}\s\d{1,5}\b", re.IGNORECASE) | |
| 294 | 299 | CITY_COUNTRY_RE = re.compile(r"^([A-ZÀ-Ý][\w'’.\- ]{1,40}),\s*([A-Za-zÀ-ÿ .]{2,40})$") |
| 300 | +MAX_VENUE_LINE_CHARS = 60 # "Hormuz Grand Hotel" — a venue line under a country heading | |
| 295 | 301 | |
| 296 | 302 | |
| 297 | 303 | def _location_from_lines(name: str, lines: list[str], href: str | None = None) -> ExtractedLocation | None: |
@@ -305,8 +311,10 @@ def _location_from_lines(name: str, lines: list[str], href: str | None = None) - | ||
| 305 | 311 | probe = (ln[:street.start()] + " " + ln[street.end():]).strip(" ,") if street else ln |
| 306 | 312 | if country is None and probe: |
| 307 | 313 | loc = parse_location(probe) |
| 308 | − if loc["country"] or loc["region"]: | |
| 309 | − city, region, country = city or loc["city"], loc["region"], loc["country"] | |
| 314 | + if loc["country"]: # the line that states the country is the authoritative "City, Country" line | |
| 315 | + city, region, country = loc["city"] or city, loc["region"], loc["country"] | |
| 316 | + elif loc["region"]: | |
| 317 | + city, region = city or loc["city"], loc["region"] | |
| 310 | 318 | if country is None: |
| 311 | 319 | c = country_code(name) |
| 312 | 320 | if c: |
@@ -315,7 +323,15 @@ def _location_from_lines(name: str, lines: list[str], href: str | None = None) - | ||
| 315 | 323 | m = CITY_COUNTRY_RE.match(name) |
| 316 | 324 | if m and country_code(m.group(2)): |
| 317 | 325 | city, country = m.group(1).strip(), country_code(m.group(2)) |
| 318 | − if city is None and country is not None and looks_like_city(name): | |
| 326 | + if country_code(name) and city is None: | |
| 327 | + # heading is a country ("Oman") — the first short line that is neither an address nor a place is the venue / office name | |
| 328 | + venue = next((ln for ln in lines if ln != address and len(ln) <= MAX_VENUE_LINE_CHARS and not parse_location(ln)["country"] | |
| 329 | + and not country_code(ln) and not any(ch.isdigit() for ch in ln)), None) | |
| 330 | + if venue: | |
| 331 | + name = venue | |
| 332 | + elif is_known_city(name): | |
| 333 | + city = name # city-states: Singapore, Monaco, Hong Kong | |
| 334 | + elif city is None and country is not None and looks_like_city(name): | |
| 319 | 335 | city = name |
| 320 | 336 | if city is None and country is None and address is None and kind == "office": |
| 321 | 337 | return None |
@@ -332,6 +348,7 @@ def extract_locations(page: NormalizedPage) -> list[ExtractedLocation]: | ||
| 332 | 348 | seen: set[str] = set() |
| 333 | 349 | |
| 334 | 350 | def add(loc: ExtractedLocation | None) -> None: |
| 351 | + loc = normalize_location(loc) if loc is not None else None | |
| 335 | 352 | if loc is None: |
| 336 | 353 | return |
| 337 | 354 | key = norm_name(loc.name) |
@@ -371,6 +388,7 @@ def extract_locations(page: NormalizedPage) -> list[ExtractedLocation]: | ||
| 371 | 388 | # ------------------------------------------------------------------------------------------------------------ news |
| 372 | 389 | |
| 373 | 390 | NEWS_NOISE_ANCHOR = re.compile(r"^(read more|learn more|more|continue reading|view all|see all|all news|all posts|next|previous|older|newer|\d+)$", re.IGNORECASE) |
| 391 | +MIN_NEWS_TITLE_CHARS = 3 # anything shorter is a glyph or a counter; the precision rule decides the rest (words / date) | |
| 374 | 392 | |
| 375 | 393 | |
| 376 | 394 | def _time_index(html: str, base_url: str) -> dict[str, Any]: |
@@ -407,14 +425,16 @@ def extract_news(page: NormalizedPage, html: str, *, surface: str, base_url: str | ||
| 407 | 425 | |
| 408 | 426 | def add(title: str, url: str, published: Any, summary: str | None = None) -> None: |
| 409 | 427 | title = normalize_whitespace(title) |
| 410 | − if len(title) < 8 or NEWS_NOISE_ANCHOR.match(title) or len(out) >= MAX_NEWS: | |
| 428 | + if len(title) < MIN_NEWS_TITLE_CHARS or NEWS_NOISE_ANCHOR.match(title) or len(out) >= MAX_NEWS: | |
| 411 | 429 | return |
| 412 | 430 | canon = canonicalize_url(url) |
| 413 | 431 | if canon in seen or canon == canonicalize_url(base_url) or is_static_asset(url): |
| 414 | 432 | return |
| 433 | + published = published or times.get(canon) | |
| 434 | + if not news_verdict(title, url=url, published_at=published).ok: | |
| 435 | + return | |
| 415 | 436 | seen.add(canon) |
| 416 | − out.append(ExtractedNewsItem(title=title[:300], url=url, published_at=published or times.get(canon), summary=(summary or None), category=category, | |
| 417 | − language=page.lang)) | |
| 437 | + out.append(ExtractedNewsItem(title=title[:300], url=url, published_at=published, summary=(summary or None), category=category, language=page.lang)) | |
| 418 | 438 | |
| 419 | 439 | for a in page.jsonld.get("articles", []) + page.microdata.get("articles", []): |
| 420 | 440 | title, url = text_of(a.get("headline")) or text_of(a.get("name")), a.get("url") or (a.get("mainEntityOfPage") if isinstance(a.get("mainEntityOfPage"), str) else None) |
@@ -425,8 +445,9 @@ def extract_news(page: NormalizedPage, html: str, *, surface: str, base_url: str | ||
| 425 | 445 | if b.kind != "news_item" or not href: |
| 426 | 446 | continue |
| 427 | 447 | lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()] |
| 428 | − title = next((ln for ln in lines if len(ln) >= 12 and not (len(ln) <= 32 and date_from_text(ln))), lines[0] if lines else "") | |
| 429 | − rest = [ln for ln in lines if ln != title and not (len(ln) <= 32 and date_from_text(ln))] | |
| 448 | + undated = [ln for ln in lines if not (len(ln) <= 32 and date_from_text(ln))] | |
| 449 | + title = next((ln for ln in undated if len(ln) >= 12), undated[0] if undated else "") | |
| 450 | + rest = [ln for ln in undated if ln != title] | |
| 430 | 451 | add(title, href, date_from_text(b.text) or times.get(canonicalize_url(href)) or date_from_url(href), " ".join(rest[:2])[:300] or None) |
| 431 | 452 | if len(out) < 3: # fallback: article-like links in the main region (same site, deeper path, long anchor) |
| 432 | 453 | base_depth = urlparse(base_url).path.strip("/").count("/") |
@@ -451,11 +472,7 @@ JOB_NOISE_ANCHOR = re.compile(r"^(apply( now)?|view( all)?( jobs| openings| role | ||
| 451 | 472 | r"browse jobs|explore|more|back|home|filter|next|previous|\d+)$", re.IGNORECASE) |
| 452 | 473 | |
| 453 | 474 | |
| 454 | −def _states_location(text: str) -> bool: | |
| 455 | − if len(text) > 80: | |
| 456 | − return False | |
| 457 | − p = parse_location(text) | |
| 458 | − return bool(p["city"] or p["country"] or p["remote"] or p["region"]) | |
| 475 | +_states_location = states_job_location | |
| 459 | 476 | |
| 460 | 477 | |
| 461 | 478 | def extract_jobs_html(page: NormalizedPage, *, base_url: str) -> list[ExtractedJob]: |
@@ -466,11 +483,14 @@ def extract_jobs_html(page: NormalizedPage, *, base_url: str) -> list[ExtractedJ | ||
| 466 | 483 | title = normalize_whitespace(title).strip(" -–—|·") |
| 467 | 484 | if not (4 <= len(title) <= 140) or JOB_NOISE_ANCHOR.match(title) or len(jobs) >= MAX_JOBS: |
| 468 | 485 | return |
| 469 | − key = (canonicalize_url(url) if url else "") + "|" + title.lower() | |
| 486 | + job = refine_job(ExtractedJob(title=title, url=url, location_text=location, department=department)) | |
| 487 | + if job is None: | |
| 488 | + return | |
| 489 | + key = (canonicalize_url(url) if url else "") + "|" + job.title.lower() | |
| 470 | 490 | if key in seen: |
| 471 | 491 | return |
| 472 | 492 | seen.add(key) |
| 473 | − jobs.append(finish_job(ExtractedJob(title=title, url=url, location_text=location, department=department))) | |
| 493 | + jobs.append(finish_job(job)) | |
| 474 | 494 | |
| 475 | 495 | jl_jobs = jobs_from_jsonld(list(page.jsonld.get("job_postings", [])) + list(page.microdata.get("job_postings", [])), page_url=base_url) |
| 476 | 496 | for j in jl_jobs: |
@@ -488,7 +508,7 @@ def extract_jobs_html(page: NormalizedPage, *, base_url: str) -> list[ExtractedJ | ||
| 488 | 508 | if not lines: |
| 489 | 509 | continue |
| 490 | 510 | title = lines[0] |
| 491 | − loc = next((ln for ln in lines[1:6] if _states_location(ln)), None) | |
| 511 | + loc = next((ln for ln in lines[1:6] if _states_location(ln) or is_known_city(ln)), None) | |
| 492 | 512 | dept = next((ln for ln in lines[1:6] if ln != loc and 2 <= len(ln) <= 40 and not _states_location(ln) and not date_from_text(ln)), None) |
| 493 | 513 | before = len(jobs) |
| 494 | 514 | add(title, href, loc, dept) |
@@ -517,6 +537,8 @@ def extract_products(page: NormalizedPage) -> list[ExtractedProduct]: | ||
| 517 | 537 | key = norm_name(name) |
| 518 | 538 | if not (2 <= len(name) <= 90) or not key or key in seen or len(out) >= MAX_PRODUCTS or NEWS_NOISE_ANCHOR.match(name): |
| 519 | 539 | return |
| 540 | + if not product_verdict(name).ok: | |
| 541 | + return | |
| 520 | 542 | seen.add(key) |
| 521 | 543 | out.append(ExtractedProduct(name=name, url=url, category=category, description=(description or None))) |
| 522 | 544 | |
modified
src/companyatlas/sdk/normalize.py
+57 −12
@@ -340,12 +340,55 @@ def _card_kind(node: LexborNode, *, allow_container: bool = False) -> str | None | ||
| 340 | 340 | return kind |
| 341 | 341 | |
| 342 | 342 | |
| 343 | +LISTING_LINK_SHARE = 0.6 # a list whose items mostly carry links is a listing (jobs, news, products), not a bullet list | |
| 344 | +LISTING_MIN_ITEMS = 2 | |
| 345 | + | |
| 346 | + | |
| 347 | +def _list_items(node: LexborNode) -> list[LexborNode]: | |
| 348 | + items: list[LexborNode] = [] | |
| 349 | + for c in node.iter(): | |
| 350 | + if c.tag in ("li", "tr"): | |
| 351 | + items.append(c) | |
| 352 | + elif c.tag in ("tbody", "thead", "tfoot"): | |
| 353 | + items.extend(x for x in c.iter() if x.tag == "tr") | |
| 354 | + return items | |
| 355 | + | |
| 356 | + | |
| 357 | +def _is_listing(node: LexborNode) -> bool: | |
| 358 | + """`ul`/`ol`/`table` whose items are card-like or mostly linked — the rows are entities, the wrapper is a container.""" | |
| 359 | + items = _list_items(node) | |
| 360 | + if len(items) < LISTING_MIN_ITEMS: | |
| 361 | + return False | |
| 362 | + if any(_card_kind(i, allow_container=True) for i in items): | |
| 363 | + return True | |
| 364 | + linked = sum(1 for i in items if i.css_first("a[href]") is not None) | |
| 365 | + return linked >= len(items) * LISTING_LINK_SHARE | |
| 366 | + | |
| 367 | + | |
| 368 | +def _card_text(node: LexborNode) -> str: | |
| 369 | + """Card text with one line per direct child element, so inline cells (`<a>title</a><span>city</span>`) do not merge into one line.""" | |
| 370 | + parts: list[str] = [] | |
| 371 | + for c in node.iter(include_text=True): | |
| 372 | + if c.tag == "-text": | |
| 373 | + t = c.text_content | |
| 374 | + if t and t.strip(): | |
| 375 | + parts.append(t) | |
| 376 | + elif c.tag in DROP_TAGS or c.tag == "-comment" or c.tag == "br" or _is_hidden(c) or _is_cookie(c): | |
| 377 | + continue | |
| 378 | + else: | |
| 379 | + parts.append(_node_text(c)) | |
| 380 | + return normalize_whitespace(html_lib.unescape("\n".join(p for p in parts if p))) | |
| 381 | + | |
| 382 | + | |
| 343 | 383 | def _is_card_container(node: LexborNode) -> bool: |
| 344 | − """A grid/list wrapper (`.products`, `.news-list`, `.team-grid`) holds ≥ 2 card-like children — segment the children instead.""" | |
| 384 | + """A grid/list wrapper (`.products`, `.news-list`, `.team-grid`, `.job-list`) holds ≥ 2 card-like children — or a listing-like list/table — | |
| 385 | + so the children are segmented instead of the wrapper becoming a single card. A feature `<ul>` inside a pricing card is not a listing.""" | |
| 345 | 386 | n = 0 |
| 346 | 387 | for child in node.iter(): |
| 347 | 388 | if child.tag not in ("div", "section", "article", "li", "a", "figure", "tr", "ul", "ol", "table"): |
| 348 | 389 | continue |
| 390 | + if child.tag in ("ul", "ol", "table") and _is_listing(child): | |
| 391 | + return True | |
| 349 | 392 | is_card = child.tag in ("ul", "ol", "table") or _card_kind(child, allow_container=True) is not None |
| 350 | 393 | if not is_card and child.tag == "div": |
| 351 | 394 | is_card = any(_card_kind(g, allow_container=True) for g in child.iter() if g.tag in ("div", "article", "li", "a")) |
@@ -388,7 +431,8 @@ class _Segmenter: | ||
| 388 | 431 | self.order += 1 |
| 389 | 432 | |
| 390 | 433 | # ---------------------------------------------------------------- traversal |
| 391 | − def walk(self, node: LexborNode, region: str, depth: int = 0) -> None: | |
| 434 | + def walk(self, node: LexborNode, region: str, depth: int = 0, hint: str | None = None) -> None: | |
| 435 | + """`hint` is the card kind of an enclosing container (`.job-list`), inherited by plain list rows / table rows underneath it.""" | |
| 392 | 436 | if depth > 300: |
| 393 | 437 | return |
| 394 | 438 | for c in node.iter(include_text=True): |
@@ -436,16 +480,16 @@ class _Segmenter: | ||
| 436 | 480 | self.add("faq", _node_text(c)) |
| 437 | 481 | continue |
| 438 | 482 | if tag in ("ul", "ol", "dl", "menu"): |
| 439 | − self.list_block(c) | |
| 483 | + self.list_block(c, hint) | |
| 440 | 484 | continue |
| 441 | 485 | if tag == "table": |
| 442 | − self.table_block(c) | |
| 486 | + self.table_block(c, hint) | |
| 443 | 487 | continue |
| 444 | 488 | if tag in ("hr", "br", "wbr"): |
| 445 | 489 | continue |
| 446 | 490 | card = _card_kind(c) if tag in ("div", "section", "article", "li", "a", "figure", "aside", "tr", "td") else None |
| 447 | 491 | if card is not None: |
| 448 | − txt = _node_text(c) | |
| 492 | + txt = _card_text(c) | |
| 449 | 493 | if txt and len(txt) <= CARD_MAX_TEXT: |
| 450 | 494 | attrs: dict[str, Any] = {} |
| 451 | 495 | href = self._first_href(c) |
@@ -464,7 +508,8 @@ class _Segmenter: | ||
| 464 | 508 | break |
| 465 | 509 | self.add("hero", txt, path="", attrs={"href": self._first_href(c)} if self._first_href(c) else {}) |
| 466 | 510 | continue |
| 467 | − self.walk(c, region, depth + 1) | |
| 511 | + container_kind = _card_kind(c, allow_container=True) if tag in ("div", "section", "article", "ul", "ol") else None | |
| 512 | + self.walk(c, region, depth + 1, container_kind or hint) | |
| 468 | 513 | |
| 469 | 514 | def _has_h1(self, node: LexborNode) -> bool: |
| 470 | 515 | return node.css_first("h1") is not None and len(_node_text(node)) < 2500 |
@@ -477,34 +522,34 @@ class _Segmenter: | ||
| 477 | 522 | |
| 478 | 523 | return absolutize(self.base_url, _attr(a, "href")) |
| 479 | 524 | |
| 480 | − def list_block(self, node: LexborNode) -> None: | |
| 525 | + def list_block(self, node: LexborNode, hint: str | None = None) -> None: | |
| 481 | 526 | items = [li for li in node.iter() if li.tag in ("li", "dt", "dd")] |
| 482 | 527 | if not items: |
| 483 | 528 | txt = _node_text(node) |
| 484 | 529 | if txt: |
| 485 | 530 | self.add("list", txt) |
| 486 | 531 | return |
| 487 | − texts = [_node_text(li) for li in items] | |
| 532 | + texts = [_card_text(li) for li in items] | |
| 488 | 533 | texts_nonempty = [t for t in texts if t] |
| 489 | 534 | if not texts_nonempty: |
| 490 | 535 | return |
| 491 | 536 | avg = sum(len(t) for t in texts_nonempty) / len(texts_nonempty) |
| 492 | 537 | has_links = sum(1 for li in items if li.css_first("a[href]") is not None) |
| 493 | − listing_like = len(items) >= 3 and (has_links >= len(items) * 0.6 or avg > 60) | |
| 538 | + listing_like = len(items) >= 3 and (has_links >= len(items) * LISTING_LINK_SHARE or avg > 60) | |
| 494 | 539 | if not listing_like or len(items) > 400: |
| 495 | 540 | self.add("list", "\n".join(texts_nonempty)[:4000]) |
| 496 | 541 | return |
| 497 | 542 | for li, t in zip(items, texts, strict=False): |
| 498 | 543 | if not t: |
| 499 | 544 | continue |
| 500 | − kind = _card_kind(li, allow_container=True) or _card_kind(node, allow_container=True) or "list" | |
| 545 | + kind = _card_kind(li, allow_container=True) or _card_kind(node, allow_container=True) or hint or "list" | |
| 501 | 546 | attrs: dict[str, Any] = {} |
| 502 | 547 | href = self._first_href(li) |
| 503 | 548 | if href: |
| 504 | 549 | attrs["href"] = href |
| 505 | 550 | self.add(kind, t[:1500], attrs=attrs) |
| 506 | 551 | |
| 507 | − def table_block(self, node: LexborNode) -> None: | |
| 552 | + def table_block(self, node: LexborNode, hint: str | None = None) -> None: | |
| 508 | 553 | rows = [tr for tr in node.css("tr")] |
| 509 | 554 | caption = node.css_first("caption") |
| 510 | 555 | cap = _node_text(caption) if caption is not None else "" |
@@ -522,7 +567,7 @@ class _Segmenter: | ||
| 522 | 567 | txt = " | ".join(c for c in cells if c) |
| 523 | 568 | if not txt: |
| 524 | 569 | continue |
| 525 | − kind = _card_kind(tr, allow_container=True) or "table" | |
| 570 | + kind = _card_kind(tr, allow_container=True) or hint or "table" | |
| 526 | 571 | attrs: dict[str, Any] = {"row": i} |
| 527 | 572 | href = self._first_href(tr) |
| 528 | 573 | if href: |
modified
src/companyatlas/services/pipeline.py
+9 −3
@@ -27,6 +27,7 @@ from typing import Any | ||
| 27 | 27 | |
| 28 | 28 | from companyatlas import archive |
| 29 | 29 | from companyatlas.config import settings |
| 30 | +from companyatlas.connectors._precision import HTML_JOB_CONNECTORS, apply_precision | |
| 30 | 31 | from companyatlas.connectors._util import is_engineering, job_fingerprint, norm_name |
| 31 | 32 | from companyatlas.db import execute, fetch_all, fetch_one, jsonb, transaction |
| 32 | 33 | from companyatlas.fetch import ( |
@@ -526,8 +527,10 @@ def _clean_label(value: str | None) -> str | None: | ||
| 526 | 527 | return v or None |
| 527 | 528 | |
| 528 | 529 | |
| 529 | −def _drop_corrupt_entities(ex: Extraction) -> None: | |
| 530 | − """Remove typed items whose labels carry replacement characters or no letters at all (mis-decoded or empty extractions).""" | |
| 530 | +def _drop_corrupt_entities(ex: Extraction, *, connector_id: str | None = None) -> None: | |
| 531 | + """Remove typed items whose labels carry replacement characters or no letters at all (mis-decoded or empty extractions), then apply the | |
| 532 | + shared precision rules (`connectors/_precision`) as the last line of defence: CTA/nav anchors, cookie categories, marketing headings, | |
| 533 | + swapped person cards. Job rules apply to HTML-scraped listings only — structured boards (ATS) are trusted.""" | |
| 531 | 534 | def ok(label: str | None) -> bool: |
| 532 | 535 | return bool(label) and "\ufffd" not in label and re.search(r"[^\W\d_]", label) is not None |
| 533 | 536 | |
@@ -551,6 +554,9 @@ def _drop_corrupt_entities(ex: Extraction) -> None: | ||
| 551 | 554 | ex.news = [n for n in ex.news if ok(n.title) and "\ufffd" not in (n.url or "")] |
| 552 | 555 | if ex.title: |
| 553 | 556 | ex.title = _clean_label(ex.title) |
| 557 | + dropped = apply_precision(ex, html_jobs=connector_id in HTML_JOB_CONNECTORS) | |
| 558 | + if dropped: | |
| 559 | + log.debug("precision rules dropped items", extra={"connector_id": connector_id, "dropped": dropped}) | |
| 554 | 560 | |
| 555 | 561 | |
| 556 | 562 | # ------------------------------------------------------------------------------------------------------------ the run |
@@ -678,7 +684,7 @@ async def run_sensor(sensor: dict[str, Any], *, fetcher: Fetcher, worker: str = | ||
| 678 | 684 | await _ledger(conn, company_id, connector.connector_id, pages, len(fetched.content)) |
| 679 | 685 | outcome.duration_ms = int((time.perf_counter() - t0) * 1000) |
| 680 | 686 | return outcome |
| 681 | − _drop_corrupt_entities(ex) | |
| 687 | + _drop_corrupt_entities(ex, connector_id=connector.connector_id) | |
| 682 | 688 | ex.normalized_hash = ex.normalized_hash or text_hash(ex.text) |
| 683 | 689 | ex.structured_hash = ex.structured_hash or _structured_hash(ex) |
| 684 | 690 | struct_hash = structural_hash(ex.blocks) |
added
tests/test_extraction_precision.py
+299 −0
@@ -0,0 +1,299 @@ | ||
| 1 | +"""Precision rules (connectors/_precision): the navigation / CTA / cookie-consent / marketing noise observed in production | |
| 2 | +(M2U64, 2026-09-13) is rejected while every real item in the fixtures is still extracted — precision must not cost recall.""" | |
| 3 | +from __future__ import annotations | |
| 4 | + | |
| 5 | +from datetime import UTC, datetime | |
| 6 | + | |
| 7 | +import pytest | |
| 8 | +from conftest import fixture_path | |
| 9 | + | |
| 10 | +from companyatlas.connectors import _precision as P | |
| 11 | +from companyatlas.fetch import file_result | |
| 12 | +from companyatlas.sdk import connector as C | |
| 13 | +from companyatlas.sdk.models import ( | |
| 14 | + ExtractedJob, | |
| 15 | + ExtractedLocation, | |
| 16 | + ExtractedNewsItem, | |
| 17 | + ExtractedPerson, | |
| 18 | + ExtractedPlan, | |
| 19 | + ExtractedProduct, | |
| 20 | + Extraction, | |
| 21 | +) | |
| 22 | +from companyatlas.services.pipeline import _drop_corrupt_entities | |
| 23 | + | |
| 24 | +BASE = "https://www.ex.example" | |
| 25 | +CFG = {"canonical_domain": "ex.example"} | |
| 26 | + | |
| 27 | + | |
| 28 | +def _run(surface: str, name: str, path: str): # type: ignore[no-untyped-def] | |
| 29 | + url = BASE + path | |
| 30 | + return C.get("generic-html-v1").extract({"url": url, "surface": surface, "config": CFG}, file_result(fixture_path("generic_html", name), url=url)) | |
| 31 | + | |
| 32 | + | |
| 33 | +# ------------------------------------------------------------------------------------------------------------ fixtures end to end | |
| 34 | + | |
| 35 | + | |
| 36 | +def test_careers_cta_anchors_rejected_real_listings_kept() -> None: | |
| 37 | + ex = _run("careers", "careers_cta_noise.html", "/nl/jobs") | |
| 38 | + titles = [j.title for j in ex.jobs] | |
| 39 | + assert titles == ["Senior Backend Software Engineer - Infrastructure", "Verpleegkundige spoedgevallen", "Data Analyst", "Accountmanager KMO", "Technicien réseau"] | |
| 40 | + assert not {"Voor starters die willen gáán", "Early careers bij Telenet group", "Meer info", "Sales", "Alle vacatures", "Waarom werken bij ons?"} & set(titles) | |
| 41 | + by = {j.title: j for j in ex.jobs} | |
| 42 | + assert by["Senior Backend Software Engineer - Infrastructure"].location_text == "London, United Kingdom" | |
| 43 | + assert by["Senior Backend Software Engineer - Infrastructure"].department == "Engineering" and by["Senior Backend Software Engineer - Infrastructure"].country == "GB" | |
| 44 | + assert by["Data Analyst"].external_id == "12933" and by["Data Analyst"].country == "DE" # id stripped from display, kept for identity | |
| 45 | + assert by["Verpleegkundige spoedgevallen"].city == "Gent" and by["Verpleegkundige spoedgevallen"].country == "BE" | |
| 46 | + assert [b.kind for b in ex.blocks].count("job_listing") == 5 | |
| 47 | + | |
| 48 | + | |
| 49 | +def test_careers_german_gender_markers_and_cta() -> None: | |
| 50 | + ex = _run("careers", "careers_de.html", "/karriere") | |
| 51 | + assert [j.title for j in ex.jobs] == ["Softwareentwickler Backend", "Projektleiter Anlagenbau", "Ausbildung zum Industriemechaniker 2027", | |
| 52 | + "Werkstudent Marketing", "Pflegefachkraft Intensivstation"] | |
| 53 | + assert [j.location_text for j in ex.jobs] == ["München", "Stuttgart", "Hamburg", "Berlin", "Köln"] # bare known cities are locations | |
| 54 | + assert ex.jobs[0].country is None # …but no country is guessed | |
| 55 | + | |
| 56 | + | |
| 57 | +def test_leadership_swapped_cards_and_contact_links() -> None: | |
| 58 | + ex = _run("leadership", "leadership_swapped.html", "/leadership") | |
| 59 | + people = {p.name: p for p in ex.people} | |
| 60 | + assert set(people) == {"Jane Doe", "Brian Jacobson", "Priya Natarajan", "Marc van der Berg", "Samuel Adebayo"} | |
| 61 | + assert people["Jane Doe"].title == "Chief Executive Officer" and people["Jane Doe"].role_category == "ceo" and people["Jane Doe"].is_executive | |
| 62 | + assert people["Samuel Adebayo"].title == "Chair of the Board" and people["Samuel Adebayo"].role_category == "chair" | |
| 63 | + assert people["Brian Jacobson"].title == "Chief Financial Officer" and people["Brian Jacobson"].role_category == "cfo" | |
| 64 | + assert people["Priya Natarajan"].title is None and people["Priya Natarajan"].role_category == "other" # "Contact" is not a title | |
| 65 | + assert people["Marc van der Berg"].role_category == "coo" | |
| 66 | + | |
| 67 | + | |
| 68 | +def test_leadership_japanese() -> None: | |
| 69 | + ex = _run("leadership", "leadership_ja.html", "/company/officers") | |
| 70 | + people = {p.name: p for p in ex.people} | |
| 71 | + assert set(people) == {"山田 太郎", "佐藤 花子", "鈴木 一郎", "高橋 美咲"} | |
| 72 | + assert people["山田 太郎"].role_category == "ceo" and people["山田 太郎"].is_executive | |
| 73 | + assert people["佐藤 花子"].role_category == "cfo" and people["佐藤 花子"].is_executive | |
| 74 | + assert people["鈴木 一郎"].role_category == "board" and not people["鈴木 一郎"].is_executive | |
| 75 | + assert people["高橋 美咲"].role_category == "vp" | |
| 76 | + | |
| 77 | + | |
| 78 | +def test_locations_cookie_categories_and_nav_cards_rejected() -> None: | |
| 79 | + ex = _run("locations", "locations_cookie_noise.html", "/locations") | |
| 80 | + locs = {loc.name: loc for loc in ex.locations} | |
| 81 | + assert set(locs) == {"Hormuz Grand Hotel", "Dubai Office", "London", "Singapore", "Rotterdam Warehouse"} | |
| 82 | + assert locs["Hormuz Grand Hotel"].country == "OM" and locs["Hormuz Grand Hotel"].city is None # "Oman | Hormuz Grand Hotel" repaired | |
| 83 | + assert locs["Dubai Office"].city == "Dubai" and locs["Dubai Office"].country == "AE" # not "Level 12" | |
| 84 | + assert locs["Rotterdam Warehouse"].city == "Rotterdam" and locs["Rotterdam Warehouse"].kind == "warehouse" and locs["Rotterdam Warehouse"].country == "NL" | |
| 85 | + assert locs["London"].address_text.startswith("1 Finsbury Avenue") and locs["Singapore"].city == "Singapore" | |
| 86 | + assert all(loc.kind != "store" for loc in ex.locations) # "…providers store data" is not a shop | |
| 87 | + | |
| 88 | + | |
| 89 | +def test_pricing_marketing_headings_rejected_real_tiers_kept() -> None: | |
| 90 | + ex = _run("pricing", "pricing_marketing_noise.html", "/pricing") | |
| 91 | + plans = {p.plan_name: p for p in ex.plans} | |
| 92 | + assert {"Free", "Starter", "Team", "Business", "Enterprise"} <= set(plans) | |
| 93 | + assert not {"Win your market with Similar Example for businesses", "Worry-free roaming.", "Unlock the full potential of your data", "Most popular", "Plans"} & set(plans) | |
| 94 | + assert plans["Free"].price == 0 and plans["Free"].price_text == "$0 forever" | |
| 95 | + assert plans["Starter"].price == 125 and plans["Starter"].billing_period == "month" and plans["Starter"].price_text == "$125 per month, billed annually" | |
| 96 | + assert plans["Team"].price == 333 and plans["Team"].currency == "EUR" and plans["Team"].price_text == "Starting at €333 / month" | |
| 97 | + assert plans["Business"].price == 1199 and plans["Business"].unit == "user" and plans["Business"].price_text == "US$ 1,199 per user / month" | |
| 98 | + assert plans["Enterprise"].contact_sales and plans["Enterprise"].price is None and plans["Enterprise"].price_text == "Talk to sales" | |
| 99 | + assert all("destina" not in (p.price_text or "") for p in ex.plans) # never a truncated sentence | |
| 100 | + | |
| 101 | + | |
| 102 | +def test_products_nav_words_and_slogans_rejected() -> None: | |
| 103 | + ex = _run("products", "products_news_noise.html", "/products") | |
| 104 | + assert [p.name for p in ex.products] == ["Atlas Metrics™", "Atlas Logs®", "Atlas Traces"] # ® / ™ kept | |
| 105 | + | |
| 106 | + | |
| 107 | +def test_news_pagination_and_category_labels_rejected() -> None: | |
| 108 | + ex = _run("newsroom", "products_news_noise.html", "/news") | |
| 109 | + titles = [n.title for n in ex.news] | |
| 110 | + assert titles == ["Acme launches Atlas AI, an assistant for cloud operations", "Q2 results", "Acme and BigCo announce strategic partnership"] | |
| 111 | + assert ex.news[1].published_at.date().isoformat() == "2026-08-28" # short title kept because a date confirms it | |
| 112 | + | |
| 113 | + | |
| 114 | +def test_existing_fixtures_recall_unchanged() -> None: | |
| 115 | + assert {p.plan_name for p in _run("pricing", "pricing.html", "/pricing").plans} == {"Starter", "Pro", "Enterprise"} | |
| 116 | + assert len(_run("leadership", "leadership.html", "/about/leadership").people) == 7 | |
| 117 | + assert len(_run("locations", "locations.html", "/company/locations").locations) == 8 | |
| 118 | + assert len(_run("careers", "careers.html", "/careers").jobs) == 5 | |
| 119 | + assert len(_run("newsroom", "newsroom.html", "/news").news) == 4 | |
| 120 | + | |
| 121 | + | |
| 122 | +# ------------------------------------------------------------------------------------------------------------ unit rules | |
| 123 | + | |
| 124 | + | |
| 125 | +@pytest.mark.parametrize("title,url,location,ok", [ | |
| 126 | + ("Meer info", "https://x.example/nl/jobs/search?page=2", None, False), | |
| 127 | + ("Voor starters die willen gáán", "https://x.example/nl/jobs/starters", None, False), | |
| 128 | + ("Early careers bij Telenet group", "https://x.example/nl/jobs/early-careers", None, False), | |
| 129 | + ("Mehr erfahren →", "https://x.example/karriere/stellen", None, False), | |
| 130 | + ("Sales", "https://x.example/nl/jobs/12990-sales", None, False), # single word | |
| 131 | + ("Wij zoeken mensen die het verschil willen maken…", "https://x.example/jobs/13001", "Mechelen", False), | |
| 132 | + ("Senior Backend Software Engineer - Infrastructure", None, "London, United Kingdom", True), | |
| 133 | + ("Regional Coordinator", "https://x.example/careers/x", None, True), # role vocabulary | |
| 134 | + ("Something Unusual", "https://x.example/jobs/12345-something-unusual", None, True), # job-like URL | |
| 135 | + ("Something Unusual", None, "Toronto, ON, Canada", True), # explicit location cell | |
| 136 | + ("Something Unusual", None, None, False), # no signal at all | |
| 137 | + ("Infirmier(ère) de nuit", None, None, True), | |
| 138 | + ("Ingeniero de datos", None, None, True), | |
| 139 | + ("ソフトウェアエンジニア", None, None, True), | |
| 140 | +]) | |
| 141 | +def test_job_verdict(title: str, url: str | None, location: str | None, ok: bool) -> None: | |
| 142 | + assert P.job_verdict(title, url=url, location=location).ok is ok | |
| 143 | + | |
| 144 | + | |
| 145 | +@pytest.mark.parametrize("raw,clean,ident", [ | |
| 146 | + ("Data Analyst (m/w/d) [12933]", "Data Analyst", "12933"), | |
| 147 | + ("Accountmanager KMO - Apply now", "Accountmanager KMO", None), | |
| 148 | + ("Technicien réseau (h/f)", "Technicien réseau", None), | |
| 149 | + ("Werkstudent Marketing (all genders)", "Werkstudent Marketing", None), | |
| 150 | + ("Product Manager (Job ID: 44812) →", "Product Manager", "44812"), | |
| 151 | + ("Senior Engineer (REQ-501)", "Senior Engineer", "REQ-501"), | |
| 152 | +]) | |
| 153 | +def test_clean_job_title(raw: str, clean: str, ident: str | None) -> None: | |
| 154 | + assert P.clean_job_title(raw) == (clean, ident) | |
| 155 | + | |
| 156 | + | |
| 157 | +@pytest.mark.parametrize("name,title,expected", [ | |
| 158 | + ("Chief Executive Officer", "Jane Doe", ("Jane Doe", "Chief Executive Officer")), # swapped card | |
| 159 | + ("Chair Emeritus", "Warner Bros. Discovery", None), # role as name, company as title | |
| 160 | + ("Brian Jacobson", "Contact", ("Brian Jacobson", None)), | |
| 161 | + ("Board of Directors", "Meet the people who govern the company", None), | |
| 162 | + ("Leadership", "Read more", None), | |
| 163 | + ("Contact", "Media relations", None), | |
| 164 | + ("Our team", None, None), | |
| 165 | + ("Marc van der Berg", "Chief Operating Officer", ("Marc van der Berg", "Chief Operating Officer")), | |
| 166 | + ("Jane Doe", "Jane leads the company since 2019 and previously ran BigCo.", ("Jane Doe", None)), # bio sentence is not a title | |
| 167 | + ("Dr. Aiko Tanaka", "Head of People", ("Dr. Aiko Tanaka", "Head of People")), | |
| 168 | + ("Jane Doe 2", "CEO", None), | |
| 169 | + ("Edgar S. Woolard, Jr.", "Key person", ("Edgar S. Woolard, Jr.", "Key person")), # abbreviation dot kept | |
| 170 | +]) | |
| 171 | +def test_normalize_person(name: str, title: str | None, expected: tuple[str, str | None] | None) -> None: | |
| 172 | + assert P.normalize_person(name, title) == expected | |
| 173 | + | |
| 174 | + | |
| 175 | +@pytest.mark.parametrize("name", [ | |
| 176 | + "Judy McGrath", "F. William McNabb III", "Michael G. McCaffery", "Catherine MacGregor", "Francis deSouza", "Calvin McDonald", | |
| 177 | + "José Vicente de los Mozos", "Alexander Trotman, Baron Trotman", "Stephen Green, Baron Green of Hurstpierpoint", "Thomas John Watson, Sr.", | |
| 178 | + "Leonardo DiCaprio", "Marc van der Berg", "María García-López", "Tom O'Neill", "山田 太郎", | |
| 179 | +]) | |
| 180 | +def test_real_world_names_are_names(name: str) -> None: | |
| 181 | + assert P.looks_like_person_name(name) | |
| 182 | + | |
| 183 | + | |
| 184 | +@pytest.mark.parametrize("name", ["Meet the team", "Chief Executive Officer", "Read more", "Warner Bros. Discovery", "Key person", "Doe, Jane", "株式会社サンプル"]) | |
| 185 | +def test_non_names_are_rejected(name: str) -> None: | |
| 186 | + assert not P.looks_like_person_name(name) | |
| 187 | + | |
| 188 | + | |
| 189 | +def test_role_category_after_swap_is_consistent() -> None: | |
| 190 | + ex = Extraction(text="", blocks=[], people=[ExtractedPerson(name="Chief Executive Officer", title="Jane Doe", role_category="other", is_executive=False)]) | |
| 191 | + P.apply_precision(ex, html_jobs=False) | |
| 192 | + assert ex.people[0].name == "Jane Doe" and ex.people[0].role_category == "ceo" and ex.people[0].is_executive | |
| 193 | + | |
| 194 | + | |
| 195 | +@pytest.mark.parametrize("loc,ok", [ | |
| 196 | + (ExtractedLocation(name="Performance & Analytics", kind="store", city="Allows use of behavioural data to optimise performance"), False), | |
| 197 | + (ExtractedLocation(name="Contact Us", city="Contact Us", country="KR"), False), | |
| 198 | + (ExtractedLocation(name="Careers", country="KR"), False), | |
| 199 | + (ExtractedLocation(name="Sign up for our newsletter", city="Paris", country="FR"), False), | |
| 200 | + (ExtractedLocation(name="Acme Regional Hub"), False), # no evidence of a place | |
| 201 | + (ExtractedLocation(name="Acme Regional Hub", kind="factory"), True), # explicit kind label | |
| 202 | + (ExtractedLocation(name="Berlin"), True), # known city | |
| 203 | + (ExtractedLocation(name="548 Market Street, Suite 200"), True), # street address | |
| 204 | + (ExtractedLocation(name="Oman", city="Hormuz Grand Hotel", country="OM"), True), | |
| 205 | + (ExtractedLocation(name="We are present in twelve countries across three continents.", country="US"), False), | |
| 206 | +]) | |
| 207 | +def test_location_verdict(loc: ExtractedLocation, ok: bool) -> None: | |
| 208 | + assert P.location_verdict(loc).ok is ok | |
| 209 | + | |
| 210 | + | |
| 211 | +def test_normalize_location_repairs_country_as_name() -> None: | |
| 212 | + fixed = P.normalize_location(ExtractedLocation(name="Oman", city="Hormuz Grand Hotel", country="OM")) | |
| 213 | + assert fixed is not None and (fixed.name, fixed.city, fixed.country) == ("Hormuz Grand Hotel", None, "OM") | |
| 214 | + kept = P.normalize_location(ExtractedLocation(name="Oman", city="Muscat", country="OM")) | |
| 215 | + assert kept is not None and (kept.name, kept.city) == ("Oman", "Muscat") # a real city stays a city | |
| 216 | + cleaned = P.normalize_location(ExtractedLocation(name="Dubai Office", city="Level 12", region="Emirates Towers", country="AE")) | |
| 217 | + assert cleaned is not None and cleaned.city is None # "Level 12" is not a city | |
| 218 | + | |
| 219 | + | |
| 220 | +@pytest.mark.parametrize("text,expected", [ | |
| 221 | + ("Stay connected in the U.S. ($13/day) and over 200 international destinations", "$13/day"), | |
| 222 | + ("$99 per user / month, billed annually", "$99 per user / month, billed annually"), | |
| 223 | + ("Starting at €333 / month", "Starting at €333 / month"), | |
| 224 | + ("US$ 1,199 per user / month", "US$ 1,199 per user / month"), | |
| 225 | + ("Talk to sales", "Talk to sales"), | |
| 226 | + ("Free", "Free"), | |
| 227 | + ("Up to 5 users and email support", None), | |
| 228 | +]) | |
| 229 | +def test_price_text_from(text: str, expected: str | None) -> None: | |
| 230 | + assert P.price_text_from(text) == expected | |
| 231 | + | |
| 232 | + | |
| 233 | +@pytest.mark.parametrize("name,ok", [ | |
| 234 | + ("Win your market with Similarweb for businesses", False), ("Worry-free roaming.", False), ("Most popular", False), ("Plans", False), | |
| 235 | + ("Unlock the full potential of your data", False), ("Talk to sales", False), ("Pro", True), ("Business Plus", True), ("Free", True), | |
| 236 | + ("Enterprise", True), ("Team (annual)", True), | |
| 237 | +]) | |
| 238 | +def test_plan_name_ok(name: str, ok: bool) -> None: | |
| 239 | + assert P.plan_name_ok(name) is ok | |
| 240 | + | |
| 241 | + | |
| 242 | +def test_plan_verdict_requires_price_contact_or_free() -> None: | |
| 243 | + assert P.plan_verdict(ExtractedPlan(plan_name="Pro", price=29.0, price_text="$29 per month")).ok | |
| 244 | + assert P.plan_verdict(ExtractedPlan(plan_name="Enterprise", contact_sales=True, price_text="Contact sales")).ok | |
| 245 | + assert P.plan_verdict(ExtractedPlan(plan_name="Free", price=0.0, price_text="Free")).ok | |
| 246 | + assert not P.plan_verdict(ExtractedPlan(plan_name="Pro", price=None, price_text="Everything you need")).ok | |
| 247 | + assert not P.plan_verdict(ExtractedPlan(plan_name="Worry-free roaming.", price=13.0, price_text="n the U.S. ($13/day) and over 200 international destina")).ok | |
| 248 | + | |
| 249 | + | |
| 250 | +@pytest.mark.parametrize("name,ok", [ | |
| 251 | + ("Overview", False), ("Learn more", False), ("All products", False), ("Solutions", False), ("Discover how Atlas helps teams ship faster.", False), | |
| 252 | + ("Atlas Metrics™", True), ("Atlas Logs®", True), ("Microsoft 365", True), ("Discover", True), ("Produits", False), ("製品一覧", False), | |
| 253 | +]) | |
| 254 | +def test_product_verdict(name: str, ok: bool) -> None: | |
| 255 | + assert P.product_verdict(name).ok is ok | |
| 256 | + | |
| 257 | + | |
| 258 | +def test_news_verdict() -> None: | |
| 259 | + assert not P.news_verdict("Read more").ok and not P.news_verdict("Older posts »").ok and not P.news_verdict("Press releases").ok | |
| 260 | + assert not P.news_verdict("3").ok and not P.news_verdict("Page 2").ok and not P.news_verdict("Actualités").ok | |
| 261 | + assert not P.news_verdict("Q2 results").ok | |
| 262 | + assert P.news_verdict("Q2 results", published_at=datetime(2026, 8, 28, tzinfo=UTC)).ok | |
| 263 | + assert P.news_verdict("Q2 results", url="https://x.example/news/2026/08/q2").ok | |
| 264 | + assert P.news_verdict("Acme launches Atlas AI, an assistant for cloud operations").ok | |
| 265 | + | |
| 266 | + | |
| 267 | +# ------------------------------------------------------------------------------------------------------------ pipeline last line of defence | |
| 268 | + | |
| 269 | + | |
| 270 | +def _noisy_extraction() -> Extraction: | |
| 271 | + return Extraction( | |
| 272 | + text="ok", blocks=[], | |
| 273 | + jobs=[ExtractedJob(title="ML Engineer"), ExtractedJob(title="Meer info", url="https://x.example/jobs"), ExtractedJob(title="Sales")], | |
| 274 | + people=[ExtractedPerson(name="Jane Doe", title="CEO"), ExtractedPerson(name="Chair Emeritus", title="Warner Bros. Discovery"), | |
| 275 | + ExtractedPerson(name="Brian Jacobson", title="Contact")], | |
| 276 | + products=[ExtractedProduct(name="Atlas Metrics"), ExtractedProduct(name="Overview")], | |
| 277 | + plans=[ExtractedPlan(plan_name="Pro", price=29.0, price_text="$29 per month"), | |
| 278 | + ExtractedPlan(plan_name="Win your market with Similarweb for businesses", contact_sales=True, price_text="Talk to sales")], | |
| 279 | + locations=[ExtractedLocation(name="Berlin", country="DE"), ExtractedLocation(name="Performance & Analytics", kind="store")], | |
| 280 | + news=[ExtractedNewsItem(title="Quarterly results published", url="https://x.example/a"), ExtractedNewsItem(title="Read more", url="https://x.example/b")], | |
| 281 | + ) | |
| 282 | + | |
| 283 | + | |
| 284 | +def test_pipeline_filter_applies_precision_for_html_connector() -> None: | |
| 285 | + ex = _noisy_extraction() | |
| 286 | + _drop_corrupt_entities(ex, connector_id="generic-html-v1") | |
| 287 | + assert [j.title for j in ex.jobs] == ["ML Engineer"] | |
| 288 | + assert [(p.name, p.title) for p in ex.people] == [("Jane Doe", "CEO"), ("Brian Jacobson", None)] | |
| 289 | + assert [p.name for p in ex.products] == ["Atlas Metrics"] | |
| 290 | + assert [p.plan_name for p in ex.plans] == ["Pro"] | |
| 291 | + assert [loc.name for loc in ex.locations] == ["Berlin"] | |
| 292 | + assert [n.title for n in ex.news] == ["Quarterly results published"] | |
| 293 | + | |
| 294 | + | |
| 295 | +def test_pipeline_filter_trusts_structured_job_boards() -> None: | |
| 296 | + ex = _noisy_extraction() | |
| 297 | + _drop_corrupt_entities(ex, connector_id="greenhouse-v1") | |
| 298 | + assert [j.title for j in ex.jobs] == ["ML Engineer", "Meer info", "Sales"] # ATS jobs are never filtered by title rules | |
| 299 | + assert [p.name for p in ex.products] == ["Atlas Metrics"] # other rules still apply | |
| 300 | ||