extraction precision: multilingual validators for jobs/people/locations/plans/products/news, extractor fixes, fixtures, audit script

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 1 mo ago (Sep 13, 2026) parent cf2a0d2 Browse files

15 changed files +1,723 −97

modified docs/CONNECTORS.md +36 −0

+ Job identity = `job_fingerprint(title, location_text, external_id | url)`; structured boards also give each job a stable block
+ key so the block diff mirrors the job delta.
++## Precision rules (`connectors/_precision.py`, `precision-v1`, 2026-09-13)
++
++Production samples showed the generic HTML connector capturing navigation / call-to-action anchors as jobs (`Meer info`, `Early careers
++bij …`), cookie-consent categories as locations (`Performance & Analytics | Allows use of behavioural data…`), swapped person cards
++(`Chair Emeritus | Warner Bros. Discovery`), marketing headings as plans (`Win your market with … | Talk to sales`) and truncated
++`price_text`. The precision module holds deterministic, multilingual (EN/FR/DE/NL/ES/IT/PT/JA) validators and normalisers that
++`generic_html.py` applies while extracting and `services/pipeline._drop_corrupt_entities` re-applies as the last line of defence for
++every connector (job rules only for `generic-html-v1` — structured boards are trusted). Every rule returns a `Verdict(ok, reason)`, so
++`scripts/audit_extractions.py` can explain rejections on stored rows. Limits are module constants (no magic numbers).
++
++| entity | rejected | required | normalised |
++|---|---|---|---|
++| jobs (HTML) | CTA / nav stoplist (`read more`, `apply now`, `meer info`, `en savoir plus`, `mehr erfahren`, `alle stellen`, `ver más`, `scopri di più`, `詳細を見る` …), category prefixes (`early careers`, `careers at …`, `why …`), single-word titles, titles ending with an ellipsis, prose-like titles (≥ 4 words with function words) without role vocabulary, > 140 chars | role vocabulary (engineer, manager, analyst, nurse, driver, `ingénieur`, `entwickler`, `verpleegkundige`, `ingeniero`, `エンジニア` … or a format marker `(m/w/d)`, `Vollzeit`), **or** a job-like URL (`/job/`, `/jobs/<slug>`, `/careers/x/y`, `/vacature`, `/stelle`, `/offre`, `/emploi`, `/empleo`, `/lavoro`, numeric id, ATS hosts), **or** an explicit location / department cell | `(m/w/d)`, `(h/f)`, `(all genders)`, `- Apply now` and bracketed ids stripped from the display title; the id becomes `external_id` (so it stays in the fingerprint). Bare known cities (`München`) count as a location cell |
++| people | names that are role titles or generic labels (`Contact`, `Team`, `Board of Directors`, `Leadership`, `お問い合わせ`), digits, < 2 or > 5 tokens | name = 2–5 capitalised tokens (particles / initials allowed) or a short CJK name without role words | name ↔ title **swapped** when the card put the title first; link-label titles (`Contact`, `Read more`, `Bio`, `LinkedIn`, `プロフィール`) and bio sentences → `title=None`; `role_category` / `is_executive` recomputed after the swap (JA titles: 代表取締役社長 → ceo, 社外取締役 → board …) |
++| locations | cookie / consent / nav vocabulary (`performance`, `analytics`, `marketing`, `necessary`, `preferences`, `cookies`, `contact us`, `careers`, `privacy`, `newsletter`, `datenschutz`, `voorkeuren` …), names > 80 chars or > 8 words, sentences | at least one of: ISO country, known city (≈ 400 major cities + exonyms, n-gram match), postal code (UK/NL/CA/numeric), street address (EN/FR/DE/NL/ES/IT patterns), explicit kind (`headquarters`, `factory`, `warehouse`, `lab`, `store` … or a kind word in the name) | `city` cleared unless it is a plausible city (`Level 12` is not); `Oman | Hormuz Grand Hotel` → name = venue, country kept; a known city parsed as region (`3089 JH Rotterdam`) is promoted to `city`; city-states keep `city` |
++| pricing plans | names > 40 chars or > 5 words, terminal punctuation, `?`, `|`, eyebrows (`Most popular`, `New`), generic headings (`Plans`, `Pricing`, `Tarifs`, `Preise`), marketing verbs (`win`, `get`, `discover`, `unlock`, `découvrez`, `entdecken` …), plans without a parsed price / `contact_sales` / free tier | — | `price_text` = the price phrase only (`$13/day`, `Starting at €333 / month`, `US$ 1,199 per user / month`, `Talk to sales`, `Free`) — never a cut sentence; the plan name is the first valid label before the price line (eyebrows skipped); a tier literally called `Free` is a valid name |
++| products | nav words (`Overview`, `Products`, `Solutions`, `All products`, `Learn more`, `Produits`, `製品一覧`), > 80 chars / > 10 words, sentences and slogans | — | `®` / `™` kept |
++| news | pagination (`Next`, `Older posts »`, `Page 2`, `3`), category labels (`Press releases`, `Blog posts`, `Actualités`, `ニュース`), `Read more` | ≥ 3 words or ≥ 15 chars **unless** `published_at` or a `/YYYY/MM/` URL confirms an article | the title is the first non-date line of the card |
++
++Normalizer changes made for these rules (`sdk/normalize.py`): a `.job-list`-style wrapper whose only child is a linked / card-like
++`<ul>` or `<table>` is a container (its rows become `job_listing` blocks, inheriting the wrapper's kind) while a feature `<ul>` inside a
++pricing card is not; card text has one line per direct child so inline `<a>title</a><span>city</span>` cells do not merge. Stripping
++gender markers changes the display title (and therefore the fingerprint) of affected HTML jobs once — expect a one-off
++removed + added pair for `(m/w/d)`-style listings on the first crawl after deployment.
++
++Fixtures reproducing the production noise: `generic_html/careers_cta_noise.html` (NL careers page: CTA anchors, single word, ellipsis
+++ 5 real listings incl. `(m/v/x)`, `[12933]`, `- Apply now`), `careers_de.html` (German `(m/w/d)` listings + `Alle Stellen` / `Mehr
++erfahren`), `leadership_swapped.html` (title-first card, `Contact` link, `Chair Emeritus | Warner Bros. Discovery`, `Board of Directors`),
++`leadership_ja.html` (Japanese officers page), `locations_cookie_noise.html` (consent categories inside a locations page, `Contact Us | KR`,
++`Oman | Hormuz Grand Hotel`, Dutch address), `pricing_marketing_noise.html` (marketing hero + `Talk to sales`, `Worry-free roaming.` +
++prose price, eyebrow, real Free / Starter / Team / Business / Enterprise), `products_news_noise.html`. Tests: `tests/test_extraction_precision.py`
++(noise gone, every real item still extracted). Audit / purge of stored rows: `scripts/audit_extractions.py` — samples HTML-sourced rows per
++table, prints a precision estimate (Wilson 95 % interval) and the rejections by reason; `--all-sources` widens the read-only audit to
++enrichment rows (Wikidata people); `--purge` deletes failing **HTML-sourced** rows of people / products / pricing_plans / locations / jobs only
++(never ATS jobs, enrichment rows or news). Run it before and after deploying to measure the effect on production.
++
+ ## Adding a connector family
+. Create `src/companyatlas/connectors/<family>.py`; subclass `Connector` (or `AtsConnector` for JSON/XML boards — implement

added fixtures/connectors/generic_html/careers_cta_noise.html +30 −0

++<!doctype html>
++<html lang="nl">
++<head><title>Jobs bij Telenet group</title></head>
++<body>
++<nav class="navbar"><a href="/">Home</a><a href="/nl/jobs">Jobs</a><a href="/nl/over-ons">Over ons</a></nav>
++<main>
++  <section class="hero"><h1>Werken bij Telenet group</h1><p>Ontdek onze openstaande vacatures.</p></section>
++  <section class="job-teasers">
++    <h2>Ontdek</h2>
++    <div class="job-card"><a href="/nl/jobs/starters"><h3>Voor starters die willen gáán</h3></a><p>Start je carrière bij ons.</p></div>
++    <div class="job-card"><a href="/nl/jobs/early-careers"><h3>Early careers bij Telenet group</h3></a><p>Stages en graduate programma's.</p></div>
++    <div class="job-card"><a href="/nl/jobs/why-us"><h3>Waarom werken bij ons?</h3></a><a href="/nl/jobs/why-us">Meer info</a></div>
++    <div class="job-card"><a href="/nl/jobs/search"><h3>Alle vacatures</h3></a><a href="/nl/jobs/search">Bekijk alle vacatures →</a></div>
++  </section>
++  <section class="job-list">
++    <h2>Openstaande vacatures</h2>
++    <ul>
++      <li class="job-item"><a href="/nl/jobs/12841-senior-backend-software-engineer">Senior Backend Software Engineer - Infrastructure</a><span class="location">London, United Kingdom</span><span class="dept">Engineering</span></li>
++      <li class="job-item"><a href="/nl/jobs/12902-verpleegkundige">Verpleegkundige spoedgevallen (m/v/x)</a><span class="location">Gent, België</span><span class="dept">Zorg</span></li>
++      <li class="job-item"><a href="/nl/jobs/12933-data-analyst">Data Analyst (m/w/d) [12933]</a><span class="location">Berlin, Deutschland</span><span class="dept">Data</span></li>
++      <li class="job-item"><a href="/nl/jobs/12950-accountmanager">Accountmanager KMO - Apply now</a><span class="location">Mechelen, België</span><span class="dept">Sales</span></li>
++      <li class="job-item"><a href="/nl/jobs/12977-technicien">Technicien réseau (h/f)</a><span class="location">Bruxelles, Belgique</span><span class="dept">Network</span></li>
++      <li class="job-item"><a href="/nl/jobs/12990-sales">Sales</a><span class="location"></span></li>
++      <li class="job-item"><a href="/nl/jobs/search?page=2">Meer info</a><span class="location"></span></li>
++      <li class="job-item"><a href="/nl/jobs/13001">Wij zoeken mensen die het verschil willen maken voor onze klanten…</a><span class="location">Mechelen</span></li>
++    </ul>
++  </section>
++</main>
++<footer><p>© 2026 Telenet group</p></footer>
++</body></html>

added fixtures/connectors/generic_html/careers_de.html +26 −0

++<!doctype html>
++<html lang="de">
++<head><title>Karriere bei Muster AG</title></head>
++<body>
++<nav class="navbar"><a href="/">Start</a><a href="/karriere">Karriere</a><a href="/unternehmen">Unternehmen</a></nav>
++<main>
++  <section class="hero"><h1>Karriere bei Muster AG</h1><p>Gestalten Sie mit uns die Zukunft.</p></section>
++  <section class="job-teasers">
++    <div class="job-card"><a href="/karriere/stellen"><h3>Alle Stellen</h3></a><a href="/karriere/stellen">Mehr erfahren</a></div>
++    <div class="job-card"><a href="/karriere/warum-wir"><h3>Warum Muster AG?</h3></a><a href="/karriere/warum-wir">Mehr erfahren →</a></div>
++    <div class="job-card"><a href="/karriere/bewerbung"><h3>Jetzt bewerben</h3></a></div>
++  </section>
++  <section class="job-list">
++    <h2>Offene Stellen</h2>
++    <ul>
++      <li class="job-item"><a href="/karriere/stellen/4471-softwareentwickler">Softwareentwickler Backend (m/w/d)</a><span>München</span><span>IT</span></li>
++      <li class="job-item"><a href="/karriere/stellen/4488-projektleiter">Projektleiter Anlagenbau (m/w/d)</a><span>Stuttgart</span><span>Engineering</span></li>
++      <li class="job-item"><a href="/karriere/stellen/4490-ausbildung">Ausbildung zum Industriemechaniker (m/w/d) 2027</a><span>Hamburg</span><span>Ausbildung</span></li>
++      <li class="job-item"><a href="/karriere/stellen/4501-werkstudent">Werkstudent Marketing (all genders)</a><span>Berlin</span><span>Marketing</span></li>
++      <li class="job-item"><a href="/karriere/stellen/4510-pflege">Pflegefachkraft Intensivstation (w/m/d)</a><span>Köln</span><span>Pflege</span></li>
++      <li class="job-item"><a href="/karriere/stellen">Weiterlesen</a><span></span></li>
++    </ul>
++  </section>
++</main>
++<footer><p>© 2026 Muster AG</p></footer>
++</body></html>

added fixtures/connectors/generic_html/leadership_ja.html +19 −0

++<!doctype html>
++<html lang="ja">
++<head><title>役員紹介 | 株式会社サンプル</title></head>
++<body>
++<nav class="navbar"><a href="/">ホーム</a><a href="/company/officers">役員紹介</a><a href="/contact">お問い合わせ</a></nav>
++<main id="main">
++  <h1>役員紹介</h1>
++  <section class="team-grid">
++    <h2>取締役</h2>
++    <div class="team-member"><h3>山田 太郎</h3><p class="role">代表取締役社長</p></div>
++    <div class="team-member"><h3>佐藤 花子</h3><p class="role">取締役 最高財務責任者（CFO）</p></div>
++    <div class="team-member"><h3>鈴木 一郎</h3><p class="role">社外取締役</p></div>
++    <div class="team-member"><h3>高橋 美咲</h3><p class="role">執行役員 技術本部長</p><a href="/company/officers/takahashi">プロフィール</a></div>
++    <div class="team-member"><h3>お問い合わせ</h3><p class="role">詳細を見る</p></div>
++    <div class="team-member"><h3>株式会社サンプル</h3><p class="role">会社概要</p></div>
++  </section>
++</main>
++<footer><p>© 2026 株式会社サンプル</p></footer>
++</body></html>

added fixtures/connectors/generic_html/leadership_swapped.html +22 −0

++<!doctype html>
++<html lang="en">
++<head><title>Leadership | Warner Example</title></head>
++<body>
++<nav class="navbar"><a href="/">Home</a><a href="/leadership">Leadership</a><a href="/contact">Contact</a></nav>
++<main id="main">
++  <h1>Our leadership</h1>
++  <section class="team-grid">
++    <h2>Executive team</h2>
++    <div class="team-member"><p class="role">Chief Executive Officer</p><h3>Jane Doe</h3><p>Jane leads the company since 2019.</p></div>
++    <div class="team-member"><h3>Brian Jacobson</h3><p class="role">Chief Financial Officer</p><a href="/contact/brian">Contact</a></div>
++    <div class="team-member"><h3>Priya Natarajan</h3><a href="/contact/priya">Contact</a></div>
++    <div class="team-member"><h3>Chair Emeritus</h3><p class="role">Warner Bros. Discovery</p></div>
++    <div class="team-member"><h3>Board of Directors</h3><p class="role">Meet the people who govern the company</p></div>
++    <div class="team-member"><h3>Leadership</h3><p class="role">Read more</p></div>
++    <div class="team-member"><h3>Contact</h3><p class="role">Media relations</p></div>
++    <div class="team-member"><h3>Marc van der Berg</h3><p class="role">Chief Operating Officer</p><a href="/bio/marc">Read bio</a></div>
++    <div class="team-member"><p class="role">Chair of the Board</p><h3>Samuel Adebayo</h3></div>
++  </section>
++</main>
++<footer><p>© 2026 Warner Example</p></footer>
++</body></html>

added fixtures/connectors/generic_html/locations_cookie_noise.html +30 −0

++<!doctype html>
++<html lang="en">
++<head><title>Our locations — Hormuz Example Group</title></head>
++<body>
++<header><nav><a href="/">Home</a><a href="/locations">Locations</a><a href="/contact-us">Contact Us</a></nav></header>
++<div class="preferences-modal" role="dialog">
++  <h2>Manage your preferences</h2>
++  <table class="preferences-table">
++    <tr class="preference location"><td>Strictly Necessary</td><td>These are required for the website to function and cannot be switched off.</td></tr>
++    <tr class="preference location"><td>Performance &amp; Analytics</td><td>Allows use of behavioural data to optimise performance. Third-party providers store aggregated data.</td></tr>
++    <tr class="preference location"><td>Functional</td><td>Enables enhanced functionality such as videos and live chat. Providers may store your settings.</td></tr>
++    <tr class="preference location"><td>Marketing</td><td>Used by advertising partners to build a profile of your interests and show relevant ads on other sites. Data is stored by the store.</td></tr>
++  </table>
++</div>
++<main>
++  <h1>Where to find us</h1>
++  <section class="offices">
++    <h2>Offices</h2>
++    <div class="office-card"><h3>Oman</h3><p>Hormuz Grand Hotel</p><p>Oman</p></div>
++    <div class="office-card"><h3>Contact Us</h3><p>Contact Us</p><p>Seoul, KR</p></div>
++    <div class="office-card"><h3>Dubai Office</h3><p>Level 12, Emirates Towers<br>Dubai, United Arab Emirates</p></div>
++    <div class="office-card"><h3>London</h3><p>1 Finsbury Avenue, London EC2M 2PF, United Kingdom</p></div>
++    <div class="office-card"><h3>Singapore</h3><p>Singapore</p></div>
++    <div class="office-card"><h3>Rotterdam Warehouse</h3><p>Waalhaven Zuidzijde 21, 3089 JH Rotterdam, Nederland</p></div>
++    <div class="office-card"><h3>Careers</h3><p>Join our team in one of our offices around the world.</p><p>Seoul, KR</p></div>
++    <div class="office-card"><h3>Sign up for our newsletter</h3><p>Receive updates from our offices around the world.</p><p>Paris, France</p></div>
++  </section>
++</main>
++<footer><p>© 2026 Hormuz Example Group</p></footer>
++</body></html>

added fixtures/connectors/generic_html/pricing_marketing_noise.html +56 −0

++<!doctype html>
++<html lang="en">
++<head><title>Pricing — Similar Example</title></head>
++<body>
++<header class="site-header"><nav><a href="/">Home</a><a href="/pricing">Pricing</a></nav></header>
++<main>
++  <section class="hero pricing-card">
++    <h1>Win your market with Similar Example for businesses</h1>
++    <p>Get the insights you need to outperform the competition.</p>
++    <a href="/contact-sales">Talk to sales</a>
++  </section>
++  <section class="pricing-grid">
++    <h2>Plans</h2>
++    <div class="pricing-card">
++      <h3>Free</h3>
++      <p class="price">$0 forever</p>
++      <ul><li>1 user</li><li>Basic reports</li></ul>
++    </div>
++    <div class="pricing-card">
++      <h3>Starter</h3>
++      <p class="price">$125 per month, billed annually</p>
++      <ul><li>1 user</li><li>3 months of data</li></ul>
++      <a href="/signup?plan=starter">Start now</a>
++    </div>
++    <div class="pricing-card">
++      <h3>Team</h3>
++      <p class="price">Starting at €333 / month</p>
++      <ul><li>3 users</li><li>15 months of data</li></ul>
++    </div>
++    <div class="pricing-card">
++      <p class="eyebrow">Most popular</p>
++      <h3>Business</h3>
++      <p class="price">US$ 1,199 per user / month</p>
++      <ul><li>5 users</li><li>Custom data</li></ul>
++    </div>
++    <div class="pricing-card">
++      <h3>Enterprise</h3>
++      <p class="price">Talk to sales</p>
++      <ul><li>Unlimited users</li><li>Dedicated success manager</li></ul>
++    </div>
++    <div class="pricing-card">
++      <h3>Worry-free roaming.</h3>
++      <p>Stay connected in the U.S. ($13/day) and over 200 international destinations with our roaming add-on for travellers.</p>
++    </div>
++    <div class="pricing-card">
++      <h3>Unlock the full potential of your data</h3>
++      <p class="price">$49 per month</p>
++    </div>
++    <div class="pricing-card">
++      <h3>Add-on: extra seats</h3>
++      <p>Bring the whole team along, more information on request.</p>
++    </div>
++  </section>
++</main>
++<footer class="site-footer"><p>© 2026 Similar Example</p></footer>
++</body></html>

added fixtures/connectors/generic_html/products_news_noise.html +29 −0

++<!doctype html>
++<html lang="en">
++<head><title>Products &amp; News — Acme Example</title></head>
++<body>
++<nav class="navbar"><a href="/">Home</a><a href="/products">Products</a><a href="/news">News</a></nav>
++<main>
++  <section class="products">
++    <h2>Products</h2>
++    <div class="product-card"><h3>Overview</h3><p>Everything we make, in one place.</p><a href="/products">All products</a></div>
++    <div class="product-card"><h3>Atlas Metrics™</h3><p>High-cardinality metrics at any scale.</p><a href="/products/metrics">Learn more</a></div>
++    <div class="product-card"><h3>Atlas Logs®</h3><p>Search petabytes in seconds.</p><a href="/products/logs">Learn more</a></div>
++    <div class="product-card"><h3>Discover how Atlas helps teams ship faster.</h3><p>Slogan card</p><a href="/why">Learn more</a></div>
++    <div class="product-card"><h3>Learn more</h3><a href="/products/more">Learn more</a></div>
++    <div class="product-card"><h3>Solutions</h3><a href="/solutions">Solutions</a></div>
++    <div class="product-card"><h3>Atlas Traces</h3><p>Distributed tracing without sampling.</p><a href="/products/traces">Learn more</a></div>
++  </section>
++  <section class="news-list">
++    <h2>Latest news</h2>
++    <article class="news-item"><time datetime="2026-09-10">September 10, 2026</time><h3><a href="/news/2026/09/acme-launches-atlas-ai">Acme launches Atlas AI, an assistant for cloud operations</a></h3></article>
++    <article class="news-item"><time datetime="2026-08-28">August 28, 2026</time><h3><a href="/news/2026/08/q2-results">Q2 results</a></h3></article>
++    <article class="news-item"><h3><a href="/news/press-releases">Press releases</a></h3></article>
++    <article class="news-item"><h3><a href="/news/category/blog">Blog posts</a></h3></article>
++    <article class="news-item"><h3><a href="/news/2026/07/acme-partners-with-bigco">Acme and BigCo announce strategic partnership</a></h3></article>
++    <article class="news-item"><h3><a href="/news?page=2">Older posts »</a></h3></article>
++    <article class="news-item"><h3><a href="/news/all">Read more news</a></h3></article>
++  </section>
++</main>
++<footer><p>© 2026 Acme Example</p></footer>
++</body></html>

added scripts/audit_extractions.py +242 −0

++"""Audit stored typed entities against the precision rules (`connectors/_precision`) — read-only by default.
++
++For each table it samples N rows (seeded, reproducible), applies the same validators the connector and the pipeline now apply, and
++prints a precision estimate (share of rows the rules accept, with a 95 % interval) plus the rows that would now be rejected, grouped
++by reason. Run it before and after a deploy to measure the effect on production.
++
++    DATABASE_URL=… .venv/bin/python scripts/audit_extractions.py                     # sample 200 rows per table
++    .venv/bin/python scripts/audit_extractions.py --sample 500 --table people,locations --show 40
++    .venv/bin/python scripts/audit_extractions.py --purge                            # delete rows failing the rules (see below)
++
++Scope: by default only rows attached to a `generic-html-v1` sensor are audited (the rows the rules were written for); `--all-sources`
++widens the read-only audit to enrichment-sourced rows (e.g. Wikidata people) as a recall check on the validators.
++`--purge` scans the *whole* table (not a sample) and deletes HTML-sourced rows that fail the validators, for `people`, `products`,
++`pricing_plans`, `locations` and `jobs` (`sensors.connector_id = 'generic-html-v1'`); it never touches ATS jobs, enrichment rows or `news_items`.
++People whose only defect is a stoplisted title ("Contact", "Read more") are kept with `title = null`. Rows the rules would *rename*
++(swapped person cards, country-as-name locations) are reported, not rewritten: the next crawl re-extracts them correctly and the
++reconciliation marks the old row `no_longer_listed`. Events derived from purged rows are history and stay untouched.
++"""
++from __future__ import annotations
++
++import argparse
++import asyncio
++import math
++from collections import Counter, defaultdict
++from collections.abc import Callable, Sequence
++from dataclasses import dataclass, field
++from typing import Any
++
++from companyatlas.connectors import _precision as P
++from companyatlas.db import dispose, execute, fetch_all, transaction
++from companyatlas.sdk.models import ExtractedLocation, ExtractedPlan
++
++DEFAULT_SAMPLE = 200
++DEFAULT_SHOW = 25
++DEFAULT_SEED = 0.42
++Z_95 = 1.96
++LABEL_WIDTH = 70
++HTML_CONNECTOR = "generic-html-v1"
++PURGE_BATCH = 500
++
++
++@dataclass(slots=True)
++class Outcome:
++    """What the rules say about one stored row."""
++    status: str                      # accept | reject | normalise
++    reason: str | None = None
++    fix: str | None = None           # normalise: what would change (for the report) — "title → null" is applied under --purge
++
++
++@dataclass(slots=True)
++class TableSpec:
++    name: str
++    label_col: str
++    select_sql: str                  # must select id + the columns `judge` reads; `{where}` and `{limit}` are filled in
++    judge: Callable[[dict[str, Any]], Outcome]
++    purgeable: bool
++    label: Callable[[dict[str, Any]], str]
++    summary_cols: Sequence[str] = field(default_factory=tuple)
++
++
++# ------------------------------------------------------------------------------------------------------------ judges (one per table)
++
++
++def _judge_person(row: dict[str, Any]) -> Outcome:
++    fixed = P.normalize_person(row["name"] or "", row.get("title"))
++    if fixed is None:
++        return Outcome("reject", P.person_verdict(row["name"] or "", row.get("title")).reason)
++    name, title = fixed
++    if name != (row["name"] or "").strip():
++        return Outcome("normalise", "name/title swapped", fix=f"name → {name!r}, title → {title!r}")
++    if (title or None) != (row.get("title") or None):
++        return Outcome("normalise", "title is a link label", fix="title → null")
++    return Outcome("accept")
++
++
++def _judge_product(row: dict[str, Any]) -> Outcome:
++    v = P.product_verdict(row["name"] or "")
++    return Outcome("accept") if v.ok else Outcome("reject", v.reason)
++
++
++def _judge_plan(row: dict[str, Any]) -> Outcome:
++    plan = ExtractedPlan(plan_name=row["plan_name"] or "", price=float(row["price"]) if row.get("price") is not None else None,
++                         price_text=row.get("price_text"), contact_sales=bool(row.get("contact_sales")))
++    v = P.plan_verdict(plan)
++    return Outcome("accept") if v.ok else Outcome("reject", v.reason)
++
++
++def _judge_location(row: dict[str, Any]) -> Outcome:
++    loc = ExtractedLocation(name=row["name"] or "", kind=row.get("kind") or "office", city=row.get("city"), region=row.get("region"),
++                            country=row.get("country"), address_text=None)
++    v = P.location_verdict(loc)
++    if not v.ok:
++        return Outcome("reject", v.reason)
++    fixed = P.normalize_location(loc)
++    if fixed is not None and fixed.name != loc.name:
++        return Outcome("normalise", "country used as the name", fix=f"name → {fixed.name!r}, city → {fixed.city!r}")
++    if fixed is not None and fixed.city != loc.city:
++        return Outcome("normalise", "city cell is not a city", fix=f"city → {fixed.city!r}")
++    return Outcome("accept")
++
++
++def _judge_job(row: dict[str, Any]) -> Outcome:
++    v = P.job_verdict(row["title"] or "", url=row.get("url"), location=row.get("location_text"), department=row.get("department"))
++    if not v.ok:
++        return Outcome("reject", v.reason)
++    clean, _ = P.clean_job_title(row["title"] or "")
++    if clean and clean != (row["title"] or "").strip():
++        return Outcome("normalise", "display title carries a marker / id", fix=f"title → {clean!r}")
++    return Outcome("accept")
++
++
++def _judge_news(row: dict[str, Any]) -> Outcome:
++    v = P.news_verdict(row["title"] or "", url=row.get("url"), published_at=row.get("published_at"))
++    return Outcome("accept") if v.ok else Outcome("reject", v.reason)
++
++
++def _lbl(*parts: Any) -> str:
++    return " | ".join(str(p) for p in parts if p not in (None, ""))
++
++
++# Every select joins `sensors` so rows can be scoped to the HTML connector (`{where}`): enrichment-sourced rows (Wikidata people…)
++# have no HTML sensor and are only ever *audited* with --all-sources, never purged.
++TABLES: dict[str, TableSpec] = {
++    "people": TableSpec("people", "name", "select t.id, t.name, t.title from people t left join sensors s on s.id = t.sensor_id {where} {limit}",
++                        _judge_person, True, lambda r: _lbl(r["name"], r.get("title"))),
++    "products": TableSpec("products", "name", "select t.id, t.name from products t left join sensors s on s.id = t.sensor_id {where} {limit}",
++                          _judge_product, True, lambda r: _lbl(r["name"])),
++    "pricing_plans": TableSpec("pricing_plans", "plan_name",
++                               "select t.id, t.plan_name, t.price, t.price_text, t.contact_sales from pricing_plans t left join sensors s on s.id = t.sensor_id "
++                               "{where} {limit}", _judge_plan, True, lambda r: _lbl(r["plan_name"], r.get("price_text"))),
++    "locations": TableSpec("locations", "name",
++                           "select t.id, t.name, t.kind, t.city, t.region, t.country from locations t left join sensors s on s.id = t.sensor_id {where} {limit}",
++                           _judge_location, True, lambda r: _lbl(r["name"], r.get("city"), r.get("country"), r.get("kind"))),
++    "jobs": TableSpec("jobs", "title",
++                      "select t.id, t.title, t.url, t.location_text, t.department from jobs t left join sensors s on s.id = t.sensor_id {where} {limit}",
++                      _judge_job, True, lambda r: _lbl(r["title"], r.get("location_text"))),
++    "news_items": TableSpec("news_items", "title",
++                            "select t.id, t.title, t.url, t.published_at from news_items t left join sensors s on s.id = t.sensor_id {where} {limit}",
++                            _judge_news, False, lambda r: _lbl(r["title"], r.get("url"))),
++}
++PURGEABLE = [t for t, spec in TABLES.items() if spec.purgeable]
++HTML_ONLY_WHERE = f"where s.connector_id = '{HTML_CONNECTOR}'"
++
++
++# ------------------------------------------------------------------------------------------------------------ reporting
++
++
++def _interval(k: int, n: int) -> tuple[float, float]:
++    """Wilson 95 % interval for a proportion k/n."""
++    if n == 0:
++        return 0.0, 0.0
++    p = k / n
++    denom = 1 + Z_95 ** 2 / n
++    centre = (p + Z_95 ** 2 / (2 * n)) / denom
++    half = Z_95 * math.sqrt(p * (1 - p) / n + Z_95 ** 2 / (4 * n * n)) / denom
++    return max(0.0, centre - half), min(1.0, centre + half)
++
++
++def _sql(spec: TableSpec, *, sample: int | None, html_only: bool) -> str:
++    where = HTML_ONLY_WHERE if (html_only or spec.name == "jobs") else ""     # jobs: ATS boards are always out of scope
++    limit = f"order by random() limit {int(sample)}" if sample else ""
++    return spec.select_sql.format(where=where, limit=limit)
++
++
++async def _audit_table(conn: Any, spec: TableSpec, *, sample: int | None, show: int, html_only: bool) -> tuple[list[str], list[str]]:
++    rows = await fetch_all(conn, _sql(spec, sample=sample, html_only=html_only))
++    outcomes = [(r, spec.judge(r)) for r in rows]
++    accepted = sum(1 for _, o in outcomes if o.status == "accept")
++    normalised = [(r, o) for r, o in outcomes if o.status == "normalise"]
++    rejected = [(r, o) for r, o in outcomes if o.status == "reject"]
++    n = len(rows)
++    lo, hi = _interval(accepted + len(normalised), n)
++    print(f"\n== {spec.name}: sampled {n} · accepted {accepted} · normalised {len(normalised)} · rejected {len(rejected)}")
++    if n:
++        print(f"   precision estimate (accepted + normalisable) = {(accepted + len(normalised)) / n:.1%}  [{lo:.1%}, {hi:.1%}]"
++              f"   rejected share = {len(rejected) / n:.1%}")
++    reasons: Counter[str] = Counter(o.reason or "?" for _, o in rejected)
++    by_reason: dict[str, list[str]] = defaultdict(list)
++    for r, o in rejected:
++        by_reason[o.reason or "?"].append(spec.label(r))
++    for reason, count in reasons.most_common():
++        print(f"   - {reason}: {count}")
++        for lbl in by_reason[reason][:show]:
++            print(f"       ✗ {lbl[:LABEL_WIDTH]}")
++    if normalised:
++        print(f"   ~ would be normalised ({len(normalised)}):")
++        for r, o in normalised[:show]:
++            print(f"       ~ {spec.label(r)[:LABEL_WIDTH]}  →  {o.fix}")
++    return [r["id"] for r, _ in rejected], [r["id"] for r, o in normalised if o.fix == "title → null"]
++
++
++async def _purge(conn: Any, spec: TableSpec, reject_ids: list[str], null_title_ids: list[str]) -> None:
++    for i in range(0, len(reject_ids), PURGE_BATCH):
++        await execute(conn, f"delete from {spec.name} where id = any(cast(:ids as text[]))", ids=reject_ids[i:i + PURGE_BATCH])
++    if spec.name == "people":
++        for i in range(0, len(null_title_ids), PURGE_BATCH):
++            await execute(conn, "update people set title = null where id = any(cast(:ids as text[]))", ids=null_title_ids[i:i + PURGE_BATCH])
++    print(f"   purged {len(reject_ids)} row(s) from {spec.name}" + (f", cleared {len(null_title_ids)} title(s)" if spec.name == "people" else ""))
++
++
++async def main(args: argparse.Namespace) -> None:
++    names = [t.strip() for t in args.table.split(",")] if args.table else list(TABLES)
++    unknown = [t for t in names if t not in TABLES]
++    if unknown:
++        raise SystemExit(f"unknown table(s): {unknown}; choose from {list(TABLES)}")
++    if args.purge:
++        names = [t for t in names if TABLES[t].purgeable]
++        print(f"PURGE mode: full scan of {names}; rows failing the precision rules will be deleted (news and ATS jobs are never touched)")
++    sample = None if args.purge else args.sample
++    html_only = args.purge or not args.all_sources                                 # a purge is always scoped to HTML-sourced rows
++    print(f"precision rules {P.PRECISION_VERSION} · sample={'all' if sample is None else sample} · seed={args.seed} · "
++          f"scope={'HTML-sourced rows (generic-html-v1)' if html_only else 'all sources'}")
++    async with transaction() as conn:
++        await execute(conn, "select setseed(:s)", s=args.seed)
++        for name in names:
++            spec = TABLES[name]
++            reject_ids, null_title_ids = await _audit_table(conn, spec, sample=sample, show=args.show, html_only=html_only)
++            if args.purge:
++                await _purge(conn, spec, reject_ids, null_title_ids)
++    print("\ndone" if args.purge else "\nread-only audit done (pass --purge to delete the rejected rows of the purgeable tables)")
++
++
++def _parse() -> argparse.Namespace:
++    ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
++    ap.add_argument("--sample", type=int, default=DEFAULT_SAMPLE, help=f"rows sampled per table (default {DEFAULT_SAMPLE})")
++    ap.add_argument("--table", type=str, default="", help=f"comma-separated subset of {list(TABLES)}")
++    ap.add_argument("--show", type=int, default=DEFAULT_SHOW, help="examples printed per reason")
++    ap.add_argument("--seed", type=float, default=DEFAULT_SEED, help="Postgres setseed() value in [-1, 1] for reproducible samples")
++    ap.add_argument("--purge", action="store_true", help=f"delete rows failing the rules in {PURGEABLE} (full scan, not a sample, HTML-sourced only)")
++    ap.add_argument("--all-sources", action="store_true", help="read-only audit over every source (enrichment rows included); ignored with --purge")
++    return ap.parse_args()
++
++
++async def _run() -> None:
++    try:
++        await main(_parse())
++    finally:
++        await dispose()
++
++
++if __name__ == "__main__":
++    asyncio.run(_run())

added src/companyatlas/connectors/_precision.py +764 −0

++"""Precision rules for typed extractions (spec §107, "never fabricate"): deterministic validators and normalisers that reject the
++navigation / call-to-action / cookie-consent / marketing noise the generic HTML connector picks up on real corporate pages, and
++repair the classic confusions (person name ↔ title swapped, country used as a location name, truncated price text).
++
++Shared by `connectors/generic_html.py` (applied while extracting) and `services/pipeline._drop_corrupt_entities` (last line of
++defence for every connector) and by `scripts/audit_extractions.py` (measurement / purge on stored rows). Every rule is a small pure
++function returning a `Verdict`, so the same code explains *why* a row is rejected. Multilingual where cheap (EN/FR/DE/NL/ES/IT/PT/JA).
++No network, no LLM.
++"""
++from __future__ import annotations
++
++import re
++import unicodedata
++from collections.abc import Iterable
++from dataclasses import dataclass
++from datetime import datetime
++from typing import Any
++
++from companyatlas.connectors._util import POSTAL_RE, country_code, date_from_url, parse_location
++from companyatlas.sdk.models import (
++    ExtractedJob,
++    ExtractedLocation,
++    ExtractedNewsItem,
++    ExtractedPerson,
++    ExtractedPlan,
++    ExtractedProduct,
++    Extraction,
++)
++from companyatlas.sdk.normalize import normalize_whitespace
++
++PRECISION_VERSION = "precision-v1"
++
++# ------------------------------------------------------------------------------------------------------------ limits (no magic numbers)
++
++MAX_JOB_TITLE_CHARS = 140
++MIN_JOB_TITLE_CHARS = 4
++MIN_JOB_TITLE_WORDS = 2
++PROSE_MIN_WORDS = 4                    # a "title" with ≥ 4 words and function words reads like a sentence
++PERSON_NAME_MIN_TOKENS, PERSON_NAME_MAX_TOKENS = 2, 5
++MIN_PERSON_NAME_CHARS, MAX_PERSON_NAME_CHARS = 4, 60
++MAX_PERSON_TITLE_CHARS = 100
++SENTENCE_TITLE_MIN_WORDS = 6           # a person "title" of ≥ 6 words ending with a full stop is a bio sentence
++MAX_LOCATION_NAME_CHARS = 80
++MAX_LOCATION_NAME_WORDS = 8
++MAX_CITY_CHARS, MAX_CITY_WORDS = 40, 4
++VENUE_MIN_WORDS = 3                    # "Hormuz Grand Hotel": ≥ 3 words in the city slot is a venue, not a city
++MAX_PLAN_NAME_CHARS, MAX_PLAN_NAME_WORDS = 40, 5
++MAX_PRICE_TEXT_CHARS = 60
++MAX_PRODUCT_NAME_CHARS, MAX_PRODUCT_NAME_WORDS = 80, 10
++NEWS_MIN_WORDS, NEWS_MIN_CHARS = 3, 15
++NEWS_MIN_CJK_CHARS = 6                 # CJK titles have no word boundaries: 6 characters already carry a headline
++MAX_DEPARTMENT_CHARS, MAX_DEPARTMENT_WORDS = 40, 4
++CITY_NGRAM_MAX = 3
++YEAR_RE = re.compile(r"^(?:19|20)\d{2}$")
++
++
++@dataclass(slots=True, frozen=True)
++class Verdict:
++    ok: bool
++    reason: str | None = None
++
++    @staticmethod
++    def accept() -> Verdict:
++        return Verdict(True, None)
++
++    @staticmethod
++    def reject(reason: str) -> Verdict:
++        return Verdict(False, reason)
++
++
++# ------------------------------------------------------------------------------------------------------------ text helpers
++
++_WORD_RE = re.compile(r"[^\W_]+", re.UNICODE)
++_CJK_RE = re.compile(r"[぀-ヿ㐀-鿿]")
++TERMINAL_PUNCT = ".!?:;,"
++TRAILING_MARKS_RE = re.compile(r"[\s→➔➡›»>\-–—|·•]+$")   # arrows, chevrons, dashes, bullets
++ELLIPSIS_RE = re.compile(r"(?:\.\.\.|…)\s*$")
++PROSE_FUNCTION_WORDS_RE = re.compile(
++    r"\b(?:who|that|which|want|wants|with|for|your|you|we|our|are|is|the|die|dat|willen|voor|met|jouw|onze|wij|zijn|qui|que|pour|avec|votre|nous|"
++    r"sont|est|les|der|das|mit|für|und|ihre|wir|sind|para|con|nuestro|somos|per|che|nostro|siamo|com|nosso|somos)\b", re.IGNORECASE)
++SENTENCE_VERB_RE = re.compile(
++    r"\b(?:allows?|enables?|helps?|uses?|provides?|lets?|makes?|gives?|ensures?|collects?|stores?|can|will|would|should|permet|permettent|utilise|"
++    r"erlaubt|ermöglicht|verwendet|gebruikt|permite|consente|permitem)\b", re.IGNORECASE)
++MARKETING_VERB_RE = re.compile(
++    r"^(?:win|get|start|discover|unlock|boost|grow|try|choose|compare|see|learn|find|build|join|save|upgrade|buy|sign up|request|book|schedule|"
++    r"contact|talk|let'?s|ready|why|how|what|explore|meet|introducing|welcome|enjoy|make|take|achieve|transform|power|scale|drive|accelerate|"
++    r"supercharge|level up|bring|stay|experience|gagnez|obtenez|découvrez|commencez|essayez|choisissez|comparez|rejoignez|débloquez|profitez|"
++    r"gewinnen|holen|starten|entdecken|testen|wählen|vergleichen|jetzt|ontdek|start|kies|vergelijk|probeer|gana|obtén|empieza|descubre|prueba|"
++    r"elige|compara|scopri|inizia|prova|scegli|confronta|ottieni|ganhe|obtenha|comece|descubra|experimente|escolha)\b", re.IGNORECASE)
++
++
++def word_count(text: str) -> int:
++    return len(_WORD_RE.findall(text))
++
++
++def key_of(text: str | None) -> str:
++    """Accent-insensitive, lower-case, single-spaced key (keeps CJK) used for stoplists and the city table."""
++    if not text:
++        return ""
++    nfkd = unicodedata.normalize("NFKD", text)
++    stripped = "".join(ch for ch in nfkd if not unicodedata.combining(ch))
++    return " ".join(_WORD_RE.findall(stripped.lower()))
++
++
++def strip_trailing_marks(text: str) -> str:
++    return TRAILING_MARKS_RE.sub("", normalize_whitespace(text)).strip()
++
++
++def has_terminal_punct(text: str) -> bool:
++    t = text.rstrip("®™ ")
++    return bool(t) and t[-1] in TERMINAL_PUNCT
++
++
++def is_prose(text: str) -> bool:
++    """A label that reads like a sentence: several words with function words, a sentence verb, or a full stop at the end."""
++    words = word_count(text)
++    if words < PROSE_MIN_WORDS:
++        return False
++    if text.rstrip().endswith((".", "!", "?")):
++        return True
++    return bool(SENTENCE_VERB_RE.search(text)) or len(PROSE_FUNCTION_WORDS_RE.findall(text)) >= 2
++
++
++def is_slogan(text: str) -> bool:
++    return bool(MARKETING_VERB_RE.match(text)) and word_count(text) >= PROSE_MIN_WORDS
++
++
++def _phrase_re(phrases: Iterable[str]) -> re.Pattern[str]:
++    alts = sorted({p.strip().lower() for p in phrases if p.strip()}, key=len, reverse=True)
++    return re.compile(r"^(?:" + "|".join(re.escape(a) for a in alts) + r")$", re.IGNORECASE)
++
++
++# ------------------------------------------------------------------------------------------------------------ role vocabulary (people)
++
++ROLE_RULES: list[tuple[str, re.Pattern[str]]] = [
++    ("founder", re.compile(r"\b(co-?founder|founder|fondat(eur|rice)|gründer(in)?|fondator[ei]|fundador(a)?|oprichter|medeoprichter)\b", re.IGNORECASE)),
++    ("ceo", re.compile(r"\b(chief executive( officer)?|ceo|pdg|président-directeur|geschäftsführer(in)?|managing director|directeur général|"
++                       r"algemeen directeur|consejero delegado|amministratore delegato|diretor executivo)\b", re.IGNORECASE)),
++    ("cfo", re.compile(r"\b(chief financial( officer)?|cfo|finanzvorstand|directeur financier|financieel directeur|director financiero|direttore finanziario)\b", re.IGNORECASE)),
++    ("cto", re.compile(r"\b(chief technology( officer)?|chief technical( officer)?|cto|directeur technique)\b", re.IGNORECASE)),
++    ("coo", re.compile(r"\b(chief operating( officer)?|coo|directeur des opérations)\b", re.IGNORECASE)),
++    ("chair", re.compile(r"\b(chair(man|woman|person)?|executive chair|présidente? du conseil|vorsitzende[rn]?|voorzitter|presidente del consejo|"
++                         r"presidente del consiglio)\b", re.IGNORECASE)),
++    ("president", re.compile(r"\b(?<!vice )(?<!vice-)president(?! of)|président(?!e? du conseil)\b", re.IGNORECASE)),
++    ("board", re.compile(r"\b(board member|member of the (supervisory |advisory )?board|non-executive director|independent director|"
++                         r"administrat(eur|rice)|aufsichtsrat|conseil d'administration|director(?= \(board)|trustee|bestuurslid|consejer[oa]|consigliere)\b", re.IGNORECASE)),
++    ("vp", re.compile(r"\b(vice[- ]president|vp|evp|svp|avp|vice-président(e)?)\b", re.IGNORECASE)),
++    ("head", re.compile(r"\b(head of|head,|general manager|gm|leiter(in)?|directeur|directrice|director|managing partner|partner|responsable|hoofd|"
++                        r"direttore|diretor(a)?|directora)\b", re.IGNORECASE)),
++]
++# Japanese titles have no word boundaries: plain substring rules, most specific first.
++ROLE_RULES_JA: list[tuple[str, re.Pattern[str]]] = [
++    ("founder", re.compile(r"創業者|共同創業者|ファウンダー")),
++    ("cfo", re.compile(r"最高財務責任者|CFO")),
++    ("cto", re.compile(r"最高技術責任者|CTO")),
++    ("coo", re.compile(r"最高執行責任者|COO")),
++    ("ceo", re.compile(r"代表取締役|最高経営責任者|社長|CEO")),
++    ("chair", re.compile(r"取締役会長|会長|議長")),
++    ("vp", re.compile(r"副社長|執行役員|バイスプレジデント")),
++    ("board", re.compile(r"社外取締役|取締役|監査役|理事")),
++    ("head", re.compile(r"本部長|事業部長|部長|統括|責任者|マネージャー|ディレクター")),
++]
++CHIEF_RE = re.compile(r"\b(chief\b|c[a-z]{1,3}o\b)|最高.{1,8}責任者", re.IGNORECASE)
++EXEC_CATEGORIES = frozenset({"ceo", "cfo", "cto", "coo", "founder", "president", "chair"})
++TITLE_WORD_RE = re.compile(r"\b(officer|manager|engineer|lead|counsel|scientist|architect|analyst|controller|secretary|treasurer|advisor|adviser|"
++                           r"strategist|evangelist|designer|emeritus|emerita|executive|principal|fellow|associate|specialist|consultant|"
++                           r"ingénieur|responsable|chargée?|gérant(e)?|associée?|manager|leiter(in)?|mitglied|vorstand|directeur|directrice|"
++                           r"adviseur|bestuurder|gerente|socio|sócio|membro|miembro)\b", re.IGNORECASE)
++TITLE_WORD_JA_RE = re.compile(r"取締役|執行役員|部長|社長|会長|役員|担当|責任者|マネージャー|ディレクター|エンジニア|顧問|監査役")
++
++
++def role_category(title: str | None) -> tuple[str, bool]:
++    """(category, is_executive) for a person's title — 'other' when nothing in the vocabulary matches."""
++    if not title:
++        return "other", False
++    for cat, pat in ROLE_RULES:
++        if pat.search(title):
++            return cat, cat in EXEC_CATEGORIES or bool(CHIEF_RE.search(title))
++    for cat, pat in ROLE_RULES_JA:
++        if pat.search(title):
++            return cat, cat in EXEC_CATEGORIES or bool(CHIEF_RE.search(title))
++    return "other", bool(CHIEF_RE.search(title))
++
++
++def looks_like_role_title(text: str) -> bool:
++    t = normalize_whitespace(text)
++    if not (2 <= len(t) <= MAX_PERSON_TITLE_CHARS):
++        return False
++    return role_category(t)[0] != "other" or bool(TITLE_WORD_RE.search(t)) or bool(TITLE_WORD_JA_RE.search(t)) or bool(CHIEF_RE.search(t))
++
++
++# ------------------------------------------------------------------------------------------------------------ people
++
++_UP, _LOW = r"A-ZÀ-ÝĀ-Ž", r"a-zà-ÿā-ž'’"
++NAME_TOKEN = (rf"(?:[{_UP}](?:['’][{_UP}])?[{_LOW}]*(?:[{_UP}][{_LOW}]+)?(?:-[{_UP}{_LOW}][{_LOW}]*)*|"     # Jane · O'Neill · McGrath · García-López
++              rf"[{_UP}]\.|"                                                                                # initial
++              rf"(?:de|van|von|der|den|ter|ten|da|di|do|dos|das|le|la|los|las|du|del|della|bin|ibn|al|el|y|e|of|op)\b|"
++              rf"(?:de|van|von|da|di|le|la|du|d|l)[{_UP}][{_LOW}]+)")                                        # deSouza · DiCaprio · LeBlanc
++NAME_SUFFIX = r"(?:,?\s+(?:Jr\.?|Sr\.?|II|III|IV|PhD|MD|MBA|CPA|Esq\.?))?"
++NAME_RE = re.compile(rf"^(?:(?:Dr|Prof|Mr|Mrs|Ms|Sir|Dame|Hon)\.?\s+)?{NAME_TOKEN}(?:\s+{NAME_TOKEN}){{{PERSON_NAME_MIN_TOKENS - 1},{PERSON_NAME_MAX_TOKENS - 1}}}"
++                     rf"{NAME_SUFFIX}$")
++NAME_SUFFIX_TAIL_RE = re.compile(r"^\s*(?:Jr\.?|Sr\.?|II|III|IV|PhD|MD|MBA|CPA|Esq\.?)\s*$", re.IGNORECASE)
++ABBREVIATION_END_RE = re.compile(r"\b(?:Jr|Sr|[A-Z])\.$")
++JA_NAME_RE = re.compile(r"^[一-鿿぀-ヿ]{1,5}(?:[\s　・]+[一-鿿぀-ヿ]{1,5})?$")
++JA_NOT_NAME_RE = re.compile(r"代表|取締役|社長|会長|役員|部長|執行|責任者|最高|担当|マネージャー|ディレクター|チーム|株式会社|有限会社|会社|事業|本部|営業|経営|"
++                            r"お問い合わせ|プロフィール|詳細|一覧|紹介|概要|採用|ニュース|ホーム")
++NOT_NAME_RE = re.compile(
++    r"\b(team|our|meet|leadership|board|executive|management|contact|about|officer|director|directors|president|manager|head|chief|founder|"
++    r"partner|partners|group|company|global|senior|vice|read|more|learn|view|profile|bio|linkedin|email|join|careers|news|press|the|and|of|for|"
++    r"at|in|on|to|with|chair|chairman|emeritus|emerita|vp|advisors?|advisory|committee|members?|staff|people|employees|équipe|direction|"
++    r"conseil|kontakt|vorstand|geschäftsführung|équipe de direction|equipo|dirección|squadra|direzione|diretoria|equipe)\b", re.IGNORECASE)
++GENERIC_PERSON_NAMES_RE = _phrase_re([
++    "contact", "contact us", "team", "our team", "the team", "meet the team", "board", "board of directors", "directors", "leadership", "our leadership",
++    "leadership team", "senior leadership", "executive team", "executive committee", "management", "management team", "our people", "advisors",
++    "advisory board", "about us", "kontakt", "unser team", "vorstand", "geschäftsführung", "aufsichtsrat", "équipe", "notre équipe", "direction",
++    "conseil d'administration", "comité de direction", "equipo", "nuestro equipo", "dirección", "consejo", "squadra", "direzione", "equipe",
++    "diretoria", "役員", "経営陣", "チーム", "お問い合わせ", "会社概要",
++])
++PERSON_TITLE_STOPLIST_RE = _phrase_re([
++    "contact", "contact us", "read more", "read bio", "view bio", "full bio", "bio", "biography", "profile", "view profile", "linkedin", "twitter", "x",
++    "email", "e-mail", "phone", "download", "download photo", "download bio", "website", "more", "learn more", "details", "en savoir plus", "voir le profil",
++    "profil", "kontakt", "lebenslauf", "mehr erfahren", "biografie", "biografía", "ver perfil", "leggi la biografia", "meer info", "lees meer",
++    "プロフィール", "詳細を見る", "詳細",
++])
++
++
++def _strip_name(text: str) -> str:
++    """Trim punctuation around a name but keep a final abbreviation dot ("Edgar S. Woolard, Jr.")."""
++    t = normalize_whitespace(text).strip(" ,:;-–—|")
++    return t if ABBREVIATION_END_RE.search(t) else t.rstrip(".").strip()
++
++
++def looks_like_person_name(text: str) -> bool:
++    """2–5 capitalised tokens (particles, initials, Mc/Mac, glued particles allowed), no digits, no role vocabulary; a peerage or
++    post-nominal tail after a comma ("…, Baron Trotman") is ignored; or a short CJK name without role words."""
++    t = _strip_name(text)
++    if not t or any(ch.isdigit() for ch in t):
++        return False
++    if _CJK_RE.search(t):
++        return bool(JA_NAME_RE.match(t)) and not JA_NOT_NAME_RE.search(t) and 2 <= len(t.replace(" ", "").replace("　", "")) <= 10
++    head, _, tail = t.partition(",")
++    core = t if (not tail or NAME_SUFFIX_TAIL_RE.match(tail)) else head.strip()
++    if not (MIN_PERSON_NAME_CHARS <= len(core) <= MAX_PERSON_NAME_CHARS):
++        return False
++    if NOT_NAME_RE.search(core) or GENERIC_PERSON_NAMES_RE.match(core):
++        return False
++    return bool(NAME_RE.match(core))
++
++
++def clean_person_title(title: str | None) -> str | None:
++    """Stoplisted link labels (Contact, Read more, LinkedIn…), bio sentences and over-long strings are not titles."""
++    if not title:
++        return None
++    t = strip_trailing_marks(title)
++    if not t or PERSON_TITLE_STOPLIST_RE.match(t) or len(t) > MAX_PERSON_TITLE_CHARS:
++        return None
++    if word_count(t) >= SENTENCE_TITLE_MIN_WORDS and t.endswith((".", "!", "?")):
++        return None
++    if is_prose(t) and not looks_like_role_title(t):
++        return None
++    return t.rstrip(".") if not t.endswith("...") else t
++
++
++def normalize_person(name: str, title: str | None) -> tuple[str, str | None] | None:
++    """Validated (name, title) — swapped when the card put the title first — or None when the row is not a person."""
++    n = _strip_name(name)
++    t = normalize_whitespace(title).strip(" ,:;-–—|") if title else None      # keep a final "." — it marks a bio sentence
++    if not looks_like_person_name(n):
++        if t and looks_like_person_name(t) and looks_like_role_title(n):
++            n, t = t, n
++        else:
++            return None
++    return n, clean_person_title(t)
++
++
++def person_verdict(name: str, title: str | None) -> Verdict:
++    fixed = normalize_person(name, title)
++    if fixed is None:
++        if looks_like_role_title(name):
++            return Verdict.reject("name is a role title")
++        if GENERIC_PERSON_NAMES_RE.match(_strip_name(name)):
++            return Verdict.reject("generic label, not a person")
++        return Verdict.reject("not a person name")
++    return Verdict.accept()
++
++
++# ------------------------------------------------------------------------------------------------------------ jobs
++
++JOB_ROLE_RE = re.compile(
++    r"\b(?:engineer|ingénieur|ingenieur|ingeniero|ingegnere|engenheiro|developer|développeur|entwickler|desarrollador|sviluppatore|desenvolvedor|"
++    r"ontwikkelaar|programmer|programmeur|manager|gestionnaire|gerente|analyst|analyste|analista|designer|director|directeur|directrice|direktor|"
++    r"diretor|direttore|specialist|spécialiste|spezialist|especialista|specialista|intern|internship|stagiaire|praktikant|praktikum|stagiair|stagista|"
++    r"tirocinio|tirocinante|becario|estagiário|estágio|alternan(?:t|ce)|apprenti|apprentice|auszubildende[rn]?|ausbildung|azubi|werkstudent|"
++    r"sales|consultant|consulente|consultor|berater|adviseur|technician|technicien|techniker|técnico|tecnico|monteur|nurse|infirmi(?:er|ère)|"
++    r"krankenpfleger|verpleegkundige|enfermer[oa]|infermier[ea]|pflegefachkraft|pflegekraft|driver|chauffeur|fahrer|conductor|autista|motorista|officer|"
++    r"lead|head|associate|coordinator|coordinat(?:eur|rice)|koordinator|coördinator|coordinador|coordinatore|coordenador|architect|architecte|"
++    r"architekt|arquitecto|architetto|arquiteto|scientist|scientifique|wissenschaftler|científico|scienziato|cientista|accountant|comptable|"
++    r"buchhalter|contable|contabile|contador|operator|opérateur|operador|operatore|mechanic|mécanicien|mechaniker|mecánico|meccanico|mecânico|"
++    r"representative|représentant|assistant|assistent|asistente|assistente|administrator|administrateur|administrador|amministratore|advisor|"
++    r"adviser|counsel|attorney|lawyer|avocat|jurist|recruiter|controller|supervisor|superviseur|planner|buyer|acheteur|einkäufer|trainee|graduate|"
++    r"student|étudiant|studierende|estudiante|responsable|chargée?|chef de|leiter(?:in)?|mitarbeiter(?:in)?|referent(?:in)?|sachbearbeiter(?:in)?|"
++    r"kaufmann|kauffrau|fachkraft|medewerker|teamleider|projectleider|projektleiter|accountmanager|jefe|addetto|impiegato|responsabile|executive|"
++    r"principal|partner|paralegal|pharmacist|physician|therapist|teacher|professor|researcher|chercheur|forscher|electrician|électricien|elektriker|"
++    r"plumber|welder|soudeur|schweißer|machinist|cook|chef|barista|cashier|clerk|agent|guard|cleaner|handler|installer|inspector|auditor|actuary|"
++    r"underwriter|banker|trader|economist|strategist|producer|editor|writer|rédacteur|redakteur|translator|marketer|merchandiser|dispatcher|steward|"
++    r"pilot|captain|technologist|veterinarian|dentist|dietitian|paramedic|caregiver|educator|industriemechaniker|softwareentwickler|elektroniker|"
++    r"mechatroniker|pfleger|erzieher(?:in)?|verkäufer(?:in)?|vendeur|vendeuse|verkoper|commercial|comercial|venditore|vendedor)s?\b|"
++    r"エンジニア|マネージャー|マネジャー|デザイナー|ディレクター|コンサルタント|スペシャリスト|アナリスト|セールス|営業|開発|正社員|契約社員|インターン|職|担当", re.IGNORECASE)
++JOB_FORMAT_RE = re.compile(r"\(\s*(?:[mfwhvdx]|all genders|alle geschlechter|tous genres)(?:\s*/\s*[mfwhvdx*])*\s*\)|\b(?:full[- ]?time|part[- ]?time|vollzeit|"
++                           r"teilzeit|temps plein|temps partiel|fulltime|parttime|cdi|cdd|freelance|\d{2,3}\s?%)\b", re.IGNORECASE)
++GENDER_MARK_RE = re.compile(r"\s*\(\s*(?:[mfwhvdx](?:\s*/\s*[mfwhvdx*]){1,3}|all genders|alle geschlechter|tous genres|m/f/d|h/f)\s*\)", re.IGNORECASE)
++APPLY_SUFFIX_RE = re.compile(r"\s*[-–—|·]\s*(?:apply(?: now)?|postuler|jetzt bewerben|bewerben|solliciteer(?: nu)?|candidati|candidatar|aplicar)\s*$", re.IGNORECASE)
++BRACKET_ID_RE = re.compile(r"\s*[\[(]\s*(?:#\s*|(?:job\s*id|ref\.?|id|req)(?:\s*[:#]\s*|\s+))?([A-Z]{0,4}[-_]?\d{3,})\s*[\])]", re.IGNORECASE)
++JOB_STOP_EXACT_RE = _phrase_re([
++    "read more", "learn more", "more info", "more information", "more", "info", "details", "view details", "view job", "view all", "view all jobs",
++    "view all openings", "view openings", "see all", "see all jobs", "see more", "see open roles", "search jobs", "search", "job search", "all jobs",
++    "all openings", "all positions", "open positions", "open roles", "openings", "jobs", "careers", "career", "apply", "apply now", "apply here",
++    "join us", "join our team", "join the team", "find out more", "discover more", "explore", "explore roles", "back", "back to top", "load more",
++    "show more", "next", "previous", "home", "filter", "filters", "sort by", "share", "save", "print", "benefits", "culture", "our culture",
++    "our values", "diversity", "faq", "hiring process", "talent community", "talent network", "job alerts", "job alert", "sign up for job alerts",
++    "meer info", "meer informatie", "lees meer", "solliciteer", "solliciteer nu", "alle vacatures", "bekijk alle vacatures", "bekijk vacature",
++    "vacatures", "ontdek meer", "meer weten", "en savoir plus", "lire la suite", "postuler", "postulez", "voir toutes les offres", "toutes les offres",
++    "voir l'offre", "voir plus", "découvrir", "nos offres", "offres d'emploi", "mehr erfahren", "mehr infos", "jetzt bewerben", "bewerben",
++    "alle stellen", "alle stellenangebote", "stellenangebote", "alle jobs", "offene stellen", "weiterlesen", "mehr anzeigen", "zur stelle", "karriere",
++    "ver más", "leer más", "aplicar", "aplica ahora", "postúlate", "ver todas las ofertas", "todas las ofertas", "más información", "saber más",
++    "empleo", "ofertas de empleo", "scopri di più", "leggi tutto", "candidati", "candidati ora", "tutte le posizioni", "tutte le offerte",
++    "maggiori informazioni", "posizioni aperte", "lavora con noi", "saiba mais", "ler mais", "candidatar", "candidate-se", "ver todas as vagas",
++    "todas as vagas", "mais informações", "vagas", "詳細を見る", "もっと見る", "応募する", "エントリー", "募集一覧", "採用情報", "募集要項",
++])
++JOB_STOP_PREFIX_RE = re.compile(r"^(?:early careers?|graduate programm?e|graduates?\b|students?\b|starters\b|careers? (?:at|bij|chez|bei)\b|jobs? (?:at|bij|chez|bei)\b|"
++                                r"working at\b|life at\b|why (?:join|work|us)\b|join us\b|about us\b|who we are\b|what we do\b|meet our\b|our (?:story|teams?|culture|"
++                                r"values|benefits|people)\b|werken bij\b|waarom\b|pourquoi\b|warum\b|karriere bei\b|travailler chez\b|arbeiten bei\b|"
++                                r"trabajar en\b|lavorare in\b|trabalhar na\b)", re.IGNORECASE)
++JOB_URL_STRONG_RE = re.compile(
++    r"(?:/jobs?/[^/?#]+|/careers?/[^/?#]+/[^/?#]+|/positions?/|/openings?/|/vacanc(?:y|ies)/[^/?#]+|/vacatures?/[^/?#]+|/stellen(?:angebote?)?/[^/?#]+|"
++    r"/stelle/|/offres?(?:-d-?emploi)?/[^/?#]+|/emplois?/[^/?#]+|/empleos?/[^/?#]+|/lavoro/[^/?#]+|/vagas?/[^/?#]+|/opportunit(?:y|ies)/[^/?#]+|"
++    r"/\d{4,}(?:[/-]|$)|[?&](?:gh_jid|jobid|job_id|jid|reqid|req_id|id)=\d+|lever\.co/|greenhouse\.io/|ashbyhq\.com/|myworkdayjobs\.com/|"
++    r"smartrecruiters\.com/|workable\.com/|recruitee\.com/|personio\.(?:de|com)/|teamtailor\.com/|bamboohr\.com/|jobvite\.com/|icims\.com/|"
++    r"taleo\.net/|successfactors\.(?:com|eu)/|breezy\.hr/|applytojob\.com/|eightfold\.ai/|phenompeople\.com/|avature\.net/|csod\.com/|oraclecloud\.com/)",
++    re.IGNORECASE)
++KNOWN_LOCATION_WORDS_RE = re.compile(r"\b(?:remote|hybrid|on-?site|télétravail|homeoffice|home office|thuiswerken|worldwide|anywhere)\b", re.IGNORECASE)
++
++
++def clean_job_title(title: str) -> tuple[str, str | None]:
++    """Display title without gender markers, "- Apply" suffixes and bracketed ids; the id found (if any) is returned for the fingerprint."""
++    t = strip_trailing_marks(title)
++    found_id: str | None = None
++    m = BRACKET_ID_RE.search(t)
++    if m:
++        found_id = m.group(1)
++        t = (t[:m.start()] + " " + t[m.end():]).strip()
++    t = GENDER_MARK_RE.sub("", t)
++    t = APPLY_SUFFIX_RE.sub("", t)
++    return normalize_whitespace(t).strip(" -–—|·,"), found_id
++
++
++def job_url_is_joblike(url: str | None) -> bool:
++    return bool(url) and bool(JOB_URL_STRONG_RE.search(url or ""))
++
++
++def states_job_location(text: str | None) -> bool:
++    if not text or len(text) > MAX_LOCATION_NAME_CHARS:
++        return False
++    p = parse_location(text)
++    return bool(p["city"] or p["country"] or p["remote"] or p["region"]) or is_known_city(text) or bool(KNOWN_LOCATION_WORDS_RE.search(text))
++
++
++def _department_like(text: str | None) -> bool:
++    return bool(text) and len(text or "") <= MAX_DEPARTMENT_CHARS and word_count(text or "") <= MAX_DEPARTMENT_WORDS and not any(ch.isdigit() for ch in text or "")
++
++
++def job_verdict(title: str, *, url: str | None = None, location: str | None = None, department: str | None = None) -> Verdict:
++    """CTA / navigation anchors are not jobs; a job needs role vocabulary, a job-like URL or an explicit location/department cell."""
++    raw = normalize_whitespace(title)
++    if ELLIPSIS_RE.search(raw):
++        return Verdict.reject("truncated title (ellipsis)")
++    t, _ = clean_job_title(raw)
++    if not (MIN_JOB_TITLE_CHARS <= len(t) <= MAX_JOB_TITLE_CHARS):
++        return Verdict.reject("title length")
++    if JOB_STOP_EXACT_RE.match(t) or JOB_STOP_PREFIX_RE.match(t):
++        return Verdict.reject("call-to-action / navigation label")
++    if word_count(t) < MIN_JOB_TITLE_WORDS and not _CJK_RE.search(t):
++        return Verdict.reject("single-word title")
++    if JOB_ROLE_RE.search(t) or JOB_FORMAT_RE.search(raw):
++        return Verdict.accept()
++    if is_prose(t):
++        return Verdict.reject("reads like a sentence")
++    if job_url_is_joblike(url) or states_job_location(location) or _department_like(department):
++        return Verdict.accept()
++    return Verdict.reject("no job-like signal (vocabulary, url, location)")
++
++
++def refine_job(job: ExtractedJob) -> ExtractedJob | None:
++    """Apply `job_verdict`, clean the display title and keep a bracketed id as `external_id` (so the fingerprint keeps it)."""
++    if not job_verdict(job.title, url=job.url, location=job.location_text, department=job.department).ok:
++        return None
++    clean, found_id = clean_job_title(job.title)
++    if clean:
++        job.title = clean
++    if found_id and not job.external_id:
++        job.external_id = found_id
++    return job
++
++
++# ------------------------------------------------------------------------------------------------------------ locations
++
++NAV_COOKIE_RE = re.compile(
++    r"\b(?:performance|analytics|analytical|marketing|functional|functionality|necessary|essential|preferences?|strictly|targeting|advertising|"
++    r"advertisement|statistics?|statistik(?:en)?|statistiques?|statistieken|estadisticas?|statistiche|tracking|cookies?|consent|privacy|privacybeleid|"
++    r"datenschutz|confidentialite|privacidad|gdpr|contact us|contact|contactez|contacto|contatti|kontakt|careers?|jobs?|sitemap|terms|legal|"
++    r"mentions legales|impressum|accessibility|newsletter|subscribe|login|log in|sign in|sign up|register|search|menu|language|select|filter|"
++    r"view all|see all|all locations|find (?:a|an|your)|more|back|home|about us|faq|help|support|settings|einstellungen|notwendig|funktional|"
++    r"necessaires?|fonctionnels?|preferences|noodzakelijk|functioneel|voorkeuren|necesarias?|funcionales?|preferencias|necessari|funzionali|"
++    r"preferenze|publicidad|pubblicita|werbung|publicite|social media|session|third party|unclassified|uncategori[sz]ed|others?)\b", re.IGNORECASE)
++LOCATION_KIND_WORD_RE = re.compile(
++    r"\b(?:headquarters|head office|hq|office|offices|bureau|büro|kantoor|oficina|ufficio|escritório|store|shop|boutique|showroom|plant|factory|usine|"
++    r"werk|fabrik|fabriek|fábrica|fabbrica|warehouse|entrepôt|lager|magazijn|almacén|lab|laboratory|laboratoire|labor|campus|branch|agence|filiale|"
++    r"succursale|niederlassung|vestiging|sede|siège|hauptsitz|data ?cent(?:er|re)|distribution cent(?:er|re))\b|本社|支社|営業所|工場|オフィス|拠点", re.IGNORECASE)
++VENUE_RE = re.compile(r"\b(?:hotel|tower|towers|building|centre|center|plaza|mall|park|campus|house|hall|street|avenue|road|floor|level|suite|"
++                      r"hôtel|gebäude|gebouw|edificio|torre|palazzo)\b", re.IGNORECASE)
++STREET_RE = re.compile(
++    r"\b\d{1,5}[a-z]?\s+[^\n,]{2,40}\b(?:street|st\.?|avenue|ave\.?|road|rd\.?|boulevard|blvd\.?|drive|dr\.?|lane|ln\.?|way|place|pl\.?|square|plaza|"
++    r"court|ct\.?|parkway|highway|straße|strasse|str\.|allee|platz|weg|rue|avenida|calle|via|piazza)\b"
++    r"|\b(?:rue|avenue|boulevard|via|calle|avenida|carrer|rua|praça)\s+[^\n,]{2,40}\s\d{1,5}\b"
++    r"|\b(?:[^\s,\d]+\s){0,2}[^\s,]{3,40}(?:straat|weg|laan|plein|kade|zijde|gracht|singel|markt|dijk|straße|strasse|str\.|allee|platz|gasse|ring|damm|ufer|"
++    r"vej|gade|gatan|vägen|katu|tie)\s+\d{1,5}[a-z]?\b", re.IGNORECASE)
++
++# ~400 major cities and business hubs with common exonyms/endonyms; keys are accent-insensitive (see `key_of`).
++_CITIES = """
++new york, los angeles, chicago, houston, phoenix, philadelphia, san antonio, san diego, dallas, san jose, austin, jacksonville, fort worth, columbus,
++charlotte, san francisco, indianapolis, seattle, denver, washington, boston, nashville, detroit, portland, las vegas, memphis, louisville, baltimore,
++milwaukee, albuquerque, tucson, fresno, sacramento, kansas city, atlanta, miami, oakland, minneapolis, cleveland, raleigh, omaha, tampa, orlando,
++pittsburgh, cincinnati, st. louis, salt lake city, richmond, new orleans, buffalo, hartford, providence, durham, boulder, palo alto, mountain view,
++menlo park, redmond, bellevue, cambridge, princeton, stamford, irvine, santa clara, sunnyvale, cupertino, redwood city, san mateo, arlington, reston,
++mclean, plano, scottsdale, tempe, ann arbor, madison, des moines, boise, honolulu, anchorage, charleston, savannah, oklahoma city, tulsa, el paso,
++long beach, colorado springs, newark, jersey city, brooklyn, manhattan, santa monica, pasadena, burbank, anaheim, riverside, henderson, reno, spokane,
++tacoma, rochester, syracuse, albany, wilmington, trenton, grand rapids, toledo, dayton, lexington, knoxville, chattanooga, huntsville, baton rouge,
++little rock, tallahassee, st. petersburg, fort lauderdale, west palm beach, boca raton, greenville, columbia, greensboro, norfolk, alexandria, bethesda,
++rockville, toronto, montréal, montreal, vancouver, calgary, edmonton, ottawa, winnipeg, québec, quebec city, hamilton, kitchener, waterloo, halifax,
++victoria, saskatoon, regina, mississauga, brampton, markham, vaughan, burnaby, surrey, laval, gatineau, kelowna, oakville, burlington,
++mexico city, ciudad de méxico, guadalajara, monterrey, puebla, tijuana, querétaro, cancún, mérida, são paulo, sao paulo, rio de janeiro, brasília, brasilia,
++belo horizonte, curitiba, porto alegre, salvador, recife, fortaleza, campinas, florianópolis, manaus, buenos aires, córdoba, rosario, mendoza, santiago,
++valparaíso, lima, bogotá, bogota, medellín, cali, cartagena, quito, guayaquil, caracas, montevideo, asunción, la paz, san josé, panama city,
++ciudad de panamá, guatemala city, san salvador, tegucigalpa, managua, santo domingo, san juan, havana, kingston, nassau,
++london, manchester, birmingham, leeds, glasgow, edinburgh, liverpool, bristol, sheffield, newcastle, nottingham, cardiff, belfast, leicester, coventry,
++oxford, reading, milton keynes, brighton, southampton, portsmouth, aberdeen, dundee, york, bath, exeter, plymouth, norwich, swindon, slough, watford,
++guildford, basingstoke, warrington, derby, sunderland, hull, bradford, dublin, cork, galway, limerick, waterford,
++paris, marseille, lyon, toulouse, nice, nantes, strasbourg, montpellier, bordeaux, lille, rennes, reims, le havre, saint-étienne, toulon, grenoble,
++dijon, angers, nîmes, villeurbanne, clermont-ferrand, le mans, aix-en-provence, brest, tours, amiens, limoges, annecy, perpignan, metz, besançon,
++orléans, rouen, mulhouse, caen, nancy, boulogne-billancourt, issy-les-moulineaux, la défense, courbevoie, neuilly-sur-seine, levallois-perret, nanterre,
++puteaux, saint-denis, versailles, sophia antipolis, cannes, monaco,
++berlin, hamburg, münchen, munich, köln, cologne, frankfurt, frankfurt am main, stuttgart, düsseldorf, dortmund, essen, leipzig, bremen, dresden, hannover,
++hanover, nürnberg, nuremberg, duisburg, bochum, wuppertal, bielefeld, bonn, münster, karlsruhe, mannheim, augsburg, wiesbaden, gelsenkirchen,
++mönchengladbach, braunschweig, chemnitz, kiel, aachen, halle, magdeburg, freiburg, krefeld, lübeck, mainz, erfurt, oberhausen, rostock, kassel, hagen,
++saarbrücken, potsdam, ludwigshafen, oldenburg, leverkusen, heidelberg, darmstadt, regensburg, ingolstadt, würzburg, ulm, wolfsburg, göttingen,
++paderborn, heilbronn, erlangen, jena, walldorf, böblingen, sindelfingen, neckarsulm, herzogenaurach, gütersloh,
++wien, vienna, graz, linz, salzburg, innsbruck, zürich, zurich, genève, geneva, genf, basel, bâle, bern, berne, lausanne, winterthur, luzern, lucerne,
++st. gallen, lugano, zug, baar, vevey, neuchâtel,
++amsterdam, rotterdam, den haag, the hague, utrecht, eindhoven, groningen, tilburg, almere, breda, nijmegen, arnhem, haarlem, amersfoort, enschede,
++apeldoorn, 's-hertogenbosch, leiden, delft, maastricht, zwolle, hilversum, hoofddorp, schiphol, amstelveen, veldhoven, brussels, bruxelles, brussel,
++antwerp, antwerpen, anvers, ghent, gent, gand, charleroi, liège, luik, bruges, brugge, namur, leuven, louvain, mechelen, mons, hasselt, kortrijk,
++zaventem, diegem, luxembourg, esch-sur-alzette,
++stockholm, göteborg, gothenburg, malmö, uppsala, västerås, linköping, lund, helsingborg, örebro, oslo, bergen, trondheim, stavanger, drammen,
++copenhagen, københavn, aarhus, århus, odense, aalborg, helsinki, helsingfors, espoo, tampere, vantaa, oulu, turku, reykjavík, reykjavik, tallinn, riga,
++vilnius, kaunas,
++madrid, barcelona, valencia, sevilla, seville, zaragoza, málaga, murcia, palma, las palmas, bilbao, alicante, valladolid, vigo, gijón, a coruña,
++granada, vitoria-gasteiz, san sebastián, donostia, pamplona, santander, lisboa, lisbon, porto, oporto, braga, coimbra, faro, funchal, roma, rome,
++milano, milan, napoli, naples, torino, turin, palermo, genova, genoa, bologna, firenze, florence, bari, catania, venezia, venice, verona, messina,
++padova, padua, trieste, brescia, parma, modena, reggio emilia, perugia, bergamo, vicenza, monza, athens, athina, thessaloniki, piraeus, nicosia,
++limassol, valletta, ljubljana, zagreb, split, belgrade, beograd, sarajevo, skopje, tirana, podgorica,
++warsaw, warszawa, kraków, krakow, cracow, łódź, lodz, wrocław, wroclaw, poznań, poznan, gdańsk, gdansk, szczecin, katowice, lublin, bydgoszcz, prague,
++praha, brno, ostrava, plzeň, bratislava, košice, budapest, debrecen, szeged, bucharest, bucurești, cluj-napoca, timișoara, iași, sofia, plovdiv,
++varna, kyiv, kiev, kharkiv, lviv, odesa, odessa, dnipro, minsk, moscow, moskva, saint petersburg, st. petersburg, novosibirsk, yekaterinburg, kazan,
++chișinău, tbilisi, yerevan, baku,
++istanbul, ankara, izmir, bursa, antalya, tel aviv, jerusalem, haifa, herzliya, petah tikva, beersheba, dubai, abu dhabi, sharjah, doha, riyadh, jeddah,
++dammam, kuwait city, manama, muscat, amman, beirut, baghdad, tehran, cairo, giza, casablanca, rabat, marrakech, tangier, tunis, algiers, lagos, abuja,
++nairobi, mombasa, accra, addis ababa, dar es salaam, kampala, kigali, johannesburg, cape town, durban, pretoria, gqeberha, luanda, maputo, lusaka,
++harare, dakar, abidjan, kinshasa, douala, yaoundé, windhoek, gaborone, antananarivo, port louis, khartoum,
++tokyo, 東京, osaka, 大阪, yokohama, 横浜, nagoya, 名古屋, sapporo, 札幌, fukuoka, 福岡, kobe, 神戸, kyoto, 京都, kawasaki, 川崎, saitama, hiroshima, 広島,
++sendai, 仙台, chiba, 千葉, seoul, 서울, busan, 부산, incheon, 인천, daegu, daejeon, gwangju, suwon, pangyo, seongnam, beijing, 北京, peking, shanghai, 上海,
++guangzhou, 广州, shenzhen, 深圳, chengdu, 成都, hangzhou, 杭州, wuhan, 武汉, xi'an, 西安, chongqing, 重庆, tianjin, 天津, nanjing, 南京, suzhou, 苏州,
++qingdao, 青岛, dalian, 大连, xiamen, 厦门, shenyang, changsha, zhengzhou, dongguan, ningbo, hong kong, 香港, macau, macao, 澳门, taipei, 台北, 臺北,
++taichung, kaohsiung, hsinchu, tainan, singapore, kuala lumpur, penang, george town, johor bahru, cyberjaya, petaling jaya, bangkok, กรุงเทพ,
++chiang mai, jakarta, surabaya, bandung, denpasar, medan, manila, makati, quezon city, cebu, taguig, bonifacio global city, pasig, hanoi, hà nội,
++ho chi minh city, hồ chí minh, saigon, da nang, phnom penh, yangon, vientiane, dhaka, chittagong, colombo, kathmandu, karachi, lahore, islamabad,
++rawalpindi, faisalabad, mumbai, bombay, delhi, new delhi, bengaluru, bangalore, hyderabad, chennai, madras, kolkata, calcutta, pune, ahmedabad, jaipur,
++surat, lucknow, kanpur, nagpur, indore, thane, bhopal, visakhapatnam, vadodara, coimbatore, kochi, cochin, gurgaon, gurugram, noida, chandigarh,
++mysore, mysuru, thiruvananthapuram, trivandrum, bhubaneswar, mohali, navi mumbai, sydney, melbourne, brisbane, perth, adelaide, canberra,
++gold coast, hobart, darwin, wollongong, geelong, auckland, wellington, christchurch, dunedin, tauranga, ulaanbaatar, almaty, astana, tashkent, bishkek
++"""
++CITY_KEYS: frozenset[str] = frozenset(key_of(c) for c in _CITIES.replace("\n", ",").split(",") if c.strip())
++
++
++def is_known_city(text: str | None) -> bool:
++    """Whole string or any 1–3-word n-gram of a short string names a city in the table."""
++    if not text:
++        return False
++    k = key_of(text)
++    if not k:
++        return False
++    if k in CITY_KEYS:
++        return True
++    if _CJK_RE.search(text):
++        return any(city in k for city in CITY_KEYS if _CJK_RE.search(city))
++    toks = k.split()
++    if len(toks) > MAX_LOCATION_NAME_WORDS:
++        return False
++    for n in range(1, min(CITY_NGRAM_MAX, len(toks)) + 1):
++        for i in range(len(toks) - n + 1):
++            if " ".join(toks[i:i + n]) in CITY_KEYS:
++                return True
++    return False
++
++
++CITY_DISTRICT_RE = re.compile(r"\s\d{1,2}$")          # "Dublin 2", "Paris 8" — a postal district after a known city
++
++
++def has_postal_code(text: str | None) -> bool:
++    if not text:
++        return False
++    return any(not YEAR_RE.match(m.group(0)) for m in POSTAL_RE.finditer(text))
++
++
++def looks_like_city_value(text: str | None) -> bool:
++    """A plausible `city` cell: short, capitalised words, no digits, no nav/cookie vocabulary, no sentence."""
++    if not text:
++        return False
++    t = normalize_whitespace(text)
++    if not (2 <= len(t) <= MAX_CITY_CHARS) or word_count(t) > MAX_CITY_WORDS:
++        return False
++    if any(ch.isdigit() for ch in t) and not (CITY_DISTRICT_RE.search(t) and is_known_city(CITY_DISTRICT_RE.sub("", t))):
++        return False                                                                   # "Level 12" no · "Dublin 2" (postal district) yes
++    if NAV_COOKIE_RE.search(key_of(t)) or is_prose(t):
++        return False
++    return t[0].isupper() or bool(_CJK_RE.search(t[0]))
++
++
++def _looks_like_venue(text: str) -> bool:
++    return word_count(text) >= VENUE_MIN_WORDS or bool(VENUE_RE.search(text))
++
++
++def location_signal(*, name: str, city: str | None, region: str | None, country: str | None, kind: str | None, address: str | None) -> str | None:
++    """The evidence that makes this a real place, or None: country / known city / postal code / street address / explicit kind label."""
++    if country:
++        return "country"
++    if is_known_city(city) or is_known_city(region) or is_known_city(name):
++        return "known city"
++    if has_postal_code(address) or (has_postal_code(name) and "," in name):
++        return "postal code"
++    if STREET_RE.search(address or "") or STREET_RE.search(name):
++        return "street address"
++    if (kind and kind not in ("office", "other")) or LOCATION_KIND_WORD_RE.search(name):
++        return "kind label"
++    return None
++
++
++def normalize_location(loc: ExtractedLocation) -> ExtractedLocation | None:
++    """Validated copy of `loc` (city cell cleaned, country-as-name repaired) or None when it is not a place."""
++    name = strip_trailing_marks(loc.name or "")
++    if not name or len(name) > MAX_LOCATION_NAME_CHARS or word_count(name) > MAX_LOCATION_NAME_WORDS:
++        return None
++    if NAV_COOKIE_RE.search(key_of(name)) or is_prose(name):
++        return None
++    city = loc.city if looks_like_city_value(loc.city) else None
++    region = loc.region
++    country = loc.country
++    name_country = country_code(name)
++    if name_country and city and key_of(city) == key_of(name) and not is_known_city(name):
++        city = None                                                                            # "Oman | Oman": a country is not its own city (Singapore is)
++    if name_country and loc.city and (city is None or _looks_like_venue(loc.city)) and not NAV_COOKIE_RE.search(key_of(loc.city)) and not is_prose(loc.city) \
++            and len(loc.city) <= MAX_LOCATION_NAME_CHARS and key_of(loc.city) != key_of(name):
++        name, city, country = normalize_whitespace(loc.city), None, country or name_country   # "Oman | Hormuz Grand Hotel" → venue named, country kept
++    if region and is_known_city(region) and looks_like_city_value(region) and (city is None or not is_known_city(city)):
++        city, region = region, None                                                            # "3089 JH Rotterdam" parsed as a region
++    if location_signal(name=name, city=city, region=region, country=country, kind=loc.kind, address=loc.address_text) is None:
++        return None
++    return ExtractedLocation(name=name, kind=loc.kind or "office", city=city, region=region, country=country, address_text=loc.address_text)
++
++
++def location_verdict(loc: ExtractedLocation) -> Verdict:
++    name = strip_trailing_marks(loc.name or "")
++    if not name:
++        return Verdict.reject("empty name")
++    if len(name) > MAX_LOCATION_NAME_CHARS or word_count(name) > MAX_LOCATION_NAME_WORDS:
++        return Verdict.reject("name too long")
++    if NAV_COOKIE_RE.search(key_of(name)):
++        return Verdict.reject("cookie/consent/navigation vocabulary")
++    if is_prose(name):
++        return Verdict.reject("name is a sentence")
++    fixed = normalize_location(loc)
++    if fixed is None:
++        return Verdict.reject("no place evidence (country, city, postal code, street, kind)")
++    return Verdict.accept()
++
++
++# ------------------------------------------------------------------------------------------------------------ pricing plans
++
++PLAN_EYEBROW_RE = _phrase_re(["most popular", "popular", "best value", "recommended", "new", "beta", "coming soon", "limited offer", "best seller",
++                              "le plus populaire", "beliebt", "am beliebtesten", "meest gekozen", "más popular", "più popolare", "mais popular", "人気",
++                              # generic section headings that are not tiers
++                              "plans", "plan", "pricing", "prices", "price", "plans & pricing", "plans and pricing", "our plans", "compare plans",
++                              "choose your plan", "tarifs", "nos tarifs", "abonnements", "preise", "unsere preise", "tarife", "prijzen", "abonnementen",
++                              "precios", "planes", "prezzi", "piani", "preços", "planos", "料金", "料金プラン", "プラン"])
++PRICE_TOKEN_RE = re.compile(
++    r"(?:(?:US\$|CA\$|C\$|A\$|NZ\$|HK\$|S\$|MX\$|R\$|USD|EUR|GBP|CAD|AUD|CHF|JPY|INR|SEK|NOK|DKK|PLN|BRL|CNY|RMB|SGD|[$€£¥₹₩₺])\s?"
++    r"\d{1,3}(?:[,.\s]\d{3})*(?:[.,]\d{1,2})?|\d{1,3}(?:[,.\s]\d{3})*(?:[.,]\d{1,2})?\s?(?:€|£|USD|EUR|GBP|CHF|kr|zł|元|\$))", re.IGNORECASE)
++PRICE_LEAD_RE = re.compile(r"(?:from|starting at|starts at|as low as|à partir de|ab|desde|a partire da|vanaf|only|just)\s*$", re.IGNORECASE)
++PRICE_UNIT = (r"(?:month|mo|year|yr|annum|week|wk|day|hour|hr|user|seat|member|license|licence|agent|editor|contact|1,?000|1k|GB|TB|request|"
++              r"transaction|call|minute|mois|an|année|monat|jahr|nutzer|benutzer|utilisateur|usuario|utente|mese|anno|mes|año|maand|jaar|gebruiker|"
++              r"dag|tag|giorno|día|dia|jour)")
++PRICE_TAIL_RE = re.compile(rf"^\s*(?:(?:/|per|a|an|each|every|par|pro|pour|por|al|je)\s*{PRICE_UNIT}\b(?:\s*(?:/|per|par|pro|pour|por)\s*{PRICE_UNIT}\b)?|"
++                           rf"(?:monthly|annually|yearly|mensuel|annuel|monatlich|jährlich|one[- ]time|lifetime|forever))?"
++                           r"(?:,?\s*billed (?:monthly|annually|yearly))?", re.IGNORECASE)
++CONTACT_PHRASE_RE = re.compile(r"\b(contact (?:us|sales)|talk to (?:us|sales|an expert)|custom pricing|custom quote|get a quote|request (?:a )?(?:quote|demo)|"
++                               r"let'?s talk|on request|upon request|sur devis|nous contacter|contactez-nous|auf anfrage|individuell|tailored|bespoke|call us)\b",
++                               re.IGNORECASE)
++FREE_TOKEN_RE = re.compile(r"^(free|gratuit|kostenlos|gratis|\$\s?0(?:\.00)?|0\s?€|€\s?0)\b", re.IGNORECASE)
++
++
++def price_text_from(text: str) -> str | None:
++    """The price phrase stated in `text` — currency + amount + period/unit, or the contact-sales / free wording — never a cut sentence."""
++    t = normalize_whitespace(text)
++    m = PRICE_TOKEN_RE.search(t)
++    if m:
++        lead = PRICE_LEAD_RE.search(t[:m.start()])
++        start = lead.start() if lead else m.start()
++        tail = PRICE_TAIL_RE.match(t[m.end():])
++        end = m.end() + (tail.end() if tail else 0)
++        return t[start:end].strip(" ,")[:MAX_PRICE_TEXT_CHARS]
++    c = CONTACT_PHRASE_RE.search(t)
++    if c:
++        phrase = c.group(1)
++        return phrase[0].upper() + phrase[1:]
++    f = FREE_TOKEN_RE.match(t)
++    if f:
++        return f.group(1)
++    return None
++
++
++def plan_name_ok(name: str | None) -> bool:
++    if not name:
++        return False
++    n = normalize_whitespace(name)
++    if not (1 <= len(n) <= MAX_PLAN_NAME_CHARS) or word_count(n) > MAX_PLAN_NAME_WORDS:
++        return False
++    if has_terminal_punct(n) or "|" in n or "?" in n or PLAN_EYEBROW_RE.match(n) or re.search(r"\d+\s?%", n):
++        return False
++    if MARKETING_VERB_RE.match(n) and word_count(n) > 1:
++        return False
++    return not PRICE_TOKEN_RE.search(n)
++
++
++def plan_verdict(plan: ExtractedPlan) -> Verdict:
++    if not plan_name_ok(plan.plan_name):
++        return Verdict.reject("plan name is a heading / marketing phrase")
++    if plan.price is None and not plan.contact_sales and not (plan.price_text and FREE_TOKEN_RE.match(plan.price_text)):
++        return Verdict.reject("no price, no contact-sales, no free tier")
++    if plan.price_text and price_text_from(plan.price_text) is None:
++        return Verdict.reject("price text does not state a price")
++    return Verdict.accept()
++
++
++# ------------------------------------------------------------------------------------------------------------ products
++
++PRODUCT_NAV_RE = _phrase_re([
++    "overview", "products", "product", "all products", "our products", "solutions", "solution", "all solutions", "services", "our services", "features",
++    "pricing", "resources", "support", "docs", "documentation", "blog", "contact", "contact us", "home", "learn more", "read more", "view all", "see all",
++    "explore", "more", "get started", "sign up", "log in", "download", "compare", "industries", "platform", "use cases", "customers", "partners",
++    "produits", "nos produits", "tous les produits", "en savoir plus", "produkte", "alle produkte", "unsere produkte", "lösungen", "mehr erfahren",
++    "producten", "alle producten", "oplossingen", "meer info", "productos", "todos los productos", "soluciones", "ver más", "prodotti", "tutti i prodotti",
++    "soluzioni", "scopri di più", "produtos", "todos os produtos", "soluções", "saiba mais", "製品", "製品一覧", "ソリューション", "サービス", "詳細", "もっと見る",
++])
++
++
++def product_verdict(name: str) -> Verdict:
++    n = strip_trailing_marks(name)
++    if not (2 <= len(n) <= MAX_PRODUCT_NAME_CHARS):
++        return Verdict.reject("name length")
++    if word_count(n) > MAX_PRODUCT_NAME_WORDS:
++        return Verdict.reject("name too long")
++    if PRODUCT_NAV_RE.match(n.rstrip("®™ ")):
++        return Verdict.reject("navigation label")
++    if has_terminal_punct(n) and word_count(n) >= 3:
++        return Verdict.reject("name is a sentence")
++    if is_slogan(n) or is_prose(n):
++        return Verdict.reject("marketing slogan")
++    return Verdict.accept()
++
++
++# ------------------------------------------------------------------------------------------------------------ news
++
++NEWS_NOISE_RE = _phrase_re([
++    "read more", "read more news", "learn more", "more", "more news", "continue reading", "view all", "see all", "all news", "all posts", "all articles",
++    "next", "previous", "older", "newer", "older posts", "newer posts", "load more", "show more", "page", "news", "press", "press releases", "press release",
++    "blog", "blog posts", "articles", "insights", "events", "media", "in the news", "case studies", "webinars", "whitepapers", "podcasts", "videos",
++    "newsroom", "latest news", "category", "categories", "archive", "archives", "tags", "share", "rss", "subscribe", "newsletter", "actualités",
++    "toutes les actualités", "communiqués de presse", "lire la suite", "en savoir plus", "alle news", "pressemitteilungen", "aktuelles", "weiterlesen",
++    "mehr erfahren", "nieuws", "alle berichten", "persberichten", "lees meer", "noticias", "todas las noticias", "notas de prensa", "leer más", "notizie",
++    "comunicati stampa", "leggi tutto", "notícias", "ler mais", "ニュース", "お知らせ", "プレスリリース", "一覧", "もっと見る",
++])
++PAGINATION_RE = re.compile(r"^(?:page\s*)?\d{1,4}$|^(?:[«»‹›<>]|\.\.\.|…)+$", re.IGNORECASE)
++
++
++def news_verdict(title: str, *, url: str | None = None, published_at: datetime | None = None) -> Verdict:
++    t = strip_trailing_marks(title)
++    if not t or PAGINATION_RE.match(t):
++        return Verdict.reject("pagination")
++    if NEWS_NOISE_RE.match(t):
++        return Verdict.reject("navigation / category label")
++    confirmed = published_at is not None or date_from_url(url) is not None
++    long_enough = word_count(t) >= NEWS_MIN_WORDS or len(t) >= NEWS_MIN_CHARS or (bool(_CJK_RE.search(t)) and len(t) >= NEWS_MIN_CJK_CHARS)
++    if not confirmed and not long_enough:
++        return Verdict.reject("too short without a date")
++    return Verdict.accept()
++
++
++# ------------------------------------------------------------------------------------------------------------ whole extraction
++
++HTML_JOB_CONNECTORS = frozenset({"generic-html-v1"})
++
++
++def apply_precision(ex: Extraction, *, html_jobs: bool) -> dict[str, int]:
++    """Filter/normalise every typed list in place; returns how many items each list lost. Job rules only for HTML-scraped listings."""
++    dropped: dict[str, int] = {}
++    if html_jobs:
++        kept_jobs = [j for j in (refine_job(j) for j in ex.jobs) if j is not None]
++        dropped["jobs"] = len(ex.jobs) - len(kept_jobs)
++        ex.jobs = kept_jobs
++    people: list[ExtractedPerson] = []
++    for p in ex.people:
++        fixed = normalize_person(p.name, p.title)
++        if fixed is None:
++            continue
++        p.name, p.title = fixed
++        p.role_category, p.is_executive = role_category(p.title)
++        people.append(p)
++    dropped["people"] = len(ex.people) - len(people)
++    ex.people = people
++    products: list[ExtractedProduct] = [p for p in ex.products if product_verdict(p.name).ok]
++    dropped["products"] = len(ex.products) - len(products)
++    ex.products = products
++    plans: list[ExtractedPlan] = [p for p in ex.plans if plan_verdict(p).ok]
++    dropped["plans"] = len(ex.plans) - len(plans)
++    ex.plans = plans
++    locations: list[ExtractedLocation] = [loc for loc in (normalize_location(loc) for loc in ex.locations) if loc is not None]
++    dropped["locations"] = len(ex.locations) - len(locations)
++    ex.locations = locations
++    news: list[ExtractedNewsItem] = [n for n in ex.news if news_verdict(n.title, url=n.url, published_at=n.published_at).ok]
++    dropped["news"] = len(ex.news) - len(news)
++    ex.news = news
++    return {k: v for k, v in dropped.items() if v}
++
++
++def describe(verdict: Verdict) -> dict[str, Any]:
++    return {"ok": verdict.ok, "reason": verdict.reason}
++
++
++__all__ = [
++    "CHIEF_RE", "CITY_KEYS", "EXEC_CATEGORIES", "HTML_JOB_CONNECTORS", "NAV_COOKIE_RE", "PRECISION_VERSION", "ROLE_RULES", "STREET_RE", "Verdict",
++    "apply_precision", "clean_job_title", "clean_person_title", "describe", "has_postal_code", "is_known_city", "is_prose", "is_slogan", "job_url_is_joblike",
++    "job_verdict", "key_of", "location_signal", "location_verdict", "looks_like_city_value", "looks_like_person_name", "looks_like_role_title", "news_verdict",
++    "normalize_location", "normalize_person", "person_verdict", "plan_name_ok", "plan_verdict", "price_text_from", "product_verdict", "refine_job",
++    "role_category", "states_job_location", "strip_trailing_marks", "word_count",
++]

modified src/companyatlas/connectors/_util.py +1 −1

				`@@ -66,7 +66,7 @@ CA_PROVINCE_CODES_SAFE = {"ON", "QC", "BC", "AB", "MB", "SK", "NS", "NB"}`
66	AMBIGUOUS_CODES = (set(US_STATES.values()) \| set(CA_PROVINCES.values())) & ISO2	66	AMBIGUOUS_CODES = (set(US_STATES.values()) \| set(CA_PROVINCES.values())) & ISO2
67	REMOTE_RE = re.compile(r"\b(remote\|work from home\|wfh\|anywhere\|distributed\|télétravail\|homeoffice\|home office)\b", re.IGNORECASE)	67	REMOTE_RE = re.compile(r"\b(remote\|work from home\|wfh\|anywhere\|distributed\|télétravail\|homeoffice\|home office)\b", re.IGNORECASE)
68	HYBRID_RE = re.compile(r"\bhybrid\b", re.IGNORECASE)	68	HYBRID_RE = re.compile(r"\bhybrid\b", re.IGNORECASE)
69	−POSTAL_RE = re.compile(r"\b[A-Z]{1,2}\d[A-Z\d]?\s\d[A-Z]{2}\b\|\b\d{4,6}(?:-\d{4})?\b\|\b[A-Z]\d[A-Z]\s\d[A-Z]\d\b")	69	+POSTAL_RE = re.compile(r"\b[A-Z]{1,2}\d[A-Z\d]?\s\d[A-Z]{2}\b\|\b\d{4}\s?[A-Z]{2}\b\|\b\d{4,6}(?:-\d{4})?\b\|\b[A-Z]\d[A-Z]\s\d[A-Z]\d\b") # UK · NL · numeric · CA
70		70
71		71
72	def country_code(text: str \| None) -> str \| None:	72	def country_code(text: str \| None) -> str \| None:

modified src/companyatlas/connectors/generic_html.py +103 −81

				`@@ -8,6 +8,7 @@ metadata, JSON-LD and links; this module adds surface-aware typed extraction:`
8		8
9	Structured data (JSON-LD / microdata) is used first, DOM heuristics second, and nothing is ever invented: no address, no country, no	9	Structured data (JSON-LD / microdata) is used first, DOM heuristics second, and nothing is ever invented: no address, no country, no
10	date that the page does not state. Every surface also yields `discovered` URLs (classified links) for the discovery feedback loop.	10	date that the page does not state. Every surface also yields `discovered` URLs (classified links) for the discovery feedback loop.
		11	+Precision rules (`connectors/_precision.py`) reject navigation / CTA / cookie-consent / marketing noise at the point of extraction.
11	"""	12	"""
12	from __future__ import annotations	13	from __future__ import annotations
13		14
				`@@ -19,6 +20,22 @@ from urllib.parse import urlparse`
19	from selectolax.lexbor import LexborHTMLParser	20	from selectolax.lexbor import LexborHTMLParser
20		21
21	from companyatlas.config import settings	22	from companyatlas.config import settings
		23	+from companyatlas.connectors._precision import (
		24	+ STREET_RE,
		25	+ is_known_city,
		26	+ looks_like_person_name,
		27	+ looks_like_role_title,
		28	+ news_verdict,
		29	+ normalize_location,
		30	+ normalize_person,
		31	+ plan_name_ok,
		32	+ plan_verdict,
		33	+ price_text_from,
		34	+ product_verdict,
		35	+ refine_job,
		36	+ role_category,
		37	+ states_job_location,
		38	+)
22	from companyatlas.connectors._util import (	39	from companyatlas.connectors._util import (
23	country_code,	40	country_code,
24	date_from_text,	41	date_from_text,
				`@@ -57,52 +74,22 @@ PRODUCT_SURFACES = {Surface.PRODUCTS, Surface.SERVICES, Surface.SOLUTIONS}`
57		74
58	# ------------------------------------------------------------------------------------------------------------ people	75	# ------------------------------------------------------------------------------------------------------------ people
59		76
60	−ROLE_RULES: list[tuple[str, re.Pattern[str]]] = [	77	+# Role vocabulary, name shape and title cleaning live in `_precision` (shared with the pipeline); re-exported here for callers/tests.
61	− ("founder", re.compile(r"\b(co-?founder\|founder\|fondat(eur\|rice)\|gründer(in)?)\b", re.IGNORECASE)),	78	+looks_like_name = looks_like_person_name
62	− ("ceo", re.compile(r"\b(chief executive( officer)?\|ceo\|pdg\|président-directeur\|geschäftsführer(in)?\|managing director\|directeur général)\b", re.IGNORECASE)),	79	+looks_like_title = looks_like_role_title
63	− ("cfo", re.compile(r"\b(chief financial( officer)?\|cfo\|finanzvorstand\|directeur financier)\b", re.IGNORECASE)),	80	+PERSON_CARD_SCAN_LINES = 4 # a person card states the name within its first lines
64	− ("cto", re.compile(r"\b(chief technology( officer)?\|chief technical( officer)?\|cto)\b", re.IGNORECASE)),	81	+PERSON_TITLE_WINDOW = 3 # …and the title within the next few
65	− ("coo", re.compile(r"\b(chief operating( officer)?\|coo)\b", re.IGNORECASE)),	82	+
66	− ("chair", re.compile(r"\b(chair(man\|woman\|person)?\|executive chair\|présidente? du conseil\|vorsitzende[rn]?)\b", re.IGNORECASE)),	83	+
67	− ("president", re.compile(r"\b(?<!vice )(?<!vice-)president(?! of)\|président(?!e? du conseil)\b", re.IGNORECASE)),	84	+def _card_person(lines: list[str]) -> tuple[str, str \| None] \| None:
68	− ("board", re.compile(r"\b(board member\|member of the (supervisory \|advisory )?board\|non-executive director\|independent director\|"	85	+ """(name, title) from a card's lines: the title may precede the name (swapped cards) or follow it; role-like lines win over prose."""
69	− r"administrat(eur\|rice)\|aufsichtsrat\|conseil d'administration\|director(?= \(board)\|trustee)\b", re.IGNORECASE)),	86	+ name_idx = next((i for i, ln in enumerate(lines[:PERSON_CARD_SCAN_LINES]) if looks_like_name(ln)), None)
70	− ("vp", re.compile(r"\b(vice[- ]president\|vp\|evp\|svp\|avp)\b", re.IGNORECASE)),	87	+ if name_idx is None:
71	− ("head", re.compile(r"\b(head of\|head,\|general manager\|gm\|leiter(in)?\|directeur\|directrice\|director\|managing partner\|partner)\b", re.IGNORECASE)),	88	+ return None
72	−]	89	+ before = [x for x in lines[:name_idx] if looks_like_title(x)]
73	−CHIEF_RE = re.compile(r"\b(chief\b\|c[a-z]{1,3}o\b)", re.IGNORECASE)	90	+ after = [x for x in lines[name_idx + 1:name_idx + 1 + PERSON_TITLE_WINDOW] if not looks_like_name(x)]
74	−EXEC_CATEGORIES = {"ceo", "cfo", "cto", "coo", "founder", "president", "chair"}	91	+ title = (before[-1] if before else None) or next((x for x in after if looks_like_title(x)), None) or (after[0] if after else None)
75	−NAME_TOKEN = r"(?:[A-ZÀ-ÝĀ-Ž](?:['’][A-ZÀ-ÝĀ-Ž])?[a-zà-ÿā-ž'’]+(?:-[A-ZÀ-ÝĀ-Ža-zà-ÿā-ž][a-zà-ÿā-ž'’])\|[A-ZÀ-ÝĀ-Ž]\.\|(?:de\|van\|von\|der\|den\|da\|di\|le\|la\|du\|del\|bin\|al)\b)"	92	+ return lines[name_idx], title
76	−NAME_RE = re.compile(rf"^(?:(?:Dr\|Prof\|Mr\|Mrs\|Ms\|Sir\|Dame\|Hon)\.?\s+)?{NAME_TOKEN}(?:\s+{NAME_TOKEN}){{1,4}}(?:,?\s+(?:Jr\.?\|Sr\.?\|II\|III\|IV\|PhD\|MD\|MBA\|CPA\|Esq\.?))?$")
77	−NOT_NAME_RE = re.compile(r"\b(team\|our\|meet\|leadership\|board\|executive\|management\|contact\|about\|officer\|director\|president\|manager\|head\|chief\|"
78	− r"founder\|partner\|group\|company\|global\|senior\|vice\|read\|more\|learn\|view\|profile\|bio\|linkedin\|email\|join\|careers\|"
79	− r"news\|press\|the\|and\|of\|for\|at\|in\|on\|to\|with)\b", re.IGNORECASE)
80	−
81	−
82	−def role_category(title: str \| None) -> tuple[str, bool]:
83	− if not title:
84	− return "other", False
85	− for cat, pat in ROLE_RULES:
86	− if pat.search(title):
87	− return cat, cat in EXEC_CATEGORIES or bool(CHIEF_RE.search(title))
88	− return "other", bool(CHIEF_RE.search(title))
89	−
90	−
91	−def looks_like_name(text: str) -> bool:
92	− t = normalize_whitespace(text).strip(" .,:;-–—\|")
93	− if not (4 <= len(t) <= 60) or any(ch.isdigit() for ch in t):
94	− return False
95	− if NOT_NAME_RE.search(t):
96	− return False
97	− return bool(NAME_RE.match(t))
98	−
99	−
100	−def looks_like_title(text: str) -> bool:
101	− t = normalize_whitespace(text)
102	− if not (2 <= len(t) <= 140):
103	− return False
104	− return role_category(t)[0] != "other" or bool(re.search(r"\b(officer\|manager\|engineer\|lead\|counsel\|scientist\|architect\|analyst\|controller\|"
105	− r"secretary\|treasurer\|advisor\|adviser\|strategist\|evangelist\|designer)\b", t, re.IGNORECASE))
106		93
107		94
108	def extract_people(page: NormalizedPage) -> list[ExtractedPerson]:	95	def extract_people(page: NormalizedPage) -> list[ExtractedPerson]:
				`@@ -110,27 +97,28 @@ def extract_people(page: NormalizedPage) -> list[ExtractedPerson]:`
110	seen: set[str] = set()	97	seen: set[str] = set()
111		98
112	def add(name: str, title: str \| None, url: str \| None = None) -> None:	99	def add(name: str, title: str \| None, url: str \| None = None) -> None:
113	− name = normalize_whitespace(name).strip(" .,:;-–—\|")	100	+ fixed = normalize_person(name, title)
		101	+ if fixed is None:
		102	+ return
		103	+ name, title = fixed
114	key = norm_name(name)	104	key = norm_name(name)
115	if not key or key in seen or len(out) >= MAX_PEOPLE:	105	if not key or key in seen or len(out) >= MAX_PEOPLE:
116	return	106	return
117	seen.add(key)	107	seen.add(key)
118	cat, is_exec = role_category(title)	108	cat, is_exec = role_category(title)
119	− out.append(ExtractedPerson(name=name[:120], title=(normalize_whitespace(title)[:160] if title else None), role_category=cat, is_executive=is_exec, url=url))	109	+ out.append(ExtractedPerson(name=name[:120], title=(title[:160] if title else None), role_category=cat, is_executive=is_exec, url=url))
120		110
121	for p in page.jsonld.get("persons", []) + page.microdata.get("persons", []):	111	for p in page.jsonld.get("persons", []) + page.microdata.get("persons", []):
122	name = text_of(p.get("name"))	112	name = text_of(p.get("name"))
123	− if name and looks_like_name(name):	113	+ if name:
124	add(name, text_of(p.get("jobTitle")) or text_of(p.get("title")), text_of(p.get("url")) if isinstance(p.get("url"), str) else None)	114	add(name, text_of(p.get("jobTitle")) or text_of(p.get("title")), text_of(p.get("url")) if isinstance(p.get("url"), str) else None)
125	for b in page.blocks:	115	for b in page.blocks:
126	if b.kind != "person":	116	if b.kind != "person":
127	continue	117	continue
128	lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]	118	lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
129	− for i, ln in enumerate(lines[:4]):	119	+ found = _card_person(lines)
130	− if looks_like_name(ln):	120	+ if found is not None:
131	− title = next((x for x in lines[i + 1:i + 4] if not looks_like_name(x) and len(x) <= 140), None)	121	+ add(found[0], found[1], b.attrs.get("href"))
132	− add(ln, title, b.attrs.get("href"))
133	− break
134	if len(out) < 2: # fallback: heading = name, next short block = title	122	if len(out) < 2: # fallback: heading = name, next short block = title
135	blocks = page.blocks	123	blocks = page.blocks
136	for i, b in enumerate(blocks):	124	for i, b in enumerate(blocks):
				`@@ -167,11 +155,11 @@ def parse_price(text: str) -> dict[str, Any] \| None:`
167	"""→ {price, currency, billing_period, unit, price_text, contact_sales} or None when the text states no price."""	155	"""→ {price, currency, billing_period, unit, price_text, contact_sales} or None when the text states no price."""
168	t = normalize_whitespace(text)	156	t = normalize_whitespace(text)
169	if CONTACT_RE.search(t) and not PRICE_RE.search(t):	157	if CONTACT_RE.search(t) and not PRICE_RE.search(t):
170	− return {"price": None, "currency": None, "billing_period": "contact", "unit": None, "price_text": t[:80], "contact_sales": True}	158	+ return {"price": None, "currency": None, "billing_period": "contact", "unit": None, "price_text": price_text_from(t) or t[:80], "contact_sales": True}
171	m = PRICE_RE.search(t)	159	m = PRICE_RE.search(t)
172	if m is None:	160	if m is None:
173	if FREE_RE.match(t):	161	if FREE_RE.match(t):
174	− return {"price": 0.0, "currency": None, "billing_period": None, "unit": None, "price_text": t[:80], "contact_sales": False}	162	+ return {"price": 0.0, "currency": None, "billing_period": None, "unit": None, "price_text": price_text_from(t) or t[:80], "contact_sales": False}
175	return None	163	return None
176	cur = m.group("cur") or m.group("cur2")	164	cur = m.group("cur") or m.group("cur2")
177	amt = m.group("amt") or m.group("amt2")	165	amt = m.group("amt") or m.group("amt2")
				`@@ -213,10 +201,28 @@ def parse_price(text: str) -> dict[str, Any] \| None:`
213	elif (u2 in YEAR_UNITS) and not period:	201	elif (u2 in YEAR_UNITS) and not period:
214	period = "year"	202	period = "year"
215	return {"price": price, "currency": CURRENCY_SYMBOLS.get(cur, cur.upper() if cur and cur.isalpha() else None), "billing_period": period, "unit": unit,	203	return {"price": price, "currency": CURRENCY_SYMBOLS.get(cur, cur.upper() if cur and cur.isalpha() else None), "billing_period": period, "unit": unit,
216	− "price_text": t[max(0, m.start() - 12):m.end() + 40].strip()[:80], "contact_sales": bool(CONTACT_RE.search(t))}	204	+ "price_text": price_text_from(t) or t[m.start():m.end()].strip()[:80], "contact_sales": bool(CONTACT_RE.search(t))}
217		205
218		206
219	−def _plan_from_lines(name: str, lines: list[str]) -> ExtractedPlan \| None:	207	+PLAN_NAME_SCAN_LINES = 3 # the plan name is one of the first lines before the price (eyebrows / marketing headings are skipped)
		208	+
		209	+
		210	+def _plan_name_from(lines: list[str], path: str) -> tuple[str \| None, list[str]]:
		211	+ """(plan name, remaining lines): the first valid short label before the price line (eyebrows such as "Most popular" are skipped).
		212	+ Otherwise the heading just above the card (its path tail) names the tier — unless that heading is generic ("Plans", "Pricing") or a
		213	+ marketing sentence, in which case the card yields no plan."""
		214	+ for i, ln in enumerate(lines[:PLAN_NAME_SCAN_LINES]):
		215	+ if plan_name_ok(ln): # "Free" is a valid tier name even though it also reads as a price
		216	+ return ln, lines[i + 1:]
		217	+ if PRICE_RE.search(ln) or parse_price(ln) is not None:
		218	+ break
		219	+ tail = path.split(" > ")[-1] if path else ""
		220	+ return (tail if plan_name_ok(tail) else None), lines
		221	+
		222	+
		223	+def _plan_from_lines(name: str \| None, lines: list[str]) -> ExtractedPlan \| None:
		224	+ if not name:
		225	+ return None
220	price_info = None	226	price_info = None
221	features: list[str] = []	227	features: list[str] = []
222	for ln in lines:	228	for ln in lines:
				`@@ -227,6 +233,8 @@ def _plan_from_lines(name: str, lines: list[str]) -> ExtractedPlan \| None:`
227	continue	233	continue
228	if 2 <= len(ln) <= 140 and not parse_price(ln) and len(features) < 25:	234	if 2 <= len(ln) <= 140 and not parse_price(ln) and len(features) < 25:
229	features.append(ln)	235	features.append(ln)
		236	+ if price_info is None and FREE_RE.match(name):
		237	+ price_info = parse_price(name) # a tier literally called "Free" with no separate price line
230	if price_info is None:	238	if price_info is None:
231	return None	239	return None
232	return ExtractedPlan(plan_name=name[:80], price=price_info["price"], price_text=price_info["price_text"], currency=price_info["currency"],	240	return ExtractedPlan(plan_name=name[:80], price=price_info["price"], price_text=price_info["price_text"], currency=price_info["currency"],
				`@@ -238,7 +246,7 @@ def extract_plans(page: NormalizedPage) -> list[ExtractedPlan]:`
238	seen: set[str] = set()	246	seen: set[str] = set()
239		247
240	def add(plan: ExtractedPlan \| None) -> None:	248	def add(plan: ExtractedPlan \| None) -> None:
241	− if plan is None:	249	+ if plan is None or not plan_verdict(plan).ok:
242	return	250	return
243	key = norm_name(plan.plan_name)	251	key = norm_name(plan.plan_name)
244	if not key or key in seen or len(out) >= MAX_PLANS:	252	if not key or key in seen or len(out) >= MAX_PLANS:
				`@@ -252,14 +260,14 @@ def extract_plans(page: NormalizedPage) -> list[ExtractedPlan]:`
252	lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]	260	lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
253	if not lines:	261	if not lines:
254	continue	262	continue
255	− name = lines[0] if parse_price(lines[0]) is None and len(lines[0]) <= 60 else (b.path.split(" > ")[-1] if b.path else "Plan")	263	+ name, rest = _plan_name_from(lines, b.path)
256	− add(_plan_from_lines(name, lines[1:] if lines[0] == name else lines))	264	+ add(_plan_from_lines(name, rest))
257	if not out: # fallback: heading followed by a price before the next heading (common in hand-rolled pricing tables)	265	if not out: # fallback: heading followed by a price before the next heading (common in hand-rolled pricing tables)
258	blocks = page.blocks	266	blocks = page.blocks
259	i = 0	267	i = 0
260	while i < len(blocks):	268	while i < len(blocks):
261	b = blocks[i]	269	b = blocks[i]
262	− if b.kind == "heading" and int(b.attrs.get("level", 2)) >= 2 and len(b.text) <= 60:	270	+ if b.kind == "heading" and int(b.attrs.get("level", 2)) >= 2 and plan_name_ok(b.text):
263	j = i + 1	271	j = i + 1
264	lines: list[str] = []	272	lines: list[str] = []
265	while j < len(blocks) and blocks[j].kind != "heading" and j - i <= 10:	273	while j < len(blocks) and blocks[j].kind != "heading" and j - i <= 10:
				`@@ -288,10 +296,8 @@ LOCATION_KIND_RULES: list[tuple[str, re.Pattern[str]]] = [`
288	("data_center", re.compile(r"\b(data ?cent(er\|re)\|datacenter\|server farm)\b", re.IGNORECASE)),	296	("data_center", re.compile(r"\b(data ?cent(er\|re)\|datacenter\|server farm)\b", re.IGNORECASE)),
289	("store", re.compile(r"\b(store\|shop\|boutique\|showroom\|outlet\|dealer(ship)?\|branch\|agence\|filiale)\b", re.IGNORECASE)),	297	("store", re.compile(r"\b(store\|shop\|boutique\|showroom\|outlet\|dealer(ship)?\|branch\|agence\|filiale)\b", re.IGNORECASE)),
290	]	298	]
291	−STREET_RE = re.compile(r"\b\d{1,5}[a-z]?\s+[^\n,]{2,40}\b(street\|st\.?\|avenue\|ave\.?\|road\|rd\.?\|boulevard\|blvd\.?\|drive\|dr\.?\|lane\|ln\.?\|way\|place\|pl\.?\|"
292	− r"square\|plaza\|court\|ct\.?\|parkway\|highway\|straße\|strasse\|str\.\|allee\|platz\|weg\|rue\|avenue\|boulevard\|avenida\|calle\|via\|piazza)\b"
293	− r"\|\b(rue\|avenue\|boulevard\|via\|calle\|avenida)\s+[^\n,]{2,40}\s\d{1,5}\b", re.IGNORECASE)
294	CITY_COUNTRY_RE = re.compile(r"^([A-ZÀ-Ý][\w'’.\- ]{1,40}),\s*([A-Za-zÀ-ÿ .]{2,40})$")	299	CITY_COUNTRY_RE = re.compile(r"^([A-ZÀ-Ý][\w'’.\- ]{1,40}),\s*([A-Za-zÀ-ÿ .]{2,40})$")
		300	+MAX_VENUE_LINE_CHARS = 60 # "Hormuz Grand Hotel" — a venue line under a country heading
295		301
296		302
297	def _location_from_lines(name: str, lines: list[str], href: str \| None = None) -> ExtractedLocation \| None:	303	def _location_from_lines(name: str, lines: list[str], href: str \| None = None) -> ExtractedLocation \| None:
				`@@ -305,8 +311,10 @@ def _location_from_lines(name: str, lines: list[str], href: str \| None = None) -`
305	probe = (ln[:street.start()] + " " + ln[street.end():]).strip(" ,") if street else ln	311	probe = (ln[:street.start()] + " " + ln[street.end():]).strip(" ,") if street else ln
306	if country is None and probe:	312	if country is None and probe:
307	loc = parse_location(probe)	313	loc = parse_location(probe)
308	− if loc["country"] or loc["region"]:	314	+ if loc["country"]: # the line that states the country is the authoritative "City, Country" line
309	− city, region, country = city or loc["city"], loc["region"], loc["country"]	315	+ city, region, country = loc["city"] or city, loc["region"], loc["country"]
		316	+ elif loc["region"]:
		317	+ city, region = city or loc["city"], loc["region"]
310	if country is None:	318	if country is None:
311	c = country_code(name)	319	c = country_code(name)
312	if c:	320	if c:
				`@@ -315,7 +323,15 @@ def _location_from_lines(name: str, lines: list[str], href: str \| None = None) -`
315	m = CITY_COUNTRY_RE.match(name)	323	m = CITY_COUNTRY_RE.match(name)
316	if m and country_code(m.group(2)):	324	if m and country_code(m.group(2)):
317	city, country = m.group(1).strip(), country_code(m.group(2))	325	city, country = m.group(1).strip(), country_code(m.group(2))
318	− if city is None and country is not None and looks_like_city(name):	326	+ if country_code(name) and city is None:
		327	+ # heading is a country ("Oman") — the first short line that is neither an address nor a place is the venue / office name
		328	+ venue = next((ln for ln in lines if ln != address and len(ln) <= MAX_VENUE_LINE_CHARS and not parse_location(ln)["country"]
		329	+ and not country_code(ln) and not any(ch.isdigit() for ch in ln)), None)
		330	+ if venue:
		331	+ name = venue
		332	+ elif is_known_city(name):
		333	+ city = name # city-states: Singapore, Monaco, Hong Kong
		334	+ elif city is None and country is not None and looks_like_city(name):
319	city = name	335	city = name
320	if city is None and country is None and address is None and kind == "office":	336	if city is None and country is None and address is None and kind == "office":
321	return None	337	return None
				`@@ -332,6 +348,7 @@ def extract_locations(page: NormalizedPage) -> list[ExtractedLocation]:`
332	seen: set[str] = set()	348	seen: set[str] = set()
333		349
334	def add(loc: ExtractedLocation \| None) -> None:	350	def add(loc: ExtractedLocation \| None) -> None:
		351	+ loc = normalize_location(loc) if loc is not None else None
335	if loc is None:	352	if loc is None:
336	return	353	return
337	key = norm_name(loc.name)	354	key = norm_name(loc.name)
				`@@ -371,6 +388,7 @@ def extract_locations(page: NormalizedPage) -> list[ExtractedLocation]:`
371	# ------------------------------------------------------------------------------------------------------------ news	388	# ------------------------------------------------------------------------------------------------------------ news
372		389
373	NEWS_NOISE_ANCHOR = re.compile(r"^(read more\|learn more\|more\|continue reading\|view all\|see all\|all news\|all posts\|next\|previous\|older\|newer\|\d+)$", re.IGNORECASE)	390	NEWS_NOISE_ANCHOR = re.compile(r"^(read more\|learn more\|more\|continue reading\|view all\|see all\|all news\|all posts\|next\|previous\|older\|newer\|\d+)$", re.IGNORECASE)
		391	+MIN_NEWS_TITLE_CHARS = 3 # anything shorter is a glyph or a counter; the precision rule decides the rest (words / date)
374		392
375		393
376	def _time_index(html: str, base_url: str) -> dict[str, Any]:	394	def _time_index(html: str, base_url: str) -> dict[str, Any]:
				`@@ -407,14 +425,16 @@ def extract_news(page: NormalizedPage, html: str, *, surface: str, base_url: str`
407		425
408	def add(title: str, url: str, published: Any, summary: str \| None = None) -> None:	426	def add(title: str, url: str, published: Any, summary: str \| None = None) -> None:
409	title = normalize_whitespace(title)	427	title = normalize_whitespace(title)
410	− if len(title) < 8 or NEWS_NOISE_ANCHOR.match(title) or len(out) >= MAX_NEWS:	428	+ if len(title) < MIN_NEWS_TITLE_CHARS or NEWS_NOISE_ANCHOR.match(title) or len(out) >= MAX_NEWS:
411	return	429	return
412	canon = canonicalize_url(url)	430	canon = canonicalize_url(url)
413	if canon in seen or canon == canonicalize_url(base_url) or is_static_asset(url):	431	if canon in seen or canon == canonicalize_url(base_url) or is_static_asset(url):
414	return	432	return
		433	+ published = published or times.get(canon)
		434	+ if not news_verdict(title, url=url, published_at=published).ok:
		435	+ return
415	seen.add(canon)	436	seen.add(canon)
416	− out.append(ExtractedNewsItem(title=title[:300], url=url, published_at=published or times.get(canon), summary=(summary or None), category=category,	437	+ out.append(ExtractedNewsItem(title=title[:300], url=url, published_at=published, summary=(summary or None), category=category, language=page.lang))
417	− language=page.lang))
418		438
419	for a in page.jsonld.get("articles", []) + page.microdata.get("articles", []):	439	for a in page.jsonld.get("articles", []) + page.microdata.get("articles", []):
420	title, url = text_of(a.get("headline")) or text_of(a.get("name")), a.get("url") or (a.get("mainEntityOfPage") if isinstance(a.get("mainEntityOfPage"), str) else None)	440	title, url = text_of(a.get("headline")) or text_of(a.get("name")), a.get("url") or (a.get("mainEntityOfPage") if isinstance(a.get("mainEntityOfPage"), str) else None)
				`@@ -425,8 +445,9 @@ def extract_news(page: NormalizedPage, html: str, *, surface: str, base_url: str`
425	if b.kind != "news_item" or not href:	445	if b.kind != "news_item" or not href:
426	continue	446	continue
427	lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]	447	lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
428	− title = next((ln for ln in lines if len(ln) >= 12 and not (len(ln) <= 32 and date_from_text(ln))), lines[0] if lines else "")	448	+ undated = [ln for ln in lines if not (len(ln) <= 32 and date_from_text(ln))]
429	− rest = [ln for ln in lines if ln != title and not (len(ln) <= 32 and date_from_text(ln))]	449	+ title = next((ln for ln in undated if len(ln) >= 12), undated[0] if undated else "")
		450	+ rest = [ln for ln in undated if ln != title]
430	add(title, href, date_from_text(b.text) or times.get(canonicalize_url(href)) or date_from_url(href), " ".join(rest[:2])[:300] or None)	451	add(title, href, date_from_text(b.text) or times.get(canonicalize_url(href)) or date_from_url(href), " ".join(rest[:2])[:300] or None)
431	if len(out) < 3: # fallback: article-like links in the main region (same site, deeper path, long anchor)	452	if len(out) < 3: # fallback: article-like links in the main region (same site, deeper path, long anchor)
432	base_depth = urlparse(base_url).path.strip("/").count("/")	453	base_depth = urlparse(base_url).path.strip("/").count("/")
				`@@ -451,11 +472,7 @@ JOB_NOISE_ANCHOR = re.compile(r"^(apply( now)?\|view( all)?( jobs\| openings\| role`
451	r"browse jobs\|explore\|more\|back\|home\|filter\|next\|previous\|\d+)$", re.IGNORECASE)	472	r"browse jobs\|explore\|more\|back\|home\|filter\|next\|previous\|\d+)$", re.IGNORECASE)
452		473
453		474
454	−def _states_location(text: str) -> bool:	475	+_states_location = states_job_location
455	− if len(text) > 80:
456	− return False
457	− p = parse_location(text)
458	− return bool(p["city"] or p["country"] or p["remote"] or p["region"])
459		476
460		477
461	def extract_jobs_html(page: NormalizedPage, *, base_url: str) -> list[ExtractedJob]:	478	def extract_jobs_html(page: NormalizedPage, *, base_url: str) -> list[ExtractedJob]:
				`@@ -466,11 +483,14 @@ def extract_jobs_html(page: NormalizedPage, *, base_url: str) -> list[ExtractedJ`
466	title = normalize_whitespace(title).strip(" -–—\|·")	483	title = normalize_whitespace(title).strip(" -–—\|·")
467	if not (4 <= len(title) <= 140) or JOB_NOISE_ANCHOR.match(title) or len(jobs) >= MAX_JOBS:	484	if not (4 <= len(title) <= 140) or JOB_NOISE_ANCHOR.match(title) or len(jobs) >= MAX_JOBS:
468	return	485	return
469	− key = (canonicalize_url(url) if url else "") + "\|" + title.lower()	486	+ job = refine_job(ExtractedJob(title=title, url=url, location_text=location, department=department))
		487	+ if job is None:
		488	+ return
		489	+ key = (canonicalize_url(url) if url else "") + "\|" + job.title.lower()
470	if key in seen:	490	if key in seen:
471	return	491	return
472	seen.add(key)	492	seen.add(key)
473	− jobs.append(finish_job(ExtractedJob(title=title, url=url, location_text=location, department=department)))	493	+ jobs.append(finish_job(job))
474		494
475	jl_jobs = jobs_from_jsonld(list(page.jsonld.get("job_postings", [])) + list(page.microdata.get("job_postings", [])), page_url=base_url)	495	jl_jobs = jobs_from_jsonld(list(page.jsonld.get("job_postings", [])) + list(page.microdata.get("job_postings", [])), page_url=base_url)
476	for j in jl_jobs:	496	for j in jl_jobs:
				`@@ -488,7 +508,7 @@ def extract_jobs_html(page: NormalizedPage, *, base_url: str) -> list[ExtractedJ`
488	if not lines:	508	if not lines:
489	continue	509	continue
490	title = lines[0]	510	title = lines[0]
491	− loc = next((ln for ln in lines[1:6] if _states_location(ln)), None)	511	+ loc = next((ln for ln in lines[1:6] if _states_location(ln) or is_known_city(ln)), None)
492	dept = next((ln for ln in lines[1:6] if ln != loc and 2 <= len(ln) <= 40 and not _states_location(ln) and not date_from_text(ln)), None)	512	dept = next((ln for ln in lines[1:6] if ln != loc and 2 <= len(ln) <= 40 and not _states_location(ln) and not date_from_text(ln)), None)
493	before = len(jobs)	513	before = len(jobs)
494	add(title, href, loc, dept)	514	add(title, href, loc, dept)
				`@@ -517,6 +537,8 @@ def extract_products(page: NormalizedPage) -> list[ExtractedProduct]:`
517	key = norm_name(name)	537	key = norm_name(name)
518	if not (2 <= len(name) <= 90) or not key or key in seen or len(out) >= MAX_PRODUCTS or NEWS_NOISE_ANCHOR.match(name):	538	if not (2 <= len(name) <= 90) or not key or key in seen or len(out) >= MAX_PRODUCTS or NEWS_NOISE_ANCHOR.match(name):
519	return	539	return
		540	+ if not product_verdict(name).ok:
		541	+ return
520	seen.add(key)	542	seen.add(key)
521	out.append(ExtractedProduct(name=name, url=url, category=category, description=(description or None)))	543	out.append(ExtractedProduct(name=name, url=url, category=category, description=(description or None)))
522		544

modified src/companyatlas/sdk/normalize.py +57 −12

Large diff (140 lines) is not loaded by default to keep the page fast.Load diff

modified src/companyatlas/services/pipeline.py +9 −3

Large diff (36 lines) is not loaded by default to keep the page fast.Load diff

added tests/test_extraction_precision.py +299 −0

Large diff (300 lines) is not loaded by default to keep the page fast.Load diff

spb/company-atlas

extraction precision: multilingual validators for jobs/people/locations/plans/products/news, extractor fixes, fixtures, audit script

Theme