crawl core: normalize/diff SDK, 15 connector families, discovery, pipeline, scheduler, repair, fixtures and tests

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 12 days ago (Sep 12, 2026) parent 8707c99

59 changed files +12,857 −53

added docs/CONNECTORS.md +70 −0

		`@@ -0,0 +1,70 @@`
	1	+# Connectors — families, SDK, fixtures, verified vendor endpoints
	2	+
	3	+A connector is a strategy for one kind of public source; a sensor binds a connector to one URL for one company
	4	+(spec §8–9, §105–106). All connectors live in `src/companyatlas/connectors/`, register themselves with the SDK
	5	+(`sdk/connector.py`) and are auto-loaded (`connectors.load_all()`); `catlas connectors` lists them and upserts the `connectors` table.
	6	+
	7	+## Families (as of `connector-v1` ids)
	8	+
	9	+\| id \| family \| picks when \| output \|
	10	+\|---\|---\|---\|---\|
	11	+\| `generic-html-v1` \| any HTML surface \| fallback for every surface (`surfaces=("*",)`) \| text + semantic blocks + surface-typed lists: people (leadership/about), plans (pricing), locations (locations/contact), news (newsroom/blog/changelog/research/IR), jobs (careers, list/table/card + JSON-LD), products (products/services/solutions); `discovered` links on every page \|
	12	+\| `sitemap-v1` \| `sitemap*.xml(.gz)`, `sitemap_index.xml`, `/sitemaps/` \| URL pattern \| URL set (bounded by `discovery_max_sitemap_urls`), classified `DiscoveredUrl`s, one block per URL (diff = new/gone URLs) \|
	13	+\| `feed-v1` \| RSS / Atom / JSON Feed \| URL pattern (`/feed`, `/rss`, `.xml`, `feed.json`) or surface `feed` \| news items (title, url, published_at, summary, language) \|
	14	+\| `jsonld-jobs-v1` \| JobPosting JSON-LD / microdata \| non-ATS `jobs_board` URLs (or explicitly by discovery) \| jobs \|
	15	+\| `greenhouse-v1` `lever-v1` `ashby-v1` `smartrecruiters-v1` `workable-v1` `workday-v1` `recruitee-v1` `personio-v1` `teamtailor-v1` \| structured job boards \| API URL pattern (set by discovery via `connectors/_util.ats_sensor_spec`) \| jobs, one stable block per job \|
	16	+\| `statuspage-v1` \| Atlassian Statuspage \| `/api/v2/summary.json` \| components / incidents as blocks + incident news \|
	17	+
	18	+Connector selection: `connector.for_surface(surface, url)` scores every registered connector (`url_pattern` match +100, `category`
	19	++20, listed `surfaces` +10, `"*"` +5, plus `priority`) and returns the best; the generic HTML connector is the floor.
	20	+
	21	+## Verified vendor endpoints (2026-09-12, one real board each, then saved as trimmed fixtures — tests never hit the network)
	22	+
	23	+\| vendor \| endpoint \| verified board \| fixture \|
	24	+\|---\|---\|---\|---\|
	25	+\| Greenhouse \| `GET https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=false` (+`/departments`, `/offices` optional) \| `stripe` \| `greenhouse/stripe_jobs.json` \|
	26	+\| Lever \| `GET https://api.lever.co/v0/postings/{token}?mode=json` (`api.eu.lever.co` for EU boards) \| `palantir` \| `lever/palantir_postings.json` \|
	27	+\| Ashby \| `GET https://api.ashbyhq.com/posting-api/job-board/{token}` \| `ashby` \| `ashby/ashby_board.json` \|
	28	+\| SmartRecruiters \| `GET https://api.smartrecruiters.com/v1/companies/{token}/postings?limit=100&offset=N` (offset pagination, ≤ 10 pages) \| `smartrecruiters` \| `smartrecruiters/smartrecruiters_postings.json` \|
	29	+\| Workable \| `GET https://apply.workable.com/api/v1/widget/accounts/{token}` \| `epignosis` \| `workable/epignosis_widget.json` \|
	30	+\| Workday \| `POST https://{tenant}.{wdN}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs` body `{"appliedFacets":{},"limit":20,"offset":N,"searchText":""}` (bounded 400 jobs) — uses `Fetcher.post_json` (SSRF-checked, governor) \| `nvidia.wd5 / NVIDIAExternalCareerSite` \| `workday/nvidia_jobs_page1.json` \|
	31	+\| Recruitee \| `GET https://{token}.recruitee.com/api/offers/` \| `vandebron` \| `recruitee/vandebron_offers.json` \|
	32	+\| Personio \| `GET https://{token}.jobs.personio.de/xml` (`<workzag-jobs>`) \| `personio` \| `personio/personio_jobs.xml` \|
	33	+\| Teamtailor \| `GET https://{career-site-host}/jobs.json` (JSON Feed 1.1 with `_jobposting`) — `{token}.teamtailor.com` boards that 404 keep the HTML careers sensor \| `career.teamtailor.com` \| `teamtailor/teamtailor_jobs_feed.json` \|
	34	+\| Statuspage \| `GET https://status.<host>/api/v2/summary.json` \| `githubstatus.com` \| `statuspage/github_summary.json` \|
	35	+
	36	+All of these are the endpoints the vendors' own public pages call (spec §13 mode C). Nothing needs a key; nothing bypasses a
	37	+challenge. Every request still goes through `fetch.Fetcher` (SSRF guard, robots.txt, per-domain governor, size caps).
	38	+
	39	+## Job normalisation (every board → `ExtractedJob`)
	40	+
	41	+`connectors/_util.finish_job` fills derived fields without overriding vendor data: `parse_location` (city / region / ISO-2
	42	+country only when stated — "San Francisco, CA" keeps region `CA` and no country because `CA` is also Canada), remote /
	43	+hybrid, `seniority_guess`, `employment_type_norm`. The pipeline sets `is_ai` (`taxonomy.AI_KEYWORDS`) and `is_engineering`.
	44	+Job identity = `job_fingerprint(title, location_text, external_id \| url)`; structured boards also give each job a stable block
	45	+key so the block diff mirrors the job delta.
	46	+
	47	+## Adding a connector family
	48	+
	49	+1. Create `src/companyatlas/connectors/<family>.py`; subclass `Connector` (or `AtsConnector` for JSON/XML boards — implement
	50	+ `parse_jobs(data, sensor)` only) and decorate with `@register`. Declare `ConnectorMeta(connector_id="<family>-v1", category=Surface…,
	51	+ fetch_mode, url_pattern, priority, accept, default_interval_s)`. Override `fetch()` only for pagination / POST; never open
	52	+ sockets — use the `Fetcher` you receive (`get`, `request`, `post_json`).
	53	+2. Verify the endpoint shape against one real public source (a handful of polite requests), then save the response trimmed to
	54	+ ~20 items under `fixtures/connectors/<family>/` (strip descriptions / personal data).
	55	+3. Add tests in `tests/test_connectors_*.py` that load the fixture with `fetch.file_result` and assert counts, ids, urls, locations.
	56	+ Tests must not touch the network (`-m live` is opt-in only).
	57	+4. If discovery should create sensors for it, extend `connectors/_util.ats_sensor_spec` (vendor → API URL + connector id + config)
	58	+ and the detection regexes in `urls.ATS_PATTERNS` / `services/discovery.EMBED_ATS_RE`.
	59	+5. Behaviour change that alters extraction materially → new id (`<family>-v2`). Observations and snapshots store the id as
	60	+ `connector_version`, so history stays reproducible; `connector.get("<family>-v1")` keeps resolving to the newest of the family.
	61	+
	62	+## Fixture policy
	63	+
	64	+* Hand-written HTML for each generic surface lives in `fixtures/connectors/generic_html/` (`pricing.html` + `pricing_v2.html`,
	65	+ `careers.html` + `careers_v2.html`, `leadership.html`, `locations.html`, `newsroom.html`, `homepage.html`, `legal_terms.html`,
	66	+ `careers_jsonld.html`); `sitemap/` and `feed/` hold XML samples. Pairs (`*_v2`) exist so the diff engine and the pipeline can be
	67	+ tested for real change semantics (price change, jobs added/removed).
	68	+* Vendor fixtures are captured responses, trimmed; keep the top-level shape intact (pagination fields included).
	69	+* Regression rule (spec §148): when changing the normalizer, the diff engine or an extractor, run the suite — the fixtures are
	70	+ the historical behaviour contract.

added docs/CRAWL.md +105 −0

		`@@ -0,0 +1,105 @@`
	1	+# Crawl core — discovery, pipeline states, scheduling, failure policies, repair
	2	+
	3	+```
	4	+companies (pending) ─▶ discovery.discover_company ─▶ sensors (pending, staggered next_run_at)
	5	+scheduler tick ─▶ claim due sensors (SKIP LOCKED) ─▶ pipeline.run_sensor ─▶ observation ─▶ snapshot ─▶ change (pending) ─▶ intelligence
	6	+```
	7	+
	8	+## Discovery (`services/discovery.py`, `catlas discover\|onboard`)
	9	+
	10	+Per company, within a ~90 s budget and ≤ ~25 requests: homepage (website variants, redirects; off-domain redirect → `domains`
	11	+row `kind=redirect`, canonical domain updated only when unambiguous and unique) → generic-html extraction of the homepage
	12	+(nav/footer/main links classified by `urls.classify_url`, `link rel=alternate` feeds, JSON-LD `sameAs`) → `robots.txt`
	13	+`Sitemap:` lines (else `/sitemap.xml`; index + ≤ 4 children, bounded by `discovery_max_sitemap_urls`) → ATS detection (links,
	14	+embedded `boards.greenhouse.io/embed/job_board?for=…`, `jobs.lever.co/…`, Ashby, Workable, Workday, Recruitee, Personio,
	15	+Teamtailor, SmartRecruiters in the homepage + best careers page; the API endpoint is verified with one request) → ≤ 12
	16	+common-path probes for missing high-value surfaces (soft-404 and redirect-to-home rejected) → subdomain probes
	17	+(`careers. jobs. news. blog. docs. developer(s). status. investors. ir. shop.` — DNS first, ≤ 6 GETs; `status.` tries
	18	+`/api/v2/summary.json`).
	19	+
	20	+Selection: one sensor per surface (best `confidence × method weight × verified × depth`), plus `jobs_board` next to `careers`
	21	+and `feed` next to `blog`/`newsroom`, capped by `discovery_max_sensors_per_company` in surface-importance order.
	22	+Sensor seed values: `discovery_confidence`, `discovery_method` (nav \| link \| sitemap \| robots \| probe \| subdomain \| ats \| feed),
	23	+`quality_score = 100 × confidence × (0.5 + 0.5 × importance) × reliability`, `base_interval_s = SURFACE_BASE_INTERVAL_S × tier
	24	+factor (1: ×0.5, 2: ×0.75, 3: ×1, 4: ×1.5)` clamped to `[min_interval_s, max_interval_s]`, `tier` letter from
	25	+`tier_for_interval`, `next_run_at` random within the first interval (or now with `--fetch-now`), `priority = 0.3 + 0.5 ×
	26	+company importance + 0.2 × surface importance`. Company → `onboarding_status` active / failed (+`onboarding_error`) /
	27	+no_website; `companies.stats.discovery` and `source_meta.same_as` filled.
	28	+
	29	+`onboard_pending` claims `queue_jobs(kind='discover')` with `FOR UPDATE SKIP LOCKED` (attempts / dead after `max_attempts`,
	30	+30 min × attempts backoff) and then `companies.onboarding_status='pending'` (marked `discovering` while claimed).
	31	+
	32	+## Pipeline states (`services/pipeline.py`, `catlas run-sensor`)
	33	+
	34	+\| outcome \| rows \| sensor \|
	35	+\|---\|---\|---\|
	36	+\| `not_modified` (HTTP 304 on etag / last-modified) \| observation `not_modified=true` \| `consecutive_unchanged++`, interval grows \|
	37	+\| `unchanged` (same `normalized_hash` and `structured_hash`) \| observation `changed=false` \| same as above, validators refreshed \|
	38	+\| `ok` (first snapshot) \| observation, snapshot v1, entities inserted (first_seen) \| baseline, no change row \|
	39	+\| `changed` \| observation, snapshot vN (`previous_snapshot_id`), entity reconciliation, `changes` row \| counters, `last_change_at`, burst on meaningful+ \|
	40	+\| `failed` \| observation with `failure_class`, `failures` row \| policy backoff, status transitions, review item \|
	41	+\| `redirected` (final registrable domain ≠ sensor domain) \| observation `REDIRECT` \| `status=redirected`, `config.redirect_url`, `review_queue(sensor_migration)` \|
	42	+\| `skipped` \| — \| domain `blocked_until` / daily budget exhausted → rescheduled \|
	43	+
	44	+Objects: raw bytes → `archive.put_bytes` (`observations.object_key`), normalized text → `text_key`, blocks JSON → `blocks_key`;
	45	+`snapshots.extracted = Extraction.structured_payload()` bounded (300 items per list, ≤ 900 KB), `extracted_summary` counters.
	46	+
	47	+Change classification: `sdk.diff.compare(previous blocks, new blocks, surface, texts, structured_delta, history)` →
	48	+`significance` → `taxonomy.change_kind` with `settings.*_threshold` → `changes.kind`; `status='pending'` for meaningful / major /
	49	+critical (consumed by `services/events`), `'archived'` for minor. Pure noise (no typed delta, `keep_noise_snapshots=false`) writes
	50	+neither snapshot nor change but updates the sensor hashes so the same noise is not re-detected.
	51	+
	52	+Entity reconciliation (scoped to the sensor): jobs by fingerprint (insert / refresh `last_seen_at` / `no_longer_listed` +
	53	+`removed_at`), people / products / locations by `name_norm`, pricing plans versioned (`superseded` + `valid_to`, `removed`),
	54	+news by canonical URL. Removals happen only when the extraction is trustworthy (≥ 1 entity extracted or a structured response)
	55	+and never when an HTML listing drops > 70 % of ≥ 10 entities (parse degradation guard → note in `structured_delta.notes`).
	56	+`structured_delta` shape is documented in `sdk/models.py`.
	57	+
	58	+## Scheduling formula
	59	+
	60	+```
	61	+unchanged / 304: current < base ? min(base, current × burst_decay) : min(max, current × stability_growth)
	62	+meaningful+: burst_interval_s (then decays back to base, one step per unchanged run)
	63	+minor: min(current, base)
	64	+failure: min(max, current × FAILURE_POLICY[class].multiplier)
	65	+next_run_at: now + interval × U(0.9, 1.1)
	66	+quality_score: EMA(α = 0.15) towards 100 × extraction confidence (success) or 0 (failure)
	67	+```
	68	+
	69	+Scheduler (`catlas schedule`): every `scheduler_tick_s` claim `status in (active, failing, pending) and next_run_at <= now()`
	70	+not claimed in the last 15 min, ordered by `priority desc, next_run_at`, `limit scheduler_claim_batch`, `FOR UPDATE SKIP LOCKED`;
	71	+run under `asyncio.Semaphore(fetch_concurrency)` (per-domain spacing / concurrency in `fetch.DomainGovernor`); `crawl_runs`
	72	+row per tick (`claimed, ok, changed, meaningful, failed, not_modified, unchanged, skipped, redirected, tick_ms`), heartbeat in
	73	+`settings_kv['scheduler:heartbeat']` (+ per-worker key); onboarding batch every 4 ticks unless `--no-onboarding`; periodic tasks
	74	+from `services/periodic` (interval + cron in `settings.tz`, APScheduler); SIGTERM/SIGINT release claims. Several schedulers on
	75	+different machines share one database safely (claims, idempotent runs, no node assumptions). `--once` runs a single tick.
	76	+
	77	+## Failure policies (`taxonomy.FAILURE_POLICY`: multiplier, failures-before-failing)
	78	+
	79	+DNS 3.0/3 · TIMEOUT 1.5/4 · HTTP_4XX 2.0/3 · HTTP_5XX 1.5/5 · BOT_CHALLENGE 4.0/2 · PARSING 2.0/3 · SCHEMA 2.0/3 · REDIRECT 2.0/2 ·
	80	+PAGE_REMOVED 4.0/2 · RATE_LIMIT 3.0/4 · ROBOTS 8.0/1 · BLOCKED_DESTINATION 8.0/1 · TOO_LARGE 4.0/2 · UNKNOWN 2.0/3.
	81	+Status transitions: `failing` at the class threshold → `stale` at `stale_after_failures` (6) → `retired` (+`retired_at`) at
	82	+`retire_after_failures` (30). `ROBOTS` → `blocked` immediately; `ROBOTS` / repeated `BOT_CHALLENGE` open a
	83	+`review_queue(kind='blocked_source')` item (never bypassed). Domain budgets (`domain_budgets`: `daily_budget`, `blocked_until`) are
	84	+honoured before any request; `used_today` resets per day; `cost_ledger` counts fetch units per company / connector and storage.
	85	+
	86	+## Repair (`services/repair.py`, `catlas repair`, periodic every 30 min)
	87	+
	88	+For `failing` / `stale` / `redirected` / repeated `PAGE_REMOVED` sensors (≤ 6 requests each): retry the URL (recovered → active,
	89	+`next_run_at=now`) → candidates for the same surface from the redirect target, the sitemap and the homepage navigation →
	90	+content-identity check against the last snapshot text (simhash Hamming ≤ 12 or fuzzy ratio ≥ 0.6) → migrate (old sensor
	91	+`retired` with `config.successor_id`, new sensor `pending` with `config.predecessor_id`, `discovery_method='repair'`; history is
	92	+never deleted) or `review_queue(kind='sensor_migration')` when uncertain. Each attempt is stamped in `config.last_repair_at`.
	93	+
	94	+## Operations cheat-sheet
	95	+
	96	+```bash
	97	+catlas connectors # registry + connectors table
	98	+catlas discover https://stripe.com # dry-run surface table (no DB writes)
	99	+catlas onboard --company <slug> --fetch-now
	100	+catlas schedule --once # one tick (tests / cron); catlas schedule for the daemon
	101	+catlas run-sensor <id\|url> [--file fixture] [--force]
	102	+catlas sensors --company <slug> [--status failing] [--surface careers]
	103	+catlas repair --dry-run
	104	+catlas stats-crawl
	105	+```

added fixtures/connectors/ashby/ashby_board.json +2027 −0

		`@@ -0,0 +1,2027 @@`
	1	+{
	2	+ "jobs": [
	3	+ {
	4	+ "id": "7458d4e9-da2e-47bd-98cb-adfda43d42b2",
	5	+ "title": "Engineering Manager - EU",
	6	+ "department": "Engineering",
	7	+ "team": "EMEA Engineering",
	8	+ "employmentType": "FullTime",
	9	+ "location": "Remote - European Union",
	10	+ "secondaryLocations": [
	11	+ {
	12	+ "location": "Spain",
	13	+ "address": {
	14	+ "postalAddress": {
	15	+ "addressRegion": "Spain",
	16	+ "addressCountry": "Spain",
	17	+ "addressLocality": "Spain"
	18	+ }
	19	+ }
	20	+ },
	21	+ {
	22	+ "location": "Italy",
	23	+ "address": {
	24	+ "postalAddress": {
	25	+ "addressRegion": "",
	26	+ "addressCountry": "Italy",
	27	+ "addressLocality": ""
	28	+ }
	29	+ }
	30	+ },
	31	+ {
	32	+ "location": "Germany",
	33	+ "address": {
	34	+ "postalAddress": {
	35	+ "addressCountry": "Germany",
	36	+ "addressLocality": ""
	37	+ }
	38	+ }
	39	+ },
	40	+ {
	41	+ "location": "Switzerland",
	42	+ "address": {
	43	+ "postalAddress": {
	44	+ "addressRegion": "",
	45	+ "addressCountry": "Switzerland",
	46	+ "addressLocality": ""
	47	+ }
	48	+ }
	49	+ },
	50	+ {
	51	+ "location": "Denmark",
	52	+ "address": {
	53	+ "postalAddress": {
	54	+ "addressRegion": "",
	55	+ "addressCountry": "Denmark",
	56	+ "addressLocality": ""
	57	+ }
	58	+ }
	59	+ },
	60	+ {
	61	+ "location": "Norway",
	62	+ "address": {
	63	+ "postalAddress": {
	64	+ "addressRegion": "",
	65	+ "addressCountry": "Norway",
	66	+ "addressLocality": ""
	67	+ }
	68	+ }
	69	+ },
	70	+ {
	71	+ "location": "Croatia",
	72	+ "address": {
	73	+ "postalAddress": {
	74	+ "addressRegion": "",
	75	+ "addressCountry": "Croatia",
	76	+ "addressLocality": ""
	77	+ }
	78	+ }
	79	+ },
	80	+ {
	81	+ "location": "Ireland",
	82	+ "address": {
	83	+ "postalAddress": {
	84	+ "addressRegion": "Ireland",
	85	+ "addressCountry": "Ireland",
	86	+ "addressLocality": "Ireland"
	87	+ }
	88	+ }
	89	+ },
	90	+ {
	91	+ "location": "Stockholm",
	92	+ "address": {
	93	+ "postalAddress": {
	94	+ "addressRegion": "Stockholm",
	95	+ "addressCountry": "Sweden",
	96	+ "addressLocality": "Stockholm"
	97	+ }
	98	+ }
	99	+ },
	100	+ {
	101	+ "location": "Romania",
	102	+ "address": {
	103	+ "postalAddress": {
	104	+ "addressRegion": "",
	105	+ "addressCountry": "Romania",
	106	+ "addressLocality": ""
	107	+ }
	108	+ }
	109	+ },
	110	+ {
	111	+ "location": "Austria",
	112	+ "address": {
	113	+ "postalAddress": {
	114	+ "addressCountry": "Austria"
	115	+ }
	116	+ }
	117	+ },
	118	+ {
	119	+ "location": "Barcelona",
	120	+ "address": {
	121	+ "postalAddress": {
	122	+ "addressRegion": "Catalonia",
	123	+ "addressCountry": "Spain",
	124	+ "addressLocality": "Barcelona"
	125	+ }
	126	+ }
	127	+ },
	128	+ {
	129	+ "location": "Netherlands",
	130	+ "address": {
	131	+ "postalAddress": {
	132	+ "addressRegion": "",
	133	+ "addressCountry": "Netherlands",
	134	+ "addressLocality": "Amsterdam"
	135	+ }
	136	+ }
	137	+ },
	138	+ {
	139	+ "location": "Portugal",
	140	+ "address": {
	141	+ "postalAddress": {
	142	+ "addressCountry": "Portugal"
	143	+ }
	144	+ }
	145	+ },
	146	+ {
	147	+ "location": "France",
	148	+ "address": {
	149	+ "postalAddress": {
	150	+ "addressRegion": "",
	151	+ "addressCountry": "France",
	152	+ "addressLocality": ""
	153	+ }
	154	+ }
	155	+ },
	156	+ {
	157	+ "location": "Berlin",
	158	+ "address": {
	159	+ "postalAddress": {
	160	+ "addressRegion": "",
	161	+ "addressCountry": "Germany",
	162	+ "addressLocality": "Berlin"
	163	+ }
	164	+ }
	165	+ },
	166	+ {
	167	+ "location": "Sweden",
	168	+ "address": {
	169	+ "postalAddress": {
	170	+ "addressRegion": "",
	171	+ "addressCountry": "Sweden",
	172	+ "addressLocality": ""
	173	+ }
	174	+ }
	175	+ },
	176	+ {
	177	+ "location": "Hungary",
	178	+ "address": {
	179	+ "postalAddress": {
	180	+ "addressRegion": "Hungary",
	181	+ "addressCountry": "Hungary",
	182	+ "addressLocality": "Budapest"
	183	+ }
	184	+ }
	185	+ },
	186	+ {
	187	+ "location": "Estonia",
	188	+ "address": {
	189	+ "postalAddress": {
	190	+ "addressRegion": "",
	191	+ "addressCountry": "Estonia",
	192	+ "addressLocality": ""
	193	+ }
	194	+ }
	195	+ }
	196	+ ],
	197	+ "publishedAt": "2024-03-04T14:29:08.532+00:00",
	198	+ "isListed": true,
	199	+ "isRemote": true,
	200	+ "workplaceType": "Remote",
	201	+ "address": {
	202	+ "postalAddress": {
	203	+ "postalCode": "",
	204	+ "addressRegion": "",
	205	+ "addressCountry": "European Union",
	206	+ "addressLocality": ""
	207	+ }
	208	+ },
	209	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/7458d4e9-da2e-47bd-98cb-adfda43d42b2",
	210	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/7458d4e9-da2e-47bd-98cb-adfda43d42b2/application"
	211	+ },
	212	+ {
	213	+ "id": "390e266b-4b6c-4490-ad74-05ff5e0bb36a",
	214	+ "title": "Product Manager, Onboarding and Growth",
	215	+ "department": "Product",
	216	+ "team": "Product",
	217	+ "employmentType": "FullTime",
	218	+ "location": "Remote - US",
	219	+ "secondaryLocations": [
	220	+ {
	221	+ "location": "Remote - Canada",
	222	+ "address": {
	223	+ "postalAddress": {
	224	+ "addressRegion": "",
	225	+ "addressCountry": "Canada",
	226	+ "addressLocality": ""
	227	+ }
	228	+ }
	229	+ }
	230	+ ],
	231	+ "publishedAt": "2026-09-11T19:28:18.199+00:00",
	232	+ "isListed": true,
	233	+ "isRemote": true,
	234	+ "workplaceType": "Remote",
	235	+ "address": {
	236	+ "postalAddress": {
	237	+ "addressRegion": "",
	238	+ "addressCountry": "United States",
	239	+ "addressLocality": ""
	240	+ }
	241	+ },
	242	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/390e266b-4b6c-4490-ad74-05ff5e0bb36a",
	243	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/390e266b-4b6c-4490-ad74-05ff5e0bb36a/application"
	244	+ },
	245	+ {
	246	+ "id": "86a60834-ba64-484d-9658-afa1bc97a957",
	247	+ "title": "Mid Market Account Executive - EMEA (French Speaking)",
	248	+ "department": "Sales",
	249	+ "team": "Sales",
	250	+ "employmentType": "FullTime",
	251	+ "location": "France",
	252	+ "secondaryLocations": [
	253	+ {
	254	+ "location": "United Kingdom",
	255	+ "address": {
	256	+ "postalAddress": {
	257	+ "addressRegion": "",
	258	+ "addressCountry": "United Kingdom",
	259	+ "addressLocality": ""
	260	+ }
	261	+ }
	262	+ },
	263	+ {
	264	+ "location": "Belgium",
	265	+ "address": {
	266	+ "postalAddress": {
	267	+ "addressRegion": "",
	268	+ "addressCountry": "Belgium",
	269	+ "addressLocality": ""
	270	+ }
	271	+ }
	272	+ },
	273	+ {
	274	+ "location": "Germany",
	275	+ "address": {
	276	+ "postalAddress": {
	277	+ "addressCountry": "Germany",
	278	+ "addressLocality": ""
	279	+ }
	280	+ }
	281	+ },
	282	+ {
	283	+ "location": "Switzerland",
	284	+ "address": {
	285	+ "postalAddress": {
	286	+ "addressRegion": "",
	287	+ "addressCountry": "Switzerland",
	288	+ "addressLocality": ""
	289	+ }
	290	+ }
	291	+ },
	292	+ {
	293	+ "location": "Ireland",
	294	+ "address": {
	295	+ "postalAddress": {
	296	+ "addressRegion": "",
	297	+ "addressCountry": "Ireland",
	298	+ "addressLocality": ""
	299	+ }
	300	+ }
	301	+ },
	302	+ {
	303	+ "location": "Austria",
	304	+ "address": {
	305	+ "postalAddress": {
	306	+ "addressCountry": "Austria"
	307	+ }
	308	+ }
	309	+ },
	310	+ {
	311	+ "location": "Spain",
	312	+ "address": {
	313	+ "postalAddress": {
	314	+ "addressCountry": "Spain"
	315	+ }
	316	+ }
	317	+ },
	318	+ {
	319	+ "location": "Stuttgart",
	320	+ "address": {
	321	+ "postalAddress": {
	322	+ "addressRegion": "Baden-Wurttemberg",
	323	+ "addressCountry": "Germany",
	324	+ "addressLocality": "Stuttgart"
	325	+ }
	326	+ }
	327	+ },
	328	+ {
	329	+ "location": "Berlin",
	330	+ "address": {
	331	+ "postalAddress": {
	332	+ "addressRegion": "",
	333	+ "addressCountry": "Germany",
	334	+ "addressLocality": "Berlin"
	335	+ }
	336	+ }
	337	+ }
	338	+ ],
	339	+ "publishedAt": "2026-09-08T16:27:26.694+00:00",
	340	+ "isListed": true,
	341	+ "isRemote": true,
	342	+ "workplaceType": "Remote",
	343	+ "address": {
	344	+ "postalAddress": {
	345	+ "addressRegion": "",
	346	+ "addressCountry": "France",
	347	+ "addressLocality": ""
	348	+ }
	349	+ },
	350	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/86a60834-ba64-484d-9658-afa1bc97a957",
	351	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/86a60834-ba64-484d-9658-afa1bc97a957/application"
	352	+ },
	353	+ {
	354	+ "id": "f40ef345-82a8-4956-9150-193b4fdf8183",
	355	+ "title": "Senior Product Designer",
	356	+ "department": "Design",
	357	+ "team": "Design",
	358	+ "employmentType": "FullTime",
	359	+ "location": "Remote - US",
	360	+ "secondaryLocations": [
	361	+ {
	362	+ "location": "Austin",
	363	+ "address": {
	364	+ "postalAddress": {
	365	+ "addressRegion": "Texas",
	366	+ "addressCountry": "USA",
	367	+ "addressLocality": "Austin"
	368	+ }
	369	+ }
	370	+ },
	371	+ {
	372	+ "location": "Los Angeles",
	373	+ "address": {
	374	+ "postalAddress": {
	375	+ "addressRegion": "California",
	376	+ "addressCountry": "USA",
	377	+ "addressLocality": "Los Angeles"
	378	+ }
	379	+ }
	380	+ },
	381	+ {
	382	+ "location": "Portland",
	383	+ "address": {
	384	+ "postalAddress": {
	385	+ "addressRegion": "Oregon",
	386	+ "addressCountry": "USA",
	387	+ "addressLocality": "Portland"
	388	+ }
	389	+ }
	390	+ },
	391	+ {
	392	+ "location": "Salt Lake City",
	393	+ "address": {
	394	+ "postalAddress": {
	395	+ "addressRegion": "Utah",
	396	+ "addressCountry": "USA",
	397	+ "addressLocality": "Salt Lake City"
	398	+ }
	399	+ }
	400	+ },
	401	+ {
	402	+ "location": "Waterloo",
	403	+ "address": {
	404	+ "postalAddress": {
	405	+ "addressRegion": "Ontario",
	406	+ "addressCountry": "Canada",
	407	+ "addressLocality": "Waterloo"
	408	+ }
	409	+ }
	410	+ },
	411	+ {
	412	+ "location": "Boston",
	413	+ "address": {
	414	+ "postalAddress": {
	415	+ "addressRegion": "Massachusetts",
	416	+ "addressCountry": "USA",
	417	+ "addressLocality": "Boston"
	418	+ }
	419	+ }
	420	+ },
	421	+ {
	422	+ "location": "Remote - Canada",
	423	+ "address": {
	424	+ "postalAddress": {
	425	+ "addressRegion": "",
	426	+ "addressCountry": "Canada",
	427	+ "addressLocality": ""
	428	+ }
	429	+ }
	430	+ },
	431	+ {
	432	+ "location": "Seattle",
	433	+ "address": {
	434	+ "postalAddress": {
	435	+ "addressRegion": "Washington",
	436	+ "addressCountry": "USA",
	437	+ "addressLocality": "Seattle"
	438	+ }
	439	+ }
	440	+ },
	441	+ {
	442	+ "location": "New York",
	443	+ "address": {
	444	+ "postalAddress": {
	445	+ "addressRegion": "New York",
	446	+ "addressCountry": "USA",
	447	+ "addressLocality": "New York"
	448	+ }
	449	+ }
	450	+ },
	451	+ {
	452	+ "location": "San Francisco",
	453	+ "address": {
	454	+ "postalAddress": {
	455	+ "addressRegion": "CA",
	456	+ "addressCountry": "USA",
	457	+ "addressLocality": "San Francisco"
	458	+ }
	459	+ }
	460	+ },
	461	+ {
	462	+ "location": "Vancouver",
	463	+ "address": {
	464	+ "postalAddress": {
	465	+ "addressRegion": "British Columbia",
	466	+ "addressCountry": "Canada",
	467	+ "addressLocality": "Vancouver"
	468	+ }
	469	+ }
	470	+ },
	471	+ {
	472	+ "location": "Denver",
	473	+ "address": {
	474	+ "postalAddress": {
	475	+ "addressRegion": "Colorado",
	476	+ "addressCountry": "USA",
	477	+ "addressLocality": "Denver"
	478	+ }
	479	+ }
	480	+ },
	481	+ {
	482	+ "location": "Toronto",
	483	+ "address": {
	484	+ "postalAddress": {
	485	+ "addressRegion": "Ontario",
	486	+ "addressCountry": "Canada",
	487	+ "addressLocality": "Toronto"
	488	+ }
	489	+ }
	490	+ },
	491	+ {
	492	+ "location": "Chicago",
	493	+ "address": {
	494	+ "postalAddress": {
	495	+ "addressRegion": "Illinois",
	496	+ "addressCountry": "USA",
	497	+ "addressLocality": "Chicago"
	498	+ }
	499	+ }
	500	+ }
	501	+ ],
	502	+ "publishedAt": "2025-12-05T22:43:22.147+00:00",
	503	+ "isListed": true,
	504	+ "isRemote": true,
	505	+ "workplaceType": "Remote",
	506	+ "address": {
	507	+ "postalAddress": {
	508	+ "addressRegion": "",
	509	+ "addressCountry": "United States",
	510	+ "addressLocality": ""
	511	+ }
	512	+ },
	513	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/f40ef345-82a8-4956-9150-193b4fdf8183",
	514	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/f40ef345-82a8-4956-9150-193b4fdf8183/application"
	515	+ },
	516	+ {
	517	+ "id": "d573471b-2005-482c-9fbf-d1df9550cb57",
	518	+ "title": "Engineering Manager - UK",
	519	+ "department": "Engineering",
	520	+ "team": "EMEA Engineering",
	521	+ "employmentType": "FullTime",
	522	+ "location": "United Kingdom",
	523	+ "secondaryLocations": [
	524	+ {
	525	+ "location": "Cardiff",
	526	+ "address": {
	527	+ "postalAddress": {
	528	+ "addressRegion": "",
	529	+ "addressCountry": "United Kingdom",
	530	+ "addressLocality": "Cardiff"
	531	+ }
	532	+ }
	533	+ },
	534	+ {
	535	+ "location": "Birmingham",
	536	+ "address": {
	537	+ "postalAddress": {
	538	+ "addressRegion": "",
	539	+ "addressCountry": "United Kingdom",
	540	+ "addressLocality": "Birmingham"
	541	+ }
	542	+ }
	543	+ },
	544	+ {
	545	+ "location": "Bristol",
	546	+ "address": {
	547	+ "postalAddress": {
	548	+ "addressRegion": "",
	549	+ "addressCountry": "United Kingdom",
	550	+ "addressLocality": "Bristol"
	551	+ }
	552	+ }
	553	+ },
	554	+ {
	555	+ "location": "Manchester",
	556	+ "address": {
	557	+ "postalAddress": {
	558	+ "addressRegion": "",
	559	+ "addressCountry": "United Kingdom",
	560	+ "addressLocality": "Manchester"
	561	+ }
	562	+ }
	563	+ },
	564	+ {
	565	+ "location": "Leeds",
	566	+ "address": {
	567	+ "postalAddress": {
	568	+ "addressRegion": "",
	569	+ "addressCountry": "United Kingdom",
	570	+ "addressLocality": "Leeds"
	571	+ }
	572	+ }
	573	+ },
	574	+ {
	575	+ "location": "Oxford",
	576	+ "address": {
	577	+ "postalAddress": {
	578	+ "addressRegion": "",
	579	+ "addressCountry": "United Kingdom",
	580	+ "addressLocality": "Oxford"
	581	+ }
	582	+ }
	583	+ },
	584	+ {
	585	+ "location": "London",
	586	+ "address": {
	587	+ "postalAddress": {
	588	+ "addressRegion": "",
	589	+ "addressCountry": "United Kingdom",
	590	+ "addressLocality": "London"
	591	+ }
	592	+ }
	593	+ },
	594	+ {
	595	+ "location": "Cambridge",
	596	+ "address": {
	597	+ "postalAddress": {
	598	+ "addressRegion": "",
	599	+ "addressCountry": "United Kingdom",
	600	+ "addressLocality": "Cambridge"
	601	+ }
	602	+ }
	603	+ },
	604	+ {
	605	+ "location": "Glasgow",
	606	+ "address": {
	607	+ "postalAddress": {
	608	+ "addressRegion": "",
	609	+ "addressCountry": "United Kingdom",
	610	+ "addressLocality": "Glasgow"
	611	+ }
	612	+ }
	613	+ },
	614	+ {
	615	+ "location": "Edinburgh",
	616	+ "address": {
	617	+ "postalAddress": {
	618	+ "addressRegion": "",
	619	+ "addressCountry": "United Kingdom",
	620	+ "addressLocality": "Edinburgh"
	621	+ }
	622	+ }
	623	+ },
	624	+ {
	625	+ "location": "Belfast",
	626	+ "address": {
	627	+ "postalAddress": {
	628	+ "addressRegion": "",
	629	+ "addressCountry": "United Kingdom",
	630	+ "addressLocality": "Belfast"
	631	+ }
	632	+ }
	633	+ }
	634	+ ],
	635	+ "publishedAt": "2025-04-01T14:39:51.428+00:00",
	636	+ "isListed": true,
	637	+ "isRemote": true,
	638	+ "workplaceType": "Remote",
	639	+ "address": {
	640	+ "postalAddress": {
	641	+ "addressRegion": "",
	642	+ "addressCountry": "United Kingdom",
	643	+ "addressLocality": ""
	644	+ }
	645	+ },
	646	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/d573471b-2005-482c-9fbf-d1df9550cb57",
	647	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/d573471b-2005-482c-9fbf-d1df9550cb57/application"
	648	+ },
	649	+ {
	650	+ "id": "7e4e8a93-c35d-43c1-a739-c53aba8b682f",
	651	+ "title": "Enterprise Account Executive - DACH",
	652	+ "department": "Sales",
	653	+ "team": "Sales",
	654	+ "employmentType": "FullTime",
	655	+ "location": "Germany",
	656	+ "secondaryLocations": [
	657	+ {
	658	+ "location": "United Kingdom",
	659	+ "address": {
	660	+ "postalAddress": {
	661	+ "addressRegion": "",
	662	+ "addressCountry": "United Kingdom",
	663	+ "addressLocality": ""
	664	+ }
	665	+ }
	666	+ },
	667	+ {
	668	+ "location": "Belgium",
	669	+ "address": {
	670	+ "postalAddress": {
	671	+ "addressRegion": "",
	672	+ "addressCountry": "Belgium",
	673	+ "addressLocality": ""
	674	+ }
	675	+ }
	676	+ },
	677	+ {
	678	+ "location": "Ireland",
	679	+ "address": {
	680	+ "postalAddress": {
	681	+ "addressRegion": "",
	682	+ "addressCountry": "Ireland",
	683	+ "addressLocality": ""
	684	+ }
	685	+ }
	686	+ },
	687	+ {
	688	+ "location": "Czech Republic",
	689	+ "address": {
	690	+ "postalAddress": {
	691	+ "addressRegion": "Czech Republic",
	692	+ "addressCountry": "Czech Republic",
	693	+ "addressLocality": "Prague"
	694	+ }
	695	+ }
	696	+ },
	697	+ {
	698	+ "location": "Austria",
	699	+ "address": {
	700	+ "postalAddress": {
	701	+ "addressCountry": "Austria"
	702	+ }
	703	+ }
	704	+ },
	705	+ {
	706	+ "location": "Poland",
	707	+ "address": {
	708	+ "postalAddress": {
	709	+ "addressRegion": "Poland",
	710	+ "addressCountry": "Poland",
	711	+ "addressLocality": "Warsaw"
	712	+ }
	713	+ }
	714	+ },
	715	+ {
	716	+ "location": "Spain",
	717	+ "address": {
	718	+ "postalAddress": {
	719	+ "addressCountry": "Spain"
	720	+ }
	721	+ }
	722	+ },
	723	+ {
	724	+ "location": "Netherlands",
	725	+ "address": {
	726	+ "postalAddress": {
	727	+ "addressRegion": "",
	728	+ "addressCountry": "Netherlands",
	729	+ "addressLocality": "Amsterdam"
	730	+ }
	731	+ }
	732	+ },
	733	+ {
	734	+ "location": "France",
	735	+ "address": {
	736	+ "postalAddress": {
	737	+ "addressRegion": "",
	738	+ "addressCountry": "France",
	739	+ "addressLocality": ""
	740	+ }
	741	+ }
	742	+ },
	743	+ {
	744	+ "location": "Hungary",
	745	+ "address": {
	746	+ "postalAddress": {
	747	+ "addressRegion": "Hungary",
	748	+ "addressCountry": "Hungary",
	749	+ "addressLocality": "Budapest"
	750	+ }
	751	+ }
	752	+ }
	753	+ ],
	754	+ "publishedAt": "2026-07-27T14:29:07.354+00:00",
	755	+ "isListed": true,
	756	+ "isRemote": true,
	757	+ "workplaceType": "Remote",
	758	+ "address": {
	759	+ "postalAddress": {
	760	+ "addressCountry": "Germany",
	761	+ "addressLocality": ""
	762	+ }
	763	+ },
	764	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/7e4e8a93-c35d-43c1-a739-c53aba8b682f",
	765	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/7e4e8a93-c35d-43c1-a739-c53aba8b682f/application"
	766	+ },
	767	+ {
	768	+ "id": "5ccfc540-35d4-4fc8-aeb6-7f11b2ad8258",
	769	+ "title": "Engineering Manager - Canada",
	770	+ "department": "Engineering",
	771	+ "team": "Americas Engineering",
	772	+ "employmentType": "FullTime",
	773	+ "location": "Remote - Canada",
	774	+ "secondaryLocations": [
	775	+ {
	776	+ "location": "Calgary",
	777	+ "address": {
	778	+ "postalAddress": {
	779	+ "addressRegion": "Alberta",
	780	+ "addressCountry": "Canada",
	781	+ "addressLocality": "Calgary"
	782	+ }
	783	+ }
	784	+ },
	785	+ {
	786	+ "location": "Vancouver",
	787	+ "address": {
	788	+ "postalAddress": {
	789	+ "addressRegion": "British Columbia",
	790	+ "addressCountry": "Canada",
	791	+ "addressLocality": "Vancouver"
	792	+ }
	793	+ }
	794	+ },
	795	+ {
	796	+ "location": "Toronto",
	797	+ "address": {
	798	+ "postalAddress": {
	799	+ "addressRegion": "Ontario",
	800	+ "addressCountry": "Canada",
	801	+ "addressLocality": "Toronto"
	802	+ }
	803	+ }
	804	+ }
	805	+ ],
	806	+ "publishedAt": "2025-11-13T23:05:17.459+00:00",
	807	+ "isListed": true,
	808	+ "isRemote": true,
	809	+ "workplaceType": "Remote",
	810	+ "address": {
	811	+ "postalAddress": {
	812	+ "addressRegion": "",
	813	+ "addressCountry": "Canada",
	814	+ "addressLocality": ""
	815	+ }
	816	+ },
	817	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/5ccfc540-35d4-4fc8-aeb6-7f11b2ad8258",
	818	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/5ccfc540-35d4-4fc8-aeb6-7f11b2ad8258/application"
	819	+ },
	820	+ {
	821	+ "id": "695b1ce6-c375-4fad-94ef-61b50a3fbd28",
	822	+ "title": "Engineering Manager - Americas",
	823	+ "department": "Engineering",
	824	+ "team": "Americas Engineering",
	825	+ "employmentType": "FullTime",
	826	+ "location": "Remote - US",
	827	+ "secondaryLocations": [
	828	+ {
	829	+ "location": "Austin",
	830	+ "address": {
	831	+ "postalAddress": {
	832	+ "addressRegion": "Texas",
	833	+ "addressCountry": "USA",
	834	+ "addressLocality": "Austin"
	835	+ }
	836	+ }
	837	+ },
	838	+ {
	839	+ "location": "Boulder",
	840	+ "address": {
	841	+ "postalAddress": {
	842	+ "addressRegion": "Colorado",
	843	+ "addressCountry": "USA",
	844	+ "addressLocality": "Boulder"
	845	+ }
	846	+ }
	847	+ },
	848	+ {
	849	+ "location": "Los Angeles",
	850	+ "address": {
	851	+ "postalAddress": {
	852	+ "addressRegion": "California",
	853	+ "addressCountry": "USA",
	854	+ "addressLocality": "Los Angeles"
	855	+ }
	856	+ }
	857	+ },
	858	+ {
	859	+ "location": "Portland",
	860	+ "address": {
	861	+ "postalAddress": {
	862	+ "addressRegion": "Oregon",
	863	+ "addressCountry": "USA",
	864	+ "addressLocality": "Portland"
	865	+ }
	866	+ }
	867	+ },
	868	+ {
	869	+ "location": "Atlanta",
	870	+ "address": {
	871	+ "postalAddress": {
	872	+ "addressRegion": "Georgia",
	873	+ "addressCountry": "USA",
	874	+ "addressLocality": "Atlanta"
	875	+ }
	876	+ }
	877	+ },
	878	+ {
	879	+ "location": "Salt Lake City",
	880	+ "address": {
	881	+ "postalAddress": {
	882	+ "addressRegion": "Utah",
	883	+ "addressCountry": "USA",
	884	+ "addressLocality": "Salt Lake City"
	885	+ }
	886	+ }
	887	+ },
	888	+ {
	889	+ "location": "Boston",
	890	+ "address": {
	891	+ "postalAddress": {
	892	+ "addressRegion": "Massachusetts",
	893	+ "addressCountry": "USA",
	894	+ "addressLocality": "Boston"
	895	+ }
	896	+ }
	897	+ },
	898	+ {
	899	+ "location": "Seattle",
	900	+ "address": {
	901	+ "postalAddress": {
	902	+ "addressRegion": "Washington",
	903	+ "addressCountry": "USA",
	904	+ "addressLocality": "Seattle"
	905	+ }
	906	+ }
	907	+ },
	908	+ {
	909	+ "location": "New York",
	910	+ "address": {
	911	+ "postalAddress": {
	912	+ "addressRegion": "New York",
	913	+ "addressCountry": "USA",
	914	+ "addressLocality": "New York"
	915	+ }
	916	+ }
	917	+ },
	918	+ {
	919	+ "location": "San Francisco",
	920	+ "address": {
	921	+ "postalAddress": {
	922	+ "addressRegion": "CA",
	923	+ "addressCountry": "USA",
	924	+ "addressLocality": "San Francisco"
	925	+ }
	926	+ }
	927	+ },
	928	+ {
	929	+ "location": "Denver",
	930	+ "address": {
	931	+ "postalAddress": {
	932	+ "addressRegion": "Colorado",
	933	+ "addressCountry": "USA",
	934	+ "addressLocality": "Denver"
	935	+ }
	936	+ }
	937	+ },
	938	+ {
	939	+ "location": "Ann Arbor",
	940	+ "address": {
	941	+ "postalAddress": {
	942	+ "addressRegion": "Michigan",
	943	+ "addressCountry": "USA",
	944	+ "addressLocality": "Ann Arbor"
	945	+ }
	946	+ }
	947	+ },
	948	+ {
	949	+ "location": "Chicago",
	950	+ "address": {
	951	+ "postalAddress": {
	952	+ "addressRegion": "Illinois",
	953	+ "addressCountry": "USA",
	954	+ "addressLocality": "Chicago"
	955	+ }
	956	+ }
	957	+ },
	958	+ {
	959	+ "location": "Raleigh",
	960	+ "address": {
	961	+ "postalAddress": {
	962	+ "addressRegion": "North Carolina",
	963	+ "addressCountry": "USA",
	964	+ "addressLocality": "Raleigh"
	965	+ }
	966	+ }
	967	+ }
	968	+ ],
	969	+ "publishedAt": "2025-11-13T23:04:14.276+00:00",
	970	+ "isListed": true,
	971	+ "isRemote": true,
	972	+ "workplaceType": "Remote",
	973	+ "address": {
	974	+ "postalAddress": {
	975	+ "addressRegion": "",
	976	+ "addressCountry": "United States",
	977	+ "addressLocality": ""
	978	+ }
	979	+ },
	980	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/695b1ce6-c375-4fad-94ef-61b50a3fbd28",
	981	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/695b1ce6-c375-4fad-94ef-61b50a3fbd28/application"
	982	+ },
	983	+ {
	984	+ "id": "cc846e56-27dd-41b5-bf5b-64d5eeb5ff1a",
	985	+ "title": "Staff Design Engineer - Americas",
	986	+ "department": "Engineering",
	987	+ "team": "Americas Engineering",
	988	+ "employmentType": "FullTime",
	989	+ "location": "Remote - US",
	990	+ "secondaryLocations": [
	991	+ {
	992	+ "location": "Austin",
	993	+ "address": {
	994	+ "postalAddress": {
	995	+ "addressRegion": "Texas",
	996	+ "addressCountry": "USA",
	997	+ "addressLocality": "Austin"
	998	+ }
	999	+ }
	1000	+ },
	1001	+ {
	1002	+ "location": "Los Angeles",
	1003	+ "address": {
	1004	+ "postalAddress": {
	1005	+ "addressRegion": "California",
	1006	+ "addressCountry": "USA",
	1007	+ "addressLocality": "Los Angeles"
	1008	+ }
	1009	+ }
	1010	+ },
	1011	+ {
	1012	+ "location": "Portland",
	1013	+ "address": {
	1014	+ "postalAddress": {
	1015	+ "addressRegion": "Oregon",
	1016	+ "addressCountry": "USA",
	1017	+ "addressLocality": "Portland"
	1018	+ }
	1019	+ }
	1020	+ },
	1021	+ {
	1022	+ "location": "Boston",
	1023	+ "address": {
	1024	+ "postalAddress": {
	1025	+ "addressRegion": "Massachusetts",
	1026	+ "addressCountry": "USA",
	1027	+ "addressLocality": "Boston"
	1028	+ }
	1029	+ }
	1030	+ },
	1031	+ {
	1032	+ "location": "Seattle",
	1033	+ "address": {
	1034	+ "postalAddress": {
	1035	+ "addressRegion": "Washington",
	1036	+ "addressCountry": "USA",
	1037	+ "addressLocality": "Seattle"
	1038	+ }
	1039	+ }
	1040	+ },
	1041	+ {
	1042	+ "location": "New York",
	1043	+ "address": {
	1044	+ "postalAddress": {
	1045	+ "addressRegion": "New York",
	1046	+ "addressCountry": "USA",
	1047	+ "addressLocality": "New York"
	1048	+ }
	1049	+ }
	1050	+ },
	1051	+ {
	1052	+ "location": "Denver",
	1053	+ "address": {
	1054	+ "postalAddress": {
	1055	+ "addressRegion": "Colorado",
	1056	+ "addressCountry": "USA",
	1057	+ "addressLocality": "Denver"
	1058	+ }
	1059	+ }
	1060	+ },
	1061	+ {
	1062	+ "location": "Chicago",
	1063	+ "address": {
	1064	+ "postalAddress": {
	1065	+ "addressRegion": "Illinois",
	1066	+ "addressCountry": "USA",
	1067	+ "addressLocality": "Chicago"
	1068	+ }
	1069	+ }
	1070	+ }
	1071	+ ],
	1072	+ "publishedAt": "2025-11-14T01:04:25.832+00:00",
	1073	+ "isListed": true,
	1074	+ "isRemote": true,
	1075	+ "workplaceType": "Remote",
	1076	+ "address": {
	1077	+ "postalAddress": {
	1078	+ "addressRegion": "",
	1079	+ "addressCountry": "United States",
	1080	+ "addressLocality": ""
	1081	+ }
	1082	+ },
	1083	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/cc846e56-27dd-41b5-bf5b-64d5eeb5ff1a",
	1084	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/cc846e56-27dd-41b5-bf5b-64d5eeb5ff1a/application"
	1085	+ },
	1086	+ {
	1087	+ "id": "97c6542c-7ff9-43e5-ac54-b77b45fc7378",
	1088	+ "title": "Staff Design Engineer - Canada",
	1089	+ "department": "Engineering",
	1090	+ "team": "Americas Engineering",
	1091	+ "employmentType": "FullTime",
	1092	+ "location": "Remote - Canada",
	1093	+ "secondaryLocations": [
	1094	+ {
	1095	+ "location": "Montreal",
	1096	+ "address": {
	1097	+ "postalAddress": {
	1098	+ "addressRegion": "Quebec",
	1099	+ "addressCountry": "Canada",
	1100	+ "addressLocality": "Montreal"
	1101	+ }
	1102	+ }
	1103	+ },
	1104	+ {
	1105	+ "location": "Vancouver",
	1106	+ "address": {
	1107	+ "postalAddress": {
	1108	+ "addressRegion": "British Columbia",
	1109	+ "addressCountry": "Canada",
	1110	+ "addressLocality": "Vancouver"
	1111	+ }
	1112	+ }
	1113	+ },
	1114	+ {
	1115	+ "location": "Toronto",
	1116	+ "address": {
	1117	+ "postalAddress": {
	1118	+ "addressRegion": "Ontario",
	1119	+ "addressCountry": "Canada",
	1120	+ "addressLocality": "Toronto"
	1121	+ }
	1122	+ }
	1123	+ }
	1124	+ ],
	1125	+ "publishedAt": "2025-11-13T22:59:40.371+00:00",
	1126	+ "isListed": true,
	1127	+ "isRemote": true,
	1128	+ "workplaceType": "Remote",
	1129	+ "address": {
	1130	+ "postalAddress": {
	1131	+ "addressRegion": "",
	1132	+ "addressCountry": "Canada",
	1133	+ "addressLocality": ""
	1134	+ }
	1135	+ },
	1136	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/97c6542c-7ff9-43e5-ac54-b77b45fc7378",
	1137	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/97c6542c-7ff9-43e5-ac54-b77b45fc7378/application"
	1138	+ },
	1139	+ {
	1140	+ "id": "ae9df091-112c-4a59-b3d1-49d6e0ed3de9",
	1141	+ "title": "Integrations Consultant - Americas",
	1142	+ "department": "Customer Success",
	1143	+ "team": "Professional Services",
	1144	+ "employmentType": "FullTime",
	1145	+ "location": "Remote - US",
	1146	+ "secondaryLocations": [],
	1147	+ "publishedAt": "2026-09-09T01:47:14.203+00:00",
	1148	+ "isListed": true,
	1149	+ "isRemote": true,
	1150	+ "workplaceType": "Remote",
	1151	+ "address": {
	1152	+ "postalAddress": {
	1153	+ "addressRegion": "",
	1154	+ "addressCountry": "United States",
	1155	+ "addressLocality": ""
	1156	+ }
	1157	+ },
	1158	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/ae9df091-112c-4a59-b3d1-49d6e0ed3de9",
	1159	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/ae9df091-112c-4a59-b3d1-49d6e0ed3de9/application"
	1160	+ },
	1161	+ {
	1162	+ "id": "6a15668c-3f98-4404-8f1f-927cddfc5af7",
	1163	+ "title": "Staff Product Engineer - Americas",
	1164	+ "department": "Engineering",
	1165	+ "team": "Americas Engineering",
	1166	+ "employmentType": "FullTime",
	1167	+ "location": "Remote - US",
	1168	+ "secondaryLocations": [
	1169	+ {
	1170	+ "location": "Austin",
	1171	+ "address": {
	1172	+ "postalAddress": {
	1173	+ "addressRegion": "Texas",
	1174	+ "addressCountry": "USA",
	1175	+ "addressLocality": "Austin"
	1176	+ }
	1177	+ }
	1178	+ },
	1179	+ {
	1180	+ "location": "Boulder",
	1181	+ "address": {
	1182	+ "postalAddress": {
	1183	+ "addressRegion": "Colorado",
	1184	+ "addressCountry": "USA",
	1185	+ "addressLocality": "Boulder"
	1186	+ }
	1187	+ }
	1188	+ },
	1189	+ {
	1190	+ "location": "Los Angeles",
	1191	+ "address": {
	1192	+ "postalAddress": {
	1193	+ "addressRegion": "California",
	1194	+ "addressCountry": "USA",
	1195	+ "addressLocality": "Los Angeles"
	1196	+ }
	1197	+ }
	1198	+ },
	1199	+ {
	1200	+ "location": "Portland",
	1201	+ "address": {
	1202	+ "postalAddress": {
	1203	+ "addressRegion": "Oregon",
	1204	+ "addressCountry": "USA",
	1205	+ "addressLocality": "Portland"
	1206	+ }
	1207	+ }
	1208	+ },
	1209	+ {
	1210	+ "location": "Salt Lake City",
	1211	+ "address": {
	1212	+ "postalAddress": {
	1213	+ "addressRegion": "Utah",
	1214	+ "addressCountry": "USA",
	1215	+ "addressLocality": "Salt Lake City"
	1216	+ }
	1217	+ }
	1218	+ },
	1219	+ {
	1220	+ "location": "Boston",
	1221	+ "address": {
	1222	+ "postalAddress": {
	1223	+ "addressRegion": "Massachusetts",
	1224	+ "addressCountry": "USA",
	1225	+ "addressLocality": "Boston"
	1226	+ }
	1227	+ }
	1228	+ },
	1229	+ {
	1230	+ "location": "Seattle",
	1231	+ "address": {
	1232	+ "postalAddress": {
	1233	+ "addressRegion": "Washington",
	1234	+ "addressCountry": "USA",
	1235	+ "addressLocality": "Seattle"
	1236	+ }
	1237	+ }
	1238	+ },
	1239	+ {
	1240	+ "location": "New York",
	1241	+ "address": {
	1242	+ "postalAddress": {
	1243	+ "addressRegion": "New York",
	1244	+ "addressCountry": "USA",
	1245	+ "addressLocality": "New York"
	1246	+ }
	1247	+ }
	1248	+ },
	1249	+ {
	1250	+ "location": "San Francisco",
	1251	+ "address": {
	1252	+ "postalAddress": {
	1253	+ "addressRegion": "CA",
	1254	+ "addressCountry": "USA",
	1255	+ "addressLocality": "San Francisco"
	1256	+ }
	1257	+ }
	1258	+ },
	1259	+ {
	1260	+ "location": "Denver",
	1261	+ "address": {
	1262	+ "postalAddress": {
	1263	+ "addressRegion": "Colorado",
	1264	+ "addressCountry": "USA",
	1265	+ "addressLocality": "Denver"
	1266	+ }
	1267	+ }
	1268	+ },
	1269	+ {
	1270	+ "location": "Ann Arbor",
	1271	+ "address": {
	1272	+ "postalAddress": {
	1273	+ "addressRegion": "Michigan",
	1274	+ "addressCountry": "USA",
	1275	+ "addressLocality": "Ann Arbor"
	1276	+ }
	1277	+ }
	1278	+ },
	1279	+ {
	1280	+ "location": "Chicago",
	1281	+ "address": {
	1282	+ "postalAddress": {
	1283	+ "addressRegion": "Illinois",
	1284	+ "addressCountry": "USA",
	1285	+ "addressLocality": "Chicago"
	1286	+ }
	1287	+ }
	1288	+ },
	1289	+ {
	1290	+ "location": "Raleigh",
	1291	+ "address": {
	1292	+ "postalAddress": {
	1293	+ "addressRegion": "North Carolina",
	1294	+ "addressCountry": "USA",
	1295	+ "addressLocality": "Raleigh"
	1296	+ }
	1297	+ }
	1298	+ }
	1299	+ ],
	1300	+ "publishedAt": "2026-04-30T22:46:28.314+00:00",
	1301	+ "isListed": true,
	1302	+ "isRemote": true,
	1303	+ "workplaceType": "Remote",
	1304	+ "address": {
	1305	+ "postalAddress": {
	1306	+ "addressRegion": "",
	1307	+ "addressCountry": "United States",
	1308	+ "addressLocality": ""
	1309	+ }
	1310	+ },
	1311	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/6a15668c-3f98-4404-8f1f-927cddfc5af7",
	1312	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/6a15668c-3f98-4404-8f1f-927cddfc5af7/application"
	1313	+ },
	1314	+ {
	1315	+ "id": "272bc3f4-5af6-4c14-b797-a424b62d306c",
	1316	+ "title": "Senior Product Engineer - Canada",
	1317	+ "department": "Engineering",
	1318	+ "team": "Americas Engineering",
	1319	+ "employmentType": "FullTime",
	1320	+ "location": "Remote - Canada",
	1321	+ "secondaryLocations": [
	1322	+ {
	1323	+ "location": "Waterloo",
	1324	+ "address": {
	1325	+ "postalAddress": {
	1326	+ "addressRegion": "Ontario",
	1327	+ "addressCountry": "Canada",
	1328	+ "addressLocality": "Waterloo"
	1329	+ }
	1330	+ }
	1331	+ },
	1332	+ {
	1333	+ "location": "Montreal",
	1334	+ "address": {
	1335	+ "postalAddress": {
	1336	+ "addressRegion": "Quebec",
	1337	+ "addressCountry": "Canada",
	1338	+ "addressLocality": "Montreal"
	1339	+ }
	1340	+ }
	1341	+ },
	1342	+ {
	1343	+ "location": "Calgary",
	1344	+ "address": {
	1345	+ "postalAddress": {
	1346	+ "addressRegion": "Alberta",
	1347	+ "addressCountry": "Canada",
	1348	+ "addressLocality": "Calgary"
	1349	+ }
	1350	+ }
	1351	+ },
	1352	+ {
	1353	+ "location": "Vancouver",
	1354	+ "address": {
	1355	+ "postalAddress": {
	1356	+ "addressRegion": "British Columbia",
	1357	+ "addressCountry": "Canada",
	1358	+ "addressLocality": "Vancouver"
	1359	+ }
	1360	+ }
	1361	+ },
	1362	+ {
	1363	+ "location": "Toronto",
	1364	+ "address": {
	1365	+ "postalAddress": {
	1366	+ "addressRegion": "Ontario",
	1367	+ "addressCountry": "Canada",
	1368	+ "addressLocality": "Toronto"
	1369	+ }
	1370	+ }
	1371	+ },
	1372	+ {
	1373	+ "location": "Edmonton",
	1374	+ "address": {
	1375	+ "postalAddress": {
	1376	+ "addressRegion": "Alberta",
	1377	+ "addressCountry": "Canada",
	1378	+ "addressLocality": "Edmonton"
	1379	+ }
	1380	+ }
	1381	+ }
	1382	+ ],
	1383	+ "publishedAt": "2026-04-30T22:46:21.621+00:00",
	1384	+ "isListed": true,
	1385	+ "isRemote": true,
	1386	+ "workplaceType": "Remote",
	1387	+ "address": {
	1388	+ "postalAddress": {
	1389	+ "addressRegion": "",
	1390	+ "addressCountry": "Canada",
	1391	+ "addressLocality": ""
	1392	+ }
	1393	+ },
	1394	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/272bc3f4-5af6-4c14-b797-a424b62d306c",
	1395	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/272bc3f4-5af6-4c14-b797-a424b62d306c/application"
	1396	+ },
	1397	+ {
	1398	+ "id": "751768c6-8d97-4e91-a57d-6b85ce4b5a37",
	1399	+ "title": "Senior Product Engineer - Americas",
	1400	+ "department": "Engineering",
	1401	+ "team": "Americas Engineering",
	1402	+ "employmentType": "FullTime",
	1403	+ "location": "Remote - US",
	1404	+ "secondaryLocations": [
	1405	+ {
	1406	+ "location": "Austin",
	1407	+ "address": {
	1408	+ "postalAddress": {
	1409	+ "addressRegion": "Texas",
	1410	+ "addressCountry": "USA",
	1411	+ "addressLocality": "Austin"
	1412	+ }
	1413	+ }
	1414	+ },
	1415	+ {
	1416	+ "location": "Boulder",
	1417	+ "address": {
	1418	+ "postalAddress": {
	1419	+ "addressRegion": "Colorado",
	1420	+ "addressCountry": "USA",
	1421	+ "addressLocality": "Boulder"
	1422	+ }
	1423	+ }
	1424	+ },
	1425	+ {
	1426	+ "location": "Los Angeles",
	1427	+ "address": {
	1428	+ "postalAddress": {
	1429	+ "addressRegion": "California",
	1430	+ "addressCountry": "USA",
	1431	+ "addressLocality": "Los Angeles"
	1432	+ }
	1433	+ }
	1434	+ },
	1435	+ {
	1436	+ "location": "Portland",
	1437	+ "address": {
	1438	+ "postalAddress": {
	1439	+ "addressRegion": "Oregon",
	1440	+ "addressCountry": "USA",
	1441	+ "addressLocality": "Portland"
	1442	+ }
	1443	+ }
	1444	+ },
	1445	+ {
	1446	+ "location": "Atlanta",
	1447	+ "address": {
	1448	+ "postalAddress": {
	1449	+ "addressRegion": "Georgia",
	1450	+ "addressCountry": "USA",
	1451	+ "addressLocality": "Atlanta"
	1452	+ }
	1453	+ }
	1454	+ },
	1455	+ {
	1456	+ "location": "Salt Lake City",
	1457	+ "address": {
	1458	+ "postalAddress": {
	1459	+ "addressRegion": "Utah",
	1460	+ "addressCountry": "USA",
	1461	+ "addressLocality": "Salt Lake City"
	1462	+ }
	1463	+ }
	1464	+ },
	1465	+ {
	1466	+ "location": "Boston",
	1467	+ "address": {
	1468	+ "postalAddress": {
	1469	+ "addressRegion": "Massachusetts",
	1470	+ "addressCountry": "USA",
	1471	+ "addressLocality": "Boston"
	1472	+ }
	1473	+ }
	1474	+ },
	1475	+ {
	1476	+ "location": "Seattle",
	1477	+ "address": {
	1478	+ "postalAddress": {
	1479	+ "addressRegion": "Washington",
	1480	+ "addressCountry": "USA",
	1481	+ "addressLocality": "Seattle"
	1482	+ }
	1483	+ }
	1484	+ },
	1485	+ {
	1486	+ "location": "New York",
	1487	+ "address": {
	1488	+ "postalAddress": {
	1489	+ "addressRegion": "New York",
	1490	+ "addressCountry": "USA",
	1491	+ "addressLocality": "New York"
	1492	+ }
	1493	+ }
	1494	+ },
	1495	+ {
	1496	+ "location": "San Francisco",
	1497	+ "address": {
	1498	+ "postalAddress": {
	1499	+ "addressRegion": "CA",
	1500	+ "addressCountry": "USA",
	1501	+ "addressLocality": "San Francisco"
	1502	+ }
	1503	+ }
	1504	+ },
	1505	+ {
	1506	+ "location": "Denver",
	1507	+ "address": {
	1508	+ "postalAddress": {
	1509	+ "addressRegion": "Colorado",
	1510	+ "addressCountry": "USA",
	1511	+ "addressLocality": "Denver"
	1512	+ }
	1513	+ }
	1514	+ },
	1515	+ {
	1516	+ "location": "Ann Arbor",
	1517	+ "address": {
	1518	+ "postalAddress": {
	1519	+ "addressRegion": "Michigan",
	1520	+ "addressCountry": "USA",
	1521	+ "addressLocality": "Ann Arbor"
	1522	+ }
	1523	+ }
	1524	+ },
	1525	+ {
	1526	+ "location": "Chicago",
	1527	+ "address": {
	1528	+ "postalAddress": {
	1529	+ "addressRegion": "Illinois",
	1530	+ "addressCountry": "USA",
	1531	+ "addressLocality": "Chicago"
	1532	+ }
	1533	+ }
	1534	+ },
	1535	+ {
	1536	+ "location": "Raleigh",
	1537	+ "address": {
	1538	+ "postalAddress": {
	1539	+ "addressRegion": "North Carolina",
	1540	+ "addressCountry": "USA",
	1541	+ "addressLocality": "Raleigh"
	1542	+ }
	1543	+ }
	1544	+ }
	1545	+ ],
	1546	+ "publishedAt": "2026-04-30T22:46:14.287+00:00",
	1547	+ "isListed": true,
	1548	+ "isRemote": true,
	1549	+ "workplaceType": "Remote",
	1550	+ "address": {
	1551	+ "postalAddress": {
	1552	+ "addressRegion": "",
	1553	+ "addressCountry": "United States",
	1554	+ "addressLocality": ""
	1555	+ }
	1556	+ },
	1557	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/751768c6-8d97-4e91-a57d-6b85ce4b5a37",
	1558	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/751768c6-8d97-4e91-a57d-6b85ce4b5a37/application"
	1559	+ },
	1560	+ {
	1561	+ "id": "55078a90-f1f0-4190-9d2a-c14d1cbc5486",
	1562	+ "title": "Product Support Specialist - Australia",
	1563	+ "department": "Customer Success",
	1564	+ "team": "Customer Support",
	1565	+ "employmentType": "FullTime",
	1566	+ "location": "Australia",
	1567	+ "secondaryLocations": [],
	1568	+ "publishedAt": "2026-02-20T17:26:27.704+00:00",
	1569	+ "isListed": true,
	1570	+ "isRemote": true,
	1571	+ "workplaceType": "Remote",
	1572	+ "address": {
	1573	+ "postalAddress": {
	1574	+ "addressRegion": "",
	1575	+ "addressCountry": "Australia",
	1576	+ "addressLocality": ""
	1577	+ }
	1578	+ },
	1579	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/55078a90-f1f0-4190-9d2a-c14d1cbc5486",
	1580	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/55078a90-f1f0-4190-9d2a-c14d1cbc5486/application"
	1581	+ },
	1582	+ {
	1583	+ "id": "b64e1498-04ae-4747-8930-a42d4ce6c047",
	1584	+ "title": "Product Support Specialist - Americas",
	1585	+ "department": "Customer Success",
	1586	+ "team": "Customer Support",
	1587	+ "employmentType": "FullTime",
	1588	+ "location": "Remote - US",
	1589	+ "secondaryLocations": [
	1590	+ {
	1591	+ "location": "Remote - Canada",
	1592	+ "address": {
	1593	+ "postalAddress": {
	1594	+ "addressRegion": "",
	1595	+ "addressCountry": "Canada",
	1596	+ "addressLocality": ""
	1597	+ }
	1598	+ }
	1599	+ }
	1600	+ ],
	1601	+ "publishedAt": "2026-02-20T13:14:50.065+00:00",
	1602	+ "isListed": true,
	1603	+ "isRemote": true,
	1604	+ "workplaceType": "Remote",
	1605	+ "address": {
	1606	+ "postalAddress": {
	1607	+ "addressRegion": "",
	1608	+ "addressCountry": "United States",
	1609	+ "addressLocality": ""
	1610	+ }
	1611	+ },
	1612	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/b64e1498-04ae-4747-8930-a42d4ce6c047",
	1613	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/b64e1498-04ae-4747-8930-a42d4ce6c047/application"
	1614	+ },
	1615	+ {
	1616	+ "id": "3b9b0141-86e7-46b2-b6b4-ed039b84cdc4",
	1617	+ "title": "Senior Software Engineer, Product Engineering - EU",
	1618	+ "department": "Engineering",
	1619	+ "team": "EMEA Engineering",
	1620	+ "employmentType": "FullTime",
	1621	+ "location": "Remote - European Union",
	1622	+ "secondaryLocations": [
	1623	+ {
	1624	+ "location": "Spain",
	1625	+ "address": {
	1626	+ "postalAddress": {
	1627	+ "addressRegion": "Spain",
	1628	+ "addressCountry": "Spain",
	1629	+ "addressLocality": "Spain"
	1630	+ }
	1631	+ }
	1632	+ },
	1633	+ {
	1634	+ "location": "Italy",
	1635	+ "address": {
	1636	+ "postalAddress": {
	1637	+ "addressRegion": "",
	1638	+ "addressCountry": "Italy",
	1639	+ "addressLocality": ""
	1640	+ }
	1641	+ }
	1642	+ },
	1643	+ {
	1644	+ "location": "Switzerland",
	1645	+ "address": {
	1646	+ "postalAddress": {
	1647	+ "addressRegion": "",
	1648	+ "addressCountry": "Switzerland",
	1649	+ "addressLocality": ""
	1650	+ }
	1651	+ }
	1652	+ },
	1653	+ {
	1654	+ "location": "Croatia",
	1655	+ "address": {
	1656	+ "postalAddress": {
	1657	+ "addressRegion": "",
	1658	+ "addressCountry": "Croatia",
	1659	+ "addressLocality": ""
	1660	+ }
	1661	+ }
	1662	+ },
	1663	+ {
	1664	+ "location": "Ireland",
	1665	+ "address": {
	1666	+ "postalAddress": {
	1667	+ "addressRegion": "",
	1668	+ "addressCountry": "Ireland",
	1669	+ "addressLocality": ""
	1670	+ }
	1671	+ }
	1672	+ },
	1673	+ {
	1674	+ "location": "Stockholm",
	1675	+ "address": {
	1676	+ "postalAddress": {
	1677	+ "addressRegion": "Stockholm",
	1678	+ "addressCountry": "Sweden",
	1679	+ "addressLocality": "Stockholm"
	1680	+ }
	1681	+ }
	1682	+ },
	1683	+ {
	1684	+ "location": "Romania",
	1685	+ "address": {
	1686	+ "postalAddress": {
	1687	+ "addressRegion": "",
	1688	+ "addressCountry": "Romania",
	1689	+ "addressLocality": ""
	1690	+ }
	1691	+ }
	1692	+ },
	1693	+ {
	1694	+ "location": "Barcelona",
	1695	+ "address": {
	1696	+ "postalAddress": {
	1697	+ "addressRegion": "Catalonia",
	1698	+ "addressCountry": "Spain",
	1699	+ "addressLocality": "Barcelona"
	1700	+ }
	1701	+ }
	1702	+ },
	1703	+ {
	1704	+ "location": "Portugal",
	1705	+ "address": {
	1706	+ "postalAddress": {
	1707	+ "addressCountry": "Portugal"
	1708	+ }
	1709	+ }
	1710	+ },
	1711	+ {
	1712	+ "location": "Berlin",
	1713	+ "address": {
	1714	+ "postalAddress": {
	1715	+ "addressRegion": "",
	1716	+ "addressCountry": "Germany",
	1717	+ "addressLocality": "Berlin"
	1718	+ }
	1719	+ }
	1720	+ },
	1721	+ {
	1722	+ "location": "Sweden",
	1723	+ "address": {
	1724	+ "postalAddress": {
	1725	+ "addressRegion": "",
	1726	+ "addressCountry": "Sweden",
	1727	+ "addressLocality": ""
	1728	+ }
	1729	+ }
	1730	+ },
	1731	+ {
	1732	+ "location": "Estonia",
	1733	+ "address": {
	1734	+ "postalAddress": {
	1735	+ "addressRegion": "",
	1736	+ "addressCountry": "Estonia",
	1737	+ "addressLocality": ""
	1738	+ }
	1739	+ }
	1740	+ }
	1741	+ ],
	1742	+ "publishedAt": "2026-04-30T19:45:44.774+00:00",
	1743	+ "isListed": true,
	1744	+ "isRemote": true,
	1745	+ "workplaceType": "Remote",
	1746	+ "address": {
	1747	+ "postalAddress": {
	1748	+ "postalCode": "",
	1749	+ "addressRegion": "",
	1750	+ "addressCountry": "European Union",
	1751	+ "addressLocality": ""
	1752	+ }
	1753	+ },
	1754	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/3b9b0141-86e7-46b2-b6b4-ed039b84cdc4",
	1755	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/3b9b0141-86e7-46b2-b6b4-ed039b84cdc4/application"
	1756	+ },
	1757	+ {
	1758	+ "id": "c3c7125d-7883-4dff-a2bf-f5a55de4a364",
	1759	+ "title": "Staff Software Engineer, Product Engineering - EU",
	1760	+ "department": "Engineering",
	1761	+ "team": "EMEA Engineering",
	1762	+ "employmentType": "FullTime",
	1763	+ "location": "Remote - European Union",
	1764	+ "secondaryLocations": [
	1765	+ {
	1766	+ "location": "Spain",
	1767	+ "address": {
	1768	+ "postalAddress": {
	1769	+ "addressRegion": "Spain",
	1770	+ "addressCountry": "Spain",
	1771	+ "addressLocality": "Spain"
	1772	+ }
	1773	+ }
	1774	+ },
	1775	+ {
	1776	+ "location": "Italy",
	1777	+ "address": {
	1778	+ "postalAddress": {
	1779	+ "addressRegion": "",
	1780	+ "addressCountry": "Italy",
	1781	+ "addressLocality": ""
	1782	+ }
	1783	+ }
	1784	+ },
	1785	+ {
	1786	+ "location": "Switzerland",
	1787	+ "address": {
	1788	+ "postalAddress": {
	1789	+ "addressRegion": "",
	1790	+ "addressCountry": "Switzerland",
	1791	+ "addressLocality": ""
	1792	+ }
	1793	+ }
	1794	+ },
	1795	+ {
	1796	+ "location": "Croatia",
	1797	+ "address": {
	1798	+ "postalAddress": {
	1799	+ "addressRegion": "",
	1800	+ "addressCountry": "Croatia",
	1801	+ "addressLocality": ""
	1802	+ }
	1803	+ }
	1804	+ },
	1805	+ {
	1806	+ "location": "Ireland",
	1807	+ "address": {
	1808	+ "postalAddress": {
	1809	+ "addressRegion": "Ireland",
	1810	+ "addressCountry": "Ireland",
	1811	+ "addressLocality": "Ireland"
	1812	+ }
	1813	+ }
	1814	+ },
	1815	+ {
	1816	+ "location": "Stockholm",
	1817	+ "address": {
	1818	+ "postalAddress": {
	1819	+ "addressRegion": "Stockholm",
	1820	+ "addressCountry": "Sweden",
	1821	+ "addressLocality": "Stockholm"
	1822	+ }
	1823	+ }
	1824	+ },
	1825	+ {
	1826	+ "location": "Romania",
	1827	+ "address": {
	1828	+ "postalAddress": {
	1829	+ "addressRegion": "",
	1830	+ "addressCountry": "Romania",
	1831	+ "addressLocality": ""
	1832	+ }
	1833	+ }
	1834	+ },
	1835	+ {
	1836	+ "location": "Barcelona",
	1837	+ "address": {
	1838	+ "postalAddress": {
	1839	+ "addressRegion": "Catalonia",
	1840	+ "addressCountry": "Spain",
	1841	+ "addressLocality": "Barcelona"
	1842	+ }
	1843	+ }
	1844	+ },
	1845	+ {
	1846	+ "location": "Portugal",
	1847	+ "address": {
	1848	+ "postalAddress": {
	1849	+ "addressCountry": "Portugal"
	1850	+ }
	1851	+ }
	1852	+ },
	1853	+ {
	1854	+ "location": "Berlin",
	1855	+ "address": {
	1856	+ "postalAddress": {
	1857	+ "addressRegion": "",
	1858	+ "addressCountry": "Germany",
	1859	+ "addressLocality": "Berlin"
	1860	+ }
	1861	+ }
	1862	+ },
	1863	+ {
	1864	+ "location": "Sweden",
	1865	+ "address": {
	1866	+ "postalAddress": {
	1867	+ "addressRegion": "",
	1868	+ "addressCountry": "Sweden",
	1869	+ "addressLocality": ""
	1870	+ }
	1871	+ }
	1872	+ },
	1873	+ {
	1874	+ "location": "Estonia",
	1875	+ "address": {
	1876	+ "postalAddress": {
	1877	+ "addressRegion": "",
	1878	+ "addressCountry": "Estonia",
	1879	+ "addressLocality": ""
	1880	+ }
	1881	+ }
	1882	+ }
	1883	+ ],
	1884	+ "publishedAt": "2026-04-30T19:45:51.749+00:00",
	1885	+ "isListed": true,
	1886	+ "isRemote": true,
	1887	+ "workplaceType": "Remote",
	1888	+ "address": {
	1889	+ "postalAddress": {
	1890	+ "postalCode": "",
	1891	+ "addressRegion": "",
	1892	+ "addressCountry": "European Union",
	1893	+ "addressLocality": ""
	1894	+ }
	1895	+ },
	1896	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/c3c7125d-7883-4dff-a2bf-f5a55de4a364",
	1897	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/c3c7125d-7883-4dff-a2bf-f5a55de4a364/application"
	1898	+ },
	1899	+ {
	1900	+ "id": "0020099f-9bb3-4da9-9808-4556564f5301",
	1901	+ "title": "Staff Software Engineer, Product Engineering - UK",
	1902	+ "department": "Engineering",
	1903	+ "team": "EMEA Engineering",
	1904	+ "employmentType": "FullTime",
	1905	+ "location": "United Kingdom",
	1906	+ "secondaryLocations": [
	1907	+ {
	1908	+ "location": "Manchester",
	1909	+ "address": {
	1910	+ "postalAddress": {
	1911	+ "addressRegion": "",
	1912	+ "addressCountry": "United Kingdom",
	1913	+ "addressLocality": "Manchester"
	1914	+ }
	1915	+ }
	1916	+ },
	1917	+ {
	1918	+ "location": "Oxford",
	1919	+ "address": {
	1920	+ "postalAddress": {
	1921	+ "addressRegion": "",
	1922	+ "addressCountry": "United Kingdom",
	1923	+ "addressLocality": "Oxford"
	1924	+ }
	1925	+ }
	1926	+ },
	1927	+ {
	1928	+ "location": "London",
	1929	+ "address": {
	1930	+ "postalAddress": {
	1931	+ "addressRegion": "",
	1932	+ "addressCountry": "United Kingdom",
	1933	+ "addressLocality": "London"
	1934	+ }
	1935	+ }
	1936	+ },
	1937	+ {
	1938	+ "location": "Cambridge",
	1939	+ "address": {
	1940	+ "postalAddress": {
	1941	+ "addressRegion": "",
	1942	+ "addressCountry": "United Kingdom",
	1943	+ "addressLocality": "Cambridge"
	1944	+ }
	1945	+ }
	1946	+ }
	1947	+ ],
	1948	+ "publishedAt": "2026-04-30T19:45:49.508+00:00",
	1949	+ "isListed": true,
	1950	+ "isRemote": true,
	1951	+ "workplaceType": "Remote",
	1952	+ "address": {
	1953	+ "postalAddress": {
	1954	+ "addressRegion": "",
	1955	+ "addressCountry": "United Kingdom",
	1956	+ "addressLocality": ""
	1957	+ }
	1958	+ },
	1959	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/0020099f-9bb3-4da9-9808-4556564f5301",
	1960	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/0020099f-9bb3-4da9-9808-4556564f5301/application"
	1961	+ },
	1962	+ {
	1963	+ "id": "472eef28-6e52-43b4-9bff-9113522890f5",
	1964	+ "title": "Senior Software Engineer, Product Engineering - UK",
	1965	+ "department": "Engineering",
	1966	+ "team": "EMEA Engineering",
	1967	+ "employmentType": "FullTime",
	1968	+ "location": "United Kingdom",
	1969	+ "secondaryLocations": [
	1970	+ {
	1971	+ "location": "Manchester",
	1972	+ "address": {
	1973	+ "postalAddress": {
	1974	+ "addressRegion": "",
	1975	+ "addressCountry": "United Kingdom",
	1976	+ "addressLocality": "Manchester"
	1977	+ }
	1978	+ }
	1979	+ },
	1980	+ {
	1981	+ "location": "Oxford",
	1982	+ "address": {
	1983	+ "postalAddress": {
	1984	+ "addressRegion": "",
	1985	+ "addressCountry": "United Kingdom",
	1986	+ "addressLocality": "Oxford"
	1987	+ }
	1988	+ }
	1989	+ },
	1990	+ {
	1991	+ "location": "London",
	1992	+ "address": {
	1993	+ "postalAddress": {
	1994	+ "addressRegion": "",
	1995	+ "addressCountry": "United Kingdom",
	1996	+ "addressLocality": "London"
	1997	+ }
	1998	+ }
	1999	+ },
	2000	+ {
	2001	+ "location": "Cambridge",
	2002	+ "address": {
	2003	+ "postalAddress": {
	2004	+ "addressRegion": "",
	2005	+ "addressCountry": "United Kingdom",
	2006	+ "addressLocality": "Cambridge"
	2007	+ }
	2008	+ }
	2009	+ }
	2010	+ ],
	2011	+ "publishedAt": "2026-04-30T19:45:47.450+00:00",
	2012	+ "isListed": true,
	2013	+ "isRemote": true,
	2014	+ "workplaceType": "Remote",
	2015	+ "address": {
	2016	+ "postalAddress": {
	2017	+ "addressRegion": "",
	2018	+ "addressCountry": "United Kingdom",
	2019	+ "addressLocality": ""
	2020	+ }
	2021	+ },
	2022	+ "jobUrl": "https://jobs.ashbyhq.com/ashby/472eef28-6e52-43b4-9bff-9113522890f5",
	2023	+ "applyUrl": "https://jobs.ashbyhq.com/ashby/472eef28-6e52-43b4-9bff-9113522890f5/application"
	2024	+ }
	2025	+ ],
	2026	+ "apiVersion": "1"
	2027	+}
		\ No newline at end of file

added fixtures/connectors/feed/blog_atom.xml +26 −0

		`@@ -0,0 +1,26 @@`
	1	+<?xml version="1.0" encoding="utf-8"?>
	2	+<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
	3	+ <title>Acme Cloud Blog</title>
	4	+ <link href="https://www.acme-cloud.example/blog"/>
	5	+ <updated>2026-09-11T10:00:00Z</updated>
	6	+ <entry>
	7	+ <title>Introducing Atlas AI</title>
	8	+ <link href="https://www.acme-cloud.example/blog/introducing-atlas-ai?utm_source=feed"/>
	9	+ <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
	10	+ <published>2026-09-10T09:00:00Z</published>
	11	+ <summary>Atlas AI is a new assistant that explains incidents in plain language.</summary>
	12	+ </entry>
	13	+ <entry>
	14	+ <title>How we cut query latency by 40%</title>
	15	+ <link href="https://www.acme-cloud.example/blog/query-latency"/>
	16	+ <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6b</id>
	17	+ <published>2026-08-21T09:00:00Z</published>
	18	+ <content type="html"><p>A deep dive into our storage engine.</p></content>
	19	+ </entry>
	20	+ <entry>
	21	+ <title>Acme Cloud is now SOC 2 Type II certified</title>
	22	+ <link href="https://www.acme-cloud.example/blog/soc2"/>
	23	+ <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6c</id>
	24	+ <published>2026-07-01T09:00:00Z</published>
	25	+ </entry>
	26	+</feed>

added fixtures/connectors/generic_html/careers.html +23 −0

		`@@ -0,0 +1,23 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head><title>Careers at Acme Cloud</title></head>
	4	+<body>
	5	+<nav><a href="/">Home</a><a href="/careers">Careers</a><a href="/about">About</a></nav>
	6	+<main>
	7	+ <section class="hero"><h1>Join us</h1><p>We are hiring across engineering, sales and operations.</p></section>
	8	+ <h2>Open positions</h2>
	9	+ <table class="jobs-table">
	10	+ <thead><tr><th>Role</th><th>Team</th><th>Location</th></tr></thead>
	11	+ <tbody>
	12	+ <tr class="job-row"><td><a href="/careers/senior-backend-engineer-1234">Senior Backend Engineer</a></td><td>Engineering</td><td>Toronto, ON, Canada</td></tr>
	13	+ <tr class="job-row"><td><a href="/careers/machine-learning-engineer-1235">Machine Learning Engineer</a></td><td>Engineering</td><td>Remote - US</td></tr>
	14	+ <tr class="job-row"><td><a href="/careers/account-executive-emea-1236">Account Executive, EMEA</a></td><td>Sales</td><td>London, United Kingdom</td></tr>
	15	+ <tr class="job-row"><td><a href="/careers/people-operations-lead-1237">People Operations Lead</a></td><td>People</td><td>Berlin, Germany</td></tr>
	16	+ <tr class="job-row"><td><a href="/careers/product-designer-1238">Product Designer</a></td><td>Design</td><td>San Francisco, CA</td></tr>
	17	+ </tbody>
	18	+ </table>
	19	+ <p><a href="https://boards.greenhouse.io/acmecloud">See all openings on our job board</a></p>
	20	+ <iframe src="https://boards.greenhouse.io/embed/job_board?for=acmecloud"></iframe>
	21	+</main>
	22	+<footer><p>© 2026 Acme Cloud</p></footer>
	23	+</body></html>

added fixtures/connectors/generic_html/careers_jsonld.html +12 −0

		`@@ -0,0 +1,12 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head><title>Jobs — Acme Cloud</title>
	4	+<script type="application/ld+json">[
	5	+ {"@context":"https://schema.org","@type":"JobPosting","title":"Staff Site Reliability Engineer","datePosted":"2026-09-01","identifier":{"@type":"PropertyValue","name":"Acme","value":"REQ-501"},
	6	+ "employmentType":"FULL_TIME","hiringOrganization":{"@type":"Organization","name":"Acme Cloud"},"url":"https://www.acme-cloud.example/careers/req-501",
	7	+ "jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Toronto","addressRegion":"ON","addressCountry":"CA"}}},
	8	+ {"@context":"https://schema.org","@type":"JobPosting","title":"Generative AI Product Manager","datePosted":"2026-09-05","identifier":{"@type":"PropertyValue","value":"REQ-502"},
	9	+ "employmentType":["FULL_TIME"],"jobLocationType":"TELECOMMUTE","url":"https://www.acme-cloud.example/careers/req-502",
	10	+ "applicantLocationRequirements":{"@type":"Country","name":"USA"},"baseSalary":{"@type":"MonetaryAmount","currency":"USD","value":{"@type":"QuantitativeValue","minValue":180000,"maxValue":230000,"unitText":"YEAR"}}}
	11	+]</script></head>
	12	+<body><main><h1>Jobs</h1><p>Two roles are open today.</p></main></body></html>

added fixtures/connectors/generic_html/careers_v2.html +23 −0

		`@@ -0,0 +1,23 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head><title>Careers at Acme Cloud</title></head>
	4	+<body>
	5	+<nav><a href="/">Home</a><a href="/careers">Careers</a><a href="/about">About</a></nav>
	6	+<main>
	7	+ <section class="hero"><h1>Join us</h1><p>We are hiring across engineering, sales and operations.</p></section>
	8	+ <h2>Open positions</h2>
	9	+ <table class="jobs-table">
	10	+ <thead><tr><th>Role</th><th>Team</th><th>Location</th></tr></thead>
	11	+ <tbody>
	12	+ <tr class="job-row"><td><a href="/careers/senior-backend-engineer-1234">Senior Backend Engineer</a></td><td>Engineering</td><td>Toronto, ON, Canada</td></tr>
	13	+ <tr class="job-row"><td><a href="/careers/machine-learning-engineer-1235">Machine Learning Engineer</a></td><td>Engineering</td><td>Remote - US</td></tr>
	14	+ <tr class="job-row"><td><a href="/careers/account-executive-emea-1236">Account Executive, EMEA</a></td><td>Sales</td><td>London, United Kingdom</td></tr>
	15	+ <tr class="job-row"><td><a href="/careers/product-designer-1238">Product Designer</a></td><td>Design</td><td>San Francisco, CA</td></tr>
	16	+ <tr class="job-row"><td><a href="/careers/ai-research-scientist-1239">AI Research Scientist</a></td><td>Research</td><td>Paris, France</td></tr>
	17	+ <tr class="job-row"><td><a href="/careers/data-platform-engineer-1240">Data Platform Engineer</a></td><td>Engineering</td><td>Toronto, ON, Canada</td></tr>
	18	+ </tbody>
	19	+ </table>
	20	+ <p><a href="https://boards.greenhouse.io/acmecloud">See all openings on our job board</a></p>
	21	+</main>
	22	+<footer><p>© 2026 Acme Cloud</p></footer>
	23	+</body></html>

added fixtures/connectors/generic_html/homepage.html +35 −0

		`@@ -0,0 +1,35 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head>
	4	+ <title>Acme Cloud — Infrastructure observability for modern teams</title>
	5	+ <meta name="description" content="Acme Cloud gives engineering teams one place to monitor, debug and optimise their infrastructure.">
	6	+ <meta property="og:site_name" content="Acme Cloud">
	7	+ <link rel="canonical" href="https://www.acme-cloud.example/">
	8	+ <link rel="alternate" type="application/atom+xml" href="https://www.acme-cloud.example/blog/atom.xml">
	9	+ <script type="application/ld+json">{"@context":"https://schema.org","@type":"Organization","name":"Acme Cloud","url":"https://www.acme-cloud.example","sameAs":["https://github.com/acme-cloud","https://www.linkedin.com/company/acme-cloud"]}</script>
	10	+</head>
	11	+<body>
	12	+<header class="global-header">
	13	+ <nav>
	14	+ <a href="/product">Product</a><a href="/solutions">Solutions</a><a href="/pricing">Pricing</a><a href="/docs">Documentation</a>
	15	+ <a href="/customers">Customers</a><a href="/company/about">About us</a><a href="/careers">Careers</a><a href="/news">Newsroom</a>
	16	+ <a href="https://developers.acme-cloud.example">Developers</a><a href="/login">Log in</a>
	17	+ </nav>
	18	+</header>
	19	+<main>
	20	+ <section class="hero"><h1>See everything. Fix anything.</h1><p>One platform for metrics, logs and traces.</p><a href="/signup">Start free</a></section>
	21	+ <section><h2>Trusted by 4,000+ teams</h2><p>From startups to the Fortune 500.</p></section>
	22	+ <section class="products">
	23	+ <h2>Products</h2>
	24	+ <div class="product-card"><h3>Atlas Metrics</h3><p>High-cardinality metrics at any scale.</p><a href="/product/metrics">Learn more</a></div>
	25	+ <div class="product-card"><h3>Atlas Logs</h3><p>Search petabytes in seconds.</p><a href="/product/logs">Learn more</a></div>
	26	+ <div class="product-card"><h3>Atlas Traces</h3><p>Distributed tracing without sampling.</p><a href="/product/traces">Learn more</a></div>
	27	+ </section>
	28	+</main>
	29	+<footer class="site-footer">
	30	+ <div><a href="/company/about">About</a><a href="/about/leadership">Leadership</a><a href="/company/locations">Locations</a><a href="/investors">Investors</a>
	31	+ <a href="/changelog">Changelog</a><a href="/security">Security</a><a href="/legal/terms">Terms of Service</a><a href="/legal/privacy">Privacy Policy</a>
	32	+ <a href="https://status.acme-cloud.example">Status</a><a href="/sustainability">Sustainability</a><a href="/partners">Partners</a></div>
	33	+ <p>© 2026 Acme Cloud, Inc.</p>
	34	+</footer>
	35	+</body></html>

added fixtures/connectors/generic_html/leadership.html +27 −0

		`@@ -0,0 +1,27 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head><title>Leadership \| Acme Cloud</title><meta property="og:title" content="Our leadership team"></head>
	4	+<body>
	5	+<nav class="navbar"><a href="/">Acme</a><a href="/about">About</a><a href="/about/leadership">Leadership</a><a href="/news">Newsroom</a></nav>
	6	+<main id="main">
	7	+ <h1>Leadership team</h1>
	8	+ <p>Meet the people guiding Acme Cloud.</p>
	9	+ <section class="team-grid">
	10	+ <h2>Executive team</h2>
	11	+ <div class="team-member"><img src="/img/jane.jpg" alt=""><h3>Jane Doe</h3><p class="role">Co-Founder & Chief Executive Officer</p><p>Jane founded Acme in 2015 after a decade at BigCo.</p></div>
	12	+ <div class="team-member"><h3>Rahul Mehta</h3><p class="role">Chief Financial Officer</p></div>
	13	+ <div class="team-member"><h3>María García-López</h3><p class="role">Chief Technology Officer</p></div>
	14	+ <div class="team-member"><h3>Tom O'Neill</h3><p class="role">SVP, Global Sales</p></div>
	15	+ <div class="team-member"><h3>Aiko Tanaka</h3><p class="role">Head of People</p></div>
	16	+ </section>
	17	+ <section class="board">
	18	+ <h2>Board of directors</h2>
	19	+ <div class="board-member"><h3>Samuel Adebayo</h3><p class="role">Chairman of the Board</p></div>
	20	+ <div class="board-member"><h3>Lin Wei</h3><p class="role">Independent Director</p></div>
	21	+ </section>
	22	+ <script type="application/ld+json">{"@context":"https://schema.org","@type":"Organization","name":"Acme Cloud","url":"https://www.acme-cloud.example",
	23	+ "sameAs":["https://www.linkedin.com/company/acme-cloud","https://twitter.com/acmecloud"],
	24	+ "employee":[{"@type":"Person","name":"Jane Doe","jobTitle":"Chief Executive Officer"}]}</script>
	25	+</main>
	26	+<footer><p>© 2025 Acme Cloud</p></footer>
	27	+</body></html>

added fixtures/connectors/generic_html/legal_terms.html +21 −0

		`@@ -0,0 +1,21 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head><title>Terms of Service — Acme Cloud</title></head>
	4	+<body>
	5	+<nav><a href="/">Home</a><a href="/legal/terms">Terms</a><a href="/legal/privacy">Privacy</a></nav>
	6	+<main>
	7	+ <h1>Terms of Service</h1>
	8	+ <p>Effective date: January 15, 2026</p>
	9	+ <h2>1. Acceptance of terms</h2>
	10	+ <p>By accessing or using the Acme Cloud services you agree to be bound by these terms. If you do not agree, do not use the services.</p>
	11	+ <h2>2. Accounts</h2>
	12	+ <p>You are responsible for safeguarding your account credentials and for all activities that occur under your account.</p>
	13	+ <h2>3. Fees and payment</h2>
	14	+ <p>Fees are billed in advance on a monthly or annual basis and are non-refundable except as required by law.</p>
	15	+ <h2>4. Termination</h2>
	16	+ <p>Either party may terminate this agreement with thirty (30) days written notice.</p>
	17	+ <h2>7. Limitation of liability</h2>
	18	+ <p>To the maximum extent permitted by law, Acme Cloud shall not be liable for indirect, incidental or consequential damages.</p>
	19	+</main>
	20	+<footer><p>© 2026 Acme Cloud</p></footer>
	21	+</body></html>

added fixtures/connectors/generic_html/locations.html +22 −0

		`@@ -0,0 +1,22 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head><title>Our offices — Acme Cloud</title></head>
	4	+<body>
	5	+<header><nav><a href="/">Home</a><a href="/company/locations">Locations</a></nav></header>
	6	+<main>
	7	+ <h1>Where we work</h1>
	8	+ <p>Acme Cloud has teams in eight cities across three continents.</p>
	9	+ <section class="offices">
	10	+ <h2>Offices</h2>
	11	+ <div class="office-card"><h3>San Francisco (Headquarters)</h3><p>548 Market Street, Suite 200<br>San Francisco, CA 94104, United States</p></div>
	12	+ <div class="office-card"><h3>New York</h3><p>New York, NY, USA</p></div>
	13	+ <div class="office-card"><h3>Toronto</h3><p>Toronto, Ontario, Canada</p></div>
	14	+ <div class="office-card"><h3>London</h3><p>1 Finsbury Avenue, London EC2M 2PF, United Kingdom</p></div>
	15	+ <div class="office-card"><h3>Berlin</h3><p>Berlin, Germany</p></div>
	16	+ <div class="office-card"><h3>Singapore</h3><p>Singapore</p></div>
	17	+ <div class="office-card"><h3>Tokyo</h3><p>Tokyo, Japan</p></div>
	18	+ <div class="office-card"><h3>Dublin Data Center</h3><p>Dublin, Ireland</p></div>
	19	+ </section>
	20	+</main>
	21	+<footer><p>© 2025 Acme Cloud</p></footer>
	22	+</body></html>

added fixtures/connectors/generic_html/newsroom.html +17 −0

		`@@ -0,0 +1,17 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head><title>Newsroom \| Acme Cloud</title><meta name="generator" content="WordPress 6.5"></head>
	4	+<body>
	5	+<nav><a href="/">Home</a><a href="/news">Newsroom</a><a href="/press-kit">Press kit</a></nav>
	6	+<main>
	7	+ <h1>Newsroom</h1>
	8	+ <section class="news-list">
	9	+ <article class="news-item"><time datetime="2026-09-10">September 10, 2026</time><h2><a href="/news/2026/09/acme-launches-atlas-ai">Acme launches Atlas AI, an assistant for cloud operations</a></h2><p>Atlas AI helps teams diagnose incidents in seconds.</p></article>
	10	+ <article class="news-item"><time datetime="2026-08-28">August 28, 2026</time><h2><a href="/news/2026/08/acme-opens-tokyo-office">Acme opens new office in Tokyo to serve APAC customers</a></h2></article>
	11	+ <article class="news-item"><time datetime="2026-07-15">July 15, 2026</time><h2><a href="/news/2026/07/series-c">Acme raises $120M Series C led by Example Ventures</a></h2></article>
	12	+ <article class="news-item"><time datetime="2026-06-02">June 2, 2026</time><h2><a href="/news/2026/06/acme-partners-with-bigco">Acme and BigCo announce strategic partnership</a></h2></article>
	13	+ </section>
	14	+ <nav class="pagination"><a href="/news?page=2">Next</a></nav>
	15	+</main>
	16	+<footer><p>© 2026 Acme Cloud</p><a href="/news/feed">RSS</a></footer>
	17	+</body></html>

added fixtures/connectors/generic_html/pricing.html +57 −0

		`@@ -0,0 +1,57 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head>
	4	+ <title>Pricing — Acme Cloud</title>
	5	+ <meta name="description" content="Simple, transparent pricing for teams of every size.">
	6	+ <link rel="canonical" href="https://www.acme-cloud.example/pricing">
	7	+ <link rel="alternate" type="application/rss+xml" title="Acme Blog" href="/blog/feed.xml">
	8	+ <script>window.dataLayer = [{"session": "abc123def456"}];</script>
	9	+ <style>.hero{color:red}</style>
	10	+</head>
	11	+<body>
	12	+<header class="site-header">
	13	+ <nav aria-label="Main">
	14	+ <a href="/">Home</a> <a href="/product">Product</a> <a href="/pricing">Pricing</a> <a href="/customers">Customers</a>
	15	+ <a href="/blog">Blog</a> <a href="/careers">Careers</a> <a href="/docs">Docs</a>
	16	+ </nav>
	17	+</header>
	18	+<div id="onetrust-consent-sdk"><div class="cookie-banner">We use cookies to improve your experience. <button>Accept all</button></div></div>
	19	+<main>
	20	+ <section class="hero">
	21	+ <h1>Pricing that scales with you</h1>
	22	+ <p>Start free, upgrade when you are ready. No credit card required.</p>
	23	+ </section>
	24	+ <section class="pricing-grid">
	25	+ <h2>Plans</h2>
	26	+ <div class="pricing-card">
	27	+ <h3>Starter</h3>
	28	+ <p class="price">$29 <span>per month</span></p>
	29	+ <ul><li>Up to 5 users</li><li>10 GB storage</li><li>Email support</li></ul>
	30	+ <a href="/signup?plan=starter">Start free trial</a>
	31	+ </div>
	32	+ <div class="pricing-card">
	33	+ <h3>Pro</h3>
	34	+ <p class="price">$99 per user / month, billed annually</p>
	35	+ <ul><li>Unlimited users</li><li>1 TB storage</li><li>Priority support</li><li>SSO & SAML</li></ul>
	36	+ <a href="/signup?plan=pro">Buy Pro</a>
	37	+ </div>
	38	+ <div class="pricing-card">
	39	+ <h3>Enterprise</h3>
	40	+ <p class="price">Contact sales</p>
	41	+ <ul><li>Custom contracts</li><li>Dedicated success manager</li><li>99.99% SLA</li></ul>
	42	+ <a href="/contact-sales">Talk to sales</a>
	43	+ </div>
	44	+ </section>
	45	+ <section class="faq">
	46	+ <h2>Frequently asked questions</h2>
	47	+ <details><summary>Can I change plans later?</summary><p>Yes, upgrades and downgrades apply on the next billing cycle.</p></details>
	48	+ <details><summary>Do you offer discounts for nonprofits?</summary><p>Yes, contact us for details.</p></details>
	49	+ </section>
	50	+ <p class="updated" aria-hidden="true">Last updated 3 minutes ago</p>
	51	+</main>
	52	+<footer class="site-footer">
	53	+ <p>© 2025 Acme Cloud, Inc. All rights reserved.</p>
	54	+ <a href="/legal/terms">Terms</a> <a href="/legal/privacy">Privacy</a> <a href="https://status.acme-cloud.example">Status</a>
	55	+</footer>
	56	+</body>
	57	+</html>

added fixtures/connectors/generic_html/pricing_v2.html +52 −0

		`@@ -0,0 +1,52 @@`
	1	+<!doctype html>
	2	+<html lang="en">
	3	+<head>
	4	+ <title>Pricing — Acme Cloud</title>
	5	+ <meta name="description" content="Simple, transparent pricing for teams of every size.">
	6	+ <link rel="canonical" href="https://www.acme-cloud.example/pricing">
	7	+</head>
	8	+<body>
	9	+<header class="site-header">
	10	+ <nav aria-label="Main">
	11	+ <a href="/">Home</a> <a href="/product">Product</a> <a href="/pricing">Pricing</a> <a href="/customers">Customers</a>
	12	+ <a href="/blog">Blog</a> <a href="/careers">Careers</a> <a href="/docs">Docs</a>
	13	+ </nav>
	14	+</header>
	15	+<main>
	16	+ <section class="hero">
	17	+ <h1>Pricing that scales with you</h1>
	18	+ <p>Start free, upgrade when you are ready. No credit card required.</p>
	19	+ </section>
	20	+ <section class="pricing-grid">
	21	+ <h2>Plans</h2>
	22	+ <div class="pricing-card">
	23	+ <h3>Starter</h3>
	24	+ <p class="price">$39 <span>per month</span></p>
	25	+ <ul><li>Up to 5 users</li><li>10 GB storage</li><li>Email support</li></ul>
	26	+ <a href="/signup?plan=starter">Start free trial</a>
	27	+ </div>
	28	+ <div class="pricing-card">
	29	+ <h3>Pro</h3>
	30	+ <p class="price">$99 per user / month, billed annually</p>
	31	+ <ul><li>Unlimited users</li><li>1 TB storage</li><li>Priority support</li><li>SSO & SAML</li></ul>
	32	+ <a href="/signup?plan=pro">Buy Pro</a>
	33	+ </div>
	34	+ <div class="pricing-card">
	35	+ <h3>Enterprise</h3>
	36	+ <p class="price">Contact sales</p>
	37	+ <ul><li>Custom contracts</li><li>Dedicated success manager</li><li>99.99% SLA</li></ul>
	38	+ <a href="/contact-sales">Talk to sales</a>
	39	+ </div>
	40	+ </section>
	41	+ <section class="faq">
	42	+ <h2>Frequently asked questions</h2>
	43	+ <details><summary>Can I change plans later?</summary><p>Yes, upgrades and downgrades apply on the next billing cycle.</p></details>
	44	+ <details><summary>Do you offer discounts for nonprofits?</summary><p>Yes, contact us for details.</p></details>
	45	+ </section>
	46	+</main>
	47	+<footer class="site-footer">
	48	+ <p>© 2026 Acme Cloud, Inc. All rights reserved.</p>
	49	+ <a href="/legal/terms">Terms</a> <a href="/legal/privacy">Privacy</a> <a href="https://status.acme-cloud.example">Status</a>
	50	+</footer>
	51	+</body>
	52	+</html>

added fixtures/connectors/greenhouse/stripe_jobs.json +547 −0

		`@@ -0,0 +1,547 @@`
	1	+{
	2	+ "jobs": [
	3	+ {
	4	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8172487",
	5	+ "data_compliance": [
	6	+ {
	7	+ "type": "gdpr",
	8	+ "requires_consent": false,
	9	+ "requires_processing_consent": false,
	10	+ "requires_retention_consent": false,
	11	+ "retention_period": null,
	12	+ "demographic_data_consent_applies": false
	13	+ }
	14	+ ],
	15	+ "education": "education_required",
	16	+ "internal_job_id": 3537052,
	17	+ "location": {
	18	+ "name": "Dublin"
	19	+ },
	20	+ "metadata": null,
	21	+ "id": 8172487,
	22	+ "updated_at": "2026-09-10T13:11:58-04:00",
	23	+ "requisition_id": "See Opening ID",
	24	+ "title": "Abuse Investigator",
	25	+ "company_name": "Stripe",
	26	+ "first_published": "2026-09-03T13:30:34-04:00",
	27	+ "language": "en",
	28	+ "application_deadline": null
	29	+ },
	30	+ {
	31	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8172510",
	32	+ "data_compliance": [
	33	+ {
	34	+ "type": "gdpr",
	35	+ "requires_consent": false,
	36	+ "requires_processing_consent": false,
	37	+ "requires_retention_consent": false,
	38	+ "retention_period": null,
	39	+ "demographic_data_consent_applies": false
	40	+ }
	41	+ ],
	42	+ "education": "education_required",
	43	+ "internal_job_id": 3537063,
	44	+ "location": {
	45	+ "name": "Seattle, San Francisco, New York City"
	46	+ },
	47	+ "metadata": null,
	48	+ "id": 8172510,
	49	+ "updated_at": "2026-09-10T13:11:58-04:00",
	50	+ "requisition_id": "See Opening ID",
	51	+ "title": "Abuse Investigator",
	52	+ "company_name": "Stripe",
	53	+ "first_published": "2026-09-09T10:50:29-04:00",
	54	+ "language": "en",
	55	+ "application_deadline": null
	56	+ },
	57	+ {
	58	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8172508",
	59	+ "data_compliance": [
	60	+ {
	61	+ "type": "gdpr",
	62	+ "requires_consent": false,
	63	+ "requires_processing_consent": false,
	64	+ "requires_retention_consent": false,
	65	+ "retention_period": null,
	66	+ "demographic_data_consent_applies": false
	67	+ }
	68	+ ],
	69	+ "education": "education_required",
	70	+ "internal_job_id": 3537062,
	71	+ "location": {
	72	+ "name": "Dublin"
	73	+ },
	74	+ "metadata": null,
	75	+ "id": 8172508,
	76	+ "updated_at": "2026-09-10T13:11:58-04:00",
	77	+ "requisition_id": "See Opening ID",
	78	+ "title": "Abuse Investigator",
	79	+ "company_name": "Stripe",
	80	+ "first_published": "2026-09-03T13:32:53-04:00",
	81	+ "language": "en",
	82	+ "application_deadline": null
	83	+ },
	84	+ {
	85	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8172503",
	86	+ "data_compliance": [
	87	+ {
	88	+ "type": "gdpr",
	89	+ "requires_consent": false,
	90	+ "requires_processing_consent": false,
	91	+ "requires_retention_consent": false,
	92	+ "retention_period": null,
	93	+ "demographic_data_consent_applies": false
	94	+ }
	95	+ ],
	96	+ "education": "education_required",
	97	+ "internal_job_id": 3537060,
	98	+ "location": {
	99	+ "name": "Remote from the US"
	100	+ },
	101	+ "metadata": null,
	102	+ "id": 8172503,
	103	+ "updated_at": "2026-09-10T13:11:58-04:00",
	104	+ "requisition_id": "See Opening ID",
	105	+ "title": "Abuse Research Engineer",
	106	+ "company_name": "Stripe",
	107	+ "first_published": "2026-09-09T10:52:09-04:00",
	108	+ "language": "en",
	109	+ "application_deadline": null
	110	+ },
	111	+ {
	112	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=7532733",
	113	+ "data_compliance": [
	114	+ {
	115	+ "type": "gdpr",
	116	+ "requires_consent": false,
	117	+ "requires_processing_consent": false,
	118	+ "requires_retention_consent": false,
	119	+ "retention_period": null,
	120	+ "demographic_data_consent_applies": false
	121	+ }
	122	+ ],
	123	+ "education": "education_required",
	124	+ "internal_job_id": 3336216,
	125	+ "location": {
	126	+ "name": "San Francisco, CA"
	127	+ },
	128	+ "metadata": null,
	129	+ "id": 7532733,
	130	+ "updated_at": "2026-09-10T13:11:58-04:00",
	131	+ "requisition_id": "See Opening ID",
	132	+ "title": "Account Executive, AI Sales",
	133	+ "company_name": "Stripe",
	134	+ "first_published": "2026-02-03T15:19:01-05:00",
	135	+ "language": "en",
	136	+ "application_deadline": null
	137	+ },
	138	+ {
	139	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8130725",
	140	+ "data_compliance": [
	141	+ {
	142	+ "type": "gdpr",
	143	+ "requires_consent": false,
	144	+ "requires_processing_consent": false,
	145	+ "requires_retention_consent": false,
	146	+ "retention_period": null,
	147	+ "demographic_data_consent_applies": false
	148	+ }
	149	+ ],
	150	+ "education": "education_required",
	151	+ "internal_job_id": 3520748,
	152	+ "location": {
	153	+ "name": "San Francisco"
	154	+ },
	155	+ "metadata": null,
	156	+ "id": 8130725,
	157	+ "updated_at": "2026-09-10T13:11:58-04:00",
	158	+ "requisition_id": "See Opening ID",
	159	+ "title": "Account Executive, AI Startups (Hunter)",
	160	+ "company_name": "Stripe",
	161	+ "first_published": "2026-08-19T14:02:07-04:00",
	162	+ "language": "en",
	163	+ "application_deadline": null
	164	+ },
	165	+ {
	166	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8077887",
	167	+ "data_compliance": [
	168	+ {
	169	+ "type": "gdpr",
	170	+ "requires_consent": false,
	171	+ "requires_processing_consent": false,
	172	+ "requires_retention_consent": false,
	173	+ "retention_period": null,
	174	+ "demographic_data_consent_applies": false
	175	+ }
	176	+ ],
	177	+ "education": "education_required",
	178	+ "internal_job_id": 3499744,
	179	+ "location": {
	180	+ "name": "SF, NYC, SEA, CHI"
	181	+ },
	182	+ "metadata": null,
	183	+ "id": 8077887,
	184	+ "updated_at": "2026-09-10T13:11:58-04:00",
	185	+ "requisition_id": "See Opening ID",
	186	+ "title": "Account Executive, Bridge ",
	187	+ "company_name": "Stripe",
	188	+ "first_published": "2026-07-22T13:15:53-04:00",
	189	+ "language": "en",
	190	+ "application_deadline": null
	191	+ },
	192	+ {
	193	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8123027",
	194	+ "data_compliance": [
	195	+ {
	196	+ "type": "gdpr",
	197	+ "requires_consent": false,
	198	+ "requires_processing_consent": false,
	199	+ "requires_retention_consent": false,
	200	+ "retention_period": null,
	201	+ "demographic_data_consent_applies": false
	202	+ }
	203	+ ],
	204	+ "education": "education_required",
	205	+ "internal_job_id": 3518194,
	206	+ "location": {
	207	+ "name": "Chicago"
	208	+ },
	209	+ "metadata": null,
	210	+ "id": 8123027,
	211	+ "updated_at": "2026-09-10T13:11:58-04:00",
	212	+ "requisition_id": "See Opening ID",
	213	+ "title": "Account Executive, Commercial (Grower)",
	214	+ "company_name": "Stripe",
	215	+ "first_published": "2026-08-11T19:45:35-04:00",
	216	+ "language": "en",
	217	+ "application_deadline": null
	218	+ },
	219	+ {
	220	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8141211",
	221	+ "data_compliance": [
	222	+ {
	223	+ "type": "gdpr",
	224	+ "requires_consent": false,
	225	+ "requires_processing_consent": false,
	226	+ "requires_retention_consent": false,
	227	+ "retention_period": null,
	228	+ "demographic_data_consent_applies": false
	229	+ }
	230	+ ],
	231	+ "education": "education_required",
	232	+ "internal_job_id": 3523367,
	233	+ "location": {
	234	+ "name": "Singapore"
	235	+ },
	236	+ "metadata": null,
	237	+ "id": 8141211,
	238	+ "updated_at": "2026-09-10T13:11:58-04:00",
	239	+ "requisition_id": "See Opening ID",
	240	+ "title": "Account Executive, Commercial Hunter",
	241	+ "company_name": "Stripe",
	242	+ "first_published": "2026-08-21T05:00:47-04:00",
	243	+ "language": "en",
	244	+ "application_deadline": null
	245	+ },
	246	+ {
	247	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=7789539",
	248	+ "data_compliance": [
	249	+ {
	250	+ "type": "gdpr",
	251	+ "requires_consent": false,
	252	+ "requires_processing_consent": false,
	253	+ "requires_retention_consent": false,
	254	+ "retention_period": null,
	255	+ "demographic_data_consent_applies": false
	256	+ }
	257	+ ],
	258	+ "education": "education_required",
	259	+ "internal_job_id": 3407094,
	260	+ "location": {
	261	+ "name": "Japan"
	262	+ },
	263	+ "metadata": null,
	264	+ "id": 7789539,
	265	+ "updated_at": "2026-09-10T13:11:58-04:00",
	266	+ "requisition_id": "See Opening ID",
	267	+ "title": "Account Executive, Commercial Hunter (Japanese Fluency)",
	268	+ "company_name": "Stripe",
	269	+ "first_published": "2026-04-17T00:47:42-04:00",
	270	+ "language": "ja",
	271	+ "application_deadline": null
	272	+ },
	273	+ {
	274	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=7893199",
	275	+ "data_compliance": [
	276	+ {
	277	+ "type": "gdpr",
	278	+ "requires_consent": false,
	279	+ "requires_processing_consent": false,
	280	+ "requires_retention_consent": false,
	281	+ "retention_period": null,
	282	+ "demographic_data_consent_applies": false
	283	+ }
	284	+ ],
	285	+ "education": "education_required",
	286	+ "internal_job_id": 3431456,
	287	+ "location": {
	288	+ "name": "Singapore"
	289	+ },
	290	+ "metadata": null,
	291	+ "id": 7893199,
	292	+ "updated_at": "2026-09-10T13:11:58-04:00",
	293	+ "requisition_id": "See Opening ID",
	294	+ "title": "Account Executive, Cross Border China",
	295	+ "company_name": "Stripe",
	296	+ "first_published": "2026-05-06T04:59:55-04:00",
	297	+ "language": "en",
	298	+ "application_deadline": null
	299	+ },
	300	+ {
	301	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8119822",
	302	+ "data_compliance": [
	303	+ {
	304	+ "type": "gdpr",
	305	+ "requires_consent": false,
	306	+ "requires_processing_consent": false,
	307	+ "requires_retention_consent": false,
	308	+ "retention_period": null,
	309	+ "demographic_data_consent_applies": false
	310	+ }
	311	+ ],
	312	+ "education": "education_required",
	313	+ "internal_job_id": 3516941,
	314	+ "location": {
	315	+ "name": "CA-Toronto, CA-Montreal, CA-Vancouver "
	316	+ },
	317	+ "metadata": null,
	318	+ "id": 8119822,
	319	+ "updated_at": "2026-09-10T13:11:58-04:00",
	320	+ "requisition_id": "See Opening ID",
	321	+ "title": "Account Executive, Enterprise (Canada) ",
	322	+ "company_name": "Stripe",
	323	+ "first_published": "2026-08-12T11:04:53-04:00",
	324	+ "language": "en",
	325	+ "application_deadline": null
	326	+ },
	327	+ {
	328	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=7825578",
	329	+ "data_compliance": [
	330	+ {
	331	+ "type": "gdpr",
	332	+ "requires_consent": false,
	333	+ "requires_processing_consent": false,
	334	+ "requires_retention_consent": false,
	335	+ "retention_period": null,
	336	+ "demographic_data_consent_applies": false
	337	+ }
	338	+ ],
	339	+ "education": "education_required",
	340	+ "internal_job_id": 3415406,
	341	+ "location": {
	342	+ "name": "Germany "
	343	+ },
	344	+ "metadata": null,
	345	+ "id": 7825578,
	346	+ "updated_at": "2026-09-10T13:11:58-04:00",
	347	+ "requisition_id": "See Opening ID",
	348	+ "title": "Account Executive, Enterprise (DACH Market) ",
	349	+ "company_name": "Stripe",
	350	+ "first_published": "2026-04-28T10:29:58-04:00",
	351	+ "language": "en",
	352	+ "application_deadline": null
	353	+ },
	354	+ {
	355	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=7993151",
	356	+ "data_compliance": [
	357	+ {
	358	+ "type": "gdpr",
	359	+ "requires_consent": false,
	360	+ "requires_processing_consent": false,
	361	+ "requires_retention_consent": false,
	362	+ "retention_period": null,
	363	+ "demographic_data_consent_applies": false
	364	+ }
	365	+ ],
	366	+ "education": "education_required",
	367	+ "internal_job_id": 3467410,
	368	+ "location": {
	369	+ "name": "US-Remote, US-San Francisco, US-Chicago, US-New York, US-Seattle, US-Texas"
	370	+ },
	371	+ "metadata": null,
	372	+ "id": 7993151,
	373	+ "updated_at": "2026-09-10T13:11:58-04:00",
	374	+ "requisition_id": "See Opening ID",
	375	+ "title": "Account Executive - Enterprise, Grower",
	376	+ "company_name": "Stripe",
	377	+ "first_published": "2026-06-09T13:38:22-04:00",
	378	+ "language": "en",
	379	+ "application_deadline": null
	380	+ },
	381	+ {
	382	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8107136",
	383	+ "data_compliance": [
	384	+ {
	385	+ "type": "gdpr",
	386	+ "requires_consent": false,
	387	+ "requires_processing_consent": false,
	388	+ "requires_retention_consent": false,
	389	+ "retention_period": null,
	390	+ "demographic_data_consent_applies": false
	391	+ }
	392	+ ],
	393	+ "education": "education_required",
	394	+ "internal_job_id": 3512080,
	395	+ "location": {
	396	+ "name": "US-San Francisco, US-Seattle, US-West Coast (Remote) "
	397	+ },
	398	+ "metadata": null,
	399	+ "id": 8107136,
	400	+ "updated_at": "2026-09-10T13:11:58-04:00",
	401	+ "requisition_id": "See Opening ID",
	402	+ "title": "Account Executive, Enterprise (Grower) ",
	403	+ "company_name": "Stripe",
	404	+ "first_published": "2026-08-10T09:19:03-04:00",
	405	+ "language": "en",
	406	+ "application_deadline": null
	407	+ },
	408	+ {
	409	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8114458",
	410	+ "data_compliance": [
	411	+ {
	412	+ "type": "gdpr",
	413	+ "requires_consent": false,
	414	+ "requires_processing_consent": false,
	415	+ "requires_retention_consent": false,
	416	+ "retention_period": null,
	417	+ "demographic_data_consent_applies": false
	418	+ }
	419	+ ],
	420	+ "education": "education_required",
	421	+ "internal_job_id": 3514819,
	422	+ "location": {
	423	+ "name": "Israel "
	424	+ },
	425	+ "metadata": null,
	426	+ "id": 8114458,
	427	+ "updated_at": "2026-09-10T13:11:58-04:00",
	428	+ "requisition_id": "See Opening ID",
	429	+ "title": "Account Executive, Enterprise Grower (Israel)",
	430	+ "company_name": "Stripe",
	431	+ "first_published": "2026-08-18T05:46:54-04:00",
	432	+ "language": "en",
	433	+ "application_deadline": null
	434	+ },
	435	+ {
	436	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8129956",
	437	+ "data_compliance": [
	438	+ {
	439	+ "type": "gdpr",
	440	+ "requires_consent": false,
	441	+ "requires_processing_consent": false,
	442	+ "requires_retention_consent": false,
	443	+ "retention_period": null,
	444	+ "demographic_data_consent_applies": false
	445	+ }
	446	+ ],
	447	+ "education": "education_required",
	448	+ "internal_job_id": 3520451,
	449	+ "location": {
	450	+ "name": "Israel"
	451	+ },
	452	+ "metadata": null,
	453	+ "id": 8129956,
	454	+ "updated_at": "2026-09-10T13:11:58-04:00",
	455	+ "requisition_id": "See Opening ID",
	456	+ "title": "Account Executive, Enterprise Grower, Platforms (Israel)",
	457	+ "company_name": "Stripe",
	458	+ "first_published": "2026-08-24T05:00:55-04:00",
	459	+ "language": "en",
	460	+ "application_deadline": null
	461	+ },
	462	+ {
	463	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8178881",
	464	+ "data_compliance": [
	465	+ {
	466	+ "type": "gdpr",
	467	+ "requires_consent": false,
	468	+ "requires_processing_consent": false,
	469	+ "requires_retention_consent": false,
	470	+ "retention_period": null,
	471	+ "demographic_data_consent_applies": false
	472	+ }
	473	+ ],
	474	+ "education": "education_required",
	475	+ "internal_job_id": 3539405,
	476	+ "location": {
	477	+ "name": "Chicago, IL "
	478	+ },
	479	+ "metadata": null,
	480	+ "id": 8178881,
	481	+ "updated_at": "2026-09-10T13:11:58-04:00",
	482	+ "requisition_id": "See Opening ID",
	483	+ "title": "Account Executive - Enterprise, Growth",
	484	+ "company_name": "Stripe",
	485	+ "first_published": "2026-09-08T11:55:39-04:00",
	486	+ "language": "en",
	487	+ "application_deadline": null
	488	+ },
	489	+ {
	490	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=7994330",
	491	+ "data_compliance": [
	492	+ {
	493	+ "type": "gdpr",
	494	+ "requires_consent": false,
	495	+ "requires_processing_consent": false,
	496	+ "requires_retention_consent": false,
	497	+ "retention_period": null,
	498	+ "demographic_data_consent_applies": false
	499	+ }
	500	+ ],
	501	+ "education": "education_required",
	502	+ "internal_job_id": 3467829,
	503	+ "location": {
	504	+ "name": "US-Chicago, US-New York"
	505	+ },
	506	+ "metadata": null,
	507	+ "id": 7994330,
	508	+ "updated_at": "2026-09-10T13:11:58-04:00",
	509	+ "requisition_id": "See Opening ID",
	510	+ "title": "Account Executive, Enterprise - Hunter",
	511	+ "company_name": "Stripe",
	512	+ "first_published": "2026-06-10T10:30:26-04:00",
	513	+ "language": "en",
	514	+ "application_deadline": null
	515	+ },
	516	+ {
	517	+ "absolute_url": "https://stripe.com/jobs/search?gh_jid=8165275",
	518	+ "data_compliance": [
	519	+ {
	520	+ "type": "gdpr",
	521	+ "requires_consent": false,
	522	+ "requires_processing_consent": false,
	523	+ "requires_retention_consent": false,
	524	+ "retention_period": null,
	525	+ "demographic_data_consent_applies": false
	526	+ }
	527	+ ],
	528	+ "education": "education_required",
	529	+ "internal_job_id": 3533661,
	530	+ "location": {
	531	+ "name": "London"
	532	+ },
	533	+ "metadata": null,
	534	+ "id": 8165275,
	535	+ "updated_at": "2026-09-10T13:11:58-04:00",
	536	+ "requisition_id": "See Opening ID",
	537	+ "title": "Account Executive, Enterprise (Hunter) ",
	538	+ "company_name": "Stripe",
	539	+ "first_published": "2026-09-10T04:03:29-04:00",
	540	+ "language": "en",
	541	+ "application_deadline": null
	542	+ }
	543	+ ],
	544	+ "meta": {
	545	+ "total": 635
	546	+ }
	547	+}
		\ No newline at end of file

added fixtures/connectors/lever/palantir_postings.json +362 −0

		`@@ -0,0 +1,362 @@`
	1	+[
	2	+ {
	3	+ "categories": {
	4	+ "commitment": "Full-time",
	5	+ "location": "London, United Kingdom",
	6	+ "team": "Administrative",
	7	+ "allLocations": [
	8	+ "London, United Kingdom"
	9	+ ]
	10	+ },
	11	+ "createdAt": 1711403416463,
	12	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nOur team of Administrative Business Partners does more than just support our leaders: we're the backbone of the busiest people at Palantir. We build positive relationships with the people we support and anticipate their needs without being asked. Our passion for helping others makes us an invaluable resource at Palantir!\n\nAs an Administrative Business Partner, you will be handling a variety of professional responsibilities, including calendaring, travel, and expenses. You are very organized and thrive off of enabling the people we support to be as productive and impactful as possible. You'll demonstrate your excellent communication skills, and exercise tact and diplomacy in helping to manage relationships with internal and external senior team members at Palantir. In this role you'll also demonstrate good judgment and critical thinking by understanding competing priorities and actioning accordingly.\n",
	13	+ "id": "ac978161-6f46-4f6b-ad9e-a258e642751c",
	14	+ "text": "Administrative Business Partner",
	15	+ "country": "GB",
	16	+ "workplaceType": "hybrid",
	17	+ "hostedUrl": "https://jobs.lever.co/palantir/ac978161-6f46-4f6b-ad9e-a258e642751c",
	18	+ "applyUrl": "https://jobs.lever.co/palantir/ac978161-6f46-4f6b-ad9e-a258e642751c/apply"
	19	+ },
	20	+ {
	21	+ "categories": {
	22	+ "commitment": "Full-time",
	23	+ "location": "London, United Kingdom",
	24	+ "team": "Dev",
	25	+ "allLocations": [
	26	+ "London, United Kingdom"
	27	+ ]
	28	+ },
	29	+ "createdAt": 1710188707256,
	30	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nBackend Software Engineers at Palantir build software at scale to transform how organisations use data. Our Software Engineers are involved throughout the product lifecycle, from idea generation, design, prototyping, and production delivery. You will collaborate closely with technical and non-technical teammates to understand our customers' problems and build products that solve them. We encourage movement across teams to share context, skills, and experience, so you'll learn about many different technologies and aspects of each product. Engineers work autonomously and make decisions independently, within a community that will support and challenge you as you grow and develop, becoming a strong technical contributor and engineering leader.\n \nYour day-to-day workflow will vary, adapting to the requirements of our users and the technical challenges that arise. One day, you may find yourself collaborating with other engineers to architect a new system that enables a novel workflow, the next you could be fine-tuning performance to enable low-latency operational outcomes.\n \nOur Product Development organisation is made up of small teams of Software Engineers. Each team focuses on a specific aspect of a product and work collaboratively to build cross functional capabilities, streamline user workflows and continuously improve our software's efficiency and reliability.\n \nWe’re hiring engineers who are passionate about solving real-world problems and empowering both developers and end-users to work optimally. If you’re motivated to develop reliable, performant, and scalable systems, and to design robust APIs and primitives, this role offers the opportunity to make a significant impact on our products and the people who use them.\n \nFrontline\n \nFoundry Software Engineers may be offered the opportunity to Frontline, an exclusive program unlike any other. This unique, short-term assignment involves being embedded with customers, allowing you to work directly with users and gain firsthand insight into how our products are used and the challenges our customers face. Unlike traditional engineering roles, Frontline immerses you in complex, ambiguous problems, empowering you to deliver impactful solutions across some of the world’s most important industries and institutions.\n \nSome of our most successful products were built on the factory floor, addressing real-world problems for the world's most important institutions. These products were developed by some of our most successful product engineers, who began their careers in roles aligned with Frontline responsibilities, gaining a deep understanding of both our technology and our customers.\n \nFrontliners operate across a broad spectrum of responsibilities, much like a startup CTO. They work in small teams to own the end-to-end execution of high-stakes projects. This spectrum ranges from discussing architecture and building custom web apps to conducting workshops with users and strategizing with customer executives. No two days are alike, as each day is diverse and impactful. By witnessing how customers engage with Foundry and experiencing these pain points firsthand, you’ll gain unique insights that feed directly back into our development process, helping to refine and enhance our products.\n",
	31	+ "id": "10dfc8bc-99ad-4ca2-ab76-853cb90a92c2",
	32	+ "text": "Backend Software Engineer - Application Development",
	33	+ "country": "GB",
	34	+ "workplaceType": "hybrid",
	35	+ "hostedUrl": "https://jobs.lever.co/palantir/10dfc8bc-99ad-4ca2-ab76-853cb90a92c2",
	36	+ "applyUrl": "https://jobs.lever.co/palantir/10dfc8bc-99ad-4ca2-ab76-853cb90a92c2/apply"
	37	+ },
	38	+ {
	39	+ "categories": {
	40	+ "commitment": "Full-time",
	41	+ "location": "New York, NY",
	42	+ "team": "Dev",
	43	+ "allLocations": [
	44	+ "New York, NY"
	45	+ ]
	46	+ },
	47	+ "createdAt": 1710191082166,
	48	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nBackend Software Engineers at Palantir build software at scale to transform how organisations use data. Our Software Engineers are involved throughout the product lifecycle, from idea generation, design, prototyping, and production delivery. You will collaborate closely with technical and non-technical teammates to understand our customers' problems and build products that solve them. We encourage movement across teams to share context, skills, and experience, so you'll learn about many different technologies and aspects of each product. Engineers work autonomously and make decisions independently, within a community that will support and challenge you as you grow and develop, becoming a strong technical contributor and engineering leader.\n \nYour day-to-day workflow will vary, adapting to the requirements of our users and the technical challenges that arise. One day, you may find yourself collaborating with other engineers to architect a new system that enables a novel workflow, the next you could be fine-tuning performance to enable low-latency operational outcomes.\n \nOur Product Development organisation is made up of small teams of Software Engineers. Each team focuses on a specific aspect of a product and work collaboratively to build cross functional capabilities, streamline user workflows and continuously improve our software's efficiency and reliability.\n \nWe’re hiring engineers who are passionate about solving real-world problems and empowering both developers and end-users to work optimally. If you’re motivated to develop reliable, performant, and scalable systems, and to design robust APIs and primitives, this role offers the opportunity to make a significant impact on our products and the people who use them.\n \nFrontline\n \nFoundry Software Engineers may be offered the opportunity to Frontline, an exclusive program unlike any other. This unique, short-term assignment involves being embedded with customers, allowing you to work directly with users and gain firsthand insight into how our products are used and the challenges our customers face. Unlike traditional engineering roles, Frontline immerses you in complex, ambiguous problems, empowering you to deliver impactful solutions across some of the world’s most important industries and institutions.\n \nSome of our most successful products were built on the factory floor, addressing real-world problems for the world's most important institutions. These products were developed by some of our most successful product engineers, who began their careers in roles aligned with Frontline responsibilities, gaining a deep understanding of both our technology and our customers.\n \nFrontliners operate across a broad spectrum of responsibilities, much like a startup CTO. They work in small teams to own the end-to-end execution of high-stakes projects. This spectrum ranges from discussing architecture and building custom web apps to conducting workshops with users and strategizing with customer executives. No two days are alike, as each day is diverse and impactful. By witnessing how customers engage with Foundry and experiencing these pain points firsthand, you’ll gain unique insights that feed directly back into our development process, helping to refine and enhance our products.\n",
	49	+ "id": "ab7e3425-81d5-4705-a7b5-cd60c8a45cdb",
	50	+ "text": "Backend Software Engineer - Application Development",
	51	+ "country": "US",
	52	+ "workplaceType": "hybrid",
	53	+ "hostedUrl": "https://jobs.lever.co/palantir/ab7e3425-81d5-4705-a7b5-cd60c8a45cdb",
	54	+ "applyUrl": "https://jobs.lever.co/palantir/ab7e3425-81d5-4705-a7b5-cd60c8a45cdb/apply"
	55	+ },
	56	+ {
	57	+ "categories": {
	58	+ "commitment": "Full-time",
	59	+ "location": "Washington, D.C.",
	60	+ "team": "Dev",
	61	+ "allLocations": [
	62	+ "Washington, D.C."
	63	+ ]
	64	+ },
	65	+ "createdAt": 1740435531334,
	66	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir's defense product vertical builds mission-critical products for the modern warfighter. We provide a complete ecosystem where customers can securely integrate and visualize their data, and build sophisticated, full-fledged programs such as common operating pictures, alert-triaging inboxes, and resource allocation planning tools driven by rich-ML models. Our customers use our defense offering to perform rich analyses that drive core operations within their organizations - these programs are relied upon for daily operations in the command centers and battlefronts of militaries across the world.\n \nBackend Software Engineers at Palantir build software at scale to transform how organizations use data. Our Software Engineers are involved throughout the product lifecycle, from idea generation, design, prototyping, and production delivery. You will collaborate closely with technical and non-technical teammates to understand our customers' problems and build products that solve them. We encourage movement across teams to share context, skills, and experience, so you'll learn about many different technologies and aspects of each product. Engineers work autonomously and make decisions independently, within a community that will support and challenge you as you grow and develop, becoming a strong technical contributor and engineering leader.\n \nYour day-to-day workflow will vary, adapting to the requirements of our users and the technical challenges that arise. One day, you may find yourself collaborating with other engineers to architect a new system that enables a novel workflow, the next you could be fine-tuning performance to enable low-latency operational outcomes.\n \nOur Product Development organization is made up of small teams of Software Engineers. Each team focuses on a specific aspect of a product and work collaboratively to build cross functional capabilities, streamline user workflows and continuously improve our software's efficiency and reliability.\n \nWe're hiring engineers who are passionate about solving real-world problems and empowering developers and end-users to do their work optimally. If you’re motivated to develop reliable, performant, scalable systems and design robust APIs and primitives, please join us.\n",
	67	+ "id": "1345438c-ebfc-4fa5-b545-30c1414f317c",
	68	+ "text": "Backend Software Engineer - Defense",
	69	+ "country": "US",
	70	+ "workplaceType": "onsite",
	71	+ "hostedUrl": "https://jobs.lever.co/palantir/1345438c-ebfc-4fa5-b545-30c1414f317c",
	72	+ "applyUrl": "https://jobs.lever.co/palantir/1345438c-ebfc-4fa5-b545-30c1414f317c/apply"
	73	+ },
	74	+ {
	75	+ "categories": {
	76	+ "commitment": "Full-time",
	77	+ "location": "Palo Alto, CA",
	78	+ "team": "Dev",
	79	+ "allLocations": [
	80	+ "Palo Alto, CA"
	81	+ ]
	82	+ },
	83	+ "createdAt": 1740435552105,
	84	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir's defense product vertical builds mission-critical products for the modern warfighter. We provide a complete ecosystem where customers can securely integrate and visualize their data, and build sophisticated, full-fledged programs such as common operating pictures, alert-triaging inboxes, and resource allocation planning tools driven by rich-ML models. Our customers use our defense offering to perform rich analyses that drive core operations within their organizations - these programs are relied upon for daily operations in the command centers and battlefronts of militaries across the world.\n \nBackend Software Engineers at Palantir build software at scale to transform how organizations use data. Our Software Engineers are involved throughout the product lifecycle, from idea generation, design, prototyping, and production delivery. You will collaborate closely with technical and non-technical teammates to understand our customers' problems and build products that solve them. We encourage movement across teams to share context, skills, and experience, so you'll learn about many different technologies and aspects of each product. Engineers work autonomously and make decisions independently, within a community that will support and challenge you as you grow and develop, becoming a strong technical contributor and engineering leader.\n \nYour day-to-day workflow will vary, adapting to the requirements of our users and the technical challenges that arise. One day, you may find yourself collaborating with other engineers to architect a new system that enables a novel workflow, the next you could be fine-tuning performance to enable low-latency operational outcomes.\n \nOur Product Development organization is made up of small teams of Software Engineers. Each team focuses on a specific aspect of a product and work collaboratively to build cross functional capabilities, streamline user workflows and continuously improve our software's efficiency and reliability.\n \nWe're hiring engineers who are passionate about solving real-world problems and empowering developers and end-users to do their work optimally. If you’re motivated to develop reliable, performant, scalable systems and design robust APIs and primitives, please join us.\n",
	85	+ "id": "a8174f9c-6f46-46b4-8e15-d1ff9e37c9eb",
	86	+ "text": "Backend Software Engineer - Defense",
	87	+ "country": "US",
	88	+ "workplaceType": "onsite",
	89	+ "hostedUrl": "https://jobs.lever.co/palantir/a8174f9c-6f46-46b4-8e15-d1ff9e37c9eb",
	90	+ "applyUrl": "https://jobs.lever.co/palantir/a8174f9c-6f46-46b4-8e15-d1ff9e37c9eb/apply"
	91	+ },
	92	+ {
	93	+ "categories": {
	94	+ "commitment": "Full-time",
	95	+ "location": "New York, NY",
	96	+ "team": "Dev",
	97	+ "allLocations": [
	98	+ "New York, NY"
	99	+ ]
	100	+ },
	101	+ "createdAt": 1740435575088,
	102	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir's defense product vertical builds mission-critical products for the modern warfighter. We provide a complete ecosystem where customers can securely integrate and visualize their data, and build sophisticated, full-fledged programs such as common operating pictures, alert-triaging inboxes, and resource allocation planning tools driven by rich-ML models. Our customers use our defense offering to perform rich analyses that drive core operations within their organizations - these programs are relied upon for daily operations in the command centers and battlefronts of militaries across the world.\n \nBackend Software Engineers at Palantir build software at scale to transform how organizations use data. Our Software Engineers are involved throughout the product lifecycle, from idea generation, design, prototyping, and production delivery. You will collaborate closely with technical and non-technical teammates to understand our customers' problems and build products that solve them. We encourage movement across teams to share context, skills, and experience, so you'll learn about many different technologies and aspects of each product. Engineers work autonomously and make decisions independently, within a community that will support and challenge you as you grow and develop, becoming a strong technical contributor and engineering leader.\n \nYour day-to-day workflow will vary, adapting to the requirements of our users and the technical challenges that arise. One day, you may find yourself collaborating with other engineers to architect a new system that enables a novel workflow, the next you could be fine-tuning performance to enable low-latency operational outcomes.\n \nOur Product Development organization is made up of small teams of Software Engineers. Each team focuses on a specific aspect of a product and work collaboratively to build cross functional capabilities, streamline user workflows and continuously improve our software's efficiency and reliability.\n \nWe're hiring engineers who are passionate about solving real-world problems and empowering developers and end-users to do their work optimally. If you’re motivated to develop reliable, performant, scalable systems and design robust APIs and primitives, please join us.\n",
	103	+ "id": "d33e0c31-ac7e-4f57-ba74-36f2df6ae2f5",
	104	+ "text": "Backend Software Engineer - Defense",
	105	+ "country": "US",
	106	+ "workplaceType": "onsite",
	107	+ "hostedUrl": "https://jobs.lever.co/palantir/d33e0c31-ac7e-4f57-ba74-36f2df6ae2f5",
	108	+ "applyUrl": "https://jobs.lever.co/palantir/d33e0c31-ac7e-4f57-ba74-36f2df6ae2f5/apply"
	109	+ },
	110	+ {
	111	+ "categories": {
	112	+ "commitment": "Full-time",
	113	+ "location": "New York, NY",
	114	+ "team": "Dev",
	115	+ "allLocations": [
	116	+ "New York, NY"
	117	+ ]
	118	+ },
	119	+ "createdAt": 1754508487190,
	120	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nSoftware Engineers at Palantir build software at scale to transform how organizations use data. Our Software Engineers are involved throughout the product lifecycle, from idea generation, design, prototyping, and production delivery. You will collaborate closely with technical and non-technical teammates to understand our customers' problems and build products that solve them. We encourage movement across teams to share context, skills, and experience, so you'll learn about many different technologies and aspects of each product. Engineers work autonomously and make decisions independently, within a community that will support and challenge you as you grow and develop, becoming a strong technical contributor and engineering leader.\n \nOur Product Development organization is made up of small teams of Software Engineers. Each team focuses on a specific aspect of a product. Our infrastructure teams are responsible for the lowest layers of our software stack, often focused on database technologies, distributed systems, large scale data systems, security, and application infrastructure. As a Software Engineer on infrastructure working on our Foundry platform, you'll contribute high-quality code to underpin Palantir Foundry and Gotham with performant, secure, and scalable building blocks, enabling products deployed to the most important institutions in the public and private sector. You'll build the foundational capabilities that power our products used by research scientists, aerospace engineers, intelligence analysts, and economic forecasters, in countries around the world.\n \nWe’re hiring engineers who are passionate about solving real-world problems and empowering both developers and end-users to work optimally. If you’re motivated to develop reliable, performant, and scalable systems, and to design robust APIs and primitives, this role offers the opportunity to make a significant impact on our products and the people who use them.\n \nFrontline\n \nFoundry Software Engineers may be offered the opportunity to Frontline, an exclusive program unlike any other. This unique, short-term assignment involves being embedded with customers, allowing you to work directly with users and gain firsthand insight into how our products are used and the challenges our customers face. Unlike traditional engineering roles, Frontline immerses you in complex, ambiguous problems, empowering you to deliver impactful solutions across some of the world’s most important industries and institutions.\n \nSome of our most successful products were built on the factory floor, addressing real-world problems for the world's most important institutions. These products were developed by some of our most successful product engineers, who began their careers in roles aligned with Frontline responsibilities, gaining a deep understanding of both our technology and our customers.\n \nFrontliners operate across a broad spectrum of responsibilities, much like a startup CTO. They work in small teams to own the end-to-end execution of high-stakes projects. This spectrum ranges from discussing architecture and building custom web apps to conducting workshops with users and strategizing with customer executives. No two days are alike, as each day is diverse and impactful. By witnessing how customers engage with Foundry and experiencing these pain points firsthand, you’ll gain unique insights that feed directly back into our development process, helping to refine and enhance our products.\n",
	121	+ "id": "6fe5515f-f677-4d98-8ac2-1775a425f5e7",
	122	+ "text": "Backend Software Engineer - Infrastructure",
	123	+ "country": "US",
	124	+ "workplaceType": "hybrid",
	125	+ "hostedUrl": "https://jobs.lever.co/palantir/6fe5515f-f677-4d98-8ac2-1775a425f5e7",
	126	+ "applyUrl": "https://jobs.lever.co/palantir/6fe5515f-f677-4d98-8ac2-1775a425f5e7/apply"
	127	+ },
	128	+ {
	129	+ "categories": {
	130	+ "commitment": "Full-time",
	131	+ "location": "London, United Kingdom",
	132	+ "team": "Dev",
	133	+ "allLocations": [
	134	+ "London, United Kingdom"
	135	+ ]
	136	+ },
	137	+ "createdAt": 1708452790131,
	138	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nBackend Software Engineers at Palantir build software at scale to transform how organizations use data. Our Software Engineers are involved throughout the product lifecycle, from idea generation, design, prototyping, and production delivery. You will collaborate closely with technical and non-technical teammates to understand our customers' problems and build products that solve them. We encourage movement across teams to share context, skills, and experience, so you'll learn about many different technologies and aspects of each product. Engineers work autonomously and make decisions independently, within a community that will support and challenge you as you grow and develop, becoming a strong technical contributor and engineering leader.\n \nOur Product Development organization is made up of small teams of Software Engineers. Each team focuses on a specific aspect of a product. Our infrastructure teams are responsible for the lowest layers of our software stack, often focused on database technologies, distributed systems, large scale data systems, security, and application infrastructure. As a Software Engineer on infrastructure, you'll contribute high-quality code to underpin Palantir Foundry and Gotham with performant, secure, and scalable building blocks, enabling products deployed to the most important institutions in the public and private sector. You'll build the foundational capabilities that power our products used by research scientists, aerospace engineers, intelligence analysts, and economic forecasters, in countries around the world.\n \nWe’re hiring engineers who are passionate about solving real-world problems and empowering both developers and end-users to work optimally. If you’re motivated to develop reliable, performant, and scalable systems, and to design robust APIs and primitives, this role offers the opportunity to make a significant impact on our products and the people who use them.\n \nFrontline\n \nFoundry Software Engineers may be offered the opportunity to Frontline, an exclusive program unlike any other. This unique, short-term assignment involves being embedded with customers, allowing you to work directly with users and gain firsthand insight into how our products are used and the challenges our customers face. Unlike traditional engineering roles, Frontline immerses you in complex, ambiguous problems, empowering you to deliver impactful solutions across some of the world’s most important industries and institutions.\n \nSome of our most successful products were built on the factory floor, addressing real-world problems for the world's most important institutions. These products were developed by some of our most successful product engineers, who began their careers in roles aligned with Frontline responsibilities, gaining a deep understanding of both our technology and our customers.\n \nFrontliners operate across a broad spectrum of responsibilities, much like a startup CTO. They work in small teams to own the end-to-end execution of high-stakes projects. This spectrum ranges from discussing architecture and building custom web apps to conducting workshops with users and strategizing with customer executives. No two days are alike, as each day is diverse and impactful. By witnessing how customers engage with Foundry and experiencing these pain points firsthand, you’ll gain unique insights that feed directly back into our development process, helping to refine and enhance our products.\n",
	139	+ "id": "f70cdff7-c62f-4b73-a136-909e5e3d1891",
	140	+ "text": "Backend Software Engineer - Infrastructure",
	141	+ "country": "GB",
	142	+ "workplaceType": "hybrid",
	143	+ "hostedUrl": "https://jobs.lever.co/palantir/f70cdff7-c62f-4b73-a136-909e5e3d1891",
	144	+ "applyUrl": "https://jobs.lever.co/palantir/f70cdff7-c62f-4b73-a136-909e5e3d1891/apply"
	145	+ },
	146	+ {
	147	+ "categories": {
	148	+ "commitment": "Full-time",
	149	+ "location": "New York, NY",
	150	+ "team": "Dev",
	151	+ "allLocations": [
	152	+ "New York, NY"
	153	+ ]
	154	+ },
	155	+ "createdAt": 1708449747920,
	156	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nBackend Software Engineers at Palantir build software at scale to transform how organizations use data. Our Software Engineers are involved throughout the product lifecycle, from idea generation, design, prototyping, and production delivery. You will collaborate closely with technical and non-technical teammates to understand our customers' problems and build products that solve them. We encourage movement across teams to share context, skills, and experience, so you'll learn about many different technologies and aspects of each product. Engineers work autonomously and make decisions independently, within a community that will support and challenge you as you grow and develop, becoming a strong technical contributor and engineering leader.\n \nOur Product Development organization is made up of small teams of Software Engineers. Each team focuses on a specific aspect of a product. Our infrastructure teams are responsible for the lowest layers of our software stack, often focused on database technologies, distributed systems, large scale data systems, security, and application infrastructure. As a Software Engineer on infrastructure, you'll contribute high-quality code to underpin Palantir Foundry and Gotham with performant, secure, and scalable building blocks, enabling products deployed to the most important institutions in the public and private sector. You'll build the foundational capabilities that power our products used by research scientists, aerospace engineers, intelligence analysts, and economic forecasters, in countries around the world.\n \nWe’re hiring engineers who are passionate about solving real-world problems and empowering both developers and end-users to work optimally. If you’re motivated to develop reliable, performant, and scalable systems, and to design robust APIs and primitives, this role offers the opportunity to make a significant impact on our products and the people who use them.\n",
	157	+ "id": "fb2d3222-dbd8-4e03-8d39-47b820e9509c",
	158	+ "text": "Backend Software Engineer - Infrastructure, Foundations",
	159	+ "country": "US",
	160	+ "workplaceType": "hybrid",
	161	+ "hostedUrl": "https://jobs.lever.co/palantir/fb2d3222-dbd8-4e03-8d39-47b820e9509c",
	162	+ "applyUrl": "https://jobs.lever.co/palantir/fb2d3222-dbd8-4e03-8d39-47b820e9509c/apply"
	163	+ },
	164	+ {
	165	+ "categories": {
	166	+ "commitment": "Full-time",
	167	+ "location": "Miami, FL",
	168	+ "team": "Administrative",
	169	+ "allLocations": [
	170	+ "Miami, FL"
	171	+ ]
	172	+ },
	173	+ "createdAt": 1780672903653,
	174	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\nOur team of Administrative Business Partners does more than just support our leaders: we’re the backbone of the busiest people at Palantir. We build positive relationships with the people we support and anticipate their needs without being asked. Our passion for helping others makes us an invaluable resource at Palantir!\n\nAs an Administrative Business Partner, you will be handling a variety of professional responsibilities, including calendaring, travel, and expenses. You are very organized and thrive off of enabling the people you support to be as productive and impactful as possible. You’ll demonstrate your excellent communication skills, and exercise tact and diplomacy in helping to manage relationships with internal and external senior team members at Palantir. In this role you'll also demonstrate good judgment and critical thinking by understanding competing priorities and actioning accordingly.\n",
	175	+ "id": "2c45b359-0d00-4c68-b13d-b9f405efb739",
	176	+ "text": "Commercial Administrative Business Partner",
	177	+ "country": "US",
	178	+ "workplaceType": "hybrid",
	179	+ "hostedUrl": "https://jobs.lever.co/palantir/2c45b359-0d00-4c68-b13d-b9f405efb739",
	180	+ "applyUrl": "https://jobs.lever.co/palantir/2c45b359-0d00-4c68-b13d-b9f405efb739/apply"
	181	+ },
	182	+ {
	183	+ "categories": {
	184	+ "commitment": "Full-time",
	185	+ "location": "Washington, D.C.",
	186	+ "team": "Administrative",
	187	+ "allLocations": [
	188	+ "Washington, D.C."
	189	+ ]
	190	+ },
	191	+ "createdAt": 1780672879478,
	192	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\nOur team of Administrative Business Partners does more than just support our leaders: we’re the backbone of the busiest people at Palantir. We build positive relationships with the people we support and anticipate their needs without being asked. Our passion for helping others makes us an invaluable resource at Palantir!\n\nAs an Administrative Business Partner, you will be handling a variety of professional responsibilities, including calendaring, travel, and expenses. You are very organized and thrive off of enabling the people you support to be as productive and impactful as possible. You’ll demonstrate your excellent communication skills, and exercise tact and diplomacy in helping to manage relationships with internal and external senior team members at Palantir. In this role you'll also demonstrate good judgment and critical thinking by understanding competing priorities and actioning accordingly.\n",
	193	+ "id": "98256288-e13a-4c3e-b6c3-1e3dd1083b9a",
	194	+ "text": "Commercial Administrative Business Partner",
	195	+ "country": "US",
	196	+ "workplaceType": "hybrid",
	197	+ "hostedUrl": "https://jobs.lever.co/palantir/98256288-e13a-4c3e-b6c3-1e3dd1083b9a",
	198	+ "applyUrl": "https://jobs.lever.co/palantir/98256288-e13a-4c3e-b6c3-1e3dd1083b9a/apply"
	199	+ },
	200	+ {
	201	+ "categories": {
	202	+ "commitment": "Full-time",
	203	+ "location": "New York, NY",
	204	+ "team": "Administrative",
	205	+ "allLocations": [
	206	+ "New York, NY"
	207	+ ]
	208	+ },
	209	+ "createdAt": 1780672683975,
	210	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\nOur team of Administrative Business Partners does more than just support our leaders: we’re the backbone of the busiest people at Palantir. We build positive relationships with the people we support and anticipate their needs without being asked. Our passion for helping others makes us an invaluable resource at Palantir!\n\nAs an Administrative Business Partner, you will be handling a variety of professional responsibilities, including calendaring, travel, and expenses. You are very organized and thrive off of enabling the people you support to be as productive and impactful as possible. You’ll demonstrate your excellent communication skills, and exercise tact and diplomacy in helping to manage relationships with internal and external senior team members at Palantir. In this role you'll also demonstrate good judgment and critical thinking by understanding competing priorities and actioning accordingly.\n",
	211	+ "id": "f163223b-db5d-4c6d-b73f-0a2eaf85583f",
	212	+ "text": "Commercial Administrative Business Partner",
	213	+ "country": "US",
	214	+ "workplaceType": "hybrid",
	215	+ "hostedUrl": "https://jobs.lever.co/palantir/f163223b-db5d-4c6d-b73f-0a2eaf85583f",
	216	+ "applyUrl": "https://jobs.lever.co/palantir/f163223b-db5d-4c6d-b73f-0a2eaf85583f/apply"
	217	+ },
	218	+ {
	219	+ "categories": {
	220	+ "commitment": "Full-time",
	221	+ "location": "New York, NY",
	222	+ "team": "Legal",
	223	+ "allLocations": [
	224	+ "New York, NY"
	225	+ ]
	226	+ },
	227	+ "createdAt": 1648073203684,
	228	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir’s in-house legal team works to proactively address legal issues so that Palantir can continue to drive positive impact in the world. As a Commercial Contracts Specialist, you will be a non-attorney partnering closely with Commercial Counsel to effectively support our rapidly growing portfolio of commercial customers. In this role, you will also work closely with Palantir’s Sales, Legal, Finance, and Engineering teams, while owning the drafting, administration, and negotiation of commercial customer contracts, and building infrastructure to support our commercial customers at scale.\n \nAs you develop expertise with our commercial contracts and build relationships with our internal partners, you will become adept at quickly researching and answering factual and interpretive contractual questions as needed. You will also support various special projects that arise at our fast-paced, mission-focused company.\n \nWe’re a team that values both creativity and teamwork - whether operating solo or collaboratively, we seek to achieve great results on challenging, time-sensitive projects. You'll be given open ended objectives and will find ways to turn them into outcomes. By focusing on the broad vision without losing sight of the details, you'll bring large and multi-phase projects to successful completion while prioritizing team outcomes over individual wins. You are able to deliver complex information in an understandable way, and can manage high touch stakeholders in a constantly shifting landscape. You are ready to become an expert on the intricate details of the contracts that enable Palantir to implement its cutting-edge technology to solve real-world problems. You are passionate about conceiving and implementing scalable systems that increase your impact, and that of the Commercial Contracting team, over time.\n",
	229	+ "id": "45241c61-11af-45b5-86a0-a5302c028d6d",
	230	+ "text": "Commercial Contracts Specialist",
	231	+ "country": "US",
	232	+ "workplaceType": "hybrid",
	233	+ "hostedUrl": "https://jobs.lever.co/palantir/45241c61-11af-45b5-86a0-a5302c028d6d",
	234	+ "applyUrl": "https://jobs.lever.co/palantir/45241c61-11af-45b5-86a0-a5302c028d6d/apply"
	235	+ },
	236	+ {
	237	+ "categories": {
	238	+ "commitment": "Full-time",
	239	+ "location": "Palo Alto, CA",
	240	+ "team": "Legal",
	241	+ "allLocations": [
	242	+ "Palo Alto, CA"
	243	+ ]
	244	+ },
	245	+ "createdAt": 1648150733190,
	246	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir’s in-house legal team works to proactively address legal issues so that Palantir can continue to drive positive impact in the world. As a Commercial Contracts Specialist, you will be a non-attorney partnering closely with Commercial Counsel to effectively support our rapidly growing portfolio of commercial customers. In this role, you will also work closely with Palantir’s Sales, Legal, Finance, and Engineering teams, while owning the drafting, administration, and negotiation of commercial customer contracts, and building infrastructure to support our commercial customers at scale.\n \nAs you develop expertise with our commercial contracts and build relationships with our internal partners, you will become adept at quickly researching and answering factual and interpretive contractual questions as needed. You will also support various special projects that arise at our fast-paced, mission-focused company.\n \nWe’re a team that values both creativity and teamwork - whether operating solo or collaboratively, we seek to achieve great results on challenging, time-sensitive projects. You'll be given open ended objectives and will find ways to turn them into outcomes. By focusing on the broad vision without losing sight of the details, you'll bring large and multi-phase projects to successful completion while prioritizing team outcomes over individual wins. You are able to deliver complex information in an understandable way, and can manage high touch stakeholders in a constantly shifting landscape. You are ready to become an expert on the intricate details of the contracts that enable Palantir to implement its cutting-edge technology to solve real-world problems. You are passionate about conceiving and implementing scalable systems that increase your impact, and that of the Commercial Contracting team, over time.\n",
	247	+ "id": "9675791e-59c3-4677-ac62-0c80c38da2eb",
	248	+ "text": "Commercial Contracts Specialist",
	249	+ "country": "US",
	250	+ "workplaceType": "hybrid",
	251	+ "hostedUrl": "https://jobs.lever.co/palantir/9675791e-59c3-4677-ac62-0c80c38da2eb",
	252	+ "applyUrl": "https://jobs.lever.co/palantir/9675791e-59c3-4677-ac62-0c80c38da2eb/apply"
	253	+ },
	254	+ {
	255	+ "categories": {
	256	+ "commitment": "Full-time",
	257	+ "location": "New York, NY",
	258	+ "team": "Legal",
	259	+ "allLocations": [
	260	+ "New York, NY"
	261	+ ]
	262	+ },
	263	+ "createdAt": 1659104802629,
	264	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir’s in‑house legal team works to proactively address legal issues so that Palantir can continue to drive positive impact in the world. As Corporate Counsel, you will serve as a key advisor on corporate governance and securities matters, leveraging your public company experience to ensure our compliance and support our growth as a public company. You will take ownership of critical areas such as investment management, trading compliance, strategic transactions, corporate governance, and public company reporting. In addition, you’ll play a central role in structuring, implementing, and administering equity compensation plans for employees globally, collaborating with cross-functional teams to align our equity programs with Palantir’s mission and values. This is a high-impact role where you will consistently work cross-functionally and with leadership teams (including the Chief Financial Officer, Chief Accounting Officer, and Chief Revenue Officer and Chief Legal Officer), and will translate complex legal requirements into clear communications and actionable strategies that support Palantir’s continued innovation and success as a public company.\n\nIn this role, you will join a team that values broad-based experience and a relentless drive to learn quickly in a dynamic environment. We prioritize practical solutions and outcomes that ensure compliance with public company requirements while maximizing benefits for our employee base and advancing Palantir’s mission. Your ability to adapt, think creatively, and work collaboratively will be essential as you take on complex issues and transactions.\n",
	265	+ "id": "1a581b97-ce5b-4b4c-84b0-dd54cf8cfe14",
	266	+ "text": "Corporate Counsel",
	267	+ "country": "US",
	268	+ "workplaceType": "hybrid",
	269	+ "hostedUrl": "https://jobs.lever.co/palantir/1a581b97-ce5b-4b4c-84b0-dd54cf8cfe14",
	270	+ "applyUrl": "https://jobs.lever.co/palantir/1a581b97-ce5b-4b4c-84b0-dd54cf8cfe14/apply"
	271	+ },
	272	+ {
	273	+ "categories": {
	274	+ "commitment": "Full-time",
	275	+ "location": "Palo Alto, CA",
	276	+ "team": "Legal",
	277	+ "allLocations": [
	278	+ "Palo Alto, CA"
	279	+ ]
	280	+ },
	281	+ "createdAt": 1651255511913,
	282	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir’s in‑house legal team works to proactively address legal issues so that Palantir can continue to drive positive impact in the world. As Corporate Counsel, you will serve as a key advisor on corporate governance and securities matters, leveraging your public company experience to ensure our compliance and support our growth as a public company. You will take ownership of critical areas such as investment management, trading compliance, strategic transactions, corporate governance, and public company reporting. In addition, you’ll play a central role in structuring, implementing, and administering equity compensation plans for employees globally, collaborating with cross-functional teams to align our equity programs with Palantir’s mission and values. This is a high-impact role where you will consistently work cross-functionally and with leadership teams (including the Chief Financial Officer, Chief Accounting Officer, and Chief Revenue Officer and Chief Legal Officer), and will translate complex legal requirements into clear communications and actionable strategies that support Palantir’s continued innovation and success as a public company.\n\nIn this role, you will join a team that values broad-based experience and a relentless drive to learn quickly in a dynamic environment. We prioritize practical solutions and outcomes that ensure compliance with public company requirements while maximizing benefits for our employee base and advancing Palantir’s mission. Your ability to adapt, think creatively, and work collaboratively will be essential as you take on complex issues and transactions.\n",
	283	+ "id": "1ef0cb77-03a2-4b60-b61d-ace924ef06e7",
	284	+ "text": "Corporate Counsel",
	285	+ "country": "US",
	286	+ "workplaceType": "hybrid",
	287	+ "hostedUrl": "https://jobs.lever.co/palantir/1ef0cb77-03a2-4b60-b61d-ace924ef06e7",
	288	+ "applyUrl": "https://jobs.lever.co/palantir/1ef0cb77-03a2-4b60-b61d-ace924ef06e7/apply"
	289	+ },
	290	+ {
	291	+ "categories": {
	292	+ "commitment": "Full-time",
	293	+ "location": "Washington, D.C.",
	294	+ "team": "Legal",
	295	+ "allLocations": [
	296	+ "Washington, D.C."
	297	+ ]
	298	+ },
	299	+ "createdAt": 1709325865697,
	300	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir’s in‑house legal team works to proactively address legal issues so that Palantir can continue to drive positive impact in the world. As Corporate Counsel, you will serve as a key advisor on corporate governance and securities matters, leveraging your public company experience to ensure our compliance and support our growth as a public company. You will take ownership of critical areas such as investment management, trading compliance, strategic transactions, corporate governance, and public company reporting. In addition, you’ll play a central role in structuring, implementing, and administering equity compensation plans for employees globally, collaborating with cross-functional teams to align our equity programs with Palantir’s mission and values. This is a high-impact role where you will consistently work cross-functionally and with leadership teams (including the Chief Financial Officer, Chief Accounting Officer, and Chief Revenue Officer and Chief Legal Officer), and will translate complex legal requirements into clear communications and actionable strategies that support Palantir’s continued innovation and success as a public company.\n\nIn this role, you will join a team that values broad-based experience and a relentless drive to learn quickly in a dynamic environment. We prioritize practical solutions and outcomes that ensure compliance with public company requirements while maximizing benefits for our employee base and advancing Palantir’s mission. Your ability to adapt, think creatively, and work collaboratively will be essential as you take on complex issues and transactions.\n",
	301	+ "id": "9804d4d1-bd2f-42d5-ae3e-3e3e51aa3830",
	302	+ "text": "Corporate Counsel",
	303	+ "country": "US",
	304	+ "workplaceType": "hybrid",
	305	+ "hostedUrl": "https://jobs.lever.co/palantir/9804d4d1-bd2f-42d5-ae3e-3e3e51aa3830",
	306	+ "applyUrl": "https://jobs.lever.co/palantir/9804d4d1-bd2f-42d5-ae3e-3e3e51aa3830/apply"
	307	+ },
	308	+ {
	309	+ "categories": {
	310	+ "commitment": "Full-time",
	311	+ "location": "New York, NY",
	312	+ "team": "Legal",
	313	+ "allLocations": [
	314	+ "New York, NY"
	315	+ ]
	316	+ },
	317	+ "createdAt": 1745438762758,
	318	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir’s in-house legal team works to proactively address legal issues so that Palantir can continue to drive positive impact in the world. We are deeply embedded in Palantir’s mission and culture, take time to connect with folks across the global Palantir community, and are viewed across the company as key to Palantir’s success. \n\nLegal Operations is the organizational backbone of the legal team at Palantir and our Deal Administrators directly support all aspects of revenue generation at the company. Our Deal Operations Administrators have a strong understanding of Palantir’s growth strategy, and are eager to implement and operationalize that strategy in our contracting processes. Our Deal Operations Administrators have a passion for helping others, making them an invaluable resource to our customer deal pipeline.\n\nAs a Deal Operations Administrator, you will be handling a wide variety of responsibilities throughout the customer contracting lifecycle. You will support Palantir’s lawyers and contracting specialists, in collaboration with our sales team and executives, to facilitate the growth of our business by managing all aspects of contract administration. As part of the Deal Operations team, you will become an expert in your specific customer vertical, take ownership for specialized scaling projects and critical initiatives as it relates to each business unit. Throughout your work, you will take initiative, operate with resourcefulness, and execute quickly and effectively the processes that drive revenue at Palantir. You’ll leverage your excellent communication skills, and exercise tact and diplomacy in helping to manage relationships with teams across Palantir. Most importantly, you have a true desire to be helpful and go above and beyond to solve challenges.\n",
	319	+ "id": "157d4289-183f-4c4d-bc77-0dbacaba4612",
	320	+ "text": "Deal Operations Administrator",
	321	+ "country": "US",
	322	+ "workplaceType": "hybrid",
	323	+ "hostedUrl": "https://jobs.lever.co/palantir/157d4289-183f-4c4d-bc77-0dbacaba4612",
	324	+ "applyUrl": "https://jobs.lever.co/palantir/157d4289-183f-4c4d-bc77-0dbacaba4612/apply"
	325	+ },
	326	+ {
	327	+ "categories": {
	328	+ "commitment": "Full-time",
	329	+ "location": "Denver, CO",
	330	+ "team": "Legal",
	331	+ "allLocations": [
	332	+ "Denver, CO"
	333	+ ]
	334	+ },
	335	+ "createdAt": 1721031773006,
	336	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir’s in-house legal team works to proactively address legal issues so that Palantir can continue to drive positive impact in the world. We are deeply embedded in Palantir’s mission and culture, take time to connect with folks across the global Palantir community, and are viewed across the company as key to Palantir’s success. \n\nLegal Operations is the organizational backbone of the legal team at Palantir and our Deal Support & Operations Administrators directly support all aspects of revenue generation at the company. Our Deal Support & Operations team have a strong understanding of Palantir’s growth strategy, and are eager to implement and operationalize that strategy in our contracting processes. Our Administrators have a passion for helping others, making them an invaluable resource to our customer deal pipeline.\n\nAs a Deal Support & Operations Administrator, you will be handling a wide variety of responsibilities throughout the customer contracting lifecycle. You will support Palantir’s lawyers and contracting specialists, in collaboration with our sales team and executives, to facilitate the growth of our business by managing all aspects of contract administration. As part of the Deal Support & Operations team, you will become an expert in your specific customer vertical, take ownership for scaling workflows and critical initiatives as it relates to each business unit. Throughout your work, you will take initiative, operate with resourcefulness, and execute quickly and effectively the processes that drive revenue at Palantir. You’ll leverage your excellent communication skills, and exercise tact and diplomacy in helping to manage relationships with teams across Palantir. Most importantly, you have a true desire to be helpful and go above and beyond to solve challenges.\n",
	337	+ "id": "403e8826-3421-43db-ac48-4ddac10cc78d",
	338	+ "text": "Deal Operations Administrator",
	339	+ "country": "US",
	340	+ "workplaceType": "hybrid",
	341	+ "hostedUrl": "https://jobs.lever.co/palantir/403e8826-3421-43db-ac48-4ddac10cc78d",
	342	+ "applyUrl": "https://jobs.lever.co/palantir/403e8826-3421-43db-ac48-4ddac10cc78d/apply"
	343	+ },
	344	+ {
	345	+ "categories": {
	346	+ "commitment": "Full-time",
	347	+ "location": "Palo Alto, CA",
	348	+ "team": "Legal",
	349	+ "allLocations": [
	350	+ "Palo Alto, CA"
	351	+ ]
	352	+ },
	353	+ "createdAt": 1716225273136,
	354	+ "descriptionPlain": "A World-Changing Company\n \nPalantir builds the world’s leading software for data-driven decisions and operations. By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving drugs, forecast supply chain disruptions, locate missing children, and more.\n\n\nThe Role\n \nPalantir’s in-house legal team works to proactively address legal issues so that Palantir can continue to drive positive impact in the world. We are deeply embedded in Palantir’s mission and culture, take time to connect with folks across the global Palantir community, and are viewed across the company as key to Palantir’s success. \n\nLegal Operations is the organizational backbone of the legal team at Palantir and our Deal Administrators directly support all aspects of revenue generation at the company. Our Deal Operations Administrators have a strong understanding of Palantir’s growth strategy, and are eager to implement and operationalize that strategy in our contracting processes. Our Deal Operations Administrators have a passion for helping others, making them an invaluable resource to our customer deal pipeline.\n\nAs a Deal Operations Administrator, you will be handling a wide variety of responsibilities throughout the customer contracting lifecycle. You will support Palantir’s lawyers and contracting specialists, in collaboration with our sales team and executives, to facilitate the growth of our business by managing all aspects of contract administration. As part of the Deal Operations team, you will become an expert in your specific customer vertical, take ownership for specialized scaling projects and critical initiatives as it relates to each business unit. Throughout your work, you will take initiative, operate with resourcefulness, and execute quickly and effectively the processes that drive revenue at Palantir. You’ll leverage your excellent communication skills, and exercise tact and diplomacy in helping to manage relationships with teams across Palantir. Most importantly, you have a true desire to be helpful and go above and beyond to solve challenges.\n",
	355	+ "id": "8c59a77a-1a57-4a2b-9d03-0b757b719015",
	356	+ "text": "Deal Operations Administrator",
	357	+ "country": "US",
	358	+ "workplaceType": "hybrid",
	359	+ "hostedUrl": "https://jobs.lever.co/palantir/8c59a77a-1a57-4a2b-9d03-0b757b719015",
	360	+ "applyUrl": "https://jobs.lever.co/palantir/8c59a77a-1a57-4a2b-9d03-0b757b719015/apply"
	361	+ }
	362	+]
		\ No newline at end of file

added fixtures/connectors/personio/personio_jobs.xml +25 −0

		`@@ -0,0 +1,25 @@`
	1	+<?xml version="1.0" encoding="UTF-8"?>
	2	+
	3	+<workzag-jobs>
	4	+
	5	+<position>
	6	+ <id>1834171</id>
	7	+ <subcompany>Personio SE & Co. KG</subcompany>
	8	+ <office>Munich</office>
	9	+ <additionalOffices>
	10	+ <office>Berlin</office>
	11	+ </additionalOffices>
	12	+ <department>Product and Tech</department>
	13	+ <recruitingCategory>Engineering</recruitingCategory>
	14	+ <name>Staff Software Engineer, Data Platform</name>
	15	+ <jobDescriptions></jobDescriptions>
	16	+ <employmentType>permanent</employmentType>
	17	+ <seniority>experienced</seniority>
	18	+ <schedule>full-time</schedule>
	19	+ <yearsOfExperience>7-10</yearsOfExperience>
	20	+ <occupation>software_and_web_development</occupation>
	21	+ <occupationCategory>it_software</occupationCategory>
	22	+ <createdAt>2024-11-13T14:10:41+00:00</createdAt>
	23	+</position>
	24	+
	25	+</workzag-jobs>
		\ No newline at end of file

added fixtures/connectors/recruitee/vandebron_offers.json +851 −0

		`@@ -0,0 +1,851 @@`
	1	+{
	2	+ "offers": [
	3	+ {
	4	+ "position": 506,
	5	+ "min_hours_per_week": "32",
	6	+ "created_at": "2026-08-26 08:52:52 UTC",
	7	+ "city": "Amsterdam",
	8	+ "guid": "c2hs6",
	9	+ "highlight": null,
	10	+ "location_question_visible": false,
	11	+ "locations_question_type": "multiple_choice",
	12	+ "options_title": "off",
	13	+ "country": "Nederland",
	14	+ "options_phone": "required",
	15	+ "updated_at": "2026-09-07 08:50:35 UTC",
	16	+ "max_hours_per_week": "40",
	17	+ "country_code": "NL",
	18	+ "experience_code": "experienced",
	19	+ "state_name": "Noord-Holland",
	20	+ "cover_image": null,
	21	+ "title": "Manager Market Operations",
	22	+ "published_at": "2026-09-01 12:17:30 UTC",
	23	+ "options_cv": "required",
	24	+ "hybrid": true,
	25	+ "category_code": "customer_service",
	26	+ "slug": "manager-market-operations-1",
	27	+ "state_code": "NH",
	28	+ "careers_url": "https://werkenbij.vandebron.nl/o/manager-market-operations-1",
	29	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/manager-market-operations-1/c/new",
	30	+ "mailbox_email": "job.c2hs6@vandebron.recruitee.com",
	31	+ "options_photo": "off",
	32	+ "remote": false,
	33	+ "location": "Amsterdam, Noord-Holland, Nederland",
	34	+ "company_name": "Vandebron",
	35	+ "locations_question_required": true,
	36	+ "locations_question": "What is your preferred work location?",
	37	+ "id": 2722598,
	38	+ "max_hours": 40,
	39	+ "on_site": false,
	40	+ "salary": {
	41	+ "max": null,
	42	+ "min": null,
	43	+ "period": null,
	44	+ "currency": null
	45	+ },
	46	+ "postal_code": "1013KS",
	47	+ "close_at": null,
	48	+ "department": "Customer Operations",
	49	+ "employment_type_code": "fulltime_permanent",
	50	+ "locations": [
	51	+ {
	52	+ "id": 62501,
	53	+ "name": "Amsterdam",
	54	+ "state": "Noord-Holland",
	55	+ "country": "Netherlands",
	56	+ "city": "Amsterdam",
	57	+ "translations": {
	58	+ "en": {
	59	+ "name": "Amsterdam",
	60	+ "city": "Amsterdam",
	61	+ "postal_code": "1013KS",
	62	+ "street": "Grote Bickersstraat 2",
	63	+ "note": null
	64	+ }
	65	+ },
	66	+ "country_code": "NL",
	67	+ "state_code": "NH",
	68	+ "postal_code": "1013KS",
	69	+ "street": "Grote Bickersstraat 2",
	70	+ "note": null
	71	+ }
	72	+ ],
	73	+ "status": "published",
	74	+ "options_salutation": "off",
	75	+ "options_cover_letter": "optional",
	76	+ "min_hours": 32,
	77	+ "tags": [],
	78	+ "education_code": "bachelor_degree"
	79	+ },
	80	+ {
	81	+ "position": 486,
	82	+ "min_hours_per_week": null,
	83	+ "created_at": "2026-08-14 08:22:34 UTC",
	84	+ "city": "Amsterdam",
	85	+ "guid": "onosp",
	86	+ "highlight": null,
	87	+ "location_question_visible": false,
	88	+ "locations_question_type": "multiple_choice",
	89	+ "options_title": "off",
	90	+ "country": "Nederland",
	91	+ "options_phone": "required",
	92	+ "updated_at": "2026-09-07 09:02:03 UTC",
	93	+ "max_hours_per_week": null,
	94	+ "country_code": "NL",
	95	+ "experience_code": "experienced",
	96	+ "state_name": "Noord-Holland",
	97	+ "cover_image": null,
	98	+ "title": "Sourcing & Pricing Analyst",
	99	+ "published_at": "2026-08-14 08:23:10 UTC",
	100	+ "options_cv": "required",
	101	+ "hybrid": true,
	102	+ "category_code": "finance",
	103	+ "slug": "sourcing-pricing-analyst",
	104	+ "state_code": "NH",
	105	+ "careers_url": "https://werkenbij.vandebron.nl/o/sourcing-pricing-analyst",
	106	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/sourcing-pricing-analyst/c/new",
	107	+ "mailbox_email": "job.onosp@vandebron.recruitee.com",
	108	+ "options_photo": "off",
	109	+ "remote": false,
	110	+ "location": "Amsterdam, Noord-Holland, Nederland",
	111	+ "company_name": "Vandebron",
	112	+ "locations_question_required": true,
	113	+ "locations_question": "What is your preferred work location?",
	114	+ "id": 2710502,
	115	+ "max_hours": null,
	116	+ "on_site": false,
	117	+ "salary": {
	118	+ "max": null,
	119	+ "min": null,
	120	+ "period": null,
	121	+ "currency": null
	122	+ },
	123	+ "postal_code": "1013KS",
	124	+ "close_at": null,
	125	+ "department": "Finance & Control",
	126	+ "employment_type_code": "fulltime_permanent",
	127	+ "locations": [
	128	+ {
	129	+ "id": 62501,
	130	+ "name": "Amsterdam",
	131	+ "state": "Noord-Holland",
	132	+ "country": "Netherlands",
	133	+ "city": "Amsterdam",
	134	+ "translations": {
	135	+ "en": {
	136	+ "name": "Amsterdam",
	137	+ "city": "Amsterdam",
	138	+ "postal_code": "1013KS",
	139	+ "street": "Grote Bickersstraat 2",
	140	+ "note": null
	141	+ }
	142	+ },
	143	+ "country_code": "NL",
	144	+ "state_code": "NH",
	145	+ "postal_code": "1013KS",
	146	+ "street": "Grote Bickersstraat 2",
	147	+ "note": null
	148	+ }
	149	+ ],
	150	+ "status": "published",
	151	+ "options_salutation": "off",
	152	+ "options_cover_letter": "optional",
	153	+ "min_hours": null,
	154	+ "tags": [],
	155	+ "education_code": "bachelor_degree"
	156	+ },
	157	+ {
	158	+ "position": 485,
	159	+ "min_hours_per_week": null,
	160	+ "created_at": "2026-08-14 07:50:39 UTC",
	161	+ "city": "Amsterdam",
	162	+ "guid": "4t36b",
	163	+ "highlight": null,
	164	+ "location_question_visible": false,
	165	+ "locations_question_type": "multiple_choice",
	166	+ "options_title": "off",
	167	+ "country": "Nederland",
	168	+ "options_phone": "required",
	169	+ "updated_at": "2026-09-07 09:01:09 UTC",
	170	+ "max_hours_per_week": null,
	171	+ "country_code": "NL",
	172	+ "experience_code": "experienced",
	173	+ "state_name": "Noord-Holland",
	174	+ "cover_image": null,
	175	+ "title": "Gross Margin Analyst",
	176	+ "published_at": "2026-08-14 07:56:36 UTC",
	177	+ "options_cv": "required",
	178	+ "hybrid": true,
	179	+ "category_code": "finance",
	180	+ "slug": "gross-margin-analyst",
	181	+ "state_code": "NH",
	182	+ "careers_url": "https://werkenbij.vandebron.nl/o/gross-margin-analyst",
	183	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/gross-margin-analyst/c/new",
	184	+ "mailbox_email": "job.4t36b@vandebron.recruitee.com",
	185	+ "options_photo": "off",
	186	+ "remote": false,
	187	+ "location": "Amsterdam, Noord-Holland, Nederland",
	188	+ "company_name": "Vandebron",
	189	+ "locations_question_required": true,
	190	+ "locations_question": "What is your preferred work location?",
	191	+ "id": 2710378,
	192	+ "max_hours": null,
	193	+ "on_site": false,
	194	+ "salary": {
	195	+ "max": null,
	196	+ "min": null,
	197	+ "period": null,
	198	+ "currency": null
	199	+ },
	200	+ "postal_code": "1013KS",
	201	+ "close_at": null,
	202	+ "department": "Finance & Control",
	203	+ "employment_type_code": "fulltime_permanent",
	204	+ "locations": [
	205	+ {
	206	+ "id": 62501,
	207	+ "name": "Amsterdam",
	208	+ "state": "Noord-Holland",
	209	+ "country": "Netherlands",
	210	+ "city": "Amsterdam",
	211	+ "translations": {
	212	+ "en": {
	213	+ "name": "Amsterdam",
	214	+ "city": "Amsterdam",
	215	+ "postal_code": "1013KS",
	216	+ "street": "Grote Bickersstraat 2",
	217	+ "note": null
	218	+ }
	219	+ },
	220	+ "country_code": "NL",
	221	+ "state_code": "NH",
	222	+ "postal_code": "1013KS",
	223	+ "street": "Grote Bickersstraat 2",
	224	+ "note": null
	225	+ }
	226	+ ],
	227	+ "status": "published",
	228	+ "options_salutation": "off",
	229	+ "options_cover_letter": "optional",
	230	+ "min_hours": null,
	231	+ "tags": [],
	232	+ "education_code": "bachelor_degree"
	233	+ },
	234	+ {
	235	+ "position": 484,
	236	+ "min_hours_per_week": null,
	237	+ "created_at": "2026-08-05 12:07:03 UTC",
	238	+ "city": "Amsterdam",
	239	+ "guid": "bw9no",
	240	+ "highlight": null,
	241	+ "location_question_visible": false,
	242	+ "locations_question_type": "multiple_choice",
	243	+ "options_title": "off",
	244	+ "country": "Nederland",
	245	+ "options_phone": "required",
	246	+ "updated_at": "2026-09-07 08:51:08 UTC",
	247	+ "max_hours_per_week": null,
	248	+ "country_code": "NL",
	249	+ "experience_code": "mid_level",
	250	+ "state_name": "Noord-Holland",
	251	+ "cover_image": null,
	252	+ "title": "Senior CX Specialist",
	253	+ "published_at": "2026-08-05 12:08:08 UTC",
	254	+ "options_cv": "required",
	255	+ "hybrid": true,
	256	+ "category_code": "marketing_pr",
	257	+ "slug": "senior-cx-specialist",
	258	+ "state_code": "NH",
	259	+ "careers_url": "https://werkenbij.vandebron.nl/o/senior-cx-specialist",
	260	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/senior-cx-specialist/c/new",
	261	+ "mailbox_email": "job.bw9no@vandebron.recruitee.com",
	262	+ "options_photo": "off",
	263	+ "remote": false,
	264	+ "location": "Amsterdam, Noord-Holland, Nederland",
	265	+ "company_name": "Vandebron",
	266	+ "locations_question_required": true,
	267	+ "locations_question": "Welke werklocatie heeft je voorkeur?",
	268	+ "id": 2701124,
	269	+ "max_hours": null,
	270	+ "on_site": false,
	271	+ "salary": {
	272	+ "max": null,
	273	+ "min": null,
	274	+ "period": null,
	275	+ "currency": null
	276	+ },
	277	+ "postal_code": "1013KS",
	278	+ "close_at": null,
	279	+ "department": "Marketing",
	280	+ "employment_type_code": "fulltime_permanent",
	281	+ "locations": [
	282	+ {
	283	+ "id": 62501,
	284	+ "name": "Amsterdam",
	285	+ "state": "Noord-Holland",
	286	+ "country": "Nederland",
	287	+ "city": "Amsterdam",
	288	+ "translations": {
	289	+ "nl": {
	290	+ "name": "Amsterdam",
	291	+ "city": "Amsterdam",
	292	+ "postal_code": "1013KS",
	293	+ "street": "Grote Bickersstraat 2",
	294	+ "note": null
	295	+ }
	296	+ },
	297	+ "country_code": "NL",
	298	+ "state_code": "NH",
	299	+ "postal_code": "1013KS",
	300	+ "street": "Grote Bickersstraat 2",
	301	+ "note": null
	302	+ }
	303	+ ],
	304	+ "status": "published",
	305	+ "options_salutation": "off",
	306	+ "options_cover_letter": "optional",
	307	+ "min_hours": null,
	308	+ "tags": [],
	309	+ "education_code": "bachelor_degree"
	310	+ },
	311	+ {
	312	+ "position": 483,
	313	+ "min_hours_per_week": null,
	314	+ "created_at": "2026-07-31 08:52:40 UTC",
	315	+ "city": "Amsterdam",
	316	+ "guid": "mf173",
	317	+ "highlight": null,
	318	+ "location_question_visible": false,
	319	+ "locations_question_type": "multiple_choice",
	320	+ "options_title": "off",
	321	+ "country": "Nederland",
	322	+ "options_phone": "required",
	323	+ "updated_at": "2026-09-07 08:51:37 UTC",
	324	+ "max_hours_per_week": null,
	325	+ "country_code": "NL",
	326	+ "experience_code": "mid_level",
	327	+ "state_name": "Noord-Holland",
	328	+ "cover_image": null,
	329	+ "title": "Customer Experience Manager",
	330	+ "published_at": "2026-07-31 08:59:50 UTC",
	331	+ "options_cv": "required",
	332	+ "hybrid": true,
	333	+ "category_code": "marketing_pr",
	334	+ "slug": "customer-experience-manager",
	335	+ "state_code": "NH",
	336	+ "careers_url": "https://werkenbij.vandebron.nl/o/customer-experience-manager",
	337	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/customer-experience-manager/c/new",
	338	+ "mailbox_email": "job.mf173@vandebron.recruitee.com",
	339	+ "options_photo": "off",
	340	+ "remote": false,
	341	+ "location": "Amsterdam, Noord-Holland, Nederland",
	342	+ "company_name": "Vandebron",
	343	+ "locations_question_required": true,
	344	+ "locations_question": "Welke werklocatie heeft je voorkeur?",
	345	+ "id": 2696453,
	346	+ "max_hours": null,
	347	+ "on_site": false,
	348	+ "salary": {
	349	+ "max": null,
	350	+ "min": null,
	351	+ "period": null,
	352	+ "currency": null
	353	+ },
	354	+ "postal_code": "1013KS",
	355	+ "close_at": null,
	356	+ "department": "Marketing",
	357	+ "employment_type_code": "fulltime_permanent",
	358	+ "locations": [
	359	+ {
	360	+ "id": 62501,
	361	+ "name": "Amsterdam",
	362	+ "state": "Noord-Holland",
	363	+ "country": "Nederland",
	364	+ "city": "Amsterdam",
	365	+ "translations": {
	366	+ "nl": {
	367	+ "name": "Amsterdam",
	368	+ "city": "Amsterdam",
	369	+ "postal_code": "1013KS",
	370	+ "street": "Grote Bickersstraat 2",
	371	+ "note": null
	372	+ }
	373	+ },
	374	+ "country_code": "NL",
	375	+ "state_code": "NH",
	376	+ "postal_code": "1013KS",
	377	+ "street": "Grote Bickersstraat 2",
	378	+ "note": null
	379	+ }
	380	+ ],
	381	+ "status": "published",
	382	+ "options_salutation": "off",
	383	+ "options_cover_letter": "optional",
	384	+ "min_hours": null,
	385	+ "tags": [],
	386	+ "education_code": "bachelor_degree"
	387	+ },
	388	+ {
	389	+ "position": 468,
	390	+ "min_hours_per_week": "32",
	391	+ "created_at": "2026-04-15 13:18:53 UTC",
	392	+ "city": "Amsterdam",
	393	+ "guid": "vu56y",
	394	+ "highlight": null,
	395	+ "location_question_visible": false,
	396	+ "locations_question_type": "multiple_choice",
	397	+ "options_title": "off",
	398	+ "country": "Nederland",
	399	+ "options_phone": "required",
	400	+ "updated_at": "2026-09-09 07:57:14 UTC",
	401	+ "max_hours_per_week": "40",
	402	+ "country_code": "NL",
	403	+ "experience_code": "mid_level",
	404	+ "state_name": "Noord-Holland",
	405	+ "cover_image": null,
	406	+ "title": "Customer Service Specialist",
	407	+ "published_at": "2026-04-15 13:19:14 UTC",
	408	+ "options_cv": "required",
	409	+ "hybrid": false,
	410	+ "category_code": "customer_service",
	411	+ "slug": "customer-service-specialist",
	412	+ "state_code": "NH",
	413	+ "careers_url": "https://werkenbij.vandebron.nl/o/customer-service-specialist",
	414	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/customer-service-specialist/c/new",
	415	+ "mailbox_email": "job.vu56y@vandebron.recruitee.com",
	416	+ "options_photo": "off",
	417	+ "remote": false,
	418	+ "location": "Amsterdam, Noord-Holland, Nederland",
	419	+ "company_name": "Vandebron",
	420	+ "locations_question_required": true,
	421	+ "locations_question": "Welke werklocatie heeft je voorkeur?",
	422	+ "id": 2567584,
	423	+ "max_hours": 40,
	424	+ "on_site": true,
	425	+ "salary": {
	426	+ "max": null,
	427	+ "min": null,
	428	+ "period": null,
	429	+ "currency": null
	430	+ },
	431	+ "postal_code": "1013KS",
	432	+ "close_at": null,
	433	+ "department": "Customer Operations",
	434	+ "employment_type_code": "fulltime_permanent",
	435	+ "locations": [
	436	+ {
	437	+ "id": 62501,
	438	+ "name": "Amsterdam",
	439	+ "state": "Noord-Holland",
	440	+ "country": "Nederland",
	441	+ "city": "Amsterdam",
	442	+ "translations": {
	443	+ "nl": {
	444	+ "name": "Amsterdam",
	445	+ "city": "Amsterdam",
	446	+ "postal_code": "1013KS",
	447	+ "street": "Grote Bickersstraat 2",
	448	+ "note": null
	449	+ }
	450	+ },
	451	+ "country_code": "NL",
	452	+ "state_code": "NH",
	453	+ "postal_code": "1013KS",
	454	+ "street": "Grote Bickersstraat 2",
	455	+ "note": null
	456	+ }
	457	+ ],
	458	+ "status": "published",
	459	+ "options_salutation": "off",
	460	+ "options_cover_letter": "optional",
	461	+ "min_hours": 32,
	462	+ "tags": [],
	463	+ "education_code": "associate_degree"
	464	+ },
	465	+ {
	466	+ "position": 466,
	467	+ "min_hours_per_week": "32",
	468	+ "created_at": "2026-04-13 13:22:59 UTC",
	469	+ "city": "Amsterdam",
	470	+ "guid": "353rp",
	471	+ "highlight": null,
	472	+ "location_question_visible": false,
	473	+ "locations_question_type": "multiple_choice",
	474	+ "options_title": "off",
	475	+ "country": "Nederland",
	476	+ "options_phone": "required",
	477	+ "updated_at": "2026-09-09 07:57:28 UTC",
	478	+ "max_hours_per_week": "40",
	479	+ "country_code": "NL",
	480	+ "experience_code": "mid_level",
	481	+ "state_name": "Noord-Holland",
	482	+ "cover_image": null,
	483	+ "title": "Energy Specialist",
	484	+ "published_at": "2026-04-13 13:24:31 UTC",
	485	+ "options_cv": "required",
	486	+ "hybrid": false,
	487	+ "category_code": "customer_service",
	488	+ "slug": "energy-specialist",
	489	+ "state_code": "NH",
	490	+ "careers_url": "https://werkenbij.vandebron.nl/o/energy-specialist",
	491	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/energy-specialist/c/new",
	492	+ "mailbox_email": "job.353rp@vandebron.recruitee.com",
	493	+ "options_photo": "off",
	494	+ "remote": false,
	495	+ "location": "Amsterdam, Noord-Holland, Nederland",
	496	+ "company_name": "Vandebron",
	497	+ "locations_question_required": true,
	498	+ "locations_question": "Welke werklocatie heeft je voorkeur?",
	499	+ "id": 2564243,
	500	+ "max_hours": 40,
	501	+ "on_site": true,
	502	+ "salary": {
	503	+ "max": null,
	504	+ "min": null,
	505	+ "period": null,
	506	+ "currency": null
	507	+ },
	508	+ "postal_code": "1013KS",
	509	+ "close_at": null,
	510	+ "department": "Customer Operations",
	511	+ "employment_type_code": "fulltime_permanent",
	512	+ "locations": [
	513	+ {
	514	+ "id": 62501,
	515	+ "name": "Amsterdam",
	516	+ "state": "Noord-Holland",
	517	+ "country": "Nederland",
	518	+ "city": "Amsterdam",
	519	+ "translations": {
	520	+ "nl": {
	521	+ "name": "Amsterdam",
	522	+ "city": "Amsterdam",
	523	+ "postal_code": "1013KS",
	524	+ "street": "Grote Bickersstraat 2",
	525	+ "note": null
	526	+ }
	527	+ },
	528	+ "country_code": "NL",
	529	+ "state_code": "NH",
	530	+ "postal_code": "1013KS",
	531	+ "street": "Grote Bickersstraat 2",
	532	+ "note": null
	533	+ }
	534	+ ],
	535	+ "status": "published",
	536	+ "options_salutation": "off",
	537	+ "options_cover_letter": "optional",
	538	+ "min_hours": 32,
	539	+ "tags": [],
	540	+ "education_code": "associate_degree"
	541	+ },
	542	+ {
	543	+ "position": 415,
	544	+ "min_hours_per_week": "16",
	545	+ "created_at": "2025-04-25 07:28:43 UTC",
	546	+ "city": "Amsterdam",
	547	+ "guid": "fw3lw",
	548	+ "highlight": null,
	549	+ "location_question_visible": false,
	550	+ "locations_question_type": "multiple_choice",
	551	+ "options_title": "off",
	552	+ "country": "Nederland",
	553	+ "options_phone": "required",
	554	+ "updated_at": "2026-08-04 10:41:44 UTC",
	555	+ "max_hours_per_week": "40",
	556	+ "country_code": "NL",
	557	+ "experience_code": "student_school",
	558	+ "state_name": "Noord-Holland",
	559	+ "cover_image": null,
	560	+ "title": "Sales Adviseur (bijbaan)",
	561	+ "published_at": "2025-08-04 12:08:15 UTC",
	562	+ "options_cv": "required",
	563	+ "hybrid": false,
	564	+ "category_code": "customer_service",
	565	+ "slug": "sales-adviseur-bijbaan-2-3",
	566	+ "state_code": "NH",
	567	+ "careers_url": "https://werkenbij.vandebron.nl/o/sales-adviseur-bijbaan-2-3",
	568	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/sales-adviseur-bijbaan-2-3/c/new",
	569	+ "mailbox_email": "job.fw3lw@vandebron.recruitee.com",
	570	+ "options_photo": "optional",
	571	+ "remote": false,
	572	+ "location": "Amsterdam, Noord-Holland, Nederland",
	573	+ "company_name": "Vandebron",
	574	+ "locations_question_required": true,
	575	+ "locations_question": "Welke werklocatie heeft je voorkeur?",
	576	+ "id": 2117052,
	577	+ "max_hours": 40,
	578	+ "on_site": true,
	579	+ "salary": {
	580	+ "max": null,
	581	+ "min": "15",
	582	+ "period": "hour",
	583	+ "currency": "EUR"
	584	+ },
	585	+ "postal_code": "1013 KS",
	586	+ "close_at": null,
	587	+ "department": "Bijbanen",
	588	+ "employment_type_code": "parttime_fixed_term",
	589	+ "locations": [
	590	+ {
	591	+ "id": 62497,
	592	+ "name": "Amsterdam, 1013 KS",
	593	+ "state": "Noord-Holland",
	594	+ "country": "Nederland",
	595	+ "city": "Amsterdam",
	596	+ "translations": {
	597	+ "nl": {
	598	+ "name": "Amsterdam, 1013 KS",
	599	+ "city": "Amsterdam",
	600	+ "postal_code": "1013 KS",
	601	+ "street": "Grote Bickersstraat 2",
	602	+ "note": null
	603	+ }
	604	+ },
	605	+ "country_code": "NL",
	606	+ "state_code": "NH",
	607	+ "postal_code": "1013 KS",
	608	+ "street": "Grote Bickersstraat 2",
	609	+ "note": null
	610	+ }
	611	+ ],
	612	+ "status": "published",
	613	+ "options_salutation": "off",
	614	+ "options_cover_letter": "optional",
	615	+ "min_hours": 16,
	616	+ "tags": [],
	617	+ "education_code": "high_school"
	618	+ },
	619	+ {
	620	+ "position": 5,
	621	+ "min_hours_per_week": "24",
	622	+ "created_at": "2022-05-20 13:10:23 UTC",
	623	+ "city": "Amsterdam",
	624	+ "guid": "2dvvr",
	625	+ "highlight": null,
	626	+ "location_question_visible": false,
	627	+ "locations_question_type": "multiple_choice",
	628	+ "options_title": "off",
	629	+ "country": "Nederland",
	630	+ "options_phone": "required",
	631	+ "updated_at": "2026-09-09 07:07:08 UTC",
	632	+ "max_hours_per_week": "40",
	633	+ "country_code": "NL",
	634	+ "experience_code": "entry_level",
	635	+ "state_name": "Noord-Holland",
	636	+ "cover_image": null,
	637	+ "title": "Customer Service Medewerker",
	638	+ "published_at": "2025-07-03 12:40:23 UTC",
	639	+ "options_cv": "required",
	640	+ "hybrid": false,
	641	+ "category_code": "customer_service",
	642	+ "slug": "customer-service-medewerker",
	643	+ "state_code": "NH",
	644	+ "careers_url": "https://werkenbij.vandebron.nl/o/customer-service-medewerker",
	645	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/customer-service-medewerker/c/new",
	646	+ "mailbox_email": "job.2dvvr@vandebron.recruitee.com",
	647	+ "options_photo": "off",
	648	+ "remote": false,
	649	+ "location": "Amsterdam, Noord-Holland, Nederland",
	650	+ "company_name": "Vandebron",
	651	+ "locations_question_required": true,
	652	+ "locations_question": "Welke werklocatie heeft je voorkeur?",
	653	+ "id": 979749,
	654	+ "max_hours": 40,
	655	+ "on_site": true,
	656	+ "salary": {
	657	+ "max": null,
	658	+ "min": "2600",
	659	+ "period": "month",
	660	+ "currency": "EUR"
	661	+ },
	662	+ "postal_code": "1013 KS",
	663	+ "close_at": null,
	664	+ "department": "Customer Operations",
	665	+ "employment_type_code": "fulltime_fixed_term",
	666	+ "locations": [
	667	+ {
	668	+ "id": 62497,
	669	+ "name": "Amsterdam, 1013 KS",
	670	+ "state": "Noord-Holland",
	671	+ "country": "Nederland",
	672	+ "city": "Amsterdam",
	673	+ "translations": {
	674	+ "nl": {
	675	+ "name": "Amsterdam, 1013 KS",
	676	+ "city": "Amsterdam",
	677	+ "postal_code": "1013 KS",
	678	+ "street": "Grote Bickersstraat 2",
	679	+ "note": null
	680	+ }
	681	+ },
	682	+ "country_code": "NL",
	683	+ "state_code": "NH",
	684	+ "postal_code": "1013 KS",
	685	+ "street": "Grote Bickersstraat 2",
	686	+ "note": null
	687	+ }
	688	+ ],
	689	+ "status": "published",
	690	+ "options_salutation": "off",
	691	+ "options_cover_letter": "off",
	692	+ "min_hours": 24,
	693	+ "tags": [],
	694	+ "education_code": "high_school"
	695	+ },
	696	+ {
	697	+ "position": 4,
	698	+ "min_hours_per_week": "16",
	699	+ "created_at": "2022-05-20 12:06:38 UTC",
	700	+ "city": "Amsterdam",
	701	+ "guid": "45ls9",
	702	+ "highlight": null,
	703	+ "location_question_visible": false,
	704	+ "locations_question_type": "multiple_choice",
	705	+ "options_title": "off",
	706	+ "country": "Nederland",
	707	+ "options_phone": "required",
	708	+ "updated_at": "2026-09-08 09:24:06 UTC",
	709	+ "max_hours_per_week": "40",
	710	+ "country_code": "NL",
	711	+ "experience_code": "student_college",
	712	+ "state_name": "Noord-Holland",
	713	+ "cover_image": null,
	714	+ "title": "Service Agent (bijbaan)",
	715	+ "published_at": "2025-10-09 10:43:27 UTC",
	716	+ "options_cv": "required",
	717	+ "hybrid": false,
	718	+ "category_code": "customer_service",
	719	+ "slug": "service-expert-bijbaan-1",
	720	+ "state_code": "NH",
	721	+ "careers_url": "https://werkenbij.vandebron.nl/o/service-expert-bijbaan-1",
	722	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/service-expert-bijbaan-1/c/new",
	723	+ "mailbox_email": "job.45ls9@vandebron.recruitee.com",
	724	+ "options_photo": "off",
	725	+ "remote": false,
	726	+ "location": "Amsterdam, Noord-Holland, Nederland",
	727	+ "company_name": "Vandebron",
	728	+ "locations_question_required": true,
	729	+ "locations_question": "Welke werklocatie heeft je voorkeur?",
	730	+ "id": 979624,
	731	+ "max_hours": 40,
	732	+ "on_site": true,
	733	+ "salary": {
	734	+ "max": null,
	735	+ "min": "15",
	736	+ "period": "hour",
	737	+ "currency": "EUR"
	738	+ },
	739	+ "postal_code": "1013KS",
	740	+ "close_at": null,
	741	+ "department": "Bijbanen",
	742	+ "employment_type_code": "parttime_fixed_term",
	743	+ "locations": [
	744	+ {
	745	+ "id": 62501,
	746	+ "name": "Amsterdam",
	747	+ "state": "Noord-Holland",
	748	+ "country": "Nederland",
	749	+ "city": "Amsterdam",
	750	+ "translations": {
	751	+ "nl": {
	752	+ "name": "Amsterdam",
	753	+ "city": "Amsterdam",
	754	+ "postal_code": "1013KS",
	755	+ "street": "Grote Bickersstraat 2",
	756	+ "note": null
	757	+ }
	758	+ },
	759	+ "country_code": "NL",
	760	+ "state_code": "NH",
	761	+ "postal_code": "1013KS",
	762	+ "street": "Grote Bickersstraat 2",
	763	+ "note": null
	764	+ }
	765	+ ],
	766	+ "status": "published",
	767	+ "options_salutation": "off",
	768	+ "options_cover_letter": "optional",
	769	+ "min_hours": 16,
	770	+ "tags": [],
	771	+ "education_code": "high_school"
	772	+ },
	773	+ {
	774	+ "position": 1,
	775	+ "min_hours_per_week": "30.0",
	776	+ "created_at": "2019-05-28 17:19:39 UTC",
	777	+ "city": "Amsterdam",
	778	+ "guid": "5nfa3",
	779	+ "highlight": null,
	780	+ "location_question_visible": false,
	781	+ "locations_question_type": "multiple_choice",
	782	+ "options_title": "off",
	783	+ "country": "Nederland",
	784	+ "options_phone": "required",
	785	+ "updated_at": "2026-08-03 10:15:36 UTC",
	786	+ "max_hours_per_week": "40.0",
	787	+ "country_code": "NL",
	788	+ "experience_code": "experienced",
	789	+ "state_name": "Noord-Holland",
	790	+ "cover_image": null,
	791	+ "title": "Open sollicitatie",
	792	+ "published_at": "2019-05-28 17:22:43 UTC",
	793	+ "options_cv": "required",
	794	+ "hybrid": false,
	795	+ "category_code": "other",
	796	+ "slug": "open-sollicitatie",
	797	+ "state_code": "NH",
	798	+ "careers_url": "https://werkenbij.vandebron.nl/o/open-sollicitatie",
	799	+ "careers_apply_url": "https://werkenbij.vandebron.nl/o/open-sollicitatie/c/new",
	800	+ "mailbox_email": "job.5nfa3@vandebron.recruitee.com",
	801	+ "options_photo": "optional",
	802	+ "remote": false,
	803	+ "location": "Amsterdam, Noord-Holland, Nederland",
	804	+ "company_name": "Vandebron",
	805	+ "locations_question_required": true,
	806	+ "locations_question": "Welke werklocatie heeft je voorkeur?",
	807	+ "id": 312680,
	808	+ "max_hours": null,
	809	+ "on_site": true,
	810	+ "salary": {
	811	+ "max": null,
	812	+ "min": null,
	813	+ "period": null,
	814	+ "currency": null
	815	+ },
	816	+ "postal_code": "1013 KS",
	817	+ "close_at": null,
	818	+ "department": "Overig",
	819	+ "employment_type_code": "fulltime_permanent",
	820	+ "locations": [
	821	+ {
	822	+ "id": 52693,
	823	+ "name": "Vandebron HQ",
	824	+ "state": "Noord-Holland",
	825	+ "country": "Nederland",
	826	+ "city": "Amsterdam",
	827	+ "translations": {
	828	+ "nl": {
	829	+ "name": "Vandebron HQ",
	830	+ "city": "Amsterdam",
	831	+ "postal_code": "1013 KS",
	832	+ "street": "Grote Bickersstraat 2a",
	833	+ "note": null
	834	+ }
	835	+ },
	836	+ "country_code": "NL",
	837	+ "state_code": "NH",
	838	+ "postal_code": "1013 KS",
	839	+ "street": "Grote Bickersstraat 2a",
	840	+ "note": null
	841	+ }
	842	+ ],
	843	+ "status": "published",
	844	+ "options_salutation": "off",
	845	+ "options_cover_letter": "optional",
	846	+ "min_hours": null,
	847	+ "tags": [],
	848	+ "education_code": "professional"
	849	+ }
	850	+ ]
	851	+}
		\ No newline at end of file

added fixtures/connectors/sitemap/sitemap_index.xml +5 −0

		`@@ -0,0 +1,5 @@`
	1	+<?xml version="1.0" encoding="UTF-8"?>
	2	+<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
	3	+ <sitemap><loc>https://www.acme-cloud.example/sitemap-pages.xml</loc><lastmod>2026-09-10T08:00:00+00:00</lastmod></sitemap>
	4	+ <sitemap><loc>https://www.acme-cloud.example/sitemap-blog.xml</loc><lastmod>2026-09-11</lastmod></sitemap>
	5	+</sitemapindex>

added fixtures/connectors/sitemap/sitemap_pages.xml +14 −0

		`@@ -0,0 +1,14 @@`
	1	+<?xml version="1.0" encoding="UTF-8"?>
	2	+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
	3	+ <url><loc>https://www.acme-cloud.example/</loc><lastmod>2026-09-01</lastmod><changefreq>daily</changefreq></url>
	4	+ <url><loc>https://www.acme-cloud.example/pricing</loc><lastmod>2026-08-20</lastmod></url>
	5	+ <url><loc>https://www.acme-cloud.example/careers</loc><lastmod>2026-09-09</lastmod></url>
	6	+ <url><loc>https://www.acme-cloud.example/about/leadership</loc><lastmod>2026-05-02</lastmod></url>
	7	+ <url><loc>https://www.acme-cloud.example/company/locations</loc></url>
	8	+ <url><loc>https://www.acme-cloud.example/news</loc><lastmod>2026-09-10</lastmod></url>
	9	+ <url><loc>https://www.acme-cloud.example/news/2026/09/acme-launches-atlas-ai</loc><lastmod>2026-09-10</lastmod></url>
	10	+ <url><loc>https://www.acme-cloud.example/legal/privacy</loc></url>
	11	+ <url><loc>https://www.acme-cloud.example/docs</loc></url>
	12	+ <url><loc>https://www.acme-cloud.example/changelog</loc><lastmod>2026-09-11</lastmod></url>
	13	+ <url><loc>https://www.acme-cloud.example/img/logo.png</loc></url>
	14	+</urlset>

added fixtures/connectors/smartrecruiters/smartrecruiters_postings.json +93 −0

		`@@ -0,0 +1,93 @@`
	1	+{
	2	+ "offset": 0,
	3	+ "limit": 100,
	4	+ "totalFound": 1,
	5	+ "content": [
	6	+ {
	7	+ "id": "744000148454651",
	8	+ "name": "Data Operations Consultant ",
	9	+ "uuid": "f4a3d5b9-5fa9-48af-ba55-30669c9c3e81",
	10	+ "jobAdId": "f9fbb59b-d9db-4c26-a4c9-959f920b370a",
	11	+ "defaultJobAd": true,
	12	+ "refNumber": "REF2025N",
	13	+ "company": {
	14	+ "identifier": "smartrecruiters",
	15	+ "name": "SmartRecruiters Inc"
	16	+ },
	17	+ "releasedDate": "2026-09-09T09:43:26.403Z",
	18	+ "location": {
	19	+ "city": "Poland",
	20	+ "region": "Remote",
	21	+ "country": "pl",
	22	+ "remote": true,
	23	+ "hybrid": false,
	24	+ "fullLocation": "Poland, Remote, Poland"
	25	+ },
	26	+ "industry": {
	27	+ "id": "computer_software",
	28	+ "label": "Computer Software"
	29	+ },
	30	+ "department": {
	31	+ "id": "5408931",
	32	+ "label": "Technical Services"
	33	+ },
	34	+ "function": {
	35	+ "id": "information_technology",
	36	+ "label": "Information Technology"
	37	+ },
	38	+ "typeOfEmployment": {
	39	+ "id": "contract",
	40	+ "label": "Contract"
	41	+ },
	42	+ "experienceLevel": {
	43	+ "id": "associate",
	44	+ "label": "Associate"
	45	+ },
	46	+ "customField": [
	47	+ {
	48	+ "fieldId": "58b7e4d6e4b0885c92cd98ee",
	49	+ "fieldLabel": "Department",
	50	+ "valueId": "5408931",
	51	+ "valueLabel": "Technical Services"
	52	+ },
	53	+ {
	54	+ "fieldId": "68f89f37181dbd53b2d51cc1",
	55	+ "fieldLabel": "SAP Cost Center",
	56	+ "valueId": "ccc63762-5f8e-43c3-b3e1-05a5ce9b24e1",
	57	+ "valueLabel": "545000309"
	58	+ },
	59	+ {
	60	+ "fieldId": "COUNTRY",
	61	+ "fieldLabel": "Country/Region",
	62	+ "valueId": "pl",
	63	+ "valueLabel": "Poland"
	64	+ },
	65	+ {
	66	+ "fieldId": "58b7e4d6e4b0885c92cd98eb",
	67	+ "fieldLabel": "Brands",
	68	+ "valueId": "default",
	69	+ "valueLabel": "SmartRecruiters Inc"
	70	+ },
	71	+ {
	72	+ "fieldId": "617043f52193e5503b7715c3",
	73	+ "fieldLabel": "SR Cost Center",
	74	+ "valueId": "e5f0ae46-b6d3-46ed-9607-12414b4c048e",
	75	+ "valueLabel": "3.06 COGS Technical Services"
	76	+ },
	77	+ {
	78	+ "fieldId": "627035b6330a2b4d25aa0882",
	79	+ "fieldLabel": "Career Site Department Labels",
	80	+ "valueId": "694b6ffd-b262-41ff-b6f7-18cc9c2354bd",
	81	+ "valueLabel": "Technology"
	82	+ }
	83	+ ],
	84	+ "visibility": "PUBLIC",
	85	+ "ref": "https://api.smartrecruiters.com/v1/companies/smartrecruiters/postings/744000148454651",
	86	+ "language": {
	87	+ "code": "en",
	88	+ "label": "English",
	89	+ "labelNative": "English (US)"
	90	+ }
	91	+ }
	92	+ ]
	93	+}
		\ No newline at end of file

added fixtures/connectors/statuspage/github_summary.json +197 −0

		`@@ -0,0 +1,197 @@`
	1	+{
	2	+ "page": {
	3	+ "id": "kctbh9vrtdwd",
	4	+ "name": "GitHub",
	5	+ "url": "https://www.githubstatus.com",
	6	+ "time_zone": "Etc/UTC",
	7	+ "updated_at": "2026-09-12T06:34:10.269Z"
	8	+ },
	9	+ "components": [
	10	+ {
	11	+ "id": "8l4ygp009s5s",
	12	+ "name": "Git Operations",
	13	+ "status": "operational",
	14	+ "created_at": "2017-01-31T20:05:05.370Z",
	15	+ "updated_at": "2026-08-17T18:23:47.907Z",
	16	+ "position": 1,
	17	+ "description": "Performance of git clones, pulls, pushes, and associated operations",
	18	+ "showcase": true,
	19	+ "start_date": null,
	20	+ "group_id": null,
	21	+ "page_id": "kctbh9vrtdwd",
	22	+ "group": false,
	23	+ "only_show_if_degraded": false
	24	+ },
	25	+ {
	26	+ "id": "4230lsnqdsld",
	27	+ "name": "Webhooks",
	28	+ "status": "operational",
	29	+ "created_at": "2019-11-13T18:00:24.256Z",
	30	+ "updated_at": "2026-08-17T16:59:38.017Z",
	31	+ "position": 2,
	32	+ "description": "Real time HTTP callbacks of user-generated and system events",
	33	+ "showcase": true,
	34	+ "start_date": null,
	35	+ "group_id": null,
	36	+ "page_id": "kctbh9vrtdwd",
	37	+ "group": false,
	38	+ "only_show_if_degraded": false
	39	+ },
	40	+ {
	41	+ "id": "0l2p9nhqnxpd",
	42	+ "name": "Visit www.githubstatus.com for more information",
	43	+ "status": "operational",
	44	+ "created_at": "2018-12-05T19:39:40.838Z",
	45	+ "updated_at": "2025-03-19T05:00:21.309Z",
	46	+ "position": 3,
	47	+ "description": null,
	48	+ "showcase": false,
	49	+ "start_date": null,
	50	+ "group_id": null,
	51	+ "page_id": "kctbh9vrtdwd",
	52	+ "group": false,
	53	+ "only_show_if_degraded": false
	54	+ },
	55	+ {
	56	+ "id": "brv1bkgrwx7q",
	57	+ "name": "API Requests",
	58	+ "status": "operational",
	59	+ "created_at": "2017-01-31T20:01:46.621Z",
	60	+ "updated_at": "2026-08-17T19:01:45.543Z",
	61	+ "position": 4,
	62	+ "description": "Requests for GitHub APIs",
	63	+ "showcase": true,
	64	+ "start_date": null,
	65	+ "group_id": null,
	66	+ "page_id": "kctbh9vrtdwd",
	67	+ "group": false,
	68	+ "only_show_if_degraded": false
	69	+ },
	70	+ {
	71	+ "id": "kr09ddfgbfsf",
	72	+ "name": "Issues",
	73	+ "status": "operational",
	74	+ "created_at": "2017-01-31T20:01:46.638Z",
	75	+ "updated_at": "2026-08-17T20:22:28.767Z",
	76	+ "position": 5,
	77	+ "description": "Requests for Issues on GitHub.com",
	78	+ "showcase": true,
	79	+ "start_date": null,
	80	+ "group_id": null,
	81	+ "page_id": "kctbh9vrtdwd",
	82	+ "group": false,
	83	+ "only_show_if_degraded": false
	84	+ },
	85	+ {
	86	+ "id": "hhtssxt0f5v2",
	87	+ "name": "Pull Requests",
	88	+ "status": "operational",
	89	+ "created_at": "2020-09-02T15:39:06.329Z",
	90	+ "updated_at": "2026-09-01T16:01:21.340Z",
	91	+ "position": 6,
	92	+ "description": "Requests for Pull Requests on GitHub.com",
	93	+ "showcase": true,
	94	+ "start_date": null,
	95	+ "group_id": null,
	96	+ "page_id": "kctbh9vrtdwd",
	97	+ "group": false,
	98	+ "only_show_if_degraded": false
	99	+ },
	100	+ {
	101	+ "id": "br0l2tvcx85d",
	102	+ "name": "Actions",
	103	+ "status": "operational",
	104	+ "created_at": "2019-11-13T18:02:19.432Z",
	105	+ "updated_at": "2026-08-27T00:26:05.957Z",
	106	+ "position": 7,
	107	+ "description": "Workflows, Compute and Orchestration for GitHub Actions",
	108	+ "showcase": true,
	109	+ "start_date": null,
	110	+ "group_id": null,
	111	+ "page_id": "kctbh9vrtdwd",
	112	+ "group": false,
	113	+ "only_show_if_degraded": false
	114	+ },
	115	+ {
	116	+ "id": "st3j38cctv9l",
	117	+ "name": "Packages",
	118	+ "status": "operational",
	119	+ "created_at": "2019-11-13T18:02:40.064Z",
	120	+ "updated_at": "2026-08-13T15:33:02.208Z",
	121	+ "position": 8,
	122	+ "description": "API requests and webhook delivery for GitHub Packages",
	123	+ "showcase": true,
	124	+ "start_date": null,
	125	+ "group_id": null,
	126	+ "page_id": "kctbh9vrtdwd",
	127	+ "group": false,
	128	+ "only_show_if_degraded": false
	129	+ },
	130	+ {
	131	+ "id": "vg70hn9s2tyj",
	132	+ "name": "Pages",
	133	+ "status": "operational",
	134	+ "created_at": "2017-01-31T20:04:33.923Z",
	135	+ "updated_at": "2026-08-26T16:49:07.311Z",
	136	+ "position": 9,
	137	+ "description": "Frontend application and API servers for Pages builds",
	138	+ "showcase": true,
	139	+ "start_date": null,
	140	+ "group_id": null,
	141	+ "page_id": "kctbh9vrtdwd",
	142	+ "group": false,
	143	+ "only_show_if_degraded": false
	144	+ },
	145	+ {
	146	+ "id": "pjmpxvq2cmr2",
	147	+ "name": "Copilot",
	148	+ "status": "operational",
	149	+ "created_at": "2022-06-21T16:04:33.017Z",
	150	+ "updated_at": "2026-08-17T21:15:46.595Z",
	151	+ "position": 10,
	152	+ "description": null,
	153	+ "showcase": true,
	154	+ "start_date": "2022-06-21",
	155	+ "group_id": null,
	156	+ "page_id": "kctbh9vrtdwd",
	157	+ "group": false,
	158	+ "only_show_if_degraded": false
	159	+ },
	160	+ {
	161	+ "id": "h2ftsgbw7kmk",
	162	+ "name": "Codespaces",
	163	+ "status": "operational",
	164	+ "created_at": "2021-08-11T16:02:09.505Z",
	165	+ "updated_at": "2026-07-14T09:51:08.862Z",
	166	+ "position": 11,
	167	+ "description": "Orchestration and Compute for GitHub Codespaces",
	168	+ "showcase": true,
	169	+ "start_date": "2021-08-11",
	170	+ "group_id": null,
	171	+ "page_id": "kctbh9vrtdwd",
	172	+ "group": false,
	173	+ "only_show_if_degraded": false
	174	+ },
	175	+ {
	176	+ "id": "cnnb39dkkk82",
	177	+ "name": "Copilot AI Model Providers",
	178	+ "status": "operational",
	179	+ "created_at": "2026-04-17T12:33:08.827Z",
	180	+ "updated_at": "2026-09-03T17:11:47.826Z",
	181	+ "position": 12,
	182	+ "description": null,
	183	+ "showcase": true,
	184	+ "start_date": "2026-04-17",
	185	+ "group_id": null,
	186	+ "page_id": "kctbh9vrtdwd",
	187	+ "group": false,
	188	+ "only_show_if_degraded": false
	189	+ }
	190	+ ],
	191	+ "incidents": [],
	192	+ "scheduled_maintenances": [],
	193	+ "status": {
	194	+ "indicator": "none",
	195	+ "description": "All Systems Operational"
	196	+ }
	197	+}
		\ No newline at end of file

added fixtures/connectors/teamtailor/teamtailor_jobs_feed.json +387 −0

		`@@ -0,0 +1,387 @@`
	1	+{
	2	+ "version": "https://jsonfeed.org/version/1.1",
	3	+ "title": "Teamtailor",
	4	+ "home_page_url": "https://career.teamtailor.com/jobs",
	5	+ "feed_url": "https://career.teamtailor.com/jobs.json",
	6	+ "items": [
	7	+ {
	8	+ "id": "a79a10f6-9f69-49b0-9f6d-8bed3d6e3a4f",
	9	+ "title": "Account Executive - UK Enterprise",
	10	+ "url": "https://career.teamtailor.com/jobs/8021334-account-executive-uk-enterprise",
	11	+ "date_published": "2026-07-06T08:58:57+02:00",
	12	+ "_jobposting": {
	13	+ "@context": "http://schema.org/",
	14	+ "@type": "JobPosting",
	15	+ "title": "Account Executive - UK Enterprise",
	16	+ "identifier": {
	17	+ "@type": "PropertyValue",
	18	+ "name": "Teamtailor",
	19	+ "value": 8021334
	20	+ },
	21	+ "datePosted": "2026-07-06T08:58:57+02:00",
	22	+ "hiringOrganization": {
	23	+ "@type": "Organization",
	24	+ "name": "Teamtailor",
	25	+ "sameAs": "https://career.teamtailor.com"
	26	+ },
	27	+ "jobLocation": [
	28	+ {
	29	+ "@type": "Place",
	30	+ "address": {
	31	+ "@type": "PostalAddress",
	32	+ "streetAddress": "16 Laystall Court",
	33	+ "addressLocality": "London",
	34	+ "postalCode": "EC1R 4",
	35	+ "addressCountry": "GB",
	36	+ "addressRegion": "Europe"
	37	+ }
	38	+ }
	39	+ ]
	40	+ }
	41	+ },
	42	+ {
	43	+ "id": "fb63bcb7-fcdf-4f35-ba55-b800505f8294",
	44	+ "title": "Mid-Market Account Executive (Outbound) - Madrid",
	45	+ "url": "https://career.teamtailor.com/jobs/7623275-mid-market-account-executive-outbound-madrid",
	46	+ "date_published": "2026-04-23T12:41:59+02:00",
	47	+ "_jobposting": {
	48	+ "@context": "http://schema.org/",
	49	+ "@type": "JobPosting",
	50	+ "title": "Mid-Market Account Executive (Outbound) - Madrid",
	51	+ "identifier": {
	52	+ "@type": "PropertyValue",
	53	+ "name": "Teamtailor",
	54	+ "value": 7623275
	55	+ },
	56	+ "datePosted": "2026-04-23T12:41:59+02:00",
	57	+ "hiringOrganization": {
	58	+ "@type": "Organization",
	59	+ "name": "Teamtailor",
	60	+ "sameAs": "https://career.teamtailor.com"
	61	+ },
	62	+ "jobLocation": [
	63	+ {
	64	+ "@type": "Place",
	65	+ "address": {
	66	+ "@type": "PostalAddress",
	67	+ "streetAddress": "Monday Chamberi",
	68	+ "addressLocality": "Madrid",
	69	+ "postalCode": "28010",
	70	+ "addressCountry": "ES",
	71	+ "addressRegion": "Europe"
	72	+ }
	73	+ }
	74	+ ]
	75	+ }
	76	+ },
	77	+ {
	78	+ "id": "07682917-d569-443b-b099-955f998f1591",
	79	+ "title": "APAC Customer Support Agent",
	80	+ "url": "https://career.teamtailor.com/jobs/8355188-apac-customer-support-agent",
	81	+ "date_published": "2026-09-09T18:08:56+02:00",
	82	+ "_jobposting": {
	83	+ "@context": "http://schema.org/",
	84	+ "@type": "JobPosting",
	85	+ "title": "APAC Customer Support Agent",
	86	+ "identifier": {
	87	+ "@type": "PropertyValue",
	88	+ "name": "Teamtailor",
	89	+ "value": 8355188
	90	+ },
	91	+ "datePosted": "2026-09-09T18:08:56+02:00",
	92	+ "hiringOrganization": {
	93	+ "@type": "Organization",
	94	+ "name": "Teamtailor",
	95	+ "sameAs": "https://career.teamtailor.com"
	96	+ },
	97	+ "baseSalary": {
	98	+ "@type": "MonetaryAmount",
	99	+ "currency": "AUD",
	100	+ "value": {
	101	+ "@type": "QuantitativeValue",
	102	+ "unitText": "MONTH",
	103	+ "minValue": "5100",
	104	+ "maxValue": "6500"
	105	+ }
	106	+ },
	107	+ "jobLocation": [
	108	+ {
	109	+ "@type": "Place",
	110	+ "address": {
	111	+ "@type": "PostalAddress",
	112	+ "streetAddress": "WeWork - Office Space & Coworking",
	113	+ "addressLocality": "Sydney",
	114	+ "postalCode": "2000",
	115	+ "addressCountry": "AU",
	116	+ "addressRegion": "Oceania"
	117	+ }
	118	+ }
	119	+ ]
	120	+ }
	121	+ },
	122	+ {
	123	+ "id": "11ef5728-a67d-4543-8b6c-75b2a7f6cd2d",
	124	+ "title": "Regional Director CSM",
	125	+ "url": "https://career.teamtailor.com/jobs/8297677-regional-director-csm",
	126	+ "date_published": "2026-09-01T12:01:07+02:00",
	127	+ "_jobposting": {
	128	+ "@context": "http://schema.org/",
	129	+ "@type": "JobPosting",
	130	+ "title": "Regional Director CSM",
	131	+ "identifier": {
	132	+ "@type": "PropertyValue",
	133	+ "name": "Teamtailor",
	134	+ "value": 8297677
	135	+ },
	136	+ "datePosted": "2026-09-01T12:01:07+02:00",
	137	+ "hiringOrganization": {
	138	+ "@type": "Organization",
	139	+ "name": "Teamtailor",
	140	+ "sameAs": "https://career.teamtailor.com"
	141	+ },
	142	+ "jobLocation": [
	143	+ {
	144	+ "@type": "Place",
	145	+ "address": {
	146	+ "@type": "PostalAddress",
	147	+ "streetAddress": "Östgötagatan 16",
	148	+ "addressLocality": "Stockholm",
	149	+ "postalCode": "116 21",
	150	+ "addressCountry": "SE",
	151	+ "addressRegion": "Sweden"
	152	+ }
	153	+ }
	154	+ ]
	155	+ }
	156	+ },
	157	+ {
	158	+ "id": "07f9a166-32f4-45f9-b0dd-8d52e0b31073",
	159	+ "title": "Senior Partnerships Manager - Toronto - Bilingual (French/ English)",
	160	+ "url": "https://career.teamtailor.com/jobs/8280874-senior-partnerships-manager-toronto-bilingual-french-english",
	161	+ "date_published": "2026-08-27T15:51:17+02:00",
	162	+ "_jobposting": {
	163	+ "@context": "http://schema.org/",
	164	+ "@type": "JobPosting",
	165	+ "title": "Senior Partnerships Manager - Toronto - Bilingual (French/ English)",
	166	+ "identifier": {
	167	+ "@type": "PropertyValue",
	168	+ "name": "Teamtailor",
	169	+ "value": 8280874
	170	+ },
	171	+ "datePosted": "2026-08-27T15:51:17+02:00",
	172	+ "hiringOrganization": {
	173	+ "@type": "Organization",
	174	+ "name": "Teamtailor",
	175	+ "sameAs": "https://career.teamtailor.com"
	176	+ },
	177	+ "jobLocation": [
	178	+ {
	179	+ "@type": "Place",
	180	+ "address": {
	181	+ "@type": "PostalAddress",
	182	+ "streetAddress": "WeWork Office Space & Coworking",
	183	+ "addressLocality": "Toronto",
	184	+ "postalCode": "M5H 4A6",
	185	+ "addressCountry": "CA",
	186	+ "addressRegion": "North America"
	187	+ }
	188	+ }
	189	+ ]
	190	+ }
	191	+ },
	192	+ {
	193	+ "id": "25b3f051-f5fd-4e43-b086-a31f49d8e1e5",
	194	+ "title": "Account Executive - Aboard by Teamtailor",
	195	+ "url": "https://career.teamtailor.com/jobs/8236014-account-executive-aboard-by-teamtailor",
	196	+ "date_published": "2026-08-18T15:02:43+02:00",
	197	+ "_jobposting": {
	198	+ "@context": "http://schema.org/",
	199	+ "@type": "JobPosting",
	200	+ "title": "Account Executive - Aboard by Teamtailor",
	201	+ "identifier": {
	202	+ "@type": "PropertyValue",
	203	+ "name": "Teamtailor",
	204	+ "value": 8236014
	205	+ },
	206	+ "datePosted": "2026-08-18T15:02:43+02:00",
	207	+ "hiringOrganization": {
	208	+ "@type": "Organization",
	209	+ "name": "Teamtailor",
	210	+ "sameAs": "https://career.teamtailor.com"
	211	+ },
	212	+ "jobLocation": [
	213	+ {
	214	+ "@type": "Place",
	215	+ "address": {
	216	+ "@type": "PostalAddress",
	217	+ "streetAddress": "Östgötagatan 16",
	218	+ "addressLocality": "Stockholm",
	219	+ "postalCode": "116 21",
	220	+ "addressCountry": "SE",
	221	+ "addressRegion": "Sweden"
	222	+ }
	223	+ }
	224	+ ]
	225	+ }
	226	+ },
	227	+ {
	228	+ "id": "3f282139-7ca5-47da-ba92-52486118f92d",
	229	+ "title": "Senior Account Executive - Chicago",
	230	+ "url": "https://career.teamtailor.com/jobs/8110423-senior-account-executive-chicago",
	231	+ "date_published": "2026-07-22T17:41:21+02:00",
	232	+ "_jobposting": {
	233	+ "@context": "http://schema.org/",
	234	+ "@type": "JobPosting",
	235	+ "title": "Senior Account Executive - Chicago",
	236	+ "identifier": {
	237	+ "@type": "PropertyValue",
	238	+ "name": "Teamtailor",
	239	+ "value": 8110423
	240	+ },
	241	+ "datePosted": "2026-07-22T17:41:21+02:00",
	242	+ "hiringOrganization": {
	243	+ "@type": "Organization",
	244	+ "name": "Teamtailor",
	245	+ "sameAs": "https://career.teamtailor.com"
	246	+ },
	247	+ "jobLocation": [
	248	+ {
	249	+ "@type": "Place",
	250	+ "address": {
	251	+ "@type": "PostalAddress",
	252	+ "streetAddress": "125 S Clark St",
	253	+ "addressLocality": "Chicago",
	254	+ "postalCode": "60603",
	255	+ "addressCountry": "US",
	256	+ "addressRegion": "North America"
	257	+ }
	258	+ }
	259	+ ]
	260	+ }
	261	+ },
	262	+ {
	263	+ "id": "f40addc7-d9ec-4535-bf88-3e12a7c96dbd",
	264	+ "title": "Customer Support Agent - Dutch/English Speaking",
	265	+ "url": "https://career.teamtailor.com/jobs/7951948-customer-support-agent-dutch-english-speaking",
	266	+ "date_published": "2026-06-22T16:45:21+02:00",
	267	+ "_jobposting": {
	268	+ "@context": "http://schema.org/",
	269	+ "@type": "JobPosting",
	270	+ "title": "Customer Support Agent - Dutch/English Speaking",
	271	+ "identifier": {
	272	+ "@type": "PropertyValue",
	273	+ "name": "Teamtailor",
	274	+ "value": 7951948
	275	+ },
	276	+ "datePosted": "2026-06-22T16:45:21+02:00",
	277	+ "hiringOrganization": {
	278	+ "@type": "Organization",
	279	+ "name": "Teamtailor",
	280	+ "sameAs": "https://career.teamtailor.com"
	281	+ },
	282	+ "baseSalary": {
	283	+ "@type": "MonetaryAmount",
	284	+ "currency": "SEK",
	285	+ "value": {
	286	+ "@type": "QuantitativeValue",
	287	+ "unitText": "MONTH",
	288	+ "value": "28000"
	289	+ }
	290	+ },
	291	+ "jobLocation": [
	292	+ {
	293	+ "@type": "Place",
	294	+ "address": {
	295	+ "@type": "PostalAddress",
	296	+ "streetAddress": "Östgötagatan 16",
	297	+ "addressLocality": "Stockholm",
	298	+ "postalCode": "116 21",
	299	+ "addressCountry": "SE",
	300	+ "addressRegion": "Sweden"
	301	+ }
	302	+ }
	303	+ ]
	304	+ }
	305	+ },
	306	+ {
	307	+ "id": "6057c3f8-17ac-43b2-9ebe-47caed0330e9",
	308	+ "title": "Mid-Market Account Executive - Chicago",
	309	+ "url": "https://career.teamtailor.com/jobs/7538468-mid-market-account-executive-chicago",
	310	+ "date_published": "2026-04-09T15:44:01+02:00",
	311	+ "_jobposting": {
	312	+ "@context": "http://schema.org/",
	313	+ "@type": "JobPosting",
	314	+ "title": "Mid-Market Account Executive - Chicago",
	315	+ "identifier": {
	316	+ "@type": "PropertyValue",
	317	+ "name": "Teamtailor",
	318	+ "value": 7538468
	319	+ },
	320	+ "datePosted": "2026-04-09T15:44:01+02:00",
	321	+ "hiringOrganization": {
	322	+ "@type": "Organization",
	323	+ "name": "Teamtailor",
	324	+ "sameAs": "https://career.teamtailor.com"
	325	+ },
	326	+ "baseSalary": {
	327	+ "@type": "MonetaryAmount",
	328	+ "currency": "USD",
	329	+ "value": {
	330	+ "@type": "QuantitativeValue",
	331	+ "unitText": "YEAR",
	332	+ "minValue": "127000",
	333	+ "maxValue": "137000"
	334	+ }
	335	+ },
	336	+ "jobLocation": [
	337	+ {
	338	+ "@type": "Place",
	339	+ "address": {
	340	+ "@type": "PostalAddress",
	341	+ "streetAddress": "125 S Clark St",
	342	+ "addressLocality": "Chicago",
	343	+ "postalCode": "60603",
	344	+ "addressCountry": "US",
	345	+ "addressRegion": "North America"
	346	+ }
	347	+ }
	348	+ ]
	349	+ }
	350	+ },
	351	+ {
	352	+ "id": "96b23e80-8b71-47cb-bf70-2e2680b91ce8",
	353	+ "title": "Enterprise Account Executive - DACH (d/f/m)",
	354	+ "url": "https://career.teamtailor.com/jobs/7309094-enterprise-account-executive-dach-d-f-m",
	355	+ "date_published": "2026-03-01T20:50:16+01:00",
	356	+ "_jobposting": {
	357	+ "@context": "http://schema.org/",
	358	+ "@type": "JobPosting",
	359	+ "title": "Enterprise Account Executive - DACH (d/f/m)",
	360	+ "identifier": {
	361	+ "@type": "PropertyValue",
	362	+ "name": "Teamtailor",
	363	+ "value": 7309094
	364	+ },
	365	+ "datePosted": "2026-03-01T20:50:16+01:00",
	366	+ "hiringOrganization": {
	367	+ "@type": "Organization",
	368	+ "name": "Teamtailor",
	369	+ "sameAs": "https://career.teamtailor.com"
	370	+ },
	371	+ "jobLocation": [
	372	+ {
	373	+ "@type": "Place",
	374	+ "address": {
	375	+ "@type": "PostalAddress",
	376	+ "streetAddress": null,
	377	+ "addressLocality": "Berlin",
	378	+ "postalCode": null,
	379	+ "addressCountry": "DE",
	380	+ "addressRegion": "Europe"
	381	+ }
	382	+ }
	383	+ ]
	384	+ }
	385	+ }
	386	+ ]
	387	+}
		\ No newline at end of file

added fixtures/connectors/workable/epignosis_widget.json +238 −0

		`@@ -0,0 +1,238 @@`
	1	+{
	2	+ "name": "Epignosis",
	3	+ "description": "(trimmed)",
	4	+ "jobs": [
	5	+ {
	6	+ "title": "Data Engineer",
	7	+ "shortcode": "E38DB16625",
	8	+ "code": "",
	9	+ "employment_type": "Full-time",
	10	+ "telecommuting": false,
	11	+ "department": "Engineering",
	12	+ "url": "https://apply.workable.com/j/E38DB16625",
	13	+ "shortlink": "https://apply.workable.com/j/E38DB16625",
	14	+ "application_url": "https://apply.workable.com/j/E38DB16625/apply",
	15	+ "published_on": "2026-09-07",
	16	+ "created_at": "2026-09-07",
	17	+ "country": "Greece",
	18	+ "city": "Athens",
	19	+ "state": "Attica",
	20	+ "education": "Bachelor's Degree",
	21	+ "experience": "Mid-Senior level",
	22	+ "function": "Engineering",
	23	+ "industry": "Computer Software",
	24	+ "locations": [
	25	+ {
	26	+ "country": "Greece",
	27	+ "countryCode": "GR",
	28	+ "city": "Athens",
	29	+ "region": "Attica",
	30	+ "hidden": false
	31	+ }
	32	+ ]
	33	+ },
	34	+ {
	35	+ "title": "Partner Marketing Lead",
	36	+ "shortcode": "F53AEDB61C",
	37	+ "code": "",
	38	+ "employment_type": "Full-time",
	39	+ "telecommuting": false,
	40	+ "department": "Marketing",
	41	+ "url": "https://apply.workable.com/j/F53AEDB61C",
	42	+ "shortlink": "https://apply.workable.com/j/F53AEDB61C",
	43	+ "application_url": "https://apply.workable.com/j/F53AEDB61C/apply",
	44	+ "published_on": "2026-09-11",
	45	+ "created_at": "2026-09-11",
	46	+ "country": "Greece",
	47	+ "city": "Athens",
	48	+ "state": "Attica",
	49	+ "education": "",
	50	+ "experience": "Mid-Senior level",
	51	+ "function": "Marketing",
	52	+ "industry": "Computer Software",
	53	+ "locations": [
	54	+ {
	55	+ "country": "Greece",
	56	+ "countryCode": "GR",
	57	+ "city": "Athens",
	58	+ "region": "Attica",
	59	+ "hidden": false
	60	+ }
	61	+ ]
	62	+ },
	63	+ {
	64	+ "title": "Sales Development Representative (Future Opportunities)",
	65	+ "shortcode": "D26AEB4351",
	66	+ "code": "",
	67	+ "employment_type": "Full-time",
	68	+ "telecommuting": false,
	69	+ "department": "Sales",
	70	+ "url": "https://apply.workable.com/j/D26AEB4351",
	71	+ "shortlink": "https://apply.workable.com/j/D26AEB4351",
	72	+ "application_url": "https://apply.workable.com/j/D26AEB4351/apply",
	73	+ "published_on": "2026-06-17",
	74	+ "created_at": "2026-04-21",
	75	+ "country": "Greece",
	76	+ "city": "Athens",
	77	+ "state": "Attica",
	78	+ "education": "Bachelor's Degree",
	79	+ "experience": "Entry level",
	80	+ "function": "Sales",
	81	+ "industry": "Computer Software",
	82	+ "locations": [
	83	+ {
	84	+ "country": "Greece",
	85	+ "countryCode": "GR",
	86	+ "city": "Athens",
	87	+ "region": "Attica",
	88	+ "hidden": false
	89	+ }
	90	+ ]
	91	+ },
	92	+ {
	93	+ "title": "Sales Development Representative (SDR) - Outbound",
	94	+ "shortcode": "5D5D9CEDF7",
	95	+ "code": "",
	96	+ "employment_type": "Full-time",
	97	+ "telecommuting": true,
	98	+ "department": "Sales",
	99	+ "url": "https://apply.workable.com/j/5D5D9CEDF7",
	100	+ "shortlink": "https://apply.workable.com/j/5D5D9CEDF7",
	101	+ "application_url": "https://apply.workable.com/j/5D5D9CEDF7/apply",
	102	+ "published_on": "2026-08-28",
	103	+ "created_at": "2026-08-28",
	104	+ "country": "United Kingdom",
	105	+ "city": "London",
	106	+ "state": "England",
	107	+ "education": "",
	108	+ "experience": "Associate",
	109	+ "function": "Sales",
	110	+ "industry": "Computer Software",
	111	+ "locations": [
	112	+ {
	113	+ "country": "United Kingdom",
	114	+ "countryCode": "GB",
	115	+ "city": "London",
	116	+ "region": "England",
	117	+ "hidden": false
	118	+ }
	119	+ ]
	120	+ },
	121	+ {
	122	+ "title": "Sales Development Representative (SDR) - Outbound (Atlanta or Florida based)",
	123	+ "shortcode": "FC2EB548ED",
	124	+ "code": "",
	125	+ "employment_type": "Full-time",
	126	+ "telecommuting": false,
	127	+ "department": "Sales",
	128	+ "url": "https://apply.workable.com/j/FC2EB548ED",
	129	+ "shortlink": "https://apply.workable.com/j/FC2EB548ED",
	130	+ "application_url": "https://apply.workable.com/j/FC2EB548ED/apply",
	131	+ "published_on": "2026-08-28",
	132	+ "created_at": "2025-09-11",
	133	+ "country": "United States",
	134	+ "city": "Atlanta",
	135	+ "state": "Georgia",
	136	+ "education": "",
	137	+ "experience": "Entry level",
	138	+ "function": "Sales",
	139	+ "industry": "Computer Software",
	140	+ "locations": [
	141	+ {
	142	+ "country": "United States",
	143	+ "countryCode": "US",
	144	+ "city": "Atlanta",
	145	+ "region": "Georgia",
	146	+ "hidden": false
	147	+ }
	148	+ ]
	149	+ },
	150	+ {
	151	+ "title": "Sales Development Representative (SDR) - Outbound (Atlanta or Florida based)",
	152	+ "shortcode": "FC2EB548ED",
	153	+ "code": "",
	154	+ "employment_type": "Full-time",
	155	+ "telecommuting": false,
	156	+ "department": "Sales",
	157	+ "url": "https://apply.workable.com/j/FC2EB548ED",
	158	+ "shortlink": "https://apply.workable.com/j/FC2EB548ED",
	159	+ "application_url": "https://apply.workable.com/j/FC2EB548ED/apply",
	160	+ "published_on": "2026-08-28",
	161	+ "created_at": "2025-09-11",
	162	+ "country": "United States",
	163	+ "city": "Florida City",
	164	+ "state": "Florida",
	165	+ "education": "",
	166	+ "experience": "Entry level",
	167	+ "function": "Sales",
	168	+ "industry": "Computer Software",
	169	+ "locations": [
	170	+ {
	171	+ "country": "United States",
	172	+ "countryCode": "US",
	173	+ "city": "Florida City",
	174	+ "region": "Florida",
	175	+ "hidden": false
	176	+ }
	177	+ ]
	178	+ },
	179	+ {
	180	+ "title": "Senior Cloud Engineer",
	181	+ "shortcode": "9ED99A8425",
	182	+ "code": "",
	183	+ "employment_type": "Full-time",
	184	+ "telecommuting": false,
	185	+ "department": "Engineering",
	186	+ "url": "https://apply.workable.com/j/9ED99A8425",
	187	+ "shortlink": "https://apply.workable.com/j/9ED99A8425",
	188	+ "application_url": "https://apply.workable.com/j/9ED99A8425/apply",
	189	+ "published_on": "2026-09-10",
	190	+ "created_at": "2026-09-10",
	191	+ "country": "Greece",
	192	+ "city": "Athens",
	193	+ "state": "Attica",
	194	+ "education": "",
	195	+ "experience": "Mid-Senior level",
	196	+ "function": "Engineering",
	197	+ "industry": "Computer Software",
	198	+ "locations": [
	199	+ {
	200	+ "country": "Greece",
	201	+ "countryCode": "GR",
	202	+ "city": "Athens",
	203	+ "region": "Attica",
	204	+ "hidden": false
	205	+ }
	206	+ ]
	207	+ },
	208	+ {
	209	+ "title": "Senior QA Automation Engineer",
	210	+ "shortcode": "76991535C6",
	211	+ "code": "",
	212	+ "employment_type": "Full-time",
	213	+ "telecommuting": false,
	214	+ "department": "Engineering",
	215	+ "url": "https://apply.workable.com/j/76991535C6",
	216	+ "shortlink": "https://apply.workable.com/j/76991535C6",
	217	+ "application_url": "https://apply.workable.com/j/76991535C6/apply",
	218	+ "published_on": "2026-09-07",
	219	+ "created_at": "2026-08-28",
	220	+ "country": "Greece",
	221	+ "city": "Athens",
	222	+ "state": "Attica",
	223	+ "education": "",
	224	+ "experience": "",
	225	+ "function": "Engineering",
	226	+ "industry": "Computer Software",
	227	+ "locations": [
	228	+ {
	229	+ "country": "Greece",
	230	+ "countryCode": "GR",
	231	+ "city": "Athens",
	232	+ "region": "Attica",
	233	+ "hidden": false
	234	+ }
	235	+ ]
	236	+ }
	237	+ ]
	238	+}
		\ No newline at end of file

added fixtures/connectors/workday/nvidia_jobs_page1.json +1164 −0

		`@@ -0,0 +1,1164 @@`
	1	+{
	2	+ "total": 2000,
	3	+ "jobPostings": [
	4	+ {
	5	+ "title": "Senior Embedded Software Engineer, DPU - Networking",
	6	+ "externalPath": "/job/US-MA-Westford/Senior-Software-Engineer--DPU---Networking_JR2017846",
	7	+ "locationsText": "US, MA, Westford",
	8	+ "postedOn": "Posted Yesterday",
	9	+ "bulletFields": [
	10	+ "JR2017846"
	11	+ ]
	12	+ },
	13	+ {
	14	+ "title": "Senior Software Engineer - DGX Cloud Production Engineering",
	15	+ "externalPath": "/job/US-CA-Remote/Senior-Production-Engineer---DGX-Cloud_JR2018101",
	16	+ "locationsText": "6 Locations",
	17	+ "postedOn": "Posted Yesterday",
	18	+ "bulletFields": [
	19	+ "JR2018101"
	20	+ ]
	21	+ },
	22	+ {
	23	+ "title": "Applied AI Engineer",
	24	+ "externalPath": "/job/US-CA-Remote/Applied-AI-Engineer_JR2018178-3",
	25	+ "locationsText": "4 Locations",
	26	+ "postedOn": "Posted Yesterday",
	27	+ "bulletFields": [
	28	+ "JR2018178"
	29	+ ]
	30	+ },
	31	+ {
	32	+ "title": "Principal Partner Enablement Product Manager, AI Infrastructure",
	33	+ "externalPath": "/job/US-CA-Santa-Clara/Principal-Partner-Enablement-Product-Manager--AI-Infrastructure_JR2017274",
	34	+ "locationsText": "US, CA, Santa Clara",
	35	+ "postedOn": "Posted Yesterday",
	36	+ "bulletFields": [
	37	+ "JR2017274"
	38	+ ]
	39	+ },
	40	+ {
	41	+ "title": "ODM Business Operations Manager",
	42	+ "externalPath": "/job/US-CA-Santa-Clara/ODM-Business-Operations-Manager_JR2006561",
	43	+ "locationsText": "US, CA, Santa Clara",
	44	+ "postedOn": "Posted Yesterday",
	45	+ "bulletFields": [
	46	+ "JR2006561"
	47	+ ]
	48	+ },
	49	+ {
	50	+ "title": "Executive Assistant",
	51	+ "externalPath": "/job/US-CA-Santa-Clara/Executive-Assistant_JR2022902",
	52	+ "locationsText": "US, CA, Santa Clara",
	53	+ "postedOn": "Posted Yesterday",
	54	+ "bulletFields": [
	55	+ "JR2022902"
	56	+ ]
	57	+ },
	58	+ {
	59	+ "title": "Senior Solutions Architect, Networking Solutions",
	60	+ "externalPath": "/job/Netherlands-Remote/Senior-Solutions-Architect--Networking-Solutions_JR2025043-1",
	61	+ "locationsText": "Netherlands, Remote",
	62	+ "postedOn": "Posted Yesterday",
	63	+ "bulletFields": [
	64	+ "JR2025043"
	65	+ ]
	66	+ },
	67	+ {
	68	+ "title": "Applied AI Engineer",
	69	+ "externalPath": "/job/US-CA-Remote/Applied-AI-Engineer_JR2018181-1",
	70	+ "locationsText": "6 Locations",
	71	+ "postedOn": "Posted Yesterday",
	72	+ "bulletFields": [
	73	+ "JR2018181"
	74	+ ]
	75	+ },
	76	+ {
	77	+ "title": "Senior Developer Relations Manager - DACH",
	78	+ "externalPath": "/job/Germany-Munich/Senior-Developer-Relations-Manager---DACH_JR2025099",
	79	+ "locationsText": "4 Locations",
	80	+ "postedOn": "Posted Yesterday",
	81	+ "bulletFields": [
	82	+ "JR2025099"
	83	+ ]
	84	+ },
	85	+ {
	86	+ "title": "Manager, Firmware",
	87	+ "externalPath": "/job/US-CA-Santa-Clara/Manager--Firmware_JR2019791",
	88	+ "locationsText": "US, CA, Santa Clara",
	89	+ "postedOn": "Posted Yesterday",
	90	+ "bulletFields": [
	91	+ "JR2019791"
	92	+ ]
	93	+ },
	94	+ {
	95	+ "title": "Manager, Third Party Events - Europe",
	96	+ "externalPath": "/job/Germany-Munich/Manager--Third-Party-Events---Europe_JR2024424",
	97	+ "locationsText": "4 Locations",
	98	+ "postedOn": "Posted Yesterday",
	99	+ "bulletFields": [
	100	+ "JR2024424"
	101	+ ]
	102	+ },
	103	+ {
	104	+ "title": "Senior Developer Relations Manager - France",
	105	+ "externalPath": "/job/France-Remote/Senior-Developer-Relations-Manager---France_JR2025100",
	106	+ "locationsText": "2 Locations",
	107	+ "postedOn": "Posted Yesterday",
	108	+ "bulletFields": [
	109	+ "JR2025100"
	110	+ ]
	111	+ },
	112	+ {
	113	+ "title": "Senior Technical Support Engineer – Slurm",
	114	+ "externalPath": "/job/US-TX-Austin/Senior-Technical-Support-Engineer---Slurm_JR2025573",
	115	+ "locationsText": "2 Locations",
	116	+ "postedOn": "Posted Yesterday",
	117	+ "bulletFields": [
	118	+ "JR2025573"
	119	+ ]
	120	+ },
	121	+ {
	122	+ "title": "Strategic Account Manager, CSP",
	123	+ "externalPath": "/job/US-CA-Santa-Clara/Strategic-Account-Manager--CSP---Networking_JR2020916",
	124	+ "locationsText": "US, CA, Santa Clara",
	125	+ "postedOn": "Posted Yesterday",
	126	+ "bulletFields": [
	127	+ "JR2020916"
	128	+ ]
	129	+ },
	130	+ {
	131	+ "title": "Media Relations Lead - France and Southern Europe",
	132	+ "externalPath": "/job/France-Remote/Media-Relations-Lead---France-and-Southern-Europe_JR2024939",
	133	+ "locationsText": "2 Locations",
	134	+ "postedOn": "Posted Yesterday",
	135	+ "bulletFields": [
	136	+ "JR2024939"
	137	+ ]
	138	+ },
	139	+ {
	140	+ "title": "Security Research Engineer, AI Safety and Security Engineering",
	141	+ "externalPath": "/job/US-CA-Santa-Clara/Security-Research-Engineer--AI-Safety-and-Security-Engineering_JR2021887",
	142	+ "locationsText": "6 Locations",
	143	+ "postedOn": "Posted Yesterday",
	144	+ "bulletFields": [
	145	+ "JR2021887"
	146	+ ]
	147	+ },
	148	+ {
	149	+ "title": "Enterprise AV Design and Collaboration Engineer",
	150	+ "externalPath": "/job/US-CA-Santa-Clara/Enterprise-AV-Design-and-Collaboration-Engineer_JR2025082",
	151	+ "locationsText": "US, CA, Santa Clara",
	152	+ "postedOn": "Posted Yesterday",
	153	+ "bulletFields": [
	154	+ "JR2025082"
	155	+ ]
	156	+ },
	157	+ {
	158	+ "title": "Senior Software Engineer, AI Agent Compute",
	159	+ "externalPath": "/job/US-CA-Remote/Senior-Software-Engineer--AI-Agent-Compute_JR2025063",
	160	+ "locationsText": "2 Locations",
	161	+ "postedOn": "Posted Yesterday",
	162	+ "bulletFields": [
	163	+ "JR2025063"
	164	+ ]
	165	+ },
	166	+ {
	167	+ "title": "Senior Staff Engineer - Employee Productivity",
	168	+ "externalPath": "/job/US-CA-Santa-Clara/Senior-Staff-Engineer---Employee-Productivity_JR2024564",
	169	+ "locationsText": "US, CA, Santa Clara",
	170	+ "postedOn": "Posted Yesterday",
	171	+ "bulletFields": [
	172	+ "JR2024564"
	173	+ ]
	174	+ },
	175	+ {
	176	+ "title": "Staff Unified Communications Engineer",
	177	+ "externalPath": "/job/US-CA-Santa-Clara/Staff-Unified-Communications-Engineer_JR2025118",
	178	+ "locationsText": "US, CA, Santa Clara",
	179	+ "postedOn": "Posted Yesterday",
	180	+ "bulletFields": [
	181	+ "JR2025118"
	182	+ ]
	183	+ }
	184	+ ],
	185	+ "facets": [
	186	+ {
	187	+ "facetParameter": "jobFamilyGroup",
	188	+ "descriptor": "Job Category",
	189	+ "values": [
	190	+ {
	191	+ "descriptor": "Engineering",
	192	+ "id": "0c40f6bd1d8f10ae43ffaefd46dc7e78",
	193	+ "count": 1746
	194	+ },
	195	+ {
	196	+ "descriptor": "Sales",
	197	+ "id": "0c40f6bd1d8f10ae43ffcac5bbec7e90",
	198	+ "count": 334
	199	+ },
	200	+ {
	201	+ "descriptor": "Operations",
	202	+ "id": "0c40f6bd1d8f10ae43ffc3fc7d8c7e8a",
	203	+ "count": 138
	204	+ },
	205	+ {
	206	+ "descriptor": "Program Manager",
	207	+ "id": "0c40f6bd1d8f10ae43ffc668c6847e8c",
	208	+ "count": 98
	209	+ },
	210	+ {
	211	+ "descriptor": "Marketing",
	212	+ "id": "0c40f6bd1d8f10ae43ffc19725ec7e88",
	213	+ "count": 91
	214	+ },
	215	+ {
	216	+ "descriptor": "Univ Employment",
	217	+ "id": "0c40f6bd1d8f10ae43ffda1e8d447e94",
	218	+ "count": 76
	219	+ },
	220	+ {
	221	+ "descriptor": "IT - Information Technology",
	222	+ "id": "0c40f6bd1d8f10ae43ffbd1459047e84",
	223	+ "count": 46
	224	+ },
	225	+ {
	226	+ "descriptor": "Research",
	227	+ "id": "0c40f6bd1d8f10ae43ffc8817cf47e8e",
	228	+ "count": 42
	229	+ },
	230	+ {
	231	+ "descriptor": "Professional Services",
	232	+ "id": "e8bdc341a93101bd5f4d2b0a1c005e36",
	233	+ "count": 38
	234	+ },
	235	+ {
	236	+ "descriptor": "Finance",
	237	+ "id": "0c40f6bd1d8f10ae43ffb5dd06f47e7e",
	238	+ "count": 22
	239	+ },
	240	+ {
	241	+ "descriptor": "Human Resources",
	242	+ "id": "0c40f6bd1d8f10ae43ffbac3680c7e82",
	243	+ "count": 8
	244	+ },
	245	+ {
	246	+ "descriptor": "Business Development",
	247	+ "id": "0c40f6bd1d8f10ae43ffac5fdfac7e76",
	248	+ "count": 8
	249	+ },
	250	+ {
	251	+ "descriptor": "Legal",
	252	+ "id": "0c40f6bd1d8f10ae43ffbf4412147e86",
	253	+ "count": 5
	254	+ },
	255	+ {
	256	+ "descriptor": "Facilities",
	257	+ "id": "0c40f6bd1d8f10ae43ffb3a6aaac7e7c",
	258	+ "count": 3
	259	+ },
	260	+ {
	261	+ "descriptor": "Administration",
	262	+ "id": "0c40f6bd1d8f10ae43ffaa1e1d6c7e74",
	263	+ "count": 2
	264	+ }
	265	+ ]
	266	+ },
	267	+ {
	268	+ "facetParameter": "workerSubType",
	269	+ "descriptor": "Job Type",
	270	+ "values": [
	271	+ {
	272	+ "descriptor": "Regular Employee",
	273	+ "id": "0c40f6bd1d8f10adf6dae161b1844a15",
	274	+ "count": 2320
	275	+ },
	276	+ {
	277	+ "descriptor": "Management",
	278	+ "id": "0c40f6bd1d8f10adf6dae2cd57444a16",
	279	+ "count": 196
	280	+ },
	281	+ {
	282	+ "descriptor": "Intern (Fixed Term)",
	283	+ "id": "0c40f6bd1d8f10adf6dae42e46d44a17",
	284	+ "count": 74
	285	+ },
	286	+ {
	287	+ "descriptor": "New College Graduate",
	288	+ "id": "ab40a98049581037a3ada55b087049b7",
	289	+ "count": 65
	290	+ },
	291	+ {
	292	+ "descriptor": "Academic (Fixed Term)",
	293	+ "id": "ec9560ee6dc71001d6f3cf17ea640000",
	294	+ "count": 1
	295	+ },
	296	+ {
	297	+ "descriptor": "Regular Employee (Fixed Term)",
	298	+ "id": "cc40d60d614c01809d56fe967144ce2c",
	299	+ "count": 1
	300	+ }
	301	+ ]
	302	+ },
	303	+ {
	304	+ "facetParameter": "timeType",
	305	+ "descriptor": "Time Type",
	306	+ "values": [
	307	+ {
	308	+ "descriptor": "Full time",
	309	+ "id": "5509c0b5959810ac0029943377d47364",
	310	+ "count": 2655
	311	+ },
	312	+ {
	313	+ "descriptor": "Part time",
	314	+ "id": "5509c0b5959810ac00299430e7947363",
	315	+ "count": 2
	316	+ }
	317	+ ]
	318	+ },
	319	+ {
	320	+ "facetParameter": "locationMainGroup",
	321	+ "values": [
	322	+ {
	323	+ "facetParameter": "locationHierarchy2",
	324	+ "descriptor": "Location Type",
	325	+ "values": [
	326	+ {
	327	+ "descriptor": "Office",
	328	+ "id": "0c3f5f117e9a0101f6422f0fe79d0000",
	329	+ "count": 2516
	330	+ },
	331	+ {
	332	+ "descriptor": "Remote",
	333	+ "id": "0c3f5f117e9a0101f63dc469c3010000",
	334	+ "count": 583
	335	+ }
	336	+ ]
	337	+ },
	338	+ {
	339	+ "facetParameter": "locationHierarchy1",
	340	+ "descriptor": "Locations",
	341	+ "values": [
	342	+ {
	343	+ "descriptor": "Armenia",
	344	+ "id": "d21cf68980ad0128000fd2c6b107c000",
	345	+ "count": 3
	346	+ },
	347	+ {
	348	+ "descriptor": "Australia",
	349	+ "id": "2fcb99c455831013ea528dce556b3224",
	350	+ "count": 12
	351	+ },
	352	+ {
	353	+ "descriptor": "Brazil",
	354	+ "id": "2fcb99c455831013ea5298c9e1f63230",
	355	+ "count": 2
	356	+ },
	357	+ {
	358	+ "descriptor": "Canada",
	359	+ "id": "2fcb99c455831013ea529c3b93ba3236",
	360	+ "count": 12
	361	+ },
	362	+ {
	363	+ "descriptor": "China",
	364	+ "id": "2fcb99c455831013ea529fe151e3323c",
	365	+ "count": 201
	366	+ },
	367	+ {
	368	+ "descriptor": "Czechia",
	369	+ "id": "2fcb99c455831013ea52a358b3d53242",
	370	+ "count": 3
	371	+ },
	372	+ {
	373	+ "descriptor": "Denmark",
	374	+ "id": "d21cf68980ad0121a67d319db107a200",
	375	+ "count": 7
	376	+ },
	377	+ {
	378	+ "descriptor": "Finland",
	379	+ "id": "2fcb99c455831013ea52a6cbc15f3248",
	380	+ "count": 5
	381	+ },
	382	+ {
	383	+ "descriptor": "France",
	384	+ "id": "2fcb99c455831013ea52aa2df70e324e",
	385	+ "count": 29
	386	+ },
	387	+ {
	388	+ "descriptor": "Germany",
	389	+ "id": "2fcb99c455831013ea52adc65f5d3254",
	390	+ "count": 53
	391	+ },
	392	+ {
	393	+ "descriptor": "Greece",
	394	+ "id": "c498fba66f4e01b7b7931c0fd400ab04",
	395	+ "count": 1
	396	+ },
	397	+ {
	398	+ "descriptor": "Hong Kong",
	399	+ "id": "2fcb99c455831013ea52b157d0d6325a",
	400	+ "count": 4
	401	+ },
	402	+ {
	403	+ "descriptor": "Hungary",
	404	+ "id": "7bcfe8379f6b01ad4d347d1ca87689a6",
	405	+ "count": 3
	406	+ },
	407	+ {
	408	+ "descriptor": "India",
	409	+ "id": "2fcb99c455831013ea52b82135ba3266",
	410	+ "count": 245
	411	+ },
	412	+ {
	413	+ "descriptor": "Israel",
	414	+ "id": "2fcb99c455831013ea52bbe14cf9326c",
	415	+ "count": 420
	416	+ },
	417	+ {
	418	+ "descriptor": "Italy",
	419	+ "id": "2fcb99c455831013ea52bf52796c3272",
	420	+ "count": 3
	421	+ },
	422	+ {
	423	+ "descriptor": "Japan",
	424	+ "id": "2fcb99c455831013ea52c29d2e923278",
	425	+ "count": 22
	426	+ },
	427	+ {
	428	+ "descriptor": "Korea",
	429	+ "id": "2fcb99c455831013ea52c61ded6e327e",
	430	+ "count": 19
	431	+ },
	432	+ {
	433	+ "descriptor": "Mexico",
	434	+ "id": "2fcb99c455831013ea52cd244532328a",
	435	+ "count": 3
	436	+ },
	437	+ {
	438	+ "descriptor": "Netherlands",
	439	+ "id": "2fcb99c455831013ea52d0e0e4583290",
	440	+ "count": 8
	441	+ },
	442	+ {
	443	+ "descriptor": "Palestine",
	444	+ "id": "970bf8c909a701f9c086d363d300fa06",
	445	+ "count": 3
	446	+ },
	447	+ {
	448	+ "descriptor": "Poland",
	449	+ "id": "2fcb99c455831013ea52d8783aa0329c",
	450	+ "count": 27
	451	+ },
	452	+ {
	453	+ "descriptor": "Romania",
	454	+ "id": "e5375aecc8d9103bebfb513cb69200d4",
	455	+ "count": 1
	456	+ },
	457	+ {
	458	+ "descriptor": "Singapore",
	459	+ "id": "2fcb99c455831013ea52df1adb7432a8",
	460	+ "count": 22
	461	+ },
	462	+ {
	463	+ "descriptor": "Spain",
	464	+ "id": "2fcb99c455831013ea52e31a43e832ae",
	465	+ "count": 14
	466	+ },
	467	+ {
	468	+ "descriptor": "Sweden",
	469	+ "id": "2fcb99c455831013ea52e6c3067832b4",
	470	+ "count": 7
	471	+ },
	472	+ {
	473	+ "descriptor": "Switzerland",
	474	+ "id": "2fcb99c455831013ea52e9ef1a0032ba",
	475	+ "count": 35
	476	+ },
	477	+ {
	478	+ "descriptor": "Taiwan",
	479	+ "id": "2fcb99c455831013ea52ed162d4932c0",
	480	+ "count": 118
	481	+ },
	482	+ {
	483	+ "descriptor": "Thailand",
	484	+ "id": "2fcb99c455831013ea52f07ea4c732c6",
	485	+ "count": 4
	486	+ },
	487	+ {
	488	+ "descriptor": "Ukraine",
	489	+ "id": "970bf8c909a7012c3733f063d3000107",
	490	+ "count": 6
	491	+ },
	492	+ {
	493	+ "descriptor": "United Arab Emirates",
	494	+ "id": "2fcb99c455831013ea52f3d5d2c232cc",
	495	+ "count": 3
	496	+ },
	497	+ {
	498	+ "descriptor": "United Kingdom",
	499	+ "id": "2fcb99c455831013ea52f785717432d2",
	500	+ "count": 49
	501	+ },
	502	+ {
	503	+ "descriptor": "United States",
	504	+ "id": "2fcb99c455831013ea52fb338f2932d8",
	505	+ "count": 1395
	506	+ },
	507	+ {
	508	+ "descriptor": "Vietnam",
	509	+ "id": "2fcb99c455831013ea52fea3f8d732de",
	510	+ "count": 36
	511	+ }
	512	+ ]
	513	+ },
	514	+ {
	515	+ "facetParameter": "locations",
	516	+ "descriptor": "Sites",
	517	+ "values": [
	518	+ {
	519	+ "descriptor": "Armenia, Remote",
	520	+ "id": "d21cf68980ad01f2f9f0b1bdb5075e07",
	521	+ "count": 2
	522	+ },
	523	+ {
	524	+ "descriptor": "Armenia, Yerevan",
	525	+ "id": "c0ed409f79231000eaff724467330000",
	526	+ "count": 2
	527	+ },
	528	+ {
	529	+ "descriptor": "Australia, Remote",
	530	+ "id": "91336993fab910af6d70362ab49cc2f7",
	531	+ "count": 12
	532	+ },
	533	+ {
	534	+ "descriptor": "Australia, Sydney",
	535	+ "id": "0d86a8e04eda011538f0ac5d90019611",
	536	+ "count": 1
	537	+ },
	538	+ {
	539	+ "descriptor": "Brazil, Remote",
	540	+ "id": "91336993fab910af6d7115f6c23cc357",
	541	+ "count": 1
	542	+ },
	543	+ {
	544	+ "descriptor": "Brazil, Sao Paulo",
	545	+ "id": "91336993fab910af6d7113d4a58cc352",
	546	+ "count": 2
	547	+ },
	548	+ {
	549	+ "descriptor": "Canada, Remote",
	550	+ "id": "91336993fab910af6d7111b9ef64c34d",
	551	+ "count": 7
	552	+ },
	553	+ {
	554	+ "descriptor": "Canada, Toronto",
	555	+ "id": "4205b8279d1110811b5e0a9820d6693a",
	556	+ "count": 7
	557	+ },
	558	+ {
	559	+ "descriptor": "China, Beijing",
	560	+ "id": "91336993fab910af6d710912773cc339",
	561	+ "count": 81
	562	+ },
	563	+ {
	564	+ "descriptor": "China, Remote",
	565	+ "id": "b4fd8272c6061070475410ad248007f0",
	566	+ "count": 1
	567	+ },
	568	+ {
	569	+ "descriptor": "China, Shanghai",
	570	+ "id": "91336993fab910af6d710b3449b4c33e",
	571	+ "count": 164
	572	+ },
	573	+ {
	574	+ "descriptor": "China, Shenzhen",
	575	+ "id": "91336993fab910af6d7106eea8f4c334",
	576	+ "count": 52
	577	+ },
	578	+ {
	579	+ "descriptor": "Czechia, Remote",
	580	+ "id": "91336993fab910af6d70ecbaa6f4c2fd",
	581	+ "count": 3
	582	+ },
	583	+ {
	584	+ "descriptor": "Denmark, Remote",
	585	+ "id": "d21cf68980ad0193b0bfc3cab2079202",
	586	+ "count": 1
	587	+ },
	588	+ {
	589	+ "descriptor": "Denmark, Roskilde",
	590	+ "id": "c498fba66f4e01c0944dde87d5005f05",
	591	+ "count": 6
	592	+ },
	593	+ {
	594	+ "descriptor": "Finland, Helsinki",
	595	+ "id": "91336993fab910af6d6fdb4a260cc22a",
	596	+ "count": 4
	597	+ },
	598	+ {
	599	+ "descriptor": "Finland, Remote",
	600	+ "id": "91336993fab910af6d6fd9177efcc225",
	601	+ "count": 3
	602	+ },
	603	+ {
	604	+ "descriptor": "France, Courbevoie",
	605	+ "id": "91336993fab910af6d6fd25f3c44c216",
	606	+ "count": 14
	607	+ },
	608	+ {
	609	+ "descriptor": "France, Remote",
	610	+ "id": "91336993fab910af6d6fd6c8476cc220",
	611	+ "count": 23
	612	+ },
	613	+ {
	614	+ "descriptor": "Germany, Berlin",
	615	+ "id": "91336993fab910af6d6fc2799054c1f3",
	616	+ "count": 5
	617	+ },
	618	+ {
	619	+ "descriptor": "Germany, Munich",
	620	+ "id": "91336993fab910af6d6fc4f707bcc1f8",
	621	+ "count": 36
	622	+ },
	623	+ {
	624	+ "descriptor": "Germany, Remote",
	625	+ "id": "91336993fab910af6d6fc96a405cc202",
	626	+ "count": 45
	627	+ },
	628	+ {
	629	+ "descriptor": "Germany, Wuerselen",
	630	+ "id": "91336993fab910af6d6fc737d3e4c1fd",
	631	+ "count": 4
	632	+ },
	633	+ {
	634	+ "descriptor": "Greece, Remote",
	635	+ "id": "970bf8c909a701babc1935ddd400b807",
	636	+ "count": 1
	637	+ },
	638	+ {
	639	+ "descriptor": "Hong Kong, STP",
	640	+ "id": "91336993fab910af6d6fc02e0a14c1ee",
	641	+ "count": 4
	642	+ },
	643	+ {
	644	+ "descriptor": "Hungary, Budapest",
	645	+ "id": "fc94d97a3a280100b73f626fd1610000",
	646	+ "count": 1
	647	+ },
	648	+ {
	649	+ "descriptor": "Hungary, Remote",
	650	+ "id": "7bcfe8379f6b017176531c86a87649a7",
	651	+ "count": 2
	652	+ },
	653	+ {
	654	+ "descriptor": "India, Bengaluru",
	655	+ "id": "91336993fab910af6d6fb9748af4c1df",
	656	+ "count": 198
	657	+ },
	658	+ {
	659	+ "descriptor": "India, Gurugram",
	660	+ "id": "f28ae3c49d2601aff203e114d301695f",
	661	+ "count": 11
	662	+ },
	663	+ {
	664	+ "descriptor": "India, Hyderabad",
	665	+ "id": "91336993fab910af6d6fb2be0974c1d0",
	666	+ "count": 41
	667	+ },
	668	+ {
	669	+ "descriptor": "India, Mumbai",
	670	+ "id": "91336993fab910af6d6fabbb968cc1c1",
	671	+ "count": 13
	672	+ },
	673	+ {
	674	+ "descriptor": "India, Pune",
	675	+ "id": "91336993fab910af6d6fb0782884c1cb",
	676	+ "count": 68
	677	+ },
	678	+ {
	679	+ "descriptor": "India, Remote",
	680	+ "id": "91336993fab910af6d6fb4ee7634c1d5",
	681	+ "count": 16
	682	+ },
	683	+ {
	684	+ "descriptor": "Israel, Beer Sheva",
	685	+ "id": "970bf8c909a701bb080521ddd400ae07",
	686	+ "count": 25
	687	+ },
	688	+ {
	689	+ "descriptor": "Israel, Raanana",
	690	+ "id": "970bf8c909a7013ea54a57dcd4008107",
	691	+ "count": 91
	692	+ },
	693	+ {
	694	+ "descriptor": "Israel, Tel Aviv",
	695	+ "id": "c7769ee377291036b08490819096b8bf",
	696	+ "count": 234
	697	+ },
	698	+ {
	699	+ "descriptor": "Israel, Tel Hai",
	700	+ "id": "970bf8c909a7016a7d893ddcd4007c07",
	701	+ "count": 12
	702	+ },
	703	+ {
	704	+ "descriptor": "Israel, Yokneam",
	705	+ "id": "970bf8c909a701c749f87bdcd4008607",
	706	+ "count": 308
	707	+ },
	708	+ {
	709	+ "descriptor": "Italy, Remote",
	710	+ "id": "91336993fab910af6d6fa203fa6cc1ad",
	711	+ "count": 3
	712	+ },
	713	+ {
	714	+ "descriptor": "Japan, Remote",
	715	+ "id": "b00b3256ed551015d42d2bebe06b02b7",
	716	+ "count": 5
	717	+ },
	718	+ {
	719	+ "descriptor": "Japan, Tokyo",
	720	+ "id": "91336993fab910af6d6f9a47b91cc19e",
	721	+ "count": 20
	722	+ },
	723	+ {
	724	+ "descriptor": "Korea, Remote",
	725	+ "id": "91336993fab910af6d71963a259cc46f",
	726	+ "count": 2
	727	+ },
	728	+ {
	729	+ "descriptor": "Korea, Seoul",
	730	+ "id": "91336993fab910af6d6f3f56714cc112",
	731	+ "count": 19
	732	+ },
	733	+ {
	734	+ "descriptor": "Mexico, Remote",
	735	+ "id": "91336993fab910af6d6f97e94b04c199",
	736	+ "count": 3
	737	+ },
	738	+ {
	739	+ "descriptor": "Netherlands, Amsterdam",
	740	+ "id": "42289dda31210100f43ca5dad0ff0000",
	741	+ "count": 2
	742	+ },
	743	+ {
	744	+ "descriptor": "Netherlands, Remote",
	745	+ "id": "5f84e8157c6a101b415772d270da2de1",
	746	+ "count": 7
	747	+ },
	748	+ {
	749	+ "descriptor": "Palestine, Rawabi",
	750	+ "id": "30350996b69b01a7dbccf4403c0174a8",
	751	+ "count": 3
	752	+ },
	753	+ {
	754	+ "descriptor": "Poland, Remote",
	755	+ "id": "91336993fab910af6d6f931ae68cc18f",
	756	+ "count": 25
	757	+ },
	758	+ {
	759	+ "descriptor": "Poland, Warsaw",
	760	+ "id": "811700f7388d0103b8ceed030d490e48",
	761	+ "count": 11
	762	+ },
	763	+ {
	764	+ "descriptor": "Romania, Remote",
	765	+ "id": "5f84e8157c6a101b435468d5d0e34bd9",
	766	+ "count": 1
	767	+ },
	768	+ {
	769	+ "descriptor": "Singapore, Pasir Panjang",
	770	+ "id": "de99d77118d51016b3861587f8f30000",
	771	+ "count": 2
	772	+ },
	773	+ {
	774	+ "descriptor": "Singapore, Remote",
	775	+ "id": "91336993fab910af6d6f8a1a368cc17b",
	776	+ "count": 11
	777	+ },
	778	+ {
	779	+ "descriptor": "Singapore, Singapore-Suntec Tower",
	780	+ "id": "c498fba66f4e01971ba26688d5008705",
	781	+ "count": 16
	782	+ },
	783	+ {
	784	+ "descriptor": "Spain, Remote",
	785	+ "id": "91336993fab910af6d6f87de293cc176",
	786	+ "count": 14
	787	+ },
	788	+ {
	789	+ "descriptor": "Sweden, Gothenburg",
	790	+ "id": "c18da66b8c3f01e4f8939069de3534f8",
	791	+ "count": 4
	792	+ },
	793	+ {
	794	+ "descriptor": "Sweden, Lund",
	795	+ "id": "91336993fab910af6d6f859de1e4c171",
	796	+ "count": 2
	797	+ },
	798	+ {
	799	+ "descriptor": "Sweden, Remote",
	800	+ "id": "91336993fab910af6d6f8342ac04c16c",
	801	+ "count": 6
	802	+ },
	803	+ {
	804	+ "descriptor": "Switzerland, Remote",
	805	+ "id": "91336993fab910af6d6f7e35a07cc162",
	806	+ "count": 25
	807	+ },
	808	+ {
	809	+ "descriptor": "Switzerland, Zurich",
	810	+ "id": "91336993fab910af6d6f80c09504c167",
	811	+ "count": 16
	812	+ },
	813	+ {
	814	+ "descriptor": "Taiwan, Hsinchu",
	815	+ "id": "91336993fab910af6d6f7933b49cc158",
	816	+ "count": 74
	817	+ },
	818	+ {
	819	+ "descriptor": "Taiwan, Remote",
	820	+ "id": "91336993fab910af6d6f56c5409cc13f",
	821	+ "count": 1
	822	+ },
	823	+ {
	824	+ "descriptor": "Taiwan, Taipei",
	825	+ "id": "91336993fab910af6d6f5eaae3acc14e",
	826	+ "count": 77
	827	+ },
	828	+ {
	829	+ "descriptor": "Thailand, Remote",
	830	+ "id": "f57d67c891a11057224421d76bc979ab",
	831	+ "count": 4
	832	+ },
	833	+ {
	834	+ "descriptor": "UAE, Dubai",
	835	+ "id": "91336993fab910af6d6f516bf58cc135",
	836	+ "count": 3
	837	+ },
	838	+ {
	839	+ "descriptor": "UAE, Remote",
	840	+ "id": "fd8f9cd69ba6019914bb053e4e01ebba",
	841	+ "count": 1
	842	+ },
	843	+ {
	844	+ "descriptor": "UK, Belfast",
	845	+ "id": "970bf8c909a7015f56c618ded400e007",
	846	+ "count": 1
	847	+ },
	848	+ {
	849	+ "descriptor": "UK, Bristol",
	850	+ "id": "91336993fab910af6d6f41d01ff4c117",
	851	+ "count": 4
	852	+ },
	853	+ {
	854	+ "descriptor": "UK, Cambridge",
	855	+ "id": "91336993fab910af6d6f445436acc11c",
	856	+ "count": 6
	857	+ },
	858	+ {
	859	+ "descriptor": "UK, Reading",
	860	+ "id": "91336993fab910af6d6f4c211db4c12b",
	861	+ "count": 10
	862	+ },
	863	+ {
	864	+ "descriptor": "UK, Remote",
	865	+ "id": "91336993fab910af6d6f4954559cc126",
	866	+ "count": 44
	867	+ },
	868	+ {
	869	+ "descriptor": "Ukraine, Kyiv",
	870	+ "id": "970bf8c909a701acf1aa08ded400db07",
	871	+ "count": 5
	872	+ },
	873	+ {
	874	+ "descriptor": "Ukraine, Remote",
	875	+ "id": "970bf8c909a70104bdbe3eddd400bd07",
	876	+ "count": 6
	877	+ },
	878	+ {
	879	+ "descriptor": "US, AL, Madison",
	880	+ "id": "91336993fab910af6d7140d48764c3b6",
	881	+ "count": 3
	882	+ },
	883	+ {
	884	+ "descriptor": "US, AL, Remote",
	885	+ "id": "91336993fab910af6d7170a1a684c41f",
	886	+ "count": 1
	887	+ },
	888	+ {
	889	+ "descriptor": "US, AR, Remote",
	890	+ "id": "df27165dab550120b41aa51fc701619d",
	891	+ "count": 4
	892	+ },
	893	+ {
	894	+ "descriptor": "US, AZ, Remote",
	895	+ "id": "91336993fab910af6d7013ca699cc2a7",
	896	+ "count": 9
	897	+ },
	898	+ {
	899	+ "descriptor": "US, CA, Remote",
	900	+ "id": "91336993fab910af6d716528e9d4c406",
	901	+ "count": 233
	902	+ },
	903	+ {
	904	+ "descriptor": "US, CA, Santa Clara",
	905	+ "id": "91336993fab910af6d702fae0bb4c2e8",
	906	+ "count": 1278
	907	+ },
	908	+ {
	909	+ "descriptor": "US, CO, Boulder",
	910	+ "id": "91336993fab910af6d712ddeebf4c38e",
	911	+ "count": 10
	912	+ },
	913	+ {
	914	+ "descriptor": "US, CO, Remote",
	915	+ "id": "91336993fab910af6d716c1bb8d4c415",
	916	+ "count": 22
	917	+ },
	918	+ {
	919	+ "descriptor": "US, CT, Remote",
	920	+ "id": "91336993fab910af6d715e41b394c3f7",
	921	+ "count": 2
	922	+ },
	923	+ {
	924	+ "descriptor": "US, DC, Remote",
	925	+ "id": "91336993fab910af6d713a0f5044c3a7",
	926	+ "count": 15
	927	+ },
	928	+ {
	929	+ "descriptor": "US, DE, Remote",
	930	+ "id": "e271d799635b01d72799f699d2d07a59",
	931	+ "count": 1
	932	+ },
	933	+ {
	934	+ "descriptor": "US, FL, Remote",
	935	+ "id": "91336993fab910af6d7020bf3b14c2c5",
	936	+ "count": 11
	937	+ },
	938	+ {
	939	+ "descriptor": "US, GA, Remote",
	940	+ "id": "91336993fab910af6d6fe4b908fcc23e",
	941	+ "count": 8
	942	+ },
	943	+ {
	944	+ "descriptor": "US, ID, Remote",
	945	+ "id": "91336993fab910af6d7143163a04c3bb",
	946	+ "count": 1
	947	+ },
	948	+ {
	949	+ "descriptor": "US, IL, Champaign",
	950	+ "id": "3f056d7dcee210447f581d1002c2e672",
	951	+ "count": 4
	952	+ },
	953	+ {
	954	+ "descriptor": "US, IL, Remote",
	955	+ "id": "91336993fab910af6d714794d4b4c3c5",
	956	+ "count": 13
	957	+ },
	958	+ {
	959	+ "descriptor": "US, MA, Remote",
	960	+ "id": "91336993fab910af6d702d89918cc2e3",
	961	+ "count": 37
	962	+ },
	963	+ {
	964	+ "descriptor": "US, MA, Westford",
	965	+ "id": "91336993fab910af6d7008ff1774c28e",
	966	+ "count": 55
	967	+ },
	968	+ {
	969	+ "descriptor": "US, MD, Remote",
	970	+ "id": "91336993fab910af6d719855647cc474",
	971	+ "count": 3
	972	+ },
	973	+ {
	974	+ "descriptor": "US, MI, Remote",
	975	+ "id": "91336993fab910af6d6fe9a03534c248",
	976	+ "count": 2
	977	+ },
	978	+ {
	979	+ "descriptor": "US, MN, Remote",
	980	+ "id": "91336993fab910af6d700b282d4cc293",
	981	+ "count": 1
	982	+ },
	983	+ {
	984	+ "descriptor": "US, MO, St. Louis",
	985	+ "id": "91336993fab910af6d7130bb158cc393",
	986	+ "count": 1
	987	+ },
	988	+ {
	989	+ "descriptor": "US, NC, Durham",
	990	+ "id": "91336993fab910af6d7022e347dcc2ca",
	991	+ "count": 65
	992	+ },
	993	+ {
	994	+ "descriptor": "US, NC, Remote",
	995	+ "id": "91336993fab910af6d7006cdf31cc289",
	996	+ "count": 27
	997	+ },
	998	+ {
	999	+ "descriptor": "US, NH, Remote",
	1000	+ "id": "91336993fab910af6d6ffd32b89cc275",
	1001	+ "count": 2
	1002	+ },
	1003	+ {
	1004	+ "descriptor": "US, NJ, Holmdel",
	1005	+ "id": "2b3c46170e30100a3d72f9d5780d7c39",
	1006	+ "count": 2
	1007	+ },
	1008	+ {
	1009	+ "descriptor": "US, NJ, Remote",
	1010	+ "id": "91336993fab910af6d7152b6749cc3de",
	1011	+ "count": 10
	1012	+ },
	1013	+ {
	1014	+ "descriptor": "US, NM, Remote",
	1015	+ "id": "91336993fab910af6d712b900a54c389",
	1016	+ "count": 1
	1017	+ },
	1018	+ {
	1019	+ "descriptor": "US, NV, Remote",
	1020	+ "id": "91336993fab910af6d7015dee21cc2ac",
	1021	+ "count": 4
	1022	+ },
	1023	+ {
	1024	+ "descriptor": "US, NY, New York",
	1025	+ "id": "d2088e737cbb01293f745dc7ce017070",
	1026	+ "count": 18
	1027	+ },
	1028	+ {
	1029	+ "descriptor": "US, NY, Remote",
	1030	+ "id": "91336993fab910af6d6ff2800b0cc25c",
	1031	+ "count": 32
	1032	+ },
	1033	+ {
	1034	+ "descriptor": "US, OH, Remote",
	1035	+ "id": "91336993fab910af6d700d4fc83cc298",
	1036	+ "count": 1
	1037	+ },
	1038	+ {
	1039	+ "descriptor": "US, OR, Hillsboro",
	1040	+ "id": "91336993fab910af6d7027195454c2d4",
	1041	+ "count": 50
	1042	+ },
	1043	+ {
	1044	+ "descriptor": "US, OR, Remote",
	1045	+ "id": "91336993fab910af6d6ff05591bcc257",
	1046	+ "count": 25
	1047	+ },
	1048	+ {
	1049	+ "descriptor": "US, PA, Remote",
	1050	+ "id": "91336993fab910af6d700193f3c4c27f",
	1051	+ "count": 3
	1052	+ },
	1053	+ {
	1054	+ "descriptor": "US, Remote",
	1055	+ "id": "16fc4607fc4310011e929f7115f90000",
	1056	+ "count": 115
	1057	+ },
	1058	+ {
	1059	+ "descriptor": "US, SC, Remote",
	1060	+ "id": "91336993fab910af6d6ffb099324c270",
	1061	+ "count": 5
	1062	+ },
	1063	+ {
	1064	+ "descriptor": "US, SD, Remote",
	1065	+ "id": "91336993fab910af6d715788d0b4c3e8",
	1066	+ "count": 1
	1067	+ },
	1068	+ {
	1069	+ "descriptor": "US, TN, Remote",
	1070	+ "id": "91336993fab910af6d7122ef682cc375",
	1071	+ "count": 5
	1072	+ },
	1073	+ {
	1074	+ "descriptor": "US, TX, Austin",
	1075	+ "id": "91336993fab910af6d702b631b94c2de",
	1076	+ "count": 186
	1077	+ },
	1078	+ {
	1079	+ "descriptor": "US, TX, Dallas",
	1080	+ "id": "10b0744835e11001032365ce4e460000",
	1081	+ "count": 3
	1082	+ },
	1083	+ {
	1084	+ "descriptor": "US, TX, Houston",
	1085	+ "id": "10b0744835e11001030c2489586f0000",
	1086	+ "count": 2
	1087	+ },
	1088	+ {
	1089	+ "descriptor": "US, TX, Remote",
	1090	+ "id": "91336993fab910af6d702939a7fcc2d9",
	1091	+ "count": 100
	1092	+ },
	1093	+ {
	1094	+ "descriptor": "US, UT, Remote",
	1095	+ "id": "91336993fab910af6d6fee30ae1cc252",
	1096	+ "count": 1
	1097	+ },
	1098	+ {
	1099	+ "descriptor": "US, UT, Salt Lake City",
	1100	+ "id": "91336993fab910af6d6ff8e9d004c26b",
	1101	+ "count": 1
	1102	+ },
	1103	+ {
	1104	+ "descriptor": "US, VA, Charlottesville",
	1105	+ "id": "91336993fab910af6d6fdfb0f2fcc234",
	1106	+ "count": 1
	1107	+ },
	1108	+ {
	1109	+ "descriptor": "US, VA, Herndon",
	1110	+ "id": "c498fba66f4e012b5a8a2e88d5007805",
	1111	+ "count": 2
	1112	+ },
	1113	+ {
	1114	+ "descriptor": "US, VA, Remote",
	1115	+ "id": "91336993fab910af6d7159be65c4c3ed",
	1116	+ "count": 16
	1117	+ },
	1118	+ {
	1119	+ "descriptor": "US, WA, Redmond",
	1120	+ "id": "91336993fab910af6d701e82d004c2c0",
	1121	+ "count": 66
	1122	+ },
	1123	+ {
	1124	+ "descriptor": "US, WA, Remote",
	1125	+ "id": "91336993fab910af6d7169a81124c410",
	1126	+ "count": 68
	1127	+ },
	1128	+ {
	1129	+ "descriptor": "US, WA, Seattle",
	1130	+ "id": "d2088e737cbb01d5e2be9e52ce01926f",
	1131	+ "count": 60
	1132	+ },
	1133	+ {
	1134	+ "descriptor": "US, WI, Remote",
	1135	+ "id": "91336993fab910af6d701c62554cc2bb",
	1136	+ "count": 1
	1137	+ },
	1138	+ {
	1139	+ "descriptor": "US, WY, Remote",
	1140	+ "id": "91336993fab910af6d72a99f46acc4f2",
	1141	+ "count": 1
	1142	+ },
	1143	+ {
	1144	+ "descriptor": "Vietnam, Hanoi",
	1145	+ "id": "dd4916e138561000a8f8afa559f50000",
	1146	+ "count": 19
	1147	+ },
	1148	+ {
	1149	+ "descriptor": "Vietnam, Ho Chi Minh City",
	1150	+ "id": "229b3d23e3f91000aa5f712809fa0000",
	1151	+ "count": 15
	1152	+ },
	1153	+ {
	1154	+ "descriptor": "Vietnam, Remote",
	1155	+ "id": "7ae28cf24c8b10874977c2d1387c19b0",
	1156	+ "count": 16
	1157	+ }
	1158	+ ]
	1159	+ }
	1160	+ ]
	1161	+ }
	1162	+ ],
	1163	+ "userAuthenticated": false
	1164	+}
		\ No newline at end of file

added src/companyatlas/commands/crawl.py +196 −0

		`@@ -0,0 +1,196 @@`
	1	+"""`catlas` crawl commands: onboard · discover · run-sensor · schedule · sensors · connectors · repair · stats-crawl."""
	2	+from __future__ import annotations
	3	+
	4	+import json
	5	+from typing import Annotated, Any
	6	+
	7	+import typer
	8	+from rich.table import Table
	9	+
	10	+from companyatlas.cli import console, out, run_async
	11	+
	12	+
	13	+def _emit(obj: Any) -> None:
	14	+ """Machine-readable line on stdout (never wrapped by the console)."""
	15	+ out.print(json.dumps(obj, default=str), soft_wrap=True, highlight=False, markup=False)
	16	+
	17	+
	18	+def _table(title: str, columns: list[str], rows: list[list[Any]]) -> None:
	19	+ t = Table(title=title, show_lines=False, header_style="bold")
	20	+ for c in columns:
	21	+ t.add_column(c, overflow="fold")
	22	+ for r in rows:
	23	+ t.add_row(*[("" if v is None else str(v)) for v in r])
	24	+ console.print(t)
	25	+
	26	+
	27	+def register(app: typer.Typer) -> None:
	28	+ @app.command()
	29	+ def onboard(limit: int = 50, company: Annotated[str \| None, typer.Option(help="slug / id / domain of one company")] = None,
	30	+ concurrency: int \| None = None, fetch_now: bool = False, dry_run: bool = False) -> None:
	31	+ """Discover surfaces and create sensors for pending companies (or one company)."""
	32	+ from companyatlas.services.discovery import onboard_pending
	33	+
	34	+ stats = run_async(onboard_pending(limit, concurrency, company_slug=company, fetch_now=fetch_now, dry_run=dry_run))
	35	+ console.print(f"[bold]onboarding[/] claimed={stats.get('claimed', 0)} active={stats.get('active', 0)} failed={stats.get('failed', 0)} "
	36	+ f"no_website={stats.get('no_website', 0)} sensors={stats.get('sensors', 0)}")
	37	+ _emit(stats)
	38	+
	39	+ @app.command()
	40	+ def discover(target: Annotated[str, typer.Argument(help="website URL or company slug")], dry_run: bool = True,
	41	+ json_out: Annotated[bool, typer.Option("--json")] = False) -> None:
	42	+ """Run discovery for a website or an existing company and print the surface table (dry-run by default)."""
	43	+ from companyatlas.db import fetch_one, transaction
	44	+ from companyatlas.fetch import Fetcher
	45	+ from companyatlas.ids import new_id, slugify
	46	+ from companyatlas.services.discovery import discover_company
	47	+ from companyatlas.urls import registrable_domain
	48	+
	49	+ async def go() -> Any:
	50	+ async with transaction() as conn:
	51	+ company = await fetch_one(conn, "select * from companies where slug = :t or id = :t or canonical_domain = :t", t=target)
	52	+ if company is None:
	53	+ if "." not in target:
	54	+ raise typer.BadParameter(f"{target!r} is neither a known company nor a website")
	55	+ website = target if target.startswith("http") else f"https://{target}"
	56	+ dom = registrable_domain(website)
	57	+ company = {"id": new_id("company"), "slug": slugify(dom.split(".")[0]), "display_name": dom, "canonical_domain": dom, "website": website, "tier": 3,
	58	+ "importance": 0.3}
	59	+ if not dry_run:
	60	+ raise typer.BadParameter("persisting discovery for an unknown website requires a seeded company — use --dry-run or seed it first")
	61	+ async with Fetcher() as fetcher:
	62	+ return await discover_company(company, fetcher=fetcher, dry_run=dry_run)
	63	+
	64	+ res = run_async(go())
	65	+ rows = [[s["surface"], s["url"], s["connector"], f"{s['confidence']:.2f}", s["method"], s["tier"], s["interval_s"] // 60, s["quality"]] for s in res.table()]
	66	+ _table(f"{res.canonical_domain} — {len(rows)} sensors · status={res.status} · {res.requests} requests · {res.duration_ms} ms",
	67	+ ["surface", "url", "connector", "conf", "method", "tier", "every (min)", "quality"], rows)
	68	+ if res.notes or res.error:
	69	+ console.print(f"[dim]notes: {'; '.join(res.notes)}[/]" + (f" [red]error: {res.error}[/]" if res.error else ""))
	70	+ if res.ats:
	71	+ console.print(f"[dim]ATS: {res.ats}[/]")
	72	+ if json_out:
	73	+ _emit({"status": res.status, "canonical_domain": res.canonical_domain, "final_url": res.final_url, "sensors": res.table(), "ats": res.ats,
	74	+ "notes": res.notes, "error": res.error, "requests": res.requests, "duration_ms": res.duration_ms})
	75	+
	76	+ @app.command("run-sensor")
	77	+ def run_sensor_cmd(ref: Annotated[str, typer.Argument(help="sensor id or URL")], file: Annotated[str \| None, typer.Option("--file", help="fixture file instead of the network")] = None,
	78	+ force: bool = False) -> None:
	79	+ """Run one sensor now (optionally against a local file) and print the outcome."""
	80	+ from companyatlas.fetch import Fetcher
	81	+ from companyatlas.services.pipeline import load_sensor, run_sensor, run_sensor_by_url_with_file
	82	+
	83	+ async def go() -> Any:
	84	+ if file:
	85	+ return await run_sensor_by_url_with_file(ref, file, force=force)
	86	+ row = await load_sensor(ref)
	87	+ if row is None:
	88	+ raise typer.BadParameter(f"sensor {ref!r} not found")
	89	+ async with Fetcher() as fetcher:
	90	+ return await run_sensor(row, fetcher=fetcher, worker="cli", force=force)
	91	+
	92	+ o = run_async(go())
	93	+ console.print(f"[bold]{o.status}[/] sensor={o.sensor_id} kind={o.kind} significance={o.significance} snapshot={o.snapshot_id} change={o.change_id} "
	94	+ f"delta={o.delta_counts} interval={o.interval_s}s next={o.next_run_at} failure={o.failure_class} {o.error or ''}")
	95	+ _emit({"status": o.status, "sensor_id": o.sensor_id, "observation_id": o.observation_id, "snapshot_id": o.snapshot_id, "change_id": o.change_id,
	96	+ "kind": o.kind, "significance": o.significance, "failure_class": o.failure_class, "error": o.error, "delta": o.delta_counts,
	97	+ "interval_s": o.interval_s, "next_run_at": o.next_run_at, "duration_ms": o.duration_ms})
	98	+
	99	+ @app.command()
	100	+ def schedule(concurrency: int \| None = None, no_onboarding: bool = False, once: bool = False, worker: str \| None = None) -> None:
	101	+ """Run the crawl scheduler: claim due sensors, run them, onboarding worker, periodic tasks. Ctrl-C to stop."""
	102	+ from companyatlas.services.scheduler import run_scheduler
	103	+
	104	+ run_async(run_scheduler(concurrency=concurrency, onboarding=not no_onboarding, once=once, worker=worker))
	105	+
	106	+ @app.command()
	107	+ def sensors(company: str \| None = None, status: str \| None = None, surface: str \| None = None, limit: int = 50,
	108	+ json_out: Annotated[bool, typer.Option("--json")] = False) -> None:
	109	+ """List sensors (filters: --company slug, --status, --surface)."""
	110	+ from companyatlas.db import fetch_all, transaction
	111	+
	112	+ async def go() -> list[dict[str, Any]]:
	113	+ async with transaction() as conn:
	114	+ return await fetch_all(conn, """
	115	+ select s.id, c.slug, s.surface, s.connector_id, s.url, s.status, s.tier, s.current_interval_s, s.next_run_at, s.last_run_at, s.quality_score,
	116	+ s.consecutive_failures, s.consecutive_unchanged, s.snapshot_count, s.change_count, s.meaningful_change_count, s.last_failure_class
	117	+ from sensors s join companies c on c.id = s.company_id
	118	+ where (:company is null or c.slug = :company or c.id = :company or c.canonical_domain = :company)
	119	+ and (:status is null or s.status = :status) and (:surface is null or s.surface = :surface)
	120	+ order by c.slug, s.surface limit :limit
	121	+ """, company=company, status=status, surface=surface, limit=limit)
	122	+
	123	+ rows = run_async(go())
	124	+ _table(f"{len(rows)} sensors", ["id", "company", "surface", "connector", "status", "tier", "every", "next", "quality", "fail", "unchg", "snaps", "chg", "mean", "url"],
	125	+ [[r["id"], r["slug"], r["surface"], r["connector_id"], r["status"], r["tier"], f"{r['current_interval_s'] // 60}m",
	126	+ r["next_run_at"].strftime("%m-%d %H:%M") if r["next_run_at"] else "", r["quality_score"], r["consecutive_failures"], r["consecutive_unchanged"],
	127	+ r["snapshot_count"], r["change_count"], r["meaningful_change_count"], r["url"]] for r in rows])
	128	+ if json_out:
	129	+ _emit(rows)
	130	+
	131	+ @app.command("connectors")
	132	+ def connectors_cmd(sync: bool = True) -> None:
	133	+ """List registered connector families and sync the `connectors` table."""
	134	+ from companyatlas.db import transaction
	135	+ from companyatlas.sdk.connector import all_connectors, sync_connectors_table
	136	+
	137	+ async def go() -> int:
	138	+ if not sync:
	139	+ return 0
	140	+ async with transaction() as conn:
	141	+ return await sync_connectors_table(conn)
	142	+
	143	+ n = run_async(go())
	144	+ _table(f"{len(all_connectors())} connectors (synced {n})", ["id", "name", "category", "fetch", "discovery", "incremental", "default interval", "url pattern"],
	145	+ [[c.meta.connector_id, c.meta.name, c.meta.category, c.meta.fetch_mode, c.meta.supports_discovery, c.meta.supports_incremental,
	146	+ f"{c.meta.default_interval_s // 3600}h", c.meta.url_pattern or ""] for c in all_connectors()])
	147	+
	148	+ @app.command()
	149	+ def repair(limit: int = 50, dry_run: bool = False) -> None:
	150	+ """Auto-repair failing / stale / redirected sensors (retry → sitemap → navigation → identity check → migrate or review)."""
	151	+ from companyatlas.services.repair import repair_batch
	152	+
	153	+ stats = run_async(repair_batch(limit, dry_run=dry_run))
	154	+ _table(f"repair: examined={stats['examined']} recovered={stats.get('recovered', 0)} migrated={stats.get('migrated', 0)} review={stats.get('review', 0)}",
	155	+ ["sensor", "surface", "action", "url", "new url", "requests"],
	156	+ [[r.get("sensor_id"), r.get("surface"), r.get("action"), r.get("url"), r.get("new_url", ""), r.get("requests", 0)] for r in stats["results"]])
	157	+ _emit({k: v for k, v in stats.items() if k != "results"})
	158	+
	159	+ @app.command("stats-crawl")
	160	+ def stats_crawl() -> None:
	161	+ """Quick crawl counters: companies by onboarding status, sensors by status/tier, observations/changes today, failures, heartbeat."""
	162	+ from companyatlas.db import fetch_all, fetch_one, transaction
	163	+
	164	+ async def go() -> dict[str, Any]:
	165	+ async with transaction() as conn:
	166	+ return {
	167	+ "companies": await fetch_all(conn, "select onboarding_status as k, count(*) as n from companies group by 1 order by 1"),
	168	+ "sensors_status": await fetch_all(conn, "select status as k, count(*) as n from sensors group by 1 order by 1"),
	169	+ "sensors_tier": await fetch_all(conn, "select tier as k, count(*) as n from sensors where status in ('active','pending','failing') group by 1 order by 1"),
	170	+ "surfaces": await fetch_all(conn, "select surface as k, count(*) as n from sensors where status <> 'retired' group by 1 order by 2 desc limit 40"),
	171	+ "today": await fetch_one(conn, """select (select count(*) from observations where fetched_at >= current_date) as observations,
	172	+ (select count(*) from observations where fetched_at >= current_date and not_modified) as not_modified,
	173	+ (select count(*) from observations where fetched_at >= current_date and failure_class is not null) as failed,
	174	+ (select count(*) from snapshots where fetched_at >= current_date) as snapshots,
	175	+ (select count(*) from changes where detected_at >= current_date) as changes,
	176	+ (select count(*) from changes where detected_at >= current_date and kind in ('meaningful','major','critical')) as meaningful,
	177	+ (select count(*) from sensors where status in ('active','failing','pending') and next_run_at <= now()) as due,
	178	+ (select count(*) from jobs where status = 'open') as jobs_open,
	179	+ (select count(*) from review_queue where status = 'open') as review_open"""),
	180	+ "failures": await fetch_all(conn, "select failure_class as k, count(*) as n from failures where at >= now() - interval '24 hours' group by 1 order by 2 desc"),
	181	+ "heartbeat": await fetch_one(conn, "select value, updated_at from settings_kv where key = 'scheduler:heartbeat'"),
	182	+ }
	183	+
	184	+ s = run_async(go())
	185	+ _table("companies", ["onboarding_status", "n"], [[r["k"], r["n"]] for r in s["companies"]])
	186	+ _table("sensors by status", ["status", "n"], [[r["k"], r["n"]] for r in s["sensors_status"]])
	187	+ _table("sensors by tier", ["tier", "n"], [[r["k"], r["n"]] for r in s["sensors_tier"]])
	188	+ _table("surfaces", ["surface", "n"], [[r["k"], r["n"]] for r in s["surfaces"]])
	189	+ _table("failures 24h", ["class", "n"], [[r["k"], r["n"]] for r in s["failures"]])
	190	+ today = s["today"] or {}
	191	+ console.print("[bold]today[/] " + " ".join(f"{k}={v}" for k, v in today.items()))
	192	+ hb = s["heartbeat"]
	193	+ if hb:
	194	+ v = hb["value"]
	195	+ console.print(f"[bold]heartbeat[/] worker={v.get('worker')} at={v.get('at')} inflight={v.get('inflight')} due={v.get('due')} tick_ms={v.get('tick_ms')}")
	196	+ _emit(s)

added src/companyatlas/connectors/_ats_base.py +62 −0

		`@@ -0,0 +1,62 @@`
	1	+"""Base class for structured job-board connectors (spec §9.3): the sensor URL is the public JSON/XML endpoint the vendor's own
	2	+career page loads. Each subclass implements `parse_jobs(data, sensor)`; the base turns the list into an `Extraction` whose blocks
	3	+mirror the jobs one-to-one (stable keys from external ids) so the block diff equals the job delta."""
	4	+from __future__ import annotations
	5	+
	6	+import re
	7	+from collections.abc import Mapping
	8	+from typing import Any, ClassVar
	9	+
	10	+from companyatlas.connectors._util import finish_job, job_blocks, jobs_text, load_json
	11	+from companyatlas.fetch import FetchResult
	12	+from companyatlas.sdk.connector import Connector
	13	+from companyatlas.sdk.models import ExtractedJob, Extraction
	14	+from companyatlas.taxonomy import Surface
	15	+
	16	+MAX_JOBS = 5000
	17	+
	18	+
	19	+class AtsConnector(Connector):
	20	+ vendor: ClassVar[str] = ""
	21	+ token_re: ClassVar[re.Pattern[str] \| None] = None
	22	+
	23	+ def token(self, sensor: Mapping[str, Any]) -> str \| None:
	24	+ cfg = sensor.get("config") or {}
	25	+ if isinstance(cfg, Mapping) and cfg.get("token"):
	26	+ return str(cfg["token"])
	27	+ if self.token_re is not None:
	28	+ m = self.token_re.search(str(sensor.get("url") or ""))
	29	+ if m:
	30	+ return m.group(1)
	31	+ return None
	32	+
	33	+ def load(self, result: FetchResult) -> Any:
	34	+ return load_json(result)
	35	+
	36	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]: # pragma: no cover - abstract
	37	+ raise NotImplementedError
	38	+
	39	+ def extract(self, sensor: Mapping[str, Any], result: FetchResult) -> Extraction:
	40	+ data = self.load(result)
	41	+ jobs = [finish_job(j) for j in self.parse_jobs(data, sensor)[:MAX_JOBS] if j.title]
	42	+ # dedupe by external id / url
	43	+ seen: set[str] = set()
	44	+ unique: list[ExtractedJob] = []
	45	+ for j in jobs:
	46	+ k = (j.external_id or j.url or f"{j.title}\|{j.location_text}").lower()
	47	+ if k in seen:
	48	+ continue
	49	+ seen.add(k)
	50	+ unique.append(j)
	51	+ header = f"{self.vendor} board — {len(unique)} open positions"
	52	+ meta = {"vendor": self.vendor, "token": self.token(sensor), "job_count": len(unique), "source_url": result.final_url,
	53	+ "structured": True, "pages": result.headers.get("x-companyatlas-pages", "1")}
	54	+ return Extraction(text=jobs_text(unique, header), blocks=job_blocks(unique, path=f"{self.vendor} jobs"), title=header,
	55	+ language=None, meta=meta, jobs=unique)
	56	+
	57	+
	58	+def surface_jobs_board() -> str:
	59	+ return str(Surface.JOBS_BOARD)
	60	+
	61	+
	62	+__all__ = ["MAX_JOBS", "AtsConnector", "surface_jobs_board"]

added src/companyatlas/connectors/_util.py +428 −0

		`@@ -0,0 +1,428 @@`
	1	+"""Shared helpers for connector families: location / country parsing (never invent), date parsing, job normalisation,
	2	+fingerprints, JSON access and the ATS vendor → API mapping used by discovery. Deterministic, no network."""
	3	+from __future__ import annotations
	4	+
	5	+import hashlib
	6	+import json
	7	+import re
	8	+from datetime import UTC, datetime
	9	+from typing import Any
	10	+from urllib.parse import urlparse
	11	+
	12	+from dateutil import parser as dtparser
	13	+
	14	+from companyatlas.fetch import FetchResult
	15	+from companyatlas.ids import stable_hash
	16	+from companyatlas.sdk.models import Block, ExtractedJob
	17	+from companyatlas.sdk.normalize import BLOCK_WEIGHTS, normalize_whitespace, normalized_text, simhash
	18	+
	19	+# ------------------------------------------------------------------------------------------------------------ countries
	20	+
	21	+_COUNTRIES: dict[str, str] = {
	22	+ "afghanistan": "AF", "albania": "AL", "algeria": "DZ", "andorra": "AD", "angola": "AO", "argentina": "AR", "armenia": "AM", "australia": "AU",
	23	+ "austria": "AT", "österreich": "AT", "azerbaijan": "AZ", "bahamas": "BS", "bahrain": "BH", "bangladesh": "BD", "belarus": "BY", "belgium": "BE",
	24	+ "belgique": "BE", "belgië": "BE", "bolivia": "BO", "bosnia and herzegovina": "BA", "bosnia": "BA", "botswana": "BW", "brazil": "BR", "brasil": "BR",
	25	+ "brunei": "BN", "bulgaria": "BG", "cambodia": "KH", "cameroon": "CM", "canada": "CA", "chile": "CL", "china": "CN", "中国": "CN",
	26	+ "colombia": "CO", "costa rica": "CR", "croatia": "HR", "cuba": "CU", "cyprus": "CY", "czech republic": "CZ", "czechia": "CZ", "denmark": "DK",
	27	+ "danmark": "DK", "dominican republic": "DO", "ecuador": "EC", "egypt": "EG", "el salvador": "SV", "estonia": "EE", "ethiopia": "ET",
	28	+ "finland": "FI", "suomi": "FI", "france": "FR", "georgia": "GE", "germany": "DE", "deutschland": "DE", "ghana": "GH", "greece": "GR",
	29	+ "guatemala": "GT", "honduras": "HN", "hong kong": "HK", "hungary": "HU", "iceland": "IS", "india": "IN", "indonesia": "ID", "iran": "IR",
	30	+ "iraq": "IQ", "ireland": "IE", "israel": "IL", "italy": "IT", "italia": "IT", "jamaica": "JM", "japan": "JP", "日本": "JP", "jordan": "JO",
	31	+ "kazakhstan": "KZ", "kenya": "KE", "kuwait": "KW", "latvia": "LV", "lebanon": "LB", "lithuania": "LT", "luxembourg": "LU", "macau": "MO",
	32	+ "malaysia": "MY", "malta": "MT", "mexico": "MX", "méxico": "MX", "moldova": "MD", "monaco": "MC", "mongolia": "MN", "montenegro": "ME",
	33	+ "morocco": "MA", "myanmar": "MM", "nepal": "NP", "netherlands": "NL", "the netherlands": "NL", "nederland": "NL", "holland": "NL",
	34	+ "new zealand": "NZ", "nicaragua": "NI", "nigeria": "NG", "north macedonia": "MK", "norway": "NO", "norge": "NO", "oman": "OM", "pakistan": "PK",
	35	+ "panama": "PA", "paraguay": "PY", "peru": "PE", "perú": "PE", "philippines": "PH", "poland": "PL", "polska": "PL", "portugal": "PT", "qatar": "QA",
	36	+ "romania": "RO", "românia": "RO", "russia": "RU", "russian federation": "RU", "rwanda": "RW", "saudi arabia": "SA", "senegal": "SN", "serbia": "RS",
	37	+ "singapore": "SG", "slovakia": "SK", "slovenia": "SI", "south africa": "ZA", "south korea": "KR", "korea": "KR", "republic of korea": "KR",
	38	+ "korea, republic of": "KR", "spain": "ES", "españa": "ES", "sri lanka": "LK", "sweden": "SE", "sverige": "SE", "switzerland": "CH", "schweiz": "CH",
	39	+ "suisse": "CH", "taiwan": "TW", "tanzania": "TZ", "thailand": "TH", "tunisia": "TN", "turkey": "TR", "türkiye": "TR", "turkiye": "TR",
	40	+ "uganda": "UG", "ukraine": "UA", "united arab emirates": "AE", "uae": "AE", "united kingdom": "GB", "uk": "GB", "u.k.": "GB", "great britain": "GB",
	41	+ "britain": "GB", "england": "GB", "scotland": "GB", "wales": "GB", "northern ireland": "GB", "united states": "US", "united states of america": "US",
	42	+ "usa": "US", "u.s.": "US", "u.s.a.": "US", "us": "US", "america": "US", "uruguay": "UY", "uzbekistan": "UZ", "venezuela": "VE", "vietnam": "VN",
	43	+ "viet nam": "VN", "zambia": "ZM", "zimbabwe": "ZW", "puerto rico": "PR", "european union": None, # type: ignore[dict-item]
	44	+}
	45	+ISO2 = {v for v in _COUNTRIES.values() if v}
	46	+ISO3 = {"USA": "US", "GBR": "GB", "DEU": "DE", "FRA": "FR", "CAN": "CA", "AUS": "AU", "JPN": "JP", "CHN": "CN", "IND": "IN", "BRA": "BR", "ESP": "ES",
	47	+ "ITA": "IT", "NLD": "NL", "SWE": "SE", "CHE": "CH", "SGP": "SG", "IRL": "IE", "MEX": "MX", "KOR": "KR", "POL": "PL", "BEL": "BE", "AUT": "AT",
	48	+ "DNK": "DK", "NOR": "NO", "FIN": "FI", "PRT": "PT", "ISR": "IL", "ARE": "AE", "NZL": "NZ", "ZAF": "ZA", "ARG": "AR", "CZE": "CZ", "HUN": "HU"}
	49	+US_STATES: dict[str, str] = {
	50	+ "alabama": "AL", "alaska": "AK", "arizona": "AZ", "arkansas": "AR", "california": "CA", "colorado": "CO", "connecticut": "CT", "delaware": "DE",
	51	+ "florida": "FL", "georgia": "GA", "hawaii": "HI", "idaho": "ID", "illinois": "IL", "indiana": "IN", "iowa": "IA", "kansas": "KS", "kentucky": "KY",
	52	+ "louisiana": "LA", "maine": "ME", "maryland": "MD", "massachusetts": "MA", "michigan": "MI", "minnesota": "MN", "mississippi": "MS",
	53	+ "missouri": "MO", "montana": "MT", "nebraska": "NE", "nevada": "NV", "new hampshire": "NH", "new jersey": "NJ", "new mexico": "NM",
	54	+ "new york": "NY", "north carolina": "NC", "north dakota": "ND", "ohio": "OH", "oklahoma": "OK", "oregon": "OR", "pennsylvania": "PA",
	55	+ "rhode island": "RI", "south carolina": "SC", "south dakota": "SD", "tennessee": "TN", "texas": "TX", "utah": "UT", "vermont": "VT",
	56	+ "virginia": "VA", "washington": "WA", "west virginia": "WV", "wisconsin": "WI", "wyoming": "WY", "district of columbia": "DC",
	57	+}
	58	+# two-letter US state codes that are NOT also ISO country codes (safe to infer US from "City, ST")
	59	+US_STATE_CODES_SAFE = {c for c in US_STATES.values()} - {"CA", "CO", "DE", "IN", "ME", "MD", "MT", "NE", "MA", "MO", "MN", "NC", "SC", "LA", "ID",
	60	+ "IL", "GA", "AL", "AR", "AZ", "KY", "PA", "TN", "VA", "MS"}
	61	+CA_PROVINCES: dict[str, str] = {"ontario": "ON", "quebec": "QC", "québec": "QC", "british columbia": "BC", "alberta": "AB", "manitoba": "MB",
	62	+ "saskatchewan": "SK", "nova scotia": "NS", "new brunswick": "NB", "newfoundland and labrador": "NL",
	63	+ "prince edward island": "PE"}
	64	+CA_PROVINCE_CODES_SAFE = {"ON", "QC", "BC", "AB", "MB", "SK", "NS", "NB"}
	65	+# two-letter tokens that are both a US state/CA province code and an ISO country code: never resolved to a country on their own
	66	+AMBIGUOUS_CODES = (set(US_STATES.values()) \| set(CA_PROVINCES.values())) & ISO2
	67	+REMOTE_RE = re.compile(r"\b(remote\|work from home\|wfh\|anywhere\|distributed\|télétravail\|homeoffice\|home office)\b", re.IGNORECASE)
	68	+HYBRID_RE = re.compile(r"\bhybrid\b", re.IGNORECASE)
	69	+POSTAL_RE = re.compile(r"\b[A-Z]{1,2}\d[A-Z\d]?\s\d[A-Z]{2}\b\|\b\d{4,6}(?:-\d{4})?\b\|\b[A-Z]\d[A-Z]\s\d[A-Z]\d\b")
	70	+
	71	+
	72	+def country_code(text: str \| None) -> str \| None:
	73	+ """ISO-2 for a country name / ISO-2 / ISO-3 token — None unless confident."""
	74	+ if not text:
	75	+ return None
	76	+ t = normalize_whitespace(text).strip(" .,;()").lower()
	77	+ if not t:
	78	+ return None
	79	+ if t in _COUNTRIES:
	80	+ return _COUNTRIES[t]
	81	+ up = t.upper()
	82	+ if len(up) == 2 and up in ISO2:
	83	+ return up
	84	+ if len(up) == 3 and up in ISO3:
	85	+ return ISO3[up]
	86	+ return None
	87	+
	88	+
	89	+def _unambiguous_country(token: str) -> str \| None:
	90	+ t = token.strip()
	91	+ if len(t) == 2 and t.upper() in AMBIGUOUS_CODES:
	92	+ return None
	93	+ return country_code(t)
	94	+
	95	+
	96	+def norm_name(text: str) -> str:
	97	+ """Entity key for people/products/locations (`name_norm`): ASCII, lowercase, single spaces, punctuation dropped."""
	98	+ from slugify import slugify as _slug
	99	+
	100	+ return _slug(text or "", lowercase=True, regex_pattern=r"[^a-z0-9]+").replace("-", " ").strip()[:120]
	101	+
	102	+
	103	+def parse_location(text: str \| None) -> dict[str, Any]:
	104	+ """'San Francisco, CA, USA' → {city, region, country, remote}. Only sets what the text states; never guesses a country."""
	105	+ out: dict[str, Any] = {"city": None, "region": None, "country": None, "remote": None}
	106	+ if not text:
	107	+ return out
	108	+ raw = normalize_whitespace(text)
	109	+ if REMOTE_RE.search(raw):
	110	+ out["remote"] = True
	111	+ elif HYBRID_RE.search(raw):
	112	+ out["remote"] = False
	113	+ cleaned = re.sub(r"$(?:remote\|hybrid\|on-?site)[^)]*$", "", raw, flags=re.IGNORECASE)
	114	+ cleaned = re.sub(r"\b(remote\|hybrid\|on-?site\|flexible\|multiple locations\|or)\b\s[-–—,/]?\s", "", cleaned, flags=re.IGNORECASE)
	115	+ cleaned = POSTAL_RE.sub(" ", cleaned).strip(" ,-–/")
	116	+ parts = [p.strip() for p in re.split(r"\s[,\|•·/;]\s\|\s+[-–—]\s+", cleaned) if p.strip()]
	117	+ if not parts:
	118	+ return out
	119	+ if len(parts) >= 2 and _unambiguous_country(parts[0]) and not _unambiguous_country(parts[-1]):
	120	+ parts.reverse() # Workday style "US, MA, Westford" → city last → first
	121	+ country = _unambiguous_country(parts[-1])
	122	+ if country:
	123	+ parts = parts[:-1]
	124	+ elif len(parts) == 1:
	125	+ # a bare token: a country name ("Germany") is a country; anything else is left alone (a city without a country is not guessed)
	126	+ return out
	127	+ out["country"] = country
	128	+ if parts:
	129	+ last = parts[-1]
	130	+ low = last.lower()
	131	+ two = last.upper() if len(last) == 2 and last.isalpha() else None
	132	+ state = US_STATES.get(low) or (two if two in US_STATE_CODES_SAFE else None)
	133	+ prov = CA_PROVINCES.get(low) or (two if two in CA_PROVINCE_CODES_SAFE else None)
	134	+ if state and (country in (None, "US")):
	135	+ out["region"], out["country"] = state, "US"
	136	+ parts = parts[:-1]
	137	+ elif prov and (country in (None, "CA")):
	138	+ out["region"], out["country"] = prov, "CA"
	139	+ parts = parts[:-1]
	140	+ elif two and two in AMBIGUOUS_CODES and len(parts) >= 2:
	141	+ out["region"] = two # "San Francisco, CA": region kept, country deliberately unknown
	142	+ parts = parts[:-1]
	143	+ elif len(parts) >= 2:
	144	+ out["region"] = last[:80]
	145	+ parts = parts[:-1]
	146	+ if parts:
	147	+ city = parts[0]
	148	+ if len(city) <= 80 and not re.search(r"\d{3,}", city):
	149	+ out["city"] = city
	150	+ return out
	151	+
	152	+
	153	+# ------------------------------------------------------------------------------------------------------------ dates
	154	+
	155	+DATE_IN_TEXT_RE = re.compile(
	156	+ r"(\d{4}-\d{2}-\d{2}(?:[T ]\d{2}:\d{2}(?::\d{2})?(?:\.\d+)?(?:Z\|[+-]\d{2}:?\d{2})?)?)\|"
	157	+ r"((?:jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|sept\|oct\|nov\|dec)[a-z]*\.?\s+\d{1,2}(?:st\|nd\|rd\|th)?,?\s+\d{4})\|"
	158	+ r"(\d{1,2}(?:st\|nd\|rd\|th)?\s+(?:jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|sept\|oct\|nov\|dec)[a-z]*\.?,?\s+\d{4})\|"
	159	+ r"(\d{1,2}[/.]\d{1,2}[/.]\d{4})", re.IGNORECASE)
	160	+DATE_IN_URL_RE = re.compile(r"/((?:19\|20)\d{2})/(0?[1-9]\|1[0-2])(?:/(0?[1-9]\|[12]\d\|3[01]))?(?:/\|$)")
	161	+
	162	+
	163	+def parse_date(value: Any) -> datetime \| None:
	164	+ """Best-effort aware datetime (UTC) from ISO strings, epoch seconds/ms, common textual dates. None when unparseable."""
	165	+ if value is None or value == "":
	166	+ return None
	167	+ if isinstance(value, datetime):
	168	+ return value if value.tzinfo else value.replace(tzinfo=UTC)
	169	+ if isinstance(value, int \| float):
	170	+ v = float(value)
	171	+ if v > 1e12:
	172	+ v /= 1000.0
	173	+ if 0 < v < 4102444800:
	174	+ return datetime.fromtimestamp(v, tz=UTC)
	175	+ return None
	176	+ s = str(value).strip()
	177	+ if not s or len(s) > 60:
	178	+ return None
	179	+ if s.isdigit():
	180	+ return parse_date(int(s))
	181	+ try:
	182	+ dt = dtparser.parse(s, fuzzy=False, dayfirst=False)
	183	+ except (ValueError, OverflowError, TypeError):
	184	+ try:
	185	+ dt = dtparser.parse(s, fuzzy=True)
	186	+ except (ValueError, OverflowError, TypeError):
	187	+ return None
	188	+ if dt.year < 1990 or dt.year > 2100:
	189	+ return None
	190	+ return dt if dt.tzinfo else dt.replace(tzinfo=UTC)
	191	+
	192	+
	193	+def date_from_text(text: str \| None) -> datetime \| None:
	194	+ if not text:
	195	+ return None
	196	+ m = DATE_IN_TEXT_RE.search(text)
	197	+ return parse_date(m.group(0)) if m else None
	198	+
	199	+
	200	+def date_from_url(url: str \| None) -> datetime \| None:
	201	+ if not url:
	202	+ return None
	203	+ m = DATE_IN_URL_RE.search(urlparse(url).path)
	204	+ if not m:
	205	+ return None
	206	+ y, mo, d = int(m.group(1)), int(m.group(2)), int(m.group(3) or 1)
	207	+ try:
	208	+ return datetime(y, mo, d, tzinfo=UTC)
	209	+ except ValueError:
	210	+ return None
	211	+
	212	+
	213	+# ------------------------------------------------------------------------------------------------------------ jobs
	214	+
	215	+SENIORITY_RULES: list[tuple[str, re.Pattern[str]]] = [
	216	+ ("intern", re.compile(r"\b(intern(ship)?\|stagiaire\|werkstudent\|working student\|apprentice\|co-op)\b", re.IGNORECASE)),
	217	+ ("c_level", re.compile(r"\b(chief\b\|\bc[a-z]o\b\|vp\b\|vice president\|evp\|svp\|head of\|director)", re.IGNORECASE)),
	218	+ ("principal", re.compile(r"\b(principal\|distinguished\|fellow)\b", re.IGNORECASE)),
	219	+ ("staff", re.compile(r"\bstaff\b", re.IGNORECASE)),
	220	+ ("lead", re.compile(r"\b(lead\|manager\|leiter)\b", re.IGNORECASE)),
	221	+ ("senior", re.compile(r"\b(senior\|sr\.?\|iii\|iv)\b", re.IGNORECASE)),
	222	+ ("junior", re.compile(r"\b(junior\|jr\.?\|entry[- ]level\|graduate\|associate\|i)\b", re.IGNORECASE)),
	223	+]
	224	+ENGINEERING_RE = re.compile(r"\b(engineer\|engineering\|developer\|software\|sre\|devops\|architect\|data scientist\|machine learning\|backend\|frontend\|"
	225	+ r"full[- ]stack\|qa\|security\|infrastructure\|platform\|embedded\|firmware\|ios\|android\|ingénieur\|entwickler)\b", re.IGNORECASE)
	226	+EMPLOYMENT_TYPES = {"full-time": "full_time", "full time": "full_time", "fulltime": "full_time", "permanent": "full_time", "part-time": "part_time",
	227	+ "part time": "part_time", "contract": "contract", "contractor": "contract", "temporary": "temporary", "temp": "temporary",
	228	+ "intern": "internship", "internship": "internship", "freelance": "contract", "apprenticeship": "internship", "seasonal": "temporary",
	229	+ "fixed-term": "contract", "fixed term": "contract", "cdi": "full_time", "cdd": "contract"}
	230	+
	231	+
	232	+def seniority_guess(title: str \| None) -> str \| None:
	233	+ if not title:
	234	+ return None
	235	+ for label, pat in SENIORITY_RULES:
	236	+ if pat.search(title):
	237	+ return label
	238	+ return None
	239	+
	240	+
	241	+def employment_type_norm(value: str \| None) -> str \| None:
	242	+ if not value:
	243	+ return None
	244	+ v = normalize_whitespace(str(value)).lower().replace("_", "-")
	245	+ if v in EMPLOYMENT_TYPES:
	246	+ return EMPLOYMENT_TYPES[v]
	247	+ for k, out in EMPLOYMENT_TYPES.items():
	248	+ if k in v:
	249	+ return out
	250	+ return v[:30]
	251	+
	252	+
	253	+def is_engineering(title: str \| None) -> bool:
	254	+ return bool(title and ENGINEERING_RE.search(title))
	255	+
	256	+
	257	+def job_fingerprint(title: str, location_text: str \| None, external_id: str \| None, url: str \| None) -> str:
	258	+ ident = (external_id or "").strip() or (url or "").split("?")[0].rstrip("/")
	259	+ return stable_hash(normalized_text(title), normalized_text(location_text or ""), ident.lower())
	260	+
	261	+
	262	+def strip_html(s: str \| None) -> str:
	263	+ if not s:
	264	+ return ""
	265	+ from selectolax.lexbor import LexborHTMLParser
	266	+
	267	+ try:
	268	+ return normalize_whitespace(LexborHTMLParser(f"<div>{s}</div>").text(separator=" "))
	269	+ except Exception: # noqa: BLE001
	270	+ return normalize_whitespace(re.sub(r"<[^>]+>", " ", s))
	271	+
	272	+
	273	+def description_hash(s: str \| None) -> str \| None:
	274	+ txt = normalized_text(strip_html(s)) if s else ""
	275	+ return hashlib.sha256(txt.encode("utf-8")).hexdigest()[:32] if txt else None
	276	+
	277	+
	278	+def finish_job(job: ExtractedJob) -> ExtractedJob:
	279	+ """Fill derived fields (seniority, remote/city/country from location text) without overriding vendor-provided values."""
	280	+ job.title = normalize_whitespace(job.title)[:300]
	281	+ if job.location_text:
	282	+ loc = parse_location(job.location_text)
	283	+ job.city = job.city or loc["city"]
	284	+ job.region = job.region or loc["region"]
	285	+ job.country = job.country or loc["country"]
	286	+ if job.remote is None:
	287	+ job.remote = loc["remote"]
	288	+ if job.remote is None and REMOTE_RE.search(job.title):
	289	+ job.remote = True
	290	+ job.seniority = job.seniority or seniority_guess(job.title)
	291	+ job.employment_type = employment_type_norm(job.employment_type)
	292	+ if job.country:
	293	+ job.country = job.country.upper()[:2]
	294	+ return job
	295	+
	296	+
	297	+def job_blocks(jobs: list[ExtractedJob], *, path: str = "Jobs") -> list[Block]:
	298	+ """One stable block per job (identity = external id / url), so the block diff mirrors the job delta exactly."""
	299	+ blocks: list[Block] = []
	300	+ seen: set[str] = set()
	301	+ for i, j in enumerate(jobs):
	302	+ ident = (j.external_id or j.url or job_fingerprint(j.title, j.location_text, None, None))
	303	+ key = f"job_listing:{hashlib.blake2b(str(ident).encode('utf-8'), digest_size=6).hexdigest()}"
	304	+ if key in seen:
	305	+ key = f"{key}#{i}"
	306	+ seen.add(key)
	307	+ text = " — ".join(x for x in (j.title, j.location_text, j.department) if x)
	308	+ blocks.append(Block(key=key, kind="job_listing", text=text, path=path, hash=hashlib.sha256(normalized_text(text).encode()).hexdigest()[:16],
	309	+ simhash=simhash(text), weight=BLOCK_WEIGHTS["job_listing"], order=i, attrs={"url": j.url, "external_id": j.external_id}))
	310	+ return blocks
	311	+
	312	+
	313	+def jobs_text(jobs: list[ExtractedJob], header: str) -> str:
	314	+ lines = [header] + [" — ".join(x for x in (j.title, j.location_text, j.department) if x) for j in jobs]
	315	+ return "\n".join(lines)
	316	+
	317	+
	318	+# ------------------------------------------------------------------------------------------------------------ JSON helpers
	319	+
	320	+
	321	+def load_json(result: FetchResult) -> Any:
	322	+ try:
	323	+ return result.json()
	324	+ except Exception as exc:
	325	+ raise ValueError(f"invalid JSON from {result.final_url}: {exc.__class__.__name__}") from exc
	326	+
	327	+
	328	+def dig(obj: Any, *path: str, default: Any = None) -> Any:
	329	+ cur = obj
	330	+ for p in path:
	331	+ if isinstance(cur, dict):
	332	+ cur = cur.get(p)
	333	+ else:
	334	+ return default
	335	+ if cur is None:
	336	+ return default
	337	+ return cur
	338	+
	339	+
	340	+def merged_result(first: FetchResult, payload: Any, *, pages: int) -> FetchResult:
	341	+ """Synthetic FetchResult holding the merged pages of a paginated public API (what we archive is what we saw)."""
	342	+ content = json.dumps(payload, ensure_ascii=False).encode("utf-8")
	343	+ res = FetchResult(url=first.url, final_url=first.final_url, status=first.status, headers=dict(first.headers), content=content,
	344	+ content_type="application/json; charset=utf-8", fetched_at=first.fetched_at, duration_ms=first.duration_ms,
	345	+ transport=first.transport, redirects=first.redirects)
	346	+ res.headers["x-companyatlas-pages"] = str(pages)
	347	+ return res
	348	+
	349	+
	350	+def text_of(v: Any) -> str \| None:
	351	+ if v is None:
	352	+ return None
	353	+ if isinstance(v, dict):
	354	+ for k in ("name", "label", "text", "title", "value"):
	355	+ if v.get(k):
	356	+ return normalize_whitespace(str(v[k]))
	357	+ return None
	358	+ if isinstance(v, list):
	359	+ return ", ".join(x for x in (text_of(i) for i in v) if x) or None
	360	+ return normalize_whitespace(str(v)) or None
	361	+
	362	+
	363	+# ------------------------------------------------------------------------------------------------------------ ATS vendor map
	364	+
	365	+WORKDAY_RE = re.compile(r"https?://([a-z0-9-]+)\.(wd\d+)\.myworkdayjobs\.com/(?:([a-z]{2}-[A-Z]{2})/)?([A-Za-z0-9_-]+)", re.IGNORECASE)
	366	+
	367	+
	368	+def ats_sensor_spec(vendor: str, token: str, board_url: str) -> tuple[str, str, dict[str, Any]] \| None:
	369	+ """(api_url, connector_id, config) for a detected ATS board — the sensor URL is the public endpoint the board's page uses."""
	370	+ v = vendor.lower()
	371	+ cfg: dict[str, Any] = {"vendor": v, "token": token, "board_url": board_url}
	372	+ if v == "greenhouse":
	373	+ return f"https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=false", "greenhouse-v1", cfg
	374	+ if v == "lever":
	375	+ region = "eu." if "jobs.eu.lever.co" in board_url else ""
	376	+ return f"https://api.{region}lever.co/v0/postings/{token}?mode=json", "lever-v1", cfg
	377	+ if v == "ashby":
	378	+ return f"https://api.ashbyhq.com/posting-api/job-board/{token}", "ashby-v1", cfg
	379	+ if v == "smartrecruiters":
	380	+ return f"https://api.smartrecruiters.com/v1/companies/{token}/postings?limit=100", "smartrecruiters-v1", cfg
	381	+ if v == "workable":
	382	+ return f"https://apply.workable.com/api/v1/widget/accounts/{token}", "workable-v1", cfg
	383	+ if v == "recruitee":
	384	+ return f"https://{token}.recruitee.com/api/offers/", "recruitee-v1", cfg
	385	+ if v == "personio":
	386	+ return f"https://{token}.jobs.personio.de/xml", "personio-v1", cfg
	387	+ if v == "teamtailor":
	388	+ host = urlparse(board_url).hostname or f"{token}.teamtailor.com"
	389	+ return f"https://{host}/jobs.json", "teamtailor-v1", cfg
	390	+ if v == "workday":
	391	+ m = WORKDAY_RE.search(board_url)
	392	+ if not m:
	393	+ return None
	394	+ tenant, wd, _locale, site = m.group(1), m.group(2), m.group(3), m.group(4)
	395	+ if site.lower() in ("wday", "en-us", "job"):
	396	+ return None
	397	+ cfg.update({"tenant": tenant, "wd": wd, "site": site})
	398	+ return f"https://{tenant}.{wd}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs", "workday-v1", cfg
	399	+ return None
	400	+
	401	+
	402	+__all__ = [
	403	+ "CA_PROVINCES",
	404	+ "DATE_IN_URL_RE",
	405	+ "REMOTE_RE",
	406	+ "US_STATES",
	407	+ "WORKDAY_RE",
	408	+ "ats_sensor_spec",
	409	+ "country_code",
	410	+ "date_from_text",
	411	+ "date_from_url",
	412	+ "description_hash",
	413	+ "dig",
	414	+ "employment_type_norm",
	415	+ "finish_job",
	416	+ "is_engineering",
	417	+ "job_blocks",
	418	+ "job_fingerprint",
	419	+ "jobs_text",
	420	+ "load_json",
	421	+ "merged_result",
	422	+ "norm_name",
	423	+ "parse_date",
	424	+ "parse_location",
	425	+ "seniority_guess",
	426	+ "strip_html",
	427	+ "text_of",
	428	+]

added src/companyatlas/connectors/ashby.py +43 −0

		`@@ -0,0 +1,43 @@`
	1	+"""Ashby job board — public endpoint behind `jobs.ashbyhq.com/<token>`:
	2	+`GET https://api.ashbyhq.com/posting-api/job-board/{token}` → {"jobs":[{id, title, department, team, employmentType, location,
	3	+secondaryLocations, publishedAt, isListed, isRemote, workplaceType, address:{postalAddress:{addressLocality, addressRegion, addressCountry}},
	4	+jobUrl, applyUrl, compensation?}]}. Verified 2026-09-12 against the `ashby` board (fixture trimmed to 20 jobs, descriptions removed)."""
	5	+from __future__ import annotations
	6	+
	7	+import re
	8	+from collections.abc import Mapping
	9	+from typing import Any
	10	+
	11	+from companyatlas.connectors._ats_base import AtsConnector
	12	+from companyatlas.connectors._util import country_code, dig, parse_date, text_of
	13	+from companyatlas.sdk.connector import ConnectorMeta, register
	14	+from companyatlas.sdk.models import ExtractedJob
	15	+from companyatlas.taxonomy import FetchMode, Surface
	16	+
	17	+
	18	+@register
	19	+class AshbyConnector(AtsConnector):
	20	+ vendor = "ashby"
	21	+ token_re = re.compile(r"api\.ashbyhq\.com/posting-api/job-board/([a-z0-9_.-]+)", re.IGNORECASE)
	22	+ meta = ConnectorMeta(connector_id="ashby-v1", name="Ashby job board", version="1", category=Surface.JOBS_BOARD, fetch_mode=FetchMode.JSON,
	23	+ default_interval_s=6 * 3600, url_pattern=r"api\.ashbyhq\.com/posting-api/job-board/", priority=50, pattern_required=True, accept="application/json",
	24	+ description="Public Ashby posting API")
	25	+
	26	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	27	+ out: list[ExtractedJob] = []
	28	+ for j in (data.get("jobs") if isinstance(data, dict) else []) or []:
	29	+ if not isinstance(j, dict) or not j.get("title") or j.get("isListed") is False:
	30	+ continue
	31	+ addr = dig(j, "address", "postalAddress", default={}) or {}
	32	+ comp = j.get("compensation") or {}
	33	+ summary = comp.get("compensationTierSummary") if isinstance(comp, dict) else None
	34	+ out.append(ExtractedJob(
	35	+ title=str(j["title"]), url=j.get("jobUrl") or j.get("applyUrl"), external_id=str(j.get("id") or ""),
	36	+ department=text_of(j.get("department")), team=text_of(j.get("team")), location_text=text_of(j.get("location")),
	37	+ city=text_of(addr.get("addressLocality")), region=text_of(addr.get("addressRegion")) or None,
	38	+ country=country_code(addr.get("addressCountry")), remote=bool(j.get("isRemote")) if j.get("isRemote") is not None else None,
	39	+ employment_type=text_of(j.get("employmentType")), posted_at=parse_date(j.get("publishedAt")),
	40	+ raw={"workplaceType": j.get("workplaceType"), "secondaryLocations": [text_of(s.get("location")) for s in (j.get("secondaryLocations") or [])
	41	+ if isinstance(s, dict)][:5], "compensation": summary},
	42	+ ))
	43	+ return out

added src/companyatlas/connectors/feed.py +91 −0

		`@@ -0,0 +1,91 @@`
	1	+"""RSS / Atom / JSON Feed connector → news items (title, url, published_at, summary, language). One block per entry keyed by the
	2	+entry's canonical URL / id so the block diff equals the entry delta. Uses feedparser (bounded input)."""
	3	+from __future__ import annotations
	4	+
	5	+import hashlib
	6	+from collections.abc import Mapping
	7	+from typing import Any
	8	+
	9	+import feedparser
	10	+
	11	+from companyatlas.connectors._util import parse_date, strip_html
	12	+from companyatlas.fetch import FetchResult
	13	+from companyatlas.sdk.connector import Connector, ConnectorMeta, register
	14	+from companyatlas.sdk.models import Block, ExtractedNewsItem, Extraction
	15	+from companyatlas.sdk.normalize import language_guess, normalized_text, simhash
	16	+from companyatlas.taxonomy import FetchMode, Surface
	17	+from companyatlas.urls import canonicalize_url
	18	+
	19	+MAX_ENTRIES = 200
	20	+CATEGORY_BY_SURFACE = {Surface.NEWSROOM: "press", Surface.BLOG: "blog", Surface.CHANGELOG: "changelog", Surface.RESEARCH: "research",
	21	+ Surface.INVESTOR_RELATIONS: "ir", Surface.FEED: "blog"}
	22	+
	23	+
	24	+def _json_feed_items(data: dict[str, Any]) -> list[ExtractedNewsItem]:
	25	+ out: list[ExtractedNewsItem] = []
	26	+ for it in (data.get("items") or [])[:MAX_ENTRIES]:
	27	+ if not isinstance(it, dict) or not it.get("title") or not it.get("url"):
	28	+ continue
	29	+ summary = strip_html(it.get("summary") or it.get("content_text") or it.get("content_html") or "")[:600] or None
	30	+ out.append(ExtractedNewsItem(title=str(it["title"])[:300], url=str(it["url"]), published_at=parse_date(it.get("date_published")),
	31	+ summary=summary, language=(it.get("language") or data.get("language") or None)))
	32	+ return out
	33	+
	34	+
	35	+@register
	36	+class FeedConnector(Connector):
	37	+ meta = ConnectorMeta(connector_id="feed-v1", name="RSS / Atom / JSON feed", version="1", category=Surface.FEED, fetch_mode=FetchMode.FEED,
	38	+ default_interval_s=2 * 3600, surfaces=(), # newsroom/blog/changelog HTML pages stay with generic-html; feeds win by URL or surface
	39	+ url_pattern=r"(/(feed\|rss\|atom\|feeds)(\.xml\|\.json\|/\|$)\|\.(rss\|atom)$\|/rss\.xml\|/feed\.xml\|/atom\.xml\|/index\.xml$\|feed\.json$)",
	40	+ priority=45, accept="application/rss+xml,application/atom+xml,application/feed+json,application/xml,text/xml;q=0.9,/;q=0.5",
	41	+ description="Syndication feeds → news items")
	42	+
	43	+ def extract(self, sensor: Mapping[str, Any], result: FetchResult) -> Extraction:
	44	+ surface = str(sensor.get("surface") or Surface.FEED)
	45	+ category = (sensor.get("config") or {}).get("category") or CATEGORY_BY_SURFACE.get(surface, "other") # type: ignore[arg-type]
	46	+ items: list[ExtractedNewsItem] = []
	47	+ title: str \| None = None
	48	+ lang: str \| None = None
	49	+ if result.is_json:
	50	+ data = result.json()
	51	+ if isinstance(data, dict):
	52	+ items = _json_feed_items(data)
	53	+ title = data.get("title")
	54	+ lang = data.get("language")
	55	+ else:
	56	+ parsed = feedparser.parse(result.content[: 8 * 1024 * 1024])
	57	+ if parsed.bozo and not parsed.entries:
	58	+ raise ValueError(f"unparseable feed: {getattr(parsed, 'bozo_exception', 'unknown error')}")
	59	+ title = parsed.feed.get("title")
	60	+ lang = parsed.feed.get("language")
	61	+ for e in parsed.entries[:MAX_ENTRIES]:
	62	+ link = e.get("link") or next((ln.get("href") for ln in e.get("links", []) if ln.get("href")), None)
	63	+ etitle = (e.get("title") or "").strip()
	64	+ if not link or not etitle:
	65	+ continue
	66	+ published = e.get("published") or e.get("updated") or e.get("created")
	67	+ summary = strip_html(e.get("summary") or (e.get("content") or [{}])[0].get("value", ""))[:600] or None
	68	+ items.append(ExtractedNewsItem(title=etitle[:300], url=link, published_at=parse_date(published), summary=summary,
	69	+ language=(e.get("language") or lang or None)))
	70	+ for it in items:
	71	+ it.category = category
	72	+ if not it.language:
	73	+ it.language = language_guess(f"{it.title} {it.summary or ''}")
	74	+ blocks: list[Block] = []
	75	+ seen: set[str] = set()
	76	+ lines: list[str] = []
	77	+ for i, it in enumerate(items):
	78	+ canon = canonicalize_url(it.url)
	79	+ if canon in seen:
	80	+ continue
	81	+ seen.add(canon)
	82	+ text = f"{it.title}\n{it.summary or ''}".strip()
	83	+ lines.append(it.title)
	84	+ blocks.append(Block(key=f"entry:{hashlib.blake2b(canon.encode('utf-8'), digest_size=8).hexdigest()}", kind="news_item", text=text, path=title or "Feed",
	85	+ hash=hashlib.sha256(normalized_text(text).encode()).hexdigest()[:16], simhash=simhash(text), weight=1.2, order=i,
	86	+ attrs={"url": canon, "published_at": it.published_at.isoformat() if it.published_at else None}))
	87	+ meta = {"feed_title": title, "entry_count": len(items), "category": category, "structured": True}
	88	+ return Extraction(text="\n".join(lines), blocks=blocks, title=title, language=(lang or "").split("-")[0] or None, meta=meta, news=items)
	89	+
	90	+
	91	+__all__ = ["FeedConnector"]

added src/companyatlas/connectors/generic_html.py +607 −0

		`@@ -0,0 +1,607 @@`
	1	+"""Generic HTML connector (spec §9.1, §107): one connector for every corporate web surface. `sdk.normalize` supplies text, blocks,
	2	+metadata, JSON-LD and links; this module adds surface-aware typed extraction:
	3	+
	4	+ leadership → people (role category, is_executive) pricing → plans (currency, period, unit, contact-sales, features)
	5	+ products/services/solutions → product cards locations → offices/stores (city / country only when stated)
	6	+ newsroom/blog/press/changelog/research/IR → news items careers → job listings (list / table / card patterns with job-like anchors)
	7	+ legal/docs/homepage/about/… → text + blocks only
	8	+
	9	+Structured data (JSON-LD / microdata) is used first, DOM heuristics second, and nothing is ever invented: no address, no country, no
	10	+date that the page does not state. Every surface also yields `discovered` URLs (classified links) for the discovery feedback loop.
	11	+"""
	12	+from __future__ import annotations
	13	+
	14	+import re
	15	+from collections.abc import Mapping
	16	+from typing import Any
	17	+from urllib.parse import urlparse
	18	+
	19	+from selectolax.lexbor import LexborHTMLParser
	20	+
	21	+from companyatlas.config import settings
	22	+from companyatlas.connectors._util import (
	23	+ country_code,
	24	+ date_from_text,
	25	+ date_from_url,
	26	+ finish_job,
	27	+ job_blocks,
	28	+ norm_name,
	29	+ parse_date,
	30	+ parse_location,
	31	+ text_of,
	32	+)
	33	+from companyatlas.connectors.jsonld_jobs import jobs_from_jsonld
	34	+from companyatlas.fetch import FetchResult
	35	+from companyatlas.sdk import normalize
	36	+from companyatlas.sdk.connector import Connector, ConnectorMeta, register
	37	+from companyatlas.sdk.models import (
	38	+ Block,
	39	+ DiscoveredUrl,
	40	+ ExtractedJob,
	41	+ ExtractedLocation,
	42	+ ExtractedNewsItem,
	43	+ ExtractedPerson,
	44	+ ExtractedPlan,
	45	+ ExtractedProduct,
	46	+ Extraction,
	47	+)
	48	+from companyatlas.sdk.normalize import NormalizedPage, normalize_whitespace
	49	+from companyatlas.taxonomy import FetchMode, Surface
	50	+from companyatlas.urls import absolutize, canonicalize_url, classify_url, is_static_asset, looks_like_trap, registrable_domain
	51	+
	52	+MAX_PEOPLE, MAX_PLANS, MAX_LOCATIONS, MAX_NEWS, MAX_JOBS, MAX_PRODUCTS, MAX_DISCOVERED = 200, 16, 300, 80, 300, 120, 120
	53	+NEWS_SURFACES = {Surface.NEWSROOM, Surface.BLOG, Surface.CHANGELOG, Surface.RESEARCH, Surface.INVESTOR_RELATIONS}
	54	+NEWS_CATEGORY = {Surface.NEWSROOM: "press", Surface.BLOG: "blog", Surface.CHANGELOG: "changelog", Surface.RESEARCH: "research",
	55	+ Surface.INVESTOR_RELATIONS: "ir"}
	56	+PRODUCT_SURFACES = {Surface.PRODUCTS, Surface.SERVICES, Surface.SOLUTIONS}
	57	+
	58	+# ------------------------------------------------------------------------------------------------------------ people
	59	+
	60	+ROLE_RULES: list[tuple[str, re.Pattern[str]]] = [
	61	+ ("founder", re.compile(r"\b(co-?founder\|founder\|fondat(eur\|rice)\|gründer(in)?)\b", re.IGNORECASE)),
	62	+ ("ceo", re.compile(r"\b(chief executive( officer)?\|ceo\|pdg\|président-directeur\|geschäftsführer(in)?\|managing director\|directeur général)\b", re.IGNORECASE)),
	63	+ ("cfo", re.compile(r"\b(chief financial( officer)?\|cfo\|finanzvorstand\|directeur financier)\b", re.IGNORECASE)),
	64	+ ("cto", re.compile(r"\b(chief technology( officer)?\|chief technical( officer)?\|cto)\b", re.IGNORECASE)),
	65	+ ("coo", re.compile(r"\b(chief operating( officer)?\|coo)\b", re.IGNORECASE)),
	66	+ ("chair", re.compile(r"\b(chair(man\|woman\|person)?\|executive chair\|présidente? du conseil\|vorsitzende[rn]?)\b", re.IGNORECASE)),
	67	+ ("president", re.compile(r"\b(?<!vice )(?<!vice-)president(?! of)\|président(?!e? du conseil)\b", re.IGNORECASE)),
	68	+ ("board", re.compile(r"\b(board member\|member of the (supervisory \|advisory )?board\|non-executive director\|independent director\|"
	69	+ r"administrat(eur\|rice)\|aufsichtsrat\|conseil d'administration\|director(?= \(board)\|trustee)\b", re.IGNORECASE)),
	70	+ ("vp", re.compile(r"\b(vice[- ]president\|vp\|evp\|svp\|avp)\b", re.IGNORECASE)),
	71	+ ("head", re.compile(r"\b(head of\|head,\|general manager\|gm\|leiter(in)?\|directeur\|directrice\|director\|managing partner\|partner)\b", re.IGNORECASE)),
	72	+]
	73	+CHIEF_RE = re.compile(r"\b(chief\b\|c[a-z]{1,3}o\b)", re.IGNORECASE)
	74	+EXEC_CATEGORIES = {"ceo", "cfo", "cto", "coo", "founder", "president", "chair"}
	75	+NAME_TOKEN = r"(?:[A-ZÀ-ÝĀ-Ž](?:['’][A-ZÀ-ÝĀ-Ž])?[a-zà-ÿā-ž'’]+(?:-[A-ZÀ-ÝĀ-Ža-zà-ÿā-ž][a-zà-ÿā-ž'’])\|[A-ZÀ-ÝĀ-Ž]\.\|(?:de\|van\|von\|der\|den\|da\|di\|le\|la\|du\|del\|bin\|al)\b)"
	76	+NAME_RE = re.compile(rf"^(?:(?:Dr\|Prof\|Mr\|Mrs\|Ms\|Sir\|Dame\|Hon)\.?\s+)?{NAME_TOKEN}(?:\s+{NAME_TOKEN}){{1,4}}(?:,?\s+(?:Jr\.?\|Sr\.?\|II\|III\|IV\|PhD\|MD\|MBA\|CPA\|Esq\.?))?$")
	77	+NOT_NAME_RE = re.compile(r"\b(team\|our\|meet\|leadership\|board\|executive\|management\|contact\|about\|officer\|director\|president\|manager\|head\|chief\|"
	78	+ r"founder\|partner\|group\|company\|global\|senior\|vice\|read\|more\|learn\|view\|profile\|bio\|linkedin\|email\|join\|careers\|"
	79	+ r"news\|press\|the\|and\|of\|for\|at\|in\|on\|to\|with)\b", re.IGNORECASE)
	80	+
	81	+
	82	+def role_category(title: str \| None) -> tuple[str, bool]:
	83	+ if not title:
	84	+ return "other", False
	85	+ for cat, pat in ROLE_RULES:
	86	+ if pat.search(title):
	87	+ return cat, cat in EXEC_CATEGORIES or bool(CHIEF_RE.search(title))
	88	+ return "other", bool(CHIEF_RE.search(title))
	89	+
	90	+
	91	+def looks_like_name(text: str) -> bool:
	92	+ t = normalize_whitespace(text).strip(" .,:;-–—\|")
	93	+ if not (4 <= len(t) <= 60) or any(ch.isdigit() for ch in t):
	94	+ return False
	95	+ if NOT_NAME_RE.search(t):
	96	+ return False
	97	+ return bool(NAME_RE.match(t))
	98	+
	99	+
	100	+def looks_like_title(text: str) -> bool:
	101	+ t = normalize_whitespace(text)
	102	+ if not (2 <= len(t) <= 140):
	103	+ return False
	104	+ return role_category(t)[0] != "other" or bool(re.search(r"\b(officer\|manager\|engineer\|lead\|counsel\|scientist\|architect\|analyst\|controller\|"
	105	+ r"secretary\|treasurer\|advisor\|adviser\|strategist\|evangelist\|designer)\b", t, re.IGNORECASE))
	106	+
	107	+
	108	+def extract_people(page: NormalizedPage) -> list[ExtractedPerson]:
	109	+ out: list[ExtractedPerson] = []
	110	+ seen: set[str] = set()
	111	+
	112	+ def add(name: str, title: str \| None, url: str \| None = None) -> None:
	113	+ name = normalize_whitespace(name).strip(" .,:;-–—\|")
	114	+ key = norm_name(name)
	115	+ if not key or key in seen or len(out) >= MAX_PEOPLE:
	116	+ return
	117	+ seen.add(key)
	118	+ cat, is_exec = role_category(title)
	119	+ out.append(ExtractedPerson(name=name[:120], title=(normalize_whitespace(title)[:160] if title else None), role_category=cat, is_executive=is_exec, url=url))
	120	+
	121	+ for p in page.jsonld.get("persons", []) + page.microdata.get("persons", []):
	122	+ name = text_of(p.get("name"))
	123	+ if name and looks_like_name(name):
	124	+ add(name, text_of(p.get("jobTitle")) or text_of(p.get("title")), text_of(p.get("url")) if isinstance(p.get("url"), str) else None)
	125	+ for b in page.blocks:
	126	+ if b.kind != "person":
	127	+ continue
	128	+ lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
	129	+ for i, ln in enumerate(lines[:4]):
	130	+ if looks_like_name(ln):
	131	+ title = next((x for x in lines[i + 1:i + 4] if not looks_like_name(x) and len(x) <= 140), None)
	132	+ add(ln, title, b.attrs.get("href"))
	133	+ break
	134	+ if len(out) < 2: # fallback: heading = name, next short block = title
	135	+ blocks = page.blocks
	136	+ for i, b in enumerate(blocks):
	137	+ if b.kind == "heading" and int(b.attrs.get("level", 3)) >= 2 and looks_like_name(b.text):
	138	+ nxt = next((x for x in blocks[i + 1:i + 3] if x.kind in ("paragraph", "other", "list", "section")), None)
	139	+ if nxt is not None and looks_like_title(nxt.text.split("\n")[0]):
	140	+ add(b.text, nxt.text.split("\n")[0])
	141	+ return out
	142	+
	143	+
	144	+# ------------------------------------------------------------------------------------------------------------ pricing
	145	+
	146	+CURRENCY_SYMBOLS = {"$": "USD", "US$": "USD", "USD": "USD", "€": "EUR", "EUR": "EUR", "£": "GBP", "GBP": "GBP", "¥": "JPY", "JPY": "JPY", "₹": "INR",
	147	+ "INR": "INR", "C$": "CAD", "CA$": "CAD", "CAD": "CAD", "A$": "AUD", "AUD": "AUD", "CHF": "CHF", "SEK": "SEK", "NOK": "NOK",
	148	+ "DKK": "DKK", "kr": "SEK", "zł": "PLN", "PLN": "PLN", "R$": "BRL", "BRL": "BRL", "MX$": "MXN", "₩": "KRW", "SGD": "SGD", "S$": "SGD",
	149	+ "HK$": "HKD", "NZ$": "NZD", "₺": "TRY", "元": "CNY", "CNY": "CNY", "RMB": "CNY"}
	150	+PRICE_RE = re.compile(r"(?:(?P<cur>US\$\|CA\$\|C\$\|A\$\|NZ\$\|HK\$\|S\$\|MX\$\|R\$\|USD\|EUR\|GBP\|CAD\|AUD\|CHF\|JPY\|INR\|SEK\|NOK\|DKK\|PLN\|BRL\|CNY\|RMB\|SGD\|[$€£¥₹₩₺])\s?"
	151	+ r"(?P<amt>\d{1,3}(?:[,.\s]\d{3})*(?:[.,]\d{1,2})?\|\d+(?:[.,]\d{1,2})?))\|"
	152	+ r"(?:(?P<amt2>\d{1,3}(?:[,.\s]\d{3})*(?:[.,]\d{1,2})?\|\d+(?:[.,]\d{1,2})?)\s?(?P<cur2>€\|£\|USD\|EUR\|GBP\|CHF\|kr\|zł\|元\|\$))")
	153	+PERIOD_RE = re.compile(r"(?:(?:per\|/\|a\|each\|every)\s*(?P<unit>month\|mo\|year\|yr\|annum\|week\|wk\|day\|hour\|hr\|user\|seat\|member\|license\|licence\|agent\|editor\|"
	154	+ r"contact\|1,?000\|1k\|GB\|TB\|request\|transaction\|call\|minute\|mois\|an\|année\|monat\|jahr)\b)\|(?P<word>monthly\|annually\|yearly\|per annum\|"
	155	+ r"billed (?:monthly\|annually\|yearly)\|mensuel\|annuel\|monatlich\|jährlich\|one[- ]time\|lifetime)", re.IGNORECASE)
	156	+CONTACT_RE = re.compile(r"\b(contact (us\|sales)\|talk to (us\|sales\|an expert)\|custom pricing\|custom quote\|get a quote\|request (a )?(quote\|demo)\|let'?s talk\|"
	157	+ r"on request\|upon request\|sur devis\|nous contacter\|contactez-nous\|auf anfrage\|individuell\|tailored\|bespoke\|call us)\b", re.IGNORECASE)
	158	+FREE_RE = re.compile(r"^(free\|gratuit\|kostenlos\|gratis\|\$\s?0(\.00)?\|0\s?€\|€\s?0)\b", re.IGNORECASE)
	159	+FROM_RE = re.compile(r"\b(from\|starting at\|starts at\|as low as\|à partir de\|ab)\b", re.IGNORECASE)
	160	+MONTH_UNITS = {"month", "mo", "monthly", "mensuel", "monatlich", "mois", "monat"}
	161	+YEAR_UNITS = {"year", "yr", "annum", "annually", "yearly", "per annum", "annuel", "jährlich", "an", "année", "jahr"}
	162	+SEAT_UNITS = {"user", "seat", "member", "license", "licence", "agent", "editor", "contact"}
	163	+USAGE_UNITS = {"1,000", "1000", "1k", "gb", "tb", "request", "transaction", "call", "minute", "hour", "hr", "day", "week", "wk"}
	164	+
	165	+
	166	+def parse_price(text: str) -> dict[str, Any] \| None:
	167	+ """→ {price, currency, billing_period, unit, price_text, contact_sales} or None when the text states no price."""
	168	+ t = normalize_whitespace(text)
	169	+ if CONTACT_RE.search(t) and not PRICE_RE.search(t):
	170	+ return {"price": None, "currency": None, "billing_period": "contact", "unit": None, "price_text": t[:80], "contact_sales": True}
	171	+ m = PRICE_RE.search(t)
	172	+ if m is None:
	173	+ if FREE_RE.match(t):
	174	+ return {"price": 0.0, "currency": None, "billing_period": None, "unit": None, "price_text": t[:80], "contact_sales": False}
	175	+ return None
	176	+ cur = m.group("cur") or m.group("cur2")
	177	+ amt = m.group("amt") or m.group("amt2")
	178	+ amt_clean = amt.replace(" ", "")
	179	+ if re.fullmatch(r"\d{1,3}(,\d{3})+(\.\d{1,2})?", amt_clean):
	180	+ amt_clean = amt_clean.replace(",", "")
	181	+ elif re.fullmatch(r"\d{1,3}(\.\d{3})+(,\d{1,2})?", amt_clean):
	182	+ amt_clean = amt_clean.replace(".", "").replace(",", ".")
	183	+ elif re.fullmatch(r"\d+,\d{1,2}", amt_clean):
	184	+ amt_clean = amt_clean.replace(",", ".")
	185	+ else:
	186	+ amt_clean = amt_clean.replace(",", "")
	187	+ try:
	188	+ price = float(amt_clean)
	189	+ except ValueError:
	190	+ return None
	191	+ tail = t[m.end():m.end() + 60]
	192	+ pm = PERIOD_RE.search(tail) or PERIOD_RE.search(t)
	193	+ period, unit = None, None
	194	+ if pm:
	195	+ u = (pm.group("unit") or pm.group("word") or "").lower()
	196	+ if u in MONTH_UNITS or u.startswith("billed monthly"):
	197	+ period = "month"
	198	+ elif u in YEAR_UNITS or "annual" in u or "yearly" in u:
	199	+ period = "year"
	200	+ elif u in ("one-time", "one time", "lifetime"):
	201	+ period = "one_time"
	202	+ elif u in SEAT_UNITS:
	203	+ unit = u
	204	+ elif u in USAGE_UNITS:
	205	+ period, unit = "usage", u
	206	+ # a second unit ("per user per month")
	207	+ for pm2 in PERIOD_RE.finditer(tail):
	208	+ u2 = (pm2.group("unit") or pm2.group("word") or "").lower()
	209	+ if u2 in SEAT_UNITS and not unit:
	210	+ unit = u2
	211	+ elif (u2 in MONTH_UNITS) and not period:
	212	+ period = "month"
	213	+ elif (u2 in YEAR_UNITS) and not period:
	214	+ period = "year"
	215	+ return {"price": price, "currency": CURRENCY_SYMBOLS.get(cur, cur.upper() if cur and cur.isalpha() else None), "billing_period": period, "unit": unit,
	216	+ "price_text": t[max(0, m.start() - 12):m.end() + 40].strip()[:80], "contact_sales": bool(CONTACT_RE.search(t))}
	217	+
	218	+
	219	+def _plan_from_lines(name: str, lines: list[str]) -> ExtractedPlan \| None:
	220	+ price_info = None
	221	+ features: list[str] = []
	222	+ for ln in lines:
	223	+ if price_info is None:
	224	+ info = parse_price(ln)
	225	+ if info is not None:
	226	+ price_info = info
	227	+ continue
	228	+ if 2 <= len(ln) <= 140 and not parse_price(ln) and len(features) < 25:
	229	+ features.append(ln)
	230	+ if price_info is None:
	231	+ return None
	232	+ return ExtractedPlan(plan_name=name[:80], price=price_info["price"], price_text=price_info["price_text"], currency=price_info["currency"],
	233	+ billing_period=price_info["billing_period"], unit=price_info["unit"], features=features, contact_sales=price_info["contact_sales"])
	234	+
	235	+
	236	+def extract_plans(page: NormalizedPage) -> list[ExtractedPlan]:
	237	+ out: list[ExtractedPlan] = []
	238	+ seen: set[str] = set()
	239	+
	240	+ def add(plan: ExtractedPlan \| None) -> None:
	241	+ if plan is None:
	242	+ return
	243	+ key = norm_name(plan.plan_name)
	244	+ if not key or key in seen or len(out) >= MAX_PLANS:
	245	+ return
	246	+ seen.add(key)
	247	+ out.append(plan)
	248	+
	249	+ for b in page.blocks:
	250	+ if b.kind != "pricing_plan":
	251	+ continue
	252	+ lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
	253	+ if not lines:
	254	+ continue
	255	+ name = lines[0] if parse_price(lines[0]) is None and len(lines[0]) <= 60 else (b.path.split(" > ")[-1] if b.path else "Plan")
	256	+ add(_plan_from_lines(name, lines[1:] if lines[0] == name else lines))
	257	+ if not out: # fallback: heading followed by a price before the next heading (common in hand-rolled pricing tables)
	258	+ blocks = page.blocks
	259	+ i = 0
	260	+ while i < len(blocks):
	261	+ b = blocks[i]
	262	+ if b.kind == "heading" and int(b.attrs.get("level", 2)) >= 2 and len(b.text) <= 60:
	263	+ j = i + 1
	264	+ lines: list[str] = []
	265	+ while j < len(blocks) and blocks[j].kind != "heading" and j - i <= 10:
	266	+ lines.extend(x.strip() for x in blocks[j].text.split("\n") if x.strip())
	267	+ j += 1
	268	+ add(_plan_from_lines(b.text, lines))
	269	+ i = j
	270	+ continue
	271	+ i += 1
	272	+ if not out: # tables: first cell = plan, another cell = price
	273	+ for b in page.blocks:
	274	+ if b.kind == "table" and "\|" in b.text:
	275	+ cells = [c.strip() for c in b.text.split("\|")]
	276	+ if len(cells) >= 2 and parse_price(cells[0]) is None and any(parse_price(c) for c in cells[1:]):
	277	+ add(_plan_from_lines(cells[0], cells[1:]))
	278	+ return out
	279	+
	280	+
	281	+# ------------------------------------------------------------------------------------------------------------ locations
	282	+
	283	+LOCATION_KIND_RULES: list[tuple[str, re.Pattern[str]]] = [
	284	+ ("headquarters", re.compile(r"\b(headquarters\|head office\|hq\|global office\|corporate office\|siège\|hauptsitz\|sede central)\b", re.IGNORECASE)),
	285	+ ("factory", re.compile(r"\b(factory\|plant\|manufacturing\|production site\|mill\|foundry\|usine\|werk\|fabrik)\b", re.IGNORECASE)),
	286	+ ("warehouse", re.compile(r"\b(warehouse\|distribution cent(er\|re)\|fulfil?lment\|logistics hub\|entrepôt\|lager)\b", re.IGNORECASE)),
	287	+ ("lab", re.compile(r"\b(lab\|laboratory\|research cent(er\|re)\|r&d\|innovation cent(er\|re))\b", re.IGNORECASE)),
	288	+ ("data_center", re.compile(r"\b(data ?cent(er\|re)\|datacenter\|server farm)\b", re.IGNORECASE)),
	289	+ ("store", re.compile(r"\b(store\|shop\|boutique\|showroom\|outlet\|dealer(ship)?\|branch\|agence\|filiale)\b", re.IGNORECASE)),
	290	+]
	291	+STREET_RE = re.compile(r"\b\d{1,5}[a-z]?\s+[^\n,]{2,40}\b(street\|st\.?\|avenue\|ave\.?\|road\|rd\.?\|boulevard\|blvd\.?\|drive\|dr\.?\|lane\|ln\.?\|way\|place\|pl\.?\|"
	292	+ r"square\|plaza\|court\|ct\.?\|parkway\|highway\|straße\|strasse\|str\.\|allee\|platz\|weg\|rue\|avenue\|boulevard\|avenida\|calle\|via\|piazza)\b"
	293	+ r"\|\b(rue\|avenue\|boulevard\|via\|calle\|avenida)\s+[^\n,]{2,40}\s\d{1,5}\b", re.IGNORECASE)
	294	+CITY_COUNTRY_RE = re.compile(r"^([A-ZÀ-Ý][\w'’.\- ]{1,40}),\s*([A-Za-zÀ-ÿ .]{2,40})$")
	295	+
	296	+
	297	+def _location_from_lines(name: str, lines: list[str], href: str \| None = None) -> ExtractedLocation \| None:
	298	+ text = " \n ".join(lines)
	299	+ kind = next((k for k, pat in LOCATION_KIND_RULES if pat.search(name) or pat.search(text[:200])), "office")
	300	+ city = region = country = address = None
	301	+ for ln in lines:
	302	+ street = STREET_RE.search(ln)
	303	+ if street and address is None and len(ln) <= 120:
	304	+ address = ln
	305	+ probe = (ln[:street.start()] + " " + ln[street.end():]).strip(" ,") if street else ln
	306	+ if country is None and probe:
	307	+ loc = parse_location(probe)
	308	+ if loc["country"] or loc["region"]:
	309	+ city, region, country = city or loc["city"], loc["region"], loc["country"]
	310	+ if country is None:
	311	+ c = country_code(name)
	312	+ if c:
	313	+ country = c
	314	+ else:
	315	+ m = CITY_COUNTRY_RE.match(name)
	316	+ if m and country_code(m.group(2)):
	317	+ city, country = m.group(1).strip(), country_code(m.group(2))
	318	+ if city is None and country is not None and looks_like_city(name):
	319	+ city = name
	320	+ if city is None and country is None and address is None and kind == "office":
	321	+ return None
	322	+ return ExtractedLocation(name=name[:120], kind=kind, city=city, region=region, country=country, address_text=address)
	323	+
	324	+
	325	+def looks_like_city(text: str) -> bool:
	326	+ t = normalize_whitespace(text)
	327	+ return 2 <= len(t) <= 40 and not any(ch.isdigit() for ch in t) and t[0].isupper() and not LOCATION_KIND_RULES[0][1].search(t)
	328	+
	329	+
	330	+def extract_locations(page: NormalizedPage) -> list[ExtractedLocation]:
	331	+ out: list[ExtractedLocation] = []
	332	+ seen: set[str] = set()
	333	+
	334	+ def add(loc: ExtractedLocation \| None) -> None:
	335	+ if loc is None:
	336	+ return
	337	+ key = norm_name(loc.name)
	338	+ if not key or key in seen or len(out) >= MAX_LOCATIONS:
	339	+ return
	340	+ seen.add(key)
	341	+ out.append(loc)
	342	+
	343	+ for a in page.jsonld.get("addresses", []) + page.jsonld.get("places", []) + page.microdata.get("addresses", []):
	344	+ addr = a.get("address") if isinstance(a.get("address"), dict) else a
	345	+ city = text_of(addr.get("addressLocality")) if isinstance(addr, dict) else None
	346	+ country = country_code(text_of(addr.get("addressCountry"))) if isinstance(addr, dict) else None
	347	+ name = text_of(a.get("name")) or ", ".join(x for x in (city, country) if x)
	348	+ if name and (city or country):
	349	+ street = text_of(addr.get("streetAddress")) if isinstance(addr, dict) else None
	350	+ add(ExtractedLocation(name=name[:120], kind="office", city=city, region=text_of(addr.get("addressRegion")) if isinstance(addr, dict) else None,
	351	+ country=country, address_text=street))
	352	+ for b in page.blocks:
	353	+ if b.kind != "location":
	354	+ continue
	355	+ lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
	356	+ if lines:
	357	+ add(_location_from_lines(lines[0], lines[1:], b.attrs.get("href")))
	358	+ if len(out) < 2: # fallback: headings naming a city/country followed by address-like lines
	359	+ blocks = page.blocks
	360	+ for i, b in enumerate(blocks):
	361	+ if b.kind == "heading" and int(b.attrs.get("level", 3)) >= 2 and (looks_like_city(b.text) or country_code(b.text)):
	362	+ lines: list[str] = []
	363	+ for nb in blocks[i + 1:i + 4]:
	364	+ if nb.kind == "heading":
	365	+ break
	366	+ lines.extend(x.strip() for x in nb.text.split("\n") if x.strip())
	367	+ add(_location_from_lines(b.text, lines))
	368	+ return out
	369	+
	370	+
	371	+# ------------------------------------------------------------------------------------------------------------ news
	372	+
	373	+NEWS_NOISE_ANCHOR = re.compile(r"^(read more\|learn more\|more\|continue reading\|view all\|see all\|all news\|all posts\|next\|previous\|older\|newer\|\d+)$", re.IGNORECASE)
	374	+
	375	+
	376	+def _time_index(html: str, base_url: str) -> dict[str, Any]:
	377	+ """canonical href → datetime for <time datetime> elements near a link (dates that the page states)."""
	378	+ out: dict[str, Any] = {}
	379	+ try:
	380	+ tree = LexborHTMLParser(html)
	381	+ except Exception: # noqa: BLE001
	382	+ return out
	383	+ for t in tree.css("time[datetime]")[:400]:
	384	+ dt = parse_date(t.attributes.get("datetime"))
	385	+ if dt is None:
	386	+ continue
	387	+ node = t
	388	+ for _ in range(7):
	389	+ if node is None or node.tag in ("body", "html"):
	390	+ break
	391	+ a = node.css_first("a[href]") if node.tag != "a" else node
	392	+ if a is not None:
	393	+ url = absolutize(base_url, a.attributes.get("href") or "")
	394	+ if url:
	395	+ out.setdefault(canonicalize_url(url), dt)
	396	+ break
	397	+ node = node.parent
	398	+ return out
	399	+
	400	+
	401	+def extract_news(page: NormalizedPage, html: str, *, surface: str, base_url: str) -> list[ExtractedNewsItem]:
	402	+ category = NEWS_CATEGORY.get(surface, "other") # type: ignore[call-overload]
	403	+ times = _time_index(html, base_url)
	404	+ out: list[ExtractedNewsItem] = []
	405	+ seen: set[str] = set()
	406	+ site = registrable_domain(base_url)
	407	+
	408	+ def add(title: str, url: str, published: Any, summary: str \| None = None) -> None:
	409	+ title = normalize_whitespace(title)
	410	+ if len(title) < 8 or NEWS_NOISE_ANCHOR.match(title) or len(out) >= MAX_NEWS:
	411	+ return
	412	+ canon = canonicalize_url(url)
	413	+ if canon in seen or canon == canonicalize_url(base_url) or is_static_asset(url):
	414	+ return
	415	+ seen.add(canon)
	416	+ out.append(ExtractedNewsItem(title=title[:300], url=url, published_at=published or times.get(canon), summary=(summary or None), category=category,
	417	+ language=page.lang))
	418	+
	419	+ for a in page.jsonld.get("articles", []) + page.microdata.get("articles", []):
	420	+ title, url = text_of(a.get("headline")) or text_of(a.get("name")), a.get("url") or (a.get("mainEntityOfPage") if isinstance(a.get("mainEntityOfPage"), str) else None)
	421	+ if title and isinstance(url, str):
	422	+ add(title, url, parse_date(a.get("datePublished")), text_of(a.get("description")))
	423	+ for b in page.blocks:
	424	+ href = b.attrs.get("href")
	425	+ if b.kind != "news_item" or not href:
	426	+ continue
	427	+ lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
	428	+ title = next((ln for ln in lines if len(ln) >= 12 and not (len(ln) <= 32 and date_from_text(ln))), lines[0] if lines else "")
	429	+ rest = [ln for ln in lines if ln != title and not (len(ln) <= 32 and date_from_text(ln))]
	430	+ add(title, href, date_from_text(b.text) or times.get(canonicalize_url(href)) or date_from_url(href), " ".join(rest[:2])[:300] or None)
	431	+ if len(out) < 3: # fallback: article-like links in the main region (same site, deeper path, long anchor)
	432	+ base_depth = urlparse(base_url).path.strip("/").count("/")
	433	+ for ln in page.links:
	434	+ if ln.region != "main" or registrable_domain(ln.url) != site or looks_like_trap(ln.url):
	435	+ continue
	436	+ depth = urlparse(ln.url).path.strip("/").count("/")
	437	+ dated = date_from_url(ln.url) or times.get(canonicalize_url(ln.url))
	438	+ if (len(ln.anchor) >= 25 and depth >= base_depth) or (dated and len(ln.anchor) >= 12):
	439	+ add(ln.anchor, ln.url, dated)
	440	+ out.sort(key=lambda n: (n.published_at is None, -(n.published_at.timestamp() if n.published_at else 0)))
	441	+ return out
	442	+
	443	+
	444	+# ------------------------------------------------------------------------------------------------------------ careers (HTML)
	445	+
	446	+JOB_URL_RE = re.compile(r"(/jobs?/\|/careers?/[^/]+\|/positions?/\|/openings?/\|/vacanc\|/opportunit\|/stellen\|/emplois?/\|/offres?/\|gh_jid=\|/job-\|lever\.co/\|"
	447	+ r"greenhouse\.io/\|ashbyhq\.com/\|myworkdayjobs\.com/.+/job/\|smartrecruiters\.com/\|workable\.com/j/\|recruitee\.com/o/\|"
	448	+ r"personio\.de/job/\|teamtailor\.com/jobs/\|bamboohr\.com/careers/\d\|jobvite\.com/\|icims\.com/jobs/\|taleo\.net/.+job)", re.IGNORECASE)
	449	+JOB_NOISE_ANCHOR = re.compile(r"^(apply( now)?\|view( all)?( jobs\| openings\| roles\| positions)?\|see (all\|more\|open)( jobs\| roles\| positions)?\|learn more\|read more\|"
	450	+ r"careers?\|jobs?\|join (us\|the team\|our team)\|open (roles\|positions)\|all (jobs\|roles\|positions\|openings)\|search jobs\|"
	451	+ r"browse jobs\|explore\|more\|back\|home\|filter\|next\|previous\|\d+)$", re.IGNORECASE)
	452	+
	453	+
	454	+def _states_location(text: str) -> bool:
	455	+ if len(text) > 80:
	456	+ return False
	457	+ p = parse_location(text)
	458	+ return bool(p["city"] or p["country"] or p["remote"] or p["region"])
	459	+
	460	+
	461	+def extract_jobs_html(page: NormalizedPage, *, base_url: str) -> list[ExtractedJob]:
	462	+ jobs: list[ExtractedJob] = []
	463	+ seen: set[str] = set()
	464	+
	465	+ def add(title: str, url: str \| None, location: str \| None, department: str \| None = None) -> None:
	466	+ title = normalize_whitespace(title).strip(" -–—\|·")
	467	+ if not (4 <= len(title) <= 140) or JOB_NOISE_ANCHOR.match(title) or len(jobs) >= MAX_JOBS:
	468	+ return
	469	+ key = (canonicalize_url(url) if url else "") + "\|" + title.lower()
	470	+ if key in seen:
	471	+ return
	472	+ seen.add(key)
	473	+ jobs.append(finish_job(ExtractedJob(title=title, url=url, location_text=location, department=department)))
	474	+
	475	+ jl_jobs = jobs_from_jsonld(list(page.jsonld.get("job_postings", [])) + list(page.microdata.get("job_postings", [])), page_url=base_url)
	476	+ for j in jl_jobs:
	477	+ add(j.title, j.url, j.location_text, j.department)
	478	+ if jobs and jobs[-1].title == normalize_whitespace(j.title):
	479	+ jobs[-1] = j
	480	+ block_jobs: list[ExtractedJob] = []
	481	+ for b in page.blocks:
	482	+ href = b.attrs.get("href")
	483	+ if b.kind != "job_listing" or not href or not JOB_URL_RE.search(href) and registrable_domain(href) == registrable_domain(base_url) and urlparse(href).path.count("/") < 2:
	484	+ continue
	485	+ lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
	486	+ if lines and " \| " in lines[0]: # table row: cells joined by " \| "
	487	+ lines = [c.strip() for c in lines[0].split(" \| ") if c.strip()] + lines[1:]
	488	+ if not lines:
	489	+ continue
	490	+ title = lines[0]
	491	+ loc = next((ln for ln in lines[1:6] if _states_location(ln)), None)
	492	+ dept = next((ln for ln in lines[1:6] if ln != loc and 2 <= len(ln) <= 40 and not _states_location(ln) and not date_from_text(ln)), None)
	493	+ before = len(jobs)
	494	+ add(title, href, loc, dept)
	495	+ if len(jobs) > before:
	496	+ block_jobs.append(jobs[-1])
	497	+ if len(jobs) < 3: # fallback: job-like anchors anywhere in the main region
	498	+ cands = [ln for ln in page.links if ln.region == "main" and JOB_URL_RE.search(ln.url) and 4 <= len(ln.anchor) <= 140
	499	+ and not JOB_NOISE_ANCHOR.match(ln.anchor) and not is_static_asset(ln.url)]
	500	+ if len(cands) >= 3:
	501	+ for ln in cands:
	502	+ add(ln.anchor, ln.url, None)
	503	+ if len(jobs) < 3 and not jl_jobs:
	504	+ return [] # not a trustworthy listing — the pipeline must not mark jobs removed on this
	505	+ return jobs
	506	+
	507	+
	508	+# ------------------------------------------------------------------------------------------------------------ products
	509	+
	510	+
	511	+def extract_products(page: NormalizedPage) -> list[ExtractedProduct]:
	512	+ out: list[ExtractedProduct] = []
	513	+ seen: set[str] = set()
	514	+
	515	+ def add(name: str, url: str \| None, description: str \| None, category: str \| None = None) -> None:
	516	+ name = normalize_whitespace(name).strip(" -–—\|·")
	517	+ key = norm_name(name)
	518	+ if not (2 <= len(name) <= 90) or not key or key in seen or len(out) >= MAX_PRODUCTS or NEWS_NOISE_ANCHOR.match(name):
	519	+ return
	520	+ seen.add(key)
	521	+ out.append(ExtractedProduct(name=name, url=url, category=category, description=(description or None)))
	522	+
	523	+ for p in page.jsonld.get("products", []) + page.microdata.get("products", []):
	524	+ name = text_of(p.get("name"))
	525	+ if name:
	526	+ add(name, p.get("url") if isinstance(p.get("url"), str) else None, text_of(p.get("description")), text_of(p.get("category")))
	527	+ for b in page.blocks:
	528	+ if b.kind != "product_card":
	529	+ continue
	530	+ lines = [ln.strip() for ln in b.text.split("\n") if ln.strip()]
	531	+ if lines:
	532	+ add(lines[0], b.attrs.get("href"), " ".join(lines[1:4])[:300], b.path.split(" > ")[-1] if b.path else None)
	533	+ return out
	534	+
	535	+
	536	+# ------------------------------------------------------------------------------------------------------------ discovery links
	537	+
	538	+
	539	+def discovered_links(page: NormalizedPage, *, base_url: str, canonical_domain: str \| None) -> list[DiscoveredUrl]:
	540	+ site = registrable_domain(canonical_domain or base_url)
	541	+ out: list[DiscoveredUrl] = []
	542	+ seen: set[str] = set()
	543	+ for ln in page.links:
	544	+ if registrable_domain(ln.url) != site or is_static_asset(ln.url) or looks_like_trap(ln.url):
	545	+ continue
	546	+ canon = canonicalize_url(ln.url)
	547	+ if canon in seen:
	548	+ continue
	549	+ surface, conf = classify_url(ln.url, anchor=ln.anchor, canonical_domain=canonical_domain or site)
	550	+ if surface in (Surface.OTHER, Surface.HOMEPAGE) or conf < settings.discovery_min_confidence:
	551	+ continue
	552	+ seen.add(canon)
	553	+ out.append(DiscoveredUrl(url=ln.url, surface=surface, confidence=conf, anchor=ln.anchor[:120] or None, method="nav" if ln.region in ("nav", "header", "footer") else "link"))
	554	+ if len(out) >= MAX_DISCOVERED:
	555	+ break
	556	+ for feed in page.feeds:
	557	+ out.append(DiscoveredUrl(url=feed, surface=Surface.FEED, confidence=0.9, method="feed"))
	558	+ return out
	559	+
	560	+
	561	+# ------------------------------------------------------------------------------------------------------------ connector
	562	+
	563	+
	564	+@register
	565	+class GenericHtmlConnector(Connector):
	566	+ meta = ConnectorMeta(connector_id="generic-html-v1", name="Generic HTML surface", version="1", category=Surface.OTHER, fetch_mode=FetchMode.HTTP,
	567	+ supports_discovery=True, default_interval_s=24 * 3600, surfaces=("*",), priority=1,
	568	+ description="Surface-aware extraction for any corporate web page")
	569	+
	570	+ def extract(self, sensor: Mapping[str, Any], result: FetchResult) -> Extraction:
	571	+ surface = str(sensor.get("surface") or Surface.OTHER)
	572	+ html = result.text
	573	+ if not result.is_html and result.is_json:
	574	+ raise ValueError("generic HTML connector received JSON")
	575	+ page = normalize.parse(html, url=result.final_url, surface=surface)
	576	+ ex = page.to_extraction()
	577	+ cfg = sensor.get("config") or {}
	578	+ canonical_domain = str(cfg.get("canonical_domain") or "") or None
	579	+ if surface == Surface.LEADERSHIP or surface == Surface.ABOUT:
	580	+ ex.people = extract_people(page)
	581	+ if surface == Surface.PRICING:
	582	+ ex.plans = extract_plans(page)
	583	+ if surface == Surface.LOCATIONS or surface == Surface.CONTACT:
	584	+ ex.locations = extract_locations(page)
	585	+ if surface in NEWS_SURFACES:
	586	+ ex.news = extract_news(page, html, surface=surface, base_url=result.final_url)
	587	+ if surface in (Surface.CAREERS, Surface.JOBS_BOARD):
	588	+ ex.jobs = extract_jobs_html(page, base_url=result.final_url)
	589	+ if ex.jobs:
	590	+ ex.blocks = [b for b in ex.blocks if b.kind != "job_listing"] + job_blocks(ex.jobs, path="Jobs")
	591	+ if surface in PRODUCT_SURFACES:
	592	+ ex.products = extract_products(page)
	593	+ ex.discovered = discovered_links(page, base_url=result.final_url, canonical_domain=canonical_domain)
	594	+ ex.meta.update({"surface": surface, "title": page.title, "headings": [h[1][:80] for h in page.headings[:30]], "link_count": len(page.links),
	595	+ "structured": bool(ex.jobs or ex.plans or ex.people or ex.locations or ex.news or ex.products)})
	596	+ return ex
	597	+
	598	+
	599	+def blocks_summary(blocks: list[Block]) -> dict[str, int]:
	600	+ out: dict[str, int] = {}
	601	+ for b in blocks:
	602	+ out[b.kind] = out.get(b.kind, 0) + 1
	603	+ return out
	604	+
	605	+
	606	+__all__ = ["GenericHtmlConnector", "blocks_summary", "discovered_links", "extract_jobs_html", "extract_locations", "extract_news", "extract_people",
	607	+ "extract_plans", "extract_products", "looks_like_name", "parse_price", "role_category"]

added src/companyatlas/connectors/greenhouse.py +41 −0

		`@@ -0,0 +1,41 @@`
	1	+"""Greenhouse job board — public endpoint used by every `boards.greenhouse.io/<token>` page:
	2	+`GET https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=false` → {"jobs":[{id, title, absolute_url, location:{name}, updated_at,
	3	+first_published, departments?, offices?, requisition_id}]}. Verified 2026-09-12 against the `stripe` board (fixture trimmed to 20 jobs)."""
	4	+from __future__ import annotations
	5	+
	6	+import re
	7	+from collections.abc import Mapping
	8	+from typing import Any
	9	+
	10	+from companyatlas.connectors._ats_base import AtsConnector
	11	+from companyatlas.connectors._util import parse_date, text_of
	12	+from companyatlas.sdk.connector import ConnectorMeta, register
	13	+from companyatlas.sdk.models import ExtractedJob
	14	+from companyatlas.taxonomy import FetchMode, Surface
	15	+
	16	+
	17	+@register
	18	+class GreenhouseConnector(AtsConnector):
	19	+ vendor = "greenhouse"
	20	+ token_re = re.compile(r"boards-api\.greenhouse\.io/v1/boards/([a-z0-9_-]+)", re.IGNORECASE)
	21	+ meta = ConnectorMeta(connector_id="greenhouse-v1", name="Greenhouse job board", version="1", category=Surface.JOBS_BOARD,
	22	+ fetch_mode=FetchMode.JSON, supports_discovery=False, supports_incremental=True, default_interval_s=6 * 3600,
	23	+ url_pattern=r"boards-api\.greenhouse\.io/v1/boards/", priority=50, pattern_required=True, accept="application/json",
	24	+ description="Public Greenhouse Job Board API (jobs list without descriptions)")
	25	+
	26	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	27	+ items = data.get("jobs") if isinstance(data, dict) else data
	28	+ out: list[ExtractedJob] = []
	29	+ for j in items or []:
	30	+ if not isinstance(j, dict) or not j.get("title"):
	31	+ continue
	32	+ depts = j.get("departments") or []
	33	+ offices = j.get("offices") or []
	34	+ out.append(ExtractedJob(
	35	+ title=str(j["title"]), url=j.get("absolute_url"), external_id=str(j.get("id") or j.get("internal_job_id") or ""),
	36	+ department=text_of(depts[0]) if depts else None, location_text=text_of(j.get("location")),
	37	+ posted_at=parse_date(j.get("first_published") or j.get("updated_at")),
	38	+ raw={"requisition_id": j.get("requisition_id"), "updated_at": j.get("updated_at"), "offices": [text_of(o) for o in offices][:5],
	39	+ "language": j.get("language")},
	40	+ ))
	41	+ return out

added src/companyatlas/connectors/jsonld_jobs.py +87 −0

		`@@ -0,0 +1,87 @@`
	1	+"""JobPosting JSON-LD / microdata on any careers page (schema.org). Shared helper `jobs_from_jsonld` is also used by the generic HTML
	2	+connector; this connector is picked when discovery has seen JobPosting structured data on a careers URL that has no ATS board."""
	3	+from __future__ import annotations
	4	+
	5	+from collections.abc import Mapping
	6	+from typing import Any
	7	+
	8	+from companyatlas.connectors._util import country_code, description_hash, finish_job, job_blocks, jobs_text, parse_date, text_of
	9	+from companyatlas.fetch import FetchResult
	10	+from companyatlas.sdk import normalize
	11	+from companyatlas.sdk.connector import Connector, ConnectorMeta, register
	12	+from companyatlas.sdk.models import ExtractedJob, Extraction
	13	+from companyatlas.taxonomy import FetchMode, Surface
	14	+
	15	+
	16	+def _place(loc: Any) -> tuple[str \| None, str \| None, str \| None, str \| None]:
	17	+ """(location_text, city, region, country) from a schema.org Place / PostalAddress / string."""
	18	+ if loc is None:
	19	+ return None, None, None, None
	20	+ if isinstance(loc, list):
	21	+ loc = loc[0] if loc else None
	22	+ if loc is None:
	23	+ return None, None, None, None
	24	+ if isinstance(loc, str):
	25	+ return loc, None, None, None
	26	+ addr = loc.get("address") if isinstance(loc, dict) else None
	27	+ if isinstance(addr, str):
	28	+ return addr, None, None, None
	29	+ addr = addr if isinstance(addr, dict) else (loc if isinstance(loc, dict) and "addressLocality" in loc else {})
	30	+ city = text_of(addr.get("addressLocality"))
	31	+ region = text_of(addr.get("addressRegion"))
	32	+ country_raw = addr.get("addressCountry")
	33	+ if isinstance(country_raw, dict):
	34	+ country_raw = country_raw.get("name")
	35	+ country = country_code(country_raw) if isinstance(country_raw, str) else None
	36	+ parts = [x for x in (city, region, (country or (country_raw if isinstance(country_raw, str) else None))) if x]
	37	+ return (", ".join(parts) or text_of(loc.get("name")) if isinstance(loc, dict) else None), city, region, country
	38	+
	39	+
	40	+def jobs_from_jsonld(items: list[dict[str, Any]] \| None, *, page_url: str) -> list[ExtractedJob]:
	41	+ out: list[ExtractedJob] = []
	42	+ for jp in items or []:
	43	+ title = text_of(jp.get("title")) or text_of(jp.get("name"))
	44	+ if not title:
	45	+ continue
	46	+ loc_text, city, region, country = _place(jp.get("jobLocation"))
	47	+ ident = jp.get("identifier")
	48	+ ext = None
	49	+ if isinstance(ident, dict):
	50	+ ext = text_of(ident.get("value")) or text_of(ident.get("name"))
	51	+ elif isinstance(ident, str \| int):
	52	+ ext = str(ident)
	53	+ remote = True if str(jp.get("jobLocationType") or "").upper() == "TELECOMMUTE" else None
	54	+ salary = jp.get("baseSalary") if isinstance(jp.get("baseSalary"), dict) else {}
	55	+ val = salary.get("value") if isinstance(salary.get("value"), dict) else {}
	56	+ emp = jp.get("employmentType")
	57	+ job = ExtractedJob(title=title, url=text_of(jp.get("url")) or page_url, external_id=ext, location_text=loc_text, city=city, region=region,
	58	+ country=country, remote=remote, employment_type=text_of(emp[0] if isinstance(emp, list) and emp else emp),
	59	+ posted_at=parse_date(jp.get("datePosted")), description_hash=description_hash(jp.get("description")),
	60	+ salary_min=val.get("minValue") or val.get("value"), salary_max=val.get("maxValue"), salary_currency=salary.get("currency"),
	61	+ salary_period=val.get("unitText"), department=text_of(jp.get("occupationalCategory")) if isinstance(jp.get("occupationalCategory"), str) else None,
	62	+ raw={"validThrough": jp.get("validThrough"), "hiringOrganization": text_of(jp.get("hiringOrganization"))})
	63	+ out.append(finish_job(job))
	64	+ return out
	65	+
	66	+
	67	+@register
	68	+class JsonLdJobsConnector(Connector):
	69	+ meta = ConnectorMeta(connector_id="jsonld-jobs-v1", name="JobPosting structured data", version="1", category=Surface.JOBS_BOARD, fetch_mode=FetchMode.HTTP,
	70	+ default_interval_s=12 * 3600, surfaces=(), priority=2,
	71	+ description="schema.org JobPosting JSON-LD / microdata (auto-picked only for non-ATS jobs_board URLs; careers HTML uses generic-html)")
	72	+
	73	+ def extract(self, sensor: Mapping[str, Any], result: FetchResult) -> Extraction:
	74	+ page = normalize.parse(result.text, url=result.final_url, surface=str(sensor.get("surface") or ""))
	75	+ items = list(page.jsonld.get("job_postings") or []) + list(page.microdata.get("job_postings") or [])
	76	+ jobs = jobs_from_jsonld(items, page_url=result.final_url)
	77	+ header = f"{page.title or 'Careers'} — {len(jobs)} structured job postings"
	78	+ ex = page.to_extraction()
	79	+ ex.jobs = jobs
	80	+ if jobs:
	81	+ ex.blocks = ex.blocks + job_blocks(jobs, path="JobPosting")
	82	+ ex.text = ex.text + "\n" + jobs_text(jobs, header)
	83	+ ex.meta.update({"job_count": len(jobs), "structured": bool(jobs)})
	84	+ return ex
	85	+
	86	+
	87	+__all__ = ["JsonLdJobsConnector", "jobs_from_jsonld"]

added src/companyatlas/connectors/lever.py +42 −0

		`@@ -0,0 +1,42 @@`
	1	+"""Lever postings — public endpoint behind `jobs.lever.co/<token>`:
	2	+`GET https://api.lever.co/v0/postings/{token}?mode=json` → [{id, text, categories:{commitment, location, team, department, allLocations},
	3	+country, workplaceType, hostedUrl, applyUrl, createdAt(ms), salaryRange?}]. EU tenants use api.eu.lever.co. Verified 2026-09-12 against the
	4	+`palantir` board (fixture trimmed to 20 postings, descriptions removed)."""
	5	+from __future__ import annotations
	6	+
	7	+import re
	8	+from collections.abc import Mapping
	9	+from typing import Any
	10	+
	11	+from companyatlas.connectors._ats_base import AtsConnector
	12	+from companyatlas.connectors._util import country_code, dig, parse_date, text_of
	13	+from companyatlas.sdk.connector import ConnectorMeta, register
	14	+from companyatlas.sdk.models import ExtractedJob
	15	+from companyatlas.taxonomy import FetchMode, Surface
	16	+
	17	+
	18	+@register
	19	+class LeverConnector(AtsConnector):
	20	+ vendor = "lever"
	21	+ token_re = re.compile(r"api\.(?:eu\.)?lever\.co/v0/postings/([a-z0-9_-]+)", re.IGNORECASE)
	22	+ meta = ConnectorMeta(connector_id="lever-v1", name="Lever postings", version="1", category=Surface.JOBS_BOARD, fetch_mode=FetchMode.JSON,
	23	+ default_interval_s=6 * 3600, url_pattern=r"api\.(eu\.)?lever\.co/v0/postings/", priority=50, pattern_required=True, accept="application/json",
	24	+ description="Public Lever postings API (mode=json)")
	25	+
	26	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	27	+ out: list[ExtractedJob] = []
	28	+ for j in data if isinstance(data, list) else []:
	29	+ if not isinstance(j, dict) or not j.get("text"):
	30	+ continue
	31	+ cats = j.get("categories") or {}
	32	+ workplace = (j.get("workplaceType") or "").lower()
	33	+ sal = j.get("salaryRange") or {}
	34	+ out.append(ExtractedJob(
	35	+ title=str(j["text"]), url=j.get("hostedUrl") or j.get("applyUrl"), external_id=str(j.get("id") or ""),
	36	+ department=text_of(cats.get("department")), team=text_of(cats.get("team")), location_text=text_of(cats.get("location")),
	37	+ country=country_code(j.get("country")), remote=True if workplace == "remote" else (False if workplace in ("hybrid", "on-site", "onsite") else None),
	38	+ employment_type=text_of(cats.get("commitment")), posted_at=parse_date(j.get("createdAt")),
	39	+ salary_min=sal.get("min"), salary_max=sal.get("max"), salary_currency=sal.get("currency"), salary_period=sal.get("interval"),
	40	+ raw={"workplaceType": j.get("workplaceType"), "allLocations": dig(cats, "allLocations", default=[])[:5] if isinstance(cats.get("allLocations"), list) else []},
	41	+ ))
	42	+ return out

added src/companyatlas/connectors/personio.py +50 −0

		`@@ -0,0 +1,50 @@`
	1	+"""Personio job XML feed — public feed behind `<token>.jobs.personio.de`:
	2	+`GET https://{token}.jobs.personio.de/xml` → <workzag-jobs><position><id/><office/><additionalOffices><office/></additionalOffices><department/>
	3	+<name/><employmentType/><seniority/><schedule/><createdAt/>…</position></workzag-jobs>. Job URL `https://{token}.jobs.personio.de/job/{id}`.
	4	+Verified 2026-09-12 against `personio` (fixture)."""
	5	+from __future__ import annotations
	6	+
	7	+import re
	8	+import xml.etree.ElementTree as ET
	9	+from collections.abc import Mapping
	10	+from typing import Any
	11	+
	12	+from companyatlas.connectors._ats_base import AtsConnector
	13	+from companyatlas.connectors._util import parse_date
	14	+from companyatlas.fetch import FetchResult
	15	+from companyatlas.sdk.connector import ConnectorMeta, register
	16	+from companyatlas.sdk.models import ExtractedJob
	17	+from companyatlas.taxonomy import FetchMode, Surface
	18	+
	19	+
	20	+@register
	21	+class PersonioConnector(AtsConnector):
	22	+ vendor = "personio"
	23	+ token_re = re.compile(r"https?://([a-z0-9-]+)\.jobs\.personio\.(?:de\|com)/xml", re.IGNORECASE)
	24	+ meta = ConnectorMeta(connector_id="personio-v1", name="Personio XML feed", version="1", category=Surface.JOBS_BOARD, fetch_mode=FetchMode.FEED,
	25	+ default_interval_s=12 * 3600, url_pattern=r"\.jobs\.personio\.(de\|com)/xml", pattern_required=True, priority=50, accept="application/xml,text/xml",
	26	+ description="Public Personio job XML feed")
	27	+
	28	+ def load(self, result: FetchResult) -> Any:
	29	+ if len(result.content) > 8 * 1024 * 1024 or b"<!ENTITY" in result.content[:4096]:
	30	+ raise ValueError("refusing suspicious XML (size or entity declarations)")
	31	+ return ET.fromstring(result.content)
	32	+
	33	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	34	+ token = self.token(sensor) or ""
	35	+ host = re.sub(r"/xml.*$", "", str(sensor.get("url") or "")) or f"https://{token}.jobs.personio.de"
	36	+ out: list[ExtractedJob] = []
	37	+ for pos in data.iter("position"):
	38	+ g = {c.tag: (c.text or "").strip() for c in pos if c.tag != "additionalOffices" and c.tag != "jobDescriptions"}
	39	+ name = g.get("name")
	40	+ if not name:
	41	+ continue
	42	+ offices = [g.get("office")] + [o.text.strip() for o in pos.iter("office") if o.text and o.text.strip() != g.get("office")]
	43	+ offices = [o for o in offices if o]
	44	+ jid = g.get("id")
	45	+ out.append(ExtractedJob(title=name, url=f"{host}/job/{jid}" if jid else None, external_id=jid or None, department=g.get("department") or None,
	46	+ location_text=offices[0] if offices else None, employment_type=g.get("schedule") or g.get("employmentType") or None,
	47	+ seniority=g.get("seniority") or None, posted_at=parse_date(g.get("createdAt")),
	48	+ raw={"recruitingCategory": g.get("recruitingCategory"), "occupation": g.get("occupation"), "subcompany": g.get("subcompany"),
	49	+ "yearsOfExperience": g.get("yearsOfExperience"), "offices": offices[:5]}))
	50	+ return out

added src/companyatlas/connectors/recruitee.py +44 −0

		`@@ -0,0 +1,44 @@`
	1	+"""Recruitee careers-site API — public, keyless endpoint behind `<token>.recruitee.com`:
	2	+`GET https://{token}.recruitee.com/api/offers/` → {"offers":[{id, slug, title, careers_url, city, country, country_code, state_name, department,
	3	+published_at, created_at, remote, hybrid, on_site, employment_type_code, category_code, experience_code, location, salary?, tags}]}.
	4	+Verified 2026-09-12 against `vandebron` (fixture trimmed to 20 offers, descriptions removed)."""
	5	+from __future__ import annotations
	6	+
	7	+import re
	8	+from collections.abc import Mapping
	9	+from typing import Any
	10	+
	11	+from companyatlas.connectors._ats_base import AtsConnector
	12	+from companyatlas.connectors._util import country_code, parse_date, text_of
	13	+from companyatlas.sdk.connector import ConnectorMeta, register
	14	+from companyatlas.sdk.models import ExtractedJob
	15	+from companyatlas.taxonomy import FetchMode, Surface
	16	+
	17	+
	18	+@register
	19	+class RecruiteeConnector(AtsConnector):
	20	+ vendor = "recruitee"
	21	+ token_re = re.compile(r"https?://([a-z0-9-]+)\.recruitee\.com/api/offers", re.IGNORECASE)
	22	+ meta = ConnectorMeta(connector_id="recruitee-v1", name="Recruitee offers", version="1", category=Surface.JOBS_BOARD, fetch_mode=FetchMode.JSON,
	23	+ default_interval_s=6 * 3600, url_pattern=r"\.recruitee\.com/api/offers", pattern_required=True, priority=50, accept="application/json",
	24	+ description="Public Recruitee careers-site API")
	25	+
	26	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	27	+ out: list[ExtractedJob] = []
	28	+ for o in (data.get("offers") if isinstance(data, dict) else []) or []:
	29	+ if not isinstance(o, dict) or not o.get("title"):
	30	+ continue
	31	+ if o.get("status") not in (None, "published"):
	32	+ continue
	33	+ sal = o.get("salary") or {}
	34	+ remote = True if o.get("remote") else (False if (o.get("on_site") or o.get("hybrid")) else None)
	35	+ out.append(ExtractedJob(
	36	+ title=str(o["title"]), url=o.get("careers_url"), external_id=str(o.get("id") or o.get("slug") or ""),
	37	+ department=text_of(o.get("department")), location_text=text_of(o.get("location")) or ", ".join(x for x in (o.get("city"), o.get("country")) if x) or None,
	38	+ city=text_of(o.get("city")), region=text_of(o.get("state_name")), country=country_code(o.get("country_code")) or country_code(o.get("country")),
	39	+ remote=remote, employment_type=text_of(o.get("employment_type_code")), posted_at=parse_date(o.get("published_at") or o.get("created_at")),
	40	+ salary_min=sal.get("min") if isinstance(sal, dict) else None, salary_max=sal.get("max") if isinstance(sal, dict) else None,
	41	+ salary_currency=sal.get("currency") if isinstance(sal, dict) else None, salary_period=sal.get("period") if isinstance(sal, dict) else None,
	42	+ raw={"category_code": o.get("category_code"), "experience_code": o.get("experience_code"), "tags": (o.get("tags") or [])[:10]},
	43	+ ))
	44	+ return out

added src/companyatlas/connectors/sitemap.py +133 −0

		`@@ -0,0 +1,133 @@`
	1	+"""Sitemap connector (spec §9.2): `urlset` and `sitemapindex` (plain or .gz), `lastmod`, bounded by `settings.discovery_max_sitemap_urls`.
	2	+The observable change is the set of URLs (new / gone since the previous snapshot — the pipeline diffs `extracted.urls`) plus `lastmod`
	3	+moves; every URL is classified with `urls.classify_url` and emitted as `DiscoveredUrl` so the discovery feedback loop can add sensors.
	4	+Child sitemaps of an index are fetched breadth-first (bounded `MAX_CHILD_SITEMAPS`)."""
	5	+from __future__ import annotations
	6	+
	7	+import gzip
	8	+import hashlib
	9	+import logging
	10	+import re
	11	+from collections.abc import Mapping
	12	+from typing import Any
	13	+from urllib.parse import urlparse
	14	+
	15	+from companyatlas.config import settings
	16	+from companyatlas.connectors._util import merged_result
	17	+from companyatlas.fetch import Fetcher, FetchResult
	18	+from companyatlas.sdk.connector import Connector, ConnectorContext, ConnectorMeta, register
	19	+from companyatlas.sdk.models import Block, DiscoveredUrl, Extraction
	20	+from companyatlas.sdk.normalize import simhash
	21	+from companyatlas.taxonomy import FetchMode, Surface
	22	+from companyatlas.urls import canonicalize_url, classify_url, is_static_asset, looks_like_trap
	23	+
	24	+log = logging.getLogger(__name__)
	25	+
	26	+MAX_CHILD_SITEMAPS = 25
	27	+LOC_RE = re.compile(r"<loc>\s(?:<!\[CDATA\[)?\s([^<\]\s]+)\s(?:\]\]>)?\s</loc>", re.IGNORECASE)
	28	+ENTRY_RE = re.compile(r"<(url\|sitemap)>(.*?)</\1>", re.IGNORECASE \| re.DOTALL)
	29	+LASTMOD_RE = re.compile(r"<lastmod>\s([^<\s]+)\s</lastmod>", re.IGNORECASE)
	30	+# Sitemap children whose names suggest low-value bulk content (products/tags/images) are visited last.
	31	+LOW_VALUE_CHILD_RE = re.compile(r"(image\|video\|tag\|category\|author\|product\|shop\|store\|collection\|attachment\|page-\d+\|post-\d+)", re.IGNORECASE)
	32	+HIGH_VALUE_CHILD_RE = re.compile(r"(page\|pages\|static\|main\|site\|news\|press\|blog\|careers?\|jobs?\|misc\|general)", re.IGNORECASE)
	33	+
	34	+
	35	+def _decode(content: bytes) -> str:
	36	+ if content[:2] == b"\x1f\x8b":
	37	+ try:
	38	+ content = gzip.decompress(content)
	39	+ except (OSError, EOFError):
	40	+ return ""
	41	+ return content.decode("utf-8", errors="replace")
	42	+
	43	+
	44	+def parse_sitemap(text: str) -> tuple[list[tuple[str, str \| None]], list[tuple[str, str \| None]]]:
	45	+ """→ (page_urls[(loc, lastmod)], child_sitemaps[(loc, lastmod)])."""
	46	+ pages: list[tuple[str, str \| None]] = []
	47	+ children: list[tuple[str, str \| None]] = []
	48	+ for m in ENTRY_RE.finditer(text):
	49	+ body = m.group(2)
	50	+ loc = LOC_RE.search(body)
	51	+ if not loc:
	52	+ continue
	53	+ lm = LASTMOD_RE.search(body)
	54	+ item = (loc.group(1).strip(), lm.group(1).strip() if lm else None)
	55	+ (children if m.group(1).lower() == "sitemap" else pages).append(item)
	56	+ if not pages and not children: # tolerate sitemaps without <url> wrappers / text sitemaps
	57	+ for m in LOC_RE.finditer(text):
	58	+ pages.append((m.group(1).strip(), None))
	59	+ if not pages:
	60	+ for line in text.splitlines():
	61	+ line = line.strip()
	62	+ if line.startswith(("http://", "https://")):
	63	+ pages.append((line, None))
	64	+ return pages, children
	65	+
	66	+
	67	+@register
	68	+class SitemapConnector(Connector):
	69	+ meta = ConnectorMeta(connector_id="sitemap-v1", name="XML sitemap", version="1", category=Surface.SITEMAP, fetch_mode=FetchMode.SITEMAP,
	70	+ supports_discovery=True, default_interval_s=24 * 3600, url_pattern=r"sitemap[^/]*\.xml(\.gz)?$\|/sitemap_index\.xml$\|/sitemaps?/",
	71	+ priority=40, accept="application/xml,text/xml,/;q=0.5", description="urlset / sitemapindex with lastmod, bounded")
	72	+
	73	+ async def fetch(self, ctx: ConnectorContext, sensor: Mapping[str, Any], fetcher: Fetcher) -> FetchResult:
	74	+ first = await fetcher.get(str(sensor["url"]), etag=sensor.get("etag"), last_modified=sensor.get("last_modified"), accept=self.meta.accept)
	75	+ pages, children = parse_sitemap(_decode(first.content))
	76	+ if not children:
	77	+ return first
	78	+ limit = settings.discovery_max_sitemap_urls
	79	+ ordered = sorted(children, key=lambda c: (0 if HIGH_VALUE_CHILD_RE.search(c[0]) else (2 if LOW_VALUE_CHILD_RE.search(c[0]) else 1), c[0]))
	80	+ merged: list[dict[str, Any]] = [{"loc": u, "lastmod": lm} for u, lm in pages]
	81	+ visited: list[str] = []
	82	+ for child_url, _lm in ordered[:MAX_CHILD_SITEMAPS]:
	83	+ if len(merged) >= limit:
	84	+ break
	85	+ try:
	86	+ res = await fetcher.get(child_url, accept=self.meta.accept)
	87	+ except Exception as exc: # noqa: BLE001 - one broken child must not fail the whole sitemap
	88	+ log.info("child sitemap skipped", extra={"child": child_url, "error": str(exc)[:200]})
	89	+ continue
	90	+ p, _c = parse_sitemap(_decode(res.content))
	91	+ merged.extend({"loc": u, "lastmod": lm} for u, lm in p[: max(0, limit - len(merged))])
	92	+ visited.append(child_url)
	93	+ payload = {"index": first.final_url, "children": [c[0] for c in ordered], "visited": visited, "urls": merged[:limit]}
	94	+ return merged_result(first, payload, pages=1 + len(visited))
	95	+
	96	+ def extract(self, sensor: Mapping[str, Any], result: FetchResult) -> Extraction:
	97	+ limit = settings.discovery_max_sitemap_urls
	98	+ if result.headers.get("x-companyatlas-pages"):
	99	+ data = result.json()
	100	+ entries = [(e.get("loc"), e.get("lastmod")) for e in data.get("urls", []) if e.get("loc")]
	101	+ children = list(data.get("children", []))
	102	+ else:
	103	+ entries, kids = parse_sitemap(_decode(result.content))
	104	+ children = [c[0] for c in kids]
	105	+ entries = entries[:limit]
	106	+ canonical_domain = str((sensor.get("config") or {}).get("canonical_domain") or urlparse(str(sensor["url"])).hostname or "")
	107	+ seen: set[str] = set()
	108	+ blocks: list[Block] = []
	109	+ discovered: list[DiscoveredUrl] = []
	110	+ by_surface: dict[str, int] = {}
	111	+ urls_out: list[dict[str, Any]] = []
	112	+ for loc, lastmod in entries:
	113	+ if not loc or is_static_asset(loc):
	114	+ continue
	115	+ canon = canonicalize_url(loc)
	116	+ if canon in seen:
	117	+ continue
	118	+ seen.add(canon)
	119	+ surface, conf = classify_url(loc, canonical_domain=canonical_domain)
	120	+ by_surface[str(surface)] = by_surface.get(str(surface), 0) + 1
	121	+ urls_out.append({"url": canon, "lastmod": lastmod, "surface": str(surface), "confidence": conf})
	122	+ key = f"url:{hashlib.blake2b(canon.encode('utf-8'), digest_size=8).hexdigest()}"
	123	+ blocks.append(Block(key=key, kind="list", text=canon, path="Sitemap", hash=hashlib.sha256(canon.encode()).hexdigest()[:16],
	124	+ simhash=simhash(canon), weight=0.6, order=len(blocks), attrs={"lastmod": lastmod}))
	125	+ if conf >= settings.discovery_min_confidence and surface not in (Surface.OTHER, Surface.SITEMAP, Surface.HOMEPAGE) and not looks_like_trap(loc):
	126	+ discovered.append(DiscoveredUrl(url=loc, surface=surface, confidence=conf, method="sitemap"))
	127	+ text = "\n".join(u["url"] for u in urls_out)
	128	+ meta = {"url_count": len(urls_out), "child_sitemaps": children[:50], "by_surface": by_surface, "truncated": len(entries) >= limit,
	129	+ "urls": urls_out, "structured": True}
	130	+ return Extraction(text=text, blocks=blocks, title=f"Sitemap — {len(urls_out)} URLs", meta=meta, discovered=discovered[:500])
	131	+
	132	+
	133	+__all__ = ["SitemapConnector", "parse_sitemap"]

added src/companyatlas/connectors/smartrecruiters.py +67 −0

		`@@ -0,0 +1,67 @@`
	1	+"""SmartRecruiters postings — public endpoint behind `careers.smartrecruiters.com/<Company>`:
	2	+`GET https://api.smartrecruiters.com/v1/companies/{token}/postings?limit=100&offset=N` → {offset, limit, totalFound, content:[{id, name,
	3	+uuid, refNumber, releasedDate, location:{city, region, country(iso2 lower), remote, hybrid, fullLocation}, department:{label}, function:{label},
	4	+typeOfEmployment:{label}, experienceLevel:{label}}]}. Paginated by offset (bounded). Verified 2026-09-12 against `smartrecruiters`."""
	5	+from __future__ import annotations
	6	+
	7	+import re
	8	+from collections.abc import Mapping
	9	+from typing import Any
	10	+
	11	+from companyatlas.connectors._ats_base import AtsConnector
	12	+from companyatlas.connectors._util import country_code, load_json, merged_result, parse_date, text_of
	13	+from companyatlas.fetch import Fetcher, FetchResult
	14	+from companyatlas.sdk.connector import ConnectorContext, ConnectorMeta, register
	15	+from companyatlas.sdk.models import ExtractedJob
	16	+from companyatlas.taxonomy import FetchMode, Surface
	17	+
	18	+PAGE_SIZE = 100
	19	+MAX_PAGES = 10
	20	+
	21	+
	22	+@register
	23	+class SmartRecruitersConnector(AtsConnector):
	24	+ vendor = "smartrecruiters"
	25	+ token_re = re.compile(r"api\.smartrecruiters\.com/v1/companies/([a-z0-9_-]+)/postings", re.IGNORECASE)
	26	+ meta = ConnectorMeta(connector_id="smartrecruiters-v1", name="SmartRecruiters postings", version="1", category=Surface.JOBS_BOARD,
	27	+ fetch_mode=FetchMode.JSON, default_interval_s=6 * 3600, url_pattern=r"api\.smartrecruiters\.com/v1/companies/",
	28	+ priority=50, pattern_required=True, accept="application/json", description="Public SmartRecruiters posting API with offset pagination")
	29	+
	30	+ async def fetch(self, ctx: ConnectorContext, sensor: Mapping[str, Any], fetcher: Fetcher) -> FetchResult:
	31	+ base = str(sensor["url"]).split("&offset=")[0]
	32	+ first = await fetcher.get(base, etag=sensor.get("etag"), last_modified=sensor.get("last_modified"), accept="application/json")
	33	+ data = load_json(first)
	34	+ content = list(data.get("content") or []) if isinstance(data, dict) else []
	35	+ total = int(data.get("totalFound") or len(content)) if isinstance(data, dict) else len(content)
	36	+ pages = 1
	37	+ offset = len(content)
	38	+ while offset < total and pages < MAX_PAGES and content:
	39	+ page = await fetcher.get(f"{base}&offset={offset}", accept="application/json")
	40	+ more = load_json(page).get("content") or []
	41	+ if not more:
	42	+ break
	43	+ content.extend(more)
	44	+ offset += len(more)
	45	+ pages += 1
	46	+ if pages == 1:
	47	+ return first
	48	+ return merged_result(first, {"totalFound": total, "content": content}, pages=pages)
	49	+
	50	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	51	+ token = self.token(sensor) or ""
	52	+ out: list[ExtractedJob] = []
	53	+ for j in (data.get("content") if isinstance(data, dict) else []) or []:
	54	+ if not isinstance(j, dict) or not j.get("name"):
	55	+ continue
	56	+ loc = j.get("location") or {}
	57	+ jid = str(j.get("id") or j.get("uuid") or "")
	58	+ url = f"https://jobs.smartrecruiters.com/{token}/{jid}" if token and jid else None
	59	+ remote = True if loc.get("remote") else (False if loc.get("hybrid") is not None or loc.get("city") else None)
	60	+ out.append(ExtractedJob(
	61	+ title=str(j["name"]).strip(), url=url, external_id=jid, department=text_of(j.get("department")),
	62	+ location_text=text_of(loc.get("fullLocation")) or ", ".join(x for x in (loc.get("city"), loc.get("region"), loc.get("country")) if x) or None,
	63	+ city=text_of(loc.get("city")), region=text_of(loc.get("region")), country=country_code(loc.get("country")), remote=remote,
	64	+ employment_type=text_of(j.get("typeOfEmployment")), seniority=None, posted_at=parse_date(j.get("releasedDate")),
	65	+ raw={"refNumber": j.get("refNumber"), "function": text_of(j.get("function")), "experienceLevel": text_of(j.get("experienceLevel"))},
	66	+ ))
	67	+ return out

added src/companyatlas/connectors/statuspage.py +54 −0

		`@@ -0,0 +1,54 @@`
	1	+"""Atlassian Statuspage summary — `GET https://status.<company>/api/v2/summary.json` → {page:{name, updated_at}, status:{indicator, description},
	2	+components:[{name, status}], incidents:[{name, status, impact, shortlink, created_at}], scheduled_maintenances:[…]}. One block per
	3	+component and per active incident; the page-level `updated_at` is noise-normalised away. Verified 2026-09-12 against githubstatus.com."""
	4	+from __future__ import annotations
	5	+
	6	+import hashlib
	7	+from collections.abc import Mapping
	8	+from typing import Any
	9	+
	10	+from companyatlas.connectors._util import load_json, parse_date, text_of
	11	+from companyatlas.fetch import FetchResult
	12	+from companyatlas.sdk.connector import Connector, ConnectorMeta, register
	13	+from companyatlas.sdk.models import Block, ExtractedNewsItem, Extraction
	14	+from companyatlas.sdk.normalize import normalized_text, simhash
	15	+from companyatlas.taxonomy import FetchMode, Surface
	16	+
	17	+
	18	+@register
	19	+class StatuspageConnector(Connector):
	20	+ meta = ConnectorMeta(connector_id="statuspage-v1", name="Statuspage summary", version="1", category=Surface.STATUS, fetch_mode=FetchMode.JSON,
	21	+ default_interval_s=6 * 3600, url_pattern=r"/api/v2/summary\.json$", pattern_required=True, priority=40, accept="application/json",
	22	+ description="Atlassian Statuspage public summary API")
	23	+
	24	+ def extract(self, sensor: Mapping[str, Any], result: FetchResult) -> Extraction:
	25	+ data = load_json(result)
	26	+ if not isinstance(data, dict):
	27	+ raise TypeError("statuspage summary is not an object")
	28	+ status = data.get("status") or {}
	29	+ components = [c for c in (data.get("components") or []) if isinstance(c, dict) and c.get("name")]
	30	+ incidents = [i for i in (data.get("incidents") or []) if isinstance(i, dict) and i.get("name")]
	31	+ maint = [m for m in (data.get("scheduled_maintenances") or []) if isinstance(m, dict) and m.get("name")]
	32	+ blocks: list[Block] = []
	33	+ lines = [f"Status: {status.get('description') or status.get('indicator') or 'unknown'}"]
	34	+
	35	+ def add(kind: str, key: str, text: str, weight: float, path: str) -> None:
	36	+ blocks.append(Block(key=key, kind=kind, text=text, path=path, hash=hashlib.sha256(normalized_text(text).encode()).hexdigest()[:16],
	37	+ simhash=simhash(text), weight=weight, order=len(blocks)))
	38	+
	39	+ add("hero", "status:overall", lines[0], 1.5, "")
	40	+ for c in components:
	41	+ txt = f"{c['name']}: {c.get('status') or 'unknown'}"
	42	+ lines.append(txt)
	43	+ add("section", f"component:{hashlib.blake2b(str(c.get('id') or c['name']).encode(), digest_size=6).hexdigest()}", txt, 1.0, "Components")
	44	+ news: list[ExtractedNewsItem] = []
	45	+ for i in incidents + maint:
	46	+ txt = f"{i['name']} — {i.get('status') or ''} ({i.get('impact') or 'n/a'})"
	47	+ lines.append(txt)
	48	+ add("news_item", f"incident:{i.get('id') or i['name']}", txt, 1.3, "Incidents")
	49	+ link = i.get("shortlink") or result.final_url
	50	+ news.append(ExtractedNewsItem(title=str(i["name"])[:300], url=str(link), published_at=parse_date(i.get("created_at")), category="other",
	51	+ summary=text_of(i.get("impact"))))
	52	+ meta = {"indicator": status.get("indicator"), "description": status.get("description"), "component_count": len(components),
	53	+ "incident_count": len(incidents), "maintenance_count": len(maint), "page_name": (data.get("page") or {}).get("name"), "structured": True}
	54	+ return Extraction(text="\n".join(lines), blocks=blocks, title=f"{meta['page_name'] or 'Status'} — {lines[0]}", meta=meta, news=news)

added src/companyatlas/connectors/teamtailor.py +47 −0

		`@@ -0,0 +1,47 @@`
	1	+"""Teamtailor career sites publish a JSON Feed at `/jobs.json` on the career-site host (`<token>.teamtailor.com` or a custom
	2	+`career.<company>.com`): {version, title, home_page_url, items:[{id, title, url, date_published, _jobposting:{identifier:{value}, datePosted,
	3	+jobLocation:[{address:{addressLocality, addressRegion, addressCountry}}], employmentType?, jobLocationType?}}]}.
	4	+Verified 2026-09-12 against https://career.teamtailor.com/jobs.json (fixture trimmed to 20 items, descriptions removed). When a career
	5	+site has no feed (404) discovery keeps the HTML careers sensor instead."""
	6	+from __future__ import annotations
	7	+
	8	+import re
	9	+from collections.abc import Mapping
	10	+from typing import Any
	11	+
	12	+from companyatlas.connectors._ats_base import AtsConnector
	13	+from companyatlas.connectors._util import country_code, dig, parse_date, text_of
	14	+from companyatlas.sdk.connector import ConnectorMeta, register
	15	+from companyatlas.sdk.models import ExtractedJob
	16	+from companyatlas.taxonomy import FetchMode, Surface
	17	+
	18	+
	19	+@register
	20	+class TeamtailorConnector(AtsConnector):
	21	+ vendor = "teamtailor"
	22	+ token_re = re.compile(r"https?://([a-z0-9-]+)\.teamtailor\.com/jobs\.json", re.IGNORECASE)
	23	+ meta = ConnectorMeta(connector_id="teamtailor-v1", name="Teamtailor jobs feed", version="1", category=Surface.JOBS_BOARD, fetch_mode=FetchMode.JSON,
	24	+ default_interval_s=6 * 3600, url_pattern=r"/jobs\.json(\?\|$)", pattern_required=True, priority=45, accept="application/feed+json,application/json",
	25	+ description="Teamtailor career-site JSON Feed (/jobs.json)")
	26	+
	27	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	28	+ out: list[ExtractedJob] = []
	29	+ for it in (data.get("items") if isinstance(data, dict) else []) or []:
	30	+ if not isinstance(it, dict) or not it.get("title"):
	31	+ continue
	32	+ jp = it.get("_jobposting") or {}
	33	+ locs = jp.get("jobLocation") or []
	34	+ if isinstance(locs, dict):
	35	+ locs = [locs]
	36	+ addr = dig(locs[0], "address", default={}) if locs and isinstance(locs[0], dict) else {}
	37	+ city, region, country = text_of(addr.get("addressLocality")), text_of(addr.get("addressRegion")), addr.get("addressCountry")
	38	+ ident = dig(jp, "identifier", "value")
	39	+ loc_type = (jp.get("jobLocationType") or "").lower()
	40	+ out.append(ExtractedJob(
	41	+ title=str(it["title"]), url=it.get("url"), external_id=str(ident or it.get("id") or ""),
	42	+ location_text=", ".join(x for x in (city, country_code(country) or text_of(country)) if x) or None, city=city,
	43	+ region=region if region and region.lower() not in ("europe", "emea", "apac", "americas") else None, country=country_code(country),
	44	+ remote=True if loc_type == "telecommute" else None, employment_type=text_of(jp.get("employmentType")),
	45	+ posted_at=parse_date(jp.get("datePosted") or it.get("date_published")), raw={"locations": len(locs)},
	46	+ ))
	47	+ return out

added src/companyatlas/connectors/workable.py +42 −0

		`@@ -0,0 +1,42 @@`
	1	+"""Workable — public widget endpoint behind `apply.workable.com/<token>`:
	2	+`GET https://apply.workable.com/api/v1/widget/accounts/{token}` → {name, description, jobs:[{title, shortcode, code, employment_type,
	3	+telecommuting, department, url, published_on, created_at, country, city, state, function, locations:[{country, countryCode, city, region}]}]}.
	4	+Verified 2026-09-12 against `epignosis` (fixture trimmed, descriptions removed)."""
	5	+from __future__ import annotations
	6	+
	7	+import re
	8	+from collections.abc import Mapping
	9	+from typing import Any
	10	+
	11	+from companyatlas.connectors._ats_base import AtsConnector
	12	+from companyatlas.connectors._util import country_code, parse_date, text_of
	13	+from companyatlas.sdk.connector import ConnectorMeta, register
	14	+from companyatlas.sdk.models import ExtractedJob
	15	+from companyatlas.taxonomy import FetchMode, Surface
	16	+
	17	+
	18	+@register
	19	+class WorkableConnector(AtsConnector):
	20	+ vendor = "workable"
	21	+ token_re = re.compile(r"apply\.workable\.com/api/v1/widget/accounts/([a-z0-9_-]+)", re.IGNORECASE)
	22	+ meta = ConnectorMeta(connector_id="workable-v1", name="Workable widget", version="1", category=Surface.JOBS_BOARD, fetch_mode=FetchMode.JSON,
	23	+ default_interval_s=6 * 3600, url_pattern=r"apply\.workable\.com/api/v1/widget/accounts/", pattern_required=True, priority=50,
	24	+ accept="application/json", description="Public Workable widget API")
	25	+
	26	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	27	+ out: list[ExtractedJob] = []
	28	+ for j in (data.get("jobs") if isinstance(data, dict) else []) or []:
	29	+ if not isinstance(j, dict) or not j.get("title"):
	30	+ continue
	31	+ locs = j.get("locations") or []
	32	+ first = locs[0] if locs and isinstance(locs[0], dict) else {}
	33	+ city, state, country = j.get("city") or first.get("city"), j.get("state") or first.get("region"), j.get("country") or first.get("country")
	34	+ out.append(ExtractedJob(
	35	+ title=str(j["title"]), url=j.get("url") or j.get("shortlink"), external_id=str(j.get("shortcode") or j.get("code") or ""),
	36	+ department=text_of(j.get("department")), location_text=", ".join(x for x in (city, state, country) if x) or None,
	37	+ city=text_of(city), region=text_of(state), country=country_code(first.get("countryCode")) or country_code(country),
	38	+ remote=bool(j.get("telecommuting")) if j.get("telecommuting") is not None else None, employment_type=text_of(j.get("employment_type")),
	39	+ posted_at=parse_date(j.get("published_on") or j.get("created_at")),
	40	+ raw={"function": j.get("function"), "experience": j.get("experience"), "industry": j.get("industry")},
	41	+ ))
	42	+ return out

added src/companyatlas/connectors/workday.py +73 −0

		`@@ -0,0 +1,73 @@`
	1	+"""Workday public career sites — the JSON search endpoint the site's own frontend calls (spec §13 mode C, provenance stored):
	2	+`POST https://{tenant}.{wdN}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs` body {"appliedFacets":{},"limit":20,"offset":N,"searchText":""}
	3	+→ {total, jobPostings:[{title, externalPath, locationsText, postedOn, bulletFields:[reqId]}]}. Paginated by offset, bounded to
	4	+`MAX_JOBS_WD` postings. Verified 2026-09-12 against nvidia.wd5 / NVIDIAExternalCareerSite (fixture = first page)."""
	5	+from __future__ import annotations
	6	+
	7	+import re
	8	+from collections.abc import Mapping
	9	+from typing import Any
	10	+
	11	+from companyatlas.connectors._ats_base import AtsConnector
	12	+from companyatlas.connectors._util import load_json, merged_result
	13	+from companyatlas.fetch import Fetcher, FetchResult
	14	+from companyatlas.sdk.connector import ConnectorContext, ConnectorMeta, register
	15	+from companyatlas.sdk.models import ExtractedJob
	16	+from companyatlas.taxonomy import FetchMode, Surface
	17	+
	18	+PAGE_LIMIT = 20
	19	+MAX_JOBS_WD = 400
	20	+API_RE = re.compile(r"https?://([a-z0-9-]+)\.(wd\d+)\.myworkdayjobs\.com/wday/cxs/([a-z0-9-]+)/([A-Za-z0-9_-]+)/jobs", re.IGNORECASE)
	21	+REQ_RE = re.compile(r"_([A-Z]{1,4}[-_]?\d{3,}[A-Z0-9-]*)$")
	22	+
	23	+
	24	+@register
	25	+class WorkdayConnector(AtsConnector):
	26	+ vendor = "workday"
	27	+ meta = ConnectorMeta(connector_id="workday-v1", name="Workday career site (public JSON)", version="1", category=Surface.JOBS_BOARD,
	28	+ fetch_mode=FetchMode.JSON, default_interval_s=12 * 3600, url_pattern=r"myworkdayjobs\.com/wday/cxs/", pattern_required=True, priority=50,
	29	+ accept="application/json", description="Public Workday CXS job search endpoint (POST, paginated)")
	30	+
	31	+ def token(self, sensor: Mapping[str, Any]) -> str \| None:
	32	+ m = API_RE.search(str(sensor.get("url") or ""))
	33	+ return f"{m.group(1)}/{m.group(4)}" if m else None
	34	+
	35	+ async def fetch(self, ctx: ConnectorContext, sensor: Mapping[str, Any], fetcher: Fetcher) -> FetchResult:
	36	+ url = str(sensor["url"])
	37	+ postings: list[dict[str, Any]] = []
	38	+ first: FetchResult \| None = None
	39	+ total = 0
	40	+ offset = 0
	41	+ pages = 0
	42	+ while True:
	43	+ body = {"appliedFacets": {}, "limit": PAGE_LIMIT, "offset": offset, "searchText": ""}
	44	+ res = await fetcher.post_json(url, body, headers={"Content-Type": "application/json"})
	45	+ pages += 1
	46	+ data = load_json(res)
	47	+ if first is None:
	48	+ first = res
	49	+ total = int(data.get("total") or 0) if isinstance(data, dict) else 0
	50	+ page = (data.get("jobPostings") if isinstance(data, dict) else None) or []
	51	+ postings.extend(p for p in page if isinstance(p, dict))
	52	+ offset += len(page)
	53	+ if not page or offset >= total or offset >= MAX_JOBS_WD:
	54	+ break
	55	+ assert first is not None
	56	+ return merged_result(first, {"total": total, "jobPostings": postings, "truncated": total > len(postings)}, pages=pages)
	57	+
	58	+ def parse_jobs(self, data: Any, sensor: Mapping[str, Any]) -> list[ExtractedJob]:
	59	+ m = API_RE.search(str(sensor.get("url") or ""))
	60	+ base = f"https://{m.group(1)}.{m.group(2)}.myworkdayjobs.com/{m.group(4)}" if m else ""
	61	+ out: list[ExtractedJob] = []
	62	+ for j in (data.get("jobPostings") if isinstance(data, dict) else []) or []:
	63	+ if not isinstance(j, dict) or not j.get("title"):
	64	+ continue
	65	+ path = str(j.get("externalPath") or "")
	66	+ bullets = [b for b in (j.get("bulletFields") or []) if isinstance(b, str)]
	67	+ req = bullets[0] if bullets else None
	68	+ if not req:
	69	+ mm = REQ_RE.search(path)
	70	+ req = mm.group(1) if mm else None
	71	+ out.append(ExtractedJob(title=str(j["title"]), url=(base + path) if base and path else None, external_id=req or path or None,
	72	+ location_text=j.get("locationsText") or None, raw={"postedOn": j.get("postedOn"), "bulletFields": bullets[:3]}))
	73	+ return out

modified src/companyatlas/fetch.py +28 −6

		`@@ -14,6 +14,7 @@ import socket`
14	14	import time
15	15	from dataclasses import dataclass, field
16	16	from datetime import UTC, datetime
	17	+from typing import Self
17	18	from urllib.parse import urljoin, urlparse
18	19	from urllib.robotparser import RobotFileParser
19	20
		`@@ -294,7 +295,7 @@ class Fetcher:`
294	295	self._http2 = http2
295	296	self._max_connections = max_connections or max(16, settings.fetch_concurrency * 2)
296	297
297		− async def __aenter__(self) -> Fetcher:
	298	+ async def __aenter__(self) -> Self:
298	299	await self.open()
299	300	return self
300	301
		`@@ -321,7 +322,22 @@ class Fetcher:`
321	322	async def get(self, url: str, *, etag: str \| None = None, last_modified: str \| None = None, min_bytes: int = 1,
322	323	retries: int \| None = None, accept: str \| None = None, rate_per_min: int \| None = None, respect_robots: bool = True,
323	324	max_bytes: int \| None = None) -> FetchResult:
	325	+ return await self.request("GET", url, etag=etag, last_modified=last_modified, min_bytes=min_bytes, retries=retries, accept=accept,
	326	+ rate_per_min=rate_per_min, respect_robots=respect_robots, max_bytes=max_bytes)
	327	+
	328	+ async def post_json(self, url: str, payload: object, , accept: str \| None = "application/json", *kw: object) -> FetchResult:
	329	+ """POST a JSON body to a public endpoint that a public page itself calls (spec §13 mode C, e.g. Workday job search)."""
	330	+ return await self.request("POST", url, json=payload, accept=accept, **kw) # type: ignore[arg-type]
	331	+
	332	+ async def request(self, method: str, url: str, *, json: object \| None = None, headers: dict[str, str] \| None = None,
	333	+ etag: str \| None = None, last_modified: str \| None = None, min_bytes: int = 1, retries: int \| None = None,
	334	+ accept: str \| None = None, rate_per_min: int \| None = None, respect_robots: bool = True,
	335	+ max_bytes: int \| None = None) -> FetchResult:
	336	+ """Same guarantees as `get()` (SSRF guard on every hop, robots, per-domain governor, size caps, failure classes) for any
	337	+ method. Non-GET requests are never retried on transport errors beyond the configured retries and never follow redirects
	338	+ across registrable domains with the body."""
324	339	client = self.client
	340	+ method = method.upper()
325	341	retries = settings.fetch_retries if retries is None else retries
326	342	try:
327	343	await validate_destination_async(url)
		`@@ -334,13 +350,14 @@ class Fetcher:`
334	350	governor.set_crawl_delay(domain, delay)
335	351	if not allowed:
336	352	raise BlockedError(f"robots.txt disallows {url}", url=url, failure=FailureClass.ROBOTS)
337		− headers: dict[str, str] = {}
	353	+ req_headers: dict[str, str] = dict(headers or {})
338	354	if etag:
339		− headers["If-None-Match"] = etag
	355	+ req_headers["If-None-Match"] = etag
340	356	if last_modified:
341		− headers["If-Modified-Since"] = last_modified
	357	+ req_headers["If-Modified-Since"] = last_modified
342	358	if accept:
343		− headers["Accept"] = accept
	359	+ req_headers["Accept"] = accept
	360	+ headers = req_headers
344	361	current = url
345	362	hops = 0
346	363	attempt = 0
		`@@ -348,7 +365,7 @@ class Fetcher:`
348	365	sem = await governor.acquire(registrable_domain(current), rate_per_min or settings.default_rate_per_min)
349	366	t0 = time.perf_counter()
350	367	try:
351		− async with client.stream("GET", current, headers=headers) as r:
	368	+ async with client.stream(method, current, headers=headers, json=json) as r:
352	369	if r.status_code == 304:
353	370	raise NotModified(int((time.perf_counter() - t0) * 1000))
354	371	if r.status_code in REDIRECT_STATUS:
		`@@ -367,6 +384,11 @@ class Fetcher:`
367	384	if registrable_domain(nxt) != registrable_domain(current):
368	385	headers.pop("If-None-Match", None)
369	386	headers.pop("If-Modified-Since", None)
	387	+ if method != "GET":
	388	+ raise FetchError(f"{method} redirected off-domain {current} → {nxt}", status=r.status_code, url=url,
	389	+ failure=FailureClass.REDIRECT)
	390	+ if r.status_code in (301, 302, 303) and method != "GET":
	391	+ method, json = "GET", None # per RFC 9110 user agents switch to GET
370	392	current = nxt
371	393	continue
372	394	if r.status_code in TRANSIENT_STATUS and attempt < retries:

added src/companyatlas/sdk/connector.py +201 −0

		`@@ -0,0 +1,201 @@`
	1	+"""Connector SDK (spec §8, §105–106): metadata, base class, registry.
	2	+
	3	+ @register
	4	+ class GreenhouseConnector(Connector):
	5	+ meta = ConnectorMeta(connector_id="greenhouse-v1", name="Greenhouse job board", version="1", category=Surface.JOBS_BOARD, …)
	6	+ def extract(self, sensor, result) -> Extraction: ...
	7	+
	8	+A connector is a strategy; a sensor row binds a connector to a URL for one company. `connector_id` (which embeds the version)
	9	+is stored on every observation/snapshot as `connector_version`, so history remains reproducible when connectors evolve — bump the
	10	+version (new id) when extraction behaviour changes materially.
	11	+
	12	+Only `fetch.Fetcher` touches the network; connectors receive it and must not open sockets themselves.
	13	+"""
	14	+from __future__ import annotations
	15	+
	16	+import importlib
	17	+import logging
	18	+import pkgutil
	19	+import re
	20	+from collections.abc import Mapping
	21	+from dataclasses import dataclass, field
	22	+from typing import Any, ClassVar
	23	+
	24	+from companyatlas.fetch import Fetcher, FetchResult
	25	+from companyatlas.sdk.models import DiscoveredUrl, Extraction
	26	+from companyatlas.taxonomy import SURFACE_BASE_INTERVAL_S, FetchMode, Surface
	27	+
	28	+log = logging.getLogger(__name__)
	29	+
	30	+
	31	+@dataclass(frozen=True, slots=True)
	32	+class ConnectorMeta:
	33	+ connector_id: str # e.g. "generic-html-v1" — stable id incl. version
	34	+ name: str
	35	+ version: str
	36	+ category: str # primary Surface value
	37	+ fetch_mode: str = FetchMode.HTTP
	38	+ supports_discovery: bool = False
	39	+ supports_incremental: bool = True # honours etag / last-modified / delta semantics
	40	+ default_interval_s: int = 86400
	41	+ surfaces: tuple[str, ...] = () # surfaces this connector can serve (empty = category only)
	42	+ url_pattern: str \| None = None # regex on the sensor URL; when it matches the connector is preferred
	43	+ pattern_required: bool = False # vendor / API connectors: never auto-selected unless the URL pattern matches
	44	+ priority: int = 10 # higher wins among candidates for the same surface
	45	+ accept: str \| None = None # Accept header override
	46	+ max_bytes: int \| None = None
	47	+ description: str = ""
	48	+
	49	+
	50	+@dataclass(slots=True)
	51	+class ConnectorContext:
	52	+ """What a connector may know about the company it is observing (read-only)."""
	53	+ company: Mapping[str, Any]
	54	+ extra: dict[str, Any] = field(default_factory=dict)
	55	+
	56	+ @property
	57	+ def canonical_domain(self) -> str:
	58	+ return str(self.company.get("canonical_domain") or "")
	59	+
	60	+
	61	+class Connector:
	62	+ meta: ClassVar[ConnectorMeta]
	63	+
	64	+ # ---------------------------------------------------------------- fetch
	65	+ async def fetch(self, ctx: ConnectorContext, sensor: Mapping[str, Any], fetcher: Fetcher) -> FetchResult:
	66	+ """Default: conditional GET using the sensor's stored validators. Raises NotModified / FetchError."""
	67	+ return await fetcher.get(str(sensor["url"]), etag=sensor.get("etag"), last_modified=sensor.get("last_modified"),
	68	+ accept=self.meta.accept, max_bytes=self.meta.max_bytes)
	69	+
	70	+ # ---------------------------------------------------------------- extract
	71	+ def extract(self, sensor: Mapping[str, Any], result: FetchResult) -> Extraction: # pragma: no cover - abstract
	72	+ raise NotImplementedError
	73	+
	74	+ # ---------------------------------------------------------------- discover (optional)
	75	+ async def discover(self, ctx: ConnectorContext, sensor: Mapping[str, Any], extraction: Extraction) -> list[DiscoveredUrl]:
	76	+ return list(extraction.discovered)
	77	+
	78	+ # ---------------------------------------------------------------- selection
	79	+ @classmethod
	80	+ def score(cls, surface: str, url: str) -> int:
	81	+ """0 = cannot serve; higher = better fit."""
	82	+ m = cls.meta
	83	+ s = 0
	84	+ if m.url_pattern and re.search(m.url_pattern, url, re.IGNORECASE):
	85	+ s += 100
	86	+ elif m.pattern_required:
	87	+ return 0
	88	+ if surface == m.category:
	89	+ s += 20
	90	+ elif surface in m.surfaces:
	91	+ s += 10
	92	+ elif "*" in m.surfaces:
	93	+ s += 5
	94	+ return s + m.priority if s > 0 else 0
	95	+
	96	+ @property
	97	+ def connector_id(self) -> str:
	98	+ return self.meta.connector_id
	99	+
	100	+ def __repr__(self) -> str:
	101	+ return f"<Connector {self.meta.connector_id}>"
	102	+
	103	+
	104	+# ------------------------------------------------------------------------------------------------------------ registry
	105	+
	106	+_REGISTRY: dict[str, type[Connector]] = {}
	107	+_INSTANCES: dict[str, Connector] = {}
	108	+_LOADED = False
	109	+
	110	+
	111	+def register(cls: type[Connector]) -> type[Connector]:
	112	+ meta = getattr(cls, "meta", None)
	113	+ if not isinstance(meta, ConnectorMeta):
	114	+ raise TypeError(f"{cls.__name__} must define `meta = ConnectorMeta(...)`")
	115	+ if meta.connector_id in _REGISTRY and _REGISTRY[meta.connector_id] is not cls:
	116	+ log.warning("connector id re-registered", extra={"connector_id": meta.connector_id})
	117	+ _REGISTRY[meta.connector_id] = cls
	118	+ _INSTANCES.pop(meta.connector_id, None)
	119	+ return cls
	120	+
	121	+
	122	+def load_all() -> None:
	123	+ """Import every `companyatlas.connectors.*` module once so decorators run."""
	124	+ global _LOADED
	125	+ if _LOADED:
	126	+ return
	127	+ _LOADED = True
	128	+ try:
	129	+ import companyatlas.connectors as pkg
	130	+ except ImportError: # pragma: no cover
	131	+ return
	132	+ for mod in pkgutil.iter_modules(pkg.__path__):
	133	+ if mod.name.startswith("_"):
	134	+ continue
	135	+ try:
	136	+ importlib.import_module(f"companyatlas.connectors.{mod.name}")
	137	+ except Exception:
	138	+ log.exception("connector module failed to import", extra={"connector_module": mod.name})
	139	+
	140	+
	141	+def get(connector_id: str) -> Connector:
	142	+ load_all()
	143	+ inst = _INSTANCES.get(connector_id)
	144	+ if inst is None:
	145	+ cls = _REGISTRY.get(connector_id)
	146	+ if cls is None:
	147	+ # tolerate an older version id (e.g. "greenhouse-v0") by falling back to the newest of the same family
	148	+ family = connector_id.rsplit("-v", 1)[0]
	149	+ candidates = sorted((k for k in _REGISTRY if k.rsplit("-v", 1)[0] == family), reverse=True)
	150	+ if not candidates:
	151	+ raise KeyError(f"unknown connector {connector_id!r}")
	152	+ cls = _REGISTRY[candidates[0]]
	153	+ inst = _INSTANCES[connector_id] = cls()
	154	+ return inst
	155	+
	156	+
	157	+def all_connectors() -> list[Connector]:
	158	+ load_all()
	159	+ return [get(k) for k in sorted(_REGISTRY)]
	160	+
	161	+
	162	+def for_surface(surface: str, url: str) -> Connector:
	163	+ """Best connector for a (surface, url) pair — ATS/JSON/feed/sitemap by URL pattern, else the generic HTML connector."""
	164	+ load_all()
	165	+ best: tuple[int, str] \| None = None
	166	+ for cid, cls in _REGISTRY.items():
	167	+ s = cls.score(surface, url)
	168	+ if s > 0 and (best is None or s > best[0] or (s == best[0] and cid < best[1])):
	169	+ best = (s, cid)
	170	+ if best is None:
	171	+ return get("generic-html-v1")
	172	+ return get(best[1])
	173	+
	174	+
	175	+def default_interval(surface: str, connector: Connector \| None = None) -> int:
	176	+ if connector is not None and connector.meta.default_interval_s:
	177	+ return min(int(SURFACE_BASE_INTERVAL_S.get(surface, 86400)), connector.meta.default_interval_s)
	178	+ return int(SURFACE_BASE_INTERVAL_S.get(surface, 86400))
	179	+
	180	+
	181	+async def sync_connectors_table(conn: Any) -> int:
	182	+ """Upsert one `connectors` row per registered connector (idempotent). Returns the number of connectors."""
	183	+ from companyatlas.db import execute
	184	+
	185	+ n = 0
	186	+ for c in all_connectors():
	187	+ m = c.meta
	188	+ await execute(conn, """
	189	+ insert into connectors (id, name, version, category, fetch_mode, supports_discovery, supports_incremental, default_interval_s)
	190	+ values (:id, :name, :version, :category, :fetch_mode, :sd, :si, :interval)
	191	+ on conflict (id) do update set name = excluded.name, version = excluded.version, category = excluded.category,
	192	+ fetch_mode = excluded.fetch_mode, supports_discovery = excluded.supports_discovery,
	193	+ supports_incremental = excluded.supports_incremental, default_interval_s = excluded.default_interval_s, updated_at = now()
	194	+ """, id=m.connector_id, name=m.name, version=m.version, category=str(m.category), fetch_mode=str(m.fetch_mode),
	195	+ sd=m.supports_discovery, si=m.supports_incremental, interval=int(m.default_interval_s))
	196	+ n += 1
	197	+ return n
	198	+
	199	+
	200	+__all__ = ["Connector", "ConnectorContext", "ConnectorMeta", "Surface", "all_connectors", "default_interval", "for_surface", "get", "load_all",
	201	+ "register", "sync_connectors_table"]

added src/companyatlas/sdk/diff.py +307 −0

		`@@ -0,0 +1,307 @@`
	1	+"""Block-level diff + significance (spec §19–20).
	2	+
	3	+ compare(before_blocks, after_blocks, surface=…, before_text=…, after_text=…, structured_delta=…, history=…) -> BlockDiff
	4	+
	5	+Matching: exact block `key` → simhash near-match (Hamming ≤ `NEAR_HAMMING`) → rapidfuzz ratio ≥ `MODIFIED_MIN_RATIO` for modified
	6	+blocks; everything else is added/removed. Moves are matched blocks whose relative order changed (LIS on positions).
	7	+
	8	+Significance ∈ [0, 1] is deterministic and explained by `reasons`: weighted changed share of content blocks (nav/footer/header do not
	9	+count), surface importance, boosts for typed deltas (prices, people, jobs, locations, products, news), novelty from the sensor's
	10	+history, and hard caps for classic noise (footer-only churn, tiny edits, counters). Bands (spec §20): < 0.20 noise · < 0.40 minor
	11	+· < 0.65 meaningful · < 0.85 major · ≥ 0.85 critical.
	12	+"""
	13	+from __future__ import annotations
	14	+
	15	+from collections.abc import Mapping
	16	+from typing import Any
	17	+
	18	+from rapidfuzz import fuzz
	19	+from rapidfuzz.distance import Indel
	20	+
	21	+from companyatlas.sdk.models import Block, BlockDelta, BlockDiff
	22	+from companyatlas.sdk.normalize import LOW_VALUE_KINDS, hamming, normalized_text
	23	+from companyatlas.taxonomy import SURFACE_IMPORTANCE
	24	+
	25	+DIFF_VERSION = "diff-v1"
	26	+
	27	+NEAR_HAMMING = 6 # simhash distance considered "same block, edited"
	28	+MODIFIED_MIN_RATIO = 0.55 # rapidfuzz ratio (0–1) for a near-match to count as modified rather than add+remove
	29	+TINY_EDIT_SIMILARITY = 0.97 # modified blocks at/above this similarity are cosmetic
	30	+BLOCK_LEN_CAP = 1500 # a single giant block must not dominate the share
	31	+TEXT_COMPARE_CAP = 60_000 # characters compared for text_delta_ratio
	32	+
	33	+# Typed-delta floors: when the pipeline reconciled real entities, the change is at least this significant (spec §20 "affected entities").
	34	+TYPED_FLOORS: dict[str, float] = {
	35	+ "plans.price_changed": 0.70, "plans.added": 0.60, "plans.removed": 0.60,
	36	+ "people.added_executive": 0.68, "people.removed_executive": 0.68, "people.added": 0.50, "people.removed": 0.50, "people.title_changed": 0.55,
	37	+ "jobs.added": 0.45, "jobs.removed": 0.42,
	38	+ "locations.new_countries": 0.70, "locations.added": 0.55, "locations.removed": 0.52,
	39	+ "products.added": 0.55, "products.removed": 0.55,
	40	+ "news.added": 0.45,
	41	+ "meta.title_changed": 0.30,
	42	+}
	43	+JOBS_SCALE_FLOOR = 0.62 # many jobs added/removed at once (≥ JOBS_SCALE_COUNT or ≥ 25 % of open jobs)
	44	+JOBS_SCALE_COUNT = 10
	45	+NOVELTY_STABLE_RUNS = 20 # consecutive unchanged runs before a change counts as "novel"
	46	+NOVELTY_BOOST = 1.10
	47	+CHURN_RATE = 0.5 # changes per observation above which the page is considered volatile
	48	+CHURN_PENALTY = 0.85
	49	+CONTENT_BASE, CONTENT_SPAN, CONTENT_EXP = 0.12, 0.72, 0.7 # pure content share maps to [0.12, 0.84] — critical needs typed deltas / novelty
	50	+NOISE_CAP = 0.19
	51	+LOW_VALUE_CAP = 0.15
	52	+
	53	+
	54	+def _content_len(b: Block) -> float:
	55	+ return float(min(len(b.text), BLOCK_LEN_CAP)) * max(0.05, b.weight)
	56	+
	57	+
	58	+def _delta(b: Block, *, before: str \| None, after: str \| None, similarity: float \| None = None) -> BlockDelta:
	59	+ return BlockDelta(key=b.key, kind=b.kind, path=b.path, before=before, after=after, weight=b.weight, similarity=similarity)
	60	+
	61	+
	62	+def _ratio(a: str, b: str) -> float:
	63	+ if not a and not b:
	64	+ return 1.0
	65	+ return fuzz.ratio(normalized_text(a)[:BLOCK_LEN_CAP * 2], normalized_text(b)[:BLOCK_LEN_CAP * 2]) / 100.0
	66	+
	67	+
	68	+def _lis_positions(seq: list[int]) -> set[int]:
	69	+ """Indices (into seq) of one longest strictly increasing subsequence — items outside it moved."""
	70	+ if not seq:
	71	+ return set()
	72	+ import bisect
	73	+
	74	+ tails: list[int] = []
	75	+ tails_idx: list[int] = []
	76	+ prev = [-1] * len(seq)
	77	+ for i, v in enumerate(seq):
	78	+ pos = bisect.bisect_left(tails, v)
	79	+ if pos == len(tails):
	80	+ tails.append(v)
	81	+ tails_idx.append(i)
	82	+ else:
	83	+ tails[pos] = v
	84	+ tails_idx[pos] = i
	85	+ prev[i] = tails_idx[pos - 1] if pos > 0 else -1
	86	+ out: set[int] = set()
	87	+ k = tails_idx[-1]
	88	+ while k != -1:
	89	+ out.add(k)
	90	+ k = prev[k]
	91	+ return out
	92	+
	93	+
	94	+def match_blocks(before: list[Block], after: list[Block]) -> tuple[list[tuple[Block, Block]], list[Block], list[Block]]:
	95	+ """Return (pairs, removed, added). Pairs include exact-key matches and near matches (simhash / fuzzy)."""
	96	+ by_key_after: dict[str, Block] = {}
	97	+ for b in after:
	98	+ by_key_after.setdefault(b.key, b)
	99	+ pairs: list[tuple[Block, Block]] = []
	100	+ used_after: set[int] = set()
	101	+ unmatched_before: list[Block] = []
	102	+ for b in before:
	103	+ a = by_key_after.get(b.key)
	104	+ if a is not None and id(a) not in used_after:
	105	+ pairs.append((b, a))
	106	+ used_after.add(id(a))
	107	+ else:
	108	+ unmatched_before.append(b)
	109	+ remaining_after = [a for a in after if id(a) not in used_after]
	110	+ # near matches: same kind, closest simhash within NEAR_HAMMING, tie-break by same path then fuzzy ratio
	111	+ still_before: list[Block] = []
	112	+ for b in unmatched_before:
	113	+ best: tuple[float, int, Block] \| None = None
	114	+ for a in remaining_after:
	115	+ if a.kind != b.kind or id(a) in used_after:
	116	+ continue
	117	+ d = hamming(a.simhash, b.simhash) if (a.simhash and b.simhash) else 64
	118	+ if d <= NEAR_HAMMING:
	119	+ score = (1.0 if a.path == b.path else 0.0, -d)
	120	+ if best is None or score > (best[0], -best[1]):
	121	+ best = (score[0], d, a)
	122	+ if best is None:
	123	+ # fuzzy fallback for short/edited blocks (bounded: only blocks of the same kind and path)
	124	+ cands = [a for a in remaining_after if a.kind == b.kind and a.path == b.path and id(a) not in used_after]
	125	+ best_r = 0.0
	126	+ best_a: Block \| None = None
	127	+ for a in cands[:60]:
	128	+ r = _ratio(b.text, a.text)
	129	+ if r > best_r:
	130	+ best_r, best_a = r, a
	131	+ if best_a is not None and best_r >= MODIFIED_MIN_RATIO:
	132	+ best = (0.0, 64, best_a)
	133	+ if best is not None:
	134	+ pairs.append((b, best[2]))
	135	+ used_after.add(id(best[2]))
	136	+ else:
	137	+ still_before.append(b)
	138	+ added = [a for a in after if id(a) not in used_after]
	139	+ return pairs, still_before, added
	140	+
	141	+
	142	+def compare(before_blocks: list[Block], after_blocks: list[Block], *, surface: str, before_text: str = "", after_text: str = "",
	143	+ structured_delta: Mapping[str, Any] \| None = None, history: Mapping[str, Any] \| None = None) -> BlockDiff:
	144	+ sd = structured_delta or {}
	145	+ hist = history or {}
	146	+ diff = BlockDiff()
	147	+ pairs, removed, added = match_blocks(before_blocks, after_blocks)
	148	+
	149	+ modified_pairs: list[tuple[Block, Block, float]] = []
	150	+ for b, a in pairs:
	151	+ if b.hash == a.hash:
	152	+ continue
	153	+ sim = _ratio(b.text, a.text)
	154	+ modified_pairs.append((b, a, sim))
	155	+ # moved: exact matches whose order is not increasing
	156	+ order_after = {id(a): a.order for _b, a in pairs}
	157	+ seq = [order_after[id(a)] for _b, a in sorted(pairs, key=lambda p: p[0].order)]
	158	+ keep = _lis_positions(seq)
	159	+ sorted_pairs = sorted(pairs, key=lambda p: p[0].order)
	160	+ for i, (b, _a) in enumerate(sorted_pairs):
	161	+ if i not in keep:
	162	+ diff.moved.append(b.key)
	163	+
	164	+ diff.added = [_delta(a, before=None, after=a.text) for a in added]
	165	+ diff.removed = [_delta(b, before=b.text, after=None) for b in removed]
	166	+ diff.modified = [_delta(a, before=b.text, after=a.text, similarity=round(sim, 4)) for b, a, sim in modified_pairs]
	167	+
	168	+ # ---------------------------------------------------------------- weighted changed share (content blocks only)
	169	+ total_before = sum(_content_len(b) for b in before_blocks if b.kind not in LOW_VALUE_KINDS)
	170	+ total_after = sum(_content_len(b) for b in after_blocks if b.kind not in LOW_VALUE_KINDS)
	171	+ total = max(total_before, total_after, 1.0)
	172	+ changed = 0.0
	173	+ low_value_changed = 0.0
	174	+ for a in added:
	175	+ if a.kind in LOW_VALUE_KINDS:
	176	+ low_value_changed += _content_len(a)
	177	+ else:
	178	+ changed += _content_len(a)
	179	+ for b in removed:
	180	+ if b.kind in LOW_VALUE_KINDS:
	181	+ low_value_changed += _content_len(b)
	182	+ else:
	183	+ changed += _content_len(b)
	184	+ for b, a, sim in modified_pairs:
	185	+ amount = max(_content_len(a), _content_len(b)) * (1.0 - sim)
	186	+ if a.kind in LOW_VALUE_KINDS:
	187	+ low_value_changed += amount
	188	+ else:
	189	+ changed += amount
	190	+ changed_share = min(1.0, changed / total)
	191	+ diff.similarity = round(max(0.0, 1.0 - changed_share), 4)
	192	+
	193	+ # ---------------------------------------------------------------- text delta ratio
	194	+ if before_text or after_text:
	195	+ nb = normalized_text(before_text)[:TEXT_COMPARE_CAP]
	196	+ na = normalized_text(after_text)[:TEXT_COMPARE_CAP]
	197	+ diff.text_delta_ratio = round(Indel.normalized_distance(nb, na), 4) if (nb or na) else 0.0
	198	+ else:
	199	+ diff.text_delta_ratio = round(changed_share, 4)
	200	+
	201	+ # ---------------------------------------------------------------- significance
	202	+ reasons: list[str] = []
	203	+ content_deltas = [d for d in (diff.added + diff.removed + diff.modified) if d.kind not in LOW_VALUE_KINDS]
	204	+ low_value_deltas = [d for d in (diff.added + diff.removed + diff.modified) if d.kind in LOW_VALUE_KINDS]
	205	+ typed_keys = _typed_keys(sd)
	206	+
	207	+ if not content_deltas and not low_value_deltas and not typed_keys and diff.text_delta_ratio < 0.001:
	208	+ diff.significance = 0.0
	209	+ diff.reasons = ["identical"] if not diff.moved else ["blocks reordered only"]
	210	+ return diff
	211	+
	212	+ importance = float(SURFACE_IMPORTANCE.get(surface, 0.3))
	213	+ importance_factor = 0.7 + 0.3 * importance
	214	+ if content_deltas:
	215	+ base = CONTENT_BASE + CONTENT_SPAN * (changed_share ** CONTENT_EXP)
	216	+ reasons.append(f"content changed share {changed_share:.3f} over {len(content_deltas)} block(s)")
	217	+ elif typed_keys:
	218	+ base = 0.2
	219	+ reasons.append("typed delta without block-level content change")
	220	+ else:
	221	+ base = 0.05
	222	+ reasons.append("only nav/header/footer blocks changed")
	223	+ sig = base * importance_factor
	224	+ reasons.append(f"surface {surface} importance {importance:.2f} (×{importance_factor:.2f})")
	225	+
	226	+ # typed floors
	227	+ floor = 0.0
	228	+ floor_reason = ""
	229	+ for key, value in typed_keys.items():
	230	+ f = TYPED_FLOORS.get(key, 0.0)
	231	+ if key in ("jobs.added", "jobs.removed"):
	232	+ open_before = int(sd.get("jobs", {}).get("open_before") or 0)
	233	+ if value >= JOBS_SCALE_COUNT or (open_before and value / open_before >= 0.25):
	234	+ f = max(f, JOBS_SCALE_FLOOR)
	235	+ if f > floor:
	236	+ floor, floor_reason = f, f"{key}={value}"
	237	+ if floor > 0:
	238	+ # a typed delta lifts the score to its floor and still rewards larger content changes above it
	239	+ sig = max(sig, floor + 0.15 * min(1.0, changed_share))
	240	+ reasons.append(f"typed delta {floor_reason} → floor {floor:.2f}")
	241	+
	242	+ # novelty / churn from sensor history
	243	+ unchanged_runs = int(hist.get("consecutive_unchanged") or 0)
	244	+ observations = int(hist.get("observation_count") or 0)
	245	+ changes = int(hist.get("change_count") or 0)
	246	+ if unchanged_runs >= NOVELTY_STABLE_RUNS:
	247	+ sig *= NOVELTY_BOOST
	248	+ reasons.append(f"novel after {unchanged_runs} unchanged runs (×{NOVELTY_BOOST})")
	249	+ elif observations >= 6 and changes / max(1, observations) > CHURN_RATE and not typed_keys:
	250	+ sig *= CHURN_PENALTY
	251	+ reasons.append(f"volatile page ({changes}/{observations} runs changed, ×{CHURN_PENALTY})")
	252	+
	253	+ # noise caps (never override typed floors)
	254	+ if not typed_keys:
	255	+ if not content_deltas:
	256	+ sig = min(sig, LOW_VALUE_CAP)
	257	+ reasons.append("nav/footer-only churn → noise cap")
	258	+ else:
	259	+ only_tiny = all(d.similarity is not None and d.similarity >= TINY_EDIT_SIMILARITY for d in content_deltas)
	260	+ if only_tiny:
	261	+ sig = min(sig, NOISE_CAP)
	262	+ reasons.append("only cosmetic edits (similarity ≥ 0.97) → noise cap")
	263	+ elif changed_share < 0.01 and len(content_deltas) <= 2 and diff.text_delta_ratio < 0.01:
	264	+ sig = min(sig, NOISE_CAP)
	265	+ reasons.append("negligible share (< 1 %) in ≤ 2 blocks → noise cap")
	266	+ elif _all_noise_tokens(content_deltas):
	267	+ sig = min(sig, NOISE_CAP)
	268	+ reasons.append("changes limited to dates/counters → noise cap")
	269	+ diff.significance = round(max(0.0, min(1.0, sig)), 4)
	270	+ diff.reasons = reasons
	271	+ return diff
	272	+
	273	+
	274	+def _typed_keys(sd: Mapping[str, Any]) -> dict[str, int]:
	275	+ """Flatten a StructuredDelta into {"jobs.added": n, "plans.price_changed": n, "people.added_executive": n, …} (non-zero only)."""
	276	+ out: dict[str, int] = {}
	277	+ for group in ("jobs", "people", "products", "plans", "locations", "news"):
	278	+ g = sd.get(group) or {}
	279	+ if not isinstance(g, Mapping):
	280	+ continue
	281	+ for k in ("added", "removed", "price_changed", "title_changed", "new_countries"):
	282	+ v = g.get(k)
	283	+ if isinstance(v, list) and v:
	284	+ out[f"{group}.{k}"] = len(v)
	285	+ if group == "people" and k in ("added", "removed"):
	286	+ execs = sum(1 for p in v if isinstance(p, Mapping) and (p.get("is_executive") or p.get("role_category") in
	287	+ ("ceo", "cfo", "cto", "coo", "founder", "president", "chair")))
	288	+ if execs:
	289	+ out[f"people.{k}_executive"] = execs
	290	+ meta = sd.get("meta") or {}
	291	+ if isinstance(meta, Mapping) and meta.get("title_changed"):
	292	+ out["meta.title_changed"] = 1
	293	+ return out
	294	+
	295	+
	296	+def _all_noise_tokens(deltas: list[BlockDelta]) -> bool:
	297	+ """True when every modified block is identical after noise normalisation (dates, counters…) — belt and braces: such blocks
	298	+ normally share the same hash and never reach the delta list, but fuzzy-matched blocks with different keys can."""
	299	+ for d in deltas:
	300	+ if d.before is None or d.after is None:
	301	+ return False
	302	+ if normalized_text(d.before) != normalized_text(d.after):
	303	+ return False
	304	+ return bool(deltas)
	305	+
	306	+
	307	+__all__ = ["DIFF_VERSION", "NEAR_HAMMING", "TYPED_FLOORS", "compare", "match_blocks"]

added src/companyatlas/sdk/normalize.py +786 −0

		`@@ -0,0 +1,786 @@`
	1	+"""HTML → semantic building blocks (spec §17–19, §107).
	2	+
	3	+ html ──parse()──▶ NormalizedPage(title, meta, lang, blocks, text, links, jsonld, microdata) ──to_extraction()──▶ Extraction
	4	+
	5	+Two layers are kept deliberately separate:
	6	+
	7	+* stored text — whitespace-normalised, but otherwise the original (what a human reads in the historical viewer);
	8	+* hash / diff layer — `normalized_text()` replaces classic noise (dates, times, "3 minutes ago", counters, csrf/nonce/session
	9	+ tokens, cache-busting query params) with stable tokens so that `text_hash`, block hashes and simhashes ignore it.
	10	+
	11	+Blocks carry a stable `key` = kind + heading path + simhash bucket (+ occurrence index) — never DOM position — so a reordered
	12	+section is a move, not an add/remove pair. Deterministic; no network; no LLM.
	13	+"""
	14	+from __future__ import annotations
	15	+
	16	+import hashlib
	17	+import html as html_lib
	18	+import json
	19	+import logging
	20	+import re
	21	+from collections import Counter
	22	+from dataclasses import dataclass, field
	23	+from typing import Any
	24	+
	25	+from selectolax.lexbor import LexborHTMLParser, LexborNode
	26	+
	27	+from companyatlas.sdk.models import Block, Extraction
	28	+
	29	+log = logging.getLogger(__name__)
	30	+
	31	+NORMALIZE_VERSION = "normalize-v1"
	32	+
	33	+# ------------------------------------------------------------------------------------------------------------ element sets
	34	+
	35	+DROP_TAGS = frozenset({"script", "style", "noscript", "svg", "iframe", "template", "canvas", "video", "audio", "source", "track", "object",
	36	+ "embed", "map", "area", "input", "select", "textarea", "option", "optgroup", "datalist", "meter", "progress", "link",
	37	+ "meta", "base", "head", "picture", "img", "dialog", "math"})
	38	+BLOCK_TAGS = frozenset({"address", "article", "aside", "blockquote", "body", "dd", "details", "div", "dl", "dt", "fieldset", "figcaption",
	39	+ "figure", "footer", "form", "h1", "h2", "h3", "h4", "h5", "h6", "header", "hr", "li", "main", "nav", "ol", "p", "pre",
	40	+ "section", "summary", "table", "tbody", "td", "tfoot", "th", "thead", "tr", "ul", "caption", "legend", "menu"})
	41	+HEADING_TAGS = ("h1", "h2", "h3", "h4", "h5", "h6")
	42	+
	43	+COOKIE_RE = re.compile(r"(cookie\|consent\|gdpr\|onetrust\|cookiebot\|truste\|didomi\|usercentrics\|osano\|cc-banner\|cc-window\|privacy-banner\|"
	44	+ r"cmp-container\|qc-cmp\|sp_message\|termly\|iubenda\|klaro\|cookieyes\|axeptio\|tarteaucitron)", re.IGNORECASE)
	45	+NAV_CLASS_RE = re.compile(r"(^\|[\s_-])(nav\|navbar\|navigation\|menu\|breadcrumbs?\|topbar\|masthead\|site-header\|global-header)([\s_-]\|$)", re.IGNORECASE)
	46	+FOOTER_CLASS_RE = re.compile(r"(^\|[\s_-])(footer\|site-footer\|global-footer\|colophon\|legal-links)([\s_-]\|$)", re.IGNORECASE)
	47	+HERO_CLASS_RE = re.compile(r"(^\|[\s_-])(hero\|jumbotron\|banner\|masthead\|splash\|intro\|cover\|landing-hero)([\s_-]\|$)", re.IGNORECASE)
	48	+CARD_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
	49	+ ("pricing_plan", re.compile(r"(^\|[\s_-])(pricing[-_ ]?(plan\|card\|tier\|column\|table\|box\|option)\|plan[-_ ]?(card\|box\|column\|tier\|item)\|tier[-_ ]?card\|"
	50	+ r"price[-_ ]?(card\|box\|column\|plan)\|package[-_ ]?(card\|box))([\s_-]\|$)", re.IGNORECASE)),
	51	+ ("job_listing", re.compile(r"(^\|[\s_-])(job\|jobs\|opening\|position\|vacancy\|posting\|role\|career)[-_ ]?(item\|card\|listing\|row\|link\|entry\|tile\|post)?([\s_-]\|$)", re.IGNORECASE)),
	52	+ ("person", re.compile(r"(^\|[\s_-])(team[-_ ]?member\|member[-_ ]?card\|person\|people[-_ ]?card\|bio\|executive\|leader\|profile[-_ ]?card\|staff\|founder\|"
	53	+ r"board[-_ ]?member\|management[-_ ]?member\|employee[-_ ]?card\|director)([\s_-]\|$)", re.IGNORECASE)),
	54	+ ("location", re.compile(r"(^\|[\s_-])(office\|location\|store\|branch\|address\|showroom\|site[-_ ]?card\|headquarters\|hq)([\s_-]\|$)", re.IGNORECASE)),
	55	+ ("news_item", re.compile(r"(^\|[\s_-])(news\|press\|article\|post\|release\|story\|blog\|announcement\|update\|publication)[-_ ]?(item\|card\|teaser\|tile\|preview\|summary\|link\|list-item\|entry)?([\s_-]\|$)", re.IGNORECASE)),
	56	+ ("product_card", re.compile(r"(^\|[\s_-])((product\|products\|catalog\|sku\|offering\|solution)[-_ ]?(card\|tile\|item\|box\|grid-item\|teaser\|entry)?\|item[-_ ]?card\|grid[-_ ]item)([\s_-]\|$)", re.IGNORECASE)),
	57	+ ("faq", re.compile(r"(^\|[\s_-])(faq\|accordion\|question\|collapsible)([\s_-]\|$)", re.IGNORECASE)),
	58	+]
	59	+CARD_MAX_TEXT = 1800
	60	+
	61	+# Weights per block kind (spec §19–20): what matters for significance. Nav/footer/cookie are kept for discovery, not for change value.
	62	+BLOCK_WEIGHTS: dict[str, float] = {
	63	+ "hero": 1.5, "pricing_plan": 1.6, "job_listing": 1.4, "person": 1.4, "product_card": 1.3, "location": 1.2, "news_item": 1.2,
	64	+ "heading": 1.0, "paragraph": 1.0, "section": 1.0, "table": 1.1, "list": 0.9, "faq": 0.8, "code": 0.7, "quote": 0.6, "other": 0.5,
	65	+ "header": 0.3, "nav": 0.2, "footer": 0.15,
	66	+}
	67	+LOW_VALUE_KINDS = frozenset({"nav", "footer", "header"})
	68	+
	69	+# ------------------------------------------------------------------------------------------------------------ noise normalisation
	70	+
	71	+_MONTH = r"(?:jan(?:uary)?\|feb(?:ruary)?\|mar(?:ch)?\|apr(?:il)?\|may\|june?\|july?\|aug(?:ust)?\|sep(?:t(?:ember)?)?\|oct(?:ober)?\|nov(?:ember)?\|dec(?:ember)?\|" \
	72	+ r"janvier\|février\|fevrier\|mars\|avril\|mai\|juin\|juillet\|août\|aout\|septembre\|octobre\|novembre\|décembre\|decembre\|" \
	73	+ r"januar\|februar\|märz\|maerz\|april\|juni\|juli\|august\|oktober\|dezember\|enero\|febrero\|marzo\|abril\|mayo\|junio\|julio\|agosto\|septiembre\|octubre\|noviembre\|diciembre)"
	74	+NOISE_RULES: list[tuple[re.Pattern[str], str]] = [
	75	+ (re.compile(r"\b\d{4}-\d{2}-\d{2}(?:[T ]\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?:Z\|[+-]\d{2}:?\d{2})?)?\b"), "<date>"),
	76	+ (re.compile(r"\b(?:\d{1,2}(?:st\|nd\|rd\|th)?\s+)?" + _MONTH + r"\.?\s+\d{1,2}(?:st\|nd\|rd\|th)?,?\s+\d{4}\b", re.IGNORECASE), "<date>"),
	77	+ (re.compile(r"\b\d{1,2}(?:st\|nd\|rd\|th)?\s+" + _MONTH + r"\.?,?\s+\d{4}\b", re.IGNORECASE), "<date>"),
	78	+ (re.compile(r"\b" + _MONTH + r"\.?\s+\d{4}\b", re.IGNORECASE), "<date>"),
	79	+ (re.compile(r"\b\d{1,2}[/.-]\d{1,2}[/.-]\d{2,4}\b"), "<date>"),
	80	+ (re.compile(r"\b\d{1,2}:\d{2}(?::\d{2})?\s*(?:am\|pm\|a\.m\.\|p\.m\.\|utc\|gmt\|est\|pst\|cet\|z)?\b", re.IGNORECASE), "<time>"),
	81	+ (re.compile(r"\b(?:\d+\|a\|an\|one\|few\|several)\s+(?:sec(?:ond)?s?\|min(?:ute)?s?\|hours?\|hrs?\|days?\|weeks?\|months?\|years?)\s+ago\b", re.IGNORECASE), "<rel>"),
	82	+ (re.compile(r"\b(?:il y a\|hace\|vor)\s+\d+\s+\w+\b", re.IGNORECASE), "<rel>"),
	83	+ (re.compile(r"\b(?:yesterday\|today\|just now\|hier\|aujourd'hui\|heute\|gestern)\b", re.IGNORECASE), "<rel>"),
	84	+ (re.compile(r"\b\d[\d,.]\s(?:k\|m)?\s*(?:views?\|comments?\|likes?\|shares?\|followers?\|visitors?\|reads?\|replies\|upvotes?\|downloads?\|stars?\|members?\|online)\b", re.IGNORECASE), "<count>"),
	85	+ (re.compile(r"(?:©\|$c$\|\bcopyright)\s(?:\d{4}\s[-–]\s*)?\d{4}\b", re.IGNORECASE), "<copyright>"),
	86	+ (re.compile(r"\b(?:\d{4}\s[-–]\s)?\d{4}\s*(?:©\|$c$)", re.IGNORECASE), "<copyright>"),
	87	+ (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", re.IGNORECASE), "<uuid>"),
	88	+ (re.compile(r"\b[0-9a-f]{24,}\b", re.IGNORECASE), "<hex>"),
	89	+ (re.compile(r"\b(?=[A-Za-z0-9_-]\d)(?=[A-Za-z0-9_-][A-Za-z])[A-Za-z0-9_-]{32,}\b"), "<token>"),
	90	+ (re.compile(r"([?&](?:v\|ver\|version\|rev\|build\|cb\|cache\|nocache\|_\|t\|ts\|timestamp\|hash\|nonce\|csrf\|token\|_token\|csrfmiddlewaretoken\|"
	91	+ r"authenticity_token\|utm_[a-z]+\|fbclid\|gclid\|sessionid\|session_id\|phpsessid\|jsessionid)=)[^&\s#\"']+", re.IGNORECASE), r"\1<q>"),
	92	+ (re.compile(r"\b(?:nonce\|csrf[-_]?token\|xsrf[-_]?token\|session[-_]?id\|request[-_]?id\|trace[-_]?id\|build[-_]?id)\s[:=]\s['\"]?[A-Za-z0-9_\-+/=.]{8,}", re.IGNORECASE), "<token>"),
	93	+]
	94	+WS_RE = re.compile(r"[ \t\r\f\v\u00a0\u1680\u2000-\u200b\u2028\u2029\u202f\u205f\u3000\ufeff]+")
	95	+NL_RE = re.compile(r"\s\n\s")
	96	+MULTI_NL_RE = re.compile(r"\n{3,}")
	97	+
	98	+
	99	+def normalize_whitespace(text: str) -> str:
	100	+ text = text.replace("\u00ad", "").replace("\ufeff", "")
	101	+ text = WS_RE.sub(" ", text)
	102	+ text = NL_RE.sub("\n", text)
	103	+ text = MULTI_NL_RE.sub("\n\n", text)
	104	+ return text.strip()
	105	+
	106	+
	107	+def normalized_text(text: str) -> str:
	108	+ """Noise-normalised, lower-cased text for hashing/diffing only. The stored text keeps the originals."""
	109	+ out = normalize_whitespace(text)
	110	+ for pat, repl in NOISE_RULES:
	111	+ out = pat.sub(repl, out)
	112	+ return WS_RE.sub(" ", out).lower().strip()
	113	+
	114	+
	115	+def text_hash(text: str) -> str:
	116	+ return hashlib.sha256(normalized_text(text).encode("utf-8")).hexdigest()
	117	+
	118	+
	119	+def structural_hash(blocks: list[Block]) -> str:
	120	+ seq = "\n".join(f"{b.kind}\|{b.key}" for b in blocks)
	121	+ return hashlib.sha256(seq.encode("utf-8")).hexdigest()
	122	+
	123	+
	124	+# ------------------------------------------------------------------------------------------------------------ simhash
	125	+
	126	+_TOKEN_RE = re.compile(r"[\w<>]+", re.UNICODE)
	127	+
	128	+
	129	+def _features(text: str) -> Counter[str]:
	130	+ toks = _TOKEN_RE.findall(text)
	131	+ feats: Counter[str] = Counter(toks)
	132	+ for i in range(len(toks) - 1):
	133	+ feats[toks[i] + " " + toks[i + 1]] += 1
	134	+ return feats
	135	+
	136	+
	137	+def simhash(text: str) -> int:
	138	+ """64-bit simhash over word uni+bigrams of the noise-normalised text (0 for empty)."""
	139	+ feats = _features(normalized_text(text))
	140	+ if not feats:
	141	+ return 0
	142	+ v = [0] * 64
	143	+ for feat, w in feats.items():
	144	+ h = int.from_bytes(hashlib.blake2b(feat.encode("utf-8"), digest_size=8).digest(), "big")
	145	+ for i in range(64):
	146	+ v[i] += w if (h >> i) & 1 else -w
	147	+ out = 0
	148	+ for i in range(64):
	149	+ if v[i] > 0:
	150	+ out \|= 1 << i
	151	+ return out
	152	+
	153	+
	154	+def hamming(a: int, b: int) -> int:
	155	+ return (a ^ b).bit_count()
	156	+
	157	+
	158	+def simhash_bucket(h: int, bits: int = 12) -> str:
	159	+ return format(h >> (64 - bits), "x") if h else "0"
	160	+
	161	+
	162	+# ------------------------------------------------------------------------------------------------------------ language guess
	163	+
	164	+_STOPWORDS: dict[str, frozenset[str]] = {
	165	+ "en": frozenset(["the", "and", "of", "to", "in", "for", "with", "on", "is", "are", "our", "we", "you", "your", "this", "that", "from", "by", "at", "as", "be"]),
	166	+ "fr": frozenset(["le", "la", "les", "et", "de", "des", "du", "en", "pour", "avec", "sur", "est", "sont", "nous", "vous", "votre", "notre", "une", "un", "dans", "par", "au", "aux", "ce", "cette"]),
	167	+ "de": frozenset(["der", "die", "das", "und", "von", "zu", "in", "für", "mit", "auf", "ist", "sind", "wir", "sie", "ihre", "unsere", "eine", "ein", "im", "den", "dem", "nicht"]),
	168	+ "es": frozenset(["el", "la", "los", "las", "y", "de", "del", "en", "para", "con", "sobre", "es", "son", "nosotros", "su", "nuestra", "una", "un", "por", "al", "como", "más"]),
	169	+ "it": frozenset(["il", "la", "gli", "le", "e", "di", "del", "della", "in", "per", "con", "su", "è", "sono", "noi", "nostro", "una", "un", "dal", "nel", "che"]),
	170	+ "pt": frozenset(["o", "a", "os", "as", "e", "de", "do", "da", "em", "para", "com", "sobre", "é", "são", "nós", "nosso", "uma", "um", "pelo", "na", "no", "que"]),
	171	+ "nl": frozenset(["de", "het", "een", "en", "van", "voor", "met", "op", "is", "zijn", "wij", "onze", "je", "jouw", "dat", "dit", "niet", "ook"]),
	172	+}
	173	+_CJK_RE = re.compile(r"[぀-ヿ]")
	174	+_HAN_RE = re.compile(r"[一-鿿]")
	175	+_HANGUL_RE = re.compile(r"[가-힯]")
	176	+_CYR_RE = re.compile(r"[Ѐ-ӿ]")
	177	+_ARAB_RE = re.compile(r"[؀-ۿ]")
	178	+
	179	+
	180	+def language_guess(text: str, html_lang: str \| None = None) -> str \| None:
	181	+ if html_lang:
	182	+ code = html_lang.strip().lower().split("-")[0].split("_")[0]
	183	+ if 2 <= len(code) <= 3 and code.isalpha():
	184	+ return code
	185	+ sample = text[:6000]
	186	+ if not sample.strip():
	187	+ return None
	188	+ if _CJK_RE.search(sample):
	189	+ return "ja"
	190	+ if _HANGUL_RE.search(sample):
	191	+ return "ko"
	192	+ if _HAN_RE.search(sample) and len(_HAN_RE.findall(sample)) > 20:
	193	+ return "zh"
	194	+ if len(_CYR_RE.findall(sample)) > 40:
	195	+ return "ru"
	196	+ if len(_ARAB_RE.findall(sample)) > 40:
	197	+ return "ar"
	198	+ words = re.findall(r"[a-zà-ÿ']+", sample.lower())
	199	+ if len(words) < 8:
	200	+ return None
	201	+ best, best_n = None, 0
	202	+ for lang, sw in _STOPWORDS.items():
	203	+ n = sum(1 for w in words if w in sw)
	204	+ if n > best_n:
	205	+ best, best_n = lang, n
	206	+ if best_n / max(1, len(words)) < 0.04:
	207	+ return None
	208	+ return best
	209	+
	210	+
	211	+# ------------------------------------------------------------------------------------------------------------ page model
	212	+
	213	+
	214	+@dataclass(slots=True)
	215	+class Link:
	216	+ url: str
	217	+ anchor: str
	218	+ region: str = "main" # nav \| header \| footer \| main
	219	+ rel: str = ""
	220	+ title: str = ""
	221	+
	222	+
	223	+@dataclass(slots=True)
	224	+class NormalizedPage:
	225	+ url: str
	226	+ title: str \| None
	227	+ lang: str \| None
	228	+ meta: dict[str, Any]
	229	+ blocks: list[Block]
	230	+ text: str # stored main text (original wording, whitespace-normalised)
	231	+ full_text: str # including nav/header/footer
	232	+ links: list[Link]
	233	+ jsonld: dict[str, list[dict[str, Any]]]
	234	+ microdata: dict[str, list[dict[str, Any]]]
	235	+ headings: list[tuple[int, str]]
	236	+ feeds: list[str] = field(default_factory=list)
	237	+ main_selector: str = "body"
	238	+
	239	+ @property
	240	+ def normalized(self) -> str:
	241	+ return normalized_text(self.text)
	242	+
	243	+ def blocks_of(self, *kinds: str) -> list[Block]:
	244	+ return [b for b in self.blocks if b.kind in kinds]
	245	+
	246	+ def to_extraction(self) -> Extraction:
	247	+ return Extraction(text=self.text, blocks=self.blocks, title=self.title, language=self.lang, meta=dict(self.meta))
	248	+
	249	+
	250	+# ------------------------------------------------------------------------------------------------------------ parsing
	251	+
	252	+
	253	+def _attr(node: LexborNode, name: str) -> str:
	254	+ try:
	255	+ v = node.attributes.get(name)
	256	+ except Exception: # noqa: BLE001
	257	+ return ""
	258	+ return (v or "").strip() if v is not None else ("" if name not in node.attributes else "true")
	259	+
	260	+
	261	+def _classes(node: LexborNode) -> str:
	262	+ return f"{_attr(node, 'id')} {_attr(node, 'class')} {_attr(node, 'role')} {_attr(node, 'data-testid')} {_attr(node, 'itemtype')}".strip()
	263	+
	264	+
	265	+def _is_hidden(node: LexborNode) -> bool:
	266	+ if node.tag in ("html", "body"):
	267	+ return False
	268	+ attrs = node.attributes
	269	+ if "hidden" in attrs and attrs.get("hidden") != "false":
	270	+ return True
	271	+ if (attrs.get("aria-hidden") or "").strip().lower() == "true":
	272	+ return True
	273	+ style = (attrs.get("style") or "").replace(" ", "").lower()
	274	+ return "display:none" in style or "visibility:hidden" in style
	275	+
	276	+
	277	+def _is_cookie(node: LexborNode) -> bool:
	278	+ cls = _classes(node)
	279	+ return bool(cls) and bool(COOKIE_RE.search(cls)) and node.tag in ("div", "section", "aside", "dialog", "footer", "header", "nav", "form")
	280	+
	281	+
	282	+def _region_of(node: LexborNode) -> str \| None:
	283	+ tag = node.tag
	284	+ role = _attr(node, "role").lower()
	285	+ cls = _classes(node)
	286	+ if tag == "nav" or role == "navigation" or NAV_CLASS_RE.search(cls):
	287	+ return "nav"
	288	+ if tag == "footer" or role == "contentinfo" or FOOTER_CLASS_RE.search(cls):
	289	+ return "footer"
	290	+ if tag == "header" or role == "banner":
	291	+ return "header"
	292	+ return None
	293	+
	294	+
	295	+def _node_text(node: LexborNode) -> str:
	296	+ """Text with block boundaries as newlines and inline elements joined by spaces."""
	297	+ parts: list[str] = []
	298	+
	299	+ def walk(n: LexborNode, depth: int) -> None:
	300	+ if depth > 400:
	301	+ return
	302	+ for c in n.iter(include_text=True):
	303	+ tag = c.tag
	304	+ if tag == "-text":
	305	+ t = c.text_content
	306	+ if t:
	307	+ parts.append(t)
	308	+ continue
	309	+ if tag in DROP_TAGS or tag == "-comment" or _is_hidden(c) or _is_cookie(c):
	310	+ continue
	311	+ if tag == "br":
	312	+ parts.append("\n")
	313	+ continue
	314	+ block = tag in BLOCK_TAGS
	315	+ if block:
	316	+ parts.append("\n")
	317	+ walk(c, depth + 1)
	318	+ if block:
	319	+ parts.append("\n")
	320	+ else:
	321	+ parts.append(" ")
	322	+
	323	+ walk(node, 0)
	324	+ return normalize_whitespace(html_lib.unescape("".join(parts)))
	325	+
	326	+
	327	+def _first_text(node: LexborNode, limit: int = 300) -> str:
	328	+ return _node_text(node)[:limit]
	329	+
	330	+
	331	+def _card_kind(node: LexborNode, *, allow_container: bool = False) -> str \| None:
	332	+ cls = _classes(node)
	333	+ if not cls:
	334	+ return None
	335	+ kind = next((k for k, pat in CARD_PATTERNS if pat.search(cls)), None)
	336	+ if kind is None:
	337	+ return None
	338	+ if not allow_container and _is_card_container(node):
	339	+ return None
	340	+ return kind
	341	+
	342	+
	343	+def _is_card_container(node: LexborNode) -> bool:
	344	+ """A grid/list wrapper (`.products`, `.news-list`, `.team-grid`) holds ≥ 2 card-like children — segment the children instead."""
	345	+ n = 0
	346	+ for child in node.iter():
	347	+ if child.tag not in ("div", "section", "article", "li", "a", "figure", "tr", "ul", "ol", "table"):
	348	+ continue
	349	+ is_card = child.tag in ("ul", "ol", "table") or _card_kind(child, allow_container=True) is not None
	350	+ if not is_card and child.tag == "div":
	351	+ is_card = any(_card_kind(g, allow_container=True) for g in child.iter() if g.tag in ("div", "article", "li", "a"))
	352	+ if is_card:
	353	+ n += 1
	354	+ if n >= 2:
	355	+ return True
	356	+ return False
	357	+
	358	+
	359	+class _Segmenter:
	360	+ def __init__(self, base_url: str):
	361	+ self.base_url = base_url
	362	+ self.blocks: list[Block] = []
	363	+ self.heading_stack: list[tuple[int, str]] = []
	364	+ self.headings: list[tuple[int, str]] = []
	365	+ self.order = 0
	366	+ self.hero_done = False
	367	+ self.text_parts: list[str] = []
	368	+
	369	+ # ---------------------------------------------------------------- helpers
	370	+ def path(self) -> str:
	371	+ return " > ".join(h[1] for h in self.heading_stack)[:200]
	372	+
	373	+ def push_heading(self, level: int, text: str) -> None:
	374	+ while self.heading_stack and self.heading_stack[-1][0] >= level:
	375	+ self.heading_stack.pop()
	376	+ self.heading_stack.append((level, text[:80]))
	377	+ self.headings.append((level, text))
	378	+
	379	+ def add(self, kind: str, text: str, *, path: str \| None = None, attrs: dict[str, Any] \| None = None, weight: float \| None = None) -> None:
	380	+ text = normalize_whitespace(text)
	381	+ if not text:
	382	+ return
	383	+ if len(text) > 6000:
	384	+ text = text[:6000]
	385	+ p = self.path() if path is None else path
	386	+ self.blocks.append(Block(key="", kind=kind, text=text, path=p, order=self.order, weight=weight or BLOCK_WEIGHTS.get(kind, 1.0),
	387	+ attrs=attrs or {}))
	388	+ self.order += 1
	389	+
	390	+ # ---------------------------------------------------------------- traversal
	391	+ def walk(self, node: LexborNode, region: str, depth: int = 0) -> None:
	392	+ if depth > 300:
	393	+ return
	394	+ for c in node.iter(include_text=True):
	395	+ tag = c.tag
	396	+ if tag == "-text":
	397	+ t = normalize_whitespace(c.text_content or "")
	398	+ if t and region == "main":
	399	+ self.add("paragraph" if len(t) > 40 else "other", t)
	400	+ elif t:
	401	+ self.add(region, t)
	402	+ continue
	403	+ if tag in DROP_TAGS or tag == "-comment" or _is_hidden(c) or _is_cookie(c):
	404	+ continue
	405	+ sub = _region_of(c) if region == "main" else None
	406	+ if sub is not None:
	407	+ txt = _node_text(c)
	408	+ if txt:
	409	+ kind = "nav" if sub == "nav" else ("footer" if sub == "footer" else "header")
	410	+ self.add(kind, txt[:1500], path="", attrs={"region": sub})
	411	+ continue
	412	+ if region != "main":
	413	+ # inside nav/header/footer: flatten into one block per top-level child
	414	+ txt = _node_text(c)
	415	+ if txt:
	416	+ self.add(region, txt[:1500], path="", attrs={"region": region})
	417	+ continue
	418	+ if tag in HEADING_TAGS:
	419	+ txt = _node_text(c)
	420	+ if txt:
	421	+ level = int(tag[1])
	422	+ self.push_heading(level, txt)
	423	+ self.add("heading", txt, path=" > ".join(h[1] for h in self.heading_stack[:-1])[:200],
	424	+ attrs={"level": level}, weight=BLOCK_WEIGHTS["heading"] * (1.3 if level == 1 else 1.0))
	425	+ continue
	426	+ if tag == "p":
	427	+ self.add("paragraph", _node_text(c))
	428	+ continue
	429	+ if tag == "blockquote":
	430	+ self.add("quote", _node_text(c))
	431	+ continue
	432	+ if tag == "pre" or (tag == "code" and len(_node_text(c)) > 80):
	433	+ self.add("code", _node_text(c))
	434	+ continue
	435	+ if tag in ("details",):
	436	+ self.add("faq", _node_text(c))
	437	+ continue
	438	+ if tag in ("ul", "ol", "dl", "menu"):
	439	+ self.list_block(c)
	440	+ continue
	441	+ if tag == "table":
	442	+ self.table_block(c)
	443	+ continue
	444	+ if tag in ("hr", "br", "wbr"):
	445	+ continue
	446	+ card = _card_kind(c) if tag in ("div", "section", "article", "li", "a", "figure", "aside", "tr", "td") else None
	447	+ if card is not None:
	448	+ txt = _node_text(c)
	449	+ if txt and len(txt) <= CARD_MAX_TEXT:
	450	+ attrs: dict[str, Any] = {}
	451	+ href = self._first_href(c)
	452	+ if href:
	453	+ attrs["href"] = href
	454	+ self.add(card, txt, attrs=attrs)
	455	+ continue
	456	+ if not self.hero_done and tag in ("section", "div", "header", "article") and (HERO_CLASS_RE.search(_classes(c)) or self._has_h1(c)):
	457	+ txt = _node_text(c)
	458	+ if txt and len(txt) <= 2500:
	459	+ self.hero_done = True
	460	+ for h in c.css("h1"):
	461	+ ht = _node_text(h)
	462	+ if ht:
	463	+ self.push_heading(1, ht)
	464	+ break
	465	+ self.add("hero", txt, path="", attrs={"href": self._first_href(c)} if self._first_href(c) else {})
	466	+ continue
	467	+ self.walk(c, region, depth + 1)
	468	+
	469	+ def _has_h1(self, node: LexborNode) -> bool:
	470	+ return node.css_first("h1") is not None and len(_node_text(node)) < 2500
	471	+
	472	+ def _first_href(self, node: LexborNode) -> str \| None:
	473	+ a = node if node.tag == "a" else node.css_first("a[href]")
	474	+ if a is None:
	475	+ return None
	476	+ from companyatlas.urls import absolutize
	477	+
	478	+ return absolutize(self.base_url, _attr(a, "href"))
	479	+
	480	+ def list_block(self, node: LexborNode) -> None:
	481	+ items = [li for li in node.iter() if li.tag in ("li", "dt", "dd")]
	482	+ if not items:
	483	+ txt = _node_text(node)
	484	+ if txt:
	485	+ self.add("list", txt)
	486	+ return
	487	+ texts = [_node_text(li) for li in items]
	488	+ texts_nonempty = [t for t in texts if t]
	489	+ if not texts_nonempty:
	490	+ return
	491	+ avg = sum(len(t) for t in texts_nonempty) / len(texts_nonempty)
	492	+ has_links = sum(1 for li in items if li.css_first("a[href]") is not None)
	493	+ listing_like = len(items) >= 3 and (has_links >= len(items) * 0.6 or avg > 60)
	494	+ if not listing_like or len(items) > 400:
	495	+ self.add("list", "\n".join(texts_nonempty)[:4000])
	496	+ return
	497	+ for li, t in zip(items, texts, strict=False):
	498	+ if not t:
	499	+ continue
	500	+ kind = _card_kind(li, allow_container=True) or _card_kind(node, allow_container=True) or "list"
	501	+ attrs: dict[str, Any] = {}
	502	+ href = self._first_href(li)
	503	+ if href:
	504	+ attrs["href"] = href
	505	+ self.add(kind, t[:1500], attrs=attrs)
	506	+
	507	+ def table_block(self, node: LexborNode) -> None:
	508	+ rows = [tr for tr in node.css("tr")]
	509	+ caption = node.css_first("caption")
	510	+ cap = _node_text(caption) if caption is not None else ""
	511	+ if len(rows) <= 1 or len(rows) > 500:
	512	+ txt = _node_text(node)
	513	+ if txt:
	514	+ self.add("table", txt[:4000])
	515	+ return
	516	+ header_cells = [_node_text(th) for th in rows[0].css("th")]
	517	+ header = " \| ".join(x for x in header_cells if x)
	518	+ base_path = self.path()
	519	+ table_path = " > ".join(x for x in (base_path, cap or header[:80]) if x)[:200]
	520	+ for i, tr in enumerate(rows):
	521	+ cells = [_node_text(td) for td in tr.iter() if td.tag in ("td", "th")]
	522	+ txt = " \| ".join(c for c in cells if c)
	523	+ if not txt:
	524	+ continue
	525	+ kind = _card_kind(tr, allow_container=True) or "table"
	526	+ attrs: dict[str, Any] = {"row": i}
	527	+ href = self._first_href(tr)
	528	+ if href:
	529	+ attrs["href"] = href
	530	+ self.add(kind, txt[:1500], path=table_path, attrs=attrs)
	531	+
	532	+
	533	+def _finalize_blocks(blocks: list[Block]) -> list[Block]:
	534	+ total = len(blocks) or 1
	535	+ seen: Counter[str] = Counter()
	536	+ for b in blocks:
	537	+ norm = normalized_text(b.text)
	538	+ b.hash = hashlib.sha256(norm.encode("utf-8")).hexdigest()[:16]
	539	+ b.simhash = simhash(b.text)
	540	+ path_h = hashlib.blake2b(b.path.lower().encode("utf-8"), digest_size=3).hexdigest() if b.path else "root"
	541	+ base = f"{b.kind}:{path_h}:{simhash_bucket(b.simhash)}"
	542	+ n = seen[base]
	543	+ seen[base] += 1
	544	+ b.key = base if n == 0 else f"{base}#{n + 1}"
	545	+ rel = b.order / total
	546	+ pos = 1.1 if rel < 0.1 else (0.9 if rel > 0.9 else 1.0)
	547	+ if b.kind in LOW_VALUE_KINDS:
	548	+ pos = 1.0
	549	+ b.weight = round(b.weight * pos, 3)
	550	+ return blocks
	551	+
	552	+
	553	+def _collect_links(tree: LexborHTMLParser, base_url: str) -> list[Link]:
	554	+ from companyatlas.urls import absolutize
	555	+
	556	+ links: list[Link] = []
	557	+ seen: set[tuple[str, str]] = set()
	558	+ for a in tree.css("a[href]"):
	559	+ href = _attr(a, "href")
	560	+ url = absolutize(base_url, href)
	561	+ if not url:
	562	+ continue
	563	+ anchor = _node_text(a)[:160] or _attr(a, "aria-label")[:160] or _attr(a, "title")[:160]
	564	+ region = "main"
	565	+ p = a.parent
	566	+ hops = 0
	567	+ while p is not None and hops < 40:
	568	+ r = _region_of(p) if p.tag not in ("html", "body") else None
	569	+ if r:
	570	+ region = r
	571	+ break
	572	+ p = p.parent
	573	+ hops += 1
	574	+ k = (url, anchor.lower())
	575	+ if k in seen:
	576	+ continue
	577	+ seen.add(k)
	578	+ links.append(Link(url=url, anchor=anchor, region=region, rel=_attr(a, "rel"), title=_attr(a, "title")))
	579	+ if len(links) >= 3000:
	580	+ break
	581	+ return links
	582	+
	583	+
	584	+def _meta(tree: LexborHTMLParser, base_url: str) -> tuple[str \| None, str \| None, dict[str, Any], list[str]]:
	585	+ from companyatlas.urls import absolutize
	586	+
	587	+ meta: dict[str, Any] = {}
	588	+ title_node = tree.css_first("title")
	589	+ title = normalize_whitespace(title_node.text()) if title_node is not None else None
	590	+ lang = _attr(tree.root, "lang") if tree.root is not None else ""
	591	+ if not lang:
	592	+ html_node = tree.css_first("html")
	593	+ lang = _attr(html_node, "lang") if html_node is not None else ""
	594	+ feeds: list[str] = []
	595	+ for m in tree.css("meta"):
	596	+ name = (_attr(m, "name") or _attr(m, "property") or _attr(m, "http-equiv")).lower()
	597	+ content = _attr(m, "content")
	598	+ if not name or not content:
	599	+ continue
	600	+ if name in ("description", "generator", "robots", "author", "keywords", "twitter:card", "theme-color") or name.startswith("og:") and name in ("og:title", "og:description", "og:type", "og:site_name", "og:locale", "og:url"):
	601	+ meta[name.replace(":", "_")] = content[:500]
	602	+ elif name in ("content-language",) and not lang:
	603	+ lang = content
	604	+ for ln in tree.css("link[rel]"):
	605	+ rel = _attr(ln, "rel").lower()
	606	+ href = _attr(ln, "href")
	607	+ if not href:
	608	+ continue
	609	+ if "canonical" in rel:
	610	+ meta["canonical"] = absolutize(base_url, href) or href
	611	+ elif "alternate" in rel:
	612	+ typ = _attr(ln, "type").lower()
	613	+ if "rss" in typ or "atom" in typ or "feed" in typ or "json" in typ and "feed" in href:
	614	+ u = absolutize(base_url, href)
	615	+ if u and u not in feeds:
	616	+ feeds.append(u)
	617	+ elif _attr(ln, "hreflang"):
	618	+ meta["hreflang_count"] = int(meta.get("hreflang_count", 0)) + 1
	619	+ return title or None, (lang or None), meta, feeds
	620	+
	621	+
	622	+# ------------------------------------------------------------------------------------------------------------ JSON-LD / microdata
	623	+
	624	+JSONLD_BUCKETS: dict[str, str] = {
	625	+ "organization": "organizations", "corporation": "organizations", "localbusiness": "organizations", "ngo": "organizations",
	626	+ "jobposting": "job_postings", "product": "products", "offer": "offers", "aggregateoffer": "offers", "service": "products",
	627	+ "softwareapplication": "products", "newsarticle": "articles", "blogposting": "articles", "article": "articles", "pressrelease": "articles",
	628	+ "techarticle": "articles", "report": "articles", "person": "persons", "postaladdress": "addresses", "place": "places",
	629	+ "breadcrumblist": "breadcrumbs", "faqpage": "faqs", "website": "websites", "webpage": "webpages", "event": "events",
	630	+}
	631	+
	632	+
	633	+def _iter_jsonld(obj: Any, out: list[dict[str, Any]], depth: int = 0) -> None:
	634	+ if depth > 8 or len(out) > 500:
	635	+ return
	636	+ if isinstance(obj, list):
	637	+ for x in obj:
	638	+ _iter_jsonld(x, out, depth + 1)
	639	+ elif isinstance(obj, dict):
	640	+ if "@graph" in obj and isinstance(obj["@graph"], list):
	641	+ _iter_jsonld(obj["@graph"], out, depth + 1)
	642	+ if "@type" in obj:
	643	+ out.append(obj)
	644	+ for k in ("mainEntity", "itemListElement", "hasPart", "member", "employee", "founder", "address", "location", "offers", "item"):
	645	+ if k in obj:
	646	+ _iter_jsonld(obj[k], out, depth + 1)
	647	+
	648	+
	649	+def extract_jsonld(tree: LexborHTMLParser) -> dict[str, list[dict[str, Any]]]:
	650	+ buckets: dict[str, list[dict[str, Any]]] = {}
	651	+ for s in tree.css('script[type="application/ld+json"], script[type="application/json+ld"]'):
	652	+ raw = (s.text() or "").strip()
	653	+ if not raw or len(raw) > 2_000_000:
	654	+ continue
	655	+ try:
	656	+ data = json.loads(raw)
	657	+ except json.JSONDecodeError:
	658	+ try:
	659	+ data = json.loads(html_lib.unescape(raw).replace("\n", " "))
	660	+ except json.JSONDecodeError:
	661	+ continue
	662	+ found: list[dict[str, Any]] = []
	663	+ _iter_jsonld(data, found)
	664	+ for item in found:
	665	+ types = item.get("@type")
	666	+ for t in (types if isinstance(types, list) else [types]):
	667	+ if not isinstance(t, str):
	668	+ continue
	669	+ key = JSONLD_BUCKETS.get(t.rsplit("/", 1)[-1].lower())
	670	+ if key:
	671	+ buckets.setdefault(key, []).append(item)
	672	+ return buckets
	673	+
	674	+
	675	+def extract_microdata(tree: LexborHTMLParser) -> dict[str, list[dict[str, Any]]]:
	676	+ buckets: dict[str, list[dict[str, Any]]] = {}
	677	+ for scope in tree.css("[itemscope][itemtype]"):
	678	+ itype = _attr(scope, "itemtype").rsplit("/", 1)[-1].lower()
	679	+ key = JSONLD_BUCKETS.get(itype)
	680	+ if not key:
	681	+ continue
	682	+ item: dict[str, Any] = {"@type": itype}
	683	+ for prop in scope.css("[itemprop]"):
	684	+ name = _attr(prop, "itemprop")
	685	+ if not name or "itemscope" in prop.attributes:
	686	+ if name and "itemscope" in prop.attributes:
	687	+ item[name] = _first_text(prop, 200)
	688	+ continue
	689	+ if prop.tag == "a" or prop.tag == "link":
	690	+ val = _attr(prop, "href")
	691	+ elif prop.tag in ("meta",):
	692	+ val = _attr(prop, "content")
	693	+ elif prop.tag == "time":
	694	+ val = _attr(prop, "datetime") or _first_text(prop, 100)
	695	+ elif prop.tag == "img":
	696	+ val = _attr(prop, "src")
	697	+ else:
	698	+ val = _first_text(prop, 300)
	699	+ if val and name not in item:
	700	+ item[name] = val
	701	+ buckets.setdefault(key, []).append(item)
	702	+ if sum(len(v) for v in buckets.values()) > 500:
	703	+ break
	704	+ return buckets
	705	+
	706	+
	707	+# ------------------------------------------------------------------------------------------------------------ main-content detection
	708	+
	709	+MAIN_SELECTORS = ("main", "[role=main]", "article", "#main", "#main-content", "#content", ".main-content", "#primary", ".site-main", ".content")
	710	+
	711	+
	712	+def _prune(tree: LexborHTMLParser) -> None:
	713	+ for tag in ("script", "style", "noscript", "svg", "iframe", "template", "canvas", "video", "audio", "object", "embed", "input", "select",
	714	+ "textarea", "option", "datalist"):
	715	+ for n in tree.css(tag):
	716	+ n.decompose()
	717	+
	718	+
	719	+def _find_main(tree: LexborHTMLParser) -> tuple[LexborNode \| None, str]:
	720	+ body = tree.body
	721	+ if body is None:
	722	+ return None, "body"
	723	+ for sel in MAIN_SELECTORS:
	724	+ try:
	725	+ n = tree.css_first(sel)
	726	+ except Exception as exc: # noqa: BLE001
	727	+ log.debug("main selector failed", extra={"selector": sel, "error": str(exc)})
	728	+ continue
	729	+ if n is not None and not _is_hidden(n) and len(_node_text(n)) >= 80:
	730	+ return n, sel
	731	+ # largest text region among body's block descendants (depth ≤ 3) that is not nav/header/footer
	732	+ best, best_len = None, 0
	733	+ candidates: list[LexborNode] = []
	734	+
	735	+ def collect(n: LexborNode, d: int) -> None:
	736	+ for c in n.iter():
	737	+ if c.tag in ("div", "section", "article", "td") and not _is_hidden(c) and _region_of(c) is None:
	738	+ candidates.append(c)
	739	+ if d < 3:
	740	+ collect(c, d + 1)
	741	+
	742	+ collect(body, 0)
	743	+ body_len = len(_node_text(body)) or 1
	744	+ for c in candidates[:400]:
	745	+ ln = len(_node_text(c))
	746	+ if ln > best_len and ln >= body_len * 0.35:
	747	+ best, best_len = c, ln
	748	+ return (best or body), ("largest" if best is not None else "body")
	749	+
	750	+
	751	+def parse(html_text: str, *, url: str = "", surface: str \| None = None) -> NormalizedPage:
	752	+ """Parse HTML into blocks + text + metadata. `surface` is a hint only (kept in meta)."""
	753	+ tree = LexborHTMLParser(html_text or "")
	754	+ title, lang, meta, feeds = _meta(tree, url)
	755	+ jsonld = extract_jsonld(tree)
	756	+ microdata = extract_microdata(tree)
	757	+ links = _collect_links(tree, url)
	758	+ _prune(tree)
	759	+ body = tree.body
	760	+ seg = _Segmenter(url)
	761	+ main_node, main_sel = _find_main(tree)
	762	+ if body is not None:
	763	+ # nav/header/footer first as low-weight blocks (in document order they usually wrap main; we walk body and let the segmenter route)
	764	+ seg.walk(body, "main")
	765	+ blocks = _finalize_blocks(seg.blocks)
	766	+ main_text = _node_text(main_node) if main_node is not None else ""
	767	+ full_text = _node_text(body) if body is not None else ""
	768	+ if not main_text:
	769	+ main_text = "\n".join(b.text for b in blocks if b.kind not in LOW_VALUE_KINDS)
	770	+ meta = {**meta, "lang": lang, "main_selector": main_sel, "jsonld_types": sorted({k for k in jsonld}), "normalize_version": NORMALIZE_VERSION}
	771	+ if surface:
	772	+ meta["surface_hint"] = surface
	773	+ if feeds:
	774	+ meta["feeds"] = feeds[:10]
	775	+ orgs = jsonld.get("organizations") or []
	776	+ if orgs:
	777	+ same_as = [s for o in orgs for s in (o.get("sameAs") or []) if isinstance(s, str)] if isinstance(orgs[0].get("sameAs", []), list) else []
	778	+ if same_as:
	779	+ meta["same_as"] = same_as[:20]
	780	+ language = language_guess(main_text, lang)
	781	+ return NormalizedPage(url=url, title=title, lang=language, meta=meta, blocks=blocks, text=main_text, full_text=full_text, links=links,
	782	+ jsonld=jsonld, microdata=microdata, headings=seg.headings, feeds=feeds, main_selector=main_sel)
	783	+
	784	+
	785	+__all__ = ["BLOCK_WEIGHTS", "LOW_VALUE_KINDS", "NORMALIZE_VERSION", "Link", "NormalizedPage", "extract_jsonld", "extract_microdata", "hamming",
	786	+ "language_guess", "normalize_whitespace", "normalized_text", "parse", "simhash", "simhash_bucket", "structural_hash", "text_hash"]

added src/companyatlas/services/discovery.py +751 −0

		`@@ -0,0 +1,751 @@`
	1	+"""Discovery engine (spec §10–11, §101–102): canonical domain → homepage → robots → sitemaps → navigation → ATS / feeds / subdomains →
	2	+URL classification → one sensor per surface.
	3	+
	4	+ discover_company(company, fetcher=…, dry_run=False) -> DiscoveryResult (never raises; ~90 s budget per company)
	5	+ onboard_pending(limit, concurrency) (consumes queue_jobs kind='discover' + pending companies)
	6	+
	7	+Every request goes through `fetch.Fetcher` (SSRF guard, robots, governor). Discovery is bounded: ≤ `MAX_PROBES` common-path probes,
	8	+≤ `MAX_SUBDOMAIN_GETS` subdomain fetches, one sitemap index (+ a few children), one ATS verification. Sensors are inserted with
	9	+`status='pending'`, staggered `next_run_at`, and a `quality_score` seeded from confidence × surface importance × fetch reliability.
	10	+"""
	11	+from __future__ import annotations
	12	+
	13	+import asyncio
	14	+import logging
	15	+import random
	16	+import re
	17	+import socket
	18	+import time
	19	+from dataclasses import dataclass, field
	20	+from datetime import UTC, datetime, timedelta
	21	+from typing import Any
	22	+from urllib.parse import urlparse
	23	+
	24	+from companyatlas.config import settings
	25	+from companyatlas.connectors._util import ats_sensor_spec
	26	+from companyatlas.db import execute, fetch_all, fetch_one, jsonb, transaction
	27	+from companyatlas.fetch import BlockedError, Fetcher, FetchError, FetchResult, NotModified
	28	+from companyatlas.ids import new_id
	29	+from companyatlas.sdk import connector as connectors
	30	+from companyatlas.sdk.models import DiscoveredUrl
	31	+from companyatlas.taxonomy import (
	32	+ SURFACE_BASE_INTERVAL_S,
	33	+ SURFACE_IMPORTANCE,
	34	+ FailureClass,
	35	+ OnboardingStatus,
	36	+ SensorStatus,
	37	+ Surface,
	38	+ tier_for_interval,
	39	+)
	40	+from companyatlas.urls import (
	41	+ absolutize,
	42	+ canonicalize_url,
	43	+ classify_url,
	44	+ detect_ats,
	45	+ is_static_asset,
	46	+ looks_like_trap,
	47	+ registrable_domain,
	48	+ same_company_host,
	49	+)
	50	+
	51	+log = logging.getLogger(__name__)
	52	+
	53	+DISCOVERY_VERSION = "discovery-v1"
	54	+COMPANY_BUDGET_S = 90.0
	55	+HOMEPAGE_MAX_BYTES = 3 * 1024 * 1024
	56	+PROBE_MAX_BYTES = 512 * 1024
	57	+MAX_PROBES = 12
	58	+MAX_SUBDOMAIN_GETS = 6
	59	+MAX_SITEMAP_CHILDREN = 4
	60	+TIER_FACTOR = {1: 0.5, 2: 0.75, 3: 1.0, 4: 1.5}
	61	+METHOD_WEIGHT = {"ats": 1.0, "feed": 0.95, "nav": 1.0, "link": 0.9, "probe": 0.92, "subdomain": 0.9, "sitemap": 0.85, "robots": 0.9, "pattern": 0.8,
	62	+ "jsonld": 0.85, "manual": 1.0}
	63	+# Surfaces worth a blind probe when navigation / sitemap did not reveal them: surface → candidate paths (ordered).
	64	+PROBE_PATHS: dict[str, tuple[str, ...]] = {
	65	+ Surface.CAREERS: ("/careers", "/jobs", "/careers/", "/company/careers", "/about/careers", "/join-us"),
	66	+ Surface.PRICING: ("/pricing", "/plans", "/pricing/"),
	67	+ Surface.NEWSROOM: ("/news", "/press", "/newsroom", "/press-releases", "/media"),
	68	+ Surface.ABOUT: ("/about", "/about-us", "/company", "/company/about"),
	69	+ Surface.LEADERSHIP: ("/leadership", "/team", "/about/leadership", "/company/leadership", "/about/team", "/management"),
	70	+ Surface.LOCATIONS: ("/locations", "/offices", "/contact", "/company/locations"),
	71	+ Surface.BLOG: ("/blog", "/insights"),
	72	+ Surface.DOCS: ("/docs", "/documentation"),
	73	+ Surface.CHANGELOG: ("/changelog", "/release-notes", "/whats-new"),
	74	+ Surface.INVESTOR_RELATIONS: ("/investors", "/investor-relations", "/ir"),
	75	+ Surface.LEGAL_TERMS: ("/terms", "/legal/terms", "/terms-of-service", "/legal"),
	76	+ Surface.LEGAL_PRIVACY: ("/privacy", "/legal/privacy", "/privacy-policy"),
	77	+}
	78	+SUBDOMAINS: tuple[tuple[str, str], ...] = (("careers", Surface.CAREERS), ("jobs", Surface.CAREERS), ("news", Surface.NEWSROOM), ("blog", Surface.BLOG),
	79	+ ("docs", Surface.DOCS), ("developer", Surface.DEVELOPER), ("developers", Surface.DEVELOPER),
	80	+ ("status", Surface.STATUS), ("investors", Surface.INVESTOR_RELATIONS), ("ir", Surface.INVESTOR_RELATIONS),
	81	+ ("shop", Surface.PRODUCTS))
	82	+# hosts that are never a company's own canonical domain even when the website redirects there
	83	+SHARED_HOSTS = ("linkedin.com", "facebook.com", "instagram.com", "twitter.com", "x.com", "youtube.com", "wixsite.com", "squarespace.com", "godaddy.com",
	84	+ "hubspot.com", "wordpress.com", "blogspot.com", "google.com", "sedo.com", "hugedomains.com", "dan.com", "afternic.com", "bluehost.com")
	85	+SOFT_404_RE = re.compile(r"(page not found\|404\|not be found\|doesn'?t exist\|no longer available\|nicht gefunden\|introuvable)", re.IGNORECASE)
	86	+EMBED_ATS_RE = re.compile(
	87	+ r"(?:boards\|job-boards)\.greenhouse\.io/(?:embed/job_board(?:/js)?\?(?:[^\"'\s]*&)?for=\|)([a-z0-9_-]{2,})\|"
	88	+ r"jobs\.(?:eu\.)?lever\.co/([a-z0-9_-]{2,})\|jobs\.ashbyhq\.com/([a-z0-9_.-]{2,})\|apply\.workable\.com/([a-z0-9_-]{2,})\|"
	89	+ r"([a-z0-9-]{2,})\.recruitee\.com\|([a-z0-9-]{2,})\.jobs\.personio\.(?:de\|com)\|([a-z0-9-]{2,})\.teamtailor\.com\|"
	90	+ r"(?:careers\|jobs)\.smartrecruiters\.com/([A-Za-z0-9_-]{2,})\|([a-z0-9-]+\.wd\d+\.myworkdayjobs\.com/[^\"'\s<>]+)", re.IGNORECASE)
	91	+EMBED_VENDORS = ("greenhouse", "lever", "ashby", "workable", "recruitee", "personio", "teamtailor", "smartrecruiters", "workday")
	92	+GH_JID_RE = re.compile(r"[?&]gh_jid=\d+", re.IGNORECASE)
	93	+
	94	+
	95	+@dataclass(slots=True)
	96	+class Candidate:
	97	+ url: str
	98	+ surface: str
	99	+ confidence: float
	100	+ method: str
	101	+ anchor: str \| None = None
	102	+ verified: bool = False # we fetched it successfully during discovery
	103	+ connector_id: str \| None = None
	104	+ config: dict[str, Any] = field(default_factory=dict)
	105	+
	106	+ @property
	107	+ def score(self) -> float:
	108	+ depth = max(0, urlparse(self.url).path.strip("/").count("/"))
	109	+ return self.confidence * METHOD_WEIGHT.get(self.method, 0.8) * (1.0 if self.verified else 0.9) * (1.0 - 0.05 * min(depth, 3))
	110	+
	111	+
	112	+@dataclass(slots=True)
	113	+class DiscoveryResult:
	114	+ company_id: str
	115	+ website: str
	116	+ final_url: str \| None = None
	117	+ canonical_domain: str = ""
	118	+ redirect_domain: str \| None = None
	119	+ candidates: list[Candidate] = field(default_factory=list)
	120	+ sensors: list[dict[str, Any]] = field(default_factory=list)
	121	+ status: str = OnboardingStatus.ACTIVE
	122	+ error: str \| None = None
	123	+ notes: list[str] = field(default_factory=list)
	124	+ subdomains: list[str] = field(default_factory=list)
	125	+ same_as: list[str] = field(default_factory=list)
	126	+ requests: int = 0
	127	+ duration_ms: int = 0
	128	+ sitemaps: list[str] = field(default_factory=list)
	129	+ ats: list[dict[str, Any]] = field(default_factory=list)
	130	+
	131	+ def table(self) -> list[dict[str, Any]]:
	132	+ return [{"surface": s["surface"], "url": s["url"], "connector": s["connector_id"], "confidence": s["discovery_confidence"], "method": s["discovery_method"],
	133	+ "interval_s": s["base_interval_s"], "tier": s["tier"], "quality": s["quality_score"]} for s in self.sensors]
	134	+
	135	+
	136	+# ------------------------------------------------------------------------------------------------------------ helpers
	137	+
	138	+
	139	+class _Budget:
	140	+ def __init__(self, seconds: float):
	141	+ self.deadline = time.monotonic() + seconds
	142	+
	143	+ @property
	144	+ def left(self) -> float:
	145	+ return self.deadline - time.monotonic()
	146	+
	147	+ def ok(self, need: float = 3.0) -> bool:
	148	+ return self.left > need
	149	+
	150	+
	151	+async def _get(fetcher: Fetcher, url: str, *, max_bytes: int = PROBE_MAX_BYTES, res: DiscoveryResult, respect_robots: bool = True,
	152	+ accept: str \| None = None) -> FetchResult \| None:
	153	+ res.requests += 1
	154	+ try:
	155	+ return await fetcher.get(url, max_bytes=max_bytes, respect_robots=respect_robots, retries=0, accept=accept)
	156	+ except NotModified:
	157	+ return None
	158	+ except (FetchError, BlockedError) as exc:
	159	+ log.debug("discovery fetch failed", extra={"url": url, "failure": str(exc.failure), "error": str(exc)[:200]})
	160	+ return None
	161	+ except Exception as exc: # noqa: BLE001
	162	+ log.debug("discovery fetch error", extra={"url": url, "error": f"{exc.__class__.__name__}: {exc}"[:200]})
	163	+ return None
	164	+
	165	+
	166	+def _is_soft_404(result: FetchResult) -> bool:
	167	+ if not result.is_html:
	168	+ return False
	169	+ head = result.text[:4000]
	170	+ m = re.search(r"<title[^>]>(.?)</title>", head, re.IGNORECASE \| re.DOTALL)
	171	+ title = m.group(1) if m else ""
	172	+ return bool(SOFT_404_RE.search(title)) or (len(result.content) < 600 and bool(SOFT_404_RE.search(head)))
	173	+
	174	+
	175	+async def _resolves(host: str) -> bool:
	176	+ def _r() -> bool:
	177	+ try:
	178	+ socket.getaddrinfo(host, 443, proto=socket.IPPROTO_TCP)
	179	+ return True
	180	+ except socket.gaierror:
	181	+ return False
	182	+ try:
	183	+ return await asyncio.wait_for(asyncio.to_thread(_r), 4.0)
	184	+ except TimeoutError:
	185	+ return False
	186	+
	187	+
	188	+def _website_variants(website: str, canonical_domain: str) -> list[str]:
	189	+ out: list[str] = []
	190	+ w = (website or "").strip()
	191	+ if w and not w.startswith(("http://", "https://")):
	192	+ w = "https://" + w
	193	+ if w:
	194	+ out.append(w)
	195	+ dom = canonical_domain.lower().removeprefix("www.")
	196	+ for u in (f"https://www.{dom}/", f"https://{dom}/", f"http://www.{dom}/", f"http://{dom}/"):
	197	+ if u not in out and canonicalize_url(u) not in {canonicalize_url(x) for x in out}:
	198	+ out.append(u)
	199	+ return out[:4]
	200	+
	201	+
	202	+def _interval_for(surface: str, tier: int) -> int:
	203	+ base = int(SURFACE_BASE_INTERVAL_S.get(surface, 86400) * TIER_FACTOR.get(int(tier or 4), 1.0))
	204	+ return max(settings.min_interval_s, min(settings.max_interval_s, base))
	205	+
	206	+
	207	+def _quality(conf: float, surface: str, verified: bool) -> float:
	208	+ importance = SURFACE_IMPORTANCE.get(surface, 0.3)
	209	+ reliability = 1.0 if verified else 0.8
	210	+ return round(100.0 * min(1.0, conf) * (0.5 + 0.5 * importance) * reliability, 1)
	211	+
	212	+
	213	+# ------------------------------------------------------------------------------------------------------------ phases
	214	+
	215	+
	216	+def _ats_from_text(text: str, links: list[str]) -> list[tuple[str, str, str]]:
	217	+ """(vendor, token, board_url) from links and embedded scripts/iframes."""
	218	+ found: dict[tuple[str, str], str] = {}
	219	+ for u in links:
	220	+ hit = detect_ats(u)
	221	+ if hit and hit[0] in EMBED_VENDORS:
	222	+ found.setdefault(hit, u)
	223	+ for m in EMBED_ATS_RE.finditer(text[:2_000_000]):
	224	+ groups = m.groups()
	225	+ for vendor, g in zip(EMBED_VENDORS, groups, strict=False):
	226	+ if g:
	227	+ token = g
	228	+ board = m.group(0)
	229	+ if vendor == "workday":
	230	+ board = "https://" + g if not g.startswith("http") else g
	231	+ hit = detect_ats(board)
	232	+ token = hit[1] if hit else g.split(".")[0]
	233	+ elif vendor == "greenhouse":
	234	+ board = f"https://boards.greenhouse.io/{token}"
	235	+ elif vendor == "lever":
	236	+ board = f"https://jobs.lever.co/{token}"
	237	+ elif vendor == "ashby":
	238	+ board = f"https://jobs.ashbyhq.com/{token}"
	239	+ elif vendor == "workable":
	240	+ board = f"https://apply.workable.com/{token}"
	241	+ elif vendor == "recruitee":
	242	+ board = f"https://{token}.recruitee.com"
	243	+ elif vendor == "personio":
	244	+ board = f"https://{token}.jobs.personio.de"
	245	+ elif vendor == "teamtailor":
	246	+ board = f"https://{token}.teamtailor.com"
	247	+ elif vendor == "smartrecruiters":
	248	+ board = f"https://careers.smartrecruiters.com/{token}"
	249	+ if token.lower() in ("embed", "js", "job_board", "www", "api", "boards"):
	250	+ continue
	251	+ found.setdefault((vendor, token), board)
	252	+ return [(v, t, b) for (v, t), b in found.items()][:4]
	253	+
	254	+
	255	+async def _phase_homepage(fetcher: Fetcher, company: dict[str, Any], res: DiscoveryResult, budget: _Budget) -> tuple[FetchResult \| None, Any]:
	256	+ canonical = str(company.get("canonical_domain") or "")
	257	+ for url in _website_variants(str(company.get("website") or ""), canonical):
	258	+ if not budget.ok(10):
	259	+ break
	260	+ res.requests += 1
	261	+ try:
	262	+ r = await fetcher.get(url, max_bytes=HOMEPAGE_MAX_BYTES, retries=1)
	263	+ except BlockedError as exc:
	264	+ res.notes.append(f"homepage blocked: {exc.failure}")
	265	+ res.error = f"{exc.failure}: {exc}"[:300]
	266	+ continue
	267	+ except FetchError as exc:
	268	+ res.error = f"{exc.failure}: {exc}"[:300]
	269	+ if exc.failure in (FailureClass.DNS, FailureClass.BLOCKED_DESTINATION):
	270	+ res.notes.append(f"{url}: {exc.failure}")
	271	+ continue
	272	+ except Exception as exc: # noqa: BLE001
	273	+ res.error = f"{exc.__class__.__name__}: {exc}"[:300]
	274	+ continue
	275	+ if not r.is_html or _is_soft_404(r):
	276	+ res.notes.append(f"{url}: not an HTML homepage")
	277	+ continue
	278	+ page_conn = connectors.get("generic-html-v1")
	279	+ sensor_stub = {"url": r.final_url, "surface": Surface.HOMEPAGE, "config": {"canonical_domain": canonical}}
	280	+ try:
	281	+ extraction = page_conn.extract(sensor_stub, r)
	282	+ except Exception as exc: # noqa: BLE001
	283	+ res.error = f"homepage parse failed: {exc}"[:300]
	284	+ continue
	285	+ return r, extraction
	286	+ return None, None
	287	+
	288	+
	289	+async def _phase_robots_sitemaps(fetcher: Fetcher, base: str, res: DiscoveryResult, budget: _Budget, canonical_domain: str) -> list[Candidate]:
	290	+ origin = f"{urlparse(base).scheme}://{urlparse(base).netloc}"
	291	+ sitemap_urls: list[str] = []
	292	+ r = await _get(fetcher, f"{origin}/robots.txt", res=res, respect_robots=False, max_bytes=256 * 1024)
	293	+ if r is not None and r.status == 200 and not r.is_html:
	294	+ for line in r.text.splitlines():
	295	+ if line.lower().startswith("sitemap:"):
	296	+ u = line.split(":", 1)[1].strip()
	297	+ if u.startswith("http") and same_company_host(u, canonical_domain) and u not in sitemap_urls:
	298	+ sitemap_urls.append(u)
	299	+ if not sitemap_urls:
	300	+ sitemap_urls = [f"{origin}/sitemap.xml"]
	301	+ cands: list[Candidate] = []
	302	+ from companyatlas.connectors.sitemap import _decode, parse_sitemap # local import: connector module
	303	+
	304	+ seen_urls = 0
	305	+ for sm_url in sitemap_urls[:3]:
	306	+ if not budget.ok(8):
	307	+ break
	308	+ r = await _get(fetcher, sm_url, res=res, max_bytes=4 * 1024 * 1024, accept="application/xml,text/xml,/;q=0.5")
	309	+ if r is None or r.status != 200 or r.is_html:
	310	+ continue
	311	+ pages, children = parse_sitemap(_decode(r.content))
	312	+ res.sitemaps.append(sm_url)
	313	+ if children and not pages:
	314	+ for child, _lm in children[:MAX_SITEMAP_CHILDREN]:
	315	+ if not budget.ok(6) or seen_urls >= settings.discovery_max_sitemap_urls:
	316	+ break
	317	+ cr = await _get(fetcher, child, res=res, max_bytes=4 * 1024 * 1024, accept="application/xml,text/xml,/;q=0.5")
	318	+ if cr is None or cr.is_html:
	319	+ continue
	320	+ p, _c = parse_sitemap(_decode(cr.content))
	321	+ pages.extend(p)
	322	+ for loc, _lm in pages[: settings.discovery_max_sitemap_urls]:
	323	+ seen_urls += 1
	324	+ if not same_company_host(loc, canonical_domain) or is_static_asset(loc) or looks_like_trap(loc):
	325	+ continue
	326	+ surface, conf = classify_url(loc, canonical_domain=canonical_domain)
	327	+ if surface in (Surface.OTHER, Surface.HOMEPAGE, Surface.SITEMAP) or conf < settings.discovery_min_confidence:
	328	+ continue
	329	+ cands.append(Candidate(url=loc, surface=str(surface), confidence=conf, method="sitemap"))
	330	+ if pages:
	331	+ break
	332	+ for sm_url in res.sitemaps[:1]:
	333	+ cands.append(Candidate(url=sm_url, surface=str(Surface.SITEMAP), confidence=0.95, method="robots", verified=True, config={"canonical_domain": canonical_domain}))
	334	+ return cands
	335	+
	336	+
	337	+async def _phase_probes(fetcher: Fetcher, base: str, have: set[str], res: DiscoveryResult, budget: _Budget, canonical_domain: str) -> list[Candidate]:
	338	+ origin = f"{urlparse(base).scheme}://{urlparse(base).netloc}"
	339	+ home_canon = canonicalize_url(base)
	340	+ cands: list[Candidate] = []
	341	+ probes = 0
	342	+ for surface, paths in PROBE_PATHS.items():
	343	+ if surface in have or probes >= MAX_PROBES or not budget.ok(6):
	344	+ continue
	345	+ for path in paths[:2]:
	346	+ if probes >= MAX_PROBES:
	347	+ break
	348	+ probes += 1
	349	+ url = origin + path
	350	+ r = await _get(fetcher, url, res=res)
	351	+ if r is None or r.status != 200 or not r.is_html or _is_soft_404(r):
	352	+ continue
	353	+ if canonicalize_url(r.final_url) == home_canon:
	354	+ continue # redirected back to the homepage: the surface does not exist
	355	+ if not same_company_host(r.final_url, canonical_domain):
	356	+ hit = detect_ats(r.final_url)
	357	+ if hit:
	358	+ res.notes.append(f"{path} → ATS {hit[0]}")
	359	+ continue
	360	+ final = canonicalize_url(r.final_url)
	361	+ s2, c2 = classify_url(final, canonical_domain=canonical_domain)
	362	+ final_path = urlparse(final).path.rstrip("/").lower()
	363	+ if not (s2 == surface or final_path.endswith(path.rstrip("/").lower()) or (s2 == Surface.OTHER and final_path)):
	364	+ continue # redirected to a different known surface (marketing redirect): not this surface
	365	+ if s2 == Surface.OTHER and not final_path.endswith(path.rstrip("/").lower()):
	366	+ continue
	367	+ conf = max(0.6, c2 if s2 == surface else 0.6)
	368	+ cands.append(Candidate(url=final, surface=str(surface), confidence=conf, method="probe", verified=True))
	369	+ have.add(surface)
	370	+ break
	371	+ return cands
	372	+
	373	+
	374	+async def _phase_subdomains(fetcher: Fetcher, canonical_domain: str, have: set[str], res: DiscoveryResult, budget: _Budget) -> list[Candidate]:
	375	+ dom = canonical_domain.lower().removeprefix("www.")
	376	+ cands: list[Candidate] = []
	377	+ gets = 0
	378	+ checked: set[str] = set()
	379	+ for sub, surface in SUBDOMAINS:
	380	+ host = f"{sub}.{dom}"
	381	+ if host in checked or gets >= MAX_SUBDOMAIN_GETS or not budget.ok(6):
	382	+ continue
	383	+ checked.add(host)
	384	+ if not await _resolves(host):
	385	+ continue
	386	+ res.subdomains.append(host)
	387	+ gets += 1
	388	+ if surface == Surface.STATUS:
	389	+ r = await _get(fetcher, f"https://{host}/api/v2/summary.json", res=res, accept="application/json", max_bytes=1024 * 1024)
	390	+ if r is not None and r.is_json:
	391	+ cands.append(Candidate(url=f"https://{host}/api/v2/summary.json", surface=str(Surface.STATUS), confidence=0.95, method="subdomain",
	392	+ verified=True, connector_id="statuspage-v1"))
	393	+ continue
	394	+ r = await _get(fetcher, f"https://{host}/", res=res)
	395	+ if r is not None and r.is_html and not _is_soft_404(r):
	396	+ cands.append(Candidate(url=r.final_url, surface=str(Surface.STATUS), confidence=0.8, method="subdomain", verified=True))
	397	+ continue
	398	+ r = await _get(fetcher, f"https://{host}/", res=res)
	399	+ if r is None or not r.is_html or _is_soft_404(r):
	400	+ continue
	401	+ hit = detect_ats(r.final_url)
	402	+ if hit:
	403	+ spec = ats_sensor_spec(hit[0], hit[1], r.final_url)
	404	+ if spec:
	405	+ cands.append(Candidate(url=spec[0], surface=str(Surface.JOBS_BOARD), confidence=0.95, method="ats", verified=False, connector_id=spec[1], config=spec[2]))
	406	+ continue
	407	+ if registrable_domain(r.final_url) != dom:
	408	+ continue
	409	+ conf = 0.9 if surface not in have else 0.75
	410	+ cands.append(Candidate(url=r.final_url, surface=str(surface), confidence=conf, method="subdomain", verified=True))
	411	+ return cands
	412	+
	413	+
	414	+async def _phase_ats(fetcher: Fetcher, home: FetchResult, home_links: list[str], careers: Candidate \| None, res: DiscoveryResult, budget: _Budget,
	415	+ canonical_domain: str) -> list[Candidate]:
	416	+ text = home.text
	417	+ links = list(home_links)
	418	+ careers_res: FetchResult \| None = None
	419	+ if careers is not None and budget.ok(8):
	420	+ careers_res = await _get(fetcher, careers.url, res=res, max_bytes=HOMEPAGE_MAX_BYTES)
	421	+ if careers_res is not None and careers_res.is_html:
	422	+ careers.verified = True
	423	+ text += "\n" + careers_res.text
	424	+ hit = detect_ats(careers_res.final_url)
	425	+ if hit:
	426	+ links.append(careers_res.final_url)
	427	+ for m in re.finditer(r"""(?:href\|src\|action\|data-url\|data-src)\s=\s["']([^"']+)["']""", careers_res.text[:1_500_000], re.IGNORECASE):
	428	+ u = absolutize(careers_res.final_url, m.group(1))
	429	+ if u:
	430	+ links.append(u)
	431	+ cands: list[Candidate] = []
	432	+ found = _ats_from_text(text, links)
	433	+ if not any(v == "greenhouse" for v, _t, _b in found) and GH_JID_RE.search(text):
	434	+ # Greenhouse-hosted jobs rendered on the company's own site (`?gh_jid=`): the board token is usually the company's slug / domain label
	435	+ label = canonical_domain.removeprefix("www.").split(".")[0].lower()
	436	+ if len(label) >= 2:
	437	+ found.append(("greenhouse", label, f"https://boards.greenhouse.io/{label}")) # verified below with one request
	438	+ for vendor, token, board in found:
	439	+ spec = ats_sensor_spec(vendor, token, board)
	440	+ if spec is None:
	441	+ continue
	442	+ api_url, connector_id, config = spec
	443	+ config["canonical_domain"] = canonical_domain
	444	+ verified = False
	445	+ if budget.ok(8) and not any(a["vendor"] == vendor for a in res.ats):
	446	+ conn = connectors.get(connector_id)
	447	+ try:
	448	+ res.requests += 1
	449	+ from companyatlas.sdk.connector import ConnectorContext
	450	+
	451	+ r = await asyncio.wait_for(conn.fetch(ConnectorContext(company={"canonical_domain": canonical_domain}), {"url": api_url, "config": config}, fetcher), 25)
	452	+ ex = conn.extract({"url": api_url, "config": config, "surface": Surface.JOBS_BOARD}, r)
	453	+ verified = True
	454	+ config["verified_job_count"] = len(ex.jobs)
	455	+ except Exception as exc: # noqa: BLE001
	456	+ res.notes.append(f"ATS {vendor}/{token} not verified: {exc.__class__.__name__}")
	457	+ continue
	458	+ res.ats.append({"vendor": vendor, "token": token, "board_url": board, "api_url": api_url, "verified": verified})
	459	+ cands.append(Candidate(url=api_url, surface=str(Surface.JOBS_BOARD), confidence=0.97 if verified else 0.85, method="ats", verified=verified,
	460	+ connector_id=connector_id, config=config))
	461	+ return cands
	462	+
	463	+
	464	+# ------------------------------------------------------------------------------------------------------------ selection
	465	+
	466	+
	467	+LOCALE_SEG_RE = re.compile(r"^/([a-z]{2})(?:[-_][a-z]{2})?(?=/\|$)", re.IGNORECASE)
	468	+
	469	+
	470	+def _locale_of(url: str) -> str \| None:
	471	+ m = LOCALE_SEG_RE.match(urlparse(url).path or "")
	472	+ return m.group(1).lower() if m else None
	473	+
	474	+
	475	+def select_sensors(cands: list[Candidate], *, company: dict[str, Any], canonical_domain: str, now: datetime, fetch_now: bool = False,
	476	+ home_url: str \| None = None) -> list[dict[str, Any]]:
	477	+ home_locale = _locale_of(home_url) if home_url else None
	478	+ best: dict[str, Candidate] = {}
	479	+ best_score: dict[str, float] = {}
	480	+ for c in cands:
	481	+ if c.confidence < settings.discovery_min_confidence and c.method not in ("ats", "robots"):
	482	+ continue
	483	+ score = c.score
	484	+ loc = _locale_of(c.url)
	485	+ if home_locale and loc and loc != home_locale:
	486	+ score *= 0.85 # prefer the homepage's language edition (/en/ over /jp/)
	487	+ if c.surface not in best or score > best_score[c.surface]:
	488	+ best[c.surface], best_score[c.surface] = c, score
	489	+ # never keep a separate feed sensor pointing at the same URL as another surface
	490	+ chosen = list(best.values())
	491	+ seen_canon: set[str] = set()
	492	+ ranked = sorted(chosen, key=lambda c: (SURFACE_IMPORTANCE.get(c.surface, 0.3) * c.score), reverse=True)
	493	+ out: list[dict[str, Any]] = []
	494	+ tier = int(company.get("tier") or 4)
	495	+ importance = float(company.get("importance") or 0.2)
	496	+ for c in ranked:
	497	+ canon = canonicalize_url(c.url)
	498	+ if canon in seen_canon:
	499	+ continue
	500	+ seen_canon.add(canon)
	501	+ connector = connectors.get(c.connector_id) if c.connector_id else connectors.for_surface(c.surface, c.url)
	502	+ base = _interval_for(c.surface, tier)
	503	+ if connector.meta.default_interval_s and connector.meta.default_interval_s < base and c.surface in (Surface.JOBS_BOARD, Surface.FEED):
	504	+ base = max(settings.min_interval_s, connector.meta.default_interval_s)
	505	+ cfg = {"canonical_domain": canonical_domain, **c.config, "discovery": {"version": DISCOVERY_VERSION, "method": c.method, "anchor": c.anchor,
	506	+ "verified": c.verified, "at": now.isoformat()}}
	507	+ out.append({
	508	+ "id": new_id("sensor"), "company_id": company["id"], "surface": c.surface, "connector_id": connector.connector_id, "url": c.url,
	509	+ "canonical_url": canon, "domain": registrable_domain(c.url), "discovery_confidence": round(min(0.99, c.confidence), 3),
	510	+ "discovery_method": c.method, "quality_score": _quality(c.confidence, c.surface, c.verified), "status": SensorStatus.PENDING,
	511	+ "tier": tier_for_interval(base), "base_interval_s": base, "current_interval_s": base,
	512	+ "next_run_at": now if fetch_now else now + timedelta(seconds=random.uniform(0, base)),
	513	+ "priority": round(min(1.0, 0.3 + 0.5 * importance + 0.2 * SURFACE_IMPORTANCE.get(c.surface, 0.3)), 3), "config": cfg,
	514	+ })
	515	+ if len(out) >= settings.discovery_max_sensors_per_company:
	516	+ break
	517	+ return out
	518	+
	519	+
	520	+# ------------------------------------------------------------------------------------------------------------ main entry
	521	+
	522	+
	523	+async def discover_company(company: dict[str, Any], *, fetcher: Fetcher, dry_run: bool = False, fetch_now: bool = False,
	524	+ budget_s: float = COMPANY_BUDGET_S) -> DiscoveryResult:
	525	+ """Full discovery for one company row. Never raises; persists sensors/domains/company status unless `dry_run`."""
	526	+ t0 = time.perf_counter()
	527	+ res = DiscoveryResult(company_id=str(company["id"]), website=str(company.get("website") or ""), canonical_domain=str(company.get("canonical_domain") or ""))
	528	+ try:
	529	+ await asyncio.wait_for(_discover(company, fetcher, res, _Budget(budget_s), fetch_now=fetch_now), budget_s + 15)
	530	+ except TimeoutError:
	531	+ res.notes.append("discovery budget exhausted")
	532	+ if not res.sensors:
	533	+ res.status, res.error = OnboardingStatus.FAILED, res.error or "timeout"
	534	+ except Exception as exc:
	535	+ log.exception("discovery crashed", extra={"company_id": company.get("id")})
	536	+ res.status, res.error = OnboardingStatus.FAILED, f"{exc.__class__.__name__}: {exc}"[:300]
	537	+ res.duration_ms = int((time.perf_counter() - t0) * 1000)
	538	+ if not dry_run:
	539	+ try:
	540	+ await persist(company, res)
	541	+ except Exception:
	542	+ log.exception("discovery persist failed", extra={"company_id": company.get("id")})
	543	+ res.status, res.error = OnboardingStatus.FAILED, "persist failed"
	544	+ return res
	545	+
	546	+
	547	+async def _discover(company: dict[str, Any], fetcher: Fetcher, res: DiscoveryResult, budget: _Budget, *, fetch_now: bool) -> None:
	548	+ canonical = str(company.get("canonical_domain") or "").lower()
	549	+ home, extraction = await _phase_homepage(fetcher, company, res, budget)
	550	+ if home is None:
	551	+ err = (res.error or "").upper()
	552	+ res.status = OnboardingStatus.NO_WEBSITE if ("DNS" in err or "BLOCKED_DESTINATION" in err or not canonical) else OnboardingStatus.FAILED
	553	+ res.error = res.error or "homepage unreachable"
	554	+ return
	555	+ res.final_url = home.final_url
	556	+ final_dom = registrable_domain(home.final_url)
	557	+ if final_dom != registrable_domain(canonical):
	558	+ res.redirect_domain = final_dom
	559	+ res.notes.append(f"website redirects to {final_dom}")
	560	+ canonical_domain = final_dom if (res.redirect_domain and not any(final_dom.endswith(h) for h in SHARED_HOSTS)) else canonical
	561	+ res.canonical_domain = canonical_domain
	562	+ res.same_as = [s for s in (extraction.meta.get("same_as") or []) if isinstance(s, str)][:20]
	563	+ now = datetime.now(UTC)
	564	+ cands: list[Candidate] = [Candidate(url=home.final_url, surface=str(Surface.HOMEPAGE), confidence=0.99, method="nav", verified=True)]
	565	+ home_links: list[str] = []
	566	+ for d in extraction.discovered:
	567	+ assert isinstance(d, DiscoveredUrl)
	568	+ home_links.append(d.url)
	569	+ if d.surface == Surface.JOBS_BOARD:
	570	+ hit = detect_ats(d.url)
	571	+ if hit:
	572	+ spec = ats_sensor_spec(hit[0], hit[1], d.url)
	573	+ if spec:
	574	+ cands.append(Candidate(url=spec[0], surface=str(Surface.JOBS_BOARD), confidence=0.9, method="ats", connector_id=spec[1], config=spec[2]))
	575	+ continue
	576	+ if d.surface == Surface.FEED:
	577	+ cands.append(Candidate(url=d.url, surface=str(Surface.FEED), confidence=d.confidence, method="feed", connector_id="feed-v1"))
	578	+ continue
	579	+ if not same_company_host(d.url, canonical_domain):
	580	+ continue
	581	+ cands.append(Candidate(url=d.url, surface=str(d.surface), confidence=d.confidence, method=d.method, anchor=d.anchor))
	582	+ # raw homepage links for ATS scanning (including off-domain)
	583	+ for m in re.finditer(r"""(?:href\|src\|action\|data-url)\s=\s["']([^"']+)["']""", home.text[:1_500_000], re.IGNORECASE):
	584	+ u = absolutize(home.final_url, m.group(1))
	585	+ if u:
	586	+ home_links.append(u)
	587	+ if budget.ok(10):
	588	+ cands.extend(await _phase_robots_sitemaps(fetcher, home.final_url, res, budget, canonical_domain))
	589	+ have = {c.surface for c in cands if c.confidence >= 0.7}
	590	+ careers = max((c for c in cands if c.surface == Surface.CAREERS), key=lambda c: c.score, default=None)
	591	+ if budget.ok(10):
	592	+ cands.extend(await _phase_ats(fetcher, home, home_links, careers, res, budget, canonical_domain))
	593	+ if budget.ok(10):
	594	+ cands.extend(await _phase_probes(fetcher, home.final_url, have, res, budget, canonical_domain))
	595	+ if budget.ok(10):
	596	+ cands.extend(await _phase_subdomains(fetcher, canonical_domain, have, res, budget))
	597	+ res.candidates = cands
	598	+ res.sensors = select_sensors(cands, company={**company, "canonical_domain": canonical_domain}, canonical_domain=canonical_domain, now=now, fetch_now=fetch_now,
	599	+ home_url=home.final_url)
	600	+ res.status = OnboardingStatus.ACTIVE if res.sensors else OnboardingStatus.FAILED
	601	+ if not res.sensors:
	602	+ res.error = res.error or "no sensors discovered"
	603	+
	604	+
	605	+# ------------------------------------------------------------------------------------------------------------ persistence
	606	+
	607	+
	608	+async def persist(company: dict[str, Any], res: DiscoveryResult) -> None:
	609	+ now = datetime.now(UTC)
	610	+ async with transaction() as conn:
	611	+ # domains: redirect target / subdomains
	612	+ if res.redirect_domain:
	613	+ await _upsert_domain(conn, company["id"], res.redirect_domain, "redirect")
	614	+ for host in res.subdomains:
	615	+ await _upsert_domain(conn, company["id"], host, "subdomain")
	616	+ new_canonical = None
	617	+ if res.redirect_domain and res.canonical_domain != str(company.get("canonical_domain") or "").lower():
	618	+ clash = await fetch_one(conn, "select id from companies where canonical_domain = :d and id <> :id", d=res.canonical_domain, id=company["id"])
	619	+ if clash is None:
	620	+ new_canonical = res.canonical_domain
	621	+ await _upsert_domain(conn, company["id"], str(company.get("canonical_domain")), "former")
	622	+ else:
	623	+ res.notes.append(f"canonical domain {res.canonical_domain} already belongs to {clash['id']} — kept {company.get('canonical_domain')}")
	624	+ inserted = 0
	625	+ for s in res.sensors:
	626	+ row = await fetch_one(conn, """
	627	+ insert into sensors (id, company_id, surface, connector_id, url, canonical_url, domain, discovery_confidence, discovery_method, quality_score,
	628	+ status, tier, base_interval_s, current_interval_s, next_run_at, priority, config)
	629	+ values (:id, :company_id, :surface, :connector_id, :url, :canonical_url, :domain, :discovery_confidence, :discovery_method, :quality_score,
	630	+ :status, :tier, :base_interval_s, :current_interval_s, :next_run_at, :priority, cast(:config as jsonb))
	631	+ on conflict (company_id, canonical_url) do update set
	632	+ discovery_confidence = greatest(sensors.discovery_confidence, excluded.discovery_confidence),
	633	+ connector_id = case when sensors.status = 'retired' then sensors.connector_id else excluded.connector_id end,
	634	+ surface = case when sensors.status in ('retired', 'paused') then sensors.surface else excluded.surface end,
	635	+ config = sensors.config \|\| excluded.config, updated_at = now()
	636	+ returning (xmax = 0) as inserted
	637	+ """, {s, "config": jsonb(s["config"]), "status": str(s["status"])})
	638	+ if row and row.get("inserted"):
	639	+ inserted += 1
	640	+ stats = {"discovery": {"version": DISCOVERY_VERSION, "at": now.isoformat(), "sensors": len(res.sensors), "inserted": inserted, "requests": res.requests,
	641	+ "duration_ms": res.duration_ms, "candidates": len(res.candidates), "sitemaps": res.sitemaps[:3], "ats": res.ats, "notes": res.notes[:10],
	642	+ "subdomains": res.subdomains}}
	643	+ source_meta_patch: dict[str, Any] = {}
	644	+ if res.same_as:
	645	+ source_meta_patch["same_as"] = res.same_as
	646	+ if res.final_url:
	647	+ source_meta_patch["final_url"] = res.final_url
	648	+ await execute(conn, """
	649	+ update companies set onboarding_status = cast(:st as text), onboarding_error = cast(:err as text), stats = stats \|\| cast(:stats as jsonb),
	650	+ source_meta = source_meta \|\| cast(:sm as jsonb), canonical_domain = coalesce(cast(:cd as text), canonical_domain),
	651	+ website = case when cast(:final as text) is not null and cast(:st as text) = 'active' then cast(:final as text) else website end,
	652	+ updated_at = now()
	653	+ where id = :id
	654	+ """, st=str(res.status), err=(res.error[:500] if res.error else None), stats=jsonb(stats), sm=jsonb(source_meta_patch), cd=new_canonical,
	655	+ final=res.final_url if res.final_url and (new_canonical or same_company_host(res.final_url, str(company.get("canonical_domain") or ""))) else None,
	656	+ id=company["id"])
	657	+
	658	+
	659	+async def _upsert_domain(conn: Any, company_id: str, domain: str, kind: str) -> None:
	660	+ await execute(conn, """
	661	+ insert into domains (id, company_id, domain, kind) values (:id, :cid, :domain, :kind)
	662	+ on conflict (domain, company_id) do update set last_seen_at = now(), kind = case when domains.kind = 'primary' then domains.kind else excluded.kind end
	663	+ """, id=new_id("domain"), cid=company_id, domain=domain.lower(), kind=kind)
	664	+
	665	+
	666	+# ------------------------------------------------------------------------------------------------------------ onboarding worker
	667	+
	668	+
	669	+async def claim_discover_jobs(limit: int, worker: str) -> list[dict[str, Any]]:
	670	+ async with transaction() as conn:
	671	+ rows = await fetch_all(conn, """
	672	+ with due as (
	673	+ select id from queue_jobs where kind = 'discover' and status = 'pending' and run_at <= now()
	674	+ order by priority desc, run_at limit :limit for update skip locked)
	675	+ update queue_jobs q set status = 'running', locked_at = now(), locked_by = :worker, attempts = attempts + 1
	676	+ from due where q.id = due.id returning q.id, q.key, q.payload, q.attempts, q.max_attempts
	677	+ """, limit=limit, worker=worker)
	678	+ return rows
	679	+
	680	+
	681	+async def finish_discover_job(job_id: str, *, ok: bool, error: str \| None, attempts: int, max_attempts: int) -> None:
	682	+ async with transaction() as conn:
	683	+ if ok:
	684	+ await execute(conn, "update queue_jobs set status = 'done', finished_at = now(), last_error = null where id = :id", id=job_id)
	685	+ elif attempts >= max_attempts:
	686	+ await execute(conn, "update queue_jobs set status = 'dead', finished_at = now(), last_error = :e where id = :id", id=job_id, e=(error or "")[:500])
	687	+ else:
	688	+ await execute(conn, """update queue_jobs set status = 'pending', locked_at = null, locked_by = null, last_error = :e,
	689	+ run_at = now() + make_interval(mins => :mins) where id = :id""", id=job_id, e=(error or "")[:500], mins=30 * attempts)
	690	+
	691	+
	692	+async def onboard_pending(limit: int = 50, concurrency: int \| None = None, *, fetcher: Fetcher \| None = None, company_slug: str \| None = None,
	693	+ fetch_now: bool = False, dry_run: bool = False, worker: str = "onboard") -> dict[str, Any]:
	694	+ """Discover pending companies: queue jobs first (SKIP LOCKED), then `companies.onboarding_status='pending'` without a job."""
	695	+ concurrency = concurrency or settings.onboarding_concurrency
	696	+ own_fetcher = fetcher is None
	697	+ fetcher = fetcher or Fetcher()
	698	+ if own_fetcher:
	699	+ await fetcher.open()
	700	+ stats = {"claimed": 0, "active": 0, "failed": 0, "no_website": 0, "sensors": 0}
	701	+ try:
	702	+ targets: list[tuple[dict[str, Any], dict[str, Any] \| None]] = []
	703	+ async with transaction() as conn:
	704	+ if company_slug:
	705	+ comp = await fetch_one(conn, "select * from companies where slug = :s or id = :s or canonical_domain = :s", s=company_slug)
	706	+ if comp is None:
	707	+ raise LookupError(f"company {company_slug!r} not found")
	708	+ targets.append((comp, None))
	709	+ if not company_slug:
	710	+ jobs = await claim_discover_jobs(limit, worker)
	711	+ for j in jobs:
	712	+ cid = (j.get("payload") or {}).get("company_id") or j["key"].removeprefix("discover:")
	713	+ async with transaction() as conn:
	714	+ comp = await fetch_one(conn, "select * from companies where id = :id or slug = :id", id=cid)
	715	+ if comp is None:
	716	+ await finish_discover_job(j["id"], ok=False, error="company not found", attempts=j["attempts"], max_attempts=j["max_attempts"])
	717	+ continue
	718	+ targets.append((comp, j))
	719	+ if len(targets) < limit:
	720	+ async with transaction() as conn:
	721	+ rows = await fetch_all(conn, """
	722	+ update companies set onboarding_status = 'discovering', updated_at = now()
	723	+ where id in (select id from companies where onboarding_status = 'pending' and id <> all(cast(:skip as text[]))
	724	+ order by importance desc, created_at limit :n for update skip locked)
	725	+ returning *""", n=limit - len(targets), skip=[c["id"] for c, _ in targets] or ["-"])
	726	+ targets.extend((r, None) for r in rows)
	727	+ stats["claimed"] = len(targets)
	728	+ sem = asyncio.Semaphore(max(1, concurrency))
	729	+
	730	+ async def one(comp: dict[str, Any], job: dict[str, Any] \| None) -> None:
	731	+ async with sem:
	732	+ res = await discover_company(comp, fetcher=fetcher, dry_run=dry_run, fetch_now=fetch_now)
	733	+ stats[str(res.status)] = stats.get(str(res.status), 0) + 1
	734	+ stats["sensors"] += len(res.sensors)
	735	+ if job is not None:
	736	+ await finish_discover_job(job["id"], ok=res.status == OnboardingStatus.ACTIVE, error=res.error, attempts=job["attempts"], max_attempts=job["max_attempts"])
	737	+ log.info("company discovered", extra={"company": comp.get("slug"), "status": str(res.status), "sensors": len(res.sensors), "requests": res.requests,
	738	+ "ms": res.duration_ms, "error": res.error})
	739	+ if fetch_now and not dry_run and res.sensors:
	740	+ from companyatlas.services.pipeline import run_sensor_ids
	741	+
	742	+ await run_sensor_ids([s["id"] for s in res.sensors], fetcher=fetcher, worker=worker)
	743	+
	744	+ await asyncio.gather(*(one(c, j) for c, j in targets))
	745	+ finally:
	746	+ if own_fetcher:
	747	+ await fetcher.close()
	748	+ return stats
	749	+
	750	+
	751	+__all__ = ["DISCOVERY_VERSION", "Candidate", "DiscoveryResult", "claim_discover_jobs", "discover_company", "onboard_pending", "persist", "select_sensors"]

added src/companyatlas/services/pipeline.py +859 −0

		`@@ -0,0 +1,859 @@`
	1	+"""The sensor pipeline (spec §17–20, §26, §60–64): one run of one sensor.
	2	+
	3	+ run_sensor(sensor_row, fetcher=…, worker=…) -> RunOutcome
	4	+
	5	+ fetch (conditional) ──▶ NotModified ─▶ observation(not_modified) + interval growth
	6	+ ──▶ failure ─▶ observation(failure_class) + failures row + policy backoff + status transitions (+ review)
	7	+ ──▶ success ─▶ archive raw ─▶ connector.extract ─▶ hashes
	8	+ unchanged ─▶ observation(changed=false) + interval growth
	9	+ changed ─▶ snapshot (version, objects, extracted) ─▶ entity reconciliation ─▶ structured_delta
	10	+ ─▶ block diff vs previous snapshot ─▶ changes row (significance, kind, status)
	11	+ ─▶ sensor counters / validators / adaptive interval / quality ─▶ company + ledgers
	12	+
	13	+Everything after the network happens in ONE transaction per sensor. Raw bytes, normalized text and blocks live in the object store
	14	+(content-addressed); rows only reference them. Nothing is ever overwritten: new snapshot versions, status columns, removed_at.
	15	+"""
	16	+from __future__ import annotations
	17	+
	18	+import asyncio
	19	+import json
	20	+import logging
	21	+import random
	22	+import time
	23	+from dataclasses import dataclass, field
	24	+from datetime import UTC, datetime, timedelta
	25	+from typing import Any
	26	+
	27	+from companyatlas import archive
	28	+from companyatlas.config import settings
	29	+from companyatlas.connectors._util import is_engineering, job_fingerprint, norm_name
	30	+from companyatlas.db import execute, fetch_all, fetch_one, jsonb, transaction
	31	+from companyatlas.fetch import BlockedError, Fetcher, FetchError, FetchResult, NotModified, classify_exception, file_result
	32	+from companyatlas.ids import new_id
	33	+from companyatlas.sdk import connector as connectors
	34	+from companyatlas.sdk.diff import DIFF_VERSION, compare
	35	+from companyatlas.sdk.models import Block, BlockDiff, Extraction, StructuredDelta
	36	+from companyatlas.sdk.normalize import structural_hash, text_hash
	37	+from companyatlas.taxonomy import (
	38	+ AI_KEYWORDS,
	39	+ FAILURE_POLICY,
	40	+ SURFACE_IMPORTANCE,
	41	+ ChangeKind,
	42	+ FailureClass,
	43	+ SensorStatus,
	44	+ Surface,
	45	+ change_kind,
	46	+ tier_for_interval,
	47	+)
	48	+from companyatlas.urls import canonicalize_url, registrable_domain
	49	+
	50	+log = logging.getLogger(__name__)
	51	+
	52	+PIPELINE_VERSION = "pipeline-v1"
	53	+DELTA_LIST_LIMIT = 200
	54	+EXTRACTED_LIST_LIMIT = 300
	55	+EXTRACTED_MAX_BYTES = 900_000
	56	+JITTER = 0.1 # ± on next_run_at
	57	+QUALITY_ALPHA = 0.15 # EMA weight for quality_score updates
	58	+HTML_SUSPICIOUS_DROP = 0.7 # HTML listings losing > 70 % of ≥ 10 entities are not trusted for removals
	59	+MIN_PREVIOUS_FOR_DROP_GUARD = 10
	60	+FETCH_TIMEOUT_FACTOR = 6 # connector.fetch (incl. pagination) may take this × http timeout
	61	+
	62	+
	63	+@dataclass(slots=True)
	64	+class RunOutcome:
	65	+ sensor_id: str
	66	+ status: str # ok \| not_modified \| unchanged \| changed \| failed \| skipped \| redirected
	67	+ observation_id: str \| None = None
	68	+ snapshot_id: str \| None = None
	69	+ change_id: str \| None = None
	70	+ failure_class: str \| None = None
	71	+ error: str \| None = None
	72	+ significance: float \| None = None
	73	+ kind: str \| None = None
	74	+ duration_ms: int = 0
	75	+ next_run_at: datetime \| None = None
	76	+ interval_s: int \| None = None
	77	+ delta_counts: dict[str, int] = field(default_factory=dict)
	78	+ sensor_status: str \| None = None
	79	+
	80	+ @property
	81	+ def ok(self) -> bool:
	82	+ return self.status in ("ok", "not_modified", "unchanged", "changed")
	83	+
	84	+
	85	+# ------------------------------------------------------------------------------------------------------------ scheduling maths
	86	+
	87	+
	88	+def _clamp(v: float) -> int:
	89	+ return int(max(settings.min_interval_s, min(settings.max_interval_s, v)))
	90	+
	91	+
	92	+def next_interval_unchanged(current: int, base: int) -> int:
	93	+ """Burst decay back to base, then slow growth towards the max (spec §16)."""
	94	+ if current < base:
	95	+ return _clamp(min(base, current * settings.burst_decay))
	96	+ return _clamp(current * settings.stability_growth)
	97	+
	98	+
	99	+def next_interval_changed(kind: ChangeKind, current: int, base: int) -> int:
	100	+ if kind in (ChangeKind.MEANINGFUL, ChangeKind.MAJOR, ChangeKind.CRITICAL):
	101	+ return _clamp(settings.burst_interval_s)
	102	+ if kind == ChangeKind.MINOR:
	103	+ return _clamp(min(current, base))
	104	+ return next_interval_unchanged(current, base)
	105	+
	106	+
	107	+def next_interval_failed(failure: str, current: int) -> int:
	108	+ mult = FAILURE_POLICY.get(failure, FAILURE_POLICY[FailureClass.UNKNOWN])[0]
	109	+ return _clamp(current * mult)
	110	+
	111	+
	112	+def _next_run(now: datetime, interval: int) -> datetime:
	113	+ return now + timedelta(seconds=interval * random.uniform(1 - JITTER, 1 + JITTER))
	114	+
	115	+
	116	+def _quality_update(current: float, target: float) -> float:
	117	+ return round(max(0.0, min(100.0, current * (1 - QUALITY_ALPHA) + target * QUALITY_ALPHA)), 2)
	118	+
	119	+
	120	+def is_ai_title(*parts: str \| None) -> bool:
	121	+ text = " " + " ".join(p.lower() for p in parts if p) + " "
	122	+ return any(k in text for k in AI_KEYWORDS)
	123	+
	124	+
	125	+# ------------------------------------------------------------------------------------------------------------ helpers
	126	+
	127	+
	128	+def _structured_hash(ex: Extraction) -> str:
	129	+ """Hash of the typed payload only (jobs/people/products/plans/locations/news), order-insensitive."""
	130	+ import hashlib
	131	+
	132	+ parts: list[str] = []
	133	+ for j in ex.jobs:
	134	+ parts.append("job\|" + job_fingerprint(j.title, j.location_text, j.external_id, j.url))
	135	+ for p in ex.people:
	136	+ parts.append(f"person\|{norm_name(p.name)}\|{(p.title or '').lower()}")
	137	+ for pr in ex.products:
	138	+ parts.append(f"product\|{norm_name(pr.name)}")
	139	+ for pl in ex.plans:
	140	+ parts.append(f"plan\|{norm_name(pl.plan_name)}\|{pl.price}\|{pl.currency}\|{pl.billing_period}\|{pl.unit}\|{pl.contact_sales}\|{'/'.join(pl.features[:25])}")
	141	+ for loc in ex.locations:
	142	+ parts.append(f"loc\|{norm_name(loc.name)}\|{loc.city}\|{loc.country}")
	143	+ for n in ex.news:
	144	+ parts.append(f"news\|{canonicalize_url(n.url)}")
	145	+ if not parts:
	146	+ return ""
	147	+ return hashlib.sha256("\n".join(sorted(parts)).encode("utf-8")).hexdigest()
	148	+
	149	+
	150	+def _bounded_extracted(ex: Extraction) -> dict[str, Any]:
	151	+ payload = ex.structured_payload()
	152	+ for k in ("jobs", "people", "products", "plans", "locations", "news"):
	153	+ lst = payload.get(k) or []
	154	+ if len(lst) > EXTRACTED_LIST_LIMIT:
	155	+ payload[k] = lst[:EXTRACTED_LIST_LIMIT]
	156	+ payload.setdefault("truncated", {})[k] = len(lst)
	157	+ meta = dict(payload.get("meta") or {})
	158	+ if isinstance(meta.get("urls"), list) and len(meta["urls"]) > 500:
	159	+ meta["urls"] = meta["urls"][:500]
	160	+ meta["urls_truncated"] = True
	161	+ payload["meta"] = meta
	162	+ raw = jsonb(payload)
	163	+ if len(raw) > EXTRACTED_MAX_BYTES:
	164	+ for k in ("jobs", "people", "products", "plans", "locations", "news"):
	165	+ payload[k] = (payload.get(k) or [])[:50]
	166	+ meta.pop("urls", None)
	167	+ payload["meta"] = meta
	168	+ payload["truncated"] = {**payload.get("truncated", {}), "reason": "size"}
	169	+ return payload
	170	+
	171	+
	172	+def _blocks_json(blocks: list[Block]) -> str:
	173	+ return json.dumps([b.to_json() for b in blocks], ensure_ascii=False, default=str)
	174	+
	175	+
	176	+def _blocks_from_json(raw: str) -> list[Block]:
	177	+ out: list[Block] = []
	178	+ try:
	179	+ data = json.loads(raw)
	180	+ except json.JSONDecodeError:
	181	+ return out
	182	+ for d in data if isinstance(data, list) else []:
	183	+ try:
	184	+ out.append(Block(key=d["key"], kind=d.get("kind", "other"), text=d.get("text", ""), path=d.get("path", ""), hash=d.get("hash", ""),
	185	+ simhash=int(d.get("simhash") or 0), weight=float(d.get("weight") or 1.0), order=int(d.get("order") or 0), attrs=d.get("attrs") or {}))
	186	+ except (KeyError, TypeError, ValueError):
	187	+ continue
	188	+ return out
	189	+
	190	+
	191	+def _guess_content_type(path: str) -> str:
	192	+ p = path.lower()
	193	+ if p.endswith(".json"):
	194	+ return "application/json; charset=utf-8"
	195	+ if p.endswith((".xml", ".rss", ".atom")):
	196	+ return "application/xml; charset=utf-8"
	197	+ return "text/html; charset=utf-8"
	198	+
	199	+
	200	+# ------------------------------------------------------------------------------------------------------------ domain budgets
	201	+
	202	+
	203	+async def _check_domain_budget(conn: Any, domain: str, now: datetime) -> tuple[bool, datetime \| None, str \| None]:
	204	+ """(allowed, resume_at, reason). Creates the row on first sight; resets `used_today` on a new day."""
	205	+ row = await fetch_one(conn, """
	206	+ insert into domain_budgets (domain, max_concurrency, requests_per_minute, daily_budget)
	207	+ values (:d, :mc, :rpm, :daily)
	208	+ on conflict (domain) do update set used_today = case when domain_budgets.budget_day < current_date then 0 else domain_budgets.used_today end,
	209	+ budget_day = greatest(domain_budgets.budget_day, current_date), updated_at = now()
	210	+ returning daily_budget, used_today, blocked_until, block_reason
	211	+ """, d=domain, mc=settings.domain_max_concurrency, rpm=settings.default_rate_per_min, daily=settings.domain_daily_budget)
	212	+ if row is None:
	213	+ return True, None, None
	214	+ if row["blocked_until"] and row["blocked_until"] > now:
	215	+ return False, row["blocked_until"], row.get("block_reason") or "domain blocked"
	216	+ if row["used_today"] >= row["daily_budget"]:
	217	+ tomorrow = datetime.combine(datetime.now(UTC).date() + timedelta(days=1), datetime.min.time(), tzinfo=UTC)
	218	+ return False, tomorrow, "daily budget exhausted"
	219	+ return True, None, None
	220	+
	221	+
	222	+async def _consume_domain_budget(conn: Any, domain: str, units: int) -> None:
	223	+ await execute(conn, "update domain_budgets set used_today = used_today + :u, updated_at = now() where domain = :d", u=units, d=domain)
	224	+
	225	+
	226	+async def _ledger(conn: Any, company_id: str, connector_id: str, units: int, size_bytes: int) -> None:
	227	+ await execute(conn, """
	228	+ insert into cost_ledger (day, dimension, key, units) values (current_date, 'fetch', :company, :u)
	229	+ on conflict (day, dimension, key) do update set units = cost_ledger.units + excluded.units
	230	+ """, company=company_id, u=units)
	231	+ await execute(conn, """
	232	+ insert into cost_ledger (day, dimension, key, units) values (current_date, 'fetch', :conn, :u)
	233	+ on conflict (day, dimension, key) do update set units = cost_ledger.units + excluded.units
	234	+ """, conn=f"connector:{connector_id}", u=units)
	235	+ if size_bytes:
	236	+ await execute(conn, """
	237	+ insert into cost_ledger (day, dimension, key, units) values (current_date, 'storage_gb', '', :gb)
	238	+ on conflict (day, dimension, key) do update set units = cost_ledger.units + excluded.units
	239	+ """, gb=size_bytes / 1e9)
	240	+
	241	+
	242	+# ------------------------------------------------------------------------------------------------------------ entity reconciliation
	243	+
	244	+
	245	+def _trustworthy(ex: Extraction, count: int) -> bool:
	246	+ return count >= 1 or bool(ex.meta.get("structured"))
	247	+
	248	+
	249	+def _suspicious_drop(ex: Extraction, previous: int, current: int) -> bool:
	250	+ if ex.meta.get("structured"):
	251	+ return False
	252	+ return previous >= MIN_PREVIOUS_FOR_DROP_GUARD and current < previous * (1 - HTML_SUSPICIOUS_DROP)
	253	+
	254	+
	255	+async def reconcile_jobs(conn: Any, *, company_id: str, sensor_id: str, ex: Extraction, now: datetime, delta: StructuredDelta) -> None:
	256	+ existing = await fetch_all(conn, "select id, fingerprint, status, title from jobs where company_id = :c and sensor_id = :s", c=company_id, s=sensor_id)
	257	+ by_fp = {r["fingerprint"]: r for r in existing}
	258	+ open_before = sum(1 for r in existing if r["status"] == "open")
	259	+ seen: set[str] = set()
	260	+ added: list[dict[str, Any]] = []
	261	+ for j in ex.jobs:
	262	+ fp = job_fingerprint(j.title, j.location_text, j.external_id, j.url)
	263	+ if fp in seen:
	264	+ continue
	265	+ seen.add(fp)
	266	+ ai = is_ai_title(j.title, j.department, j.team)
	267	+ row = by_fp.get(fp)
	268	+ params = {"c": company_id, "s": sensor_id, "fp": fp, "title": j.title[:300], "dept": j.department, "team": j.team, "loc": j.location_text,
	269	+ "city": j.city, "region": j.region, "country": (j.country or None), "remote": j.remote, "et": j.employment_type, "sen": j.seniority,
	270	+ "skills": list(j.skills or [])[:30], "smin": j.salary_min, "smax": j.salary_max, "scur": j.salary_currency, "sper": j.salary_period,
	271	+ "url": j.url, "dh": j.description_hash, "posted": j.posted_at, "ai": ai, "eng": is_engineering(j.title), "raw": jsonb(j.raw or {}),
	272	+ "ext": j.external_id, "now": now}
	273	+ if row is None:
	274	+ await execute(conn, """
	275	+ insert into jobs (id, company_id, sensor_id, external_id, fingerprint, title, department, team, location_text, city, region, country, remote,
	276	+ employment_type, seniority, skills, salary_min, salary_max, salary_currency, salary_period, url, description_hash, posted_at,
	277	+ first_seen_at, last_seen_at, status, is_ai, is_engineering, raw)
	278	+ values (:id, :c, :s, :ext, :fp, :title, :dept, :team, :loc, :city, :region, :country, :remote, :et, :sen, cast(:skills as text[]), :smin, :smax,
	279	+ :scur, :sper, :url, :dh, :posted, :now, :now, 'open', :ai, :eng, cast(:raw as jsonb))
	280	+ on conflict (company_id, fingerprint) do update set sensor_id = excluded.sensor_id, last_seen_at = excluded.last_seen_at, status = 'open',
	281	+ removed_at = null, url = coalesce(excluded.url, jobs.url), is_ai = excluded.is_ai
	282	+ """, id=new_id("job"), **params)
	283	+ added.append({"title": j.title, "url": j.url, "location_text": j.location_text, "country": j.country, "remote": j.remote, "department": j.department, "is_ai": ai})
	284	+ else:
	285	+ await execute(conn, """
	286	+ update jobs set last_seen_at = :now, status = 'open', removed_at = null, title = :title, location_text = coalesce(:loc, location_text),
	287	+ city = coalesce(:city, city), region = coalesce(:region, region), country = coalesce(:country, country), remote = coalesce(:remote, remote),
	288	+ url = coalesce(:url, url), is_ai = :ai, salary_min = coalesce(:smin, salary_min), salary_max = coalesce(:smax, salary_max),
	289	+ posted_at = coalesce(posted_at, :posted)
	290	+ where id = :id
	291	+ """, id=row["id"], **params)
	292	+ if row["status"] != "open":
	293	+ added.append({"title": j.title, "url": j.url, "location_text": j.location_text, "country": j.country, "remote": j.remote, "department": j.department,
	294	+ "is_ai": ai, "relisted": True})
	295	+ removed: list[dict[str, Any]] = []
	296	+ missing = [r for r in existing if r["status"] == "open" and r["fingerprint"] not in seen]
	297	+ if missing and _trustworthy(ex, len(seen)) and not _suspicious_drop(ex, open_before, len(seen)):
	298	+ rows = await fetch_all(conn, """
	299	+ update jobs set status = 'no_longer_listed', removed_at = :now, last_seen_at = last_seen_at
	300	+ where id = any(cast(:ids as text[])) returning title, url, location_text, country, remote, department, is_ai
	301	+ """, ids=[r["id"] for r in missing], now=now)
	302	+ removed = [dict(r) for r in rows]
	303	+ elif missing:
	304	+ delta.setdefault("notes", []).append(f"jobs: {len(missing)} missing not marked removed (untrusted extraction)")
	305	+ if added or removed or open_before != len(seen):
	306	+ delta["jobs"] = {"added": added[:DELTA_LIST_LIMIT], "removed": removed[:DELTA_LIST_LIMIT], "open_before": open_before, "open_after": len(seen),
	307	+ "ai_added": sum(1 for a in added if a.get("is_ai")), "ai_removed": sum(1 for r in removed if r.get("is_ai"))}
	308	+
	309	+
	310	+async def reconcile_named(conn: Any, *, table: str, company_id: str, sensor_id: str, sensor_url: str, items: list[Any], ex: Extraction, now: datetime,
	311	+ delta: StructuredDelta, key: str) -> None:
	312	+ """people / products / locations share the same shape: unique (company_id, name_norm), first/last_seen, removed_at, status."""
	313	+ listed_status = "listed"
	314	+ existing = await fetch_all(conn, f"select id, name_norm, status, title from {table} where company_id = :c and sensor_id = :s"
	315	+ if table == "people" else f"select id, name_norm, status from {table} where company_id = :c and sensor_id = :s",
	316	+ c=company_id, s=sensor_id)
	317	+ by_norm = {r["name_norm"]: r for r in existing}
	318	+ seen: set[str] = set()
	319	+ added: list[dict[str, Any]] = []
	320	+ title_changed: list[dict[str, Any]] = []
	321	+ for it in items:
	322	+ name = getattr(it, "name", None)
	323	+ if not name:
	324	+ continue
	325	+ nn = norm_name(name)
	326	+ if not nn or nn in seen:
	327	+ continue
	328	+ seen.add(nn)
	329	+ row = by_norm.get(nn)
	330	+ if table == "people":
	331	+ payload = {"name": name, "title": it.title, "role_category": it.role_category, "is_executive": it.is_executive}
	332	+ if row is None:
	333	+ await execute(conn, """
	334	+ insert into people (id, company_id, sensor_id, name, name_norm, title, role_category, is_executive, first_seen_at, last_seen_at, status, source_url)
	335	+ values (:id, :c, :s, :name, :nn, :title, :rc, :ex, :now, :now, 'listed', :url)
	336	+ on conflict (company_id, name_norm) do update set sensor_id = excluded.sensor_id, last_seen_at = excluded.last_seen_at, status = 'listed',
	337	+ removed_at = null, title = coalesce(excluded.title, people.title), role_category = coalesce(excluded.role_category, people.role_category),
	338	+ is_executive = excluded.is_executive
	339	+ """, id=new_id("person"), c=company_id, s=sensor_id, name=name[:200], nn=nn, title=it.title, rc=it.role_category, ex=bool(it.is_executive), now=now,
	340	+ url=it.url or sensor_url)
	341	+ added.append(payload)
	342	+ else:
	343	+ if row["status"] != listed_status:
	344	+ added.append({**payload, "relisted": True})
	345	+ elif it.title and row.get("title") and norm_name(it.title) != norm_name(row["title"]):
	346	+ title_changed.append({"name": name, "before": row["title"], "after": it.title})
	347	+ await execute(conn, """update people set last_seen_at = :now, status = 'listed', removed_at = null, title = coalesce(:title, title),
	348	+ role_category = coalesce(:rc, role_category), is_executive = :ex where id = :id""",
	349	+ id=row["id"], now=now, title=it.title, rc=it.role_category, ex=bool(it.is_executive))
	350	+ elif table == "products":
	351	+ payload = {"name": name, "url": it.url}
	352	+ if row is None:
	353	+ await execute(conn, """
	354	+ insert into products (id, company_id, sensor_id, name, name_norm, category, description, url, first_seen_at, last_seen_at, status)
	355	+ values (:id, :c, :s, :name, :nn, :cat, :desc, :url, :now, :now, 'listed')
	356	+ on conflict (company_id, name_norm) do update set sensor_id = excluded.sensor_id, last_seen_at = excluded.last_seen_at, status = 'listed',
	357	+ removed_at = null, url = coalesce(excluded.url, products.url), description = coalesce(excluded.description, products.description)
	358	+ """, id=new_id("product"), c=company_id, s=sensor_id, name=name[:200], nn=nn, cat=it.category, desc=(it.description or None), url=it.url, now=now)
	359	+ added.append(payload)
	360	+ else:
	361	+ if row["status"] != listed_status:
	362	+ added.append({**payload, "relisted": True})
	363	+ await execute(conn, "update products set last_seen_at = :now, status = 'listed', removed_at = null, url = coalesce(:url, url) where id = :id",
	364	+ id=row["id"], now=now, url=it.url)
	365	+ else: # locations
	366	+ payload = {"name": name, "city": it.city, "country": it.country, "kind": it.kind}
	367	+ if row is None:
	368	+ await execute(conn, """
	369	+ insert into locations (id, company_id, sensor_id, kind, name, name_norm, city, region, country, first_seen_at, last_seen_at, status, source_url)
	370	+ values (:id, :c, :s, :kind, :name, :nn, :city, :region, :country, :now, :now, 'listed', :url)
	371	+ on conflict (company_id, name_norm) do update set sensor_id = excluded.sensor_id, last_seen_at = excluded.last_seen_at, status = 'listed',
	372	+ removed_at = null, city = coalesce(excluded.city, locations.city), country = coalesce(excluded.country, locations.country)
	373	+ """, id=new_id("location"), c=company_id, s=sensor_id, kind=it.kind or "office", name=name[:200], nn=nn, city=it.city, region=it.region,
	374	+ country=(it.country or None), now=now, url=sensor_url)
	375	+ added.append(payload)
	376	+ else:
	377	+ if row["status"] != listed_status:
	378	+ added.append({**payload, "relisted": True})
	379	+ await execute(conn, "update locations set last_seen_at = :now, status = 'listed', removed_at = null where id = :id", id=row["id"], now=now)
	380	+ removed: list[dict[str, Any]] = []
	381	+ missing = [r for r in existing if r["status"] == listed_status and r["name_norm"] not in seen]
	382	+ if missing and _trustworthy(ex, len(seen)) and not _suspicious_drop(ex, len(existing), len(seen)):
	383	+ cols = "name, title, role_category, is_executive" if table == "people" else ("name, url" if table == "products" else "name, city, country, kind")
	384	+ rows = await fetch_all(conn, f"update {table} set status = 'no_longer_listed', removed_at = :now where id = any(cast(:ids as text[])) returning {cols}",
	385	+ ids=[r["id"] for r in missing], now=now)
	386	+ removed = [dict(r) for r in rows]
	387	+ elif missing:
	388	+ delta.setdefault("notes", []).append(f"{key}: {len(missing)} missing not marked removed (untrusted extraction)")
	389	+ if added or removed or title_changed:
	390	+ entry: dict[str, Any] = {"added": added[:DELTA_LIST_LIMIT], "removed": removed[:DELTA_LIST_LIMIT]}
	391	+ if table == "people" and title_changed:
	392	+ entry["title_changed"] = title_changed[:DELTA_LIST_LIMIT]
	393	+ if table == "locations":
	394	+ prior = await fetch_all(conn, "select distinct country from locations where company_id = :c and country is not null and status = 'listed' and first_seen_at < :now",
	395	+ c=company_id, now=now)
	396	+ known = {r["country"] for r in prior}
	397	+ new_countries = sorted({a["country"] for a in added if a.get("country") and a["country"] not in known})
	398	+ if new_countries:
	399	+ entry["new_countries"] = new_countries
	400	+ delta[key] = entry
	401	+
	402	+
	403	+async def reconcile_plans(conn: Any, *, company_id: str, sensor_id: str, sensor_url: str, ex: Extraction, now: datetime, delta: StructuredDelta) -> None:
	404	+ current = await fetch_all(conn, """select id, plan_norm, plan_name, currency, billing_period, price, unit, features, contact_sales, version_no
	405	+ from pricing_plans where company_id = :c and sensor_id = :s and status = 'current'""", c=company_id, s=sensor_id)
	406	+ by_norm = {r["plan_norm"]: r for r in current}
	407	+ seen: set[str] = set()
	408	+ added: list[dict[str, Any]] = []
	409	+ price_changed: list[dict[str, Any]] = []
	410	+ for pl in ex.plans:
	411	+ nn = norm_name(pl.plan_name)
	412	+ if not nn or nn in seen:
	413	+ continue
	414	+ seen.add(nn)
	415	+ row = by_norm.get(nn)
	416	+ feats = list(pl.features or [])[:40]
	417	+ same = row is not None and (row["price"] is None) == (pl.price is None) and (row["price"] is None or abs(float(row["price"]) - float(pl.price)) < 1e-9) \
	418	+ and (row["currency"] or None) == (pl.currency or None) and (row["billing_period"] or None) == (pl.billing_period or None) \
	419	+ and bool(row["contact_sales"]) == bool(pl.contact_sales) and list(row["features"] or []) == feats
	420	+ if same:
	421	+ await execute(conn, "update pricing_plans set last_seen_at = :now where id = :id", id=row["id"], now=now)
	422	+ continue
	423	+ version = 1
	424	+ if row is not None:
	425	+ version = int(row["version_no"]) + 1
	426	+ await execute(conn, "update pricing_plans set status = 'superseded', valid_to = :now where id = :id", id=row["id"], now=now)
	427	+ if (row["price"] is None) != (pl.price is None) or (row["price"] is not None and abs(float(row["price"]) - float(pl.price)) >= 1e-9):
	428	+ before = float(row["price"]) if row["price"] is not None else None
	429	+ pct = round((pl.price - before) / before * 100, 2) if (before and pl.price is not None) else None
	430	+ price_changed.append({"plan_name": pl.plan_name, "before": before, "after": pl.price, "currency": pl.currency or row["currency"],
	431	+ "billing_period": pl.billing_period or row["billing_period"], "pct": pct})
	432	+ else:
	433	+ added.append({"plan_name": pl.plan_name, "price": pl.price, "currency": pl.currency, "billing_period": pl.billing_period})
	434	+ await execute(conn, """
	435	+ insert into pricing_plans (id, company_id, sensor_id, plan_name, plan_norm, currency, billing_period, price, price_text, unit, features, contact_sales,
	436	+ version_no, valid_from, first_seen_at, last_seen_at, status, source_url)
	437	+ values (:id, :c, :s, :name, :nn, :cur, :bp, :price, :pt, :unit, cast(:feats as jsonb), :cs, :v, :now, :now, :now, 'current', :url)
	438	+ """, id=new_id("plan"), c=company_id, s=sensor_id, name=pl.plan_name[:200], nn=nn, cur=pl.currency, bp=pl.billing_period, price=pl.price, pt=pl.price_text,
	439	+ unit=pl.unit, feats=jsonb(feats), cs=bool(pl.contact_sales), v=version, now=now, url=sensor_url)
	440	+ removed: list[dict[str, Any]] = []
	441	+ missing = [r for r in current if r["plan_norm"] not in seen]
	442	+ if missing and _trustworthy(ex, len(seen)):
	443	+ rows = await fetch_all(conn, """update pricing_plans set status = 'removed', valid_to = :now where id = any(cast(:ids as text[]))
	444	+ returning plan_name, price, currency, billing_period""", ids=[r["id"] for r in missing], now=now)
	445	+ removed = [dict(r) for r in rows]
	446	+ if added or removed or price_changed:
	447	+ delta["plans"] = {"added": added, "removed": removed, "price_changed": price_changed}
	448	+
	449	+
	450	+async def reconcile_news(conn: Any, *, company_id: str, sensor_id: str, ex: Extraction, now: datetime, delta: StructuredDelta) -> None:
	451	+ added: list[dict[str, Any]] = []
	452	+ seen: set[str] = set()
	453	+ for n in ex.news[:500]:
	454	+ canon = canonicalize_url(n.url)
	455	+ if canon in seen:
	456	+ continue
	457	+ seen.add(canon)
	458	+ row = await fetch_one(conn, """
	459	+ insert into news_items (id, company_id, sensor_id, url, canonical_url, title, summary, category, published_at, first_seen_at, language)
	460	+ values (:id, :c, :s, :url, :canon, :title, :summary, :cat, :pub, :now, :lang)
	461	+ on conflict (company_id, canonical_url) do nothing returning id
	462	+ """, id=new_id("news"), c=company_id, s=sensor_id, url=n.url[:2000], canon=canon[:2000], title=n.title[:300], summary=(n.summary or None), cat=n.category,
	463	+ pub=n.published_at, now=now, lang=n.language)
	464	+ if row is not None:
	465	+ added.append({"title": n.title, "url": n.url, "published_at": n.published_at.isoformat() if n.published_at else None, "category": n.category})
	466	+ if added:
	467	+ delta["news"] = {"added": added[:DELTA_LIST_LIMIT], "count": len(added)}
	468	+
	469	+
	470	+async def reconcile(conn: Any, *, company_id: str, sensor: dict[str, Any], ex: Extraction, now: datetime, previous_meta: dict[str, Any] \| None) -> StructuredDelta:
	471	+ delta: StructuredDelta = {}
	472	+ sid, url = str(sensor["id"]), str(sensor["url"])
	473	+ surface = str(sensor.get("surface") or "")
	474	+ if ex.jobs or surface in (Surface.JOBS_BOARD, Surface.CAREERS):
	475	+ await reconcile_jobs(conn, company_id=company_id, sensor_id=sid, ex=ex, now=now, delta=delta)
	476	+ if ex.people or surface == Surface.LEADERSHIP:
	477	+ await reconcile_named(conn, table="people", company_id=company_id, sensor_id=sid, sensor_url=url, items=ex.people, ex=ex, now=now, delta=delta, key="people")
	478	+ if ex.products or surface in (Surface.PRODUCTS, Surface.SERVICES, Surface.SOLUTIONS):
	479	+ await reconcile_named(conn, table="products", company_id=company_id, sensor_id=sid, sensor_url=url, items=ex.products, ex=ex, now=now, delta=delta, key="products")
	480	+ if ex.locations or surface == Surface.LOCATIONS:
	481	+ await reconcile_named(conn, table="locations", company_id=company_id, sensor_id=sid, sensor_url=url, items=ex.locations, ex=ex, now=now, delta=delta, key="locations")
	482	+ if ex.plans or surface == Surface.PRICING:
	483	+ await reconcile_plans(conn, company_id=company_id, sensor_id=sid, sensor_url=url, ex=ex, now=now, delta=delta)
	484	+ if ex.news:
	485	+ await reconcile_news(conn, company_id=company_id, sensor_id=sid, ex=ex, now=now, delta=delta)
	486	+ meta: dict[str, Any] = {"language": ex.language}
	487	+ if previous_meta:
	488	+ if (previous_meta.get("title") or None) != (ex.title or None) and (previous_meta.get("title") or ex.title):
	489	+ meta["title_changed"] = {"before": previous_meta.get("title"), "after": ex.title}
	490	+ if (previous_meta.get("description") or None) != (ex.meta.get("description") or None):
	491	+ meta["description_changed"] = True
	492	+ delta["meta"] = meta
	493	+ return delta
	494	+
	495	+
	496	+def _delta_counts(delta: StructuredDelta) -> dict[str, int]:
	497	+ out: dict[str, int] = {}
	498	+ for k, v in delta.items():
	499	+ if isinstance(v, dict):
	500	+ for kk in ("added", "removed", "price_changed", "title_changed", "new_countries"):
	501	+ if isinstance(v.get(kk), list) and v[kk]:
	502	+ out[f"{k}_{kk}"] = len(v[kk])
	503	+ return out
	504	+
	505	+
	506	+# ------------------------------------------------------------------------------------------------------------ the run
	507	+
	508	+
	509	+async def run_sensor(sensor: dict[str, Any], *, fetcher: Fetcher, worker: str = "local", force: bool = False, result: FetchResult \| None = None,
	510	+ collection_method: str = "live") -> RunOutcome:
	511	+ """One complete run. Never raises for fetch/extract problems (they become observations); DB errors propagate."""
	512	+ t0 = time.perf_counter()
	513	+ sid = str(sensor["id"])
	514	+ now = datetime.now(UTC)
	515	+ outcome = RunOutcome(sensor_id=sid, status="failed")
	516	+ domain = str(sensor.get("domain") or registrable_domain(str(sensor["url"])))
	517	+ company_id = str(sensor["company_id"])
	518	+ cfg = dict(sensor.get("config") or {})
	519	+ connector = connectors.get(str(sensor["connector_id"]))
	520	+
	521	+ # ---- admission: domain budget / block (own short transaction)
	522	+ if result is None:
	523	+ async with transaction() as conn:
	524	+ allowed, resume_at, reason = await _check_domain_budget(conn, domain, now)
	525	+ if not allowed:
	526	+ resume = (resume_at or now + timedelta(hours=1)) + timedelta(seconds=random.uniform(60, 900))
	527	+ await execute(conn, "update sensors set next_run_at = :n, claimed_by = null, claimed_at = null, updated_at = now() where id = :id", n=resume, id=sid)
	528	+ outcome.status, outcome.error, outcome.next_run_at = "skipped", reason, resume
	529	+ outcome.duration_ms = int((time.perf_counter() - t0) * 1000)
	530	+ return outcome
	531	+ company = await fetch_one(conn, "select id, canonical_domain, tier, importance, slug from companies where id = :id", id=company_id)
	532	+ else:
	533	+ async with transaction() as conn:
	534	+ company = await fetch_one(conn, "select id, canonical_domain, tier, importance, slug from companies where id = :id", id=company_id)
	535	+ ctx = connectors.ConnectorContext(company=company or {"id": company_id})
	536	+
	537	+ # ---- network (outside any transaction)
	538	+ fetched: FetchResult \| None = result
	539	+ failure: tuple[str, str, int \| None] \| None = None
	540	+ not_modified = False
	541	+ fetch_ms = 0
	542	+ if fetched is None:
	543	+ req_sensor = {**sensor, "etag": None, "last_modified": None} if force else sensor
	544	+ tf = time.perf_counter()
	545	+ try:
	546	+ fetched = await asyncio.wait_for(connector.fetch(ctx, req_sensor, fetcher), settings.http_timeout_s * FETCH_TIMEOUT_FACTOR)
	547	+ except NotModified as nm:
	548	+ not_modified = True
	549	+ fetch_ms = nm.duration_ms
	550	+ except (FetchError, BlockedError) as exc:
	551	+ failure = (str(exc.failure), str(exc)[:500], exc.status)
	552	+ except TimeoutError:
	553	+ failure = (str(FailureClass.TIMEOUT), "connector fetch timed out", None)
	554	+ except Exception as exc: # noqa: BLE001
	555	+ failure = (str(classify_exception(exc)), f"{exc.__class__.__name__}: {exc}"[:500], None)
	556	+ fetch_ms = fetch_ms or int((time.perf_counter() - tf) * 1000)
	557	+
	558	+ # ---- redirect to another registrable domain (keeps the observation, parks the sensor)
	559	+ if fetched is not None and result is None and registrable_domain(fetched.final_url) != registrable_domain(str(sensor["url"])):
	560	+ failure = (str(FailureClass.REDIRECT), f"redirected off-domain to {fetched.final_url}", fetched.status)
	561	+ cfg["redirect_url"] = fetched.final_url
	562	+
	563	+ async with transaction() as conn:
	564	+ base_interval = int(sensor.get("base_interval_s") or 86400)
	565	+ current = int(sensor.get("current_interval_s") or base_interval)
	566	+ quality = float(sensor.get("quality_score") or 50.0)
	567	+ obs_id = new_id("observation")
	568	+ outcome.observation_id = obs_id
	569	+
	570	+ if not_modified:
	571	+ interval = next_interval_unchanged(current, base_interval)
	572	+ nxt = _next_run(now, interval)
	573	+ await execute(conn, """
	574	+ insert into observations (id, sensor_id, company_id, fetched_at, status_code, duration_ms, transport, final_url, not_modified, changed,
	575	+ connector_version, collection_method, worker)
	576	+ values (:id, :sid, :cid, :now, 304, :ms, 'http', :url, true, false, :cv, :cm, :worker)
	577	+ """, id=obs_id, sid=sid, cid=company_id, now=now, ms=fetch_ms, url=sensor["url"], cv=connector.connector_id, cm=collection_method, worker=worker)
	578	+ quality = _quality_update(quality, 100.0)
	579	+ await execute(conn, """
	580	+ update sensors set status = 'active', last_run_at = :now, last_success_at = :now, last_status = 304, last_failure_class = null, last_error = null,
	581	+ consecutive_failures = 0, consecutive_unchanged = consecutive_unchanged + 1, observation_count = observation_count + 1,
	582	+ current_interval_s = :iv, tier = :tier, next_run_at = :nxt, quality_score = :q, claimed_by = null, claimed_at = null, updated_at = now()
	583	+ where id = :id
	584	+ """, now=now, iv=interval, tier=tier_for_interval(interval), nxt=nxt, q=quality, id=sid)
	585	+ await execute(conn, "update companies set last_observed_at = :now, first_observed_at = coalesce(first_observed_at, :now) where id = :id", now=now, id=company_id)
	586	+ await _consume_domain_budget(conn, domain, 1)
	587	+ await _ledger(conn, company_id, connector.connector_id, 1, 0)
	588	+ outcome.status, outcome.next_run_at, outcome.interval_s, outcome.sensor_status = "not_modified", nxt, interval, "active"
	589	+ outcome.duration_ms = int((time.perf_counter() - t0) * 1000)
	590	+ return outcome
	591	+
	592	+ if failure is not None:
	593	+ await _record_failure(conn, sensor=sensor, cfg=cfg, company_id=company_id, obs_id=obs_id, failure=failure, now=now, fetch_ms=fetch_ms,
	594	+ connector_id=connector.connector_id, worker=worker, current=current, quality=quality, outcome=outcome,
	595	+ collection_method=collection_method, final_url=fetched.final_url if fetched else None)
	596	+ await _consume_domain_budget(conn, domain, 1)
	597	+ await _ledger(conn, company_id, connector.connector_id, 1, 0)
	598	+ outcome.duration_ms = int((time.perf_counter() - t0) * 1000)
	599	+ return outcome
	600	+
	601	+ assert fetched is not None
	602	+ pages = int(fetched.headers.get("x-companyatlas-pages", "1") or 1)
	603	+ object_key, _stored, _created = archive.put_bytes(fetched.content)
	604	+
	605	+ # ---- extract
	606	+ try:
	607	+ ex = connector.extract(sensor, fetched)
	608	+ except Exception as exc: # noqa: BLE001
	609	+ failure = (str(FailureClass.PARSING), f"extract failed: {exc.__class__.__name__}: {exc}"[:500], fetched.status)
	610	+ await _record_failure(conn, sensor=sensor, cfg=cfg, company_id=company_id, obs_id=obs_id, failure=failure, now=now, fetch_ms=fetch_ms,
	611	+ connector_id=connector.connector_id, worker=worker, current=current, quality=quality, outcome=outcome,
	612	+ collection_method=collection_method, final_url=fetched.final_url, object_key=object_key, size=len(fetched.content),
	613	+ content_type=fetched.content_type)
	614	+ await _consume_domain_budget(conn, domain, pages)
	615	+ await _ledger(conn, company_id, connector.connector_id, pages, len(fetched.content))
	616	+ outcome.duration_ms = int((time.perf_counter() - t0) * 1000)
	617	+ return outcome
	618	+
	619	+ ex.normalized_hash = ex.normalized_hash or text_hash(ex.text)
	620	+ ex.structured_hash = ex.structured_hash or _structured_hash(ex)
	621	+ struct_hash = structural_hash(ex.blocks)
	622	+ unchanged = (not force and sensor.get("last_normalized_hash") == ex.normalized_hash and (cfg.get("last_structured_hash") or "") == ex.structured_hash
	623	+ and sensor.get("last_snapshot_id") is not None)
	624	+ etag, last_mod = fetched.etag, fetched.last_modified
	625	+
	626	+ await execute(conn, """
	627	+ insert into observations (id, sensor_id, company_id, fetched_at, status_code, duration_ms, transport, final_url, redirects, not_modified, changed,
	628	+ content_hash, normalized_hash, structural_hash, object_key, size_bytes, content_type, connector_version, collection_method, worker)
	629	+ values (:id, :sid, :cid, :now, :st, :ms, :tr, :url, :redir, false, :changed, :ch, :nh, :sh, :ok, :size, :ct, :cv, :cm, :worker)
	630	+ """, id=obs_id, sid=sid, cid=company_id, now=now, st=fetched.status, ms=fetch_ms, tr=fetched.transport, url=fetched.final_url, redir=fetched.redirects,
	631	+ changed=not unchanged, ch=fetched.sha256, nh=ex.normalized_hash, sh=struct_hash, ok=object_key, size=len(fetched.content),
	632	+ ct=fetched.content_type[:100], cv=connector.connector_id, cm=collection_method, worker=worker)
	633	+ await _consume_domain_budget(conn, domain, pages)
	634	+ await _ledger(conn, company_id, connector.connector_id, pages, len(fetched.content))
	635	+ extraction_conf = 100.0 if (ex.meta.get("structured") or len(ex.text) > 200) else 60.0
	636	+
	637	+ if unchanged:
	638	+ interval = next_interval_unchanged(current, base_interval)
	639	+ nxt = _next_run(now, interval)
	640	+ quality = _quality_update(quality, extraction_conf)
	641	+ await execute(conn, """
	642	+ update sensors set status = 'active', last_run_at = :now, last_success_at = :now, last_status = :st, last_failure_class = null, last_error = null,
	643	+ consecutive_failures = 0, consecutive_unchanged = consecutive_unchanged + 1, observation_count = observation_count + 1,
	644	+ current_interval_s = :iv, tier = :tier, next_run_at = :nxt, quality_score = :q, etag = coalesce(:etag, etag),
	645	+ last_modified = coalesce(:lm, last_modified), last_content_hash = :ch, claimed_by = null, claimed_at = null, updated_at = now()
	646	+ where id = :id
	647	+ """, now=now, st=fetched.status, iv=interval, tier=tier_for_interval(interval), nxt=nxt, q=quality, etag=etag, lm=last_mod, ch=fetched.sha256, id=sid)
	648	+ await execute(conn, "update companies set last_observed_at = :now, first_observed_at = coalesce(first_observed_at, :now) where id = :id", now=now, id=company_id)
	649	+ outcome.status, outcome.next_run_at, outcome.interval_s, outcome.sensor_status = "unchanged", nxt, interval, "active"
	650	+ outcome.duration_ms = int((time.perf_counter() - t0) * 1000)
	651	+ return outcome
	652	+
	653	+ # ---- changed (or first run / forced): snapshot + reconciliation + diff
	654	+ prev_id = sensor.get("last_snapshot_id")
	655	+ prev = await fetch_one(conn, "select id, version_no, blocks_key, text_key, title, extracted from snapshots where id = :id", id=prev_id) if prev_id else None
	656	+ prev_blocks: list[Block] = []
	657	+ prev_text = ""
	658	+ if prev is not None:
	659	+ try:
	660	+ if prev.get("blocks_key") and archive.exists(prev["blocks_key"]):
	661	+ prev_blocks = _blocks_from_json(archive.get_text(prev["blocks_key"]))
	662	+ if prev.get("text_key") and archive.exists(prev["text_key"]):
	663	+ prev_text = archive.get_text(prev["text_key"])
	664	+ except OSError:
	665	+ log.warning("previous snapshot objects unreadable", extra={"sensor_id": sid, "snapshot_id": prev_id})
	666	+ prev_meta = None
	667	+ if prev is not None:
	668	+ pm = (prev.get("extracted") or {}).get("meta") if isinstance(prev.get("extracted"), dict) else {}
	669	+ prev_meta = {"title": prev.get("title"), "description": (pm or {}).get("description")}
	670	+
	671	+ delta = await reconcile(conn, company_id=company_id, sensor=sensor, ex=ex, now=now, previous_meta=prev_meta)
	672	+ history = {"consecutive_unchanged": sensor.get("consecutive_unchanged") or 0, "observation_count": sensor.get("observation_count") or 0,
	673	+ "change_count": sensor.get("change_count") or 0}
	674	+ diff: BlockDiff = compare(prev_blocks, ex.blocks, surface=str(sensor["surface"]), before_text=prev_text, after_text=ex.text,
	675	+ structured_delta=delta, history=history) if prev is not None else BlockDiff()
	676	+ kind = change_kind(diff.significance, noise=settings.noise_threshold, meaningful=settings.meaningful_threshold, major=settings.major_threshold,
	677	+ critical=settings.critical_threshold) if prev is not None else ChangeKind.NOISE
	678	+ typed = any(k in delta and any(isinstance(v, list) and v for v in delta[k].values()) for k in ("jobs", "people", "products", "plans", "locations", "news")
	679	+ if isinstance(delta.get(k), dict))
	680	+ keep_snapshot = prev is None or kind != ChangeKind.NOISE or typed or settings.keep_noise_snapshots
	681	+ snap_id: str \| None = None
	682	+ change_id: str \| None = None
	683	+ version_no = int(sensor.get("snapshot_count") or 0) + 1
	684	+ if keep_snapshot:
	685	+ snap_id = new_id("snapshot")
	686	+ text_key, _s, _c = archive.put_text(ex.text)
	687	+ blocks_key, _s, _c = archive.put_text(_blocks_json(ex.blocks))
	688	+ extracted = _bounded_extracted(ex)
	689	+ await execute(conn, """
	690	+ insert into snapshots (id, sensor_id, company_id, observation_id, previous_snapshot_id, version_no, fetched_at, content_hash, normalized_hash,
	691	+ structural_hash, object_key, text_key, blocks_key, extracted, extracted_summary, title, language, size_bytes, text_length,
	692	+ block_count, content_type, connector_version, collection_method)
	693	+ values (:id, :sid, :cid, :oid, :prev, :v, :now, :ch, :nh, :sh, :ok, :tk, :bk, cast(:ex as jsonb), cast(:sum as jsonb), :title, :lang, :size, :tl, :bc,
	694	+ :ct, :cv, :cm)
	695	+ """, id=snap_id, sid=sid, cid=company_id, oid=obs_id, prev=prev_id, v=version_no, now=now, ch=fetched.sha256, nh=ex.normalized_hash, sh=struct_hash,
	696	+ ok=object_key, tk=text_key, bk=blocks_key, ex=jsonb(extracted), sum=jsonb(ex.summary()), title=(ex.title or None), lang=ex.language,
	697	+ size=len(fetched.content), tl=len(ex.text), bc=len(ex.blocks), ct=fetched.content_type[:100], cv=connector.connector_id, cm=collection_method)
	698	+ if prev is not None:
	699	+ change_id = new_id("change")
	700	+ status = "pending" if kind in (ChangeKind.MEANINGFUL, ChangeKind.MAJOR, ChangeKind.CRITICAL) else "archived"
	701	+ await execute(conn, """
	702	+ insert into changes (id, sensor_id, company_id, surface, snapshot_before, snapshot_after, detected_at, significance, kind, blocks_added,
	703	+ blocks_removed, blocks_modified, blocks_moved, text_delta_ratio, similarity, diff, structured_delta, status, diff_version)
	704	+ values (:id, :sid, :cid, :surface, :before, :after, :now, :sig, :kind, :ba, :br, :bm, :bmv, :tdr, :sim, cast(:diff as jsonb), cast(:sd as jsonb),
	705	+ :status, :dv)
	706	+ """, id=change_id, sid=sid, cid=company_id, surface=str(sensor["surface"]), before=prev_id, after=snap_id, now=now, sig=diff.significance,
	707	+ kind=str(kind), ba=len(diff.added), br=len(diff.removed), bm=len(diff.modified), bmv=len(diff.moved), tdr=diff.text_delta_ratio,
	708	+ sim=diff.similarity, diff=jsonb(diff.to_json()), sd=jsonb(delta), status=status, dv=DIFF_VERSION)
	709	+ meaningful = prev is not None and kind in (ChangeKind.MEANINGFUL, ChangeKind.MAJOR, ChangeKind.CRITICAL)
	710	+ if prev is None:
	711	+ interval = current
	712	+ else:
	713	+ interval = next_interval_changed(kind, current, base_interval)
	714	+ nxt = _next_run(now, interval)
	715	+ quality = _quality_update(quality, extraction_conf)
	716	+ cfg["last_structured_hash"] = ex.structured_hash
	717	+ cfg.pop("redirect_url", None)
	718	+ await execute(conn, """
	719	+ update sensors set status = 'active', last_run_at = :now, last_success_at = :now, last_status = :st, last_failure_class = null, last_error = null,
	720	+ consecutive_failures = 0, consecutive_unchanged = case when :meaningful then 0 else consecutive_unchanged + 1 end,
	721	+ observation_count = observation_count + 1, snapshot_count = case when :kept then snapshot_count + 1 else snapshot_count end,
	722	+ change_count = case when :haschange then change_count + 1 else change_count end,
	723	+ meaningful_change_count = case when :meaningful then meaningful_change_count + 1 else meaningful_change_count end,
	724	+ last_change_at = case when :haschange then :now else last_change_at end,
	725	+ last_meaningful_change_at = case when :meaningful then :now else last_meaningful_change_at end,
	726	+ last_content_hash = :ch, last_normalized_hash = :nh, last_structural_hash = :sh, last_snapshot_id = coalesce(:snap, last_snapshot_id),
	727	+ etag = :etag, last_modified = :lm, current_interval_s = :iv, tier = :tier, next_run_at = :nxt, quality_score = :q,
	728	+ config = config \|\| cast(:cfg as jsonb), claimed_by = null, claimed_at = null, updated_at = now()
	729	+ where id = :id
	730	+ """, now=now, st=fetched.status, meaningful=meaningful, kept=keep_snapshot, haschange=change_id is not None, ch=fetched.sha256, nh=ex.normalized_hash,
	731	+ sh=struct_hash, snap=snap_id, etag=etag, lm=last_mod, iv=interval, tier=tier_for_interval(interval), nxt=nxt, q=quality,
	732	+ cfg=jsonb({"last_structured_hash": ex.structured_hash, "redirect_url": None}), id=sid)
	733	+ await execute(conn, """
	734	+ update companies set last_observed_at = :now, first_observed_at = coalesce(first_observed_at, :now),
	735	+ last_change_at = case when :meaningful then :now else last_change_at end,
	736	+ stats = stats \|\| jsonb_build_object('observations', coalesce((stats->>'observations')::int, 0) + 1,
	737	+ 'snapshots', coalesce((stats->>'snapshots')::int, 0) + case when :kept then 1 else 0 end,
	738	+ 'changes', coalesce((stats->>'changes')::int, 0) + case when :haschange then 1 else 0 end,
	739	+ 'meaningful_changes', coalesce((stats->>'meaningful_changes')::int, 0) + case when :meaningful then 1 else 0 end),
	740	+ updated_at = now()
	741	+ where id = :id
	742	+ """, now=now, meaningful=meaningful, kept=keep_snapshot, haschange=change_id is not None, id=company_id)
	743	+ outcome.status = "changed" if prev is not None else "ok"
	744	+ outcome.snapshot_id, outcome.change_id = snap_id, change_id
	745	+ outcome.significance = diff.significance if prev is not None else None
	746	+ outcome.kind = str(kind) if prev is not None else None
	747	+ outcome.next_run_at, outcome.interval_s, outcome.sensor_status = nxt, interval, "active"
	748	+ outcome.delta_counts = _delta_counts(delta)
	749	+ outcome.duration_ms = int((time.perf_counter() - t0) * 1000)
	750	+ log.info("sensor run", extra={"sensor_id": sid, "surface": sensor.get("surface"), "status": outcome.status, "kind": outcome.kind, "significance": outcome.significance,
	751	+ "delta": outcome.delta_counts, "interval_s": interval, "ms": outcome.duration_ms})
	752	+ return outcome
	753	+
	754	+
	755	+async def _record_failure(conn: Any, *, sensor: dict[str, Any], cfg: dict[str, Any], company_id: str, obs_id: str, failure: tuple[str, str, int \| None], now: datetime,
	756	+ fetch_ms: int, connector_id: str, worker: str, current: int, quality: float, outcome: RunOutcome, collection_method: str,
	757	+ final_url: str \| None = None, object_key: str \| None = None, size: int \| None = None, content_type: str \| None = None) -> None:
	758	+ fclass, message, status_code = failure
	759	+ sid = str(sensor["id"])
	760	+ await execute(conn, """
	761	+ insert into observations (id, sensor_id, company_id, fetched_at, status_code, duration_ms, transport, final_url, not_modified, changed, failure_class, error,
	762	+ object_key, size_bytes, content_type, connector_version, collection_method, worker)
	763	+ values (:id, :sid, :cid, :now, :st, :ms, 'http', :url, false, false, :fc, :err, :ok, :size, :ct, :cv, :cm, :worker)
	764	+ """, id=obs_id, sid=sid, cid=company_id, now=now, st=status_code, ms=fetch_ms, url=final_url or sensor["url"], fc=fclass, err=message, ok=object_key, size=size,
	765	+ ct=(content_type or "")[:100] or None, cv=connector_id, cm=collection_method, worker=worker)
	766	+ await execute(conn, "insert into failures (id, sensor_id, company_id, at, failure_class, status_code, message, url) values (:id, :sid, :cid, :now, :fc, :st, :msg, :url)",
	767	+ id=new_id("failure"), sid=sid, cid=company_id, now=now, fc=fclass, st=status_code, msg=message[:500], url=sensor["url"])
	768	+ failures = int(sensor.get("consecutive_failures") or 0) + 1
	769	+ _mult, threshold = FAILURE_POLICY.get(fclass, FAILURE_POLICY[FailureClass.UNKNOWN])
	770	+ prev_status = str(sensor.get("status") or "active")
	771	+ status = prev_status if prev_status in (SensorStatus.PAUSED, SensorStatus.RETIRED) else SensorStatus.ACTIVE
	772	+ review_kind: str \| None = None
	773	+ if fclass == FailureClass.REDIRECT and cfg.get("redirect_url"):
	774	+ status, review_kind = SensorStatus.REDIRECTED, "sensor_migration"
	775	+ elif fclass == FailureClass.ROBOTS:
	776	+ status, review_kind = SensorStatus.BLOCKED, "blocked_source"
	777	+ elif fclass == FailureClass.BOT_CHALLENGE:
	778	+ status = SensorStatus.FAILING if failures >= threshold else status
	779	+ review_kind = "blocked_source" if failures >= threshold else None
	780	+ elif failures >= settings.retire_after_failures:
	781	+ status = SensorStatus.RETIRED
	782	+ elif failures >= settings.stale_after_failures:
	783	+ status = SensorStatus.STALE
	784	+ elif failures >= threshold:
	785	+ status = SensorStatus.FAILING
	786	+ interval = next_interval_failed(fclass, current)
	787	+ nxt = _next_run(now, interval)
	788	+ quality = _quality_update(quality, 0.0)
	789	+ retire = status == SensorStatus.RETIRED
	790	+ await execute(conn, """
	791	+ update sensors set status = :status, last_run_at = :now, last_status = :st, last_failure_class = :fc, last_error = :err, consecutive_failures = :cf,
	792	+ consecutive_unchanged = 0, observation_count = observation_count + 1, current_interval_s = :iv, tier = :tier, next_run_at = :nxt, quality_score = :q,
	793	+ config = config \|\| cast(:cfg as jsonb), retired_at = case when :retire then coalesce(retired_at, :now) else retired_at end,
	794	+ claimed_by = null, claimed_at = null, updated_at = now()
	795	+ where id = :id
	796	+ """, status=str(status), now=now, st=status_code, fc=fclass, err=message[:500], cf=failures, iv=interval, tier=tier_for_interval(interval), nxt=nxt, q=quality,
	797	+ cfg=jsonb({k: v for k, v in cfg.items() if k in ("redirect_url",)}), retire=retire, id=sid)
	798	+ if review_kind:
	799	+ existing = await fetch_one(conn, "select id from review_queue where kind = :k and ref_id = :r and status = 'open'", k=review_kind, r=sid)
	800	+ if existing is None:
	801	+ await execute(conn, """insert into review_queue (id, kind, ref_id, company_id, payload) values (:id, :k, :r, :cid, cast(:p as jsonb))""",
	802	+ id=new_id("review"), k=review_kind, r=sid, cid=company_id,
	803	+ p=jsonb({"sensor_id": sid, "url": sensor["url"], "surface": sensor.get("surface"), "failure_class": fclass, "message": message[:300],
	804	+ "redirect_url": cfg.get("redirect_url"), "consecutive_failures": failures}))
	805	+ await execute(conn, "update companies set last_observed_at = :now where id = :id", now=now, id=company_id)
	806	+ outcome.status = "redirected" if status == SensorStatus.REDIRECTED else "failed"
	807	+ outcome.failure_class, outcome.error, outcome.next_run_at, outcome.interval_s, outcome.sensor_status = fclass, message, nxt, interval, str(status)
	808	+
	809	+
	810	+# ------------------------------------------------------------------------------------------------------------ convenience entry points
	811	+
	812	+
	813	+async def load_sensor(sensor_ref: str) -> dict[str, Any] \| None:
	814	+ async with transaction() as conn:
	815	+ row = await fetch_one(conn, "select * from sensors where id = :ref", ref=sensor_ref)
	816	+ if row is None:
	817	+ row = await fetch_one(conn, "select * from sensors where canonical_url = :c order by created_at limit 1", c=canonicalize_url(sensor_ref))
	818	+ if row is None:
	819	+ row = await fetch_one(conn, "select * from sensors where url = :u order by created_at limit 1", u=sensor_ref)
	820	+ return row
	821	+
	822	+
	823	+async def run_sensor_ids(ids: list[str], *, fetcher: Fetcher, worker: str = "local", force: bool = False, concurrency: int \| None = None) -> list[RunOutcome]:
	824	+ sem = asyncio.Semaphore(max(1, concurrency or settings.fetch_concurrency))
	825	+ out: list[RunOutcome] = []
	826	+
	827	+ async def one(sid: str) -> None:
	828	+ row = await load_sensor(sid)
	829	+ if row is None:
	830	+ out.append(RunOutcome(sensor_id=sid, status="failed", error="sensor not found"))
	831	+ return
	832	+ async with sem:
	833	+ try:
	834	+ out.append(await run_sensor(row, fetcher=fetcher, worker=worker, force=force))
	835	+ except Exception as exc:
	836	+ log.exception("sensor run crashed", extra={"sensor_id": sid})
	837	+ out.append(RunOutcome(sensor_id=sid, status="failed", error=f"{exc.__class__.__name__}: {exc}"[:300]))
	838	+ async with transaction() as conn:
	839	+ await execute(conn, "update sensors set claimed_by = null, claimed_at = null, next_run_at = now() + interval '30 minutes' where id = :id", id=sid)
	840	+
	841	+ await asyncio.gather(*(one(s) for s in ids))
	842	+ return out
	843	+
	844	+
	845	+async def run_sensor_by_url_with_file(sensor_ref: str, path: str, *, worker: str = "file", force: bool = False, content_type: str \| None = None) -> RunOutcome:
	846	+ """Run a sensor against a fixture file instead of the network (`catlas run-sensor --file`)."""
	847	+ row = await load_sensor(sensor_ref)
	848	+ if row is None:
	849	+ raise LookupError(f"sensor {sensor_ref!r} not found")
	850	+ res = file_result(path, url=str(row["url"]), content_type=content_type or _guess_content_type(path))
	851	+ return await run_sensor(row, fetcher=Fetcher(), worker=worker, force=force, result=res)
	852	+
	853	+
	854	+def surface_importance(surface: str) -> float:
	855	+ return float(SURFACE_IMPORTANCE.get(surface, 0.3))
	856	+
	857	+
	858	+__all__ = ["PIPELINE_VERSION", "RunOutcome", "is_ai_title", "load_sensor", "next_interval_changed", "next_interval_failed", "next_interval_unchanged", "reconcile",
	859	+ "run_sensor", "run_sensor_by_url_with_file", "run_sensor_ids"]

added src/companyatlas/services/repair.py +276 −0

		`@@ -0,0 +1,276 @@`
	1	+"""Sensor auto-repair (spec §60, §189): failing / stale / redirected / PAGE_REMOVED sensors.
	2	+
	3	+ retry ─▶ (redirect target) ─▶ re-fetch sitemap ─▶ rediscover navigation ─▶ candidate URL for the same surface
	4	+ ─▶ content-identity check (simhash / fuzzy ratio against the last snapshot text) ─▶ migrate (old retired with
	5	+ config.successor_id, new sensor with config.predecessor_id, history kept) or review_queue(kind='sensor_migration').
	6	+
	7	+Registered as a periodic task (every 30 min, bounded batch). Bounded requests per sensor; never raises out of the batch.
	8	+"""
	9	+from __future__ import annotations
	10	+
	11	+import logging
	12	+import re
	13	+from datetime import UTC, datetime
	14	+from typing import Any
	15	+from urllib.parse import urlparse
	16	+
	17	+from rapidfuzz import fuzz
	18	+
	19	+from companyatlas import archive
	20	+from companyatlas.config import settings
	21	+from companyatlas.db import execute, fetch_all, fetch_one, jsonb, transaction
	22	+from companyatlas.fetch import BlockedError, Fetcher, FetchError, FetchResult, NotModified
	23	+from companyatlas.ids import new_id
	24	+from companyatlas.sdk import connector as connectors
	25	+from companyatlas.sdk import normalize
	26	+from companyatlas.sdk.normalize import hamming, normalized_text, simhash
	27	+from companyatlas.services.periodic import periodic
	28	+from companyatlas.taxonomy import SensorStatus, Surface
	29	+from companyatlas.urls import canonicalize_url, classify_url, registrable_domain, same_company_host
	30	+
	31	+log = logging.getLogger(__name__)
	32	+
	33	+REPAIR_VERSION = "repair-v1"
	34	+MAX_REQUESTS_PER_SENSOR = 6
	35	+IDENTITY_HAMMING = 12
	36	+IDENTITY_RATIO = 0.6
	37	+MIN_CANDIDATE_CONF = 0.7
	38	+SOFT_404_RE = re.compile(r"(page not found\|404\|not be found\|doesn'?t exist\|no longer available)", re.IGNORECASE)
	39	+
	40	+
	41	+class _Session:
	42	+ def __init__(self, fetcher: Fetcher):
	43	+ self.fetcher = fetcher
	44	+ self.requests = 0
	45	+
	46	+ async def get(self, url: str, , max_bytes: int = 2 1024 * 1024, accept: str \| None = None) -> FetchResult \| None:
	47	+ if self.requests >= MAX_REQUESTS_PER_SENSOR:
	48	+ return None
	49	+ self.requests += 1
	50	+ try:
	51	+ return await self.fetcher.get(url, max_bytes=max_bytes, retries=0, accept=accept)
	52	+ except NotModified:
	53	+ return None
	54	+ except (FetchError, BlockedError) as exc:
	55	+ log.debug("repair fetch failed", extra={"url": url, "failure": str(exc.failure)})
	56	+ return None
	57	+ except Exception as exc: # noqa: BLE001
	58	+ log.debug("repair fetch error", extra={"url": url, "error": str(exc)[:200]})
	59	+ return None
	60	+
	61	+
	62	+def _soft_404(res: FetchResult) -> bool:
	63	+ m = re.search(r"<title[^>]>(.?)</title>", res.text[:4000], re.IGNORECASE \| re.DOTALL)
	64	+ return bool(m and SOFT_404_RE.search(m.group(1)))
	65	+
	66	+
	67	+async def _last_text(sensor: dict[str, Any]) -> str:
	68	+ sid = sensor.get("last_snapshot_id")
	69	+ if not sid:
	70	+ return ""
	71	+ async with transaction() as conn:
	72	+ row = await fetch_one(conn, "select text_key from snapshots where id = :id", id=sid)
	73	+ if row and row.get("text_key") and archive.exists(row["text_key"]):
	74	+ try:
	75	+ return archive.get_text(row["text_key"])
	76	+ except OSError:
	77	+ return ""
	78	+ return ""
	79	+
	80	+
	81	+def content_identity(previous_text: str, candidate_text: str) -> tuple[bool, float]:
	82	+ """(same_content, score). simhash distance first, fuzzy ratio second — both on noise-normalised text."""
	83	+ if not previous_text or not candidate_text:
	84	+ return False, 0.0
	85	+ d = hamming(simhash(previous_text), simhash(candidate_text))
	86	+ ratio = fuzz.ratio(normalized_text(previous_text)[:20000], normalized_text(candidate_text)[:20000]) / 100.0
	87	+ return (d <= IDENTITY_HAMMING or ratio >= IDENTITY_RATIO), round(max(ratio, 1 - d / 64), 3)
	88	+
	89	+
	90	+async def _candidates(session: _Session, sensor: dict[str, Any], company: dict[str, Any]) -> list[tuple[str, float, str]]:
	91	+ """(url, confidence, method) for the sensor's surface: redirect target, sitemap entries, navigation links."""
	92	+ surface = str(sensor["surface"])
	93	+ canonical = str(company.get("canonical_domain") or sensor["domain"])
	94	+ out: list[tuple[str, float, str]] = []
	95	+ cfg = sensor.get("config") or {}
	96	+ if cfg.get("redirect_url"):
	97	+ out.append((str(cfg["redirect_url"]), 0.9, "redirect"))
	98	+ origin = f"https://{urlparse(str(company.get('website') or 'https://' + canonical)).netloc or canonical}"
	99	+ # sitemap
	100	+ async with transaction() as conn:
	101	+ sm = await fetch_one(conn, "select url from sensors where company_id = :c and surface = 'sitemap' and status <> 'retired' order by quality_score desc limit 1",
	102	+ c=company["id"])
	103	+ sm_url = str(sm["url"]) if sm else f"{origin}/sitemap.xml"
	104	+ res = await session.get(sm_url, max_bytes=4 * 1024 * 1024, accept="application/xml,text/xml,/;q=0.5")
	105	+ if res is not None and not res.is_html:
	106	+ from companyatlas.connectors.sitemap import _decode, parse_sitemap
	107	+
	108	+ pages, children = parse_sitemap(_decode(res.content))
	109	+ if children and not pages:
	110	+ child = await session.get(children[0][0], max_bytes=4 * 1024 * 1024, accept="application/xml,text/xml,/;q=0.5")
	111	+ if child is not None:
	112	+ pages, _ = parse_sitemap(_decode(child.content))
	113	+ for loc, _lm in pages[: settings.discovery_max_sitemap_urls]:
	114	+ if not same_company_host(loc, canonical):
	115	+ continue
	116	+ s, c = classify_url(loc, canonical_domain=canonical)
	117	+ if s == surface and c >= MIN_CANDIDATE_CONF:
	118	+ out.append((loc, c * 0.9, "sitemap"))
	119	+ # navigation
	120	+ home = await session.get(origin + "/")
	121	+ if home is not None and home.is_html:
	122	+ page = normalize.parse(home.text, url=home.final_url)
	123	+ for ln in page.links:
	124	+ if not same_company_host(ln.url, canonical):
	125	+ continue
	126	+ s, c = classify_url(ln.url, anchor=ln.anchor, canonical_domain=canonical)
	127	+ if s == surface and c >= MIN_CANDIDATE_CONF:
	128	+ out.append((ln.url, c, "nav"))
	129	+ # dedupe, exclude the broken URL itself
	130	+ seen = {canonicalize_url(str(sensor["url"]))}
	131	+ uniq: list[tuple[str, float, str]] = []
	132	+ for url, conf, method in sorted(out, key=lambda x: -x[1]):
	133	+ cu = canonicalize_url(url)
	134	+ if cu in seen:
	135	+ continue
	136	+ seen.add(cu)
	137	+ uniq.append((url, conf, method))
	138	+ return uniq[:5]
	139	+
	140	+
	141	+async def migrate_sensor(sensor: dict[str, Any], new_url: str, *, reason: str, confidence: float, identity_score: float) -> str:
	142	+ """Retire the old sensor (successor_id) and create the replacement (predecessor_id). Returns the new sensor id."""
	143	+ now = datetime.now(UTC)
	144	+ surface = str(sensor["surface"])
	145	+ connector = connectors.for_surface(surface, new_url)
	146	+ new_id_ = new_id("sensor")
	147	+ old_cfg = dict(sensor.get("config") or {})
	148	+ new_cfg = {k: v for k, v in old_cfg.items() if k in ("canonical_domain", "vendor", "token")}
	149	+ new_cfg.update({"predecessor_id": sensor["id"], "repair": {"version": REPAIR_VERSION, "reason": reason, "confidence": confidence, "identity": identity_score,
	150	+ "at": now.isoformat(), "from_url": sensor["url"]}})
	151	+ async with transaction() as conn:
	152	+ existing = await fetch_one(conn, "select id from sensors where company_id = :c and canonical_url = :u", c=sensor["company_id"], u=canonicalize_url(new_url))
	153	+ if existing is not None:
	154	+ new_id_ = str(existing["id"])
	155	+ await execute(conn, "update sensors set status = case when status = 'retired' then 'pending' else status end, next_run_at = now(), updated_at = now() where id = :id",
	156	+ id=new_id_)
	157	+ else:
	158	+ await execute(conn, """
	159	+ insert into sensors (id, company_id, surface, connector_id, url, canonical_url, domain, discovery_confidence, discovery_method, quality_score, status, tier,
	160	+ base_interval_s, current_interval_s, next_run_at, priority, config)
	161	+ values (:id, :c, :surface, :conn, :url, :curl, :domain, :conf, 'repair', :q, 'pending', :tier, :base, :base, now(), :prio, cast(:cfg as jsonb))
	162	+ """, id=new_id_, c=sensor["company_id"], surface=surface, conn=connector.connector_id, url=new_url, curl=canonicalize_url(new_url),
	163	+ domain=registrable_domain(new_url), conf=round(confidence, 3), q=max(30.0, float(sensor.get("quality_score") or 50) * 0.9),
	164	+ tier=sensor.get("tier") or "D", base=int(sensor.get("base_interval_s") or 86400), prio=float(sensor.get("priority") or 0.5), cfg=jsonb(new_cfg))
	165	+ await execute(conn, """update sensors set status = 'retired', retired_at = coalesce(retired_at, now()), claimed_by = null, claimed_at = null,
	166	+ config = config \|\| cast(:cfg as jsonb), updated_at = now() where id = :id""",
	167	+ id=sensor["id"], cfg=jsonb({"successor_id": new_id_, "retired_reason": reason}))
	168	+ await execute(conn, "update review_queue set status = 'resolved', resolution = :r, resolved_at = now() where ref_id = :id and status = 'open'",
	169	+ id=sensor["id"], r=f"migrated to {new_id_}")
	170	+ return new_id_
	171	+
	172	+
	173	+async def repair_sensor(sensor: dict[str, Any], *, fetcher: Fetcher, dry_run: bool = False) -> dict[str, Any]:
	174	+ session = _Session(fetcher)
	175	+ sid = str(sensor["id"])
	176	+ result: dict[str, Any] = {"sensor_id": sid, "surface": sensor["surface"], "url": sensor["url"], "action": "none", "requests": 0}
	177	+ async with transaction() as conn:
	178	+ company = await fetch_one(conn, "select id, canonical_domain, website from companies where id = :id", id=sensor["company_id"])
	179	+ if company is None:
	180	+ result["action"] = "orphan"
	181	+ return result
	182	+ # 1. retry the URL itself
	183	+ res = await session.get(str(sensor["url"]))
	184	+ if res is not None and res.status == 200 and registrable_domain(res.final_url) == registrable_domain(str(sensor["url"])) and not (res.is_html and _soft_404(res)):
	185	+ result["action"] = "recovered"
	186	+ if not dry_run:
	187	+ async with transaction() as conn:
	188	+ await execute(conn, """update sensors set status = 'active', consecutive_failures = 0, next_run_at = now(), claimed_by = null, claimed_at = null,
	189	+ config = config - 'redirect_url', updated_at = now() where id = :id""", id=sid)
	190	+ result["requests"] = session.requests
	191	+ return result
	192	+ if res is not None and registrable_domain(res.final_url) != registrable_domain(str(sensor["url"])):
	193	+ sensor = {sensor, "config": {(sensor.get("config") or {}), "redirect_url": res.final_url}}
	194	+ # 2–4. candidates + identity check
	195	+ previous = await _last_text(sensor)
	196	+ cands = await _candidates(session, sensor, company)
	197	+ result["candidates"] = [{"url": u, "confidence": c, "method": m} for u, c, m in cands]
	198	+ best: tuple[str, float, str, float] \| None = None
	199	+ for url, conf, method in cands[:3]:
	200	+ r = await session.get(url)
	201	+ if r is None or r.status != 200 or (r.is_html and _soft_404(r)):
	202	+ continue
	203	+ text = normalize.parse(r.text, url=r.final_url).text if r.is_html else r.text
	204	+ same, score = content_identity(previous, text)
	205	+ result.setdefault("checked", []).append({"url": url, "identity": score, "same": same})
	206	+ if same and (best is None or score > best[3]):
	207	+ best = (r.final_url, conf, method, score)
	208	+ elif best is None and method == "redirect" and conf >= 0.9 and not previous:
	209	+ best = (r.final_url, conf, method, 0.0)
	210	+ result["requests"] = session.requests
	211	+ if best is not None:
	212	+ result["action"] = "migrated" if not dry_run else "would_migrate"
	213	+ result["new_url"] = best[0]
	214	+ if not dry_run:
	215	+ result["new_sensor_id"] = await migrate_sensor(sensor, best[0], reason=best[2], confidence=best[1], identity_score=best[3])
	216	+ return result
	217	+ # uncertain: one strong candidate → human review; none → leave the sensor to its failure policy
	218	+ if cands and not dry_run:
	219	+ async with transaction() as conn:
	220	+ existing = await fetch_one(conn, "select id from review_queue where kind = 'sensor_migration' and ref_id = :r and status = 'open'", r=sid)
	221	+ if existing is None:
	222	+ await execute(conn, "insert into review_queue (id, kind, ref_id, company_id, payload) values (:id, 'sensor_migration', :r, :c, cast(:p as jsonb))",
	223	+ id=new_id("review"), r=sid, c=sensor["company_id"], p=jsonb({"sensor_id": sid, "url": sensor["url"], "surface": sensor["surface"],
	224	+ "candidates": result["candidates"], "checked": result.get("checked", [])}))
	225	+ result["action"] = "review"
	226	+ elif cands:
	227	+ result["action"] = "would_review"
	228	+ async with transaction() as conn:
	229	+ await execute(conn, "update sensors set config = config \|\| cast(:c as jsonb), updated_at = now() where id = :id", id=sid,
	230	+ c=jsonb({"last_repair_at": datetime.now(UTC).isoformat(), "last_repair_action": result["action"]}))
	231	+ return result
	232	+
	233	+
	234	+async def repair_batch(limit: int = 50, *, fetcher: Fetcher \| None = None, dry_run: bool = False) -> dict[str, Any]:
	235	+ own = fetcher is None
	236	+ fetcher = fetcher or Fetcher()
	237	+ if own:
	238	+ await fetcher.open()
	239	+ stats: dict[str, Any] = {"examined": 0, "recovered": 0, "migrated": 0, "review": 0, "none": 0, "results": []}
	240	+ try:
	241	+ async with transaction() as conn:
	242	+ rows = await fetch_all(conn, """
	243	+ select * from sensors
	244	+ where (status in ('failing', 'stale', 'redirected') or (status = 'active' and last_failure_class = 'PAGE_REMOVED' and consecutive_failures >= 2))
	245	+ and (config->>'last_repair_at' is null or (config->>'last_repair_at')::timestamptz < now() - interval '1 day')
	246	+ and surface <> :sitemap
	247	+ order by quality_score desc, last_run_at nulls first limit :n
	248	+ """, n=limit, sitemap=str(Surface.SITEMAP))
	249	+ for row in rows:
	250	+ stats["examined"] += 1
	251	+ try:
	252	+ r = await repair_sensor(row, fetcher=fetcher, dry_run=dry_run)
	253	+ except Exception as exc:
	254	+ log.exception("repair failed", extra={"sensor_id": row.get("id")})
	255	+ r = {"sensor_id": row.get("id"), "action": "error", "error": str(exc)[:200]}
	256	+ key = r["action"].removeprefix("would_")
	257	+ stats[key] = stats.get(key, 0) + 1
	258	+ stats["results"].append(r)
	259	+ finally:
	260	+ if own:
	261	+ await fetcher.close()
	262	+ return stats
	263	+
	264	+
	265	+@periodic("sensor-repair", every_s=1800, initial_delay_s=120)
	266	+async def repair_task() -> None:
	267	+ stats = await repair_batch(limit=50)
	268	+ if stats["examined"]:
	269	+ log.info("repair batch", extra={k: v for k, v in stats.items() if k != "results"})
	270	+
	271	+
	272	+def sensor_status_values() -> tuple[str, ...]:
	273	+ return tuple(s.value for s in SensorStatus)
	274	+
	275	+
	276	+__all__ = ["REPAIR_VERSION", "content_identity", "migrate_sensor", "repair_batch", "repair_sensor", "repair_task"]

added src/companyatlas/services/scheduler.py +215 −0

		`@@ -0,0 +1,215 @@`
	1	+"""`catlas schedule` — the long-running crawl process (spec §15–16, §62–63, §124–128).
	2	+
	3	+Every `settings.scheduler_tick_s` the scheduler claims due sensors from Postgres (`for update skip locked`, so any number of
	4	+scheduler processes on any machines can share one database), runs them through `pipeline.run_sensor` under a global semaphore
	5	+(the per-domain governor lives in `fetch.Fetcher`), releases claims, writes a `crawl_runs` row and a heartbeat in `settings_kv`.
	6	+It also hosts the onboarding worker (`discovery.onboard_pending`) and every periodic task registered through
	7	+`services.periodic` (interval + cron via APScheduler in `settings.tz`). Claims older than `CLAIM_TTL` are considered abandoned.
	8	+"""
	9	+from __future__ import annotations
	10	+
	11	+import asyncio
	12	+import logging
	13	+import os
	14	+import signal
	15	+import socket
	16	+import time
	17	+from datetime import UTC, datetime
	18	+from typing import Any
	19	+
	20	+from companyatlas.config import settings
	21	+from companyatlas.db import execute, fetch_all, jsonb, transaction
	22	+from companyatlas.fetch import Fetcher
	23	+from companyatlas.ids import new_id
	24	+from companyatlas.services import periodic
	25	+from companyatlas.services.pipeline import RunOutcome, run_sensor
	26	+
	27	+log = logging.getLogger(__name__)
	28	+
	29	+CLAIM_TTL = "15 minutes"
	30	+IDLE_RUN_ROW_EVERY = 40 # write a crawl_runs row on idle ticks only every N ticks
	31	+ONBOARD_BATCH = 20
	32	+ONBOARD_EVERY_TICKS = 4
	33	+
	34	+
	35	+def worker_name() -> str:
	36	+ return f"{socket.gethostname().split('.')[0]}:{os.getpid()}"
	37	+
	38	+
	39	+async def claim_due_sensors(worker: str, limit: int) -> list[dict[str, Any]]:
	40	+ async with transaction() as conn:
	41	+ return await fetch_all(conn, f"""
	42	+ with due as (
	43	+ select id from sensors
	44	+ where status in ('active', 'failing', 'pending') and next_run_at <= now()
	45	+ and (claimed_at is null or claimed_at < now() - interval '{CLAIM_TTL}')
	46	+ order by priority desc, next_run_at
	47	+ limit :limit for update skip locked)
	48	+ update sensors s set claimed_by = :worker, claimed_at = now() from due where s.id = due.id returning s.*
	49	+ """, limit=limit, worker=worker)
	50	+
	51	+
	52	+async def release_claims(worker: str) -> None:
	53	+ async with transaction() as conn:
	54	+ await execute(conn, "update sensors set claimed_by = null, claimed_at = null where claimed_by = :w", w=worker)
	55	+
	56	+
	57	+async def heartbeat(worker: str, payload: dict[str, Any]) -> None:
	58	+ async with transaction() as conn:
	59	+ for key in ("scheduler:heartbeat", f"scheduler:heartbeat:{worker}"):
	60	+ await execute(conn, """insert into settings_kv (key, value) values (:k, cast(:v as jsonb))
	61	+ on conflict (key) do update set value = excluded.value, updated_at = now()""", k=key, v=jsonb(payload))
	62	+
	63	+
	64	+async def record_run(kind: str, worker: str, started_at: datetime, stats: dict[str, Any], error: str \| None = None) -> None:
	65	+ async with transaction() as conn:
	66	+ await execute(conn, """insert into crawl_runs (id, kind, worker, started_at, finished_at, stats, error)
	67	+ values (:id, :kind, :worker, :started, now(), cast(:stats as jsonb), :error)""",
	68	+ id=new_id("crawl_run"), kind=kind, worker=worker, started=started_at, stats=jsonb(stats), error=error)
	69	+
	70	+
	71	+class Scheduler:
	72	+ def __init__(self, *, concurrency: int \| None = None, onboarding: bool = True, worker: str \| None = None, claim_batch: int \| None = None):
	73	+ self.concurrency = max(1, concurrency or settings.fetch_concurrency)
	74	+ self.onboarding = onboarding
	75	+ self.worker = worker or worker_name()
	76	+ self.claim_batch = claim_batch or settings.scheduler_claim_batch
	77	+ self.stop = asyncio.Event()
	78	+ self.inflight = 0
	79	+ self.ticks = 0
	80	+ self.fetcher = Fetcher()
	81	+ self._onboard_task: asyncio.Task[Any] \| None = None
	82	+ self._aps: Any = None
	83	+
	84	+ # ------------------------------------------------------------------ one tick
	85	+ async def tick(self) -> dict[str, Any]:
	86	+ started = datetime.now(UTC)
	87	+ t0 = time.perf_counter()
	88	+ sensors = await claim_due_sensors(self.worker, self.claim_batch)
	89	+ stats = {"claimed": len(sensors), "ok": 0, "changed": 0, "meaningful": 0, "failed": 0, "not_modified": 0, "unchanged": 0, "skipped": 0, "redirected": 0}
	90	+ sem = asyncio.Semaphore(self.concurrency)
	91	+
	92	+ async def one(row: dict[str, Any]) -> None:
	93	+ async with sem:
	94	+ self.inflight += 1
	95	+ try:
	96	+ outcome: RunOutcome = await run_sensor(row, fetcher=self.fetcher, worker=self.worker)
	97	+ except Exception as exc:
	98	+ log.exception("sensor run crashed", extra={"sensor_id": row.get("id")})
	99	+ stats["failed"] += 1
	100	+ async with transaction() as conn:
	101	+ await execute(conn, """update sensors set claimed_by = null, claimed_at = null, last_error = :e,
	102	+ next_run_at = now() + interval '30 minutes' where id = :id""", id=row["id"], e=f"{exc.__class__.__name__}: {exc}"[:500])
	103	+ return
	104	+ finally:
	105	+ self.inflight -= 1
	106	+ if outcome.status in ("ok", "changed", "unchanged", "not_modified"):
	107	+ stats["ok"] += 1
	108	+ if outcome.status == "changed":
	109	+ stats["changed"] += 1
	110	+ if outcome.kind in ("meaningful", "major", "critical"):
	111	+ stats["meaningful"] += 1
	112	+ elif outcome.status in stats:
	113	+ stats[outcome.status] += 1
	114	+
	115	+ if sensors:
	116	+ await asyncio.gather(*(one(s) for s in sensors))
	117	+ stats["tick_ms"] = int((time.perf_counter() - t0) * 1000)
	118	+ self.ticks += 1
	119	+ if sensors or self.ticks % IDLE_RUN_ROW_EVERY == 0:
	120	+ await record_run("scheduler_tick", self.worker, started, stats)
	121	+ due = await self._due_count()
	122	+ await heartbeat(self.worker, {"worker": self.worker, "at": datetime.now(UTC).isoformat(), "inflight": self.inflight, "due": due, "tick_ms": stats["tick_ms"],
	123	+ "claimed": stats["claimed"], "concurrency": self.concurrency, "tasks": periodic.snapshot()})
	124	+ log.info("tick", extra={"worker": self.worker, **stats, "due": due})
	125	+ return stats
	126	+
	127	+ async def _due_count(self) -> int:
	128	+ async with transaction() as conn:
	129	+ rows = await fetch_all(conn, "select count(*) as n from sensors where status in ('active','failing','pending') and next_run_at <= now()")
	130	+ return int(rows[0]["n"]) if rows else 0
	131	+
	132	+ # ------------------------------------------------------------------ onboarding
	133	+ async def _maybe_onboard(self) -> None:
	134	+ if not self.onboarding or (self._onboard_task and not self._onboard_task.done()):
	135	+ return
	136	+ from companyatlas.services.discovery import onboard_pending
	137	+
	138	+ async def go() -> None:
	139	+ try:
	140	+ stats = await onboard_pending(ONBOARD_BATCH, settings.onboarding_concurrency, fetcher=self.fetcher, worker=self.worker)
	141	+ if stats.get("claimed"):
	142	+ await record_run("onboarding", self.worker, datetime.now(UTC), stats)
	143	+ except Exception:
	144	+ log.exception("onboarding batch failed")
	145	+
	146	+ self._onboard_task = asyncio.create_task(go())
	147	+
	148	+ # ------------------------------------------------------------------ periodic tasks
	149	+ def _start_periodic(self) -> None:
	150	+ loaded = periodic.load_task_modules()
	151	+ try:
	152	+ from apscheduler.schedulers.asyncio import AsyncIOScheduler
	153	+ from apscheduler.triggers.cron import CronTrigger
	154	+ from apscheduler.triggers.interval import IntervalTrigger
	155	+ except ImportError: # pragma: no cover
	156	+ log.warning("apscheduler unavailable — periodic tasks disabled")
	157	+ return
	158	+ aps = AsyncIOScheduler(timezone=settings.tz)
	159	+ for task in periodic.tasks().values():
	160	+ if task.cron:
	161	+ trigger: Any = CronTrigger.from_crontab(task.cron, timezone=settings.tz)
	162	+ else:
	163	+ trigger = IntervalTrigger(seconds=float(task.every_s or 60), start_date=datetime.now(UTC).timestamp() + task.initial_delay_s and None)
	164	+ aps.add_job(task.run, trigger=trigger, id=task.name, name=task.name, max_instances=1, coalesce=True, misfire_grace_time=60)
	165	+ aps.start()
	166	+ self._aps = aps
	167	+ log.info("periodic tasks started", extra={"modules": loaded, "tasks": list(periodic.tasks())})
	168	+
	169	+ # ------------------------------------------------------------------ main loop
	170	+ async def run(self, *, once: bool = False) -> None:
	171	+ await self.fetcher.open()
	172	+ loop = asyncio.get_running_loop()
	173	+ for sig in (signal.SIGINT, signal.SIGTERM):
	174	+ try:
	175	+ loop.add_signal_handler(sig, self.stop.set)
	176	+ except (NotImplementedError, RuntimeError): # pragma: no cover - non-unix
	177	+ pass
	178	+ try:
	179	+ from companyatlas.sdk.connector import sync_connectors_table
	180	+
	181	+ async with transaction() as conn:
	182	+ await sync_connectors_table(conn)
	183	+ if not once:
	184	+ self._start_periodic()
	185	+ log.info("scheduler started", extra={"worker": self.worker, "concurrency": self.concurrency, "tick_s": settings.scheduler_tick_s, "onboarding": self.onboarding})
	186	+ while not self.stop.is_set():
	187	+ try:
	188	+ if self.onboarding and (self.ticks % ONBOARD_EVERY_TICKS == 0):
	189	+ await self._maybe_onboard()
	190	+ await self.tick()
	191	+ except Exception:
	192	+ log.exception("tick failed")
	193	+ if once:
	194	+ if self._onboard_task:
	195	+ await self._onboard_task
	196	+ break
	197	+ try:
	198	+ await asyncio.wait_for(self.stop.wait(), settings.scheduler_tick_s)
	199	+ except TimeoutError:
	200	+ pass
	201	+ finally:
	202	+ if self._aps is not None:
	203	+ self._aps.shutdown(wait=False)
	204	+ if self._onboard_task and not self._onboard_task.done():
	205	+ self._onboard_task.cancel()
	206	+ await release_claims(self.worker)
	207	+ await self.fetcher.close()
	208	+ log.info("scheduler stopped", extra={"worker": self.worker, "ticks": self.ticks})
	209	+
	210	+
	211	+async def run_scheduler(*, concurrency: int \| None = None, onboarding: bool = True, once: bool = False, worker: str \| None = None) -> None:
	212	+ await Scheduler(concurrency=concurrency, onboarding=onboarding, worker=worker).run(once=once)
	213	+
	214	+
	215	+__all__ = ["Scheduler", "claim_due_sensors", "heartbeat", "record_run", "release_claims", "run_scheduler", "worker_name"]

modified src/companyatlas/urls.py +47 −47

		`@@ -17,7 +17,7 @@ _extract = tldextract.TLDExtract(suffix_list_urls=(), fallback_to_snapshot=True)`
17	17	TRACKING_PREFIXES = ("utm_", "ref", "fbclid", "gclid", "dclid", "msclkid", "mc_cid", "mc_eid", "_hs", "hsa_", "igshid", "yclid", "_ga",
18	18	"_gl", "source", "campaign", "mkt_tok", "trk", "cmpid", "s_kwcid", "ef_id", "sessionid", "session_id", "phpsessid",
19	19	"jsessionid", "sid", "cid", "icid", "ncid", "spm", "srsltid")
20		−SESSION_PATH_RE = re.compile(r";jsessionid=[^/?#]+", re.I)
	20	+SESSION_PATH_RE = re.compile(r";jsessionid=[^/?#]+", re.IGNORECASE)
21	21	MULTI_SLASH_RE = re.compile(r"/{2,}")
22	22	STATIC_EXT = (".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp", ".ico", ".css", ".js", ".mjs", ".woff", ".woff2", ".ttf", ".eot", ".mp4",
23	23	".mp3", ".webm", ".mov", ".zip", ".gz", ".tar", ".dmg", ".exe", ".pkg", ".apk", ".ics", ".xlsx", ".pptx", ".docx")
		`@@ -85,7 +85,7 @@ def absolutize(base: str, href: str) -> str \| None:`
85	85
86	86	# ------------------------------------------------------------------------------------------------------ crawl-trap heuristics
87	87
88		−TRAP_PARAM_RE = re.compile(r"(^\|[?&])(page\|p\|offset\|start\|sort\|order\|filter\|facet\|color\|size\|price\|min\|max\|year\|month\|day\|date\|q\|s\|search)=", re.I)
	88	+TRAP_PARAM_RE = re.compile(r"(^\|[?&])(page\|p\|offset\|start\|sort\|order\|filter\|facet\|color\|size\|price\|min\|max\|year\|month\|day\|date\|q\|s\|search)=", re.IGNORECASE)
89	89	CALENDAR_RE = re.compile(r"/(19\|20)\d{2}/(0?[1-9]\|1[0-2])(/\|$)")
90	90
91	91
		`@@ -106,40 +106,40 @@ def looks_like_trap(url: str) -> bool:`
106	106
107	107	_R = re.compile
108	108	RULES: list[tuple[Surface, re.Pattern[str] \| None, re.Pattern[str] \| None, float]] = [
109		− (Surface.PRICING, _R(r"/(pricing\|plans\|plans-and-pricing\|pricing-plans\|tarifs\|preise\|precios\|prezzi\|价格)(/\|$)", re.I), _R(r"^(pricing\|plans( & pricing\| and pricing)?\|see pricing\|view pricing\|tarifs\|preise\|precios)$", re.I), 0.95),
110		− (Surface.JOBS_BOARD, _R(r"(boards\.greenhouse\.io\|job-boards\.greenhouse\.io\|jobs\.lever\.co\|jobs\.ashbyhq\.com\|jobs\.smartrecruiters\.com\|careers\.smartrecruiters\.com\|myworkdayjobs\.com\|apply\.workable\.com\|jobs\.jobvite\.com\|recruiting\.paylocity\.com\|bamboohr\.com/careers\|breezy\.hr\|recruitee\.com\|personio\.(de\|com)\|teamtailor\.com\|icims\.com\|taleo\.net\|successfactors\.com\|eightfold\.ai\|phenom\.com\|wd\d\.myworkdaysite\.com\|careers-page\.com\|homerun\.co\|pinpointhq\.com\|rippling-ats\.com\|jobs\.gem\.com)", re.I), None, 0.97),
111		− (Surface.CAREERS, _R(r"/(careers?\|jobs?\|join(-us\|us)?\|work-with-us\|work-for-us\|working-at\|open-positions\|opportunities\|vacancies\|recruit(ing\|ment)?\|employment\|karriere\|stellen\|emplois?\|carri[eè]res?\|empleo\|trabaja-con-nosotros\|lavora-con-noi\|saiyou\|採用)(/\|$)", re.I), _R(r"^(careers?\|jobs?\|join (us\|the team\|our team)\|work (with\|for\|at) us\|open (roles\|positions)\|we'?re hiring\|hiring\|opportunities\|vacancies\|karriere\|emplois?\|carri[eè]res?\|empleo)$", re.I), 0.93),
112		− (Surface.NEWSROOM, _R(r"/(news(room)?\|press(-releases?\|room\|-center\|-centre)?\|media(-center\|-centre\|-room)?\|announcements\|releases\|actualit[eé]s\|presse\|noticias\|prensa\|ニュース)(/\|$)", re.I), _R(r"^(news(room)?\|press( releases?\| room\| center)?\|media( center\| room)?\|announcements\|in the news\|actualit[eé]s\|presse\|noticias)$", re.I), 0.9),
113		− (Surface.INVESTOR_RELATIONS, _R(r"/(investors?\|investor-relations\|ir\|shareholders?\|financials?\|earnings\|sec-filings\|annual-reports?\|investisseurs\|investoren\|inversores)(/\|$)\|^https?://(ir\|investors?\|investor)\.", re.I), _R(r"^(investors?\|investor relations\|shareholders?\|financials?\|ir\|investisseurs\|investoren)$", re.I), 0.92),
114		− (Surface.LEADERSHIP, _R(r"/(leadership\|management(-team)?\|executive(s\|-team\|-leadership)?\|our-team\|the-team\|team\|board(-of-directors)?\|directors\|founders\|people\|who-we-are\|direction\|equipe\|équipe\|equipo\|vorstand\|management-board\|governance/(board\|leadership))(/\|$)", re.I), _R(r"^(leadership( team)?\|management( team)?\|executive (team\|leadership)\|our (team\|leadership\|people)\|meet the team\|board of directors\|founders\|team\|direction\|équipe)$", re.I), 0.85),
115		− (Surface.LOCATIONS, _R(r"/(locations?\|offices?\|our-offices\|where-we-are\|global-presence\|worldwide\|stores?\|store-locator\|find-a-store\|branches\|dealers?\|showrooms?\|sites\|standorte\|bureaux\|ubicaciones\|拠点)(/\|$)", re.I), _R(r"^(locations?\|our (locations\|offices)\|offices?\|where we are\|global presence\|find a store\|store locator\|branches\|standorte\|bureaux)$", re.I), 0.88),
116		− (Surface.CHANGELOG, _R(r"/(changelog\|change-log\|changes\|release-notes\|releases\|whats-new\|what's-new\|updates\|product-updates)(/\|$)\|^https?://(changelog\|releases\|updates)\.", re.I), _R(r"^(changelog\|release notes\|what'?s new\|product updates\|updates\|releases)$", re.I), 0.9),
117		− (Surface.API, _R(r"/(api\|apis\|api-reference\|api-docs\|reference)(/\|$)\|^https?://api-?docs?\.", re.I), _R(r"^(api( reference\| docs\| documentation)?\|apis\|rest api\|graphql)$", re.I), 0.82),
118		− (Surface.DEVELOPER, _R(r"/(developers?\|dev\|devs\|developer-portal\|platform\|sdks?\|integrations?\|build)(/\|$)\|^https?://(developers?\|dev\|build)\.", re.I), _R(r"^(developers?\|developer (portal\|center\|hub)\|for developers\|sdks?\|integrations?\|build)$", re.I), 0.8),
119		− (Surface.DOCS, _R(r"/(docs\|documentation\|help-center\|help\|guides?\|manuals?\|knowledge-base\|kb\|learn\|tutorials?)(/\|$)\|^https?://(docs\|documentation\|help\|support\|kb\|learn\|guides?)\.", re.I), _R(r"^(docs\|documentation\|guides?\|help center\|knowledge base\|manuals?\|tutorials?\|learn)$", re.I), 0.8),
120		− (Surface.STATUS, _R(r"^https?://(status\|health\|uptime\|trust)\.\|/(status\|system-status\|service-status)(/\|$)", re.I), _R(r"^(status\|system status\|service status\|status page)$", re.I), 0.85),
121		− (Surface.BLOG, _R(r"/(blog\|blogs\|insights\|stories\|articles\|journal\|magazine\|perspectives\|thinking\|ideas\|posts\|editorial\|le-blog)(/\|$)\|^https?://(blog\|insights\|stories\|medium)\.", re.I), _R(r"^(blog\|insights\|stories\|articles\|journal\|perspectives\|ideas\|our blog)$", re.I), 0.82),
122		− (Surface.RESEARCH, _R(r"/(research\|labs?\|science\|publications\|papers\|whitepapers?\|reports\|studies)(/\|$)\|^https?://(research\|labs?\|science)\.", re.I), _R(r"^(research\|labs?\|publications\|whitepapers?\|reports\|science)$", re.I), 0.78),
123		− (Surface.PRODUCTS, _R(r"/(products?\|product-catalog\|catalog(ue)?\|shop\|store\|collections\|portfolio\|offerings\|our-products\|produits\|produkte\|productos\|prodotti\|製品)(/\|$)\|^https?://(shop\|store\|products?)\.", re.I), _R(r"^(products?\|our products\|product catalog\|catalog(ue)?\|shop\|store\|portfolio\|offerings\|produits\|produkte)$", re.I), 0.82),
124		− (Surface.SERVICES, _R(r"/(services?\|what-we-do\|capabilities\|expertise\|offerings\|our-services\|prestations\|leistungen\|servicios\|servizi)(/\|$)", re.I), _R(r"^(services?\|our services\|what we do\|capabilities\|expertise\|leistungen\|prestations)$", re.I), 0.78),
125		− (Surface.SOLUTIONS, _R(r"/(solutions?\|use-cases\|platform\|technology\|technologies\|features)(/\|$)", re.I), _R(r"^(solutions?\|use cases\|platform\|features)$", re.I), 0.7),
126		− (Surface.INDUSTRIES, _R(r"/(industries\|industry\|sectors?\|markets?\|verticals?\|who-we-serve)(/\|$)", re.I), _R(r"^(industries\|sectors?\|markets?\|who we serve\|verticals?)$", re.I), 0.7),
127		− (Surface.CUSTOMERS, _R(r"/(customers?\|customer-stories\|case-studies\|success-stories\|clients?\|references\|testimonials\|showcase\|wall-of-love)(/\|$)", re.I), _R(r"^(customers?\|customer stories\|case studies\|success stories\|clients?\|our customers\|testimonials)$", re.I), 0.78),
128		− (Surface.PARTNERS, _R(r"/(partners?\|partnerships?\|partner-program\|alliances\|ecosystem\|marketplace\|resellers?\|channel)(/\|$)\|^https?://(partners?\|marketplace)\.", re.I), _R(r"^(partners?\|partnerships?\|partner program\|alliances\|ecosystem\|become a partner\|marketplace)$", re.I), 0.78),
129		− (Surface.SECURITY, _R(r"/(security\|trust(-center\|-portal)?\|compliance\|privacy-and-security)(/\|$)\|^https?://(security\|trust)\.", re.I), _R(r"^(security\|trust( center)?\|compliance\|trust & safety)$", re.I), 0.8),
130		− (Surface.SUSTAINABILITY, _R(r"/(sustainability\|esg\|responsibility\|corporate-responsibility\|csr\|impact\|environment\|climate\|social-impact\|citizenship\|purpose\|d[eé]veloppement-durable\|nachhaltigkeit\|sostenibilidad)(/\|$)\|^https?://(sustainability\|esg\|impact)\.", re.I), _R(r"^(sustainability\|esg\|(corporate \|social )?responsibility\|csr\|impact\|environment\|climate\|our impact\|purpose)$", re.I), 0.82),
131		− (Surface.LEGAL_PRIVACY, _R(r"/(privacy(-policy\|-notice\|-statement)?\|privacypolicy\|datenschutz\|confidentialit[eé]\|privacidad\|cookie-policy\|cookies)(/\|$)", re.I), _R(r"^(privacy( policy\| notice\| statement)?\|datenschutz(erkl[aä]rung)?\|politique de confidentialit[eé]\|cookie policy)$", re.I), 0.9),
132		− (Surface.LEGAL_TERMS, _R(r"/(terms(-of-(service\|use\|sale\|business))?\|tos\|legal\|legal-notice\|terms-and-conditions\|conditions\|eula\|agb\|mentions-l[eé]gales\|cgu\|cgv\|aviso-legal\|impressum\|acceptable-use(-policy)?\|aup)(/\|$)", re.I), _R(r"^(terms( of (service\|use\|sale))?\|terms (and\|&) conditions\|legal( notice)?\|agb\|mentions l[eé]gales\|impressum\|eula\|acceptable use policy)$", re.I), 0.88),
133		− (Surface.SUPPORT, _R(r"/(support\|customer-support\|customer-service\|contact-support\|faq\|faqs\|community\|forum)(/\|$)\|^https?://(community\|forum\|faq)\.", re.I), _R(r"^(support\|customer (support\|service\|care)\|faqs?\|community\|forum\|help & support)$", re.I), 0.72),
134		− (Surface.CONTACT, _R(r"/(contact(-us\|us\|-sales)?\|get-in-touch\|reach-us\|kontakt\|contactez-nous\|contacto\|お問い合わせ)(/\|$)", re.I), _R(r"^(contact( us\| sales)?\|get in touch\|talk to (us\|sales)\|kontakt\|contactez-nous\|contacto)$", re.I), 0.85),
135		− (Surface.ABOUT, _R(r"/(about(-us\|us\|-company\|-[a-z0-9-]+)?\|company\|our-company\|our-story\|who-we-are\|mission\|history\|overview\|corporate\|a-propos\|qui-sommes-nous\|[uü]ber-uns\|unternehmen\|sobre-nosotros\|chi-siamo\|会社概要\|企業情報)(/\|$)", re.I), _R(r"^(about( us\| the company)?\|company\|our (company\|story\|mission\|history)\|who we are\|mission\|history\|overview\|a propos\|qui sommes-nous\|[uü]ber uns\|unternehmen)$", re.I), 0.85),
136		− (Surface.FEED, _R(r"/(feed\|rss\|atom\|feeds)(\.xml\|/\|$)\|\.(rss\|atom)$\|/rss\.xml\|/feed\.xml\|/atom\.xml", re.I), _R(r"^(rss\|atom\|feed\|subscribe via rss)$", re.I), 0.9),
137		− (Surface.SITEMAP, _R(r"/sitemap[^/]*\.xml(\.gz)?$\|/sitemap_index\.xml$\|/sitemaps?/", re.I), None, 0.95),
	109	+ (Surface.PRICING, _R(r"/(pricing\|plans\|plans-and-pricing\|pricing-plans\|tarifs\|preise\|precios\|prezzi\|价格)(/\|$)", re.IGNORECASE), _R(r"^(pricing\|plans( & pricing\| and pricing)?\|see pricing\|view pricing\|tarifs\|preise\|precios)$", re.IGNORECASE), 0.95),
	110	+ (Surface.JOBS_BOARD, _R(r"(boards\.greenhouse\.io\|job-boards\.greenhouse\.io\|jobs\.lever\.co\|jobs\.ashbyhq\.com\|jobs\.smartrecruiters\.com\|careers\.smartrecruiters\.com\|myworkdayjobs\.com\|apply\.workable\.com\|jobs\.jobvite\.com\|recruiting\.paylocity\.com\|bamboohr\.com/careers\|breezy\.hr\|recruitee\.com\|personio\.(de\|com)\|teamtailor\.com\|icims\.com\|taleo\.net\|successfactors\.com\|eightfold\.ai\|phenom\.com\|wd\d\.myworkdaysite\.com\|careers-page\.com\|homerun\.co\|pinpointhq\.com\|rippling-ats\.com\|jobs\.gem\.com)", re.IGNORECASE), None, 0.97),
	111	+ (Surface.CAREERS, _R(r"/(careers?\|jobs?\|join(-us\|us)?\|work-with-us\|work-for-us\|working-at\|open-positions\|opportunities\|vacancies\|recruit(ing\|ment)?\|employment\|karriere\|stellen\|emplois?\|carri[eè]res?\|empleo\|trabaja-con-nosotros\|lavora-con-noi\|saiyou\|採用)(/\|$)", re.IGNORECASE), _R(r"^(careers?\|jobs?\|join (us\|the team\|our team)\|work (with\|for\|at) us\|open (roles\|positions)\|we'?re hiring\|hiring\|opportunities\|vacancies\|karriere\|emplois?\|carri[eè]res?\|empleo)$", re.IGNORECASE), 0.93),
	112	+ (Surface.NEWSROOM, _R(r"/(news(room)?\|press(-releases?\|room\|-center\|-centre)?\|media(-center\|-centre\|-room)?\|announcements\|releases\|actualit[eé]s\|presse\|noticias\|prensa\|ニュース)(/\|$)", re.IGNORECASE), _R(r"^(news(room)?\|press( releases?\| room\| center)?\|media( center\| room)?\|announcements\|in the news\|actualit[eé]s\|presse\|noticias)$", re.IGNORECASE), 0.9),
	113	+ (Surface.INVESTOR_RELATIONS, _R(r"/(investors?\|investor-relations\|ir\|shareholders?\|financials?\|earnings\|sec-filings\|annual-reports?\|investisseurs\|investoren\|inversores)(/\|$)\|^https?://(ir\|investors?\|investor)\.", re.IGNORECASE), _R(r"^(investors?\|investor relations\|shareholders?\|financials?\|ir\|investisseurs\|investoren)$", re.IGNORECASE), 0.92),
	114	+ (Surface.LEADERSHIP, _R(r"/(leadership\|management(-team)?\|executive(s\|-team\|-leadership)?\|our-team\|the-team\|team\|board(-of-directors)?\|directors\|founders\|people\|who-we-are\|direction\|equipe\|équipe\|equipo\|vorstand\|management-board\|governance/(board\|leadership))(/\|$)", re.IGNORECASE), _R(r"^(leadership( team)?\|management( team)?\|executive (team\|leadership)\|our (team\|leadership\|people)\|meet the team\|board of directors\|founders\|team\|direction\|équipe)$", re.IGNORECASE), 0.85),
	115	+ (Surface.LOCATIONS, _R(r"/(locations?\|offices?\|our-offices\|where-we-are\|global-presence\|worldwide\|stores?\|store-locator\|find-a-store\|branches\|dealers?\|showrooms?\|sites\|standorte\|bureaux\|ubicaciones\|拠点)(/\|$)", re.IGNORECASE), _R(r"^(locations?\|our (locations\|offices)\|offices?\|where we are\|global presence\|find a store\|store locator\|branches\|standorte\|bureaux)$", re.IGNORECASE), 0.88),
	116	+ (Surface.CHANGELOG, _R(r"/(changelog\|change-log\|changes\|release-notes\|releases\|whats-new\|what's-new\|updates\|product-updates)(/\|$)\|^https?://(changelog\|releases\|updates)\.", re.IGNORECASE), _R(r"^(changelog\|release notes\|what'?s new\|product updates\|updates\|releases)$", re.IGNORECASE), 0.9),
	117	+ (Surface.API, _R(r"/(api\|apis\|api-reference\|api-docs\|reference)(/\|$)\|^https?://api-?docs?\.", re.IGNORECASE), _R(r"^(api( reference\| docs\| documentation)?\|apis\|rest api\|graphql)$", re.IGNORECASE), 0.82),
	118	+ (Surface.DEVELOPER, _R(r"/(developers?\|dev\|devs\|developer-portal\|platform\|sdks?\|integrations?\|build)(/\|$)\|^https?://(developers?\|dev\|build)\.", re.IGNORECASE), _R(r"^(developers?\|developer (portal\|center\|hub)\|for developers\|sdks?\|integrations?\|build)$", re.IGNORECASE), 0.8),
	119	+ (Surface.DOCS, _R(r"/(docs\|documentation\|help-center\|help\|guides?\|manuals?\|knowledge-base\|kb\|learn\|tutorials?)(/\|$)\|^https?://(docs\|documentation\|help\|support\|kb\|learn\|guides?)\.", re.IGNORECASE), _R(r"^(docs\|documentation\|guides?\|help center\|knowledge base\|manuals?\|tutorials?\|learn)$", re.IGNORECASE), 0.8),
	120	+ (Surface.STATUS, _R(r"^https?://(status\|health\|uptime\|trust)\.\|/(status\|system-status\|service-status)(/\|$)", re.IGNORECASE), _R(r"^(status\|system status\|service status\|status page)$", re.IGNORECASE), 0.85),
	121	+ (Surface.BLOG, _R(r"/(blog\|blogs\|insights\|stories\|articles\|journal\|magazine\|perspectives\|thinking\|ideas\|posts\|editorial\|le-blog)(/\|$)\|^https?://(blog\|insights\|stories\|medium)\.", re.IGNORECASE), _R(r"^(blog\|insights\|stories\|articles\|journal\|perspectives\|ideas\|our blog)$", re.IGNORECASE), 0.82),
	122	+ (Surface.RESEARCH, _R(r"/(research\|labs?\|science\|publications\|papers\|whitepapers?\|reports\|studies)(/\|$)\|^https?://(research\|labs?\|science)\.", re.IGNORECASE), _R(r"^(research\|labs?\|publications\|whitepapers?\|reports\|science)$", re.IGNORECASE), 0.78),
	123	+ (Surface.PRODUCTS, _R(r"/(products?\|product-catalog\|catalog(ue)?\|shop\|store\|collections\|portfolio\|offerings\|our-products\|produits\|produkte\|productos\|prodotti\|製品)(/\|$)\|^https?://(shop\|store\|products?)\.", re.IGNORECASE), _R(r"^(products?\|our products\|product catalog\|catalog(ue)?\|shop\|store\|portfolio\|offerings\|produits\|produkte)$", re.IGNORECASE), 0.82),
	124	+ (Surface.SERVICES, _R(r"/(services?\|what-we-do\|capabilities\|expertise\|offerings\|our-services\|prestations\|leistungen\|servicios\|servizi)(/\|$)", re.IGNORECASE), _R(r"^(services?\|our services\|what we do\|capabilities\|expertise\|leistungen\|prestations)$", re.IGNORECASE), 0.78),
	125	+ (Surface.SOLUTIONS, _R(r"/(solutions?\|use-cases\|platform\|technology\|technologies\|features)(/\|$)", re.IGNORECASE), _R(r"^(solutions?\|use cases\|platform\|features)$", re.IGNORECASE), 0.7),
	126	+ (Surface.INDUSTRIES, _R(r"/(industries\|industry\|sectors?\|markets?\|verticals?\|who-we-serve)(/\|$)", re.IGNORECASE), _R(r"^(industries\|sectors?\|markets?\|who we serve\|verticals?)$", re.IGNORECASE), 0.7),
	127	+ (Surface.CUSTOMERS, _R(r"/(customers?\|customer-stories\|case-studies\|success-stories\|clients?\|references\|testimonials\|showcase\|wall-of-love)(/\|$)", re.IGNORECASE), _R(r"^(customers?\|customer stories\|case studies\|success stories\|clients?\|our customers\|testimonials)$", re.IGNORECASE), 0.78),
	128	+ (Surface.PARTNERS, _R(r"/(partners?\|partnerships?\|partner-program\|alliances\|ecosystem\|marketplace\|resellers?\|channel)(/\|$)\|^https?://(partners?\|marketplace)\.", re.IGNORECASE), _R(r"^(partners?\|partnerships?\|partner program\|alliances\|ecosystem\|become a partner\|marketplace)$", re.IGNORECASE), 0.78),
	129	+ (Surface.SECURITY, _R(r"/(security\|trust(-center\|-portal)?\|compliance\|privacy-and-security)(/\|$)\|^https?://(security\|trust)\.", re.IGNORECASE), _R(r"^(security\|trust( center)?\|compliance\|trust & safety)$", re.IGNORECASE), 0.8),
	130	+ (Surface.SUSTAINABILITY, _R(r"/(sustainability\|esg\|responsibility\|corporate-responsibility\|csr\|impact\|environment\|climate\|social-impact\|citizenship\|purpose\|d[eé]veloppement-durable\|nachhaltigkeit\|sostenibilidad)(/\|$)\|^https?://(sustainability\|esg\|impact)\.", re.IGNORECASE), _R(r"^(sustainability\|esg\|(corporate \|social )?responsibility\|csr\|impact\|environment\|climate\|our impact\|purpose)$", re.IGNORECASE), 0.82),
	131	+ (Surface.LEGAL_PRIVACY, _R(r"/(privacy(-policy\|-notice\|-statement)?\|privacypolicy\|datenschutz\|confidentialit[eé]\|privacidad\|cookie-policy\|cookies)(/\|$)", re.IGNORECASE), _R(r"^(privacy( policy\| notice\| statement)?\|datenschutz(erkl[aä]rung)?\|politique de confidentialit[eé]\|cookie policy)$", re.IGNORECASE), 0.9),
	132	+ (Surface.LEGAL_TERMS, _R(r"/(terms(-of-(service\|use\|sale\|business))?\|tos\|legal\|legal-notice\|terms-and-conditions\|conditions\|eula\|agb\|mentions-l[eé]gales\|cgu\|cgv\|aviso-legal\|impressum\|acceptable-use(-policy)?\|aup)(/\|$)", re.IGNORECASE), _R(r"^(terms( of (service\|use\|sale))?\|terms (and\|&) conditions\|legal( notice)?\|agb\|mentions l[eé]gales\|impressum\|eula\|acceptable use policy)$", re.IGNORECASE), 0.88),
	133	+ (Surface.SUPPORT, _R(r"/(support\|customer-support\|customer-service\|contact-support\|faq\|faqs\|community\|forum)(/\|$)\|^https?://(community\|forum\|faq)\.", re.IGNORECASE), _R(r"^(support\|customer (support\|service\|care)\|faqs?\|community\|forum\|help & support)$", re.IGNORECASE), 0.72),
	134	+ (Surface.CONTACT, _R(r"/(contact(-us\|us\|-sales)?\|get-in-touch\|reach-us\|kontakt\|contactez-nous\|contacto\|お問い合わせ)(/\|$)", re.IGNORECASE), _R(r"^(contact( us\| sales)?\|get in touch\|talk to (us\|sales)\|kontakt\|contactez-nous\|contacto)$", re.IGNORECASE), 0.85),
	135	+ (Surface.ABOUT, _R(r"/(about(-us\|us\|-company\|-[a-z0-9-]+)?\|company\|our-company\|our-story\|who-we-are\|mission\|history\|overview\|corporate\|a-propos\|qui-sommes-nous\|[uü]ber-uns\|unternehmen\|sobre-nosotros\|chi-siamo\|会社概要\|企業情報)(/\|$)", re.IGNORECASE), _R(r"^(about( us\| the company)?\|company\|our (company\|story\|mission\|history)\|who we are\|mission\|history\|overview\|a propos\|qui sommes-nous\|[uü]ber uns\|unternehmen)$", re.IGNORECASE), 0.85),
	136	+ (Surface.FEED, _R(r"/(feed\|rss\|atom\|feeds)(\.xml\|/\|$)\|\.(rss\|atom)$\|/rss\.xml\|/feed\.xml\|/atom\.xml", re.IGNORECASE), _R(r"^(rss\|atom\|feed\|subscribe via rss)$", re.IGNORECASE), 0.9),
	137	+ (Surface.SITEMAP, _R(r"/sitemap[^/]*\.xml(\.gz)?$\|/sitemap_index\.xml$\|/sitemaps?/", re.IGNORECASE), None, 0.95),
138	138	]
139	139
140	140	# Anchors that are almost always navigation noise, never surfaces.
141	141	_NOISE_ANCHOR = re.compile(r"^(home\|back\|next\|previous\|prev\|more\|read more\|learn more\|skip to content\|menu\|close\|login\|log in\|sign in\|sign up\|"
142		− r"register\|search\|cart\|account\|download\|share\|print\|top\|en\|fr\|de\|es\|it\|ja\|zh\|português\|english\|français\|deutsch)$", re.I)
	142	+ r"register\|search\|cart\|account\|download\|share\|print\|top\|en\|fr\|de\|es\|it\|ja\|zh\|português\|english\|français\|deutsch)$", re.IGNORECASE)
143	143
144	144
145	145	def classify_url(url: str, *, anchor: str \| None = None, title: str \| None = None, canonical_domain: str \| None = None) -> tuple[Surface, float]:
		`@@ -178,21 +178,21 @@ def classify_url(url: str, *, anchor: str \| None = None, title: str \| None = Non`
178	178
179	179
180	180	ATS_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
181		− ("greenhouse", re.compile(r"https?://(?:boards\|job-boards)\.greenhouse\.io/([a-z0-9_-]+)", re.I)),
182		− ("greenhouse", re.compile(r"https?://boards-api\.greenhouse\.io/v1/boards/([a-z0-9_-]+)", re.I)),
183		− ("lever", re.compile(r"https?://jobs\.(?:eu\.)?lever\.co/([a-z0-9_-]+)", re.I)),
184		− ("ashby", re.compile(r"https?://jobs\.ashbyhq\.com/([a-z0-9_.-]+)", re.I)),
185		− ("smartrecruiters", re.compile(r"https?://(?:careers\|jobs)\.smartrecruiters\.com/([a-z0-9_-]+)", re.I)),
186		− ("workable", re.compile(r"https?://apply\.workable\.com/([a-z0-9_-]+)", re.I)),
187		− ("workday", re.compile(r"https?://([a-z0-9-]+)\.(wd\d+)\.myworkdayjobs\.com/(?:[a-z]{2}-[A-Z]{2}/)?([A-Za-z0-9_-]+)", re.I)),
188		− ("recruitee", re.compile(r"https?://([a-z0-9-]+)\.recruitee\.com", re.I)),
189		− ("personio", re.compile(r"https?://([a-z0-9-]+)\.jobs\.personio\.(?:de\|com)", re.I)),
190		− ("teamtailor", re.compile(r"https?://([a-z0-9-]+)\.teamtailor\.com", re.I)),
191		− ("bamboohr", re.compile(r"https?://([a-z0-9-]+)\.bamboohr\.com/careers", re.I)),
192		− ("breezy", re.compile(r"https?://([a-z0-9-]+)\.breezy\.hr", re.I)),
193		− ("jobvite", re.compile(r"https?://jobs\.jobvite\.com/([a-z0-9_-]+)", re.I)),
194		− ("pinpoint", re.compile(r"https?://([a-z0-9-]+)\.pinpointhq\.com", re.I)),
195		− ("rippling", re.compile(r"https?://ats\.rippling\.com/([a-z0-9_-]+)", re.I)),
	181	+ ("greenhouse", re.compile(r"https?://(?:boards\|job-boards)\.greenhouse\.io/([a-z0-9_-]+)", re.IGNORECASE)),
	182	+ ("greenhouse", re.compile(r"https?://boards-api\.greenhouse\.io/v1/boards/([a-z0-9_-]+)", re.IGNORECASE)),
	183	+ ("lever", re.compile(r"https?://jobs\.(?:eu\.)?lever\.co/([a-z0-9_-]+)", re.IGNORECASE)),
	184	+ ("ashby", re.compile(r"https?://jobs\.ashbyhq\.com/([a-z0-9_.-]+)", re.IGNORECASE)),
	185	+ ("smartrecruiters", re.compile(r"https?://(?:careers\|jobs)\.smartrecruiters\.com/([a-z0-9_-]+)", re.IGNORECASE)),
	186	+ ("workable", re.compile(r"https?://apply\.workable\.com/([a-z0-9_-]+)", re.IGNORECASE)),
	187	+ ("workday", re.compile(r"https?://([a-z0-9-]+)\.(wd\d+)\.myworkdayjobs\.com/(?:[a-z]{2}-[A-Z]{2}/)?([A-Za-z0-9_-]+)", re.IGNORECASE)),
	188	+ ("recruitee", re.compile(r"https?://([a-z0-9-]+)\.recruitee\.com", re.IGNORECASE)),
	189	+ ("personio", re.compile(r"https?://([a-z0-9-]+)\.jobs\.personio\.(?:de\|com)", re.IGNORECASE)),
	190	+ ("teamtailor", re.compile(r"https?://([a-z0-9-]+)\.teamtailor\.com", re.IGNORECASE)),
	191	+ ("bamboohr", re.compile(r"https?://([a-z0-9-]+)\.bamboohr\.com/careers", re.IGNORECASE)),
	192	+ ("breezy", re.compile(r"https?://([a-z0-9-]+)\.breezy\.hr", re.IGNORECASE)),
	193	+ ("jobvite", re.compile(r"https?://jobs\.jobvite\.com/([a-z0-9_-]+)", re.IGNORECASE)),
	194	+ ("pinpoint", re.compile(r"https?://([a-z0-9-]+)\.pinpointhq\.com", re.IGNORECASE)),
	195	+ ("rippling", re.compile(r"https?://ats\.rippling\.com/([a-z0-9_-]+)", re.IGNORECASE)),
196	196	]
197	197
198	198

added tests/conftest.py +120 −0

		`@@ -0,0 +1,120 @@`
	1	+"""Shared test helpers: fixture paths, a network-free FakeFetcher, and a DB availability check for integration tests."""
	2	+from __future__ import annotations
	3	+
	4	+import asyncio
	5	+from dataclasses import dataclass, field
	6	+from datetime import UTC, datetime
	7	+from pathlib import Path
	8	+from typing import Any, Self
	9	+
	10	+import pytest
	11	+
	12	+from companyatlas.fetch import FetchError, FetchResult, NotModified
	13	+from companyatlas.taxonomy import FailureClass
	14	+
	15	+ROOT = Path(__file__).resolve().parents[1]
	16	+FIXTURES = ROOT / "fixtures" / "connectors"
	17	+
	18	+
	19	+def fixture_path(*parts: str) -> str:
	20	+ return str(FIXTURES.joinpath(*parts))
	21	+
	22	+
	23	+def fixture_bytes(*parts: str) -> bytes:
	24	+ return FIXTURES.joinpath(*parts).read_bytes()
	25	+
	26	+
	27	+def fixture_text(*parts: str) -> str:
	28	+ return FIXTURES.joinpath(*parts).read_text(encoding="utf-8")
	29	+
	30	+
	31	+def make_result(url: str, content: bytes \| str, *, content_type: str = "text/html; charset=utf-8", status: int = 200, final_url: str \| None = None,
	32	+ headers: dict[str, str] \| None = None) -> FetchResult:
	33	+ body = content.encode("utf-8") if isinstance(content, str) else content
	34	+ return FetchResult(url=url, final_url=final_url or url, status=status, headers={k.lower(): v for k, v in (headers or {}).items()}, content=body,
	35	+ content_type=content_type, fetched_at=datetime.now(UTC), duration_ms=1, transport="fake")
	36	+
	37	+
	38	+@dataclass
	39	+class FakeFetcher:
	40	+ """Network-free stand-in for `fetch.Fetcher`: maps URL → (status, content_type, body) or an exception. Records every request."""
	41	+ routes: dict[str, Any] = field(default_factory=dict)
	42	+ calls: list[str] = field(default_factory=list)
	43	+ default_status: int = 404
	44	+
	45	+ def add(self, url: str, body: bytes \| str, *, content_type: str = "text/html; charset=utf-8", status: int = 200, final_url: str \| None = None) -> None:
	46	+ self.routes[url] = (status, content_type, body, final_url)
	47	+
	48	+ def add_error(self, url: str, exc: Exception) -> None:
	49	+ self.routes[url] = exc
	50	+
	51	+ async def open(self) -> None:
	52	+ return None
	53	+
	54	+ async def close(self) -> None:
	55	+ return None
	56	+
	57	+ async def __aenter__(self) -> Self:
	58	+ return self
	59	+
	60	+ async def __aexit__(self, *exc: object) -> None:
	61	+ return None
	62	+
	63	+ async def get(self, url: str, **kw: Any) -> FetchResult:
	64	+ self.calls.append(url)
	65	+ hit = self.routes.get(url) or self.routes.get(url.rstrip("/")) or self.routes.get(url + "/")
	66	+ if hit is None:
	67	+ raise FetchError(f"http 404 for {url}", status=404, url=url, failure=FailureClass.PAGE_REMOVED)
	68	+ if isinstance(hit, Exception):
	69	+ if isinstance(hit, NotModified):
	70	+ raise hit
	71	+ raise hit
	72	+ status, ctype, body, final = hit
	73	+ if status >= 400:
	74	+ raise FetchError(f"http {status} for {url}", status=status, url=url, failure=FailureClass.HTTP_4XX if status < 500 else FailureClass.HTTP_5XX)
	75	+ return make_result(url, body, content_type=ctype, status=status, final_url=final)
	76	+
	77	+ async def request(self, method: str, url: str, **kw: Any) -> FetchResult:
	78	+ return await self.get(url, **kw)
	79	+
	80	+ async def post_json(self, url: str, payload: Any, **kw: Any) -> FetchResult:
	81	+ self.calls.append(f"POST {url}")
	82	+ return await self.get(url, **kw)
	83	+
	84	+
	85	+@pytest.fixture
	86	+def fake_fetcher() -> FakeFetcher:
	87	+ return FakeFetcher()
	88	+
	89	+
	90	+def _db_available() -> bool:
	91	+ async def probe() -> bool:
	92	+ from companyatlas.db import dispose, fetch_val, transaction
	93	+
	94	+ try:
	95	+ async with transaction() as conn:
	96	+ await fetch_val(conn, "select 1")
	97	+ return True
	98	+ except Exception: # noqa: BLE001
	99	+ return False
	100	+ finally:
	101	+ await dispose()
	102	+
	103	+ try:
	104	+ return asyncio.run(probe())
	105	+ except Exception: # noqa: BLE001
	106	+ return False
	107	+
	108	+
	109	+_DB_OK: bool \| None = None
	110	+
	111	+
	112	+@pytest.fixture
	113	+def db(): # type: ignore[no-untyped-def]
	114	+ """Skip integration tests when the local Postgres is unavailable."""
	115	+ global _DB_OK
	116	+ if _DB_OK is None:
	117	+ _DB_OK = _db_available()
	118	+ if not _DB_OK:
	119	+ pytest.skip("local Postgres not available")
	120	+ return True

added tests/test_classify.py +112 −0

		`@@ -0,0 +1,112 @@`
	1	+from __future__ import annotations
	2	+
	3	+import pytest
	4	+
	5	+from companyatlas.connectors._util import ats_sensor_spec, country_code, job_fingerprint, parse_date, parse_location
	6	+from companyatlas.connectors.generic_html import looks_like_name, parse_price, role_category
	7	+from companyatlas.sdk import connector as C
	8	+from companyatlas.taxonomy import Surface
	9	+from companyatlas.urls import classify_url, detect_ats
	10	+
	11	+
	12	+@pytest.mark.parametrize("url,anchor,surface", [
	13	+ ("https://stripe.com/pricing", "Pricing", Surface.PRICING),
	14	+ ("https://stripe.com/jobs", "Jobs", Surface.CAREERS),
	15	+ ("https://boards.greenhouse.io/stripe", None, Surface.JOBS_BOARD),
	16	+ ("https://stripe.com/newsroom", "Newsroom", Surface.NEWSROOM),
	17	+ ("https://investors.example.com/", None, Surface.INVESTOR_RELATIONS),
	18	+ ("https://example.com/about/leadership", "Leadership", Surface.LEADERSHIP),
	19	+ ("https://example.com/legal/privacy", None, Surface.LEGAL_PRIVACY),
	20	+ ("https://example.com/blog/atom.xml", None, Surface.FEED),
	21	+ ("https://example.com/sitemap.xml", None, Surface.SITEMAP),
	22	+])
	23	+def test_classify_url(url: str, anchor: str \| None, surface: Surface) -> None:
	24	+ s, conf = classify_url(url, anchor=anchor, canonical_domain="example.com")
	25	+ assert s == surface and conf >= 0.55
	26	+
	27	+
	28	+def test_connector_selection() -> None:
	29	+ assert C.for_surface("jobs_board", "https://boards-api.greenhouse.io/v1/boards/stripe/jobs?content=false").connector_id == "greenhouse-v1"
	30	+ assert C.for_surface("jobs_board", "https://api.lever.co/v0/postings/palantir?mode=json").connector_id == "lever-v1"
	31	+ assert C.for_surface("jobs_board", "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs").connector_id == "workday-v1"
	32	+ assert C.for_surface("feed", "https://x.com/blog/feed.xml").connector_id == "feed-v1"
	33	+ assert C.for_surface("sitemap", "https://x.com/sitemap_index.xml").connector_id == "sitemap-v1"
	34	+ assert C.for_surface("status", "https://status.x.com/api/v2/summary.json").connector_id == "statuspage-v1"
	35	+ assert C.for_surface("pricing", "https://x.com/pricing").connector_id == "generic-html-v1"
	36	+ assert C.for_surface("legal_terms", "https://x.com/terms").connector_id == "generic-html-v1"
	37	+ assert C.get("greenhouse-v0").connector_id == "greenhouse-v1" # older ids fall back to the family's newest
	38	+ # vendor connectors are pattern-bound: a non-ATS jobs_board URL never gets an ATS parser
	39	+ assert C.for_surface("jobs_board", "https://x.com/jobs").connector_id == "jsonld-jobs-v1"
	40	+ assert C.for_surface("status", "https://status.x.com/").connector_id == "generic-html-v1"
	41	+
	42	+
	43	+def test_ats_detection_and_specs() -> None:
	44	+ assert detect_ats("https://boards.greenhouse.io/stripe") == ("greenhouse", "stripe")
	45	+ spec = ats_sensor_spec("workday", "nvidia", "https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite")
	46	+ assert spec and spec[0].endswith("/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs") and spec[1] == "workday-v1"
	47	+ assert ats_sensor_spec("teamtailor", "acme", "https://career.acme.com/jobs")[0] == "https://career.acme.com/jobs.json"
	48	+ assert ats_sensor_spec("lever", "x", "https://jobs.eu.lever.co/x")[0].startswith("https://api.eu.lever.co/")
	49	+
	50	+
	51	+@pytest.mark.parametrize("text,city,region,country,remote", [
	52	+ ("San Francisco, CA", "San Francisco", "CA", None, None), # CA is ambiguous → no country guess
	53	+ ("Austin, TX", "Austin", "TX", "US", None),
	54	+ ("Toronto, ON, Canada", "Toronto", "ON", "CA", None),
	55	+ ("US, MA, Westford", "Westford", "MA", "US", None),
	56	+ ("Remote - US", None, None, "US", True),
	57	+ ("Germany", None, None, "DE", None),
	58	+ ("Paris, France (Hybrid)", "Paris", None, "FR", False),
	59	+ ("Dublin", None, None, None, None),
	60	+ ("London EC2M 2PF, United Kingdom", "London", None, "GB", None),
	61	+])
	62	+def test_parse_location(text: str, city: str \| None, region: str \| None, country: str \| None, remote: bool \| None) -> None:
	63	+ loc = parse_location(text)
	64	+ assert (loc["city"], loc["region"], loc["country"], loc["remote"]) == (city, region, country, remote)
	65	+
	66	+
	67	+def test_country_code_and_dates() -> None:
	68	+ assert country_code("United Kingdom") == "GB" and country_code("usa") == "US" and country_code("DEU") == "DE" and country_code("Atlantis") is None
	69	+ assert parse_date("2026-09-10T09:00:00Z").isoformat().startswith("2026-09-10T09:00")
	70	+ assert parse_date(1711403416463).year == 2024
	71	+ assert parse_date("Posted Yesterday") is None or parse_date("Posted Yesterday").year >= 2020
	72	+ assert parse_date("garbage") is None
	73	+
	74	+
	75	+def test_job_fingerprint_is_stable() -> None:
	76	+ a = job_fingerprint("Senior Engineer", "Toronto, ON", "123", None)
	77	+ b = job_fingerprint("senior engineer", "toronto, on", "123", "https://x/other")
	78	+ assert a == b and a != job_fingerprint("Senior Engineer", "Toronto, ON", "124", None)
	79	+
	80	+
	81	+@pytest.mark.parametrize("title,category,is_exec", [
	82	+ ("Chief Executive Officer", "ceo", True), ("Co-Founder & CEO", "founder", True), ("Chief Financial Officer", "cfo", True),
	83	+ ("SVP, Global Sales", "vp", False), ("Head of People", "head", False), ("Chairman of the Board", "chair", True),
	84	+ ("Independent Director", "board", False), ("Chief People Officer", "other", True), ("Software Engineer", "other", False),
	85	+])
	86	+def test_role_category(title: str, category: str, is_exec: bool) -> None:
	87	+ assert role_category(title) == (category, is_exec)
	88	+
	89	+
	90	+def test_looks_like_name() -> None:
	91	+ assert looks_like_name("María García-López") and looks_like_name("Tom O'Neill") and looks_like_name("Dr. Jane Doe")
	92	+ assert not looks_like_name("Meet the team") and not looks_like_name("Chief Executive Officer") and not looks_like_name("Read more")
	93	+
	94	+
	95	+@pytest.mark.parametrize("text,price,currency,period,unit,contact", [
	96	+ ("$29 per month", 29.0, "USD", "month", None, False),
	97	+ ("€1.299,00 / Jahr", 1299.0, "EUR", "year", None, False),
	98	+ ("£12 per user / month, billed annually", 12.0, "GBP", "month", "user", False),
	99	+ ("From $1,000/year", 1000.0, "USD", "year", None, False),
	100	+ ("Contact sales", None, None, "contact", None, True),
	101	+ ("Free", 0.0, None, None, None, False),
	102	+ ("CHF 49 monthly", 49.0, "CHF", "month", None, False),
	103	+ ("$0.10 per 1,000 requests", 0.10, "USD", "usage", "1,000", False),
	104	+])
	105	+def test_parse_price(text: str, price: float \| None, currency: str \| None, period: str \| None, unit: str \| None, contact: bool) -> None:
	106	+ info = parse_price(text)
	107	+ assert info is not None
	108	+ assert (info["price"], info["currency"], info["billing_period"], info["unit"], info["contact_sales"]) == (price, currency, period, unit, contact)
	109	+
	110	+
	111	+def test_parse_price_none_for_prose() -> None:
	112	+ assert parse_price("Up to 5 users and email support") is None

added tests/test_connectors_ats.py +111 −0

		`@@ -0,0 +1,111 @@`
	1	+from __future__ import annotations
	2	+
	3	+import json
	4	+
	5	+import pytest
	6	+from conftest import FakeFetcher, fixture_bytes, fixture_path, make_result
	7	+
	8	+from companyatlas.fetch import file_result
	9	+from companyatlas.sdk import connector as C
	10	+from companyatlas.sdk.connector import ConnectorContext
	11	+
	12	+CASES = [
	13	+ ("greenhouse-v1", ("greenhouse", "stripe_jobs.json"), "https://boards-api.greenhouse.io/v1/boards/stripe/jobs?content=false", 20, "Abuse Investigator", "8172487"),
	14	+ ("lever-v1", ("lever", "palantir_postings.json"), "https://api.lever.co/v0/postings/palantir?mode=json", 20, "Administrative Business Partner", None),
	15	+ ("ashby-v1", ("ashby", "ashby_board.json"), "https://api.ashbyhq.com/posting-api/job-board/ashby", 20, "Engineering Manager - EU", None),
	16	+ ("smartrecruiters-v1", ("smartrecruiters", "smartrecruiters_postings.json"), "https://api.smartrecruiters.com/v1/companies/smartrecruiters/postings?limit=100", 1,
	17	+ "Data Operations Consultant", "744000148454651"),
	18	+ ("workable-v1", ("workable", "epignosis_widget.json"), "https://apply.workable.com/api/v1/widget/accounts/epignosis", 7, "Data Engineer", "E38DB16625"),
	19	+ ("workday-v1", ("workday", "nvidia_jobs_page1.json"), "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs", 20,
	20	+ "Senior Embedded Software Engineer, DPU - Networking", "JR2017846"),
	21	+ ("recruitee-v1", ("recruitee", "vandebron_offers.json"), "https://vandebron.recruitee.com/api/offers/", 11, "Manager Market Operations", None),
	22	+ ("personio-v1", ("personio", "personio_jobs.xml"), "https://personio.jobs.personio.de/xml", 1, "Staff Software Engineer, Data Platform", "1834171"),
	23	+ ("teamtailor-v1", ("teamtailor", "teamtailor_jobs_feed.json"), "https://career.teamtailor.com/jobs.json", 10, "Account Executive - UK Enterprise", "8021334"),
	24	+]
	25	+
	26	+
	27	+@pytest.mark.parametrize("cid,fixture,url,count,first_title,first_ext", CASES)
	28	+def test_ats_connector_extracts_jobs(cid: str, fixture: tuple[str, str], url: str, count: int, first_title: str, first_ext: str \| None) -> None:
	29	+ con = C.get(cid)
	30	+ assert C.for_surface("jobs_board", url).connector_id == cid
	31	+ ct = "application/xml" if fixture[1].endswith(".xml") else "application/json"
	32	+ ex = con.extract({"url": url, "surface": "jobs_board", "config": {}}, file_result(fixture_path(*fixture), url=url, content_type=ct))
	33	+ assert len(ex.jobs) == count
	34	+ assert ex.jobs[0].title == first_title
	35	+ if first_ext:
	36	+ assert ex.jobs[0].external_id == first_ext
	37	+ assert all(j.url for j in ex.jobs)
	38	+ assert len(ex.blocks) == count and all(b.kind == "job_listing" for b in ex.blocks)
	39	+ assert ex.meta["structured"] is True and ex.meta["vendor"] == con.vendor # type: ignore[attr-defined]
	40	+ for j in ex.jobs:
	41	+ assert j.country is None or (len(j.country) == 2 and j.country.isupper())
	42	+
	43	+
	44	+def test_greenhouse_fields() -> None:
	45	+ ex = C.get("greenhouse-v1").extract({"url": "https://boards-api.greenhouse.io/v1/boards/stripe/jobs?content=false", "config": {}},
	46	+ file_result(fixture_path("greenhouse", "stripe_jobs.json"), url="x", content_type="application/json"))
	47	+ j = ex.jobs[0]
	48	+ assert j.url.startswith("https://stripe.com/jobs/") and j.location_text == "Dublin" and j.posted_at is not None
	49	+
	50	+
	51	+def test_workday_paginates_via_post(fake_fetcher: FakeFetcher) -> None:
	52	+ url = "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs"
	53	+ page = json.loads(fixture_bytes("workday", "nvidia_jobs_page1.json"))
	54	+ page["total"] = 25
	55	+ fake_fetcher.add(url, json.dumps(page), content_type="application/json")
	56	+ con = C.get("workday-v1")
	57	+
	58	+ async def go(): # type: ignore[no-untyped-def]
	59	+ return await con.fetch(ConnectorContext(company={}), {"url": url, "config": {}}, fake_fetcher) # type: ignore[arg-type]
	60	+
	61	+ import asyncio
	62	+
	63	+ res = asyncio.run(go())
	64	+ assert res.headers["x-companyatlas-pages"] == "2" # 25 total, 20 per page → 2 POSTs
	65	+ assert fake_fetcher.calls.count(f"POST {url}") == 2
	66	+ ex = con.extract({"url": url, "config": {}}, res)
	67	+ assert ex.jobs and ex.jobs[0].external_id == "JR2017846"
	68	+ assert ex.jobs[0].url.endswith("_JR2017846")
	69	+
	70	+
	71	+def test_smartrecruiters_pagination(fake_fetcher: FakeFetcher) -> None:
	72	+ base = "https://api.smartrecruiters.com/v1/companies/acme/postings?limit=100"
	73	+ p1 = {"offset": 0, "limit": 100, "totalFound": 2, "content": [{"id": "1", "name": "A", "location": {"city": "Berlin", "country": "de"}}]}
	74	+ p2 = {"offset": 1, "limit": 100, "totalFound": 2, "content": [{"id": "2", "name": "B", "location": {"city": "Paris", "country": "fr"}}]}
	75	+ fake_fetcher.add(base, json.dumps(p1), content_type="application/json")
	76	+ fake_fetcher.add(base + "&offset=1", json.dumps(p2), content_type="application/json")
	77	+ import asyncio
	78	+
	79	+ con = C.get("smartrecruiters-v1")
	80	+ res = asyncio.run(con.fetch(ConnectorContext(company={}), {"url": base, "config": {"token": "acme"}}, fake_fetcher)) # type: ignore[arg-type]
	81	+ ex = con.extract({"url": base, "config": {"token": "acme"}}, res)
	82	+ assert [j.title for j in ex.jobs] == ["A", "B"] and ex.jobs[1].country == "FR"
	83	+ assert ex.jobs[0].url == "https://jobs.smartrecruiters.com/acme/1"
	84	+
	85	+
	86	+def test_statuspage_and_feed_and_sitemap() -> None:
	87	+ sp = C.get("statuspage-v1").extract({"url": "https://www.githubstatus.com/api/v2/summary.json", "config": {}},
	88	+ file_result(fixture_path("statuspage", "github_summary.json"), url="x", content_type="application/json"))
	89	+ assert sp.meta["indicator"] == "none" and sp.meta["component_count"] >= 5 and sp.blocks[0].kind == "hero"
	90	+ fd = C.get("feed-v1").extract({"url": "https://www.acme-cloud.example/blog/atom.xml", "surface": "feed", "config": {}},
	91	+ file_result(fixture_path("feed", "blog_atom.xml"), url="x", content_type="application/atom+xml"))
	92	+ assert fd.news[0].title == "Introducing Atlas AI" and fd.news[0].published_at.year == 2026 and fd.news[0].category == "blog"
	93	+ assert len(fd.blocks) == 3
	94	+ sm = C.get("sitemap-v1").extract({"url": "https://www.acme-cloud.example/sitemap-pages.xml", "surface": "sitemap", "config": {"canonical_domain": "acme-cloud.example"}},
	95	+ file_result(fixture_path("sitemap", "sitemap_pages.xml"), url="x", content_type="application/xml"))
	96	+ assert sm.meta["url_count"] == 10 # the .png is dropped
	97	+ surfaces = {d.surface.value for d in sm.discovered}
	98	+ assert {"pricing", "careers", "leadership", "locations", "newsroom", "changelog"} <= surfaces
	99	+ idx = C.get("sitemap-v1")
	100	+ from companyatlas.connectors.sitemap import parse_sitemap
	101	+
	102	+ pages, children = parse_sitemap(fixture_bytes("sitemap", "sitemap_index.xml").decode())
	103	+ assert not pages and len(children) == 2 and children[0][1].startswith("2026-09-10")
	104	+ assert idx.meta.supports_discovery
	105	+
	106	+
	107	+def test_feed_json_feed_variant() -> None:
	108	+ body = json.dumps({"version": "https://jsonfeed.org/version/1.1", "title": "T", "items": [{"id": "1", "title": "Hello world post", "url": "https://x.example/p/1",
	109	+ "date_published": "2026-01-02T00:00:00Z", "summary": "s"}]})
	110	+ ex = C.get("feed-v1").extract({"url": "https://x.example/feed.json", "surface": "blog", "config": {}}, make_result("https://x.example/feed.json", body, content_type="application/feed+json"))
	111	+ assert ex.news[0].title == "Hello world post" and ex.news[0].category == "blog"

added tests/test_connectors_generic.py +96 −0

		`@@ -0,0 +1,96 @@`
	1	+from __future__ import annotations
	2	+
	3	+from conftest import fixture_path
	4	+
	5	+from companyatlas.fetch import file_result
	6	+from companyatlas.sdk import connector as C
	7	+
	8	+BASE = "https://www.acme-cloud.example"
	9	+CFG = {"canonical_domain": "acme-cloud.example"}
	10	+
	11	+
	12	+def _run(surface: str, name: str, path: str): # type: ignore[no-untyped-def]
	13	+ url = BASE + path
	14	+ return C.get("generic-html-v1").extract({"url": url, "surface": surface, "config": CFG}, file_result(fixture_path("generic_html", name), url=url))
	15	+
	16	+
	17	+def test_pricing_plans() -> None:
	18	+ ex = _run("pricing", "pricing.html", "/pricing")
	19	+ plans = {p.plan_name: p for p in ex.plans}
	20	+ assert set(plans) == {"Starter", "Pro", "Enterprise"}
	21	+ assert plans["Starter"].price == 29 and plans["Starter"].currency == "USD" and plans["Starter"].billing_period == "month"
	22	+ assert plans["Pro"].unit == "user" and "SSO & SAML" in plans["Pro"].features
	23	+ assert plans["Enterprise"].contact_sales and plans["Enterprise"].billing_period == "contact" and plans["Enterprise"].price is None
	24	+ assert {d.surface.value for d in ex.discovered} >= {"careers", "docs", "legal_terms", "feed"}
	25	+
	26	+
	27	+def test_leadership_people() -> None:
	28	+ ex = _run("leadership", "leadership.html", "/about/leadership")
	29	+ people = {p.name: p for p in ex.people}
	30	+ assert people["Jane Doe"].role_category == "ceo" and people["Jane Doe"].is_executive
	31	+ assert people["María García-López"].role_category == "cto"
	32	+ assert people["Tom O'Neill"].role_category == "vp" and not people["Tom O'Neill"].is_executive
	33	+ assert people["Samuel Adebayo"].role_category == "chair" and people["Lin Wei"].role_category == "board"
	34	+ assert "Meet the people" not in people
	35	+
	36	+
	37	+def test_locations_never_invent() -> None:
	38	+ ex = _run("locations", "locations.html", "/company/locations")
	39	+ locs = {loc.name: loc for loc in ex.locations}
	40	+ assert locs["San Francisco (Headquarters)"].kind == "headquarters" and locs["San Francisco (Headquarters)"].country == "US"
	41	+ assert locs["San Francisco (Headquarters)"].address_text.startswith("548 Market Street")
	42	+ assert locs["Toronto"].country == "CA" and locs["Toronto"].region == "ON" and locs["Toronto"].address_text is None
	43	+ assert locs["Dublin Data Center"].kind == "data_center" and locs["Dublin Data Center"].country == "IE"
	44	+ assert locs["London"].city == "London" and locs["London"].country == "GB"
	45	+
	46	+
	47	+def test_newsroom_items_with_dates() -> None:
	48	+ ex = _run("newsroom", "newsroom.html", "/news")
	49	+ assert len(ex.news) == 4
	50	+ assert ex.news[0].title.startswith("Acme launches Atlas AI") and ex.news[0].published_at.date().isoformat() == "2026-09-10"
	51	+ assert ex.news[1].published_at.date().isoformat() == "2026-08-28" # from <time datetime>
	52	+ assert all(n.category == "press" for n in ex.news)
	53	+ assert all(not n.title.startswith(("September", "August")) for n in ex.news)
	54	+
	55	+
	56	+def test_careers_html_jobs_and_removed_semantics() -> None:
	57	+ ex = _run("careers", "careers.html", "/careers")
	58	+ titles = [j.title for j in ex.jobs]
	59	+ assert titles == ["Senior Backend Engineer", "Machine Learning Engineer", "Account Executive, EMEA", "People Operations Lead", "Product Designer"]
	60	+ ml = ex.jobs[1]
	61	+ assert ml.remote is True and ml.country == "US" and ml.department == "Engineering"
	62	+ assert ex.jobs[0].city == "Toronto" and ex.jobs[0].country == "CA" and ex.jobs[0].seniority == "senior"
	63	+ assert ex.jobs[4].country is None # "San Francisco, CA" — no guess
	64	+ assert [b.kind for b in ex.blocks].count("job_listing") == 5
	65	+ ex2 = _run("careers", "careers_v2.html", "/careers")
	66	+ assert "AI Research Scientist" in [j.title for j in ex2.jobs] and "People Operations Lead" not in [j.title for j in ex2.jobs]
	67	+
	68	+
	69	+def test_careers_jsonld_jobs() -> None:
	70	+ ex = _run("careers", "careers_jsonld.html", "/jobs")
	71	+ assert [j.external_id for j in ex.jobs] == ["REQ-501", "REQ-502"]
	72	+ assert ex.jobs[0].country == "CA" and ex.jobs[0].city == "Toronto"
	73	+ assert ex.jobs[1].remote is True and ex.jobs[1].salary_min == 180000 and ex.jobs[1].salary_currency == "USD"
	74	+ via_connector = C.get("jsonld-jobs-v1").extract({"url": BASE + "/jobs", "surface": "careers", "config": CFG}, file_result(fixture_path("generic_html", "careers_jsonld.html"), url=BASE + "/jobs"))
	75	+ assert len(via_connector.jobs) == 2 and via_connector.meta["structured"]
	76	+
	77	+
	78	+def test_non_listing_page_yields_no_jobs() -> None:
	79	+ ex = _run("careers", "legal_terms.html", "/careers")
	80	+ assert ex.jobs == [] # < 3 job-like anchors → untrusted → nothing
	81	+
	82	+
	83	+def test_homepage_products_and_discovery() -> None:
	84	+ home = _run("homepage", "homepage.html", "/")
	85	+ assert home.title.startswith("Acme Cloud") and home.language == "en"
	86	+ found = {d.surface.value for d in home.discovered}
	87	+ assert {"pricing", "careers", "newsroom", "leadership", "locations", "investor_relations", "changelog", "legal_terms", "legal_privacy", "feed", "docs"} <= found
	88	+ assert all(d.confidence >= 0.55 for d in home.discovered)
	89	+ prods = _run("products", "homepage.html", "/")
	90	+ assert [p.name for p in prods.products] == ["Atlas Metrics", "Atlas Logs", "Atlas Traces"]
	91	+
	92	+
	93	+def test_legal_is_text_and_blocks_only() -> None:
	94	+ ex = _run("legal_terms", "legal_terms.html", "/legal/terms")
	95	+ assert not (ex.jobs or ex.people or ex.plans or ex.locations or ex.news or ex.products)
	96	+ assert "Limitation of liability" in ex.text and [b.kind for b in ex.blocks].count("heading") >= 5

added tests/test_diff.py +92 −0

		`@@ -0,0 +1,92 @@`
	1	+from __future__ import annotations
	2	+
	3	+from conftest import fixture_text
	4	+
	5	+from companyatlas.sdk import normalize as N
	6	+from companyatlas.sdk.diff import compare
	7	+from companyatlas.taxonomy import ChangeKind, change_kind
	8	+
	9	+URL = "https://www.acme-cloud.example/pricing"
	10	+
	11	+
	12	+def _page(name: str, url: str = URL) -> N.NormalizedPage:
	13	+ return N.parse(fixture_text("generic_html", name), url=url)
	14	+
	15	+
	16	+def test_same_page_twice_is_zero() -> None:
	17	+ a = _page("pricing.html")
	18	+ d = compare(a.blocks, a.blocks, surface="pricing", before_text=a.text, after_text=a.text)
	19	+ assert d.significance == 0.0 and d.is_empty and d.reasons == ["identical"]
	20	+
	21	+
	22	+def test_footer_year_change_is_noise() -> None:
	23	+ html = fixture_text("generic_html", "pricing.html")
	24	+ a, b = N.parse(html, url=URL), N.parse(html.replace("© 2025", "© 2026"), url=URL)
	25	+ d = compare(a.blocks, b.blocks, surface="pricing", before_text=a.text, after_text=b.text)
	26	+ assert d.significance < 0.20
	27	+ assert change_kind(d.significance) is ChangeKind.NOISE
	28	+
	29	+
	30	+def test_nav_only_churn_is_noise() -> None:
	31	+ html = fixture_text("generic_html", "pricing.html")
	32	+ b = N.parse(html.replace('<a href="/docs">Docs</a>', '<a href="/docs">Docs</a> <a href="/partners">Partners</a>'), url=URL)
	33	+ a = N.parse(html, url=URL)
	34	+ d = compare(a.blocks, b.blocks, surface="pricing", before_text=a.text, after_text=b.text)
	35	+ assert d.significance < 0.20
	36	+ assert any("nav/footer" in r for r in d.reasons)
	37	+
	38	+
	39	+def test_price_change_with_typed_delta_is_major() -> None:
	40	+ a, b = _page("pricing.html"), _page("pricing_v2.html")
	41	+ delta = {"plans": {"added": [], "removed": [], "price_changed": [{"plan_name": "Starter", "before": 29, "after": 39, "currency": "USD", "billing_period": "month", "pct": 34.5}]}}
	42	+ d = compare(a.blocks, b.blocks, surface="pricing", before_text=a.text, after_text=b.text, structured_delta=delta)
	43	+ assert d.significance >= 0.65
	44	+ assert len(d.modified) == 1 and d.modified[0].kind == "pricing_plan"
	45	+ assert any("typed delta plans.price_changed" in r for r in d.reasons)
	46	+
	47	+
	48	+def test_tiny_text_edit_without_typed_delta_is_noise() -> None:
	49	+ html = fixture_text("generic_html", "pricing.html")
	50	+ a = N.parse(html, url=URL)
	51	+ b = N.parse(html.replace("No credit card required.", "No credit card needed."), url=URL)
	52	+ d = compare(a.blocks, b.blocks, surface="pricing", before_text=a.text, after_text=b.text)
	53	+ assert d.significance < 0.20
	54	+
	55	+
	56	+def test_new_job_listings_are_meaningful() -> None:
	57	+ a = _page("careers.html", "https://www.acme-cloud.example/careers")
	58	+ b = _page("careers_v2.html", "https://www.acme-cloud.example/careers")
	59	+ delta = {"jobs": {"added": [{"title": "AI Research Scientist"}, {"title": "Data Platform Engineer"}], "removed": [{"title": "People Operations Lead"}],
	60	+ "open_before": 5, "open_after": 6}}
	61	+ d = compare(a.blocks, b.blocks, surface="careers", before_text=a.text, after_text=b.text, structured_delta=delta)
	62	+ assert change_kind(d.significance) in (ChangeKind.MEANINGFUL, ChangeKind.MAJOR)
	63	+ assert len(d.added) >= 2 and len(d.removed) >= 1
	64	+
	65	+
	66	+def test_whole_redesign_is_major() -> None:
	67	+ a = _page("legal_terms.html", "https://www.acme-cloud.example/")
	68	+ b = _page("homepage.html", "https://www.acme-cloud.example/")
	69	+ d = compare(a.blocks, b.blocks, surface="homepage", before_text=a.text, after_text=b.text)
	70	+ assert change_kind(d.significance) in (ChangeKind.MAJOR, ChangeKind.CRITICAL)
	71	+ assert d.text_delta_ratio > 0.5
	72	+
	73	+
	74	+def test_reordered_blocks_are_moves_not_add_remove() -> None:
	75	+ html = fixture_text("generic_html", "legal_terms.html")
	76	+ a = N.parse(html, url=URL)
	77	+ sec2 = html[html.index("<h2>2. Accounts</h2>"):html.index("<h2>3. Fees")]
	78	+ sec3 = html[html.index("<h2>3. Fees"):html.index("<h2>4. Termination")]
	79	+ b = N.parse(html.replace(sec2 + sec3, sec3 + sec2), url=URL)
	80	+ d = compare(a.blocks, b.blocks, surface="legal_terms", before_text=a.text, after_text=b.text)
	81	+ assert not d.added and not d.removed and not d.modified
	82	+ assert d.moved
	83	+
	84	+
	85	+def test_novelty_boost_and_churn_penalty_are_deterministic() -> None:
	86	+ a = _page("careers.html", "https://www.acme-cloud.example/careers")
	87	+ b = _page("careers_v2.html", "https://www.acme-cloud.example/careers")
	88	+ base = compare(a.blocks, b.blocks, surface="careers", before_text=a.text, after_text=b.text).significance
	89	+ novel = compare(a.blocks, b.blocks, surface="careers", before_text=a.text, after_text=b.text, history={"consecutive_unchanged": 50}).significance
	90	+ churn = compare(a.blocks, b.blocks, surface="careers", before_text=a.text, after_text=b.text, history={"observation_count": 20, "change_count": 15}).significance
	91	+ assert novel > base > churn
	92	+ assert base == compare(a.blocks, b.blocks, surface="careers", before_text=a.text, after_text=b.text).significance

added tests/test_discovery.py +79 −0

		`@@ -0,0 +1,79 @@`
	1	+from __future__ import annotations
	2	+
	3	+import asyncio
	4	+
	5	+import pytest
	6	+from conftest import FakeFetcher, fixture_bytes, fixture_text
	7	+
	8	+from companyatlas.services import discovery as D
	9	+from companyatlas.taxonomy import OnboardingStatus, Surface
	10	+
	11	+BASE = "https://www.acme-cloud.example"
	12	+COMPANY = {"id": "co_test", "slug": "ztest-acme", "display_name": "Acme Cloud", "canonical_domain": "acme-cloud.example", "website": BASE + "/", "tier": 2, "importance": 0.5}
	13	+
	14	+
	15	+def _fake_site(ff: FakeFetcher) -> None:
	16	+ ff.add(BASE + "/", fixture_text("generic_html", "homepage.html"))
	17	+ ff.add(BASE + "/robots.txt", "User-agent: *\nAllow: /\nSitemap: https://www.acme-cloud.example/sitemap.xml\n", content_type="text/plain")
	18	+ ff.add(BASE + "/sitemap.xml", fixture_bytes("sitemap", "sitemap_index.xml"), content_type="application/xml")
	19	+ ff.add(BASE + "/sitemap-pages.xml", fixture_bytes("sitemap", "sitemap_pages.xml"), content_type="application/xml")
	20	+ ff.add(BASE + "/sitemap-blog.xml", "<urlset></urlset>", content_type="application/xml")
	21	+ ff.add(BASE + "/careers", fixture_text("generic_html", "careers.html"))
	22	+ ff.add(BASE + "/pricing", fixture_text("generic_html", "pricing.html"))
	23	+ ff.add("https://boards-api.greenhouse.io/v1/boards/acmecloud/jobs?content=false", fixture_bytes("greenhouse", "stripe_jobs.json"), content_type="application/json")
	24	+
	25	+
	26	+@pytest.fixture(autouse=True)
	27	+def _no_dns(monkeypatch: pytest.MonkeyPatch) -> None:
	28	+ async def never(host: str) -> bool:
	29	+ return False
	30	+ monkeypatch.setattr(D, "_resolves", never)
	31	+
	32	+
	33	+def test_discover_company_dry_run(fake_fetcher: FakeFetcher) -> None:
	34	+ _fake_site(fake_fetcher)
	35	+ res = asyncio.run(D.discover_company(COMPANY, fetcher=fake_fetcher, dry_run=True)) # type: ignore[arg-type]
	36	+ assert res.status == OnboardingStatus.ACTIVE, res.error
	37	+ surfaces = {s["surface"]: s for s in res.sensors}
	38	+ assert {"homepage", "pricing", "careers", "jobs_board", "newsroom", "leadership", "locations", "sitemap", "legal_terms", "legal_privacy", "feed"} <= set(surfaces)
	39	+ gh = surfaces["jobs_board"]
	40	+ assert gh["connector_id"] == "greenhouse-v1" and gh["config"]["token"] == "acmecloud" and gh["config"].get("verified_job_count") == 20
	41	+ assert surfaces["careers"]["url"] == BASE + "/careers" and surfaces["careers"]["connector_id"] == "generic-html-v1"
	42	+ assert surfaces["sitemap"]["connector_id"] == "sitemap-v1"
	43	+ assert surfaces["feed"]["connector_id"] == "feed-v1"
	44	+ # tier 2 → ×0.75 on the surface base interval, clamped
	45	+ assert surfaces["pricing"]["base_interval_s"] == int(12 * 3600 * 0.75)
	46	+ assert surfaces["homepage"]["tier"] in ("C", "D") and 0 < surfaces["pricing"]["quality_score"] <= 100
	47	+ assert all(s["status"] == "pending" for s in res.sensors)
	48	+ assert len(res.sensors) <= 40 and len({s["canonical_url"] for s in res.sensors}) == len(res.sensors)
	49	+ # bounded politeness: no more than ~25 requests for a whole company
	50	+ assert res.requests <= 25
	51	+ assert all(c.startswith(("https://www.acme-cloud.example", "https://boards-api.greenhouse.io")) for c in fake_fetcher.calls)
	52	+
	53	+
	54	+def test_discover_no_website(fake_fetcher: FakeFetcher) -> None:
	55	+ from companyatlas.fetch import FetchError
	56	+ from companyatlas.taxonomy import FailureClass
	57	+
	58	+ for u in (BASE + "/", "https://www.acme-cloud.example/", "https://acme-cloud.example/", "http://www.acme-cloud.example/", "http://acme-cloud.example/"):
	59	+ fake_fetcher.add_error(u, FetchError("dns", url=u, failure=FailureClass.DNS))
	60	+ res = asyncio.run(D.discover_company(COMPANY, fetcher=fake_fetcher, dry_run=True)) # type: ignore[arg-type]
	61	+ assert res.status == OnboardingStatus.NO_WEBSITE and not res.sensors
	62	+
	63	+
	64	+def test_redirect_to_other_domain_is_recorded(fake_fetcher: FakeFetcher) -> None:
	65	+ fake_fetcher.add(BASE + "/", fixture_text("generic_html", "homepage.html"), final_url="https://www.acme-group.example/")
	66	+ fake_fetcher.add("https://www.acme-group.example/robots.txt", "", content_type="text/plain")
	67	+ res = asyncio.run(D.discover_company(COMPANY, fetcher=fake_fetcher, dry_run=True)) # type: ignore[arg-type]
	68	+ assert res.redirect_domain == "acme-group.example" and res.canonical_domain == "acme-group.example"
	69	+ assert any("redirects" in n for n in res.notes)
	70	+
	71	+
	72	+def test_select_sensors_caps_and_prefers_best_candidate() -> None:
	73	+ from datetime import UTC, datetime
	74	+
	75	+ cands = [D.Candidate(url=BASE + "/pricing", surface=str(Surface.PRICING), confidence=0.7, method="sitemap"),
	76	+ D.Candidate(url=BASE + "/plans", surface=str(Surface.PRICING), confidence=0.95, method="nav", verified=True),
	77	+ D.Candidate(url=BASE + "/x", surface=str(Surface.OTHER), confidence=0.3, method="nav")]
	78	+ rows = D.select_sensors(cands, company=COMPANY, canonical_domain="acme-cloud.example", now=datetime.now(UTC))
	79	+ assert len(rows) == 1 and rows[0]["url"] == BASE + "/plans" and rows[0]["discovery_method"] == "nav"

added tests/test_normalize.py +80 −0

		`@@ -0,0 +1,80 @@`
	1	+from __future__ import annotations
	2	+
	3	+from conftest import fixture_text
	4	+
	5	+from companyatlas.sdk import normalize as N
	6	+
	7	+
	8	+def test_noise_normalisation_replaces_dates_times_counters_tokens() -> None:
	9	+ s = "Updated 3 minutes ago on 2024-05-01 at 10:34 PM · March 5, 2024 · © 2024 Acme · 1,234 views · token=abcdef0123456789abcdef0123456789 ?v=123&x=1"
	10	+ n = N.normalized_text(s)
	11	+ assert "<rel>" in n and "<date>" in n and "<time>" in n and "<count>" in n and "<copyright>" in n and "<hex>" in n and "?v=<q>" in n
	12	+ assert "2024" not in n.replace("<date>", "")
	13	+
	14	+
	15	+def test_text_hash_ignores_noise_but_not_prices() -> None:
	16	+ assert N.text_hash("Price $29 per month, updated 2 days ago") == N.text_hash("Price $29 per month, updated 5 hours ago")
	17	+ assert N.text_hash("Price $29 per month") != N.text_hash("Price $39 per month")
	18	+
	19	+
	20	+def test_simhash_near_duplicates_and_hamming() -> None:
	21	+ a = N.simhash("The quick brown fox jumps over the lazy dog near the river bank today")
	22	+ b = N.simhash("The quick brown fox jumps over the lazy dog near the river bank tonight")
	23	+ c = N.simhash("Completely different content about pricing plans and enterprise contracts")
	24	+ assert N.hamming(a, b) < N.hamming(a, c)
	25	+ assert N.simhash("") == 0
	26	+
	27	+
	28	+def test_parse_pricing_page_blocks_meta_links() -> None:
	29	+ page = N.parse(fixture_text("generic_html", "pricing.html"), url="https://www.acme-cloud.example/pricing")
	30	+ assert page.title == "Pricing — Acme Cloud"
	31	+ assert page.lang == "en"
	32	+ assert page.meta["description"].startswith("Simple")
	33	+ assert page.meta["canonical"] == "https://www.acme-cloud.example/pricing"
	34	+ assert page.feeds == ["https://www.acme-cloud.example/blog/feed.xml"]
	35	+ kinds = [b.kind for b in page.blocks]
	36	+ assert kinds.count("pricing_plan") == 3
	37	+ assert "hero" in kinds and "nav" in kinds and "footer" in kinds and "faq" in kinds
	38	+ # script/style/cookie banner/hidden content dropped
	39	+ assert "dataLayer" not in page.text and "We use cookies" not in page.text and "Last updated" not in page.text
	40	+ # nav/footer are low weight, pricing plans high
	41	+ assert max(b.weight for b in page.blocks if b.kind == "pricing_plan") > max(b.weight for b in page.blocks if b.kind in ("nav", "footer"))
	42	+ regions = {ln.region for ln in page.links}
	43	+ assert {"nav", "footer", "main"} <= regions
	44	+ assert page.main_selector == "main"
	45	+
	46	+
	47	+def test_block_keys_are_stable_under_reordering() -> None:
	48	+ html = fixture_text("generic_html", "pricing.html")
	49	+ a = N.parse(html, url="https://x.example/pricing")
	50	+ # swap the Starter and Pro cards
	51	+ swapped = html.replace('<h3>Starter</h3>', '<h3>TMP</h3>').replace('<h3>Pro</h3>', '<h3>Starter</h3>').replace('<h3>TMP</h3>', '<h3>Pro</h3>')
	52	+ b = N.parse(swapped, url="https://x.example/pricing")
	53	+ keys_a = {blk.key for blk in a.blocks if blk.kind == "pricing_plan"}
	54	+ keys_b = {blk.key for blk in b.blocks if blk.kind == "pricing_plan"}
	55	+ # names moved but content per card changed, so at least the Enterprise card key is identical
	56	+ assert keys_a & keys_b
	57	+ assert N.structural_hash(a.blocks) != N.structural_hash(b.blocks) or [x.key for x in a.blocks] == [x.key for x in b.blocks]
	58	+
	59	+
	60	+def test_jsonld_and_microdata_extraction() -> None:
	61	+ page = N.parse(fixture_text("generic_html", "leadership.html"), url="https://www.acme-cloud.example/about/leadership")
	62	+ assert "organizations" in page.jsonld
	63	+ assert page.meta["same_as"][0].startswith("https://www.linkedin.com/")
	64	+ assert page.jsonld["persons"][0]["name"] == "Jane Doe"
	65	+ micro = N.extract_microdata(N.LexborHTMLParser('<div itemscope itemtype="https://schema.org/Person"><span itemprop="name">Ada Lovelace</span><span itemprop="jobTitle">CTO</span></div>'))
	66	+ assert micro["persons"][0]["name"] == "Ada Lovelace"
	67	+
	68	+
	69	+def test_language_guess() -> None:
	70	+ assert N.language_guess("whatever", "fr-CA") == "fr"
	71	+ assert N.language_guess("Nous sommes une entreprise qui offre des services pour les clients et les partenaires dans le monde") == "fr"
	72	+ assert N.language_guess("We are a company that offers services for customers and partners in the world with our team") == "en"
	73	+ assert N.language_guess("これは日本語のテキストです") == "ja"
	74	+
	75	+
	76	+def test_card_containers_are_split_into_children() -> None:
	77	+ page = N.parse(fixture_text("generic_html", "newsroom.html"), url="https://www.acme-cloud.example/news")
	78	+ assert [b.kind for b in page.blocks].count("news_item") == 4
	79	+ home = N.parse(fixture_text("generic_html", "homepage.html"), url="https://www.acme-cloud.example/")
	80	+ assert [b.kind for b in home.blocks].count("product_card") == 3

added tests/test_pipeline.py +190 −0

		`@@ -0,0 +1,190 @@`
	1	+"""Integration: the pipeline against the local Postgres with a temporary `ztest-` company (skipped when the DB is unavailable)."""
	2	+from __future__ import annotations
	3	+
	4	+import asyncio
	5	+from typing import Any
	6	+
	7	+import pytest
	8	+from conftest import FakeFetcher, fixture_path
	9	+
	10	+from companyatlas.fetch import FetchError, NotModified, file_result
	11	+from companyatlas.taxonomy import FailureClass
	12	+
	13	+BASE = "https://www.acme-cloud.example"
	14	+
	15	+
	16	+async def _setup() -> tuple[str, dict[str, str]]:
	17	+ from companyatlas.db import execute, jsonb, transaction
	18	+ from companyatlas.ids import new_id
	19	+
	20	+ cid = new_id("company")
	21	+ slug = f"ztest-pipe-{cid[-6:].lower()}"
	22	+ dom = f"{slug}.example"
	23	+ sensors: dict[str, str] = {}
	24	+ async with transaction() as conn:
	25	+ await execute(conn, """insert into companies (id, slug, display_name, canonical_domain, website, tier, importance, onboarding_status)
	26	+ values (:id, :slug, 'Pipeline test', :dom, :web, 3, 0.4, 'active')""", id=cid, slug=slug, dom=dom, web=f"https://{dom}/")
	27	+ for surface, url, con in [("pricing", BASE + "/pricing", "generic-html-v1"), ("careers", BASE + "/careers", "generic-html-v1"),
	28	+ ("jobs_board", "https://boards-api.greenhouse.io/v1/boards/ztest/jobs?content=false", "greenhouse-v1"),
	29	+ ("leadership", BASE + "/about/leadership", "generic-html-v1"), ("locations", BASE + "/company/locations", "generic-html-v1"),
	30	+ ("newsroom", BASE + "/news", "generic-html-v1"), ("legal_terms", BASE + "/legal/terms", "generic-html-v1")]:
	31	+ sid = new_id("sensor")
	32	+ await execute(conn, """insert into sensors (id, company_id, surface, connector_id, url, canonical_url, domain, base_interval_s, current_interval_s, status, config)
	33	+ values (:id, :c, :s, :con, :u, :u, :d, 43200, 43200, 'pending', cast(:cfg as jsonb))""",
	34	+ id=sid, c=cid, s=surface, con=con, u=url, d=dom, cfg=jsonb({"canonical_domain": "acme-cloud.example", "token": "ztest"}))
	35	+ sensors[surface] = sid
	36	+ return cid, sensors
	37	+
	38	+
	39	+async def _teardown(cid: str) -> None:
	40	+ from companyatlas.db import dispose, execute, transaction
	41	+
	42	+ async with transaction() as conn:
	43	+ await execute(conn, "delete from companies where id = :c", c=cid)
	44	+ await execute(conn, "delete from domain_budgets where domain like 'ztest-%'")
	45	+ await dispose()
	46	+
	47	+
	48	+async def _sensor(sid: str) -> dict[str, Any]:
	49	+ from companyatlas.db import fetch_one, transaction
	50	+
	51	+ async with transaction() as conn:
	52	+ row = await fetch_one(conn, "select * from sensors where id = :id", id=sid)
	53	+ assert row is not None
	54	+ return row
	55	+
	56	+
	57	+async def _run(sid: str, path: str, ct: str = "text/html", fetcher: Any = None): # type: ignore[no-untyped-def]
	58	+ from companyatlas.services.pipeline import run_sensor
	59	+
	60	+ row = await _sensor(sid)
	61	+ return await run_sensor(row, fetcher=fetcher or FakeFetcher(), worker="pytest", result=file_result(path, url=row["url"], content_type=ct))
	62	+
	63	+
	64	+async def _q(sql: str, **params: Any) -> list[dict[str, Any]]:
	65	+ from companyatlas.db import fetch_all, transaction
	66	+
	67	+ async with transaction() as conn:
	68	+ return await fetch_all(conn, sql, **params)
	69	+
	70	+
	71	+@pytest.mark.usefixtures("db")
	72	+def test_pipeline_end_to_end() -> None:
	73	+ async def scenario() -> None:
	74	+ cid, s = await _setup()
	75	+ try:
	76	+ gh = fixture_path("greenhouse", "stripe_jobs.json")
	77	+ # ---- baseline runs (version 1, no change rows, entities inserted)
	78	+ o = await _run(s["pricing"], fixture_path("generic_html", "pricing.html"))
	79	+ assert o.status == "ok" and o.snapshot_id and not o.change_id and o.delta_counts == {"plans_added": 3}
	80	+ # ---- same content again → unchanged, no snapshot, interval grows ×1.25
	81	+ o = await _run(s["pricing"], fixture_path("generic_html", "pricing.html"))
	82	+ assert o.status == "unchanged" and o.snapshot_id is None and o.interval_s == int(43200 * 1.25)
	83	+ # ---- price change → new snapshot version, change row pending, plan superseded, burst interval
	84	+ o = await _run(s["pricing"], fixture_path("generic_html", "pricing_v2.html"))
	85	+ assert o.status == "changed" and o.kind in ("major", "critical") and o.change_id and o.delta_counts.get("plans_price_changed") == 1
	86	+ assert o.interval_s == 900
	87	+ plans = await _q("select plan_name, price, status, version_no from pricing_plans where company_id = :c order by plan_name, version_no", c=cid)
	88	+ starter = [p for p in plans if p["plan_name"] == "Starter"]
	89	+ assert [(int(p["price"]), p["status"], p["version_no"]) for p in starter] == [(29, "superseded", 1), (39, "current", 2)]
	90	+ chg = (await _q("select * from changes where company_id = :c and surface = 'pricing'", c=cid))[0]
	91	+ assert chg["status"] == "pending" and chg["structured_delta"]["plans"]["price_changed"][0]["after"] == 39 and chg["diff"]["counts"]["modified"] == 1
	92	+ snaps = await _q("select version_no, previous_snapshot_id from snapshots where sensor_id = :s order by version_no", s=s["pricing"])
	93	+ assert [x["version_no"] for x in snaps] == [1, 2] and snaps[1]["previous_snapshot_id"] == (await _q("select id from snapshots where sensor_id = :s and version_no = 1", s=s["pricing"]))[0]["id"]
	94	+ # ---- careers: jobs added / removed with no_longer_listed
	95	+ await _run(s["careers"], fixture_path("generic_html", "careers.html"))
	96	+ o = await _run(s["careers"], fixture_path("generic_html", "careers_v2.html"))
	97	+ assert o.delta_counts == {"jobs_added": 2, "jobs_removed": 1}
	98	+ jobs = await _q("select title, status, removed_at, is_ai from jobs where sensor_id = :s order by title", s=s["careers"])
	99	+ by = {j["title"]: j for j in jobs}
	100	+ assert by["People Operations Lead"]["status"] == "no_longer_listed" and by["People Operations Lead"]["removed_at"] is not None
	101	+ assert by["AI Research Scientist"]["is_ai"] and by["AI Research Scientist"]["status"] == "open"
	102	+ assert len(jobs) == 7
	103	+ # ---- untrusted extraction never mass-removes: a page with no job listing leaves the 6 open jobs alone
	104	+ o = await _run(s["careers"], fixture_path("generic_html", "legal_terms.html"))
	105	+ open_after = await _q("select count(*) n from jobs where sensor_id = :s and status = 'open'", s=s["careers"])
	106	+ assert open_after[0]["n"] == 6
	107	+ # ---- structured board: baseline + unchanged
	108	+ o = await _run(s["jobs_board"], gh, "application/json")
	109	+ assert o.status == "ok" and o.delta_counts == {"jobs_added": 20}
	110	+ o = await _run(s["jobs_board"], gh, "application/json")
	111	+ assert o.status == "unchanged"
	112	+ # ---- other surfaces baseline
	113	+ for surf, fn in (("leadership", "leadership.html"), ("locations", "locations.html"), ("newsroom", "newsroom.html"), ("legal_terms", "legal_terms.html")):
	114	+ o = await _run(s[surf], fixture_path("generic_html", fn))
	115	+ assert o.status == "ok", (surf, o.error)
	116	+ assert (await _q("select count(*) n from people where company_id = :c and is_executive", c=cid))[0]["n"] == 4
	117	+ assert (await _q("select count(*) n from locations where company_id = :c and country = 'US'", c=cid))[0]["n"] == 2
	118	+ assert (await _q("select count(*) n from news_items where company_id = :c", c=cid))[0]["n"] == 4
	119	+ # ---- failure path: fetch error → observation + failures row + backoff; repeated → failing
	120	+ from companyatlas.services.pipeline import run_sensor
	121	+
	122	+ ff = FakeFetcher()
	123	+ ff.add_error(BASE + "/legal/terms", FetchError("http 404", status=404, url=BASE + "/legal/terms", failure=FailureClass.PAGE_REMOVED))
	124	+ row = await _sensor(s["legal_terms"])
	125	+ o = await run_sensor(row, fetcher=ff, worker="pytest") # type: ignore[arg-type]
	126	+ assert o.status == "failed" and o.failure_class == "PAGE_REMOVED" and o.interval_s == min(7 * 86400, int(row["current_interval_s"] * 4.0))
	127	+ o = await run_sensor(await _sensor(s["legal_terms"]), fetcher=ff, worker="pytest") # type: ignore[arg-type]
	128	+ assert o.sensor_status == "failing"
	129	+ fails = await _q("select failure_class from failures where sensor_id = :s", s=s["legal_terms"])
	130	+ assert len(fails) == 2
	131	+ # ---- not modified path
	132	+ ff2 = FakeFetcher()
	133	+ ff2.add_error(BASE + "/news", NotModified(5))
	134	+ o = await run_sensor(await _sensor(s["newsroom"]), fetcher=ff2, worker="pytest") # type: ignore[arg-type]
	135	+ assert o.status == "not_modified"
	136	+ obs = await _q("select not_modified from observations where sensor_id = :s order by fetched_at desc limit 1", s=s["newsroom"])
	137	+ assert obs[0]["not_modified"] is True
	138	+ # ---- robots block → review queue item
	139	+ ff3 = FakeFetcher()
	140	+ from companyatlas.fetch import BlockedError
	141	+
	142	+ ff3.add_error(BASE + "/company/locations", BlockedError("robots", url=BASE + "/company/locations", failure=FailureClass.ROBOTS))
	143	+ o = await run_sensor(await _sensor(s["locations"]), fetcher=ff3, worker="pytest") # type: ignore[arg-type]
	144	+ assert o.sensor_status == "blocked"
	145	+ assert (await _q("select count(*) n from review_queue where ref_id = :s and kind = 'blocked_source'", s=s["locations"]))[0]["n"] == 1
	146	+ # ---- company + ledger touched
	147	+ comp = (await _q("select stats, first_observed_at, last_change_at from companies where id = :c", c=cid))[0]
	148	+ assert comp["first_observed_at"] is not None and comp["last_change_at"] is not None and comp["stats"]["meaningful_changes"] >= 2
	149	+ led = await _q("select units from cost_ledger where day = current_date and dimension = 'fetch' and key = :c", c=cid)
	150	+ assert led and led[0]["units"] >= 10
	151	+ finally:
	152	+ await _teardown(cid)
	153	+
	154	+ asyncio.run(scenario())
	155	+
	156	+
	157	+@pytest.mark.usefixtures("db")
	158	+def test_scheduler_claims_and_releases() -> None:
	159	+ async def scenario() -> None:
	160	+ from companyatlas.services.scheduler import claim_due_sensors, release_claims
	161	+
	162	+ cid, _sensors = await _setup()
	163	+ try:
	164	+ rows = await claim_due_sensors("pytest-worker", 100)
	165	+ mine = [r for r in rows if r["company_id"] == cid]
	166	+ assert len(mine) == 7 and all(r["claimed_by"] == "pytest-worker" for r in mine)
	167	+ again = await claim_due_sensors("pytest-worker-2", 100)
	168	+ assert not [r for r in again if r["company_id"] == cid] # already claimed
	169	+ await release_claims("pytest-worker")
	170	+ assert (await _q("select count(*) n from sensors where company_id = :c and claimed_by is null", c=cid))[0]["n"] == 7
	171	+ finally:
	172	+ await _teardown(cid)
	173	+
	174	+ asyncio.run(scenario())
	175	+
	176	+
	177	+@pytest.mark.usefixtures("db")
	178	+def test_connectors_table_sync() -> None:
	179	+ async def scenario() -> None:
	180	+ from companyatlas.db import dispose, transaction
	181	+ from companyatlas.sdk.connector import all_connectors, sync_connectors_table
	182	+
	183	+ async with transaction() as conn:
	184	+ n = await sync_connectors_table(conn)
	185	+ assert n == len(all_connectors()) >= 14
	186	+ rows = await _q("select id from connectors where id = 'generic-html-v1'")
	187	+ assert rows
	188	+ await dispose()
	189	+
	190	+ asyncio.run(scenario())
	191