spb/company-atlas

1# Connectors — families, SDK, fixtures, verified vendor endpoints23A **connector** is a strategy for one *kind* of public source; a **sensor** binds a connector to one URL for one company4(spec §8–9, §105–106). All connectors live in `src/companyatlas/connectors/`, register themselves with the SDK5(`sdk/connector.py`) and are auto-loaded (`connectors.load_all()`); `catlas connectors` lists them and upserts the `connectors` table.67## Families (as of `connector-v1` ids)89| id | family | picks when | output |10|---|---|---|---|11| `generic-html-v1` | any HTML surface | fallback for every surface (`surfaces=("*",)`) | text + semantic blocks + surface-typed lists: people (leadership/about), plans (pricing), locations (locations/contact), news (newsroom/blog/changelog/research/IR), jobs (careers, list/table/card + JSON-LD), products (products/services/solutions); `discovered` links on every page |12| `sitemap-v1` | `sitemap*.xml(.gz)`, `sitemap_index.xml`, `/sitemaps/` | URL pattern | URL set (bounded by `discovery_max_sitemap_urls`), classified `DiscoveredUrl`s, one block per URL (diff = new/gone URLs) |13| `feed-v1` | RSS / Atom / JSON Feed | URL pattern (`/feed`, `/rss`, `.xml`, `feed.json`) or surface `feed` | news items (title, url, published_at, summary, language) |14| `jsonld-jobs-v1` | JobPosting JSON-LD / microdata | non-ATS `jobs_board` URLs (or explicitly by discovery) | jobs |15| `greenhouse-v1` `lever-v1` `ashby-v1` `smartrecruiters-v1` `workable-v1` `workday-v1` `recruitee-v1` `personio-v1` `teamtailor-v1` | structured job boards | API URL pattern (set by discovery via `connectors/_util.ats_sensor_spec`) | jobs, one stable block per job |16| `statuspage-v1` | Atlassian Statuspage | `/api/v2/summary.json` | components / incidents as blocks + incident news |1718Connector selection: `connector.for_surface(surface, url)` scores every registered connector (`url_pattern` match +100, `category`19+20, listed `surfaces` +10, `"*"` +5, plus `priority`) and returns the best; the generic HTML connector is the floor.2021## Verified vendor endpoints (2026-09-12, one real board each, then saved as trimmed fixtures — tests never hit the network)2223| vendor | endpoint | verified board | fixture |24|---|---|---|---|25| Greenhouse | `GET https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=false` (+`/departments`, `/offices` optional) | `stripe` | `greenhouse/stripe_jobs.json` |26| Lever | `GET https://api.lever.co/v0/postings/{token}?mode=json` (`api.eu.lever.co` for EU boards) | `palantir` | `lever/palantir_postings.json` |27| Ashby | `GET https://api.ashbyhq.com/posting-api/job-board/{token}` | `ashby` | `ashby/ashby_board.json` |28| SmartRecruiters | `GET https://api.smartrecruiters.com/v1/companies/{token}/postings?limit=100&offset=N` (offset pagination, ≤ 10 pages) | `smartrecruiters` | `smartrecruiters/smartrecruiters_postings.json` |29| Workable | `GET https://apply.workable.com/api/v1/widget/accounts/{token}` | `epignosis` | `workable/epignosis_widget.json` |30| Workday | `POST https://{tenant}.{wdN}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs` body `{"appliedFacets":{},"limit":20,"offset":N,"searchText":""}` (bounded 400 jobs) — uses `Fetcher.post_json` (SSRF-checked, governor) | `nvidia.wd5 / NVIDIAExternalCareerSite` | `workday/nvidia_jobs_page1.json` |31| Recruitee | `GET https://{token}.recruitee.com/api/offers/` | `vandebron` | `recruitee/vandebron_offers.json` |32| Personio | `GET https://{token}.jobs.personio.de/xml` (`<workzag-jobs>`) | `personio` | `personio/personio_jobs.xml` |33| Teamtailor | `GET https://{career-site-host}/jobs.json` (JSON Feed 1.1 with `_jobposting`) — `{token}.teamtailor.com` boards that 404 keep the HTML careers sensor | `career.teamtailor.com` | `teamtailor/teamtailor_jobs_feed.json` |34| Statuspage | `GET https://status.<host>/api/v2/summary.json` | `githubstatus.com` | `statuspage/github_summary.json` |3536All of these are the endpoints the vendors' own public pages call (spec §13 mode C). Nothing needs a key; nothing bypasses a37challenge. Every request still goes through `fetch.Fetcher` (SSRF guard, robots.txt, per-domain governor, size caps).3839## Job normalisation (every board → `ExtractedJob`)4041`connectors/_util.finish_job` fills derived fields without overriding vendor data: `parse_location` (city / region / ISO-242country **only when stated** — "San Francisco, CA" keeps region `CA` and no country because `CA` is also Canada), remote /43hybrid, `seniority_guess`, `employment_type_norm`. The pipeline sets `is_ai` (`taxonomy.AI_KEYWORDS`) and `is_engineering`.44Job identity = `job_fingerprint(title, location_text, external_id | url)`; structured boards also give each job a stable block45key so the block diff mirrors the job delta.4647## Adding a connector family48491. Create `src/companyatlas/connectors/<family>.py`; subclass `Connector` (or `AtsConnector` for JSON/XML boards — implement50   `parse_jobs(data, sensor)` only) and decorate with `@register`. Declare `ConnectorMeta(connector_id="<family>-v1", category=Surface…,51   fetch_mode, url_pattern, priority, accept, default_interval_s)`. Override `fetch()` only for pagination / POST; never open52   sockets — use the `Fetcher` you receive (`get`, `request`, `post_json`).532. Verify the endpoint shape against **one** real public source (a handful of polite requests), then save the response trimmed to54   ~20 items under `fixtures/connectors/<family>/` (strip descriptions / personal data).553. Add tests in `tests/test_connectors_*.py` that load the fixture with `fetch.file_result` and assert counts, ids, urls, locations.56   Tests must not touch the network (`-m live` is opt-in only).574. If discovery should create sensors for it, extend `connectors/_util.ats_sensor_spec` (vendor → API URL + connector id + config)58   and the detection regexes in `urls.ATS_PATTERNS` / `services/discovery.EMBED_ATS_RE`.595. Behaviour change that alters extraction materially → new id (`<family>-v2`). Observations and snapshots store the id as60   `connector_version`, so history stays reproducible; `connector.get("<family>-v1")` keeps resolving to the newest of the family.6162## Fixture policy6364* Hand-written HTML for each generic surface lives in `fixtures/connectors/generic_html/` (`pricing.html` + `pricing_v2.html`,65  `careers.html` + `careers_v2.html`, `leadership.html`, `locations.html`, `newsroom.html`, `homepage.html`, `legal_terms.html`,66  `careers_jsonld.html`); `sitemap/` and `feed/` hold XML samples. Pairs (`*_v2`) exist so the diff engine and the pipeline can be67  tested for real change semantics (price change, jobs added/removed).68* Vendor fixtures are captured responses, trimmed; keep the top-level shape intact (pagination fields included).69* Regression rule (spec §148): when changing the normalizer, the diff engine or an extractor, run the suite — the fixtures are70  the historical behaviour contract.71