Connectors — families, SDK, fixtures, verified vendor endpoints
A connector is a strategy for one kind of public source; a sensor binds a connector to one URL for one company
(spec §8–9, §105–106). All connectors live in src/companyatlas/connectors/, register themselves with the SDK
(sdk/connector.py) and are auto-loaded (connectors.load_all()); catlas connectors lists them and upserts the connectors table.
Families (as of connector-v1 ids)
| id | family | picks when | output |
|---|---|---|---|
generic-html-v1 |
any HTML surface | fallback for every surface (surfaces=("*",)) |
text + semantic blocks + surface-typed lists: people (leadership/about), plans (pricing), locations (locations/contact), news (newsroom/blog/changelog/research/IR), jobs (careers, list/table/card + JSON-LD), products (products/services/solutions); discovered links on every page |
sitemap-v1 |
sitemap*.xml(.gz), sitemap_index.xml, /sitemaps/ |
URL pattern | URL set (bounded by discovery_max_sitemap_urls), classified DiscoveredUrls, one block per URL (diff = new/gone URLs) |
feed-v1 |
RSS / Atom / JSON Feed | URL pattern (/feed, /rss, .xml, feed.json) or surface feed |
news items (title, url, published_at, summary, language) |
jsonld-jobs-v1 |
JobPosting JSON-LD / microdata | non-ATS jobs_board URLs (or explicitly by discovery) |
jobs |
greenhouse-v1 lever-v1 ashby-v1 smartrecruiters-v1 workable-v1 workday-v1 recruitee-v1 personio-v1 teamtailor-v1 |
structured job boards | API URL pattern (set by discovery via connectors/_util.ats_sensor_spec) |
jobs, one stable block per job |
statuspage-v1 |
Atlassian Statuspage | /api/v2/summary.json |
components / incidents as blocks + incident news |
Connector selection: connector.for_surface(surface, url) scores every registered connector (url_pattern match +100, category
+20, listed surfaces +10, "*" +5, plus priority) and returns the best; the generic HTML connector is the floor.
Verified vendor endpoints (2026-09-12, one real board each, then saved as trimmed fixtures — tests never hit the network)
| vendor | endpoint | verified board | fixture |
|---|---|---|---|
| Greenhouse | GET https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=false (+/departments, /offices optional) |
stripe |
greenhouse/stripe_jobs.json |
| Lever | GET https://api.lever.co/v0/postings/{token}?mode=json (api.eu.lever.co for EU boards) |
palantir |
lever/palantir_postings.json |
| Ashby | GET https://api.ashbyhq.com/posting-api/job-board/{token} |
ashby |
ashby/ashby_board.json |
| SmartRecruiters | GET https://api.smartrecruiters.com/v1/companies/{token}/postings?limit=100&offset=N (offset pagination, ≤ 10 pages) |
smartrecruiters |
smartrecruiters/smartrecruiters_postings.json |
| Workable | GET https://apply.workable.com/api/v1/widget/accounts/{token} |
epignosis |
workable/epignosis_widget.json |
| Workday | POST https://{tenant}.{wdN}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs body {"appliedFacets":{},"limit":20,"offset":N,"searchText":""} (bounded 400 jobs) — uses Fetcher.post_json (SSRF-checked, governor) |
nvidia.wd5 / NVIDIAExternalCareerSite |
workday/nvidia_jobs_page1.json |
| Recruitee | GET https://{token}.recruitee.com/api/offers/ |
vandebron |
recruitee/vandebron_offers.json |
| Personio | GET https://{token}.jobs.personio.de/xml (<workzag-jobs>) |
personio |
personio/personio_jobs.xml |
| Teamtailor | GET https://{career-site-host}/jobs.json (JSON Feed 1.1 with _jobposting) — {token}.teamtailor.com boards that 404 keep the HTML careers sensor |
career.teamtailor.com |
teamtailor/teamtailor_jobs_feed.json |
| Statuspage | GET https://status.<host>/api/v2/summary.json |
githubstatus.com |
statuspage/github_summary.json |
All of these are the endpoints the vendors' own public pages call (spec §13 mode C). Nothing needs a key; nothing bypasses a
challenge. Every request still goes through fetch.Fetcher (SSRF guard, robots.txt, per-domain governor, size caps).
Job normalisation (every board → ExtractedJob)
connectors/_util.finish_job fills derived fields without overriding vendor data: parse_location (city / region / ISO-2
country only when stated — "San Francisco, CA" keeps region CA and no country because CA is also Canada), remote /
hybrid, seniority_guess, employment_type_norm. The pipeline sets is_ai (taxonomy.AI_KEYWORDS) and is_engineering.
Job identity = job_fingerprint(title, location_text, external_id | url); structured boards also give each job a stable block
key so the block diff mirrors the job delta.
Adding a connector family
- Create
src/companyatlas/connectors/<family>.py; subclassConnector(orAtsConnectorfor JSON/XML boards — implementparse_jobs(data, sensor)only) and decorate with@register. DeclareConnectorMeta(connector_id="<family>-v1", category=Surface…, fetch_mode, url_pattern, priority, accept, default_interval_s). Overridefetch()only for pagination / POST; never open sockets — use theFetcheryou receive (get,request,post_json). - Verify the endpoint shape against one real public source (a handful of polite requests), then save the response trimmed to
~20 items under
fixtures/connectors/<family>/(strip descriptions / personal data). - Add tests in
tests/test_connectors_*.pythat load the fixture withfetch.file_resultand assert counts, ids, urls, locations. Tests must not touch the network (-m liveis opt-in only). - If discovery should create sensors for it, extend
connectors/_util.ats_sensor_spec(vendor → API URL + connector id + config) and the detection regexes inurls.ATS_PATTERNS/services/discovery.EMBED_ATS_RE. - Behaviour change that alters extraction materially → new id (
<family>-v2). Observations and snapshots store the id asconnector_version, so history stays reproducible;connector.get("<family>-v1")keeps resolving to the newest of the family.
Fixture policy
- Hand-written HTML for each generic surface lives in
fixtures/connectors/generic_html/(pricing.html+pricing_v2.html,careers.html+careers_v2.html,leadership.html,locations.html,newsroom.html,homepage.html,legal_terms.html,careers_jsonld.html);sitemap/andfeed/hold XML samples. Pairs (*_v2) exist so the diff engine and the pipeline can be tested for real change semantics (price change, jobs added/removed). - Vendor fixtures are captured responses, trimmed; keep the top-level shape intact (pagination fields included).
- Regression rule (spec §148): when changing the normalizer, the diff engine or an extractor, run the suite — the fixtures are the historical behaviour contract.