# Adding a connector

This is the operational guide for building a production-grade RareIndex connector. A connector is not complete because it returns one listing (SPEC §35). It is complete when it has reliable discovery, pagination, normalisation, stable ids, error handling, rate limiting, retries, fixtures, tests, health, logging, persistence, a refresh class, source metadata and documentation.

# 0. Before writing code: the source registry

Every source starts as an entry in the source registry (schema: packages/connectors/src/sources.ts, SPEC §4). Write entries as an array in data/sources/entries/<your-group>.json; pnpm sources:build merges all fragments over data/sources/sources.json (generated, do not edit by hand) and regenerates SOURCES.md. Likewise, per-host policies go in connectors/domains.d/<your-group>.json as { "domains": { "host": {…} } }. Research and verify live (curl with the honest bot UA, then Firecrawl, then Scrapfly):

• robots.txt (adapters.parseRobots) — which paths are allowed for * / our UA; Crawl-delay.
• Terms of use URL; anything that forbids automated access of public pages → legal note; login walls, paywalls, CAPTCHAs protecting private data → status blocked, never circumvented (SPEC §36).
• Public API? Official docs URL, auth type. Prefer it over any HTML.
• Structured data: __NEXT_DATA__, JSON-LD Product (adapters.productsFromHtml), XHR JSON the page itself loads, sitemaps (adapters.discoverFromSitemap), RSS, bulk files, Shopify products.json, WooCommerce Store API.
• Pagination style, JS rendering requirement, anti-bot (Cloudflare/Akamai/PerimeterX), rate limits.
• Which fields exist: sold price, sale date, fees/buyer premium, seller, grading, cert number, SKU/UPC/EAN.

Record what you found (verified: true, verifiedAt). Set status honestly: implemented, partial, gated (needs a key we do not hold), planned, blocked, rejected (+ statusReason).

# 1. Scaffold

pnpm connector:new <id> --url https://www.example.com --name "Example" --engine api|firecrawl|scrapfly \
  [--adapter shopify|woocommerce] --type marketplace|auction_house|dealer|pricing_guide|grading_company|catalog \
  --categories pokemon,magic_the_gathering --country US --currency USD --kind listing|sale|auction_lot|price_observation|catalog_item|population_report

Creates connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts,README.md}, data/fixtures/<id>/, a sources.json stub, and rebuilds connectors/registry.json.

Folder = primary engine: api/ (direct HTTP/JSON/HTML, official APIs, bulk files, Shopify/Woo adapters), firecrawl/ (rendered markdown/HTML), scrapfly/ (anti-bot/JS/geo). Shared helpers live in _lib-style folders (connectors/api/_lib, _auction-lib, _wlib, _luxury-lib, _carlib) — reuse them; add to them rather than duplicating parsers. Framework-level adapters live in packages/connectors/src/adapters.

# 2. meta.json (the registry entry)

Key fields (ConnectorMetaSchema in packages/connectors/src/types.ts):

Rate limits, concurrency, timeouts and engine policies are not in meta.json: they live in connectors/domains.json (SPEC §16), keyed by host. Add an entry for every new domain.

# 3. crawl(ctx) — acquisition

async *crawl(ctx: CrawlContext): AsyncIterable<RawRecordInput>

• Use ctx.fetch(url, opts) for every request: it routes engines, scores quality, records costs, applies the per-host gate (concurrency, interval, circuit breaker) and the crawl budget. await this.throttle(url) before each fetch adds the connector's own politeness on top.
• Pass expect + parse so the router can score the page and fall back to the next engine when the parser finds nothing (SPEC §11 automatic escalation). Do not set engines wider than enginePriority.
• Yield one raw record per source page or item, payload = the source-shaped data you parsed (JSON from the API / rows you extracted), snapshot = raw HTML/markdown when useful for debugging (stored on disk when large; SPEC §24). externalId = stable id from the source (item id, lot id, seed:pN).
• Resumable: read ctx.options.cursor, call await ctx.setCursor({...}) at every checkpoint (page, seed, date). In mode: 'backfill' also call await ctx.progress({ page, totalPages, itemsProcessed, reachedDate }) and finish with setCursor({ done: true }) — the crawler turns that into connector_backfills progress (SPEC §9).
• Respect ctx.options.limit (this.reached(ctx, count)) and ctx.signal.
• Report problems, never swallow them: ctx.anomaly('page_fetch_failed' | 'parse_failure_page' | 'schema_drift' | 'pagination_failure' | 'price_parse_failure' | 'selector_missing', detail).
• Prefer adapters: adapters.discoverFromSitemap, adapters.numberedPages/cursorPages/offsetPages, adapters.productsFromHtml (JSON-LD), adapters.parseFeed (RSS), adapters.extractPdfPages (PDF price lists), html.jsonLd / nextData / inlineJson. Shopify/WooCommerce shops need no code: extend ShopifyStoreConnector / WooCommerceStoreConnector and put collections → taxonomy mapping in config.

# 4. normalize(raw) — pure, deterministic, fixture-testable

async normalize(raw: RawRecordLike): Promise<NormalizedRecord[]>

Output kinds (packages/shared/src/schemas.ts): sale (realised transaction), listing (asking price — never a sale, §111), auction_lot (upcoming/live lot), price_observation (guide value), catalog_item, population_report, news_item.

Rules:

• Validate the payload with the connector's Zod schema; throw on unknown shapes (the normalizer records the failure on the raw row — that is the schema-drift signal).
• Native currency, source dates. saleDate is the source's date, never the fetch time. Hammer vs total: set buyerPremiumIncluded and keep hammer/BP in attributes.metadata (SPEC §20).
• Identify: attributes.categorySlug (taxonomy slug), name, set/setCode, number, year, variant, brand, reference, language, identifiers (sku, upc/ean/isbn/jan, style_code, lego_set_number, scryfall_id, cardmarket_id, tcgplayer_id, pcgs_number, comics_org_id, …) — see DETERMINISTIC_IDS in workers/entity-resolution/canonical-key.ts; add new deterministic ids there when a source has one.
• Grading: grade.grader (taxonomy grader slug), grade.grade, qualifier, certificationNumber (parseGradeFromTitle from @rareindex/taxonomy). Cert numbers feed the provenance graph (SPEC §22).
• confidence 0–1 = how sure the extraction and identification are. Titles alone → 0.6–0.75; structured ids → 0.9+. Never 1.0 for scraped data.
• Bundles/lots of many items: isBundle: true (excluded from valuation, kept for audit).
• Unknown → null. Never guess a year, grade or price (§192).

# 5. Fixtures and tests (SPEC §14)

• data/fixtures/<id>/<name>.json = { raw: { url, externalId, kind, engine, fetchedAt, payload }, expect: { minCount, kinds, requiredFields, first }, note } captured from a live page/API (write a small capture script next to the connector or reuse connectors/api/_lib/capture*.ts). Large HTML → <name>.html, referenced by payload.snapshot. Trim payloads to what normalize needs; keep at least: one normal page, one edge case (sold-out, graded, multi-currency, bundle), and one pagination/empty page when relevant.
• index.test.ts: runFixtureSuite(connector, it, expect) (schema validity, prices > 0, currency, dates, dedupe keys) plus parser unit tests on saved HTML/markdown/JSON (title parsing, grade parsing, price and date parsing in the source's locale, pagination detection, currency).
• Live smoke: pnpm ri crawl <id> --mode probe --limit 5 (needs .env), or the _lib/smoke.ts pattern.
• Run: pnpm vitest run connectors/<engine>/<id> · pnpm --filter @rareindex/source-connectors run typecheck.

# 6. Register, document, ship

1. pnpm registry (rebuilds connectors/registry.json; prints completeness warnings) · pnpm sources:build (validates sources.json, regenerates docs/connectors/SOURCES.md).
2. Add the host to connectors/domains.json.
3. pnpm db:seed mirrors the registry into the connectors/sources tables (cursor/state preserved).
4. The worker schedules it by refreshFrequencyMinutes; /admin/connectors/<id> → Test (probe 25), Run now, Backfill, Pause/Resume/Maintenance, logs, last payload, last parsed record.

# 7. Compliance checklist (SPEC §36)

• Public pages only. No login, no paywall, no CAPTCHA solving, no private/personal data (seller names are kept only when the source shows them publicly as part of the listing; never emails/phones/addresses).
• robots.txt respected for the paths crawled; honest UA (RareIndexBot) unless domains.json says a browser UA is needed and terms allow it; Scrapfly ASP only on public pages.
• Rate limits per domain; backfills bounded (backfillMaxPages); archives refreshed monthly at most.
• Attribution: sourceUrl on every record; termsUrl in meta.