# Adding a connector

This is the operational guide for building a **production-grade** RareIndex connector. A connector is
not complete because it returns one listing (SPEC §35). It is complete when it has reliable discovery,
pagination, normalisation, stable ids, error handling, rate limiting, retries, fixtures, tests, health,
logging, persistence, a refresh class, source metadata and documentation.

## 0. Before writing code: the source registry

Every source starts as an entry in the source registry (schema: `packages/connectors/src/sources.ts`,
SPEC §4). Write entries as an array in `data/sources/entries/<your-group>.json`; `pnpm sources:build` merges
all fragments over `data/sources/sources.json` (generated, do not edit by hand) and regenerates `SOURCES.md`.
Likewise, per-host policies go in `connectors/domains.d/<your-group>.json` as `{ "domains": { "host": {…} } }`. Research and **verify live** (curl with the honest bot UA, then Firecrawl, then Scrapfly):

- robots.txt (`adapters.parseRobots`) — which paths are allowed for `*` / our UA; `Crawl-delay`.
- Terms of use URL; anything that forbids automated access of public pages → `legal` note; login walls,
  paywalls, CAPTCHAs protecting private data → **status `blocked`**, never circumvented (SPEC §36).
- Public API? Official docs URL, auth type. Prefer it over any HTML.
- Structured data: `__NEXT_DATA__`, JSON-LD Product (`adapters.productsFromHtml`), XHR JSON the page
  itself loads, sitemaps (`adapters.discoverFromSitemap`), RSS, bulk files, Shopify `products.json`,
  WooCommerce Store API.
- Pagination style, JS rendering requirement, anti-bot (Cloudflare/Akamai/PerimeterX), rate limits.
- Which fields exist: sold price, sale date, fees/buyer premium, seller, grading, cert number, SKU/UPC/EAN.

Record what you found (`verified: true`, `verifiedAt`). Set `status` honestly: `implemented`, `partial`,
`gated` (needs a key we do not hold), `planned`, `blocked`, `rejected` (+ `statusReason`).

## 1. Scaffold

```bash
pnpm connector:new <id> --url https://www.example.com --name "Example" --engine api|firecrawl|scrapfly \
  [--adapter shopify|woocommerce] --type marketplace|auction_house|dealer|pricing_guide|grading_company|catalog \
  --categories pokemon,magic_the_gathering --country US --currency USD --kind listing|sale|auction_lot|price_observation|catalog_item|population_report
```

Creates `connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts,README.md}`, `data/fixtures/<id>/`,
a `sources.json` stub, and rebuilds `connectors/registry.json`.

Folder = primary engine: `api/` (direct HTTP/JSON/HTML, official APIs, bulk files, Shopify/Woo adapters),
`firecrawl/` (rendered markdown/HTML), `scrapfly/` (anti-bot/JS/geo). Shared helpers live in `_lib`-style
folders (`connectors/api/_lib`, `_auction-lib`, `_wlib`, `_luxury-lib`, `_carlib`) — reuse them; add to
them rather than duplicating parsers. Framework-level adapters live in `packages/connectors/src/adapters`.

## 2. meta.json (the registry entry)

Key fields (`ConnectorMetaSchema` in `packages/connectors/src/types.ts`):

| field | meaning |
|---|---|
| `id`, `sourceId` | slug; `sourceId` = `sources.json` id (several connectors may share one source) |
| `sourceType` | marketplace · auction_house · dealer · pricing_guide · grading_company · catalog · … |
| `enginePriority` | engines the router may use, in order (`["api"]`, `["firecrawl","scrapfly"]`, `["scrapfly"]`) |
| `categories` | taxonomy slugs (`data/taxonomy/categories.json`) the connector yields |
| `regions`, `country`, `languages`, `currency` | geography; `currency` = native currencies emitted |
| `supports*` / `capabilities` | what the connector does (SPEC §5); `capabilities` is derived from the flags and may add `cert_lookup`, `seller_data`, `price_guide`, `historical_backfill` |
| `refreshFrequencyMinutes` / `refreshClass` | HOT 1–5 min · ACTIVE 15–60 · NORMAL 6–24 h · ARCHIVE weekly+ (SPEC §17) |
| `historicalDepth` | none · months · years · decades — drives backfill campaigns |
| `requires` | env vars (API keys). Missing → connector reported `DISABLED`, never crashes |
| `acquisitionMethod` | human label shown in the admin ("official API", "Shopify products.json", "Firecrawl markdown", "Scrapfly render") |
| `trustScore` | 0–1 baseline trust of the source (auction house with realised prices ≈ 0.9; price guide ≈ 0.7) |
| `accessNotes` | **mandatory**: exactly which public pages/endpoints are read, robots findings, rate limit, anti-bot behaviour, what is deliberately NOT fetched and why. This is the compliance record. |
| `config` | seeds, mappings, page counts — everything an operator may tune in `/admin/connectors/<id>` |

Rate limits, concurrency, timeouts and engine policies are **not** in meta.json: they live in
`connectors/domains.json` (SPEC §16), keyed by host. Add an entry for every new domain.

## 3. crawl(ctx) — acquisition

```ts
async *crawl(ctx: CrawlContext): AsyncIterable<RawRecordInput>
```

- Use `ctx.fetch(url, opts)` for **every** request: it routes engines, scores quality, records costs,
  applies the per-host gate (concurrency, interval, circuit breaker) and the crawl budget.
  `await this.throttle(url)` before each fetch adds the connector's own politeness on top.
- Pass `expect` + `parse` so the router can score the page and fall back to the next engine when the
  parser finds nothing (SPEC §11 automatic escalation). Do not set `engines` wider than `enginePriority`.
- Yield **one raw record per source page or item**, `payload` = the source-shaped data you parsed
  (JSON from the API / rows you extracted), `snapshot` = raw HTML/markdown when useful for debugging
  (stored on disk when large; SPEC §24). `externalId` = stable id from the source (item id, lot id, `seed:pN`).
- **Resumable**: read `ctx.options.cursor`, call `await ctx.setCursor({...})` at every checkpoint (page,
  seed, date). In `mode: 'backfill'` also call `await ctx.progress({ page, totalPages, itemsProcessed, reachedDate })`
  and finish with `setCursor({ done: true })` — the crawler turns that into `connector_backfills` progress (SPEC §9).
- Respect `ctx.options.limit` (`this.reached(ctx, count)`) and `ctx.signal`.
- Report problems, never swallow them: `ctx.anomaly('page_fetch_failed' | 'parse_failure_page' | 'schema_drift' | 'pagination_failure' | 'price_parse_failure' | 'selector_missing', detail)`.
- Prefer adapters: `adapters.discoverFromSitemap`, `adapters.numberedPages/cursorPages/offsetPages`,
  `adapters.productsFromHtml` (JSON-LD), `adapters.parseFeed` (RSS), `adapters.extractPdfPages` (PDF price lists),
  `html.jsonLd / nextData / inlineJson`. Shopify/WooCommerce shops need **no code**: extend
  `ShopifyStoreConnector` / `WooCommerceStoreConnector` and put collections → taxonomy mapping in `config`.

## 4. normalize(raw) — pure, deterministic, fixture-testable

```ts
async normalize(raw: RawRecordLike): Promise<NormalizedRecord[]>
```

Output kinds (`packages/shared/src/schemas.ts`): `sale` (realised transaction), `listing` (asking price —
never a sale, §111), `auction_lot` (upcoming/live lot), `price_observation` (guide value), `catalog_item`,
`population_report`, `news_item`.

Rules:
- Validate the payload with the connector's Zod schema; throw on unknown shapes (the normalizer records
  the failure on the raw row — that is the schema-drift signal).
- **Native currency, source dates.** `saleDate` is the source's date, never the fetch time. Hammer vs
  total: set `buyerPremiumIncluded` and keep hammer/BP in `attributes.metadata` (SPEC §20).
- Identify: `attributes.categorySlug` (taxonomy slug), `name`, `set`/`setCode`, `number`, `year`,
  `variant`, `brand`, `reference`, `language`, `identifiers` (sku, upc/ean/isbn/jan, style_code, lego_set_number,
  scryfall_id, cardmarket_id, tcgplayer_id, pcgs_number, comics_org_id, …) — see `DETERMINISTIC_IDS` in
  `workers/entity-resolution/canonical-key.ts`; add new deterministic ids there when a source has one.
- Grading: `grade.grader` (taxonomy grader slug), `grade.grade`, `qualifier`, `certificationNumber`
  (`parseGradeFromTitle` from `@rareindex/taxonomy`). Cert numbers feed the provenance graph (SPEC §22).
- `confidence` 0–1 = how sure the extraction and identification are. Titles alone → 0.6–0.75; structured
  ids → 0.9+. Never 1.0 for scraped data.
- Bundles/lots of many items: `isBundle: true` (excluded from valuation, kept for audit).
- Unknown → `null`. Never guess a year, grade or price (§192).

## 5. Fixtures and tests (SPEC §14)

- `data/fixtures/<id>/<name>.json` = `{ raw: { url, externalId, kind, engine, fetchedAt, payload }, expect: { minCount, kinds, requiredFields, first }, note }`
  captured from a **live** page/API (write a small `capture` script next to the connector or reuse
  `connectors/api/_lib/capture*.ts`). Large HTML → `<name>.html`, referenced by `payload.snapshot`.
  Trim payloads to what `normalize` needs; keep at least: one normal page, one edge case (sold-out, graded,
  multi-currency, bundle), and one pagination/empty page when relevant.
- `index.test.ts`: `runFixtureSuite(connector, it, expect)` (schema validity, prices > 0, currency, dates,
  dedupe keys) **plus** parser unit tests on saved HTML/markdown/JSON (title parsing, grade parsing, price
  and date parsing in the source's locale, pagination detection, currency).
- Live smoke: `pnpm ri crawl <id> --mode probe --limit 5` (needs `.env`), or the `_lib/smoke.ts` pattern.
- Run: `pnpm vitest run connectors/<engine>/<id>` · `pnpm --filter @rareindex/source-connectors run typecheck`.

## 6. Register, document, ship

1. `pnpm registry` (rebuilds `connectors/registry.json`; prints completeness warnings) · `pnpm sources:build`
   (validates `sources.json`, regenerates `docs/connectors/SOURCES.md`).
2. Add the host to `connectors/domains.json`.
3. `pnpm db:seed` mirrors the registry into the `connectors`/`sources` tables (cursor/state preserved).
4. The worker schedules it by `refreshFrequencyMinutes`; `/admin/connectors/<id>` → **Test** (probe 25),
   **Run now**, **Backfill**, **Pause/Resume/Maintenance**, logs, last payload, last parsed record.

## 7. Compliance checklist (SPEC §36)

- Public pages only. No login, no paywall, no CAPTCHA solving, no private/personal data (seller names are
  kept only when the source shows them publicly as part of the listing; never emails/phones/addresses).
- robots.txt respected for the paths crawled; honest UA (`RareIndexBot`) unless `domains.json` says a
  browser UA is needed and terms allow it; Scrapfly ASP only on public pages.
- Rate limits per domain; backfills bounded (`backfillMaxPages`); archives refreshed monthly at most.
- Attribution: `sourceUrl` on every record; `termsUrl` in meta.