TypeScript 61.9%
HTML 37.2%
SQL 0.7%
1# Adding a connector23This is the operational guide for building a **production-grade** RareIndex connector. A connector is4not complete because it returns one listing (SPEC §35). It is complete when it has reliable discovery,5pagination, normalisation, stable ids, error handling, rate limiting, retries, fixtures, tests, health,6logging, persistence, a refresh class, source metadata and documentation.78## 0. Before writing code: the source registry910Every source starts as an entry in the source registry (schema: `packages/connectors/src/sources.ts`,11SPEC §4). Write entries as an array in `data/sources/entries/<your-group>.json`; `pnpm sources:build` merges12all fragments over `data/sources/sources.json` (generated, do not edit by hand) and regenerates `SOURCES.md`.13Likewise, per-host policies go in `connectors/domains.d/<your-group>.json` as `{ "domains": { "host": {…} } }`. Research and **verify live** (curl with the honest bot UA, then Firecrawl, then Scrapfly):1415- robots.txt (`adapters.parseRobots`) — which paths are allowed for `*` / our UA; `Crawl-delay`.16- Terms of use URL; anything that forbids automated access of public pages → `legal` note; login walls,17 paywalls, CAPTCHAs protecting private data → **status `blocked`**, never circumvented (SPEC §36).18- Public API? Official docs URL, auth type. Prefer it over any HTML.19- Structured data: `__NEXT_DATA__`, JSON-LD Product (`adapters.productsFromHtml`), XHR JSON the page20 itself loads, sitemaps (`adapters.discoverFromSitemap`), RSS, bulk files, Shopify `products.json`,21 WooCommerce Store API.22- Pagination style, JS rendering requirement, anti-bot (Cloudflare/Akamai/PerimeterX), rate limits.23- Which fields exist: sold price, sale date, fees/buyer premium, seller, grading, cert number, SKU/UPC/EAN.2425Record what you found (`verified: true`, `verifiedAt`). Set `status` honestly: `implemented`, `partial`,26`gated` (needs a key we do not hold), `planned`, `blocked`, `rejected` (+ `statusReason`).2728## 1. Scaffold2930```bash31pnpm connector:new <id> --url https://www.example.com --name "Example" --engine api|firecrawl|scrapfly \32 [--adapter shopify|woocommerce] --type marketplace|auction_house|dealer|pricing_guide|grading_company|catalog \33 --categories pokemon,magic_the_gathering --country US --currency USD --kind listing|sale|auction_lot|price_observation|catalog_item|population_report34```3536Creates `connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts,README.md}`, `data/fixtures/<id>/`,37a `sources.json` stub, and rebuilds `connectors/registry.json`.3839Folder = primary engine: `api/` (direct HTTP/JSON/HTML, official APIs, bulk files, Shopify/Woo adapters),40`firecrawl/` (rendered markdown/HTML), `scrapfly/` (anti-bot/JS/geo). Shared helpers live in `_lib`-style41folders (`connectors/api/_lib`, `_auction-lib`, `_wlib`, `_luxury-lib`, `_carlib`) — reuse them; add to42them rather than duplicating parsers. Framework-level adapters live in `packages/connectors/src/adapters`.4344## 2. meta.json (the registry entry)4546Key fields (`ConnectorMetaSchema` in `packages/connectors/src/types.ts`):4748| field | meaning |49|---|---|50| `id`, `sourceId` | slug; `sourceId` = `sources.json` id (several connectors may share one source) |51| `sourceType` | marketplace · auction_house · dealer · pricing_guide · grading_company · catalog · … |52| `enginePriority` | engines the router may use, in order (`["api"]`, `["firecrawl","scrapfly"]`, `["scrapfly"]`) |53| `categories` | taxonomy slugs (`data/taxonomy/categories.json`) the connector yields |54| `regions`, `country`, `languages`, `currency` | geography; `currency` = native currencies emitted |55| `supports*` / `capabilities` | what the connector does (SPEC §5); `capabilities` is derived from the flags and may add `cert_lookup`, `seller_data`, `price_guide`, `historical_backfill` |56| `refreshFrequencyMinutes` / `refreshClass` | HOT 1–5 min · ACTIVE 15–60 · NORMAL 6–24 h · ARCHIVE weekly+ (SPEC §17) |57| `historicalDepth` | none · months · years · decades — drives backfill campaigns |58| `requires` | env vars (API keys). Missing → connector reported `DISABLED`, never crashes |59| `acquisitionMethod` | human label shown in the admin ("official API", "Shopify products.json", "Firecrawl markdown", "Scrapfly render") |60| `trustScore` | 0–1 baseline trust of the source (auction house with realised prices ≈ 0.9; price guide ≈ 0.7) |61| `accessNotes` | **mandatory**: exactly which public pages/endpoints are read, robots findings, rate limit, anti-bot behaviour, what is deliberately NOT fetched and why. This is the compliance record. |62| `config` | seeds, mappings, page counts — everything an operator may tune in `/admin/connectors/<id>` |6364Rate limits, concurrency, timeouts and engine policies are **not** in meta.json: they live in65`connectors/domains.json` (SPEC §16), keyed by host. Add an entry for every new domain.6667## 3. crawl(ctx) — acquisition6869```ts70async *crawl(ctx: CrawlContext): AsyncIterable<RawRecordInput>71```7273- Use `ctx.fetch(url, opts)` for **every** request: it routes engines, scores quality, records costs,74 applies the per-host gate (concurrency, interval, circuit breaker) and the crawl budget.75 `await this.throttle(url)` before each fetch adds the connector's own politeness on top.76- Pass `expect` + `parse` so the router can score the page and fall back to the next engine when the77 parser finds nothing (SPEC §11 automatic escalation). Do not set `engines` wider than `enginePriority`.78- Yield **one raw record per source page or item**, `payload` = the source-shaped data you parsed79 (JSON from the API / rows you extracted), `snapshot` = raw HTML/markdown when useful for debugging80 (stored on disk when large; SPEC §24). `externalId` = stable id from the source (item id, lot id, `seed:pN`).81- **Resumable**: read `ctx.options.cursor`, call `await ctx.setCursor({...})` at every checkpoint (page,82 seed, date). In `mode: 'backfill'` also call `await ctx.progress({ page, totalPages, itemsProcessed, reachedDate })`83 and finish with `setCursor({ done: true })` — the crawler turns that into `connector_backfills` progress (SPEC §9).84- Respect `ctx.options.limit` (`this.reached(ctx, count)`) and `ctx.signal`.85- Report problems, never swallow them: `ctx.anomaly('page_fetch_failed' | 'parse_failure_page' | 'schema_drift' | 'pagination_failure' | 'price_parse_failure' | 'selector_missing', detail)`.86- Prefer adapters: `adapters.discoverFromSitemap`, `adapters.numberedPages/cursorPages/offsetPages`,87 `adapters.productsFromHtml` (JSON-LD), `adapters.parseFeed` (RSS), `adapters.extractPdfPages` (PDF price lists),88 `html.jsonLd / nextData / inlineJson`. Shopify/WooCommerce shops need **no code**: extend89 `ShopifyStoreConnector` / `WooCommerceStoreConnector` and put collections → taxonomy mapping in `config`.9091## 4. normalize(raw) — pure, deterministic, fixture-testable9293```ts94async normalize(raw: RawRecordLike): Promise<NormalizedRecord[]>95```9697Output kinds (`packages/shared/src/schemas.ts`): `sale` (realised transaction), `listing` (asking price —98never a sale, §111), `auction_lot` (upcoming/live lot), `price_observation` (guide value), `catalog_item`,99`population_report`, `news_item`.100101Rules:102- Validate the payload with the connector's Zod schema; throw on unknown shapes (the normalizer records103 the failure on the raw row — that is the schema-drift signal).104- **Native currency, source dates.** `saleDate` is the source's date, never the fetch time. Hammer vs105 total: set `buyerPremiumIncluded` and keep hammer/BP in `attributes.metadata` (SPEC §20).106- Identify: `attributes.categorySlug` (taxonomy slug), `name`, `set`/`setCode`, `number`, `year`,107 `variant`, `brand`, `reference`, `language`, `identifiers` (sku, upc/ean/isbn/jan, style_code, lego_set_number,108 scryfall_id, cardmarket_id, tcgplayer_id, pcgs_number, comics_org_id, …) — see `DETERMINISTIC_IDS` in109 `workers/entity-resolution/canonical-key.ts`; add new deterministic ids there when a source has one.110- Grading: `grade.grader` (taxonomy grader slug), `grade.grade`, `qualifier`, `certificationNumber`111 (`parseGradeFromTitle` from `@rareindex/taxonomy`). Cert numbers feed the provenance graph (SPEC §22).112- `confidence` 0–1 = how sure the extraction and identification are. Titles alone → 0.6–0.75; structured113 ids → 0.9+. Never 1.0 for scraped data.114- Bundles/lots of many items: `isBundle: true` (excluded from valuation, kept for audit).115- Unknown → `null`. Never guess a year, grade or price (§192).116117## 5. Fixtures and tests (SPEC §14)118119- `data/fixtures/<id>/<name>.json` = `{ raw: { url, externalId, kind, engine, fetchedAt, payload }, expect: { minCount, kinds, requiredFields, first }, note }`120 captured from a **live** page/API (write a small `capture` script next to the connector or reuse121 `connectors/api/_lib/capture*.ts`). Large HTML → `<name>.html`, referenced by `payload.snapshot`.122 Trim payloads to what `normalize` needs; keep at least: one normal page, one edge case (sold-out, graded,123 multi-currency, bundle), and one pagination/empty page when relevant.124- `index.test.ts`: `runFixtureSuite(connector, it, expect)` (schema validity, prices > 0, currency, dates,125 dedupe keys) **plus** parser unit tests on saved HTML/markdown/JSON (title parsing, grade parsing, price126 and date parsing in the source's locale, pagination detection, currency).127- Live smoke: `pnpm ri crawl <id> --mode probe --limit 5` (needs `.env`), or the `_lib/smoke.ts` pattern.128- Run: `pnpm vitest run connectors/<engine>/<id>` · `pnpm --filter @rareindex/source-connectors run typecheck`.129130## 6. Register, document, ship1311321. `pnpm registry` (rebuilds `connectors/registry.json`; prints completeness warnings) · `pnpm sources:build`133 (validates `sources.json`, regenerates `docs/connectors/SOURCES.md`).1342. Add the host to `connectors/domains.json`.1353. `pnpm db:seed` mirrors the registry into the `connectors`/`sources` tables (cursor/state preserved).1364. The worker schedules it by `refreshFrequencyMinutes`; `/admin/connectors/<id>` → **Test** (probe 25),137 **Run now**, **Backfill**, **Pause/Resume/Maintenance**, logs, last payload, last parsed record.138139## 7. Compliance checklist (SPEC §36)140141- Public pages only. No login, no paywall, no CAPTCHA solving, no private/personal data (seller names are142 kept only when the source shows them publicly as part of the listing; never emails/phones/addresses).143- robots.txt respected for the paths crawled; honest UA (`RareIndexBot`) unless `domains.json` says a144 browser UA is needed and terms allow it; Scrapfly ASP only on public pages.145- Rate limits per domain; backfills bounded (`backfillMaxPages`); archives refreshed monthly at most.146- Attribution: `sourceUrl` on every record; `termsUrl` in meta.147