# RareIndex connector layer

The connector layer is the data-acquisition half of RareIndex: hundreds of marketplaces, auction houses, dealers, grading databases, price guides and catalogs → one canonical, deduplicated, historically deep market dataset. This folder is its documentation.

Document	What it covers
ARCHITECTURE.md	SDK, engines (HTTP → Firecrawl → Scrapfly), router, domain policies, circuit breaker, adapters, pipeline, storage
ADDING_A_CONNECTOR.md	Step-by-step contract for a production-grade connector (scaffold, meta, crawl, normalize, fixtures, tests, compliance)
SOURCES.md	Generated catalogue of every source researched: implemented, partial, gated, blocked, rejected — with access facts
ENTITY_RESOLUTION.md	How source records become canonical assets/variants; deterministic ids, canonical keys, fuzzy rules, confidence
HEALTH_MONITORING.md	Health statuses, metrics, schema-drift detectors, alerts, the `/admin/connectors` explorer and JSON API
BACKFILL.md	Resumable historical backfill campaigns: state, CLI, admin controls, refresh classes

# Quick reference

bash

pnpm connector:new <id> --url https://… --name "…" [--adapter shopify|woocommerce] …   # scaffold (SPEC §15)
pnpm registry                 # rebuild connectors/registry.json from meta.json files (+ completeness warnings)
pnpm registry:check --strict  # CI: fail when a connector lacks tests/fixtures/accessNotes
pnpm sources:build            # validate data/sources/** and regenerate SOURCES.md
pnpm connectors:test          # vitest over every connector (fixture suites + parser tests)
pnpm ri crawl <id> --mode probe --limit 5   # live smoke test (needs DATABASE_URL)
pnpm ri backfill <id> [--status|--pause|--reset]
pnpm ri certs [--grader psa]  # verify cert numbers through cert_lookup connectors
pnpm ri health [--probe]

Layout:

text

packages/connectors/src/   SDK: types, base, router, engines/, domains (policies), circuit, adapters/, health, sources, testing
connectors/<engine>/<id>/  one folder per connector: meta.json · index.ts · index.test.ts · README.md
connectors/registry.json   GENERATED registry (§106)          connectors/domains.json + domains.d/   per-host policies (§16)
data/fixtures/<id>/        real captured payloads used by tests      data/sources/sources.json + entries/   source catalogue (§4)
workers/crawler            runs, cursors, backfills, scheduler       workers/normalizer · entity-resolution · health · certs-verify
apps/web /admin/connectors explorer · /admin/coverage dashboard · /api/admin/connectors · /api/admin/coverage

# Principles

Public data, lawfully (SPEC §36): no logins, paywalls, CAPTCHA solving or personal data; robots and rate limits respected; every connector carries accessNotes explaining exactly what is read.
Structure before scraping: official API → bulk file/feed → embedded JSON → Firecrawl → Scrapfly.
Raw is immutable, normalised is auditable, canonical is deduplicated (§108). Everything keeps its source URL, engine, parser and connector version.
Never invent data (§192): native currency, source dates, hammer vs total labelled, null when unknown, confidence < 1 for anything scraped.
Fault isolation (§26): per-host gates and circuit breakers; one broken site never stalls the pipeline.
Every wave makes the next one cheaper: adapters (Shopify, WooCommerce, sitemaps, RSS, PDF, JSON-LD), shared parsing libs, the scaffolder and the fixture suite are the leverage.