TypeScript 61.9%
HTML 37.2%
SQL 0.7%
1# RareIndex connector layer23The connector layer is the data-acquisition half of RareIndex: hundreds of marketplaces, auction houses,4dealers, grading databases, price guides and catalogs → one canonical, deduplicated, historically deep5market dataset. This folder is its documentation.67| Document | What it covers |8|---|---|9| [ARCHITECTURE.md](./ARCHITECTURE.md) | SDK, engines (HTTP → Firecrawl → Scrapfly), router, domain policies, circuit breaker, adapters, pipeline, storage |10| [ADDING_A_CONNECTOR.md](./ADDING_A_CONNECTOR.md) | Step-by-step contract for a production-grade connector (scaffold, meta, crawl, normalize, fixtures, tests, compliance) |11| [SOURCES.md](./SOURCES.md) | Generated catalogue of every source researched: implemented, partial, gated, blocked, rejected — with access facts |12| [ENTITY_RESOLUTION.md](./ENTITY_RESOLUTION.md) | How source records become canonical assets/variants; deterministic ids, canonical keys, fuzzy rules, confidence |13| [HEALTH_MONITORING.md](./HEALTH_MONITORING.md) | Health statuses, metrics, schema-drift detectors, alerts, the `/admin/connectors` explorer and JSON API |14| [BACKFILL.md](./BACKFILL.md) | Resumable historical backfill campaigns: state, CLI, admin controls, refresh classes |1516## Quick reference1718```bash19pnpm connector:new <id> --url https://… --name "…" [--adapter shopify|woocommerce] … # scaffold (SPEC §15)20pnpm registry # rebuild connectors/registry.json from meta.json files (+ completeness warnings)21pnpm registry:check --strict # CI: fail when a connector lacks tests/fixtures/accessNotes22pnpm sources:build # validate data/sources/** and regenerate SOURCES.md23pnpm connectors:test # vitest over every connector (fixture suites + parser tests)24pnpm ri crawl <id> --mode probe --limit 5 # live smoke test (needs DATABASE_URL)25pnpm ri backfill <id> [--status|--pause|--reset]26pnpm ri certs [--grader psa] # verify cert numbers through cert_lookup connectors27pnpm ri health [--probe]28```2930Layout:3132```33packages/connectors/src/ SDK: types, base, router, engines/, domains (policies), circuit, adapters/, health, sources, testing34connectors/<engine>/<id>/ one folder per connector: meta.json · index.ts · index.test.ts · README.md35connectors/registry.json GENERATED registry (§106) connectors/domains.json + domains.d/ per-host policies (§16)36data/fixtures/<id>/ real captured payloads used by tests data/sources/sources.json + entries/ source catalogue (§4)37workers/crawler runs, cursors, backfills, scheduler workers/normalizer · entity-resolution · health · certs-verify38apps/web /admin/connectors explorer · /admin/coverage dashboard · /api/admin/connectors · /api/admin/coverage39```4041## Principles42431. **Public data, lawfully** (SPEC §36): no logins, paywalls, CAPTCHA solving or personal data; robots and44 rate limits respected; every connector carries `accessNotes` explaining exactly what is read.452. **Structure before scraping**: official API → bulk file/feed → embedded JSON → Firecrawl → Scrapfly.463. **Raw is immutable, normalised is auditable, canonical is deduplicated** (§108). Everything keeps its47 source URL, engine, parser and connector version.484. **Never invent data** (§192): native currency, source dates, hammer vs total labelled, `null` when unknown,49 confidence < 1 for anything scraped.505. **Fault isolation** (§26): per-host gates and circuit breakers; one broken site never stalls the pipeline.516. **Every wave makes the next one cheaper**: adapters (Shopify, WooCommerce, sitemaps, RSS, PDF, JSON-LD),52 shared parsing libs, the scaffolder and the fixture suite are the leverage.53