# RareIndex connector layer

The connector layer is the data-acquisition half of RareIndex: hundreds of marketplaces, auction houses,
dealers, grading databases, price guides and catalogs → one canonical, deduplicated, historically deep
market dataset. This folder is its documentation.

| Document | What it covers |
|---|---|
| [ARCHITECTURE.md](./ARCHITECTURE.md) | SDK, engines (HTTP → Firecrawl → Scrapfly), router, domain policies, circuit breaker, adapters, pipeline, storage |
| [ADDING_A_CONNECTOR.md](./ADDING_A_CONNECTOR.md) | Step-by-step contract for a production-grade connector (scaffold, meta, crawl, normalize, fixtures, tests, compliance) |
| [SOURCES.md](./SOURCES.md) | Generated catalogue of every source researched: implemented, partial, gated, blocked, rejected — with access facts |
| [ENTITY_RESOLUTION.md](./ENTITY_RESOLUTION.md) | How source records become canonical assets/variants; deterministic ids, canonical keys, fuzzy rules, confidence |
| [HEALTH_MONITORING.md](./HEALTH_MONITORING.md) | Health statuses, metrics, schema-drift detectors, alerts, the `/admin/connectors` explorer and JSON API |
| [BACKFILL.md](./BACKFILL.md) | Resumable historical backfill campaigns: state, CLI, admin controls, refresh classes |

## Quick reference

```bash
pnpm connector:new <id> --url https://… --name "…" [--adapter shopify|woocommerce] …   # scaffold (SPEC §15)
pnpm registry                 # rebuild connectors/registry.json from meta.json files (+ completeness warnings)
pnpm registry:check --strict  # CI: fail when a connector lacks tests/fixtures/accessNotes
pnpm sources:build            # validate data/sources/** and regenerate SOURCES.md
pnpm connectors:test          # vitest over every connector (fixture suites + parser tests)
pnpm ri crawl <id> --mode probe --limit 5   # live smoke test (needs DATABASE_URL)
pnpm ri backfill <id> [--status|--pause|--reset]
pnpm ri certs [--grader psa]  # verify cert numbers through cert_lookup connectors
pnpm ri health [--probe]
```

Layout:

```
packages/connectors/src/   SDK: types, base, router, engines/, domains (policies), circuit, adapters/, health, sources, testing
connectors/<engine>/<id>/  one folder per connector: meta.json · index.ts · index.test.ts · README.md
connectors/registry.json   GENERATED registry (§106)          connectors/domains.json + domains.d/   per-host policies (§16)
data/fixtures/<id>/        real captured payloads used by tests      data/sources/sources.json + entries/   source catalogue (§4)
workers/crawler            runs, cursors, backfills, scheduler       workers/normalizer · entity-resolution · health · certs-verify
apps/web /admin/connectors explorer · /admin/coverage dashboard · /api/admin/connectors · /api/admin/coverage
```

## Principles

1. **Public data, lawfully** (SPEC §36): no logins, paywalls, CAPTCHA solving or personal data; robots and
   rate limits respected; every connector carries `accessNotes` explaining exactly what is read.
2. **Structure before scraping**: official API → bulk file/feed → embedded JSON → Firecrawl → Scrapfly.
3. **Raw is immutable, normalised is auditable, canonical is deduplicated** (§108). Everything keeps its
   source URL, engine, parser and connector version.
4. **Never invent data** (§192): native currency, source dates, hammer vs total labelled, `null` when unknown,
   confidence < 1 for anything scraped.
5. **Fault isolation** (§26): per-host gates and circuit breakers; one broken site never stalls the pipeline.
6. **Every wave makes the next one cheaper**: adapters (Shopify, WooCommerce, sitemaps, RSS, PDF, JSON-LD),
   shared parsing libs, the scaffolder and the fixture suite are the leverage.