TypeScript 61.9%
HTML 37.2%
SQL 0.7%
1# Connector architecture23## 1. Layers45```6 ┌────────────────────────── connectors/<engine>/<id> ──────────────────────────┐7 source ──────► │ crawl(ctx) → RawRecordInput (source-shaped payload, snapshot, externalId) │8 │ normalize() → NormalizedRecord (sale · listing · auction_lot · price_observation │9 │ · catalog_item · population_report · news_item) │10 │ healthCheck(), lookup(url) (scanner / cert verification) │11 └───────────────▲──────────────────────────────────────────────────────────────────┘12 │ SDK (packages/connectors)13 ctx.fetch ── Router ── HostGates (concurrency · interval · circuit breaker, per host, from domains.json)14 │ engines in priority order, each result scored by the connector's parser (quality ≥ 0.6)15 ├─ api/feed direct HTTP (honest UA, retries/backoff, conditional requests, binary)16 ├─ firecrawl v2 scrape/search/map (markdown + rawHtml + JSON extraction)17 └─ scrapfly render_js + ASP + geo (public pages only)18 adapters: shopify · woocommerce · sitemap/robots · rss · pdf · schema.org · pagination19 budget store (crawl_state): content hash + change-interval estimate → shouldFetch()20 │21 workers/crawler/run.ts ── raw_records (immutable, content-hash dedupe, snapshots on disk), connector_runs,22 connector_backfills (resumable campaigns), engine stats (statuses, blocked, latency)23 workers/normalizer ── normalized_records + connector_field_stats (schema drift) + price-distribution anomaly24 workers/entity-resolution ─ assets / asset_variants (identifiers → canonical key → fuzzy) → sales · listings ·25 auction_lots · price_observations · population_reports · certificates (+ sightings)26 workers/health ── connector_health snapshots (UP/DEGRADED/BROKEN/DISABLED/MAINTENANCE)27 workers/certs-verify ── cert_lookup connectors verify cert numbers seen on marketplaces28```2930## 2. The connector contract (`packages/connectors/src/types.ts`)3132```ts33interface RareIndexConnector {34 meta: ConnectorMeta; // registry entry (id, source, engines, categories, geography, capabilities, refresh class, requires…)35 version; parserVersion; // bump on crawl / mapping changes; stored on every raw row36 crawl(ctx): AsyncIterable<RawRecordInput>; // resumable through ctx.options.cursor + ctx.setCursor / ctx.progress37 normalize(raw): Promise<NormalizedRecord[]>; // pure, deterministic, fixture-tested38 healthCheck(ctx): Promise<ConnectorHealth>; // default: derived from recent runs (+ light probe when fresh)39 lookup?(url, ctx): Promise<RawRecordInput[]>; // single-URL resolution (scanner, cert verification)40 urlPatterns?: RegExp[];41}42```4344SPEC §5 method names map onto this surface: `discover/search/fetchListing/fetchProduct/fetchSoldListings/45fetchAuctionResults/fetchPopulationData/fetchCatalog` are what `crawl()` does for the record kinds the46connector declares in `capabilities`; `refresh()` is the scheduler; `fetchSeller()` is the `seller`/47`sellerReputation` fields (capability `seller_data`). Capabilities are explicit (`meta.capabilities`, derived48from the `supports*` flags plus `cert_lookup`, `price_guide`, `seller_data`, `historical_backfill`, `url_lookup`).4950## 3. Engines and routing (SPEC §10–11)5152`createRouter()` tries the connector's `enginePriority` (narrowed by the domain policy) in order:53541. **api/feed** — `createHttpEngine`: honest `RareIndexBot` UA (browser UA only where `domains.json`55 says `userAgent: "browser"`), exponential backoff on 429/5xx, `Retry-After`, `responseType json|text|binary`.562. **firecrawl** — `/v2/scrape` (markdown + rawHtml, optional JSON schema/prompt extraction, `waitFor`,57 `location`), `/v2/search`, `/v2/map` (sitemap-like discovery). Credits recorded in `costs`.583. **scrapfly** — `/scrape` with `render_js`, `asp`, `country`, `rendering_wait`, extraction models.5960Each result is scored: the connector's `parse()` returns which expected fields were found (weights §199) or,61without a parser, a document heuristic. `isChallengePage()` detects Cloudflare/Akamai/PerimeterX/DataDome62interstitials → counted as `blocked` and escalated. 404/410 never escalate. All attempts feed63`engineStats[engine] = { attempts, success, credits, ms, statuses{code:n}, blocked, circuitOpen }` and the64cost ledger (`onAttempt`). The **strategy that succeeds per domain** is visible in `connector_runs.engine_stats`65and in health (`firecrawl_success_rate`, `scrapfly_fallback_rate`).6667## 4. Per-domain policy and fault isolation (SPEC §16, §26)6869`connectors/domains.json` (+ `domains.d/*.json` fragments) → `policyFor(url)`: `minIntervalMs`, `concurrency`,70`timeoutMs`, `maxRetries`, allowed `engines`, Firecrawl/Scrapfly options, `country/locale/currency`,71`crawlDepth`, `backfillMaxPages`, `userAgent`, `circuitFailures`, `circuitCooldownMs`. Resolution: defaults →72parent domain → exact host. `HostGates` (one per process, shared by all connectors) enforces the concurrency73cap and interval per host and opens a **circuit** after N consecutive failures; while open, requests fail fast74(`circuitOpen` counted), then a single half-open trial closes it again. Health probes bypass the gate.7576## 5. Adapters (SPEC §1, §5)7778`packages/connectors/src/adapters/`:7980- `ShopifyStoreConnector` — `/collections/<handle>/products.json` (250/page), variants → listings, SKU/barcode81 → `upc/ean/jan/isbn`, grade/condition from titles, `lookup()` for `/products/<handle>`; config-only connectors.82- `WooCommerceStoreConnector` — Store API `/wp-json/wc/store/v1/products`, minor-unit prices with currency.83- `discoverFromSitemap` / `parseRobots` / `robotsAllows` — sitemap indexes, gz, lastmod filters, robots rules.84- `parseFeed` — RSS 2.0 / Atom. `extractPdfPages/extractPdfText/parsePricesRealised` — PDF price lists (pdfjs).85- `productsFromHtml/primaryProduct/metaTags` — schema.org Product/Offer JSON-LD and Open Graph.86- `numberedPages/cursorPages/offsetPages/withParams` — pagination loops (page numbers, cursors, offsets/infinite scroll).87- `html.jsonLd/nextData/inlineJson` — embedded JSON.8889## 6. Storage (SPEC §24, §27)9091| table | role |92|---|---|93| `sources`, `connectors` (`meta` jsonb = registry entry, `status` active/paused/maintenance/disabled, `config` incl. cursor) | registry mirror + runtime state |94| `connector_runs` | one row per run: pages, records, dupes, engine stats (statuses/blocked/latency), anomalies, cost, cursor |95| `connector_backfills` | resumable campaigns: last_cursor, pages/items processed, total_pages, reached_date, percent, errors, retry_count |96| `connector_health` | latest health snapshot (JSON `ConnectorHealth`) |97| `connector_field_stats` | daily field-presence counters per connector (schema drift) |98| `crawl_state` | per-URL budget: content hash, change interval, next fetch, failures |99| `costs` | Firecrawl/Scrapfly/AI credits per connector |100| `raw_records` | immutable captures (payload JSON, `snapshot_ref` to disk for large HTML, engine, parser/connector version, http status) |101| `normalized_records` | canonical-shaped staging rows with match decision (method, confidence, reject reason) |102| `assets`, `asset_variants`, `sets`, `brands` | canonical graph |103| `sales`, `listings`, `listing_events`, `auctions`, `auction_lots`, `price_observations`, `population_reports`, `images`, `news` | observations |104| `certificates`, `certificate_sightings` | cert numbers across sources over time (provenance graph, SPEC §22) |105| `sales_multi_currency`, `listings_multi_currency` (views) | USD/CAD/EUR/GBP/JPY at the FX rate of the sale/observation date (SPEC §19) |106107All large tables carry date indexes suitable for range partitioning (`sales(sale_date)`, `raw_records(fetched_at)`,108`price_observations(observation_date)`); ids are text prefixes so partitions can be added by migration without109changing application code.110111## 7. Scheduling (SPEC §17)112113`refreshFrequencyMinutes` ⇄ refresh class: HOT ≤ 5 min · ACTIVE ≤ 60 · NORMAL ≤ 24 h · ARCHIVE weekly+.114The scheduler tick (every minute) enqueues due incremental crawls (priority high/medium/low), continues running115backfill campaigns, skips gated connectors (missing `requires`) and connectors that already have a queued or116running crawl. Each run is time-boxed (`RI_MAX_RUN_MINUTES`, default 40) and resumes from its cursor.117118## 8. Observability (SPEC §25)119120Structured pino logs with `component`/`connector` fields; `connector_run_finished` events; per-run engine121stats; cost ledger; health snapshots every 30 min; `/api/admin/connectors` and `/api/admin/coverage` JSON for122external monitors (MacLustr apps). OpenTelemetry export can wrap the pino logger and the queue handlers123without touching connectors.124