TypeScript 61.9%
HTML 37.2%
SQL 0.7%
1# Historical backfills23Historical sales are the most valuable data RareIndex holds (SPEC §9). Backfills crawl a source's archive4backwards, deeply, and **never restart from zero after an interruption**.56## 1. Model78`connector_backfills` — one row per campaign:910| column | meaning |11|---|---|12| `status` | running · paused · completed · failed |13| `backfill_start_date`, `backfill_end_date` | targeted source-date range (optional) |14| `reached_date` | oldest source date reached so far |15| `pages_processed`, `total_pages`, `last_successful_page`, `percent` | progress (percent only when the total is known) |16| `items_processed` | raw records captured by the campaign |17| `last_cursor` | connector-defined cursor (page, seed, date, token…) — the resume point |18| `runs`, `errors`, `retry_count`, `last_error` | bookkeeping; the campaign fails after 8 hard errors |1920A backfill is a sequence of time-boxed runs (`RI_MAX_RUN_MINUTES`, default 40) in `mode: 'backfill'`. Each run21receives the campaign's `last_cursor`, the connector calls `ctx.setCursor()` at every checkpoint and22`ctx.progress({ page, totalPages, itemsProcessed, reachedDate })`, and signals completion with23`setCursor({ done: true })`. The scheduler continues running campaigns automatically (backfill jobs have a24lower queue priority than incremental crawls so live data is never starved).2526## 2. Operating2728```bash29pnpm ri backfill <id> # start or continue the campaign, run one time-boxed slice now30pnpm ri backfill <id> --status # campaign row31pnpm ri backfill <id> --pause # scheduler stops continuing it32pnpm ri backfill <id> --reset # abandon the campaign and start from scratch (explicit operator decision)33pnpm ri backfill <id> --start 2015-01-01 --end 2020-12-3134```3536Admin: **Backfill** button (start/continue), **Pause backfill**, **Reset backfill**; progress bar in the37explorer; campaign table on the connector page (`/admin/connectors/<id>`).3839## 3. Connector requirements4041- Deep sources declare `historicalDepth: months|years|decades` in meta.json (capability `historical_backfill`).42- Pagination must be resumable from `ctx.options.cursor` (page number, offset, cursor token, date window, seed).43- `backfillMaxPages` from `connectors/domains.json` bounds one run; archives are refreshed at most monthly44 (refresh class ARCHIVE) — historical result pages almost never change.45- Realised prices keep the source's sale date and currency; buyer's premium is labelled.4647## 4. Refresh classes (SPEC §17)4849| class | cadence | typical |50|---|---|---|51| HOT | 1–5 min | major marketplaces' newly-listed feeds (only with an official API) |52| ACTIVE | 15–60 min | live auction lots ending soon, marketplace listings |53| NORMAL | 6–24 h | dealer inventories, price guides, catalogs, population reports (daily/weekly) |54| ARCHIVE | weekly–monthly | auction archives, bulk catalog files, completed backfills |5556`refreshFrequencyMinutes` in meta.json sets the class; operators can override the cadence and priority per57connector in the admin config panel without redeploying.58