# Historical backfills

Historical sales are the most valuable data RareIndex holds (SPEC §9). Backfills crawl a source's archive backwards, deeply, and never restart from zero after an interruption.

# 1. Model

connector_backfills — one row per campaign:

column	meaning
`status`	running · paused · completed · failed
`backfill_start_date`, `backfill_end_date`	targeted source-date range (optional)
`reached_date`	oldest source date reached so far
`pages_processed`, `total_pages`, `last_successful_page`, `percent`	progress (percent only when the total is known)
`items_processed`	raw records captured by the campaign
`last_cursor`	connector-defined cursor (page, seed, date, token…) — the resume point
`runs`, `errors`, `retry_count`, `last_error`	bookkeeping; the campaign fails after 8 hard errors

A backfill is a sequence of time-boxed runs (RI_MAX_RUN_MINUTES, default 40) in mode: 'backfill'. Each run receives the campaign's last_cursor, the connector calls ctx.setCursor() at every checkpoint and ctx.progress({ page, totalPages, itemsProcessed, reachedDate }), and signals completion with setCursor({ done: true }). The scheduler continues running campaigns automatically (backfill jobs have a lower queue priority than incremental crawls so live data is never starved).

# 2. Operating

bash

pnpm ri backfill <id>                 # start or continue the campaign, run one time-boxed slice now
pnpm ri backfill <id> --status        # campaign row
pnpm ri backfill <id> --pause         # scheduler stops continuing it
pnpm ri backfill <id> --reset         # abandon the campaign and start from scratch (explicit operator decision)
pnpm ri backfill <id> --start 2015-01-01 --end 2020-12-31

Admin: Backfill button (start/continue), Pause backfill, Reset backfill; progress bar in the explorer; campaign table on the connector page (/admin/connectors/<id>).

# 3. Connector requirements

Deep sources declare historicalDepth: months|years|decades in meta.json (capability historical_backfill).
Pagination must be resumable from ctx.options.cursor (page number, offset, cursor token, date window, seed).
backfillMaxPages from connectors/domains.json bounds one run; archives are refreshed at most monthly (refresh class ARCHIVE) — historical result pages almost never change.
Realised prices keep the source's sale date and currency; buyer's premium is labelled.

# 4. Refresh classes (SPEC §17)

class	cadence	typical
HOT	1–5 min	major marketplaces' newly-listed feeds (only with an official API)
ACTIVE	15–60 min	live auction lots ending soon, marketplace listings
NORMAL	6–24 h	dealer inventories, price guides, catalogs, population reports (daily/weekly)
ARCHIVE	weekly–monthly	auction archives, bulk catalog files, completed backfills

refreshFrequencyMinutes in meta.json sets the class; operators can override the cadence and priority per connector in the admin config panel without redeploying.