# Historical backfills

Historical sales are the most valuable data RareIndex holds (SPEC §9). Backfills crawl a source's archive
backwards, deeply, and **never restart from zero after an interruption**.

## 1. Model

`connector_backfills` — one row per campaign:

| column | meaning |
|---|---|
| `status` | running · paused · completed · failed |
| `backfill_start_date`, `backfill_end_date` | targeted source-date range (optional) |
| `reached_date` | oldest source date reached so far |
| `pages_processed`, `total_pages`, `last_successful_page`, `percent` | progress (percent only when the total is known) |
| `items_processed` | raw records captured by the campaign |
| `last_cursor` | connector-defined cursor (page, seed, date, token…) — the resume point |
| `runs`, `errors`, `retry_count`, `last_error` | bookkeeping; the campaign fails after 8 hard errors |

A backfill is a sequence of time-boxed runs (`RI_MAX_RUN_MINUTES`, default 40) in `mode: 'backfill'`. Each run
receives the campaign's `last_cursor`, the connector calls `ctx.setCursor()` at every checkpoint and
`ctx.progress({ page, totalPages, itemsProcessed, reachedDate })`, and signals completion with
`setCursor({ done: true })`. The scheduler continues running campaigns automatically (backfill jobs have a
lower queue priority than incremental crawls so live data is never starved).

## 2. Operating

```bash
pnpm ri backfill <id>                 # start or continue the campaign, run one time-boxed slice now
pnpm ri backfill <id> --status        # campaign row
pnpm ri backfill <id> --pause         # scheduler stops continuing it
pnpm ri backfill <id> --reset         # abandon the campaign and start from scratch (explicit operator decision)
pnpm ri backfill <id> --start 2015-01-01 --end 2020-12-31
```

Admin: **Backfill** button (start/continue), **Pause backfill**, **Reset backfill**; progress bar in the
explorer; campaign table on the connector page (`/admin/connectors/<id>`).

## 3. Connector requirements

- Deep sources declare `historicalDepth: months|years|decades` in meta.json (capability `historical_backfill`).
- Pagination must be resumable from `ctx.options.cursor` (page number, offset, cursor token, date window, seed).
- `backfillMaxPages` from `connectors/domains.json` bounds one run; archives are refreshed at most monthly
  (refresh class ARCHIVE) — historical result pages almost never change.
- Realised prices keep the source's sale date and currency; buyer's premium is labelled.

## 4. Refresh classes (SPEC §17)

| class | cadence | typical |
|---|---|---|
| HOT | 1–5 min | major marketplaces' newly-listed feeds (only with an official API) |
| ACTIVE | 15–60 min | live auction lots ending soon, marketplace listings |
| NORMAL | 6–24 h | dealer inventories, price guides, catalogs, population reports (daily/weekly) |
| ARCHIVE | weekly–monthly | auction archives, bulk catalog files, completed backfills |

`refreshFrequencyMinutes` in meta.json sets the class; operators can override the cadence and priority per
connector in the admin config panel without redeploying.