# RareIndex

**The global market for collectibles.** RareIndex.io is a market-intelligence, search, pricing and
index platform for collectible assets: trading cards, sports cards, watches, sneakers, LEGO, comics,
video games, toys, coins, art, wine and, over time, every category the collectibles economy contains.
It continuously collects public market data from marketplaces, auction houses, dealers, grading
companies, manufacturer catalogs and archives, resolves it into one canonical asset graph, and turns
it into research-grade data: valuations with confidence levels, comparable sales, live listings,
rarity and liquidity scores, price histories, and the RareIndex family of indices (`RARE` and its
category subindices).

RareIndex is an intelligence platform, not a marketplace aggregator. Every number shown is traceable
to its source, every estimate carries a sample size and a confidence level, and when the evidence is
insufficient the platform says "Data unavailable" rather than inventing a figure. The full product
mandate is in [`docs/SPEC-original.md`](docs/SPEC-original.md); the condensed working version that
guides implementation is [`CLAUDE.md`](CLAUDE.md).

## Architecture

TypeScript end-to-end in a pnpm monorepo: Next.js 16 web app, Fastify public API, PostgreSQL 17
(pg_trgm, unaccent, pgvector) as system of record, pg-boss workers for the ingestion pipeline, and a
connector framework routing Direct API → Firecrawl → Scrapfly with quality scoring and health
metrics. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the layout and data flow, and
[`docs/adr/`](docs/adr/) for the decisions behind it.

```
apps/web · apps/api · packages/{shared,database,taxonomy,connectors,valuation,indices,search,ai}
workers/ · connectors/ (+ registry.json) · data/{taxonomy,fixtures} · docs/ · scripts/
```

Data flow: `raw_records` (immutable) → `normalize()` → `normalized_records` → entity resolution →
`assets` / `asset_variants` → dedupe and price validation (flags + audit log, nothing deleted) →
`sales` / `listings` / `auction_lots` / `price_observations` → valuations and stats → indices.

## Quick start

Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with the `pgvector` extension available.

```bash
pnpm install
createdb rareindex
cp .env.example .env            # fill DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, AI and Resend keys
pnpm db:migrate && pnpm db:seed # creates extensions + tables, seeds taxonomy, graders, indices, registry

pnpm dev:web                    # Next.js on http://localhost:3000
pnpm dev:api                    # public REST API (/v1)
pnpm worker                     # pipeline workers (crawl, normalize, resolve, value, index)
pnpm ri --help                  # operational CLI (run a connector, recompute valuations, health…)
```

Useful scripts: `pnpm typecheck`, `pnpm test`, `pnpm build`, `pnpm db:generate` (after editing
`packages/database/src/schema/*`), `pnpm tsx scripts/build-registry.ts` (rebuilds
`connectors/registry.json` from per-connector `meta.json` files).

## Adding a connector

A new source needs only four things: connector metadata (`meta.json`), an extraction schema, a pure
`normalize()` adapter and real captured fixtures with tests. The rest of the pipeline works
unchanged. Conventions and the module contract are in [`connectors/README.md`](connectors/README.md).

## Data honesty rules

- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions or listings.
- Price-guide values are stored as observations, never as transactions.
- Valuations are published only with sample size, confidence, low/high range and timestamp.
- Outliers are flagged and audited, never silently deleted.
- Sale dates come from the source; crawl time is never substituted.
- Historical prices are converted with the exchange rate of the sale date.

Details: [`docs/adr/0002-data-honesty.md`](docs/adr/0002-data-honesty.md).

## Deployment

Production runs on a MacLustr node (PostgreSQL 17 + pgvector) deployed through the `mld` gateway:
PM2 processes `rareindex-web`, `rareindex-api`, `rareindex-worker` and an ngrok tunnel serving
**https://www.rareindex.io**. Secrets live only in the node's `.env` / the `mld` manifest.

## Disclaimers

RareIndex is an information platform. Valuations are estimates derived from observed public data.
Listing prices are not confirmed transactions. Past performance does not guarantee future results.
RareIndex does not authenticate items. Third-party names, marks and content belong to their owners
and are attributed to their sources; RareIndex respects source terms, robots policies, rate limits,
copyright, privacy and database rights, and never bypasses authentication, paywalls or CAPTCHAs.

## License

Proprietary — © 2026 RareIndex. All rights reserved.