# RareIndex

The global market for collectibles. RareIndex.io is a market-intelligence, search, pricing and index platform for collectible assets: trading cards, sports cards, watches, sneakers, LEGO, comics, video games, toys, coins, art, wine and, over time, every category the collectibles economy contains. It continuously collects public market data from marketplaces, auction houses, dealers, grading companies, manufacturer catalogs and archives, resolves it into one canonical asset graph, and turns it into research-grade data: valuations with confidence levels, comparable sales, live listings, rarity and liquidity scores, price histories, and the RareIndex family of indices (RARE and its category subindices).

RareIndex is an intelligence platform, not a marketplace aggregator. Every number shown is traceable to its source, every estimate carries a sample size and a confidence level, and when the evidence is insufficient the platform says "Data unavailable" rather than inventing a figure. The full product mandate is in docs/SPEC-original.md; the condensed working version that guides implementation is CLAUDE.md.

# Architecture

TypeScript end-to-end in a pnpm monorepo: Next.js 16 web app, Fastify public API, PostgreSQL 17 (pg_trgm, unaccent, pgvector) as system of record, pg-boss workers for the ingestion pipeline, and a connector framework routing Direct API → Firecrawl → Scrapfly with quality scoring and health metrics. See docs/ARCHITECTURE.md for the layout and data flow, and docs/adr/ for the decisions behind it.

text

apps/web · apps/api · packages/{shared,database,taxonomy,connectors,valuation,indices,search,ai}
workers/ · connectors/ (+ registry.json) · data/{taxonomy,fixtures} · docs/ · scripts/

Data flow: raw_records (immutable) → normalize() → normalized_records → entity resolution → assets / asset_variants → dedupe and price validation (flags + audit log, nothing deleted) → sales / listings / auction_lots / price_observations → valuations and stats → indices.

# Quick start

Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with the pgvector extension available.

bash

pnpm install
createdb rareindex
cp .env.example .env            # fill DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, AI and Resend keys
pnpm db:migrate && pnpm db:seed # creates extensions + tables, seeds taxonomy, graders, indices, registry

pnpm dev:web                    # Next.js on http://localhost:3000
pnpm dev:api                    # public REST API (/v1)
pnpm worker                     # pipeline workers (crawl, normalize, resolve, value, index)
pnpm ri --help                  # operational CLI (run a connector, recompute valuations, health…)

Useful scripts: pnpm typecheck, pnpm test, pnpm build, pnpm db:generate (after editing packages/database/src/schema/*), pnpm tsx scripts/build-registry.ts (rebuilds connectors/registry.json from per-connector meta.json files).

# Adding a connector

A new source needs only four things: connector metadata (meta.json), an extraction schema, a pure normalize() adapter and real captured fixtures with tests. The rest of the pipeline works unchanged. Conventions and the module contract are in connectors/README.md.

# Data honesty rules

No fabricated data, ever: no invented sales, prices, populations, market caps, auctions or listings.
Price-guide values are stored as observations, never as transactions.
Valuations are published only with sample size, confidence, low/high range and timestamp.
Outliers are flagged and audited, never silently deleted.
Sale dates come from the source; crawl time is never substituted.
Historical prices are converted with the exchange rate of the sale date.

Details: docs/adr/0002-data-honesty.md.

# Deployment

Production runs on a MacLustr node (PostgreSQL 17 + pgvector) deployed through the mld gateway: PM2 processes rareindex-web, rareindex-api, rareindex-worker and an ngrok tunnel serving https://www.rareindex.io. Secrets live only in the node's .env / the mld manifest.

# Disclaimers

RareIndex is an information platform. Valuations are estimates derived from observed public data. Listing prices are not confirmed transactions. Past performance does not guarantee future results. RareIndex does not authenticate items. Third-party names, marks and content belong to their owners and are attributed to their sources; RareIndex respects source terms, robots policies, rate limits, copyright, privacy and database rights, and never bypasses authentication, paywalls or CAPTCHAs.

spb/rareindex