www.rareindex.io · Methodology · API · Screener · Auctions · contact@spboucher.ai

# RareIndex

RareIndex is the market-data and intelligence layer for physical collectibles — the terminal the collectibles economy never had. It continuously collects public market evidence (marketplaces, auction houses, dealers, price guides, grading companies, manufacturer catalogs), resolves it into a canonical asset graph and turns it into research-grade data: a valuation with a confidence level and a range, comparable sales, live asks, market depth, liquidity and rarity scores, auction intelligence with buyer-pays costs, price histories and the RARE family of indices.

A marketplace tells you "this item is for sale for $4,000". RareIndex tells you:

This asset has a RareIndex Valuation of $3,180 (high confidence, 42 comparable sales of the same variant across five sources), the market is up 12.4 % over 12 months, typical time-to-sale is 18 days, liquidity is above the category average, and the current $2,650 ask is 16.7 % below fair value — while a $472 slab of a different grade matched to the same card is a data anomaly, not a deal.

Every number is traceable to its source, carries a sample size, a confidence and a freshness, and when the evidence is not there the platform says "Not enough data" instead of inventing a figure.

# Table of contents

1. What RareIndex answers
2. Live platform at a glance
3. Product surface
4. Data model — the canonical asset graph
5. Methodology in brief
6. Architecture
7. Repository layout
8. Pipeline and workers
9. Connectors
10. Public API
11. Quick start
12. Operator CLI
13. Testing and quality gates
14. Deployment
15. Brand
16. Data honesty rules
17. Roadmap
18. Author, hosting and contact
19. Disclaimers and license

# What RareIndex answers

# Live platform at a glance

Production figures on 2026-09-11 (PostgreSQL on the MacLustr node):

# Product surface

Public pages (Next.js 16 App Router, server-rendered, dense financial-terminal design, dark mode first-class):

# Data model — the canonical asset graph

Marketplace titles are never assets. Every observation is resolved to a canonical asset (assets: category, family, franchise, brand, series, set, number, year, edition, variant, language, region, identifiers…) and to an asset variant (asset_variants: grader, grade, qualifier, condition, completeness, size — e.g. psa|10, raw|near_mint, cgc|9.8|black_label, psa|unknown for a slab whose grade could not be read). Variants never share a valuation: a PSA 10, a raw copy and a sealed box are three markets.

Evidence tables keep the three kinds of numbers strictly apart (§205 of the spec):

• sales — observed transactions: native price + currency, price_usd at the sale-date ECB rate (fx_rate, fx_date), all_in_usd = buyer-pays price (hammer + estimated buyer premium with a fee_basis), status valid / flagged / excluded, confidence, flags, dedupe_key.
• listings — asks: lifecycle (first_seen_at, last_seen_at, availability, listing_events), discount_to_riv = (ask − RIV) / RIV only when the comparison passed the gates, flags riv_anomaly / riv_review.
• auction_lots — estimates, current bid, hammer, USD conversions, all_in_bid_usd, all_in_estimate_low_usd, bid_vs_riv, estimate_vs_riv, assessment_verdict.
• price_observations — price-guide values, always labelled "guide", never mixed with transactions.

Derived layers: valuations (append-only history with method breakdown), asset_stats / variant_stats (current read model), price_snapshots (daily series), grade_premiums, population_reports, certificates + certificate_sightings, indices / index_values / index_constituents, category_snapshots, radar_findings, correlations, benchmarks. Schema: packages/database/src/schema/*, 8 migrations in packages/database/migrations/.

# Methodology in brief

Full text: docs/METHODOLOGY.md and the live Methodology Center, which imports its thresholds from the code so it cannot drift.

• RIV (RareIndex Valuation) — ensemble of robust estimators over the variant's valid transactions (exponentially-weighted mean, medians 5/10/20, trust- and recency-weighted mean, trimmed mean), median of the core; 365-day window extended to 3 years when thin (confidence capped); falls back to grade-adjusted comparables, then to guide observations, each with capped confidence. Inputs are buyer-pays prices (coalesce(all_in_usd, price_usd)).
• Confidence — 0.35·sample + 0.30·(1 − dispersion) + 0.20·recency + 0.15·source trust; labels High ≥ 0.75, Medium ≥ 0.5, Low > 0.2, else Insufficient.
• Range — low / fair / high from p25 / RIV / p75, at least ±5 %.
• Representative variant — the default (raw / base) variant when it has a transaction-based RIV with ≥ 5 sales and confidence ≥ 0.5, else the qualifying variant with the most sales. Changes, ATH / ATL and drawdown are measured on that variant's own series.
• Ask vs RIV and the anomaly gate — an ask is compared only with a transaction-based RIV of the same variant (≥ 5 sales, confidence ≥ 0.5, match ≥ 0.7). Asks below 0.1× or above 10× RIV are flagged Data/identity anomaly; asks more than 50 % below RIV are held for review; a deal is between −10 % and −50 %. Auction bids are never asks. Deal Score is multiplicative in depth, confidence, sample, liquidity and match quality.
• Buyer premiums — per-house marginal schedules (data/fees/auction-houses.json, 57 houses, each rate labelled published / approximate / none, default 22 % when unknown); VAT, duties and shipping are explicitly excluded.
• Liquidity Score — sales per month, listing depth, source count, days between sales, ask-sold spread. Rarity Score — population, production quantity, sales and listings per year. Momentum / Trending with quality gates (canonical identity, ≥ 5 sales, plausible move ≤ ±500 %).
• Outliers — modified z-score > 3.5 on log prices, flagged and audited, never deleted; bundles excluded.
• Sales verification — heuristic label verified / likely / unverified / excluded from status, confidence, source type and trust; documented as heuristic.
• Indices — chain-linked trimmed equal-weight log returns, eligibility confidence ≥ 0.4 and ≥ 3 sales in 12 months, single-return clamp ln 2.5, base 1000 on 2024-01-01, minimum 10 constituents (RARE: 25 across published subindices); unpublished indices are shown as "in development".
• FX — ECB reference rates (frankfurter), sale-date rate for history, native values never overwritten.

# Architecture

TypeScript end-to-end in a pnpm monorepo.

                 ┌──────────────┐   HTTPS (MacLustr Tunnel: WireGuard + Caddy)
   browsers ───▶ │ apps/web     │   Next.js 16 · App Router · webpack build · Tailwind v4
                 │  :8210       │──┐  /api/v1/* rewritten to the API
                 └──────────────┘  │
                 ┌──────────────┐  │
   developers ─▶ │ apps/api     │◀─┘  Fastify · /v1 · API keys (hashed) · tiers · rate limits · CSV
                 │  :8211       │
                 └──────┬───────┘
                        │ SQL (drizzle-orm, raw sql for hot paths)
                 ┌──────▼───────────────────────────────────────────────┐
                 │ PostgreSQL 17 · pg_trgm · unaccent · pgvector · pgboss│
                 └──────▲───────────────────────────────────────────────┘
                        │
                 ┌──────┴───────┐   pg-boss queues: crawl.run · normalize.batch · resolve.batch
                 │ workers/     │   valuation.asset/rebuild · auctions.assess · indices.daily
                 │ (tsx)        │   snapshots.daily · radar.scan · health.compute · fx.sync
                 └──────┬───────┘   benchmarks.sync · images.process · listings.expire · certs.verify · account.jobs
                        │
                 ┌──────▼───────┐   212 connectors · router Direct API → Firecrawl → Scrapfly
                 │ connectors/  │   per-host policies, circuit breakers, quality scoring, fixtures
                 └──────────────┘

Packages: shared (schemas, money, stats, ids), database (Drizzle schema, migrations, seed), taxonomy (136 categories, condition scales, graders, grade parser, variant keys, indices), connectors (SDK, router, adapters Shopify / WooCommerce / sitemap / RSS / PDF / schema.org, health), valuation (RIV, outliers, grade premiums, scores, ask assessment, fees, auction assessment, depth, liquidation, verification), indices (chain-linked, repeat-sales, correlations), search (FTS + trigram + NL parser), ai (provider router Anthropic / OpenAI-compatible, cost ledger), notify (Resend e-mail).

Decisions: docs/adr/. Layout and data flow: docs/ARCHITECTURE.md.

# Repository layout

rareindex/
├── apps/
│   ├── web/                 Next.js site (src/app routes, components, lib/queries, brand assets, OG images)
│   └── api/                 Fastify public API (/v1), OpenAPI generator, key minting script
├── packages/
│   ├── shared/  database/  taxonomy/  connectors/  valuation/  indices/  search/  ai/  notify/
├── workers/                 pg-boss workers + operator CLI (cli.ts): crawler, normalizer, entity-resolution,
│                            valuation, auctions, indices, image-processing, account jobs, fees backfill…
├── connectors/              api/ · firecrawl/ · scrapfly/ — one folder per source (meta.json, index.ts, tests)
│                            registry.json (generated), domains.json + domains.d/ (host policies)
├── data/
│   ├── taxonomy/            categories.json · conditions.json · graders.json · indices.json
│   ├── fees/                auction-houses.json — buyer-premium schedules
│   ├── sources/             443 catalogued sources → sources.json → docs/connectors/SOURCES.md
│   └── fixtures/            real captured payloads for every connector (465 JSON, 22 HTML)
├── docs/                    ARCHITECTURE · METHODOLOGY · API · openapi.json · SPEC-original · adr/ · connectors/
├── scripts/                 build-registry · build-sources · connector-new · gen-openapi · ri
└── CLAUDE.md                condensed product mandate and engineering rules

# Pipeline and workers

source → raw_records (immutable, content-hashed)
       → normalize()            connector adapter → NormalizedRecord (sale | listing | auction_lot |
                                price_observation | catalog_item | population_report | news_item), Zod-validated
       → entity resolution      identifiers → canonical key → constrained trigram → create; grade re-parsed
                                from titles (refineGrade), variant ensured (never raw for a slab)
       → canonical writes       sales (dedupe key, sale-date FX, buyer-pays price), listings (lifecycle events),
                                auction lots, observations, populations, certificates
       → valuation              per-variant RIV, outlier flags, ask-vs-RIV with the anomaly gate,
                                representative variant, asset_stats / variant_stats / price_snapshots
       → auctions.assess        USD + all-in bid / estimate + bid vs RIV every 20 minutes
       → indices / snapshots    RARE + subindices, category snapshots, correlations, benchmarks
       → radar / alerts         record sales, price discrepancies, auctions below RIV, reappearances,
                                ultra-low populations; member alerts, digests, price targets

Operational safeguards: per-run time budgets, one crawl per connector at a time, orphan-run recovery, per-host concurrency and circuit breakers, challenge-page detection, schema-drift and result-collapse detectors feeding connector_health, audit log for every flag or move.

# Connectors

212 implemented connectors (179 api, 43 firecrawl, 8 scrapfly folders; 20 countries, 125 categories) over 443 catalogued sources (201 implemented, 10 gated behind API keys, 108 blocked by access controls that RareIndex will not circumvent, 108 rejected with a reason). The catalogue is docs/connectors/SOURCES.md.

A new source needs four things: meta.json, an extraction schema, a pure normalize() adapter and real fixtures with tests. Scaffold with pnpm connector:new <id> --url … [--adapter shopify|woocommerce], then pnpm registry, pnpm sources:build, pnpm db:seed. Guide: docs/connectors/ADDING_A_CONNECTOR.md.

Routing is Direct API → structured feed → Firecrawl → Scrapfly → manual review, with a quality gate (title 20 %, price 20 %, status 15 %, date 10 %, images 10 %, identifiers 10 %, description 5 %, category 5 %, currency 5 %). RareIndex never bypasses logins, paywalls, CAPTCHAs or other access controls.

# Public API

Base URL https://www.rareindex.io/api/v1 (proxied to Fastify). Bearer keys ri_live_… are stored hashed; anonymous calls get the public tier. Every list supports cursor pagination and ?format=csv; responses carry meta.as_of and attribution.

Reference: docs/API.md, docs/openapi.json, live at /api-docs.

# Quick start

Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with pgvector.

pnpm install
createdb rareindex
cp .env.example .env              # DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, ANTHROPIC_API_KEY, RESEND_API_KEY, SESSION_SECRET, ADMIN_TOKEN
pnpm db:migrate && pnpm db:seed   # extensions, tables, taxonomy, graders, indices, connector registry

pnpm dev:web                      # http://localhost:3000
pnpm dev:api                      # http://localhost:8211/v1
pnpm worker                       # pipeline workers
pnpm ri stats                     # operator CLI

Build and verify: pnpm typecheck, pnpm test, pnpm build. Schema change: edit packages/database/src/schema/*, pnpm db:generate, commit the migration.

# Operator CLI

pnpm ri <command> (on the node: node node_modules/tsx/dist/cli.mjs workers/cli.ts <command>):

# Testing and quality gates

• Unit tests (vitest) in every package and app: valuation (RIV, outliers, premiums, ask gate, fees, auction assessment, depth, liquidation, verification), taxonomy (grade parser, variant keys), search parser, indices, shared money/stats, workers, API, web helpers.
• 213 connector test files run every fixture through the schema, price, currency and date checks (pnpm connectors:test).
• pnpm -r run typecheck (TypeScript strict), pnpm --filter @rareindex/web run lint.
• Data-quality gates are code, not conventions: anomaly band, review threshold, plausible-change bound, representative-variant rule, numeric overflow guard, Data/identity anomaly and Needs review labels in the UI.

# Deployment

Since 2026-09-14 production runs on the OVH server BHS128b (ssh BHS128b, ubuntu@51.161.112.85, Ubuntu 24.04, 16 threads / 128 GB), outside mld: code in ~/apps/rareindex, PostgreSQL 17 native (db/role rareindex/rareindex, pgvector + pg_trgm + unaccent, en_US.UTF-8, ~150 GB), image store ~/apps/rareindex/data/images (270 GB, ~5 M files: orig + w96/w192/w384/w768/w1200, two-level hex sharding), PM2 under systemd (pm2-ubuntu) with ~/apps/pm2.rareindex.config.cjs generated from the mld manifest by ~/apps/.manifests/gen-pm2.py (secrets, 0600). PM2 processes: rareindex-web (8210), rareindex-api (8211), rareindex-worker (cron 04:30 + autorestart). Published through the MacLustr Tunnel: BHS64 Caddy → wg1 10.67.0.62:8210 (mlt add www.rareindex.io BHS128b:8210). Release on BHS128b: rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data --exclude logs . BHS128b:apps/rareindex/ then ssh BHS128b 'cd apps/rareindex && pnpm install --frozen-lockfile && DATABASE_URL=postgres://rareindex:rareindex@127.0.0.1:5432/rareindex pnpm db:migrate && DATABASE_URL=postgres://rareindex:rareindex@127.0.0.1:5432/rareindex NEXT_PUBLIC_SITE_URL=https://www.rareindex.io API_URL=http://127.0.0.1:8211 pnpm --filter @rareindex/web build && pm2 restart rareindex-api rareindex-web rareindex-worker && pm2 save'. pnpm ≥ 10 needs onlyBuiltDependencies (esbuild, sharp, @tailwindcss/oxide, unrs-resolver) in pnpm-workspace.yaml on a fresh install. Former home (kept as a cold copy, retired from mld with --keep-dir): MacLustr node M3U96a (Mac Studio, PostgreSQL 17 + pgvector), orchestrated by the mld gateway; the migration moved a 179 GB database (parallel pg_dump -Fd -j 4, 14 GB compressed, restored with pg_restore -j 8 in 21 min) and the 270 GB image store (rsync streams capped at 20 MB/s each so the Mac's 1 Gbit/s link stayed usable).

Release procedure: commit → git archive HEAD synced to the node → pnpm db:migrate → pnpm --filter @rareindex/web run build (detached, ~15 min: 146 static pages) → pm2 restart. Never delete .next under a running server; /data is rendered on demand with a one-hour cache because its aggregates exceed the build-time prerender budget.

# Brand

The mark is a cut stone — crown, girdle, pavilion — whose facets are drawn by an ascending index line: rarity and market data in one shape. Ink #0B0F19, gold #F1D08A → #C9963F, mint #2FB673 → #5FE3A1. Wordmark RareIndex with "Index" in gold. Source files in apps/web/public/brand/ (mark, light and dark lockups, README); favicons, PWA icons, favicon.ico and the Apple touch icon are generated from the mark by apps/web/scripts/brand-assets.mjs; the site-wide share image is rendered by apps/web/src/app/opengraph-image.tsx.

# Data honesty rules

• No fabricated data, ever: no invented sales, prices, populations, market caps, auctions, bids or listings.
• Observed data, model estimates, user data and external asks are always distinguished (§205).
• Price-guide values are observations, never transactions; a listing price is never a market value.
• Valuations publish only with sample size, confidence, range and timestamp; missing evidence reads "Not enough data".
• Outliers are flagged and audited, never deleted; every repair (regrade, flags) is in audit_log.
• Sale dates come from the source; historical prices use the exchange rate of the sale date; native values are never overwritten.
• Hammer prices are never compared with marketplace prices without the buyer's premium; the fee basis is shown.
• Neutral wording for anomalies ("anomalous transaction", "needs review"); RareIndex never accuses anyone of fraud and never claims to authenticate items.

Details: docs/adr/0002-data-honesty.md.

# Roadmap

Delivered on 2026-09-11: valuation safety (anomaly gate, variant-aware asks, representative variant, grade repair), screener, methodology center, market depth and time-to-sell, sale verification labels, buyer premiums and auction intelligence, brand. Next, in order of leverage:

1. Entity resolution for unique objects (auction lots on generic assets), certificate-number and multi-token search, unaccent in FTS.
2. fx_rate / fx_date on listings, observations and lots; per-lot bid counts for houses that omit them.
3. Index history reconstruction from transactions (ri value --all --history), repeat-sales publication.
4. Population reports through grader partnerships; cross-grader empirical model.
5. Portfolio overhaul (fees, liquidity horizon, confidence aggregation), Scanner multi-item and visual embeddings.
6. Cross-border arbitrage with FX, fees and duties; regional price indices.
7. Natural-language query engine over the screener; webhooks; API tier enforcement and shared rate-limit store.

# Author, hosting and contact

RareIndex is designed, built and operated by Simon-Pierre Boucher (Québec, Canada). It is hosted on MacLustr — www.maclustr.io — a distributed Apple-silicon compute cluster operated by the same team.

• General contact: contact@spboucher.ai
• Data corrections and source requests: data@rareindex.io · API and partnerships: hello@rareindex.io · Rights holders: legal@rareindex.io

# Disclaimers and license

RareIndex is an information platform. Valuations are estimates derived from observed public data. Listing prices and auction estimates are not confirmed transactions. Past performance does not guarantee future results. RareIndex does not authenticate items and does not provide investment advice. Third-party names, marks and content belong to their owners and are attributed to their sources; RareIndex respects source terms, robots policies, rate limits, copyright, privacy and database rights.

Proprietary — © 2026 Simon-Pierre Boucher / RareIndex. All rights reserved.