CLAUDE.md
RareIndex.io
Global Collectibles Market Intelligence, Search, Pricing & Index Platform
RareIndex.io must become the global reference platform for discovering, tracking, pricing, comparing, analyzing, and researching collectible assets.
The ambition is significantly larger than a marketplace aggregator.
RareIndex should become: the Bloomberg Terminal, Google, CoinMarketCap, Zillow, and StockX of the global collectibles economy.
The platform must continuously collect public market data from marketplaces, auction houses, dealers, specialized websites, historical databases, grading companies, collector communities, manufacturer catalogs, archives, and other relevant sources.
RareIndex should cover as many collectible categories as technically possible.
The architecture must be built around: Firecrawl · Scrapfly · direct APIs when available · structured feeds · official datasets · auction databases · marketplace connectors · AI extraction · multimodal image analysis · entity resolution · historical sale normalization · deduplication · valuation models · global collectible indices · category indices · individual asset price histories · collection tracking · alerts · comparable-sales analysis · rarity analysis · grading normalization · liquidity analysis · AI-assisted identification.
The platform language must be English-first.
The application must be: extremely fast · visually exceptional · mobile-first · responsive · data dense without feeling cluttered · professional · credible · research-grade · scalable to hundreds of millions of records.
Repository companion docs:
docs/ARCHITECTURE.md(what exists and how it fits),docs/adr/(decisions),connectors/README.md(how to add a source). Read them before changing structure.
1. CORE PRODUCT VISION
RareIndex should answer questions such as: What is this collectible worth? How rare is it? Has its value increased? What are the latest comparable sales? Where is it currently for sale? Which marketplace has the cheapest listing? What is the global market price? What is the price trend? What is the liquidity? How often does it sell? Is the market accelerating or declining? What grade commands the highest premium? Is this listing unusually cheap or expensive? Is the collectible likely misidentified? How many examples exist? What are similar collectibles? What are the most valuable objects in this category? What collectibles are trending globally? Which categories are outperforming? Where are collectors moving capital? How has Pokémon performed versus watches? How has LEGO performed versus the S&P 500? What are the best performing collectibles over 1Y / 5Y / 10Y? What is the most liquid collectible market? What items are becoming scarce? What discontinued products are appreciating rapidly?
RareIndex must transform fragmented collectible markets into structured financial-style data.
2. PRIMARY PLATFORM MODULES
Main navigation: 1 Home · 2 Explore · 3 Markets · 4 Categories · 5 RareIndex · 6 Trending · 7 Sales · 8 Listings · 9 Auctions · 10 Price History · 11 Collections · 12 Watchlist · 13 Compare · 14 Scanner · 15 AI Research · 16 Market Map · 17 Grading · 18 Auction Calendar · 19 News · 20 Data · 21 API · 22 About
3. RAREINDEX GLOBAL INDEX
Flagship index: RareIndex Global Collectibles Index, ticker RARE. Subindices: RARE-TCG, RARE-SPORT, RARE-WATCH, RARE-SNEAKER, RARE-LEGO, RARE-COMIC, RARE-GAME, RARE-ART, RARE-LUXURY, RARE-WINE, RARE-COIN, RARE-TOY, RARE-MUSIC, RARE-AUTO, RARE-TECH, RARE-BOOK, RARE-MILITARIA, RARE-SPACE, RARE-FOSSIL, RARE-DESIGN.
Each index should support: current value · daily change · weekly change · monthly change · YTD · 1 year · 3 years · 5 years · 10 years · all time · volatility · drawdown · liquidity score · transaction count · median sale price · average sale price · market capitalization estimate · number of tracked assets · category breadth · momentum.
4. COLLECTIBLE CATEGORY UNIVERSE
The architecture MUST NOT hardcode a small number of collectible categories. Use an extensible taxonomy (data/taxonomy/categories.json, categories table, taxonomy_proposals).
Every category should support: category · subcategory · franchise · brand · series · model · release · variant · year · edition · condition · grade · grading company · certification number · provenance · country · material · size · color · serial number · production quantity · rarity · population · original MSRP · estimated value · latest sale · listing price · currency · marketplace · images · description · source URLs.
5–100. CATEGORY MASTER LIST
RareIndex should attempt to cover all of the following (each is a taxonomy family; details of sets, brands, tracked attributes and graders are captured in the taxonomy files and the connectors):
5 Trading Cards (Pokémon — all eras, languages, promos, error/shadowless/1st edition, graded PSA/BGS/CGC/SGC/TAG/raw; Magic: The Gathering — Alpha→current, reserved list, promos, foils, serialized; Yu-Gi-Oh!; One Piece; Disney Lorcana; other TCGs) · 6 Sports Cards (all sports, Topps/Panini/Upper Deck/Bowman/Fleer/Donruss/Score/Leaf/Futera; rookies, numbered, parallels, autos, patches, 1/1, plates) · 7 Comics (Marvel, DC, Image, Dark Horse, IDW, Boom, Archie, Valiant, Dynamite, EC; keys, variants, pedigrees; CGC/CBCS/raw) · 8 Manga · 9 Video Games (all platforms; sealed/graded/CIB/loose/prototypes) · 10 Gaming Hardware · 11 Arcade & Pinball · 12 Sneakers · 13 Watches · 14 Luxury Handbags · 15 Fashion & Streetwear · 16 LEGO · 17 Funko · 18 Designer Toys · 19 Action Figures · 20 Vintage Toys · 21 Dolls · 22 Plush · 23 Model Cars · 24 Model Trains · 25 Automobiles · 26 Motorcycles · 27 Automotive Memorabilia · 28 Formula 1 · 29 Sports Memorabilia · 30 Coins (PCGS/NGC/ANACS/ICCS) · 31 Banknotes · 32 Stamps · 33 Books · 34 Historical Documents · 35 Autographs · 36 Music · 37 Music Memorabilia · 38 Musical Instruments · 39 Movie Memorabilia · 40 Movie Posters · 41 Animation Art · 42 Art · 43 Contemporary Art (do not hardcode value assumptions) · 44 Photography · 45 Cameras · 46 Apple Collectibles · 47 Vintage Computers · 48 Computer Hardware · 49 Mobile Phones · 50 Keyboards · 51 Audio Equipment · 52 Wine · 53 Whisky · 54 Rum · 55 Cognac · 56 Perfume · 57 Jewelry · 58 Gemstones (certification + provenance) · 59 Minerals · 60 Meteorites (weight + classification) · 61 Fossils (legally traded; provenance and jurisdiction flags) · 62 Antiques · 63 Porcelain · 64 Glass & Crystal · 65 Silver · 66 Clocks · 67 Pens · 68 Lighters · 69 Advertising Collectibles · 70 Disney · 71 Sanrio · 72 Star Wars · 73 Marvel · 74 DC · 75 Harry Potter · 76 Lord of the Rings · 77 Warhammer · 78 Gundam · 79 Travel · 80 Airline · 81 Aviation · 82 Space · 83 Militaria (legal; compliance restrictions implemented) · 84 Medals · 85 Olympic · 86 Maps · 87 Postcards · 88 License Plates · 89 Pins · 90 Board Games · 91 Chess · 92 Playing Cards · 93 Casino Memorabilia (where legal) · 94 Political Memorabilia (neutral presentation only) · 95 Royal Memorabilia · 96 Scientific Instruments · 97 Typewriters · 98 Vending Machines · 99 Historic Corporate Memorabilia · 100 Digital Collectibles (separate valuation model).
101. FUTURE CATEGORY DISCOVERY
RareIndex must automatically discover new collectible categories: crawl marketplaces → identify recurring product clusters → detect categories missing from the taxonomy → measure transaction volume → propose taxonomy nodes → admin queue → approval → auto-generate source connectors and schemas. The taxonomy must evolve continuously.
102. CONNECTOR ARCHITECTURE
Every source uses the standardized interface (packages/connectors/src/types.ts):
interface RareIndexConnector {
meta: ConnectorMeta // id, name, categories, countries, languages…
crawl(ctx): AsyncIterable<RawRecordInput>
normalize(raw): Promise<NormalizedRecord[]>
healthCheck(ctx): Promise<ConnectorHealth>
}103. FIRECRAWL
Preferred engine for websites reliably parsed with HTML / markdown / structured extraction / sitemap discovery / crawling / search / dynamic rendering: category pages, search results, public catalogs, auction archives, sold-item databases, dealer inventory, pricing guides, manufacturer archives, grading info, encyclopedias, editorial market data. Output is converted into canonical JSON.
104. SCRAPFLY
Secondary extraction engine, difficult-site connector, dynamic rendering layer, proxy/geo routing layer, anti-bot-compatible access layer where legally permitted, fallback when ordinary extraction fails.
Routing: Direct API → Structured Feed → Firecrawl → Scrapfly → Browser Extraction → Manual Review.
Do NOT attempt to circumvent authentication, paywalls, CAPTCHAs, access controls, or other security protections unlawfully. Respect applicable laws, source terms, robots policies where applicable, copyright, privacy, rate limits, authentication boundaries.
105. CONNECTOR FALLBACK ENGINE
Every connector produces health metrics (ConnectorHealth: status, success_rate_24h, pages_attempted/success, firecrawl_success_rate, scrapfly_fallback_rate, parse_failure_rate, last_success, schema_version). Automatically detect page redesigns, selector failures, missing fields, duplicate explosion, abnormal price distributions, blocked requests, empty pages, unexpected redirects, currency parsing problems.
106. CONNECTOR REGISTRY
/connectors/registry.json (generated from per-connector meta.json): id, displayName, sourceType, enginePriority, categories, regions, languages, currency, supportsListings/Sold/Auctions/Images, refreshFrequencyMinutes.
107. SOURCE TYPES
marketplace · auction_house · dealer · pricing_guide · grading_company · manufacturer · museum · archive · catalog · collector_database · community · forum · news · analytics_provider · registry · official_source
108. RAW INGESTION LAYER
Never directly write scraped records into final product tables. Source → Raw payload → Extraction → Raw JSON → Normalization → Entity Resolution → Deduplication → Price Validation → Canonical Asset Match → Sale/Listing Database → Indices & Analytics. Raw records are immutable.
109–111. CANONICAL MODELS
Canonical asset (assets + asset_variants), sale (sales: source, url, type, date, price, currency, price_usd, buyer premium, condition, grade, location, images, raw title, confidence) and listing (listings: asking price, seller, source, location, shipping, condition, grade, dates, availability, quantity, seller reputation, images, description). Listing price MUST NOT automatically equal market value.
112. ENTITY RESOLUTION
Different sources describe the same collectible differently; they must resolve to the same canonical asset variant using deterministic identifiers, model numbers, card numbers, sets, year, UPC/ISBN/SKU, certification, serial, visual similarity, text and multimodal embeddings, LLM verification.
113–114. IMAGE INTELLIGENCE & SCANNER
Image pipelines for duplicate detection, identification, edition recognition, condition estimation, box/seal recognition; multimodal embeddings stored in pgvector. The Scanner accepts photos or a marketplace URL and estimates category, item, edition, year, variation, condition, grade range, rarity, value range, sales, comps and listings — always with confidence, never claiming professional authentication.
115–124. PRICE ENGINE & SCORES
Never use only the average. Compute latest sale, median 5/10/20, volume-weighted, trimmed mean, exponentially weighted, comparable-, grade- and condition-adjusted prices → RIV (RareIndex Valuation) with low/high estimate and confidence. Flag (never silently delete) outliers, fake/cancelled/bundle sales, misidentified items; keep audit logs. Category-specific condition scales; grading companies tracked separately with empirical premiums (PSA 10 ≠ BGS 10 ≠ CGC 10). Population reports → gem rate, scarcity, Population Momentum. Rarity Score 0–100 (unknown when data missing). Liquidity Score 0–100. Momentum 7d/30d/90d/1y. Value Opportunity Score (analytical data, not investment advice). Market capitalization only with methodology and confidence.
125–136. PAGES, SEARCH, PORTFOLIO, ALERTS, AUCTIONS, COMPS, CURRENCY
Category market pages (/markets/<slug>), world-class asset pages (header, metrics, charts, tabs Overview/Sales/Listings/Grades/Population/Images/History/Comparables/AI Analysis/Sources), natural-language global search, conversational AI research over structured internal data (no hallucination), collection portfolios with analytics, watchlists, alerts, auction calendar, comparable sales with similarity scores, price distribution (min/p25/median/p75/max/trimmed mean). Store native currency + historical FX + USD/CAD/EUR/GBP/JPY; never convert historical sales at today's rate.
137–159. PLATFORM
Internationalization later (canonical data language neutral). Hybrid search (PostgreSQL FTS + vectors, OpenSearch later). PostgreSQL + object storage + Redis/queues + vector storage. Event-driven pipeline (page_discovered, page_crawled, listing_created, listing_updated, sale_detected, entity_matched, valuation_updated, index_updated). Dynamic crawl scheduling by source importance. Full data provenance. Source trust score. /admin/connectors control center. AI connector repair with fixture validation. Connector test suite with fixtures. Marketplace and image deduplication (cross_listing_group_id, perceptual hashes). Fraud signals (analytical, never definitive). Data Quality 0–100. Market heatmap, trending algorithm, RareIndex 100, record sales, Rare Radar, market news, public API /v1/... with tiers, exports (CSV/JSON/Excel/PDF).
160–166. DESIGN
Institutional financial platform × modern luxury marketplace (Bloomberg, Sotheby's, Christie's, StockX, Linear, Stripe, Apple). Neutral base; accents only for gains, losses, index identity, rarity, alerts; dark mode. Homepage: "The Global Market for Collectibles" — search — live tape — stats — discovery grid (Trending Now, Biggest Movers, Record Sales, Rare Finds, Auctions Ending Soon, Most Watched, Newly Added, Market Insights). Mobile is not an afterthought. Performance: homepage interactive < 2 s, cached search < 300 ms, optimized images, SSR, CDN caching.
167–171. TECH, AI ROUTER, COST, BUDGET, CHANGE DETECTION
Next.js/TypeScript/React/Tailwind/TanStack; Node/TypeScript services; PostgreSQL; Redis/queues; object storage; Firecrawl + Scrapfly; AI provider abstraction (ModelProvider { complete, extract, embed, vision }) never locked to one vendor; cost tracking per connector/category/source/record/user/endpoint (/admin/costs); scraping budget engine (ETag, Last-Modified, content hashing, change-frequency estimates); listing change detection (New, Price Changed, Sold, Removed, Relisted, Auction Ended) with event history.
172–179. BACKFILL, SEO, AUTH, PRIVACY, COMPLIANCE
Aggressive but legitimate historical backfills with real source timestamps. Indexable SEO pages with Schema.org (Product, Offer, AggregateOffer, CollectionPage, Article, BreadcrumbList). Auth: email, Google, Apple, passkeys. Private collections private by default. Disclaimers: valuations are estimates; listing prices are not confirmed transactions; past performance does not guarantee future results; RareIndex does not authenticate items. Source attribution. Prohibited automation: no bypassing logins/paywalls/CAPTCHAs/authentication controls, no private data scraping, no unlawful circumvention.
180–183. PHASES
Phase 1 foundation: Pokémon, Sports Cards, Magic, Yu-Gi-Oh!, Watches, Sneakers, LEGO, Comics, Video Games, Funko/Designer Toys — canonical models, connectors, sale ingestion, listings, search, price engine, asset pages, category pages. Phase 2: coins, handbags, wine, whisky, cars, music, toys, books, art + portfolio, alerts, scanner, indices. Phase 3: all categories, AI category discovery, international sources, API, institutional analytics, backfills, research terminal. Phase 4: RareIndex Terminal (screener, correlation matrix, liquidity scanner, volatility ranking, historical returns, factor analysis, anomaly scanner).
184–191. TERMINAL, FACTORS, CORRELATION, BENCHMARKS, DATA SCIENCE, CONFIDENCE
Screener filters; collectible factors (research relationships without asserting causality); category correlation matrix; benchmarking vs S&P 500, NASDAQ, gold, Bitcoin, inflation, real estate (explain liquidity/methodology differences); research models (hedonic, nearest-neighbor comps, gradient boosting, Bayesian, time-series smoothing, repeat-sales, hierarchical, multimodal embeddings). Every statistic shows sample size, confidence, last update.
192. DO NOT INVENT DATA
Absolute rule. Never fabricate sale prices, production counts, grading populations, market caps, auctions, listing availability. When data is missing, "Data unavailable" is better than hallucination.
193–194. OBSERVABILITY & SECURITY
Structured logs, traces, metrics, error reporting (crawl latency, parsing errors, DB latency, AI cost, connector health, queue lag, valuation failures). Strict secrets management, server-side credentials only, encryption, rate limits, abuse prevention, input validation. Never expose Firecrawl, Scrapfly, AI keys or database credentials.
195. REPOSITORY STRUCTURE
rareindex/
├── apps/ (web, api, admin→ inside web /admin for Phase 1)
├── packages/ (database, taxonomy, valuation, search, indices, ai, connectors, shared)
├── workers/ (crawler, normalizer, entity-resolution, valuation, image-processing, indices)
├── connectors/ (firecrawl, scrapfly, api, feeds, registry.json)
├── data/ (taxonomy, fixtures)
├── docs/ · scripts/ · CLAUDE.md196. ENGINEERING RULES FOR CLAUDE CODE
1 Inspect the existing repository first. 2 Never overwrite working functionality unnecessarily. 3 Prefer reusable abstractions. 4 Avoid monolithic files. 5 Keep connector logic isolated. 6 Keep source-specific logic outside the canonical domain model. 7 Add tests for every connector. 8 Add migrations for DB changes. 9 Keep TypeScript strict. 10 Validate external payloads. 11 Use Zod. 12 Log extraction failures. 13 Never swallow exceptions silently. 14 Build responsive UI from the beginning. 15 Optimize database indexes. 16 Avoid premature microservices. 17 Document major architectural decisions. 18 Never hardcode credentials. 19 Never invent fake production data. 20 Preserve traceability from every normalized datapoint to its source.
197. CONNECTOR IMPLEMENTATION RULE
A new marketplace requires only: connector metadata · extraction schema · normalization adapter · test fixtures. The rest of the ingestion pipeline works automatically.
198–199. ROUTER & EXTRACTION QUALITY
fetchCollectiblePage: official API → Firecrawl (quality ≥ MIN) → Scrapfly (quality ≥ MIN) → requiresReview. Quality weights: Title 20% · Price 20% · Status 15% · Date 10% · Images 10% · Description 5% · Identifiers 10% · Category 5% · Currency 5%.
200–202. ULTIMATE GOAL, PRINCIPLE, MANDATE
One canonical asset graph containing every collectible, every important variation, every observable listing, every verified historical sale, and every meaningful market signal that can legally and reliably be obtained. Always prioritize data quality over superficial features; admit insufficient evidence rather than display a number; build the intelligence platform, not a generic marketplace. Build incrementally but architect from day one for millions of models, hundreds of millions of observations, thousands of sources, global currencies, multilingual descriptions and billions of image references. RareIndex is the global data layer for collectible assets.
Repository conventions (added by the implementation)
- Package manager
pnpm;pnpm typecheck,pnpm test,pnpm build. Workers run withtsx. - Secrets only in
.env(git-ignored) or the PM2 config on the server (BHS128b:~/apps/pm2.rareindex.config.cjs, generated from the formermldmanifest kept in~/apps/.manifests/). Never in code or fixtures. - Production (since 2026-09-14): OVH server BHS128b, outside
mld— Postgres 17 native, image store~/apps/rareindex/data/images(270 GB), PM2 under systemd, routehttps://www.rareindex.io → wg1 10.67.0.62:8210. Release procedure and details inREADME.md→ Deployment. The M3U96a copy is a cold copy. - Add a connector:
pnpm connector:new <id> --url … [--adapter shopify|woocommerce]→connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts,README.md}+ real fixtures indata/fixtures/<id>/, source entry indata/sources/entries/<group>.json, host policy inconnectors/domains.d/<group>.json; thenpnpm registry,pnpm sources:build,pnpm db:seed. Guide:docs/connectors/ADDING_A_CONNECTOR.md; catalogue:docs/connectors/SOURCES.md. - DB changes: edit
packages/database/src/schema/*, runpnpm db:generate, commit the SQL migration. - UI: dense, neutral, tabular numbers (
numutility),<Unavailable/>for missing evidence, every number with its sample size / confidence / freshness when it is an estimate. - Member accounts: email + password, email verification code and multi-factor code delivered through Resend (
RESEND_API_KEY,EMAIL_FROM).