> Verbatim product mandate as provided on 2026-09-07. The repository CLAUDE.md is the condensed working version; this file is the authoritative full text. # CLAUDE.md ## RareIndex.io Global Collectibles Market Intelligence, Search, Pricing & Index Platform RareIndex.io must become the global reference platform for discovering, tracking, pricing, comparing, analyzing, and researching collectible assets. The ambition is significantly larger than a marketplace aggregator. RareIndex should become: The Bloomberg Terminal, Google, CoinMarketCap, Zillow, and StockX of the global collectibles economy. The platform must continuously collect public market data from marketplaces, auction houses, dealers, specialized websites, historical databases, grading companies, collector communities, manufacturer catalogs, archives, and other relevant sources. RareIndex should cover as many collectible categories as technically possible. The architecture must be built around: - Firecrawl - Scrapfly - direct APIs when available - structured feeds - official datasets - auction databases - marketplace connectors - AI extraction - multimodal image analysis - entity resolution - historical sale normalization - deduplication - valuation models - global collectible indices - category indices - individual asset price histories - collection tracking - alerts - comparable-sales analysis - rarity analysis - grading normalization - liquidity analysis - AI-assisted identification The platform language must be English-first. The application must be: - extremely fast - visually exceptional - mobile-first - responsive - data dense without feeling cluttered - professional - credible - research-grade - scalable to hundreds of millions of records ## 1. CORE PRODUCT VISION RareIndex should answer questions such as: - What is this collectible worth? - How rare is it? - Has its value increased? - What are the latest comparable sales? - Where is it currently for sale? - Which marketplace has the cheapest listing? - What is the global market price? - What is the price trend? - What is the liquidity? - How often does it sell? - Is the market accelerating or declining? - What grade commands the highest premium? - Is this listing unusually cheap? - Is this listing unusually expensive? - Is the collectible likely misidentified? - How many examples exist? - What are similar collectibles? - What are the most valuable objects in this category? - What collectibles are trending globally? - Which categories are outperforming? - Where are collectors moving capital? - How has Pokémon performed versus watches? - How has LEGO performed versus the S&P 500? - What are the best performing collectibles over 1Y / 5Y / 10Y? - What is the most liquid collectible market? - What items are becoming scarce? - What discontinued products are appreciating rapidly? RareIndex must transform fragmented collectible markets into structured financial-style data. ## 2. PRIMARY PLATFORM MODULES Main navigation: 1. Home 2. Explore 3. Markets 4. Categories 5. RareIndex 6. Trending 7. Sales 8. Listings 9. Auctions 10. Price History 11. Collections 12. Watchlist 13. Compare 14. Scanner 15. AI Research 16. Market Map 17. Grading 18. Auction Calendar 19. News 20. Data 21. API 22. About ## 3. RAREINDEX GLOBAL INDEX Create a flagship index: RareIndex Global Collectibles Index Ticker: RARE Possible subindices: - RARE-TCG - RARE-SPORT - RARE-WATCH - RARE-SNEAKER - RARE-LEGO - RARE-COMIC - RARE-GAME - RARE-ART - RARE-LUXURY - RARE-WINE - RARE-COIN - RARE-TOY - RARE-MUSIC - RARE-AUTO - RARE-TECH - RARE-BOOK - RARE-MILITARIA - RARE-SPACE - RARE-FOSSIL - RARE-DESIGN Each index should support: - current value - daily change - weekly change - monthly change - YTD - 1 year - 3 years - 5 years - 10 years - all time - volatility - drawdown - liquidity score - transaction count - median sale price - average sale price - market capitalization estimate - number of tracked assets - category breadth - momentum ## 4. COLLECTIBLE CATEGORY UNIVERSE The architecture MUST NOT hardcode a small number of collectible categories. Use an extensible taxonomy. Every category should support: - category - subcategory - franchise - brand - series - model - release - variant - year - edition - condition - grade - grading company - certification number - provenance - country - material - size - color - serial number - production quantity - rarity - population - original MSRP - estimated value - latest sale - listing price - currency - marketplace - images - description - source URLs ## 5. CATEGORY MASTER LIST RareIndex should attempt to cover all of the following. ### 5.1 Trading Cards Pokémon Track: - Base Set - Base Set 2 - Jungle - Fossil - Team Rocket - Gym Heroes - Gym Challenge - Neo Genesis - Neo Discovery - Neo Revelation - Neo Destiny - Legendary Collection - Expedition - Aquapolis - Skyridge - EX era - Diamond & Pearl - Platinum - HeartGold SoulSilver - Black & White - XY - Sun & Moon - Sword & Shield - Scarlet & Violet - Japanese expansions - Korean editions - Chinese editions - promotional cards - tournament cards - trophy cards - staff cards - prerelease cards - stamped cards - error cards - shadowless - first edition - unlimited - reverse holo - holo - secret rare - illustration rare - special illustration rare - gold cards - rainbow rares - alternate arts Grades: - PSA - BGS - CGC - SGC - TAG - raw ### 5.2 Magic: The Gathering Track: - Alpha - Beta - Unlimited - Revised - Arabian Nights - Antiquities - Legends - The Dark - Fallen Empires - Ice Age - Alliances - Mirage - Visions - Weatherlight - Tempest - Urza block - Mercadian Masques - Invasion - Odyssey - Onslaught - Mirrodin - Kamigawa - Ravnica - Lorwyn - Zendikar - Innistrad - Return to Ravnica - Theros - Khans - Battle for Zendikar - Kaladesh - Dominaria - War of the Spark - Modern Horizons - Commander - Secret Lair - serialized cards - artist proofs - reserved list - Judge promos - foils - etched foils - masterpieces - special treatments ### 5.3 Yu-Gi-Oh! Track: - Legend of Blue Eyes - Metal Raiders - Spell Ruler - Pharaoh's Servant - Labyrinth of Nightmare - Legacy of Darkness - tournament packs - championship cards - prize cards - Ghost Rare - Ultimate Rare - Collector Rare - Starlight Rare - Quarter Century - first edition - unlimited - Japanese cards - Korean cards - promotional cards ### 5.4 One Piece Card Game Track: - booster sets - starter decks - promotional cards - manga rares - alternate arts - tournament cards - championship cards - serialized cards - Japanese variants - English variants ### 5.5 Disney Lorcana Track: - First Chapter - Rise of the Floodborn - Into the Inklands - Ursula's Return - Shimmering Skies - future sets - Enchanted cards - promos - tournament cards - graded cards ### 5.6 Other TCGs Include: - Dragon Ball Super - Dragon Ball Masters - Digimon - Flesh and Blood - Weiss Schwarz - Final Fantasy TCG - Star Wars Unlimited - Star Wars CCG - Star Wars Destiny - Naruto - My Hero Academia - UniVersus - MetaZoo - Cardfight Vanguard - Battle Spirits - Force of Will - KeyForge - Chaotic - Duel Masters - Altered - Sorcery Contested Realm - Riftbound - Grand Archive - Gundam Card Game - Marvel trading cards - DC trading cards - Garbage Pail Kids - Topps entertainment cards - non-sport cards ## 6. SPORTS CARDS Sports: - baseball - basketball - hockey - American football - soccer - Formula 1 - UFC - boxing - golf - tennis - cricket - wrestling - NASCAR - Olympics - lacrosse - rugby Brands: - Topps - Panini - Upper Deck - Bowman - Fleer - Donruss - Score - Leaf - Futera Track: - rookie cards - numbered cards - parallels - refractors - autographs - patches - memorabilia cards - game-used cards - one-of-one cards - printing plates - redemptions - inserts - short prints - super short prints ## 7. COMICS Publishers: - Marvel - DC - Image - Dark Horse - IDW - Boom Studios - Archie - Valiant - Dynamite - EC Comics Track: - first appearances - first issues - key issues - variants - retailer exclusives - convention exclusives - signed copies - sketch covers - newsstand variants - direct editions - pedigree copies Grades: - CGC - CBCS - raw ## 8. MANGA Track: - first editions - first printings - limited printings - signed copies - Japanese original editions - English editions - French editions - box sets - special editions - promotional volumes Series examples: - Dragon Ball - One Piece - Naruto - Bleach - Berserk - Vagabond - Slam Dunk - Akira - Attack on Titan - Demon Slayer - Jujutsu Kaisen - Chainsaw Man - Hunter x Hunter - JoJo - Death Note - Sailor Moon ## 9. VIDEO GAMES Platforms: - Atari - Intellivision - ColecoVision - NES - Famicom - SNES - Super Famicom - Nintendo 64 - GameCube - Wii - Wii U - Switch - Game Boy - Game Boy Color - Game Boy Advance - Nintendo DS - 3DS - Sega Master System - Genesis - Mega Drive - Saturn - Dreamcast - Game Gear - PlayStation - PS2 - PS3 - PS4 - PS5 - PSP - Vita - Xbox - Xbox 360 - Xbox One - Xbox Series - Neo Geo - PC Engine - TurboGrafx - MSX - Commodore - Amiga - PC games Track: - sealed games - graded games - CIB - loose - prototypes - review copies - developer builds - kiosk demos - press kits - collector editions - limited editions - regional variants - misprints ## 10. GAMING HARDWARE Track: - consoles - handhelds - controllers - accessories - development kits - prototypes - retail demo units - special editions - limited editions - anniversary editions - transparent consoles - promotional consoles ## 11. ARCADE & PINBALL Track: - arcade cabinets - PCBs - conversion kits - marquees - control panels - pinball machines - prototypes - tournament machines - promotional units - replacement art - original parts ## 12. SNEAKERS Brands: - Nike - Jordan - Adidas - Yeezy - New Balance - ASICS - Reebok - Puma - Converse - Vans - Salomon - Hoka - BAPE - Supreme collaborations Track: - SKU - size - colorway - release date - original retail - resale - deadstock - used - samples - friends & family - player editions - prototypes - collaborations - region exclusives ## 13. WATCHES Brands: - Rolex - Patek Philippe - Audemars Piguet - Richard Mille - Vacheron Constantin - Omega - Cartier - Tudor - Breitling - IWC - Jaeger-LeCoultre - A. Lange & Söhne - Breguet - Blancpain - Zenith - Grand Seiko - Seiko - TAG Heuer - Panerai - Hublot - F.P. Journe - MB&F - De Bethune - Greubel Forsey - Urwerk Track: - reference - dial - bracelet - movement - year - production period - serial - papers - box - service history - auction history - provenance ## 14. LUXURY HANDBAGS Brands: - Hermès - Chanel - Louis Vuitton - Dior - Gucci - Goyard - Fendi - Bottega Veneta - Prada - Celine - Loewe - Saint Laurent Models: - Birkin - Kelly - Constance - Chanel Classic Flap - Chanel 2.55 - Louis Vuitton trunks - Speedy - Neverfull - rare runway pieces Track: - leather - hardware - size - year - stamp - condition - color - rarity ## 15. FASHION & STREETWEAR Brands: - Supreme - BAPE - Palace - Stüssy - Kith - Off-White - Chrome Hearts - Fear of God - Stone Island - Arc'teryx - Kapital - Visvim - Neighborhood - Undercover Track: - release - season - collaboration - size - condition - retail price - resale price ## 16. LEGO Track: - sealed sets - opened complete sets - minifigures - promotional sets - employee exclusives - convention exclusives - LEGO House exclusives - retired sets - prototypes Themes: - Star Wars - Technic - Icons - Ideas - Architecture - Modular Buildings - Castle - Pirates - Space - Harry Potter - Marvel - DC - Lord of the Rings - Indiana Jones - Ninjago - Bionicle - Monkie Kid ## 17. FUNKO Track: - Funko Pop - Soda - Wacky Wobblers - prototypes - convention exclusives - chase editions - vaulted products - signed products - error variants ## 18. DESIGNER TOYS Brands/artists: - Bearbrick - Medicom - KAWS - Kidrobot - Superplastic - Mighty Jaxx - Pop Mart - Labubu - Molly - Skullpanda - Dimoo - Banksy-related editions - Murakami collectibles ## 19. ACTION FIGURES Track: - Hot Toys - Sideshow - NECA - Hasbro - Mattel - McFarlane - Bandai - Tamashii Nations - S.H. Figuarts - Mafex - Mezco - Nendoroid - Figma Franchises: - Star Wars - Marvel - DC - Transformers - G.I. Joe - TMNT - Masters of the Universe - Power Rangers - Dragon Ball - One Piece - Gundam - Pokémon ## 20. VINTAGE TOYS Track: - Kenner - Matchbox - Hot Wheels - Dinky Toys - Corgi - Barbie - Fisher-Price - LEGO vintage - Playmobil - Transformers - G.I. Joe - Star Wars - He-Man - TMNT - My Little Pony - Care Bears - Polly Pocket - Micro Machines - Tamagotchi - Furby - Beanie Babies ## 21. DOLLS Track: - Barbie - Blythe - American Girl - Bratz - Monster High - antique dolls - bisque dolls - porcelain dolls - artist dolls - limited editions ## 22. PLUSH Track: - Steiff - Jellycat - Pokémon - Disney - Sanrio - Squishmallows - Beanie Babies - Build-A-Bear exclusives - vintage plush ## 23. MODEL CARS Brands: - Hot Wheels - Matchbox - Tomica - Mini GT - AUTOart - Kyosho - Minichamps - BBR - CMC - Spark - Tarmac Works - GreenLight - Maisto Scales: - 1:64 - 1:43 - 1:24 - 1:18 - 1:12 - 1:8 ## 24. MODEL TRAINS Track: - Märklin - Lionel - Hornby - Bachmann - Kato - Atlas - Fleischmann - vintage railway models - locomotives - cars - accessories - limited sets ## 25. AUTOMOBILES Collectible vehicle categories: - classic cars - supercars - hypercars - JDM - muscle cars - homologation specials - rally cars - race cars - concept cars - celebrity cars - low-production models Brands: - Ferrari - Porsche - Lamborghini - McLaren - Bugatti - Pagani - Koenigsegg - Mercedes-Benz - BMW - Audi - Ford - Chevrolet - Dodge - Nissan - Toyota - Honda - Mazda - Subaru - Mitsubishi ## 26. MOTORCYCLES Brands: - Ducati - Harley-Davidson - Honda - Yamaha - Kawasaki - Suzuki - BMW Motorrad - Triumph - Indian - MV Agusta Track: - rare models - production numbers - race provenance - celebrity ownership - original condition ## 27. AUTOMOTIVE MEMORABILIA Track: - steering wheels - race-used parts - engines - badges - hood ornaments - dealership signs - manuals - brochures - license plates - racing suits - helmets - signed parts ## 28. FORMULA 1 Track: - race-used helmets - gloves - suits - bodywork - steering wheels - signed memorabilia - cards - programs - tickets - model cars - team-issued clothing Drivers: - Ayrton Senna - Michael Schumacher - Lewis Hamilton - Max Verstappen - Sebastian Vettel - Fernando Alonso - Alain Prost - Niki Lauda - future drivers dynamically ## 29. SPORTS MEMORABILIA Sports: - hockey - baseball - basketball - football - soccer - tennis - golf - boxing - UFC - Olympics Assets: - jerseys - game-worn jerseys - match-issued jerseys - helmets - gloves - shoes - balls - pucks - bats - sticks - trophies - medals - tickets - autographs - photographs ## 30. COINS Track: - ancient coins - Roman coins - Greek coins - medieval coins - US coins - Canadian coins - European coins - Asian coins - bullion coins - commemoratives - proof coins - mint errors - pattern coins Grading: - PCGS - NGC - ANACS - ICCS ## 31. BANKNOTES Track: - world currency - historical notes - rare serial numbers - replacement notes - error notes - specimen notes - proof notes - obsolete currency ## 32. STAMPS Track: - world stamps - classic issues - rare errors - inverted centers - first-day covers - postal history - blocks - sheets - proofs ## 33. BOOKS Track: - first editions - first printings - signed editions - presentation copies - association copies - manuscripts - illuminated manuscripts - incunabula - banned editions - limited editions Authors across: - literature - fantasy - science fiction - philosophy - economics - science - children's literature - comic literature ## 34. HISTORICAL DOCUMENTS Track: - letters - manuscripts - government documents - presidential documents - royal documents - treaties - maps - deeds - certificates - newspapers - telegrams - handwritten notes ## 35. AUTOGRAPHS Categories: - politicians - scientists - astronauts - musicians - actors - athletes - authors - royalty - entrepreneurs - historical personalities ## 36. MUSIC Track: - vinyl - shellac - CDs - cassettes - master tapes - acetates - test pressings - promo copies - signed albums - rare pressings - colored vinyl - box sets Artists dynamically discovered. ## 37. MUSIC MEMORABILIA Track: - guitars - stage-used instruments - handwritten lyrics - posters - concert tickets - backstage passes - costumes - tour merchandise - signed items - awards - gold records ## 38. MUSICAL INSTRUMENTS Track: - Fender - Gibson - Martin - Gretsch - Rickenbacker - PRS - vintage synthesizers - Moog - Roland - Yamaha - Sequential - drum machines - microphones - effects pedals - amplifiers ## 39. MOVIE MEMORABILIA Track: - screen-used props - costumes - scripts - storyboards - production artwork - posters - lobby cards - miniatures - masks - weapons props - signed items ## 40. MOVIE POSTERS Track: - one sheets - half sheets - inserts - lobby cards - international posters - original theatrical posters - rare variants - recalled posters ## 41. ANIMATION ART Track: - production cels - backgrounds - concept art - key animation - storyboards - model sheets - signed cels Studios: - Disney - Warner Bros. - Studio Ghibli - Toei - Gainax - Madhouse - Production I.G - Pixar - DreamWorks ## 42. ART Track: - paintings - drawings - sculptures - prints - lithographs - screenprints - etchings - photographs - editions - multiples - artist proofs ## 43. CONTEMPORARY ART Track artists such as: - Banksy - KAWS - Takashi Murakami - Damien Hirst - Jeff Koons - Yayoi Kusama - Jean-Michel Basquiat - Keith Haring Do not hardcode value assumptions. ## 44. PHOTOGRAPHY Track: - signed prints - vintage prints - limited editions - artist proofs - contact sheets - negatives - historic photographs ## 45. CAMERAS Track: - Leica - Hasselblad - Rolleiflex - Contax - Nikon - Canon - Olympus - Polaroid - Fujifilm Include: - rare bodies - lenses - prototypes - military cameras - limited editions ## 46. APPLE COLLECTIBLES Track: - Apple I - Apple II - Macintosh - Lisa - Newton - iMac - iBook - PowerBook - Power Macintosh - iPod - iPhone - iPad - Apple Watch - prototypes - engineering samples - sealed devices - packaging - manuals - posters - Think Different campaign items - employee memorabilia ## 47. VINTAGE COMPUTERS Track: - Commodore - Amiga - Atari - IBM - NeXT - SGI - DEC - Sun Microsystems - Sinclair - Tandy - TRS-80 - Altair - early PCs - early laptops ## 48. COMPUTER HARDWARE Track collectible: - CPUs - GPUs - motherboards - sound cards - development hardware - engineering samples - vintage memory - rare graphics accelerators - boxed hardware - first-generation hardware ## 49. MOBILE PHONES Track: - early mobile phones - Nokia - Motorola - BlackBerry - Palm - first-generation iPhones - prototypes - limited editions - sealed products ## 50. KEYBOARDS Track: - vintage IBM keyboards - mechanical keyboards - custom keyboards - group buys - artisan keycaps - rare switches - prototype boards ## 51. AUDIO EQUIPMENT Track: - turntables - amplifiers - receivers - tape machines - speakers - headphones - vacuum tube equipment - studio equipment Brands: - Marantz - McIntosh - Technics - Nakamichi - Pioneer - Sony - JBL - Klipsch - Sennheiser ## 52. WINE Regions: - Bordeaux - Burgundy - Champagne - Rhône - Loire - Tuscany - Piedmont - Napa - Rioja - Mosel - Port Track: - producer - vintage - bottle size - provenance - fill level - storage - auction prices - critic score - lot composition ## 53. WHISKY Track: - Scotch - Japanese whisky - bourbon - Irish whiskey - Canadian whisky - closed distilleries - limited editions - single casks - numbered releases ## 54. RUM Track: - vintage rum - closed distilleries - limited editions - single casks - historic bottlings ## 55. COGNAC Track: - vintage Cognac - rare houses - limited editions - historic bottles ## 56. PERFUME Track: - vintage bottles - discontinued fragrances - limited editions - rare formulations - sealed bottles - Lalique bottles - Guerlain - Chanel - Dior - Hermès ## 57. JEWELRY Track: - Cartier - Van Cleef & Arpels - Tiffany - Bulgari - Graff - Harry Winston - vintage jewelry - signed jewelry - Art Deco jewelry ## 58. GEMSTONES Track: - diamonds - colored diamonds - rubies - sapphires - emeralds - alexandrite - spinel - opal - jadeite Capture certification and provenance. ## 59. MINERALS Track: - museum-quality specimens - crystals - rare minerals - type specimens - fluorescent minerals - historical specimens ## 60. METEORITES Track: - lunar meteorites - Martian meteorites - iron meteorites - pallasites - chondrites - witnessed falls - historic meteorites Capture weight and classification. ## 61. FOSSILS Track legally traded: - ammonites - trilobites - dinosaur fossils - dinosaur teeth - mammoth remains - shark teeth - fish fossils - plant fossils Include provenance and jurisdiction flags. ## 62. ANTIQUES Categories: - furniture - clocks - mirrors - ceramics - porcelain - silverware - glass - lighting - decorative arts - scientific instruments ## 63. PORCELAIN Brands: - Meissen - Sèvres - Wedgwood - Royal Copenhagen - Royal Worcester - Limoges - Herend ## 64. GLASS & CRYSTAL Track: - Baccarat - Lalique - Murano - Daum - Steuben - Waterford - antique glass ## 65. SILVER Track: - Tiffany - Christofle - Georg Jensen - sterling objects - antique silver - tea services - flatware ## 66. CLOCKS Track: - mantel clocks - carriage clocks - marine chronometers - astronomical clocks - grandfather clocks - industrial clocks - railway clocks ## 67. PENS Track: - Montblanc - Namiki - Pelikan - Parker - Waterman - Sailor - Aurora - limited editions - vintage fountain pens ## 68. LIGHTERS Track: - Zippo - S.T. Dupont - Dunhill - Cartier - vintage advertising lighters ## 69. ADVERTISING COLLECTIBLES Track: - Coca-Cola - Pepsi - Michelin - Texaco - Shell - Esso - automotive brands - tobacco historical advertising - neon signs - enamel signs - posters - clocks - vending machines ## 70. DISNEY COLLECTIBLES Track: - pins - park exclusives - animation art - figurines - costumes - props - attraction memorabilia - vintage toys - limited editions ## 71. SANRIO Track: - Hello Kitty - My Melody - Kuromi - Cinnamoroll - rare collaborations - vintage items - Japanese exclusives ## 72. STAR WARS Track virtually everything collectible: - toys - cards - comics - props - costumes - posters - LEGO - games - books - autographs - production memorabilia ## 73. MARVEL Track: - comics - cards - toys - statues - posters - movie props - LEGO - autographs ## 74. DC Same broad collectible structure. ## 75. HARRY POTTER Track: - books - first editions - props - costumes - LEGO - cards - wands - signed memorabilia ## 76. LORD OF THE RINGS Track: - books - first editions - props - miniatures - collectibles - autographs - LEGO ## 77. WARHAMMER Track: - miniatures - discontinued models - boxed armies - limited editions - Forge World - painted armies - tournament-winning armies - books - rulebooks - vintage miniatures ## 78. GUNDAM Track: - Gunpla - Perfect Grade - Master Grade - Real Grade - limited editions - event exclusives - Premium Bandai - vintage kits ## 79. TRAVEL COLLECTIBLES Track: - postcards - magnets - hotel memorabilia - airline memorabilia - cruise memorabilia - travel posters - luggage labels - tickets ## 80. AIRLINE COLLECTIBLES Track: - safety cards - menus - uniforms - aircraft models - amenity kits - china - silverware - seats - signs - retired airline memorabilia ## 81. AVIATION Track: - aircraft parts - cockpit instruments - propellers - aircraft models - pilot equipment - military aviation memorabilia - historic documentation ## 82. SPACE Track: - NASA memorabilia - mission patches - astronaut autographs - flown items - spacecraft components - mission documentation - Apollo memorabilia - Shuttle memorabilia - Soviet space memorabilia ## 83. MILITARIA Track legal historical collectibles: - medals - uniforms - helmets - insignia - documents - patches - flags - equipment - trench art Compliance restrictions must be implemented. ## 84. MEDALS Track: - military decorations - Olympic medals - sporting medals - civic medals - commemorative medals ## 85. OLYMPIC COLLECTIBLES Track: - medals - torches - pins - tickets - posters - uniforms - mascots - signed memorabilia ## 86. MAPS Track: - antique maps - atlases - globes - exploration maps - nautical charts - city plans - railway maps ## 87. POSTCARDS Track: - real photo postcards - historic cities - disasters - transportation - military - exhibitions - world's fairs ## 88. LICENSE PLATES Track: - antique plates - low numbers - vanity plates - diplomatic plates - historic jurisdictions - rare designs ## 89. PIN COLLECTIONS Track: - Disney - Olympics - Hard Rock - political - corporate - travel - military - sports - conventions ## 90. BOARD GAMES Track: - first editions - discontinued games - Kickstarter editions - prototypes - deluxe editions - signed editions ## 91. CHESS Track: - antique chess sets - tournament sets - artist sets - luxury sets - signed boards - historic pieces ## 92. PLAYING CARDS Track: - casino decks - designer decks - historical decks - rare printing variants - sealed decks ## 93. CASINO MEMORABILIA Only where legally appropriate: - casino chips - tokens - playing cards - signs - historic casino items ## 94. POLITICAL MEMORABILIA Track historically significant: - campaign buttons - posters - signs - signed documents - inauguration memorabilia - ballots - campaign literature Neutral presentation only. ## 95. ROYAL MEMORABILIA Track: - coronation items - signed photographs - correspondence - porcelain - medals - royal warrants - ceremonial objects ## 96. SCIENTIFIC INSTRUMENTS Track: - microscopes - telescopes - laboratory instruments - calculators - slide rules - surveying instruments - navigational instruments ## 97. TYPEWRITERS Track: - Underwood - Olivetti - IBM - Royal - Hermes - rare models - author-owned machines ## 98. VENDING MACHINES Track: - Coca-Cola - Pepsi - candy machines - cigarette machines - arcade vending - antique dispensers ## 99. HISTORIC CORPORATE MEMORABILIA Track: - early company stock certificates - employee badges - promotional objects - prototypes - corporate gifts - early packaging ## 100. DIGITAL COLLECTIBLES Potentially support separately: - NFTs - blockchain collectibles - tokenized physical assets - game skins - CS skins - digital trading cards - virtual items These markets require a separate valuation model due to extreme volatility. ## 101. FUTURE CATEGORY DISCOVERY RareIndex must automatically discover new collectible categories. Create an AI workflow that: 1. crawls marketplaces 2. identifies recurring product clusters 3. detects categories not represented in the taxonomy 4. measures transaction volume 5. proposes new taxonomy nodes 6. sends proposed categories to an admin queue 7. allows approval 8. automatically generates source connectors and schemas The taxonomy must evolve continuously. ## 102. CONNECTOR ARCHITECTURE RareIndex requires a professional connector framework. Every source must use a standardized connector interface. Example conceptual interface: ```ts interface RareIndexConnector { id: string name: string category: string[] countries: string[] languages: string[] crawl(): Promise normalize(records: RawRecord[]): Promise healthCheck(): Promise } ``` ## 103. FIRECRAWL Firecrawl should be the preferred connector engine for websites that can be reliably parsed using: - HTML - markdown extraction - structured data extraction - sitemap discovery - page crawling - search - dynamic rendering where supported Use Firecrawl for: - category pages - search results - public catalog pages - auction archives - sold-item databases - dealer inventory - pricing guides - manufacturer archives - grading information - collectible encyclopedias - editorial market data Firecrawl output should be converted into canonical JSON. ## 104. SCRAPFLY Scrapfly should act as: 1. secondary extraction engine 2. difficult-site connector 3. dynamic rendering layer 4. proxy/geo routing layer 5. anti-bot-compatible access layer where legally permitted 6. fallback when ordinary extraction fails Routing logic: ``` Direct API ↓ Structured Feed ↓ Firecrawl ↓ Scrapfly ↓ Browser Extraction ↓ Manual Review ``` Do NOT attempt to circumvent authentication, paywalls, CAPTCHAs, access controls, or other security protections unlawfully. Respect: - applicable laws - source terms - robots policies where applicable - copyright - privacy - rate limits - authentication boundaries ## 105. CONNECTOR FALLBACK ENGINE Every connector should produce health metrics: ```json { "connector": "example", "status": "healthy", "success_rate_24h": 0.982, "pages_attempted": 18752, "pages_success": 18414, "firecrawl_success_rate": 0.91, "scrapfly_fallback_rate": 0.07, "parse_failure_rate": 0.02, "last_success": "timestamp", "schema_version": "3.2" } ``` Automatically detect: - page redesigns - selector failures - missing fields - duplicate explosion - abnormal price distributions - blocked requests - empty pages - unexpected redirects - currency parsing problems ## 106. CONNECTOR REGISTRY Create: /connectors/registry.json Each connector: ```json { "id": "source-id", "displayName": "Source", "sourceType": "marketplace", "enginePriority": [ "api", "firecrawl", "scrapfly" ], "categories": [], "regions": [], "languages": [], "currency": [], "supportsListings": true, "supportsSold": true, "supportsAuctions": false, "supportsImages": true, "refreshFrequencyMinutes": 30 } ``` ## 107. SOURCE TYPES Support: - marketplace - auction_house - dealer - pricing_guide - grading_company - manufacturer - museum - archive - catalog - collector_database - community - forum - news - analytics_provider - registry - official_source ## 108. RAW INGESTION LAYER Never directly write scraped records into final product tables. Pipeline: ``` Source ↓ Raw HTML/API payload ↓ Extraction ↓ Raw JSON ↓ Normalization ↓ Entity Resolution ↓ Deduplication ↓ Price Validation ↓ Canonical Asset Match ↓ Sale/Listing Database ↓ Indices & Analytics ``` Raw records should be immutable. ## 109. CANONICAL ASSET MODEL Example: ```json { "asset_id": "rare_xxxxx", "category": "trading_cards", "subcategory": "pokemon", "brand": "Pokemon", "franchise": "Pokemon", "set": "Base Set", "name": "Charizard", "year": 1999, "number": "4/102", "variant": "1st Edition Holo", "language": "English", "grade": 10, "grader": "PSA", "certification_number": null, "population": null, "rarity": null, "images": [], "identifiers": {}, "metadata": {} } ``` ## 110. SALE MODEL ```json { "sale_id": "sale_xxxxx", "asset_id": "rare_xxxxx", "source": "source", "source_url": "", "sale_type": "auction", "sale_date": "", "sale_price": 0, "currency": "USD", "sale_price_usd": 0, "buyer_premium_included": false, "condition": "", "grade": "", "location": "", "image_urls": [], "raw_title": "", "confidence": 0.98 } ``` ## 111. LISTING MODEL Track: - asking price - seller - source - location - shipping - condition - grade - listing date - update date - availability - quantity - seller reputation where public - images - description Listing price MUST NOT automatically equal market value. ## 112. ENTITY RESOLUTION This is one of the most important parts of RareIndex. Different sources may describe the same collectible differently. Example: ``` 1999 Pokemon Base Charizard 1st PSA 10 Pokémon 1999 First Edition Charizard Holo PSA GEM MT 10 1999 Pokemon Game Charizard-Holo #4 PSA 10 ``` These must resolve to the same canonical asset variant. Use: - deterministic identifiers - manufacturer model numbers - card numbers - sets - release year - UPC - ISBN - SKU - certification - serial - visual similarity - text embeddings - multimodal embeddings - LLM verification ## 113. IMAGE INTELLIGENCE Images are fundamental. Build image pipelines for: - duplicate detection - visual identification - edition recognition - card recognition - shoe identification - watch reference recognition - LEGO identification - comic issue recognition - coin recognition - autograph comparison - condition estimation - box/seal recognition Generate multimodal embeddings. Store vectors in a vector database. ## 114. RAREINDEX SCANNER Allow users to: - upload a photo - take a smartphone photo - upload multiple photos - paste marketplace URL RareIndex AI should estimate: - category - item - edition - year - variation - likely condition - grade range where reasonable - rarity - market value - price range - latest sales - comparable sales - current listings Never claim professional authentication from AI alone. Display confidence. ## 115. PRICE ENGINE Never use only the average price. Calculate: - latest sale - median 5 sales - median 10 sales - median 20 sales - volume-weighted price - trimmed mean - exponentially weighted price - comparable-adjusted price - grade-adjusted price - condition-adjusted price RareIndex Fair Value: RIV = RareIndex Valuation Display: - RIV - low estimate - high estimate - confidence score ## 116. OUTLIER DETECTION Remove or flag: - obvious data errors - fake sales - cancelled auctions - bundle sales - incomplete items - incorrectly identified items - extremely abnormal prices - suspicious self-trades where detectable Never silently delete data. Maintain audit logs. ## 117. CONDITION NORMALIZATION Build category-specific condition scales. Cards: - raw - graded Games: - loose - CIB - sealed - graded Sneakers: - new - deadstock - lightly used - used Watches: - unworn - excellent - good - fair - poor - parts Books: - fine - very good - good - fair - poor Use category schemas. ## 118. GRADING NORMALIZATION Track grading company separately. Never treat: PSA 10 == BGS 10 == CGC 10 automatically. Measure market premiums empirically. Example: ```json { "grader": "PSA", "grade": "10", "market_multiplier": 2.31, "sample_size": 148 } ``` ## 119. POPULATION REPORTS Where grading population data is available, calculate: - total graded population - population by grade - gem rate - highest grade - grade scarcity - population growth Create: Population Momentum to show whether supply is expanding rapidly. ## 120. RARITY SCORE Create a RareIndex Rarity Score: 0–100 Possible factors: - production quantity - surviving population - graded population - number listed - listing frequency - sale frequency - historical scarcity - variant rarity - production errors - region exclusivity - promotional distribution Do not fabricate production counts. Use unknown when unavailable. ## 121. LIQUIDITY SCORE RareIndex Liquidity Score Factors: - transactions per month - number of marketplaces - number of active listings - bid depth if available - sale frequency - days between transactions - spread between asking and sold prices Scale: 0–100 ## 122. MOMENTUM SCORE Measure: - 7d - 30d - 90d - 1y Use: - price acceleration - transaction acceleration - search interest - listing velocity - auction activity ## 123. VALUE OPPORTUNITY SCORE Detect collectibles where listings are materially below estimated fair value. Example: ``` RIV Fair Value $3,850 Current Listing $2,950 Discount -23.4% Confidence 91% ``` Label this as analytical data, not investment advice. ## 124. MARKET CAPITALIZATION Estimate collectible market capitalization cautiously. Possible methods: Estimated population × representative market value Must display methodology and confidence. Never imply impossible precision. ## 125. CATEGORY MARKET PAGES Example: /markets/pokemon Display: - market index - market cap estimate - volume - number of sales - top gainers - top losers - highest volume - trending - biggest auctions - most expensive cards - newest records - population trends - listings - news ## 126. ASSET PAGE Every collectible should get a world-class asset page. Header: - title - hero image - category - year - variant - grade Metrics: - RareIndex Fair Value - latest sale - daily change - 30-day change - 1-year change - all-time high - all-time low - liquidity - rarity - active listings - sales last 30d Charts: - price history - transaction volume - active listings - grade distribution - population history - marketplace distribution Tabs: - Overview - Sales - Listings - Grades - Population - Images - History - Comparables - AI Analysis - Sources ## 127. GLOBAL SEARCH Search must handle natural-language requests. Examples: ``` 1999 Charizard PSA 10 Rolex Daytona Panda sealed Mario 64 first edition Harry Potter UK Michael Jordan rookie PSA 10 LEGO Millennium Falcon retired ``` Search across: - canonical assets - listings - historical sales - categories - brands - franchises - sets - auction houses ## 128. AI SEARCH Provide a conversational research interface. Examples: ``` Find Pokémon cards under $5,000 that have increased more than 30% in the last year. Compare Rolex Daytona versus Patek Nautilus over five years. Find sealed Nintendo games with fewer than 20 public sales in the last year. What collectible category has the strongest momentum this month? ``` AI must query structured internal data instead of hallucinating. ## 129. COLLECTION PORTFOLIO Users can create portfolios. Track: - acquisition date - purchase price - source - grade - serial/cert - photos - notes - current estimated value - gain/loss - portfolio allocation - category exposure Dashboard: ``` Collection Value $127,450 Cost Basis $91,200 Unrealized Gain $36,250 Return 39.7% ``` ## 130. COLLECTION ANALYTICS Show: - historical portfolio value - category diversification - concentration - best performer - worst performer - liquidity - rarity distribution - grading distribution ## 131. WATCHLIST Users can watch: - assets - categories - brands - sets - marketplaces - auctions ## 132. ALERTS Alerts: - price below threshold - price above threshold - new listing - new auction - auction ending - new record sale - unusual volume - market move - rare item appearance - population update ## 133. AUCTION CALENDAR Aggregate upcoming auctions. Views: - today - this week - this month Filters: - category - auction house - price - region ## 134. COMPARABLE SALES Every valuation should expose comps. For each comparable: - image - item - sale date - sale amount - source - grade - condition - similarity score ## 135. PRICE DISTRIBUTION Show: - minimum - 25th percentile - median - 75th percentile - maximum - trimmed mean Avoid hiding dispersion. ## 136. CURRENCY Store: - native transaction currency - historical FX rate - normalized USD - normalized CAD - EUR - GBP - JPY Allow users to choose display currency. Never convert historical sales using today's exchange rate. ## 137. INTERNATIONALIZATION Support later: - English - French - Japanese - Spanish - German - Italian - Chinese - Korean Canonical data remains language neutral. ## 138. SEARCH INFRASTRUCTURE Use a hybrid search system: - PostgreSQL - full-text search - vector search - optional OpenSearch/Elasticsearch at scale Ranking combines: - semantic relevance - lexical relevance - popularity - category match - historical volume - recency ## 139. DATABASE Recommended: - PostgreSQL - TimescaleDB where useful - object storage - Redis - vector storage Core tables: - assets - variants - brands - franchises - categories - listings - sales - auctions - sources - connectors - grades - population_reports - price_snapshots - valuations - indices - users - collections - collection_items - watchlists - alerts - images ## 140. EVENT-DRIVEN PIPELINE Use queues. Possible stack: - Redis Streams - BullMQ - Kafka at larger scale Events: - page_discovered - page_crawled - listing_created - listing_updated - sale_detected - entity_matched - valuation_updated - index_updated ## 141. CRAWL SCHEDULING Refresh cadence based on source importance. Examples: High-frequency: - active marketplaces - auctions ending soon - major card markets Medium-frequency: - dealers - auction catalogs - grading data Low-frequency: - historical archives - manufacturer catalogs Dynamic scheduling should prioritize sources with changing data. ## 142. DATA PROVENANCE Every displayed data point should be traceable. Store: - source - URL - crawl timestamp - original currency - parsed value - normalized value - parser version - connector version ## 143. SOURCE TRUST SCORE Score each source. Factors: - official source - historical consistency - sale verification - price reliability - duplicate rates - error rates ## 144. ADMIN CONNECTOR CENTER Create internal page: /admin/connectors Display: - connector - health - pages/min - errors - last crawl - records - duplicates - Firecrawl rate - Scrapfly rate - parser confidence - schema drift Buttons: - run - pause - retry - inspect - recrawl - edit mapping ## 145. AI CONNECTOR REPAIR When extraction breaks: 1. capture failed HTML 2. capture screenshot if permitted 3. compare old/new structure 4. ask LLM to identify schema changes 5. generate suggested parser patch 6. run fixture tests 7. deploy only after validation ## 146. CONNECTOR TEST SUITE Every source connector requires fixtures. Tests: - title extraction - price extraction - currency extraction - date extraction - image extraction - pagination - sold status - auction status - canonical URL - duplicate handling ## 147. MARKETPLACE DEDUPLICATION The same collectible may be cross-listed. Detect using: - title similarity - seller - images - price - serial numbers - descriptions - timestamps Generate: cross_listing_group_id ## 148. IMAGE DEDUPLICATION Compute: - perceptual hash - image embedding - similarity Detect reused marketplace photos. ## 149. FRAUD SIGNALS RareIndex can surface analytical warnings: - suspiciously low price - stolen/reused image likelihood - newly created seller when available - inconsistent description - wrong variant - fake certification formatting - duplicate certification Never claim definitive fraud without evidence. ## 150. DATA QUALITY SCORE Every record should receive: Data Quality: 0–100 Factors: - source trust - field completeness - entity confidence - image match - date confidence - price confidence ## 151. MARKET HEATMAP Create a visual global market map. Show performance by category. Example: ``` Pokemon +12.4% Sports Cards +4.8% Watches -2.1% Sneakers -7.3% LEGO +9.7% Video Games +6.1% Comics +3.5% Coins +1.2% ``` ## 152. TRENDING ALGORITHM Score: ``` Trending = Price Momentum × Volume Momentum × Search Momentum × Listing Momentum × News Momentum ``` Normalize values. ## 153. RAREINDEX 100 Create: RareIndex 100 The 100 most important collectible assets globally. Potential categories represented: - cards - watches - sneakers - LEGO - games - comics - coins - luxury goods - art - cars Do not select based purely on price. Consider: - liquidity - historical importance - data availability - global recognition - transaction frequency ## 154. RECORD SALES Create: /records Track: - highest collectible sale - highest Pokémon sale - highest sports card sale - highest game sale - highest comic sale - highest LEGO sale - highest watch sale - highest sneaker sale Use verified transactions only. ## 155. NEWLY DISCOVERED ITEMS Create: Rare Radar Detect: - first listing in years - ultra-low population - rare auction appearance - unusual price discrepancy - previously undocumented variation ## 156. MARKET NEWS Aggregate relevant collectible news. Categories: - auction results - record sales - grading changes - new releases - market trends - discoveries - collector events AI summarize with links to source. ## 157. PUBLIC API Future API: ``` GET /v1/assets/search GET /v1/assets/:id GET /v1/assets/:id/sales GET /v1/assets/:id/listings GET /v1/assets/:id/history GET /v1/categories GET /v1/indices GET /v1/markets GET /v1/trending ``` ## 158. API TIERS Possible: - free - hobby - professional - research - enterprise Rate limits configurable. ## 159. DATA EXPORT Allow exports: - CSV - JSON - Excel - PDF reports Professional users may export historical comps. ## 160. DESIGN LANGUAGE RareIndex must look like an institutional financial platform blended with a modern luxury marketplace. Avoid: - childish marketplace design - excessive gradients - crypto-style visual noise - giant empty cards - excessive rounded corners - generic AI SaaS appearance Target aesthetic: - Bloomberg - Sotheby's - Christie's - StockX - Linear - Stripe - Apple - modern financial terminals ## 161. COLOR SYSTEM Suggested neutral base: - white - off-white - near-black - dark charcoal - subtle gray Use accent colors only for: - gains - losses - index identity - rarity - alerts Support dark mode. ## 162. HOMEPAGE Hero: The Global Market for Collectibles Subtitle: Discover, price, track and analyze collectible assets across the world's marketplaces and auction houses. Search field: Search cards, watches, games, sneakers, LEGO, art... Below hero: - RareIndex Global - total tracked assets - indexed sales - marketplaces - latest transactions - trending categories ## 163. HOMEPAGE LIVE TAPE Create a financial-style ticker. Example: ``` RARE GLOBAL +1.28% POKÉMON +3.42% WATCHES -0.71% LEGO +2.11% COMICS +0.39% VIDEO GAMES +1.74% ``` ## 164. DISCOVERY GRID Homepage sections: - Trending Now - Biggest Movers - Record Sales - Rare Finds - Auctions Ending Soon - Most Watched - Newly Added - Market Insights ## 165. MOBILE DESIGN Mobile is not an afterthought. Must support: - camera scanner - swipeable charts - watchlists - collection tracking - price alerts - instant search - compact market cards Target: excellent experience on modern iPhone and Android devices. ## 166. PERFORMANCE Targets: - homepage interactive under 2 seconds on fast connection - cached search results under 300 ms where possible - optimized image delivery - lazy loading - server rendering - aggressive CDN caching ## 167. TECH STACK Recommended: Frontend: - Next.js - TypeScript - React - Tailwind - shadcn/ui - TanStack Query - TanStack Table Backend: - Node.js/TypeScript or Python services - PostgreSQL - Redis - object storage - queue workers Scraping: - Firecrawl - Scrapfly AI: provider abstraction supporting multiple models. Do not lock the architecture to one AI vendor. ## 168. AI MODEL ROUTER Create abstraction: ```ts interface ModelProvider { complete() extract() embed() vision() } ``` Possible roles: Fast/cheap models: - normalization - classification - schema matching Strong models: - difficult entity resolution - connector repair - research - image analysis ## 169. COST CONTROL Track AI and crawler cost per: - connector - category - source - record - user - API endpoint Create: /admin/costs ## 170. SCRAPING BUDGET ENGINE Avoid recrawling unchanged content. Use: - ETag - Last-Modified - content hashing - URL fingerprinting - change frequency estimates ## 171. CHANGE DETECTION For listing pages: ``` New Listing Price Changed Sold Removed Relisted Auction Ended ``` Store event history. ## 172. HISTORICAL DATA BACKFILL RareIndex should aggressively build historical datasets from legitimate publicly accessible archives. Store raw source timestamps. Never pretend crawl date equals sale date. ## 173. SEO Generate indexable pages for: - collectible - set - product - model - category - brand - year - marketplace - auction - price history Example: /pokemon/1999-base-set-charizard-4-1st-edition Metadata: - title - canonical description - latest price - chart preview ## 174. STRUCTURED DATA Use Schema.org where applicable: - Product - Offer - AggregateOffer - CollectionPage - Article - BreadcrumbList ## 175. USER AUTHENTICATION Support: - email - Google - Apple - passkeys ## 176. USER PRIVACY Do not sell private collection data. Private collections remain private by default. Allow explicit public portfolio option. ## 177. COMPLIANCE RareIndex is an information platform. Required disclaimers: - valuations are estimates - listing prices are not confirmed transactions - past performance does not guarantee future results - RareIndex does not authenticate items unless explicitly offering a separate verified authentication service Respect: - copyright - trademarks - privacy - database rights - marketplace terms - auction terms - API agreements - applicable scraping laws - regional regulations ## 178. SOURCE ATTRIBUTION Show source attribution where necessary. Do not falsely represent third-party content as RareIndex-owned. ## 179. PROHIBITED AUTOMATION Do not build systems designed to: - bypass login requirements - evade paywalls - defeat CAPTCHA protections - evade explicit authentication controls - access private user accounts - scrape private personal data - circumvent technical restrictions unlawfully Firecrawl and Scrapfly are extraction infrastructure, not authorization to violate access restrictions. ## 180. PHASE 1 Build foundation. Categories: 1. Pokémon 2. Sports Cards 3. Magic 4. Yu-Gi-Oh! 5. Watches 6. Sneakers 7. LEGO 8. Comics 9. Video Games 10. Funko / Designer Toys Build: - canonical models - connectors - sale ingestion - listings - search - price engine - asset pages - category pages ## 181. PHASE 2 Add: - coins - luxury handbags - wine - whisky - cars - music - toys - books - art Add: - portfolio - alerts - scanner - RareIndex indices ## 182. PHASE 3 Expand toward all collectible categories. Add: - AI category discovery - international sources - API - institutional analytics - historical backfills - research terminal ## 183. PHASE 4 RareIndex Terminal. Advanced screens: - market screener - correlation matrix - liquidity scanner - volatility ranking - historical returns - collectible factor analysis - price anomaly scanner ## 184. RAREINDEX SCREENER Filters: - category - price - market cap - rarity - liquidity - 7d performance - 30d performance - 1y performance - transaction volume - population - listings ## 185. COLLECTIBLE FACTORS Potential factors: - scarcity - brand - nostalgia - age - liquidity - momentum - grading - provenance - celebrity relevance - franchise strength Research relationships without presenting causality unless supported. ## 186. CORRELATION ENGINE Calculate collectible category correlation. Example matrix: ``` Cards Watches LEGO Games Art Cards 1.00 Watches 0.21 1.00 LEGO 0.38 0.14 1.00 Games 0.61 0.09 0.42 1.00 Art 0.08 0.37 0.10 0.05 1.00 ``` ## 187. BENCHMARKING Allow comparison against: - S&P 500 - NASDAQ - gold - Bitcoin - inflation - real estate indices Clearly explain differences in liquidity and methodology. ## 188. DATA SCIENCE Research models: - hedonic pricing - nearest-neighbor comps - gradient boosting - Bayesian valuation - time-series smoothing - repeat-sales indices - hierarchical models - multimodal embeddings ## 189. HEDONIC PRICING Useful for: - watches - cards - cars - sneakers - LEGO - games Features can capture: - age - edition - grade - rarity - brand - franchise - set - production - condition ## 190. REPEAT-SALES INDEX When the same identifiable asset trades multiple times, use repeat-sales methods to reduce composition bias. ## 191. DATA CONFIDENCE Every RareIndex statistic should have: - sample size - confidence - last update Example: ``` RareIndex Value $8,450 Confidence: High Sales Used: 37 Last Updated: 4 minutes ago ``` ## 192. DO NOT INVENT DATA Absolute rule. Never fabricate: - sale prices - production counts - grading populations - market caps - auctions - listing availability When data is missing: Data unavailable is better than hallucination. ## 193. OBSERVABILITY Use: - structured logs - traces - metrics - error reporting Track: - crawl latency - parsing errors - DB latency - AI cost - connector health - queue lag - valuation failures ## 194. SECURITY Use: - strict secrets management - server-side API credentials - encryption in transit - encryption at rest where appropriate - rate limits - abuse prevention - input validation Never expose: - Firecrawl keys - Scrapfly keys - AI keys - database credentials ## 195. REPOSITORY STRUCTURE Suggested: ``` rareindex/ ├── apps/ │ ├── web/ │ ├── api/ │ └── admin/ │ ├── packages/ │ ├── database/ │ ├── taxonomy/ │ ├── valuation/ │ ├── search/ │ ├── indices/ │ ├── ai/ │ ├── connectors/ │ └── shared/ │ ├── workers/ │ ├── crawler/ │ ├── normalizer/ │ ├── entity-resolution/ │ ├── valuation/ │ ├── image-processing/ │ └── indices/ │ ├── connectors/ │ ├── firecrawl/ │ ├── scrapfly/ │ ├── api/ │ ├── feeds/ │ └── registry.json │ ├── data/ │ ├── taxonomy/ │ └── fixtures/ │ ├── docs/ ├── scripts/ └── CLAUDE.md ``` ## 196. ENGINEERING RULES FOR CLAUDE CODE When implementing RareIndex: 1. Inspect the existing repository first. 2. Never overwrite working functionality unnecessarily. 3. Prefer reusable abstractions. 4. Avoid monolithic files. 5. Keep connector logic isolated. 6. Keep source-specific logic outside the canonical domain model. 7. Add tests for every connector. 8. Add migrations for DB changes. 9. Keep TypeScript strict. 10. Validate external payloads. 11. Use Zod or equivalent schemas. 12. Log extraction failures. 13. Never swallow exceptions silently. 14. Build responsive UI from the beginning. 15. Optimize database indexes. 16. Avoid premature microservices unless scale requires them. 17. Document major architectural decisions. 18. Never hardcode credentials. 19. Never invent fake production data. 20. Preserve traceability from every normalized datapoint to its source. ## 197. CONNECTOR IMPLEMENTATION RULE Every new marketplace should require creating only: - connector metadata - extraction schema - normalization adapter - test fixtures The rest of the ingestion pipeline should automatically work. ## 198. FIRECRAWL → SCRAPFLY ROUTER Implement a connector orchestration service. Pseudo-flow: ```ts async function fetchCollectiblePage(url, source) { const apiResult = await tryOfficialApi(source, url) if (apiResult.success) return apiResult const firecrawl = await tryFirecrawl(url) if ( firecrawl.success && firecrawl.qualityScore >= MIN_QUALITY ) { return firecrawl } const scrapfly = await tryScrapfly(url) if ( scrapfly.success && scrapfly.qualityScore >= MIN_QUALITY ) { return scrapfly } return { success: false, requiresReview: true } } ``` ## 199. EXTRACTION QUALITY Quality score based on expected fields. Example: ``` Title 20% Price 20% Status 15% Date 10% Images 10% Description 5% Identifiers 10% Category 5% Currency 5% ``` ## 200. RAREINDEX ULTIMATE GOAL RareIndex should eventually represent the complete global collectible economy as structured, queryable market data. A user should be able to type almost any collectible object in the world and immediately see: - what it is - what edition it is - how rare it is - its history - its latest sales - its current listings - what it is worth - how liquid it is - how its value has changed - where it is currently available - how it compares with similar objects - whether its market is gaining or losing momentum RareIndex must not merely aggregate listings. It must structure the collectible economy. The long-term objective is: One canonical asset graph containing every collectible, every important variation, every observable listing, every verified historical sale, and every meaningful market signal that can legally and reliably be obtained. RareIndex.io should become the default destination whenever someone asks: "What is this collectible worth?" or: "What is happening in the collectibles market?" ## 201. PRODUCT PRINCIPLE When choosing between: - more superficial features and - better data quality, always prioritize data quality. When choosing between: - displaying a number and - admitting insufficient evidence, admit insufficient evidence. When choosing between: - a generic marketplace and - a collectible intelligence platform, always build the intelligence platform. RareIndex should feel less like eBay and more like a professional market-data terminal built specifically for collectors. ## 202. FINAL MANDATE FOR CLAUDE CODE Build RareIndex incrementally but architect it from day one for massive expansion. Do not reduce the project to the initial ten categories. Those categories are only the launch wedge. The architecture must assume that RareIndex will eventually contain: - millions of collectible models - tens or hundreds of millions of listings - hundreds of millions of historical observations - thousands of data sources - hundreds of category families - global currencies - multilingual descriptions - billions of images or image references - continuous price updates Every technical decision should be evaluated against that end state. The finished platform should feel unprecedented in scope. RareIndex is the global data layer for collectible assets.