README: full platform documentation with the RareIndex brand (logo, product surface, data model, methodology, architecture, pipeline, connectors, API, CLI, deployment, roadmap, author and hosting on MacLustr)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +386 −56
modified
README.md
+386 −56
@@ -1,88 +1,418 @@ | |||||
| 1 | +<p align="center"> | ||||
| 2 | + <picture> | ||||
| 3 | + <source media="(prefers-color-scheme: dark)" srcset="apps/web/public/brand/rareindex-logo-dark.svg"> | ||||
| 4 | + <img src="apps/web/public/brand/rareindex-logo.svg" alt="RareIndex — The Global Market for Collectibles" width="520"> | ||||
| 5 | + </picture> | ||||
| 6 | +</p> | ||||
| 7 | + | ||||
| 8 | +<p align="center"> | ||||
| 9 | + <a href="https://www.rareindex.io"><strong>www.rareindex.io</strong></a> · | ||||
| 10 | + <a href="https://www.rareindex.io/methodology">Methodology</a> · | ||||
| 11 | + <a href="https://www.rareindex.io/api-docs">API</a> · | ||||
| 12 | + <a href="https://www.rareindex.io/screener">Screener</a> · | ||||
| 13 | + <a href="https://www.rareindex.io/auctions">Auctions</a> · | ||||
| 14 | + <a href="mailto:contact@spboucher.ai">contact@spboucher.ai</a> | ||||
| 15 | +</p> | ||||
| 16 | + | ||||
| 17 | +<p align="center"> | ||||
| 18 | + <img alt="TypeScript" src="https://img.shields.io/badge/TypeScript-strict-3178c6"> <img alt="Next.js 16" src="https://img.shields.io/badge/Next.js-16-000000"> <img alt="Fastify" src="https://img.shields.io/badge/Fastify-%2Fv1-202020"> <img alt="PostgreSQL 17" src="https://img.shields.io/badge/PostgreSQL-17%20%2B%20pgvector-336791"> <img alt="pg-boss" src="https://img.shields.io/badge/workers-pg--boss-4b5563"> <img alt="connectors" src="https://img.shields.io/badge/connectors-212-c9963f"> <img alt="hosted on MacLustr" src="https://img.shields.io/badge/hosted%20on-MacLustr-0b0f19"> | ||||
| 19 | +</p> | ||||
| 20 | + | ||||
| 1 | # RareIndex | 21 | # RareIndex | ||
| 2 | 22 | ||||
| 3 | −**The global market for collectibles.** RareIndex.io is a market-intelligence, search, pricing and | 23 | +**RareIndex is the market-data and intelligence layer for physical collectibles** — the terminal the | ||
| 4 | −index platform for collectible assets: trading cards, sports cards, watches, sneakers, LEGO, comics, | 24 | +collectibles economy never had. It continuously collects public market evidence (marketplaces, | ||
| 5 | −video games, toys, coins, art, wine and, over time, every category the collectibles economy contains. | 25 | +auction houses, dealers, price guides, grading companies, manufacturer catalogs), resolves it into a | ||
| 6 | −It continuously collects public market data from marketplaces, auction houses, dealers, grading | 26 | +canonical asset graph and turns it into research-grade data: a valuation with a confidence level and | ||
| 7 | −companies, manufacturer catalogs and archives, resolves it into one canonical asset graph, and turns | 27 | +a range, comparable sales, live asks, market depth, liquidity and rarity scores, auction intelligence | ||
| 8 | −it into research-grade data: valuations with confidence levels, comparable sales, live listings, | 28 | +with buyer-pays costs, price histories and the **RARE** family of indices. | ||
| 9 | −rarity and liquidity scores, price histories, and the RareIndex family of indices (`RARE` and its | 29 | + | ||
| 10 | −category subindices). | 30 | +A marketplace tells you *"this item is for sale for $4,000"*. RareIndex tells you: | ||
| 11 | − | 31 | + | ||
| 12 | −RareIndex is an intelligence platform, not a marketplace aggregator. Every number shown is traceable | 32 | +> *This asset has a RareIndex Valuation of $3,180 (high confidence, 42 comparable sales of the same | ||
| 13 | −to its source, every estimate carries a sample size and a confidence level, and when the evidence is | 33 | +> variant across five sources), the market is up 12.4 % over 12 months, typical time-to-sale is 18 | ||
| 14 | −insufficient the platform says "Data unavailable" rather than inventing a figure. The full product | 34 | +> days, liquidity is above the category average, and the current $2,650 ask is 16.7 % below fair | ||
| 15 | −mandate is in [`docs/SPEC-original.md`](docs/SPEC-original.md); the condensed working version that | 35 | +> value — while a $472 slab of a different grade matched to the same card is a data anomaly, not a | ||
| 16 | −guides implementation is [`CLAUDE.md`](CLAUDE.md). | 36 | +> deal.* | ||
| 37 | + | ||||
| 38 | +Every number is traceable to its source, carries a sample size, a confidence and a freshness, and | ||||
| 39 | +when the evidence is not there the platform says **"Not enough data"** instead of inventing a figure. | ||||
| 40 | + | ||||
| 41 | +--- | ||||
| 42 | + | ||||
| 43 | +## Table of contents | ||||
| 44 | + | ||||
| 45 | +1. [What RareIndex answers](#what-rareindex-answers) | ||||
| 46 | +2. [Live platform at a glance](#live-platform-at-a-glance) | ||||
| 47 | +3. [Product surface](#product-surface) | ||||
| 48 | +4. [Data model — the canonical asset graph](#data-model--the-canonical-asset-graph) | ||||
| 49 | +5. [Methodology in brief](#methodology-in-brief) | ||||
| 50 | +6. [Architecture](#architecture) | ||||
| 51 | +7. [Repository layout](#repository-layout) | ||||
| 52 | +8. [Pipeline and workers](#pipeline-and-workers) | ||||
| 53 | +9. [Connectors](#connectors) | ||||
| 54 | +10. [Public API](#public-api) | ||||
| 55 | +11. [Quick start](#quick-start) | ||||
| 56 | +12. [Operator CLI](#operator-cli) | ||||
| 57 | +13. [Testing and quality gates](#testing-and-quality-gates) | ||||
| 58 | +14. [Deployment](#deployment) | ||||
| 59 | +15. [Brand](#brand) | ||||
| 60 | +16. [Data honesty rules](#data-honesty-rules) | ||||
| 61 | +17. [Roadmap](#roadmap) | ||||
| 62 | +18. [Author, hosting and contact](#author-hosting-and-contact) | ||||
| 63 | +19. [Disclaimers and license](#disclaimers-and-license) | ||||
| 64 | + | ||||
| 65 | +--- | ||||
| 66 | + | ||||
| 67 | +## What RareIndex answers | ||||
| 68 | + | ||||
| 69 | +| Question | Where | | ||||
| 70 | +|---|---| | ||||
| 71 | +| What is this collectible, exactly (set, number, variant, language, grade, grader)? | Canonical assets and variants, Scanner (photo / URL / text) | | ||||
| 72 | +| What is it worth, and how confident are we? | **RIV** — RareIndex Valuation with low / fair / high range, confidence label, sample size, method breakdown | | ||||
| 73 | +| What has it actually sold for? | Verified sales timeline with source links, buyer-pays price, verification label | | ||||
| 74 | +| What are sellers asking right now? | Live listings, ask vs RIV with the anomaly gate, market depth ±5 / 10 / 20 % | | ||||
| 75 | +| How liquid is it, how fast could it sell? | Liquidity Score 0–100, days on market, time-to-sale by asking band, fast / typical / patient sell prices | | ||||
| 76 | +| How rare is it? | Rarity Score from population reports, production quantity and market frequency | | ||||
| 77 | +| Is that auction lot cheap once fees are counted? | Auction intelligence: all-in bid or estimate (hammer + buyer's premium) vs RIV, per-house fee schedules | | ||||
| 78 | +| Is the market rising or falling? | RARE global index and 20 subindices, category markets, market map, screener | | ||||
| 79 | +| Which collectibles match my criteria? | `/screener` with presets and CSV / JSON export | | ||||
| 80 | +| What is unusual right now? | Rare Radar: record sales, price discrepancies, reappearances, auctions below RIV | | ||||
| 81 | + | ||||
| 82 | +## Live platform at a glance | ||||
| 83 | + | ||||
| 84 | +Production figures on 2026-09-11 (PostgreSQL on the MacLustr node): | ||||
| 85 | + | ||||
| 86 | +| Measure | Value | | ||||
| 87 | +|---|---| | ||||
| 88 | +| Canonical assets | 767 830 | | ||||
| 89 | +| Sales (transactions) | 2 228 672 | | ||||
| 90 | +| Live listings | 657 107 | | ||||
| 91 | +| Price observations (guide values, never transactions) | 14 476 398 | | ||||
| 92 | +| Auction lots tracked | 39 502 (≈ 22 100 live or upcoming) | | ||||
| 93 | +| Sources catalogued / connectors implemented | 443 / 212 | | ||||
| 94 | +| Taxonomy nodes | 136 categories, 96 families | | ||||
| 95 | +| Indices | RARE + 20 subindices (published only above their constituent minimum) | | ||||
| 96 | + | ||||
| 97 | +## Product surface | ||||
| 98 | + | ||||
| 99 | +Public pages (Next.js 16 App Router, server-rendered, dense financial-terminal design, dark mode first-class): | ||||
| 100 | + | ||||
| 101 | +| Route | Purpose | | ||||
| 102 | +|---|---| | ||||
| 103 | +| `/` | Hero + universal search, live system stats, RARE index panel, market snapshot, Trending, Record Sales, Rare Radar, Value Opportunities (gated), Ending auctions, Latest sales, categories | | ||||
| 104 | +| `/explore`, `/search` | Filterable asset browser and hybrid search (FTS + trigram + natural-language parsing, ⌘K palette) | | ||||
| 105 | +| `/asset/[slug]` | Asset terminal: hero with RIV range and score explainers, price history (sales as dots, RIV overlay with band), valuation breakdown and evidence, market depth & time to sell, sales, listings, **auctions**, grades (grade curve), population, images, history, comparables, sources | | ||||
| 106 | +| `/screener` | Filter builder (category, grader/grade, RIV, confidence, liquidity, rarity, 1M/1Y change, sales, listings, spread, drawdown), presets, sortable columns, CSV / JSON export | | ||||
| 107 | +| `/markets`, `/markets/[slug]`, `/categories`, `/market-map` | Category terminals, movers, activity, structure, treemap heatmap | | ||||
| 108 | +| `/rareindex`, `/rareindex/[ticker]` | Indices, constituents, correlations, benchmarks (S&P 500, gold, BTC, CPI…) | | ||||
| 109 | +| `/auctions`, `/auctions/house/[slug]`, `/auctions/calendar` | Open / ending-soon / below-RIV lots with all-in cost, house pages with buyer-premium schedules and objective stats, global calendar | | ||||
| 110 | +| `/sales`, `/listings`, `/records`, `/radar`, `/trending`, `/news`, `/grading` | Market feeds and intelligence | | ||||
| 111 | +| `/compare`, `/price-history`, `/set/[slug]`, `/brand/[brand]` | Comparison and entity pages | | ||||
| 112 | +| `/scanner`, `/research` | AI identification from photo / URL / text with confidence; AI Research answering only from structured data through inspectable tools | | ||||
| 113 | +| `/methodology`, `/data`, `/api-docs`, `/about` | Methodology center (wired to the code's constants), coverage & exports, API reference, company | | ||||
| 114 | +| Account | Collections (portfolio with cost basis, gains, allocation, concentration, insurance schedule, CSV import/export), watchlists, alerts (11 types incl. `auction_below_riv`), price targets, saved searches, Deal Radar, My Index, public profiles `/u/[handle]` | | ||||
| 115 | +| Admin `/admin` | Pipeline overview, connectors explorer, coverage, data quality, taxonomy proposals, events, audit, costs | | ||||
| 116 | + | ||||
| 117 | +## Data model — the canonical asset graph | ||||
| 118 | + | ||||
| 119 | +Marketplace titles are never assets. Every observation is resolved to a canonical **asset** | ||||
| 120 | +(`assets`: category, family, franchise, brand, series, set, number, year, edition, variant, language, | ||||
| 121 | +region, identifiers…) and to an **asset variant** (`asset_variants`: grader, grade, qualifier, | ||||
| 122 | +condition, completeness, size — e.g. `psa|10`, `raw|near_mint`, `cgc|9.8|black_label`, `psa|unknown` | ||||
| 123 | +for a slab whose grade could not be read). Variants never share a valuation: a PSA 10, a raw copy and | ||||
| 124 | +a sealed box are three markets. | ||||
| 125 | + | ||||
| 126 | +Evidence tables keep the three kinds of numbers strictly apart (§205 of the spec): | ||||
| 127 | + | ||||
| 128 | +- `sales` — observed transactions: native `price` + `currency`, `price_usd` at the **sale-date** ECB | ||||
| 129 | + rate (`fx_rate`, `fx_date`), `all_in_usd` = buyer-pays price (hammer + estimated buyer premium with a | ||||
| 130 | + `fee_basis`), `status` valid / flagged / excluded, `confidence`, `flags`, `dedupe_key`. | ||||
| 131 | +- `listings` — asks: lifecycle (`first_seen_at`, `last_seen_at`, `availability`, `listing_events`), | ||||
| 132 | + `discount_to_riv` = (ask − RIV) / RIV **only when the comparison passed the gates**, flags | ||||
| 133 | + `riv_anomaly` / `riv_review`. | ||||
| 134 | +- `auction_lots` — estimates, current bid, hammer, USD conversions, `all_in_bid_usd`, | ||||
| 135 | + `all_in_estimate_low_usd`, `bid_vs_riv`, `estimate_vs_riv`, `assessment_verdict`. | ||||
| 136 | +- `price_observations` — price-guide values, always labelled "guide", never mixed with transactions. | ||||
| 137 | + | ||||
| 138 | +Derived layers: `valuations` (append-only history with method breakdown), `asset_stats` / | ||||
| 139 | +`variant_stats` (current read model), `price_snapshots` (daily series), `grade_premiums`, | ||||
| 140 | +`population_reports`, `certificates` + `certificate_sightings`, `indices` / `index_values` / | ||||
| 141 | +`index_constituents`, `category_snapshots`, `radar_findings`, `correlations`, `benchmarks`. | ||||
| 142 | +Schema: `packages/database/src/schema/*`, 8 migrations in `packages/database/migrations/`. | ||||
| 143 | + | ||||
| 144 | +## Methodology in brief | ||||
| 145 | + | ||||
| 146 | +Full text: [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) and the live | ||||
| 147 | +[Methodology Center](https://www.rareindex.io/methodology), which imports its thresholds from the | ||||
| 148 | +code so it cannot drift. | ||||
| 149 | + | ||||
| 150 | +- **RIV (RareIndex Valuation)** — ensemble of robust estimators over the variant's valid | ||||
| 151 | + transactions (exponentially-weighted mean, medians 5/10/20, trust- and recency-weighted mean, | ||||
| 152 | + trimmed mean), median of the core; 365-day window extended to 3 years when thin (confidence | ||||
| 153 | + capped); falls back to grade-adjusted comparables, then to guide observations, each with capped | ||||
| 154 | + confidence. Inputs are **buyer-pays prices** (`coalesce(all_in_usd, price_usd)`). | ||||
| 155 | +- **Confidence** — 0.35·sample + 0.30·(1 − dispersion) + 0.20·recency + 0.15·source trust; labels | ||||
| 156 | + High ≥ 0.75, Medium ≥ 0.5, Low > 0.2, else Insufficient. | ||||
| 157 | +- **Range** — low / fair / high from p25 / RIV / p75, at least ±5 %. | ||||
| 158 | +- **Representative variant** — the default (raw / base) variant when it has a transaction-based RIV | ||||
| 159 | + with ≥ 5 sales and confidence ≥ 0.5, else the qualifying variant with the most sales. Changes, | ||||
| 160 | + ATH / ATL and drawdown are measured on that variant's own series. | ||||
| 161 | +- **Ask vs RIV and the anomaly gate** — an ask is compared only with a transaction-based RIV of the | ||||
| 162 | + **same variant** (≥ 5 sales, confidence ≥ 0.5, match ≥ 0.7). Asks below 0.1× or above 10× RIV are | ||||
| 163 | + flagged **Data/identity anomaly**; asks more than 50 % below RIV are held for **review**; a deal is | ||||
| 164 | + between −10 % and −50 %. Auction bids are never asks. Deal Score is multiplicative in depth, | ||||
| 165 | + confidence, sample, liquidity and match quality. | ||||
| 166 | +- **Buyer premiums** — per-house marginal schedules (`data/fees/auction-houses.json`, 57 houses, | ||||
| 167 | + each rate labelled published / approximate / none, default 22 % when unknown); VAT, duties and | ||||
| 168 | + shipping are explicitly excluded. | ||||
| 169 | +- **Liquidity Score** — sales per month, listing depth, source count, days between sales, ask-sold | ||||
| 170 | + spread. **Rarity Score** — population, production quantity, sales and listings per year. | ||||
| 171 | + **Momentum / Trending** with quality gates (canonical identity, ≥ 5 sales, plausible move ≤ ±500 %). | ||||
| 172 | +- **Outliers** — modified z-score > 3.5 on log prices, flagged and audited, never deleted; bundles | ||||
| 173 | + excluded. | ||||
| 174 | +- **Sales verification** — heuristic label verified / likely / unverified / excluded from status, | ||||
| 175 | + confidence, source type and trust; documented as heuristic. | ||||
| 176 | +- **Indices** — chain-linked trimmed equal-weight log returns, eligibility confidence ≥ 0.4 and ≥ 3 | ||||
| 177 | + sales in 12 months, single-return clamp ln 2.5, base 1000 on 2024-01-01, minimum 10 constituents | ||||
| 178 | + (RARE: 25 across published subindices); unpublished indices are shown as "in development". | ||||
| 179 | +- **FX** — ECB reference rates (frankfurter), sale-date rate for history, native values never | ||||
| 180 | + overwritten. | ||||
| 17 | 181 | ||||
| 18 | ## Architecture | 182 | ## Architecture | ||
| 19 | 183 | ||||
| 20 | −TypeScript end-to-end in a pnpm monorepo: Next.js 16 web app, Fastify public API, PostgreSQL 17 | 184 | +TypeScript end-to-end in a pnpm monorepo. | ||
| 21 | −(pg_trgm, unaccent, pgvector) as system of record, pg-boss workers for the ingestion pipeline, and a | 185 | + | ||
| 22 | −connector framework routing Direct API → Firecrawl → Scrapfly with quality scoring and health | 186 | +``` | ||
| 23 | −metrics. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the layout and data flow, and | 187 | + ┌──────────────┐ HTTPS (MacLustr Tunnel: WireGuard + Caddy) | ||
| 24 | −[`docs/adr/`](docs/adr/) for the decisions behind it. | 188 | + browsers ───▶ │ apps/web │ Next.js 16 · App Router · webpack build · Tailwind v4 | ||
| 189 | + │ :8210 │──┐ /api/v1/* rewritten to the API | ||||
| 190 | + └──────────────┘ │ | ||||
| 191 | + ┌──────────────┐ │ | ||||
| 192 | + developers ─▶ │ apps/api │◀─┘ Fastify · /v1 · API keys (hashed) · tiers · rate limits · CSV | ||||
| 193 | + │ :8211 │ | ||||
| 194 | + └──────┬───────┘ | ||||
| 195 | + │ SQL (drizzle-orm, raw sql for hot paths) | ||||
| 196 | + ┌──────▼───────────────────────────────────────────────┐ | ||||
| 197 | + │ PostgreSQL 17 · pg_trgm · unaccent · pgvector · pgboss│ | ||||
| 198 | + └──────▲───────────────────────────────────────────────┘ | ||||
| 199 | + │ | ||||
| 200 | + ┌──────┴───────┐ pg-boss queues: crawl.run · normalize.batch · resolve.batch | ||||
| 201 | + │ workers/ │ valuation.asset/rebuild · auctions.assess · indices.daily | ||||
| 202 | + │ (tsx) │ snapshots.daily · radar.scan · health.compute · fx.sync | ||||
| 203 | + └──────┬───────┘ benchmarks.sync · images.process · listings.expire · certs.verify · account.jobs | ||||
| 204 | + │ | ||||
| 205 | + ┌──────▼───────┐ 212 connectors · router Direct API → Firecrawl → Scrapfly | ||||
| 206 | + │ connectors/ │ per-host policies, circuit breakers, quality scoring, fixtures | ||||
| 207 | + └──────────────┘ | ||||
| 208 | +``` | ||||
| 209 | + | ||||
| 210 | +Packages: `shared` (schemas, money, stats, ids), `database` (Drizzle schema, migrations, seed), | ||||
| 211 | +`taxonomy` (136 categories, condition scales, graders, grade parser, variant keys, indices), | ||||
| 212 | +`connectors` (SDK, router, adapters Shopify / WooCommerce / sitemap / RSS / PDF / schema.org, health), | ||||
| 213 | +`valuation` (RIV, outliers, grade premiums, scores, ask assessment, fees, auction assessment, depth, | ||||
| 214 | +liquidation, verification), `indices` (chain-linked, repeat-sales, correlations), `search` | ||||
| 215 | +(FTS + trigram + NL parser), `ai` (provider router Anthropic / OpenAI-compatible, cost ledger), | ||||
| 216 | +`notify` (Resend e-mail). | ||||
| 217 | + | ||||
| 218 | +Decisions: [`docs/adr/`](docs/adr/). Layout and data flow: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md). | ||||
| 219 | + | ||||
| 220 | +## Repository layout | ||||
| 221 | + | ||||
| 222 | +``` | ||||
| 223 | +rareindex/ | ||||
| 224 | +├── apps/ | ||||
| 225 | +│ ├── web/ Next.js site (src/app routes, components, lib/queries, brand assets, OG images) | ||||
| 226 | +│ └── api/ Fastify public API (/v1), OpenAPI generator, key minting script | ||||
| 227 | +├── packages/ | ||||
| 228 | +│ ├── shared/ database/ taxonomy/ connectors/ valuation/ indices/ search/ ai/ notify/ | ||||
| 229 | +├── workers/ pg-boss workers + operator CLI (cli.ts): crawler, normalizer, entity-resolution, | ||||
| 230 | +│ valuation, auctions, indices, image-processing, account jobs, fees backfill… | ||||
| 231 | +├── connectors/ api/ · firecrawl/ · scrapfly/ — one folder per source (meta.json, index.ts, tests) | ||||
| 232 | +│ registry.json (generated), domains.json + domains.d/ (host policies) | ||||
| 233 | +├── data/ | ||||
| 234 | +│ ├── taxonomy/ categories.json · conditions.json · graders.json · indices.json | ||||
| 235 | +│ ├── fees/ auction-houses.json — buyer-premium schedules | ||||
| 236 | +│ ├── sources/ 443 catalogued sources → sources.json → docs/connectors/SOURCES.md | ||||
| 237 | +│ └── fixtures/ real captured payloads for every connector (465 JSON, 22 HTML) | ||||
| 238 | +├── docs/ ARCHITECTURE · METHODOLOGY · API · openapi.json · SPEC-original · adr/ · connectors/ | ||||
| 239 | +├── scripts/ build-registry · build-sources · connector-new · gen-openapi · ri | ||||
| 240 | +└── CLAUDE.md condensed product mandate and engineering rules | ||||
| 241 | +``` | ||||
| 242 | + | ||||
| 243 | +## Pipeline and workers | ||||
| 25 | 244 | ||||
| 26 | ``` | 245 | ``` | ||
| 27 | −apps/web · apps/api · packages/{shared,database,taxonomy,connectors,valuation,indices,search,ai} | 246 | +source → raw_records (immutable, content-hashed) | ||
| 28 | −workers/ · connectors/ (+ registry.json) · data/{taxonomy,fixtures} · docs/ · scripts/ | 247 | + → normalize() connector adapter → NormalizedRecord (sale | listing | auction_lot | | ||
| 248 | + price_observation | catalog_item | population_report | news_item), Zod-validated | ||||
| 249 | + → entity resolution identifiers → canonical key → constrained trigram → create; grade re-parsed | ||||
| 250 | + from titles (refineGrade), variant ensured (never raw for a slab) | ||||
| 251 | + → canonical writes sales (dedupe key, sale-date FX, buyer-pays price), listings (lifecycle events), | ||||
| 252 | + auction lots, observations, populations, certificates | ||||
| 253 | + → valuation per-variant RIV, outlier flags, ask-vs-RIV with the anomaly gate, | ||||
| 254 | + representative variant, asset_stats / variant_stats / price_snapshots | ||||
| 255 | + → auctions.assess USD + all-in bid / estimate + bid vs RIV every 20 minutes | ||||
| 256 | + → indices / snapshots RARE + subindices, category snapshots, correlations, benchmarks | ||||
| 257 | + → radar / alerts record sales, price discrepancies, auctions below RIV, reappearances, | ||||
| 258 | + ultra-low populations; member alerts, digests, price targets | ||||
| 29 | ``` | 259 | ``` | ||
| 30 | 260 | ||||
| 31 | −Data flow: `raw_records` (immutable) → `normalize()` → `normalized_records` → entity resolution → | 261 | +Operational safeguards: per-run time budgets, one crawl per connector at a time, orphan-run recovery, | ||
| 32 | −`assets` / `asset_variants` → dedupe and price validation (flags + audit log, nothing deleted) → | 262 | +per-host concurrency and circuit breakers, challenge-page detection, schema-drift and | ||
| 33 | −`sales` / `listings` / `auction_lots` / `price_observations` → valuations and stats → indices. | 263 | +result-collapse detectors feeding `connector_health`, audit log for every flag or move. | ||
| 264 | + | ||||
| 265 | +## Connectors | ||||
| 266 | + | ||||
| 267 | +212 implemented connectors (179 `api`, 43 `firecrawl`, 8 `scrapfly` folders; 20 countries, | ||||
| 268 | +125 categories) over 443 catalogued sources (201 implemented, 10 gated behind API keys, 108 blocked | ||||
| 269 | +by access controls that RareIndex will not circumvent, 108 rejected with a reason). The catalogue is | ||||
| 270 | +[`docs/connectors/SOURCES.md`](docs/connectors/SOURCES.md). | ||||
| 271 | + | ||||
| 272 | +A new source needs four things: `meta.json`, an extraction schema, a pure `normalize()` adapter and | ||||
| 273 | +real fixtures with tests. Scaffold with `pnpm connector:new <id> --url … [--adapter shopify|woocommerce]`, | ||||
| 274 | +then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide: | ||||
| 275 | +[`docs/connectors/ADDING_A_CONNECTOR.md`](docs/connectors/ADDING_A_CONNECTOR.md). | ||||
| 276 | + | ||||
| 277 | +Routing is Direct API → structured feed → Firecrawl → Scrapfly → manual review, with a quality gate | ||||
| 278 | +(title 20 %, price 20 %, status 15 %, date 10 %, images 10 %, identifiers 10 %, description 5 %, | ||||
| 279 | +category 5 %, currency 5 %). RareIndex never bypasses logins, paywalls, CAPTCHAs or other access | ||||
| 280 | +controls. | ||||
| 281 | + | ||||
| 282 | +## Public API | ||||
| 283 | + | ||||
| 284 | +Base URL `https://www.rareindex.io/api/v1` (proxied to Fastify). Bearer keys `ri_live_…` are stored | ||||
| 285 | +hashed; anonymous calls get the public tier. Every list supports cursor pagination and `?format=csv`; | ||||
| 286 | +responses carry `meta.as_of` and attribution. | ||||
| 287 | + | ||||
| 288 | +| Endpoint | Description | | ||||
| 289 | +|---|---| | ||||
| 290 | +| `GET /v1/assets/search` · `/v1/assets/:id` | Search and asset detail (id or slug) | | ||||
| 291 | +| `GET /v1/assets/:id/sales` · `/listings` · `/history` | Evidence and daily series (native + USD + all-in + fee basis) | | ||||
| 292 | +| `GET /v1/assets/:id/depth` | Market depth ±5/10/20 %, days on market, time-to-sale bands, fair buy / sell ladder | | ||||
| 293 | +| `GET /v1/assets/:id/auctions` · `GET /v1/auctions/lots` | Lots with estimates, bids, all-in cost, `bid_vs_riv`, verdict; filters `below_riv`, `house`, `ending_within_hours`, `category` | | ||||
| 294 | +| `GET /v1/indices` · `/v1/indices/:ticker/history` | RARE and subindices | | ||||
| 295 | +| `GET /v1/markets` · `/v1/markets/:slug` · `/v1/categories` · `/v1/trending` · `/v1/sales/latest` · `/v1/records` · `/v1/stats` | Markets and feeds | | ||||
| 296 | +| `GET /v1/openapi.json` | OpenAPI 3.1 | | ||||
| 297 | + | ||||
| 298 | +Reference: [`docs/API.md`](docs/API.md), [`docs/openapi.json`](docs/openapi.json), live at | ||||
| 299 | +[/api-docs](https://www.rareindex.io/api-docs). | ||||
| 34 | 300 | ||||
| 35 | ## Quick start | 301 | ## Quick start | ||
| 36 | 302 | ||||
| 37 | −Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with the `pgvector` extension available. | 303 | +Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pgvector`. | ||
| 38 | 304 | ||||
| 39 | ```bash | 305 | ```bash | ||
| 40 | pnpm install | 306 | pnpm install | ||
| 41 | createdb rareindex | 307 | createdb rareindex | ||
| 42 | −cp .env.example .env # fill DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, AI and Resend keys | 308 | +cp .env.example .env # DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, ANTHROPIC_API_KEY, RESEND_API_KEY, SESSION_SECRET, ADMIN_TOKEN | ||
| 43 | −pnpm db:migrate && pnpm db:seed # creates extensions + tables, seeds taxonomy, graders, indices, registry | 309 | +pnpm db:migrate && pnpm db:seed # extensions, tables, taxonomy, graders, indices, connector registry | ||
| 44 | 310 | ||||
| 45 | −pnpm dev:web # Next.js on http://localhost:3000 | 311 | +pnpm dev:web # http://localhost:3000 | ||
| 46 | −pnpm dev:api # public REST API (/v1) | 312 | +pnpm dev:api # http://localhost:8211/v1 | ||
| 47 | −pnpm worker # pipeline workers (crawl, normalize, resolve, value, index) | 313 | +pnpm worker # pipeline workers | ||
| 48 | −pnpm ri --help # operational CLI (run a connector, recompute valuations, health…) | 314 | +pnpm ri stats # operator CLI | ||
| 49 | ``` | 315 | ``` | ||
| 50 | 316 | ||||
| 51 | −Useful scripts: `pnpm typecheck`, `pnpm test`, `pnpm build`, `pnpm db:generate` (after editing | 317 | +Build and verify: `pnpm typecheck`, `pnpm test`, `pnpm build`. Schema change: edit | ||
| 52 | −`packages/database/src/schema/*`), `pnpm tsx scripts/build-registry.ts` (rebuilds | 318 | +`packages/database/src/schema/*`, `pnpm db:generate`, commit the migration. | ||
| 53 | −`connectors/registry.json` from per-connector `meta.json` files). | 319 | + | ||
| 320 | +## Operator CLI | ||||
| 54 | 321 | ||||
| 55 | −## Adding a connector | 322 | +`pnpm ri <command>` (on the node: `node node_modules/tsx/dist/cli.mjs workers/cli.ts <command>`): | ||
| 56 | 323 | ||||
| 57 | −A new source needs only four things: connector metadata (`meta.json`), an extraction schema, a pure | 324 | +| Command | Purpose | | ||
| 58 | −`normalize()` adapter and real captured fixtures with tests. The rest of the pipeline works | 325 | +|---|---| | ||
| 59 | −unchanged. Conventions and the module contract are in [`connectors/README.md`](connectors/README.md). | 326 | +| `connectors` · `crawl <id> [--mode probe\|incremental\|backfill]` · `backfill <id…> [--start-only]` | Sources and crawls | | ||
| 327 | +| `normalize [--connector id]` · `resolve [--limit N]` | Pipeline stages | | ||||
| 328 | +| `value [--asset id\|--all] [--history]` · `premiums` · `regrade [--dry-run] [--recheck]` | Valuations, grade premiums, grade repair (moves mis-varianted rows, audited) | | ||||
| 329 | +| `fees --backfill [--force]` · `auctions --assess [--all]` | Buyer-pays prices on sales, auction lot assessment | | ||||
| 330 | +| `index` · `snapshots` · `radar` · `benchmarks` · `fx [--backfill]` | Indices, snapshots, Rare Radar, benchmarks, exchange rates | | ||||
| 331 | +| `certs [--backfill]` · `images` · `expire` · `health [--probe]` · `stats` · `run-all` · `worker` | Certificates, images, listing expiry, health, stats, full cycle | | ||||
| 332 | + | ||||
| 333 | +## Testing and quality gates | ||||
| 334 | + | ||||
| 335 | +- Unit tests (vitest) in every package and app: valuation (RIV, outliers, premiums, ask gate, fees, | ||||
| 336 | + auction assessment, depth, liquidation, verification), taxonomy (grade parser, variant keys), search | ||||
| 337 | + parser, indices, shared money/stats, workers, API, web helpers. | ||||
| 338 | +- 213 connector test files run every fixture through the schema, price, currency and date checks | ||||
| 339 | + (`pnpm connectors:test`). | ||||
| 340 | +- `pnpm -r run typecheck` (TypeScript strict), `pnpm --filter @rareindex/web run lint`. | ||||
| 341 | +- Data-quality gates are code, not conventions: anomaly band, review threshold, plausible-change | ||||
| 342 | + bound, representative-variant rule, numeric overflow guard, `Data/identity anomaly` and | ||||
| 343 | + `Needs review` labels in the UI. | ||||
| 344 | + | ||||
| 345 | +## Deployment | ||||
| 346 | + | ||||
| 347 | +Production runs on the **MacLustr** cluster ([www.maclustr.io](https://www.maclustr.io)) — node | ||||
| 348 | +M3U96a (Mac Studio, PostgreSQL 17 + pgvector local) — orchestrated by the `mld` gateway and published | ||||
| 349 | +through the MacLustr Tunnel (WireGuard + Caddy on an OVH gateway) at **https://www.rareindex.io**. | ||||
| 350 | +PM2 processes: `rareindex-web` (8210), `rareindex-api` (8211), `rareindex-worker`. Secrets live only | ||||
| 351 | +in the `mld` manifest. | ||||
| 352 | + | ||||
| 353 | +Release procedure: commit → `git archive HEAD` synced to the node → `pnpm db:migrate` → | ||||
| 354 | +`pnpm --filter @rareindex/web run build` (detached, ~15 min: 146 static pages) → `pm2 restart`. | ||||
| 355 | +Never delete `.next` under a running server; `/data` is rendered on demand with a one-hour cache | ||||
| 356 | +because its aggregates exceed the build-time prerender budget. | ||||
| 357 | + | ||||
| 358 | +## Brand | ||||
| 359 | + | ||||
| 360 | +<p align="center"> | ||||
| 361 | + <img src="apps/web/public/brand/rareindex-mark.svg" alt="RareIndex mark" width="96"> | ||||
| 362 | +</p> | ||||
| 363 | + | ||||
| 364 | +The mark is a cut stone — crown, girdle, pavilion — whose facets are drawn by an ascending index | ||||
| 365 | +line: rarity and market data in one shape. Ink `#0B0F19`, gold `#F1D08A → #C9963F`, mint | ||||
| 366 | +`#2FB673 → #5FE3A1`. Wordmark **Rare<span>Index</span>** with "Index" in gold. Source files in | ||||
| 367 | +[`apps/web/public/brand/`](apps/web/public/brand/) (mark, light and dark lockups, README); favicons, | ||||
| 368 | +PWA icons, `favicon.ico` and the Apple touch icon are generated from the mark by | ||||
| 369 | +`apps/web/scripts/brand-assets.mjs`; the site-wide share image is rendered by | ||||
| 370 | +`apps/web/src/app/opengraph-image.tsx`. | ||||
| 60 | 371 | ||||
| 61 | ## Data honesty rules | 372 | ## Data honesty rules | ||
| 62 | 373 | ||||
| 63 | −- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions or listings. | 374 | +- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions, bids or listings. | ||
| 64 | −- Price-guide values are stored as observations, never as transactions. | 375 | +- Observed data, model estimates, user data and external asks are always distinguished (§205). | ||
| 65 | −- Valuations are published only with sample size, confidence, low/high range and timestamp. | 376 | +- Price-guide values are observations, never transactions; a listing price is never a market value. | ||
| 66 | −- Outliers are flagged and audited, never silently deleted. | 377 | +- Valuations publish only with sample size, confidence, range and timestamp; missing evidence reads "Not enough data". | ||
| 67 | −- Sale dates come from the source; crawl time is never substituted. | 378 | +- Outliers are flagged and audited, never deleted; every repair (`regrade`, flags) is in `audit_log`. | ||
| 68 | −- Historical prices are converted with the exchange rate of the sale date. | 379 | +- Sale dates come from the source; historical prices use the exchange rate of the sale date; native values are never overwritten. | ||
| 380 | +- Hammer prices are never compared with marketplace prices without the buyer's premium; the fee basis is shown. | ||||
| 381 | +- Neutral wording for anomalies ("anomalous transaction", "needs review"); RareIndex never accuses anyone of fraud and never claims to authenticate items. | ||||
| 69 | 382 | ||||
| 70 | Details: [`docs/adr/0002-data-honesty.md`](docs/adr/0002-data-honesty.md). | 383 | Details: [`docs/adr/0002-data-honesty.md`](docs/adr/0002-data-honesty.md). | ||
| 71 | 384 | ||||
| 72 | −## Deployment | 385 | +## Roadmap | ||
| 73 | 386 | ||||
| 74 | −Production runs on a MacLustr node (PostgreSQL 17 + pgvector) deployed through the `mld` gateway: | 387 | +Delivered on 2026-09-11: valuation safety (anomaly gate, variant-aware asks, representative variant, | ||
| 75 | −PM2 processes `rareindex-web`, `rareindex-api`, `rareindex-worker` and an ngrok tunnel serving | 388 | +grade repair), screener, methodology center, market depth and time-to-sell, sale verification labels, | ||
| 76 | −**https://www.rareindex.io**. Secrets live only in the node's `.env` / the `mld` manifest. | 389 | +buyer premiums and auction intelligence, brand. Next, in order of leverage: | ||
| 77 | 390 | ||||
| 78 | −## Disclaimers | 391 | +1. Entity resolution for unique objects (auction lots on generic assets), certificate-number and | ||
| 392 | + multi-token search, `unaccent` in FTS. | ||||
| 393 | +2. `fx_rate` / `fx_date` on listings, observations and lots; per-lot bid counts for houses that omit them. | ||||
| 394 | +3. Index history reconstruction from transactions (`ri value --all --history`), repeat-sales publication. | ||||
| 395 | +4. Population reports through grader partnerships; cross-grader empirical model. | ||||
| 396 | +5. Portfolio overhaul (fees, liquidity horizon, confidence aggregation), Scanner multi-item and visual embeddings. | ||||
| 397 | +6. Cross-border arbitrage with FX, fees and duties; regional price indices. | ||||
| 398 | +7. Natural-language query engine over the screener; webhooks; API tier enforcement and shared rate-limit store. | ||||
| 79 | 399 | ||||
| 80 | −RareIndex is an information platform. Valuations are estimates derived from observed public data. | 400 | +## Author, hosting and contact | ||
| 81 | −Listing prices are not confirmed transactions. Past performance does not guarantee future results. | 401 | + | ||
| 82 | −RareIndex does not authenticate items. Third-party names, marks and content belong to their owners | 402 | +RareIndex is designed, built and operated by **Simon-Pierre Boucher** (Québec, Canada). | ||
| 83 | −and are attributed to their sources; RareIndex respects source terms, robots policies, rate limits, | 403 | +It is hosted on **MacLustr** — [www.maclustr.io](https://www.maclustr.io) — a distributed | ||
| 84 | −copyright, privacy and database rights, and never bypasses authentication, paywalls or CAPTCHAs. | 404 | +Apple-silicon compute cluster operated by the same team. | ||
| 85 | 405 | ||||
| 86 | −## License | 406 | +- General contact: [contact@spboucher.ai](mailto:contact@spboucher.ai) | ||
| 407 | +- Data corrections and source requests: data@rareindex.io · API and partnerships: hello@rareindex.io · Rights holders: legal@rareindex.io | ||||
| 408 | + | ||||
| 409 | +## Disclaimers and license | ||||
| 410 | + | ||||
| 411 | +RareIndex is an information platform. Valuations are estimates derived from observed public data. | ||||
| 412 | +Listing prices and auction estimates are not confirmed transactions. Past performance does not | ||||
| 413 | +guarantee future results. RareIndex does not authenticate items and does not provide investment | ||||
| 414 | +advice. Third-party names, marks and content belong to their owners and are attributed to their | ||||
| 415 | +sources; RareIndex respects source terms, robots policies, rate limits, copyright, privacy and | ||||
| 416 | +database rights. | ||||
| 87 | 417 | ||||
| 88 | −Proprietary — © 2026 RareIndex. All rights reserved. | 418 | +Proprietary — © 2026 Simon-Pierre Boucher / RareIndex. All rights reserved. | ||
| 89 | 419 | ||||