spb/rareindex

1# RareIndex methodology23RareIndex turns fragmented public collectibles data into structured market data. This page explains4how every number on the platform is produced, what evidence it rests on, and where its limits are.5Nothing on RareIndex is invented: every sale, listing, guide price, population count and index value6traces back to a source record with its URL, capture time, parser version and connector version.78## 1. Data sources and pipeline910Data enters through connectors (marketplaces, auction houses, price guides, grading companies,11catalogs). Each capture is stored immutably as a raw record, then normalised into a canonical shape,12matched to a canonical asset (entity resolution), de-duplicated, validated and only then written to13the sales, listings and observations tables that feed valuations and indices.1415Three kinds of price evidence are kept strictly apart:1617| Evidence | Meaning | Used for |18| --- | --- | --- |19| **Sale** | An observed transaction with a source date and price | RIV, indices, records, liquidity |20| **Listing** | An asking price; not a transaction | Value opportunity, spread, supply signals |21| **Price observation** | A published guide value (e.g. a marketplace's market price) | Fallback estimate only, flagged as guide-based |2223Currency: every price is stored in its native currency together with the USD conversion at the24**historical** ECB reference rate of the transaction date. Display currencies are converted at view25time; historical sales are never converted at today's rate.2627## 2. Entity resolution2829Descriptions of the same object differ across sources ("1999 Pokemon Base Charizard 1st PSA 10" vs30"Pokémon Base Set Charizard Holo 1st Edition #4/102 PSA GEM MT 10"). Records are resolved in order:31deterministic identifiers (catalog ids, style codes, set numbers, references, UPC/ISBN) → a32deterministic canonical key built from structured attributes per category → trigram similarity of33titles constrained by hard discriminators (card number, set, year) → creation of a new asset. The34match method and confidence are stored on every record, and grade/condition slices are kept as35variants of the asset (PSA 10, CIB, deadstock size 10 …) so that different grades are never mixed.3637## 3. RIV — RareIndex Valuation3839RIV is an ensemble estimate of the current market value of a variant, computed from verified40transactions in the last 365 days (extended to three years when fewer than five recent sales exist,41in which case confidence is capped).4243The estimators are: latest sale, medians of the last 5/10/20 sales, a trimmed mean (10 % each tail),44an exponentially weighted mean (90-day half-life) and a weighted mean using source trust,45identification confidence and recency. RIV is the median of the core estimators. The low/high band46comes from the inter-quartile range of the sales used (widened to at least ±5 %).4748When a variant has fewer than three sales, RIV may fall back to **grade-adjusted comparables**49(sales of other grades of the same asset, adjusted by empirical grade premiums) and, as a last50resort, to **guide observations**. Fallback estimates are labelled and their confidence is capped at510.5 (comps) or 0.45 (guide-based). With no evidence at all, RareIndex shows *Data unavailable*.5253### Confidence5455Confidence (0–1, shown as High ≥ 0.75, Medium ≥ 0.5, Low > 0.2, Insufficient otherwise) combines56sample size (log-scaled, 40 sales ≈ maximum), dispersion of log prices (robust MAD), recency of the57latest sale and the trust/identification quality of the sales used. Every RIV is displayed with its58sample size, confidence and computation time.5960### Outliers6162Within a variant, sales whose modified z-score on log price exceeds 3.5 are **flagged**, not deleted;63bundles and multi-quantity lots are excluded from pricing. Every flag is written to the audit log with64its reason.6566### Grade premiums6768For each category, grader and grade, RareIndex measures the median ratio between graded and raw69sales of the same assets (at least eight paired assets). PSA 10, BGS 10 and CGC 10 are never assumed70equivalent; premiums are only ever derived from paired evidence.7172### Buyer premiums and the buyer-pays price7374A hammer price and a marketplace price are not the same number: the buyer of an auction lot also75pays the house's buyer premium. Every sale therefore carries `all_in_usd` = `price_usd` plus the76premium when the record is hammer-only (`buyer_premium_included = false`, or unknown for a listed77auction house), computed from the schedule in `data/fees/auction-houses.json` with marginal tiers,78minimums, caps and fixed fees, tier thresholds converted into the schedule currency at the sale-date79rate. `fee_basis` records how the figure was obtained — `included`, `added_published`,80`added_approximate`, `added_default` (house unknown, 22 %), `none` (no premium: marketplaces,81ComicLink, Yahoo! Auctions Japan…) or `unknown`. RIV, ATH/ATL, grade premiums, index volumes and82the auction assessments all use `coalesce(all_in_usd, price_usd)`; the native `price` and83`price_usd` are never overwritten. When a valuation's inputs include estimated premiums the84valuation says so ("n % of inputs include an estimated buyer premium"). Live lots are assessed the85same way: current bid (only when at least one bid exists) or low estimate + premium → all-in cost,86compared with the RIV of the lot's own variant through the ask gates (`bid_vs_riv`,87`estimate_vs_riv`, verdict). VAT or sales tax on the premium, import duties and shipping are **not**88modelled and are stated as such.8990## 4. Scores9192- **Liquidity (0–100)**: sales per month, active listings, number of sources, median days between93  sales and the spread between asks and RIV.94- **Rarity (0–100)**: graded population, production quantity when documented, and observed95  sale/listing frequency. Rarity is *null* when no supply signal exists — it is never guessed.96- **Momentum (−100…100)** over 7d/30d/90d/1y: price change against the RIV history blended with97  transaction-volume acceleration.98- **Value opportunity**: `(ask − RIV) / RIV` of an active ask against the RIV of its **own variant**99  (negative = below the valuation), shown only when the gate below passes. Analytical information,100  not investment advice.101- **Data quality (0–100)**: attribute completeness, source trust, identification confidence, images102  and transaction evidence.103- **Trending**: geometric blend of price, volume and listing momenta.104105### Ask vs RIV and the anomaly gate106107An asking price is compared with a valuation only when the comparison is legitimate (the live108constants are exported from `packages/valuation/src/scores.ts` and rendered on `/methodology`):109110- same variant — a slab whose grade could not be read is kept in a "grader · grade unknown" variant111  and is never compared against the raw valuation;112- the RIV rests on transactions (comps-only and guide-only estimates never qualify an ask);113- RIV confidence ≥ 0.5 and ≥ 5 transactions used; listing identification confidence ≥ 0.7;114- plausibility band: an ask below 0.1× or above 10× RIV is flagged `riv_anomaly` and shown as115  **Data/identity anomaly** — a wrong variant, lot, currency or identity until reviewed — never as a116  deal.117118Verdicts: **deal** ≤ −10 %, **fair** between −10 % and +10 %, **premium** ≥ +10 %. The **Deal Score**119(0–100) multiplies discount depth, RIV confidence, sample size, liquidity and match confidence, so a120large discount on an illiquid or poorly identified asset cannot score high. Discounts are recomputed121from scratch on every valuation pass; a stale discount never survives a changed valuation. Period122changes beyond ±500 % are treated as data artefacts and are not published as movers.123124### Auction fees and all-in cost125126A hammer price is not what a buyer pays. Before any comparison with marketplace prices or with the127RIV, RareIndex adds the house's buyer's premium (`data/fees/auction-houses.json`: marginal tiers,128minimum/maximum, fixed fees, per-house confidence `published` / `approximate` / `none`; houses not on129file fall back to a labelled default of 22 %). The basis is stored with every figure130(`sales.fee_basis`, `auction_lots.fee_basis`): *premium included*, *premium added (published /131≈ estimated / default)*, *no buyer premium* (fixed-price marketplaces, dealers, houses without a132premium) or *fees unknown* (connector silent and house not on file — the price is kept as recorded133and flagged). VAT/sales tax on the premium, import duties and shipping are excluded and stated as such.134135Live lots are assessed hourly on the same basis: current bid (or the low estimate, labelled "est.",136when no bid has been placed) → USD at the current ECB rate → + premium → compared with the RIV of137the lot's variant under the listing gates (transaction-based RIV, ≥ 5 sales, confidence ≥ 0.5,138plausibility band, −50 % review threshold). Opening prices without a bid are never called deals.139140## 5. Indices141142Each subindex (RARE-TCG, RARE-WATCH, …) is a **chain-linked** index of daily RIV changes across its143constituents: assets with a valuation confidence of at least 0.4 and at least three sales in the last144twelve months (one variant per asset, the most traded). Each day the index moves by the trimmed,145equal-weight average of constituents' log returns; composition can change daily without breaking146the level. Base = 1000 on 2024-01-01. An index publishes nothing until it has its minimum number of147constituents (10 for subindices, 25 for RARE), and every point stores its constituent count and148coverage. Indices that have not met their minimum are listed as *in development* outside the primary149ticker.150151**RARE**, the flagship index, is the transaction-weighted average of subindex daily returns and152publishes only once at least 25 constituents exist across the published subindices.153154A **repeat-sales** estimator (pairs of sales of the same variant, monthly periods, least squares)155is provided for research views; it is less exposed to composition bias but requires many pairs.156157Historical index and asset series are reconstructed from transactions only: for each week since the158first sale, the valuation is recomputed with the sales known at that date.159160## 6. Market capitalisation161162Category market capitalisation is estimated as Σ (graded population × RIV) **only** for assets with163a published population report; the estimate carries a coverage-based confidence label and is never164shown as a precise figure.165166## 7. Benchmarks and correlations167168Benchmark series (S&P 500, NASDAQ, Bitcoin, CPI, Case-Shiller) come from public datasets (FRED,169stooq). Correlations between subindices are Pearson coefficients of daily log returns over 30/90/365170days; collectibles trade far less frequently than securities, so daily comparisons should be read171with that in mind.172173## 8. Rare Radar and records174175Record sales are the highest verified transactions per category. Radar findings (price176discrepancies, first listings in years, ultra-low populations) are derived mechanically from stored177transactions, listings and population reports with the underlying evidence attached.178179## 9. Freshness and limits180181Connectors run on schedules tuned to each source; every statistic shows its last update. Coverage182is uneven by design: categories with few public transactions show fewer valuations and no index183rather than a fabricated one.184