SPB Git forge
spboucher.ai ↗

spb/rareindex

Public
54commits 1branches 0releases
7.1 MBsize
maindefault branch
10 days agolast push
TypeScript 61.9% HTML 37.2% SQL 0.7%

README: full platform documentation with the RareIndex brand (logo, product surface, data model, methodology, architecture, pipeline, connectors, API, CLI, deployment, roadmap, author and hosting on MacLustr)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed 13 days ago (Sep 11, 2026) parent e0d6c47

1 changed file +386 −56

modified README.md +386 −56
@@ -1,88 +1,418 @@
1 +<p align="center">
2 + <picture>
3 + <source media="(prefers-color-scheme: dark)" srcset="apps/web/public/brand/rareindex-logo-dark.svg">
4 + <img src="apps/web/public/brand/rareindex-logo.svg" alt="RareIndex — The Global Market for Collectibles" width="520">
5 + </picture>
6 +</p>
7 +
8 +<p align="center">
9 + <a href="https://www.rareindex.io"><strong>www.rareindex.io</strong></a> ·
10 + <a href="https://www.rareindex.io/methodology">Methodology</a> ·
11 + <a href="https://www.rareindex.io/api-docs">API</a> ·
12 + <a href="https://www.rareindex.io/screener">Screener</a> ·
13 + <a href="https://www.rareindex.io/auctions">Auctions</a> ·
14 + <a href="mailto:contact@spboucher.ai">contact@spboucher.ai</a>
15 +</p>
16 +
17 +<p align="center">
18 + <img alt="TypeScript" src="https://img.shields.io/badge/TypeScript-strict-3178c6"> <img alt="Next.js 16" src="https://img.shields.io/badge/Next.js-16-000000"> <img alt="Fastify" src="https://img.shields.io/badge/Fastify-%2Fv1-202020"> <img alt="PostgreSQL 17" src="https://img.shields.io/badge/PostgreSQL-17%20%2B%20pgvector-336791"> <img alt="pg-boss" src="https://img.shields.io/badge/workers-pg--boss-4b5563"> <img alt="connectors" src="https://img.shields.io/badge/connectors-212-c9963f"> <img alt="hosted on MacLustr" src="https://img.shields.io/badge/hosted%20on-MacLustr-0b0f19">
19 +</p>
20 +
1 21 # RareIndex
2 22
3 −**The global market for collectibles.** RareIndex.io is a market-intelligence, search, pricing and
4 −index platform for collectible assets: trading cards, sports cards, watches, sneakers, LEGO, comics,
5 −video games, toys, coins, art, wine and, over time, every category the collectibles economy contains.
6 −It continuously collects public market data from marketplaces, auction houses, dealers, grading
7 −companies, manufacturer catalogs and archives, resolves it into one canonical asset graph, and turns
8 −it into research-grade data: valuations with confidence levels, comparable sales, live listings,
9 −rarity and liquidity scores, price histories, and the RareIndex family of indices (`RARE` and its
10 −category subindices).
11 −
12 −RareIndex is an intelligence platform, not a marketplace aggregator. Every number shown is traceable
13 −to its source, every estimate carries a sample size and a confidence level, and when the evidence is
14 −insufficient the platform says "Data unavailable" rather than inventing a figure. The full product
15 −mandate is in [`docs/SPEC-original.md`](docs/SPEC-original.md); the condensed working version that
16 −guides implementation is [`CLAUDE.md`](CLAUDE.md).
23 +**RareIndex is the market-data and intelligence layer for physical collectibles** — the terminal the
24 +collectibles economy never had. It continuously collects public market evidence (marketplaces,
25 +auction houses, dealers, price guides, grading companies, manufacturer catalogs), resolves it into a
26 +canonical asset graph and turns it into research-grade data: a valuation with a confidence level and
27 +a range, comparable sales, live asks, market depth, liquidity and rarity scores, auction intelligence
28 +with buyer-pays costs, price histories and the **RARE** family of indices.
29 +
30 +A marketplace tells you *"this item is for sale for $4,000"*. RareIndex tells you:
31 +
32 +> *This asset has a RareIndex Valuation of $3,180 (high confidence, 42 comparable sales of the same
33 +> variant across five sources), the market is up 12.4 % over 12 months, typical time-to-sale is 18
34 +> days, liquidity is above the category average, and the current $2,650 ask is 16.7 % below fair
35 +> value — while a $472 slab of a different grade matched to the same card is a data anomaly, not a
36 +> deal.*
37 +
38 +Every number is traceable to its source, carries a sample size, a confidence and a freshness, and
39 +when the evidence is not there the platform says **"Not enough data"** instead of inventing a figure.
40 +
41 +---
42 +
43 +## Table of contents
44 +
45 +1. [What RareIndex answers](#what-rareindex-answers)
46 +2. [Live platform at a glance](#live-platform-at-a-glance)
47 +3. [Product surface](#product-surface)
48 +4. [Data model — the canonical asset graph](#data-model--the-canonical-asset-graph)
49 +5. [Methodology in brief](#methodology-in-brief)
50 +6. [Architecture](#architecture)
51 +7. [Repository layout](#repository-layout)
52 +8. [Pipeline and workers](#pipeline-and-workers)
53 +9. [Connectors](#connectors)
54 +10. [Public API](#public-api)
55 +11. [Quick start](#quick-start)
56 +12. [Operator CLI](#operator-cli)
57 +13. [Testing and quality gates](#testing-and-quality-gates)
58 +14. [Deployment](#deployment)
59 +15. [Brand](#brand)
60 +16. [Data honesty rules](#data-honesty-rules)
61 +17. [Roadmap](#roadmap)
62 +18. [Author, hosting and contact](#author-hosting-and-contact)
63 +19. [Disclaimers and license](#disclaimers-and-license)
64 +
65 +---
66 +
67 +## What RareIndex answers
68 +
69 +| Question | Where |
70 +|---|---|
71 +| What is this collectible, exactly (set, number, variant, language, grade, grader)? | Canonical assets and variants, Scanner (photo / URL / text) |
72 +| What is it worth, and how confident are we? | **RIV** — RareIndex Valuation with low / fair / high range, confidence label, sample size, method breakdown |
73 +| What has it actually sold for? | Verified sales timeline with source links, buyer-pays price, verification label |
74 +| What are sellers asking right now? | Live listings, ask vs RIV with the anomaly gate, market depth ±5 / 10 / 20 % |
75 +| How liquid is it, how fast could it sell? | Liquidity Score 0–100, days on market, time-to-sale by asking band, fast / typical / patient sell prices |
76 +| How rare is it? | Rarity Score from population reports, production quantity and market frequency |
77 +| Is that auction lot cheap once fees are counted? | Auction intelligence: all-in bid or estimate (hammer + buyer's premium) vs RIV, per-house fee schedules |
78 +| Is the market rising or falling? | RARE global index and 20 subindices, category markets, market map, screener |
79 +| Which collectibles match my criteria? | `/screener` with presets and CSV / JSON export |
80 +| What is unusual right now? | Rare Radar: record sales, price discrepancies, reappearances, auctions below RIV |
81 +
82 +## Live platform at a glance
83 +
84 +Production figures on 2026-09-11 (PostgreSQL on the MacLustr node):
85 +
86 +| Measure | Value |
87 +|---|---|
88 +| Canonical assets | 767 830 |
89 +| Sales (transactions) | 2 228 672 |
90 +| Live listings | 657 107 |
91 +| Price observations (guide values, never transactions) | 14 476 398 |
92 +| Auction lots tracked | 39 502 (≈ 22 100 live or upcoming) |
93 +| Sources catalogued / connectors implemented | 443 / 212 |
94 +| Taxonomy nodes | 136 categories, 96 families |
95 +| Indices | RARE + 20 subindices (published only above their constituent minimum) |
96 +
97 +## Product surface
98 +
99 +Public pages (Next.js 16 App Router, server-rendered, dense financial-terminal design, dark mode first-class):
100 +
101 +| Route | Purpose |
102 +|---|---|
103 +| `/` | Hero + universal search, live system stats, RARE index panel, market snapshot, Trending, Record Sales, Rare Radar, Value Opportunities (gated), Ending auctions, Latest sales, categories |
104 +| `/explore`, `/search` | Filterable asset browser and hybrid search (FTS + trigram + natural-language parsing, ⌘K palette) |
105 +| `/asset/[slug]` | Asset terminal: hero with RIV range and score explainers, price history (sales as dots, RIV overlay with band), valuation breakdown and evidence, market depth & time to sell, sales, listings, **auctions**, grades (grade curve), population, images, history, comparables, sources |
106 +| `/screener` | Filter builder (category, grader/grade, RIV, confidence, liquidity, rarity, 1M/1Y change, sales, listings, spread, drawdown), presets, sortable columns, CSV / JSON export |
107 +| `/markets`, `/markets/[slug]`, `/categories`, `/market-map` | Category terminals, movers, activity, structure, treemap heatmap |
108 +| `/rareindex`, `/rareindex/[ticker]` | Indices, constituents, correlations, benchmarks (S&P 500, gold, BTC, CPI…) |
109 +| `/auctions`, `/auctions/house/[slug]`, `/auctions/calendar` | Open / ending-soon / below-RIV lots with all-in cost, house pages with buyer-premium schedules and objective stats, global calendar |
110 +| `/sales`, `/listings`, `/records`, `/radar`, `/trending`, `/news`, `/grading` | Market feeds and intelligence |
111 +| `/compare`, `/price-history`, `/set/[slug]`, `/brand/[brand]` | Comparison and entity pages |
112 +| `/scanner`, `/research` | AI identification from photo / URL / text with confidence; AI Research answering only from structured data through inspectable tools |
113 +| `/methodology`, `/data`, `/api-docs`, `/about` | Methodology center (wired to the code's constants), coverage & exports, API reference, company |
114 +| Account | Collections (portfolio with cost basis, gains, allocation, concentration, insurance schedule, CSV import/export), watchlists, alerts (11 types incl. `auction_below_riv`), price targets, saved searches, Deal Radar, My Index, public profiles `/u/[handle]` |
115 +| Admin `/admin` | Pipeline overview, connectors explorer, coverage, data quality, taxonomy proposals, events, audit, costs |
116 +
117 +## Data model — the canonical asset graph
118 +
119 +Marketplace titles are never assets. Every observation is resolved to a canonical **asset**
120 +(`assets`: category, family, franchise, brand, series, set, number, year, edition, variant, language,
121 +region, identifiers…) and to an **asset variant** (`asset_variants`: grader, grade, qualifier,
122 +condition, completeness, size — e.g. `psa|10`, `raw|near_mint`, `cgc|9.8|black_label`, `psa|unknown`
123 +for a slab whose grade could not be read). Variants never share a valuation: a PSA 10, a raw copy and
124 +a sealed box are three markets.
125 +
126 +Evidence tables keep the three kinds of numbers strictly apart (§205 of the spec):
127 +
128 +- `sales` — observed transactions: native `price` + `currency`, `price_usd` at the **sale-date** ECB
129 + rate (`fx_rate`, `fx_date`), `all_in_usd` = buyer-pays price (hammer + estimated buyer premium with a
130 + `fee_basis`), `status` valid / flagged / excluded, `confidence`, `flags`, `dedupe_key`.
131 +- `listings` — asks: lifecycle (`first_seen_at`, `last_seen_at`, `availability`, `listing_events`),
132 + `discount_to_riv` = (ask − RIV) / RIV **only when the comparison passed the gates**, flags
133 + `riv_anomaly` / `riv_review`.
134 +- `auction_lots` — estimates, current bid, hammer, USD conversions, `all_in_bid_usd`,
135 + `all_in_estimate_low_usd`, `bid_vs_riv`, `estimate_vs_riv`, `assessment_verdict`.
136 +- `price_observations` — price-guide values, always labelled "guide", never mixed with transactions.
137 +
138 +Derived layers: `valuations` (append-only history with method breakdown), `asset_stats` /
139 +`variant_stats` (current read model), `price_snapshots` (daily series), `grade_premiums`,
140 +`population_reports`, `certificates` + `certificate_sightings`, `indices` / `index_values` /
141 +`index_constituents`, `category_snapshots`, `radar_findings`, `correlations`, `benchmarks`.
142 +Schema: `packages/database/src/schema/*`, 8 migrations in `packages/database/migrations/`.
143 +
144 +## Methodology in brief
145 +
146 +Full text: [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) and the live
147 +[Methodology Center](https://www.rareindex.io/methodology), which imports its thresholds from the
148 +code so it cannot drift.
149 +
150 +- **RIV (RareIndex Valuation)** — ensemble of robust estimators over the variant's valid
151 + transactions (exponentially-weighted mean, medians 5/10/20, trust- and recency-weighted mean,
152 + trimmed mean), median of the core; 365-day window extended to 3 years when thin (confidence
153 + capped); falls back to grade-adjusted comparables, then to guide observations, each with capped
154 + confidence. Inputs are **buyer-pays prices** (`coalesce(all_in_usd, price_usd)`).
155 +- **Confidence** — 0.35·sample + 0.30·(1 − dispersion) + 0.20·recency + 0.15·source trust; labels
156 + High ≥ 0.75, Medium ≥ 0.5, Low > 0.2, else Insufficient.
157 +- **Range** — low / fair / high from p25 / RIV / p75, at least ±5 %.
158 +- **Representative variant** — the default (raw / base) variant when it has a transaction-based RIV
159 + with ≥ 5 sales and confidence ≥ 0.5, else the qualifying variant with the most sales. Changes,
160 + ATH / ATL and drawdown are measured on that variant's own series.
161 +- **Ask vs RIV and the anomaly gate** — an ask is compared only with a transaction-based RIV of the
162 + **same variant** (≥ 5 sales, confidence ≥ 0.5, match ≥ 0.7). Asks below 0.1× or above 10× RIV are
163 + flagged **Data/identity anomaly**; asks more than 50 % below RIV are held for **review**; a deal is
164 + between −10 % and −50 %. Auction bids are never asks. Deal Score is multiplicative in depth,
165 + confidence, sample, liquidity and match quality.
166 +- **Buyer premiums** — per-house marginal schedules (`data/fees/auction-houses.json`, 57 houses,
167 + each rate labelled published / approximate / none, default 22 % when unknown); VAT, duties and
168 + shipping are explicitly excluded.
169 +- **Liquidity Score** — sales per month, listing depth, source count, days between sales, ask-sold
170 + spread. **Rarity Score** — population, production quantity, sales and listings per year.
171 + **Momentum / Trending** with quality gates (canonical identity, ≥ 5 sales, plausible move ≤ ±500 %).
172 +- **Outliers** — modified z-score > 3.5 on log prices, flagged and audited, never deleted; bundles
173 + excluded.
174 +- **Sales verification** — heuristic label verified / likely / unverified / excluded from status,
175 + confidence, source type and trust; documented as heuristic.
176 +- **Indices** — chain-linked trimmed equal-weight log returns, eligibility confidence ≥ 0.4 and ≥ 3
177 + sales in 12 months, single-return clamp ln 2.5, base 1000 on 2024-01-01, minimum 10 constituents
178 + (RARE: 25 across published subindices); unpublished indices are shown as "in development".
179 +- **FX** — ECB reference rates (frankfurter), sale-date rate for history, native values never
180 + overwritten.
17 181
18 182 ## Architecture
19 183
20 −TypeScript end-to-end in a pnpm monorepo: Next.js 16 web app, Fastify public API, PostgreSQL 17
21 −(pg_trgm, unaccent, pgvector) as system of record, pg-boss workers for the ingestion pipeline, and a
22 −connector framework routing Direct API → Firecrawl → Scrapfly with quality scoring and health
23 −metrics. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the layout and data flow, and
24 −[`docs/adr/`](docs/adr/) for the decisions behind it.
184 +TypeScript end-to-end in a pnpm monorepo.
185 +
186 +```
187 + ┌──────────────┐ HTTPS (MacLustr Tunnel: WireGuard + Caddy)
188 + browsers ───▶ │ apps/web │ Next.js 16 · App Router · webpack build · Tailwind v4
189 + │ :8210 │──┐ /api/v1/* rewritten to the API
190 + └──────────────┘ │
191 + ┌──────────────┐ │
192 + developers ─▶ │ apps/api │◀─┘ Fastify · /v1 · API keys (hashed) · tiers · rate limits · CSV
193 + │ :8211 │
194 + └──────┬───────┘
195 + │ SQL (drizzle-orm, raw sql for hot paths)
196 + ┌──────▼───────────────────────────────────────────────┐
197 + │ PostgreSQL 17 · pg_trgm · unaccent · pgvector · pgboss│
198 + └──────▲───────────────────────────────────────────────┘
199 + │
200 + ┌──────┴───────┐ pg-boss queues: crawl.run · normalize.batch · resolve.batch
201 + │ workers/ │ valuation.asset/rebuild · auctions.assess · indices.daily
202 + │ (tsx) │ snapshots.daily · radar.scan · health.compute · fx.sync
203 + └──────┬───────┘ benchmarks.sync · images.process · listings.expire · certs.verify · account.jobs
204 + │
205 + ┌──────▼───────┐ 212 connectors · router Direct API → Firecrawl → Scrapfly
206 + │ connectors/ │ per-host policies, circuit breakers, quality scoring, fixtures
207 + └──────────────┘
208 +```
209 +
210 +Packages: `shared` (schemas, money, stats, ids), `database` (Drizzle schema, migrations, seed),
211 +`taxonomy` (136 categories, condition scales, graders, grade parser, variant keys, indices),
212 +`connectors` (SDK, router, adapters Shopify / WooCommerce / sitemap / RSS / PDF / schema.org, health),
213 +`valuation` (RIV, outliers, grade premiums, scores, ask assessment, fees, auction assessment, depth,
214 +liquidation, verification), `indices` (chain-linked, repeat-sales, correlations), `search`
215 +(FTS + trigram + NL parser), `ai` (provider router Anthropic / OpenAI-compatible, cost ledger),
216 +`notify` (Resend e-mail).
217 +
218 +Decisions: [`docs/adr/`](docs/adr/). Layout and data flow: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
219 +
220 +## Repository layout
221 +
222 +```
223 +rareindex/
224 +├── apps/
225 +│ ├── web/ Next.js site (src/app routes, components, lib/queries, brand assets, OG images)
226 +│ └── api/ Fastify public API (/v1), OpenAPI generator, key minting script
227 +├── packages/
228 +│ ├── shared/ database/ taxonomy/ connectors/ valuation/ indices/ search/ ai/ notify/
229 +├── workers/ pg-boss workers + operator CLI (cli.ts): crawler, normalizer, entity-resolution,
230 +│ valuation, auctions, indices, image-processing, account jobs, fees backfill…
231 +├── connectors/ api/ · firecrawl/ · scrapfly/ — one folder per source (meta.json, index.ts, tests)
232 +│ registry.json (generated), domains.json + domains.d/ (host policies)
233 +├── data/
234 +│ ├── taxonomy/ categories.json · conditions.json · graders.json · indices.json
235 +│ ├── fees/ auction-houses.json — buyer-premium schedules
236 +│ ├── sources/ 443 catalogued sources → sources.json → docs/connectors/SOURCES.md
237 +│ └── fixtures/ real captured payloads for every connector (465 JSON, 22 HTML)
238 +├── docs/ ARCHITECTURE · METHODOLOGY · API · openapi.json · SPEC-original · adr/ · connectors/
239 +├── scripts/ build-registry · build-sources · connector-new · gen-openapi · ri
240 +└── CLAUDE.md condensed product mandate and engineering rules
241 +```
242 +
243 +## Pipeline and workers
25 244
26 245 ```
27 −apps/web · apps/api · packages/{shared,database,taxonomy,connectors,valuation,indices,search,ai}
28 −workers/ · connectors/ (+ registry.json) · data/{taxonomy,fixtures} · docs/ · scripts/
246 +source → raw_records (immutable, content-hashed)
247 + → normalize() connector adapter → NormalizedRecord (sale | listing | auction_lot |
248 + price_observation | catalog_item | population_report | news_item), Zod-validated
249 + → entity resolution identifiers → canonical key → constrained trigram → create; grade re-parsed
250 + from titles (refineGrade), variant ensured (never raw for a slab)
251 + → canonical writes sales (dedupe key, sale-date FX, buyer-pays price), listings (lifecycle events),
252 + auction lots, observations, populations, certificates
253 + → valuation per-variant RIV, outlier flags, ask-vs-RIV with the anomaly gate,
254 + representative variant, asset_stats / variant_stats / price_snapshots
255 + → auctions.assess USD + all-in bid / estimate + bid vs RIV every 20 minutes
256 + → indices / snapshots RARE + subindices, category snapshots, correlations, benchmarks
257 + → radar / alerts record sales, price discrepancies, auctions below RIV, reappearances,
258 + ultra-low populations; member alerts, digests, price targets
29 259 ```
30 260
31 −Data flow: `raw_records` (immutable) → `normalize()` → `normalized_records` → entity resolution →
32 −`assets` / `asset_variants` → dedupe and price validation (flags + audit log, nothing deleted) →
33 −`sales` / `listings` / `auction_lots` / `price_observations` → valuations and stats → indices.
261 +Operational safeguards: per-run time budgets, one crawl per connector at a time, orphan-run recovery,
262 +per-host concurrency and circuit breakers, challenge-page detection, schema-drift and
263 +result-collapse detectors feeding `connector_health`, audit log for every flag or move.
264 +
265 +## Connectors
266 +
267 +212 implemented connectors (179 `api`, 43 `firecrawl`, 8 `scrapfly` folders; 20 countries,
268 +125 categories) over 443 catalogued sources (201 implemented, 10 gated behind API keys, 108 blocked
269 +by access controls that RareIndex will not circumvent, 108 rejected with a reason). The catalogue is
270 +[`docs/connectors/SOURCES.md`](docs/connectors/SOURCES.md).
271 +
272 +A new source needs four things: `meta.json`, an extraction schema, a pure `normalize()` adapter and
273 +real fixtures with tests. Scaffold with `pnpm connector:new <id> --url … [--adapter shopify|woocommerce]`,
274 +then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide:
275 +[`docs/connectors/ADDING_A_CONNECTOR.md`](docs/connectors/ADDING_A_CONNECTOR.md).
276 +
277 +Routing is Direct API → structured feed → Firecrawl → Scrapfly → manual review, with a quality gate
278 +(title 20 %, price 20 %, status 15 %, date 10 %, images 10 %, identifiers 10 %, description 5 %,
279 +category 5 %, currency 5 %). RareIndex never bypasses logins, paywalls, CAPTCHAs or other access
280 +controls.
281 +
282 +## Public API
283 +
284 +Base URL `https://www.rareindex.io/api/v1` (proxied to Fastify). Bearer keys `ri_live_…` are stored
285 +hashed; anonymous calls get the public tier. Every list supports cursor pagination and `?format=csv`;
286 +responses carry `meta.as_of` and attribution.
287 +
288 +| Endpoint | Description |
289 +|---|---|
290 +| `GET /v1/assets/search` · `/v1/assets/:id` | Search and asset detail (id or slug) |
291 +| `GET /v1/assets/:id/sales` · `/listings` · `/history` | Evidence and daily series (native + USD + all-in + fee basis) |
292 +| `GET /v1/assets/:id/depth` | Market depth ±5/10/20 %, days on market, time-to-sale bands, fair buy / sell ladder |
293 +| `GET /v1/assets/:id/auctions` · `GET /v1/auctions/lots` | Lots with estimates, bids, all-in cost, `bid_vs_riv`, verdict; filters `below_riv`, `house`, `ending_within_hours`, `category` |
294 +| `GET /v1/indices` · `/v1/indices/:ticker/history` | RARE and subindices |
295 +| `GET /v1/markets` · `/v1/markets/:slug` · `/v1/categories` · `/v1/trending` · `/v1/sales/latest` · `/v1/records` · `/v1/stats` | Markets and feeds |
296 +| `GET /v1/openapi.json` | OpenAPI 3.1 |
297 +
298 +Reference: [`docs/API.md`](docs/API.md), [`docs/openapi.json`](docs/openapi.json), live at
299 +[/api-docs](https://www.rareindex.io/api-docs).
34 300
35 301 ## Quick start
36 302
37 −Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with the `pgvector` extension available.
303 +Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pgvector`.
38 304
39 305 ```bash
40 306 pnpm install
41 307 createdb rareindex
42 −cp .env.example .env # fill DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, AI and Resend keys
43 −pnpm db:migrate && pnpm db:seed # creates extensions + tables, seeds taxonomy, graders, indices, registry
308 +cp .env.example .env # DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, ANTHROPIC_API_KEY, RESEND_API_KEY, SESSION_SECRET, ADMIN_TOKEN
309 +pnpm db:migrate && pnpm db:seed # extensions, tables, taxonomy, graders, indices, connector registry
44 310
45 −pnpm dev:web # Next.js on http://localhost:3000
46 −pnpm dev:api # public REST API (/v1)
47 −pnpm worker # pipeline workers (crawl, normalize, resolve, value, index)
48 −pnpm ri --help # operational CLI (run a connector, recompute valuations, health…)
311 +pnpm dev:web # http://localhost:3000
312 +pnpm dev:api # http://localhost:8211/v1
313 +pnpm worker # pipeline workers
314 +pnpm ri stats # operator CLI
49 315 ```
50 316
51 −Useful scripts: `pnpm typecheck`, `pnpm test`, `pnpm build`, `pnpm db:generate` (after editing
52 −`packages/database/src/schema/*`), `pnpm tsx scripts/build-registry.ts` (rebuilds
53 −`connectors/registry.json` from per-connector `meta.json` files).
317 +Build and verify: `pnpm typecheck`, `pnpm test`, `pnpm build`. Schema change: edit
318 +`packages/database/src/schema/*`, `pnpm db:generate`, commit the migration.
319 +
320 +## Operator CLI
54 321
55 −## Adding a connector
322 +`pnpm ri <command>` (on the node: `node node_modules/tsx/dist/cli.mjs workers/cli.ts <command>`):
56 323
57 −A new source needs only four things: connector metadata (`meta.json`), an extraction schema, a pure
58 −`normalize()` adapter and real captured fixtures with tests. The rest of the pipeline works
59 −unchanged. Conventions and the module contract are in [`connectors/README.md`](connectors/README.md).
324 +| Command | Purpose |
325 +|---|---|
326 +| `connectors` · `crawl <id> [--mode probe\|incremental\|backfill]` · `backfill <id…> [--start-only]` | Sources and crawls |
327 +| `normalize [--connector id]` · `resolve [--limit N]` | Pipeline stages |
328 +| `value [--asset id\|--all] [--history]` · `premiums` · `regrade [--dry-run] [--recheck]` | Valuations, grade premiums, grade repair (moves mis-varianted rows, audited) |
329 +| `fees --backfill [--force]` · `auctions --assess [--all]` | Buyer-pays prices on sales, auction lot assessment |
330 +| `index` · `snapshots` · `radar` · `benchmarks` · `fx [--backfill]` | Indices, snapshots, Rare Radar, benchmarks, exchange rates |
331 +| `certs [--backfill]` · `images` · `expire` · `health [--probe]` · `stats` · `run-all` · `worker` | Certificates, images, listing expiry, health, stats, full cycle |
332 +
333 +## Testing and quality gates
334 +
335 +- Unit tests (vitest) in every package and app: valuation (RIV, outliers, premiums, ask gate, fees,
336 + auction assessment, depth, liquidation, verification), taxonomy (grade parser, variant keys), search
337 + parser, indices, shared money/stats, workers, API, web helpers.
338 +- 213 connector test files run every fixture through the schema, price, currency and date checks
339 + (`pnpm connectors:test`).
340 +- `pnpm -r run typecheck` (TypeScript strict), `pnpm --filter @rareindex/web run lint`.
341 +- Data-quality gates are code, not conventions: anomaly band, review threshold, plausible-change
342 + bound, representative-variant rule, numeric overflow guard, `Data/identity anomaly` and
343 + `Needs review` labels in the UI.
344 +
345 +## Deployment
346 +
347 +Production runs on the **MacLustr** cluster ([www.maclustr.io](https://www.maclustr.io)) — node
348 +M3U96a (Mac Studio, PostgreSQL 17 + pgvector local) — orchestrated by the `mld` gateway and published
349 +through the MacLustr Tunnel (WireGuard + Caddy on an OVH gateway) at **https://www.rareindex.io**.
350 +PM2 processes: `rareindex-web` (8210), `rareindex-api` (8211), `rareindex-worker`. Secrets live only
351 +in the `mld` manifest.
352 +
353 +Release procedure: commit → `git archive HEAD` synced to the node → `pnpm db:migrate` →
354 +`pnpm --filter @rareindex/web run build` (detached, ~15 min: 146 static pages) → `pm2 restart`.
355 +Never delete `.next` under a running server; `/data` is rendered on demand with a one-hour cache
356 +because its aggregates exceed the build-time prerender budget.
357 +
358 +## Brand
359 +
360 +<p align="center">
361 + <img src="apps/web/public/brand/rareindex-mark.svg" alt="RareIndex mark" width="96">
362 +</p>
363 +
364 +The mark is a cut stone — crown, girdle, pavilion — whose facets are drawn by an ascending index
365 +line: rarity and market data in one shape. Ink `#0B0F19`, gold `#F1D08A → #C9963F`, mint
366 +`#2FB673 → #5FE3A1`. Wordmark **Rare<span>Index</span>** with "Index" in gold. Source files in
367 +[`apps/web/public/brand/`](apps/web/public/brand/) (mark, light and dark lockups, README); favicons,
368 +PWA icons, `favicon.ico` and the Apple touch icon are generated from the mark by
369 +`apps/web/scripts/brand-assets.mjs`; the site-wide share image is rendered by
370 +`apps/web/src/app/opengraph-image.tsx`.
60 371
61 372 ## Data honesty rules
62 373
63 −- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions or listings.
64 −- Price-guide values are stored as observations, never as transactions.
65 −- Valuations are published only with sample size, confidence, low/high range and timestamp.
66 −- Outliers are flagged and audited, never silently deleted.
67 −- Sale dates come from the source; crawl time is never substituted.
68 −- Historical prices are converted with the exchange rate of the sale date.
374 +- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions, bids or listings.
375 +- Observed data, model estimates, user data and external asks are always distinguished (§205).
376 +- Price-guide values are observations, never transactions; a listing price is never a market value.
377 +- Valuations publish only with sample size, confidence, range and timestamp; missing evidence reads "Not enough data".
378 +- Outliers are flagged and audited, never deleted; every repair (`regrade`, flags) is in `audit_log`.
379 +- Sale dates come from the source; historical prices use the exchange rate of the sale date; native values are never overwritten.
380 +- Hammer prices are never compared with marketplace prices without the buyer's premium; the fee basis is shown.
381 +- Neutral wording for anomalies ("anomalous transaction", "needs review"); RareIndex never accuses anyone of fraud and never claims to authenticate items.
69 382
70 383 Details: [`docs/adr/0002-data-honesty.md`](docs/adr/0002-data-honesty.md).
71 384
72 −## Deployment
385 +## Roadmap
73 386
74 −Production runs on a MacLustr node (PostgreSQL 17 + pgvector) deployed through the `mld` gateway:
75 −PM2 processes `rareindex-web`, `rareindex-api`, `rareindex-worker` and an ngrok tunnel serving
76 −**https://www.rareindex.io**. Secrets live only in the node's `.env` / the `mld` manifest.
387 +Delivered on 2026-09-11: valuation safety (anomaly gate, variant-aware asks, representative variant,
388 +grade repair), screener, methodology center, market depth and time-to-sell, sale verification labels,
389 +buyer premiums and auction intelligence, brand. Next, in order of leverage:
77 390
78 −## Disclaimers
391 +1. Entity resolution for unique objects (auction lots on generic assets), certificate-number and
392 + multi-token search, `unaccent` in FTS.
393 +2. `fx_rate` / `fx_date` on listings, observations and lots; per-lot bid counts for houses that omit them.
394 +3. Index history reconstruction from transactions (`ri value --all --history`), repeat-sales publication.
395 +4. Population reports through grader partnerships; cross-grader empirical model.
396 +5. Portfolio overhaul (fees, liquidity horizon, confidence aggregation), Scanner multi-item and visual embeddings.
397 +6. Cross-border arbitrage with FX, fees and duties; regional price indices.
398 +7. Natural-language query engine over the screener; webhooks; API tier enforcement and shared rate-limit store.
79 399
80 −RareIndex is an information platform. Valuations are estimates derived from observed public data.
81 −Listing prices are not confirmed transactions. Past performance does not guarantee future results.
82 −RareIndex does not authenticate items. Third-party names, marks and content belong to their owners
83 −and are attributed to their sources; RareIndex respects source terms, robots policies, rate limits,
84 −copyright, privacy and database rights, and never bypasses authentication, paywalls or CAPTCHAs.
400 +## Author, hosting and contact
401 +
402 +RareIndex is designed, built and operated by **Simon-Pierre Boucher** (Québec, Canada).
403 +It is hosted on **MacLustr** — [www.maclustr.io](https://www.maclustr.io) — a distributed
404 +Apple-silicon compute cluster operated by the same team.
85 405
86 −## License
406 +- General contact: [contact@spboucher.ai](mailto:contact@spboucher.ai)
407 +- Data corrections and source requests: data@rareindex.io · API and partnerships: hello@rareindex.io · Rights holders: legal@rareindex.io
408 +
409 +## Disclaimers and license
410 +
411 +RareIndex is an information platform. Valuations are estimates derived from observed public data.
412 +Listing prices and auction estimates are not confirmed transactions. Past performance does not
413 +guarantee future results. RareIndex does not authenticate items and does not provide investment
414 +advice. Third-party names, marks and content belong to their owners and are attributed to their
415 +sources; RareIndex respects source terms, robots policies, rate limits, copyright, privacy and
416 +database rights.
87 417
88 −Proprietary — © 2026 RareIndex. All rights reserved.
418 +Proprietary — © 2026 Simon-Pierre Boucher / RareIndex. All rights reserved.
89 419