README: full platform documentation with the RareIndex brand (logo, product surface, data model, methodology, architecture, pipeline, connectors, API, CLI, deployment, roadmap, author and hosting on MacLustr)

1

+

2

+ <picture>

3

+ <source media="(prefers-color-scheme: dark)" srcset="apps/web/public/brand/rareindex-logo-dark.svg">

4

+ <img src="apps/web/public/brand/rareindex-logo.svg" alt="RareIndex — The Global Market for Collectibles" width="520">

5

+ </picture>

6

+

7

+

8

+

9

+ <a href="https://www.rareindex.io">www.rareindex.io</a> ·

10

+ <a href="https://www.rareindex.io/methodology">Methodology</a> ·

11

+ <a href="https://www.rareindex.io/api-docs">API</a> ·

12

+ <a href="https://www.rareindex.io/screener">Screener</a> ·

13

+ <a href="https://www.rareindex.io/auctions">Auctions</a> ·

14

+ <a href="mailto:contact@spboucher.ai">contact@spboucher.ai</a>

15

+

16

+

17

+

18

+ <img alt="TypeScript" src="https://img.shields.io/badge/TypeScript-strict-3178c6"> <img alt="Next.js 16" src="https://img.shields.io/badge/Next.js-16-000000"> <img alt="Fastify" src="https://img.shields.io/badge/Fastify-%2Fv1-202020"> <img alt="PostgreSQL 17" src="https://img.shields.io/badge/PostgreSQL-17%20%2B%20pgvector-336791"> <img alt="pg-boss" src="https://img.shields.io/badge/workers-pg--boss-4b5563"> <img alt="connectors" src="https://img.shields.io/badge/connectors-212-c9963f"> <img alt="hosted on MacLustr" src="https://img.shields.io/badge/hosted%20on-MacLustr-0b0f19">

19

+

20

+

1

21

# RareIndex

2

22

3

−**The global market for collectibles.** RareIndex.io is a market-intelligence, search, pricing and

4

−index platform for collectible assets: trading cards, sports cards, watches, sneakers, LEGO, comics,

5

−video games, toys, coins, art, wine and, over time, every category the collectibles economy contains.

6

−It continuously collects public market data from marketplaces, auction houses, dealers, grading

7

−companies, manufacturer catalogs and archives, resolves it into one canonical asset graph, and turns

8

−it into research-grade data: valuations with confidence levels, comparable sales, live listings,

9

−rarity and liquidity scores, price histories, and the RareIndex family of indices (`RARE` and its

10

−category subindices).

11

−

12

−RareIndex is an intelligence platform, not a marketplace aggregator. Every number shown is traceable

13

−to its source, every estimate carries a sample size and a confidence level, and when the evidence is

14

−insufficient the platform says "Data unavailable" rather than inventing a figure. The full product

15

−mandate is in [`docs/SPEC-original.md`](docs/SPEC-original.md); the condensed working version that

16

−guides implementation is [`CLAUDE.md`](CLAUDE.md).

23

+**RareIndex is the market-data and intelligence layer for physical collectibles** — the terminal the

24

+collectibles economy never had. It continuously collects public market evidence (marketplaces,

25

+auction houses, dealers, price guides, grading companies, manufacturer catalogs), resolves it into a

26

+canonical asset graph and turns it into research-grade data: a valuation with a confidence level and

27

+a range, comparable sales, live asks, market depth, liquidity and rarity scores, auction intelligence

28

+with buyer-pays costs, price histories and the **RARE** family of indices.

29

+

30

+A marketplace tells you *"this item is for sale for $4,000"*. RareIndex tells you:

31

+

32

+> *This asset has a RareIndex Valuation of $3,180 (high confidence, 42 comparable sales of the same

33

+> variant across five sources), the market is up 12.4 % over 12 months, typical time-to-sale is 18

34

+> days, liquidity is above the category average, and the current $2,650 ask is 16.7 % below fair

35

+> value — while a $472 slab of a different grade matched to the same card is a data anomaly, not a

36

+> deal.*

37

+

38

+Every number is traceable to its source, carries a sample size, a confidence and a freshness, and

39

+when the evidence is not there the platform says **"Not enough data"** instead of inventing a figure.

40

+

41

+---

42

+

43

+## Table of contents

44

+

45

+1. [What RareIndex answers](#what-rareindex-answers)

46

+2. [Live platform at a glance](#live-platform-at-a-glance)

47

+3. [Product surface](#product-surface)

48

+4. [Data model — the canonical asset graph](#data-model--the-canonical-asset-graph)

49

+5. [Methodology in brief](#methodology-in-brief)

50

+6. [Architecture](#architecture)

51

+7. [Repository layout](#repository-layout)

52

+8. [Pipeline and workers](#pipeline-and-workers)

53

+9. [Connectors](#connectors)

54

+10. [Public API](#public-api)

55

+11. [Quick start](#quick-start)

56

+12. [Operator CLI](#operator-cli)

57

+13. [Testing and quality gates](#testing-and-quality-gates)

58

+14. [Deployment](#deployment)

59

+15. [Brand](#brand)

60

+16. [Data honesty rules](#data-honesty-rules)

61

+17. [Roadmap](#roadmap)

62

+18. [Author, hosting and contact](#author-hosting-and-contact)

63

+19. [Disclaimers and license](#disclaimers-and-license)

64

+

65

+---

66

+

67

+## What RareIndex answers

68

+

69

+| Question | Where |

70

+|---|---|

71

+| What is this collectible, exactly (set, number, variant, language, grade, grader)? | Canonical assets and variants, Scanner (photo / URL / text) |

72

+| What is it worth, and how confident are we? | **RIV** — RareIndex Valuation with low / fair / high range, confidence label, sample size, method breakdown |

73

+| What has it actually sold for? | Verified sales timeline with source links, buyer-pays price, verification label |

74

+| What are sellers asking right now? | Live listings, ask vs RIV with the anomaly gate, market depth ±5 / 10 / 20 % |

75

+| How liquid is it, how fast could it sell? | Liquidity Score 0–100, days on market, time-to-sale by asking band, fast / typical / patient sell prices |

76

+| How rare is it? | Rarity Score from population reports, production quantity and market frequency |

77

+| Is that auction lot cheap once fees are counted? | Auction intelligence: all-in bid or estimate (hammer + buyer's premium) vs RIV, per-house fee schedules |

78

+| Is the market rising or falling? | RARE global index and 20 subindices, category markets, market map, screener |

79

+| Which collectibles match my criteria? | `/screener` with presets and CSV / JSON export |

80

+| What is unusual right now? | Rare Radar: record sales, price discrepancies, reappearances, auctions below RIV |

81

+

82

+## Live platform at a glance

83

+

84

+Production figures on 2026-09-11 (PostgreSQL on the MacLustr node):

85

+

86

+| Measure | Value |

87

+|---|---|

88

+| Canonical assets | 767 830 |

89

+| Sales (transactions) | 2 228 672 |

90

+| Live listings | 657 107 |

91

+| Price observations (guide values, never transactions) | 14 476 398 |

92

+| Auction lots tracked | 39 502 (≈ 22 100 live or upcoming) |

93

+| Sources catalogued / connectors implemented | 443 / 212 |

94

+| Taxonomy nodes | 136 categories, 96 families |

95

+| Indices | RARE + 20 subindices (published only above their constituent minimum) |

96

+

97

+## Product surface

98

+

99

+Public pages (Next.js 16 App Router, server-rendered, dense financial-terminal design, dark mode first-class):

100

+

101

+| Route | Purpose |

102

+|---|---|

103

+| `/` | Hero + universal search, live system stats, RARE index panel, market snapshot, Trending, Record Sales, Rare Radar, Value Opportunities (gated), Ending auctions, Latest sales, categories |

104

+| `/explore`, `/search` | Filterable asset browser and hybrid search (FTS + trigram + natural-language parsing, ⌘K palette) |

105

+| `/asset/[slug]` | Asset terminal: hero with RIV range and score explainers, price history (sales as dots, RIV overlay with band), valuation breakdown and evidence, market depth & time to sell, sales, listings, **auctions**, grades (grade curve), population, images, history, comparables, sources |

106

+| `/screener` | Filter builder (category, grader/grade, RIV, confidence, liquidity, rarity, 1M/1Y change, sales, listings, spread, drawdown), presets, sortable columns, CSV / JSON export |

107

+| `/markets`, `/markets/[slug]`, `/categories`, `/market-map` | Category terminals, movers, activity, structure, treemap heatmap |

108

+| `/rareindex`, `/rareindex/[ticker]` | Indices, constituents, correlations, benchmarks (S&P 500, gold, BTC, CPI…) |

109

+| `/auctions`, `/auctions/house/[slug]`, `/auctions/calendar` | Open / ending-soon / below-RIV lots with all-in cost, house pages with buyer-premium schedules and objective stats, global calendar |

110

+| `/sales`, `/listings`, `/records`, `/radar`, `/trending`, `/news`, `/grading` | Market feeds and intelligence |

111

+| `/compare`, `/price-history`, `/set/[slug]`, `/brand/[brand]` | Comparison and entity pages |

112

+| `/scanner`, `/research` | AI identification from photo / URL / text with confidence; AI Research answering only from structured data through inspectable tools |

113

+| `/methodology`, `/data`, `/api-docs`, `/about` | Methodology center (wired to the code's constants), coverage & exports, API reference, company |

114

+| Account | Collections (portfolio with cost basis, gains, allocation, concentration, insurance schedule, CSV import/export), watchlists, alerts (11 types incl. `auction_below_riv`), price targets, saved searches, Deal Radar, My Index, public profiles `/u/[handle]` |

115

+| Admin `/admin` | Pipeline overview, connectors explorer, coverage, data quality, taxonomy proposals, events, audit, costs |

116

+

117

+## Data model — the canonical asset graph

118

+

119

+Marketplace titles are never assets. Every observation is resolved to a canonical **asset**

120

+(`assets`: category, family, franchise, brand, series, set, number, year, edition, variant, language,

121

+region, identifiers…) and to an **asset variant** (`asset_variants`: grader, grade, qualifier,

122

123

+for a slab whose grade could not be read). Variants never share a valuation: a PSA 10, a raw copy and

124

+a sealed box are three markets.

125

+

126

+Evidence tables keep the three kinds of numbers strictly apart (§205 of the spec):

127

+

128

+- `sales` — observed transactions: native `price` + `currency`, `price_usd` at the **sale-date** ECB

129

+ rate (`fx_rate`, `fx_date`), `all_in_usd` = buyer-pays price (hammer + estimated buyer premium with a

130

+ `fee_basis`), `status` valid / flagged / excluded, `confidence`, `flags`, `dedupe_key`.

131

+- `listings` — asks: lifecycle (`first_seen_at`, `last_seen_at`, `availability`, `listing_events`),

132

+ `discount_to_riv` = (ask − RIV) / RIV **only when the comparison passed the gates**, flags

133

+ `riv_anomaly` / `riv_review`.

134

+- `auction_lots` — estimates, current bid, hammer, USD conversions, `all_in_bid_usd`,

135

+ `all_in_estimate_low_usd`, `bid_vs_riv`, `estimate_vs_riv`, `assessment_verdict`.

136

+- `price_observations` — price-guide values, always labelled "guide", never mixed with transactions.

137

+

138

+Derived layers: `valuations` (append-only history with method breakdown), `asset_stats` /

139

+`variant_stats` (current read model), `price_snapshots` (daily series), `grade_premiums`,

140

+`population_reports`, `certificates` + `certificate_sightings`, `indices` / `index_values` /

141

+`index_constituents`, `category_snapshots`, `radar_findings`, `correlations`, `benchmarks`.

142

+Schema: `packages/database/src/schema/*`, 8 migrations in `packages/database/migrations/`.

143

+

144

+## Methodology in brief

145

+

146

+Full text: [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) and the live

147

+[Methodology Center](https://www.rareindex.io/methodology), which imports its thresholds from the

148

+code so it cannot drift.

149

+

150

+- **RIV (RareIndex Valuation)** — ensemble of robust estimators over the variant's valid

151

+ transactions (exponentially-weighted mean, medians 5/10/20, trust- and recency-weighted mean,

152

+ trimmed mean), median of the core; 365-day window extended to 3 years when thin (confidence

153

+ capped); falls back to grade-adjusted comparables, then to guide observations, each with capped

154

+ confidence. Inputs are **buyer-pays prices** (`coalesce(all_in_usd, price_usd)`).

155

+- **Confidence** — 0.35·sample + 0.30·(1 − dispersion) + 0.20·recency + 0.15·source trust; labels

156

+ High ≥ 0.75, Medium ≥ 0.5, Low > 0.2, else Insufficient.

157

+- **Range** — low / fair / high from p25 / RIV / p75, at least ±5 %.

158

+- **Representative variant** — the default (raw / base) variant when it has a transaction-based RIV

159

+ with ≥ 5 sales and confidence ≥ 0.5, else the qualifying variant with the most sales. Changes,

160

+ ATH / ATL and drawdown are measured on that variant's own series.

161

+- **Ask vs RIV and the anomaly gate** — an ask is compared only with a transaction-based RIV of the

162

+ **same variant** (≥ 5 sales, confidence ≥ 0.5, match ≥ 0.7). Asks below 0.1× or above 10× RIV are

163

+ flagged **Data/identity anomaly**; asks more than 50 % below RIV are held for **review**; a deal is

164

+ between −10 % and −50 %. Auction bids are never asks. Deal Score is multiplicative in depth,

165

+ confidence, sample, liquidity and match quality.

166

+- **Buyer premiums** — per-house marginal schedules (`data/fees/auction-houses.json`, 57 houses,

167

+ each rate labelled published / approximate / none, default 22 % when unknown); VAT, duties and

168

+ shipping are explicitly excluded.

169

+- **Liquidity Score** — sales per month, listing depth, source count, days between sales, ask-sold

170

+ spread. **Rarity Score** — population, production quantity, sales and listings per year.

171

+ **Momentum / Trending** with quality gates (canonical identity, ≥ 5 sales, plausible move ≤ ±500 %).

172

+- **Outliers** — modified z-score > 3.5 on log prices, flagged and audited, never deleted; bundles

173

+ excluded.

174

+- **Sales verification** — heuristic label verified / likely / unverified / excluded from status,

175

+ confidence, source type and trust; documented as heuristic.

176

+- **Indices** — chain-linked trimmed equal-weight log returns, eligibility confidence ≥ 0.4 and ≥ 3

177

+ sales in 12 months, single-return clamp ln 2.5, base 1000 on 2024-01-01, minimum 10 constituents

178

+ (RARE: 25 across published subindices); unpublished indices are shown as "in development".

179

+- **FX** — ECB reference rates (frankfurter), sale-date rate for history, native values never

180

+ overwritten.

17

181

18

182

## Architecture

19

183

20

−TypeScript end-to-end in a pnpm monorepo: Next.js 16 web app, Fastify public API, PostgreSQL 17

21

−(pg_trgm, unaccent, pgvector) as system of record, pg-boss workers for the ingestion pipeline, and a

22

−connector framework routing Direct API → Firecrawl → Scrapfly with quality scoring and health

23

−metrics. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the layout and data flow, and

24

−[`docs/adr/`](docs/adr/) for the decisions behind it.

184

+TypeScript end-to-end in a pnpm monorepo.

185

+

186

+```

187

+ ┌──────────────┐ HTTPS (MacLustr Tunnel: WireGuard + Caddy)

188

+ browsers ───▶ │ apps/web │ Next.js 16 · App Router · webpack build · Tailwind v4

189

+ │ :8210 │──┐ /api/v1/* rewritten to the API

190

+ └──────────────┘ │

191

+ ┌──────────────┐ │

192

+ developers ─▶ │ apps/api │◀─┘ Fastify · /v1 · API keys (hashed) · tiers · rate limits · CSV

193

+ │ :8211 │

194

+ └──────┬───────┘

195

+ │ SQL (drizzle-orm, raw sql for hot paths)

196

+ ┌──────▼───────────────────────────────────────────────┐

197

+ │ PostgreSQL 17 · pg_trgm · unaccent · pgvector · pgboss│

198

+ └──────▲───────────────────────────────────────────────┘

199

+ │

200

+ ┌──────┴───────┐ pg-boss queues: crawl.run · normalize.batch · resolve.batch

201

+ │ workers/ │ valuation.asset/rebuild · auctions.assess · indices.daily

202

+ │ (tsx) │ snapshots.daily · radar.scan · health.compute · fx.sync

203

+ └──────┬───────┘ benchmarks.sync · images.process · listings.expire · certs.verify · account.jobs

204

+ │

205

+ ┌──────▼───────┐ 212 connectors · router Direct API → Firecrawl → Scrapfly

206

+ │ connectors/ │ per-host policies, circuit breakers, quality scoring, fixtures

207

+ └──────────────┘

208

+```

209

+

210

+Packages: `shared` (schemas, money, stats, ids), `database` (Drizzle schema, migrations, seed),

211

+`taxonomy` (136 categories, condition scales, graders, grade parser, variant keys, indices),

212

+`connectors` (SDK, router, adapters Shopify / WooCommerce / sitemap / RSS / PDF / schema.org, health),

213

+`valuation` (RIV, outliers, grade premiums, scores, ask assessment, fees, auction assessment, depth,

214

+liquidation, verification), `indices` (chain-linked, repeat-sales, correlations), `search`

215

+(FTS + trigram + NL parser), `ai` (provider router Anthropic / OpenAI-compatible, cost ledger),

216

+`notify` (Resend e-mail).

217

+

218

+Decisions: [`docs/adr/`](docs/adr/). Layout and data flow: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).

219

+

220

+## Repository layout

221

+

222

+```

223

+rareindex/

224

+├── apps/

225

+│ ├── web/ Next.js site (src/app routes, components, lib/queries, brand assets, OG images)

226

+│ └── api/ Fastify public API (/v1), OpenAPI generator, key minting script

227

+├── packages/

228

+│ ├── shared/ database/ taxonomy/ connectors/ valuation/ indices/ search/ ai/ notify/

229

+├── workers/ pg-boss workers + operator CLI (cli.ts): crawler, normalizer, entity-resolution,

230

+│ valuation, auctions, indices, image-processing, account jobs, fees backfill…

231

+├── connectors/ api/ · firecrawl/ · scrapfly/ — one folder per source (meta.json, index.ts, tests)

232

+│ registry.json (generated), domains.json + domains.d/ (host policies)

233

+├── data/

234

+│ ├── taxonomy/ categories.json · conditions.json · graders.json · indices.json

235

+│ ├── fees/ auction-houses.json — buyer-premium schedules

236

+│ ├── sources/ 443 catalogued sources → sources.json → docs/connectors/SOURCES.md

237

+│ └── fixtures/ real captured payloads for every connector (465 JSON, 22 HTML)

238

+├── docs/ ARCHITECTURE · METHODOLOGY · API · openapi.json · SPEC-original · adr/ · connectors/

239

+├── scripts/ build-registry · build-sources · connector-new · gen-openapi · ri

240

+└── CLAUDE.md condensed product mandate and engineering rules

241

+```

242

+

243

+## Pipeline and workers

25

244

26

245

```

27

−apps/web · apps/api · packages/{shared,database,taxonomy,connectors,valuation,indices,search,ai}

28

−workers/ · connectors/ (+ registry.json) · data/{taxonomy,fixtures} · docs/ · scripts/

246

+source → raw_records (immutable, content-hashed)

247

+ → normalize() connector adapter → NormalizedRecord (sale | listing | auction_lot |

248

+ price_observation | catalog_item | population_report | news_item), Zod-validated

249

+ → entity resolution identifiers → canonical key → constrained trigram → create; grade re-parsed

250

+ from titles (refineGrade), variant ensured (never raw for a slab)

251

+ → canonical writes sales (dedupe key, sale-date FX, buyer-pays price), listings (lifecycle events),

252

+ auction lots, observations, populations, certificates

253

+ → valuation per-variant RIV, outlier flags, ask-vs-RIV with the anomaly gate,

254

+ representative variant, asset_stats / variant_stats / price_snapshots

255

+ → auctions.assess USD + all-in bid / estimate + bid vs RIV every 20 minutes

256

+ → indices / snapshots RARE + subindices, category snapshots, correlations, benchmarks

257

+ → radar / alerts record sales, price discrepancies, auctions below RIV, reappearances,

258

+ ultra-low populations; member alerts, digests, price targets

29

259

```

30

260

31

−Data flow: `raw_records` (immutable) → `normalize()` → `normalized_records` → entity resolution →

32

−`assets` / `asset_variants` → dedupe and price validation (flags + audit log, nothing deleted) →

33

−`sales` / `listings` / `auction_lots` / `price_observations` → valuations and stats → indices.

261

+Operational safeguards: per-run time budgets, one crawl per connector at a time, orphan-run recovery,

262

+per-host concurrency and circuit breakers, challenge-page detection, schema-drift and

263

+result-collapse detectors feeding `connector_health`, audit log for every flag or move.

264

+

265

+## Connectors

266

+

267

+212 implemented connectors (179 `api`, 43 `firecrawl`, 8 `scrapfly` folders; 20 countries,

268

+125 categories) over 443 catalogued sources (201 implemented, 10 gated behind API keys, 108 blocked

269

+by access controls that RareIndex will not circumvent, 108 rejected with a reason). The catalogue is

270

+[`docs/connectors/SOURCES.md`](docs/connectors/SOURCES.md).

271

+

272

+A new source needs four things: `meta.json`, an extraction schema, a pure `normalize()` adapter and

273

+real fixtures with tests. Scaffold with `pnpm connector:new <id> --url … [--adapter shopify|woocommerce]`,

274

+then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide:

275

+[`docs/connectors/ADDING_A_CONNECTOR.md`](docs/connectors/ADDING_A_CONNECTOR.md).

276

+

277

+Routing is Direct API → structured feed → Firecrawl → Scrapfly → manual review, with a quality gate

278

+(title 20 %, price 20 %, status 15 %, date 10 %, images 10 %, identifiers 10 %, description 5 %,

279

+category 5 %, currency 5 %). RareIndex never bypasses logins, paywalls, CAPTCHAs or other access

280

+controls.

281

+

282

+## Public API

283

+

284

+Base URL `https://www.rareindex.io/api/v1` (proxied to Fastify). Bearer keys `ri_live_…` are stored

285

+hashed; anonymous calls get the public tier. Every list supports cursor pagination and `?format=csv`;

286

+responses carry `meta.as_of` and attribution.

287

+

288

+| Endpoint | Description |

289

+|---|---|

290

+| `GET /v1/assets/search` · `/v1/assets/:id` | Search and asset detail (id or slug) |

291

+| `GET /v1/assets/:id/sales` · `/listings` · `/history` | Evidence and daily series (native + USD + all-in + fee basis) |

292

+| `GET /v1/assets/:id/depth` | Market depth ±5/10/20 %, days on market, time-to-sale bands, fair buy / sell ladder |

293

+| `GET /v1/assets/:id/auctions` · `GET /v1/auctions/lots` | Lots with estimates, bids, all-in cost, `bid_vs_riv`, verdict; filters `below_riv`, `house`, `ending_within_hours`, `category` |

294

+| `GET /v1/indices` · `/v1/indices/:ticker/history` | RARE and subindices |

295

+| `GET /v1/markets` · `/v1/markets/:slug` · `/v1/categories` · `/v1/trending` · `/v1/sales/latest` · `/v1/records` · `/v1/stats` | Markets and feeds |

296

+| `GET /v1/openapi.json` | OpenAPI 3.1 |

297

+

298

+Reference: [`docs/API.md`](docs/API.md), [`docs/openapi.json`](docs/openapi.json), live at

299

+[/api-docs](https://www.rareindex.io/api-docs).

34

300

35

301

## Quick start

36

302

37

−Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with the `pgvector` extension available.

303

+Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pgvector`.

38

304

39

305

```bash

40

306

pnpm install

41

307

createdb rareindex

42

−cp .env.example .env # fill DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, AI and Resend keys

43

−pnpm db:migrate && pnpm db:seed # creates extensions + tables, seeds taxonomy, graders, indices, registry

308

+cp .env.example .env # DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, ANTHROPIC_API_KEY, RESEND_API_KEY, SESSION_SECRET, ADMIN_TOKEN

309

+pnpm db:migrate && pnpm db:seed # extensions, tables, taxonomy, graders, indices, connector registry

44

310

45

−pnpm dev:web # Next.js on http://localhost:3000

46

−pnpm dev:api # public REST API (/v1)

47

−pnpm worker # pipeline workers (crawl, normalize, resolve, value, index)

48

−pnpm ri --help # operational CLI (run a connector, recompute valuations, health…)

311

+pnpm dev:web # http://localhost:3000

312

+pnpm dev:api # http://localhost:8211/v1

313

+pnpm worker # pipeline workers

314

+pnpm ri stats # operator CLI

49

315

```

50

316

51

−Useful scripts: `pnpm typecheck`, `pnpm test`, `pnpm build`, `pnpm db:generate` (after editing

52

−`packages/database/src/schema/*`), `pnpm tsx scripts/build-registry.ts` (rebuilds

53

−`connectors/registry.json` from per-connector `meta.json` files).

317

+Build and verify: `pnpm typecheck`, `pnpm test`, `pnpm build`. Schema change: edit

318

+`packages/database/src/schema/*`, `pnpm db:generate`, commit the migration.

319

+

320

+## Operator CLI

54

321

55

−## Adding a connector

322

+`pnpm ri <command>` (on the node: `node node_modules/tsx/dist/cli.mjs workers/cli.ts <command>`):

56

323

57

−A new source needs only four things: connector metadata (`meta.json`), an extraction schema, a pure

58

−`normalize()` adapter and real captured fixtures with tests. The rest of the pipeline works

59

−unchanged. Conventions and the module contract are in [`connectors/README.md`](connectors/README.md).

324

+| Command | Purpose |

325

+|---|---|

326

327

+| `normalize [--connector id]` · `resolve [--limit N]` | Pipeline stages |

328

+| `value [--asset id\|--all] [--history]` · `premiums` · `regrade [--dry-run] [--recheck]` | Valuations, grade premiums, grade repair (moves mis-varianted rows, audited) |

329

+| `fees --backfill [--force]` · `auctions --assess [--all]` | Buyer-pays prices on sales, auction lot assessment |

330

+| `index` · `snapshots` · `radar` · `benchmarks` · `fx [--backfill]` | Indices, snapshots, Rare Radar, benchmarks, exchange rates |

331

+| `certs [--backfill]` · `images` · `expire` · `health [--probe]` · `stats` · `run-all` · `worker` | Certificates, images, listing expiry, health, stats, full cycle |

332

+

333

+## Testing and quality gates

334

+

335

+- Unit tests (vitest) in every package and app: valuation (RIV, outliers, premiums, ask gate, fees,

336

+ auction assessment, depth, liquidation, verification), taxonomy (grade parser, variant keys), search

337

+ parser, indices, shared money/stats, workers, API, web helpers.

338

+- 213 connector test files run every fixture through the schema, price, currency and date checks

339

+ (`pnpm connectors:test`).

340

+- `pnpm -r run typecheck` (TypeScript strict), `pnpm --filter @rareindex/web run lint`.

341

+- Data-quality gates are code, not conventions: anomaly band, review threshold, plausible-change

342

+ bound, representative-variant rule, numeric overflow guard, `Data/identity anomaly` and

343

+ `Needs review` labels in the UI.

344

+

345

+## Deployment

346

+

347

+Production runs on the **MacLustr** cluster ([www.maclustr.io](https://www.maclustr.io)) — node

348

+M3U96a (Mac Studio, PostgreSQL 17 + pgvector local) — orchestrated by the `mld` gateway and published

349

+through the MacLustr Tunnel (WireGuard + Caddy on an OVH gateway) at **https://www.rareindex.io**.

350

+PM2 processes: `rareindex-web` (8210), `rareindex-api` (8211), `rareindex-worker`. Secrets live only

351

+in the `mld` manifest.

352

+

353

+Release procedure: commit → `git archive HEAD` synced to the node → `pnpm db:migrate` →

354

+`pnpm --filter @rareindex/web run build` (detached, ~15 min: 146 static pages) → `pm2 restart`.

355

+Never delete `.next` under a running server; `/data` is rendered on demand with a one-hour cache

356

+because its aggregates exceed the build-time prerender budget.

357

+

358

+## Brand

359

+

360

+

361

+ <img src="apps/web/public/brand/rareindex-mark.svg" alt="RareIndex mark" width="96">

362

+

363

+

364

+The mark is a cut stone — crown, girdle, pavilion — whose facets are drawn by an ascending index

365

+line: rarity and market data in one shape. Ink `#0B0F19`, gold `#F1D08A → #C9963F`, mint

366

+`#2FB673 → #5FE3A1`. Wordmark **RareIndex** with "Index" in gold. Source files in

367

+[`apps/web/public/brand/`](apps/web/public/brand/) (mark, light and dark lockups, README); favicons,

368

+PWA icons, `favicon.ico` and the Apple touch icon are generated from the mark by

369

+`apps/web/scripts/brand-assets.mjs`; the site-wide share image is rendered by

370

+`apps/web/src/app/opengraph-image.tsx`.

60

371

61

372

## Data honesty rules

62

373

63

−- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions or listings.

64

−- Price-guide values are stored as observations, never as transactions.

65

−- Valuations are published only with sample size, confidence, low/high range and timestamp.

66

−- Outliers are flagged and audited, never silently deleted.

67

−- Sale dates come from the source; crawl time is never substituted.

68

−- Historical prices are converted with the exchange rate of the sale date.

374

+- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions, bids or listings.

375

+- Observed data, model estimates, user data and external asks are always distinguished (§205).

376

+- Price-guide values are observations, never transactions; a listing price is never a market value.

377

+- Valuations publish only with sample size, confidence, range and timestamp; missing evidence reads "Not enough data".

378

+- Outliers are flagged and audited, never deleted; every repair (`regrade`, flags) is in `audit_log`.

379

+- Sale dates come from the source; historical prices use the exchange rate of the sale date; native values are never overwritten.

380

+- Hammer prices are never compared with marketplace prices without the buyer's premium; the fee basis is shown.

381

+- Neutral wording for anomalies ("anomalous transaction", "needs review"); RareIndex never accuses anyone of fraud and never claims to authenticate items.

69

382

70

383

Details: [`docs/adr/0002-data-honesty.md`](docs/adr/0002-data-honesty.md).

71

384

72

−## Deployment

385

+## Roadmap

73

386

74

−Production runs on a MacLustr node (PostgreSQL 17 + pgvector) deployed through the `mld` gateway:

75

−PM2 processes `rareindex-web`, `rareindex-api`, `rareindex-worker` and an ngrok tunnel serving

76

−**https://www.rareindex.io**. Secrets live only in the node's `.env` / the `mld` manifest.

387

+Delivered on 2026-09-11: valuation safety (anomaly gate, variant-aware asks, representative variant,

388

+grade repair), screener, methodology center, market depth and time-to-sell, sale verification labels,

389

+buyer premiums and auction intelligence, brand. Next, in order of leverage:

77

390

78

−## Disclaimers

391

+1. Entity resolution for unique objects (auction lots on generic assets), certificate-number and

392

+ multi-token search, `unaccent` in FTS.

393

+2. `fx_rate` / `fx_date` on listings, observations and lots; per-lot bid counts for houses that omit them.

394

+3. Index history reconstruction from transactions (`ri value --all --history`), repeat-sales publication.

395

+4. Population reports through grader partnerships; cross-grader empirical model.

396

+5. Portfolio overhaul (fees, liquidity horizon, confidence aggregation), Scanner multi-item and visual embeddings.

397

+6. Cross-border arbitrage with FX, fees and duties; regional price indices.

398

+7. Natural-language query engine over the screener; webhooks; API tier enforcement and shared rate-limit store.

79

399

80

−RareIndex is an information platform. Valuations are estimates derived from observed public data.

81

−Listing prices are not confirmed transactions. Past performance does not guarantee future results.

82

−RareIndex does not authenticate items. Third-party names, marks and content belong to their owners

83

−and are attributed to their sources; RareIndex respects source terms, robots policies, rate limits,

84

−copyright, privacy and database rights, and never bypasses authentication, paywalls or CAPTCHAs.

400

+## Author, hosting and contact

401

+

402

+RareIndex is designed, built and operated by **Simon-Pierre Boucher** (Québec, Canada).

403

+It is hosted on **MacLustr** — [www.maclustr.io](https://www.maclustr.io) — a distributed

404

+Apple-silicon compute cluster operated by the same team.

85

405

86

−## License

406

+- General contact: [contact@spboucher.ai](mailto:contact@spboucher.ai)

407

+- Data corrections and source requests: data@rareindex.io · API and partnerships: hello@rareindex.io · Rights holders: legal@rareindex.io

408

+

409

+## Disclaimers and license

410

+

411

+RareIndex is an information platform. Valuations are estimates derived from observed public data.

412

+Listing prices and auction estimates are not confirmed transactions. Past performance does not

413

+guarantee future results. RareIndex does not authenticate items and does not provide investment

414

+advice. Third-party names, marks and content belong to their owners and are attributed to their

415

+sources; RareIndex respects source terms, robots policies, rate limits, copyright, privacy and

416

+database rights.

87

417

88

418

89

419

spb/rareindex

Theme