SPB Git
spboucher.ai

spb/rareindex

Public

snapshot final BHS128b 2026-09-26 avant retrait de rareindex (disque plein : base 351 Go + images 299 Go) — docs PENDING-SCHEMA, README, CLAUDE.md, pnpm-workspace

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Sep 27, 2026) parent d4d0eca Browse files

7 changed files +150 −396

7 changed files
Unified Split
modified CLAUDE.md +1 −2
@@ -182,8 +182,7 @@ One canonical asset graph containing every collectible, every important variatio
182 ## Repository conventions (added by the implementation) 182 ## Repository conventions (added by the implementation)
183 183
184 - Package manager `pnpm`; `pnpm typecheck`, `pnpm test`, `pnpm build`. Workers run with `tsx`. 184 - Package manager `pnpm`; `pnpm typecheck`, `pnpm test`, `pnpm build`. Workers run with `tsx`.
185−- Secrets only in `.env` (git-ignored) or the PM2 config on the server (`BHS128b:~/apps/pm2.rareindex.config.cjs`, generated from the former `mld` manifest kept in `~/apps/.manifests/`). Never in code or fixtures. 185+- Secrets only in `.env` (git-ignored) or the `mld` manifest on the cluster. Never in code or fixtures.
186−- Production (since 2026-09-14): OVH server **BHS128b**, outside `mld` — Postgres 17 native, image store `~/apps/rareindex/data/images` (270 GB), PM2 under systemd, route `https://www.rareindex.io → wg1 10.67.0.62:8210`. Release procedure and details in `README.md` → Deployment. The M3U96a copy is a cold copy.
187 - Add a connector: `pnpm connector:new <id> --url … [--adapter shopify|woocommerce]` → `connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts,README.md}` + real fixtures in `data/fixtures/<id>/`, source entry in `data/sources/entries/<group>.json`, host policy in `connectors/domains.d/<group>.json`; then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide: `docs/connectors/ADDING_A_CONNECTOR.md`; catalogue: `docs/connectors/SOURCES.md`. 186 - Add a connector: `pnpm connector:new <id> --url … [--adapter shopify|woocommerce]` → `connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts,README.md}` + real fixtures in `data/fixtures/<id>/`, source entry in `data/sources/entries/<group>.json`, host policy in `connectors/domains.d/<group>.json`; then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide: `docs/connectors/ADDING_A_CONNECTOR.md`; catalogue: `docs/connectors/SOURCES.md`.
188 - DB changes: edit `packages/database/src/schema/*`, run `pnpm db:generate`, commit the SQL migration. 187 - DB changes: edit `packages/database/src/schema/*`, run `pnpm db:generate`, commit the SQL migration.
189 - UI: dense, neutral, tabular numbers (`num` utility), `<Unavailable/>` for missing evidence, every number with its sample size / confidence / freshness when it is an estimate. 188 - UI: dense, neutral, tabular numbers (`num` utility), `<Unavailable/>` for missing evidence, every number with its sample size / confidence / freshness when it is an estimate.
modified README.md +56 −394
@@ -1,426 +1,88 @@
1−<p align="center">
2− <picture>
3− <source media="(prefers-color-scheme: dark)" srcset="apps/web/public/brand/rareindex-logo-dark.svg">
4− <img src="apps/web/public/brand/rareindex-logo.svg" alt="RareIndex — The Global Market for Collectibles" width="520">
5− </picture>
6−</p>
7−
8−<p align="center">
9− <a href="https://www.rareindex.io"><strong>www.rareindex.io</strong></a> ·
10− <a href="https://www.rareindex.io/methodology">Methodology</a> ·
11− <a href="https://www.rareindex.io/api-docs">API</a> ·
12− <a href="https://www.rareindex.io/screener">Screener</a> ·
13− <a href="https://www.rareindex.io/auctions">Auctions</a> ·
14− <a href="mailto:contact@spboucher.ai">contact@spboucher.ai</a>
15−</p>
16−
17−<p align="center">
18− <img alt="TypeScript" src="https://img.shields.io/badge/TypeScript-strict-3178c6"> <img alt="Next.js 16" src="https://img.shields.io/badge/Next.js-16-000000"> <img alt="Fastify" src="https://img.shields.io/badge/Fastify-%2Fv1-202020"> <img alt="PostgreSQL 17" src="https://img.shields.io/badge/PostgreSQL-17%20%2B%20pgvector-336791"> <img alt="pg-boss" src="https://img.shields.io/badge/workers-pg--boss-4b5563"> <img alt="connectors" src="https://img.shields.io/badge/connectors-212-c9963f"> <img alt="hosted on MacLustr" src="https://img.shields.io/badge/hosted%20on-MacLustr-0b0f19">
19−</p>
20−
21 # RareIndex 1 # RareIndex
22 2
23−**RareIndex is the market-data and intelligence layer for physical collectibles** — the terminal the 3+**The global market for collectibles.** RareIndex.io is a market-intelligence, search, pricing and
24−collectibles economy never had. It continuously collects public market evidence (marketplaces, 4+index platform for collectible assets: trading cards, sports cards, watches, sneakers, LEGO, comics,
25−auction houses, dealers, price guides, grading companies, manufacturer catalogs), resolves it into a 5+video games, toys, coins, art, wine and, over time, every category the collectibles economy contains.
26−canonical asset graph and turns it into research-grade data: a valuation with a confidence level and 6+It continuously collects public market data from marketplaces, auction houses, dealers, grading
27−a range, comparable sales, live asks, market depth, liquidity and rarity scores, auction intelligence 7+companies, manufacturer catalogs and archives, resolves it into one canonical asset graph, and turns
28−with buyer-pays costs, price histories and the **RARE** family of indices. 8+it into research-grade data: valuations with confidence levels, comparable sales, live listings,
29− 9+rarity and liquidity scores, price histories, and the RareIndex family of indices (`RARE` and its
30−A marketplace tells you *"this item is for sale for $4,000"*. RareIndex tells you: 10+category subindices).
31− 11+
32−> *This asset has a RareIndex Valuation of $3,180 (high confidence, 42 comparable sales of the same 12+RareIndex is an intelligence platform, not a marketplace aggregator. Every number shown is traceable
33−> variant across five sources), the market is up 12.4 % over 12 months, typical time-to-sale is 18 13+to its source, every estimate carries a sample size and a confidence level, and when the evidence is
34−> days, liquidity is above the category average, and the current $2,650 ask is 16.7 % below fair 14+insufficient the platform says "Data unavailable" rather than inventing a figure. The full product
35−> value — while a $472 slab of a different grade matched to the same card is a data anomaly, not a 15+mandate is in [`docs/SPEC-original.md`](docs/SPEC-original.md); the condensed working version that
36−> deal.* 16+guides implementation is [`CLAUDE.md`](CLAUDE.md).
37−
38−Every number is traceable to its source, carries a sample size, a confidence and a freshness, and
39−when the evidence is not there the platform says **"Not enough data"** instead of inventing a figure.
40−
41−---
42−
43−## Table of contents
44−
45−1. [What RareIndex answers](#what-rareindex-answers)
46−2. [Live platform at a glance](#live-platform-at-a-glance)
47−3. [Product surface](#product-surface)
48−4. [Data model — the canonical asset graph](#data-model--the-canonical-asset-graph)
49−5. [Methodology in brief](#methodology-in-brief)
50−6. [Architecture](#architecture)
51−7. [Repository layout](#repository-layout)
52−8. [Pipeline and workers](#pipeline-and-workers)
53−9. [Connectors](#connectors)
54−10. [Public API](#public-api)
55−11. [Quick start](#quick-start)
56−12. [Operator CLI](#operator-cli)
57−13. [Testing and quality gates](#testing-and-quality-gates)
58−14. [Deployment](#deployment)
59−15. [Brand](#brand)
60−16. [Data honesty rules](#data-honesty-rules)
61−17. [Roadmap](#roadmap)
62−18. [Author, hosting and contact](#author-hosting-and-contact)
63−19. [Disclaimers and license](#disclaimers-and-license)
64−
65−---
66−
67−## What RareIndex answers
68−
69−| Question | Where |
70−|---|---|
71−| What is this collectible, exactly (set, number, variant, language, grade, grader)? | Canonical assets and variants, Scanner (photo / URL / text) |
72−| What is it worth, and how confident are we? | **RIV** — RareIndex Valuation with low / fair / high range, confidence label, sample size, method breakdown |
73−| What has it actually sold for? | Verified sales timeline with source links, buyer-pays price, verification label |
74−| What are sellers asking right now? | Live listings, ask vs RIV with the anomaly gate, market depth ±5 / 10 / 20 % |
75−| How liquid is it, how fast could it sell? | Liquidity Score 0–100, days on market, time-to-sale by asking band, fast / typical / patient sell prices |
76−| How rare is it? | Rarity Score from population reports, production quantity and market frequency |
77−| Is that auction lot cheap once fees are counted? | Auction intelligence: all-in bid or estimate (hammer + buyer's premium) vs RIV, per-house fee schedules |
78−| Is the market rising or falling? | RARE global index and 20 subindices, category markets, market map, screener |
79−| Which collectibles match my criteria? | `/screener` with presets and CSV / JSON export |
80−| What is unusual right now? | Rare Radar: record sales, price discrepancies, reappearances, auctions below RIV |
81−
82−## Live platform at a glance
83−
84−Production figures on 2026-09-11 (PostgreSQL on the MacLustr node):
85−
86−| Measure | Value |
87−|---|---|
88−| Canonical assets | 767 830 |
89−| Sales (transactions) | 2 228 672 |
90−| Live listings | 657 107 |
91−| Price observations (guide values, never transactions) | 14 476 398 |
92−| Auction lots tracked | 39 502 (≈ 22 100 live or upcoming) |
93−| Sources catalogued / connectors implemented | 443 / 212 |
94−| Taxonomy nodes | 136 categories, 96 families |
95−| Indices | RARE + 20 subindices (published only above their constituent minimum) |
96−
97−## Product surface
98−
99−Public pages (Next.js 16 App Router, server-rendered, dense financial-terminal design, dark mode first-class):
100−
101−| Route | Purpose |
102−|---|---|
103−| `/` | Hero + universal search, live system stats, RARE index panel, market snapshot, Trending, Record Sales, Rare Radar, Value Opportunities (gated), Ending auctions, Latest sales, categories |
104−| `/explore`, `/search` | Filterable asset browser and hybrid search (FTS + trigram + natural-language parsing, ⌘K palette) |
105−| `/asset/[slug]` | Asset terminal: hero with RIV range and score explainers, price history (sales as dots, RIV overlay with band), valuation breakdown and evidence, market depth & time to sell, sales, listings, **auctions**, grades (grade curve), population, images, history, comparables, sources |
106−| `/screener` | Filter builder (category, grader/grade, RIV, confidence, liquidity, rarity, 1M/1Y change, sales, listings, spread, drawdown), presets, sortable columns, CSV / JSON export |
107−| `/markets`, `/markets/[slug]`, `/categories`, `/market-map` | Category terminals, movers, activity, structure, treemap heatmap |
108−| `/rareindex`, `/rareindex/[ticker]` | Indices, constituents, correlations, benchmarks (S&P 500, gold, BTC, CPI…) |
109−| `/auctions`, `/auctions/house/[slug]`, `/auctions/calendar` | Open / ending-soon / below-RIV lots with all-in cost, house pages with buyer-premium schedules and objective stats, global calendar |
110−| `/sales`, `/listings`, `/records`, `/radar`, `/trending`, `/news`, `/grading` | Market feeds and intelligence |
111−| `/compare`, `/price-history`, `/set/[slug]`, `/brand/[brand]` | Comparison and entity pages |
112−| `/scanner`, `/research` | AI identification from photo / URL / text with confidence; AI Research answering only from structured data through inspectable tools |
113−| `/methodology`, `/data`, `/api-docs`, `/about` | Methodology center (wired to the code's constants), coverage & exports, API reference, company |
114−| Account | Collections (portfolio with cost basis, gains, allocation, concentration, insurance schedule, CSV import/export), watchlists, alerts (11 types incl. `auction_below_riv`), price targets, saved searches, Deal Radar, My Index, public profiles `/u/[handle]` |
115−| Admin `/admin` | Pipeline overview, connectors explorer, coverage, data quality, taxonomy proposals, events, audit, costs |
116−
117−## Data model — the canonical asset graph
118−
119−Marketplace titles are never assets. Every observation is resolved to a canonical **asset**
120−(`assets`: category, family, franchise, brand, series, set, number, year, edition, variant, language,
121−region, identifiers…) and to an **asset variant** (`asset_variants`: grader, grade, qualifier,
122−condition, completeness, size — e.g. `psa|10`, `raw|near_mint`, `cgc|9.8|black_label`, `psa|unknown`
123−for a slab whose grade could not be read). Variants never share a valuation: a PSA 10, a raw copy and
124−a sealed box are three markets.
125−
126−Evidence tables keep the three kinds of numbers strictly apart (§205 of the spec):
127−
128−- `sales` — observed transactions: native `price` + `currency`, `price_usd` at the **sale-date** ECB
129− rate (`fx_rate`, `fx_date`), `all_in_usd` = buyer-pays price (hammer + estimated buyer premium with a
130− `fee_basis`), `status` valid / flagged / excluded, `confidence`, `flags`, `dedupe_key`.
131−- `listings` — asks: lifecycle (`first_seen_at`, `last_seen_at`, `availability`, `listing_events`),
132− `discount_to_riv` = (ask − RIV) / RIV **only when the comparison passed the gates**, flags
133− `riv_anomaly` / `riv_review`.
134−- `auction_lots` — estimates, current bid, hammer, USD conversions, `all_in_bid_usd`,
135− `all_in_estimate_low_usd`, `bid_vs_riv`, `estimate_vs_riv`, `assessment_verdict`.
136−- `price_observations` — price-guide values, always labelled "guide", never mixed with transactions.
137−
138−Derived layers: `valuations` (append-only history with method breakdown), `asset_stats` /
139−`variant_stats` (current read model), `price_snapshots` (daily series), `grade_premiums`,
140−`population_reports`, `certificates` + `certificate_sightings`, `indices` / `index_values` /
141−`index_constituents`, `category_snapshots`, `radar_findings`, `correlations`, `benchmarks`.
142−Schema: `packages/database/src/schema/*`, 8 migrations in `packages/database/migrations/`.
143−
144−## Methodology in brief
145−
146−Full text: [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) and the live
147−[Methodology Center](https://www.rareindex.io/methodology), which imports its thresholds from the
148−code so it cannot drift.
149−
150−- **RIV (RareIndex Valuation)** — ensemble of robust estimators over the variant's valid
151− transactions (exponentially-weighted mean, medians 5/10/20, trust- and recency-weighted mean,
152− trimmed mean), median of the core; 365-day window extended to 3 years when thin (confidence
153− capped); falls back to grade-adjusted comparables, then to guide observations, each with capped
154− confidence. Inputs are **buyer-pays prices** (`coalesce(all_in_usd, price_usd)`).
155−- **Confidence** — 0.35·sample + 0.30·(1 − dispersion) + 0.20·recency + 0.15·source trust; labels
156− High ≥ 0.75, Medium ≥ 0.5, Low > 0.2, else Insufficient.
157−- **Range** — low / fair / high from p25 / RIV / p75, at least ±5 %.
158−- **Representative variant** — the default (raw / base) variant when it has a transaction-based RIV
159− with ≥ 5 sales and confidence ≥ 0.5, else the qualifying variant with the most sales. Changes,
160− ATH / ATL and drawdown are measured on that variant's own series.
161−- **Ask vs RIV and the anomaly gate** — an ask is compared only with a transaction-based RIV of the
162− **same variant** (≥ 5 sales, confidence ≥ 0.5, match ≥ 0.7). Asks below 0.1× or above 10× RIV are
163− flagged **Data/identity anomaly**; asks more than 50 % below RIV are held for **review**; a deal is
164− between −10 % and −50 %. Auction bids are never asks. Deal Score is multiplicative in depth,
165− confidence, sample, liquidity and match quality.
166−- **Buyer premiums** — per-house marginal schedules (`data/fees/auction-houses.json`, 57 houses,
167− each rate labelled published / approximate / none, default 22 % when unknown); VAT, duties and
168− shipping are explicitly excluded.
169−- **Liquidity Score** — sales per month, listing depth, source count, days between sales, ask-sold
170− spread. **Rarity Score** — population, production quantity, sales and listings per year.
171− **Momentum / Trending** with quality gates (canonical identity, ≥ 5 sales, plausible move ≤ ±500 %).
172−- **Outliers** — modified z-score > 3.5 on log prices, flagged and audited, never deleted; bundles
173− excluded.
174−- **Sales verification** — heuristic label verified / likely / unverified / excluded from status,
175− confidence, source type and trust; documented as heuristic.
176−- **Indices** — chain-linked trimmed equal-weight log returns, eligibility confidence ≥ 0.4 and ≥ 3
177− sales in 12 months, single-return clamp ln 2.5, base 1000 on 2024-01-01, minimum 10 constituents
178− (RARE: 25 across published subindices); unpublished indices are shown as "in development".
179−- **FX** — ECB reference rates (frankfurter), sale-date rate for history, native values never
180− overwritten.
181 17
182 ## Architecture 18 ## Architecture
183 19
184−TypeScript end-to-end in a pnpm monorepo. 20+TypeScript end-to-end in a pnpm monorepo: Next.js 16 web app, Fastify public API, PostgreSQL 17
185− 21+(pg_trgm, unaccent, pgvector) as system of record, pg-boss workers for the ingestion pipeline, and a
186−``` 22+connector framework routing Direct API → Firecrawl → Scrapfly with quality scoring and health
187− ┌──────────────┐ HTTPS (MacLustr Tunnel: WireGuard + Caddy) 23+metrics. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the layout and data flow, and
188− browsers ───▶ │ apps/web │ Next.js 16 · App Router · webpack build · Tailwind v4 24+[`docs/adr/`](docs/adr/) for the decisions behind it.
189− │ :8210 │──┐ /api/v1/* rewritten to the API
190− └──────────────┘ │
191− ┌──────────────┐ │
192− developers ─▶ │ apps/api │◀─┘ Fastify · /v1 · API keys (hashed) · tiers · rate limits · CSV
193− │ :8211 │
194− └──────┬───────┘
195− │ SQL (drizzle-orm, raw sql for hot paths)
196− ┌──────▼───────────────────────────────────────────────┐
197− │ PostgreSQL 17 · pg_trgm · unaccent · pgvector · pgboss│
198− └──────▲───────────────────────────────────────────────┘
199− │
200− ┌──────┴───────┐ pg-boss queues: crawl.run · normalize.batch · resolve.batch
201− │ workers/ │ valuation.asset/rebuild · auctions.assess · indices.daily
202− │ (tsx) │ snapshots.daily · radar.scan · health.compute · fx.sync
203− └──────┬───────┘ benchmarks.sync · images.process · listings.expire · certs.verify · account.jobs
204− │
205− ┌──────▼───────┐ 212 connectors · router Direct API → Firecrawl → Scrapfly
206− │ connectors/ │ per-host policies, circuit breakers, quality scoring, fixtures
207− └──────────────┘
208−```
209−
210−Packages: `shared` (schemas, money, stats, ids), `database` (Drizzle schema, migrations, seed),
211−`taxonomy` (136 categories, condition scales, graders, grade parser, variant keys, indices),
212−`connectors` (SDK, router, adapters Shopify / WooCommerce / sitemap / RSS / PDF / schema.org, health),
213−`valuation` (RIV, outliers, grade premiums, scores, ask assessment, fees, auction assessment, depth,
214−liquidation, verification), `indices` (chain-linked, repeat-sales, correlations), `search`
215−(FTS + trigram + NL parser), `ai` (provider router Anthropic / OpenAI-compatible, cost ledger),
216−`notify` (Resend e-mail).
217−
218−Decisions: [`docs/adr/`](docs/adr/). Layout and data flow: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
219−
220−## Repository layout
221−
222−```
223−rareindex/
224−├── apps/
225−│ ├── web/ Next.js site (src/app routes, components, lib/queries, brand assets, OG images)
226−│ └── api/ Fastify public API (/v1), OpenAPI generator, key minting script
227−├── packages/
228−│ ├── shared/ database/ taxonomy/ connectors/ valuation/ indices/ search/ ai/ notify/
229−├── workers/ pg-boss workers + operator CLI (cli.ts): crawler, normalizer, entity-resolution,
230−│ valuation, auctions, indices, image-processing, account jobs, fees backfill…
231−├── connectors/ api/ · firecrawl/ · scrapfly/ — one folder per source (meta.json, index.ts, tests)
232−│ registry.json (generated), domains.json + domains.d/ (host policies)
233−├── data/
234−│ ├── taxonomy/ categories.json · conditions.json · graders.json · indices.json
235−│ ├── fees/ auction-houses.json — buyer-premium schedules
236−│ ├── sources/ 443 catalogued sources → sources.json → docs/connectors/SOURCES.md
237−│ └── fixtures/ real captured payloads for every connector (465 JSON, 22 HTML)
238−├── docs/ ARCHITECTURE · METHODOLOGY · API · openapi.json · SPEC-original · adr/ · connectors/
239−├── scripts/ build-registry · build-sources · connector-new · gen-openapi · ri
240−└── CLAUDE.md condensed product mandate and engineering rules
241−```
242−
243−## Pipeline and workers
244 25
245 ``` 26 ```
246−source → raw_records (immutable, content-hashed) 27+apps/web · apps/api · packages/{shared,database,taxonomy,connectors,valuation,indices,search,ai}
247− → normalize() connector adapter → NormalizedRecord (sale | listing | auction_lot | 28+workers/ · connectors/ (+ registry.json) · data/{taxonomy,fixtures} · docs/ · scripts/
248− price_observation | catalog_item | population_report | news_item), Zod-validated
249− → entity resolution identifiers → canonical key → constrained trigram → create; grade re-parsed
250− from titles (refineGrade), variant ensured (never raw for a slab)
251− → canonical writes sales (dedupe key, sale-date FX, buyer-pays price), listings (lifecycle events),
252− auction lots, observations, populations, certificates
253− → valuation per-variant RIV, outlier flags, ask-vs-RIV with the anomaly gate,
254− representative variant, asset_stats / variant_stats / price_snapshots
255− → auctions.assess USD + all-in bid / estimate + bid vs RIV every 20 minutes
256− → indices / snapshots RARE + subindices, category snapshots, correlations, benchmarks
257− → radar / alerts record sales, price discrepancies, auctions below RIV, reappearances,
258− ultra-low populations; member alerts, digests, price targets
259 ``` 29 ```
260 30
261−Operational safeguards: per-run time budgets, one crawl per connector at a time, orphan-run recovery, 31+Data flow: `raw_records` (immutable) → `normalize()` → `normalized_records` → entity resolution →
262−per-host concurrency and circuit breakers, challenge-page detection, schema-drift and 32+`assets` / `asset_variants` → dedupe and price validation (flags + audit log, nothing deleted) →
263−result-collapse detectors feeding `connector_health`, audit log for every flag or move. 33+`sales` / `listings` / `auction_lots` / `price_observations` → valuations and stats → indices.
264−
265−## Connectors
266−
267−212 implemented connectors (179 `api`, 43 `firecrawl`, 8 `scrapfly` folders; 20 countries,
268−125 categories) over 443 catalogued sources (201 implemented, 10 gated behind API keys, 108 blocked
269−by access controls that RareIndex will not circumvent, 108 rejected with a reason). The catalogue is
270−[`docs/connectors/SOURCES.md`](docs/connectors/SOURCES.md).
271−
272−A new source needs four things: `meta.json`, an extraction schema, a pure `normalize()` adapter and
273−real fixtures with tests. Scaffold with `pnpm connector:new <id> --url … [--adapter shopify|woocommerce]`,
274−then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide:
275−[`docs/connectors/ADDING_A_CONNECTOR.md`](docs/connectors/ADDING_A_CONNECTOR.md).
276−
277−Routing is Direct API → structured feed → Firecrawl → Scrapfly → manual review, with a quality gate
278−(title 20 %, price 20 %, status 15 %, date 10 %, images 10 %, identifiers 10 %, description 5 %,
279−category 5 %, currency 5 %). RareIndex never bypasses logins, paywalls, CAPTCHAs or other access
280−controls.
281−
282−## Public API
283−
284−Base URL `https://www.rareindex.io/api/v1` (proxied to Fastify). Bearer keys `ri_live_…` are stored
285−hashed; anonymous calls get the public tier. Every list supports cursor pagination and `?format=csv`;
286−responses carry `meta.as_of` and attribution.
287−
288−| Endpoint | Description |
289−|---|---|
290−| `GET /v1/assets/search` · `/v1/assets/:id` | Search and asset detail (id or slug) |
291−| `GET /v1/assets/:id/sales` · `/listings` · `/history` | Evidence and daily series (native + USD + all-in + fee basis) |
292−| `GET /v1/assets/:id/depth` | Market depth ±5/10/20 %, days on market, time-to-sale bands, fair buy / sell ladder |
293−| `GET /v1/assets/:id/auctions` · `GET /v1/auctions/lots` | Lots with estimates, bids, all-in cost, `bid_vs_riv`, verdict; filters `below_riv`, `house`, `ending_within_hours`, `category` |
294−| `GET /v1/indices` · `/v1/indices/:ticker/history` | RARE and subindices |
295−| `GET /v1/markets` · `/v1/markets/:slug` · `/v1/categories` · `/v1/trending` · `/v1/sales/latest` · `/v1/records` · `/v1/stats` | Markets and feeds |
296−| `GET /v1/openapi.json` | OpenAPI 3.1 |
297−
298−Reference: [`docs/API.md`](docs/API.md), [`docs/openapi.json`](docs/openapi.json), live at
299−[/api-docs](https://www.rareindex.io/api-docs).
300 34
301 ## Quick start 35 ## Quick start
302 36
303−Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pgvector`. 37+Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with the `pgvector` extension available.
304 38
305 ```bash 39 ```bash
306 pnpm install 40 pnpm install
307 createdb rareindex 41 createdb rareindex
308−cp .env.example .env # DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, ANTHROPIC_API_KEY, RESEND_API_KEY, SESSION_SECRET, ADMIN_TOKEN 42+cp .env.example .env # fill DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, AI and Resend keys
309−pnpm db:migrate && pnpm db:seed # extensions, tables, taxonomy, graders, indices, connector registry 43+pnpm db:migrate && pnpm db:seed # creates extensions + tables, seeds taxonomy, graders, indices, registry
310 44
311−pnpm dev:web # http://localhost:3000 45+pnpm dev:web # Next.js on http://localhost:3000
312−pnpm dev:api # http://localhost:8211/v1 46+pnpm dev:api # public REST API (/v1)
313−pnpm worker # pipeline workers 47+pnpm worker # pipeline workers (crawl, normalize, resolve, value, index)
314−pnpm ri stats # operator CLI 48+pnpm ri --help # operational CLI (run a connector, recompute valuations, health…)
315 ``` 49 ```
316 50
317−Build and verify: `pnpm typecheck`, `pnpm test`, `pnpm build`. Schema change: edit 51+Useful scripts: `pnpm typecheck`, `pnpm test`, `pnpm build`, `pnpm db:generate` (after editing
318−`packages/database/src/schema/*`, `pnpm db:generate`, commit the migration. 52+`packages/database/src/schema/*`), `pnpm tsx scripts/build-registry.ts` (rebuilds
319− 53+`connectors/registry.json` from per-connector `meta.json` files).
320−## Operator CLI
321 54
322−`pnpm ri <command>` (on the node: `node node_modules/tsx/dist/cli.mjs workers/cli.ts <command>`): 55+## Adding a connector
323 56
324−| Command | Purpose | 57+A new source needs only four things: connector metadata (`meta.json`), an extraction schema, a pure
325−|---|---| 58+`normalize()` adapter and real captured fixtures with tests. The rest of the pipeline works
326−| `connectors` · `crawl <id> [--mode probe\|incremental\|backfill]` · `backfill <id…> [--start-only]` | Sources and crawls | 59+unchanged. Conventions and the module contract are in [`connectors/README.md`](connectors/README.md).
327−| `normalize [--connector id]` · `resolve [--limit N]` | Pipeline stages |
328−| `value [--asset id\|--all] [--history]` · `premiums` · `regrade [--dry-run] [--recheck]` | Valuations, grade premiums, grade repair (moves mis-varianted rows, audited) |
329−| `fees --backfill [--force]` · `auctions --assess [--all]` | Buyer-pays prices on sales, auction lot assessment |
330−| `index` · `snapshots` · `radar` · `benchmarks` · `fx [--backfill]` | Indices, snapshots, Rare Radar, benchmarks, exchange rates |
331−| `certs [--backfill]` · `images` · `expire` · `health [--probe]` · `stats` · `run-all` · `worker` | Certificates, images, listing expiry, health, stats, full cycle |
332−
333−## Testing and quality gates
334−
335−- Unit tests (vitest) in every package and app: valuation (RIV, outliers, premiums, ask gate, fees,
336− auction assessment, depth, liquidation, verification), taxonomy (grade parser, variant keys), search
337− parser, indices, shared money/stats, workers, API, web helpers.
338−- 213 connector test files run every fixture through the schema, price, currency and date checks
339− (`pnpm connectors:test`).
340−- `pnpm -r run typecheck` (TypeScript strict), `pnpm --filter @rareindex/web run lint`.
341−- Data-quality gates are code, not conventions: anomaly band, review threshold, plausible-change
342− bound, representative-variant rule, numeric overflow guard, `Data/identity anomaly` and
343− `Needs review` labels in the UI.
344−
345−## Deployment
346−
347−**Since 2026-09-14 production runs on the OVH server BHS128b** (`ssh BHS128b`, ubuntu@51.161.112.85, Ubuntu 24.04,
348−16 threads / 128 GB), outside `mld`: code in `~/apps/rareindex`, PostgreSQL 17 native (db/role `rareindex`/`rareindex`,
349−pgvector + pg_trgm + unaccent, en_US.UTF-8, ~150 GB), image store `~/apps/rareindex/data/images` (270 GB, ~5 M files:
350−`orig` + `w96/w192/w384/w768/w1200`, two-level hex sharding), PM2 under systemd (`pm2-ubuntu`) with
351−`~/apps/pm2.rareindex.config.cjs` generated from the mld manifest by `~/apps/.manifests/gen-pm2.py` (secrets, 0600).
352−PM2 processes: `rareindex-web` (8210), `rareindex-api` (8211), `rareindex-worker` (cron 04:30 + autorestart). Published through the
353−MacLustr Tunnel: BHS64 Caddy → wg1 `10.67.0.62:8210` (`mlt add www.rareindex.io BHS128b:8210`).
354−Release on BHS128b: `rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data --exclude logs . BHS128b:apps/rareindex/`
355−then `ssh BHS128b 'cd apps/rareindex && pnpm install --frozen-lockfile && DATABASE_URL=postgres://rareindex:rareindex@127.0.0.1:5432/rareindex pnpm db:migrate && DATABASE_URL=postgres://rareindex:rareindex@127.0.0.1:5432/rareindex NEXT_PUBLIC_SITE_URL=https://www.rareindex.io API_URL=http://127.0.0.1:8211 pnpm --filter @rareindex/web build && pm2 restart rareindex-api rareindex-web rareindex-worker && pm2 save'`.
356−pnpm ≥ 10 needs `onlyBuiltDependencies` (esbuild, sharp, @tailwindcss/oxide, unrs-resolver) in `pnpm-workspace.yaml` on a fresh install.
357−Former home (kept as a cold copy, retired from `mld` with `--keep-dir`): MacLustr node M3U96a (Mac Studio, PostgreSQL 17 + pgvector),
358−orchestrated by the `mld` gateway; the migration moved a 179 GB database (parallel `pg_dump -Fd -j 4`, 14 GB compressed, restored with
359−`pg_restore -j 8` in 21 min) and the 270 GB image store (rsync streams capped at 20 MB/s each so the Mac's 1 Gbit/s link stayed usable).
360−
361−Release procedure: commit → `git archive HEAD` synced to the node → `pnpm db:migrate` →
362−`pnpm --filter @rareindex/web run build` (detached, ~15 min: 146 static pages) → `pm2 restart`.
363−Never delete `.next` under a running server; `/data` is rendered on demand with a one-hour cache
364−because its aggregates exceed the build-time prerender budget.
365−
366−## Brand
367−
368−<p align="center">
369− <img src="apps/web/public/brand/rareindex-mark.svg" alt="RareIndex mark" width="96">
370−</p>
371−
372−The mark is a cut stone — crown, girdle, pavilion — whose facets are drawn by an ascending index
373−line: rarity and market data in one shape. Ink `#0B0F19`, gold `#F1D08A → #C9963F`, mint
374−`#2FB673 → #5FE3A1`. Wordmark **Rare<span>Index</span>** with "Index" in gold. Source files in
375−[`apps/web/public/brand/`](apps/web/public/brand/) (mark, light and dark lockups, README); favicons,
376−PWA icons, `favicon.ico` and the Apple touch icon are generated from the mark by
377−`apps/web/scripts/brand-assets.mjs`; the site-wide share image is rendered by
378−`apps/web/src/app/opengraph-image.tsx`.
379 60
380 ## Data honesty rules 61 ## Data honesty rules
381 62
382−- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions, bids or listings. 63+- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions or listings.
383−- Observed data, model estimates, user data and external asks are always distinguished (§205). 64+- Price-guide values are stored as observations, never as transactions.
384−- Price-guide values are observations, never transactions; a listing price is never a market value. 65+- Valuations are published only with sample size, confidence, low/high range and timestamp.
385−- Valuations publish only with sample size, confidence, range and timestamp; missing evidence reads "Not enough data". 66+- Outliers are flagged and audited, never silently deleted.
386−- Outliers are flagged and audited, never deleted; every repair (`regrade`, flags) is in `audit_log`. 67+- Sale dates come from the source; crawl time is never substituted.
387−- Sale dates come from the source; historical prices use the exchange rate of the sale date; native values are never overwritten. 68+- Historical prices are converted with the exchange rate of the sale date.
388−- Hammer prices are never compared with marketplace prices without the buyer's premium; the fee basis is shown.
389−- Neutral wording for anomalies ("anomalous transaction", "needs review"); RareIndex never accuses anyone of fraud and never claims to authenticate items.
390 69
391 Details: [`docs/adr/0002-data-honesty.md`](docs/adr/0002-data-honesty.md). 70 Details: [`docs/adr/0002-data-honesty.md`](docs/adr/0002-data-honesty.md).
392 71
393−## Roadmap 72+## Deployment
394−
395−Delivered on 2026-09-11: valuation safety (anomaly gate, variant-aware asks, representative variant,
396−grade repair), screener, methodology center, market depth and time-to-sell, sale verification labels,
397−buyer premiums and auction intelligence, brand. Next, in order of leverage:
398−
399−1. Entity resolution for unique objects (auction lots on generic assets), certificate-number and
400− multi-token search, `unaccent` in FTS.
401−2. `fx_rate` / `fx_date` on listings, observations and lots; per-lot bid counts for houses that omit them.
402−3. Index history reconstruction from transactions (`ri value --all --history`), repeat-sales publication.
403−4. Population reports through grader partnerships; cross-grader empirical model.
404−5. Portfolio overhaul (fees, liquidity horizon, confidence aggregation), Scanner multi-item and visual embeddings.
405−6. Cross-border arbitrage with FX, fees and duties; regional price indices.
406−7. Natural-language query engine over the screener; webhooks; API tier enforcement and shared rate-limit store.
407−
408−## Author, hosting and contact
409−
410−RareIndex is designed, built and operated by **Simon-Pierre Boucher** (Québec, Canada).
411−It is hosted on **MacLustr** — [www.maclustr.io](https://www.maclustr.io) — a distributed
412−Apple-silicon compute cluster operated by the same team.
413 73
414−- General contact: [contact@spboucher.ai](mailto:contact@spboucher.ai) 74+Production runs on a MacLustr node (PostgreSQL 17 + pgvector) deployed through the `mld` gateway:
415−- Data corrections and source requests: data@rareindex.io · API and partnerships: hello@rareindex.io · Rights holders: legal@rareindex.io 75+PM2 processes `rareindex-web`, `rareindex-api`, `rareindex-worker` and an ngrok tunnel serving
76+**https://www.rareindex.io**. Secrets live only in the node's `.env` / the `mld` manifest.
416 77
417−## Disclaimers and license 78+## Disclaimers
418 79
419 RareIndex is an information platform. Valuations are estimates derived from observed public data. 80 RareIndex is an information platform. Valuations are estimates derived from observed public data.
420−Listing prices and auction estimates are not confirmed transactions. Past performance does not 81+Listing prices are not confirmed transactions. Past performance does not guarantee future results.
421−guarantee future results. RareIndex does not authenticate items and does not provide investment 82+RareIndex does not authenticate items. Third-party names, marks and content belong to their owners
422−advice. Third-party names, marks and content belong to their owners and are attributed to their 83+and are attributed to their sources; RareIndex respects source terms, robots policies, rate limits,
423−sources; RareIndex respects source terms, robots policies, rate limits, copyright, privacy and 84+copyright, privacy and database rights, and never bypasses authentication, paywalls or CAPTCHAs.
424−database rights. 85+
86+## License
425 87
426−Proprietary — © 2026 Simon-Pierre Boucher / RareIndex. All rights reserved. 88+Proprietary — © 2026 RareIndex. All rights reserved.
added docs/PENDING-SCHEMA-account.md +41 −0
@@ -0,0 +1,41 @@
1+# Pending schema changes — member accounts (agent F1)
2+
3+Applied locally with `drizzle-kit push` on the `rareindex_account` database; **no migration file was
4+generated** (per instructions). Generate one migration after merging (`pnpm db:generate`).
5+
6+## Modified tables (`packages/database/src/schema/users.ts`)
7+
8+- `users`: + `handle text unique`, `avatar_url text`, `bio text`, `mfa_enabled boolean not null default false`,
9+ `totp_secret_enc text` (AES-256-GCM, key derived from `SESSION_SECRET`), `always_ask_code boolean not null default true`,
10+ `password_changed_at timestamptz`, `pending_email text`, `deleted_at timestamptz`, `purge_after timestamptz`.
11+- `sessions`: + `ip text`, `last_seen_at timestamptz`, `revoked_at timestamptz`.
12+- `collections`: + `kind text not null default 'collection'` (collection | wishlist | vault | sold), `budget_usd numeric(18,4)`, `color text`.
13+- `collection_items`: + `tags jsonb not null default '[]'`, `condition text`, `manual_value_usd numeric(18,4)`, `sold_at date`, `sold_price_usd numeric(18,4)`.
14+- `watchlist_items`: + `label text`, `note text`, `target_price_usd numeric(18,4)`, `baseline_usd numeric(18,4)`.
15+- `alerts`: + `name text`, `cooldown_minutes integer not null default 1440`, `trigger_count integer not null default 0`; `channel` now accepts `both`.
16+
17+## New tables (`packages/database/src/schema/account.ts`, exported from `schema/index.ts`)
18+
19+| table | purpose |
20+|---|---|
21+| `auth_codes` | hashed one-time codes: verify_email, mfa_email, password_reset, change_email, new_device (expiry, attempts, consumed_at, payload) |
22+| `recovery_codes` | 10 single-use MFA recovery codes per user (hashed) |
23+| `trusted_devices` | 30-day second-factor skip per browser (hashed token, label, ip, expiry, revoked_at) |
24+| `login_events` | sign-in history (outcome, method, ip, user agent) |
25+| `rate_limits` | fixed-window counters keyed by string (works across PM2 instances) |
26+| `notifications` | in-app inbox (kind, title, body, href, payload, read_at, emailed_at) |
27+| `saved_searches` | named search URLs, notify flag, last run/count |
28+| `price_targets` | per-user buy/sell RIV targets with baseline, hit_at, notified_at |
29+| `uploads` | member files (avatars, item photos) stored under `RI_DATA_DIR/uploads/<userId>/` |
30+| `user_badges` | data-derived badges with evidence (recomputed daily) |
31+
32+## Notes for the merge
33+
34+- `apps/web/src/lib/db.ts` now re-exports tables explicitly from the `schema` namespace (Turbopack could
35+ not enumerate `export *` through the packages' `.js`-suffixed re-exports) and exports `Database` type.
36+- `apps/web/next.config.ts`: `@rareindex/notify` added to `transpilePackages`; root `.env` loaded via
37+ `process.loadEnvFile`; `webpack.resolve.extensionAlias` so `next dev/build --webpack` resolves
38+ `./x.js` → `x.ts` inside workspace packages. **Turbopack cannot resolve those imports** — build with
39+ `next build --webpack` or drop the `.js` suffixes in packages (tsconfig `moduleResolution: bundler`).
40+- `packages/notify/src/index.ts` uses explicit named exports for the same reason.
41+- `workers/package.json`: + `@rareindex/database`, `@rareindex/notify`, `drizzle-orm`; `workers/tsconfig.json` added.
added docs/PENDING-SCHEMA-ai.md +14 −0
@@ -0,0 +1,14 @@
1+# Pending schema changes — agent F2 (ai / scanner / research)
2+
3+New file `packages/database/src/schema/ai.ts`, exported from `schema/index.ts`. Applied locally with
4+`drizzle-kit push` on `rareindex_ai`; **no migration generated** (to be folded into the next
5+`pnpm db:generate` at integration).
6+
7+| Table | Purpose |
8+|---|---|
9+| `scanner_sessions` | Scanner submissions (photo/url/text), model guess + confidence, candidates shown, user pick (feedback loop), thumbnails, cost |
10+| `research_sessions` | AI Research threads keyed by anonymous cookie (`anon_id`) or `user_id` |
11+| `research_messages` | Messages with transparent tool-call trace, usage and cost |
12+| `ai_quotas` | Per-day counters for anonymous AI features (key = ip hash or user id) |
13+
14+No changes to existing tables. AI spend is written to the existing `costs` table (`kind='ai'`).
added docs/PENDING-SCHEMA-images.md +18 −0
@@ -0,0 +1,18 @@
1+# Pending schema changes — images pipeline (agent R, 2026-09-07)
2+
3+Applied locally with `drizzle-kit push`; the parent generates the migration.
4+
5+Table `images` — new columns and indexes:
6+
7+| column | type | note |
8+|---|---|---|
9+| `status` | text not null default `'unchecked'` | `unchecked` · `ok` · `dead` (404/410/not an image) · `blocked` (400/401/403/429) · `error` (network/timeout, retried after 6 h) |
10+| `checked_at` | timestamptz | last validation |
11+| `bytes` | integer | original size |
12+| `content_type` | text | original MIME |
13+| `cache_key` | text | `sha1(url)`; key of `RI_DATA_DIR/images/{orig,w96,w192,w384,w768,w1200}/<k[0:2]>/<k>.*` |
14+| `error` | text | last failure reason (≤200 chars) |
15+
16+Indexes: `images_cache_key_idx (cache_key)`, `images_status_idx (status, checked_at)`.
17+
18+No other tables touched. `workers/lib/queue.ts` gained the job name `images.process`.
added docs/PENDING-SCHEMA.md +11 −0
@@ -0,0 +1,11 @@
1+# Pending schema changes (to be folded into the next generated migration)
2+
3+Applied locally with `drizzle-kit push`; regenerate one migration at integration with `pnpm db:generate`.
4+
5+## packages/database/src/schema/ingestion.ts — `normalized_records`
6+- **added** `seq integer not null default 0` — position of the record within `normalize(raw)` output (one raw row commonly yields a catalog item + several price observations).
7+- **changed** unique index `normalized_records_raw_uq`: `(raw_record_id)` → `(raw_record_id, kind, seq)`.
8+
9+## Notes
10+- No other table/column changes were needed by the pipeline; the `assets_title_trgm` GIN index declared in the schema was missing from the local DB and got created by the push (it is already in migration 0000).
11+- pg-boss creates and manages its own schema `pgboss` at worker start (not part of Drizzle migrations).
modified pnpm-workspace.yaml +9 −0
@@ -3,3 +3,12 @@ packages:
3 - packages/* 3 - packages/*
4 - workers 4 - workers
5 - connectors 5 - connectors
6+allowBuilds:
7+ esbuild: true
8+ unrs-resolver: true
9+onlyBuiltDependencies:
10+ - esbuild
11+ - sharp
12+ - "@tailwindcss/oxide"
13+ - unrs-resolver
14+ - better-sqlite3
6 15
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere