snapshot final BHS128b 2026-09-26 avant retrait de rareindex (disque plein : base 351 Go + images 299 Go) — docs PENDING-SCHEMA, README, CLAUDE.md, pnpm-workspace · 8fd0330 · rareindex

snapshot final BHS128b 2026-09-26 avant retrait de rareindex (disque plein : base 351 Go + images 299 Go) — docs PENDING-SCHEMA, README, CLAUDE.md, pnpm-workspace

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 1 mo ago (Sep 27, 2026) parent d4d0eca Browse files

7 changed files +150 −396

modified CLAUDE.md +1 −2

		`@@ -182,8 +182,7 @@ One canonical asset graph containing every collectible, every important variatio`
182	182	## Repository conventions (added by the implementation)
183	183
184	184	- Package manager `pnpm`; `pnpm typecheck`, `pnpm test`, `pnpm build`. Workers run with `tsx`.
185		−- Secrets only in `.env` (git-ignored) or the PM2 config on the server (`BHS128b:~/apps/pm2.rareindex.config.cjs`, generated from the former `mld` manifest kept in `~/apps/.manifests/`). Never in code or fixtures.
186		−- Production (since 2026-09-14): OVH server BHS128b, outside `mld` — Postgres 17 native, image store `~/apps/rareindex/data/images` (270 GB), PM2 under systemd, route `https://www.rareindex.io → wg1 10.67.0.62:8210`. Release procedure and details in `README.md` → Deployment. The M3U96a copy is a cold copy.
	185	+- Secrets only in `.env` (git-ignored) or the `mld` manifest on the cluster. Never in code or fixtures.
187	186	- Add a connector: `pnpm connector:new <id> --url … [--adapter shopify\|woocommerce]` → `connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts,README.md}` + real fixtures in `data/fixtures/<id>/`, source entry in `data/sources/entries/<group>.json`, host policy in `connectors/domains.d/<group>.json`; then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide: `docs/connectors/ADDING_A_CONNECTOR.md`; catalogue: `docs/connectors/SOURCES.md`.
188	187	- DB changes: edit `packages/database/src/schema/*`, run `pnpm db:generate`, commit the SQL migration.
189	188	- UI: dense, neutral, tabular numbers (`num` utility), `<Unavailable/>` for missing evidence, every number with its sample size / confidence / freshness when it is an estimate.

modified README.md +56 −394

		`@@ -1,426 +1,88 @@`
1		−<p align="center">
2		− <picture>
3		− <source media="(prefers-color-scheme: dark)" srcset="apps/web/public/brand/rareindex-logo-dark.svg">
4		− <img src="apps/web/public/brand/rareindex-logo.svg" alt="RareIndex — The Global Market for Collectibles" width="520">
5		− </picture>
6		−</p>
7		−
8		−<p align="center">
9		− <a href="https://www.rareindex.io"><strong>www.rareindex.io</strong></a> ·
10		− <a href="https://www.rareindex.io/methodology">Methodology</a> ·
11		− <a href="https://www.rareindex.io/api-docs">API</a> ·
12		− <a href="https://www.rareindex.io/screener">Screener</a> ·
13		− <a href="https://www.rareindex.io/auctions">Auctions</a> ·
14		− <a href="mailto:contact@spboucher.ai">contact@spboucher.ai</a>
15		−</p>
16		−
17		−<p align="center">
18		− <img alt="TypeScript" src="https://img.shields.io/badge/TypeScript-strict-3178c6"> <img alt="Next.js 16" src="https://img.shields.io/badge/Next.js-16-000000"> <img alt="Fastify" src="https://img.shields.io/badge/Fastify-%2Fv1-202020"> <img alt="PostgreSQL 17" src="https://img.shields.io/badge/PostgreSQL-17%20%2B%20pgvector-336791"> <img alt="pg-boss" src="https://img.shields.io/badge/workers-pg--boss-4b5563"> <img alt="connectors" src="https://img.shields.io/badge/connectors-212-c9963f"> <img alt="hosted on MacLustr" src="https://img.shields.io/badge/hosted%20on-MacLustr-0b0f19">
19		−</p>
20		−
21	1	# RareIndex
22	2
23		−RareIndex is the market-data and intelligence layer for physical collectibles — the terminal the
24		−collectibles economy never had. It continuously collects public market evidence (marketplaces,
25		−auction houses, dealers, price guides, grading companies, manufacturer catalogs), resolves it into a
26		−canonical asset graph and turns it into research-grade data: a valuation with a confidence level and
27		−a range, comparable sales, live asks, market depth, liquidity and rarity scores, auction intelligence
28		−with buyer-pays costs, price histories and the RARE family of indices.
29		−
30		−A marketplace tells you "this item is for sale for $4,000". RareIndex tells you:
31		−
32		−> *This asset has a RareIndex Valuation of $3,180 (high confidence, 42 comparable sales of the same
33		−> variant across five sources), the market is up 12.4 % over 12 months, typical time-to-sale is 18
34		−> days, liquidity is above the category average, and the current $2,650 ask is 16.7 % below fair
35		−> value — while a $472 slab of a different grade matched to the same card is a data anomaly, not a
36		−> deal.*
37		−
38		−Every number is traceable to its source, carries a sample size, a confidence and a freshness, and
39		−when the evidence is not there the platform says "Not enough data" instead of inventing a figure.
40		−
41		−---
42		−
43		−## Table of contents
44		−
45		−1. [What RareIndex answers](#what-rareindex-answers)
46		−2. [Live platform at a glance](#live-platform-at-a-glance)
47		−3. [Product surface](#product-surface)
48		−4. [Data model — the canonical asset graph](#data-model--the-canonical-asset-graph)
49		−5. [Methodology in brief](#methodology-in-brief)
50		−6. [Architecture](#architecture)
51		−7. [Repository layout](#repository-layout)
52		−8. [Pipeline and workers](#pipeline-and-workers)
53		−9. [Connectors](#connectors)
54		−10. [Public API](#public-api)
55		−11. [Quick start](#quick-start)
56		−12. [Operator CLI](#operator-cli)
57		−13. [Testing and quality gates](#testing-and-quality-gates)
58		−14. [Deployment](#deployment)
59		−15. [Brand](#brand)
60		−16. [Data honesty rules](#data-honesty-rules)
61		−17. [Roadmap](#roadmap)
62		−18. [Author, hosting and contact](#author-hosting-and-contact)
63		−19. [Disclaimers and license](#disclaimers-and-license)
64		−
65		−---
66		−
67		−## What RareIndex answers
68		−
69		−\| Question \| Where \|
70		−\|---\|---\|
71		−\| What is this collectible, exactly (set, number, variant, language, grade, grader)? \| Canonical assets and variants, Scanner (photo / URL / text) \|
72		−\| What is it worth, and how confident are we? \| RIV — RareIndex Valuation with low / fair / high range, confidence label, sample size, method breakdown \|
73		−\| What has it actually sold for? \| Verified sales timeline with source links, buyer-pays price, verification label \|
74		−\| What are sellers asking right now? \| Live listings, ask vs RIV with the anomaly gate, market depth ±5 / 10 / 20 % \|
75		−\| How liquid is it, how fast could it sell? \| Liquidity Score 0–100, days on market, time-to-sale by asking band, fast / typical / patient sell prices \|
76		−\| How rare is it? \| Rarity Score from population reports, production quantity and market frequency \|
77		−\| Is that auction lot cheap once fees are counted? \| Auction intelligence: all-in bid or estimate (hammer + buyer's premium) vs RIV, per-house fee schedules \|
78		−\| Is the market rising or falling? \| RARE global index and 20 subindices, category markets, market map, screener \|
79		−\| Which collectibles match my criteria? \| `/screener` with presets and CSV / JSON export \|
80		−\| What is unusual right now? \| Rare Radar: record sales, price discrepancies, reappearances, auctions below RIV \|
81		−
82		−## Live platform at a glance
83		−
84		−Production figures on 2026-09-11 (PostgreSQL on the MacLustr node):
85		−
86		−\| Measure \| Value \|
87		−\|---\|---\|
88		−\| Canonical assets \| 767 830 \|
89		−\| Sales (transactions) \| 2 228 672 \|
90		−\| Live listings \| 657 107 \|
91		−\| Price observations (guide values, never transactions) \| 14 476 398 \|
92		−\| Auction lots tracked \| 39 502 (≈ 22 100 live or upcoming) \|
93		−\| Sources catalogued / connectors implemented \| 443 / 212 \|
94		−\| Taxonomy nodes \| 136 categories, 96 families \|
95		−\| Indices \| RARE + 20 subindices (published only above their constituent minimum) \|
96		−
97		−## Product surface
98		−
99		−Public pages (Next.js 16 App Router, server-rendered, dense financial-terminal design, dark mode first-class):
100		−
101		−\| Route \| Purpose \|
102		−\|---\|---\|
103		−\| `/` \| Hero + universal search, live system stats, RARE index panel, market snapshot, Trending, Record Sales, Rare Radar, Value Opportunities (gated), Ending auctions, Latest sales, categories \|
104		−\| `/explore`, `/search` \| Filterable asset browser and hybrid search (FTS + trigram + natural-language parsing, ⌘K palette) \|
105		−\| `/asset/[slug]` \| Asset terminal: hero with RIV range and score explainers, price history (sales as dots, RIV overlay with band), valuation breakdown and evidence, market depth & time to sell, sales, listings, auctions, grades (grade curve), population, images, history, comparables, sources \|
106		−\| `/screener` \| Filter builder (category, grader/grade, RIV, confidence, liquidity, rarity, 1M/1Y change, sales, listings, spread, drawdown), presets, sortable columns, CSV / JSON export \|
107		−\| `/markets`, `/markets/[slug]`, `/categories`, `/market-map` \| Category terminals, movers, activity, structure, treemap heatmap \|
108		−\| `/rareindex`, `/rareindex/[ticker]` \| Indices, constituents, correlations, benchmarks (S&P 500, gold, BTC, CPI…) \|
109		−\| `/auctions`, `/auctions/house/[slug]`, `/auctions/calendar` \| Open / ending-soon / below-RIV lots with all-in cost, house pages with buyer-premium schedules and objective stats, global calendar \|
110		−\| `/sales`, `/listings`, `/records`, `/radar`, `/trending`, `/news`, `/grading` \| Market feeds and intelligence \|
111		−\| `/compare`, `/price-history`, `/set/[slug]`, `/brand/[brand]` \| Comparison and entity pages \|
112		−\| `/scanner`, `/research` \| AI identification from photo / URL / text with confidence; AI Research answering only from structured data through inspectable tools \|
113		−\| `/methodology`, `/data`, `/api-docs`, `/about` \| Methodology center (wired to the code's constants), coverage & exports, API reference, company \|
114		−\| Account \| Collections (portfolio with cost basis, gains, allocation, concentration, insurance schedule, CSV import/export), watchlists, alerts (11 types incl. `auction_below_riv`), price targets, saved searches, Deal Radar, My Index, public profiles `/u/[handle]` \|
115		−\| Admin `/admin` \| Pipeline overview, connectors explorer, coverage, data quality, taxonomy proposals, events, audit, costs \|
116		−
117		−## Data model — the canonical asset graph
118		−
119		−Marketplace titles are never assets. Every observation is resolved to a canonical asset
120		−(`assets`: category, family, franchise, brand, series, set, number, year, edition, variant, language,
121		−region, identifiers…) and to an asset variant (`asset_variants`: grader, grade, qualifier,
122		−condition, completeness, size — e.g. `psa\|10`, `raw\|near_mint`, `cgc\|9.8\|black_label`, `psa\|unknown`
123		−for a slab whose grade could not be read). Variants never share a valuation: a PSA 10, a raw copy and
124		−a sealed box are three markets.
125		−
126		−Evidence tables keep the three kinds of numbers strictly apart (§205 of the spec):
127		−
128		−- `sales` — observed transactions: native `price` + `currency`, `price_usd` at the sale-date ECB
129		− rate (`fx_rate`, `fx_date`), `all_in_usd` = buyer-pays price (hammer + estimated buyer premium with a
130		− `fee_basis`), `status` valid / flagged / excluded, `confidence`, `flags`, `dedupe_key`.
131		−- `listings` — asks: lifecycle (`first_seen_at`, `last_seen_at`, `availability`, `listing_events`),
132		− `discount_to_riv` = (ask − RIV) / RIV only when the comparison passed the gates, flags
133		− `riv_anomaly` / `riv_review`.
134		−- `auction_lots` — estimates, current bid, hammer, USD conversions, `all_in_bid_usd`,
135		− `all_in_estimate_low_usd`, `bid_vs_riv`, `estimate_vs_riv`, `assessment_verdict`.
136		−- `price_observations` — price-guide values, always labelled "guide", never mixed with transactions.
137		−
138		−Derived layers: `valuations` (append-only history with method breakdown), `asset_stats` /
139		−`variant_stats` (current read model), `price_snapshots` (daily series), `grade_premiums`,
140		−`population_reports`, `certificates` + `certificate_sightings`, `indices` / `index_values` /
141		−`index_constituents`, `category_snapshots`, `radar_findings`, `correlations`, `benchmarks`.
142		−Schema: `packages/database/src/schema/*`, 8 migrations in `packages/database/migrations/`.
143		−
144		−## Methodology in brief
145		−
146		−Full text: [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) and the live
147		−[Methodology Center](https://www.rareindex.io/methodology), which imports its thresholds from the
148		−code so it cannot drift.
149		−
150		−- RIV (RareIndex Valuation) — ensemble of robust estimators over the variant's valid
151		− transactions (exponentially-weighted mean, medians 5/10/20, trust- and recency-weighted mean,
152		− trimmed mean), median of the core; 365-day window extended to 3 years when thin (confidence
153		− capped); falls back to grade-adjusted comparables, then to guide observations, each with capped
154		− confidence. Inputs are buyer-pays prices (`coalesce(all_in_usd, price_usd)`).
155		−- Confidence — 0.35·sample + 0.30·(1 − dispersion) + 0.20·recency + 0.15·source trust; labels
156		− High ≥ 0.75, Medium ≥ 0.5, Low > 0.2, else Insufficient.
157		−- Range — low / fair / high from p25 / RIV / p75, at least ±5 %.
158		−- Representative variant — the default (raw / base) variant when it has a transaction-based RIV
159		− with ≥ 5 sales and confidence ≥ 0.5, else the qualifying variant with the most sales. Changes,
160		− ATH / ATL and drawdown are measured on that variant's own series.
161		−- Ask vs RIV and the anomaly gate — an ask is compared only with a transaction-based RIV of the
162		− same variant (≥ 5 sales, confidence ≥ 0.5, match ≥ 0.7). Asks below 0.1× or above 10× RIV are
163		− flagged Data/identity anomaly; asks more than 50 % below RIV are held for review; a deal is
164		− between −10 % and −50 %. Auction bids are never asks. Deal Score is multiplicative in depth,
165		− confidence, sample, liquidity and match quality.
166		−- Buyer premiums — per-house marginal schedules (`data/fees/auction-houses.json`, 57 houses,
167		− each rate labelled published / approximate / none, default 22 % when unknown); VAT, duties and
168		− shipping are explicitly excluded.
169		−- Liquidity Score — sales per month, listing depth, source count, days between sales, ask-sold
170		− spread. Rarity Score — population, production quantity, sales and listings per year.
171		− Momentum / Trending with quality gates (canonical identity, ≥ 5 sales, plausible move ≤ ±500 %).
172		−- Outliers — modified z-score > 3.5 on log prices, flagged and audited, never deleted; bundles
173		− excluded.
174		−- Sales verification — heuristic label verified / likely / unverified / excluded from status,
175		− confidence, source type and trust; documented as heuristic.
176		−- Indices — chain-linked trimmed equal-weight log returns, eligibility confidence ≥ 0.4 and ≥ 3
177		− sales in 12 months, single-return clamp ln 2.5, base 1000 on 2024-01-01, minimum 10 constituents
178		− (RARE: 25 across published subindices); unpublished indices are shown as "in development".
179		−- FX — ECB reference rates (frankfurter), sale-date rate for history, native values never
180		− overwritten.
	3	+The global market for collectibles. RareIndex.io is a market-intelligence, search, pricing and
	4	+index platform for collectible assets: trading cards, sports cards, watches, sneakers, LEGO, comics,
	5	+video games, toys, coins, art, wine and, over time, every category the collectibles economy contains.
	6	+It continuously collects public market data from marketplaces, auction houses, dealers, grading
	7	+companies, manufacturer catalogs and archives, resolves it into one canonical asset graph, and turns
	8	+it into research-grade data: valuations with confidence levels, comparable sales, live listings,
	9	+rarity and liquidity scores, price histories, and the RareIndex family of indices (`RARE` and its
	10	+category subindices).
	11	+
	12	+RareIndex is an intelligence platform, not a marketplace aggregator. Every number shown is traceable
	13	+to its source, every estimate carries a sample size and a confidence level, and when the evidence is
	14	+insufficient the platform says "Data unavailable" rather than inventing a figure. The full product
	15	+mandate is in [`docs/SPEC-original.md`](docs/SPEC-original.md); the condensed working version that
	16	+guides implementation is [`CLAUDE.md`](CLAUDE.md).
181	17
182	18	## Architecture
183	19
184		−TypeScript end-to-end in a pnpm monorepo.
185		−
186		−```
187		− ┌──────────────┐ HTTPS (MacLustr Tunnel: WireGuard + Caddy)
188		− browsers ───▶ │ apps/web │ Next.js 16 · App Router · webpack build · Tailwind v4
189		− │ :8210 │──┐ /api/v1/* rewritten to the API
190		− └──────────────┘ │
191		− ┌──────────────┐ │
192		− developers ─▶ │ apps/api │◀─┘ Fastify · /v1 · API keys (hashed) · tiers · rate limits · CSV
193		− │ :8211 │
194		− └──────┬───────┘
195		− │ SQL (drizzle-orm, raw sql for hot paths)
196		− ┌──────▼───────────────────────────────────────────────┐
197		− │ PostgreSQL 17 · pg_trgm · unaccent · pgvector · pgboss│
198		− └──────▲───────────────────────────────────────────────┘
199		− │
200		− ┌──────┴───────┐ pg-boss queues: crawl.run · normalize.batch · resolve.batch
201		− │ workers/ │ valuation.asset/rebuild · auctions.assess · indices.daily
202		− │ (tsx) │ snapshots.daily · radar.scan · health.compute · fx.sync
203		− └──────┬───────┘ benchmarks.sync · images.process · listings.expire · certs.verify · account.jobs
204		− │
205		− ┌──────▼───────┐ 212 connectors · router Direct API → Firecrawl → Scrapfly
206		− │ connectors/ │ per-host policies, circuit breakers, quality scoring, fixtures
207		− └──────────────┘
208		−```
209		−
210		−Packages: `shared` (schemas, money, stats, ids), `database` (Drizzle schema, migrations, seed),
211		−`taxonomy` (136 categories, condition scales, graders, grade parser, variant keys, indices),
212		−`connectors` (SDK, router, adapters Shopify / WooCommerce / sitemap / RSS / PDF / schema.org, health),
213		−`valuation` (RIV, outliers, grade premiums, scores, ask assessment, fees, auction assessment, depth,
214		−liquidation, verification), `indices` (chain-linked, repeat-sales, correlations), `search`
215		−(FTS + trigram + NL parser), `ai` (provider router Anthropic / OpenAI-compatible, cost ledger),
216		−`notify` (Resend e-mail).
217		−
218		−Decisions: [`docs/adr/`](docs/adr/). Layout and data flow: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
219		−
220		−## Repository layout
221		−
222		−```
223		−rareindex/
224		−├── apps/
225		−│ ├── web/ Next.js site (src/app routes, components, lib/queries, brand assets, OG images)
226		−│ └── api/ Fastify public API (/v1), OpenAPI generator, key minting script
227		−├── packages/
228		−│ ├── shared/ database/ taxonomy/ connectors/ valuation/ indices/ search/ ai/ notify/
229		−├── workers/ pg-boss workers + operator CLI (cli.ts): crawler, normalizer, entity-resolution,
230		−│ valuation, auctions, indices, image-processing, account jobs, fees backfill…
231		−├── connectors/ api/ · firecrawl/ · scrapfly/ — one folder per source (meta.json, index.ts, tests)
232		−│ registry.json (generated), domains.json + domains.d/ (host policies)
233		−├── data/
234		−│ ├── taxonomy/ categories.json · conditions.json · graders.json · indices.json
235		−│ ├── fees/ auction-houses.json — buyer-premium schedules
236		−│ ├── sources/ 443 catalogued sources → sources.json → docs/connectors/SOURCES.md
237		−│ └── fixtures/ real captured payloads for every connector (465 JSON, 22 HTML)
238		−├── docs/ ARCHITECTURE · METHODOLOGY · API · openapi.json · SPEC-original · adr/ · connectors/
239		−├── scripts/ build-registry · build-sources · connector-new · gen-openapi · ri
240		−└── CLAUDE.md condensed product mandate and engineering rules
241		−```
242		−
243		−## Pipeline and workers
	20	+TypeScript end-to-end in a pnpm monorepo: Next.js 16 web app, Fastify public API, PostgreSQL 17
	21	+(pg_trgm, unaccent, pgvector) as system of record, pg-boss workers for the ingestion pipeline, and a
	22	+connector framework routing Direct API → Firecrawl → Scrapfly with quality scoring and health
	23	+metrics. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the layout and data flow, and
	24	+[`docs/adr/`](docs/adr/) for the decisions behind it.
244	25
245	26	```
246		−source → raw_records (immutable, content-hashed)
247		− → normalize() connector adapter → NormalizedRecord (sale \| listing \| auction_lot \|
248		− price_observation \| catalog_item \| population_report \| news_item), Zod-validated
249		− → entity resolution identifiers → canonical key → constrained trigram → create; grade re-parsed
250		− from titles (refineGrade), variant ensured (never raw for a slab)
251		− → canonical writes sales (dedupe key, sale-date FX, buyer-pays price), listings (lifecycle events),
252		− auction lots, observations, populations, certificates
253		− → valuation per-variant RIV, outlier flags, ask-vs-RIV with the anomaly gate,
254		− representative variant, asset_stats / variant_stats / price_snapshots
255		− → auctions.assess USD + all-in bid / estimate + bid vs RIV every 20 minutes
256		− → indices / snapshots RARE + subindices, category snapshots, correlations, benchmarks
257		− → radar / alerts record sales, price discrepancies, auctions below RIV, reappearances,
258		− ultra-low populations; member alerts, digests, price targets
	27	+apps/web · apps/api · packages/{shared,database,taxonomy,connectors,valuation,indices,search,ai}
	28	+workers/ · connectors/ (+ registry.json) · data/{taxonomy,fixtures} · docs/ · scripts/
259	29	```
260	30
261		−Operational safeguards: per-run time budgets, one crawl per connector at a time, orphan-run recovery,
262		−per-host concurrency and circuit breakers, challenge-page detection, schema-drift and
263		−result-collapse detectors feeding `connector_health`, audit log for every flag or move.
264		−
265		−## Connectors
266		−
267		−212 implemented connectors (179 `api`, 43 `firecrawl`, 8 `scrapfly` folders; 20 countries,
268		−125 categories) over 443 catalogued sources (201 implemented, 10 gated behind API keys, 108 blocked
269		−by access controls that RareIndex will not circumvent, 108 rejected with a reason). The catalogue is
270		−[`docs/connectors/SOURCES.md`](docs/connectors/SOURCES.md).
271		−
272		−A new source needs four things: `meta.json`, an extraction schema, a pure `normalize()` adapter and
273		−real fixtures with tests. Scaffold with `pnpm connector:new <id> --url … [--adapter shopify\|woocommerce]`,
274		−then `pnpm registry`, `pnpm sources:build`, `pnpm db:seed`. Guide:
275		−[`docs/connectors/ADDING_A_CONNECTOR.md`](docs/connectors/ADDING_A_CONNECTOR.md).
276		−
277		−Routing is Direct API → structured feed → Firecrawl → Scrapfly → manual review, with a quality gate
278		−(title 20 %, price 20 %, status 15 %, date 10 %, images 10 %, identifiers 10 %, description 5 %,
279		−category 5 %, currency 5 %). RareIndex never bypasses logins, paywalls, CAPTCHAs or other access
280		−controls.
281		−
282		−## Public API
283		−
284		−Base URL `https://www.rareindex.io/api/v1` (proxied to Fastify). Bearer keys `ri_live_…` are stored
285		−hashed; anonymous calls get the public tier. Every list supports cursor pagination and `?format=csv`;
286		−responses carry `meta.as_of` and attribution.
287		−
288		−\| Endpoint \| Description \|
289		−\|---\|---\|
290		−\| `GET /v1/assets/search` · `/v1/assets/:id` \| Search and asset detail (id or slug) \|
291		−\| `GET /v1/assets/:id/sales` · `/listings` · `/history` \| Evidence and daily series (native + USD + all-in + fee basis) \|
292		−\| `GET /v1/assets/:id/depth` \| Market depth ±5/10/20 %, days on market, time-to-sale bands, fair buy / sell ladder \|
293		−\| `GET /v1/assets/:id/auctions` · `GET /v1/auctions/lots` \| Lots with estimates, bids, all-in cost, `bid_vs_riv`, verdict; filters `below_riv`, `house`, `ending_within_hours`, `category` \|
294		−\| `GET /v1/indices` · `/v1/indices/:ticker/history` \| RARE and subindices \|
295		−\| `GET /v1/markets` · `/v1/markets/:slug` · `/v1/categories` · `/v1/trending` · `/v1/sales/latest` · `/v1/records` · `/v1/stats` \| Markets and feeds \|
296		−\| `GET /v1/openapi.json` \| OpenAPI 3.1 \|
297		−
298		−Reference: [`docs/API.md`](docs/API.md), [`docs/openapi.json`](docs/openapi.json), live at
299		−[/api-docs](https://www.rareindex.io/api-docs).
	31	+Data flow: `raw_records` (immutable) → `normalize()` → `normalized_records` → entity resolution →
	32	+`assets` / `asset_variants` → dedupe and price validation (flags + audit log, nothing deleted) →
	33	+`sales` / `listings` / `auction_lots` / `price_observations` → valuations and stats → indices.
300	34
301	35	## Quick start
302	36
303		−Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with `pgvector`.
	37	+Requirements: Node ≥ 22, pnpm 11, PostgreSQL 17 with the `pgvector` extension available.
304	38
305	39	```bash
306	40	pnpm install
307	41	createdb rareindex
308		−cp .env.example .env # DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, ANTHROPIC_API_KEY, RESEND_API_KEY, SESSION_SECRET, ADMIN_TOKEN
309		−pnpm db:migrate && pnpm db:seed # extensions, tables, taxonomy, graders, indices, connector registry
	42	+cp .env.example .env # fill DATABASE_URL, FIRECRAWL_API_KEY, SCRAPFLY_API_KEY, AI and Resend keys
	43	+pnpm db:migrate && pnpm db:seed # creates extensions + tables, seeds taxonomy, graders, indices, registry
310	44
311		−pnpm dev:web # http://localhost:3000
312		−pnpm dev:api # http://localhost:8211/v1
313		−pnpm worker # pipeline workers
314		−pnpm ri stats # operator CLI
	45	+pnpm dev:web # Next.js on http://localhost:3000
	46	+pnpm dev:api # public REST API (/v1)
	47	+pnpm worker # pipeline workers (crawl, normalize, resolve, value, index)
	48	+pnpm ri --help # operational CLI (run a connector, recompute valuations, health…)
315	49	```
316	50
317		−Build and verify: `pnpm typecheck`, `pnpm test`, `pnpm build`. Schema change: edit
318		−`packages/database/src/schema/*`, `pnpm db:generate`, commit the migration.
319		−
320		−## Operator CLI
	51	+Useful scripts: `pnpm typecheck`, `pnpm test`, `pnpm build`, `pnpm db:generate` (after editing
	52	+`packages/database/src/schema/*`), `pnpm tsx scripts/build-registry.ts` (rebuilds
	53	+`connectors/registry.json` from per-connector `meta.json` files).
321	54
322		−`pnpm ri <command>` (on the node: `node node_modules/tsx/dist/cli.mjs workers/cli.ts <command>`):
	55	+## Adding a connector
323	56
324		−\| Command \| Purpose \|
325		−\|---\|---\|
326		−\| `connectors` · `crawl <id> [--mode probe\\|incremental\\|backfill]` · `backfill <id…> [--start-only]` \| Sources and crawls \|
327		−\| `normalize [--connector id]` · `resolve [--limit N]` \| Pipeline stages \|
328		−\| `value [--asset id\\|--all] [--history]` · `premiums` · `regrade [--dry-run] [--recheck]` \| Valuations, grade premiums, grade repair (moves mis-varianted rows, audited) \|
329		−\| `fees --backfill [--force]` · `auctions --assess [--all]` \| Buyer-pays prices on sales, auction lot assessment \|
330		−\| `index` · `snapshots` · `radar` · `benchmarks` · `fx [--backfill]` \| Indices, snapshots, Rare Radar, benchmarks, exchange rates \|
331		−\| `certs [--backfill]` · `images` · `expire` · `health [--probe]` · `stats` · `run-all` · `worker` \| Certificates, images, listing expiry, health, stats, full cycle \|
332		−
333		−## Testing and quality gates
334		−
335		−- Unit tests (vitest) in every package and app: valuation (RIV, outliers, premiums, ask gate, fees,
336		− auction assessment, depth, liquidation, verification), taxonomy (grade parser, variant keys), search
337		− parser, indices, shared money/stats, workers, API, web helpers.
338		−- 213 connector test files run every fixture through the schema, price, currency and date checks
339		− (`pnpm connectors:test`).
340		−- `pnpm -r run typecheck` (TypeScript strict), `pnpm --filter @rareindex/web run lint`.
341		−- Data-quality gates are code, not conventions: anomaly band, review threshold, plausible-change
342		− bound, representative-variant rule, numeric overflow guard, `Data/identity anomaly` and
343		− `Needs review` labels in the UI.
344		−
345		−## Deployment
346		−
347		−Since 2026-09-14 production runs on the OVH server BHS128b (`ssh BHS128b`, ubuntu@51.161.112.85, Ubuntu 24.04,
348		−16 threads / 128 GB), outside `mld`: code in `~/apps/rareindex`, PostgreSQL 17 native (db/role `rareindex`/`rareindex`,
349		−pgvector + pg_trgm + unaccent, en_US.UTF-8, ~150 GB), image store `~/apps/rareindex/data/images` (270 GB, ~5 M files:
350		−`orig` + `w96/w192/w384/w768/w1200`, two-level hex sharding), PM2 under systemd (`pm2-ubuntu`) with
351		−`~/apps/pm2.rareindex.config.cjs` generated from the mld manifest by `~/apps/.manifests/gen-pm2.py` (secrets, 0600).
352		−PM2 processes: `rareindex-web` (8210), `rareindex-api` (8211), `rareindex-worker` (cron 04:30 + autorestart). Published through the
353		−MacLustr Tunnel: BHS64 Caddy → wg1 `10.67.0.62:8210` (`mlt add www.rareindex.io BHS128b:8210`).
354		−Release on BHS128b: `rsync -az --exclude node_modules --exclude .git --exclude 'apps/web/.next' --exclude data --exclude logs . BHS128b:apps/rareindex/`
355		−then `ssh BHS128b 'cd apps/rareindex && pnpm install --frozen-lockfile && DATABASE_URL=postgres://rareindex:rareindex@127.0.0.1:5432/rareindex pnpm db:migrate && DATABASE_URL=postgres://rareindex:rareindex@127.0.0.1:5432/rareindex NEXT_PUBLIC_SITE_URL=https://www.rareindex.io API_URL=http://127.0.0.1:8211 pnpm --filter @rareindex/web build && pm2 restart rareindex-api rareindex-web rareindex-worker && pm2 save'`.
356		−pnpm ≥ 10 needs `onlyBuiltDependencies` (esbuild, sharp, @tailwindcss/oxide, unrs-resolver) in `pnpm-workspace.yaml` on a fresh install.
357		−Former home (kept as a cold copy, retired from `mld` with `--keep-dir`): MacLustr node M3U96a (Mac Studio, PostgreSQL 17 + pgvector),
358		−orchestrated by the `mld` gateway; the migration moved a 179 GB database (parallel `pg_dump -Fd -j 4`, 14 GB compressed, restored with
359		−`pg_restore -j 8` in 21 min) and the 270 GB image store (rsync streams capped at 20 MB/s each so the Mac's 1 Gbit/s link stayed usable).
360		−
361		−Release procedure: commit → `git archive HEAD` synced to the node → `pnpm db:migrate` →
362		−`pnpm --filter @rareindex/web run build` (detached, ~15 min: 146 static pages) → `pm2 restart`.
363		−Never delete `.next` under a running server; `/data` is rendered on demand with a one-hour cache
364		−because its aggregates exceed the build-time prerender budget.
365		−
366		−## Brand
367		−
368		−<p align="center">
369		− <img src="apps/web/public/brand/rareindex-mark.svg" alt="RareIndex mark" width="96">
370		−</p>
371		−
372		−The mark is a cut stone — crown, girdle, pavilion — whose facets are drawn by an ascending index
373		−line: rarity and market data in one shape. Ink `#0B0F19`, gold `#F1D08A → #C9963F`, mint
374		−`#2FB673 → #5FE3A1`. Wordmark Rare<span>Index</span> with "Index" in gold. Source files in
375		−[`apps/web/public/brand/`](apps/web/public/brand/) (mark, light and dark lockups, README); favicons,
376		−PWA icons, `favicon.ico` and the Apple touch icon are generated from the mark by
377		−`apps/web/scripts/brand-assets.mjs`; the site-wide share image is rendered by
378		−`apps/web/src/app/opengraph-image.tsx`.
	57	+A new source needs only four things: connector metadata (`meta.json`), an extraction schema, a pure
	58	+`normalize()` adapter and real captured fixtures with tests. The rest of the pipeline works
	59	+unchanged. Conventions and the module contract are in [`connectors/README.md`](connectors/README.md).
379	60
380	61	## Data honesty rules
381	62
382		−- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions, bids or listings.
383		−- Observed data, model estimates, user data and external asks are always distinguished (§205).
384		−- Price-guide values are observations, never transactions; a listing price is never a market value.
385		−- Valuations publish only with sample size, confidence, range and timestamp; missing evidence reads "Not enough data".
386		−- Outliers are flagged and audited, never deleted; every repair (`regrade`, flags) is in `audit_log`.
387		−- Sale dates come from the source; historical prices use the exchange rate of the sale date; native values are never overwritten.
388		−- Hammer prices are never compared with marketplace prices without the buyer's premium; the fee basis is shown.
389		−- Neutral wording for anomalies ("anomalous transaction", "needs review"); RareIndex never accuses anyone of fraud and never claims to authenticate items.
	63	+- No fabricated data, ever: no invented sales, prices, populations, market caps, auctions or listings.
	64	+- Price-guide values are stored as observations, never as transactions.
	65	+- Valuations are published only with sample size, confidence, low/high range and timestamp.
	66	+- Outliers are flagged and audited, never silently deleted.
	67	+- Sale dates come from the source; crawl time is never substituted.
	68	+- Historical prices are converted with the exchange rate of the sale date.
390	69
391	70	Details: [`docs/adr/0002-data-honesty.md`](docs/adr/0002-data-honesty.md).
392	71
393		−## Roadmap
394		−
395		−Delivered on 2026-09-11: valuation safety (anomaly gate, variant-aware asks, representative variant,
396		−grade repair), screener, methodology center, market depth and time-to-sell, sale verification labels,
397		−buyer premiums and auction intelligence, brand. Next, in order of leverage:
398		−
399		−1. Entity resolution for unique objects (auction lots on generic assets), certificate-number and
400		− multi-token search, `unaccent` in FTS.
401		−2. `fx_rate` / `fx_date` on listings, observations and lots; per-lot bid counts for houses that omit them.
402		−3. Index history reconstruction from transactions (`ri value --all --history`), repeat-sales publication.
403		−4. Population reports through grader partnerships; cross-grader empirical model.
404		−5. Portfolio overhaul (fees, liquidity horizon, confidence aggregation), Scanner multi-item and visual embeddings.
405		−6. Cross-border arbitrage with FX, fees and duties; regional price indices.
406		−7. Natural-language query engine over the screener; webhooks; API tier enforcement and shared rate-limit store.
407		−
408		−## Author, hosting and contact
409		−
410		−RareIndex is designed, built and operated by Simon-Pierre Boucher (Québec, Canada).
411		−It is hosted on MacLustr — [www.maclustr.io](https://www.maclustr.io) — a distributed
412		−Apple-silicon compute cluster operated by the same team.
	72	+## Deployment
413	73
414		−- General contact: [contact@spboucher.ai](mailto:contact@spboucher.ai)
415		−- Data corrections and source requests: data@rareindex.io · API and partnerships: hello@rareindex.io · Rights holders: legal@rareindex.io
	74	+Production runs on a MacLustr node (PostgreSQL 17 + pgvector) deployed through the `mld` gateway:
	75	+PM2 processes `rareindex-web`, `rareindex-api`, `rareindex-worker` and an ngrok tunnel serving
	76	+https://www.rareindex.io. Secrets live only in the node's `.env` / the `mld` manifest.
416	77
417		−## Disclaimers and license
	78	+## Disclaimers
418	79
419	80	RareIndex is an information platform. Valuations are estimates derived from observed public data.
420		−Listing prices and auction estimates are not confirmed transactions. Past performance does not
421		−guarantee future results. RareIndex does not authenticate items and does not provide investment
422		−advice. Third-party names, marks and content belong to their owners and are attributed to their
423		−sources; RareIndex respects source terms, robots policies, rate limits, copyright, privacy and
424		−database rights.
	81	+Listing prices are not confirmed transactions. Past performance does not guarantee future results.
	82	+RareIndex does not authenticate items. Third-party names, marks and content belong to their owners
	83	+and are attributed to their sources; RareIndex respects source terms, robots policies, rate limits,
	84	+copyright, privacy and database rights, and never bypasses authentication, paywalls or CAPTCHAs.
	85	+
	86	+## License
425	87
426		−Proprietary — © 2026 Simon-Pierre Boucher / RareIndex. All rights reserved.
	88	+Proprietary — © 2026 RareIndex. All rights reserved.

added docs/PENDING-SCHEMA-account.md +41 −0

		`@@ -0,0 +1,41 @@`
	1	+# Pending schema changes — member accounts (agent F1)
	2	+
	3	+Applied locally with `drizzle-kit push` on the `rareindex_account` database; **no migration file was
	4	+generated** (per instructions). Generate one migration after merging (`pnpm db:generate`).
	5	+
	6	+## Modified tables (`packages/database/src/schema/users.ts`)
	7	+
	8	+- `users`: + `handle text unique`, `avatar_url text`, `bio text`, `mfa_enabled boolean not null default false`,
	9	+ `totp_secret_enc text` (AES-256-GCM, key derived from `SESSION_SECRET`), `always_ask_code boolean not null default true`,
	10	+ `password_changed_at timestamptz`, `pending_email text`, `deleted_at timestamptz`, `purge_after timestamptz`.
	11	+- `sessions`: + `ip text`, `last_seen_at timestamptz`, `revoked_at timestamptz`.
	12	+- `collections`: + `kind text not null default 'collection'` (collection \| wishlist \| vault \| sold), `budget_usd numeric(18,4)`, `color text`.
	13	+- `collection_items`: + `tags jsonb not null default '[]'`, `condition text`, `manual_value_usd numeric(18,4)`, `sold_at date`, `sold_price_usd numeric(18,4)`.
	14	+- `watchlist_items`: + `label text`, `note text`, `target_price_usd numeric(18,4)`, `baseline_usd numeric(18,4)`.
	15	+- `alerts`: + `name text`, `cooldown_minutes integer not null default 1440`, `trigger_count integer not null default 0`; `channel` now accepts `both`.
	16	+
	17	+## New tables (`packages/database/src/schema/account.ts`, exported from `schema/index.ts`)
	18	+
	19	+\| table \| purpose \|
	20	+\|---\|---\|
	21	+\| `auth_codes` \| hashed one-time codes: verify_email, mfa_email, password_reset, change_email, new_device (expiry, attempts, consumed_at, payload) \|
	22	+\| `recovery_codes` \| 10 single-use MFA recovery codes per user (hashed) \|
	23	+\| `trusted_devices` \| 30-day second-factor skip per browser (hashed token, label, ip, expiry, revoked_at) \|
	24	+\| `login_events` \| sign-in history (outcome, method, ip, user agent) \|
	25	+\| `rate_limits` \| fixed-window counters keyed by string (works across PM2 instances) \|
	26	+\| `notifications` \| in-app inbox (kind, title, body, href, payload, read_at, emailed_at) \|
	27	+\| `saved_searches` \| named search URLs, notify flag, last run/count \|
	28	+\| `price_targets` \| per-user buy/sell RIV targets with baseline, hit_at, notified_at \|
	29	+\| `uploads` \| member files (avatars, item photos) stored under `RI_DATA_DIR/uploads/<userId>/` \|
	30	+\| `user_badges` \| data-derived badges with evidence (recomputed daily) \|
	31	+
	32	+## Notes for the merge
	33	+
	34	+- `apps/web/src/lib/db.ts` now re-exports tables explicitly from the `schema` namespace (Turbopack could
	35	+ not enumerate `export *` through the packages' `.js`-suffixed re-exports) and exports `Database` type.
	36	+- `apps/web/next.config.ts`: `@rareindex/notify` added to `transpilePackages`; root `.env` loaded via
	37	+ `process.loadEnvFile`; `webpack.resolve.extensionAlias` so `next dev/build --webpack` resolves
	38	+ `./x.js` → `x.ts` inside workspace packages. Turbopack cannot resolve those imports — build with
	39	+ `next build --webpack` or drop the `.js` suffixes in packages (tsconfig `moduleResolution: bundler`).
	40	+- `packages/notify/src/index.ts` uses explicit named exports for the same reason.
	41	+- `workers/package.json`: + `@rareindex/database`, `@rareindex/notify`, `drizzle-orm`; `workers/tsconfig.json` added.

added docs/PENDING-SCHEMA-ai.md +14 −0

		`@@ -0,0 +1,14 @@`
	1	+# Pending schema changes — agent F2 (ai / scanner / research)
	2	+
	3	+New file `packages/database/src/schema/ai.ts`, exported from `schema/index.ts`. Applied locally with
	4	+`drizzle-kit push` on `rareindex_ai`; no migration generated (to be folded into the next
	5	+`pnpm db:generate` at integration).
	6	+
	7	+\| Table \| Purpose \|
	8	+\|---\|---\|
	9	+\| `scanner_sessions` \| Scanner submissions (photo/url/text), model guess + confidence, candidates shown, user pick (feedback loop), thumbnails, cost \|
	10	+\| `research_sessions` \| AI Research threads keyed by anonymous cookie (`anon_id`) or `user_id` \|
	11	+\| `research_messages` \| Messages with transparent tool-call trace, usage and cost \|
	12	+\| `ai_quotas` \| Per-day counters for anonymous AI features (key = ip hash or user id) \|
	13	+
	14	+No changes to existing tables. AI spend is written to the existing `costs` table (`kind='ai'`).

added docs/PENDING-SCHEMA-images.md +18 −0

		`@@ -0,0 +1,18 @@`
	1	+# Pending schema changes — images pipeline (agent R, 2026-09-07)
	2	+
	3	+Applied locally with `drizzle-kit push`; the parent generates the migration.
	4	+
	5	+Table `images` — new columns and indexes:
	6	+
	7	+\| column \| type \| note \|
	8	+\|---\|---\|---\|
	9	+\| `status` \| text not null default `'unchecked'` \| `unchecked` · `ok` · `dead` (404/410/not an image) · `blocked` (400/401/403/429) · `error` (network/timeout, retried after 6 h) \|
	10	+\| `checked_at` \| timestamptz \| last validation \|
	11	+\| `bytes` \| integer \| original size \|
	12	+\| `content_type` \| text \| original MIME \|
	13	+\| `cache_key` \| text \| `sha1(url)`; key of `RI_DATA_DIR/images/{orig,w96,w192,w384,w768,w1200}/<k[0:2]>/<k>.*` \|
	14	+\| `error` \| text \| last failure reason (≤200 chars) \|
	15	+
	16	+Indexes: `images_cache_key_idx (cache_key)`, `images_status_idx (status, checked_at)`.
	17	+
	18	+No other tables touched. `workers/lib/queue.ts` gained the job name `images.process`.

added docs/PENDING-SCHEMA.md +11 −0

		`@@ -0,0 +1,11 @@`
	1	+# Pending schema changes (to be folded into the next generated migration)
	2	+
	3	+Applied locally with `drizzle-kit push`; regenerate one migration at integration with `pnpm db:generate`.
	4	+
	5	+## packages/database/src/schema/ingestion.ts — `normalized_records`
	6	+- added `seq integer not null default 0` — position of the record within `normalize(raw)` output (one raw row commonly yields a catalog item + several price observations).
	7	+- changed unique index `normalized_records_raw_uq`: `(raw_record_id)` → `(raw_record_id, kind, seq)`.
	8	+
	9	+## Notes
	10	+- No other table/column changes were needed by the pipeline; the `assets_title_trgm` GIN index declared in the schema was missing from the local DB and got created by the push (it is already in migration 0000).
	11	+- pg-boss creates and manages its own schema `pgboss` at worker start (not part of Drizzle migrations).

modified pnpm-workspace.yaml +9 −0

		`@@ -3,3 +3,12 @@ packages:`
3	3	- packages/*
4	4	- workers
5	5	- connectors
	6	+allowBuilds:
	7	+ esbuild: true
	8	+ unrs-resolver: true
	9	+onlyBuiltDependencies:
	10	+ - esbuild
	11	+ - sharp
	12	+ - "@tailwindcss/oxide"
	13	+ - unrs-resolver
	14	+ - better-sqlite3
6	15

spb/rareindex

snapshot final BHS128b 2026-09-26 avant retrait de rareindex (disque plein : base 351 Go + images 299 Go) — docs PENDING-SCHEMA, README, CLAUDE.md, pnpm-workspace

Theme