TypeScript 61.9%
HTML 37.2%
SQL 0.7%
1# RareIndex — Architecture (Phase 1 foundation)23This document is the practical companion to `CLAUDE.md` (the product & engineering mandate). It4describes what exists in the repository, how the pieces fit, and the decisions taken so far.56## Monorepo78pnpm workspaces, TypeScript strict everywhere, ESM.910```11apps/web Next.js 16 (App Router, React 19, Tailwind v4) — public site, account, admin12apps/api Fastify — public REST API /v1 (API keys, tiers, rate limits, exports)13packages/shared ids, money/FX helpers, canonical Zod schemas, stats helpers, logger, env14packages/database Drizzle ORM schema (55 tables), migrations, client, seed15packages/taxonomy extensible category tree, condition scales, graders, index definitions (data/taxonomy)16packages/connectors connector framework: interface, Firecrawl/Scrapfly/HTTP engines, router, quality, health, fixtures17packages/valuation price engine (RIV), outliers, grade premiums, scores18packages/indices index computation (chain-linked, repeat-sales), category snapshots, correlations19packages/search hybrid search (FTS + trigram + optional vectors) and natural-language query parsing20packages/ai ModelProvider abstraction (Anthropic, OpenAI-compatible), extraction, embeddings, vision21connectors/ source connectors grouped by primary engine (api/, feeds/, firecrawl/, scrapfly/) + registry.json22workers/ pipeline processes: crawler, normalizer, entity-resolution, valuation, image-processing, indices23data/taxonomy seed JSON for the taxonomy24data/fixtures captured real payloads per connector for tests25scripts/ operational CLI (`ri`), registry builder, deploy helpers26docs/ this file, ADRs, methodology27```2829## Data flow (never write scraped data into product tables directly — §108)3031```32connector.crawl() → raw_records (immutable, content-hash dedupe, snapshot on disk)33 → connector.normalize(raw) → normalized_records (staging, auditable)34 → entity resolution (identifiers → canonical key → fuzzy → embeddings/LLM)35 → assets / asset_variants (+ sets, brands)36 → dedupe + price validation (flags, audit_log; nothing deleted)37 → sales / listings / auction_lots / price_observations (native currency + USD at historical FX)38 → valuations, asset_stats, price_snapshots39 → indices (index_values), category_snapshots, radar_findings40```4142Queue: pg-boss (PostgreSQL-backed) behind a tiny abstraction so Redis Streams/Kafka can replace it43at scale (§140). Domain events are also appended to `events`.4445## Engines & routing (§198)4647`packages/connectors/src/router.ts`: `api/feed (direct HTTP) → firecrawl → scrapfly → requiresReview`.48Each engine result is scored (§199) by the connector's parser through `FetchOptions.parse/expect`.49Engines are only used on publicly accessible pages, with an honest user agent and rate limiting (§179).5051## Connector contract (§102, §197)5253`connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts}` + `data/fixtures/<id>/*.json`.54`scripts/build-registry.ts` assembles `connectors/registry.json` from the `meta.json` files.55`packages/database/src/seed.ts` mirrors the registry into `sources`/`connectors` tables.5657## Canonical model highlights5859- `assets` = the object independent of grade/condition; `asset_variants` = grade/condition/size slice60 (PSA 10, CIB, deadstock size 10). Sales, listings and valuations reference both.61- `canonical_key` (deterministic normalised key) + `identifiers` (JSONB of external ids) power entity62 resolution; merges are recorded in `merged_from` and `audit_log`.63- Prices: `NUMERIC(18,4)` native + `price_usd` + `fx_rate`/`fx_date`. FX from ECB (frankfurter) stored64 in `fx_rates`; historical conversions always use the rate of the sale date (§136).65- Price-guide numbers (Scryfall, TCG APIs) are `price_observations`, never `sales`.66- Every statistic exposes sample size, confidence and last update (§191); unknown stays NULL (§192).6768## Deployment6970Target: MacLustr node with PostgreSQL 17 + pgvector, deployed through the `mld` gateway (M1M32).71Processes (PM2): `rareindex-web` (Next.js), `rareindex-api` (Fastify), `rareindex-worker` (pipeline),72`rareindex-ngrok` (www.rareindex.io). Secrets live only in the mld manifest / `.env` on the node.7374## Decisions log7576See `docs/adr/`.77