# RareIndex — Architecture (Phase 1 foundation)

This document is the practical companion to CLAUDE.md (the product & engineering mandate). It describes what exists in the repository, how the pieces fit, and the decisions taken so far.

# Monorepo

pnpm workspaces, TypeScript strict everywhere, ESM.

text

apps/web            Next.js 16 (App Router, React 19, Tailwind v4) — public site, account, admin
apps/api            Fastify — public REST API /v1 (API keys, tiers, rate limits, exports)
packages/shared     ids, money/FX helpers, canonical Zod schemas, stats helpers, logger, env
packages/database   Drizzle ORM schema (55 tables), migrations, client, seed
packages/taxonomy   extensible category tree, condition scales, graders, index definitions (data/taxonomy)
packages/connectors connector framework: interface, Firecrawl/Scrapfly/HTTP engines, router, quality, health, fixtures
packages/valuation  price engine (RIV), outliers, grade premiums, scores
packages/indices    index computation (chain-linked, repeat-sales), category snapshots, correlations
packages/search     hybrid search (FTS + trigram + optional vectors) and natural-language query parsing
packages/ai         ModelProvider abstraction (Anthropic, OpenAI-compatible), extraction, embeddings, vision
connectors/         source connectors grouped by primary engine (api/, feeds/, firecrawl/, scrapfly/) + registry.json
workers/            pipeline processes: crawler, normalizer, entity-resolution, valuation, image-processing, indices
data/taxonomy       seed JSON for the taxonomy
data/fixtures       captured real payloads per connector for tests
scripts/            operational CLI (`ri`), registry builder, deploy helpers
docs/               this file, ADRs, methodology

# Data flow (never write scraped data into product tables directly — §108)

text

connector.crawl()  → raw_records (immutable, content-hash dedupe, snapshot on disk)
                   → connector.normalize(raw) → normalized_records (staging, auditable)
                   → entity resolution (identifiers → canonical key → fuzzy → embeddings/LLM)
                   → assets / asset_variants (+ sets, brands)
                   → dedupe + price validation (flags, audit_log; nothing deleted)
                   → sales / listings / auction_lots / price_observations (native currency + USD at historical FX)
                   → valuations, asset_stats, price_snapshots
                   → indices (index_values), category_snapshots, radar_findings

Queue: pg-boss (PostgreSQL-backed) behind a tiny abstraction so Redis Streams/Kafka can replace it at scale (§140). Domain events are also appended to events.

# Engines & routing (§198)

packages/connectors/src/router.ts: api/feed (direct HTTP) → firecrawl → scrapfly → requiresReview. Each engine result is scored (§199) by the connector's parser through FetchOptions.parse/expect. Engines are only used on publicly accessible pages, with an honest user agent and rate limiting (§179).

# Connector contract (§102, §197)

connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts} + data/fixtures/<id>/*.json. scripts/build-registry.ts assembles connectors/registry.json from the meta.json files. packages/database/src/seed.ts mirrors the registry into sources/connectors tables.

# Canonical model highlights

assets = the object independent of grade/condition; asset_variants = grade/condition/size slice (PSA 10, CIB, deadstock size 10). Sales, listings and valuations reference both.
canonical_key (deterministic normalised key) + identifiers (JSONB of external ids) power entity resolution; merges are recorded in merged_from and audit_log.
Prices: NUMERIC(18,4) native + price_usd + fx_rate/fx_date. FX from ECB (frankfurter) stored in fx_rates; historical conversions always use the rate of the sale date (§136).
Price-guide numbers (Scryfall, TCG APIs) are price_observations, never sales.
Every statistic exposes sample size, confidence and last update (§191); unknown stays NULL (§192).

# Deployment

Target: MacLustr node with PostgreSQL 17 + pgvector, deployed through the mld gateway (M1M32). Processes (PM2): rareindex-web (Next.js), rareindex-api (Fastify), rareindex-worker (pipeline), rareindex-ngrok (www.rareindex.io). Secrets live only in the mld manifest / .env on the node.

# Decisions log

See docs/adr/.