# RareIndex — Architecture (Phase 1 foundation)

This document is the practical companion to `CLAUDE.md` (the product & engineering mandate). It
describes what exists in the repository, how the pieces fit, and the decisions taken so far.

## Monorepo

pnpm workspaces, TypeScript strict everywhere, ESM.

```
apps/web            Next.js 16 (App Router, React 19, Tailwind v4) — public site, account, admin
apps/api            Fastify — public REST API /v1 (API keys, tiers, rate limits, exports)
packages/shared     ids, money/FX helpers, canonical Zod schemas, stats helpers, logger, env
packages/database   Drizzle ORM schema (55 tables), migrations, client, seed
packages/taxonomy   extensible category tree, condition scales, graders, index definitions (data/taxonomy)
packages/connectors connector framework: interface, Firecrawl/Scrapfly/HTTP engines, router, quality, health, fixtures
packages/valuation  price engine (RIV), outliers, grade premiums, scores
packages/indices    index computation (chain-linked, repeat-sales), category snapshots, correlations
packages/search     hybrid search (FTS + trigram + optional vectors) and natural-language query parsing
packages/ai         ModelProvider abstraction (Anthropic, OpenAI-compatible), extraction, embeddings, vision
connectors/         source connectors grouped by primary engine (api/, feeds/, firecrawl/, scrapfly/) + registry.json
workers/            pipeline processes: crawler, normalizer, entity-resolution, valuation, image-processing, indices
data/taxonomy       seed JSON for the taxonomy
data/fixtures       captured real payloads per connector for tests
scripts/            operational CLI (`ri`), registry builder, deploy helpers
docs/               this file, ADRs, methodology
```

## Data flow (never write scraped data into product tables directly — §108)

```
connector.crawl()  → raw_records (immutable, content-hash dedupe, snapshot on disk)
                   → connector.normalize(raw) → normalized_records (staging, auditable)
                   → entity resolution (identifiers → canonical key → fuzzy → embeddings/LLM)
                   → assets / asset_variants (+ sets, brands)
                   → dedupe + price validation (flags, audit_log; nothing deleted)
                   → sales / listings / auction_lots / price_observations (native currency + USD at historical FX)
                   → valuations, asset_stats, price_snapshots
                   → indices (index_values), category_snapshots, radar_findings
```

Queue: pg-boss (PostgreSQL-backed) behind a tiny abstraction so Redis Streams/Kafka can replace it
at scale (§140). Domain events are also appended to `events`.

## Engines & routing (§198)

`packages/connectors/src/router.ts`: `api/feed (direct HTTP) → firecrawl → scrapfly → requiresReview`.
Each engine result is scored (§199) by the connector's parser through `FetchOptions.parse/expect`.
Engines are only used on publicly accessible pages, with an honest user agent and rate limiting (§179).

## Connector contract (§102, §197)

`connectors/<engine>/<id>/{meta.json,index.ts,index.test.ts}` + `data/fixtures/<id>/*.json`.
`scripts/build-registry.ts` assembles `connectors/registry.json` from the `meta.json` files.
`packages/database/src/seed.ts` mirrors the registry into `sources`/`connectors` tables.

## Canonical model highlights

- `assets` = the object independent of grade/condition; `asset_variants` = grade/condition/size slice
  (PSA 10, CIB, deadstock size 10). Sales, listings and valuations reference both.
- `canonical_key` (deterministic normalised key) + `identifiers` (JSONB of external ids) power entity
  resolution; merges are recorded in `merged_from` and `audit_log`.
- Prices: `NUMERIC(18,4)` native + `price_usd` + `fx_rate`/`fx_date`. FX from ECB (frankfurter) stored
  in `fx_rates`; historical conversions always use the rate of the sale date (§136).
- Price-guide numbers (Scryfall, TCG APIs) are `price_observations`, never `sales`.
- Every statistic exposes sample size, confidence and last update (§191); unknown stays NULL (§192).

## Deployment

Target: MacLustr node with PostgreSQL 17 + pgvector, deployed through the `mld` gateway (M1M32).
Processes (PM2): `rareindex-web` (Next.js), `rareindex-api` (Fastify), `rareindex-worker` (pipeline),
`rareindex-ngrok` (www.rareindex.io). Secrets live only in the mld manifest / `.env` on the node.

## Decisions log

See `docs/adr/`.