# Entity resolution and deduplication

The same collectible appears on twenty sites under twenty names. Entity resolution (SPEC §7) maps every
normalised record to one canonical `assets` row (the object) and one `asset_variants` row (grade / condition /
completeness / size slice). Deduplication (SPEC §8) prevents the same observation from counting twice.
Code: `workers/entity-resolution/{canonical-key,resolver,writers,certificates}.ts`.

## 1. Deterministic rules first, AI last

Order inside `resolveAsset(attributes)`:

1. **Deterministic identifiers** (`DETERMINISTIC_IDS` in `canonical-key.ts`): `scryfall_id`, `oracle_id`,
   `tcgplayer_id`, `cardmarket_id`, `pokemontcg_id`, `ygo_id`, `psa_spec_id`, `style_code`, `sku`, `upc`, `ean`,
   `jan`, `isbn`, `lego_set_number`, `bricklink_id`, `brickset_id`, `pricecharting_id`, `reference` (watches),
   `discogs_release_id`, `pcgs_number`, `comics_org_id`, `ebay_epid`, `asin`, `vin`, … A hit must also agree on
   the **variant** when both sides state one (card-level ids are shared by Holo / 1st Edition printings).
   Confidence 0.99.
2. **Canonical key** — a family-specific normalised key built from structured attributes
   (`buildCanonicalKey`): cards `category|set|number|name|variant|language`; watches `category|brand|reference|variant`;
   sneakers `style_code` or `brand|model|color|year`; LEGO `set number|variant`; video games
   `platform|name|region|variant`; comics `publisher|series|issue|variant|year`; coins `country|year|name|mint`;
   wine/whisky `brand|name|vintage|size`. Text is normalised (`normalizeForMatch`: lower-case, accents stripped,
   punctuation collapsed, multilingual-safe). Confidence 0.96–0.97.
3. **Fuzzy** — pg_trgm similarity on the composed title within the same category, then `fuzzyCandidateScore`
   applies hard discriminators: different card number / reference / model → 0; year difference > 1 → 0;
   set Jaccard < 0.3 → 0; reference-driven families (watches, sneakers, LEGO, cameras, audio) refuse a match
   when only one side has a reference. Accepted at ≥ 0.72, confidence ≤ 0.9.
4. **Create** a new asset (`deterministicId('asset', canonicalKey)`), enriching sets/brands.

Embeddings (pgvector, `asset_embeddings`) and LLM verification are wired as later stages for low-confidence
candidates when `OPENAI_API_KEY`/`ANTHROPIC_API_KEY` are configured; they never merge below the thresholds above.
**Ambiguous items are never merged automatically**: they stay separate assets (a duplicate is cheaper than a
wrong merge) and surface in `/admin/data-quality` for manual match (`manualMatchAction`).

## 2. Variants

`ensureVariant(assetId, grade, condition, size)` → `variantKey({ grader, grade, qualifier, condition, completeness, size })`.
Grading companies are distinct (PSA 10 ≠ BGS 10 ≠ CGC 10); condition is normalised through the category's
condition scale (`normalizeCondition`).

## 3. Confidence metadata (SPEC §23)

Every normalised record carries `confidence` (extraction + identification); the writer stores
`min(record.confidence, resolution.confidence)` on sales/listings and a `dataQuality` 0–100 score
(completeness × source trust × confidence). `normalized_records.match_method/match_confidence` keep the
decision auditable. Flags (`low_identification_confidence`, `bundle`, `zero_price`) are stored, never deleted.

## 4. Deduplication (SPEC §8)

| what | mechanism |
|---|---|
| same page captured twice | `raw_records` unique `(connector_id, content_hash)`; `contentHash = sha256(kind|externalId|payload)` |
| same sale from two URLs / relist pages | `sales.dedupe_key = sha256(sale|source|externalId∨url|date|price)` unique |
| same listing re-seen | `listings` unique `(source_id, external_id)` → updates + `listing_events` (price_changed, relisted, sold, removed) |
| same guide value | `price_observations.dedupe_key = (obs|source|externalId|priceKind|variant|date)` |
| same lot across result pages | `auction_lots.url` unique, upsert |
| cross-posted listing across marketplaces | `cross_listing_group_id` (same asset + same images/phash + same seller/price window) |
| duplicate images | `images.url` unique + perceptual hash `phash` (image-processing worker) |
| identical certification numbers | `certificates (grader, cert_number)` unique; every appearance is a `certificate_sightings` row |
| renamed listings | matched through identifiers/canonical key, not the title |

## 5. Certification numbers (SPEC §22)

When a sale/listing/lot carries `grade.certificationNumber` + `grade.grader`, `recordCertificate()` normalises the
number (digits/letters, grader-specific sanity), upserts `certificates` (first/last seen, sightings, sources,
last price, verify URL from `data/taxonomy/graders.json`) and inserts a sighting. The hourly `certs.verify` job
calls `cert_lookup` connectors (`meta.config.grader`, exported `certUrl(cert)`) on unverified certs and stores
the grader's answer in `certificates.verification` — the beginning of a provenance graph per slab.

## 6. Adding identifiers

A source that exposes a stable id should emit it in `attributes.identifiers` under a documented key and the key
must be appended to `DETERMINISTIC_IDS` (one place). Keys proposed by the 2026-09-08 wave: `cardmarket_id`
(already), `numista_id`, `gcd_issue_id`/`comics_org_id`, `psa_cert`, `cgc_cert`, `bgs_cert`, `sgc_cert`,
`pcgs_cert`, `ngc_cert`, `pmg_cert`, `ebay_item_id`, `shopify_product_id` (per-shop, not deterministic across
sources — kept in metadata only).