# Entity resolution and deduplication

The same collectible appears on twenty sites under twenty names. Entity resolution (SPEC §7) maps every normalised record to one canonical assets row (the object) and one asset_variants row (grade / condition / completeness / size slice). Deduplication (SPEC §8) prevents the same observation from counting twice. Code: workers/entity-resolution/{canonical-key,resolver,writers,certificates}.ts.

# 1. Deterministic rules first, AI last

Order inside resolveAsset(attributes):

Deterministic identifiers (DETERMINISTIC_IDS in canonical-key.ts): scryfall_id, oracle_id, tcgplayer_id, cardmarket_id, pokemontcg_id, ygo_id, psa_spec_id, style_code, sku, upc, ean, jan, isbn, lego_set_number, bricklink_id, brickset_id, pricecharting_id, reference (watches), discogs_release_id, pcgs_number, comics_org_id, ebay_epid, asin, vin, … A hit must also agree on the variant when both sides state one (card-level ids are shared by Holo / 1st Edition printings). Confidence 0.99.
Canonical key — a family-specific normalised key built from structured attributes (buildCanonicalKey): cards category|set|number|name|variant|language; watches category|brand|reference|variant; sneakers style_code or brand|model|color|year; LEGO set number|variant; video games platform|name|region|variant; comics publisher|series|issue|variant|year; coins country|year|name|mint; wine/whisky brand|name|vintage|size. Text is normalised (normalizeForMatch: lower-case, accents stripped, punctuation collapsed, multilingual-safe). Confidence 0.96–0.97.
Fuzzy — pg_trgm similarity on the composed title within the same category, then fuzzyCandidateScore applies hard discriminators: different card number / reference / model → 0; year difference > 1 → 0; set Jaccard < 0.3 → 0; reference-driven families (watches, sneakers, LEGO, cameras, audio) refuse a match when only one side has a reference. Accepted at ≥ 0.72, confidence ≤ 0.9.
Create a new asset (deterministicId('asset', canonicalKey)), enriching sets/brands.

Embeddings (pgvector, asset_embeddings) and LLM verification are wired as later stages for low-confidence candidates when OPENAI_API_KEY/ANTHROPIC_API_KEY are configured; they never merge below the thresholds above. Ambiguous items are never merged automatically: they stay separate assets (a duplicate is cheaper than a wrong merge) and surface in /admin/data-quality for manual match (manualMatchAction).

# 2. Variants

ensureVariant(assetId, grade, condition, size) → variantKey({ grader, grade, qualifier, condition, completeness, size }). Grading companies are distinct (PSA 10 ≠ BGS 10 ≠ CGC 10); condition is normalised through the category's condition scale (normalizeCondition).

# 3. Confidence metadata (SPEC §23)

Every normalised record carries confidence (extraction + identification); the writer stores min(record.confidence, resolution.confidence) on sales/listings and a dataQuality 0–100 score (completeness × source trust × confidence). normalized_records.match_method/match_confidence keep the decision auditable. Flags (low_identification_confidence, bundle, zero_price) are stored, never deleted.

# 4. Deduplication (SPEC §8)

what	mechanism
same page captured twice	`raw_records` unique `(connector_id, content_hash)`; `contentHash = sha256(kind
same sale from two URLs / relist pages	`sales.dedupe_key = sha256(sale
same listing re-seen	`listings` unique `(source_id, external_id)` → updates + `listing_events` (price_changed, relisted, sold, removed)
same guide value	`price_observations.dedupe_key = (obs
same lot across result pages	`auction_lots.url` unique, upsert
cross-posted listing across marketplaces	`cross_listing_group_id` (same asset + same images/phash + same seller/price window)
duplicate images	`images.url` unique + perceptual hash `phash` (image-processing worker)
identical certification numbers	`certificates (grader, cert_number)` unique; every appearance is a `certificate_sightings` row
renamed listings	matched through identifiers/canonical key, not the title

# 5. Certification numbers (SPEC §22)

When a sale/listing/lot carries grade.certificationNumber + grade.grader, recordCertificate() normalises the number (digits/letters, grader-specific sanity), upserts certificates (first/last seen, sightings, sources, last price, verify URL from data/taxonomy/graders.json) and inserts a sighting. The hourly certs.verify job calls cert_lookup connectors (meta.config.grader, exported certUrl(cert)) on unverified certs and stores the grader's answer in certificates.verification — the beginning of a provenance graph per slab.

# 6. Adding identifiers

A source that exposes a stable id should emit it in attributes.identifiers under a documented key and the key must be appended to DETERMINISTIC_IDS (one place). Keys proposed by the 2026-09-08 wave: cardmarket_id (already), numista_id, gcd_issue_id/comics_org_id, psa_cert, cgc_cert, bgs_cert, sgc_cert, pcgs_cert, ngc_cert, pmg_cert, ebay_item_id, shopify_product_id (per-shop, not deterministic across sources — kept in metadata only).