TypeScript 61.9%
HTML 37.2%
SQL 0.7%
1# Entity resolution and deduplication23The same collectible appears on twenty sites under twenty names. Entity resolution (SPEC §7) maps every4normalised record to one canonical `assets` row (the object) and one `asset_variants` row (grade / condition /5completeness / size slice). Deduplication (SPEC §8) prevents the same observation from counting twice.6Code: `workers/entity-resolution/{canonical-key,resolver,writers,certificates}.ts`.78## 1. Deterministic rules first, AI last910Order inside `resolveAsset(attributes)`:11121. **Deterministic identifiers** (`DETERMINISTIC_IDS` in `canonical-key.ts`): `scryfall_id`, `oracle_id`,13 `tcgplayer_id`, `cardmarket_id`, `pokemontcg_id`, `ygo_id`, `psa_spec_id`, `style_code`, `sku`, `upc`, `ean`,14 `jan`, `isbn`, `lego_set_number`, `bricklink_id`, `brickset_id`, `pricecharting_id`, `reference` (watches),15 `discogs_release_id`, `pcgs_number`, `comics_org_id`, `ebay_epid`, `asin`, `vin`, … A hit must also agree on16 the **variant** when both sides state one (card-level ids are shared by Holo / 1st Edition printings).17 Confidence 0.99.182. **Canonical key** — a family-specific normalised key built from structured attributes19 (`buildCanonicalKey`): cards `category|set|number|name|variant|language`; watches `category|brand|reference|variant`;20 sneakers `style_code` or `brand|model|color|year`; LEGO `set number|variant`; video games21 `platform|name|region|variant`; comics `publisher|series|issue|variant|year`; coins `country|year|name|mint`;22 wine/whisky `brand|name|vintage|size`. Text is normalised (`normalizeForMatch`: lower-case, accents stripped,23 punctuation collapsed, multilingual-safe). Confidence 0.96–0.97.243. **Fuzzy** — pg_trgm similarity on the composed title within the same category, then `fuzzyCandidateScore`25 applies hard discriminators: different card number / reference / model → 0; year difference > 1 → 0;26 set Jaccard < 0.3 → 0; reference-driven families (watches, sneakers, LEGO, cameras, audio) refuse a match27 when only one side has a reference. Accepted at ≥ 0.72, confidence ≤ 0.9.284. **Create** a new asset (`deterministicId('asset', canonicalKey)`), enriching sets/brands.2930Embeddings (pgvector, `asset_embeddings`) and LLM verification are wired as later stages for low-confidence31candidates when `OPENAI_API_KEY`/`ANTHROPIC_API_KEY` are configured; they never merge below the thresholds above.32**Ambiguous items are never merged automatically**: they stay separate assets (a duplicate is cheaper than a33wrong merge) and surface in `/admin/data-quality` for manual match (`manualMatchAction`).3435## 2. Variants3637`ensureVariant(assetId, grade, condition, size)` → `variantKey({ grader, grade, qualifier, condition, completeness, size })`.38Grading companies are distinct (PSA 10 ≠ BGS 10 ≠ CGC 10); condition is normalised through the category's39condition scale (`normalizeCondition`).4041## 3. Confidence metadata (SPEC §23)4243Every normalised record carries `confidence` (extraction + identification); the writer stores44`min(record.confidence, resolution.confidence)` on sales/listings and a `dataQuality` 0–100 score45(completeness × source trust × confidence). `normalized_records.match_method/match_confidence` keep the46decision auditable. Flags (`low_identification_confidence`, `bundle`, `zero_price`) are stored, never deleted.4748## 4. Deduplication (SPEC §8)4950| what | mechanism |51|---|---|52| same page captured twice | `raw_records` unique `(connector_id, content_hash)`; `contentHash = sha256(kind|externalId|payload)` |53| same sale from two URLs / relist pages | `sales.dedupe_key = sha256(sale|source|externalId∨url|date|price)` unique |54| same listing re-seen | `listings` unique `(source_id, external_id)` → updates + `listing_events` (price_changed, relisted, sold, removed) |55| same guide value | `price_observations.dedupe_key = (obs|source|externalId|priceKind|variant|date)` |56| same lot across result pages | `auction_lots.url` unique, upsert |57| cross-posted listing across marketplaces | `cross_listing_group_id` (same asset + same images/phash + same seller/price window) |58| duplicate images | `images.url` unique + perceptual hash `phash` (image-processing worker) |59| identical certification numbers | `certificates (grader, cert_number)` unique; every appearance is a `certificate_sightings` row |60| renamed listings | matched through identifiers/canonical key, not the title |6162## 5. Certification numbers (SPEC §22)6364When a sale/listing/lot carries `grade.certificationNumber` + `grade.grader`, `recordCertificate()` normalises the65number (digits/letters, grader-specific sanity), upserts `certificates` (first/last seen, sightings, sources,66last price, verify URL from `data/taxonomy/graders.json`) and inserts a sighting. The hourly `certs.verify` job67calls `cert_lookup` connectors (`meta.config.grader`, exported `certUrl(cert)`) on unverified certs and stores68the grader's answer in `certificates.verification` — the beginning of a provenance graph per slab.6970## 6. Adding identifiers7172A source that exposes a stable id should emit it in `attributes.identifiers` under a documented key and the key73must be appended to `DETERMINISTIC_IDS` (one place). Keys proposed by the 2026-09-08 wave: `cardmarket_id`74(already), `numista_id`, `gcd_issue_id`/`comics_org_id`, `psa_cert`, `cgc_cert`, `bgs_cert`, `sgc_cert`,75`pcgs_cert`, `ngc_cert`, `pmg_cert`, `ebay_item_id`, `shopify_product_id` (per-shop, not deterministic across76sources — kept in metadata only).77