spb/datacenterindex

1# Entity reconciliation23> **2026-09-12 — claim-first layer.** Numeric fields (capacity, investment) now flow through the claim store described in [CLAIMS.md](CLAIMS.md): scope + semantics + evidence sentence + field-level authority, sanity engine, then the merge policy below. External-id folding is allowlist-only (`IDENTIFYING_EXTERNAL_ID_KEYS`), candidates are relevance-ordered (same operator → same name → nearest), and `rule:campus-vs-building` now links the building to its campus (`parent_facility_id`, `record_scope`) instead of opening a duplicate review. `provenance.is_winner` marks the observation behind each displayed value; `run_id` ties every write to a connector run (rollback).45How a normalized record coming out of a connector becomes (or updates) a canonical facility, operator, project,6cloud region or IXP — and how conflicting values are merged. Implementation: `apps/worker/src/ingest/`7(`match.ts` holds the pure scoring/merge logic, `facilities.ts` wires it to Postgres).89## Guarantees1011- **Nothing is invented and nothing is lost.** Every incoming value is written to `provenance` even when it does12  not win the merge; ambiguous matches are kept as new records *and* queued for review, never discarded.13- **Idempotent.** Re-ingesting the same document yields 0 created rows, 0 events and the same provenance rows14  (refreshed `last_observed`). Connector keys (`peeringdb:fac:12`) are mapped in `entity_keys`.15- **Atomic per batch, tolerant per entity.** A batch runs in one transaction; each entity runs in a savepoint,16  so a bad record is rolled back and counted as `rejected` without failing the batch. Dry runs roll back everything.1718### Concurrency1920Connectors run in parallel, so two batches may describe the same operator at the same moment. Operator21resolution takes a transaction-scoped advisory lock on the normalized (canonical) name before looking up or22inserting, which serializes creation across batches; `mergeOperators()` folds duplicates that slipped through23before the lock existed. Facilities are *not* locked (a dataset batch can carry thousands of them and Postgres'24lock table is finite): two connectors creating the same unseen facility within the same seconds can still25produce a pair of records that only a later near-duplicate sweep or an admin merge (`mergeFacilities`) will fold.2627## Facilities2829### Resolution order30311. **Connector key** — `entity_keys.key = record.key` → same facility (following `merged_into`). Always wins.322. **Shared external ids** — any `externalIds` entry (`peeringdb_fac`, `osm`, `wikidata`, …) already stored on a33   live facility → same facility.343. **Candidate scoring** — candidates are live facilities that (a) lie in a 40 km bounding box around the incoming35   coordinates, or (b) share the normalized name (or an alias) in the same country, or (c) belong to the same36   operator in the same country (so facility codes like `DC12` can be compared). Up to 400 candidates are scored.3738### Score3940Weighted sum, weights redistributed over the signals available on both sides:4142| Signal | Weight | Value |43|---|---|---|44| Name | 0.40 | 1 for equal normalized names (noise words removed), equal code-preserving names or an alias hit; else max(token Jaccard, 0.85 × containment) |45| Facility codes (`DC12`, `FR5`, `LD8`) | 0.15 | 1 when both names carry codes and one matches, 0 when they conflict; skipped when either side has no code |46| Operator | 0.20 | 1 same operator, 0.5 unknown on either side, 0 different |47| Distance | 0.15 | `1 − 0.4·d/r` inside the match radius `r = matchRadiusKm(precisionA, precisionB)` (exact 250 m … metro 40 km), decaying to 0 at 4r; 0.6 for "same city" without coordinates |48| Address | 0.10 | normalized address equality → 1; same house number → ≥ 0.75; different house numbers → ≤ 0.4; else token Jaccard |4950Rules applied on top of the weighted sum:5152- **Operator + code rule**: same operator, matching facility code and geographically compatible (inside the radius or53  no coordinates) → the name signal is raised to 0.95. This is what merges `DC12` into `Equinix DC12`.54- **Code conflict**: same operator but conflicting codes (`DC12` vs `DC13`) caps the name signal at 0.45.55- **Different countries** cap the score at 0.30.56- **Too far**: precise coordinates more than 4× the match radius apart cap the score at 0.50.57- **Different known operators never merge** — the score is capped below the pending threshold — unless the normalized58  name *and* the normalized address are both exact (an operator change on the same building).5960### Decision thresholds6162| Score | Decision | What happens |63|---|---|---|64| ≥ 0.92 | **auto-merge** | The incoming record updates the candidate; `entity_matches` gets an `auto_merged` row (score + reasons) for audit. |65| 0.60 – 0.92 | **pending** | A **new facility is created** with `confidence = unverified`, and an `entity_matches` row with `status = pending`, the candidate JSON (incl. `createdFacilityId`), the matched facility id, the score and the reasons. Nothing is lost; the admin decides. |66| < 0.60 | **create** | New facility (`auto_created` row recorded when a candidate scored ≥ 0.30, for tuning). |6768Why "create + pending" rather than "hold": the record is visible immediately (flagged unverified), its provenance is69preserved, and approving the match later is a pure fold (`mergeFacilities`). Holding would hide real facilities for70days when the review queue is slow.7172### Admin review flow7374`GET /api/admin/matches?status=pending` lists the queue. For each row:7576- **Approve** → `mergeFacilities(createdFacilityId, matchedFacilityId)`: keys, aliases, provenance (deduplicated on77  entity/field/source/url), tenants, IXPs, projects and events are moved to the surviving facility; the created one78  gets `merged_into`; its external ids are folded; derived fields are recomputed; the match row becomes `approved`.79- **Reject** → the match row becomes `rejected`; the created facility simply stays a separate record and its80  confidence is recomputed on the next observation (it is no longer forced to `unverified`).8182A facility keeps `confidence = unverified` for as long as a pending match references it as `createdFacilityId`.8384### Field merge policy8586Each incoming field is compared with the observation that currently backs the stored value (from `provenance`,87`is_current = true`, joined with `sources.kind`):88891. Empty stored value → take the incoming one.902. Same source re-observing → take it (a source may correct itself).913. Measured beats estimate (`isEstimate`), whatever the source.924. Otherwise higher **authority** wins: operator / government / filing 4 · utility / cloud provider 3.5 · registry 3 ·93   dataset 2.5 · community 2 · secondary 1.5 · news 1 (estimates −1.5). Ties → most recent observation.9495Special cases:9697- **MW** (`itCapacityMw`, `totalPowerMw`, `plannedPowerMw`): a non-primary source (dataset, community, secondary,98  news) never overwrites a figure from an operator / government / filing / utility / cloud-provider / registry99  source. `mw_is_estimate` is true only when every current MW observation is an estimate.100- **Coordinates**: never replaced by a less precise point (`PRECISION_RANK`: exact > parcel > street > approximate >101  city > metro > unknown). Same precision → newest wins (same source always refreshes its own point).102- **Flags** `is_ai`, `is_hyperscale`: true sticks. A facility operated by a hyperscaler is hyperscale.103- **Certifications, aliases, external ids**: accumulated (set union).104- **Name**: same policy as other fields; the previous name becomes an alias.105- **Status / type**: `unknown` never overwrites a known value.106107### Derived fields (recomputed after every write)108109- `completeness` 0–100: geo 15 (exact/parcel/street; 8 for city/metro/approximate), operator 10, address 10,110  status 10, MW 20 (14 when estimate; planned-only 12/8), type 5, opened 10, website 5, description 5, tenants/IXPs 10.111- `confidence` = `computeConfidence()` with the best source kind as base and the number of distinct primary source112  kinds as corroborations; `verified` needs ≥ 2 primary kinds, `estimated` when every observation is an estimate.113- `source_count` = distinct sources with current provenance; `last_verified` = latest observation by a primary source.114- `metro_id` = nearest seeded metro whose radius covers the point (same country), else the metro whose alias matches115  the city; `country_iso2` falls back to the metro's country; `geohash` (precision 7).116117## Operators118119Order: connector key → external ids → curated canonical table (`canonical-operators.ts`: ~175 brands with aliases,120e.g. Interxion/Telx/DuPont Fabros → Digital Realty, RagingWire/e-shelter/Gyron/NetMagic → NTT Global Data Centers)121→ exact normalized name → alias match (case-insensitive) → trigram similarity ≥ 0.92 **and** the same website122domain → create. Kind inference: curated kind, else caller hint (carrier / cloud), else name heuristics.123Hyperscalers (AWS, Microsoft, Google, Meta, Oracle, Alibaba Cloud, Tencent Cloud, Apple, IBM, Huawei Cloud) get124`kind = hyperscaler`; `is_cloud_provider` / `is_carrier` are set from the table or the role hint and only ever turn125on. Incoming names that differ from the canonical name are appended to `aliases`. News text can only *link* to126curated or already-known operators; it never creates one.127128## Projects129130Key → external ids → same `sourceUrl` → (same country, same or unknown operator, normalized-name trigram131similarity ≥ 0.85). Timeline rows are deduplicated on `(project, date, type, sha256(description))`. Status changes132emit `project_status_changed`; planned MW, expected opening, investment and operator changes follow the same event133rules as facilities. `last_update` moves whenever a column or a timeline row changes.134135## Cloud regions, IXPs, campuses, tenants136137- Cloud regions are unique on `(provider, code)`; the provider is resolved as an operator with the cloud hint.138- IXPs: key → external ids → normalized name (or long name) in the same country. `facility_ixps` links come from the139  IXP's `facilityKeys` (resolved through `entity_keys`) or from a facility page's `ixps` list.140- Campuses: key → external ids → normalized name with compatible operator/country.141- Tenants: `carriers` / `cloudProviders` on a facility resolve to operators (carrier / cloud hint, `AS12345` parsed142  into `asn`) and fill `facility_tenants`; `carriers_count` / `networks_count` are recomputed.143144## Events145146Tracked facility fields: status, IT/total/planned MW, opened/announced/construction-start dates, operator, owner,147facility type, name. Significance: status 90 · operator 85 · owner 70 · MW change ≥ 20 % 80 else 50 · dates 60 ·148others 20 (first-time values are capped at 50; enrichments of name/type/operator are not events).149Fingerprint = `sha256(entityType|entityId|eventType|JSON(newValue)|day)` — the same change observed by several pages150on the same day is one event. `facility_discovered` (65 when MW ≥ 50 or in pipeline, else 40) and151`project_announced` (75 when ≥ 100 MW planned) are emitted for new records. News items keyed by URL emit one event152per article (fingerprint on the URL) when they carry an event type with significance ≥ 40.153