spb/datacenterindex

1# Architecture23## Stores4- **PostgreSQL** (truth): countries, metros, operators, campuses, facilities (+aliases, entity_keys), cloud_regions, ixps, facility_ixps, facility_tenants, projects (+timeline), sources, connectors (+runs), documents (+versions), provenance, events, news_items, entity_matches, rankings, daily_metrics, admin_sessions, system_alerts.5- **ClickHouse** (append-only analytics): crawl_log, observations, page_changes, metrics, entity_daily, api_requests.6- **Redis**: BullMQ queues (`dci:crawl`, `dci:ingest`, `dci:maintenance`), rate-limit + locks, API cache.7- **MinIO (S3)**: raw bodies `raw/<connector>/<doc_id>/<content_hash>.<ext>.zst`, PDFs, exports.89## Crawl flow10Scheduler (per connector group interval, adaptive by `change_frequency_score`) → Connector.discover → URL registry (`documents`, fingerprint, classify, priority) → priority queue → fetch (robots, rate limit, conditional GET, escalation L1→L4, budgets) → archive raw → content fingerprint → if changed: extract → normalize → validate → reconcile (entity_matches) → persist entities + provenance + entity_keys → diff old/new field values → events (+ ClickHouse observations / page_changes) → recompute `documents.next_check`.1112## Entity resolution (facilities)13Score candidates by: same connector key (exact) → external ids → normalized name equality / facility code match (DC12, FR5) → same operator + distance ≤ radius(precision) → address similarity → alias table. ≥ 0.92 auto-merge, 0.6–0.92 → `entity_matches` pending (admin approves), < 0.6 → create. Never merge across different operators unless name+address exact.1415## Confidence16`computeConfidence()` in core: source kind base (operator/government/filing/utility/cloud/registry = high; dataset/community/secondary/news = moderate), extraction certainty, corroborations (≥2 independent primary → verified), conflicts, age.1718## Change detection19`contentFingerprint` (noise-stripped) per fetch; on change store a `document_versions` row with line diff + `detectedChanges` (field-level from normalized entity diff), significance 0–100 (status change 90, MW change 80, date change 60, operator change 85, text-only 10–30).2021## Two-node deployment22- **BHS128 (data/public)**: edge Caddy :8300 → web :8310 / api :8311; postgres, clickhouse, redis, minio bound to `10.68.0.1` (private WireGuard) + docker network. Public traffic arrives via the BHS64 gateway over WireGuard `wg1` (10.67.0.60).23- **BHS64b (crawl)**: worker + scheduler containers reaching the data services over `dci0` (10.68.0.2 → 10.68.0.1). No public ports.24