Architecture
Stores
- PostgreSQL (truth): countries, metros, operators, campuses, facilities (+aliases, entity_keys), cloud_regions, ixps, facility_ixps, facility_tenants, projects (+timeline), sources, connectors (+runs), documents (+versions), provenance, events, news_items, entity_matches, rankings, daily_metrics, admin_sessions, system_alerts.
- ClickHouse (append-only analytics): crawl_log, observations, page_changes, metrics, entity_daily, api_requests.
- Redis: BullMQ queues (
dci:crawl,dci:ingest,dci:maintenance), rate-limit + locks, API cache. - MinIO (S3): raw bodies
raw/<connector>/<doc_id>/<content_hash>.<ext>.zst, PDFs, exports.
Crawl flow
Scheduler (per connector group interval, adaptive by change_frequency_score) → Connector.discover → URL registry (documents, fingerprint, classify, priority) → priority queue → fetch (robots, rate limit, conditional GET, escalation L1→L4, budgets) → archive raw → content fingerprint → if changed: extract → normalize → validate → reconcile (entity_matches) → persist entities + provenance + entity_keys → diff old/new field values → events (+ ClickHouse observations / page_changes) → recompute documents.next_check.
Entity resolution (facilities)
Score candidates by: same connector key (exact) → external ids → normalized name equality / facility code match (DC12, FR5) → same operator + distance ≤ radius(precision) → address similarity → alias table. ≥ 0.92 auto-merge, 0.6–0.92 → entity_matches pending (admin approves), < 0.6 → create. Never merge across different operators unless name+address exact.
Confidence
computeConfidence() in core: source kind base (operator/government/filing/utility/cloud/registry = high; dataset/community/secondary/news = moderate), extraction certainty, corroborations (≥2 independent primary → verified), conflicts, age.
Change detection
contentFingerprint (noise-stripped) per fetch; on change store a document_versions row with line diff + detectedChanges (field-level from normalized entity diff), significance 0–100 (status change 90, MW change 80, date change 60, operator change 85, text-only 10–30).
Two-node deployment
- BHS128 (data/public): edge Caddy :8300 → web :8310 / api :8311; postgres, clickhouse, redis, minio bound to
10.68.0.1(private WireGuard) + docker network. Public traffic arrives via the BHS64 gateway over WireGuardwg1(10.67.0.60). - BHS64b (crawl): worker + scheduler containers reaching the data services over
dci0(10.68.0.2 → 10.68.0.1). No public ports.