# DataCenterIndex.io — repo guide (read first)

⚠️ PRODUCTION RETIRÉE LE 2026-10-02. Le serveur OVH qui hébergeait cette app a été retiré du MacLustr (résiliation OVH ; seule la passerelle BHS64 est conservée). Plus aucune instance ne tourne, le site public ne répond plus et la route MacLustr Tunnel a été supprimée. Code final = ce dépôt (spbgit, branche main). Sauvegardes à froid : dumps Postgres + état sur M3U96b:~/ovh-retired-20261002/, manifestes/secrets/PM2 sur le laptop ~/Desktop/Cluster/secrets/ovh-retired-20261002/ (0600). Les procédures de déploiement ci-dessous sont historiques.

Product: the global index of data center infrastructure (facilities, campuses, operators, cloud regions, metros, countries, projects, IXPs, power, connectivity) with a live change feed. Feels like Our World in Data × a Bloomberg terminal for digital infrastructure × a global infrastructure map. The product is the structured graph and its history, not the crawler.

Pipeline: discover → fetch → archive → extract → normalize → reconcile → validate → version → detect changes → publish. Never scrape → dump → show.

# Non-negotiables

Never invent data. No fake facilities, capacities, coordinates, dates. A figure that is an estimate is flagged isEstimate. City-level coordinates are geo_precision = city, never exact.
Provenance on every field (provenance table: source, URL, first/last observed, retrieved, confidence, method).
Legal crawling: robots.txt respected, no auth/paywall/CAPTCHA bypass, rate limits per host, conditional GET, content hashing — never refetch unchanged pages with premium credits. Attribution + license recorded per source (sources.license/attribution).
Escalation: L1 direct HTTP → L2 direct + browser identity → L3 Firecrawl → L4 Scrapfly rendered. Escalate only when blocked / JS shell. Premium daily budgets (DCI_SCRAPFLY_DAILY_BUDGET, DCI_FIRECRAWL_DAILY_BUDGET).
Secrets only in env / deploy secrets, never in git, never sent to the browser.
Mobile-first UI (320–430 px first), then tablet/desktop. Industrial, premium, restrained: no neon, no fake terminal, no giant hero, no card soup.

# Layout

text

packages/core        types (entities, provenance, events), ids, hashing, normalize (MW, dates, status, money), geo, confidence, diff, ssrf, api-types (API contract)
packages/db          Drizzle schema (Postgres) + plain SQL migrations (migrations/*.sql, runner src/migrate.ts) + ClickHouse client/DDL
packages/connectors  connector SDK: types, YAML config schema, fetchers (direct/firecrawl/scrapfly + escalation), robots, ratelimit, discovery (sitemap/rss/links), extract (JSON-LD, embedded JSON, selectors, PDF), classify, registry (parsers/implementations), GenericConnector
config/connectors/*.yaml   one file per source (declarative; `parser:` or `implementation:` for code-backed logic)
apps/worker          runtime: ConnectorContext, document registry + storage (MinIO), change detection, ingest (reconcile/persist/events), scheduler (BullMQ), rankings/metrics, CLI `pnpm dci …`
apps/api             Fastify REST `/api/v1/*` (public) + `/api/admin/*` (token) — see packages/core/src/api-types.ts
apps/web             Next 16 (App Router, Tailwind v4, MapLibre) — SSR for content, lazy map/charts
deploy/              Dockerfiles, compose (data node + crawl node), edge Caddy, backups, monitoring
docs/                ARCHITECTURE.md, CONNECTORS.md (authoring guide), DEPLOY.md

# Conventions

pnpm workspaces, TypeScript strict, ESM, verbatimModuleSyntax (use import type). Run pnpm typecheck before finishing.
Ids: prefixed (fac_, op_, prj_, evt_, doc_…) via newId() / stableId() in @dci/core. Slugs unique per table.
Drizzle: casing: "snake_case". sql\${arr}`spreads arrays → usetextArray()orsql.raw. db.execute` returns timestamps as strings.
Partial dates are strings: 2027, 2027-06, 2027-06-15, 2027-Q2, 2027-H1 (parsePartialDate, formatPartialDate).
Connector keys are connector-scoped and stable (peeringdb:fac:12), mapped in entity_keys for idempotent re-ingestion.
Events are deduplicated by fingerprint (entity + type + new value + day).
API envelope { data, meta, sources }. Web reads NEXT_PUBLIC_API_URL server-side (API_URL_INTERNAL preferred when set).
Ports: web 8310 (Next, loopback in prod), api 8311, edge 8300 (Caddy in compose, bound to the WireGuard IP). Public: https://www.datacenterindex.io via BHS64 gateway.
Local dev: Postgres postgres://dci:dci@127.0.0.1:5432/dci, Redis 6379, ClickHouse 8123, MinIO 9000 (see .env.example, deploy/dev-compose.yml).

# Connector authoring (short)

config/connectors/<id>.yaml (id, name, domain, kind, license/attribution, fetch levels, discovery, schedule, extractors).
Declarative fields when possible; otherwise a parser in apps/worker/src/connectors/<group>/<id>.ts (registered in <group>/index.ts register() via registerParser), or a full implementation (registerImplementation) for APIs/datasets.
Validate against LIVE pages: pnpm dci run <id> --dry-run --limit 5 must yield valid entities (no errors in the ValidationReport). Nothing ships unverified.
Document the source (what, license, update cadence, quirks) in docs/connectors/<id>.md.