spb/datacenterindex

1# @dci/api — DataCenterIndex REST API23Fastify 5 service exposing the public read API (`/api/v1/*`) consumed by `apps/web`, and a token-protected admin4API (`/api/admin/*`) for connector operations, document forensics, reconciliation and curation. Response types5are the contract in `packages/core/src/api-types.ts`.67```bash8pnpm --filter @dci/api dev        # tsx watch, port API_PORT (8311)9pnpm --filter @dci/api start10pnpm --filter @dci/api typecheck11pnpm --filter @dci/api test       # vitest: query interpretation + integration against the local Postgres (.env)12```1314Docs: `GET /api/v1/docs` (Swagger UI), `GET /api/v1/openapi.json`.1516## Conventions1718- Envelope on every `/api/v1` response: `{ data, meta, sources }`. `meta` carries `total/page/perPage`, `generatedAt`19  and, where a figure is derived, `methodology`. `sources` (detail endpoints) lists the `SourceRef`s behind the20  provenance / events shown.21- Errors: `{ error, statusCode, details? }` — 400 with zod issues for bad query/body, 404 for unknown entities and22  routes, 401 for admin, 429 for rate limits.23- Caching: in-process LRU + Redis (`dci:api:cache:*`), key = route + normalized query, TTL per route family24  (dashboard 60 s, lists 120 s, details 300 s, map 300 s, events/news 30 s, search 60 s, sitemap 600 s). Headers:25  `Cache-Control: public, max-age, s-maxage=<ttl>, stale-while-revalidate`, weak `ETag` (independent of26  `generatedAt`; `If-None-Match` → 304), `X-Cache: HIT|MISS`. Disable with `DCI_API_CACHE=0`. Admin writes bust27  the affected prefixes (`/datacenters`, `/dashboard`, `/map`, `/events`).28- Ids or slugs are accepted everywhere a `:slug` appears; a facility whose `merged_into` is set redirects to its29  survivor in the detail endpoint and is hidden from lists.30- MW methodology: `mw = COALESCE(it_capacity_mw, total_power_mw, planned_power_mw)`. `knownMw` sums31  operational/expansion/partially_operational facilities, `constructionMw` under_construction, `plannedMw`32  rumored/proposed/announced/permitting/approved/delayed (planned figure first). Per-facility estimates keep their33  `mwIsEstimate` flag; nothing is invented.34- Rate limit: 240 req/min per IP on public routes (`X-RateLimit-*` headers), 60/min on admin. Client IP is taken35  from `X-Forwarded-For` / `X-Real-IP` only when the peer is a trusted proxy (`DCI_TRUSTED_PROXIES`, default36  127/8, 10/8, 172.16/12, ::1).37- Logging: pino JSON on stdout. Every request is also batched into ClickHouse `api_requests` (never fatal;38  `DCI_API_CH_LOG=0` disables). `/api/metrics` exposes Prometheus text (requests by route/status, latency39  histogram, cache hits/misses, memory cache size, pool max).4041## Public routes (`/api/v1`)4243| Route | Returns | Notes |44|---|---|---|45| `GET /datacenters` | `FacilitySummary[]` | `FacilityFilters` from api-types: `q, country (csv iso2), region, metro, operator, status (csv), type (csv), min_mw, max_mw, opened_from/to, planned_from/to (pipeline statuses by expected year), cloud (provider slug: tenant role cloud or cloud region in same metro), hyperscale, colocation, ai, renewable, has_mw, has_geo, confidence (csv), sort (name|mw|updated|opened|completeness), order, page, per_page ≤ 100`. Excludes merged rows. |46| `GET /datacenters/:idOrSlug` | `FacilityDetail` | aliases, owner, campus, tenants (carriers / cloud), IXPs, cloud regions in the metro (12), nearby ≤ 25 km (8, haversine), linked projects (facility_id or same operator + metro), current provenance, last 50 events, `sourceHistory`, externalIds. |47| `GET /operators` | `OperatorSummary[]` | `q, kind, country, sort (facilities|name|mw), order, page, per_page`; counts aggregated live from facilities (stats jsonb as fallback). |48| `GET /operators/:slug` | `OperatorDetail` | facilities paginated with `?fPage` (50). |49| `GET /countries` | `CountrySummary[]` | countries with ≥ 1 facility or cloud region; `?all=1` for every row; `?region=`. |50| `GET /countries/:slugOrIso2` | `CountryDetail` | top operators, metros, cloud regions, recent projects/events, growth by opened year, status/type breakdowns, announced investment, rankings positions, facilities (`?fPage`). |51| `GET /metros` | `MetroSummary[]` | `?country=&q=`. |52| `GET /metros/:slug` | `MetroDetail` | operators, cloud providers/regions, IXPs, facilities (`?fPage`), projects, events, `constraints` (news mentions with power/land/water/grid/moratorium keywords; `[]` when none), growth, rankings. |53| `GET /cloud-regions` | `CloudRegionSummary[]` | `?provider=&country=&status=`. |54| `GET /cloud-regions/:slug` | summary + metro, sibling regions, facilities in metro | |55| `GET /ixps`, `GET /ixps/:slug` | `IxpSummary[]` / summary + facilities | `?country=&q=`. |56| `GET /projects` | `ProjectSummary[]` | `status (csv), country, operator, metro, min_mw, ai, q, sort (updated|mw|announced|opening), order, page`. |57| `GET /projects/pipeline` | aggregates | by status, by expected year, top 15 countries, totals. |58| `GET /projects/:slug` | `ProjectDetail` | timeline sorted by `partialDateSortKey`, statusHistory from `project_status_changed` events. |59| `GET /events` | `EventDTO[]` | `type (csv), country, operator (slug), metro, project, entity_type, entity_id, min_significance, since, page, per_page ≤ 100`; `entity {slug,name}` resolved per entity type; rejected events hidden. |60| `GET /events/:id` | `EventDTO` | |61| `GET /news` | news items | `country, operator, since, q, page`. |62| `GET /map` | `MapResponse` | `zoom, bbox=w,s,e,n, status, type, operator, country, min_mw, max_mw, ai, hyperscale, cloud_regions=1`. zoom < 5 → country clusters (centroid from `countries` or facility mean), 5–8 → grid clusters (`gridCell` computed in SQL), ≥ 9 → points capped at 5 000 (else zoom-8 clusters, `mode: "clusters"`). Only facilities with coordinates; `p` = geo precision. |63| `GET /search?q=` | `SearchResponse` | interprets country / operator (trigram) / status / `NN MW` / type words; hits across facilities (tsvector + trigram), operators, countries, metros, cities (`href=/datacenters?q=<city>`), cloud regions, projects, IXPs (≤ 30); `facilities` block (20 + total) when the query has text or filters. |64| `GET /rankings`, `GET /rankings/:key?limit=` | ranking list / `Ranking` | current rows from the `rankings` table (computed by the worker). |65| `GET /dashboard` | `Dashboard` | live counts + MW by status, events 24h/7d, sources/documents, last crawl, capacity over time (daily_metrics → fallback opened year), pipeline by year, AI expansion, latest events, new projects, recently verified; top lists from rankings with live fallback. |66| `GET /stats/timeseries?metric=&dim=global&days=365` | `{day, value}[]` | without `metric`: available metrics. |67| `GET /sources`, `GET /sources/:id` | `SourceRef` + counts | documents, provenance rows, facilities, breakdowns. |68| `GET /sitemap/:kind?page=` | `{slug, updatedAt}[]` | kinds `facilities, operators, countries, metros, projects, cloud-regions`; 5 000 per page. |6970System: `GET /api/health` (no I/O), `GET /api/ready` (Postgres ping, Redis reported), `GET /api/metrics`.7172## Admin routes (`/api/admin`, header `x-dci-admin-token: $DCI_ADMIN_TOKEN`)7374Constant-time token compare; 401 otherwise, 503 when the token is not configured. All responses `no-store`.7576| Route | Purpose |77|---|---|78| `GET /connectors` | `ConnectorHealthDTO[]` from `connectors` + last run + document aggregates + ClickHouse `crawl_log` (avg ms, cost by fetcher, 7 days; run stats fallback). |79| `GET /connectors/:id` | config, last 30 runs, documents by page type, error samples. |80| `POST /connectors/:id/run` `{task: crawl|discover|full|reprocess, group?, limit?, force?}` | BullMQ job on queue `crawl` (prefix `dci` → Redis `dci:crawl:*`), jobId `manual__<id>__<ts>`. |81| `POST /connectors/:id/pause` / `resume` | toggles `connectors.paused`. |82| `PATCH /connectors/:id/schedule` `{group: interval}` | validates intervals, merges into `connectors.schedule` and rewrites the `schedule` section of `config/connectors/<id>.yaml` (backup `<id>.yaml.bak-<ts>`). |83| `GET /runs?connector=&limit=`, `GET /runs/:id` | runs (with log). |84| `GET /documents?connector=&pageType=&status=error|changed|quarantined&q=&page=` | URL registry. |85| `GET /documents/:id` | row + versions + provenance referencing it + resolved `entity_refs`. |86| `GET /documents/:id/raw?version=` | decompressed body from MinIO (zstd, gzip fallback), original content-type, capped 5 MB (`X-Truncated`). |87| `GET /documents/:id/versions/:vid/diff` | diff summary + detected changes (+ previous version). |88| `POST /documents/:id/reprocess` / `refetch` / `quarantine {quarantined}` | enqueue (`urls` + `force` in job data) / toggle. |89| `GET /matches?status=pending` | `entity_matches` with candidate + matched facility summaries. |90| `POST /matches/:id/approve` | points the candidate key at the matched facility; if the candidate had created a separate facility it is merged (aliases, provenance, tenants, IXPs, keys, projects, events, document refs moved; `merged_into` set). |91| `POST /matches/:id/reject` | keeps the separate facility. |92| `PATCH /facilities/:id` | curation of `name, status, facility_type, lat, lng, geo_precision, it_capacity_mw, total_power_mw, planned_power_mw, mw_is_estimate, opened_on, operator_id, is_ai, is_hyperscale, description (+note)`; writes `src_manual` provenance rows (source auto-created, kind registry) and a `status_changed` / `capacity_changed` / `facility_updated` event. Hand-entered coordinates default to `geo_precision = approximate`. |93| `POST /facilities/:id/merge {into}` | merge duplicate. |94| `PATCH /events/:id {reviewStatus}` | review. |95| `POST /devtool/fetch {url, level?, maxLevel?, renderJs?, country?, waitForSelector?, userAgent?}` | fetch through the escalation stack (SSRF policy enforced by the fetchers): status, fetcher, credits, html (≤ 1.5 MB), text (≤ 200 kB), markdown, title, JSON-LD, embedded JSON, classification, geo, address, 50 links. |96| `POST /devtool/preview {url|html, extractor, defaults?, connectorId?}` | runs one `ExtractorConfig` (validated) through `GenericConnector` extract → normalize → validate. |97| `POST /devtool/validate-config {yaml}` | `connectorConfigSchema` result + warnings. |98| `POST /devtool/save-config {id, yaml, version?}` | writes the YAML (backup), bumps `parserVersion`, upserts `connectors`. |99| `POST /devtool/run-sample {yaml, urls[] ≤ 10}` | per-URL fetch + extract + normalize + validate with a test context; field coverage matrix. |100| `GET /devtool/configs`, `GET /devtool/configs/:id` | YAML inventory / raw text. |101| `GET /ops` | worker heartbeat (`dci:worker:status`), queue counts, premium budgets (Redis keys or ClickHouse credits today), DB size + table estimates, ClickHouse/MinIO/Redis health, open `system_alerts`, last 20 runs, API cache stats. |102| `POST /maintenance/:task` (`rankings|metrics|refresh-stats`) | BullMQ job on `maintenance` (`{kind, task}`). |103| `POST /alerts/:id/resolve`, `POST /cache/invalidate {prefix}` | housekeeping. |104105## Environment106107| Variable | Default | Role |108|---|---|---|109| `API_PORT` / `API_HOST` | `8311` / `0.0.0.0` | listen address |110| `DATABASE_URL` | `postgres://dci:dci@127.0.0.1:5432/dci` | Postgres (via `@dci/db`; `PG_POOL_MAX` 16) |111| `REDIS_URL` | `redis://127.0.0.1:6379/0` | cache + BullMQ producers + worker status |112| `CLICKHOUSE_URL` / `CLICKHOUSE_DB` | `http://127.0.0.1:8123` / `dci` | request log, connector cost stats (optional) |113| `S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_REGION` | MinIO dev values | raw document bodies |114| `DCI_ADMIN_TOKEN` | — | admin API token (admin disabled when missing) |115| `DCI_CONFIG_DIR` | `<repo>/config/connectors` | connector YAML directory |116| `DCI_WORKER_CONNECTORS_DIR` | `<repo>/apps/worker/src/connectors` | optional: registers worker parsers for the dev tool |117| `DCI_API_CACHE`, `DCI_API_CACHE_ENTRIES` | `1`, `800` | response cache toggle / LRU size |118| `DCI_API_RATE_LIMIT`, `DCI_ADMIN_RATE_LIMIT` | `240`, `60` | per-minute per IP |119| `DCI_TRUSTED_PROXIES` | `127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,::1/128` | proxies allowed to set `X-Forwarded-For` |120| `DCI_API_CH_LOG` | `1` | ClickHouse request logging |121| `DCI_LOG_LEVEL` | `info` | pino level |122| `NEXT_PUBLIC_SITE_URL` | `https://www.datacenterindex.io` | OpenAPI server URL |123124In development the repo-root `.env` is loaded automatically (`process.loadEnvFile`, never overriding set variables).125126## Layout127128```129src/main.ts              entry (env, graceful shutdown)130src/app.ts               Fastify app: plugins, error handling, hooks, route registration131src/env.ts               typed env132src/cache.ts / redis.ts  LRU + Redis response cache; ioredis clients133src/queues.ts            BullMQ producers (crawl / maintenance)134src/storage.ts           MinIO reader (zstd / gzip)135src/metrics.ts           Prometheus registry     src/telemetry.ts  ClickHouse api_requests batch136src/lib/                 http (envelope, ETag, errors), params (zod), route (publicGet), sql (fragments), rows (coercion), dto (mappers), resolve (slug/id, entity refs), source-history137src/repositories/        facilities, operators, countries, metros, cloud-regions, ixps, projects, events, map, search, rankings, dashboard, misc (stats/sources/news/sitemap), admin/{connectors,merge}138src/search/interpret.ts  query interpretation (pure, tested)139src/routes/              public/{facilities,graph,activity,discovery}, admin/{index,connectors,documents,matches,curation,devtool,ops}, system140```141