SPB Git forge
38commits 1branches 0releases
338.7 MBsize
maindefault branch
3 h agolast push
HTML 53.9% TypeScript 44.5% JavaScript 0.6% SQL 0.5%
13.2 KB

# @dci/api — DataCenterIndex REST API

Fastify 5 service exposing the public read API (/api/v1/*) consumed by apps/web, and a token-protected admin API (/api/admin/*) for connector operations, document forensics, reconciliation and curation. Response types are the contract in packages/core/src/api-types.ts.

bash
pnpm --filter @dci/api dev        # tsx watch, port API_PORT (8311)
pnpm --filter @dci/api start
pnpm --filter @dci/api typecheck
pnpm --filter @dci/api test       # vitest: query interpretation + integration against the local Postgres (.env)

Docs: GET /api/v1/docs (Swagger UI), GET /api/v1/openapi.json.

# Conventions

  • Envelope on every /api/v1 response: { data, meta, sources }. meta carries total/page/perPage, generatedAt and, where a figure is derived, methodology. sources (detail endpoints) lists the SourceRefs behind the provenance / events shown.
  • Errors: { error, statusCode, details? } — 400 with zod issues for bad query/body, 404 for unknown entities and routes, 401 for admin, 429 for rate limits.
  • Caching: in-process LRU + Redis (dci:api:cache:*), key = route + normalized query, TTL per route family (dashboard 60 s, lists 120 s, details 300 s, map 300 s, events/news 30 s, search 60 s, sitemap 600 s). Headers: Cache-Control: public, max-age, s-maxage=<ttl>, stale-while-revalidate, weak ETag (independent of generatedAt; If-None-Match → 304), X-Cache: HIT|MISS. Disable with DCI_API_CACHE=0. Admin writes bust the affected prefixes (/datacenters, /dashboard, /map, /events).
  • Ids or slugs are accepted everywhere a :slug appears; a facility whose merged_into is set redirects to its survivor in the detail endpoint and is hidden from lists.
  • MW methodology: mw = COALESCE(it_capacity_mw, total_power_mw, planned_power_mw). knownMw sums operational/expansion/partially_operational facilities, constructionMw under_construction, plannedMw rumored/proposed/announced/permitting/approved/delayed (planned figure first). Per-facility estimates keep their mwIsEstimate flag; nothing is invented.
  • Rate limit: 240 req/min per IP on public routes (X-RateLimit-* headers), 60/min on admin. Client IP is taken from X-Forwarded-For / X-Real-IP only when the peer is a trusted proxy (DCI_TRUSTED_PROXIES, default 127/8, 10/8, 172.16/12, ::1).
  • Logging: pino JSON on stdout. Every request is also batched into ClickHouse api_requests (never fatal; DCI_API_CH_LOG=0 disables). /api/metrics exposes Prometheus text (requests by route/status, latency histogram, cache hits/misses, memory cache size, pool max).

# Public routes (/api/v1)

Route Returns Notes
GET /datacenters FacilitySummary[] FacilityFilters from api-types: `q, country (csv iso2), region, metro, operator, status (csv), type (csv), min_mw, max_mw, opened_from/to, planned_from/to (pipeline statuses by expected year), cloud (provider slug: tenant role cloud or cloud region in same metro), hyperscale, colocation, ai, renewable, has_mw, has_geo, confidence (csv), sort (name
GET /datacenters/:idOrSlug FacilityDetail aliases, owner, campus, tenants (carriers / cloud), IXPs, cloud regions in the metro (12), nearby ≤ 25 km (8, haversine), linked projects (facility_id or same operator + metro), current provenance, last 50 events, sourceHistory, externalIds.
GET /operators OperatorSummary[] `q, kind, country, sort (facilities
GET /operators/:slug OperatorDetail facilities paginated with ?fPage (50).
GET /countries CountrySummary[] countries with ≥ 1 facility or cloud region; ?all=1 for every row; ?region=.
GET /countries/:slugOrIso2 CountryDetail top operators, metros, cloud regions, recent projects/events, growth by opened year, status/type breakdowns, announced investment, rankings positions, facilities (?fPage).
GET /metros MetroSummary[] ?country=&q=.
GET /metros/:slug MetroDetail operators, cloud providers/regions, IXPs, facilities (?fPage), projects, events, constraints (news mentions with power/land/water/grid/moratorium keywords; [] when none), growth, rankings.
GET /cloud-regions CloudRegionSummary[] ?provider=&country=&status=.
GET /cloud-regions/:slug summary + metro, sibling regions, facilities in metro
GET /ixps, GET /ixps/:slug IxpSummary[] / summary + facilities ?country=&q=.
GET /projects ProjectSummary[] `status (csv), country, operator, metro, min_mw, ai, q, sort (updated
GET /projects/pipeline aggregates by status, by expected year, top 15 countries, totals.
GET /projects/:slug ProjectDetail timeline sorted by partialDateSortKey, statusHistory from project_status_changed events.
GET /events EventDTO[] type (csv), country, operator (slug), metro, project, entity_type, entity_id, min_significance, since, page, per_page ≤ 100; entity {slug,name} resolved per entity type; rejected events hidden.
GET /events/:id EventDTO
GET /news news items country, operator, since, q, page.
GET /map MapResponse zoom, bbox=w,s,e,n, status, type, operator, country, min_mw, max_mw, ai, hyperscale, cloud_regions=1. zoom < 5 → country clusters (centroid from countries or facility mean), 5–8 → grid clusters (gridCell computed in SQL), ≥ 9 → points capped at 5 000 (else zoom-8 clusters, mode: "clusters"). Only facilities with coordinates; p = geo precision.
GET /search?q= SearchResponse interprets country / operator (trigram) / status / NN MW / type words; hits across facilities (tsvector + trigram), operators, countries, metros, cities (href=/datacenters?q=<city>), cloud regions, projects, IXPs (≤ 30); facilities block (20 + total) when the query has text or filters.
GET /rankings, GET /rankings/:key?limit= ranking list / Ranking current rows from the rankings table (computed by the worker).
GET /dashboard Dashboard live counts + MW by status, events 24h/7d, sources/documents, last crawl, capacity over time (daily_metrics → fallback opened year), pipeline by year, AI expansion, latest events, new projects, recently verified; top lists from rankings with live fallback.
GET /stats/timeseries?metric=&dim=global&days=365 {day, value}[] without metric: available metrics.
GET /sources, GET /sources/:id SourceRef + counts documents, provenance rows, facilities, breakdowns.
GET /sitemap/:kind?page= {slug, updatedAt}[] kinds facilities, operators, countries, metros, projects, cloud-regions; 5 000 per page.

System: GET /api/health (no I/O), GET /api/ready (Postgres ping, Redis reported), GET /api/metrics.

# Admin routes (/api/admin, header x-dci-admin-token: $DCI_ADMIN_TOKEN)

Constant-time token compare; 401 otherwise, 503 when the token is not configured. All responses no-store.

Route Purpose
GET /connectors ConnectorHealthDTO[] from connectors + last run + document aggregates + ClickHouse crawl_log (avg ms, cost by fetcher, 7 days; run stats fallback).
GET /connectors/:id config, last 30 runs, documents by page type, error samples.
POST /connectors/:id/run `{task: crawl discover
POST /connectors/:id/pause / resume toggles connectors.paused.
PATCH /connectors/:id/schedule {group: interval} validates intervals, merges into connectors.schedule and rewrites the schedule section of config/connectors/<id>.yaml (backup <id>.yaml.bak-<ts>).
GET /runs?connector=&limit=, GET /runs/:id runs (with log).
`GET /documents?connector=&pageType=&status=error changed
GET /documents/:id row + versions + provenance referencing it + resolved entity_refs.
GET /documents/:id/raw?version= decompressed body from MinIO (zstd, gzip fallback), original content-type, capped 5 MB (X-Truncated).
GET /documents/:id/versions/:vid/diff diff summary + detected changes (+ previous version).
POST /documents/:id/reprocess / refetch / quarantine {quarantined} enqueue (urls + force in job data) / toggle.
GET /matches?status=pending entity_matches with candidate + matched facility summaries.
POST /matches/:id/approve points the candidate key at the matched facility; if the candidate had created a separate facility it is merged (aliases, provenance, tenants, IXPs, keys, projects, events, document refs moved; merged_into set).
POST /matches/:id/reject keeps the separate facility.
PATCH /facilities/:id curation of name, status, facility_type, lat, lng, geo_precision, it_capacity_mw, total_power_mw, planned_power_mw, mw_is_estimate, opened_on, operator_id, is_ai, is_hyperscale, description (+note); writes src_manual provenance rows (source auto-created, kind registry) and a status_changed / capacity_changed / facility_updated event. Hand-entered coordinates default to geo_precision = approximate.
POST /facilities/:id/merge {into} merge duplicate.
PATCH /events/:id {reviewStatus} review.
POST /devtool/fetch {url, level?, maxLevel?, renderJs?, country?, waitForSelector?, userAgent?} fetch through the escalation stack (SSRF policy enforced by the fetchers): status, fetcher, credits, html (≤ 1.5 MB), text (≤ 200 kB), markdown, title, JSON-LD, embedded JSON, classification, geo, address, 50 links.
`POST /devtool/preview {url html, extractor, defaults?, connectorId?}`
POST /devtool/validate-config {yaml} connectorConfigSchema result + warnings.
POST /devtool/save-config {id, yaml, version?} writes the YAML (backup), bumps parserVersion, upserts connectors.
POST /devtool/run-sample {yaml, urls[] ≤ 10} per-URL fetch + extract + normalize + validate with a test context; field coverage matrix.
GET /devtool/configs, GET /devtool/configs/:id YAML inventory / raw text.
GET /ops worker heartbeat (dci:worker:status), queue counts, premium budgets (Redis keys or ClickHouse credits today), DB size + table estimates, ClickHouse/MinIO/Redis health, open system_alerts, last 20 runs, API cache stats.
POST /maintenance/:task (`rankings metrics
POST /alerts/:id/resolve, POST /cache/invalidate {prefix} housekeeping.

# Environment

Variable Default Role
API_PORT / API_HOST 8311 / 0.0.0.0 listen address
DATABASE_URL postgres://dci:dci@127.0.0.1:5432/dci Postgres (via @dci/db; PG_POOL_MAX 16)
REDIS_URL redis://127.0.0.1:6379/0 cache + BullMQ producers + worker status
CLICKHOUSE_URL / CLICKHOUSE_DB http://127.0.0.1:8123 / dci request log, connector cost stats (optional)
S3_ENDPOINT, S3_BUCKET, S3_ACCESS_KEY, S3_SECRET_KEY, S3_REGION MinIO dev values raw document bodies
DCI_ADMIN_TOKEN — admin API token (admin disabled when missing)
DCI_CONFIG_DIR <repo>/config/connectors connector YAML directory
DCI_WORKER_CONNECTORS_DIR <repo>/apps/worker/src/connectors optional: registers worker parsers for the dev tool
DCI_API_CACHE, DCI_API_CACHE_ENTRIES 1, 800 response cache toggle / LRU size
DCI_API_RATE_LIMIT, DCI_ADMIN_RATE_LIMIT 240, 60 per-minute per IP
DCI_TRUSTED_PROXIES 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,::1/128 proxies allowed to set X-Forwarded-For
DCI_API_CH_LOG 1 ClickHouse request logging
DCI_LOG_LEVEL info pino level
NEXT_PUBLIC_SITE_URL https://www.datacenterindex.io OpenAPI server URL

In development the repo-root .env is loaded automatically (process.loadEnvFile, never overriding set variables).

# Layout

text
src/main.ts              entry (env, graceful shutdown)
src/app.ts               Fastify app: plugins, error handling, hooks, route registration
src/env.ts               typed env
src/cache.ts / redis.ts  LRU + Redis response cache; ioredis clients
src/queues.ts            BullMQ producers (crawl / maintenance)
src/storage.ts           MinIO reader (zstd / gzip)
src/metrics.ts           Prometheus registry     src/telemetry.ts  ClickHouse api_requests batch
src/lib/                 http (envelope, ETag, errors), params (zod), route (publicGet), sql (fragments), rows (coercion), dto (mappers), resolve (slug/id, entity refs), source-history
src/repositories/        facilities, operators, countries, metros, cloud-regions, ixps, projects, events, map, search, rankings, dashboard, misc (stats/sources/news/sitemap), admin/{connectors,merge}
src/search/interpret.ts  query interpretation (pure, tested)
src/routes/              public/{facilities,graph,activity,discovery}, admin/{index,connectors,documents,matches,curation,devtool,ops}, system