spb/datacenterindex

1# osm-overpass — OpenStreetMap data centers via the Overpass API23| | |4|---|---|5| id / implementation | `osm-overpass` (`apps/worker/src/connectors/datasets/osm-overpass.ts`) |6| kind / mode / priority | community / dataset / 3 |7| endpoints | primary https://overpass-api.de/api/interpreter, fallback https://overpass.private.coffee/api/interpreter (kumi.systems replied 504 after 204 s during verification; maps.mail.ru is another working public instance) |8| entities | `NormalizedFacility`, key `osm:<type>/<id>` (e.g. `osm:way/123`), `externalIds.osm` |9| schedule | `dataset: monthly` |10| requests per run | one Overpass query per world tile (13 tiles, `params.maxBboxesPerRun`), sequential, ≥ 10 s apart |11| license | **ODbL 1.0 — © OpenStreetMap contributors** (attribution mandatory) |12| attribution | "© OpenStreetMap contributors, ODbL 1.0" — must be displayed wherever OSM-derived facts are shown; derived databases are share-alike |1314## License (verified 2026-09-11)1516https://www.openstreetmap.org/copyright:1718> "OpenStreetMap is open data, licensed under the Open Data Commons Open Database License (ODbL) by the OpenStreetMap Foundation (OSMF). In summary: You are free to copy, distribute, transmit and adapt our data, as long as you credit OpenStreetMap and its contributors. If you alter or build upon our data, you may distribute the result only under the same license. […] Where you use OpenStreetMap data, you are required to do the following two things: Provide credit to OpenStreetMap by displaying our attribution notice. Make clear that the data is available under the Open Database License."1920Implication for DataCenterIndex: facts sourced from OSM keep their `provenance.sourceId` so the UI/API can show "© OpenStreetMap contributors" next to them and the export layer can honour share-alike for OSM-derived subsets.2122## Access policy — why `respectRobots: false`2324`https://overpass-api.de/robots.txt` disallows `/api/` for `User-agent: *` (aimed at search-engine crawlers). The Overpass API is a query service whose terms of use are the "Commons" guidance (https://dev.overpass-api.de/overpass-doc/en/preface/commons.html, retrieved 2026-09-11):2526> "As a broad guideline to stay within safety margins, users are expected to send a maximum of about 10000 requests per day and keep their download volume below about 1 GB per day." — problematic behaviours listed: sending the same request tens of thousands of times a day, fetching elements one by one millions of times, "Stiching bounding boxes to scrape the full data of the complete world", using the public instance as an app backend.2728We send ≈ 13 requests and ≈ 3 MB per monthly pass with the bot UA, sequentially, honouring load shedding (429/504 → backoff, fallback, resumable cursor). `overpass.kumi.systems` has no robots.txt (404). **Decision to confirm by the owner**: `fetch.respectRobots: false` in the YAML applies to this API host only.2930## Query3132```33[out:json][timeout:180][maxsize:536870912];34( nwr["telecom"="data_center"](S,W,N,E); nwr["building"="data_center"](S,W,N,E); );35out center tags;36```3738Tiles (a partition of lat −60…85, shared edges, no overlap): na-west, na-east, atlantic-north, latam, eu-west, eu-east, africa-west, africa-east, mena, central-asia, south-asia, east-asia, sea-oceania (see `WORLD_BBOXES`). The connector keeps a cursor (`osm-overpass:cursor` in connector state) advanced after each successfully extracted tile, so an interrupted run resumes at the failed tile and `params.maxBboxesPerRun` can spread a full pass over several runs (dry runs never write state).3940Fetch policy: primary → fallback → primary (`params.maxAttempts`, default 3), backoff `params.backoffMs × attempt` between attempts, only for load-shedding signals (timeout/connection, 429, 502, 503, 504). A tile that still fails is logged and its cursor is not advanced.4142## Tag mapping (never invents)4344| field | tags |45|---|---|46| `name` | `name` → `name:en`; if absent, `operator` (+ `ref`, e.g. "Equinix LD4"). Features with neither name nor operator are skipped |47| `aliases` | `name:en` (when different), `alt_name`, `short_name`, `official_name`, `old_name`, `brand`, `operator:short` |48| `operatorName` / `ownerName` | `operator` → `brand` / `owner` |49| `address` | `addr:housenumber` + `addr:street` (number first for US/CA/GB/IE/AU/NZ/FR/IN/ZA/SG/HK/MY/PH or when country unknown), else `addr:street` or `addr:housename` |50| `city`, `regionName`, `postalCode` | `addr:city`, `addr:state`/`addr:province`, `addr:postcode` |51| `countryIso2` | `addr:country` only when it is an ISO-2 code (present on ~8 % of features). **Otherwise null: ingest must resolve the country from the coordinates / metro** |52| `geo` | node lat/lon, or `center` of ways/relations → precision `exact`, source `community:osm` |53| `openedOn` | `start_date` → `opening_date` via `parsePartialDate`; years < 1950 dropped (that is the building's date, not the DC's) |54| `totalPowerMw` | only `data_center:power` and only if it says MW (`parseMw`), e.g. "16 MW", "77.4MW". `power=*` is never used (it is the OSM power-infrastructure key) |55| `status` | `operational` by default (OSM maps existing features); `construction=*`/`building=construction` → `under_construction`; `disused`/`abandoned` → `closed` |56| `facilityType` | `inferFacilityType(name + description + operator)` or null (not defaulted to colocation) |57| `website` | `website` → `contact:website` → `url` (http(s) only) |58| `externalIds` | `osm: "way/123"`, `wikidata: Q…`, `operator_wikidata: Q…`, `osm_ref` |59| `facts` | `totalPowerMw` (`osm-tag:data_center:power`), `openedOn` (`osm-tag:start_date`) |6061## Verified live (2026-09-11)6263```64node node_modules/tsx/dist/cli.mjs scripts/try-connector.ts config/connectors/osm-overpass.yaml --limit 3 --json /tmp/osm-overpass.json65tile na-west        :   563 features →   472 kept ( 91 without name/operator), 6.0 s66tile na-east        : 1 337 features → 1 053 kept (284 without name/operator), 13.9 s67tile atlantic-north :    25 features →    22 kept (  3 without name/operator), 0.8 s68TOTAL 1 547 entities, 1 547 valid, 0 rejected, 0 warnings, 25 s (10 s pacing included)69```7071Coverage of the 1 547: all have coordinates (`exact`), 1 225 an operator, 929 an address, 410 a website, 44 an opening date, 37 an `addr:country`; status 1 539 operational / 8 under_construction; no `data_center:power` in MW in these tiles (Europe has a few: "1.2 MW", "16 MW", "77.4MW").72A first `--limit 2` run met a 504 on na-east (load shedding) and the then-fallback kumi.systems timed out — hence the alternating retry sequence and the fallback change. An exploratory Europe query (35,-12,72,40) returned 1 786 features in 38 s / 632 kB, 275 without name/operator — a full world pass should yield roughly 4 000–6 000 facilities.73Samples: `osm:node/3732415309` "The Pittock Internet Exchange" (Portland OR, operator Alco Properties, `internet_exchange`), `osm:node/1825907839` "Southwest Cyberport" (Albuquerque, alias SWCP, website).7475## Quirks / gaps7677- No `addr:country` on most features → `countryIso2` null with a validation *warning* only when coordinates are also missing (never the case for Overpass `out center`). The ingest/reconcile step must derive the country from coordinates.78- Overpass load shedding is real (504 on the second tile during verification; "Rate limit: 2" slots per IP). Keep `minDelayMs ≥ 10 s`, do not parallelise, keep `maxBboxesPerRun` small if runs are frequent.79- Public mirrors are volunteer-run and fluctuate (`kumi.systems` 504 after 204 s, `private.coffee` 21 s for a tiny query, `maps.mail.ru` 2 s but no robots.txt/terms page); the connector alternates primary/fallback instead of depending on any single one. `overpass.osm.ch` is Switzerland-only — unusable.80- OSM also has `office=telecommunication`, `man_made=data_center` (rare) and `telecom=exchange` — not queried (precision over recall).81- Multiple OSM features can represent one facility (building way + node POI). They get distinct keys; reconciliation (same name/operator within 250 m) merges them.82