spb/internetpressure

1# Probe ↔ Ingestion protocol (v1)23The probe agent (Go, `services/probe-agent`) talks only to the ingestion API (`apps/api`, FastAPI) over HTTPS.4Base URL in production: `https://www.internetpressure.io/ingest/v1`. Nothing else is required on the probe side.56## Authentication — signed requests78Every probe has a `probe_id` (e.g. `ca-qc-01`) and a 32-byte hex `key` issued once by the admin API. Requests carry:910| Header | Value |11|---|---|12| `X-IP-Probe` | `probe_id` |13| `X-IP-Timestamp` | Unix seconds (UTC) at the moment of sending |14| `X-IP-Signature` | lowercase hex `HMAC-SHA256(key_bytes, canonical)` |15| `Content-Encoding` | `gzip` on `POST /batch` (body is gzip-compressed JSON) |16| `Content-Type` | `application/json` |17| `User-Agent` | `InternetPressureProbe/<version> (+https://www.internetpressure.io/probes)` |1819`canonical = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + sha256_hex(raw_body_bytes_as_sent)` where PATH is the20request path without query string (e.g. `/ingest/v1/batch`) and the hash is over the compressed bytes for gzip21bodies, or `sha256_hex("")` for GET. `key_bytes` = the hex key decoded to bytes. The server rejects skew > 300 s22and replays (same probe + timestamp + body hash within 10 min).2324## GET /ingest/v1/config2526Returns the probe's assignment. The probe calls it at start-up, then every `schedule.config_refresh_seconds`, and27immediately when a batch response carries a different `config_version`.2829```json30{31  "server_time": "2026-09-12T05:10:04.120Z",32  "config_version": "2026-09-12T04:00:00Z-7f3a",33  "probe": { "probe_id": "ca-qc-01", "name": "Québec City (Bell)", "region": "na-east", "country": "CA",34             "city": "Québec", "provider": "Bell Canada", "asn": 577, "lat": 46.8, "lon": -71.2, "enabled": true },35  "schedule": { "tiers": {"1": 20, "2": 45, "3": 180}, "dns_every": 60, "ping_every": 30, "traceroute_every": 900,36                "batch_flush_seconds": 10, "max_batch": 500, "config_refresh_seconds": 300,37                "boost": { "targets": ["aws-us-east-1"], "factor": 0.5, "until": "2026-09-12T05:25:00Z" } },38  "resolvers": [ {"id": "system", "address": ""}, {"id": "google", "address": "8.8.8.8:53"},39                 {"id": "cloudflare", "address": "1.1.1.1:53"}, {"id": "quad9", "address": "9.9.9.9:53"} ],40  "targets": [41    { "target_id": "cloudflare-www", "name": "Cloudflare", "hostname": "www.cloudflare.com",42      "url": "https://www.cloudflare.com/", "ip": null, "port": 443, "category": "cdn", "provider": "cloudflare",43      "service_id": "cloudflare", "country": null, "region": "global", "importance": 5, "tier": 1,44      "checks": ["http", "dns", "ping"], "traceroute": true }45  ]46}47```4849Rules the agent follows:50- `tier` → HTTP check interval `schedule.tiers[tier]` seconds (±10 % jitter, spread uniformly so a probe never bursts).51- `checks` may contain `http`, `dns`, `ping`, `tcp`. `dns` runs against every configured resolver every `dns_every`52  seconds; `ping` every `ping_every`; `traceroute: true` targets get one traceroute every `traceroute_every`.53- `boost`: for listed targets multiply intervals by `factor` until `until`.54- A target with `"ip"` set is dialed at that IP (hostname still used for TLS SNI / Host).55- If `probe.enabled` is false the agent idles (health only) and re-checks config.56- Measurements must look like ordinary lightweight client traffic: one request, no retries inside a check, no57  parallel hammering of the same host, HTTP body read capped at 16 KiB, keep-alive disabled (so TCP/TLS are measured).5859## POST /ingest/v1/batch6061Gzip-compressed JSON:6263```json64{65  "probe_id": "ca-qc-01",66  "agent_version": "0.1.0",67  "sent_at": "2026-09-12T05:10:14.001Z",68  "measurements": [ /* ≤ max_batch */ ],69  "traceroutes": [ /* optional */ ],70  "health": { /* optional, at most once per minute */ }71}72```7374### Measurement7576```json77{78  "ts": "2026-09-12T05:10:04.123Z",79  "target_id": "cloudflare-www",80  "kind": "http",                       // "http" | "dns" | "ping" | "tcp"81  "ok": true,82  "error": "",                          // "" or short code (see below)83  "dns_ms": 12.8, "tcp_ms": 23.1, "tls_ms": 35.4, "ttfb_ms": 79.2, "total_ms": 90.1,   // http (null when N/A)84  "http_status": 200, "http_proto": "HTTP/2.0", "tls_version": "TLS1.3",              // http85  "resolved_ip": "104.16.123.96",86  "resolver": "cloudflare", "dns_rcode": "NOERROR", "dns_answers": ["104.16.123.96","104.16.124.96"], // dns87  "sent": 5, "received": 5, "packet_loss": 0.0, "rtt_min_ms": 11.9, "rtt_avg_ms": 12.4, "rtt_max_ms": 13.0, "jitter_ms": 0.4 // ping/tcp88}89```9091Only the fields relevant to `kind` need to be present; the others may be omitted or null. Timestamps are UTC RFC 333992with milliseconds. `packet_loss` is 0–1.9394Error codes: `dns_fail`, `dns_timeout`, `dns_servfail`, `dns_nxdomain`, `tcp_timeout`, `tcp_refused`, `tcp_reset`,95`tls_fail`, `tls_cert`, `http_timeout`, `http_5xx`, `http_4xx`, `reset`, `unreachable`, `icmp_unavailable`, `other`.96`ok` for `http` means: connection + TLS succeeded and status < 500. `ok` for `dns` means rcode NOERROR with ≥1 answer.97`ok` for `ping`/`tcp` means received ≥ 1.9899### Traceroute100101```json102{ "ts": "…", "target_id": "cloudflare-www", "dest_ip": "104.16.123.96", "reached": true, "hop_count": 11,103  "total_ms": 30.2, "route_hash": "3f1c…",              // sha1 over "ip1|ip2|*|ip4…" (unanswered hops kept as *)104  "hops": [ {"n": 1, "ip": "192.168.2.1", "rtt_ms": 1.2}, {"n": 2, "ip": "*", "rtt_ms": null} ] }105```106107### Health108109```json110{ "ts": "…", "agent_version": "0.1.0", "uptime_s": 86400, "buffered": 0, "spool_bytes": 0,111  "measurements_total": 120345, "errors_total": 12, "clock_offset_ms": -14, "rss_mb": 21.4, "goroutines": 18,112  "capabilities": ["http", "dns", "ping", "traceroute"], "os": "darwin", "arch": "arm64",113  "identity": { "public_ip": "76.70.60.50", "asn": 577, "org": "Bell Canada", "country": "CA", "city": "Québec",114                "lat": 46.79, "lon": -71.35, "source": "ipinfo.io" } }115```116117`identity` is best-effort (ipinfo.io / ip-api.com, refreshed hourly). Only the public IP, ASN and city-level118coordinates are ever sent — never a street address. The server stores it on the probe record and displays119city-level location only. `clock_offset_ms` = local clock − server clock, estimated from `server_time` in responses120(half RTT correction).121122### Response123124```json125{ "accepted": 231, "rejected": 0, "config_version": "…", "server_time": "…", "boost": { … } }126```127128`4xx` on auth/signature → do not retry the same batch (it is spooled and re-sent with a fresh signature only if the129failure was `401 skew`). `5xx`/network error → spool to disk, exponential backoff 5 s → 5 min, retry oldest first.130131## GET /ingest/v1/agent/latest132133`{ "version": "0.1.3", "assets": { "darwin-arm64": { "url": "…", "sha256": "…" }, "linux-amd64": { … } } }`134— optional self-update: download, verify, atomically replace the binary, exit 0 (the supervisor restarts it).135136## Local health endpoint (on the probe)137138`http://127.0.0.1:9381/healthz` → `{"ok":true,"probe_id":…,"buffered":…,"last_flush":…}` and `/metrics`139(Prometheus text). Not exposed publicly.140