# Probe ↔ Ingestion protocol (v1)

The probe agent (Go, services/probe-agent) talks only to the ingestion API (apps/api, FastAPI) over HTTPS. Base URL in production: https://www.internetpressure.io/ingest/v1. Nothing else is required on the probe side.

# Authentication — signed requests

Every probe has a probe_id (e.g. ca-qc-01) and a 32-byte hex key issued once by the admin API. Requests carry:

canonical = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + sha256_hex(raw_body_bytes_as_sent) where PATH is the request path without query string (e.g. /ingest/v1/batch) and the hash is over the compressed bytes for gzip bodies, or sha256_hex("") for GET. key_bytes = the hex key decoded to bytes. The server rejects skew > 300 s and replays (same probe + timestamp + body hash within 10 min).

# GET /ingest/v1/config

Returns the probe's assignment. The probe calls it at start-up, then every schedule.config_refresh_seconds, and immediately when a batch response carries a different config_version.

{
  "server_time": "2026-09-12T05:10:04.120Z",
  "config_version": "2026-09-12T04:00:00Z-7f3a",
  "probe": { "probe_id": "ca-qc-01", "name": "Québec City (Bell)", "region": "na-east", "country": "CA",
             "city": "Québec", "provider": "Bell Canada", "asn": 577, "lat": 46.8, "lon": -71.2, "enabled": true },
  "schedule": { "tiers": {"1": 20, "2": 45, "3": 180}, "dns_every": 60, "ping_every": 30, "traceroute_every": 900,
                "batch_flush_seconds": 10, "max_batch": 500, "config_refresh_seconds": 300,
                "boost": { "targets": ["aws-us-east-1"], "factor": 0.5, "until": "2026-09-12T05:25:00Z" } },
  "resolvers": [ {"id": "system", "address": ""}, {"id": "google", "address": "8.8.8.8:53"},
                 {"id": "cloudflare", "address": "1.1.1.1:53"}, {"id": "quad9", "address": "9.9.9.9:53"} ],
  "targets": [
    { "target_id": "cloudflare-www", "name": "Cloudflare", "hostname": "www.cloudflare.com",
      "url": "https://www.cloudflare.com/", "ip": null, "port": 443, "category": "cdn", "provider": "cloudflare",
      "service_id": "cloudflare", "country": null, "region": "global", "importance": 5, "tier": 1,
      "checks": ["http", "dns", "ping"], "traceroute": true }
  ]
}

Rules the agent follows:

• tier → HTTP check interval schedule.tiers[tier] seconds (±10 % jitter, spread uniformly so a probe never bursts).
• checks may contain http, dns, ping, tcp. dns runs against every configured resolver every dns_every seconds; ping every ping_every; traceroute: true targets get one traceroute every traceroute_every.
• boost: for listed targets multiply intervals by factor until until.
• A target with "ip" set is dialed at that IP (hostname still used for TLS SNI / Host).
• If probe.enabled is false the agent idles (health only) and re-checks config.
• Measurements must look like ordinary lightweight client traffic: one request, no retries inside a check, no parallel hammering of the same host, HTTP body read capped at 16 KiB, keep-alive disabled (so TCP/TLS are measured).

# POST /ingest/v1/batch

Gzip-compressed JSON:

{
  "probe_id": "ca-qc-01",
  "agent_version": "0.1.0",
  "sent_at": "2026-09-12T05:10:14.001Z",
  "measurements": [ /* ≤ max_batch */ ],
  "traceroutes": [ /* optional */ ],
  "health": { /* optional, at most once per minute */ }
}

# Measurement

{
  "ts": "2026-09-12T05:10:04.123Z",
  "target_id": "cloudflare-www",
  "kind": "http",                       // "http" | "dns" | "ping" | "tcp"
  "ok": true,
  "error": "",                          // "" or short code (see below)
  "dns_ms": 12.8, "tcp_ms": 23.1, "tls_ms": 35.4, "ttfb_ms": 79.2, "total_ms": 90.1,   // http (null when N/A)
  "http_status": 200, "http_proto": "HTTP/2.0", "tls_version": "TLS1.3",              // http
  "resolved_ip": "104.16.123.96",
  "resolver": "cloudflare", "dns_rcode": "NOERROR", "dns_answers": ["104.16.123.96","104.16.124.96"], // dns
  "sent": 5, "received": 5, "packet_loss": 0.0, "rtt_min_ms": 11.9, "rtt_avg_ms": 12.4, "rtt_max_ms": 13.0, "jitter_ms": 0.4 // ping/tcp
}

Only the fields relevant to kind need to be present; the others may be omitted or null. Timestamps are UTC RFC 3339 with milliseconds. packet_loss is 0–1.

Error codes: dns_fail, dns_timeout, dns_servfail, dns_nxdomain, tcp_timeout, tcp_refused, tcp_reset, tls_fail, tls_cert, http_timeout, http_5xx, http_4xx, reset, unreachable, icmp_unavailable, other. ok for http means: connection + TLS succeeded and status < 500. ok for dns means rcode NOERROR with ≥1 answer. ok for ping/tcp means received ≥ 1.

# Traceroute

{ "ts": "…", "target_id": "cloudflare-www", "dest_ip": "104.16.123.96", "reached": true, "hop_count": 11,
  "total_ms": 30.2, "route_hash": "3f1c…",              // sha1 over "ip1|ip2|*|ip4…" (unanswered hops kept as *)
  "hops": [ {"n": 1, "ip": "192.168.2.1", "rtt_ms": 1.2}, {"n": 2, "ip": "*", "rtt_ms": null} ] }

# Health

{ "ts": "…", "agent_version": "0.1.0", "uptime_s": 86400, "buffered": 0, "spool_bytes": 0,
  "measurements_total": 120345, "errors_total": 12, "clock_offset_ms": -14, "rss_mb": 21.4, "goroutines": 18,
  "capabilities": ["http", "dns", "ping", "traceroute"], "os": "darwin", "arch": "arm64",
  "identity": { "public_ip": "76.70.60.50", "asn": 577, "org": "Bell Canada", "country": "CA", "city": "Québec",
                "lat": 46.79, "lon": -71.35, "source": "ipinfo.io" } }

identity is best-effort (ipinfo.io / ip-api.com, refreshed hourly). Only the public IP, ASN and city-level coordinates are ever sent — never a street address. The server stores it on the probe record and displays city-level location only. clock_offset_ms = local clock − server clock, estimated from server_time in responses (half RTT correction).

# Response

{ "accepted": 231, "rejected": 0, "config_version": "…", "server_time": "…", "boost": { … } }

4xx on auth/signature → do not retry the same batch (it is spooled and re-sent with a fresh signature only if the failure was 401 skew). 5xx/network error → spool to disk, exponential backoff 5 s → 5 min, retry oldest first.

# GET /ingest/v1/agent/latest

{ "version": "0.1.3", "assets": { "darwin-arm64": { "url": "…", "sha256": "…" }, "linux-amd64": { … } } } — optional self-update: download, verify, atomically replace the binary, exit 0 (the supervisor restarts it).

# Local health endpoint (on the probe)

http://127.0.0.1:9381/healthz → {"ok":true,"probe_id":…,"buffered":…,"last_flush":…} and /metrics (Prometheus text). Not exposed publicly.