# Probe ↔ Ingestion protocol (v1)

The probe agent (Go, `services/probe-agent`) talks only to the ingestion API (`apps/api`, FastAPI) over HTTPS.
Base URL in production: `https://www.internetpressure.io/ingest/v1`. Nothing else is required on the probe side.

## Authentication — signed requests

Every probe has a `probe_id` (e.g. `ca-qc-01`) and a 32-byte hex `key` issued once by the admin API. Requests carry:

| Header | Value |
|---|---|
| `X-IP-Probe` | `probe_id` |
| `X-IP-Timestamp` | Unix seconds (UTC) at the moment of sending |
| `X-IP-Signature` | lowercase hex `HMAC-SHA256(key_bytes, canonical)` |
| `Content-Encoding` | `gzip` on `POST /batch` (body is gzip-compressed JSON) |
| `Content-Type` | `application/json` |
| `User-Agent` | `InternetPressureProbe/<version> (+https://www.internetpressure.io/probes)` |

`canonical = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + sha256_hex(raw_body_bytes_as_sent)` where PATH is the
request path without query string (e.g. `/ingest/v1/batch`) and the hash is over the compressed bytes for gzip
bodies, or `sha256_hex("")` for GET. `key_bytes` = the hex key decoded to bytes. The server rejects skew > 300 s
and replays (same probe + timestamp + body hash within 10 min).

## GET /ingest/v1/config

Returns the probe's assignment. The probe calls it at start-up, then every `schedule.config_refresh_seconds`, and
immediately when a batch response carries a different `config_version`.

```json
{
  "server_time": "2026-09-12T05:10:04.120Z",
  "config_version": "2026-09-12T04:00:00Z-7f3a",
  "probe": { "probe_id": "ca-qc-01", "name": "Québec City (Bell)", "region": "na-east", "country": "CA",
             "city": "Québec", "provider": "Bell Canada", "asn": 577, "lat": 46.8, "lon": -71.2, "enabled": true },
  "schedule": { "tiers": {"1": 20, "2": 45, "3": 180}, "dns_every": 60, "ping_every": 30, "traceroute_every": 900,
                "batch_flush_seconds": 10, "max_batch": 500, "config_refresh_seconds": 300,
                "boost": { "targets": ["aws-us-east-1"], "factor": 0.5, "until": "2026-09-12T05:25:00Z" } },
  "resolvers": [ {"id": "system", "address": ""}, {"id": "google", "address": "8.8.8.8:53"},
                 {"id": "cloudflare", "address": "1.1.1.1:53"}, {"id": "quad9", "address": "9.9.9.9:53"} ],
  "targets": [
    { "target_id": "cloudflare-www", "name": "Cloudflare", "hostname": "www.cloudflare.com",
      "url": "https://www.cloudflare.com/", "ip": null, "port": 443, "category": "cdn", "provider": "cloudflare",
      "service_id": "cloudflare", "country": null, "region": "global", "importance": 5, "tier": 1,
      "checks": ["http", "dns", "ping"], "traceroute": true }
  ]
}
```

Rules the agent follows:
- `tier` → HTTP check interval `schedule.tiers[tier]` seconds (±10 % jitter, spread uniformly so a probe never bursts).
- `checks` may contain `http`, `dns`, `ping`, `tcp`. `dns` runs against every configured resolver every `dns_every`
  seconds; `ping` every `ping_every`; `traceroute: true` targets get one traceroute every `traceroute_every`.
- `boost`: for listed targets multiply intervals by `factor` until `until`.
- A target with `"ip"` set is dialed at that IP (hostname still used for TLS SNI / Host).
- If `probe.enabled` is false the agent idles (health only) and re-checks config.
- Measurements must look like ordinary lightweight client traffic: one request, no retries inside a check, no
  parallel hammering of the same host, HTTP body read capped at 16 KiB, keep-alive disabled (so TCP/TLS are measured).

## POST /ingest/v1/batch

Gzip-compressed JSON:

```json
{
  "probe_id": "ca-qc-01",
  "agent_version": "0.1.0",
  "sent_at": "2026-09-12T05:10:14.001Z",
  "measurements": [ /* ≤ max_batch */ ],
  "traceroutes": [ /* optional */ ],
  "health": { /* optional, at most once per minute */ }
}
```

### Measurement

```json
{
  "ts": "2026-09-12T05:10:04.123Z",
  "target_id": "cloudflare-www",
  "kind": "http",                       // "http" | "dns" | "ping" | "tcp"
  "ok": true,
  "error": "",                          // "" or short code (see below)
  "dns_ms": 12.8, "tcp_ms": 23.1, "tls_ms": 35.4, "ttfb_ms": 79.2, "total_ms": 90.1,   // http (null when N/A)
  "http_status": 200, "http_proto": "HTTP/2.0", "tls_version": "TLS1.3",              // http
  "resolved_ip": "104.16.123.96",
  "resolver": "cloudflare", "dns_rcode": "NOERROR", "dns_answers": ["104.16.123.96","104.16.124.96"], // dns
  "sent": 5, "received": 5, "packet_loss": 0.0, "rtt_min_ms": 11.9, "rtt_avg_ms": 12.4, "rtt_max_ms": 13.0, "jitter_ms": 0.4 // ping/tcp
}
```

Only the fields relevant to `kind` need to be present; the others may be omitted or null. Timestamps are UTC RFC 3339
with milliseconds. `packet_loss` is 0–1.

Error codes: `dns_fail`, `dns_timeout`, `dns_servfail`, `dns_nxdomain`, `tcp_timeout`, `tcp_refused`, `tcp_reset`,
`tls_fail`, `tls_cert`, `http_timeout`, `http_5xx`, `http_4xx`, `reset`, `unreachable`, `icmp_unavailable`, `other`.
`ok` for `http` means: connection + TLS succeeded and status < 500. `ok` for `dns` means rcode NOERROR with ≥1 answer.
`ok` for `ping`/`tcp` means received ≥ 1.

### Traceroute

```json
{ "ts": "…", "target_id": "cloudflare-www", "dest_ip": "104.16.123.96", "reached": true, "hop_count": 11,
  "total_ms": 30.2, "route_hash": "3f1c…",              // sha1 over "ip1|ip2|*|ip4…" (unanswered hops kept as *)
  "hops": [ {"n": 1, "ip": "192.168.2.1", "rtt_ms": 1.2}, {"n": 2, "ip": "*", "rtt_ms": null} ] }
```

### Health

```json
{ "ts": "…", "agent_version": "0.1.0", "uptime_s": 86400, "buffered": 0, "spool_bytes": 0,
  "measurements_total": 120345, "errors_total": 12, "clock_offset_ms": -14, "rss_mb": 21.4, "goroutines": 18,
  "capabilities": ["http", "dns", "ping", "traceroute"], "os": "darwin", "arch": "arm64",
  "identity": { "public_ip": "76.70.60.50", "asn": 577, "org": "Bell Canada", "country": "CA", "city": "Québec",
                "lat": 46.79, "lon": -71.35, "source": "ipinfo.io" } }
```

`identity` is best-effort (ipinfo.io / ip-api.com, refreshed hourly). Only the public IP, ASN and city-level
coordinates are ever sent — never a street address. The server stores it on the probe record and displays
city-level location only. `clock_offset_ms` = local clock − server clock, estimated from `server_time` in responses
(half RTT correction).

### Response

```json
{ "accepted": 231, "rejected": 0, "config_version": "…", "server_time": "…", "boost": { … } }
```

`4xx` on auth/signature → do not retry the same batch (it is spooled and re-sent with a fresh signature only if the
failure was `401 skew`). `5xx`/network error → spool to disk, exponential backoff 5 s → 5 min, retry oldest first.

## GET /ingest/v1/agent/latest

`{ "version": "0.1.3", "assets": { "darwin-arm64": { "url": "…", "sha256": "…" }, "linux-amd64": { … } } }`
— optional self-update: download, verify, atomically replace the binary, exit 0 (the supervisor restarts it).

## Local health endpoint (on the probe)

`http://127.0.0.1:9381/healthz` → `{"ok":true,"probe_id":…,"buffered":…,"last_flush":…}` and `/metrics`
(Prometheus text). Not exposed publicly.