# ip-probe — InternetPressure.io probe agent

Single static Go binary that measures HTTP / DNS / ICMP / traceroute against the targets assigned by the
ingestion API and ships signed, gzip-compressed batches. Implements `docs/PROBE-PROTOCOL.md` (v1) exactly.

```
services/probe-agent/
├── main.go, agent.go, once.go          CLI + wiring (run / once / check-config / --version)
├── internal/
│   ├── protocol/     wire types (config, batch, measurement, traceroute, health, error codes)
│   ├── signer/       HMAC-SHA256 request signature (+ test vectors, see below)
│   ├── config/       probe.yaml (+ IP_PROBE_* env) — flat YAML subset parser, no third-party dependency
│   ├── client/       signed HTTP client (GET /config, POST /batch, GET /agent/latest) + clock offset EWMA
│   ├── sched/        priority timer wheel: intervals, jitter, spread, boost, concurrency + per-host limits
│   ├── checks/http   httptrace timings, fresh connection, no redirects, 16 KiB body cap, error mapping
│   ├── checks/dns    one A query per resolver (system resolver or miekg/dns UDP + TCP on truncation)
│   ├── checks/ping   unprivileged ICMP echo (udp4/udp6) with tcp-connect fallback; also the "tcp" check
│   ├── checks/traceroute  system traceroute binary + parser (macOS / Linux) + route_hash
│   ├── batcher/      queue → JSON → gzip → signed POST; health ≤ 1/min; spool + backoff 5 s → 5 min
│   ├── spool/        data_dir/spool/<unixnano>.json.gz, 200 MiB cap, oldest first
│   ├── identity/     ipinfo.io → ip-api.com (public IP, ASN, org, country, city, 2-decimal lat/lon)
│   ├── health/       counters, /healthz, /metrics (Prometheus text)
│   └── update/       optional self-update (sha256-verified, atomic rename, exit 0)
├── deploy/launchd/io.internetpressure.probe.plist · deploy/systemd/ip-probe.service · deploy/install.sh
├── Makefile · VERSION · probe.example.yaml
└── dist/  (make build) ip-probe-darwin-arm64, ip-probe-linux-amd64, SHA256SUMS
```

Dependencies: Go standard library, `github.com/miekg/dns`, `golang.org/x/net` (icmp, ipv4, ipv6). Nothing else.

## Build & test

```bash
make build          # CGO_ENABLED=0, -trimpath, -ldflags "-s -w -X main.version=$(cat VERSION)" → dist/
make test           # go test ./...
make lint           # gofmt + go vet
make once T=www.cloudflare.com   # build for this host and run every check once (no server needed)
```

## Commands

| Command | Purpose |
|---|---|
| `ip-probe run [--config F]` | Run the agent (default command). |
| `ip-probe once --target HOST [--url U] [--ip IP] [--port N] [--no-traceroute] [--no-ping]` | Run http + dns (all resolvers) + ping + traceroute once and print JSON. Works without a config file or server. |
| `ip-probe check-config [--config F]` | Validate the configuration, print it with the key redacted. |
| `ip-probe --version` | `ip-probe 0.1.0 (darwin/arm64, go1.26.3)` |

Config lookup order: `--config`, `$IP_PROBE_CONFIG`, `/etc/internetpressure/probe.yaml`, `~/.internetpressure/probe.yaml`.

## Configuration (`probe.yaml`)

```yaml
probe_id: ca-qc-01
key: <64 hex chars>                       # HMAC key issued by the admin API
ingest_url: https://www.internetpressure.io/ingest/v1
data_dir: /var/lib/internetpressure       # spool/ + config.json (last good config) + update state; 0700
listen: 127.0.0.1:9381                    # local /healthz + /metrics
log_level: info                           # debug | info | warn | error
allow_self_update: true
resolvers_override: []                    # optional: ["system", "google=8.8.8.8:53", "1.1.1.1"]
max_concurrency: 8                        # global cap on concurrent checks
```

Every key can be overridden by `IP_PROBE_<UPPER_KEY>` (lists comma-separated), e.g.
`IP_PROBE_PROBE_ID=ca-qc-01 IP_PROBE_KEY=… ip-probe run`. The file is a flat YAML document (scalars, inline or
block lists, comments) parsed without a YAML library — nested mappings are rejected.

Logs go to stderr: human-readable text on a terminal, JSON lines otherwise (launchd / journald).

## How signing works

Every request carries `X-IP-Probe`, `X-IP-Timestamp` (Unix seconds) and `X-IP-Signature`:

```
canonical  = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + sha256_hex(body_bytes_as_sent)
signature  = lowercase_hex( HMAC-SHA256( hex_decode(key), canonical ) )
```

* `PATH` is the request path without query string (`/ingest/v1/batch`).
* For `POST /batch` the body is gzip → the hash covers the **compressed bytes** exactly as sent. Spooled batches
  are the compressed bytes, so a retry re-signs with a fresh timestamp without touching the payload.
* For GET the body hash is `sha256_hex("")` = `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`.
* The timestamp is the local clock corrected by the estimated offset (`clock_offset_ms`, see below), so a probe
  with a drifting clock still passes the ±300 s window. On `401 … skew …` the agent resyncs from the response
  `Date` header and retries the batch once.

### Test vector (also in `internal/signer/signer_test.go`; computed independently with Python `hmac`)

| | |
|---|---|
| key (hex) | `000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f` |
| probe_id | `ca-qc-01` |
| timestamp | `1789189804` (2026-09-12T05:10:04Z) |
| **POST** path | `/ingest/v1/batch` |
| body (raw, not gzipped for the vector) | `{"probe_id":"ca-qc-01","agent_version":"0.1.0","measurements":[]}` |
| sha256(body) | `9fd6962cedcf4a7aedb13c38f7f63137fdbbbf3703b447ebd761234981847c65` |
| canonical | `POST\n/ingest/v1/batch\n1789189804\n9fd6962c…847c65` |
| **signature** | `111075ee8d4c20723598c8f75d5ad89433f0385ff965f123302d2b9411280723` |
| **GET** path | `/ingest/v1/config` (body empty) |
| canonical | `GET\n/ingest/v1/config\n1789189804\ne3b0c442…52b855` |
| **signature** | `3798599e7f1bed2dab170d2aacf5f5288ec7cc591a64bd37285d3ea3c3450a8b` |

Python reference:

```python
import hmac, hashlib
key = bytes.fromhex("000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f")
canonical = f"POST\n/ingest/v1/batch\n1789189804\n{hashlib.sha256(body).hexdigest()}".encode()
hmac.new(key, canonical, hashlib.sha256).hexdigest()   # == 111075ee…0723
```

## Behaviour summary

* **Start-up**: load config → start `/healthz` → apply `data_dir/config.json` (last good config, offline start)
  → `GET /config` with backoff 5 s → 5 min until it succeeds (warn, never fatal) → refresh every
  `config_refresh_seconds` (min 30 s) or immediately when a batch response carries another `config_version`.
  Target adds/removes/edits and interval changes apply live.
* **Scheduler**: single timer wheel. `http` every `tiers[tier]`, `dns` every `dns_every` (one query per resolver,
  sequential), `ping` every `ping_every`, `traceroute` every `traceroute_every`. ±10 % jitter, first run
  uniformly spread over one interval, `boost` factor honoured until `until`, floor of **10 s** (server values
  below are clamped and logged). Global limit `max_concurrency`, never two checks on the same hostname at once,
  never two traceroutes at once. Panics in a check are recovered and counted.
* **http**: fresh connection (keep-alive disabled, no proxy, HTTP/2 attempted), `httptrace` gives `dns_ms`,
  `tcp_ms`, `tls_ms`; `ttfb_ms` = start → first byte, `total_ms` = start → body (≤ 16 KiB) read. Redirects are
  not followed (3xx is `ok`). `ok = connected ∧ TLS ok ∧ status < 500`; 4xx keeps `ok=true` with
  `error="http_4xx"`, 5xx → `ok=false, error="http_5xx"`. Pinned `ip` is dialed while the hostname stays in SNI
  and `Host`. Error mapping: `dns_fail`, `tcp_timeout`, `tcp_refused`, `tcp_reset`, `tls_fail`, `tls_cert`,
  `http_timeout`, `reset`, `other`.
* **dns**: `system` → OS resolver; others → UDP A query, EDNS0 1232, 3 s, retried once over TCP only when
  truncated. `ok = NOERROR ∧ ≥1 answer`; answers sorted.
* **ping**: 5 echoes, 200 ms apart, 2 s each, through an unprivileged ICMP datagram socket. If the socket cannot
  be opened the measurement becomes kind `tcp` (5 sequential connects to `port`) with `error="icmp_unavailable"`
  when successful, `unreachable` when nothing answered. `jitter_ms` = mean absolute successive difference.
* **traceroute**: `traceroute -n -q 1 -w 2 -m 30 <ip>` (60 s cap), hops with `*`, `route_hash` = sha1 of
  `ip1|ip2|*|…`, `reached` when the last hop is the destination, `total_ms` = RTT of the last answered hop.
* **Batching**: flush every `batch_flush_seconds` or at `max_batch`; health block at most once per minute.
  Network error / 5xx / 429 → gzip batch spooled to `data_dir/spool/`, backoff 5 s → 5 min, oldest first when
  the server is back. 200 MiB spool cap drops the oldest files. Other 4xx → dropped and logged.
* **Shutdown** (SIGTERM/SIGINT): stop scheduling, wait for in-flight checks, one final POST with a 5 s timeout,
  spool whatever remains.
* **Identity**: ipinfo.io (fallback ip-api.com) at start and hourly; only public IP, ASN, org, country, city and
  coordinates rounded to 2 decimals are kept.
* **Self-update**: every 6 h `GET /agent/latest`; a different version with an asset for `GOOS-GOARCH` is
  downloaded to `data_dir/update.tmp`, sha256-verified, `chmod 0755`, renamed over the running binary, then the
  agent exits 0 and the supervisor restarts it. Each version is attempted at most once (`data_dir/update.last`).

## Local endpoints

* `GET http://127.0.0.1:9381/healthz` →
  `{"ok":true,"probe_id":"…","version":"0.1.0","buffered":0,"spool_bytes":0,"last_flush":"…","last_config":"…","config_version":"…","targets":250,"clock_offset_ms":-14,"uptime_s":…,"capabilities":[…]}`
* `GET /metrics` → `ip_probe_measurements_total{kind}`, `ip_probe_errors_total{code}`, `ip_probe_buffered`,
  `ip_probe_spool_bytes`, `ip_probe_flush_failures_total`, `ip_probe_check_panics_total`,
  `ip_probe_clock_offset_ms`, `ip_probe_targets`, `ip_probe_uptime_seconds`, `ip_probe_rss_mb`, `ip_probe_goroutines`, `ip_probe_info`.

## Install

```bash
make build
sudo deploy/install.sh ca-qc-01 <64-hex-key> https://www.internetpressure.io/ingest/v1
```

The installer is idempotent: it installs `dist/ip-probe-<os>-<arch>` to `/usr/local/bin/ip-probe`, writes
`/etc/internetpressure/probe.yaml` (0600; kept if id/key/url are unchanged), creates `/var/lib/internetpressure`
(0700) and (re)starts the service. Re-running upgrades the binary and restarts.

* **macOS**: LaunchDaemon `/Library/LaunchDaemons/io.internetpressure.probe.plist` (`KeepAlive`, `RunAtLoad`,
  `UserName` = `$IP_PROBE_USER` or the invoking user, logs in `/var/log/internetpressure-probe.log`).
  Unprivileged ICMP and UDP traceroute work for any user.
* **Linux**: system user `ip-probe`, `/etc/systemd/system/ip-probe.service` (`Restart=always`, `RestartSec=5`),
  `/etc/sysctl.d/60-ip-probe.conf` sets `net.ipv4.ping_group_range = 0 2147483647` so ICMP needs no
  capability. `traceroute` must be installed (`apt install traceroute`).

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `config fetch failed … http 401` | Wrong `key`/`probe_id`, or clock skew > 300 s (fix NTP). The agent keeps retrying and runs from the cached config meanwhile. |
| `ping` measurements have `kind: "tcp"` and `error: "icmp_unavailable"` | Linux without `net.ipv4.ping_group_range` (installer sets it; otherwise `sysctl -w net.ipv4.ping_group_range="0 2147483647"`). |
| No traceroutes | No `traceroute` binary at `/usr/sbin/traceroute` (macOS) / `/usr/bin/traceroute` (Linux). Capability list in `/healthz` omits `traceroute`. |
| `interval below minimum, clamped` | Server asked for < 10 s; the agent enforces the 10 s floor (ethics rule). |
| `batch spooled … server unavailable` | Normal during an outage: `ls /var/lib/internetpressure/spool`; drained oldest-first when the server returns. |
| `local clock differs from server by more than 120 s` | Enable NTP; signatures still pass because timestamps are offset-corrected, but measurements are timestamped with the local clock. |
| Debug a target | `ip-probe once --target host.example --url https://host.example/path` |
| Where do logs go? | macOS `/var/log/internetpressure-probe.log`; Linux `journalctl -u ip-probe -f`. JSON lines, `log_level: debug` for per-batch details. |

## Notes on the protocol implementation

* Every wire field name/type follows `docs/PROBE-PROTOCOL.md`. Fields not relevant to a `kind` are omitted
  (pointer + `omitempty`), which the protocol allows ("omitted or null"); `total_ms`/`rtt_ms` in traceroutes are
  explicit `null` when unknown.
* `ttfb_ms` is measured from the start of the check (before DNS), i.e. the conventional time-to-first-byte, so
  `dns_ms + tcp_ms + tls_ms ≤ ttfb_ms ≤ total_ms` as in the protocol's example values.
* `http_4xx` is reported with `ok=true` (connection + TLS fine, status < 500), per the `ok` definition.
* On the ICMP → TCP fallback the measurement is `kind:"tcp"` and, when at least one connect succeeded,
  `error:"icmp_unavailable"` with `ok:true` — an informational code so the server can tell a real `tcp` check
  from a degraded `ping`.
* `rss_mb` is computed from `runtime.MemStats` (`Sys − HeapReleased`), a close upper bound of the true RSS.