# ip-probe — InternetPressure.io probe agent

Single static Go binary that measures HTTP / DNS / ICMP / traceroute against the targets assigned by the ingestion API and ships signed, gzip-compressed batches. Implements docs/PROBE-PROTOCOL.md (v1) exactly.

services/probe-agent/
├── main.go, agent.go, once.go          CLI + wiring (run / once / check-config / --version)
├── internal/
│   ├── protocol/     wire types (config, batch, measurement, traceroute, health, error codes)
│   ├── signer/       HMAC-SHA256 request signature (+ test vectors, see below)
│   ├── config/       probe.yaml (+ IP_PROBE_* env) — flat YAML subset parser, no third-party dependency
│   ├── client/       signed HTTP client (GET /config, POST /batch, GET /agent/latest) + clock offset EWMA
│   ├── sched/        priority timer wheel: intervals, jitter, spread, boost, concurrency + per-host limits
│   ├── checks/http   httptrace timings, fresh connection, no redirects, 16 KiB body cap, error mapping
│   ├── checks/dns    one A query per resolver (system resolver or miekg/dns UDP + TCP on truncation)
│   ├── checks/ping   unprivileged ICMP echo (udp4/udp6) with tcp-connect fallback; also the "tcp" check
│   ├── checks/traceroute  system traceroute binary + parser (macOS / Linux) + route_hash
│   ├── batcher/      queue → JSON → gzip → signed POST; health ≤ 1/min; spool + backoff 5 s → 5 min
│   ├── spool/        data_dir/spool/<unixnano>.json.gz, 200 MiB cap, oldest first
│   ├── identity/     ipinfo.io → ip-api.com (public IP, ASN, org, country, city, 2-decimal lat/lon)
│   ├── health/       counters, /healthz, /metrics (Prometheus text)
│   └── update/       optional self-update (sha256-verified, atomic rename, exit 0)
├── deploy/launchd/io.internetpressure.probe.plist · deploy/systemd/ip-probe.service · deploy/install.sh
├── Makefile · VERSION · probe.example.yaml
└── dist/  (make build) ip-probe-darwin-arm64, ip-probe-linux-amd64, SHA256SUMS

Dependencies: Go standard library, github.com/miekg/dns, golang.org/x/net (icmp, ipv4, ipv6). Nothing else.

# Build & test

make build          # CGO_ENABLED=0, -trimpath, -ldflags "-s -w -X main.version=$(cat VERSION)" → dist/
make test           # go test ./...
make lint           # gofmt + go vet
make once T=www.cloudflare.com   # build for this host and run every check once (no server needed)

# Commands

Config lookup order: --config, $IP_PROBE_CONFIG, /etc/internetpressure/probe.yaml, ~/.internetpressure/probe.yaml.

# Configuration (probe.yaml)

probe_id: ca-qc-01
key: <64 hex chars>                       # HMAC key issued by the admin API
ingest_url: https://www.internetpressure.io/ingest/v1
data_dir: /var/lib/internetpressure       # spool/ + config.json (last good config) + update state; 0700
listen: 127.0.0.1:9381                    # local /healthz + /metrics
log_level: info                           # debug | info | warn | error
allow_self_update: true
resolvers_override: []                    # optional: ["system", "google=8.8.8.8:53", "1.1.1.1"]
max_concurrency: 8                        # global cap on concurrent checks

Every key can be overridden by IP_PROBE_<UPPER_KEY> (lists comma-separated), e.g. IP_PROBE_PROBE_ID=ca-qc-01 IP_PROBE_KEY=… ip-probe run. The file is a flat YAML document (scalars, inline or block lists, comments) parsed without a YAML library — nested mappings are rejected.

Logs go to stderr: human-readable text on a terminal, JSON lines otherwise (launchd / journald).

# How signing works

Every request carries X-IP-Probe, X-IP-Timestamp (Unix seconds) and X-IP-Signature:

canonical  = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + sha256_hex(body_bytes_as_sent)
signature  = lowercase_hex( HMAC-SHA256( hex_decode(key), canonical ) )

• PATH is the request path without query string (/ingest/v1/batch).
• For POST /batch the body is gzip → the hash covers the compressed bytes exactly as sent. Spooled batches are the compressed bytes, so a retry re-signs with a fresh timestamp without touching the payload.
• For GET the body hash is sha256_hex("") = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
• The timestamp is the local clock corrected by the estimated offset (clock_offset_ms, see below), so a probe with a drifting clock still passes the ±300 s window. On 401 … skew … the agent resyncs from the response Date header and retries the batch once.

# Test vector (also in internal/signer/signer_test.go; computed independently with Python hmac)

Python reference:

import hmac, hashlib
key = bytes.fromhex("000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f")
canonical = f"POST\n/ingest/v1/batch\n1789189804\n{hashlib.sha256(body).hexdigest()}".encode()
hmac.new(key, canonical, hashlib.sha256).hexdigest()   # == 111075ee…0723

# Behaviour summary

• Start-up: load config → start /healthz → apply data_dir/config.json (last good config, offline start) → GET /config with backoff 5 s → 5 min until it succeeds (warn, never fatal) → refresh every config_refresh_seconds (min 30 s) or immediately when a batch response carries another config_version. Target adds/removes/edits and interval changes apply live.
• Scheduler: single timer wheel. http every tiers[tier], dns every dns_every (one query per resolver, sequential), ping every ping_every, traceroute every traceroute_every. ±10 % jitter, first run uniformly spread over one interval, boost factor honoured until until, floor of 10 s (server values below are clamped and logged). Global limit max_concurrency, never two checks on the same hostname at once, never two traceroutes at once. Panics in a check are recovered and counted.
• http: fresh connection (keep-alive disabled, no proxy, HTTP/2 attempted), httptrace gives dns_ms, tcp_ms, tls_ms; ttfb_ms = start → first byte, total_ms = start → body (≤ 16 KiB) read. Redirects are not followed (3xx is ok). ok = connected ∧ TLS ok ∧ status < 500; 4xx keeps ok=true with error="http_4xx", 5xx → ok=false, error="http_5xx". Pinned ip is dialed while the hostname stays in SNI and Host. Error mapping: dns_fail, tcp_timeout, tcp_refused, tcp_reset, tls_fail, tls_cert, http_timeout, reset, other.
• dns: system → OS resolver; others → UDP A query, EDNS0 1232, 3 s, retried once over TCP only when truncated. ok = NOERROR ∧ ≥1 answer; answers sorted.
• ping: 5 echoes, 200 ms apart, 2 s each, through an unprivileged ICMP datagram socket. If the socket cannot be opened the measurement becomes kind tcp (5 sequential connects to port) with error="icmp_unavailable" when successful, unreachable when nothing answered. jitter_ms = mean absolute successive difference.
• traceroute: traceroute -n -q 1 -w 2 -m 30 <ip> (60 s cap), hops with *, route_hash = sha1 of ip1|ip2|*|…, reached when the last hop is the destination, total_ms = RTT of the last answered hop.
• Batching: flush every batch_flush_seconds or at max_batch; health block at most once per minute. Network error / 5xx / 429 → gzip batch spooled to data_dir/spool/, backoff 5 s → 5 min, oldest first when the server is back. 200 MiB spool cap drops the oldest files. Other 4xx → dropped and logged.
• Shutdown (SIGTERM/SIGINT): stop scheduling, wait for in-flight checks, one final POST with a 5 s timeout, spool whatever remains.
• Identity: ipinfo.io (fallback ip-api.com) at start and hourly; only public IP, ASN, org, country, city and coordinates rounded to 2 decimals are kept.
• Self-update: every 6 h GET /agent/latest; a different version with an asset for GOOS-GOARCH is downloaded to data_dir/update.tmp, sha256-verified, chmod 0755, renamed over the running binary, then the agent exits 0 and the supervisor restarts it. Each version is attempted at most once (data_dir/update.last).

# Local endpoints

• GET http://127.0.0.1:9381/healthz → {"ok":true,"probe_id":"…","version":"0.1.0","buffered":0,"spool_bytes":0,"last_flush":"…","last_config":"…","config_version":"…","targets":250,"clock_offset_ms":-14,"uptime_s":…,"capabilities":[…]}
• GET /metrics → ip_probe_measurements_total{kind}, ip_probe_errors_total{code}, ip_probe_buffered, ip_probe_spool_bytes, ip_probe_flush_failures_total, ip_probe_check_panics_total, ip_probe_clock_offset_ms, ip_probe_targets, ip_probe_uptime_seconds, ip_probe_rss_mb, ip_probe_goroutines, ip_probe_info.

# Install

make build
sudo deploy/install.sh ca-qc-01 <64-hex-key> https://www.internetpressure.io/ingest/v1

The installer is idempotent: it installs dist/ip-probe-<os>-<arch> to /usr/local/bin/ip-probe, writes /etc/internetpressure/probe.yaml (0600; kept if id/key/url are unchanged), creates /var/lib/internetpressure (0700) and (re)starts the service. Re-running upgrades the binary and restarts.

• macOS: LaunchDaemon /Library/LaunchDaemons/io.internetpressure.probe.plist (KeepAlive, RunAtLoad, UserName = $IP_PROBE_USER or the invoking user, logs in /var/log/internetpressure-probe.log). Unprivileged ICMP and UDP traceroute work for any user.
• Linux: system user ip-probe, /etc/systemd/system/ip-probe.service (Restart=always, RestartSec=5), /etc/sysctl.d/60-ip-probe.conf sets net.ipv4.ping_group_range = 0 2147483647 so ICMP needs no capability. traceroute must be installed (apt install traceroute).

# Troubleshooting

# Notes on the protocol implementation

• Every wire field name/type follows docs/PROBE-PROTOCOL.md. Fields not relevant to a kind are omitted (pointer + omitempty), which the protocol allows ("omitted or null"); total_ms/rtt_ms in traceroutes are explicit null when unknown.
• ttfb_ms is measured from the start of the check (before DNS), i.e. the conventional time-to-first-byte, so dns_ms + tcp_ms + tls_ms ≤ ttfb_ms ≤ total_ms as in the protocol's example values.
• http_4xx is reported with ok=true (connection + TLS fine, status < 500), per the ok definition.
• On the ICMP → TCP fallback the measurement is kind:"tcp" and, when at least one connect succeeded, error:"icmp_unavailable" with ok:true — an informational code so the server can tell a real tcp check from a degraded ping.
• rss_mb is computed from runtime.MemStats (Sys − HeapReleased), a close upper bound of the true RSS.