spb/internetpressure

1# ip-probe — InternetPressure.io probe agent23Single static Go binary that measures HTTP / DNS / ICMP / traceroute against the targets assigned by the4ingestion API and ships signed, gzip-compressed batches. Implements `docs/PROBE-PROTOCOL.md` (v1) exactly.56```7services/probe-agent/8├── main.go, agent.go, once.go          CLI + wiring (run / once / check-config / --version)9├── internal/10│   ├── protocol/     wire types (config, batch, measurement, traceroute, health, error codes)11│   ├── signer/       HMAC-SHA256 request signature (+ test vectors, see below)12│   ├── config/       probe.yaml (+ IP_PROBE_* env) — flat YAML subset parser, no third-party dependency13│   ├── client/       signed HTTP client (GET /config, POST /batch, GET /agent/latest) + clock offset EWMA14│   ├── sched/        priority timer wheel: intervals, jitter, spread, boost, concurrency + per-host limits15│   ├── checks/http   httptrace timings, fresh connection, no redirects, 16 KiB body cap, error mapping16│   ├── checks/dns    one A query per resolver (system resolver or miekg/dns UDP + TCP on truncation)17│   ├── checks/ping   unprivileged ICMP echo (udp4/udp6) with tcp-connect fallback; also the "tcp" check18│   ├── checks/traceroute  system traceroute binary + parser (macOS / Linux) + route_hash19│   ├── batcher/      queue → JSON → gzip → signed POST; health ≤ 1/min; spool + backoff 5 s → 5 min20│   ├── spool/        data_dir/spool/<unixnano>.json.gz, 200 MiB cap, oldest first21│   ├── identity/     ipinfo.io → ip-api.com (public IP, ASN, org, country, city, 2-decimal lat/lon)22│   ├── health/       counters, /healthz, /metrics (Prometheus text)23│   └── update/       optional self-update (sha256-verified, atomic rename, exit 0)24├── deploy/launchd/io.internetpressure.probe.plist · deploy/systemd/ip-probe.service · deploy/install.sh25├── Makefile · VERSION · probe.example.yaml26└── dist/  (make build) ip-probe-darwin-arm64, ip-probe-linux-amd64, SHA256SUMS27```2829Dependencies: Go standard library, `github.com/miekg/dns`, `golang.org/x/net` (icmp, ipv4, ipv6). Nothing else.3031## Build & test3233```bash34make build          # CGO_ENABLED=0, -trimpath, -ldflags "-s -w -X main.version=$(cat VERSION)" → dist/35make test           # go test ./...36make lint           # gofmt + go vet37make once T=www.cloudflare.com   # build for this host and run every check once (no server needed)38```3940## Commands4142| Command | Purpose |43|---|---|44| `ip-probe run [--config F]` | Run the agent (default command). |45| `ip-probe once --target HOST [--url U] [--ip IP] [--port N] [--no-traceroute] [--no-ping]` | Run http + dns (all resolvers) + ping + traceroute once and print JSON. Works without a config file or server. |46| `ip-probe check-config [--config F]` | Validate the configuration, print it with the key redacted. |47| `ip-probe --version` | `ip-probe 0.1.0 (darwin/arm64, go1.26.3)` |4849Config lookup order: `--config`, `$IP_PROBE_CONFIG`, `/etc/internetpressure/probe.yaml`, `~/.internetpressure/probe.yaml`.5051## Configuration (`probe.yaml`)5253```yaml54probe_id: ca-qc-0155key: <64 hex chars>                       # HMAC key issued by the admin API56ingest_url: https://www.internetpressure.io/ingest/v157data_dir: /var/lib/internetpressure       # spool/ + config.json (last good config) + update state; 070058listen: 127.0.0.1:9381                    # local /healthz + /metrics59log_level: info                           # debug | info | warn | error60allow_self_update: true61resolvers_override: []                    # optional: ["system", "google=8.8.8.8:53", "1.1.1.1"]62max_concurrency: 8                        # global cap on concurrent checks63```6465Every key can be overridden by `IP_PROBE_<UPPER_KEY>` (lists comma-separated), e.g.66`IP_PROBE_PROBE_ID=ca-qc-01 IP_PROBE_KEY=… ip-probe run`. The file is a flat YAML document (scalars, inline or67block lists, comments) parsed without a YAML library — nested mappings are rejected.6869Logs go to stderr: human-readable text on a terminal, JSON lines otherwise (launchd / journald).7071## How signing works7273Every request carries `X-IP-Probe`, `X-IP-Timestamp` (Unix seconds) and `X-IP-Signature`:7475```76canonical  = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + sha256_hex(body_bytes_as_sent)77signature  = lowercase_hex( HMAC-SHA256( hex_decode(key), canonical ) )78```7980* `PATH` is the request path without query string (`/ingest/v1/batch`).81* For `POST /batch` the body is gzip → the hash covers the **compressed bytes** exactly as sent. Spooled batches82  are the compressed bytes, so a retry re-signs with a fresh timestamp without touching the payload.83* For GET the body hash is `sha256_hex("")` = `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`.84* The timestamp is the local clock corrected by the estimated offset (`clock_offset_ms`, see below), so a probe85  with a drifting clock still passes the ±300 s window. On `401 … skew …` the agent resyncs from the response86  `Date` header and retries the batch once.8788### Test vector (also in `internal/signer/signer_test.go`; computed independently with Python `hmac`)8990| | |91|---|---|92| key (hex) | `000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f` |93| probe_id | `ca-qc-01` |94| timestamp | `1789189804` (2026-09-12T05:10:04Z) |95| **POST** path | `/ingest/v1/batch` |96| body (raw, not gzipped for the vector) | `{"probe_id":"ca-qc-01","agent_version":"0.1.0","measurements":[]}` |97| sha256(body) | `9fd6962cedcf4a7aedb13c38f7f63137fdbbbf3703b447ebd761234981847c65` |98| canonical | `POST\n/ingest/v1/batch\n1789189804\n9fd6962c…847c65` |99| **signature** | `111075ee8d4c20723598c8f75d5ad89433f0385ff965f123302d2b9411280723` |100| **GET** path | `/ingest/v1/config` (body empty) |101| canonical | `GET\n/ingest/v1/config\n1789189804\ne3b0c442…52b855` |102| **signature** | `3798599e7f1bed2dab170d2aacf5f5288ec7cc591a64bd37285d3ea3c3450a8b` |103104Python reference:105106```python107import hmac, hashlib108key = bytes.fromhex("000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f")109canonical = f"POST\n/ingest/v1/batch\n1789189804\n{hashlib.sha256(body).hexdigest()}".encode()110hmac.new(key, canonical, hashlib.sha256).hexdigest()   # == 111075ee…0723111```112113## Behaviour summary114115* **Start-up**: load config → start `/healthz` → apply `data_dir/config.json` (last good config, offline start)116  → `GET /config` with backoff 5 s → 5 min until it succeeds (warn, never fatal) → refresh every117  `config_refresh_seconds` (min 30 s) or immediately when a batch response carries another `config_version`.118  Target adds/removes/edits and interval changes apply live.119* **Scheduler**: single timer wheel. `http` every `tiers[tier]`, `dns` every `dns_every` (one query per resolver,120  sequential), `ping` every `ping_every`, `traceroute` every `traceroute_every`. ±10 % jitter, first run121  uniformly spread over one interval, `boost` factor honoured until `until`, floor of **10 s** (server values122  below are clamped and logged). Global limit `max_concurrency`, never two checks on the same hostname at once,123  never two traceroutes at once. Panics in a check are recovered and counted.124* **http**: fresh connection (keep-alive disabled, no proxy, HTTP/2 attempted), `httptrace` gives `dns_ms`,125  `tcp_ms`, `tls_ms`; `ttfb_ms` = start → first byte, `total_ms` = start → body (≤ 16 KiB) read. Redirects are126  not followed (3xx is `ok`). `ok = connected ∧ TLS ok ∧ status < 500`; 4xx keeps `ok=true` with127  `error="http_4xx"`, 5xx → `ok=false, error="http_5xx"`. Pinned `ip` is dialed while the hostname stays in SNI128  and `Host`. Error mapping: `dns_fail`, `tcp_timeout`, `tcp_refused`, `tcp_reset`, `tls_fail`, `tls_cert`,129  `http_timeout`, `reset`, `other`.130* **dns**: `system` → OS resolver; others → UDP A query, EDNS0 1232, 3 s, retried once over TCP only when131  truncated. `ok = NOERROR ∧ ≥1 answer`; answers sorted.132* **ping**: 5 echoes, 200 ms apart, 2 s each, through an unprivileged ICMP datagram socket. If the socket cannot133  be opened the measurement becomes kind `tcp` (5 sequential connects to `port`) with `error="icmp_unavailable"`134  when successful, `unreachable` when nothing answered. `jitter_ms` = mean absolute successive difference.135* **traceroute**: `traceroute -n -q 1 -w 2 -m 30 <ip>` (60 s cap), hops with `*`, `route_hash` = sha1 of136  `ip1|ip2|*|…`, `reached` when the last hop is the destination, `total_ms` = RTT of the last answered hop.137* **Batching**: flush every `batch_flush_seconds` or at `max_batch`; health block at most once per minute.138  Network error / 5xx / 429 → gzip batch spooled to `data_dir/spool/`, backoff 5 s → 5 min, oldest first when139  the server is back. 200 MiB spool cap drops the oldest files. Other 4xx → dropped and logged.140* **Shutdown** (SIGTERM/SIGINT): stop scheduling, wait for in-flight checks, one final POST with a 5 s timeout,141  spool whatever remains.142* **Identity**: ipinfo.io (fallback ip-api.com) at start and hourly; only public IP, ASN, org, country, city and143  coordinates rounded to 2 decimals are kept.144* **Self-update**: every 6 h `GET /agent/latest`; a different version with an asset for `GOOS-GOARCH` is145  downloaded to `data_dir/update.tmp`, sha256-verified, `chmod 0755`, renamed over the running binary, then the146  agent exits 0 and the supervisor restarts it. Each version is attempted at most once (`data_dir/update.last`).147148## Local endpoints149150* `GET http://127.0.0.1:9381/healthz` →151  `{"ok":true,"probe_id":"…","version":"0.1.0","buffered":0,"spool_bytes":0,"last_flush":"…","last_config":"…","config_version":"…","targets":250,"clock_offset_ms":-14,"uptime_s":…,"capabilities":[…]}`152* `GET /metrics` → `ip_probe_measurements_total{kind}`, `ip_probe_errors_total{code}`, `ip_probe_buffered`,153  `ip_probe_spool_bytes`, `ip_probe_flush_failures_total`, `ip_probe_check_panics_total`,154  `ip_probe_clock_offset_ms`, `ip_probe_targets`, `ip_probe_uptime_seconds`, `ip_probe_rss_mb`, `ip_probe_goroutines`, `ip_probe_info`.155156## Install157158```bash159make build160sudo deploy/install.sh ca-qc-01 <64-hex-key> https://www.internetpressure.io/ingest/v1161```162163The installer is idempotent: it installs `dist/ip-probe-<os>-<arch>` to `/usr/local/bin/ip-probe`, writes164`/etc/internetpressure/probe.yaml` (0600; kept if id/key/url are unchanged), creates `/var/lib/internetpressure`165(0700) and (re)starts the service. Re-running upgrades the binary and restarts.166167* **macOS**: LaunchDaemon `/Library/LaunchDaemons/io.internetpressure.probe.plist` (`KeepAlive`, `RunAtLoad`,168  `UserName` = `$IP_PROBE_USER` or the invoking user, logs in `/var/log/internetpressure-probe.log`).169  Unprivileged ICMP and UDP traceroute work for any user.170* **Linux**: system user `ip-probe`, `/etc/systemd/system/ip-probe.service` (`Restart=always`, `RestartSec=5`),171  `/etc/sysctl.d/60-ip-probe.conf` sets `net.ipv4.ping_group_range = 0 2147483647` so ICMP needs no172  capability. `traceroute` must be installed (`apt install traceroute`).173174## Troubleshooting175176| Symptom | Cause / fix |177|---|---|178| `config fetch failed … http 401` | Wrong `key`/`probe_id`, or clock skew > 300 s (fix NTP). The agent keeps retrying and runs from the cached config meanwhile. |179| `ping` measurements have `kind: "tcp"` and `error: "icmp_unavailable"` | Linux without `net.ipv4.ping_group_range` (installer sets it; otherwise `sysctl -w net.ipv4.ping_group_range="0 2147483647"`). |180| No traceroutes | No `traceroute` binary at `/usr/sbin/traceroute` (macOS) / `/usr/bin/traceroute` (Linux). Capability list in `/healthz` omits `traceroute`. |181| `interval below minimum, clamped` | Server asked for < 10 s; the agent enforces the 10 s floor (ethics rule). |182| `batch spooled … server unavailable` | Normal during an outage: `ls /var/lib/internetpressure/spool`; drained oldest-first when the server returns. |183| `local clock differs from server by more than 120 s` | Enable NTP; signatures still pass because timestamps are offset-corrected, but measurements are timestamped with the local clock. |184| Debug a target | `ip-probe once --target host.example --url https://host.example/path` |185| Where do logs go? | macOS `/var/log/internetpressure-probe.log`; Linux `journalctl -u ip-probe -f`. JSON lines, `log_level: debug` for per-batch details. |186187## Notes on the protocol implementation188189* Every wire field name/type follows `docs/PROBE-PROTOCOL.md`. Fields not relevant to a `kind` are omitted190  (pointer + `omitempty`), which the protocol allows ("omitted or null"); `total_ms`/`rtt_ms` in traceroutes are191  explicit `null` when unknown.192* `ttfb_ms` is measured from the start of the check (before DNS), i.e. the conventional time-to-first-byte, so193  `dns_ms + tcp_ms + tls_ms ≤ ttfb_ms ≤ total_ms` as in the protocol's example values.194* `http_4xx` is reported with `ok=true` (connection + TLS fine, status < 500), per the `ok` definition.195* On the ICMP → TCP fallback the measurement is `kind:"tcp"` and, when at least one connect succeeded,196  `error:"icmp_unavailable"` with `ok:true` — an informational code so the server can tell a real `tcp` check197  from a degraded `ping`.198* `rss_mb` is computed from `runtime.MemStats` (`Sys − HeapReleased`), a close upper bound of the true RSS.199