0004 — Agents connect outbound over the LAN with mTLS; commands are signed by the orchestrator
Context (observed). The public entry (Caddy on the gateway) terminates TLS, so client certificates cannot reach the backend through it. 12 Macs have no WireGuard tunnel; all are on one LAN whose DHCP addresses have already been renumbered once.
Decision.
telemetry-ingestlistens on the host node (LAN + tunnel address), TLS 1.3, client certificates issued by a private CA created by the product. It is never routed by the public gateway.- Agents open outbound connections only; they listen on nothing. They try a list of endpoints and authenticate the server by certificate (fixed name
ingest.mcc.internal), not by address. - Enrollment uses a single-use token (15 min) and a CSR; the private key never leaves the node. Certificates last 30 days and renew over mTLS.
- Commands are signed (Ed25519) by the orchestrator, which holds the only signing key. The ingest service merely transports them: if it is compromised it can drop or delay commands but cannot forge or alter one. Agents verify signature, recipient, expiry (≤ 120 s) and a persisted nonce, then apply their local allowlist.
Consequences. A node outside the LAN (the Linux gateway) reaches ingest through its tunnel address. A revoked agent is cut off within seconds (certificate serial checked per request).
Amendment (2026-11-18) — signed HTTPS through the public origin is the default transport
Measured. On node m4ml (macOS 26.7), a Go binary started as a LaunchAgent gets connect: no route to host when it
dials LAN peers (192.168.2.x), while the same binary succeeds from an SSH session. From the LaunchAgent the Internet
and the WireGuard hub are reachable. This is macOS Local Network Privacy: consent can only be given in the graphical
session of each Mac, per binary, and is not available to launchd daemons. Direct-LAN mTLS is therefore unusable for a
launchd-started agent on these machines.
Decision. Agents talk to https://clustr.spboucher.ai/agent/v1/… (public address, not subject to that
restriction, reachable from every node including those without a tunnel). Because the gateway terminates TLS, client
certificates cannot be used at the TLS layer; instead every request carries a proof of possession of the agent's
enrolled certificate key: the certificate, a timestamp, a nonce and an ECDSA signature over method, URI, timestamp,
nonce and body hash. telemetry-ingest verifies chain, validity, revocation, freshness, replay and signature before
processing. control-api only relays /agent/v1/ to ingest on loopback and strips user credentials; it cannot forge
an agent request (no agent key) nor a command (no orchestrator key). The enrollment CLI still checks the private CA
fingerprint returned at enrollment. The mTLS listener remains available as an alternative endpoint type for
environments where direct connections are permitted.
Trade-off. Telemetry now depends on the Internet path to the gateway. During an outage agents buffer up to 6 hours and replay in order; nothing is lost within that window, and the dashboard shows nodes as stale/offline — which is also what the operator would experience, since the UI uses the same entry point.