Agent protocol (v1)
Wire protocol between mcc-agent (on every node) and telemetry-ingest. Types are defined once, in
services/internal/protocol (report.go, metrics.go, envelope.go, reqsig.go), and shared by the agent, the
ingest service and the orchestrator. This document describes behaviour; the Go types are the schema.
1. Principles
- The agent never listens on a port and never runs as root. Every connection is outbound.
- The agent's private key (ECDSA P-256) is generated on the node and never leaves it.
- The ingest service authenticates agents but cannot command them: commands are signed (Ed25519) by the orchestrator, verified by the agent, then checked against the node owner's local policy. A compromised ingest service, proxy or network can delay or drop commands; it cannot forge, alter or replay one.
- A value that cannot be measured is declared (
availability), never invented.
2. Two transports, one protocol
Both transports serve exactly the same routes (/agent/v1/*) and the same handlers; only the way an agent is
authenticated differs. An endpoint string selects the transport:
| Endpoint form | Transport | Server authentication | Agent authentication |
|---|---|---|---|
https://host[:port] (URL) |
signed requests over ordinary HTTPS — default in production | WebPKI (system roots, TLS ≥ 1.2) | proof of possession of the enrolled key on every request (headers below) |
host:port |
direct mTLS, TLS 1.3 only | private CA, fixed server name ingest.mcc.internal |
client certificate, checked against the agents table on every request |
Why two. Measured on a real node (macOS 26.7): a binary started by launchd gets no route to host towards LAN peers
(macOS Local Network Privacy; it cannot be granted headlessly), while the public Internet and the WireGuard hub stay
reachable. Direct mTLS on the LAN therefore does not work for a launchd-managed agent. In production agents use
https://clustr.spboucher.ai; control-api reverse-proxies /agent/v1/ to the signed listener of the ingest
service (MCC_INGEST_SIGNED_LISTEN, default 127.0.0.1:8734, plain HTTP, loopback). TLS ends at the public front,
so a client certificate cannot authenticate the connection: the signature does, per request. The mTLS listener
(MCC_INGEST_LISTEN, default 127.0.0.1:8731) remains available as an optional endpoint type.
http:// endpoints are accepted for loopback addresses only (development, tests).
EnrollResponse.endpoints is an ordered list mixing both forms (taken verbatim from MCC_INGEST_ADVERTISE). The
agent uses one endpoint at a time; a transport failure, a 5xx or a redirect moves it to the next one for the following
call; the last endpoint that answered is remembered on disk (last-endpoint) and tried first after a restart.
2.1 Signed-request headers
| Header | Value |
|---|---|
X-MCC-Agent-Cert |
base64 (std) of the DER client certificate |
X-MCC-Agent-Ts |
unix time in milliseconds |
X-MCC-Agent-Nonce |
16 random bytes, base64url without padding |
X-MCC-Agent-Sig |
base64 (std) ECDSA-P256-SHA256 signature (ASN.1 DER) |
The signature covers exactly these bytes (protocol.RequestSigningString):
mcc-agent-request-v1 \n METHOD \n request-URI \n ts \n nonce \n lowercase-hex(SHA-256(body))request-URI is the path plus ? and the raw query when there is one; body is the bytes sent on the wire (after
gzip when the body is compressed; empty for GET).
Server checks, in this order, before the body is parsed (it is only read, size-capped at 1 MiB, to be hashed):
headers present and well-formed → certificate chains to the agent CA with clientAuth usage and is within validity
→ CN / SAN URI name an agent → certificate serial belongs to a non-revoked row of agents (lookup cached ≤ 5 s,
same cache as mTLS) → |now − ts| ≤ 90 s → signature → nonce not already used by this agent. Any failure is 401
with a machine-readable code, and nothing is written:
signature_headers_missing, certificate_invalid, agent_revoked, timestamp_out_of_window,
signature_invalid, nonce_replayed.
Failures are counted per client IP + presented certificate (30 per minute, then 429), so one misbehaving agent
behind a shared address does not lock out the others. The client IP comes from X-Forwarded-For only when the TCP
peer is loopback (the front).
Replay protection and its limit: the nonce cache is in memory (per agent, kept 190 s, bounded). A restart of the
ingest service empties it, which reduces the protection to the ±90 s timestamp window. This is acceptable because
every agent call is idempotent — reports by (agent, seq), command lifecycle calls by command id and state — and
commands carry their own Ed25519 signature and nonce, verified on the node (§7).
2.2 mTLS
ClientAuth = VerifyClientCertIfGiven with the private CA pool: a certificate from another CA fails the handshake;
a request without certificate reaches only /enroll (others answer 401 client_certificate_required). The server
certificate is issued at start by the same CA (SAN DNS = MCC_INGEST_SERVER_NAME, 72 h, key in memory only,
re-issued automatically). Server certificates carry serverAuth only and agent certificates clientAuth only: a
stolen agent key cannot impersonate the service.
3. PKI
Private CA, ECDSA P-256, created on first start in MCC_SECRETS_DIR (agent-ca.key, agent-ca.crt, files 0600,
directory 0700). The CA certificate, its sha256:<hex> fingerprint (SHA-256 of the DER certificate), the server name
and the advertised endpoints are published in settings('pki.ca') for the control API (install hint).
Agent certificates: 30 days, CN = agent id, SAN URI mcc://agent/<agent-id>, ExtKeyUsage clientAuth only. The
subject and SANs of the CSR are ignored: the CA decides the identity. Only P-256 keys are signed.
4. Enrollment — POST /agent/v1/enroll
mcc-agent enroll --endpoints https://clustr.spboucher.ai[,host:port…] --token <token> --ca-fingerprint sha256:<hex>Request: header X-MCC-Enroll-Token, body EnrollRequest {csr_pem, inventory, agent_version} (≤ 256 KiB).
Response: EnrollResponse {agent_id, node_id, cert_pem, ca_pem, command_keys, endpoints, server_name, interval_s}.
- The token is single-use (15 min by default, only its SHA-256 is stored) and is consumed atomically
(
UPDATE … WHERE consumed_at IS NULL AND revoked_at IS NULL AND expires_at > now() RETURNING): of N concurrent presentations exactly one succeeds. Rate limit: 6 per minute per client IP. - Binding to the expected control plane: on a
host:portendpoint the CA is pinned during the TLS handshake (the server chain must contain the certificate whose SHA-256 is--ca-fingerprint, and the leaf must verify under it). On a URL endpoint TLS is ordinary WebPKI and the agent compares--ca-fingerprintwith the SHA-256 ofca_pemin the response. In both cases a mismatch aborts before anything is written to the state directory. - Node resolution: by
hardware_uuid(IOPlatformUUID / machine-id) first — if that hardware is already registered under another alias, the existing node and alias are kept and the fact is written in the audit detail; otherwise by the token's alias — a manually pre-registered node (hardware_uuid NULL) is adopted; an alias owned by different hardware is refused (409 alias_in_use, token not consumed); otherwise the node is created (registered_via='agent'). A retired node is refused (409 node_retired). - Re-enrollment revokes the previous active agent of the node (
revoked_reason='re-enrolled'). - Audit:
agent.enrolled(actor_kindagent); refusals:agent.enroll_denied.
State directory (~/.mcc-agent, 0700; files 0600): identity.json, agent.key (key + certificate, replaced
atomically together), agent.crt, ca.crt, command-keys.json, seq, spool/, nonces.log,
commands.journal, status.json, last-endpoint, optional policy.json and FREEZE.
5. Renewal — POST /agent/v1/renew
Body RenewRequest {csr_pem} (a new key), answer RenewResponse {cert_pem, ca_pem, command_keys}. Triggered
by ReportAck.renew_cert (server: less than 15 days left) or by the agent itself at half of the certificate's life,
at most one attempt per 10 minutes. The previous certificate stays accepted for 24 h (or until its own expiry), so
a lost response cannot lock the agent out. The CA is never replaced by a renewal. Renewal is also how an agent
learns rotated command keys (it is attempted when a command names an unknown key id). Audit: agent.cert_renewed.
Revocation: agents.revoked_at is checked on every request (cache ≤ 5 s) and every few seconds on open
long-polls; a revoked agent gets 401 agent_revoked within seconds, keeps buffering (bounded) and retries slowly.
6. Reports — POST /agent/v1/report
Body: protocol.Report, gzip accepted (≤ 1 MiB on the wire, ≤ 4 MiB decompressed). Every 10 s by default
(ReportAck.interval_s, 5–300 s).
{ v, seq, ts, agent_version, buffered?, samples[{m,l?,v}], availability{metric: state}, processes?[],
services?{managers[], items[]}, inventory?, volumes?[], inference?{items[]}, nas?[], capabilities?[] }seqincreases for the life of an identity and survives restarts (reserved on disk in blocks of 64: gaps are normal, reuse never happens). Idempotence:(agent, seq)is recorded inagent_report_seqs; a duplicate is acknowledged (200) and not applied again.agents.last_seqis the highest seen.- Strict validation, whole report rejected with
400(the agent drops it — it would never succeed): unknown metric (unknown_metric), non-finite value, label rules (labeled ⇔ label present, ≤ 64 bytes, no control characters), duplicate(metric, label), cardinality (≤ 8 labels fornet.*, ≤ 12 fordisk.*, ≤ 400 samples, ≤ 15 processes, ≤ 200 services, ≤ 12 volumes), availability states, timestamp more than 5 min in the future (clock_skew) or older than the raw retention (too_old), credentials in a volumeremote. Unknown JSON fields are ignored (forward compatibility). - One transaction per report: samples (one multi-row insert,
ON CONFLICT DO NOTHING),node_state, node and agent bookkeeping, services, inference servers, volumes, inventory, NAS observations, events. - Live state only moves forward. A report that is
buffered, or older than the storednode_state.ts, contributes samples only: it never overwritesnode_state, services, volumes, inventory or NAS state. node_state.values: unlabeled metric → number; labeled metric →{label: number}. Metrics have different intervals, so values present replace the stored ones, metrics listed inavailabilityare removed, others are kept.availabilityis the complete map of metrics currently not measured.servicesfollowsServiceSetsemantics: a manager listed inmanagerswas observed completely — its services absent fromitemsare marked gone (gone_at); a manager not listed was not observed and its services are left untouched.inference(when present) andvolumes(when non-empty) are complete observations too.capabilitiesof a live report is authoritative (absent = none: frozen agent or deny-all policy).nas: results of the probes requested byReportAck.nas_targets; only registerednas_targetsare recorded, innas_observationsper node;nas_targets.stateis the aggregate of the observations of the last 5 minutes.- Events (
events.Publish, SSE outbox):node.state{node_id, ts, values (headline subset), availability}at most once per live report;service.changed{node_id, service_id, manager, name, state, previous_state, gone}only on an actual change;node.updated{node_id, reason}on enrollment and inventory change.
Answer ReportAck {seq, server_time, interval_s, want_inventory, nas_targets, renew_cert}.
Offline buffer: an undelivered report is written to spool/ (one fsynced file per report, atomic rename), bounded
to 6 h and 20 MiB, oldest dropped first. Once a live report succeeds, buffered reports are replayed oldest first at a
paced rate beside the live ones (buffered=true).
7. Commands
Database contract: see the header of 0002_command_contract.sql. The orchestrator inserts agent_commands
(pending) and notifies mcc_commands with the agent id; the ingest service transports and records; it notifies
mcc_command_updates with the command id after each change. Only the orchestrator writes actions.
| Route | Effect |
|---|---|
GET /agent/v1/commands?wait=25 |
long-poll (≤ 55 s). Returns CommandsResponse {commands[SignedCommand], cancel[ids]}. Pending, unexpired commands become delivered. A command is handed out once; there is no redelivery. |
POST /agent/v1/commands/{id}/ack |
CommandAck {accepted, reason}: delivered → acked, or → finished with ack_accepted=false on refusal |
POST /agent/v1/commands/{id}/events |
one CommandOutput {stream, data} or an array; requires acked. Lines are redacted again by the server, stored in action_events (kind output), bounded to 2 000 lines / 512 KiB per command (then one truncation marker), published as action.output {action_id, command_id, node_id, lines[{id,stream,data}], truncated} |
POST /agent/v1/commands/{id}/result |
CommandResult {status, exit_code?, error?, result?, duration_s}: → finished (accepted from acked, and from delivered for crash recovery) |
An agent can only see and act on commands addressed to itself: any other id answers 404 command_not_found.
Out-of-order calls answer 409 command_state.
Agent-side pipeline for each envelope, in order — a failure at any step is a refusal (accepted=false with the
reason) and nothing runs:
- Kill switch:
<state-dir>/FREEZEexists → reasonfrozen. protocol.Verify: known key id, Ed25519 signature over"mcc-agent-command-v1\0" || payload, recipient = this agent, lifetime ≤ 120 s, not expired, not issued in the future (30 s skew), nonce never seen. The nonce journal (nonces.log) is fsynced before acceptance and kept until the command can no longer be valid, so a replay is refused across restarts.- Local policy (
policy.json, §8) and strict parameter validation (unknown fields refused). - Limits: at most 2 commands at a time, one per target → reason
busy…. - Journal
started(fsync), thenack accepted=true. If the acceptance cannot be delivered, the command is not executed. - Execution with the
timeout_sdeadline, output cap, redaction; thenresult.
After an agent crash, a command journaled started without an end is reported failed with
result.outcome_unknown=true at the next start and is never executed again.
Cancellation: ids in cancel (and job.cancel {command_id}) stop a running command cooperatively — SIGTERM to its
process group, SIGKILL after 10 s — and it ends cancelled.
7.1 Command types and parameters
| Type | Parameters | Notes |
|---|---|---|
inventory.refresh |
{} |
fresh inventory sent with the next report |
diagnostic.run |
{profile: basic|network|disk} |
fixed read-only programs (uptime, df, sw_vers/uname, vm_stat/free, netstat -rn, ifconfig/ip, gateway ping, diskutil list/lsblk) |
service.health_check |
{manager, name, port?} |
manager state + optional local TCP connect; result healthy |
service.logs.tail |
{manager, name, lines? ≤ 500} |
pm2 log files, launchd StandardOutPath/StandardErrorPath of a user agent, journalctl -u, docker logs; ≤ 500 lines / 256 KiB |
service.start / stop / restart |
{manager, name} |
service must match the local allowlist; pm2, launchd user domain (kickstart, bootstrap, bootout), systemctl --no-ask-password, docker |
agent.restart |
{} |
result delivered, then exit status 75; launchd KeepAlive / systemd Restart=always restart it |
cache.purge |
{cache} |
logical name from the local policy; contents deleted through an os.Root (cannot leave the directory, symlinks never followed) |
job.run |
{template, params{}} |
argv template of the local policy with typed parameters; no shell |
job.cancel |
{command_id} |
cooperative stop of a running command |
system.reboot / system.shutdown |
{} |
only if the policy allows it and sudo -n works; scheduled one minute ahead so the result is delivered |
Every program is started without a shell, in its own process group, with a minimal environment (PATH, C locale,
HOME, USER, plus the variables declared by a job template — nothing inherited from the agent).
8. Local policy — policy.json
The node owner's allowlist (see services/internal/agent/policy). Absent file: inventory.refresh,
diagnostic.run, service.health_check, service.logs.tail. Present but invalid, or writable by group/others:
every command is refused. mcc-agent policy check validates it. Contents: allow (types), services (per
manager, exact names or globs), cache_roots + caches (name → absolute path that must stay under a root, also
after symlink resolution), jobs (name → argv with {param} placeholders; parameters typed
string (mandatory regex) / int (range) / enum / bool; values starting with - refused unless allowed;
each value substituted once inside its own argv element), allow_reboot, allow_shutdown (default false).
The allowed types are declared to the server as capabilities.
9. Redaction
Applied by the agent before sending and again by the server before storing: PEM blocks, Authorization/cookie
headers and bearer tokens, NAME=value / "name": "value" where the name suggests a secret (password, token,
secret, key, credential, auth), --password x style flags, URL and //user:pass@host userinfo, hex runs ≥ 32 and
base64-like runs ≥ 40 characters. Process command lines are never collected; pm2 env blocks are never decoded
(only PORT); inference server arguments are read only to extract --model and --port.
10. Server-side housekeeping
metric_catalog seeded from protocol.Catalog at start; daily partitions of samples_raw (from now − retention to
now + 3 days, created on demand for buffered data), monthly partitions of samples_1m / samples_1h; rollups
raw → 1 min → 1 h every 20 s under an advisory lock, driven by rollup_state watermarks, idempotent, with a
trailing window and rollup_dirty flags for late (buffered) data; retention; events.Prune; heartbeat in
service_heartbeats; GET /healthz on MCC_INGEST_HEALTH_LISTEN (loopback only).
11. Failure modes
| Situation | Behaviour |
|---|---|
| Ingest or network unreachable | agent buffers on disk (6 h / 20 MiB), tries the next endpoint, replays without duplicates |
| Database down | every route answers 503 (Retry-After); the service keeps running and recovers by itself; agents buffer. An outage never looks like a revocation |
Report permanently invalid (400, 413) |
dropped by the agent and logged; the queue is not blocked |
| Agent revoked | 401 agent_revoked within seconds; long-polls closed; the agent keeps buffering and retries slowly |
| Agent clock wrong | signed transport: 401 timestamp_out_of_window; reports: 400 clock_skew beyond +5 min |
| Command not acknowledged | never re-sent; it expires and the orchestrator closes the action |
| Agent crash during a command | reported failed, outcome_unknown, never re-run |
| Ingest restart (signed transport) | request nonce cache lost: replay window = ±90 s; harmless (idempotent calls) |
| Certificate renewal response lost | previous certificate accepted for 24 h; the agent retries |
| Local policy file broken | every command refused; reported by mcc-agent status |
12. Versioning and compatibility
v(reports, commands) is1; the URL prefix is/agent/v1. A report with anothervis refused (400 unsupported_version); a command with anothervis refused by the agent.- Within v1, changes are additive only: receivers ignore unknown JSON fields; optional fields may be added; no field changes meaning. New metrics get new catalog ids (ids are never reused); an agent sending a metric the server does not know has its report refused, so the server is upgraded before the agents.
- New command types are refused by older agents (
not allowed/not implemented) and are not listed in theircapabilities: the orchestrator must check capabilities before sending. - A breaking change means
v = 2under/agent/v2, served alongside v1 during the migration. - The request-signature domain (
mcc-agent-request-v1) and the command-signature domain (mcc-agent-command-v1) are versioned separately from the payloads.