SPB Git

Repository Public

AGENT-PROTOCOL.md 20.5 KB

# Agent protocol (v1)

Wire protocol between mcc-agent (on every node) and telemetry-ingest. Types are defined once, in services/internal/protocol (report.go, metrics.go, envelope.go, reqsig.go), and shared by the agent, the ingest service and the orchestrator. This document describes behaviour; the Go types are the schema.

# 1. Principles

  • The agent never listens on a port and never runs as root. Every connection is outbound.
  • The agent's private key (ECDSA P-256) is generated on the node and never leaves it.
  • The ingest service authenticates agents but cannot command them: commands are signed (Ed25519) by the orchestrator, verified by the agent, then checked against the node owner's local policy. A compromised ingest service, proxy or network can delay or drop commands; it cannot forge, alter or replay one.
  • A value that cannot be measured is declared (availability), never invented.

# 2. Two transports, one protocol

Both transports serve exactly the same routes (/agent/v1/*) and the same handlers; only the way an agent is authenticated differs. An endpoint string selects the transport:

Endpoint form Transport Server authentication Agent authentication
https://host[:port] (URL) signed requests over ordinary HTTPS — default in production WebPKI (system roots, TLS ≥ 1.2) proof of possession of the enrolled key on every request (headers below)
host:port direct mTLS, TLS 1.3 only private CA, fixed server name ingest.mcc.internal client certificate, checked against the agents table on every request

Why two. Measured on a real node (macOS 26.7): a binary started by launchd gets no route to host towards LAN peers (macOS Local Network Privacy; it cannot be granted headlessly), while the public Internet and the WireGuard hub stay reachable. Direct mTLS on the LAN therefore does not work for a launchd-managed agent. In production agents use https://clustr.spboucher.ai; control-api reverse-proxies /agent/v1/ to the signed listener of the ingest service (MCC_INGEST_SIGNED_LISTEN, default 127.0.0.1:8734, plain HTTP, loopback). TLS ends at the public front, so a client certificate cannot authenticate the connection: the signature does, per request. The mTLS listener (MCC_INGEST_LISTEN, default 127.0.0.1:8731) remains available as an optional endpoint type.

http:// endpoints are accepted for loopback addresses only (development, tests).

EnrollResponse.endpoints is an ordered list mixing both forms (taken verbatim from MCC_INGEST_ADVERTISE). The agent uses one endpoint at a time; a transport failure, a 5xx or a redirect moves it to the next one for the following call; the last endpoint that answered is remembered on disk (last-endpoint) and tried first after a restart.

# 2.1 Signed-request headers

Header Value
X-MCC-Agent-Cert base64 (std) of the DER client certificate
X-MCC-Agent-Ts unix time in milliseconds
X-MCC-Agent-Nonce 16 random bytes, base64url without padding
X-MCC-Agent-Sig base64 (std) ECDSA-P256-SHA256 signature (ASN.1 DER)

The signature covers exactly these bytes (protocol.RequestSigningString):

text
mcc-agent-request-v1 \n METHOD \n request-URI \n ts \n nonce \n lowercase-hex(SHA-256(body))

request-URI is the path plus ? and the raw query when there is one; body is the bytes sent on the wire (after gzip when the body is compressed; empty for GET).

Server checks, in this order, before the body is parsed (it is only read, size-capped at 1 MiB, to be hashed): headers present and well-formed → certificate chains to the agent CA with clientAuth usage and is within validity → CN / SAN URI name an agent → certificate serial belongs to a non-revoked row of agents (lookup cached ≤ 5 s, same cache as mTLS) → |now − ts| ≤ 90 s → signature → nonce not already used by this agent. Any failure is 401 with a machine-readable code, and nothing is written:

signature_headers_missing, certificate_invalid, agent_revoked, timestamp_out_of_window, signature_invalid, nonce_replayed.

Failures are counted per client IP + presented certificate (30 per minute, then 429), so one misbehaving agent behind a shared address does not lock out the others. The client IP comes from X-Forwarded-For only when the TCP peer is loopback (the front).

Replay protection and its limit: the nonce cache is in memory (per agent, kept 190 s, bounded). A restart of the ingest service empties it, which reduces the protection to the ±90 s timestamp window. This is acceptable because every agent call is idempotent — reports by (agent, seq), command lifecycle calls by command id and state — and commands carry their own Ed25519 signature and nonce, verified on the node (§7).

# 2.2 mTLS

ClientAuth = VerifyClientCertIfGiven with the private CA pool: a certificate from another CA fails the handshake; a request without certificate reaches only /enroll (others answer 401 client_certificate_required). The server certificate is issued at start by the same CA (SAN DNS = MCC_INGEST_SERVER_NAME, 72 h, key in memory only, re-issued automatically). Server certificates carry serverAuth only and agent certificates clientAuth only: a stolen agent key cannot impersonate the service.

# 3. PKI

Private CA, ECDSA P-256, created on first start in MCC_SECRETS_DIR (agent-ca.key, agent-ca.crt, files 0600, directory 0700). The CA certificate, its sha256:<hex> fingerprint (SHA-256 of the DER certificate), the server name and the advertised endpoints are published in settings('pki.ca') for the control API (install hint).

Agent certificates: 30 days, CN = agent id, SAN URI mcc://agent/<agent-id>, ExtKeyUsage clientAuth only. The subject and SANs of the CSR are ignored: the CA decides the identity. Only P-256 keys are signed.

# 4. Enrollment — POST /agent/v1/enroll

text
mcc-agent enroll --endpoints https://clustr.spboucher.ai[,host:port…] --token <token> --ca-fingerprint sha256:<hex>

Request: header X-MCC-Enroll-Token, body EnrollRequest {csr_pem, inventory, agent_version} (≤ 256 KiB). Response: EnrollResponse {agent_id, node_id, cert_pem, ca_pem, command_keys, endpoints, server_name, interval_s}.

  • The token is single-use (15 min by default, only its SHA-256 is stored) and is consumed atomically (UPDATE … WHERE consumed_at IS NULL AND revoked_at IS NULL AND expires_at > now() RETURNING): of N concurrent presentations exactly one succeeds. Rate limit: 6 per minute per client IP.
  • Binding to the expected control plane: on a host:port endpoint the CA is pinned during the TLS handshake (the server chain must contain the certificate whose SHA-256 is --ca-fingerprint, and the leaf must verify under it). On a URL endpoint TLS is ordinary WebPKI and the agent compares --ca-fingerprint with the SHA-256 of ca_pem in the response. In both cases a mismatch aborts before anything is written to the state directory.
  • Node resolution: by hardware_uuid (IOPlatformUUID / machine-id) first — if that hardware is already registered under another alias, the existing node and alias are kept and the fact is written in the audit detail; otherwise by the token's alias — a manually pre-registered node (hardware_uuid NULL) is adopted; an alias owned by different hardware is refused (409 alias_in_use, token not consumed); otherwise the node is created (registered_via='agent'). A retired node is refused (409 node_retired).
  • Re-enrollment revokes the previous active agent of the node (revoked_reason='re-enrolled').
  • Audit: agent.enrolled (actor_kind agent); refusals: agent.enroll_denied.

State directory (~/.mcc-agent, 0700; files 0600): identity.json, agent.key (key + certificate, replaced atomically together), agent.crt, ca.crt, command-keys.json, seq, spool/, nonces.log, commands.journal, status.json, last-endpoint, optional policy.json and FREEZE.

# 5. Renewal — POST /agent/v1/renew

Body RenewRequest {csr_pem} (a new key), answer RenewResponse {cert_pem, ca_pem, command_keys}. Triggered by ReportAck.renew_cert (server: less than 15 days left) or by the agent itself at half of the certificate's life, at most one attempt per 10 minutes. The previous certificate stays accepted for 24 h (or until its own expiry), so a lost response cannot lock the agent out. The CA is never replaced by a renewal. Renewal is also how an agent learns rotated command keys (it is attempted when a command names an unknown key id). Audit: agent.cert_renewed.

Revocation: agents.revoked_at is checked on every request (cache ≤ 5 s) and every few seconds on open long-polls; a revoked agent gets 401 agent_revoked within seconds, keeps buffering (bounded) and retries slowly.

# 6. Reports — POST /agent/v1/report

Body: protocol.Report, gzip accepted (≤ 1 MiB on the wire, ≤ 4 MiB decompressed). Every 10 s by default (ReportAck.interval_s, 5–300 s).

text
{ v, seq, ts, agent_version, buffered?, samples[{m,l?,v}], availability{metric: state}, processes?[],
  services?{managers[], items[]}, inventory?, volumes?[], inference?{items[]}, nas?[], capabilities?[] }
  • seq increases for the life of an identity and survives restarts (reserved on disk in blocks of 64: gaps are normal, reuse never happens). Idempotence: (agent, seq) is recorded in agent_report_seqs; a duplicate is acknowledged (200) and not applied again. agents.last_seq is the highest seen.
  • Strict validation, whole report rejected with 400 (the agent drops it — it would never succeed): unknown metric (unknown_metric), non-finite value, label rules (labeled ⇔ label present, ≤ 64 bytes, no control characters), duplicate (metric, label), cardinality (≤ 8 labels for net.*, ≤ 12 for disk.*, ≤ 400 samples, ≤ 15 processes, ≤ 200 services, ≤ 12 volumes), availability states, timestamp more than 5 min in the future (clock_skew) or older than the raw retention (too_old), credentials in a volume remote. Unknown JSON fields are ignored (forward compatibility).
  • One transaction per report: samples (one multi-row insert, ON CONFLICT DO NOTHING), node_state, node and agent bookkeeping, services, inference servers, volumes, inventory, NAS observations, events.
  • Live state only moves forward. A report that is buffered, or older than the stored node_state.ts, contributes samples only: it never overwrites node_state, services, volumes, inventory or NAS state.
  • node_state.values: unlabeled metric → number; labeled metric → {label: number}. Metrics have different intervals, so values present replace the stored ones, metrics listed in availability are removed, others are kept. availability is the complete map of metrics currently not measured.
  • services follows ServiceSet semantics: a manager listed in managers was observed completely — its services absent from items are marked gone (gone_at); a manager not listed was not observed and its services are left untouched. inference (when present) and volumes (when non-empty) are complete observations too.
  • capabilities of a live report is authoritative (absent = none: frozen agent or deny-all policy).
  • nas: results of the probes requested by ReportAck.nas_targets; only registered nas_targets are recorded, in nas_observations per node; nas_targets.state is the aggregate of the observations of the last 5 minutes.
  • Events (events.Publish, SSE outbox): node.state {node_id, ts, values (headline subset), availability} at most once per live report; service.changed {node_id, service_id, manager, name, state, previous_state, gone} only on an actual change; node.updated {node_id, reason} on enrollment and inventory change.

Answer ReportAck {seq, server_time, interval_s, want_inventory, nas_targets, renew_cert}.

Offline buffer: an undelivered report is written to spool/ (one fsynced file per report, atomic rename), bounded to 6 h and 20 MiB, oldest dropped first. Once a live report succeeds, buffered reports are replayed oldest first at a paced rate beside the live ones (buffered=true).

# 7. Commands

Database contract: see the header of 0002_command_contract.sql. The orchestrator inserts agent_commands (pending) and notifies mcc_commands with the agent id; the ingest service transports and records; it notifies mcc_command_updates with the command id after each change. Only the orchestrator writes actions.

Route Effect
GET /agent/v1/commands?wait=25 long-poll (≤ 55 s). Returns CommandsResponse {commands[SignedCommand], cancel[ids]}. Pending, unexpired commands become delivered. A command is handed out once; there is no redelivery.
POST /agent/v1/commands/{id}/ack CommandAck {accepted, reason}: delivered → acked, or → finished with ack_accepted=false on refusal
POST /agent/v1/commands/{id}/events one CommandOutput {stream, data} or an array; requires acked. Lines are redacted again by the server, stored in action_events (kind output), bounded to 2 000 lines / 512 KiB per command (then one truncation marker), published as action.output {action_id, command_id, node_id, lines[{id,stream,data}], truncated}
POST /agent/v1/commands/{id}/result CommandResult {status, exit_code?, error?, result?, duration_s}: → finished (accepted from acked, and from delivered for crash recovery)

An agent can only see and act on commands addressed to itself: any other id answers 404 command_not_found. Out-of-order calls answer 409 command_state.

Agent-side pipeline for each envelope, in order — a failure at any step is a refusal (accepted=false with the reason) and nothing runs:

  1. Kill switch: <state-dir>/FREEZE exists → reason frozen.
  2. protocol.Verify: known key id, Ed25519 signature over "mcc-agent-command-v1\0" || payload, recipient = this agent, lifetime ≤ 120 s, not expired, not issued in the future (30 s skew), nonce never seen. The nonce journal (nonces.log) is fsynced before acceptance and kept until the command can no longer be valid, so a replay is refused across restarts.
  3. Local policy (policy.json, §8) and strict parameter validation (unknown fields refused).
  4. Limits: at most 2 commands at a time, one per target → reason busy….
  5. Journal started (fsync), then ack accepted=true. If the acceptance cannot be delivered, the command is not executed.
  6. Execution with the timeout_s deadline, output cap, redaction; then result.

After an agent crash, a command journaled started without an end is reported failed with result.outcome_unknown=true at the next start and is never executed again.

Cancellation: ids in cancel (and job.cancel {command_id}) stop a running command cooperatively — SIGTERM to its process group, SIGKILL after 10 s — and it ends cancelled.

# 7.1 Command types and parameters

Type Parameters Notes
inventory.refresh {} fresh inventory sent with the next report
diagnostic.run {profile: basic|network|disk} fixed read-only programs (uptime, df, sw_vers/uname, vm_stat/free, netstat -rn, ifconfig/ip, gateway ping, diskutil list/lsblk)
service.health_check {manager, name, port?} manager state + optional local TCP connect; result healthy
service.logs.tail {manager, name, lines? ≤ 500} pm2 log files, launchd StandardOutPath/StandardErrorPath of a user agent, journalctl -u, docker logs; ≤ 500 lines / 256 KiB
service.start / stop / restart {manager, name} service must match the local allowlist; pm2, launchd user domain (kickstart, bootstrap, bootout), systemctl --no-ask-password, docker
agent.restart {} result delivered, then exit status 75; launchd KeepAlive / systemd Restart=always restart it
cache.purge {cache} logical name from the local policy; contents deleted through an os.Root (cannot leave the directory, symlinks never followed)
job.run {template, params{}} argv template of the local policy with typed parameters; no shell
job.cancel {command_id} cooperative stop of a running command
system.reboot / system.shutdown {} only if the policy allows it and sudo -n works; scheduled one minute ahead so the result is delivered

Every program is started without a shell, in its own process group, with a minimal environment (PATH, C locale, HOME, USER, plus the variables declared by a job template — nothing inherited from the agent).

# 8. Local policy — policy.json

The node owner's allowlist (see services/internal/agent/policy). Absent file: inventory.refresh, diagnostic.run, service.health_check, service.logs.tail. Present but invalid, or writable by group/others: every command is refused. mcc-agent policy check validates it. Contents: allow (types), services (per manager, exact names or globs), cache_roots + caches (name → absolute path that must stay under a root, also after symlink resolution), jobs (name → argv with {param} placeholders; parameters typed string (mandatory regex) / int (range) / enum / bool; values starting with - refused unless allowed; each value substituted once inside its own argv element), allow_reboot, allow_shutdown (default false). The allowed types are declared to the server as capabilities.

# 9. Redaction

Applied by the agent before sending and again by the server before storing: PEM blocks, Authorization/cookie headers and bearer tokens, NAME=value / "name": "value" where the name suggests a secret (password, token, secret, key, credential, auth), --password x style flags, URL and //user:pass@host userinfo, hex runs ≥ 32 and base64-like runs ≥ 40 characters. Process command lines are never collected; pm2 env blocks are never decoded (only PORT); inference server arguments are read only to extract --model and --port.

# 10. Server-side housekeeping

metric_catalog seeded from protocol.Catalog at start; daily partitions of samples_raw (from now − retention to now + 3 days, created on demand for buffered data), monthly partitions of samples_1m / samples_1h; rollups raw → 1 min → 1 h every 20 s under an advisory lock, driven by rollup_state watermarks, idempotent, with a trailing window and rollup_dirty flags for late (buffered) data; retention; events.Prune; heartbeat in service_heartbeats; GET /healthz on MCC_INGEST_HEALTH_LISTEN (loopback only).

# 11. Failure modes

Situation Behaviour
Ingest or network unreachable agent buffers on disk (6 h / 20 MiB), tries the next endpoint, replays without duplicates
Database down every route answers 503 (Retry-After); the service keeps running and recovers by itself; agents buffer. An outage never looks like a revocation
Report permanently invalid (400, 413) dropped by the agent and logged; the queue is not blocked
Agent revoked 401 agent_revoked within seconds; long-polls closed; the agent keeps buffering and retries slowly
Agent clock wrong signed transport: 401 timestamp_out_of_window; reports: 400 clock_skew beyond +5 min
Command not acknowledged never re-sent; it expires and the orchestrator closes the action
Agent crash during a command reported failed, outcome_unknown, never re-run
Ingest restart (signed transport) request nonce cache lost: replay window = ±90 s; harmless (idempotent calls)
Certificate renewal response lost previous certificate accepted for 24 h; the agent retries
Local policy file broken every command refused; reported by mcc-agent status

# 12. Versioning and compatibility

  • v (reports, commands) is 1; the URL prefix is /agent/v1. A report with another v is refused (400 unsupported_version); a command with another v is refused by the agent.
  • Within v1, changes are additive only: receivers ignore unknown JSON fields; optional fields may be added; no field changes meaning. New metrics get new catalog ids (ids are never reused); an agent sending a metric the server does not know has its report refused, so the server is upgraded before the agents.
  • New command types are refused by older agents (not allowed / not implemented) and are not listed in their capabilities: the orchestrator must check capabilities before sending.
  • A breaking change means v = 2 under /agent/v2, served alongside v1 during the migration.
  • The request-signature domain (mcc-agent-request-v1) and the command-signature domain (mcc-agent-command-v1) are versioned separately from the payloads.
Files main
Menu

Theme