Repository Public
1# Agent protocol (v1)23Wire protocol between `mcc-agent` (on every node) and `telemetry-ingest`. Types are defined once, in4`services/internal/protocol` (`report.go`, `metrics.go`, `envelope.go`, `reqsig.go`), and shared by the agent, the5ingest service and the orchestrator. This document describes behaviour; the Go types are the schema.67## 1. Principles89- The agent **never listens** on a port and **never runs as root**. Every connection is outbound.10- The agent's private key (ECDSA P-256) is generated on the node and never leaves it.11- The ingest service **authenticates agents** but **cannot command them**: commands are signed (Ed25519) by the12 orchestrator, verified by the agent, then checked against the node owner's local policy. A compromised ingest13 service, proxy or network can delay or drop commands; it cannot forge, alter or replay one.14- A value that cannot be measured is declared (`availability`), never invented.1516## 2. Two transports, one protocol1718Both transports serve exactly the same routes (`/agent/v1/*`) and the same handlers; only the way an agent is19authenticated differs. An endpoint string selects the transport:2021| Endpoint form | Transport | Server authentication | Agent authentication |22|---|---|---|---|23| `https://host[:port]` (URL) | **signed requests** over ordinary HTTPS — default in production | WebPKI (system roots, TLS ≥ 1.2) | proof of possession of the enrolled key on **every request** (headers below) |24| `host:port` | **direct mTLS**, TLS 1.3 only | private CA, fixed server name `ingest.mcc.internal` | client certificate, checked against the `agents` table on every request |2526Why two. Measured on a real node (macOS 26.7): a binary started by launchd gets `no route to host` towards LAN peers27(macOS Local Network Privacy; it cannot be granted headlessly), while the public Internet and the WireGuard hub stay28reachable. Direct mTLS on the LAN therefore does not work for a launchd-managed agent. In production agents use29`https://clustr.spboucher.ai`; `control-api` reverse-proxies `/agent/v1/` to the signed listener of the ingest30service (`MCC_INGEST_SIGNED_LISTEN`, default `127.0.0.1:8734`, plain HTTP, loopback). TLS ends at the public front,31so a client certificate cannot authenticate the connection: the signature does, per request. The mTLS listener32(`MCC_INGEST_LISTEN`, default `127.0.0.1:8731`) remains available as an optional endpoint type.3334`http://` endpoints are accepted for loopback addresses only (development, tests).3536`EnrollResponse.endpoints` is an ordered list mixing both forms (taken verbatim from `MCC_INGEST_ADVERTISE`). The37agent uses one endpoint at a time; a transport failure, a 5xx or a redirect moves it to the next one for the following38call; the last endpoint that answered is remembered on disk (`last-endpoint`) and tried first after a restart.3940### 2.1 Signed-request headers4142| Header | Value |43|---|---|44| `X-MCC-Agent-Cert` | base64 (std) of the DER client certificate |45| `X-MCC-Agent-Ts` | unix time in milliseconds |46| `X-MCC-Agent-Nonce` | 16 random bytes, base64url without padding |47| `X-MCC-Agent-Sig` | base64 (std) ECDSA-P256-SHA256 signature (ASN.1 DER) |4849The signature covers exactly these bytes (`protocol.RequestSigningString`):5051```52mcc-agent-request-v1 \n METHOD \n request-URI \n ts \n nonce \n lowercase-hex(SHA-256(body))53```5455`request-URI` is the path plus `?` and the raw query when there is one; `body` is the bytes sent on the wire (after56gzip when the body is compressed; empty for GET).5758Server checks, in this order, before the body is parsed (it is only read, size-capped at 1 MiB, to be hashed):59headers present and well-formed → certificate chains to the agent CA with `clientAuth` usage and is within validity60→ CN / SAN URI name an agent → certificate serial belongs to a **non-revoked** row of `agents` (lookup cached ≤ 5 s,61same cache as mTLS) → `|now − ts| ≤ 90 s` → signature → nonce not already used by this agent. Any failure is `401`62with a machine-readable code, and nothing is written:6364`signature_headers_missing`, `certificate_invalid`, `agent_revoked`, `timestamp_out_of_window`,65`signature_invalid`, `nonce_replayed`.6667Failures are counted per client IP + presented certificate (30 per minute, then `429`), so one misbehaving agent68behind a shared address does not lock out the others. The client IP comes from `X-Forwarded-For` only when the TCP69peer is loopback (the front).7071Replay protection and its limit: the nonce cache is in memory (per agent, kept 190 s, bounded). A restart of the72ingest service empties it, which reduces the protection to the ±90 s timestamp window. This is acceptable because73every agent call is idempotent — reports by `(agent, seq)`, command lifecycle calls by command id and state — and74commands carry their own Ed25519 signature and nonce, verified on the node (§7).7576### 2.2 mTLS7778`ClientAuth = VerifyClientCertIfGiven` with the private CA pool: a certificate from another CA fails the handshake;79a request without certificate reaches only `/enroll` (others answer `401 client_certificate_required`). The server80certificate is issued at start by the same CA (SAN DNS = `MCC_INGEST_SERVER_NAME`, 72 h, key in memory only,81re-issued automatically). Server certificates carry `serverAuth` only and agent certificates `clientAuth` only: a82stolen agent key cannot impersonate the service.8384## 3. PKI8586Private CA, ECDSA P-256, created on first start in `MCC_SECRETS_DIR` (`agent-ca.key`, `agent-ca.crt`, files 0600,87directory 0700). The CA certificate, its `sha256:<hex>` fingerprint (SHA-256 of the DER certificate), the server name88and the advertised endpoints are published in `settings('pki.ca')` for the control API (install hint).8990Agent certificates: 30 days, CN = agent id, SAN URI `mcc://agent/<agent-id>`, ExtKeyUsage `clientAuth` only. The91subject and SANs of the CSR are ignored: the CA decides the identity. Only P-256 keys are signed.9293## 4. Enrollment — `POST /agent/v1/enroll`9495```96mcc-agent enroll --endpoints https://clustr.spboucher.ai[,host:port…] --token <token> --ca-fingerprint sha256:<hex>97```9899Request: header `X-MCC-Enroll-Token`, body `EnrollRequest {csr_pem, inventory, agent_version}` (≤ 256 KiB).100Response: `EnrollResponse {agent_id, node_id, cert_pem, ca_pem, command_keys, endpoints, server_name, interval_s}`.101102- The token is single-use (15 min by default, only its SHA-256 is stored) and is consumed **atomically**103 (`UPDATE … WHERE consumed_at IS NULL AND revoked_at IS NULL AND expires_at > now() RETURNING`): of N concurrent104 presentations exactly one succeeds. Rate limit: 6 per minute per client IP.105- Binding to the expected control plane: on a `host:port` endpoint the CA is **pinned during the TLS handshake**106 (the server chain must contain the certificate whose SHA-256 is `--ca-fingerprint`, and the leaf must verify under107 it). On a URL endpoint TLS is ordinary WebPKI and the agent compares `--ca-fingerprint` with the SHA-256 of108 `ca_pem` in the response. In both cases a mismatch aborts before anything is written to the state directory.109- Node resolution: by `hardware_uuid` (IOPlatformUUID / machine-id) first — if that hardware is already registered110 under another alias, the existing node and alias are kept and the fact is written in the audit detail; otherwise by111 the token's alias — a manually pre-registered node (hardware_uuid NULL) is adopted; an alias owned by different112 hardware is refused (`409 alias_in_use`, token not consumed); otherwise the node is created113 (`registered_via='agent'`). A retired node is refused (`409 node_retired`).114- Re-enrollment revokes the previous active agent of the node (`revoked_reason='re-enrolled'`).115- Audit: `agent.enrolled` (actor_kind `agent`); refusals: `agent.enroll_denied`.116117State directory (`~/.mcc-agent`, 0700; files 0600): `identity.json`, `agent.key` (key + certificate, replaced118atomically together), `agent.crt`, `ca.crt`, `command-keys.json`, `seq`, `spool/`, `nonces.log`,119`commands.journal`, `status.json`, `last-endpoint`, optional `policy.json` and `FREEZE`.120121## 5. Renewal — `POST /agent/v1/renew`122123Body `RenewRequest {csr_pem}` (a **new** key), answer `RenewResponse {cert_pem, ca_pem, command_keys}`. Triggered124by `ReportAck.renew_cert` (server: less than 15 days left) or by the agent itself at half of the certificate's life,125at most one attempt per 10 minutes. The previous certificate stays accepted for 24 h (or until its own expiry), so126a lost response cannot lock the agent out. The CA is never replaced by a renewal. Renewal is also how an agent127learns rotated command keys (it is attempted when a command names an unknown key id). Audit: `agent.cert_renewed`.128129Revocation: `agents.revoked_at` is checked on every request (cache ≤ 5 s) and every few seconds on open130long-polls; a revoked agent gets `401 agent_revoked` within seconds, keeps buffering (bounded) and retries slowly.131132## 6. Reports — `POST /agent/v1/report`133134Body: `protocol.Report`, gzip accepted (≤ 1 MiB on the wire, ≤ 4 MiB decompressed). Every 10 s by default135(`ReportAck.interval_s`, 5–300 s).136137```138{ v, seq, ts, agent_version, buffered?, samples[{m,l?,v}], availability{metric: state}, processes?[],139 services?{managers[], items[]}, inventory?, volumes?[], inference?{items[]}, nas?[], capabilities?[] }140```141142- `seq` increases for the life of an identity and survives restarts (reserved on disk in blocks of 64: gaps are143 normal, reuse never happens). **Idempotence**: `(agent, seq)` is recorded in `agent_report_seqs`; a duplicate is144 acknowledged (`200`) and not applied again. `agents.last_seq` is the highest seen.145- Strict validation, whole report rejected with `400` (the agent drops it — it would never succeed): unknown metric146 (`unknown_metric`), non-finite value, label rules (labeled ⇔ label present, ≤ 64 bytes, no control characters),147 duplicate `(metric, label)`, cardinality (≤ 8 labels for `net.*`, ≤ 12 for `disk.*`, ≤ 400 samples, ≤ 15148 processes, ≤ 200 services, ≤ 12 volumes), availability states, timestamp more than 5 min in the future149 (`clock_skew`) or older than the raw retention (`too_old`), credentials in a volume `remote`. Unknown JSON fields150 are ignored (forward compatibility).151- One transaction per report: samples (one multi-row insert, `ON CONFLICT DO NOTHING`), `node_state`, node and agent152 bookkeeping, services, inference servers, volumes, inventory, NAS observations, events.153- **Live state only moves forward.** A report that is `buffered`, or older than the stored `node_state.ts`,154 contributes samples only: it never overwrites `node_state`, services, volumes, inventory or NAS state.155- `node_state.values`: unlabeled metric → number; labeled metric → `{label: number}`. Metrics have different156 intervals, so values present replace the stored ones, metrics listed in `availability` are removed, others are157 kept. `availability` is the complete map of metrics currently not measured.158- `services` follows `ServiceSet` semantics: a manager listed in `managers` was observed completely — its services159 absent from `items` are marked gone (`gone_at`); a manager not listed was not observed and its services are left160 untouched. `inference` (when present) and `volumes` (when non-empty) are complete observations too.161- `capabilities` of a live report is authoritative (absent = none: frozen agent or deny-all policy).162- `nas`: results of the probes requested by `ReportAck.nas_targets`; only registered `nas_targets` are recorded, in163 `nas_observations` per node; `nas_targets.state` is the aggregate of the observations of the last 5 minutes.164- Events (`events.Publish`, SSE outbox): `node.state` `{node_id, ts, values (headline subset), availability}` at165 most once per live report; `service.changed` `{node_id, service_id, manager, name, state, previous_state, gone}`166 only on an actual change; `node.updated` `{node_id, reason}` on enrollment and inventory change.167168Answer `ReportAck {seq, server_time, interval_s, want_inventory, nas_targets, renew_cert}`.169170Offline buffer: an undelivered report is written to `spool/` (one fsynced file per report, atomic rename), bounded171to 6 h and 20 MiB, oldest dropped first. Once a live report succeeds, buffered reports are replayed oldest first at a172paced rate beside the live ones (`buffered=true`).173174## 7. Commands175176Database contract: see the header of `0002_command_contract.sql`. The orchestrator inserts `agent_commands`177(`pending`) and notifies `mcc_commands` with the agent id; the ingest service transports and records; it notifies178`mcc_command_updates` with the command id after each change. Only the orchestrator writes `actions`.179180| Route | Effect |181|---|---|182| `GET /agent/v1/commands?wait=25` | long-poll (≤ 55 s). Returns `CommandsResponse {commands[SignedCommand], cancel[ids]}`. Pending, unexpired commands become `delivered`. **A command is handed out once; there is no redelivery.** |183| `POST /agent/v1/commands/{id}/ack` | `CommandAck {accepted, reason}`: `delivered → acked`, or `→ finished` with `ack_accepted=false` on refusal |184| `POST /agent/v1/commands/{id}/events` | one `CommandOutput {stream, data}` or an array; requires `acked`. Lines are redacted again by the server, stored in `action_events` (kind `output`), bounded to 2 000 lines / 512 KiB per command (then one truncation marker), published as `action.output` `{action_id, command_id, node_id, lines[{id,stream,data}], truncated}` |185| `POST /agent/v1/commands/{id}/result` | `CommandResult {status, exit_code?, error?, result?, duration_s}`: `→ finished` (accepted from `acked`, and from `delivered` for crash recovery) |186187An agent can only see and act on commands addressed to itself: any other id answers `404 command_not_found`.188Out-of-order calls answer `409 command_state`.189190Agent-side pipeline for each envelope, in order — a failure at any step is a refusal (`accepted=false` with the191reason) and nothing runs:1921931. Kill switch: `<state-dir>/FREEZE` exists → reason `frozen`.1942. `protocol.Verify`: known key id, Ed25519 signature over `"mcc-agent-command-v1\0" || payload`, recipient =195 this agent, lifetime ≤ 120 s, not expired, not issued in the future (30 s skew), **nonce never seen**. The nonce196 journal (`nonces.log`) is fsynced before acceptance and kept until the command can no longer be valid, so a197 replay is refused across restarts.1983. Local policy (`policy.json`, §8) and strict parameter validation (unknown fields refused).1994. Limits: at most 2 commands at a time, one per target → reason `busy…`.2005. Journal `started` (fsync), then `ack accepted=true`. If the acceptance cannot be delivered, the command is not201 executed.2026. Execution with the `timeout_s` deadline, output cap, redaction; then `result`.203204After an agent crash, a command journaled `started` without an end is reported `failed` with205`result.outcome_unknown=true` at the next start and is **never executed again**.206207Cancellation: ids in `cancel` (and `job.cancel {command_id}`) stop a running command cooperatively — SIGTERM to its208process group, SIGKILL after 10 s — and it ends `cancelled`.209210### 7.1 Command types and parameters211212| Type | Parameters | Notes |213|---|---|---|214| `inventory.refresh` | `{}` | fresh inventory sent with the next report |215| `diagnostic.run` | `{profile: basic\|network\|disk}` | fixed read-only programs (`uptime`, `df`, `sw_vers`/`uname`, `vm_stat`/`free`, `netstat -rn`, `ifconfig`/`ip`, gateway ping, `diskutil list`/`lsblk`) |216| `service.health_check` | `{manager, name, port?}` | manager state + optional local TCP connect; result `healthy` |217| `service.logs.tail` | `{manager, name, lines? ≤ 500}` | pm2 log files, launchd `StandardOutPath`/`StandardErrorPath` of a user agent, `journalctl -u`, `docker logs`; ≤ 500 lines / 256 KiB |218| `service.start` / `stop` / `restart` | `{manager, name}` | service must match the local allowlist; pm2, launchd user domain (`kickstart`, `bootstrap`, `bootout`), `systemctl --no-ask-password`, docker |219| `agent.restart` | `{}` | result delivered, then exit status **75**; launchd `KeepAlive` / systemd `Restart=always` restart it |220| `cache.purge` | `{cache}` | logical name from the local policy; contents deleted through an `os.Root` (cannot leave the directory, symlinks never followed) |221| `job.run` | `{template, params{}}` | argv template of the local policy with typed parameters; no shell |222| `job.cancel` | `{command_id}` | cooperative stop of a running command |223| `system.reboot` / `system.shutdown` | `{}` | only if the policy allows it **and** `sudo -n` works; scheduled one minute ahead so the result is delivered |224225Every program is started without a shell, in its own process group, with a minimal environment (`PATH`, C locale,226`HOME`, `USER`, plus the variables declared by a job template — nothing inherited from the agent).227228## 8. Local policy — `policy.json`229230The node owner's allowlist (see `services/internal/agent/policy`). Absent file: `inventory.refresh`,231`diagnostic.run`, `service.health_check`, `service.logs.tail`. Present but invalid, or writable by group/others:232**every command is refused**. `mcc-agent policy check` validates it. Contents: `allow` (types), `services` (per233manager, exact names or globs), `cache_roots` + `caches` (name → absolute path that must stay under a root, also234after symlink resolution), `jobs` (name → `argv` with `{param}` placeholders; parameters typed235`string` (mandatory regex) / `int` (range) / `enum` / `bool`; values starting with `-` refused unless allowed;236each value substituted once inside its own argv element), `allow_reboot`, `allow_shutdown` (default false).237The allowed types are declared to the server as `capabilities`.238239## 9. Redaction240241Applied by the agent before sending and again by the server before storing: PEM blocks, `Authorization`/cookie242headers and bearer tokens, `NAME=value` / `"name": "value"` where the name suggests a secret (password, token,243secret, key, credential, auth), `--password x` style flags, URL and `//user:pass@host` userinfo, hex runs ≥ 32 and244base64-like runs ≥ 40 characters. Process command lines are never collected; pm2 `env` blocks are never decoded245(only `PORT`); inference server arguments are read only to extract `--model` and `--port`.246247## 10. Server-side housekeeping248249`metric_catalog` seeded from `protocol.Catalog` at start; daily partitions of `samples_raw` (from now − retention to250now + 3 days, created on demand for buffered data), monthly partitions of `samples_1m` / `samples_1h`; rollups251raw → 1 min → 1 h every 20 s under an advisory lock, driven by `rollup_state` watermarks, idempotent, with a252trailing window and `rollup_dirty` flags for late (buffered) data; retention; `events.Prune`; heartbeat in253`service_heartbeats`; `GET /healthz` on `MCC_INGEST_HEALTH_LISTEN` (loopback only).254255## 11. Failure modes256257| Situation | Behaviour |258|---|---|259| Ingest or network unreachable | agent buffers on disk (6 h / 20 MiB), tries the next endpoint, replays without duplicates |260| Database down | every route answers `503` (`Retry-After`); the service keeps running and recovers by itself; agents buffer. An outage never looks like a revocation |261| Report permanently invalid (`400`, `413`) | dropped by the agent and logged; the queue is not blocked |262| Agent revoked | `401 agent_revoked` within seconds; long-polls closed; the agent keeps buffering and retries slowly |263| Agent clock wrong | signed transport: `401 timestamp_out_of_window`; reports: `400 clock_skew` beyond +5 min |264| Command not acknowledged | never re-sent; it expires and the orchestrator closes the action |265| Agent crash during a command | reported `failed`, `outcome_unknown`, never re-run |266| Ingest restart (signed transport) | request nonce cache lost: replay window = ±90 s; harmless (idempotent calls) |267| Certificate renewal response lost | previous certificate accepted for 24 h; the agent retries |268| Local policy file broken | every command refused; reported by `mcc-agent status` |269270## 12. Versioning and compatibility271272- `v` (reports, commands) is `1`; the URL prefix is `/agent/v1`. A report with another `v` is refused273 (`400 unsupported_version`); a command with another `v` is refused by the agent.274- Within v1, changes are additive only: receivers ignore unknown JSON fields; optional fields may be added; no field275 changes meaning. New metrics get new catalog ids (ids are never reused); an agent sending a metric the server does276 not know has its report refused, so **the server is upgraded before the agents**.277- New command types are refused by older agents (`not allowed` / `not implemented`) and are not listed in their278 `capabilities`: the orchestrator must check capabilities before sending.279- A breaking change means `v = 2` under `/agent/v2`, served alongside v1 during the migration.280- The request-signature domain (`mcc-agent-request-v1`) and the command-signature domain (`mcc-agent-command-v1`)281 are versioned separately from the payloads.282