SPB Git

Repository Public

0010-at-most-once-commands.md 3.6 KB 31 lines markdown

1# 0010 — At-most-once command delivery; an unknown outcome is a state, never a retry23**Status.** Accepted. Recorded on 2026-11-18 from the implementation: the rule was stated in the design brief (§4) and in ADR 0009 ("actions are never re-issued after a crash") but had no decision record of its own.45**Context.** Actions change real machines (restart a service, purge a cache, reboot a node). Between the orchestrator and an agent there are a database, a transport service, a public gateway and a network that can each fail after a command has left and before its result has come back. In that situation the platform cannot know whether the command ran.67**Options.** (a) At-least-once: re-send until acknowledged, and require every command to be idempotent on the node. (b) At-most-once: send once, and report honestly when the outcome is not known. (c) Exactly-once through a two-phase protocol with the agent.89**Decision.** At-most-once, enforced at three levels.1011- *Schema.* One action owns at most one command: unique index `agent_commands_one_per_action` (`services/internal/db/migrations/0006_ops.sql`). The command row, the `queued → dispatched` transition and the deadline are written in one transaction (`dispatchOne`, `services/internal/orchestrator/orchestrator.go`).12- *Transport.* `telemetry-ingest` hands a command out once (`pending → delivered`); there is no redelivery (`services/internal/ingest/commands.go`).13- *Agent.* A signed envelope is accepted once: its nonce is fsynced to `nonces.log` before acceptance, and a command journaled `started` without an end is reported `failed` with `outcome_unknown=true` at the next start, never executed again (`services/internal/agent/nonce`, `services/internal/agent/journal`).1415Outcomes are classified by what is actually known:1617| Situation | Final state | Meaning |18|---|---|---|19| never delivered before `expires_at` | `failed`, `error_code=agent_unreachable` | it certainly did not run |20| refused by the agent (policy, signature, freeze, busy) | `failed`, `error_code=agent_refused` | it did not run |21| delivered, no result before the deadline | `timed_out`, `result.outcome_unknown=true` | it may or may not have run; it will not be sent again |22| result arrives after the action was closed | state unchanged; the late result is recorded next to it | history is not rewritten |2324Retrying is a decision of a person (a new action, new idempotency key) or of the job runner under its own bounded policy, which retries only when the previous outcome is known and nothing is left running (`services/internal/jobs/runner.go`).2526**Why not the others.** (a) would require proving every command type idempotent on every service manager, and a duplicated `service.stop` or `system.reboot` is exactly the incident this product must not cause. (c) cannot remove the uncertainty either — the last message can always be lost — it only moves it.2728**Consequences.** Operators see "outcome unknown" and must look before acting again; the UI and API expose `outcome_unknown` explicitly. Idempotency keys protect the *request* side (the same `POST /actions` replayed returns the same action), independently of this rule.2930**Evidence.** `TestCrashRestartNeverIssuesASecondCommand`, `TestTwoInstancesNoDuplicates`, `TestDeliveredWithoutResultIsOutcomeUnknown`, `TestUndeliveredCommandIsAgentUnreachable`, `TestLateResultIsRecordedNotRewritten` (`services/internal/orchestrator/orchestrator_test.go`); `TestUnknownOutcomeAfterAgentCrash` (`services/internal/ingest/signed_test.go`); `TestNoRetryAfterUnknownOutcomeOrRefusal` (`services/internal/jobs/runner_test.go`); `tests/e2e/local-stack.sh` ("exactly one command issued").31
Files main
Menu

Theme