# Operations

Paths are the production ones (docs/deployment.md); for the dev stack replace ~/agent-os-data by /tmp/aos-dev-data and PM2 by scripts/dev-up.sh / scripts/dev-down.sh.

# First run

1. Read the one-time setup token on the host — it is never logged or printed: ssh <node> 'cat ~/agent-os-data/setup-token' (file mode 0600; it disappears once the administrator exists).
2. Open https://agent.spboucher.ai → the setup page: token, username, a passphrase (≥ 12 characters).
3. Enrol a second factor at once (TOTP or a passkey). It is mandatory: until one exists the session can only reach the enrolment endpoints. Add a second device in Settings → Security as a spare.
4. System page: the worker heartbeat, the database and both provider keys must be green. A provider shown "missing" means its key is not in .env (see below); missions on that provider end blocked.
5. Settings: default model / effort, mission budget (default USD 15), daily budget (default USD 60 — a hard cap for all missions together), approval timeout (default 72 h; an expired approval is a denial).
6. Create a project (new empty repository, an existing folder under the workspace root, or a clone) and a first small mission with one verifiable done criterion.

# Where keys live

The browser never receives a key: the server keeps booleans, scrubs key values from every response, WebSocket frame and log line, and the worker's sandbox gets a from-scratch environment.

# Updating (remote-first)

On the node, in ~/apps/agent-os:

git pull                                     # or edit there, then commit + push to spbgit
pnpm install --frozen-lockfile               # only if the lockfile changed
pnpm -r --workspace-concurrency=1 build
node packages/kernel/dist/migrate-cli.js     # DATABASE_URL from .env: set -a; . ./.env; set +a  (operator shell only)
pm2 restart agent-os-worker                  # in-flight missions resume (see below)
pm2 restart agent-os-server

Before a release run scripts/ci.sh (and scripts/ci.sh --e2e on a machine with dev keys). An engine change that alters the order or the requests of journaled steps makes in-flight runs fail with NonDeterminismError on their next claim: let running missions finish (or cancel them) before deploying such a change.

# Backup and restore

• Nightly at 04:05: agent-os-backup → ~/backups/agent-os/agent-os-<timestamp>.tar.gz.enc (AES-256, PBKDF2; the last 14 are kept). Content: pg_dump of agent_os + an index of the artifact store (size and hash per artifact). Not saved, on purpose: artifact bytes, the workspace (it is git — push the branches you care about), .env, provider keys.
• Manual: AOS_DATA_DIR=~/agent-os-data AOS_ENV_FILE=~/apps/agent-os/.env ~/apps/agent-os/deploy/backup.sh
• Restore drill (never into the live database):
  bashCopy

  createdb -O agent_os agent_os_restore && psql -d agent_os_restore -c 'create extension if not exists pgcrypto'
  deploy/restore.sh ~/backups/agent-os/agent-os-<timestamp>.tar.gz.enc postgres://…/agent_os_restore --check-artifacts

  To restore for real: stop both processes, restore into an empty agent_os, start the server, then the worker. Runs that were running at backup time resume from their journal; a step whose effect is unknown asks you.

# After a worker crash

Nothing to repair. The worker holds no state: a run it was executing stays running with a lease (90 s, heartbeat every 30 s). When the worker comes back (PM2 restarts it; dev: scripts/dev-up.sh --worker) it claims the run once the lease has expired and replays the journal:

• completed steps return their recorded result (nothing is executed twice);
• the step that was in flight is handled by its effect class: reads, model calls and workspace-confined writes are executed again in place; git_push is reconciled against the remote; a network-enabled shell command is never retried blindly — a question appears in Approvals ("it ran — continue" / "it did not run — run it" / "abort").

While no worker is alive the mission still reads "running"; the System page shows the worker as down (no heartbeat for 60 s). Streamed text of an interrupted model turn is discarded and streamed again.

# Stop everything

1. pm2 stop agent-os-worker (dev: scripts/dev-down.sh --worker) — no model call, tool or shell command can start any more; commands already running are killed with the worker.
2. Cancel the missions so that they do not resume with the next worker: in the console (mission → Cancel), or for all of them at once:
   bashCopy

   cd ~/apps/agent-os && set -a && . ./.env && set +a && cd apps/worker && node --input-type=module -e "
     import { createKernel, requestCancel } from '@aos/kernel';
     const k = createKernel();
     const r = await k.pool.query(\"SELECT id FROM missions WHERE status IN ('queued','running','waiting')\");
     for (const m of r.rows) console.log(m.id, JSON.stringify(await requestCancel(m.id, k)));
     await k.close();"

   (tested on the dev stack; it runs from apps/worker so that @aos/kernel resolves). requestCancel cancels queued / waiting runs at once (and running ones whose lease has expired — the case when the worker is stopped) and marks the others; pending approvals and questions are dropped.
3. The daily budget in Settings is the other brake: at the cap every model call is refused until the next day.

# Routine checks

• System page: worker heartbeat, database latency, key presence, counts; GET /api/v1/health for probes.
• Spend page: today vs the cap, 7 days, by model (from the journal — real numbers only).
• Logs: pm2 logs agent-os-worker / agent-os-server (JSON, no prompt, no model output, no key).
• Disk: ~/agent-os-data/artifacts grows with offloaded tool outputs; there is no artifact GC yet.
• Old data: no retention job yet for events / steps; plan one before the tables get large.