Repository Public
1# Operations23Paths are the production ones (`docs/deployment.md`); for the dev stack replace `~/agent-os-data` by4`/tmp/aos-dev-data` and PM2 by `scripts/dev-up.sh` / `scripts/dev-down.sh`.56## First run781. Read the one-time setup token **on the host** — it is never logged or printed:9 `ssh <node> 'cat ~/agent-os-data/setup-token'` (file mode 0600; it disappears once the administrator exists).102. Open `https://agent.spboucher.ai` → the setup page: token, username, a passphrase (≥ 12 characters).113. **Enrol a second factor at once** (TOTP or a passkey). It is mandatory: until one exists the session12 can only reach the enrolment endpoints. Add a second device in Settings → Security as a spare.134. System page: the worker heartbeat, the database and both provider keys must be green. A provider14 shown "missing" means its key is not in `.env` (see below); missions on that provider end `blocked`.155. Settings: default model / effort, mission budget (default USD 15), **daily budget** (default USD 60 —16 a hard cap for all missions together), approval timeout (default 72 h; an expired approval is a denial).176. Create a project (new empty repository, an existing folder under the workspace root, or a clone) and a18 first small mission with one verifiable done criterion.1920## Where keys live2122| Secret | Location | Change it |23|---|---|---|24| Provider keys | node: `~/.config/agent-os/keys.env` (0600) → copied into `~/apps/agent-os/.env` at deploy | edit `keys.env`, redeploy (or edit `.env`), `pm2 restart agent-os-worker agent-os-server` |25| `DATABASE_URL`, `AOS_SECRET_KEY` | manifest on M1M32 (0600) → `.env` on the node | never rotate `AOS_SECRET_KEY` casually: enrolled TOTP secrets are encrypted with it |26| Backup passphrase | node: `~/.config/agent-os/backup.pass` (0600, created by the first backup) | keep a copy off the node: without it the archives are unreadable |2728The browser never receives a key: the server keeps booleans, scrubs key values from every response,29WebSocket frame and log line, and the worker's sandbox gets a from-scratch environment.3031## Updating (remote-first)3233On the node, in `~/apps/agent-os`:3435```bash36git pull # or edit there, then commit + push to spbgit37pnpm install --frozen-lockfile # only if the lockfile changed38pnpm -r --workspace-concurrency=1 build39node packages/kernel/dist/migrate-cli.js # DATABASE_URL from .env: set -a; . ./.env; set +a (operator shell only)40pm2 restart agent-os-worker # in-flight missions resume (see below)41pm2 restart agent-os-server42```4344Before a release run `scripts/ci.sh` (and `scripts/ci.sh --e2e` on a machine with dev keys).45An engine change that alters the order or the requests of journaled steps makes **in-flight** runs fail46with `NonDeterminismError` on their next claim: let running missions finish (or cancel them) before47deploying such a change.4849## Backup and restore5051- Nightly at 04:05: `agent-os-backup` → `~/backups/agent-os/agent-os-<timestamp>.tar.gz.enc`52 (AES-256, PBKDF2; the last 14 are kept). Content: `pg_dump` of `agent_os` + an **index** of the artifact53 store (size and hash per artifact). Not saved, on purpose: artifact bytes, the workspace (it is git —54 push the branches you care about), `.env`, provider keys.55- Manual: `AOS_DATA_DIR=~/agent-os-data AOS_ENV_FILE=~/apps/agent-os/.env ~/apps/agent-os/deploy/backup.sh`56- Restore drill (never into the live database):57 ```bash58 createdb -O agent_os agent_os_restore && psql -d agent_os_restore -c 'create extension if not exists pgcrypto'59 deploy/restore.sh ~/backups/agent-os/agent-os-<timestamp>.tar.gz.enc postgres://…/agent_os_restore --check-artifacts60 ```61 To restore for real: stop both processes, restore into an empty `agent_os`, start the server, then the worker.62 Runs that were `running` at backup time resume from their journal; a step whose effect is unknown asks you.6364## After a worker crash6566Nothing to repair. The worker holds no state: a run it was executing stays `running` with a lease67(90 s, heartbeat every 30 s). When the worker comes back (PM2 restarts it; dev: `scripts/dev-up.sh --worker`)68it claims the run once the lease has expired and replays the journal:6970- completed steps return their recorded result (nothing is executed twice);71- the step that was in flight is handled by its effect class: reads, model calls and workspace-confined72 writes are executed again in place; `git_push` is reconciled against the remote; a network-enabled73 shell command is **never** retried blindly — a question appears in Approvals74 ("it ran — continue" / "it did not run — run it" / "abort").7576While no worker is alive the mission still reads "running"; the System page shows the worker as down77(no heartbeat for 60 s). Streamed text of an interrupted model turn is discarded and streamed again.7879## Stop everything80811. `pm2 stop agent-os-worker` (dev: `scripts/dev-down.sh --worker`) — no model call, tool or shell command82 can start any more; commands already running are killed with the worker.832. Cancel the missions so that they do not resume with the next worker: in the console (mission → Cancel),84 or for all of them at once:85 ```bash86 cd ~/apps/agent-os && set -a && . ./.env && set +a && cd apps/worker && node --input-type=module -e "87 import { createKernel, requestCancel } from '@aos/kernel';88 const k = createKernel();89 const r = await k.pool.query(\"SELECT id FROM missions WHERE status IN ('queued','running','waiting')\");90 for (const m of r.rows) console.log(m.id, JSON.stringify(await requestCancel(m.id, k)));91 await k.close();"92 ```93 (tested on the dev stack; it runs from `apps/worker` so that `@aos/kernel` resolves). `requestCancel`94 cancels queued / waiting runs at once (and running ones whose lease has expired — the case when the95 worker is stopped) and marks the others; pending approvals and questions are dropped.963. The daily budget in Settings is the other brake: at the cap every model call is refused until the next day.9798## Routine checks99100- System page: worker heartbeat, database latency, key presence, counts; `GET /api/v1/health` for probes.101- Spend page: today vs the cap, 7 days, by model (from the journal — real numbers only).102- Logs: `pm2 logs agent-os-worker` / `agent-os-server` (JSON, no prompt, no model output, no key).103- Disk: `~/agent-os-data/artifacts` grows with offloaded tool outputs; there is no artifact GC yet.104- Old data: no retention job yet for `events` / `steps`; plan one before the tables get large.105