# AKOS operations runbook

Deployed node: **M4M64c** (`~/apps/akos`, PM2 `akos-web`, `akos-worker`, `akos-proxy`), public URL **https://akos.spboucher.ai** via the MacLustr Tunnel (BHS64 Caddy → wg1 10.67.0.33:8520). Orchestrated by `mld` from M1M32 (manifest `M1M32:~/dispatch/apps/akos.json`). Remote-first: the deployed directory is the git repository (origin spbgit `akos.git`).

## Daily commands
```bash
mld status --live | grep akos          # placement + health
ssh M4M64c 'export PATH=/opt/homebrew/bin:$PATH; pm2 ls | grep akos; pm2 logs akos-web --lines 50 --nostream'
curl -s https://akos.spboucher.ai/api/v1/health
akos status && akos doctor             # from any machine with the CLI + token
```
If the laptop cannot reach M4M64c directly (Tailscale relay hiccup seen 2026-11-01), use the jump alias `ssh akosnode` (ProxyJump M1M32 → 192.168.2.103).

## Processes (PM2, restart on boot via `pm2 startup` + `mld heal` every 5 min)
| process | command | listens | role |
|---|---|---|---|
| akos-web | `.venv/bin/python -m akos web` | 127.0.0.1:8520 + wg1:8520 (never LAN) | UI, REST `/api/v1`, MCP `/mcp`, `/docs` |
| akos-worker | `.venv/bin/python -m akos worker` | — | jobs (discover/compile/verify/export/drift/backup/maintenance) + scheduler |
| akos-proxy | `.venv/bin/python -m akos proxy` | 127.0.0.1:8521 | governed egress for sandboxed workers |

Restart: `pm2 restart akos-web akos-worker akos-proxy`. Logs: JSON lines in `~/.pm2/logs/akos-*-out.log` (secrets redacted at the logger).

## Change procedure (remote-first)
1. `ssh M4M64c` → `cd ~/apps/akos && git status --short && git log --oneline -3`.
2. Edit on the node (or scp round-trip), then `make test` (offline, uses `akos_test`) and `make test-node` (sandbox + backup drill).
3. `pm2 restart akos-web akos-worker akos-proxy`; `curl 127.0.0.1:8520/api/v1/health`; `curl https://akos.spboucher.ai/api/v1/health`.
4. `git add -A && git commit -m "…" && git push origin main` on the node.
5. Web UI changes: `cd web && pnpm build` on the node (dist is served by akos-web; excluded from git/rsync).
Migrations: add `akos/migrations/000N_*.sql` (append-only); applied automatically at `akos-web` start and by `python -m akos migrate`.

## Secrets
- `~/apps/akos/.env` (0600) holds `SESSION_SECRET`, bootstrap admin password, `DATABASE_URL`; copy in `~/Desktop/Cluster/secrets/akos/.env` (laptop, outside git). Rotate `SESSION_SECRET` = all sessions logged out (tokens unaffected).
- Provider credentials live in the macOS login keychain of M4M64c (service `akos`); AKOS stores `vault://keychain/akos/<name>` only. Onboard: UI → Credential Vault → Create (value is write-only) or `POST /api/v1/credentials`. Rotate/revoke from the same page; every use is audited.
- To list keychain items: `security find-generic-password -s akos -a <name>` (value with `-w`; avoid in shared terminals).

## Backups
- Nightly 03:40 UTC (worker scheduler) → `scripts/backup.sh`: `pg_dump -Fc` + artifacts tarball + sha256 → `~/backups/akos/` (keep 14) → rsync `M3U96b:~/backups/akos/`.
- Manual: `POST /api/v1/ops/backup` (admin) or `bash scripts/backup.sh` on the node. Status: Operations page or `GET /api/v1/ops/backups`.
- Restore drill (tested by `make test-node`): `bash scripts/restore.sh ~/backups/akos/akos-<ts>.dump akos_restore_test [artifacts.tgz]`. To restore the live DB: stop PM2 processes, `scripts/restore.sh <dump> akos`, extract artifacts into `data/`, start processes.

## Observability
- `GET /api/v1/ops/health` (DB latency, migrations, live workers, proxy, sandbox, disk), `GET /api/v1/ops/metrics` (p50/p95 execution latency, job stats, egress decisions, by-API), `GET /api/v1/audit/verify` (hash chain).
- Every request/execution carries `X-Correlation-Id`; logs are JSON with `cid`.
- `mld heal` restarts missing PM2 processes/routes; `agentd` sees the site through the registry.

## Recovery
| Failure | Action |
|---|---|
| akos-web down | `pm2 restart akos-web`; check `.env` and Postgres (`brew services list`). Public 502 page is served by Caddy meanwhile. |
| worker crash mid-job | automatic: stale lock (no heartbeat 120 s) is requeued by any worker; dead-letter after `max_attempts` (`POST /jobs/{id}/retry`). Executions stuck in `intent/running` > 15 min are marked failed by `maintenance`. |
| Postgres down | `brew services restart postgresql@17`; `akos-web` fails closed (no execution without persisted intent). |
| disk full | prune `~/backups/akos`, `data/artifacts` is content-addressed (never delete while referenced). |
| tunnel route lost | `mld tunnel route akos` (or `mld heal`). |
| node loss | `mld move akos --to M2M32c` (has Postgres 17 + pgvector): restore the latest dump + artifacts there first, copy `.env`, then move. |

## Rollback
`mld tunnel rm akos` → `pm2 delete akos-web akos-worker akos-proxy` → (optional) `dropdb akos` after a backup → remove `~/apps/akos`, manifest, keychain items (`security delete-generic-password -s akos -a <name>`), `claude mcp remove akos -s user` on clients.