# Operator runbook

## Daily checks
- `GET /api/health` → `db: ok`, `redis: ok`, `stream.connected: true`, `stream.last_msg_age < 10`.
- `/settings` → Kraken status `online`, clock drift < 2 s; `/market` → data-quality badges green (lag_bars 0–1).
- `pm2 ls` on the node: `crypto-trade-bot-api|web|worker` online, restarts not climbing.

## Common operations
| Task | Command |
|---|---|
| Apply migrations | `ktrade migrate` |
| One-shot OHLC pull | `ktrade backfill` |
| Probe key permissions (never trades) | `ktrade preflight` or Settings → Preflight |
| Change owner password | `ktrade set-owner-password` → put hash in env → restart api |
| Verify audit chain | `ktrade audit-verify` |
| Kill switch | Risk Center → toggle (audited) — blocks all new orders; cancels eligible open orders in Phase 4/5 |
| Add a pair to the universe | edit `KTRADE_UNIVERSE`, restart worker (pairs refresh immediately) |

## Failure modes
- **WS disconnected / checksum mismatches climbing**: worker reconnects with backoff; if persistent, restart the
  worker; check Kraken status page. Backtests are unaffected (REST data).
- **Stale candles (lag_bars > 2)**: REST backfill failing → see worker logs `ingest.backfill.error`; rate-limit waits
  are logged as `kraken.ratelimit.wait`.
- **Postgres down**: API health reports `db: error`; gateway defaults to deny; restart `postgresql@17`.
- **Exchange outage**: kill switch still blocks new orders locally; *cancellation of resting exchange orders cannot be
  guaranteed while the exchange is unreachable* (Phase 5 note).

## Backups & retention
- Postgres: nightly `pg_dump ktrade` (add to PM2 cron on the node); candles are re-accumulable only forward in time —
  **back up `candles` to keep history**.
- Retention: trades 30 d, book samples 14 d (worker `retention_loop`).

## Deployment (MacLustr)
Manifest `M1M32:~/dispatch/apps/crypto-trade-bot.json`; `mld deploy crypto-trade-bot` (post_sync builds venv + web);
route `https://crypto-trade-bot.app → <node>:8471` on BHS64. Remote-first: edit on the node, `git push` to spbgit.