# Operations runbook

Routine procedures for the production instance on M5M36. Diagnosis by symptom is in troubleshooting.md; deployment mechanics in deployment.md; recovery in backup-recovery.md.

Conventions: commands run on the node as the cluster operator account (ssh M5M36), in ~/apps/claude-remote unless stated. mld … commands run from the gateway (or its laptop wrapper). Nothing here needs sudo.

# Reference

# Daily / weekly checks

There is no external alerting: these checks are manual.

# Start, stop, restart

pm2 restart claude-remote-server      # browsers reconnect; sessions and tasks untouched
pm2 restart claude-remote-runtime     # attach clients dropped, then re-attached; tmux untouched
pm2 stop claude-remote-server         # the public site stops answering (gateway error)
pm2 logs claude-remote-server --lines 100 --nostream

• Both can be restarted in one command, as in the update procedure. The server reconnects to the runtime by itself with a 1–10 s backoff, in whatever order they come up.
• Restart the server after every rebuild of the web application (the CSP hashes are computed at start).
• Never run tmux -L claude-remote kill-server on production: it ends every session and task process. Sessions become RECOVERABLE, running tasks LOST.
• From the gateway, mld heal restarts missing processes and checks the route; it also runs every 5 minutes on its own.

# Planned node reboot

1. Tell yourself what will be lost: every running process inside sessions and tasks.
2. In the application, let running tasks finish or cancel them; note which Claude sessions matter.
3. Reboot. The node logs in automatically; PM2 and mld heal bring the processes back.
4. Check health, then open the Sessions page: former live sessions are RECOVERABLE. Restart the ones you need (claude --resume for known conversations) and close the others.

# Update the application

Remote-first, on the node — full procedure in deployment.md:

cd ~/apps/claude-remote && git status
# edit
pnpm -r build
set -a && . ./.env && set +a && node apps/server/dist/migrate.js
pm2 restart claude-remote-runtime claude-remote-server
curl -fsS http://127.0.0.1:8620/api/v1/health
git add -A && git commit -m "…" && git push origin main

# Sessions and tasks from the shell

tmux -L claude-remote list-sessions            # cr-<uuid> = terminal sessions, crt-<uuid> = tasks
tmux -L claude-remote capture-pane -p -t '=cr-<uuid>:' | tail -40      # read-only look at a pane
ls ~/claude-remote-data/tasks/<id>/            # command, log, exit, meta.json

Prefer the application for terminating sessions and cancelling tasks: it records the state transition and the audit event. A tmux session killed by hand shows up as RECOVERABLE (or the task as LOST).

Do not tmux attach to a cr-* session from an SSH terminal while a browser is using it: the window follows the most recently used client's size.

# Backups

set -a && . ./.env && set +a && deploy/backup.sh          # manual backup; prints "backup ok: <file> (<bytes>)"
pm2 logs claude-remote-backup --lines 20 --nostream       # result of the nightly run

Restore drill and disaster recovery: backup-recovery.md. Recommended monthly: one restore drill into a scratch database. Outstanding: copy archives off the node; store the passphrase elsewhere.

# Access management

# Configuration changes

# Data housekeeping

Automatic (daily janitor in the server): lifecycle events (90 days by default, user setting), node metrics (14 days), login attempts (30 days), audit events (400 days), notifications (90 days), expired auth sessions (7 days after expiry).

Manual:

• ~/claude-remote-data/tasks/<id>/ directories are never deleted by the application. Remove old ones by hand when disk space matters (the task row keeps its metadata; its log is then gone).
• Archived sessions stay in the database; there is no purge of terminal_sessions rows.
• Unregistering a project never deletes its files; remove the directory in a shell if wanted.

# Incident notes

When something unexpected happens, record: time, what the System page showed, pm2 list, the last 100 lines of both logs (they contain no secrets, cookies or terminal data by construction), and the audit entries around the time. Add the outcome to implementation-log.md.

# Limitations / not verified

• No monitoring or alerting beyond mld heal and the cluster's own node telemetry; a failed backup or a full disk is not signalled.
• The planned-reboot procedure has not been executed on M5M36.
• mld tunnel rm and the redeploy-after-manifest-change path are cluster-orchestrator operations taken from the cluster documentation; they were not re-run for this document.
• Task directory cleanup and session purge are manual; no tooling exists for them.
• Rotation procedures for CR_RUNTIME_TOKEN and CR_SECRET_KEY are derived from make-manifest.py and the code; they were not rehearsed.