# Backup and recovery

Source: deploy/backup.sh, deploy/restore.sh, deploy/make-manifest.py.

Status on 2026-11-10: the first nightly-format backup was produced on M5M36 (about 20 KB, the database being almost empty) and a restore drill into a scratch database succeeded (schema restored, row counts printed). Backups are not yet copied off the node.

# What is backed up

deploy/backup.sh runs as the PM2 process claude-remote-backup every night at 03:35 (and can be run by hand). It writes ~/backups/claude-remote/claude-remote-YYYYmmdd-HHMMSS.tar.gz.enc (mode 0600) and keeps the 14 most recent archives (KEEP).

Encryption: openssl enc -aes-256-cbc -pbkdf2 -iter 200000 -salt with the passphrase file ~/.config/claude-remote/backup.pass (0600), created with 48 random bytes on the first run. Without that file a backup cannot be restored.

# What is NOT backed up

# Restore

deploy/restore.sh <archive.tar.gz.enc> <target DATABASE_URL> [--with-claude-config]

• Decrypts with $PASS_FILE (default ~/.config/claude-remote/backup.pass), prints the archive date.
• pg_restore --no-owner --no-privileges --exit-on-error into the target, which must exist and be empty.
• Prints the row counts of users, projects, terminal_sessions, audit_events.
• --with-claude-config additionally extracts claude-config.tar into $HOME (overwrites the listed files).
• It does not restore task-meta.tar; extract it by hand into $CR_DATA_DIR if task metadata on disk matters.

# Restore drill (do this periodically; it does not touch production)

ssh M5M36
cd ~/apps/claude-remote
createdb cr_restore_drill
deploy/restore.sh "$(ls -1t ~/backups/claude-remote/*.enc | head -1)" postgres://localhost/cr_restore_drill
dropdb cr_restore_drill

Expected output: archive created at: …, four name=count lines, restore ok.

# Recommended and not yet done

1. Copy the archives to another machine. They currently stay on M5M36, next to the database they protect. For example a nightly rsync -a ~/backups/claude-remote/ <other-node>:backups/claude-remote/ towards a node the M5M36 key can reach. Not configured.
2. Keep the passphrase somewhere else (owner step 5 of the first-run list), and not next to the off-node copies.
3. Protect the gateway manifest: it is the only other copy of CR_SECRET_KEY.

# Recovery scenarios

# Disaster recovery runbook: M5M36 lost → new node

Prerequisites that must exist outside M5M36: a backup archive, the backup passphrase, the gateway manifest. If the archive or the passphrase is missing, the database cannot be recovered: deploy fresh, run the first-run setup again, and re-create projects by cloning.

1. Choose and prepare the node (criteria in node-selection.md): always-on Mac Studio, tunnel peer, Node, pnpm, PM2. Install tmux, ripgrep, postgresql@17 (started as a Homebrew service) and the Claude Code CLI (npm install -g @anthropic-ai/claude-code).
2. Regenerate the manifest on the gateway for the new node, keeping the secrets: python3 make-manifest.py --node <NEW> > ~/dispatch/apps/claude-remote.json.new, review, then replace the manifest. The generator reuses CR_SECRET_KEY, CR_RUNTIME_TOKEN, DATABASE_URL and CR_EXEC_HOSTS from the existing file.
3. Create the database by hand on the new node. Because DATABASE_URL is reused, the manifest's post_sync hook does not contain the role/database creation commands (they are emitted only when a new password is generated). Create the role claude_remote with the password from the manifest's DATABASE_URL, the database claude_remote owned by it, and create extension pgcrypto.
4. Deploy the code from the git server (claude-remote.git) through mld with the new manifest (mld deploy claude-remote --node <NEW>), then adopt the directory again. The build and the migrations run on the node; the server starts with an empty schema.
5. Restore the database. Copy the archive and the passphrase file to the node, then: pm2 stop claude-remote-server claude-remote-runtime; dropdb claude_remote && createdb -O claude_remote claude_remote (plus pgcrypto); PASS_FILE=<path> deploy/restore.sh <archive> "$DATABASE_URL" --with-claude-config; pm2 start claude-remote-runtime claude-remote-server. The restored schema_migrations table makes the next migration run a no-op unless newer migrations exist. With an administrator in the database, the server deletes the setup token at start.
6. Public route. mld deploy repoints https://claude.spboucher.ai to the new node; DNS does not change. Check https://claude.spboucher.ai/api/v1/health.
7. Log in. Password and passkeys work as before (the WebAuthn RP ID is the unchanged hostname). TOTP works only if CR_SECRET_KEY is the original one.
8. Re-establish what is not in the backup: run /login in a Claude session; generate a new SSH key on the node, authorise it on the git server (git-shell only) and on the CR_EXEC_HOSTS nodes, and remove the lost node's key from those authorized_keys; re-enter MCP servers and their API keys; rewrite the sanitized global CLAUDE.md. The runtime re-installs its hook entries by itself.
9. Projects. Project rows are restored but their directories are not. Re-clone each repository into the same relative path under ~/workspace (the project table stores root label + relative path).
10. Sessions and tasks. Sessions that were live at backup time become RECOVERABLE; because the conversation transcripts were not backed up, restarting them with --resume is not expected to find the conversation — close them and start new sessions. Tasks that were running become LOST.
11. Re-enable backups (the PM2 process is part of the manifest), restore the passphrase file at ~/.config/claude-remote/backup.pass so that old and new archives share one passphrase, run a backup and a restore drill.
12. Revoke all browser sessions from Settings → Security if the lost node's disk may be read by someone else, and change the password.

# Break-glass: all second factors lost

There is no recovery code and no reset endpoint. The login logic sends an account with no second factor to enrolment after a correct password, so the manual procedure on the node is:

-- psql "$DATABASE_URL"   (password must still be known)
UPDATE users SET totp_secret_enc = NULL, totp_last_counter = NULL;
DELETE FROM devices;
UPDATE auth_sessions SET revoked_at = now() WHERE revoked_at IS NULL;

then log in with the password and enrol a new factor immediately. Anyone with shell access to the node account can do this, which is consistent with the trust model in security.md. A lost password has no supported procedure other than restoring a backup or re-running setup on an empty database.

# Limitations / not verified

• The disaster-recovery runbook has not been rehearsed on a second node. Steps 2–5 follow from reading make-manifest.py and restore.sh; step 4's exact mld invocation depends on the orchestrator.
• The break-glass SQL is derived from the login code path and was not exercised.
• Only the restore drill into a scratch database was actually performed (once, with an almost empty database).
• AES-256-CBC provides confidentiality but no integrity check; a corrupted archive is detected only when tar or pg_restore fails.
• There is no monitoring of backup success: a failing nightly run is visible only in pm2 logs claude-remote-backup and by the absence of a new file.
• Recovery point objective is up to 24 hours for metadata; project files have no backup beyond git.

# Codex Remote instance

The second instance (see deployment.md, "Codex Remote instance") is backed up by the same two scripts, selected by the environment variable CR_APP_NAME (default claude-remote). Every name derives from it; CR_AGENT (claude | codex) selects which agent configuration is saved and, when unset, is inferred (codex-remote ⇒ codex, anything else ⇒ claude). With neither variable set the scripts behave exactly as described above.

The manifest sets CR_APP_NAME=codex-remote in the environment of codex-remote-backup. Retention (KEEP, 14) applies per instance: each run prunes only the archives carrying its own prefix.

# What is and is not backed up (Codex)

codex-config.tar holds, relative to the Codex home ($CODEX_HOME if set, otherwise ~/.codex), an explicit allow-list — each item only if it exists:

• config.toml
• AGENTS.md
• prompts/

Never in the archive:

auth.json, *.sqlite*, *.db, *.jsonl and *.log are additionally excluded by name, so such a file placed inside prompts/ is not archived either. config.toml is saved as it is: if API keys were written into it (for example in an MCP server's env table), they are in the archive — encrypted, like the rest. The other rows of "What is NOT backed up" above apply with the Codex names (~/workspace is shared by the two instances and is in neither backup).

# Restore (Codex)

CR_APP_NAME=codex-remote deploy/restore.sh <archive.tar.gz.enc> <target DATABASE_URL> [--with-codex-config]

• PASS_FILE defaults to ~/.config/codex-remote/backup.pass.
• --with-codex-config extracts codex-config.tar into the Codex home ($CODEX_HOME or ~/.codex, created if missing), overwriting config.toml, AGENTS.md and the files of prompts/. A restored config.toml carries the notify entry it had at backup time.
• --with-agent-config is accepted by both instances and means the configuration of that instance's agent. Passing the other agent's option (--with-claude-config to the Codex instance or the reverse) is refused before anything is restored.

Restore drill, as above with the Codex names:

ssh M5M36
cd ~/apps/codex-remote
createdb cxr_restore_drill
CR_APP_NAME=codex-remote deploy/restore.sh "$(ls -1t ~/backups/codex-remote/*.enc | head -1)" postgres://localhost/cxr_restore_drill
dropdb cxr_restore_drill

The recovery scenarios and the disaster-recovery runbook apply to Codex Remote with the names substituted (make-manifest.py --agent codex, role and database codex_remote, codex-remote-* processes, CR_APP_NAME=codex-remote … restore.sh … --with-codex-config, codex login instead of /login).

# Limitations / not verified (Codex)

• Tested on a development machine only, with a temporary HOME, a stub pg_dump / pg_restore / psql and a fake ~/.codex: the archive contained config.toml, AGENTS.md, prompts/ and not auth.json; the restore put them back. No backup or restore of a real codex_remote database has been performed.