Session persistence
What survives which failure, exactly. The rule that governs everything:
A terminal session's process lives in the tmux server on the node. It survives anything that does not stop that tmux server. When the tmux server is gone, the process and its memory are gone; what remains is the session's metadata and, for Claude sessions, the conversation that Claude Code saved on disk. Claude Remote never claims that process memory survives.
Source: apps/runtime/src/{tmux,sessions,core-attach,core-task-manager,auth}.ts,
apps/server/src/modules/sessions/{service,index}.ts, apps/web/src/components/terminal/controller.ts.
Where state lives
| State | Location | Lost when |
|---|---|---|
| Running process (Claude Code or shell), its memory, its child processes | tmux server on the node | the tmux server stops (host reboot, power loss, kill-server) or the process exits |
| Pane history (≤ 20 000 lines) and current screen | tmux server memory | same |
| Session metadata: id, title, project, working directory, kind, state, launch arguments, conversation id, exit code | PostgreSQL terminal_sessions |
never by a failure (restored from backup after a disk loss) |
| Lifecycle events | PostgreSQL terminal_events |
retention (default 90 days) |
| Claude conversation transcript | ~/.claude/projects/…/*.jsonl, written by Claude Code |
never by Claude Remote |
| Browser scrollback | xterm.js memory in the tab | page reload (rebuilt from the tmux history on the next attach) |
| Task process | tmux session crt-<uuid> |
tmux server stops |
| Task command, log, exit code, usage | $CR_DATA_DIR/tasks/<id>/ |
never by a failure |
Guarantees per failure
| Event | Process and its memory | What the user sees | Action needed |
|---|---|---|---|
| Browser refresh or tab closed | Kept | New attach: history replay (up to CR_REPLAY_LINES), then the live screen |
none |
| Network drop / Wi-Fi ↔ cellular change | Kept | "Reconnecting"; automatic re-attach | none |
| iOS suspends the tab or the PWA | Kept | Re-attach when the page becomes visible again | none |
| Auth session expired or revoked | Kept | Socket closed with 4401; redirect to login | log in again, re-open the session |
claude-remote-server restarted |
Kept | Sockets closed (1001), automatic re-attach once the server answers | none |
claude-remote-runtime restarted or killed |
Kept | Sockets closed (1001 or 4503), automatic re-attach once the runtime is back; hooks keep working | none |
| PostgreSQL restarted or briefly unavailable | Kept | API errors during the outage; an open terminal stream keeps flowing | none |
| Slow client saturating the stream for 30 s | Kept | Socket closed with 4429, automatic retry | none |
| Session process exits by itself | Ended (normal) | STOPPED (exit 0) or FAILED (non-zero) with the exit code; "Session ended" with a Restart button — never an automatic new session |
optional: start a new session |
| tmux session killed outside the application | Lost | RECOVERABLE |
Restart |
| Host reboot, power loss, tmux server killed | Lost | every live session becomes RECOVERABLE once the application is back |
Restart each session you need |
| Disk loss / node loss | Lost | service down until rebuilt | backup-recovery.md |
Details
Browser refresh, network drop, iOS suspension. Closing a WebSocket kills only that tmux attach
client. The web client reconnects with exponential backoff and jitter (0.5 s → 10 s) and immediately on
visibilitychange, online and pageshow; when the page becomes visible with a socket that looks open,
it sends a ping and reconnects if no answer arrives within 2.5 s. Three unanswered pings (one every 5 s)
also mark the connection dead. Every new attach starts from a reset terminal, gets the history replay,
then the live screen. Keystrokes typed while disconnected are dropped, not queued. The client never
creates or restarts a session by itself.
Server restart. The server holds no terminal state. On start it re-subscribes to the runtime's event
stream and runs the reconciler, which re-reads the truth from the runtime (attached counts, exits). While
the runtime is unreachable the reconciler changes nothing and the runtime component is reported OFFLINE.
Runtime restart. The tmux server is a separate daemon; shutting the runtime down closes attach PTYs
only. A new runtime finds the same cr-* sessions. Because surviving sessions carry the hook token in
their environment, the token file is reused when at least one live cr-* session exists, so
WAITING_FOR_INPUT keeps working; it is regenerated only when no session survived. Tested with SIGKILL
of a real runtime process (runtime/src/main.test.ts) and end to end
(server/src/integration/real-runtime.test.ts — "runtime down ⇒ OFFLINE, states untouched; restart ⇒
same tmux sessions, hooks still accepted").
Database restart. Terminal bytes do not touch the database. State transitions that fail during the outage are caught up by the reconciler (every 15 s) afterwards. Open WebSockets survive a failed periodic auth re-check (it is retried on the next tick).
Host reboot → RECOVERABLE. After a reboot the node logs in automatically, the PM2 processes are brought
back (cluster-standard PM2 startup; mld heal restarts missing ones every 5 minutes), and the runtime
starts a fresh, empty tmux server. The reconciler then finds database sessions in a live state that the healthy runtime does
not know:
| State before | State after |
|---|---|
RUNNING, DETACHED, WAITING_FOR_INPUT |
RECOVERABLE |
STARTING |
FAILED |
STOPPING |
STOPPED |
Restarting a RECOVERABLE session (POST /sessions/:id/reconnect {restart:true}; "Restart" in the UI)
starts a fresh process with the same session id, title and working directory:
| Session kind | Command |
|---|---|
claude, conversation id known (learned from a hook or given at creation) |
claude --resume <conversationId> |
claude, conversation id unknown |
claude --continue (most recent conversation in that directory) |
shell |
a new login shell |
What comes back is the saved conversation, loaded by Claude Code. What does not come back: anything that
was in process memory — a tool call in flight, background shell commands started by Claude, unsaved
editor buffers inside the terminal, shell variables and history of a shell session, the pane scrollback.
A RECOVERABLE session can also be closed (STOPPED) without restarting.
Sessions are never restarted automatically.
Tasks
| Event | Running task |
|---|---|
| Browser closed, server restart, runtime restart | Keeps running; the runtime rebuilds task state from tasks/<id>/meta.json, the exit file and tmux at startup; the log is written by a sink process independent of the runtime |
| Host reboot / tmux server lost | LOST (no exit file, runner gone). End time left empty. Never retried automatically |
| Task finished while the runtime was down | Final state and exit code are read from the exit file at the next start |
Evidence
| Guarantee | Test |
|---|---|
| Detach on close, re-attach to the same shell | apps/server/src/integration/real-runtime.test.ts — "shell session: terminal I/O, resize, detach on close, reattach to the same shell" |
| History fully replayed on attach | apps/runtime/src/scrollback.test.ts — "a fresh attach gets the whole history back: 300 numbered lines, none missing, none twice" |
| Runtime killed, session intact | apps/runtime/src/main.test.ts — "SIGKILL of a real runtime process leaves the session running…" |
tmux session killed ⇒ RECOVERABLE |
real-runtime.test.ts — "tmux session killed behind the server ⇒ RECOVERABLE (process lost, metadata kept)" |
tmux server gone ⇒ RECOVERABLE, restart = fresh process, same id |
real-runtime.test.ts — "host reboot (tmux server gone) ⇒ RECOVERABLE; reconnect {restart:true} ⇒ fresh process, same id" |
| Restart reuses id, cwd and resume arguments | apps/server/src/modules/sessions/sessions.test.ts — "healthy runtime without the session ⇒ RECOVERABLE; restart reuses id, cwd and resume args" |
| Browser reconnects after a network drop | apps/web/e2e/auth-terminal.spec.ts — "network drop: the terminal reconnects by itself and keeps working"; e2e-real/real-backend.spec.ts (reload re-attach, socket drop) |
| Task survives a runtime restart | apps/runtime/src/tasks.test.ts — "a running task keeps running while the runtime is down and is picked up again" |
Limitations / not verified
- A real reboot of M5M36 was not performed. "Host reboot" is covered by killing the tmux server in
tests; PM2 resurrection and auto-login after a real power cycle rely on the cluster's standard node
configuration and
mld heal. - iOS suspension was not tested on a real device (WebKit emulation only).
- The PostgreSQL-restart row is derived from the code paths (terminal bytes never touch the database; failed auth re-checks keep the socket); it was not exercised by a test.
claude --resume/--continueafter a restart was verified as an argument list, not as a resumed paid conversation.- The pane history is capped at 20 000 lines by tmux and the replay at
CR_REPLAY_LINES/ 2 MiB; older output is not recoverable from the terminal. - Session processes share the node's fate: there is no migration of a live session to another node.