spb/claude-remote

1# Session persistence23What survives which failure, exactly. The rule that governs everything:45> A terminal session's process lives in the tmux server on the node. It survives anything that does not6> stop that tmux server. When the tmux server is gone, the process and its memory are gone; what remains7> is the session's metadata and, for Claude sessions, the conversation that Claude Code saved on disk.8> Claude Remote never claims that process memory survives.910Source: `apps/runtime/src/{tmux,sessions,core-attach,core-task-manager,auth}.ts`,11`apps/server/src/modules/sessions/{service,index}.ts`, `apps/web/src/components/terminal/controller.ts`.1213## Where state lives1415| State | Location | Lost when |16|---|---|---|17| Running process (Claude Code or shell), its memory, its child processes | tmux server on the node | the tmux server stops (host reboot, power loss, `kill-server`) or the process exits |18| Pane history (≤ 20 000 lines) and current screen | tmux server memory | same |19| Session metadata: id, title, project, working directory, kind, state, launch arguments, conversation id, exit code | PostgreSQL `terminal_sessions` | never by a failure (restored from backup after a disk loss) |20| Lifecycle events | PostgreSQL `terminal_events` | retention (default 90 days) |21| Claude conversation transcript | `~/.claude/projects/…/*.jsonl`, written by Claude Code | never by Claude Remote |22| Browser scrollback | xterm.js memory in the tab | page reload (rebuilt from the tmux history on the next attach) |23| Task process | tmux session `crt-<uuid>` | tmux server stops |24| Task command, log, exit code, usage | `$CR_DATA_DIR/tasks/<id>/` | never by a failure |2526## Guarantees per failure2728| Event | Process and its memory | What the user sees | Action needed |29|---|---|---|---|30| Browser refresh or tab closed | **Kept** | New attach: history replay (up to `CR_REPLAY_LINES`), then the live screen | none |31| Network drop / Wi-Fi ↔ cellular change | **Kept** | "Reconnecting"; automatic re-attach | none |32| iOS suspends the tab or the PWA | **Kept** | Re-attach when the page becomes visible again | none |33| Auth session expired or revoked | **Kept** | Socket closed with 4401; redirect to login | log in again, re-open the session |34| `claude-remote-server` restarted | **Kept** | Sockets closed (1001), automatic re-attach once the server answers | none |35| `claude-remote-runtime` restarted or killed | **Kept** | Sockets closed (1001 or 4503), automatic re-attach once the runtime is back; hooks keep working | none |36| PostgreSQL restarted or briefly unavailable | **Kept** | API errors during the outage; an open terminal stream keeps flowing | none |37| Slow client saturating the stream for 30 s | **Kept** | Socket closed with 4429, automatic retry | none |38| Session process exits by itself | Ended (normal) | `STOPPED` (exit 0) or `FAILED` (non-zero) with the exit code; "Session ended" with a Restart button — never an automatic new session | optional: start a new session |39| tmux session killed outside the application | **Lost** | `RECOVERABLE` | Restart |40| Host reboot, power loss, tmux server killed | **Lost** | every live session becomes `RECOVERABLE` once the application is back | Restart each session you need |41| Disk loss / node loss | **Lost** | service down until rebuilt | [backup-recovery.md](backup-recovery.md) |4243### Details4445**Browser refresh, network drop, iOS suspension.** Closing a WebSocket kills only that `tmux attach`46client. The web client reconnects with exponential backoff and jitter (0.5 s → 10 s) and immediately on47`visibilitychange`, `online` and `pageshow`; when the page becomes visible with a socket that looks open,48it sends a ping and reconnects if no answer arrives within 2.5 s. Three unanswered pings (one every 5 s)49also mark the connection dead. Every new attach starts from a reset terminal, gets the history replay,50then the live screen. Keystrokes typed while disconnected are dropped, not queued. The client never51creates or restarts a session by itself.5253**Server restart.** The server holds no terminal state. On start it re-subscribes to the runtime's event54stream and runs the reconciler, which re-reads the truth from the runtime (attached counts, exits). While55the runtime is unreachable the reconciler changes nothing and the runtime component is reported `OFFLINE`.5657**Runtime restart.** The tmux server is a separate daemon; shutting the runtime down closes attach PTYs58only. A new runtime finds the same `cr-*` sessions. Because surviving sessions carry the hook token in59their environment, the token file is reused when at least one live `cr-*` session exists, so60`WAITING_FOR_INPUT` keeps working; it is regenerated only when no session survived. Tested with SIGKILL61of a real runtime process (`runtime/src/main.test.ts`) and end to end62(`server/src/integration/real-runtime.test.ts` — "runtime down ⇒ OFFLINE, states untouched; restart ⇒63same tmux sessions, hooks still accepted").6465**Database restart.** Terminal bytes do not touch the database. State transitions that fail during the66outage are caught up by the reconciler (every 15 s) afterwards. Open WebSockets survive a failed periodic67auth re-check (it is retried on the next tick).6869**Host reboot → `RECOVERABLE`.** After a reboot the node logs in automatically, the PM2 processes are brought70back (cluster-standard PM2 startup; `mld heal` restarts missing ones every 5 minutes), and the runtime71starts a fresh, empty tmux server. The reconciler then finds database sessions in a live state that the healthy runtime does72not know:7374| State before | State after |75|---|---|76| `RUNNING`, `DETACHED`, `WAITING_FOR_INPUT` | `RECOVERABLE` |77| `STARTING` | `FAILED` |78| `STOPPING` | `STOPPED` |7980Restarting a `RECOVERABLE` session (`POST /sessions/:id/reconnect {restart:true}`; "Restart" in the UI)81starts a **fresh process** with the same session id, title and working directory:8283| Session kind | Command |84|---|---|85| `claude`, conversation id known (learned from a hook or given at creation) | `claude --resume <conversationId>` |86| `claude`, conversation id unknown | `claude --continue` (most recent conversation in that directory) |87| `shell` | a new login shell |8889What comes back is the saved conversation, loaded by Claude Code. What does not come back: anything that90was in process memory — a tool call in flight, background shell commands started by Claude, unsaved91editor buffers inside the terminal, shell variables and history of a shell session, the pane scrollback.92A `RECOVERABLE` session can also be closed (`STOPPED`) without restarting.9394Sessions are never restarted automatically.9596## Tasks9798| Event | Running task |99|---|---|100| Browser closed, server restart, runtime restart | Keeps running; the runtime rebuilds task state from `tasks/<id>/meta.json`, the `exit` file and tmux at startup; the log is written by a sink process independent of the runtime |101| Host reboot / tmux server lost | `LOST` (no exit file, runner gone). End time left empty. Never retried automatically |102| Task finished while the runtime was down | Final state and exit code are read from the `exit` file at the next start |103104## Evidence105106| Guarantee | Test |107|---|---|108| Detach on close, re-attach to the same shell | `apps/server/src/integration/real-runtime.test.ts` — "shell session: terminal I/O, resize, detach on close, reattach to the same shell" |109| History fully replayed on attach | `apps/runtime/src/scrollback.test.ts` — "a fresh attach gets the whole history back: 300 numbered lines, none missing, none twice" |110| Runtime killed, session intact | `apps/runtime/src/main.test.ts` — "SIGKILL of a real runtime process leaves the session running…" |111| tmux session killed ⇒ `RECOVERABLE` | `real-runtime.test.ts` — "tmux session killed behind the server ⇒ RECOVERABLE (process lost, metadata kept)" |112| tmux server gone ⇒ `RECOVERABLE`, restart = fresh process, same id | `real-runtime.test.ts` — "host reboot (tmux server gone) ⇒ RECOVERABLE; reconnect {restart:true} ⇒ fresh process, same id" |113| Restart reuses id, cwd and resume arguments | `apps/server/src/modules/sessions/sessions.test.ts` — "healthy runtime without the session ⇒ RECOVERABLE; restart reuses id, cwd and resume args" |114| Browser reconnects after a network drop | `apps/web/e2e/auth-terminal.spec.ts` — "network drop: the terminal reconnects by itself and keeps working"; `e2e-real/real-backend.spec.ts` (reload re-attach, socket drop) |115| Task survives a runtime restart | `apps/runtime/src/tasks.test.ts` — "a running task keeps running while the runtime is down and is picked up again" |116117## Limitations / not verified118119- A real reboot of M5M36 was **not** performed. "Host reboot" is covered by killing the tmux server in120  tests; PM2 resurrection and auto-login after a real power cycle rely on the cluster's standard node121  configuration and `mld heal`.122- iOS suspension was not tested on a real device (WebKit emulation only).123- The PostgreSQL-restart row is derived from the code paths (terminal bytes never touch the database;124  failed auth re-checks keep the socket); it was not exercised by a test.125- `claude --resume` / `--continue` after a restart was verified as an argument list, not as a resumed paid126  conversation.127- The pane history is capped at 20 000 lines by tmux and the replay at `CR_REPLAY_LINES` / 2 MiB; older128  output is not recoverable from the terminal.129- Session processes share the node's fate: there is no migration of a live session to another node.130