SPB Git

Repository Public

TEST_REPORT.md 10.5 KB

# SP Term — Test report

Date: 2026-11-17 · Machine: MacBook Pro, Apple M5 Max (18 cores), 48 GB, macOS 27.0 (26A5425a), Xcode 27.2, Swift 6.4.2 · One display (built-in, 1312×848 pt @2x).

All numbers below are copied from the output of the listed commands. Failures met on the way are listed too.

# Summary

Suite Command Result
SPTermKit (core) Scripts/kit.sh test SPTermKit 98 tests passed (PTY 17, terminal core 34, workspace 19, placement 13, storage 10, themes 5)
SPRemoteKit Scripts/kit.sh test SPRemoteKit 65 tests passed (SSH 39, git 26)
SPIntelligence Scripts/kit.sh test SPIntelligence 56 tests passed (parameterized: 118 risk cases, 28 redaction cases, 27 loopback hosts)
App integration (hosted in the app) Scripts/build.sh test 38 tests passed (workspace 8, displays 6, broadcast 3, paste/clipboard 5, assistant 6, remote/registry 7, performance 3)
End-to-end on the running app Scripts/smoke.sh 29 checks; six runs, from 23 to 29 passed; last run on the signed Release build: 29/29 (see "End-to-end")
Total automated tests 257 unit/integration tests, 0 failing at the last run

Test code: 4 800 lines for 11 700 lines of source.

# What each suite proves

# PTY (PTYProcessTests, real processes)

Output and exit code; the child has a real controlling terminal (tty, test -t); interactive shell echo; initial window size seen by stty size; SIGWINCH delivered with the new size; Ctrl+C interrupts the foreground job and not the shell (status=130); Ctrl+Z suspends and job control works in zsh; Ctrl+D is end-of-file; Unicode round trip (accents, CJK, ZWJ emoji); working directory and environment (nothing inherited implicitly); terminate ends and reaps the session; releasing the object hangs up and reaps (no zombie, no orphan); the child inherits no descriptor of the app; launch failures throw; 200 000 lines are delivered completely before the exit is reported; 300 KB of input is accepted without blocking; libproc probes.

# Emulation engine conformance (ConformanceTests, SwiftTerm 1.99.2 headless)

Cursor addressing and clamping; erase/insert/delete line; SGR basic, bright, 256 and true colour (both ; and : forms); bold/italic/underline/inverse/strike; alternate screen kept separate and restored; scrolling region; bracketed paste, mouse and focus-report modes; cursor position and device attribute reports; OSC 0/2 titles; OSC 8 hyperlinks; OSC 52 write is handed to the host and read returns nothing by default; wide, combining, ZWJ and flag sequences; wrap and reflow (the line under the cursor is left to the shell, completed lines re-wrap both ways); bounded scrollback; a corpus of malformed sequences and invalid UTF-8; 60 fuzz rounds of 4 000 random chunks with random resizes. These are checks of the integrated engine, not of an emulator written for SP Term.

# Session integration (SessionIntegrationTests, real zsh/sh)

Commands run with TERM/COLORTERM/TERM_PROGRAM; title reports; directory tracking by probing and by OSC 7; busy detection and foreground process name; exit status and message; view resize reaches stty size; restart keeps scrollback and starts a new pid in the last directory; launch failure is reported in the pane; terminal-generated replies are not treated as user input; theme, font and size application including a missing font.

# Layout and workspace model

Recursive splits, same-axis flattening, exact tiling without overlap for 2/3/4/6/8 panes, removal and collapse, divider and keyboard resizing with minimum sizes, geometric focus navigation, swap/move, normalization of damaged trees, Codable round trip, 200 random operation sequences preserving all invariants; tab repair; every preset consistent; snapshot instantiation with fresh ids.

# Multi-display placement (simulated monitors)

Same display kept; display removed → window brought back with its relative position; runtime rescue moves only unreachable windows; 500 random configuration changes (1–3 displays, a projector, random frames) never leave a window unreachable; no display → no placement; resolution change rescales; display id change matched by name and size; cascade stays inside the visible frame; slot mapping; Dual Monitor on one and two displays; tiling. Hosted versions of the same scenarios run against real NSWindow objects: windows opened on a chosen display, moved between displays, rescued after an unplug with their sessions still alive, restored when the saved display is gone.

# Storage

Round trip with 0600/0700 permissions and no temp file left; a corrupt file is set aside, not deleted; a newer schema is refused and preserved; migration from schema 1; damaged layouts repaired; tolerant settings decoding with safe defaults; snippet rendering that cannot escape its quotes; history opt-in, private mode, masking, bounding.

# SSH and git

Config parsing fixtures; every command-builder security rule; hostile host names refused; known-hosts inspector; error classification; read-only git on temporary repositories; the allow-list refusing mutating subcommands. In the app: ssh is started directly (no shell), arguments never mention host-key or password options, a failing connection is diagnosed and can be reconnected. One real session to a LAN host was opened by hand through the palette: remote hostname and uname -m answered, exit produced "Connection ended" with a Reconnect button. No command was sent to any remote host by the app itself.

# Safety (hosted)

Broadcast is off by default, the toggle only opens a confirmation, input is mirrored only after it, a destructive line is held for the other panes and a standard confirmation is refused, cancelling clears the line, automatic terminal replies are never mirrored. Paste policy including escape injection with confirmations disabled; sanitized paste never presses Return. OSC 52 write asks by default and leaves the clipboard untouched; read is refused. Assistant: no request without preview and explicit send (counted on a mock transport), masked payload, the API key never in the body or in descriptions, suggestions never executed on arrival or on Insert, execution only through the consent gate, typed phrase for destructive commands, local-only refuses before any I/O, terminal text attached only when chosen, keys never written to disk.

# End-to-end on the running application

XCUITest cannot be used on this toolchain (Xcode 27.2 on macOS 27.0 times out "enabling automation mode"), so Scripts/smoke.sh drives the real app with System Events and observes results from outside (files written by commands typed into panes, window counts, the saved layout). 29 checks: launch; command execution on a real tty; environment; job control; Ctrl+C; Ctrl+D; vim writes a file; top; less; python sees a tty; split right/down with three distinct PTYs; keyboard focus navigation; zoom changes the TTY size; new tab; new window; split inherits the directory (a path with a space); opening a folder with the app; native full screen; window resize changes the TTY size; quit; no shell left behind; layout saved with three windows; 0600/0700 permissions; relaunch restores three windows; restored sessions are new processes in the saved directory.

Run Pace Result
1 0.5 s between keys 25 passed, 4 failed — 3 were mistakes in the script (quoting; Return sent as the word "return")
2 0.5 s 27 passed, 2 failed — "new tab" and "split inherits directory": the command keys never reached the app
3 1.0 s 28 passed, 1 failed — the first-window check timed out; every later check passed
4 1.0 s, signed Release build 28 passed, 1 failed — again the first-window check
5 1.0 s, signed Release build 23 passed, 6 failed — a keystroke was lost in the vim step and the following checks typed into vim
6 1.0 s, signed Release build, script made tolerant to a lost key (forces full-screen programs to quit before the next check) 29 passed, 0 failed

After run 5, vim was exercised alone on the signed build: the file was written correctly. (A first attempt at that isolated check failed because System Events mistypes digits on this keyboard layout — the file contained the text without the digit.)

The two failures of run 2 were investigated by replaying the same sequences step by step with the app state read after each step: every action worked (zoom toggled six times in a row, tab and split created each time). The losses appear only when System Events injects keys quickly; other automation was active on the machine during these runs. They are reported as an unreliability of the test harness, not as verified application defects — but they are not proven harmless either.

# Defects found by testing and fixed

Found by Defect Fix
PTY tests (Ctrl+Z) zsh had no controlling terminal: opening the slave after POSIX_SPAWN_SETSID does not acquire it on macOS launch through forkpty + exec-mode posix_spawn
PTY tests exit of very short programs could be missed (process source registered after the exit) registration handler + end-of-file triggered polling
PTY tests (crash, signal 5) releasing a suspended dispatch write source the source is cancelled and recreated instead of suspended
Screenshot after a split keyboard stayed in the previous pane while the new one was highlighted the terminal container claims first responder when its view enters the window
Screenshot of an ended SSH pane status bar showed "Exited" twice and a local directory for a remote session status bar corrected
Benchmark reading the whole 200 000-line buffer in a polling loop stalled the parser (a flaw of the test, 0.4 MB/s) completion observed through a sentinel file and the visible rows
Theme tests secondary text below 4.5:1 on the two light themes darker secondary text on light themes

# Not tested

  • More than one physical display (only simulated displays were available); Spaces.
  • VoiceOver; the app was not run with VoiceOver on.
  • Mouse gestures: divider dragging, tab and pane drag & drop, ⌘-click on links, text selection.
  • Quick terminal hotkey, notifications, the Finder Services entry, the session export save panel.
  • Real AI providers (no request was sent), real agent CLIs inside panes.
  • fish, nvim, htop, nano (first three not installed on this machine); tmux attach entry.
  • Intel execution: the x86_64 slice is built and signed but was not run (no Intel Mac, no Rosetta run).
  • macOS 14 and 15: the deployment target is 14.0 but the app only ran on macOS 27.0.
  • Long-duration use, sleep/wake, very large numbers of sessions.
Files main
Menu

Theme