SPB Git

Repository Public

python-runtime.md 25.7 KB

# Python runtime

ORIX Sheet does not embed a Python interpreter. Python Studio runs a script in a separate process of an interpreter that is already installed on the Mac, talks to it over two extra pipes, buffers everything the script wants to write, and applies those writes to the workbook only when the run succeeded. By default the interpreter is wrapped in a sandbox-exec profile ("restricted mode").

Package Role Imports
Packages/SheetPython interpreter discovery, process spawning, the RPC host, the sandbox profile, limits, trust store, audit log, the orix Python module (Resources/python) SheetCore, Foundation, Darwin, CryptoKit
Packages/FeaturePython Python Studio (editor, console, tables, plots, changes), the consent logic, the workbook host, the write applier SheetCore, SheetUI, SheetPython

Source: Packages/SheetPython/Package.swift, Packages/FeaturePython/Package.swift.

The decision not to embed CPython (crash isolation, cancellation, no several-hundred-megabyte runtime to sign and notarize) is recorded in docs/research/technology-evaluation.md. The security analysis written by the package author is Packages/SheetPython/SECURITY.md; this document describes the mechanism and repeats the guarantees with their evidence. The application-wide threat model is in ../security.md.

# What runs where

flowchart TD
    STUDIO["Python Studio (main actor)<br/>consent check, then run"] --> RUNNER["PythonRunner<br/>one run at a time"]
    RUNNER -->|posix_spawn| SBX["sandbox-exec -p profile<br/>restricted mode only"]
    SBX --> PY["python -u -X utf8 -I -B runner.py<br/>own process group, fds 0-4 only"]
    PY -->|fd 4 requests| HOST["ActiveRun: RPC host<br/>answers on fd 3"]
    PY -->|fd 1 and 2| CONSOLE["Console<br/>capped at 4 MiB"]
    HOST -->|reads| WB["Live workbook<br/>read under the engine lock"]
    HOST -->|buffered writes| RESULT["PythonRunResult<br/>applied only if the run succeeded,<br/>as one undo step"]

SVG fallback: Python runtime processes and pipes

Piece Where it runs
Consent decision, write application, UI main actor (PythonStudioModel, PythonConsent, PythonWriteApplier)
Workbook reads for the script main actor (SessionPythonHost), holding session.engine.lock
Process supervision: pipe readers, timeout timer, memory watchdog, waitpid background threads and dispatch sources of ActiveRun
User code, runner.py, orix.py the child interpreter process

Nothing in either package starts an interpreter on a script when a workbook is opened, when the Studio appears or when a script is selected. PythonFeature.register() registers the pane and the menu action (shortcut ⌥⌘P: the action declares (key: "p", modifiers: [.option]) and App/Sources/MainMenu.swift adds ⌘ to every action shortcut), schedules a background sweep of stale run directories and installs a willTerminate observer. Interpreter discovery starts when the Studio is first shown.

# Interpreter discovery

PythonDiscovery.candidatePaths lists, in order: paths the user added, the managed virtual environment, /opt/homebrew/bin/python3, /usr/local/bin/python3, the versioned python3.N executables of those two directories (newest first), python.org framework builds, pyenv and conda locations under the home directory and /opt, and finally /usr/bin/python3 — the last only when developer tools are installed, because the shim otherwise opens an "install the command line tools" dialog.

Each existing executable is run once with the bundled probe.py (-I -B, an environment of three variables, 8-second timeout). The probe reports version, real executable, prefixes, site-packages directories, virtual-environment status and the versions of the packages of the scientific set (numpy, pandas, scipy, matplotlib, statsmodels, scikit-learn, polars, sympy, duckdb, pyarrow, seaborn). Interpreters older than 3.9 are discarded (isSupported).

Discovery runs interpreters found at well-known locations. It runs no workbook content.

# Managed environment and package installation

PythonEnvironmentManager.createVenv() creates ~/Library/Application Support/ORIX Sheet/python/venv with python -m venv --system-site-packages; this works offline.

PythonEnvironmentManager.install(package:) is the only operation of the Python integration that uses the network. It runs

text
<venv>/bin/python3 -I -m pip install --disable-pip-version-check --no-input -- <requirement>

unsandboxed, with a 900-second timeout. pip downloads from PyPI and executes the package's build and installation code. Two guards exist: the requirement must match a strict pattern (a project name, optional extras, optional version constraints; no options, URLs or paths; at most 120 characters), and the Studio shows a confirmation with the exact command before calling it. SECURITY.md states that real installs are not exercised by the tests.

# The run, step by step

sequenceDiagram
    participant U as Python Studio
    participant R as PythonRunner
    participant P as Python process
    U->>U: consent check (PythonConsent)
    U->>R: run(code, host, options)
    R->>R: sandbox self-test, run directory, config.json
    R->>P: posix_spawn (fds 0-4 only)
    P->>P: setrlimit, parent watchdog, import orix
    P->>R: read_range (fd 4)
    R->>U: host.readRange on the main actor
    R-->>P: rows as JSON (fd 3)
    P->>R: write (buffered, not applied)
    R-->>P: ok
    P->>R: done (status, error, limits)
    R-->>U: PythonRunResult + audit entry
    U->>U: if succeeded: validate and apply all writes as one undo step

SVG fallback: Sequence of one Python run

  1. Refusals first. PythonRunner.run refuses (and writes an audit entry with outcome refused) when the interpreter executable is missing or when restricted mode is requested and the sandbox cannot be applied. A runner executes one script at a time (alreadyRunning).
  2. Run directory. PreparedRun creates orix-python-<UUID> in the user's temporary directory with mode 0700 and the sub-directories figures, mpl, work and tmp. It copies runner.py and orix.py into it, writes the user's code as script.py and the options as config.json.
  3. Environment. Built from an allow-list, not inherited: LANG, LC_ALL, LC_CTYPE, TZ are copied when present; then PATH=/usr/bin:/bin:/usr/sbin:/sbin, HOME=<run>/work, TMPDIR=<run>/tmp, XDG_CACHE_HOME=<run>/tmp, MPLBACKEND=Agg, MPLCONFIGDIR=<run>/mpl, ORIX_SHEET=1. Caller-supplied extras whose names start with PYTHON or DYLD_ are dropped.
  4. Spawn. ChildProcess uses posix_spawn directly because Foundation's Process cannot map extra descriptors. Flags: POSIX_SPAWN_SETPGROUP (the child leads a new process group), POSIX_SPAWN_SETSIGDEF, POSIX_SPAWN_SETSIGMASK, POSIX_SPAWN_CLOEXEC_DEFAULT. The child inherits exactly five descriptors: 0 = /dev/null, 1 and 2 = output pipes, 3 = answers from the app, 4 = requests to the app.
  5. Launch line. [/usr/bin/sandbox-exec -p <profile>] <python> -u -X utf8 -I -B <run>/runner.py <run>. -I ignores PYTHON* variables and keeps the per-user site directory and the script directory off sys.path; -B writes no .pyc; -u makes the console stream.
  6. Inside the child (runner.py, trusted code): apply setrlimit caps → start the parent watchdog thread → load orix.py as the module orix and connect it to descriptors 3 and 4 → compile and exec the user's script as __main__ → capture open matplotlib figures → send done.
  7. Supervision (ActiveRun.pump): three reader threads for stdout, stderr and the RPC pipe; a timeout timer; a 0.2-second watchdog that samples the process tree and its memory; a thread blocked in waitpid. When the interpreter exits, the whole family receives SIGKILL.
  8. Result. PythonRunResult carries stdout/stderr, the buffered writes, figure URLs, table previews, the error with its line number, protocol warnings and the limits the runner reports as applied. One audit entry is appended.
  9. Clean-up. The run directory is removed; when figures were produced only figures/ is kept until the caller calls discard() — Python Studio reads the PNG files into memory and discards at once.

# RPC over the extra pipes

The protocol is newline-delimited JSON. The script side (orix._Channel.call) writes one request object with an op and an increasing id on descriptor 4 and blocks reading one answer line on descriptor 3. The host answers {"id":n,"ok":true,"result":…} or {"id":n,"ok":false,"error":"…"}; a refused request raises orix.OrixError in the script.

op Direction of data Host behaviour
read_range app → script host.readRange(sheet, a1); answer holds sheet, resolved a1, date1904, errors, date_columns, rows
active_sheet, sheets, selection, used_range app → script answered from the live session
write script → app validated and buffered; never applied by SheetPython
figure script → app accepted only for a .png directly inside this run's figures directory; at most 200
table script → app a preview for the Tables tab; at most 5,000 preview rows kept per table, at most 200 tables
done script → app exit status, error info, applied rlimits
anything else — refused, recorded as a warning (at most 50 warnings)

Source: Packages/SheetPython/Sources/SheetPython/ActiveRun.swift (handle).

Cell values cross the boundary as PythonCell: null, number, string, boolean, or {"error":"#DIV/0!"}. Non-finite numbers are sent as null. Range answers are serialised by hand-written code (Wire.rangeAnswer) rather than JSONSerialization, because they are the hot path.

Host-side limits of the protocol:

Limit Value Constant
Cells returned by one read 8,000,000 SessionPythonHost.maxCellsPerRead
Cells buffered in writes per run 20,000,000 ActiveRun.maxBufferedCells
One write must fit 1,048,576 × 16,384 from its anchor; rows must be rectangular ActiveRun.handle
Number-format hint at most 64 characters, inside the written block ActiveRun.handle
One request line 512 MiB; larger lines are dropped with a warning ActiveRun.maxLineBytes
stdout + stderr kept 4 MiB (the rest is drained and dropped, with a notice) PythonRunOptions.maxOutputBytes

Source: Packages/SheetPython/Sources/SheetPython/ActiveRun.swift, Model.swift; Packages/FeaturePython/Sources/FeaturePython/SessionHost.swift.

# Reads

SessionPythonHost.read runs on the main actor and holds the calculation engine's lock for the duration of the copy, so a background recalculation cannot be observed half-way (the lock is described in calculation-graph.md and overview.md). Whole-row and whole-column references are clipped to the used range. The script receives computed values, not formulas. A column is flagged as a date column when every number in it carries a date/time number format; orix then converts those serials to datetime. Cells holding an array or a lambda are sent as the #CALC! error.

A script can read every sheet of the open workbook, hidden sheets included.

# Writes

Writes are buffered in the host and returned in PythonRunResult.writes. PythonStudioModel.finish applies them only when result.succeeded (exit status 0, no error, no timeout, no cancellation, no memory kill):

  1. PythonWriteApplier.plan resolves every write against the workbook without changing anything: the sheet must still exist, the block must fit, and with formulas: true every string must parse with FormulaParser in canonical syntax. Any failure cancels the whole batch ("Nothing was written").
  2. session.perform("Run Python Script") applies all planned writes through CalcEngine.set, cell by cell, as one undoable step.
  3. Number-format hints (dates, times) are applied only to cells that do not already have a date/time format.

Because writes are deferred, a read made after a write in the same run still sees the sheet as it was before the run. A failed, stopped or timed-out run writes nothing; the Changes tab then lists what the script wanted to write, marked as not applied.

# The orix module

orix.py (version string 1.0) is copied into every run directory and loaded by the runner under the name orix. pandas, numpy and polars are optional: only the methods that need them import them.

API Meaning
orix.active_sheet(), orix.sheet(name), orix.sheets() Sheet objects (sheet(name) matches case-insensitively)
orix.selection() the current selection as a Range
Sheet.name, Sheet.range(address), sheet["A1:D10"], Sheet.used_range() ranges of a sheet
Range.address, Range.shape resolved address, (rows, cols)
Range.values, Range.value list of rows; a single cell gives a scalar. Blank cells are None, error cells are CellError (a str subclass)
Range.to_numpy(dtype=None) numpy array
Range.to_dataframe(header=True, index=None, errors="nan") pandas DataFrame; errors="nan" turns error cells into missing values, "keep" keeps them
Range.to_polars(header=True) polars DataFrame
Range.write(value, index=None, header=True, transpose=False) buffers a write anchored at the range's top-left cell (scalars, lists, arrays, DataFrames, Series…)
Range.write_formula(formula) buffers formulas (strings starting with =)
orix.show(obj=None, name=None) DataFrames/arrays → Tables tab (first preview_rows rows, default 200); matplotlib figures → Plots tab
orix.to_serial(value, date1904=False), orix.from_serial(serial, date1904=False) date/time ⇄ serial number
orix.MAX_ROWS, orix.MAX_COLS, orix.RESTRICTED 1,048,576, 16,384, whether the run is restricted
orix.OrixError, orix.CellError request refused by the app; error value read from a cell

Source: Packages/SheetPython/Sources/SheetPython/Resources/python/orix.py.

Date conversion follows the workbook's date system, including the 1900 leap-year quirk: from_serial maps serials 59 and 60 both to 1900-02-28, and to_serial subtracts one day below serial 61 (see ../mathematics/numerical-precision.md). Time-zone-aware datetimes are written as their wall-clock time; the zone is dropped.

Open matplotlib figures are captured automatically at the end of the script (_capture_open_figures), so plt.show() is not required; the backend is forced to Agg.

# Restricted mode

Restricted mode is on at every launch of the app. PythonSandbox.profile generates an SBPL profile per run, in the style "allow default, then deny":

Rule in the profile Purpose
(deny network*), (deny system-socket) no network, unless allowNetwork is set (not exposed in the 1.0 UI)
(deny mach-lookup) + allow-list of 9 services user database, logging, preferences, fonts only
(deny appleevent-send), (deny lsopen), (deny job-creation) no scripting of other apps, no launching through Launch Services or launchd
(deny signal) + (allow signal (target same-sandbox)) cannot signal processes outside the sandbox
(deny process-exec*) + allow for the interpreter executable and its installation trees no other program can be executed
(deny file-write*) + allow for the run directory, explicitly granted writable paths, /dev/null, /dev/dtracehelper no writes elsewhere
(deny file-read* …) for /Users, /Volumes, /Network, /Library/Keychains, /private/var/root, /private/tmp, /private/var/tmp, the per-user temp/cache tree then re-allowed for the run directory, the interpreter's prefixes and site-packages, and explicitly granted paths

Source: Packages/SheetPython/Sources/SheetPython/Sandbox.swift (profile, deniedReadRoots, machAllowList).

Fail closed. Before the first restricted run of a launch, unavailabilityReason() really starts /usr/bin/sandbox-exec -p "(version 1)(allow default)(deny network*)" /usr/bin/true. If the launcher is missing or cannot apply a profile, restricted runs are refused with sandboxUnavailable; they never fall back to unrestricted.

# Verified guarantees

SECURITY.md marks each claim with the test that asserts it against a real interpreter. The same probe script is asserted to succeed without the sandbox, so the tests measure the sandbox and not an unrelated failure. Observations were made on macOS 27.0 (26A5425a) with Homebrew Python 3.14.4 and Xcode Python 3.9.6.

Guarantee in restricted mode Test (in Packages/SheetPython/Tests/SheetPythonTests)
No outbound TCP (even to 127.0.0.1), no UDP send, no listening socket RestrictedModeTests.restrictedModeDeniesWhatItClaims
No file creation or modification outside the run directory and granted writable paths same
No reading of /Users/**, /private/tmp, /Volumes, nor of the per-user temp/cache tree outside the run directory same
Granted readablePaths are readable but not writable; writablePaths are writable same
No other program can be executed (curl, ls, a shell via os.system, osascript) same
The interpreter may re-execute itself (multiprocessing); children inherit the sandbox same
No signal to processes outside the sandbox same
Missing or failing sandbox-exec → the run is refused RestrictedModeTests.restrictedModeRefusesToRunWithoutTheSandbox
Network returns only with the explicit option networkCanBeGrantedExplicitly
numpy, pandas, matplotlib, scipy, statsmodels, scikit-learn, polars, duckdb, pyarrow, sympy work under the profile scientificStackWorksRestricted

Source: Packages/SheetPython/SECURITY.md §5 (test names as given there).

# What restricted mode does not guarantee

  • Rules without a dedicated test. (deny mach-lookup), (deny appleevent-send), (deny lsopen), (deny job-creation) and the read denials for /private/var/tmp, /Network, /Library/Keychains and /private/var/root are in the profile but are not covered by a dedicated test. SECURITY.md says not to present them to users as guarantees.
  • Reads outside the denied roots. /Applications, /usr, /opt, /Library (except Keychains), /System, /private/etc remain readable.
  • The workbook itself. The script can read the whole open workbook through orix. The sandbox stops it from sending that data over the network or to a file, not from reading it.
  • It is not a virtual machine. sandbox-exec is Apple's Seatbelt launcher; Apple marks it deprecated. The script and native extension modules run on the real kernel.
  • Memory. RLIMIT_AS, RLIMIT_DATA and RLIMIT_RSS are not enforceable for a running interpreter on macOS (asserted by addressSpaceLimitIsNotEnforceableButTheWatchdogIs). The host instead samples the physical footprint of the process tree every 0.2 s and kills it above maxMemoryBytes; a script can overshoot between two samples.
  • Fast daemonisation. A process that detaches faster than one 0.2-second sampling interval and is re-parented to launchd could survive termination; in restricted mode it stays inside the sandbox and under RLIMIT_CPU.
  • Disk. Each file is capped by RLIMIT_FSIZE, but the number of files in the run directory is not.
  • Unrestricted mode has no sandbox at all. Only the resource limits and the environment isolation below apply; the script has the user's file and network access.

# Limits in both modes

Limit Default Mechanism
Wall-clock time 60 s (user setting, stored in UserDefaults) host timer → SIGTERM to the group and every observed descendant, SIGKILL after 0.5 s
Stop button, task cancellation — same termination path
CPU time per process 300 s RLIMIT_CPU, soft = n (SIGXCPU), hard = n + 1 (SIGKILL)
File size 1 GiB RLIMIT_FSIZE, soft = hard
Open files 256 RLIMIT_NOFILE, soft = hard
Core dumps 0 RLIMIT_CORE
Memory of the process tree 8 GiB host watchdog (0.2 s sampling)
Output 4 MiB host keeps the first bytes, drains the rest
App quits — PythonRunner.killAllRunningScripts() on NSApplication.willTerminate: SIGKILL to every family, run directories removed
App crashes — a daemon thread in runner.py polls getppid() every 0.5 s and kills its own process group when re-parented
Stale run directories older than 24 h removed at a later launch by removeStaleRunDirectories()

Source: Packages/SheetPython/Sources/SheetPython/Model.swift (PythonRunOptions), ActiveRun.swift, Runner.swift, Resources/python/runner.py; Packages/FeaturePython/Sources/FeaturePython/StudioModel.swift (timeout setting).

The rlimits are applied by runner.py before the user's code is compiled. .pth files of the interpreter's own site-packages run earlier (inside the sandbox in restricted mode). A script that blocks in native code without releasing the GIL delays the parent watchdog thread; RLIMIT_CPU still applies.

The only file that travels between runs is matplotlib's fontlist-v<N>.json cache (regular file, under 8 MiB, never overwritten once present), kept in ~/Library/Application Support/ORIX Sheet/python/cache/ with separate copies for restricted and unrestricted runs.

Scripts are stored in the workbook as data, in attachments named python/<id>.json; loading only decodes JSON. The single entry point to execution is PythonStudioModel.run(), which evaluates PythonConsent.requirement first.

Situation Consent sheet shown?
Script arrived with the workbook, first run in this session yes — full code, interpreter, mode, time limit and SHA-256 are displayed
Same script, exact text previously recorded with "Always trust this script" for this workbook and this mode no
Script created by the user in this session, restricted mode no
Any script, unrestricted mode, not yet reviewed for unrestricted mode yes
Script already reviewed in this session ("Run once" or "Always trust"), then edited by the user no, until the app quits

Source: Packages/FeaturePython/Sources/FeaturePython/Consent.swift (requirement, resolve).

Further rules implemented in StudioModel+Run.swift and StudioModel.swift:

  • The answer to the sheet is valid only for the exact text and mode that were displayed. If the script changed while the sheet was open, nothing runs.
  • "Run once" records nothing. "Always trust" stores the SHA-256 of the exact code together with a workbook identifier (a UUID kept in the attachment python/workbook-id) and whether unrestricted execution was approved. Any change to the text gives a different hash.
  • Turning restricted mode off requires its own confirmation, once per app launch (unrestrictedConfirmedThisSession). Approval for restricted mode never covers an unrestricted run.

The last row of the table is a deliberate usability choice documented in the source: after a review, the user's own edits run without a new prompt for the rest of the session.

The trust store is ~/Library/Application Support/ORIX Sheet/python-trust.json (mode 0600). It is an ordinary user file: a script run unrestricted could edit it. A restricted script can neither read nor write it, because the home directory is denied.

# Audit log

PythonRunner.run appends one JSON line per run, and per refused run, to ~/Library/Application Support/ORIX Sheet/python-audit.jsonl (created with mode 0600). Fields: timestamp, workbook name, script name, SHA-256 of the code, interpreter path and version, restricted flag, network flag, duration, exit status, outcome (ok, error, timeout, cancelled, killed, refused), error summary and number of writes. When the file exceeds 4 MiB, the most recent half is kept.

It is a record for the user, not a tamper-proof log, for the same reason as the trust store.

# Relationship with the assistant

The on-device assistant can propose Python code (intelligence.md). That code is text: it is syntax-checked by compile() in a separate isolated interpreter and never executed by the assistant. "Open in Python Studio" copies the code to the clipboard and shows the Studio pane (AssistantModel.openInPythonStudio); the user pastes it into the editor, and running it is subject to the consent rules above like any other script.

# Tests and measurements

Packages/SheetPython/Tests contains 57 @Test functions and Packages/FeaturePython/Tests 40 (counts of test functions in the source on 2026-11-21; see docs/reports/tests.json for executed totals when present). The tests start real interpreters and need at least one Python 3.9+ on the machine. How to run them is described in ../development/testing.md.

No benchmark of the Python exchange (rows per second over the pipes, start-up latency) is stored in the repository at the time of writing; none is quoted here.

# Not implemented

  • No bundled or embedded interpreter: Python Studio requires a Python 3.9+ installation.
  • No user-defined worksheet functions in Python, and no formula that calls Python; scripts are run explicitly from the Studio.
  • No UI for allowNetwork, readablePaths, writablePaths or includeUserSite; these PythonRunOptions exist for callers and tests only. Packages installed with pip install --user are therefore not importable from the Studio.
  • No persistent interpreter session between runs: every run is a fresh process.
  • No package installation in restricted mode, and no installation into interpreters other than the managed environment.
Files main
Menu

Theme