Persistence
How a workbook gets to and from disk: the native .orixsheet package, saving through NSDocument,
the "import as untitled" policy for every other format, and the export preflight.
Code: Packages/SheetIO (formats, no UI), App/Sources/WorkbookDocument.swift (the NSDocument
subclass), App/Sources/CompatibilityReport.swift (the report sheet). The byte-level specification
of the native format is Packages/SheetIO/NATIVE-FORMAT.md; this document summarises it and
describes how the application uses it. Per-format capabilities are in
../compatibility/formats.md.
The policy in one paragraph
Only .orixsheet documents are edited in place. Opening any other format (XLSX, XLSM, ODS, CSV, TSV,
JSON, JSON Lines) imports it into an untitled ORIX workbook: the source file is never written
again, Save asks for a new location and always writes .orixsheet, and producing a foreign format
is an explicit File ▸ Export To command preceded by a report of what the target format cannot hold.
The reason is stated in the class comment of WorkbookDocument: saving can then never silently drop
what the foreign format cannot represent.
flowchart TD
Native["Open .orixsheet"] --> ReadN["NativeFormat.read<br/>verify manifest hashes"]
Foreign["Open .xlsx .xlsm .ods .csv .tsv .json"] --> ReadF["WorkbookReader.load<br/>format detected from content"]
ReadN --> Trust["Session with cached values trusted<br/>no recalculation"]
ReadF --> Recalc["Session recalculated in full"]
Recalc --> Untitled["fileURL cleared, type set to native<br/>document marked edited"]
Untitled --> Report["Import report<br/>issues, macros, external links"]
Trust --> Edit["Editing"]
Report --> Edit
Edit --> Save["Save and autosave<br/>NativeFormat.write only"]
Edit --> Export["Export To<br/>preflight, report, then write"]
SVG fallback:
The .orixsheet package
A document is a directory that Finder shows as one file: the exported type
ai.spboucher.orix.sheet.workbook conforms to com.apple.package, and the document type sets
LSTypeIsPackage (App/Resources/Info.plist).
Budget.orixsheet/
├── manifest.json format id, versions, dates, SHA-256 + size of every other part
├── workbook.json sheet list, defined names, style table, properties, attachment index
├── sheets/
│ ├── <id>.cells binary cell store of the sheet with that id (chunked, zlib)
│ └── <id>.meta.json layout and features of the sheet (SheetMeta)
├── attachments/ Workbook.attachments (scripts, queries, workflows…), one file each
├── preserved.plist optional: raw parts kept from an imported XLSX (charts, VBA, pivots…)
└── preview.png optional: thumbnail| Part | Encoding | Content |
|---|---|---|
manifest.json |
JSON, sorted keys | format = orixsheet, formatVersion = 1, minReaderVersion = 1, app, appVersion, created, modified, and parts: path → {sha256, size} for every other file |
workbook.json |
JSON | sheets in tab order (id, name, part paths, cellCount), defined names, the style table in index order, date1904, active sheet, default font, document properties, attachment index, hasPreservedParts |
sheets/<id>.meta.json |
JSON | column widths, row heights, hidden/filtered rows, outline levels, merges, frozen panes, tables, conditional formats, validations, comments, hyperlinks, autofilter, embedded objects (payload in base64), print settings, view flags, tab colour, hidden/protected flags |
sheets/<id>.cells |
binary, little-endian | magic OXCL, version 1, cell count, then independent zlib chunks of 50,000 cells, each with its own string table and CRC-32 |
attachments/<n>-<name> |
raw bytes | file name derived from the index and a sanitised suffix ([A-Za-z0-9._-], last 48 characters) so that an attachment name can never become a path |
preserved.plist |
binary property list | PreservedParts: bytes of XLSX parts ORIX Sheet does not model, with the sheet ids they belong to; never interpreted or executed |
Source: Packages/SheetIO/NATIVE-FORMAT.md; constants checked in Native/NativeSchema.swift
(currentVersion = 1), Native/NativeCells.swift (cellsPerChunk = 50_000,
maxChunkRawBytes = 256 << 20) and Native/NativeFormat.swift (safeFileName).
Design points worth knowing:
- Sheet parts are named after
Sheet.id, which is stable across renames and reordering; tab order is the order of thesheetsarray inworkbook.json. - Formulas are stored as canonical text (en-US syntax, no leading
=) together with the cached value, the spill range and the volatile flag; identical texts in a chunk are parsed once. A document therefore opens without recalculation. - Numbers: integral values below 9·10¹⁵ are stored as zig-zag varints, everything else as the
8 bytes of the IEEE-754 double, so −0.0, subnormals and NaN payloads round-trip. In the JSON parts,
non-finite numbers are the strings
"+inf","-inf","nan", and dates are seconds since 2001-01-01 UTC as a double. - What is not stored: LAMBDA values held by a cell (saved as
#CALC!, recomputed on load, reported as alossissue), the dependency graph and dirty flags (rebuilt), the private sheet-id counter (restored to "largest id + 1").
Integrity
Every part is hashed with SHA-256 (CryptoKit) at save time. On load, each part listed in the manifest is compared by size and hash; in addition each cell chunk carries its own CRC-32.
| Situation | Default behaviour | With LoadOptions.strictIntegrity |
|---|---|---|
workbook.json missing or undecodable |
error corrupted(part: "workbook.json", …) |
same |
workbook.json hash mismatch but decodable |
opens, warning |
error |
| a cell chunk fails its CRC, zlib or structure check | that chunk is skipped, other chunks and sheets load; loss issue naming the sheet, part and chunk |
error |
a .cells part missing or truncated |
the sheet opens with what could be read (possibly empty); loss |
error |
a .meta.json part missing or undecodable |
cells load, the sheet layout is reset to defaults; loss |
error |
| attachment missing / changed | loss / warning |
error for a hash mismatch or missing part |
manifest missing or unreadable (and workbook.json present) |
opens without verification; warning |
error |
directory with neither manifest nor workbook.json |
error: not an ORIX Sheet document | same |
minReaderVersion newer than this build |
error unsupportedVersion |
same |
formatVersion newer, minReaderVersion readable |
opens, unknown content ignored, warning that it would be lost on save |
same |
Source: NativeFormat.read(parts:options:progress:); behaviour covered by
NativeFormatTests.damagedPartsAreReportedPreciselyAndTheRestIsRecovered and
versioningAndForwardCompatibility.
The application opens documents with the default options, that is, in recovery mode: a damaged
document opens with whatever could be read, and the issues are shown in the report sheet (see
"Known gaps" for a wording problem in that case). NativeFormat.verify(url:) checks a package
against its manifest without loading it; the app test testNativeSaveReopenIsLossless uses it.
The reader does not trust sizes found in the file: chunk lengths are bounded (256 MiB raw),
decompression stops at the declared size, indices are range-checked, a package may contain at most
100,000 files, symbolic links inside the package are skipped, and LoadOptions.maxCells
(200 million) caps the total.
The integrity scheme detects accidental damage. It is not a signature: anyone who edits a part can also rewrite the manifest. See ../security.md.
Versioning
formatVersion is the schema the writer used; minReaderVersion the oldest reader that can open the
file without misreading it. Unknown JSON keys and unknown files in the package are ignored. Lists
whose elements may gain new variants (tables, conditionalFormats, validations, objects) are
decoded element by element (LossyArray), so one unknown element is skipped and counted instead of
making the sheet unreadable. NativeFormat.migrations is the hook that upgrades the parts of an
older package before decoding; version 1 is the first schema, so the table is empty and no migration
has ever been exercised on a real older file.
Saving
What NativeFormat.write guarantees
The package is built completely in memory (build), written into a hidden sibling directory
.<name>.saving-<uuid>, and then swapped in with FileManager.replaceItemAt (or moved when the
destination does not exist). On any failure before the swap the temporary directory is removed and
the previous document is untouched. A successful save replaces the whole package, so parts of deleted
sheets do not linger.
NativeFormatTests.interruptedSaveLeavesTheOriginalIntact injects a failure just before the swap and
checks that the document on disk is byte-for-byte the previous version, that it still verifies, and
that no temporary directory is left; it also checks that a failed XLSX save leaves the existing file
intact (single-file formats use Data.write(options: .atomic)).
How the application saves
WorkbookDocument overrides the URL-based pair read(from:ofType:) / write(to:ofType:):
sequenceDiagram
participant U as User or autosave
participant D as NSDocument machinery
participant W as WorkbookDocument
participant E as CalcEngine lock
participant N as NativeFormat
U->>D: save
D->>W: write(to url, ofType native)
W->>W: set properties.modified and created
W->>E: lock
W->>N: write(workbook, to url, preserved)
N->>N: build parts and SHA-256 manifest
N->>N: write hidden sibling directory
N->>N: replaceItemAt or move
N-->>W: issues (discarded)
W->>E: unlock
W-->>D: return
D-->>U: document saved
SVG fallback:
Facts from the code:
- Only the native type is writable:
writableTypes(for:)returns[nativeType]andfileNameExtension(forType:saveOperation:)returnsorixsheet, whatever the save operation. - Reading and writing happen on the main actor (
MainActor.assumeIsolatedinside thenonisolatedoverrides;canConcurrentlyReadDocumentsreturnsfalse;canAsynchronouslyWrite(to:ofType:for:)is not overridden). A save therefore blocks the UI for its duration; the measured write time is 0.58 s per million cells (../compatibility/formats.md). - The engine lock is held for the whole write, so a background recalculation cannot store values while the cells are being serialised.
properties.modifiedis set at every save,properties.createdat the first one; the manifest'sappVersionis the bundle'sCFBundleShortVersionString.- The
PreservedPartsof an imported Excel file are passed to every save, so they travel inside the.orixsheetpackage and are available to a later XLSX export. - The issues returned by the writer are discarded on a native save. The only issue the native writer can produce is the LAMBDA-value loss described above.
NativeWriteOptions.previewis never set by the application: documents saved by the app contain nopreview.png, and there is no Quick Look extension in the project.
NativeFormat also offers a FileWrapper API (fileWrappers(for:), read(fileWrapper:)), covered
by NativeFormatTests.fileWrappersPreviewAttachmentsAndPreservedParts. The application does not use
it: WorkbookDocument overrides the URL-based read(from:ofType:) and write(to:ofType:), as
Packages/SheetIO/NATIVE-FORMAT.md states.
Autosave and versions
WorkbookDocument.autosavesInPlace returns true. Nothing else about autosaving is customised:
preservesVersions, autosavingFileType, autosavesDrafts and the autosave interval keep their
AppKit defaults, and NSSupportsSuddenTermination / NSSupportsAutomaticTermination are false in
Info.plist. Consequently:
- a saved
.orixsheetdocument is autosaved in place by AppKit and has no "Save As"; the File menu offers Save…, Duplicate, Rename…, Move To… and Revert To Saved (App/Sources/MainMenu.swift); - change tracking has two sources. Undoable changes:
WorkbookSession.performregisters one undo action per transaction with the document'sUndoManager, which marks the document edited. Non-undoable workbook changes:WorkbookDocument.adoptsetssession.onChange = { updateChangeCount(.changeDone) }, and the features callsession.onChange?()after storing something in the workbook outside the undo stack (a Python script, a transform pipeline, a figure setting, scenario data, ignored auditor findings, a workflow attachment), so those changes also mark the document edited; - untitled documents (new, from a template, or imported) are handled by AppKit's standard untitled-document autosave.
Not verified in the repository: no test exercises an actual autosave cycle, the system version
browser, Revert To, crash recovery of an untitled document, or iCloud Drive / file-coordination
scenarios. What is tested is the write/read pair called directly (testNativeSaveReopenIsLossless)
and the atomicity of NativeFormat.write. The behaviour described in the list above is AppKit's
documented behaviour for autosavesInPlace, not something this project measured.
The app is not sandboxed (ENABLE_APP_SANDBOX: NO in project.yml), so no security-scoped bookmark
handling is involved.
Opening
readOnMain(from:ofType:):
- Determines the format with
FileFormat.detect(url:)— content first (package directory, ZIP entries,mimetype, compound-file magic), then extension — and only falls back to the type name given by AppKit. A file with a misleading extension is opened for what it is. - Calls
WorkbookReader.load(url:format:)with defaultLoadOptions. - Refuses a workbook without any sheet.
- Keeps
preserved,issues,hasMacros,hasExternalLinksfrom theLoadResult. - Creates the session: for
.orixsheet, cached values are trusted (CalcEngine(workbook:trustCachedValues: true): the graph is built, nothing is scheduled); for any other format the workbook is recalculated in full by ORIX Sheet's engine (session.recalculateAll()), so displayed values are ours, not the cached values of the producing application.
Loading is synchronous on the main thread; WorkbookReader.load accepts a progress callback but the
document class does not pass one, so there is no progress indicator while a large file opens.
Import as untitled
For a foreign format, makeWindowControllers() then:
- sets
fileURL = nil,fileType = nativeType,displayName= the source name without extension; - calls
updateChangeCount(.changeDone), so closing the window asks whether to save; - shows the import report 0.4 s later when there are issues, macros or external links.
The app test testForeignFilesAreImportedAsUntitledAndExportIsPreflighted checks that fileURL is
nil, that the type is native, that formulas were recalculated and that the source file's bytes are
unchanged.
The report (CompatibilityReport) lists issues sorted by severity — Lost (loss), Changed
(warning), Note (info) — with feature, location and detail, and a Copy Report button.
Repeated issues are merged by IssueCollector into one line with "(×N)" and up to five locations.
Two fixed warnings are added by the document: macros are present and are never run (kept only for an
.xlsm export); the workbook refers to other files and external data is not fetched. The report can
be reopened with File ▸ Import Report….
Info.plist declares the role Editor (rank Owner) for the native workbook type only. The Excel,
delimited-text, OpenDocument and JSON types are declared with the role Viewer, which matches the
behaviour described here: imported, never saved in place.
Export and preflight
File ▸ Export To offers XLSX, XLSM, ODS, CSV (active sheet), TSV (active sheet), JSON (active
sheet), PDF (active sheet) and PDF (all visible sheets). JSON Lines can be read, and written by
SheetIO, but has no export menu entry.
WorkbookDocument.export(as:):
- commits a pending cell edit (and aborts if the entry is invalid);
- runs
WorkbookWriter.preflight(workbook, format:, preserved:)— a pure function that computes what the target cannot keep, without writing anything; - if the list is empty, shows the save panel directly; otherwise shows the report with Cancel / Export Anyway;
- writes under the engine lock with
WorkbookWriter.save. CSV, TSV and JSON export the active sheet; CSV and TSV write values as displayed (through the number-format engine), not raw values; - shows "Exported " in the status bar. The document itself is not modified, renamed or marked clean.
What preflight reports, by target:
| Target | Reported as loss |
Reported as warning / info |
|---|---|---|
.orixsheet |
cells holding a LAMBDA value | — |
.xlsx / .xlsm |
VBA project when saving as .xlsx; embedded objects (ORIX charts, figures, pivots); attachments; text over 32,767 characters; LAMBDA values; more than 65,530 hyperlinks on a sheet; colour scales with other than 2 or 3 points; invalid or duplicate sheet names, no sheet, grid overflow (saving will fail) |
more than about 65,490 cell formats; threaded comments written as notes; charts or pivots of a deleted sheet dropped |
.ods |
conditional formats, validations, tables (written as plain ranges), autofilter, objects, outline, page setup, sheet-scoped names, attachments | 1904 date system converted; tab colour |
.csv .tsv .json |
other sheets with content; formulas exported as values; tables, validations, comments, hyperlinks, objects; defined names; content preserved from Excel | formatting not part of the format |
Source: Packages/SheetIO/Sources/SheetIO/Preflight.swift and WorkbookIO.swift
(WorkbookWriter.preflight).
Preflight is conservative about what it scans: the XLSX preflight visits every cell once (text length, LAMBDA values). The writers themselves return the same kind of issue list after writing (for instance truncated text with the cell address); the export path shows the preflight list before writing and does not display the post-write list.
PDF export does not go through SheetIO: it uses the print renderer
(grid-engine.md) and writes the data atomically.
Known gaps
- Blocking I/O: open, save and export run synchronously on the main thread without progress UI.
- No preview or Quick Look:
preview.pngis supported by the format but never written by the app. - Autosave, versions, revert and crash recovery are untested (see above).
- Report wording for native documents: when a native document opens with integrity issues, the
same sheet is shown with the title "Imported from" followed by an empty name and the message "The
file was converted into an ORIX Sheet workbook", because
showImportReport()has a single wording intended for foreign files. - Issues of a native save are not surfaced (LAMBDA values stored as
#CALC!). - Integrity is not authenticity: hashes are recomputable by anyone who edits the package.
- No encryption or password protection of documents; encrypted ZIP archives and password-protected Office files are refused on import.
- No file locking or merge: two applications writing the same package are arbitrated by
NSDocument's standard conflict detection only, which this project has not tested.