SPB Git

Repository Public

persistence.md 19.7 KB

# Persistence

How a workbook gets to and from disk: the native .orixsheet package, saving through NSDocument, the "import as untitled" policy for every other format, and the export preflight.

Code: Packages/SheetIO (formats, no UI), App/Sources/WorkbookDocument.swift (the NSDocument subclass), App/Sources/CompatibilityReport.swift (the report sheet). The byte-level specification of the native format is Packages/SheetIO/NATIVE-FORMAT.md; this document summarises it and describes how the application uses it. Per-format capabilities are in ../compatibility/formats.md.

# The policy in one paragraph

Only .orixsheet documents are edited in place. Opening any other format (XLSX, XLSM, ODS, CSV, TSV, JSON, JSON Lines) imports it into an untitled ORIX workbook: the source file is never written again, Save asks for a new location and always writes .orixsheet, and producing a foreign format is an explicit File ▸ Export To command preceded by a report of what the target format cannot hold. The reason is stated in the class comment of WorkbookDocument: saving can then never silently drop what the foreign format cannot represent.

flowchart TD
    Native["Open .orixsheet"] --> ReadN["NativeFormat.read<br/>verify manifest hashes"]
    Foreign["Open .xlsx .xlsm .ods .csv .tsv .json"] --> ReadF["WorkbookReader.load<br/>format detected from content"]
    ReadN --> Trust["Session with cached values trusted<br/>no recalculation"]
    ReadF --> Recalc["Session recalculated in full"]
    Recalc --> Untitled["fileURL cleared, type set to native<br/>document marked edited"]
    Untitled --> Report["Import report<br/>issues, macros, external links"]
    Trust --> Edit["Editing"]
    Report --> Edit
    Edit --> Save["Save and autosave<br/>NativeFormat.write only"]
    Edit --> Export["Export To<br/>preflight, report, then write"]

SVG fallback: Document flow: open, import, save, export

# The .orixsheet package

A document is a directory that Finder shows as one file: the exported type ai.spboucher.orix.sheet.workbook conforms to com.apple.package, and the document type sets LSTypeIsPackage (App/Resources/Info.plist).

text
Budget.orixsheet/
├── manifest.json          format id, versions, dates, SHA-256 + size of every other part
├── workbook.json          sheet list, defined names, style table, properties, attachment index
├── sheets/
│   ├── <id>.cells         binary cell store of the sheet with that id (chunked, zlib)
│   └── <id>.meta.json     layout and features of the sheet (SheetMeta)
├── attachments/           Workbook.attachments (scripts, queries, workflows…), one file each
├── preserved.plist        optional: raw parts kept from an imported XLSX (charts, VBA, pivots…)
└── preview.png            optional: thumbnail
Part Encoding Content
manifest.json JSON, sorted keys format = orixsheet, formatVersion = 1, minReaderVersion = 1, app, appVersion, created, modified, and parts: path → {sha256, size} for every other file
workbook.json JSON sheets in tab order (id, name, part paths, cellCount), defined names, the style table in index order, date1904, active sheet, default font, document properties, attachment index, hasPreservedParts
sheets/<id>.meta.json JSON column widths, row heights, hidden/filtered rows, outline levels, merges, frozen panes, tables, conditional formats, validations, comments, hyperlinks, autofilter, embedded objects (payload in base64), print settings, view flags, tab colour, hidden/protected flags
sheets/<id>.cells binary, little-endian magic OXCL, version 1, cell count, then independent zlib chunks of 50,000 cells, each with its own string table and CRC-32
attachments/<n>-<name> raw bytes file name derived from the index and a sanitised suffix ([A-Za-z0-9._-], last 48 characters) so that an attachment name can never become a path
preserved.plist binary property list PreservedParts: bytes of XLSX parts ORIX Sheet does not model, with the sheet ids they belong to; never interpreted or executed

Source: Packages/SheetIO/NATIVE-FORMAT.md; constants checked in Native/NativeSchema.swift (currentVersion = 1), Native/NativeCells.swift (cellsPerChunk = 50_000, maxChunkRawBytes = 256 << 20) and Native/NativeFormat.swift (safeFileName).

Design points worth knowing:

  • Sheet parts are named after Sheet.id, which is stable across renames and reordering; tab order is the order of the sheets array in workbook.json.
  • Formulas are stored as canonical text (en-US syntax, no leading =) together with the cached value, the spill range and the volatile flag; identical texts in a chunk are parsed once. A document therefore opens without recalculation.
  • Numbers: integral values below 9·10¹⁵ are stored as zig-zag varints, everything else as the 8 bytes of the IEEE-754 double, so −0.0, subnormals and NaN payloads round-trip. In the JSON parts, non-finite numbers are the strings "+inf", "-inf", "nan", and dates are seconds since 2001-01-01 UTC as a double.
  • What is not stored: LAMBDA values held by a cell (saved as #CALC!, recomputed on load, reported as a loss issue), the dependency graph and dirty flags (rebuilt), the private sheet-id counter (restored to "largest id + 1").

# Integrity

Every part is hashed with SHA-256 (CryptoKit) at save time. On load, each part listed in the manifest is compared by size and hash; in addition each cell chunk carries its own CRC-32.

Situation Default behaviour With LoadOptions.strictIntegrity
workbook.json missing or undecodable error corrupted(part: "workbook.json", …) same
workbook.json hash mismatch but decodable opens, warning error
a cell chunk fails its CRC, zlib or structure check that chunk is skipped, other chunks and sheets load; loss issue naming the sheet, part and chunk error
a .cells part missing or truncated the sheet opens with what could be read (possibly empty); loss error
a .meta.json part missing or undecodable cells load, the sheet layout is reset to defaults; loss error
attachment missing / changed loss / warning error for a hash mismatch or missing part
manifest missing or unreadable (and workbook.json present) opens without verification; warning error
directory with neither manifest nor workbook.json error: not an ORIX Sheet document same
minReaderVersion newer than this build error unsupportedVersion same
formatVersion newer, minReaderVersion readable opens, unknown content ignored, warning that it would be lost on save same

Source: NativeFormat.read(parts:options:progress:); behaviour covered by NativeFormatTests.damagedPartsAreReportedPreciselyAndTheRestIsRecovered and versioningAndForwardCompatibility.

The application opens documents with the default options, that is, in recovery mode: a damaged document opens with whatever could be read, and the issues are shown in the report sheet (see "Known gaps" for a wording problem in that case). NativeFormat.verify(url:) checks a package against its manifest without loading it; the app test testNativeSaveReopenIsLossless uses it.

The reader does not trust sizes found in the file: chunk lengths are bounded (256 MiB raw), decompression stops at the declared size, indices are range-checked, a package may contain at most 100,000 files, symbolic links inside the package are skipped, and LoadOptions.maxCells (200 million) caps the total.

The integrity scheme detects accidental damage. It is not a signature: anyone who edits a part can also rewrite the manifest. See ../security.md.

# Versioning

formatVersion is the schema the writer used; minReaderVersion the oldest reader that can open the file without misreading it. Unknown JSON keys and unknown files in the package are ignored. Lists whose elements may gain new variants (tables, conditionalFormats, validations, objects) are decoded element by element (LossyArray), so one unknown element is skipped and counted instead of making the sheet unreadable. NativeFormat.migrations is the hook that upgrades the parts of an older package before decoding; version 1 is the first schema, so the table is empty and no migration has ever been exercised on a real older file.

# Saving

# What NativeFormat.write guarantees

The package is built completely in memory (build), written into a hidden sibling directory .<name>.saving-<uuid>, and then swapped in with FileManager.replaceItemAt (or moved when the destination does not exist). On any failure before the swap the temporary directory is removed and the previous document is untouched. A successful save replaces the whole package, so parts of deleted sheets do not linger.

NativeFormatTests.interruptedSaveLeavesTheOriginalIntact injects a failure just before the swap and checks that the document on disk is byte-for-byte the previous version, that it still verifies, and that no temporary directory is left; it also checks that a failed XLSX save leaves the existing file intact (single-file formats use Data.write(options: .atomic)).

# How the application saves

WorkbookDocument overrides the URL-based pair read(from:ofType:) / write(to:ofType:):

sequenceDiagram
    participant U as User or autosave
    participant D as NSDocument machinery
    participant W as WorkbookDocument
    participant E as CalcEngine lock
    participant N as NativeFormat
    U->>D: save
    D->>W: write(to url, ofType native)
    W->>W: set properties.modified and created
    W->>E: lock
    W->>N: write(workbook, to url, preserved)
    N->>N: build parts and SHA-256 manifest
    N->>N: write hidden sibling directory
    N->>N: replaceItemAt or move
    N-->>W: issues (discarded)
    W->>E: unlock
    W-->>D: return
    D-->>U: document saved

SVG fallback: Sequence of a native save

Facts from the code:

  • Only the native type is writable: writableTypes(for:) returns [nativeType] and fileNameExtension(forType:saveOperation:) returns orixsheet, whatever the save operation.
  • Reading and writing happen on the main actor (MainActor.assumeIsolated inside the nonisolated overrides; canConcurrentlyReadDocuments returns false; canAsynchronouslyWrite(to:ofType:for:) is not overridden). A save therefore blocks the UI for its duration; the measured write time is 0.58 s per million cells (../compatibility/formats.md).
  • The engine lock is held for the whole write, so a background recalculation cannot store values while the cells are being serialised.
  • properties.modified is set at every save, properties.created at the first one; the manifest's appVersion is the bundle's CFBundleShortVersionString.
  • The PreservedParts of an imported Excel file are passed to every save, so they travel inside the .orixsheet package and are available to a later XLSX export.
  • The issues returned by the writer are discarded on a native save. The only issue the native writer can produce is the LAMBDA-value loss described above.
  • NativeWriteOptions.preview is never set by the application: documents saved by the app contain no preview.png, and there is no Quick Look extension in the project.

NativeFormat also offers a FileWrapper API (fileWrappers(for:), read(fileWrapper:)), covered by NativeFormatTests.fileWrappersPreviewAttachmentsAndPreservedParts. The application does not use it: WorkbookDocument overrides the URL-based read(from:ofType:) and write(to:ofType:), as Packages/SheetIO/NATIVE-FORMAT.md states.

# Autosave and versions

WorkbookDocument.autosavesInPlace returns true. Nothing else about autosaving is customised: preservesVersions, autosavingFileType, autosavesDrafts and the autosave interval keep their AppKit defaults, and NSSupportsSuddenTermination / NSSupportsAutomaticTermination are false in Info.plist. Consequently:

  • a saved .orixsheet document is autosaved in place by AppKit and has no "Save As"; the File menu offers Save…, Duplicate, Rename…, Move To… and Revert To Saved (App/Sources/MainMenu.swift);
  • change tracking has two sources. Undoable changes: WorkbookSession.perform registers one undo action per transaction with the document's UndoManager, which marks the document edited. Non-undoable workbook changes: WorkbookDocument.adopt sets session.onChange = { updateChangeCount(.changeDone) }, and the features call session.onChange?() after storing something in the workbook outside the undo stack (a Python script, a transform pipeline, a figure setting, scenario data, ignored auditor findings, a workflow attachment), so those changes also mark the document edited;
  • untitled documents (new, from a template, or imported) are handled by AppKit's standard untitled-document autosave.

Not verified in the repository: no test exercises an actual autosave cycle, the system version browser, Revert To, crash recovery of an untitled document, or iCloud Drive / file-coordination scenarios. What is tested is the write/read pair called directly (testNativeSaveReopenIsLossless) and the atomicity of NativeFormat.write. The behaviour described in the list above is AppKit's documented behaviour for autosavesInPlace, not something this project measured.

The app is not sandboxed (ENABLE_APP_SANDBOX: NO in project.yml), so no security-scoped bookmark handling is involved.

# Opening

readOnMain(from:ofType:):

  1. Determines the format with FileFormat.detect(url:) — content first (package directory, ZIP entries, mimetype, compound-file magic), then extension — and only falls back to the type name given by AppKit. A file with a misleading extension is opened for what it is.
  2. Calls WorkbookReader.load(url:format:) with default LoadOptions.
  3. Refuses a workbook without any sheet.
  4. Keeps preserved, issues, hasMacros, hasExternalLinks from the LoadResult.
  5. Creates the session: for .orixsheet, cached values are trusted (CalcEngine(workbook:trustCachedValues: true): the graph is built, nothing is scheduled); for any other format the workbook is recalculated in full by ORIX Sheet's engine (session.recalculateAll()), so displayed values are ours, not the cached values of the producing application.

Loading is synchronous on the main thread; WorkbookReader.load accepts a progress callback but the document class does not pass one, so there is no progress indicator while a large file opens.

# Import as untitled

For a foreign format, makeWindowControllers() then:

  • sets fileURL = nil, fileType = nativeType, displayName = the source name without extension;
  • calls updateChangeCount(.changeDone), so closing the window asks whether to save;
  • shows the import report 0.4 s later when there are issues, macros or external links.

The app test testForeignFilesAreImportedAsUntitledAndExportIsPreflighted checks that fileURL is nil, that the type is native, that formulas were recalculated and that the source file's bytes are unchanged.

The report (CompatibilityReport) lists issues sorted by severity — Lost (loss), Changed (warning), Note (info) — with feature, location and detail, and a Copy Report button. Repeated issues are merged by IssueCollector into one line with "(×N)" and up to five locations. Two fixed warnings are added by the document: macros are present and are never run (kept only for an .xlsm export); the workbook refers to other files and external data is not fetched. The report can be reopened with File ▸ Import Report….

Info.plist declares the role Editor (rank Owner) for the native workbook type only. The Excel, delimited-text, OpenDocument and JSON types are declared with the role Viewer, which matches the behaviour described here: imported, never saved in place.

# Export and preflight

File ▸ Export To offers XLSX, XLSM, ODS, CSV (active sheet), TSV (active sheet), JSON (active sheet), PDF (active sheet) and PDF (all visible sheets). JSON Lines can be read, and written by SheetIO, but has no export menu entry.

WorkbookDocument.export(as:):

  1. commits a pending cell edit (and aborts if the entry is invalid);
  2. runs WorkbookWriter.preflight(workbook, format:, preserved:) — a pure function that computes what the target cannot keep, without writing anything;
  3. if the list is empty, shows the save panel directly; otherwise shows the report with Cancel / Export Anyway;
  4. writes under the engine lock with WorkbookWriter.save. CSV, TSV and JSON export the active sheet; CSV and TSV write values as displayed (through the number-format engine), not raw values;
  5. shows "Exported " in the status bar. The document itself is not modified, renamed or marked clean.

What preflight reports, by target:

Target Reported as loss Reported as warning / info
.orixsheet cells holding a LAMBDA value —
.xlsx / .xlsm VBA project when saving as .xlsx; embedded objects (ORIX charts, figures, pivots); attachments; text over 32,767 characters; LAMBDA values; more than 65,530 hyperlinks on a sheet; colour scales with other than 2 or 3 points; invalid or duplicate sheet names, no sheet, grid overflow (saving will fail) more than about 65,490 cell formats; threaded comments written as notes; charts or pivots of a deleted sheet dropped
.ods conditional formats, validations, tables (written as plain ranges), autofilter, objects, outline, page setup, sheet-scoped names, attachments 1904 date system converted; tab colour
.csv .tsv .json other sheets with content; formulas exported as values; tables, validations, comments, hyperlinks, objects; defined names; content preserved from Excel formatting not part of the format

Source: Packages/SheetIO/Sources/SheetIO/Preflight.swift and WorkbookIO.swift (WorkbookWriter.preflight).

Preflight is conservative about what it scans: the XLSX preflight visits every cell once (text length, LAMBDA values). The writers themselves return the same kind of issue list after writing (for instance truncated text with the cell address); the export path shows the preflight list before writing and does not display the post-write list.

PDF export does not go through SheetIO: it uses the print renderer (grid-engine.md) and writes the data atomically.

# Known gaps

  • Blocking I/O: open, save and export run synchronously on the main thread without progress UI.
  • No preview or Quick Look: preview.png is supported by the format but never written by the app.
  • Autosave, versions, revert and crash recovery are untested (see above).
  • Report wording for native documents: when a native document opens with integrity issues, the same sheet is shown with the title "Imported from" followed by an empty name and the message "The file was converted into an ORIX Sheet workbook", because showImportReport() has a single wording intended for foreign files.
  • Issues of a native save are not surfaced (LAMBDA values stored as #CALC!).
  • Integrity is not authenticity: hashes are recomputable by anyone who edits the package.
  • No encryption or password protection of documents; encrypted ZIP archives and password-protected Office files are refused on import.
  • No file locking or merge: two applications writing the same package are arbitrated by NSDocument's standard conflict detection only, which this project has not tested.
Files main
Menu

Theme