Repository Public
1# Persistence23How a workbook gets to and from disk: the native `.orixsheet` package, saving through `NSDocument`,4the "import as untitled" policy for every other format, and the export preflight.56Code: `Packages/SheetIO` (formats, no UI), `App/Sources/WorkbookDocument.swift` (the `NSDocument`7subclass), `App/Sources/CompatibilityReport.swift` (the report sheet). The byte-level specification8of the native format is `Packages/SheetIO/NATIVE-FORMAT.md`; this document summarises it and9describes how the application uses it. Per-format capabilities are in10[../compatibility/formats.md](../compatibility/formats.md).1112## The policy in one paragraph1314Only `.orixsheet` documents are edited in place. Opening any other format (XLSX, XLSM, ODS, CSV, TSV,15JSON, JSON Lines) **imports** it into an untitled ORIX workbook: the source file is never written16again, *Save* asks for a new location and always writes `.orixsheet`, and producing a foreign format17is an explicit *File ▸ Export To* command preceded by a report of what the target format cannot hold.18The reason is stated in the class comment of `WorkbookDocument`: saving can then never silently drop19what the foreign format cannot represent.2021<!-- diagram: persistence-document-flow -->22```mermaid23flowchart TD24 Native["Open .orixsheet"] --> ReadN["NativeFormat.read<br/>verify manifest hashes"]25 Foreign["Open .xlsx .xlsm .ods .csv .tsv .json"] --> ReadF["WorkbookReader.load<br/>format detected from content"]26 ReadN --> Trust["Session with cached values trusted<br/>no recalculation"]27 ReadF --> Recalc["Session recalculated in full"]28 Recalc --> Untitled["fileURL cleared, type set to native<br/>document marked edited"]29 Untitled --> Report["Import report<br/>issues, macros, external links"]30 Trust --> Edit["Editing"]31 Report --> Edit32 Edit --> Save["Save and autosave<br/>NativeFormat.write only"]33 Edit --> Export["Export To<br/>preflight, report, then write"]34```35SVG fallback: 3637## The `.orixsheet` package3839A document is a directory that Finder shows as one file: the exported type40`ai.spboucher.orix.sheet.workbook` conforms to `com.apple.package`, and the document type sets41`LSTypeIsPackage` (`App/Resources/Info.plist`).4243```44Budget.orixsheet/45├── manifest.json format id, versions, dates, SHA-256 + size of every other part46├── workbook.json sheet list, defined names, style table, properties, attachment index47├── sheets/48│ ├── <id>.cells binary cell store of the sheet with that id (chunked, zlib)49│ └── <id>.meta.json layout and features of the sheet (SheetMeta)50├── attachments/ Workbook.attachments (scripts, queries, workflows…), one file each51├── preserved.plist optional: raw parts kept from an imported XLSX (charts, VBA, pivots…)52└── preview.png optional: thumbnail53```5455| Part | Encoding | Content |56|---|---|---|57| `manifest.json` | JSON, sorted keys | `format` = `orixsheet`, `formatVersion` = 1, `minReaderVersion` = 1, `app`, `appVersion`, `created`, `modified`, and `parts`: path → `{sha256, size}` for every other file |58| `workbook.json` | JSON | sheets in tab order (`id`, `name`, part paths, `cellCount`), defined names, the style table in index order, `date1904`, active sheet, default font, document properties, attachment index, `hasPreservedParts` |59| `sheets/<id>.meta.json` | JSON | column widths, row heights, hidden/filtered rows, outline levels, merges, frozen panes, tables, conditional formats, validations, comments, hyperlinks, autofilter, embedded objects (payload in base64), print settings, view flags, tab colour, hidden/protected flags |60| `sheets/<id>.cells` | binary, little-endian | magic `OXCL`, version 1, cell count, then independent zlib chunks of 50,000 cells, each with its own string table and CRC-32 |61| `attachments/<n>-<name>` | raw bytes | file name derived from the index and a sanitised suffix (`[A-Za-z0-9._-]`, last 48 characters) so that an attachment name can never become a path |62| `preserved.plist` | binary property list | `PreservedParts`: bytes of XLSX parts ORIX Sheet does not model, with the sheet ids they belong to; never interpreted or executed |6364Source: `Packages/SheetIO/NATIVE-FORMAT.md`; constants checked in `Native/NativeSchema.swift`65(`currentVersion = 1`), `Native/NativeCells.swift` (`cellsPerChunk = 50_000`,66`maxChunkRawBytes = 256 << 20`) and `Native/NativeFormat.swift` (`safeFileName`).6768Design points worth knowing:6970- **Sheet parts are named after `Sheet.id`**, which is stable across renames and reordering; tab71 order is the order of the `sheets` array in `workbook.json`.72- **Formulas are stored as canonical text** (en-US syntax, no leading `=`) together with the cached73 value, the spill range and the volatile flag; identical texts in a chunk are parsed once. A74 document therefore opens without recalculation.75- **Numbers**: integral values below 9·10¹⁵ are stored as zig-zag varints, everything else as the76 8 bytes of the IEEE-754 double, so −0.0, subnormals and NaN payloads round-trip. In the JSON parts,77 non-finite numbers are the strings `"+inf"`, `"-inf"`, `"nan"`, and dates are seconds since78 2001-01-01 UTC as a double.79- **What is not stored**: LAMBDA values held by a cell (saved as `#CALC!`, recomputed on load,80 reported as a `loss` issue), the dependency graph and dirty flags (rebuilt), the private sheet-id81 counter (restored to "largest id + 1").8283### Integrity8485Every part is hashed with SHA-256 (CryptoKit) at save time. On load, each part listed in the86manifest is compared by size and hash; in addition each cell chunk carries its own CRC-32.8788| Situation | Default behaviour | With `LoadOptions.strictIntegrity` |89|---|---|---|90| `workbook.json` missing or undecodable | error `corrupted(part: "workbook.json", …)` | same |91| `workbook.json` hash mismatch but decodable | opens, `warning` | error |92| a cell chunk fails its CRC, zlib or structure check | that chunk is skipped, other chunks and sheets load; `loss` issue naming the sheet, part and chunk | error |93| a `.cells` part missing or truncated | the sheet opens with what could be read (possibly empty); `loss` | error |94| a `.meta.json` part missing or undecodable | cells load, the sheet layout is reset to defaults; `loss` | error |95| attachment missing / changed | `loss` / `warning` | error for a hash mismatch or missing part |96| manifest missing or unreadable (and `workbook.json` present) | opens without verification; `warning` | error |97| directory with neither manifest nor `workbook.json` | error: not an ORIX Sheet document | same |98| `minReaderVersion` newer than this build | error `unsupportedVersion` | same |99| `formatVersion` newer, `minReaderVersion` readable | opens, unknown content ignored, `warning` that it would be lost on save | same |100101Source: `NativeFormat.read(parts:options:progress:)`; behaviour covered by102`NativeFormatTests.damagedPartsAreReportedPreciselyAndTheRestIsRecovered` and103`versioningAndForwardCompatibility`.104105The application opens documents with the default options, that is, in **recovery mode**: a damaged106document opens with whatever could be read, and the issues are shown in the report sheet (see107"Known gaps" for a wording problem in that case). `NativeFormat.verify(url:)` checks a package108against its manifest without loading it; the app test `testNativeSaveReopenIsLossless` uses it.109110The reader does not trust sizes found in the file: chunk lengths are bounded (256 MiB raw),111decompression stops at the declared size, indices are range-checked, a package may contain at most112100,000 files, symbolic links inside the package are skipped, and `LoadOptions.maxCells`113(200 million) caps the total.114115The integrity scheme detects accidental damage. It is **not** a signature: anyone who edits a part116can also rewrite the manifest. See [../security.md](../security.md).117118### Versioning119120`formatVersion` is the schema the writer used; `minReaderVersion` the oldest reader that can open the121file without misreading it. Unknown JSON keys and unknown files in the package are ignored. Lists122whose elements may gain new variants (`tables`, `conditionalFormats`, `validations`, `objects`) are123decoded element by element (`LossyArray`), so one unknown element is skipped and counted instead of124making the sheet unreadable. `NativeFormat.migrations` is the hook that upgrades the parts of an125older package before decoding; version 1 is the first schema, so the table is empty and no migration126has ever been exercised on a real older file.127128## Saving129130### What `NativeFormat.write` guarantees131132The package is built completely in memory (`build`), written into a hidden sibling directory133`.<name>.saving-<uuid>`, and then swapped in with `FileManager.replaceItemAt` (or moved when the134destination does not exist). On any failure before the swap the temporary directory is removed and135the previous document is untouched. A successful save replaces the whole package, so parts of deleted136sheets do not linger.137138`NativeFormatTests.interruptedSaveLeavesTheOriginalIntact` injects a failure just before the swap and139checks that the document on disk is byte-for-byte the previous version, that it still verifies, and140that no temporary directory is left; it also checks that a failed XLSX save leaves the existing file141intact (single-file formats use `Data.write(options: .atomic)`).142143### How the application saves144145`WorkbookDocument` overrides the URL-based pair `read(from:ofType:)` / `write(to:ofType:)`:146147<!-- diagram: persistence-save-sequence -->148```mermaid149sequenceDiagram150 participant U as User or autosave151 participant D as NSDocument machinery152 participant W as WorkbookDocument153 participant E as CalcEngine lock154 participant N as NativeFormat155 U->>D: save156 D->>W: write(to url, ofType native)157 W->>W: set properties.modified and created158 W->>E: lock159 W->>N: write(workbook, to url, preserved)160 N->>N: build parts and SHA-256 manifest161 N->>N: write hidden sibling directory162 N->>N: replaceItemAt or move163 N-->>W: issues (discarded)164 W->>E: unlock165 W-->>D: return166 D-->>U: document saved167```168SVG fallback: 169170Facts from the code:171172- **Only the native type is writable**: `writableTypes(for:)` returns `[nativeType]` and173 `fileNameExtension(forType:saveOperation:)` returns `orixsheet`, whatever the save operation.174- **Reading and writing happen on the main actor** (`MainActor.assumeIsolated` inside the175 `nonisolated` overrides; `canConcurrentlyReadDocuments` returns `false`;176 `canAsynchronouslyWrite(to:ofType:for:)` is not overridden). A save therefore blocks the UI for its177 duration; the measured write time is 0.58 s per million cells178 ([../compatibility/formats.md](../compatibility/formats.md)).179- **The engine lock is held for the whole write**, so a background recalculation cannot store values180 while the cells are being serialised.181- `properties.modified` is set at every save, `properties.created` at the first one; the manifest's182 `appVersion` is the bundle's `CFBundleShortVersionString`.183- The `PreservedParts` of an imported Excel file are passed to every save, so they travel inside the184 `.orixsheet` package and are available to a later XLSX export.185- The issues returned by the writer are discarded on a native save. The only issue the native writer186 can produce is the LAMBDA-value loss described above.187- `NativeWriteOptions.preview` is never set by the application: documents saved by the app contain188 **no `preview.png`**, and there is no Quick Look extension in the project.189190`NativeFormat` also offers a `FileWrapper` API (`fileWrappers(for:)`, `read(fileWrapper:)`), covered191by `NativeFormatTests.fileWrappersPreviewAttachmentsAndPreservedParts`. The application does not use192it: `WorkbookDocument` overrides the URL-based `read(from:ofType:)` and `write(to:ofType:)`, as193`Packages/SheetIO/NATIVE-FORMAT.md` states.194195### Autosave and versions196197`WorkbookDocument.autosavesInPlace` returns `true`. Nothing else about autosaving is customised:198`preservesVersions`, `autosavingFileType`, `autosavesDrafts` and the autosave interval keep their199AppKit defaults, and `NSSupportsSuddenTermination` / `NSSupportsAutomaticTermination` are `false` in200`Info.plist`. Consequently:201202- a saved `.orixsheet` document is autosaved in place by AppKit and has no "Save As"; the File menu203 offers *Save…*, *Duplicate*, *Rename…*, *Move To…* and *Revert To Saved* (`App/Sources/MainMenu.swift`);204- change tracking has two sources. Undoable changes: `WorkbookSession.perform` registers one undo205 action per transaction with the document's `UndoManager`, which marks the document edited.206 Non-undoable workbook changes: `WorkbookDocument.adopt` sets207 `session.onChange = { updateChangeCount(.changeDone) }`, and the features call `session.onChange?()`208 after storing something in the workbook outside the undo stack (a Python script, a transform209 pipeline, a figure setting, scenario data, ignored auditor findings, a workflow attachment), so210 those changes also mark the document edited;211- untitled documents (new, from a template, or imported) are handled by AppKit's standard212 untitled-document autosave.213214**Not verified in the repository**: no test exercises an actual autosave cycle, the system version215browser, *Revert To*, crash recovery of an untitled document, or iCloud Drive / file-coordination216scenarios. What is tested is the write/read pair called directly (`testNativeSaveReopenIsLossless`)217and the atomicity of `NativeFormat.write`. The behaviour described in the list above is AppKit's218documented behaviour for `autosavesInPlace`, not something this project measured.219220The app is not sandboxed (`ENABLE_APP_SANDBOX: NO` in `project.yml`), so no security-scoped bookmark221handling is involved.222223## Opening224225`readOnMain(from:ofType:)`:2262271. Determines the format with `FileFormat.detect(url:)` — content first (package directory, ZIP228 entries, `mimetype`, compound-file magic), then extension — and only falls back to the type name229 given by AppKit. A file with a misleading extension is opened for what it is.2302. Calls `WorkbookReader.load(url:format:)` with default `LoadOptions`.2313. Refuses a workbook without any sheet.2324. Keeps `preserved`, `issues`, `hasMacros`, `hasExternalLinks` from the `LoadResult`.2335. Creates the session: for `.orixsheet`, cached values are trusted234 (`CalcEngine(workbook:trustCachedValues: true)`: the graph is built, nothing is scheduled); for any235 other format the workbook is **recalculated in full** by ORIX Sheet's engine236 (`session.recalculateAll()`), so displayed values are ours, not the cached values of the producing237 application.238239Loading is synchronous on the main thread; `WorkbookReader.load` accepts a progress callback but the240document class does not pass one, so there is no progress indicator while a large file opens.241242### Import as untitled243244For a foreign format, `makeWindowControllers()` then:245246- sets `fileURL = nil`, `fileType = nativeType`, `displayName` = the source name without extension;247- calls `updateChangeCount(.changeDone)`, so closing the window asks whether to save;248- shows the import report 0.4 s later when there are issues, macros or external links.249250The app test `testForeignFilesAreImportedAsUntitledAndExportIsPreflighted` checks that `fileURL` is251`nil`, that the type is native, that formulas were recalculated and that the source file's bytes are252unchanged.253254The report (`CompatibilityReport`) lists issues sorted by severity — *Lost* (`loss`), *Changed*255(`warning`), *Note* (`info`) — with feature, location and detail, and a *Copy Report* button.256Repeated issues are merged by `IssueCollector` into one line with "(×N)" and up to five locations.257Two fixed warnings are added by the document: macros are present and are never run (kept only for an258`.xlsm` export); the workbook refers to other files and external data is not fetched. The report can259be reopened with *File ▸ Import Report…*.260261`Info.plist` declares the role `Editor` (rank `Owner`) for the native workbook type only. The Excel,262delimited-text, OpenDocument and JSON types are declared with the role `Viewer`, which matches the263behaviour described here: imported, never saved in place.264265## Export and preflight266267*File ▸ Export To* offers XLSX, XLSM, ODS, CSV (active sheet), TSV (active sheet), JSON (active268sheet), PDF (active sheet) and PDF (all visible sheets). JSON Lines can be read, and written by269`SheetIO`, but has no export menu entry.270271`WorkbookDocument.export(as:)`:2722731. commits a pending cell edit (and aborts if the entry is invalid);2742. runs `WorkbookWriter.preflight(workbook, format:, preserved:)` — a pure function that computes275 what the target cannot keep, without writing anything;2763. if the list is empty, shows the save panel directly; otherwise shows the report with *Cancel* /277 *Export Anyway*;2784. writes under the engine lock with `WorkbookWriter.save`. CSV, TSV and JSON export the *active*279 sheet; CSV and TSV write values **as displayed** (through the number-format engine), not raw280 values;2815. shows "Exported <name>" in the status bar. The document itself is not modified, renamed or282 marked clean.283284What preflight reports, by target:285286| Target | Reported as `loss` | Reported as `warning` / `info` |287|---|---|---|288| `.orixsheet` | cells holding a LAMBDA value | — |289| `.xlsx` / `.xlsm` | VBA project when saving as `.xlsx`; embedded objects (ORIX charts, figures, pivots); attachments; text over 32,767 characters; LAMBDA values; more than 65,530 hyperlinks on a sheet; colour scales with other than 2 or 3 points; invalid or duplicate sheet names, no sheet, grid overflow (saving will fail) | more than about 65,490 cell formats; threaded comments written as notes; charts or pivots of a deleted sheet dropped |290| `.ods` | conditional formats, validations, tables (written as plain ranges), autofilter, objects, outline, page setup, sheet-scoped names, attachments | 1904 date system converted; tab colour |291| `.csv` `.tsv` `.json` | other sheets with content; formulas exported as values; tables, validations, comments, hyperlinks, objects; defined names; content preserved from Excel | formatting not part of the format |292293Source: `Packages/SheetIO/Sources/SheetIO/Preflight.swift` and `WorkbookIO.swift`294(`WorkbookWriter.preflight`).295296Preflight is conservative about what it scans: the XLSX preflight visits every cell once (text297length, LAMBDA values). The writers themselves return the same kind of issue list after writing298(for instance truncated text with the cell address); the export path shows the preflight list before299writing and does not display the post-write list.300301PDF export does not go through `SheetIO`: it uses the print renderer302([grid-engine.md](grid-engine.md)) and writes the data atomically.303304## Known gaps305306- **Blocking I/O**: open, save and export run synchronously on the main thread without progress UI.307- **No preview or Quick Look**: `preview.png` is supported by the format but never written by the app.308- **Autosave, versions, revert and crash recovery are untested** (see above).309- **Report wording for native documents**: when a native document opens with integrity issues, the310 same sheet is shown with the title "Imported from" followed by an empty name and the message "The311 file was converted into an ORIX Sheet workbook", because `showImportReport()` has a single wording312 intended for foreign files.313- **Issues of a native save are not surfaced** (LAMBDA values stored as `#CALC!`).314- **Integrity is not authenticity**: hashes are recomputable by anyone who edits the package.315- **No encryption or password protection** of documents; encrypted ZIP archives and316 password-protected Office files are refused on import.317- **No file locking or merge**: two applications writing the same package are arbitrated by318 `NSDocument`'s standard conflict detection only, which this project has not tested.319