SPB Git

Repository Public

persistence.md 14.5 KB

# Persistence

# The native format: .orixdoc

Code: Packages/ORIXDocsKit/Sources/DocFormats/NativeContainer.swift, SchemaMigration.swift. Tests: Tests/DocFormatsTests/NativeContainerTests.swift.

# Why a single ZIP file

An .orixdoc document is one ZIP archive, not a directory package. A package looks like a folder to everything that is not macOS: mail clients, most non-Apple cloud folders and web uploads split it, zip it behind the user's back or refuse it. A single file survives all of them, and the ZIP layer is the defensive reader the DOCX engine already uses (DocContainer.ZipReader: entry-count, size and compression-ratio limits, no path traversal, no ZIP64, no encryption).

# Layout

Entry Stored as Content
mimetype first entry, uncompressed application/vnd.orix.docs+zip. Being first and stored, the string sits at byte offset 38 of the file, so the type can be recognised without a ZIP reader (the OpenDocument / EPUB convention).
manifest.json deflate Container version, schema version, generator, creation date, and one record per other entry.
document.json deflate JSONEncoder output of DocumentModel: sorted keys, ISO-8601 dates. Assets and preserved parts are not in it.
assets/<id>.<ext> stored for PNG / JPEG / GIF / HEIC / WebP, deflate otherwise One entry per DocumentModel.assets value.
preserved/<part name> deflate One entry per DocumentModel.preservedParts value (uninterpreted parts of an imported DOCX), under the original part name.

Asset ids and DOCX part names are not always valid, portable entry names (a part name starts with /, an id may contain a space or a slash). The entry name is therefore only a readable hint: the manifest maps each entry to its asset id or original part name (key) and records the media type of assets. Ids are hex-escaped outside [A-Za-z0-9_-]; a part name that cannot be an entry name becomes preserved/_partN.bin.

# The manifest

(Sizes and dates below are illustrative.)

json
{
  "containerVersion": 1,
  "created": "2026-11-21T06:30:51.169Z",
  "entries": [
    { "name": "mimetype", "role": "mimetype", "sha256": "…", "size": 29 },
    { "name": "document.json", "role": "document", "sha256": "…", "size": 18342 },
    { "key": "photo", "mimeType": "image/jpeg", "name": "assets/photo.jpg", "role": "asset", "sha256": "…", "size": 707 },
    { "key": "/word/vbaProject.bin", "name": "preserved/word/vbaProject.bin", "role": "preserved", "sha256": "…", "size": 600 }
  ],
  "generator": { "name": "ORIX Docs", "version": "1.0" },
  "schemaVersion": 1
}

Every entry other than the manifest itself is listed with its byte size and the SHA-256 (CryptoKit) of its uncompressed bytes. Two versions are recorded and checked independently:

  • containerVersion — the layout described on this page (currently 1);
  • schemaVersion — the version of document.json, equal to DocumentModel.schemaVersion.

# Dates

Dates are ISO-8601 strings in UTC with a fractional part of up to nine digits (2026-11-21T06:30:51.123456789Z), trailing zeros removed. Nine digits are more than a Date can resolve, so a date read back is bit-identical to the one written; a plain second-precision ISO-8601 date is accepted on reading.

# Reading

NativeContainer.read is strict. In order:

  1. files larger than FormatLimits.maxContainerBytes (1 GiB) are refused (FormatError.tooLarge), as is anything ZipLimits refuses;
  2. the first entry must be mimetype with the expected content, otherwise the file is "not an ORIX Docs document" (FormatError.unreadable);
  3. manifest.json must exist and decode; a containerVersion or schemaVersion newer than this build throws FormatError.unsupportedVersion;
  4. every manifest entry must exist, have the recorded size and the recorded SHA-256 (the ZIP CRC-32 is checked by the reader underneath);
  5. document.json is parsed, migrated (below) and decoded.

Any failure in steps 3–5 that is not a version problem throws FormatError.integrityCheckFailed. Entries present in the archive but absent from the manifest are tolerated and ignored (reported as one .info warning): a future version, or another tool, may add a thumbnail or a signature without breaking older readers. An unlisted assets/… entry is not loaded.

NativeContainer.validate runs the same checks and returns every problem as a sentence, instead of stopping at the first.

# Recovery

NativeContainer.recover is the best-effort path offered when read fails. It does not need the ZIP central directory (the part a truncated download loses): it walks the local file headers from the start of the file and keeps every entry whose data is complete, noting whether its CRC-32 holds.

  • document.json is used when it is intact, and also when its checksum fails but it still parses and decodes — with a warning asking the user to check the text;
  • a manifest that fails its own CRC is ignored (it cannot vouch for anything), and the loss of checksums is reported;
  • assets and preserved parts that are missing, incomplete or fail their checksum are skipped, one .dropped warning each; images whose asset is gone are counted in a Missing images warning;
  • a file whose document.json cannot be recovered throws: recovery never invents a document.

A sound file recovers to exactly the document read returns, with no warning.

# Schema versions and migrations

Only schema version 1 exists today. There is no older format in the wild, so SchemaMigrator.standard contains no migration step.

The pipeline is nevertheless in place and runs on every read, strict or recovering:

  • document.json is first parsed as an untyped JSON tree ([String: Any]);
  • its schemaVersion is read; a version newer than the build throws FormatError.unsupportedVersion;
  • for each version from the document's up to the current one, the step registered for that source version transforms the tree into the next version; the pipeline itself updates schemaVersion after each step;
  • the result is decoded into DocumentModel; when at least one step ran, an .info warning tells the user the document was upgraded.

Steps work on the JSON tree, not on Swift types, so a step written today keeps working after the model has moved on. A missing step for an old version is an error ("no migration is available from document schema N"), never a silent best guess.

Because no real older schema exists, the tests exercise the pipeline with an injected migration table (SchemaMigrator(steps:targetVersion:), internal) and deliberately invented "version 0" and "version −1" layouts: a renamed key, a moved key, steps run in sequence, a failing step, a missing step.

When schema 2 is introduced:

  1. bump DocumentModel.currentSchemaVersion;
  2. add 1: { tree in … } to SchemaMigrator.standard;
  3. add a fixture written by the last schema-1 build and a test that reads it.

# What the format does not do

  • No encryption, no digital signature. SHA-256 here detects corruption, not tampering: someone who can rewrite the file can rewrite the manifest.
  • No ZIP64: a document is limited to what a classic ZIP holds (4 GiB per entry and for the archive), and the reader's own limits are lower.
  • No incremental save: the whole file is rewritten on each save.
  • Writing is deterministic for a given document, generator and creation date (fixed ZIP timestamps, sorted keys and entries), which makes files diffable and cacheable.

# Autosave, versions and recovery

Code: App/Sources/Document/ORIXDocument.swift, DocumentSession.swift (ModelStore), App/Sources/App/AppDelegate.swift. Checks: the scenario "save, reopen and autosave through NSDocument" in App/Sources/App/SelfTest.swift; Scripts/recovery-test.sh.

# NSDocument

The application is a classic AppKit document application: ORIXDocument subclasses NSDocument, and NSDocumentController owns opening and the list of recent documents. The reason is that autosave in place, versions, revert, duplicate, rename, move and file coordination then come from the system instead of being rewritten. The document class itself is small:

Override Value or behaviour
autosavesInPlace true
preservesVersions true
autosavesDrafts true
writableTypes(for:) The native type only (ai.spboucher.orix.docs.document); other formats are written through Export.
canAsynchronouslyWrite(to:ofType:for:) true
data(ofType:) The native container of the current model.
read(from:ofType:) Imports any supported format (below).

# What AppKit provides

The File menu items are wired straight to NSDocument actions in MainMenu.build (AppDelegate.swift): Save (save(_:)), Duplicate (duplicate(_:)), Rename (rename(_:)), Move To (move(_:)), Revert To ▸ Last Saved Version (revertToSaved(_:)) and Browse All Versions (browseVersions(_:)). The versions browser, the storage of versions, renaming and moving are provided by AppKit; ORIX Docs adds no code to them beyond supplying and reading the file data.

# Autosave delay

applicationWillFinishLaunching sets NSDocumentController.shared.autosavingDelay to 5 seconds, or to the value of the ORIXAutosaveDelay default when it is set (never less than 1 second). AppKit autosaves that long after a change, and at other moments it chooses.

# Background writing from a snapshot

DocumentSession runs on the main actor and owns the editor. NSDocument may call data(ofType:) on a background thread, so the two meet in ModelStore, a lock-protected holder of the latest DocumentModel:

  • DocumentSession.documentChanged() stores the model after every edit;
  • data(ofType:) reads it under the lock, stamps meta.modified on its copy and encodes the container.

Because the model is a value type, what the save thread receives is a consistent snapshot; later edits on the main thread do not affect it. The provisional text of an input-method composition is not stored until it is committed, so an autosave during a composition writes the document as it was before the composition.

# Foreign files open as untitled documents

read(from:ofType:) chooses the format from the file extension (falling back to the type), imports the file with DocumentConverter.importDocument, and records the import warnings and the source format in the store. When the window is created (makeWindowControllers), a document that came from a foreign format (DOCX, RTF, Markdown, HTML, plain text, PDF) has its fileURL cleared, its type set to the native type and its change count marked (.changeReadOtherContents), keeping the original's name as display name. The result is an untitled, edited document: saving asks for a new .orixdoc location and the original file is never written. The document types other than the native one are declared with the Viewer role in project.yml.

Import warnings are kept in the session and shown from the status bar; the warnings sheet opens by itself when at least one warning is more than informational.

# Damaged native files

For .orixdoc, DocumentConverter.importDocument first calls the strict NativeContainer.read. If it throws, it tries NativeContainer.recover (see "Recovery" above); when recovery succeeds, the document opens with an extra warning, "Damaged document: the file failed its integrity check (…). What could be read was recovered; save it under a new name.", which is of the .dropped kind and therefore opens the warnings sheet. When recovery also fails, the strict error is shown and nothing opens.

A recovered document is still a native document: unlike an imported foreign file it stays bound to the damaged file, so autosave in place will rewrite that file after the next edit. The warning text asks the user to save under a new name; the code does not enforce it.

# Revert and version restore

Revert and restoring a version make AppKit call read(from:ofType:) on a document that already has a window. The read fills the store, then, on the main thread, DocumentSession.reload(_:) replaces the editor, empties the layout cache and lays the document out again.

# External modification

ORIX Docs contains no file-presenter code of its own: detecting that another program changed the file, and asking the user what to do when saving over it, is NSDocument's standard behaviour.

What the repository shows is one check in the self-test: after a save and an autosave, the test overwrites the file from outside with different content, lets the run loop turn for 1.5 seconds, and asserts that the file still holds the external content — that is, the open document did not silently write over it during that interval. The test does not make a further edit, does not trigger a save afterwards, and does not exercise the conflict alert.

# Recovery after a kill

Scripts/recovery-test.sh checks that work survives a process that gets no chance to save:

  1. launch the built app with -ORIXTemplate blank -ORIXSaveTo <file> so the application itself creates a saved document, and record the file's SHA-256;
  2. launch it again with -ORIXOpen <file> -ORIXAutosaveDelay 2 -ORIXType "Unsaved but recoverable 42": the text is typed through real key events (SelfTest.type), with no explicit save;
  3. wait (9 seconds by default, ORIX_RECOVERY_WAIT), then kill -9 the process;
  4. launch a third time with -ORIXOpen <file> -ORIXDumpText <out> and compare the text read back.

It prints whether the file changed on disk before the kill, the text found, and PASS or FAIL.

What it proves when it passes: with autosave in place, text typed into a saved document is on disk within the autosave delay, in a file that opens normally, without any save command or orderly quit.

What it does not prove: anything about an untitled document that was never saved (AppKit's draft autosave and crash restoration; the script disables state restoration with -ApplePersistenceIgnoreState YES); the fate of text typed less than the autosave delay before the kill; a kill in the middle of a write; power loss; or the versions browser. Scripts/test.sh runs the script and copies its last line to docs/reports/test-results.json (key recovery), which is the only record of a run in the repository. The XCUITest testAutosaveSurvivesAKilledProcess (UITests/EditorUITests.swift) describes the same scenario; see testing.md for why that suite was not run.

Files main
Menu

Theme