SPB Git

Repository Public

persistence.md 28.0 KB

# Persistence

ORIX DJ keeps three kinds of state on disk:

  1. the library, one SQLite database with tracks, analysis, collections, history, automation scripts and named JSON documents;
  2. projects, .orixdjproject packages that capture a session (decks, mixer, effects, routing, scripts, MIDI mapping) and reference audio by path;
  3. preferences in UserDefaults, and one secret, the user's Anthropic API key, in the macOS Keychain.

Audio files are never copied, moved, modified or deleted by any of these.

Sources:

  • Packages/ORIXDJCore/Sources/ORIXLibrary/: SQLiteDatabase.swift, LibraryStore.swift, LibraryStore+Collections.swift, LibraryStore+Scripts.swift, SmartCrateSQL.swift, MediaImporter.swift, Models.swift
  • Packages/ORIXDJCore/Sources/ORIXDJEngine/Project.swift, Session.swift
  • App/Sources/App/AppModel.swift, AppModel+Projects.swift, AppSettings.swift, ORIXDJApp.swift
  • Tests: Packages/ORIXDJCore/Tests/ORIXLibraryTests/ (45 tests in 10 suites, all passing in the run of 2026-11-21) and projectsRoundTripAndReportMissingMedia in Tests/ORIXDJEngineTests/SessionTests.swift

Related documents: overview.md, analysis.md and waveform.md for what the analysis blob contains, automation.md for scripts, recording.md for recorded files, ai-integration.md and ../reports/security.md for the API key, ../development/testing.md.

# Where data lives

The data root is ~/Library/Application Support/ORIX DJ/ (AppModel.init), or the directory given with the launch argument -ORIXDataDir <dir>, which UI tests and screenshot scripts use. SessionDirectories defines its content:

Path under the root Content
Library.sqlite (plus SQLite's -wal and -shm side files) The library database
Library.sqlite.corrupt-<yyyyMMdd-HHmmss> A corrupt database moved aside by the recovery path, if that ever happened
Recordings/ Files written by the recorder (see recording.md)
Demo Tracks/ Six WAV files synthesised on request by DJSession.installDemoTracks (16-bit, 44.1 kHz, generated by SignalSynth; no samples)
Autosave.orixdjproject/ The autosave project

Elsewhere:

Location Content
UserDefaults (standard domain; suite ai.spboucher.orix.dj.test when -ORIXDataDir is used) AppSettings: audio device UID, sample rate, buffer size, output routing, EQ mode, deck count and mode, waveform palette and zoom, appearance, recording format, Anthropic settings (not the key) and the list of verified model ids
macOS Keychain, generic password, kSecAttrAccessibleWhenUnlockedThisDeviceOnly The Anthropic API key (AnthropicClient.swift)
Wherever the user saves them .orixdjproject packages, exported .orixdj scripts, script backups

The project defines no entitlements file and the app is not sandboxed, which is why file references are plain (not security-scoped) bookmarks.

# SQLite wrapper

SQLiteDatabase is a thin class over the system SQLite3 C API; the package has no third-party dependencies (Package.swift links sqlite3 and Apple frameworks only). The source does not record a comparison with other persistence layers; what it does show is that the wrapper is about 200 lines and exposes exactly what the store needs:

  • one connection opened with SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX, extended result codes on, busy timeout 5000 ms;
  • a cache of prepared statements keyed by SQL text (flushed when it exceeds 256 entries);
  • typed binding through SQLiteValue (null, integer, real, text, blob); the number of bound values must equal the statement's parameter count or the call throws LibraryError.invalidArgument;
  • run, query, first, scalarInt, and transaction, which issues BEGIN IMMEDIATE and COMMIT or ROLLBACK; nested calls join the outer transaction;
  • dates stored as REAL seconds since the reference date, booleans as integers.

LibraryStore owns the connection and serialises every access with an NSRecursiveLock. All public methods are synchronous, callable from any thread, and throw LibraryError. Writes go through write, which wraps the body in a transaction and, after the outermost write succeeds, increments changeCount and yields to every changes() AsyncStream (buffering policy bufferingNewest(1), so a slow observer gets one pending event and not a backlog).

Connection settings applied at open: PRAGMA journal_mode = WAL, PRAGMA foreign_keys = ON, PRAGMA synchronous = NORMAL, and for file databases PRAGMA quick_check(1).

# Schema

The schema version is PRAGMA user_version. LibraryStore.migrations is an append-only array of SQL scripts; entry n upgrades version n to n + 1 inside a transaction. The current build has two migrations, so LibraryStore.schemaVersion == 2.

Version Adds
1 tracks, track_analysis, track_artwork, crates, crate_tracks, playlists, playlist_tracks, smart_crates, sessions, history, settings
2 scripts, script_versions, documents

A database whose user_version is higher than the build supports is refused with LibraryError.unsupportedSchema(found:supported:); it is not treated as corruption and is not modified (MigrationTests.newerSchemaIsRefusedAndNotTreatedAsCorruption). An older database is upgraded in place keeping its data (oldSchemaIsUpgradedInPlaceKeepingData).

flowchart LR
    tracks["tracks"] --> ta["track_analysis (zlib JSON blob)"]
    tracks --> art["track_artwork (JPEG thumbnail)"]
    crates["crates"] --> ct["crate_tracks"]
    tracks --> ct
    playlists["playlists"] --> pt["playlist_tracks (position)"]
    tracks --> pt
    sessions["sessions"] --> history["history"]
    tracks --> history
    smart["smart_crates (rules as JSON)"] -. "compiled to a WHERE clause" .-> tracks
    scripts["scripts"] --> sv["script_versions"]
    documents["documents (kind, name, json)"]
    settings["settings (key, value)"]

# Tables

Table Key columns Notes
tracks id autoincrement, path TEXT UNIQUE File facts (bookmark, duration, sample_rate, channels, bit_rate, format, file_size, dates), tags (title, artist, album, genre, year, comment), user data (rating 0 to 5, favorite, tags as a JSON array, play_count, last_played), analysis scalars (bpm, bpm_confidence, grid_first_beat, grid_downbeat_index, key, camelot, key_confidence, loudness_lufs, peak_dbfs, gain_db, analysis_version), missing, cues and loops as JSON arrays, and derived search columns (title_f, artist_f, album_f, genre_f, tags_index, search_text)
track_analysis track_id primary key, references tracks with ON DELETE CASCADE One opaque blob per track: the full AnalysisResult (beat grid, transients, key, loudness, waveform summary) as JSON compressed with zlib (DJSession.encode)
track_artwork track_id primary key, cascade JPEG thumbnail, longest side at most 300 pixels
crates, crate_tracks crate_tracks(crate_id, track_id) primary key, WITHOUT ROWID, both cascade A crate is a set: adding a track twice is ignored
playlists, playlist_tracks playlist_tracks.id, position Ordered; the same track may appear several times; positions are kept dense
smart_crates match (all or any), rules JSON Rules are stored, not results
sessions, history history.session_id cascade One history row per play
settings key primary key, WITHOUT ROWID String values; currently used for midi.activeProfile
scripts id; unique index on name COLLATE NOCASE source, tags, favorite, created_at, updated_at, origin (user, preset, ai), search_text
script_versions UNIQUE (script_id, version), cascade Append-only history of sources
documents UNIQUE (kind, name) Named JSON documents; json must parse as JSON or the save is refused

Indices: tracks_title(title_f), tracks_artist(artist_f), tracks_bpm(bpm), tracks_date_added(date_added), crate_tracks_track(track_id), playlist_tracks_order(playlist_id, position), playlist_tracks_track(track_id), history_played(played_at), history_session(session_id, played_at), history_track(track_id), scripts_name(name COLLATE NOCASE).

List queries (tracks(query:)) select the 36 track columns and never load the analysis blob or the artwork; those are fetched per track on demand with analysisBlob(for:) and artwork(for:).

DocumentKind names four document kinds: effect-chain (saved effect chains), midi-mapping (MIDI mapping profiles), visual-graph (visual builder graphs) and session-recovery. The last one is defined but no code writes it; session recovery uses the autosave project described below.

# Search and sorting

There is no FTS virtual table. Free-text search uses a derived column:

  • LibraryText.fold lower-cases and removes diacritics and width differences ("Beyoncé" becomes "beyonce"). On every write search_text is set to the folded concatenation of title, artist, album, genre, tags and comment, and title_f, artist_f, album_f, genre_f hold the folded fields.
  • TrackQuery.search is folded and split on whitespace; every word adds t.search_text LIKE ? ESCAPE '\' with the value %word%, the %, _ and \ characters of the word being escaped by LibraryText.likeEscaped. Words are AND-ed.
  • Because the pattern starts with %, SQLite cannot use an index for it: a search is a scan of tracks. PerformanceAndConcurrencyTests.twentyThousandTracksInsertAndSearchTimings runs insert, list, search, count, paged sort and a smart query on 20 000 tracks and prints the timings; its assertions are deliberately loose (batch insert under 30 s, search under 3 s, full list under 10 s, in a debug build). The printed timings were not captured in the report used for this document.
  • Sort keys (TrackQuery.SortKey) map to fixed ORDER BY clauses in orderClause. Unset values (no BPM, never played, no key) sort last in both directions; the key sort orders by Camelot number then letter.
  • Scopes (TrackQuery.Scope): all, crate, playlist, smartCrate, favorites, history, unanalysed (analysis_version = 0) and missing.

# Smart crates and SQL injection

A smart crate is a list of SmartRule values (field, comparison, value, optional upperValue) combined with all (AND) or any (OR). SmartCrate.predicate() in SmartCrateSQL.swift compiles it to a SQLPredicate: a SQL boolean expression and its bound parameters.

Field kind Fields SQL shape
Number bpm, rating, year, duration, playCount column > ?, column BETWEEN ? AND ?, equality as ABS(column - ?) < 0.005
Folded text title, artist, album, genre, key, camelot, format column = ?, column LIKE ? ESCAPE '\', comparisons on the folded column
Tag tag t.tags_index LIKE ?; tags are stored folded between U+001F separators so equals matches a whole tag
Flag favorite, analysed fixed expression or its negation, no parameter
Age in days dateAdded comparisons of t.date_added with a computed cut-off date

Injection is prevented by construction, not by escaping:

  • The SQL text is assembled only from string literals in the source: column names come from a switch over the SmartRule.Field enum, operators from a switch over SmartRule.Comparison. No rule value is ever interpolated into the SQL string.
  • Every rule value travels as a SQLiteValue parameter bound with sqlite3_bind_*.
  • Numeric fields parse the value with Double(...) and reject non-numeric text with LibraryError.invalidArgument; boolean fields accept only true/yes/1/on and false/no/0/off.
  • LIKE patterns escape %, _ and \, so user text cannot widen a match.
  • Rules are compiled once at creation and update time (createSmartCrate, updateSmartCrate), so an invalid rule is never stored.

SmartCrateTests.injectionStringsAreBoundNotInterpolated runs seven attack strings (among them x' OR '1'='1, "; DELETE FROM tracks; --, 1; DROP TABLE tracks, ') OR 1=1 --) through every field and comparison, checks that the generated SQL never contains the attempt, that the table still has its five rows, and that such strings are stored and matched literally as titles, artists, genres, tags and crate names. SearchAndSortTests.searchTreatsLikeWildcardsAndQuotesLiterally covers the free-text search.

The same binding rule holds for the rest of the store. The only identifiers interpolated into SQL are internal constants (table names passed by the store's own methods, the column lists).

# Tracks: upsert semantics

upsertTrack is keyed on path. For an existing row it replaces the file facts and the tags read from the file and clears missing, and keeps everything produced by the user or the analyser: id, dateAdded, lastPlayed, playCount, rating, favourite, tags, cues, loops and the analysis. BPM and key read from file tags are taken only while the track has not been analysed (analysis_version == 0). removeTracks deletes rows (and, by cascade, crate and playlist memberships, history, analysis and artwork) and never touches the audio file (TrackTests.removeTracksNeverDeletesAudioFiles).

normalised(_:) is applied on every write: rating clamped to 0...5, play count to at least 0, tags trimmed and de-duplicated ignoring case and diacritics, format lower-cased, and cues reduced to one per pad for pad indices 0...7.

# Scripts

Automation scripts are rows of scripts; every stored change of the source appends a row to script_versions.

  • createScript makes names unique without failing: a taken name becomes "Name 2", "Name 3" (case-insensitive).
  • saveScript(id:source:expectedUpdatedAt:) is an optimistic save. If the stored updated_at differs from the value the caller read, it throws LibraryError.conflict and stores nothing; the app then tells the user to use Save As (AppModel.saveActiveDocument). Saving an identical source is a no-op.
  • restoreScriptVersion does not rewrite history: the restored source is appended as a new version.
  • exportScript, importScript and backupScripts read and write .orixdj text files (UTF-8, atomic writes): a comment header starting with // ORIX DJ Script v1 with name, tags, origin, favourite and ISO 8601 dates, closed by // ---, followed by the source. A file without the header is imported whole and named after the file. Import never overwrites an existing script.
  • Built-in presets are not rows: DJSession resolves a script name in the library first and falls back to PresetLibrary.

These behaviours are covered by ScriptStoreTests (version history, stale save, unique names, search, export and import round trip, backup).

# Corruption recovery

LibraryStore.init(url:) opens the file, runs quick_check(1) and the migrations. If SQLite reports SQLITE_CORRUPT or SQLITE_NOTADB (or the quick check does not return ok), the file is renamed Library.sqlite.corrupt-<timestamp> (its -wal and -shm files follow), a fresh database is created, and the store reports recoveredFromCorruption == true with corruptBackupURL. CorruptionTests covers a file of garbage and a database damaged in the middle.

Two points are not handled today: no code in the app reads recoveredFromCorruption, so the user is not told that the library was reset, and nothing attempts to salvage rows from the damaged file. There is also no periodic backup of Library.sqlite.

# Importing media

MediaImporter is a stateless value type. It references files in place.

# Formats

MediaImporter.supportedExtensions is mp3, wav, aif, aiff, flac, m4a, aac, caf (ALAC is carried in .m4a). Decoding for playback and analysis goes through AVAudioFile (TrackBuffer.load), so the practical limit is what AVAudioFile opens on the running macOS. MediaImporterFormatTests.everySupportedExtensionOpensWithAVAudioFile writes a half-second fixture per format and opens it both with AVAudioFile and with the importer:

Format Verified by the test on macOS 27.0
WAV, AIFF (.aiff and .aif), CAF written and opened; asserted
AAC in M4A, ALAC in M4A written and opened; asserted
FLAC written and opened; asserted
MP3 macOS has no MP3 encoder, so the test builds a stream of silent frames by hand; opening it is asserted
Raw AAC (.aac, ADTS) attempted and reported in the test log, but not in the list the test asserts; treat as not verified

Real MP3 music files were not part of the automated tests. The 60 files of docs/reports/real-music-analysis.tsv were decoded through the same TrackBuffer.load path with 0 failures; that report does not record their formats.

TrackBuffer.load decodes the whole file into memory as planar 32-bit float at the file's sample rate and refuses files longer than TrackBuffer.maxSeconds (150 minutes).

# Scan

scan(urls:recursive:) expands files and folders into a sorted, de-duplicated list of files with a supported extension. Hidden files and folders and the contents of packages (for example .app bundles) are skipped; the scan stops when the task is cancelled. It filters by extension only: an empty .mp3 is listed and rejected later by validation.

# Metadata

readMetadata(url:):

  1. validate checks the extension, that the path is an existing readable regular file, and that its size is not zero.
  2. AVURLAsset provides hasProtectedContent, duration, estimated data rate and the stream description (sample rate, channel count).
  3. If the asset left sample rate, channels or duration unknown, AVAudioFile(forReading:) is tried; the source calls it "the reference for what the engine can decode".
  4. Tags are read from the asset's common and format-specific metadata: title, artist, album, genre (ID3v1 numeric genres are resolved to names), year, comment, BPM, initial key (including free-form initialkey, key, bpm, tempo items) and artwork. A BPM outside 0...1000 is dropped. A key tag that is a Camelot code ("8A") is stored in camelot, anything else in key.
  5. Without a title tag, the file name is used; "Artist - Title" (hyphen, en dash or em dash with spaces) is split.
  6. Artwork is reduced with ImageIO to a JPEG whose longest side is at most artworkMaxPixelSize = 300 (quality 0.8).
  7. A plain bookmark is created with URL.bookmarkData(options: []).

Bit rate falls back to fileSize * 8 / duration when the container does not report one.

# Malformed and unusable files

Every failure is a typed LibraryError, collected per file in ImportSummary.failures; one bad file never stops an import.

Situation Error
Extension not supported unsupportedFormat(path:)
Zero-byte file emptyFile(path:)
Missing file, or a directory with an audio extension unreadableFile(path:, reason: "no such file")
No read permission unreadableFile(path:, reason: "permission denied")
DRM-protected content protectedContent(path:)
Bytes that are not audio, a header with no samples, no audio track unreadableFile(path:, reason:) with the framework's reason
Task cancelled cancelled, and ImportSummary.cancelled = true

validationRejectsUnusableFilesWithClearErrors checks these cases, including 4096 bytes of garbage named .wav, .mp3, .m4a and .flac and a WAV header without samples. The DRM path is checked only for its message text; no protected file was available to test the detection itself.

After an import DJSession.importURLs calls analyzePending(), which analyses every track in the unanalysed scope, one at a time, on a utility queue, and can be cancelled.

# Bookmarks, moved and missing files

MediaImporter.resolve(track:) looks for the media through the stored bookmark first (which follows moves and renames on the same volume), then at the stored path. Files in the Trash count as missing. resolve(track:updating:) also persists the outcome: a moved file is relocated (LibraryStore.relocate updates path, bookmark, size, date and format), a stale bookmark is refreshed, and missing is set or cleared. DJSession.performLoad runs this before every load and returns SessionError.missingMedia instead of failing later. Covered by movedAndRenamedFileIsFoundThroughItsBookmark and missingMediaIsDetectedAndRecoversWhenTheFileReturns.

# Analysis storage

DJSession.analyze stores an analysis in two places with one call to LibraryStore.setAnalysis:

  • scalar columns of tracks (BPM, confidences, grid first beat and downbeat index, key short name, Camelot code, loudness, peak, suggested gain, analysis_version), so that lists, sorting and smart crates work without decoding anything;
  • the complete AnalysisResult in track_analysis, used when a track is loaded on a deck.

A stored analysis is reused only when track.analysisVersion == AnalysisResult.currentVersion (1); otherwise the track is analysed again during the load. "Analyse again" in the library sets analysis_version to 0 and restarts the background analyser. When a track has a stored BPM, the deck's grid is built from the tracks columns (gridFor), which is where manual grid edits made through DJSession.setGrid live.

Cue points are written by DJSession.persistCues (on quit for every deck, and by Auto DJ after a transition) and restored at load. See the known issue about the main cue at the end of this document.

# Projects

# Format

A project is a directory with the extension orixdjproject, declared in App/Info.plist as the exported type ai.spboucher.orix.dj.project conforming to com.apple.package, so Finder shows it as a single file.

text
My Set.orixdjproject/
    project.json          the ProjectDocument, pretty-printed JSON with sorted keys and ISO 8601 dates
    Scripts/
        <name>.orixdj     one plain-text copy of each captured script

project.json is the only file read back. The Scripts/ copies are a convenience for reading or recovering scripts outside the app; ProjectStore.load takes the script sources from the JSON.

# Content of ProjectDocument

Field Content
formatVersion ProjectDocument.currentFormat, currently 1
languageVersion The ORIX Script language version at save time
name, notes, savedAt Metadata
decks One ProjectDeck per deck: trackPath, trackTitle, a file bookmark, position, cuePoint, hotcues, loopStart, loopEnd, rate (stored as 1 when sync is on), keylock, keyShift, sync, quantize, and values, a dictionary of every other writable deck parameter of the script schema (volume, EQ, filter and so on)
mixer, crossfaderCurve Every writable mixer parameter, and the curve name
racks Five EffectChain values: one per deck and the master
masterLeft, masterRight, cueLeft, cueRight, splitCue Output routing
autoDJ Auto DJ settings
scripts Name and source of the library scripts open in the editor at save time
midiProfile The active MIDI mapping profile
autoDJQueue, recordings Present in the format; ProjectStore.capture does not fill them in the current build

# Save, load, apply

  • ProjectStore.save builds the complete package in a temporary directory next to the destination and then swaps it in (FileManager.replaceItemAt, or a move when the destination does not exist); on any error the temporary directory is removed and ProjectError.io is thrown. An interrupted save therefore leaves the previous package intact.
  • ProjectStore.load first decodes only formatVersion. A value above currentFormat throws ProjectError.newerFormat(version), so a project from a newer build is reported as such and not as corruption. A missing or invalid project.json throws ProjectError.unreadable. There is one format version so far and therefore no migration code.
  • ProjectStore.apply stops all automations, pauses the decks, restores mixer, effect chains, routing, Auto DJ settings and the MIDI profile, creates or updates the scripts in the library, and loads each deck's track asynchronously through the normal session path. A file that is no longer at its path is looked up through the bookmark; if it cannot be found its path is added to ProjectRestoreReport.missingMedia and the other decks still load. After a deck loads, the cue point, hot cues, position and rate are restored.

projectsRoundTripAndReportMissingMedia saves a session with two decks, mixer and effect settings, a hot cue, a script and a MIDI profile, saves it twice to the same URL, restores it into a fresh session with one audio file deleted, and checks the restored values, the report (loadedDecks == [0], one missing path, one script) and the three error cases.

loopStart and loopEnd are saved but apply does not restore them: a loop that was active when the project was saved is not active after opening it.

# Autosave and recovery

  • AppModel.autosave() captures the session as a project named "Autosave" into Autosave.orixdjproject in the data root. It runs every 900 ticks of the 30 Hz refresh timer, that is every 30 s, and once more in AppModel.shutdown() (called from applicationWillTerminate), in both cases only if at least one deck has a track. Errors are ignored (try?).
  • shutdown() also stops a running recording and calls persistCues for every deck.
  • At the next launch hasRecoverableSession is true when the autosave loads and at least one of its decks references a track. While all decks are empty, the Studio shows a "Restore Previous Session" button; nothing is restored without that click. recoverSession() opens the autosave like any project and clears projectURL, so a later Save asks for a location and does not overwrite the autosave.
  • Because the save is atomic, a crash during an autosave leaves the previous autosave. A crash between autosaves loses at most the last 30 s of deck, mixer and effect state. Unsaved text in script editor tabs is not part of the autosave: only scripts already saved to the library are captured, with their saved source.

Recovery after a real crash has not been tested end to end; the automated test covers project save, load and apply.

# What is never copied or stored

Item Verified in
Audio files are referenced by path and bookmark; import, projects and autosave copy no audio MediaImporter ("nothing is copied, moved or modified"), ProjectDeck holds trackPath and bookmark only, removeTracksNeverDeletesAudioFiles
The Anthropic API key is not in the library, in UserDefaults, in projects or in logs AppSettings stores AnthropicSettings and model ids only; ProjectDocument has no credential field; the key is read and written only through the Keychain helper in AnthropicClient.swift, tested by keychainStoresAndMasksTheKey
Removing a track from the library never deletes its file LibraryStore.removeTracks
Recordings are not embedded in projects ProjectDocument.recordings is a list of names and is left empty

A project is therefore not self-contained: moved to another Mac, it restores mixer, effects, scripts and MIDI mapping, and reports the audio files it cannot find.

# Known issues found while documenting

  • Main cue point is not persisted in the library. DJSession.persistCues writes the main cue as a CuePoint with index: -1, but LibraryStore.normalisedCues keeps only indices 0...7, so the entry is dropped on write and the c.index == -1 branch in performLoad never runs. Hot cues persist (tested); the main cue is restored only from projects.
  • Comment and behaviour differ in DJSession.analyze. Its comment says manual tempo or grid edits are kept, but setAnalysis replaces every analysis column including bpm and the grid fields. In practice analysis runs only for tracks without a current analysis, so an edit is overwritten only by "Analyse again".
  • DocumentKind.sessionRecovery, Track.savedLoops and LibraryStore.setSavedLoops exist in the schema and API but are not used by the app.
  • Corruption recovery is silent (see above).
Files main
Menu

Theme