Persistence
ORIX DJ keeps three kinds of state on disk:
- the library, one SQLite database with tracks, analysis, collections, history, automation scripts and named JSON documents;
- projects,
.orixdjprojectpackages that capture a session (decks, mixer, effects, routing, scripts, MIDI mapping) and reference audio by path; - preferences in
UserDefaults, and one secret, the user's Anthropic API key, in the macOS Keychain.
Audio files are never copied, moved, modified or deleted by any of these.
Sources:
Packages/ORIXDJCore/Sources/ORIXLibrary/:SQLiteDatabase.swift,LibraryStore.swift,LibraryStore+Collections.swift,LibraryStore+Scripts.swift,SmartCrateSQL.swift,MediaImporter.swift,Models.swiftPackages/ORIXDJCore/Sources/ORIXDJEngine/Project.swift,Session.swiftApp/Sources/App/AppModel.swift,AppModel+Projects.swift,AppSettings.swift,ORIXDJApp.swift- Tests:
Packages/ORIXDJCore/Tests/ORIXLibraryTests/(45 tests in 10 suites, all passing in the run of 2026-11-21) andprojectsRoundTripAndReportMissingMediainTests/ORIXDJEngineTests/SessionTests.swift
Related documents: overview.md, analysis.md and waveform.md for what the analysis blob contains, automation.md for scripts, recording.md for recorded files, ai-integration.md and ../reports/security.md for the API key, ../development/testing.md.
Where data lives
The data root is ~/Library/Application Support/ORIX DJ/ (AppModel.init), or the directory given with the launch argument -ORIXDataDir <dir>, which UI tests and screenshot scripts use. SessionDirectories defines its content:
| Path under the root | Content |
|---|---|
Library.sqlite (plus SQLite's -wal and -shm side files) |
The library database |
Library.sqlite.corrupt-<yyyyMMdd-HHmmss> |
A corrupt database moved aside by the recovery path, if that ever happened |
Recordings/ |
Files written by the recorder (see recording.md) |
Demo Tracks/ |
Six WAV files synthesised on request by DJSession.installDemoTracks (16-bit, 44.1 kHz, generated by SignalSynth; no samples) |
Autosave.orixdjproject/ |
The autosave project |
Elsewhere:
| Location | Content |
|---|---|
UserDefaults (standard domain; suite ai.spboucher.orix.dj.test when -ORIXDataDir is used) |
AppSettings: audio device UID, sample rate, buffer size, output routing, EQ mode, deck count and mode, waveform palette and zoom, appearance, recording format, Anthropic settings (not the key) and the list of verified model ids |
macOS Keychain, generic password, kSecAttrAccessibleWhenUnlockedThisDeviceOnly |
The Anthropic API key (AnthropicClient.swift) |
| Wherever the user saves them | .orixdjproject packages, exported .orixdj scripts, script backups |
The project defines no entitlements file and the app is not sandboxed, which is why file references are plain (not security-scoped) bookmarks.
SQLite wrapper
SQLiteDatabase is a thin class over the system SQLite3 C API; the package has no third-party dependencies (Package.swift links sqlite3 and Apple frameworks only). The source does not record a comparison with other persistence layers; what it does show is that the wrapper is about 200 lines and exposes exactly what the store needs:
- one connection opened with
SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX, extended result codes on, busy timeout 5000 ms; - a cache of prepared statements keyed by SQL text (flushed when it exceeds 256 entries);
- typed binding through
SQLiteValue(null,integer,real,text,blob); the number of bound values must equal the statement's parameter count or the call throwsLibraryError.invalidArgument; run,query,first,scalarInt, andtransaction, which issuesBEGIN IMMEDIATEandCOMMITorROLLBACK; nested calls join the outer transaction;- dates stored as
REALseconds since the reference date, booleans as integers.
LibraryStore owns the connection and serialises every access with an NSRecursiveLock. All public methods are synchronous, callable from any thread, and throw LibraryError. Writes go through write, which wraps the body in a transaction and, after the outermost write succeeds, increments changeCount and yields to every changes() AsyncStream (buffering policy bufferingNewest(1), so a slow observer gets one pending event and not a backlog).
Connection settings applied at open: PRAGMA journal_mode = WAL, PRAGMA foreign_keys = ON, PRAGMA synchronous = NORMAL, and for file databases PRAGMA quick_check(1).
Schema
The schema version is PRAGMA user_version. LibraryStore.migrations is an append-only array of SQL scripts; entry n upgrades version n to n + 1 inside a transaction. The current build has two migrations, so LibraryStore.schemaVersion == 2.
| Version | Adds |
|---|---|
| 1 | tracks, track_analysis, track_artwork, crates, crate_tracks, playlists, playlist_tracks, smart_crates, sessions, history, settings |
| 2 | scripts, script_versions, documents |
A database whose user_version is higher than the build supports is refused with LibraryError.unsupportedSchema(found:supported:); it is not treated as corruption and is not modified (MigrationTests.newerSchemaIsRefusedAndNotTreatedAsCorruption). An older database is upgraded in place keeping its data (oldSchemaIsUpgradedInPlaceKeepingData).
flowchart LR
tracks["tracks"] --> ta["track_analysis (zlib JSON blob)"]
tracks --> art["track_artwork (JPEG thumbnail)"]
crates["crates"] --> ct["crate_tracks"]
tracks --> ct
playlists["playlists"] --> pt["playlist_tracks (position)"]
tracks --> pt
sessions["sessions"] --> history["history"]
tracks --> history
smart["smart_crates (rules as JSON)"] -. "compiled to a WHERE clause" .-> tracks
scripts["scripts"] --> sv["script_versions"]
documents["documents (kind, name, json)"]
settings["settings (key, value)"]
Tables
| Table | Key columns | Notes |
|---|---|---|
tracks |
id autoincrement, path TEXT UNIQUE |
File facts (bookmark, duration, sample_rate, channels, bit_rate, format, file_size, dates), tags (title, artist, album, genre, year, comment), user data (rating 0 to 5, favorite, tags as a JSON array, play_count, last_played), analysis scalars (bpm, bpm_confidence, grid_first_beat, grid_downbeat_index, key, camelot, key_confidence, loudness_lufs, peak_dbfs, gain_db, analysis_version), missing, cues and loops as JSON arrays, and derived search columns (title_f, artist_f, album_f, genre_f, tags_index, search_text) |
track_analysis |
track_id primary key, references tracks with ON DELETE CASCADE |
One opaque blob per track: the full AnalysisResult (beat grid, transients, key, loudness, waveform summary) as JSON compressed with zlib (DJSession.encode) |
track_artwork |
track_id primary key, cascade |
JPEG thumbnail, longest side at most 300 pixels |
crates, crate_tracks |
crate_tracks(crate_id, track_id) primary key, WITHOUT ROWID, both cascade |
A crate is a set: adding a track twice is ignored |
playlists, playlist_tracks |
playlist_tracks.id, position |
Ordered; the same track may appear several times; positions are kept dense |
smart_crates |
match (all or any), rules JSON |
Rules are stored, not results |
sessions, history |
history.session_id cascade |
One history row per play |
settings |
key primary key, WITHOUT ROWID |
String values; currently used for midi.activeProfile |
scripts |
id; unique index on name COLLATE NOCASE |
source, tags, favorite, created_at, updated_at, origin (user, preset, ai), search_text |
script_versions |
UNIQUE (script_id, version), cascade |
Append-only history of sources |
documents |
UNIQUE (kind, name) |
Named JSON documents; json must parse as JSON or the save is refused |
Indices: tracks_title(title_f), tracks_artist(artist_f), tracks_bpm(bpm), tracks_date_added(date_added), crate_tracks_track(track_id), playlist_tracks_order(playlist_id, position), playlist_tracks_track(track_id), history_played(played_at), history_session(session_id, played_at), history_track(track_id), scripts_name(name COLLATE NOCASE).
List queries (tracks(query:)) select the 36 track columns and never load the analysis blob or the artwork; those are fetched per track on demand with analysisBlob(for:) and artwork(for:).
DocumentKind names four document kinds: effect-chain (saved effect chains), midi-mapping (MIDI mapping profiles), visual-graph (visual builder graphs) and session-recovery. The last one is defined but no code writes it; session recovery uses the autosave project described below.
Search and sorting
There is no FTS virtual table. Free-text search uses a derived column:
LibraryText.foldlower-cases and removes diacritics and width differences ("Beyoncé" becomes "beyonce"). On every writesearch_textis set to the folded concatenation of title, artist, album, genre, tags and comment, andtitle_f,artist_f,album_f,genre_fhold the folded fields.TrackQuery.searchis folded and split on whitespace; every word addst.search_text LIKE ? ESCAPE '\'with the value%word%, the%,_and\characters of the word being escaped byLibraryText.likeEscaped. Words are AND-ed.- Because the pattern starts with
%, SQLite cannot use an index for it: a search is a scan oftracks.PerformanceAndConcurrencyTests.twentyThousandTracksInsertAndSearchTimingsruns insert, list, search, count, paged sort and a smart query on 20 000 tracks and prints the timings; its assertions are deliberately loose (batch insert under 30 s, search under 3 s, full list under 10 s, in a debug build). The printed timings were not captured in the report used for this document. - Sort keys (
TrackQuery.SortKey) map to fixedORDER BYclauses inorderClause. Unset values (no BPM, never played, no key) sort last in both directions; thekeysort orders by Camelot number then letter. - Scopes (
TrackQuery.Scope):all,crate,playlist,smartCrate,favorites,history,unanalysed(analysis_version = 0) andmissing.
Smart crates and SQL injection
A smart crate is a list of SmartRule values (field, comparison, value, optional upperValue) combined with all (AND) or any (OR). SmartCrate.predicate() in SmartCrateSQL.swift compiles it to a SQLPredicate: a SQL boolean expression and its bound parameters.
| Field kind | Fields | SQL shape |
|---|---|---|
| Number | bpm, rating, year, duration, playCount |
column > ?, column BETWEEN ? AND ?, equality as ABS(column - ?) < 0.005 |
| Folded text | title, artist, album, genre, key, camelot, format |
column = ?, column LIKE ? ESCAPE '\', comparisons on the folded column |
| Tag | tag |
t.tags_index LIKE ?; tags are stored folded between U+001F separators so equals matches a whole tag |
| Flag | favorite, analysed |
fixed expression or its negation, no parameter |
| Age in days | dateAdded |
comparisons of t.date_added with a computed cut-off date |
Injection is prevented by construction, not by escaping:
- The SQL text is assembled only from string literals in the source: column names come from a
switchover theSmartRule.Fieldenum, operators from aswitchoverSmartRule.Comparison. No rule value is ever interpolated into the SQL string. - Every rule value travels as a
SQLiteValueparameter bound withsqlite3_bind_*. - Numeric fields parse the value with
Double(...)and reject non-numeric text withLibraryError.invalidArgument; boolean fields accept onlytrue/yes/1/onandfalse/no/0/off. LIKEpatterns escape%,_and\, so user text cannot widen a match.- Rules are compiled once at creation and update time (
createSmartCrate,updateSmartCrate), so an invalid rule is never stored.
SmartCrateTests.injectionStringsAreBoundNotInterpolated runs seven attack strings (among them x' OR '1'='1, "; DELETE FROM tracks; --, 1; DROP TABLE tracks, ') OR 1=1 --) through every field and comparison, checks that the generated SQL never contains the attempt, that the table still has its five rows, and that such strings are stored and matched literally as titles, artists, genres, tags and crate names. SearchAndSortTests.searchTreatsLikeWildcardsAndQuotesLiterally covers the free-text search.
The same binding rule holds for the rest of the store. The only identifiers interpolated into SQL are internal constants (table names passed by the store's own methods, the column lists).
Tracks: upsert semantics
upsertTrack is keyed on path. For an existing row it replaces the file facts and the tags read from the file and clears missing, and keeps everything produced by the user or the analyser: id, dateAdded, lastPlayed, playCount, rating, favourite, tags, cues, loops and the analysis. BPM and key read from file tags are taken only while the track has not been analysed (analysis_version == 0). removeTracks deletes rows (and, by cascade, crate and playlist memberships, history, analysis and artwork) and never touches the audio file (TrackTests.removeTracksNeverDeletesAudioFiles).
normalised(_:) is applied on every write: rating clamped to 0...5, play count to at least 0, tags trimmed and de-duplicated ignoring case and diacritics, format lower-cased, and cues reduced to one per pad for pad indices 0...7.
Scripts
Automation scripts are rows of scripts; every stored change of the source appends a row to script_versions.
createScriptmakes names unique without failing: a taken name becomes "Name 2", "Name 3" (case-insensitive).saveScript(id:source:expectedUpdatedAt:)is an optimistic save. If the storedupdated_atdiffers from the value the caller read, it throwsLibraryError.conflictand stores nothing; the app then tells the user to use Save As (AppModel.saveActiveDocument). Saving an identical source is a no-op.restoreScriptVersiondoes not rewrite history: the restored source is appended as a new version.exportScript,importScriptandbackupScriptsread and write.orixdjtext files (UTF-8, atomic writes): a comment header starting with// ORIX DJ Script v1with name, tags, origin, favourite and ISO 8601 dates, closed by// ---, followed by the source. A file without the header is imported whole and named after the file. Import never overwrites an existing script.- Built-in presets are not rows:
DJSessionresolves a script name in the library first and falls back toPresetLibrary.
These behaviours are covered by ScriptStoreTests (version history, stale save, unique names, search, export and import round trip, backup).
Corruption recovery
LibraryStore.init(url:) opens the file, runs quick_check(1) and the migrations. If SQLite reports SQLITE_CORRUPT or SQLITE_NOTADB (or the quick check does not return ok), the file is renamed Library.sqlite.corrupt-<timestamp> (its -wal and -shm files follow), a fresh database is created, and the store reports recoveredFromCorruption == true with corruptBackupURL. CorruptionTests covers a file of garbage and a database damaged in the middle.
Two points are not handled today: no code in the app reads recoveredFromCorruption, so the user is not told that the library was reset, and nothing attempts to salvage rows from the damaged file. There is also no periodic backup of Library.sqlite.
Importing media
MediaImporter is a stateless value type. It references files in place.
Formats
MediaImporter.supportedExtensions is mp3, wav, aif, aiff, flac, m4a, aac, caf (ALAC is carried in .m4a). Decoding for playback and analysis goes through AVAudioFile (TrackBuffer.load), so the practical limit is what AVAudioFile opens on the running macOS. MediaImporterFormatTests.everySupportedExtensionOpensWithAVAudioFile writes a half-second fixture per format and opens it both with AVAudioFile and with the importer:
| Format | Verified by the test on macOS 27.0 |
|---|---|
WAV, AIFF (.aiff and .aif), CAF |
written and opened; asserted |
| AAC in M4A, ALAC in M4A | written and opened; asserted |
| FLAC | written and opened; asserted |
| MP3 | macOS has no MP3 encoder, so the test builds a stream of silent frames by hand; opening it is asserted |
Raw AAC (.aac, ADTS) |
attempted and reported in the test log, but not in the list the test asserts; treat as not verified |
Real MP3 music files were not part of the automated tests. The 60 files of docs/reports/real-music-analysis.tsv were decoded through the same TrackBuffer.load path with 0 failures; that report does not record their formats.
TrackBuffer.load decodes the whole file into memory as planar 32-bit float at the file's sample rate and refuses files longer than TrackBuffer.maxSeconds (150 minutes).
Scan
scan(urls:recursive:) expands files and folders into a sorted, de-duplicated list of files with a supported extension. Hidden files and folders and the contents of packages (for example .app bundles) are skipped; the scan stops when the task is cancelled. It filters by extension only: an empty .mp3 is listed and rejected later by validation.
Metadata
readMetadata(url:):
validatechecks the extension, that the path is an existing readable regular file, and that its size is not zero.AVURLAssetprovideshasProtectedContent, duration, estimated data rate and the stream description (sample rate, channel count).- If the asset left sample rate, channels or duration unknown,
AVAudioFile(forReading:)is tried; the source calls it "the reference for what the engine can decode". - Tags are read from the asset's common and format-specific metadata: title, artist, album, genre (ID3v1 numeric genres are resolved to names), year, comment, BPM, initial key (including free-form
initialkey,key,bpm,tempoitems) and artwork. A BPM outside 0...1000 is dropped. A key tag that is a Camelot code ("8A") is stored incamelot, anything else inkey. - Without a title tag, the file name is used; "Artist - Title" (hyphen, en dash or em dash with spaces) is split.
- Artwork is reduced with ImageIO to a JPEG whose longest side is at most
artworkMaxPixelSize = 300(quality 0.8). - A plain bookmark is created with
URL.bookmarkData(options: []).
Bit rate falls back to fileSize * 8 / duration when the container does not report one.
Malformed and unusable files
Every failure is a typed LibraryError, collected per file in ImportSummary.failures; one bad file never stops an import.
| Situation | Error |
|---|---|
| Extension not supported | unsupportedFormat(path:) |
| Zero-byte file | emptyFile(path:) |
| Missing file, or a directory with an audio extension | unreadableFile(path:, reason: "no such file") |
| No read permission | unreadableFile(path:, reason: "permission denied") |
| DRM-protected content | protectedContent(path:) |
| Bytes that are not audio, a header with no samples, no audio track | unreadableFile(path:, reason:) with the framework's reason |
| Task cancelled | cancelled, and ImportSummary.cancelled = true |
validationRejectsUnusableFilesWithClearErrors checks these cases, including 4096 bytes of garbage named .wav, .mp3, .m4a and .flac and a WAV header without samples. The DRM path is checked only for its message text; no protected file was available to test the detection itself.
After an import DJSession.importURLs calls analyzePending(), which analyses every track in the unanalysed scope, one at a time, on a utility queue, and can be cancelled.
Bookmarks, moved and missing files
MediaImporter.resolve(track:) looks for the media through the stored bookmark first (which follows moves and renames on the same volume), then at the stored path. Files in the Trash count as missing. resolve(track:updating:) also persists the outcome: a moved file is relocated (LibraryStore.relocate updates path, bookmark, size, date and format), a stale bookmark is refreshed, and missing is set or cleared. DJSession.performLoad runs this before every load and returns SessionError.missingMedia instead of failing later. Covered by movedAndRenamedFileIsFoundThroughItsBookmark and missingMediaIsDetectedAndRecoversWhenTheFileReturns.
Analysis storage
DJSession.analyze stores an analysis in two places with one call to LibraryStore.setAnalysis:
- scalar columns of
tracks(BPM, confidences, grid first beat and downbeat index, key short name, Camelot code, loudness, peak, suggested gain,analysis_version), so that lists, sorting and smart crates work without decoding anything; - the complete
AnalysisResultintrack_analysis, used when a track is loaded on a deck.
A stored analysis is reused only when track.analysisVersion == AnalysisResult.currentVersion (1); otherwise the track is analysed again during the load. "Analyse again" in the library sets analysis_version to 0 and restarts the background analyser. When a track has a stored BPM, the deck's grid is built from the tracks columns (gridFor), which is where manual grid edits made through DJSession.setGrid live.
Cue points are written by DJSession.persistCues (on quit for every deck, and by Auto DJ after a transition) and restored at load. See the known issue about the main cue at the end of this document.
Projects
Format
A project is a directory with the extension orixdjproject, declared in App/Info.plist as the exported type ai.spboucher.orix.dj.project conforming to com.apple.package, so Finder shows it as a single file.
My Set.orixdjproject/
project.json the ProjectDocument, pretty-printed JSON with sorted keys and ISO 8601 dates
Scripts/
<name>.orixdj one plain-text copy of each captured scriptproject.json is the only file read back. The Scripts/ copies are a convenience for reading or recovering scripts outside the app; ProjectStore.load takes the script sources from the JSON.
Content of ProjectDocument
| Field | Content |
|---|---|
formatVersion |
ProjectDocument.currentFormat, currently 1 |
languageVersion |
The ORIX Script language version at save time |
name, notes, savedAt |
Metadata |
decks |
One ProjectDeck per deck: trackPath, trackTitle, a file bookmark, position, cuePoint, hotcues, loopStart, loopEnd, rate (stored as 1 when sync is on), keylock, keyShift, sync, quantize, and values, a dictionary of every other writable deck parameter of the script schema (volume, EQ, filter and so on) |
mixer, crossfaderCurve |
Every writable mixer parameter, and the curve name |
racks |
Five EffectChain values: one per deck and the master |
masterLeft, masterRight, cueLeft, cueRight, splitCue |
Output routing |
autoDJ |
Auto DJ settings |
scripts |
Name and source of the library scripts open in the editor at save time |
midiProfile |
The active MIDI mapping profile |
autoDJQueue, recordings |
Present in the format; ProjectStore.capture does not fill them in the current build |
Save, load, apply
ProjectStore.savebuilds the complete package in a temporary directory next to the destination and then swaps it in (FileManager.replaceItemAt, or a move when the destination does not exist); on any error the temporary directory is removed andProjectError.iois thrown. An interrupted save therefore leaves the previous package intact.ProjectStore.loadfirst decodes onlyformatVersion. A value abovecurrentFormatthrowsProjectError.newerFormat(version), so a project from a newer build is reported as such and not as corruption. A missing or invalidproject.jsonthrowsProjectError.unreadable. There is one format version so far and therefore no migration code.ProjectStore.applystops all automations, pauses the decks, restores mixer, effect chains, routing, Auto DJ settings and the MIDI profile, creates or updates the scripts in the library, and loads each deck's track asynchronously through the normal session path. A file that is no longer at its path is looked up through the bookmark; if it cannot be found its path is added toProjectRestoreReport.missingMediaand the other decks still load. After a deck loads, the cue point, hot cues, position and rate are restored.
projectsRoundTripAndReportMissingMedia saves a session with two decks, mixer and effect settings, a hot cue, a script and a MIDI profile, saves it twice to the same URL, restores it into a fresh session with one audio file deleted, and checks the restored values, the report (loadedDecks == [0], one missing path, one script) and the three error cases.
loopStart and loopEnd are saved but apply does not restore them: a loop that was active when the project was saved is not active after opening it.
Autosave and recovery
AppModel.autosave()captures the session as a project named "Autosave" intoAutosave.orixdjprojectin the data root. It runs every 900 ticks of the 30 Hz refresh timer, that is every 30 s, and once more inAppModel.shutdown()(called fromapplicationWillTerminate), in both cases only if at least one deck has a track. Errors are ignored (try?).shutdown()also stops a running recording and callspersistCuesfor every deck.- At the next launch
hasRecoverableSessionis true when the autosave loads and at least one of its decks references a track. While all decks are empty, the Studio shows a "Restore Previous Session" button; nothing is restored without that click.recoverSession()opens the autosave like any project and clearsprojectURL, so a later Save asks for a location and does not overwrite the autosave. - Because the save is atomic, a crash during an autosave leaves the previous autosave. A crash between autosaves loses at most the last 30 s of deck, mixer and effect state. Unsaved text in script editor tabs is not part of the autosave: only scripts already saved to the library are captured, with their saved source.
Recovery after a real crash has not been tested end to end; the automated test covers project save, load and apply.
What is never copied or stored
| Item | Verified in |
|---|---|
| Audio files are referenced by path and bookmark; import, projects and autosave copy no audio | MediaImporter ("nothing is copied, moved or modified"), ProjectDeck holds trackPath and bookmark only, removeTracksNeverDeletesAudioFiles |
The Anthropic API key is not in the library, in UserDefaults, in projects or in logs |
AppSettings stores AnthropicSettings and model ids only; ProjectDocument has no credential field; the key is read and written only through the Keychain helper in AnthropicClient.swift, tested by keychainStoresAndMasksTheKey |
| Removing a track from the library never deletes its file | LibraryStore.removeTracks |
| Recordings are not embedded in projects | ProjectDocument.recordings is a list of names and is left empty |
A project is therefore not self-contained: moved to another Mac, it restores mixer, effects, scripts and MIDI mapping, and reports the audio files it cannot find.
Known issues found while documenting
- Main cue point is not persisted in the library.
DJSession.persistCueswrites the main cue as aCuePointwithindex: -1, butLibraryStore.normalisedCueskeeps only indices0...7, so the entry is dropped on write and thec.index == -1branch inperformLoadnever runs. Hot cues persist (tested); the main cue is restored only from projects. - Comment and behaviour differ in
DJSession.analyze. Its comment says manual tempo or grid edits are kept, butsetAnalysisreplaces every analysis column includingbpmand the grid fields. In practice analysis runs only for tracks without a current analysis, so an edit is overwritten only by "Analyse again". DocumentKind.sessionRecovery,Track.savedLoopsandLibraryStore.setSavedLoopsexist in the schema and API but are not used by the app.- Corruption recovery is silent (see above).