Repository Public
1# Testing23This page lists what the automated tests of ORIX DJ verify, with the tolerances written in the assertions, what they print, which system services they touch, and what is not covered by any test. All automated tests live in the Swift package `Packages/ORIXDJCore`; the app target has a UI-test bundle that is currently empty.45Related pages: [setup](setup.md), [performance](performance.md), [audio engine](../architecture/audio-engine.md), [DSP](../architecture/dsp.md), [analysis](../architecture/analysis.md), [automation](../architecture/automation.md), [persistence](../architecture/persistence.md), [AI integration](../architecture/ai-integration.md), [accessibility report](../reports/accessibility.md), [security report](../reports/security.md).67## How to run89```sh10Scripts/build.sh core-test # filtered: failures, summaries, measurement lines11cd Packages/ORIXDJCore && swift test --no-parallel # full output12```1314The tests use the Swift Testing framework (`import Testing`, `@Suite`, `@Test`, `#expect`). `--no-parallel` runs the six test targets one after the other; several suites are additionally marked `.serialized` because they measure time, share the CoreMIDI server, or share a static stub.1516## Result of the run used for this document1718Source: `/tmp/orixdj-docs-test.out`, the filtered output of `swift test --no-parallel` run on 2026-11-21 on the build machine (Apple M5 Max, macOS 27.0, Swift 6.4.2, debug build).1920| Test target | Tests | Suites | Result | Duration |21|---|---|---|---|---|22| `ORIXScriptTests` | 47 | 7 | passed | 0.730 s |23| `ORIXMIDITests` | 50 | 4 | passed | 3.321 s |24| `ORIXLibraryTests` | 45 | 10 | passed | 3.190 s |25| `ORIXDSPTests` | 38 | 9 | passed | 49.749 s |26| `ORIXDJEngineTests` | 38 | 6 | passed | 110.607 s |27| `ORIXAnalysisTests` | 23 | 8 | passed | 71.770 s |28| Total | 241 | 44 | 241 passed, 0 failed | |2930A parameterised `@Test(arguments:)` counts as one test in these totals, so the number of executed cases is higher than 241 (for example `CheckerTests.rejects` runs 49 source/diagnostic pairs and five `GenericEffectTests` run once per effect kind, 18 each). The per-suite counts in the tables below are source counts (`@Test` attributes per `@Suite`, counted with a script); they add up to the run totals of each target.3132## Why audio can be tested without a device3334`DJEngine.renderOffline(frames:channels:)` runs the same render core as the Core Audio callback and returns the output as arrays, and `AutomationHost.renderOffline(frames:)` does the same while also running the automation scheduler in step with the rendered samples. No audio device is opened. A test builds an engine with `makeEngine()` (48 kHz, 256-frame buffers, master limiter off, linear crossfader curve, 4096 frames rendered first so the limiter bypass cross-fade is over), loads a synthesised signal from `SignalSynth` (sine, sweep, pink or white noise, click track, or a complete synthetic track with drums, bass and chords), renders, and measures the result: RMS level, zero-crossing frequency, onset positions, sample-by-sample difference. Results are deterministic to the sample, which is what makes tolerances such as "maximum error below 1e-6" possible. See [audio engine](../architecture/audio-engine.md).3536The consequence is stated under "Not covered" below: the real-time thread, the device and the D/A path are outside these tests.3738## ORIXDSPTests (38 tests, 9 suites)3940Files: `EffectsTests.swift`, `EffectBenchmarkTests.swift`, `EffectTestSupport.swift` (the `Rig` that renders arrays through one effect in blocks while advancing the transport, plus measurement helpers; test rate 48 kHz).4142| Suite | Tests | What is verified (tolerances from the assertions) |43|---|---|---|44| `GenericEffectTests` | 6 | Factory builds all 18 kinds with default values. For every kind: finite output and peak <= 12 at defaults, all-min, all-max and each parameter at each extreme, at 48 kHz and (finite only) 44.1 kHz. NaN, infinity and +/-1e9 parameter values plus NaN / infinite audio samples leave the unit finite with peak <= 40. A disabled unit is bit-transparent, also after enable then disable. Enable / disable produces no step larger than the steady wet plus dry sample delta plus 0.01, and the first enabled sample is within 0.02 of the dry signal. Output with 512-frame and 64-frame blocks differs by less than 2e-3 (0.05 for the gate, whose edge may land one sample apart) |45| `SpaceEffectTests` | 6 | Delay: first repeat exactly on the expected sample for four tempo / time / feedback combinations, amplitude within 1e-3, feedback ratio within 2e-3, less than 1e-6 between repeats; ping-pong alternates sides. Delay and echo time changes on a steady tone stay within 1.1 times the steady delta plus 1e-4. Echo repeats on the grid and each repeat is darker (band-energy ratio below 0.7 of the previous). Reverb: energy per 100 ms window decreases monotonically over 1.5 s for four sizes, RT60 grows by more than 15 % per size step, between 0.2 s and 6 s; nothing before the pre-delay (peak below 1e-6); width 0 is mono within 1e-6 |46| `ModulationEffectTests` | 3 | Flanger delay sweeps between 37...60 and 280...328 samples. Phaser without feedback preserves level within 0.5 dB and modulates a fixed tone by a factor above 4 at 50 % mix. Chorus stays bounded, spreads energy around the carrier, is stereo, and leaves pitch alone at depth 0 |47| `DistortionEffectTests` | 4 | Peak does not grow with drive (factor 1.005). Near transparent without drive (within 1 dB), third harmonic present, second harmonic below 0.002, tone control darkens. Bitcrusher uses between 2^bits - 1 and 2^bits levels for 2, 3, 4, 6, 8 bits, peak <= 1, silence stays silent; sample-and-hold runs are multiples of the factor |48| `FilterEffectTests` | 4 | Low-pass and high-pass: -3.01 dB at cutoff within 0.3 dB, pass band within 0.1 / 0.2 dB, -40 dB one decade away within 1.5 dB, 12 dB per octave within 1.5 dB, gain at cutoff equals Q (12.04 dB within 0.3 dB for Q = 4). Band-pass matches the analogue response within 0.2 dB at centre and 1 to 1.5 dB a decade away. A full-range cutoff jump on a steady tone produces no step above 1.5 times the steady delta |49| `RhythmEffectTests` | 8 | Tremolo follows the beat-locked LFO within 0.03 and its period within 2 samples for three tempo / rate / start-beat cases; phase follows a transport jump; square shape is open half the time within 0.02. Gate: zero wrong samples outside the 8 ms de-click window over more than 40000 checked, period within 1.5 samples, duty within 0.02. Autopan positions at quarter cycles within 0.02 and never boosts. Beat repeat: output exactly periodic (below 1e-6), the loop equals the last slice heard before enabling (below 1e-6), the seam is cross-faded, the capture refreezes on every enable |50| `DynamicsEffectTests` | 4 | Compressor static curve within 1 dB for five threshold / ratio / level cases; make-up gain within 0.5 dB; slower attack lets more than twice the level through; release recovers. Limiter never exceeds the ceiling plus 0.1 dB for ceilings -1, -6, -24, 0 dB on a +12 dB programme with spikes, and does not over-reduce by more than 3 dB; on a steady +12 dB sine the output is a sine at the ceiling within 0.1 dB with third harmonic below 0.3 %; below the ceiling the unit is a pure 240-sample delay (error below 1e-5) |51| `NoiseEffectTests` | 2 | Added noise is concentrated around the cutoff (more than 8 times the band energy a factor 5 away), stereo, and leaves the input tone within 0.01; level -60 dB is silence, a 20 dB level step measures 20 dB within 0.5 dB |52| `EffectBenchmarkTests` | 1 | CPU share of every effect over 20 s of audio in 512-frame blocks, best of three passes; the only assertion is a loose ceiling of 20 % of real time per effect. Prints a table (debug or release build is stated in its header) |5354## ORIXAnalysisTests (23 tests, 8 suites)5556Files: `AccuracyTests.swift`, `AnalyzerTests.swift`. All audio is synthesised at 44.1 kHz unless stated.5758| Suite | Tests | What is verified |59|---|---|---|60| `AccuracyTests` | 1 | `accuracyReport`: 10 tempi (70 to 174 BPM) as click tracks and in four synthetic styles (40 tracks), and 24 keys in three styles (72 tracks), 40 s each. Regression floors: every click track correct with BPM error below 0.02 and phase error below 2 ms; at least 32 of 40 tracks at the correct tempo; no track classified "other" or without grid (only octave errors are tolerated); correct tracks within 0.03 BPM; mean phase error of the three drum styles below 3 ms; at least 80 % of downbeats correct; at least 64 of 72 keys exact |61| `LoudnessTests` | 4 | A -20 dBFS stereo 1 kHz sine reads -20 LUFS within 0.15 at 44.1 and 48 kHz; BS.1770 reference levels (one channel -3.01, both channels 0.0, within 0.1); absolute and relative gating (within 0.1 and 0.2); the suggested gain never lets the peak exceed -1 dBFS |62| `WaveformTests` | 2 | Band separation of the three-band waveform for 60 Hz, 1 kHz and 6 kHz sines; 150 columns per second; overview reduction preserves peaks |63| `RobustnessTests` | 5 | Silence gives no grid and no key and encodes to JSON; inputs of 0 to 40000 samples, mismatched channel lengths and sample rate 0 do not crash; a steady sine has no grid and noise has no grid or confidence below 0.4; cancellation throws `CancellationError`; progress is monotonic from 0 to 1 |64| `TempoBehaviourTests` | 4 | Stability above 0.9 for steady clicks, below 0.4 for an accelerating 90 to 150 BPM track, below 0.6 for a random-walk tempo; `bpmRange` folds 140 to 70 and 60 to 120 within 0.05 BPM; 48 kHz and 22.05 kHz, swing and mono give 126 BPM within 0.05, phase within 6 ms and the right key; 20 transients within 3 ms of the clicks |65| `ModelTests` | 4 | Key names and the full Camelot table, compatibility rules, transposition, beat-grid conversions (1e-12), JSON round trip of a result |66| `SynthTests` | 2 | The synthesiser used by the tests is deterministic, seeded, bounded (peak 0.891 within 1e-3) and its elementary signals are what they claim |67| `TimingTests` | 1 | Analysis of a synthesised 5-minute track: BPM within 0.01 of 126; in release builds only, best of three under 3.0 s |6869## ORIXLibraryTests (45 tests, 10 suites)7071Files: `LibraryStoreTests.swift`, `MediaImporterTests.swift`, `ScriptStoreTests.swift`, `TestSupport.swift`.7273| Suite | Tests | What is verified |74|---|---|---|75| `MigrationTests` | 4 | A fresh database is at `LibraryStore.schemaVersion` and persists tracks, cues, loops, tags, analysis blob and settings across reopen; a schema-1 file is upgraded in place keeping data; a newer schema (999) is refused with `unsupportedSchema` and not treated as corruption; a closed store throws |76| `CorruptionTests` | 2 | A garbage file is moved aside byte for byte to `library.sqlite.corrupt-...` and a fresh library starts; a database overwritten after its header is detected and recovered |77| `TrackTests` | 6 | Upsert is unique on path and keeps rating, favourite, tags and play count; analysed BPM is not overwritten by tags; setters clamp; artwork and analysis blobs cascade on removal; removing a track never deletes the audio file; play history and sessions, with rollback of a failed play; scopes (favourites, unanalysed, missing) |78| `SearchAndSortTests` | 4 | Search ignores case and diacritics, ANDs words across fields, treats LIKE wildcards and quotes literally; every sort key in both directions, including harmonic order |79| `CollectionTests` | 4 | Crates are sets, playlists keep order and allow duplicates, library removal cascades; settings and documents; change counter and change stream |80| `SmartCrateTests` | 5 | Every comparison (equals, contains, greater, less, between, age in days), all / any, stored crates; injection strings are bound as parameters and leave the table intact; invalid rules rejected |81| `PerformanceAndConcurrencyTests` | 3 | 20000 tracks: exact result counts and loose time bounds (insert under 30 s, search under 3 s, list under 10 s) meant to catch a quadratic path; 12 threads times 150 iterations of mixed reads and writes with every write landing exactly once and gap-free script versions; two stores on the same file |82| `MediaImporterFormatTests` | 1 | WAV, AIFF, CAF, AAC and ALAC in M4A, and FLAC are written with `AVAudioFile`, reopened and read by the importer (44.1 kHz, 2 channels, duration 0.5 s within 0.1); `.aif`; a hand-built silent MP3 stream opens; the supported extension list is exactly `mp3, wav, aif, aiff, flac, m4a, aac, caf` |83| `MediaImporterTests` | 10 | Scan skips hidden files, packages and unsupported types and honours cancellation; metadata falls back to the file name; tags and artwork are read from a tagged M4A and thumbnailed; unusable files are rejected with clear errors; a folder import copies or modifies nothing and re-import refreshes instead of duplicating; a moved and renamed file is found through its bookmark; missing media is detected and recovers |84| `ScriptStoreTests` | 6 | Every source change appends a version and restoring appends too; a stale save throws a conflict and overwrites nothing; names stay unique case-insensitively; search over name, tags and source; export / import round trip with the `// ORIX DJ Script v1` header; backup of every script to a folder |8586## ORIXMIDITests (50 tests, 4 suites)8788Files: `MIDIParserTests.swift`, `MIDIMappingEngineTests.swift`, `MIDIMappingProfileTests.swift`, `MIDIManagerSystemTests.swift`, `TestSupport.swift`.8990| Suite | Tests | What is verified |91|---|---|---|92| `MIDIParserTests` | 12 | Single messages, running status (also across buffers), several messages per packet, real-time bytes inside a message, SysEx skipped in four variants, system common cancels running status, bytes and UMP round trip, out-of-range fields masked. Fuzzing: 4000 random byte buffers never crash, stay in range and parse identically when fed in random chunks; 200000 random UMP words never crash |93| `MIDIMappingEngineTests` | 19 | Absolute 7-bit and 14-bit, range and invert, relative encodings (two's complement, signed bit, sensitivity), trigger, toggle, momentary, inverted buttons, device scope, soft takeover in all its cases, and no non-finite output for degenerate values |94| `MIDIMappingProfileTests` | 8 | JSON round trip with stable bytes, tolerant decoding of hand-written JSON, rejection of newer formats and garbage, validation reports, file export / import |95| `MIDIManagerSystemTests` (`.serialized`) | 11 | Against the real CoreMIDI server with virtual endpoints: the server is reachable; notes, CC and pitch bend arrive from a virtual source; unsupported events are counted as dropped; a legacy byte packet arrives decoded; an active profile produces target events with device scoping and soft takeover; MIDI learn picks the control that moved most; hot-plug of sources and destinations reconnects automatically; a selected subset connects only chosen sources; `send` delivers the expected bytes to a virtual destination, also for a batch larger than one packet; 1000 messages are all received and mapped (latency printed; only a sanity bound of 1 s on the median); stop and restart |9697## ORIXScriptTests (47 tests, 7 suites)9899Files: `LanguageTests.swift`, `PresetTests.swift`, `DocsTests.swift`.100101| Suite | Tests | What is verified |102|---|---|---|103| `LexerTests` | 4 | Unit literals (`dB`, `Hz`, `kHz`, `bars`, fractions of a beat, `%`, `ms`, `s`, `bpm`), comments, strings and locations, `E-UNIT` and `E-LEX`, highlighting of invalid source |104| `ParserTests` | 5 | The product-brief example compiles with the expected decks and a length of 32 beats; syntax errors carry line and column; the parser recovers and reports several errors; empty and unbounded scripts are rejected; nesting depth is bounded (`E-LIMIT`) |105| `CheckerTests` | 5 | 49 wrong statements each produce the expected diagnostic code (type, range, path, deck, read-only, effect, ramp, limits, events, arguments ...); "Did you mean" messages; capability validation for a two-deck rig; functions, recursion and purity; implicit linear gain (50 % is -6.0206 dB within 1e-3) |106| `RuntimeTests` | 21 | In the simulator: sequential timing, absolute `at` blocks, ramps reaching targets exactly (1e-9; frequencies on a logarithmic scale), curves, a direct write cancelling a ramp, loops, `every` on a grid, concurrent procedures, events (track end, beat, bar, trigger), conditions, quantised start, seconds following the tempo, run-time errors cancelling and bypassing effects, budgets (`E-BUDGET`, `E-RATE`, `E-UNBOUNDED`, `E-LIMIT`), clamping, stop / pause / resume, deterministic random, and 3000 + 500 malformed inputs that never crash |107| `ToolTests` | 5 | Debugger breakpoints and stepping, console parsing, the formatter, autocomplete and documentation lookup, every schema entry reachable from source |108| `PresetTests` | 3 | At least 20 presets with unique names and one per category; each compiles without errors or warnings, simulates without error, warning or clamped value, finishes, and is formatted canonically; "Eight Bar Transition" does what its name says beat by beat; every path of the generated reference compiles (more than 150 checked) |109| `DocsTests` | 4 | The manifest lists at least 29 existing chapters of more than 400 characters with no duplicate; every `orixdj` example in `docs/orix-script` compiles and simulates (at least 60), every `orixdj-error` example is rejected, every `console` line parses; the four generated chapters are byte-identical to the generator output; every preset name and 30 error codes appear in the documentation |110111The user-facing counterpart of these suites is the language documentation, starting at [introduction](../orix-script/introduction.md), with [errors](../orix-script/errors.md) and [presets](../orix-script/presets.md).112113## ORIXDJEngineTests (38 tests, 6 suites)114115Files: `EngineAudioTests.swift`, `SessionTests.swift`.116117| Suite | Tests | What is verified |118|---|---|---|119| `PlaybackTests` | 14 | Unity playback equals the source sample for sample after the 4 ms start fade (error below 1e-6, no clipped block). Fader, trim and master gains within 0.02 dB. -60 dB is exact silence. EQ boost, cut and kill (0.3 to 1 dB in band, below -50 dB at full kill). Channel filters (-3.01 dB at the corner within 0.2 dB). Crossfader curves `smooth`, `linear`, `cut` within 0.05 dB. Varispeed: 1250 Hz within 0.05 Hz at rate 1.25. Sample-rate conversion from 44.1 kHz keeps pitch within 0.02 Hz and level within 0.01 dB. Key lock at rates 0.8 to 1.25: pitch within 0.5 Hz of 440, playhead speed equals the rate, level within 1 dB. Key shift within 0.2 %. Key lock keeps click timing within 12 ms. Reverse is sample-accurate (1e-5), brake stops the deck. Track end stops and emits `trackEnd`. Pause and seek on a loud 80 Hz tone produce no step above 0.03 |120| `DeckFeatureTests` | 7 | A 2-beat loop repeats every 24000 samples (within 4 samples at the seam, exactly 48000 over two passes); quantised cue and hot cues; sync brings a 126 BPM deck to 120 BPM within 0.05 with worst phase error below 3 ms; the master clock follows the master deck; 4-channel routing puts the pre-fader cue bus on outputs 3 and 4, cue mix and split cue work; the master limiter holds -1 dBFS and the clip counter reports clipping without it; effect rack order, enable, move, removal, bypass and the slot limit |121| `AutomationLiveTests` | 5 | Script actions land on the grid of the running engine within 2 samples with zero late events; a 4-beat ramp is linear in dB (within 0.25 dB at the midpoint, 0.05 dB at the end) with a largest gain step below 0.0005; a failing script leaves audio running, is recorded as failed and bypasses the effect it enabled; the console drives the real engine; events reach handlers |122| `PerformanceTests` | 1 | Renders 10 s with four key-locked decks and 15 effects and prints the render cost; the only assertion is that more than 1000 blocks were rendered |123| `SessionTests` (`.serialized`) | 6 | Demo tracks, analysis cache, loading, history, a busy deck refusing a load, cues surviving a reload, missing media reported. Recording equals the master output sample for sample (maximum error exactly 0) with zero overruns, and every export format reopens with the expected length; lossless exports are within one 16-bit or 24-bit step; normalising to -1 dBFS lands within 0.01 dB. Auto DJ performs two real transitions on a queue with a script that compiles and no run-time error. The planner is deterministic and every transition style compiles at 1, 4, 8 and 16 bars. Projects round-trip and report missing media; corrupt and newer projects fail cleanly. The control surface scales every writable parameter and MIDI events drive the engine |124| `AnthropicTests` (`.serialized`) | 5 | See "Real system services" below |125126## Measurement blocks printed by the tests127128The tests print fixed-format blocks so that numbers can be copied into reports without retyping. `Scripts/build.sh core-test` keeps every line containing `=== ORIX`, which means single-line blocks survive the filter and multi-line blocks show only their header.129130| Block | Printed by | Value in today's run |131|---|---|---|132| `=== ORIX-DOCS ===` | `DocsTests.everyExampleCompilesAndRuns` | 150 examples compiled and simulated, 59 error examples rejected, 59 console lines parsed |133| `=== ORIX-SYNC ===` | `DeckFeatureTests.syncMatchesTempoAndPhase` | mean 0.0066 ms, worst 0.0208 ms over 22 beats (offline render, synthetic click tracks) |134| `=== ORIX-AUTOMATION-TIMING ===` | `AutomationLiveTests.scriptActionsLandOnExactSamples` | tracked 3, late 0, maxLate 0 samples; worst click offset 0 samples |135| `=== ORIX-RENDER-COST (debug build) ===` | `PerformanceTests.renderCostFullLoad` | 1875 blocks of 256 frames, mean 3098 µs, max 69046 µs, budget 5333 µs, load 58.1 %. This is an unoptimised debug build rendered off the real-time thread: it is not a performance figure. Release numbers are in [performance](performance.md) and `docs/reports/engine-benchmark.txt` |136| `=== ORIX-ANALYSIS-ACCURACY ===` | `AccuracyTests.accuracyReport` | multi-line; see below |137| `=== ORIX-ANALYSIS-TIMING ===` | `TimingTests.fiveMinuteTrackTiming` | multi-line; see below |138139Other lines are printed without the `=== ORIX` marker and are therefore absent from the filtered output: the per-effect CPU table (`ORIX DJ effects, CPU cost ...`), the reverb RT60 values, the tempo-stability values, `[ORIXLibrary timing, debug build]`, `[ORIXLibrary format support on ...]` and `[ORIXMIDI latency]`. Their values were not available at the time of writing; run `swift test --no-parallel` unfiltered to see them.140141The body of the two analysis blocks comes from a companion capture of the same test target, `/tmp/orixdj-docs-analysis.out` (a separate run of `ORIXAnalysisTests` on the same day and machine, debug build):142143| Group | n | Correct tempo | Double | Half | Mean phase error | Max phase error | Downbeat |144|---|---|---|---|---|---|---|---|145| click | 10 | 10 | 0 | 0 | 0.068 ms | 0.113 ms | n/a |146| house | 10 | 9 | 1 | 0 | 0.949 ms | 1.001 ms | 9/9 |147| techno | 10 | 10 | 0 | 0 | 1.023 ms | 1.089 ms | 10/10 |148| breakbeat | 10 | 9 | 0 | 1 | 0.951 ms | 1.024 ms | 9/9 |149| ambient | 10 | 8 | 2 | 0 | 7.201 ms | 16.340 ms | 8/8 |150| all tracks | 40 | 36 | 3 | 1 | 2.572 ms | 16.340 ms | 36/36 |151152Octave error rate on tracks: 4/40 = 10.0 % (87.5 BPM house read as 175, 140 BPM breakbeat as 70, 70 and 87.5 BPM ambient as 140 and 174.83). Key: 71 of 72 exact, one relative-key confusion (house D flat major read as B flat minor). Timing block: 5 minutes of 44.1 kHz stereo analysed in 0.747 s (best of three, single thread, debug build), 126.0002 BPM against a truth of 126, first beat 0.0811 s against 0.0800 s, key G minor, -14.95 LUFS.153154These are results on synthetic audio. Results on 60 real tracks are in `docs/reports/real-music-analysis.tsv` and discussed in [analysis](../architecture/analysis.md); there is no ground truth for those tracks, so that file shows plausibility, not accuracy.155156## Real system services used by the tests157158| Service | Where | Detail |159|---|---|---|160| CoreMIDI server | `MIDIManagerSystemTests` | Real `MIDIClient`, virtual sources (`MIDISourceCreateWithProtocol`) and virtual destinations created by the test process, named `ORIXDJ Test ...` with a random suffix. Helpers throw instead of skipping, so a machine without a reachable MIDI server fails the suite. No physical controller is involved |161| SQLite files | `ORIXLibraryTests`, `SessionTests` | Real database files in unique temporary directories (`TempDirectory`, `tempRoot()`), plus in-memory stores for the pure logic tests |162| AVFoundation codecs | `MediaImporterFormatTests`, `SessionTests` | Audio fixtures are encoded and decoded by the system (`AVAudioFile`, `AVAssetExportSession` for tagged M4A); recording and export files are written and reopened |163| Keychain | `AnthropicTests.keychainStoresAndMasksTheKey` | Saves, replaces, reads and deletes a generic password in the user's real Keychain under a unique service `ai.spboucher.orix.dj.tests.<UUID>`, and checks the masked form `sk-ant-…abcd` |164| Network | none | The Anthropic client is tested with `StubProtocol`, a `URLProtocol` subclass registered in an ephemeral `URLSessionConfiguration`. It intercepts requests inside the test process and answers with canned HTTP statuses and server-sent-event bodies. No socket is opened and nothing reaches `api.anthropic.com`. (It is an in-process stub, not a listening HTTP server.) |165166What `AnthropicTests` verifies through that stub: the request goes to `https://api.anthropic.com/v1/messages` with the key in the `x-api-key` header only, `anthropic-version: 2026-10-21`, streaming on, the configured effort, a cached system block, and none of `temperature`, `top_p`, `top_k`, `thinking`, `budget_tokens`; text deltas are streamed in order and usage is read; HTTP 401, 402, 403, 404, 429 (with `retry-after`), 400 and 529 map to typed errors, with 529 tried three times in total; `refusal` and `max_tokens` stop reasons are errors; a deployment rejecting the fallback parameter gets one retry without it; a missing key or an exhausted hourly budget never reaches the transport; the model list is paginated; proposals returned by the model are compiled and simulated locally before being offered. No Anthropic API key was available during development, so the client has never been run against the live API.167168## Not covered169170| Area | Status |171|---|---|172| Listening tests | None. Nobody has listened critically to the engine output, the time stretcher, the effects, or the two offline mixes produced with `orix-dj-bench mix`. All audio assertions are numerical |173| Real-time thread and audio devices | The Core Audio callback, device start, sample-rate changes and device removal are not exercised by `swift test`. `orix-dj-bench live` measures the real render thread manually (see [setup](setup.md)); it is not part of the test suite |174| Multi-channel interfaces | No multi-channel interface was attached. 4-channel cue routing is verified only by offline rendering (`routingCueBusAndSplitCue`) |175| MIDI hardware | No controller was attached. CoreMIDI is tested with virtual endpoints only; controller-specific behaviour, LED feedback on a real device and driver quirks are untested |176| Live Anthropic API | Not tested; stub transport only |177| Long live sessions | No soak test. The longest renders are offline (10 s to a few minutes of audio); memory growth, drift or overloads over hours of real playback have not been observed |178| Real music accuracy | No ground truth for tempo or key on commercial tracks |179| User interface | No automated UI test (next section). Layout, keyboard shortcuts, drag and drop, the Metal waveform renderer and window restoration are checked by hand or by screenshot |180| Accessibility | No VoiceOver walkthrough and no Accessibility Inspector audit has been done; see the [accessibility report](../reports/accessibility.md) |181| Localisation | No test of the French (fr-CA) strings; the string catalog is not in the repository at the time of writing |182| Release artifacts | Signing, notarisation and Gatekeeper are checked by the release scripts, not by tests, and that run has not happened yet; see the [release checklist](../release/checklist.md) |183| Other Macs | Only the build machine (arm64, macOS 27.0). macOS 15 at the deployment target was not tested |184185## UI tests: current state186187`project.yml` defines the target `ORIXDJUITests` (`bundle.ui-testing`, sources `UITests/`, depends on the app, automatic signing with "Apple Development"), and the scheme `ORIXDJ` lists it as its only test target. At the time of writing (re-read on 2026-11-21, after the file was replaced during this documentation pass) `UITests/ORIXDJUITests.swift` contains one `XCTestCase` class with nine test methods. Every test launches an isolated, silent instance with `-ORIXDataDir /tmp/orixdj-uitest-<uuid> -ORIXSilent YES -ORIXDemo YES`, English language and locale, and `-ApplePersistenceIgnoreState YES`: the real engine renders, there is no audio device, and the tracks are the synthesised demo tracks.188189| Test | What it asserts |190|---|---|191| `testEverySectionOpens` | Clicking each of the ten `section-<name>` buttons shows that section's root element (`library-root`, `effects-rack`, `autodj-root`, `automation-studio`, `docs-root`, `claude-root`, `recording-root`, `midi-root`, `performance`, `studio`) within 10 s |192| `testTransportDrivesTheEngine` | `play-B` becomes enabled within 30 s, its label changes from "Play deck B" to "Pause deck B" after a click, `sync-B` reports the value "on", `cue-B` returns the deck to paused, and deck A (started by the demo) reads "Pause deck A" |193| `testConsoleExecutesRealCommands` | Typing `deck.B.volume = -8dB` then `deck.B.volume` in the console prints a text containing "-8 dB"; `deck.Z.play()` prints a text containing `E-DECK` |194| `testSimulatorRunsAPreset` | With the preset "Eight Bar Transition" and the simulator tab, the elements `automation-simulator-timeline` and `automation-simulator-state` appear |195| `testLiveRunAsksBeforeTouchingAPlayingMix` | With "Filter Sweep" loaded and deck A playing, `automation-run` shows a confirmation button "Run on the Live Mix"; after confirming, the status strip reports a running automation; `emergency-stop` produces the `notice` element |196| `testDocumentationSearch` | Typing "crossfader" in `docs-search` shows `docs-search-results` and at least one `docs-search-hit` |197| `testRecordingShowsAnIndicatorAndStops` | `record-indicator` is absent before recording, appears after `recording-toggle` is clicked, disappears after a second click, and `recording-takes` exists |198| `testEffectsRack` | Adding "Echo" from `effects-add` creates `effect-power-echo` with value "off"; a click turns it "on"; `emergency-stop` returns it to "off" |199| `testClaudeStudioWithoutAKeyExplainsItself` | Without a key Claude Studio shows `claude-setup` and `claude-open-settings`, and the Studio section still works |200201These tests check that elements exist, that labels and values change, and that two safety behaviours hold (confirmation before a live run, emergency stop). They do not check audio output, layout, VoiceOver behaviour or keyboard navigation. **The documentation author did not run them**: no result of `Scripts/build.sh uitest` is reported here, and whether the nine tests pass on this machine is not verified in this document.202203## Manual and semi-automatic checks204205`Scripts/check.sh` is a repository hygiene check, not a test of behaviour: author headers, no signing material or audio in git, no secrets or private addresses, network APIs confined to the Anthropic client and its two UI folders, no logging of the API key, no privileged or process-spawning API. It exits with status 1 on any finding and is the first step of `Scripts/release.sh`. When it was run while writing this page it reported one finding, `missing header: Scripts/shot.sh`: the `Author:` line of that script is on line 12 and the check reads the first 9 lines.206207`Scripts/shot.sh` is a screenshot tool: it launches an isolated instance (`-ORIXDataDir`, `-ORIXSilent YES`, `-ORIXDemo YES`) on one section, waits, and captures the window by ID. Combined with the feature launch arguments it reproduces a given screen in light or dark appearance for visual review. A person looks at the image; nothing compares it to a reference.208209Two Debug-only self-checks can be triggered by launch argument and read on standard output: `-ORIXBuilderSelfCheck YES` prints `ORIX-BUILDER-CHECK PASS|FAIL` lines for the visual builder round trip, and `-ORIXMIDIDemo YES` pushes real CoreMIDI messages from a virtual source through the running app.210211`orix-dj-bench check <file.orixdj>` compiles and simulates a script from the command line, and `orix-dj-bench bench|live|analyze|mix` produce the measured reports; see [setup](setup.md) and [performance](performance.md).212