SPB Git

Repository Public

waveform.md 15.4 KB 177 lines markdown

1# Waveform display23ORIX DJ never draws raw audio samples. The analyser reduces every track to a small three-band summary once, the summary is stored with the track, and both waveform views draw from it: the scrolling deck waveform with a single Metal fragment-shader pass, the whole-track overview with a SwiftUI `Canvas`.45Sources:67- `Packages/ORIXDJCore/Sources/ORIXAnalysis/Waveform.swift` (`WaveformSummary`, `WaveformBuilder`)8- `App/Sources/Features/Waveforms/WaveformRenderer.swift` (Metal renderer and shader source)9- `App/Sources/Features/Waveforms/WaveformViews.swift` (`ScrollingWaveform`, `MetalWaveform`, `OverviewWaveform`, `OverviewCache`)10- `App/Sources/App/AppSettings.swift` (`WaveformPalette`), `App/Sources/DesignSystem/Tokens.swift` (deck and hot-cue colours, `ORIX.Palette.waveformWell`)1112Related documents: [analysis.md](analysis.md) for the rest of the analysis pass, [persistence.md](persistence.md) for how the summary is stored, [deck-engine.md](deck-engine.md) for the deck state the renderer reads, [../design/design-system.md](../design/design-system.md) for the colour system and [../development/performance.md](../development/performance.md) for measured costs.1314## Data flow1516```mermaid17flowchart TD18    A["Decoded audio (TrackBuffer, planar Float32)"] --> B["WaveformBuilder.build"]19    B --> C["WaveformSummary: low, mid, high, peak at 150 columns per second"]20    C --> D["AnalysisResult, JSON then zlib, table track_analysis"]21    C --> E["LoadedTrack.waveform on the deck"]22    D --> E23    E --> F["WaveformRenderer.upload: RGBA8 texture, 4096 columns per row"]24    F --> G["waveFragment: one full-screen triangle per frame"]25    H["DeckStatus: position, grid, cue, hot cues, loop"] --> G26    E --> I["WaveformSummary.overview(columns:), max-pooled to the view width"]27    I --> J["OverviewWaveform: three filled paths in a SwiftUI Canvas"]28    H --> J29```3031## Summary format3233`WaveformSummary` is a `Codable`, `Sendable` struct with four byte arrays of equal length and a rate:3435| Field | Content | Scaling |36|---|---|---|37| `columnsPerSecond` | Columns per second of audio. `150` for a fresh analysis (`WaveformBuilder.columnsPerSecond`), lower after `overview(columns:)`. | `Double` |38| `low` | Peak magnitude of the band below 200 Hz in the column | `255 * sqrt(amplitude)`, rounded, clamped to 255 |39| `mid` | Peak magnitude of the 200 Hz to 2500 Hz band | same |40| `high` | Peak magnitude of the band above 2500 Hz | same |41| `peak` | Full-band sample peak over both channels | linear, `255 * abs(amplitude)`, 255 = 0 dBFS |4243How the columns are made (`WaveformBuilder.build`):4445- A column covers `sampleRate / 150` samples: 294 samples at 44.1 kHz, 320 at 48 kHz. The column count is `ceil(frames * 150 / sampleRate)`.46- `peak` is computed per channel with `vDSP_maxmgv` and the larger of left and right is kept.47- The band values are computed on the mono mix (`0.5 * (left + right)`, built once by `TrackAnalyzer`). The mix is filtered by 8th-order Butterworth filters, each realised as four biquad sections (`WaveformBuilder.sections`, Q values `0.5098`, `0.6013`, `0.9000`, `2.5629`; 48 dB per octave, -3 dB at the crossover) and run through `vDSP_biquad`:48  - low: low-pass at `lowCrossover = 200` Hz;49  - mid: high-pass at 200 Hz followed by low-pass at `highCrossover = 2500` Hz;50  - high: high-pass at 2500 Hz.51  The crossover frequency is clamped to `0.45 * sampleRate`.52- Each band value is the maximum magnitude of the filtered signal within the column, stored square-root scaled so that quiet material keeps visible height in 8 bits.53- The filters are causal IIR filters applied once, forward. Their group delay is not compensated, so band envelopes lag the `peak` array slightly, most at the low crossover. The delay has not been measured.5455Size, from the constants: the four arrays take 4 bytes per column, 600 bytes per second of audio, 180 000 bytes for a five-minute track before compression.5657`overview(columns:)` returns a max-pooled copy for the whole-track view: output column `c` is the maximum of source columns `c*n/columns ..< (c+1)*n/columns`; `columnsPerSecond` is scaled accordingly. It returns `self` when `columns >= count` and an empty summary when `columns <= 0`.5859### Storage6061The summary is not stored on its own. It is a member of `AnalysisResult`, which `DJSession.encode` serialises with `JSONEncoder` and compresses with zlib (`NSData.compressed(using: .zlib)`); the blob goes to the `track_analysis` table of the library database (see [persistence.md](persistence.md)). Because the arrays are `[UInt8]`, the JSON holds them as arrays of numbers; the compressed size was not measured. When a track is loaded, `DJSession.storedAnalysis(for:)` decodes the blob if `track.analysisVersion == AnalysisResult.currentVersion` (currently `1`) and the summary is handed to the deck as `LoadedTrack.waveform`. A track without a current analysis is analysed during the load.6263### What the tests verify6465`WaveformTests` in `Packages/ORIXDJCore/Tests/ORIXAnalysisTests/AnalyzerTests.swift`:6667| Test | Assertion |68|---|---|69| `bandSeparation` | A 60 Hz sine gives `low > 170`, `mid < 0.1 * low`, `high < 0.02 * low`; a 1 kHz sine gives `mid > 170` with `low` and `high` below `0.1 * mid`; a 6 kHz sine gives `high > 170`, `low < 0.02 * high`, `mid < 0.1 * high`. For amplitude 0.5 the band value is within 4 of 180 (`255 * sqrt(0.5)`) and the linear peak within 2 of 127.5. The first and last 15 columns (100 ms) are skipped because of filter transients. |70| `columnCountAndOverview` | 10 s of audio gives exactly 1500 columns at `columnsPerSecond == 150`; `overview(columns: 100)` has 100 columns at 10 columns per second, preserves the global maximum, and its first column equals the maximum of the first 15 source columns; clicks every 0.5 s appear every 75 columns with silence (`peak == 0`) between them. |71| `silenceGivesNoGrid` (RobustnessTests) | 20 s of silence gives 3000 columns, all zero. |7273## Scrolling waveform (Metal)7475### Structure7677`ScrollingWaveform` (SwiftUI) hosts `MetalWaveform`, an `NSViewRepresentable` around an `MTKView`. Its coordinator is a `WaveformRenderer`, one per deck view, which is the view's `MTKViewDelegate`. The Metal device (`MTLCreateSystemDefaultDevice()`), the render pipeline and the command queue are `static` and shared by all decks. The shader is a Metal Shading Language string compiled at first use with `device.makeLibrary(source:options:)`; there is no `.metal` file in the app bundle.7879The view is configured with `colorPixelFormat = .bgra8Unorm`, `enableSetNeedsDisplay = false` and `isPaused = false`, so MetalKit drives `draw(in:)` continuously at `preferredFramesPerSecond`.8081### Texture8283`WaveformRenderer.upload` copies the summary into one `rgba8Unorm` texture, without mipmaps, usage `.shaderRead`:8485| Channel | Content |86|---|---|87| R | `low` |88| G | `mid` |89| B | `high` |90| A | `peak` |9192The texture is `textureWidth = 4096` texels wide and `ceil(columns / 4096)` rows high; column `i` lives at texel `(i % 4096, i / 4096)`. A five-minute track (45 000 columns) needs 11 rows. The upload happens when the deck's `LoadedTrack` changes or its waveform column count changes, not per frame. When no track is loaded a 1x1 zero texture is bound.9394The alpha channel is uploaded but the current fragment shader reads only `.rgb`: the linear `peak` array is not drawn by the scrolling view.9596### Per-frame work9798Each frame the renderer reads `engine.deckStatus(deck)`, fills a `WaveformUniforms` block made only of `float4` members (so the Swift and Metal layouts agree) and draws one triangle that covers the viewport (`waveVertex` generates it from `vertex_id`; there are no vertex buffers and no per-frame geometry). Everything visible is computed per pixel in `waveFragment`:99100| Uniform | Content |101|---|---|102| `view` | drawable width and height in pixels, playhead position in seconds, seconds per pixel |103| `data` | columns per second, column count, texture width, track duration |104| `grid` | BPM, first beat in seconds, downbeat index, cue point in seconds (`-1` when no track) |105| `loop` | loop start, loop end, loop active flag, gain (always `1` at present) |106| `hotA`, `hotB` | eight hot-cue times in seconds, `-1` when unset |107| `low`, `mid`, `high` | band colours from the selected palette |108| `deck` | deck colour, used for the loop region and loop edges |109| `background` | well colour, default `(0.02, 0.027, 0.055)` |110| `misc` | backing scale factor, playing flag (not used by the shader) |111112The fragment shader, in drawing order:1131141. Maps the pixel to track time: `t = playhead + (x - width / 2) * secondsPerPixel`. The playhead is fixed at the horizontal centre and the audio scrolls under it. Vertical position is folded to `y = abs(2 * py / height - 1)`, so the waveform is mirrored about the centre line.1152. Lightens the background by `0.012` outside `0...duration`, so the start and end of the track are visible.1163. Tints an active loop region with the deck colour (16 %).1174. Beat grid: for `bpm > 0`, and only when beats are more than `7 * scale` pixels apart, draws a line at every beat of the constant-tempo grid. Lines whose index modulo 4 matches `downbeatIndex` are drawn at 42 % white, other beats at 14 %.1185. Waveform: reads up to 8 columns spanning the pixel (`taps = clamp(ceil(secondsPerPixel * columnsPerSecond), 1, 8)`) with `texture.read` (integer texel fetch, no filtering) and keeps the per-band maximum. Band heights are `low`, `0.86 * mid`, `0.72 * high` of the half height, times `0.94`; they are composited in the order low, mid, high with a one-pixel anti-aliased edge. Audio left of the playhead is dimmed to 62 %.1196. Centre line at 10 % white.1207. Loop edges in the deck colour; the cue point as an orange line (`1.0, 0.62, 0.04`), full strength in the outer 45 % of the height and 55 % elsewhere.1218. Hot cues: a thin line in one of eight fixed colours plus a small flag at the top (`9 * scale` pixels), so that hot cues differ from the cue point by shape as well as colour.1229. Playhead: a white vertical line at the centre.123124### Zoom and scrubbing125126- Zoom is `AppSettings.waveformZoom`, the number of seconds visible across the view: default `6`, clamped to `1...32`, stored in `UserDefaults` under `ui.waveformZoom`. It is one setting for all decks. It changes with a pinch (`MagnifyGesture`) or the two magnifier buttons (factor 1.5).127- Dragging horizontally scrubs: the translation is converted to seconds with the current zoom and sent to `engine.seek(deck:seconds:)`, so the real playhead moves.128- There is no free scrolling away from the playhead; the view is always centred on the play position.129130### Frame pacing131132At the end of every `draw(in:)` the renderer sets `view.preferredFramesPerSecond` to `60` while the deck is playing and `15` otherwise. The deck position comes from `DJEngine.deckStatus`, read on the main thread each frame; the renderer does not interpolate between engine updates. Frame time and GPU cost of this pass have not been measured (see [../development/performance.md](../development/performance.md)).133134## Overview waveform135136`OverviewWaveform` shows the whole track in a SwiftUI `Canvas`; it does not use Metal.137138- `OverviewCache` builds three closed `Path` outlines (low, mid, high) from `waveform.overview(columns: Int(viewWidth))`, mirrored about the centre, with height factors `0.94`, `0.8` and `0.66`. The paths are rebuilt only when the track, the view size or the column count changes.139- Each redraw fills the well (`ORIX.Palette.waveformWell`, `0x05070E` in both appearances), the three paths in the palette colours, a 45 % black veil over the played part, the loop region in the deck colour (35 % when active, 15 % otherwise), the cue point (`0xFF9F0A`), hot cues as half-height marks in `ORIX.hotCueColors`, and a 2-point white playhead.140- A click or drag seeks to the proportional position. For assistive technologies the view is one adjustable element whose increment and decrement actions seek by one fiftieth of the duration; its value is the played percentage.141- The overview does not draw the beat grid.142143## Palettes144145`WaveformPalette` (`App/Sources/App/AppSettings.swift`) defines the band colours as sRGB hex. The choice is stored in `UserDefaults` under `ui.palette` and applies to both the scrolling and the overview waveform.146147| Case | Title in the interface | Low | Mid | High |148|---|---|---|---|---|149| `spectrum` (default) | Spectrum (red / green / blue) | `0xFF4B3A` | `0x3BDC6E` | `0x4D8DFF` |150| `ember` | Ember (amber / white / cyan) | `0xFF8A1F` | `0xF2F4FB` | `0x22D3EE` |151| `colorBlindSafe` | Colour-blind safe (blue / orange / yellow) | `0x0072B2` | `0xE69F00` | `0xF0E442` |152| `mono` | Monochrome | `0x6E7592` | `0xA3AAC2` | `0xF2F4FB` |153154The `colorBlindSafe` values are the blue, orange and yellow of the Okabe-Ito palette. The default `spectrum` palette separates low and mid by red against green, which is the pair most affected by the common colour-vision deficiencies; `colorBlindSafe` and `mono` avoid that pair. In all palettes the bands also differ by height (`1.0`, `0.86`, `0.72` of the scale) and by stacking order, so colour is not the only cue. None of the palettes has been checked with a colour-vision simulator or by affected users (see [../reports/accessibility.md](../reports/accessibility.md)).155156Other colours are not part of the palette setting:157158- Deck colour: `ORIX.deck(index)` in `Tokens.swift`, converted to sRGB and passed as the `deck` uniform.159- Hot cues: the overview uses `ORIX.hotCueColors`; the Metal shader has its own constant array of eight colours written as floats. The two lists were written to match but are maintained separately.160- Cue point: orange in both views.161162## When Metal is unavailable163164There is no software fallback for the scrolling waveform. If `MTLCreateSystemDefaultDevice()` returns `nil`, or the shader fails to compile (the error is logged with `NSLog`), `WaveformRenderer.pipeline` is `nil` and `draw(in:)` returns before encoding anything: the scrolling view stays empty. The SwiftUI overlays (zoom buttons, the "Drop a track here" and "No waveform" messages) and scrubbing still work, and the overview waveform is unaffected because it is drawn with `Canvas`. This path has not been exercised: every Mac that can run the deployment target (macOS 15, Apple silicon) has a Metal device.165166## Limitations167168- Fixed resolution of 150 columns per second (6.7 ms). Zooming in to 1 s across a wide Retina view shows several pixels per column; the shader does not interpolate between columns.169- At most 8 columns are sampled per pixel. When one pixel spans more than 8 columns (seconds per pixel above about 53 ms, for example 32 s shown in a drawable narrower than 600 pixels) some columns are skipped and short peaks can be missed.170- 8 bits per value with square-root scaling for the bands; heights are not a linear amplitude scale.171- The band split is fixed (200 Hz and 2500 Hz) and is not the mixer's EQ; band envelopes carry uncompensated filter delay.172- The bands are computed from the mono mix; stereo content that cancels in the mix is under-represented in the bands (the unused `peak` array is computed per channel).173- The beat grid is the constant-tempo grid of [analysis.md](analysis.md); the display cannot show tempo changes inside a track.174- Time and position are passed to the shader as 32-bit floats. For very long tracks at maximum zoom the precision of the playhead time approaches the size of a pixel; this has not been tested near the 150-minute limit of `TrackBuffer.maxSeconds`.175- The `gain` uniform is constant: there is no user control of waveform height.176- No listening or visual comparison against other software has been done; the correctness claims above are those of the unit tests listed.177
Files main
Menu

Theme