SPB Git

Repository Public

waveform.md 15.4 KB

# Waveform display

ORIX DJ never draws raw audio samples. The analyser reduces every track to a small three-band summary once, the summary is stored with the track, and both waveform views draw from it: the scrolling deck waveform with a single Metal fragment-shader pass, the whole-track overview with a SwiftUI Canvas.

Sources:

  • Packages/ORIXDJCore/Sources/ORIXAnalysis/Waveform.swift (WaveformSummary, WaveformBuilder)
  • App/Sources/Features/Waveforms/WaveformRenderer.swift (Metal renderer and shader source)
  • App/Sources/Features/Waveforms/WaveformViews.swift (ScrollingWaveform, MetalWaveform, OverviewWaveform, OverviewCache)
  • App/Sources/App/AppSettings.swift (WaveformPalette), App/Sources/DesignSystem/Tokens.swift (deck and hot-cue colours, ORIX.Palette.waveformWell)

Related documents: analysis.md for the rest of the analysis pass, persistence.md for how the summary is stored, deck-engine.md for the deck state the renderer reads, ../design/design-system.md for the colour system and ../development/performance.md for measured costs.

# Data flow

flowchart TD
    A["Decoded audio (TrackBuffer, planar Float32)"] --> B["WaveformBuilder.build"]
    B --> C["WaveformSummary: low, mid, high, peak at 150 columns per second"]
    C --> D["AnalysisResult, JSON then zlib, table track_analysis"]
    C --> E["LoadedTrack.waveform on the deck"]
    D --> E
    E --> F["WaveformRenderer.upload: RGBA8 texture, 4096 columns per row"]
    F --> G["waveFragment: one full-screen triangle per frame"]
    H["DeckStatus: position, grid, cue, hot cues, loop"] --> G
    E --> I["WaveformSummary.overview(columns:), max-pooled to the view width"]
    I --> J["OverviewWaveform: three filled paths in a SwiftUI Canvas"]
    H --> J

# Summary format

WaveformSummary is a Codable, Sendable struct with four byte arrays of equal length and a rate:

Field Content Scaling
columnsPerSecond Columns per second of audio. 150 for a fresh analysis (WaveformBuilder.columnsPerSecond), lower after overview(columns:). Double
low Peak magnitude of the band below 200 Hz in the column 255 * sqrt(amplitude), rounded, clamped to 255
mid Peak magnitude of the 200 Hz to 2500 Hz band same
high Peak magnitude of the band above 2500 Hz same
peak Full-band sample peak over both channels linear, 255 * abs(amplitude), 255 = 0 dBFS

How the columns are made (WaveformBuilder.build):

  • A column covers sampleRate / 150 samples: 294 samples at 44.1 kHz, 320 at 48 kHz. The column count is ceil(frames * 150 / sampleRate).
  • peak is computed per channel with vDSP_maxmgv and the larger of left and right is kept.
  • The band values are computed on the mono mix (0.5 * (left + right), built once by TrackAnalyzer). The mix is filtered by 8th-order Butterworth filters, each realised as four biquad sections (WaveformBuilder.sections, Q values 0.5098, 0.6013, 0.9000, 2.5629; 48 dB per octave, -3 dB at the crossover) and run through vDSP_biquad:
    • low: low-pass at lowCrossover = 200 Hz;
    • mid: high-pass at 200 Hz followed by low-pass at highCrossover = 2500 Hz;
    • high: high-pass at 2500 Hz. The crossover frequency is clamped to 0.45 * sampleRate.
  • Each band value is the maximum magnitude of the filtered signal within the column, stored square-root scaled so that quiet material keeps visible height in 8 bits.
  • The filters are causal IIR filters applied once, forward. Their group delay is not compensated, so band envelopes lag the peak array slightly, most at the low crossover. The delay has not been measured.

Size, from the constants: the four arrays take 4 bytes per column, 600 bytes per second of audio, 180 000 bytes for a five-minute track before compression.

overview(columns:) returns a max-pooled copy for the whole-track view: output column c is the maximum of source columns c*n/columns ..< (c+1)*n/columns; columnsPerSecond is scaled accordingly. It returns self when columns >= count and an empty summary when columns <= 0.

# Storage

The summary is not stored on its own. It is a member of AnalysisResult, which DJSession.encode serialises with JSONEncoder and compresses with zlib (NSData.compressed(using: .zlib)); the blob goes to the track_analysis table of the library database (see persistence.md). Because the arrays are [UInt8], the JSON holds them as arrays of numbers; the compressed size was not measured. When a track is loaded, DJSession.storedAnalysis(for:) decodes the blob if track.analysisVersion == AnalysisResult.currentVersion (currently 1) and the summary is handed to the deck as LoadedTrack.waveform. A track without a current analysis is analysed during the load.

# What the tests verify

WaveformTests in Packages/ORIXDJCore/Tests/ORIXAnalysisTests/AnalyzerTests.swift:

Test Assertion
bandSeparation A 60 Hz sine gives low > 170, mid < 0.1 * low, high < 0.02 * low; a 1 kHz sine gives mid > 170 with low and high below 0.1 * mid; a 6 kHz sine gives high > 170, low < 0.02 * high, mid < 0.1 * high. For amplitude 0.5 the band value is within 4 of 180 (255 * sqrt(0.5)) and the linear peak within 2 of 127.5. The first and last 15 columns (100 ms) are skipped because of filter transients.
columnCountAndOverview 10 s of audio gives exactly 1500 columns at columnsPerSecond == 150; overview(columns: 100) has 100 columns at 10 columns per second, preserves the global maximum, and its first column equals the maximum of the first 15 source columns; clicks every 0.5 s appear every 75 columns with silence (peak == 0) between them.
silenceGivesNoGrid (RobustnessTests) 20 s of silence gives 3000 columns, all zero.

# Scrolling waveform (Metal)

# Structure

ScrollingWaveform (SwiftUI) hosts MetalWaveform, an NSViewRepresentable around an MTKView. Its coordinator is a WaveformRenderer, one per deck view, which is the view's MTKViewDelegate. The Metal device (MTLCreateSystemDefaultDevice()), the render pipeline and the command queue are static and shared by all decks. The shader is a Metal Shading Language string compiled at first use with device.makeLibrary(source:options:); there is no .metal file in the app bundle.

The view is configured with colorPixelFormat = .bgra8Unorm, enableSetNeedsDisplay = false and isPaused = false, so MetalKit drives draw(in:) continuously at preferredFramesPerSecond.

# Texture

WaveformRenderer.upload copies the summary into one rgba8Unorm texture, without mipmaps, usage .shaderRead:

Channel Content
R low
G mid
B high
A peak

The texture is textureWidth = 4096 texels wide and ceil(columns / 4096) rows high; column i lives at texel (i % 4096, i / 4096). A five-minute track (45 000 columns) needs 11 rows. The upload happens when the deck's LoadedTrack changes or its waveform column count changes, not per frame. When no track is loaded a 1x1 zero texture is bound.

The alpha channel is uploaded but the current fragment shader reads only .rgb: the linear peak array is not drawn by the scrolling view.

# Per-frame work

Each frame the renderer reads engine.deckStatus(deck), fills a WaveformUniforms block made only of float4 members (so the Swift and Metal layouts agree) and draws one triangle that covers the viewport (waveVertex generates it from vertex_id; there are no vertex buffers and no per-frame geometry). Everything visible is computed per pixel in waveFragment:

Uniform Content
view drawable width and height in pixels, playhead position in seconds, seconds per pixel
data columns per second, column count, texture width, track duration
grid BPM, first beat in seconds, downbeat index, cue point in seconds (-1 when no track)
loop loop start, loop end, loop active flag, gain (always 1 at present)
hotA, hotB eight hot-cue times in seconds, -1 when unset
low, mid, high band colours from the selected palette
deck deck colour, used for the loop region and loop edges
background well colour, default (0.02, 0.027, 0.055)
misc backing scale factor, playing flag (not used by the shader)

The fragment shader, in drawing order:

  1. Maps the pixel to track time: t = playhead + (x - width / 2) * secondsPerPixel. The playhead is fixed at the horizontal centre and the audio scrolls under it. Vertical position is folded to y = abs(2 * py / height - 1), so the waveform is mirrored about the centre line.
  2. Lightens the background by 0.012 outside 0...duration, so the start and end of the track are visible.
  3. Tints an active loop region with the deck colour (16 %).
  4. Beat grid: for bpm > 0, and only when beats are more than 7 * scale pixels apart, draws a line at every beat of the constant-tempo grid. Lines whose index modulo 4 matches downbeatIndex are drawn at 42 % white, other beats at 14 %.
  5. Waveform: reads up to 8 columns spanning the pixel (taps = clamp(ceil(secondsPerPixel * columnsPerSecond), 1, 8)) with texture.read (integer texel fetch, no filtering) and keeps the per-band maximum. Band heights are low, 0.86 * mid, 0.72 * high of the half height, times 0.94; they are composited in the order low, mid, high with a one-pixel anti-aliased edge. Audio left of the playhead is dimmed to 62 %.
  6. Centre line at 10 % white.
  7. Loop edges in the deck colour; the cue point as an orange line (1.0, 0.62, 0.04), full strength in the outer 45 % of the height and 55 % elsewhere.
  8. Hot cues: a thin line in one of eight fixed colours plus a small flag at the top (9 * scale pixels), so that hot cues differ from the cue point by shape as well as colour.
  9. Playhead: a white vertical line at the centre.

# Zoom and scrubbing

  • Zoom is AppSettings.waveformZoom, the number of seconds visible across the view: default 6, clamped to 1...32, stored in UserDefaults under ui.waveformZoom. It is one setting for all decks. It changes with a pinch (MagnifyGesture) or the two magnifier buttons (factor 1.5).
  • Dragging horizontally scrubs: the translation is converted to seconds with the current zoom and sent to engine.seek(deck:seconds:), so the real playhead moves.
  • There is no free scrolling away from the playhead; the view is always centred on the play position.

# Frame pacing

At the end of every draw(in:) the renderer sets view.preferredFramesPerSecond to 60 while the deck is playing and 15 otherwise. The deck position comes from DJEngine.deckStatus, read on the main thread each frame; the renderer does not interpolate between engine updates. Frame time and GPU cost of this pass have not been measured (see ../development/performance.md).

# Overview waveform

OverviewWaveform shows the whole track in a SwiftUI Canvas; it does not use Metal.

  • OverviewCache builds three closed Path outlines (low, mid, high) from waveform.overview(columns: Int(viewWidth)), mirrored about the centre, with height factors 0.94, 0.8 and 0.66. The paths are rebuilt only when the track, the view size or the column count changes.
  • Each redraw fills the well (ORIX.Palette.waveformWell, 0x05070E in both appearances), the three paths in the palette colours, a 45 % black veil over the played part, the loop region in the deck colour (35 % when active, 15 % otherwise), the cue point (0xFF9F0A), hot cues as half-height marks in ORIX.hotCueColors, and a 2-point white playhead.
  • A click or drag seeks to the proportional position. For assistive technologies the view is one adjustable element whose increment and decrement actions seek by one fiftieth of the duration; its value is the played percentage.
  • The overview does not draw the beat grid.

# Palettes

WaveformPalette (App/Sources/App/AppSettings.swift) defines the band colours as sRGB hex. The choice is stored in UserDefaults under ui.palette and applies to both the scrolling and the overview waveform.

Case Title in the interface Low Mid High
spectrum (default) Spectrum (red / green / blue) 0xFF4B3A 0x3BDC6E 0x4D8DFF
ember Ember (amber / white / cyan) 0xFF8A1F 0xF2F4FB 0x22D3EE
colorBlindSafe Colour-blind safe (blue / orange / yellow) 0x0072B2 0xE69F00 0xF0E442
mono Monochrome 0x6E7592 0xA3AAC2 0xF2F4FB

The colorBlindSafe values are the blue, orange and yellow of the Okabe-Ito palette. The default spectrum palette separates low and mid by red against green, which is the pair most affected by the common colour-vision deficiencies; colorBlindSafe and mono avoid that pair. In all palettes the bands also differ by height (1.0, 0.86, 0.72 of the scale) and by stacking order, so colour is not the only cue. None of the palettes has been checked with a colour-vision simulator or by affected users (see ../reports/accessibility.md).

Other colours are not part of the palette setting:

  • Deck colour: ORIX.deck(index) in Tokens.swift, converted to sRGB and passed as the deck uniform.
  • Hot cues: the overview uses ORIX.hotCueColors; the Metal shader has its own constant array of eight colours written as floats. The two lists were written to match but are maintained separately.
  • Cue point: orange in both views.

# When Metal is unavailable

There is no software fallback for the scrolling waveform. If MTLCreateSystemDefaultDevice() returns nil, or the shader fails to compile (the error is logged with NSLog), WaveformRenderer.pipeline is nil and draw(in:) returns before encoding anything: the scrolling view stays empty. The SwiftUI overlays (zoom buttons, the "Drop a track here" and "No waveform" messages) and scrubbing still work, and the overview waveform is unaffected because it is drawn with Canvas. This path has not been exercised: every Mac that can run the deployment target (macOS 15, Apple silicon) has a Metal device.

# Limitations

  • Fixed resolution of 150 columns per second (6.7 ms). Zooming in to 1 s across a wide Retina view shows several pixels per column; the shader does not interpolate between columns.
  • At most 8 columns are sampled per pixel. When one pixel spans more than 8 columns (seconds per pixel above about 53 ms, for example 32 s shown in a drawable narrower than 600 pixels) some columns are skipped and short peaks can be missed.
  • 8 bits per value with square-root scaling for the bands; heights are not a linear amplitude scale.
  • The band split is fixed (200 Hz and 2500 Hz) and is not the mixer's EQ; band envelopes carry uncompensated filter delay.
  • The bands are computed from the mono mix; stereo content that cancels in the mix is under-represented in the bands (the unused peak array is computed per channel).
  • The beat grid is the constant-tempo grid of analysis.md; the display cannot show tempo changes inside a track.
  • Time and position are passed to the shader as 32-bit floats. For very long tracks at maximum zoom the precision of the playhead time approaches the size of a pixel; this has not been tested near the 150-minute limit of TrackBuffer.maxSeconds.
  • The gain uniform is constant: there is no user control of waveform height.
  • No listening or visual comparison against other software has been done; the correctness claims above are those of the unit tests listed.
Files main
Menu

Theme