SPB Git

Repository Public License

visualization.md 6.0 KB 103 lines markdown

1<!--2ORIX Audio3File: docs/architecture/visualization.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->78# Visualization910Three plots — waveform, spectrum, spectrogram — in a live form (while recording) and a file form (in the workstation). All are drawn from measured data; with no audio flowing they show a flat line or an empty plot.1112## Rendering technology: why not Metal1314The brief asked for Metal "when justified by performance". It was evaluated and **not used**.1516| Work per frame | Measured cost (Mac, Apple M5 Max, release build) |17|---|---|18| Spectrum frame: 4096-point FFT, smoothing, peak hold, bands | 15.1 µs |19| Rasterise a 512 × 512 spectrogram tile through a palette table | 0.65 ms |20| Waveform summary for 60 s of audio | 14 ms, once |2122At 60 frames per second a frame is 16.7 ms. The heaviest of these, rasterising a tile, is done once per tile and cached as a `CGImage`; scrolling and zooming then draw images. With that budget a GPU pipeline would add shaders, texture management and a second code path without a measured benefit, so the app uses SwiftUI `Canvas` for vector plots and Core Graphics images for spectrogram tiles.2324Caveat: these numbers come from a Mac. An older iPhone is several times slower, and nothing has been profiled on one. If device profiling shows dropped frames, the natural next step is a Metal view that uploads the dB tiles as textures and applies the palette in a fragment shader — the tile store already keeps numeric dB values separate from their colouring for exactly that reason.2526## Live views2728<p align="center"><img src="../assets/diagrams/realtime-dsp.svg" alt="Real-time analysis thread" width="760"></p>2930The analysis thread writes into `LiveVisuals`; views read it inside a `TimelineView`. Two details matter:3132- **Rates are decoupled.** Analysis follows the audio; drawing follows the display (60 Hz for the waveform, 30 Hz for spectrum, spectrogram and meters; 4–5 Hz with Reduce Motion).33- **Data is read as view input, not inside the drawing closure.** SwiftUI redraws a `Canvas` when its inputs change. An earlier version fetched the buckets inside the closure, the inputs looked constant, and the waveform froze while audio kept flowing. The fetch now happens in the `TimelineView` body.3435The live waveform shows the latest buckets right-aligned at 3 points per bar at true scale, with ±0.5 and ±1.0 FS guides.3637## File waveform3839<p align="center"><img src="../assets/diagrams/waveform-rendering.svg" alt="Waveform rendering" width="760"></p>4041<details><summary>Mermaid source</summary>4243```mermaid44flowchart LR45  S[PCM samples] --> B[512-sample bucket: min, max, RMS]46  S --> F[4096-point Hann FFT ending at the bucket] --> Bands[7 band powers]47  B --> P[Pyramid of levels]48  Bands --> P49  P --> Sel[Pick the coarsest level with one bucket per bar]50  Sel --> R[Re-bin to bars] --> C[Canvas]51```52</details>5354`WaveformSummary.buckets(for:points:)` returns the coarsest level that still has at least one bucket per bar; `WaveformRendering.rebin` merges to exactly the number of bars. Gestures are shared by the waveform and the spectrogram through `TimelineState`: pinch to zoom about the pinch point, drag to pan (or to select, in selection mode), tap to move the playhead. Zoom is limited to 50 ms across the view.5556Quiet recordings are magnified vertically when their peak is below 0.7 FS, up to 16×, and the factor is drawn on the plot.5758| Style | Encoding |59|---|---|60| Frequency colours | one colour per bar: the band colours mixed by each band's share of RMS amplitude |61| Band shares | the bar divided bottom to top into seven segments in proportion to the same shares |62| Amplitude | minimum to maximum in the accent colour |63| RMS and peak envelope | RMS bar over a dimmer peak bar |6465Colour mixing uses amplitude shares rather than power shares: with power, a strong bass component would hide everything else. The underlying band powers are unchanged, and analysis reports use power.6667## Spectrum plot6869A dB grid every 20 dB, decade ticks on the logarithmic axis, a strip under the axis showing the seven bands, the spectrum filled with a gradient in which each horizontal position takes its band's colour, a peak-hold trace, a dashed line at the dominant frequency and a draggable cursor with a frequency and level readout. The vertical range is −120…0 dBFS for amplitude and power and −150…−20 dBFS/Hz for PSD.7071## Spectrogram7273<p align="center"><img src="../assets/diagrams/spectrogram-tiles.svg" alt="Spectrogram tile processing" width="760"></p>7475<details><summary>Mermaid source</summary>7677```mermaid78flowchart LR79  Z[Visible range and width] --> L[Level L: hop = fft/4 × 2^L]80  L --> K[Tile keys, 256 columns each]81  K --> R[Read the mono range] --> T[STFT, hop ≤ one FFT length]82  T --> M[Max-pool to the level's column width] --> Cache[(dB tiles, 28 kept)]83  Cache --> LUT[Palette table] --> I[CGImage] --> Canvas84```85</details>8687`SpectrogramTileStore` is an actor. A level's columns are `fftSize/4 × 2^L` samples wide. The transform never hops further than one FFT length, and coarser levels take the maximum over consecutive columns, so no audio is skipped at any zoom. Tiles hold numeric dB values; changing the palette, axis or range re-rasterises cached tiles without recomputing a transform. The store keeps 28 tiles; the view keeps images for what is on screen.8889Tapping reads the value under the finger from the tile data and shows time, frequency and dBFS. A legend under the plot shows the palette with the dB values at its ends.9091| Palette | Notes |92|---|---|93| ORIX Spectrum | ink → violet → blue → cyan → green → lime → amber → warm white, lightness rising |94| Scientific | viridis anchors |95| Thermal | inferno anchors |96| Monochrome | black to white |97| High Contrast | deliberate hard steps |98| Colorblind-friendly | cividis anchors |99100## Image export101102The waveform is rendered at 3200 × 1200 pixels with its title, duration and band legend. The spectrogram is computed for the whole file at up to 3200 columns and 1024 rows and saved as PNG, with the palette and dB range in the file name.103
Files main
Menu

Theme