SPB Git

Repository Public

chart-engine.md 25.5 KB

# Chart engine

Charts in ORIX Sheet are produced by a pure model-and-drawing package with no AppKit or SwiftUI dependency, and placed in the grid by a feature package that adds caching, an inspector, an interactive viewer and a multi-panel figure editor. The same laid-out scene is replayed into a bitmap for the screen, into the PDF context for printing, and into an SVG writer for export.

Package Role Imports
Packages/SheetCharts ChartSpec, data resolution, statistics, scales, layout, the display list, CoreGraphics and SVG canvases, FigureSpec, chart suggestions SheetCore; CoreGraphics, CoreText, ImageIO, UniformTypeIdentifiers
Packages/FeatureCharts object renderers for the grid, ChartRenderCache, ChartSnapshot, chart inspector, chart viewer, ORIX Figure Studio, export commands SheetCore, SheetUI, SheetCharts, SheetFormat

Source: Packages/SheetCharts/Package.swift, Packages/FeatureCharts/Package.swift.

Swift Charts is not used. docs/research/technology-evaluation.md records the reason: it cannot export vector PDF/SVG nor be rendered off-screen deterministically.

# Pipeline

flowchart TD
    SPEC["ChartSpec<br/>JSON payload of a SheetObject"] --> RESOLVE["ChartData.resolve<br/>numbers only, missing stays missing"]
    SRC["ChartDataSource<br/>workbook, snapshot or literals"] --> RESOLVE
    RESOLVE --> BUILD["SceneBuilder<br/>statistics, scales, ticks, layout"]
    BUILD --> SCENE["ChartScene<br/>ChartLayout + display list"]
    SCENE -->|replay| CG["CGChartCanvas<br/>screen bitmap, PNG, PDF, print"]
    SCENE -->|replay| SVG["SVGChartCanvas<br/>standalone SVG"]
    SCENE -->|layout only| HIT["Hit testing, tooltips,<br/>linked selection, accessibility text"]

SVG fallback: Chart pipeline from spec to canvases

The design rule stated in the package README is that every mark derives from data: nothing decorative is drawn, and a value that cannot be represented is left out and never replaced by zero.

# 1. ChartSpec

ChartSpec is Codable, Sendable, Hashable and versioned (schema, currently 1). It is stored as the JSON payload of a SheetObject whose kind is "chart". Decoding is tolerant: stored JSON is overlaid on the defaults of the current model, so payloads lacking keys still load.

There are 21 chart kinds: line, bar (horizontal), column, area, scatter, bubble, histogram, boxPlot, violin, heatmap, waterfall, candlestick, pie, donut, radar, treemap, surface3D, contour, density, timeSeries, combo.

Source: Packages/SheetCharts/Sources/SheetCharts/Model/ChartSpec.swift (enum ChartKind).

A spec holds: series[] (name, the references used by the kind — values, x, size, open/high/low/close, lower/upper — style, primary/secondary axis, per-series mark override for combo charts, trendline, error bars, data labels); categories and rowLabels; xAxis, yAxis, y2Axis (title, unit, min/max, major step, minor ticks, scale linear/log10/log2/date, number-format code, gridlines, reversed, label rotation); legend; annotations (text, arrow, reference line, band, in data coordinates); theme; options.

A DataRef is .range(sheet:range:) with the sheet addressed by name, .literal([Double]) or .labels([String]).

FeatureCharts adds one flag outside the spec: pinTheme, written as a leading "orixPinTheme":true key of the JSON payload, which ChartSpec.decoded ignores. Without it a chart follows the app's appearance (the theme's dark variant in dark mode); with it the chart keeps its own theme.

# 2. Data resolution

ChartData.resolve(spec, source:) turns references into numbers through a ChartDataSource, a three-member protocol (values, shape, date1904). Three sources exist:

Source Use
WorkbookDataSource reads computed cell values straight from a Workbook; clips open-ended references such as A:A to the stored extent; takes a synchronize closure because Workbook is not thread-safe
ChartSnapshot (FeatureCharts) an immutable, hashable copy of exactly the cells a chart reads; used by the grid cache and background rendering
LiteralDataSource specs that only use literals and labels (samples, tests)

The missing-value rule, quoted from the README and applied throughout the marks:

Only numeric cells holding a finite number are data. Blank cells, text (including text that looks like a number), booleans, error values and NaN / ±∞ are missing.

Missing values are never plotted as zero: line marks break (unless options.connectGaps), bar marks leave the slot empty, stacks do not move, statistics exclude the value. ChartData.skippedCells counts the non-blank cells that were skipped. Further rules: values ≤ 0 are skipped on log axes; pie, donut and treemap draw strictly positive values only; a candle needs all four prices; a bubble needs a size > 0. A chart with nothing to draw shows a short note instead of an empty frame.

# 3. Layout and the display list

ChartRenderer.scene(spec, data:size:measure:viewport:formatter:) is a pure function: the same inputs give the same scene. It creates a SceneBuilder, which dispatches on the kind — pie/donut, radar, treemap and 3D surface have their own builders; every other kind goes through the Cartesian model (cartesianModel() then buildCartesian) — and accumulates two things:

  • a ChartLayout: plot rectangle, axis scales and ticks with their label frames, legend entries, colour-bar frame, every text frame, and hit-test data;
  • a display list of DrawOp values: fill, stroke, text, gradient, image, save, restore, clip, beginGroup, endGroup.

Together they form a ChartScene, which is Sendable. scene.draw(in:) replays the list into any ChartCanvas. The source comment states the cost model: building a scene is the expensive part; drawing it is a replay.

Text measurement is needed during layout, so the builder takes a TextMeasurer; both canvases implement it, and CoreTextMeasurer is the stand-alone version used when no canvas exists yet (background layout). Font matching goes through a process-wide locked cache (FontCache), because, as the README notes, concurrent first-time font lookups from many threads can stall in the system font service.

Coordinates are points with the origin at the top-left and y pointing down. Paths are ChartPath values (move, line, cubic curve, close); circles and arcs are approximated with cubic Béziers.

# 4. Canvases

Canvas Output Notes
CGChartCanvas any CGContext: view drawing, bitmap (PNG via ImageIO), PDF, the print context CGChartCanvas(context:height:flipped:)
SVGChartCanvas standalone SVG 1.1 text real <text> elements; groups named series-N, grid, axes, legend…; fonts referenced by family name, not embedded

Source: Packages/SheetCharts/Sources/SheetCharts/Canvas/CGChartCanvas.swift, SVGChartCanvas.swift; Packages/SheetCharts/README.md (section "Export formats").

Convenience entry points: ChartRenderer.draw, pngData(…scale:), pdfData, svgString. PDF output is single-page vector with text kept as text; the one exception is a heatmap with more than 4,096 cells, which embeds a raster with one pixel per cell (GridMarks.swift, rows * cols <= 4096).

# Statistical conventions

Charts that compute something from the data follow these conventions. The right-hand column is the reference the package's tests compare against; the reference numbers are generated by Tests/SheetChartsTests/Generators/gen_fixtures.py and gen_colormaps.py (numpy 2.4.4, scipy 1.17.1, matplotlib, per the README).

Topic Convention Reference used in tests
Linear ticks steps 1-2-5 × 10^k, at least two ticks inside the range, automatic ends extended to the enclosing ticks matplotlib.ticker.MaxNLocator(steps=[1,2,5,10])
Log ticks integer powers of the base; 2…9 minors; 1-2-5 multiples under one decade numpy
Calendar ticks seconds to years from serial dates through DateSerial (1900 and 1904 systems); months and years on the first day, weeks on Mondays Python datetime
Histogram bins auto, freedmanDiaconis, sturges, scott, fixedCount, fixedWidth; bins are [a, b) with the last one closed; auto = min(max(FD, range / 2√n), Sturges); at most 2,000 bins numpy.histogram_bin_edges, numpy.histogram
Normalisation count, density (count / (n · width)), probability numpy density=True
Quartiles Hyndman–Fan type 7 inclusive (default; the convention of Excel QUARTILE.INC), type 6 exclusive, type 2 averaged, type 8 medianUnbiased numpy.quantile methods linear, weibull, averaged_inverted_cdf, median_unbiased
Box plot box Q1–Q3, median line, whiskers to the most extreme observation within 1.5 · IQR (Tukey) or to min/max, the rest are outliers; optional mean matplotlib.cbook.boxplot_stats
KDE (violin, density) Gaussian kernel; bandwidth silverman = 0.9 · min(s, IQR / 1.34) · n^(−1/5) (default), scott = s · n^(−1/5), or fixed; s is the sample standard deviation; no estimate for n < 2 or zero spread scipy.stats.gaussian_kde
Stacking positive values stack upward from zero, negative values downward; 100 % mode divides by the sum of absolute values of the category numpy cumulative sums
Trendlines least squares: linear, polynomial (degree clamped to 1…8, Householder QR on centred/scaled x), exponential, logarithmic, power, trailing moving average. R² is computed in the space of the fit. On category axes x = 1, 2, 3… numpy.polyfit
Error bars fixed, percent of the value, k · sample standard deviation of the series, k · standard error (s / √n), custom ranges closed forms
Contours marching squares on grid nodes (inside = value ≥ level, saddles resolved by the cell mean); filled bands are polygon clips of the same piecewise-linear surface matplotlib / contourpy iso-line lengths, analytic circle
Treemap squarified (Bruls, Huizing & van Wijk); areas proportional to values property tests
Bubbles area proportional to size (diameter ∝ √size) closed form
Waterfall values are deltas; indices in waterfallTotals show the running sum from zero closed form
Colour maps viridis, magma, plasma, inferno, cividis, RdBu, coolwarm, BrBG, Blues, Greys — 65 anchors each, linearly interpolated; diverging maps centred on zero unless bounds are given matplotlib, within 4/255
3D surface static orthographic projection (azimuth, elevation), quads painted far-to-near by mean depth; no perspective, lighting or interaction none

Source: Packages/SheetCharts/README.md (section "Statistical conventions"); constants checked in Stats/Binning.swift (maxBins = 2000) and Stats/Regression.swift (min(max(degree, 1), 8)).

Two conventions differ from what an Excel user may expect and are flagged as such by the package:

  • Error-bar kind stddev draws ± k·s around each point, not around the series mean as Excel does.
  • Trendline R² is computed in the space of the fit (for example on ln y for an exponential fit); the README describes this as Excel's convention.

These statistics are local to the chart engine. The statistics behind worksheet functions and the analysis tools are separate code with their own validation; see ../mathematics/reference-tests.md.

# Decimation

Large series are reduced before drawing, without changing what is visible at the output resolution:

  • Lines longer than 2,000 points are reduced per horizontal bucket of 0.5 pt to at most four points — first, lowest, highest, last, in order (Decimation.minMaxIndices). Every bucket keeps its exact vertical extent. options.decimation = false disables it; smoothed and area series are not decimated.
  • Scatter series above 20,000 opaque markers skip a marker that falls on an already occupied 0.5 pt cell (Decimation.occupancyIndices).
  • Hit-test data always contains every point.

Measured by chart-gallery and recorded in the package README (release build, 640 × 400 pt, best of three; resolve + layout + draw + encode):

100,000 points PNG 2x PDF SVG
line (decimated) 40–45 ms 8–9 ms 5–6 ms
line, decimation off 1.4–1.7 s — —
scatter 0.26–0.57 s 0.29–0.85 s 0.10–0.18 s

Source: Packages/SheetCharts/README.md (section "Decimation"). The README gives ranges over two runs and notes that the machine was shared with other builds; it does not state the machine model or date for this table. The numbers were not re-measured for this document.

# Charts in the grid: caching

The grid (grid-engine.md) does not know about charts. ChartsFeature.register() registers an ObjectRenderer for the object kinds "chart" and "figure" with the workspace (the extension mechanism is described in overview.md); the grid calls draw(object, session, context, rect, dark) for each visible object, and open on double-click.

Both renderers go through ChartRenderCache.shared (main actor). One cache entry per object id holds four levels, each rebuilt only when its inputs change:

flowchart TD
    DRAW["ObjectRenderer.draw<br/>called by the grid or by printing"] --> PAY["Payload bytes changed?<br/>decode ChartSpec again"]
    PAY --> SNAP["contentRevision moved?<br/>capture a new ChartSnapshot"]
    SNAP -->|differs| DROP["Snapshot differs:<br/>drop scene and image"]
    SNAP -->|equal| KEY["RenderKey<br/>size, pixel scale, dark"]
    DROP --> KEY
    KEY -->|off-screen| VEC["Not drawing to the screen:<br/>replay scene as vectors"]
    KEY -->|cached| BLIT["Image current for the key:<br/>blit cached CGImage"]
    KEY -->|slow chart| ASYNC["Last render over 30 ms:<br/>blit stale image,<br/>render in the background"]
    KEY -->|else| SYNC["Otherwise:<br/>layout and rasterise now"]

SVG fallback: Chart render cache decision flow

  1. Payload. If the object's payload bytes differ from the cached ones, the spec is decoded again. The stale image is kept so that a slow chart does not blank while it re-renders.
  2. Snapshot. When session.contentRevision has moved, a new ChartSnapshot is captured: an immutable copy of the cells of every range the chart references. Two snapshots compare equal exactly when the chart would draw the same data, so editing unrelated cells costs one snapshot comparison and nothing else. Only a differing snapshot drops the scene and the image.
  3. Scene. Layout at the object's natural size (figures use their own size) and appearance.
  4. Image. A CGImage rasterised for a RenderKey — width, height, device pixels per point quantised to 1/64, dark flag. Bitmaps larger than 16,384 pixels on a side are not produced.

Locking. Capturing a snapshot reads the workbook, which the calculation thread may be writing (calculation-graph.md). The cache uses lock.try() on the engine lock: the grid calls renderers with the lock free, while the print renderer calls them on the same thread while already holding the (non-recursive) lock. When the lock is busy because a background recalculation is in progress and a snapshot already exists, the previous data is drawn and refreshed on the redraw that follows the recalculation.

Slow charts. When the previous render of an entry took longer than asyncThreshold (0.030 s), the stale image is drawn at once (with interpolation) and layout plus rasterisation move to a concurrent background queue (ai.spboucher.orix.charts.render). A generation counter discards results that were overtaken by a newer payload or snapshot; on completion the session is invalidated so the grid redraws.

Capacity is 96 entries; when it is exceeded the least recently used entries are evicted until three quarters of the capacity remain. The cache keeps hit/miss counters and timings (Stats) for tests and diagnostics. No measured hit or miss times are stored in the repository at the time of writing, so none are quoted here.

# Structural edits

Chart ranges are addressed by sheet name and are not part of the formula dependency graph. ChartStructure.install(on:) registers a CalcEngine.structuralObservers entry that rewrites the JSON payloads of chart and figure objects when rows or columns are inserted or deleted, a block is moved, or a sheet is renamed. It runs inside the undo snapshot of the edit, so undo restores the previous payloads. When a referenced range is deleted entirely, the chart keeps its last range rather than losing the series. The observer is installed lazily: the first time a chart is drawn or a chart command runs, and for every new session through the sessionObservers hook set up in App/Sources/Features.swift.

# Vector printing

When the destination context does not draw to the screen (NSGraphicsContext.currentContextDrawingToScreen() is false — printing and PDF export of a sheet), the cache does not use the bitmap: it replays the scene as vectors through CGChartCanvas(context:height:flipped: true). Printed charts therefore keep text as text and paths as paths, with the same heatmap exception as in PDF export. PrintRenderer in SheetUI calls the same registered object renderers as the grid; no separate chart code exists for printing.

Figures are publication artefacts and are never re-themed: the cache forces the light appearance for them and preserves their aspect ratio inside the object frame, whereas charts fill their frame.

# Interactive viewer

Double-clicking a chart opens the Chart Viewer pane (ChartCanvasView), which draws the scene as vectors and uses the ChartLayout for interaction:

  • hover tooltips: the mark under the pointer, or all series at the nearest x (hitTest, items(alignedWith:));
  • zoom (scroll wheel, pinch, rectangle drag, +/= keys) and pan (pan mode or ⌥-drag), implemented by passing a ChartViewport back into ChartRenderer.scene; zooming out past the whole chart returns to the automatic view;
  • linked selection: ChartLinking.sourceCell maps a mark back to the cell it was drawn from. Marks that summarise several cells (histogram bins, density curves, box plots, violins) and inline data have no source cell;
  • a what-if slider that drives one numeric input cell: while the knob moves, the cell is written outside the undo history so formulas recalculate and the chart redraws; when the drag ends the original value is restored and the final value committed once, so a drag is exactly one undo step.

ChartAccessibility.describe generates a text alternative from the data (kind, title, number of series and categories, minimum/maximum/last per series). Accessibility of the application as a whole is covered in ../development/accessibility.md.

# Number formatting

SheetCharts has a small built-in axis formatter (automatic decimals from the tick step, thousands separators from 10,000, ×10ⁿ beyond 10⁹ or below 10⁻⁴, and a subset of format codes). Its automatic labels are English/invariant. To get the application's number formats, FeatureCharts passes a ChartValueFormatter closure (ChartFormat.formatter) that delegates explicit format codes to SheetFormat's NumberFormat with the current locale and the workbook's date system. An empty or General code falls back to the built-in formatter.

# Themes

ChartTheme.presets contains ten themes: Light, Dark, Okabe–Ito and its dark variant, Publication, Minimal, Financial, Presentation (light and dark), Monochrome. ChartPalette additionally carries Tol "bright" and Tableau 10. theme.variant(dark:) switches surfaces and palette. In dark mode FeatureCharts replaces the stock dark background with the ORIX dark surface (0x171B2B); the application palette is documented in ../design/design-system.md.

Source: Packages/SheetCharts/Sources/SheetCharts/Model/ChartTheme.swift (presets), Packages/FeatureCharts/Sources/FeatureCharts/Core/ChartPayload.swift (effectiveTheme).

# Figure Studio

A FigureSpec (object kind "figure", same tolerant JSON coding as charts) places FigurePanels — each a ChartSpec with row, column, spans and a label — on a rows × cols grid of a figure whose size is given in points, with millimetre conversion (pointsPerMillimetre = 72 / 25.4). One FigureTypography (family and base size, from which the size ladder derives) and one ChartTheme apply to every panel. Panel titles move to a header line next to the panel label ("(a)", "A"…). shareX / shareY unify the automatic domains of continuous axes per column / row and keep the axis title on the outer panels only; panels keep their own tick labels.

FigureRenderer lays each panel out with ChartRenderer.scene and composes the panel scenes into a FigureScene, which replays into the same canvases (each panel translated, clipped and wrapped in a panel-N group). It offers scene, draw, pngData, pdfData, svgString and exportData.

Six templates are built in:

Template id Name
academic-single Academic paper — single column
academic-double Academic paper — double column
scientific Scientific publication
financial Financial report
presentation Business presentation (16:9)
dashboard Dashboard

Source: Packages/SheetCharts/Sources/SheetCharts/Figure/FigureSpec.swift (FigureTemplates.all).

The README is explicit that templates are size, grid and typography presets only; they do not claim compliance with the requirements of any journal or publisher.

The Figure Studio pane (FeatureCharts/Figure) edits a draft FigureSpec, previews it, and inserts or updates the figure object in the sheet.

# Export

What Formats Resolution
One chart PNG, PDF, SVG PNG at scale pixels per point (default 2)
One figure PNG, PDF, SVG PNG at a DPI of the figure's physical size (default 300; the export preset of a template stores its own format and DPI)
"Export All Charts…" one file per chart and figure of the workbook, named after the object file names are sanitised (path separators and control characters removed, at most 80 characters, made unique)

Source: Packages/FeatureCharts/Sources/FeatureCharts/Render/ChartExport.swift.

# Chart suggestions

ChartSuggest.suggest(for:in:) is deterministic code, not AI. It trims the selection to its content, detects a header row and a category column, and proposes charts best-first (time series for dates, column/bar/pie for one measure, grouped and stacked columns for several, histogram/box plot/density for one long numeric column, scatter/bubble for two/three numeric columns, heatmap/contour/surface for a bare numeric matrix, candlestick for Open/High/Low/Close headers). It samples at most 2,000 rows (sampleRows). The chart picker uses it; five kinds also have direct shortcuts (column, line, scatter, pie, histogram on ⌥⌘1…5, per ChartBuilder.quickKinds).

The on-device assistant has a separate "Suggest Charts" task that returns textual ideas only; see intelligence.md.

# Tests

Packages/SheetCharts/Tests contains 43 @Test functions, three of them parameterised (over the 21 kinds, per the README), and Packages/FeatureCharts/Tests contains 51 (counts of test functions in the source on 2026-11-21; executed totals are in docs/reports/tests.json when present). The README describes their scope: numeric algorithms against the Python references; every kind rendered to SVG (parsed with XMLParser), PDF and PNG on normal, empty, single-point, all-blank, negative, extreme-magnitude and dirty data; every kind in every theme and at extreme sizes; structural checks (marks per datum, positions through the axis scale within 0.5 pt, stack sums, bin geometry, wedge angles, bubble areas); absence of clipped or overlapping text on the gallery charts at three sizes; vector-PDF heuristics. See ../development/testing.md.

chart-gallery <dir> writes a PNG (2x), SVG and PDF per chart kind, dark PNGs, option variants and multi-panel figures, and prints timings; charts-preview is the preview executable of the feature package (../development/setup.md).

# Known limitations

From the package README, checked against the code where a constant is involved:

  • The 3D surface is a static projection; the painter's algorithm can mis-order quads on steep, self-occluding surfaces.
  • Contour charts have no in-line level labels; heatmap and contour colour bars have no title.
  • No legend for bubble sizes; one legend per chart; long legends are truncated.
  • Markers of one series are filled as compound paths: with fillOpacity < 1, overlapping markers of the same series do not darken each other.
  • Text is single-line and left-to-right; long labels are truncated with an ellipsis. No rich text, no wrapping.
  • SVG and PDF reference fonts by family; a viewer without the font substitutes it.
  • Histograms of several series share the bin edges of the pooled data and are overlaid, not dodged or stacked.
  • Stacking applies to columns, bars, areas and category-axis lines, not to continuous-x series.
  • XLSX charts: whether charts are read from or written to .xlsx files is a property of the file layer and is documented in ../compatibility/formats.md and ../compatibility/excel.md, not here. The chart model itself is native JSON and is stored as such in .orixsheet files (persistence.md).
Files main
Menu

Theme