SPB Git

Repository Public

chart-engine.md 25.5 KB 257 lines markdown

1# Chart engine23Charts in ORIX Sheet are produced by a pure model-and-drawing package with no AppKit or SwiftUI dependency, and placed in the grid by a feature package that adds caching, an inspector, an interactive viewer and a multi-panel figure editor. The same laid-out scene is replayed into a bitmap for the screen, into the PDF context for printing, and into an SVG writer for export.45| Package | Role | Imports |6|---|---|---|7| `Packages/SheetCharts` | `ChartSpec`, data resolution, statistics, scales, layout, the display list, CoreGraphics and SVG canvases, `FigureSpec`, chart suggestions | `SheetCore`; CoreGraphics, CoreText, ImageIO, UniformTypeIdentifiers |8| `Packages/FeatureCharts` | object renderers for the grid, `ChartRenderCache`, `ChartSnapshot`, chart inspector, chart viewer, ORIX Figure Studio, export commands | `SheetCore`, `SheetUI`, `SheetCharts`, `SheetFormat` |910Source: `Packages/SheetCharts/Package.swift`, `Packages/FeatureCharts/Package.swift`.1112Swift Charts is not used. `docs/research/technology-evaluation.md` records the reason: it cannot export vector PDF/SVG nor be rendered off-screen deterministically.1314## Pipeline1516<!-- diagram: chart-engine-pipeline -->17```mermaid18flowchart TD19    SPEC["ChartSpec<br/>JSON payload of a SheetObject"] --> RESOLVE["ChartData.resolve<br/>numbers only, missing stays missing"]20    SRC["ChartDataSource<br/>workbook, snapshot or literals"] --> RESOLVE21    RESOLVE --> BUILD["SceneBuilder<br/>statistics, scales, ticks, layout"]22    BUILD --> SCENE["ChartScene<br/>ChartLayout + display list"]23    SCENE -->|replay| CG["CGChartCanvas<br/>screen bitmap, PNG, PDF, print"]24    SCENE -->|replay| SVG["SVGChartCanvas<br/>standalone SVG"]25    SCENE -->|layout only| HIT["Hit testing, tooltips,<br/>linked selection, accessibility text"]26```27SVG fallback: ![Chart pipeline from spec to canvases](../assets/diagrams/chart-engine-pipeline.svg)2829The design rule stated in the package README is that **every mark derives from data**: nothing decorative is drawn, and a value that cannot be represented is left out and never replaced by zero.3031### 1. ChartSpec3233`ChartSpec` is `Codable`, `Sendable`, `Hashable` and versioned (`schema`, currently 1). It is stored as the JSON payload of a `SheetObject` whose kind is `"chart"`. Decoding is tolerant: stored JSON is overlaid on the defaults of the current model, so payloads lacking keys still load.3435There are **21 chart kinds**: line, bar (horizontal), column, area, scatter, bubble, histogram, boxPlot, violin, heatmap, waterfall, candlestick, pie, donut, radar, treemap, surface3D, contour, density, timeSeries, combo.3637Source: `Packages/SheetCharts/Sources/SheetCharts/Model/ChartSpec.swift` (`enum ChartKind`).3839A spec holds: `series[]` (name, the references used by the kind — `values`, `x`, `size`, `open/high/low/close`, `lower/upper` — style, primary/secondary axis, per-series mark override for combo charts, trendline, error bars, data labels); `categories` and `rowLabels`; `xAxis`, `yAxis`, `y2Axis` (title, unit, min/max, major step, minor ticks, scale linear/log10/log2/date, number-format code, gridlines, reversed, label rotation); `legend`; `annotations` (text, arrow, reference line, band, in data coordinates); `theme`; `options`.4041A `DataRef` is `.range(sheet:range:)` with the sheet addressed **by name**, `.literal([Double])` or `.labels([String])`.4243`FeatureCharts` adds one flag outside the spec: `pinTheme`, written as a leading `"orixPinTheme":true` key of the JSON payload, which `ChartSpec.decoded` ignores. Without it a chart follows the app's appearance (the theme's dark variant in dark mode); with it the chart keeps its own theme.4445### 2. Data resolution4647`ChartData.resolve(spec, source:)` turns references into numbers through a `ChartDataSource`, a three-member protocol (`values`, `shape`, `date1904`). Three sources exist:4849| Source | Use |50|---|---|51| `WorkbookDataSource` | reads **computed** cell values straight from a `Workbook`; clips open-ended references such as `A:A` to the stored extent; takes a `synchronize` closure because `Workbook` is not thread-safe |52| `ChartSnapshot` (FeatureCharts) | an immutable, hashable copy of exactly the cells a chart reads; used by the grid cache and background rendering |53| `LiteralDataSource` | specs that only use literals and labels (samples, tests) |5455The missing-value rule, quoted from the README and applied throughout the marks:5657> Only numeric cells holding a finite number are data. Blank cells, text (including text that looks like a number), booleans, error values and NaN / ±∞ are *missing*.5859Missing values are never plotted as zero: line marks break (unless `options.connectGaps`), bar marks leave the slot empty, stacks do not move, statistics exclude the value. `ChartData.skippedCells` counts the non-blank cells that were skipped. Further rules: values ≤ 0 are skipped on log axes; pie, donut and treemap draw strictly positive values only; a candle needs all four prices; a bubble needs a size > 0. A chart with nothing to draw shows a short note instead of an empty frame.6061### 3. Layout and the display list6263`ChartRenderer.scene(spec, data:size:measure:viewport:formatter:)` is a pure function: the same inputs give the same scene. It creates a `SceneBuilder`, which dispatches on the kind — pie/donut, radar, treemap and 3D surface have their own builders; every other kind goes through the Cartesian model (`cartesianModel()` then `buildCartesian`) — and accumulates two things:6465- a `ChartLayout`: plot rectangle, axis scales and ticks with their label frames, legend entries, colour-bar frame, every text frame, and hit-test data;66- a **display list** of `DrawOp` values: `fill`, `stroke`, `text`, `gradient`, `image`, `save`, `restore`, `clip`, `beginGroup`, `endGroup`.6768Together they form a `ChartScene`, which is `Sendable`. `scene.draw(in:)` replays the list into any `ChartCanvas`. The source comment states the cost model: building a scene is the expensive part; drawing it is a replay.6970Text measurement is needed during layout, so the builder takes a `TextMeasurer`; both canvases implement it, and `CoreTextMeasurer` is the stand-alone version used when no canvas exists yet (background layout). Font matching goes through a process-wide locked cache (`FontCache`), because, as the README notes, concurrent first-time font lookups from many threads can stall in the system font service.7172Coordinates are points with the origin at the top-left and y pointing down. Paths are `ChartPath` values (move, line, cubic curve, close); circles and arcs are approximated with cubic Béziers.7374### 4. Canvases7576| Canvas | Output | Notes |77|---|---|---|78| `CGChartCanvas` | any `CGContext`: view drawing, bitmap (PNG via ImageIO), PDF, the print context | `CGChartCanvas(context:height:flipped:)` |79| `SVGChartCanvas` | standalone SVG 1.1 text | real `<text>` elements; groups named `series-N`, `grid`, `axes`, `legend`…; fonts referenced by family name, not embedded |8081Source: `Packages/SheetCharts/Sources/SheetCharts/Canvas/CGChartCanvas.swift`, `SVGChartCanvas.swift`; `Packages/SheetCharts/README.md` (section "Export formats").8283Convenience entry points: `ChartRenderer.draw`, `pngData(…scale:)`, `pdfData`, `svgString`. PDF output is single-page vector with text kept as text; the one exception is a heatmap with more than 4,096 cells, which embeds a raster with one pixel per cell (`GridMarks.swift`, `rows * cols <= 4096`).8485## Statistical conventions8687Charts that compute something from the data follow these conventions. The right-hand column is the reference the package's tests compare against; the reference numbers are generated by `Tests/SheetChartsTests/Generators/gen_fixtures.py` and `gen_colormaps.py` (numpy 2.4.4, scipy 1.17.1, matplotlib, per the README).8889| Topic | Convention | Reference used in tests |90|---|---|---|91| Linear ticks | steps 1-2-5 × 10^k, at least two ticks inside the range, automatic ends extended to the enclosing ticks | `matplotlib.ticker.MaxNLocator(steps=[1,2,5,10])` |92| Log ticks | integer powers of the base; 2…9 minors; 1-2-5 multiples under one decade | numpy |93| Calendar ticks | seconds to years from serial dates through `DateSerial` (1900 and 1904 systems); months and years on the first day, weeks on Mondays | Python `datetime` |94| Histogram bins | `auto`, `freedmanDiaconis`, `sturges`, `scott`, `fixedCount`, `fixedWidth`; bins are `[a, b)` with the last one closed; `auto` = min(max(FD, range / 2√n), Sturges); at most 2,000 bins | `numpy.histogram_bin_edges`, `numpy.histogram` |95| Normalisation | count, density (count / (n · width)), probability | numpy `density=True` |96| Quartiles | Hyndman–Fan type 7 `inclusive` (default; the convention of Excel `QUARTILE.INC`), type 6 `exclusive`, type 2 `averaged`, type 8 `medianUnbiased` | `numpy.quantile` methods linear, weibull, averaged_inverted_cdf, median_unbiased |97| Box plot | box Q1–Q3, median line, whiskers to the most extreme observation within 1.5 · IQR (Tukey) or to min/max, the rest are outliers; optional mean | `matplotlib.cbook.boxplot_stats` |98| KDE (violin, density) | Gaussian kernel; bandwidth `silverman` = 0.9 · min(s, IQR / 1.34) · n^(−1/5) (default), `scott` = s · n^(−1/5), or fixed; s is the sample standard deviation; no estimate for n < 2 or zero spread | `scipy.stats.gaussian_kde` |99| Stacking | positive values stack upward from zero, negative values downward; 100 % mode divides by the sum of absolute values of the category | numpy cumulative sums |100| Trendlines | least squares: linear, polynomial (degree clamped to 1…8, Householder QR on centred/scaled x), exponential, logarithmic, power, trailing moving average. R² is computed in the space of the fit. On category axes x = 1, 2, 3… | `numpy.polyfit` |101| Error bars | fixed, percent of the value, k · sample standard deviation of the series, k · standard error (s / √n), custom ranges | closed forms |102| Contours | marching squares on grid nodes (inside = value ≥ level, saddles resolved by the cell mean); filled bands are polygon clips of the same piecewise-linear surface | matplotlib / contourpy iso-line lengths, analytic circle |103| Treemap | squarified (Bruls, Huizing & van Wijk); areas proportional to values | property tests |104| Bubbles | **area** proportional to size (diameter ∝ √size) | closed form |105| Waterfall | values are deltas; indices in `waterfallTotals` show the running sum from zero | closed form |106| Colour maps | viridis, magma, plasma, inferno, cividis, RdBu, coolwarm, BrBG, Blues, Greys — 65 anchors each, linearly interpolated; diverging maps centred on zero unless bounds are given | matplotlib, within 4/255 |107| 3D surface | static orthographic projection (azimuth, elevation), quads painted far-to-near by mean depth; no perspective, lighting or interaction | none |108109Source: `Packages/SheetCharts/README.md` (section "Statistical conventions"); constants checked in `Stats/Binning.swift` (`maxBins = 2000`) and `Stats/Regression.swift` (`min(max(degree, 1), 8)`).110111Two conventions differ from what an Excel user may expect and are flagged as such by the package:112113- Error-bar kind `stddev` draws ± k·s around **each point**, not around the series mean as Excel does.114- Trendline R² is computed in the space of the fit (for example on ln y for an exponential fit); the README describes this as Excel's convention.115116These statistics are local to the chart engine. The statistics behind worksheet functions and the analysis tools are separate code with their own validation; see [../mathematics/reference-tests.md](../mathematics/reference-tests.md).117118## Decimation119120Large series are reduced before drawing, without changing what is visible at the output resolution:121122- **Lines** longer than 2,000 points are reduced per horizontal bucket of 0.5 pt to at most four points — first, lowest, highest, last, in order (`Decimation.minMaxIndices`). Every bucket keeps its exact vertical extent. `options.decimation = false` disables it; smoothed and area series are not decimated.123- **Scatter** series above 20,000 opaque markers skip a marker that falls on an already occupied 0.5 pt cell (`Decimation.occupancyIndices`).124- Hit-test data always contains every point.125126Measured by `chart-gallery` and recorded in the package README (release build, 640 × 400 pt, best of three; resolve + layout + draw + encode):127128| 100,000 points | PNG 2x | PDF | SVG |129|---|---|---|---|130| line (decimated) | 40–45 ms | 8–9 ms | 5–6 ms |131| line, decimation off | 1.4–1.7 s | — | — |132| scatter | 0.26–0.57 s | 0.29–0.85 s | 0.10–0.18 s |133134Source: `Packages/SheetCharts/README.md` (section "Decimation"). The README gives ranges over two runs and notes that the machine was shared with other builds; it does not state the machine model or date for this table. The numbers were not re-measured for this document.135136## Charts in the grid: caching137138The grid ([grid-engine.md](grid-engine.md)) does not know about charts. `ChartsFeature.register()` registers an `ObjectRenderer` for the object kinds `"chart"` and `"figure"` with the workspace (the extension mechanism is described in [overview.md](overview.md)); the grid calls `draw(object, session, context, rect, dark)` for each visible object, and `open` on double-click.139140Both renderers go through `ChartRenderCache.shared` (main actor). One cache entry per object id holds four levels, each rebuilt only when its inputs change:141142<!-- diagram: chart-engine-cache -->143```mermaid144flowchart TD145    DRAW["ObjectRenderer.draw<br/>called by the grid or by printing"] --> PAY["Payload bytes changed?<br/>decode ChartSpec again"]146    PAY --> SNAP["contentRevision moved?<br/>capture a new ChartSnapshot"]147    SNAP -->|differs| DROP["Snapshot differs:<br/>drop scene and image"]148    SNAP -->|equal| KEY["RenderKey<br/>size, pixel scale, dark"]149    DROP --> KEY150    KEY -->|off-screen| VEC["Not drawing to the screen:<br/>replay scene as vectors"]151    KEY -->|cached| BLIT["Image current for the key:<br/>blit cached CGImage"]152    KEY -->|slow chart| ASYNC["Last render over 30 ms:<br/>blit stale image,<br/>render in the background"]153    KEY -->|else| SYNC["Otherwise:<br/>layout and rasterise now"]154```155SVG fallback: ![Chart render cache decision flow](../assets/diagrams/chart-engine-cache.svg)1561571. **Payload.** If the object's payload bytes differ from the cached ones, the spec is decoded again. The stale image is kept so that a slow chart does not blank while it re-renders.1582. **Snapshot.** When `session.contentRevision` has moved, a new `ChartSnapshot` is captured: an immutable copy of the cells of every range the chart references. Two snapshots compare equal exactly when the chart would draw the same data, so editing unrelated cells costs one snapshot comparison and nothing else. Only a differing snapshot drops the scene and the image.1593. **Scene.** Layout at the object's natural size (figures use their own size) and appearance.1604. **Image.** A `CGImage` rasterised for a `RenderKey` — width, height, device pixels per point quantised to 1/64, dark flag. Bitmaps larger than 16,384 pixels on a side are not produced.161162Locking. Capturing a snapshot reads the workbook, which the calculation thread may be writing ([calculation-graph.md](calculation-graph.md)). The cache uses `lock.try()` on the engine lock: the grid calls renderers with the lock free, while the print renderer calls them on the same thread while already holding the (non-recursive) lock. When the lock is busy because a background recalculation is in progress and a snapshot already exists, the previous data is drawn and refreshed on the redraw that follows the recalculation.163164Slow charts. When the previous render of an entry took longer than `asyncThreshold` (0.030 s), the stale image is drawn at once (with interpolation) and layout plus rasterisation move to a concurrent background queue (`ai.spboucher.orix.charts.render`). A generation counter discards results that were overtaken by a newer payload or snapshot; on completion the session is invalidated so the grid redraws.165166Capacity is 96 entries; when it is exceeded the least recently used entries are evicted until three quarters of the capacity remain. The cache keeps hit/miss counters and timings (`Stats`) for tests and diagnostics. No measured hit or miss times are stored in the repository at the time of writing, so none are quoted here.167168### Structural edits169170Chart ranges are addressed by sheet name and are not part of the formula dependency graph. `ChartStructure.install(on:)` registers a `CalcEngine.structuralObservers` entry that rewrites the JSON payloads of chart and figure objects when rows or columns are inserted or deleted, a block is moved, or a sheet is renamed. It runs inside the undo snapshot of the edit, so undo restores the previous payloads. When a referenced range is deleted entirely, the chart keeps its last range rather than losing the series. The observer is installed lazily: the first time a chart is drawn or a chart command runs, and for every new session through the `sessionObservers` hook set up in `App/Sources/Features.swift`.171172## Vector printing173174When the destination context does not draw to the screen (`NSGraphicsContext.currentContextDrawingToScreen()` is false — printing and PDF export of a sheet), the cache does not use the bitmap: it replays the scene as vectors through `CGChartCanvas(context:height:flipped: true)`. Printed charts therefore keep text as text and paths as paths, with the same heatmap exception as in PDF export. `PrintRenderer` in `SheetUI` calls the same registered object renderers as the grid; no separate chart code exists for printing.175176Figures are publication artefacts and are never re-themed: the cache forces the light appearance for them and preserves their aspect ratio inside the object frame, whereas charts fill their frame.177178## Interactive viewer179180Double-clicking a chart opens the Chart Viewer pane (`ChartCanvasView`), which draws the scene as vectors and uses the `ChartLayout` for interaction:181182- hover tooltips: the mark under the pointer, or all series at the nearest x (`hitTest`, `items(alignedWith:)`);183- zoom (scroll wheel, pinch, rectangle drag, `+`/`=` keys) and pan (pan mode or ⌥-drag), implemented by passing a `ChartViewport` back into `ChartRenderer.scene`; zooming out past the whole chart returns to the automatic view;184- linked selection: `ChartLinking.sourceCell` maps a mark back to the cell it was drawn from. Marks that summarise several cells (histogram bins, density curves, box plots, violins) and inline data have no source cell;185- a what-if slider that drives one numeric input cell: while the knob moves, the cell is written outside the undo history so formulas recalculate and the chart redraws; when the drag ends the original value is restored and the final value committed once, so a drag is exactly one undo step.186187`ChartAccessibility.describe` generates a text alternative from the data (kind, title, number of series and categories, minimum/maximum/last per series). Accessibility of the application as a whole is covered in [../development/accessibility.md](../development/accessibility.md).188189## Number formatting190191`SheetCharts` has a small built-in axis formatter (automatic decimals from the tick step, thousands separators from 10,000, `×10ⁿ` beyond 10⁹ or below 10⁻⁴, and a subset of format codes). Its automatic labels are English/invariant. To get the application's number formats, `FeatureCharts` passes a `ChartValueFormatter` closure (`ChartFormat.formatter`) that delegates explicit format codes to `SheetFormat`'s `NumberFormat` with the current locale and the workbook's date system. An empty or `General` code falls back to the built-in formatter.192193## Themes194195`ChartTheme.presets` contains ten themes: Light, Dark, Okabe–Ito and its dark variant, Publication, Minimal, Financial, Presentation (light and dark), Monochrome. `ChartPalette` additionally carries Tol "bright" and Tableau 10. `theme.variant(dark:)` switches surfaces and palette. In dark mode `FeatureCharts` replaces the stock dark background with the ORIX dark surface (`0x171B2B`); the application palette is documented in [../design/design-system.md](../design/design-system.md).196197Source: `Packages/SheetCharts/Sources/SheetCharts/Model/ChartTheme.swift` (`presets`), `Packages/FeatureCharts/Sources/FeatureCharts/Core/ChartPayload.swift` (`effectiveTheme`).198199## Figure Studio200201A `FigureSpec` (object kind `"figure"`, same tolerant JSON coding as charts) places `FigurePanel`s — each a `ChartSpec` with row, column, spans and a label — on a `rows × cols` grid of a figure whose size is given in points, with millimetre conversion (`pointsPerMillimetre = 72 / 25.4`). One `FigureTypography` (family and base size, from which the size ladder derives) and one `ChartTheme` apply to every panel. Panel titles move to a header line next to the panel label ("(a)", "A"…). `shareX` / `shareY` unify the automatic domains of continuous axes per column / row and keep the axis title on the outer panels only; panels keep their own tick labels.202203`FigureRenderer` lays each panel out with `ChartRenderer.scene` and composes the panel scenes into a `FigureScene`, which replays into the same canvases (each panel translated, clipped and wrapped in a `panel-N` group). It offers `scene`, `draw`, `pngData`, `pdfData`, `svgString` and `exportData`.204205Six templates are built in:206207| Template id | Name |208|---|---|209| `academic-single` | Academic paper — single column |210| `academic-double` | Academic paper — double column |211| `scientific` | Scientific publication |212| `financial` | Financial report |213| `presentation` | Business presentation (16:9) |214| `dashboard` | Dashboard |215216Source: `Packages/SheetCharts/Sources/SheetCharts/Figure/FigureSpec.swift` (`FigureTemplates.all`).217218The README is explicit that **templates are size, grid and typography presets only; they do not claim compliance with the requirements of any journal or publisher.**219220The Figure Studio pane (`FeatureCharts/Figure`) edits a draft `FigureSpec`, previews it, and inserts or updates the figure object in the sheet.221222## Export223224| What | Formats | Resolution |225|---|---|---|226| One chart | PNG, PDF, SVG | PNG at `scale` pixels per point (default 2) |227| One figure | PNG, PDF, SVG | PNG at a DPI of the figure's physical size (default 300; the export preset of a template stores its own format and DPI) |228| "Export All Charts…" | one file per chart and figure of the workbook, named after the object | file names are sanitised (path separators and control characters removed, at most 80 characters, made unique) |229230Source: `Packages/FeatureCharts/Sources/FeatureCharts/Render/ChartExport.swift`.231232## Chart suggestions233234`ChartSuggest.suggest(for:in:)` is deterministic code, not AI. It trims the selection to its content, detects a header row and a category column, and proposes charts best-first (time series for dates, column/bar/pie for one measure, grouped and stacked columns for several, histogram/box plot/density for one long numeric column, scatter/bubble for two/three numeric columns, heatmap/contour/surface for a bare numeric matrix, candlestick for Open/High/Low/Close headers). It samples at most 2,000 rows (`sampleRows`). The chart picker uses it; five kinds also have direct shortcuts (column, line, scatter, pie, histogram on ⌥⌘1…5, per `ChartBuilder.quickKinds`).235236The on-device assistant has a separate "Suggest Charts" task that returns textual ideas only; see [intelligence.md](intelligence.md).237238## Tests239240`Packages/SheetCharts/Tests` contains 43 `@Test` functions, three of them parameterised (over the 21 kinds, per the README), and `Packages/FeatureCharts/Tests` contains 51 (counts of test functions in the source on 2026-11-21; executed totals are in `docs/reports/tests.json` when present). The README describes their scope: numeric algorithms against the Python references; every kind rendered to SVG (parsed with `XMLParser`), PDF and PNG on normal, empty, single-point, all-blank, negative, extreme-magnitude and dirty data; every kind in every theme and at extreme sizes; structural checks (marks per datum, positions through the axis scale within 0.5 pt, stack sums, bin geometry, wedge angles, bubble areas); absence of clipped or overlapping text on the gallery charts at three sizes; vector-PDF heuristics. See [../development/testing.md](../development/testing.md).241242`chart-gallery <dir>` writes a PNG (2x), SVG and PDF per chart kind, dark PNGs, option variants and multi-panel figures, and prints timings; `charts-preview` is the preview executable of the feature package ([../development/setup.md](../development/setup.md)).243244## Known limitations245246From the package README, checked against the code where a constant is involved:247248- The 3D surface is a static projection; the painter's algorithm can mis-order quads on steep, self-occluding surfaces.249- Contour charts have no in-line level labels; heatmap and contour colour bars have no title.250- No legend for bubble sizes; one legend per chart; long legends are truncated.251- Markers of one series are filled as compound paths: with `fillOpacity < 1`, overlapping markers of the same series do not darken each other.252- Text is single-line and left-to-right; long labels are truncated with an ellipsis. No rich text, no wrapping.253- SVG and PDF reference fonts by family; a viewer without the font substitutes it.254- Histograms of several series share the bin edges of the pooled data and are overlaid, not dodged or stacked.255- Stacking applies to columns, bars, areas and category-axis lines, not to continuous-x series.256- XLSX charts: whether charts are read from or written to `.xlsx` files is a property of the file layer and is documented in [../compatibility/formats.md](../compatibility/formats.md) and [../compatibility/excel.md](../compatibility/excel.md), not here. The chart model itself is native JSON and is stored as such in `.orixsheet` files ([persistence.md](persistence.md)).257
Files main
Menu

Theme