SPB Git

Repository Public License

storage.md 11.4 KB

# Storage

Source: Packages/ORIXMonitorCore/Sources/ORIXMonitorCore/Storage/TimeSeriesStore.swift.

TimeSeriesStore is an actor around one SQLite database. It keeps numeric history in three retention tiers, answers downsampled range queries, reports gaps, and holds an event log (alerts, bookmarks, system and network events, process spikes).

# Location

Case Path
Normal run ~/Library/Application Support/ORIX Monitor/telemetry.sqlite (resolved with FileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask))
-ORIXDataDir <dir> <dir>/telemetry.sqlite
Tests, benchmarks TimeSeriesStore(url: nil) opens a private :memory: database

The directory is created if missing. With WAL mode SQLite also creates telemetry.sqlite-wal and telemetry.sqlite-shm. .gitignore excludes all three patterns.

If the database cannot be opened, MonitorModel keeps running without a store (storeError is shown in Settings › History).

# Connection settings

Applied at open (sqlite3_open_v2 with SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX):

sql
PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
PRAGMA auto_vacuum=INCREMENTAL;

# Schema

sql
CREATE TABLE IF NOT EXISTS raw(
  metric INTEGER NOT NULL, ts INTEGER NOT NULL, v REAL NOT NULL,
  PRIMARY KEY(metric, ts)) WITHOUT ROWID;

CREATE TABLE IF NOT EXISTS minute(
  metric INTEGER NOT NULL, ts INTEGER NOT NULL,
  mn REAL, mx REAL, av REAL, n INTEGER,
  PRIMARY KEY(metric, ts)) WITHOUT ROWID;

CREATE TABLE IF NOT EXISTS hour(
  metric INTEGER NOT NULL, ts INTEGER NOT NULL,
  mn REAL, mx REAL, av REAL, n INTEGER,
  PRIMARY KEY(metric, ts)) WITHOUT ROWID;

CREATE TABLE IF NOT EXISTS events(
  id INTEGER PRIMARY KEY AUTOINCREMENT, ts REAL NOT NULL, kind TEXT NOT NULL,
  severity INTEGER NOT NULL, metric INTEGER, code TEXT NOT NULL, payload TEXT NOT NULL);

CREATE INDEX IF NOT EXISTS events_ts ON events(ts);
Column Meaning
metric MetricID.rawValue (stable integer key: 1 = cpu.total, 10 = memory.used, …, 90 = self.cpu, 91 = self.memory)
raw.ts Unix time in milliseconds (rounded)
minute.ts, hour.ts Unix time in seconds of the start of the bucket (floored to 60 s / 3600 s)
v The value in the metric's stored unit
mn, mx, av, n Minimum, maximum, arithmetic mean and number of raw samples in the bucket
events.ts Unix time in seconds (floating point)
events.kind EventKind: alert, finding, bookmark, system, network, process
events.severity Severity: 0 info, 1 warning, 2 critical
events.payload JSON object of string → string

There is no schema-version table and no migration code: the schema is the first and only version.

# Retention tiers

RetentionPolicy (seconds each tier is kept):

Tier (StoreTier) Content Default retention Settings choices
raw Every sample as produced by the scheduler 6 hours 1 h, 6 h, 24 h, 3 days
minute One row per metric per minute: min, avg, max, count 14 days 7, 14, 30, 90 days
hour One row per metric per hour: min, avg, max, count 730 days (2 years) 90 days, 1 year, 2 years, 5 years

Events are pruned with the hour retention.

# Write path

flowchart TD
    A["append(values, date)"] --> F{"value is finite?"}
    F -- "no" --> X["dropped"]
    F -- "yes" --> P["pendingRaw buffer"]
    F -- "yes" --> M["roll minute bucket"]
    F -- "yes" --> H["roll hour bucket"]
    M --> C["closedBuckets when the bucket start changes"]
    H --> C
    P --> FL["flush()"]
    C --> FL
    M --> FL
    H --> FL
    FL --> T["one BEGIN IMMEDIATE transaction"]
    T --> R["INSERT OR REPLACE raw"]
    T --> AG["INSERT OR REPLACE minute and hour"]
    T --> PR{"more than 1 h since last prune?"}
    PR -- "yes" --> PN["prune() and incremental_vacuum"]
  • append(_:at:) touches no disk. Each finite value is added to pendingRaw and to the running Bucket (start, min, max, sum, count) of its metric in both the minute and the hour tier. NaN and infinite values are dropped (nonFiniteValuesAreDropped).
  • roll computes the bucket start as floor(time / step) × step. If the metric's current bucket has a different start, it is moved to closedBuckets and a new one begins.
  • Relaunch inside a bucket. When there is no in-memory bucket for a metric (first value after launch), loadBucket reads the row already stored for that bucket start and continues it, so a relaunch within the same minute or hour merges with the earlier partial aggregate instead of overwriting it (historySurvivesRelaunchAndPartialBucketsMerge).
  • flush(now:) writes, in one transaction, all pending raw rows, all closed buckets, and the current open buckets (INSERT OR REPLACE, so an open bucket is rewritten with its latest aggregate at each flush). The stored average is sum / count.
  • The engine calls flush every flushInterval (30 s) and on stop(). series(...) and statistics() also flush first, so a query never misses buffered samples.

# Durability

At most one flush period of raw samples (30 s by default) is held in memory. A normal quit flushes through AppDelegate.applicationShouldTerminate → MonitorModel.shutdown() → TelemetryEngine.stop(). A crash or kill loses the unflushed samples; Scripts/bench.sh notes this explicitly.

# Pruning

prune(now:) runs at most once an hour (checked at the end of flush), and immediately after setRetention(_:) resets the timer:

sql
DELETE FROM raw    WHERE ts < (now − retention.raw) × 1000;
DELETE FROM minute WHERE ts <  now − retention.minute;
DELETE FROM hour   WHERE ts <  now − retention.hour;
DELETE FROM events WHERE ts <  now − retention.hour;
PRAGMA incremental_vacuum;

eraseAll() clears the in-memory buffers, deletes every row of the four tables and runs VACUUM. It is reached from Settings › History › "Erase All History…" behind a confirmation dialog.

# Read path and downsampling

series(_:from:to:maxPoints:tier:now:) returns a Series { metric, tier, step, points, gaps } where each SeriesPoint is { time, min, max, avg, count } and time is the start of the bucket.

# Tier selection

SeriesMath.tier(from:to:now:retention:), unless the caller forces a tier:

Condition Tier
span ≤ 2 hours and from is within raw retention raw
span ≤ 7 days and from is within minute retention minute
otherwise hour

So a one-hour range from two days ago is answered from the minute tier (raw data that old no longer exists).

# Bucket width

SeriesMath.step(span:tier:maxPoints:rawInterval:):

text
native = (tier == raw) ? max(rawInterval, 1) : tier.step        // 60 or 3600
wanted = span / maxPoints
step   = wanted > native ? ceil(wanted / native) × native : native

The step is never finer than the tier and always a whole multiple of the tier's native resolution. rawInterval is kept in sync with the configured sampling interval by TelemetryEngine.update(_:). Raw buckets are never narrower than one second.

Examples (from stepNeverFinerThanTier): 1 h on the minute tier at 600 points → 60 s; 24 h on the minute tier at 600 points → 180 s; 2 h raw at a 2 s interval and 600 points → 12 s.

# Aggregation

Raw tier:

sql
SELECT (ts / :stepMs) * :stepMs, MIN(v), MAX(v), AVG(v), COUNT(*)
FROM raw WHERE metric=? AND ts>=? AND ts<? GROUP BY 1 ORDER BY 1;

Minute and hour tiers:

sql
SELECT (ts / :step) * :step, MIN(mn), MAX(mx), SUM(av * n) / SUM(n), SUM(n)
FROM <tier> WHERE metric=? AND ts>=? AND ts<? GROUP BY 1 ORDER BY 1;

Properties that follow, and that downsamplingPreservesExtremesAndWeightedMean checks:

  • Extremes survive. The min of mins and the max of maxes are exact, so a one-sample spike is still visible at any zoom level.
  • Means are sample-weighted. A bucket with 30 samples weighs 30 times a bucket with one. Series.average applies the same weighting across returned points.
  • Sample counts are preserved, which lets reports state how many measurements a figure rests on.

Callers and their point budgets: History page 480, reports 360, findings 2000, default 600.

# Gap detection

SeriesMath.gaps(points:from:to:step:tolerance:) with tolerance = 2.5:

  • No point at all → one gap covering the whole range.
  • first.time − from > 2.5 × step → a leading gap [from, first.time].
  • For consecutive points with b.time − a.time > 2.5 × step → a gap [a.time + step, b.time].
  • to − (last.time + step) > 2.5 × step → a trailing gap. The store passes min(to, now) so the future part of a range is not reported as missing.

Gaps are reported, never filled. Their consumers:

Consumer Behaviour
HistoryChart Shades each gap with a RectangleMark; splits the line and the min–max band into segments at holes wider than 2.5 steps
History page statistics "Gaps: n · total duration"
PDFReportRenderer.chart Shades gaps and breaks the line with the same 2.5-step rule
Markdown / PDF reports "Data gaps" section: count, total missing, longest, per metric
JSON / CSV JSON includes gaps per series; CSV simply has no rows for missing buckets
Detectors.sustained Ends an episode when consecutive points are more than 3 steps apart

A gap means "no sample was stored": the app was not running, the Mac was asleep, sampling was paused, history recording was off, or the metric could not be measured (for example a rate just after a discontinuity).

# Events

Method Purpose
record(_:) Inserts a StoredEvent; returns the row id
events(kinds:from:to:limit:) Newest first, filtered by kind and time, default limit 500
deleteEvent(id:) Removes one event (bookmark deletion)
deleteEvents(kinds:) Removes all events of the given kinds (alert history "Clear History")

Event codes written by the app are listed in telemetry. The finding kind exists in EventKind but nothing writes events of that kind: findings are recomputed from history on demand and are not persisted.

# Statistics

statistics() returns StoreStatistics { rawRows, minuteRows, hourRows, events, fileBytes, earliest }. fileBytes is the size of the database file plus its -wal file; earliest is the oldest hour bucket. Shown in Settings › History and used for the History page subtitle ("Collected locally since …").

# Limitations

  • No encryption. The database is a plain SQLite file protected by the user account's file permissions. It contains numeric series and event payloads (process names in process.spike, interface names in network events, bookmark notes); it contains no IP addresses and no per-process history.
  • SQL errors are not surfaced. exec, run and prepare ignore failures after open; a failed write is silent.
  • Timestamps are wall-clock. A manual clock change moves where subsequent rows land. Rates are unaffected (they use the monotonic clock), but a backwards clock jump can overwrite rows with identical (metric, ts) keys.
  • No migration path exists yet for future schema changes.
  • Per-process data is not stored, only the process count and process.spike events.
  • Minute and hour buckets are aligned to Unix time, not to the local time zone; hour buckets therefore start on UTC hour boundaries.
Files main
Menu

Theme