SPB Git

Repository Public License

telemetry.md 11.1 KB

# Telemetry engine

Source: Packages/ORIXMonitorCore/Sources/ORIXMonitorCore/Scheduler/TelemetryEngine.swift.

TelemetryEngine is an actor that owns one sampling loop. Each iteration ("tick") reads the collectors that are due, normalises the readings into stored metrics, evaluates alerts, buffers the values in the store and publishes a TelemetrySnapshot.

# Configuration

TelemetryConfiguration (Codable, persisted by AppSettings under the telemetry key):

Field Default Meaning Settings control (range)
interval 2 s Fast tier while a window is visible "While a window is visible" (1–10 s)
backgroundInterval 5 s Fast tier when only the menu bar is showing "Menu bar only" (2–30 s)
processInterval 4 s Process-table tier "Process list" (2–30 s)
slowInterval 30 s Volumes and battery tier "Volumes and battery" (10–120 s)
batterySaver true Doubles the fast interval on battery or in Low Power Mode "Halve the sampling rate on battery and in Low Power Mode"
bestEffortMetrics true Reads undocumented IORegistry properties (GPU utilization, battery detail) "Read undocumented system properties"
historyEnabled true Writes samples to the store "Record history"
flushInterval 30 s Seconds between store flushes not exposed

TelemetryConfiguration.intervalRange is 1...10.

# Scheduler tiers

All tiers are driven by the same loop; there is one timer, not one per collector.

Tier Collectors Condition evaluated on each tick
Fast CPUCollector.sample, MemoryCollector.sample, DiskCollector.activity, NetworkCollector.sample, GPUCollector.statistics (only if bestEffortMetrics), ThermalLevel.current, ProcessCollector.selfUsage Every tick
Process ProcessCollector.sample, then ProcessSpikeTracker.update wantsProcesses and now − lastProcess ≥ processInterval × (planned / max(interval, 0.5)), or first tick
Slow DiskCollector.volumes, PowerCollector.sample(includeBestEffort:) now − lastSlow ≥ slowInterval, or first tick, or after invalidateSlowReadings()
Flush TimeSeriesStore.flush historyEnabled and now − lastFlush ≥ flushInterval

Notes:

  • The process tier scales with the fast tier: when the effective interval is stretched (background, battery), the process interval is stretched by the same factor planned / interval. With the defaults, processes are sampled every 4 s in the foreground on AC power and every 10 s when only the menu bar is showing.
  • The slow tier is not scaled. It can be forced early: MonitorModel calls invalidateSlowReadings() on power-state changes, thermal-state changes, volume mount/unmount and network-path changes.
  • On ticks where the process table is not sampled, TelemetrySnapshot.processes is nil and consumers keep the previous sample. process.count is still stored from the last known count.
  • Outside the engine, MonitorModel.ingest refreshes Wi-Fi link data and DNS configuration every fifth snapshot, and findings every 300 s.

# Adaptive interval

SchedulerMath.effectiveInterval(_:foreground:onBattery:lowPower:):

text
base   = foreground ? interval : max(interval, backgroundInterval)
scaled = (batterySaver && (onBattery || lowPower)) ? base × 2 : base
result = clamp(scaled, 0.5 … 60)
Situation (defaults) Effective interval
Window visible, AC power 2 s
Menu bar only, AC power 5 s
Window visible, on battery or Low Power Mode 4 s
Menu bar only, on battery or Low Power Mode 10 s
Battery saver off 2 s / 5 s regardless of power

"Foreground" is decided by MonitorModel.updateForeground(): true when at least one titled, normal-level window is visible and not occluded. onBattery and lowPower come from the most recent PowerSample (source == .battery, lowPowerMode), so a power change takes effect once the slow tier has re-read the power source — which invalidateSlowReadings() forces on the next tick.

The interval in force is carried by every snapshot (TelemetrySnapshot.interval) and displayed in the sidebar footer ("Sampling every … s") and in Settings › Sampling.

# Loop and timing

swift
loop = Task { [weak self] in
    while !Task.isCancelled {
        guard let self else { return }
        let wait = await self.tick()
        try? await Task.sleep(for: .seconds(wait), tolerance: .seconds(wait * 0.1))
    }
}
  • The sleep starts after the tick finishes, so the real period is the planned interval plus collection time. Rates never assume the planned interval: every collector divides by the monotonic time actually elapsed since its previous reading.
  • MonotonicClock.now() is clock_gettime_nsec_np(CLOCK_MONOTONIC_RAW) in seconds. It includes time asleep, which is what makes sleep detectable.
  • tick(now:date:) accepts injected times, which the engine tests use to simulate gaps deterministically.

# Normalisation

Normalizer.values(cpu:memory:disk:bootVolume:network:processCount:power:thermal:gpu:selfCPU:selfMemory:) maps samples to [MetricValue]:

Input Stored metrics Omitted when
CPUSample cpuTotal, cpuUser, cpuSystem (fractions × 100), loadAverage1 No baseline yet (first tick, after a reset)
MemorySample memUsed, memApp, memWired, memCompressed, memCached, memFree, memPressure, swapUsed; pageIns, pageOuts Page rates omitted on the first sample
DiskActivitySample diskRead, diskWrite No rate yet, or counter reset
Root VolumeInfo diskFree No volume flagged isRoot
NetworkSample netDown, netUp No rate yet, or no counted interface produced a rate
PowerSample batteryCharge, batteryPower, batteryTemperature No battery; or the individual field is nil
ThermalLevel thermalState Never (always present)
[GPUStatistics] gpuUtilization = maximum deviceUtilization over devices Best-effort readings off, or the driver publishes none
Process count processCount Process table never sampled yet
Self usage selfCPU, selfMemory selfCPU omitted without a baseline

The unit test normalizerOmitsWhatWasNotMeasured asserts that with every optional input nil only thermalState and selfMemory are produced.

# Snapshot stream

snapshots() returns an AsyncStream<TelemetrySnapshot> created with .bufferingNewest(1). Each subscriber gets its own continuation; a slow consumer sees the latest snapshot and silently misses older ones instead of building a backlog. Termination removes the continuation.

TelemetrySnapshot carries the typed samples, the stored values, alert transitions of this tick, currently firing rules, collectionTime (milliseconds the collectors took), sequence, and afterDiscontinuity.

# Discontinuities: sleep, wake, pause, gaps

A discontinuity is any moment after which "the delta since the previous reading" would be meaningless.

stateDiagram-v2
    [*] --> NoBaseline
    NoBaseline --> Sampling: first tick stores baselines
    Sampling --> Sampling: tick, rates from deltas
    Sampling --> NoBaseline: sleep or wake announced
    Sampling --> NoBaseline: gap detected from the monotonic clock
    Sampling --> Paused: sampling paused
    Paused --> NoBaseline: sampling resumed

Three triggers lead to resetBaselines():

Trigger Detected by Event recorded
Sleep announced NSWorkspace.willSleepNotification → noteDiscontinuity(code: "system.sleep") system.sleep (kind system)
Wake announced NSWorkspace.didWakeNotification → noteDiscontinuity(code: "system.wake") system.wake
Unannounced hole (process suspended, missed notification) SchedulerMath.isDiscontinuity(elapsed:planned:) inside tick() system.gap with payload seconds
Sampling paused by the user setPaused(true) none

isDiscontinuity is elapsed > max(planned × 4, planned + 10). With a 2 s interval the limit is 12 s; with 30 s it is 120 s.

resetBaselines():

  • calls reset() on the CPU, memory, disk, network and process collectors (their previous counters and timestamps are dropped);
  • clears the self-usage baseline and lastTick;
  • sets pendingDiscontinuity.

Consequences on the next tick:

  • Rate-type metrics are absent (no baseline), not zero and not a giant spike. Level-type metrics (memory amounts, pressure, thermal state, battery charge, available space) are present because they need no baseline.
  • The snapshot has afterDiscontinuity == true. MonitorModel.ingest then trims every live series to its last point and clears the per-core history, so live charts do not join samples across the sleep with a straight line.
  • While paused, tick() returns immediately without sampling; nothing is stored, and the store later reports the period as a gap.

Alert hold timers are protected separately: AlertEngine.maxGap (three sampling intervals, at least 30 s) resets a breach that was not observed continuously. See alerts. Stored series report the hole as a SeriesGap. See storage.

The engine test sleepWakeDropsBaselinesAndLogsTheEvent covers both the announced and the unannounced path.

# Events written by the engine

Code Kind When Payload
system.sleep, system.wake system noteDiscontinuity(code:) —
system.gap system Hole detected in tick() seconds
alert.fired, alert.resolved alert Each AlertTransition rule, value, threshold, comparison, held
process.spike process A ProcessSpikeTracker episode ended name, pid, peak, duration
bookmark bookmark MonitorModel.addBookmark(note:) note
network.online, network.offline network NWPathMonitor path change (in MonitorModel) interfaces

Events are recorded even when historyEnabled is off; that switch gates only store.append and the periodic flush.

# Self-measurement

Every tick calls ProcessCollector.selfUsage() (one proc_pid_rusage on the app's own PID) and stores self.cpu (percent of one core over the last interval) and self.memory (physical footprint). collectionTime measures the wall time of the collector calls. These feed the "ORIX Monitor overhead" widget, Settings › About, and Scripts/bench.sh. See performance.

# Limitations

  • One loop means one cadence for the fast tier: CPU, memory, disk, network, GPU and thermal state cannot be sampled at different rates from each other.
  • The interval cannot go below 0.5 s (effectiveInterval clamp); the settings UI offers 1 s as the minimum.
  • A tick whose collection takes long delays the next one; there is no catch-up. The elapsed-time arithmetic keeps rates correct, but sample spacing becomes irregular.
  • Sleep shorter than the discontinuity limit that is also not announced by NSWorkspace would be averaged into one long interval. No such case has been observed; the notifications are the primary mechanism.
Files main
Menu

Theme