SPB Git

Repository Public License

telemetry.md 11.1 KB 177 lines markdown

1<!--2ORIX Monitor3File: docs/architecture/telemetry.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->78# Telemetry engine910Source: `Packages/ORIXMonitorCore/Sources/ORIXMonitorCore/Scheduler/TelemetryEngine.swift`.1112`TelemetryEngine` is an actor that owns one sampling loop. Each iteration ("tick") reads the collectors that are due, normalises the readings into stored metrics, evaluates alerts, buffers the values in the store and publishes a `TelemetrySnapshot`.1314## Configuration1516`TelemetryConfiguration` (Codable, persisted by `AppSettings` under the `telemetry` key):1718| Field | Default | Meaning | Settings control (range) |19|---|---|---|---|20| `interval` | 2 s | Fast tier while a window is visible | "While a window is visible" (1–10 s) |21| `backgroundInterval` | 5 s | Fast tier when only the menu bar is showing | "Menu bar only" (2–30 s) |22| `processInterval` | 4 s | Process-table tier | "Process list" (2–30 s) |23| `slowInterval` | 30 s | Volumes and battery tier | "Volumes and battery" (10–120 s) |24| `batterySaver` | `true` | Doubles the fast interval on battery or in Low Power Mode | "Halve the sampling rate on battery and in Low Power Mode" |25| `bestEffortMetrics` | `true` | Reads undocumented IORegistry properties (GPU utilization, battery detail) | "Read undocumented system properties" |26| `historyEnabled` | `true` | Writes samples to the store | "Record history" |27| `flushInterval` | 30 s | Seconds between store flushes | not exposed |2829`TelemetryConfiguration.intervalRange` is `1...10`.3031## Scheduler tiers3233All tiers are driven by the same loop; there is one timer, not one per collector.3435| Tier | Collectors | Condition evaluated on each tick |36|---|---|---|37| Fast | `CPUCollector.sample`, `MemoryCollector.sample`, `DiskCollector.activity`, `NetworkCollector.sample`, `GPUCollector.statistics` (only if `bestEffortMetrics`), `ThermalLevel.current`, `ProcessCollector.selfUsage` | Every tick |38| Process | `ProcessCollector.sample`, then `ProcessSpikeTracker.update` | `wantsProcesses` and `now − lastProcess ≥ processInterval × (planned / max(interval, 0.5))`, or first tick |39| Slow | `DiskCollector.volumes`, `PowerCollector.sample(includeBestEffort:)` | `now − lastSlow ≥ slowInterval`, or first tick, or after `invalidateSlowReadings()` |40| Flush | `TimeSeriesStore.flush` | `historyEnabled` and `now − lastFlush ≥ flushInterval` |4142Notes:4344- The process tier scales with the fast tier: when the effective interval is stretched (background, battery), the process interval is stretched by the same factor `planned / interval`. With the defaults, processes are sampled every 4 s in the foreground on AC power and every 10 s when only the menu bar is showing.45- The slow tier is not scaled. It can be forced early: `MonitorModel` calls `invalidateSlowReadings()` on power-state changes, thermal-state changes, volume mount/unmount and network-path changes.46- On ticks where the process table is not sampled, `TelemetrySnapshot.processes` is `nil` and consumers keep the previous sample. `process.count` is still stored from the last known count.47- Outside the engine, `MonitorModel.ingest` refreshes Wi-Fi link data and DNS configuration every fifth snapshot, and findings every 300 s.4849## Adaptive interval5051`SchedulerMath.effectiveInterval(_:foreground:onBattery:lowPower:)`:5253```54base   = foreground ? interval : max(interval, backgroundInterval)55scaled = (batterySaver && (onBattery || lowPower)) ? base × 2 : base56result = clamp(scaled, 0.5 … 60)57```5859| Situation (defaults) | Effective interval |60|---|---|61| Window visible, AC power | 2 s |62| Menu bar only, AC power | 5 s |63| Window visible, on battery or Low Power Mode | 4 s |64| Menu bar only, on battery or Low Power Mode | 10 s |65| Battery saver off | 2 s / 5 s regardless of power |6667"Foreground" is decided by `MonitorModel.updateForeground()`: true when at least one titled, normal-level window is visible and not occluded. `onBattery` and `lowPower` come from the most recent `PowerSample` (`source == .battery`, `lowPowerMode`), so a power change takes effect once the slow tier has re-read the power source — which `invalidateSlowReadings()` forces on the next tick.6869The interval in force is carried by every snapshot (`TelemetrySnapshot.interval`) and displayed in the sidebar footer ("Sampling every … s") and in Settings › Sampling.7071## Loop and timing7273```swift74loop = Task { [weak self] in75    while !Task.isCancelled {76        guard let self else { return }77        let wait = await self.tick()78        try? await Task.sleep(for: .seconds(wait), tolerance: .seconds(wait * 0.1))79    }80}81```8283- The sleep starts after the tick finishes, so the real period is the planned interval plus collection time. Rates never assume the planned interval: every collector divides by the monotonic time actually elapsed since its previous reading.84- `MonotonicClock.now()` is `clock_gettime_nsec_np(CLOCK_MONOTONIC_RAW)` in seconds. It includes time asleep, which is what makes sleep detectable.85- `tick(now:date:)` accepts injected times, which the engine tests use to simulate gaps deterministically.8687## Normalisation8889`Normalizer.values(cpu:memory:disk:bootVolume:network:processCount:power:thermal:gpu:selfCPU:selfMemory:)` maps samples to `[MetricValue]`:9091| Input | Stored metrics | Omitted when |92|---|---|---|93| `CPUSample` | `cpuTotal`, `cpuUser`, `cpuSystem` (fractions × 100), `loadAverage1` | No baseline yet (first tick, after a reset) |94| `MemorySample` | `memUsed`, `memApp`, `memWired`, `memCompressed`, `memCached`, `memFree`, `memPressure`, `swapUsed`; `pageIns`, `pageOuts` | Page rates omitted on the first sample |95| `DiskActivitySample` | `diskRead`, `diskWrite` | No rate yet, or counter reset |96| Root `VolumeInfo` | `diskFree` | No volume flagged `isRoot` |97| `NetworkSample` | `netDown`, `netUp` | No rate yet, or no counted interface produced a rate |98| `PowerSample` | `batteryCharge`, `batteryPower`, `batteryTemperature` | No battery; or the individual field is `nil` |99| `ThermalLevel` | `thermalState` | Never (always present) |100| `[GPUStatistics]` | `gpuUtilization` = maximum `deviceUtilization` over devices | Best-effort readings off, or the driver publishes none |101| Process count | `processCount` | Process table never sampled yet |102| Self usage | `selfCPU`, `selfMemory` | `selfCPU` omitted without a baseline |103104The unit test `normalizerOmitsWhatWasNotMeasured` asserts that with every optional input `nil` only `thermalState` and `selfMemory` are produced.105106## Snapshot stream107108`snapshots()` returns an `AsyncStream<TelemetrySnapshot>` created with `.bufferingNewest(1)`. Each subscriber gets its own continuation; a slow consumer sees the latest snapshot and silently misses older ones instead of building a backlog. Termination removes the continuation.109110`TelemetrySnapshot` carries the typed samples, the stored values, alert transitions of this tick, currently firing rules, `collectionTime` (milliseconds the collectors took), `sequence`, and `afterDiscontinuity`.111112## Discontinuities: sleep, wake, pause, gaps113114A discontinuity is any moment after which "the delta since the previous reading" would be meaningless.115116```mermaid117stateDiagram-v2118    [*] --> NoBaseline119    NoBaseline --> Sampling: first tick stores baselines120    Sampling --> Sampling: tick, rates from deltas121    Sampling --> NoBaseline: sleep or wake announced122    Sampling --> NoBaseline: gap detected from the monotonic clock123    Sampling --> Paused: sampling paused124    Paused --> NoBaseline: sampling resumed125```126127Three triggers lead to `resetBaselines()`:128129| Trigger | Detected by | Event recorded |130|---|---|---|131| Sleep announced | `NSWorkspace.willSleepNotification` → `noteDiscontinuity(code: "system.sleep")` | `system.sleep` (kind `system`) |132| Wake announced | `NSWorkspace.didWakeNotification` → `noteDiscontinuity(code: "system.wake")` | `system.wake` |133| Unannounced hole (process suspended, missed notification) | `SchedulerMath.isDiscontinuity(elapsed:planned:)` inside `tick()` | `system.gap` with payload `seconds` |134| Sampling paused by the user | `setPaused(true)` | none |135136`isDiscontinuity` is `elapsed > max(planned × 4, planned + 10)`. With a 2 s interval the limit is 12 s; with 30 s it is 120 s.137138`resetBaselines()`:139140- calls `reset()` on the CPU, memory, disk, network and process collectors (their previous counters and timestamps are dropped);141- clears the self-usage baseline and `lastTick`;142- sets `pendingDiscontinuity`.143144Consequences on the next tick:145146- Rate-type metrics are **absent** (no baseline), not zero and not a giant spike. Level-type metrics (memory amounts, pressure, thermal state, battery charge, available space) are present because they need no baseline.147- The snapshot has `afterDiscontinuity == true`. `MonitorModel.ingest` then trims every live series to its last point and clears the per-core history, so live charts do not join samples across the sleep with a straight line.148- While paused, `tick()` returns immediately without sampling; nothing is stored, and the store later reports the period as a gap.149150Alert hold timers are protected separately: `AlertEngine.maxGap` (three sampling intervals, at least 30 s) resets a breach that was not observed continuously. See [alerts](alerts.md). Stored series report the hole as a `SeriesGap`. See [storage](storage.md).151152The engine test `sleepWakeDropsBaselinesAndLogsTheEvent` covers both the announced and the unannounced path.153154## Events written by the engine155156| Code | Kind | When | Payload |157|---|---|---|---|158| `system.sleep`, `system.wake` | `system` | `noteDiscontinuity(code:)` | — |159| `system.gap` | `system` | Hole detected in `tick()` | `seconds` |160| `alert.fired`, `alert.resolved` | `alert` | Each `AlertTransition` | `rule`, `value`, `threshold`, `comparison`, `held` |161| `process.spike` | `process` | A `ProcessSpikeTracker` episode ended | `name`, `pid`, `peak`, `duration` |162| `bookmark` | `bookmark` | `MonitorModel.addBookmark(note:)` | `note` |163| `network.online`, `network.offline` | `network` | `NWPathMonitor` path change (in `MonitorModel`) | `interfaces` |164165Events are recorded even when `historyEnabled` is off; that switch gates only `store.append` and the periodic flush.166167## Self-measurement168169Every tick calls `ProcessCollector.selfUsage()` (one `proc_pid_rusage` on the app's own PID) and stores `self.cpu` (percent of one core over the last interval) and `self.memory` (physical footprint). `collectionTime` measures the wall time of the collector calls. These feed the "ORIX Monitor overhead" widget, Settings › About, and `Scripts/bench.sh`. See [performance](../development/performance.md).170171## Limitations172173- One loop means one cadence for the fast tier: CPU, memory, disk, network, GPU and thermal state cannot be sampled at different rates from each other.174- The interval cannot go below 0.5 s (`effectiveInterval` clamp); the settings UI offers 1 s as the minimum.175- A tick whose collection takes long delays the next one; there is no catch-up. The elapsed-time arithmetic keeps rates correct, but sample spacing becomes irregular.176- Sleep shorter than the discontinuity limit that is also not announced by `NSWorkspace` would be averaged into one long interval. No such case has been observed; the notifications are the primary mechanism.177
Files main
Menu

Theme