SPB Git

Repository Public License

alerts.md 12.5 KB

# Alerts and findings

Two independent mechanisms watch the metrics:

Alerts Findings
Source Analysis/AlertEngine.swift Analysis/IntelligenceEngine.swift
Input The values of the current tick Stored history (minute and hour tiers) and process.spike events
Timing Every tick, in real time On demand; MonitorModel refreshes every 300 s and when the user asks
Output AlertTransition (fired / resolved), persisted as events Finding values, recomputed each time, not persisted
User control Rule editor on the Alerts page Thresholds shown on the Insights page ("What is checked"); not editable in the UI

# Alert rules

AlertRule:

Field Meaning
id UUID
metric A MetricID
comparison .above (value > threshold) or .below (value < threshold); strict inequalities
threshold In the metric's stored unit (percent 0…100, bytes, bytes per second, level)
duration Seconds the condition must hold before the alert fires
cooldown Seconds before the same rule may fire again, counted from the moment it fired (default 900)
severity info, warning, critical
isEnabled Disabled rules are not evaluated

# Default rules (AlertRule.defaults)

Metric Condition Hold Cooldown Severity Enabled
cpuTotal above 90 % 120 s 900 s warning yes
memPressure above 0.5 (warning or higher) 60 s 900 s warning yes
memPressure above 1.5 (critical) 20 s 900 s critical yes
diskFree below 20 GB (20 × 10⁹ bytes) 60 s 6 h warning yes
batteryCharge below 15 % 30 s 1800 s warning yes
thermalState above 1.5 (serious or higher) 30 s 900 s critical yes
netDown above 50 MB/s (50 × 10⁶ bytes/s) 60 s 900 s info no

Level metrics use half-step thresholds (0.5, 1.5, 2.5) so that "above 1.5" means "at level 2 or higher"; AlertText.condition renders them as "at or above Serious".

The rule editor (RuleEditor) offers these metrics: CPU usage, memory pressure, memory used, swap used, startup volume available, disk read, disk write, network download, network upload, battery charge, thermal state, GPU utilization. Hold is adjustable from 0 to 3600 s in 10 s steps, cooldown from 1 minute to 24 hours.

# State machine

AlertEngine is a Sendable struct with no I/O. It keeps one State per rule: breachStart, lastSeen, firing, lastFired.

stateDiagram-v2
    [*] --> Idle
    Idle --> Holding: value matches the condition
    Holding --> Idle: value stops matching
    Holding --> Idle: no evaluation for more than maxGap
    Holding --> Firing: held for duration, cooled down, not snoozed
    Holding --> Holding: still matching, waiting
    Firing --> Firing: still matching, no repeat
    Firing --> Idle: value stops matching, emits resolved
    Firing --> Idle: rule disabled, emits resolved

evaluate(_ values: [MetricID: Double], at date: Date) -> [AlertTransition] processes each rule in order:

  1. Disabled rule. If it was firing, emit resolved (value = current value or NaN). Reset the state, keeping only lastFired.
  2. Metric missing this tick. Skip the rule: the state is left untouched and lastSeen is not updated.
  3. Gap reset. If now − lastSeen > maxGap, clear breachStart. Time during which the engine did not observe the metric — sleep, paused sampling, a long hole — never counts as time spent beyond the threshold. maxGap defaults to 30 s and TelemetryEngine.tick() sets it to max(30, 3 × effective interval) on every tick.
  4. Condition matches.
    • breachStart is set if not already; held = now − breachStart.
    • The alert fires when all of these hold: not already firing; held ≥ duration; now − lastFired ≥ cooldown (or it never fired); the engine is not snoozed.
    • On firing: firing = true, lastFired = now, emit fired with the value and held.
  5. Condition does not match. If firing, emit resolved with held = time since breachStart. Clear firing and breachStart.

After the loop, states of rules that no longer exist are discarded.

# Behaviours worth knowing

Behaviour Explanation Test
One alert per sustained problem While firing, further matching ticks emit nothing firesOnlyAfterTheHoldDuration
A dip resets the hold timer Any non-matching sample clears breachStart aDipResetsTheHoldTimer
Exact threshold does not match Comparisons are strict belowComparisonAndExactThreshold
Cooldown counts from the firing time After resolving, the rule stays quiet until lastFired + cooldown. The condition keeps being tracked, so if it has held long enough when the cooldown ends, the alert fires at once resolvesThenRespectsCooldown
Snooze suppresses, it does not forget snoozedUntil blocks firing only; hold timers keep running, so an ongoing breach fires on the first tick after the snooze ends snoozeSuppressesThenFires
Sleep does not count An hour asleep leaves held at zero on wake missingMetricAndSleepGap
Disabling resolves A firing rule that is switched off emits resolved disablingAFiringRuleResolvesIt
Snooze does not resolve An alert already firing when the snooze starts stays in firing by construction (snooze is only checked on the firing transition)

Interaction between maxGap and long sampling intervals. Rules are evaluated once per tick. A fixed 30 s allowance would clear every hold timer whenever the effective interval reaches about 30 s (menu bar only at a long interval, doubled on battery), so that only rules with a zero duration could fire. The engine therefore scales the allowance with the interval in force; the test holdTimerSurvivesLongSamplingIntervals covers the case.

# From transition to user

flowchart LR
    T["AlertTransition"] --> EV["events table: alert.fired / alert.resolved"]
    T --> SN["TelemetrySnapshot.alerts and firingRules"]
    SN --> MM["MonitorModel.firing"]
    MM --> SB["Sidebar badge on Alerts"]
    MM --> MB["Menu bar dot and panel list"]
    MM --> AP["Alerts page: Active now"]
    SN --> NO["Notification Center, fired only, if authorised"]
    EV --> HI["Alerts page: History"]
    EV --> RE["Incident and system reports"]
  • Persistence. TelemetryEngine.tick() records every transition as a StoredEvent of kind alert, code alert.fired or alert.resolved, with payload rule, value, threshold, comparison, held.
  • Notifications. Optional. MonitorModel.notify posts a UNNotificationRequest only for fired transitions, only when the user enabled notifications and macOS authorised them. The request identifier is the rule's UUID, so a repeat replaces the previous notification instead of stacking. Critical rules play the default sound.
  • Snooze. The Alerts page offers 15 minutes, 1 hour and 8 hours (MonitorModel.snoozedUntil → TelemetryEngine.snooze(until:)). The snooze is held in memory and does not survive a relaunch.
  • Rule state is in memory. Hold timers, firing flags and cooldown timestamps are not persisted; after a relaunch every rule starts idle.

# Findings

TelemetryEngine.findings(window:thresholds:now:) gathers the inputs for a 24-hour window (default) and calls IntelligenceEngine.analyse(_:thresholds:):

Input Query
CPU, memory pressure, swap, network down/up, thermal state minute tier over the window, up to 2000 points (one-minute buckets for 24 h)
Startup volume available space hour tier over the last 14 days
Process spikes process.spike events in the window (limit 2000)

Each Finding carries its evidence: kind, severity, metric, start, end, threshold, peak, mean, sampleCount, optional subject (process name), optional projected date, occurrences.

# Detectors and thresholds

IntelligenceThresholds defaults: cpuPercent = 85, cpuMinutes = 10, pressureMinutes = 5, swapGrowthBytes = 2 GiB, diskHorizonDays = 30, networkFloor = 5 MiB/s, processEpisodes = 3.

Finding (FindingKind) Detector Rule Severity
sustainedCPU Detectors.sustained Bucket average of cpu.total above 85 % for at least 10 minutes critical if the episode lasts ≥ 30 minutes (3 × the minimum), else warning
memoryPressure Detectors.sustained Bucket average of the pressure level above 0.5 for at least 5 minutes critical if the peak level is 2, else warning
swapGrowth first vs last point last.avg − first.avg ≥ 2 GiB over the window, with at least 3 points warning
diskExhaustion Detectors.exhaustion Least-squares line through hourly available space; needs ≥ 6 hours of history and a negative slope; projected exhaustion within 30 days critical if within 7 days, else warning. An estimate; shown with the "Estimate" badge
networkAnomaly Detectors.anomalies One-minute buckets above max(5 MiB/s, median + 8 × max(MAD, 0.1 × median)); needs at least 30 buckets of baseline. Evaluated separately for download and upload info
thermalPressure Detectors.sustained Bucket average of the thermal level above 1.5 (serious or critical) for at least 60 s critical if the peak is 3, else warning
processSpikes grouping of process.spike events by name The same process name has at least 3 spike episodes in the window info

Detector details:

  • Detectors.sustained(_:step:above:minDuration:) scans points in order, accumulating a run while avg > threshold. A point below the threshold closes the run; so does a hole wider than three bucket widths — absence of data is never counted as evidence. A run becomes an Episode if last.time + step − first.time ≥ minDuration. Its mean is sample-weighted and its peak is the maximum of bucket maxima.
  • Detectors.linearFit needs at least three points with a spread in time.
  • Detectors.exhaustion returns (date, bytesPerDay) where secondsLeft = last.avg / −slope. Flat or growing free space, or less than minSpan (6 h) of history, gives no projection.
  • Detectors.anomalies uses the median and the median absolute deviation (MAD) of the series itself; the 0.1 × median term prevents a perfectly flat series from flagging tiny variations, and the absolute floor prevents idle-level traffic from ever being called anomalous.
  • ProcessSpikeTracker (fed on every process tick) starts a run when a process shows cpuPercent ≥ 80 (percent of one core), extends it while the same PID and name stay at or above the threshold, and emits one Spike when the process drops below it or disappears, provided the run lasted ≥ 30 s. The engine stores it as a process.spike event.

Findings are sorted by severity (critical first), then by most recent end.

# Presentation

FindingText (app) turns each finding into a title, a sentence quoting the numbers (threshold, duration, mean, peak, occurrences, projected date) and a suggested action. The Insights page lists the evidence fields (metric, start, duration, samples) under each finding. Findings whose interval overlaps a report's period are included in that report.

The Insights page states the policy in the UI: "A single unusual reading never produces a finding, and no finding claims a hardware fault."

# Limitations

  • Findings are not persisted and not notified; they exist only as the result of the latest analysis. Closing the app discards them, and they are recomputed from history next time.
  • The analysis window is fixed at 24 hours in the app (the function accepts another window, but nothing passes one).
  • Finding thresholds have no editor. They can only differ from the defaults if the thresholds value in user defaults is changed by other means.
  • networkAnomaly reports one finding per direction covering the first to the last outlier, with the count of outlier minutes; it does not group outliers into separate episodes.
  • swapGrowth compares the first and last point only; swap that rose and fell back within the window is not reported.
  • Alert rules compare a single metric with a constant. There are no compound conditions, no rate-of-change rules and no per-process rules.
  • Alert state, snooze and cooldown timers reset on relaunch.
  • maxGap is not configurable from the UI; it follows the sampling interval automatically (three intervals, never less than 30 s).
Files main
Menu

Theme