Repository Public License
1<!--2ORIX Monitor3File: docs/architecture/alerts.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->78# Alerts and findings910Two independent mechanisms watch the metrics:1112| | Alerts | Findings |13|---|---|---|14| Source | `Analysis/AlertEngine.swift` | `Analysis/IntelligenceEngine.swift` |15| Input | The values of the current tick | Stored history (minute and hour tiers) and `process.spike` events |16| Timing | Every tick, in real time | On demand; `MonitorModel` refreshes every 300 s and when the user asks |17| Output | `AlertTransition` (fired / resolved), persisted as events | `Finding` values, recomputed each time, not persisted |18| User control | Rule editor on the Alerts page | Thresholds shown on the Insights page ("What is checked"); not editable in the UI |1920## Alert rules2122`AlertRule`:2324| Field | Meaning |25|---|---|26| `id` | `UUID` |27| `metric` | A `MetricID` |28| `comparison` | `.above` (value > threshold) or `.below` (value < threshold); strict inequalities |29| `threshold` | In the metric's stored unit (percent 0…100, bytes, bytes per second, level) |30| `duration` | Seconds the condition must hold before the alert fires |31| `cooldown` | Seconds before the same rule may fire again, counted from the moment it fired (default 900) |32| `severity` | `info`, `warning`, `critical` |33| `isEnabled` | Disabled rules are not evaluated |3435### Default rules (`AlertRule.defaults`)3637| Metric | Condition | Hold | Cooldown | Severity | Enabled |38|---|---|---|---|---|---|39| `cpuTotal` | above 90 % | 120 s | 900 s | warning | yes |40| `memPressure` | above 0.5 (warning or higher) | 60 s | 900 s | warning | yes |41| `memPressure` | above 1.5 (critical) | 20 s | 900 s | critical | yes |42| `diskFree` | below 20 GB (20 × 10⁹ bytes) | 60 s | 6 h | warning | yes |43| `batteryCharge` | below 15 % | 30 s | 1800 s | warning | yes |44| `thermalState` | above 1.5 (serious or higher) | 30 s | 900 s | critical | yes |45| `netDown` | above 50 MB/s (50 × 10⁶ bytes/s) | 60 s | 900 s | info | **no** |4647Level metrics use half-step thresholds (0.5, 1.5, 2.5) so that "above 1.5" means "at level 2 or higher"; `AlertText.condition` renders them as "at or above Serious".4849The rule editor (`RuleEditor`) offers these metrics: CPU usage, memory pressure, memory used, swap used, startup volume available, disk read, disk write, network download, network upload, battery charge, thermal state, GPU utilization. Hold is adjustable from 0 to 3600 s in 10 s steps, cooldown from 1 minute to 24 hours.5051## State machine5253`AlertEngine` is a `Sendable` struct with no I/O. It keeps one `State` per rule: `breachStart`, `lastSeen`, `firing`, `lastFired`.5455```mermaid56stateDiagram-v257 [*] --> Idle58 Idle --> Holding: value matches the condition59 Holding --> Idle: value stops matching60 Holding --> Idle: no evaluation for more than maxGap61 Holding --> Firing: held for duration, cooled down, not snoozed62 Holding --> Holding: still matching, waiting63 Firing --> Firing: still matching, no repeat64 Firing --> Idle: value stops matching, emits resolved65 Firing --> Idle: rule disabled, emits resolved66```6768`evaluate(_ values: [MetricID: Double], at date: Date) -> [AlertTransition]` processes each rule in order:69701. **Disabled rule.** If it was firing, emit `resolved` (value = current value or NaN). Reset the state, keeping only `lastFired`.712. **Metric missing this tick.** Skip the rule: the state is left untouched and `lastSeen` is not updated.723. **Gap reset.** If `now − lastSeen > maxGap`, clear `breachStart`. Time during which the engine did not observe the metric — sleep, paused sampling, a long hole — never counts as time spent beyond the threshold. `maxGap` defaults to 30 s and `TelemetryEngine.tick()` sets it to `max(30, 3 × effective interval)` on every tick.734. **Condition matches.**74 - `breachStart` is set if not already; `held = now − breachStart`.75 - The alert fires when all of these hold: not already firing; `held ≥ duration`; `now − lastFired ≥ cooldown` (or it never fired); the engine is not snoozed.76 - On firing: `firing = true`, `lastFired = now`, emit `fired` with the value and `held`.775. **Condition does not match.** If firing, emit `resolved` with `held` = time since `breachStart`. Clear `firing` and `breachStart`.7879After the loop, states of rules that no longer exist are discarded.8081### Behaviours worth knowing8283| Behaviour | Explanation | Test |84|---|---|---|85| One alert per sustained problem | While firing, further matching ticks emit nothing | `firesOnlyAfterTheHoldDuration` |86| A dip resets the hold timer | Any non-matching sample clears `breachStart` | `aDipResetsTheHoldTimer` |87| Exact threshold does not match | Comparisons are strict | `belowComparisonAndExactThreshold` |88| Cooldown counts from the firing time | After resolving, the rule stays quiet until `lastFired + cooldown`. The condition keeps being tracked, so if it has held long enough when the cooldown ends, the alert fires at once | `resolvesThenRespectsCooldown` |89| Snooze suppresses, it does not forget | `snoozedUntil` blocks firing only; hold timers keep running, so an ongoing breach fires on the first tick after the snooze ends | `snoozeSuppressesThenFires` |90| Sleep does not count | An hour asleep leaves `held` at zero on wake | `missingMetricAndSleepGap` |91| Disabling resolves | A firing rule that is switched off emits `resolved` | `disablingAFiringRuleResolvesIt` |92| Snooze does not resolve | An alert already firing when the snooze starts stays in `firing` | by construction (snooze is only checked on the firing transition) |9394**Interaction between `maxGap` and long sampling intervals.** Rules are evaluated once per tick. A fixed 30 s allowance would clear every hold timer whenever the effective interval reaches about 30 s (menu bar only at a long interval, doubled on battery), so that only rules with a zero duration could fire. The engine therefore scales the allowance with the interval in force; the test `holdTimerSurvivesLongSamplingIntervals` covers the case.9596## From transition to user9798```mermaid99flowchart LR100 T["AlertTransition"] --> EV["events table: alert.fired / alert.resolved"]101 T --> SN["TelemetrySnapshot.alerts and firingRules"]102 SN --> MM["MonitorModel.firing"]103 MM --> SB["Sidebar badge on Alerts"]104 MM --> MB["Menu bar dot and panel list"]105 MM --> AP["Alerts page: Active now"]106 SN --> NO["Notification Center, fired only, if authorised"]107 EV --> HI["Alerts page: History"]108 EV --> RE["Incident and system reports"]109```110111- **Persistence.** `TelemetryEngine.tick()` records every transition as a `StoredEvent` of kind `alert`, code `alert.fired` or `alert.resolved`, with payload `rule`, `value`, `threshold`, `comparison`, `held`.112- **Notifications.** Optional. `MonitorModel.notify` posts a `UNNotificationRequest` only for `fired` transitions, only when the user enabled notifications and macOS authorised them. The request identifier is the rule's UUID, so a repeat replaces the previous notification instead of stacking. Critical rules play the default sound.113- **Snooze.** The Alerts page offers 15 minutes, 1 hour and 8 hours (`MonitorModel.snoozedUntil` → `TelemetryEngine.snooze(until:)`). The snooze is held in memory and does not survive a relaunch.114- **Rule state is in memory.** Hold timers, firing flags and cooldown timestamps are not persisted; after a relaunch every rule starts idle.115116## Findings117118`TelemetryEngine.findings(window:thresholds:now:)` gathers the inputs for a 24-hour window (default) and calls `IntelligenceEngine.analyse(_:thresholds:)`:119120| Input | Query |121|---|---|122| CPU, memory pressure, swap, network down/up, thermal state | minute tier over the window, up to 2000 points (one-minute buckets for 24 h) |123| Startup volume available space | hour tier over the last **14 days** |124| Process spikes | `process.spike` events in the window (limit 2000) |125126Each `Finding` carries its evidence: `kind`, `severity`, `metric`, `start`, `end`, `threshold`, `peak`, `mean`, `sampleCount`, optional `subject` (process name), optional `projected` date, `occurrences`.127128### Detectors and thresholds129130`IntelligenceThresholds` defaults: `cpuPercent = 85`, `cpuMinutes = 10`, `pressureMinutes = 5`, `swapGrowthBytes = 2 GiB`, `diskHorizonDays = 30`, `networkFloor = 5 MiB/s`, `processEpisodes = 3`.131132| Finding (`FindingKind`) | Detector | Rule | Severity |133|---|---|---|---|134| `sustainedCPU` | `Detectors.sustained` | Bucket average of `cpu.total` above 85 % for at least 10 minutes | critical if the episode lasts ≥ 30 minutes (3 × the minimum), else warning |135| `memoryPressure` | `Detectors.sustained` | Bucket average of the pressure level above 0.5 for at least 5 minutes | critical if the peak level is 2, else warning |136| `swapGrowth` | first vs last point | `last.avg − first.avg ≥ 2 GiB` over the window, with at least 3 points | warning |137| `diskExhaustion` | `Detectors.exhaustion` | Least-squares line through hourly available space; needs ≥ 6 hours of history and a negative slope; projected exhaustion within 30 days | critical if within 7 days, else warning. **An estimate**; shown with the "Estimate" badge |138| `networkAnomaly` | `Detectors.anomalies` | One-minute buckets above `max(5 MiB/s, median + 8 × max(MAD, 0.1 × median))`; needs at least 30 buckets of baseline. Evaluated separately for download and upload | info |139| `thermalPressure` | `Detectors.sustained` | Bucket average of the thermal level above 1.5 (serious or critical) for at least 60 s | critical if the peak is 3, else warning |140| `processSpikes` | grouping of `process.spike` events by name | The same process name has at least 3 spike episodes in the window | info |141142Detector details:143144- **`Detectors.sustained(_:step:above:minDuration:)`** scans points in order, accumulating a run while `avg > threshold`. A point below the threshold closes the run; so does a hole wider than three bucket widths — *absence of data is never counted as evidence*. A run becomes an `Episode` if `last.time + step − first.time ≥ minDuration`. Its mean is sample-weighted and its peak is the maximum of bucket maxima.145- **`Detectors.linearFit`** needs at least three points with a spread in time.146- **`Detectors.exhaustion`** returns `(date, bytesPerDay)` where `secondsLeft = last.avg / −slope`. Flat or growing free space, or less than `minSpan` (6 h) of history, gives no projection.147- **`Detectors.anomalies`** uses the median and the median absolute deviation (MAD) of the series itself; the `0.1 × median` term prevents a perfectly flat series from flagging tiny variations, and the absolute floor prevents idle-level traffic from ever being called anomalous.148- **`ProcessSpikeTracker`** (fed on every process tick) starts a run when a process shows `cpuPercent ≥ 80` (percent of one core), extends it while the same PID and name stay at or above the threshold, and emits one `Spike` when the process drops below it or disappears, provided the run lasted ≥ 30 s. The engine stores it as a `process.spike` event.149150Findings are sorted by severity (critical first), then by most recent end.151152### Presentation153154`FindingText` (app) turns each finding into a title, a sentence quoting the numbers (threshold, duration, mean, peak, occurrences, projected date) and a suggested action. The Insights page lists the evidence fields (metric, start, duration, samples) under each finding. Findings whose interval overlaps a report's period are included in that report.155156The Insights page states the policy in the UI: "A single unusual reading never produces a finding, and no finding claims a hardware fault."157158## Limitations159160- Findings are not persisted and not notified; they exist only as the result of the latest analysis. Closing the app discards them, and they are recomputed from history next time.161- The analysis window is fixed at 24 hours in the app (the function accepts another window, but nothing passes one).162- Finding thresholds have no editor. They can only differ from the defaults if the `thresholds` value in user defaults is changed by other means.163- `networkAnomaly` reports one finding per direction covering the first to the last outlier, with the count of outlier minutes; it does not group outliers into separate episodes.164- `swapGrowth` compares the first and last point only; swap that rose and fell back within the window is not reported.165- Alert rules compare a single metric with a constant. There are no compound conditions, no rate-of-change rules and no per-process rules.166- Alert state, snooze and cooldown timers reset on relaunch.167- `maxGap` is not configurable from the UI; it follows the sampling interval automatically (three intervals, never less than 30 s).168