SPB Git

Repository Public License

testing.md 20.0 KB 195 lines markdown

1<!--2ORIX Monitor3File: docs/development/testing.md4Author: Simon-Pierre Boucher5Contact: contact@spboucher.ai6-->78# Testing910ORIX Monitor has two test layers:1112| Layer | Location | Framework | Runs with |13|---|---|---|---|14| Core tests | `Packages/ORIXMonitorCore/Tests/ORIXMonitorCoreTests/` | Swift Testing (`@Suite`, `@Test`, `#expect`) | `Scripts/build.sh test` (`swift test`) |15| Interface tests | `UITests/ORIXMonitorUITests.swift` | XCTest UI testing | `Scripts/build.sh uitest` (`xcodebuild test`) |1617`Scripts/build.sh all` runs the core tests, a Debug build and the interface tests in that order. `Scripts/release.sh` runs the core tests before building a release.1819## Results2021**Results recorded on 2026-11-21** on the development Mac (MacBook Pro `Mac17,6`, Apple M5 Max, macOS 27.0, Xcode 27.2), on the source that produced the notarized 1.0.0 (build 1).2223| Suite | Command | Result |24|---|---|---|25| Core | `Scripts/build.sh test` (`swift test`) | **81 tests in 16 suites passed**, about 2.5 s |26| Interface | `Scripts/build.sh uitest` | **13 executed: 12 passed, 1 skipped, 0 failed** |2728- The skipped test is `testMenuBarExtraOpensPanel`: it verified that the status item exists and carries a CPU reading, then skipped the click because macOS hides the item behind the display notch on this Mac. The panel was therefore not opened by an automated click.29- `testAccessibilityAuditOfEveryPage` passes by design (it records the audit's notes instead of failing); its output is discussed in [`../reports/accessibility.md`](../reports/accessibility.md). It takes about 12 minutes, most of the suite's run time.30- The interface tests need the screen to themselves. Runs made while another automation was driving the same Mac produced "not hittable" failures that disappeared when the suite ran alone; those runs are not counted above.31- Earlier runs of the same suite found two real defects, both fixed before release: the app aborted on the Processes page in a narrow window (now covered by `testProcessesPageSurvivesANarrowWindow`), and processes of other users were missing from the table (now covered by the `launchd` assertions in `processTableContainsThisProcessWithPlausibleFigures`).32- Not run: any test on macOS 15, on an Intel Mac, or on a Mac with several displays.3334## Testing principles35361. **Arithmetic is tested without the system.** Every formula lives in a pure function (`CPUMath`, `MemoryMath`, `CounterMath`, `NetworkMath`, `DiskMath`, `ProcessMath`, `PowerMath`, `SchedulerMath`, `SeriesMath`, `Detectors`) and is checked with hand-computed inputs, including the awkward cases: counter wraparound, counter reset, zero elapsed time, inconsistent kernel counters.372. **Collectors are tested against the real machine with invariants, not expected values.** A live reading cannot be compared with a constant, but it can be required to satisfy identities (fractions sum to 1, used + cached + free = physical) and bounds.383. **Time is injected.** `TelemetryEngine.tick(now:date:)`, `AlertEngine.evaluate(_:at:)`, `TimeSeriesStore.append(_:at:)` and the `now:` parameters of queries take explicit times, so sleep, gaps, cooldowns and retention are tested deterministically, without waiting.394. **The store is tested in memory.** `TimeSeriesStore(url: nil)` opens a private database; one test uses a temporary file to cover relaunch.405. **Interface tests run an isolated instance** (`-ORIXUITest YES`, a temporary `-ORIXDataDir`) that never touches real preferences or history.416. **Honesty properties are asserted explicitly**: unknown values are absent rather than zero, unavailable sensors are declared, gaps are reported rather than filled.4243## Core tests4445### `MetricMathTests.swift` — arithmetic of every collector4647| Suite | Test | Verifies |48|---|---|---|49| `CPUMathTests` | `loadSplitsTicksByState` | User (with nice folded in), system and idle fractions from tick deltas; they sum to 1 |50| | `tickCounterWraparoundIsHandled` | 32-bit tick counters that wrap between two readings still give the right fractions |51| | `noElapsedTicksGivesZeroNotNaN` | Identical readings and empty inputs yield `.zero` |52| | `aggregateWeighsCoresByTicksNotByAverageOfRatios` | The whole-Mac figure sums ticks before dividing (25 %, not 50 %, in the example) |53| | `aggregateRejectsCoreCountChange` | A changed core count gives `.zero` rather than a misaligned result |54| `MemoryMathTests` | `accountingFollowsActivityMonitorConvention` | app, wired, compressed, used, cached and free from page counters; the identity with physical memory |55| | `inconsistentCountersNeverUnderflow` | Impossible counter combinations clamp instead of underflowing |56| | `pressureLevelMapping` | Kernel values 1, 2, 4 (and 0) map to normal, warning, critical; ordering |57| `CounterMathTests` | `plainDelta`, `thirtyTwoBitWraparound`, `counterResetIsNotAWrap`, `rateNeedsPositiveElapsedTime` | Deltas, single-wrap tolerance for narrow counters, resets returning `nil`, and rates being `nil` without elapsed time or delta |58| `NetworkMathTests` | `throughputFromByteCounters` | Byte deltas over elapsed time |59| | `interfaceResetYieldsNoRateInsteadOfASpike` | A re-created interface yields no rate |60| | `legacyThirtyTwoBitCountersWrap` | The 32-bit path |61| | `interfaceClassification` | Loopback, Wi-Fi, Ethernet, tunnel, bridge, "other" (`awdl0`); which kinds count toward totals |62| `DiskMathTests` | `throughputAndOperations` | Byte and operation rates |63| | `detachedAndReattachedDiskGivesNoRate` | A counter reset yields no rate |64| | `volumeUsage` | `used` and `usedFraction` |65| `ProcessMathTests` | `cpuPercentIsPerCore` | 1.5 s of CPU in 1 s is 150 %; backwards counters and zero elapsed time give `nil` |66| | `machTimeConversion` | Mach units to nanoseconds with the 125/3 timebase, and no overflow for large values |67| | `spikeTrackerReportsFinishedEpisodesOnce` | `ProcessSpikeTracker` reports a finished episode once with its duration and peak, and ignores a short burst |68| `PowerMathTests` | `signedRegistryValues` | Unsigned registry integers reinterpreted as signed |69| | `nestedBatteryLayout` | The macOS 27 `AppleSmartBattery` shape: capacities under `BatteryData`, no temperature, `PowerTelemetryData.SystemLoad` |70| | `flatBatteryLayoutAndImplausibleTemperature` | The flat layout; an out-of-range temperature is rejected; an empty registry gives all-`nil` fields |71| | `derivedBatteryFigures` | Battery watts and capacity ratio, and their absence when inputs are missing |72| | `gpuStatisticsParsing` | `PerformanceStatistics` parsing, clamping to 100, missing keys, missing dictionary |73| `SchedulerMathTests` | `intervalAdaptsToVisibilityAndPower` | `effectiveInterval` for foreground / background, battery, Low Power Mode, saver off, and an interval longer than the background interval |74| | `discontinuityDetection` | The `max(4 × planned, planned + 10)` rule on both sides of the limit |75| | `normalizerOmitsWhatWasNotMeasured` | With nothing measured, only thermal state and own memory are produced |76| | `catalogueDescribesEveryStoredMetric` | Every `MetricID` has a source and a method, keys are unique and round-trip, GPU utilization is classified best effort |7778### `StoreTests.swift` — the time-series store7980| Suite | Test | Verifies |81|---|---|---|82| `SeriesMathTests` | `tierSelection` | Raw for one recent hour, minute for a day, hour for a month, and minute for an hour that is older than raw retention |83| | `stepNeverFinerThanTier` | Bucket width is a whole multiple of the tier's resolution and respects the point budget |84| | `gapDetection` | Empty range, contiguous points, leading and interior gaps with their exact bounds |85| `TimeSeriesStoreTests` | `rawSamplesRoundTrip` | Samples come back with the right tier, count, extremes and mean, and no gap |86| | `minuteAndHourAggregatesAreExact` | Minute and hour rows hold the exact min, max, mean and count |87| | `downsamplingPreservesExtremesAndWeightedMean` | A single spike survives downsampling; the overall mean equals the exact mean of all samples |88| | `gapsAreReportedNotFilled` | A ten-minute hole produces one gap of the right duration and no invented point |89| | `nonFiniteValuesAreDropped` | NaN and infinity are not stored |90| | `retentionPrunesEachTier` | Each tier is pruned by its own retention |91| | `historySurvivesRelaunchAndPartialBucketsMerge` | A database file reopened by a new store keeps its samples, and a bucket begun before the relaunch is continued, not overwritten |92| | `timestampsKeepMillisecondPrecision` | Sub-second samples are all kept; raw query buckets are floored at one second |93| | `eventsRoundTripFilterAndDelete` | Events round-trip (including non-ASCII payloads), filter by kind and time, delete individually; `eraseAll` empties everything |9495### `AnalysisTests.swift` — alerts, detectors, findings, reports9697| Suite | Test | Verifies |98|---|---|---|99| `AlertEngineTests` | `firesOnlyAfterTheHoldDuration` | No alert before the hold time; one `fired` transition with value and held time; no repeat while still above |100| | `aDipResetsTheHoldTimer` | One sample below the threshold restarts the hold |101| | `resolvesThenRespectsCooldown` | `resolved` on recovery; no re-fire during the cooldown; re-fire after it |102| | `belowComparisonAndExactThreshold` | "Below" rules; a value equal to the threshold does not match |103| | `snoozeSuppressesThenFires` | Nothing fires while snoozed; the alert fires once the snooze ends |104| | `missingMetricAndSleepGap` | A missing metric does nothing; an hour without evaluation does not count as time above the threshold |105| | `disablingAFiringRuleResolvesIt` | Disabling emits `resolved` |106| | `holdTimerSurvivesLongSamplingIntervals` | With a 40 s sampling interval and a gap allowance that follows it, a 60 s rule still fires (regression for hold timers being cleared at long intervals) |107| | `defaultRulesAreSane` | Default rules exist, have unique identifiers and cooldowns of at least a minute |108| `DetectorTests` | `sustainedEpisodes` | Exact start, end, duration, peak, mean and sample count of an episode; a too-short one is ignored |109| | `aDataGapEndsAnEpisode` | Two runs separated by a hole are not joined |110| | `linearFitRecoversSlope` | Slope and intercept of a known line; `nil` with two points |111| | `exhaustionProjection` | Projection date and bytes per day for a known decline; no projection for growing space or short history |112| | `anomaliesNeedABaselineAndAFloor` | One outlier found; none below the absolute floor; none with too little baseline; `median` |113| `IntelligenceEngineTests` | `quietSystemProducesNoFinding` | A calm history yields no finding |114| | `everyFindingCarriesItsEvidence` | A constructed history yields the six expected finding kinds with their duration, peak, threshold, sample count, severity, projection date, subject and occurrences; ordering by severity; unique identifiers |115| `ReportTests` | `markdownContainsUnitsMethodsGapsAndLimitations` | The Markdown report contains the source API, sampling interval, gaps, limitations and findings sections, escapes pipes, and omits the host name unless requested |116| | `levelMetricsAreNamedNotNumbered` | Memory pressure and thermal state are printed as names, through the localiser; other metrics keep their unit |117| | `csvIsLongFormatAndMachineReadable` | Header, column count, stable metric key, ISO timestamp, locale-independent numbers, CSV quoting |118| | `jsonRoundTrips` | A report decodes back from its JSON with points, gaps and findings intact |119| | `frenchCanadianFormattingAndLocalisation` | `fr_CA` number, percent and byte formatting; the localiser closure is applied; the "no stored samples" limitation comes first for an empty report |120| | `comparisonTable` | The percentage change between two reports |121| | `everyReportKindListsStoredMetrics` | Every report kind has metrics and a title |122123### `LiveCollectorTests.swift` — against the real machine124125These suites are `.serialized`, because they read global system state and burn CPU deliberately.126127| Suite | Test | Verifies |128|---|---|---|129| `LiveCollectorTests` | `cpuNeedsABaselineThenReportsEveryCore` | No sample without a baseline; then one load per logical core, each in 0…1 with fractions summing to 1; three load averages; `reset()` drops the baseline |130| | `topologyMatchesTheKernel` | Logical core count equals `ProcessInfo.processorCount`; group cores sum to the total |131| | `memoryAccountingAddsUp` | Physical memory matches `ProcessInfo`; used + cached + free = physical; no rate on the first sample, non-negative rates on the second; swap used ≤ swap total |132| | `volumesIncludeTheStartupVolume` | A root volume mounted at `/` with a `disk…` BSD name; available and free never exceed total |133| | `diskActivityHasCountersAndThenRates` | Devices are listed; no rate first, non-negative rates next; a physical device has read bytes |134| | `networkInterfacesAndRates` | Loopback is present and up; interface names are unique; no rate first; rates non-negative; no `%scope` suffix in addresses |135| | `processTableContainsThisProcessWithPlausibleFigures` | The test process is listed with statistics, the right parent, path, user and a recent start time; after burning CPU its percentage and CPU time rise plausibly; `launchd` is listed as root **without statistics** (when not running as root) and restricted counts are consistent |136| | `cpuTimeAgreesWithGetrusage` | The Mach-time conversion of own CPU time agrees with `getrusage` |137| | `powerThermalAndGPUReadingsAreCoherent` | Battery fields are within plausible ranges, or absent without a battery; best-effort fields are absent when disabled; thermal level in 0…3; at least one Metal device; utilization within 0…100 |138| | `capabilityMatrixIsHonest` | Capability identifiers are unique; nothing classified unavailable is marked present; fans and CPU temperature are unavailable; CPU load is present |139| | `systemInformation` | Memory matches `ProcessInfo`; uptime is positive; model and OS version are non-empty |140| `EngineTests` | `engineLifecycleStreamsAndPersists` | The real loop streams snapshots: the first has no CPU value (no invented baseline), later ones do; sequence numbers, collection time, own usage and interval are coherent; samples reach the store |141| | `sleepWakeDropsBaselinesAndLogsTheEvent` | After an announced wake and after an unannounced hole, rate values are absent, `afterDiscontinuity` is set, and `system.wake` / `system.gap` events are stored |142| | `pausedEngineSamplesNothingAndAlertsReachTheStore` | A paused tick produces no snapshot; a rule that always matches fires, appears in `firingRules`, and is stored as an `alert.fired` event; removing the rule clears it |143| | `findingsRunOverStoredHistory` | Forty minutes of stored high CPU produce a sustained-CPU finding through the engine |144145Because these tests read the machine they run on, they assume a normal macOS user session: more than twenty processes, a mounted startup volume, a loopback interface and a Metal device. Several assertions are conditional (battery present or not, running as root or not).146147## Interface tests — `ORIXMonitorUITests`148149Every test launches a fresh app instance with `-ORIXUITest YES`, a per-test temporary `-ORIXDataDir`, `-ORIXInterval 1`, `-ORIXPrivacy YES`, `-ApplePersistenceIgnoreState YES` and English as the language. The temporary directory is removed in `tearDown`. Pages are located through accessibility identifiers (`sidebar.<section>`, `page.<section>`, `widget.<kind>`, `toolbar.pause`, …).150151| Test | Verifies |152|---|---|153| `testFirstLaunchShowsDashboardAndStartsSampling` | The Overview page appears, the sidebar reports "Sampling every …", the core widgets exist, and a real percentage reaches a sidebar badge |154| `testEveryPageOpens` | Each of the thirteen sections opens from the sidebar and shows a characteristic text |155| `testUnavailableSensorsAreDeclaredNotInvented` | The Thermal & Sensors page shows "Not available through supported system interfaces." and no "rpm" value |156| `testDashboardEditingAddsAndRemovesAWidget` | In edit mode the "ORIX Monitor overhead" widget can be added from the menu and removed again |157| `testProcessesPageSurvivesANarrowWindow` | Regression test: at 860 × 560 the Processes page used to abort the app (a split-view constraint loop); the app must keep running and the table must reappear after navigating away and back |158| `testProcessExplorerListsAndFilters` | The table and count appear; searching "ORIX Monitor" finds the app itself; a nonsense query shows "0 shown of …" |159| `testAlertRuleCanBeAddedAndSnoozed` | The default enabled-rule count is shown; adding a rule through the editor increases it; snoozing for 15 minutes shows "Snoozed until" |160| `testReportPreviewIsBuiltFromStoredSamples` | After a few seconds of sampling, refreshing the report shows the System Overview title and the Limitations section, and the four export buttons are enabled |161| `testSettingsWindowAndPauseCommand` | The Settings window opens from the app menu, each of its six tabs can be selected, it closes; the toolbar pause button toggles "Sampling paused" / "Sampling every" |162| `testMenuBarExtraOpensPanel` | A status item exists and carries a CPU reading in its accessibility title; if macOS has not hidden the item, clicking it opens the panel with "Top processes", and "Open ORIX Monitor" returns to the Overview. Skipped (not failed) when the item is not hittable |163| `testLightAndDarkAppearance` | The app launches on the CPU page in both appearances; a screenshot of each is attached to the test result |164| `testWindowResizingKeepsContentReachable` | After the app resizes its own window to 900 × 600, the CPU widget is still hittable and the Memory and Network pages still show their content |165| `testHistorySurvivesRelaunch` | After sampling, quitting through the menu (which flushes), and relaunching on History with the same data directory, the page reports "Collected locally since …" and a non-zero sample count |166167Requirements and caveats for the interface tests:168169- They drive a real GUI session and need the macOS permissions XCTest UI testing normally requires.170- `testMenuBarExtraOpensPanel` depends on the menu bar having room for the status item (it can be hidden behind the notch or by a full menu bar); it then skips the click-through part.171- They use real timing (`sleep`, `waitForExistence`) and so are slower and more environment-sensitive than the core tests.172- The UI-test target is signed with an "Apple Development" identity (`project.yml`).173174## Other checks175176| Check | Command | What it verifies |177|---|---|---|178| Repository hygiene | `Scripts/check.sh` | Author headers on first-party sources; no secret-like strings; no certificates, keys, databases or `.env` files |179| Signature | `Scripts/sign.sh` | Developer ID authority, Hardened Runtime, timestamp, identifier, strict verification, no `get-task-allow` |180| Translation coverage | `Scripts/localize.py --missing` | Lists keys without a French value |181| Manual validation against system tools | `swift run orix-probe [seconds]` | Prints one reading of every collector, to compare by eye with Activity Monitor, `vm_stat`, `df`, `netstat -ib` and similar |182183## Not covered184185- **No unit tests for the app layer.** `MonitorModel`, `AppSettings`, `PDFReportRenderer`, `MenuBarRenderer` and the SwiftUI views are exercised only through the interface tests; there is no app unit-test target in `project.yml`.186- **PDF output is not verified** beyond the export button being enabled. No test opens or inspects a generated PDF.187- **No snapshot or pixel-comparison tests.** `testLightAndDarkAppearance` attaches screenshots for a human to look at.188- **No automated accessibility audit** and no contrast test.189- **French is not exercised in the interface tests** (they force English). French formatting is covered at the `ValueFormatter` level only.190- **Real sleep and wake are not triggered**; they are simulated through `noteDiscontinuity` and injected clock values.191- **Notification delivery, launch at login and process termination** are not tested.192- **Hardware-dependent paths are tested only as far as the test machine allows**: no desktop Mac (no battery), no external or network volume, no VPN, no eGPU, no Intel Mac.193- **No performance assertions** in the test suites; overhead is measured separately (see [performance](performance.md)).194- **No code-coverage figure is collected.**195
Files main
Menu

Theme