SPB Git

Repository Public License

testing.md 20.0 KB

# Testing

ORIX Monitor has two test layers:

Layer Location Framework Runs with
Core tests Packages/ORIXMonitorCore/Tests/ORIXMonitorCoreTests/ Swift Testing (@Suite, @Test, #expect) Scripts/build.sh test (swift test)
Interface tests UITests/ORIXMonitorUITests.swift XCTest UI testing Scripts/build.sh uitest (xcodebuild test)

Scripts/build.sh all runs the core tests, a Debug build and the interface tests in that order. Scripts/release.sh runs the core tests before building a release.

# Results

Results recorded on 2026-11-21 on the development Mac (MacBook Pro Mac17,6, Apple M5 Max, macOS 27.0, Xcode 27.2), on the source that produced the notarized 1.0.0 (build 1).

Suite Command Result
Core Scripts/build.sh test (swift test) 81 tests in 16 suites passed, about 2.5 s
Interface Scripts/build.sh uitest 13 executed: 12 passed, 1 skipped, 0 failed
  • The skipped test is testMenuBarExtraOpensPanel: it verified that the status item exists and carries a CPU reading, then skipped the click because macOS hides the item behind the display notch on this Mac. The panel was therefore not opened by an automated click.
  • testAccessibilityAuditOfEveryPage passes by design (it records the audit's notes instead of failing); its output is discussed in ../reports/accessibility.md. It takes about 12 minutes, most of the suite's run time.
  • The interface tests need the screen to themselves. Runs made while another automation was driving the same Mac produced "not hittable" failures that disappeared when the suite ran alone; those runs are not counted above.
  • Earlier runs of the same suite found two real defects, both fixed before release: the app aborted on the Processes page in a narrow window (now covered by testProcessesPageSurvivesANarrowWindow), and processes of other users were missing from the table (now covered by the launchd assertions in processTableContainsThisProcessWithPlausibleFigures).
  • Not run: any test on macOS 15, on an Intel Mac, or on a Mac with several displays.

# Testing principles

  1. Arithmetic is tested without the system. Every formula lives in a pure function (CPUMath, MemoryMath, CounterMath, NetworkMath, DiskMath, ProcessMath, PowerMath, SchedulerMath, SeriesMath, Detectors) and is checked with hand-computed inputs, including the awkward cases: counter wraparound, counter reset, zero elapsed time, inconsistent kernel counters.
  2. Collectors are tested against the real machine with invariants, not expected values. A live reading cannot be compared with a constant, but it can be required to satisfy identities (fractions sum to 1, used + cached + free = physical) and bounds.
  3. Time is injected. TelemetryEngine.tick(now:date:), AlertEngine.evaluate(_:at:), TimeSeriesStore.append(_:at:) and the now: parameters of queries take explicit times, so sleep, gaps, cooldowns and retention are tested deterministically, without waiting.
  4. The store is tested in memory. TimeSeriesStore(url: nil) opens a private database; one test uses a temporary file to cover relaunch.
  5. Interface tests run an isolated instance (-ORIXUITest YES, a temporary -ORIXDataDir) that never touches real preferences or history.
  6. Honesty properties are asserted explicitly: unknown values are absent rather than zero, unavailable sensors are declared, gaps are reported rather than filled.

# Core tests

# MetricMathTests.swift — arithmetic of every collector

Suite Test Verifies
CPUMathTests loadSplitsTicksByState User (with nice folded in), system and idle fractions from tick deltas; they sum to 1
tickCounterWraparoundIsHandled 32-bit tick counters that wrap between two readings still give the right fractions
noElapsedTicksGivesZeroNotNaN Identical readings and empty inputs yield .zero
aggregateWeighsCoresByTicksNotByAverageOfRatios The whole-Mac figure sums ticks before dividing (25 %, not 50 %, in the example)
aggregateRejectsCoreCountChange A changed core count gives .zero rather than a misaligned result
MemoryMathTests accountingFollowsActivityMonitorConvention app, wired, compressed, used, cached and free from page counters; the identity with physical memory
inconsistentCountersNeverUnderflow Impossible counter combinations clamp instead of underflowing
pressureLevelMapping Kernel values 1, 2, 4 (and 0) map to normal, warning, critical; ordering
CounterMathTests plainDelta, thirtyTwoBitWraparound, counterResetIsNotAWrap, rateNeedsPositiveElapsedTime Deltas, single-wrap tolerance for narrow counters, resets returning nil, and rates being nil without elapsed time or delta
NetworkMathTests throughputFromByteCounters Byte deltas over elapsed time
interfaceResetYieldsNoRateInsteadOfASpike A re-created interface yields no rate
legacyThirtyTwoBitCountersWrap The 32-bit path
interfaceClassification Loopback, Wi-Fi, Ethernet, tunnel, bridge, "other" (awdl0); which kinds count toward totals
DiskMathTests throughputAndOperations Byte and operation rates
detachedAndReattachedDiskGivesNoRate A counter reset yields no rate
volumeUsage used and usedFraction
ProcessMathTests cpuPercentIsPerCore 1.5 s of CPU in 1 s is 150 %; backwards counters and zero elapsed time give nil
machTimeConversion Mach units to nanoseconds with the 125/3 timebase, and no overflow for large values
spikeTrackerReportsFinishedEpisodesOnce ProcessSpikeTracker reports a finished episode once with its duration and peak, and ignores a short burst
PowerMathTests signedRegistryValues Unsigned registry integers reinterpreted as signed
nestedBatteryLayout The macOS 27 AppleSmartBattery shape: capacities under BatteryData, no temperature, PowerTelemetryData.SystemLoad
flatBatteryLayoutAndImplausibleTemperature The flat layout; an out-of-range temperature is rejected; an empty registry gives all-nil fields
derivedBatteryFigures Battery watts and capacity ratio, and their absence when inputs are missing
gpuStatisticsParsing PerformanceStatistics parsing, clamping to 100, missing keys, missing dictionary
SchedulerMathTests intervalAdaptsToVisibilityAndPower effectiveInterval for foreground / background, battery, Low Power Mode, saver off, and an interval longer than the background interval
discontinuityDetection The max(4 × planned, planned + 10) rule on both sides of the limit
normalizerOmitsWhatWasNotMeasured With nothing measured, only thermal state and own memory are produced
catalogueDescribesEveryStoredMetric Every MetricID has a source and a method, keys are unique and round-trip, GPU utilization is classified best effort

# StoreTests.swift — the time-series store

Suite Test Verifies
SeriesMathTests tierSelection Raw for one recent hour, minute for a day, hour for a month, and minute for an hour that is older than raw retention
stepNeverFinerThanTier Bucket width is a whole multiple of the tier's resolution and respects the point budget
gapDetection Empty range, contiguous points, leading and interior gaps with their exact bounds
TimeSeriesStoreTests rawSamplesRoundTrip Samples come back with the right tier, count, extremes and mean, and no gap
minuteAndHourAggregatesAreExact Minute and hour rows hold the exact min, max, mean and count
downsamplingPreservesExtremesAndWeightedMean A single spike survives downsampling; the overall mean equals the exact mean of all samples
gapsAreReportedNotFilled A ten-minute hole produces one gap of the right duration and no invented point
nonFiniteValuesAreDropped NaN and infinity are not stored
retentionPrunesEachTier Each tier is pruned by its own retention
historySurvivesRelaunchAndPartialBucketsMerge A database file reopened by a new store keeps its samples, and a bucket begun before the relaunch is continued, not overwritten
timestampsKeepMillisecondPrecision Sub-second samples are all kept; raw query buckets are floored at one second
eventsRoundTripFilterAndDelete Events round-trip (including non-ASCII payloads), filter by kind and time, delete individually; eraseAll empties everything

# AnalysisTests.swift — alerts, detectors, findings, reports

Suite Test Verifies
AlertEngineTests firesOnlyAfterTheHoldDuration No alert before the hold time; one fired transition with value and held time; no repeat while still above
aDipResetsTheHoldTimer One sample below the threshold restarts the hold
resolvesThenRespectsCooldown resolved on recovery; no re-fire during the cooldown; re-fire after it
belowComparisonAndExactThreshold "Below" rules; a value equal to the threshold does not match
snoozeSuppressesThenFires Nothing fires while snoozed; the alert fires once the snooze ends
missingMetricAndSleepGap A missing metric does nothing; an hour without evaluation does not count as time above the threshold
disablingAFiringRuleResolvesIt Disabling emits resolved
holdTimerSurvivesLongSamplingIntervals With a 40 s sampling interval and a gap allowance that follows it, a 60 s rule still fires (regression for hold timers being cleared at long intervals)
defaultRulesAreSane Default rules exist, have unique identifiers and cooldowns of at least a minute
DetectorTests sustainedEpisodes Exact start, end, duration, peak, mean and sample count of an episode; a too-short one is ignored
aDataGapEndsAnEpisode Two runs separated by a hole are not joined
linearFitRecoversSlope Slope and intercept of a known line; nil with two points
exhaustionProjection Projection date and bytes per day for a known decline; no projection for growing space or short history
anomaliesNeedABaselineAndAFloor One outlier found; none below the absolute floor; none with too little baseline; median
IntelligenceEngineTests quietSystemProducesNoFinding A calm history yields no finding
everyFindingCarriesItsEvidence A constructed history yields the six expected finding kinds with their duration, peak, threshold, sample count, severity, projection date, subject and occurrences; ordering by severity; unique identifiers
ReportTests markdownContainsUnitsMethodsGapsAndLimitations The Markdown report contains the source API, sampling interval, gaps, limitations and findings sections, escapes pipes, and omits the host name unless requested
levelMetricsAreNamedNotNumbered Memory pressure and thermal state are printed as names, through the localiser; other metrics keep their unit
csvIsLongFormatAndMachineReadable Header, column count, stable metric key, ISO timestamp, locale-independent numbers, CSV quoting
jsonRoundTrips A report decodes back from its JSON with points, gaps and findings intact
frenchCanadianFormattingAndLocalisation fr_CA number, percent and byte formatting; the localiser closure is applied; the "no stored samples" limitation comes first for an empty report
comparisonTable The percentage change between two reports
everyReportKindListsStoredMetrics Every report kind has metrics and a title

# LiveCollectorTests.swift — against the real machine

These suites are .serialized, because they read global system state and burn CPU deliberately.

Suite Test Verifies
LiveCollectorTests cpuNeedsABaselineThenReportsEveryCore No sample without a baseline; then one load per logical core, each in 0…1 with fractions summing to 1; three load averages; reset() drops the baseline
topologyMatchesTheKernel Logical core count equals ProcessInfo.processorCount; group cores sum to the total
memoryAccountingAddsUp Physical memory matches ProcessInfo; used + cached + free = physical; no rate on the first sample, non-negative rates on the second; swap used ≤ swap total
volumesIncludeTheStartupVolume A root volume mounted at / with a disk… BSD name; available and free never exceed total
diskActivityHasCountersAndThenRates Devices are listed; no rate first, non-negative rates next; a physical device has read bytes
networkInterfacesAndRates Loopback is present and up; interface names are unique; no rate first; rates non-negative; no %scope suffix in addresses
processTableContainsThisProcessWithPlausibleFigures The test process is listed with statistics, the right parent, path, user and a recent start time; after burning CPU its percentage and CPU time rise plausibly; launchd is listed as root without statistics (when not running as root) and restricted counts are consistent
cpuTimeAgreesWithGetrusage The Mach-time conversion of own CPU time agrees with getrusage
powerThermalAndGPUReadingsAreCoherent Battery fields are within plausible ranges, or absent without a battery; best-effort fields are absent when disabled; thermal level in 0…3; at least one Metal device; utilization within 0…100
capabilityMatrixIsHonest Capability identifiers are unique; nothing classified unavailable is marked present; fans and CPU temperature are unavailable; CPU load is present
systemInformation Memory matches ProcessInfo; uptime is positive; model and OS version are non-empty
EngineTests engineLifecycleStreamsAndPersists The real loop streams snapshots: the first has no CPU value (no invented baseline), later ones do; sequence numbers, collection time, own usage and interval are coherent; samples reach the store
sleepWakeDropsBaselinesAndLogsTheEvent After an announced wake and after an unannounced hole, rate values are absent, afterDiscontinuity is set, and system.wake / system.gap events are stored
pausedEngineSamplesNothingAndAlertsReachTheStore A paused tick produces no snapshot; a rule that always matches fires, appears in firingRules, and is stored as an alert.fired event; removing the rule clears it
findingsRunOverStoredHistory Forty minutes of stored high CPU produce a sustained-CPU finding through the engine

Because these tests read the machine they run on, they assume a normal macOS user session: more than twenty processes, a mounted startup volume, a loopback interface and a Metal device. Several assertions are conditional (battery present or not, running as root or not).

# Interface tests — ORIXMonitorUITests

Every test launches a fresh app instance with -ORIXUITest YES, a per-test temporary -ORIXDataDir, -ORIXInterval 1, -ORIXPrivacy YES, -ApplePersistenceIgnoreState YES and English as the language. The temporary directory is removed in tearDown. Pages are located through accessibility identifiers (sidebar.<section>, page.<section>, widget.<kind>, toolbar.pause, …).

Test Verifies
testFirstLaunchShowsDashboardAndStartsSampling The Overview page appears, the sidebar reports "Sampling every …", the core widgets exist, and a real percentage reaches a sidebar badge
testEveryPageOpens Each of the thirteen sections opens from the sidebar and shows a characteristic text
testUnavailableSensorsAreDeclaredNotInvented The Thermal & Sensors page shows "Not available through supported system interfaces." and no "rpm" value
testDashboardEditingAddsAndRemovesAWidget In edit mode the "ORIX Monitor overhead" widget can be added from the menu and removed again
testProcessesPageSurvivesANarrowWindow Regression test: at 860 × 560 the Processes page used to abort the app (a split-view constraint loop); the app must keep running and the table must reappear after navigating away and back
testProcessExplorerListsAndFilters The table and count appear; searching "ORIX Monitor" finds the app itself; a nonsense query shows "0 shown of …"
testAlertRuleCanBeAddedAndSnoozed The default enabled-rule count is shown; adding a rule through the editor increases it; snoozing for 15 minutes shows "Snoozed until"
testReportPreviewIsBuiltFromStoredSamples After a few seconds of sampling, refreshing the report shows the System Overview title and the Limitations section, and the four export buttons are enabled
testSettingsWindowAndPauseCommand The Settings window opens from the app menu, each of its six tabs can be selected, it closes; the toolbar pause button toggles "Sampling paused" / "Sampling every"
testMenuBarExtraOpensPanel A status item exists and carries a CPU reading in its accessibility title; if macOS has not hidden the item, clicking it opens the panel with "Top processes", and "Open ORIX Monitor" returns to the Overview. Skipped (not failed) when the item is not hittable
testLightAndDarkAppearance The app launches on the CPU page in both appearances; a screenshot of each is attached to the test result
testWindowResizingKeepsContentReachable After the app resizes its own window to 900 × 600, the CPU widget is still hittable and the Memory and Network pages still show their content
testHistorySurvivesRelaunch After sampling, quitting through the menu (which flushes), and relaunching on History with the same data directory, the page reports "Collected locally since …" and a non-zero sample count

Requirements and caveats for the interface tests:

  • They drive a real GUI session and need the macOS permissions XCTest UI testing normally requires.
  • testMenuBarExtraOpensPanel depends on the menu bar having room for the status item (it can be hidden behind the notch or by a full menu bar); it then skips the click-through part.
  • They use real timing (sleep, waitForExistence) and so are slower and more environment-sensitive than the core tests.
  • The UI-test target is signed with an "Apple Development" identity (project.yml).

# Other checks

Check Command What it verifies
Repository hygiene Scripts/check.sh Author headers on first-party sources; no secret-like strings; no certificates, keys, databases or .env files
Signature Scripts/sign.sh Developer ID authority, Hardened Runtime, timestamp, identifier, strict verification, no get-task-allow
Translation coverage Scripts/localize.py --missing Lists keys without a French value
Manual validation against system tools swift run orix-probe [seconds] Prints one reading of every collector, to compare by eye with Activity Monitor, vm_stat, df, netstat -ib and similar

# Not covered

  • No unit tests for the app layer. MonitorModel, AppSettings, PDFReportRenderer, MenuBarRenderer and the SwiftUI views are exercised only through the interface tests; there is no app unit-test target in project.yml.
  • PDF output is not verified beyond the export button being enabled. No test opens or inspects a generated PDF.
  • No snapshot or pixel-comparison tests. testLightAndDarkAppearance attaches screenshots for a human to look at.
  • No automated accessibility audit and no contrast test.
  • French is not exercised in the interface tests (they force English). French formatting is covered at the ValueFormatter level only.
  • Real sleep and wake are not triggered; they are simulated through noteDiscontinuity and injected clock values.
  • Notification delivery, launch at login and process termination are not tested.
  • Hardware-dependent paths are tested only as far as the test machine allows: no desktop Mac (no battery), no external or network volume, no VPN, no eGPU, no Intel Mac.
  • No performance assertions in the test suites; overhead is measured separately (see performance).
  • No code-coverage figure is collected.
Files main
Menu

Theme