SPB Git

Repository Public License

testing.md 15.2 KB 191 lines markdown

1<!-- ORIX Clean / File: docs/development/testing.md / Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai -->2# Testing34Two layers: Swift Testing suites for the core package, and XCUITest scenarios that drive the built application5on a disposable disk image. This document lists what each test proves, as read from the test sources.67## Running89```sh10Scripts/build.sh core-test                       # swift test in Packages/ORIXCleanCore11Scripts/build.sh uitest                          # all UI scenarios on a fresh disk image12ORIX_ONLY=testReportExportWritesAFile Scripts/build.sh uitest   # a single UI scenario13```1415## The safety rule1617**Destructive tests run only on temporary directories and disk images.** No test removes, moves or modifies a18file in a real user folder.1920- Core tests create a `Sandbox`: a new directory `orixclean-tests-<label>-<id>` under `NSTemporaryDirectory()`.21- Tests that need another volume create a `TestVolume`: an APFS disk image made with `hdiutil`, attached at a22  mount point inside the temporary directory, detached and deleted when the object is released.23- Most deletion tests inject a `FakeTrasher`, which moves items into a `bin` folder inside the sandbox instead24  of the user's Trash and can be told to fail for chosen paths.25- Two tests use the real `SystemTrasher` on purpose (`moveToTheRealTrashAndRestore`,26  `readOnlyVolumeFailsCleanlyAndFullVolumeStillTrashes`). They trash only files they created in the sandbox or27  on the test image; the first restores its item afterwards.28- `SafetyPolicy` is constructed with the sandbox as home, or with the fictitious `/Users/tester`.29- Read-only assertions are made against real system paths in a few places (the signature of30  `/System/Applications/Mail.app`, the presence of a startup volume); nothing is written there.31- UI tests launch the application with `-ORIXHome /tmp/orix-uitest/home`, a folder on a 900 MB APFS disk image32  created and destroyed by `Scripts/build.sh uitest`. Cleanup scenarios really move files to the Trash of that33  image. The history store is in memory (`-ORIXStore memory`).3435## Core tests3637`Packages/ORIXCleanCore/Tests/ORIXCleanCoreTests/` declares 56 `@Test` functions in 9 suites (counted in the38sources on 2026-11-21).3940### ScannerTests (16)4142| Test | What it proves |43|---|---|44| `sizesMatchTheKernel` | File and folder counts, logical and allocated totals equal the sums from `stat`, for sizes around the block boundary; the allocated total equals `du -sk` |45| `unicodeAndLongNames` | Accented, CJK, emoji, decomposed and 240-character names are returned exactly as on disk; extension parsing works |46| `pathsLongerThanPathMax` | A tree deeper than 1,500 bytes of path is scanned without issue through the `openat` fallback |47| `symlinkLoopsAreNotFollowed` | A link to the root, a link to a folder and a dangling link are recorded as links with no children; content is counted once |48| `hardLinksAreCountedOnce` | Three names of one inode count as three files but one file's worth of bytes; two are flagged duplicates |49| `apfsClonesReportSharedBlocks` | Clones are flagged as possibly sharing blocks, report full allocated size and near-zero private size, and share a clone id |50| `sparseFilesHaveSmallAllocation` | A 200 MB sparse file reports its logical size and a small allocated size equal to `stat` |51| `permissionDeniedIsReportedAndScanContinues` | An unreadable folder yields one `permissionDenied` issue and the `unreadable` flag; the rest is scanned |52| `missingAndNonDirectoryRoots` | The right `ScanError` is thrown for a missing root and for a file |53| `filesChangingAndDisappearingDuringScan` | 25 scans while another thread deletes and recreates folders: no crash, only `vanished` or `ioError` issues, and the root total always equals the sum over files |54| `largeDirectoryTree` | 32,000 files in 203 folders, including one folder of 12,000 entries, are all counted |55| `packagesAreFlagged` | `.app` and `.photoslibrary` are packages; their contents are hidden from "largest files" unless requested |56| `cancellationAndPause` | A cancelled scan returns `cancelled`; a paused scan makes no progress until resumed, then completes |57| `excludedPathsAndRemoval` | Excluded paths are flagged `skipped`; `markRemoved` subtracts sizes and counts from ancestors and hides the subtree |58| `mountedVolumeIsABoundaryAndRemovalIsAnError` | A volume mounted inside the root is a `boundary` and is not entered unless `stayOnDevice` is off; the volume can be scanned directly; after ejection the mount point is an empty folder |59| `queriesAndCategories` | Largest files and folders, filters by category, age, name and size, category totals, `Measure.item`, and volume lookup |6061### TreemapTests (1)6263| Test | What it proves |64|---|---|65| `areasAreProportionalAndInsideTheParent` | Cells stay inside the rectangle, areas are proportional to sizes, leaves do not overlap, 50 tiny files become one aggregate cell with the right count and size, and viewport culling reduces the cell count |6667### DuplicateTests (9)6869| Test | What it proves |70|---|---|71| `exactDuplicatesAreGroupedAndDifferentFilesAreNot` | Identical files are grouped, same-size different files are not, redundant size is correct, the digest equals `shasum -a 256`, and the minimum size filter works |72| `differencesOutsideTheSampledWindowsAreCaughtByTheFullHash` | One changed byte outside the three sampled windows passes the sample hash but is caught by the full hash and by the byte comparison |73| `collisionsInThePreliminaryFilterDoNotCreateFalseGroups` | With a sampler that always collides, groups are still decided by the full hash |74| `hardLinksAreOneFileNotDuplicates` | Two names of one inode are not a duplicate pair; with a real copy, the linked file lists its other name and removing it is estimated to free nothing |75| `clonesAreDuplicatesThatShareStorage` | A clone pair is one group flagged as sharing storage, with a recoverable range from near zero to the full size |76| `largeFilesAndUnreadableFiles` | Two 48 MB real copies are hashed in full and grouped; an unreadable file does not break the run |77| `aFileThatChangedSinceTheScanIsDropped` | A file whose size changed after the scan is excluded |78| `packageContentsAreSkippedUnlessAsked` | Identical files inside two applications are ignored by default and found when requested |79| `cancellationReturnsNoGroups` | A cancelled run returns no groups |8081### SimilarImageTests (3)8283| Test | What it proves |84|---|---|85| `resizedAndRecompressedCopiesAreGroupedAndUnrelatedImagesAreNot` | A picture, its resized recompressed JPEG and its PNG form one group; seven unrelated pictures form none |86| `thoroughModeFindsCrops` | Thorough mode groups a picture with its cropped version |87| `corruptImagesAreCountedNotCrashed` | An undecodable `.jpg` is counted as unreadable and nothing is grouped |8889### SafetyPolicyTests (2)9091| Test | What it proves |92|---|---|93| `protectedLocationsAreDenied` | 32 paths are refused: system areas, top-level and standard folders, another user's home, the keychain, volume roots, the inside of an application and of a photo library, relative paths, `..`, the Data-volume alias, `/tmp` |94| `ordinaryItemsAreAllowed` | 13 ordinary paths are allowed, including a whole application, a whole photo library, items inside the Trash and items on another volume |9596### DeletionTests (11)9798| Test | What it proves |99|---|---|100| `nothingIsRemovedWithoutAuthorization` | An item in the plan but not in the authorization is skipped and stays on disk |101| `highRiskNeedsExplicitAcknowledgement` | A high-risk item is skipped unless acknowledged |102| `moveToTheRealTrashAndRestore` | With the real Trash: the item leaves its place, exists at the reported Trash path, is restored with identical content, and cannot be restored twice |103| `restoreRefusesToOverwrite` | Restore returns `destinationOccupied` when a new item exists at the original path |104| `revalidationSkipsVanishedReplacedAndChangedItems` | Items that vanished, were replaced by another inode, or changed size are skipped |105| `duplicatesAreVerifiedAgainstTheKeptCopy` | A true duplicate is removed; a copy that drifted, one whose kept copy is missing, and a hard link of the kept copy are skipped |106| `aGroupIsNeverEmptied` | Two duplicates each naming the other as the kept copy: neither is removed |107| `partialFailureIsReportedPerItem` | One failure does not stop the run; a volume without Trash yields `requiresManualAction` and the item stays |108| `protectedPathsRunningAppsPermanentModeAndCancellation` | Home, Documents and Library are refused; data of a running application requires manual action; permanent mode is refused outside the Trash category and works inside it; a cancelled engine skips everything |109| `readOnlyVolumeFailsCleanlyAndFullVolumeStillTrashes` | On a read-only image nothing is removed and the failure is reported; on a full image the move to that volume's Trash still succeeds and the Trash inventory sees it |110| `planDeduplicatesNestedItems` | A folder listed twice together with items inside it yields one plan item with correct totals |111112### CatalogTests (6)113114| Test | What it proves |115|---|---|116| `cachesSectionListsOnlyDocumentedSafeLocations` | Application caches, a browser cache and logs are listed; `com.apple.*`, CloudKit, tiny entries and developer caches are not; owner, apps to close, risk and sizes are set; every item passes the safety policy |117| `developerSectionSplitsPerProjectAndMarksArchivesForReview` | DerivedData is one low-risk item per project; archives are `review`; npm and Homebrew caches are found; every item passes the safety policy |118| `everyRuleIsDocumented` | Every catalogue rule has a reason, a consequence and a regeneration note, and rule ids are unique |119| `buildArtifactsNeedTheirManifest` | `node_modules` and `.build` are proposed only next to their manifest; look-alikes are not |120| `downloadsAreClassified` | Disk image, installer, archive, incomplete download and other items are classified; hidden files are ignored; an extracted sibling is detected and mentioned |121| `trashInventory` | Items and sizes of a Trash folder are listed; an unreadable Trash is reported as not accessible |122123### ApplicationTests (3)124125| Test | What it proves |126|---|---|127| `discoveryAssociatedFilesAndUninstallPlan` | Bundle facts are read; associated files are matched with the right confidence; look-alike names are not matched; name-only matches become high risk; documents are never included |128| `systemApplicationsAreNeverOffered` | A system application yields no uninstall items; its signature is read as Apple and valid |129| `leftoversAreInferredConservatively` | Only the identifier with no installed application, no shared vendor prefix, no Apple prefix and no Launch Services registration is reported |130131### HistoryAndReportTests (3)132133| Test | What it proves |134|---|---|135| `historyStoreRoundTrip` | Samples, the 15-minute de-duplication, the trend slope, watched folders, folder samples, operations and documents round-trip; data survives reopening; erase empties samples and operations |136| `recommendationsComeOnlyFromEvidence` | Empty findings give no recommendation; low space, large DerivedData and a growth trend each appear only when their data is present |137| `reportRenderers` | Markdown escapes table pipes; CSV quotes fields and neutralizes formulas; JSON decodes with all sections; PDF output starts with `%PDF-`; cleanup results carry status and note |138139### WorkflowTests (2)140141| Test | What it proves |142|---|---|143| `validationForbidsRemovalWithoutReview` | Removal without review, removal after an intervening collection, a second removal after one review, duplicates without a scan and an empty workflow are all rejected; the built-in workflows are valid |144| `duplicateWorkflowKeepsOneCopyAndStopsWhenReviewIsDeclined` | A declined review removes nothing; an approved one removes two of three copies and leaves the unrelated file |145146## UI tests147148`UITests/ORIXCleanUITests.swift` has 8 scenarios. `Scripts/build.sh uitest` regenerates the project, creates149the disk image at `/tmp/orix-uitest`, builds `orix-bench` in release mode, writes the sample home folder at150scale 0.25, runs the tests, then detaches and deletes the image. Tests are skipped when the sample folder is151missing. The application is launched in English (`-AppleLanguages (en)`, `-AppleLocale en_CA`), and each152scenario keeps window screenshots as attachments.153154| Scenario | What it checks |155|---|---|156| `testFirstLaunchShowsOnboardingAndNothingIsScanned` | Onboarding appears on first launch; after "Get Started" the Overview is shown with no scan and no selection |157| `testScanStorageMapSearchAndSidebarNavigation` | A scan produces the map and its list; search finds a file and shows a match count; Large Files lists the largest file |158| `testDuplicatesAndSimilarImages` | Duplicate groups appear with "3 copies" and a "shares storage" label; similar-image groups appear |159| `testCachesDeveloperDownloadsAndTrashLists` | Each of the four lists shows its expected fixture item; nothing is preselected |160| `testUninstallPreviewCancelThenRealCleanupAndPutBack` | Selecting removes nothing; cancelling the preview removes nothing; confirming moves the application to the Trash; Put Back restores it; the selection bar disappears |161| `testReportExportWritesAFile` | Exporting a report writes a file that starts with `%PDF-` |162| `testActionStopsWhenTheReviewIsDeclined` | Running the built-in cache action and cancelling its preview leaves the cache folder in place and logs "Nothing was removed" |163| `testSettingsLightModeHistoryAndWindowResize` | Light appearance, the History section, the Settings window, and the layout at 980 by 620 and at 1800 by 1100 |164165## Known flakiness166167- **Keyboard focus.** UI tests can fail when another process takes keyboard focus during a run (another168  automation session, a system dialog, a notification that activates an application). Typed text then goes to169  the wrong window or a click lands on an inactive one. The helpers reduce this: `tap` re-activates the170  application and retries up to four times, and `type` checks `hasKeyboardFocus` before typing. They do not171  eliminate it. Run UI tests on a machine that is otherwise idle, and rerun a single scenario with `ORIX_ONLY`172  before treating a failure as a regression.173- **Shared paths.** The image is always mounted at `/tmp/orix-uitest`, so two UI test runs cannot overlap on174  one machine.175- **Timing.** Scenarios wait up to 40 seconds for analyses; a heavily loaded machine can exceed that.176- **Core tests using `hdiutil`.** Two core tests create disk images. When `hdiutil` is unavailable they record177  an issue instead of passing silently.178- **Unreadable-file test.** `largeFilesAndUnreadableFiles` accepts either outcome for the locked file's listing,179  because the sampling stage can drop it before the full hash.180181## What is not tested182183- Intel execution, and macOS 15 and 26: tests were run on Apple silicon with macOS 27.0 only.184- Real TCC prompts and Full Disk Access transitions (the probe is exercised only against the current state).185- External disks, HDDs and network volumes; non-APFS filesystems.186- VoiceOver or Voice Control behaviour; UI tests use accessibility identifiers, which shows the elements are187  exposed but not that the experience is good.188- The French interface in UI tests.189- The release scripts (signing, notarization, packaging).190- Emptying the Trash through the UI (permanent mode is covered at engine level).191
Files main
Menu

Theme