SPB Git

Repository Public License

performance.md 8.3 KB 175 lines markdown

1<!-- ORIX Clean / File: docs/development/performance.md / Author: Simon-Pierre Boucher / Contact: contact@spboucher.ai -->2# Performance34Measurements of the core engines, how they were taken, how to repeat them, and what was not measured. Every5figure in this document was measured; the one estimate is labelled as such.67## Environment89| Item | Value |10|---|---|11| Date | 2026-11-21 |12| Machine | MacBook Pro `Mac17,6`, Apple M5 Max, 18 cores, 48 GB |13| System | macOS 27.0 (26A5425a) |14| Storage | Internal APFS SSD |15| Toolchain | Xcode 27.2, Swift 6.4.2 |16| Binary | `orix-bench`, release build |17| Cache state | Warm file cache (each tree had been read before the timed run) |1819## Methodology2021- All figures come from `orix-bench`, which calls the same `ORIXCleanCore` engines as the application.22- **Duration** is `ScanResult.duration` (wall clock, from the start of traversal to the end of aggregation) or23  the corresponding `duration` of the duplicate and similar-image results.24- **CPU seconds** are user plus system time from `getrusage(RUSAGE_SELF)` around the scan.25- **RSS** is `resident_size` from `task_info(MACH_TASK_BASIC_INFO)` read after the scan, with the tree still in26  memory. It includes the whole process, not only the tree.27- **Objects** are files plus folders. Throughput is objects divided by duration.28- **Treemap** time is the average of 100 consecutive layouts of the same tree.29- Each figure is from the runs reported here; no statistical treatment (variance, confidence interval) was30  done. Treat differences of a few percent as noise.31- Warm cache means metadata and file contents were already in memory. A first scan after boot, and any hashing32  of files that are not cached, will be slower and bound by the storage device.3334## Scan3536| Tree | Files | Folders | Threads | Duration | Throughput | Peak RSS |37|---|---|---|---|---|---|---|38| `~/Library/Developer` | 985,558 | 699,525 | 8 | 17.4 s | 96,757 objects/s | 202 MB (about 125 bytes per object) |39| `/Applications` | 448,727 | 64,278 | see below | 2.92 to 4.07 s | | about 77 MB |40| `/System/Library` | 296,089 | 159,923 | not recorded | 3.27 s | | |4142The `/System/Library` scan reported 68 unreadable folders; they are listed as issues and excluded from totals.4344Accuracy: on a tree of about 31,000 objects the allocated total reported by the scanner matched `du -sk`45exactly. The same equality is asserted on a small tree by the test `sizesMatchTheKernel`.4647### Thread count4849`/Applications`, same tree, warm cache:5051| Threads | Wall time | CPU time |52|---|---|---|53| 1 | 3.45 s | 3.4 s |54| 2 | 3.49 s | 6.8 s |55| 3 | 3.04 s | 8.9 s |56| 4 | 2.92 s | 11.2 s |57| 6 | 2.96 s | 16.7 s |58| 8 | 4.07 s | 23.4 s |5960Reading the table:6162- Wall time barely moves: the best run (4 threads, 2.92 s) is 15 % faster than one thread.63- CPU time grows almost linearly with the thread count: eight threads use about seven times the CPU of one64  and are slower than one in wall time.65- Directory enumeration is bound by the filesystem, not by the number of cores. Batches are also committed66  to the tree under a single lock, which is consistent with the flat wall time; lock contention was not67  profiled.6869**Decision: the default is 3 threads.** Three threads reach about 96 % of the best measured throughput70(2.92 / 3.04) for 38 % of the CPU time of eight (8.9 / 23.4). The default drops to 1 thread when the thermal71state is serious or critical, or when Low Power Mode is on (`FileScanner.run`). `ScanOptions.workers` overrides72it; `Measure.item` uses 2 for the small directory measurements made while listing candidates.7374The `~/Library/Developer` figure above was taken with 8 threads; that tree was not measured again with 3.7576## Memory model7778Scan results live in memory only. The tree costs:7980| Part | Cost |81|---|---|82| `ScanNode` | 64 bytes per object, in one contiguous array |83| Names | UTF-8 bytes of each name, back to back, no terminator |84| Clone ids | one dictionary entry per file that may share blocks |85| Transient | hard-link and directory "seen" sets during the scan, one 256 KiB buffer per thread, array growth slack |8687Measured: 202 MB resident for 1,685,083 objects, about 125 bytes per object for the whole process.8889**Estimate, not a measurement:** a scan of the whole startup volume of the test machine (about 15 million90inodes) at 125 to 160 bytes per object would need roughly 1.9 to 2.4 GB. This was not run. The per-object range91extrapolates from one measured tree; a volume with longer names or many hard links may fall outside it.9293Hard limits in the data structure: node indexes are `Int32` and name offsets are `UInt32`, so a tree is limited94to about 2.1 billion objects and 4 GiB of names. Past either limit the scanner records an issue and drops the95directory rather than overflowing.9697## Treemap9899| Tree | Viewport | Cells produced | Layout time |100|---|---|---|---|101| 467,907 nodes | 1600 by 1000 | 1,798 | 0.162 ms (average of 100 runs) |102103The layout visits only folders whose rectangle is large enough to open and merges invisible children into one104cell, so its cost follows what is visible, not the size of the tree. This measurement is the reason the map is105drawn with SwiftUI `Canvas` and no Metal renderer was written. Drawing time (text and gradients for the cells)106was not measured separately.107108## Duplicate hashing109110| Input | Data hashed | Duration | Threads | Cache |111|---|---|---|---|---|112| Sample folder, scale 1 | 685 MB | 0.069 s | 6 | warm |113114**This is a warm-cache figure.** The sample files had just been written and were served from memory. It shows115that SHA-256 (CryptoKit) is not the bottleneck on this processor; it says nothing about cold reads. On cold116data, hashing speed is bound by the read throughput of the disk, and the engine sets `F_NOCACHE` so that a117large run does not displace other cached data. No cold-cache run was measured.118119## Similar images120121| Input | Images | Duration | Mode |122|---|---|---|---|123| Sample folder, scale 1 | 31 | 0.21 s (about 150 images/s) | fast |124125Fast mode computes a thumbnail and a 64-bit hash per image and runs Vision only for pairs whose hashes are126close. Thorough mode (Vision for every image and every pair) was not timed. Pair comparison is quadratic in127the number of images.128129## Sample folder130131`orix-bench fixture <dir>` at scale 1 writes 182 files totalling 1.89 GB. UI tests use scale 0.25.132133## Reproducing134135```sh136cd Packages/ORIXCleanCore137swift build -c release138B=.build/release/orix-bench139140$B scan ~/Library/Developer 8        # path, optional thread count (0 or omitted = default)141for n in 1 2 3 4 6 8; do $B scan /Applications $n; done142$B scan /System/Library143$B treemap /Applications             # average layout time over 100 runs at 1600 by 1000144$B fixture /tmp/orix-demo/home       # deletes and rewrites the target directory145$B dupes /tmp/orix-demo/home         # optional minimum size in bytes as a third argument146$B similar /tmp/orix-demo/home147```148149Run each command twice and keep the second result to match the warm-cache condition.150151| Command | Output fields |152|---|---|153| `scan` | `files`, `dirs`, `issues`, `seconds`, `objects_per_s`, `cpu_s`, `rss_mb`, `bytes_per_object`, then `logical`, `allocated`, `private`, `sharing` |154| `dupes` | `candidates`, `groups`, `hashed_bytes`, `seconds`, `mb_per_s`, `redundant`, `rss_mb` |155| `similar` | `images`, `groups`, `unreadable`, `seconds`, `images_per_s` |156| `treemap` | `nodes`, `cells`, `layout_ms` |157| `sample <db> [interval]` | records real startup-volume samples into a history database until stopped |158| `fixture <dir> [scale]` | writes the sample folder |159160`scan /` uses `ScanOptions.startupVolume()`, as the application does.161162## Not measured163164- **Intel Macs.** The Release build is universal, but the `x86_64` slice was not run.165- **macOS 15 and macOS 26.** The application is built for 15.0 and later and was run on 27.0 only.166- **External, HDD and network volumes.** Nothing beyond a small APFS disk image was exercised. Seek-bound and167  latency-bound storage may favour a different thread count.168- **Cold-cache scans and cold-cache hashing.**169- **A full-disk scan of the test machine**, in time or memory (see the estimate above).170- **Battery impact** and energy use.171- **Interface responsiveness**: frame times of the map, list scrolling with large results, memory of the172  application process as opposed to `orix-bench`.173- **VoiceOver with a real user** (see `docs/ACCESSIBILITY.md`).174- **Thorough image mode**, and image libraries larger than the sample.175
Files main
Menu

Theme