Archive engine (archive) — compatibility
Package Packages/EngineArchive, module EngineArchive, factory makeEngines(). Built on a bundled, static
libarchive 3.8.11 with liblzma 5.8.4 and libzstd 1.5.7 (EmbeddedEngines/libarchive, 2.2 MB; zlib, bzip2 and iconv
from the macOS SDK; AES / PBKDF2 / digests from CommonCrypto). The system /usr/lib/libarchive is not used.
Extraction to disk is the engine's own code (openat-confined), not libarchive's archive_write_disk.
State on 2026-11-21 (macOS 27.0, arm64): 119 paths as registered by the app (makeEngines()), 33 tests, all
passing (swift test: Test run with 33 tests in 4 suites passed). ArchiveEngine(declaresFileSources: true)
(makeEngines(declaringFileSources: true), used by the tests) declares 1 396: the same 119 plus one creation and
one compression path per catalog format (119 single-file formats today). The parameterised matrix test runs a real
conversion through each of the 1 396.
Formats
| Format | Extract / list / verify | Create | Notes |
|---|---|---|---|
| ZIP | yes (ZIP64; deflate and store tested — the bzip2 / lzma / xz / zstd methods are in the library but untested; ZipCrypto and AES with the password) | yes (deflate or store, ZIP64 when needed, optional AES-256 or ZipCrypto) | names UTF-8; non-UTF-8 names are read as CP437 and reported |
| 7z | yes (LZMA / LZMA2 and store tested — bzip2, deflate, PPMd are in the library but untested; no encrypted 7z) | yes (LZMA2 or store; single-threaded; no encryption) | |
| RAR | experimental — RAR 4 and RAR 5, see below; no encrypted RAR | no (no free encoder exists; never declared) | |
| TAR | yes (ustar, pax, GNU long names) | yes (pax-restricted: ustar unless a field needs pax) | |
| TAR + gzip / bzip2 / xz / zstd | yes | yes | |
| single gzip / bzip2 / xz / zstd stream | yes (one file, named after the archive minus its last extension) | yes (one file) | the stream stores no name |
Declared paths
All fidelity: lossless except verification (partial: the output is a report). Status supported except the 10
paths that read RAR (experimental).
| # | Source | Destination | Operation | Test |
|---|---|---|---|---|
| 12 | zip, 7z, rar, tar, tgz, tbz2, txz, tzst, gz, bz2, xz, zst | folder | archive.extract |
MatrixTests.everyDeclaredPathConvertsAndValidates (fixtures made by other tools, see "Fixtures"), RoundTripTests, SecurityTests |
| 7 | folder (+ any further files / folders: acceptsMultipleSources) |
zip, 7z, tar, tgz, tbz2, txz, tzst | archive.create |
matrix, RoundTripTests.createThenExtractGivesBackTheTree (also read back by unzip / bsdtar) |
| 833 (inflated variant only) | each of the 119 single-file formats of the catalog (+ further files / folders) | zip, 7z, tar, tgz, tbz2, txz, tzst | archive.create |
matrix: one generated file per source format, stored, read back, CRC compared |
| 4 | folder pseudo-source = any single regular file (appliesToAnyFile) |
gz, bz2, xz, zst | archive.compress |
matrix, app self-test |
| 444 (inflated variant only) | each single-file format of the catalog except gz, bz2, xz, zst, tgz, tbz2, txz, tzst (111 formats) | gz, bz2, xz, zst | archive.compress |
matrix: compressed, decompressed, CRC compared with the source |
| 49 | zip, 7z, rar, tar, tgz, tbz2, txz, tzst | each other of zip, 7z, tar, tgz, tbz2, txz, tzst | — (plain conversion) | matrix: converted, then extracted by unzip / bsdtar and compared with the original tree |
| 12 | gz, bz2, xz, zst | each other of the four | — | matrix: decoded by gzip / bzip2 / Homebrew xz / zstd when present, and by the engine |
| 11 | zip, 7z, tar, tgz, tbz2, txz, tzst, gz, bz2, xz, zst | same format | archive.recompress |
matrix |
| 24 | the 12 readable formats | txt, json | archive.verify |
matrix, ListingTests.verificationFindsDamageInsideAnArchive |
How any file reaches an archive without inflating the registry. The app registers the 119-path engine. The 7
folder → archive creation paths accept several sources, and the 4 folder → gz / bz2 / xz / zst compression paths
are marked appliesToAnyFile: the core offers the former for any selection of files and folders and the latter for
any single regular file (ConversionCapabilityResolver.genericOperations), and enqueues the job with
sourceFormat: .folder. A folder handed to a stream compression path is refused with a message naming .tar.gz.
The three usd / usda / usdc → zip paths are declared again (in the inflated variant): the core now accepts a ZIP
whose content looks like a ZIP-based document where a ZIP was announced. An archive holding only empty folders is
extracted (the core accepts a folder with at least one entry of any kind).
About the 1 277 per-format "any file" paths of the inflated variant. Putting a file in an archive, or compressing it as a stream, does not look
at what the file is. The core plans a job from the detected format of the first source, so for a dropped .pdf to
offer "Create ZIP archive" there has to be a pdf → zip path: the engine declares one per catalog format, generated
by a loop over FormatCatalog.all, and the matrix test stores one generated file of each format through each of
them (the content is arbitrary bytes named with the format's extension — the engine never parses it). They inflate
the registry's pair count without representing 1 263 distinct pieces of work; ArchiveEngine(declaresFileSources: false) declares only the 115 other paths, and the app then enqueues loose files with sourceFormat: .folder
(tested: looseFilesCanBeArchivedThroughTheFolderPathWhateverTheEngineDeclares). The lead decides which to register.
A file whose format the core cannot detect at all cannot be planned by the core (not an engine matter); enqueued
with sourceFormat: .folder it is archived fine.
Limitations shown to the user (per path family)
- Extract: entries whose name climbs out with
..are left out; absolute paths, drive letters and invisible characters are cleaned; links pointing outside, devices and FIFOs are not created; owner and group are not restored; set-user-ID, set-group-ID and sticky bits are removed; nothing nested is extracted. ZIP: needs the password for protected entries; a symbolic link whose target is itself encrypted cannot be decoded and is left out. 7z: password-protected archives cannot be opened. Single streams: one file, named after the archive. - Create / convert, by destination: owner and group not stored, set-ID and sticky bits not stored, hard links become separate files (create) or are left out when the destination is not TAR (convert). ZIP: no extended attributes or resource forks; encryption covers contents only — names, sizes and link targets stay readable. 7z: no extended attributes, no encryption, one core. TAR family: extended attributes only on request. Streams: neither name nor permissions.
- RAR (all 10 paths): verified only with stored (uncompressed) RAR 4 and RAR 5 archives; compressed, solid and multi-volume archives untested; encrypted RAR cannot be opened; RAR cannot be created.
- Verify: what the format lets a reader check (see "Integrity verification").
Options
compression.level (store for ZIP and 7z only, fast, normal, maximum; mapped to deflate 1/6/9, LZMA2 1/6/9,
gzip 1/6/9, bzip2 1/6/9, xz 1/6/9, zstd 1/3/19), deterministic, exclude.macos (default on), root.keep,
xattrs (TAR family), symlinks (keep / skip), quarantine (default on), encrypt, encryption (aes256 /
zipcrypto), limit.total.mb, limit.entry.mb, limit.entries, limit.ratio, and password.
password is a secret. The descriptor is an ordinary text option named password (nothing was added to the
contract): the app must mask it, must not write it to the history store, to presets, to workflows or to logs. The
engine never logs it and never puts it in an error or warning (a test checks the messages of wrong-password
errors). It is offered only where it does something: reading ZIP, and writing ZIP with encrypt.
Reproducible archives (deterministic): entries sorted by the UTF-8 bytes of their names (always the case),
modification times set to 1980-01-01 00:00:00 UTC, permissions normalised to 644 / 755, no owner, no gzip
timestamp, single-threaded xz and zstd. Tested for the seven container formats: two trees with the same content
but different dates, modes and creation order give byte-identical archives (and different ones without the
option). ZIP stores DOS local time: the bytes are reproducible on machines in the same time zone, not across time
zones; TAR and 7z are not affected.
Passwords and encryption — what was tested
| Case | Result |
|---|---|
| ZIP AES-256 written by the engine, read by the engine | works (encryptedZipRoundTripsAndRefusesWrongPasswords); non-ASCII password included |
ZIP AES-256 written by the engine, read by system bsdtar --passphrase |
works (ASCII password) |
ZIP ZipCrypto written by the engine, read by the engine and by bsdtar |
works |
ZIP ZipCrypto written by /usr/bin/zip -P, read by the engine |
works, except symbolic links: Info-ZIP encrypts link targets and libarchive returns them undeciphered; such links are left out and reported (symlinks-encrypted) instead of being created with garbage targets |
| wrong password / no password | encryptedInput ("The password is not the right one…" / "…protected by a password. Enter it…"), no partial output |
| encrypted ZIP → other format | decrypted with the password; the result is not encrypted (unless ZIP + encrypt) |
| symbolic links in a ZIP the engine encrypts | their targets are not encrypted by libarchive; said in the path limitations |
| AES ZIP written by another tool (WinZip, 7-Zip) | not tested: no such tool on this Mac. The engine's AES archives are readable by bsdtar (libarchive 3.7.4), which is the only cross-check available |
| encrypted 7z, encrypted RAR | not tested (none could be produced here). libarchive has no decryption for either; its error is mapped to encryptedInput ("uses an encryption this app cannot open") |
No password guessing, no bypass.
Security — every item tested with crafted archives (SecurityTests)
Hostile TAR and ZIP archives are assembled byte by byte in the tests; hostile 7z archives are written through libarchive directly, bypassing the engine. After each extraction the tests compare a snapshot of everything outside the destination before and after.
| Threat | Behaviour | Test |
|---|---|---|
../, a/../../x, ..\..\x |
entry rejected and reported (never "resolved") | pathTraversalNeverLeavesTheDestination (tar, zip, 7z, tgz) |
absolute paths, C:\…, UNC |
root removed, extracted relatively, reported as sanitised | same |
| control characters, bidi overrides (U+202E…), BOM, U+FFFD in names | replaced by _, reported |
same |
| NUL in a name | the C string ends there; what remains is sanitised like any name | namesWithNulBytesNormalisationAndCaseCollisions |
overlong component (> 255 bytes), path > 4 096 bytes, depth > 128, ..namedfork |
entry rejected | pathTraversal…, sanitiserRules |
| names equal after Unicode normalisation or case folding (same file on APFS) | the last entry wins, reported (duplicates-replaced); counts stay exact |
namesWith… |
| file where a folder is, folder where a file is, link over a file, file over a link | the later entry is rejected; nothing is replaced by a different kind of thing | namesWith…, linkAttacksAreBlocked |
| symlink to an absolute path / climbing out, then a file "through" it | the link is not created; the file lands in a real folder inside | linkAttacksAreBlocked |
| symlink that stays inside, then a file through it | the write is refused: every path component is opened with openat(O_NOFOLLOW) |
same |
| links harmless one by one that escape together; a link made dangerous by a later entry | links are re-resolved on the finished tree until stable; those that leave the root are removed | same |
| link to nothing | not kept (its meaning would be decided by whatever is created there later), reported symlinks-dangling |
danglingLinksAreNotKeptAndThatIsReported |
hard link to an absolute path, to ../outside, to a missing entry |
refused: a hard link may only point at a regular file this extraction created | linkAttacksAreBlocked (checks the outside file's link count) |
| devices, FIFOs, sockets | never created, reported | devicesFifosAndSetIDBitsAreNeverCreated |
| set-user-ID / set-group-ID / sticky | removed, reported; no chown ever; files get at least u+rw, folders u+rwx |
same |
| decompression bomb | stops when content exceeds limit.ratio × archive size (default 1000, judged after 32 MB), total (100 GB), per file (50 GB), entry count (1 000 000); 7z / ZIP / RAR are refused before writing when the announced total is over the limit; partial output removed. Also applied to conversion and verification, and to the gzip pre-check |
decompressionBombsStopEarlyAndLeaveNothingBehind (400 MB of zeros in < 200 kB, as tzst, 7z, txz), entryCountAndSizeLimitsApply |
| nested archives | an archive inside an archive is extracted as a file | entryCountAndSizeLimitsApply |
| truncated, damaged, random files | error, nothing left in the work folder; never a crash | truncatedAndCorruptedArchivesFailCleanly (11 formats × 6 kinds of damage), mutatedArchivesNeverCrashOrEscape (150 mutations each of zip, 7z, tar, tgz, rar) |
| disk full | announced size compared with free space before extracting ZIP / 7z / RAR; free space re-checked every 64 MB for all formats (256 MB reserve) | entryCountAndSizeLimitsApply (the check function; a full disk was not simulated) |
| quarantine | if the archive file carries com.apple.quarantine, the same value is set on every extracted file and folder |
quarantineFlagFollowsTheArchive |
| cancellation | checked at every data block; extract / convert / verify / create stop within 2 s on a 2–3 GB job and leave nothing | cancellationIsPromptAndRemovesPartialOutput |
Two bugs of this kind were found by these tests while the engine was written and are fixed: a double free of the libarchive handle when opening failed (crash on any unreadable archive), and a wrong file count when a name repeats.
Not covered: a time-of-check / time-of-use race against another local process modifying the work folder during
extraction is prevented for path components (descriptors, O_NOFOLLOW, O_EXCL), but the final link re-check and
the folder metadata pass use paths below the root descriptor's folder; the work folder is private (mode 0700).
Integrity verification (archive.verify, ArchiveVerifier)
Every file entry is read to the end. Output: a .txt or .json report (per file: path, size, CRC-32 computed,
ok / failed with the reason; totals; what the format allows to check); ArchiveVerifier.verify is the same thing
as a public API. A failed verification is not a failed job: the report is delivered, with a caution warning
and facts["passed"] = "no". A wrong password throws.
| Format | What is checked | Notes |
|---|---|---|
| ZIP | CRC-32 by the decoder and the CRC computed here against the central directory read by the engine's own small reader | independent of libarchive |
| 7z | CRC-32 by the decoder | |
| RAR | CRC-32 by the decoder for compressed files; libarchive does not check stored files, so the engine compares its own CRC with the file headers, read by its own small RAR 4 / RAR 5 header reader (single-volume, unencrypted archives only; otherwise no comparison) | found by test |
| gzip, tar.gz | libarchive does not check the gzip CRC-32 / length trailer (a TODO in its source): a damaged file decompressed "successfully". The engine runs zlib over the file first (also before extracting or converting) |
found by test; costs one extra decompression pass |
| bzip2, xz, zstd (and tar.*) | the stream's own checks, by the decoder | zstd streams written by the engine carry a checksum |
| TAR | header checksums and structure only: TAR stores no checksum of file data, and the report says so |
Listing API (ArchiveBrowser.list) and inspect
ArchiveBrowser.list(url, format:, password:, offset:, limit:, scanLimit:, timeLimit:) returns one page of entries
(index, stored path, kind, size, compressed size — ZIP only —, modification date, encrypted flag, link target,
permissions, whether the name is safe) and a summary of the whole archive (format and filter names, entry / file /
folder / link counts, total size, archive size, compression ratio, encrypted entries, ZIP comment, number of
unsafe names, reader warnings, and whether the scan was complete). The call is stateless: streaming formats are
read again from the start for each page (headers only for ZIP / 7z / RAR / TAR; a compressed TAR must be
decompressed to be listed). inspect uses it with a 5-second / 200 000-entry bound and says when totals are
unknown; an archive that cannot be listed because it is encrypted is reported as encrypted rather than as an
error (code path untested for 7z and RAR: no encrypted one could be produced here).
The uncompressed size of a single stream is not known without decompressing it and is reported as unknown.
Fixtures: who wrote the archives the tests read
| Format | Written by | Read back by (besides the engine) |
|---|---|---|
| zip | /usr/bin/zip, /usr/bin/ditto, hand-assembled |
/usr/bin/unzip, bsdtar |
| tar, tar.gz, tar.bz2, tar.xz | /usr/bin/tar (bsdtar 3.5.3, libarchive 3.7.4), hand-assembled TAR |
/usr/bin/tar |
| 7z | /usr/bin/tar --format 7zip — the same library family as the engine, an older version; there is no other 7z tool on this Mac |
/usr/bin/tar |
| tar.zst, .zst, .xz | Homebrew zstd and xz when installed (they were on this machine: the run printed selfMade=[]); otherwise the engine itself — macOS ships neither command, and the system bsdtar has no zstd of its own |
same |
| gz, bz2 | /usr/bin/gzip, /usr/bin/bzip2 |
same |
| rar | assembled in the test from the published RAR 4 and RAR 5 format descriptions, stored method, with correct header and file CRCs | — |
RAR. No .rar file exists on this Mac and none was downloaded. The two fixtures are built byte by byte
(Craft.rar4, Craft.rar5 in the tests): they list, extract, verify and convert correctly, damage is detected,
and a wrong header CRC is refused. That proves the path works for stored entries only. RAR's real compression
(LZSS / PPMd in RAR 4, the RAR 5 algorithm), solid archives and multi-volume sets are decoded by libarchive's own
code, which these tests never exercise: hence experimental. Encrypted RAR is not supported by libarchive.
Measured
EngineArchiveBench (release build, swift build -c release --product EngineArchiveBench), one process per format
so that the peak memory belongs to that format. Apple M5 Max (18 cores), macOS 27.0, internal SSD, 2026-11-21.
The machine was busy with other sessions during the runs (load average 27–28), so these are lower bounds, and
the single-threaded encoders (ZIP, 7z, gzip) suffered most. Each number is one run.
Tree: 1 032 MB in 2 017 files — 40 % log text, 30 % JSON-lines records with random keys, 30 % random bytes
(incompressible), 2 000 small files. Throughput is MB of uncompressed data per wall-clock second, through the engine
(archive.create from the folder, then archive.extract of the result); validation is not included.
| Format | Level | Archive | Create | Extract | Peak RSS (process) |
|---|---|---|---|---|---|
| zip | normal (deflate 6) | 398.6 MB | 112–123 MB/s (8.4–9.2 s) | 1 377–1 532 MB/s (0.7 s) | 15 MB |
| zip | fast (deflate 1) | 414.4 MB | 163 MB/s (6.3 s) | 1 331 MB/s | 14 MB |
| 7z | normal (LZMA2 6) | 359.5 MB | 2.6 MB/s (400 s) | 241 MB/s (4.3 s) | 106 MB |
| 7z | fast (LZMA2 1) | 378.7 MB | 14 MB/s (74 s) | 342 MB/s (3.0 s) | 22 MB |
| tar.zst | normal (zstd 3, 8 threads) | 397.7 MB | 1 754–2 628 MB/s (0.4–0.6 s) | 934–1 039 MB/s (1.0–1.1 s) | 240 MB |
| tar.xz | normal (xz 6, 4 threads) | 359.7 MB | 8 MB/s (126 s) | 224 MB/s (4.6 s) | 629 MB |
| tar.xz | fast (xz 1, 4 threads) | 378.9 MB | 40 MB/s (26 s) | 224 MB/s (4.6 s) | 83 MB |
| tar.gz | normal (gzip 6) | 398.3 MB | 100–102 MB/s (10 s) | 804–868 MB/s (1.2–1.3 s, includes the gzip checksum pass) | 15 MB |
| tar | — | 1 033.3 MB | 1 766 MB/s (0.6 s) | 2 592 MB/s (0.4 s) | not isolated (this run also generated the tree) |
Reading: ZIP, gzip and TAR stream through a 1 MB buffer and use almost no memory. zstd's 240 MB and xz's 629 MB
are the multi-threaded encoders' buffers (the decoders need far less; the peak shown is the process's, reached
during creation). 7z at Normal is very slow: libarchive's 7z writer runs LZMA2 on one core, and it is the price
of the smallest archive here; the path's limitation text says so, and fast is five times quicker for 5 % more
size. The first tar.gz run showed a 414 MB peak during extraction — the gzip checksum pass was accumulating
temporary buffers; fixed, re-measured at 15 MB.
The whole test suite (1 378 path conversions, round trips, security, listing) runs in 6–10 s.
Evaluated and not implemented
- RAR creation: impossible without RARLAB's proprietary encoder. Never declared.
- 7z and RAR decryption, 7z encryption: not in libarchive. Not implemented separately.
- Multi-volume archives (
.z01,.7z.001,.part1.rar): a single file is read; no volume joining. - Self-extracting archives, XAR / pkg, cpio, ISO, CAB, LHA, ar, lz4, lzip, compress (
.Z): libarchive can read several of these; none is enabled or declared (no catalog format, no test). - AppleDouble merging:
__MACOSX/._nameand._nameentries are ordinary entries for this engine (libarchive's macOS-specific folding is switched off so that nothing disappears silently); by default they are left out as housekeeping and counted, or kept as files withexclude.macosoff. They are never merged back into extended attributes / resource forks. - Extended attributes in ZIP / 7z, ACLs, file flags, creation dates: not stored, not restored.
- Updating an archive in place (add / delete entries), archive comments (written), splitting.
- Per-entry compressed size outside ZIP: libarchive does not expose it.
- Multi-threaded ZIP / 7z compression: libarchive's writers are single-threaded (see the 7z numbers).
- ZIP name encodings other than UTF-8 and CP437 (Shift-JIS, GBK ZIPs from old Windows tools): read as CP437 and reported; no encoding option.
Known defects and things for the lead
-
7z creation writes a temporary file outside the work folder. libarchive's 7z writer spools the compressed stream in an anonymous (already unlinked) file in
$TMPDIR, then copies it into the archive. Nothing remains afterwards, but the space is needed twice during the job and it is not undercontext.workDirectory. It cannot be redirected without changing the process environment. -
Core output check versus ZIP contents.
OutputValidatorrejects a.zipwhose content the detector classifies as a ZIP-based document (USDZ when the first entry is a USD layer; DOCX / XLSX / PPTX when[Content_Types].xmlis present; EPUB / ODF whenmimetypesays so). Archiving such a folder as ZIP fails in the core after the engine succeeded. Fix in the core: accept those detections where.zipis expected. -
Core folder check. A folder output must contain at least one file, so an archive holding only empty folders is reported by the engine as "contains no file to extract" instead of being extracted. And
validateDirectorycan judge a dangling link "outside" (seen when the target path runs through a regular file); the engine no longer produces dangling links, which also avoids this. -
Converting an archive whose entry sizes are unknown (not seen with any fixture; possible for streamed ZIPs read from a pipe) spools that entry in the work folder before writing it to TAR / 7z.
-
Extracting a
.gz/.tar.gzreads the file twice (checksum pass, then extraction). -
Stored ZIP entries carry a data descriptor. A ZIP written with
compression.level = storehas "extended local header" entries; LibreOffice refuses such a package (seen when zipping an OpenDocument tree with Store: "source file could not be loaded"; the same tree zipped with Normal opens). Ordinary unzip tools read both. Items 2 and 3 above are fixed in the core (2026-11-21).