--- project: modelmap document: expH_capture_cost_frontier — analysis (run #1) author: Simon-Pierre Boucher contact: contact@spboucher.ai website: https://modelmap.io created: 2026-08-12 modified: 2026-08-12 status: reviewed --- # Analysis — expH run #1 (storage formats + hook overhead) Run: `results/expH_capture_cost_frontier/20260812T052441Z/results.json` Hardware: Apple M5 Max, 48 GB unified memory, macOS 27.0 · torch 2.13.0 (MPS) · MLX 0.32.0 · Python 3.14.4. Hypothesis registered before the run in `hypothesis.md`; config, seed, commit and full manifest embedded in the JSON. ```text Hypothesis : (A) raw mmap ≥ 2× zarr on random-batch reads (warm); (B) retain-capture < 2× plain inference on both backends; copy+write dominates capture cost. Falsification criterion : (A) dies at ratio < 1.25×; (B) dies at retain > 2× or total > 3×. Method / Baseline : as registered (see hypothesis.md) — 3 repeats/cell, plain-forward baselines, sequential-scan references. Result : (A) CONFIRMED. Random-batch reads: raw-mmap 4.09 GB/s, safetensors(mmap) 4.26, zarr-uncompressed 1.27, zarr-zstd 0.38 → mmap/zarr ratio 3.2× (uncompressed) and 10.8× (zstd), both past the 2× bar. Writes: safetensors 8.10 GB/s (single large write), raw-mmap chunked 2.82, zarr-uncompressed 1.49, zarr-zstd 0.54. Seq scans: 3.9–7.3 GB/s all formats. (B) CONFIRMED, asymmetrically. torch-MPS: plain 46.7 ms/fwd (±1.6) → retain 57.2 (1.22×) → retain+copy+write 68.8 (1.47×). MLX: plain 36.2 (±2.6) → retain 36.9 (1.02×) → +copy+write 40.2 (1.11×). Retain never exceeds 1.25×; totals never exceed 1.5× — far under the 2×/3× kill lines. The "copy+write dominates" sub-claim holds on MLX (+3.3 ms vs +0.7 ms retain) but on torch-MPS the two costs are comparable (+10.5 ms retain, +11.6 ms copy+write) — retain itself is expensive on MPS, matching its eager hook materialization. Bonus observation (not registered, Level 0): MLX baseline is 1.29× faster than torch-MPS on the identical architecture, and MLX capture is nearly free (2%), consistent with lazy-graph retention. Interpretation : Level 0–1. Within-machine replication only (3 repeats/cell, tight variance ≤ ±4 ms); single hardware, single size class, warm cache, synthetic model — all declared in advance. Design decisions this licenses NOW: (1) activation stores are mmap-backed flat files (raw or safetensors container for self-description) — zarr is eliminated for the SAE-shuffle pattern unless cold-cache reverses the ordering; (2) MLX is the preferred capture backend, with torch-MPS as the compatibility path; (3) capture-cost is NOT the bottleneck at this scale — the 4 TB/500M-token storage wall (notes §4.9) remains the real constraint. Next experiment : run #2 = cold-cache pass (purge/F_NOCACHE) to test whether the mmap>zarr ordering survives; run #3 = same capture modes on a real 0.5B checkpoint via mlx-lm vs transformers-MPS; then replicate the whole grid on ≥2 cluster Macs (M3U96a, M2U64) to promote the cost table toward Level 1 cross-hardware. ``` ## Notes - zarr numbers use chunk = one write-block (4096×4096); different chunk geometries could narrow the gap — a registered caveat, not tested here. - The torch-MPS retain overhead (22%) is per-layer-output retention of ~100 MB/forward; capture pipelines that subsample layers will scale it down roughly linearly. - All timing cells and raw seconds are in the results JSON with the hardware manifest; the figure-ready aggregation is deferred to the atlas pipeline. --- # Analysis — expH run #2 (cold cache, second hardware) — **HYPOTHESIS FALSIFIED** Run: `results/expH_capture_cost_frontier/*_run2_M3U96a/results.json` Hardware: Apple M3 Ultra (Mac Studio, MacLustr M3U96a), 96 GB, macOS. Store enlarged to 200k×4096 fp16 (~1.6 GB); `sudo purge` before **every** timed read repetition. Hypothesis registered before the run. ```text Hypothesis : warm ordering survives cold — mmap ≥ 1.5× zarr on random-batch reads. Falsification criterion : dies if any zarr variant beats mmap (ratio <1.25×). Result : FALSIFIED — the ordering INVERTS cold. Random-batch (cold): zarr-uncompressed 0.62 GB/s, safetensors(mmap) 0.32, zarr-zstd 0.19, raw-mmap 0.14 → zarr-uncompressed beats mmap 4.4×. Sequential (cold): zarr-uncompressed 3.47, zarr-zstd 3.18, raw-mmap 0.64, safetensors 0.53. Interpretation : Level 0–1, and the mechanism is legible: cold mmap reads are page-fault-driven — effectively ~8–16 KiB random reads at queue depth 1, which localvm's cold-cache SSD grid already showed run at ~0.07–0.14 GB/s. zarr instead read()s whole 32 MiB chunks, exactly the ≥256 KiB / high-QD regime where Apple NVMe delivers. The warm-cache run #1 numbers measured memory, not disk. Cross-project agreement (modelmap run #2 ↔ localvm expH) is strong convergent evidence for the real rule: ***IO granularity decides, not the container.*** Design decision REVISED: for stores larger than RAM, either use chunked containers or access mmap in chunk-aligned batched reads; never row-level random access. Caveat: run #1 was M5 Max, run #2 M3 Ultra — the warm-vs-cold contrast within run #2's own machine is what carries the claim; hardware confound noted for the cross-run comparison. Next experiment : run #4 (registered idea, not yet run): mmap with chunk-sorted, ≥1 MiB batched reads + F_RDAHEAD vs zarr-uncompressed, cold, same machine — decides the final store design. Also replicate run #2 on the M5 Max laptop for the hardware axis. ``` --- # Analysis — expH run #3 (real 4-bit checkpoint via mlx-lm) — CONFIRMED Run: `results/expH_capture_cost_frontier/20260812T061742Z/results.json` Model: mlx-community/Qwen3-0.6B-4bit (28 layers, d_model 1024), prefill of a 901-token prompt, M5 Max 48 GB. Hypothesis registered before the run. ```text Hypothesis : retain ≤ 1.15× plain prefill; retain+write ≤ 1.5×. Result : CONFIRMED, stronger than predicted. plain 28.5 ms/prefill (31,644 tok/s) → retain 28.6 ms (1.004× — capture is FREE) → retain+copy+write 36.4 ms (1.28×, 24,733 tok/s). Interpretation : Level 0–1 (3 repeats, single machine/model). The synthetic run #1 MLX result (1.02×) transfers to a real quantized checkpoint. This run is also the project's first QUANTIZED-model activation capture in Python — the capability the Phase 1 survey found nowhere (notes §4.9). Engineering note: quantized embeddings pack weight shapes, so d_model must be inferred from a runtime activation, and bf16 activations must be cast in MLX before NumPy conversion (numpy has no bfloat16). Next experiment : same three modes at 1.7B/4B to trace the overhead curve vs model size; then wire this Tap pattern into src/modelmap/capture as the standard MLX capture layer (expA depends on it). ```