SPB Git
spboucher.ai

spb/wp33_uqo

Public MIT

Energy analysis script, README status and headline table

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 1, 2026) parent 7c1fb41 Browse files

6 changed files +150 −4

6 changed files
Unified Split
modified README.md +12 −2
@@ -13,8 +13,14 @@
13 13 - Live cluster inventory: 21 reachable Macs (M1–M6, 8–96 GB), **1 GbE everywhere**, no active Thunderbolt link → distributed gates use TCP; RDMA/JACCL blocked pending cabling.
14 14
15 15 ## Status
16 −Version 0.1 (2026-11-01), **protocol + preliminary** working paper; not submitted. Gates A–B complete, Gate C in progress, D–G pending
17 −(see `research/LOG.md`). Nothing in the paper is reported without a raw log under `benchmarks/raw/`.
16 +Version 0.1 (2026-11-01), **preliminary** working paper; not submitted. Gates A–E complete, Gate F (medium model, two nodes) and the
17 +energy study running at the time of writing, Gate G audit done for the existing logs (see `research/LOG.md`). Nothing in the paper
18 +is reported without a raw log under `benchmarks/raw/`; `artifacts/manifest.json` lists every artifact with its SHA-256.
19 +
20 +**Verdicts so far (all with bootstrap CIs, `benchmarks/processed/audit.json`):** H1 not supported on OLMoE/laptop NVMe (best predictor ties:
21 +the install path, not prediction, is the bottleneck); H4 not supported (decode at batch 1 is launch-bound; fusion ±2 %); the remote
22 +tier over 1 GbE is a capacity mechanism, not a latency one (−66 tok/s for odd-expert placement); the expert-contiguous storage layout
23 +is the one clear win (cold NVMe 8 → 36 tok/s).
18 24
19 25 ## What this paper does
20 26 Designs, implements and measures TITAN-MoE against five pre-stated hypotheses (causal prefetch, topology-aware placement,
@@ -29,6 +35,10 @@ constraint: placement decides speed, never which experts run.
29 35 | TITAN, 32 slots, LRU, demand loading (page cache) | 56–80 | 0.73–0.81 | same (before lazy-install fix) |
30 36 | TITAN, 16 slots, LRU, demand loading (page cache) | 51–73 | 0.48–0.67 | `benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl` |
31 37 | TITAN, 16 slots + transition prefetch (d = 1, m = 8) | 35–56 | 0.63–0.73 | same |
38 +| TITAN, 16 slots, cold NVMe, contiguous layout, no prefetch | 36.4 | 0.53 | `benchmarks/raw/gateC/cold_nvme_laptop/sweep.jsonl` |
39 +| TITAN, 16 slots, cold NVMe + linear probe (recall@8 0.76) | 31.8 | 0.78 | `.../attribution.jsonl` |
40 +| Qwen3-30B-A3B on M3U96a: mlx-lm / TITAN all-resident | 121.7 / 121.6 | — | `benchmarks/raw/gateC/qwen3_m3u96a/baseline.jsonl` |
41 +| OLMoE two nodes (M3U96a + M3U96b, 1 GbE), 32 slots: local / odd experts remote | 81 / 15 | — | `benchmarks/raw/gateE/two_node_m3u96a_m3u96b.jsonl` |
32 42
33 43 ## Repository layout
34 44 ```
added benchmarks/raw/gateG/energy_laptop/bench.jsonl +32 −0
@@ -0,0 +1,32 @@
1 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:25:06", "prompt_tokens": 24, "gen_tokens": 256, "ttft_ms": 188.12183407135308, "prompt_tps": 235.37786372458933, "gen_tps": 470.4318210844757, "decode_tps_wall": 468.6594703396594, "itl_ms_p50": 2.112540998496115, "itl_ms_p95": 2.2170000011101365, "itl_ms_p99": 2.5747499894350767, "itl_ms_mean": 2.1337454234633375, "total_s": 0.7322269170545042, "peak_mem_gb": 3.96874552, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "t_start_epoch": 1793514305.306269, "t_end_epoch": 1793514306.040073, "t_first_epoch": 1793514305.4943907}
2 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:25:06", "prompt_tokens": 18, "gen_tokens": 256, "ttft_ms": 16.4172500371933, "prompt_tps": 1121.8069190605863, "gen_tps": 466.3281340508769, "decode_tps_wall": 464.5175390868041, "itl_ms_p50": 2.1014580270275474, "itl_ms_p95": 2.209290978498757, "itl_ms_p99": 2.349124988541007, "itl_ms_mean": 2.1527712429672774, "total_s": 0.565373916993849, "peak_mem_gb": 3.968181484, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "t_start_epoch": 1793514306.0402071, "t_end_epoch": 1793514306.607169, "t_first_epoch": 1793514306.0566244}
3 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:25:07", "prompt_tokens": 14, "gen_tokens": 256, "ttft_ms": 15.293208067305386, "prompt_tps": 937.2384934901097, "gen_tps": 463.60241928593786, "decode_tps_wall": 461.8049920430233, "itl_ms_p50": 2.15233291964978, "itl_ms_p95": 2.2360829170793295, "itl_ms_p99": 2.5505420053377748, "itl_ms_mean": 2.165416176156963, "total_s": 0.5674743329873309, "peak_mem_gb": 3.968181844, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "t_start_epoch": 1793514306.607254, "t_end_epoch": 1793514307.1763232, "t_first_epoch": 1793514306.6225471}
4 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:25:07", "prompt_tokens": 19, "gen_tokens": 256, "ttft_ms": 17.32750004157424, "prompt_tps": 1119.7713283980459, "gen_tps": 441.9768996846985, "decode_tps_wall": 440.2591374342207, "itl_ms_p50": 2.1481250878423452, "itl_ms_p95": 3.0132500687614083, "itl_ms_p99": 3.9645839715376496, "itl_ms_mean": 2.271389540777925, "total_s": 0.5965318329399452, "peak_mem_gb": 3.968173196, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array in descending ord", "t_start_epoch": 1793514307.176417, "t_end_epoch": 1793514307.773642, "t_first_epoch": 1793514307.1937447}
5 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:25:08", "prompt_tokens": 18, "gen_tokens": 256, "ttft_ms": 18.827666994184256, "prompt_tps": 984.7074360217741, "gen_tps": 461.9654113835424, "decode_tps_wall": 460.17860925704383, "itl_ms_p50": 2.1511659724637866, "itl_ms_p95": 2.315332996658981, "itl_ms_p99": 2.8839580481871963, "itl_ms_mean": 2.17306928197835, "total_s": 0.5729603338986635, "peak_mem_gb": 3.96607196, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "t_start_epoch": 1793514307.7737489, "t_end_epoch": 1793514308.348369, "t_first_epoch": 1793514307.7925766}
6 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:25:08", "prompt_tokens": 14, "gen_tokens": 256, "ttft_ms": 14.978124992921948, "prompt_tps": 956.5400847156083, "gen_tps": 464.05664489251785, "decode_tps_wall": 462.2590770176442, "itl_ms_p50": 2.144083962775767, "itl_ms_p95": 2.2526669781655073, "itl_ms_p99": 2.5600000517442822, "itl_ms_mean": 2.163289050918584, "total_s": 0.5666168329771608, "peak_mem_gb": 3.968480906, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "t_start_epoch": 1793514308.348459, "t_end_epoch": 1793514308.9166908, "t_first_epoch": 1793514308.3634372}
7 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:25:09", "prompt_tokens": 19, "gen_tokens": 256, "ttft_ms": 16.884125070646405, "prompt_tps": 1149.1501371051957, "gen_tps": 457.21528412730925, "decode_tps_wall": 455.43643758431836, "itl_ms_p50": 2.146625076420605, "itl_ms_p95": 2.2882079938426614, "itl_ms_p99": 2.655582968145609, "itl_ms_mean": 2.1956960784782673, "total_s": 0.5767866250826046, "peak_mem_gb": 3.972462288, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array in descending ord", "t_start_epoch": 1793514308.916773, "t_end_epoch": 1793514309.4952042, "t_first_epoch": 1793514308.9336572}
8 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:25:10", "prompt_tokens": 24, "gen_tokens": 256, "ttft_ms": 19.740415969863534, "prompt_tps": 1239.560065911956, "gen_tps": 454.95511202383386, "decode_tps_wall": 453.1933131077652, "itl_ms_p50": 2.148041967302561, "itl_ms_p95": 2.5877500884234905, "itl_ms_p99": 3.46941698808223, "itl_ms_mean": 2.2065638902359734, "total_s": 0.5824142079800367, "peak_mem_gb": 3.9681691, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "t_start_epoch": 1793514309.495293, "t_end_epoch": 1793514310.079181, "t_first_epoch": 1793514309.5150332}
9 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:25:10", "prompt_tokens": 18, "gen_tokens": 256, "ttft_ms": 17.16316700913012, "prompt_tps": 1069.8047641670712, "gen_tps": 467.71559575039373, "decode_tps_wall": 465.89695236670747, "itl_ms_p50": 2.133708097971976, "itl_ms_p95": 2.2149579599499702, "itl_ms_p99": 2.4115419946610928, "itl_ms_mean": 2.146397384486216, "total_s": 0.5644945000531152, "peak_mem_gb": 3.96848097, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "t_start_epoch": 1793514310.0792608, "t_end_epoch": 1793514310.645349, "t_first_epoch": 1793514310.096424}
10 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:25:11", "prompt_tokens": 24, "gen_tokens": 256, "ttft_ms": 19.290167023427784, "prompt_tps": 1263.942342994187, "gen_tps": 458.9146981283141, "decode_tps_wall": 457.1315544926368, "itl_ms_p50": 2.1307909628376365, "itl_ms_p95": 2.216291963122785, "itl_ms_p99": 2.4344170233234763, "itl_ms_mean": 2.1875540862846026, "total_s": 0.5771164590260014, "peak_mem_gb": 3.96903448, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "t_start_epoch": 1793514310.6454232, "t_end_epoch": 1793514311.224113, "t_first_epoch": 1793514310.6647134}
11 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:25:11", "prompt_tokens": 14, "gen_tokens": 256, "ttft_ms": 14.892042032442987, "prompt_tps": 960.5873066247185, "gen_tps": 466.69212812497847, "decode_tps_wall": 464.8837312732684, "itl_ms_p50": 2.134124981239438, "itl_ms_p95": 2.2295420058071613, "itl_ms_p99": 2.934625023044646, "itl_ms_mean": 2.151075489910356, "total_s": 0.5634162919595838, "peak_mem_gb": 3.970299104, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "t_start_epoch": 1793514311.2242, "t_end_epoch": 1793514311.7892108, "t_first_epoch": 1793514311.239092}
12 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:25:12", "prompt_tokens": 19, "gen_tokens": 256, "ttft_ms": 16.762166982516646, "prompt_tps": 1153.9220930435374, "gen_tps": 468.6168932968543, "decode_tps_wall": 466.7935856987825, "itl_ms_p50": 2.1224169759079814, "itl_ms_p95": 2.2076249588280916, "itl_ms_p99": 2.8056249720975757, "itl_ms_mean": 2.142274509841467, "total_s": 0.5630421669920906, "peak_mem_gb": 3.972462288, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array in descending ord", "t_start_epoch": 1793514311.789291, "t_end_epoch": 1793514312.353961, "t_first_epoch": 1793514311.8060532}
13 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 2, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:25:12", "prompt_tokens": 14, "gen_tokens": 256, "ttft_ms": 14.997999998740852, "prompt_tps": 953.1535891574458, "gen_tps": 457.16755235425677, "decode_tps_wall": 455.39228044100975, "itl_ms_p50": 2.1347920410335064, "itl_ms_p95": 2.2339170100167394, "itl_ms_p99": 2.674165996722877, "itl_ms_mean": 2.1959089842971924, "total_s": 0.574954790994525, "peak_mem_gb": 3.966071884, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "t_start_epoch": 1793514312.354039, "t_end_epoch": 1793514312.930638, "t_first_epoch": 1793514312.369037}
14 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 2, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:25:13", "prompt_tokens": 18, "gen_tokens": 256, "ttft_ms": 16.436457983218133, "prompt_tps": 1115.8523078335347, "gen_tps": 467.1782922784267, "decode_tps_wall": 465.3530578443966, "itl_ms_p50": 2.135958056896925, "itl_ms_p95": 2.2165830014273524, "itl_ms_p99": 2.3358339676633477, "itl_ms_mean": 2.148906047017697, "total_s": 0.5644074999727309, "peak_mem_gb": 3.9681691, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "t_start_epoch": 1793514312.930718, "t_end_epoch": 1793514313.4966679, "t_first_epoch": 1793514312.9471545}
15 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 2, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:25:14", "prompt_tokens": 19, "gen_tokens": 256, "ttft_ms": 16.946540912613273, "prompt_tps": 1144.5438406020976, "gen_tps": 456.19956920266856, "decode_tps_wall": 454.43275762925106, "itl_ms_p50": 2.1413330687209964, "itl_ms_p95": 2.241290989331901, "itl_ms_p99": 2.6564159197732806, "itl_ms_mean": 2.200545588343897, "total_s": 0.5780856659403071, "peak_mem_gb": 3.966071948, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array in descending ord", "t_start_epoch": 1793514313.496758, "t_end_epoch": 1793514314.076493, "t_first_epoch": 1793514313.5137045}
16 +{"backend": "mlx_lm", "capacity": null, "policy": null, "predictor": null, "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 2, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:25:14", "prompt_tokens": 24, "gen_tokens": 256, "ttft_ms": 18.928042030893266, "prompt_tps": 1289.9007274930318, "gen_tps": 468.3260225667729, "decode_tps_wall": 466.507967877797, "itl_ms_p50": 2.1314169280231, "itl_ms_p95": 2.211374929174781, "itl_ms_p99": 2.265332965180278, "itl_ms_mean": 2.1435861096845246, "total_s": 0.565542500000447, "peak_mem_gb": 3.968936176, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "t_start_epoch": 1793514314.076572, "t_end_epoch": 1793514314.643809, "t_first_epoch": 1793514314.0955}
17 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:25:23", "prompt_tokens": 24, "gen_tokens": 256, "ttft_ms": 595.1724579790607, "prompt_tps": 93.17831212423292, "gen_tps": 450.5807717575956, "decode_tps_wall": 448.8982520941604, "itl_ms_p50": 2.2037079324945807, "itl_ms_p95": 2.3027079878374934, "itl_ms_p99": 2.6057499926537275, "itl_ms_mean": 2.2276763060111917, "total_s": 1.1632299160119146, "peak_mem_gb": 7.517514916, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "t_start_epoch": 1793514322.462275, "t_end_epoch": 1793514323.6272619, "t_first_epoch": 1793514323.0574474, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35840, "stall_s": 0.0, "hits_device": 35840, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
18 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:25:24", "prompt_tokens": 18, "gen_tokens": 256, "ttft_ms": 18.090958008542657, "prompt_tps": 1019.7939183808035, "gen_tps": 458.06426813974963, "decode_tps_wall": 456.28107786330935, "itl_ms_p50": 2.187499892897904, "itl_ms_p95": 2.273250021971762, "itl_ms_p99": 2.4198340252041817, "itl_ms_mean": 2.191631537040367, "total_s": 0.5769569999538362, "peak_mem_gb": 3.976689572, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "t_start_epoch": 1793514323.627431, "t_end_epoch": 1793514324.206025, "t_first_epoch": 1793514323.6455219, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35072, "stall_s": 0.0, "hits_device": 35072, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
19 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:25:24", "prompt_tokens": 14, "gen_tokens": 256, "ttft_ms": 14.778750017285347, "prompt_tps": 968.9307506542386, "gen_tps": 470.7078236129177, "decode_tps_wall": 468.88837558741795, "itl_ms_p50": 2.12320894934237, "itl_ms_p95": 2.2106250980868936, "itl_ms_p99": 2.3337500169873238, "itl_ms_mean": 2.1327037565117957, "total_s": 0.5586182079277933, "peak_mem_gb": 3.978955704, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "t_start_epoch": 1793514324.2061288, "t_end_epoch": 1793514324.7663832, "t_first_epoch": 1793514324.2209077, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 34560, "stall_s": 0.0, "hits_device": 34560, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
20 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:25:25", "prompt_tokens": 19, "gen_tokens": 256, "ttft_ms": 17.6326249493286, "prompt_tps": 1095.9983784392155, "gen_tps": 471.51769111164896, "decode_tps_wall": 469.6899921301727, "itl_ms_p50": 2.11891601793468, "itl_ms_p95": 2.209958038292825, "itl_ms_p99": 2.3376670433208346, "itl_ms_mean": 2.1290638862981224, "total_s": 0.5605439159553498, "peak_mem_gb": 3.976772116, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array in descending ord", "t_start_epoch": 1793514324.7664902, "t_end_epoch": 1793514325.328651, "t_first_epoch": 1793514324.784123, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35200, "stall_s": 0.0, "hits_device": 35200, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
21 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:25:25", "prompt_tokens": 18, "gen_tokens": 256, "ttft_ms": 16.823415993712842, "prompt_tps": 1087.1808515492182, "gen_tps": 461.21673178735455, "decode_tps_wall": 459.4311758689545, "itl_ms_p50": 2.1194169530645013, "itl_ms_p95": 2.246459014713764, "itl_ms_p99": 2.3375418968498707, "itl_ms_mean": 2.1766045765366915, "total_s": 0.5718575830105692, "peak_mem_gb": 3.968251684, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "t_start_epoch": 1793514325.328746, "t_end_epoch": 1793514325.902508, "t_first_epoch": 1793514325.3455696, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35072, "stall_s": 0.0, "hits_device": 35072, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
22 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:25:26", "prompt_tokens": 14, "gen_tokens": 256, "ttft_ms": 16.51325006969273, "prompt_tps": 864.7891649966291, "gen_tps": 450.47356937470596, "decode_tps_wall": 448.7347988533801, "itl_ms_p50": 2.2017499431967735, "itl_ms_p95": 2.314708079211414, "itl_ms_p99": 2.6200830470770597, "itl_ms_mean": 2.2284877450004514, "total_s": 0.5847776250448078, "peak_mem_gb": 3.968173196, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "t_start_epoch": 1793514325.902622, "t_end_epoch": 1793514326.489092, "t_first_epoch": 1793514325.9191353, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 34560, "stall_s": 0.0, "hits_device": 34560, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
23 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:25:27", "prompt_tokens": 19, "gen_tokens": 256, "ttft_ms": 18.321583978831768, "prompt_tps": 1055.5238367339955, "gen_tps": 456.1814140653704, "decode_tps_wall": 454.41450373834437, "itl_ms_p50": 2.187666017562151, "itl_ms_p95": 2.2581249941140413, "itl_ms_p99": 2.3323330096900463, "itl_ms_mean": 2.2006339845521485, "total_s": 0.5794832500396296, "peak_mem_gb": 3.96817326, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array in descending ord", "t_start_epoch": 1793514326.489198, "t_end_epoch": 1793514327.070356, "t_first_epoch": 1793514326.5075195, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35200, "stall_s": 0.0, "hits_device": 35200, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
24 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:25:27", "prompt_tokens": 24, "gen_tokens": 256, "ttft_ms": 20.011833985336125, "prompt_tps": 1216.9305429183698, "gen_tps": 466.0705356397449, "decode_tps_wall": 464.2588224316587, "itl_ms_p50": 2.1361669059842825, "itl_ms_p95": 2.2514170268550515, "itl_ms_p99": 2.2703330032527447, "itl_ms_mean": 2.15397091381544, "total_s": 0.5692744170082733, "peak_mem_gb": 3.968485162, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "t_start_epoch": 1793514327.070466, "t_end_epoch": 1793514327.6414561, "t_first_epoch": 1793514327.090478, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35840, "stall_s": 0.0, "hits_device": 35840, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
25 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:25:28", "prompt_tokens": 18, "gen_tokens": 256, "ttft_ms": 18.294042092747986, "prompt_tps": 1001.8808681486292, "gen_tps": 464.40025140517963, "decode_tps_wall": 462.60461575074055, "itl_ms_p50": 2.1379999816417694, "itl_ms_p95": 2.2337499540299177, "itl_ms_p99": 2.3762499913573265, "itl_ms_mean": 2.1616731998601963, "total_s": 0.5695207080570981, "peak_mem_gb": 3.970369328, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "t_start_epoch": 1793514327.641553, "t_end_epoch": 1793514328.2128031, "t_first_epoch": 1793514327.659847, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35072, "stall_s": 0.0, "hits_device": 35072, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
26 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:25:28", "prompt_tokens": 24, "gen_tokens": 256, "ttft_ms": 21.27758297137916, "prompt_tps": 1145.1178739580555, "gen_tps": 462.03186883030514, "decode_tps_wall": 460.2449167497562, "itl_ms_p50": 2.1430000197142363, "itl_ms_p95": 2.2282080026343465, "itl_ms_p99": 2.303167013451457, "itl_ms_mean": 2.172756207851218, "total_s": 0.5753304159734398, "peak_mem_gb": 3.96817326, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "t_start_epoch": 1793514328.212903, "t_end_epoch": 1793514328.790012, "t_first_epoch": 1793514328.2341807, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35840, "stall_s": 0.0, "hits_device": 35840, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
27 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:25:29", "prompt_tokens": 14, "gen_tokens": 256, "ttft_ms": 15.425458084791899, "prompt_tps": 925.0897329873465, "gen_tps": 463.4110778789221, "decode_tps_wall": 461.6119853386972, "itl_ms_p50": 2.140500000678003, "itl_ms_p95": 2.2345000179484487, "itl_ms_p99": 2.3074999917298555, "itl_ms_mean": 2.166321568245835, "total_s": 0.5678374579874799, "peak_mem_gb": 3.968173196, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "t_start_epoch": 1793514328.790124, "t_end_epoch": 1793514329.35961, "t_first_epoch": 1793514328.8055494, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 34560, "stall_s": 0.0, "hits_device": 34560, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
28 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:25:29", "prompt_tokens": 19, "gen_tokens": 256, "ttft_ms": 18.710667034611106, "prompt_tps": 1031.9426998024403, "gen_tps": 463.21106015100696, "decode_tps_wall": 461.4111730331186, "itl_ms_p50": 2.143666031770408, "itl_ms_p95": 2.22250004298985, "itl_ms_p99": 2.319999970495701, "itl_ms_mean": 2.167264380327919, "total_s": 0.5713630840182304, "peak_mem_gb": 3.96817326, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array in descending ord", "t_start_epoch": 1793514329.3597162, "t_end_epoch": 1793514329.932776, "t_first_epoch": 1793514329.3784268, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35200, "stall_s": 0.0, "hits_device": 35200, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
29 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 2, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:25:30", "prompt_tokens": 14, "gen_tokens": 256, "ttft_ms": 15.392875066027045, "prompt_tps": 928.2486308730157, "gen_tps": 462.22972249169266, "decode_tps_wall": 460.43820176684613, "itl_ms_p50": 2.1432500798255205, "itl_ms_p95": 2.242750022560358, "itl_ms_p99": 2.3874170146882534, "itl_ms_mean": 2.1718441175442993, "total_s": 0.5692131250398234, "peak_mem_gb": 3.966076044, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "t_start_epoch": 1793514329.932936, "t_end_epoch": 1793514330.50381, "t_first_epoch": 1793514329.9483287, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 34560, "stall_s": 0.0, "hits_device": 34560, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
30 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 2, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:25:31", "prompt_tokens": 18, "gen_tokens": 256, "ttft_ms": 18.278000061400235, "prompt_tps": 1000.6925920853079, "gen_tps": 461.85046685233317, "decode_tps_wall": 460.05774178287794, "itl_ms_p50": 2.151958062313497, "itl_ms_p95": 2.234625048004091, "itl_ms_p99": 2.319667022675276, "itl_ms_mean": 2.173640195955979, "total_s": 0.5725562500301749, "peak_mem_gb": 3.96817326, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "t_start_epoch": 1793514330.503915, "t_end_epoch": 1793514331.078241, "t_first_epoch": 1793514330.522193, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35072, "stall_s": 0.0, "hits_device": 35072, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
31 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 2, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:25:31", "prompt_tokens": 19, "gen_tokens": 256, "ttft_ms": 18.404333968646824, "prompt_tps": 1049.8107592380431, "gen_tps": 460.615534664283, "decode_tps_wall": 458.8322514676528, "itl_ms_p50": 2.1516670240089297, "itl_ms_p95": 2.2910829866304994, "itl_ms_p99": 2.730917069129646, "itl_ms_mean": 2.1794457490756813, "total_s": 0.5741629999829456, "peak_mem_gb": 3.968485162, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array in descending ord", "t_start_epoch": 1793514331.0783691, "t_end_epoch": 1793514331.654183, "t_first_epoch": 1793514331.0967734, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35200, "stall_s": 0.0, "hits_device": 35200, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
32 +{"backend": "titan", "capacity": 64, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "fused_upgate": false, "nocache": false, "max_tokens": 256, "split": "heldout", "tag": "energy", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out benchmarks/raw/gateG/energy_laptop/bench.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "7c1fb41"}, "seed": 0, "warmup": false, "repeat": 2, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:25:32", "prompt_tokens": 24, "gen_tokens": 256, "ttft_ms": 20.47570899594575, "prompt_tps": 1189.8515750321321, "gen_tps": 459.3398270654975, "decode_tps_wall": 457.5545624763554, "itl_ms_p50": 2.160334028303623, "itl_ms_p95": 2.314708079211414, "itl_ms_p99": 2.6427090633660555, "itl_ms_mean": 2.185531698313414, "total_s": 0.5777862920658663, "peak_mem_gb": 3.96819798, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "t_start_epoch": 1793514331.654289, "t_end_epoch": 1793514332.23374, "t_first_epoch": 1793514331.6747646, "cache_hit_rate": 1.0, "cache_misses": 0, "cache_hits": 35840, "stall_s": 0.0, "hits_device": 35840, "layers_complete": 16, "prefetch": null, "host_time_s": {"sync": 0.0, "prefetch": 0.0, "ensure": 0.0, "build": 0.0}, "ssd_bytes": 0, "ssd_read_s": 0.0}
added benchmarks/raw/gateG/energy_laptop/power.jsonl +38 −0
@@ -0,0 +1,38 @@
1 +{"t": 1793514296.549512, "elapsed_ns": 1020739166, "cpu_mW": 13782.2, "gpu_mW": 173.404, "ane_mW": 0.0, "combined_mW": 13955.6, "thermal_pressure": "Nominal"}
2 +{"t": 1793514297.5711622, "elapsed_ns": 1021713416, "cpu_mW": 11452.3, "gpu_mW": 42.0862, "ane_mW": 0.0, "combined_mW": 11494.4, "thermal_pressure": "Nominal"}
3 +{"t": 1793514298.592243, "elapsed_ns": 1021348208, "cpu_mW": 10512.6, "gpu_mW": 46.0176, "ane_mW": 0.0, "combined_mW": 10558.6, "thermal_pressure": "Nominal"}
4 +{"t": 1793514299.6141, "elapsed_ns": 1021586208, "cpu_mW": 13192.2, "gpu_mW": 37.1971, "ane_mW": 0.0, "combined_mW": 13229.4, "thermal_pressure": "Nominal"}
5 +{"t": 1793514300.635915, "elapsed_ns": 1021119458, "cpu_mW": 11902.6, "gpu_mW": 33.2968, "ane_mW": 0.0, "combined_mW": 11935.9, "thermal_pressure": "Nominal"}
6 +{"t": 1793514301.656907, "elapsed_ns": 1021850083, "cpu_mW": 30909.6, "gpu_mW": 62.6315, "ane_mW": 0.0, "combined_mW": 30972.3, "thermal_pressure": "Nominal"}
7 +{"t": 1793514302.6781352, "elapsed_ns": 1021119541, "cpu_mW": 15155.9, "gpu_mW": 35.2554, "ane_mW": 0.0, "combined_mW": 15191.2, "thermal_pressure": "Nominal"}
8 +{"t": 1793514304.720781, "elapsed_ns": 1021519166, "cpu_mW": 14041.8, "gpu_mW": 22.5155, "ane_mW": 0.0, "combined_mW": 14064.3, "thermal_pressure": "Nominal"}
9 +{"t": 1793514304.722431, "elapsed_ns": 1020628625, "cpu_mW": 13774.8, "gpu_mW": 88.1809, "ane_mW": 0.0, "combined_mW": 13863.0, "thermal_pressure": "Nominal"}
10 +{"t": 1793514305.742136, "elapsed_ns": 1022123666, "cpu_mW": 14294.7, "gpu_mW": 294.485, "ane_mW": 0.0, "combined_mW": 14589.2, "thermal_pressure": "Nominal"}
11 +{"t": 1793514306.763233, "elapsed_ns": 1021290749, "cpu_mW": 14808.7, "gpu_mW": 7506.19, "ane_mW": 0.0, "combined_mW": 22314.9, "thermal_pressure": "Nominal"}
12 +{"t": 1793514307.785113, "elapsed_ns": 1021253166, "cpu_mW": 13126.0, "gpu_mW": 158.629, "ane_mW": 0.0, "combined_mW": 13284.7, "thermal_pressure": "Nominal"}
13 +{"t": 1793514308.805221, "elapsed_ns": 1020404875, "cpu_mW": 18136.9, "gpu_mW": 57959.3, "ane_mW": 0.0, "combined_mW": 76096.3, "thermal_pressure": "Nominal"}
14 +{"t": 1793514309.8271391, "elapsed_ns": 1021151874, "cpu_mW": 18217.7, "gpu_mW": 29507.9, "ane_mW": 0.0, "combined_mW": 47725.5, "thermal_pressure": "Nominal"}
15 +{"t": 1793514310.8490849, "elapsed_ns": 1021604083, "cpu_mW": 17000.7, "gpu_mW": 29584.8, "ane_mW": 0.0, "combined_mW": 46585.6, "thermal_pressure": "Nominal"}
16 +{"t": 1793514312.8931289, "elapsed_ns": 1021821625, "cpu_mW": 17463.9, "gpu_mW": 30285.1, "ane_mW": 0.0, "combined_mW": 47749.0, "thermal_pressure": "Nominal"}
17 +{"t": 1793514312.895048, "elapsed_ns": 1021975708, "cpu_mW": 17156.0, "gpu_mW": 29919.5, "ane_mW": 0.0, "combined_mW": 47075.5, "thermal_pressure": "Nominal"}
18 +{"t": 1793514313.913878, "elapsed_ns": 1021681250, "cpu_mW": 16979.9, "gpu_mW": 167.371, "ane_mW": 0.0, "combined_mW": 17147.2, "thermal_pressure": "Nominal"}
19 +{"t": 1793514314.9348638, "elapsed_ns": 1021290000, "cpu_mW": 17336.9, "gpu_mW": 60122.0, "ane_mW": 0.0, "combined_mW": 77458.9, "thermal_pressure": "Nominal"}
20 +{"t": 1793514315.955905, "elapsed_ns": 1021708666, "cpu_mW": 14308.4, "gpu_mW": 21942.7, "ane_mW": 0.0, "combined_mW": 36251.0, "thermal_pressure": "Nominal"}
21 +{"t": 1793514316.977277, "elapsed_ns": 1021247958, "cpu_mW": 6331.47, "gpu_mW": 59.7308, "ane_mW": 0.0, "combined_mW": 6391.2, "thermal_pressure": "Nominal"}
22 +{"t": 1793514317.999382, "elapsed_ns": 1021032541, "cpu_mW": 5807.85, "gpu_mW": 50.9288, "ane_mW": 0.0, "combined_mW": 5858.78, "thermal_pressure": "Nominal"}
23 +{"t": 1793514319.020412, "elapsed_ns": 1021403958, "cpu_mW": 8238.66, "gpu_mW": 47.9732, "ane_mW": 0.0, "combined_mW": 8286.63, "thermal_pressure": "Nominal"}
24 +{"t": 1793514321.062876, "elapsed_ns": 1021503916, "cpu_mW": 9368.54, "gpu_mW": 61.6738, "ane_mW": 0.0, "combined_mW": 9430.21, "thermal_pressure": "Nominal"}
25 +{"t": 1793514321.0646698, "elapsed_ns": 1021483583, "cpu_mW": 9745.63, "gpu_mW": 60.696, "ane_mW": 0.0, "combined_mW": 9806.32, "thermal_pressure": "Nominal"}
26 +{"t": 1793514322.0847821, "elapsed_ns": 1020966958, "cpu_mW": 12243.3, "gpu_mW": 123.412, "ane_mW": 0.0, "combined_mW": 12366.7, "thermal_pressure": "Nominal"}
27 +{"t": 1793514323.106275, "elapsed_ns": 1021548249, "cpu_mW": 17042.8, "gpu_mW": 96.9117, "ane_mW": 0.0, "combined_mW": 17139.7, "thermal_pressure": "Nominal"}
28 +{"t": 1793514324.127588, "elapsed_ns": 1021649124, "cpu_mW": 18386.0, "gpu_mW": 2853.23, "ane_mW": 0.0, "combined_mW": 21239.2, "thermal_pressure": "Nominal"}
29 +{"t": 1793514325.148823, "elapsed_ns": 1021652625, "cpu_mW": 17275.0, "gpu_mW": 29309.4, "ane_mW": 0.0, "combined_mW": 46584.3, "thermal_pressure": "Nominal"}
30 +{"t": 1793514326.170733, "elapsed_ns": 1021347041, "cpu_mW": 13658.4, "gpu_mW": 168.405, "ane_mW": 0.0, "combined_mW": 13826.8, "thermal_pressure": "Nominal"}
31 +{"t": 1793514327.1920068, "elapsed_ns": 1021359833, "cpu_mW": 15589.0, "gpu_mW": 61123.4, "ane_mW": 0.0, "combined_mW": 76712.4, "thermal_pressure": "Nominal"}
32 +{"t": 1793514329.235394, "elapsed_ns": 1021287333, "cpu_mW": 17349.7, "gpu_mW": 30531.1, "ane_mW": 0.0, "combined_mW": 47880.7, "thermal_pressure": "Nominal"}
33 +{"t": 1793514329.237072, "elapsed_ns": 1021679500, "cpu_mW": 16122.5, "gpu_mW": 30605.5, "ane_mW": 0.0, "combined_mW": 46728.0, "thermal_pressure": "Nominal"}
34 +{"t": 1793514330.25682, "elapsed_ns": 1021666166, "cpu_mW": 15308.3, "gpu_mW": 31092.3, "ane_mW": 0.0, "combined_mW": 46400.7, "thermal_pressure": "Nominal"}
35 +{"t": 1793514331.278759, "elapsed_ns": 1021706583, "cpu_mW": 15422.2, "gpu_mW": 31004.0, "ane_mW": 0.0, "combined_mW": 46426.2, "thermal_pressure": "Nominal"}
36 +{"t": 1793514332.299674, "elapsed_ns": 1021573166, "cpu_mW": 15170.7, "gpu_mW": 31157.8, "ane_mW": 0.0, "combined_mW": 46328.5, "thermal_pressure": "Nominal"}
37 +{"t": 1793514333.3209531, "elapsed_ns": 1021180500, "cpu_mW": 17384.8, "gpu_mW": 29319.0, "ane_mW": 0.0, "combined_mW": 46703.8, "thermal_pressure": "Nominal"}
38 +{"t": 1793514334.343148, "elapsed_ns": 1021683166, "cpu_mW": 7718.64, "gpu_mW": 564.754, "ane_mW": 0.0, "combined_mW": 8283.39, "thermal_pressure": "Nominal"}
modified benchmarks/scripts/bench_decode.py +3 −2
@@ -33,7 +33,7 @@ def env_record():
33 33
34 34 def run_one(model, tok, prompt, max_tokens):
35 35 """Return timing record for one greedy generation."""
36 − t0 = time.perf_counter()
36 + t0 = time.perf_counter(); t_start_epoch = time.time()
37 37 first = None
38 38 stamps = []
39 39 text = []
@@ -52,7 +52,8 @@ def run_one(model, tok, prompt, max_tokens):
52 52 "ttft_ms": (first - t0) * 1e3, "prompt_tps": last.prompt_tps, "gen_tps": last.generation_tps,
53 53 "decode_tps_wall": (len(stamps) - 1) / (stamps[-1] - stamps[0]) if len(stamps) > 1 else None,
54 54 "itl_ms_p50": pct(0.5), "itl_ms_p95": pct(0.95), "itl_ms_p99": pct(0.99), "itl_ms_mean": statistics.mean(itl) if itl else None,
55 − "total_s": stamps[-1] - t0, "peak_mem_gb": last.peak_memory, "text": "".join(text)[:200]}
55 + "total_s": stamps[-1] - t0, "peak_mem_gb": last.peak_memory, "text": "".join(text)[:200],
56 + "t_start_epoch": t_start_epoch, "t_end_epoch": time.time(), "t_first_epoch": t_start_epoch + (first - t0)}
56 57
57 58
58 59 def main():
added benchmarks/scripts/run_energy_laptop.sh +21 −0
@@ -0,0 +1,21 @@
1 +#!/bin/bash
2 +# H5 first evidence (single node): powermetrics package power sampled at 1 s while bench_decode runs four configurations.
3 +cd "$(dirname "$0")/../.."
4 +M=$(ls -d ~/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/*)
5 +OUT=benchmarks/raw/gateG/energy_laptop; mkdir -p $OUT
6 +echo "$SUDO_PW" | sudo -S -v 2>/dev/null
7 +sudo -n powermetrics --samplers cpu_power,gpu_power,thermal -i 1000 -f plist 2>/dev/null | .venv/bin/python benchmarks/scripts/power_sampler.py --out $OUT/power.jsonl &
8 +PM=$!
9 +sleep 8 # idle baseline
10 +.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend mlx_lm --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out $OUT/bench.jsonl 2>&1 | grep -v Fetching | tail -1
11 +sleep 5
12 +.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 64 --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out $OUT/bench.jsonl 2>&1 | grep -v Fetching | tail -1
13 +sleep 5
14 +echo "$SUDO_PW" | sudo -S purge 2>/dev/null; sleep 3
15 +.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 16 --nocache --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out $OUT/bench.jsonl 2>&1 | grep -v Fetching | tail -1
16 +sleep 5
17 +echo "$SUDO_PW" | sudo -S purge 2>/dev/null; sleep 3
18 +.venv/bin/python benchmarks/scripts/bench_decode.py --model $M --backend titan --capacity 16 --nocache --predictor probe --m 8 --traces benchmarks/raw/gateC/traces_olmoe_v2_laptop --families general,code --split heldout --repeats 3 --max-tokens 256 --tag energy --out $OUT/bench.jsonl 2>&1 | grep -v Fetching | tail -1
19 +sleep 8 # idle tail
20 +sudo -n pkill powermetrics; kill $PM 2>/dev/null
21 +echo ENERGY-DONE
added scripts/analyze_energy.py +44 −0
@@ -0,0 +1,44 @@
1 +#!/usr/bin/env python3
2 +"""Joules per generated token from a powermetrics sample file and the benchmark records that carry epoch timestamps.
3 +Energy of a request = trapezoid integral of combined package power over [t_first_token, t_end] (decode phase) minus the idle
4 +baseline power measured before the first run (so that only the incremental energy of decoding is attributed to tokens);
5 +both gross and net values are reported. Writes benchmarks/processed/energy_<tag>.json and paper/tables/energy.tex."""
6 +import json, os, statistics, sys
7 +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "benchmarks", "scripts"))
8 +from power_sampler import energy_joules
9 +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
10 +
11 +def main(d="benchmarks/raw/gateG/energy_laptop", tag="laptop"):
12 + d = os.path.join(ROOT, d); pw = os.path.join(d, "power.jsonl"); bj = os.path.join(d, "bench.jsonl")
13 + pts = [json.loads(l) for l in open(pw)]; recs = [json.loads(l) for l in open(bj) if l.strip()]
14 + t_first_run = min(r["t_start_epoch"] for r in recs)
15 + idle = [p["combined_mW"] for p in pts if p["t"] < t_first_run - 1 and p.get("combined_mW") is not None]
16 + idle_mW = statistics.median(idle) if idle else None
17 + groups = {}
18 + for r in recs:
19 + if r.get("warmup"): continue
20 + k = (r["backend"], r.get("capacity"), r.get("predictor") or "none", bool(r.get("nocache")))
21 + e = energy_joules(pw, r["t_first_epoch"], r["t_end_epoch"])
22 + if e is None: continue
23 + dur = r["t_end_epoch"] - r["t_first_epoch"]; n = r["gen_tokens"]
24 + groups.setdefault(k, []).append({"J_per_token_gross": e / n, "J_per_token_net": (e - idle_mW * 1e-3 * dur) / n if idle_mW else None,
25 + "mean_W": e / dur, "tok_s": r["gen_tps"], "n": n})
26 + out = {"idle_mW": idle_mW, "samples": len(pts), "configs": []}
27 + for k, v in groups.items():
28 + out["configs"].append({"backend": k[0], "capacity": k[1], "predictor": k[2], "cold": k[3], "runs": len(v),
29 + "J_per_token_gross_median": statistics.median(x["J_per_token_gross"] for x in v),
30 + "J_per_token_net_median": statistics.median(x["J_per_token_net"] for x in v) if idle_mW else None,
31 + "mean_W_median": statistics.median(x["mean_W"] for x in v), "tok_s_median": statistics.median(x["tok_s"] for x in v)})
32 + os.makedirs(os.path.join(ROOT, "benchmarks/processed"), exist_ok=True)
33 + json.dump(out, open(os.path.join(ROOT, f"benchmarks/processed/energy_{tag}.json"), "w"), indent=1)
34 + L = ["\\begin{table}[t]", "\\centering\\small", f"\\caption{{Energy per generated token on the laptop (M5 Max), OLMoE 4-bit, 256-token greedy decode, held-out general/code prompts; package power from \\code{{powermetrics}} at 1 s; idle baseline {idle_mW/1e3:.1f} W. Source: \\code{{benchmarks/raw/gateG/energy\\_laptop/}}.}}", "\\label{tab:energy}",
35 + "\\begin{tabular}{llrrrrr}", "\\toprule", "Backend & residency & runs & tok/s & mean W & J/token (gross) & J/token (net of idle) \\\\", "\\midrule"]
36 + for c in sorted(out["configs"], key=lambda c: -c["tok_s_median"]):
37 + res = "all resident" if c["backend"] == "mlx_lm" or (c["capacity"] or 0) >= 64 else f"{c['capacity']} slots{', cold NVMe' if c['cold'] else ''}{', ' + c['predictor'] if c['predictor'] != 'none' else ''}"
38 + L.append(f"{c['backend'].replace('_', '-')} & {res} & {c['runs']} & {c['tok_s_median']:.1f} & {c['mean_W_median']:.1f} & {c['J_per_token_gross_median']:.3f} & {c['J_per_token_net_median']:.3f} \\\\")
39 + L += ["\\bottomrule", "\\end{tabular}", "\\end{table}"]
40 + open(os.path.join(ROOT, "paper/tables/energy.tex"), "w").write("\n".join(L) + "\n")
41 + print(json.dumps(out, indent=1))
42 +
43 +if __name__ == "__main__":
44 + main(*sys.argv[1:])
45
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere