1
Fetching 6 files: 0%| | 0/6 [00:00<?, ?it/s]
Fetching 6 files: 100%|██████████| 6/6 [00:00<00:00, 5124.38it/s]2==========3Mixture-of-experts models are memory-bound during decoding because they often involve processing large datasets that exceed the available GPU memory, leading to paging or swapping to disk, which significantly slows down computation. Each expert model may require significant model parameters and intermediate data storage, exacerbating memory constraints. Additionally, the frequent loading and unloading of models from memory during inference can introduce overhead, further limiting performance.4==========5Prompt: 33 tokens, 45.049 tokens-per-sec6Generation: 85 tokens, 493.904 tokens-per-sec7Peak memory: 4.110 GB8