DeltaMoE: Memory-Efficient Inference for Merged Mixture of Experts with Delta Compression

Boyko Borisov, Xiaozhe Yao, Nezihe Merve Gürel, Ana Klimovic

Track: long paper (up to 4 pages)

Keywords: Sparse Mixture of Experts, Compression, Structured Sparsity, Quantization, Model Serving

TL;DR: We propose DeltaMoE, a training-free data-efficient model compression and fast serving pipeline.

Abstract: Sparse Mixture of Experts (SMoEs) have emerged as an efficient architecture for large language models. While recent community efforts have focused on merging multiple models to create SMoEs, deploying these merged models remains challenging due to their substantial memory requirements. In this paper, we present DeltaMoE, a training-free delta compression pipeline that enables efficient deployment of SMoE models through structured sparsity and quantization. Our evaluation shows that DeltaMoE achieves up to a $2.34\times$ compression ratio and $2.57\times$ throughput improvement. DeltaMoE is also scalable with the number of experts, making it particularly suitable for large SMoE models.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 39