XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference

Daniil Vankov, Nikita Ivkin, Jaime Campos Salas, Kyle R. Ulrich, Xiang song, Ashish Khetan, George Karypis

Keywords: Mixture of Expert, pruning, speculative decoding, inference optimization

TL;DR: Online batch-aware greedy expert selection for MoE inference

Abstract: Mixture-of-Experts (MoE) architectures are increasingly used to efficiently scale large language models. However, in production inference, request batching and speculative decoding significantly amplify expert activation, eroding these efficiency benefits. We address this issue by modeling batch-aware expert selection as a modular optimization problem and designing efficient greedy algorithms for different deployment settings. The proposed method, namely XShare, requires no retraining and dynamically adapts to each batch by maximizing the total gating score of selected experts. It delivers end-to-end throughput improvements of 10-37% across diverse deployment scenarios (single- or mixed-batch, with or without speculative decoding) while preserving baseline accuracy. On mixed-batch workloads, our method Pareto-dominates the baseline on both throughput and accuracy.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 93