SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models

Juntong Wu, Jialiang Cheng, Fuyu Lv, OU Dan, Li Yuan

Keywords: MoE Model, Inference Acceleration, Batch Decoding, Expert Re-routing

TL;DR: We propose SERE, a Similarity-based Expert Re-routing method for Efficient batch decoding in MoE models.

Abstract: Mixture-of-Experts (MoE) architectures employ sparse activation to deliver faster training and inference with higher accuracy than dense LLMs. However, in production serving, MoE models require batch inference to optimize hardware efficiency, which may cause excessive expert activation and thus slow the memory-bound decoding stage. To address the fundamental tension between batch decoding and expert sparsity, we present **SERE**, a **S**imilarity-based **E**xpert **R**e-routing method for **E**fficient batch decoding in MoE models. SERE dynamically reduces the number of active experts in an input-aware manner by re-routing tokens from secondary experts to their most similar primary counterparts. It also leverages similarity patterns to identify and preserve critical experts, thereby preventing capability loss. Notably, SERE avoids static expert pruning or merging, instead enabling dynamic expert skipping based on batch-level expert redundancy. Additionally, we provide an efficient custom CUDA kernel for SERE, enabling plug-and-play use in vLLM with only a single-line code change. Extensive experiments on various complex reasoning benchmarks demonstrate that SERE achieves up to $2.0\times$ speedup with minimal quality loss, providing a practical solution for cost-efficient and latency-sensitive large-scale MoE deployment.

Primary Area: infrastructure, software libraries, hardware, systems, etc.

Submission Number: 2060