$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts

ACL ARR 2025 February Submission6383 Authors

Abstract:

The Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance.
In conventional MoE, each expert is treated as entirely independent, and experts are combined in a discrete space. As a result, when the number of experts increases, it becomes difficult to train each expert effectively.
To stabilize training while increasing the number of experts, we propose $\infty$-MoE that selects a portion of the parameters of large FFNs based on continuous values sampled for each token.
By considering experts in a continuous space, this approach allows for an infinite number of experts while maintaining computational efficiency. Experiments show that a GPT-2 Small-based $\infty$-MoE model, with 129M active and 186M total parameters, achieves comparable performance to a dense GPT-2 Medium with 350M parameters. Adjusting the number of sampled experts at inference time allows for a flexible trade-off between accuracy and speed, with an improvement of up to 2.5\% in accuracy over conventional MoE.

Paper Type: Short

Research Area: Language Modeling

Research Area Keywords: Mixture of Experts, Sparse Routing, Transformer Architectures

Contribution Types: Approaches low compute settings-efficiency, Theory

Languages Studied: English

Submission Number: 6383