Probing Semantic Routing in Large Mixture-of-Expert Models

ACL ARR 2025 May Submission4729 Authors

Abstract:

The past year has seen an increase in the number of large ($>100B$ parameter) mixture-of-expert (MoE) models in the open domain. With the increase in size has come an increase in capabilities, with the largest models achieving new state-of-the-art on mathematics and reasoning benchmarks.
Although the benefit of the MoE architecture can be understood through an engineering efficiency lens, prior work has also investigated whether MoE models exhibit functional differentiation along architectural lines, by examining patterns in the routing behavior of MoE modules.
This work investigates whether expert routing in large MoE models is related to the \textit{semantics} of the inputs. We develop a pair of controlled experiments to identify whether routing occurs on the basis of \textit{semantics}, i.e. the meaning of the inputs.
The first experiment compares the activations on lexically identical but semantically distinct inputs, by using pairs of sentences where a common target word either has the same meaning or not. The second experiment compares activations where context is kept constant but the target word is substituted for semantically equivalent/distinct words. By comparing the rate of overlap in expert routing between sentence pairs where meaning is kept the same versus sentence pairs where meaning is changed, we identify whether expert routing is made on the basis of semantics.
Our experiments show clear and statistically significant evidence of \textit{semantic routing} in large MoE models; inputs with the same meaning have greater overlap in routed experts than inputs with differing meaning.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: model architectures, polysemy

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 4729