HarmoMoE: Unifying Domain-Specialized Experts into a Mixture-of-Experts Model under Privacy Constraints

Weisen Jiang, Shuhao Chen, Sinno Jialin Pan

Keywords: Mixture of Experts, Privacy-Preserving Learning

Abstract: Mixture-of-Experts (MoE) models offer a powerful way to scale capacity, but existing designs typically assume centralized access to all training data. In many real-world scenarios, however, data is distributed across clients from different domains and cannot be shared due to privacy constraints, making it challenging to build a unified and generalizable MoE. We propose HarmoMoE, a framework that unifies domain-specialized experts into a single MoE without sharing private data. HarmoMoE combines relevance-weighted DPP proxy selection with a context-aware router, ensuring that experts trained on both private and proxy data remain compatible and effectively coordinated. Experiments on CV and NLP show that HarmoMoE consistently outperforms recent methods such as BTX and FlexOlmo.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 10257