Emergent Mixture-of-Experts: Can Dense Pre-trained Transformers Benefit from Emergent Modular Structures?
Keywords: Modular networks, Mixture-of-Experts, Emergent Modularity
TL;DR: We validate transforming a pre-trained dense model into its modular counterpart can enjoy the benefits of modularity, like mitigating negative transfer, and effectively boosting ID and OOD generalization abilities.
Abstract: Incorporating modular designs into neural networks demonstrates superior out-of-generalization, learning efficiency, etc.
Existing modular neural networks are generally $\textit{explicit}$ because their modular architectures are pre-defined, and individual modules are expected to implement distinct functions.
Conversely, recent works reveal that there exist $\textit{implicit}$ modular structures in standard pre-trained transformers, namely $\textit{Emergent Modularity}$.
They indicate that such modular structures exhibit during the early pre-training phase and are totally spontaneous.
However, most transformers are still treated as monolithic models with their modular natures underutilized.
Therefore, given the excellent properties of explicit modular architecture, we explore $\textit{whether and how dense pre-trained transformers can benefit from emergent modular structures.}$
To study this question, we construct $\textbf{E}$mergent $\textbf{M}$ixture-$\textbf{o}$f-$\textbf{E}$xperts (EMoE).
Without introducing additional parameters, EMoE can be seen as the modular counterpart of the original model and can be effortlessly incorporated into downstream task tuning.
Extensive experiments on various downstream tasks (visions and languages) and models (from 22M to 1.5B) demonstrate that EMoE effectively boosts in-domain and out-of-domain generalization abilities.
Further analysis and ablation studies suggest that EMoE mitigates negative knowledge transfer and is robust to various configurations.
Supplementary Material: zip
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 1834
Loading