Sparse Mixture-of-Experts are Domain Generalizable LearnersDownload PDF

05 Oct 2022, 00:12 (modified: 08 Nov 2022, 08:17)NeurIPS 2022 Workshop DistShift PosterReaders: Everyone
Keywords: domain generalization, mixture-of-experts, algorithmic alignment, visual attributes
TL;DR: We theoretically investigate the impact of backbone architecture on DG. We propose a novel SOTA model Generalizable Mixture-of-Experts (GMoE) for DG.
Abstract: In domain generalization (DG), most existing methods focused on the loss function design. This paper proposes to explore an orthogonal direction, i.e., the design of the backbone architecture. It is motivated by an empirical finding that transformer-based models trained with empirical risk minimization (ERM) outperform CNN-based models employing state-of-the-art (SOTA) DG algorithms on multiple DG datasets. We develop a formal framework to characterize a network's robustness to distribution shifts by studying its architecture's alignment with the correlations in the dataset. This analysis guides us to propose a novel DG model built upon vision transformers, namely \emph{Generalizable Mixture-of-Experts (GMoE)}. Experiments on DomainBed demonstrate that GMoE trained with ERM outperforms SOTA DG baselines by a large margin.
1 Reply