In Search of Lost Domain Generalization

Ishaan Gulrajani, David Lopez-Paz

Keywords: domain generalization, reproducible research

Abstract:

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training.
While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions---datasets, network architectures, and model selection criteria---render fair comparisons difficult.
The goal of this paper is to understand how useful domain generalization algorithms are in realistic settings.
As a first step, we realize that model selection is non-trivial for domain generalization tasks, and we argue that algorithms without a model selection criterion remain incomplete.
Next we implement DomainBed, a testbed for domain generalization including seven benchmarks, fourteen algorithms, and three model selection criteria.
When conducting extensive experiments using DomainBed we find that when carefully implemented and tuned, ERM outperforms the state-of-the-art in terms of average performance.
Furthermore, no algorithm included in DomainBed outperforms ERM by more than one point when evaluated under the same experimental conditions.
We hope that the release of DomainBed, alongside contributions from fellow researchers, will streamline reproducible and rigorous advances in domain generalization.

One-sentence Summary: Our ERM baseline achieves state-of-the-art performance across many domain generalization benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ facebookresearch/DomainBed](https://github.com/facebookresearch/DomainBed) + [ 8 community implementations](https://paperswithcode.com/paper/?openreview=lQdXeXDoWtI)

Data: [Colored MNIST](https://paperswithcode.com/dataset/colored-mnist), [DomainNet](https://paperswithcode.com/dataset/domainnet), [Office-Home](https://paperswithcode.com/dataset/office-home), [PACS](https://paperswithcode.com/dataset/pacs)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2007.01434/code)