Finding lost DG: Explaining domain generalization via model complexity

Da Li, Henry Gouk, Timothy Hospedales

Keywords: domain generalisation, rademacher complexity

Abstract:

The domain generalization (DG) problem setting challenges a model trained on multiple known data distributions to generalise well on unseen data distributions. Due to its practical importance, a large number of methods have been proposed to address this challenge. However most of this work is empirical, as the DG problem is hard to model formally; and recent evaluations have cast doubt on existing methods’ practical efficacy -- in particular compared to a well chosen empirical risk minimisation baseline.
We present a novel learning-theoretic generalisation bound for DG that bounds novel domain generalisation performance in terms of the model’s Rademacher complexity. Based on this, we conjecture that the causal factor behind existing methods’ efficacy or lack thereof is a variant of the standard empirical risk-predictor complexity tradeoff, and demonstrate that their performance variability can be explained in these terms. Algorithmically, this analysis suggests that domain generalisation should be achieved by simply performing regularised ERM with a leave-one-domain-out cross-validation objective. Empirical results on the DomainBed benchmark corroborate this.

One-sentence Summary: We derive generalisation bounds for the Domain Generalisation problem setting, and then provide experimental evidence that existing DG approaches work because they implicitly control model capacity.