Fishr: Invariant Gradient Variances for Out-of-distribution Generalization

Alexandre Rame, Corentin Dancette, Matthieu Cord

Keywords: Deep Learning, Computer Vision, Domain Generalization

Abstract: Learning robust models that generalize well under changes in the data distribution is critical for real-world applications. To this end, there has been a growing surge of interest to learn simultaneously from multiple training domains - while enforcing different types of invariance across those domains. Yet, all existing approaches fail to show systematic benefits under controlled evaluation protocols. In this paper, we introduce a new regularization - named Fishr - that enforces domain invariance in the space of the gradients of the loss: specifically, the domain-level variances of gradients are matched across training domains. Our approach is based on the close relations between the gradient covariance, the Fisher Information and the Hessian of the loss: in particular, we show that Fishr eventually aligns the domain-level loss landscapes locally around the final weights. Extensive experiments demonstrate the effectiveness of Fishr for out-of-distribution generalization. Notably, Fishr improves the state of the art on the DomainBed benchmark and performs consistently better than Empirical Risk Minimization. Our code is available anonymously at https://anonymous.4open.science/r/fishr-anonymous-EBB6/.

One-sentence Summary: To improve the generalization properties of deep networks when data comes from multiple domains, we introduce a new invariance criterion that matches the domain-level variances of gradients.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2109.02934/code)