DAIR: Data Augmented Invariant Regularization

Tianjian Huang, Shaunak Ashish Halbe, Chinnadhurai Sankar, Pooyan Amini, Satwik Kottur, Alborz Geramifard, Meisam Razaviyayn, Ahmad Beirami

Keywords: data augmentation, domain shift, adversarial training

Abstract: While deep learning through empirical risk minimization (ERM) has succeeded at achieving human-level performance at a variety of complex tasks, ERM generalizes poorly to distribution shift. This is partly explained by overfitting to spurious features such as background in images or named entities in natural language. Synthetic data augmentation followed by empirical risk minimization (DA-ERM) is a simple yet powerful solution to remedy this problem. In this paper, we propose data augmented invariant regularization (DAIR). The idea of DAIR is based on the observation that the model performance (loss) is desired to be consistent on the augmented sample and the original one. DAIR introduces a regularizer on DA-ERM to penalize such loss inconsistency. Both theoretically and through empirical experiments, we show that a particular form of the DAIR regularizer consistently performs well in a variety of settings. We prove convergence guarantees for DAIR. We apply it to multiple real-world unsupervised and supervised learning problems involving domain shift. Our experiments show that DAIR consistently outperforms ERM and DA-ERM with little marginal cost. Furthermore, DAIR is competitive with state-of-the-art methods specifically designed for these problems.

One-sentence Summary: We propose a simple regularization technique for imposing invariance, that is competitive with state-of-the-art problem-specific techniques in a variety of problem setups including domain shift and adversarial learning.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2110.11205/code)