Consistency Regularization for Domain Generalization with Logit Attribution Matching

Han Gao, Kaican Li, Weiyan Xie, Zhi LIN, Yongxiang Huang, Luning Wang, Caleb Chen Cao, Nevin L. Zhang

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Domain generalization, Consistency regularization, Causal

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: In this paper, we show that CR can further improve DG performance on top of targeted DA and we propose a novel CR method called Logit Attribution Matching (LAM) that have multiple advantages over previous CR methods.

Abstract: Domain generalization (DG) is about training models that generalize well to unseen domains that follow different distributions than the training domains. It has recently been shown that an effective way to achieve good DG performance is targeted data augmentation, which randomizes spurious factors while preserving robustly predictive factors in training examples. Data augmentation (DA) naturally leads to paired training examples that share the same semantic contents, which can be utilized via consistency regularization (CR). In this paper, we show that CR can further boost DG performance on top of targeted DA. We also propose a novel CR-based DG method called Logit Attribution Matching (LAM). In comparison with previous CR-based DG methods, a key advantage of LAM is that it leverages class labels often associated with semantic sharing (SS) pairs. Empirically we find that LAM consistently outperforms previous CR-based DG methods on benchmarks with multiple classes. In fact, it is the only one that can consistently improve the model DG performance over the targeted DA on all evaluated datasets. To justify the CR-based approach to DG theoretically, we establish conditions for optimal DG in a causal framework and explain how CR is related to those conditions.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1705