Can Variance-Based Regularization Improve Domain Generalization?

Chuanlong Xie, Ruichen Li, Qishi Dong, Liwei Wang, Zhenguo Li

Keywords: Variance-Based Regularization, Domain Generalization, Robustness

Abstract: If there is no prior information, domain generalization with only access to multi-domain training data relies on guessing what the test data is. In this work, we consider mild assumptions that there is a distribution over domains and the out-of-distribution data is generated by the shift of the domain distribution. We study a domain-level variance-based regularizer. We show that the variance-regularized method can locally approximate the group distributionally robust optimization and embed the local information into the objective function as a weighting scheme. By taking the empirical domain distribution as an anchor of the location, we propose a weighting correction scheme and provide theoretical guarantees of in-distribution generalization. Compared to the Empirical Risk Minimization, we prove the potential benefits of our proposed method but do not observe consistent improvements in general.