Towards Understanding Feature Learning in Out-of-Distribution Generalization

ICML 2023 Workshop SCIS Submission29 Authors

Keywords: Feature Learning, Out-of-Distribution Generalization, Invariant Learning

TL;DR: We show ERM learns both invariant and spurious features and propose a new algorithm to learn richer features than ERM for facilitating OOD generalization.

Abstract:

A common explanation for the failure of out-of-distribution (OOD) generalization is that the model trained with empirical risk minimization
(ERM) learns spurious features instead of invariant features. However, several recent studies challenged this explanation and found that deep networks may have already learned sufficiently good features for OOD generalization. Despite the contradictions at first glance, we theoretically show that ERM essentially learns both spurious and invariant features, while ERM tends to learn spurious features faster if the spurious correlation is stronger. Moreover, when fed the ERM learned features to the OOD objectives, the invariant feature learning quality significantly affects the final OOD performance, as OOD objectives rarely learn new features. Therefore, ERM feature learning can be a bottleneck to OOD generalization. To alleviate the reliance, we propose Feature Augmented Training (FAT), to enforce the model to learn richer features ready for OOD generalization. FAT iteratively augments the model to learn new features while retaining the already learned
features. In each round, the retention and augmentation operations are performed on different subsets of the training data that capture distinct features. Extensive experiments show that FAT effectively learns richer features thus boosting the performance of various OOD objectives

Submission Number: 29