Rich feature learning via diversification

XI LENG, Yongqiang Chen, Xiaoying Tang, Yatao Bian

Track: regular paper (up to 6 pages)

Keywords: Out-of-distribution generalization, rich feature learning

TL;DR: We explore Rich Feature Learning (RFL) in Out-of-Distribution tasks. We formally define feature richness and then propose a simple, effective RFL method, which we theoretially state its efficacy and then empirically validate it.

Abstract: Rich Feature Learning (RFL) aims to extract all beneficial features from the training distribution and has demonstrated significant efficacy in Out-of-Distribution (OOD) generalization. Despite its success, a precise and comprehensive definition of ``richness'' remains elusive. Through an in-depth analysis of RFL algorithms and empirical risk minimization (ERM), the standard OOD baseline, we identify feature diversity as the key differentiator driving RFL's superior OOD performance. Building on this insight, we formally define rich features as those that exhibit both high informativeness and diversity. Leveraging this foundation, we propose Diversity-fOunded Rich fEature lEarniNg (DOREEN), a simple yet highly effective RFL algorithm. We theoretically demonstrate that DOREEN not only realizes the benefits of RFL but also addresses the limitations of prior RFL algorithms. Extensive experiments validate that DOREEN learns richer features and consistently enhances OOD performance across various OOD objectives.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Format: Yes, the presenting author will attend in person if this work is accepted to the workshop.

Funding: Yes, the presenting author of this submission falls under ICLR’s funding aims, and funding would significantly impact their ability to attend the workshop in person.

Presenter: ~XI_LENG1

Submission Number: 24