OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?

ICLR 2025 Conference Submission1304 Authors

Keywords: OOD generalization, distribution shifts, algorithm selection, learning to learn

TL;DR: Choosing the right learning algorithm for the right dataset in OOD generalization is crucial but elusive. We learn to a priori predict which algorithm could produce the most robust model.

Abstract:

Out-of-distribution (OOD) generalization is challenging because distribution shifts come in many forms. A multitude of learning algorithms exist and each can improve performance in *specific* OOD situations. We posit that much of the challenge of OOD generalization lies in *choosing the right algorithm for the right dataset*. However, such algorithm selection is often elusive under complex real-world shifts. In this work, we formalize the task of *algorithm selection for OOD generalization* and investigate whether it could be approached by learning.
We propose a solution, dubbed OOD-Chameleon that formulates the task as a supervised classification over candidate algorithms. We construct a *dataset of datasets* to learn from, which represents diverse types, magnitudes and combinations of shifts (covariate shift, label shift, spurious correlations). We train the model to predict the relative performance of algorithms given a dataset's characteristics. This enables *a priori* selection of the best learning strategy, i.e. without training various models as needed with traditional model selection.
Our experiments show that the adaptive selection outperforms any individual algorithm and simple selection heuristics, on unseen datasets of controllable and realistic image data. Inspecting the model shows that it learns non-trivial data/algorithms interactions, and reveals the conditions for any one algorithm to surpass another. This opens new avenues for (1) enhancing OOD generalization with existing algorithms, and (2) gaining insights into the applicability of existing algorithms with respect to datasets' properties.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1304