Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift
Keywords: meta-learning, distribution shift, distributional robustness, test time adaptation
Abstract: A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution. However, this assumption is violated in almost all practical applications: machine learning systems are regularly tested under distribution shift, due to temporal correlations, particular end users, or other factors. In this work, we consider the setting where the training data are structured into groups and test time shifts correspond to changes in the group distribution. Prior work has approached this problem by attempting to be robust to all possible test time distributions, which may degrade average performance. In contrast, we propose to use ideas from meta-learning to learn models that are adaptable, such that they can adapt to shift at test time using a batch of unlabeled test points. We acquire such models by learning to adapt to training batches sampled according to different distributions, which simulate structural shifts that may occur at test time. Our primary contribution is to introduce the framework of adaptive risk minimization (ARM), a formalization of this setting that lends itself to meta-learning. We develop meta-learning methods for solving the ARM problem, and compared to a variety of prior methods, these methods provide substantial gains on image classification problems in the presence of shift.
One-sentence Summary: We meta-learn models that can use unlabeled test data to adapt to group distribution shift.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2007.02931/code)
Reviewed Version (pdf): https://openreview.net/references/pdf?id=QCjTAFdCt2
13 Replies
Loading