Group-Label-Free Validation for Spuriously Correlated Data

Yawen Zhao, Nan Ye, Miao Xu

Keywords: Spurious correlations; model selection; robustness

TL;DR: We propose SIEVE, a plug-and-play method that constructs group-aware validation sets without any group labels, enabling robust model selection under spurious correlations.

Abstract:

Deep learning models are known to be sensitive to spurious correlations between irrelevant features and the labels.
These spurious features can negatively affect the model’s generalization and robustness, particularly for groups consisting of examples without spurious correlations.
Early approaches address this issue by requiring group labels in the training set, and more recent methods aim to reduce reliance on group labels in training; however, many state-of-the-art approaches still require a validation set with group annotations for hyperparameter tuning or model selection, which are often unavailable or costly to obtain.
In this work, we propose SIEVE, a plug-and-play module that constructs a group-aware validation set for robust model evaluation under spurious correlations, without using any group annotations. SIEVE identifies confusing training examples based on feature-space similarity, and iteratively separates them into spurious and non-spurious subsets based on differing loss dynamics patterns, which we discovered in our data analysis.
The selected samples are assigned pseudo group labels and used as a surrogate validation set for model selection. Our method is annotation-efficient, easy to implement, and compatible with existing methods that rely on group-labeled validation sets for hyperparameter tuning and model selection. Experiments on benchmark datasets demonstrate that SIEVE enables robust model selection without access to group labels, achieving performance competitive with methods that use true group annotations.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 6979