Towards Real-world Debiasing: Rethinking Evaluation, Challenge, and Solution

ICLR 2026 Conference Submission25195 Authors

Keywords: spurious correlation, dataset bias, debias

TL;DR: In this work, we revisit the task of debiasing under real-world scenarios, proposing systematic evaluation framework, challenges, and solutions for real-world debiasing.

Abstract: Spurious correlations in training data significantly hinder the generalization capability of machine learning models when faced with distribution shifts, leading to the proposition of numberous debiasing methods. However, it remains to be asked: Do existing benchmarks for debiasing really represent biases in the real world? Recent works attempt to address such concerns by sampling from real-world data (instead of synthesizing) according to some predefined biased distributions to ensure the realism of individual samples. However, the realism of the biased distribution is more critical yet challenging and underexplored due to the complexity of real-world bias distributions. To tackle the problem, we propose a fine-grained framework for analyzing biased distributions, based on which we empirically and theoretically identify key characteristics of biased distributions in the real world that are poorly represented by existing benchmarks. Towards applicable debiasing in the real world, we further introduce two novel real-world-inspired biases to bridge this gap and build a systematic evaluation framework for real-world debiasing, RDBench. Furthermore, focusing on the practical setting of debiasing w/o bias label, we find real-world biases pose a novel Sparse bias capturing challenge to the existing paradigm. We propose a simple yet effective approach named Debias in Destruction (DiD), to address the challenge, whose effectiveness is validated with extensive experiments on 8 datasets of various biased distributions.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 25195