Bias as a Virtue: Rethinking Generalization under Distribution Shifts

Ruixuan Chen, Wentao Li, Jiahui Xiao, Yuchen Li, Yimin Tang, Xiaonan Wang

Keywords: Distribution shifts, Out-of-distribution generalization, Bias-variance tradeoff, Model selection, Statistical diversity, Optimal transport, Adaptive Distribution Bridge

TL;DR: This paper demonstrates that strategically increasing in-distribution bias can improve out-of-distribution generalization, and introduces the Adaptive Distribution Bridge (ADB) framework to achieve this by managing training data diversity

Abstract: Machine learning models often degrade when deployed on data distributions different from their training data. Challenging conventional validation paradigms, we demonstrate that higher in-distribution (ID) bias can lead to better out-of-distribution (OOD) generalization. Our Adaptive Distribution Bridge (ADB) framework implements this insight by introducing controlled statistical diversity during training, enabling models to develop bias profiles that effectively generalize across distributions. Empirically, we observe a robust negative correlation where higher ID bias corresponds to lower OOD error—a finding that contradicts standard practices focused on minimizing validation error. Evaluation on multiple datasets shows our approach significantly improves OOD generalization. ADB achieves robust mean error reductions of up to 26.8% compared to traditional cross-validation, and consistently identifies high-performing training strategies, evidenced by percentile ranks often exceeding 83.4%. Our work provides both a practical method for improving generalization and a theoretical framework for reconsidering the role of bias in robust machine learning.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 22883