Unraveling the Complexities of Simplicity Bias: Mitigating and Amplifying Factors

Published: 07 Nov 2023, Last Modified: 15 Dec 2023M3L 2023 PosterEveryoneRevisionsBibTeX
Keywords: simplicity bias, generalization, machine learning, implicit bias
Abstract: The success of neural networks depends on the generalization ability, while Shah et al. conclude that the inherent bias towards simplistic features, a phenomenon called *Simplicity Bias*, hurts generalization by preferring simple but noisy features to complex yet predictive ones. We aim to understand the scenarios when simplicity bias occurs more severely and the factors that help mitigate its effects. We show that many traditional insights such as increasing training size and increasing informative feature dimensions are not as effective as balancing the modes of our data distribution, distorting the simplistic features, or even searching for a good initialization. Our empirical results reveal intriguing factors of simplicity bias, and we call for future investigations to a more thorough understanding of simplicity bias and its interplay with the related fields.
Submission Number: 83