Simplicity bias leads to amplified performance disparities

Samuel Bell, Levent Sagun

Keywords: fairness, model bias, dataset bias, bias amplification, simplicity bias

Abstract:

The simple idea that not all things are equally difficult has surprising implications when applied in a fairness context.
In this work we explore how "difficulty" is model-specific, such that different models find different parts of a dataset challenging.
When difficulty correlates with group information, we term this difficulty disparity.
Drawing a connection with recent work exploring the inductive bias towards simplicity of SGD-trained models, we show that when such a disparity exists, it is further amplified by commonly-used models.
We quantify this amplification factor across a range of settings aiming towards a fuller understanding of the role of model bias. We also present a challenge to the simplifying assumption that ``fixing'' a dataset is sufficient to ensure unbiased performance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Social Aspects of Machine Learning (eg, AI safety, fairness, privacy, interpretability, human-AI interaction, ethics)

TL;DR: We introduce difficulty disparity and difficulty amplification, where a model's bias towards simplicity results in disparate performance between groups.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2212.06641/code)