The Limits of Fairness Gains Under Scaling in Vision Models

ICLR 2026 Conference Submission21122 Authors

Keywords: Scaling Laws, fairness, vision models

Abstract: Recent advances in computer vision indicate that increasing dataset size and model parameters substantially enhance model performance. Scaling laws derived from these observations provide valuable guidance for the design and optimization of large vision models. However, the impact of scaling on fairness within these models has yet to be systematically investigated. Here we empirically show that scaling model parameters and dataset size can improve fairness for certain protected attributes in downstream tasks. Our results demonstrate that, when using a loss function that jointly optimizes for utility and fairness, there exists a critical threshold in scaling beyond which fairness gains plateau. While scaling enhances fairness for some attributes, it does not eliminate disparities. These results emphasize that fairness in vision models requires more than scaling. Fairness techniques must be incorporated early in model development to address structural disparities and improve outcomes for all groups. This is especially crucial in sensitive domains such as medical imaging, where achieving equal representation and unbiased performance across diverse populations is essential for ethical and effective deployment.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 21122